Challenge in visualizing large coronavirus clusters in US data I'm trying to take the data on large coronavirus clusters in the US and visualize them to show the sizes and the different settings (prisons, healthcare facilities, etc).  I want to show the difference between the different settings.
If the sizes were more similar, I'd try to show a stacked bar chart (with size as horizontal axis and count as vertical axis).  Unfortunately, that's not working well because some clusters are much bigger than others.
The first few lines of my data look like (there are lots of aged care facilities with 50 cases):
size category
50 agedcare
50 agedcare
50 agedcare
50 agedcare
50 agedcare
50 agedcare
50 agedcare

and the bottom looks like  (the prisons and meat packing facilities have huge outbreaks)
931 prisons
981 prisons
1028 prisons
1031 meat
1051 prisons
1065 prisons
1098 meat
1107 prisons
1283 prisons
1362 prisons
1374 prisons
1791 prisons
2439 prisons

Here is a visualization of the smaller sizes
I can do some binning and I get this:

But it's still hard to immediately see that some of these setting types have small outbreaks while others have much larger ones.
Any suggestions on how to visualize would help (I use python primarily if that matters)

How about a small multiples style visualization? A nice 2 x 3 grid would cover the six categories here. Size on X, frequency on Y.
This approach is one of the clearest ways to show this kind of data. The stacked histogram is difficult to interpret because the bars of the same color do not have common starting points. If you make six histograms arranged in a rectangle with common scale on the Y, you can quickly visualize the distribution for each category.
Here is a plot made with some simulated data like yours. The top image has all six categories on one plot, like yours. The bottom image facets each category into its own histogram with common scale. You can much more easily compare the six categories.


Assessing Group Similarities and Dissimilarities Post PCA The goal is to assess similarity and dissimilarity between 6 known groups.

The original data began with the 6 known groups and 2,700+ variables all on a scale of 0 to 100.

I have performed PCA to reduce the 2700+ variables into 5 principal components using the dudi.pca function from the ade4 package in R. Here are the Eigenvalues for the components:

      eigenvalue variance.percent cumulative.variance.percent
Dim.1   998.3274        36.635867                    36.63587
Dim.2   670.1278        24.591848                    61.22771
Dim.3   482.2372        17.696776                    78.92449
Dim.4   352.2806        12.927728                    91.85222
Dim.5   222.0270         8.147781                   100.00000

I would now like to assess the distances between the 6 known groups. Is this done as simply as generating a distance matrix using each group's coordinates for each of the principal components? If so, I am leaning towards using Manhattan distance to get the absolute distance.

Here are the coordinates of each group:

           Dim.1       Dim.2       Dim.3        Dim.4        Dim.5
Group 1    69.019038    7.940190    0.4985599  - 6.847178     0.3964117
Group 2   -16.302322  -25.965373  -29.3084201  -23.013430     9.9183010
Group 3   -26.313850   50.159662    6.9486408  -10.713924     5.2883152
Group 4   -12.800767  -26.211432   39.5067264  - 8.775551   - 8.8840592
Group 5   - 9.228404    2.648632  -20.4297314   16.685426   -26.8559444
Group 6   - 4.373694   -8.571679    2.7842244   32.664657    20.1369757

If not, what would be the appropriate way to assess individual similarity/ dissimilarity post PCA?

What does the distance between groups mean for your problem? Answering that problem would help you pick a distance metric.

Assuming you pick e.g. Euclidean distance, one simple way to find inter-group distance would be to first calculate the group centroid, i.e. average location of each group by averaging the locations of each group together.  You can do this by just averaging all the 5x1 vectors belonging to a group, and repeat for each group.  Then calculate the Euclidean distance between the resulting 6 centroids, giving you a 6x6 distance matrix.


Training with less data Most problem with machine learning projects I have faced is the lack of data. The samples available are enough to disqualify rule based approach but not enough for a neural network to train.
For example, to train a neural network (or even fine tune a pretrained model) on a new entity in an NER system takes a thousands of different records. And the requirement of these thousands of records is to have enough variations to avoid overfitting.
Generally as human we can detect patterns by carefully observing the data, however it becomes humanly not possible to detect all patterns in inputs, and that is where deep learning comes into play of automatically detecting the patterns to make a hypothesis.
Now my question is, what are possible ways, using which limited data can be used to train a neural network with limited data. Let me add some inputs from my side, which I believe are not sufficient:

Data Augmentation : for images rotating, scaling and skewing. For textual data, repeating text with some masking and embedding/synonym replacement.
what else?

As rightly pointed out by @erwan, it is a bad idea to use data augmentation with 'text data'
The problem of 'training with less data' can be approached in many ways, here I enlist two ways which helped me with significant impact:
(a) One approach would be to use semi-supervised approach. There are open sourced language models trained on insanely massive datasets, that can be used to perform a specific task like custom NER or sentence classification. Transfer learning is more useful today then ever.
(b) Anyways if we want to go ahead with Data augmentation, 'Sentence‐Chain Based Seq2seq Model for Corpus Expansion' is one of the proven methods to proceed. Please find the link to paper here.
I will strongly recommend to experiment with BERT before moving to Corpus Expansion. For introduction to transfer learning, BERT etc. please visit here.


Train Accuracy vs Test Accuracy vs Confusion matrix After I developed my predictive model using Random Forest I get the following metrics:

        Train Accuracy ::  0.9764634601043997
        Test Accuracy  ::  0.7933284397683713
         Confusion matrix  [[28292  1474]
                            [ 6128   889]]

This is the results from this code:

  training_features, test_features, training_target, test_target, = train_test_split(df.drop(['bad_loans'], axis=1),
                                                  df['target'],
                                                  test_size = .3,
                                                  random_state=12)
clf = RandomForestClassifier()
trained_model = clf.fit(training_features, training_target)
trained_model.fit(training_features, training_target)
predictions = trained_model.predict(test_features)

Train Accuracy: accuracy_score(training_target, trained_model.predict(training_features))
Test Accuracy: accuracy_score(test_target, predictions)
Confusion Matrix: confusion_matrix(test_target, predictions)

However I'm getting a little confuse to interpret and explain this values.

What exactly this 3 measures tell me about my model?

Thanks!

Definitions

Accuracy: The amount of correct classifications / the total amount
of classifications.
The train accuracy: The accuracy of a model on examples it was constructed on.
The test accuracy is the accuracy of a model on examples it hasn't seen.
Confusion matrix: A tabulation of the predicted class (usually
vertically) against the actual class (thus horizontally).

Overfitting

What I would make up of your results is that your model is overfitting. You can tell that from  the large difference in accuracy between the test and train accuracy. Overfitting means that it learned rules  specifically for the train set, those rules do not generalize well beyond the train set.

Your confusion matrix tells us how much it is overfitting, because your largest class makes up over 90% of the population. Assuming that you test and train set have a similar distribution, any useful model would have to score more than 90% accuracy:  A simple 0R-model would. Your model scores just under 80% on the test set.

In depth look at the confusion matrix

If you would look at the confusion matrix relatively (in percentages) it would look like this:

               Actual    TOT
               1    2
Predicted 1 | 77% | 4% | 81%
Predicted 2 | 17% | 2% | 19%
TOT         | 94% | 6% |

You can infer from the total in the first row that your model predicts Class 1 81% of the time, while the actual occurrence of Class 1 is 94%. Hence your model is underestimating this class. It could be the case that it learned specific (complex) rules on the train set, that work against you in the test set.

It could also be worth noting that even though the false negatives of Class 1 (17%-point, row 2, column 1)) are hurting your overall performance most, the false negatives of Class 2 (4%-point, row 1 column 2) are actually more common with respect to the total population of the respective classes (94%, 6%). This means that your model is bad at predicting Class 1, but even worse at predicting Class 2. The accuracy just for Class 1 is 77/99 while the accuracy for Class 2 is 2/6.


Is ImageNet 1k a subset of ImageNet 22k There are 2 different ImageNet datasets: ImageNet 1k usually referred to in papers as just ImageNet and the full ImageNet dataset also called ImageNet 22k. Is the ImageNet 1k a subset of the 22k? And if so are the 1k categories represented the same way in the 22k version or maybe images of a certain class in the 1k version are aggregated from several different classes in the 22k version (like different dog breeds are merged into the "dog" class)? Can I somehow cut the 1k version from the 22k version and say the rest of the 22k version except the 1k version is a completely different dataset or even an out of distribution dataset for the 1k version? Any links to some trusted sources would be cool. Thanks

Yes, the full ImageNet is a superset of the 1k version (see Benchmarking Neural Network Robustness to Common Corruptions and Perturbations). The 1k synsets (a.k.a. classes) corresponding to the 1k version are also present in the full 22k version. So there are 21841 classes in total, 1000 of them are exactly the same as in the 1k version (and images are exactly the same), and the other 20841 can be treated as OOD to the 1k version dataset (at least they are treated so in one of the papers by Hendrycks et al, Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty.


Preprocessing in Data mining? I am still new to data mining but I really want (and need) to learn it so badly. I know that before I can actually process my data in softwares like WEKA, I need to do some filtering like cleaning the data, integrating, transforming, etc to actually get your data cleaned from any kind of duplicate, missing value, noise, etc. But I only know all of these theoretically.

The problem I have now is I have a very big set of data that I need to filter first before going on to the processing part. But I don't know where to start. Fyi, my dataset is very big that the usual spreadsheet programs like Ms Excel, Libre Office, WPS,etc can't open it. I have to use Linux terminal and commands to count the number of rows, columns etc.

What do I do in the preprocessing start? How do I 'clean' my data? I have been thinking to use Linux commands to do all of these, but I am also wondering how real data scientists clean their data. Do they do these manually or they already have some sort of software to help them? Because seriously I don't know where to start or to do. Every reference I found in the internet only explain things theoretically. ANd I need something more practical to help me understand.

What do I do with my dataset? Help please?

The preprocessing will depend on what your data is like: textual? numerical? If in whatever analysis you want to do the data is not allowed to have duplicates, you'll have to filter those out. If not, they'll get to stay, etc.
If you already know exactly what you want to do theoretically and just need to find some software, python pandas is very good. Load the data, clean it, group it, perform elementary analysis.


finding maximum depth of random forest given the number of features How do we find maximum depth of Random Forest if we know the number of features ?

This is needed for regularizing random forest classifier.

The maximum depth of a forest is a parameter which you set yourself. If you're asking how do you find the optimal depth of a tree given a set of features then this is through cross-validation. For example, create 5 rf's with 5 different tree depths and see which one performs the best on the validation set.


What is the "dying ReLU" problem in neural networks? Referring to the Stanford course notes on Convolutional Neural Networks for Visual Recognition, a paragraph says:

  "Unfortunately, ReLU units can be fragile during training and can
  "die". For example, a large gradient flowing through a ReLU neuron
  could cause the weights to update in such a way that the neuron will
  never activate on any datapoint again. If this happens, then the
  gradient flowing through the unit will forever be zero from that point
  on. That is, the ReLU units can irreversibly die during training since
  they can get knocked off the data manifold. For example, you may find
  that as much as 40% of your network can be "dead" (i.e. neurons that
  never activate across the entire training dataset) if the learning
  rate is set too high. With a proper setting of the learning rate this
  is less frequently an issue."

What does dying of neurons here mean?

Could you please provide an intuitive explanation in simpler terms.

A "dead" ReLU always outputs the same value (zero as it happens, but that is not important) for any input. Probably this is arrived at by learning a large negative bias term for its weights.

In turn, that means that it takes no role in discriminating between inputs. For classification, you could visualise this as a decision plane outside of all possible input data.

Once a ReLU ends up in this state, it is unlikely to recover, because the function gradient at 0 is also 0, so gradient descent learning will not alter the weights. "Leaky" ReLUs with a small positive gradient for negative inputs (y=0.01x when x < 0 say) are one attempt to address this issue and give a chance to recover.

The sigmoid and tanh neurons can suffer from similar problems as their values saturate, but there is always at least a small gradient allowing them to recover in the long term.


Detecting Offensive Text Content in English and German I am working on an NLP project about classifying offensive text data in social media. By offensive I especially mean threat words that one say to another.

Some examples:

"Stop doing this or else you will pay for it."
"Just wait until you see what's coming"
"I will break your legs next time I see you."

As an initial approach, I considered semantic and syntactic keyword matching. However, doing this on this very problem seemed harder because threating is an action and it is expressed in so many different ways.

My main goal is classifying text data by offensive/non-offensive text by using Machine Learning and Deep Learning algorithms. After weeks of online searching, I could not find a ready-to-use dataset. I considered manually labelling the data. However, I don't know where I should start.
What would be the best approach for making progress in this task? I also plan to do this in both English and German languages.
Also, below is a related article for fully understanding of the problem:
Deep learning for detecting inappropriate content in text

Toxic comment classification challenge might be a good place to start.  It contains a set of comments and 6 binary classifications indicting if it's a toxic comment and of which type.

I imagine this would be a sufficient start.


How can I include prior knowledge about monotonic features into a neural network? Let's make an example: I want to build a neural network which should predict if a person is obese. It gets the following features:

Weight
Height
Number of hours the person makes sports
superspecial health index
foobar

The output of the model is a probability that the given person is obese. The higher, the more likely.

Now I know, if everything else stays the same, that a higher weight should always be the same or a higher output probability. But I don't know the exact relationship, only that it changes monothonically.

How can I include this knowledge into the network and force it to show the desired behaviour?

If you plug this kind of data into a standard network, e.g. an MLP, you will usually hope that the model actually extracts this information itself. You could introduce a dummy variable that encodes this information, but you run the risk that the model learns to just follow the dummy variable and doesn't learn its own powerful abstractions and feastures from the data.

[EDITED: ] An example could be to create a weight based variable that is normalised to other physical characteristics, such as DV = weight / (height + waist circumfrence). This should then, according to your assumptions, scale nicely with the output obesity.

Another way that people might include prior information into a model, is to use a Probabilistic modelling, which incorporates ideas from Bayesian statistics. You can do things such as define a prior distribution for your outputs, conditioned on your inputs - essentially allowing you to provide information to the model (such as weight being correlated with obesity, ceteris paribus) - this then nudges the model to go along these lines.

If you would like to get into it, there are already some great libraries to make it really easy:

Stan - which has interfaces to many languages: Python, R, State etc.
Edward - probabilistic programming with modern GPU accelration and deep learning integration (Tensorflow and Keras).

A Linear regression example
A Bayesian Neural Network examples

This seems to be a nice overview of some of the methods and tools, but I haven't read through it all.

Another way that I could suggest is to play with the architecture of your model and the idea of auxilliary models. Have a look at my recent answer on a question talking about the Inception model from Szegedy et al..

The idea is that you have branches coming off the model at train time only, which also make predictions and produce error to be backpropagated through the preceding weights.

You could make a side model that predicts the obesity, based on e.g. the input weights and perhaps some related features extracted from the first layer of your neural network. This would make the idea or importance of this relationship more prominent during training and the weights would subsequently be tuned to take that into account.

At test time, you simply ignore these auxilliary branches.


How to design two different neural nets for actor and critic RL? In order to have an actor critic RL model there are two things to be satisfied .

Value approximation function should converge to a local minimum

$$\sum_s d^{\pi}(s) \sum_a \pi(s,a)[Q^{\pi}(s,a) - f_w(s,a)]\frac{\partial f_w(s,a)}{\partial w} = 0$$

The following condition should be satisfied with the parameterization

$$\frac{\partial f_w(s,a)}{\partial w} = \frac{\partial \pi(s,a)}{\partial \theta} \frac{1}{ \pi(s,a)} $$

So specifically how can we design a model to meet the second condition?

Update

here I want to highlight the value function approximation in actor-critic methods . We need to optimize the critic also as we did for the Q learning but following the on policy which is taking the TD error according to the actor. Here I will put the final equation of actor critic.

Here simply we can take the critic neural net's output as the state value function or the f above. So how to satisfy the condition 2 ?

There are many papers out there that deal with neural networks and RL. This blog will give a very good insight on a Policy Gradient Network: Deep RL with PG

Now for your question. You really need to be familiar on how we train a neural network. A simple one for classification. If you check the derivations and how the weights are getting updated things will be very clear to you on how you can implement the above.

I will describe it to you as simple as possible so you get the link. A Neural Network in a very broad sense consists of nested functions. The function that contains all the others is the one at your output layer. In case of Stochastic Policy Gradients this is your Boltzmann function. So your output layer will take all the previous layer's outputs and will pass them through the Boltzmann function. In the case of NN the parametrization comes from all previous layers.

The blog link I sent you describes a very nice and simple example of a vanilla Policy Gradient with NN (REINFORCE algorithm). By using the code (plus your understanding on feedforward networks) you will see that the gradients are multiplied by the reward. It is a very good exercise!

For Actor-Critic, you need in general a network performing PG (stochastic or deterministic) -- you Actor -- and a network that will give you the reward signal (like the simple case in the blog. However, for various reasons, instead of the actual reward we use another network that estimates the reward by performing Q-learning as in Deep-Q learning (minimizing square error between estimated reward and true reward).

Hope this helps!


Understanding XG Boost Training (Multi class classification) I have been working with XG boost for classification (multi class classification : 6 classes)
I use 5 fold CV to train and validate my model.

Please refer to the paramters, which i had used in my model.

 params    = {"objective": 'multi:softprob', "eta": 0.1, "max_depth": 7,
               "min_child_weight": 4,"silent": 1, "subsample": 0.8,
               "colsample_bytree": 0.8, "num_class" : 6, "gamma" : 0,
               "eval_metric" : 'merror', "seed": 0}

I plotted the training and testing error for each fold in a 5 fold CV.

Questions:

What can I understand/interpret from the training & test loss graph ?
Training error reduces to zero, but the testing error reduces over a period & become idle.

I am not sure whether the model is overfitting ?
How can I reduces the error b/w training & testing, only through hyperparam tunning or
this is how the XG Boost model works ?

What can I understand/interpret from the training & test loss graph?

This checking out the quality of the model. If the train and test set loss decreasing according to the number of epoch in the same way(i.e plots should overlap each other), that means that model is good. Otherwise, we have a problem. In your graph plots fastly separates(from x-value moreover 10 or 20). In this case we could say that's overfitting.

  I am not sure whether the model is overfitting ?

It's typical overfitting.

  How can I reduces the error b/w training & testing, only through hyperparam tunning or this is how the XG Boost model works ?

More data.
Feature selection.
Cross-validation.
Wide range hyperparameters to explore.
Random search.


Sales prediction of an Item So, I've been trying to implement my first algorithm to predict the (sales/month) of a single product,  I've been using linear regression since that was what were recommended to me. I'm using data from the past 42 months, being the first 34 months as training set, and the remaining 8 as validation.

I've been trying to use 4 features to start:

Month number(1~12)
Average price that the product was sold during that month
Number of devolutions previous month
Number of units sold previous month

Here are images with graphs comparing the Real Data x Predicted Data and a Error x number of elements graph:

So far the results are not good at all (as shown in the images above), the algorithm can't even get the training set right. I tried to use higher degrees polynomials, and the regularization parameter, it seems to make it worse.

Then, I would like to know if there is a better approach for this problem, or what could I do to improve the performance.

Thanks a lot in advance!

Based on the information given by you. I'm assuming you have performed multiple linear regression ie multiple features and one response feature to be predicted.

First, apply PCA on all of your features except the response variable you want to predict. In your case the four features you mentioned. Then transform it into a 2 component matrix using PCA. Once you are done with that plot the new Matrix you formed with the response features as a scatter plot.So effectively a 3D scatter plot.

When you generate this scatter plot you will be able to visualize much better on which regression you have to use. You can decide for yourself if it is linear or not. Depending on how many outliers you are comfortable with.


Wrong result when solving: "chance that two random cards differ in color and value?" I'm trying to build a simulation for this question:

"There are 50 cards of 5 different colors. Each color has cards
numbered between 1 to 10. You pick 2 cards at random. What is the
probability that they are not of same color and also not of same
number?"

(From Glassdoor)
I should have a result like "73%" but with my code I get (consistently) "72%" or "71.8%".
Here is my code:
import numpy as np

# Building a deck of 10 cards for each of the 5 colors
cards = np.array([c+str(n) for c in ("A", "B", "C", "D", "E") for n in range(1, 11)])

def random_cards_differ():
"""Returns True if two random cards differ"""
    a, b = np.random.choice(cards, 2, replace=False)
    if a[0] != b[0] and a[1] != b[1]:
        return True
    else:
        return False

nb_success = 0
nb_tries = 100000

for i in range(nb_tries):
    if random_cards_differ():
        nb_success += 1

print(nb_success / nb_tries)
>>> 0.71892

Is this normal? Is there a mistake in my code or is it a "random gotcha" caused by some seed or something else?

There's an error in your code:
cards = np.array([c+str(n) for c in ("A", "B", "C", "D", "E") for n in range(1, 11)])

will produce "A10" and "A1" among other values, and
if a[0] != b[0] and a[1] != b[1]:

will return true when a=A10 and b=A1, for example. This is why you're probably consistently underestimating the number of differences.
An easy fix would be to use:
cards = np.array([c+str(n) for c in ("A", "B", "C", "D", "E") for n in range(0, 10)])

instead, which is more readable anyway.  But if I were doing this, I might use itertools.product.
Even after this fix it's normal to not always get exactly the theoretical value, but it's bad if there's a bias (i.e. consistent under- or over- estimation).


What is batch size is required for training 30000 images in keras cnn? I am working on the 30000 images. I kept the batch size as 670 and its showing the 0.76 accuracy. Now I want to improve my accuracy up to 0.92, so for that one what should be the batch size is required?

Changing the batch size might change things, but I would doubt an increase in acciracy of 16% by change to batch size alone, given you have 30,000 images.

I would recommend playing with the learning rate in conjunction with batch size or at least allowing your model to train for more epochs. You can use the following points to guide your trials.

Changing the batch size to something smaller will generally:

Make the learning curve more volatile, because each loss update is the average of a smaller set of samples, meaning the variation between batches will likely increase
Make learning faster because the model will get bigger (but maybe more biased) updates for weights in each step
Decrease overall model accuracy - assuming the model with larger batches were to be allowed to train for much longer, given the first two points above

There has been a lot of research published on the effects of batch size, so maybe have a quick look through some papers (like this one) for the general outcomes.

Also have a look at the answer to this question about the effects of batch size on DNNs.


Colab can not connect to GPU from a python file I am trying to run a github deep learning repository in Colab but I can not connect the python files to colab GPU.

I can connect to GPU when writing a script in the colab notebook e.g. when I run this cod from a notebook cell :

import os, torch
print('Torch', torch.__version__, 'CUDA', torch.version.cuda)
print('Device:', torch.device('cuda:0'))
print(torch.cuda.is_available())

I get:

Torch 1.4.0 CUDA 10.1
Device: cuda:0
True

but when I run it from a file called myExample.py  e.g. using

!python myExample.py

I get:

Torch 1.4.0 CUDA 10.1
Device: cuda:0
False

Is there any solution to this?

Why dont you import the file into notebook and run your function.

Eg: if your example.py has something like this

if __name__ == '__main__':
    myfunction()

Then in your colab,

import os
os.chdir("path/to/cloned/dir/")
import example
example.myfunction()


Neural Network - exercise I am currently learning the concept of neural networks by myself. I am working with a very good pdf from http://neuralnetworksanddeeplearning.com/chap1.html

I also did a few exercises but there is one exercise I really don't understand.

Task: There is a way of determining the bitwise representation of a digit by adding an extra layer to the three-layer network above. The extra layer converts the output from the previous layer into a binary representation, as illustrated in the figure below. Find a set of weights and biases for the new output layer. Assume that the first 3 layers of neurons are such that the correct output in the third layer (i.e., the old output layer) has activation at least 0.99, and incorrect outputs have activation less than 0.01.

I found also the solution, as can be seen in the second image

I understand why the matrix has to have this shape, but I really struggle to understand the step, where the user calculates

0.99 + 3*0.01
4*0.01

PS: I know that the question was answered in Extra output layer in a neural network (Decimal to binary). However, the specific step I am looking for was not answered.

But what happens if the input has 0.99 and 0.01?

indicates the point: if the third layer is perfect, then we're already done; but we need to account for small variations in the third layer.  We need to understand how far off we can get after multiplying by $W$.

Now, it appears that the answer gets this a little wrong.  If $\vec{x}$ consists of one entry at least 0.99 and nine entries of at most 0.01, then $W\vec{x}$ is off by some number depending on the ten columns of $W$, not the four rows.  And we need lower bounds for the correct slots, and upper bounds for the incorrect slots (rather than upper bounds for both).

For example, if $\vec{x}=\langle 0.05, 0.997, 0.1, 0.01, \dotsc\rangle$, we'd get

$$W\vec{x} =  \begin{pmatrix}
 0.997 +0.01 + \dotsb \\
 0.1 +0.01 + \dotsb \\
\vdots
\end{pmatrix}.$$

We need to use bias/activation/cutoffs to "push" this to the desired $\langle 1,0,0,0\rangle$ (and similarly for other $\vec{x}$).  Well, the entries where we desire a 1 are at least 0.99 in the relevant position and at least 0 elsewhere (seemingly the activations are supposed to make everything positive?), so we get at least 0.99.  The entries where we desire a 0 are at most 0.01 in every non-zero position, so we get at most $5*0.01$ (the first row of $W$ has the most nonzero entries, 5).  So setting any cutoff (with relevant a bias and/or activation function) between 0.05 and 0.99 will suffice.

[Another example, since that might have been too wordy.  Say the output in the third row is
$$\vec{x}=\langle 0, 0.01, 0.99, 0.01, 0, 0.01, 0.01, 0.01, 0, 0.01\rangle.$$
Then
$$W\vec{x} = \langle 0.05, 1.02, 0.03, 0.01\rangle.$$
You can see how I've cooked up $\vec{x}$ to get the largest possible first entry in $W\vec{x}$ given the constraints on $\vec{x}$.]


Replace this for loop with a pandas vector operation I am running this for loop in python to create a list of differences of means of two dataframes, and then using that list to generate a random values under a normal distribution.
df_means = []

for _ in range(10000):
    df_sample = df_common.sample(1000, replace=True)
    d_mean = df_sample[df_sample.doggo == True].retweet_count.mean()
    nd_mean = df_sample[df_sample.doggo == False].retweet_count.mean()
    df_means.append(d_mean - nd_mean)

# under null hypothesis mean will be 0
nulls = np.random.normal(0,np.std(df_means), 10000)

However, given that it is a for loop, this is taking a long time to run. Is there a way I can replace the for loop with a vector operation to speed up code execution?

Try using list comprehension:
df_means = [np.diff(x)[0] for x in [df_common.sample(1000, replace = True).groupby(["doggo"]).retweet_count.mean().values.tolist() for _ in range(10000)]


How does cost function change by choice of activation function (ReLU, Sigmoid, Softmax)? I am new to ML and as I take courses for the area DL, I am wondering, by our choice of activation function for the last layer, whether we take sigmoid, relu or softmax, would the formula for calculation of cost function change?

I am grateful for every good reply I can get, have a nice day! :)

You need to discriminate between two types of neural networks. If your output variable is continous you can use linear, ReLU, tanh, logistic-sigmoid,... as activation functions, because these functions map continous inputs to continous outputs. If your output is discrete / categorical you can use the signum (binary) or softmax activation (multiclass) function as activation function for the output layer.

The cost function is often a function that is comparing the real outputs $y_n$ and the predicted outputs $\hat{y}(x_n)$ for the input $x_n$ for all $n=1,...,N$. Let us introduce the comparison function $D(y_n,\hat{y}(x_n))$. The comparison function has a low value if the predicted output is almost equal to the real output and high if the outputs are not similar. Assuming all the observations are equivalently important, we could sum the values of comparison function applied on all observations and obtain the integrated loss

$$J=\sum_{n=1}^ND(y_n,\hat{y}(x_n))$$

for the whole data set.

In order to see the influence of the activation function $g$ in the last layer we summarize the transfer function from the input to the last layer as $f(x_n)$.  Then the predicted output $\hat{y}(x_n)$ can be written as

$$\hat{y}(x_n)=g(f(x_n)).$$

Hence, the activation function at the output has an effect on the integrated loss $J$. For example if you choose the $\tanh$ as output activation you will bound your outputs in the intervall $(-1,1)$ which will be a bad choice if your outputs can be from $\mathbb{R}$ and your cost function will probably have a very high value while training. A better choice would be a linear activation function at the output layer.


What to choose for null hypothesis and alternate hypothesis? In this article which explains types of t-tests
for “Independent Two-Sample t-test” example:

We can confirm that the t-statistic is again less than the t-critical
value so we fail to reject the null hypothesis. Hence, we can conclude
that there is no difference between the mean screen size of both
samples.

Null hypothesis was that the average screen size of the sample does not differ from 10 cm.
Alternate hypothesis was that the average screen size differs. And also the data is only about sizes of screens.
Why can't null hypothesis be that the average screen size differs. and alternate hypothesis, that the average screen size doesn't differ.
Not just this example, in every article I have read, every author only told what was chosen for null and alternate hypothesis. But, no one talked about why they chose what they chose. I always think that, why can’t the statement chosen for alternate hypothesis be chosen for null hypothesis and vice versa. And the data won’t have much information too. It’s always up to the article writer to what to choose for null and alternate hypothesis.
So, my question is, are there some set of rules, that have to be followed to choose a statement/condition for hypothesis. Or what actually causes a particular statement to be chosen for hypothesis. Or am I missing something.

You always choose the statement that you want to disprove as Null hypothesis. You can either reject it in favour of alternative hypothesis or not reject it. We don't say we accept null hypothesis. Because we don't have enough evidence for the null hypothesis to be true. It might be true, might be not. But with significant t/p value we can reject null hypothesis in favour of alternative hypothesis
Like in this example, if you don't get any significant value after the experiment you do not reject that the average screen size of the sample does not differ from 10 cm.But you don't have any proof to accept it either.
Always choose the statement you want to disprove as null hypothesis.


Confusion between precision and recall I have a machine learning model that try to fingerprint the functions in a binary file with a corpus. Final output of upon inputing a binary file is a table with one to one mapping between the binary function and the corpus function as follows-:

As you could see from the names, some of the functions are correct while the others are incorrect. Is there a way to calculate precision and recall for the above result? I understand that precision and recall make sense if I am doing other ML tasks such as image classification. Using a confusion matrix will help to calculate both the metrics easily. However, I am confused and feel that I could not do such measures as it is just one to one mapping which is either true or false. If precision and recall does not make sense, is there any other metrics I could use to evaluate the model? Thank you!

First of all, precision and recall are not specific to image classification; they are relevant wherever there are two distinct "positive" and "negative" classes (for example, when you test an e-mail for "spam/not-spam", or a blood sample for "has virus/does not have virus").
You can read more on this question on Cross Validated, but to sum it up - precision is the probability that a sample is positive if a test said it is, and recall is the probability that a positive sample will be reported as positive by the test.  False positives mess up your precision, and false negatives mess up your recall.
Now, your task appears to be one of multi-class classification - with at least 17 classes, from your example. I wouldn't go with precision/recall for this - you can only do it pair-wise for pairs of classes. You can, however, plot a CxC confusion matrix (where C is the number of classes), and investigate where your models tend to miss. There's an implementation in SKLearn (link).
If you need a single-number metric, I'd start with just accuracy (and develop from there). Following Nuclear Wang's comment, I'd also suggest looking at Cohen's Kappa (see explanation on Cross Validated) to better account for class imbalance.
To read more on multi-class classification, see this question. I'd also recommend this blog post on Towards Data Science


Different number of features in train vs test I'm doing the titanic exercise on kaggle and there is a categorical Cabin attribute that has a lot of different strings: C41, C11, B20 etc. (about 100).

To be able to train my model I'm converting it to numerical attributes (using pandas get_dummies()). So in the end I get 100+ attributes.

On the test dataset however, there are less cabins, so I'll end up with fewer attributes.

I did something like this to make them equal (create columns that are in the training set and delete those that aren't):

for column in X.columns:
    if column not in X_test.columns:
        X_test[column] = 0

for column in X_test.columns:
    if column not in X.columns:
        X_test.drop([column], axis=1, inplace=True)

but I know it is not a good thing. So how else should I approach it?

I tried removing the cabin column altogether but my model performs better on test data with that column.

You could concatenate your train and test datasets, crete dummy variables and then separate them dataset.

Something like this:

train_objs_num = len(train)
dataset = pd.concat(objs=[train, test], axis=0)
dataset = pd.get_dummies(dataset)
train = copy.copy(dataset[:train_objs_num])
test = copy.copy(dataset[train_objs_num:])


Data manipulation in SQL I need to write a SQL query where I need the data for the year 2014 from May-Dec and complete data (Jan-Dec) of the year 2015. Can anyone help me with this query?

Use following WHERE predicate

-- data (Jan-Dec) of the year 2015
year = 2015  OR
-- data for the year 2014 from May-Dec
year = 2014 and month in ('May','Jun','Jul','Aug','Sep','Oct','Nov','Dec')

Adjust the column names and/or litaral values as required.
If the month is stored as number you may use month between 5 and 12


No, you can't exclude that there are some clusters in this data.

Maybe they:
- can not scored by CH and Silhouette
- can not be found by k-means (but by other algorithms)
- require better preprocessing
- were destroyed by the preprocessing you applied
- exist only in subspaces

But I would agree that none of the k-means results results you got was good on Silhouette. Which is not surprising given 128 variables.


Image clustering by similarity measurement (CW-SSIM) I'm trying to use scikit-learn and pyssim for clustering a set of images - less than 100.

The end goal is to place the images into several buckets (clusters) according to the calculated similarity measures - CW-SSIM.

The task seems to be trivial, but I can't figure out the best way to handle "similarity based" clustering in scikit-learn. K-Means clustering looks like a good choice, but it doesn't accept any "comparison functions" or custom distance functions.

So how to handle the comparison based (similarity based) clustering in scikit-learn?

I was thinking about "comparison matrices" with 1 (similar) or 0 (not similar) per cell according to the calculated CW-SSIM similarity values. This matrix will be used for fitting into K-Means clustering. But then we will face the scalability issue, because such matrix will have dimensions equal to the amount of images ... which might grow to 1+ million in the future.

If there is an easier option in R than in Python, then I'm ready to review as well.

Thanks in advance.

UPDATE from Jan 18, 2016

I've created some code on GitHub about this topic: https://github.com/llvll/imgcluster

This project also includes IP[y] Notebook with step-by-step instructions and extra comments: https://github.com/llvll/imgcluster/blob/master/ip%5By%5D/imgcluster.ipynb

I would use a regular clustering algorithm and replace the objective function, which is usually the MSE, with a differentiable loss function of your choice. Another way is to learn an embedding that optimizes your similarity metric using a neural network and just cluster that.

If you would rather do similarity-based clustering, here are some papers:

A Similarity-Based Robust Clustering Method
A Discriminative Framework for Clustering via Similarity Functions
Similarity-Based Clustering by Left-Stochastic Matrix Factorization

sklearn implements two similarity clustering methods: Affinity propagation, and spectral clustering.


How can I plot data after cluster it into two cluster? I have some unlabeled data with 15 attributes and 30161 instances. I took 70% data for clustering and cluster those data into 2 cluster. I also added one column after clustering named 'cluster' and the column '<=50k' was label column. Now I need to plot those data except label column, How can I plot it ?
Here's the sample of my code and data.

Data

cols_x=train.columns[0:14]
cols_y = train.columns[14]
X = train[cols_x]
y = train[cols_y]

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30,random_state=10)

kmeans = KMeans(n_clusters=2)
train["cluster"] = kmeans.fit_predict(train[X_train.columns[0:14]])
train.head(10)

KMeans on such data is meaningless if used carelessly. It looks as if your result is based on whether the ID is larger than 200000 or not.

Rather than copy and pasting example code from the internet now also to visualize, I think you should spend more time thinking about what you are actually computing. Because solving the wrong problem doesn't help.


How to find the feature regions where each label is the most expected when using decision trees? Given a decision tree for classification for example this one:

What is the way to find the feature domain (petal and sepal width and length) where a sample would most likely occur in the feature space for each class?
It is clear here that for Setosa it is when petal length is less or equal to 2.45.
However, where I am confused is how to think in more complex cases. For example, let's take Versicolor:
I am hesitating between 2 choices or take every path that leads to Versicolor or just choose the domain (considering the path) that leads to the leaf with the most samples.
I don't necessarily care about this example, I want to know the general case and how to think to solve that problem.
Thanks

It seems that you want to achieve something like this :

Where you can see the instances, classes and the predicted and the cutoffs for the rules. The exemple is taken from : https://jakevdp.github.io/PythonDataScienceHandbook/05.08-random-forests.html You might want to find one that is interactive (plotly ?) so you can get the rules that interest you by hoovering your mouse above the graph.
Note that this appraoch has some problems :

It only work with two variables at a time. You migth need to plot similar graphs for all your couple of features.
It only work for simple classification trees. It might start to get more difficult to interpret the plots and the rules if your output is continuous.


Heavy regression loss for false non 0 prediction My regression should predict values  >=0
But a wrongly predicted value >0(e.g. 0.001 instead of 0) is much worse then a a slight missprediction of 0.001 (e.g. 0.002 instead of 0.003)

I am thinking about a costume loss function that weights the false non zeros to return a big loss.

Is there a more elegant way ?

Edit:

this is my unelegant sollution:

def custom_loss(y,yh):
    if y == 0:
        loss = tf.math.minimum(100,yt*100000)
    else :
        loss = losses.mean_squared_error(yh,y)

    return loss

One option would be Huber Loss which could be setup to increase the weight for some types of errors compared to other errors.


How to add a filter/query/condition in Plotly I want to add a condition to the column where attractionName.value_counts() >=165. How can I add a query/filter/condition to a DataFrame before I plot the data.
fig = px.bar(rwgor, x="attractionName", y="attraction score summary", color="attraction score summary", title="Long-Form Input")
fig.show()

With the above code I can plot all values in the column attractionName. I only want to plot values whose value_counts() is grater than 165.

Assuming your DataFrame is rwgor, you need to filter it like this:
rwgor = rwgor.loc[rwgor.attractionName >=165]
fig = px.bar(rwgor, x="attractionName", y="attraction score summary", color="attraction score summary", title="Long-Form Input")
fig.show()

or
fig = px.bar(rwgor.loc[rwgor.attractionName >=165], x="attractionName", y="attraction score summary", color="attraction score summary", title="Long-Form Input")
fig.show()

Finally, note that I used rwgor.attractionName >=165 and not rwgor.attractionName.value_counts() >=165, according your plot description.


Interpreting confidence interval results for datasets I have created a dataset automatically and wanted to clarify my interpretation of the amount of noise using the confidence interval.
I selected a random sample and manually annotated the sample and found that 98% of the labels were correct. Based on these values I then calculated the confidence interval at 99% which gave a lower bound of 0.9614 and upper bound of 0.9949. Does this mean that the noise in the overall dataset is between the lower and upper bound and is then from 0.005% to 0.038%?

No, that isn't what it means.
For one thing it is not clear what parameter the confidence interval that you calculated is for.
In any case, some care is needed in the interpretation of (frequentist) confidence intervals.
In frequentist statistics, a confidence interval is random, and the parameter that the interval is for is fixed. In the case of a 99% interval this means that if the data were collected again, many times, and the confidence interval re-calculated each time, then 99 times out of 100 it would contain the true value of the parameter. This is the only technically correct interpretation of the frequentist confidence interval. It is often interpreted, incorrectly, as being an interval that contains the parameter with 99% probability, and that appears to be the interpretation that you are using.


Since the daily dataset does not contain labels, you could aggregate the daily data into annual and then do the join. It sounds like a (binary) classification problem, which can be done using methods such as logistic regression. You will however have to handle missing values caused by the left join, one method would be imputing them. Or just doing an inner join if the missing data is random without patterns (e.g. companies of certain type don't have missing data more often than the other types) and if there's enough data that is not missing.


R - time series decomposition without detection of seasonality I have a time series dataset with 200 data points. I have decomposed it using the function below:

dat2 = ts(dat1, frequency = 4)
decomposeDat = decompose(dat2, "multiplicative")

I get 4 components: trend, seasonal, cyclic and irregularity. But when I check if there is seasonality present in the dataset with frequency "4", Rstudio says that there is no seasonality for this frequency. The check is performed with the following code:

dat2 = ts(dat1, frequency = 4)
fit <- tbats(dat2)
seasonal <- !is.null(fit$seasonal)
seasonal

seasonal returns FALSE meaning that there is no seasonality with frequency 4.

Can someone explain, why can I decompose it into a seasonality component when no seasonality is present from the check mentioned above?

Without seeing your data it is hard to tell whether there is seasonality or not. The decompose() function will try to find seasonality using a different approach than tbats() as discussed in this post and the user comments of this blog post by the author of tbats.

I would recommend to post your data and discuss what you are trying to achieve to get more guidance.


Next event prediction in the sequence I have a sequence of event and would like to predict the next one. The training data looks like this:

Ev1,Ev2,Ev5,Ev6,Ev7
Ev1,Ev6,Ev99
Ev4,Ev3,Ev6

So, the idea is to get Ev7 given Ev1,Ev2,Ev5,Ev6

The problem is that the number of "final" events is very high (100K). I tried to look into neural network but it means the last layer would need to have size of 100K.

Can anybody point to any other ways how to do that or any examples?

Hidden Markov Model (HMM) is a method that you could try out. They have been used to model very large sequences (like gene sequences).


Are there any examples other than anomaly detection where unsupervised deep learning could be useful? I am new to deep learning and its concepts. After reading a while I understood that unsupervised deep learning techniques usually try to reconstruct the input data(probably with less number of dimensions using encoder-decoder) and train the network by optimizing reconstruction error. But I am unable to image how these could be used for solving real life tasks(other than anomaly detection, for example clustering).

Note: You can correct me, if my understanding about the unsupervised deep learning techniques is wrong.

Recommendation Systems

In recommendation systems the idea of extracting signal and using to make recommendations is very common, e.g, Alternating Least Squares and Singular Value Decomposition approaches.

Autoencoder fits quite well, reducing dimensionality (the encoder part) should help extract signal. The weights in the network represent the behavior of all the users that we trained on, but we don't want to capture all of it, we want to capture just the most import parts. Size of output of encoder (or bottleneck) controls the amount of dimensionality reduction.

Using it is quite simple:

1) You train the encoder-decoder model on, say, user movie ratings, where user ratings are input and output.

2) To generate recommendations for a user, just pass through user's current ratings through encoder-decoder, and you have the scores that can be used to make recommendations.


Difference between OLS(statsmodel) and Scikit Linear Regression I have a question about two different methods from different libraries which seems doing same job. I am trying to make linear regression model.

Here is the code which I using statsmodel library with OLS :

X_train, X_test, y_train, y_test = cross_validation.train_test_split(x, y, test_size=0.3, random_state=1)

x_train = sm.add_constant(X_train)
model = sm.OLS(y_train, x_train)
results = model.fit()

print "GFT + Wiki / GT  R-squared", results.rsquared

This print out GFT + Wiki / GT  R-squared 0.981434611923

and the second one is scikit learn library Linear model method:

model = LinearRegression()
model.fit(X_train, y_train)

predictions = model.predict(X_test)

print 'GFT + Wiki / GT R-squared: %.4f' % model.score(X_test, y_test)

This print out GFT + Wiki / GT R-squared: 0.8543

So my question is the both method prints our R^2 result but one is print out 0.98 and the other one is 0.85.

From my understanding, OLS works with training dataset. So my questions,

Is there a way that work with test data set with OLS ?
Is the traning data set score gives us any meaning(In OLS we didn't use test data set)? From my past knowledge we have to work with test data.
What is the difference between OLS and scikit linear regression. Which one we use for calculating the score of the model ?

Thanks for any help.

So my question is the both method prints our R^2 result but one is print out 0.98 and the other one is 0.85.

There is a lot of information missing here, but it looks like the first example could be finding the correlation between the predicted and actual values of data from your training set and the second example is finding the correlation between the predicted and actual values of your testing set. This would also explain why the correlation in the second result is lower. The performance of a model on novel (testing) data should nearly always be poorer than on the training dataset that was used to fit the model.

  Is there a way that work with test data set with OLS ?

The statsmodels documentation describes a predict() method that can be used to make predictions on inputs from a design matrix.  You should be able to use this method on your test data once you have fit the model.

  Is the traning data set score gives us any meaning(In OLS we didn't use test data set)? From my past knowledge we have to work with test data.

Evaluation on training data can often be used as a quick check for "best case" generalization performance.  However, this is often only used as a diagnostic to ensure your model has fit properly and should not be taken to be a true indicator of generalization.


How can I use a multivariable list of strings as a single X variable / feature in a Neural Network Model Here is an example of what I am trying to do.

I am using a bunch of X variables to predict a Y.

X1 = Number of days since applying first medication
X1.1 = Total times the medication should have been applied (calculated using frequency in prescription)
X1.2 = Total times the medication was applied (calculated)

X2 = Number of days since applying first medication
X2.1 = Total times the medication should have been applied (calculated using frequency in prescription)
X2.2 = Total times the medication was applied (calculated)

X3 = Number of days since applying first medication
X3.1 = Total times the medication should have been applied (calculated using frequency in prescription)
X3.2 = Total times the medication was applied (calculated)

Y = Skin clear 0 / 1

I have some sample data as below:

Now, the X1.1 and 1.2 are descriptive of X1. As a separate input to the Neural Network, they are meaningless.

I am trying to find, how I can tell the model that X1, X1.1 and X1.2 together is a single input with multiple dimensions.

I use the below code:

classifier = Sequential()

classifier.add(Dense(units = 14, kernel_initializer = 'uniform', activation = 'relu', input_dim = 27))

classifier.add(Dense(units = 14, kernel_initializer = 'uniform', activation = 'relu'))

classifier.add(Dense(units = 1, kernel_initializer = 'uniform', activation = 'sigmoid'))

classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])

classifier.fit(X_train, y_train, batch_size = 25, epochs = 500)

Any help will be really appreciated.

For your problem you can easily use PCA to remove the redundant features and only keep the viable information. Since you are using neural networks, you anyways wont have any interpretation layer, so PCA will not be a problem.


Difference in model performance measures of train and test data sets I am using CART classification technique by dividing a dataset into train and test sets. I have been using Mis-classification error, KS by rank ordering, AUC and Gini as MPMs(model performance measures). The problem I am facing is that the MPM values are quite far apart.

Dataset
Metadata

I have tried with minsplit equal to anywhere from 20 to 1400 and minbucket from 5 to 100 but couldn't get expected results. I have also tried oversampling/undersampling through ROSE package but without any improvement. Moreover, the mis-classification error increased a lot. Following code is through which I could get the best values, but they were not enough.

#Reading Data
pdata = read.csv("PL_XSELL.csv", header = TRUE)

#Converting ACC_OP_DATE from type factor to date
pdata$ACC_OP_DATE<-as.Date(pdata$ACC_OP_DATE, format = "%d-%m-%Y")

#Paritioning the data into training and test dataset
set.seed(2000)
n=nrow(pdata)
split= sample(c(TRUE, FALSE), n, replace=TRUE, prob=c(0.70, 0.30))
ptrain = pdata[split, ]
ptest = pdata[!split,]

#CART Model
#Taking the minsplit, minbucket values as low as possible, so that pruning
#can be done later. Higher values didn't allow any scope for pruning
r.ctrl = rpart.control(minsplit=20, minbucket = 5,  cp = 0, xval = 10)

#Calling the rpart function to build the tree
cartModel <- rpart(formula = TARGET ~ .,
    data = ptrain[,-1], method = "class",
    control = r.ctrl)

#Pruning Tree Code
cartModel<- prune(cartModel, cp= 0.00225  ,"CP")

#Predicting class and scores
ptrain$predict.class <- predict(cartModel, ptrain, type="class")
ptrain$predict.score <- predict(cartModel, ptrain, type="prob")

Results that I got-: Train data Mis-classification error-.103 AUC - 0.679 KS - 0.259 Gini - 0.313

Test data Mis-classification error-.113 AUC - 0.664 KS - 0.226 Gini - 0.307

Is it due to the dataset or am I doing something wrong. I am new to Data Analytics. It is a part of my academic project, so I need to use CART technique only. I will put separate questions for Random Forest and Neural Networks. Kindly help.

Your model suffers of a slight overfitting, however it doesn't seem too dramatic.

Performance on train set are always better than test set if random sampled (when you have statistically significant volumes)

Maybe you can reduce the gap of performance by controlling the CP parameter, try setting a higher cp when you prune the tree (like 0.01) or by using the parameter maxdepth that prune according to the length of the tree.


Feature Scaling and normalization in cross-validation set I have a question that normally, when we are making a training set and a final test set,

  we would compute the mean and standard deviation for preprocessing
  using the training data and use it to standardize (transform) the test
  data.

So, when we are making a training data, a cross-validation data and a final test data, shouldn't we compute mean and std for preprocessing using the training data alone and then standardize the validation set and test set using it? And, if I'm right how can we do it in keras, since in keras like in the below code, taken from kaggle( by Francois Chollet MNIST tutorial):

simple-deep-mlp-with-keras by Francois Chollet

# pre-processing: divide by max and substract mean
scale = np.max(X_train)
X_train /= scale
X_test /= scale

mean = np.std(X_train)
X_train -= mean
X_test -= mean

model.fit(X_train, y_train, nb_epoch=10, batch_size=16, validation_split=0.1, show_accuracy=True, verbose=2)

The validation set is created using the fit method and thus, it is also used in computing the preprocessing mean and std. So, how to create validation set before and use only the training data for feature scaling and normalization? And which method is the right method?

Also, in the above code scale and mean are being computed for each column respectively and is being used for normalization , should we compute mean and scale(standard deviation or range) for each pixel or for the entire image? How does that affect?

Answering your last question first: your mean and std should be computed over the entire image. So for an RGB image you will get a mean/std vector of 3 values for the entire dataset (calculated per channel). Normalizing per pixel is wrong and would result in loss of information (you need to think of an image as a single discreet function, you can't treat each part of it separately without changing its overall meaning).

Regarding normalizing the validation set, you can manually create the validation set and use the "validation_data" argument in the fit method:

# Creating the validation set:
X_valid = X_train[int(0.9*len(X_train)):]
X_train = X_train[:int(0.9*len(X_train))]
y_valid = y_train[int(0.9*len(y_train)):]
y_train = y_train[:int(0.9*len(y_train))]

# pre-processing: divide by max and subtract mean

scale = np.max(X_train)
X_train /= scale
X_valid /= scale
X_test /= scale

mean = np.std(X_train)
X_train -= mean
X_valid -= mean
X_test -= mean

model.fit(X_train, y_train, batch_size=16, epochs=10, validation_data=(X_valid, y_valid), verbose=2)


pandas groupby and sort values I am studying for an exam and encountered this problem from past worksheets:

This is the data frame called 'contest' with granularity as each submission of question from each contestant in the math contest. The question is

and the answer is in red. I get why that works, but why is the 4th choice wrong? I really can't figure it out - please help.
*please let me know if this is not allowed as a post in this community.
for full description of the problem:

In this question, we will be looking at the contest dataframe which contains data from a
math contest in 2019. In the contest, each participant had a total of five questions. The participants submit each question separately and each row of the DataFrame records a particular submission of one of the contestants by some participant. The Timestamp column specifies the
time a given problem is submitted by a participant; each timestamp is discretized to the minute
and has been properly converted to a Pandas datetime object with pd.to datetime.
The Contestant column contains the id-name pair of each participant. The Question
column contains the question that was submitted. The Correct column tells us if the answer
given in the submission is correct (1) or not (0). Assume each participant can have several
submissions for the same problem, but they can only submit one question per minute.

If the participant has answered the question 2 before the question 1, you will lose the information on question one by using .agg("first") in the 4th option


Where does the sum of squared errors function in neural networks come from? Training a basic multilayer perceptron neural network boils down to minimizing some kind of error function. Often the sum of squared errors is chosen as a this error function, but where does this function come from?

I always thought this function was chosen because it makes sense intuitively. However, recently I learned that this is only partly true and there is more behind it.

Bishop wrote in one of his papers that the sum of squared errors function can be derived from the principle of maximum likelihood. Furthermore he wrote that the squared error therefore makes the assumption that the noise on the target value has a Gaussian distribution.

I am not sure what he means with that. How does the sum of squared errors relate to the maximum likelihood principle in the context of neural networks?

you can trace the squared error in statistics through multivariate calculus all the way to Pythagorus. You are basically calculating the 'effective length' of the error, the hypotenuse, among errors from multiple variables $(X_1 - X_2)^2 + (Y_1 - Y_2)^2 + ...$ like in a triangle.

But where did the square root go?

Somebody realized that calculating roots of multiple variables over multiple iterations is computationally very expensive. So they decided to drop it. Checkout the squared Euclidean distance here for more details

How would a cubic error or a logarithmic error affect the outcome?

It just takes more time to converge because they are not as accurate. But we do see logarithmic errors over squares such as logistic regression where it is more optimal

All in all it is a simple case of optimization


Do we really need `` tokens? I am wondering, do we really need <unk> tokens?
Why do we limit our vocabulary?

Is it for speed? Accuracy?

If we disable all limitations, what do you predict happens?

The <unk> tags can simply be used to tell the model that there is stuff, which is not semantically important to the output. This is a choice made via the selection of a dictionary. If the word is not in the dictionary we have chosen, then we are saying we have no valid representation for that word (or we are simply not interested).

Other tags are commonly used to groups such thing together, not only (j)unk.

For example <EMOJI> might replace any token that is found in our list of deined emojis. We are keeping some information, i.e. that there is a symbol representing emotion of some kind, but we are neglecting exactly which emotion. You can think of many more examples where this might be helpful, or you just don't have the right (labelled) data to make the most of the contents semantically.


Probabilistic Machine Learning model to match spatial data I have spatial data from multiple sources. This data consists of ID, lat, long, and time.

My goal is that given a new lat-long, the model needs to return (preferably with a probability) the data points that match the new lat-long. This matching should be based on the features (such as lat, long, timestamp).

I could only think of clustering. ie. Cluster the dataset and try to predict which cluster the new data belongs to. The drawback is that if the cluster has a lot of points then its hard to accurately pin point to which point in the cluster matches the closest to the new point.

Is there any other ways to do this? Any probabilistic model (HMM?).

So you have an existing dataset;
$$\mathbf{X} = \{ [id_i, latitude_i, longitude_i, time_i] : i \in \{1,...,n\} \} $$
And you receive a new sample;
$$ \mathbf{x}_* = [latitude, longitude, now] $$
And you want to determine a probability of which datapoints in $\mathbf{X}$ match $\mathbf{x_*}$?

For fear of pointing out the obvious why not just use K-nearest-neighbours, where $K=1$. Obviously you need to establish a distance metric that accounts for time, or at least equate time to longitude and latitude difference, or you could just ignore time altogether.

Distance Metric

The notion of nearest is obviously associated with some measure of distance. Considering a map one can easily associate a point on a map with a coordinate system and the notion of distance is intuitive. Given a sample data point you can just find the nearest datapoint to it on the map.

But what if you have another dimension that is not intuitively obvious. This is often specific to the problem. Suppose you were searching for two criminals in a city, and criminal Adam was last seen at coordinates (0,0) 1 week ago, and criminal Brian was last seen at coordinates (1,1) 1 day ago. Now you have a new sighting at (0.25, 0.25). This is closer in geography to (0,0) than (1,1) but Brian was more recently seen so perhaps this is the more likely lead to allocate resources for a search?

When using K-nearest neighbors you might need some kind of transformation function to convert distances to probabilities. There may be numerous choices, google softmax as one possible starting point.


Extracting information with corresponding fields I have large pool of scanned county documents. I need to extract information like document title, borrower name&address, lender name&address etc.

The text is like this
Eg: the deed of trust, between abc llc, a limited company, whose address is XXXXXX, herein called "borrower", and  xyz, whose address is XXXXX,herein called "lender".

I used Named entity recognition method to  extract the names, it works well. but how would i know which name is borrower and which one is lender? can anyone help me

you’re definitely on the right track with NER. As for determining the ‘class’ of what you’ve extracted, I think you have two major options:

Train a new entity type for each.
Use a set of rules that examine the term in context to determine the class.

I think you might have more luck with 2, if the language surrounding the terms is pretty static. Also, I haven’t watched it yet, but I thought this might help as well - seems to be about exactly your problem domain. https://www.youtube.com/watch?v=KrXJmaSHBJU.


What are the pros and cons of zero padding in a convolution layer? TensorFlow's conv2d() operation lets you choose between "VALID" (without padding) and "SAME" (with zero-padding). I suppose all other frameworks let you do the same. I'm trying to understand the pros and cons of zero padding: when would you want to use it, or not to use it?

So far, my understanding is that if the filter size is large relative to the input image size, then without zero padding the output image will be much smaller, and after a few layers you will be left with just a few pixels. So to maintain a reasonably sized output, you need zero-padding + stride 1. Is this the main reason for using zero-padding? Is it preferable to avoid it when you can, for example when the filter size is small relative to the input image size?

I summarized from some other web pages:

Pros: Deep network because the output dimension could be constant after convolution

Saving image border information

Cons:

Waste of computational resources


Regex-style pattern-matching for time series This is more of a "what technology/library would you use for this?" question than anything else.

I have categorical time series data, and need to match cases in these time series to known patterns. For example, State A, followed by State B within six months, followed by multiple periods of State A, followed by an optional period of State B, followed by State C.

It's quite hard to interrogate e.g. an SQL database or pandas dataframe for sequences of rows that match a pattern like this. How you'd do this "in theory" is build a pattern-matching finite state machine, but I didn't know a relevant library for one. Then I realised I've been using pattern-matching finite state machines for years in the form of regular expressions.

So, what I've been doing is converting my time series to strings, defining my pattern as a regex, and matching them that way. It works extraordinarily well. In spite of this, it feels hacky. Surely there should be some proper way to do this.

Is there? What technology/data structure/library would let me define finite automata for time series and match them based on that definition?

If you need optional parts in the pattern then regular expressions were specifically built for those kind of problems. You could probably speed up regular expressions by using loops in Cython, but the time spent programming it is not really worth it.

If your pattern is fixed then there are faster ways to do pattern matching. OpenCV uses Descrete Fourier Transformation to match a template. It is very fast and is available for Python and C++. It is optimized to work with 2D images, and can also work with 32 bit floats. How DFT works

Documentation for template matching in OpenCV


Heat map and correlation among variables I would have a question on heat map and correlation among variables.
I created this heat map, looking at possible correlation among variables and target. I got very small values.
I wanted to set a small threshold, e.g., 0.05, for selecting features.
Do you think it makes sense, or should I exclude all of them?

From the info you provide, it seems you are carrying feature selection based on the correlation between your predictor variables and the target.
This is correct as a type of feature selection (see here) in the family of univariate filter selection, although not the only one. It is fast and intuitive, although you can have a look at other methods. You might also be interested in:

variance threshold selection (also per input feature, univariate filter method): it assumes that higher variance in a feature values could mean more prediction power
sequential backward selection (look here): it means more performance cost, but features are judged in subsets (not independently as above) and is ok if you don't have many features (as it seems to be)

There are many other strategies for feature selection (you might want to check for this source)


Hyperparameter tuning in multiclass classification problem: which scoring metric? I'm working with an imbalanced multi-class dataset. I try to tune the parameters of a DecisionTreeClassifier, RandomForestClassifier and a GradientBoostingClassifier using a randomized search and a bayesian search.

For now, I used just accuracy for the scoring which is not really applicable for assessing my models performance (which I'm not doing). Is it also not suitable for parameter tuning?

I found that for example recall_micro and recall_weighted yield the same results as accuracy. This should be the same for other metrics like f1_micro.

So my question is: Is the scoring relevant for tuning? I see that recall_macro leads to lower results since it doesn't take the number of samples per class into account. So which metric should I use?

You should use the same metric to evaluate and to tune the classifiers. If you wull evaluate the final classifier using accuracy, then you must use accuracy to tune the hyper parameters. If you think you should use macro-averaged F1 as the final evaluation of the classifier, use it also to tune them.
On a side, for multiclass problems I have not yet heard any convincing argument not to use accuracy, but that is just me.


Confusion with cosine similarity In information retrieval when we calculate the cosine similarity between the query features vector and the document features vector we penalize the unseen words in the query.

Example if we have two documents with features vectors

d1 = [1,1,1,0,0]
d2 = [0,1,1,1,0]

We can see that the two documents have the second feature so if we want to search for the second feature with query vector: q = [0,1,0,0,0] then the cosine similarity between q and d1,d2 will be $1/√3$, and not 1 because that we penalize the other features that we have not mention in the query.

From this discussion I don't understand why penalize it is a good Idea.

Is penalizing unseen features good?
Is there another similarity measure that does not penalize them?

Of course cosine similarity is not proper for searching a specific features in documents! To do this, you can exactly using dot product, as it will ignore zero features in query vector from documents.

Cosine similarity, in the current context, can be used to finding similarity between two documents. So, all features can be important to finding similarity. It means, if there is not a feature in query vector, but there is in  a document, or vice versa, these two are 100 percent similar. Hence, it makes sense.


Quantifying the Reproducibility of LDA Models I am working on a text mining project where I'm using Latent Dirichlet Allocation to study a corpus of documents. I'm currently in the process of optimizing my parameters to get the best models for my client. My biggest concern at this point is whether my models are reproducible or not. I understand that LDA models are probabilistic, so separate models run on the same data under the same parameters are not guaranteed to be identical.

However, for a model to be considered reliable you would expect duplicate models to come out looking roughly the same. A more reproducible model would be a more accurate representation of the text it is modeling. I wrote a script to find the average similarity between two distinct models (found here: https://github.com/HarryBaker/gensim/tree/topic2topic_seperate_models). The idea behind it is that I try to force a one to one relationship between two models' topics, and then find the average similarity of each match. The assumption is that a reproducible model would have a clear bijection between a duplicate.

From my early research it seems like training a model for longer increases the similarity of duplicate models. (Models trained under 500 iterations were more similar than those trained under 150 passes).

What I'm wondering is if there's been any papers or studies done on the reproducibility of LDA models, or if anyone has any ideas. I do not want to fix a seed. I don't want to be able to output an identical model--I want to be confident that for any model I create, a model trained under the same data and parameters will be almost the same.

As you already noticed, Topic Models aren't reproducible due to the probabilistic algorithm. I did some topic models on a large corpus (10k documents, 30M words), and generally, between two different topic models about 10% of the topics were incomparable. My corpus is mainly English, but contains some Latin texts, and for a given number of topics I had between 1 and 3 topics consisting of Latin language words predominantly.

What to do now depends on what you want to learn from your Topic Model. When you want to "interpret" the Topic Model (using methods from the humanities), the best thing is to commit to one topic model (i.e., fixing the random seed).

When you want to study some derived measures (like topic diversity), make sure that the derived measures taken from different topic models are "stable", i.e., centred around one value with small statistical fluctuations.

You can also try to create an ensemble of topic models and try some techniques to get an "average topic model" out of it. I have no experience yet in this business and cannot suggest techniques here.


Why does adding data augmentation decrease training accuracy a tiny bit? Before data augmentation, my model clearly overfits and hits a 100% training accuracy and a 52% validation accuracy. When only adding data augmentation with Keras, as a regularization technique, it achieves a 95% training accuracy with slower convergence and a 80% validation accuracy (which is a way better result). But why does the training accuracy gets reduced by around 5%?
If somebody could provide the link to a research paper or explain the reasoning behind this, it would be greatly appreciated!

The obvious reasons why data augmentation might reduce the train accuracy is -
As you know, deep learning models are data hungry. If the model don't get enough data to recognize the patterns then it will try to memorize the dataset. Bigger models tend to memorize the data instead of finding patterns, because they are big enough to do so. When model memorizes the training data it will definitely perform very good on training set and poorly on validation set.
And as you said data augmentation is a regularization technique. In regularization your model weights are penalized more to make sure they don't over fit. As a result, your model cannot perform well on training set (depending on how much regularzation is used), but as an advantage model will try to find generalized patterns in the dataset and this will also help at the time of validation.
I could find one research paper which has exhaustive experiments about data augmentation and regularization.


What is the advantage of a tensorflow.data.Dataset over a tensorflow.Tensor? I have my own input data class. It has x and y as well as test and train values (1 Tensor for each combination). I noticed there is a Dataset class built in to TensorFlow. What is the advantage of this class over a regular Tensor? Is it mainly around handling large datasets / laziness? It doesn't appear to have features tailored for x vs y data, or test vs train. All my data fits into memory so I am not clear it would be beneficial to use the built in class over my current one. Of course, the first assumption is it would be foolish not to use the built in class.

The main advantage is in domains where you can't fit all of your data into memory.
However, I've seen improvements in performance even in cases where I have all my data into memory. I think two reasons contribute to this:

One is caching, where some operations (e.g. a mapping OP) will be cached and performed only in the first epoch. This, obviously is applicable if you have such a function.
Another one is prefetching. While a the model is being trained on a batch in the GPU, the CPU loads and prepares the next batch. This can help save a lot of time.

Some other capabilities are allowing for the vectorization of user defined functions (e.g. for data augmentation) and their parallelization.
You can take a look at some benchmarks here. They are a bit unrelated, as they refer to cases where the dataset isn't all loaded into memory, but they might interest you nevertheless.


Extracting text from few areas on product label I'm trying to achieve algorithm that will extract text from few areas(marked with red color) on label(similar to attached image) and QR code on a single photo taken with mobile camera so label may be in a little bit different position. I'm struggling with choosing and executing good enough solution to accomplish it. I was wondering about labeling images with marked boxes around my area of intersts and then put it into some CNN. I've encounter some interesting paper on text spotting but I'm not sure if it would be good solution for my problem.
I will be more than thankful for any help and suggestions. Thanks in advance :)

If the image will always be the same and your ROI will always be in the same location, then an RNN with CV2 is not necessary, you can mark the areas that you need to extract the texts. I even recommend you more like JaidedAI
/
EasyOCR.
Here is the google colab of the papers you sent: Information extraction


Found input variables with inconsistent numbers of samples I would appreciate if you could let me know how to resolve this error:
Code:
X = np.array(pd.read_csv('my_X_table1-1c.csv',header=None).values)
y = np.array(pd.read_csv('my_y_table1-1c.csv',header=None).values.ravel())
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=7)

def Ridgecv(alpha):
    return cross_val_score(Ridge(alpha=float(alpha), random_state=2),
                           X_train, y_train, 'mae', cv=5).mean()

The error is related to X_train, y_train:
ValueError: Found input variables with inconsistent numbers of samples: [1052, 1052, 3]

It seems that I missed the word "scoring". In fact, the extra 3 was related to the number of characters of 'mae'.

def Ridgecv(alpha):
    return cross_val_score(Ridge(alpha=float(alpha), random_state=2),
                           X_train, y_train, scoring='mae', cv=5).mean()


What method is recommended after outliers removal? I have a data of mice reaction times. In every session, there are some trials in which the mouse "decides of a break" and responds after a long time to these specific trials.
I was thinking of applying outlier removal on my data. and the data does look better (I used a Matlab function which removed all data above of below 3 IQR's above and below median).
After doing that I got a histogram which is more similar to a normal distribution (below an example picture of one of my sessions).
My question is:

After applying my outlier removal, how should I analyze the remaining data?
Should I consider the Median (together with IQR as standard error mean)?
Or should I consider the mean (together with $ \frac{\sigma}{\sqrt n}$ as standard error mean)?

Remark: I have very little knowledge in statistics, so, I there are mistakes above (for example my standard error mean definition as IQR or $ \frac{\sigma}{\sqrt n}$ is not correct), I would be grateful if you'll let me know.
Thanks!
Edit: The purpose of my analysis is to show that under certain conditions, the response times of the mice will be faster then under other conditions.
fig 1: Data before outlier removal

fig 2: Data after outlier removal

based on this information I would recommend to use

Wilcoxon signed-rank test

or

paired Student's t test

This depends on your sample size and distribution. To test if your data set is normally distributed you can use the Jarque-Bera test.

I didn't work with Matlab yet, but  I guess all the tests should be implemented in Matlab.

From that point you could evaluate the impact of the condition on response time.
Visualize your data (X on Y) - Scatter plot / Heatmap (if multivariate data).

From here you could start building a model

starting with linear regression
going into more advanced models/approaches -- random forests, bootstrapping etc.

Added

If you only have binary variables (condition / no condition) - you need to set up dummy variables to perform a regression model (1 / 0) . Data then could look like:

 obs  response-time   any_condition  condition_1   condition_2   condition_3
  1   12.54           1              1             0             0
  2   19.34           0              0             0             0
  3   13.32           1              1             1             0
  4   14.7            0              0             0             0

If you have multiple different "state" conditions, I would recommend to use a control variable ("any_condition") - to see if conditions (not specifying which, have an impact) on the response time.

I hope that helps.


Negative loss, 100% accuracy I have a very small dataset with 567 images. I've used pretrained model resnet50 in transfer learning. But whenever i fit my model it's giving me 100% accuracy , 100% validation accuracy and in per epoch loss and validation loss goes to negative. The evaluation shows me output with 100% accuracy and -5915788.088888888  loss.This is preety unexpected result. What's the matter ? Is it possible or my model is getting bad output ? I have used the model like below :

def Transfer():
    model = Sequential()
    model.add(ResNet50(include_top = False, pooling = 'avg', weights = 'imagenet'))
    model.add(Dropout(0.4))
    model.add(Dense(3, activation = 'softmax'))

    model.layers[0].trainable = False
    sgd = optimizers.SGD(lr = 0.01, decay = 1e-6, momentum = 0.9, nesterov = True)
    model.compile(optimizer = sgd, loss = 'categorical_crossentropy', metrics = ['accuracy'])
return model

categorical_crossentropy should never be negative. The most common reason it becomes negative is incorrect data encoding. It is best if all features and targets are scaled between 0 and 1.


Training a model with Cross-Validation I'm training a model with CV and then I'm testing the predictions on a new test set.
Am I doing the right thing or is it necessary to test the predictions on a new dataset using Cross-Validation?
Thank you!

Everybody understands how to perform $k$-fold cross-validation but there is often quite a lot of confusion about where/how to use it. So thanks for this good question :)
First, cross-validation is a statistical method for evaluation, not for training:

Of course training is performed during cross-validation, but it is performed $k$ times and therefore there are $k$ models produced during the process. Each of these models is meant to be applied only to its corresponding test set, and in general it would be a mistake to use one of these models after the CV process (it would be an even bigger mistake to select the best of these $k$ models). If a final model is needed, it should be trained on the whole training set independently from the CV process.
The purpose of CV is to obtain a reliable evaluation of the performance of a method by minimizing the chance factor caused by splitting the data between training and test set. The variations across folds can also be used to assess the stability of the method: if there are large variations in the performance, the method is not stable and the resulting model is likely to overfit since it depends too much on the training set.

When to use a separate test set?

If the purpose is only to evaluate a single method reliably, then there is no need to train a final model or to evaluate it on a separate test set.
However quite often one uses CV to evaluate not one but multiple methods: with different algorithms, different hyper-parameters values, different features, etc. These cases fall in the category of parameter tuning, which is a kind of training: the goal is indeed to find the optimal values for some parameters, even if the parameters appear to be part of the design (e.g. some preprocessing option). Like any training stage, there is a risk of overfitting: using CV makes the risk quite low, but even with CV the risk exists (especially with a high number of methods). This means that the selection of the best method based on CV performance might be due to chance, this is why the proper methodology in this setting is to (1) select the best method according to CV performance; (2) train the final model using only this method on the full training set; (3) evaluate on a separate test set. This ensures that the performance obtained on the test set is reliable.


How to calculate VC-dimension? Im studying machine learning, and I would like to know how to calculate VC-dimension.

For example:

$h(x)=\begin{cases} 1 &\mbox{if } a\leq x \leq b \\
 0 & \mbox{else } \end{cases} $, with parameters $(a,b) ∈ R^2$.

What is the VC-dimension of it?

The VC dimension is an estimate for the capability of a binary classifier. If you can find a set of $n$ points, so that it can be shattered by the classifier (i.e. classify all possible $2^n$ labelings correctly) and you cannot find any set of $n+1$ points that can be shattered (i.e. for any set of $n+1$ points there is at least one labeling order so that the classifier can not seperate all points correctly), then the VC dimension is $n$.

In your case, first consider two points $x_1$ and $x_2$, such that $x_1 < x_2$. Then there are are $2^2=4$ possible labelings

$x_1:1$, $x_2:1$
$x_1:0$, $x_2:0$
$x_1:1$, $x_2:0$
$x_1:0$, $x_2:1$

All labelings can be achieved through the classifier $h$ by setting the parameters $a<b \in R$ such that

$a<x_1<x_2<b$
$x_1<x_2<a<b$
$a<x_1<b<x_2$
$x_1<a<x_2<b$

respectively. (Actually, $x_1 < x_2$ can be assumed w.l.o.g. but it is enough to find one set that can be shattered.)

Now, consider three arbitrary(!) points $x_1$, $x_2$, $x_3$ and w.l.o.g. assume $x_1<x_2<x_3$, then you can't achieve the labeling (1,0,1). As in case 3 above, the labels $x_1$:1 and $x_2$:0 imply $a<x_1<b<x_2$. Which implies $x_3$ > b and therefore the label of $x_3$ has to be 0. Thus, the classifier cannot shatter any set of three points and therefore the VC dimension is 2.

-

Maybe it becomes clearer with a more useful classifier. Let's consider hyperplanes (i.e. lines in 2D).

It is easy to find a set of three points that can be classified correctly no matter how they are labeled:

For all $2^3=8$ possible labelings we can find a hyperplane that separates them perfectly.

However, we cannot find any set of 4 points so that we could classify all $2^4=16$ possible labelings correctly. Instead of a formal proof, I try to present a visual argument:

Assume for now, that the 4 points form a figure with 4 sides. Then it is impossible to find a hyperplane that can separate the points correctly if we label the opposite corners with the same label:

If they don't form a figure with 4 sides, there are two "boundary cases": The "outer" points must either form a triangle or all form a straight line. In the case of the triangle, it is easy to see that the labeling where the "inner" point (or the point between two corners) is labeled different from the others can't be achieved:

In the case of a line segment, the same idea applies. If the end points are labeled differently than one of the other points, they cannot be separated by a hyperplane.

Since we covered all possible formations of 4 points in 2D, we can conclude that there are no 4 points that can be shattered. Hence, the VC dimension must be 3.


Get the average time between first and 2nd call (postgresql) I have the following data in table where I want to calculate the average time between 1st and 2nd call.
I know how to get the average, but I have a though time to figure out how to subtract the 2nd from 1st attempt since it is in the same column and I am more familiar to subtracting things between columns.

If you had a table containing only the first call details and another table containing only those of the second call you'd be able to join them, right? Then the two times would be in separate columns. You don't have to copy data or split the table to achieve this. It can be done logically by referencing the table twice in the query and using a self-join.
select
    AVG(second_call.time_of_call - first_call.time_of_call)
from TheTable as first_call
inner join TheTable as second_call
    on first_call.customer_id = second_call.customer_id
where first_call.call_attempt = 1
and second_call.call_attempt = 2;

Logically I split the table into two, one called first_call and one called second_call. The former only contains rows for call_attempt 1 and the latter for call attempt 2. This is enforced in the WHERE clause. I'm assuming each customer will only ever have one row for each call attempt number.
Then I join these tables on the customer_id. I use an inner join so only customers that have both first second calls will be processed. It would be a good idea to profile the data to see how this affects the quality of the calculation.
You'll likely want to change the subtraction inside the AVG() to get the format you need.
If run-time performance is critical it may be more run-time efficient to write this using window functions rather than a self-join, depending on what indexes exist and the cardinalities.


Naive Bayes Predict type = 'raw' returning NA I have build a naive bayes model for text classification.It is predicting correctly.But it is returning 'NA' in prediction results if i put 'type = raw'.i have seen some results in stackoverflow to add some noise.when i do that i am getting all A category as 0's and all B category as 1's.How can i get correct probabilities in naive bayes?

library('tm');
library('e1071');
library('SparseM');
Sample_data <- read.csv("products.csv");
traindata <- as.data.frame(Sample_data[1:60,c(1,2)]);
testdata <- as.data.frame(Sample_data[61:80,c(1,2)]);
trainvector <- as.vector(traindata$Description);
testvector <- as.vector(testdata$Description);
trainsource <- VectorSource(trainvector);
testsource <- VectorSource(testvector);
traincorpus <- Corpus(trainsource);
testcorpus <- Corpus(testsource);
traincorpus <- tm_map(traincorpus,stripWhitespace);
 traincorpus <- tm_map(traincorpus,tolower);
 traincorpus <- tm_map(traincorpus, removeWords,stopwords("english"));
traincorpus<- tm_map(traincorpus,removePunctuation);
 testcorpus <- tm_map(testcorpus,stripWhitespace);
 testcorpus <- tm_map(testcorpus,tolower);
 testcorpus <- tm_map(testcorpus, removeWords,stopwords("english"));
 testcorpus<- tm_map(testcorpus,removePunctuation);
trainmatrix <- t(TermDocumentMatrix(traincorpus));
testmatrix <- t(TermDocumentMatrix(testcorpus));
model <- naiveBayes(as.matrix(trainmatrix),as.factor(traindata$Group));
results <- predict(model,as.matrix(testmatrix))

I am assuming that you are referring to this Stackoverflow post that mentions to add noise to the data since the error seems to be coming when there is one (or small) instance of a class in the dataset. Is that the case with the training data? If what you're trying to predict is a rare-event, then a suggestion might be to balance the training data by oversampling the rare class (hence adding noise).

Provided the above is not working, another suggestion is to remove infrequent terms in your term-document-matrix using the function removeSparseTerms.

Going beyond, given the amount of training data you have, it would be good to evaluate if the term document matrix with the words it contains or frequency of specific words is sufficient to differentiate the classes. If not, you should consider adding new features to describe the dataset.

Few suggestions:

count of positive/negative words or a sentiment index that ranges from -1 to 1, if relevant for your data
types of words in dataset (index or count of adjectives or nouns or verbs), again depending on your problem & data
rather than using term-document-matrix, try noun-phrases

Finally, I'm assuming that your test data contains records for both classes. If not, it is difficult to evaluate the model.

Hope that helps. If you could formulate your question more clearly with the data problem and provide some examples of the data, that would help.