Remove noise by clustering on which step of pre-processing is better? I am working on a classification task. The dataset is a UCI data set about machine learning with 200 observations and 2 classes.

Part of my model includes the following preprocessing steps:

remove missing values
normalize between 0 and 1
remove outlier
smoothing
remove trend from data
SMOTE

I would like to use a clustering method to remove noisy data points. The question is, at which step should this happen?

Looking at your different steps, the important thing to do is check which step would be affected by outliers.

Removing missing values is not affected because this step is not dependent on other data points present (or not) in the dataset.
However, normalizing your data is. Indeed, let's say your outliers contain extreme values, this will affect the normalized values of the non-outlier data points.

Therefore, intuitively, I would perform your noise removal at the very start or after step 1.

Ultimately, you should see what works better for your task. Perhaps removing outliers doesn't help as much as you'd expect. Same with your pre-processing. Feel free to experiment!


Undersampling for credit card fraud detection before or after Train/Test Split I have a credit card dataset with 98% transactions are Non-Fraud and 2% are fraud.
I have been trying to undersample the majotrity class before train and test split and get very good recall and precision on the test set.
When I do the undersampling only on training set and test on the independent set I get a very poor precision but the same recall!
My question is :

Should I undersample before splitting into train and test , will this mess with the distribution of the dataset and not be representative of the real world?

Or does the above logic only apply when oversampling?

Do the undersampling after the train/test split and only in the train split: you want to somehow "weigth" your learning algorithm in order to prevent it to be biased towards the majority class, indeed, other techniques can be applied, and always on the train split, what eventually is the data for the learning algorithm. But... your test set should represent the real data distribution. In this case, you will asses if whatever technique you have applied to prevent a biased algorithm towards the majority class has not eventually affected its performance on it.
Some other approaches for this problem (fraud vs non-fraud):

Anomally detection
One-class classifies: One Class SVM


Information leakage when train/test are truly i.i.d.? I am well aware that to avoid information leakage, it is recommended to fit any transformation (e.g., standardization or imputation based on the median value) on the training dataset and applying it to the test datasets. However. I am not clear what is the risk of applying these transformations to the entire dataset prior to train/test split if the the data is iid and the train/test split is indeed random?
For example, if the original data set has certain statistical characteristics(e.g., mean, median, and std) then I would expect a random data spilt with generate a train and test datasets that have the same statistical characteristics. Therefore, standardizing the entire datasets and then  splitting should produce the same results as splitting the dataset, standardizing based on the train database and transforming the test dataset. The same argument can be made for imputation based on the median value.
Am I missing something?

Well, keep in mind that when you standardize/impute data you're estimating parameters. Given the conditions that you've defined and having enough data such that the estimates are good, then I don't think it should matter to use the training data or all the data (as a matter of fact, the estimate of the parameter using training data should be very similar to the parameter using all the data).
However, when the datasets are small these estimates may have high variance, and if you want your cross-validation to be reliable you might want to take it into account. In general, I think for scaling and missing imputation this should not be an issue, but if you're doing some things that are more prone to overfitting like target encoding, or imputing using a model instead of a single parameter, then you should be more careful.
In order to not even think about having enough data or not, the cross-validation framework tries to treat the test data as what the serving data should be when putting the model into production, and then you'll always be alright.


How do I add together multiple columns based on header names? Here's what my dataframe looks like

Server  Performance123      Performance456
server1       1024             0
server2       110              0
server3       0                1024

Here's what I want it to ultimately look like

Server        Tier1
server1       1024
server2       110
server3       1024

Now obviously I could just add the two columns together but I can't be sure what the "123" or "456" part of the CSV I'm importing will look like as it's the last part of the UID of the datastore.

I've tried this

df['Tier 1'] = df.filter(like='Performance')

But I can't assign that as a new column in the dataframe. Any advice? I'm a bit of a newcomer to this so ran out of ideas quickly.

Well after much searching and a lot of errors I have the below working code

df['Tier 1'] = df.groupby(df.columns.str.contains('Performance'),axis=1).sum()[1]


Should I modify my data to reflect time dependence of my rules? Say I have 100 points. Each of these points have a date associated with them and have a "yes" label. I can artificially add "no" points to create more data. Now, I want, using these 100 "yes" points + X "no" points, to predict if a given set of features (a new point) will be categorized "yes" or "no".

I can develop a model that does so.

Now, let's imagine that some "rules of the game" can change. Something that in 1967 was a "yes" point, becomes a "no" point after a (unknown) rule change in 1989. Most factors stay the same, but one change causes the point to change classification drastically.

Can I modify my data to add more importance to recent dates (that is, duplicate recent values so that in "importance", 1 value from 2010 equals 3 values from 1990 and 5 values from 1970 for example), because it's a better representation of our prediction for a future point? Or is this a terrible idea?

Basically, to simplify, if my data is 5 points, labeled "1939", "1967", 1980", "1982", "2010", can I artificially modify it to be 12 points, "1939", "1967", 1980", "1980", "1982", "1982", "1982", "2010", "2010", "2010", "2010", "2010"?

Of course, chosing the "importance" of a given more recent point is pretty difficult to begin with.

Maybe I misunderstood something, but if you have feature vector with N features, it would be ok to add one more feature reflected the time. Then there could be two instances with the same first N features, but the different (N+1)-th and as a result the answer will differ too: "yes" for one year and "no" for another. However, for some instances answer will stay the same, which is also possible. So, I guess your idea might work, if there's actually some connection and meaningful "pattern" in real-life data.


Unable to understand the meaning of following lines of the research paper for image segmentation I am implementing a paper on image segmentation. It is based on the slight modification of the u-net architecture.
The paper is based on encoder and decoder steps
Following are the lines of the paper which I am unable to understand.

In the decoder path, features from the encoder path at the same
resolution are fused through concatenation.

and then the architecture of the network is given-

Layer .  Output size .  Filter Size    Stride Dropout
Inputs .  _ .              _ .           _ .    _
Conv1
Conv2
Max Pool1
Conv3
Conv4
Max Pool2
Conv5
Conv6
Max Pool3
Conv7
Conv8
Max Pool4
Conv9
Conv10
Upsampling1
Conv11
Conv12
Upsampling2
Conv11
Conv12
Upsampling3
Conv11
Conv12
Upsampling4
Conv11
Conv12
Output

I am not adding the other columns values for conciseness.
Anyone who want to read the whole paper, it is this. The required lines are in the Table 1.

Generally, if you look at image segmentation models, they have two main paths, what the author of your paper calls encoder and decoder paths.

The role of the encoder is to contract the size of the image while extracting meaningful information, while the decoder restores the contracted image to its original dimensions. However, a lot of information is lost during contraction. To overcome this issue the decoder takes some features directly from the contraction path via skip connections.

A certain decoder layer's input is actually the concatenation of two things:

The output of its previous layer
The output of the respective layer (the one with the same dimensions) of the encoder

This becomes more clear if you look at the following image:

The left part of the network is the encoder and the right is the decoder. The skip connections are the grey horizontal lines that go directly from the encoder to the decoder.

These can't appear the way you portray the model because you show only the layers and not any information about how they are connected. I think you assume the network should be sequential, but segmentation models usually aren't.


Using a pre trained CNN classifier and apply it on a different image dataset How would you optimize a pre-trained neural network to apply it to a separate problem? Would you just add more layers to the pre-trained model and test it on your data set?

For example, if the task was to use a CNN to classify wallpaper groups, I'm sure that it wouldn't work to directly classify off a pre-trained network trained on images cats and dogs, even though both are image classifiers.

Do Read - When Transfer Learning is Disadvantageous ?(thanks to @media)
(looks very informative to me, so added here to make this answer a complete one...)
Answer to your Question..(starts here)

Transfer Learning is What you are Looking for..

When we are given a Deep Learning task, say, one that involves training a Convolutional Neural Network (Covnet) on a dataset of images, our first instinct would be to train the network from scratch. However, in practice, deep neural networks like Covnet has a huge number of parameters, often in the range of millions. Training a Covnet on a small dataset (one that is smaller than the number of parameters) greatly affects the Covnet’s ability to generalize, often result in overfitting.

Therefore, more often in practice, one would fine-tune existing networks that are trained on a large dataset like the ImageNet (1.2M labeled images) by continue training it (i.e. running back-propagation) on the smaller dataset we have. Provided that our dataset is not drastically different in context to the original dataset (e.g. ImageNet), the pre-trained model will already have learned features that are relevant to our own classification problem.

When to fine tune Models?

In general, if our dataset is not drastically different in context from the dataset which the pre-trained model is trained on, we should go for fine-tuning. Pre-trained network on a large and diverse dataset like the ImageNet captures universal features like curves and edges in its early layers, that are relevant and useful to most of the classification problems.

Of course, if our dataset represents some very specific domain, say for example, medical images or Chinese handwritten characters, and that no pre-trained networks on such domain can be found, we should then consider training the network from scratch.

One other concern is that if our dataset is small, fine-tuning the pre-trained network on a small dataset might lead to overfitting, especially if the last few layers of the network are fully connected layers, as in the case for VGG network. Speaking from my experience, if we have a few thousand raw samples, with the common data augmentation strategies implemented (translation, rotation, flipping, etc), fine-tuning will usually get us a better result.

If our dataset is really small, say less than a thousand samples, a better approach is to take the output of the intermediate layer prior to the fully connected layers as features (bottleneck features) and train a linear classifier (e.g. SVM) on top of it. SVM is particularly good at drawing decision boundaries on a small dataset.

Fine-tuning Techniques

Below are some general guidelines for fine-tuning implementation:

The common practice is to truncate the last layer (softmax layer) of the pre-trained network and replace it with our new softmax layer that are relevant to our own problem. For example, pre-trained network on ImageNet comes with a softmax layer with 1000 categories.

If our task is a classification on 10 categories, the new softmax layer of the network will be of 10 categories instead of 1000 categories. We then run back propagation on the network to fine-tune the pre-trained weights. Make sure cross validation is performed so that the network will be able to generalize well.

Use a smaller learning rate to train the network. Since we expect the pre-trained weights to be quite good already as compared to randomly initialized weights, we do not want to distort them too quickly and too much. A common practice is to make the initial learning rate 10 times smaller than the one used for scratch training.

It is also a common practice to freeze the weights of the first few layers of the pre-trained network. This is because the first few layers capture universal features like curves and edges that are also relevant to our new problem. We want to keep those weights intact. Instead, we will get the network to focus on learning dataset-specific features in the subsequent layers.

You need to train them again in this case as if I am not wrong then the wallpapers is not a class of Image-net models.. It won't be tough to build a model from scratch to do so(preferably a shallower one will also do here..)

Source of my answer is translation from these amazing courses..

CS231N
fast.ai

For Further Readings,

There is a difference between using pre-trained weights and transfer learning.,
Karpathy's  Latest NN's Training Recepie BlogPost..


Sequence Batching in RNNs I'm wondering why sequence batching in RNNs's target value loops back (I'm not sure what you call it), but let's take for example:

We want to learn a sequence of numbers (our input) from 1 to 16:

$$
        \begin{bmatrix}
        1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 & 11 & 12 & 13 & 14 & 15 & 16
        \end{bmatrix}
$$

Batches: 2, Sequence Length: 4

First, we can divide the data to 2 batches:

$$
        \begin{bmatrix}
        1 & 2 & 3 & 4 & 5 & 6 & 7 & 8\\
        9 & 10 & 11 & 12 & 13 & 14 & 15 & 16
        \end{bmatrix}
$$

Then we can divide this into mini batches:

$$
        \begin{bmatrix}
        1 & 2 & 3 & 4\\
        9 & 10 & 11 & 12
        \end{bmatrix}
$$
$$
        \begin{bmatrix}
        5 & 6 & 7 & 8\\
        13 & 14 & 15 & 16
        \end{bmatrix}
$$

Then we need to create targets for the inputs, and intuitively we want to to targets to be the next value of the input, so:

$$
        \begin{bmatrix}
        2 & 3 & 4 & 5\\
        10 & 11 & 12 & 13
        \end{bmatrix}
$$

However, this is not what I usually see, instead I see the last value in a mini batch is swapped with the first value:

$$
        \begin{bmatrix}
        2 & 3 & 4 & 1\\
        10 & 11 & 12 & 9
        \end{bmatrix}
$$

So what is the intuition in doing so?

Since if we want to learn the sequence of 1, 2, 3, 4, but 1 was given as the target for the value 3, so 4 was not learnt but instead of 1.

It makes no sense to re-order inputs in the general case because the order might matter. In your example it does not; you can shuffle the columns as long as the corresponding outputs remain the same.

I've seen the input reversed, which is a less arbitrary transformation than the one you cite, to improve prediction in sequence-to-sequence models, though that's not set in stone, either.


Random forest confusion matrix encountered invalid values I am doing classificaion using random forest classifier in python (scikit learn).
I have many different databases, each one has 33 observations and the prediction is based on 600 columns.
The script is iteration which run the classifier and then create confusion matrix for each.
When I run the script it works but for some databases I get the next error message:

:25: RuntimeWarning: invalid value encountered in true_divide   matrix = matrix.astype('float') /
matrix.sum(axis=1)[:, np.newaxis]

also the results look weird:

I understand that this might happen if I have Null values in my data, but I have used dropna to make sure that there are no null values:
df=df.dropna(axis=0,how='any')

This is my script for the confusion matrix iteration:
for h in dfch:
    print('Hour:',h)
    print('')
    list_dates=dfch[h]['date'].unique()
#     print(h)
#     print(list_dates)
    for d in list_dates:
        print('date:',d)
        print('hour:',h)
        dfhd=dfch[h]
        dfhd=dfhd.loc[dfhd['date']==d]
        print('database size for hour',h,'date',d,'is',len(dfhd))
        X=dfhd.iloc[:, 4:]
        y=dfhd.iloc[:,2:3]
        #split the data
        X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
        #reshape the y_train to fit the the model
        y_train=y_train.values.ravel()
        #fit the model
        rfc.fit(X_train,y_train)
        rfc_pred=rfc.predict(X_test)
        print('')
        # Get and reshape confusion matrix data
        matrix = confusion_matrix(y_test, rfc_pred)
        matrix = matrix.astype('float') / matrix.sum(axis=1)[:, np.newaxis]
        # Build the plot
        plt.figure(figsize=(16,7))
        sns.set(font_scale=1.4)
        sns.heatmap(matrix, annot=True, annot_kws={'size':10},cmap=plt.cm.Greens, linewidths=0.2)
        # Add labels to the plot
        class_names = ['high','medium','low']
        tick_marks = np.arange(len(class_names))
        tick_marks2 = tick_marks + 0.5
        plt.xticks(tick_marks, class_names, rotation=25)
        plt.yticks(tick_marks2, class_names, rotation=0)
        plt.xlabel('Predicted label')
        plt.ylabel('True label')
        plt.title('Confusion Matrix for Random Forest Model')
        plt.show()

        score=rfc.score(X_test,y_test)

My question : How can it happen that it will be divided in 0/null? (there is no null in the databases), and how does it still display the confusion matrix if it fails due to dividing by 0? How canI solve it?

Most probably, sum across one of the rows is coming out as zero in this code
matrix = matrix.astype('float') / matrix.sum(axis=1)[:, np.newaxis]

It is throwing a runtime warning and only that particular cell will be np.inf. Rest all division will be fine. That's why the plot is showing some data.
 You may see the same in this sample code
import numpy as np
matrix = np.array([[1, 2, 3, 0, -6],[6, 7, 8, -4, 10]]) #Sum zero for 1st row

matrix.astype('float') / matrix.sum(axis=1)[:, np.newaxis]

/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:4: RuntimeWarning: invalid value encountered in true_divide after removing the cwd from sys.path.
array([[        inf,         inf,         inf,         nan,        -inf],
       [ 0.22222222,  0.25925926,  0.2962963 , -0.14814815,  0.37037037]])

As a solution, you should add a constant in the denominator.1 will be best
import numpy as np
matrix = np.array([[1, 2, 3, 0, -6],[6, 7, 8, -4, 10]]) #Sum zero for 1st row

matrix.astype('float') / (matrix.sum(axis=1)[:, np.newaxis] + 1)


setting class weights for imbalanced dataset, how using EarlyStopping? I want to train a CNN with Early Stopping (Keras).
The data set is imbalanced, so I have set class_weights to 'balanced' like follows:

class_weights=class_weight.comput_class_weight('balanced', np.unique(y_train),y_train)

I would like also use Early Stopping, and here is the problem.
Which metric for monitor should I use?
Because I have still balanced the data set with the class weights, shouldn't it be ok when using val_loss?

Hoping for hints and thank you in advance

Early stopping is not directly affected by imbalanced data.
Also comput_class_weight effects the training, not the evaluation.
As far as picking a metric for evaluating imbalanced data, it depends on the specific problem. The most common choices are F-score, precision, and recall. It appears that you are using scikit-learn which offers a weighted option, finding average weighted by class support.


Why is my neural network not learning? I am using the Keras library (with Python 3.6) to create a neural network.

My network maintains a constant overall maximum accuracy of 62.5%, over 16 training samples.

In what ways can I increase this accuracy?

Should I increase the number of training samples, or will restricting some of the data in the training samples help? Or something else that I might not know of?

Any help is greatly appreciated.

Here are the layers of my neural network:

# Build neural network
# Neural net with multiple layers
model = Sequential()

model.add(Dense(32, input_dim=17, init='uniform', activation='sigmoid'))
model.add(Dense(64, init='uniform', activation='relu'))
model.add(Dense(64, init='uniform', activation='relu'))
model.add(Dense(64, init='uniform', activation='relu'))
model.add(Dense(32, init='uniform', activation='relu'))
model.add(Dense(16, init='uniform', activation='sigmoid'))
model.add(Dense(4, init='uniform', activation='sigmoid'))
model.add(Dense(1, init='uniform', activation='sigmoid'))

# Compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

# Fit model
history = model.fit(X, Y, validation_split=0.46, nb_epoch=150, batch_size=3)

A neural network is the wrong approach for a problem with a small training set. Even if you only have 2 features that are very representative of your function then 16 feature are not sufficient.

As a very general rule of thumb I use 100 examples for each feature in my dataset. This then increases exponentially with every single different class you expect. 16 instances is not enough to train a neural network. You will always have huge error margins when applying your model on a testing set. Even more problematic is the fact that you are using a very deep neural network. This will require even more training instances to properly learn the function.

I suggest you use a general machine learning technique such as SVM. This will likely result in better result. Try these techniques instead and see what results you get: k-NN, kernel SVM, k-means clustering.

But, be warned 16 training instances is still very little.


Software Testing for Data Science in R I often use Nose, Tox or Unittest when testing my python code, specially when it has to be integrated with other modules or other pieces of code. However, now that I've found myself using R more than python for ML modelling and development. I realized that I don't really test my R code (And more importantly I really don't know how to do it well). So my question is, what are good packages that allow you to test R code in a similar manner as Nose, Tox or Unittest do in Python. Additional references such as tutorials will be greatly appreciated as well.

Bonus points for packages in R similar to

Hypothesis

or
Feature Forge

Related Talk:

Trey Causey: Testing for Data Scientists

Packages for unit testing and assertive testing that are actively maintained:
Packages for unit testing

testthat: more information on how to use you can find here or on github
Runit: Cran page

Packages for assertions:

assertthat: info on github
assertive: Assertive has a lot of subpackages available in case you do not need all of them. check on cran
assertr: info on github
ensurer: info on github
tester: info on github

It is a matter of preference what you want to use for assertions.
Read this bioconductor page for more info on the difference between RUnit and testthat.


How can I measure if a population has the same distribution as other? Is there a population similarity index of some kind which could help me determine if two populations in two different datasets are the same or at least similar?

The datasets have the exact same number of variables.

I would like a measure which assess the similarity variable-by-variable, but an overall measure works too.

The context of my question is that I would like to know if a variable is suitable or not for using in a classification method given the similarity of it between populations.

What I'm gonna say might seem too simple, but I assume, it might not be bad to fit a multivariate distribution (like Gaussian) to each distribution and then figure out what the mean and covariance matrix are. Mean might not depict so much information but the variance and correlation that are exposed in the covariance matrix might be helpful.

For the case of determining if a variable is useful or not, analyzing its correlation with other parameters might turn out to be useful.

(These are my own thoughts)


3D visualisation and post analysis tool I am looking for a 3D plotting and post analysis tool, specifically which can generate figures like the following. It is preferred the tool has a Linux distribution. The figure is copied from here.

Plotly is an excellent tool for 2D and 3D data. It is quite simple to use with python as well.
https://plotly.com/python/


What model is suitable for classification of a small data set? I have a dataset that consists of 365 records, and I want to apply a classification model on it (binary classification).

As an output, in addition to the classification labels, I want to retrieve the classification confidence for each instance.

I don't know how to deal with such a case. Can I use, for example, linear classifiers (SVM, logistic regression) with this small dataset? Because, I want to retrieve the classification confidence as well.

I read that decision trees can be a good classifier for small datasets, but how can I retrieve the classification confidence with it?

The dataset consists of tweets, each classified as positive or negative (from a sentiment perspective), and my feature vector consists of 2400 features (combination between word2vec embeddings and other features).

Also, do you recommend me to use word2vec embeddings with such a small dataset? I think the classifier can't learn something from them using small dataset.

The question whether to use a linear classifier depends less on the number of samples you have in your dataset and more whether your dataset is linearly separable (by the way, SVMs can be non-linear with the kernel trick).

Now with regards to confidence in the classification, In SVMs there is a method that calculates the probability that a given sample belongs to a particular class using Platt scaling ("Original Paper"). This is the approach that is used in sklearn's SVM confidence implementation. You can read more about it in the following link:

How To Compute Confidence Measure For SVM Classifiers

In both SVMs and linear regression models you can calculate the distance of a sample from the border and treat it as a confidence measurement (but it is not exactly that).

With decision trees I'm not an expert but a similar question was posted and answered in the following link:

Decision tree, how to understand or calculate the probability/confidence of prediction result

I would strongly recommend using some known embedding method like the word2vec, since as you mentioned, your dataset is too small for your model to be able to properly learn an encoding of context and vocabulary from.


Logistic regression cost function In Aurelien Geron's  book I found this line

This cost function makes sense because –log(t) grows very large when t approaches
 0, so the cost will be large if the model estimates a probability close to 0 for a positive instance, and it will also be very large if the model estimates a probability close to 1
for a negative instance. On the other hand, – log(t) is close to 0 when t is close to 1, so
the cost will be close to 0 if the estimated probability is close to 0 for a negative
instance or close to 1 for a positive instance, which is precisely what we want.

What I dont get is, How will  the cost will be large if the model estimates a probability close to 0 for a positive instance, and it will also be very large if the model estimates a probability close to 1
for a negative instance?

The cost function of the Logistic Regression derived via Maximum    Likelihood  Estimation:

If y = 1 (positive): i) cost = 0 if prediction is correct (i.e. h=1), ii) cost $\rightarrow \infty $ if $h_{\theta}(x)\rightarrow 0$.
If y = 0 (negative): i) cost = 0 if prediction is correct (i.e. h=0), ii) cost $\rightarrow \infty$ if $(1-h_{\theta}(x))\rightarrow 0$.

The intuition is that larger mistakes should get larger penalties.
Further readings, 1,2,3,4.


Compare cross validation and test set results I am having a hard time understanding the results of a cross validation test and a test run on a test set.
First I made the following pipeline:
pipe=Pipeline([('clf',DecisionTreeClassifier(random_state=0))])
Then I use cross validation on a scaled training set(75% of the original dataset):
>>> cross_val_score(pipe, X_train_scaled,Y_train,cv=7).mean()
0.7257796129913106
I then fit the pipeline with the training data and run the classifier on the training data.
>>> pipe.fit(X_train_scaled,Y_train)
>>> pipe.score(X_train_scaled,Y_train)
0.7734339749126584
Finally, I checked the models performance on the test set:
pipe.score(X_test_scaled, Y_test)
0.941353836876225
Question 1: have I done the right steps, do I even need to run the pipeline on the training data for the training data score?
Question 2: why is the test data so much more accurate than the cross validated one. Is the data underfitted, or is it okay for this to happen ?

At first sight the steps seem to be correct, nevertheless:

you did not tell how you split your dataset intro training and test sets, in case it has some influence in your final score values
you might be interested in the sklearn GridSearchCV option , with which you carry out cross validation similar to what you did but you do not need to manually refit the model on the whole train dataset, because you have the option with the refit parameter (choose the scoring metric considered so as to select the best model from the grid search cross-validation process already refit on the whole train dataset wothout splits), as follows:
Basically, the steps for your example could be:

split your dataset to leave some rows out for a final validation

apply grid search cross-validation on the rest of the data (your train set), something like:
dt_clf = DecisionTreeClassifier(random_state=0, class_weight="balanced")
hiperparams_search_space = {'criterion': ["gini", "entropy"], 'max_depth': [4,
5, 6, 8], 'min_samples_leaf': [2, 3, 5]}
dec_tree_cross_val_clf = GridSearchCV(dt_clf, hiperparams_search_space, cv=10,
scoring=['accuracy', 'recall', 'precision', 'roc_auc'],
refit='recall',return_train_score=True, n_jobs=-1)
dec_tree_cross_val_clf.fit(X_train, y_train)
where for each scoring metric, you would have train and test values across your k-folds in a dataframe via dec_tree_cross_val_clf.cv_results_:

use clf.best_estimator_ to make predictions on your validation set to calculate a final score metric:
dec_tree_cross_val_clf_best_est = dec_tree_cross_val_clf.best_estimator_


Keras cNN Transfer Model: Reduce Final Model Size? I'm working with multiple cNNs to be ran on mobile devices.  If I create these cNNs from scratch (black n white, 256x256), I'm able to produce a binary classification model of about 10mb, which is a size that would work great on a model device.

However, using transfer learning such as VG16 Imagenet gives my much better classification accuracy.  But, the model size is closer 100mb!  That is 10x as big as from scratch.  I have to take those grayscale images from scratch and convert them to 3 channel, which accounts for some of the size.  Also, freezing less layers, like 10 instead of 14 reduces the size, but very slightly.

Any suggestion on a way I can still leverage Keras Transfer learning and produce a much smaller model (like a way there a transfer learning model that is B&W)?

Here is my Transfer baseline model:

def baseline_model_func():
vgg_model = VGG16(weights='imagenet', include_top=False, input_shape=(input_dim, input_dim, 3))

for layer in vgg_model.layers[:10]:
    layer.trainable = False

x = vgg_model.output
x = Flatten()(x) # use global pooling
x = Dense(1024, activation="relu")(x)
x = Dropout(0.5)(x)
x = Dense(1024, activation="relu")(x)
predictions = Dense(1, activation="sigmoid")(x)
model_final = Model(inputs = vgg_model.input, outputs = predictions)
model_final.compile(loss = "binary_crossentropy", optimizer = optimizers.SGD(lr=0.0001, momentum=0.9),
                    metrics=["accuracy"])
return model_final

`

Here is my cNN from scratch baseline model:

def baseline_model_func():
model = Sequential()
model.add(Conv2D(filters=16, kernel_size=2, padding='same', activation='relu',
                        input_shape=(input_dim, input_dim, channel_numbers)))
model.add(MaxPooling2D(pool_size=2))
model.add(Conv2D(filters=32, kernel_size=2, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=2))
model.add(Conv2D(filters=64, kernel_size=2, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=2))
model.add(Conv2D(filters=128, kernel_size=2, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=2))
model.add(Conv2D(filters=256, kernel_size=2, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=2))
model.add(Conv2D(filters=512, kernel_size=2, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=2))
model.add(Dropout(0.3))
model.add(Flatten())
model.add(Dense(250, activation='relu'))
model.add(Dropout(0.4))
model.add(Dense(500, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss = "binary_crossentropy", optimizer = "adam", metrics=["accuracy"])
return model

```

There's actually some pretty interesting research on this topic. Key words would be model compression or CNN pruning (doing things like reducing the model size by removing filters with low activations).

Another interesting method uses a technique called teacher-student networks. Basically, this involves training the big network (the teacher) and using the output of the teacher as the labels when you train the small student network. This usually serves to improve the student over training with the ground truth.

For example, if you were training on MNIST, you'd have 10 classes and a softmax output of the teacher network that might look like [0, 0, 0, 0, 0.33, 0, 0, 0, 0, 0.67], in the case of a 9 that looks like a 4, when the original ground truth was [0, 0, 0, 0, 0, 0, 0, 0, 0, 1].

I'm not totally current on this research so give these a google first, there might be some better best practices out there. Hope this at least gets you going in the right direction.


How can we extract fields from images? I am making an document parser which extracts data fields from the documents and store them in a structured way. Each field in my dataset is horizontal which is easy to extract.

But the model fails on following type of example -

Is there any way to extract invoice number and date from such images.

I have a similar use-case and a working product based on tensorflow object-detection api and pytesseract for OCR. On top of the extracted text, I perform regex for validation of the extracted information and cleaning it to meet requirements of other processes.

Steps:
1. Annotate images with some tool like labelimg.
I annotated a set of 1K images, similar to yours, with 23 different classes. The dataset is unbalanced with some classes appearing almost in every image to some classes appearing in only as few as 60. However, there are ways to ensure that this imbalance does not affect the performance of the network.
2. Choose a model from tf model zoo (I use this frcnn model) and retrain the last two layers using transfer learning.
3. Export the inference graph, perform object detection to identify the region of interest, and run OCR on the region of interest to extract the text.
I'd recommend storing the extracted data in a dictionary with class of the object as key and the extracted text as value.
4. Finally, have regex validate the text in the extracted field and perform any manipulation/transformation that is necessary.

The trained model can be deployed to production with help of tfserving. The same trained network can be deployed into a mobile app as well - look for tutorials on tensorflowlite for this.

Hope my answer helps you! I had a tough (but interesting) time gathering the knowledge required to get a production grade product that currently serves hundreds of request everyday. I would recommend reading completely all the links I have shared in this answer, and feel free for more questions. Good luck!


Calculate coefficient w* I'm learning ML from Bishop's book. But I don't know that How should I calculate w* in the below picture.

Just looked into the book. It was an example of the data that is presented in Fig 1.4. Numpy is a good package to derive the sum of squares fit to the data. This is an optimization problem. Look at the notes section of numpy.polynomial.polynomial.polyfit. I think the data is not given (you can digitize it!). The book is showing some qualitative behavior to give some intuition.


Filtering outliers in Apache Spark based on calculations of previous values I'm processing geospatial data using Spark 2.0 Dataframes with the following schema:

root
 |-- date: timestamp (nullable = true)
 |-- lat: double (nullable = true)
 |-- lon: double (nullable = true)
 |-- accuracy: double (nullable = true)
 |-- track_id: long (nullable = true)

I have seen that there are jumps of the location signal to a complete different place. The strange thing is, that the signal then remains for a certain time, say aound 25 seconds or 5 samples at the remote location and then jumps back to where I stand.

I'd like to remove these outliers by calculating the speed between the current and the "last valid record" by calculating the speed between the points.
If the speed is above a given threshold, the current record should be dropped and the "last valid record" remains the same.
If the speed is below the threshold the current record is added to the result data frame and becomes the new "last valid record".

I'm using Spark 2.0 with Dataframes.

Any suggestions of how to implement this strategy or any better strategy are highly appreciated. Thanks.

PS: I asked the same questions in stackoverflow, with a concrete implementation. But, since I'm not sure if this is the right approach, and do not want bias the answers to a certain Spark method, I ask here for any suggestions.
https://stackoverflow.com/questions/41002844/how-to-filter-outlier-rows-from-spark-dataframe-based-on-distance-to-previous-va

This is actually a general problem with time-series data: you have some logic to implement based on one or more values in the series.  You always have two choices:

Feed the time series through some module that calculates as each data point arrives
Use the "spreadsheet method" to calculate a series of columns eventually arriving at the goal

The advantage of the first approach is you can use the same module to process your real-time data.  The advantage of the second approach is that it's very fast and usually easier to implement.

Since you're already in a Spark Dataset, here's the strategy:

Calculate a speed column: $p_t - p_{t-1}$ where $p$ is the position
Calculate a "jump" column: 1 if the speed is over a certain threshold, -1 if under, 0 otherwise
Calculate a "jumpsum" column: the cumulative sum of the jump column
Bad data will have a jumpsum of 1; filter them out

Here's how you do it:

import org.apache.spark.sql.expressions.Window
import org.apache.spark.sql.functions._
import org.apache.spark.sql.SparkSession

val ss: SparkSession = SparkSession.builder.getOrCreate()

// note the file must be on each executor in the same directory
val ds = ss.read
   .option("header", "true")
   .option("inferSchema", "true")
   .csv("file:///home/peter/data.csv")

val w = Window.partitionBy().orderBy("datetime")
val threshold = 10
def jump(v: Double): Int = if (v > threshold) 1 else if (v < -threshold) -1 else 0
val sqlJump = udf(jump _)

val cleanDS = ds
    .withColumn("speed", $"position" - lag($"position", 1).over(w.rowsBetween(-1, -1)))
    .withColumn("jump", sqlJump($"speed"))
    .withColumn("jumpsum", sum($"jump").over(w.rowsBetween(Long.MinValue, 0)))

Here's what the output Dataset looks like (I didn't remove the bad rows so you can see the calculation):

+--------+--------+-----+----+-------+
|datetime|position|speed|jump|jumpsum|
+--------+--------+-----+----+-------+
|       1|       1| null|null|   null|
|       2|       1|    0|   0|      0|
|       3|       1|    0|   0|      0|
|       4|       1|    0|   0|      0|
|       5|       1|    0|   0|      0|
|       6|       2|    1|   0|      0|
|       7|       1|   -1|   0|      0|
|       8|       1|    0|   0|      0|
|       9|      46|   45|   1|      1|
|      10|      45|   -1|   0|      1|
|      11|      48|    3|   0|      1|
|      12|      45|   -3|   0|      1|
|      13|       1|  -44|  -1|      0|
|      14|       2|    1|   0|      0|
|      15|       1|   -1|   0|      0|
+--------+--------+-----+----+-------+

The "data.csv" is just the first two columns of that Dataset:

datetime,position
1,1
2,1
3,1
4,1

...etc.

All that's left to do is filter out jumpsum === 1.


Need help in deriving Policy Evaluation (Prediction) Policy Evaluation is computing the state-value function for an arbitary policy $\pi$.(suton & barto book).
Now

\begin{equation}
v_{\pi} = E_{\pi}[\;G_{t}\,|\;S_{t}=s] \qquad\qquad\qquad\qquad(4.1)\\\
        =  E_{\pi}[R_{t+1}+\gamma G_{t+1} |S_{t}=s]\qquad\qquad\quad(4.2)\\
        =  E_{\pi}[R_{t+1}+\gamma v_{\pi}(S_{t+1})| S_{t}=s]\qquad\qquad(4.3)\\
        = \sum_{a} \pi(a|s) \sum_{s^{'},r} p(s^{'},r|s,a)[r+\gamma v_{\pi}(s^{'})]\quad(4.4)
\end{equation}

Can someone help me to figure out from (4.3) to (4.4). And If equation (4.4) is bellman's equation for state value function, then how is it different from below equation
\begin{equation}
v_{\pi} = \sum_{a\in A} \pi(a|s)[R_{s}^{a}+\gamma \sum_{s'\in S} P_{ss^{'}}^{a}v_{\pi}({s^{'})}]
\end{equation}

Going from step (4.3) to (4.4) is turning the expectation (dependent on following policy $\pi$) into a more concrete calculation. To do this, you must resolve your random variables ($R_{t+1}$ and $S_{t+1}$) into the specific values from the finite sets of $\mathcal{R}$ and $\mathcal{S}^+$ - where individual set members are noted $r$ and $s$.

Recall that $\pi(a|s)$ is the probability of taking action $a$ given state $s$, so by taking a sum of any function depending on $(s,a)$ e.g. $\sum_{a} \pi(a|s)F(s,a)$ you will get the expected value of $F(s,a)$ starting from state $s$ and following policy $\pi$.

So you could write (4.3b) as:

$$v(s) = \sum_{a} \pi(a|s)\mathbb{E}_{\pi}[R_{t+1}+\gamma v_{\pi}(S_{t+1})| S_{t}=s, A_t=a]\qquad\qquad(4.3b)\\$$

Similarly, now you have written the expectation when given $s$ and $a$ (so you have all the possible cases of $a$ available to work with), you can use the transition and reward probabilities to fully resolve the expectation and substitute the random variables ($R_{t+1}$ and $S_{t+1}$) for their MDP model distributions described in $p(s',r|s,a)$, leading to equation (4.4). E.g.

$$v(s) = \sum_{a} \pi(a|s)\sum_{s',r} p(s',r|s,a)\mathbb{E}_{\pi}[R_{t+1} + \gamma v_{\pi}(S_{t+1})| S_{t+1}=s', R_{t+1}=r]\qquad\qquad(4.3c)\\$$

$$\qquad = \sum_{a} \pi(a|s)\sum_{s',r} p(s',r|s,a)(r + \gamma v_{\pi}(s'))\qquad\qquad(4.4)\\$$

Your last equation is a slightly different formulation of the same result, where:

$R_s^a$ is the expected reward when taking action $a$ in state $s$. It is equal to $\sum_{s',r} p(s',r|s,a)r$. It is also equal to $\mathbb{E}[R_{t+1}|S_t=s, A_t=a]$, by definition and independently of $\pi$.
$P_{ss'}^a$ is the state transition probability - the probability of ending up in state $s'$ when taking action $a$ in state $s$. It is equal to $\sum_r p(s',r|s,a)$ given $s'$

Technically using $R_s^a$ loses some information about the dynamics of the underlying MDP. Although not anything important to reinforcement learning, which deals with maximising expected reward, so in some ways it is more convenient to already characterise the MDP with expected rewards.


How to choose inputs to maximize reward for an ordered dataset? Consider the following problem, simplified:

Let's say that you have some data where you have only 5 columns/attributes.
One of them can be seen as a reward. How well we performed at a given run.
The other two attributes, let's say x and y are our input to the system. A human hand picked those x, y values and they were recorded.
Another two attributes are observations from sensors. These are not inputs and we have no control over them. Let's call them ob1 and ob2.
All are real values (nothing is discrete).

One idea is to look at this dataset as an ordered dataset.
We are asked to choose input values x and y that maximize our reward.

We have available ~70.000 instances of this five-tupled dataset

This is one approach that comes into mind but not sure if it is correct in principle or the simpler one.
We could build a predictor which takes as input the four attributes (x, y, ob1, ob2) and has as target the reward.
Then try a repetitive process for the inputs by using reinfocement learning (?) to get the inputs that maximize the reward?..

It seems like a reinforcement learning question with continuous action space. (ob1, ob2) is the observation, (x, y) the action, and reward the reward.

You may refer to this paper from DeepMind, which provides a general deep Q-learning way solving this type of question, including some physics tasks.

However, as I don't have your data, I'm not sure if you have sufficient instances. ~70,000 instances with 4 features is (roughly) sufficient for general ML problems, but in reinforcement learning world, people use a simulation environment which is able to generate infinite samples, thus ~70,000 instances may be the bottleneck.

If you encounter such problem, you may try classic reinforcement learning (without "deep") techniques.


Plotting libraries for Scala on Zeppelin My main question is it looks like Zeppelin limit the display of the results to on 1000, I know that I can change this number but when I change it Zeppelin become slow. And it looks like the default plotting tool of Zeppelin also plot the first 1000 results.
Is there a configuration or a way to make the plotting tool plot all the data?

If no, is there any equivalent for Matlibplot to Scala on Zeppelin?

Vegas is a library that strives to substitute for matplotlib in Zeppelin/Spark environments. It works on Spark DataFrames.


Why is 10 considered the default value for k-fold cross-validation? I understand very well what k-fold cross-validation is. In my studies, and at work, I've always heard something along the lines of:

  We most often use k=10 because evidence shows it's the best value for k. Smaller values don't give as good estimates, and larger values don't provide much better results either.

I intuitively can wrap my head around this. However, I cannot seem to find any research that went into declaring k=10 as the default. How would one go on to prove that 10 is the best value?

I understand the effects of using a smaller k versus a larger k in terms of bias. But why is it 10? And no 5 or 20? How could one end up on the number 10?

The statement:

  We most often use k=10 because evidence shows it's the best value for k. Smaller values don't give as good estimates, and larger values don't provide much better results either.

Is just categorically false. The reason people default to K=10 is because they don't know how changing K effects their estimates of the generalization error and they (like you) heard somewhere along the line that K=10 was good.

To understand what makes a good value of K (and whether or not K=10 is in fact better than say K=9 or K=11) you need to understand what changing this value has on your estimate.

As K decrease the bias in your estimate increases. This is because with lower values of K you are training on less data. For example K = 2 trains on only half the data, thus you will have a pessimistic bias in your estimate since you've decreased the amount of data available for your model to learn from. K = 3 trains on two thirds of your data, more data available to train on, better performance.

It used to be thought that there was a bias/variance trade-off in that a decrease in K would cause a decrease in variance (to go along with your increased bias) and while this is partially true it does not always hold. Lower values of K will have lower variance due to the fact that your training sets are less correlated. Think of the extreme example where K = N (leave one out). All of the training sets will look extremely similar, meaning that the estimate you receive is highly dependent on the sample you have to train on. If you were to draw many samples from the population and estimate error using leave one out you would have large variance in your estimates because of the variance between samples. This was the original reasoning for believing there existed a bias/variance trade-off with the choice of K.

However, this post outlines that this is not the case and that there is no universal truth as to what happens to the variance as K increases or decreases. Some studies show the variance increases with K, some show it decreases with K.

https://stats.stackexchange.com/questions/61783/bias-and-variance-in-leave-one-out-vs-k-fold-cross-validation/357749#357749

The other thing to consider is computational complexity. If you are dealing with datasets with millions of records it may be infeasible to use a very large value of K, especially if you are doing nested and/or repeated cross validation. So many people make their choice of K based simply on time to compute.


Clustering with constraints I have a dataset of this form:

chrX posX labelX

where chrX refers to the chromosome number, posX refers to location, and labelX is a categorical variable with 3 labels.
For example:

chr1 3223 1
chr1 3200 2
chr1 3100  1
chr1 1000 2
chr2 1000 1
chr1 3210 3

I want to be cluster these data with certain constraints (thus, this is no longer what 'clustering' typically is), such that, for each chromosome, the sites which are within, say $\pm 30$ positions are together, the constraint being they should have unique labels.

For example, an expected output would be:

chr1 : 3223(1) 3200(2) 3210(3) Notice how sites like chr1 3100 1 are not part of the 'cluster'.

One approach would be to simply calculate pairwise differences between all such sites, and grow the cluster taking care of unique labels.

Are there more elegant approaches to solve such problems?

Each clustering algorithm in my knwledge is based on distance, which is calculated as pairwise differences. So I'm not sure if it is easy to find a general more elegant approach.

In you case you can calculate the distance and than override it based on your constraint.

Here an simple example reduced to one numerical dimension with distance threshold 30.

 nlst <- c(1,3,5, 36,39,42, 1001,1003,1005, 1036,1039,1042)

 d <- dist(nlst, method="euclidean")
 dr <- as.dist(ifelse(as.matrix(d) > 30, NA, 0))

     1  2  3  4  5  6  7  8  9 10 11
 2   0
 3   0  0
 4  NA NA NA
 5  NA NA NA  0
 6  NA NA NA  0  0
 7  NA NA NA NA NA NA
 8  NA NA NA NA NA NA  0
 9  NA NA NA NA NA NA  0  0
 10 NA NA NA NA NA NA NA NA NA
 11 NA NA NA NA NA NA NA NA NA  0
 12 NA NA NA NA NA NA NA NA NA  0  0

A scan through the points collecting those with distance zero will give the result clustering.

This should be always possible if your constraints are local (i.e. depending only on the two points for which you calculate the distance).


How to account for reduced student capacity when calculating program retention? I work at a nonprofit youth center that has 2 distinctive programs throughout the year. Our afterschool program runs during the school year, and our summer camp program runs during the summer.
I am working on calculating our student retention between Afterschool 2019-20 and SummerCamp 2020 to see which students we retained, however, Between AS2019-20 and SC2020, we reduced our maximum student capacity by 10. I figure that this definitely impacts the retention rate calculation, but I have no idea how to account for this reduction in capacity. Does this impact anything, or am I overthinking?
I have to assume it would be some sort of calculation like returned students / eligible students = retention and then maybe dividing that percentage by the quotient of SC2020 capacity / AS2019-20 capacity = proportion to account for the proportion. Is this correct?

Rationale
Some of the terms are a little vague, particularly what you refer to as eligible students and returned students. I'll set some variables for clarity, but tell me if I defined them incorrectly.
I assume them to mean:

eligible students $ = A $ being the set of all students in the after-school program 2019-2020

returning students $ = A\cap S $ where $S$ is the set of all of summer camp 2020 students
Now we can define retention rate to be $\frac{|A\cap S|}{|A|}$. I base this on the US government definition of university student retention rate, which I think is pretty similar, but please correct me if it's not.

The reasons as to why you might have $|A\cap S|<|A|$ are irrelevant.

Example
Let's say you have $|A|=100$ so that, in your situation, $|S|=90$. For simplicity, let's also say that there are no newcomers to the mix.
The retention is, quite literally, how many students you retained. If you retained $90$ students, then the retention rate is $90/100$. Even though $|S|=|A|-10$, the number of students you retained $|A\cap S|$ as a percentage of the number of students in the original program $|A|$ is still $90\%$.
It wouldn't make sense to normalize this to $100\%$.


Survey papers on Natural Language Understanding What are some good survey or review papers on the state of the art of machine learning approaches to grammatical/syntactic structure of natural languages?

“Grammar as a foreign language” (2015) by Vinyals et al. is not a review paper but is a significant breakthrough by applying sequence-to-sequence deep learning.

"Advances in natural language processing" (2015) by Hirschberg and Manning provides a general overview, including parsing.

"Recent Trends in Deep Learning Based Natural Language Processing" (2018) by Young et al. covers popular deep learning approaches and lists current benchmarks for parsing.


Gaussian Naive Bayes (GaussianNB) classifier not working with large number of features I'm trying to make a partial fitting with GuassianNB here's small snippet of my code

classfier = GaussianNB()
classfier.fit(X_init, y_init)
for i in range(X_train.shape[0]):
    if('some condition'):
        classfier = GaussianNB()
        classfier.partial_fit(X_train[i, :].reshape(1, -1), y_train[i].reshape(1, -1).ravel(), classes = np.unique(y_train))
    else:
        classfier.partial_fit(X_train[i, :].reshape(1, -1), y_train[i].reshape(1, -1).ravel())

Everything works correctly until the condition is true, it seems that the classifier for some reason stops learning and starts making arbitrary predictions (predicts all zeros)

tn, fp, fn, tp = confusion_matrix(y_test, pred).ravel()
print(tn, fp, fn, tp)

>>> 1324 0 1031 0

I thought that maybe my classifier made predictions after reinitializing it (the condition is satisfied right before the loop ends) but after I checked it, it made at least 900 partial fittings before the loop ends with a both 0 and 1 labels.

I'm so confused, what's happening? Thanks

Edit: I figured out what's wrong

it turned out that the problem was that my dataset has so many features (around 40000 as it was originally a string of reviews and vectorized it with tfidf) And for some reason that made GaussianNB classifier break down. I'm trying to know why GaussianNB doesn't with a big number of features. If anyone knows, I would appreciate the help. Thanks

It is not the number of features that is the problem with Gaussian Naive Bayes (GaussianNB). It is the decision boundary that GaussianNB is learning. Naive Bayes is constrained to the learn the marginal distribution of the data because "naive" assumption. Often times the conditional distribution is useful to make predictions. Given the performance of GaussianNB on that dataset, it might make sense to apply a different classifier that can learn a different decision boundary.


What are the disadvantages of having a left skewed distribution? I'm currently working on a classification problem and I've a numerical column which is left skewed. i've read many posts where people are recommending to take log transformation or boxcox transformation to fix the left skewness.

So I was wondering what would happen If I left the skewness as it is and continue with my model building? Are there any advantages of fixing skewness for classification problem (knn, logistic regression)?

There are issues that will depend on specific features of your data and analytic approach, but in general skewed data (in either direction) will degrade some of your model's ability to describe more "typical" cases in order to deal with much rarer cases which happen to take extreme values.
Since "typical" cases are more common than extreme ones in a skewed data set, you are losing some precision with the cases you'll see most often in order to accommodate cases that you'll see only rarely. Determining a coefficient for a thousand observations which are all between [0,10] is likely to be more precise than for 990 observations between [0,10] and 10 observations between [1,000, 1,000,000]. This can lead to your model being less useful overall.
"Fixing" skewness can provide a variety of benefits, including making analysis which depends on the data being approximately Normally distributed possible/more informative. It can also produce results which are reported on a sensible scale (this is very situation-dependent), and prevent extreme values (relative to other predictors) from over- or underestimating the influence of the skewed predictor on the predicted classification.
You can test this somewhat (in a non-definitive way, to be sure) by training models with varying subsets of your data: everything you've got, just as it is, your data without that skewed variable, your data with that variable but excluding values outside of the "typical" range (though you'll have to be careful in defining that), your data with the skewed variable distribution transformed or re-scaled, etc.
As for fixing it, transformations and re-scaling often make sense. But I cannot emphasize enough:
Fiddling with variables and their distributions should follow from properties of those variables, not your convenience in modelling.
Log-transforming skewed variables is a prime example of this:

If you really think that a variable operates on a geometric scale,
and you want your model to operate on an arithmetic scale, then log
transformation can make a lot of sense.
If you think that variable operates on an arithmetic scale, but you
find its distribution inconvenient and think a log transformation
would produce a more convenient distribution, it may make sense to
transform. It will change how the model is used and interpreted,
usually making it more dense and harder to interpret clearly, but
that may or may not be worthwhile. For example, if you take the log of a numeric outcome and the log of a numeric predictor, the result has to be interpreted as an elasticity between them, which can be awkward to work with and is often not what is desired.
If you think that a log transformation would be desirable for a
variable, but it has a lot of observations with a value of 0, then
log transformation isn't really an option for you, whether it would
be convenient or not. (Adding a "small value" to the 0 observations
causes lots of problems-- take the logs of 1-10, and then 0.0 to
1.0).


Any pythonic way to auto determine imbalance class problem, specially in multiclass scenario? A data is imbalanced if a target class proportions are unequal and typically, heavily biased. But, what is the exact measurement of this heavy bias?
Before applying imbalance techniques (SMOTE, ADASYN, Tree classifiers,  etc.), I want to auto detect if problem belongs to imbalance class problem.

Depending on your dataset, I would say you can set the threshold yourself.
For example, a 50% split is a perfect balance so by definition any deviation away would be an imbalanced problem. That being said, this is quite typical, so it would make more sense to set your threshold to be somewhere around 80% imbalance or higher.
Count your classes and set a threshold, e.g.
threshold = 80
imbalance_measure = (minority_class/majority_class)*100
if imbalance_measure >= threshold:
    print("Imbalanced problem")
else:
    print("Balanced problem")

In fact, if you data is suitable you can test this yourself through cross-validation
Some further relevant documents here


Determine how each feature contribute to XGBoost Classification so for a summary of what I have done:

My dataset has 5 classes and 10 parameters. I used XGBclassifer from sklearn to investigate if I could use those 10 parameters to predict the class of each data point. After training and fitting the XGBclassifier, I checked feature_importances_ and found out that 2/10 parameters played a key role in the classification.

So my question is:

Can I find out exactly how those 2 parameters contribute to the classification of each specific class? For example, can I find the cut-off values for parameter 1 and parameter 2 that will result in the prediction of class 1?

I am thinking of performing unsupervised clustering using those 2 parameters and k value = 5. Afterwards, I can just eyeball the approximate cutoff values. However, I worry that the 5 clusters will not correspond closely to the 5 groups.

Thanks a lot in advanced

If you have just then parameters and two of them are important. You can plot the trees and see the threshold for each of the parameters.

from xgboost import XGBClassifier
from xgboost import plot_tree
import matplotlib.pyplot as plt

# fit the model
model = XGBClassifier().fit(X, y)
# plot single tree
plot_tree(model)
plt.show()

The above code just plots the first tree. You can plot, for example, 4th boosted tree in sequences using the following line of code

plot_tree(model, num_trees=3)

Note that in each tree, you might have a different threshold for each parameter in an ensembling method.


Pandas DataFrame: Aggregating multi-level groups by matching keys I have some data that looks like this;

data.head()

    stock  date  binNum    volume
0  stock0  d120       2  249500.0
1  stock0  d120       3   81500.0
2  stock0  d120       4   79000.0
3  stock0  d120       5  244000.0
4  stock0  d120       6  175000.0

I can get the average volume for each bin across all days for a particular stock with the following code;

stock0 = data[(data['stock'] == 'stock0')]
binGroups = stock0[['binNum', 'volume']].groupby('binNum', sort=False)
stock0vol = binGroups.aggregate({'volume': np.mean}).reset_index()
stock0vol.head()

   binNum         volume
0       2  174095.238095
1       3  100428.571429
2       4   79880.952381
3       5   73642.857143
4       6   69761.904762

I would like to apply this to all stocks. The result will be a table with a stock column but no date column (since it is an aggregation across all days). Something like this;

    stock  binNum         volume
0  stock0       2  174095.238095
1  stock0       3  100428.571429
2  stock0       4   79880.952381
3  stock0       5   73642.857143
4  stock0       6   69761.904762

I can do this by putting the above code in a loop and bolting on the rows one by one, but I am sure there is a more elegant way to do it via grouping and aggregation. Can anyone shed some light please?

Sorry, I realised all I needed to do was to add another level of grouping, but the aggregation is the same;

stockBinGroups = data.groupby(['stock', 'binNum'], sort=False)
binAveVol = stockBinGroups.aggregate({'volume': np.mean}).reset_index()
binAveVol.head()

    stock  binNum         volume
0  stock0       2  174095.238095
1  stock0       3  100428.571429
2  stock0       4   79880.952381
3  stock0       5   73642.857143
4  stock0       6   69761.904762


Test set larger than train set There is a two class dataset with 1121 values in total, having 230 from same class and 891 from the other class. The training set is choosen as 230+230=460 from both classes and the test set as the entire 1121 data.
1)Accuracy values are less than 0,50 even some are as low as 0,18 and 0,20. Does this make sense? For a two class outcome, there is more chance for an accurate prediction if I toss a coin. Can there be an accuracy of less than 0.50 for a two class prediction?
2)When both test-train set is choosen from the 460 class balanced rows and k-fold(1:10) is made, the accuracy levels are considerably higher, up to 0,90.
3)Can the difference between the results be because the test set is much larger than the train set?

Do I correctly understand, that the test data is the whole dataset, whereas training is only a subset of it? Training and test data must not overlap. The test is a measure of quality on unseen, unfamiliar data.
In the case of inbalanced data and two class classification the naive classifier, predicting always the most probable class has the quality 891 / 1121. Any sensbible model should beat this score.
To handle inbalanced data you can use several approaches - undersampling the majority class, oversampling the minority https://machinelearningmastery.com/random-oversampling-and-undersampling-for-imbalanced-classification/. Also many classifiers have the attribute weights, which can be set to balanced - this would penalize larger for mistakes on minority class.
In the case of imbalanced data measure not only accuracy, but precision and recall as well https://en.wikipedia.org/wiki/Precision_and_recall


Support Vector classifier perform well with input features rather than transformed features in contrast to ANN-BP, random forest (other classifiers) I am working on stock data with 5 raw features (OHLCV). Using few transformations used by technical analysts, have created 20 more features giving different kinds of indications.
When trying to predict the direction of the next few days, the behaviour shown by SVM is opposite to other classifiers. (SVM is giving better results when 5 raw inputs are considered while other methods (as expected) give better results when generated features are being used)

I completely understand that market data is very random and noisy. This is a theoretical exercise being done to understand the behaviours of different ML methods.

Thanks for the answer and it completely makes sense that results will be driven by data set in question!!

Is there any theory that says that SVM gives better results as compared to other ML algos like NN and RF when features are lower in number?
Another reason that I can think the reason for this behaviour is attributed to the information loss during transformations which were done.

Though SVMs results almost remained the same but for other algorithms improvement was significant(8-10%). Means transformed data showed patterns that RF and NN were able to recognize and do the classification while it hardly affected SVM outputs.


How to understand equations in research papers? I was searching for latent class logit model for conjoint analysis. i found a paper which has equations for this model.

I have co-workers who knows how to decipher the meaning of these equations and write these algorithms in any language, from scratch. One of my co-worker told me that he understood what is written in this paper and can write the algorithm in R. My co-workers mostly have masters degree in engineering. I have done graduation in accounting and fortunately landed in this job of analytics. I am very keen to learn this art/science of deciphering equations in research paper.  Can anyone suggest which discipline is it or lets say any recommendation of online course or books that can help me learn this? I will be indebted for life.

Thanks!

Study multiclass logistic regression; it's similar and has many tutorials. Good books include Larry Wasserman's All of Statistics, and An Introduction to Statistical Learning.

The way to understand research papers is simply to read more of them. Whenever you encounter something you don't understand follow the references or look it in one of the aforementioned books.


Need help understanding LSTMs' backpropagation and carousel of error I'm having a hard time trying to derive the maths behind LSTMs and vanishing gradients.

I had a of help from LSTM forward and backward pass, but I got stuck in page 11 from LSTM forward and backward pass.

Given the image:

We can form system of equations,
$$
        \begin{bmatrix}
        a^t \\
        i^t \\
        f^t \\
        o^t \\
        \end{bmatrix}
=
        \begin{bmatrix}
        tanh(W_cx^t+U_ch^{t-1}) \\
        \sigma(W_iX^t+U_ih^{t-1}) \\
        \sigma(W_fx^t+U_fh^{t-1}) \\
        \sigma(W_ox^t+U_oh^{t-1}) \\
        \end{bmatrix}
=
        \begin{bmatrix}
        tanh(\hat  a^t) \\
        \sigma (\hat i^t) \\
        \sigma (\hat f^t) \\
        \sigma (\hat o^t) \\
        \end{bmatrix}
$$
We can then represent this as $z$:

$$
z=
        \begin{bmatrix}
        \hat  a^t \\
        \hat i^t \\
        \hat f^t \\
        \hat o^t \\
        \end{bmatrix}
=
        \begin{bmatrix}
        W^c & U^c \\
        W^i & U^i \\
        W^f & U^f \\
        W^o & U^o \\
        \end{bmatrix}
*
        \begin{bmatrix}
        x^t \\
        h^{t-1} \\
        \end{bmatrix}
$$

We can find out the backprop derivation for $z$ from page 10 from LSTM forward and backward pass

$$
\delta z=
        \begin{bmatrix}
        \delta \hat  a^t \\
        \delta \hat i^t \\
        \delta \hat f^t \\
        \delta \hat o^t \\
        \end{bmatrix}
=
        \begin{bmatrix}
        \delta a^t \odot (1-tanh^2(\hat a^t)) \\
        \delta i^t \odot i^t \odot (1-i^t) \\
        \delta f^t \odot f^t \odot (1-f^t) \\
        \delta o^t \odot o^t \odot (1-o^t) \\
        \end{bmatrix}
$$

However the next part at page 11 from LSTM forward and backward pass is where I'm confused.

Given $\delta z$, we need to find $\delta W$, $\delta h^{t-1}$,

1) The author wrote down $\delta I^t = W^T * \delta z$:

If we do some linear algebra variables moves:

$$z = W^T * I^t$$

Multiply both sides with $I^{t^{-1}}$

$$I^{t^{-1}} z = W^T$$

Multiply both sides with $z^{-1}$

$$I^{t^{-1}} = z^{-1} W^T$$

Somehow this doesn't match with the author's formula?

2) Let's ignore 1), and try to solve for $\delta I$

$$
\delta I =
        \begin{bmatrix}
        \delta x^t \\
        \delta h^{t-1} \\
        \end{bmatrix}
$$
$$
\delta I = \frac{dE}{dI} =
        \begin{bmatrix}
        \frac{dE}{dx^t} \\
        \frac{d}{dh^{t-1}} \\
        \end{bmatrix}
$$

But $\frac{d}{dh^{t-1}}$ depends on a lot of the equations in $z$

Do I solve for them individually and add them up?

Note: $\frac{dE}{d\hat i_t}$ can be found at page 10 from LSTM forward and backward pass

$$h_{t-1}^{i_t}=\frac{dE}{dh_{t-1}^{i_t}}=\frac{dE}{d\hat i_t}\frac{d\hat i_t}{h_{t-1}}=\frac{dE}{d\hat i_t}\frac{d}{dh_{t-1}}i_t(1-i_t)$$

Replace $i_t$ with $\sigma(W_iX^t+U_ih^{t-1})$, replace $\frac{dE}{d\hat i_t}$ with $\delta \hat i_t \delta i_t$

$$=\delta \hat i_t \delta i_t \frac{d}{dh_{t-1}} \sigma(W_iX^t+U_ih^{t-1}) (1-\sigma(W_iX^t+U_ih^{t-1}))$$

It looks solvable, then my question is I would get 4 equations like the above, do I add them all together in the end to get $\delta h^{t-1}$? For example:

$$
\delta I = \frac{dE}{dI} =
        \begin{bmatrix}
        \frac{dE}{dx^t} \\
        \frac{d}{dh^{t-1}} \\
        \end{bmatrix}
=
        \begin{bmatrix}
        ignore \\
        h_{t-1}^{i_t}+h_{t-1}^{a_t}+h_{t-1}^{f_t}+h_{t-1}^{o_t} \\
        \end{bmatrix}
$$

Since my logic was to find the total error contributed from $h_{t-1}$ so you need to add them together?

3) Finding for $W$ looks even like a bigger task, however, I'm not sure where to start on this?

4) How does this relate to the error carousel? I mean after derivation of all the weights and $h_{t-1}$, I'm not sure how this leads to avoidance of vanishing gradients? I read somewhere that the weights are constant 1 or something along the lines like that?

I know this is kinda long, but feel free to ask for clarification if my question does not make sense. I think I've tried to solve for this almost for half a month now.

Appreciate any sort of guidance. Thanks.

Part 1: Derivation

It's important to note that $\delta z \neq z^{-1}$.

So on page 11 when he says he plans to derive $\delta I^t, \delta h^{t-1}$, and $\delta w^t$ he's invoking the chain rule rather than inverse matrix multiplication.

Let's derive $\delta I^t$.

$\delta I^t = \frac{\partial E}{\partial I^t}$ by definition
$\delta I^t = \frac{\partial z^t}{\partial I^t} * \frac{\partial E}{\partial z^t} $ by the chain rule

2.1.1 Recall that $z^t = W * I^t$

2.1.2 Therefore $\frac{\partial z^t}{\partial I^t} = W^T$

2.2.1 $\frac{\partial E}{\partial z^t} = \delta z^t$ by definition
Therefore $\delta I^t = W^T * \delta z^t$

The author continues by noting that since $I^t = \begin{bmatrix}x^t\\h^{t-1}\end{bmatrix}$, that $\delta h^{t-1}$ can be retrieved from $\delta I^t$ where  $\delta h^{t-1}$ would be the dx1 terms after the first nx1 terms.

Along the same vein to derive $\delta W^t$,

$\delta W^t = \frac{\partial E}{\partial W^t}$ by definition
$\delta W^t = \frac{\partial E}{\partial z^t} * \frac{\partial z^t}{\partial W^t} $ by the chain rule

2.1.1 Recall that $z^t = W * I^t$

2.1.2 Therefore $\frac{\partial z^t}{\partial W^t} = (I^t)^T$

2.2.1 $\frac{\partial E}{\partial z^t} = \delta z^t$ by definition
Therefore $\delta W^t = \delta z^t * (I^t)^T$

Part 2: Constant Error Carousels

The following explanation draws heavily from the 14th slide of this lecture, and from the LSTM section of this blog, which I would highly recommend referencing. Consider this portion of my answer the TL;DR version of these two links.

The Constant Error Carousel (CEC) as you might well know is the magic of the LSTM in that it prevents vanishing gradients. It's denoted as follows:

$c_{t+1} = c_t *$ forget gate + new input $*$ input gate

In the case of regular RNNs during backpropagation, the derivative of an activation function, such as a sigmoid, will be less than one. Therefore over time, the repeated multiplication of that value against the weights $f'(x) * W$ will lead to a vanishing gradient.

In the case of an LSTM, we only multiply the cell state by a forget gate, which acts as both the weights and the activation function for the cell state. As long as that forget gate equals one, the information from the previous cell state passes through unchanged.

So in our case, the parameters we derived $h_t$ and $h_{t-1}$ are the hidden states which as part of the input vector $I^t$, will be filtered through the forget gate, and affect the cell state accordingly. The $\delta W$ we solved for is the parameter we use to update all our weights/gates.


In a pandas dataframe can I convert my column values into numbers? So I am new to all this. I was wondering in pandas can I convert my column values into numbers?
I'll try and give an example to explain what I mean
So say for example I have a column called, 'animals', in this column I have six different animals but I want to convert them to numerical values so just as simple as 1,2,3,4,5,6 for each of the different animals. How would I go about doing this??

Well, one way i like to handle this problem (which is a common problem, at least in daily job life) is to convert each possibility in a column with binary value. Let me elaborate a bit. Let's say you have your column animals with 3 possibilities : dog, cat, and horse. You explode your column in 3 differents columns : colDog, colCat and colHorse. And you fill your new columns based on the value of the column animals. For example : if you have dog in the first row, you put 1 in the column colDog, etc.
The problem with handling categorical data with numerical value instead of binary is that you create a hierarchical order between your values. If dog is 1, cat is 2 and horse is 3, then horse will have more impact than cat and dog. Or i think you just want to represent your categories.


using neural networks to predict set of charactertics Say i have a matrix with m rows and n features where m is the number of people on say an dating website such as tinder. So n could be age, sex, location,job... etc these kinds of features.

My output 'Y' would be a features vector containing features of a person they have previously liked/matched. So it would have m rows and x features(columns) ( x can be age, sex, job, etc...). I would like to predict/suggest a possible match given this training data of X and Y.

Is it possible to use any machine learning/neural networks to predict a SET of features so Y being a matrix rather than a single columned vector? I have seen examples where given some data X we can predict Y but Y is normally a classification such as 0/1 or some sort of wine classification from (1-3) for example. What if it was a set of features? is it possible.

Yes, you can generate multiple features as output of network.

In this example, network will have X outputs. 1 output will have most probable gender , another will have most probable age and so on.

Input_1= Input(shape=(n_features, ))

x = Dense(100, activation='relu')(Input_1)
x = Dense(100, activation='relu')(x)
x = Dense(100, activation='relu')(x)

out1 = Dense(1,  activation='linear')(x)
out2 = Dense(1,  activation='linear')(x)
out3 = Dense(1,  activation='linear')(x)
..
..
outX = Dense(1,  activation='linear')(x)

model = Model(inputs=Input_1, outputs=[out1,out2,out3,...,outX])
model.compile(optimizer = "rmsprop", loss = 'mse')

Complete example of this :

https://www.pyimagesearch.com/2018/06/04/keras-multiple-outputs-and-multiple-losses/


Are there enough databases for all learning tasks? this might be a silly question but I guess the answer comes with experience in this field. I'm just wondering if today, with the internet overflowing with data and specifically with images (maybe not tagged), are there a lot of examples for learning tasks (specifically ones that takes images as inputs) without a proper database for those tasks?

I'm asking because my experience in this field is limited and I only tried to work with some more standard learning tasks that have a proper size database and I'm wondering if the need for data (or tagged data) is still an issue. So, if you can give some examples for tasks that still require a proper database of images, that will be great. Thanks.

A naive observer might be tempted to think that with the vast amounts of data being collected, there's enough to answer all learning tasks. But this is misleading at least for the following reasons:

There's no such thing as "all learning". Humanity has not yet perceived what could be the limits of knowledge. With the boldly stated ambitions of AI to match human intelligence, this also transfers to the lack of limits to what could machine learning aim to learn.

One feature that a lot of data scientists downplay, but that comes as heritage from traditional science fields, is that sound research starts with identifying what you want to achieve and focus data collection on that objective. In other words, in an ideal data analysis setup, the analyst would have a say in how fit-for-purpose data is collected.

A big part of machine learning is preprocessing data in order to clean, normalise or annotate it. This could also be done to pre-existing data as a way to adapt it for certain - previously unenvisaged - purposes. If the data is not collected with attention to how it is going to be used, it would need more preprocessing, and thus - generally speaking - would be less reliable due to the accumulation of steps where noise could be introduced.

These three arguments are only a scratch on the surface of the topic, but they come to show that even if it is sometimes useful, the widespread data collection does not always respond to all questions one might want to ask. Even if in machine learning it sounds a bit more complicated, it all boils down to this amazing quote from Alice in Wonderland:

“Alice: Would you tell me, please, which way I ought to go from here?
The Cheshire Cat: That depends a good deal on where you want to get to.
Alice: I don't much care where.
The Cheshire Cat: Then it doesn't much matter which way you go.
Alice: ...So long as I get somewhere.
The Cheshire Cat: Oh, you're sure to do that, if only you walk long enough.”

Examples:

Comprehending information from random receipts or invoices.
Automatically judging football situations.
Telling Google (think captcha) if that thing on the image is a street sign, a bridge or a number.


Multiclass Classification and log_loss I hope I can make this clear with few lines of code/explanation.
I've a 16K list of texts, labelled over 30 different classes that were ran through different classifiers; my Prediction and the Ground truth match on average at 94%.
I am now after measuring something extra (not sure what should I measure on top of an F1_score minimum as I'm still learning, though) and I came across the log_loss from sklearn, which result I understand it range between 0 and 1.
When ran against my prediction, however, the results is 1.48xxx, which is in fact higher.
In trying to understand what was wrong.
I have explored the result of ComplementNB.predict_proba that is required for the log_loss, and the value matches the one of my prediction array.
Below some code:
from sklearn.metrics import log_loss

y = ... # This is my array of value that is my source of truth

labels = numpy.unique(y)
label_ary = [idx for gt in y for idx, lbl in enumerate(labels) if gt == lbl]

print(f'The log loss is {log_loss(label_ary, clf.predict_proba(X.toarray()))}')

Whether I use label_ary or y, in both the circumstance I am obtaining the same value, meaning that some conversion inside the log_loss is already happening.
I'm not sure whether it me misinterpreting the results, or the specific of the function.
What am I doing wrong?
Thanks

Interpretability of log loss
Log loss isn't necessarily between the range [0; 1] - it only expects input to be in this range. Take a look at this example:
$$
y_{pred} = 0.1 \\
y_{true} = 1.0 \\
log\_loss = -(log(y_{pred}) * y_{true} + (1 - y_{true}) * log(1 - y_{pred})) = -(log(0.1) * 1.0) = 2.302
$$
In an extreme case log loss can even be equal to infinity. So there is nothing wrong with the code and also there aren't many interesting things you can derive from the fact that log_loss is lower or greater than 1. What you can do with it is the same as with any loss function - compare it to a similar model with different hyperparameters and choose the one with the lowest average loss as your best model (a process called hyperparameter optimization).
When to use loss and when f1 score?
Let's say you have a dataset and a classification problem which you want to solve. You know that you can create a statistical model which returns probabilities of a given class. You also know that there is (hypothetically) an algorithm which classifies based on some heuristics which requires no training. You would like to know which of these is the best for your problem. What you do, if we simplify a little, is:

Split your dataset to train, validation and test sets.
Use your train set to train the model
While training the model calculate loss for train and validation set in each epoch (if you're not using deep neural networks you can and should use cross validation).
Plot loss for train and validation set and see if your model is biased (high train loss and high validation loss) or overfitted (low train loss and high validation loss). The lower loss for validation set the better.
Do 3. and 4. multiple times for different hyperparameters and select one with the lowest validation set loss. You now have a trained statistical model.
Now use f1 score to compare your model to the algorithm you also know about. The higher score the better. Notice that assuming that the algorithm returns classes and not probabilities if it is incorrect for even one example its log loss function will be equal to infinity. This is why we can't use log loss as a metric to compare these two methods.

In short, you should use loss as a metric during training/validation process to optimize parameters and hyperparameters and f1 score (and possibly many more metrics for example Area Under Curve) during test process to select the best method to solve your problem. This way it's possible to compare different methods to solve the problem - even ones which don't use machine learning at all.


How to order the data with respect to data type I am having large data set (82 variables)
Is there any way to arrange data such a way that I have to get all numerical variables firstly then categorical variables so that I can run hypothesis testing or exploratory data analysis(eda) by having loop.

If not is there any way to do eda in simpler manner because I can't check the correlation or chi.square test for each variable

Let your data frame be df. First get the numeric columns:

num_col = df.select_dtypes('number').columns

Then get the remaining columns.

non_num_col = set(df.columns) - set(df.select_dtypes('number').columns)

Merge as required.

df = pd.concat([df[num_col], df[list(non_num_col)]], axis=1)

The columns are now in the desired sequence.


Cleaning input data with pd.get_dummies() What is the advantage of converting a series like

>>> df
   Color
0    Red
1   Blue
2  Green
3    Red

To a multiple series like the below?

>>> pd.get_dummies(df)
   Color_Blue  Color_Green  Color_Red
0           0            0          1
1           1            0          0
2           0            1          0
3           0            0          1

One could as well have a hot encoded values for the Color Column as below?

>>> labels=list(set(df.Color))
>>> pd.DataFrame(df.Color.map({x:labels.index(x) for x in labels}).rename('Color_Code'))
   Color_Code
0           1
1           2
2           0
3           1

I know syntactically pd.get_dummies looks much simpler, but somehow I want to lean towards a lesser number of features than more number of features...

Categorical features need to be converted to numerical values. They are various ways to do that. I would recommend reading this blog and this one to learn what are the advantages and disadvantages of choosing each. The first method you showed is called one-hot encoding that you can get easily by pd.get_dummies as you mentioned, and frequently used by people in algorithms like XGBoost. One of the biggest downfall of the first method OHE, is increasing often unnecessarily the dimensionality of your feature space. Other cons could be not being a good representative of feature space as well as problem of missing features in test data. The second method is LabelEncoding, usually used in target variables or when you have very few categorical features.

The problem surfaces when you start having a large number of categorical features (>50 % of feature space) with each feature having many sub levels. Here you need more clever way of feature encoding like Target-based mean encoding (smoothing, you can find some kernels in Kaggle like this). If you are Python user, I have found a fairly new package under scikit-learn-contrib offering a wide range of Categorical Encoding Methods. This Kdnuggets post also compares some of the aspect of using different methods.


Solutions for Continuous Online Cluster Identification? Let me show you an example of a hypothetical online clustering application:

At time n points 1,2,3,4 are allocated to the blue cluster A and points b,5,6,7 are allocated to the red cluster B.

At time n+1 a new point a is introduced which is assigned to the blue cluster A but also causes the point b to be assigned to the blue cluster A as well.

In the end points 1,2,3,4,a,b belong to A and points 5,6,7 to B. To me this seems reasonable.

What seems simple at first glance is actually a bit tricky - to maintain identifiers across time steps. Let me try to make this point clear with a more borderline example:

The green point will cause two blue and two red points to be merged into one cluster which I arbitrarily decided to color blue - mind this is already my human heuristical thinking at work!

A computer to make this decision will have to use rules. For example when points are merged into a cluster then the identity of the cluster is determined by the majority. In this case we would face a draw - both blue and red might be valid choices for the new (here blue colored) cluster.

Imagine a fifth red point close to the green one. Then the majority would be red (3 red vs 2 blue) so red would be a good choice for the new cluster - but this would contradict the even clearer choice of red for the rightmost cluster as those have been red and probably should stay that way.

I find it fishy to think about this. At the end of the day I guess there are no perfect rules for this - rather heuristics optimizing some stability criterea.

This finally leads to my questions:

Does this "problem" have a name that it can be referred to?
Are there "standard" solutions to this and ...
... is there maybe even an R package for that?

Reasonable Inheritance of Cluster Identities in Repetitive Clustering

Stability-Plasticity Dilemma, Learning Rates, and Forgetting Algorithms:

First, let me say that this is a really great question and is the type of thought provoking stuff that really improves one's understanding of ML algorithms.

Does this "problem" have a name that it can be referred to?

This is generally referred to as "stability".  What's funny is that stability is actually a useful concept in regular clustering i.e. not online. The "stability" of the algorithm is often chosen as a selection criterion for whether the right number of clusters have been selected.  More specifically, the online clustering stability issue that you have described is referred to as the stability-plasticity dilemma.

Are there "standard" solutions to this and ...

First, the big picture answer is that many online clustering algorithm are surprisingly stable when they have been well trained with a large cohort of initial data.  However, its still a problem if you want to really nail down the cluster identities of points while allowing the algorithm to react to new data.  The trickiness of you point is briefly addressed in Introduction to Machine Learning By Ethem Alpaydin.  On page 319 he derives the online k-means algorithm through the application of stochastic gradient descent, but mentions that the stability-plasticity dilemma arises when choosing a value for the learning rate.  A small learning rate results in stability, but the system looses adaptability where as a larger learning rate gains adaptability, but looses cluster stability.

I believe the best path forward is to choose an implementation of online clustering which allows you to control the stochastic gradient descent algorithm and then choose the learning rate so that you maximize stability and adaptability as best as you can using a sound cross-validation procedure.

Another method that I've seen employed is some sort of forgetting algorithm e.g. forgetting older points as the data stream matures.  This allows for a fairly stable system on fast time scales and allows for evolution on slower time scales.  Adaptive Resonance Theory was created to try to solve the stability-plasticity dilemma.  You might find this article interesting.

I'm not well-versed enough in R to suggest an algorithm, but I suggest you look for a mini-batch k-means algorithm that allows you to control the learning rate in its stochastic gradient descent algorithm.

I hope this helps!


How does the test data gets collected? I'm learning the linear regression now. I used R to build linear model upon a set of train model and try to predict() data based on the test data.

My question: I understand how train data gets collected. But what's with the test data? How did it get collected? Is the test data built, or is it collected, or is it predicted or what?

PS: I'm learning data science by method of self-learning, so I lack the structural in my knowledge. I might know something on one place while lack the knowledge at another place. Please forgive and guide. Thanks. :)

In general, you randomly splits the available data into following 3 sets. Train, validation and test. [Different ratios could be used depending on total amount of data at hand and the difficulty of the problem.] You can start with a simple 80%/10%/10% split.
You use the first two sets to build your model. So you would use the 80% split of your data to lets say build a logistic regression model. Now you use the 10% validation set to see how good your model is. You can iterate on this process until you are satisfied with the validation data set performance of the model.

Now, you use the last set (the "test" split) to see how your model would generalize on an "unseen" data set. [Here unseen means, your model never had access to "see" this data while in the learning phase]. You never use use this 3rd set to glean knowledge about how your model could be improved. Think of this set as a set your customer has, and you don't have access to, and to whom you are going to deliver your model.

Once you get a hang of this concept also learn more about how you can do cross-validation if you have limited data and to improve confidence on your model.


Best practice for short sentences in a deep learning network In a deep learning network (CNN or RNN), we might use word embeddings such as FastText, Glove, etc. to represent the input text. My question is:

If I'm working on a data from Twitter, and I have a variety of text lengths, as in the picture below (example):

sometime I have a few sentences of a length larger than 150 and the average length of the rest of the sentences is 48.

Here, what I noticed in some implementations online that they pad the short sentences with "PAD" word to increase their size to reach the length of the largest sentence (Red spaces in the picture), where this PAD word is filled with a random value. Is it a good practice to do so?

If my dataset contains 5000 tweets, and I have about 20 tweets with a length larger than 150 , the random value will affect the classification task with a high ratio since most of the sentences will be padded with a random value.

Yes, when you have different sequences with different length you can pad all the sequences so that they have all the length of the longest one. is a good practice and doesn t impact the learning mode. Another solution could be online learnig. With online learning you give to the model 1 sample at the time (batch_size =1) and in keras you can train the model with batch with different lengths.

With online learning your computation is slower, but theoretically your model need less epochs to learn.

If you want to pad you should pad with all 0s and not random values.


This changed in the latest version of Bokeh (I guess 0.12.7). This is the new way of doing it.
from bokeh.io import show, output_file
from bokeh.models import ColumnDataSource, FactorRange
from bokeh.plotting import figure

output_file("bars.html")

fruits = ['Apples', 'Pears', 'Nectarines', 'Plums', 'Grapes', 'Strawberries']
years = ['2015', '2016', '2017']

data = {'fruits' : fruits,
        '2015'   : [2, 1, 4, 3, 2, 4],
        '2016'   : [5, 3, 3, 2, 4, 6],
        '2017'   : [3, 2, 4, 4, 5, 3]}

# this creates [ ("Apples", "2015"), ("Apples", "2016"), ("Apples", "2017"), ("Pears", "2015), ... ]
x = [ (fruit, year) for fruit in fruits for year in years ]
counts = sum(zip(data['2015'], data['2016'], data['2017']), ()) # like an hstack

source = ColumnDataSource(data=dict(x=x, counts=counts))

p = figure(x_range=FactorRange(*x), plot_height=250, title="Fruit Counts by Year",
           toolbar_location=None, tools="")

p.vbar(x='x', top='counts', width=0.9, source=source)

p.y_range.start = 0
p.x_range.range_padding = 0.1
p.xaxis.major_label_orientation = 1
p.xgrid.grid_line_color = None

show(p)