On which step should use SMOTE technique for over sampling? I want to use SMOTE technique for over sampling but I don't know on which step on pre-processing I should use it.

My preprocessing steps are:

Missing values
Removing Outliers
Smoothing Data

Should I use SMOTE before all of these steps or its better to use it after these steps?

If you are using python, you can't use SMOTE in the presence of null values.

In this case:

Remove Outliers
Smooth Data
Impute null values (there are some smart options for that in R: using random forests to impute)
SMOTE

Removing outliers first let you do better smoothing and imputing.


You can check the type of activation in layer config.

model.layers[idx].get_config(), where idx is the index of desired layer.


Clarification on the Keras Recurrent Unit Cell I paste below the Keras documentation on Recurrent layer

model= Sequential()
model.add(LSTM(32, input_shape=(10, 64)))

# now model.output_shape == (None, 32)
# note: `None` is the batch dimension.

What does the first argument of LSTM 32 mean here? Is it creating 32 LSTM blocks (By block I mean consisting of input, forget and output gate)? Could you please explain the meaning of the first argument and how that contributes to the output dimension of the LSTM both when return sequences is True and when return sequences is False

The notation used for LSTM is quite confusing, and this took me some time to get my head around this as well. When you see this graphic (often used to explain RNNs):

You need to consider that X is a sequence of data the timestep t, it's not just a single scalar input value (as we're used to with feed-forward networks); it's an array / tensor of data. The diagram shows how there is an output at time-step t, but that it then also feeds into the next time-step, t+1, when the next array/tensor is then fed in.

A better/clearer way (in my opinion) is to look at it is like this:

So a LSTM 'cell' is actually what you might consider a layer. And a unit (one of the circles) is one of these:

which you can consider a neuron in a hidden layer.

Initially, I thought this was a cell, but it isn't, it's a single unit. So when you specify 32 units, for example, you're actually saying how many of these units (neurons) you want in the cell (layer).

This is what gives the model its capacity to learn the data that's presented to it. And just like hidden layers/neurons in a feed-forward, it's a hyperparameter that you'll need to experiment with; too low and the model will under-fit, too high and it will over-fit.


Random Forest where objective is not to replicate past classifications Normally when I train a random forest to classify observations into multi-class buckets the objective is to correctly predict which bucket an observation will fall into based on historical (training) data.

Instead, I want to train the random forest to classify observations on some other criteria, such as profit maximization. Is this possible?

Here's an example:

Imagine that we have data on Dropbox subscriptions over the past 1 year. Some Dropbox leads (potential customers) had 1 of 3 possible coupons, other leads did not have a coupon.

In this scenario I want to determine which (if any) coupon I should should offer to a lead to maximize net revenue, considering their likelihood of purchasing a subscription, their predicted retention (# of months they will continue their subscription) and the effect of the coupon on purchase price.

Theoretically some leads who are likely to purchase a subscription and be retained will not need a coupon to do so. Others might produce higher net revenue from a coupon for "$5 off per month" or "first month free", etc.

I presume the dependent variable should still be coupon type, i.e.

Y = No Coupon, Coupon A, Coupon B, Coupon C, Coupon D

Is it possible to make the random forest work in this way? You can consider this question language-agnostic, though if I have a choice I will try to do this in R.

I know that in the case of eXtreme Gradient Boosting in either R or Python I can specify a custom objective function.

Your target variable should always reflect what you're trying to optimize (maximize or reduce), so if you want to maximize revenue, you should make revenue your target variable. I would use the coupons as independent variables. When you want to see whether to offer a coupon or not for a particular customer, feed a new record into your model for every permutation of coupon / no coupon and see which has the highest prediction. There might be other/better ways of doing this, but I think this should get what you're looking for.


Which (naive) NLP method for correlating human messages in chatrooms? Suppose an online chatroom is filled with many "alt" accounts - i.e, multiple accounts are being controlled by one user, a troll. This user leverages multiple accounts in order to steer the conversation in particular directions to suit their needs. This user may also be altering their mode of speech per account, to avoid detection.
Suppose I want to detect these alt accounts by using some kind of NLP classifier. What would the best method be? Where "best" means effective, but also relatively simple to set up - as in something you could do in python with relatively basic sklearn modules.
What I've tried so far is to collect the 100 most frequent words used by each user, and just throw that corpus into a sklearn.feature_extraction.text.TfidfVectorizer, and then looking at the pairwise similarity. These are mostly stop words, words which most NLP articles online are telling me to ignore. But I think these basic (almost unconsciously-used) words are less likely to be subject to obfuscation. The user may, for example, be applying varying spellings of less-common words (realize vs realise etc) - but the user could not plausibly have alternative spellings for ["the", "of", "you", "one"] etc.
Like I said, that isn't an "advanced" approach, but it's an example of the general level of complexity that I'm willing to undertake. Is there are any validity to that approach? If not, can you suggest a better one?

This is a stylometry task, more exactly some form of author verification/identification task. In case you want to dig deeper, the PAN workshop series is a good source of datasets and methods.
About your method:

Your intuition is correct about selecting the most frequent words, in particular stop words: writing style is better characterized by the patterns in frequent grammatical constructs than by the choice of content words. However depending on the size of the data you might want to be more flexible about the number of words: if you have enough data, you should probably take more than the top 100 frequent words. If you don't, well... it might not work very well. Note also that some successful methods don't use the top frequent words but rather a range of middle frequency words.
Using TFIDF is controversial for style identification. It's often done simply with the term frequency instead.

Other features you could consider:

words n-grams, typically bigrams or even trigrams if there is enough data.
characters n-grams (usually trigrams) have the surprising property to be very robust features for style detection. If you use these do not apply any tokenizer or remove punctuation signs.


How to find combined feature contributions in a random forest algorithm I am currently working on a pancreatic cancer dataset which has numerous features including smoking, family history, age, etc. I have used the feature_importances_ method in sklearn to find and plot the individual feature importances, but wanted to know how to find the importance of combined variables (e.g. are those who smoke and have a family history of pancreatic cancer much more likely than others?). I know it is not as simple as adding the individual feature contributions for the combined features, and wanted to know if there is a specific method/function that can be used to do that. I have also used treeinterpreter to find the feature contributions.

My recommendation would be to use Partial Dependence Plots, which show the marginal effect one or two features have on the predicted outcome.  They plot the average predictions over a range of values of the predictor(s) you specify, with the influence of all other predictors "averaged out".  If you are familiar with linear regression, it is like plotting the predicted value of the target for different values of a predictor multiplied by the estimated weight/coefficient.

In your question, you are interested in two predictors (smoking and family history), so you would plot a two-way partial dependence plot showing the dependence of the target variable on joint values of smoking and family history.

There is a PDPBox package you can install that produces a variety of very nice plots and has some tutorials: https://github.com/SauceCat/PDPbox.  It claims to support all Scikit-Learn algorithms.

There is also a Cross Validated Q&A on interpreting PDP's for Random Forests (produced with R): https://stats.stackexchange.com/questions/121383/interpreting-y-axis-of-a-partial-dependence-plots


why do we need row sampling in random forests? In random forests, where our estimators are decision trees, we do column (feature) sampling without replacement within an estimator, and with replacement  in between estimators. This is perfectly fine as we are trying to reduce the high variance of individual decision trees.

But what is the need to do row sampling?

Usually more the data, the better it is for a model to learn, and even if i dont have any computational resource limitation, why do we have to do row sampling in estimators for random forest classifier?

First I think your understanding of “column sampling” is incorrect.  Random forest try’s a subset of features for each split.  It does not sample without replacement within an individual tree.
Random forest samples rows with replacement (bootstrap samples) to remove correlation between the decision trees.  Think about it, if you didn’t do this even though you create each split based on only a subset of features, your trees would end up looking fairly similar (or at least more similar than if you bootstrapped).  You do have larger bias due to creating trees based only on about ~63% unique values but the decrease in variance by having more uncorrelated trees makes up for it.


Which kNN model to chose? I am trying to tune the "n_neighbors" for a kNN model andI have the following problem :

Based on the mean cross validation score the optimal kNN model should be the one with 10 neighbors.

On the other hand, when I plot the "scores vs neighbors" graphs I see that there are models whose score distance between the training and the test data is much smaller ( for instance the model with 20 neighbors ).

I am new to ML and this is still very confusing to me.. but should I stick to the 10 neighbors model, or is the 20 neighbors model better ? How do I decide ?
Any help is much appreciated.

Here is my code and the graphs :

best_score = 0
neighbors = np.arange(1,31)
all_train_scores = []
all_test_scores = []

for n_neighbors in neighbors :

        reg = KNeighborsRegressor(n_neighbors = n_neighbors, metric = 'manhattan')

        score = cross_val_score(reg, X_train, y_train, cv = 5)
        score = np.mean(score)

        if score > best_score :
            best_score = score
            optimal_choice = {'n_neighbors' : n_neighbors}

        reg.fit(X_train, y_train)
        train_score = reg.score(X_train, y_train)
        test_score = reg.score(X_test, y_test)

        all_train_scores = np.append(all_train_scores, train_score)
        all_test_scores = np.append(all_test_scores, test_score)

It depends on couple of things but one of the important ones is how big is your set.

Note that the difference is 0.01 in R squared, so if dataset is small taking 20 samples do determine class might be costly, on the other hand if you can afford it 10 samples might be to little do ensure class membership.

Depending on your next usage I would weigh in data size.


Feature selection for gene expression dataset I am searching for a feature selection algorithm which selects features that are:

relevant to discriminate groups of samples (for each sample a group label is provided)
endowed with high variance across all the samples

This should be applied to gene expression dataset, in which each sample has a group label, therefore it should be possible to select for each group a set of features to be checked against.

I have now two candidates:

selecting features by the feature importance result of a Random Forest classifier
using the Minimum Redundancy Maximum Relevance (mRMR) algorithm

However, I am unsure of which may be the best or if there are better candidates for this purpose.

If the algorithm is implemented in Python scikit-learn it would be a plus.

It would be helpful if you described your dataset more.  Gene expression datasets seem to often have very high dimensionality and Lasso regularized logistic regression is a popular method to approach this problem.  This paper takes it a little further and might help you out:
http://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-14-198

Random forest can generally certainly provide a meaningful importance ranking, but it also depends on what your dataset looks like.

mRMR sounds like it is specifically designed for identifying gene characteristics, so definitely give it a try.

There's also Principle Component Analysis which is also used for gene expression data.

Lots of options, but your questions is not detailed enough to go any further, and providing code as a solution at this point isn't realistic. The documentation for Python scikit-learn has many good explanations and examples.


Dueling DQN - Advantage Stream, why use average and not the tanh? For Dueling DQN (page 5), why do authors use an average for Advantage stream, and don't simply "activate" the Advantage stream (with a $tanh$ for example)?

Would "activating" work in theory, and is it a similar idea to what the authors intended to achieve, or am I missing the point?

To remind, this is the equation, which Produces a Q-value for the taken action $a$ by combining the Value stream with the Advantage stream:

$$Q(s,a; \theta, \alpha, \beta) = V(s; \theta, \beta) + \biggl( A(s, a; \theta, \alpha) - \frac{1}{N}\sum_{a'}^{N}A(s, a'; \theta, \alpha) \biggr)$$

where

$s$ is the current state we are in
$a$ is the action we've decided to take
$a'$ is one of any action we could have taken (including the action we've taken)
$\theta$ are parameters (weights) of the network before the "splitting" into two separate streams
$\alpha$ are the parameters of Advantage stream
$\beta$ are the parameters of the Value stream

The goal of these models is to estimate the value of each action choice. They chose to use the average function to estimate the update value because the average function produces a single scalar influenced by each value.

Tanh function is not an appropriate nonlinear activation function. The tanh function only takes a scalar as an input, thus would not weight each value.

The authors tried an appropriate nonlinear activation function for many values - softmax:

  We also experimented with a softmax version of equation (8), but found it to deliver similar results to the simpler module of equation (9).


Avoiding trivial solution with sparse output I have a network whose output is a large vector ( ~ 2000 let's say ). The ground truth output is very sparse and binary- that is out of the 2000 most of the values will be 0 and just a handful will be 1. Just to clarify, each sample will have 1's at different locations with some uneven likelihood of there being a 1 at any given neuron.

I am pretty sure that if I run this network it will quickly find the trivial solutions of outputting all 0 and just stay there.

How might you suggest to nudge the network to avoid this trivial solution?

I thought of somehow weighting the samples in such a way that each sample will give a high weight to the neurons that are equal to 1. This seems very hard to implement efficiently. How can I easily implement this in tensorflow/keras efficiently?

Another Idea is to set all the 0's to random centered at .5 and slowly decrease them towards zero as the network trains.

Other ideas?

Some interesting suggestions I got from others:

split the network so that each subnetwork is in charge of a subset of the classes.
use the Dice loss or jaccard loss which are pretty good for this situation.
pre-train with the less prevalent classes only and add classes over epochs. This needs to be done very carefully.


how to increase the coverage of wordnet using wikipedia I want to increase the coverage of wordnet by increasing the synset count.

Is there a way in which I can proceed and increase sysnets. I am trying to work on Hindi wordnet with Princeton wordnet  as the  standard

You should look at Open Multilingual Wordnet (http://compling.hss.ntu.edu.sg/omw/).

They used wiktionary (not wikipedia, because wikipedia lemmas are nouns only) to get translations and synsets. EDIT: In the extended version, there is also a Hindi wordnet with 3,593 synsets, see http://compling.hss.ntu.edu.sg/omw/summx.html

There is a non-open Hindi wordnet at https://catalog.ldc.upenn.edu/LDC2008L02.


Validation set performance increased, test set performance decreased I am training a CNN model for a three-class classification problem. To do this, I'm gradually unfreezing more convolutional blocks of a pre-trained Resnet-18 network. The thing is that after unfreezing a block (let's say block 3, and preceding blocks), the performance on the validation set did improve but the performance on the test dataset did not improve (as opposed to block 2, and preceding blocks).

I'm now wondering whether or not it is 'justified' (so to speak) to keep unfreezing blocks to see how this affects performance on the test dataset. I feel like I'm just capitalising on some kind of luck that the model fits the test dataset better just by chance.

In short, I guess my question is whether I should choose my models on validation performance or test performance?

Don't optimize your model for the test set during development. You shouldn't even consider using your model on the test data until you're done tweaking your it, since the point of the test set is to have a ready example of "unseen" data for your model.

The idea with a train/validation/test split is that you could evaluate all the different iterations your model went through (that you feel like testing) against this completely unseen set of data and not have to worry about whether you were optimizing for fit to this data because it's completely new to your evaluation process.

No decisions were made during development to ensure that any of the tested models would perform well on this set because you're only checking how well you did after you're done fussing with your models. It's as clean a reference point for model performance as you can get.

If the only constraint for your final model choice is performance, then you would pick the model that performed the best on your test data. But I'd highly recommend using the train/validation/test split as intended.


Hyper parameters tuning XGBClassifier I am working on a highly imbalanced dataset for a competition.

The training data shape is : (166573, 14)

train['outcome'].value_counts()

0    159730
1      6843

I am using XGBClassifier for building model and the only parameter I manually set is scale_pos_weight : 23.34 (0 value counts / 1 value counts)

and it's giving around 82% under AUC metric.

I guess I can get much accuracy if I hypertune all other parameters.

XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
       colsample_bytree=1, gamma=0, learning_rate=0.1, max_delta_step=0,
       max_depth=3, min_child_weight=1, missing=None, n_estimators=100,
       n_jobs=1, nthread=None, objective='binary:logistic', random_state=0,
       reg_alpha=0, reg_lambda=1, scale_pos_weight=23.4, seed=None,
       silent=True, subsample=1)

I tried GridSearchCV but it's taking a lot of time to complete on my local machine and I am not able to get any result back.

clf = XGBClassifier()
grid = GridSearchCV(clf,
                    params, n_jobs=-1,
                    scoring="roc_auc",
                    cv=3)

grid.fit(X_train, y_train)
print("Best: %f using %s" % (grid.best_score_, grid.best_params_))

What others parameters should I target to tune considering higly imbalanced dataset and how to run it so that I can actually get some results back?

1)Random search if often better than grid
https://www.analyticsindiamag.com/why-is-random-search-better-than-grid-search-for-machine-learning/
Try:

#clf = RandomizedSearchCV(clf ,param_distributions = params, cv=kfold, scoring="accuracy", n_jobs= 10, verbose = 1)

2) For unbalanced data set:

Resampling: undersampling or oversampling
create new data points

https://towardsdatascience.com/methods-for-dealing-with-imbalanced-data-5b761be45a18


Do i need to handle missing values before EDA? I am working on a data set and there is an interesting column with missing values, but I don't want to discard the rows (so as not to lose data from other columns) or do imputation (so as not to change the data). Can I work with the dataframe with a column with missing values during exploratory data analysis and only take the slides with no values missing when plotting something with this specific column?

If the number of rows includes missing values are very small according to sample size I recommend dismissing it.
But if you decide to keep them according to not lose any information, you can do a bunch of things according to the feature that involves a null value.
You should understand the pattern of the feature column well before deciding the filling method below.

You can change the null value as;

mean of the column
median of the column
same as above or below
just zero
most repeated value along the column
etc.

If there is any categorical feature, you can group by a feature like gender and can do the same thing as above. For example, if there is a NaN height value for a male you can fill it with the mean of the heights of males etc.
Besides all you can decide to discard the whole column with:

Checking the correlation between the column and the dependent variable
Checking the column's representation level of source data with PCA


what should i do if my target variable is categorical when using decision tree? (many categorical variables) all,

i'm trying to classify a set of features to belong to a particular company (my dependent variable). my independent variables are a mixture of continuous and categorical features.

my data-set i am training on is labelled data with the label being the company ( the dependent var). I am not sure how i should go about dealing with my dependent variable should i use one hot encoding on the the entire data-set and then split it into training and test?

i am uncertain whether to onehot encode before i feed into decsion tree.

This is called multiclass classification, and the encoding needed for the target variable depends on what package and model you're using.  You may be expected to one-hot encode (e.g. neural networks usually have an output neuron for each class), ordinal encode (e.g. most(?) sklearn multiclass classifiers), or leave them as strings (most R models, I'd guess?).


Clustering sequences of sentence embeddings I have a sequence of events, right now I am not worried about their actual times, just the order. This is a sequence of web page views.

I have modelled my data as the following, where each element represents the category of the web page.

user_sequence = ['A', 'A', 'B', 'C', ...]

Following this I used the code and approach from this paper: Sequence Graph Transform

My question is how could I represent more complex data in my sequences, for example, we have an embedding representing the page content, along with features including the dwell of the user on each page.

So to summarise, the goal is to do process sequences of rich event data in an unsupervised manner.

The approach taken for anyone coming across this has been to first cluster each web page / node before passing it into SGT. This means we can encode more information into the sequence prior to using SGT.


Choice of time series models I built a model for time series in order to forecast new values.

What is the best way to choose the correct model?

Are better criteria like AIC and BIC or the comparison between prediction errors? In this last case I have to split dataset in train and test while in the first case is not necessary, reducing time of implementation.

Thanks!

First thing first, when ever you use Time Series data you call it as Forecasting not Prediction as it is time dependent. To understand why you can go through this link
Metrics to compare models
When you are trying to compare between models you need to use AIC,BIC, AUC etc values. you can go though this link to understand better
Metrics to Access the Model
When you are accessing the performance of the model then you need to check for Error Rates(RMSE, MAE, MAPE, MSE etc). Yes, in this case you need to divide the data into Train and Test to access the model. You can go through this link for better understanding
Improve the Forecast
To take it to the next level, you can use an ensemble to get better the result. This might or might not decrease the error rate. In most of the cases it is helpful. You can combine 2/3 moderately performing hotels outcome to get best results i.e., Ensemble Model.


Why do a lot of people use ipython notebook over python file when doing analyzing data? Is it the same in industry? I have seen that a lot of people write code in ipython notebook when doing statistical analysis on data, apart from easy visualization after each step rather than running the whole code every time on a .py file. What are the other advantages,if there are any?

Also in industry which one is used more often?

iPython notebooks are great for some cases. I use them because of:

Easy in-place editing and immediate execution, very friendly for quick and experimental stuff
In-place visualization. Also, ability to have multiple figures on the same page, compare them, re-run figures, move the cells. Much more convenient than multiple and independent OpenCV's imshow windows.
They are more convenient for step-by-step iterative analysis
Since they are collection of code blocks prepended with a description and followed with the results, they are great for tutorials and guides.

You can compare the notebooks to test stands, where you can quickly assemble something working to demonstrate to the others, while Python modules are more like building blocks to be used either for demonstration or for production later. Thus, combination of both is very powerful.


Significant overfitting with CV I working on a binary classification task. The dataset is quite small ~1800 rows and ~60 columns. There are no duplicates in the rows. I am comparing different classifiers amongst the canonical ones: random forest, logistic regression, boosted tree and SVC. I am training the hyperparameters by a CV on 90% (train) with 10% held out to measure the generalization error (test). The dataset is slightly unbalances (1 to 3 ratio of classes) hence I used a stratified fold for all splits. I also use roc-auc as a metric for my CV.

I get the following results for roc-auc score and accuracy:

 DummyClassifier
Train
ROC-AUC score: 0.50000
Accuracy: 0.69705
Test
ROC-AUC score: 0.50000
Accuracy: 0.69545

 LogisticRegression
Train
ROC-AUC score: 0.88459
Accuracy: 0.78666
Test
ROC-AUC score: 0.72559
Accuracy: 0.69545

 RandomForestClassifier
Train
ROC-AUC score: 1.00000
Accuracy: 0.99695
Test
ROC-AUC score: 0.81748
Accuracy: 0.80455

 XGBClassifier
Train
ROC-AUC score: 1.00000
Accuracy: 0.99949
Test
ROC-AUC score: 0.80617
Accuracy: 0.79545

 SVC
Train
ROC-AUC score: 0.89900
Accuracy: 0.83248
Test
ROC-AUC score: 0.73515
Accuracy: 0.73182

There is always a significant gap between train and test scores. I am clearly overfitting. I guess it is a consequence of the low number of rows but I am not sure about what to do about that? Force the CV grid search for hyperparameters to a range with strong regularization?

For the problem of overfitting, you could look train models that employ regularization. For instance this examples shows how to regularize an SVM.

Another thing I noted is that you have used the tag "unbalanced-classes". If that is the case, accuracy isn't a very good metric. While AUC is good at this, I've personally had trouble with this metric in the past. My suggestion would be to include a metirc like F1-score and most importantly in each case calculate the confusion matrix. This will show you if you are missing one class more than the other. If that is the case you might want to incorporate an oversampling method (e.g. SMOTE) into your pipeline.


Few activation functions handling various problems - neural networks How can a few activation functions in neural networks handle so many different problems?

I know some basics theory behind ANN, but I can't get what functions like the sigmoid function etc. have in common with for example image classification?

Image classification and other task can be expressed as function approximations and, in theory, neural networks can approximate (almost) any function (given few assumptions on the activation function (see Universal Approx. Theorem)).

However, in practice, not all functions fulfilling these assumptions work equally well. Popular activation functions usually share some properties that allow neural networks to learn efficiently in practice, e.g. they are continuously differentiable (for gradient descent), close to the identity near zero (accelerates initial learning from small random weights) and so on.

Activation functions like the sigmoid function are not directly related to image classification or any other tasks. Rather, they allow for efficient training of neural networks, which, in turn, can represent a wide variety of tasks using different architectures and cost functions.


How to show percentage (text) next to the horizontal bars in matplotlib? I am using seaborn's countplot to show count distribution of 2 categorical data. Fine it works but I want the percentages to show on top of the bars for each of the plot. Please how do I do it?

fig, ax = plt.subplots(1, 2)
sns.countplot(y = df['current_status'], ax=ax[0]).set_title('Current Occupation')
sns.countplot(df['gender'], ax=ax[1]).set_title('Gender distribution')

I have made edits based on the comments made but I can't get the percentages to the right of horizontal bars. This is what I have done.

total = len(df['current_status'])*1.
ax = sns.countplot(y="current_status", data=df)
plt.title('Distribution of  Configurations')
plt.xlabel('Number of Axles')

for p in ax.patches:
        ax.annotate('{:.1f}%'.format(100*p.get_height()/total), (p.get_y()+0.1, p.get_height()+5))
_ = ax.set_xticklabels(map('{:.1f}%'.format, 100*ax.xaxis.get_majorticklocs()/total))

Here is a working example to add a text to the right of horizontal bars:

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns

df = pd.DataFrame(np.array([['a'], ['a'], ['b']]), columns=['current_status'])
ax = sns.countplot(y="current_status", data=df)
plt.title('Distribution of  Configurations')
plt.xlabel('Number of Axles')

total = len(df['current_status'])
for p in ax.patches:
        percentage = '{:.1f}%'.format(100 * p.get_width()/total)
        x = p.get_x() + p.get_width() + 0.02
        y = p.get_y() + p.get_height()/2
        ax.annotate(percentage, (x, y))

plt.show()

Output:

You can move the texts around by changing the formula for x and y. For example, to place them on top:

x = p.get_x() + p.get_width() / 2
y = p.get_y() - 0.02

Here is a general guideline for annotating axes.


Accuracy noise patterns during model training I'm training a logistic regression model on a small dataset. I have about 1300 samples that I split into a training and a testing set (70% and 30% respectively).

The training seems ok, however when I plot the accuracy of my model w.r.t. the epoch, some repeating noisy patterns appear at the end well after the accuracy is stabilised (after 800 epochs, see images below).

The training is done with an Adam optimizer. I'm using a learning rate of 0.04 and a weight decay of 0.07 that I found after doing a random search.

Is it something that may happen when training and is without consequence or does it reflect an issue with my data / training / software implementation ?

It is likely that in your training phase you are reaching local minima, then since you 'persist' at each epoch you get off of your point and then after the 800 epochs you reach it again.

Look at the image below, imagine you reach the blue point, but you keep persisting, looking for another point, then you will be off of your track but it will eventually come back. The pattern will be more clear in logistic regression since theoretically you are applying gradient descent to a simpler function than a complex neural network.


Does MLP always find local minimum In linear regression we use the following cost function which is a convex function:

We Use the following cost function

in logistic regression because the preceding cost function is not convex whenever the hypothesis (h) is logistic function. We have changed the equation of cost function to have a convex shape to find its global (the only one which exists). There is a fact that I can not understand. In Multi Layer Perceptrons ANNs I have seen a lot that they can be stuck in local minimums. Why is that? We have used this cost function for each perceptron and gotten the rules for updating the values for the weights in back propagation algorithm; So why do we stuck?

The loss functions are only simple convex functions with respect to the weight parameters (and specific data) when there is a single layer. More exactly, they can proven to be always convex with respect to the weights in the simple models (linear or logistic regression), but not with respect to weights of deeper networks.

You can prove that there must be more than one minimum in a network with 2 or more layers - and thus the loss function cannot be convex - by considering swapping the weights around when you have found a minimum value. Unlike with a single layer network, it is possible to swap the weights around that feed into the hidden layer whilst maintaining the same output. For example, you can swap the weights between input and hidden layer so that values of neuron output 1 and neuron output 2 are reversed. Then you can also swap the weights feeding out of those neurons to the output so that the network still outputs the same value. The network would have a different set of weights, but generate the same outputs, and so this new permutation of weights is also at a minimum for the loss function. It is the "same" network, but the weight matrices are different. It is clear that there must be very many fully equivalent solutions all at the true minimum.

Here's a worked example. If you have a network with 2 inputs, 2 neurons in the hidden layer, and a single output, and you found that the following weight matrices were a minimum:

$W^{(1)} = \begin{bmatrix} -1.5 & 2.0 \\ 1.7 & 0.4 \end{bmatrix}$

$W^{(2)} = \begin{bmatrix} 2.3 & 0.8 \end{bmatrix}$

Then the following matrices provide the same solution (the network outputs the same values for all inputs):

$W^{(1)} = \begin{bmatrix} 1.7 & 0.4 \\ -1.5 & 2.0 \end{bmatrix}$

$W^{(2)} = \begin{bmatrix} 0.8 & 2.3 \end{bmatrix}$

As we said the first set of 6 parameters was a solution/minimum, then the
second set of 6 parameters must also be a solution (because it outputs the same). The loss function therefore has 2 minima with respect to the weights. In general for a MLP with one hidden layer containing $n$ neurons, there are $n!$ permutations of weights that produce identical outputs. That means that there are at least $n!$ minima.

Although this does not prove that there are worse local minima, it definitely shows that the loss surface must be much more complex than a simple convex function.


Negative examples for a Yes/No image classification neural network I'm trying to retrain a neural network using transfer learning that can classify whether an image has a certain object, say, a car. My positive sample dataset is quite small, only 2500~ images. It works really well with "regular" binary classification (2500 images of cars/2500 images of flowers and it has to differentiate between those two) but the problem is that I'm not sure how to make it classify for all types of images, or how to make it 2500 images of cars/2500 random images and it'd have to classify whether Yes - the image is a car or No - it's not a car. What would those random images be? Thanks!

Have you tried creating a dataset with 2500 images of cars and ~2500 images of random stuff and trained it first?
if so you should try that and see how it works, if it doesnt work you might have to add more images in the "distractor" class.
Provide more information about what you find after you do this experiment and may be we can help further?


Recurrent Neural Network (LSTM) not converging during optimization I am trying to train a RNN with text from wikipedia but I having having trouble getting the RNN to converge. I have tried increasing the batch size but it doesn't seem to be helping. All data is one hot encoded before being used and I am using the Adam optimizer which is implemented like this.

   for k in M.keys(): ##For k in weights
        M[k] = beta1 * M[k] + (1-beta1)*grad[k]
        R[k] = beta2 *R[k] + (1-beta2)*grad[k]**2
        m_k = M[k] / (1-beta1**n)
        r_k = R[k] / (1-beta2**n)
        model[k] = model[k] - alpha * m_k / np.sqrt(r_k + 1e-8)

Beta1 is set to 0.9, beta2 to 0.999 and alpha is set to 0.001. When I train it for 50,000 I get very high fluctuation of the cost and it never seems to significantly decrease (only sometimes due to the fluctuations (and I catch the weights with the lowest cost)).
After sketching the cost of iterations I get a graph like this:

It seems to be increasing on average only seeming to decrease to the the large fluctuations. What can I change to have better success and have it converge?

Thanks for any help

I think the problem lies with your text processing to one-hot vectors.
Try using embeddings instead of one-hot vectors.
An embedding is also a n-dimesional vector that allows words with similar meanings to have similar vectorial representation.
One-hot vectors don't have such information. For them, there's a set of words of say cardinality c , then each vector is cx1 . The 1 in the vector just represents its location. It's like bit manipulation in this sense.
So no semantic meaning is preserved. For e.g., in your corpus, there may be words like adore and love. Both have similar meaning. But one-hot vector for love and adore may be far away, depending upon where in set love and adore are mentioned.

But if you use embeddings, words with similar meanings will have similar representations in predefined vector space.

With this, your LSTM will learn dependencies better and will start converging.

Hope this helps :)


Best frequent itemset package in python Could anyone please recommend a good frequent itemset package in python? I only need to find frequent itemset, no need of finding the association rules. Thanks!

I also recommend MLXtend library for frequent itemsets.
usage example:
dataset = [['Milk', 'Onion', 'Nutmeg', 'Kidney Beans', 'Eggs', 'Yogurt'],
           ['Dill', 'Onion', 'Nutmeg', 'Kidney Beans', 'Eggs', 'Yogurt'],
           ['Milk', 'Apple', 'Kidney Beans', 'Eggs'],
           ['Milk', 'Unicorn', 'Corn', 'Kidney Beans', 'Yogurt'],
           ['Corn', 'Onion', 'Onion', 'Kidney Beans', 'Ice cream', 'Eggs']]

te = TransactionEncoder()

te_ary = te.fit(dataset).transform(dataset)

df = pd.DataFrame(te_ary, columns=te.columns_)

frequent_itemsets = apriori(df, min_support=0.1, use_colnames=True)

print frequent_itemsets


VC dimension of half spaces over the real line I'm studying VC dimension and I'm having a little difficulty understanding it. I read lots of explanations, but when I come across this simple exercise I did not get a good intuition. The problem is this:

Find the VC dimension for the following case,

  $h (x) = \mathbb{1}_{{a < x}}$, with parameter $a, X \in R$.

The resolution is as follows:

  VC-dimension = 1.

  (a) It can shatter point $0$, by choosing a to be $2$ and $-2$.

  (b) It can not shatter any two points ${x1, x2}, x1 < x2$, because the
  labeling $x1 = 1$ and $x2=0$ can not be realized.

My doubts are:

1) In item (a) what is parameter $a$? I looked for similar exercises, but there was no concrete explanation about this parameter, what is its function? In this item it takes as equals $2$ and $-2$, with $-2 < 0$ ok, but $2 < 0$, I did not understand, from this conclusion, why it is correct.

2) In item (b) I did not understand the inequality, as he came to the conclusion that $x1 < x2$ can not be realized, why is the inequality $x1 > x2$ not valid?

First you need to understand how to calculate the VC-dimension. There are two conditions for the VC-dimension to be $n$ (here $n=1$):

You need to find one set of $n$ points that can be shattered (i.e. classify all possible $|\{0,1\}|^n=2^n$ labelings correctly) and
It must be impossible to find any set of $n+1$ points that can be shattered.

Now to your questions:

1) $a$ defines the family of possible decision functions in your classification problem. In your case, $a$ defines all possible half spaces along the real axis. That means all points greater than $a$ a classified as $1$ and all points less than $a$ are classified as $0$.

First they show that one point can be shattered. And they choose that point to be $x=0$. They could choose another point but it is enough to show it for $x=0$.

To shatter $x=0$ you need to find an $a$ so that $h(0)=0$ and an $a$ such that $h(0)=1$. (Those are the $2^1=2$ possible labels.)

If the label is $0$ you can choose any $a>0$ to label $x$ correctly. $2$ is one example. With $a=2$ all $h(x)$ will return $0$ for all points less than $2$ (which includes $x=0$).

If the label is $1$ you can similarly choose any $a<0$. $-2$ is one example, but you could also choose $-0.01$ or $-1232132158932$. In the case $a=-2$, $h(x)$ will return 1 for all $x>-2$ (which again includes $x=0$).

So you fulfilled the first condition.

2) Now they show the second condition. But maybe they left out some of the details. Let me try to fill them in.

To show the second condition, you need to pick any two points and show that it is impossible to find $a$ to classify all possible labels correctly. Let's call the two points $x1$ and $x2$. The $2^2=4$ possible labels are: $\{0,0\},\{0,1\},\{1,0\},\{1,1\}$.

Clearly, if $x1=x2$ it follows that $h(x1)=h(x2)$ and therefore the labels must be the same and and only $\{0,0\}$ and $\{1,1\}$ are possible but not $\{0,1\},\{1,0\}$.

Therefore, we can move on to the case where $x1 \neq x2$. Since the naming of $x1$ and $x2$ is arbitrary we can assume that $x1 < x2$ ("without loss of generality" because if $x1 > x2$ we can just rename $x2$ as $x1$ and $x1$ as $x2$ and we would have $x1 <x2$ again).

Now, in this case it is also impossible to achieve all four possible labeling combinations. In particular they show that the labeling $\{1,0\}$ is impossible.
If the label of $x1$ is $1$ then it follows (similarly to 1) with $x=0$) that $a$ must be less than $x1$. But then $a < x1 < x2$ and therefore $h(x2)$ must also be $1$ and can't be $0$.

(You could now move on to the case $x1 > x2$ but as I tried to explain above that's basically the same idea, but now with the labeling $\{0,1\}$).

See this question for a very similar example with intervals instead of half lines.


Find non overlapping area between two kde plots in python I was attempting to determine whether a feature is important or not base on its kde distribution for target variable. I am aware how to plot the kde plot and guess after looking at the plots, but is there a more formal doing this? Such as can we calculate the area of non overlapping area between two curves?
When I googled for the area between two curves there are many many links but none of them could solve my exact problem.
NOTE:
The main aim of this plot is to find whether the feature is important or not. So, please suggest me further if I am missing any hidden concepts here.
What I am trying to do is set some threshold such as 0.2, if the non-overlapping area > 0.2, then assert that the feature is important, otherwise not.
MWE:
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

df = sns.load_dataset('titanic')

x0 = df.loc[df['survived']==0,'fare']
x1 = df.loc[df['survived']==1,'fare']

sns.kdeplot(x0,shade=1)
sns.kdeplot(x1,shade=1)

Output

Similar links

Fill area of overlap between two normal distributions in seaborn matplotlib
Python: Overlap between two functions (PDF of kde and normal)
Fill area-between two curves in python

There are different ways to measure the similarity between two functions.

One option is to define the overlap between both functions as their dot-product:

# ensure both functions are normalized (self-overlap = 1)
x0 /= np.dot(x0, x0)
x1 /= np.dot(x1,x1)
overlap = np.dot(x0,x1)

Instead of multiplying the individual function values as above, you may calculate their difference and take, for example, the mean. This is similar to a loss function in machine learning:

d = np.absolute(x0 - x1)
mae = np.mean(d)    # mean absolute error
mse = np.mean(d**2) # mean square error

If the data is represented on different grid this approach won't work. But you can interpolate your functions and represent them on a new, common grid. A basic example is available in the SciPy documentation. The interpolated data can then be used in the above code snippets.


R - newdata has X rows but variables have X rows I have a dataset dimensions 1142obs in 454 variables.
I've used 'caret' to separate into training and testing datasets.
training =858 obs of 99 var
testing =284obs of 99 var

I make a linear regression model like so (apologies in advance for the extra $ sign before response, which I include here otherwise it won't show correctly):

lm1 <-lm(training$$response ~ training$balloons, data=training)

I then try to make a prediction like so:

lm.predict <-predict(lm1, newdata=testing)

R comes up with:

Warning message:
'newdata' had 284 rows but variables found have 858 rows

lm.predict also generates a vector 858 numbers long, instead of 284.
Any ideas?

Cut down your problem to a minimal reproducible example and supply data or generate it. For your problem I can reproduce with a data frame with two columns (we don't care if its 99 or 2 columns):

training = data.frame(response=runif(858), balloons=runif(858))
testing = data.frame(response=runif(284), balloons=runif(284))

Now you are fitting the model thus, and getting this error:

lm1 <-lm(training$response ~ training$balloons, data=training)
lm.predict <-predict(lm1, newdata=testing)
## Warning message:
## 'newdata' had 284 rows but variables found have 858 rows

Your problem is because you are specifying vectors in the model (ie columns of training using the $ column notation) and also specifying a data frame in the data argument (which is ignored). You get the same error if you leave it out:

> lm1 <-lm(training$response ~ training$balloons)
> lm.predict <-predict(lm1, newdata=testing)
Warning message:
'newdata' had 284 rows but variables found have 858 rows

If your observations are coming from a data frame then you don't need to specify the data as vectors, you give a formula which is evaluated in the context of the data frame:

> lm1 <-lm(response ~ balloons, data=training)

and then:

> lm.predict <-predict(lm1, newdata=testing)

which is 284 predictions as expected.


XGBoost Log Loss different from GridSearchCV Log Loss I have a classification problem where i am trying to predict if the data returns a 1 or 0. So you're classic binary classification. I have my set of data that I have split into the dependent variables (ones i'm training on) and the independent variable (my target that i'm predicting, either a 0 or 1). I am using log loss as the scoring metric for my model.

Firstly I am using the cv function in xgboost to figure out the number of estimators I need as it  stops when the log loss hasn't improved over 50 rounds. I then train my model and predict. My code is below:

def modelfit(alg, dtrain, dtarget, useTrainCV=True, cv_folds=5, early_stopping_rounds=50):

    if useTrainCV:
        # gets the xgb parameters specifically.
        xgb_param = alg.get_xgb_params()

        # this is the internal xgb dataframe that is for efficiency. We map the training data to the labels.
        xgtrain = xgb.DMatrix(dtrain.values, label=dtarget)

        # this performs cross validation on the dataset. As our data is not really time dependent we can afford to cross
        # validate. It stops when it hasnt improved for 50 rounds. This is only for determining n_estimators
        cvresult = xgb.cv(xgb_param, xgtrain, num_boost_round=alg.get_params()['n_estimators'], nfold=cv_folds,
            metrics='logloss', early_stopping_rounds=early_stopping_rounds)

        print(f'Optimal n_estimators - {cvresult.shape[0]}')

        # this sets the most optimal n_estimators parameter into the booster.
        alg.set_params(n_estimators=cvresult.shape[0])

    # fit the algorithm on the data and set evaluation metric
    alg.fit(dtrain.values, dtarget, eval_metric='logloss', eval_set=[(dtrain.values, dtarget)])

    print(alg.evals_result())

    # predict training set:
    dtrain_predictions = alg.predict(dtrain.values)
    print(dtrain_predictions)
    dtrain_predprob = alg.predict_proba(dtrain.values)[:,1]

    # print model report:
    print("\nModel Report")
    print("Log Loss Score (Train): %f" % metrics.log_loss(dtarget, dtrain_predprob))

I then run this function on this particular XGBoostClassifier:

#Choose all predictors
xgb1 = XGBClassifier(
 learning_rate =0.1,
 n_estimators=1000,
 max_depth=5,
 min_child_weight=1,
 gamma=0,
 subsample=0.8,
 colsample_bytree=0.8,
 objective= 'binary:logistic',
 scale_pos_weight=1,
 nthread=-1,
 seed=27)

modelfit(xgb1, X, y)

The logloss value that is returned is: 0.577496 and number of estimators is 65.

I then turn to GridSearchCV to tune the other parameters and I start with:

param_test1 = {
 'max_depth' : range(1,10),
 'min_child_weight' : range(1,6)
}

Note how the original max depth and min child weight are contained within these ranges that i used in xgb1 classifier.

xgb2 = XGBClassifier(
        learning_rate =0.1,
        n_estimators=65,
        max_depth=5,
        min_child_weight=1,
        gamma=0,
        subsample=0.8,
        colsample_bytree=0.8,
        objective= 'binary:logistic',
        nthread=-1,
        scale_pos_weight=1,
        seed=27
)

gsearch1 = GridSearchCV(
    estimator = xgb2,
    param_grid = param_test1, scoring='neg_log_loss', n_jobs=-1, cv=5
)

gsearch1.fit(X, y)
gsearch1.best_params_, gsearch1.best_score_

However this returns me with:

(
{'max_depth': 1, 'min_child_weight': 1}, -0.6275341839742403
)

So my question is how has the grid search said the best parameters are max_depth = 1 and min_child_weight = 1 and the log loss is 0.628 when previously before using GridSearchCV my model returned a better log loss of 0.577 with max_depth = 5 and min_child_weight = 1?

Any help would be appreciated please. Thanks!

Your modelfit prints the training score, but GridSearchCV bases its decisions on the out-of-fold average (and in particular best_score_ is an out-of-fold average score).  This is an unfair comparison, and in particular your 0.577 is probably quite optimistically biased.


Here's an easy solution. The sort order changes, but that shouldn't be difficult to address if you really care:

import pandas as pd

df = pd.DataFrame(a)
df.columns = ['a','b','c','d']
df.index = df.a.astype(str) # to preserve correspondence
df.groupby('a').min()['b']

a
13.0    25.0
24.0    87.0
25.0    29.0
27.0    25.0
62.0    62.0
64.0    57.0
Name: b, dtype: float64

Edit: I think you meant to name your array y instead of a. This works for me:

 from decimal import Decimal

 y=np.array([[196512, 28978, Decimal('12.7805170314276')],
    [196512, 34591, Decimal('12.8994111000000')] ,
    [196512, 13078, Decimal('12.9135746000000')] ,
    [196641, 114569, Decimal('12.9267705000000')] ,
    [196641, 118910, Decimal('12.8983353775637')] ,
    [196641, 100688, Decimal('12.9505091000000')]])

 df = pd.DataFrame(y)
 df.columns = ['a','b','c']
 df.index = df.a.astype(str)
 dd=df.groupby('a').min()['c']

In [210]: dd
Out[210]:
a
196512    12.7805170314276
196641    12.8983353775637
Name: c, dtype: object


Gradient descent multidimensional linear regression - does learning rate affects concurrency? I wonder if gradient descent for multidimensional regression always finds the right result? I feel like this doesn't always have to be true. I have done some calculations and actually got correct results but not for all learning rates I tried to specify. It's not the case that it is too big, cause sometimes with even having too small learning rates I get wrong results. Please tell my if I make mistake in the calculations, or I'm wrong with divergence of the hypothesis.

If you use a convex loss function you always have one optimum point, and you will always be able to find it.

  I have done some calculations and I get correct results but not for all learning rates I tried to specify.

The reason is that you may have selected a large learning rate which may lead to overshoot the optimum. It also may be so small that you don't reach to the desired point and it all depends on your data. If you use the simple gradient descent algorithm, it is an incident that can be seen a lot. You can use other optimisation techniques like Adam which is much more better and you rarely see these problems.


How to test/train a model for realtime data with new data points and classes in a ML pipeline First, For a text classification problem, if I have trained the model on 2 classes and it gives good accuracy. Now, when I use the model in real-time, there is a completely new class from a totally different business team which I need to use the trained model and test this data on. Here, obviously the model will have worse results, and also the pre-processing or any other data cleaning could be way different. So, the question is - Is the only possible way is to re-train the model again?

Second, In a segmentation problem, the approach is to use K-Means clustering and group customers. On a live environment, how do we handle new data points? How do we know which group does the data point belong to? Coz the model has been standardized/normalized properly and then fitted. The same cannot be expected for a new data point. How do we handle this?

In both cases you need to know that the already trained model is not written on the stone. Specially the second question "needs" change of model.

To be more detailed, you are talking about Online Learning in general. Many ML algorithms have online versions. Here, for example, you see the online version of K-means for second question. But in general, online learning is nothing more than adapting a model to the situation in which data comes one sample after the other (like your question). So for basic usage, you can simply adapt your model respectively in a regular manner. this can not be fully online but with a time gap you can retrain the model with new data points which are not assigned to any cluster yet. For fully online-algorithms you can just refer to the link above and start researching.

For the first question I have a meta-question. How do you know that you have a totally new class of texts? So first you need to detect the change and then react correspondingly. To detect the presence of a new class, I propose looking at the logit of the ML algorithm. For instance, if it is a probabilistic approach, you may look at probability of assignment to any present class and say if all less than a threshold and the number of out-of-vocabulary words of new document according to current vocabulary is above a threshold, I create a new class. Not state-of-the-art but for basic usages it will work I assume. Not much to do with preprocessing in a very online manner if it needs a big change. Preprocessing should either be simple and general or very specific according to the class. The first does not need a change and the second needs a bunch of samples from new class to be decided. leave it simple and general for all (like removing strange punctuations, lower-casing, etc.)


Text PreProcessing not improving accuracy I was wondering if TfidfVectorizer() from scikit-learn and its methods fit_transform/transform already do language preprocessing like lowercase/lemmatization/removing punctuation. I am using Imdbs review dump to predict if a review is positive or negative and I wasn't preprocessing initially but recently started to add it in. My accuracy score is 84% using SGDClassifier but whether or not I preprocess the training/testing dataset it seems to be the same?

You can find that information on the class's documentation, here's the link.

Quick summary:

lowercasing: True by default.
lemmatization: I think the closest parameter is preprocessor; it's
set to None by default but you can specify as the value a callable
object of your own and then the vectorizer will apply it on every
word to do the lemmatization, e.g.NLTK's WordNet lemmatizer.
tokenizer: None by default (because the class expects its input to
be already tokenized words, so no need for this unless that's not
so).
stopwords: None by default (but you can specify a list of strings as
the argument and then the class will filter out any words in that
list).

I agree that it is suspicious that you're getting the same Accuracy, generally it is expected to go up when using TFIDF. Make sure you're

1) sending the right input to the TfIdfVectorizer class (it takes a list: list: str, dataset: sentence: each word),

2) using its output (perhaps some variable in your code is still pointing to the non-TFIDF representation instead of the TFIDF one?)

I think it would not be theoretically impossible for SGDClassifier to approximate TFIDF behavior over a sufficiently large number of training iterations with the right learning rate (if all the stopwords' weights are effectively minimized while optimizing loss, TFIDF might have little effect at that point). It would be good to know what parameters you're using and/or to see the code to help you pinpoint the issue.


Multiply weights after using dropout in training - PyTorch I have a Pytorch regression model as follows:

    model = nn.Sequential(nn.Linear(n_in, n_h_1),
                      nn.ReLU(),
                      nn.Dropout(p=0.1),
                      nn.Linear(n_h_1, n_h_2),
                      nn.ReLU(),
                      nn.Linear(n_h_2, n_out))

As you can see, I have used a Dropout regularization layer with dropout probability of 0.1.
When the training is completed, we should disable the dropout. This can be achieved using model.eval(). Reference to  this StackOverflow answer and other resources, we should multiply the output of hidden layer with  (1-p) during inferencing of model.

My question is: do we have to do this manually using pytorch or the library handles this itself  after applying model.eval()?
If we have to do it ourselves, how can we achieve thisfor the given model?

PyTorch handles this with scaling the output of the dropout layer at training time with this probability:

So it handles this itself after applying model.eval(). You can lookup their code here:


Where did this NY Times op-ed get his Google Search data? I hope this is a question appropriate for SO.
The article in question: http://www.nytimes.com/2015/01/25/opinion/sunday/seth-stephens-davidowitz-searching-for-sex.html
As far as I can tell, the only publicly available data from Google Search is through their Trends API.  The help page states that

The numbers on the graph reflect how many searches have been done for a particular term, relative to the total number of searches done on Google over time. They don't represent absolute search volume numbers, because the data is normalized and presented on a scale from 0-100.

However in the article, the author reports (absolute) "average monthly searches".  The source is stated as:

All monthly search numbers are approximate and derived from anonymous and aggregate web activity.
Source: analysis of Google data by (author)

So, how did he get this "anonymous and aggregate web activity"?

Google AdWords.  That has absolute search volumes.


How to obtain and load a good initial data set for object localization? I'm looking for a good data set for training a CNN based network to do object localization (i.e. a data set with class labels and bounding box data).

What is a good initial data set to use?

How can I obtain and load all the data (images, labels, bounding boxes)?

I'm using python, so a python-friendly data source is ideal.

The ones that are available out of the box -

Open Images Dataset (Object Detection Track)
MS COCO dataset
Indoor Objects Detection
ILSVRC dataset

Create your own

Search for the object of interest on Google Images (like bikes, cars etc.)
Use the Fatkun Chrome browser plugin to download all images at once
Use LabelImg to annotate and generate a custom dataset
Use pre-trained models like SSD MobileNet to re-train your own models

Might not be an exhaustive one, but it will definitely get you started. Cheers!


Technical name for this data wrangling process? Multiple columns into multi-factor single column What is the technical name for the following data wrangling process? I want to collapse Table A into Table B. (To make the data suitable for ANOVA.)

Table A:

ArmyVet_ID  Served_WW2  Served_KoreanWar    Served_VietnamWar
110001          1              0                    0
110002          1              0                    0
110004          0              1                    0
110005          0              1                    0
110009          0              0                    1
110010          0              0                    1

Table B:

ArmyVet_ID    Served
110001          WW2
110002          WW2
110004          KoreanWar
110005          KoreanWar
110009          VietnamWar
110010          VietnamWar

Also, the question of how to do the above conversion using R has been asked to death on SO. However, there seem to be way too many ways to do it. If anyone's figured out the absolutely best way to do it (quickest, easiest), I'd appreciate pointers.

Update after correct answer marked below: It turns out that Table A is called "wide format" and B is called "long format".

It is usually called reshaping! For a great description of the process, see this walkthrough, or read up on Hadley Wickham's documentation for the reshape package!


Comparison between addition and multiplication function in deep neural network? I designed a specific Convolution Neural Network to study in the area of image processing. The network has a part that there are two tensors which have to be transformed into a tensor in order to be fed to the next layer. This situation happen in several points of the network. In fact, there are several operations such as addition, multiplication, etc. The results of the network are a bit better when I  use the addition pyramid pooling module (the second image between two convolution) and multiply function (in last step of the network). I used tf.math.add and tf.math.multiply which perform the operation element-wisely. The whole network is shown in the first image.

The second image represents the pyramid pooling module which includes several scale images.

I am looking forward to the addition and multiplication function's attribute in a deep neural network.

The question is:

Why does the addition function (between conv1 and conv2) indicate better final performance in Accuracy (precision) and mean Intersection of Union(mIoU) compared to multiplication and concatenation when I unify two tensors into one tensor?

The observation is very interesting you report, since concatenation and addition are practically the same. A nice explanation can be found in https://distill.pub/2018/feature-wise-transformations/ .


How to use decode_predictions() for non-Imagenet models..? I know that decode_predictions() works for only imagenet dataset(1000 classes) for the models like VGG16 etc.
But condiser my scenario.

My Scenario:

I used vgg16 pretrained model, and added my own weights.

So this turns out to be a non-Imagenet model. I have mentioned classes=9 as i trained my previous model with 9 classes only.

So now to find the predictions, i could use predict() method and then my print(answer) would give the corresponding class label.
But actually i need the class name to be printed.
Is that possible to get class name ?? If it is, can anyone explain me how.?

In Deep learning when you are performing prediction you will get prediction in your case it is an array with 9 probabilities in them. So first perform following operation on them.

import numpy as np

prediction = model.prediction(test_data)
# prediction will contain [[0.6, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05]]

prediction = np.argmax(prediction[0])
# Now predition hold the index of the (0.6) i.e max probability value

Now after that you should have a dictionary which contains key as 0, 1, 2, .. 8 and values as classname1, classname2, ...classname9

This is the way you will get the class name as output


Why the loss is nan by using linear activation function in the last layer? I want to use neural network to solve a simple regression problem, and I try to program by myself accroding to lecture Backpropagation and Neural Networks .
However, I meet loss divergence problem.

My neural network can be descried as:

$l_{1}=\frac{1}{1+e^{-(W_0 x + b_0)}}$

$l_{2}={W_1 l_1 + b_1}$

And loss is:

$loss = {(y-l_2)^T(y - l_2)}$

import numpy as np
import matplotlib.pyplot as plt

np.random.seed(1)
x_pre = 2*np.random.normal(size = (1000,1))
y = x_pre**2 + -0.2*np.cos(3*np.pi*x_pre)
y = y.reshape(y.shape[0],1)

def standard(data):
    mu = np.mean(data)
    std = np.var(data)
    return (data - mu)/std
x = standard(x_pre)

ndim = 10
w0 = 2*np.random.random((1,ndim))-1
w1 = 2*np.random.random((ndim,1))-1
np.random.seed(10)
b0 = np.random.normal(size = (1000,ndim))
b1 = np.random.normal(size = (1000,1))
lr = 1
for j in range(4):
    l1 = 1/(1+ np.exp(-(np.dot(x,w0)+ b0)))
    l2 = np.dot(l1,w1)+ b1#1/(1+ np.exp(-(np.dot(l1,w1)+ b1)))
    l2_delta= np.mean(y - l2)*2*(y-l2)#mse
    l1_delta = l2_delta.dot(w1.T)*(l1*(1-l1))
    w1 += lr*l1.T.dot(l2_delta)
    w0 += lr*x.T.dot(l1_delta)
    b0 += lr* l1_delta
    b1 += lr*l2_delta
    print('loss',np.mean(y - l2))
l1 = 1/(1+ np.exp(-(np.dot(x,w0))))
l2 = 1/(1+ np.exp(-np.dot(l1,w1)))
y_hap =  l2

print(np.sum(np.square(y-y_hap)))
plt.plot(np.arange(len(y)),y,'r')
plt.plot(np.arange(len(y_hap)),y_hap,'g')
plt.show()

output result:

loss 3.485080512494127
loss -30525.316587393125
loss -3293457250652.145
loss -5.777133209515429e+28

Anyone knows how to solve it?

There seem a lot of initialization problems.

There ain't too many biases. Its just 1 per layer. Your shape of
biases seems wrong.
You may have got the network right in shapes since it doesn't return shape errors. But bear in mind, it seems highly likely #1 is
true.
Same goes to w0 and w1.

Consider checking all dims again. If you need, I'm attaching a sample Neural Network implementation in python.

https://raw.githubusercontent.com/MasterSkepticista/NEAT_py/master/network.py

This is as it happens to be taught in Andrew Ng's course.


Comparing Non-deterministic Binary Classifiers I have two classifiers which I am implementing, and they are both non-deterministic in the sense that they can each give different results (FPR and TPR) when you run them multiple times. I would like to compare these two algorithms to evaluate their performance. How do I go about this? Usually what people do most times is to run the classifier till they get the best FPR and TPR values, then they publish the results. But the problem with this approach is that it might not be a good representation of the performance of such a classifier. This is what I planned on doing so far, but don't know if that is correct:

Split my evaluation data into train and test, and after training, predict using the test data to get the FPR and TPR, then repeat this prediction for 99 more times to form 100 FPR and TPR readings, then take an average of this. To get an ROC, use the mean FPR and TPR.  OR
Use k fold cross validation of say k=3 or 10 on the data, and this will return 3 or 10 different values for TPR and FPR, then I will take the mean to get the mean FPR and TPR, and also use this mean for plotting the mean ROC.

Which of the two methods I stated above is ok? And if they are both wrong, what do you suggest I do. Thanks.

A good strategy is to do n times k-fold cross validation, which should give a pretty good estimation of the average performance of both algorithms. What this means is you perform your k-fold cross validation n times, with different random folds each time, and average the results. People commonly use k = n = 10, but the higher the values of k and n the better.


Inverse Relationship Between Precision and Recall I made some search to learn precision and recall and I saw some graphs represents inverse relationship between precision and recall and I started to think about it to clarify subject. I wonder the inverse relationship always hold? Suppose I have a binary classification problem and there are positive and negative labeled classes. After training some of the actual positive examples are predicted as true positives and some of them false negatives and some of the actual negative examples are predicted as true negatives and some of them false positives. To calculate precision and recall I use these formulas:
$$Precision = \frac{TP}{TP + FP}$$ and $$Recall = \frac{TP}{TP + FN}$$ If I decrease false negatives then true positives increases and in that case don't precision and recall both increase?

If we decrease the false negative (select more positives), recall always increases, but precision may increase or decrease. Generally, for models better than random, precision and recall have an inverse relationship (@pythinker's answer), but for models worse than random, they have a direct relationship (@kbrose's example).

It is worth noting that we can artificially build a sample that causes a model which is better-than-random on true distribution to perform worse-than-random, so we are assuming that the sample resembles the true distribution.

Recall

We have
$$TP = P - FN$$
therefore, recall would be
$$r = \frac{P-FN}{P} = 1- \frac{FN}{P}$$
which always increases by decrease in $FN$.

Precision

For precision, the relation is not as straightforward. Lets start with two examples.

First case: decrease in precision, by decrease in false negative:

label   model prediction
1       0.8
0       0.2
0       0.2
1       0.2

For threshold $0.5$ (false negative = $\{(1, 0.2)\}$),

$$p = \frac{1}{1+0}=1$$

For threshold $0.0$ (false negative = $\{\}$),

$$p = \frac{2}{2+2}=0.5$$

Second case: increase in precision, by decrease in false negative (the same as @kbrose example):

label   model prediction
0       1.0
1       0.4
0       0.1

For threshold $0.5$ (false negative = $\{(1, 0.4)\}$),

$$p = \frac{0}{0+1}=0$$

For threshold $0.0$ (false negative = $\{\}$),

$$p = \frac{1}{1+2}=0.33$$

It is worth noting that ROC curve for this case is

Analysis of precision based on ROC curve

When we lower the threshold, false negative decreases, and true positive [rate] increases, which is equivalent to moving to the right in ROC plot. I did a simulation for better-than-random, random, and worse-than-random models, and plotted ROC, recall, and precision:

As you can see, by moving to the right, for better-than-random model, precision decreases, for random model, precision has substantial fluctuations, and for worse-than-random model precision increases. And there are slight fluctuations in all three cases.  Therefore,

  By increase in recall, if model is better than random, precision generally decreases. If mode is worse than random, precision generally increases.

Here is the code for simulation:

import numpy as np
from sklearn.metrics import roc_curve
from matplotlib import pyplot

np.random.seed(123)
count = 2000
P = int(count * 0.5)
N = count - P
# first half zero, second half one
y_true = np.concatenate((np.zeros((N, 1)), np.ones((P, 1))))

title = 'Better-than-random model'
# title = 'Random model'
# title = 'Worse-than-random model'
if title == 'Better-than-random model':
    # GOOD: model output increases from 0 to 1 with noise
    y_score = np.array([p + np.random.randint(-1000, 1000)/3000
                        for p in np.arange(0, 1, 1.0 / count)]).reshape((-1, 1))
elif title == 'Random model':
    # RANDOM: model output is purely random
    y_score = np.array([np.random.randint(-1000, 1000)/3000
                        for p in np.arange(0, 1, 1.0 / count)]).reshape((-1, 1))
elif title == 'Worse-than-random model':
    # SUB RANDOM: model output decreases from 0 to -1 (worse than random)
    y_score = np.array([-p + np.random.randint(-1000, 1000)/1000
                        for p in np.arange(0, 1, 1.0 / count)]).reshape((-1, 1))

# calculate ROC (fpr, tpr) points
fpr, tpr, thresholds = roc_curve(y_true, y_score)
# calculate recall, precision, and accuracy for corresponding thresholds
# recall = TP / P
recall = np.array([np.sum(y_true[y_score > t])/P
                   for t in thresholds]).reshape((-1, 1))
# precision = TP / (TP + FP)
precision = np.array([np.sum(y_true[y_score > t])/np.count_nonzero(y_score > t)
                      for t in thresholds]).reshape((-1, 1))
# accuracy = (TP + TN) / (P + N)
accuracy = np.array([(np.sum(y_true[y_score > t]) + np.sum(1 - y_true[y_score < t]))
                     /len(y_score)
                      for t in thresholds]).reshape((-1, 1))

# Sort performance measures from min tpr to max tpr
index = np.argsort(tpr)
tpr_sorted = tpr[index]
recall_sorted = recall[index]
precision_sorted = precision[index]
accuracy_sorted = accuracy[index]

# visualize
fig, ax = pyplot.subplots(3, 1)
fig.suptitle(title, fontsize=12)

line = np.arange(0, len(thresholds))/len(thresholds)
ax[0].plot(fpr, tpr, label='ROC', color='purple')
ax[0].plot(line, line, '--', label='random', color='black')
ax[0].set_xlabel('fpr')
ax[0].legend(loc='center left', bbox_to_anchor=(1, 0.5))
ax[1].plot(line, recall, label='recall', color='blue')
ax[1].plot(line, precision, label='precision', color='red')
ax[1].plot(line, accuracy, label='accuracy', color='black')
ax[1].set_xlabel('1 - threshold')
ax[1].legend(loc='center left', bbox_to_anchor=(1, 0.5))
ax[2].plot(tpr_sorted, recall_sorted, label='recall', color='blue')
ax[2].plot(tpr_sorted, precision_sorted, label='precision', color='red')
ax[2].plot(tpr_sorted, accuracy_sorted, label='accuracy', color='black')
ax[2].set_xlabel('tpr (1 - fnr)')
ax[2].legend(loc='center left', bbox_to_anchor=(1, 0.5))

fig.tight_layout()
fig.subplots_adjust(top=0.88)
pyplot.show()


Feature Scaling and Mean Normalization In my machine learning class these two methods were discussed and mentioned that both should be used. I have a couple questions about this:

1) Can I mix and match these two approaches? e.g. Feature Scale x1 and Mean Normalize x2?

2) How do you determine which of these options to apply? It seems that either could accomplish the task of increasing your convergence rates... I suppose you just need to know your data set to understand which will reliably reduce your values while leaving as few outliers as possible?

Mean normalization is a form of feature scaling, so these are not really two different approaches. Feature scaling is just a more general term. What kind of feature scaling, e.g. mean normalization,  you need to use ultimately depends on the data.


Optimizer for Convolutional neural network What is the best optimizer for Convolutional neural network (CNN)?

Can I use RMSProp for CNN or only for RNN?

Yes, you can use the same optimizers you are familiar with for CNNs.

I don't think that there is a best optimizer for CNNs. The most popular in my opinion is Adam. However some people like to use a plain SGD optimizer with custom parameters.

An excellent article explaining the differences between most popular gradient descent based optimizers can be found here.


Implementing back translation as a data augmentation for text classification Since back translation English->other language -> English seems like quite a useful data augmentation technique , I wanted to experiment with it. E.g. it occurred to me that languages from very different language families (but very well supported for economic reasons such as Chinese, Russian, Spanish, Korean, Arabic...) could make for a diverse set of effects occurring in the back translation.

Commercial translation APIs would be a straightforward way of doing this, but without free API key or budget from my organization (would not qualify as academic) that's quickly quite expensive for a private thing.

Pretrained translation models would seem like an obvious alternative (I have a GPU for inference, but clearly that's not enough to train all the models from scratch), but I could e.g. not find those for any OpenNMT variant. Are there any recommendations from others that have used this approach?

One option, which I have discovered, is back-translation via the Unsupervised Data Augmentation repository made public by Google Research. This is based on this paper.


What does images per second mean when benchmarking Deep Learning GPU? I've been reviewing performance of several NVIDIA GPU's and I see that typically results are presented in terms of "images per second" that can be processed. Experiments are typically being performed on classical network architectures such as Alex Net or GoogLeNet.

I'm wondering if a given number of images per second, say 15000, means that 15000 images can be processed by iteration or for fully learning the network with that amount of images?. I suppose that if I have 15000 images and want to calculate how fast will a given GPU train that network, I would have to multiply by certain values of my specific configuration (for example number of iterations). In case this is not true, is there a default configuration being used for this tests?

Here an example of benchmark Deep Learning Inference on P40 GPUs (mirror)

I'm wondering if a given number of images per second, say 15000, means that 15000 images can be processed by iteration or for fully learning the network with that amount of images?.

Typically they specify somewhere whether they talk about the forward (a.k.a. inference a.k.a. test) time, e.g. from the page you mentioned in your question:

Another example from https://github.com/soumith/convnet-benchmarks (mirror):


Is it possible to integrate Keras and TensorFlow code? I want to use the pre-trained VGG16 model of Keras, along with another TensorFlow model. I want to take the output from one of the layers of VGG16 in Keras, put it into the TensorFlow model and train only the latter. Is this possible if I use a TensorFlow backend for Keras?

A naive approach to integrate Keras and Tensorflow:

input_img = tf.placeholder(tf.float32, (1,200,300,3), name='input_img')
vgg19 = tf.keras.applications.VGG19(weights='imagenet', include_top=False)
output = vgg19(input_img)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    output_val = sess.run(output, {input_img:
                          np.expand_dims(img,0)})

output_val.shape, output_val.mean()


Public dataset for account to account payments I'm looking for a dataset which contains account to account payments (bank transfers). Ideally, this dataset would contain labeled data for transactions or accounts known to be victims of phishing attacks. In this scenario, the account holder enters and authenticates the transfer but has been tricked into making an undesired purchase or into sending the funds to an undesired recipient.
This could be a public dataset, or optionally I could collaborate confidentially on a private dataset and would sign the necessary confidentiality agreements.
I've looked at the repositories listed here already:
Publicly Available Datasets
I do know this credit card fraud dataset well, and it's the closest to what I'm searching for, but does not fill the requirements above: https://www.kaggle.com/mlg-ulb/creditcardfraud
For experts in this area, a more technical way to describe this fraud scenario is "authorized push payments".

There is this dataset and if it doesnt match, maybe you could contact the authors
at Brussels University for more information.
https://data.world/raghu543/credit-card-fraud-data

The datasets contains transactions made by credit cards in September
2013 by european cardholders. This dataset presents transactions that
occurred in two days, where we have 492 frauds out of 284,807
transactions.

This one uses simulated data but sounds more like what you are looking for. There is also an academic paper that goes with it.
https://www.kaggle.com/ntnu-testimon/banksim1

We ran BankSim for 180 steps (approx. six months), several times and
calibrated the parameters in order to obtain a distribution that get
close enough to be reliable for testing. We collected several log
files and selected the most accurate. We injected thieves that aim to
steal an average of three cards per step and perform about two
fraudulent transactions per day. We produced 594643 records in total.
Where 587443 are normal payments and 7200 fraudulent transactions.
Since this is a randomised simulation the values are of course not
identical to original data.


Do the filters in deconvolution layer same as filters in convolution? I am trying to understand how the deconvolution works in Convolutional Neural Network for image segmentation problem. I have seen this definition:

Filters used in deconvolution is just the transpose of the convolution matrix.

Does this mean that the convolution layer and deconvolution layer shares the same filters? Do we train a separate set of filters for deconvolution layer?

Here, I think, you can find good visuals and explanations for convolution/deconvolution arithmetic.


Batch Size of Stateful LSTM in keras My Model is defined as below:
## defining the model
batch_size = 1

def my_model():

   input_x = Input(batch_shape=(batch_size, look_back, 4), name='input')
   drop = Dropout(0.5)

   lstm_1 = LSTM(100, return_sequences=True, batch_input_shape=(batch_size, look_back, 4), name='3dLSTM', stateful=True)(input_x)
   lstm_1_drop = drop(lstm_1)
   lstm_2 = LSTM(100, batch_input_shape=(batch_size, look_back, 4), name='2dLSTM', stateful=True)(lstm_1_drop)
   lstm_2_drop = drop(lstm_2)

   y1 = Dense(1, activation='relu', name='op1')(lstm_2_drop)
   y2 = Dense(1, activation='relu', name='op2')(lstm_2_drop)

   model = Model(inputs=input_x, outputs=[y1,y2])
   model.compile(loss='mse', optimizer='adam',metrics=['mse'])
   model.summary()
   return model

model = my_model()

It is a Stateful LSTM model with batch size =1. My model.fit looks like this :
# Train the model
history = model.fit(
    x_train,
    [y_11_train,y_22_train],
    epochs=1,
    batch_size=batch_size,
    verbose=0,
    shuffle=False)

model.reset_states()

My model runs well and outputs results. But I am unable to increase or alter the batch_size as flexibly as we could do when the model is in stateless condition.
As in for bigger size of dataset the model seems to be training forever as the batch_size here is just 1. And as we know we can't randomly put any batch_size value for stateful LSTM as it needs to be a divisible factor.
I have gone through some blogs which describes changing batch_size by using different batch sizes for training and predicting using the get_weights() and set_weights() functions in the Keras API ref: https://machinelearningmastery.com/use-different-batch-sizes-training-predicting-python-keras/, still it seems to be that the batch_size that is being used here is less.
My question is : Cant we use batch size like 32, 64, 128 in stateful LSTMs? If yes then how do I implement it in my above given model, if not then what are the alternatives?
Looking for valuable suggestions.
Post Edit
In stateful LSTM model.reset_states() should be after every epoch, Hence I set the resetting of the states after each epoch in the following way:
for i in range(100):
    start = time.time()
    history = model.fit(x_train, [y_11_train,y_22_train], epochs=1, batch_size=batch_size, verbose=0, shuffle=False)
    model.reset_states()

    print("Epoch",i, time.time()-start,"s")

I solved the problem this way:

I realized that I needed to find the HCF (highest common factor) of both the length of x_train and x_test. For this I wrote a simple HCF python code the output of it is the batch_size.

Hence data set of different size will have different batch sizes.

Following is the code I used to find the HCF:

def computeHCF(x, y):
    if x > y:
        smaller = y
    else:
        smaller = x
    for i in range(1, smaller+1):
        if((x % i == 0) and (y % i == 0)):
            hcf = i

    return hcf

batch_size= computeHCF(x_train.shape[0], x_test.shape[0])

And use this batch_size  both while defining the model, fitting the model(model.fit(..)) & predicting(model.predict(...)).

NOTE: We need to specify batch size while predicting the model. As:

model.predict(x_test, batch_size=batch_size)

Hope this helps someone!


What commercial software should a data scientist purchase? I am setting up a work computer and have free reign here. My typical go-to software packages are all freely available, such as Rstudio and Anaconda.

I have thought about investing in commercial BI software such as Tableau or Spotfire, but nobody else in the group (or myself) is a proficient user. Are there other obvious choices I am missing?

At the outset, much of what this group is working on will be exploratory. They are needing some justification to invest further into "data science".

If you are looking for a technical package I would definitely go with Mathematica. It has functions for a very wide range of disciplines that come standard with it (no extra packages to buy). This allows you to expand into techniques as you grow without having to change platforms or buy extra packages.  It also has a great interactive format in CDF with both a free and professional reader. The documentation is interactive as well which really helps a lot.

However, you have to be open to a little bit of structure, typing & chaining functions together, and things like this.  Once you get started it comes very quickly.


Around how much difference between two classes can be accounted as an imbalanced dataset? I am building a binary classification model which has 17K values as class A and 10K values as class B. I want to know when a dataset can face the issues of "IMBALANCED Dataset" ?

In general there will be not a hard rule about this, but this dataset seems to be like balanced.
The point about disbalanced is that you have to keep in mind that the accuracy of your model will have a different starting point. For this python has the function to run a baseline with dummy classification. https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html. This gives you a good indication what is your baseline (if you choose the the strategy to most_frequent). In your case, any prediction model which always forecast class A, would have a accuracy of: 17/(10+17) = 0,6296... , so any binary classification with a score under this value would be really bad


How to choose our data set wisely? I have a couple of questions and I was wondering if you could answer them.

I have a bunch of images of the cars (side view only). I would like to train a model with those images. My objects of interest are 3 types of trucks that have different trailers. I rarely see two target objects at once in an image (maybe $1\over2$ in every 1000 images). However, I do see other types of cars that I do not want to detect.

My questions are:

Do you think I should tackle this problem as a detection task or classification task? (For example, should I consider multilabel classification or omit those pictures).

Should I also include other vehicles that I do not want to detect in my training dataset? Let's say that I do not assign a bounding box to them, but include them in training dataset just to make the system robust. (I trained YOLO with 200 images, sometimes the trained model confused and detected the wrong object that is not in any of classes, this happens when training with 2000 images per class? This is due to a small number of dataset or it is because of not including those images with no bounding boxes)?

Thank you in advance!

You have got nothing to lose by framing it as a multi-label detection task. All you have to do is remove the final Softmax layer and format your ground truth as multi-hot, and you can regress multiple labels directly. Mean Squared Error and Sum of Absolute Differences are neutrally biased with respect to false positives and false negatives if the data are balanced, whereas categorical cross-entropy is biased for higher recall and lower precision and usually does not work as well once the Softmax is removed, so choose your loss function accordingly.

Definitely include more objects in your training set that you don't want to detect--this will reduce overfitting by forcing the detection template(s) to improve their specificity.


Friedman H'statistic for interaction I'm interested in interaction effects in Random Forest. I try to implement Friedman H'statistic basically developped for Gradient boosting Friendman, 2001.
But I have no idea how calculate (44) on the real data.

If anybody has some examples or explanations?

In your article $\hat F$ is defined by Eq. (39). Although it's a bit abstract, it says that the partial dependency function $\hat F_j$ is obtained by averaging over all variables but a subset $s$.

As an example, the one-variable dependency function for the variable $1$ is given by $$\hat F_1(x_1) = \frac{1}{n}\sum_{i=1}^n \hat f \left (x_{1}, x_{i2}, \dots, x_{im} \right ) ,$$ where $\hat f$ is your model prediction function (Random Forest in your case), $n$ is the number of samples, $m$ is the number of features (I've included $\hat f$ instead of $\hat F$ for the predictor to avoid confusion). Here $x_1$ is a variable, and the $x_{i2}, \dots, x_{im}$ values are taken from your samples.

For example, if you have two variables and two samples $(x_1=0,x_2=0)$ and $(x_1=1, x_2=1$) in your dataset (this is a very dummy, pedagogical example), the partial dependence function for the first variable reads:
$$ \hat F_1 (x_1) = \frac{1}{2}\left [ \hat f(x_1, x_2=0) + \hat f(x_1, x_2=1) \right ]. $$

Similarly, you can compute the two-variables partial dependence function as
$$\hat F_{12}(x_1, x_2) = \frac{1}{n}\sum_{i=1}^n \hat f \left (x_{1}, x_{2}, \dots, x_{im} \right )$$ (same as before, but note that the $i$ index is not under $x_2$ anymore).

The two equation I've just made explicit are the only two terms that appear in the equation (44), thus you should be able to compute the $H$-statistic with them.

Let me know if it is still unclear, and check out this for more exegesis on the $H$-statistic.


using "OneVsRestClassifier" from sklearn in Python to tune a customized binary classification into a multi-class classification I have binary classification method name Fclassifier I need to apply it in a multi-class classification problem, this classifier doesn't have any decision_function (or predict_proba) and its core only accept {1,-1} as class labels! I am trying to use OneVsRestClassifier multi-classifier from sklearn so that I can tune the Fclassifier into multi-class classifier:

fm = FClassification(n_iter=1000, init_stdev=0.1, rank=2, random_state=123,l2_reg_w=0, l2_reg_V=0, l2_reg=0, step_size=0.1)
OneVsRestClassifier(fm).fit(X_train,y_train)

but I am facing the following error:

  RuntimeError: Cannot clone object FClassification(init_stdev=0.1,
  l2_reg=None, l2_reg_V=0, l2_reg_w=0,
           n_iter=1000, random_state=123, rank=2, step_size=0.1), as the constructor does not seem to set parameter l2_reg_V

do you think if it is the problem of Fclassifier that can not satisfy the expectation of OneVsRestClassifier and if I should define a customized predict_proba which can fit my data to the classifier in order to solve the problem? or if the error may have other reasons and solutions?

Hi you need to implement get_params method to export all your hyper-parameters or inherit your classifier from sklearn.base.BaseEstimator since under hood of OneVsRestClassifier, every classifier is being distributed. so your classifier is being cloned.


One Hot encoding for large number of values How do we use one hot encoding if the number of values which a categorical variable can take is large ?

In my case it is 56 values. So as per usual method I would have to add 56 columns (56 binary features) in the training dataset which will immensely increase the complexity and hence the training time.

So how do we deal with such cases ?

If you really care about the number of dimensions, you still can try to apply a dimensionality reduction algorithm, such as PCA (Principal Component Analysis) or LDA (Linear Discriminant Analysis), after your one hot encoding.

But know that "56 features" isn't really large and it's highly common in the industry to have thousands, millions or even billions of features.


Replacing column values in pandas I have a data frame which has three columns as shown below. There are about 10,000 entries in the data frame and there are duplicates as well.
Hospital_ID   District_ID  Employee
Hospital 1    District 19   5
Hospital 1    District 19   10
Hospital 1    District 19   6
Hospital 2    District 10   50
Hospital 2    District 10   51

Now I want to remove the duplicates but I want to replace the values in my original data frame by their mean so that it should look like this:
Hospital 1    District 19   7.0000
Hospital 2    District 10   50.5000

As Emre already mentioned, you may use the groupby function. After that, you should apply reset_index to move the MultiIndex to the columns:
import pandas as pd

df = pd.DataFrame( [ ['Hospital 1', 'District 19', 5],
                     ['Hospital 1', 'District 19', 10],
                     ['Hospital 1', 'District 19', 6],
                     ['Hospital 2', 'District 10', 50],
                     ['Hospital 2', 'District 10', 51]], columns = ['Hospital_ID', 'District_ID', 'Employee'] )

df = df.groupby( ['Hospital_ID', 'District_ID'] ).mean()

which gives you:
  Hospital_ID  District_ID  Employee
0  Hospital 1  District 19       7.0
1  Hospital 2  District 10      50.5


Simple Object Detection I want to create a simple object detection tool. So basically an image will be provided to the tool and from that, it has to detect the number of objects.

For example

An image of a dining table which has certain items present on it such as plates, cups, forks, spoons, bottles etc.

The tool has to only identify the number of objects irrespective of the type of object. After identifying it should return the position of the object with its size so that I can draw a border over it.

I don't want to use any library or API present such as Tenser Flow, OpenCV etc.

If the process is very difficult to be created without using an API then the number of/type of objects which it will count as an object can also be limited but since this project will be for my educational/learning purpose can anyone help me understand the logic using which this can be achieved? For eg, it may ignore a napkin present in the table to be counted as an object.

First, you have to have an object detector for recognizing different objects. After that, you have to post-process the outcomes of your detector to count the numbers. Basically, you cannot recognize all objects due to the fact that the number of labels that a detector finds is limited. I highly recommend you using built in models in the libraries you mentioned due to this reason that training detector models are very time consuming and need appropriate hardwre. If you want to learn something, take a look at papers that already exist and try to implement them. For this case there are numerous studies that I separate them in this way.

Pixel-wise classification based techniques
YOLO-based techniques

This is a general classification that there may not be any consensus on that but is something that shows you the mainstream directions.

I guess the ImageNet data-set contains the label clock. Try to use them alongside another label after implementing the model you want. Don't train your model using all the labels which takes to much time.


why variational inference in variational autoencoder if dealing with simple graphical model As far as I am reading about VAE, I always see a graphical model of Z --> X,

I know P(X) can be intractable, but that would be if we would have many dependencies. But here is only Z. So why variational inference at first place?:)

I'm not sure if I get the question correctly, but we use variational inference, because we do have many dependencies. X and Z are usually high-dimensional vectors with complex relationships.