Why is vanishing gradient a problem? Let us say we are using a neural network with $4$ layers with $50,30,20,10$ neurons each. The problem of vanishing gradient would mean that the rate of change of parameters associated with the earlier layer (say first) would be significantly lower than the rate of change of the parameters associated with the later on layers (say 4th).
Now the cost function spans across $n$-dimensions where $n$ is the total number of the parameters and we try to find the minimum of this cost function by varying it across these dimensions.
My question is why does it not lead to the conclusion that the cost function is nearly flat in the dimensions created by the parameters associated with the earlier layers.
Here is the help I can provide on this: this is from neuralnetworksanddeeplearning.com

One response to vanishing (or unstable) gradients is to wonder if they're really such a problem. Momentarily stepping away from neural nets, imagine we were trying to numerically minimize a function $f(x)$ of a single variable. Wouldn't it be good news if the derivative $f′(x)$ was small? Wouldn't that mean we were already near an extremum? In a similar way, might the small gradient in early layers of a deep network mean that we don't need to do much adjustment of the weights and biases?
Of course, this isn't the case. Recall that we randomly initialized the weight and biases in the network. It is extremely unlikely our initial weights and biases will do a good job at whatever it is we want our network to do. To be concrete, consider the first layer of weights in a $[784,30,30,30,10]$ network for the MNIST problem. The random initialization means the first layer throws away most information about the input image. Even if later layers have been extensively trained, they will still find it extremely difficult to identify the input image, simply because they don't have enough information. And so it can't possibly be the case that not much learning needs to be done in the first layer. If we're going to train deep networks, we need to figure out how to address the vanishing gradient problem.

Specific problems from the excerpt:

It is extremely unlikely our initial weights and biases will do a good job at whatever it is we want our network to do. Why? Why cant the cost function be nearly flat in the dimensions created by the earlier weights. To be precise why is the explanation not the the last 3 layers are enough to do such a bangup job that they always find weights corresponding to a very small proximity of whatever random weights we initialized the first layer to. Potato pothatoe. i am asking the same thing in different words to make sure i convey myself

To be concrete, consider the first layer of weights in a $[784,30,30,30,10]$ network for the MNIST problem. The random initialization means the first layer throws away most information about the input image. What does this mean? Kindly explain. from what i understand it is that we cant trace back the second layer inputs unambiguously to the first layer inputs. but, even if we are initializing the weights randomly, we still know what those weights are so how is it that we are throwing away the information?

why does it not lead to the conclusion that the cost function is nearly flat in the dimensions created by the parameters associated with the earlier layers.

Your conclusion sounds very reasonable - but only in the neighborhood where we calculated the gradient.

E.g. (this is an unrealistic example, but bear with me) consider the case in which the cost function is $C(x,y)=100x^2+y^2$, while $x$ is a weight in a later layer and $y$ is a weight in an earlier layer.
I used https://academo.org/demos/contour-plot/ to draw a contour map of $C$, and then drew $4$ steps (arrows starting at the top left) that a gradient descent algorithm might take.
(For an explanation about contour lines and why they are perpendicular to the gradient, see videos 1 and 2 by the legendary 3Blue1Brown.)

The gradient descent algorithm only sees the slopes where gradients are calculated, so here it thinks that the cost function is quite flat in the dimension of $y$:

Imagine a scenario in which the arrows above are even more densely packed, i.e. the steps make an even smaller progress in the dimension of $y$. This would make the gradient descent algorithm really slow.

Here is another intuitive example for an unfortunate case in which the gradient tells us "the cost function is nearly flat in this dimension, so there is no need to move in that dimension".

$\frac{\partial C}{\partial y}$ is very small here, so a gradient descent algorithm would go right very slowly:

As for your other questions, I would try to clarify Nielsen's explanation.

Let's say we have a $[784,30,30,30,10]$ network for the MNIST problem, and it has a severe vanishing gradient problem.
Therefore, after we randomly initialize the weights in the first hidden layer (i.e. the first $30$ neurons layer, whose weight matrix's size is $30\times 784$), they are virtually stuck that way (as the gradient in the earliest layer is extremely small).

Now, consider an unfortunate case in which we randomly initialize the weights in the first hidden layer such that the weights of some pixels are so small, that these pixels' input values are virtually discarded.
To clarify, these input values are multiplied by very small weights, so their weighted value is always close to $0$, regardless of the input value. The weighted value of each input value is the only thing the network knows about the input, and so it is as if these input values were discarded.

In other words, some of the pixels in the input image are invisible to the network.
To demonstrate that, I painted these "invisible pixels" in red:

Let's see how our network sees some MNIST image:

Is it a $6$? Is it a $5$? Even a human like me finds it difficult to say for sure..
What about a picture of an $8$ or $9$? would we be able to identify the digit then?

Nielsen claims that it would be impossible to effectively train such network to identify digits, as it simply has no access to some crucial information - the invisible pixels.


How many parameters in a Conv2d Layer? I was following andrew-ng coursera course on deep learning and there's a question that has been asked there which I couldn't figure out the answer for?

Suppose your input is a 300 by 300 color (RGB) image, and you use a convolutional layer with 100 filters that are each 5x5. How many parameters does this hidden layer have (including the bias parameters)?

2501
2600
7500
7600

The right answer is the fourth.

From this, the formula to calculate the number of parameters in a convolutional layer is (nml+1)*k with n = m = 5, k = 100, l = 3 and +1 for the bias.


Bayesian networks in scikit-learn? I am trying to understand and use Bayesian Networks. I see that there are many references to Bayes in scikit-learn API, such as Naive Bayes, Bayesian regression, BayesianGaussianMixture etc. On searching for python packages for Bayesian network I find bayespy and pgmpy.

Is it possible to work on Bayesian networks in scikit-learn?

If you wish to understand and use Bayesian networks, you can try OpenMarkov, an open-source tool. I recommend you having a look at its tutorial.


Text Similarities: which nlp methods to use? I have data where there is text for each user A visiting a business B. I want to find similarity between each user using their text.

Question 1: Which NLP method should I start with?

I have tried cosine similarity, I know its a great method but it's very slow in computing similarities.

I have thought about extracting important information from text (for example: Term Extraction, I don't want to use LDA) and computing similarities between that information.

Question 2: Any suggestions?

There are many ways to see how texts are similar, but this will depend on your use case.

Semantics

Nowadays, word embeddings are getting popular. Like it was suggested in the comments, you could use Doc2Vec to transform your sentence into a vector and calculate the cosine distance from each sentence.

The idea with these learned embeddings is that you kind of encode the meaning of the sentence by looking how words are used in context. So, to sentences would be similar if words are used in the same manner in a context.

However, this could potentially be difficult to interpret.

For example:

Good tattoo shop. Clean space.
Good pizza restaurant. Large space.
Terrible tattoo shop. Dirty space.

Which of these are semantically close? It will depend a lot on your training and your own judgment to tell if the results are useful to you.

Sentiment

Maybe you are interested in saying that sentences are similar if their sentiment is positive or negative (or on some other scale).

For example:

Suppose you can have sentiment on a scale $[-1,1]$, where $-1$ is negative, $0$ is neutral and $1$ is positive.

Good tattoo shop. Clean space.
    Sentiment = 0.7

Good pizza restaurant. Large space.
    Sentiment = 0.75

Terrible tattoo shop. Dirty space.
    Sentiment = -0.9

There are many resources online on how to do this. Here is an answer with some links.

BoW

The bag-of-words is a very simple approach, but depending on what you are doing, it could do the trick. Suppose you want to say that sentences are similar if they are about a business in the same industry.

You can make a simple dictionary for every industry:

word_per_industry = {
    'restaurants' = ['restaurant', 'food', 'chef', 'dish', 'salad', 'lunch'],
    'tatto_shop' = ['clean', 'dirty', 'art', 'sterlized']
    .
    .
    .
}

Then, per sentences, you can count which business has more words in the sentence, and if sentences are from the same industry they are similar.
Of course, you could also make lists based on another characteristic, not necessarily on industries.


What should I check if model accuracy is no better than baseline level(random guess) I have a data with only 8 columns:

id
created_time
employee_id
rank position
hourly price
num_work_completed
work_category
hired

Hired is the target variable with 1 representing hired and 0 representing not hired, and it is imbalanced with 5.7% hired(1) which makes the baseline accuracy 94.3%
I am trying to build model that predict whether a employee will be hired.
After I finished the EDA, feature engineering(dealing with NAs, encoding categorical variables, normalizing numeric variables), I used 80/20 as the splitting rule and built random forest with rank_position,hourly_price, num_work_completed, work_category_dummy

clf=RandomForestClassifier(n_estimators=100,class_weight=balanced)
clf.fit(X_train,y_train)
y_pred=clf.predict(X_test)

However the model's accuracy(Test accuracy) turned out to be 93% while the baseline is 94.3%.
The training accuracy is 99%. Compared to test accuracy 94.3%, I don't think there's a over-fitting problem
The logistic regression also has the same problem.
Based on correlation blot, most independent variables have pretty weak relationship with target variable smaller than +/- 0.3.
what should I do next to improve my model accuracy?
I tried parameter tuning but it doesn't help a lot.

This is a common problem with rare events modelling, and your options are relatively limited (as far as I am aware, at least). It may well be the case that the features you're using are not very informative with respect to predicting these outcomes.

The major issue is that your predictors, in the context of this model, are not very informative. The model tries to balance false positives and false negatives, but with so few true positives any mistakenly-predicted positive outcome will have a large effect on your classification accuracy.

It seems likely in this case that your predictors do not offer enough information to predict outcomes well. You may have reached the ceiling of what this model can do. This could be an artifact of the rarity of the "hired" outcome in your data set, or it could simply be that the relationship between these predictors and the outcome is weak.

There are a few options, involving use of different techniques (like a Firth regression, designed for rare events modelling). But using different predictors may be the best option, if it is possible to do so. Not every event can be modelled well with some arbitrary set of features, and it may be that you've found one of those.


Confused about how to graph my high dimensional dataset with Kmeans PLEASE NO SKLEARN ANSWERS

So I have a dataset that is very high dimensional, and I am very confused about to convert it into a form that can be used to plot with Kmeans. Here is an example of what my dataset looks like:

Blk Students %       White Students %     Total Endowment    Full time Students     Tuition Revenue

     35                    60               $4,000,000             50000              $4,999,999
     50                    50               $5,888,888             67899              $200,000,000
     .                     .                    .                     .
     .                     .                    .                     .

.

I have normalized my variables, converted everything to integers, etc... But I am confused about how I am supposed to take this high dimensional dataset and plot it on a 2D scatterplot for kmeans? Thanks in advance.

To answer a question asked in the comments: there are 60 more columns than shown here.

Your k-means should be applied in your high dimensional space. It does not need to be applied in 2D and will give you poorer results if you do this. Once you obtain the cluster label for each instance then you can plot it in 2D. However, we live in a 3D world thus we can only visualize 3D, 2D and 1D spatial dimensions. This means you can at most plot 3 variables in a spatial context, then you can maybe use the color of your points as a fourth dimension. If you really want to stretch it you can use the size of your points for a 5th dimension. But these plots will quickly get very convoluted.

You can use dimensionality reduction techniques to project your high dimensional data onto 2 dimensions. What you are looking to do is perform some projection or feature compression (both of those terms mean the same thing in this context) onto a 2D plane while maintaining relative similarity. Many of these techniques exist each optimizing a different aspect of relative "closeness".

The rest of this answer is taken from here.

The following code will show you 4 different algorithms which exist which can be used to plot high dimensional data in 2D. Although these algorithms are quite powerful you must remember that through any sort of projection a loss of information will result. Thus you will likely have to tune the parameters of these algorithms in order to best suit it for your data. In essence a good projection maintains relative distances between the in-groups and the out-groups.

The Boston dataset has 13 features and a continuous label $Y$ representing a housing price. We have 339 instances.

from sklearn.datasets import load_boston
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
%matplotlib inline
from sklearn.manifold import TSNE, SpectralEmbedding, Isomap, MDS

bostonboston  ==  load_bostonload_bo ()
X = boston.data
Y = boston.target

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, shuffle= True)

# Embed the features into 2 features using TSNE# Embed
X_embedded_iso  = Isomap(n_components=2).fit_transform(X)
X_embedded_mds  = MDS(n_components=2, max_iter=100, n_init=1).fit_transform(X)
X_embedded_tsne = TSNE(n_components=2).fit_transform(X)
X_embedded_spec = SpectralEmbedding(n_components=2).fit_transform(X)

print('Description of the dataset: \n')

print('Input shape : ', X_train.shape)
print('Target shape: ', y_train.shape)

plt.plot(Y)
plt.title('Distribution of the prices of the homes in the Boston area')
plt.xlabel('Instance')
plt.ylabel('Price')
plt.show()

print('Embed the features into 2 features using Spectral Embedding: ', X_embedded_spec.shape)
print('Embed the features into 2 features using TSNE: ', X_embedded_tsne.shape)

fig = plt.figure(figsize=(12,5),facecolor='w')
plt.subplot(1, 2, 1)
plt.scatter(X_embedded_iso[:,0], X_embedded_iso[:,1], c = Y, cmap = 'hot')
plt.title('2D embedding using Isomap \n The color of the points is the price')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.colorbar()
plt.tight_layout()

plt.subplot(1, 2, 2)
plt.scatter(X_embedded_mds[:,0], X_embedded_mds[:,1], c = Y, cmap = 'hot')
plt.title('2D embedding using MDS \n The color of the points is the price')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.colorbar()
plt.show()
plt.tight_layout()

fig = plt.figure(figsize=(12,5),facecolor='w')
plt.subplot(1, 2, 1)
plt.scatter(X_embedded_spec[:,0], X_embedded_spec[:,1], c = Y, cmap = 'hot')
plt.title('2D embedding using Spectral Embedding \n The color of the points is the price')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.colorbar()
plt.tight_layout()

plt.subplot(1, 2, 2)
plt.scatter(X_embedded_tsne[:,0], X_embedded_tsne[:,1], c = Y, cmap = 'hot')
plt.title('2D embedding using TSNE \n The color of the points is the price')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.colorbar()
plt.show()
plt.tight_layout()

The target $Y$ looks like:

The projected data using the 4 techniques is shown below. The color of the points represents the housing price.

You can see that these 4 algorithms resulted in vastly different plots, but they all seemed to maintain the similarity between the targets. There are more options than these 4 algorithms of course. Another useful term for these techniques is called manifolds, embeddings, etc.

Check out the sklearn page: http://scikit-learn.org/stable/modules/classes.html#module-sklearn.manifold.


Normalising data with multiple methods When training a neural network, I appreciate that data normalisation helps training. However, is it a good idea to normalise the data in multiple ways. For instance, is it a good idea to apply z-score normalisation on min-max normalised data? That is if the input data is already normalised to [0, 1], is it a good idea to train on the z-scores of that?

Depends on the dataset, and what you're looking for.

It is possible to do multiple data transformations. Standardization is often the safest choice, but only when the data is normally distributed.

If you have normalized the data, you can always apply standardisation. Standardization does not mutate the data set, but rescaling does create a new data set. From the standardized data set you can go back to the original data set.


How is this tensorflow command interpreted? I am reading an introductory tutorial on tensorflow here, and I'm confused about the code that defines an input layer for a word2vec embedding:

# Headline input: meant to receive sequences of 100 integers, between 1 and 10000.
# Note that we can name any layer by passing it a "name" argument.
main_input = Input(shape=(100,), dtype='int32', name='main_input')

Afaik, there are 10.000 words in the library, and the input is a string of 100 words (a news headline). One word is represented as a 32 bit int.

But it seems to me that too little information is passed to this Input layer:

What does "shape=(100,)" mean? why does it omit the second argument?
how does it know how to deal with "dtype='int32' "? If I saw that without context, I would guess that it interprets each of the 100 32bit ints as numbers rather than one-shot word vectors, so I would guess it would create a 100 neuron layer, where each neuron receives 1 input of 32 bits which would be the input to the neuron's activation function. But what we really want is a neural net with 10000 neurons (one for each word) times 100 (for each word in the sequence), right?

What am I missing?

This question is very much about code syntax of the Keras Input layer (Tensorflow backend)...

shape=(100,) meaning - The shape variable takes as input a tuple of integers, not including the batch size. For instance, shape=(100,) indicates that the expected input will be batches of 100-dimensional vectors.
It knows how to deal with "dtype='int32'" because that is how the Input class was written (I have no better answer). The data type variable takes as input a string (float32, float64, int32...).

The official documentation of the Input layer: tf.keras.layers.Input

This link has all the answers you are looking for.


what is criterion in flcuster of scipy package? Could some one explain what does criterion of fcluster indicate? I tried to read the documentation but I am unable to understand. What does maxclust criterion indicate?

Welcome to the community!
You may want to refer to a tutorial on Agglomerative Hierarchical Clustering before reading this answer. My explanation is more practical.
Assume the data below:
from scipy.cluster.hierarchy import ward, fcluster
from scipy.spatial.distance import pdist
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.text import TextPath

X = [[0, 0], [0, 1], [1, 0],
     [0, 4], [0, 3], [1, 4],
     [4, 0], [3, 0], [4, 1],
     [4, 4], [3, 4], [4, 3]]
X = np.array(X)
Z = ward(pdist(X))

plt.plot(X[:,0],X[:,1],'o')

We have 12 points in 2 dimensions. This is how they are distributed:

Now let's see how maxclust affects the clustering. Please note that I named clusters by numbers and plot the name of cluster each data belongs to, instead of its point.
for tt in [1, 3, 5, 9]:
    plt.figure(figsize=(8,8))
    plt.title('t={}'.format(str(tt)))
    plt.xlim((-1,5))
    plt.ylim((-1,5))
    memberships = fcluster(Z, t=tt, criterion='maxclust')
    for ii in range(len(memberships)):
        path = TextPath((X[ii,0],X[ii,1]), str(memberships[ii]))
        plt.plot(X[ii,0],X[ii,1],marker=path,c='b',markersize=50)

    plt.show()

Results are shown below. I explain each:
t is the parameter which limits number of clusters i.e. you can have maximum t clusters in output. If t=1, any other criteria will be ignored as all data points have to be in the same cluster.

So far so good. Let's say t=3. Now the fun starts. maxclust finds an optimal distance between each pair of points which are going to be in the same cluster. If t=3, without that distance, you get unbalanced clusters. In the figure below we have 4 obvious clusters and if I want 3 clusters out, two of them will merge and cause a cluster which is "wrong" (let's say wrong for simplicity). The maxclust distance threshold, prevents the algorithm to do so as it rejects far points being in the same cluster. So you see the data is clustered to 2 clusters instead of 3 which makes perfect sense. Please note that, yes, we have bigger clusters. But they are "right". It is like we zoomed out and see the clusters from a higher perspective.

Let's go with t=5. Now the algorithm is allowed to find 4 clusters. Topology tells you that finding "right clusters" i.e. clusters in which each pair of points have a relatively small distance (in compare with distances to other clusters), is possible. This means that maxclust can find an optimal distance threshold and t lets us have 4 clusters. So it works!

The last but not least is t=9. No we can have clusters more than the actual number needed (which was 4). Now maxclust tries to find its minimum distance between pairs so that points in same clusters are closer to each other than to other points in other clusters. As you see in each real cluster, The distance between center points to other two is smaller than the distance between those two diagonal points (as this distance is identical, clustering algorithms choose members randomly). So you see that each real cluster is divided to two, due to the effect of combination of maxclust and t

Hope it helped. Good Luck!


How do I crop faces with a neural network? I am looking to build a facial recognition system and realized I could probably pump up accuracy by first cropping the faces. I know I can use a Haar Cascade classifier to do this but would prefer to use a deep learning approach, as that is my current area of focus. To do this do I simply create a CNN with inputs being an image and outputs being the X, Y, Width, Height? Or is there more to it? I have tried this approach with an Adam optimizer and around 600 pictures but results haven't been great. Any ideas on how to improve this?

You are looking for a fully convolution neural network trained for image segmentation. The architecture is typically convolutions followed by de-convolutional layers with out pooling.

The forward inference serves to extract information with convolutions. The de-convolutions serve to resize the tensors to the size of the inputs height and width. A fully connected layer then classifies each pixel from the input. Deconvolutions demonstration.

Unfortunately you need pixel wise labels to train a network to do this. You could look for a pretrained model that already segments faces from images or you could grind out hand labeling for a 1000 images and use transfer learning (there are online tools for doing this such as Label box.


Dropout classifcation I have a data set of recruiting pipeline information that has 12 columns of categorical variables (they range from binary variables like gender to non-binary variables like the name of the school the applicant attended). The last column (13) contains a "Yes" or "No" value that tells if the person is still in the recruiting process.

The idea here is to build a model that can predict (reasonably well) the likelihood of the person withdrawing from the recruiting process at any given stage (there are many stages and this field is one of the 12 independent variables captured).

I was thinking of using logistic regression to create the model but all the predictor variables are categorical, which I hear, doesn't bode well for logistic regression. Another factor is that some fields have missing values (like gender) that I cannot reasonably input/capture accurate data for atm.

Given my situation, what approach do you think would be a good starting point?

Was thinking of random forest but not sure if there's a better way to go about tackling this problem.

There isn't a particular approach or model that is definitively the best. You might want to try multiple types of models. A good starting point would be to visualize your data. If there are many non-linear relationships between your data and the the outputs you are predicting then you might want to start with a tree based model. It will be much less work to make it perform well. If the relationship between your variables and the outcomes you are predicting are fairly linear then a logistic regression model is a good approach.

If you have to have a silver bullet model to solve all your problems then I recommend trying xgboost. Use grid search and you will have a good model.

I actually worked on a similar project for my university. Aggressive feature engineering combined with logistic regression performed and xgboost were the two best models.


Reinforcement Learning (Q Learning) I was reading a paper on traffic flow optimization using Multi-Agent Q learning. the paper proposes the following method:
Deploy a Reinforcement learning controller at each intersection with traffic lights.

first the Q value equation is:

$Q^{t}(s,a) = (1- \alpha )Q^{t-1}(s,a) +  \alpha (R{t} +  \gamma max_{a}(Q^{t-1}(s,a))$

second the state is:
the sum of vehicle queues lengths at the current intersection and one hop intersections

third the action space is:

here the actions represent the possible moves of vehicles at the intersection.

forth the reward at time t is

$R^{t} = -( w_1\sum q_{current intersection} + w_2\sum q_{neighbors} )$

where the q refers to vehicle queues lengths, w1 and w2 are constants.

fifth there is the algorithm in the image below where it acquires the action required for maximizing the Q value

What I am trying to understand is, the reward calculation does not take an action as a parameter. How does it choose an action properly.

I am a newbie to reinforcement learning, so please if you find my question naive, cosedire referring me to a proper textbook.
Thanks

What I am trying to understand is, the reward calculation does not take an action as a parameter. How does it choose an action properly.

The reward function does not choose the action. It gives the immediate consequences for your previous action - and in most cases is a consequence of all previous actions, states and random factors in the environment. Although it should only depend on most recent state, previous action plus a random factor at most.

In your case reward depends directly on the resulting state = how many cars at each junction after taking an action, and not directly on the previous action = which traffic flow to enable. It does still depend on the action indirectly, via however the environment works. But that does not need to be expressed mathematically (as a model of the system) for Q learning to work. Although not stated, the reward probably also depends on a random factor outside of your control = how many cars turn up at each junction.

It is the Q value which eventually ranks the different actions and allows you to select the best action. The value $Q^t(s, a)$ gives you the current best estimate for future rewards (for a continuous problem, such as yours, it is common to have a discount factor, $\gamma$, to give more weight to immediate rewards).

There are multiple ways to select the next action in reinforcement learning, depending on the problem, your learning algorithm etc. In the example pseudo-code you copied, the selection process is using Upper Confidence Bound (or UCB), which is a relatively sophisticated way of balancing exploration of actions that might be better than your current best estimate, versus simply using the "best" action so far. If you always took the "best" action, in many cases this would cause a problem because you would never update your knowledge of what the other actions do.

One of the standard textbooks for all this is Sutton & Barto, Reinforcement Learning: An Introduction. The notation in that book is slightly different to what you have used, but it explains the concepts involved in your problem in easy-to-understand detail.


Calculating Univariate and MultiVariate Logistic Regression with Python I have a simple data set of a number of variables and a single binary dependent variable.  The data is stored in a data frame.  When I use python's statsmodels.api and logit.fit() on the dataframe I am presented with a table detailing p values and confidence intervals etc for each of the variables.  I need to calculate both univariate and multivariate p values and confidence intervals for each variable, however I am unsure what logit.fit is calculating - multivariate?  If so how do I calculate univariate values - maybe just analyse a single variable at a time?  Sample output below:

==============================================================================
Dep. Variable:           Vehicle        No. Observations:                11540
Model:                          Logit   Df Residuals:                    11515
Method:                           MLE   Df Model:                           24
Date:                Thu, 29 Aug 2019   Pseudo R-squ.:                 0.05443
Time:                        11:57:39   Log-Likelihood:                -7463.8
converged:                       True   LL-Null:                       -7893.4
                                        LLR p-value:                6.082e-166
=========================================================================================
                            coef    std err          z      P>|z|      [0.025      0.975]
-----------------------------------------------------------------------------------------
Red                    0.0084      0.001      6.880      0.000       0.006       0.011
Green                  0.1345      0.041      3.293      0.001       0.054       0.215

I case somebody is searching for the same thing.  The answer is fairly obvious.  The results as presented are a multivariate analysis.  To achieve a univariate analysis process a single variable at a time.


If I use Gibbs sampling with a Bayesian model, what do I have to check is memoryless? Right now I am trying to better understand how Bayesian modeling works with just the basics.  I found through reading tutorials that some very basic Bayesian models like Bayesian Hierarchical Modeling use something called the "Gibbs sampling algorithm", which is a Markov Chain Monte Carlo Method algorithm.

I know that, if I am going to do anything with Markov Chains, then I have to test a data or parameter violates the assumption of memoryless.  However, I am uncertain what exactly do I need to test for memorylessness.  Is it a parameter, the dataset, the response variable, a particular residual or error distribution.... what is it that has to be memoryless if you use Gibbs Sampling?

(P.P.S.) I did find a paper on that test's memoryless for Bayesian models but its really confusing.  I am a little familiar with how to test memorylessness for basic data, but I am lost on more complex algorithms like Gibbs Sampling.

It is best practice to check for memoryless. It is also common to assume the process is memoryless, then use Gibbs sampling.


machine learning mode are suitable for using word features as x, and y as click rate of text Hi I want to extract words of english texts using click rate with machine learning model. Now I know the click rate of text, and I know How to extract words(unigram) of each text, for example, there are about 10000 texts,and the click rate of each text is provided. How to extract words features for click rate. How to extract key words and compute the importance of each word for computing click rate.

You can build a dictionary of character sequences (words) and for each instance of text you will count the occurrence of these words. You can either use groupings of characters n-grams or words them selves using bag-of-words.

n-grams

n-grams is a feature extraction technique for language based data. It segments the Strings such that roots of words can be found, ignoring verb endings, pluralities etc...

The segmentation works as follows:

The String: Hello World

2-gram: "He", "el", "ll", "lo", "o ", " W", "Wo", "or", "rl", "ld"
3-gram: "Hel", "ell", "llo", "lo ", "o W", " Wo", "Wor", "orl", "rld"
4-gram: "Hell", "ello", "llo ", "lo W", "o Wo", " Wor", "Worl", "orld"

Thus in your example, if we use 4-grams, truncations of the word Hello would appear to be the same. And this similarity would be captured by your features.

Bag-of-words

Bag-of-Words builds a dictionary of the words it has seen during the training phase. Then using the word the frequency of each word in the example a vector is created. This can then be used with any standard machine learning technique.


How to identify the feature that make the model misclassifed in text classification Hi I am working on social media financial THAI text classification, the problem with this one is the confused classes, the misclassified prediction has a pattern that consistent as a pair.
and I want to know how can I identify the word/feature that responsible for the misclassified and lead the model to confuse with the prediction?

I think we would need more information about the model architecture you are using and the features you engineered for training.
Something I like to do when starting to work on text classification problems is to gradually increase the complexity of the architecture and start from simple approaches.
So if you are using tf-idf features and some sort of probabilistic model, such as logistic regression you may have a look at libraries such as Lime or eli5 that can help you make sense and explain the prediction of your classification model.
In your case you may then look at the top-features of the misclassified text and increase your understanding of why the model derived at its decision.


How to debug data analysis? I've came across the following problem, that I recon is rather typical.

I have some large data, say, a few million rows. I run some non-trivial analysis on it, e.g. an SQL query consisting of several sub-queries. I get some result, stating, for example, that property X is increasing over time.

Now, there are two possible things that could lead to that:

X is indeed increasing over time
I have a bug in my analysis

How can I test that the first happened, rather than the second? A step-wise debugger, even if one exists, won't help, since intermediate results can still consist of millions of lines.

The only thing I could think of was to somehow generate a small, synthetic data set with the property that I want to test and run the analysis on it as a unit test. Are there tools to do this? Particularly, but not limited to, SQL.

Here is a suggestion:

Code your analysis in such a way that it can be run on sub-samples.
Code a complementary routine which can sample, either randomly, or by time, or by region, or ...  This may be domain-specific. This is where your knowledge enters.
Combine the two and see if the results are stable across subsamples.


Storing and collecting data I wanted to ask you which coding language I should use. I am pretty new to coding and could really use some help.
As I mentioned above I want to create a survey and afterward store my results in a database.
Details:
The survey should be available for a longer period of time (a few years) and the results should be transferred daily to the database.
The survey starts with a general identification (name, age, gender, etc.).
An ID is then assigned to the participants (in case they want to add more answers to the survey AFTER they have completed it).
Afterward, the identification of the actual survey takes place. In this part, I would like to query recurring events. For example, classes attended at the university with subcategories (date, GPA, etc.) Each candidate has attended 1 to x classes.
Here is my first problem, how can I achieve that every candidate can enter all his classes?
For example, is there the following possibility that I ask the following question after each class: Have you attended any other classes? If "NO", the survey is declared closed, if "YES", he can enter another class.
The next problem, I have is that I want to make sure that different people name the same class the same way. So I want to have the possibility that as soon as you are participant 2, you get the already entered classes of participant 1 as a suggestion.
Then I would like to transfer the information into a database. (if possible with MySQL).
And if possible I also would like to visualize it.
I just wanted to ask you which coding language you would recommend.
So far I have little coding experience with Python and Java.

There are several separate issues:

Survey design
Collection
Data querying
Visualization

Each issue has its own best practices for solutions.

Generally, survey design and collection are independent of coding language. Data querying from a database is done with SQL. Visualization can be done with or without a coding language.

Python and Java can help to automate the process.


How to increase accuracy of model from tensorflow model zoo? Situation:

My dataset is 70k images of people wearing clothes. Images are labeled: bbox position and class. There are 10 classes. I did 80:20 split. Categories are balanced with exception of one category, but I can accept poor performance on one category.

Goal is cloth recognition in images. When I feed image of person wearing pants and tshirt, I want to see two bboxes of these clothes.

My problems:

I already trained few models from tf model zoo. I did over 100k steps on ssd mobilenet v1 and faster rcnn resnet 101.

Problem with ssd is that it won't converge. Loss is not getting below stable 2 and accuracy is bad. Problem with faster rcnn is that loss is below 1 but it's varying a lot and sometimes it jumps over 1.

What I've done:

I tried different batch sizes for ssd with no luck. FRCNN is locked with batch size 1. I improved dataset multiple times. I went from 50 unbalanced classes to 10 balanced classes. I didn't tweak hyperparameters from models configs besides batch size.

My access to strong gpu is limited for me so I can't just randomly try different hyperparams combinations with hope that it will work. Could you suggest me few things that I can do in order to improve my models? I would be very thankful.

Are you training the models from scratch? If yes, then can you try using pre-trained models and fine-tune for your specific dataset.

You'll have the experiment with hyperparameters. Learning rate and optimizer (e.g., sgd, adam, rmsprop, adadelta) will have largest effect on model training and performance.


How to fit a KNN and then a linear regression with those neighbors? How do I fit a KNN to get the $k$ nearest neighbors and then aggregate the those neighbors into a fit using a linear regression (instead of a weighted average) in Scikit-Learn?
I've tried creating a pipeline using KNeighborsTransformer and then LinearRegression but that doesn't appear to be doing the right thing.

KNeighborsTransformer only gives you the indices of the nearest neighbors and the distances.  You need to do more work to retrieve the points to fit your linear regression.
Here's a draft that appears to be working:
from sklearn.neighbors import NearestNeighbors
from sklearn.base import RegressorMixin, BaseEstimator, clone
from sklearn.linear_model import Lasso
from sklearn.utils import check_X_y
import numpy as np

class LocalLinearRegressor(RegressorMixin, BaseEstimator):
    def __init__(self, n_neighbors=10, linear_model=Lasso()):
        self.n_neighbors = n_neighbors
        self.linear_model = linear_model

    def fit(self, X, y=None):
        "Fits the neighbors search."
        X, y = check_X_y(X, y)
        self._fit_X = X
        self._fit_y = y
        self.neighbor_search = NearestNeighbors(n_neighbors=self.n_neighbors)
        self.neighbor_search.fit(X)
        self.local_regressors_ = {}
        return self

    def predict(self, X):
        """Fits linear regressions on the k nearest training points to predict new values.

        We don't fit these linear regressions at fit time because there would be so many.
        However, we do save the regressions as we see them to speed up predictions.
        """
        neighbors = self.neighbor_search.kneighbors(X, return_distance=False)
        ksets, mapper = np.unique(neighbors, return_inverse=True, axis=0)
        for kset in ksets:
            if tuple(kset) in self.local_regressors_:
                continue
            local_X = self._fit_X[kset, :]
            local_y = self._fit_y[kset]
            self.local_regressors_[tuple(kset)] = clone(self.linear_model).fit(local_X, local_y)
        return np.array([
            self.local_regressors_[tuple(ksets[mapper[i]])].predict(X[i, :].reshape(1, -1))[0]
            for i in range(X.shape[0])
        ])

And here's a Colab notebook showing it in action.


Why does an imbalanced data set badly effect distance measures like Mahalanobis? I'm relatively new to data science and I am struggling to understand why the Mahalanobis distance (or any other distance measure) applied to an imbalanced data-set becomes inaccurate. I have a data set that consists of three classes A, B and C. There are 100 observations for class A, 60 for class B, and 20 for class C. When I calculate the Mahalanobis distance between each class, the results do not appear consistent with my PCA (principal component analysis) plot. In the PCA plot, class C is the most separate class; however, the Mahalanobis distance does not reflect this.
For balanced data sets, i.e., where classes A, B and C have the same number of observations, this has never been an issue. The Mahalanobis distance has always quite accurately reflected the results of PCA for balanced data.
I have read some similar questions and answers on here about why imbalanced data must be handled carefully for classification algorithms, but is this the same for distance measures? From what I can tell the Mahalanobis distance doesn't explicitly depend on sample size.
Therefore, I ask why does this measure lose reliability for imbalanced data?

Mahalanobis distance is defined as a distance between a point and a distribution. The key is how you define the distribution and I would say the imbalance of classses is not the problem in itself here. What could be a problem is mahalanobis is sensitive to initialization and the sample sizes of your classes are not that big. You could check the covariance matrix if it is reasonable for you task.
About more general question -- the distance between two points obviously does not depend on size of the classes. If we talk about distance between a point and a set, then it could affect the result: i.e. you defined the distance as the distance to the closest point in set, then obviously the more points, the bigger chance to get the closer point.


Data sets that have strings and numerical data all in one column Here's a code I wrote for pre-processing a data set. It works
import numpy as np
import pandas as pd
from sklearn import svm

%matplotlib inline
import matplotlib.pyplot as plt

from sklearn.impute import SimpleImputer
import seaborn as sns; sns.set(font_scale=1.2)

stock=pd.read_csv("C:/Users/Dulangi/Downloads/winequality-red.csv")
stock.head()
X= stock.iloc[:,0:5].values
y= stock.iloc[:,5].values

g=sns.lmplot('alcohol','quality',data=stock,height=7, truncate=True, scatter_kws={"s":100})
imputer = SimpleImputer( strategy = "mean")

imputer = imputer.fit(X[:,1:2])
imputer.fit_transform(X[:,1:2])

imputer = imputer.fit(X[:,4:5])
imputer.fit_transform(X[:,4:5])

I want to know what if i have both strings and numeric data in one column, how to pre-process such data to have all numeric data?

Normally pandas should identify numeric types automatically. If it doesn't in your case, there seems to be a formatting issue.

Instead of reading everything into string, I'd rather try to enable pandas to read the types directly in the correct form. First I'd try to pass the column types in a dict to dtype, as in:

pd.read_csv(file_name, dtype={'int_column_name': 'int32', 'float_column_name': 'float32'})

If that is not applicable, because the format is not recognized by pandas automatically, you might want to try some other options.

E.g. if it's just the decimal point or the thousand separator that is different in your data, you can set it over the corresponding keywords in read_csv (thousands and/or decimal).

If that isn't sufficient because you have some special formatting etc., you can also pass your own converter to parse the string into the data type you require for the column, like this as an example of how you could parse numeric values with a currency unit:

import re
mon_re=re.compile('(?P<value>[0-9.]*)([^0-9].*)?')
def strip_off_currency(currency_string):
    m=mon_re.match(currency_string)
    if m is not None:
        return np.float32(m.group('value'))
    else:
        return np.NaN

pd.read_csv(file_name, dtype={'int_column_name': 'int32', 'float_column_name': 'float32'}, converters={'monetary_ammount': strip_off_currency})

The reason why I would prefer this way over reading everything into memory and processing it there is, that it needs less memory and probably also is faster if you process the values just once (at least if you don't need to pass converters).


Python library to compute some metrics for multioutput-multiclass classification task Is there any Python library that provides ready-to-use metrics to analyze the performance of a classifier for a multioutput-multiclass classification task? scikit-learn doesn't have this option yet (as stated in the documentation and in the corresponding feature request on GitHub).

Late answer!  Sorry if I am repeating things you know...

I am sure somebody has worked on this formally, but least in my short industry experience, it might just be easier to roll your own solution here.  I recently had to do this for classifying businesses (into restaurants, or dentists, or doctors, or amusement parks etc). Because calling a dentist a doctor was a relatively minor error compared to calling a doctor an amusement park, certainly all mistakes are not equal.  My colleague decided to make a different ROC curve for each true class, and call things a "hit" if the classification was right, and a "miss" if the classification was wrong, and so he could leverage typical scikit-learn stuff, but it still only told part of the picture.

With all that in mind, we can still be quantitative.  Perhaps you could decide on a misclassification matrix pentaly, where cell (i,j) is the penalty for calling an "i" a "j", with 0 penalties on the diagonal for correct classifications.  Then you could at least compare models (at one particular set of thresholds) by their cumulative error or squared error etc.

It's already a little tough striking a tradeoff between false positives and false negatives in a binary classification problem - and it depends very heavily on your subject matter and what risks you are willing to tolerate. Do you want to be able to say "we aren't confident and therefore we won't guess"?  This complicated things further for me, because we were trading off accuracy with automation, so "not sures" were also unfortunate, but in a different sense.


Class Imbalance and Cost-Sensitive Learning XGBoost I'm fairly new to data science and machine learning and have been trying to read a bit more on methods like boosting for one of the projects I am working on. The investigator on this project is interested in utilizing xgboost for this project, but there are a couple of main issues that we anticipate once we get the data.

There will be class imbalance, as in way fewer 1's than 0's. I've seen that people handle this by using the scale_pos_weight parameter in xgboost, but from my understanding, xgboost allows for custom objective functions. I was wondering how people come up with custom objective functions and the basis behind their chosen objective function. I'm interested in a customized objective function that penalizes false negatives more and was wondering if there's anything out there on the web that's recommended. We are dealing with a classification problem, so by default we would be using a log-loss function. I am hoping there is some sort of custom log-loss function out there that is commonly used.

I apologize if I am using the wrong terminology, as I'm still very new to this. Thanks so much!

XGBoost has several parameters to tune for imbalance datasets. You wouldn't mess with the objective function from my knowledge. You can find them below:

scale_pos_weight
max_delta_step
min_child_weight

Another thing to consider is to resample the dataset. We talk about Undersampling, Oversampling and Ensemble sampling. I think I was using the imbalanced-learn Python library for that. Things can get even more creative there. For instance, create many XGBoost trees on the same undersampled class instances but each XGBoost tree holds another copy (or randomly sampled copy) of your oversampled class instances. Then you can average your results.

In the end, you will have to try and see for yourself.


Why training error is larger than validation error after PCA? We have 4000 features and we are applying Principal Component Analysis to reduce them a small number of features from 20 to 100.

We are performing linear regression.

Both training and validation errors are worse than without any use of PCA.

However we notice that the training error is not lower and actually has become slightly larger than the validation error.

Is this something expected?

Edit:

The variance ratio of the first principal components is high enough in comparison to the others.

What we are noticing is that if you use all of the 4000 principal components then the gap between training error and validation is larger and as you repeat the process and you reduce the principal components to 1000, 200, 100, 10, etc. then the training error comes closer to the validation error (even though both are being increased as expected)

The basic principle that helps us implement PCA is it's ability to explain the variation, so it depends how much PC you are taking and how much variation they are explaining. They are definitely not explaining 100% of variation so there are good chance that Validation Error is higher.

Before applying PCA you need to understand why we apply it? We apply it to reduce computational effort and practical implementation of problem. The idea of PCA is not to give better result than actual variables in problem.

Better results are generally available with complete list of variables.


Drastic shift in feature importance upon adding other features I have a model (GBDT) where adding a feature X is not important (according to SHAP), but when I add other features, and add X again, now feature X is the second most important!
What could explain that? How do I investigate what is going on?

There's a good chance that it's a sign of overfitting: the fact that the importance of the features is not stable can be considered as an indication that the model itself is not stable, and this typically happens when it doesn't have enough information in the data to be sure how to use the features. As a result minor variations in the features or data cause the model to change a lot, because it uses features which occur by chance. A way to investigate is to reduce the number of features: if the model becomes more stable this way, it confirms overfitting (and the performance on the test set should be the same or better).
[edited] There is also a possibility that the new features are really useful for the model, causing it to use the whole set of features in a very different way since it can take advantage of new combinations of features. For instance, suppose we have a model which predicts a disease based on features which represent the patient's symptoms, then we add features which represent the patient's age and gender. Let's assume that having a particular symptom at a particular age is a strong indicator of the disease, so this would mean that this symptom feature was not very useful on its own, but becomes more important with the additional feature.
In such a case I would probably expect to observe a significant improvement in performance on the test set when the new features are added.


Should I include all dummy variables or N-1 dummy variables (keep one as reference) in neural networks I have a categorical variable with N factor levels (e.g. gender has two levels) in classification problem. I have converted it into dummy variables (male and female).
I have to use neural network (nnet) to classify. I have two options -

Include any N-1 dummy variables in the input data (e.g. include either male or female). In statistical models, we use N-1 dummy variables.
Include all N dummy variables (e.g. include both male and female)

Can someone please highlight the pros and cons of both options in predictive power and interpretability

I will answer the question in the Neural Network context (i.e. I won't talk about regularization in regression algorithms handles this problem).
The problem of not dropping one of the encoded variables will cause multicollinearity. That is, one of the variables can be estimated using others. e.g. If you have a variable that says whether a person is female or not, why would you need another variable which says whether a person is male or not. However, the main thing that people forgot to say while answering this question is that multicollinearity is not actually a big problem unless you need the interpretation of your variables. Multicollinearity will cause your coefficients to be false, but it won't affect your predictions. Thus, it should not be a big problem unless you need to explain your model in terms of your variables. Another possibility is that, if the covariance between your variables in the training set and test set would be different, then your predictions would be affected, and you would have incorrect results. However, if you shuffle your dataset good enough (which is your assumption) and split the train and test set (also validation) correctly then covariance should be the same in both. In other words, the relation between those correlated variables can be assumed to stay the same in the train and test set. Thus, you can safely assume that your predictions are correct.


Predicting next medical condition from past conditions in claims data I am currently working with a large set of health insurance claims data that includes some laboratory and pharmacy claims. The most consistent information in the data set, however, is made up of diagnosis (ICD-9CM) and procedure codes (CPT, HCSPCS, ICD-9CM).

My goals are to:

Identify the most influential precursor conditions (comorbidities) for a medical condition like chronic kidney disease;
Identify the likelihood (or probability) that a patient will develop a medical condition based on the conditions they have had in the past;
Do the same as 1 and 2, but with procedures and/or diagnoses.
Preferably, the results would be interpretable by a doctor

I have looked at things like the Heritage Health Prize Milestone papers and have learned a lot from them, but they are focused on predicting hospitalizations.

So here are my questions: What methods do you think work well for problems like this? And, what resources would be most useful for learning about data science applications and methods relevant to healthcare and clinical medicine?

EDIT #2 to add plaintext table:

CKD is the target condition, "chronic kidney disease", ".any" denotes that they have acquired that condition at any time, ".isbefore.ckd" means they had that condition before their first diagnosis of CKD.  The other abbreviations correspond with other conditions identified by ICD-9CM code groupings.  This grouping occurs in SQL during the import process. Each variable, with the exception of patient_age, is binary.

I've never worked with medical data, but from general reasoning I'd say that relations between variables in healthcare are pretty complicated. Different models, such as random forests, regression, etc. could capture only part of relations and ignore others. In such circumstances it makes sense to use general statistical exploration and modelling.

For example, the very first thing I would do is finding out correlations between possible precursor conditions and diagnoses. E.g. in what percent of cases chronic kidney disease was preceded by long flu? If it is high, it doesn't always mean causality, but gives pretty good food for thought and helps to better understand relations between different conditions.

Another important step is data visualisation. Does CKD happens in males more often than in females? What about their place of residence? What is distribution of CKD cases by age? It's hard to grasp large dataset as a set of numbers, plotting them out makes it much easier.

When you have an idea of what's going on, perform hypothesis testing to check your assumption. If you reject null hypothesis (basic assumption) in favour of alternative one, congratulations, you've made "something real".

Finally, when you have a good understanding of your data, try to create complete model. It may be something general like PGM (e.g. manually-crafted Bayesian network), or something more specific like linear regression or SVM, or anything. But in any way you will already know how this model corresponds to your data and how you can measure its efficiency.

As a good starting resource for learning statistical approach I would recommend Intro to Statistics course by Sebastian Thrun. While it's pretty basic and doesn't include advanced topics, it describes most important concepts and gives systematic understanding of probability theory and statistics.


What are advantages or disadvantages of training deep learning model from scratch? I want to know advantages and disadvantages.Also, What are advantages and disadvantages of transfer leraning?

(Suggestions and edits will be appreciated)

let us discuss advantages of training a deep learning model from scratch:

Building and training NN from scratch is of a great use in the research field.
You will know your model to the most basics and can modify it in case needed as per the requirements.
It will be more efficient in terms of size and training time(because there may be less number of layers as compared to any pretrained model trained for more than one purpose).Best example is VGG16.In case of VGG16 it is trained for 1000 different categories and even if we want only 2 categories we have to train the whole model,because we can't change the layers without deep knowledge about the architecture of VGG16.
For any beginner(like me) it will be beneficial to start building from scratch and the use the pre-trained model.
Don't require mastery in Deep Learning to use pretrained models.

Disadvantages:

Many pre-trained models are trained for less or mode different purposes,so may not be suitable in some cases.
It will take large amount of resources(time and computation power) to train big models from scratch.


When to stop showing content on recommendation engines? Let's take an example. I log into my Netflix account and see that it's suggesting the show Friends to me. But I have no interest in watching Friends. So I ignore it. The next time I login, it suggests Friends again and I ignore again. This goes on for quite a few number of logins until Netflix finds content that is probably more relevant to me than Friends.

My question is this: Is there a way to use the information that I ignored Friends for say 5 consecutive logins and inject some bias into the recommendation algorithm such that after 5 impressions and no clicks, it would stop recommending that content to me?

Note. This is a general question about recommendation engines and the use of Netflix and Friends is just to put things into perspective.

Interesting question, but I would say this really depends on the recommendation system itself and the purpose of your goal. Let's think it through:

Indeed we could collect the number of logins and create a relative feature of times ignored until chosen. This allows us also to inspect the amount of time a user ignores a show. On the first look, this makes perfect sense to me. However, this will only work if the user decides each time for a new show, from my own experiance I can say this is not valid for all user :D!

Therefore we would need to include also the probability or the fact of the user choosing a new show for given login. To keep things simple let's say we collect only those logins in which the user is deciding for a new tv-show or movie. Then our new feature would make way more sense and is a bit closer to what we can expect from reality.

But there is more. As far as I know (please correct if I'm wrong), Netflix has a sort of prediction or classification on what is the best picture to display for a certain user on each tv-show. In case the pictures change over time, our login feature from above is less useful. Maybe the image made the trick to convince the user of a tv-show or movie - honestly I don't have a clue how to measure this straight away, except maybe with another feature or collected value if the image was different than the last time.

So far so good. Let's say we collect the following:

selected/not selected for each movie or tv-show recommended in a
session the user picked a new tv show or movie
picture changed for each tv show or movie recommended after the last
time the user picked a newly recommended item.
And maybe to improve our picture algorithm also what kind of picture
could have had an impact on the user's decision

Sounds like a plan. But the more I think about it the more comes to my mind, that often my wife is watching on the same Netflix account and our tastes are pretty different for tv-shows or movies we watch in absence of the other. Now, one could say: "yes buddy, this is why netflix offers you profiles". But honestly, I'm not into watching tv that much, so I usually don't care about the profile I'm using. Hence I won't be the only one who shares his profile with a person with another preference, this could have serious influence on our recommendation as well. The profile will be mixed and the precision of our recommendation system will probably decrease.

I'm not sure if this is valid for the mass of Netflix users. We don't have any numbers, so I would assume this is valid for more or less than 30% (just guessing). In this case, you could try to track active user behavior. Like on what kind of recommendations does the user stops for how long, on which tv-shows is he interested in how many seasons it has, how often did the user hover over a movie or show?

The more we think about it the more edge cases and conditions come to my mind. Since I personally don't have a lot of experience with such advanced recommendation systems I would assume that from a technical point of view it would make sense to split the system into two parts. One which gives us a baseline of a larger amount of predictions and one which adjusts the recommendation live, while the user is behaving on the open application to adapt current data and to decide which profile is relevant and applicable for the real person behind the user.

This is really interesting. I'm curious what others (probably with more experience) thing about these questions and about what I provided as an idea.


Why seq2seq models are superior to simple LSTMs? It is common knowledge in the field of Deep Learning that the most powerful Recurrent architecture is the sequence-to-sequence, or seq2seq, for pretty much any task (to time series forecasts, to machine translation, to text generation).

Why? What are the underlying mathematical reasons for which an LSTM encoder-decoder architecture would outperform more canonical RNNs? Is it in the generation of dense latent representations? Is it about the comparatively higher number of parameters? Any hint is appreciated.

Encoder-decoder architectures are not simply "more powerful" than a mere LSTM.
LSTMs can't be used (in their standard configuration) for general sequence transduction tasks. On the other hand, encoder-decoder architectures are conditional autoregressive models, that is, they generate a sequence element by element conditioning on another sequence. This difference is what justifies the rather different use cases for simple LSTMs and encoder-decoder architectures.

Among LSTM-based encoder-decoder architectures, we should distinguish different types by the way information is passed from the encoder to the decoder. The simplest form is to simply pass the last hidden state of the encoder LSTM to the first decoder LSTM; this implies that all the information from the input sequence is "compressed" into a fixed-length vector, which is known to be an information bottleneck. More complex forms include using attention mechanisms, where the encoder hidden states at each time step are combined into a different previous context vector for each decoder LSTM; here there is no bottleneck, and normally their results are much better.


What Unsupervised Machine Learning algorithms can be used to detect Fraud in insurance? I'm about to start a project regarding Fraud Detection in Insurance but there is no dependant variable to train the model and classify it.

Please provide the algorithms that I can use to detect Fraud in both Auto and Health insurance industry.

Thanks in advance!

Fraud, by definition, is a supervised concept.

No unsupervised algorithm will reliably detect this. At best you'll get some suspicious cases with unsupervised approaches such as anomaly detection.


Training CNN for Regression Background: I am using CNN to predict forces acting on a circular particle in a granular medium. Based on the magnitude of the forces, particle exhibits different patterns on its surface. The images are greyscale, 64-by-64 pixels. You can see different pictures with the magnitude of the corresponding force on the x-axis attached below.

My attempt at solution: I am relatively new to deep learning and data science and decided to use a simple conv net to run a regression. My code is provided below. I tried to fit the model using adam optimizer and mse as a loss function, but it takes forever and sometimes aborts execution by itself. What could be the problem? I am running it on a PC with 8GB RAM, 1TB SSD, Intel i7 CPU, and GTX 1080 GPU.

def build_model():

model = Sequential()
model.add(Conv2D(64, kernel_size=(3, 3), strides = (1,1),
             padding = 'valid', activation='relu',
             input_shape=input_shape))

model.add(Conv2D(64, kernel_size = (3, 3), strides = (1,1),
                 padding = 'valid', activation='relu'))

model.add(MaxPooling2D(pool_size=(2, 2), strides = (2,2)))

model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(53824, activation='relu'))
model.add(Dense(53824, activation='relu'))
model.add(Dense(53824, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(1, activation='linear'))

return model

Although building neural network models is admittedly still an art rather than a science, there are some (unwritten) rules, at least for initial approaches to a problem, such as yours here (I guess).

One of them is that dense layers with 50,000 nodes are too large, and AFAIK I have never seen such large layers in practice; multiply this x3 (layers), and no wonder your code takes forever.

I would certainly suggest to experiment with a dense layer size between 100 - 1000, and even start with less than 3 dense layers. Reducing your 1st CNN layer to 32 is also certainly an option.


Detecting text region from an image So I'm working on a document processing AI and I already have a character recognition model which performs decently well. Now the problem is, how do I feed each character to the model in order to make predictions. Sliding window is one technique to segment arbitrary sequences of the image but it might not work in many cases as the words may get cut off. Can anyone suggest a robust way to detect text regions in an image? I've been researching about MSER but they claim to work well on grayscale images and not RGB. I know that we can convert RGB to grayscale but that is not performant enough.

You can use OpenCV's Scene Text Detection https://docs.opencv.org/3.3.1/d4/d61/group__text.html


How to down-weight non-words in text classification? Background:

Documents coming in as well as training set have gone through Apache Tika with Tesseract for inline images.  This works well, except when it doesn't.  Many docs are old, scanned images and what Tika extracts is gibberish.

Using Spark on Hadoop and either ML or MLlib (haven't settled, though I like ML better).

So far getting best results from a pipeline using Naive Bayes that removes Stopwords, tokenizes and Countvectorizes features (no Tf-Idf).  Total bag-of-words approach.  Next best is using ML to tokenize, Tf with Idf and into LogisticregressionwithLBFGS.

Anyway, the thought occurred to me that the model uses many docs that are junk.  Literally just strings of gibberish like "mmmmmmmm aaannnammmmrrr hdhhdhhhhhjjj..."

This isn't good, but since I'm operating at scale it's just what happened.  Certainly I could pick through 10,000 training docs and remove the bad examples, but there has to be an easier way.  Is there?

The title of this question belies my brainstorm that there might be a way to discount, downweight or outright ignore tokens that aren't discernable by a dictionary.  Is there?

Open to any and all advice or approaches to get better precision out of this model.

Thanks

Usually, those non-sensical words are not problematic because they appear in one or two documents, and people usually just filter words with such low frequencies.

Are you not able to do this? Or do your non-sensical words appear that much?

Anyhow, your suggestion is what people use:

  outright ignore tokens that aren't discernable by a dictionary. Is there?

Exactly. There is no magic way to know if a word is English or not. What word processors do is to use a dictionary, as you yourself suggested.

In python, before stemming, you could filter based on pyenchant.

import enchant
d = enchant.Dict("en_US")
d.check('hello')  # True
d.check('mmmmmmmm')  # False

I bet this would be good enough.

But, if there are a lot of false negatives, you could ask the dictionary for the most similar words and then apply a word distance measure. Here I will use Levenshtein distance  as implemented in python:

>>> r = d.suggest('helo')
>>> r
['hole', 'help', 'helot', 'hello', 'halo', 'hero', 'hell', 'held', 'helm', 'he lo', 'he-lo', 'heel', 'loathe', 'Helios', 'helicon']
>>> min([LD('helo', w) for w in r])
1

The distance will be higher for the non-sensical words you suggested:

>>> d.suggest('mmmmmmmm')
['mammogram', "Mammon's"]
>>> min([LD('mmmmmmmm', w) for w in d.suggest('mmmmmmmm')])
5
>>> min([LD('aaannnammmmrrr', w) for w in d.suggest('aaannnammmmrrr')])
12
>>> min([LD('hdhhdhhhhhjjj', w) for w in d.suggest('hdhhdhhhhhjjj')])
10

To recap:

filter words that appear in less than 3 documents
if that's not enough or it is not a possibility for your use-case, use a dictionary

I think this last suggestion of mine of using Levenshtein distance to the closest English words is overkill.

If your documents involve a lot of urban words or hip terms, you could also filter based on the number of Google results, instead of using a common dictionary. Your word hdhhdhhhhhjjj, for instance, gives me only two Google results, one of which is this stackexchange. Data science is about being creative. :)

There probably are heuristics or statistics based on the number of consonants in a word, and letter combinations, which you could use to probabilistic shrink your documents' dictionary, but I wouldn't go there. It will be too much work and very brittle.


PCA - Error minimization and Variance Maximization I'm studying the PCA algorithm and the theory behind it. I think I understood how does it work and the idea of dimension reduction of the data in order to find a new feature (component) that maximizes the variance of the data and minimizes the error.

My question is : in this algorithm , are the maximum variance and the minimum error reached at the same moment ?

In this example the magenta/black line is the solution of my PCA. So I find the 1 dimension vector that reduces my 2 dimensions dataset. I found this vector because the error (length of the red lines) is minimized and the variance (distance among the red projected points) is maximized.

So if I need to apply this algorithm , if I use either only the avg error minimization , will I reach the same result of the case in which I use the variance maximization principle ?

Thanks.

Yes, the two formulations where you either capture the maximum amount of variance of the data while reducing the dimension or where you try to minimize the distance of the data from the selected subspace (i.e. reconstruction error) are equivalent.


How to interpret statsmodel output - logit? I ran a logit model using statsmodel api available in Python. I have few questions on how to make sense of these

1) What's the difference between summary and summary2 output?

2) Why is the AIC and BIC score in the range of 2k-3k? I read online that lower values of AIC and BIC indicates good model. Is my model doing good? Is there any optimal range for AIC and BIC?

3) As you can see covariance Type is non-robust. What is it and should I be concerned about it?

4) Is there any other field/item in the output that I should pay attention to?

5) You can see below how certain significant variables like X2,X8,X45 have very low coefficients. How can they be significant and still have very low or near to zero coefficient? Is it normal?

This is the output that I got

Summary output

Dep. Variable:  vae_flag        No. Observations:   3298
Model:  Logit                   Df Residuals:   3241
Method: MLE                     Df Model:   56
Date:   Mon, 30 Dec 2019        Pseudo R-squ.:  0.3347
Time:   21:18:36                Log-Likelihood: -1392.2
converged:  True                LL-Null:    -2092.7
Covariance Type: nonrobust      LLR p-value:    3.894e-256

Summary2 output

Model:  Logit                   Pseudo R-squared:   0.335
Dependent Variable: op_flag     AIC:    2898.4259
Date:   2019-12-30 21:18        BIC:    3246.1870
No. Observations:   3298        Log-Likelihood: -1392.2
Df Model:   56                  LL-Null:    -2092.7
Df Residuals:   3241            LLR p-value:    3.8937e-256
Converged:  1.0000              Scale:  1.0000
No. Iterations: 7.0000

Significant variables

      coef  std err   z     P>|z|   [0.025  0.975]
x2   0.0321 0.060   11.227  0.000   0.558    0.794
x6   2.2996 0.095   24.332  0.000   2.114    2.485
x7  -1.8795 0.082   -22.835 0.000   -2.041  -1.718
x8   0.0002 0.058   2.116   0.034   0.009    0.237
x16  0.2693 0.059   4.564   0.000   0.154    0.385
x33 -0.3138 0.139   -2.254  0.024   -0.587  -0.041
x34  0.4644 0.137   3.392   0.001   0.196    0.733
x45  0.0088 0.052   2.306   0.021   0.018    0.221
x52 -0.1755 0.087   -2.007  0.045   -0.347  -0.004
x55 -0.0982 0.050   -1.965  0.049   -0.196  -0.000

Very brief answer (no way to go into details here):

Aparently the two calls produce different tables containing slightly different statistics.
AIC and BIC compare nested models. So if you have some model and you add or remove some variables (for instance), you may compare AIC, BIC. There is no universal "okay" range in terms of overall figures. Even with a low(er) AIC, BIC, you can have a "bad" model. So AIC, BIC really is about comparing "similar" models against each other.
There are robust standard errors, which are computed in a different way than "normal" standard errors. I think this indicates that "normal" standard errors are calculated.
Not really. The pseudo R-squ. may give you some idea about model fit, but it is a little different from "normal" R-squared and I don't find it too useful.
The value of the coefficient is not directly related to significance (small values can be significant and vice versa). Significance is calculated from standard errors/t-/z-statistics. Note that your coefficients are log-odds (NOT marginal effects). In case you want to obtain marginal effects, you need to look for some package (like "margins" in R/Stata) or you do this by hand.

Overall I recommend to have a good read about logistic regression since you seem to be uncertain about basic concepts.


Which machine learning algorithm to choose? I want to choose an unsupervised algorithm which learns to predict $n$ outputs from the data, for eg. 4 coordinates (pixels) in an image. What algorithm should I choose? I think it's a 2-class classification to divide the set of points in the image as belonging to output (1) or not (0), maybe logistic regression to give the probabilities of a point being the output point. But I am confused because classification algorithms are part of supervised algorithms where we have labelled data. Should I use clustering to find 2 groups of points that can be output or not? Maybe anomaly detection to find the 4 odd points out?

If you want to use an unsupervised method i.e if your data is not labelled with classes, then something like k-means clustering may be your best bet to find patterns in the data.

Alternatively, if you want to do anomaly detection, 2 possible options are

Assuming your data is normally distributed, you could fit a Gaussian to your data and calculate the Probability Density Function (PDF). Once you have the PDF, you can set a threshold probability, below which a data point could be classified as an anomaly
If you have enough data, use a Variational Autoencoder neural network. Very roughly speaking, you train this on all the data you deem to be “normal” (the neural network learns how to reconstruct the input data in the output), and then when anomalous cases are passed to the network, it can’t reconstruct it. If the network can’t reconstruct it accurately, the data is an anomaly.


Decision boundary for KMeans clustering algorithm Below code is used for plotting decision boundary for KMeans clustering. I understand contour in general, my question is, how does plot (contour method) know where to draw decision boundary based on predicted Z values, without knowing X and Y coordinates?
 xx, yy = np.meshgrid(np.linspace(mins[0], maxs[0], resolution),
                         np.linspace(mins[1], maxs[1], resolution))
 Z = clusterer.predict(np.c_[xx.ravel(), yy.ravel()])
 Z = Z.reshape(xx.shape)

 plt.contourf(Z, extent=(mins[0], maxs[0], mins[1], maxs[1]),
                cmap="Pastel2")
 plt.contour(Z, extent=(mins[0], maxs[0], mins[1], maxs[1]),
                linewidths=1, colors='k')

Can you post a picture of the plot here and maybe some reproducible code? My understanding with the info you've given is that the contour plot is not actually plotting a decision boundary. In fact, I think its not really helpful to think about k-means as producing a decision boundary. At the end of iteration you have K centroids and 'classify' points based on which centroid it is closest to.


How to treat the undefined values which make sense? I'm currently trying to create a few features to improve the performances of a model. One of those features that I would like to create corresponds to the difference in days between a customer's purcharse and his last one. To create this feature is not a problem. However, I don't know which value to set if this is the first purcharse of a customer. Which value should I set and, more generally, how to treat these cases ?

   customer_id date_purchase  diff_last_purchase  first_purchase
0            1    2018.02.12                 NaN               1
1            1    2018.02.18                 6.0               0
2            2    2018.02.25                 NaN               1
3            3    2018.03.15                 NaN               1
4            3    2018.03.18                 3.0               0

Recently had a discussion on the same topic at work. It boiled down to encoding missing values as impossible values (negatives, very high) or as information inferred from the dataset (mean, median). Some more sophisticated methods use models build on rest of data (non-missing columns) to predict missing ones.

If using tree-based approach setting it to -1 should be fine as a start, as as there might be observations where previous purchase was in same day (0).

With mean-trending models (linear regression), setting it to the mean might be also fine, but you need to calculate the mean on the train set only and propagate it to the test set separately.


Does not using more filters in deeper CNN creates more images? For example, we have applied 32 filters to a single image. Then it created 32 different images (stack of convolutional values).
And in the second layer, if we apply 64 filters, are all these filters going to be applied on all those 32 images? If so, then it will create 64*32 numbers of output or  I am understanding wrong?
I have become confused because when I studied keras documentation, it says that using 64 filters it will create 64 outputs. If anybody enlights me on how the second or deeper layer works in CNN briefly it will be helpful for me.

No, your understanding is not correct.
Each of the 64 filters of the second layer will be applied to each of the 32 channels from the output of the first layer, resulting in 64 channels in the output of the second layer.
When the input of a convolutional layer has multiple channels, the convolution filter itself has the same number of channels. In your example, if we are using $3\times3$ filters, each filter in the second layer will be a tensor of dimensions $3\times3\times32$. Therefore, the filter "covers" the full depth of the input. Then, you simply perform the element-wise multiplication of the filter with the overlapping region in the input and add all the resulting elements together. Applying just 1 filter, we obtain a result with 1 channel.
This way, the number of channels of the output of a convolutional layer is the same as the number of filters in the convolution.


Why is the cosine distance used to measure the similatiry between word embeddings? While computing the similarity between the words, cosine similarity or distance is computed on word vectors. Why aren't other distance metrics such as Euclidean distance suitable for this task.
Let us consider 2 vectors a and b. Where, a = [-1,2,-3] and b = [-3,6,-9], here b = 3*a, i.e, both the vectors have same direction but different magnitude. The cosine similarity between a and b is 1, indicating they are identical. While the euclidean distance between a and b is 7.48.
Does this mean the magnitude of the vectors is irrelevant for computing the similarity in the word vectors?

You're asking two questions here.

Does this mean the magnitude of the vectors is irrelevant?

Yes. Cosine distance is $ D_{cos} =  \frac{A \cdot B}{\|A\|\|B\|} $, which just comes from the definition of inner product, $A \cdot B = \|A\|\|B\|\cos\theta$.

Why is the cosine distance used? Or, to think of it another way, why is the answer to (1) a desirable property to have in a distance metric?

In a word embedding, we choose a dimensionality $d$ for the embedding. This is the number of components in our embedding space. The components (or, linear combinations of the components) are meant to encode some kind of semantic meaning. Classic examples are like that the vector for "queen" plus the vector for "man" should be near the vector for "king". That sort of thing. There's a direction that roughly corresponds to "royalty" and a direction for gender.
Look at your example, where $b = 3a$, $a=[-1,2,-3], b=[-3,6,-9]$. This is a perfect illustration of why we use cosine similarity. They have very different magnitudes but point in the same direction. They have cosine distance 1, and we want that because it means that they have the same relative proportion of each component.
If we use euclidean distance, $a$ and $b$ are $\sim7.48$ units apart.  It would be easy to find another vector $c$ that is around the same distance from $a$ as $b$ is, in a completely different direction. If our space has been learned properly, $c$ should have completely different semantic meaning from $b$, but they're both the same distance from $a$. The euclidean distance doesn't measure the similarity that we want very well.


How can I do classification with categorical data which is not fixed? I have a classification problem with both categorical and numerical data. The problem I'm facing is that my categorical data is not fixed, that means that the new candidate whose label I want to predict may have a new category which was not observed beforehand.

For example, if my categorical data was sex, the only possible labels would be female, male and other, no matter what. However, my categorical variable is city so it could happen that the person I am trying to predict has a new city that my classifier has never seen.

I am wondering if there is a way to do the classification in these terms or if I should do the training again considering this new categorical data.

It is very good question; in fact this problem has been around for a while and I have not yet found the perfect solution. Yet more than happy to share my experience:

Avoid one-hot-encode as much as possible (contrary to what was suggested above). The reasoning is that it won't work. A model with one-hot-encode features only works when all those sublevels had been existed in the training data. The model won't be able to do prediction, unless somehow it is manually tweaked. If you search you will find many people ran into this issue when splitting their data into train/test, and faced the issue of some sublevels of a particular feature were not present in the training split and subsequently failed to do prediction on the test. Said aside, if you have very high cardinal features (maybe like your city with let's say 200 city names), this will increase the dimensionality of your data unnecessarily! If for some reasons you may need to do one-hot-encoding, just keep these in mind.
Use Other Encoding Methods. Maybe try learning more about other methods that are robust to this issue, at least for the time being like target-based coding, hashing (see some references below). If you are with Python there is a nice package offer mant encoding options. You may be surprised to see that other simple methods often works just fine.
Retrain your model. Theoretically when learning your train/set should have had the same distribution (mostly this is thought of as target distribution, but can be true about variables as well). Now with new items comes into play, your test (unseen) data distribution has changed. Then it is best to retrain the model again so that those new cities will be accounted for.
Put Newly Added Subcategories (and least frequent ones) to Others. While earlier point is true theoretically, it is very likely that the test distribution (of that particular category) won't change that drastically in most of the cases e.g. very few items goes top of the categories in the training set. Perhaps like in your case, you may have 100 cities in the city feature, and just very few new ones comes over time. What I would consider would be looking at the let's X-quantile of that particular category, and put the least frequent ones into Others subcategory. Assuming your newly added data point is only little, it will very much goes into the Others group. You will certainly lose a level of granularity by doing this one, but once again the point of learning is that not only your model learn the training data, but most importantly to be able to generalize very well on unseen data and if those new added categories are very data points, grouping them altogether into the Others group won't hurt.
Other Recent Not-Yet Mature Solutions like Cat2Vec (borrowed from Word2Vec from NLP) or Similarity Encoding. These are very recent, check the paper for the former and its github and an example (based on Word2Vec) in Kaggle, and this paper for the latter and its implementation. The idea of the former is to convert categories to vectors. As much as I have to say it really make sense to work, but I have no experience using it. The latter, so-called dirty_cat, looks quite promising and easy to use. Whether it is robust to having unseen cardinal category in your test data is not clear to me!

P.S.: I would like to add that the idea of city to a geographical location given in the first comment is really nice and it is actually not complicated as they are many Python API e.g. by Google or HERE that allows you to do that. But it is noted that this is just a way to engineer new features and certainly not to be replaced by the city feature itself.

Interesting references to check first, second, third, fourth (no particular order!)

All above-mentioned points are practical solutions rather concretely theoretically correct, and surely subject to further discussions. And I am more than happy to learn more.


Lack of activation function in output layer at regression? I supposed that the output layer should have certain kind of activation function (preferably linear or tanh) for regression, but I recently read that in case of regression this is not necessary. This seems me reasonable, after all, the node(s) of the output layer produce(s) numeric values themselves. Which solution is better: with or without activation func?

Activation "linear" is identical to "no activation function". The term "linear output layer" also means precisely "the last layer has no activation function". Whether you use one or the other term might be down to how your NN library implements it. You may also see it described either way around in documents, but it is exactly the same thing mathematically:

$$a^{out}_j = b^{out}_j + \sum_{i=1}^{N^{hidden}} W_{ij}a^{hidden}_i$$

Where $a$ values are activation, $b$ are biases, $W$ is weight matrix.

For a regression problem with a mean squared error objective, this is the most common approach.

There is nothing stopping you using other activation functions. They might help you if they match the target variable distribution. About the only rule is that your network output should be able to cover possible values of the target variable. So if the target variable is always between -1.0 and 1.0, with higher density around 0.0, perhaps tanh could also work for you.


How fbprophet cross validation works I am facing some issues to understand how cross_validation function works in fbprophet packages.

I have a time series of 68 days (only business days) grouped by 15min and a certain metric :

00:00 5
00:15 2
00:30 10
etc
23:45 26

And I really don’t know how to set up my cross_validation function.

I have understand that initial parameter is the training data, but what are period and horizon ?

I have 6528 rows, I want to forecast the next day so my period is 96 ? Because one day = 96 step of 15 min.

Please can you explain me how I have to fill period and horizon parameters. The official documentation of Facebook Prophet is not very understandable.

Thanks a lot.

Think about how you would verify the model on your own. You could train it on data in the past, stopping before the present, and then ask the model to predict for a period that you already had data for. That way you can check how accurate it is (either by eye or with error metrics such as MAE, MAPE, RMSE, etc), and adjust accordingly.

cross_validation just automates this process. The first parameter you give is your trained model m (not the data). You then also give the prediction horizon - how frequently you want to predict (in your case '15min', assuming Python). You may then give an initial (how long to train before starting the tests) and a period (how frequently to stop and do a prediction). If you don't give them, Prophet will assign defaults of initial = 3 * horizon, and cutoffs every half a horizon. You then have a long running series of validations, each time predicting forward and calculating the error (you can use the other fbprophet.diagnostics tool, performance_metrics for this). This is somewhat akin to k-fold cross-validation in non-time-series machine learning.

The documentation on this is actually pretty good. See also this blog post for an example in Python.


Pytorch: understanding the purpose of each argument in the forward function of nn.TransformerDecoder According to https://pytorch.org/docs/stable/generated/torch.nn.TransformerDecoder.html, the forward function of nn.TransformerDecoder contemplates the following arguments:

tgt – the sequence to the decoder (required).
memory – the sequence from the last layer of the encoder (required).
tgt_mask – the mask for the tgt sequence (optional).
memory_mask – the mask for the memory sequence (optional).
tgt_key_padding_mask – the mask for the tgt keys per batch (optional).
memory_key_padding_mask – the mask for the memory keys per batch (optional).

Unfortunately, Pytorch's official documentation on the function isn't exactly very thorough at this point (April 2021), in terms of the expected dimensions of each tensor and when it does or doesn't make sense to use each of the optional arguments.
For example, in previous conversations it was explained to me that tgt_mask is usually a square matrix used for self attention masking to prevent future tokens from leaking into the prediction of past tokens. Similarly, tgt_key_padding_mask is used for masking padding tokens (which happens when you pad a batch of sequences of different lengths so that they can fit into a single tensor). In light of this, it makes total sense to use tgt_mask in the decoder, but I wouldn't be so sure about tgt_key_padding_mask. What would be the point of masking target padding tokens? Isn't it enough to simply ignore the predictions associated to padding tokens during training (say, you could do something like nn.CrossEntropyLoss(ignore_index=PADDING_INDEX) and that's it)?
More generally, and considering that the current documentation is not as thorough as one would like it to be, I would like to know what the purpose is of each argument of nn.TransformerDecoder's forward function, when it makes sense to use each of the optional arguments, and if there are nuances in the usage one should keep in mind when switching between training and inference modes.

About the need for tgt_key_padding_mask
While padding is usually applied after the normal tokens (i.e. right padding), it is perfectly fine to apply it before normal tokens (i.e. left padding). For instance, fairseq supports parameter left_pad to specify precisely this.
For left padding to be handled correctly, you must mask the padding tokens, because the self-attention mask would not prevent the hidden states of the normal token positions to depend on the padding positions.
About the meaning of each argument
I think the best documentation is in MultiHeadAttention.forward:

key_padding_mask – if provided, specified padding elements in the key will be ignored by the attention. When given a binary mask and a value is True, the corresponding value on the attention layer will be ignored. When given a byte mask and a value is non-zero, the corresponding value on the attention layer will be ignored
attn_mask – 2D or 3D mask that prevents attention to certain positions. A 2D mask will be broadcasted for all the batches while a 3D mask allows to specify a different mask for the entries of each batch.

With that information and knowing where keys, values and queries come from in each multi-head attention block, it should be clear the purpose of each parameter in nn.TransformerDecoder.forward. Also, the documentation of MultiheadAttention.forward contains info about the expected shapes.