Gradient computation I am beginner in data-science. I am trying to understand this PyTorch code for gradient computation using custom autograd function:

class MyReLU(torch.autograd.Function):

@staticmethod
def forward(ctx, x):

    ctx.save_for_backward(x)
    return x.clamp(min=0)

def backward(ctx, grad_output):

    x, = ctx.saved_tensors
    grad_x = grad_output.clone()
    grad_x[x < 0] = 0
    return grad_x

However, I don't understand this line : grad_x[x < 0] = 0. Can anyone explain this part?

The example you find is calculating the gradient for the ReLU function, whose gradient is
$$\text{ReLU}'(x)=\left\{
\begin{array}{c l}
     1 & \text{if } x>0\\
     0 & \text{if } x<0
\end{array}\right.$$

Therefore when x<0, you make the gradient 0 by grad_x[x < 0] = 0.


Output trained parameters of Keras model If I am training on a GRU model, is there a way I can output the learnt parameters so that when I train next time with more data, I can initialize with those learnt parameters as a starting point?

There is a way to do this. See the documentation! It is very good.

I assume you mean that you build a network using Keras (which contains recurrent GRU layers) and would like to save the model after some training, then restart from the same point e.g. with new data or just to push the model further.

Example

Assuming you want to analyse some images of shape (150, 150, 3):

from keras.models import Model, load_model
from keras.layers import Input, GRU, Dense

my_model = Model()

my_input = Input(shape=(150, 150, 3))    # using Tensorflow's dims: 'channels last'
gru_layer - GRU()(input)
output = Dense(32)(gru_layer)

# tell the Model class which layers are the start and end of the model
model = Model(inputs=my_input, outputs=my_output)

# save it!
my_model.save('my_model.h5')

# some time later.... a new python session
my_reloaded_model = load_model('my_model.h5')

# continue using the model

Explanation

At the point the model is built, we can compile and train it the usual way e.g. my_model.fit(...)

Once training is finished, we can save the model two different ways:

just the model weights, using my_model.save_weights('my_weights.h5')
the whole model and its meta data, using my_model.save('my_model.h5')

In either case, you will need the library h5py (see the documentation)

Option 2 preserves more information from the model. Quoting the documentation, it saves:

  the architecture of the model, allowing to re-create the model
  the weights of the model
  the training configuration (loss, optimizer)
  the state of the optimizer, allowing to resume training exactly where you left off.

To continue on the with model where you ended and saved, it is as simple as:

my_model = keras.models.load_model('my_models.h5')

Now you can train it further or introspect the model and so on.

Links

Details on the Model class
Details of GRU
Details of saving models


RNNs for time series prediction - what configurations would make sense My question here is mostly about general-intuition logic: when using a RNN (LSTM) for predicting a time series, and you have the goal of, for example, predicting at 100 steps ahead a series of one single feature, what kinds of configurations would make sense for the layer sizes of a simple input | hidden | output RNN, and of the window size (assuming you want to look at more than one point at one so you pick a "window" / "interval"?

More precisely:

use only 1 input neuron and look at one point at a time, or pick a window/interval? - obviously that by looking only at 1 point, you lean entirely on the memory aspect of the network for detecting useful patterns... would this be a good thing?
when using a window, does it make sense to use a set of overlapping-windows, eg. to slide a window of 10 points one point at a time, or non-overlapping windows?
should there number of units in the hidden layer roughly equal the number of steps you aim to predict ahead? (also, similar thing with the relationship between window size and steps to predict ahead)

Or, if answering this is too hard  or time consuming, where could one get the intuitions for answering such questions?

(Because obviously the space of possible configurations is huge, and unless you have a ton of time or resource, you need some intuitions so you can start form a configuration that "roughly works"...)

should there number of units in the hidden layer roughly equal the number of steps you aim to predict ahead? (also, similar thing with the relationship between window size and steps to predict ahead)

Don't forget that a single neuron can saturate, over time - it can assume values closer and closer to -1 or to 1. That means you get nice flexibility, across several consecutive timesteps. Therefore, you don't need number of neurons to be as many as the number of timesteps.

You can verify it by plotting a single neuron of a simple RNN on the graph-paper, with pen and paper. For every square (timestep) draw neuron's activation and previous activation. At some particular timestep assume the input is no longer 0 (for example -1) and observe how the function fluctuates into the wave from that timestep onwards. Pretty flexible for a single neuron to be honest. Even more entangling were we to graph 2 neurons. That wave of only a few neurons can hold and encode a lot of information (exponentially more)

  when using a window, does it make sense to use a set of overlapping-windows, eg. to slide a window of 10 points one point at a time, or non-overlapping windows?
  This is similar to another question:
  sliding window leads to overfitting in LSTM?

Overall, sliding window is a bad approach for LSTM that leads to overfitting. A better approach would be to scan sequence backwards and forwards before outputting an answer. This is Bidirectional LSTM, or even looking into Attention. If you need to output an answer every frame relying on a neuron as is would be reasonable - after all LSTMs are quite good at remembering things due to the "accumulation" and the entangling which allows for oscillations (I've mentioned them above).

  you need some intuitions so you can start form a configuration that "roughly works"...)

Quoting a pretty much legendary post:

  in sum, for most problems, one could probably get decent performance (even without a second optimization step) by setting the hidden layer configuration using just two rules: (i) number of hidden layers equals one; and (ii) the number of neurons in that layer is the mean of the neurons in the input and output layers.

Also a bonus, how to query LSTM might be useful for future thought - LSTM querying approach


Is it vital to use weighted metrics with weighted loss functions? When one has an imbalanced machine learning problem, let's say a binary classification problem where class 0 is a majority class and class 1 is a minority class. Here class 1 is the most important class for you. Now your dataset is composed of 100 samples: 70 samples from class 0 and 30 samples from class 1.
In this case, usually, one defines and uses a weighted binary cross-entropy loss function to calculate the loss.
classweights = torch.FloarTensor([1.000, 1.300])
weighted_loss = torch.nn.BCELoss(weight = classweights)

For calculation of metrics like F1-score, is it required that these weights are used?
sample_classweights = sklearn.utils.class_weight.compute_sample_weight(class_weight, y_true)
classification_report = sklearn.metrics.classification_report(y_true, y_pred, sample_weight=sample_classweights)

Any hint or help is appreciated.

Calculation of loss and the calculation of metrics on the test set are two different entities. Usually, the weighted loss function is used to weight one of the class (you can use higher weights to the important class in balance or unbalanced class distribution).
For metrics like F1, its always safer to use multiple measures in unbalance class distribution. It would be good to keep a check on macro as well as weighted-F1.
For example: a confusion matrix of the following order:
predicted labels:     0   1
true label 0:      | 70 | 0 |
true label 1:      | 30 | 0 |

Has macro-F1: 0.41 and weighted-F1: 0.58
Further, Aconfusion matrix of the following order:
predicted labels:     0   1
true label 0:      | 99 | 0 |
true label 1:      | 1  | 0 |

Has macro-F1: 0.50 and weighted-F1: 0.99
Both these cases are unbalance where 70 and 90 samples belong to class 0 in 1st and 2nd case respectively. However, when the dataset is too unbalanced, and model predictions are biased towards majority class, then weighted F1 becomes higher.


Identifying importance of each feature in deep model I have a deep model and I want to figure out which feature has the maximum influence on predicted result. For this I train the model with all the features I think are important, during prediction I set all features =0 one by one keeping the rest unchanged so I could figure out which is the least important of all. On predicting the results of these tampered test set on the trained model I get the same(changes at 5th-6th decimal place) F1 score, Recall and Precision.

Can some one explain me where I am wrong?

This could mean that your features are heavily correlated. As a result, when you set one variable to 0, there is always another one that carries the same information.
Try to set several features to 0 to see if you have the same issue. If yes, then it might be something else (maybe your code).

You might also be interested in this paper, you'll find several methods to assess feature importance.


Training algorithm for finding relationships within a data set Given a dataset that looks like the following:

-----------------------------------------------------------
|Category|Phrase         |Related 1       |Related 2  |...|
-----------------------------------------------------------
|Art     |Art exhibitions|                |           |   |
|Art     |Art galleries  |Art exhibitions |Art museum |   |
|Art     |Art museum     |Art galleries   |           |   |
|Sports  |Football       |Sports coaching |NFL        |   |
|Sports  |Basketball     |NBA             |           |   |
|Sports  |Sports coaching|                |           |   |
-----------------------------------------------------------

If I have an input phrase, how can I list possible related phrases inside the dataset? For example, I would relate "art exhibitions" to "art museum" and "art galleries". The dataset has some relationships filled out already (I am not sure if this counts as usable training data or if I have to transform it further) and I am looking to fill in the blanks, that is find relationships for some rows.

Eventually we would want to be able to support input phrases that don't exist yet in this dataset and establish relationships for them based on historical info, which is why I think I should be looking at machine learning.

What is the best algorithm/approach I can make use of for this problem?

Given your example, if you want to group phrases based on terms, a simple approach would be to use grep to find rows that contain specific letter sequences (e.g. "art", "ball"). For example, say your table is a csv file (myfile.csv), where each line is a word or phrase (e.g. "Football", "Jazz music"). We can use a UNIX terminal to get all lines that contain "ball":

grep -n ball myfile.csv | cut -d: -f1 > myresults.txt

This would yield a file (myresults.txt) that containts the lines numbers of phrases that contain "ball". You can also apply grep to a data frame version of your table using R or Python, so you can loop over different letter sequences and save the found relationships to a separate column of the data frame or a separate data frame or vector.


What to do when seed has a big impact on model performance? I have a training procedure set up for an image recognition task. Each time I train a model, I record training loss, validation loss, validation precision and validation recall.
Recently I switched from EfficientNet to a ResNet-based model. Both models are pretrained, so weight initialization is deterministic. With the old model I ran 5 experiments (each on 5-folds) with exactly the same parameters, varying only the seed and got around 0.001 standard deviation for validation loss. With the new model I ran 3 experiments (also with 5 folds) with exactly the same parameters and got a standard deviation of 0.028 for validation loss.
This high variance makes it very hard to interpret the effects of changing non-seed parameters when running new experiments with the new model, since in many cases the differences in performance sit within one standard deviation. Experiments take multiple hours each, so it's not feasible to run multiple experiments for each condition.
What does one do in cases like this?

It's just a simple idea that I often recommend to understand what happens in cases like this:
You could try to do an ablation study, where you train the model with varying sizes of training data (e.g. 10%, 20%,..., 100% of your full training data). Given that your problem is instability, it would be even better to do every size several times with different random subsets. Then you plot the performance as a function of the data size, for example with boxplots in order to visualize the variance. The evolution of the performance (and its variance) would show you whether there is enough training data given the complexity of the model: if yes, the curve increases and then starts stabilizing around its maximum. If not, the curve is still in a phase of increase when it reaches 100%, meaning that more data would be needed to allow the model to stabilize.
Of course the disadvantage is that you need to run the training/testing many times, so this might not be practical.


Reinforcement learning: negative reward (punish) illegal actions? If you train an agent using reinforcement learning (with Q-function in this case), should you give a negative reward (punish) if the agent proposes illegal actions for the presented state?

I guess over time if you only select from between the legal actions, the illegal ones would eventually drop out, but would punishing them cause them to drop out sooner and possibly cause the agent to explore more possible legal actions sooner?

To expand on this further; say you're training an autonomous vehicle, and the output is drive direction (forward or reverse) and speed. Say for the scenario you're in, the vehicle must drive between a speed range, e.g. 20mph min, 40mph max, what do you do in the scenario where the agent gives an action to drive forward but gives a speed below the minimum speed? Or another example, say you're training to play a game, and the agent proposes an illegal action which it cannot perform.

I can't proceed with the action because it's illegal, so what do I do?  How do I proceed with training in that situation? I will of course enter the min/max speeds as part of the state given to the agent, but how do I prevent it from proposing actions that are illegal, and how do I proceed with training when it does?

I think that you should specify better what an illegal action is. Suppose to me in a highway with a self-driving car. I don't which are your legal limits values but suppose that they are between 20mph and 40mph. Of course, the car itself could drive at less than 20mph and let us suppose that its maximum possible velocity is 60mph. If your self-driving car is driving at less of 20mph or between 40mph and 60mph you should give a negative reward for each time-step it is outside legal limits. Instead, if your self-driving car is at more than 60mph you have a problem in your environment because 60mph is a physical limitation and should be handled by the environment. An easy solution would be to clip the action between 0mph and 60mph and give negative reward if it's in the range 0mph to 20mph or if it is in the range 40mph to 60mph.


How to trust the labels generated using ML models? I have a dataset of patient records. But I do not know whether he is +ve for a cancer or not. So, I do not have the labels in my dataset.

Now I can run a machine learning models like clustering to generate labels.

For ex: I can run clustering to group the two classes based on similarity and find out who all belong to +ve and -ve class.

Of course, we cannot sit and manual review the patients' data to know whether he is actually +ve for cancer or not.

So when we generate labels via machine learning models like clustering above, is it a recommended approach?

Is it used in industries/real time where people don't have ground truth and only rely on labels based on ML models?

How can we trust these labels generated?

If it's a human I know that it can be trusted. But how do we trust these labels.

Are things like this being used in Industries and how do they tackle the trust issue?

So when we generate labels via machine learning models like clustering above, is it a recommended approach? Only if you can really make highly distinct 2 clusters/groups. This will be highy unlikely, especially for complicated and high dimensional datasets. One of the reasons is that clustering algorithms are just weaker than the supervised algorithms. If you can find a good representation (have a look at representation learning from Bengio), i.e. highly discriminative embeddings, than it might work.

Is it used in industries/real time where people don't have ground truth and only rely on labels based on ML models? Its an option, one can definately try it, but not rely on it.

How can we trust these labels generated? As long as you can validate it with out of fold set with ground truths, or with humans looking at the clusters, there is no problem.

Are things like this being used in Industries and how do they tackle the trust issue? Its one of the possible solutions, personally I always try first transfer learning. Especially for problems like yours, chances are there is already some pretrained model. Only thing you need is some labeling tool, for 1000 samples (it takes a couple of hours to do it but its worth it). Have a look at this tool.


Reinforcement Learning model always gives different output I am trying to build a reinforcement learning model for hardware capacity optimisation. The state of the model would input like CPU capacity utilisation, memory utilisation. The model is supposed to predict what should be the CPU, memory etc I need to provision for my environment. The model uses DQN at its core and reward mechanism is based on the current capacity used.
The challenge I have is that every time I run with same input state and reward mechanism, I am getting different combination of hardware to be provisioned.
Is it possible that RL might give different output for same set of inputs and reward?

Reinforcement learning in principle should not be the source of stochasiticy, i.e. given exactly the same inputs the dqn model and the rl agent should give exactly the same output.
The source of stochasticity in your setting may be the random seed. Make sure to set it once and share it with all libraries
that have a stochastic aspect (i.e. network initialization, batch selection, your simulator if you are using one).


What parameters to use when normalising training, validation, and testing data? I know a similar post was made here, but I wanted to ask some follow up questions. I am conducting a cross-validation search to find values of a set of hyper-parameters and need to normalise the data.
If we split up the data as follows:

'Training' (call this set 'A' for now) and testing data
Split the 'training' into training (call this set 'B' for now) and validation sets

what parameters should be used when normalising the datasets?
Am I correct in thinking that:

We normalise dataset 'B' and then extract the means and standard deviations on it
We then normalise the validation set using those parameters obtained from set 'B'
Once we have used the validation set to find my hyper-parameters with cross-validation, then we normalise set 'A' and extract its parameters
Use the parameters from set 'A' to normalise the testing set

Is this correct, or have I misunderstood something? I know this is basic, but I can't seem to find a straightforward answer to this anywhere?

I am not exactly sure what you mean by "what parameters should be used when normalizing datasets."
However, it is important to note:
Normalization is a preprocessing step that you do to some or all of the parameters of your model before constructing the model.
But in answer to your question:
You always normalize the same parameters used in both the train and the test set (otherwise how would you be able to compare the results?).


Is Faster RCNN the same thing as VGG-16, RESNET-50, etc... or not? My understanding is that Faster RCNN is an architecture for performing object detection. It finds objects in an image and classifies them. My understanding is also that VGG-16, RESNET-50, etc... also find objects in images and classify them. Are VGG-16, RESNET-50, etc... object detection architectures in the same way that Faster RCNN is? I often see VGG-16, RESNET-50, etc... as the "backbone" for Faster RCNN and am seriously confused by the literature. Thanks in advance!

Image Classification Models are commonly referred as a combination of feature extraction and classification sub-modules. Where the total model excluding last layer is called feature extractor, and the last layer is called classifier.
Popular Image Classification Models are: Resnet, Xception, VGG, Inception, Densenet and Mobilenet.

Object Detection Models are more combination of different sub-models rather than single end to end connected models, as you mentioned it is more like an architecture. Object detection model contains a feature extraction model, region proposal network, classification and regression models. Here the above mentioned classification models (Resnet50, VGG, etc) excluding all dense layers are used as a feature extractors.
Some of the popular object detection models are: Region-based Convolutional Neural Networks(R-CNN), Spatial Pyramid Pooling(SPP-net), Fast R-CNN, Faster R-CNN, Regression-based object detectors, YOLO(You only Look Once),Single Shot Detector(SSD).


What if outliers still exist after variable transformation? I have a variable with a skewed distribution.

I applied BoxCox transformation and now the variable follows a Gaussian distribution. But, as seen in the image below in the boxplot, outliers still exist.

My question is:
Although after transformation, the variable distribution is nearly Gaussian, if there are still outliers, should we still select this transformation?
Or should we decide to use other techniques such as discretization in order to capture all outliers?

No right way in all cases. I have dealt with outliers from statistics and business problem view.

Are the outliers in a segment the business is expanding into and expect more people in these "outlier" areas? In this case these are not outliers from the business view and probably should be kept.
Are the outliers in a segment the business is retreating from and expect fewer people in these areas. Possibly want to get rid of these records.
Are these outlier records just outlier in this feature or in others? If they are just far from the mean in 1 feature, might want to keep them.
Discretizing - I am not a fan since the model is losing information. But can add an indicator variable and try the model. Might want to try multiple bucket approaches but I still think the model should see the real numbers.
I built models where the model would always get these outliers correct or incorrect regardless of if they were in the model training or not. So including these was a moot point. Want to make sure they are in a validation set to check. Make an outlier validation set.
Is the business treating these people different regardless of the model. For example, if this is a marketing model, the business might be targeting high income people regardless of what the model says. So including these in the model may penalize lower income people. Try building the model with and without.
I am sure I am missing other techniques that I have done. Outliers do not mean wrong and they do not mean right. They can be looked at from a purely statistical view. But in building a model since we should know something about the business problem and have access to subject matter experts, we can probably have a better informed decision. And always test. Might need to build multiple models.


What's the relationship between an SVM and hinge loss? My colleague and I are trying to wrap our heads around the difference between logistic regression and an SVM. Clearly they are optimizing different objective functions. Is an SVM as simple as saying it's a discriminative classifier that simply optimizes the hinge loss? Or is it more complex than that? How do the support vectors come into play? What about the slack variables? Why can't you have deep SVM's the way you can't you have a deep neural network with sigmoid activation functions?

They are both discriminative models, yes. The logistic regression loss function is conceptually a function of all points. Correctly classified points add very little to the loss function, adding more if they are close to the boundary. The points near the boundary are therefore more important to the loss and therefore deciding how good the boundary is.

SVM uses a hinge loss, which conceptually puts the emphasis on the boundary points. Anything farther than the closest points contributes nothing to the loss because of the "hinge" (the max) in the function. Those closest points are the support vectors, simply. Therefore it actually reduces to picking a boundary that creates the largest margin -- distance to closest point. The theory is that the boundary case is all that really matters to generalization.

The downside is that hinge loss is not differentiable, but that just means it takes more math to discover how to optimize it via Lagrange multipliers. It doesn't really handle the case where data isn't linearly separable. Slack variables are a trick that lets this possibility be incorporated cleanly into the optimization problem.

You can use hinge loss with "deep learning", e.g. http://arxiv.org/pdf/1306.0239.pdf


Statistical test to infer if brand preference depends on income For the data in image can anyone please suggest the right statistical test. Want to test if brand preference depends on income level.

The percentages adds upto 100% row wise meaning the values in a row are percentage share of different brands in total purchase amount of an income group.

You are going to need to know the sample size as well and multiply it back out from the percentages into frequency counts.

After that, my suggestion is you do a Chi Square Test for independence, since you have two categorical variables.

Either that or you can do an ANOVA (you technically have 3 variables: something for the percentage like a yes/no, brand, and income) , but you'll have to structure you data differently to do that and you will need the sample detail. You'll also need to convert your categorical variables into dummy variables.


Gradient Descent Convergence I'm a double major in Math and CS interested in Machine Learning. I'm currently taking the popular Coursera course by Prof. Andrew. He's talking and explaining Gradient Descent but I can't avoid noticing a few things. With my math background, I know that if I'm trying to find the global min/max of a function, I must first find all the critical points first. The course talks about convergence of GD, but is it really guaranteed to converge to the global min? How do I know it won't get stuck at a saddle point? Wouldn't be safer to do a 2nd derivative test to test it? If my function is differentiable it seems reasonable it converges to a local min, but not to the global min. I have tried looking for a better explanation but everyone seems to take it for granted without questioning. Can someone point me in the right direction?

Gradient Descent does not always converge to Global minima. It only converges if function is convex and learning rate is appropriate.

For most real life problems, function will have local minimums and we need to run training multiple times. One of the reason is to avoid local minima.


How to use different classes of words in CountVectorizer() Suppose I have a piece of writing and I want to assign probabilities to different genres (classes) based on its contents. For example

  Text #1 : Comedy 10%, Horror 50%, Romance 1%

  Text #2 : Comedy 40%, Horror 3%, Romance 30%

We have given keywords in each class through which we make a comparison. Below is the code that explains this scenario better

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

# Comedy
keywords_1 = ['funny', 'amusing', 'humorous', 'hilarious', 'jolly']

# Horror
keywords_2 = ['horror', 'fear', 'shock', 'panic', 'scream']

# Romance
keywords_3 = ['romantic', 'intimate', 'passionate', 'love', 'fond']

text = ('funny hilarious fear passionate')

cv1 = CountVectorizer(vocabulary = keywords_1)
data = cv1.fit_transform([text]).toarray()
vec1 = np.array(data)
vec2 = np.array([[1, 1, 1, 1, 1]])
print(cosine_similarity(vec1, vec2))

cv2 = CountVectorizer(vocabulary = keywords_2)
data = cv2.fit_transform([text]).toarray()
vec1 = np.array(data)
vec2 = np.array([[1, 1, 1, 1, 1]])
print(cosine_similarity(vec1, vec2))

cv3 = CountVectorizer(vocabulary = keywords_3)
data = cv3.fit_transform([text]).toarray()
vec1 = np.array(data)
vec2 = np.array([[1, 1, 1, 1, 1]])
print(cosine_similarity(vec1, vec2))

The problem with this approach is that vocabulary in CountVectorizer() doesn't consider different word classes (Nouns, Verbs, Adjectives, Adverbs, plurals, etc.) of a word in a text. For example, let's say we have keywords list as below

  keywords_1 =  [(...),
  ('amusement', 'amusements', 'amuse', 'amuses', 'amused', 'amusing'),
  (...),
  ('hilarious', 'hilariously')
  (...)]

and we want to compute similarity as follows

cv1 = CountVectorizer(vocabulary = keywords_1)
data = cv1.fit_transform([text]).toarray()
vec1 = np.array(data) # [[f1, f2, f3, f4, f5]]) # fi is the count of number of keywords matched in a sublist
vec2 = np.array([[n1, n2, n3, n4, n5]]) # ni is the size of sublist
print(cosine_similarity(vec1, vec2))

How can we modify the above code to capture this scenario. Any advice is appreciated.

First of all your question is about stemming words as mentioned in the other answer which can be found in any Python NLP library such as Spacy or NLTK.

The other point to mention here is that despite the other answer, what libraries has as Stop Words list is not actually stop word! Do no remove them! In NLP stop words should be extracted based on working corpus not based on a predefined list. In practice removing this kind of stop words usually reduces the performance on specific domain corpuses.

The Third point is that depending on the classifier and loss function you use, TF-IDF might be better than Count Vectorizer. I suppose it works better specially if Log Loss is the cost function but I am not sure. Just give it a try.


Why activation function is not needed during the runtime of an Word2Vec model In Word2Vec trainable model, there are two different weight matrix. The matrix $W$ from input-to-hidden layer and the matrix $W'$ from hidden-to-output layer.

Referring to this article, I understand that the reason we have the matrix $W'$ is basically to compensate for the lack of activation function in the output layer. As activation function is not needed during runtime, there is no activation function in the output layer. But we need to update the input-to-hidden layer weight matrix $W$ through backpropagation to eventually reach to the word embedding most suitable for our usecase. So there is this weight matrix $W'$ in the output layer.

But my question is why activation function is not needed during the runtime? Can anyone please explain?

From this StackOverflow Question

  while no activation is explicitly formulated, we could consider it to
  be a linear classification function. It appears that the dependencies
  that the word2vec models try to model can be achieved with a linear
  relation between the input words.

  Adding a non-linear activation function allows the neural network to
  map more complex functions, which could in turn lead to fit the input
  onto something more complex that doesn't retain the dependencies
  word2vec seeks.


Deep learning based Resume Parser and Scoring I want to know if Deep learning can be used for Resume Parsing and scoring of the resume.
Currently what I am doing is extracting the text from pdf or image using OCR/tesseract and finding the features like Email, Mobile No, Skills, Tenure, No of Companies, Awards etc from the text. So I have close to 100 features which are important for scoring the resume.
Can we do similar thing using Deep learning and will the accuracy be better ?
Any starting point/document/blog/github link which can help me get started on this.
I have gone through this link but this doesn't not have code to start with.

Resume parsing and scoring can be done by identifying relevant keyword in the text. NER (Named Entity Recognition) is a technique to identify relevant keywords in text. There are different framework available to work with NER and NLP as a whole. I will suggest you to look into spacy
to train a custom NER model.


Why do the deconv outputs of layer >= 2 of Zeiler&Fergus look so unrealistic? Reading the Zeiler&Fergus paper (my summary), I wonder how exactly they trained the deconv net. What was their data?

I think for one CNN which they want to analyze, they train exactly one deconv net (in contrast to training one deconv net per layer). The featuers (inputs) of the deconv net are the activations of the layer they want to analyze. The output they train them on are the activations that actually was the input of the layer they want to analyze. So although they have one deconv-net in total, they train it layer-wise. So for each training run, the weights of only one deconv layer are adjusted.

However, I wonder why the images look that unrealistic:

Is it gray because MSE is the training objective? Why aren't the first layer filter outputs gray then, too?

The reconstructed images from deconvnet represent the patterns in the original images which are very likely to strongly activate a particular featuremap at a particular convolution layer.

The gray portion in the reconstructed images shows that the values of those pixels in input images are not likely to strongly activate that featuremap.

There is no such gray portion in the first layer filters because first layer filter are representing the very simple patterns(e.g. edges) and those are very likely to present in input image at everywhere.


Characterisitcs of the data set for a binary classification problem I want to build a classifier for my problem statement and for that I don't have data. So while doing data acquisition, what should be the minimum sample size? And would it be a good practice if I label each observation myself to build a valid data set? (I cannot automate the process of labeling observation to each class while doing data acquisition and manually doing that takes up lot of time)

Unfortunately you're not going to be able to do much without at least 200-300 records. You're going to be limited to simple (i.e. mostly linear) models until your dataset expands to at least 1,000. Anything less than 1,000 will require very thorough cross validation, and if you're not careful you'll be at risk of building a model that easily overfits.

@EricLecoutre makes a great point that you should use Amazon's Mechanical Turk. It usually costs just a penny or two per record and could save you a lot of time.


Package that is similar to R's caret? Recently I stumbled upon a R package that works similar like caret, but I can't remember what it's name was and I can't find it. It seemed to be less well known, but at least as extensive. Someone able to help?

Edit: I search for this other package to widen my horizon. There is no specific other reason. This question is specific however, since I am not looking for a general recommendation, but I am looking for this specific package I have in mind. I will recognize the package once I read its functions and descriptions. I had searched for a couple of hours myself and did not find it by searching for "package similar to caret". That is why I need help.

MLR is similar to caret, MLR offers a high-level interface to various statistical and machine learning packages.

According to the package description:

  Interface to a large number of classification and regression
  techniques, including machine-readable parameter descriptions. There
  is also an experimental extension for survival analysis, clustering
  and general, example-specific cost-sensitive learning. Generic
  resampling, including cross-validation, bootstrapping and subsampling.
  Hyperparameter tuning with modern optimization techniques, for single-
  and multi-objective problems. Filter and wrapper methods for feature
  selection. Extension of basic learners with additional operations
  common in machine learning, also allowing for easy nested resampling.
  Most operations can be parallelized.

Please check the Reference and Homepage.


Convolution Neural Network Loss and performance I have a set of about ~100,000 training examples. Ratio of positive to negative example is roughly 1:2. The true ratio is more like 1:100 so this represents a major downsampling of the negative class. It is also a very noisy dataset - the examples were automatically generated through distant supervision. Each example represents a set of sentences and has 700 columns. Number of rows may vary from 10 to 100 (maybe even more).
I used a Convolution Neural Network in Tensorflow to train my model (model architecture similar to the one described here)  with only 2 epochs and stored the loss, f-score , precision and recall every 10 steps. I evaluated the model on a validation set (which too was generated automatically through distant supervision with negative class downsampling resulting in pos:neg ratio of ~1:2) every 100 steps. Here are the hyperparameters:
batch size: 60 for train, 100 for validation
epochs: 2
convolution filter sizes: 700x5, 700x6, 700x7, 700x8, 700x9, 700x10
number of convolution filters per filter size: 125 (so total of 750 filters)
dropout: 0.5
l2reg: 0.001
lr: 0.001

I'm seeing some strange behavior with the model and I don't understand why.
My training precision, recall and f-score go over 0.95 in about a 100 steps (6000 examples) and then plateaus. The loss falls down from 0.8 to 0.2 in about 200 steps and then fluctuates between 0.1 and 0.4.
On the validation set my precision, recall and f-score are over 0.95 starting from the first time I evaluate it on the 100th step. Loss fall slightly from 0.3 to 0.2.
When I evaluated on a real-world test set (without downsampling negative class so it has the true ratio of pos:neg), the actual precision and recall were 0.37 and 0.85.
My results are not making any sense to me. I use tensorflow metrics for calculating training precision, recall and fscore and scikit-learn metrics for calculation validation precision, recall and fscore. I can't find anything wrong in the code but I don't understand why I should have such results unless there is a bug. I would have understood having low precision and recall all through - the class imbalance favors the negative class and my set is noisy. However, I am very confused about why I'm having such misleadingly high scores all through..
Given that my dev dataset is also noisy and generated in the same manner as the train set, the dev results might just be useless and it is possible that the model is overfitting the noisy set. But I still don't understand why the scores are so high so soon. Also, if overfitting is the issue, do you think I should make the dropout even higher?
I've attached a screenshot of the graphs and would really appreciate your thoughts on this. Blue is train and red is dev. Thanks a lot!

Precision

Your step change in precision looks to be almost entirely explained by the change in positive class frequency. It is reasonable to expect the proportion of false positives to increase when increasing the proportion of negative examples. Even if you assume your cv results were perfect, then you would see some increase.

As an example, assume you have cv results representative of test results
 - which means same distribution before random under-sampling, and no over-fit to the cv set.

Say you measured precision at 0.97 with a t:f ratio of 1:2, and for the sake of simplicity that this was due to the following confusion table:

      Predicted:  T    F
Real T           97    3
Real F            3  197

What precision should you expect when going to the real distribution? That is the same as multiplying the bottom row of the confusion table by 50. Precision is $\frac{TP}{TP+FP}$, so your expected precision would be $\frac{97}{97+150} \approx 0.39$

Recall

The same effect does not impact recall, because it is about the ratio between true positive and false negative. So when you change the ratio of positive to negative classes, in theory recall should be unaffected.

In your case, recall has been affected, but a lot less than precision. that is promising. A drop from 0.95 to 0.85 between cv and test is not great perhaps, but it doesn't point to a really major problem, just room for improvement.

There are a few possible causes. The ones that I can think of are:

Your test set might be too small, so estimates of precision and recall have large error. So in fact there is no problem . . .
Test distribution might be different to train and cv set.
Train/CV set split might allow some data leakage (e.g. they share some common features such as data about the same person, and should be split by that common feature). In which case CV estimates could be too high.
Your mechanism for under-sampling the negative class may be biased.

What to do?

First of all, these results are unlikely to be anything directly do with faults in the model, and are not informed much by the training curves. They are also not that bad out of context (i.e. they are much better than simply guessing which items are in the positive class) - the question is more whether you could improve on them, and what the costs are to you for the different types of error. It might be worth you actually assigning real-world comparable costs to each type of error, to help decide whether your model is successful/useful and to pick the best model later on.

One thing from the training curves is that your cv and training loss look pretty close. It implies you are not over-fitting to the training data (or you should check to a train/cv data leak). You may have room to add more parameters and improve the model in general.

It is possible you could make the model even better with different hyper-parameter choices, feature engineering etc, and that would improve the scores. There is no general advice for that though, it depends on what you have available.

It might be worth experimenting with training on the unbalanced training set (take the raw data without undersampling) and instead weighting the loss function, so that costs are larger for inaccurate classification of positive class. This is not guaranteed to fix your problem, but will increase the amount of data you use for training.

Otherwise, you should investigate whether any of the possible causes listed above is likely and try to apply fixes.

Finally, in this situation, it is not unheard of to have a four-way data split:

A ratio-adjusted set split two ways:

Training data
CV or "Dev" set A

A same as production set split two ways:

CV or "Dev" set B
Test set

CV set A is used to perform early stopping and low-level model selection.

CV set B is used to perform high-level model selection against production metric.

Test set is used to assess the chosen "best" model without bias.


Tools for ML on csv files and jsons So much of what we export is in CSV and JSON files.

Is there any useful tools you know of that can automatically perform data analysis on flat file formats. For example

Basic statistics if numeric types: avg, stdev, mode, median
Column type and Cardinality detection
Find relationships between columns, if column A is X, then column B is always Y

Any of these things would be useful, even if there'd be an intermediary step of just loading the flat file into some software...

In R you can load the csv file easily by using the method read.csv and get the summary of the data using the method summary, you will get the most of the basic statistics of each columns like Min, 1st Qu, Median, Mean, 3rd Qu, Max and NA counts for numeric/integer rows and for string columns you will also get the some statics of count of different type of string if it has repeated multiple i.e factors.

Sample code:

Loading the csv file and check the few datasets:

> df <- read.csv("SampleData.csv")
> head(df)
  name age       Type
1    A  34     Active
2    B  56 Cancelled
3    C  12     Active
4    D  32 Cancelled
5   Z   34     Active

Basic statistics if numeric types: avg, stdev, mode, median

> summary(df)
 name        age               Type
 A :1   Min.   :12.0   Active    :3
 B :1   1st Qu.:32.0   Cancelled :2
 C :1   Median :34.0
 D :1   Mean   :33.6
 Z :1   3rd Qu.:34.0
        Max.   :56.0
>

Column type and Cardinality detection

> sapply(df, class)
   name       age      Type
  "factor" "integer"  "factor"
>
> nrow(df)
[1] 5
> ncol(df)
[1] 3
>

Find relationships between columns, if column A is X, then column B is always Y

For this you have to do some Basic calculation and you can go through this cookbook: http://www.cookbook-r.com/

I have attached Sample CSV Image here


Scaling label encoded values for Linear Algorithms I have encoded categorical variables to numerical values. As we know that for feeding values to Linear Algorithms like SVM or KNN, we scale the values for columns having large variations.

I have three label encoded columns, one of which has unique values from 1-3, another has 1-150 and another has 0,1,-1 where -1 represents missing values. How will the MinMaxScaler affect here? Or it is not needed to scale these columns?
If not, how can I avoid these specific columns? Generally we scale the whole dataframe.

P.S. I'm using LinearSVC algorithm.

Sounds like you should use a OneHotEncoder instead of a LabelEncoder since you are trying to encode non-ordinal data such as missing values. Also, one hot encoded values don't get affected by the MinMaxScaler, so that should be fine.

If MinMaxScaling makes sense is dependent on the categories. If your categories are ordinal, like [1,2,3] is [low, medium, high], then it still makes sense to use a LabelEncoder with a MinMaxScaler.

But if you have non-ordinal categorical values, like [-1,0,1] for [MISSING, DOG, CAT], then it would be better to use a OneHotEncoder instead of forcing ordinality with a LabelEncoder. Otherwise the algorithms you will use will make the assumption that the distance MISSING-DOG is longer than MISSING-CAT, which makes no sense.


Explanation of the F beta formula The F beta formula according to the wikipedia is "The weighted harmonic mean of precision and recall". I can not understand why in the left part of equation there is beta and in the right one is beta^2:

To my mind if I claim that Precision is 5 times important than Recall:
F beta = (1+beta)/(beta/P+1/R)=(1+beta)PR/(beta*R+P), where beta=0.2.

Is this right?

That's a great question, because on its face it seems like the weight should be $\beta$ alone, and, it should be in front of recall. The answer is in the text from which that reference is taken, on page 133: http://www.dcs.gla.ac.uk/Keith/pdf/Chapter7.pdf

The definition is designed to make the metric indifferent to a change in precision or recall when $P/R = \beta$. That is, $F_\beta$ increases by the same amount when either precision or recall increases, at the point where precision is already $\beta$ times bigger than recall.

The definition does indeed weight recall more highly as you can verify. Honestly on re-reading the text above, I was confused, because I don't see how it makes sense to think of "equilibrium" as the point where precision is much bigger, if recall matters more.

I plugged in the formula to Wolfram Alpha, and:

Hm. These are only equal if $R/P = \beta$! I think the paper may have misstated this then, or else I've really missed something. It's a formula whose value changes at the same rate with respect to precision or recall, when recall is already $\beta$ times larger, and in that sense it corresponds to treating recall as $\beta$ time more important.


Hash function for text documents that maps similar documents to the same value I have a web site that process text documents (typically 10-100 pages) submitted by users.  Each time a user submits a document, I'd like to store a hash of the document, but I'd like similar documents to map to the same hash value.  I essentially want to know whether a user is resubmitting a slightly changed document or a new document.

I don't store the documents, so I can only compare hash values and I can't compare the documents to each other.

I've done a lot of reading about MinHash and LSH, but these all seem to be based on having a corpus of a large number of documents and then finding similar documents within the corpus.  I think these don't work for me because I need to compute my hash vector on a single document at a time without knowing anything about other documents.

In some ways I feel like this should be an easy problem.  Something like computing a hash of a bag-of-words vector, but I'm struggling to figure out a good way to do this.

My comparison is based on text and not meaning so I don't need anything like word embeddings.

Hashing the unique copies of anything, including documents, is most commonly called fingerprinting.

Picking the fingerprinting hash function depends on your use case. For your use case, pick a rolling, non-cryptographic hash function. One of the simplest examples is Rabin–Karp algorithm. Once applied, similar documents will have similar hash values.

Another issue is comparing hash values to identify near-duplicates. Exact nearest-neighbors algorithms work best but are not scalable. Approximate nearest-neighbors algorithms are scalable but can have errors. Locality-sensitive hashing (LSH) is an example of an approximate nearest-neighbors search algorithm. You'll have to decide the trade-off between scale and potential errors.


Difference of Activation Functions in Neural Networks in general I have studied the activation function types for neural networks. The functions themselves are quite straightforward, but the application difference is not entirely clear.

It's reasonable that one differentiates between logical and linear type functions, depending on the desired binary/continuous output but what is the advantage of sigmoid function over the simple linear one?

ReLU is especially difficult to understand for me, for instance: what is the point to use a function that behaves like linear in case of positive inputs but is "flat" in case of negatives? What is the intuition behind this? Or is it just a simple trial-error thing, nothing more?

A similar question was asked on CV: Comprehensive list of activation functions in neural networks with pros/cons.

I copy below one of the answers:

  One such a list, though not much exhaustive:
  http://cs231n.github.io/neural-networks-1/

  Commonly used activation functions

  Every activation function (or non-linearity) takes a single number
  and performs a certain fixed mathematical operation on it. There are
  several activation functions you may encounter in practice:

           Left: Sigmoid non-linearity
  squashes real numbers to range between [0,1] Right: The tanh
  non-linearity squashes real numbers to range between [-1,1].

  Sigmoid. The sigmoid non-linearity has the mathematical form $\sigma(x) = 1 / (1 + e^{-x})$ and is shown in the image above on
  the left. As alluded to in the previous section, it takes a
  real-valued number and "squashes" it into range between 0 and 1. In
  particular, large negative numbers become 0 and large positive numbers
  become 1. The sigmoid function has seen frequent use historically
  since it has a nice interpretation as the firing rate of a neuron:
  from not firing at all (0) to fully-saturated firing at an assumed
  maximum frequency (1). In practice, the sigmoid non-linearity has
  recently fallen out of favor and it is rarely ever used. It has two
  major drawbacks:

  Sigmoids saturate and kill gradients. A very undesirable property of the sigmoid neuron is that when the neuron's activation
  saturates at either tail of 0 or 1, the gradient at these regions is
  almost zero. Recall that during backpropagation, this (local) gradient
  will be multiplied to the gradient of this gate's output for the whole
  objective. Therefore, if the local gradient is very small, it will
  effectively "kill" the gradient and almost no signal will flow through
  the neuron to its weights and recursively to its data. Additionally,
  one must pay extra caution when initializing the weights of sigmoid
  neurons to prevent saturation. For example, if the initial weights are
  too large then most neurons would become saturated and the network
  will barely learn.
  Sigmoid outputs are not zero-centered. This is undesirable since neurons in later layers of processing in a Neural Network (more on
  this soon) would be receiving data that is not zero-centered. This has
  implications on the dynamics during gradient descent, because if the
  data coming into a neuron is always positive (e.g. $x > 0$
  elementwise in $f = w^Tx + b$)), then the gradient on the weights
  $w$ will during backpropagation become either all be positive, or
  all negative (depending on the gradient of the whole expression
  $f$). This could introduce undesirable zig-zagging dynamics in the
  gradient updates for the weights. However, notice that once these
  gradients are added up across a batch of data the final update for the
  weights can have variable signs, somewhat mitigating this issue.
  Therefore, this is an inconvenience but it has less severe
  consequences compared to the saturated activation problem above.

  Tanh. The tanh non-linearity is shown on the image above on the right. It squashes a real-valued number to the range [-1, 1]. Like the
  sigmoid neuron, its activations saturate, but unlike the sigmoid
  neuron its output is zero-centered. Therefore, in practice the tanh
  non-linearity is always preferred to the sigmoid nonlinearity. Also
  note that the tanh neuron is simply a scaled sigmoid neuron, in
  particular the following holds: $ \tanh(x) = 2 \sigma(2x) -1  $.

           Left: Rectified Linear
  Unit (ReLU) activation function, which is zero when x &lt 0 and then
  linear with slope 1 when x &gt 0. Right: A plot from Krizhevsky
  et al. (pdf) paper indicating the 6x improvement in convergence
  with the ReLU unit compared to the tanh unit.

  ReLU. The Rectified Linear Unit has become very popular in the last few years. It computes the function $f(x) = \max(0, x)$. In
  other words, the activation is simply thresholded at zero (see image
  above on the left). There are several pros and cons to using the
  ReLUs:

  (+) It was found to greatly accelerate (e.g. a factor of 6 in Krizhevsky et
  al.) the
  convergence of stochastic gradient descent compared to the
  sigmoid/tanh functions. It is argued that this is due to its linear,
  non-saturating form.
  (+) Compared to tanh/sigmoid neurons that involve expensive operations (exponentials, etc.), the ReLU can be implemented by simply
  thresholding a matrix of activations at zero.
  (-) Unfortunately, ReLU units can be fragile during training and can "die". For example, a large gradient flowing through a ReLU neuron
  could cause the weights to update in such a way that the neuron will
  never activate on any datapoint again. If this happens, then the
  gradient flowing through the unit will forever be zero from that point
  on. That is, the ReLU units can irreversibly die during training since
  they can get knocked off the data manifold. For example, you may find
  that as much as 40% of your network can be "dead" (i.e. neurons that
  never activate across the entire training dataset) if the learning
  rate is set too high. With a proper setting of the learning rate this
  is less frequently an issue.

  Leaky ReLU. Leaky ReLUs are one attempt to fix the "dying ReLU" problem. Instead of the function being zero when x < 0, a leaky ReLU will instead have a small negative slope (of 0.01, or so). That is, the function computes $f(x) = \mathbb{1}(x < 0) (\alpha x) + \mathbb{1}(x>=0) (x) $ where $\alpha$ is a small constant. Some people report success with this form of activation function, but the results are not always consistent. The slope in the negative region can also be made into a parameter of each neuron, as seen in PReLU neurons, introduced in Delving Deep into Rectifiers, by Kaiming He et al., 2015. However, the consistency of the benefit across tasks is presently unclear.

  Maxout. Other types of units have been proposed that do not have the functional form $f(w^Tx + b)$ where a non-linearity is applied
  on the dot product between the weights and the data. One relatively
  popular choice is the Maxout neuron (introduced recently by
  Goodfellow et
  al.) that
  generalizes the ReLU and its leaky version. The Maxout neuron computes
  the function $\max(w_1^Tx+b_1, w_2^Tx + b_2)$. Notice that both
  ReLU and Leaky ReLU are a special case of this form (for example, for
  ReLU we have $w_1, b_1 = 0$). The Maxout neuron therefore enjoys
  all the benefits of a ReLU unit (linear regime of operation, no
  saturation) and does not have its drawbacks (dying ReLU). However,
  unlike the ReLU neurons it doubles the number of parameters for every
  single neuron, leading to a high total number of parameters.

  This concludes our discussion of the most common types of neurons and
  their activation functions. As a last comment, it is very rare to mix
  and match different types of neurons in the same network, even though
  there is no fundamental problem with doing so.

  TLDR: "What neuron type should I use?" Use the ReLU non-linearity, be careful with your learning rates and possibly
  monitor the fraction of "dead" units in a network. If this concerns
  you, give Leaky ReLU or Maxout a try. Never use sigmoid. Try tanh, but
  expect it to work worse than ReLU/Maxout.

  License:
  The MIT License (MIT)

  Copyright (c) 2015 Andrej Karpathy

  Permission is hereby granted, free of charge, to any person obtaining
  a copy of this software and associated documentation files (the
  "Software"), to deal in the Software without restriction, including
  without limitation the rights to use, copy, modify, merge, publish,
  distribute, sublicense, and/or sell copies of the Software, and to
  permit persons to whom the Software is furnished to do so, subject to
  the following conditions:

  The above copyright notice and this permission notice shall be
  included in all copies or substantial portions of the Software.

  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
  IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
  CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
  TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
  SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.*


Cluster algorithm to group events in more general domains I've a list of 1,300 news events, represented by only three terms coming from running LDA topic model on thousands of tweets. Here's some of them as an example:

['manchester,bony,city', 'attack,claims,responsibility', 'police,officers,nypd',
 'goal,arsenal,liverpool', 'test,pakistan,sunday', 'obama,ukraine,merkel', ...]

I need to group them in more general domains (Politics, Sport, Health, Economy, etc.).

Which kind of clustering algorithms could I use (in Python)?

Or maybe, can I use LDA topic model, even if I don't have documents but only three words?

Some creative post-processing can be done. For instance applying Named Entity Recognition and simplify some parts (Manchester is a City). Using Knowledge Graph Analysis also gives some meta-info e.g. mapping your word to Wikipedia graph or using DBpedia may help you to recognize Named Entitiy categories (Obama and Merkel are politicians and NER does not necessarily capture their profession).

Note that the combination of named entity recognition and knowledge-base (Wikipedia, DBpedia, etc.) mapping is called Entity Linking.

Regardless of statistical learning for NLP techniques, all structures above are actually graphs so they can give you also the semantic similarity measure based on which you can use other clustering algorithms like Spectral Clustering and go to a semantically higher level of clustering.

Hope it helps :)


Could someone explain to me how back-prop is done for the generator in a GAN? I'm not very familiar with neural networks, however, I though I understood the concept of back propagation as starting from the error in the output layer. Say, we have 3 neurons in the output layer and their respective values end up being:

[1 0.5 0.3]

And we wished to obtain values

[0 1 0]

So we can compute a vector of errors between the two:

[-1, +0.5, -0.3] (Not necessarily with the - operation, but you get the point)

And back propagate from there. However, in a the generator of a GAN, it seems to me that the output layer has a bunch of neurons (representative to whatever size the entity we want to generate is) but the error is based only on a % of images the discriminator classified wrongfully. So how exactly do we do back-prop for the generator ?

The only human readable examples of GAN I've found use frameworks and libraries like autoGrad that kind of obfuscate the problem for me :/

Is there a simple way to explain how this backprop can be done ? (Like, in code, not with an "intuitive example" or "simple 4 variable equation where each of the 4 variable packs 50 years of mathematics behind it")

There is really nothing special about the backpropagation algorithm in a generative adversarial network (GAN). It is the same as that of a convolutional neural network (CNN), as CNNs are usually what the generator and discriminator of the GAN are made of. I will assume the MNIST toy example for the explanation and I will provide code to get a GAN working below.

The GAN

A GAN is composed of a discriminator and a generator. Each of these are held constant while training the other. We will thus alternate between training the discriminator and the generator. This is done separately. Training the discriminator is much easier so lets look at that, then we will look at training the generator as you asked in your question.

Training the discriminator

The discriminator has two output nodes and is used to distinguish real instances from artificial instances. To train the discriminator we will generate $m$ instances using forward passes from the generator, these are the artificial instances, their label will be $y = 0$. To generate these we simply pass a noise vector as the input to the model. We will also use $m$ instances from the real data these will have labels $y = 1$.

We can then see that the discriminator is trained exactly the same way as a basic classification CNN with 2 output nodes. I will describe the training process of a CNN below.

Training the generator

When we train the generator we will keep the discriminator fixed, this is necessary as to not saturate our parameters and make the discriminator too strong to ever beat. So what we essentially have is a CNN (the generator) connected to another CNN (the discriminator). The connecting nodes between these two models is going to be output which will generate the desired images once trained. Do note that the generator wants the instances it will produce in this case to be classified by the discriminator as being from the real distribution, so they will have labels $y = 1$.

All together this is just a CNN and the backpropagation will be computed in the exact same way. First we pass through a noise vector, it goes through the generator, some random image gets produced at its output, that then goes through the discriminator and gets classified as $artificial$. But, we are expecting the discriminator to be fooled in this case, so this is an error, it should have been labeled as $real$. We then use backpropagation to get the error contribution of each model parameter. We will then use gradient descent to update all the parameters associated with the generator.

Backpropagation

This is a method used to compute the contribution of each parameter on the error term. We then use gradient descent to update these parameters such that the next pass through should result in a lower loss rate. Picking the right loss function is essential for this process. For classification tasks, as is the case with a GAN, we typically choose binary cross entropy as defined by

$L = - ylog(\hat{y}) - (1-y)log(1-\hat{y})$

and over $N$ instances as is typically the case for stochastic gradient descent the loss function is

$L(w) = - \frac{1}{N} \sum_{n = 1}^N \big[ y_n log(\hat{y}_n) + (1-y_n)log(1-\hat{y}_n) \big] $

where $y$ is the true label and $\hat{y}$ is the predicted label.

Backpropagation in deep neural networks

Take a look at this answer here which describes the process of using backpropagation and gradient descent to train a single neuron perceptron, and then a multi-layered network.

The only difference is we are using the binary entropy loss function here which has a different derivative with respect to $\hat{y}$. This becomes

$\frac{\partial L}{\partial \hat{y}} = - \frac{1}{N} \sum_{n = 1}^N \Big[ \frac{y}{\hat{y}} - \frac{1-y}{1-\hat{y}} \Big]$

You will then backpropagate this loss through the network by using the chain rule with the activation functions you selected at each layer.

Backpropagation in a CNN

Please refer to this answer for extensive details and a derivation of backpropagation for a CNN. The process is very similar to that for a deep neural network. However, the CNN uses the cross-correlation function at each layer, so you need to backpropagate the loss function through the derivative of this function. That question asks about the inner working of a CNN with two outputs very much like our discriminator.

Finally, in order to train the generator, just imagine the same process with more convolutional layers ahead of it. When we have the associated gradient $\nabla L(w)$ for each parameters, only apply the gradient descent algorithm to the parameters that are a part of the generator model. I am sure that through these derivations you will see that the number of intermediate nodes between layers does not affect the backpropagation algorithm. It remains the same process. You should also be convinced that for a non-human the intermediate nodes that are the generated image from the GAN, are no different from any intermediate nodes in the model. We simply train these intermediate nodes in such a way that we can perceive some meaning, such as generating instances of the MNIST dataset.


When a relational database has better performance than a no relational When a relational database, like MySQL, has better performance than a no relational, like MongoDB?

I saw a question on Quora other day, about why Quora still uses MySQL as their backend, and that their performance is still good.

It depends on your data and what you're doing with it. For example, if the processing you have to do requires transactions to synchronize across nodes, it will likely be faster to use transactions implemented in an RDBMS rather than implementing it yourself on top of NoSQL databases which don't support it natively.


Confidence intervals in multivariate linear regression I am fitting my data to a multivariate linear regression $Y = BX + \Xi$, where the response is bivariate $Y\in R^{n\times 2}$, and the predictor is uni-variate but elevated to the projective plane to account for the intercept $X\in R^{n\times 2}$.

Now, finding the best fit reduces to $\hat B = (X^T X)^{-1}X^T Y$.

But I am interested in finding a $0.7$ confidence region around $\hat B$. How do I do that?

Looking at https://en.wikipedia.org/wiki/Simple_linear_regression :

This t-value has a Student's t-distribution with $n-2$ degrees of freedom. Using it we can construct a confidence interval for $\beta$:

$$ \beta \in \left[\widehat\beta - s_{\widehat\beta} t^*_{n - 2},\ \widehat\beta + s_{\widehat\beta} t^*_{n - 2}\right] $$

at confidence level $1-\gamma$, where $t^*_{n - 2}$ is the $(1-\frac{\gamma}{2})$-th quantile of the $t_{n−2}$ distribution.


Random forest Classfication I am a beginner in KNIME and I need to predict attribute I have some questions :

1- How can I choose the most related attribute to predict the target attribute?

2- can I choose the attributes randomly then compare the accuracy?

3- can I use nominal attributes or only numerical attributes?

1) You don't, at least initially. Use all of the variables you have, and through proper cross validation/model tuning (in particular, tuning the number of variables you randomly select to try in each split of the tree, and maybe stuff like minimum node size and maximum size of individual trees) let the random forest find the most predictive variables for you.

If you have a lot of candidate variables, then maybe try an wrapper based method like permutation importance, recursive feature elimination, etc. though for now I'd avoid this especially if you are a beginner (these methods tend to take a long time, and may not be useful/offer marginal improvement).

2) What is the purpose of this? As a means of feature selection? Random forests already kind of do this; they randomly select a subset of your variables at each split in a tree. Again, I would stick to letting the random forest do its feature selection for you for now.

3) Both. Nominal attributes are often one-hot encoded/dummy encoded (create k-1 dummy columns, k = number of categories in the nominal variable) which I encourage you to search up. Basically, you are encoding a nominal attribute into a numerical one.

You can also encode nominal variables through other means (like target encoding) but again, if you are a beginner let's avoid that for now because you can severely screw up your model validation by doing this.


How to fix inconsistent (variable spelling) categorical data and "fill in" missing data I am a newbie data science engr. My first challenge is to (1) normalize inconsistent values in categorical features and (2) fill any missing information.

To describe inconsistency lets say we have a country field and there can be multiple entries for USA. e.g. USA, United States of America, US, US-50, United States, America, etc.

How do I normalize all these entries and just say USA for all entries. I can think of some set of full text search rules to match against using lucene or something. Is there any specific technical term for this problem?

I'm also new to Data Science, so my approach may be laughably naive.  But I work as a "terminology analyst" who prepares and maintains standardized vocabularies and code sets (which are meant to avoid the very problem you describe), so at least I'm describing things I've actually paid for.  In fact, when my program was first established, a slide presentation called "The State of the State File" was prepared describing a problem very similar to yours: The file used to populate the data entry pick list of states contained hundreds of different ways the 50 US states could be represented in text.

To answer your question: I would call the name of the problem "unstructured data" or "unstandardized data."  We call the process of fixing it "data standardization."  The outcome of such standardization is a "reference terminology," which is a list of standardized terms as well as synonyms (aka "search terms") for each standard term.  Sometimes alphanumeric codes are assigned to each standardized term, resulting in a "code set."

The approach my organization has taken to standardizing data is rudimentary and hardly qualifies as any sophisticated data science: we run a DISTINCT query to get all the different text strings used to represent concepts.  Then that list needs to be reviewed to see what different text strings represent the same thing.  If your list is too large to review each text string in the amount of time you have, then you need to get frequency data for the number of times each distinct text string occurs. Sort the list of terms from highest to lowest number of occurrences, and work your way down the list as far as you can in the time alotted.

Sounds simplistic, but sometimes you need to make judgement calls about equality or subsumption.  For example, if one of your text strings is "the continental united states" (excluding Alaska and Hawaii) do you lump that under "USA" or does it get its own category?

Your result would be a reference terminology of a finite number of standardized country names, and all the synonyms for each country.

One place to start is ISO-3166, a set of codes "for the names of countries, dependent territories, special areas of geographical interest, and their principal subdivisions (e.g., provinces or states)."  You may be able to match each of your text strings to one of these established codes.


Where can I find a trained neural network data to play with? This is the trained neural network for the XOR operator:

Can I find something like a trained network for recognizing hand writing digits somewhere on the internet? Is there an "official format" for trained neural networks?

What about the super popular and classical MINIST? Tensorflow has a nice tutorial:

  https://www.tensorflow.org/get_started/mnist/beginners

Just follow the instructions!


How do I build a recommend system based on user's past purchases? I am exploring approaches to build a model that shows personalized search results (with or without query) for a fashion eCommerce platform. For that I am first working on coming up with a bunch of products for each user and their corresponding likelihood to buy it.
I have user's purchase history i.e the list of all the products with the client has bought with brand name and the dress category that the product belongs to (shoe or top etc.)
So I want to populate client's search results with the items that the client is most likely to purchase based on what he has bought in the past. So I am trying to build a model that estimates the probability that the user is going to like a suggested item. The products are part of a larger product inventory.
Is this a content based filtering problem? Currently I am trying to create each client's vector profile based on his liking towards a brand or not? Is this the current way.

Content-based filtering is relevant whenever you have features to learn from (features about the products \ user). Sound like this is not your case as you mostly rely on the historical choices \ preferences of your user base. Sounds like a classical use case for Collaborative-Filtering (CF) based approach.
There are many possible CF implementations, most common ones are:
Memory-based: User to User \ Item to Item based similarity
Model-based: Matrix factorization
Of course, there are much more implementation options but these ones could give you a good starting point


Is each form of word classification also considered to be '(named) entity recognition'? In an article that I am writing, I focus on word classification. A typical task that involves word classification is (named) entity recognition. Entity recognition is a rather broad task and seems to cover other sub-tasks as well.
Therefore, it seems fair to me to use the terms interchangeably.
Is this a fair assumption?

Is this a fair assumption?

No: Named Entity Recognition (NER) is a specific task which consists in detecting named entities. The more general term for this kind of task in Machine Learning is sequence labelling, because it's not only about classifying words but annotating a sequence of instances in which order matters (e.g. words).
It's true that NER is certainly the most famous task of this kind, but there are other important ones, for instance Part-Of-Speech (POS) tagging.


Finding predictions using biglm without finding errors I'm using the biglm R package for linear regression.

click/impression is the output required. But the test data does not contain click and impression.
The predict function of biglm gives the error

  Error in eval(expr, envir, enclos) : object 'click' not found

I assume this is because predict tries to compute the standard errors also.

Is there a method to just obtain the predictions ? I tried assigning values to se.fit and type attributes, but I get the same error.

biglm calls model.frame which "all the variables in the formula are included in the data frame" see documentation for model.frame. This is the issue that comes up when predict is called on the biglm class. It looks for those values in the predict function. To get around this you can just create a variable and encode it with 0. See below...

data(trees)
ff<-log(Volume)~log(Girth)+log(Height)

chunk1<-trees[1:10,]
chunk2<-trees[11:20,]
chunk3<-trees[21:31,]

a <- biglm(ff,chunk1)
summary(a)

#produces same error
chunk2 <- select(chunk2, -Volume)
predict(a, chunk2)

#Fixed
chunk2$Volume <- 0
predict(a, chunk2)


TypeError: 'GridSearchCV' object is not callable - how do I use a pickle of an SVM (Scikit-learn)? I have created an SVM in Scikit-learn for classification. It works; it prints out either 1 or 0 depending on the class. I converted it to a pickle file and tried to use it, but I am receiving this error:

TypeError: 'GridSearchCV' object is not callable
(occurs during the last line of the program)

How can I overcome this?

Code:

import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB, GaussianNB
from sklearn import svm
from sklearn.model_selection import GridSearchCV
import numpy as np
from sklearn.externals import joblib
from joblib import load
import pickle

dataframe = pd.read_csv("emails.csv")

x = dataframe["text"]
y = dataframe["spam"]
x_train,y_train = x[0:5724],y[0:5724]

cv = CountVectorizer()
features = cv.fit_transform(x_train)

tuned_parameters = {'kernel': ['rbf','linear'], 'gamma': [1e-3, 1e-4],
                 'C': [1, 10, 100, 1000]}

model = GridSearchCV(svm.SVC(), tuned_parameters)

file = open("finalized_model.sav",'rb')
model = pickle.load(file)
file.close()

X = pd.read_csv("ExampleSingleEmail.csv")
model(cv.transform(X))

Most sklearn model instances are fitted using the fit method.
Same goes here - https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html.

Change the last line to model.fit(cv.transform(X)) and it should work.


Threshold Value for Random Forest Classifier Is there any way to set a threshold value that can be set for voting results for Random Forest Classifier in sklearn package in order to prevent any misclassification for incoming test data.

Yes and the most important parameter is the tree depth. It's a pre-pruning technique that allows to prevent overfitting. Specifically for sklearn is:

estimator.tree_.max_depth

I suggest you to perform GridSearch on max_depth:

params = {'max_depth':[1,50]}
gs = GridSearchCV(DecisionTreeClassifier(), params)
gs.fit(X,y)

where X is you training set containing instances
and y are the labels.

There are packages that also support post-pruning like this one https://svaante.github.io/decision-tree-id3/


Rearranging data frame from column names to key value pairs If I have data like this in R:

> a = data.frame(a = c(1, 2),
               b = c(3, 4))
> a
  a b
1 1 3
2 2 4

but I would like to have it like this:

> b = data.frame(k = c("a", "a", "b", "b"),
+                v = c(1, 2, 3, 4))
> b
  k v
1 a 1
2 a 2
3 b 3
4 b 4

How can I convert the data this way?

This is a simple gather() using package tidyr:

> library(tidyr)
>
> gather(a, key = "k", value = "v")
  k v
1 a 1
2 a 2
3 b 3
4 b 4


Keras ModelCheckpoint Callback returning weights only even though both save_best_only & save_weights_only are set to False I am trying to understand what happens here when I use the Keras ModelCheckpoint callback without setting either save_best_only & save_weights_only to True.
i.e.
from keras.callbacks import ModelCheckpoint
mc = ModelCheckpoint(filepath = './checkpoint/to/save.h5')

According to Tensorflow, both save weights and save best are both set to False by default.
tf.keras.callbacks.ModelCheckpoint(
    filepath, monitor='val_loss', verbose=0, save_best_only=False,
    save_weights_only=False, mode='auto', save_freq='epoch',
    options=None, **kwargs
)

So when none are initialized to True, what gets saved?

Keras should be accessed as tf.keras now with tf2 ,so your import should be written as
tf.keras.callbacks.ModelCheckpoint

Keras own documentation as well as tf api documentation can be easily accessed for this purpose. Keras ModelCheckpoint class mentions the following arguments in official docs:
save_best_only: if save_best_only=True, it only saves when the model is considered the "best" and the latest best model according to the quantity monitored will not be overwritten. If filepath doesn't contain formatting options like {epoch} then filepath will be overwritten by each new better model.
save_weights_only: if True, then only the model's weights will be saved (model.save_weights(filepath)), else the full model is saved (model.save(filepath)).
So both 'save_best_only and save_weights_only'  have default value as False and will save all weights and full model if not True.


Why does the overfitting decreases if we choose K to be large in K-nearest neighbors? I am studying machine learning and I am focusing on  K-nearest neighbors . I have understood the algorithm, but I have still a doubt, which is on how to choose the K for the number of neighbors.
I have seen that choosing $K=1$ lads to a frafmented decision boundary, while if I choose a larger value of K, I obtain a smoother boundary:

I think that I have lear the reason of why this happens.
But I have also studied that if $K=1$ we do overfitting, whiile if we increase $K$ the overfitting decreases, but why?
So, why does the overfitting decreases if we choose K to be large in K-nearest neighbors?

Overfitting is "The production of an analysis which corresponds too closely or exactly to a particular set of data, and may therefore fail to fit additional data or predict future observations reliably." (Oxford dictionary)
When you fit a ML model, you use a dataset that you assume is a sample of the real statistical distribution you want to model. This means that the dataset does not fully reflect the real distribution and will contain anomalies, exceptions or some randomness. An overfitted model would stick too much to the particularities of this dataset and be too variant, instead of learning statistical patterns that would smooth these particularities and bring the model closer to the real statistical distribution.
In the case of KNN, K controls the size of the neighborhood used to model the local statistical properties. A very small value for K makes the model more sensitive to local anomalies and exceptions, giving too many weight to these particular points. On the contrary, a too large value of K would make the model ignore the local structure of the distribution you try to learn, and produce an underfitted model.
Let's take an example to illustrate this:
In a given city, you want to predict if a household has low or high income because you have data about most of them, but not all. You decide to use KNN and predict the income category of the households with no data according to the majority category among the K closest neighbors. This is likely to work well because we know that housing prices tend to create low or high income neighborhoods.
If you use a small K, let's say K=1 (you predict based on the closest neighbor), you might end up with these kind of predictions:

In a low income neighborhood, you wrongly predict one househlod to have a high income because its closest neighbor has a high income. Indeed, this neighbor refused to move out from his childhood house though he could afford living anywhere else.
In a high income neighborhood, you wrongly predict one household to have a low income because its closest neighborhood has a low income. Indeed, though he couldn't afford the rent, the house belongs to the family since ever.

The model overfits on these particular cases. Now if you increase the number of neighbors you compare to (= increase K), you will get a majority of either low or high income households in the neighborhood and correctly classify these examples.
Finally, if you increase K too much, you will have an underfitted model. In our example, if we increase K indefinitely we will end up taking all households in the city as neighbors, and basically always predict the category that is the majority category in the city, ignoring local particularities.


Finding best match/score between 2 datasets I am looking for a potential matching algorithm to apply to 2 datasets (DS1,DS2), that would provide a score for all DS1 x DS2.
To exemplifies the problem: imagine a pool of job seekers are looking for jobs. What would the jobs that match the job seeker profile, and what could the job seekers that match a job.
Any pointer or documentation you could direct me to help me with this?

inputs:
jobseekers (DS1)
ID | name | skills | bio | ...
u1 | alex | C#     | candidate bio blah ...
u2 | john | JVM,AWS| ...
u3 | emma | AWS,CSS| ...

jobs (DS2)
ID | name            | spec                | skills needed | ...
j1 | C# engineer     | a job spec blah ... | C#,AWS
j2 | Java developer  | a job spec blah ... | JVM,AWS
j3 | VueJS developer | a job spec blah ... | CSS

outputs:
scoring matrix
   j1  j2  j3
u1 0.2 0.1 0.3
u2 0.3 0.4 0.5
u3 0.6 0.1 0.3

Welcome to the site! It is hard to answer this without more details (how much data do you have? On what parameters do you want to score the data?).
If you have some previous training data, you can look at recommendation algorithms with side information.
If you don't have any training data, then you essentially want to compute text similarity between the job spec and candidate's skills. For this read up on text similarity measures, cosine similarity etc. The first step in building such unsupervised algorithms is representing your text data in terms of vectors, basics of which are explained here.


Time Series - Values recorded every 10 minutes - Fill missing values I have a time series data from a sensor that records value periodically - sometimes - every 10 minute period, other times every 5 minute period etc. I have to find out anomalies in real time (as and when data comes) based on breach of a static threshold value.

Question is related to the minutes fall within the period and do not have any recordings.

Let's assume today the sensor is recording values every 10 minutes (APPROXIMATELY) as shown below.
What is the best possible way to handle the data in between the periodic interval. In below example, should I do a back fill with value 12 for all the points from 13:02 to 13:09?

2019-04-25T13:01:00  10,
2019-04-25T13:02:00  NAN,
2019-04-25T13:03:00  NAN,
2019-04-25T13:04:00  NAN,
2019-04-25T13:05:00  NAN,
2019-04-25T13:06:00  NAN,
2019-04-25T13:07:00  NAN,
2019-04-25T13:08:00  NAN,
2019-04-25T13:09:00  NAN,
2019-04-25T13:10:00  12,
2019-04-25T13:11:00  NAN,
2019-04-25T13:12:00  NAN,
2019-04-25T13:13:00  NAN,
2019-04-25T13:14:00  NAN,
2019-04-25T13:15:00  NAN,
2019-04-25T13:16:00  NAN,
2019-04-25T13:17:00  NAN,
2019-04-25T13:18:00  NAN,
2019-04-25T13:19:00  11,
2019-04-25T13:20:00  NAN

One option would be to interpolate the missing values. There are various methods depending on your use case (ie. linear, cubic, etc.)

https://docs.scipy.org/doc/scipy/reference/tutorial/interpolate.html

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.interpolate.html

You could also drop the missing values df.dropna() and just use the known values you have for thresholding.

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dropna.html


Deciding about dimensionality reduction, classification and clustering? Could you please help me to understand it because I'm not sure if I got it correctly.

Let's say I have a dataset, of persons, with 100 features, various characteristics like height, weight, age, etc. I want to classify if are normal or abnormal. By abnormal I mean if a 20 years old man is 170cm and 150kg to identify it as abnormal.

Should I use Self Organising Map to reduce the dimensionality (these 100 features) and then K-means to classify them into normal and abnormal? Is that a correct approach? Or I can use just K-NN to classify them into normal - abnormal without any dimensionality reduction?

How many features can I use with K-NN? All the examples I've found so far use just two.

And if later I'd like to find why this person has included into the abnormal class how can I find that this happened because of these two features, his weight according to his height?

Should I use Self Organising Map to reduce the dimensionality?

Not the first choice. For dimensionality reduction you have more handy and basic methods like PCA and NMF (if you don't have negative values) or Archetypal Analysis which are recommended to be used first.

and then K-means to classify them into normal and abnormal?

Red alarm! Kmeans is not for classification but clustering. Be careful about the fundamental understanding of what you are about to do otherwise you get confused. If you have labels K-NN for instance is a supervised (thus proper) method.

Or I can use just K-NN to classify them into normal - abnormal without any dimensionality reduction?

Impossible. As you already know you have the problem called Curse of Dimensionality where the euclidean distances are highly distorted. You need to reduce the dimenstionality before K-NN. The only class of methods I know which may not need dimentionality reduction are Kernel Methods e.g. SVM.

How many features can I use with K-NN?

Once I examined the Curse of Dimentionality with K-Means in MATLAB and I saw after 7 dimensions everything was distorted. It of course depends on the nature of the data (distribution,etc) but this is what I got from my experiment.

And if later I'd like to find why this person has included into the abnormal class how can I find that this happened because of these two features, his weight according to his height?

This question is so fuzzy and about the goal of Machine Learning. In Machine Learning you get a set of answers (called labels) and try to learn them. You are not supposed to say if they are right or not. So if you are said a specific guy is abnormal you should ask the one who provided labels (in the terminology we call him expert) about the reason. If a guy with a certain weight and hight is called abnormal you only know this weight and hight indicate abnormal people. Why? Well, who knows?

If I did not get the last question correctly please drop me a line in comments. Would be glad to help.

My Suggestion

I recommend you to apply PCA to your data and take the first 5 PCs and feed them to K-NN or LDA. You can plot some of these PCs against each other and see how they discriminate your classes. It gives you a good impression of what is happening. You can also plot the variance or entropy of all features beforehand and drop those features whose variance or entropy is lower than a threshold.

Good Luck :)


Getting count of frequent itemsets in Python mlxtend I have a dataset:

[['Apple', 'Beer', 'Chicken', 'Rice'],
['Apple', 'Beer', 'Rice'],
['Apple', 'Beer'],
['Apple', 'Bananas']]

I am using the frequent itemsets tools, specifically apriori, to find frequent datasets.  I convert to a OneHotDataset, then to a Data Frame:

oht = OnehotTransactions()
oht_ary = oht.fit(tempList).transform(tempList)
oht_ary

df = pd.DataFrame(oht_ary, columns=oht.columns_)
df

Then I use the Apriori function:

frequent_itemsets = apriori(df, min_support=0.01, use_colnames=True)

I see this table, which is what I expect:

support                      itemsets
0      1.00                       [Apple]
1      0.25                     [Bananas]
2      0.75                        [Beer]
3      0.25                     [Chicken]
4      0.50                        [Rice]
5      0.25              [Apple, Bananas]
6      0.75                 [Apple, Beer]
7      0.25              [Apple, Chicken]
8      0.50                 [Apple, Rice]
9      0.25               [Beer, Chicken]
10     0.50                  [Beer, Rice]
11     0.25               [Chicken, Rice]
12     0.25        [Apple, Beer, Chicken]
13     0.50           [Apple, Beer, Rice]
14     0.25        [Apple, Chicken, Rice]
15     0.25         [Beer, Chicken, Rice]
16     0.25  [Apple, Beer, Chicken, Rice]

The question I have is, is there any way to print a suppport "count" of occurrences of the datasets-meaning how many times that itemset appears in the transactions?  One thing I messed around with from the mlxtend site was this being added to add an additional column of length but couldn't get a count piece:

frequent_itemsets['length'] = frequent_itemsets['itemsets'].apply(lambda x: len(x))

EDIT:  Ok I think I am confusing folks.  What I would like to see is:

      support     itemsets    count
0      1.00       [Apple]       4
1      0.25      [Bananas]      1
2      0.75         [Beer]      3
3      0.25       [Chicken]     1

I guess I could just create a column that uses the total support and multiplies by the relative support, but was looking for a more systematic way to do it.

consider the master dataframe on which the apriori algorithm is applied as df1

  ls1=frequent_itemsets['support']*len(df1)
  frequent_itemsets.insert(loc=2,column='Count',value=ls1)

Now count is appended in your frequent_itemsets


How to create a table to display relative frequencies of selected words (eg. with, can, will) from any text corpus in nltk package in python Interested words are ['with', 'can', 'will']
gutenberg corpus is the corpus we would like to search on.
Expected output is
                           can  could    may  might should   will  would
        austen-emma.txt    270    825    213    322    366    559    815
  austen-persuasion.txt    100    444     87    166    185    162    351
       austen-sense.txt    206    568    169    215    228    354    507
          bible-kjv.txt    213    165   1024    475    768   3807    443
        blake-poems.txt     20      3      5      2      6      3      3
     bryant-stories.txt     75    154     18     23     38    144    110
burgess-busterbrown.txt     23     56      3     17     13     19     46
      carroll-alice.txt     57     73     11     28     27     24     70
    chesterton-ball.txt    131    117     90     69     75    198    139
   chesterton-brown.txt    126    170     47     71     56    111    132
chesterton-thursday.txt    117    148     56     71     54    109    116
  edgeworth-parents.txt    340    420    160    127    271    517    503
 melville-moby_dick.txt    220    215    230    183    181    379    421
    milton-paradise.txt    107     62    116     98     55    161     49
 shakespeare-caesar.txt     16     18     35     12     38    129     40
 shakespeare-hamlet.txt     33     26     56     28     52    131     60
shakespeare-macbeth.txt     21     15     30      5     41     62     42
     whitman-leaves.txt     88     49     85     26     42    261     85

Reference: https://www.nltk.org/book/ch02.html

Finding frequency of words for the Gutenberg Corpus. You need to just put Gutenberg text files as input.

cfd = nltk.ConditionalFreqDist(
    (fileids, word)
    for fileids in gutenberg.fileids()
    for word in gutenberg.words(fileids))
modals = ['with', 'can', 'will']
cfd.tabulate(fileids, samples=modals)

Output :
                        with  can will
        austen-emma.txt 1187  270  559
  austen-persuasion.txt  643  100  162
       austen-sense.txt  971  206  354
          bible-kjv.txt 5961  213 3807
        blake-poems.txt   59   20    3
     bryant-stories.txt  276   75  144
burgess-busterbrown.txt   59   23   19
      carroll-alice.txt  175   57   24
    chesterton-ball.txt  799  131  198
   chesterton-brown.txt  726  126  111
chesterton-thursday.txt  544  117  109
  edgeworth-parents.txt 1304  340  517
 melville-moby_dick.txt 1659  220  379
    milton-paradise.txt  876  107  161
 shakespeare-caesar.txt  131   16  129
 shakespeare-hamlet.txt  212   33  131
shakespeare-macbeth.txt  134   21   62
     whitman-leaves.txt 1111   88  261

Hope This answers your question :)