Gradient descent around optimal loss surface All the loss surface used in examples have some of bowl shape that decrease drastically far from the optimal and decrease slowly around the optimal flat point.
My questions are:

Has all the loss surface flat area around the optimal thus causing small gradient updates?

Most of loss curve when I trained some model shows some drastic decrease in loss and shows the decreasing become slow and the loss graph become flatten. Why the loss updates (amount of the gradient) become small ?

Based on how the gradient descent optimization algorithm works (which you can find extensevily explained in this link), find below the answers to your questions:

Has all the loss surface flat area around the optimal thus causing small gradient updates?

as far as the updated weights approximate the optimal weight values (and this happens on what you call flat area of the loss function surface), the weights updates are smaller and smaller, and this is because the derivative of such loss function on each component becomes smaller (i.e. that nearly flat area is less steep):

source
and in terms of the gradient descent algorithm details below:

source

Most of loss curve when I trained some model shows some drastic decrease in loss and shows the decreasing become slow and the loss graph become flatten. Why the loss updates (amount of the gradient) become small?

this second question (if I got it right you are asking about the learning curves) is a direct consequence of the point above: when the weights become near the optimal values when converging, the amount of weight update become smaller and the loss curve becomes nearly (with some oscillations) flat at around the minimum value, something like:

source


Reg. Pandas factorize() -Hi Experts-

I just read about factorise() function in Pandas. Using this I'm able to encode (enumerate) my string values into numbers. But, now I'm not able to understand what  numbers corresponds to what string.

Ex.

df['product_name'] # Ex. A, B, C

df['product_name'] = df['product_name'].factorize()[0]
df['product_name'] # Ex. 0, 1, 2

Just illustration, not actual o/p -

A - 0
B - 1
C - 2

How can i get this? Please advise.

-Curious newbie :)

From the documnetation

  Encode the object as an enumerated type or categorical variable.

  This method is useful for obtaining a numeric representation of an
  array when all that matters is identifying distinct values. factorize
  is available as both a top-level function pandas.factorize(), and as a
  method Series.factorize() and Index.factorize().

The examples section goes on to show that the output of the factorize method actually returns two things:

labels - referring to the new values for each of your classes
uniques - essentially the mapping back to your original labels

In your line of code:

df['product_name'] = df['product_name'].factorize()[0]

The part at the end: [0] means you are only taking the labels, throwing away the uniques that map back to your input.

If you keep both by making the same line:

df['product_name'], mapping = df['product_name'].factorize()

You could now do the rest of your work and end up with a results column full with the factorised output, you can use this line to get the original values back from those factorized labels:

mapped_back_to_product_name = mapping.take(results)

I suggest reading the documentation to get more information on how best to use the method  :-)


Correlation between specific columns of a data set I have a CSV file which has 150 columns belonging to 7 categories but I want a correlation between 2 categories. The categories are movies and music, 12 and 19 columns respectively.

Is there a way to plot a correlation matrix or a correlation graph between two of the categories and selected columns?

For example, 19 columns on x and 12 columns on y. Or summing 12 and 19 columns and having a correlation between only 31 columns instead of 150.

I'm using Python. Which packages could help me?

I recommend you to use the following example and try to manipulate the arguments and adjust them for your work:

from matplotlib import cm
cmap = cm.get_cmap('gnuplot')
scatter = pd.scatter_matrix(YOUR_TRAINING_DATA, c = YOUR_LABELS_OF_TRAINING, marker = 'o', s = 40, hist_kwds = {'bins':15}, figsize = (12, 12), cmap = cmap)


cross validation for small dataset I have a dataset of 39 medical MR images, and I have to build a model to classify the tumor type. so is it suitable to use k-fold cross validation for validating the model? if so, what would be the number of K?

Given the size of your data set, the best approach to cross validation is the
Leave-One-Out method. You haven't discussed the language or package you used for your model, but generally speaking you set the $k$ equal to the number of records.  In your case, that 39.  This will cause your model to train on 38 instances and predict the 39th, with each instance eventually will receive a classification.


Do Data Scientists prefer MACS? I see many Data Science (DS) tutorials done on MACS, and many DS blogs recommend MACS as the best developing platform, thus the quote "Data Science is statistics on a Mac" came more than once into my mind.

I'm quite fascinated by MACS (be it iMAC or MacbookPro) but I never could get a valid reason to why Data Scientists in particular use them (if that is true, of course)

All who I asked said: "because it has Unix", but I hardly manage to accept it as a valid reason (at least for me). The hardware is very weak for Machine Learning processes, and the OS is inefficient and cumbersome (no file path, no drag and drop photos and music files, windows don't fully open by default nor close, etc)

The only benefit I found so far in MACoS (or Linux for this matter), is that when running parallel processes, the RAM is being shared, unlike on windows, but quoting one of my lecturers: "you need to be mad to switch OS for that reason only"

Would be grateful if you could enlighten me on the subject.

This question is highly subjective, like asking what colour keyboard is best for programming faster.

A large number of Computer Science research at universities is done using Unix based operating systems (normally a Linux variant like Ubuntu) because it is completely open source and allows the user to tweak things as they please. Windows never allowed this officially. Apple's OSX doesn't either. However, OSX is based on Unix like Linux, and so there are many more similarities in every day usage between Linux and OSX than Linux and Windows. Many of the poster-boys/girls from data science have an academic background, where they likely used Linux.

Because many of the latest research ideas come from universities (where I claim most researchers use Linux), it is common to see open-source projects that have only been tested and packaged for Linux usage. Making them run on OSX is a smaller jump than going for Windows, as OSX and Linux are more alike.

Other guesses as to why many pick Apple products, are that they generally have higher build quality, they have a reputation for many thing just working (though IMO, Microsoft and Linux have become almost as good in that respect) and they are marketed as being luxury consumer products - and so are visually appealing. I'm sure there are books written on how social psychology with respect to this. All this ends up with the price being somewhat inflated for the actual hardware parts that you get.

In the end, someone who wants to get work done and has the necessary skills  will manage it on a 10 year old netbook running Windows. Likewise, owning a Linux server with 8 GPUs and a Macbook Pro to remotely connect to that server is no guarantee of good work. I for one have worked on all three platforms, for data science as well as other tasks, and although I have my preferences, the operating system rarely poses the biggest problem.


What is the easiest way to scale a data science project based on scikit stack? This is an issue for all Data Scientists who have worked with this stack:

python
scikit-learn
scipy-stats
matplotlib
etc.

We are looking for ways to have a project already implemented in the aforementioned stack scale for very large datasets by doing the minimum amount of work

Counter examples would be to rewrite everything in Tensorflow framework or use industry tools that are unrelated with Python.

You generally don't.  Scikit-learn is primarily aimed to help new data scientists quickly get comfortable with data science

That being said, some strategies for scaling are discussed here: http://scikit-learn.org/stable/modules/scaling_strategies.html

This includes using out-of-core models, reducing data size with PCA, and various incremental learners

Besides that, your best bet is to use a beefier computer

Also, remember that once a model is trained, it can be pickled and shared.  Training/testing is usually the time/cpu consuming process.  So, once you have a model, you should be able to implement it on machines less beefy than the train/test machine


What do you pass for the cv parameter in the sklearn method cross_val_score In sklearn, there is a method for cross validation called cross_val_score. One of the parameters of this method is 'cv'.

I understand in cross validation, there is no splitting the data into training and validation (70-30 split). Instead, you split the data into 'k' subsamples, then train it on the K-1 subsamples and validate using the kth sample.
And repeat it for each of the 'k' subsamples.

So is this cv = k, i.e the number of subsamples in which you split the training data?

It determines the splitting strategy used by sklearn.

The default (“none”) is 3-fold CV.

Doc


Should I perform cross validation only on the training set? I am working with a dataset that I downloaded from Kaggle. The data set is already divided into two CSVs for Train and Test.

I built a model using the training set because I imported the train CSV into a Jupyter Notebook. I predicted using the Train CSV itself. I would like to perform cross validation. Should I perform cross validation on the train CSV and split it into two parts again Train and Test? Or, should I import a new CSV file Test and combine both CSVs into ONE?

Chapter 5 of "Introduction to Statistical Learning" covers CV and bootstrap. I strongly recommend to read this chapter, since sampling methods are extremely relevant in practice.

Cross validation (CV) usually means that you split some training dataset in k pieces in order to generate different train/validation sets. By doing so you can see how well a model learns (and is able to make predictions) on different samples of a training dataset.

During training and model tuning, your model should not see the test data! The idea is that you reserve a truly "exogenous" dataset (never used during training) to test how well your model does in the end. If you use your test set during training, information may leak from your test set to the model, and you cannot demonstrate exogenous validity of your model anymore (becasue information from the test set leaked to your model).

In the image below, you would make four different splits on your training set, where blue would represent training cases and orange the validation cases. You run the same regression/classification on each of the four blue parts, and obtain predictions on each of the four orange parts.

There are different strategies to deal with data characteristics. In order to balance groups etc, you can use a stratified sampling strategy. See an sklearn implementation here.

Since sampling strategies can be relevant, it is advisable to let some tool like sklearn do the CV splits. Most tools/methods come with some CV options if relevant.


Text Mining with Naive Bayes I'm implementing prediction code for courses of computing fields using Naive Bayes classifier. The output is to predict whether the course is (management, design, database, analysis,…9 classes). I have only about 100 records of courses descriptions. I label 50 by search keyword and I will use them as 80% of them as train data and the 20% as test data where the other 50 courses to predict by the classifier. The classes are not balanced I can’t to change anything in data.

The accuracy is 75% how can I increase the accuracy?

Is it small for training dataset?

Is there any reason you have to use Naive Bayes? While Naive Bayes does handle multi-class modeling quite nicely, sometimes the "naive" assumption that each word in the text is independent of the others is too naive, especially given the size of your training set. While you may be able to increase the accuracy a little bit by doing more text processing, I wouldn't expect to see significant improvement, especially given the training size.

Since Naive Bayes itself doesn't have much parameter tweaking, if you want to try more processing of the data itself to improve performance, you can try things like text count vectorization or TF-IDF vectorization to represent the text data more holistically than just keyword flagging.

If you are able to implement other models, I would take a look at the scikit learn multi-class models for ideas. Tree based methods are also inherently multiclass, and have more parameters to tweak, so implementing something like Random Forests may suit you better.

As for the training size itself, that does seem a little small. Especially if you are using text data, the more features you include, the more data you need to be able to stand a chance of detecting a signal in the data.


How i can generate the probabilistic graph for my dataset? Im doing my academic project. im having the base paper for reference  the paper is IEEE paper "effective and efficient clustering methods for correlated probabilistic graph". i wish to do this in R tool. in this paper two algorithm are implemented. i like to implement the peedr algorithm in the paper. how can i give the input for that algorithm.? suggest the packgages in R tool
 the paper can be found here
http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6570474

Søren Højsgaard has many quality resources for graphical models in R. He has a tutorial "Graphical Models with R" and a list of CRAN packages.

Additionally, mclust is one of the best clustering packages in R.


changing cost function in xgboost I'm using the newest version of xgboost package in python 2.7 and based on my problem, I'm going to change xgboost cost function to use my own defined cost function. Couple of questions:

In which part of the xgboost package should I find and change the cost function?
After changing the cost function, how can I add the updated code to the xgboost package to use the new training function instead of xgboost's pre-defined function?

This is discussed in stackoverflow, just to recap:

Define your your customized cost function, e.g.:

 def new_cost(y_pred, y_true):
    # perform calculation for new cost
    return 'new_cost', score

Then pass it to the feval argument (see official doc) for training like:

model = xgboost.train(params, dtrain, num_rounds, watchlist, feval=new_cost)

There are many examples out there if you search harder for more concrete use cases.


Can we select features by examining scatterplots? Suppose, I have a data set with eight features in my hand. I want to find features to predict the diamonds, hearts, clubs, spades.

------------------------------------------------------------------------------
|   f1   |   f2   |   f3    |   f4    |    f5    |   f6    |   f7    |  f8
+--------+--------+---------+---------+---------------------------------------
|        |        |         |         |          |         |         |

f1 column is for class-labels and the rest of them are features.

First, I have rendered scatter plots by taking the features f2 and f3 and it looks like the following,

Later, I have rendered scatter plots by taking the features f3 and f4 and it looks like the following,

If $Red$ = Diamonds, $Blue$ = Spades, $Magenta$ = Hearts, and $Black$ = Clubs,

I have the following questions,

(1) Why are some plots cohesive and others are separated?

(2) What do these two plots tell about those four types of cards?

(3) Which two features, among f2, f3, and f4, would you choose for further experiments and why?

Like one of the commenters, I also question the utility of these scatterplots in this situation.

What you've described is a standard multi-class classification problem. Your outcomes are labeled (clubs, spades, etc.), and you have seven features (f2-f8) with which to predict the outcomes.

So, simply try putting all of the features into a standard classification model (e.g., logistic regression), and see what comes out. If you believe that some features are unimportant, you can eliminate those features from the model, and examine the cross-validation score (accuracy, f1-score, etc.) to see if the feature was truly predictive or not. For a linear model, looking at the coefficients and their standard errors can also be useful in this regard. Another approach for feature selection for generalized linear models is to examine the deviance. Hill/Gelman write in their book (Data Analysis using Regression and Multilevel/Hierarchical Models) that "when an informative predictor is added to a model, we expect the deviance to decrease by more than 1. When $k$ predictions are added to a model, we expect the deviance to decrease by more than $k$".

What you should not do is to examine scatter plots to determine which features are important. The fundamental problem is that even though the data might be well-separated in seven dimensions (i.e., using all seven features), this separation might not show up well in a scatter plot since this plot is only two-dimensional.


Regime classification pre model training? i don't have a formal background in this so please bear with me.
This is what my dataset looks like :

I'm interested in modeling the first variable using the rest as explanatory.
A simple OLS yields a more or less satisfying model but inuitively, i know that there are k (between 2 and 4) regimes where the correlation between my Y and certain variables switch completely making the one full OLS across the dataset sort of irrelevant as we sort of smooth all of these regimes by doing this.
I attempted differentiating the regimes using correlation similarity with a k-means algorithm in R, the result is the coloring in the pictures.
i was hoping that dividing the dataset this way would reveal more "straight lines" where the OLS would be more "fitting".
I wonder if i'm going in the right way, would be grateful if anyone could point to better ways at achieving the fundamental problem : Modeling Y~X while taking into account that there are different regimes which require a likely different coefficient or (model altogether ?)
EDIT : I also tried MARS models but the result is not satisfying at all from a "physics" sense of how these variables are supposed to actually work together + changing the training set slightly yields changes the coefficients too much.

Just a couple thoughts:

It looks like these "regimes" could be represented as a latent variable:  you could probably design a bayesian model in which the OLS model depends on the value of this latent variable. This means that the model would still be trained only with the observed features, but would internally predict the value of the regime and this value would determine the parameters of the OLS model.
A more direct approach for this kind of case-by-case setting would be to use decision trees (or random forests), since they can handle independent models in different branches. However I'm not sure how to make decision trees and linear regression work together (or if it's possible at all).


Are there any algorithms for solid polygon clustering? I'm looking for something like K-Means for dividing solid polygons into regions. K-Means clusters discrete points. But I want to cluster (that is, partition) the points of solid polygons.

I don't see any problems with implementing K-Means extension for this case but I want be sure before reinventing the wheel.

So questions:

Are there any algorithms for solid polygon clustering?
Are there any implementations (preferably in javascript)?

I looked at GIS clustering but everything I found is about polygon clustering for zooming. And a polygon is a convex hull of geo markers and deeply inside it is about clustering discrete points.

Finally I just implemented solution as I planned and described it in Medium post.
This is "K-Means for solid polygons". I made a playground on my github.io.

A bit more details of solid clustering approach.

K-Means complexity is O(nkdi) or O(nk) per iteration without d,

n is the number of d-dimensional vectors (to be clustered)
k the number of clusters
i the number of iterations needed until convergence.

Complexity estimation of solid clustering

Suppose that clustered polygon has no self-intersections.

k - number of clusters
m - number of points in clustered polygon (to not mix with K-Means n).

Step complexity estimations

Building Voronoi diagram for k centers with Fortune's algorithm: O(k log(k)). Expected that you have centers and cells as result.
Intersect each Voronoi cell with clustered polygon.
I use algorithm from polygon-clipping package.

From description:

  "The Martinez-Rueda-Feito polygon clipping algorithm is used to
  compute the result in O((n+k) log(n)) time, where n is the total
  number of edges in all polygons involved and k is the number of
  intersections between edges."

Total number of edges for us is m and edges in Voronoi cell Tmp. In complex polygon case m >> Tmp. I don't know how to estimate number of intersections but I think with big m it can be ignored.

So full estimation of this step O(m log(m) k).

Complexities of polygon area and center of weight are O(m).

What is the number of polygon points estimation?

In worst case one Voronoi cell polygon has (k-1) edges.
Clustered polygon has m edges.
As I can imagine in the worst case intersection of the worst Voronoi cell and the worst polygon will produce 3n/2 edges polygon.
As a result the upper boundary of this step is O(nk).

Centers of weight are new centers - move to step 1.

Final complexity estimation is O(k log(k) + m log(m) k + m k) => O(m log(m) k) per iteration.

What does it mean?

If we want to solve the same problem of solid polygon clustering with K-Means then we need to discretize original polygon to K-Means points.

K-Means result will be less accurate.
If n >> m log(m) then K-Means will be slower than solid
clustering.

Interesting points

K-Means includes density as part of clustering process. Some points may be heavier than other.
Solid clustering ignores density so weights of all clustered points are equal.
We can see bottleneck effect in solid clustering. Clustering tries to spread cluster centers to have equal density. But cluster center can't jump out of its polygon so it slowly moves from dense area to spatial. In certain conditions we can see the same effect in K-Means.

I think this bottleneck problem may be significantly reduced with center jumps - jump center of the smallest cluster to the biggest one.


Why does applying PCA on targets causes underfitting? The goal:

I am new to machine learning and experimenting with neural networks.
I would like to build a network that takes as an input a series of 5 images and predicts the next image. My data set is completely artificial, just for my experimentation. As an illustration, here are a couple of examples of input and expected output:

The images of the data points and of the targets are from the same source: the target image of a data point appears in other data points, and vice-versa.

What I have done:

For now I have built a perceptron with one hidden layer and the output layer gives the pixels of the prediction. The two layers are dense and made of sigmoid neurons and I used the mean squared error as the objective. As the images are fairly simple and do not vary much, this does the job well: with 200-300 examples and 50 hidden units, I get a good error value (0.06) and good predictions on test data. The network is trained with gradient descent (with learning rate scaling). Here are the kinds of learning curves I get, and the error evolution with number of epochs:

What I am trying to do:

This is all good, but now I would like to reduce the dimensionality of the data set so that it would scale to bigger images and more examples. So I applied PCA. However I did not apply it on the list of data points but on the list of images, for two reasons:

On the data set as a whole, the convariance matrix would be 24000x24000, which does not fit in the memory of my laptop;
By doing on the images, I can also compress the targets, since they are made of the same images.

As the images look all similar, I managed to reduce their size from 4800 (40x40x3) to 36 while loosing only 1e-6 of the variance.

What does not work:

When I feed my reduced data set and its reduced targets to the network, the gradient descent converges very fast to a high error (around 50 !). You can see the equivalent plots as the ones above:

I had not imagined that a learning curve could start at a high value, and then go down and back up... And what are the usual causes of gradient descent stopping so fast? Could it be linked to parameter initialization (I use GlorotUniform, the default of the lasagne library).

Then I noticed that if I feed the reduced data but the original (uncompressed) targets, I get back the initial performance. So it seems that applying PCA on the target images was not a good idea. Why is that? After all, I just multiplied the inputs and the targets by the same matrix, so the training input and target are still linked in a manner that the neural network should be able to figure out, no? What am I missing?

Even if I introduce and extra layer of 4800 units so that there is the same total number of sigmoid neurons, I get the same results. To sum up, I have tried:

24000 pixels => 50 sigmoids => 4800 sigmoids (= 4800 pixels)
180 "pixels" => 50 sigmoids => 36 sigmoids (= 36 "pixels")
180 "pixels" => 50 sigmoids => 4800 sigmoids (= 4800 pixels)
180 "pixels" => 50 sigmoids => 4800 sigmoids => 36 sigmoids (= 36 "pixels")
180 "pixels" => 50 sigmoids => 4800 sigmoids => 36 linear (= 36 "pixels")

(1) and (3) work fine; but not (2), (4) and (5), and I don't understand why. In particular, since (3) works, (5) should be able to find the same parameters as (3) and the eigen vectors in the last linear layer. Is that not possible for a neural network?

First, thanks for the edits to your original question since we now know that you are applying the same transformation to all of your data.

Q: Why do perceptrons perform so much better than generalized linear models for some problems? A: Because they are inherently nonlinear models, with a great deal of flexibility.  The drawback is that the additional knobs require more data to correctly tune.

Big picture:

Less data can cause high-bias.  High-bias can be overcome by more data.  You have reduced your data from a 4800 feature dataset to a 38 feature dataset, so should expect to see increased bias.  Neural networks require more data than models without hidden layers.

Linearity vs. Nonlinearity

Your artificial neural network (perceptron) is an inherently nonlinear model, yet you are deciding to remove features from your dataset with a linear model (PCA).  The presence of a single hidden layer explicitly creates second order terms in your data, and there are two additional nonlinear transformations (input==>hidden and hidden==>prediction), which add sigmoidal nonlinearity at each step.

The fact that the training data and the target data are both multiplied by the same matrix to achieve the reduced dimensionality really just means that your perceptron needs to learn the matrix if it wants to reconstruct nonlinear aspects of the original data.  This requires more data or for you to reduce less of the dimensionality.

An Experiment:

I suggest trying an experiment where you perform second order polynomial feature extraction on the full dataset and then performing PCA on that enhanced dataset.  See how many features you end up with while retaining 99% of the variance of the enhanced dataset.  If it is larger than the dimensionality of your initial dataset then stick with the un-enhanced and un-reduced dataset.  If it is betweeen the dimensionality of the original data and 38 then try training the perceptron with that data.

A better idea:

Rather than using the (linear) variance to determine the feature reduction of your PCA projection, try training and cross validating your model with different amounts of PCA dimensionality reduction.  You will likely find that there is a sweet spot for a given set of images.  For instance, an SVM on the MNIST digit data performs best when the 784 pixel features are reduced down to 50 linearly independent features using PCA.  Though this is not obvious from analyzing the variance of the principal components.

Other options:

There are nonlinear dimensionality reduction techniques, such as isomap.  You could investigate the use of nonlinear feature reduction since you are clearly loosing information with the linear PCA that you have applied.

You could also look into image specific feature extraction techniques to add some nonlinearity before reducing the dimensionality.

Hope this helps!


Calculate probability vector from sample data I'd like to compare two distributions using Jensen-Shannon Divergence metric. To do this, I need two probability vectors to plug into distance.jensenshannon(p, q). From the scipy.spatial documentation.

  scipy.spatial.distance.jensenshannon(p, q, base=None)[source]

  Parameters:

  p(N,) array_like left probability vector

  q(N,) array_like right probability vector

Question

How can I calculate probability vectors from sample data?

Example:

from scipy.spatial import distance
import numpy as np

x1 = np.random.normal(size=100)
x2 = np.random.normal(size=100)

p =

q =

jsd_metric = distance.jensenshannon(p, q)

Can I accomplish this using scipy.stats.norm.pdf()?

p = scipy.stats.norm.pdf(x1)
q = scipy.stats.norm.pdf(x2)

Probability vectors are meant for discrete random variables. Your data is drawn from a continuous distribution. In order to get a probability vector from data, you need to bin it into a normalized histogram. This gets you a probability vector. You probably want to make sure though, that bin center and width for samples from both distributions are the same.

I guess you could use the pdf to calculate the probability vectors too, if you knew the pdfs of the distributions. But if you only have access to data, i.e. samples from the distributions, this is not the case.


Histograms in Machine Learning I have a large data set with over 100k samples and I want to  predict a continuous target feature from 4 other continuous features using Scikit Learn.
For this project, I would like to visualize and analyze the data using both 1 dimensional and two dimensional histograms. I know how to plot histograms and I know what a histogram means/displays mathematically but how can I make good use of it in order to analyze my data?

One thing that comes to mind is that I could spot regions with outliers, but this doesn't seem so useful/efficient (correct me if I'm wrong).

So what are useful ways to use histograms for analyzing Machine Learning data?

Thanks

I would suggest you, other than simple histograms, to visualize how variables are associated with each other using a pairplot from seaborn.pairplot(). This will let you check how correlated your explanatory variables are with each other. Multicollinearity can be a problem that you can solve using dimensionality reduction, for example.

Outliers might not be a problem, but you can't say before running any model. On that, I suggest you to run the same model more than once, with and without outliers. Also, always normalize your data, this might affect the "outlierness" of an observation.


Algorithm to calculate nerual network training time? Before starting a new machine learning side project, it would be very useful to estimate how long it will take to run 1, 10, 100, 1k epochs. A crude estimate is more than sufficient (i.e. 1 epoch would take 1 second, 10 seconds, 1 minute, 1 hour, etc..).

Given the variables below, can you recommend any heuristics that could provide an estimate?

Problem type (e.g. Image Segmentation)
Model type (e.g. PyTorch Unet)
Dataset (e.g. 10k images, 512x512)
Compute (e.g AWS p2.xlarge)
Library (e.g. PyTorch)

Is an empirical method (e.g train on smaller subsets of the data and scale accordingly) a better approach to solving this problem?

The best way to know is to train two epochs. The first epoch often takes longest, because data loading takes place with some caching.

The second epoch will give you an accurate time for each epoch.

Things to help the guesses that you might include in your question:

What are the specifications of p2.xlarge?
Where are your images stored? on the instance EBS storage? s3 storage? A separate file-system?
In which format is your data stored?
Perhaps the number of parameters in the model, but that can also be misleading!


Find the best interpolation method for missing observations I have a database  which has measurements of objects every day every hour.
However, some data is missing and I don't have measurements for all the hours.
in order to get over this challenge I have used different interpolations methods in order to create this missing data (with pandas).
So now I have several databases with those interpolations methods, and I need only one.
My question is how can I determine which interpolation is the best interpolation method?
I have researched the internet but mainly found explanation abut how to interpolate data  but not how to choose the best method and how can I visualize it.

The most basic method that springs to mind is split of a test set:
Take the data where you have recorded all variables which you might need to extrapolate in another set, and split of a percentage of that and "mask" or hide the variable you wish to interpolate in this split (maybe using the data from the other part of the split if you're using some sort of trained interpolation).
Compare the results of the different interpolation methods you are using with the actual values (that you've taken out) on a metric that suits your purpose for the data best (e.g. mean squared error, mean absolute error, logistic loss, or maybe even the outcome of some machine learning method trained on the dataset).
That way, you'll find the interpolation method that best suits your data + problem.
One thing to keep in mind is that your masking should follow the same (if any) patterns that your actual missing data has: e.g. if it only happens on certain time periods, your masking method should try to follow that pattern if possible.


Dataframe size is null? I have a function in which I want to represent my data like this:

Input is a column:[ 123 125 11 122 ...]
Output:  123  125
         125  11
         11   122

The function in python is like that:

def create_dataset(dataset, look_back=1):
dataX, dataY = [], []
for i in range(len(dataset)-look_back-1):
    a = dataset[i:(i+look_back), 0]
    dataX.append(a)
    dataY.append(dataset[i + look_back, 0])
return np.array(dataX), np.array(dataY)

My dataset is represented as 1-grams of integers in a csv file, so I was obliged to use the transpose () of the dataframe to use this function. The problem is that I find the dataframe size null, and the train and test data (after spliting) also empty.

the code is:

dataframe = pd.read_csv("train2.csv")

print(dataframe.shape)  # (0,150)
print("--------\n")
#dataframe= np.asarray(dataframe)
dataframe = dataframe.transpose()
print(dataframe.shape)   #(150,0)
print(dataframe.size)  # 0

dataset = dataframe.values
dataset = dataset.astype('float32')

# split into train and test sets
train_size = int(len(dataset) * 0.67)
print(len(dataset))
#print (train_size)
test_size = len(dataset) - train_size
train, test = dataset[0:train_size,:], dataset[train_size:len(dataset),:]
print(len(train), len(test))

print(dataset[0:train_size,:])  #[]
print(train)                    # []
print(test)                     # []

# reshape into X=t and Y=t+1
look_back = 1
trainX, trainY = create_dataset(train, look_back)
testX, testY = create_dataset(test, look_back)

Any solution please?

Your first dataframe.shape is showing you that you have no rows.  Pandas is probably treating your data as just the column names, so try pd.read_csv('train2.csv', header=None).


word2vec - log in the objective softmax function I'm reading a TensorFlow tutorial on Word2Vec models and got confused with the objective function. The base softmax function is the following:

$P(w_t|h) = softmax(score(w_t, h) = \frac{exp[score(w_t, h)]}{\Sigma_v exp[score(w',h)]}$, where $score$ computes the compatibility of word $w_t$ with the context $h$ (a dot product is commonly used). We train this model by maximizing its log-likelihood on the training set, i.e. by maximizing $ J_{ML} = \log P(w_t|h) = score(w_t,h) - \log \bigl(\Sigma_v exp[score(w',h)\bigr)$

But why $\log$ disappeared from the $score(w_t,h)$ term?

No, the logartihm doesn't disappear. From the equation
 ,
When you want to calculate
 ,
it essentially means calculating ,
Now ,
So ,

as .


How do I interpret loss in a neural network? I am studying how to evaluate the performances of a convolutional neural network, and in particular I have seen that we have to look both at accuracy and loss. I don't understand why do we have to look also at the loss, and honestly I haven' t understand really clearly what the loss is. I have understood that it is something that we want to minimize and the lower the loss, the better the performances are.

But also I have seen that if the loss is to low we are overfitting, and also that there is a sort of relationship between loss and accuracy, but I have not clear what it really means.

To me this loss concept and how to interpret it both alone and with respect to the accuracy seems an abstract concept at the moment.

Can somebody help me clarify this concepts? Thanks in advance.

[EDIT] For example, I obtain this plot :

How do I interpret this?

From what I can see, it seems that the loss decreases both for train set and test set at the same rate, and we have that if this happens the model is good, otherwise we encouter overfitting or also underfitting. But I don't understand, why we have this? So why if the are close and decrease at same pace we are doing well, otherwise not?

Thanks.

Thinking in terms of optimisation, loss is convergence indicator

NN can be expressed as an optimization problem.

In the context of an optimization algorithm, the function used to evaluate a candidate solution (i.e. a set of weights) is referred to as the objective function. When we minimise this function we also refer to it as loss function

Cross-entropy and mean squared error are the two main types of loss functions to use when training neural networks.

Graph that you plotted indicates just convergance behavior on two different sets of our optimisation problem. If they diverge too much, we are overfitting. If they stop, we stopped learning.


rows to columns in data.table R (or Python) This is something I can't achieve with the reshape2 library for R. I have the following data:

 zone       code        literal
 1: A         14           bicl
 2: B         14           bicl
 3: B         24          calso
 4: A         51           mara
 5: B         51           mara
 6: A         125           gan
 7: A         143          carc
 8: B         143          carc

i.e.: each zone has 4 codes with its corresponding literal. I would like to transform it to a dataset with one column for each of the four codes and one column for each of the four literals:

 zone  code1 literal1   code2 literal2   code3 literal3   code4 literal4
 1: A    14     bicl      51     mara     125      gan      143   carc
 2: B    14     bicl      24    calso      51     mara      143   carc

Any easy way to achieve this in R? If not, I would also be comfortable with a solution in Python.

Just two lines in plain R

X <- read.table(header = TRUE, text = "
zone       code        literal
 A         14           bicl
 B         14           bicl
 B         24          calso
 A         51           mara
 B         51           mara
 A         125           gan
 A         143          carc
 B         143          carc")

X$time <- ave(X$code, X$zone, FUN = seq_along)
reshape(X, direction = "wide", timevar = "time", idvar = "zone", sep = "")

# output
  zone code1 literal1 code2 literal2 code3 literal3 code4 literal4
1    A    14     bicl    51     mara   125      gan   143     carc
2    B    14     bicl    24    calso    51     mara   143     carc


Matrix Confusion - Get Model Precision I've this matrix confusion:

[9779  107]
[2227  148]

What is the accuracy of my model? My doubt is because the confusion matrix is calculated based on Test dataset so how can it evaluate the accuracy of my model?

Thanks!

A confusion matrix gives you the following:

[TP, FP]
[FN, TN]

where TP = 'true positives'; FP = 'false positives'; FN = 'false negatives'; TN = 'true negatives'.

You can read more here: http://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/

By taking TP+TN and dividing by TP+FP+FN+TN, you can get the classification accuracy of your model. In your case, that means (9779+148)/(9779+107+2227+148) = about 81%

More details:

This type of confusion matrix is used for binary classification.

A 'true positive' in this case means a true instance of class 0; that is, the model predicts that a given example belongs to class 0 and it really does belong to class 0.
A 'true negative' means a true instance of class 1.
A 'false positive' means the model predicts that the example belongs to class 0 when it really belongs to class 1;
A 'false negative' means the model predicts that the example belongs to class 1 when it really belongs to class 0.


How to interpret Correlation along with Coefficients of multiple linear regression ?£ I have 10000 samples. There are 4 independent variables and 1 dependent variable.
The independent variables are all centered with 0 mean.

I found the correlation coefficients between each of these variables which are as below:

I used linear regression model and below is the summary of that model:

Now, based on the coefficients of the predictor variables in the linear regression model, I have been asked to find the significant predictor (s).
Based on just the correlation values, I was thinking X 4 will be the significant predictor but its regression coefficient says a different story altogether. (x4 has the least coefficient value in lm summary output). Can you help me understand what exactly is the correct way to identify the significant predictor (s)?
Also, Additionally even if I remove the x4 variable from the lm model, the Residual standard error remains the same which kinda re-iterates the fact the it is not a significant predictor? Is
my understanding correct here?
Also, I ran the VarImp function available in R which again returned a smaller value for x4.
> varImp(lm_df, scale = TRUE)
     Overall
x1 33.673993
x2 34.858260
x3 33.820908
x4  1.969445

The  table of correlation coefficients shows the pairwise correlation between the variables in your data set: on a range from 0 (no correlation) to 1 (full correlation), to what extent does variation in one variable explain variation in the other variable?

The coefficients from the regression table, on the other hand, describe the relation between y and the different x's all else being equal. For example, the coefficient estimate of x1 tells you that, provided x2 to x4 are held constant, y is expected to change by .959 units when x1 changes by 1 unit. To understand the relation between your y and the different x's, these coefficients are usually more informative than pairwise correlations.

In addition to the strength of the effect (the site is the coefficients) which can be used to assess the substantive significance of a predictor variable, the regression table gives p-values that are used to judge the statistical significance of a predictor variable. A low p-value (conventionally below .05) suggests statistical significance (meaning a low probability of observing your data under the null hypothesis).

So looking at your regression output, you can see that all four predictor variables have a statistically significant relation with y. X1 to x3 are more significant than x4, both in terms of statistical and substantive significance. The contrast between the strong pairwise correlation between x4 and y and the small regression coefficient is due to multicollinearity between the x's: x4 covaries with the other predictors, and it is this covariance which accounts for the correlation with y.


Multiclass Regression for density prediction This is my first question in the DS-community, so I'm happily willing to accept any kind of (meta-) advice :).

I have time-series data for a set of users (~100), whereas each 15 min it is logged to which antenna (~80) they were connected (similar to cell phone connections).

Based on this data, I created a density vector, which for some time t (i.e. 01.01.2016 at 06:00) counts how many users are connected to which antenna.

Such a density object (for same day and time as given above) might look like this:

100: 5;
101: 2;
102: 3;
103: 0;

whereas the first number refers to some ID of the antenna, and the 2nd number refers to the number of users connected to the antenna.

I'm planning to feed this time-series data to a recurrent neural network.

The results should be the predicted number of users connected to the antenna at the next time step (so every 15 minutes). So it might predict for 01.01.2016 at 06:15:

100: 7;
101: 0;
102: 1;
103: 2;

Now I'm wondering what should the output layer be like? Regarding number of neurons and activation function especially. I've been reading quite a lot about multinomial logistic regression but some confirmation would be nice.

If it should output the predicted number of users per antenna, it should have probably the same number of neurons; so 80 it would be, just as in a multi-class classification scenario with a softmax activation function.

So what I need is a different activation function, but even after reading quite a bit, I couldn't wrap my head around it yet.

i.e. Get multiple output from Keras proposes to use a linear activation function, but in their case, they tried to predict the next 3 values, by using regression; whereas I am trying to predict the next 1 value for a set of antennas.

PS: For constructing the Neural Network, I'm using (Tensorflow-) Keras.

PPS: For feeding the neural network, I would generate the density vector for all time steps, and then feed batch-wise with batch_size = 80 (number of antennas). Out of curiosity: I happen to have only 1 feature, so the input_shape probably be (batch_size, 1, 1*80); If I was to have 2 features, would it then be (batch_size, 1, 2*80)?

PPPS: Not even quite sure how to name this problem. I think it would probably be called a (time-series) multiclass regression problem but I couldn't find any example with the same name (left aside the multinomial logistic regression).

If you want to predict the raw number of users, then this is a classical regression problem. Set an output layer with a node for each antenna, and no activation function.

If instead you need to predict a probability distribution / frequency on all antennas, use a softmax activation, so that each output vector would sum up to 1.


Isolation Forest Prediction To compute anomaly score we use whole dataset to test or only test dataset? Can someone please help.
And with streaming data we doesn't train so in that we are using whole dataset to find accuracy. So how would we compare both without streaming and with streaming.

Thank you.

The training dataset is only used to fit/train the model. The training will extract information from your training data and store it in your model so after training you no longer need it. You should never include training data when you later check the performance/accuracy of your model, that is only the test set.

For predictions and computation of anomaly scores you only need to use the dataset you actually want to check for anomalies/outliers.

For streaming you should train first on offline data which only contains inliers or has a know ratio of outliers. Once it is trained you can use it to make predictions on new streamed data without comparing to your offline data.


Existing pre-trained NLP models to detect if a text input is a question I would like to quickly filter text data into question and non-questions. Using the presence of question mark in the text is too crude. Are there any existing models I can use to aid me with my task?

There are multiple threads on Stack Overflow talking about this. Couple of examples can be found below:

Determine if a sentence is an inquiry
NLTK. Detecting whether a sentence is Interogative or Not?

You can also take a look at Natty. It's a bot on Stack Overflow, which reviews answers to old questions. You can see how effective it is in terms of detecting questions (every bot needs training, and Natty has been trained for quite a while). However it is written in Java, it may give you some ideas. Like detecting keywords alongside checking for presence of question mark.


Multiple formulas for r squared eval metric for regression I was came across different formulas for R squared on different articles.

  R Squared = 1 - RSS/TSS

  R Squared = ESS/TSS

RSS -Residual sum of squares.
TSS - Total sum of squares.
ESS - Explained sum of squares.

Can any one explain which one should be the correct one.

Both are correct. R squared tells you how many variance of the dependent variable is explained by your model. This can be written in different ways.
It is like to say that the probability of being alive is 1 minus the probability of dying.

ESS=TSS-RSS

https://en.m.wikipedia.org/wiki/Coefficient_of_determination

[1]: https://i.stack.imgur.com/QlM6O.jpg


One Challenging Neural Network Question and MATLAB Simulation I see this example ON NN notes:

how this solution is achieved? is there any clear trick or usual step for solving this question quickly?I think the solution in the question be wrong. no idea anyway... please be aware that this is PhD entrance exam question and we should think the information is not enough in the question.
some idea tell us that 5, 5 is answer because for NN there is no difference on condition (1) and (2) in the question.
some idea tell us there is 3, 3 solution, but with complex neurons (this is not should used here in general, because this is general question even with lack of information)
some idea tell us there is at least 5, 5 solution because we cannot model with 3 nodes here... (not for first reason)
here one implementation of these structure in MATLAB for each case with 3 or 5 nodes and I think with $5$ we get the better result.

$A) 4, 5$
$B)  5, 3$
$C)  5, 4$
$D)  5, 5$

I believe the solutions should be a total of 3 neurons for each question. This is because two neurons are sufficient to calculate the lines that separate all the points. Then, an extra neuron can be used to decide to which class does a point belong to. Let's see why:
1st Question: Output of the neurons $=1$ or $=0$
First, we should note that both classes are separable if we use two lines:

Let's call these lines $l_1$ and $l_2$ (the order doesn't matter). Because of the fact that a line depends only on 2 parameters (let's call them $k$ and $c$), we know that the points $(x_1,x_2)$ that belong to these red and blue lines will  satisfy:
$$ l_1 = x_1 + k_1\,x_2 + c_1 = 0 \\
l_2 = x_1 + k_2\,x_2 + c_2 = 0$$
If a point $(x_1,x_2)$ doesn't belong to these red or blue lines (like all the points in the picture), then the above equalities would not hold.
In this situation, we have that the points lying at one side of the line will lead to a quantity $l_i > 0$, and the points to the other side will lead to a quantity $l_i < 0$ (with $l_i$ being $l_1$ or $l_2$).
Given this, we know that the first thing a neuron computes is the weighted sum of its inputs using its weights ($w_i$) and bias ($b_i$):
$$ w_ix_1+w'_ix_2+b_i$$
This means that we can calculate the previous lines using two neurons. This can be achieved by setting $w_i = 1$, $w'_i=k_i$ and $b_i = c_i$. Just like in the next NN:

So, for example, the first neuron will be calculating the value of $l_1$ and the second neuron will be calculating the value of $l_2$.
As said earlier, all the points that need to be classified will have values of $l_i$ $>0$ or $<0$, so we can use an activation function that returns a neuron output of $1$ if $l_i>0$ and $0$ if $l_i<0$.
This way, we will end up with four possible combinations of outputs:
$$ \begin{align}
(0,0)\,\, \text{and} \,\, (1,1)\quad &\text{For one class}\\
   (1,0)\,\, \text{and} \,\,(0,1)\quad &\text{For the other class}
\end{align}$$
Thereby we can use this to finally know to which class does a point belong to. Note that the first class has these values repeated $\Rightarrow$ If we substract them we will end up with a value of $0$. However, if we substract the values of the second class, we would end up with $1$ or $-1$.
Hence we can use an activation function at the output neuron that outputs a value of $1$ when the substraction of its inputs is $0$ and that outputs a value of $0$ otherwise, achieving the correct classification. Thereby, our final NN would be like:

2nd Question: Output of the neurons $=1$ or $=-1$
The only thing that changes w.r.t. the previous reasoning is that the neurons will output $-1$ when $l_i<0$ (instead of outputing $0$). This way, we would have the next four cases with the hidden layer output:
$$ \begin{align}
(-1,-1)\,\, \text{and} \,\, (1,1)\quad &\text{For one class}\\
(1,-1)\,\, \text{and} \,\,(-1,1)\quad &\text{For the other class}
\end{align}$$
Where again, the first class has these values repeated and the second class doesn't $\Rightarrow$ we can still use the previous NN to correctly classify the data.
Edit regarding the comments
The above explanation respects the constraints given by the question (neuron outputs $\in\{0,1\}$ or $\in\{-1,1\}$), but it also assumes that we are able to choose freely how is the activation function w.r.t. its inputs.
Below in the comments we have talked about using simpler activation functions, concretely perceptrons, which have the next behaviour:
$$ \text{output} = \begin{cases}
1 \text{ if } \mathbf{w\cdot x}+ b > 0\\
0 \text{ otherwise}
\end{cases}$$
Perceptrons have already been considered above in the answer for the first two neurons, which are able to transform our initial data to XOR data.
Given this, it's important to note that 3 perceptrons are enough to model any logical function (as argued in posts like this one from Medium). This means that with the above reasoning we would need 3 more neurons, making a total of 5 neurons needed.


Blind feature engineering I received a dataset for analysis that had ~100 numeric columns with anonymous column names($X1$, $X2$, $X3$, etc...) and asked to do a binary classification.  My resulting classification algorithm using a SVM had good accuracy (> 95%), but I was unable to do much in the way of feature engineering or feature generation other than the standard scaling, null-value replacement, etc, since I had no intuition about the columns.
Is there any standard logic as to how to do some sort of automated feature generation, i.e. some simple mathematical combinations of various columns to create new, useful features?  Does this sort of thing have any mathematical basis for linear or tree-based models?  Or is feature engineering only really meaningful when one has intuition based on the column names...

I think your question has multiple answers. Let's start with a tool created with this purpose named featuretools. You can read more about it on the excellent documentation, but just to give you an intuition, think about a dataset that is on transaction level, that is, you have one row a new transaction, and then you want to make predictions based on customer level, you will create a series of features based on transactions, for example:

mean, max, min, sum, std of transactions values
number of transactions
number of unique products bought
...

This tool will automate all these tasks based on entity sets and relationships between your datasets.

Now, think about your situation, you might use PolynomialFeatures from scikit-learn to generate interactions and polynomial features that is more automated than other approaches.

Having said that, based on your score, in a real application, using these approaches, or trying to improve performance using more features, is overwhelming and not useful. We use kaggle as our standard training, and I think it is really good, but business requires explanation as well, and they will be happy with less predictive algorithm if they can trust on it.


which deep learning text classifier is good for health data I have a data set like this:

postID  Sentence                                         drugYesOrNo
1       He went out with his friends
2       He behaved nicely while talking with me
3       He stopped using drugs after a while                 1
4       He did not meet any friend during last week
1       He slowly cut usage of drugs                         1
2       He smiled like he is good
3       He did not seem happy with his situation

As you see there is two features. the first feature is our sentence and the second feature shows is this sentence is a sign that patient has stopped his drug or not.

the first column shows that sentence which are part of a paragraph. for example HERE sentence 1-4 are one paragraph in which we have splited them to see which sentence exactly show the stopping of drugs.
so sentence 3 in first paragraph shows this.

In the second case, sentence 1-3 is part of a paragraph. here sentence one shows that this person stopped using drugs(which is not good the person should continue)

so my goal is to apply a deep learning text classifier on my text data and make a model and so when I received  A NEW PARAGRAPH, I will be able to predict if the person has stopped his drugs or not.

first question, with this case study, which deep learning text classifier may work best?

Secondly, as you see we have cut the paragraph in to series of the sentences. but in reality we will give a paragraph to test the model. in your idea what will be the best approach to deal with this?

the thing that came to my mind is that while testing and receiving a paragraph, we again split the paragraph to sentences and give those sentences to the model but I am not sure it is a good approach.

we have 900 of these sentences, again Im not sure with this much of data it will be goof to apply deep learning classifier on it.

I appreciate it if you give me your point of view:)

Update after reading comments

I asked a couple of guys to make such a dataset for me. I mean looking at paragraph, spliting, then saying which sentence has that meaning(stopping drugs or not). what if I did not ask them to explicitly say which sentence does have that meaning and just point which paragraph does have that meaning(stopping drugs or not). do you think labeling exactly which sentence does have that meaning has been a good idea rather than which paragraph does have that meaning? I hope I am clear enough :)

Yes, you should split the paragraph to sentences and give those sentences to the model. Your deep structure should be like this:

In the first layer, you must put a word embedding layer to represent a sentence as a sequence of vectors. In the second layer, you must put LSTM to be able to model your sequence vector as a single vector. Now, you can add successive layers with linear, relu or sigmoid activation functions to make your model deeper. In the last layer, you must use sigmoid activation function to do binary classification.


How to weigh imbalanced softlabels? The target is a probability between N classes, I don't want it to predict the class with the highest probability but the 'actual' probability per class.

For example:

|    | Class 1 | Class 2 | Class 3 |
------------------------------------
|  1 |     0.9 |    0.05 |    0.05 |
|  2 |     0.2 |     0.8 |       0 |
|  3 |     0.3 |     0.3 |     0.4 |
|  4 |     0.7 |       0 |     0.3 |
------------------------------------
|  + |     2.1 |    1.15 |    0.75 | <- correct this imbalance?
| >0 |       4 |       3 |       3 | <- or this one?

Some classes have 'more' samples in the sense that the sum of probabilities is higher than other classes. Do I have to balance this out with weights in the loss function? Or do I only correct for the imbalance in >0 as normally?

If you have imbalanced classes (for example, if you have 3 classes and 100 examples of class 1 and 1000 examples of class 2 and 5000 examples of class 3), then yes, I would weight the loss function (I would use weighted categorical cross-entropy).

If you mean some classes have a higher probability than others, then this is normal and expected behaviour. For example, if you were doing a 10-class classification problem like on MNIST, and you're trying to predict a given image, if the image has some rounded sections then it's much more likely to be a 3 or an 8 than a 1.


BERT :dropout(): argument 'input' (position 1) must be Tensor, not str I am new to NLP and would like to build a BERT model for sentiment analysis so I am following this tutorial.
However, I am getting the error below:
F.softmax(model(input_ids, attention_mask), dim = 1)

When I would like to execute this cell I get the error:
 dropout(): argument 'input' (position 1) must be Tensor, not str

Which transformer version are you using? I had to pin mine to transformer == 3.5.1 to mitigate that problem, when the hugging face team updated their transformer to 4.0 things started to break.
Hope it helps


Which value of output should be taken in multiple sessions of training Neural Network Suppose I am using Neural Network for a 2 class classification. After training the network with the training set, I want to predict the class label of a dataset with no class label. Now with retraining, the same dataset gives different result. For example in one training session, a sample gave the output of it belonging to class 1 while in the other session it gave the output of it belonging to class 2. Then which value should be taken as the correct one?

This is normal behaviour of most classifiers. You are not guaranteed 100% accuracy in machine learning, and a direct consequence is that classifiers make mistakes. Different classifiers, even if trained on the same data, can make different mistakes. Neural networks with different starting weights will often converge to slightly different results each time.

Also, perhaps in your problem the classification is an artificial construct over some spectrum (e.g. "car" vs "van" or "safe" vs "dangerous") in which case the mistake in one case is entirely reasonable and expected?

You should use the value from the classifier that you trust the most. To establish which one that is, use cross-validation on a hold-out set (where you know the true labels), and use the classifier with the best accuracy, or other metric, such as logloss or area under ROC. Which metric you should prefer depends on the nature of your problem, and the consequences of making a mistake.

Alternatively, you could look at averaging the class probabilities to determine the best prediction - perhaps one classifier is really confident in the class assignment, and the other is not, so an average will go with the first classifier. Some kind of model aggregation will often boost accuracy, and is common in e.g. Kaggle competitions when you want the highest possible score and don't mind the extra effort and cost. However, if you want to use aggregation to solve your problem, again you should test your assumptions using validation and a suitable metric so you know whether or not it is really an improvement.


Spark algorithm to make a link analysis I'm doing my master thesis on Big Data Analytics. I'm trying to develop a algorithm to identify associations between some products in a supermakert. Imagine that I've this dataset:

Purchase_ID Product_ID  Purchase_Value
    1       2           4.5
    1       3           1.2
    2       3           1.4
    2       1           3.5
    2       2           7.3
    3       2           0.5
    3       3           1.0

What I want to conclude is that:
"Every people that by Product_ID 2 also buy 3"

Anyone knows If exists any code algorithm available to use in Spark Mllib? I already search on internet but I didn't found anything...

Anyone can help me?

Many thanks!

This sounds like a classic frequent pattern mining problem, where you're trying to find sets of items that are frequently found together within users' sets of purchases.

Start here: https://spark.apache.org/docs/latest/mllib-frequent-pattern-mining.html


How to predict probabilities in xgboost? The below predict function is giving -ve values as well so it cannot be probabilities.

param <- list(max.depth = 5, eta = 0.01,  objective="binary:logistic",subsample=0.9)
bst <- xgboost(param, data = x_mat, label = y_mat,nround = 3000)

pred_s <- predict(bst, x_mat_s2)

I google & tried pred_s <- predict(bst, x_mat_s2,type="response")
but it didn't work.

Question

How to predict probabilities instead?

Just use predict_proba instead of predict. You can leave the objective as binary:logistic.


Using RNN (LSTM) for Gesture Recognition System I'm trying to build a gesture recognition system for classifying ASL (American Sign Language) Gestures, so my input is supposed to be a sequence of frames either from a camera or a video file then it detects the sequence and maps it to it's corresponding class (sleep, help, eat, run, etc.)

The thing is I've already built a similar system but for static images (no motion included), it was useful for translating alphabets only in which building a CNN was a straight forward task, as the hand doesn't move so much and the data set structure was also manageable as I was using keras and maybe still intending to do so (every folder contained a set of images for a particular sign and the name of the folder is the class name of this sign ex: A, B, C,..)

My question here, how can I organize my data set to be able to input it into a RNN in keras and what certain functions should I use to effectively train my model and any necessary parameters, some people suggested using TimeDistributed class but I don't have a clear idea on how to use it for my favor, and take in account the input shape of every layer in the network.

also considering that my data set would consist of images, I'll probably need a convolutional layer, how would it be feasible to combine the conv layer into the LSTM one (I mean in terms of code).

For example I imagine my data set to be something like this

Folder named 'Run' contains 3 folders 1, 2 and 3, each folder corresponds to it's frame in the sequence

So Run_1 will contain some set of images for the first frame, Run_2 for the second frame and Run_3 for the third, my model's objective is to be trained with this sequence to output the word Run.

Based on my understanding, this seems to boil down to a multi-class classification problem (each set/sequence of hand gestures eventually corresponding to a word/symbol). An RNN/LSTM/GRU architecture is a great place to start, but will be computationally expensive and rather slow to train. However, whether you absolutely need one or more of these depends on how your data is structured, and whether you can get away with treating your images as a set, rather than a sequence.

Here are some potential ideas:

Start by setting up a CNN layer for each "RUN_*" folder, along with
dropout and/or flattening between each layer.
Start by setting up a single long vector representing all stages of
the sequence, then add a couple of layers of CNNs, also with
dropout and/or flattening.

This will take some exploration and tuning on your end to see what works the best, and you will need to choose with evaluation metrics you want to optimize for (e.g. precision/recall)

The reason I'm suggesting a CNN is because, in certain cases, CNNs can provide similar performance to a recurrent model, but at a fraction of the cost. Unfortunately, it's tough to tell with little to no information about you data.


What machine learning technique should I use in medical problem What are the best machine learning techniques to classify responders to a medicine if I have:

Clinical data with ~200 features (age, education, marital status etc.)
Gene data with around 250K features (genom data (snips) taken from the patient (DNA analysis))
Number of obs. ~ 4K (the data is taken from a study on 4000 patients).

Please advise.

Typically you need a much more exact phrasing (in more mathematical terms) of the question to ask.

Will a patient respond to medicine X? What is the likelihood? To what amount will patient repond to medicine X? Is patient in the group that is expected to respond to medicine X?
Are slightly diffent questions that may impact choice of technique.

Furthermore, your data plays an important role. Are you missing data? Have you normalized already? Do you expect 'Marital status' or 'Education level' to have an impact on medicine efficiency? (It might if certain groups take medicine at home, but it may be less likely when taking under doctor supervision)

A priori you determine how to measure or quantize the success of a model (typically prediction accuracy).

Then typically you try a few machine learning techniques, and make a model from the most successful one.


Structure of NN for input data with drop out In financial markets, there is a simple problem of trading calendars varying across different countries. For example, Sweden observes Sweden National Day and Norway has Whit Monday. Typically, what happens then is that a time-series in the equity market that was closed for a holiday 'catches up' the next day when the market is open again.
For example:
+-----------------+------------+------------+------------+------------+
| Date            | SEK 1      | SEK 2      | NOK 1      | NOK 2      |
+-----------------+------------+------------+------------+------------+
| Date 1          | + 0.2%     | +0.4%      | +0.3%      | +0.6%      |
| Date 2          | + 1.1%     | +0.7%      | +0.3%      | +2.1%      |
| Date 3          | -3.2%      | -2.9%      | NaN        | NaN        |
| Date 4          | +0.1%      | -0.2%      | -2.8%      | -1.6%      |
+-----------------+------------+------------+------------+------------+

The objective of my model is to adjust the NOK returns for dates 3 and 4 since they have been distorted by the NOK holiday on Date 3. To do this I will use as many good dates as I have, such as date 1 and build a large dataset by randomly dropping out some data, adjusting the following the date and using the dropped out adjustments as the input and obviously I know the supervised output as the real data.
I feel that this would be well suited for a neural network, but I have never constructed one before with dropped out input.
Would it be appropriate to just build network where input data-points that are missing have no impact on the weights for that round? Are there any neural networks (or all) that do this by default?

There is no Special reason that neural networks are suitable, if anything it could be too much (depending on the complexities of your time-series).
In any case, you need to deal with you Nan values and if you are looking for a fast model, that imputes that automatically, I would suggest lgbm. Since it will impute NAN with values that reduce the out of fold loss the most (making the highly optimal imputation).


Convolutional layer dropout layer in keras According to classical paper

http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf

dropout operation affects not only training step but also test step - we need to multiply all neuron output weights by probability p.

But in keras library I found the following implementation of dropout operation:

retain_prob = 1. - level

...

random_tensor = rng.binomial(x.shape, p=retain_prob, dtype=x.dtype)

...

x *= random_tensor
x /= retain_prob
return x

(see https://github.com/fchollet/keras/blob/master/keras/backend/theano_backend.py)

Why x is divided by retain_prob while it should be multiplied?
Or I'm just confused, and multiplying weights is equivelent to dividing output value?

Let's clarify few things about dropout. And Neil Slater is to credit for this answer since his comments helped formulate a more clear explanation.

First of all, dropout is a regularization method, it is usually only applied during training (although it can be used in prediction as an approximation to a Bayesian Neural Network as is explained by Yarin Gal's paper). As you might have understood, its goal is to limit overfitting. That is, to train a model that will generalize better to newly unseen data samples. So it has nothing to do with testing but everything to do with training.

Second, the reason why you might have seen the output multiplied by $p$ at prediction time is a trick used with the very basic implementation of dropout referred to as vanilla dropout. At prediction time (or testing time if you prefer this wording) no need to drop anymore, but it is needed to scale the outputs by $p$. The reason is that because at training time, dropout was perform with a probability $p$, then the output need to be scaled to adjust for $p$ at prediction time.

Third, inverted dropout (which is the dropout implementation used in all serious DL libraries) does not need to scale the output at prediction time because scaling by $p$ is actually already performed at training time (by dividing by $p$). Therefore, no need for any trick at prediction time!

Finally, the concept is nicely explained in these videos from the Udacity Deep Learning course or this course from Stanford.

I hope this is clear enough. :)


How do you choose an appropriate $k$ to achieve $k$-anonymity for data? How do you choose an appropriate $k$ to achieve $k$-anonymity for a data? What methods exist that are agnostic to the business context for the problem?

In most cases $k$ emerges from the volume and nature of data, plus trhe anonymity method used. Rarely does one have explicit control over $k$, except implicitly through these options.
Think of $k$ as a score instead of as a parameter.
It is possible, for example, some records will have higher $k$-anonymity than others. Then the average $k$ counts, or even the minimum.
If anonymity is a requirement, then the highest possible value of $k$ is what is needed. Since for each record there are only $k-1$ similar records, so methods can be used to exhaustively find the anonymised info, thus the highest possible $k$ is needed in order to slow down this process and make it practically impossible.
Of course the maximum $k$ is achieved when all data columns are anonymised, but this creates useless data, so the tradeoff between useful data and maximum anonymity results in a range of $k$ values to achieve (and this depends on the actual nature and volume of data).


What is the standard for dealing with null values in plotting? There's really no "standard," I assume, but how do you deal with null values when plotting data? An example of what I'm talking about is listed below:

Suppose I'm plotting data on number of toys by color. Nearly half of these toys in my database have null values for color. Do you guys include the toys with null values in your visualization? How do you address that issue? Saying "nearly half of the toys didn't have colors listed" doesn't really add much to the conversation.

Imo, It depends on two major factors:

What is the purpose of the plot
Does the null value indicate incompleteness (one of the values, but unknown which one), or does it indicate a last final option, (like: multiple colors)

Imagine if the purpose of the plot is to compare which toy colors are most popular to a particular person/group.

If null indicates multi-colored toys, it may be important to include the data but rename it to something more meaningful. (Multi-colored)

If null indicates that we just don't know the color data for those toys, it would probably be best to exclude that data from the plot. However, with such a large percentage, it would be good to note that "x%" were excluded from your plot because they had no color information associated with them. Perhaps purple is in such a majority because this person/group really likes purple, or maybe its because purple toys almost never made it into the null category for whatever reason.

The same line of thinking should be applied to whatever the purpose of the plot is. If the null values are relevant to the purpose, include them. If not, don't - and maybe take a note of how much data is excluded if it could influence the results substantially (such as this case)


training data in supervised models, case of linear regresion I have a newbie question about the case of linear regression or other supervised models for prediction. Imagine that I have the following dataset represented by the X array:

So we have n observations and p features, when I want to train a linear regression model I need to follow the formula:

which at the end can be transformed into the following summation:

At this point I have a doubt, let´s suppose that we have 3000 registers and 10 features with one variable for prediction. So, in my linear regression model (if I want it to program from scratch and not use any library) I should apply the summation formula for the 12 features 3000 times to fit the correct weights of theta. Is it like that? Any detailed explanation would be really helpful.

Thanks

no. of features $p$
no. of observations (training examples) $n$

In your example, $p = 10$ and $n = 3000$

So, what you said is right, you have to perform the following equation 3000 times.
$$h(x) = \sum_{i=0}^p \theta_{i}x_{i
}$$

and, see that the summation runs from $0$ to $p$, so a total of $p+1$ multiplications, with the first one $\theta_{0}$ being bias, and $x_{0}$ being $1$ always.

If you have all $\theta_{i}$ values, at inference time for getting prediction for each test example, you perform the equation above, but the objective of linear regression is to find the values of $\theta$. This is nothing but called as fitting the model.

And the equation for finding $\theta_{i}$ values is

$$\theta_{i} := \theta_{i} - \frac{\alpha}{n} \sum_{j=1}^{n}(h^{j}(x)- y^{j})x^{j}$$

You can observe that calculation of $\theta$ also has the first equation $h(x)$,

where $\alpha$ is the Learning rate.


Categorical Variables - Classification I have a categorical variable, country which takes on values like India, US, Pakistan etc. I am currently using a linear SLM for a classification task.

So my country value varies from 1-20. How should this be a feature in the classification task. Should i have a one hot vector like (1,0,0..) for us and assign this vector 20 weights, or should i have integer from 1_20 and assign a single weight? I am using scikit learn. Does the answer depend on classifier?

The answer depends less on the classifier and more on the nature of the variable. In your case One Hot Encoding might be the best answer.

Label Encoding (Replacing categorical variables with integers) is useful when the variable is ordinal, i.e. it has a sense of order. For example the days of the week or the months of the year. Since they follow a fixed order, you can encode January as 1 and February as 2 and so on. The classifier would interpret Feb as being greater than Jan in some way (which is okay for a task like weather prediction and so on).

Can your countries be considered to be ordinal? If not, One Hot Encode them.


A3C - Turning action probabilities into intensities I'm experimenting with using an A3C network to learn to play old Atari video games. My network outputs a set of probabilities for each possible action (e.g. left, right, shoot), and I use this information to determine which action to take in the game.

However I have gotten to thinking how would one go about playing a game with non-binary actions. For example, steering a car left or right with a wheel instead of keyboard keys. I thought about simply translating probabilities into intensities (e.g. if I have values of 1.0/0.0 for left right, then make the hardest left turn possible, but make a much more gradual turn if my values are 0.6/0.4) but I'm not sure if this makes sense mathematically, or even in practice.

Is there a standard approach to doing this?

In general, with Policy Gradients (PG) you can have two 'flavors' of them: Stochastic and Deterministic.

In your case, you can build a NN that outputs continuous actions or one that approximates the sufficient statistics of a probability distribution that you will sample the actions from. The main references are: Theoretical Proofs and NNs with DPG.

There are no rules on which one to use. There are many examples in the papers I cite that DPG outperforms SPG but also the opposite. Sometimes you really want the output to be deterministic. For example, in pricing: If a client comes in a store and you use SGD for assigning prices, the client will encounter every single time different prices (small differences but still differences).

In my personal experience I find it a bit hard to stabilize the DPG with NNs and you need also to come up with a good exploration strategy. However, once the network is stabilized you get the continuous action values that could control your vehicle. This is a detailed example of controlling a car with DPG using tensorflow and Keras.

For general theoretical exploration about the two frameworks I suggest you   this Master Thesis. It provides a concise overview of the two methods (no NN implementations). You can look up also the answer I gave here if you decide to get on board with PGs.


why is the model prediction output in keras lstm imdb, a vector? If you run the example of the LSTM sentiment classification task example in keras

  https://github.com/fchollet/keras/blob/master/examples/imdb_lstm.py

and add

p = model.predict(x_test[0])
print(p.shape)

you get:
(80,1)

.. why? I thought the point of the model was to classify each sample as 0 or 1 as per each of the y_train elements (binary classification)

x_test is of shape (25000, 80), that means 25000 samples with each sample corresponding to a sentence with a maximum of 80 tokens (see maxlen).
Accordingly x_test[0] is a vector of size 80.

model.predict does work on batches of samples, so you can call all model.predict(x_test) you will get a vector of n predictions where n is the number of test samples, i.e. 25000 in that case.
You could also predict the outcomes for just a part of the test set with slicing, e.g. model.predict(x_test[:10]) for the 10 first samples.

What happens when you are calling this function on just one sample x_test[0] is that the input is interpreted as a batch of 80 samples each with just one token.
(To the best of my knowledge, this "interpretation" happens somewhere down the function tree in keras.engine.training._standardize_inpupt_data.)
The model "doesn't know" the length of the input samples beforehand and thus doen't complain about these very short samples.
So you get 80 predictions for all these samples as if they were all sentences with just one token.

To get a prediction for the first sample, you could use model.predict(x_test[:1]). So you predict a batch containing only one sample and get the corresponding batch of one prediction.


Combining heterogeneous data sets for more powerful machine learning Suppose we have two data sets of movie reviews; one from IMDB and one from Rotten Tomatoes (RT). Each entry has a written-review and a score attached to it. The concatenated datasets might look like

Movie|Score|Review|Site

The Lion King|9.8|"This is one of the best movies ever!"|IMDB

The Lion King|4.5|"This move absolutely made my childhood"|RT

etc.

Our task is to predict scores from reviews. This is similar to the Kaggle IMDB sentiment analysis challenge, except we're trying to predict a continuous variable from two data sources. For a single data set, we might train a model in Keras with an embedding and lstm layer. In R:

model <- keras_model_sequential() %>%
   layer_embedding(input_dim = features, output_dim = 32) %>%
   layer_lstm(units = 32) %>%
   layer_dense(units = 1, activation = "linear")

where features is the number of words in our corpus.

How do we combine the IMDB and Rotten Tomatoes data sets to maximize our predictions?

My first thought is to include the site column as a categorical variable and then let the model learn how to best combine them. Would this cause a skew in the predictions?

We could also try multi-task learning, but most of these methods seem to be designed to predict repeated observations. Such as 5 different people reviewing the same 100 movies.

Some things to consider:

IMDB and RT don't have all the same movies (i.e. not a repeated observation problem)
There are words local to each dataset. i.e. the word "rotten" might be a frequent word on RT, but never appears in the IMDB dataset.
The outcome measures are different. RT uses a 5 star scale, whereas IMDB uses a 10 star one.

How do we deal with this heterogeneity?

Following your considerations:

  IMDB and RT don't have all the same movies

I would build a larger dataset of movies, aggregate all observations available, and run a model on that.

  There are words local to each dataset. i.e. the word "rotten" might be
  a frequent word on RT, but never appears in the IMDB dataset.

I think this doesn't constitute a problem once you aggregate data in a larger dataset. At that point you can run some NLP method on the whole aggregated corpus. Let me know what you think.

  The outcome measures are different. RT uses a 5 star scale, whereas
  IMDB uses a 10 star one. How do we deal with this heterogeneity?

You just need to uniform the scales on a given interval like [0, 1]. Then all votes could be compared.


Possible Challenges for a Data Science Escape Room Dear Data Science Community,

as my project for my bachelorthesis I am working on a concept for a physical Data Science Escape Room. The goal of the escape room is to create awareness for data science / the importance of data in general and motivate "escapees" to learn more about data science.

The escape room is going to be a physical room, that needs to be escaped by completing different challenges related to data science. The target group are people who are already working with data in form of reports or similar, but have no or little to none background in data science.

Possible challenges for this escape room could use simple machine learning algorithms that need to be trained, or easy statistical concepts. Players could also need to curate data, which then needs to be integrated to be usable afterwards.

I am looking for ideas or further input for possible challenges for this escape room, and was hoping to have some data science experts give creative input!

Thank you!

Fun! Cool idea, but it seems like a big undertaking. Anyways, here is my suggestion:

Summary

A manual fit of a 2D logistic regression which only becomes solvable once some previous step is completed.

Explaination

I imagine being greeted by a screen looking something like this:

The two boxes are inputs where the escapees can input numbers. If they try inputting at random it will be possible to figure out that the numbers represent $B_0$ and $B_1$ in: $$Y = B_0 + B_1 X1$$

This will still leave them confused since they will only see something like this:

However, when some other challenge in the room is finished the real challenge is revealed by colorizing the dots.

Then they just need to find the proper intercept and coefficient to finish challenge.

I think this would work quite well if there are enough samples so that you won't be able to hack it by trying values before they know the colors. If the challenge is too easy you could make them fit polynomials instead.


Evaluating performance of classifier on lopsided dataset I have a binary classifier that I would like to evaluate the performance of. It's been both trained and tested on a data set where the ratio of true to false labels is lopsided. This means that while it's quite poor at correctly guessing true, its overall performance on the test set looks very good when using a metric such as right_guesses/total.

What is a better metric to use? Preferably, one where the true false labels account for the same percentage of the score although their numbers are unequal.

sklearn has weighted accuracy score which works just fine:

sklearn.metrics.balanced_accuracy_score()


Find value and return row in excel spreadsheet with over 1 mil rows I have an excel spreadsheet that has over 1 million rows and about 120 columns. I frequently need to find a value or a part of a value (value is name or number) and return the row where the value is found. I only need values from about 20 columns so I could drop out the extra columns but that does not solve the problem of excel being very slow in finding the value and it is very difficult to read information in the row.

What would you recommend to use for this task? Ideally I would like to enter the value and get the row in some sort of a report but not necessary as long as it is well readable.

I was thinking of using Power BI but got kind of stuck and in the end felt that it wasn't the right tool for the task.

I am open for all suggestions!

the problem of excel being very slow in finding the value and it is
  very difficult to read information in the row

What is "slow" for you (in seconds) and how fast would you like the search to be?

In a lot of cases, the performance can be considerably improved by using INDEX and MATCH to search for values in Excel, instead of using VLOOKUP (which is known to be much slower in comparison).

Alternatively, you could install SQL Server Express on your machine, import the Excel file into a table, and use some SQL statements to find the values you want.

Power BI is a visualization tool, and doesn't seem appropriate for what you're describing which is preparing the data. You could use Power BI after, of course, to plot the final results.


Can we use GAEs with A3C asynchronous reinforcement learning? The Generalized Advantage Estimator seems very effective with algorithms like PPO in reinforcement learning. But most of the A3C algorithms I have seen use Averaged Advantage estimates. Is there any reason?

Sure can:

https://github.com/ikostrikov/pytorch-a3c

The reason is the GAE paper was released after A3C, so the paper implementation, which is what most people will look at, doesn't use GAE.

Relevant lines here: https://github.com/ikostrikov/pytorch-a3c/blob/master/train.py#L95-L98


How to make a neural network generalizes better? I designed a neural network model with large number of output predicted by softmax function. However, I want categorize all the outputs into 5 outputs without modifying the architecture of other layers. The model performs well in the first case but when I decrease the number of output it loses accuracy and get a bad generalization. My question is: Is there a method to make my model performs well even if there is just 5 outputs? for example: adding dropout layer before output layer, using other activation function, etc.

First make sure your dataset is labelled properly in 5 distinct classes from 0 to 4 and no more.
Then add dropout layer in between each layer with prob of 0.1 and gradually increase till 0.5 until you find  a good val score.
In your optimizer add weight decay term with value around 0.1.
Find some appropriate regularizer for your data type online.


Understanding Youtube recommender (candidate generation step) I'm trying to understand Deep Neural Networks for YouTube Recommendations.
Their candidate generation step outputs top N items

via softmax (with negative sampling) at training time .
via nearestneighbor at serving time.

I guess $v_j$ represents, (from softmax layer to nearest neighbor index)
topn videos you get via softmax, and represent them in the original encoding (same encoding you used for the input (used for embedded video watches))
apparently, $v_j$ are in the different encoding from the input encodings.
The softmax layer outputs a multinomial distribution over the same 1M video classes with a dimension of 256 (which can be thought of as a separate output video embedding)
I'm trying to understand what they mean by interpreting softmax output as a separate output video embedding.  I thought softmax layer that outputs 1M classes has dimension of 1M, where does 256 came from?  (It's the same question as How to create a multi-dimensional softmax output in Tensorflow? and I don't think it has been answered there..)

user vector $u$ is the output of the final ReLU unit, although I'm not sure what this user vector is used for.

I guess in serving time, to pick the top N for a given user, user vector $u$ is used by nearest-neighbor. But my understanding of nearest-neighbor is for a given vector, it finds nearest vectors in the same dimension. (such as given an movie, find nearest movies). However here, you are given a user and need to find topn videos. How does that work?
My best guess is that, for a given user, u get a user vector as the ReLU output, then find user-user nearest neighbor, and combine their topn items obtained in the training time. But it's just a guess..

$v_j$ is the learned vector of weights (dimension 256) that connect the last hidden layer (dimension 256) to the corresponding output node for video class $j$. The last hidden layer is the user embedding vector $u$.

The paper uses a vocabulary $V$ of 1M video classes so the deep neural net learns 1M vectors of weights that connect the last hidden layer to each class: $[v_1, \ldots, v_{1M}]$. Negative sampling is used to train such a model with so many classes.

Below softmax function on page 2 section 3.1 with vj and $u$.

$$P(w_t=i|U,C) = \frac{e^{v_i u}}{\sum_{j \in V e^{v_{j}u}}}$$

I believe $v_j$ is "thought of as a separate output video embedding" because $v_j$ can be interpreted as a compressed representation of video j in a 256 dimensional vector space.

Since $e^x$ is monotonically increasing, we just care about the dot product $v_iu$ in the numerator of equation above. This is why "the scoring problem reduces to a nearest neighbor search in the dot product space" where we simply find the nearest neighbors $v$ to $u$.


r"""What does it mean?""" I see r""" followed by a comment in quite a few of the source codes for PyTorch and the one I am looking at is here. What is the r for?

An r in front of a string makes it a raw string literal. In short it means that any \ in that string will not be seen as an escape character, like when you write \n for newline.

Some more info can be found here:
What exactly do “u” and “r” string flags do, and what are raw string literals?


Collection Of Variable Length Sequences and Descriptions: A Search Problem I have a tough problem and need some advice:

Suppose I have a collection of variable length sequences, many of which are unique -- imagine the moves to a chess game, eg

  d4 Nf6
  c4 g6
  Nc3 Bg7
  e4 d6
  Nf3 O-O
  Be2 c5
  O-O Bg4

...

and for each item in this collection, I have another collection of descriptions generated by humans (think comments - comment_1: "cool game", comment_2: "awesome sacrifice")

The goal is to mine the associations between the comments and the sequences to tag the sequences with human readable labels for search purposes.

I've thought about topic modeling for label generation + clustering/grouping the sequences, but I cannot figure out how to do something like cluster the games. I have millions of examples of the sequences if that helps. Any idea how I can measure the distance/similarity between sequences like this? Some kind of embedding? I've thought about trying a word2vec / doc2vec approach, but haven't tested yet.

Ideally I'd be able to input an unseen sequence and suggest labels/human-readable description for this sequence as well.

While an RNN using one-hot encoded moves is possible, I would suggest that your model needs to understand chess (or similar complex games) at a deeper level to be able to associate comments to positions.

I would encode the position itself (eg a layered representation like in Alpha-Zero paper), and pass those through a conv-RNN to model the temporal relationship between positions corresponding to annotations. (Maybe you can find a pretrained model for evaluation as a baseline).

I read a fair amount of chess commentary, and one thing that is common is that there are cases of single move as well as comments about several moves at a time (think opening lines, or forced continuations), which are fairly discrete (my 'move 10 sacrifice' and my 'move 15 blunder' are not related by a hidden state). This leads me to think that instead of an RNN model, you are really trying to first Partition then Embed moves in a comment space - which then leads to a model that is first an RNN which may learn likely partitions of moves to be commented on, then another model, perhaps conditional autoencoder, is trained jointly on (position | comment) subsequences (you should still have plenty of features to work with at this point, so no worry about a weak signal).

The 'search' functionality here points not at a pure deep learning model, since you want to find specific instances of positions, rather than a typical position with that comment (ie sampling a latent space). This leads me to think that you create an explicit data-structure to map points in your final embedding space to their positions. So if you wanted to search 'Nice Sacrifice', model would find closest points in 'comment space' and return those games using something like cosine distance, or this last step is clustering of the (position | comment) space.

Either way, interesting problem, try the simpler thing first.