Is there any activation function that can make batch normalization obsolete? While learning batch normalization, I was thinking why can't we solve the "gradient scale problem" by using an appropriate activation function ?

Like can't we delay and scale the activation function instead of scaling the whole dataset and ensure that the variance is preserve through it ?

What you describe sounds a lot like Scaled-Exponential Linear Units (SELUs), that are the core of Self-Normalizing Neural Networks, which where presented at NIPS 2017.

A short summary from here is that:

  If the mean and variance of the input is in certain range, then the mean and variance of the output should (1) also in that range and (2) converge to a fixed point after iteratively applying the activation function.

You might want to have a look at the reddit post comments. If you want to fully understand them, you can go ahead with the 90 page-long appendix of the arxiv preprint.

They got a lot of attention when they were presented, but I think they have not delivered up to the expectations, as no one seems to be talking about them lately on the internet.


Data quality improvement as a part of preprocessing: Imputation I have a python pandas dataframe representing a superset. The data contains a lot of nulls which I want to overwrite with real values.

the superset has:

both numerical and categorical data
some nulls for most attributes
multi class attributes (attributes can have multiple values)
It is not time dependent
each row is a unique person

It would be neat to use machine learning to fill in the nulls, any recommendations on how I can do this?

(I guess that I can tranform the categorical data to numerical if required)

Firstly, filling a column which has most values as null might not make sense. You'll have to look at the nature of the business problem you face. You can try dropping those columns. You can drop the rows which contain null values if the number of null values for columns in that row fall below a particular threshold (based on your data, of course).
Use: dataset.dropna(inplace=True) to achieve this

That being said, there are multiple ways you can fill (impute) null values:

You can try filling values based on the mean (or median or mode): dataset.fillna(dataset.mean(), inplace=True). However this approach has it's limitations if your column is non-numeric.

If you want to use an sklearn implementation, try using the Imputer method.

from sklearn.preprocessing import Imputer

impute = Imputer(missing_values='NaN', strategy='mean', axis=0)

impute.fit(dataset)

dataset = impute.transform(dataset)

For categorical columns, try using fillna with the most frequent value in a column:

dataset[column].fillna(value=df['column1'].value_counts().index[0], inplace =True)

more information on this method can be found here: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.fillna.html

Try using a KNN to fill in missing values. There's a package called fancyimpute which will allow you to do this:

from fancyimpute import KNN

dataset = pd.DataFrame(KNN(number_of_neighbors).complete(dataset))

You'll need to choose the number of neighbors appropriately. Also, fancyimpute needs a numpy array as input.

There are many other methods, but these should do the trick.


How to impute right-censored data I have a dataset of vectors representing movement with various characteristics. Some vectors represents the movement that was stopped by external factor and therefore, observed value for length of such a vector  (v_length) is incomplete (marked as incomplete == 1). The data looks like below:

# A tibble: 10 x 9
   v_length incomplete v_angle    x0    y0    x1    y1 type    vap
      <dbl>      <dbl>   <dbl> <dbl> <dbl> <dbl> <dbl> <chr> <dbl>
 1     1.70          1   0.869  66.6   0.5  67.7   1.8 A         0
 2     1.82          1  -0.165  37.4  65.6  39.2  65.3 B         0
 3     2.57          1   0.236  61.3  49.7  58.8  49.1 A         0
 4     3.14          1   1.18   57.8  40.5  59    43.4 A         0
 5    12.6           0   0.119  52.5  33.7  65    35.2 A         0
 6    20.5           0  -0.847  65.3  32.3  78.9  16.9 A         0
 7    33.0           0  -0.180  77.5  13.7  45    19.6 A         0
 8    14.1           0  -0.780  45    19.6  35    29.5 B         0
 9     2.97          0   1.00   35    29.5  33.4  27   B         0
10     6.59          0   0.732  33.4  27    38.3  31.4 A         0

I want to impute a v_length for incomplete observations (incomplete==1). My first idea was to use some parametric survival model (e.g. Weibull). But as I'm not experienced in Survival analysis I have been struggling with a good setup. My first doubt is if it is correct to use v_length as one of the predictors as well? It doesn't make sense at first sight, but the predictions for the model without v_length as one of predictors looks very strange:

The idea behind inclusion is to help the model know what was the observed vector length, so it can predict a value higher than that. After inclusion of v_length in predictors the output looks like below:

However, we still have plenty of values lower than actual vector length, while I obviously don't want a model to predict a lower value than observed.

So here's my question: is Weibull survival model suitable for this task? What's the correct setup if so? What are the other methods suitable for imputation of right-censored data?

You can't put v_length in the regression - that'd be a form of data leakage. However, you are right to be thinking about "how to tell the model that I've already observed some length". This can be accomplished with some survival analysis math. For censored observations, what you want is

$S(l \;|\; l > \text{observed v_length})= P(L > l \;|\; L > \text{observed v_length})$.

Let's analyze this:

$P(L > l \;|\; L > \text{observed v_length}) = \frac{P(L > l \;\text{and}\; L > \text{observed v_length})} {P(l > \text{observed v_length})}$

$\;\;\;\;\;\;=\frac{P(L > l)} {P(L > \text{observed v_length})} = \frac{S(l)}{S(\text{observed v_length})} $

This gives you a new survival curve that you can use to do imputation (take the median or mean of the new survival curve).

In python's lifelines package, this calculation is done behind the scenes when the conditional_after argument is used in the prediction methods, see docs here. I'm not sure if R packages have something like this.

It also doesn't make sense to plot the observed vs predicted, because some observed values are truncated (censored) - hence a difference in observed and predicted could be due to censoring rather than bad prediction. For example, using your values above, I may plot the point (1.70, 25.0). The 1.70 is censored, and the 25.0 is the predicted value. There is a big difference between these values, but we don't know if that difference is just because of the censoring or if our prediction is just really off.

My advice would be to focus on finding a good model that maximizes the out-of-sample likelihood or AIC (Why likelihood? It's a good measure of survival fit).


Does it make sense to normalize? My data consists of multiple NHL seasons where the total number of goals scored by each player is kept track.

I intend to compare the distribution of goals scored between each season.

My thinking is that I need to normalize goals scored before I can properly compare distributions since not every season has had the same number of games played.

Is it in fact necessary to normalize the data in order to fairly compare the distributions?

First, let's look at the term normalization.

  In statistics and applications of statistics, normalization can have a range of meanings. In the simplest cases, normalization of ratings means adjusting values measured on different scales to a notionally common scale, often prior to averaging. In more complicated cases, normalization may refer to more sophisticated adjustments where the intention is to bring the entire probability distributions of adjusted values into alignment.

Your NHL goals example probably does not need "adjusting values measured on different scales to a notionally common scale."

Think of the handful of players who had a high-scoring beginning of the season, and then was injured and sat out the majority of games. Would you want to normalize his goals up proportionally to the rest of the league?


Cheat Sheet of UNIX commands for Data Science I was looking for a cheat sheet of UNIX commands, which are specifically usable for data science. I mean an introduction to very basic commands (starting from cd, ls, pwd, to some still simple but usable for data - e.g. wc, a few simple things with pipes, ssh, maybe s3cmd, etc), with some minimalistic examples.

I couldn't find one; or at least, nothing close to:

Git Cheat Sheet
Regular Expressions Cheat Sheet
series of R Cheat Sheets

I am pretty much aware of datascienceatthecommandline.com, but my goal would be to have a 1-2 A4 pages printout, with really simple stuff.

If there is one (but I missed it), I would be grateful for posting it as an answer. If there isn't, then I am asking to post pieces of such list.

(If Moderator permits, we may create one list with Community Wiki.)

EDIT: The closest thing I've found is here: Useful Unix commands for data science (very nice, but missing the very introductory part).

I would recommend this one - http://www.cheat-sheets.org/saved-copy/ubunturef.pdf

Its comprehensive, but not data science specific.

There are a number of great cheat sheets here that may be what you are looking for:https://dzone.com/refcardz

If you are wanting to create your own, this is a very nice tool: http://www.cheatography.com/


Keras Neural Network training is stuck (gets stuck around epoch 6) I have a training dataset with 8000 rows, and I am trying to train a Keras Neural Network on it, using 100 epochs. However, the training process gets stuck around epoch 6 every time, as shown below. I'm not sure if it's because of my computer (8GB RAM Macbook Pro) or because of some inappropriately set parameters for my model. Thanks!

    import keras
    from keras.models import Sequential
    from keras.layers import Dense

    classifier = Sequential()

    classifier.add(Dense(activation="relu", input_dim=11, units=6, kernel_initializer="uniform"))

    classifier.add(Dense(activation="relu", units=6, kernel_initializer="uniform"))

    classifier.add(Dense(activation="sigmoid", units=1, kernel_initializer="uniform"))

    classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])

    classifier.fit(X_train, y_train, batch_size = 10, nb_epoch = 100)

8000/8000 [==============================] - 3s 348us/step - loss: 0.3890 - acc: 0.8367
Epoch 2/100
8000/8000 [==============================] - 3s 329us/step - loss: 0.3894 - acc: 0.8376
Epoch 3/100
8000/8000 [==============================] - 3s 314us/step - loss: 0.3887 - acc: 0.8370
Epoch 4/100
8000/8000 [==============================] - 3s 346us/step - loss: 0.3895 - acc: 0.8379
Epoch 5/100
8000/8000 [==============================] - 3s 320us/step - loss: 0.3887 - acc: 0.8370 - ETA: 2s - loss: 0.4087 - acc: 0.8077
Epoch 6/100
7650/8000 [===========================>..] - ETA: 0s - loss: 0.3881 - acc: 0.8363
//STUCK HERE

I do get the following message on my terminal window though (even though I'm using jupyter notebook):

2017-12-24 20:45:30.464660: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.

I had a similar problem. I guess the reason is incompatibility of Keras with your current anaconda. Just update it.


Input layer is incompatible even when dimensions (apparently) match I am making a sequential neural network for classification, with 3 dense layers, which will be trained on a simple synthetic dataset. The description of dataset is as follows:

Data and class labels are integers. They are 2000 each.
There is only a single feature column (populated by np.arange(2000) * 3)
There is only a single label which indicates last digit of number (populated by np.arange(2000) *3 % 10).

After making the model, I am encountering the following error when calling model.fit():

ValueError: Input 0 of layer sequential is incompatible with the
layer: expected axis -1 of input shape to have value 1500 but received
input with shape (100, 1)

I have uploaded the commented Jupyter Notebook for this code on Google Collab: https://colab.research.google.com/drive/14v92NTBxIEIFJh2BhybfqhawHYIBvKnm?usp=sharing
Any suggestion about how to fix this error and get reasonable accuracy on training set?

You set the input shape to (1500, 2) whereas your data only contains a single feature. You should therefore change the shape to (1,) or (None, 1) to match the shape of the input data.


Can someone explain the lambda collocation metric? I'm trying to understand the lambda collocation scoring metric from the quanteda package in R, but the explanation in the documentation is incredibly difficult to understand.

Can anyone explain it better?

Having an example worked out on a toy set of text would help too, like something with 5-10 words.  This line (and the one right before it) is especially confusing:

  The ni are the counts in a 2^K contingency table whose dimensions are defined by the ci.

I will try to illustrate the lambda collocation metric, first we have to define a function c(x, z) that receives two K-word expressions where x is the K-word expression you want to score and z corresponds to all possible values of a K-word expression in the corpus. c(x, z) returns a binary vector (j1, ..., jk) such that ji = 1 if xi == zi, 0 otherwise. Now a binary vector can represent a decimal number up to  2^K = M, so c(x, z) = ci means that the binary vector returned was equal to the binary representation of the decimal number i. Now as it says on the documentation ni corresponds to the number of times the c(x, z) function was equal to ci through the corpus. Now an example, consider the following sentence:

  this sentence is an example for this sentence

The target expression is "this sentence" so K=2. You have the following possible values for z:

this sentence
sentence is
an example
example for
for this
this in
in this
this sentence

Given that K=2, the possible values for c(x, z) are 00 = 0, 01 = 1, 10 = 2 and 11 = 3. Then considering the target expression and all the values of z above you have the following, (for x = "this sentence"):

c(x, this sentence) = 11 = c3
c(x, sentence is) = 00 = c0
c(x, an example) = 00 = c0
c(x, example for) = 00 = c0
c(x, for this) = 00 = c0
c(x, this in) = 10 = c2
c(x, in this) = 00 = c0
c(x, this sentence) = 11 = c3

So we have 5 times c0 which means n0 = 5, 2 times c3 means n3 = 2 and n2 = 1. Finally bi is the number of bits equals to 1 in c1 which b0 = 0, b1 = 1, b2 = 1, b3 = 2. This completes all the values you need for computing the lambda collocation scoring metric. Also take into account you could add smoothing to ni to avoid 0 counts. Now with respect to

  The ni are the counts in a 2^K contingency table whose dimensions are defined by the ci.

The definition of a contingency table for the array data is simply the number of times a given value appears (i.e. its frequency). See this and the usage of the table command for more information. For the example above we have the following array data:

c3 c0 c0 c0 c0 c2 c0 c3

So the corresponding contingency table is:

c0 | c1 | c2 | c3
-----------------
 5 |  0 |  1 |  2


How to determine best parameters after grid search? I'm optimizing the parameters for a single layer MLP. I've chosen to vary 4 parameters: hidden layer size, tolerance, activation, and regularization weights. Each of these has 4 possible values it can take (4^4 = 256 combinations).

So the question is, how does one determine that a set of parameters are statistically significantly better than another?

My stats is a little rusty, but my first thought was n-way ANOVA with 4 factors and 4 degrees of freedom in each factor. Is there something better?

I too was in your place when I started using Neural networks. There are so many hyper-parameters to choose, and each take on many values. Like Emre said, you need to check the model which is giving best metric score on your data (Cross validation set). The parameter values of that model will be your optimized values. You can also check this link


How to find optimized x values (input features) after training in deep learning? I did deep learning training by Keras.

I have done the training part by model.fit If I do model.predict, it only gives me y value. But I want to know x (input features) that gives the best y value, which means the optimized x values after training.

Is there any way to fetch the optimized x values (input features) in Keras or else?

I really appreciate your help.

Thank you.

If I understand your question correctly, you want to find the best features to optimize your model's performance?

I think what you're looking for is feature selection, which is an entire subject within machine learning/statistics. There is a whole tag on here about feature selection, you'd be better off asking your question with that. I would poke around a little online to learn more about feature selection, and if after that you still want help here you should provide more information about the problem you're trying to solve, the data you're working with, and more details about your model.


Grouping/clustering similar words python I have a question regarding grouping of similar words for example I have list of words give below:

artificialintelligence
Artificial Intelligence
AI
Machine Learning
ML
Data Analytics
Data & Analytics

I want to group these words into [Artificial intelligence, machine Learning, Data Analytics]

I have used difflib.get_close_matches() but that does not give me desired result
For example this is how difflib group:
Information Technology': ['Information Technology','Mobile Technology',
  'newtechnology']

I have also used fuzz.token_set_ratio() but that also does not provide me the desired result.  Neither Levenshtein does.

If there is any Machine learning algorithm or any python library please let me know.

Thank you

You won't be able to just solve this with an algorithm.

Instead you will need to solve this more data, about language and abbreviations. This is also known as world models background data etc.


What is a simple image classification ANN architecture and a simple data set? I'm looking for a neural network architecture, which gives pretty good performance on an image classification challenge and is still simple to implement in a framework and also reasonably fast to train. Do you have suggestions?

I'm new the neural network implementation, and in the end, I'd like to do some "research-style" playing with it.

Moreover, a pointer to an image classification data set which is not too tough for a simple network would be great. However, not something like MNIST which is so easy that almost any classifier does a good job.

PS: Which neural network framework do you recommend if I'd like to modify the way the network is trained?

I'd advice going through cs231n course, which covers this.

A simple but good architecture is using a feed forward net with one hidden layer and ReLU.

As for dataset - CIFAR-10 is good, if you don't like MNIST.

Tensorflow is quite easy to use framework.


Convolution layer dimensions in deeper layers? I am trying to understand the CNN network dimensions:

layer                   activation shape    number of weights

Input                   (25, 64, 1)         0
Conv2D(7x7, 1, 100)     (25, 64, 100)       4.900 (7x7x100)
MaxPool2D               (13, 32, 100)       0
Conv2D(5x5, 1, 150)     (13, 32, 150)       375.000 (5x5x150x100)
...

The input dimension is 25x64, then first Conv layer applies 100 convolutions - so the output dimension is 25x64x100. Max pooling reduces this to 13x32x100.

Second Conv layer applies 150 convolutions. What's unclear to me is the dimension after the second Conv layer. Shouldn't it be 13x32x150x100 (instead of just 13x32x150)? How is the convolution layer applied on a 3D input?

Number of weights for the first Conv layer is 7x7x100, and for the second layer it is 5x5x150x100, meaning that weights are saved for each of the 100 input layers from the previous conv. This also leads me to think that output dimension should be 4D.

The output shapes for convolutional layers are calculated in the following way:

Let's say that the input shape for some convolutional layer is WxHxC:

W - width
H - height
C - channels

Now, assume that you have only one convolutional kernel, eg. size 5x5 (width and height). That kernel will actually be of size 5x5xC (C is the number of channels in the input shape) because one kernel must multiply all channels in the input when it is fixed in some place of the input. As you know, for that one fixed position, you get only one output number. When you repeat this for all input positions, you get only one output feature map (ie. with shape WxHx1 assuming that you keep the input dimensions by using padding ...).

In order to get more feature maps on the output, you need to repeat the process with new convolutional kernels (all those kernels must have the channel number equal to input shape channels). So, if you repeat this K times, your output feature map will have dimensions WcHxK. And the size of the kernel for that convolutional layer is said to be 5x5xCxK.

You can consult the following image:

In the image, one convolutional kernel (in the image it's called "Convnet Filter") outputs one output feature map (in the image it's called "One Feature Map").

Hence, in your case, for the first convolutional layer, the size of kernel is 7x7x1x100, and the output shape is 25x64x100. For the second convolutional layer, kernel size is 5x5x100x150 and the output shape is 13x32x150.

The above-described convolution is the "classic" convolution. There are other types of convolution, eg. Depthwise separable convolution, which use different ways of calculation. Consult A Basic Introduction to Separable Convolutions tutorial for more information about different types of convultion (it includes also descrption of "classic" convolution in the "Normal Convolution" section).


Tips on how to create box whisker plot for huge data set I have a huge dataset (about 2 million lines) that I want to visualize to have an idea of how spread the data is. The problem now is when I create a box whisker plot the resulting graph is not legible due to the huge amount of data.
Is there any trick to be able to successfully create a box whisker plot for huge data set in the way that is readable?

Hope you are aware of data sampling methods. They exist to solve problems such as these. They are of several types like;
Probability Sampling Methods

Simple random sampling
Systematic sampling
Stratified sampling
Clustered sampling

Non-Probability Sampling Methods

Convenience sampling
Quota sampling
Judgement (or Purposive) Sampling
Snowball sampling

When sampling the data, be careful of the bias.
Bias in sampling
There are five important potential sources of bias that should be considered when selecting a sample, irrespective of the method used. Sampling bias may be introduced when:

Any pre-agreed sampling rules are deviated from
People in hard-to-reach groups are omitted
Selected individuals are replaced with others, for example if they are difficult to contact
There are low response rates
An out-of-date list is used as the sample frame (for example, if it excludes people who have recently moved to an area)

Basis of the above detail, I think there is no need to use the entire dataset for visualization purpose. Consider an example, each country conducts a census survey to estimate its population of people. Such a dataset is huge both in size and in complexity. Do you think those statisticians use the complete dataset for visualization? I strongly doubt it. They use sampling methods.
Edit
Python Code
# load required libraries
import pandas as pd
# create some dummy data
df = pd.DataFrame({'num_legs': [2, 4, 8, 0],
                   'num_wings': [2, 0, 0, 0],
                   'num_specimen_seen': [10, 2, 1, 8]},
                  index=['falcon', 'dog', 'spider', 'fish'])
print("## Original data ##\n",df)

## Original data ##
         num_legs  num_wings  num_specimen_seen
falcon         2          2                 10
dog            4          0                  2
spider         8          0                  1
fish           0          0                  8

Sampling methods:

Simple random sampling: Extract 3 random rows

    print("\n Simple random sample")
    print(df.sample(3, random_state=10))
    Simple random sample
            num_legs  num_wings  num_specimen_seen
    spider         8          0                  1
    falcon         2          2                 10
    fish           0          0                  8

1.1. Simple random sampling: Extract 3 random elements from the Series
df['num_legs']
print("\n Simple random sample of particular column\n",df['num_legs'].sample(n=3, random_state=10))

Simple random sample of particular column
spider    8
falcon    2
fish      0
Name: num_legs, dtype: int64

1.2. Random sampling with replacement
print("\nA random 50% sample of the DataFrame with replacement:")
print(df.sample(frac=0.5, replace=True, random_state=10))
A random 50% sample of the DataFrame with replacement:
     num_legs  num_wings  num_specimen_seen
dog         4          0                  2
dog         4          0                  2

1.3. Random upsampling with replacement
print("\nAn upsample sample of the DataFrame with replacement")
print(df.sample(frac=2, replace=True, random_state=10))
An upsample sample of the DataFrame with replacement
        num_legs  num_wings  num_specimen_seen
dog            4          0                  2
dog            4          0                  2
falcon         2          2                 10
fish           0          0                  8
falcon         2          2                 10
dog            4          0                  2
fish           0          0                  8
falcon         2          2                 10

1.4. Random sampling with weights
print("\nUsing a DataFrame column as weights. Rows with larger value in the num_specimen_seen column are more likely to be sampled.")
print(df.sample(n=2, weights='num_specimen_seen', random_state=10))
# Using a DataFrame column as weights. Rows with larger value in the num_specimen_seen column are more likely to be sampled.
        num_legs  num_wings  num_specimen_seen
fish           0          0                  8
falcon         2          2                 10

Stratified Random Sampling

    print("\nStratified Random Sampling")
    print(df.groupby('num_legs', group_keys=False).apply(lambda x:x.sample(min(len(x), 2))))

Stratified Random Sampling

        num_legs  num_wings  num_specimen_seen
fish           0          0                  8
falcon         2          2                 10
dog            4          0                  2
spider         8          0                  1

This brief example should get you started!


Troubleshooting Neural Network Implementation I've been going through the Standford/Coursera Machine Learning course; and it's been going pretty well. I'm really more interested in the understanding of the topic than getting the grade from the course and as such I'm trying to write all the code in a programming language I'm more fluent in (something I can easily dig down into the roots of).
The way I learn best is via working through problems, so I've implemented a neural network and it doesn't work. I seem to get the same probability of each class irrespective of the test example (for example 0.45 of class 0, 0.55 of class 1, irrespective of the input values). Strangely this isn't the case if I remove all the hidden layers.
Here's a brief run through of what I do;
Set all Theta's (weights) to a small random number

for each training example

set activation 0 on layer 0 as 1 (bias)
set layer 1 activations = inputs

forward propagate;
 Z(j+1) = Theta(j) x activation(j)  [matrix operations]
 activation(j+1) = Sigmoid function (Z(j+1)) [element wise sigmoid]

Set Hx = final layer activations

Set bias of each layer (activation 0,0) = 1

[back propagate]
calculate delta;
delta(last layer) = activation(last layer) - Y  [Y is the expected answer from training set]

delta(j) = transpose(Theta(j)) x delta(j+1) .* (activation(j) .*(Ones - activation(j))

[where ones is a matrix of 1's in every cell; and .* is the element wise multiplication]
[Don't calculate delta(0) since there ins't one for input layer]

DeltaCap(j) = DeltaCap(j) + delta(j+1) x transpose(activation(j))

Next [End for]

Calculate D;

D(j) = 1/#Training * DeltaCap(j) (for j = 0)

D(j) = 1/#Training * DeltaCap(j) + Lambda/#Training * Theta(j) (for j = 0)

[calculate cost function]

J(theta) = -1/#training * Y*Log(Hx) + (1-Y)*log(1-Hx) + lambda/ (2 * #training) * theta^2

Recalculate Theta

Theta = Theta - alpha * D

That's probably not a great deal to go on. If someone can tell me if there's any major flaw in my code that would be fantastic, otherwise some general idea of where I might be going wrong/how to debug something like that that would also be great.
EDIT:
Here's a quick image of the network (including a test case of inputs and responses) (this is after 1 million iterations of gradient descent);

The data set I've used is two exam scores as the x's and the success/failure of getting into a university as the y. Clearly two test scores of 0 would mean failure in getting into university however the network suggests 56% chance of getting it with 0's as inputs.
Edit #2;
I've ran a gradient checking algorithm with the following sort of results;
Numerical calculation: -0.0074962585205895493
Value from propagation: 0.62021047431540277
Numerical calculation: 0.0032635827218463476
Value from propagation: -0.39564819922432665
etc. Clearly there's something wrong here; I'm going to work through it.

Are your inputs scaled? Not doing so can cause the weights to immediately blow up.

It's common practice to preprocess your input to be between -1 and 1 or at least in that range. Otherwise you are running the risk of your gradients exploding or vanishing (covered here).


KNN scoring low compared to Logistic regression in MNIST challenge KNN gives me a score of 0.76100 while it shows 94% accuracy for my training data (splitted with test_size =0.3) in my jupyter notebook while logistic regression gives me a score of 0.91485 with an accuracy of 92 %.I do not understand the reason. Please can anybody help me or suggest anything.Here is my model:

   from sklearn.neighbors import KNeighborsClassifier
   knn=KNeighborsClassifier(n_neighbors=1)
   knn.fit(X_train,y_train)
   pred = knn.predict(X_test)

below is the classification report :

                    precision    recall  f1-score   support

               0       0.94      0.99      0.96      1213
               1       0.95      0.99      0.97      1422
               2       0.95      0.92      0.93      1258
               3       0.92      0.94      0.93      1284
               4       0.93      0.94      0.94      1209
               5       0.93      0.91      0.92      1121
               6       0.96      0.97      0.97      1242
               7       0.93      0.93      0.93      1315
               8       0.97      0.89      0.93      1227
               9       0.91      0.91      0.91      1309

        accuracy                           0.94     12600
       macro avg       0.94      0.94      0.94     12600
    weighted avg       0.94      0.94      0.94     12600

Confusion matrix :

[[1190    2    2    4    1    2   10    0    0    2]
 [   2 1404    7    3    2    1    2    0    1    0]
 [  14   11 1157   25    8    0   10   15   11    7]
 [   0    1   19 1191    1   34    1   18   12    7]
 [   2    8    6    0 1125    2    4   11    3   48]
 [   2    2    3   36    5 1030   16    2   15   10]
 [  18    4    4    1    4    9 1200    0    2    0]
 [   1    8    7    6   10    1    0 1224    3   55]
 [   8   14    8   30   13   31    9    6 1093   15]
 [   4    2    4    6   38    6    0   51    8 1190]]

Thanks in advance

If you are referring score as test accuracy, your knn is overfitting a lot because of 1 nearest neighbour. Please do it with high k values and also try to do cross validation to get best model.


Problem in constructing co-occurence matrix I have a basic doubt in construction of co-occurence matrix from text. Coocurence matrix X is defined as :

Xij : number of times word i occurs in context of word j in a window of size w.

Now is this context limited to sentence i.e do we context word of a main word in a sentence only or across the whole document?

For ex:

Consider the corpus :

human interface computer
survey user computer system response time
eps user interface system
system human system eps
user response time
trees
graph trees
graph minors trees
graph minors survey
I like graph and stuff
I like trees and stuff
Sometimes I build a graph
Sometimes I build trees

Now for word graph and context window 10 do we look for only 4th last line or do we search for words across line also considering whole doc as one line?

Whether to window only over sentences or over the whole corpus depends on the context of your problem and on the structure of your corpus. In this corpus the order of your sentences does not matter, if they were switched around it would still be the same corpus. This means that you should stick to only the lines and not see it as one line. If your corpus would be a book for example, spilling over sentences makes a lot more sense. If you are unsure, try both, it is just a hyperparameter of your model.


How predictions of level 1 models become training set of a new model in stacked generalization. In stacked generalization, if I understood well, we divide the training set into train/test set. We use train set to train M models, and make predictions on test set. Then we use the predictions as input of a new model. Thus, the new training set will have M features corresponding to the M models predictions. Finally, we use the last model to make the final predictions. First, is my understanding correct ? If so, how is it possible to use the last model to make predictions as it has different features.

You have created a model pipeline and must run all trained models ("lower level" ones first) in order to make a prediction on new data using the stack.

With test data set, it is slightly easier, since you can store the predictions from the "level 1" models when testing them, and only run the final model across this stored data.

In addition to your brief description, usually to avoid bias from re-using training data, you would use k-fold cross-validation or similar mechanism, and your training data for the final model should be the cv predictions from each model. You do not want to use the training predictions from those models, because they are likely to be overfit whilst "level 1" test and production predictions will not be, and this would introduce population differences between train and test data in your "level 2" model.

It is also quite a common variation to use the M new features from your "level 1" models alongside some or all of the original features. This gives the meta model more data to base its decision on when deciding the relative weights between the first stage models (assuming this top-level model is non-linear).


NoSQL Comparison - Is this part of my Job? For the more experienced Data Scientists here, i was asked to perform a case study on how Redis / HBase etc performs, compared to each other.How does data science play a role into this? Note that there will be no actual data involved.

I am not very familiar with Redis, but as far as I understand the two are based on different principles and used for different things. Redis is in-memory while HBase is not. They are solutions to different problems. Maybe you should start by describing what kind of problems they are suitable for.


Approaching a multi-class classification problem but without labels I am working on a business problem where I have a movie description dataset. In this dataset I've columns as - Movie title, Movie plot summary, Date of Release. Now based on this information and using machine learning I want to predict which category the movie falls into. For example The Conjuring should fall into Horror and Thriller i.e a multiclass classification problem. Now the problem is I don't have a label column besides the movie description and other info. Now I want my model to predict which categories a movie(unseen to model) should fall into. I have decided 5 labels that I want to consider - Horror, Thriller, Comedy, Romantic and Emotional. So, I want the dataset to look like this -

Conjuring| Description | Title | Horror,Thriller

The notebook| Description| Title | Romantic,Emotional

I believe if I want to proceed this problem as a classification problem then I have to think of some way to create labels to existing dataset by some script and logic. If not supervised then maybe if I can do clustering first and then based on where the data point lies I can do classification later on.

What I have tried ?

Once I decided what my 5 labels should be, I made 50 synonyms for each and then iterated the description of the movies and based on the number of occurrence of words I made frequency and based on majority of the occurrence I decided which category a movie should fall into. Very bad results from this approach.

I used K means clusters from the data and tried to extract information from the clusters. Could not get very meaningful information though.

To be very honest I am pretty clueless and just want a direction how to approach this problem.

Your specific problem can be solved by Googling.

Here is a solution that

Searches the "imdb [year] [movie name]" in Google,
Finds its IMDb address and fetches the IMDb page, and then
Searches for the genres inside the IMDb page.

I have changed "romantic" to "romance", and "emotional" to "drama" to match the IMDb vocabulary.

from requests import get
import re

titles=["2013+Conjuring", "2004+The+notebook"]
genres = ['horror', 'thriller', 'comedy', 'romance', 'drama']
matched_genres = {}
for title in titles:
  query = "https://www.google.com/search?q=imdb+" + title
  print(query)
  search_result = get(query).text.lower()
  imdb_id = re.findall("https://www.imdb.com/title/(tt\d+)/", search_result)[0]
  imdb_address = "https://www.imdb.com/title/%s/" % imdb_id
  print(imdb_address)
  imdb_result = get(imdb_address).text.lower()
  matched_genres[title] = []
  for genre in genres:
    # find ">genre<" inside tags
    if imdb_result.find(">%s<" % genre) > -1:
      matched_genres[title].append(genre)

print(matched_genres)

Output

https://www.google.com/search?q=imdb+2013+Conjuring
https://www.imdb.com/title/tt1457767/
https://www.google.com/search?q=imdb+2004+The+notebook
https://www.imdb.com/title/tt0332280/
{'2013+Conjuring': ['horror', 'thriller'], '2004+The+notebook': ['romance', 'drama']}

This solution could be improved by

Querying movie titles in parallel,
Directly querying IMDb API,
Handling edge cases (for example when the first IMDb url is irrelevant or no IMDb page is found, etc.),

etc.


Using Pandas, how do I strip part of a cell from an Excel file? Thanks in advance, I have been struggling with this for a couple of hours now. I have a column named 'CaseNumber'.  the data in that column is structured as 'Case ID Number - 00123456'.

I am trying to create a loop that will go through the entire sheet and strip 'Case ID Number - 00'. So the final output would just be the 6 digit number set.

So far I have found the attribute .strip('Case ID Number - 00') but I cannot find the logic to create the for loop.

I appreciate any insight.

String operations on particular columns (or series) can be done by appending .str to the series. You can use the following command to get your output:

>>> df['Case Number'].str.strip('Case ID Number - 00')

You can refer to the documentation here Working with text.


What does the Ip mean in the Bayesian Ridge Regression formula? From http://scikit-learn.org/stable/modules/linear_model.html#bayesian-ridge-regression, they gave the bayesian ridge distribution as this:

$p(w|\lambda) = \mathcal{N}(w|0,\lambda^{-1}{I_{p}})$

And there is a variable $I_p$ but it's unexplained what does the $I_p$ refer to?

Also, the variable $\mathcal{N}$ is unexplained but I'm not sure whether I've guessed correctly but is that the Gaussian prior as described in the Bayesian regression section above the Bayesian Ridge?

$\mathcal{N}$ does indeed denote a (multivariate) normal / Gaussian distribution. $I_p$ is just an identity matrix of dimension $p$. So this a matrix with $\lambda^{-1}$ along the diagonal. Read this as the covariance matrix, so this is a spherical Gaussian (0 covariance between different dimensions) where each variable has variance $\lambda^{-1}$.


Should normalization be applied? I have more then 100 columns with the values of 1-0. But the two features at the end as seen in the below image, have different values then the rest. Should I rescale the values in the last two columns into a range of [0,1]?
Note : I use K-Means algorithm.

Let $x \in \mathbb{R}^{d}$ be a feature vector (one row of your table).
Your data has the problem that the distribution per dimension is very different, in particular $x_{1} \in \{0,1\}$ and $x_{d} \in \mathbb{R}$.
Using K-Means means to consider each data point in the euclidean space where the similarity (or distance) is measured. However, the main problem arises when using the euclidean space in your context!
If we measure a distance in the euclidean space for $x,y \in \mathcal{X}$, then $||x-y|| = \sqrt{\sum_{i = 1}^{d}(x_{i}-y_{i})^{2}}$. To simplify the examination, let us now consider the squared euclidean loss $||x-y||^{2} = \sum_{i = 1}^{d}(x_{i}-y_{i})^{2} $ and let $e_{i}:= (x_{i}-y_{i})^2$ be the squared deviation per dimension.
Then $||x-y||^{2} =\sum_{i = 1}^{d}e_{i}$. For a column $i$ with binary values we have $e_{i} \in [0,1]$. However, for the last two columns we have $e_{i} \in \mathbb{R}$.
Therefore, the last two columns have already a higher weight on the distance calculation $||x-y||^{2} =\sum_{i = 1}^{d}e_{i}$, which might not represent the true importance of these dimensions. Even more, by using the euclidean space you implicitly assume that the importance of the features are comparable to each other, which might not be the case.
So what are your options:
1.) Min-Max scaling (so that also the last two dimensions are in $[0,1]$).
You can do that, but you should be aware of the impact: You could say from the perspective of the distance measurement, all dimensions have now equal weight, which might be misleading again.
2.) Standardizing your data
You can do that. It is often better than min-max scaling as you maintain the distribution per dimension. Still you should be aware of the impact: You could say from the perspective of the distance measurement, all dimensions still have kind of an equal weight, which might be misleading again.
3.) You can perform L2 normalization and use the cosine similarity.
It results in the same issues..
Therefore, if the importance of some features is very different, or if the importance of different features cannot be determined, e.g. they are not comparable, using K-Means (or measurements in the euclidean space) is  basically a bad idea.
4.)
Recall that Random forest is invariant to these issues: If $\mathcal{X}$ is a set of data points that is classified by an inner node of a decision tree that has been trained by the random forest algorithm, then the algorithm provides a threshold $\theta \in \mathbb{R}$ and a dimension $d' \in \{1,\ldots,d\}$. The inner node splits the data into $\{x \in \mathcal{X} \mid x_{d'} \leq \theta \}$ and $\{x \in \mathcal{X} \mid x_{d'} > \theta \}$.
In other words, it learns at each inner node of a decision tree a weak classifier that is applied to a single dimension, thereby avoiding to define a comparable importance weight for each dimension.
You can use a random forest to build a distance measurement (see Wikipedia), and apply K-Means afterwards.
5.) Use K-Means variants.
6.) Use PCA to project your data to a space $\mathbb{R}^{d'}$ with $d' < d$, then apply K-means.
7.) Use an autoencoder to learn a better embedding. Then apply K-means.


How to change parameters in LSTM for multivariate binary classification time series? https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/
I am trying to follow this tutorial's code with a slightly different dataset, where the predictor is a 0 or 1: presence or absence of something. My data is continuous time series data, and I have other continuous variables that are inputs.
For the most part, sans column numbers and names, my code is almost identical to the one in that tutorial. However, my predicted values show up like this on the plot:

And upon changing the "Dense" line from this:
model.add(Dense(1))

into:
model.add(Dense(1, activation='sigmoid'))

and the compile line from:
model.compile(loss='mae', optimizer='adam')

to:
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

I thought this was all I needed to change to make the predictions binary, but my predictions plot still keeps giving decimals and nonbinaries, as the plot above shows. What else do I need to account for in a classification problem in LSTM?

The sigmoid activation gives you a number between 0 and 1 (you can consider it as the probability of item i to be of class 1). To get hard predictions (0s and 1s) you just use a rule that says "I consider that the item is predicted positive if the probability is above a certain threshold". For example, for a threshold of 0.5, you just map every predicted value above 0.5 to 1 and the rest to 0.


Identifying sequences of actions required to complete tasks, based on data of completed tasks I have a list of about 20 tasks that a machine need to be performed by a machine. Each tasks consists of a sequence of 3 to 5 actions that must be executed sequentially to complete a task. There are a total of 50 different actions.

I want an algorithm to learn which actions to execute to complete the tasks. To learn this I have a dataset of approximately 1000 samples. Each sample consists of a task with corresponding actions.

Given a task $T_n$, the machine needs to come up with the sequence of actions to complete the task.

Sample dataset ($T$ for tasks out of 20, $A$ for actions out of 50):
$T_1$ - $(A_5 \rightarrow A_1 \rightarrow A_3)$
$T_2$ - $(A_7\rightarrow A_12\rightarrow A_4\rightarrow A_1)$
$T_1$ - $(A_5\rightarrow A_3\rightarrow A_{13})$

What is the best way to solve this?

Based on my knowledge of linear regression and classification, I believe that both are not the right approach to solve the task at hand.

EDIT 1:

One way I can think to solve this problem.
The machine doesn't have to come up with the whole sequence instantly.

If a new task has to be done, the machine has an option to choose the next action (first action) from, say, 5 available actions it can perform. After performing this action, it would have, say, 5 more actions and it has to choose it's next action to complete. And go on doing these actions to complete the given task.

Is this kind of a finite state machine?

What is the best way to solve this?
Based on my knowledge of linear regression and classification, I believe that both are not the right approach to solve the task at hand.

Not sure about the best way, but one indeed can formulate this kind of problem as a classification problem. I assume that additionally to the data included in your sample dataset you also have information about the environment for all 1,000 observations.
For example, one can reshape the dataset as follows.
Underlying data (I've inserted env as a placeholder for the information about environment, you might have several variables describing the environment):
|task |env |action1 |action2 |action3 |action4 |action5 |
|:----|:---|:-------|:-------|:-------|:-------|:-------|
|T1   |E1  |A5      |A1      |A3      |END     |NA      |
|T2   |E2  |A7      |A12     |A4      |A1      |END     |
|T1   |E3  |A5      |A3      |A13     |END     |NA      |

Transformed dataset that can be passed to the classifier
|task |env | num_action|prev_action |action |
|:----|:---|----------:|:-----------|:------|
|T1   |E1  |          1|START       |A5     |
|T2   |E2  |          1|START       |A7     |
|T1   |E3  |          1|START       |A5     |
|T1   |E1  |          2|A5          |A1     |
|T2   |E2  |          2|A7          |A12    |
|T1   |E3  |          2|A5          |A3     |
|T1   |E1  |          3|A1          |A3     |
|T2   |E2  |          3|A12         |A4     |
|T1   |E3  |          3|A3          |A13    |
|T1   |E1  |          4|A3          |END    |
|T2   |E2  |          4|A4          |A1     |
|T1   |E3  |          4|A13         |END    |
|T2   |E2  |          5|A1          |END    |

Then one can apply any multiclass classification (e.g. multinomial logistic regression) and try to predict the column action using the first four columns as predictors.


Difference between AlphaGo's policy network and value network I was reading a high level summary about Google's AlphaGo, and I came across the terms "policy network" and "value network". At a high level, I understand that the policy network is used to suggest moves and the value network is used to, "Reduce the depth of the search tree [and estimate] the winner in each position in place of searching all the way to the end of the game."
These two networks seem redundant to me. What is the policy network doing if it's not using the value network to prune its policies? It seems pretty clear that the value network is a deep learning neural network; is the policy network just a theoretical abstraction and not an actual neural network? The target variable for the value network seems to be win/loss. Is there a target variable for the policy network; if so, what is it? What is the policy network trying to optimize?
The full pdf of Google's paper, published in Nature, can be found here.

In brief each net has a different purpose as you mentioned:

The value network was used at the leaf nodes to reduce the depth of the tree search.
The policy network was used to reduce the breadth of the search from a node (guiding towards promising immediate actions).

In general, you can use value function methods to find an optimal policy or directly search in the policy space to optimize a parametrized policy function (of course there are pros and cons). You can use function approximators (e.g Deep Nets) in each case. I see that mainly you are confused about the policy net so I focus my answer into this.

The policy net was first:

trained to do the moves that most likely a human would do given a board state (so input is a board state and output is a histogram that shows the probability of each action given that state). The net can approximate the probability function underlying the mapping from states to actions. It is reasonable to think to start building your policy from available data after all. After supervised training using experts moves the policy net could play the game sufficient (although far from a Master's level). Simply, you attempted to capture the general pattern of action selection of professional players.

Then,

it was trained in games with opponent itself, in order to optimize the previous-learned policy. This time its weights were updated using the REINFORCE algorithm. By doing this, you update the net parameters towards maximization of expected reward. Eventually you have a net that not only selects the actions like a professional player but also towards winning the game (However it cannot plan!).

After this step, they approximated the value function of a bit more noisy version of the learned policy, by regression (input is the state board and target the result of the game). You can use this network to affect the leaf node evaluation.

Conceptually speaking, the policy net gives you a probability over actions, but this doesn't indicate that you will end up in a good, for winning the game, state. AlphaGo had some "blind spots" and during the tournament did some really bad moves but also one exceptional move that a human could never had thought.

Finally you can use your planning algorithm (MCTS) in combination with these nets. Why we took all these steps? Briefly, the simple MCTS without any "intuition" would have failed.


Data mining- Clustering techniques I have a project for comparison between clustering techniques using the data set of SSA for birth names from 1910-2013 years for the different states.

I have finished applying my clustering techniques on my data set and the output of the clusters were the clusters of the states for each year.

Now I can know from my results; which states are close to each other with the birth names and which were not. By looking

I would like to have interesting results to make my project report interesting, Which states are similar in birth names are not enough to make the read be excited about my project.

My questions:

any ideas of what can be learned from my project?
How can I show the comparison between the clustering techniques? anyway other than seeing how Homogeneous the clusters are?

It's common to perform clustering on certain features, and then use characteristics such as mean differences between other features (not used in clustering) to show how clusters differ.  It's not always meaningful to say that clusters differ merely by lower distances between cluster centers and objects assigned to them (as in k-means).

As an example, you might acquire other features (variables) for the states and then average each feature among objects assigned to each cluster.  Then show bar charts of averages for those variables among all the clusters to assist in explaining how the clusters differ.  You might also test for significantly different means of the supplemental features across clusters.


How to get rid of some parts in a string variable? The original string is like this "965 - Vehicle parking area", how do I get rid of the "965 - " part in R? (There are two spaces before and after the dash.)
Thank you!

One possibility is to use gsub function to find the first ' - ' and remove everything before and also remove the dash with the space before and after that.

> tmp
[1] "965 - Vehicle parking area"
> gsub("^.*? - ","", tmp)
[1] "Vehicle parking area"

If there are two spaces:

> tmp <- "965  -  Vehicle parking area"
> sub("^.*?  -  ","", tmp)
[1] "Vehicle parking area"


What is the difference between an autoencoder and an encoder-decoder? I want to know if there is a difference between an autoencoder and an encoder-decoder.

According to section 2.3 of this paper:

  Auto-encoders are special cases of encoder-decoder models in which the input and output are the same.

The same section of the paper describes encoder-decoder as follows:

  Encoder-Decoder models are a family of models which learn to map data-points from an input domain to an output domain via a two-stage network: The encoder, represented by an encoding function z = f(x), compresses the input into a latent-space representation; the decoder, y = g(z), aims to predict the output from the latent space representation.


How to decide the number of primary components for PCA Background

Trying to identify the number of primary components to use (k) for PCA for MNIST aiming at 95%.

from sklearn.datasets import fetch_openml
mnist = fetch_openml('mnist_784', version=1)

# Split data into training and test
X, y = mnist["data"], mnist["target"]
X_train, y_train = X[:60000], y[:60000]

COVERAGE=0.95

If I follow Coursera Machine Learning - Principal Component Analysis Algorithm it is 67.

from sklearn.preprocessing import StandardScaler
X_centered = StandardScaler().fit_transform(X_train - X_train.mean(axis=0))
covariance_matrx = X_centered.T.dot(X_centered)
U, s, Vt= sp.linalg.svd(covariance_matrx)

calculated_coverages = ((s ** 2) / (len(s) -1)).cumsum()
calculated_coverages = calculated_coverages / calculated_coverages[-1]
k = np.argmax(np.array(calculated_coverages) >= COVERAGE)
print("k-th component to cover {0} is {1}".format(calculated_coverages[k], k))

  k-th component to cover 0.9507022719172283 is 66

However, if I use explained_variance_ratio_ from scikit learn, it is 154.

from sklearn.decomposition import PCA
pca = PCA()
pca.fit(X_train)

contributions = pca.explained_variance_ratio_
coverages = pca.explained_variance_ratio_.cumsum()
k = np.argmax(coverages >= COVERAGE)

print("k-th primary compoent for 95% coverage is {}".format(k + 1))

  k-th primary compoent for 95% coverage is 154

When I look at scikit-learn/sklearn/decomposition/_pca.py, it looks the logic is the same.

    U, S, V = linalg.svd(X, full_matrices=False)
    # flip eigenvectors' sign to enforce deterministic output
    U, V = svd_flip(U, V)

    components_ = V

    # Get variance explained by singular values
    explained_variance_ = (S ** 2) / (n_samples - 1)
    total_var = explained_variance_.sum()
    explained_variance_ratio_ = explained_variance_ / total_var
    singular_values_ = S.copy()  # Store the singular values.

Question

Please help understand why they are different.

Related

Deciding on the number of components in PCA
Python scikit learn pca.explained_variance_ratio_ cutoff

There are two ways to perform the PCA:

Compute the eigenvalue decomposition of the covariance matrix $\Sigma$
Compute the singular value decomposition of the data matrix $X$

Numerically, you can do both by calling svd() on either of them, as for positive semi-definite matrices (like $\Sigma$) svd() gives you the eigenvalue decomposition.

There is a difference, though, when it comes to interpreting the result:

The singular values in s are the eigenvalues of $\Sigma$, i.e. the variances along the PCs
The singular values in s are the singular values of $X$, i.e. the square roots of the variances along the PCs

In sklearn they go with method 2. Hence they need to square the singular values to compute the coverage. In coursera they go with method 1 so no need to square s. In the slide you show and in the video you linked they just sum the values up.

Without having run your code, my guess would be that if you change the line

calculated_coverages = ((s ** 2) / (len(s) -1)).cumsum()

to

calculated_coverages = (s / (len(s) -1)).cumsum()

you will get better results.

Addendum: On second thought I'm not sure how StandardScaler() impacts the results of the PCA either. When comparing, make sure that it is applied in both your implementation of the PCA and the implementation provided by sklearn (and maybe leave a comment if this mattered or not, pretty please ;)).


Tips and tricks for designing time-series variational autoencoders I am new to VAEs but find them quite fascinating. I was wondering if anyone might have any tips or tricks regarding, how one should build the encoder and decoder layers w.r.t. to time-series data.

Suppose my inputs are $x \in \mathbb{R}^{1\times1000}$ so they have a thousand dimensions. The data is highly-nonlinear and rather noisy. I have a couple of tens of thousand $x$:s.

With that in mind, what are some general guidelines (because I cannot find any, owing to the VAE field being somewhat particular and very new):

How deep should my encoder and decoder network be? Are there any good guidelines?
Should one use fully connected dense networks, or stacked conv1Dnets?
What activation functions are good choices?
Can we say anything about the 'best' dimensionality of the latent dimension?
Like everyone else I imagine, my loss function is the sum of a reconstruction term, and the KL divergence regularization term. Is there something else one should consider?
Batch normalization? Currently, on my problem, it doesn't make a blind bit of difference. But it should. Should one always use batch-norm?
In the decoder layer, is it better to up-sample before returning to the original input dimension?

Any tips or tricks would be most welcome. I have naturally tried a few of these, but I cannot discern a trend at present.

I can speak from a more theoretical point of view, but honestly I haven't had much success with VAEs.

1) How deep should my encoder and decoder network be? Are there any good guidelines?

That depends entirely on your dataset. If you have highly nonlinear data, then a deep network should do well. The successive nonlinearities allow you to capture higher-order correlations in the input, which might be good for your situation.

Do you have any idea of the number of truly independent dimensions in your dataset? Have you done something like PCA (a linear autoencoder)? That might tell you where to aim your dimensionality of your bottleneck layer, and that might help you determine your layer sizes to get there.

2) Should one use fully connected dense networks, or stacked conv1Dnets?

I'd say try both.

3) What activation functions are good choices?

Hugo Larochelle once said that you should always start with ReLUs. See if you get a good enough result with them, as they aren't as prone to the problem of exploding/vanishing gradients, which is something you're going to face with time series data. Remember to initalize them with small positive values, though, to avoid dead neurons.

[EDIT: actually, vanishing and exploding gradients can still be an issue with ReLUs, it's just mitigated a bit. Also, initialize the bias with small positive values. Andrej Karpathy said so in a lecture]

4) Can we say anything about the 'best' dimensionality of the latent dimension?

In some contexts yes, we can. The 'best' dimensionality will be the one that results in the highest lossless compression. That is, the one that compresses your input data the most without losing information when you reconstruct. Good luck finding that optimum, though.

5) Like everyone else I imagine, my loss function is the sum of a reconstruction term, and the KL divergence regularization term. Is there something else one should consider?

You could incorporate various types of regularization that will modify your loss function. You could use dropout, for example, or L1 or L2 regularization.

EDIT: you could also consider an attention-based model, for example see this discussion.

6) Batch normalization? Currently, on my problem, it doesn't make a blind bit of difference. But it should. Should one always use batch-norm?

'Always' is a strong term. I would say not always, but batch normalization ought to be a good choice. Have another look at your implementation before you give up on it.

7) In the decoder layer, is it better to up-sample before returning to the original input dimension?

That's the default presentation - an autoencoder is usually shown as a symmetric construct from the input to the hidden to the reconstruction. I would say experiment with not doing so, but I don't know if one is better than the other.


Why are RNNs used in some computer vision problems? I am learning computer vision. When I was going through implementations of various computer vision projects, some OCR problems used GRU or LSTM, while some did not. I understand that RNNs are used only in problems where input data is a sequence, like audio or text.
So, in kernels of MNIST on kaggle almost no kernel has used RNNs and almost every repository for OCR on IAM dataset on GitHub has used GRU or LSTMs. Intuitively, written text in an image is a sequence, so RNNs were used. But, so is the written text in MNIST data. So, when exactly is it that RNNs(or GRUs or LSTMs) need to be used in computer vision and when don't?

When I think about RNNs applied to Computer Vision, two main research areas of Deep Learning come up to my mind:

Image Captioning: Neural Networks trained to produce descriptions of images. In that case, you have a Conv ecoder that processed pixel data, and an RNN decoder that produces a description.

Video processing (I don't have a better term). Anything regarding video data requires a mixture of CNNs and RNNs. In general: you process each frame with Conv layers, and the sequence of processed image frames is then fed into some RNN layer that performs the final task, that could be classification, segmentation, self driving cars, ... you name it.


How would you optimize this code? I have the following code written using the pandas library. I would like to know if there are ways to optimize the code.

for column in df:
 for index, row in df[column].iteritems():
    if type(row) == str:
        if 'R$' in row:
            n = row.replace('R$', '')
            n = n.replace(' ', '')
            n = n.replace('.', '')
            df[column].iloc[index] = float(n)

Just want to remove unwanted string parts.

You can replace the symbol in dataframe without iterating yourself.

df = df.replace({'R\$': ''}, regex=True)

Then change the type of columns that can be numeric. If you don't know which are those columns, use this that will automatically change the type to numeric and ignore those that cannot be changed.

df = df.apply(pd.to_numeric, errors='ignore')

When you use replace and in general many other pandas features, it doesn't update your dataframe. It creates a new, temporary dataframe. So, you either need to assign it back to your original dataframe or use inplace=True wherever it is available, like:

df.replace({'R\$': ''}, regex=True, inplace = True)

$ is a special character in regex, so you need to escape it. That's why the backslash before it.

import pandas as pd

dic = {'feature1': 'R$ aaa bb', 'feature2': 1}
df = pd.DataFrame(dic, index=[0,1])

print(df)

>>  feature1  feature2
0  R$ aaa bb         1
1  R$ aaa bb         1

df = df.replace({'R\$': ''}, regex=True)

print(df)

>>  feature1  feature2
0   aaa bb         1
1   aaa bb         1


Bagging vs pasting in ensemble learning This is a citation from "Hands-on machine learning with Scikit-Learn, Keras and TensorFlow" by Aurelien Geron:

"Bootstrapping introduces a bit more diversity in the subsets that each predictor is trained on, so bagging ends up with a slightly higher bias than pasting, but this also means that predictors end up being less correlated so the ensemble’s variance is reduced."

I can't understand why bagging, as compared to pasting, results in higher bias and lower variance. Can anyone provide an intuitive explanation of this?

Let's say we have a set of 40 numbers from 1 to 40.
We have to pick 4 subsets of 10 numbers.
Case 1 - Bagging -
We will pick the first number, put it back, and then pick the next. This makes all the draw independent and consequently have very little correlation.
So, if you make a Tree on the first 10 samples and another Tree on the next, both the trees will have little correlation and high variance(among them) (more independent splits).
At the same time, because of selection with replacement the data points will be repeated [~63% unique] [Ref], which will increase the bias of individual Trees.

In case of bagging, the sample size is equal to the size of the dataset but we just created this scenario to compare it with Pasting.

Same logic goes for splitting with Random Features subset i.e. RandomForest.
It might be possible that a Split on a particular Feature may result in the correlated next split(always). So if we randomly pick a subset of features before each split, then this will further reduce the Correlation. But again, we will have increased Bias.
Case 2 - Pasting -
Here, because of selection without replacement, the data points in each sample will be unique which will result in lesser bias of individual Trees.


Build a corpus for machine translation I want to train an LSTM with attention for translation between French and a "rare" language. I say rare because it is an african language with less digital content, and especially databases with seq to seq like format. I have found somewhere a dataset, but in terms of quality, both french and native language sentences where awfully wrong. When I used this dataset, of course my translations where damn funny ...
So I decided to do some web scraping to build myself my parallel corpus and it might be useful for research in the future.
It worked well and I managed to collect some good articles from a website containing some articles (monthly, since 2016 in both languages). Now the tricky part is putting everything into sentence to sentence format. I did a trial with a text and its translation just by tokenizing into sentence and I noticed that for example I had 23 sentences for French and 24 for native language.
Further checking showed that some small differences where notices in both languages, like a sentence where a comma was replaced in the other language by a dot.
So my question is :
Is it mandatory to put my articles into sentence-French to sentence-Native language format ? Or can I let it as text / paragraphs ?

What you would typically do in your case is to apply a sentence alignment tool. Some popular options for that are:

hunalign: a classical tool that relies on a bilingual dictionary.
bleualign: it aligns based on the BLEU score similarity
vecalign: it is based on sentence embeddings, like LASER's.

I suggest you take a look at the preprocessing applied for the ParaCrawl corpus. In the article you can find an overview of the most popular methods for each processing step.
A different option altogether, as you suggest, is to translate at the document level. However, most NMT models are constrained in the length of the input text they accept, so if you go for document-level translation, you must ensure that your NMT system can handle such very long inputs. An example of NMT system that can be used for document-level NMT out of the box is Marian NMT with its gradient-checkpointing feature.


Using a part of a trained model in a custom loss function -Tensorflow I want to write a custom loss function that uses the intermediate result of a trained discriminator.
the loss function compares images.
the loss function is for recovering the latent vector of an image from a gan.
im relatively new to this.

visual representation

im using a reference code to test it out.
https://github.com/utkd/gans/blob/master/cifar10dcgan.ipynb
for full reference code im using

below is an example
https://m.youtube.com/watch?v=dCKbRCUyop8
Watch at 17:30

below is the discriminator code

def get_discriminator(input_layer):
  '''
  Requires the input layer as input, outputs the model and the final layer
  '''

  hid = Conv2D(128, kernel_size=3, strides=1, padding='same')(input_layer)
  hid = BatchNormalization(momentum=0.9)(hid)
  hid = LeakyReLU(alpha=0.1)(hid)

  hid = Conv2D(128, kernel_size=4, strides=2, padding='same')(hid)
  hid = BatchNormalization(momentum=0.9)(hid)
  hid = LeakyReLU(alpha=0.1)(hid)

  hid = Conv2D(128, kernel_size=4, strides=2, padding='same')(hid)
  hid = BatchNormalization(momentum=0.9)(hid)
  hid = LeakyReLU(alpha=0.1)(hid)

for my loss function i want to use the intermediate result from the layer above

  hid = Conv2D(128, kernel_size=4, strides=2, padding='same')(hid)
  hid = BatchNormalization(momentum=0.9)(hid)
  hid = LeakyReLU(alpha=0.1)(hid)

  hid = Flatten()(hid)
  hid = Dropout(0.4)(hid)
  out = Dense(1, activation='sigmoid')(hid)

  model = Model(input_layer, out)

  model.summary()

  return model, out

below is the code im planning to use

zp = tf.Variable(np.random.normal(size=(1,l_size)), dtype=tf.float32)

start_img = Image.open(folder + "foo_00.png")
start_img.resize((img_x, img_y), Image.ANTIALIAS)
start_img_np = np.array(start_img)/255

fz = tf.Variable(start_img_np, tf.float32)
fz = tf.expand_dims(fz, 0)
fz = tf.cast(fz,tf.float32)
# variable 'generator' = trained model that is loaded.

# Define the optimization problem
fzp = generator(zp)
loss = tf.losses.mean_squared_error(labels=fz, predictions=fzp)

here is where i want it to go something like

fzpD= discriminator_intermediate(fpz)
fzD= discriminator_intermediate(fz)
loss = tf.losses.mean_squared_error(labels=fzD, predictions=fzpD)
```

Model class allows you define multiple outputs: official tensorflow (keras) documentation.

...
hid = LeakyReLU(alpha=0.1)(hid)  # the layer you want to use
intermediate = hid
...
model = Model(input_layer, outputs=[out, intermediate])

Then, if you train using model.fit or model.fit_generator, you simply need to provide the labels as a tuple of (expected_output, expected_intermediate_layer).


Eigenvalues of covariance matrix are negative I'm working on the PCA of the mnist dataset, and I get a very strange result, I created a matrix whose rows are flattened mnist images, When I try to compute the eigenvalues of the covariance matrix, I get some negative values. But the covariance matrix is positive semi-definite.
np.linalg.eigvals(np.dot(mnistBis[:, 0:20].T, mnistBis[:, 0:20])) # mnistBis.shape=(60000, 784)

array([ 4.79599869e+02, -1.19628465e+02,  9.68398702e+01,  1.88726171e-01,
        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,
        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,
        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,
        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00])

Modification:
Here is the full code
import tensorflow.keras.datasets.mnist as mnist
import matplotlib.pyplot as plt
import numpy as np

(x_train, y_train), (x_test, y_test) = mnist.load_data()
mnistBis = np.reshape(x_train, (-1, 28*28))
np.linalg.eigvalsh(np.dot(mnistBis[:, 0:20].T, mnistBis[:, 0:20]))

```

You seem to be using a generic eignvalue solver. Perhaps your matrix is badly conditioned and the aglorithm cannot pick it up. Lots of zero eigenvalues with few massive ones certainly seem suspicious. Have you tried solvers specific for hermitian/symmetric matricies? eigh & eigvalsh
Also, I would z-score all your features, and drop all features with zero variance. This will set an upper bound on maximum possible eigenvalue since trace of your matrix will be the number of the features, and since positive definite matrix has only positive eigenvalues, the largest eigenvalue will be no larger than the trace.


What is the “learning” step in Gaussian Naive Bayes classification? For conditionally independent features $f_i$, Naive Bayes Classification gives me the classifier

$Classifier(f) := \arg \max_{k} P(C=k) · ∏^n_{i=1} P(f_i|C=k)$

for classes $k$. I understand that for Gaussian Naive Bayes, I can assume normally distributed features, yielding

$Classifier(f) := \arg \max_k P(C=k) · ∏^n_{i=1} \frac{1}{\sqrt{(2πσ_{k,i})}} e^{( -\frac{(f_i - μ_{k,i})^2}{2σ_{k,i})}}$

where $μ_{k,i}$ is the mean of class $k$ and feature $f_i$ (and similiar for variance $σ_{k,i}$).

But where is the "learning step" in this whole procedure?

I'm assuming you're asking about the intuition behind Naive Bayes (NB). For the sake of clarity I'm considering only categorical features. Gaussian NB is simply an application of NB to numerical features (assumed to be normally distributed).

During training every $p(f_i|C_k)$ is calculated by counting how often the feature value $f_i$ is associated with class $C_k$ among all the other possible features values associated with $C_k$:

This is done by counting how often $f_i$ and $C_k$ appear together across all the instances. That's how NB generalizes: the fact that a feature appears with a particular class is just an example, but the fact it appears proportionally more often with class A than class B forms a pattern.
The probability $p(f_i|C_k)$ represents how important $f_i$ is within class $C_k$.

When predicting the class for a new instance:

NB "weighs all the pros and cons" for this instance to be predicted as $C_k$ by combining all the $p(f_i|C_k)$ corresponding to this instance (in the sense that some of the probabilities $p(f_i|C_k)$ are low and some are high, so their product reflects the combination of "pros" and "cons" indications).
But even if $p(f_i|A) > p(f_i|B)$, it doesn't imply that $f_i$ is a strong indication of class $A$, because maybe this is due to class $A$ being less frequent than class $B$. This is taken into account in the prior $p(C_k)$, which gives more importance to a frequent class than a rare one (this is the basis of the Bayes theorem).

These last two points show how NB uses the "knowledge" of the trained model to make a prediction about any unknown instance.


Confusion about neural network architecture for the actor critic reinforcement learning algorithm I am trying to understand the implementation of the actor critic reinforcement learning algorithm. According to this, there should be just one neural network with two heads for the action probabilities and the state values. It is also true in their tensorflow implementation here.

However, if I refer to the tensorflow implementations such as this and this, they use two nn and update them separately.

So, my question is what is the correct way to go about the neural network for the actor critic algorithm?

Both architectures work. It is probably more common to use two separate networks for simpler problems, and a combined network for a more complex problem, such as one involving machine vision.

In general, you need the two functions - policy and value - to be separate. There is no reason to expect them to have too much in common, in terms of the overall mapping from state to their output. However, if the state requires a lot of non-linear interpretation to get meaningful features, such as image, audio or video input, then it could be an overall benefit if the two functions share the lower-level feature representations. Not only will this encourage better generic learning of the low-level features (because they are effectively being trained with twice the data per time step compared to if they were separate), but the calculations should be faster too.

For a similar reason, if the agent works from natural images, you could use either pre-processed features from upper layers of some model trained on ImageNet, or you could start the network initialised with the first layers of such a model. This might apply for other RL agents, such as DQN too - if this works, then clearly sharing that part of the network when you have more than one function to work on could work also.

If your state data is simpler, e.g. a few positions and velocities, or one/both policy and value have a simple relationship to state, then a joint network may be less useful.

When the policy and value functions are in a shared network, they may still both have more than one dedicated layer, as it is expected that there is not any simple linear relationship between the two functions (a NN could learn this by itself, but the assumption is good enough that it may as well be hard-coded by splitting the network into two branches with maybe a couple of hidden layers each before output).


How would be plotted the logistic functions associated to a multiclass logistic classifier in a X-Y plot? Logistic function is well known to be a good binary classificator, as it can be easily shown with this image (let x be the dot product of [x1,x2] and [w1, w2]): enter image description here

I am currently learning how a multiclass logistic classifier works, and this time with x1 and x2 as axes, three classes can be represented as follows (from this link):

According to several sites, both the "one vs the rest" and the multinomial logistic classifiers, would have one logistic regression for each of these 3 classes, with the difference of training them "together" or separately. If this is true, and we assume the next figure to be related to the previous example, I can't figure out how these 3 regressions (as in the first example) would be plotted in there.

As much I could figure, in the "one vs the rest" case, how to draw the logistic function for blue and yellow categories against the rest, but how would it be for the red category against the rest?

If you are asking how to reproduce those plots, the following technique works to visualize the decision boundaries for any model:

Generate a dense grid of coordinates that fill your plotting area
Score each point on your grid
Plot your results, coloring observations based on your model's predictions. This serves as the background.
Overlay your test data as a scatter plot

In your link, they generate the grid using np.meshgrid and constrtuct the background as a contour plot. The lines representing the decision boundaries of each respective 1-vs-all classifier is plotted using a closed form solution for logistic decision boundaries.

EDIT: To answer the question in your comment, you don't have a single $X$ dimension, and consequently your model output doesn't correspond to a curve as simple as this: it's a 3D surface. The simplest solution would be to just apply the strategy I suggested earlier (and which is also described in your link) but instead of calling .predict() to construct the background coloration, you'd call .predict_proba() and use a sequential color map (e.g. the default veridis) so color intensity corresponds to your class likelihood, giving you something which should look like this. Alternatively, you could plot surface curves. Both of these solutions are projections of the $Y$ axis, $P(Y|x1,x2)$, onto the $X_1$-$X_2$ plane. If that isn't satisfactory, you could pass scored results through PCA to combine your $X$ dimensions into a single $X$ feature -- call this $PC_1$ -- and then plot $PC_1$ vs. $P(Y|x1,x2)$. Personally I think this is significantly less informative, but it would give you an x vs. y plot.


Convergence in Hartigan-Wong k-means method and other algorithms I have been trying to understand the different k-means clustering algorithms mainly that are implemented in the stats package of the R language.

I understand the Lloyd's algorithm and MacQueen's online algorithm. The way I understand them is as follows:

Lloyd's Algorithm:

Initially ‘k’ random observations are chosen that will serve as the centroids of the ‘k’ clusters. Then the following steps occur in iteration till the centroids converge.

The Euclidean distance between each observation and the chosen centroids is calculated.
The observations that are closest to each centroids are tagged within ‘k’ buckets.
The mean of all the observations in each bucket serves as new centroids.
The new centroids replace the old centroids and the iteration goes back to step 1 if the old and new centroids have not converged.

The conditions to converge are the following: the old and the new centroids are exactly identical, the difference between the centroids is small (of the order of 10^-3) or the maximum number of iterations (10 or 100) are reached.

MacQueen's Algorithm:

This is an online version where the first 'k' instances are chosen as centroids.

Then each instance is placed in buckets depending on which centroid is closest to that instance. The respective centroid is recalculated.

Repeat this step till each instance is placed in the appropriate bucket.

This algorithm only has one iteration and the loop goes on for 'x' instances

Hartigan-Wong Algorithm:

Assign all the points/instances to random buckets and calculate the respective centroid.
Starting from the first instance find the nearest centroid and assing that bucket. If the bucket changed then recalculate the new centroids i.e. the centroid of the newly assigned bucket and the centroid of the old bucket assignment as those are two centroids that are affected by the change
Loop through all the points and get new centroids.
Do a second iteration of points 2 and 3 which performs sort of a clean-up operation and reassigns stray points to correct buckets.

So this algorithm performs 2 iterations before we see the convergence result.

Now, I am unsure if what I think in point 4 in the Hartigan-Wong algorithm is the correct method of the algorithm. My question is, if the following method for Hartigan-Wong is the correct method to implement k-means? Are there only two iterations for this method? if not, what is the condition for convergence (when to stop)?

Another possible implementation explanation what I understand is.

Assign all the points/instances to random buckets and calculate the respective centroid.
Starting from the first instance find the nearest centroid and assign that bucket. If the bucket changed then recalculate the new centroids i.e. the centroid of the newly assigned bucket and the centroid of the old bucket assignment as those are two centroids that are affected by the change.
Once there is a change in the bucket for any point, go back to the first instance and repeat the steps again.
The iteration ends when all the instances are iterated and none of the points change buckets.

This way there are a lot of iterations that start from the beginning of the dataset again and again every time when an instance changes buckets.

Any explanations would be helpful and please let me know if I my understanding for any of these methods is wrong.

H-W's algorithm, from the 1979 paper, takes as input initial clusters. However, the authors suggest a method for obtaining them in their last section. They write that it is guaranteed that no cluster will be empty after the initial assignment in the subroutine. It goes as follows:

Compute the overall mean $\bar{x}$.
Order the observations with respect to their distance to $\bar{x}$, that is $||x_i - \bar{x}||_2$ (in ascending order I guess?).
Take the points in position $\{ 1 + (L-1) [M/K] \}$, where $L=1, \dots, K$, as initial centroids. ($[\ \cdot\ ]$ most probably refer to the floor function, hence the $1$ at the beginning.)

As for the main algorithm, it is described in a paper called Hartigan's K-Means Versus Lloyd's K-Means-Is It Time for a Change? by N Slonim, E Aharoni, K Crammer, published in 2013 by AJCAI. Note that this version simply uses a random initial partition. It goes as follows.

For the vectors $x \in \mathcal{X}$ and a target number of clusters $K$,

Set $\mathcal{C}$ to be a random partition of $\mathcal{X}$ into $K$ clusters and compute the centroid vector associated to each $C \in \mathcal{C}$, denote them $v_C$.
Scan $\mathcal{X}$ in a random order, and for all $x \in \mathcal{X}$

2.1 Set the stopping indicator to $s = 1$

2.1. Tentatively remove $x$ from its cluster $C$, letting $C^{-} = C \setminus \{ x \}$.

2.2 Find
\begin{align*}
C^+ = \Big\{ \mathrm{argmin}_{C^* \in (\mathcal{C} \setminus C) \cup C^{-}}\ \frac{1}{n} d(x,v_C^*) + \frac{1}{n} \sum_{y \in C^*} [d(y,v_{C^* \cup x}) - d(y,v_{C^*})] \Big\} \cup \{ x \}
\end{align*}

2.3 If $C^{+} \neq C$, then reset $C \leftarrow C^{-}$ and $C^* \leftarrow C^{+}$ and update their respective vectors $v_{C}$ and $v_{C^*}$. Also reset $s \leftarrow 0$.
If $s=0$, go back to 2.

Here I make a slight abuse of notation letting $C^*$ be the solution of the $\mathrm{argmin}$ in 2.2. This step, 2.2, simply computes the change in the loss obtained if $x$ is added to $C^*$, as opposed to it staying alone in its own cluster (once removed). $d$ stands for distance. Finally, the particular ways to compute the updated vectors $v_C$, $v_{C^* \cup \{x \}}$ and so on is provided in the paper cited above. They can be done in linear time.

I think the answers to all your questions are implicit in the above algorithm... However, I still have to make sure this implementation of the algorithm is standard. In particular if it is the one implemented in R. Any comments/edit are welcomed.


nested CV feature selection I have a small dataset of 150 records with 25 features (too small to do train/test).
I'm using nested cv for both hyperparameter tuning and feature selection. 10cv in the outer loop, 5 cv in the inner loop.
Eventually i'm getting 10 sets of hyperparameters and 10 sets of selected features.
If i'm publishing my results, how eventually would i know which features should be selected for a model to be tried on an external data. (Currently i don't have another external data to test the model)
thank you

Assuming the feature selection method is always the same, on an external data (or final training set), you would simply apply the exact same method. The actual set of selected features does not matter.
If there was any difference in the selection method, for instance if you are selecting different number of features, you would do like with any other hyper-parameter: select the best model according to the nested CV process, then apply the same hyper-parameters (including for instance number of features) when training the final model.


Algorithms and tools for ranking text as a job description We're working on a ranking texts by degree of the similarity to vacancies. We have 4-year data set (≈1M texts) of custom search feeds from social network. We also have vacancies (≈30K) manually selected from this set.

Now the vacancies are selected in 2 stages:

We make several specific search queries (i.e. "need translator") on the social network by API, receiving and combining the search feeds into one.
Then we look through each post in it for real vacancies. Generally, it should contain a request for interpreter services. Often it contains some job details (such as a place, salary, subject, deadlines, languages, contacts) and stylistic features of speech (i.e. greetings, exclamations, request to write in private, ask for recommendations, etc.).

Before the point 2, we would like to have first in our final moderation feed the texts, that are most likely to be vacancies. We're going to try ranking by using the Naive Bayes classifier or the Natural Language Processing, but it's not as simple as it seems: there are a lot of various tools, schemes, models.

The question is, what algorithms and tools could help in our case? What we should pay attention to?

If I understand the problem correctly, there are several approaches that can be taken.

We'll begin with the least deep learning oriented solution first and slowly move into the spectrum of Deep Learning. This choice is primarily for cost and difficulty reasons. The amount of bugs that you can find with a deep learning solution can be very difficult to understand in certain situations and that results in slower prototyping and ultimately slower time to production.

At the very base, your problem deals with ranking the jobs with respect to how relevant they are to a particular job. The literature for ranking problems is vast and I will suffix my answer with links to these papers or studies. I will also say that instead of dealing with this as a ranking problem there may be another way of going about it. However, this is hinged on the type of data you possess. If you know that each query is certainly looking for a particular type of job. Then, you can break this down into a multi-task problem, where one of the tasks is figuring out whether there is a vacancy or not and the second is figuring out if the vacancy relates in any way to the search query.

Solution 1: Take the dataset and handcraft a set of features that you think are relevant to the process of classifying the output. To do this effectively, you should know certain core features that you think relate the input features to the final label (here I use input features to represent the features that you think represent your features well. For example, some features that may be relevant is information about the person or entity posting the title (in stage 2), if they are a well known company in the domain (this can be accesses through some intelligent web scraping and text retrieval methods). Then, it is possibly more likely for it to be a posting, right? Let us assume now that you have constructed this set of features. Now, for each of the set of features, you have a corresponding label to whether there is a vacancy or not. To turn this approach into a ranking problem is very simple. All you have to do is observe the confidence (the softmax probability is good enough) and the one which has a higher probability is ranked higher). This solution is an easy one to accomplish using even simple feed forward neural networks. You would also benefit from testing out simple classifiers like SVMs or Random Forests. It must also be remembered that the actual sentence must be embedded into some numeric space, like a sequence of vectors in $R^n$. This solution does not detail all of the explicit steps but provides a general framework with which you should be able to attack the problem.

Drawbacks The above method requires a lot of time spent in handcrafting the feature set and there is going to be a lot of iteration because the feature set is never going to be perfect. There is always going to be something that you possibly missed out. Secondly training a good word embedding is not a very easy task. You may have to spend a lot of time getting that running, especially if you want the embedding to be somehow related to your words.

Solution 2 This solution begins to touch the fence of Deep Learning. This solution is a comparison of the attention encodings within the source utterance and the target utterance. The source utterance is, a phrase like "need janitor". Now you should train an attention model over the sequence of words and observe what is the word(s) being given the maximum attention to. Once you know this, the problem becomes a question of consolidating the attentions in some meaningful manner and comparing the distance between the source and target through some metric space. So you should certainly pay attention to attention based models (haha!). Again, there are going to be hiccups in the process of figuring out how much attention actually affects the output and whether it is a good model for your data. Distances between attention will return a vector and you can consider the $\mathcal{L}_2$ norm or any other map from $\mathbb{R^n} \rightarrow \mathbb{R}$.

Drawbacks Attention distances are a good metric, however, they are extremely data centric. For example if the ads are very long, it may be true that the model does not pick up the appropriate words to attend to.

Now we get to some resources that may be useful to go through in order to understand exactly how the different components of the answer work.

https://en.wikipedia.org/wiki/Learning_to_rank
http://www.shivani-agarwal.net/Events/SDM-10-Tutorial/sdm10-tutorial.pdf
https://towardsdatascience.com/learning-to-rank-with-python-scikit-learn-327a5cfd81f
http://times.cs.uiuc.edu/course/598f16/l2r.pdf
https://www.stat.uchicago.edu/~lekheng/meetings/mathofranking/slides/agarwal.pdf

Now for the posts about attention:

http://www.wildml.com/2016/01/attention-and-memory-in-deep-learning-and-nlp/
http://akosiorek.github.io/ml/2017/10/14/visual-attention.html
https://medium.com/syncedreview/a-brief-overview-of-attention-mechanism-13c578ba9129

Finally, my last component of the answer will deal with evaluating exactly how well you have done using metrics. I have added some metrics above in my Answer, however there is another metric which in particular represents the similarity, syntactically between the input query and the output query.

https://en.wikipedia.org/wiki/BLEU


Regarding Training loss and validation loss I am trying to use keras for multi label news classification. I am a beginner in machine learning so please bear with  me.

Here is my training loss vs validation loss diagram:

I understand that my model is overfitting.
On the test set, I have precision of 0.89 , recall of 0.82, f1score 0.83. This according to me is high precision and high recall. So this means a good classification right ?

Further, this is the parameter settings:

My question is why in the test set there is high precision and high recall but the model is overfitting in validation set. Isn't it odd? or am i missing something?

I would have uploaded my dataset. But the dataset excel file is 800 MB.

Note that I don't have a seperate training set, validation set and test set. I have a single file. On the first column are the articles and in the second columns are the labels.

I used sklearn.train_test_split twice so that the training set is 60% test set is 20% and validation set is 20%.

To the learning-curves look exactly like what you would expect. The training-loss goes down to zero. That means your model is sufficient to fit the data. If the training-loss would get stuck somewhere, that would mean the model is not able to fit the data. So, your model is flexible enough. Furthermore the validation-loss goes down first until it reaches a minimum and than starts to rise again. If your validation set is large enough and representative you can argue that your model starts to overfit the data.

Next step could be to prevent the model to overfit. You can do that by adding some kind of regularization, dropout, batch-normalization.

Considering the test set. Instead of using two distinct values try to decide for one metric that does what you want. For example the F1 score. It is hard to say whether these values are good or not. It ultimately depends on what you define as "good enough". If 90% is sufficient for you than that is ok,  but if you need a model that is 99.9% accurate it would not be good enough. Apply the metric to all sets training, validation and test. The validation error should be a good estimator of the test error unless you use it to fit your data. If you use it to identify the optimal number of training rounds it must be considered part of the training set and you would expect the error to be lower than in the test set. Again, if the test set error is bigger, usually something is off.

If your validation and test errors are very different from each other that probably means they sample different regions in the space of your data or the sets are not big enough so that stochastic effects play a role.


How are pictures pre processed before being used as ML data So I was watching this YouTube video
So basically the professor used ML to generate random faces in order to create data for a Kaggle challenge.
When I looked into the data file, I was expecting to see *.jpg or png files, but all I found was CSV files and based on his video, his model generated some png files.
what happened? how did he turn jpg into CSV? can we not use jpg?

First images are turned into pixel values. If it is a color image we have RGB combination of values. It it is black and white then the pixel values for each image in the dataset are unsigned integers in the range between black and white, or 0 and 255. This is the first transformation into numbers.
After that we apply an embedding to it to convert it into an array of numbers. We then use this embedding for further processing.
Images cannot directly be used because AI/ML algorithm deals with numbers only.
You may want to take a simpler image base like the MNIST which is a simple 0-9 representation on a 28*28 pixel image and then play around with recognition algorithms for a better understanding.