In the context of natural language processing, can anyone give a concrete example of True Positive, True Negative, False Positive, False Negative? Google post gives a interesting explanation about True Positive, True Negative, False Positive, False Negative

  True Positive (TP): Reality: A wolf threatened. Shepherd said: "Wolf."
  Outcome: Shepherd is a hero.

  True Negative (TN): Reality: No wolf threatened. Shepherd said: "No
  wolf." Outcome: Everyone is fine.

  False Positive (FP): Reality: No wolf threatened. Shepherd said:
  "Wolf." Outcome: Villagers are angry at shepherd for waking them up.

  False Negative (FN): Reality: A wolf threatened. Shepherd said: "No
  wolf." Outcome: The wolf ate all the sheep.

In the context of CV, the classifier predicts if an image contains cat

True Positive (TP):
Reality: an image contains cat.
classifier predicts: cat.

True Negative (TN):
Reality: an image does not contains cat.
classifier predicts: no cat.

False Positive (FP):
Reality: an image does not contains cat.
classifier predicts: cat.

False Negative (FN):
Reality: an image contains cat.
classifier predicts: no cat.

Can anyone gives a concrete example of TP、TN、FP、FN like above, in the context of natural language processing?

Imagine a hot news classifier.

True Positive (TP):
Reality: a piece of hot news.
classifier predicts: hot.

True Negative (TN):
Reality: not a piece of hot news.
classifier predicts: not hot.

False Positive (FP):
Reality: not a piece of hot news.
classifier predicts: hot.

False Negative (FN):
Reality: a piece of hot news.
classifier predicts: not hot.


Which type of model to use and what's my target variable I am a beginner in Data Science field, so sorry if my question is too basic.

The task is to build an ad bidding model for online marketing which allows you to deliver targeted ads to the right people.
A part of the given data is

I don't have any additional info about the task. Is my target variable the variable spent ?
And if yes which is the best method to follow to predict the target variable?

I would say the variable 'spent' is your target variable. I'd suggest you to analyse the distribution of each feature and target to check if, eventually a linear regression could be appropriate.
You might try different other methods however, like a decision tree regression, which does not require any assumptions as it doesn't estimate parameters.
What I would suggest you to do, howeveer, is to concentrate on the distributions of parameters to check for eventual unbalances and make groups to make estimation robust.


Why should I use data augmentation as Keras layer I've seen in several Tensorflow/Keras tutorials that data augmentation functions are added as keras layers. When I converted my Keras Python model (for production purpose) to TensorflowJS I faced the issue that e.g. the RandomFlip layer is not available in TensorflowJS. So I have to use the ImageDataGenerator.
I don't understand why should I put the data augmentation layers at all in the model, I mean when I want to use my model in production I'll still have them in my model which doesn't make sense for me.

Data Augmentation is basically a technique to increase the diversity of your training set by applying random (but realistic) transformations such as image rotation. Training deep learning neural network models on more data can result in more skillful models, and the augmentation techniques can create variations of the images that can improve the ability of the fit models to generalize what they have learned to new images. So you can decrease the rate of overfitting or take some precautions for it.
The Keras deep learning neural network library provides the capability to fit models using image data augmentation via the ImageDataGenerator class. By using this class you are able to modify the dataset by using its a lot of functionality.


Error when checking : expected dense_1_input to have shape (None, 5) but got array with shape (200, 1) I've trained a model with Keras, saved it and when I'm trying to apply it on new data, I'm encountering an error :

ValueError: Error when checking : expected dense_1_input to have shape (None, 5) but got array with shape (200, 1)

Here's the code for training and saving the trained model :

# Import necessary modules
import numpy as np  # numpy is just used for reading the data
import keras
from keras.layers import Dense
from keras.models import Sequential
from keras.models import load_model # To save and load model

# fix random seed for reproducibility
seed = 7
np.random.seed(seed)

# load the dataset
dataset = np.loadtxt("modiftrain.csv", delimiter=";")

# split into input (X) and output (Y) variables
X = dataset[:,0:5]
Y = dataset[:,5]

# create model
model = Sequential()

# Add the first layer
# input_dim=   has to be the number of input variables.
# It represent the number of inputs in the first layer,one per column
model.add(Dense(12, input_dim=5, activation='relu'))

# Add the second layer
model.add(Dense(8, activation='relu'))

# Add the output layer
model.add(Dense(1, activation='sigmoid'))

# Compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

# Fit the model
model.fit(X, Y, epochs=150, batch_size=10)

# Evaluate the model
scores = model.evaluate(X, Y)
print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))

# Save the model
model.save('model_file.h5')

Here's the code for predicting :

import numpy as np
from keras.models import load_model # To save and load model

# Load the model
my_model = load_model('model_file.h5')

# Load the test data file and make predictions on it
predictions = my_model.predict(np.loadtxt("modiftest.csv", delimiter=";"))

print(predictions.shape)

my_predictions=my_model.predict(predictions)

print(my_predictions)

Here comes the error :

Traceback (most recent call last):
  File "predict01.py", line 14, in <module>
    my_predictions=my_model.predict(X_predictions)
  File "C:\Users\Philippe\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\models.py", line 913, in predict
    return self.model.predict(x, batch_size=batch_size, verbose=verbose)
  File "C:\Users\Philippe\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\engine\training.py", line 1695, in predict
    check_batch_axis=False)
  File "C:\Users\Philippe\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\engine\training.py", line 144, in _standardize_input_data
    str(array.shape))
ValueError: Error when checking : expected dense_1_input to have shape (None, 5) but got array with shape (200, 1)

Thank you for your help.

I had a very similar issue when executing predict/predict_classes;

  ...
    classes = model.predict(XpredictInputData)
  File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 1006, in predict
    return self.model.predict(x, batch_size=batch_size, verbose=verbose)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1772, in predict
    check_batch_axis=False)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 153, in _standardize_input_data
    str(array.shape))
ValueError: Error when checking : expected dense_1_input to have shape (None, 100) but got array with shape (100, 1)

The problem was occurring because the input array to keras predict was 1 dimensional (caused by a combination of a) numpy.genfromtxt producing a 1d array from a 1 row data file and b) a broadcasting failure). Keras predict appears to require a two dimensional array, even if there is only 1 prediction to be made.

I solved the issue by adding the following check before executing predict;

if (XpredictInputData.ndim == 1):
    XpredictInputData = numpy.array([XpredictInputData])


Does scikit-learn have forward selection/stepwise regression algorithm? I'm working on the problem with too many features and training my models takes way too long. I implemented forward selection algorithm to choose features.

However, I was wondering does scikit-learn have forward selection/stepwise regression algorithm?

No, sklearn doesn't seem to have a forward selection algorithm. However, it does provide recursive feature elimination, which is a greedy feature elimination algorithm similar to sequential backward selection. See the documentation here:

http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html


Similarity metric for clustering I am working on a clustering problem. I am not able to find the right similarity metric for my system.

I have n nodes with ordered vector (eg: [1,0,0,0,1,0,.....] "1" represents represent presence of ith object and "0" represents absence of the object). I want to cluster them.

Cosine is for continuous values. It's not the most appropriate thing here.

For binary values, look at

Simple matching distance
Hamming distance
Jaccard distance
Many many many more.

Don't expect clustering to just work. If you have binary data, you only have very few bits of information. The resulting distances will show discrete steps, and clustering will likely still fail to provide good results. Roughly, you have "identical", "one difference", "two differences", and then probably everything already is connected.
It's because of the data, and a not very well defined task (what would be a good cluster?).


Is this the way to obtain the same individuals for x_test and y_test? x_train, x_test = train_test_split(x, test_size = 0.3,random_state=250)
y_train, y_test = train_test_split(y, test_size = 0.3,random_state=250)

Is this the way to coincide the same individuals in x_test and y_test as well as x_train and y_train?

train_test_split has this capability built-in.  Just pass all of the data in the first go like:

Code:

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.3, random_state=250)

Test Code:

from sklearn.model_selection import train_test_split

x = list(range(100))
y = list(range(100, 200))
x_train, x_test, y_train, y_test = train_test_split(
    x, y, test_size=0.3, random_state=250)

assert(all(xx == yy - 100 for xx, yy in zip(x_train, y_train)))
assert(all(xx == yy - 100 for xx, yy in zip(x_test, y_test)))


Keras Embedding Layer main paper I need to explain the word embedding layer of Keras in my paper, mathematically. I know that keras initialize the embedding vectors randomly and then update the parameters using the optimizer specified by the programmer. I want to explain my architecture in an academic paper, therefore I need to explain each layer with a formal formula.
Is there a paper that explains the method in details to reference it? or can you please guide me to how to compose the method formally?

Thanks a lot

I think I found the answer. The embedding layer in keras is nothing more than a set of vectors for distinct words. The keras embedding layer initializes the word embedding with some random values (the default values from a uniform distribution)and then updates the values when train the whole network. Therefore, there is no need to compose the model in details, because the backpropagation do it perfectly :)


Correct way to set Spark variables in jupyter notebook I need to set a couple of variables in my Jupyter Notebook where I have a pre-existing sparkContext and sqlContext and I am doing it wrong. If I don't include an
    sc.stop()
, I get an error that I am trying to instantiate a second context. If I do include it, I get an error that I am trying to call methods on a stopped context.

Can someone tell me the correct way to set these variables?

Here is my code:

from pyspark import SparkConf, SparkContext
from pyspark.sql import HiveContext
from pyspark.sql import SQLContext
conf = (SparkConf()
       .set("spark.yarn.executor.memoryOverhead", "4096")
       .set("spark.kryoserializer.buffer.max.mb", "1024"))

sc.stop()
sc = SparkContext(conf=conf)
sqlContext = SQLContext.getOrCreate(sc)

When you run Spark in the shell the SparkConf object is already created for you. As stated in the documentation once a SparkConf object is passed to Spark, it  can no longer be modified by the user. So stopping it and creating a new one is actually the right way to do it.

However, this should now be possible for Spark 2.0 and higher.


How to test hypothesis? I have a table named app_satisfaction which has user_id, satisfaction, # of people they've invited.

I've grouped by satisfaction and found that on averge.
people in satisfaction ="BAD" group invted 2.25 people,
"GOOD" group invited 2.09 people, and
"EXECELLENT" group invited 1.89 people.

So my hypothesis is people who dislike the app are more likely to invite people since inviting people gives them free coupon and they do not like to spend their own money on app they dislike.

I have a problem, just by looking at average invite in each group it seems unreasonable to draw conclusion. Also there are more people in "GOOD", "EXCELLENT" group compared to "BAD" group.

How can I test my hypothesis? what are the approaches one might take in real world problems?

As far as I understand, you have factors ("bad, "good" etc) and continuous "invitations". If you want to compare two groups you could use a t-test (e.g. Wilcoxon). If you want to compare all of the groups, you could use a simple linear regression of form:

$$ invitations = \beta_0 satisfaction_1 + \beta_1 satisfaction_2 + ... + u.$$

R example:

library("e1071")
iris = iris

table(iris$Species)
#iris = iris[!(iris$Species=="versicolor"),]

library(dplyr)

iris %>%
  group_by(Species) %>%
  summarise_at(vars(Sepal.Length), funs(mean(., na.rm=TRUE)))

Result (means):

# A tibble: 3 x 2
  Species    Sepal.Length
  <fct>             <dbl>
1 setosa             5.01
2 versicolor         5.94
3 virginica          6.59

Compare two groups:

# Two-samples Wilcoxon test
wilcox.test(iris$Sepal.Length[iris$Species=="setosa"], iris$Sepal.Length[iris$Species=="virginica"])
# The p-value is less than the significance level alpha = 0.05. We can conclude that Sepal Length is significantly different

Result:

    Wilcoxon rank sum test with continuity correction

data:  iris$Sepal.Length[iris$Species == "setosa"] and iris$Sepal.Length[iris$Species == "virginica"]
W = 38.5, p-value < 2.2e-16
alternative hypothesis: true location shift is not equal to 0

Regression:

# Simple linear regression
summary(lm(Sepal.Length~Species, data=iris))
# p-values are smaller than 0.05 which means each factor's contribution is statistically different from the intercept

Result:

Call:
lm(formula = Sepal.Length ~ Species, data = iris)

Residuals:
    Min      1Q  Median      3Q     Max
-1.6880 -0.3285 -0.0060  0.3120  1.3120

Coefficients:
                  Estimate Std. Error t value Pr(>|t|)
(Intercept)         5.0060     0.0728  68.762  < 2e-16 ***
Speciesversicolor   0.9300     0.1030   9.033 8.77e-16 ***
Speciesvirginica    1.5820     0.1030  15.366  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.5148 on 147 degrees of freedom
Multiple R-squared:  0.6187,    Adjusted R-squared:  0.6135
F-statistic: 119.3 on 2 and 147 DF,  p-value: < 2.2e-16

The interesting bit here is Pr(>|t|). If the number in this column is smaller 0.05, you can say that the factor is significantly different from the intercept (which is the base category, in this case "setosa").

In this application, the column Estimate directly gives you the mean of "setosa" for the intercept. The effect for "versicolor" is 0.9300, where 5.0060+0.9300=5.936, which is the mean for "versicolor" and so on.


How to identify word in a sentence representing the song genre? I am training a model to identify a word that represents a song genre given a sentence.  For example, the model is given a sentence "Beethoven songs are part of the classical genre."  The model will categorize the word, "classical" as "song genre".  Can I use an RNN classifier to do this or is there another algorithm that can accomplish this task?

You are describing named-entity recognition (NER) which seeks to locate and classify named entities mentioned in text into pre-defined categories. In this case, the pre-defined categories are music genres.
There are many algorithms for NER. In addition to Recurrent Neural Networks (RNN),  Hidden Markov Models (HMMs) and CRF (Conditional Random Field) are often used.
In most cases, the algorithm is not the most difficult part of NER. The difficult part of NER is creating a large, accurate annotated training corpus.


Why is KNN better at K-Fold Cross Validation than XGBoost or Random Forest? I've been running K-Fold cross validation multiple times for KNN, random forest and XGBoost.
KNN can complete sklearn's cross_val_score, so much faster consistently.
They all use the same preprocessed data, all have the same test/train split with a random state, etc.
When ran individually the timings are within 10% of each other.
When ran using sklearn's cross val score, KNN can do 50 k-folds in less than 10 minutes, when the other two take over an hour each.  (this is consistent, i've ran it 5 times for all 3 algorithms)
Mathematically, or maybe due to sklearn's code, is there a reason for this? (The dataset is 200,000 by 100 columns, at a 25% test/train split)

It's not that KNN is better at k-fold cross validation.
Its just that KNN doesn't do any training apart from storing a footprint of the training data within the model.
KNN's logic resides in its inference step i.e. the predict() call where it determines the k nearest neighbors for the newly supplied instance from previously supplied training data and makes a prediction for the label.
So it's likely that KNN is faster than most other models for small to medium sized datasets.


How gradients are learned for pooling layers in Convolution Neural Network? Assuming we could compute a layerwise Hessian of the error function when training a
neural network, the error sub-surface of pooling layers will be flat.??

Is that correct?

There is no weights to be learnt for pooling layer but for eg. max pool can have different values at different iterations? Will that effect the error surface?

1) Will the error subsurface be flat?

One cannot find the hessian or any such error subsurface of the pooling layer because common pooling layers like max and avg pooling do not have any parameters to learn (as you mentioned)!

2) But, we can speak for the effect of the pooling layers on the error surface of the previous layers. The effect is different for different types of pooling layers. For max pooling, it generates a relatively sparser gradient flow for the parameters on the previous layers as only a few of the output from the previous layers are selected during forward propagation. Whereas for average pooling, its allows for a  more smooth gradient flow to all the learnable parameters on the previous layers.


visually explication of dense layer of keras? I am learning keras and I would like to know how the dense layers of keras are visually interpreted, I do not understand very well the concepts and arguments of

model.add (Dense (512, activation = 'relu', input_shape = (784,)))

I know that input_shape = (784,) would be drawn as 784 input neurons ,but I don't understand what 512 is?

You can find more details about the arguments of Dense Layer in Keras Sequential model here.

Example

# as first layer in a sequential model:
model = Sequential()
model.add(Dense(32, input_shape=(16,)))
# now the model will take as input arrays of shape (*, 16)
# and output arrays of shape (*, 32)

# after the first layer, you don't need to specify
# the size of the input anymore:
model.add(Dense(32))

So basically 512 represents dimensionality of the output space of your model.


word2vec: usefulness of context vectors in classification I've been working on a NN-based classification system that accepts document vectors as input. I can't really talk about what I'm specifically training the neural net on, so i'm hoping for a more general answer.
Up to now, the word vectors I've been using (specifically, the gloVe function from the text2vec package for R) have been target vectors. Up to now I wasn't aware that the word2vec training produced context vectors, and quite frankly I'm not sure what exactly they represent. (It's not part of the main question, but if anybody could point me to resources on what context vectors are for and what they do, that would be greatly appreciated)
My question is, how useful are these context word vectors in any kind of classification scheme? Am I missing out on useful information to feed into the neuralnet?
How would, qualitatively speaking, these four schemes fare?

Target word vectors only.
Context word vectors only.
Averaged target and context vectors.
Concatenated vectors (i.e. a 100-vector word2vec model ends up with a length of 200)

So, from what I understand from the question, you want to get an idea of how word2vec works so you can assess how well the resulting context vectors from this model will help discriminate between words by their meaning.
Word2vec works on the premise of the distributional hypothesis which essentially states that words which appear in soimikar contexts  will have similar meanings (e.g. the dog ate the food/ the cat ate the food : both dog and cat appear in the same context so they are semantically close to each other)
So word2vec formulates this by a CBOW model which effectively is a feedforward neural network, which takes in the surrounding context of the target word as a series of one hot encoded vectors and aims to predict the target word (there is an assumption made of course where the contexts are treated as a bag of words and therefore assumes that a word’s meaning is not related to the word ordering of its context).
After training this, the models weights are then used to dorm the word embeddings, which represents a words meaning in semantic space. (For further reference)


How can we calculate AUC for a simple decision tree? The setup is simple: binary classification using a simple decision tree, each node of the tree has a single threshold applied on a single feature. In general, building a ROC curve requires moving a decision threshold over different values and computing the effect of that change on the true positive rate and the false positives rate of predictions. What's that decision threshold in the case of a simple fixed decision tree?

In order to build the ROC curve and AUC (Area under curve) you have to have a binary classifier which provides you at classification time, the distribution (or at least a score), not the classification label. To give you an example, suppose you have a binary classification model, with classes $c1$ and $c2$. For a given instance, your classifier would have to return a score for $c1$ and another for $c2$. If this score is a probability-like (preferrable), than something like $p(c1)$, $p(c2)$ would work. In plain English is translated like "I (the model) classify this instance as $c1$ with probability $p(c1)$, and as $c2$ with probability  $p(c2)=1-p(c1)$".
This applies for all type of classifiers, not only for decision trees. Having these scores you can than compute ROC or AUC by varying a threshold on $p(c1)$ values, from the smallest to the greatest value.
Now, if you have an implementation of a decision tree and you want to change that implementation to return scores instead of labels you have to compute those values. The most used way for decision trees is to use the proportion of classes from the leaf nodes. So, for example you have built a decision tree and when you predict the class for an instance you arrive at a leaf node which have (stored from the learning phase) 10 instances of class $c1$ and 15 instances of class $c2$, you can use the ratios as the scores. So, in our example, you would return $p(c1) = 10 / (10+15) = 0.4$ probability of class $c1$ and $p(c2) = 15/(10+15)=0.6$ probability of being class $c2$.
For further reading on the ROC curves, the best and inspiring source of information I found to be the Tom Fawcett paper called An Introduction to ROC Analysis, it's solid gold on this topic.


What is the better way to predict classes for the models developed using the functional API in Keras We can predict the class for new data instances using the Sequential classification model in Keras using the predict_classes() function. What is the way to predict the class for models that developed using the functional API?

For example, I have a model (functional API based) with sigmoid activation on the last layer to get probabilities in a multi-label classification. When I apply model.predict(), I got a series of probabilities even though the loss is binary_crossentropy.

I understand that I can do this classification manually e.g. following approach.

test_predict_proba = model.predict(x_test, batch_size=batch_size)
class_predict = (test_predicted_proba > 0.5).astype(int)

I am wondering is there any standard procedure to do the same?

Is this a multi-label problem or a pure classification problem? If it is just classification change the activation function in the final layer to softmax. When you do predictions select the output with the highest probability as the class. Alternatively use model.evaluate(.....). versus model.predict Documentation is here,


Visualizing Variance / Standard Deviation for categories Data Structure:

Method Category   Variance for X
1         A             20
1         B             14
1         C             16
2         A             14
2         B             19

Where X was not used for classification, but is evaluation criteria. The objective is select method which produces classification with minimum possible variance for X for most of class / overall least variance.

My question: is there some standard (or obscure) method of visualizing variance for large number of categories?

I know this may not exactly correspond to what you need, but my first idea would not be to visualize the variances. Instead, I would define a metric for each method.

For instance, considering all results obtained with a given, you have a vector of outputs (one Variance for X for each Category), then you can simply compute a p-norm of this vector, and compare it to the norm obtained for other methods:

$$\left\lVert x\right\rVert_p = \left( \sum_{i=1}^n x_i^p \right)^{1/p} $$

$p=2$ gives you the euclidian norm, you can increase or decrease $p$ depending on if you want to raise attention on maximal values or the average.

In terms of visualization, you could simply plot (in a bar plot for instance) a few norms that you selected (orders 1, 2, and infinite for instance).

For a pure visualization, you could plot the data in violin plots. For instance in python:

import seaborn as sns
sns.violinplot(x="Method", y="Variance for X", data=your_data_as_df)

Each violin would give you an idea of how the data is distributed over all categories.


How to ensure same encoding pattern? I created a XGBRegressor model with certain encoded 'object' dtypes in the data. Now if I want to run the model with new set of data which is freshly encoded it's giving wrong predictions. How to ensure that the new dataset is encoded in the same  way as was the train data? Or any other solution to this problem?

You can save the encoding and use them to encode the new data. Only thing to make sure is not to have new data which was not there in saved encoding. You can save them as pickle files if you are using Python.


Why could my DDQN get significantly worse after beating the game repeatedly? I've been trying to train a DDQN to play OpenAI Gym's CartPole-v1, but found that although it starts off well and starts getting full score (500) repeatedly (at around 600 episodes in the pic below), it then seems to go off the rails and do worse the more it plays.

I'm pretty new to ML so I'm not really sure what could cause this so I'm not sure how to start debugging (I've tried tweaking some of the hyper-parameters, but nothing seems to stop this trend).
If it helps, here's the (probably) relevant parts of my agent:
def _build_model(self):
    model = Sequential()
    model.add(Dense(24, input_dim=self.state_size, activation="relu"))
    model.add(Dense(24, activation="relu"))
    model.add(Dense(self.action_size, activation="linear"))
    model.compile(optimizer=Adam(lr=self.learning_rate), loss="mse")
    return model

def get_action(self, state):
    # Use random exploration for the current rate.
    if np.random.rand() < self.epsilon:
        return random.randrange(self.action_size)

    # Otherwise use the model to predict the rewards and select the max.
    q_values = self.model.predict(state)
    return np.argmax(q_values[0])

def replay(self, batch_size):
    if len(agent.memory) < minibatch_size:
        return

    # Decay the exploration rate.
    self.epsilon *= self.epsilon_decay
    self.epsilon = max(self.epsilon_min, self.epsilon)

    minibatch = random.sample(self.memory, minibatch_size)

    state_batch, q_values_batch = [], []
    for state, action, reward, next_state, done in minibatch:
        # Get predictions for all actions for the current state.
        q_values = self.model.predict(state)

        # If we're not done, add on the future predicted reward at the discounted rate.
        if done:
            q_values[0][action] = reward
        else:
            f = self.target_model.predict(next_state)
            future_reward = max(self.target_model.predict(next_state)[0])
            q_values[0][action] = reward + self.gamma * future_reward

        state_batch.append(state[0])
        q_values_batch.append(q_values[0])

    # Re-fit the model to move it closer to this newly calculated reward.
    self.model.fit(np.array(state_batch), np.array(q_values_batch), batch_size=batch_size, epochs=1, verbose=0)

    self.update_weights()

def update_weights(self):
    weights = self.model.get_weights()
    target_weights = self.target_model.get_weights()

    for i in range(len(target_weights)):
        target_weights[i] = weights[i] * self.tau + target_weights[i] * (1 - self.tau)

    self.target_model.set_weights(target_weights)

And the full notebook is here.

This is called "catastrophic forgetting" and can be a serious problem in many RL scenarios.

If you trained a neural network to recognise cats and dogs and did the following:

Train it for many epochs on a full dataset until you got a high accuracy.
Continue to train it, but remove all the cat pictures.

Then in a relatively short space of time, the NN would start to lose accuracy. It would forget what a cat looks like. It would learn that its task was to switch the dog prediction as high as possible, just because on average everything in the training population was a dog.

Something very similar happens in your DQN experience replay memory. Once it gets good at a task, it may only experience success. Eventually, only successful examples are in its memory. The NN forgets what failure looks like (what the states are, and what it should predict for their values), and predicts high values for everything.

When something bad happens and the NNs high predicted value is completely wrong, the error can be high, and the NN may have incorrectly "linked" its state representations so that it cannot distinguish which parts of the feature space are the cause of this. This creates odd effects in terms of what it learns about values of all states. Often the NN will behave incorrectly for a few episodes but then re-learn optimal behaviour. But it is also possible that it completely breaks and never recovers.

There is lots of active research into catastrophic forgetting and I suggest you search that term to find out some of the many types of mitigation you could use.

For Cartpole, I found a very simple hack made the learning very stable. Simply keep aside some percentage of replay memory stocked with the initial poor performing random exploration. Reserving say 10% to this long term memory is enough to make learning in Cartpole rock solid, as the NN always has a few examples of what not to do. The idea unfortunately does not scale well to more complex environments, but it is a nice demonstration. For a more sophisticated look at similar solutions you could see the paper "The importance of experience replay database composition in deep reinforcement learning"


How to detect the match precision of OneVsRestClassifier I've improved my text classification to topic module, from simple word2vec to piped tfidf and OneVsRestClassifier (using sklearn). It does improve the classification but with word2vec I was able to calculate the match percentage for each topic and with OneVsRestClassifier i get a match or no match to a specific topic. Is there a way to see with OneVsRestClassifier what was the percentage of the classification?

P.S.
I am not talking about evaluating the performance of the training but the actual real time matching percentage.

Yes, of course.

Assuming that you have used sklearn's OneVsRestClassifier and so you have a decision function for example a Support Vector Classifier with say linear kernel. Use set_params to change probability key to True, default is False. Use this in the OneVsRestClassifier classifier and then go with the inbuilt function predict_proba like

from sklearn.multiclass import OneVsRestClassifier
from sklearn.svm import SVC
mod = OneVsRestClassifier(SVC(kernel='linear').set_params(probability=True)).fit(samples,classes)
print mod.predict_proba(np.array([your_sample_vector]).reshape(1,-1))

Edit:

You can use your old LinearSVC with decision_function to find the distance from the hyperplane and convert them to probabilities like

mod = OneVsRestClassifier(LinearSVC()).fit(sample,clas)
proba = (1./(1.+np.exp(-mod.decision_function(np.array(your_test_array).reshape(1,-1)))))
proba /= proba.sum(axis=1).reshape((proba.shape[0], -1))\
print proba

Now you don't need tuning the parameters, I guess. :)


Which GUI library to use with Deep Learning I have completed basics Deep Learning course from coursera using Tensorflow and Keras.
Now I want to apply GUI to it.
So which library should i learn:
1.PyQt
2.Kivy
3.Tkinter
Are there libraries which can help to easily create deep learning projects.

It really depends on how far you want to go.
If you are very serious about GUI apps, PyQt is the only way to go.
Qt5 is the gold standard for cross-platform GUI right now.
But, for basic applications, you are good to go with Tkinter.
I have never used Kivy, and I don't know many people who use it.


Need for Dense layer in Text Classifcation While creating a model for text classification, what is the need for a Dense Layer? I noticed in multiple examples the following is the structure. A softmax is what required right instead of the Dense Layer?
model = tf.keras.Sequential([
    tf.keras.layers.Embedding(encoder.vocab_size, 64),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)),
    tf.keras.layers.Dense(1)
])

Consider the following sentence in 5 class classification:
"movie is good" . The model structure could be:
a = activation_unit
emb= embedding_vector(word)

a0 -> emb("movie") ->a1->emb("is") ->a2->emb("good") ->a3, and

sample_y = softmax(np.dot(Wya,a3))

and
sample_y = [0.1,0.2,0.2,0.4,0.1]

which says the sentence belongs to "class 4". So where is the need for a "Dense Layer"? Can anyone please explain this

In neural networks meant for classification, you need a linear layer before the softmax to project the internal representation, which has some dimensionality $d_i$, to the output space, which has dimensionality $d_o$ equal the number of choices (5 in your case).
So you either place a Dense(5) layer after the BiLSTM or you take the output of the BiLSTM "manually" and implement the projection.
The code above has some strange things:

Uses numpy.dot to multiply the output of the BiLSTM. Is this a typo and you actually meant tf.dot or tf.matmul?
The model ends with a tf.keras.layers.Dense(1), maybe because it was originally meant for binary classification.
Has both a Dense layer and then a dot product (i.e. matrix multiplication). These two operations are equivalent to a single Dense layer, so it is pointless to have both.

So yo answer your question: assuming that the np.dot actually means a tf matrix multiplication, then the Dense layer in the model is pointless.


XGBoost: Can the features in test data be a subset of the features used to train the model? Is it a problem if the test data only has a subset of the features that are used to train the xgboost model? All my predictor variables (except 1) are factors, so one hot encoding is done before converting it into xgb.DMatrix. So the different levels of the factor variables become the features and my test doesn't have all of these features, only a subset of it.

At the moment, while running my model on test data in R, I 'm running into an error saying that "Features names stored in object and newdata are different!".

I'm new in the field, so any help would be much appreciated. Thanks!

All the variables used to train the model must be present in the test set.

This is because you used all the variables to create the rules. Hence we would need those to score them.

If you are using python to do one hot encoding using fit or fit_transform functions in sklearn you will use the same object to transform the test set using transform function.

This will ensure the variables to be consistent in test and train.


Analyzing customer response I'm new to ML. I'm taking over a Classification project which involves analyzing data for customers which returned a product and I need to determine the return reason (~10 categories).
This data was captured at the counter, and could include words like: LGTM (Looks good to me) NFF (No fault found), etc.

I have a training set of 1000 records and when using Google Prediction API I get a "classificationAccuracy" value of "0.82" and 10 labels.

Questions:
1. Any recommended API to analyze this type of data?.
2. What is a good  "classificationAccuracy" value?

Thank you

http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html

You can use the above tutorial to get acquaintance with text classification. Afterwards it should be easier to formulate nontrivial questions to move even further.


how to compare two groups I need to compare two groups of people, where the independent variable is having a college degree, and the dependent variable is the income.

The problem is that if I divide the whole population of the study into two groups,
one of them has signally more persons, so the mean is affected by the size of each group and the value of the outliers.

How can I demonstrate or reject the theory that having a college degree
among the population of the study, ensures a better income?

I assume you are performing an independent samples t-test. That the Ns are different isn't necessarily a problem -- the mean is an unbiased estimator -- but just how different are the sample sizes?
As you describe, you are probably violating the homogeneity of variance assumption.

T-tests tend to be fairly robust to this violation, but you might consider having some outlier treatment and/or using a nonparametric test.


Approach to semantic similarity between documents I was wondering what approach people would take, or point me in the right direction on this challenge I have set myself. I am pretty new at this, I have covered some area but want to expand my skillset.

Say you have an abstract from a research paper, which is a summarised form of information of a larger document, can you calculate, from a list of papers, which research paper this abstract belongs to?

Please note: I am not asking how to summarise the research paper. Also, note that the abstract information doesn't necessarily take the same form as the research paper, but is semantically similar.

Would you, encode both datasets with a something like doc2vec to try to get the semantic meaning of the texts and then use cosine similarity?

Would the semantics of the numbers used in these papers get lost in the vectorizing?

IN this case, would a custom encoder work the best, or do you think USE or doc2vec would fair better?

Forgive my basic questions, I just wanted to explore things before I started coding!

If I understand correctly, you're trying to map abstracts to their research papers.

Here is a simple starting point:

Compute a TF IDF model using the entire corpus (all abstracts + research papers). Use this model to transform your abstracts and research papers into a weighted vector representation. Under the TF IDF weighting scheme, these documents will be represented by vectors that point in the direction of the words that are most discriminating for them. Put another way, these vectors point towards the words that are the best at telling you what the document is about. So if two vectors are close to each other, then the two documents are likely to be about the same topic because they are using similar words.

This is where cosine similarity comes in. Take an abstract, and then iterate across all the research paper vectors, computing the cosine similarity between them. Then, map that abstract to whichever research paper has the highest cosine similarity. Repeat this for each abstract.

Once you have this simple baseline approach, you can start looking at more sophisticated models for capturing semantic similarity.


How do we define a linearly separable problem? When we talk about Perceptrons, we say that they are limited for approximating functions that are linearly separable, while Neural Networks that use non-linear transformations are not.

I am having trouble understanding this idea of linear separability. Specifically, does it apply only for binary classification or it generalizes for N classes? If so, how does a linear decision boundary for lets say 4 classes even look like?

1.) Perceptron is a non-linear transformation!
2.) Linear seperable function is only defined for boolean functions, see Wikipedia.
Therefore, yes, the statement is meant only for binary classification.
3.) For general functions, see the universal approximation theorem.


If you print those shapes using below for loop

weights_m=model.get_weights()
for i in range(8):
  print(weights_m[i].shape)

you will get output as

(3, 3, 1, 32)
(32,)
(3, 3, 32, 64)
(64,)
(9216, 128)
(128,)
(128, 10)
(10,)

so we will get one layer weight and bias. we have a total of 4 layers(2 conv + 2 dense) so 8 weight vectors.


PySpark v Pandas Dataframe Memory Issue Suppose I have a csv file with 20k rows, which I import into Pandas dataframe. I then run models like Random Forest or Logistic Regression from sklearn package and it runs fine. However, when I import into PySpark dataframe format and run the same models (Random Forest or Logistic Regression) from PySpark packages, I get a memory error and I have to reduce the size of the csv down to say 3-4k rows. Why does this happen? Is this a conceptual problem or am I coding it wrong somewhere?

For Pandas dataframe, my sample code is something like this:

df=pd.read_csv("xx.csv")

features=TfIdf().fit(df['text'])
....
RandomForest.fit(features,labels)

And for PySpark, I'm first reading the file like this:

data = sqlContext.read.format('com.databricks.spark.csv').options(header='true', inferschema='true')\
.load('xx.csv')
data.show()
from pyspark.ml.feature import RegexTokenizer, StopWordsRemover, CountVectorizer
from pyspark.ml.classification import LogisticRegression
# regular expression tokenizer
regexTokenizer = RegexTokenizer(inputCol="converted_text", outputCol="words", pattern="\\W")
# stop words
add_stopwords = ["http","https","amp","rt","t","c","the"]
stopwordsRemover = StopWordsRemover(inputCol="words", outputCol="filtered").setStopWords(add_stopwords)
# bag of words count
countVectors = CountVectorizer(inputCol="filtered", outputCol="features", vocabSize=10000, minDF=5)
from pyspark.ml import Pipeline
from pyspark.ml.feature import OneHotEncoder, StringIndexer, VectorAssembler
label_stringIdx = StringIndexer(inputCol = "Complaint-Status", outputCol = "label")
pipeline = Pipeline(stages=[regexTokenizer, stopwordsRemover, countVectors, label_stringIdx])
# Fit the pipeline to training documents.
pipelineFit = pipeline.fit(data)
dataset = pipelineFit.transform(data)
dataset.show(5)
(trainingData, testData) = dataset.randomSplit([0.7, 0.3], seed = 100)
print("Training Dataset Count: " + str(trainingData.count()))
print("Test Dataset Count: " + str(testData.count()))
from pyspark.ml.classification import RandomForestClassifier
rf = RandomForestClassifier(labelCol="label", \
                            featuresCol="features", \
                            numTrees = 100, \
                            maxDepth = 4, \
                            maxBins = 32)
# Train model with Training Data
rfModel = rf.fit(trainingData)
predictions = rfModel.transform(testData)
predictions.filter(predictions['prediction'] == 0) \
    .select("converted_text","Complaint-Status","probability","label","prediction") \
    .orderBy("probability", ascending=False) \
    .show(n = 10, truncate = 30)

I was trying for lightgbm, only changing the .fit() part:

from mmlspark import LightGBMClassifier
lgb = LightGBMClassifier(learningRate=0.3,
                           numIterations=100,
                           numLeaves=31)
lgb_model=lgb.fit(trainingData)
predictions = lgb_model.transform(testData)
predictions.filter(predictions['prediction'] == 0) \
    .select("converted_text","Complaint-Status","probability","label","prediction") \
    .orderBy("probability", ascending=False) \
    .show(n = 10, truncate = 30)
from pyspark.ml.evaluation import MulticlassClassificationEvaluator
evaluator = MulticlassClassificationEvaluator(predictionCol="prediction")
evaluator.evaluate(predictions)

And the dataset has hardly 5k rows inside the csv files. Why is it happening? How can I solve it?

While I can't tell you why Spark is so slow (it does come with overheads, and it only makes sense to use Spark when you have 20+ nodes in a big cluster and data that does not fit into RAM of a single PC - unless you use distributed processing, the overheads will cause such problems. For example, your program first has to copy all the data into Spark, so it will need at least twice as much memory. Probably even three copies: your original data, the pyspark copy, and then the Spark copy in the JVM. In the worst case, the data is transformed into a dense format when doing so, at which point you may easily waste 100x as much memory because of storing all the zeros).

Use an appropriate - smaller - vocabulary.

There is no use in including every single word, as most of them will never score well in the decision trees anyway!

It's safe to assume that you can omit both very frequent (stop-) words, as well as rare words (using them would be overfitting anyway!). So use min_df=10 and max_df=1000 or so.


rectangular markers in bubble plot (Python) I would like to make a 'bubble' plot, but with rectangular markers. Is it possible to do this in Python?

On x-axis should be the day of the week (Mon, Tue, Wed,...), on y-axis - counts, how many times people came to a restaurant, the area of rectangles - the number of people came to a restaurant. A height of rectangles is always the same, only a width changes.

So, one rectangle on the plot shows how many people how many times came to a restaurant on a particular day of the week. As axis y may have values > 30, bubbles are not suitable and I consider that rectangles works better.

A sketch of what I would like to draw is below.

Is it possible to draw rectangular markers or is there another good way for visualizing the same concept?

UPDATE:
I've already tried box and violin plots.
The problem with boxplots is that in my case a height of a box is quite small and a tail is long. There are a lot of points indicating outliers. I would prefer to see some difference in outliers instead of a single point. With a bubble plot, it is possible to use some customized function for bubble size, e,g, logarithm.

I can imagine a transposed boxplot() working well... in a figure with seven subplots...

https://github.com/steadfordium/ga_pre/blob/master/bar_as_rectangular_bubble.ipynb


How can I draw bar graph in python on aggregated data? Normally when I draw bar plot its simple as

import matplotlib.pyplot as plt
from pylab import rcParams
import seaborn as sb

%matplotlib inline
rcParams['figure.figsize'] = 5, 4
sb.set_style('whitegrid')

x = range(1, 10)
y = [1,2,3,4,0.5,4,3,2,1]

plt.bar(x, y)

When I aggregate data on basis of age feature with the following command

data_ag = data.groupby('age')['age'].count()

It returns the output that is the people belonging to a particular age e.g. 11 people with age 22 years.

age
22     11
23      8
27     28
28      1
29     70
30     13
31     45
Name: age, dtype: int64

How can I treat those as x and y points to draw a bar plot?

x = # what should I write here for age data
y = # what for count

plt.bar(x, y)

import pandas as pd

df = pd.Series(data=[11, 8, 28, 1, 70, 13, 45],
               index=[22, 23, 27, 28, 29, 30, 31],
               name="age").rename_axis("age", axis=0)

x = df.index
y = df.values

plt.bar(x, y)


Should we convert independent continous variables (features) to categorical variable before using decision tree like classifier? Consider I have one dependent variable to predict 'Attitude' which can take three values 'Positive/Negative/Neutral'.
I have following independent variables or features- Age, Height, Gender, Income etc. I trying to predict Attitude using decision tree classifier.

Attitude ~ Age + Height + Gender + Income (Decision Tree)

I am getting >90% accuracy for the when tree depth is 15. As tree is dividing on continuous variables (i.e. Age, Income and Height) again and again to get leaf with pure classes.

Is this problem of overfitting? Should I convert the continuous variables into categorical variables (like range classes)?

There is no need to split continuous variables because the tree already does that automatically. The only way you can test for overfitting is by either using a holdout set or by doing cross validation. If you are overfitting, changing a continuous variable to a categorical variable likely won't make a difference. If you get the sense that you're overfitting, you should reduce the depth of your tree.


Predictive model to maximize sum of dependent variable? I am trying to classify cars for a towing company. Junky  cars earn more when  sent to the junkyard, and the more valuable cars should earn more at the auction, despite the auction fee.  Creating a logistic regression that takes into account Make, Model, Mileage,  Year and Run status helps us improve the accuracy of which cars should go where, but a difficulty arises: Sometimes, a car that would be classified as junk can actually be an outlier, and sell for a lot of money. So to optimize our model, we don't really care that much whether we are right or wrong on an individual car, so much as we maximize our bottom line.
All of the models I have seen (Logistic regression, RF, linear regression) make predictions on a line by line basis.
What would be a good model to try and maximize the aggregate sum of the predictions?
Below is a reprex of my data, as well as basic code I used.
What I actually tried until now is to look at past data, and classify, in hindsight, what should have been done, based on prices that were earned in the auction vs available junk prices.  I then ran a glm against that classification to predict the future. As mentioned above, my code improved the accuracy of our decisions, and would have sent more cars to junk correctly, but  some  that we classified as junk sold for so much in the auction that  it wasn't worth sending any to junk.
What is the proper way to approach this?
cars <- structure(list(YearOfCar = c(2009L, 2009L, 2003L, 2004L),
    Make = c("Hyundai", "Lexus", "Ford", "Toyota"), Model =
    c("Sonata", "GS 350", "F-250 Super Duty","Camry"), PickUpState =
    c("MN", "LA", "MA", "NJ"), Auction_Result = c(650,625,425, 1500),
    Auction_Fee = c(144.25, 373.54, 213.5, 187), Mileage = c(116120L,
    198900L, 140241L, 312927L), Runs = structure(c(1L, 1L, 1L, 2L),
    .Label = c("No", "Yes"), class = "factor"),   junkyard_Offer =
    c(230L, 235L, 140L, 300L), Date = structure(c(17592,  17707,
    17674, 17583), class = "Date")), row.names = 3:6, class =
    "data.frame")
cars$hindsight <- ifelse(cars$Auction_Result-
    cars$Auction_Fee>cars$junkyard_Offer,1,0)
glmodel <- glm(hindsight~Make+Model+Mileage+Runs, data = cars,
    family="binomial")
prediction <- predict(glmodel, cars, type = 'response')
prediction_classifier <-  if_else(prediction>.501,1,0)
cars$prediction_results <- ifelse(prediction_classifier==1,
    cars$Auction_Result-cars$Auction_Fee,cars$junkyard_Offer)

Interesting problem which potentially involves many aspects of ML, here are a few thoughts:

At first sight I thought that this looks more like an optimization problem, not a regular classification problem. In this case I would suggest to maybe try things like genetic learning, because it can find an optimal assignment for individual elements which maximizes a global cost or reward function.
However I suspect that the problem is not entirely well defined: "but some that we classified as junk sold for so much in the auction that it wasn't worth sending any to junk.": I'm clueless about the business model but wouldn't it damage the reputation of the company if they were trying to sell all the junk cars at an auction? If I'm right then it matters that only a small proportion of cars are sent to the auction, and then the problem might be related to resource allocation (which is a specific kind of optimization problem).
Another way to look at it is that the vast majority of the cars are junk. Thus it might be relevant to see the problem as anomaly detection, in the sense that one tries to pick the rare cases out of a sea of regular cases. The same idea could be implemented as one-class classification, the idea being that the model can identify all the common cases (junk) and anything else is potentially valuable.

Finally it's certainly worth investigating in the data if the valuable cars can actually be found from the features: would a human expert with only the information in the features be able to correctly classify a car? I could imagine that for instance if a particular car is valuable because it was used in a famous movie, it doesn't help to just know its model and mileage.  It would also be useful to check the relation between how common a car model is and its junk/valuable status, this could be an important indicator to take into account in the model via a feature. In the most simple case, it might even be possible to detect potentially valuable cars just by looking at this indicator... in which case there's no need for ML at all.


Making a labelled training data set We are developing a classification system, where the categories are fixed, but many of them are inter-related.

For example, we have a category called, "roads" and another one called "traffic". We believe that the model will be confused by the text samples, which could be in roads category and also in traffic.

Some of our text samples are suitable for multi class labelling too.  For example, "There is a garbage dump near the footpath. The footpath is broken completely". This text could be categorized into garbage bucket or footpath bucket.

We are going to build a training set for this classifier, by manually annotating the text. So, can we put multiple labels for one issue? How should we deal with  text with multiple labels for it? Should they be added into all categories to which it is tagged to, as training sample ?

For example, "There is a garbage dump near the footpath. The footpath is broken completely". This text could be categorized into garbage bucket or footpath bucket. So, should this text be added as a training sample for garbage and footpath? How should we consider the labels?

Can you please give your insights?

Generally with multiple classes you have to make a distinction between exclusive and inclusive groups. The simplest cases are "all classes are exclusive" (predict only one class), and "all classes are compatible" (predict list of classes that apply).

Either way, label the classes as you would want your trained model to predict them. If you expect your classifier to predict an example is in both garbage and footpath, then you should label such an example with both. If you want it to disambiguate between them, then label with a single correct class.

To train a classifier to predict multiple target classes at once, it is usually just a matter of picking the correct objective function and a classifier with architecture that can support it.

For example, with a neural network, you would avoid using a "softmax" output which is geared towards predicting a single class - instead you might use a regular "sigmoid" function and predict class membership on a simple threshold on each output.

You can get also more sophisticated perhaps with a pipeline model if your data can be split into several exclusive groups - predict the group in the first stage, and have multiple group-specific models predicting the combination of classes in each group in a second stage. This may be overkill for your problem, although it may still be handy if it keeps your individual models simple (e.g. they could all be logistic regression, and the first stage may gain some accuracy if the groups are easier to separate).


Should you perform feature scaling/mean normalization when predicting with a model? Should you also perform feature scaling/mean normalization when predicting with a model, that was trained and tested on with feature scaling/mean normalization?

Yes, testing data should follow the same preprocessing as the training data. Otherwise, testing data will have nothing comparable with what the algorithm learned, leading to (very) bad performances.

note: In Sklearn, the Pipeline class helps you to respect the fundamentals of ML modeling like data leakage and applying the same transformations to train and test set.


how to calculate quadratic features in computer vision Neural Network I am recently watching some tutorials for deep learning from Dr Andrew Ng on Youtube. Link is hereThe Youtube video

There is a concept of number of features in convolutional neural network in TensorFlow's tutorial.

I don't quite understand why the feature is 32 or 64 here in conv layer1 or layer2?

Then I came to the video, there is also the concept of Quadratic features. It is calculated as 3 million however. But how is it calculated?

Are the two features related in concept?

Sorry for being so late in the response. I have just read your message.

The instructor is using $\frac{x^2}2$. So $\frac{2500\times 2500}2$, and this will get approx $3$ millions features.

BR


which Model to apply on panel data where unique id has 6-8 records and total records are 2,000,000? I am new to such panel data where I have multiple observation for same ID in different Quarter and I am not sure what kind of machine learning algorithm I can apply.
I have data from Q1-18 till Q4-2020
I have 2,000,000 rows and 200,000 unique id and 20 columns
For each id I have only 6-8 past quarter values, max quarter for each id are 8 quarters and for some id I have only 6 quarters where few quarter value are not available for that id
Below is the basic idea of what my data set look like
Quarter - respective business quarter for that year
Target - is the sales volume in ratio
I am trying to Predict - Target column for 2021 Q1 quarter
I have 8-10 different numeric columns and state , quarter and ID as category columns
I would appreciate if someone could suggest me what kind of modelling could be performed

Your data is a good candidate for a mixed effect model.
You have two potential random effects that look to be crossed: I see ID as one and state as another, as it seems that any ID can belong to any state.
Essentially, you can leverage the fact that some states and some IDs will have less data than others (and in general, some states and IDs will vary more than others). States and IDs who exhibit such characteristics will have their predictions bolstered by other states and IDs (called partial pooling). You also need to use time as a variable (perhaps convert quarter to 2019.25, 2019.5, etc. or use two time variables year, quarter) because you wish to predict the next period.
I think generalized linear mixed effect models can scale to 2 million observations. There are also machine learning methods that allow for mixed effects: mixed effect random forests (MixRF), mixed effect gradient boosting (GPBoost/mboost), and of course you can go full Bayesian as well (though you will likely need to use an approximation method since your data is large). The ML models may scale better.


Log scaling whole dataset I am log scaling my whole dataset in log base 10. When I do this I get -infitity for the minimum value.
I am wondering how I can get rid of this -infinity?
I have been advised to add a small value number inside the log to get rid of this like:
log10(dataset + 0.01)

This does get rid of the -infinity but I'm unsure does this skew the data or is it the wrong thing to do?
Also,if this is the right method, does anyone have any tips on how to decide what number to use e.g. 0.01, 0.1, or 0.001?

I would recommend taking a look at the transformation you did:

Consider whether you would want to add a really small number like 0.0000001 to zero before taking the log. Probably not, as you will get a large negative number. However, there should be little to no difference in results once you start adding large enough numbers like 0.1. I would recommend trying each and looking at a density plot of the resulting data. Investigate which distribution looks the least skewed and go with it.


Why is there such a mismatch between the Model's predicted probability and theoretical probability in logistic regression? I am trying to do Logistic Regression using SAS Enterprise Miner.
My Independent variables are

CPR/Inc (Categorical 1 to 7)
OD/Inc (Categorical 1 to 4)
Insurance (Binary 0 or 1)
Income Loss (Binary 0 or 1)
Living Arrangement (Categorical 1 to 7)
Employment Status (categorical 1 to 8)

My Dependent Variable is Default (Binary 0 or 1)

The following is the output from running Regression Model.

Analysis of Maximum Likelihood Estimates

                                  Standard          Wald
Parameter       DF    Estimate       Error    Chi-Square    Pr > ChiSq    Exp(Est)

Intercept        1     -0.4148      0.0645         41.30        <.0001       0.660
CPR___Inc  1     1     -0.8022      0.1051         58.26        <.0001       0.448
CPR___Inc  2     1     -0.4380      0.0966         20.57        <.0001       0.645
CPR___Inc  3     1      0.3100      0.0871         12.68        0.0004       1.363
CPR___Inc  4     1    -0.00304      0.0898          0.00        0.9730       0.997
CPR___Inc  5     1      0.1331      0.0885          2.26        0.1324       1.142
CPR___Inc  6     1      0.1694      0.0881          3.70        0.0546       1.185
Emp_Status 1     1     -0.2289      0.1006          5.18        0.0229       0.795
Emp_Status 2     1      0.4061      0.0940         18.66        <.0001       1.501
Emp_Status 3     1     -0.2119      0.1004          4.46        0.0347       0.809
Emp_Status 4     1      0.1100      0.0963          1.30        0.2534       1.116
Emp_Status 5     1     -0.2280      0.1007          5.12        0.0236       0.796
Emp_Status 6     1      0.3761      0.0943         15.91        <.0001       1.457
Emp_Status 7     1     -0.3337      0.1026         10.59        0.0011       0.716
Inc_Loss   0     1     -0.1996      0.0449         19.76        <.0001       0.819
Insurance  0     1      0.1256      0.0559          5.05        0.0246       1.134
Liv_Arran  1     1     -0.1128      0.0916          1.52        0.2178       0.893
Liv_Arran  2     1      0.2576      0.0880          8.57        0.0034       1.294
Liv_Arran  3     1      0.0235      0.0904          0.07        0.7950       1.024
Liv_Arran  4     1      0.0953      0.0887          1.16        0.2825       1.100
Liv_Arran  5     1     -0.0493      0.0907          0.29        0.5871       0.952
Liv_Arran  6     1     -0.3732      0.0966         14.93        0.0001       0.689
OD___Inc   1     1     -0.2136      0.0557         14.72        0.0001       0.808
OD___Inc   2     1     -0.0279      0.0792          0.12        0.7248       0.973
OD___Inc   3     1     -0.0249      0.0793          0.10        0.7534       0.975

Now I used this Model to Score a new set of data. An example row of my new data is

CPR - 7
OD - 4
Living Arrangement - 4
Employment Status - 4
Insurance - 0
Income Loss - 1

For this sample row, the model predicted output (Probability of default = 1) as 0.7335
To check this manually, I added the estimates

Intercept + Emp Status 4 + Liv Arran 4 + Insurance 0
-0.4148   + 0.1100  +   0.0953   +   0.1256    =   -0.0839

Odds ratio = Exponential(-0.0839) = 0.9195

Hence probability = 0.9195 / (1 + 0.9195)  =   0.4790

I am unable to understand why there is such a mismatch between the Model's predicted probability and theoretical probability.

Any help would be much appreciated .
Thanks

It's probably due to the effect coding of categorical predictors. Eg, the regression coefficient for CPR = 7 is not zero but -(sum of regression coefficients for the other 6 levels). I guess EM should have an option to switch it to reference coding, then your way of computing the predicted probability should work.


For text match problem, what is the different between question-question match and question-answer match? I know question-question match is a text similarity problem.

What about question-answer or question-doc match? It is used in information retrieval.

question-question match is indeed text similarity. But how do you define question-answer similarity?

Thank you!!

You can refer to the paper "A Deep Look into Neural Ranking Models for
Information Retrieval" to get more discussion about different matching tasks.


Paper: What's the difference between Layer Normalization, Recurrent Batch Normalization (2016), and Batch Normalized RNN (2015)? So, recently there's a Layer Normalization paper. There's also an implementation of it on Keras.

But I remember there are papers titled Recurrent Batch Normalization (Cooijmans, 2016) and Batch Normalized Recurrent Neural Networks (Laurent, 2015). What's the difference between those three?

There is this Related Work section I don't understand:

  Batch normalization has been previously extended to recurrent neural networks [Laurent et al., 2015, Amodei et al., 2015, Cooijmans et al., 2016]. The previous work [Cooijmans et al., 2016] suggests the best performance of recurrent batch normalization is obtained by keeping independent normalization statistics for each time-step. The authors show that initializing the gain parameter in the recurrent batch normalization layer to 0.1 makes significant difference in the final performance of the model. Our work is also related to weight normalization [Salimans and Kingma, 2016]. In weight normalization, instead of the variance, the L2 norm of the incoming weights is used to normalize the summed inputs to a neuron. Applying either weight normalization or batch normalization using expected statistics is equivalent to have a different parameterization of the original feed-forward neural network. Re-parameterization in the ReLU network was studied in the Pathnormalized SGD [Neyshabur et al., 2015]. Our proposed layer normalization method, however, is not a re-parameterization of the original neural network. The layer normalized model, thus, has different invariance properties than the other methods, that we will study in the following section

Layer normalization (Ba 2016): Does not use batch statistics. Normalize using the statistics collected from all units within a layer of the current sample. Does not work well with ConvNets.
Recurrent Batch Normalization (BN) (Cooijmans, 2016; also proposed concurrently by Qianli Liao & Tomaso Poggio, but tested on Recurrent ConvNets, instead of RNN/LSTM): Same as batch normalization. Use different normalization statistics for each time step. You need to store a set of mean and standard deviation for each time step.
Batch Normalized Recurrent Neural Networks (Laurent, 2015): batch normalization is only applied between the input and hidden state, but not between hidden states. i.e., normalization is not applied over time.
Streaming Normalization (Liao et al. 2016) : it summarizes existing normalizations and overcomes most issues mentioned above. It works well with ConvNets, recurrent learning and online learning (i.e., small mini-batch or one sample at a time):
Weight Normalization (Salimans and Kingma 2016): whenever a weight is used, it is divided by its $L2$ norm first, such that the resulting weight has $L2$ norm $1$. That is, output $y = x*(w/|w|)$, where $x$ and $w$ denote the input and weight respectively. A scalar scaling factor $g$ is then multiplied to the output $y = y*g$. But in my experience $g$ seems not essential for performance (also downstream learnable layers can learn this anyway).
Cosine Normalization (Luo et al. 2017): weight normalization is very similar to cosine normalization, where the same $L2$ normalization is applied to both weight and input: $y = (x/|x|)*(w/|w|)$. Again, manual or automatic differentiation can compute appropriate gradients of $x$ and $w$.

Note that both Weight and Cosine Normalization have been extensively used (called normalized dot product) in the 2000s in a class of ConvNets called HMAX (Riesenhuber 1999) to model biological vision. You may find them interesting.

Ref: The HMAX Model Reference

Ref: Cortical Network Simulator Reference

Ref: Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks,  Luo Chunjie, Zhan jianfeng, Wang lei, Yang Qiang


Which accuracy metric of a ML classifier can maximize map@K of a recommender system for an unbalanced dataset? I have to build a recommender system & it will be evaluated using map@10 criteria.
I have rolled up the data/rows at user-item level & is using Gradient Boosting in scikit learn to build the model.

Challenge

But there is a big class imbalance where only 6%( out of 40 K use-item pairs) have actually purchased the item in the entire base. Given this challenge i want to know the accuracy metric I should use to tune the parameters of GBM model.

Question

I thought about using recall, precision, f1 score etc. to tune the hyperparameters of GBM. But which one will maximize map@10 given the class imbalance?

Thanks

If you want to stay in Python, I would suggest going for xgboost instead of GradientBoostingClassifier - among other advantages, it supports map / ndcg as metrics. If you are considering R, gbm supports both map and ndcg.


Street address clustering? I have a huge dataset of addresses. I have another data stream that contains addresses that I need to match against those in the original dataset. As all the addresses are user-provided, matching them is not trivial. For example, 10 John Smith Square could be represented as 10 J.S. Square, J. Smith Sq 10, possibly even as only John Smith (which obviously misses the number and on top of that can collide with John Smith Street).

I am wondering if there are some known ways to reducing this ambiguity. My intuition is that some clustering should be possible, allowing me to match incoming addresses from the second dataset to the clusters of the first. I want to get to clusters that separate different street numbers, rather than put all addresses on a single street together.

I'm recently considering employing some sort of (possibly sparse) autoencoders, but was not able to find any literature to suggest that anyone else has tried a similar approach. So my question is, how can I employ autoencoders to cluster street address labels if anyone has tried anything of the sort? Otherwise, what alternative approaches would you suggest.

I've already have a dictionary to reduce the vocabulary by trivial substitutions (e.g. Square to Sq), but it remains to be tested whether this actually improves results or not, depending on what further analysis is applied after that. My current representation of the address strings is with bag-of-words, as I don't think word order changes much.

My instinct would say to start with a standard dataset and map all addresses ‒ both from your first and second datasets ‒ to that. In the standard dataset, each physical location would have a unique address.

The obvious source for the standard dataset is OpenStreetMap. You can query and dowload data from it using the Overpass API. Its query language, just like the whole OSM project, is well documented in my experience. Overpass Turbo is the user-friendly way with a graphical user interface to get started with the querying. Once you have developed a query, you can download data using e.g. wget, you just have to convert the query into the right URL at the Overpass API Convert Form by entering it into the window Overpass API Convert Form and selecting compact Overpass QL.

  wget --timeout=0 http://overpass-api.de/api/interpreter?data=[your query in compact Overpass QL form] -O filename

In OSM, some addresses are associated with buildings, which are ways (closed polygons), and some are nodes inside the buildings' outlines, so that's a bit of complication.

This all sounds doable. However, the OSM is not complete, so it very much depends on your area of interest whether it's worthwhile to use OpenStreetMap.


Best way to fix the size of a sentence [Sentiment Analysis] I am working on a project that is about Natural Language Processing. However I am stuck at the point which is I have a ANN that has fixed size of input neurons.

I am trying to do sentiment analysis with using Imdb movie review set. To able to do that, firstly, I calculated the word embeddings for each word with creating a word-context matrix then applied SVD. So I have the word embedding matrix. But I do not know the best way to compress sentence's vector (which contains embeddings for each word in the sentence) into a fixed size to be able to feed the neural net. I tried PCA but result was not satisfying.

Any help?

The easiest way is to average the word- embeddings. This works quite well.
Another thing you can try is to represent each document as a bag of words - i.e. - to have a vector in the size of your vocabulary, where each element in the vector represents the number of times a certain word had been mentioned in your document (for example, the first element in the vector will represent how many times the word a was mentioned, and so on).
Afterwords, to reduce the size of the vector you can use techniques like LDA, SVD, or autoencoders.


Find periodicity of a signal using python I have a dataset that contains occurrences of the Kettle in a single-occupancy house for the duration of a month. In this dataset, ‘ts’ column indicates the unix-timestamp (this can be converted to date and
time) that the kettle was turned ON; and the ‘second’ column indicates the duration in seconds that the kettle was ON.
My requirement - Find routines in the data, evaluate their periodicities and variabilities. For instance, a routine could be “breakfast”, its periodicity might be every day at a certain time and variability might be a time window within the routine occurred.
What I've tried and I'm a bit unsure of - I plotted the autocorrelation of the virgin signal with different amounts of lag and found there was no significant correlation with any of its lagged versions. But on looking at the plot of this signal, I see there are peaks every day- morning time. Also, I've tried autocorrelation on the denoised(denoised using KNN) version and the under-sampled versions(retaining just first and last samples for each day of the month) and found that the denoised signal also isn't correlated with its lagged versions, but the under-sampled signal is.
I'm unsure first of all about how to find periodicity of a signal like this, and if the autocorrelation plot lets us make any conclusions about periodicity.
from statsmodels.tsa.stattools import acf
from statsmodels.graphics.tsaplots import plot_acf

# Compute the acf array
acf_array = acf(q4['seconds'])
print(acf_array)

# Plot the acf function
plot_acf(q4['seconds'],alpha=0.05)
plt.show()

More code showing what I've read can be found at - this link.

One option could be applying the Fourier transform which transforms time to the frequency domain. It is often easy to find recurring patterns when signals are represented in the frequency domain.


bagging vs. pasting in ensemble learning I am bit confused about two concepts.
From my understanding Bagging is when each data is replaced after each choice. so for example for each subset of data you pick one from population, replace it then pick one again, etc... and this is repeated for each subset of data.
But for pasting people say it is sampling without replacement however does that mean you can't have same data on any subset? I thought it picks one subset w/o replacement but when it replaces all data when picking another subset, no?

Actually I think that you are mostly correct, except that in my understanding "with/without replacement" applies only to selecting one subset, not across subsets. This means that if we have a training set of instances $T=\{t_1,..,t_N\}$:

With bagging, a particular sample can contain duplicate instances, in other words it is not a subset of $T$ but a multiset where an instance $t_i$ may occur several times. Of course the probability to pick the same instance $t_i$ $n$ times decreases quickly when $n$ increases, depending on the size $N$.
With pasting a sample is a subset of $T$ and cannot contain the same instance $t_i$ twice. However another sample is drawn again from the full set $T$, which means that it is possible for an instance to be selected in several different samples.

In theory, the size of a sample can be higher than $N$ with bagging but not with pasting.
Note that I'm referring to multiset for the sake of clarity but formally $T$ itself is not a set since in theory it may contain the same instance twice.


Regression and Machine learning models for panel data I have a panel data which consists of the sales of different companies across different years. Can someone tell me what kinds of regressiona and ML methods we use for forecasting the sales?

Thanks

Sales data is Time-series data. You should use the method of Time-series forecasting.

Below is a list of useful links to start with -

Analytics Vidhya
Yashuseth blog
Kaggle blog

This is a good book to grasp the concepts -
Forecasting: Principles and Practice

Also, do check this similar question.
Stack Exchange


Is Mac OS the most suitable OS for a beginner Data Scientist? As far as I understand a data scientist should learn both Excel and Bash. And the golden mean is this case is Mac OS, because it has both. And later I can install Ubuntu on Macbook.

You will not find the most suitable OS for data science. It will come down to preference and what tools you will use. Data science is broad, what tools people work with will vary, some will have support in one OS and less in another.

All OSs are suitable and used by data scientists. Often you don't get to pick yourself. But very few things are exclusive to one OS, so it all works out. I would say you should pick the one you are most comfortable with. You want to spend more time learning data science and less time learning a new OS.


Does the performance of GBM methods profit from feature scaling? I know that feature scaling is an important pre-processing step for creating artificial neural network models.
But what about Gradient Boosting Machines, such as LightGBM, XGBoost or CatBoost? Does their performance profit from feature scaling? If so, why and how?

Scaling doesn't affect the performance of any tree-based method, not for lightgbm, xgboost, catboost or even a decision tree.
This post that elaborates on the topic, but mainly the issue is that decision trees split the feature space based on binary decisions like "is this feature bigger than this value?", and if you scale your data, the decisions might look different, as they are done on the scaled space, but the results should be the same.
As an example, a decision tree should split in the same way the data if you change units (a particular case of scaling). Let's say you want to use weight of a person to predict if someone is under 18 (as a binary classifier). If given the weight in grams, the decision tree might do something like: if weight < 5000 grams, then the person is under 18. If you change the units to kilograms, the decision tree will do: if weight < 5 kg, then the person is under 18.
To sum up, the splits will be equivalent under any scaling or, in general, any increasing linear transformation.


Is it bad to have a lot of one class of Data [K-NN classifier]? I am trying to train a sklearn K-NN classifier on a labeled text dataset (in Irish). There are 5 classes, 0-4, but there is a lot of variation between how many there are in each class.
What I have done is I've gotten a corpus of Irish text, iterated through every word and stripped a few letters from it based on a linguistic form it took (or not). The problem is, class 4 (which means no action was performed) accounts for 16.5M out of 20.1M entries and it goes all the way down to class 3 with only 36,000 entries.
Gathering more data probably won't help as this basically represents the proportion of times these forms of words appear in real life.
Is this bad for classification and will it bias the classifier in any way? If it does, is that bias actually of help?
Any help is appreciated.
Justin

Just imagine it practically, If class A data is 90% and class B data is 10% ,then if you just randomly classify the label you prediction as class A then your accuracy will be 90%.
So biased data will lead your model to be biased over the class which has more data as it will give better predictions in your model.


Extract features from Decision tree leaf nodes Recently came across a coursera course on "How to win Kaggle competitions" where they explain how we can engineer a categorical feature from each leaf node of the decision tree.

[Video Link][1]

I cannot understand this concept. Any suggestion or pointers towards understanding  this will be great.

For example assume the following random training data:

Gender  Age Sample_Ftre

 M      25   1.5
 M      26   1.5
 F      28   1.5
 F      27   1.5
 M      26   1.5

Can anyone explain what will be the value of new engineered_feature from the decision tree and how to calculate it.

I will describe my personal approach to achieve what is taught in the Course you mentioned, a mind blowing course I think!

To generate DT features for your training set:

First of all, split your training data in k-folds. It is necessary to avoid overfitting. Then:

Hide a fold and build a DT using the (k-1) remaining folds. Control the number of leaf nodes in your DT using the max_leaf_nodes paramenter in sklearn. Set it as the number of categories (levels) you want in your new categorical feature.
Predict output for the held out fold. When you are predicting using a DT, each observation will fall in a leaf. So, the values for your new feature will be the index of the leaf each observation ends up in. You can get this index using the apply command in sklearn.

Repeat these 2 steps for each one of the k folds.

To generate DT features for your test set:

Build a DT using the entire training set. Don't forget to set the max_leaf_nodes parameter as the same used above.
Predict output for the test set. Get the leaf index for each test observation.

Now you have a new categorical feature containing the leaf indexes in the predictions of the DT.

Problems with this procedure

Since you cannot control the sequence of splits in each DT, the indexes found in each CV fold may no match. For example, category "0" found in the first fold may not be same category "0" found in the second fold.
At the same time, it is not recommended (following the advises in the Course) building a DT using the entire training set and then predicting leaf indexes for the training and test sets using the same DT. This may lead to overfitting, because you are introducing target information in your training set.

So, what can we do?

Instead of using the leaf index as the feature value you can count the number of nodes each observation goes through until it falls in a leaf, and use it as the feature value. In single words, it is the path length from the root node to the leaf node. I think it is a more informative value. This is the principle behind Isolation Forest.


Why LSTM models do not require labels for each step? For time related problems like, for example, stock prediction:

Let's say we have 300 days of data, 10 features, and one target: the price.

Why, for the training, we only need the price of the 300th day?
I know this is the way LSTM models work, but wouldn't it be useful to take into account the price of the 299 other days for the model?

Time-series analysis has two main goals:

To identifying the nature of the phenomenon represented by the sequence of observations, and
To forecast (predicting future values in time for a variable).

NOTE: In this respect, models try to get pattern using historical values. Do not confuse here between forecasting (predicting future values) and prediction.

For time-series prediction(forecasting) problems, the models ( ARIMA, LSTM) try to extract trend, seasonality, and residual from the list of historical values e.g. price from 300 historical dates or time. Therefore, there is no need to look for dependent variables here.
We must be clear, the time series forecasting algorithm extrapolate historical trend to near and far future. They do not predict target variable based on dependent variables. This is the main reason you will see time-series models only work with single variable who's historical data-points become input and future data points become target.

Reference:

http://www.statsoft.com/Textbook/Time-Series-Analysis


What is the meaning of 2D vectors? I keep hearing people say something like:

  lets say you have a 1 dimensional vector of a person that just has his age.
  Then you add another dimension which is his height, so you have a 2 dimensional vector

But I also heard things like

  A 2d vector is a rank 2 tensor

Can someone clarify which is the correct one?

Is it

[1, 1]

or

[[1,1],
 [1,1]]

"Tensor" is a very general term, meaning N-dimensional array. A tensor can be 0-dimensional (a scalar), 1-dimensional (a vector), 2-dimensional (a matrix), and so on... Tensor is a very encompassing term, meaning all of the things listed above. Sometimes, people use "tensor" to refer to high dimensional objects, as explained here:

  While, technically, all of the above constructs are valid tensors, colloquially when we speak of tensors we are generally speaking of the generalization of the concept of a matrix to N ≥ 3 dimensions. We would, then, normally refer only to tensors of 3 dimensions or more as tensors, in order to avoid confusion [...]

But please keep in mind that this is just a practice, and N-dimensional array is technically a tensor.

Coming to your question, if a tensor is more like:

[1, 1]

or

[[1,1],
 [1,1]]

First, they look like Python list and list of lists, respectively. Python lists are not mathematical objects, strictly speaking. Provided you imported numpy, it would be more correct to describe them as:

numpy.array([ 1, 1 ])

and

numpy.array([[ 1, 1 ],
             [ 1, 1 ]])

Now these objects have the mathematical properties of vectors-matrices-tensors. I think they could both be thought as tensors of different kinds and ranks. The first is of rank 1, the other of rank 2.

Other useful resources on tensors are this question, and this explanation.


How do you choose the min and max for the min-max normalization on a histogram classifier? Please let me know what to do when there is a value in the testing set is bigger than the max value used to min-max normalize the training set building a histogram classifier.

Do I go back and change the bounds of the min-max normalization for the training set? Wouldn't that violate the notion that your training set should generalize to any testing set on its own and that you should retroactively change the what was done during the classifier building on the training set based on future testing sets that you are not supposed to know?

Do I change the bounds of the min-max normalization to the the min and max of the testing set? But, you are supposed to use the same transformation on the testing set as the training set, right?

Or, do I just let there be a bin on either side of the normalized histogram that such that everything that gets (not actually) normalized above (below) the interval [0,1] goes into the bin for all values (below) the interval?

Or, do I just exclude values that get transformed outside of the histogram's interval?

None of these seem right.  Please let me know if I am missing an option.

First build the min-max normalization on the entire data set. Then apply different workloads to your train and test split(s) separately. If you believe your data set will change over time you could consider estimating the extreme min and max if you are still interested in this min-max normalization.


PRML Bishop - Doubt in Entropy Formula (Section 1.6, equation 1.94) Please refer to the following excerpt from "Pattern Recognition and Machine Learning" Bishop.

  We can understand this alternative view of entropy by considering a
  set of N identical objects that are to be divided amongst a set of
  bins, such that there are $n_i$ objects in the $i^{th}$ bin. Consider
  the number of different ways of allocating the objects to the bins.
  There are N ways to choose the first object, $(N - 1)$ ways to choose
  the second object, and so on, leading to a total of N! ways to
  allocate all N objects to the bins, where N! (pronounced ‘factorial N
  ’) denotes the product N \times (N - 1) $\times \ldots \times 2 \times
> 1$.  However, we don’t wish to distinguish between rearrangements of
  objects within each bin. In the $i^{th}$ bin there are $n_{i}!$ ways
  of reordering the objects, and so the total number of ways of
  allocating the N objects to the bins is given by:
  $${\bf W} = \frac{N!}{\prod_{i}n_{i}!}$$ which is called multiplicity.

I am confused about the bold-faced part.

If bin 1 were to have n=1 object and bin 2 were to have 2 objects and so on, then the number of options for placing the objects in $i^{th}$ bin would be like - (with successive bin have only remaining objects to choose from):

$$^{N}C_{1} * ^{N-1}C_{2} * ^{N-3}C_{3}* ^{N-6}C_{4} \cdot \cdot $$

Were am I missing the point?

P.S.: If each bin has only one object, then I can understand N!.

You are not missing anything. You simply found another derivation of the same relationship.

We know that

$$^{n}C_{k} = \frac{n!}{k!(n-k)!} $$

If we expand the terms in your product,

$$^{N}C_{1} * ^{N-1}C_{2} * ^{N-3}C_{3}* ^{N-6}C_{4} \cdot \cdot $$

, we get

$$\frac{n!}{1!(n-1)!} \cdot \frac{(n-1)!}{2!(n-3)!} \cdot \frac{(n-3)!}{3!(n-6)!} \cdot \frac{(n-6)!}{4!(n-10)!} \dots $$

If we simplify the factorials,

$$\frac{n}{1!} \cdot \frac{(n-1)(n-2)}{2!} \cdot \frac{(n-3)(n-4)(n-5)}{3!} \cdot \frac{(n-6)(n-7)(n-8)(n-9)}{4!} \dots $$

And now it's obvious that this is

$$ \frac{n!}{\prod_i i!} $$


Why to adjust class weights instead of simply finding the best threshold? In a binary supervised classification where classes 1 and 0 have different number of samples in training, it’s very common to find tutorials about adjusting class weights, over and under sampling for imbalanced data sets. In a situation where there are enough samples for both classes (e.g. not an anomaly detection), why would one adjust class weights or balance the training data if in the end you’ll have to adjust a threshold anyway?

If there are enough samples of both classes, I don't think it makes a lot of sense. I've been in kaggle competitions with very imbalanced datasets, like:

Fraud detection
Home default risk
Quora insincere questions

And none of the top solutions used any kind of treatment of the imbalance, as there were enough samples for both classes.
I've also done myself, several fraud-related models with high imbalance, and using no imbalance solution proved better than all the other options.


How does SQL Server Analysis Services compare to R? This may be too broad of a question with heavy opinions, but I really am finding it hard to seek information about running various algorithms using SQL Server Analysis Service Data Mining projects versus using R. This is mainly because all the data science guys I work with don't have any idea about SSAS because no one seems to use it. :)

The Database Guy

Before I start, let me clarify. I am a database guy and not a data scientist. I work with people who are data scientist who mainly use R. I assist these guys with creating large data sets where they can analyze and crunch data.

My objective here is to leverage a tool that came with SQL Server that no one is really leveraging because no one seems to have a clue about how it works in comparison to other methods and tools such as R, SAS, SSPS and so forth in my camp.

SSAS

I have never really used SQL Server Analysis Services (SSAS) outside of creating OLAP cubes. Those who know SSAS, you can also perform data mining tasks on cubes or directly on the data in SQL Server.

SSAS Data Mining comes with a range of algorithm types:

Classification algorithms predict one or more discrete variables,
based on the other attributes in the dataset.
Regression algorithms predict one or more continuous variables, such
as profit or loss, based on other attributes in the dataset.
Segmentation algorithms divide data into groups, or clusters, of
items that have similar properties.
Association algorithms find correlations between different attributes
in a dataset. The most common application of this kind of algorithm
is for creating association rules, which can be used in a market
basket analysis.
Sequence analysis algorithms summarize frequent sequences or episodes
in data, such as a Web path flow.

Predicting Discrete Columns

With these different algorithm options, I can start making general predictions from the data such as finding out simply who is going to buy a bike based on a predictable column, Bike Buyers, against an input column, Age. The histogram shows that the age of a person helps distinguish whether that person will purchase a bicycle.

Predicting Continuous Columns

When the Microsoft Decision Trees algorithm builds a tree based on a continuous predictable column, each node contains a regression formula. A split occurs at a point of non-linearity in the regression formula. For example, consider the following diagram.

Comparison

With some of that said, it seems I can run a range of algorithms on the data and also have various functions available to me in SSAS to run against the data. It also seems I can develop my own algorithms in Visual Studio and deploy them to SSAS (if I'm not mistaken).

So, what am I missing here in regards to languages and tools from R? Is it just that they have more flexibility to deploy and edit complex algorithms versus SSAS etc?

In my opinion, it seems that SSAS makes more sense for someone who:

has significantly invested in Microsoft's technology stack and platform;
prefer point-and-click interface (GUI) to command line;
focus on data warehousing (OLAP cubes, etc.);
has limited needs in terms of statistical methods and algorithms variety;
has limited needs in cross-language integration;
doesn't care much about openness, cross-platform integration and vendor lock-in.

You can find useful this blog post by Sami Badawi. However, note that the post is not recent, so some information might be outdated. Plus, the post contains an initial review, which might be not very accurate or comprehensive. If you're thinking about data science, while considering staying within Microsoft ecosystem, I suggest you to take a look at Microsoft's own machine learning platform Azure ML. This blog post presents a brief comparison of (early) Azure ML and SSAS.


How to set class weights for imbalanced classes in Keras? I know that there is a possibility in Keras with the class_weights parameter dictionary at fitting, but I couldn't find any example. Would somebody so kind to provide one?

By the way, in this case the appropriate praxis is simply to weight up the minority class proportionally to its underrepresentation?

You could simply implement the class_weight from sklearn:

Let's import the module first

from sklearn.utils import class_weight

In order to calculate the class weight do the following

class_weights = class_weight.compute_class_weight('balanced',
                                                 np.unique(y_train),
                                                 y_train)

Thirdly and lastly add it to the model fitting

model.fit(X_train, y_train, class_weight=class_weights)

Attention: I edited this post and changed the variable name from class_weight to class_weights in order to not to overwrite the imported module. Adjust accordingly when copying code from the comments.