How to apply AdaBoost to more "complex" (non-binary) classifications/data fitting? In this article, Chris McKinlay says he used AdaBoost to choose the proper "importances" of questions he answered on okcupid.

If you haven't read and don't want to read the article, or are unfamiliar with okcupid and the question system, here's the data and problem he had:
The goal is to "match" as highly as possible with as many users as possible, each of whom may have answered an arbitrary number of questions. These questions may have between 2 and 4 answers each, and for the sake of simplicity, let's pretend that the formula for a match% $\ M $ between you and another user is given by

$\ M = Q_a/Q_c $

Where $\ Q_c $ is the number of questions you and the other user have in common, and

$\ Q_a $ is the number of questions you both answered with the same value.

The real formula is slightly more complex, but the approach would be the same regarding "picking" a correct answer (he actually used boosting to find the ideal "importance" to place on a given question, rather than the right answer).
In any case, the point is you want to pick a certain value for each question, such that you maximize your match% with as many users as possible - something you might quantify by the sum of $\ M $ over all users.

Now I've watched the MIT course on AI up to and including the lecture on boosting, but I don't understand how you would apply it to a problem like this. Honestly I don't even know where to begin with choosing rules for the weak learners. I don't have any "rules" about what values to choose for each question (if the user is under 5'5, choose A, etc) - I'm just trying to fit the data I have.

Is this not the way boosting is supposed to be used? Is there likely some other optimization left out of how he figured this out?

AdaBoost is a supervised learning method; it starts with a table of 'correct' answers and generates a predictive model for a target, which is known. It is then possible to inspect this model to figure out how it works, what it judged was more important. With that in mind, here is my guess for what he did:

First, he created a training dataset. The dataset was created, according to the article, like this:

  Now he’d do the same for love. First he’d need data. While his
  dissertation work continued to run on the side, he set up 12 fake
  OkCupid accounts and wrote a Python script to manage them. The script
  would search his target demographic (heterosexual and bisexual women
  between the ages of 25 and 45), visit their pages, and scrape their
  profiles for every scrap of available information: ethnicity, height,
  smoker or nonsmoker, astrological sign—“all that crap,” he says.

  To find the survey answers, he had to do a bit of extra sleuthing.
  OkCupid lets users see the responses of others, but only to questions
  they’ve answered themselves. McKinlay set up his bots to simply answer
  each question randomly—he wasn’t using the dummy profiles to attract
  any of the women, so the answers didn’t matter—then scooped the
  women’s answers into a database. "

So he used python scripts to collect lots of information! At the end of this, he had a table of data, where each row had three pieces of information:
- A bot's answer to all the questions, and their weights
- A woman's answer to all the questions
- Their match percentage.

On this, he could use AdaBoost to create a predictive model, predicting the match percentage from the available information. The weak learners were probably decision stumps, greedily choosing one variable at a time to split on, that's the standard that people refer to when talking about AdaBoost.

Once the predictive model, was in place, it could be used for optimization - determining which weights to put on the questions when holding all other variables constant, maximizing the average match percentage to women in his target audience.

Of course, this is just a guess. The article doesn't have much detail. But it's a potential way to use AdaBoost for that purpose.


Is this a data issue, or a model issue? A Keras binary classification model I've been trying to create a binary classification model that predicts wether there will be a train delay based on the train and time. Here is a link to the data

The issue I'm having is that my accuracy goes to 94.07 in the first 5 epochs. Meanwhile, my class prediction will always be 0 and never 1.

From what I understand, this is "Accuracy Paradox". A symptom of Class Imbalance. To combat this, I implemented Kfold.

kfold = StratifiedKFold(n_splits=10,shuffle=True)
cvs_scores = []

for train,test in kfold.split(X,Y):

  history = model.fit(X[train],Y[train],epochs=50,batch_size=15, shuffle = False, verbose = 1)
  scores = model.evaluate(X[test],Y[test],verbose=0)

  print("%s: %.2f%%" % (model.metrics_names[1],scores[1]*100))
  cvs_scores.append(scores[1] * 100)

print("%.2f%% (+/- %.2f%%)" % (numpy.mean(cvs_scores),numpy.std(cvs_scores)))

No luck. Still had the same issue as before.

Here is how I import my data:

raw_data = pd.read_csv('MTA_DELAY_DATA_DUMP - Sheet1.csv')

X = raw_data.iloc[1:-2,0:2].dropna().values
Y = raw_data.iloc[1:-2,2:3].dropna().astype(int).values

My Model:

model = Sequential()
model.add(Dense(32, kernel_initializer='uniform', activation='relu',input_dim =2))
model.add(Dense(16, kernel_initializer='uniform', activation='relu'))
model.add(Dense(8,  kernel_initializer='uniform', activation='relu'))
model.add(Dense(1,  kernel_initializer='uniform', activation='sigmoid'))

model.compile(loss='binary_crossentropy', optimizer=Adam(lr=0.001),metrics=['accuracy'])
history = model.fit(X,Y,epochs=150,batch_size=15, shuffle = False, verbose = 1)

I tried assigning class weights to balance the data out. Even manually deleting 0's in the data, but nothing seems to result in accurate predictions. Am I doing something wrong in the model, or is this simply data that cannot be utilized by machine learning?

Possible problems:

You have classes 0 and 0.1. You want to make them 0 and 1 since the formula for the loss function
$$-{(y\log(p) + (1 - y)\log(1 - p))}$$
expects $y$ to be 0 or 1.
shuffle = False why would you want to turn shuffling off?
The classes are unbalanced. You have ~100 examples of class 1  and ~20000 total examples. So if you just predict class of everything to be 0 you will be right (20000-100)/20000 = 99.5% times. You can try these tricks and see if they help.


Gradients for bias terms in backpropagation I was trying to implement neural network from scratch to understand the maths behind it. My problem is completely related to backpropagation  when we take derivative with respect to bias) and I derived all the equations used in backpropagation. Now every equation is matching with the code for neural network except for that the derivative with respect to biases.

z1=x.dot(theta1)+b1

h1=1/(1+np.exp(-z1))
z2=h1.dot(theta2)+b2
h2=1/(1+np.exp(-z2))

dh2=h2-y
#back prop

dz2=dh2*(1-dh2)
H1=np.transpose(h1)
dw2=np.dot(H1,dz2)
db2=np.sum(dz2,axis=0,keepdims=True)

I looked up online for the code, and i want to know
why do we add up the matrix and then the scalar db2=np.sum(dz2,axis=0,keepdims=True) is subtracted from the original bias, why not the matrix as a whole is subtracted. Can anyone help me to give some intuion behind it. If i take partial derivative of loss with respect to bias it will give me upper gradient only which is dz2 because  z2=h1.dot(theta2)+b2 h1 and theta will be 0 and b2 will be 1. So the upper term will be left.

b2+=-alpha*db2

I would like to explain the meaning of db2=np.sum(dz2,axis=0,keepdims=True) as it also confused me once and it didn't get answered.

The derivative of L (loss) w.r.t. b is the upstream derivative multiplied with the local derivate:
$$
\frac{ \partial L}{\partial \mathbf{b}} = \frac{ \partial L}{\partial Z} \frac{ \partial Z}{\partial \mathbf{b}}
$$

If we have multiple samples Z and L are both matrices. b is still a vector.

The local derivative is simply a vector of ones:
$$
\frac{ \partial Z}{\partial \mathbf{b}} = \frac{\partial}{\partial \mathbf{b}} W \times X + \mathbf{b} = \mathbf{1}
$$

That means our complete derivative is a matrix multiplication, that looks as follows (e.g. 2 samples with 3 outputs):
$$
\frac{\partial L}{\partial Z} \times \mathbf{1} =
\begin{bmatrix}
. &.  &. \\
. &.  &.
\end{bmatrix}
\begin{bmatrix}
1\\
1\\
1\\
\end{bmatrix}
$$

Note that this is the sum of the rows.

And that's where db2=np.sum(dz2, axis=0, keepdims=True) comes from. It is simply an abbreviation for the matrix multiplication of the local and the upstream derivatives.


Find the optimal n_estimator by looping the model accuracy indicator in random forest algorithm - python i'm trying to find the best n_estimator value on a Random Forest ML model by running this loop:

for i in r:
   RF_model_i = RandomForestClassifier(criterion="gini",   n_estimators=i, oob_score=True)
   RF_model_i.id = [i]  # dynamically add fields to objects
   RF_model_i.fit(X_train, y_train)
   y_predict_i =   RF_model_i.predict(X_test)
   accuracy_i = [accuracy_score(y_test, y_predict_i), i]
   results.append(accuracy_i) # put the result on a list within the for-loop

question #1 ** What i like to understand is if this could be a good way to understand how to decide the n_estimator parameter and
possibly why (i'm not so sure it is)
2.**question #2 If what i'm doing have some sense, could be a good idea extend this loop to all of other main parameters?

What i obtain is this:
a level of accuracy for a number of estimator associated with which i runned the random forest algorithm

Thank you

The number of trees in a random forest doesn't really need to be tuned, at least not in the same way as other hyperparameters.  Adding more trees just stabilizes the results (you're averaging more samples from a distribution of trees); you want enough trees to get stable results, and adding more won't hurt except for computational resources.

More directly (question 1), you could instead train the RF with, say, 1000 n_estimators, then just grab the predictions from each tree and average only the first 100 of them, the first 200, etc.  Since the trees are built independently, this is essentially the same as training repeatedly on 100, then 200, etc. trees from scratch (but of course faster).

More generally (question 2), yes, this is one-dimensional grid search.  Hyperparameters may be interdependent though, so doing independent 1D searches will probably be suboptimal.  Look into full grid search, random search, and maybe more advanced hyperparameter optimization methods.


Keras neural network entirely different results on two different hardwares (one is AWS) I have a Keras model. which is defined as follows:

nn_model = Sequential()
nn_model.add(Dense(300, activation="relu",input_shape=(4,))) # because we have 4 features
nn_model.add(Dropout(0.3))
nn_model.add(Dense(150, activation="relu"))
nn_model.add(Dense(50, activation="relu"))
nn_model.add(Dense(30, activation="relu"))
nn_model.add(Dense(1))

optimizer = optimizers.RMSprop(0.001)
nn_model.compile(loss='mse',optimizer=optimizer,metrics=['mae'])

This model is used as a regressor for predicting a certain pattern.
For the SAME training and testing data AND SAME code, the model is making two entirely different predictions on two different hardwares. In one case, the weights become entirely corrupt and makes flat predictions (training data has no flat curve).

Versions of libraries being used are same on both the machines.

Here are the specs of two machines:

Correct predictions machine:
Lenovo ThinkPad E570
model type: Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz

Wrong prediction (AWS VM):
Machine type: c5.large

Can anyone point out any reason of which this might be happening?

Consistency in models / predictions in Keras depend on the following:

Software Stack: The most uncommon problem, sometimes there are differences in different version of the numerical software which may change results.

Difference in Backends: Keras supports the TensorFlow backend, the Theano backend, and the CNTK backend. All of which will yield slightly different models. In the python console you can check the version with a message similar to:

Using TensorFlow backend.

You can switch backends using the following guide: Backend utilities | Keras

Setting seeds: In order to have consistency among models you need to set random seeds accordingly. In the case of Keras and Tensorflow backend, you need to set two random seeds:

In numpy

from numpy.random import seed

seed(123)

In tensorflow

from tensorflow import set_random_seed

set_random_seed(123)

There are other more intricate details for setting seeds in certain parts of the graph in Tensorflow explained better on this Stack Overflow answer and the set_random_seed() API documantation on Tensorflow's website.

Additional randomness added by GPUs: According to this MachineLearningMastery article there is even more randomness being added to a model when GPUs are used.

Intuitively in your problem, the source of randomness comes from not having set the appropriate seeds in numpy and Tensorflow, so start from there and start exploring the other options and documentations. Also remember that if you're using a different backend (Theano or CNTK), setting seeds will require a different function to be called.
An additional recommendation is to look for bugs in the script, version of downloaded binaries, warnings and errors during installation in case the result are too different. Something could be broken if one instance doesn't perform as expected and a fresh install might be necessary.


Training Validation Testing set split for facial expression dataset I am using Convolutional Neural Networks (CNN) and I just want to ask if the way I split my training/validation/testing set is correct.

I have a total of 55 subjects. I plan to split them into 80–10–10 for training (45 subjects), validation (unseen 5 subjects), testing (unseen 6 subjects).

Should the validation set consists of unseen subjects as well? Or can I shuffle the whole training set and use a part of it (10-20%) as validation set?

I have read that using N-Fold cross-validation, the whole training set (instances) are shuffled then split into N-folds and the model is trained and averaged N times. However, in the case for Neural Networks or CNN, we don't use cross-validation since it is very computationally expensive.

I'm just wondering which is correct since using a validation set of unseen subjects, my model starts to overfit after 3-5 epochs and doesn't learn at all. On the other hand, if I use 10-20% of the training set as my validation set, my model learns with reasonable accuracy (45-50%) using a 3-layer CNN but when tested with the unseen testing set, my top-1 accuracy is around 15-16% only.

Thank you very much.

NB This advice assumes your goal is to recognise expression in pictures of any person, and not just people from your training data.

  Should the validation set consists of unseen subjects as well?

Yes. This will give you the most accurate measure of performance in the task you want to use the network in, in order to choose the best generalisation and take it forward to testing.

You would only use a simple random split if the end goal of your trained network is to recognise expressions from images of the people in the training set.

  Or can I shuffle the whole training set and use a part of it (10-20%) as validation set?

No. If you take random samples where the same face appears in training and cv, you will get an over-estimate of generalisation. I have seen this effect first hand in the Kaggle State Farm Distracted Driver contest, and you should see it discussed in the forums there. It might be helpful for ideas to improve performance too.

  if I use 10-20% of the training set as my validation set, my model learns with reasonable accuracy (45-50%) using a 3-layer CNN

This is a data leak between train and cross validation - the network has learnt to correctly classify expressions in images of people it has already seen, and that is what you are measuring by taking a split in this way. It is not surprising that the test result does not match the promising values from cross validation.

  I'm just wondering which is correct since using a validation set of unseen subjects, my model starts to overfit after 3-5 epochs and doesn't learn at all

You are over-fitting regardless of how you split train,CV. When you split the CV incorrectly, then in addition to over-fitting, the data leak gives you bad guidance.

It looks like that you have very little diversity in the training data, whilst training an image classifier from scratch requires a lot of data. Consider:

Collecting more labelled data, perhaps some other dataset you can download.
Adding a lot of regularisation to your model (e.g. multiple dropout layers)
Using a pre-trained image classifier network (e.g. VGG-19 or Inception) as a starting point and fine-tuning it for your classification task.
Use full k-fold cross-validation regardless of the computational cost, to mitigate problems of using small training set. This won't help solve your training problems directly, but will give you a better shot at tuning your network hyper-parameters once you solve that issue.


How should the hyper parameters be defined for the other algorithms defined for sklearn_crfsuite.CRF In the example given for sklearn_crfsuite, the parameter space that needs to be passed to a cross_validating class like RandomizedSearchCV is defined as below.

params_space = {
    'c1': scipy.stats.expon(scale=0.5),
    'c2': scipy.stats.expon(scale=0.05),
}

In the example the algorithm that is shown is "lbfgs". Other supported algorithms in sklearn_crfsuite are "l2sgd", "ap", "pa", "arow". How should the variable params_space be defined in those cases?

C1 and C2 are coefficients for L1 and L2 regularization, respectively. You can use the same definition of params_space regardless of the optimization algorithm.


Can I classify a healthy population into clusters found in an injured population? Just a bit of context: We are applying Machine Learning algorithms in the field of Human Biomechanics. In a previous project, my colleagues identified three different subgroups in an injured population performing a movement utilising hierarchical clustering. In order to be able to classify new subjects, we developed a classification algorithm (using the same dataset) that was able to allocate each case to the predefined cluster.

We currently are working in another study that involves new comparing injured and healthy subjects performing the same task in which the clusters were found.

My questions are, is there any way to know to what extent clusters found in injured athletes apply to healthy athletes? If I used the same algorithm and features, what could be the effects/consequences? can I justify it? What are the limitations?

Any answer or directions to literature will be greatly appreciated as I am struggling to find any similar situation in the literature/internet.

Thanks everyone.

Cheers!

The dataset (population) that you used to define the clusters was a different one from the one you are looking at now (healthy population). That means you can’t use your clusters to categorize healthy people just like that, but your research question is not what cluster each person belongs to.

If your question is to see whether people are consistent in their movement strategy then you can use the information from the clusters you have, or any other clusters you just made up. The cluster assignment is a proxy (a simplified representation) of each measurement, and you’re trying to see if they are consistent from the cluster numbers of each measurement.

This seems to me to lead to results that are difficult to interpret. What if healthy people are consistently very close to a border between two clusters? Consistent, but jittering between the clusters.

I would say you need to drop the idea of using a cluster as a proxy. Let’s say your measurement is how fast people can run a mile, time after time. To know if they’re consistent, you would just look at the standard deviation for each person.

I guess your problem is that you have many measurements in each example, and you’re trying to simplify by using a cluster number rather than the actual measurement. That’s dimensionality reduction, to a single dimension, which is extreme, and I wouldn’t use clustering for it. Is it really necessary to reduce the dimensions, or would it be possible to define a consistency metric on the original data? If it’s not and you need to reduce the dimensions, I would first turn to PCA and if that’s not enough, maybe an embedding from an auto-encoder.


Neural network reaching local optima I was recently trying to train a convolutional neural network to classify people as Hispanic or white (for learning purposes). I couldn't find a good dataset of just those two races, so I had to manually scrape images from the web. I ended up with 48 total images, 24 of them Hispanic and 24 white. When I trained my network, I tried many different architectures and hyperparameters, but the accuracy stayed at exactly 50%. I learned this was because the network was outputting 1 every time, classifying every image as Hispanic because it found a local optima for the cost function. I believe this is because of my 50/50 data split, although I may be wrong. If if is because of the split, how does this not happen in the real world? For example, if I had 10 million images, half of them white and the other half Hispanic, how does the network avoid falling prey to the same trap?

First of all your dataset is very small to train a deep learning model you would need at least 10k images to train the model and produce good results. Secondly, if you still think that your cost function gets stuck in local minima then you might consider using Nesterov momentum.


Can overfitting occur in Advanced Optimization algorithms? while taking an online course on machine learning by Andrew Ng on coursera, I came across a topic called overfitting. I know it can occur when gradient descent is used in linear or logistic regression but can it occur when Advanced Optimization algorithms such as "Conjugate gradient", "BFGS", and "L-BFGS" are used ?

There is no technique that will eliminate the risk of overfitting entirely. The methods you've listed are all just different ways of fitting a linear model. A linear model will have a global minimum, and that minimum shouldn't change regardless of the flavor of gradient descent that you're using (unless you're using regularization), so all of the methods you've listed would overfit (or underfit) equally.

Moving from linear models to more complex models, like deep learning, you're even more at risk of seeing overfitting. I've had plenty of convoluted neural networks that badly overfit, even though convolution is supposed to reduce the chance of overfitting substantially by sharing weights. In summary, there is no silver bullet for overfitting, regardless of model family or optimization technique.


When I include validation_data=(x_val, y_val) in model.fit_generator, should I create another test dataset for accuracy measures? While modelling in keras, often I see the usage of validation_data=(x_val, y_val) in model.fit_generator where (x_val, y_val) generally forms 10% of the dataset. While training, is it that the model takes hint from the validation loss calculated on (x_val, y_val) and I need to create another test_data for measuring accuracies in the end? Or, I can use the same (x_val, y_val) in input of model.fit_generator and measuring the accuracy in the end.

The confusion arises from the fact that we are often advised to create training, validation, test datasets while modelling. If the validation dataset has been used to judge some parameters like when to stop(early stopping), etc; wouldn't it be unfair to use the validation_data to calculate the various measures of accuracy?

You are correct in saying that it would be unfair - and if avoidable, you shouldn't do it.

In order to truly be able to claim (in a statistical sense) that a model achieves e.g. 90% accuracy, the test must be performed on unseen data. That is where your test data should be used.

Training a neural network requires the validation data (as you mentioned, within the fit_generator method of a Keras model), in order to compute errors and steer the weights in the right direction. The final accuracy you report needs to be on data to which the the training pipeline has never been exposed.

Creating a training/validation/test split is advised where possible; however, it can sometimes be a challenge, due to things like lack of data and imbalanced datasets. You can try things such as cross-validation - here is an example using Keras. Her is another example, in a similar question.


Should we use one hot encoder class in data having 2 as maximum numeric representation of categorical feature in each column? I am testing the Play Golf data set using Decision tree classifier:

I am splitting the data into Outlook, Temp, Humidity and windy as features, and Play Golf as target feature.

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

# Importing the dataset
dataset = pd.read_excel('data.xlsx')
X = dataset.iloc[:, 0:4]
y = dataset.iloc[:, 4]

To test the DecisionTreeClassifier I need to transform categorical data into numeric:

from sklearn.preprocessing import LabelEncoder
lb = LabelEncoder()

X['Outlook'] = lb.fit_transform(X['Outlook'])
X['Temp'] = lb.fit_transform(X['Temp'])
X['Humidity'] = lb.fit_transform(X['Humidity'])
X['Windy'] = lb.fit_transform(X['Windy'])
y = lb.fit_transform(y)

The result is:

ix  Otk T   H   W
0   1   1   0   0
1   1   1   0   1
2   0   1   0   0
3   2   2   0   0
4   2   0   1   0
5   2   0   1   1
6   0   0   1   1
7   1   2   0   0
8   1   0   1   0
9   2   2   1   0
10  1   2   1   1
11  0   2   0   1
12  0   1   1   0
13  2   2   0   1

Should I use OneHotEncoder() class afterwards? Or 2 isn't that big to do so?

The purpose of one-hot encoding is to binarize categorical labels so that your model doesn't learn a spurious ordinal relationship.

For example if you had categories Red, Yellow, Blue, it would be bad to encode Red as 0, Yellow as 1, and Blue as 2 because your model might accidentally "learn" that Red < Yellow < Blue.  In this case you should use one-hot encoding.

But for some categorical features, an ordinal encoding makes sense.  For example, your dataset has the feature "Temperature", which can be Cool, Mild, or Hot.  In this case, the relationship Cool < Mild < Hot is semantically meaningful, so you can probably get away with ordinal encoding if you want.

However, you still have to ensure that the ordinal relationship is preserved in your encoding, and scikit-learn's LabelEncoder does not take care of this for you.  To illustrate, look at how "Temperature" is being encoded:

Hot => 1
Mild => 2
Cool => 0

That's no good!  You want Hot to have the highest value.  Instead you should use scikit's OrdinalEncoder and explicitly tell it how to order the categories:

from sklearn.preprocessing import LabelEncoder, OrdinalEncoder

outlook_encoder = OrdinalEncoder(categories=['Rainy', 'Overcast', 'Sunny'])
X['Outlook'] = outlook_encoder.fit_transform(X['Outlook'])

temp_encoder = OrdinalEncoder(categories=['Cool', 'Mild', 'Hot'])
X['Temp'] = temp_encoder.fit_transform(X['Temp'])

# humidity and wind are binary features, so we don't need the ordinal encoder

humidity_encoder = LabelEncoder()
X['Humidity'] = humidity_encoder.fit_transform(X['Humidity'])

windy_encoder = LabelEncoder()
X['Windy'] = windy_encoder.fit_transform(X['Windy'])

Although there do exist ordinal relationships in most of your categorical features, you should still feel free to use one-hot encoding.  It's always a safe bet when you're dealing with categorical features.

Since your dataset is small, you should experiment with both and see which gives the best results!


How to create a complex Gaussian random noise with a specific covariance matrix I am trying to generate a complex Gaussian white noise, with zero mean and the covariance matrix of them is going to be a specific matrix which is assumed to be given.

Assume i to be a point on the grid of x axis, where there are N points on the axis. The problem is to generate a complex valued random noise at each point (let's call the random value at the point i as $y_i$), which obeys Gaussian distribution and has a covariance matrix of,

$E((y_i)(y_j^*)) = c_{ij}$

where $c_{ij}$ is a given diagonal NxN matrix. * is complex conjugate, E represents the expectation value.

I am trying to generate this on Python.

Might be a bit late, but if you're still looking for an answer, here it is. You can use np.random.multivariate_normal. You need to provide the said covariance matrix as an argument to the function. You can find the documentation here.


How to use multiple cross-section observations per subject for churn prediction? Recently I have started to teach myself about machine learning and I have ran into a dataset, which got me a bit confused.
Dataset: The subjects of the dataset are university students (student ID == "Key" feature), and each observation is a summary of their semester (grade averages, ECTS taken and completed, etc.) plus their general programme-related data (enrollment and scholarship status, date of enrollment, programme code, etc.). The data is in hungarian, but in the context of the issue, it is not important to understand the meaning of the feature names and values. Below is an example of an observation:

My goal: I want to build a model, which predicts student churn.
Problem: The dataset contains a single or multiple observations per student, based on the number of university semesters, and the observation periods are not consistent between the students, since it is based on the individual date of enrollment.

In the example picture above, you can see, that student no.1 has 7 observations (=7 semesters completed) and started his programme on 2009.09.10 (Képzés jogviszony kezdete == Date of programme enrollment), while student no.2 has 3 observations and started his programme on 2008.09.12.
I am wondering, should I use only one observation (e.g.: the last completed university semester) per student ,or does it make sense to use all observations per student?
Thank you for the feedbacks in advance!
(Also, I am new on the forum, so if you have any constructive criticism regarding the content and the format of my question, please, share with me.)

If you know SQL, then my method might work. I'll break it out into subqueries to make it a little more understandable. I don't know your columns so this might not translate perfectly.
The point is to have 1 query identify all active students at a particular point in time. The point of time you choose is arbitrary, but it must have enough time forwards enough to the end of your data so that you can establish a forward-looking window to satisfy churn or not. I am calling this the snapshot_date. Don't have students who churned before the shapshot_date in this resultset at all.
Then, you join this to a subquery which uses that shapshot_date and looks forward for the churn condition. This returns only students who churned, but churned within the window that you decide. If they churned in the future beyond your window, treat them as if they have not churned.
The very first query that returns a list of active students might still return multiple rows for each student. You handle this by deciding how to aggregate those rows into a single row for each customer. You might end up creating a lot of good variables for your model this way. I added a few as an example.
Then, your SQL would look similar to this:
SELECT active_students.*,
case when student_churned is not null then 1 else 0 end as target_student_churn
FROM
    (
        SELECT Key,
        DateDiff(snapshot_date, date_of_enrollment, "Days") as time_since_enrollment,
        min(date_of_enrollment) as date_of_enrollment,
        max(felev) as max_felev,
        COUNT(DISTINCT course) as distinct_courses_taken,
        ...
        FROM table
        WHERE date_of_enrollment < snapshot_date
        AND {student has not churned as of snapshot date}
        GROUP BY Key
    ) as active_students
LEFT OUTER JOIN
    (
    SELECT Key
    FROM table
    WHERE {student churned within hard coded time period AFTER snapshot date}
    GROUP BY Key
    ) as student_churned
ON active_students.Key = student_churned.Key;

You may wonder then how this accounts for seasonality over time if you chose your snapshot date randomly. Well, just change the snapshot_date and repeat while you continue to stack the results on top of one another. Having the snapshot date as a column will help you extract the "season" from each resultset.
Now the students might be repeated again in the final dataset. So, when you build the model you want to make sure that the students' rows fall into the same validation or holdout dataset. So use GroupKFold or something like that. I hope this helps.


Using doccano for Aspect Based Sentiment Analysis annotation Currently looking for a good tool to annotate sentences regarding aspects and their respective sentiment polarities.

I'm using SemEval Task 4 as a reference. The following is an example in the training dataset:

<sentence id="2005">
    <text>it is of high quality, has a killer GUI, is extremely stable, is highly expandable, is bundled with lots of very good applications, is easy to use, and is absolutely gorgeous.</text>
    <aspectTerms>
        <aspectTerm term="quality" polarity="positive" from="14" to="21"/>
        <aspectTerm term="GUI" polarity="positive" from="36" to="39"/>
        <aspectTerm term="applications" polarity="positive" from="118" to="130"/>
        <aspectTerm term="use" polarity="positive" from="143" to="146"/>
    </aspectTerms>
</sentence>

Can I easily use doccano for such a task? Or would I be better off using some other tool, such as brat?

You can use sequence labeling feature to annotate the text:


Is ensemble learning using different classifier combination another name for Boosting? For implementation I am following the Matlab code for AdaBoost. Based on my understanding, AdaBoost uses weak classifiers known as base classifiers and creates several instances of it. For example, a weak classifier is a decision tree. So, AdaBoost can create maximum N decision trees (where N = number of samples) and combine the prediction results. This is a Homogeneous boosting method. But I have seen some examples such as this one in Matlab and ensemble-toolbox which have confused me. Can somebody please explain the following concepts with respect to the implementations and what is going on in the code?

1) Does the Matlab code for AdaBoost combine different classifiers? The combination method is unclear to me - -whether they do sum or majority voting or something else.

If they are combining several classifiers, then technically it is a Heterogenous ensemble method and the term for it is stacking and not boosting. Please correct me where I am wrong.
In Boosting methods,  the based classifiers are the same. But the given in Matlab code for AdaBoost combines different classifiers, I am not sure.

2) Is ensemble learning or the example in the ensemble toolbox the same as the Adaptive Boosting Matlab code (second link)? Is ensemble learning the same as Adaptive boost?

Ensemble learning combines predictions from multiple learners. Boosting methods are one way to form an ensemble. Stacking is another. The important difference between boosting and stacking (and other ensemble methods) is that boosting applies a number of weak learners sequentially and then produces a final result via a weighted majority vote.

The learners in stacking can also be combined as a weighted average or vote (or by another "meta" model) but they can be more or less independent.

In boosting, each weak learner (usually the same, yes) modifies the data for the next learner in the sequence. E.g. in AdaBoost the weight of each learner and also each sample in the data depends on the misclassification rate of the previous weak learner to the effect that the next learner focuses more on previously misclassified samples.

Boosting methods typically vary by how these weight updates are performed.


Loading Street View House Numbers Dataset in Python I am trying to work with the Street View House Numbers Dataset in Python. To load the .mat file digitStruct.mat in python I am using H5py

import h5py
labels_file = './sv/train/digitStruct.mat'
f = h5py.File(labels_file)
struct= f.values()
names = struct[1].values()
print(names[1][1].value)

I get [<HDF5 object reference>] but I need to know the actual string

If you already have a <HDF5 object reference>, and you want to know the real data, here is one approach:

import h5py
labels_file = './sv/train/digitStruct.mat'
f = h5py.File(labels_file)
ref = f.get('/digitStruct/name').value[0][0]  # <HDF5 object reference>
print(f[ref].value)

In my environment, f[ref].value shows:

In [84]: f[ref].value
Out[84]:
array([[ 49],
       [ 46],
       [112],
       [110],
       [103]], dtype=uint16)

For more information on object references, you can refer to http://docs.h5py.org/en/latest/refs.html


Ways to deal with longitude/latitude feature I am working on a fictional dataset with 25 features. Two of the features are latitude and longitude of a place and others are pH values, elevation, windSpeed etc with varying ranges. I can perform normalization on the other features but how do I approach latitude/longitude features?

Edit: This is a problem to predict agriculture yield. I would think lat/long is very important since locations can be vital in prediction and hence the dilemma.

Lat long coordinates have a problem that they are 2 features that represent a three dimensional space. This means that the long coordinate goes all around, which means the two most extreme values are actually very close together. I've dealt with this problem a few times and what I do in this case is map them to x, y and z coordinates. This means close points in these 3 dimensions are also close in reality. Depending on the use case you can disregard the changes in height and map them to a perfect sphere. These features can then be standardized properly.

To clarify (summarised from the comments):

x = cos(lat) * cos(lon)
y = cos(lat) * sin(lon),
z = sin(lat)


What is a manifold for Unsupervised Learning? I've been watching Dr. G. Hinton lectures on Neural Networks in Machine Learning, and in one of the lectures he explains what the goals of Unsupervised Learning are.

I am having trouble understanding the part where high-dimensional inputs such as images live on or near a low-dimensional manifold (or several such manifolds). What is a manifold exactly, and why is this the case?

Thanks!

Depends who you ask but generally mannifold is just some  structure in a (high) dimensional space having finite dimension : a line, a curve, a plane, a rock, a sphere, a ball, a cylinder, a torus, a "blob"... something like this :

If you ask a mathematician
they would say that it is a general term describing "a curve" (dimension 1) or "surface" (dimension 2), or a 3D object (dimension 3)... for any possible finite dimension $n$. A one dimensional manifold is simply a curve (line, circle...). A two dimensional manifold is simply a surface (plane, sphere, torus, cylinder...). A three dimensional manifold is a "full object" (ball, full cube, the 3D space around us...).

TO ANSWER YOUR QUESTION what he is referring is that information in this high-dimensional space can be compressed and perserved in some smaller space. (remember images when quantified live in some space that contains information about them)


Match a two items from two different receipts I have two different invoices or receipts. One is a Purchase order one is something like a receipt(acknowledgement).

Suppose I have ordered(PO) Wine:

White Wine
Red Wine
Rose Wine

And I receive the acknowledgement as:

Wine Red Jacobs Creek
White Wine
Winter's Hill Estate Dry Rose

I want to match the strings (items) in the Purchase Order and the Invoice.

Can you suggest me ways to do it.

I have tried vectorization using Count Vectorization Alg
Then have used distance measures to calculate similarity using:
'dice', 'rogerstanimoto', 'yule', 'hamming', 'jaccard', 'braycurtis', 'canberra', 'cityblock', 'correlation', 'cosine', 'euclidean', and 'minkowski'

The problem is the position of Words.

Red Wine is will not be similar to Wine Red. But that should not be the case.

I have tried Word2Vec Algorithm too but as this is not language technically just Nouns. It did not work.

You can try some approximate string matching which gives a confidence score. For example, you can try out with Levenshtein distance, but adjusted with the length of the strings using a probabilistic model; or, you can try out with Jaccard similarity on 3-grams and special treatment on word boundaries, and then calibrated into probabilities. Now you have an n by n matrix of probabilities.

After this you need to perform a matching. The final likelihood is the product of each single probability. To maximize it, you can try to maximize the sum of logs of individual probabilities. Having taken logarithms of each probability, this now becomes an (additive) assignment problem which has implementations in R or Python.


LightGBM - Why Exclusive Feature Bundling (EFB)? I'm currently studying GBDT and started reading LightGBM's research paper.

In section 4. they explain the Exclusive Feature Bundling algorithm, which aims at reducing the number of features by regrouping mutually exclusive features into bundles, treating them as a single feature. The researchers emphasize the fact that one must be able to retrieve the original values of the features from the bundle.

Question: If we have a categorical feature that has been one-hot encoded, won't this algorithm simply reverse the one-hot encoding to a numeric encoding, thereby cancelling all the benefits of our previous encoding? (suppression of hierarchy between categories etc.)

I've read that paper so many times before in so many ways. What I can say on the matter is that the paper does not describe explicitly what the framework particularly does. It just gives an hint of their intuitive idea of bundling of the features in an efficient way. But specificly, it does not say that it does a 'reversion of one-hot-encoding' in particular to your question.

I tried giving categorical inputs directly and as one-hot-encoded to compare the time that it takes to compute. There was a significant difference: giving directly was all better in multiple datasets compared to giving as one-hot-encoded.

Possibilities:

1)It is possible that LightGBM Framework can find out that we give the features as one-hot-encoded from the sparsity, it is possible that the algorithm does not treat one-hot-encoded with EFB.

2)It is also possible that LightGBM uses EFB on one-hot-encoded samples but it may be harmful, or not good as EFB on direct categorical inputs. (I go for this one)

But still, I do not think that EFB will reverse one-hot-encoding since EFB is explained as a unique way of treating the categorical features. But it possibly 'bundles the unbundled  features' when treating one-hot-encoded inputs.

I used the word 'probably' so much times out of implicitness of the paper. What I can advice to you is that to send an e-mail to one of the authors of the paper, I do not think that they would refuse to explain it. Or if you are brave, go for the GitHub Repo of LightGBM, to check the codes by yourself.
I hope that I could give you an insight. If you come up with an exact answer on the matter, please let me know. Please do not hesitate to further discuss this, I'll be around. Good luck, have fun!


How synthetic sampling of data helps classifier to predict better result? I am currently working on data imbalance using SMOTE for binary and other algorithms for the multi-class problem.

I have the idea how to create the synthetic example to bring noticeable accuracy on a given dataset.

I want to go into deep and understand how a classifier, especially SVM handle the data with the synthetic example to classify more accurately. It would also helpful to know for other techniques like boosting algorithms, random forest etc.

Any kind of guidance on above question will be very much helpful.

I have already asked this question on StackExchange here but didn't get any answer.

I think that in order to understand how the SVM handles the new synthetic
data, you should look at the loss function SVM uses, i.e. hinge loss and the behavior on an imbalanced dataset. Intuitively this function will try to fit the hyperplane that best separates the data. For example imagine you have a dataset that is not linearly separable and you have to classes A (with a 1000 samples) and B (with 50 samples). So the SVM will "move" the hyperplane to classify right all samples of A even if this means failing to classify correctly some samples of class B.Then when you create new samples of the classes B the classifier will try to find a balance between the error of the both classes. A more detailed explanation can be found in the 6.3 section fo this article

In general all classifiers tend to classify correctly the most dominating class because of the loss function they use. One way of dealing with it is generating synthetic data, other way is using a cost sensitive classifier, this is the cost of misclassification of the minority class is higher than the majority class. More details can be found in the following links:

https://svds.com/learning-imbalanced-classes/

http://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/


Multiclass classification task where each class is present only once in the test set I have a multiclass classification problem where, in the test set, there is only one entry for each possible class. In my particular problem we want to guess the author of a text, and we have 20 different authors. The test set contains 20 texts, one of each author.

I have many texts of each authors in the training data. I can't change the content of the test set, it's these specific texts that I need to classify.

Does this kind of task has a name, so I can google previous work on it more easily ? I'm not talking about author identification specifically, but having 1 entry for each class.
Do you have any suggestion of algorithm that would work well for this task ? Maybe something that gives ranked results or a confidence value ?

Thank you

When talking about the test data, we can have any number of examples of each class for inference. It doesn't matter if it has 20 classes with one example each.

You can use a CNN classifier for this type of a problem but if one book can be written by more than one author you should use sigmoid activation in the last layer rather than using softmax.

You can see this article which address an usecase similar to yours - Link


In supervised learning, what does "Estimating $p(y \vert x)$" mean? I read chapter 5.1.3 of Joshua Bengio's deeplearning book, which says:
supervised learning involve observing examples of random vectors $\textbf{x}$ and associated value or vector $\textbf{y}$ and learning to predict $\textbf{y}$ from $\textbf{x}$ by estimating $p(\textbf{y} \vert \textbf{x})$.

What does $p(\textbf{y} \vert \textbf{x})$ mean?

From basic statistics, I know that $p(\textbf{y} \vert \textbf{x})=\frac{p(\textbf{y},\textbf{x})}{p(\textbf{x})}$.
How do we find $p(\textbf{y},\textbf{x})$ and  $p(\textbf{x})$?

You are correct that
\begin{equation*}
p(y|\mathbf{x})=\dfrac{p(\mathbf{x},y)}{p(\mathbf{x})}.
\end{equation*}
Similarly, we can write the joint probability $p(\mathbf{x},y)$ as follows:
\begin{equation*}
p(\mathbf{x},y)=p(\mathbf{x}|y)\cdot p(y)
\end{equation*}
From the above two equations, we obtain
\begin{equation*}
p(y|\mathbf{x})=\dfrac{p(\mathbf{x}|y)\cdot p(y)}{p(\mathbf{x})}
\end{equation*}
In the context of supervised learning, the variable $y$ is used to denote the class labels, and  the vector $\mathbf{x}$ for measurement vector or feature vector. For the purpose of discussion, let us assume that the class label $y$ takes values in the set $\{1,2\}$, where $1$ denotes  male and $2$ denotes female. Similarly, $\mathbf{x}$ is a measurement vector on two variables say, $(x_{1},x_{2})$, where $x_{1}$ stands for height and $x_{2}$ stands for weight of individuals.

$p(y|\mathbf{x})$ denote the posterior density for $y$ given the observation $\mathbf{x}$. For example, $p(1|\mathbf{x})$ means that given the observation  $\mathbf{x}$, what is the probability that sample belongs to the class males. Similarly, we can interpret $p(2|\mathbf{x})$.
$p(\mathbf{x}|y)$ stands for the class conditional probability density. For example, $p(\mathbf{x}|1)$ denote the probability density for males and $p(\mathbf{x}|2)$ denote the probability  density for females respectively. In supervised learning, these class conditional densities are usually known in advance. Finally $p(y)$ denote the prior probability for the class label $y$. For example, $p(1)$ denotes the probability that an individual/example selected randomly from the population is a male.

Assuming that, the prior probabilities for the classes and class conditional densities are known, Bayes rule says, assign an observation $x_{0}$ whose class label is not known, to the class, say, male, if
\begin{equation*}
p(\mathbf{x_{0}}|1)p(1)>p(\mathbf{x_{0}}|2)p(2).
\end{equation*}
Note that, $p(\mathbf{x})$ is not used in the classifier. Classification rules usually require class conditional densities.


Tensorflow regression predicting 1 for all inputs I'm trying to predict a UPDRS score (regression) from the parkinsons dataset. I wanted to start with a fairly basic structure and the add in more layers/techniques to improve performance but no matter what I do the network constantly predicts 1 for every input.

I've been banging my head against this problem for a couple of days now with no success. Originally I was using RELU and thought it was a case of dead neurons but using leaky RELU and monitoring the outputs I can see them firing.

Changing the network structure or training steps always results with the same horrible MSE of 896.39105. This leads me to believe the mistake is somewhere in the normalization but when I check the intermediate values they all seem fine.

Can anyone tell me where I've messed up?

import tensorflow as tf
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.cross_validation import train_test_split
from sklearn.preprocessing import normalize
from sklearn.preprocessing import StandardScaler

def leaky_relu(x, alpha):
  return tf.nn.relu(x) - alpha * tf.nn.relu(-x)

raw = pd.read_csv('parkinsons_updrs.data')
print("CSV loaded successfully")

#Split into features and targets
features = raw.drop(['subject#','test_time','motor_UPDRS','total_UPDRS'], axis = 1).as_matrix()
targets = raw['total_UPDRS'].get_values()

scale = StandardScaler(with_mean=0, with_std=1)
scale.fit(features, targets)
scaled_feats = scale.transform(features)

print("pre-processing complete")

#Split data into test/train
data_train, data_test, targets_train, targets_test = train_test_split(scaled_feats,targets,test_size = 0.2,random_state=123)

print("Train/Test split complete")

#Need to reshape to match placeholder shape
targets_train = targets_train.reshape(targets_train.shape[0],-1)
targets_test = targets_test.reshape(targets_test.shape[0],-1)

#Network Structure
X = tf.placeholder(tf.float32, [None,features.shape[1]])
y = tf.placeholder(tf.float32, [None,1])
prob = tf.placeholder(tf.float32)

w1 = tf.Variable(tf.random_normal([features.shape[1],100]))
w2 = tf.Variable(tf.random_normal([100,1]))

b1 = tf.Variable(tf.random_normal([100]))
b2 = tf.Variable(tf.random_normal([1]))

layer1 = tf.nn.dropout(leaky_relu(tf.add(tf.matmul(X,w1),b1),0.2),prob)
out = tf.nn.dropout(tf.nn.tanh(tf.add(tf.matmul(layer1,w2),b2)),prob)

loss = tf.reduce_mean(tf.square(out - y))
train_step = tf.train.AdamOptimizer(0.001).minimize(loss)

sess = tf.Session()
sess.run(tf.global_variables_initializer())

print("Training...")

minibatch = False
batchsize = 512

for i in range(1000):
    if minibatch:
        for x in range(0,targets_train.shape[0],batchsize):
            sess.run(train_step, feed_dict={X:data_train[x:x+batchsize],y:targets_train[x:x+batchsize],prob:0.5})
    else:
        sess.run(train_step, feed_dict={X:data_train,y:targets_train,prob:0.5})
    if i%25==0:
        acc = sess.run(loss, feed_dict={X: data_train, y: targets_train,prob:1})
        print ("Training accuracy: %.5f" % (acc))
        #print(sess.run(layer1,feed_dict={X: data_train, y: targets_train,prob:1})[0])

test_acc = sess.run(loss, feed_dict={X: data_test, y: targets_test,prob:1})
print ("Test accuracy: %.5f" % (test_acc))

#Check if still predicting 1
output = sess.run(out,feed_dict={X: data_test, y: targets_test,prob:1})
print(sum(output)==len(output))

Ok, I've managed to get a decent improvement. Currently producing an MSE of 20 for training and 30 for test. This is after converting all layers to relu and increasing their size.

The problem appeared the activation function on the output. I was under the impression that a sigmoid output layer was common practice.

I was also normalizing before the test/train split and leaking information into the test set! (though this likely wasn't the cause of my problem)

Here is my 'improved' code:

import pandas as pd
import numpy as np
from scipy import stats
from sklearn.cross_validation import train_test_split
from sklearn import preprocessing

raw = pd.read_csv('parkinsons_updrs.data')
print("CSV loaded successfully")

events = raw.shape[0]
#drop events where any feature has a zscore > 3 for that feature
raw = raw[(np.abs(stats.zscore(raw)) &lt 3).all(axis=1)]
print("Removed " + str(events-raw.shape[0]) + " outlier events")

features = raw.drop(['subject#','test_time','motor_UPDRS','total_UPDRS'], axis = 1)
targets = raw['total_UPDRS'].get_values()
print("pre-processing complete")

#Split data into test/train
data_train, data_test, targets_train, targets_test = train_test_split(features,targets,test_size = 0.33,random_state=123)
print("Train/Test split complete")

#Normalize data to mean 0 and std 1
#We normalize after the split to avoid leaking information to the training set
data_train = (data_train-data_train.mean())/data_train.std()
data_test = (data_test-data_test.mean())/data_test.std()

targets_train = targets_train.reshape(targets_train.shape[0],-1)
targets_test = targets_test.reshape(targets_test.shape[0],-1)

#Network Structure
X = tf.placeholder(tf.float32, [None,18])
y = tf.placeholder(tf.float32, [None,1])
prob = tf.placeholder_with_default(1.0, shape=())

w1 = tf.Variable(tf.random_normal([18,72]))
w2 = tf.Variable(tf.random_normal([72,36]))
w3 = tf.Variable(tf.random_normal([36,1]))

b1 = tf.Variable(tf.random_normal([72]))
b2 = tf.Variable(tf.random_normal([36]))
b3 = tf.Variable(tf.random_normal([1]))

layer1 = tf.nn.dropout(tf.nn.relu(tf.add(tf.matmul(X,w1),b1)),prob)
layer2 = tf.nn.dropout(tf.nn.relu(tf.add(tf.matmul(layer1,w2),b2)),prob)
out = tf.nn.relu(tf.add(tf.matmul(layer2,w3),b3))

loss = tf.losses.mean_squared_error(y, out)

train_step = tf.train.AdamOptimizer(0.005).minimize(loss)

sess = tf.Session()
sess.run(tf.global_variables_initializer())

print("Training...")

minibatch = True
batchsize = 512

for i in range(10000):
    if minibatch:
        for x in range(0,targets_train.shape[0],batchsize):
            sess.run(train_step, feed_dict={X:data_train[x:x+batchsize],y:targets_train[x:x+batchsize],prob:0.5})
    else:
        sess.run(train_step, feed_dict={X:data_train,y:targets_train,prob:0.5})
    if i%25==0:
        acc = sess.run(loss, feed_dict={X: data_train, y: targets_train})
        print ("Training accuracy: %.5f" % (acc))

test_acc = sess.run(loss, feed_dict={X: data_test, y: targets_test})
print ("Test accuracy: %.5f" % (test_acc))

output = sess.run(out,feed_dict={X: data_test, y: targets_test})

error = abs(output-targets_test)

print(max(error))
print(sum(error)/len(error))
print(min(error))

I'd be very interested to know why this happened. Both tanh and sigmoid get 'stuck' in the 900s whereas relu sorts itself out and carries on.


Which model is better for incremental learning? I'm trying to implement face recognition. I'm planning to use some model (like DeepFace) to extract discriminative features and then use a classifier to recognize the faces. I'm confused as to which classifier to use.

My setup is as follows:
Initially, I'll have a few labelled samples and I train a classifier. Then I start getting more labelled samples and I want to update my classifier with the new data. Also, the newer samples may contain new classes (faces). So, I want my classifier to be able to learn to classify new classes as well. My question is which classifier is better suited for this task. I would also prefer to get multiple predictions with a confidence score.

K Nearest Neighbors algorithm: Here I can use distance as some form of confidence score.
As presented here, I can use SVM: Train on the initial data and retain support vectors alone. Repeat the process as and when new data comes up.

But, I have no idea if the above 2 classifiers are suited for the task at hand or how they would perform. Any inputs regarding this is greatly appreciated. I'm also open to any other classifiers as well. Thanks!

I will say, it's an either Or situation
You can pick one of "Incremental/Online" training Or "addition of new class".
 You may do a fine-tuning approach with a Neural network by adjusting the o/p layer and training the last few layers. But this approach expects the new data to be quite similar to the training set.
KNN - Can do the online stuff but it doesn't do training. It simply calculates all the distances at the time of prediction. So, no reduction in computing. But you might have to compromise on accuracy if it is not the best
Scikit-Learn SGD Classifier can help with online training but can't support new classes

For classification, a somewhat important thing to note is that although a stateless feature extraction routine may be able to cope with new/unseen attributes, the incremental learner itself may be unable to cope with new/unseen targets classes.
SGDClassifier

Other models e.g. SVM/DT doesn't support incremental learning naturally. Though there are suggested approaches on the internet. But may not be simple. See these references.
journalofbigdata$\hspace{1cm}$
A Good SE read
I am not sure how will you figure it out if it is a new Class unless you have a separate arrangement because the model will predict it an existing Class anyway. Assuming you have a setup for that. We may try below approach -
Till the time, the prediction is within "known class" - do an Online training.
When the data is for a "New Class", do a full-data training.
 With this approach, you can reduce the frequency of full-data training.
A simple Neural network can be a good candidate for both purposes. You can also get class probability with it.
Please see these useful links too
Creme
Keras + online learning


what do we actually study in machine learning? I just started learning machine learning and learned a few basic algorithms and there is one stupid doubt in my mind and I am unable to find the answer of it. What do we actually study in machine learning algorithms that solves kinds of problems based on the data given to the algorithms?

Take an example of linear regression:

We give data.
We tell machine how to calculate error.
Using gradient descent also we are telling machine how to optimize and based on all above some optimized parameters come out.

Why there is actually a "learning" word in this algorithm? Although we are the telling how to perform steps and we have provided all the mathematics in this algorithm.

I want to point out one aspect which is not adequately reflected in the previous answers. When you start with linear regression, you basically I) propose some model (a way how to represent the relation between $y$ and $X$), e.g. a linear function, and II) you specify a loss function (the sum of squared residuals). The aim is to find coefficients (or weights), which map $X$ to $y$ in a way so that there is no “better way” to predict $y$ (viz. minimise the sum of square residuals).

In the case of linear regression, you have two options: solve the problem analytically or by gradient decent. Optimal coefficients/weights (vector beta) are given by:

$$ (X'X)^{-1} X'y=\beta. $$

So there is no need for gradient decent in this case. It is just simple matrix algebra.

Here is a minimal example in R, demonstrating how it works:

Source: https://www.r-bloggers.com/regression-via-gradient-descent-in-r/

# Input some data
x0 <- c(1,1,1,1,1)
x1 <- c(1,2,3,4,5)
x2 <- c(8,4,3,1,8)
x <- as.matrix(cbind(x0,x1,x2))
y <- as.matrix(c(3,7,5,11,14))

# (X'X)^-1 X'y
beta1 = solve(t(x)%*%x) %*% t(x)%*%y

# R's regression command
beta2 = summary(lm(y ~ x[, 2:3]))

# Gradient decent
m <- nrow(y)
grad <- function(x, y, theta) {
  gradient <- (1/m)* (t(x) %*% ((x %*% t(theta)) - y))
  return(t(gradient))
}

# Define gradient descent update algorithm
grad.descent <- function(x, maxit){
  theta <- matrix(c(0, 0, 0), nrow=1) # Initialize the parameters

  alpha = .05 # Learning rate
  for (i in 1:maxit) {
    theta <- theta - alpha  * grad(x, y, theta)
  }
  return(theta)
}

# Results
print(grad.descent(x,2000))
print(beta1)
print(beta2)

Results:

> print(grad.descent(x,2000))
            x0     x1        x2
[1,] -0.672238 2.6459 0.1530299
> print(beta1)
         [,1]
x0 -0.6722955
x1  2.6459103
x2  0.1530343
> print(beta2)

Call:
lm(formula = y ~ x[, 2:3])

Residuals:
      1       2       3       4       5
-0.1979  1.7683 -2.7245  0.9356  0.2185

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  -0.6723     3.3565  -0.200   0.8598
x[, 2:3]x1    2.6459     0.7677   3.447   0.0748 .
x[, 2:3]x2    0.1530     0.3897   0.393   0.7325
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 2.399 on 2 degrees of freedom
Multiple R-squared:  0.8561,    Adjusted R-squared:  0.7122
F-statistic: 5.949 on 2 and 2 DF,  p-value: 0.1439

So as you can see, there is not much of a machine involved in solving the problem (apart of doing the math of course). So why machine learning? Increasing computational power has made it possible to solve very complex problems using gradient decent. The idea is that you depart from simple solutions and look for better, more complex ways to map $X$ to $y$.

You can think about changing the base procedure, e.g. use decision trees instead of a linear function. Since one tree may be only a weak learner, you can combine (or ensemble) trees to a random forest. Or you go even further and use boosting, in which you repeatedly use information contained in the residual to improve your fit. Usually very small trees are used to do this, but you can boost on linear regression as well.

Here is a little example for L2 boosting:

(from: Boosting Algorithms: Regularization, Prediction and Model Fitting)

library(glmnet)

# Simple boosting example from:
# P. Bühlmann, T. Hothorn (2007), "Boosting Algorithms: Regularization, Prediction and Model Fitting", Statistical Science 22(4), p. 477-505.
# L2-Boosting from Section 3.3 (p. 483)

#######################
# Data
x = matrix(c(130,165,150,150,140,198,307,350,318,304,302,429),nrow=6,ncol=2,byrow = TRUE)
y = c(18,15,18,16,17,15)

# Parameter
# This works like a learning rate
nu = 0.1
# Early stopping (SSR value to stop)
es = 7.58765583743327
# Max. iterations (of boosting)
maxiter = 10000

#######################
# OLS benchmark model
olsdata = data.frame(cbind(y,x))
ols = lm(y~.,data=olsdata)
olspreds = predict(ols,newdata=olsdata)
olsssr = sum((y-olspreds)^2)
print(paste0("OLS SSE: ",olsssr))

#######################
# Boosting

# Initialize f0
f0 = mean(y)
f0ssr = sum((y-f0)^2)

# Lists to store results per boosting iteration
ssrlist = list()
bstep = list()

# Boosting (p. 483, Sec. 3.3, L2-Boosting)
for (n in seq(1:maxiter)){
  # I) Get residual
  if(n==1){f=f0} # initialize in first step
  u=y-f          # get residual from estimation
  # II) Fit residual / predict, I use ridge (alpha=0)
  reg = glmnet(x,u,alpha=0, lambda=2)
  g = predict(reg, newx=x, type="link")
  # III) Update f
  f=f+nu*g
  # Print feedback
  cat(paste0("Step: ", n," SSR: ", sum(u^2), "\n" ))
  # Save SSR/iter to list
  ssrlist[[n]] <- sum(u^2)
  bstep[[n]] <- n
  # Early stopping rule
  if(sum(u^2)<es){break}
}

#######################
# End statement
cat("\n ~~~ RESULTS ~~~ \n")
cat(paste0("OLS SSR:           ", olsssr, "\n"))
cat(paste0("Initialised SSR:   ", f0ssr, "\n"))
cat(paste0("Last SSR boosting: ", ssrlist[length(ssrlist)], " [after ",bstep[length(bstep)], " iterations] \n"))

The problem here is that you do not only need to solve one regression, but 10000 (in the example here). In real world applications, the amount of math that has to be done is immense (counts for boosting and neural nets as well).

So what is machine learning? You want to let “the machine learn things” (recognise patterns in data, recognise images, make prediction based on that etc) and you will likely be unable to do this learning task on your own using pen and paper. Improved computing capacity over time has enabled us to solve very complex problems. And because of this cheap resource, new tools and methods develop. You can do things today on your laptop which literally seemed impossible in the late 1990s.

Of course you need to tell “the machine” how to learn. This is a lot of statistics but also computer science. Neural nets for example have mostly been advanced by computer scientists. Ultimately, the machine is merely a tool, but a very useful one.


Using Vowpal Wabbit for NER The Vowpal Wabbit (VW) apparently supports sequence tagging functionality via SEARN. The problem is that I cannot find anywhere detailed parameter list with explanations and with some examples. The best I could find is Zinkov's blog entry with a very short example. The main wiki page barely mentions SEARN.

In the checked out source code I found demo folder with some NER sample data. Unfortunately, the script running all the tests does not show how to run on this data. At least it was informative enough to see what is the expected format: almost the same as standard VW data format, except that entries are separated by blank lines (this is important).

My current understanding is to run the following command:

cat train.txt | vw -c --passes 10 --searn 25 --searn_task sequence \
--searn_passes_per_policy 2 -b 30 -f twpos.vw

where

--searn 25 - the total number of NER labels (?)

--searn_task sequence - sequence tagging task (?)

--searn_passes_per_policy 2 - not clear what it does

Other parameters are standard to VW and need no additional explanation. Perhaps there are more parameters specific to SEARN? What is their importance and impact? How to tune them? Any rules of thumb?

Any pointers to examples will be appreciated.

John Langford's documentation on GitHub could help. You can find something on the Learning to Search Sub System page.


selecting a number of neurons specifically for RL If I would like to use DQN to train my Reinforcement-Learning agent, how do I select the number of neurons?

In Supervised Learning, selecting too few or too many can result in either  a too low potential (lack of neurons) or in a poor generalization due to overfitting (for example, due to too many neurons). We can use Genetic Algorithm to determine the best combination of hyperparameters instead of trying by hand.

However, in RL - is generalization even a thing? After all, we are training a spider to run by moving its limbs in a most efficient way, an arm to grab a box in the fastest way, etc.

Once learnt, are we even allowed to change the environment and expect agent to perform well, generalizing to new environment?

And thus, is selecting neurons even a thing for RL, or can we just give it a maximum possible number of neurons and layers, letting it learn correct movement of creature's limbs?

Deep reinforcement learning (DRL) methods such as DQN involve deep learning as a large memory that is not expensive. The neural network learns in a supervised way to provide good estimates; and DQN is not an exception.

Hyper-parameters optimization for the neural network in DQN is similar to that of fully supervised learning. you should try various hyper-parameters[ number of layers, neurons,...etc] until obtaining a good solution.

Evolutionary algorithms can help you find appropriate hyper-parameters. Recently there are some published papers reported using evolutionary algorithms to train the neural network in RL.

are we allowed to change the environment and expect agent to perform well, generalizing to new environment?

Yes,the main idea of DQN is to allow generalization, see DeepMind publications such as [1]. And generalization to unseen environments is possible : see [2], [3]

As you can see in the literature, the boundaries of the generalization differ form one work to another. Generalization might mean the ability to perform well for different experiments at the same environment. Some researcher work on building agents that generalize to unseen environments which is more difficult and in this case the design of the neural network and the reinforcement learning algorithm play important role in determining the generalization ability.

is selecting neurons even a thing for RL, can we just give it a max possible number of neurons and layers, letting it learn correct movement of creature's limbs?

Generalization is an issue,  large number of neurons can overfit (not generalize well) DQN involves neural network in order to allow generalization. The ability of the neural network to generalize to new cases (and new experiments/environments) depends on the approximation that the neural network provides which depends on the neural networks parameters and hyper-parameters.

For building a huge neural network : as a concept you can increase the depth of the neural network to get better results. However, the question is do you have enough data? in RL language, you need more different examples for the larger network to train. Theoretically, having infinite neural networks with infinite number of different experiments/environments will lead to the best results. however, we are not patient enough to wait for very long training time and most probably you don't have large number of different environments/experiments to teach the neural network. hence the standard way to do this is by starting with a small network (few neurons and 1 or 2 layers) and train it then expand it gradually step by step and stop when the network can't generalize well.

you can think of the size of the neural network as a water tank as long you have large amounts of water(experiments) you need larger tank to occupy it. A large network without sufficient number of learning experiments leads to worse results specially when the the agent face new unseen experiments.

:[1] https://web.stanford.edu/class/psych209/Readings/MnihEtAlHassibis15NatureControlDeepRL.pdf

:[2] https://www.semanticscholar.org/paper/Virtual-to-real-deep-reinforcement-learning%3A-of-for-Tai-Paolo/494af0cedf1abb2454d457d0a89e21b983233276

:[3] https://arxiv.org/pdf/1612.05533.pdf


Given is the result of the model performance. Help me with this MCQ You also evaluate your model on the test set, and find the following:

Human-level performance 0.1%
Training set error  2.0%
Dev set error   2.1%
Test set error  7.0%
What does this mean? (Check the two best options.)

You have underfit to the dev set.
You should get a bigger test set.
You should try to get a bigger dev set.
You have overfit to the dev set.

I think answer are options 3) and 4)

Clearly there seems to be overfiting towards dev set as the dev set error is low and almost equal to training set error.

And  increasing the number of samples helps minimizing overfiting problem..


Classification with feature not available at time of model creation I have problem statement to predict the probability of solving a task depending on multiple features for e.g. when the task was created, the time needed to work on a task, etc Please find a dummy snippet attached

task_id  date_time_open    time_needed   day_created  time_created   status

aa      12/09/2019             20 hrs     Tuesday        3 pm      done
cc      17/10/2019             4 hrs      Friday        10 pm      not_done

I know I can run a classification model to identify the class. However, things complicate when  I add a time dimension to it since the data set now gets an added feature which highly impacts the status

The task was scanned at suppose 7 pm and a new feature added for 7 pm

task_id  date_time_tsk_open    time_needed     day_created  time_created    status_7pm     status

    aa      12/09/2019               20 hrs     tuesday       3pm              done           done
    cc      17/10/2019               4 hrs      friday        10 pm            done           not_done
    dd      19/10/2019               6 hrs      friday        2 pm             done          done
    ff      19/10/2019               9 hrs      Monday        4 pm             not_done      not_done

The task id was again scanned at a fixed interval of 1 hr and added new features to data

task_id  date_time_tsk_open    time_needed     day_created  time_created    status_8pm     status

    aa      12/09/2019               20 hrs     tuesday       3pm              done         done
    cc      17/10/2019               4 hrs      friday        10 pm            not_done     not_done
    dd      19/10/2019               6 hrs      friday        2 pm             done            done
    ff      19/10/2019               9 hrs      Monday        4 pm             not_done        not_done

The final prediction of status == resolved / un_resolved in my understanding should be based on features including status_7pm and status_8pm.

How should the data structure for training such a classification model look like to generate a prediction at time 9 pm for sample task ff respectively

  task_id  date_time_tsk_open    time_needed     day_created  time_created    status_7pm status_8pm     status

    ff      19/10/2019               9 hrs      Monday        4 pm            not_done           not_done      not_done

I assume the classification model should be trained on all status_1, status_2 ....status_8pm to classify  status. Or would the model be trained every time in memory once it gets a new column updated status every hour

It seems the simplest way to go would be to build a line at each time step after 'time created'. With 'status_n-1', 'status_n'. That will allow to deal  with the notion of time in rows. You also might want to :

Deal with time relatively : instead of considering status at a given hour, you probably want to work with status since the creation of the task.
You will need to deal with task ponderation (longer tasks will get more rows) one way or another. You may add some ponderation in your model, based on 1/(expected length).
Add some features : to me it is unclear what feature you use for prediction. As is, you seems to be trying to predict time of completion based on start time / expected length. You won't learn much except for which tasks takes more than expected / some daily effects. I think this can be achieved more efficiently / more clearly with simple statistics.


Clustering: How to cluster multiple CSV files that each represent a Steam user I am currently about to do clustering analysis regarding Steam users activity. So I have a thousands of CSV’s, each representing a Steam user and his/her purchased games (with ID and genre). I am planning to use k-modes clustering because my data is categorical, and I have found a python library k-modes. This is what a CSV file looks like, representing a random steam user.

I have researched a lot online, but it seems that many people have all their data in one CSV file, just like this example of stock exchange data.

How could I read each file and represent it as one unique data then do clustering on all the data among the thousands of CSV files?

You're likely going to have to do a little data wrangling to get the data in a better format.

I'm assuming each file has a varying amount of rows, one for each game the user purchased? So, you could create one big matrix with the rows representing users and the columns representing games, and create an indicator matrix for to map purchases to games. Then potentially add additional columns per user including additional insight, like:

the number/percent of games purchased for each category(action, RPG, ...)
total purchases

Having data in that format may lend to more useful application of clustering algorithms.


Max-pooling vs. zero padding: Loosing spatial information When it comes to convolutional neural networks there are normally many papers recommending different strategies. I have heard people say that it is an absolute must to add padding to the images before a convolution, otherwise to much spatial information is lost. On the other hand they are happy to use pooling, normally max-pooling, to reduce the size of the images. I guess the thought here is that max pooling reduces the spatial information but also reduces the sensitivity to relative positions, so it is a trade-off?

I have heard other people saying that zero-padding does not keep more information, just more empty data. This is because by adding zeros you will not get a reaction from your kernel anyway when part of the information is missing.

I can imagine that zero-padding works if you have big kernels with "scrap values" in the edges and the source of activation centered in a smaller region of the kernel?

I would be happy to read some papers about the effect of down-sampling using pooling contra not using padding, but I can't find much about it. Any good recommendations or thoughts?

Figure: Spatial down-sampling using convolution contra pooling (Researchgate)

Pooling (max/mean/etc) has two primary benefits: it significantly reduces computational complexity (at the cost of potentially important data) and it helps the network achieve spatial invariance by making the spatial relativity between features less relevant.

However, the spatial invariance achieved through pooling is not always ideal.    For instance, given a CNN trained to identify images of automobiles, it would probably classify an image containing a hood/trunk, tires, windows, car doors, etc. as an automobile, regardless of the components' relative positioning.

On the other hand, I do not know of significant costs to using padding.  Zero padding is critical for deep networks; without it, our volumes would rapidly collapse through the layers.  It therefore helps to maintain desirable volume sizes and to preserve the border data.  It also helps the geometry of CNNs to work out smoothly by preventing mismatched dimensions between layers.

You might find this paper interesting.  Hinton and his team propose a new network called Capsule Network (CapsNet) that addresses the problems with pooling and provides an alternative solution for acquiring spatial invariance.  One of the many exciting parts of their algorithm is that the location of features relative to one another is a primary asset in the classification process, bringing computers closer to a "true understanding" of objectness.


ML, Statistics and Mathematics I have just started getting my hands wet in ML and every time I try delving deeper into the concepts/code, I face the challenges of the mathematics and its cryptic notations.
Coming from a Computer Science background, I do understand bit of them but majority goes tangent.
Say, for example below formulae from this page -

I try and really want to understand them but somehow get confused and leave it everytime.
Can you please suggest how to start with it? Any starting pointers or advise please.

It is quite true that papers or books use notations that sometimes seem obvious to people who are used to dealing with the mathematical aspects, but are meaningless for the others. Ways of understanding the math include:

Following theoretical courses or trainings
Reading dedicated books
Asking people around you
Asking people on forums such as this one, or Cross Validated for stats formulae
Getting it by yourself upon re-reading parts of the paper/book you didn't get at the first time

There are some notations/conventions that are implicitly accepted in data science / machine learning papers, such as:

Using $X$ as input, $y$ as output, $\theta$ as model parameters
Using $\hat{y}$ for the estimator of the true $y$
Assuming that vectors are column vectors

The list would be too long to include here.

Regarding the example above, what we face is a constrained optimization.

The $max$ statement means that we are looking for a maximum value of the expression that follows. What is below (namely, the $\Delta_{ij}$ values) the $max$ is the list of "free" parameters that change the value of the expression.

The $max$ statement is prefixed by $arg$, which means that we do not have interest in the expression's maximum value, but rather in the $\Delta_{ij}$ set that leads to that value.

Then we face a $s.t.$ statement, because this is no ordinary optimization, we also have to respect the several constraints expressed after $s.t.$. Those can be inequations, equations, belonging constraints, etc., either explicit ($\Delta_{ij} > 0$) or more implicit.


Are weights of a neural network reset between epochs? If an epoch is defined as the neural network training process after seeing the whole training data once. How is it that when starting the next epoch, the loss is almost always smaller than the first one? Does this mean that after an epoch the weights of the neural network are not reset? and each epoch is not a standalone training process?

An epoch is not a standalone training process, so no, the weights are not reset after an epoch is complete. Epochs are merely used to keep track of how much data has been used to train the network. It's a way to represent how much "work" has been done.

Epochs are used to compare how "long" it would take to train a certain network regardless of hardware. Indeed, if a network takes 3 epochs to converge, it will take 3 epochs to converge, regardless of hardware. If you had used time, it would be less meaningful as one machine could maybe do 1 epoch in 10 minutes, and another setup might only do 1 epoch in 45 minutes.

Neural networks (sadly) are usually not able to learn enough by seeing the data once, which is why multiple epochs are often required. Think about it as if you were studying a syllabus for a course. Once you finished the syllabus (first epoch), you go over it again to understand it even better (epoch 2, epoch 3, etc.)


Where is the cost parameter C in the RBF kernel in SVM? RBF kernel using SVM depends on two parameters C and gamma. If the equation of the kernel RBF as the following:

$K(X,X')= \exp(\gamma||X-X'||^2)$

In the equation I can see where can I use gamma, but I can't find the C parameter.
So, can enybody tell me please?

Thanks in advance,

The C parameter in SVMs doesn't have to do anything with the kernel function. C is the penalty associated to the instances which are either misclassified or violates the maximal margin. As you may know already, SVM returns the maximum margin for the linearly separable datasets (in the kernel space).

It might be the case that the dataset is not linearly separable. In this case the corresponding SVM quadratic program is unsolvable. To make it solvable, we redefine the quadratic program to return the maximum margin with respect to some error cost C for the points violating the maximum margin (misclassified samples also violate the maximum margin). Below you can see a simple example I found on the Internet. This uses linear kernel but the story is the same for other non-linear kernels as well.

The effect C has on the classification result is shown in the picture below:

As you can see, if you set the error penalty to be very high, you will end up with less misclassifications and fewer support vectors (the highlighted points are SVs). However, if you set it too high when you are using non-linear kernels (specially Gaussian) you might overfit.

If you are interested in the theory behind it, here is what we are trying to optimize:

If we write its dual, we'll get:

See how convenient it is! No C in the kernel function!

In other words, you are mapping the samples to a higher dimensional space using the kernel function, then solving it using some penalty term C.


Is it possible to use deep learning on small number of samples but each sample has a large amount data? I have around 50 subjects and each subject has weeks worth of time series data.  The task is to identify whether a subject has as specific condition.  We have been running wavelet transformations in 30s windows across the time series and then summarizing across spectral band groupings to get a data matrix of subject x spectral band (50x6).  Although this method does reasonably well at detecting the condition, I'm nervous that this may wash away important temporal information.

Is it possible to use the wavelet transformations as images and feed this through a convolutional network?

To clarify, there are 50 subjects each with ~350 30s intervals of data.  The wavelets are run on each 30s interval producing 350 wavelet transformations per subject.  The wavelet transformations (previously) are summarized into power per spectral band, however, I would like to use each wavelet transformation as an image so each subject would now have ~350 images.

You have a database of 50x6 currently.

When you run a exercise of deep learning on images, you have n (images) x (width(px) x height(px) x 3) variables.

You have similarly characterized dataset when you use the wavelet transforms, so it is possible. You have to be very cautious because you will have more variables than n, so the danger of overfitting increases a lot.


Is the number of bidirectional LSTMs in encoder-decoder model equal to the maximum length of input text/characters? I'm confused about this aspect of RNNs while trying to learn how seq2seq encoder-decoder works at https://machinelearningmastery.com/configure-encoder-decoder-model-neural-machine-translation/.

It seems to me that the number of LSTMs in the encoder would have to be the same as number of words in the text (if word embeddings are being used) or characters in the text (if char embeddings are being used). For char embeddings, each embedding would correspond to 1 LSTM in 1 direction and 1 encoder hidden state.

Is this understanding correct?

E.g. If we have another model that uses encoder-decoder for a different application (say text-to-speech synthesis described here https://ai.googleblog.com/2017/12/tacotron-2-generating-human-like-speech.html) tha uses 256 LSTMs in each direction of the bidirectional-encoder, does it mean the input to this encoder is limited to 256 characters of text?

Can the decoder output has to be same length as the encoder input or can it be different? If different what factor describes what the decoder output length should be?

There is a slight problem with the semantics that you have used.
RNN is a 'recurring' neural net. LSTM is a type of RNN. So when you say "number of LSTMs in the encoder would have to be the same as number of words in the text", I suppose what you mean to say is that the number of time-steps or number of NN's in the (single) LSTM layer is the same as the number of words in the sentence. If this is what you actually meant, then you are correct. For e.g. If you have a 20 word tweet and a set of 10000 such tweets. You can use a single LSTM layer with num of timesteps=20 to process this. You do NOT use 20 LSTM layers.
This should answer your second question also. The decoder too does not have 20 LSTMs. It has probably 1 LSTM layer with 20 time-steps.
The decoder output length need not match exactly with the encoder output length. In other words it need not be a word for word translation. The models typically use BOS and EOS markers to start and stop processing.


What does many low important feature indicate? I have a dataset where I am focusing on binary classification problem. In total,I have around 60 features in my dataset

When I used Xgboost Feature Importance, I was able to see that the top 5 features account for 42% whereas rest of the of the 50 features account for 40-49 % (each feature about 1%) and remaing 8-10 features have zero importance or less than 1% of importance.

This is my best paramter list for Xgboost after gridsearch

op_params = {'alpha': [10],
 'as_pandas': [True],
 'colsample_bytree': [0.5],
 'early_stopping_rounds': [100],
 'learning_rate': [0.04],
 'max_depth': [6],
 'metrics': ['auc'],
 'num_boost_round': [10000],
 'objective': ['reg:logistic'],
 'scale_pos_weight': [3.08],
 'seed': [123],
 'subsample': [0.75]}

Since I have many low importance features, should I try to use them all in my model to increase the model metrics?

When I built the model only with top 5 features, I was able to get 80% accuracy.

I am trying to understand is it even useful to make use of these low importance feature for prediction?

Shown below is my feature importance in descending order

Do they even really help?

Any insights would really be helpful

Adding low-value features might not help you surpass your current accuracy.
Getting good quality data and add more data to the dataset or training for more epochs if it doesn't converge might help you get more accuracy.


Relation between hyperparamters and training set for an object detection model I have 2 instances of an object detection model. The only difference between these two models is the training data used:

The first model was trained with a small training set
The second model was trained on a larger training set than the first one

The first model was trained on the following hyperparameters:

Number of iterations: 250k
Batch Size: 10
Learning Rate: warms up to 0.001 and decreases to 0.0002 after 150k iterations

Since the second model has more training data, I assumed I need to change the hyperparameters a bit. So I tried training the second model on the following hyperparamters:

Number of iterations: 600k
Batch Size: 10
Learning Rate: warms up to 0.001 and decreases to 0.0002 after 400k iterations

When I measure the mAP for both models on a testing set, the first model vastly outperforms the second model.

model 1 mAP: 0.924

model 2 mAP: 0.776

This leads me to my question for you:

How would the hyperparameters (batch size, learning rate etc) change when the size of your training set increases? What factors need to be considered for this increase in training set size, in order to get the most optimal model possible?

Any and all responses will be greatly helpful. Thank you :)

A major difference between the first and the second model you trained is the size of the data assuming that the model is not pretrained. Increased data, of course, needs increased epochs. According, the batch size must also increase.

Batch Size:

While training on the smaller dataset, a batch size of 10 yielded better results. The errors were averaged over 10 samples and then back-propagated through the model. But now for the larger dataset, the batch size remains the same and hence only little optimization occurs as the error is averaged over 10 samples only ( which is relatively smaller for a large dataset ).

Learning Rate:

For the larger dataset, the number of epochs is increased. The purpose of the learning rate is to scale the gradients of the loss with respect to the parameter. A smaller learning rate always helps as it prevents the overshooting of the minima of the loss function. I would insist you increase the learning rate so that the optimization does not diminish as we are having a larger number of epochs. Gradually decrease the learning rate, as the loss approaches its minima.

  If you are training a popular architecture ( like Inception, VGG, etc. ) and that too on datasets like ImageNet, COCO with little modifications, you should definitely read the research papers published on various problems as they would provide a better start to the training.


Formal math notation of masked vector I'm struggling to write my algorithm in a concise and correct way.
The following is an explanation for an optimizer's update step of part of a vector of weights (not a matrix in my case).

I have a vector $\alpha \in \mathbb{R}^d$, and a set $S$ that includes some indices $1\leq i \leq d$ ($S \subseteq \{1,\dots, d\}$).
Now, I want to denote that $\alpha$ is 0 for every index $i\in S$, and otherwise it's the value as in $\alpha_i$.
At first I denoted it $\alpha_S$, but I'm not sure it is properly defined or understandable.

I could use the following notation:

$\alpha_S =
  \begin{cases}
    \alpha_j        & j \in S\\
    0               & j \notin S
  \end{cases}$

But its line height is twice the size, and I want to avoid that.

Is there any other formal, simplistic way to notate this correctly? Maybe some kind of a masking vector to be multiplied with $\alpha$?

Thanks!

Check out the indicator function $1_S(\cdot)$. In your case it would be fined as

$$1_S: \{1, \ldots, d\} \rightarrow \{0, 1\}, j \mapsto \begin{cases}
1 & j \in S \\
0 & \, \text{else}
\end{cases}.$$

Multiplying this function with the respective values should give you what you are looking for.

Edit: If I understand your comment correctly, the vector you are looking for is

$$ \alpha_s = \sum\limits_{i = 1}^{d} \alpha_i e_i 1_{S^C}(i). $$

Here $e_i$ denotes the i-th unit vector of $\mathbb{R}^d$, $S^C$ is the complement $\{1, \ldots, d\} \setminus S$ in $\{1, \ldots, d\}$ and $1_{S^C}$ is the indicator function that is $1$ for $i \notin S$ and $0$ for $i \in S$.


Dealing with pre-trained model for grayscale images I would like to do Transfer Learning using one of the novel networks such as VGG, ResNet, Inception, etc.
The problem is that my images are grayscale (1 channel) since all the above mentioned models were trained on ImageNet dataset (which consists of RGB images).
One of the solutions is to repeat the image array 3 times to make it 3 channel.
Is this really the only solution for that? Is it a good solution? Are there any other solutions?

print(grayscale_batch.shape)  # (64, 224, 224)

rgb_batch = np.repeat(grayscale_batch[..., np.newaxis], 3, -1)
print(rgb_batch.shape)  # (64, 224, 224, 3)


Making predictions with missing numeric data I don't know much about how to deal with missing data.  When the data is categorical it doesn't seem too bad, if I one-hot encode it without dropping any of the actual categories, the missing data is accounted for in the model because all the associated one-hot columns are zero. But for numeric data, I don't like the options as I understand them: omit the entire observation, or impute a value. Are there any other choices?

I'd lean toward omitting the data (in the data I am working on it would lead to ~2% data loss), but this doesn't help if you are trying to make predictions when data is missing, and I I can't find much useful discussion addressing this aspect of the problem (mostly it is around training the model).

I guess you could train multiple models that deal with all the permutations of missing features, and select the model matching the available predictive data?  But that seems likely to be too unwieldy and time consuming to be practical. (The data set I'm working on now is artificial; it has 100 features, 96 of which are numeric, and all of which [both training and prediction sets] have from 2-20 missing values.)

Fwiw, my plan if I have to impute data is to regress using the 98% of complete training/prediction data, predict mean for each missing value, then randomize it based on the variance/CI.

I guess this boils down two questions/requests:

1) As in bold above re omission/imputation: are there any other choices?

2) General advice on how to proceed, given the predictions must be based on missing data?

-- Edit/Add 15-Mar-20 --
I would particularly like to know what other options are, and what is considered best practice, when implementing models such as regression that do not cope with missing values?

2 Questions.

Yes there are certain algos that deal with it internally. For example, lightGBM will ignore missing values during a split, then allocate them to whichever side reduces the loss the most. Check section 3.2 here Or Catboost for example also

Try to save/impute information as much as possible, if there is just too much missing values (30%+) than just drop the column. You will do more harm injecting the noise while imputing. (NOTE 30% is approximation based on experience, some datasets require less or more)