Recommending college degrees based on high school subjects I have a list of our college students' high school courses.

I want to recommend college degrees to current high school students based on their courses - that is, predict a class based on a vector. For instance someone who studied algebra, calculus and statistics could be recommended software engineering, accountancy or mathematics.

There seem to be a few approaches I could take: market basket analysis, collaborative filtering, clustering or even neural networks.

I can structure my data in a sparse matrix of courses, with each row having a class representing the student's eventual degree, e.g.:

DEGREE       English    Calc    Algebra    Geography    History ... etc
Soft.Eng.    0          1       1          0            0
Comms        1          0       0          1            1
Mech.Eng.    1          1       1          0            0

How should I approach this?

This problem is perfectly suited for a neural network. Your model will have 40 input nodes (this is fine), then you will have some arbitrary hidden layers, you need to tune this, and 20 outputs. After the training process you can even get a probability for each of them. This can be used to rank suggestions for potential students!

How to do this

Load the data to memory

Depending on the means by which your data is stored this step will be different. However, the goal is to go from the raw file source to either Python Numpy array or a Pandas DataFrame. I will assume your data is structured as follows

and is saved as a .csv file.

Let's load our data into an X and Y matrix. We will be encoding the labels for 'degree' as values. Make sure these are all well spelled otherwise a new label will be created for the mispelled ones.

import pandas as pd
import numpy as np

df = pd.read_csv('test1.csv')
df['Degree'] = df['Degree'].astype('category')
df["Degree_encoding"] = df["Degree"].cat.codes

X = np.asarray(df.loc[:, df.columns != 'Degree'])
Y = df['Degree_encoding']

print(X.shape)
print(Y.shape)

  (39, 7) (39,)

Applying a neural network to this dataset

First we will split the data randomly into a training and testing set. This is used to evaluate our model while maintaining that we are not overfitting. Then we identify the number of output classes. Then we reshape the input matrices such that they have a channel in their last dimension, this is how data flows through the model. Then we will categorize our outputs as one-hot encoded vectors.

from sklearn.model_selection import train_test_split
import keras

# Split the data
x_train, x_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, shuffle= True)

# The known number of output classes.
num_classes = len(set(df["Degree_encoding"]))
# Input dimensions
shape = X.shape[1::]

# We need to add a channels dimension to our data
# Channels go last for TensorFlow backend in Keras
x_train_reshaped = x_train.reshape((x_train.shape[0],) + shape)
x_test_reshaped = x_test.reshape((x_test.shape[0],) + shape)
input_shape = shape
print(input_shape)

# Convert class vectors to binary class matrices. This uses 1 hot encoding.
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

We then design our model

model = Sequential()
model.add(Dense(32,
                 activation='relu',
                 input_shape=input_shape))
model.add(Dense(128, activation='relu'))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

You can use model.summary() to get a description of these layers. Then we are ready to train our model!

epochs = 100
batch_size = 128
model.fit(x_train_reshaped, y_train,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test_reshaped, y_test))


Function growing faster for negative inputs than for positives I am working on a regression problem where I want to model the loss function in a way that it "punishes" to big errors much more than small errors (so I am in the realm of exponential functions) but also in a way that is punishes a negative error much more than a positive error.
So for example:

Prediction off by +4.0: is a problem, but still ok
Prediction off by +0.5: not a big deal
Prediction off by -0.5: is a problem, but still ok
Prediction off by -4.0: is a major problem

My problem is that I cant find a good function to describe this. x squared and so do not have the higher values for negative inputs that I am looking for.
My best workaround for now is to just move the whole function to the right (x-2)^2, but there must be something better?

This should be possible using a piecewise exponential loss, something like this:
$
f(x) =
\begin{cases}
x^2 & x < 0\\
\lambda x^2 & x \ge 0
\end{cases}
$
with $0 < \lambda < 1$. A $\lambda$ of around 0.02 should roughly give you the scale you want.


Best practice to test (unit and integration) a neural network implementation I'm implementing my own neural network (with the term implementing i mean writing the code that run the neural network itself, not training).

I implemented it for didactical purpose, but i does not know how to proceede for a validation of what i have done.

So i would ask if there is some best practice for neural network test (eg : is there a dataset of neural nets with weight, input and expected output)

Make your code testable. You should be able to write plenty of tests for individual components without having to mock up an entire training pipeline for each one. A "property-based testing" library like Hypothesis can be helpful for this.
Train a standard model on a standard dataset and make sure the accuracy numbers are similar.
See here for an in-depth discussion, with more and better advice than I've been able to provide.


What is the purpose of untrainable weights in Keras In this page it is mentioned that when trainable=false, the weight won't be updated and is used for optimization, too. But I still do not understand how it can be useful? (e.g., I want to find out the best number of neurons or the best drop out rate, can it help?)

One common application is to freeze an embedding layer. Freezing this layer will prevent the embedding from updating its weight which can be a desirable thing, especially for a text embedding layer.

There also exist designs where updating a weight during a certain batch is not wanted.  For example some GAN implementations only want to train a model during the combined model phase and therefore freezes the Generator and Discriminator layers for batches.

You can also see this used for stacked autoencoder where someone may train one layer at a time.  Here is a quick link.


Openai Spaces for a modified environment I have a 2-dimensional array of normalized data. I am using

space = np.array([0,1,...366],[0,0.000001,.....1])

I need to fit this as an observation space in reinforcement learning. I have extended the open ai gym and created a custom made environment. How to fit in this 2-dimensional array in openAI spaces.
Can I use Box, DiscreteSpace or MultiDiscrete space? Can anyone help me with a sample code to fit this in observation space?

In your case it seems you simply can return a 2D vector and extract out the components for that. You can take a look at the MountainCar example under classic control envs for a fully working case. But something like:

vec_1_low = 0
vec_1_high = 366
vec_2_low = 0
vec_2_high = 1

space_low = np.array([vec_1_low, vec_2_low])

space_high = np.array([vec_1_high, vec_2_high])

obs = spaces.Box(space_low, space_high, dtype=np.float32)

Then:

vec1_obs, vec2_obs = obs.sample()


Why results of statsmodel logreg is different from scikit-learn logreg? I am trying to do a binary classification. I have only 6 input variables and one output variables. Label 1 is 1554 records and Label 0 is 3558 records.

As you can see below, the metrics that I get from these two are different. I am not sure what other info I can share more about this issue. I just tried to do the classification through different methods

# code for statsmodel logreg

model = smm.Logit(y_train, X_train_std)    #std indicates standardized inputs
result=model.fit()
result.summary()
y_pred = result.predict(X_test_std)
y_pred[y_pred > 0.5] = 1
y_pred[y_pred < 0.5] = 0
cm = confusion_matrix(y_test, y_pred)
print(cm)
print("Accuracy is ", accuracy_score(y_test, y_pred))
print(classification_report(y_test, y_pred))
print("ACU score is ",roc_auc_score(y_test, y_pred))
print("Recall score is",recall_score(y_test,y_pred))
print("Precision score is",precision_score(y_test,y_pred))
print("F1 score is",f1_score(y_test,y_pred))

# code for scikit-learn

#log reg optimized parameters
op_param_grid = {'C': [0.01],'class_weight':['balanced'],'penalty': ['l1'], 'solver': ['saga'],'max_iter':[200]}
logreg=LogisticRegression(random_state=41)
logreg_cv=GridSearchCV(logreg,op_param_grid,cv=10,scoring='f1')
logreg_cv.fit(X_train_std,y_train)

May I know why is this happening and which one should I rely upon?

Is it not right to expect the same results on both approach? Do they work differently?

Can anyone help?

As far as I can tell, you use a "normal" Logit in one approach and a Logit with L1 penalty in the other case (penalty': ['l1']), which is called "Lasso". In Statsmodels L1 penalty would be implemented like stated in the docs.

Lasso and "normal" Logit are two different approaches. In the first case, (some) parameters are shrunken and can be set to zero, while with normal Logit, no parameter is shrunken. Lasso is often used to improve fit with noisy $x$, to do feature selection, or in case of high dimensional data (lot of $x$). Compare Chapters 4.3 and 6.2 in Introduction to Statistical Learning.

ISLR comes with Python labs. This can give you a good starting point how to apply one or the other method in a proper way.


Determine relationship between users and age? I would like to understand how to find an association between users, spam and email's age.
My dataset looks like as follows:
User      Spam      Age (yr)
porn_23    1         1
Mary_g     0         6
cricket_s54 0        4
rewuoiou   1         0
pure75     1         2
giogio35   0         10
viv3roe    1         1

I am looking at the correlation using Pearson. Is it right? I would like to determine the correlation between age and user: spam email should likely come from users having recent email's addresses (fake account / email).

If you are using pandas, all you need to do is:
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

corrMatrix = df.corr()

Then you can print the correlation matrix and also plot it using seaborn or any other plotting method.
sns.heatmap(corrMatrix, annot=True)
plt.show()

Hope this helps.


How to cope with new/unseen targets classes in incremental learning algorithms According to scikit-learn documentation1, the sklearn incremental learner itself may be unable to cope with new/unseen targets classes. Is there any available python machine learning library which support this functionality?

Scikit-multiflow is a package designed for incremental learning for multi-output/multi-label problems.


SAS PROC means (two variants together) PROC means data=d mean;
var a;
class b; var a;
run;

I want to perform the "PROC means" for continuous "var a":
1) in general and
2) by classes.
But it performed by the classes only.
How to make procedure for "var a" here in general too?

P.S. SAS WARNING: Analysis variable "a" was defined in a previous statement, duplicate definition will be ignored.

Just run

PROC means data=d mean; var a; run;

for overall data


How can I infer psychological intentions from email corpus using text mining? I'm working with Enron email data set, and try to get some psychological intentions about email marketing. I know that we can derive Sentimental analysis like emotions and mode and opinion mining like positive or negative through text data. what else can I do regarding human psychological intentions?

DISCLAIMER : I'm not sure that this will be the answer you are looking for but I'll give a try.

According to Noam Chomsky

  Language is a process of free creation; its laws and principles are
  fixed, but the manner in which the principles of generation are used
  is free and infinitely varied. Even the interpretation and use of
  words involves a process of free creation.

On one hand, data or text mining the combined large lexical data resources of linguistics and psychology suggests that the linguistic variable of 'word frequency' should be rehabilitated in psychology.

But you have to remember that, correlation does not imply causation.

This said, there is not much way to apply cross text mining with psychological intuition. In other words, use cases are not too many.

You can always try quantitative text analysis. It is a set of techniques stemming from the social sciences where either a human judge or a computer extracts semantic or grammatical relationships between words in order to find out the meaning or stylistic patterns of, usually, a casual personal text for the purpose of psychological profiling etc.

And by psychological profiling, I'm talking about offender profiling per example.

You can get more information upon that on the following websites :

Criminal Investigative Research & Analysis
http://www.psychlotron.org.uk/ article  about criminal profiling.
Here is a very intersting article about criminal profiling for beginners.

You can also try to take a look at this presentation by Harold Baize who talks about the intersection of text mining in the content analysis of psychology.

I hope this can help!


Repeated features in Neural Networks with tabular data When using algorithms like linear regression or least-squares methods, having repeated or highly correlated features can be harmful for the model. For tree based models, they are generally not too strongly affected by highly correlated features. There are no numeric stability issues as with least squares.
But what happens with Neural Networks? Most of the literature on NN is made for images, signal and there is not too much about tabular data.
Having repeated features in a tabular data Neural Network model, does it harm the accuracy? Or NN are able to select features?

Strictly theoretically it makes no difference on accuracy.
Here is why: We already know mathematically that NN can approximate any function. So lets say that we have Input X. X is highly correlated, than we can apply a decorrelation technique out there. Main Thing is, you get X` that has different numerical representation. Most likely more difficult for NN to learn to map to Outputs y. But still in Theory you can Change the architecure, Train for longer and you can still get the same Approximation, i.e. Accuracy.
Now, Theory and Praxis are same in Theory but different in Praxis, and I suspect that this Adjustments of Architecture etc will be much more costly in reality depending on the dataset.


Mapping column values of one DataFrame to another DataFrame using a key with different header names I have two data frames df1 and df2 which look something like this.

    cat1    cat2  cat3
0   10       25     12
1   11       22     14
2   12       30     15

   all_cats  cat_codes
0   10       A
1   11       B
2   12       C
3   25       D
4   22       E
5   30       F
6   14       G

I would like a DataFrame where each column in df1 is created but replaced with cat_codes. Column header names are different. I have tried join and merge but my number of rows are inconsistent. I am dealing with huge number of samples (100,000). My output should ideally be this:

    cat1    cat2  cat3
0    A        D     C
1    B        E     Y
2    C        F     Z

The resulting columns should be appended to df1.

You can convert df2 to a dictionary and use that to replace the values in df1

cat_1 = [10, 11, 12]
cat_2 = [25, 22, 30]
cat_3 = [12, 14, 15]

df1 = pd.DataFrame({'cat1':cat_1, 'cat2':cat_2, 'cat3':cat_3})

all_cats = [10, 11, 12, 25, 22, 30, 15]
cat_codes = ['A', 'B', 'C', 'D', 'E', 'F', 'G']

df2 = pd.DataFrame({'all_cats':all_cats, 'cat_codes':cat_codes})

rename_dict = df2.set_index('all_cats').to_dict()['cat_codes']

df1 = df1.replace(rename_dict)

If you still have some values that aren't in your dictionary and want to replace them with Z, you can use a regex to replace them.

df1.astype('str').replace({'\d+': 'Z'}, regex=True)


IndexError: index 804 is out of bounds for axis 0 with size 800 i installed a self driving car project from superdatascience site , when i open the map using terminal after a while the map window close up or it closes directly after i maximize the map window and it gives me this error :

    [INFO   ] [Base        ] Leaving application in progress...
 Traceback (most recent call last):
   File "map.py", line 235, in <module>
     CarApp().run()
   File "/usr/lib/python2.7/dist-packages/kivy/app.py", line 826, in run
     runTouchApp()
   File "/usr/lib/python2.7/dist-packages/kivy/base.py", line 502, in runTouchApp
     EventLoop.window.mainloop()
   File "/usr/lib/python2.7/dist-packages/kivy/core/window/window_sdl2.py", line 727, in mainloop
     self._mainloop()
   File "/usr/lib/python2.7/dist-packages/kivy/core/window/window_sdl2.py", line 460, in _mainloop
     EventLoop.idle()
   File "/usr/lib/python2.7/dist-packages/kivy/base.py", line 340, in idle
     self.dispatch_input()
   File "/usr/lib/python2.7/dist-packages/kivy/base.py", line 325, in dispatch_input
     post_dispatch_input(*pop(0))
   File "/usr/lib/python2.7/dist-packages/kivy/base.py", line 231, in post_dispatch_input
     listener.dispatch('on_motion', etype, me)
   File "kivy/_event.pyx", line 707, in kivy._event.EventDispatcher.dispatch
   File "/usr/lib/python2.7/dist-packages/kivy/core/window/__init__.py", line 1360, in on_motion
     self.dispatch('on_touch_down', me)
   File "kivy/_event.pyx", line 707, in kivy._event.EventDispatcher.dispatch
   File "/usr/lib/python2.7/dist-packages/kivy/core/window/__init__.py", line 1376, in on_touch_down
     if w.dispatch('on_touch_down', touch):
   File "kivy/_event.pyx", line 707, in kivy._event.EventDispatcher.dispatch
   File "/usr/lib/python2.7/dist-packages/kivy/uix/widget.py", line 460, in on_touch_down
     if child.dispatch('on_touch_down', touch):
   File "kivy/_event.pyx", line 707, in kivy._event.EventDispatcher.dispatch
   File "map.py", line 181, in on_touch_down
     sand[int(touch.x),int(touch.y)] = 1
 IndexError: index 620 is out of bounds for axis 1 with size 600

the code i have in the brain ( ai.py file ) is

    class Dqn():

    def select_action(self, state):
            probs = F.softmax(self.model(Variable(state, volatile = True))*100) # T=100
        action = probs.multinomial(num_samples=1)
            return action.data[0,0]

the code i have in the map ( map.py file ) is

class MyPaintWidget(Widget):

    def on_touch_down(self, touch):
        global length, n_points, last_x, last_y
        with self.canvas:
            Color(0.8,0.7,0)
            d = 10.
            touch.ud['line'] = Line(points = (touch.x, touch.y), width = 10)
            last_x = int(touch.x)
            last_y = int(touch.y)
            n_points = 0
            length = 0
            sand[int(touch.x),int(touch.y)] = 1

what should i do or change ??

One of the values being returned by touch.x or touch.y are bigger than the dimensions of the sand array.

You could try adding a try/except block around it and print a useful message - just as an example, as I have no idea what this programme is doing (in the on_touch_down function):

def on_touch_down(self, touch)
...
...
    try:
        sand[int(touch.x), int(touch.y)] = 1
    except IndexError:
        print("Touchdown did not occur within the end-zone! No goal!")


Which tribe does Probabilistic Graphical Models fall under? Pedro Domingos in  "The Master Algorithm" listed five tribes of machine learning algorithms:

Symbolists
Connectionists
Evolutionaries
Bayesians
Analogizers

Which category do probabilistic graphical models fall under?

From wikipedia (https://en.wikipedia.org/wiki/Graphical_model):

  A graphical model or probabilistic graphical model (PGM) is a
  probabilistic model for which a graph expresses the conditional
  dependence structure between random variables. They are commonly used
  in probability theory, statistics—particularly Bayesian statistics—and
  machine learning.

In that case, would it be "Bayesian"? What about something like Conditional Random Field? Is that Bayesian as well?

Probabilistic Graphical Models (PGMs) are:

Connectionist: RBMs are PGMs and neural networks (source)
Bayesian: Bayes Networks are bayesian (Wikipedia article)
Symbolist: Markov Logic Networks (source)
Analogizers and Evolutionaries: According to Domingos, they are also in Markov Logic Networks.

So the answer is that you can't simply categorize such a general technique as probabilistic graphical models in a single one of those categories.
See also: "The Five Tribes of Machine Learning (And What You Can Learn from Each)," Pedro Domingos


Adding more features in SVC leading to worse performance, even w/ regularization I have a relatively small dataset of 30 samples with binary labels (16 positive and 14 negative).  I also have five continuous features for each these samples.  I'm trying to use the support-vector classifier (SVC) for this task.  I tested the performance of different feature combinations and regularization strengths in the classification task using leave-one-out cross-validation.

One odd thing that I found is that if I took feature A and used it alone for classification, I might get, say 87% classification accuracy.  If I use feature B in isolation, I might get 60% classification accuracy (i.e., same as majority classifier baseline).  But then combining all the features, I would get only 63% classification accuracy.  This is despite performing a search across a large range of regularization strengths.

In case it matters, I'm using the sklearn SVC implementation, and varying the regularization parameter C.

Is this sort of behavior typical with an SVC classifier?  I'm not too familiar with this support vector algorithms in general.

30 samples probably just isn't enough. The more features you want to use, the more samples you'll need - otherwise you'll get huge model instability and overfitting (especially with bad/useless features). With only 30 samples you'll probably get the "best" results with 1 or 2 carefully selected features. Get 100 or 200 samples, then try again with 5 features.

Also make sure you're standardizing your features - for example by removing the mean and scaling to unit variance. SVMs don't like features that are a lot larger than other features.


Binary Cross Entropy | Manual scalars I am wanting to make print statements "showing my working out" of Binary Cross Entropy loss function, that works with scalar inputs, not arrays.
import math as m
import numpy as np

y_ans = 1
y_pred = 1

print("Cost(y,y^) = -yln(y^) - (1-y)-y^ln(1-y^)")
print('Cost('+str(y_ans)+', '+str(y_pred)+') = -'+str(y_ans)+'ln('+str(y_pred)+') -('+str(1-y_ans)+')ln('+str(1-y_pred)+')')
#ans = round((-y_ans*m.log(y_pred)) - ((1-y_ans)*m.log(1-y_pred)), 2)
ans = round((-y_ans*np.log(y_pred)) - ((1-y_ans)*np.log(1-y_pred)), 2)
print('Cost('+str(y_ans)+', '+str(y_pred)+') = '+str(ans))

The answer should be 0, but I get NaN
Cost(y,y^) = -yln(y^) - (1-y)-y^ln(1-y^)
Cost(1, 1) = -1ln(1) -(0)ln(0)
/home/runner/MalwareAnalysis/BinaryCrossEntropy.py:10: RuntimeWarning: divide by zero encountered in log
  ans = round((-y_ans*np.log(y_pred)) - ((1-y_ans)*np.log(1-y_pred)), 2)
/home/runner/MalwareAnalysis/BinaryCrossEntropy.py:10: RuntimeWarning: invalid value encountered in double_scalars
  ans = round((-y_ans*np.log(y_pred)) - ((1-y_ans)*np.log(1-y_pred)), 2)
Cost(1, 1) = nan

print((-y_ans*np.log(y_pred)))
-0.0
print((1-y_ans)*np.log(1-y_pred))
nan

I'm fairly new to Python.

np.nan_to_num() does the trick, as @Oxbowerce pointed out.
import numpy as np
np.nan_to_num(np.log(1-y_pred))


how to design sigma(std) vector to have values above > 0 in VAE autoencoder I am trying to understand how to design mu and sigma vector in VAE autoencoder, as we know std value can not be larger than 0. If I use linear function, the value can be lower than zero, so I think Relu is the only activation function to apply in order to get sigma (Std) vector larger than zero. Would this be correct understanding?

Yeah, almost, except that Relu might cause problems ones it goes to zero (having zero variance is a little odd and the gradient goes to zero as well). Usually you'd go with

$$ \sigma = \exp(\rho) $$

where $\rho$ is coming from the encoder.


ValueError: Numpy arrays that you are passing to your model is not the size the model expected I am trying to perform concatenation on the Bidirectinal LSTM layer. I have my model defined like this:

embedding_layer=Embedding(words,300,input_length=trainDataVecs.shape[1],weights=[embedding_matrix])

LSTM_word_1 = Bidirectional(LSTM(80, activation='tanh', dropout_W = 0.25, dropout_U = 0.25,return_sequences=True))
model1 = Input(shape=(trainDataVecs.shape[1],),dtype='int32')
embedded_sequences1=embedding_layer(model1)
lstm_word_out_1 = LSTM_word_1(embedded_sequences1)

LSTM_word_2 = Bidirectional(LSTM(80, activation='tanh', dropout_W = 0.25, dropout_U = 0.25,return_sequences=True))
model2 = Input(shape=(trainDataVecs.shape[1],),dtype='int32')
embedded_sequences2=embedding_layer(model2)
lstm_word_out_2 = LSTM_word_2(embedded_sequences2)

conc=concatenate([MaxPooling1D()(lstm_word_out_1),AveragePooling1D()(lstm_word_out_2)])

outp = Dense(classes_num, activation='sigmoid')(conc)
model=Model(input=[model1,model2],output=outp)

I am fitting like this:

history=model.fit(trainDataVecs,Y_train,shuffle=True,batch_size=512,epochs=10,validation_data=(valDataVecs,valid_Y))

But I am getting the error like:
ValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 2 array(s), but instead got the following list of 1 arrays
PS: The training and validation vectors are arrays. But still I am getting the error. I think it might be because of the model definition but I am not sure.

Your network has two Inputs, so you need to pass in two arrays.  From the rest of the code, perhaps you want to pass the same data through both strands, in which case you can just duplicate the input:

history=model.fit([trainDataVecs, trainDataVecs], ...


Purpose of converting continuous data to categorical data I was reading through a notebook tutorial working with the Titanic dataset, linked here, and noticed that they highly favored ordinal data to continuous data.

For example, they converted both the Age and Fare features into ordinal data bins.

I understand that categorizing data like this is helpful when doing data analytics manually, as fewer categories makes data easier to understand from a human perspective. But intuitively, I would think that doing this would cause our data to lose precision, thus leading to our model losing precision or accuracy.

Can someone explain when is converting numerical data to ordinal data is appropriate, and the underlying statistics of why it is effective?

Your intuition is generally correct - in many cases, premature discretization of continuous variables is undesirable. Doing so throws away potentially meaningful data, and the result can be highly dependent on exactly how you bucket the continuous variables, which is usually done rather arbitrarily. Bucketing people by age decade, for example, implies that there is more similarity between a 50-year-old and a 59-year-old than there is between a 59-year-old and a 60-year-old. There can be some advantages in statistical power to doing this, but if your binning doesn't reflect natural cutpoints in the data, you may just be throwing away valuable information.

You can find a very similar question here:

https://stats.stackexchange.com/questions/68834/what-is-the-benefit-of-breaking-up-a-continuous-predictor-variable?noredirect=1&lq=1


Dataset from sequence of messages I have a sorted dataset by datestamp which looks like this:

user    message
A       Hi.
B       Hello.
B       How are you?
A       I am stuck.
B       How can I help you?

What I want is to create a pandas df that would look like this:

user  message       reply
A     Hi.           Hello.
A     Hi.           How are you?
B     Hello.        I am stuck.
B     How are you?  I am stuck.
A     I am stuck.   How can I help you?

For each message, I want to find all of the replies. That means that I want the messages after current one but from the other user. How can I do this with pandas? Let's only consider a binary case of 2 users A and B.

First, find out when the user switch and give a separate id to each message group:

df['group_id'] = ((df['user'] != df['user'].shift()).cumsum())

user              message  group_id
   A                  Hi.         1
   B               Hello.         2
   B         How are you?         2
   A          I am stuck.         3
   B  How can I help you?         4

Then groupby each group_id and aggregate a list of the messages for each id. By shifting these messages by -1 we recive the replies for each group_id:

df_reply = df.groupby('group_id')['message'].agg(list)
df_reply = df_reply.shift(-1).reset_index().rename(columns={'message': 'reply'})

 group_id                   reply
        1  [Hello., How are you?]
        2           [I am stuck.]
        3   [How can I help you?]
        4                     NaN

The replies can then by merged back into the original dataframe. The reply lists are exploded to ensure a single reply per row:

df.merge(df_reply, on='group').explode('reply').drop('group', axis=1).dropna()

The final result:

user       message                reply
   A           Hi.               Hello.
   A           Hi.         How are you?
   B        Hello.          I am stuck.
   B  How are you?          I am stuck.
   A   I am stuck.  How can I help you?


Meaning of Instance Space and Concept Class, (PAC Learnable) I'm studying Probably approximately correct learning, and I don't understand what an Instance Space and a Concept is.

I have see that wikipedia https://en.wikipedia.org/wiki/Probably_approximately_correct_learning  provides various examples, but it's still rather an abstract concept.

Could you provide me with an intuitive definition and some tangible examples?

This kind of language is typically associated within a field of math called computational learning theory (clt).  CLT is inherently abstract since it's trying to derive general observation about unknown target functions.  That's why you get generic terms like instance space and concepts.  Broadly speaking an instance space is typically just the domain of X (your sample population) and an instance of that space is a particular x with various features.  For example, X may be the set of all people and an instance of x will be an individual person who is 30 female and medium height etc.

The concept space on the other hand is subset of the domain X such that X -> {0, 1} or in other words a subset that maps X via Boolean function as either 0 or 1.  For example, Our concept space can be people who walk to work.  Every person X can either say yes they walk to work (and thus x -> 1) or no (x -> 0).

If this is still too abstract for you, I would recommend stepping back and building a more foundation  footing on basic machine learning before you start trying to study them as broad generic objects.


Cross validation and evaluation: neural network loss function continuously decreases in cross-validation I am evaluating a neural network model using cross validation in 2 different ways ( A & B ) that I thought were equivalent.

Evaluation type A :
For each cross validation loop, the model is instantiated and fitted.
Evaluation type B :
I instantiate the model once and then that instantiated model is fitted for each loop of the cross validation procedure.

I am using the metric mean absolute error (MAE).
Question: Why do I get a continuously decreasing MAE over cross-validation loops when using type B evaluation and not when using type A evaluation?
Code and details
First I generate synthetic data :
from sklearn.datasets import make_regression

X , y = make_regression( n_samples = 1000 , n_features = 10 , n_informative = 5 , n_targets = 1 , random_state = 2 )

I then define a function to get a model ( neural network ) :
from keras.models import Sequential
from keras.layers import Dense

def get_model( n_nodes_hidden_layer , n_inputs , n_outputs ) :
   model = Sequential()
   model.add( Dense( n_nodes_hidden_layer , input_dim = n_inputs , kernel_initializer = 'he_uniform' , activation = 'relu' ) )
   model.add( Dense( n_outputs ) )
   model.compile( loss = 'mae' , optimizer = 'adam' )
   return model

After that I define 2 evaluation functions using :
from sklearn.model_selection import RepeatedKFold
from sklearn.metrics import mean_absolute_error

Type A evaluation function :
def evaluate_model_A( X , y ) :
   results = list()
   cv = RepeatedKFold( n_splits = 10 , n_repeats = 1 , random_state = 999 )

   for train_ix, test_ix in cv.split( X ) :
     X_train, X_test = X[ train_ix ] , X[ test_ix ]
     y_train, y_test = y[ train_ix ] , y[ test_ix ]

     model = get_model( 20 , 10 , 1 )
     model.fit( X_train , y_train , epochs = 100 , verbose = 0 )

     y_test_pred = model.predict( X_test )
     mae = mean_absolute_error( y_test , y_test_pred )
     results.append( mae )

     print( f'mae : {mae}' )

   return results

Type B evaluation function :
 def evaluate_model_B( model , X , y ) :
   results = list()
   cv = RepeatedKFold( n_splits = 10 , n_repeats = 1 , random_state = 999 )

   for train_ix, test_ix in cv.split( X ) :
     X_train, X_test = X[ train_ix ] , X[ test_ix ]
     y_train, y_test = y[ train_ix ] , y[ test_ix ]

     model.fit( X_train , y_train , epochs = 100 , verbose = 0 )

     y_test_pred = model.predict( X_test )
     mae = mean_absolute_error( y_test , y_test_pred )
     results.append( mae )

     print( f'mae : {mae}' )

   return results

Before using type B evaluation function I need to instantiate the model because it is an argument of the function :
model = get_model( 20 , 10 , 1 )

What I do not understand is the fact that while using type B evaluation function the MAE is decreasing for each cross validation loop which is not the case with type A evaluation function.
Is this specific to neural networks?
Note : when I am using a RandomForestRegressor() the phenomenon does not show up.

In the evaluation type B approach, your neural network weights and biais are not reset before each loop of cross-validation. The neural network is then learning from one loop to another, so you see the MAE continuously decreasing.
A solution is to store your weights and biais before fitting the model and load them at each loop so they have the same init.
You can use those methods to do so:
model.save_weights('model.h5') # right after model instantiation
model.load_weights('model.h5') # in the loop before fitting

In evaluation type A, because you instantiate the model in the loop, weights and biais are reset so you don't see the phenomenon.


Neural Network for regression with one dependent and one independent variable I am trying to make a simple neural network with one dependent and one independent variable. Could you maybe give me a tutorial or help me with the implementation of a neural network with one dependent and one independent variable. So far I have the following code, however my predictions are not good although the error is minimized. Should I scale X and Y or do I have some mistake?

Thank you in advance

import tensorflow as tf
import matplotlib.pyplot as plt
import numpy as np

x=[(i*i)+0.2 for i in range(1000)]
y=[i for i in range(1000)]

x_train=np.reshape(x,(-1,1))
y_train=np.reshape(y,(-1,1))
x_test=x_train[:,-10:]
y_test=y_train[:,-10:]
plt.scatter(x_train,y_train)
plt.show()

X=tf.placeholder(tf.float32,[None,1])
Y=tf.placeholder(tf.float32,[None,1])

n_inputs=1
n_hidden_1=20
n_hidden_2=20
n_outputs=1

weights={
    "h1": tf.Variable(tf.random_normal([n_inputs,n_hidden_1])),
    "h2": tf.Variable(tf.random_normal([n_hidden_1,n_hidden_2])),
    "out": tf.Variable(tf.random_normal([n_hidden_2,n_outputs]))
}

biases={
    "b1": tf.Variable(tf.random_normal([n_hidden_1])),
    "b2": tf.Variable(tf.random_normal([n_hidden_2])),
    "out": tf.Variable(tf.random_normal([n_outputs]))
}

def neural_net(x):
    layer_1=tf.add(tf.matmul(x,weights["h1"]),biases["b1"])
    layer_1=tf.nn.relu(layer_1)
    layer_2=tf.add(tf.matmul(layer_1,weights["h2"]),biases["b2"])
    layer_2=tf.nn.relu(layer_2)
    layer_3=tf.matmul(layer_2,weights["out"])+biases["out"]
    return layer_3

Y_pred=neural_net(X)

loss=tf.losses.mean_squared_error(X,Y_pred)
optimizer=tf.train.AdamOptimizer(learning_rate=0.01)
train_op=optimizer.minimize(loss)

epochs=1000
init=tf.global_variables_initializer()
with tf.Session() as sess:
    sess.run(init)
    for i in range(epochs):
        sess.run(train_op,feed_dict={X:x_train,Y:y_train})
        loss_op=sess.run(loss,feed_dict={X:x_train,Y:y_train})
        if i%10==0:
            print("Epoch "+str(i)+" loss "+str(loss_op))
    pred=sess.run(Y_pred,feed_dict={X:x_test})
    plt.plot(pred,color="red")
    plt.plot(y_test,color="blue")
    plt.show()
    plt.scatter(pred,y_test)
    plt.show()
    for i in range(len(pred)):
        print(str(pred[i])+" "+str(y_test[i]))

Your predictions are not actually that bad. At the very last line of your code, print the expected value too at each line (that is x_test[i]= y_test[i]^2+0.2).


TensorFlow1.15, multi-GPU-1-machine, how to set batch_size? The input function code:

    def input_fn(params):
        """The actual input function."""
        batch_size = FLAGS.train_batch_size

        name_to_features = {
            "input_ids":
                tf.FixedLenFeature([max_seq_length], tf.int64),
            "input_mask":
                tf.FixedLenFeature([max_seq_length], tf.int64),
            "segment_ids":
                tf.FixedLenFeature([max_seq_length], tf.int64),
            "masked_lm_positions":
                tf.FixedLenFeature([max_predictions_per_seq], tf.int64),
            "masked_lm_ids":
                tf.FixedLenFeature([max_predictions_per_seq], tf.int64),
            "masked_lm_weights":
                tf.FixedLenFeature([max_predictions_per_seq], tf.float32),
            "next_sentence_labels":
                tf.FixedLenFeature([1], tf.int64),
        }

        # For training, we want a lot of parallel reading and shuffling.
        # For eval, we want no shuffling and parallel reading doesn't matter.
        if is_training:
            d = tf.data.Dataset.from_tensor_slices(tf.constant(input_files))
            d = d.repeat()
            d = d.shuffle(buffer_size=len(input_files))

            # `cycle_length` is the number of parallel files that get read.
            cycle_length = min(num_cpu_threads, len(input_files))

            # `sloppy` mode means that the interleaving is not exact. This adds
            # even more randomness to the training pipeline.
            d = d.apply(
                tf.contrib.data.parallel_interleave(
                    tf.data.TFRecordDataset,
                    sloppy=is_training,
                    cycle_length=cycle_length))
            d = d.shuffle(buffer_size=100)
        else:
            d = tf.data.TFRecordDataset(input_files)
            # Since we evaluate for a fixed number of steps we don't want to encounter
            # out-of-range exceptions.
            d = d.repeat()

        # We must `drop_remainder` on training because the TPU requires fixed
        # size dimensions. For eval, we assume we are evaluating on the CPU or GPU
        # and we *don't* want to drop the remainder, otherwise we wont cover
        # every sample.
        d = d.apply(
            tf.contrib.data.map_and_batch(
                lambda record: _decode_record(record, name_to_features),
                batch_size=batch_size,
                num_parallel_batches=num_cpu_threads,
                drop_remainder=True))
        d = d.prefetch(10)
        return d

The mirrow strategy code:

    distribution = tf.contrib.distribute.MirroredStrategy(
        devices=["device:GPU:%d" % i for i in range(FLAGS.n_gpus)],
        # num_gpus=4,
        cross_tower_ops=tf.distribute.HierarchicalCopyAllReduce())
    run_config = RunConfig(
        train_distribute=distribution,
        # eval_distribute=dist_strategy,
        log_step_count_steps=log_every_n_steps,
        model_dir=FLAGS.output_dir,
        save_checkpoints_steps=FLAGS.save_checkpoints_steps)

    model_fn = model_fn_builder(
        bert_config=bert_config,
        init_checkpoint=FLAGS.init_checkpoint,
        learning_rate=FLAGS.learning_rate,
        num_train_steps=FLAGS.num_train_steps,
        num_warmup_steps=FLAGS.num_warmup_steps,
        use_tpu=FLAGS.use_tpu,
        use_one_hot_embeddings=FLAGS.use_tpu)

    # If TPU is not available, this will fall back to normal Estimator on CPU
    # or GPU.
    estimator = Estimator(
        model_fn=model_fn,
        params={},
        config=run_config)

The problem is that if I have 4 GPU. Each GPU could run 8 batchsize. I set batch_size = 8 not 32. batch_size = 32 will OOM.

Am I right? Will the data be distributed to 4 GPU with different batches?

Tensorflow handles batches differently on distribution strategies if you're using Keras, Estimator, or custom training loops.

Since you are using TF1.15 Estimator with MirroredStrategy in one worker (1 machine), each replica (one per GPU) will receive a batch size of FLAGS.train_batch_size. So, if you have 4 GPUs, then the global batch size will be 4 * FLAGS.train_batch_size.

Here's the explanation:

  In Estimator, however, the user provides an input_fn and have full control over how they want their data to be distributed across workers and devices. We do not do automatic splitting of batch, nor automatically shard the data across different workers. The provided input_fn is called once per worker, thus giving one dataset per worker. Then one batch from that dataset is fed to one replica on that worker, thereby consuming N batches for N replicas on 1 worker. In other words, the dataset returned by the input_fn should provide batches of size PER_REPLICA_BATCH_SIZE. And the global batch size for a step can be obtained as PER_REPLICA_BATCH_SIZE * strategy.num_replicas_in_sync.

Source: TF1.X Distribution Strategy Notebook


How to compute inner product between two networks'parameters Consider a neural network with $f(x) = w^T_2 \sigma(w^T_1 x) $ where $\sigma(.)$ is a activation function such as ReLU. $w_2 \in R^{d \times k}, w_1 \in R^{k \times o}$ are two matrices. I would like to compute the inner product between two initialization of model's parameters $\theta =(w_2, w_1)$ and $\theta'=(w'_2, w'_1)$.
Should we stack all elements of networks parameter into a single vector, i.e $\theta, \theta'$ will be a big vector with the number of entries equal to $w\times k + k \times o$. Then we just compute the inner product between two vectors. Is that a good way to compute the inner product between two model' parameters to measure how similarity they are ?

You can compute the Frobenius norm between the two matrices. Also look up this related article on distance measures between matrices/


When to use Dense, Conv1/2D, Dropout, Flatten, and all the other layers? I have a binary classification problem and want to build a NN model which classifies the data whether class 0 or class 1.

My actual implementation looks like the following:

# Split dataset in train and test data
X_train, X_test, Y_train, Y_test = train_test_split(normalized_X, Y, test_size=0.3, random_state=seed)

# Build the model
model = Sequential()
model.add(Dense(23, input_dim=45, kernel_initializer='normal', activation='relu'))
model.add(Dense(1, kernel_initializer='normal', activation='sigmoid'))

# Compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

# Fit the model
history = model.fit(X_train, Y_train, validation_split=0.3, epochs=200, batch_size=5, verbose=1, callbacks=[tensorboard, time_callback])

and I get a val_accuracy of 79.85% at the last training epoch.

I used a confusion matrix to evaluate the model:

y_pred = model.predict(X_test)
y_pred =(y_pred>0.5)
list(y_pred)

cm = confusion_matrix(Y_test, y_pred)
print(cm)

and I get these values:
[[ 622  205]
 [ 216 1055]]

which makes 79.93% (approximately the same as the val_accuracy on the last epoch) of right predicted classes (622 TN + 1055 TP).

Now my question is: How to improve my NN so that I get above that? Okay, I am using 1 hidden Dense Layer, with 23 nodes. When to use Dense layers, and when to use Conv2D or Dropout, or any of the other layers of Keras?

I am classifying numerical data. Here is how my data looks like (the dataframe separated in 2 photos, because it's too wide for just 1):

PS: the categorical features were One-Hot-Encoded using:

basic_df = pd.get_dummies(basic_df, columns=['industry', 'weekday', 'category_name', 'page_name', 'type'])

And the label column is 'successful'.

What are Dense layers and when are they useful?

Dense layers are used when association can exist among any feature to any other feature in data point. Since between two layers of size $n_{1}$ and $n_{2}$, there can $n_{1} * n_{2}$ connections and these are referred to as Dense.

Where does Conv layers come in and when are they useful?

Coming to the conv layers, these are important when nearby associations among the features matter, example object detection. Neighborhoods matter to classify or detect. It is very less likely that the pixels at the opposite corners(very far away) are somehow helpful in these use cases. Filters does this job of getting associations among neighborhoods. This answer is great at understanding difference between 1D and 2D convolutions. I really dont want to repeat it.

Dropout and Flatten

Dropout is a way of cutting too much association among features by dropping the weights (edges) at a probability. The original paper from Hinton et.al is a quick and great read to grasp it. Reducing associations can be applied among any layers which stops weight
updation for the edge. The another key difference here is it has no weights associated with it. It is just there dropping things.

Flatten layers are used when you got a multidimensional output and you want to make it linear to pass it onto a Dense layer. If you are familiar with numpy, it is equivalent to numpy.ravel. An output from flatten layers is passed to an MLP for classification or regression task you want to achieve. No weighting are associated with these too. It is just flattening the hell out.

To improve the model

Try more layers, change activation functions, try batching. If there is any chance of improving features and creating any higher order ones, do it. Too many variables to speculate.

This question is real broad, but hope I covered it well.


Can a Logsitic Regression model continue making predictions after removing predictions from the data set? I have a logistic regression model that predicts churn (0 vs. 1). I was asked to use the model to predict on a historical group of non-churners, remove anyone who was marked as a churner, and then increase one variable while keeping the rest constant to see how the prediction changes. Interestingly, it predicts zero churns after removing this first cohort of predicted churn. Increasing this one variable seems to have no impact after the initial cohort is removed, although this variable has the greatest feature importance in the model. Is this simply functioning as expected?

Here is the sample code being used:

r1_pred = logisticRegr_balanced.predict(dfa)
dfa['Churn Prediction'] = r1_pred
print("Non-Churners: "+str(len(dfa[dfa['Churn Prediction']==0])))
print("Churners: "+str(len(dfa[dfa['Churn Prediction']==1])))
print("Percent Churn: " +str((len(dfa[dfa['Churn Prediction']==1]))/len(dfa['Churn Prediction'])))

Results in:

Non-Churners: 70611
Churners: 19609
Percent Churn: 0.21734648636665926

Then I create a new dataframe with only the survivors, and increment the "Customer Life" variable by 30 days.

dfa_30 = dfa[dfa['Churn Prediction']==0]
dfa_30 = dfa_60.drop('Churn Prediction', axis=1, inplace=False)
dfa_30.CustomerLife = dfa_30.CustomerLife + 30

r30_pred = logisticRegr_balanced.predict(dfa_30)
dfa_30['Churn Prediction'] = r30_pred
print("Non-Churners: "+str(len(dfa_30[dfa_30['Churn Prediction']==0])))
print("Churners: "+str(len(dfa_30[dfa_30['Churn Prediction']==1])))
print("Percent Churn: " +str((len(dfa_30[dfa_30['Churn Prediction']==1]))/len(dfa_30['Churn Prediction'])))

Results in:

Non-Churners: 70611
Churners: 0
Percent Churn: 0.0

Is the model no longer able to predict churn because it has categorized everything on the binary scale in the first prediction, so all that's left are "permanent" survivors?

The model is able to predict churn on any sample. The issue is that increasing the „customer life“ by 30 days does not lead to any change in the prediction (based on what the model has learned).

I think you can check two things:

A) More of a fundamental thing, try to improve the model, e.g. by applying regularization to get a better fit on you features. GLMNET is the thing to go for in this case: https://web.stanford.edu/~hastie/glmnet_python/

B) Check how „sensitive“ your predictions are to changes in features. You could plot or look at predictions in case you increase „customer life“ by say 10, 20, 30, ..., 100 or so days. Remember that you basically predict the probability of churn. So it is possible that (some) probabilities at 30 days of customer life are just below 50%. If you gradually look at increases of customer life, you get a good idea of how churn changes in X. This is kind of a marginal effect (dy/dX).


assigning weight for attributes in dataset How can I assign weight for attributes in dataset before training model and which best method to achieve that in python?

Only for certain models.

For example for mlp you have this possibility:

The docs show you the attributes in use.

  Attributes:
  ...

  coefs_ : list, length n_layers - 1
  The ith element in the list represents the weight matrix corresponding to > layer i.

  intercepts_ : list, length n_layers - 1
  The ith element in the list represents the bias vector corresponding to layer > i + 1.

Just build your classifier clf=MLPClassifier(solver="sgd") and set coefs_ and intercepts_ before calling clf.fit().


Can anyone explain me the difference between Factor Anaysis and PCA? Is PCA and Factor Analysis same? Both are used for Data dimension reduction but theoretically I am not able to find the difference between them?

I did FA in SPSS to reduce number of variables in my data set. I think the same think can be done using PCA as well. Never did PCA so wanted to know exact difference between two techniques.

Principal components don't have to make sense, they are just artificial numbers, summarized data if you will, used e.g. for visualizations

With factor analysis you try to find a latent variable (an underlying mechanism) that explains the data. Factor analysis is often used in psychology, eg the personality trait extraversion (the factor) explains certain behaviours (eg certain behaviours reported in a questionnaire)


Dividing the weights obtained on an already standardized data set by the standard deviation of the features? (Ridge regression) I'm trying to understand a code snippet from my lecture on Machine Learning (see the code below).

It extracts the mean and standard deviation of the features and uses them to 'normalize' (standardize) the features. The data X contains all-ones in the first column. The variable t is the output value on which we train the weights.
First the mean of the output is obtained so that ridge regression can be applied without a bias term (intercept), then the all-ones column is removed and the data is normalized. The weights are obtained by the formula:

What I don't understand is that the weights obtained by using the normalized data are once again divided by the standard deviation of the features W[1:, 0] = Wn[:, 0] * Sinv. Can someone help me out? Thanks!

PS: I know normalizing is something else and I actually mean standardizing, but my professor uses normalizing instead for some reason..

Here's the code:

def NormalisedRidgeRegression(X, t, p_lambda=0.0):
    K = X.shape[1] - 1

    M = np.mean(X[:, 1:], axis=0)
    S = np.std(X[:, 1:], axis=0)
    Sinv = 1 / S

    targetmean = np.mean(t)
    tn = t - targetmean

    Xn = np.dot(X[:, 1:] - M, np.diag(Sinv))

    Wn = np.dot(np.dot(np.linalg.pinv(np.dot(Xn.T, Xn) + p_lambda * np.eye(K)), Xn.T), tn)
    W = np.zeros((K + 1, 1))
    W[1:, 0] = Wn[:, 0] * Sinv
    W[0, 0] = targetmean
    y_hat = np.dot(X, W)
    return W, y_hat

I found my answer myself.. I'm not sure if I should delete the question, so I'll just share:

Say for a given data point before normalising we have
a*x + b*z =  y Normalising makes this a * x/std(x) + b * z/std(z) = y This means that for standard deviations greater than 1 the features are shrunk and therefore the weights are blown up. In the code the y_hat values are obtained using the original data, which means the coefficients have to be shrunk to reflect the relationship between the output and the original feature values.


How to sort two dimensional array in Java? I have a two dimensional double array. I need to sort my based column 1 in descending order. How can I sort my two dimensional array in descending order.

Sample data and code are the following:

package mypro.com;

public class SortDoubleArrary {
    public static void main(String agrs[]) throws Exception{

        double[][] testdatset=new double[5][2];

        testdatset[0][0]=20.0;
        testdatset[0][1]=0.20;

        testdatset[1][0]=100.0;
        testdatset[1][1]=0.50;

        testdatset[2][0]=10.0;
        testdatset[2][1]=0.95;

        testdatset[3][0]=220.0;
        testdatset[3][1]=0.35;

        testdatset[4][0]=140.0;
        testdatset[4][1]=0.10;
    }
}

Expected output looks like the following

 10.0    0.95
 100.0   0.5
 220.0   0.35
 20.0    0.2
 140.0   0.1

There is an overloaded sort method in java.util.Arrays class which takes two arguments: the array to sort and a java.util.Comparator object.

You can add the following lines of code with your program to get the expected result.

import java.util.Arrays;
import java.util.Comparator;

Arrays.sort(testdatset, new Comparator<double[]>() {
        @Override
        public int compare(double[] o1, double[] o2) {
            return Double.compare(o2[1], o1[1]);
        }
    });


What is the name of this statistical interaction? What is the name of the following statistical / informational interaction:

given A, I know exactly what B is.
given B, I know to some extent what A is.

I'm not looking for a probability but rather something like correlation.
Something that tells me that I don't need to include B as a feature when A
is already a feature. A correlation heatmap wouldn't give me this information, but there must be some computation that tells me "Don't include B, it's worthless."

To give some intuition: A could be an item_id and B an item_category.

Or am I wrong? Is it not worthless at all?

You might want to look at conditional entropy, H(A|B) and H(B|A).


Why does spark.ml.feautures.Word2Vec vectorize sentences instead of single words? In the process of understanding how Word2Vec in Spark differs from gensim one, I got very confused by the example presented in the Spark docs (reference link:
https://spark.apache.org/docs/2.2.0/ml-features.html#word2vec) and I was wondering why instead of transforming single words, they transform entire sentences. Isn't word2vec purpose to embed single words into a vector space? Why they embed entire sentences? How can one properly train word2vec and then applying to single words in Spark?

Spark (naively) uses average of vectors for all words in the document as representation of the document. Check the API documentation little carefully.

"The Word2VecModel transforms each document into a vector using the average of all words in the document ; this vector can then be used as features for prediction, document similarity calculations, etc."

If you are interested specifically in vector for a word (and not document), you can check getVectors method which will return data-frame of word and vector. The API behaviour does give rise to confusion though as it assumes that everyone wants to use averaging by default.


Find shared properties of a cluster samples I have a dataset which contains ~15 features. With the elbow method, I found out that the optimal number of clusters is probably four. Therefore, I applied the K-means algorithm with four clusters. Now, I would like to understand why these clusters have been formed the way they are. In other words, I would like to know what are the shared properties of the points of a specific cluster.

My idea is the following:

Let's pretend that C1 are the coordinates of the centroid of the first cluster and that P1 and P2 are two points of this cluster.

$$
   C1 =
   \begin{pmatrix}
    5\\
    2\\
    4\\
    \end{pmatrix}
$$

$$
    P1 =
    \begin{pmatrix}
    8\\
    2\\
    6\\
    \end{pmatrix}
    P2 =
    \begin{pmatrix}
    9\\
    2\\
    0\\
    \end{pmatrix}
$$

If we compute the average distance of the different coordinates of P1 and P2 we obtain this:

$$
    DistAverage =
    \begin{pmatrix}
    ((8-5)+(9-5))/2\\
    ((2-2)+(2-2))/2\\
    ((6-4)+(4-0))/2\\
    \end{pmatrix}
    =
    \begin{pmatrix}
    3.5\\
    0\\
    3\\
    \end{pmatrix}
$$

Would this mean that the second feature is a "shared property" of the points of this cluster (since the average distance is 0) ?

I hope that the question was clear enough.

Obviously you can check the variance of each attribute.

But unless the data is badly scaled, there will likely need the combination of attributes to explain the differences of clusters.


train half model in one time and other in another time I am using keras library to build the convolutional neural network model and tensorflow as a backend so what I wanted to do is that to train the model in one point time then save that model and again use that trained model to train itself to improve the accuracy. So is it possible then how should I do?

You can save a keras model during the training and then load it up again. To save a model you can do so using a json format, and then to save the weights you can do so using a callback as follows:

Let's make some random model,

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

Then we can save the model that we created itself

# Save the model
model_json = model.to_json()
with open("weights/model.json", "w") as json_file:
    json_file.write(model_json)

And we can use a callback to save the weights at some given checkpoint. In this example they are saved each time we get better model performance.

# Save the weights using a checkpoint.
filepath="weights/weights-improvement-{epoch:02d}-{val_acc:.2f}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list = [checkpoint]

epochs = 4
batch_size = 128
# Fit the model weights.
model.fit(x_train_reshaped, y_train_binary,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          callbacks=callbacks_list,
          validation_data=(x_test_reshaped, y_test_binary))

Then we can load the model and weights in the future using

# load model saved as a json and create model
json_file = open('weights/model.json', 'r')
loaded_model_json = json_file.read()
json_file.close()
loaded_model = model_from_json(loaded_model_json)

# load weights into new model
loaded_model.load_weights("weights/weights-improvement-04-0.99.hdf5")
print("Loaded model from disk")

# evaluate loaded model on test data
#loaded_model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
predictions = loaded_model.predict_classes(x_test_reshaped[0:10], verbose=0)
show_imgs(x_test[0:10])
print('Predicted classes: ', predictions)

Once you have the loaded weights, then you can resume training by once again calling the fit function

epochs = 4
batch_size = 128
# Fit the model weights.
loaded_model.fit(x_train_reshaped, y_train_binary,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          callbacks=callbacks_list,
          validation_data=(x_test_reshaped, y_test_binary))

This will perform 4 more epochs of training.


Learning rate of 0 still changes weights in Keras I just trained a model (SGD) with keras and was wondering why the change of accuracy and loss from epoch to epoch doesn't really decrease that much when I lower the learning rate. So I tested what happens when I set the learning rate to 0 and to my surprise, accuracy and loss still changed from epoch to epoch and I can't find an explanation for that. Does anyone know why this could be happening?

If your learning rate is set lower, training will progress very slowly because you are making very tiny updates to the weights. However, if you set learning rate higher, it can cause undesirable divergent behavior in your loss function. So when you set learning rate lower you need to set higher number of epochs.
The reason for change when you set learning rate to 0 is beacuse of Batchnorm. If you have batchnorm in your model, remove it and try.
Look at these link, link


LDA topic model has 0-weight topics, is that normal? While experimenting with different number of topics for the Gensim implementation of LDA, I found that for a high number of topics, the output often consists of topics with all weights equal to zero. Is this an indication of an implementation mistake or is this normal and just an indication that I should use fewer topics?

It's normal: LDA tries to maximize the likelihood of the data according to the parameters by finding the right probabilities for the parameters. Usually at the beginning increasing the number of topics allows the model to separate topics more precisely and therefore obtain a higher likelihood. But at some point (depending on the data), increasing the number of topics cannot help the model anymore because the topics are already separated to the maximum and using all the topics would actually decrease the likelihood.
So it's a sign that you don't need that many topics. Note that it doesn't mean that the number of "used" topics is optimal for the application, it's often a balance to find.


Correlation between Time Series Indicators ( Stock Prices ) I am new to time series analysis and I am currently tackling a stock market prediction problem.

I have a set of market indicators (such as Bollinger Bands, ADX etc) which are derived from the time dependent Open, High, Low and Close prices ( in Dollars ).

I need to find correlations of this indicators with the Open, Low, High and Close prices over time and drop the features which are not correlated enough ( less than 0.70 )

I am working on python. Using Pandas I have tried the pandas.dataframe.corr() method as well, but I want to know if Pearson and Spearman correlation fucntions in pandas serve my purpose ? Is it the right way or is there another way of finding correct correlations ?

Thanks.

For time series, correlation is a different. A variable might b related past N values of other variables.

This article explains theory behind finding relationships in time series (Skip to section "Stationarity in Time Series") : https://www.quantstart.com/articles/Serial-Correlation-in-Time-Series-Analysis

This is an implementation in Python : https://machinelearningmastery.com/gentle-introduction-autocorrelation-partial-autocorrelation/


Design a branching CNN So I have a CNN task at hand. Let's say I have a dataset containing pictures of a number of classes, their attributes and sub attributes and so on.

What is the state of art/How can we design a system which first predicts the class, then based on that class the attributes which are specific to that class e.g claws present in birds but not in humans and so on.

How is such a system designed with minimum resource utilization?

Note: My question is not about prediction of classes and attributes but it is the design of a system which will predict the class then go predict the specific attributes belonging to the class and so on.

I'm not sure what the state of the art is right now, but I had a similar problem for which I designed a neural network. What I did was make a model that predicted the class probabilities from the inputs (in your case the image) and then made another model that took both the normal inputs and my probability predictions. I saw a very large increase in the accuracy for the second part after I had fed the class predictions into the feature prediction section.

You could design this as one model with two outputs where the class is an intermediate layer but still an output. Your loss function would be adding the loss for the class prediction and the feature prediction. Weighting the loss for the class prediction would cause the model to lean more towards correctly predicting the class over the other features. If you must absolutely predict the class first and then the features, you can first train the model on the class loss, then train it again with the feature loss after freezing the weights associated with/preceding the class predictions.


How to dive into data science quickly? I'm new here - I hope this kind of question is legit.

I have a background in mathematics - my field is probability theory, where I am quite strong (top 2% of University).

I have little experience in programming but I guess I get things done once I know what is out there (i.e. what program languages can do for me).
I want to learn more about data science and dive into the field particular when it comes to programming. I'd also like to get going on manageable yet interesting data science related tasks (building some "usefull" statistics skill might be desirable, too).

I know kaggle.com are there other places I could look into?
Any projects I might take on?

Thanks a lot. :)

If you're new to both machine learning and programming, try taking a look at this guy's tutorials: http://www.pyimagesearch.com/. I've been using them for a while, and I think they're great because of how visual, hands-on, and practical they are. For example, he has facial recognition and handwriting recognition lessons. I'm using what I learned in his tutorials to build a self driving remote control car (similar to this: https://zhengludwig.wordpress.com/projects/self-driving-rc-car/).


Can one perform Feature Selection on a subset of training data? I have a training data set with almost one million rows and I am considering eight features initially. My machine learning model will be Random Forest regressor.
In Section 3.4.7 of "Feature Engineering and Selection: A Practical Approach for Predictive Models" there is this:

Finally, with large amounts of data, alternative data usage schemes might be a good idea. Instead of a simple training/test split, multiple splits can be created for specific purposes. For example, a specific split of the data can be used to determine the relevant predictors for the model prior to model building using the training set. This would reduce the need to include the expensive feature selection steps inside of resampling.

Based on this and for finding relevant features, I want to do an exhaustive search over all possible subsets of initial set of features on 10% of my training data (~100k rows). The evaluation of each subset will be done by 5fold cross-validating random forest with the same hyperparameters for all of the subsets.
Why 10 %? ans: to reduce computation time and because I think 100k rows would give me a reliable set of features. I have no reference to support my choices. Please let me know if you have one.
After finding THE best subset, I go on and do hyperparameter tuning using 100% of my training data.
Am I doing something wrong? Could you suggest me some references to look for vis-a-vis the parts that I am doing wrong?
I searched about this "specific split" that Max Kuhn talks about, couldn't find any similar suggestion.
More details:
Why do I want to do feature selection if I only have eight features, especially considering that RF is somewhat insensitive to redundant/irrelevant features (with respect to predictive performance, not feature importance/selection)?
Apart from the general idea of decreasing the computational time by using less features, the features are meteorological variables that may not be available/accessible easily and everywhere. So if I am to build a model that can be used in other case studies and research, it would make it easier to use if it has less variables.
Using RF for feature selection inside another method is kind of more useful I think because it can also consider the interaction effects implicitly.
I want to do all these things in an academic work, that's why I need to justify all the steps and decisions that I make. Here I am mostly concerned about the fact that I am using only 100k of my rows for feature selection. One might say, why 100k? why not 200k? and etc.
Another doubt of mine is: since I do a supervised feature selection, I should do the tuning on the remaining 90% of training data, not on 100%. Do you agree?

After finding THE best subset doesn't make sense. Your approach is not correct in this way.
If you cannot handle all data you may have to follow one of approaches below:
1- Perform clustering and chose 10% (the more the better, based on what your computer can handle) from each cluster so all 10 subsets may have similar distribution. As a result, the models theoretically should have similar performance. This approach is useful if you can find clusters in your data. Don't forget to split each subset to training/test subsets!
2- If no clusters found, develop 10 models for each subset and use the average of outputs (bagging), and see how it works.
Hyper parameter tuning is an expensive task, even for small datasets; so it may not be a good idea to do it using not 100% but 70-80% of your data as the training set.
Another thing that you should consider is the ML algorithm you choose. Random Forest may or may not be a good choice. So try different algorithms on small subsets to have an idea which algorithm may work better for your problem. Then think about hyper parameter tuning and feature selection.


Is mutual information symmetric? Why is mutual information symmetric, meaning why does I(A,B) = I(B,A)? Isnt the definition of mutual information, I(A,B), something like "the reduction of entropy in A when given B"? P(A|B) doesnt equal P(B|A) right?

Given the definition for mutual information

$$I(X;Y) = \sum_{y \in Y} \sum_{x \in X}
  p(x,y) \log{ \left(\frac{p(x,y)}{p(x)\,p(y)}
  \right) },$$

it follows from rearrangement of the summands

$$I(Y; X) =
  \sum_{x \in X} \sum_{y \in Y}
  p(x,y) \log{ \left(\frac{p(x,y)}{p(x)\,p(y)}
  \right) }.$$

Hence $I(X; Y) = I(Y; X)$.

Edit

  "$I(X;Y)$ measures the average reduction in uncertainty of $X$ that results from knowing $Y$"

If you interpret $H(X) - H(X|Y)$ where $H(X)$ is the marginal entropy of $X$ and $H(X|Y)$ the conditional entropy of $X$ given $Y$ as the reduction in uncertainty, then it is equal to $H(Y) - H(Y|X)$ and $I(X; Y)$ (see Wikipedia).


Why after adding categorical data the Linear Regression fails? Based on a training set we applied a simple Linear Regression on some attributes that all were numeric.

Now we have more attributes in terms of categories and of course we applied one-hot-encoding to transform the categories to binary attributes

Take for example this simple python code:

X_train, X_test, y_train, y_test = train_test_split(X, y, train_size = 0.8, test_size=0.2)

model = LinearRegression(normalize=True).fit(X_train, y_train)

printErrorMetrics(trueTargets=y_test, predictions=model.predict(X_test))

When the table X has only the original numeric attributes the scores from the printErrorMetrics function (RMSE, etc.) are all good enough

We were expecting better results after adding the one-hot-encoded categories but the results are so worse that the method does not seem to work.

Are we missing anything?

Do we need to preprocess the data after adding the one-hot-encoded columns/attributes?

One possible reason is that when you use one-hot-encoding for categorical data, you should set the intercept property in the function to be False:

model = LinearRegression(fit_intercept=False, normalize=True).fit(X_train, y_train)

This will avoid the dummy variable trap:
http://www.algosome.com/articles/dummy-variable-trap-regression.html

You could also use dummy encoding to avoid this problem:
http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html


Evaluate imbalanced classification model on balanced testing sample Why it would be too optimistic to compute presicion, recall and f1-score to evaluate a model trained for imbalanced classification on a balanced testing sample ?

The test set should represent what your model will encounter in practice when you use it to make predictions. It serves as a resource to demonstrate that what you did will actually work.

In case your test set does not represent what the true problem is or will be in the future, you can‘t say much on how well your model works.

If you test on a balanced set while you have trained on a unbalanced set (which is harder to tackle), your test predictions will likely look better or worse than what you could reasonably expect with the „real“ unbalanced problem. So you cannot say anything with this test about the quality of your model.


How does two convolution layers make computation quadratic increase in GoogLeNet? In GoogLeNet papaer, "For example, in a deep vision network, if two convolutional layers are chained, any uniform increase in the number of their filters results in a quadratic increase of computation."

But, I don't know how a quadratic increase is possible when the number of filters increase.

For any CNN layer $n$ with $F_n$ filters and kernel size $K_n^{height},K_n^{width}$ the number of parameters to train is
$$F_{n-1} \cdot F_n \cdot K_n^{height} \cdot K_n^{width} + F_n$$

where the first summand stands for the weights and the second for the bias parameters (if $n$ is the first CNN layer then $F_{n-1}$ equals the number of input channels). And for the special case of

$F_{n-1} = F_n$ (i.e. same number of filters for layer $n$ and layer $n-1$)
and $K_n^{height} = K_n^{width}$ (i.e. square shaped Kernel)

the number of parameters for layer $n$ is $F_n^2 \cdot K_n^2 + F_n$. Accordingly, you have a non-linear dependency of trainable parameters with regards to the number of filters.


XGBoost: # rounds is equal to n_estimators? I'm running a regression XGBoost model and trying to prevent over-fitting by watching the train and test error using this code:

eval_set = [(X_train, y_train), (X_test, y_test)]

xg_reg = xgb.XGBRegressor(booster='gbtree',
                          objective ='reg:squarederror',
                          max_depth = 6,
                          n_estimators = 100,
                          min_child_weight = 1,
                          learning_rate = 0.05,
                          seed = 1,early_stopping_rounds = 10)

xg_reg.fit(X_train,y_train,eval_metric="rmse", eval_set = eval_set, verbose = True)

This prints out as follows:

[93]    validation_0-rmse:0.233752      validation_1-rmse:0.373165
[94]    validation_0-rmse:0.2334        validation_1-rmse:0.37314
[95]    validation_0-rmse:0.232194      validation_1-rmse:0.372643
[96]    validation_0-rmse:0.231809      validation_1-rmse:0.372675
[97]    validation_0-rmse:0.231392      validation_1-rmse:0.372702
[98]    validation_0-rmse:0.230033      validation_1-rmse:0.372244
[99]    validation_0-rmse:0.228548      validation_1-rmse:0.372253

However, I've noticed the number of training rounds printed out and in the evals_results always equals the n_estimators.

In [92]: len(results['validation_0']['rmse'])
Out[92]: 100

If I change the number of trees to 600, the # of rounds goes up to 600, etc.  I was under the impression that what's being printed is the metric result from each round of training, which includes training all the trees at once.

What is going on here?  Is each layer of trees considered a separate training round?

For gradient boosting, there really is no concept of a "layer of trees" which I think is where the confusion is happening. Each iteration (round) of boosting fits a single tree to the negative gradient of some loss function (calculated using the predictions from the updated model in all prior iterations), in your case, root mean squared error. The algorithm does not fit an ensemble of trees at each iteration. Each tree is then added (with optimal weighting) to all prior fitted trees + optimal weights in previous iterations to come up with final predictions.

The validation scores you see are, as a result, the score of your complete model up to that iteration of tree fitted. So this line here for instance:

[93]    validation_0-rmse:0.233752      validation_1-rmse:0.373165

is the performance of your model on validation_ 0 and validation_1 for a model that has fit 93 trees on past gradients generated in prior iterations.


I need help on how I would go about solving this problem, my apologies for the vague title I'm a beginner in the Data Science field and I'm trying to find a way to solve this problem, but I'm not sure where to start.
Here's the situation:
I have a battery that stores and discharges energy into my house from the power grid. Energy is produced by different generation techniques throughout the day and so I have data on the emission impact of using electricity at different hours. I also have data on the energy I've used in the past year. My goal is to charge and discharge this battery in the most effective way for reducing the emission impact of my home.
If anyone can point me in the right direction for what I should know and use to solve this problem I would be greatly appreciative!

I agree that it looks like an optimization problem, but the parameters are not completely clear. Here are some vague ideas I have about the design of such a system:

Let's assume a unit of time for which you have data and by which you attempt to predict things, for instance by the hour.
I think the first part needs to be about predicting energy needs by unit of time, based on past data and using any relevant factors as features (e.g. temperature, week day or not, etc.). I think this would be a reasonably standard time series regression problem.
The second part would be the optimization of energy consumption or carbon footprint: assuming we know the energy needs for a given period of time by unit of time, what are the optimal times for storing/discharging the battery? I don't know enough about the technical factors here, but I guess it's possible to simulate the target (say carbon footprint) based on different scenarios, and then select the optimal scenario. It seems to me that the problem at this stage could be simple enough to simulate all the scenarios and find the exact optimal times, that would probably correspond to a grid search. but in case this is too complex there are ML techniques which could estimate them, for instance genetic algorithms.


How to Predict the future values of time horizon with Keras? I just built this LSTM neural network with Keras

    import numpy as np
    import pandas as pd
    from sklearn import preprocessing
    from keras.layers.core import Dense, Dropout, Activation
    from keras.activations import linear
    from keras.layers.recurrent import LSTM
    from keras.models import Sequential
    from matplotlib import pyplot

    #read and prepare data from datafile
    data_file_name = "DailyDemand.csv"
    data_csv = pd.read_csv(data_file_name, delimiter = ';',header=None, usecols=[1,2,3,4,5])
    yt = data_csv[1:]
    data = yt
    data.columns = ['MoyenneTransactHier', 'MaxTransaction', 'MinTransaction','CountTransaction','Demand']
    # print (data.head(10))
    pd.options.display.float_format = '{:,.0f}'.format
    data = data.dropna ()
    y=data['Demand'].astype(int)
    cols=['MoyenneTransactHier', 'MaxTransaction', 'MinTransaction','CountTransaction']
    x=data[cols].astype(int)

    #scaling data
    scaler_x = preprocessing.MinMaxScaler(feature_range =(-1, 1))
    x = np.array(x).reshape ((len(x),4 ))
    x = scaler_x.fit_transform(x)
    scaler_y = preprocessing.MinMaxScaler(feature_range =(-1, 1))
    y = np.array(y).reshape ((len(y), 1))
    y = scaler_y.fit_transform(y)
    print("longeur de y",len(y))
    # Split train and test data
    train_end = 80
    x_train=x[0: train_end ,]
    x_test=x[train_end +1: ,]
    y_train=y[0: train_end]
    y_test=y[train_end +1:]
    x_train=x_train.reshape(x_train.shape +(1,))
    x_test=x_test.reshape(x_test.shape + (1,))

    print("Data well prepared")
    print ('x_train shape ', x_train.shape)
    print ('y_train', y_train.shape)

    #Design the model - LSTM Network
    seed = 2016
    np.random.seed(seed)
    fit1 = Sequential ()
    fit1.add(LSTM(
        output_dim = 4,
        activation='tanh',
        input_shape =(4, 1)))
    fit1.add(Dense(output_dim =1))
    fit1.add(Activation(linear))
    #rmsprop or sgd
    batchsize = 1
    fit1.compile(loss="mean_squared_error",optimizer="rmsprop")
    #train the model
    fit1.fit(x_train , y_train , batch_size = batchsize, nb_epoch =20, shuffle=True)

    print(fit1.summary ())

    #Model error
    score_train = fit1.evaluate(x_train ,y_train ,batch_size =batchsize)
    score_test = fit1.evaluate(x_test , y_test ,batch_size =batchsize)
    print("in  train  MSE = ",round(score_train,4))
    print("in test  MSE = ",round(score_test ,4))

    #Make prediction
    pred1=fit1.predict(x_test)
    pred1 = scaler_y.inverse_transform(np.array(pred1).reshape ((len(pred1), 1)))
    real_test = scaler_y.inverse_transform(np.array(y_test).reshape ((len(y_test), 1))).astype(int)

    #save prediction
    testData = pd.DataFrame(real_test)
    preddData = pd.DataFrame(pred1)
    dataF = pd.concat([testData,preddData], axis=1)
    dataF.columns =['Real demand','Predicted Demand']
    dataF.to_csv('Demandprediction.csv')

    pyplot.plot(pred1, label='Forecast')
    pyplot.plot(real_test,label='Actual')
    pyplot.legend()
    pyplot.show()

then it generates this result:

After building and training a good model on the historical data, I don't know how I can generate the prediction for future values? For example the demand of the next 10 days. Data are daily.

NB: this is an example of how the data is shaped, the green is the label and the yellow one are the features.
after dropna() (delete null values) it remains 100 data rows, I've used 80 in the training and the 20 in the test.

This answer goes a little bit in a different direction, but I hope it still answers your question. It uses the idea of a rolling forecast/prediction.

Because you use the word horizon, I will assume you mean that you would like to predict 10 days into the future at a given time step. There are a few ways of doing this. With this kind of time-series problem, it is common to make the assumption that only a certain history will influence the next few time steps (neglecting seasonal effects).

Example in words:

So in your case, you might use e.g. the previous 60 days, and predict the next 10.
Taking your 100 rows of data as an example, this means you can actually make (100 - 60 - 9) = 31 predictions, each prediction of 10 time steps ahead (we will need these 31 predictive_blocks later). From 100 rows we lose the first 60 to fit the first model. Of the remaining 40 rows of data, we can predict 10 steps ahead (rows 61-70), then we shift the whole thing one row further and repeat. The last prediction of 10 future points would be for rows 91-100. After that we cannot predict 10 steps anymore, so we stop - and this is why we have to subtract that extra 9. [There are of course ways to continue making prediction, as to use all the data]

Example with a thousand words:

Let me paint the picture; to help explain the idea of a shifting window prediction.

For each train set (e.g. from t=0 to t=5 in red - train set 1), you want to predict the following H time steps (corresponding to t=6 in orange - test set 1). In this, your horizon is simply one i.e. H=1.

From what I understand, you would like to predict the next 10 days, meaning you need H=10.

In order to try this with your example, I think you will need to make two changes.

Change #1

The shape of your train and test sets will need to match the new horizon. Each sample of your model input (the x_train and x_test can stay the same as before. However, each sample in your test set will have to contain the next H=10 values of the label, not just a single value.

Here is a rough example of how you might do this:

# Define our horizon
H = 10

# Create data split, using values from my example above
window_size = 60
num_pred_blocks = 31    # as computed above

# Loop over the train and test samples to create the sliding window sets
x_train = []
y_train = []
for i in range(num_pred_blocks):
    x_train_block = x_train[i:(i + window_size)]    # 31 blocks of 60 * num-columns
    x_train.append(x_train_block)
    y_train_block = y_train[(i + window_size):(i + window_size + H)]    # 31 blocks of 10 * 1
    y_train.append(y_train_block)

Because you are doing out-of-sample testing, your predictions are already interesting to look analyse. Once this runs, you can then create the equivalent test datasets with the new data you mentioned.

Without knowing your data too well, I don't know if your should be predicting the y-values of the same row as the input, or of the following row. Additionally, depending on your data, you could be including the past values of y in each of the x_train blocks. In this case you'd simply swap x for the whole table i.e. data[cols], where new_cols = ['Demand'] + cols.

Change #2

You will need to make the model reflect this horizon, by forcing it to output H values.

Here is an example of how to specify the model:

# Define our horizon
H = 10

# Create the model using the parameterised horizon
fit1 = Sequential ()
fit1.add(LSTM(output_dim = 4, activation='tanh', input_shape =(4, 1)))
fit1.add(Dense(output_dim=30, activation='sigmoid')
fit1.add(Dense(output_dim=H))    # our horizon is produced!

Note:
In your model specification, you don't need to add the final linear Activation, as the preceding Dense layer by default includes a linear activation. See the excellent documentation here.

This is a big topic and there are many things that you could try out. I agree with the comments on your question, that you will need a lot more data to allow an RNN to make a meaning representation of the model.

If you are not just doing this to learn about LSTMs etc., another practical approach might be to look into simpler time-series models such as an ARIMA model (do not be intimidated by the complicated name - it is much simpler than an LSTM). Such models can be constructed quite easily with Python, using the statsmodels package, which has a nice implementation.


sklearn - overfitting problem I'm looking for recommendations as to the best way forward for my current machine learning problem

The outline of the problem and what I've done is as follows:

I have 900+ trials of EEG data, where each trial is 1 second long. The ground truth is known for each and classifies state 0 and state 1 (40-60% split)
Each trial goes through preprocessing where I filter and extract power of certain frequency bands, and these make up a set of features (feature matrix: 913x32)
Then I use sklearn to train the model. cross_validation is used where I use a test size of 0.2. Classifier is set to SVC with rbf kernel, C = 1, gamma = 1 (I've tried a number of different values)

You can find a shortened version of the code here: http://pastebin.com/Xu13ciL4

My issues:

When I use the classifier to predict labels for my test set, every prediction is 0
train accuracy is 1, while test set accuracy is around 0.56
my learning curve plot looks like this:

Now, this seems like a classic case of overfitting here. However, overfitting here is unlikely to be caused by a disproportionate number of features to samples (32 features, 900 samples). I've tried a number of things to alleviate this problem:

I've tried using dimensionality reduction (PCA) in case it is because I have too many features for the number of samples, but accuracy scores and learning curve plot looks the same as above. Unless I set the number of components to below 10, at which point train accuracy begins to drop, but is this not somewhat expected given you're beginning to lose information?
I have tried normalizing and standardizing the data. Standardizing (SD = 1) does nothing to change train or accuracy scores. Normalizing (0-1) drops my training accuracy to 0.6.
I've tried a variety of C and gamma settings for SVC, but they don't change either score
Tried using other estimators like GaussianNB, even ensemble methods like adaboost. No change
Tried explcitly setting a regularization method using linearSVC but didn't improve the situation
I tried running the same features through a neural net using theano and my train accuracy is around 0.6, test is around 0.5

I'm happy to keep thinking about the problem but at this point I'm looking for a nudge in the right direction. Where might my problem be and what could I do to solve it?

It's entirely possible that my set of features just don't distinguish between the 2 categories, but I'd like to try some other options before jumping to this conclusion. Furthermore, if my features don't distinguish then that would explain the low test set scores, but how do you get a perfect training set score in that case? Is that possible?

To see if SVM can capture any signal at all, try to balance your data: create training and test sets that consist of exactly 50% positive and 50% negative samples (i.e., by subsampling randomly from whichever one is bigger). Also standardize the data (subtract the mean and divide by standard deviation).

(For balancing, you might try changing the class_weight parameter in sklearn, but we found the manual method (subsampling) to work better.)


Follow Up Question - Why use fit when already have fit_transform? This is a follow up question to: What's the difference between fit and fit_transform in scikit-learn models?
I want to know why should we use fit at all when we have fit_transform which is much faster than using fit and transform separately? After all we will always transform the training data after fitting it. Do we have any use of fit all by itself?

It probably is fairly rare to need to use fit and not instead fit_transform for a sklearn transformer.  It nevertheless makes sense to keep the method separate: fitting a transformer is learning relevant information about the data, while transforming produces an altered dataset.  Fitting still makes sense for sklearn predictors, and only some of those (particularly clusterers and outlier detectors) provide a combined fit_predict.
I can think of at least one instance where a transformer gets fitted but does not (immediately) transform data, but it is internal.  In KBinsDiscretizer, if encode='onehot', then an internal instance of OneHotEncoder is created, and at fit time for the discretizer, the encoder is fitted (to dummy data) just to prepare it to transform future data.  Transforming the data given to KBinsDiscretizer.fit would be wasteful at this point.
Finally, one comment on your post:

we have fit_transform which is much faster than using fit and transform separately

In most (but not all) cases, fit_transform is literally the same as fit(X, y).transform(X), so this should not be faster.


How can give weight to feature before PCA I wonder how can I give weight to my feature before employing PCA. I mean somehow weighted PCA. Because I know that one of the features is better than others and want to give importance to it in creating components (It is not possible to select only that feature. I should have others impact too)

PCA is unsupervised method for finding the most important components. I don't see a reason why you should want add a weight. If you know what features are important, why use PCA at all? Or perform PCA on the features where you are unsure about the importance.

Further, components are created in directions with highest variance and the importance is measured by eigenvalues. So I can imagine you can somehow increase the variance (like Jan van der Vegt proposed). But that's a sorcery with very questionable output.


My ADALINE model using Gradient Descent is increasing error on each iteration I have used the Iris Dataset's 1st and 3rd Column for the features. and the labels of Iris Setosa (-1) and Iris Versicolor (1). I am using ADALINE as a simple classification model for my dataset. I am using gradient descent as the cost minimizing function. But on every iteration the error increases. What am I doing wrong in the python code?

import numpy as np
import pandas as pd

class AdalineGD(object):

    def __init__(self, eta = 0.01, n_iter = 50):
        self.eta = eta
        self.n_iter = n_iter

    def fit (self, X, y):
        """Fit training data."""

        self.w_ = np.random.random(X.shape[1])
        self.cost_ = []
        print ('Initial weights are: %r' %self.w_)
        for i in range(self.n_iter):
            output = self.net_input(X)
            print ("On iteration %d, output is: %r" %(i, output))
            errors = output - y
            print("On iteration %d, Error is: %r" %(i, errors))
            self.w_ += self.eta * X.T.dot(errors)
            print ('Weights on iteration %d: %r' %(i, self.w_))
            cost = (errors**2).sum() / 2.0
            self.cost_.append(cost)
            print ("On iteration %d, Cost is: %r" %(i, cost))
            prediction = self.predict(X)
            print ("Prediction after iteration %d is: %r" %(i, prediction))
            input()
        return self

    def net_input(self, X):
        """Calculate net input"""
        return X.dot(self.w_)

    def activation(self, X):
        """Computer Linear Activation"""
        return self.net_input(X)

    def predict(self, X):
        """Return class label after unit step"""
        return np.where(self.activation(X) >= 0.0, 1, -1)

####### END OF THE CLASS ########
#importing the Iris Dataset
df = pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data", header = None)
y = df.iloc[0:100, 4].values
y = np.where(y == 'Iris-setosa', -1, 1)
X = df.iloc[0:100, [0, 2]].values
#Adding the ones column to the X matrix
X = np.insert(X, 0,  np.ones(X.shape[0]), axis = 1)
ada = AdalineGD(n_iter = 20, eta = 0.001).fit(X, y)

I think something is wrong here.

self.w_ += self.eta * X.T.dot(errors)

You are going to the positive to the gradient while you should be doing is going to the negative direction of it

self.w_ -= self.eta * X.T.dot(errors)

or

self.w_ += -self.eta * X.T.dot(errors)

see this for more clarification.


Several independent variables based on the same underlying data I got a data containing, among others, two feature variables, which are based from the same underlying data (i.e. have mutual information), but they convey different information/message. How to handle such cases?

Since, logically, they will be highly correlated, it would make sense to only use one of them, preferably the one which conveys more information. But:

Is this the correct approach, or do we actually lose a valuable
information by not including it?
If including it is the correct approach, is there anything else needed to be done and/or checked to prevent messing up the model
(since they will be highly correlated)?

Example 1:

Let's say we have a feature which can be pair of any number from 1 to 3, e.g. (1,1), (3,2), (2,1), etc.
And we also have another feature which tells us how many ones (i.e., 1) are in the previous feature, so for the previous cases this would correspond to 2, 0, 1, etc.
Although this second feature does not provide us with any new information not already present in the first feature per se (i.e. can be deduced from the first feature), it does have some special meaning, i.e. lets say that the number of ones is expected to influence the results (dependent variable).

Example 2:

One variable is a discrete/integer value, and the other one is 0 if the value of the first feature is below some specific value, and 1 if higher or the same.
Just as in the Example 1, the second feature has some special meaning.

For predictive power, in general, including both shouldn't be a problem. But there is a lot of nuance here.

Foremost, if predictive power isn't all you care about: if you're making statistical inferences, or care about explainability and feature importances, then including both can cause issues. Briefly, your model may split the importance of the underlying variable across all the derived ones.

In some cases, it might not help at all: a tree model on your second example can already easily discover the derived variable given only the original.  It may be actively harmful: adding too many of these derived variables might provide noise for your model to overfit to, rather than useful signal.

In some cases, it might help a lot: in your first example, a linear classifier won't see the derived feature at all from the original, and a tree model would require several consecutive splits to see it. A neural network could build it, but it's not clear whether the training process would find it.