Time Series segmentation I have a time series graph that is segmented into a few parts based on the maintenance day. You can think of it like vertical lines appearing out of the x axis which symbolize maintenance at the date. I would like to find out the relationship between the segmentation to predict what would happen to the signal after a maintenance day. Is there any way to tackle that problem? How should i solve it?

Considering that you are trying to predict values in the future according to a set of edges on you graph, you can do this in several ways. Humans predict some events of the future looking at the past, and you can model this with traditional time series forecasting methods, such as ARMA. You can build time series aggregating values for each segmentation and apply your forecasting.


Using Keras to Predict a Function Following a Normal Distribution I am currently trying to predict a function that has a shape similar to that of a normal distribution.  It is defined as:

(4 γΩ )/((γ+2nγ)^2+(4Δ^2+2Ω^2))

I have tried to use relu, sigmoid, and tanh activation functions.

I have also tried mae, mse and binary_crossentropy loss functions.

I have also tried adam, rmsprop and sgd optimizers, and have also played around with learning rates.

Nothing seems to work. When I perform a regression analysis, I'm told that the error percentage, mse and rmse values are low.

But when I plot the predictions, it just shows a linear function.

Any suggestions?

My architecture looks like:

df=pd.read_csv('DataGen(N=1,Gamma=0.1,Omega=5).csv')
df.head()
dataset=df.values
X=dataset[:1800000,0].reshape(-1,1)
Y=dataset[:1800000,2].reshape(-1,1)
X2=dataset[1800001:,0].reshape(-1,1)
Y2=dataset[1800001:,2].reshape(-1,1)
scaler= MinMaxScaler(feature_range=(0,1))
X_min=scaler.fit_transform(X)
Y_min=scaler.fit_transform(Y.reshape(-1,1))

X_test=scaler.fit_transform(X2)
Y_test=scaler.fit_transform(Y2.reshape(-1,1))

seed=7
np.random.seed(seed)
X_tr,X_val, Y_tr, Y_val = train_test_split(X,Y,test_size=0.1, random_state=seed)

def NN():
model= Sequential()
model.add(Dense(4, input_dim=4, activation='tanh',kernel_initializer = 'normal'))
model.add(BatchNormalization())
model.add(Dense(16, activation='tanh',kernel_initializer = 'normal'))
model.add(BatchNormalization())
model.add(Dense(32, activation='tanh',kernel_initializer = 'normal'))
model.add(BatchNormalization())
model.add(Dense(64, activation='tanh',kernel_initializer = 'normal'))
model.add(BatchNormalization())
model.add(Dense(128, activation='tanh',kernel_initializer = 'normal'))
model.add(BatchNormalization())
model.add(Dense(64, activation='tanh',kernel_initializer = 'normal'))
model.add(BatchNormalization())
model.add(Dense(32, activation='tanh',kernel_initializer = 'normal'))
model.add(BatchNormalization())
model.add(Dense(16, activation='tanh',kernel_initializer = 'normal'))
model.add(BatchNormalization())
model.add(Dense(3, activation='tanh',kernel_initializer = 'normal'))
model.add(BatchNormalization())
model.add(Dense(1,kernel_initializer = 'normal'))

model.compile(loss='mean_absolute_error', optimizer='adam', metrics=['MAE'])
return model

My plotted results look kind of like this:

The blue points are the expected values and the red points are predictions.

My guess is that the problem is with your first layer: model.add(Dense(1, input_dim=1, activation='tanh',kernel_initializer = 'normal')). With a single hidden unit, a lot of information from the input layer will not be available to subsequent layers.


Would like to know some Machine Learning applications for Civil Engineering I've thought about some, like predicting Concrete's strength by it's composition, but would like some ideas on the theme. One other thing I was thinking was traffic, how would I use machine learning to predict something related to streets and roads... Maybe something related to hydraulics on a building. I'm really beginning on machine learning so I'm not so good at knowing how can I use it. Although it's amazing. Thank's a lot guys.

This is something I've experimented with substantially.  Predicting the compressive strength of concrete based on it's mix design is a great example, and fairly challenging - I used the concrete compressive strength dataset available at the UCI ML repository to help me better understand how to manipulate ML algorithms to get good predictive results.  Here's a guy who took the same data set and used BigML's tools to do an analysis.

I've also used ML to predict payitem bid prices on highway construction contracts to help improve cost estimation.  For example, I set out to determine the unit price for a specific item, like guardrail, that contractors are likely to bid given a contract's location, time of year, total value, relevant cost indices, etc.  It took a lot of time and experimentation, but I achieved surprisingly good predictive results.

Regarding your specific topics of interest, I imagine you'll have to just do a lot of searching and reviewing master's theses on related topics, taking time to learn the mathematical models and theories that they employ.

If you want to know where to get started, my learning path involved the following:

Learning whatever I could about neural networks and machine learning
theory (guys like Jeff Heaton proved helpful, here)
Taking Andrew Ng's Machine Learning course on Coursera.
Downloading and getting to know the R scripting language
Reading whitepapers / theses on applying ML to specific
applications.
Developing a solid, rigorous understanding of basic statistics theory, especially linear regression.

There's a lot of different ways to approach the topic, though.  Websites like BigML.com are a good way to get your feet wet doing ML without having a rigorous mathematical / statistical understanding, but it'll only get you so far.

In the end, you'll really need to devote time to getting your head wrapped around the math and programming in languages like R or Python.  It's also important to remember that while ML seems to favor complexity over simplicity (in many cases non-linear problems benefit from novel and sometimes non-intuitive approaches), sometimes the simplest approach is best.  It's really a question of how accurate the final model really needs to be.


Do non-parametric models always overfit without regularization? Let's scope this to just classification.
It's clear that if you fully grow out a decision tree with no regularization (e.g. max depth, pruning), it will overfit the training data and get full accuracy down to Bayes error*.
Is this universally true for all non-parametric methods?
*Assuming the model has access to the "right" features.

No - non-parametric methods only means that the method does not assume a function form of the data. There are non-parametric methods such as Random Forest that do not always overfit. In fact nonparametric methods could underfit, it could lack the ability to fit the training data. An example of this would be a decision stump.


Weights for keywords in a set of documents using Term Frequency and Inverse Document Frequency I have a set of 270 documents. I have a list of keywords that are spread across these 270 documents. I would like to rank these keywords based on the TR-IDF method. From what I read, this method helps to determine the importance of a word in a particular document as the TR of a word is of one particular document.

Could someone tell me how can I extend this approach to get the importance of a word across all 270 documents?

Eg: The word "abc" and "xyz" and spread across all the documents. So
 I can have a document matrix(word and the tr-idf) for "abc" and "xyz"
 for all the documents. Now how do I decide which word is important
 overall, not just in one document?

TF - IDF stands for  term frequency–inverse document frequency
TF counts the frequency of a term / total #terms in a given document. For each term in a document, this value changes.
IDF counts the log of ratio of total document / term appearing in #documents . This value is constant for a given unique term. Greater the idf value for a term, higher its significance.
Example:

  Document 1: This is a sample example.

  Document 2: This is another example.

Lets calculate for term = "is":
TF(is, Document 1) = 1/5
TF(is, Document 2) = 1/4
IDF(is) = log(2/2) = 0
TFIDF = TF*IDF
TFIDF(is, document 1) = (1/5)*0 = 0
TFIDF(is, document 2) = (1/4)*0 = 0
It means that the term "is" is not a significant term in the list of documents(corpus).
Lets consider term = "another"
TF(another, document 1) = 0/5 = 0
TF(another, document 2) = 1/4
IDF(another) = log(2/1) = 0.301
TFIDF(another, document 1) = 0*0.301 = 0
TFIDF(another, document 2) = (1/4)*0.301
You can observe from both examples that TF varies per document while IDF is constant.
You can convert your entire 270 documents into term document matrix.
Demo in python:

import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer

d = pd.Series(['this is a sample example','this is another example'])
df = pd.DataFrame(d)
tfidf_vectorizer = TfidfVectorizer(analyzer='word', min_df=0)
# if you want, say only top 2 features(terms)
# tfidf_vectorizer = TfidfVectorizer(analyzer='word', min_df=0, max_features=2, max_df = 3)
# Terms with given below:
# occurred in too many documents (max_df, tfidf score = 3)
# occurred in too few documents (min_df, tfidf score = 0)
# cut off by feature selection (max_features, tfidf score = 2).
tfidf = tfidf_vectorizer.fit_transform(df[0])
print tfidf_vectorizer.vocabulary_
# output: {u'this': 4, u'sample': 3, u'is': 2, u'example': 1, u'another': 0}
print tfidf_vectorizer.idf_
# output(constant): [ 1.40546511  1.          1.          1.40546511  1.        ]
print tfidf
# output:
#(0, 1)        0.448320873199    Document 1, term = example
#(0, 3)        0.630099344518    Document 1, term = sample
#(0, 2)        0.448320873199    Document 1, term = is
#(0, 4)        0.448320873199    Document 1, term = this
#(1, 0)        0.630099344518    Document 2, term = another
#(1, 1)        0.448320873199    Document 2, term = example
#(1, 2)        0.448320873199    Document 2, term = is
#(1, 4)        0.448320873199    Document 2, term = this

Source:
https://en.wikipedia.org/wiki/Tf%E2%80%93idf
http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html


Reasoning for temporal difference update rule In TD(0) learning where the value function is given by $V(s) = w^T\phi(s)$ where $w$ is a weight vector and $\phi$ is a feature map, the weight update is given by $w_{t+1} = w_{t} + \eta\delta_{t+1}\phi(s_{t})$, where $\eta$ is the learning rate and $\delta$ is the temporal difference error. The temporal difference error is given by $\delta_{t+1} = r_{t}+\gamma V_{t+1} -V_{t}$, where $r$ is the reward and $\gamma$ is the discount factor.  Note that the weight update is proportional to the old state features. This update rule can be thought of as trying to make $V_{t}$ closer to $r_t +\gamma V_{t+1}$ to make the value function more self consistent.

However, it is also possible to make the value of $\gamma V_{t+1}$ closer to $V_{t}-r_{t}$ by making the weight update rule $w_{t+1} = w_{t} - \frac{\eta}{\gamma}\delta_{t+1}\phi(s_{t+1})$. It is also possible to do something in the middle like $w_{t+1} = w_{t} + \frac{\eta}{2}\delta_{t+1}(\phi(s_{t})-\frac{1}{\gamma}\phi(s_{t+1}))$.

Are these viable weight update rules? If not, why? What properties do the learning algorithms corresponding to these update rules have?

First of all the weights update is derived using gradient descent. So the proper form of update is the first one you have. This is derived using math and satisfies that the updates are towards minimizing the Mean Square Error between true value and approximated one. For the true value we use a biased sample of the true value at the next time step plus the reward obtained at the current timestep: $r_t + \gamma \hat{v}_{t+1}$ which is your TD target (what you try to approximate). So the tricky part in RL with FA is that you try to approximate something which is also an approximation of a true quantity.

I am not very sure what do you mean by making the value of γVt+1 closer to Vt−rt. For linear approximation, as you stated at the beginning of your question, the update needs to have the following form in order to reduce the MSE of the true value function: $\Delta w=\eta(v_\pi-\hat{v}_w)\phi(s)$. As I mentioned above because in RL we don't have knowledge of the true value, but only reward signals, we substitute the target (true value) with the target form I described in the previous paragraph.

Your first proposed update, first of all, eliminates the discount factor which ensures that in an infinite horizon scenario the sum of rewards converges into a real value and not to infinity. Even if you consider episodic tasks the sign change makes your weights updating into a direction that doesn't follow the gradient of the MSE. In your second update again you use a an arbitrary rule for updating and not derived through gradient descent. By the way cutting in half the learning rate does not help in any sense as again you can define a new learning rate.

I refer you to these documents in which you can clarify how the update rule is defined and why it has that particular form and what are the optimality criteria and the role of the discount factor in RL:

RL with FA,
2.3 Optimality Criteria and Discounting

Hope this helps!


Binary classification for variable outputs I am creating a binary classifier in Keras and here's the code

model = Sequential()
model.add(Dense(30, input_dim=60, activation='relu'))
model.add(Dense(1, activation='tanh'))
model.compile(loss='mse', optimizer='adam', metrics=['accuracy'])

The problem though, is that I have variable targets, i.e. they lie between -1 and 1. And here is a histogram of the distibution of my targets

I know that this still, could work very well for me, and the model would probably be able to predict the variable outputs, the input data is very complex, and the model might not be able to train to predict the exact outputs. I am perfectly okay of the model cannot predict the exact values, as a long as the sign of the output [-1,1] is correct.

So I thought what if the loss function gave zero loss for a prediction that has the correct sign, but gave whatever the required binary loss is, if the output has the wrong sign, this way, during training it would give importance to the highly incorrect predictions, but as soon as the sign of the predictions are right (just what I need), it doesn't consider them anymore, hopefully giving a better result. The catch here is, that if I just trained it to predict the signs of the targets, it would give all of the inputs equal importance

So my question is, does any such loss function exist in Keras or anywhere else, or do I have to create my own loss function? If I have to create my own loss function, how do I do it?

Actually what you need is a linear regression model and not a binary classifier. You can trying to predict continuous variables using a classifier. So we need to build a regression model which predicts continuous variables in a range i.e from -1 to 1.

You can do this with some changes in your code.

model = Sequential()
model.add(Dense(30, input_dim=60, activation='relu'))
model.add(Dense(1))
model.compile(loss='mse', optimizer='adam', metrics=['accuracy'])

So basically we maintain a linear activation function in the output layer for regression.

Tip:

Instead of accuracy as the metric, use mae or mean absolute error to monitor the model's progress.


Do we need to use off-policy methods for policy shaping? Let's say that there is a reinforcement learning task and an agent in a environment. I want a human teacher to manually modify the policy of the agent (policy shaping) to speed up the learning of the agent. Do I have to use off-policy methods or I can get away with on-policy? Why?

You can use Policy Gradients for this which is on-policy. One of the most straightforward ways to do that is to provide the teacher/expert's probabilities over actions at each time-step. Then you can use the usual policy gradient loss plus the cross entropy between expert's actions distribution and agent's action distribution:

$$\mathcal{L(\boldsymbol{\theta})} =(R_t-V_{\boldsymbol{\theta}}(\boldsymbol{s}_t)) \log \pi_{\boldsymbol{\theta}}(a_t|\boldsymbol{s}_t) + \lambda H(\pi_{expert},\pi_{\boldsymbol{\theta}})$$

You could get a look at this paper for basic ways to affect on-policy learning in PG methods.

On the why this works, I will give you a descriptive explanation: Cross entropy is related to KL divergence which is a "measure" of the distance between two distributions. With the extra term in the Loss function we are trying to minimize the distance between the expert's distribution and agent's distribution (i.e. making the agent's one more similar to the expert's one).


Time Series Classification for loan data I have multiple columns for loan installment repayment. As there is a field for month of repayment, I want to predict if the customer is going to pay next month's installment or not.
As I have multiple variables and target variable as installment paid (Y/N), despite repayment being dependent on time variable, i.e., installments paid in past months, I'm looking to solve this problem with time series classification.
Any references will be appreciated.

It sounds like you are interested in multivariate time series forecasting. Below are some references to an example that sounds similar -- using multiple input variables to forecast air quality.

Kaggle competition- see Discussion tab for some nice examples and, well, discussion.
Corresponding tutorial


Scikit learn linear regression - learning rate and epoch adjustment I am trying to learn linear regression using ordinary least squares and gradient descent from scratch.
I read the documentation for the Scikit learn function and I do not see a means to adjust the learning rate or the epoch with the sklearn.linear_model.LinearRegression class.
Is there a standard learning rate for the linear regression model?
Epochs I am assuming are determined by the change in the error function and are thus dependent on the dataset, with some predetermined maximum amount before convergence.
Are there any other good packages that have a linear regression model that I can test my own model against with adjustable learning rates and epochs?

A linear regression model $y=\beta X+u$ can be solved in one "round" by using $(X'X)^{-1}X'y=\hat{\beta}$. It can also be solved using gradient descent but there is no need to adjust something like a learning rate or the number of epochs since the solver (usually) converges without much trouble.
Here is a minimal example in R:
x0 <- c(1,1,1,1,1)
x1 <- c(1,2,3,4,5)
x2 <- c(8,4,3,1,8)
x <- as.matrix(cbind(x0,x1,x2))
y <- as.matrix(c(3,7,5,11,14))

x
y

# (X'X)^-1 X'y
beta1 = solve(t(x)%*%x) %*% t(x)%*%y

# R's regression command
beta2 = summary(lm(y ~ x[, 2:3]))

# Gradient decent
m <- nrow(y)
grad <- function(x, y, theta) {
  gradient <- (1/m)* (t(x) %*% ((x %*% t(theta)) - y))
  return(t(gradient))
}

# define gradient descent update algorithm
grad.descent <- function(x, maxit){
  theta <- matrix(c(0, 0, 0), nrow=1) # Initialize the parameters

  alpha = .05 # set learning rate
  for (i in 1:maxit) {
    theta <- theta - alpha  * grad(x, y, theta)
  }
  return(theta)
}

# results without feature scaling
print(grad.descent(x,2000))
beta1
beta2


How to build a hybrid graph with CNN and RNN Architecture? For an instance, we have M matrix of size n*k, I would like to treat each matrix independently and perform the convolution operation at each one in order to introduce to an LSTM layer the outputs of the convolution layer. can I treat each matrix independently or do I have to use the channel technique and how I can provide each LSTM cell? you find the architecture link below, thank you in advance!!

I write an answer here since I don't have enough reputation to write a comment.

What about treating your M matrices as sequence, feeding them to a TimeDistributed(Conv2D) layer(s) and then to an LSTM layer? In this way the Conv layer should extract feature maps that are treated by the LSTM in a sequence-like fashion, updating/accumulating the states up to the M matrix. Afterwards you should put a Dense layer with sigmoid/softmax activation and get class probabilities.


What happens to a machine learning technique (specifically Decision Tress and Logistic Regression) if the validation dataset has a new category? Let's suppose I have a dataset which has a categorical variable and the problem I am solving is a classification one.

This categorical variable var has ['A','B','C'] as the possible set of data.

What happens to a decision tree if a new category 'D' is seen only in the validation data set (meaning: absolutely new data)? Supposing the variable var is a feature used in the tree.

With the decision tree:

Does it give an error? The decision tree stops the path and returns the not-final-node's probability?

With the Logistic Regression:

The dummy variables are zero for all the categories (I suppose), and then the model runs normally?

There are no categorical value support in the decision trees used in scikit-learn. Either the values are just numbers, or they have been one-hot-encoded.

One-hot encoding with an unseen input value will change the width of the feature array, which will give an error when running decision_tree.predict(X).

Without one-hot encoding the output will most likely be the same as for the value closest to it. Because the tree learns rules on form feature[2] > 3 to distinguish say a value of 3 from a value of 4 (during training). So a value of 4,5,6... etc will be treated the same way.
There are no feature[2] > 4 or feature[2] < 5 rules, since there was no data for that to cause splits that could be evaluate/optimize during training.

So if you want an error also when not using one-hot-encoding, you will have to check the range of features before you pass it to the Decision Tree.


How to choose the best optimiser in Deep Learning algorithms? I have been training a CNN to classify 4 faulty (acoustic emission/250KhZ) signals. I have no problem in implementing the algorithm using tensorflow libraries, but I am confused with which optimizer to choose. I have been trying out different optimizers, for Adam optimizer and the gradient descent optimizer algorithm gave the same results in test classification accuracy(~84% to 85%) on k-fold cross-validation. Results are same for both optimizers, is it because of the data or because of training on huge data(64000 samples/fault condition)?

Are there any specific guidelines to choose a particular optimizer for kind of data? Or else, is it randomly by trial and error?

I found this helpful.

I usually stick with Adam, and rarely attempt SGD. Simple words: try to go with the recent optimiser implemented on popular datasets and that has been implemented by the framework which claims to be better than others.

To specifically say that this optimiser is best for this kind of data is subjective, and I haven't read any relevant material related to that. In future, If anyone finds it, please do comment or edit this post.

Hope this helps.


How to do give input to CNN when doing a text processing? As a signal processing engineering and being new to NLP, I am confused with giving input to CNN network.

With my knowledge of CNN, I am trying to build a classifier for ethnicity with inputs as text (last name(LN), middle name(MN), first name(FN)).
I have a list of 8,000,000 samples with last, middle, first names and class information

array = [['person1_LN','person1_MN','person1_FN','Person1_class'],
         ['person1_LN','person1_MN','person1_FN','Person2_class'],
         ....]

I want to apply conv layer (CL) followed by Pooling Layer(PL) on LN,MN,FN respectively.

Text processing example demonstrates with the sentences to convert to word embedding. I am trying to understand this code snippet

W = tf.Variable(tf.random_uniform([vocab_size, embedding_size], -1.0, 1.0), name='W')
self.embedded_chars = tf.nn.embedding_lookup(W, self.input_x)
self.embedded_chars_expanded = tf.expand_dims(self.embedded_chars, -1)

This tutorial says training to be done using word2vec.
Reading these two blogs I could understand nothing. In Either of the cases, my data is not a sentence, moreover in the second case on what things I have to train?

Will the CL operate if I directly give the input of words without word embeddings?
If not any example on how to embed the words in my case to give the input to CNN?

So, here the question is asking how you use text in a CNN architecture.
Just like when we process images with a CNN, the text data will also be a 2D matrix, where the rows will represent the text features and the columns the sequence of characters (which make up a name) for example.
Now, of course, if we simply use one-hot encoding, we get data sparsity, which is not not particularly computational efficient, especially when using CNNs. So, preferably, we would use word embeddings, which you have described in your post. Here, word embeddings aim to collapse the high dimensionality of the input so that reduces the chance of data sparsity.
However, there is a potential problem with using word embeddings in your problem. The principal behind word embeddings is that it will assign similar word embeddings to words which can be easily replaced in similar contexts (e.g. Sam is kind & Simon is kind) and that don't affect the coherence of a sentence. As you can see, when it comes to proper nouns, we might end up with similar vectors for all names since they are very interchangeable in any context and do not affect the overall coherence of the sentence.
Therefore, although this is contradictory, I would advise using one-hot encoded sequence of characters as a start point for this problem and then when you implement this, you might come up with a more effective way to represent the names.


How to add jpg images information as a column in a data frame I have jpg images stored in a folder. For ex: 11_lion_king.jpg,22_avengers.jpg etc.

I have a data frame as below:

data_movie.head()

movie_id  genre
11       ['action','comedy]
22       ['animation',comedy]
..........

I want to add a new column movie_image into the data_movie data frame with the jpg information mapped correctly with movie_id column as shown below:

movie_id  genre                movie_image
11       ['action','comedy]    11_lion_king.jpg
22       ['animation',comedy]  22_avengers.jpg
.........

Help will be appreciated.

I assume you a list of the filenames called movie_images

# Could get filenames with:
# import os;  movie_images = os.listdir("./folder/with/images/")
movie_filenames = ["11_lion_king.jpg", "22_avengers.jpg"]

First create a mapping between the ID values and the filenames:

# Use the "_" to split the filename and take the first items, the ID
mapping = {f.split("_")[0]: f for f in movie_filenames}    # <-- a dictionary-comprehension

Now add a column of some empty values (whatever you like) that will hold the movie_image values:

data_movie["movie_image"] = pd.Series()    # will be filled with NaN values until populated

Now iterate over this mapping, inserting the movie filenames for the correct movie IDs:

for movie_id, movie_image_filename in mapping.items():
    df.loc[df.movie_id == movie_id, "movie_image"] = movie_image_filename

This should produce the output dataframe you described.

As a side note (in case you are ever tempted): never load the actual images into a  pandas dataframe. It is best to load them as NumPy arrays or something similar. Pandas DataFrames are in essence just annotated NumPy arrays anyway.


Random forest with zero precision for unbalanced test data Apologies if this is a basic question.

I have a very unbalanced dataset in which the records are labelled by one of two  classes, class1 (negative class) and class2 (positive class):

class 1: 1.5 million records

class 2: 100 thousand records

So there is essentially a 1:15 ratio between them.

Because I am dealing with time dependent observations, I have divided the train/test data in the following way:

train data:

class1: 900 thousand

class2: ~100 thousand

test data:
class1: 600 thousand

class2: 25

Thus having 62% of training data and 38% of test data, the test data being more recent.

I am using a Random Forest Classifier as a model, and have given a class weight of 1 to class1 and 9 to class2, to compensate for the data imbalance.

I read that to measure the quality of a scenario with unbalanced data, I should use a Precision vs Recall curve.

The problem is, none of the 25 data points from class2  are classified correctly, and thus the true_positives = 0.

In a case like this, how can I evaluate my model effectively?
Should I undersample class1 from my test data?
Are there more parameters besides class weights that I should try to tune?

Thank you for your time.

You mentioned that you're dealing with time-dependent observations, and I'm guessing that's why you chose a train-test split for evaluation.  If you continue along this path, then I think you just need a larger sample of class2 in your test set.

How did you come up with 25 examples of class2 in your testing set?  You should be using stratified sampling to produce your train-test sets.
It looks like you started with a 60-40 train-test split with class1.  If so, then your datasets should have the following distributions:

Train: 900,000 of class 1, 60,000 of class2

Test: 600,000 of class1, 40,000 of class2

Since you have a decent amount of data, you might want to train with more than 60%.  An 80-20 split would probably provide a better estimate of your model's final accuracy.

In any case, I think you can do better than train-test splitting!  There's a modified version of cross-validation designed for time-dependent data.  And scikit-learn has an implementation: TimeSeriesSplit

If you go this route, then you will probably want to use different weights for your classes.  Since class1 outnumbers class2 15:1, then you could assign a weight of 1 to class1 and 15 to class2 (assuming you care equally about the two classes).

To answer a few more of your questions:

  Should I undersample class1 from my test data?

I don't think this will help since you're already adjusting for the class imbalance with sample weighting.

  Are there more parameters besides class weights that I should try to tune?

You should always tune model hyperparameters :) For Random Forests, the most salient parameters are probably the number of trees, the max depth, and minimum impurity decrease.


Question about scaling in (R) Principal Components Regression I'm reding Introduction to Statistical Learning (ISL), Ch. 6 Principle Component Regression. When I look at the Labs, I wonder how scaling is handled in R in this case.
The ISL-Lab "Chapter 6 Lab 3: PCR and PLS Regression" suggests the following approach:
pcr.fit=pcr(Salary~., data=Hitters,subset=train,scale=TRUE, validation="CV")
pcr.pred=predict(pcr.fit,x[test,],ncomp=7)

Here train and test are lists of row indexes, indicating test and training data, like:
> train
  [1]  70  98 150 237  53 232 243

Question:
During training the data is scaled by scale=TRUE. However, scaling appears to be only used for subset=train if I'm not wrong (which seems correct).
However, when predicting on x[test,], no scaling seems to take place. My take would be that the same scaling should be applied to the test data as was used to scale the training data in order to produce correct predictions.
Am I wrong here? Or is there anything going on in the background with pcr() which makes scaling the test set unnecessary?
Unfortunately the DOCS come with no details on scale() with pcr().
What is the correct approach here?

I agree that if scaling is used on the training data, it should also be used on the test data.
However, from what I see in the pcr documentation there is not scale option in the function. This seems to be confirmed by running the code from the example in the documentation:
pcr: Prinicpal component regression transfer function models
> library(analogue)
> data(ImbrieKipp)
> data(SumSST)
> pcr(ImbrieKipp, SumSST, tranFun = Hellinger, scale = TRUE)

    Principal Component Regression Model

Call:
pcr(x = ImbrieKipp, y = SumSST, tranFun = Hellinger, scale = TRUE)

No. of Components: 27

RMSE (Apparent):
     PC1      PC2      PC3      PC4      PC5      PC6      PC7      PC8      PC9     PC10     PC11
2.381215 1.707588 1.680896 1.679774 1.608903 1.535145 1.507995 1.496939 1.496908 1.432142 1.426444
    PC12     PC13     PC14     PC15     PC16     PC17     PC18     PC19     PC20     PC21     PC22
1.405155 1.391348 1.349172 1.349172 1.315284 1.313187 1.311801 1.291201 1.206484 1.188438 1.187503
    PC23     PC24     PC25     PC26     PC27
1.171215 1.170947 1.170380 1.162497 1.162355
> pcr(ImbrieKipp, SumSST, tranFun = Hellinger, scale = FALSE)

    Principal Component Regression Model

Call:
pcr(x = ImbrieKipp, y = SumSST, tranFun = Hellinger, scale = FALSE)

No. of Components: 27

RMSE (Apparent):
     PC1      PC2      PC3      PC4      PC5      PC6      PC7      PC8      PC9     PC10     PC11
2.381215 1.707588 1.680896 1.679774 1.608903 1.535145 1.507995 1.496939 1.496908 1.432142 1.426444
    PC12     PC13     PC14     PC15     PC16     PC17     PC18     PC19     PC20     PC21     PC22
1.405155 1.391348 1.349172 1.349172 1.315284 1.313187 1.311801 1.291201 1.206484 1.188438 1.187503
    PC23     PC24     PC25     PC26     PC27
1.171215 1.170947 1.170380 1.162497 1.162355

I did check the the data were not standardised to begin with !
So in your case I would simply use scale() on the data before running pcr

Edit to address the point raised in comments. In the pcr function there is no warning or error if you use a completely fictitious variable:
> pcr(ImbrieKipp, SumSST, tranFun = Hellinger, SomethingIjustMadeUp = TRUE)

    Principal Component Regression Model

Call:
pcr(x = ImbrieKipp, y = SumSST, tranFun = Hellinger, SomethingIjustMadeUp = TRUE)

No. of Components: 27

RMSE (Apparent):
     PC1      PC2      PC3      PC4      PC5      PC6      PC7      PC8      PC9     PC10     PC11
2.381215 1.707588 1.680896 1.679774 1.608903 1.535145 1.507995 1.496939 1.496908 1.432142 1.426444
    PC12     PC13     PC14     PC15     PC16     PC17     PC18     PC19     PC20     PC21     PC22
1.405155 1.391348 1.349172 1.349172 1.315284 1.313187 1.311801 1.291201 1.206484 1.188438 1.187503
    PC23     PC24     PC25     PC26     PC27
1.171215 1.170947 1.170380 1.162497 1.162355

No warning or error


Is there a general guideline for experience replay size, and how to store? I am trying to use deep Q-learning on color images of size (224 x 224 x 3). I have read that implementations of DQN use experience replay sizes around 50,000. If my calculation is correct, this is over 56 Gigabytes for my data (two images per tuple, one image for state and next_state, totaling 100,000 images). Is this correct?

If yes, how can I go about this without that much RAM?

If yes, how can I go about this without that much RAM?

Your value of 56GB seems correct to me, assuming you include a multiplier of 4 for the "4 frames per state representation" used in the DQN/Atari paper. However you should note that in the original paper, the images were converted to greyscale and downsampled to 110×84 prior to use in state representation. This made them 16 times smaller than the frames in your problem, so the whole data set would fit into 4GB.

If you must use larger images, you could store them on disk - maybe in a database - and load on demand. That may unfortunately create an I/O bottleneck that slows learning, although you should still be able to work with it. You can parallelise fetching from the database for mini-batches with the learning process, and this is similar to the mini-batch generators used for things like ImageNet training. You can also work on improving disk performance using optimisations such as parallel disk arrays or SSDs.

You could also pre-process the frames using a hidden layer embedding from a generic computer vision network trained on e.g. ImageNet, and store that representation, not the raw pixel values. This may limit self-discovery of important low-level features by the learning agent, but then again it may still be worth a shot, as lower-level image features are often very similar across different problems, and transfer learning in e.g. image classification has been quite successful using that approach.

More likely practical answers used by RL researchers at least up to a certain scale is one of:

Store less states in replay memory. Base the size of replay on the memory you have available. Yes this may compromise the learning, but there is no special magic about the number 50,000 and if you are optimising resource use you may have to decide between how efficiently a system learns with 10,000 fast replay memory size or 50,000 slower I/O-based replay memory size.
Buy more memory. The big name research labs working in deep RL are funded well enough that they can afford to throw money at this problem. One estimate for how much AlphaGo Zero hardware cost is $25million, so you can imagine that loading a few machines with 128GB+ RAM if they thought it necessary for any reason on any other flagship project would not be a major blocker.

If you look at what OpenAI are doing at the cutting edge of video game playing, you can see that their hardware setup is equally monstrous. It is not clear whether they have an issue with storing experience as they use a different algorithm, or needing RAM in general, but if they did, it is also clear they could quite happily finance maximum RAM on their training rigs.

Do note I am not a RL researcher myself (just a hobbyist) and have not asked anyone else what they would do when faced with this problem.


Best way to implement 10-fold cross validation for CNN model I was performing a binary classification problem with 15000 RGB images using a scratch build CNN model. While it comes to evaluate the model, I can do it in two ways:

Splitting data Train and Test and use 10 fold cross-validation for the training data. Later with the best model, I would use the unseen Test data. In this way I got appx. 91.5% avg. accuracy for both test and validation.
Just use 10 fold cross-validation and got 92.5% avg accuracy(slightly better result than the previous one.)

Which option would be the best for reporting the performance of my model in the research article?
TIA

In my point of view, it should be Option-I
 Primary logic is to test your Model on unseen data.
Reason for that is,
When you executed your K-Fold and let's say got a score ~87% and then you tweaked your Hyperparameters.
In this way, we actually leak the test data(K-Fold test set) information to the learning process, and eventually, the process will overfit to the test data too if this hit-try happens multiple times.
So, the Option-I score should be closer to the score on future new data.


Keras models break when I add batch normalization I'm creating the model for a DDPG agent (keras-rl version) but i'm having some trouble with errors whenever I try adding in batch normalization in the first of two networks.

Here is the creation function as i'd like it to be:

def buildDDPGNets(actNum, obsSpace):
    actorObsInput = Input(shape = (1,) + obsSpace, name = "actor_obs_input")
    a = Flatten()(actorObsInput)
    a = Dense(600, use_bias = False)(a)
    a = BatchNormalization()(a)
    a = Activation("relu")(a)
    a = Dense(300, use_bias = False)(a)
    a = BatchNormalization()(a)
    a = Activation("relu")(a)
    a = Dense(actNum)(a)
    a = Activation("tanh")(a)   # Bipdeal walker applies torque (-1, 1).
    actor = Model(inputs = [actorObsInput], outputs = a)
    criticActInput = Input(shape = (actNum,), name = "critic_action_input")
    criticObsInput = Input(shape = (1,) + obsSpace, name = "critic_obs_input")
    c = Flatten()(criticObsInput)
    c = Dense(600, use_bias = False)(c)
    c = BatchNormalization()(c)
    c = Activation("relu")(c)
    c = Concatenate()([c, criticActInput])
    c = Dense(300, use_bias = False)(c)
    c = BatchNormalization()(c)
    c = Activation("relu")(c)
    c = Dense(1)(c)
    c = Activation("linear")(c)
    critic = Model(inputs = [criticActInput, criticObsInput], outputs = c)
    return (criticActInput, actor, critic)

This causes me to get the following error:

    InvalidArgumentError: You must feed a value for placeholder tensor 'actor_obs_input' with dtype float and shape [?,1,24]
     [[{{node actor_obs_input}}]]

However, If I remove the batch normalization from the first network (a) and do not change the second network (c) at all, It runs as it should.

    a = Flatten()(actorObsInput)
    a = Dense(600, use_bias = False)(a)
    #a = BatchNormalization()(a)
    a = Activation("relu")(a)
    a = Dense(300, use_bias = False)(a)
    #a = BatchNormalization()(a)
    a = Activation("relu")(a)
    a = Dense(actNum)(a)

Its also notable that if I do it the other way around (remove bn from c and leave it in a) the error still occurs.
Any idea as to why that's happening?
Its odd because it runs fine with batch norm in the critic, just not the actor.
The models are being used by keras-rl DDPG agent btw.

Update:
I've tried rewriting it with the sequential object instead of the functional api. Didn't help. Still got the error with no change. I'm beginning to think this is some sort of problem with keras's batch normalize class when being applied to systems of multiple models.

This link describes the cause of the problem. Its because we try to feed a placeholder defined as an Input layer using another input layer. You can modify your code as follows:

def buildDDPGNets(actNum, obsSpace):

  actorObsInput = Input(shape = (1,) + obsSpace, name = "actor_obs_input")
  criticActInput = Input(shape = (actNum,), name = "critic_action_input")

  flattened_obs = Flatten()(actorObsInput)
  a = Dense(600, use_bias = False)(flattened_obs)
  a = BatchNormalization()(a)
  a = Activation("relu")(a)
  a = Dense(300, use_bias = False)(a)
  a = BatchNormalization()(a)
  a = Activation("relu")(a)
  a = Dense(actNum)(a)
  a = Activation("tanh")(a)   # Bipdeal walker applies torque (-1, 1).
  actor = Model(inputs = [actorObsInput], outputs = a)

  c = Dense(600, use_bias = False)(flattened_obs)
  c = BatchNormalization()(c)
  c = Activation("relu")(c)
  c = Concatenate()([c, criticActInput])
  c = Dense(300, use_bias = False)(c)
  c = BatchNormalization()(c)
  c = Activation("relu")(c)
  c = Dense(1)(c)
  c = Activation("linear")(c)
  critic = Model(inputs = [criticActInput, criticObsInput], outputs = c)

  return (criticActInput, actor, critic)


Outlier detection for Disk Space Usage I would like to do outlier or anomaly detection on the disk free space data. Sample dataset as below (I don't have any label dataset):

date                   free_space (GB)
2019-05-15 09:00:00    102.65
2019-05-15 09:05:00    102.69
2019-05-15 09:10:00    103.11
2019-05-15 09:15:00    102.58
2019-05-15 09:20:00    102.55

I would like to detect whether the new value of disk space is an outlier or not. There are several outlier analysis methods (ref link):

Box plot analysis
Based on Z score
IQR based analysis (this is similar to box plot analysis)

Above methods are more statistical approach to detect outlier. There are several ways using an unsupervised machine learning algorithm to detect outlier (ref link). For example,

K-mean
Markov Chain
Isolation Forest etc.

Which method is suitable for the above dataset? Should I implement unsupervised based machine learning algorithm or statistical approach?

Given the specific context of detecting abnormal changes in the amount of free space, I'd suggest that you use the variation over time instead of the raw amount. For instance:

date                   free_space  variation
2019-05-15 09:00:00    102.65      NA
2019-05-15 09:05:00    102.69      0.04
2019-05-15 09:10:00    103.11      0.42
2019-05-15 09:15:00    102.58      -0.53
2019-05-15 09:20:00    102.55      -0.03

Whatever method you use, the variation is a much more relevant information to detect an unusual change than the raw size. You could also use a time window, e.g. calculate the variation over the last 30 minutes.

Personally I would simply use a heuristic for something like this: if the absolute value of the variation is higher than a threshold then label as outlier. The threshold could be a percentage of the size of the disk, e.g. 5%.


Earlystopping in multi-output deep learning When working with a neural network with more than one output, what is generally advised as the best strategy for early-stopping the training process?

Given that I am currently monitoring the net validation loss (validation loss from n different output neurons added together), and have noticed my training early-stopping before convergence in one or more of my losses, I am curious as to what the best practices in this situation are.

Should one monitor the net validation loss, or is there any way to implement early-stopping such that m out of n (m goes from 1 to n) outputs' validation losses are monitored and training stops at convergence of all/oscillations within a certain threshold.
How would one implement this without risking overtraining some output nodes while preventing others from under-training?

I am a newcomer to this field, if you hadn't guessed already so please bear with me if this question is not extremely well put together!

How would one implement this without risking overtraining some output nodes while preventing others from under-training?

This isn't done with early stopping. There are rather other techniques that could prevent that some main nodes are trained in your model such as Dropout. By using Dropout your model can't rely on some main nodes, because they could get shut off at any time. So with Dropout the model  should learn to balance the weights throughout the nodes.

  Should one monitor the net validation loss, or is there any way to implement early-stopping such that m out of n (m goes from 1 to n) outputs' validation losses are monitored and training stops at convergence of all/oscillations within a certain threshold.

You noted that n in your case refers to the number of output nodes. Generally, that is the number of classes in your classifier which is mostly denoted by c. So, in multi-class classification you wouldn't have c different validation losses but average them to obtain one mean validation loss for the whole model.

  How would one implement ...

There is no gold rule to how to implement early stopping.
The simplest way to do it is as follows:

Set a so called patience i.e. after how many epochs do we stop if the loss doesn't improve (usually set to 10)
After each epoch check your validation loss
Then select the model patience epochs before you stopped, because that was the best performing model.

You see that this is not the best approach and there could be better ones. In this paper, some of them are suggested.

  ... training stops all/oscillations within a certain threshold

Could be another idea, that would have to be checked. I wouldn't recommend it though, because with SGD you still have oscillations at the very beginning.


From logistic regression to XGBoost - selecting features to run the model with I have been asked to look at XGBoost (as implemented in R, and with a maximum of around 50 features) as an alternative to an already existing but not developed by me logistic regression model created from a very large set of credit risk data containing a few thousand predictors.
The documentation surrounding the logistic regression is very well prepared, and as such track has been kept of the reasons for exclusion of each variable. Among those are:

automated data audit (through internal tool) - i.e. detected excessive number of missings, or incredibly low variance, etc.;
lack of monotonic trend - for u-shaped variables after attempts at coarse classing;
high correlation (>70%) - on raw level or after binning;
low GINI / Information Value - on raw level or after binning;
low representativeness - assessed through population stability index, PSI;
business logic / expert judgement.

A huge number of the variables are derived (incl. aggregates like min / max / avg of the standard deviation of other predictors) and some have been deemed too synthetic for inclusion. We have decided to not use those in XGBoost either.
The regression was initially ran with 44 predictors (the output of a stepwise procedure), whereas the final approved model includes only 10.
Because I am rather new to XGBoost, I was wondering whether the feature selection process differs substantially from what has already been done in preparation for the logistic regression, and what some rules / good practices would be.
Based on what I have been reading, perfect correlation and missing values are both automatically handled in XGBoost. I suspect monotonicity of trend should not be a concern (as the focus, unlike in regression, is on non-linear relations), hence binning is likely out; otherwise I am a bit unsure about handling of u-shaped variables. Since GINI is used in deciding on the best split in decision trees in general under the CART (“Classification and Regression Trees”) approach, maybe this is one criterion that it is worth keeping.
I have been entertaining the idea of perusing our internal automated data audit tool, removing std aggregates (too synthetic as per above), removing low GINI and low PSI variables, potentially treating for very high (95+%) correlation, and then applying lasso / elastic net and taking it from there. I am aware that Boruta is relevant here but as of now still have no solid opinion on it.

One of the main advantages of tree-based methods is that "high correlation" is not so much of a problem (when compared to linear models). Another advantage is that there is no parametric assumtion behind the model. Thus you have a good chance that some of the variables you dropped to do the Logit (high correlation/u-shaped) can be used with boosting.
WRT to raw levels and/or low representativeness, it could be that some features are actually useful in the boosting process. I would at least give it a try. Check the feature importance to exclude features with little predictive power.
Missings carry no information, so exclude features with lot of missings. Bossting essentially has the same problems as Logit here. Same is true for features which are excluded by expert advice. However, you could still see if some of the features help. This is a matter of testing.
Note that xgboost requires a matrix in which factors are encoded as dummies. This is highly relevant when you work with R.
You can add early stopping to boost until no more progress is made.
Regarding your list, my feeling (not knowing the data in detail) would be...
Can be used with xgboost:

lack of monotonic trend - for u-shaped variables after attempts at coarse classing;
high correlation (>70%) - on raw level or after binning;

Worth a try:

low GINI / Information Value - on raw level or after binning;
low representativeness - assessed through population stability index, PSI;

Exclude:

business logic / expert judgement.
automated data audit (through internal tool) - i.e. detected excessive number of missings, or incredibly low variance, etc.;

Some ideas for the model:
There are many hyperparameters that can/must be tuned. Make sure you pick the right "Learning Task Parameters" for your problem (likely a right skewed distribution). Have a look at the docs, there are some good hints. Also try regulation (alpha, lambda). PLay with max_depth (usually between 5 and 8 work well). Too small trees do not fit well, too large trees overfit. Look at subsample to fight overfitting by making the tree booster "stochastic". Also look at the learning rate eta. Often a low(er) learning rate works find with large(er) data.
You can also add custom losses which can be helpful. I played around with that in R and you can find some code (Python is not so different) here: https://github.com/Bixi81/R-ml
As an alternative to xgboost, you could also look at LightGBM https://github.com/Bixi81/Python-ml/blob/master/boosting_regression_boston.py


PReLU Activation Function update rule I just finished reading Delving Deep into Rectifiers paper. This paper proposes a new activation function called PReLU. Maybe it is obvious, because the paper did not mention it, but I want to know when is the parameter of a PReLU updated? Is it updated before weight update or after weight update?

Is it updated before weight update or after weight update?

It doesn't usually matter, provided your NN framework has separate gradient calculation stage from parameter update stage.

The gradients should be collected with all weights and parameters at their current values (during back-propagation). Then the parameters are updated, using those gradients. There should be no interaction between the parameters during the update step.

However, if you have implemented the training code yourself from scratch, and have interleaved the gradient calculations and weight updates, then this only works in a pure online situation with weights updated as the gradients are calculated. In that scenario, you should probably update the weights for a layer before updating the PReLU parameter for the same layer, to prevent changes in the activation function altering the gradients during your calculations (this is not a concern with usual weight updates).

Alternatively, separate your gradient calculations from the weight update steps. This is more flexible and allows you to use other more advanced optimisations and layer designs.


Why is Keras fit_generator different from fit? I would like to use a custom generator so that I can implement custom augmentations on my dataset in Keras. However, I built a generic generator (without augmentation) and am confused why it is performing way worse than the built-in fit function.

My generator looks like this:

def data_generator_plain(features, labels, batch_size):

    batch_features = np.zeros((batch_size, IMG_SIZE, IMG_SIZE, NUM_CHANNELS), dtype=np.float64)
    batch_labels = np.zeros((batch_size, NUM_KEYPOINTS * 2), dtype=np.float64)

    while True:
        steps = len(batch_features) // batch_size
        for i in range(steps):
            for j in range(batch_size):
                batch_features[j] = features[(i*batch_size)+j]
                batch_labels[j] = labels[(i*batch_size)+j]

            yield batch_features, batch_labels

and when i call model.fit_generator(data_generator_plain(X_train, y_train, BATCH_SIZE), steps_per_epoch=X_train.shape[0]//BATCH_SIZE, epochs=NUM_EPOCHS, validation_data=(X_val, y_val)) the val_accuracy ends around 73% while the basic model.fit(X_train, y_train, epochs=NUM_EPOCHS, batch_size=BATCH_SIZE, validation_data=(X_val, y_val)) val_accuracy ends around 93%.

Is my generator implementation correct? In theory, it should perform the same as the built-in fit function, right? I imagine it is not showing the algorithm all the examples it needs to. Thank you!

The Keras fit_generator takes in a python generator as an input to train the model over an array of training data. Usually, these generators are formed using built-in Keras functions such as ImageDataGenerator that takes in the image array as inputs and performs user-defined operations (such as augmentations, normalization, transformations, etc.) on the same in real time. This considerably reduces the memory consumption.

For more, refer Keras image preprocessing docs


Papers on anger detection in dialogues I am interested in anger detection in dialogues and I want to study multiple methods like LSTM, CNN, etc. Are there any good research papers or books about this subject?

DeepMoji is a fun project that came out of MIT, which predicts emojis that are most related to an input sentence. Gimmicks aside, it seems perfectly adequate for your task of anger detection.

Paper
Blog post
Pretrained models: PyTorch or Keras.


What type of a problem is this? The problem: I have a huge dataset which is more than 200GBs in size. The dataset contains around 200 columns (predictors). The task at hand is to predict which product to promote to the customer so that he/she ultimately buys it and thus maximizes the revenue of the company.

It is a cross selling situation but I need to look at product recommendation as well as revenue maximization.

From what I understand, revenue maximization would require a regression model. (Correct me if I'm wrong)

I am not sure how to recommend products since there are more than 2000 unique products. (Dummy coding would require tremendous amount of time and resources I feel).

Due to the sheer size of the data, I am planning to use Python for handling the data. (Suggestions about R also welcome)

PS: Forgive me if the problem seems too basic, but I have just started learning

[UPDATE]:

The data is in long(narrow) format
I can also use R in order to tackle this
Products are identified by their unique product ID (2000+ unique product IDs)
Headers:
date | time | pid | cust_id | ... | amount | tax | net_revenue
Net revenue: Continuous variable
Product ID: Continuous but to be treated as nominal
Cust_id: Continuous but to be treated as nominal

If I understand your question correctly, you need to recommend one product out of 2000. You want to pick the one with the highest revenue expectation. This can be decomposed in the probability of making the sale times the revenue given that is a sale. The revenue for a sale is known, the probability is what is needed. I assume you have historic data of which customers (with the given features) bought which of these products. Now you can train regression models to estimate the probability of a sale for a specific product. Logistic Regression and Neural Networks with a sigmoid activation functions are well suited for unbiased probability estimates, by optimizing the log-loss cost function.

With this amount of data I propose to start on a subset of your data and using Logistic Regression. Since this is a generalized linear model you will likely need to do some feature engineering to increase performance. Since you mentioned you use Python, scikit-learn has a lot of functions for Logistic Regression optimization and feature engineering so take a look at that.

Once you have trained 2000 models, for a new recommendation you would run the features through the models, get all the probabilities, multiply them with revenue corresponding to this product and pick the highest one.


sklearn nmf - question about its use This is a complete noob question, I'm new to Python, and I understand the basics of NMF, but when implementing it in sklearn it seems a bit convoluted and I wanted to ask whether anyone knows how to exploit the results.

I've imported data from a file, and created and shaped an array, and now have implemented some initial NMF code I took from the sklearn guide online.

nmf = NMF(init='nndsvd', max_iter=2000, random_state=0, shuffle=False, solver='cd',verbose=0)
W = nmf.fit_transform(myfile);
H = nmf.components_;

#X_new = np.array(?)
#W_new = model.transform(X_new)

But here's the bit I'm getting lost - and why I commented out the last two lines - where does X_new come from, am I meant to be imported another matrix for comparison - I thought the point of NMF was to identify key features of the matrix under analysis?

Any help greatly appreciated.

The API allows two types of operations: Learn and transform a matrix under analysis. Or alternatively, learn the factorization on one matrix (myfile / X) and then apply the same transformation on a new matrix (X_new), having the same number of columns (but potentially different number of rows).

Think of it as learning the basis transformation on one set of input data (.fit method) and then apply the same basis transformation on new set of data (.transform method).

If you are interested in transforming the same matrix (myfile in your case), use only the fit_transform method instead of transform.


What is the difference between SVM and logistic regression? While reading the book by Aurelien Geron, I noticed that both logistic regression and SVM predict classes in exactly the same way, so I suspect there must be something that I am missing. In the Logistic regression chapter we can read:

  $σ(t) < 0.5$ when $t < 0$, and $σ(t) ≥ 0.5$ when $t ≥ 0$, so a Logistic Regression model predicts $1$ if $θ^T · x$ is positive, and $0$ if it is negative.

Similarly, in the SVM chapter:

  The linear SVM classifier model predicts the class of a new instance x by simply computing the decision function $w^T · x + b = w_1 x_1 + ⋯ + w_n x_n + b$: if the result is positive, the predicted class $ŷ$ is the positive class ($1$), or else it is the negative class ($0$).

I know that one way they could be different is because of the loss function they use: while log loss is used in logistic regression, SVM uses hinge loss to optimize the cost function. However, I would like to get this thing completely clear. How are the two models actually different?

Both logistic regression and SVM are linear models under the hood, and both implement a linear classification rule:

$$f_{\mathbf{w},b}(\mathbf{x}) = \mathrm{sign}(\mathbf{w}^T \mathbf{x} + b)$$

Note that I am regarding the "primal", linear form of the SVM here.

In both cases the parameters $\mathbf{w}$ and $b$ are estimated by minimizing a certain function, and, as you correctly noted, the core difference between the models boils down to the use of different optimization objectives. For logistic regression:

$$(\mathbf{w}, b) = \mathrm{argmin}_{\mathbf{w},b} \sum_i \log(1+e^{-z_i}),$$

where $z_i = y_if_{\mathbf{w},b}(\mathbf{x}_i)$.

For SVM:

$$(\mathbf{w}, b) = \mathrm{argmin}_{\mathbf{w},b} \sum_i (1-z_i)_+ + \frac{1}{2C}\Vert \mathbf{w} \Vert^2$$

Note that the regularization term $\Vert \mathbf{w} \Vert^2$ may just as well be added to the logistic regression objective - this will result in regularized logistic regression.

You do not have to limit yourself to $\ell_2$-norm as the regularization term. Replace it with $\Vert \mathbf{w} \Vert_1$ in the SVM objective, and you will get $\ell_1$-SVM. Add both $\ell_1$ and $\ell_2$ regularizers to get the "elastic net regularization". In fact, feel free to pick your favourite loss, add your favourite regularizer, and voila - help yourself to a freshly baked machine learning algorithm.

This is not a coincidence. Any machine learning modeling problem can be phrased as the task of finding a probabilistic model $M$ which describes a given dataset $D$ sufficiently well. One general method for solving such a task is the technique of maximum a-posteriori (MAP) estimation, which suggests you should always choose the most probable model given the data:

$$M^* = \mathrm{argmax}_M P(M|D).$$

Using the Bayes rule and remembering that $P(D)$ is constant when the data is fixed:

\begin{align*}
\mathrm{argmax}_M P(M|D) &= \mathrm{argmax}_M \frac{P(D|M)P(M)}{P(D)} \\
 &= \mathrm{argmax}_M P(D|M)P(M) \\
&= \mathrm{argmax}_M \log P(D|M)P(M) \\
&= \mathrm{argmax}_M \log P(D|M) + \log P(M) \\
&= \mathrm{argmin}_M (-\log P(D|M)) +  (-\log P(M))
\end{align*}

Observe how the loss turns out to be just another name for the (minus) log-likelihood of the data (under the chosen model) and the regularization penalty is the log-prior of the model. For example, the familiar $\ell_2$-penalty is just the minus logarithm of the Gaussian prior on the parameters:

$$ -\log\left((2\pi)^{-m/2}e^{-\frac{1}{2\sigma^2}\Vert \mathbf{w} \Vert^2}\right) = \mathrm{const} + \frac{1}{2\sigma^2}\Vert \mathbf{w} \Vert^2$$

Hence, another way to describe the difference between SVM and logistic regression (or any other model), is that these two postulate different probabilitic models for the data. In logistic regression the data likelihood is given via the Bernoulli distribution (with $p$=sigmoid), while the model prior is uniform (or simply ignored). In SVM the data likelihood is modeled via some $\mathrm{exp}(-\mathrm{hinge})$ distribution (not sure it even has a name, but I hope you get the idea that undoing the minus-logarithm would always bring you back to $P(D|M)$, up to a constant), and the model prior is the Gaussian.

In practice, the two models have different properties, of course. For example, SVM has sparse dual representations, which makes it possible to kernelize it efficiently. Logistic regression, on the other hand, is usually well-calibrated (which is not the case with SVM). Hence, you choose the model based on your needs (or, if you are unsure, on whatever cross-validation tells you).


finding optimal solution $w$ and classification accuracy Suppose you are given $6$ one-dimensional points: $3$ with negative labels $x_1 = −1$, $x_2 = 0$, $x_3 = 1$ and $3$ with positive labels $x_4 = −3$, $x_5 = −2$, $x_6 = 3$. In this question, we first
compare the performance of linear classifier with or without kernel. Then we solve for
the maximum margin classifier using SVM.
Consider a linear classifier of form $f(x) = sign(w_1x+w_0)$. Write down the optimal
value of $w$ and its classification accuracy on the above 6 points. There might be more than one optimal solution, writing down one of them is enough.

My attempt:
I understand that the data isn't linearly separable and that there will be some error, but I don't get how to get the optimal value of $w$. Do I minimize $f(x)$? But how do I take the derivative of $f(x)$? Any guidance would be appreciated, I'm a little lost.

The best that we can do is to classify $5$ points correctly and sacrifice one point.

We want to classify $-1,0,1$ as negative and $-3,-2$ as positivie.  (we have to sacrifice $3$).

The boundary with maximum margin would be in the middle of $-2$ and $-1$. That is $-\frac32$.

$$sign(-(2x+3))=sign(-2x-3).$$


VM image for data science projects As there are numerous tools available for data science tasks, and it's cumbersome to install everything and build up a perfect system.

Is there a Linux/Mac OS image with Python, R and other open-source data science tools installed and available for people to use right away? An Ubuntu or a light weight OS with latest version of Python, R (including IDEs), and other open source data visualization tools installed will be ideal. I haven't come across one in my quick search on Google.

Please let me know if there are any or if someone of you have created one for yourself? I assume some universities might have their own VM images. Please share such links.

There is another choice which popular recently: docker(https://www.docker.com). Docker is a container and let you create/maintain a working environment very easily and fast.

install essential tools for data science in python

https://registry.hub.docker.com/u/ceshine/python-datascience/

use r language to do data science

https://github.com/rocker-org/rocker

Hope that would help you.


NLP text autoencoder that generates text in poetic meter I would like to create an NLP autoencoder that happens to only generate text that conforms to a poetic meter, for example 'iambic pentameter'. That is, the output should be a series of clauses which are 10 syllables long and are read aloud with the verbal stresses as 'duh-DUH duh-DUH duh-DUH duh-DUH duh-DUH'. Shakespeare and his contemporaries used this format for much of their poetry.

The only method I can think of is to collect lines of text which are in this format, and then train an autoencoder on these. Since the network only emits iambic pentameter while training, it should also emit iambic pentameter while predicting. Given a large enough corpus, this seems like it should work.

One problem is that this training only accepts iambic pentameter. To rewrite other text into IP, the training needs to include variations of text in IP and train (not IP sentence) to emit (IP sentence). Generating these variations is straightforward. This crosses the line from training "sentence embeddings" to training "thought embeddings".

Are there other ways to solve this problem? Is there a way to directly alter the sentence embedding space? For example, variational sampling works by applying a nonlinear transformation on the embedding space.

Note: the CMU Pronouncing Dictionary supplies pronunciation stresses for over 100k words, usable for classifying meter:
http://www.speech.cs.cmu.edu/cgi-bin/cmudict

Note: this is a personal hobby project to teach myself deep learning & NLP- I really do not know whether it is achievable with current NLP technology.

A starting point that comes to mind is creating a cost function for a sentence being in IP. Now, while normally this is a binary affair (either a sentence is in IP or not - or so I would assume), this does not lend itself readily to the task. You should devise a cost  function that measures how close your sentence is to IP (so I assume that sentence with 11 syllables would have a lower cost than one with 12. Verbal stresses and their associated costs are left to you to figure out ;) ).

After you have a cost function, you should take some sort of pretrained deep neural network that can translate from one language to another, such as those that use Word2Vec, set it to translate from English to English, and train it with two costs - one for keeping the meaning of the text (probably with another instance of word2vec or something to that effect), the other for being in IP. The relative weights of the two costs should be determined experimentally (you may even want to change them while training).

This is of course a shot in the dark. More research and experimentation will be required, but I hope this may serve you as a starting point in your endeavor.

Be advised that while this is an intriguing project, in my opinion it might not be the best choice for beginners.


Confusion with using different classes in neural networks (training vs testing) I am new to deep learning and I am confused about having a neural network trained on certain classes and tested on different ones. Suppose I want to have a convolutional neural network that learns authorship attribution (Identifying the author of a certain text). For example, this model is trained on 10,000 authors/classes over hundreds of thousands of texts. How does the model generalize what it learnt to predict the author of a new author/class in a test set? Will it not be trained to identify the authors it was initially trained on and not the new one? For example, a CNN that learns whether an image is a cat or a dog is trained as such and when testing it predicts the same classes.

You are correct, most models in general don't perform well when asked to predict something they haven't seen before. Normally when you are splitting the data into Train and Test datasets, you want to include all examples in each dataset. In the train_test_split library, stratify is used to do this for you.
In the real world, sometimes the new class is not available during training, such as if you are predicting future data that may have a new label. When this occurs, typically your model won't categorize that correctly until you have built up several examples of that data and then rebuilt your model including these new examples in your dataset.
In the meantime, you can try to create a category named "other" and train your model to use this category when it can't categorize into a known class.


Distributed k-means in Spark I want to implement K-means algorithm in Spark. I am looking for a starting point and I found Berkeley's naive implementation. However, is that distributed?

I mean I see no mapreduce operations. Or maybe, when submitted in Spark, the framework actually makes the needed tricks under the hood to distribute the algorithm?

I also found that Spark shows mapreduce the exit and I am using Spark 1.6.

EDIT: This code produces a runtime error, check here.

In that link you posted, you can look at the python full solution here at the end and go through it to see what all is distributed.
In short, some parts are distributed, like reading data from the file, but the very important parts like the distance computation are not.

Running down, we see:

  sc = SparkContext("local[6]", "PythonKMeans")

This instantiates the context and creates a local cluster which the jobs will be submitted to

  lines = sc.textFile(..)

This is still setting up. No operations have taken place yet. You can verify this by putting timing statements in the code

  data = lines.map(lambda x: (x.split("#")[0], parseVector(x.split("#")[1])))

The lambda here will be applied to lines, so this operation will split the file in parallel. Note that the actual line also has a cache() at the end (see cache]). data is just a reference to the spark object in memory. (I may be wrong here, but I think the operation still doesn't happen yet)

  count = data.count()

This forces the parallel computation to start, and the count to be stored. At the end, the reference data is still valid, and we'll use it for further computations. I'll stop with detailed explanations here, but wherever data is being used is a possible parallel computation. The python code itself is single threaded, and interfaces with the Spark cluster.

An interesting line is:

  tempDist = sum(np.sum((centroids[x] - y) ** 2) for (x, y) in newCentroids.iteritems())

centroids is an object in python memory, as is newCentroids. So, at this point, all computations are being done in memory (and on the client, typically clients are slim, i.e. have limited capabilities, or the client is an SSH shell, so the computers resources are shared. You should ideally never do any computation here), so no parallelization is being used. You could optimize this method further by doing this computation in parallel. Ideally you want the python program to never directly handle individual points' $x$ and $y$ values.


Why this TensorFlow Transformer model has Linear output instead of Softmax? I am checking this official TensorFlow tutorial on a Transformer model for Portuguese-English translation.
I am quite surprised that when the Transformer is created, their final output is a Dense layer with linear activation, instead of Softmax. Why is that the case? In the original paper Attention is All You Need the image is pretty clear, there is a Softmax layer just at the end (Fig.1, p. 3).
How can you justify this difference, when your task involves building a language model and your Loss is based on sparse categorical crossentropy?

The key is precisely in the definition of the loss:
loss_object = tf.keras.losses.SparseCategoricalCrossentropy(
    from_logits=True, reduction='none')

As you can see, the loss is created with the flag from_logits=True which means that the input to the loss is not a probability distribution, but unnormalized log probabilities, namely "logits", which is precisely the result of the final projection, before any softmax.
When from_logits is true, the softmax itself is handled inside the loss, combining it with the sparse categorical cross-entropy into a more numerically stable form.
From the docs:

from_logits:  Whether y_pred is expected to be a logits tensor. By default, we assume that y_pred encodes a probability distribution.
Note - Using from_logits=True may be more numerically stable.


Can we quantify how position within search results is related to click-through probability? Suppose, for example, that the first search result on a page of Google search results is swapped with the second result. How much would this change the click-through probabilities of the two results? How much would its click-through probability drop if the fifth search result was swapped with the sixth?

Can we say something, with some level of assurance, about how expected click-through probabilities change if we do these types of pairwise swaps within pages of search results?

What we seek is a measure of the contribution to click-through rates made specifically by position bias.

Likely, how position ranking would affect the sales in Amazon or other online shopping website? If we cast the sales into two parts, the product quality and its ranking effect.

sales = alpha*quality + beta*position + epsilon

How can we quantify the beta?

you might want to look at this paper
Predicting Clicks: Estimating the Click-Through Rate for New Ads

  Whenever an ad is displayed on the search results page, it has
  some chance of being viewed by the user. The farther down the
  page an ad is displayed, the less likely it is to be viewed. As a
  simplification, we consider the probability that an ad is clicked on
  to be dependent on two factors: a) the probability that it is viewed,
  and b) the probability that it is clicked on, given that it is viewed:


How to execute a git project with different input data? I would like to run this project and receive result from this project

However I have problem to execute the step

  mailcorpus.json created by executing the sql script
  data/get_mail_corpus.sql on the Apache database

because this is the sql query

  https://github.com/collab-uniba/Personality-Recognition-in-SD/blob/master/data/get_mail_corpus.sql

SET group_concat_max_len=18446744073709547520;

SELECT

CONCAT(

'[',

GROUP_CONCAT(

JSON_OBJECT(

'id',messages.message_id,

'mailing_list_url',messages.mailing_list_url,

'type_of_recipient',messages_people.type_of_recipient,

'email_address',messages_people.email_address,

'message_body',messages.message_body,

'is_response_of',messages.is_response_of

) SEPARATOR ',\r'),

']') AS list

FROM messages

LEFT JOIN messages_people

ON messages.message_id = messages_people.message_id

AND messages.mailing_list_url = messages_people.mailing_list_url

WHERE email_address IN ('dr@gorsvet.kz', 'mathews@uk2.net', 'd.jentsch@fu-berlin.de', 'atul.soman@microfocus.com', 'kkrugler_lists@transpac.com', 'hcorg@minions.org.pl', 'bruno.ronchetti@mac.com', 'christophe.thiebaud@sap.com', 'kc.baltz@copart.com', 'havanki4j@gmail.com', 'joseph.obernberger@ngc.com', 'goran@roseen.se', 'hboutemy@apache.org', 'johann@gyger.name', 'fredrik@jonson.org', 'jose-marcio.martins@mines-paristech.fr', 'dmytro.kostiuchenko@gmail.com', 'jochen.wiedmann@softwareag.com', 'guyskk@qq.com', 'alejandroscandroli@gmail.com', '1983-01-06@gmx.net', 'jstrayer@proofpoint.com', 'daniel@degu.cl', 'hontvari@flyordie.com', 'david@andl.org', 'gholmes@pinsightmedia.com', 'dennis.geurts@luminis.eu', 'carlspring@gmail.com', 'jozef.koval@protonmail.ch', 'gabriele.ctn@gmail.com', 'ashish.disawal@evivehealth.com', 'aarslan2@anadolu.edu.tr', 'kshukla@yahoo-inc.com', 'alex.huang@gmail.com', 'markus@openindex.io', 'marohn@sipgate.de', 'ea@apache.org', 'krzysztof.szalast@gmail.com', 'kfoskey@tpg.com.au', 'krauss@in.tum.de', 'kenhans@hotmail.co.uk', 'jokin.c@odeian.com', 'eitch@eitchnet.ch', 'm.kristian@web.de', 'bob.sandiford@sirsidynix.com', 'asandstrom2@eastlink.ca', 'kelly@apache.org', 'antony.blakey@linkuistics.com.au', 'ashawki@hotmail.com', 'h.klocker-mark@tsn.at');

but the database is not available to make the query. How is it possible to use my own input and take the rinal results?

Apparently the program expects a JSON file (probably) like this:

[
    {
    "id": ".......",
    "mailing_list_url": "......",
    "type_of_recipient": "......",
    "email_address": ".......",
    "message_body": "......",
    "is_response_of": "......"
    },
    {
    "id": ".......",
    "mailing_list_url": "......",
    "type_of_recipient": "......",
    "email_address": ".......",
    "message_body": "......",
    "is_response_of": "......"
    },

    ...

    {
    "id": ".......",
    "mailing_list_url": "......",
    "type_of_recipient": "......",
    "email_address": ".......",
    "message_body": "......",
    "is_response_of": "......"
    }
]

So if your data can be somehow converted to this that would be ideal, I guess. Otherwise you might have to play with the code.

Alternatively you could contact the authors by opening an issue.


What do you call the ratio of positive to negative samples? I am working with a binary classifier and I want to express the "balance" or "skewness" of the training data using a metric.

I want to reflect this ratio in a report, like this:

Accuracy: 80%
Recall: 78%
Precision: 62%
*The Ratio of Positive to Negative Samples*: 62%

I feel like there's probably a standard name for the metric expressing "The Ratio of Positive to Negative Samples".

My main question is: What is the name of this metric?

(This question assumes that there is a standard name for this metric.)

Extra Information:

Here are some example values for the metric and their interpretations:

1.0 = The sample (of training data) is balanced.
0.5 = There are twice as many negative samples as positive samples.
2.0 = There are twice as many positive samples as negative samples.

My secondary question is: Is there another metric that can be used to form similarly meaningful interpretations?

You can use class ratio / sample class ratio. Which will make it more intuitive for any reader while going through the details.

As its not used for model performance analysis hence I think we don’t have a metric name for this.


Analyzing survey data for predictions I've got survey data that resembles:

|-------------| Q1a | Q1b | Q1c | Q2a | Q2b | Q2c | Classification
| Respondent  | 1   | 0   | 0   | 1   | 0   | 0   | Red
| Respondent  | 0   | 0   | 1   | 1   | 0   | 0   | Green
| Respondent  | 0   | 1   | 0   | 0   | 0   | 1   | Yellow

I am trying to predict the classification for new respondents. Currently I'm using a Naive Bayes, and getting pretty bad accuracy (~20%). I don't have much training data, and the training data is hand scraped from non-standard sources (internal company procedures are a mess here).

I'm looking for other ways to predict the classification.

I'm thinking about assigning weights to each question, and magically predicting the result based on those, somehow. Although I don't really know where to start learning about how to do that, and whether it's appropriate for this data. I have very little background in this :(

Any ideas or tips on predicting the classification column with no training data?

Can you give a bit more information on the size of the data you're training on (and if it's really 6 parameters you're basing the predictions off of)?  If it's really 6 questions with binary answers (1, 0 as you suggest), then there are 2^6 (i.e. 64) unique answer combinations, and to determine a probability for them you'll want a multiple entries per combination.  Standard error scales like 1/sqrt(n) so for 10% accuracy you'll need roughly 6,400 inputs which given your description, sounds like more data than you may have.  You may want to invest time into automating data collection.

If on the other hand, you have a reasonably large data set and are hoping for some alternative models, both boosted decision trees and random forest models sound like good candidates for this problem.


How to create a multi label classification network in Keras if I have the training data with various accuracy? I'm trying to create a neural network that finds the most effective treatment for each patient.
I have a medical database for training. The inputs are histological and pathological data (mostly 0/1 data of having some diseases, conditions) and the outputs are the treatments. And there is an accuracy score, which is the patient's reaction to the treatment (this shows how good was the treatment for the symptoms, so how accurate was the input-output pair).
How can I create a multi label classification network that finds the best treatment if the training data pairs are not always totally correct, rather having these variable accuracy scores?

One approach is to have a dummy class that represents no treatment and use the accuracy score via a threshold (lower than threshold) correspond to no treatment at all.
Threshold as used above becomes a new hyper-parameter and you have new input-output pairs that are now exact (depending on threshold).


How do I use a custom stopwords filter in the Java Weka API? I am using the Java Weka API to build a classification model. I can use the builtin stopwords filter. However, I need to use a custom filter for my problem. I do not know how to use a custom stopwords filter in the Java Weka API.

First of all you have to prepare a text file for your custom stopwords. Then you can use the following code:

import weka.filters.unsupervised.attribute.StringToWordVector;

StringToWordVector filter = new StringToWordVector(10000);
filter.setStopwords(new File(".../stopwords.txt"));

Hope it will help you.


Is fitting two RandomForestClassifiers 500 trees each and average their predicted probabilities on the test set more performant than one with 1000? If I fit two RandomForestClassifiers 500 trees each and average their predicted probabilities on the test set, would it have better results than fitting a RandomForestClassifier with 1000 trees and use it to get test set probabilities?
As these algorithms are random based I would say that their performance should be roughly the same?
I am okay with some math to prove it, or any other way that might prove it.

With a toy dataset, I obtained slightly better results with the RF 1000 trees
import pandas as pd
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

from scipy.stats import ks_2samp

import matplotlib.pyplot as plt
plt.style.use("seaborn-whitegrid")

n_models = 2
threshold = .5

X, y = load_breast_cancer(return_X_y= True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 42, test_size = .3)

rForest1000 = RandomForestClassifier(n_estimators= 1000, random_state= 42, oob_score= True).fit(X_train, y_train)
preds1000 = rForest1000.predict_proba(X_test)[:,1]

roc_score = roc_auc_score(y_true = y_test, y_score= preds1000)

print(f"Test set score for 1K trees is : {round(roc_score, 4)}")

ls = list()
for i in range(n_models):
    model = RandomForestClassifier(n_estimators= 500, random_state= i).fit(X_train, y_train)
    ls.append(model.predict_proba(X_test)[:,1])
preds = np.array(ls).mean(axis =0)

roc_score = roc_auc_score(y_true = y_test, y_score= preds)

print(f"Test set score for 500trees X 2 avg is : {round(roc_score, 4)}")

fig, ax = plt.subplots(1,2, figsize = (12,5))

ax[0].hist(preds1000[y_test == 0], color = "darkgreen", alpha = .5)
ax[0].hist(preds1000[y_test == 1], color = "darkred", alpha = .5)

ax[0].set_title(f"Predictions distribution with 1K Trees RF\n KS: {[round(ks_2samp(preds1000[y_test == 0], preds1000[y_test == 1])[0],3)]}")

ax[1].hist(preds[y_test == 0], color = "darkgreen", alpha = .5)
ax[1].hist(preds[y_test == 1], color = "darkred", alpha = .5)

ax[1].set_title(f"Predictions distribution with X2 500 Trees RF Trees RF\n KS: {[round(ks_2samp(preds[y_test == 0], preds[y_test == 1])[0],3)]}");


Same confusion matrix when changing DecisionTreeClassifier parameters I'm trying to build my first Decision Tree Classifier using the Iris dataset in the sklearn library. This is my first sample code:
from sklearn.datasets import load_iris
from sklearn.metrics import accuracy_score
from sklearn.metrics import f1_score
from sklearn.model_selection import cross_val_score
from sklearn.metrics import confusion_matrix
from sklearn import tree
import numpy as np
import graphviz

iris = load_iris()

clf_ex1 = tree.DecisionTreeClassifier(criterion="entropy",random_state=300,min_samples_leaf=5,
                                      class_weight={0:1,1:10,2:10})

np.random.seed(0)

indices = np.random.permutation(len(iris.data))
indices_training=indices[:-10]
indices_test=indices[-10:]

iris_X_train = iris.data[indices_training]
iris_y_train = iris.target[indices_training]
iris_X_test  = iris.data[indices_test]
iris_y_test  = iris.target[indices_test]

clf_ex1 = clf_ex1.fit(iris_X_train, iris_y_train)

predicted_y_test = clf_ex1.predict(iris_X_test)

print(confusion_matrix(iris_y_test, predicted_y_test))

print("Predictions:")
print(predicted_y_test)
print("True classes:")
print(iris_y_test)
print("--------")
print(iris.target_names)

# print some metrics results
acc_score = accuracy_score(iris_y_test, predicted_y_test)
print("--------")
print("Accuracy score: "+ str(acc_score))
print("--------")
f1=f1_score(iris_y_test, predicted_y_test, average='macro')
print("F1 score: "+str(f1))
print("--------")

scores = cross_val_score(clf_ex1, iris.data, iris.target, cv=5)
print(scores)

dot_data = tree.export_graphviz(clf_ex1, out_file=None,
                         feature_names=iris.feature_names,
                         class_names=iris.target_names,
                         filled=True, rounded=True,
                         special_characters=True)
graph = graphviz.Source(dot_data)
graph

As you can see, in my DecisionTreeClassifier function I set the weight of the classes by giving a major value to the second one and the third one and I've given 300 to the random_state parameter. Then, I made another example by changing these parameters on this way:
clf_ex2 = tree.DecisionTreeClassifier(criterion="entropy",random_state=300,min_samples_leaf=5,
                                  class_weight={0:1,1:1,2:10})

and on this way:
clf_ex3 = tree.DecisionTreeClassifier(class_weight=None, criterion='entropy',
                                  max_depth=2,
                                  max_leaf_nodes=None,
                                  min_samples_leaf=15,
                                  min_samples_split=5,
                                  random_state=100,
                                  splitter='best')

The problem is that all the values that I print (the confusion matrix, the accuracy, the predicted_y_test and the f1 score) do not change between the three codes. The only value that gets affected is the Cross Validation Score. Why?

I ran your script and this is what was returned for all 3 confusion matrices:
[[4 0 0]
 [0 3 1]
 [0 0 2]]

This confusion matrix indicates to me that your model is working wonderfully on the testing data. There is a single exception where your model predicts the wrong class (the non-diagonal "1").
Since you are giving the same random seed every time, your training and testing indices are the same for all 3 models you are investigating. Meaning: they are all being trained on the same data and tested on the same data.
It is quite possible that, in this test data set, 9 out of 10 samples are very easy to classify, while there is 1 case where it is much more non-trivial. The contrast between trivial and non-trivial might be so stark that adjusting hyperparameters in your DecisionTree isn't yielding any difference in performance.
So this could explain why your output is the same for all those fields you mentioned.
As for the cross_val_score - this method randomly mixes your entire dataset and gives you 5 successive CV trials. As this process is actually random from your first model through to your third (it doesn't depend on your np.random.seed(0)), the splits are different and you will yield similar (but not identical) results.
As a general note - be careful how you're currently using cross_val_score. In your current implementation you are feeding a pre-trained model to a method where it expects an un-trained model, and you are asking it to cross validate. As a result it's likely that cross_val_score is re-training your model in a way that you might not be intending.


Get bounding boxes for adjacent instances of a single class in image I have a dataset with thousands of music score pages and manually annotated bounding boxes for the individual bars:

My objective is now to train a DNN that should ultimately be able to get these bounding boxes on its own. First idea was to use something like the Region Proposal Network (RPN) from Faster R-CNN on top of ResNet or VGG, but I am unsure if this still works because the "objectness" is rather high for almost each section of the page. Plus the regions are mostly touching each other but rarely overlap. Number of bars is roughly somewhere between 1 and 250 per page.

Additionally, the number of systems (=rows of bars) per page is oftentimes not changing between subsequent pages. This might be a very helpful info that RPN would miss. Maybe introduce some sort of recurrency?

Is there anything out there that would be more tailored to my specific problem? Any advise on a better fitting architecture or further tweaks would be highly appreciated.

EDIT:
Some more extreme examples:

My first thought would be not to full deep learning on this - It is hard to see but it looks like your regions are bound by vertical lines with many horizontal ones spanning those regions. You can try doing just simple canny filters to detect those lines (maybe with [opencv] - [https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_houghlines/py_houghlines.html]), then find the points where horizontal and vertical lines intersect to form vertical bounds for regions.

Another idea that may help is the sweep-plane algorithm,:[https://scicomp.stackexchange.com/questions/8895/vertical-and-horizontal-segments-intersection-line-sweep]
I am just spitballing here, but where notes and horizontals meet will form connected regions. Finding connected regions that contain horizantal lines gets you part of the way. Then slicing those with the output of the vertical line detector (maybe it is a search over similar length groups starting with longest vertical using the tree-based strategy of sweep-plane) is worth a try.

On the lines of the RPN, I have had good experience with SSD for a similar problem (detecting individual drawings on an architectural drawing). SSD differs in that it returns something like 8K proposals with confidences, and then a second pass of tuning the confidence threshold and finding non-overlapping regions got me pretty close, but my intuition says that your dataset is structured enough to have another answer.

I am curious how many pages you have in the dataset. If you have less than a few thousand annotated lets say, it may be harder to train a big neural net, and would lean toward the canny filter/hough transform direction. Also are the pages that are annotated represent a diverse enough sample of the production data?


Back propagation on matrix of weights I am trying to implement a Neural Network for binary classification using python and numpy only.

My network structure is as follows:
input features: 2 [1X2] matrix
Hidden layer1: 5 neurons [2X5] matrix
Hidden layer2: 5 neurons [5X5] matrix
Output layer: 1 neuron [5X1]matrix
I have used the sigmoid activation function in all the layers.

Now lets say I use binary cross entropy as my loss function. How do I do the back propagation on these matrices to update weights?

class Layer():
    def __init__(self,number_of_neurons,number_of_inputs):
        self.weights=np.random.rand(number_of_neurons, number_of_inputs)
        self.bias=np.random.rand(number_of_neurons,1)

class NeralNetwork():
        def __init__(self, layer1, layer2,layer3):
            self.layer1 = layer1
            self.layer2 = layer2
            self.layer3 = layer3

    def sigmoid(self,x):
        return 1 / (1 + np.exp(-x))

    def derivative_sigmoid(self,x):
        return x*(1-x)

    def get_cost_value(self,Y_hat, Y):
        m = Y_hat.shape[1]
        cost = -1 / m * (np.dot(Y, np.log(Y_hat).T) + np.dot(1 - Y, np.log(1 - Y_hat).T))
        return np.squeeze(cost)

    def get_cost_derivative(self,Y_hat,Y):
        return  - (np.divide(Y, Y_hat) - np.divide(1 - Y, 1 - Y_hat))

    def train(self,inputs,labels,epocs):
        for epoc in range(1,epocs+1):
            z1=np.dot(self.layer1.weights,inputs)+self.layer1.bias
            a1=self.sigmoid(z1)
            z2=np.dot(self.layer2.weights,a1)+self.layer2.bias
            a2=self.sigmoid(z2)
            #print(a2.shape)
            z3=np.dot(self.layer3.weights,a2)+self.layer3.bias
            a3=self.sigmoid(z3)
            #print(a3.shape)
            if epoc%100 is 0:
                print(a3)

            cost=self.get_cost_value(a3,labels)
            #print(cost)

            layer3_delta=self.derivative_sigmoid(a3)*self.get_cost_derivative(a3,labels)
            print(layer3_delta.shape)
            Dw_layer3=np.dot(layer3_delta,a2.T)
            Db_layer3=layer3_delta
            #print(Dw_layer3.shape)
            layer2_delta=np.dot(self.layer3.weights.T,layer3_delta)*self.derivative_sigmoid(a2)
            #print(layer2_delta.shape)
            Dw_layer2=np.dot(layer2_delta,a1.T)
            Db_layer2=layer2_delta

            layer1_delta=np.dot(self.layer2.weights.T,layer2_delta)*self.derivative_sigmoid(a1)
            Dw_layer1=np.dot(layer1_delta,inputs.T)
            Db_layer1=layer1_delta
            #print(Dw_layer1)

            self.layer1.weights-=((1/epoc)*Dw_layer1)
            self.layer2.weights-=((1/epoc)*Dw_layer2)
            self.layer3.weights-=((1/epoc)*Dw_layer3)

            self.layer1.bias-=((1/epoc)*Db_layer1)
            self.layer2.bias-=((1/epoc)*Db_layer2)
            self.layer3.bias-=((1/epoc)*Db_layer3)

So far I have tried to implement this as shown above. But there seems to be a mistake because, after training, the network doesn't seem to have learned. Please let me know if you have any inputs.

Your question needs a wide explanation.

  I suggest to go through this YouTube video DNN Backpropagation
  It will explain how backpropagation
  works in case of DNN.