Which learning rate should I choose? I'm training a segmentation model, Unet++, on 2d images and I am now trying to find the optimal learning rate.
The backbone of the model is Resnet34, I use Adam optimizer and the loss function is the dice loss function.
Also, I use a few callbacksfunctions:
callbacks = [
            keras.callbacks.EarlyStopping(monitor='val_loss', patience=15, verbose=1, min_delta=epsilon, mode='min'),
            keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=3, verbose=1, mode='min', cooldown=0, min_lr=1e-8),
            keras.callbacks.ModelCheckpoint(model_save_path, save_weights_only=True, save_best_only=True, mode='min'),
            keras.callbacks.ReduceLROnPlateau(),
            keras.callbacks.CSVLogger(logger_save_path)
        ]

I plotted the curves of training loss over epochs for a few learning rates:

The validation loss and training loss seem to decrease slowly. However, the validation loss isn't oscillating (it is almost always decreasing).

The validation and training losses decreased quickly on first 2/3 epochs. After 6 or 7 epochs, the validation loss increases again.

I have a few questions (I hope it is not too much):

*What is normally the best way to find the learning rate i.e.

How many epochs should I wait before considering that the learning rate isn't good?
What are the criteria on the loss function to determine if a learning rate is "good"?

Is there a big difference if I use a small learning (which still converges) instead of the "optimal" learning rate ?
Is it normal that the validation loss function oscillates over the training?
Which learning rate should I use according to my results?

Even a partial response would help me a lot.

I am afraid the that besides learning rate, there are a lot of values for you to make a choice for over a lot of hyperparameters, especially if you’re using ADAM Optimization, etc.
A principled order of importance for tuning is as follows

Learning rate
Momentum term , num of hidden units in each layer, batch size.
Number of hidden layers, learning rate decay.

To tune a set of hyperparameters, you need to define a range that makes sense for each parameter. Given a number of different values you want to try according to your budget, you could choose a hyperparameter value from a random sampling.
Specifically to learning rate investigation though, you may want to try a wide range of values, e.g. from 0.0001 to 1, and so you can avoid sampling random values directly from 0.0001 to 1. You can instead go for $x=[-4,0]$ for $a=10^x$ essentially following a logarithmic scale.
As far as number of epochs go, you should set an early stopping callback with patience~=50, depending on you "exploration" budget. This means, you give up training with a certain learning rate value if there is no improvement for a defined number of epochs.
Parameter tuning for neural networks is a form of art, one could say. For this reason I suggest you look at basic methodologies for non-manual tuning, such as GridSearch and RandomSearch which are implemented in the sklearn package. Additionally, it may be worth looking at more advanced techniques such as bayesian optimisation with Gaussian processes and Tree Parzen Estimators. Good luck!

Randomized Search for parameter tuning in Keras

Define function that creates model instance

# Model instance
input_shape = X_train.shape[1]

def create_model(n_hidden=1, n_neurons=30, learning_rate = 0.01, drop_rate = 0.5, act_func = 'ReLU',
                act_func_out = 'sigmoid',kernel_init = 'uniform', opt= 'Adadelta'):
    model = Sequential()
    model.add(Dense(n_neurons, input_shape=(input_shape,), activation=act_func,
                   kernel_initializer = kernel_init))
    model.add(BatchNormalization())
    model.add(Dropout(drop_rate))
    # Add as many hidden layers as specified in nl
    for layer in range(n_hidden):
    # Layers have nn neurons model.add(Dense(nn, activation='relu'
        model.add(Dense(n_neurons, activation=act_func, kernel_initializer = kernel_init))
        model.add(BatchNormalization())
        model.add(Dropout(drop_rate))
    model.add(Dense(1, activation=act_func_out, kernel_initializer = kernel_init))
    opt= Adadelta(lr=learning_rate)
    model.compile(loss='binary_crossentropy',optimizer=opt, metrics=[f1_m])
    return model

Define parameter search space

params = dict(n_hidden= randint(4, 32),
              epochs=[50], #, 20, 30],
              n_neurons= randint(512, 600),
              act_func=['relu'],
              act_func_out=['sigmoid'],
              learning_rate= [0.01, 0.1, 0.3, 0.5],
              opt = ['adam','Adadelta', 'Adagrad','Rmsprop'],
              kernel_init = ['uniform','normal', 'glorot_uniform'],
              batch_size=[256, 512, 1024, 2048],
              drop_rate= [np.random.uniform(0.1, 0.4)])

Wrap Keras model with sklearn API and instantiate random search

model = KerasClassifier(build_fn=create_model)
random_search = RandomizedSearchCV(model,  params, n_iter=5, scoring='average_precision',
                                   cv=5)

Search for optimal hyperparameters

random_search_results = random_search.fit(X_train, y_train,
                                          validation_data =(X_test, y_test),
                                          callbacks=[EarlyStopping(patience=50)])


ExtraTreesRegressor criterion As I understand, ExtraTreesRegressor from sklearn works by doing random splits instead of minimizing a metric like gini for classification or mae for regression.
I don't understand why there's a criterion parameter, as the criterion for the splits should be random.
Is it just for code compatibility, or am I missing something?

No, extremely-random trees does still optimize splits.  It does only pick one random splitting point for each feature (out of those randomly chosen max_features) but then which feature is actually used for the split depends on the criterion chosen.
https://scikit-learn.org/stable/modules/ensemble.html#extremely-randomized-trees


Can i build an image classification model where each image has multiple labels? If I am building a model where I need to predict the vehicle, color of it, and make of it, then can I use all the labels for a single image and build my model around it.

Like for a single image of a vehicle which is a car (car1.jpg) will have labels like - Sedan(Make), Blue(Color) and Car(Type of vehicle). Can I make a single model for this or I will have to make 3 separate models for this problem.

One model will work in this case.
Lets put it this way, let's say we are trying to solve multi-class(lets say 5) classification problem in neural networks we will have 5 neurons in the final dense layer and ideally use softmax as activation function and categorical_crossentrophy as loss function.
Now coming back to multi-label classification lets take an example:
Your car1 has labels like Sedan,Blue,and Type(Good).
Your car2 has labels like SUV,Black, and type(Bad).
You will these many neurons in your final layer lets say here we have 6 different types of labels for 2 cars, and we do one hot encoding for the target variables.

The difference is we use sigmoid and binary_crossentrophy as activation and loss functions. This will give probabilities of the labels for which each image belong to.
Any questions, please post.


Is it possible feed BERT to seq2seq encoder/decoder NMT (for low resource language)? I'm working on NMT model which the input and the target sentences are from the same language (but the grammar differs). I'm planning to pre-train and use BERT since I'm working on small dataset and low/under resource language. so is it possible to feed BERT to the seq2Seq encoder/decoder?

Sure, why not? An encoder/decoder is basically agnostic to the format of the token vectors, whether they be derived via Word2Vec, BERT, GPT2, etc.

The more challenging aspect of this, should you not have figured it out already, might be finding a pretrained embedding model for your low resource language. Given a small dataset, training your own from scratch seems infeasible. You could potentially find a BERT pretrained over a similar language (e.g. within the same family or grammatical structure) and fine-tune the last layer according to your dataset. Conventional BERT is trained over the English language, with some BERTs spinning off in other languages.


Plotting multivariate linear regression For practicing linear regression, I am generating some synthetic data samples as follows.
First it generates 2000 samples with 3 features (represented by x_data). Then it generates y_data (results as real y) by a small simulation. i.e. by assuming a linear dependence model: imaginary weights (represented by w_real), bias (represented by b_real), and adding some noise.
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

#create some test data and simulate results
x_data = np.random.randn(2000,3)
w_real = [0.3,0.5,0.1]
b_real = -0.2

noise = np.random.randn(1,2000)*0.1
y_data = np.matmul(w_real,x_data.T) + b_real + noise

print(len(x_data))
print(len(y_data[0]))

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

x1 = x_data[:,0]
x2 = x_data[:,1]
x3 = x_data[:,2]
ax.scatter3D(x1, x2, x3, c=x3, cmap='Greens');

plt.show()

#actual implementation of liner regression
#compute y_pred, compare with y_data above etc etc
#assume more code here

exit()

I am trying to visualize above simulated samples (x_data and y_data) using matplotlib. I was able to plot x_data as displayed in following image. I would also like to visualize simulated results (y_data) on this plot, may be with a different color. Motivation behind it is to visualize relation between x & y. How can I plot it so?
Data dimensions:

x_data: $2000 \times 3$
y_data: $2000 \times 1$

Here are how sample data is displayed by above sample,

Here is an example of what I would like to achieve. The image shows two different hypothesis represented by straight lines, in my case I would like to draw a single line representing y_data.

With more than two variables, you have a dimension problem. Here, with 3 variables and one output you would need a 4 dimensionnal graph, which is not possible unless you use some trick.

1. Reduce the dimension of your problem

Generally speaking, if you need to observe a problem for which the dimension is too big, you may want to reduce its dimension. Observe relations with only one or two variable. Of course this means that you will have some difficulties to to observe more complex relationships.

For your exemple, that would mean plotting independently for (X1,X2), (X2,X3) and (X1,X3):

ax.scatter(x1, x2, y_data[0], c=y_data[0], cmap='viridis');

To be honest, this is not ideal as some point may recover others. This can be adressed by adding some transparancy to the points (parameter alpha), but it doesn't improve the visualisation that much. I would recommend to start with 1D plot (y against another variable), to really understand what is happening:

plt.scatter(x1, y_data[0], c=y_data[0], cmap='viridis');

2. Use color and make the graph interactive

One way to add a 4th dimension to graph is to make the use of color. It has some limitations (you need a good color scale : one that would still render if printed in B&W, one that is color-blind friendly). Indeed, It won't apply to more than 3 variables.

For your exemple, that would mean something like :

ax.scatter3D(x1, x2, x3, c=y_data[0], cmap='viridis');

This faces the readability problem as above (but I find it better as the colour bring some information instead of repeating what is on the vertical axis).

An option is to make the graph interactive, with something like plotly. (More info here: https://plot.ly/python/3d-scatter-plots/)

3. use contour curves

Another approach for adding a dimension to a graph would be to plot contour curves, which represent an ensemble of X values that give the same y). Note that you won't get any "single line representing y_data". Generally speaking I am quite sure this would not render well in 3D (plotting an ensemble of 3D curves), except maybe for your linear regression problem (you would get an ensemble of 3D plane). Again, the main option is to plot a reduced version of your problem, i.e. 2D plots with 2D contour curves.

One main requirement of this approach is that you need to provide the relationships between X and y, which is unknow. So you have to build a model and adapt it to what you want to plot.

For linear regression you would get something like :

Get the estimated model :

w_est = [0.29,0.51,0.09]
b_est = -0.19

def output_X1_X2(X1, X2):
    return X1*w_est[0] + X2*w_est[1] + 0 * w_est[2] + b_est

Set the values for plotting :

x1_plot = np.linspace(-3, 3, 50)
x2_plot = np.linspace(-3, 3, 50)

X1_plot, X2_plot = np.meshgrid(x1_plot, x2_plot)
Y = output_X1_X2(X1_plot, X2_plot)

Plot the output and associated contour :

contours = plt.contour(X1_plot, X2_plot, Y, 20, colors='black')
plt.clabel(contours, inline=True, fontsize=8)
plt.imshow(Y, extent=[0, 3, 0, 3], origin='lower',
           cmap='viridis')
plt.colorbar();

You get a graph with different value of y for X1 and X2. The main drawbacks are : you don't see the interaction with X3, you have to set a given X3 (0 here). Meaning that you have to plot similar graphs with (X2,X3) and (X1,X3), plus you have to make the set aside variable move to value other than 0. Even if this could be automated, it can rapidly be a pain with lots of variables.


Recommendation Engine - Content based and Collaborative recommendation? I am building a recommendation system for hotel accommodation. I scraped data from online booking portal and now my data has Name of the hotel, review, description and location.

I built a simple content-based recommendation. I combined the hotel name, location and description. Then, used NLP and TF-IDF to convert into vectors. I used cosine similarity for recommending the hotel name.

I have taken a hotel name as an input and model recommends the hotels. I have the following questions,

1) If I wanted to find hotel recommendations based on keywords instead of hotel names, do you have any suggestions on how I can do it? For example, Instead of giving hotel name, can I give the location name or typing IBIS (IBIS Singapore - > is a complete hotel name)

2) Second, I don't have a hotel ID, how to do Collaborative recommendation system with my present dataset?

Please advise

Not sure what your production requirements are for your collaborative filtering system, but from a research standpoint, I have had a good experience with the Surprise Package for Python.


How to perform feature engineering on unknown features? I am participating on a kaggle competition. The dataset has around 100 features and all are unknown (in terms of what actually they represent). Basically they are just numbers.

People are performing a lot of feature engineering on these features. I am wondering how exactly one is able perform feature engineering on features which are unknown? Can somebody please help me understand this and some tips on how can I perform feature engineering on unknown features?

You do not need domain knowledge (the knowledge of what your data mean) in order to do feature engineering (finding more expressive ways of framing your data).

As Tu N. explained, you can find "quick and dirty" combinations of features that could be helpful pretty easily. Given an output $y$ and an individual feature $x$, you can take the following transforms, $x' \in \{e^x, \log(x), x^2, x^3, \tanh(x)\}$. A quick check of the usefulness of the transformation is if the correlation between $\{y,x'\}$ is higher than the correlation between $\{y,x\}$.

Warning on correlation: Correlation does not show everything and depending on the model you are using (highly non-linear such as NN or RF) and interaction with other variables, a change in correlation could mean nothing.

However, if you are using a simple linear model like logistic regression, it is an OK indicator of perfomance. The best way to evaluate such a transformation, however, as noted by Fokhruz Zaman, would be to build a model with and without your transformed feature, and see how the validation error (on your Cross-Validation folds) evolves.

It is rather easy to spot single-feature transformations this way. Those apply to a lot of data, where a more expressive relation between your input and output could be on a different scale. For exemple the relationship between Income and "Happiness" appears to be logarithmic, but you would never record the log of a participant income directly.

Finding combinations of feature is more difficult. For a start, if you want to test every addition of 2 features, and you have $D$ features, you have an order of $D^2$ transformations to test. In order to find such transformations, you can apply a nonlinear model (such as NN or RF) to the problem and try to see what it is that it is learning. If you can identify what an intermediate layer in a NN is doing, you can pre-compute its result and add it as a new feature. It will not need to compute it again, and it will probably try to learn something new.

It can be hard to interpret the internal representation of a NN, or even interpret feature importance in a Random Forest. An easier, and probably more suited method for this purpose, model would be Boosting with decision trees. There are a lot of libraries implementing Boosting, and if you are into Kaggle competition as your post seem to imply, XGBoost seems used by a lot of participants, so you might find some help/tutorials on what I'm going to describe.

First, run your boosting algorithm using only stumps, 1-level decision trees. Stumps are very weak, but Boosting makes it a reasonnable model. This will act as your baseline. Depending on the library you are using, you should be able to display pretty easily which are the most used features, and you should plot them against the response (or do an histogram if the response is categorical) to identify some pattern. This might give you an intuition on what would be a good single feature transformation.

Next, run the Boosting algorithm with 2-level decision trees. This model is a lot more complex than the previous one; if two variables taken together have more power than taken individually, this model should outperform your previous one (again, not in term of training error, but on validation error!). Based on this, you should be able to extract which variable are often used together, and this should lead you to potential multi-feature transformations.

On related material, I would advise the following videos as they are easy to follow

Data Agnosticism: Feature Engineering Without Domain Expertise; SciPy 2013 Presentation: An example of how "simple" feature engineering can be.
Peter Prettenhofer - Gradient Boosted Regression Trees in scikit-learn: The speaker touches quicly on analysing feature importance at the end.


Change raw count to a proportion based on a filter in Tableau In Tableau, I commonly want to carry out a task like distinguishing 2 groups by various measures.

The example below shows the difference in the count of 2 treatment and control for variable levels of a factor (measure):

This view is misleading though, because the overall group sizes of treatment and control are not equal.

What I'd like to do is to represent this as a proportion for each levels of the factor (measure) instead of using just a count. For instance, perhaps 25% of the treatment group is in Business Development whereas only 10% of the control group is in Business Development.

Is this possible to do in Tableau, or does it strictly require pre-processing?

Yes, you can do this without reworking your, and as always, there are few ways to do it. Without seeing exact data, I would try setting table calculation to percent of total and then alter compute with to Table Down / Table Across depending on the data layout


kNN - what happens if more than K observation have the same distance to the centroid of the cluster EDIT It was pointed out in the Answers-section that I am confusing k-means and kNN. Indeed I was thinking about kNN but wrote k-means since I'm still new to this topic and confuse the terms quite often. So here is the changed question.

I was looking at kNN today and something struck me as odd or - to be more precise - something that I was unable to find information about namely the following situation.

Imagine that we pick kNN for some dataset. I want to remain as general as possible, thus $k$ will not be specified here. Further we select, at some point, an observation where the number of neighbors that fulfill the requirement to be in the neighbourhood are actually more than the specified $k$.

What criterion/criteria should be applied here if we are restricted to use the specific K and thus cannot alter the structure of the neighborhood (number of neighbors). Which observations will be left out and why? Also is this a problem that occurs often, or is it something of an anomaly?

K-means does not make an assumption regarding how many observations should be assigned to each cluster. K is simply the number of clusters one chooses to generate. During each iteration, each observation is assigned to the cluster having the nearest mean. There is no assumption that all clusters should have a comparable number of observations assigned (i.e., for N observations, there is no expectation that each cluster should have ~ N/K observations assigned).

It is quite possible that the numbers of observations in the various clusters are highly imbalanced. This can be due to the distribution of the data, the number of clusters chosen (K), or even how the cluster means are initialized.


Same TF-IDF Vectorizer for 2 data inputs I am trying to work on Dataset released by quora, to identify if Question1 has similar intent as of Question2
The dataset looks like:

id|question1|question2|is_duplicate
0|What is the step by step guide to invest in share market in india|What is the step by step guide to invest in share market?|0

I am trying to refer to Abhishek Thakur's feature to get started.
It says:

As per my understanding the python code for sklearn would be:
    from sklearn.feature_extraction.text import TfidfVectorizer
    import pandas as pd
    tfidf_vectorizer = TfidfVectorizer()
    data['tf_idf_q1'] = tfidf_vectorizer.fit_transform(data.question1)
    data['tf_idf_q2'] = tfidf_vectorizer.fit_transform(data.question2)

data['tf_idf_q1] and data['tf_idf_q2] will refer to 2 models for each question as in 1st part of the image.
I am not sure how would i achieve second part? Do I fit_transform the vectorizer with first question and then transform the second question?
Or do I merge 2 questions and then get a vectorizer?
Something like below:
merged_questions = pd.DataFrame(data['question1'].map(str) + data['question2'].map(str))
data['tf_idf_q1_q2'] = tfidf_vectorizer.fit_transform(merged_questions)

Any inputs are greatly appreciated.

You can use something like this

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
tfidf_vectorizer = TfidfVectorizer()
raw_data = pd.DataFrame(*raw_data, columns = ['id', 'is_identical', 'q1', 'q2'])
data['tf_idf_q1'] = tfidf_vectorizer.fit_transform(data['q1'])
data['tf_idf_q2'] = tfidf_vectorizer.fit_transform(data['q2'])
data_for_model = data[['tf_idf_q1', 'tf_idf_q2', 'is_identical']]
X = data_for_model[['tf_idf_q1', 'tf_idf_q2']].as_matrix()
Y = data_for_model['is_identical'].as_matrix()
model = Sklearn.LogisticRegression()
model.fit(X, Y)

Combined Model - here you actually learn the transformation for all questions. Then transform each one(question) separately to create features for your model training.

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
tfidf_vectorizer = TfidfVectorizer()
raw_data = pd.DataFrame(*raw_data, columns = ['id', 'is_identical', 'q1', 'q2'])
tf_train_data = pd.concat([data['q1'], data['q2']])
trained_tf_idf_transformer = tfidf_vectorizer.fit_transform(tf_train_data)
data['tf_idf_q1'] = trained_tf_idf_transformer.transform(data['q1'])
data['tf_idf_q2'] = trained_tf_idf_transformer.transform(data['q2'])
data_for_model = data[['tf_idf_q1', 'tf_idf_q2', 'is_identical']]
X = data_for_model[['tf_idf_q1', 'tf_idf_q2']].as_matrix()
Y = data_for_model['is_identica'l].as_matrix()
model = Sklearn.LogisticRegression()
model.fit(X, Y)


Any cluster algo can cluster time series datasets based on variation ratio(or quantity)? I learn machine learning from sciki and read its documents.

Clustering cluster groups based on the euclidean distance and filter them by different ways ex: guassian distribution, or mean-shift...etc.

But none of clustering algo cluster samples based on the variation ratio

EX: below every items has there sold numbers of everyday.

Item,D1,D2
A,1,5
B,10,50
C,4,70

The variation ratio below:
A:500%
B:500%
C:1750%

So A and B are the same group, C isn't.

Any cluster algo can cluster time series datasets based on variation ratio(or quantity)?

Extract features such as variation ratio
Cluster the extracted features instead of the raw data


Can learning algorithms take in data along with their uncertainty? (chaining ML algorithms along with errors) How to chain statistical methods (estimators or classifiers) taking into account the uncertainty (error) of the previous step?
Ex: Consider a pipeline, where housing prices are estimated from census and geographical data and are fed into another algorithm to estimate credit scores. How does the error in estimating housing prices fed into credit scores estimator and factor in the overall error?
I think that if you just consider the output values and not the error of the previous step, the error of the current estimator will be lesser and will be misleading. The uncertainty is not being propagated forward in this pipeline, hence the uncertainty at the end is only because of the last step.
I'm new to Machine Learning and in the introductory books or courses I didn't come across any discussion about this topic. If anyone can point me to good resources to learn more about this, I'll be happy.

One option is to use Bayesian methods in machine learning.
Assume the input to any stage in a machine learning pipeline is a prior distribution and the output of any stage is a posterior distribution. Then it would be possible to chain stages together with uncertainty propagated through the system through the distributions.


Suggestion on Preprocessing dataset I am trying to preprocess my dataset and needs some suggestion on it.

The training data shape is : (166573, 14)

The distribution of features :

As you can see, only the first 4 columns go to different max values. Rest of the columns have either 1 or 0 value (max: 1, min: 0)

**Null Handling: **

I have dropped claims_daysaway column as most of the values are NULL and replace tier's NaN values with its mean value.

Scaling features:

I have scaled features where max value varies and left others untouched.

X['scaled_distance']= sc.fit_transform(X['distance'].values.reshape(-1,1))
X['scaled_visit_count'] = sc.fit_transform(X['visit_count'].values.reshape(-1,1))
X['scaled_tier'] = sc.fit_transform(X['tier'].values.reshape(-1,1))

Is this right approach? or should I scale all features?

Depends on what is your goal with the data.

If you are trying to model your variable "outcome" as a classification problem (I am supposing this) then it depends on the modelling technique you are expecting to use.

Some techniques use scalled data as input because they have a functional form which allows them to perform better if the scale is the same for the variables (the canonic example could be Neural Networks).

Some techniques do not need to have scalled inputs, mostly because their functional form does not depend on relationships (interactions) with other variables just as random trees.

There are other techniques like logistic regression where scalling is optional because they work fine without scalling, the result is the same but sometimes you need to compare the resulting $\beta$'s and the only way to do this is if the $X$s have the same scale.

The process you are doing is ok, but make sure you actually need to do it.


Pandas, R, and interoperable data frame formats I want to have a single format which is compatible both with R and Python's Pandas (and preferably, other things like Stata and SAS), for which I will not ever need to impute whether things are numbers or dates.

Does such a thing exist? Or has every package got their own system (Pickle, RDS, Stata .dta, .sasb7dat) for the purpose of lock-in?

Is your data in a data frame?  If so, feather is what you're looking for.


How to determine the function is linear in linear regression problem? I know that the first degree of the polynomial equation is considered as a linear function.

But, I found some things confusing in linear regression.

1. f(x)= w1 x1+ w2 x2 + W3 x3 --> linear function
2. f(x)= w1 x1+ w2 x2 + W3 x1 x3 --> is it linear? if not, then why?
3. f(x)= w1 x1+ w2 x2 + W3 W4 x3 --> is it linear? if not, then why?
4. f(x)= w1 x1+ w2 x2 + W3 x3 x3 --> it is not linear.

Please help.

A function $x = (x_1, \dots, x_n) \mapsto f(x_1, \dots, x_n) \in \mathbb{R}$  is linear if and only if for every $a,b \in \mathbb{R}$ it holds that $f(a (x_1, \dots, x_n) + b (\xi_1, \dots, \xi_n)) = a  f(x_1, \dots, x_n) + b f(\xi_1, \dots,\xi_n)$.

In your second example try computing $f(a x + b \xi)$ to see that the last term $w_3 x_1 x_3$ will make $f(a x + b \xi) \neq a f(x) + b f(\xi)$.

The third  example has a coefficient which is expressed as a product, but the function remains linear (just rename $w_3 w_4$ to $w$).


Is information lost when converting an image to a single vector for analysis? On several tutorials on object and pattern recognition in images (such as the typical mnist digit recognition problem), when working with a sample of $m$ images of $n \times n$ pixels each, the first step is to build a matrix of $m \times (n * n)$, where each image is converted into a single row-vector with all of its pixels, one next to the other. Different techniques are then applied to this matrix, such as PCA.

My question is: when converting a two-dimensional image to a single-dimensional vector by placing each image row next to another, the fact of which pixels are neighbours (in the same column, different row) seems to be lost. For instance, in image:

a b c
d e f
g h i

Which is converted to:

a b c d e f g h i

The fact that b and e were next to each other seems to be lost.

Is that conclusion correct? Does that fact even matter for analysis? Are there other ways to represent the image that do preserve that piece of information?

Apologies in advance if the question is too vague or basic, I'll be glad to reformulate if needed. Thank you.

The short answer is it depends on two things: what definition of information you use and if you really use that information.

From information theory point of view, if your transformation is reversible than the information is there. This happens because you can apply the inverse transformation to recover the original. So nothing is lost. This is similar with what happens in various lose-less compression algorithms.

So, information is there, but is encoded in another place, specifically in the transformation function. As a consequence, to learn also the position you have to create those new features to express that information. Not also that when you have a classical learner this information is not understood by learner. Take for example a linear model. There is no difference between $f_1 = \beta_0 +\beta_1 x_1 + \beta_2 x_2$ and $f_2 = \beta_2 x_2 + \beta_0 + \beta_1 x_1$. So learning with a simple linear model is impossible to use directly this locality information. This happens with most learners. So, even in the original space the locality is an information in your head, and not in the data.

On the other hand if your neural net is tailored for images and expects to receive patches of images where the order of the values is important than yes, your neural net perhaps understands those relations and use them. Even so, perhaps that custom neural net has a custom way to specify the location information, so what you will have to do would be to encode this information in a format specific to that custom neural net.

On the other hand if we are talking about a standard back propagation neural net, the order is irrelevant. The model is simply not able to use the location.

As a conclusion, if you have a standard generic learning algorithm your only your only chance would be to encode this locality information somehow in the features. For example (this is a made-up example, I do not even know if that would word) one can add a new set of input variable, one for each square of 4 adjacent pixels and use the average of pixels intensity as value. You do not made that explicitly enough, but you created a new set of features which is created using the locality information.


Sequences of time series data with only 1 output classification So I'm facing a problem where I have a sequence (30h of data with 10sec intervals) and which is labeled to a class (3 different classes) for the whole sequence. I'm used to work with time series who have an output for each time sample in the sequence and now I'm block with this.

So far I have thought of performing a statistical study of the sequence and create a dataset where each row contains the results of that study from a sequence and the target. This way I'll have a dataset with a Y for each X (but I'll lose a lot of information from the sequences)

I come here to see if any of you have faced a similar problem before or if you have any ideas of how to approach this.

With only one output, there is no task for a supervised learning algorithm for classification. Your algorithm can only predict your only one class, you do not give any other situations of other classes for your algorithm to work on; a classification algorithm will need at least two classes. I have strong experience with time-series learning problems, but it seems that your problem has nothing to do with whether your data is time-series.

However, depending on your task, you can implement unsupervised learning which does not require any labels. You can seperate your data into clusters which have similar properties as time-windows. Then you can predict with new (or test) samples that which cluster should it be from, what it is alike.


Difference between Memory and Disk space I was watching this machine learning course by Andrew NG where he states memory requirement and disk space requirement separately while referring to machine learning algorithms.
What exactly is the difference and how are they occupied?

What you need to remember is :

Disk space : slow to read/write but cheap, any computer has > 100Go of hard disk
Memory : fast to read/write but expensive, usually a few Go

When you run an algorithm you usually use only RAM. In deep learning your dataset is often too big so you store it in the disk space and load pieces of it into RAM when needed for training.
There are tons of resources on the internet if you want to better understand how they differ (for example https://www.differencebetween.com/difference-between-hard-disk-and-vs-ram/)


Linear Regression Coefficient Calculation class LR:

    def __init__(self, x, y):
        self.x = x
        self.y = y
        self.xmean = np.mean(x)
        self.ymean = np.mean(y)
        self.x_xmean = self.x - self.xmean
        self.y_ymean = self.y - self.ymean
        self.covariance = sum(self.x_xmean * self.y_ymean)
        self.variance = sum(self.x_xmean * self.x_xmean)

    def getYhat(self, input_x):
        input_x = np.array(input_x)
        return self.intercept + self.slope * input_x

    def getCoefficients(self):
        self.slope = self.covariance/self.variance
        self.intercept = self.ymean - (self.xmean * self.slope)
        return self.intercept, self.slope

I am using the above class to calculate intercept and slope for a Simple Linear Regression.  However, I would like to tweak it to make it work for Multiple Linear Regression as well, but WITHOUT using matrix formula $(XX^T)^{-1}X^TY$.

Please suggest.

While I am not sure if you need the calculations done within the class specifically, there is a relatively more simple way to extract the intercept and slope coefficients using the linear_model from sklearn and pandas if it is of use to you.

Suppose we have the following variables:

y: [-0.006,-0.001,0.015,0.017,-0.0019,-0.005]
x1: [-0.018,-0.008,0.011,0.017,-0.008,-0.002]
x2: [-0.04,-0.003,0.012,0.011,-0.004,-0.009]
x3: [-0.06,-0.007,0.3,0.09,-0.005,-0.006]

Now, let's run a linear regression using sklearn:

from pandas import DataFrame
from sklearn import linear_model
import statsmodels.api as sm

dataset = {'y': [-0.006,-0.001,0.015,0.017,-0.0019,-0.005],
                'x1': [-0.018,-0.008,0.011,0.017,-0.008,-0.002],
                'x2': [-0.04,-0.003,0.012,0.011,-0.004,-0.009],
                'x3': [-0.06,-0.007,0.3,0.09,-0.005,-0.006]
                }

df = DataFrame(dataset,columns=['y','x1','x2','x3'])

X = df[['x1','x2','x3']]
Y = df['y']

# Regression Model
regr = linear_model.LinearRegression()
regr.fit(X, Y)

print('Intercept: \n', regr.intercept_)
print('Coefficients: \n', regr.coef_)

Once we do this, our intercept and slope coefficients are then printed:

>>> print('Intercept: \n', regr.intercept_)
Intercept:
 0.0022491408670789535
>>> print('Coefficients: \n', regr.coef_)
Coefficients:
 [ 0.62742415 -0.06618899  0.02384715]

Hope you find this of use if you are simply looking to extract the intercept and slope coefficients.


Decision Tree Classifier with Majority vote (Is 50% a majority vote) I have a homework question that requires me to create a 3 level tree (root, intermediate and leaf) with majority vote and expand the tree if 3 level tree is not possible. I have to use ID3 algorithm. Its a manual question rather than coding assignment and dataset contains 16, 4 features and a class label with T, F as their values.
I started calculating entropies and information gain for different nodes. When I complete the 3 level tree, I can assign the classes at the leaf with 50%, 100%, 100% and 75% majority vote or confidence. My questions are

is 50% really a majority vote?
Do I need to expand on that part of the tree that has 50% confidence
or I can just assign the classes arbitrarily?

Regards

The goal of ID3 is to get the purest nodes possible ( ironically that is what contributes to its problem of overfitting), so 50% is not pure at all, the data under that node is equally likely to be in one of the classes which makes peedicition tricky, it would be better to grow the tree further and find nodes which are more pure than atleast 50%.


What is the best way to visualize range of data? If I a have data like a bunch of classes where each class can have a different number of students, and each student has a grade (0-100), for example like this:
tdf = pd.DataFrame({'classes':["a", "a", "a", "b", "b", "b","b","b", "c", "c"],
                    'grades':[78, 28, 10, 3, 79, 91, 65, 51, 75, 89]})

What is the best way to visualize the range of grades within each class? I want to show that in class A, there is a range between 10 and 78, in B there is 3 and 91, in C there is 75 and 89. I was thinking of boxplot or violin plot but when I tried them I am not really sure. Is there such thing as a bar chart that can show range? Will that make sense?
I am not sure if this is the right place to ask, so here I am.

I believe boxplot or violin plot is a good idea and you could overlay datapoints with a bit of jitter to the former. See below an example in seaborn taken from a relevant question:
import seaborn as sns
import matplotlib.pyplot as plt

tips = sns.load_dataset("tips")

sns.violinplot(x="day", y="total_bill", data=tips, color="0.8")
sns.stripplot(x="day", y="total_bill", data=tips, jitter=True, zorder=1)
plt.show()


Forecasting multiple time series with a single model I have a dataset with sales numbers for ~500 different markets (assume different cities or regions) and need monthly forecasts for each market. Instead of building 500 different models, I'm interested in just training a single model with a single set of params to prevent me from having to manually feature engineer each market.

Can someone point me to some resources for this problem? I went through the similar Stack questions and they weren't as helpful for this particularly.

Specifically, I know that LSTMs are sometimes used here, but would classical approaches be effective as well?

You can treat the markets as a categorical feature in a tree based (using decision tree as a weak leaner) ensemble model such as random forest or gradient boosting. Some applications are:

Sales for retail outlets of a major European Pharmacy
retailing company
Predicting the sales of
products from different outlets
Multi-Store multi-product forecasting


Classifying Letters using CNN - Help so some context, I'm trying to develop an OCR (for fun) and for that reason I decided to first find text within a page, parse it in to letters within the text and from there try and classify the letters that were extracted one by one.

For the classification I'm trying to build a CNN based upon the following database:
http://www.ee.surrey.ac.uk/CVSSP/demos/chars74k/

and the following CNN architecture:

 # CNN architecture
input = Input(shape=(32, 32, 1))
conv1 = Conv2D(32, (3, 3))(input)
pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)
drop1 = Dropout(0.5)(pool1)
conv2 = Conv2D(64, (3, 3))(drop1)
pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)
drop2 = Dropout(0.5)(pool2)
flatten = Flatten()(drop2)
fully_connected = Dense(512, activation='relu')(flatten)
output = Dense(62)(fully_connected)
model = Model(inputs=input, outputs=output)
model.compile(loss='sparse_categorical_crossentropy',optimizer='Adam', metrics=['accuracy'])
print(model.summary())
model.fit(X, y, epochs=10 , validation_split=0.3)

However, all I seen to get no matter what I try (epochs, batch size, validation split) etc is 0 accuracy.... So assuming my database and labels are fine... what could be going wrong?

So there are some things you could consider.

First of all, normally datasets like this (or for example mnist) will be one hot encoded. So you have a vector with the length of all classes, each position stands for one class and is 1 if it is the class and 0 otherwise.
As you are predicting values between zero and one, the activation of your output should do the same. So the first thing i would change is the activation function of the last layer to sigmoid. It is linear in your case.

Other things to consider are data preprocessing like normalizing pixel values between 0 and 1. Also no data preprocessing at all can lead to unstable training it is unlikely to cause no learning at all.


Spark Mllib - FPG-Growth - Machine Learning Is the FPG-Growth an Machine Learning algorithm? Because I'm looking at this code:

import org.apache.spark.mllib.fpm.FPGrowth
import org.apache.spark.rdd.RDD

val data = sc.textFile("data/mllib/sample_fpgrowth.txt")

val transactions: RDD[Array[String]] = data.map(s => s.trim.split(' '))

val fpg = new FPGrowth()
  .setMinSupport(0.2)
  .setNumPartitions(10)
val model = fpg.run(transactions)

model.freqItemsets.collect().foreach { itemset =>
  println(itemset.items.mkString("[", ",", "]") + ", " + itemset.freq)
}

val minConfidence = 0.8
model.generateAssociationRules(minConfidence).collect().foreach { rule =>
  println(
    rule.antecedent.mkString("[", ",", "]")
      + " => " + rule.consequent .mkString("[", ",", "]")
      + ", " + rule.confidence)
}

And I'm not seeing how the algoritm can learn because it don't have an train, test or validation set...

Thanks

FP-growth is a frequent pattern association rule learning algorithm. Thus it's a rule based machine learning algorithm.

When you call the following :

val model = fpg.run(transactions)

You are actually creating and Frequent Pattern model without generating candidates. So actually you'll need to generate afterwards if you need to use them with :

model.generateAssociationRules(minConfidence)

Now, concerning the usual flow of building and validating such models, your code doesn't deal with that. It's usually done through quality measures variations.

It can also be done through Feature extraction techniques.

With such techniques, you should consider the following. For each association rule, you'll need to measure the improvements in accuracy that a commonly used predictor can obtain from an additional feature, constructed according to the exceptions to the rule. In other terms, you'll have to select a reference set of rules that should help your model perform better. I strongly advice you to read this paper about the topic.

So now what does that mean ? This means that you'll need to implement that pipeline yourself because it's not implemented in Spark yet.

I hope that this answers your question.


Distance measure for ternary feature I have a data set consisting of 100 features each of which are ternary: values of -1 if it exists in one category, 0 if it doesn't exist, and 1 if it exists  in the second category. For example

F1 F2 F3 ... F90 F91 F92 ... F99 F100
0  0  0  ... 1   -1  0   ... 0   -1
0  -1 0  ... -1   0  1   ... 0   0

The data is very sparse, ~20 of the 100 features have values of -1 or 1 for each row of data. I want to find similar rows of data through a heatmap visualization and dendrogram but I was confused on whether to use Euclidean distance or Cityblock distance. I'm quite new to data mining and while reading the scipy pages, I found many distance measures which I have no idea means what. Is there a good distance measure for my type of data set?

Since apparently each feature is encoding something about two different categories, I would suggest that you should replace that with two features.  Your two features would be $(x,y)$ where $x$ is 0 or 1 according to whether it exists in the first category, and $y$ is 0 or 1 according to whether it exists in the second category.  In other words, instead of -1, 0, and 1, you would use $(1,0)$, $(0,0)$, and $(0,1)$, respectively.  I think that is closer to the true data and might give better results.  It might also make your results easier to interpret.

Then you can try both distance metrics and see which you find more helpful -- but it might not make a large difference.


Jupyter Notebook - Change HTTP Headers I am using jupyter notebook to host my machine learning codes and notebooks.

Jupyter Notebook server is sending some HTTP headers like Server = TornadoServer/5.1.1
I want to remove this header. Also want to add some HTTP headers like Cache-Control etc.
Can someone please tell me :

What kind of Http server jupyter notebook runs
Which configuration file controls the HTTP headers sent by Jupyter notebook
Any link which explains how to change/add/remove the HTTP headers

Current jupyter notebook is running on Ubuntu server on https.

You can generate a config file in your home directory by running:

jupyter notebook --generate-config

which should leave you with a file ~/.jupyter, called jupyter_notebook_config.py.

Have a look at all the possible configurations in the Jupyter documentation.

In addition to many server options, there is specifically an options to override Tornado configuration.

  NotebookApp.tornado_settings : Dict Default: {}

  Supply overrides for the tornado.web.Application that the Jupyter
  notebook uses.


Replacing column values in pandas with specific column with multiple database operation? Need to replace database of the column in specific refine query with multiple operations as mention in below image. Trying such operation as an individual, but can't understand which method to use can make in one column with multiple operations.

Using Boolean methods to justify results but how can I do in one line code of python to get a replacement of refining/ categorized values to a specific column.

  import pandas as pd
         db = pd.read_csv('Database Path')
         db.head(2)

stoppage time in minutes  | Activity
-------------------------120 | Stopped
-------------------------240 | Stopped

#Need to refine below code for specific requirement as mention above:

db['New Column(Time Category)'] =

db['Stoppage time in minutes'] < 120 OR
db['Stoppage time in minutes'] > 120 & db['Stoppage time in minutes'] < 240 OR
db['Stoppage time in minutes'] > 240 & db['Stoppage time in minutes'] < 360 OR

# So Need Result like as below:

stoppage time in minutes  | Activity | New Column(Time Category)
-------------------------120 | Stopped  | <2 HRS
-------------------------240 | Stopped  | 2 - 4 HRS

Require this solution due to a pool of high numbers of databases.

You can use the python ternary operator with list comprehension:

df['Buckets(HRS)'] =
 ['<2hrs' if x<=120 else '2-4' if 120>x<=240 else '4-6' if 240>x<=360 else '>6' for x in db['Stoppage time in minutes']]


TF-IDF Regression & Machine Learning I would like to take a set of documents where each document already has an assigned popularity change value ("trending", i.e. -10%, +25%, etc.), and create a machine learning model that would be able to predict a "trending" value for a new document.

After creating TF-IDFs for all documents, dropping some of the rarest words that only appear in 1 document, possibly removing stop words, etc., I'm a little lost. Is it the right approach to use some kind of sentiment analysis model that would train the TF-IDF vectors on the popularity change values? I am also thinking about how to train a regular regression model for this task, but it doesn't seem like an easy choice to integrate with TD-IDF data.

I'm not sure what you already know, so I'll answer how I'd do it. If you do not understand things, or already know some things I say, please leave a comment.

I would tackle it as a multi-class problem if your output is categorical, hence indeed only $10\%$, $25\%$ and so on. If you want to treat it as a regression problem, I'd say you need continuous deltas of popularity, so you need all kinds of values as the delta, and not only a fixed number of them. The training data should dictate how you approach the problem. Note that you can always reduce the regression to a multi-class problem (which asks for an instance which of the deltas it should assign) by bucketing to intervals.

For argument's sake let's assume you have multi-class data. One document has one of the classes, i.e. $10\%$. You then go ahead and encode these classes into a binary vector, which probably even is a part of the library you're using.

You'd then go ahead and split your data with the hold-out method to have some vlalidation examples set aside to see how your model actually works. You then go ahead and train some model, say a random forest. With the validation data you calculate some measure of error and see if it is enough. If not you go ahead and choose other models to train, other features to add and so on.

I'd suggest to try use another approach than bag-of-words such as word2vec or GloVe. You can also add the sentiment classifiers values as feature columns. You could then compare the different features and see how it performs.

Let me know if this helps you or if I misunderstood your question.


Eliminate low quality predictions in a classification task Here is some background on the problem. My aim is to classify text into some categories. I would like to get only good quality predictions from the model. If the model is not confident, I would like to classify the text manually.

Let's consider the example provided in http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html so that it's reproducible. In the following example, a classification model is trained and is fitted for test documents. One of the test document is - 'what the heck is this?'. I understand that model is returning the class which has highest probability. However, when the model is unsure, I would like to label the text as 'unable to classify'

from sklearn.naive_bayes import MultinomialNB
clf = MultinomialNB().fit(X_train_tfidf, twenty_train.target)

docs_new = ['God is love', 'OpenGL on the GPU is fast', 'what the heck is this?']
X_new_counts = count_vect.transform(docs_new)
X_new_tfidf = tfidf_transformer.transform(X_new_counts)

predicted = clf.predict(X_new_tfidf)

for doc, category in zip(docs_new, predicted):
    print('%r => %s' % (doc, twenty_train.target_names[category]))

Output

'God is love' => soc.religion.christian
'OpenGL on the GPU is fast' => comp.graphics
'what the heck is this?' => soc.religion.christian

Predicting the probabilities

Here are the probabilities of prediction. Documents 1 and 2 have somewhat clear winner. However, the third document doesn't have one. I have about 100 classes and I would be hesitant to set a manual threshold.

clf.predict_proba(X_new_tfidf)
array([[ 0.16297502,  0.03828016,  0.03737814,  0.76136668],
       [ 0.16387956,  0.36874738,  0.2364763 ,  0.23089675],
       [ 0.28288106,  0.17035852,  0.2484853 ,  0.29827513]])

An alternative to thresholding the classification probability would be to set a threshold on the ratio between the highest reported probability and the second highest reported probability. For example, a threshold of 2 would be interpretable as: "Only retain classifications where the likelihood of the class assignment is at least twice as likely as the next most likely class."


How to perform polynomial landmark detection with deep learning I am trying to build a system to segment vehicles using a deep convolutional neural network. I am familiar with predicting a set amount of points (i.e. ending a neural architecture with a Dense layer with 4 neurons to predict 2 points(x,y) coords for both). However, vehicles come in many different shapes and sizes and one vehicle may require more segmentation points than another. How can I create a neural network that can have different amounts of output values? I imagine I could use a RNN of some sort but would like a little guidance. Thank you

For example, in the following image the two vehicles have a different number of labeled keypoints.

Landmarks are nice when you have a fixed amount for every image but I don't think it is the right approach for your problem. Instead I think you should look into models that segment images by applying a mask. A place to start could be to look into Mask R-CNN.

Here is the paper: Mask R-CNN

Here is how to train a ready made Mask R-CNN implementation with your own dataset using Keras: Tutorial


where can I get access to AlexNet? A lot of websites on CNN for large datasets of images talk about starting with the pretrained model for 1.2 million images in 1000 categories available via AlexNet / Imagenet. These sites seem to imply that this dataset is freely available, but I'm having trouble actually getting access to it. For example, I tried going to https://github.com/deep-diver/AlexNet but couldn't get the code in alexnet.ipynb to run. Consider the following code:

for f in data_file.iterdir():
    data = pickle.loads(f.read_bytes(), encoding='bytes')
    if 'meta' in str(f):
        meta = data
    elif 'test' in str(f):
        test = data
    elif 'batch' in str(f.name):
        train_dic = merge(train_dic, data)

This seems to imply that there should be some data files which include the text strings 'meta', 'test', and/or 'batch', but I couldn't seem to find those files.

Where is the best place to get access to this dataset--ideally somewhere where there is good documentation as to how to download/use it?

The ImageNet data is online: http://www.image-net.org/.

The data descriptions says:

  14,197,122 images, 21841 synsets indexed

However, you don't need to train a model on all of the data from scratch, since you can use pre-trained models, e.g. in TF/Keras. See the docs: https://keras.io/applications/.

I don't think the original Alexnet is available, but a lot of other models are ready to load from Keras. You may also find a Alexnet version stored as .hdf5 or so online (which you could load with Keras/TF).

I don't know what you actually want to do, but here is a simple example of how to use a pretrained model (VGG16 in this case) in Keras.


Outlier detection with sklearn I've been reading the sklearn documentation on outlier detection, and even the examples provided by the documentation. Once I have fitted my dataset, all I can do is apply the predict function to the estimator in order to find outliers. However, I would like to get the probabilities that the point is an outlier. Can this be done in sklearn? Is there an R package to do it?

I don't even know if there is a theoretical foundation of the outlier detection methods used by sklearn that allows you to give probabilities. If not, what is the criterium that tells you what is an outlier and what is not? Does it consider probabilities or some kind of non-probabilistic scores?

Any piece of information will be appreciated.

Edit

I would like the outlier method to consider the multivariate distribution of the data. I think that univariate detection methods are rather poor.

One of the easiest and quick way would be to plot a 'boxplot' where the "dots" above the upper whisker(T shaoed line) of the boxplot denotes an outlier(refer to the attached image). I hope this was what you are searching for. Do let me know if otherwise, may be I can help.


How does CNN doing bounding box regression and what do features and weights represent for? I knew that, in the house price logistical regression problem, the weights and features represent the "importance" of factor or coefficients of feature variables respectively, then minimize LSR loss can get the value of coefficients, and question are:

How does CNN doing bounding box regression?

I actually did a lot of googling to find an intuitive explanation, but no luck.

What do features and weights represent for in BBR?

I think it couldn't be $T$, $L$, $W$, and $H$ because these absolute values will vary a lot due to distance/scale and perspective difference,but the ratio of $\frac{W}{H}$ is a reasonable feature (for me to understand) due to it is a relative value.

It depends on the concrete model, but we can consider the most popular single-stage model for object detection: SSD.

It has a set of default bounding boxes (prior boxes), for each of them it predicts:

Probability distribution over the set of classes (this is a classification problem, solved with crossentropy loss)
Offsets for the default location of the prior box (x, y coordinates for the center) and its height/width. This is a regression problem optimized with smoothed L1 loss (localization loss):

The exact definition is a bit more complicated, as it includes variance and exponents, but the general idea is like this. Also, depending on the bounding box encoding type, we may predict offsets not for the (cx, cy, w, h), but for (xmin, ymin, xmax, ymax).

The predicted shape offsets for prior boxes are not absolute values, of course. These values are relative to the default size of the prior box. The exact formulas for decoding are:


Stationary time series for clustering algorithms I have a set of time series data that I would like to feed into a clustering algorithm (like k-means, using dynamic time warping as the distance function). After standardizing the data with mean 0 and variance 1, the k-means classifier generated a batch of centroids that seemed to fit the data pretty well.

The only question I have is whether the data should be stationary. Models such as ARIMA require for the data to be stationary due to the nature of it. However, the data I want to cluster is mortgage rates as a function of time, which could be subject to seasonal trends, which could be useful when clustering other future time series data.

The question is: do clustering algorithms for time series data generally require for the data to be stationary?

Have you tried replacing the entire timeseries with some meta data of it - clustering on each point of a series seems to be a bit exzessive. Especially as then if your timeseries are not synchronized they might not be considered similar - eventhough they might be for the purpose of this exercise (Imagine two sinus curves but one is slightly phase shifted -- If you would want to cluster by frequency then these would be equal).

So you could try to do something similar to Fourier transformations Fourier Transformations, linear fits, etc and use this meta data as input for the clustering.


Estimating destination according to previous data I need an advice. I can resume my problem like that :

I have some travels in a database, for example :

Person1 travelled from CityA to CityB on Date1
Person1 travelled from CityB to CityC on Date2
Person2 travelled from CityB to CityD on Date3
...

We can consider that these cities are in the complete graph.

Now, according to all the travels in the database, I would like to know where a PersonX is likely to go. I can know when he come from (or not).

I don't know if I should use machine learning, data-mining or graph theory.

This is a spatio-temporal clustering problem that is likely best solved with a Markov model. You could reasonable group this into machine learning or data mining. Develop your model using machine learning and then (the data mining part) leverage those pattern recognition techniques (that have been developed in machine learning). I think there are at least one or two threads on this over at Cross-Validated that go into more detail.

Here are a couple of papers to look at if you are just getting started.

Using GPS to learn significant locations and predict movement across multiple users

Predicting Future Locations with Hidden Markov Models


Anomaly detection in Time Series Data - Help Required I am looking for algorithms on Anomaly detection for time series data. It is uni-variate analysis, considering single parameter (inlet pressure) of air compressor sensor data. The objective is to detect if there is any significant deviation in parameter value in real time.

If any one has worked on similar projects, please share your thoughts.

Welcome to DS forum. There are many methods available for anomaly detection of uni-variate, low dimension and high dimension data sets.

Supervised algorithms can be used if the past anomalies are recorded with flag variables and there is significant examples of flagged anomalies.

In unsupervised algorithms the key aspect of anomaly is which subset of the larger data set is considered when looking for anomolies i.e., global vs local and point vs contextual

Here are some useful links to get you started, anomaly detection in machine generated data and introductory overview of time-series based anomaly detection algorithms.

Hope this helps.


Visualize performance, % of goal implementation I want to visualize goal achievment progress.
This is my first idea:

use area chart to show progress in current metric
use horizontal band to show the goal value
colorize areas under/above the band into "positive" and "negative" colors

Is this approach informative enough? Are there better choises?

Additional info:

charts made in Tableau
two data sources: metric progress & goals

What information should be shown on the charts are based on what the viewer wants. From your sample charts, the x axis indicates 'date'. For time serial analysis, people usually want to know the trending of the metrics and to predict the performance in the future.

Here are 2 suggestions:

People may want to know how it will perform in the future. Add a trendline. Trendline could help people to predict the performance in the future.
http://onlinehelp.tableausoftware.com/current/pro/online/mac/en-us/trendlines.html
People may want to know how far away from goal. When people mouseover on the points on the metric line, you can try to show how far from current metric to the goal.

Try to understand what information people want to know and visualise them. Hope my answer could help you.


Text classification for data with multiple labels per observation I have a dataset of tweets that has been labeled by multiple people. So the columns look something like:

Tweet_ID, Coder_1_Classification, Coder_2_Classification, etc.

The idea is to build a tweet classifier based on the labels. How should I input this data into the classifier? I was thinking of taking the plurality response, but is there a way I can use all of the data?

Also, there is partial coverage, so not every labeler labeled every tweet.

There are different options:

Multi-class classification: each tweet can be labelled with several classes
Majority voting: each tweet is labelled with the class chosen most often
Consider tweets with (too many) different labels as ambiguous and remove them from the dataset
...

Additionally it's usually useful to calculate the inter-annotator agreement, because this indicates how subjective the task is and therefore how accurate one can expect the predictions to be.


Neural Network unseen data performance I started dabbling in neural networks quite recently and encountered a situation which is quite strange (at least with my limited knowledge).

The problem I'm using a NN is a regression problem which tried to predict the sales of a product for a particular kind of promotion in an FMCG.

Although the data is not strictly time series, it still has some time related attributes like a prediction may depend on a similar kind of promotion last year (which I've modelled using feature engineering).

Now to the problem I'm having:

I took the data from 2015-2017,augumented it by adding small amount of noise, shuffled it, and ran it through a neural network(I think the architecture is not important, but let me know if it is, I'll try and post it).

Optimizer: Adam with a decay
Loss Function : Volume weighted mape
Validation set : Randomly selected 20% of the data

The network trained well and gave me errors as small as 8%. And both training and validation set errors decreased together and did not indicate an overfit.

But the kicker is that when I applied the algorithm to new unseen data(2018, first half), the errors rose to 55%. .

I tried researching the issue on Google but didn't find anything useful. What is happening here? Am I doing something wrong?

Did you split your „original“ data into Train/Dev/Test?
if not, try splitting your data into three parts. Train the model on your Train set, tune your hyperparameters on your Dev set and test your network on the Test set. The validation error doesn’t give an unbiased error, because the hyperparameters are tuned to it. That’s why you should always test your model on a given Test set. If the error is also low for the Test set I would guess you have another problem:
Without knowing too much about your approach and assuming you have no bugs, my guess is that the high error is because your Validation and Test set come from different distributions. I assume that maybe sales in 2015-2017 are quite different to those in 2018.
You should also take a look at Andrew Ng's explanation of this particular problem.


If the entire production dataset is known is advance, should I use it when fitting normalization parameters? I found in multiple sources the recommendation not to fit the normalization parameters on the combined train/test dataset when evaluating the model, to prevent data leakage.

I presume this recommendation assumes prediction data would flow to the model online, and the entire data is not known in advance.

Does this recommendation also apply to when the entire data is known in advance?
And even if not, I guess it is possible to refit the model every time we get new data, with normalization parameters acheived from the entire dataset seen until now?

What are the pros and cons of normalizing over the entire dataset before fitting (as opposed to only on the train set)?

The point of evaluating your model is to assess how it will perform when it is used on unseen data. Normalizing over the entire dataset causes a data leakage (information coming from the test set). As a result, you will get a biased evaluation of your model.

In any machine learning task, the end goal is to use your model on unseen data to see how much the model has learned and how well it can generalize to unseen data. In that regard, there is no pros for normalizing over the entire dataset, and you should avoid doing so!


Transformation of categorical variables (binary vs numerical) When using categorical encoding, I see some authors use arbitrary numerical transformation while others use binary transformation. For example, if I have  a feature vector with values A, B and c. The first method will transom A,B and C to numeric values such 1,2 and 3 respectively, other researches use (1,0,0), (0,1,0) and (0,0,1).

What is the difference between the first method and the second one?

The only difference I can think of is, if you use binary values, the size of the training/testing data will increase linearly according to how many values you have, which may slow down the performance, while the first one will keep the size unchanged.

Does either of these methods will effect the accuracy of your machine learning model (or classifier)?

While using one-hot (binary) encoding certainly takes more space, it also implies an independence assumption among the data. On the other hand, using integers such as 1, 2 and 3 implies some kind of a relationship between them.

The problem that you mention of linear increase in size with one-hot encoding is common and can be treated by using something such as an embedding. An embedding also helps define a sense of distance among different datapoints.

https://en.wikipedia.org/wiki/Word_embedding


Topic Similarity Measure | Multi-class Text Classification Model I am trying to build a multi class text-classifier that classifies whether the tweet belongs to one of the categories ( Advise or Science or others )
let the input be any tweet like this ,
Input :

The goal of teaching should not be to help the students learn how to
memorize and spit out information under academic pressure. Brain

The purpose of teaching is to inspire the desire for learning in them and
make them able to think, understand, and question.

#maths #richardfeyman

Output :
( Learning : 98% , others : 1 % , Science : 1 % )
For now i came up with this
Idea 1 :
Maybe i can use LDA to get the topics of the tweet and perform semantic matching between these labels ( learning , science , others ) so one with the highest score can be choosen as right one (argMax(...) ).
What do you think about the above idea ?
Can anyone please enlighten me or point me in the right direction .

LDA is amazing for topic modelling, but naming the topics (regardless of what algorithm you use) is inevitably a disappointing process, because the labels you’re thinking of are very subjective and the algorithm has little respect for what words we want to go in each topic :p

The way I would approach your problem is:

Train an LDA topic model, using grid search rather than intuition to select the right number of topics
Transform your training examples into k-dimensional vectors using your trained topic model
Add your training vectors to a k-d tree and classify new tweets using a k-nearest neighbours approach


Feature importance difference in two similar machine learning models Situation 1:
I have trained a text classification model (Model 1) which gives me a probability of true class as X. I have also trained a classification model (Model 2) using only the categorical and numeric data. Both the models are used to predict the same true class; just the features differ.
I  used a random forest classifier on the probabilities returned by Model 1 and Model 2(taking them as input features) and got similar performance metrics(Accuracy, Precision recall). feature importance was 49% for model 1 and 51% for model 2.
Situation 2:
I used the probability X of text classification model as an input feature to the Model 2(which contained categorical and numeric features). The performance was almost similar to Situation 1 but here the feature importance of the final model indicated that text model probability had higher importance around 68% and rest of the features had lesser importance.
I want to understand the difference in feature importance of both situations.

In 2nd case, you are not comparing apple to apple.
Let's say we have 4 Features and all are equally good [Also, No interaction].
Case I -
We created two Models using 2 Features each
Model I - F1/F2
Model II- F3/F4
Comparing these two Models as Features will give you an Idea about F1/F2-combined compared to F3/F4-combined. This is your Case - I.
Case II -
If you will compare F1/F2-combined, F3(alone), and F4(alone), definitely F1/F2 combined will have high importance.
 The output probability of a Model using two Features is basically holding the information of both the Features together.


How to create and format an image dataset from scratch for machine learning? I've only worked with ML with .csv formats. I've worked with image formats too but only premade imagesets (MNIST,etc). If I were to create an imageset from scratch, how are the class labels typically formated? Would I have to manually title the image of a jpeg?

Best,
Jeremy

Looking at existing challenges around and their data format (for example http://www.kaggle.com/c/datasciencebowl/data) I would say put the images in a folder per class. You can use the file names for the index.


Expectation Maximization Algorithm (EM) for Gaussian Mixture Models (GMMs) I'm trying to apply the Expectation Maximization algorithm (EM) to a Gaussian Mixture Model (GMM) using Python and NumPy. The PDF document I am basing my implementation on can be found here.
Below are the equations:
$\mathrm{E}-\text{step:}$
$$w_{ik} = \frac{\pi_k \cdot p_k(x_i|z_k, \mu_k, \Sigma_k)}{\sum_{m=1}^{K} \pi_m \cdot p_m(x_i|z_m, \mu_m, \Sigma_m)}, \; [1]$$
$\text{where:}$
$${\displaystyle (2\pi )^{-{1}}|{\Sigma_k}|^{-{\frac {1}{2}}}\,\mathrm e^{-{\frac {1}{2}}(x_i -{\mu_k})^{\!{\mathsf {T}}}{{\Sigma_k }}^{-1}(x_i -{\mu_k})}.}
$$
$\mathrm{M}-\text{step:}$
$$\pi_k^{\text{new}} = \frac{N_k}{N}, \; [2]$$
$\text{where:}$
$$N_k = \sum_{i=1}^{N} w_{ik}.$$
$$\mu_k^{\text{new}} = \frac{1}{N_k} \sum_{i=1}^{N} w_{ik} \cdot x_i, \; [3]$$
$$\Sigma_k^{\text{new}} = \frac{1}{N_k} \sum_{i=1}^N w_{ik} (x_i - \mu_k)(x_i - \mu_k)^\mathsf {T}. \; [4]$$
When applying the algorithm I get the mean of the first and second cluster equal to:
array([[2.50832195],
       [2.51546208]])

When the actual vector means for the first and second cluster are, respectively:
array([[0],
       [0]])

and:
array([[5],
       [5]])

The same thing happens when getting the values of the covariance matrices I get:
array([[7.05168736, 6.17098629],
       [6.17098629, 7.23009494]])

When it should be:
array([[1, 0],
       [0, 1]])

for both clusters.
Here is the code:
np.random.seed(1)

# first cluster
X_11 = np.random.normal(0, 1, 1000)
X_21 = np.random.normal(0, 1, 1000)

# second cluster
X_12 = np.random.normal(5, 1, 1000)
X_22 = np.random.normal(5, 1, 1000)

X_1 = np.concatenate((X_11,X_12), axis=None)
X_2 = np.concatenate((X_21,X_22), axis=None)

# data matrix of k x n dimensions (2 x 2000 dimensions)
X = np.concatenate((np.array([X_1]),np.array([X_2])), axis=0)

# multivariate normal distribution function gives n x 1 vector (2000 x 1 vector)
def normal_distribution(x, mu, sigma):
  mvnd = []
  for i in range(np.shape(x)[1]):
    gd = (2*np.pi)**(-2/2) * np.linalg.det(sigma)**(-1/2) * np.exp((-1/2) * np.dot(np.dot((x[:,i:i+1]-mu).T, np.linalg.inv(sigma)), (x[:,i:i+1]-mu)))
    mvnd.append(gd)
  return np.reshape(np.array(mvnd), (np.shape(x)[1], 1))

# Initialized parameters
sigma_1 = np.array([[10, 0],
                    [0, 10]])
sigma_2 = np.array([[10, 0],
                    [0, 10]])
mu_1 = np.array([[10],
                 [10]])
mu_2 = np.array([[10],
                 [10]])
pi_1 = 0.5
pi_2 = 0.5

Sigma_1 = np.empty([2000, 2, 2])
Sigma_2 = np.empty([2000, 2, 2])

for i in range(10):
  # E-step:
  w_i1 = (pi_1*normal_distribution(X, mu_1, sigma_1))/(pi_1*normal_distribution(X, mu_1, sigma_1) + pi_2*normal_distribution(X, mu_2, sigma_2))
  w_i2 = (pi_2*normal_distribution(X, mu_2, sigma_2))/(pi_1*normal_distribution(X, mu_1, sigma_1) + pi_2*normal_distribution(X, mu_2, sigma_2))
  # M-step:
  pi_1 = np.sum(w_i1)/2000
  pi_2 = np.sum(w_i2)/2000
  mu_1 = np.array([(1/(np.sum(w_i1)))*np.sum(w_i1.T*X, axis=1)]).T
  mu_2 = np.array([(1/(np.sum(w_i2)))*np.sum(w_i2.T*X, axis=1)]).T
  for i in range(2000):
    Sigma_1[i:i+1, :, :] = w_i1[i:i+1,:]*np.dot((X[:,i:i+1]-mu_1), (X[:,i:i+1]-mu_1).T)
    Sigma_2[i:i+1, :, :] = w_i2[i:i+1,:]*np.dot((X[:,i:i+1]-mu_2), (X[:,i:i+1]-mu_2).T)
    sigma_1 = (1/(np.sum(w_i1)))*np.sum(Sigma_1, axis=0)
    sigma_2 = (1/(np.sum(w_i2)))*np.sum(Sigma_2, axis=0)

Would really appreciate if someone could point out the mistake in my code or in my misunderstanding of the algorithm.

One reason why you aren't getting fitted values close to the true values could be the initial values of the parameters used.
It's likely what you have found is a local maxima. You have to try a number of initial starts and then pick the one with that gives the highest likelihood.


Identify objects (bus) on the map based on coordinates (lat, lon) Let's say I have an android app that frequently sends current GPS location of the user. If person is driving with bus, I can easily get GPS location of the bus and display it on the map and update it accordingly every 30 seconds.
Let's say that instead of one bus there are 3 buses driving along same route. My problem is to display real time location of every bus on the route. In this case there are X users on every bus all having my android application installed on their phones. Here is the picture that could represent the problem:

As you can see, every mobile application will send GPS location of the user to my server. Server receives and saves data.
Based on the server data I should create a web app that should display map whith position of every bus in real time.
One possible way to do this is to to try to cluster GPS data and based on clusters to conclude where is position of every bus. This approach has few problems:

This algorithm could be slow if there is lot of data (then I can't
update position of every bus every 30 seconds) Second problem is that
I don't know number of clusters (buses). There might be 2, 3, 4 or N
buses

Is there any other better approach to this problem?

You might build a logic of distance. This will raise an issue with buses passing each other. For curiosity,  can't you identify source of signal?

To be honest, me thinking of same problem but from moving objects identifying (computer vision), so you can consider approaches out there.

Good luck.


Mutation on demand in genetic algorithms genetic algorithm usually use a "mutation rate" to control the rate of chromosome mutation. Most of the researchers at researchgate recommend to keep this rate low in order to converge quickly, to be able to find local optima and not to make the optimization a random walk. However, I see one major problem with keeping a low mutation rate. If the breeding at one point doesn't result into "new" children/chromosomes, the algorithm will run very inefficiently. Assume that all individuals in the population have the same chromosome, then crossovers will result into the same individuals until a mutation occurs in one of the chromosomes. With a low rate this can take quite long (until then the fitness function with the same chromosomes is repated again and again).

Isn't it much more efficient to mutate on purpose as soon as children/chromosomes would repeat in the next generation in order to assure that we explore new solution space? Wouldn't this make a "mutation rate" obsolete and the entire algorithm more efficient or do I miss a major point here?

If the breeding at one point doesn't result into "new" children/chromosomes, the algorithm will run very inefficiently. Assume that all individuals in the population have the same chromosome, then crossovers will result into the same individuals until a mutation occurs in one of the chromosomes

This is correct, but in theory this case should happen only when the algorithm has already converged, meaning that the chromosome is actually optimal. If this happens by chance before reaching an optimal solution, then it's probably because the size of the population is too low and should be higher in order to maintain diversity and allow mutations from time to time.

For example, if the population size is 100 and the mutation rate is 0.02 then there are in average two mutations for a particular gene at every generation. This maintains a bit of diversity, as the mutated gene is likely to be reproduced at the next generation if it turns out to be beneficial.

My experience with genetic algorithms is that many variants are possible and imho it's a good idea to explore various variants with a dataset, including non-standard parameters such as this idea of "automatically mutating", because why not? The danger I can imagine with that is that this could cause too much variation and prevent or slow down the convergence, but testing is the best way to see what happens.


Regression with LSTM network: use multiple time series as input I've spent a few days on this and am starting to think I'm missing the obvious solution as this doesn't seem like a very uncommon problem.
As an example dataset: I have 100 measurements with each a different constant speed as target and a length of 10000 data points. Every measurement has additional 30 features that are temperatures, airflow, humidity, etc. My goal is to predict the speed depending on the features/conditions.
My initial dataset so far has the shape (100, 10000, 30) as in (amount of measurements, length of measurements, amount of features) which I would convert with a sliding window shape. All the solutions I have found so far create one big dataset which (in my understanding) would mean that the independent measurements are treated as one single measurement.
As the hard transitions between the measurements harm my predictions: How can I train the network without creating one big measurement that includes these 'transition mistakes'?
I want to use a larger part for training and evaluation and then test a few of the remaining measurements on their own. Therefore I presume my input_shape has to fit a single measurement (10000, 30) and the output target as speed has (10000, 1). How do I transform my data into the window shape while keeping the measurements separate and insert it into my network for training/testing?
I've tried using the batch_input_shape but couldn't find a solution how to incorporate this with the sliding window and existing input_shape so I'm not sure if this could even solve my problem. (I've also thought about training a model each per measurement and trying to ensemble them but that seems way too complicated for this kind of problem.)

I think what you're looking for is to change your dataset into a supervised learning problem. Keras has its own TimeSeriesGenerator but that wont be of use as it only has a single step output. Someone created an alternate version of the TimeSeriesGenerator. Using this, you can specify the inputs and outputs per sample to change your data into a supervised learning problem which will then be able to fit seamlessly in your LSTM network.
Hope that helps.


What are the possible approaches to fixing Overfitting on a CNN? Currently I am trying to make a cnn that would allow for age detection on facial images. My dataset has the following shape where the images are grayscale.

(50000, 120, 120) - training
(2983, 120, 120) - testing

And my model currently looks like the following - I've been testing/trying different methods.

    model = Sequential()
    model.add(Conv2D(64, kernel_size=3, use_bias=False,
                     input_shape=(size, size, 1)))
    model.add(BatchNormalization())
    model.add(Activation("relu"))

    model.add(Conv2D(32, kernel_size=3, use_bias=False))
    model.add(BatchNormalization())
    model.add(Activation("relu"))

    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.25))
    model.add(Flatten())

    model.add(Dense(128, use_bias=False))
    model.add(BatchNormalization())
    model.add(Activation("relu"))

    model.add(Dropout(0.5))
    model.add(Dense(10, activation='softmax'))

    #TODO: Add in a lower learning rate - 0.001
    adam = optimizers.adam(lr=0.01)
    model.compile(optimizer=adam, loss='categorical_crossentropy',
                  metrics=['accuracy'])
    model.fit(x_train, y_train, validation_data=(x_test, y_test),
              epochs=number_of_epochs, verbose=1)

After running my data on just 10 epochs I started to initially see decent values but at the end of the run my results were the following and it has me concerned that my model is definitely over fitting.

How many epochs: 10
Train on 50000 samples, validate on 2939 samples
Epoch 1/10
50000/50000 [==============================] - 144s 3ms/step - loss: 1.7640 - acc: 0.3625 - val_loss: 1.6128 - val_acc: 0.4100
Epoch 2/10
50000/50000 [==============================] - 141s 3ms/step - loss: 1.5815 - acc: 0.4059 - val_loss: 1.5682 - val_acc: 0.4059
Epoch 3/10
50000/50000 [==============================] - 141s 3ms/step - loss: 1.5026 - acc: 0.4264 - val_loss: 1.6673 - val_acc: 0.4158
Epoch 4/10
50000/50000 [==============================] - 141s 3ms/step - loss: 1.3996 - acc: 0.4641 - val_loss: 1.5618 - val_acc: 0.4209
Epoch 5/10
50000/50000 [==============================] - 141s 3ms/step - loss: 1.2478 - acc: 0.5226 - val_loss: 1.6530 - val_acc: 0.4066
Epoch 6/10
50000/50000 [==============================] - 141s 3ms/step - loss: 1.0619 - acc: 0.5954 - val_loss: 1.6661 - val_acc: 0.4086
Epoch 7/10
50000/50000 [==============================] - 141s 3ms/step - loss: 0.8695 - acc: 0.6750 - val_loss: 1.7392 - val_acc: 0.3770
Epoch 8/10
50000/50000 [==============================] - 141s 3ms/step - loss: 0.7054 - acc: 0.7368 - val_loss: 1.8634 - val_acc: 0.3743
Epoch 9/10
50000/50000 [==============================] - 141s 3ms/step - loss: 0.5876 - acc: 0.7848 - val_loss: 1.8785 - val_acc: 0.3767
Epoch 10/10
50000/50000 [==============================] - 141s 3ms/step - loss: 0.5012 - acc: 0.8194 - val_loss: 2.2673 - val_acc: 0.3981
Model Saved

I assume the issue might be related to the number of images I have for each output class, but other then that I am a bit stuck in moving forward. Is there something wrong in my understanding/implementation? Any advice or critique would be well appreciated this is more of a learning project for me.

To deal with overfitting, you need to use regularization during the training:

Weight regularization - The first thing you have to do (practically always) is to use regularization on the weights of the model. L1 or L2 regularization update the general loss function by adding another term known as the regularization term. As a result thee values of weights decrease because it assumes that a neural network with smaller weights leads to simpler models. Therefore, it will also reduce overfitting.
If you are not sure what you need, just use L2.

Keras - Usage of regularizers
Dropout - Add dropout layers after dense layers (by the way, there are also advantages to using dropout after the convolution layers, it helps with occlusions). Just make sure not to use it at the final dense layer (the one with the same size as the number of classes).
Data Augmentation - The simplest way to reduce overfitting is to increase the size of the training data. Use data augmentation to potentially expend your training set to "infinity". Keras's data augmentation is really simple an easy to use:

Keras Image Preprocessing

If you implement these 3 steps, you will see drastic improvements (probably even just after the first one).

Further corrections and improvements (nothing to do with overfitting):

Your batch normalization layer should come after the non-linear activation, or more accurately, it needs to come before the next convolution layer.
Add an additional dense layer or 2 (only if the results are not good enough).


Churn Prediction problem I am a data science neophyte struggling to solve a churn prediction problem, i would be grateful to have someone help with the regression model

I really like this article because it gives an overview of common approaches to churn prediction as well as a new state-of-the-art method.  There is also code available. However, the state-of-the-art method uses a custom loss function built on top of recurrent neural nets, which is a complex place to dive in.


How to model user choice probability: binary model vs multi class model Let's say Morpheus has multiple users to offer colored pills(from an infinite set of colored pills), there are in total 3 unique colored pills(red, blue, green) Morpheus can offer. The trick is, Morpheus can offer only one pill to a user and the user has a choice to either take the pill or deny it. (Also, user's decisions are independent of each other)
Now Morpheus wants to be smart about his offer and wants to model the user such that the user selects the pill he is offering.
The users are moody and there is some uncertainty that they would randomly take a choice.
Rejection can be because of multiple unknown reasons such as I didn't like the color of the pill, I will choose the pill later, I want to understand more about this pill, Show me other pills before I decide
Now there are two ways I can think of modeling this:

Treating this as binary classification
Treating this as multi-class classification

When I treat this as binary classification, I pass pill color as feature with other user features to the model, and my output is the probability of user taking or rejecting a pill given the pill color.
Morpheus can then offer the pill color with the highest probability. This will use both Accept and Reject decisions of a user while modeling, but there is some uncertainty and the same type of users can accept or reject randomly.
When I treat this as a multi-class classification, where I try to predict the pill color itself. I would not use the rejected case in my training and would only consider cases when the user chose something.
In this way, I can reduce uncertainty in this case but would have to completely ignore rejected cases. Morpheus then can either use softmax or sigmoid for each class and take argmax to get the best choice to offer.
I am not sure if there are other ways to model this problem, but out of these two which can be a better way?

This is a textbook multi-armed bandit problem where Morpheus needs to learn the correct policy about offering pills. As you’ve said the Neos are independent, and making the assumption that there is a best pill overall, we need an algorithm that will experiment with each of the pills to find out which one is most likely to be accepted. This is the same as having three one armed bandit slot machines and trying to find out which one will pay out most frequently.
In the case where the Neos are observable (so that we have some information about each Neo and can predict what pill they would like based on their characteristics) this becomes a contextual bandit problem. This is the basic form of reinforcement learning problems
In a contextual bandit problem, you need to balance exploration (trying out offering different pills to different Neos to find out what they like) with exploitation (choosing what seems to be the best pill based on what we saw so far). This is why straight-up supervised multinomial classification approaches (as in e.g. Benji Albert’s answer) will struggle to converge: they don’t explore the “action space” (i.e. try out a bunch of responses) sufficiently in order to generate a variety of training examples for themselves.


What is an intuitive explanation for the log loss cost function? I would really appreciate if someone could explain the log loss cost function And the use of it in measuring a classification model performance.

I have read a few articles but most of them concentrate on mathematics and not on intuitive explanation and also a basic implementation using python with a small dataset would really helpful, So I can understand it better.

It would really help many looking for the same here. Thanks.

Ok, so here is how it works. Say you want to classify animals, and you have cats, dogs and birds. This means that your model will output 3 units in the form of a vector (call it a list if you prefer).

Each element of the list represents and animal, so for example

Position 0 represents how likely the input is to be a cat
Position 1 represents how likely the input is to be a dog
Position 2 represents how likely the input is to be a bird.

Now imagine you get an input that is a bird, in a happy world, your algorithm should output a vector like

[0, 0, 1]

That is, the input has 0% chance of being a cat, a 0% chance of being a dog and a 100% chance of being a bird.

I reality, this is not so simple, and most likely your output would be something like this

[0.15, 0.1, 0.75]

This means 15% chance of being a cat, 10% chance of being a dog and 75% chance of being a bird. Now, notice that this means your algorithm will still consider the input as a bird, so in terms of classification, sure the output would be correct... but it would not be as correct as it had predicted 100% chance of a bird.

So, the intuition is that the logloss measures how far away you are from perfection, where perfection would be identifying the correct label with a 100% chance and the incorrect labels with a 0% chance.

Final word of advice: Do NOT be afraid of math, you will really need to get the grasp of it at some point, do not let the sum terms  intimidate you, after all they just represent loops in programming.

UPDATE

Let's dive into the math, specially to demystify it.

The logloss formula is given by

$$ LogLoss = - \frac{1}{n} \sum\limits_{i=1}^n [y_i \cdot log(\hat{y_i}) + (1-y_i) \cdot log(1-\hat{y_i}) ] $$

Where

$n$ represents the number of examples, in our case i will use 2 examples.
$y_i$ represents the correct answer for example $i$
$\hat{y}_i$ represents our prediction for example $i$

So, for our example we have two examples, (remember that the examples are denoted by $y_i$) which are

[0, 1, 0] # Example 1: This means the correct answer is dog
[1, 0, 0] # Example 2: This means the correct answer is cat

Now, let's go for our predictions, remember that the predictions are denoted by $\hat{y}$ let's say they are

[0.1,  0.6,  0.3]
[0.85, 0.05, 0.1]

And let's apply the scary formula here. First notice that $$\sum\limits_{i=1}^n$$ just means sum all the elements from i=1 to $n$, in our case, $n$ is the number of examples, and we have two of those.

So, for $i=1$ we have

$[y_i \cdot log_e(\hat{y_i}) + (1-y_i) \cdot log_e(1-\hat{y_i}) ]$

For $i=1$

term1 = [0, 1, 0] * log([0.1, 0.6, 0.3]) + (1-[0, 1, 0]) * log(1 - [0.1, 0.6, 0.3])

For $i=2$

term2 = [1, 0, 0] * log([0.85, 0.05, 0.1]) + (1-[1, 0, 0]) * log (1-[0.85, 0.05, 0.1])

And finally we have

log_loss = (-1/2) * (term1 + term2)

Using sklearn log_loss, the answer is 0.3366

Now, do not get too lost in the math here, just notice that it is not THAT hard, and also do understand that the loss function here tells you essentially how wrong you are, or if you prefer, it measures the "distance" from perfection. I strongly recommend you to code the logloss by yourself (normally numpy is a good option to do so :)


Autoencoder train and test accuracy shooting to 99% on few epochs I am trying to train an autoencoder for dimensionality reduction and hopefully for anomaly detection. My data specifications are as follows.

Unlabeled
1 million data points
9 features

I am trying to reduce it to 2 compressed features so I can have better visualization for clustering.
My autoencoder is as follows where latent_dim = 2 and input_dim = 9
class Autoencoder(tf.keras.Model):
  def __init__(self,latent_dim,input_dim):
        super(Autoencoder32x, self).__init__()
        self.latent_dim = latent_dim
        self.input_dim = input_dim
        self.dropout_factor = 0.5
        self.encoder = Sequential([
                                  # Dense(16, activation='relu', input_shape=(self.input_dim,)),
                                   #Dropout(self.dropout_factor),
                                   Dense(8, activation='relu'),
                                   Dropout(self.dropout_factor),
                                   Dense(4, activation='relu'),
                                   Dropout(self.dropout_factor),
                                   Dense(self.latent_dim, activation='relu')
                                   ])
        self.decoder = Sequential([
                                   Dense(4, activation='relu', input_shape=(self.latent_dim,)),
                                   Dropout(self.dropout_factor),
                                   Dense(8, activation='relu'),
                                   Dropout(self.dropout_factor),
                                   #Dense(16, activation='relu'),
                                   #Dropout(self.dropout_factor),
                                   Dense(self.input_dim, activation=None)
                                   ])

  def call(self, inputs):
    encoder_out = self.encoder(inputs)
    return self.decoder(encoder_out)

Model compilation
ae_train_x, ae_test_x, ae_train_y, ae_test_y = train_test_split(scaled_df[COLUNMS_FOR_AUTOENCODER], scaled_df[COLUNMS_FOR_AUTOENCODER], test_size=0.33)
autoencoder = Autoencoder(latent_dim=2,input_dim=9)
autoencoder.compile(loss='mse', optimizer='adam',metrics=['accuracy'])

Finally training
ae_history = autoencoder_10_32x.fit(ae_train_x, ae_train_y, validation_data=(ae_test_x, ae_test_y), epochs=50)

Output of training
Epoch 1/50
22255/22255 [==============================] - 38s 2ms/step - loss: 0.3330 - accuracy: 0.9646 - val_loss: 0.2816 - val_accuracy: 0.9999
Epoch 2/50
22255/22255 [==============================] - 38s 2ms/step - loss: 0.2664 - accuracy: 0.9999 - val_loss: 0.2818 - val_accuracy: 0.9999
Epoch 3/50
22255/22255 [==============================] - 38s 2ms/step - loss: 0.2649 - accuracy: 0.9999 - val_loss: 0.2845 - val_accuracy: 0.9999

What could be the problem? I think the network is learning to just pass the values. But that should not be possible with the bottleneck and dropout layers. I have also decreased layers but still the result is same. How can I fix it?

The issue turned out to be MinMaxScaler. The issue was sorted after using standardization instead of normalization. (I am not sure as I did not completely verify but best possible guess as compressed output was mostly mapping one feature) One of the feature X had much higher values after normalization then the other features (probably due to outliers in other features). So the the feature X pretty much mostly contributed to the Accuracy.