Why are discriminative models denoted as $P\left(Y|X\right)$? In general, we can speak we know deterministic and probabilistic models. Discriminative ones are deterministic, while generative one is probabilistic.

But I have been reading about the difference between discriminative and generative models where the difference was expressed as:

$P\left(Y|X\right)$ for D. models

and $P\left(X|Y\right) \cdot P\left(Y\right)$ for G. models

My question is: Why do we denote $P\left(Y|X\right)$ for Discriminative models, since these models are completely deterministic, build on top of composite functions with no stochastic processes? Isn't this confusing?

Discriminative models output some tensor $Y$ describing a label given some input tensor $X$ describing some input data. The method of mapping $X$ to $Y$ might be deterministic or not - there's no need for it to be (consider the example of taking a deterministic model and adding a small amount of random noise to it, so that it is perhaps $1\%$ less accurate and more random). The accuracy of the model is the average accuracy over the validation/testing data. The model estimates the probability that the label is $Y$ given the input data is $X: P\left(Y|X\right)$.

Generative models generate a tensor that is drawn from a learned distribution that is similar (as a result of the learning) to the distribution of some training data. For sequence generation, the probability of a generated tensor ($X$, say) at time t is dependent on the probability that the tensor generated at time step $t-1$ was correct ($P\left(Y\right)$, say) and the probability of the new tensor ($X$) if the previous one is taken as given (probability of $X$ given $Y$, or $P\left(X|Y\right)$). The probability of the recently-generated tensor is $P\left(X|Y\right) \cdot P\left(Y\right)$.

I don't really like the notation in the text you're reading, and I agree with you that it's confusing. However, in answer to your question "Why do we denote $P\left(Y|X\right)$ for Discriminative models": model outputs are always estimates and should ideally reflect the fact that the input data doesn't perfectly predict the class. That is, even deterministic models should communicate their estimates in the language of probability; for example even a deterministic model might give an output that can be interpreted as "$80\%$ of the time, input data with these features means the class is C"


What is the best practice to test a ETL pipeline? In traditional software development practice, before going into production, a piece of code should go through various stages of testing (unit test, integration test, user acceptance test) to secure the stability of the software.

A ETL pipeline, as a piece of code, should also go through these testing steps to build a healthy system.

However due to the nature of ETL process, traditional testing technique may not be applicable.

Is there any reference or guideline specifically focus on testing on ETL pipeline?

I have been writing and testing ETL pipelines for a few years now and there are typically two types of pipelines.

Code only pipelines written in python or whatever
GUI pipelines using tools like SSIS or Informatica

The first set you can test like any code, unit and integration tests and the second you can only really test using integration tests I.e. you deploy the code and run it in a test environment.

I expand on how to unit test code pipelines here: https://the.agilesql.club/2019/07/how-do-we-test-etl-pipelines-part-one-unit-tests/

Essentially you will want unit and integration tests and also monitoring which is a form of continuous testing in production :)


Can feature importance change a lot between models? I have a random forest classifier and Multinomial Naive Bayes. For feature importance, I used gini index for random forest and for Multinomial Naive Bayes I used the coefficients of each feature. Then normalized to compare the two lists but there is a big difference between the two. Is this normal? Something like

RF vs NB

A - C
B - D
C - A
D - B

Is this normal?

It is not surprising.

First, you are using different measures of feature importance. It’s like measuring the importance of people (or simply sorting them) using their a) weight, b) height, c) wealth and d) IQ. With a and b you might get quite similar results, but these results are likely to be different from results obtained with c and d.

Second, the performance of your models is likely to be different. In extreme case, output of one of your models could be completely rubbish (in your case it is more likely to be NB). Then the feature importance metrics produced with such model is not credible. In less extreme scenarios when the difference in models‘ performance is not so dramatic the trustworthiness of importances produced by two different models is more comparable. Still the importances might be quite different due to the first argument, i.e. different language used to capture the importance.

Edit:

You have not asked about it in your question, but there are feature importance  approaches which are model agnostic, i.e. can be applied to any predictive model. For example, check the permutation importance approach, described in chapter "15.3.2 Variable Importance" in The Elements of Statistical Learning


Transformer architecture not working on toy problem My transformer is not working on a toy problem.

Toy problem
Input : Sequence of random integer, one-hot-encoded. Example :
[[0, 0, 0, 1, 0, 0, 0, 0, 0, 0],
 [0, 0, 1, 0, 0, 0, 0, 0, 0, 0],
 [0, 0, 0, 0, 0, 0, 0, 0, 1, 0],
 [0, 0, 0, 0, 0, 0, 1, 0, 0, 0],
 [0, 0, 0, 1, 0, 0, 0, 0, 0, 0]
]

Output : 1 if the first random integer is < 5, 0 otherwise. For the given example, the output would be 1.

Problem
Running my Transformer on this toy problem works perfectly. However, running with a bigger size (one-hot encoding on 800 dimension) does not work anymore : the network always output similar results, no matter the input.
Note : I did change the condition of the output to be x < 800, in order to have class balance.
Why my transformer architecture is not working with this variation of the toy problem ?

Another problem
I also tried to use 2 inputs (similarly generated, with size = 10) and generate the output based on these 2 inputs (1 if the first random integer of both input is < 5 or both > 5, 0 otherwise)

I am not sure but maybe the excess zeroes are the culprit. I guess the results you are getting are just the accumulation of the biases. I would suggest converting your one-hot vectors to dense vectors using an Embedding Layer.


Recreating the sum symbol using python I am currently reading a white paper relating to Expectation-Maximisation (EM) and would like to encode a formula so I can play with it in order to help my understanding. The formula in question is a sum over values and shown below;

I am wondering what the advice on the best way to acheive this would be. I am currently thinking about using a for loop to create some kind of cumulative value as I iterate from t = 1 to m.

In most cases, I would go for NumPy.
Implement a Python function f(t) that calculates the $t$-th summand.
Then run

import numpy as np
result = np.array([f(t) for t in range(1,m+1)]).sum()

This will be very fast, unless $m$ is so large that [f(t) for t in range(1,m+1)] does not fit into memory. In this case, I would follow your approach and use a for-loop:

result = 0
for t in range(1, m+1):
    result += f(t)


Training multiple keras models and combining outputs to determine losses I'm trying to predict the future states of a 1D travelling wave (square, triangle and sawtooth) using a deep learning setup in Keras. The waves are discretised in a 1024 data points. As this gives a quite high input dimensionality for an RNN, the 1024 inputs are reduced to a low dimensional latent space of 10 dimensions using a convolutional autoencoder setup. The RNN is given 5 previous time step latent inputs to predict the next time step. I have trained the autoencoder and RNN seperately and obtained pretty decent results, but ideally could be improved. The following figure gives an example of how I trained the models in a decoupled manner.

My idea to improve this model by coupling the training process. Here the autoencoder is trained with the same loss as the RNN. The following picture gives an idea of what i'm trying to achieve.

My hope is that the latent space will be more suitable for time predictions using this setup, as the autoencoder is penalised for creating a latent space which is hard to do time predictions with.

My problem is that I don't know how to implement the combining of the models in this manner in Keras. Could anyone help me with this?

P.S.

I have asked another question about this topic in the following link which gives some examples of my project and on the decoupled training results.

Other question on Data Science Stack Exchange

In keras a model can have multiple outputs, for example:

model = Model(inputs=[a1, a2], outputs=[b1, b2, b3])

Aand if you have multiple outputs you can have multiple losses, for example:

model.compile(loss=[loss1, loss2, loss3], optimizer='sgd')

By doing this, when you train and backpropagate your model you are using all the losses to train your model since you are minimizing all the losses. In this way you can couple your two models as you suggested.

Additionally compile has the parameter loss_weights in which you can change the weights of the different losses.


Custom loss function which is included gradient in Keras I want to make a custom loss function.
Concretely, I use a 2D Convolutional neural network in Keras.
So far, I've made various custom loss function by adding to losses.py.
However, in this case, I encountered the trouble which is explained later.

I have attempted to make a regressor for image tasks.
The data shape of both input & output are (1,128,128,2), where 1 is mini-batch size, 128 is pixel number and 2 is the number of the channels.

Anyway, I want to add new loss function to this task.
I want to compute the difference of pixel gradients between answer & predicted values and add to loss function.
I tried to make such like below.

def continuity(y_true, y_pred):
    import tensorflow as tf
    import numpy as np
    dx = dy = 1/128
    gridSetting = (128,128)

    u_true = y_true[0,:,:,0]
    v_true = y_true[0,:,:,1]
    u_pred = y_pred[0,:,:,0]
    v_pred = y_pred[0,:,:,1]

    cont_true = np.zeros((127,127))

    for j in range(127):
        for i in range(127):
            cont_true[i,j] = (u_true[i+1,j]-u_true[i,j])/dx + (v_true[i,j+1]-v_true[i,j])/dy

    cont_pred = np.zeros((127,127))
    for j in range(127):
        for i in range(127):
            cont_pred[i,j] = (u_pred[i+1,j]-u_pred[i,j])/dx + (v_pred[i,j+1]-v_pred[i,j])/dy

    cont_true = K.variable(value=cont_true, dtype='float64')
    cont_pred = K.variable(value=cont_pred, dtype='float64')
    cont = K.mean(K.abs(cont_true - cont_pred), axis=-1)
    mse = K.mean(K.square(y_pred - y_true), axis=-1)
    cont_mse = cont+mse
    return cont_mse

Error is written as below.

Traceback (most recent call last):
File "DSC_multi-scale_2D.py", line 107, in <module>
autoencoder.compile(optimizer='adam', loss='continuity')
File "/home/----/anaconda3/envs/tensorflow-only/lib/python3.6/site-packages/keras/engine/training.py", line 332, in compile
sample_weight, mask)
File "/home/----/anaconda3/envs/tensorflow-only/lib/python3.6/site-packages/keras/engine/training_utils.py", line 403, in weighted
score_array = fn(y_true, y_pred)
File "/home/----/anaconda3/envs/tensorflow-only/lib/python3.6/site-packages/keras/losses.py", line 86, in continuity
cont_true[i,j] = (u_true[i+1,j]-u_true[i,j])/dx + (v_true[i,j+1]-v_true[i,j])/dy
ValueError: setting an array element with a sequence.

Probably, in losses.py, the data type of u_true, v_true... doesn't support numpy.

How can I debug/fix this problem?

I think it is necessary to perform all operations using the backend versions, allowing Keras to perform backpropagation on every step of the function. You use the common + for example.... try K.add, which should work (based on the available arithmetic operation of the tensorflow backend).

Also, have a look at a related question, where some of the mechanics around creating a custom loss function in Keras are discussed.

The two main loops in your function that compute the gradients should be candidates for vecotisation, where you could compute the differences in one operation. Try something along these lines:

diffs = K.subtract(u_true[i+1, j], u_true[i, j])

and then the quotient in a second operation:

quot = K.divide(diffs, dx)

and then of course the same for v_true and dy.


What is the difference between these 2 Python Pandas queries I am trying to make sense out of some data, here is the Pandas data frame:

In [1]: import pandas as pd
In [2]: df = pd.read_csv('c1.csv')
In [3]: df.head(2)
Out[3]:
   year     city        country  avg_temp
0  1849  Abidjan  Côte D'Ivoire     25.58
1  1850  Abidjan  Côte D'Ivoire     25.52

All I want is to know if the year is greater than 2000 and  avg_temp is less than 20, then what's the mean() and I get 2 different answers:

In [10]: (df.query('year > 2000')['avg_temp'] < 20).mean()
Out[10]: 0.5629877369007804

In [11]: ((df.year > 2000) & (df.avg_temp < 20)).mean()
Out[11]: 0.03540828203222504

In [12]: ((df['year'] > 2000) & (df['avg_temp'] < 20)).mean()
Out[12]: 0.03540828203222504

I can't figure out which one is correct

Depends on what you are trying to do. What is the mean that you are trying to find. Your query return the mean of the number of correct matching rows divided by all the rows in the dataset.

Query 10: It first filters the data out on year > 2000 and then check the total number of rows in this filtered dataset which have ['avg_temp'] < 20. So if your whole dataset had 100 rows, your filtered dataset now will have lets say 10 rows. After the avg_temp condition, only 5 rows match that criterion. So the mean will be 0.5

Query 11 and 12:

It filters the original dataframe on both the conditions at the same time. SO from the sample example as above, your total number of rows in the dataset is 100 and the number of rows that match the criterion are still 5 so the mean will be 0.05.

So it depends on what you are trying to do here.


How can the performance of a neural network vary considerably without changing any parameters? I am training a neural network with 1 sigmoid hidden layer and a linear output layer. The network simply approximates a cosine function. The weights are initiliazed according to Nguyen-Widrow initialization and the biases are initialized to 1. I am using MATLAB as a platform.

Running the network a number of times without changing any parameters, I am getting results (mean squared error) which range from 0.5 to 0.5*10^-6. I cannot understand how the results can even vary that much, I'd imagine there would at least be a narrower and more consistent window of errors.

What could be causing such a large variance?

In general, there is no guarantee that ANNs such as a multi-layer Perceptron network will converge to the global minimum squared error (MSE) solution. The final state of the network can be heavily dependent on how the network weights are initialized. Since most initialization schemes (including Nguyen-Widrow) use random numbers to generate the initial weights, it is quite possible that some initial states will result in convergence on local minima, whereas others will converge on the MSE solution.


How to train more models on 2 GPUs with Keras? I got 2 GPUs of type NVIDIA GTX 1070 Ti. I would like to train more models on them in such a way that half of the models are trained on one GPU only, and half on the other, at the same time. So as training goes, one model goes to GPU1, the next model goes to GPU2, ... I don't want to train one model on the two GPUs. I use Keras - Python with TensorFlow back-end. Can you please recommend resources where I can see how to do this? Most examples/articles online cover the case only if you want to distribute one model on the two GPUs. Thank you.

I would just create two separate scripts with one set of models that target one gpu and the other set of models target the other gpu. Then run the scripts as separate processes.That would easily get around Python's GIL.


What is the intuition behind using LSTM for classification tasks? LSTM is good for sequence prediction, because it can remember the previous context. What is the rationale behind using it in classification tasks ? In particular, they have used it for the following task:

  The Large Movie Review Dataset (often referred to as the IMDB dataset)
  contains 25,000 highly-polar movie reviews (good or bad) for training
  and the same amount again for testing. The problem is to determine
  whether a given movie review has a positive or negative sentiment.

In other words, can the classification problem be reduced to sequence prediction problem in some way?

I'll go for you questions one by one:

  What is the rationale behind using it in classification tasks?

LSTM Networks for classifcation tasks are mainly due to NLP. In particular, Sentiment Analysis. What is the sentiment/polarization of a tweet? By "understanding" natural language, a model can guess whether it is positive or negative towards a given issue.
Since the sequence of words is fundamental in order to understand natural language, you need to capture the sequence of words in order to understand the text, that you then need to then estimate the sentiment.

--

  Can the classification problem be reduced to sequence prediction
  problem in some way?

I think NLP made the two tasks very related to each other, even though in my opinion they can still be analytically separated, so I wouldn't say that one can be reduced to the other.

However let's say that, once you have implemented a model that is very good at estimating the sentiment of a text, you can then make it generate positive/negative texts. In other words, they are closely related (thanks to NLP) but they are not the same.

Hope this helps, otherwise let me know.


input and output to fully connected layer I have an input vector of size 384. It is to be inputted to FC layer with one hidden layer. The output should be a vector of 3.
How is it possible? What is the math behind the calculations in FC layer?

A Fully Connected network looks like this:

You input layer can be of any size, and your output layer can be of any size as well.

The hidden layer in the middle has a size that you need to define.

Each layer is a Fully Connected layer because it has a connection to all nodes in the previous layer. So output nodes have connections to all nodes in the hidden layer. And, all nodes in the hidden layer have access to all nodes in the input layer.


Why adding combinations of features would increase performance of linear SVM? I have a dataset of ~5000 elements represented by vectors composed by ~30 binary values (0 or 1)
on which I am performing binary classification with SVM with linear kernel (I use the Scikit learn lib).

For curiosity, I tried to add an extra feature that consists in a AND between two others (remember that all my features are boolean). The result was that the performance of the SVM improved. I was surprised by this improvement because the AND operation is equivalent to a multiplication, therefore I would expect that my SVM, as every linear classifier, was somehow naturally already taking into account mutiplications between features.

What is wrong with my theoretic understanding of SVM ?

Multiplication is not a linear operation. Your linear SVM constructs a (hyper-)plane
$$
w_0 = w_1 x_1 + w_2 x_2
$$
for some weights $w_0, w_1, w_2.$

By introducing the AND-feature, you add another dimension:
$$
w_0 = w_1 x_1 + w_2 x_2 + w_3 x_1 x_2.
$$

It might well be that your two-dimensional data set is not linearly separable, but the three-dimensional data set is.

A small addition: Would adding the OR-feature increase performance even further? No, because it is a linear combination of the other three features: $x \vee y = x + y - (x \wedge y)$ where $\vee$ is OR and $\wedge$ is AND.


Representing cyclical features as sin/cos components I'm working on a prediction project where we have a lot cyclical features such as hour of the day, weekday, month, day of year, etc etc.
After some searching I decided to follow the advice here.

Now I have the sin and cos component for every cyclical feature as a separate feature, so month becomes month_sin and month_cos.
However, I don't know for sure whether the model can deal with this correlation, as both components need to be equally weighted in order for the feature to make sense. The model assigns different weights to the sin and cos components after training though. My intuition tells me that this is bad, but I'm not sure what to do about it.

Currently gbm (R) gives the best results. For a gradient boosting model, is it better to force equal weights on the two correlated features, or is it better to let the model figure it out even if it results in different weights on the two components? Or would you suggest an entirely different approach?

as both components need to be equally weighted in order for the feature to make sense

That is not the case.

For instance if $\text{sin}(\theta)$ of the cyclical feature is weighted strongly, it means that the original feature has the strongest positive effect on output at $\theta = \frac{\pi}{2}$.

If the two features are weighted equally, then the focus is around $\theta = \frac{\pi}{4}$

In general, you should expect different weights to apply to the different components, depending on which hours of the day maximise the target variable and which minimise it.

  For a gradient boosting model, is it better to force equal weights on the two correlated features, or is it better to let the model figure it out even if it results in different weights on the two components?

Definitely better to let the model figure it out in this case.

Your main concern would be whether a one-hot-encoded representation might be better than a cyclical representation for your problem. One-hot allows for arbitrary relationships to each hour, but adds more dimensions - and so may require more examples of each time. Cyclical has less dimensions, but is more likely to include non-linear effects, if for instance min and max effect on output target variable are not exactly 12 hours apart - thus it may require a more complex model.


Connecting Infusionsoft data to Google data studio I want to create a Google Data studio dashboard from Infusion soft data.
The main problem are the connectors - there are multiple tools that provide direct connectors but they are paid solutions like Klipfolio, Clicdata, Grow etc.

If a direct connection is not possible, I want to use some combination of Google sheets and Zapier or other free tools to create a data flow that can be constantly refreshed for data coming in from "infusionsoft" to "Google data studio"

Questions summary:

Is there any direct way of connecting Infusionsoft data or API with Google data studio (maybe with some Database like MySQL/postgresql/cloud sql etc?)
What are some free mid-tier tools I can use to create this data connection?(Zapier, google sheets, excel etc)

All your suggestions are highly appreciated.

I'm starting to look at this now, to start with simply on batched/CSV export/import process pushing data into Google BigQuery first and using it as a source in DataStudio. I've thought about how to automate and I'm not sure Zapier will cut it - it doesn't seem to have an action for Contact 'updates' for one, which is a showstopper for us (though you don't mention what objects you want to query).

Another option would be to use the Novak Solutions Data Warehouse service to get everything into SQL and either query directly from DataStudio or replicate into BigQuery using StitchData or similar.

If you want a cheaper option still, it could be done via the Infusionsoft API, but you'd have the time and cost of creating the synchronisation tasks and managing changes. If you have a limited subset of objects (eg. Contacts + Tags) this might be cost effective.


Equation of Linear Regression I performed Linear Regression on the data set with two attributes (Salary and YearsExperience) using orange data mining tool.

Salary is a dependent variable and YearsExperience is an independent variable.

Can I find the equation : Salary = b0+YearsExperience*b1.

Depending on what language you are using you can just print the model. In R it looks like this:

summary(my.model)

The output will look like this, or similar:

##
## Call:
## lm(formula = dist ~ speed.c, data = cars)
##
## Residuals:
##     Min      1Q  Median      3Q     Max
## -29.069  -9.525  -2.272   9.215  43.201
##
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)
## (Intercept)  42.9800     2.1750  19.761  < 2e-16 ***
## YearsExp     3.9324     0.4155   9.464 1.49e-12 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 15.38 on 48 degrees of freedom
## Multiple R-squared:  0.6511, Adjusted R-squared:  0.6438
## F-statistic: 89.57 on 1 and 48 DF,  p-value: 1.49e-12

Your betas are under the "Estimate Std." Column. Your beta0 is (Intercept) and your beta1 is YearsExp (or whatever your variable is) etc... If you have more than one variable there will be more in this column for you to see.

After you get the betas you can write a function to apply your model to new data like this:

y-hat <- 42.9800 + YearsExp*3.9.324

Model summary copypastad w/ variable name edit from: https://feliperego.github.io/blog/2015/10/23/Interpreting-Model-Output-In-R


adding a supervising process during knn process I am trying to improve my KNN regression process (I would like to use sklearn / python, but it doesn't matter).I would like to improve my results and to gain insight. Here is an example:
I have data measured from an electric motor: an input voltage (U) and current (I) and an output torque (T) and speed (S).
First intend is a simple approach where I'm giving those data in the state to a KNN algorithm and I use the results. But every result is not logical (even if they are statistically close).
if you add a knowledge layer during the process, a "human in the loop approach", it must be better! For example here, you know that the input power is Pin = UI and the output power is Pout = TS. The efficiency of the system eff = Pout / Pin and the efficiency cannot be higher than 1. Whereas KNN results can generate results with eff > 1.
My question is, how to use this knowledge (additional condition/human in the loop approach) during my KNN learning process phase to improve my results? Do I continue to learn on the initial data or transformed data? Do I modify the learning process? Is it possible to add "supervisor condition" that are influencing the learning process?
Thank you for your help!

KNN Regressor, per se, does not have a dedicated learning process as it just takes weighted average of the dependent variable of the k-nearest neighbors from your training set of the test datapoint whose dependent variable you have to predict.

You can try using Support Vector Regressor or MLP (Neural Network) Regressor offered by scikit-learn and see if the performance has increased with respect to number of test cases where predictions resulted in efficiency > 1.

If you are adamant on incorporating this condition in your learning process, you can use Keras and create a custom loss function which returns a high value if your efficiency results in a value greater than 1. Although the training will be sensitive to this value and higher values can shutdown the neurons, resulting in all zeros output. But if you are careful you will be able to get sensible predictions with efficiency < 1.

def custom_loss(voltage, current, torque, speed):
    def loss(y_true, y_pred):
        #your logic here
    return loss

model = Sequential()
model.add(Dense(16, kernel_regularizer=regularizers.l2(0.001), activation='relu'))
model.add(Dense(1))
model.compile(loss=custom_loss(input_1, input_2, input_3, input_4), optimizer='adam')


Really bad value of Val loss I am using GTZAN dataset to make a CNN and classify by musical genres.
I'm getting very good results except Val. loss (See Image)

I am processing the audio files using Librosa, obtaining the spectogram and then using the power_to_db function.
This is my CNN Model:
class CNNModel(object):

def __init__(self, config, X):
    self.filters = 32 # number of convolutional filters to use
    self.pool_size = (2, 2)  # size of pooling area for max pooling
    self.kernel_size = (3, 3)  # convolution kernel size
    self.nb_layers = 4
    self.input_shape = (128, 625, 1) # cambiar por x.shape

def build_model(self, nb_classes):

    model = Sequential()
    model.add(
            Conv2D(
                self.filters,
                self.kernel_size,
                padding ='same',
                input_shape = self.input_shape))
    model.add(BatchNormalization(axis=1))
    model.add(Activation('relu'))

    model.add(
            Conv2D(
                self.filters,
                self.kernel_size))
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size = self.pool_size))
    model.add(Dropout(0.25))

    model.add(
        Conv2D(
            self.filters + 32,
            self.kernel_size,
            padding ='same'))
    model.add(Activation('relu'))

    model.add(
        Conv2D(
            self.filters + 32,
            self.kernel_size,
            padding ='same'))
    model.add(MaxPooling2D(pool_size = self.pool_size))
    model.add(Dropout(0.25))

    model.add(Flatten())
    model.add(Dense(512))
    model.add(Activation('relu'))
    model.add(Dropout(0.5))
    model.add(Dense(nb_classes))
    model.add(Activation("softmax")) #mirar

    return model

I leave the link of my github in case you want to see the whole code.
Every song is (128, 625) Shape, I used MinMaxScale to Scale the data.
This is my loss function and my optimizer
loss = losses.categorical_crossentropy,
optimizer = optimizers.SGD(lr=0.001, momentum=0, decay=1e-5, nesterov=True)

I have read about overfitting and that seems to be the cause but I do not know how to solve it at the code level.
Update 1: With Dropout(0.9) I get this results:

Thank you

You can test with higher values of Dropout (0.5,0.7,0.9) and/or try L1/L2 Regularization to combat overfitting  : Keras Regularizers.

Update:
You can play with a combination of l1/l2 regularization and dropout for your convolutional and FC layers. Start with low values of lambda (0.001) and increase it thereon. A common practice is to have dropout only in the FC layers, see if it helps your problem.

Also, from the looks of your loss curve, your model probably hasn't converged.You can train it until your validation loss starts going up/becomes stable.


Choosing a learning rate I'm currently working on implementing Stochastic Gradient Descent, SGD, for neural nets using back-propagation, and while I understand its purpose I have some questions about how to choose values for the learning rate.

Is the learning rate related to the shape of the error gradient, as it dictates the rate of descent?
If so, how do you use this information to inform your decision about a value?
If it's not what sort of values should I choose, and how should I choose them?
It seems like you would want small values to avoid overshooting, but how do you choose one such that you don't get stuck in local minima or take to long to descend?
Does it make sense to have a constant learning rate, or should I use some metric to alter its value as I get nearer a minimum in the gradient?

In short: How do I choose the learning rate for SGD?

Is the learning rate related to the shape of the error gradient, as
it dictates the rate of descent?

In plain SGD, the answer is no. A global learning rate is used which is indifferent to the error gradient. However, the intuition you are getting at has inspired various modifications of the SGD update rule.

If so, how do you use this information to inform your decision about a value?

Adagrad is the most widely known of these and scales a global learning rate η on each dimension based on l2 norm of the history of the error gradient gt on each dimension:

Adadelta is another such training algorithm which uses both the error gradient history like adagrad and the weight update history and has the advantage of not having to set a learning rate at all.

If it's not what sort of values should I choose, and how should I choose them?

Setting learning rates for plain SGD in neural nets is usually a
process of starting with a sane value such as 0.01 and then doing cross-validation
to find an optimal value. Typical values range over a few orders of
magnitude from 0.0001 up to 1.

It seems like you would want small values to avoid overshooting, but
how do you choose one such that you don't get stuck in local minima
or take too long to descend? Does it make sense to have a constant learning rate, or should I use some metric to alter its value as I get nearer a minimum in the gradient?

Usually, the value that's best is near the highest stable learning
rate and learning rate decay/annealing (either linear or
exponentially) is used over the course of training. The reason behind this is that early on there is a clear learning signal so aggressive updates encourage exploration while later on the smaller learning rates allow for more delicate exploitation of local error surface.


Choosing the right data mining method to find the effect of each parameter over the target I am dealing with a lot of categorical data right now and I would like to use an appropriate data mining method in any tool [preferably R] to find the effect of each parameter [categorical parameters] over my target variable. To give a brief notion about the data that am dealing with, my target variable denotes the product type [say, disposables and non-disposables] and I have parameters like root cause,symptom,customer name, product name etc. As my target can be considered as a binary value, I tried to find the combination of values leading to the desired categories using Apriori but, I have more than 2 categories in that attribute and I want to use all of them and find the effect of the mentioned parameters over each category. I really wanted to try SVM and use hyperplanes to separate the content and get n-dimensional view. But, I do not have enough knowledge to validate the technique, functions am using to do the analysis. Currently I have like 9000 records and each of them represents a complaint from the user. There are lot of columns available in the dataset which is what I am trying to use to determine the target variable [ myForumla <- Target~. ] I tried with just 4 categorical columns too. Not getting a proper result.

Can just the categorical variables be used to develop a SVM model and get visualization with n hyper planes? Is there any appropriate data mining technique available for dealing with just the categorical data?

You can try Bayesian belief networks (BBNs). BBNs can easily handle categorical variables and give you the picture of the multivariable interactions. Furthermore, you may use sensitivity analysis to observe how each variable influences your class variable.

Once you learn the structure of the BBN, you can identify the Markov blanket of the class variable. The variables in the Markov blanket of the class variable is a subset of all the variables, and you may use optimization techniques to see which combination of values in this Markov blanket maximizes your class prediction.


In supervised learning, how to get info from correlation? I am trying to build a classification model so I have tried to check the correlation between the features.

Here Loan_Status is my target variable.

I just don't know how to extract information from this? Please help.
I have questions like.
Is -0.0047 corelation of ApplicantIncome with Loan Status useful?

To help you, that shows the correlations between features and each other feature. For example, the number one in the image that you gave, is shown all along the diagonal part of the matrix. The ones represent the 100% linear correlation with one of the features and one of the other features. This image might help:

As you can see negative correlations mean that as one feature value increases, the other feature value decreases. A correlation of 0 means that the features appear to have no linear correlation.

If you want to merely concentrate on the correlations between the label and the features, then here is some python code (the language I assume you are using) to help you:

# Your data should be a pandas core dataframe

yourdata = ...

# To find correlations, use the corr() function
corr_matrix = yourdata.corr()
corr_matrix["your label"].sort_values(ascending=False)
# This should print out a correlation list.
# If it doesn't then wrap the last line of code in print( )
# You are going to notice that some features will be missing from the list.
# That is because the corr() function does not return any discrete features.
# If you still have every feature, then every one of your features are continuous.


Naive bayes expectation maximization vs logistic regression for binary classification Assuming I'm dealing with binary classification.
For what kind of data Naive bayes using expectation maximization would give a better solution and for what kind of data logistic regression would be the better choice?

On a very high level -
Naive Bayes is a probabilistic model and it is scale invariant. That means scaling the data won't affect your model's performance. It is mostly used for Natural Language Processing based use cases.
Logistic Regression is a geometric model based on decision boundary. This means scaling the data will affect your model's performance. It is mostly used for traditional row column based datasets.
In Depth explanation can be found here -
https://dataespresso.com/en/2017/10/24/comparison-between-naive-bayes-and-logistic-regression/


Using a past time series to predict how a present time series will pan out? Let's say that I have past data indicating how some time series panned out. Now I also have the beginnings of a new time series that I expect to pan out in a similar trend to the old one.

What are some general data science / machine learning / statistics techniques I can use to predict the rest of the new time series? What about using some sort of online model to iteratively predict the old model's trend, and then somehow applying that to the new model?

Here you go: https://otexts.com/fpp2

But my experience is you should start using GBM based models for time series forecasting to get really good forecasts.

Check out XGBoost, Prophet, BSTS models for time series forecasting.


Trouble representing a problem I have a problem and I'm having trouble representing it - first I thought I should use graph theory (nodes and edges) and now I'm not sure.

My data is some tanks names and it's volumes, those tanks are connected by pipelines which I have the names and length.

------(pipeline 1)------.-----(pipeline 2)------.----(pipeline 3)---
  |                     |    |                           |         |
[R tank 1]        [S tank 1] [S tank 2]            (pipeline 4) [S tank 3]
                                                         |
                                                     [S tank 4]

R tank is sink (receiver) and S tank is source (sender)

Problem is the pipe names change doesn't occur where there is a tank - they change name because historical reasons, size or connections...

So if I want to graphically show that S tank 2 is connected to pipeline 2 at point X and pipeline 2 connects to pipeline and the content goes to R tank 1, how should I do this? (I think the point X may not be relevant but if I had some way to get the distance travelled would be great).

I would definitely use a graph (Though, this clearly depends on the final application, maybe you could add more information)

For the nodes, you should consider as nodes not only tanks but also points were pipelines change name or bifurcate. For instance, following your example:

    e1     e2        e3          e4        e5          e7
    +---+----------.---------+----------.----------+-----------
    |          |         |                     |          |
[R tank 1]  [S tank 1] [S tank 2]              |e6     [S tank 3]
                                               |
                                           [S tank 4]

Now adding the nodes:

    e1 n2  e2     n4   e3   n6   e4    n8  e5     n9   e7
n1 +---+----------.---------+----------.----------+----------+
   |          |         |                     |              |
  n3         n5        n7                     |e6           n11
                                              |
                                             n10

Lastly, you need some kind of mapping.
Some of the nodes will map to tanks:

[R tank 1]     n3
[S tank 1]     n5
[S tank 2]     n7
[S tank 4]     n10
[S tank 3]     n11

And the pipelines will be represented by paths in the graph

Pipeline 1     e1 e2
Pipeline 2     e3 e4
Pipeline 3     e5 e7
Pipeline 4     e6


Would Deep Q Learning work for a finite horizon problem? I want to apply Deep Q Learning to a problem, which has a clear finite horizon definition, like:

$$V(s) = \mathbb{E}[r_1 + r_2]$$

Since the horizon is finite, I do not use reward discounting. My action spaces at the time steps 1 and 2 do change and also the policies are therefore clearly not stationary as well. But the state and reward transitions are deterministic as: $r_i(s_i,a_i)$ and $s_{i+1}(s_i,a_i)$. The state space is also very large, so tabular methods are out of question. So the value function can be defined as:

$$V(s) = \sum_{a_1}\sum_{a_2}\left(r_1 + r_2\right)\pi_1(a_1|s_1=s)\pi_2(a_2|s_2(s_1=s,a_1))$$

since the only randomness comes from the policy distributions. (I dropped the reward functions' depedency on the states and actions for clarity). My question is, would Deep Q Learning work for such a finite horizon case? I plan to use two separate MLPs for the Q functions at time steps 1 and 2. I know that the Bellman Optimality can be shown both for the finite and infinite cases; in the finite case, it does not even need the stationarity of the distributions. But all examples on DQNs I came accross during my research solves problems with the infitine horizon assumption. So I just wanted ask for some insight there.

DQN solves optimal control problems for maximising average reward. Although it typically uses discounted reward, the discount factor is not part of the setting - instead it is part of the solution hyperparameters, and usually set quite high e.g. 0.99 - when using function approximators.

The TD target used in DQN is a problem for you:

$$G_{t:t+1} = R_{t+1} + \gamma \text{max}_{a'}Q(S_{t+1},a')$$

as it relies on a Bellman equation that no longer holds for the given value functions. In your case, there does not seem to be any way to express the TD target for time $t$ by referencing other Q values, as you would need to then subtract future rewards, which is very ungainly. Instead, you could use a simple truncated Monte Carlo return

$$G_{t:t+3} = R_{t+1} + R_{t+2} + R_{t+3}$$

There are a few different ways you could do this, but the simplest and closest to DQN would IMO be to:

Store trajectories in order in the experience replay table
For each item in the training mini-batch:

Pick a start state/action pair to assess $s_t, a_t$ randomly from replay table
Check that $a_{t+1}$ and $a_{t+2}$ are maximising actions in your current target policy for $s_{t+1}$ and $s_{t+2}$, reject the sample if not (note you don't need to check or reject $a_t$, and that is how your code learns about exploratory actions)
Calculate the TD target, $g_t = r_{t+1} + r_{t+2} + r_{t+3}$
Your training data for that example is $s_t, a_t, g_t$

The checking for maximising action parts could be quite slow, so you might prefer to simplify the approach and not use off-policy. Alternatively, if $epsilon$ is low enough you could just store the three step returns directly in the experience replay table (wait until you have the data from $t+3$ before storing data for $t$) and ignore the fact that some returns are from exploratory actions, thus noisy/biased . . . this approach is used in n-step returns in DQN "Rainbow" version and works well enough in practice on the Atari problems despite being on shaky theoretical ground.

Note I am using the convention $s_t, a_t, r_{t+1}, s_{t+1}$ to represent a step in the trajectory, whilst in the question you appear to be using $s_t, a_t, r_t, s_{t+1}$ with a different reward index. You will need to convert back if you want to stick with your convention.


How to use features when predicting aggregation? Let's say I have to predict the total monthly sales of a store. I have the data in the following format:

I don't think I can use this data directly for predicting the sales for a month, as in it's current form there is no target variable present to predict. We would have to sum the sales for each month and only then will we get our target variable (the sales that we want to predict).

Please see when I group the sales by month (Jan and Feb):

Now the above is data I can use with ml algos to predict the Total_Sale which is my target variable. But, since now I have grouped the sale by month, I have lost all the features like Gender, Age, Product etc... I cannot explain these features and see their impact on the sale.

Is there any way around this?

Thanks

You can pass the entire month's data as a single sample, which should predict the sales of that month. It would look like an index with multiple indexes. Something like MultiIndex in python's Pandas library.

Another idea would be to still try predicting the sales every day, then just sum up all predictions at the end to create your monthly aggregation.

Using something like a neural network may be overkill - it depends how much data you have - but will offer the flexibility to easily train a model with multiple inputs and a single output, as is your case.


GradientBoostingRegressor Text Classifier I am working to build a text classifier using a Boosting method from sklearn. It is performing quite well, at around 97% accuracy on my test data. However, the problem I am seeing is that if I input text that clearly does not fall into a predefined category, it will randomly assign it to a certain classification with a high probability score
For example:
X_train = df_train.text
X_test = df_test.text
y_train = df_train.label
y_test = df_test.label

boosting = Pipeline([('vect', CountVectorizer()),
                  ('tfidf', TfidfTransformer()),
                  ('boosting', GradientBoostingClassifier()),])
boosting = boosting.fit(df_train.text, df_train.label)

list of categories -> {'A': fruit, 'B': animal, 'C': car, 'D': person, 'E': dessert, 'F': place}
docs = ['this is not category']

boosting.predict_proba(docs).tolist()

Output:
[[0.0016872033185414193,
  0.9915417761339475,
  0.0016865302624752719,
  0.0016921961567993337,
  0.0016974399174602914,
  0.001694854210776211]]

You can see that the second category is receiving a .99 probability when it is clearly not fitting into any of the options. Regardless of what I put through it, could be "fdahsjfkasl" it will return the same probability score for that second category
The model works so well for text that could logically fit into a category (not only performing well on test data, but also on new/random text too), but i need a way to handle text that does not, so that it can be labeled "Not a category" or something like it.
Does anyone have any suggestions?

This will always happen since in multi-output classification (OnevsOne or OnevsRest) classes are mutually exclusive i.e probabilities will always sum up to one.
If you want independent probabilities you could try rewrite your target so that this is a multi-output classification (not a multi class) problem
toy example with iris data:
from sklearn.datasets import load_iris
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.multioutput import MultiOutputClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder

X, y = load_iris(return_X_y= True)

X_train, X_test, y_train, y_test = train_test_split(X,y,random_state= 42, stratify= y)

model = RandomForestClassifier(random_state= 42).fit(X_train,y_train)

pd.DataFrame(model.predict_proba(X_test)).head(5)

Output:

Rewriting in a multi-output classification:
y_train = y_train.reshape(-1, 1)
y_test = y_test.reshape(-1, 1)

encoder = OneHotEncoder(handle_unknown= "ignore").fit(y_train)

y_train_multiclass = np.array(encoder.transform(y_train).todense())
y_test_multiclass = np.array(encoder.transform(y_test).todense())

model = MultiOutputClassifier(estimator= RandomForestClassifier(random_state= 42)).fit(X_train,y_train_multiclass)

pred = np.array(model.predict_proba(X_test))

pred[:,:,1].shape

prob0 = pred[:,:, 0].T
prob1 = pred[:,:, 1].T

pd.DataFrame(prob0).head(5)

Output:

Yo can see that in this approach the probabilities might not sum up to one, so it may happen that if you pass a document that does not belong to any category I should return 0 for all the classes
Other option would be to use a neural net model and changing the utput via the activation function of the output layer as mention on this answer:
Should estimated probabilities from multi class classification sum to 1


Q-learning why do we subtract the Q(s, a) term during update? I can't understand the meaning of $-Q(s_t, a_t)$ term in the Q-learning algorithm, and can't find explanation to it either.

Everything else makes sence. The q-learning algorithm is an off-policy algorithm, unlike SARSA. The Bellman equation describes q-learning as follows:

$$Q(s_t, a_t) \leftarrow Q(s_t, a_t) + \alpha\left[ r_t + \gamma \cdot argmax(Q(s'_t, a'_t)) - Q(s_t, a_t) \right] $$

"The q value for action $a$ taken in state $s$ at time $t$ becomes equal to: that same q-value plus small amount of: currently received reward (reward could be zero) with some amount $\gamma$ of the best Q-value available in the new state, minus our current value

To me, this $-Q(s_t, a_t)$ term at the very end is redundant. ...If we set gamma to $0.8$, the future rewards will decay anyway.

Yes, if instead we now set $\gamma=0$, then the $-Q(s_t, a_t)$ term will drag our value down - is there a case when it would be useful and what would the result be?

Edit:

wikipeda uses a slighly different form of the bellman equation

$$Q(s_t, a_t) \leftarrow (1-\alpha)\cdot Q(s_t, a_t) + \alpha\left[ r_t + \gamma \cdot argmax(Q(s'_t, a'_t)) \right] $$

It's the same equation as the one above, because we can multiply $Q(s_t, a_t)$ with an $\alpha$ and then factor it out, obtaining the first equation.

This representation makes me kind-off understand that we are linearly interpolating from current to newer Q-value, but I can't tie it to the original representation ...In the original representation (the first equation), it magically seems that gamma would be enough - can someone clear it up for me?

The wikipedia formulation does indeed show you a better view of how the update rule for action values is constructed:

$$Q(s_t, a_t) \leftarrow (1-\alpha)\cdot Q(s_t, a_t) + \alpha\left[ r_t + \gamma \cdot \text{max}_{a'}(Q(s_{t+1}, a')) \right] $$

. . . here you can see that you are taking a weighted average between $Q(s_t, a_t)$ and $ r_t + \gamma \cdot \text{max}_{a'}(Q(s_{t+1}, a')$. An aside: In both cases you have written $\text{argmax}$, where the actual term is $\text{max}$.

The intuitive reason for the update is because the second term contains a new estimate of the true value of Q from the environment - the immediate reward $r_t$, and the actual state transition that occurred $s_{t+1}$ are observed data that are part of the new estimate. The term  $r_t + \gamma \cdot \text{max}_{a'}(Q(s_{t+1}, a'))$ is usually called the Temporal Difference Target or just TD Target.

  In the original representation (the first equation), it magically seems that gamma would be enough - can someone clear it up for me?

The $\gamma$ term is the discount rate, and nothing to do with convergence of Q-learning or SARSA. It is a term used to control preference for more immediate rewards (low values) or longer-term rewards (high values), and as such is part of the problem definition. It is not a learning parameter like $\alpha$, the learning rate. And in fact $\gamma = 0$ is possible (rarely used, but meaning that only next reward is important); $\gamma = 1$ is also possible and quite a common choice for episodic problems. Clearly when using $\gamma = 1$, there is no "decay of future rewards" happening.

What is actually happening is that the TD Target is a stochastic estimate for the correct Q action value (in fact with all TD learning such as SARSA and Q-learning, this is a biased estimate, although the bias should reduce as the system approaches convergence). If you rename $Q(s,a)$ as $Q_{so\_far}$ and $r_t + \gamma \cdot \text{max}_{a'}(Q(s_{t+1}, a'))$ as $Q_{latest\_estimate}$, you get the idea of the update:

$$Q_{updated} \leftarrow (1-\alpha)Q_{so\_far} + \alpha Q_{latest\_estimate}$$

which is the same as

$$Q_{updated} \leftarrow Q_{so\_far} + \alpha(Q_{latest\_estimate} - Q_{so\_far})$$

You might ask "Why is the TD Target an estimate of $Q(s_t, a_t)$?"

The answer can be derived from the definition of Q, and is called the Bellman equation - in this case the Bellman equation for action value under the optimal policy $\pi^*$ (anther aside: what you have called the Bellman equation is not, although it is related). I won't give the full derivation here, but in essence the definition of Q is "the expected total reward when starting with state $s_t$ and action $a_t$ then following the optimal policy form then on", or

$$q_{*}(s, a) = \mathbb{E}_{\pi^*}[\sum_{k=0}^{\infty}\gamma^k R_{t+k+1} | S_t = s, A_t = a]$$

and from this definition, you can extract the first reward term and write out $q_*(s_t, a_t)$ in terms of $q_*(s_{t+1}, a_{t+1})$.

$$q_{*}(s, a) = \sum_{r,s'} p(r,s'|s,a)(r + \gamma \text{max}_{a'} q_{*}(s', a'))$$

Where $p(r,s'|s,a)$ is the transition probability of getting reward $r$ and ending up in state $s'$ when starting in state $s$ and taking action $a$.

Note this is an equality, not an update rule. Q-learning and SARSA turn it into an update rule by sampling the right hand side, and using that sample to update the current best estimate of the left-hand term. Note that by taking a sample, you don't need to know $p(r,s'|s,a)$ . . . that is what is meant by "model free" learning, as if you knew $p(r,s'|s,a)$ then you would have a model of the underlying Markov Decision Process (MDP).


How to search collection of podcasts (.mp3 files)? I have a collection of podcasts, in particular the "Talkingmachines" Podcasts ("Human conversation about machine learning"), see http://www.thetalkingmachines.com/blog/

There are 22 episodes, each 1 hour long. MP3 files are available on my local disk.

Now  when I want to search the audio files for the episode where they talk about, say, the kmeans++ algorithm , what should I do?

Specifically, I want to search for that episode where a male voice (Ryan) talks about kmeans++ in the first half of the podcast.

Again, what should I do? Is there a desktop tool available that can do this search? Should I upload the files to some webservice that can do audio recognition?

Transcribe then topic model the recordings. This will let you know which podcast talks about the subject of interest, then you can search the transcript. If you really need it to pinpoint the moment when the subject was discussed, you'll need a temporal topic model like so but that would add a good bit of complexity.

Google has a good service for the first part. You can use an open source library like gensim for the second part.


Interactive Data Visualisations So I have finally gotten a hold of ggplot2 and ggvis packages and tried out shiny too to have interactive graphics and visualizations. but i have been using Power BI before and came to R as a an good alternative for creating effective visualizations. But there is one thing that has been missing in R which is to be able to filter out all visualizations on a page on the basis of a mouse click in one of the visualizations on the page. I understand how we can create slicers and other ways to filter graphics using shiny or better with ggvis.

So can anyone suggest a package or something i have missed in the above packages to help me achieve this.

You could give the plotlypackage a try. You basically code up the plot you want first with ggplot2 and then call ggplotly() at the end, which will render an interactive version of it. You get zooming in on mouse selection for example and labels of points on hover. It has a pretty extensive documentation too. I can try and help you out with a specific example if you post some code. Another cool package you can check out is trelliscopejs. hth


Text classification problem using Python or R I am a novice in machine learning and new to NLP. I am looking for ideas on how to solve the below two problems.

I have a dataset with two columns, "Titles" and "Description". Titles column has names of clinical lab tests and description column has description about results of corresponding laboratory test( can be seen below). There many titles specific to a particular lab test.

Title                       Description
Complete blood test         RBC: Normocytic and Normochromic
COMPLETE Blood test\        Platelets: Adequate on the smear
Blood glucose               COLOUR - COLOURLESS
Complete blood picture      WBC: Total and Differential counts are within normal limits

I have only shared a small part of the data frame.

Problem 1: I have manually grouped the similar looking titles. For example, I have grouped the titles in the above data frame as "Blood Test". Is there a way to use NLP technique to group similar looking titles ( As shown below).

Problem 2: Based on the description, i have manually labeled a particular outcome for a lab test as normal or abnormal. Again i am looking for a way to do this without having to manually label the outcome (As shown below).

 Title                       Description                           Outcome
Blood Test                   RBC: Normocytic and Normochromic       Normal
Blood Test                   Platelets: Adequate on the smear       Normal
Blood Test                   COLOUR - COLOURLESS                    Normal
Blood Test                   WBC: Total and Differential counts     Normal

Any suggestions or resources to help me get started would be appreciated.

Take a supervised approach:

For each group "Blood test", "Stool test" etc,

Take a subset of your rows as a training set, say 200 rows.
Create a new numeric or logical column, "IsBloodTest" or similar for this new subset of your dataset

For each row in these 200

Classify them manually: if it is a blood test, assign 1, if not assign 0 to "IsBloodTest"
Split the "Description" column into word vectors. This creates a Document-Term matrix with only 1 or 0 values in the cells (Word is present/not present)
Classify the rows of the sparse new matrix (omitting your own newly manually created attribute) with the Multinomial NaiveBayes algorithm. This assigns another column with "0/1" prediction, say "PredictedBloodTest"  to  your training set.
With these 0/1 values in the two new columns you can build a confusion matrix.

Perform Cross validation experiments to check your confusion matrix, or use an extra validation set. Label more rows manually if necessary.

If the classification result is good enough, use it for the remaining 1000s of rows from your data set.

This procedure is simple, but it might get you results faster than learning more sophisticated approaches first.


Why use convolutional NNs for a visual inspection task over classic CV template matching? I had an interesting discussion come up based on a project we were working on: why use a CNN visual inspection system over a template matching algorithm?

Background: I had shown a demo of a simple CNN vision system (webcam + laptop) that detected if a particular type of object was "broken"/defective or not - in this case, a PCB circuit board. My CNN model was shown examples of the proper and broken circuit boards (about 100 images of each) on a static background. Our model used the first few conv/maxpool layers of pre-trained VGG16 (on imagenet), and then we added a few more trainable convs/pools, with a few denses, leading to a dim-3 one hot encoded vectored output for classification: (is_empty, has_good_product, has_defective_product).

The model trained pretty easily and reached 99% validation acc no problems; we also trained with various data augmentation since we know our dataset was small. In practice, it worked about 9 times out of 10, but a few random translations/rotations of the same circuit board would occasionally put it in the opposite class. Perhaps more aggressive data augmentation would have helped. Anyways, for a prototype concept project we were happy.

Now we were presenting to another engineer and his colleague, and he brought up the argument that NNs are overkill for this, should just use template matching, why would one want to do CNNs?

We didn't have a great answer for why our approach could be better in certain applications (e.g. other parts to inspect). Some points we brought up:

1) More robust to invariances (through e.g. data augmentation)

2) Can do online learning to improve the system (e.g. human can tell the software which examples it got wrong)

3) No need to set thresholds like in classical computer vision algorithms
What do you guys think, are there more advantages for a CNN system for this type of inspection task? In what cases would it be better than template matching?

A few more random ideas for when deep NNs could be the tech for the job: for systems that require 3D depth sensing as part of the input, or any type of object that can be deformed/stretched/squished but still be "good" and not defective (e.g. a stuffed animal, wires, etc). Curious to hear your thoughts :)

The answer depends on the task.  Template matching can work for some tasks but not for all.  CNNs potentially have the ability to generalize to unseen inputs that don't match any of your templates, so can potentially generalize better.

But whether CNNs will beat template matching will depend on the specific task and what specifically you're trying to achieve.  This is an empirical science; ultimately, the way you find out which works better is to try them both -- or learn from others who have tried them (e.g., by reading the literature).  I don't think you're going to find some theory or taxonomy that is going to substitute for empirical evaluation on real-world data.


How do I get the name of Sagemaker Estimator's job I'm having a stumbling block with SageMaker. How do I know what my job name is?

For example:

mnist_estimator = MXNet(entry_point='/home/ec2-user/sample-notebooks/sagemaker-python-sdk/mxnet_mnist/mnist.py',
                        role=role,
                        output_path=model_artifacts_location,
                        code_location=custom_code_upload_location,
                        train_instance_count=1,
                        train_instance_type='ml.m4.xlarge',
                        base_job_name=’foo’,
                        hyperparameters={'learning_rate': 0.1})

and then when I call fit() it prints out:

  INFO: Creating training job with name: foo-2018-01-10-20-13-57-893

and when I look in my S3 bucket I see:

  2018-01-10 15:20:45     411784 artifacts/foo-2018-01-10-20-13-57-893/output/model.tar.gz

So my job name is "foo-2018-01-10-20-13-57-893" and not "foo", but how can I get this from python? I guess I'm looking for a way of extracting that from the Estimator itself, but I'm just blanking on it. I'm reading the python source but that's not helping me, probably because I'm just learning python also.

The _current_job_name contains the name of the job. So in the example from the question:

print(mnist_estimator._current_job_name)

would print

foo-2018-01-10-20-13-57-893


I think this covers your issue in the Keras documentation
https://keras.io/callbacks/#create-a-callback

class LossHistory(keras.callbacks.Callback):
    def on_train_begin(self, logs={}):
        self.losses = []

    def on_batch_end(self, batch, logs={}):
        self.losses.append(logs.get('loss'))

model = Sequential()
model.add(Dense(10, input_dim=784, kernel_initializer='uniform'))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer='rmsprop')

history = LossHistory()
model.fit(x_train, y_train, batch_size=128, epochs=20, verbose=0, callbacks=[history])

print(history.losses)
# outputs
'''
[0.66047596406559383, 0.3547245744908703, ..., 0.25953155204159617, 0.25901699725311789]
```


How to interpret linear trends in residuals? I am trying to compare companies in the same industry and see how Profit and Number of employees correlated. My linear regression looks something like this:

Given the nature of the dataset, the model describes offices with low-profit better than the offices with a high profit. This is understandable since there are fewer samples for very profitable offices and a lot of samples for less profitable offices.
However, when I check for the residuals I get something like this:

The graphs show linear patterns for offices with lower profits. I am not certain how to interpret these trends.
Question:
What do linear trends in residuals indicate? How can the model be adjusted to handle them?

When I see it correctly in the top figure, there is some " bunching" in your data, meaning that there are a number of companies with the same (or very similar) number of employees. Since you appear to run a regression with only one independent variable (right hand side), this bunching will be visible in the residual.
Your model is:
$$ y_i = \beta_0 + \beta_1 x_i + u_i$$
Now say $\beta_0 = 1$ and $\beta_1 = 0.1$, and with "bunching" in $y$ you will get something like:
y     x      y_hat  u_hat
--------------------------
5     10     2      3
5     20     3      2
5     30     4      1
5     40     5      0

So the linear nature of the model will be reflected in the residual given that there is "bunching" in the data.
Note that using a log-log approach (which is perfectly okay) will change the natural interpretation of the estimated coefficients. In a log-log case, you will interprete the results as "a one percent change in $x$ will be associated with a $\beta_1$ percent change in $y$" (all other things equal).
Since the number of emplyees is bounded (no negative employees -> "count data"), things like Poisson regression could be worth a try.
Overall your model may be underspecified. If you can, include additional $x$-variables in your model, so to better reflect the data generating process. See page 114 in this book.


AUC ROC Threshold Setting in heavy imbalance I am doing binary logistic regression on a dataset with very heavy class imbalance. Class 1 is only 1% of data. When I train logistic regressor without class weights I get ROC AUC Score of 0.6269. Which is decent. However, when I see my confusion matrix I see that my model never predicted any 1's at all. So why is my AUC so high? I though AUC is meant to be a good measure for such a case.

Confusion matrix
 Predicted      0    All
True
0          32109  32109
1           1223   1223
All        33332  33332

I know Confusion matrix makes the probability threshold 0.5, so is score saying there is some threshold for which model will give higher recall? How can I get this threshold?

      Class  precision    recall  f1-score   support

       0       0.96      1.00      0.98     32109
       1       0.00      0.00      0.00      1223

I only care about precision and recall of class 1.

-->ROC curves should be used when there are roughly equal numbers of observations for each class.

-->Precision-Recall curves should be used when there is a moderate to large class imbalance.

enter code here
 from sklearn.metrics import precision_recall_curve
 precision, recall, thresholds = precision_recall_curve(y_test, probs)
 auc = auc(recall, precision)


Feature engineering on distributions Something that has always bothered me is summarizing distributions when feature engineering for a machine learning process. Does anybody have best practices for this?

Example: Imagine a dataset showing orders of products by customers. You want to summarize customer behaviour.

In that example, if we're to only focus on customers' order value, the order value for each customer would have some distribution D(x).

Now, can create variables describing features of each customer's distributions (min, mean, median, max, quartiles, IQR, etc.), but are there best practices around which features tend to provide the most information upon extraction? Moreover, is there some way to contain the information of a distribution in a single variable?

At the end each distribution can be described by a function with parameters. Can be a Gaussian, polynomial etc. In principle you can choose functions that only have one free parameter and fit this one. Depending on your data you might be able to guess a function class that seems to fit the underlying distributions. You can then use the fit parameters as input for your data set. Example: Fit a Gaussian and use mean, normalization and sigma = 3 parameters for your model.


Kohen Kappa Coefficient of Naive Bayes with 62% overall accuracy is better than Logistic Regression with 98% accuracy? I have been trying to evaluate my models used on fire systems dataset with a huge imbalance in the dataset. Most models failed to predict any true positives correctly however naive Bayes managed to do that but with a very high rate of False Positive. I had run the experiments on both the confusion matrix and classification report for both can be seen below. The same dataset and train/test split was used with both of the datasets

 Naive Bayes Confusion Matrix and Classification Report

     [[TN=732 FP=448]
     [FN=2   TP=15]]

          precision    recall  f1-score   support

       0       1.00      0.62      0.76      1180
       1       0.03      0.88      0.06        17

accuracy                               0.62      1197
macro avg          0.51      0.75      0.41      1197
weighted avg       0.98      0.62      0.75      1197

Logistic Regression Confusion Matrix and Classification Report

     [[TN=1180 FP=0]
     [FN=17   TP=0]]

          precision    recall  f1-score   support

       0       0.99      1.00      0.99      1180
       1       0.00      0.00      0.00        17

accuracy                              0.98      1197
macro avg          0.49      0.50     0.50      1197
weighted avg       0.97      0.99     0.98      1197

However I got the Kohen Kappa Coefficient for these models and I am quite confused on how to interpret the values. Please find values below

Logistic Regression=0.0
Naive Bayes=0.03

These values indicate very slight agreement. But why is the value of Naive Bayes slightly better than Logistic regression ?

Logistic Regression is only predicting one class (in this case the negative class)! Because of the high imbalance in the data, this model gives a high accuracy score. This metric, however, isn't reliable for imbalanced datasets. A more proper metric like Cohen's Kappa penalizes this behavior.

Naive Bayes, on the other hand, tries to predict both classes. It misses a lot more predictions this way, but it's Kappa is higher.


How does Phrases in Gensim work? I am using Gensim Phrases to detct n-grams in my text. Thus, I am interested in knowing the mechanism that Phrases uses to detect these n-grams in the text. Can someone please explain the mechanism used in Phrases in simple terms?

As the gensim tool cites the very famous paper by Mikolov - "Distributed Representations of Words and Phrases..." using which it is implemented. In the paper if you look at the section "4 Learning Phrases" they give a nice explanation of how n-grams are calculated (Equation 6).

So, if want to count bigrams this formula is straight-forward; score(wi, wj) is the score between any two words occuring together. But when counting trigrams, 'wi' will be a bigram and 'wj' will be a word. And same follows for any number of grams.


Applying bayesian methods to a simple neural network This is a really simple neural network with backprop. If one had to apply bayesian "inferences" to update the weights and biases, what would change in the code.

#Forward Propogation
hidden_layer_input1=np.dot(X,wh)
hidden_layer_input=hidden_layer_input1 + bh               # linear transformation
hiddenlayer_activations = sigmoid(hidden_layer_input)     # non-linear transformation
output_layer_input1=np.dot(hiddenlayer_activations,wout)
output_layer_input= output_layer_input1+ bout             # linear transformation
output = sigmoid(output_layer_input)

#Backpropagation
E = y-output
slope_output_layer = derivatives_sigmoid(output)
slope_hidden_layer = derivatives_sigmoid(hiddenlayer_activations)
derivative_output = E * slope_output_layer
Error_at_hidden_layer = derivative_output.dot(wout.T)
derivative_hiddenlayer = Error_at_hidden_layer * slope_hidden_layer
wout += hiddenlayer_activations.T.dot(d_output) *lr
bout += np.sum(derivative_output, axis=0,keepdims=True) *lr
wh += X.T.dot(d_hiddenlayer) *lr                          # update weight
bh += np.sum(d_hiddenlayer, axis=0,keepdims=True) *lr     # update bias

If my understanding is correct of using bayes method to derive the weights and biases, it would help in getting to those values faster, leaving behind unnecessary loops for weights which would not converge.

Apart from that, coding and observing it is the best bet to get a real perception.

Would like an understanding.

Bayesian models specify priors to inform and constrain the models and get uncertainty estimation in form of a posterior distribution. Non-Bayesian Deep Learning computes a scalar value for weights and biases at each layer. Bayesian Deep Learning calculates a posterior distribution of weights and biases at each layer which better estimates uncertainty but increases computational cost.

Edward is a Python package for Bayesian inference, including Deep Learning.


CSR scipy matrix does not update after updating its values I have the following code in python:

import numpy as np
from scipy.sparse import csr_matrix
M = csr_matrix(np.ones([2, 2],dtype=np.int32))
print(M)
print(M.data.shape)
for i in range(np.shape(M)[0]):
    for j in range(np.shape(M)[1]):
        if i==j:
            M[i,j] = 0
print(M)
print(M.data.shape)

The output of the first 2 prints is:

  (0, 0)    1
  (0, 1)    1
  (1, 0)    1
  (1, 1)    1
(4,)

The code is changing the value of the same index (i==j) and setting the value to zero.
After executing the loops then the output of the last 2 prints is:

  (0, 0)    0
  (0, 1)    1
  (1, 0)    1
  (1, 1)    0
(4,)

If I understand the concept of sparse matrices correctly, it should not be the case. It should not show me the zero values and the output of last 2 prints should be like this:

  (0, 1)    1
  (1, 0)    1
(2,)

Does anyone have explanation for this? Am I doing something wrong?

Sparse matrix is a matrix that has most of its elements as zero. Matrix on the other side of it, is Dense. csr_matrix creates an all zero element matrix and if you are updating the elements as

m[i,j] = 0.

Well, you're not really doing anything as elements are already zero. For example

import numpy as np
from scipy.sparse import csr_matrix
mat = csr_matrix((2,2), dtype = np.int16).toarray()
print mat,"\n"
for i in range(np.shape(mat)[0]):
    for j in range(np.shape(mat)[1]):
        mat[i,j] = i+j
print mat

gives

[[0 0]
 [0 0]]

[[0 1]
 [1 2]]

which is true, the first matrix,an all zero matrix created and the second, is the updated one.

Edit:

After substituting when $i==j$, the resultant matrix still has four elements but this time with $(0,0)$ and $(1,1)$ indices being $0$. So the result is

(0, 0)    0
(0, 1)    1
(1, 0)    1
(1, 1)    0
(4,)

But explaining your expected result

(0, 1)    1
(1, 0)    1
(2,)

The result has basically two elements with $(0,1)$ and $(1,0)$ indices being $1$ and $1$ and shape is just $(2,)$ when flattened which does not make sense as matrix is still of size $(4,)$ . So python output is basically giving all the values in the matrix with index. If I am rightly understanding what you are assuming, since the matrix is sparse, you are expecting it to give only the values where the elements are non-zero, well this does not do that.

Hope it helps.


Softmax function result for already normalized probabilities Isn't the aim of softmax function normalizing the probabilities such that they all sum to 1? So when we apply this method to the already normalized numbers, it would change them. what do these new outcomes depict? Since they are changed, are they reliable?

from scipy import special
scipy.special.softmax([0.4,0.6])

gives:

 array([0.450166, 0.549834])

Summing to 1 is just one property of the softmax function.

The softmax function takes the exponential of each value and divides it by the sum of the exponentials of all values. This tends to cluster values towards the mean, as you've seen in your example.

While the outputs of a softmax look and smell like probabilities, their values don't actually correspond to the likelihood of sampling from that distribution. They're better off thought of as the confidence of your prediction.


Is there any book for modern optimization in Python? I was reading Modern Optimization with R (Use R!) and wondering if a book like this exists in Python too? To be precise something that covers stochastic gradient descent and other advanced optimization techniques. Many thanks!

You should be able to translate code written in one language -- even pseudo-code -- to another, so I see no reason to avoid books for R. If you want one specifically for python, there's Machine Learning in Action by Peter Harrington.

One of scikit-learn's core committers is a releasing a book in October: Introduction to Machine Learning with Python: A Guide for Data Scientists.


Isolation Forest height limit absent in SkLearn implementation In the original publication of the Isolation Forest algorithm, the authors mention a height limit parameter to control the granularity of the algorithm. I did not find that explicit parameter on the Sklearn implementation of the algorithm, and I was wondering whether it is possible to control granularity in some other way?

Unfortunately, it seems like there is no hlim parameter incorporated into sklearn.ensemble.IsolationForest. Calculation of anomaly score is just based on the depth each point settles to and by the average path length. The only way to tune in a bit is by using contamination which calculates the threshold needed to set for anomaly score.

To achieve the granularity that was given in the original paper using hlim=6 to detect a cluster of small points, using a lot of estimators may solve the problem (still depends heavily on how you sample data from smaller cluster into a lot of estimators). But if that small cluster of data is very less in number, I don't think this idea works and there is nothing much we can do from the current implementation in sklearn. Hope this helps.


Do I need a datalake in my use case? My web application stores usage data, for example:

tickets opened an closed
tasks executed
user scores

etc. I need to show dashboards and reports for usage and performance trends, like:

How many tickets where opened/closed in a period?
what is the average tasks execution time?
who is the most active user?
which user gets the best score?

etc.
the set of questions is open, we can invent more.

Does this scenario fall under a datawarehouse classic approach?
Data are coming from a relational database with a specific data model.

Somebody is asking me to create a datalake, just giving him all my raw data.
I don't understand how does a datalake fit in this picture. With no model and relationship my data are usueless.

Also, I don't have a stream of events: I have tables in a relational database.
For example a row in a "task" table will have an "open time" column and a "close time" column.

I'm confused about the need or advantages of creating a datalake vs a traditional datawarehouse approach

The point of a "Data Lake" is to store unstructured data together, along with the related meta-data, all in one place. The data is usually stored in the format in which it was recorded i.e. the raw format. Data is then only extracted and processed for usage as required.

There is a nice article that compares data warehouses and data lakes in an unbiased and practical manner.

From your description of your data and the problems you want to solve, I don't personally think a data lake would be required.

Perhaps your colleague is planning to create a data lake for larger purposes, e.g. to house the data for many projects such as yours, which might also want to use your data? There are the issues of security, data ownership and data stewardship to think about in this case. Questions to answer would be:

who actually owns the data?
who is allowed to use what data?
who is allowed to view what data?
who is responsible for ensuring data is available at all times?


Predicting outcome of games - how to make team 1/2 interchangable Currently I'm writing some models to predict outcomes of team games, but I'm having a hard time featurizing the teams so that the order of teams doesn't matter.

For example, currently I duplicate all the data and reverse it to reduce the effect of ordering on the outcome, but that's not proving perfect. If I have "Ravens vs Panthers = Panthers win", I also put in "Panthers vs Ravens = Panthers wins".

When I then go to my model and ask for the output of Panthers vs Ravens and Ravens vs Panthers, I get different (but similar) results. For example the probabilities of Panther's winning may be 75% in the first ane 72% in the second.

How else might I make it 100% ambiguous as to which team is team 1 and which is team 2?

Home field advantage is incredibly important in sports. If you always put the home team in the first column, your model will adjust the bias term accordingly. I've been doing this successfully for NBA and MLB models for several years now. Making the first team the home team also eliminates the need for an explicit home indicator variable and you'll be able to use "half" of the data like you've described.

For rare neutral games like the Superbowl, you would need to create a mirror image of the data. In other words, present two rows for each game (team-A,team-B;team-B,team-A). When it's time to score you model, you can just arbitrarily pick one of the two records because they represent mirror opposite of the same thing. I've done this before for March Madness basketball games (all neutral courts), and this year (2016) my March Madness model was in the top 10% in Kaggle, so the technique works.

Home field advantage is just one of the variables you'll need to make a good predictive model, but that's an answer for another question.


Automatically Detect Valid/Interesting pivot/drill downs of a dataset Imagine you are given a tabular data set with a limited set of columns and rows and you are asked to find the valid/interesting pivot configuration by exploring the data. The brute-force option is to calculate all possible pivot configuration and somehow score the result features (such as its sparsity, count,...) and pick the ones that are scored higher. This is obviously very time-consuming.

I understand that my definition of "Valid/Interesting" is fuzzy in here but is there a more science-based approach (say using correlations, column cardinality,...) to automatically find the good pivot configuration of a given dataset? Any pointer is highly appreciated.

The automated generation of pivot tables is a well-studied topic. In fact, Google has a patent on the technology.

The general approach is:

Identify low-cardinality, categorical data features that have many repetitions. Those features become the row and column candidates for the pivot table.
Identify numerical data that are the candidates for the data (or fact) cells in the pivot table.
Select an aggregation function (e.g., count, sum, average, …).
Select an evaluation method. The most common evaluation methods are human rating or look-a-like modeling with historical data.

Generating useful pivot tables typically requires domain knowledge.


Can somebody explain me the meaning of this sentence? (Color Similarity - Selective Search Algorithm) This is a sentence from this article :

Color similarity: Computing a 25-bin histogram for each channel of an image, concatenating them together, and obtaining a final descriptor that is 25×3=75-d. Color similarity of any two regions is measured by the histogram intersection distance.

But I don't understand it. Can someone explain with an example?

If you have two images, you first start to make histograms of the values (0-255) in the three color channels (red, green, and blue). In the article 25 bins are used, meaning that the values are assigned to one of 25 ranges. The second step is to then concatenate the the three single channel histograms to one histogram for the full image. Since each image channel histogram has 25 bins, the concatenates histogram will have 75 bins (and thus is of length 75). When you do this for both images you want to compare you will now have two histograms. To get a similarity score you compute the intersection of the two histograms, i.e. the percentage of overlap. The following link from the article provides a more visual explanation on this.


Does standardization result in normal distribution? I have a question about standardization (subtract mean, divide by standard deviation) of data consisting of different features with different ranges.
I read some information that seemed to be contradictory.

Standardization

does not alter the fundamental shape of the data
but it gives the data the properties of a normal distribution: mean of 0 and standard deviation of 1

So, does it change the normal distribution or not?

No, standardization does not change the shape of the distribution. It centers the distribution by subtracting the mean and scales it by dividing by the standard deviation.


Open source alternative to google's Auto ML Auto ML receives raw(ish) data and generates a supervised learning model from it. That is, it automates feature engineering and model selection/tuning.

What open source alternatives currently exist that when given raw data and told which variable to target, and will produce an ML model autonomously?

AutoSklearn, AutoViML, TPOT, H2O are good choices. They can do feature engineering, but I would suggest to build your own feature based on domain knowledge.


What is the best way to encode features when clustering data? I have a dataset with numerical and categorical features. I am trying to run a k-means algorithm to find clusters of data.

What is the best way to encode categorical features?

I have been doing one hot encoding but I guess that it can be easily improved.

Try Hierarchical Clustering, Kmeans will not serve the purpose if you have categorical data.


Dealing with the test set of imbalanced data I am working on a problem dealing with unbalanced data that has a very specific request.

I would like to know the following:

When I have an imbalanced dataset and I do train test split, the test samples also have an imbalance. How can I at least balance the test set so that when I do the confusion matrix equal numbers are tested?

Here's an example (Classification):

If I have three groups: A 60,B 30,C 20. For simplicity, the test set is 10%. So, instead of having a train/test set of 54/6, 27/3, 18/2, I would like 58/2, 28/2 and 18/2 OR 54/2, 27/2, 18/2.

How can I do this?

For a case like this, it would probably be the easiest to do the train/test split manually. Here is a quick example with Pandas:

import pandas as pd
import numpy as np

# Create an empty dataframe with 60*A, 30*B, 20*C and some random target values
df = pd.DataFrame(columns=['category'])
df['category'] = (["A"] * 60 + ["B"] * 30 + ["C"] * 20)
df['target'] = np.random.rand(len(df))
df.category.value_counts()

> A    60
> B    30
> C    20
> Name: category, dtype: int64

# Obtain the size of the least represented category
smallest_category_size = np.min(df.category.value_counts().values)

# Size of the test split of the smallest category
test_size = 0.1

# Test count (2 in this case)
test_count = int(test_size*smallest_category_size)

# New train/test dataframes
train_df = pd.DataFrame(columns=df.columns)
test_df = pd.DataFrame(columns=df.columns)

for cat in df.category.unique():
    # Select only a part of the dataset containing one category
    cat_df = df[df.category == cat]

    # Create and shuffle boolean mask
    mask = np.array((len(cat_df) - test_count) * [True] + test_count * [False])
    np.random.shuffle(mask)

    # Append the selected values to the train/test
    train_df = train_df.append(cat_df[mask])
    test_df = test_df.append(cat_df[~mask])

train_df.category.value_counts()

> A    58
> B    28
> C    18
> Name: category, dtype: int64

test_df.category.value_counts()

> A    2
> B    2
> C    2
> Name: category, dtype: int64


decision trees on mix of categorical and real value parameters I have about 3,000,000 samples and each sample is described by a list of size about 20. Some elements in this list are categorical, for example name of cities, day of week, etc. (some categories have a large number of options, for example one category is url with more than 700,000 unique elements in my dataset!). Also some elements have real values, for example for time of day.

My data is labelled (2 categories,) and I need to train a classifier for test data. I am inclined towards decision tree or random forest since they seem to be a good choice for this type of problem.

Now my questions are:

1) How do I pre-process categorical data? one-hot-encoding seem to be the right choice but given that some of my categories have huge number of possible values, one hot encoder will produce very long words! am I correct?

2) How do I combine data from different categories? For example data from category 'cities' with data from 'urls', since they have different lengths. Do I simply concatenate them?

3) How can I combine categorical data with real valued data, for example 'name of cities' with 'time of day' to produce one matrix that can then be passed to decision tree classifier?

4) Are there any special normalisation, etc. that I have to do before passing data to classifier?

I plan to use python and Scikit for this task.

Many thanks for your help.

Unfortunately scikit-learn does not handle categorical features well - they must be encoded.  One-hot-encoding is great and can be implemented in sklearn very easily.  So if you have an array of feature columns, X, and a class label vector, y

from sklearn import OneHotEncoder
ohe = OneHotEncoder(categorical_features=[0])
X = ohe.fit_transform(X).toarray()

will perform the encoding for you (The categorical_features attribute gives the index of the feature you want encoded).  More to your point though, if your feature has many levels, one-hot encoding can leave you with a sparse and inefficient X array.  In this case it may be beneficial to do a count transform, possibly replacing each level with its corresponding log-odds ratio.

Count Transforms

The link above does a more thorough job explaining it, but, in layman's terms, for each categorical feature you:

count the number of times a level(category) belongs to class0 or
class1
add 0.5 to each count (this takes care of instances when a level belongs strictly to one class)
calculate the probability that it belongs to either class.
calculate the log of the odds ratio
replace each level with its corresponding log odds ratio

In this manner you get numerical values representing each of the feature's many levels without creating a dummy variable for each level.  Still, 20 variables is a lot, and you may want to use PCA for data compression and dimension reduction.

One of the great things about tree based methods in general is that they do not require standardization.  That being said, from a data science perspective, I would train several different models (tree based methods, svms, logistic regression, knn, etc.), and see which one yields the best results.  All methods besides CART benefit from scaling, so try

from sklearn.preprocessing import StandardScaler
stdsc = StandardScaler()
X_train_std = stdsc.fit_transform(X_train)
X_test_std = stdsc.transform(X_test)

I hope that answers most of what you're asking.  I'm not sure why you would want to concatenate two very different variables like cities and urls, but if you encode them or do a count transform your X array should be just fine and you can leave them separated.  Also, you shouldn't have to worry about the length of a given variable - python is object oriented (if you're getting error messages complaining about 'length', you've likely just confused your indices).


An error with respect to filter weights in CNN during the backpropagation Let's say a convolutional layer takes an input $X$ with dimensions of 5x100x100 and applies 10 filters $F$ 5x5x5, thus produces an output $O$ 10 feature maps 96x96.

During the backpropagation the layer receives $\frac{dE}{dO}$ of shape 10x96x96.

My question is how to compute $\frac{dE}{dF}$ ?

According to that article $\frac{dE}{dF}$ can be calculated as convolution between $X$ and $\frac{dE}{dO}$. Unfortunately, the article does not cover a case with multiple filters and multiple input channels.

Since $X$ has shape 5x100x100 and $\frac{dE}{dO}$ has shape 10X96x96 the depth of $X$ equals to 5 and the depth of $\frac{dE}{dO}$ equals to 10. So the depth dimension does not match. How to compute convolution in that case ?

One possible option is padding the lower dimensional object to match the size of the higher dimensional object. Padding is most commonly done so the input and output volume will always have the same spatial dimensions. The additional cells could be padded with any value. It is most common to use zero or the mean of the other cells, depending on the filter operation.


Should I expect major performance improvements by scaling my features? I'm trying to decide whether I should scale my features & responses for training, and I'm in a situation where I can't just try both scaling and not scaling.

My features currently have an std around 0.05, and the behavior of the timeseries I'm studying is very much scale dependent (0.5 means a VERY different thing than a 0.05 in terms of what's happening in the market).

Can I expect major improvements in performance, optimizer-wise, if I scaled my features to have std of 1?

My models are different combinations of MLP and 1d conv, and my algorithm is gradient descent with the Adam optimizer.

Thank you! [and yes; I'm another one of those people who are trying to forecast the stock market]

Louis is right that I should try both. But I'm leaving the question up just in case someone comes along and gives a mathematical proof of exactly normalization is useful /useless.

It's impossible to know whether the performance will improve without knowing what algorithm you are using. Even then the only way to tell is to try both.

That being said, I can't think of any scenario that standardisation would hurt the performance.


How to add 'other' as one group to clustering algorithm inference pipeline I have few clustering algorithms tuned having 5 cluster. I want 6th cluster if new data does not belong initial  5 cluster fall in 6th  cluster.
6th cluster [ say other category] consist of all data point which does not belong to 5 cluster.
P.S.:- initial whatever data is give is belong those 5 cluster. so say, kmean algorithms with number cluster as 5. during inference I wanted add 6th cluster so any which does belong given cluster can put this category depending on threshold distance. I have textual data. do let me which clustering algorithms i should go with dbscan, som etc..

Clustering doesn't work like this: for example k-means assigns an instance to the closest centroid, and since there is always a closest centroid there is a always a cluster that an instance "belongs to".
So you need a different approach if you plan to have the possibility of in instance "not in any group":

redo the clustering on the full set of instances
apply a first step which detects outliers
train a one-class classification model for every cluster