Can a Neural Network Measure the Random Error in a Linear Series? I have been trying to develop a neural network to measure the error in a linear series. What I would like the model to do is infer a linear regression line and then measure the mean absolute error around that line.

I have tried a number of neural network model configurations, including recurrent configurations, but the network learns a weak relationship and then overfits. I have also tried L1 and L2 regularization but neither work.

Any thoughts? Thanks!

Below is the code I am using to simulate the data and a fit sample model:

import numpy as np, matplotlib.pyplot as plt

from keras import layers
from keras.models import Sequential
from keras.optimizers import Adam
from keras.backend import clear_session

## Simulate the data:

np.random.seed(20190318)

X = np.array(()).reshape(0, 50)

Y = np.array(()).reshape(0, 1)

for _ in range(500):

    i = np.random.randint(100, 110) # Intercept.

    s = np.random.randint(1, 10) # Slope.

    e = np.random.normal(0, 25, 50) # Error.

    X_i = np.round(i + (s * np.arange(0, 50)) + e, 2).reshape(1, 50)

    Y_i = np.sum(np.abs(e)).reshape(1, 1)

    X = np.concatenate((X, X_i), axis = 0)

    Y = np.concatenate((Y, Y_i), axis = 0)

## Training and validation data:

split = 400

X_train = X[:split, :-1]
Y_train = Y[:split, -1:]

X_valid = X[split:, :-1]
Y_valid = Y[split:, -1:]

print(X_train.shape)
print(Y_train.shape)
print()
print(X_valid.shape)
print(Y_valid.shape)

## Graph of one of the series:

plt.plot(X_train[0])

## Sample model (takes about a minute to run):

clear_session()

model_fnn = Sequential()
model_fnn.add(layers.Dense(512, activation = 'relu', input_shape = (X_train.shape[1],)))
model_fnn.add(layers.Dense(512, activation = 'relu'))
model_fnn.add(layers.Dense(  1, activation = None))

# Compile model.

model_fnn.compile(optimizer = Adam(lr = 1e-4), loss = 'mse')

# Fit model.

history_fnn = model_fnn.fit(X_train, Y_train, batch_size = 32, epochs = 100, verbose = False,
    validation_data = (X_valid, Y_valid))

## Sample model learning curves:

loss_fnn = history_fnn.history['loss']
val_loss_fnn = history_fnn.history['val_loss']
epochs_fnn = range(1, len(loss_fnn) + 1)

plt.plot(epochs_fnn, loss_fnn, 'black', label = 'Training Loss')
plt.plot(epochs_fnn, val_loss_fnn, 'red', label = 'Validation Loss')
plt.title('FNN: Training and Validation Loss')
plt.legend()
plt.show()

UPDATE:

## Predict.

Y_train_fnn = model_fnn.predict(X_train)
Y_valid_fnn = model_fnn.predict(X_valid)

## Evaluate predictions with training data.

plt.scatter(Y_train, Y_train_fnn)
plt.xlabel("Actual")
plt.ylabel("Predicted")

## Evaluate predictions with training data.

plt.scatter(Y_valid, Y_valid_fnn)
plt.xlabel("Actual")
plt.ylabel("Predicted")

This problem is naturally hard. The underlying function that we try to learn is
$$\mathbf{X}=i+s+\mathbf{e} \rightarrow Y=\left \| \mathbf{X} - i - s \right \|_1 = \left \| \mathbf{e} \right \|_1=\sum_d|e_d|$$

where $i$ and $s$ are unknown random variables. For large $i$ and $s$, $\left \| \mathbf{e} \right \|_1$ is naturally hard to recover from $\mathbf{X}$. I found a working example (training error almost zero) by setting the intercept $i$ and slope $s$ to zero!, drastically shrinking the network size to work better with a small sample size (800), and increased the number of epochs to 800, which was crucial. Also, (true value, error) is plotted at the end for training data.

You can work up from this point to see the effect of increasing $i$ and $s$ on performance.

import numpy as np, matplotlib.pyplot as plt

from keras import layers
from keras.models import Sequential
from keras.optimizers import Adam
from keras.backend import clear_session

## Simulate the data:

np.random.seed(20190318)

dimension = 50

X = np.array(()).reshape(0, dimension)

Y = np.array(()).reshape(0, 1)

for _ in range(1000):

    i = 0   # np.random.randint(100, 110) # Intercept.

    s = 0   # np.random.randint(1, 10) # Slope.

    e = np.random.normal(0, 25, dimension) # Error.

    X_i = np.round(i + (s * np.arange(0, dimension)) + e, 2).reshape(1, dimension)

    Y_i = np.sum(np.abs(e)).reshape(1, 1)

    X = np.concatenate((X, X_i), axis = 0)

    Y = np.concatenate((Y, Y_i), axis = 0)

## Training and validation data:

split = 800

X_train = X[:split, :-1]
Y_train = Y[:split, -1:]

X_valid = X[split:, :-1]
Y_valid = Y[split:, -1:]

print(X_train.shape)
print(Y_train.shape)
print()
print(X_valid.shape)
print(Y_valid.shape)

## Graph of one of the series:

plt.plot(X_train[0])

## Sample model (takes about a minute to run):

clear_session()

model_fnn = Sequential()
model_fnn.add(layers.Dense(dimension, activation = 'relu', input_shape = (X_train.shape[1],)))
model_fnn.add(layers.Dense(dimension, activation = 'relu'))
model_fnn.add(layers.Dense(  1, activation = 'linear'))

# Compile model.

model_fnn.compile(optimizer = Adam(lr = 1e-4), loss = 'mse')

# Fit model.

history_fnn = model_fnn.fit(X_train, Y_train, batch_size = 32, epochs = 800, verbose = True,
    validation_data = (X_valid, Y_valid))

# Sample model learning curves:

loss_fnn = history_fnn.history['loss']
val_loss_fnn = history_fnn.history['val_loss']
epochs_fnn = range(1, len(loss_fnn) + 1)

plt.figure(1)
offset = 5
plt.plot(epochs_fnn[offset:], loss_fnn[offset:], 'black', label = 'Training Loss')
plt.plot(epochs_fnn[offset:], val_loss_fnn[offset:], 'red', label = 'Validation Loss')
plt.title('FNN: Training and Validation Loss')
plt.legend()

## Predict.

plt.figure(2)

Y_train_fnn = model_fnn.predict(X_train)

## Evaluate predictions with training data.
sorted_index = Y_train.argsort(axis=0)
Y_train_sorted = np.reshape(Y_train[sorted_index], (-1, 1))
Y_train_fnn_sorted = np.reshape(Y_train_fnn[sorted_index], (-1, 1))
plt.plot(Y_train_sorted, Y_train_sorted - Y_train_fnn_sorted)
plt.xlabel("Y(true) train")
plt.ylabel("Y(true) - Y(predicted) train")
plt.show()


How to check for "statistical significance" of categorical feature in black box models Let's say we have a categorical feature $X_i$ and we have build a black-box classification model like xgboost with $X_i$ as one of many predictors. We'd like to ask a question: does $X_i$ affects the overall prediction and, if so, how much?

In particular $X_i$ could be:

a dichotomous variable
a n-level variable where we are interested in the potential difference between two particular levels

In white-box models like linear regression we have tests to obtain statistical significance. But can we obtain statistical-significance-alike with black box models? Does any tool from explainable artifficial intelligence field is applicable to that? Or would it be better to just perform standard t-test on the output probabilities predictions?

First, you have to encode the feature. Models only take numerical features.

Then assessing the solution:

You can either see the feature importance of the model
Or use an XAI tool that will help you to understand the predictions. I normally use SHAP (SHapley Additive exPlanations):is a game theoretic approach to explain the output of any machine learning model. It connects optimal credit allocation with local explanations using the classic Shapley values from game theory and their related extensions.

https://github.com/slundberg/shap


Detecting original vs. edited (reposted / recompressed) image I'm trying to create something to help solve this problem:

My goal is that, given two images where one is an edit of the other, produce a system that outputs which one is most likely to be the original.
I have attempted to solve the problem in the following way.

Collect a data set of random images that one may find in their social media blog. (10,000 images)

Apply random lossy transformations to the image: resizing (with various scaling algorithms), cropping, lossy compression (JPEG)

Preprocess input into training data:

Metadata:

Collect file size, resolution, file format, lossy compression quality

Add jitter (to prevent overfitting by recognizing specific images), jitter varies per 8x8 sample

Output is a fixed number of floats per datum

Input pixels:

Break the image into 8x8 chunks (the size is chosen to coincide with JPEG MCU size)

Select a fixed number of chunks with the most entropy (changes in pixels)

Select a color channel

Divide by 255.0

Output is 64 floats (one per pixel)

DCT values:

Using the pixel data from above, perform a DCT transform

Take the absolute value of the DCT value

Output is 64 floats

Feed the above data into a neural network.
I don't know if it makes sense or not, but I started with a straight-forward model and did a simple hill-climbing hypersearch which added/removed/edited layers, and it produced the following model:

Each sample (and associated metadata) is fed as a single input.

Fit the model to the labels 0.0 or 1.0, with 0.0 representing original (less noise) and 1.0 representing edited (more noise).
Even though these labels don't describe any quality of the image itself, the goal is to use them to indicate some kind of ordering. The idea here is to "pull" the weights towards 0 for images known to have less noise and towards 1 for those with more noise, which I hope will cause the model to output higher values for images which have been edited.

Currently the model's accuracy is 88% (i.e. it correctly produces higher values for our edited versions for 88% of input image pairs).
My question is if there is a better way to represent ordering in this case other than labeling pairs 0 and 1. One potential issue with this is that the model is penalized if it outputs a score >1 or <0 (though currently avoided using sigmoid activation if that makes sense). Also, if there are other obvious improvements in the whole process. Thank you!

My question is if there is a better way to represent ordering in this case other than labeling pairs 0 and 1.

I solved this by creating an artificial comparator layer, which takes two inputs, and outputs 0 if the first is smaller or 1 if the first is greater. This allows backpropagation to correctly push the two away from each other within the same batch.

Also, if there are other obvious improvements in the whole process.

See my other question: Producing a confidence output to use in a weighted average layer


Set value for column based on two other columns in pandas dataframe I have a dataframe that has contracts with different order dates and I need to create a new column that assign a number to each contract if it has more than one order date. For example my sample dataframe looks something like this:
df = pd.DataFrame({'contract': ['123A','123A','123A','123A','123B','123B','123C'],'prod': ['X1','M1','V1','D1','A1','B1','C1'],'date':['2019-04-17','2019-07-02','2019-04-17','2019-07-02','2019-04-17','2019-09-01','2019-08-02'],'revenue': [5688,113932,5688,49157,5002,892,9000]})

I need my final table to have another column with a unique contract id for each date. My final table from above should look something like this:

contract
date
header_contract

123A
2019-04-17
123A_0

123A
2019-07-02
123A_1

123A
2019-04-17
123A_0

123A
2019-08-02
123A_2

I have the following code that does what I need on a smaller dataset:
contracts_num = df['contract'].unique()
for cm in contracts_num:
    for idx,val in enumerate(df[df['contract'] == cm]['contract'].dt.date.unique()):
        df.loc[((df['contract'] == cm) & (df['contract'] == str(val))),'contract'] = df['contract'] + '_' + str(idx)

I'm trying to do it on a much larger dataset (around 50,000 contracts) and it's taking a really long time. Is there anyway to make it more efficient?

You can use groupby together with shift and cumsum as follows:
df['header_contract'] = df['contract'] + '_' + df.sort_values(['contract', 'date']).\
  groupby('contract')["date"].\
  apply(lambda x: (x.shift() != x).cumsum()).astype(str)

In the apply, x.shift() != x is used to create a new series of booleans corresponding to if the date has changed in the next row or not. cumsum will then create a cumulative sum (treating all True as 1) which creates the suffixes for each group. This is then merged with the contract names to create the new column.
Result:
  contract prod       date  revenue header_contract
0     123A   X1 2019-04-17     5688          123A_1
1     123A   M1 2019-07-02   113932          123A_2
2     123A   V1 2019-04-17     5688          123A_1
3     123A   D1 2019-07-02    49157          123A_2
4     123B   A1 2019-04-17     5002          123B_1
5     123B   B1 2019-09-01      892          123B_2
6     123C   C1 2019-08-02     9000          123C_1


Is there any possibility to apply deep dreaming in data augmentation? I looked into the deep dreaming concepts and feel like this has the potential for data generation. But i'm not sure how possible this concept is. Any thoughts regarding this?

TL;DR Not usually, because Deep Dream ignores the distribution of your input population.
You cannot extract additional informative data using generators, only remix and recombine what you already have. There may be a few situations where that is useful because you can use a generator trained on other data to bring in extra relevant information that was not in your main dataset.

Deep Dream logically alters an input structure to maximise some measure within a neural network based on neuron activations. This might be the activation of a single neuron, or of a layer of neurons.
I have worked with Deep Dream scripts based on Google's Inception v5 network which was trained on ImageNet, just for fun.
Here's a relatively extreme example of output, with a few specific neuron activatioons maximised and a long zoom so that the data is essentially only a super-stimulous for those artifical neurons:

One thing you can say with reasonable certainty is that this image does not look like anything you might draw at random from ImageNet, from any category. It would be a poor choice as augmentation for ImageNet classification tasks, because it is not from the target population.
The same is true in general for all obvious Deep Dream modifications. The modifications do not look natural and produce inputs that would not reasonably be in the input distribution. Once you are clearly outside the input distribution for your use case, then learning how to process those inputs may not give you any benefit related to the goals of your machine learning task.
More subtle changes might be useful, if on inspection it is clear that they should not change the categorisation. This might be similar augmentation to adding noise, and if the alterations were chosen carefully, might help defend against adversarial inputs. In fact generating adversarial inputs is an almost identical process to Deep Dream, usually focusing on the output categories and updates that look like high frequency noise so that humans tend not to see the difference.
There are better data generating models that attempt to create data that could have come from the input population. Two major ones are variational autoencoders (VAE) and generative adversarial networks (GAN). These might be useful for data generation.
However, if you are considering somehow "bootstrapping" a small dataset by training one of these generators on it then using its output to augment your data, it is unlikely that this will work. The quality of the output will usually depend on its training data such that generated data will not truly add information that you did not already have in the original training set. In most cases you will better off looking at simpler logical transformations of data that you are confident will not change the target label (or that change it in predictable ways) - for image-based CNNs that might include changing rotation, crop, brightness/contrast and other basic image manipulations.
There might be a few cases in which working with a generator helps. For instance if you can train it on far more data because you have lots of unlabelled data, or dat afrom a different but similar task. Then the quality of generated inputs could be high enough that you effectively get to make use of unsupervised learning from the same dataset. You might still find it easier to do transfer learning, or train an autoencoder and re-use its first few layers though.


Can one property name be used twice in the same branch of a DecisionTreeRegressor? I am using this dataset for the analysis (Generated using make_regression of sklearn library)

I was trying to learn the DecisionTreeRegression algorithm of sklearn library. I used the following code to fit the regressor.

from sklearn.tree import DecisionTreeRegressor as DTR

regressor1 = DTR(max_depth=2)
regressor1.fit(X,y)

y_pred1 = regressor1.predict(X)

These are the leaf node values that I got,

It seems like, the decision tree first did a split on prop 2 at -1.0923644932716892 for the root node then on the right child of the root it again did another split on prop 2 at 0.0340153523120978.

But what I learnt about Decision Trees is that is a split is done at a property then in the same branch the property should not be used again. Then why the sklearn library is doing this thing?

Good job looking at the tree and understanding what has happened.

There is no problem splitting on the same feature multiple times. A continuous feature has many split points available. The tree continues to subset and refine. The split criteria shows what will be the "best" greedy split at this point. If a feature is income, perhaps the best split is \$100,000. Then on the high side, there is another split for \$10,000,000 since those people behave differently from the \$100,100 income people.

Even a categorical variable may split again. For example black and blonde hair go left, all other hair color go right. Later splitting black and blonde is the best split available.

I saw a research project where scikit learn code was adjusted to give split priority to features that were already used to reduce the number of features in the model which may happen in trees on spurious interactions due to greediness. It worked well.


How to solve this regression problem? I have a multivariable regression problem which basically forms a plane in 3D space. There may be other independent variables at play, but if I can approach the problem with 2 independent variables then I should be able to scale to include the rest.

Broadly, it's a resource problem. I have a required resource (z), which depends on time (x) and number of registered users (y). Registered users is a function of time, but I need the time component because the resource requirement is different at different times of the day. I want to be able to predict the resource requirement at a given time of day if our users continue to registered at a known rate.

Here is a chart of the data:

This is obviously not a linear regression problem, so how do I approach this?

If you want to predict, you need an underlying model of the behavior. Then you can perform least-squares fitting to determine the model parameters. Linear regression just assumes that you have a linear relationship and predicts from that relationship - there's nothing magic about it, except that the model is simple and often effective. Polynomial regression gives bad behavior when extrapolating, so I wouldn't advise that.

Building a model function is usually a trial-and-error process (unless you have some prior knowledge about your system, which I guess you don't). I'd suggest looking at the locus of (x, y) points where z is maximized. It looks like they have an exponential decay relationship. A model like y = exp(-x/x0) might fit the data rather closely. Once you've determined the value of x0 from the data, you can rebuild a model for z.


Product Prediction to group of customers I have multiple groups of customer, say for segment 1 as shown in the pictures, I have a list of products that I can choose the cross-sell to that group.
Consider the product has multiple features, e.g., prices, types, how do I come up with a model that recommend top 3 products for the segment of customers?

The difficuties lie in:
For segment of customers, we have range of values (e.g., for ALL_FEES we have \$13-\$4835), how to model that?

This is an interesting problem statement. If we have products purchased by all the users, We can try the following

Perform Collaborative filtering for each metric e.g. ALL_FEES, ASSETS
While performing Collaborative Filtering, we use userGroups instead of users
For arriving at the top product recommendations for each user group, we choose products that have been recommended in most of the collaborative filtering metric model outputs


Why does CV yield lower score? My training accuracy was better than my test accuracy, hence I thought my model was over-fitted and tried Cross-validation. The model further degraded. Is that my input data need to be sanitised further and of better quality?

Please share your thoughts what could be getting wrong here.

My data distribution:

Code snippets...

My function get_score:

def get_score(model, X_train, X_test, y_train, y_test):
    model.fit(X_train, y_train.values.ravel())
    pred0 = model.predict(X_test)
    return accuracy_score(y_test, pred0)

Logic:

print('*TRAIN* Accuracy Score => '+str(accuracy_score(y_train, m.predict(X_train)))) # LinearSVC() used
print('*TEST* Accuracy Score => '+str(accuracy_score(y_test, pred))) # LinearSVC() used

print("... Cross Validation begins...")

y0 = pd.DataFrame(y)
y0.reset_index(drop=True, inplace=True)

print(X.shape)
print(y0.shape)

kf = KFold(n_splits = 10)

e = []
for train_index, test_index in kf.split(X):
    X_train, X_test, y_train, y_test = X.iloc[train_index], X.iloc[test_index],y0.iloc[train_index], y0.iloc[test_index]
    print(train_index, test_index)
    e.append(get_score(LinearSVC(random_state=777),X_train, X_test, y_train, y_test))

print("Finally :: "+str(np.mean(e)))

Output:

*TRAIN* Accuracy Score => 0.9451327433628318
*TEST* Accuracy Score => 0.6597345132743363
... Cross Validation begins...
(9040, 6458)
(9040, 1)
[ 904  905  906 ... 9037 9038 9039] [  0   1   2 ... 901 902 903]
[   0    1    2 ... 9037 9038 9039] [ 904  905  906 ... 1805 1806 1807]
[   0    1    2 ... 9037 9038 9039] [1808 1809 1810 ... 2709 2710 2711]
[   0    1    2 ... 9037 9038 9039] [2712 2713 2714 ... 3613 3614 3615]
[   0    1    2 ... 9037 9038 9039] [3616 3617 3618 ... 4517 4518 4519]
[   0    1    2 ... 9037 9038 9039] [4520 4521 4522 ... 5421 5422 5423]
[   0    1    2 ... 9037 9038 9039] [5424 5425 5426 ... 6325 6326 6327]
[   0    1    2 ... 9037 9038 9039] [6328 6329 6330 ... 7229 7230 7231]
[   0    1    2 ... 9037 9038 9039] [7232 7233 7234 ... 8133 8134 8135]
[   0    1    2 ... 8133 8134 8135] [8136 8137 8138 ... 9037 9038 9039]
Finally :: 0.32499999999999996
>>>

Edit -1- Adding values of "e"

[0.08075221238938053, 0.413716814159292, 0.05752212389380531, 0.15376106194690264, 0.14712389380530974, 0.4668141592920354, 0.6946902654867256, 0.7112831858407079, 0.33738938053097345, 0.18694690265486727]

Edit -2- Adding shuffle=True parameter to KFold()

Result:

[   0    1    2 ... 9037 9038 9039] [   4    5   10 ... 9007 9011 9024]
[   0    1    2 ... 9037 9038 9039] [  21   43   44 ... 9018 9035 9036]
[   0    2    3 ... 9037 9038 9039] [   1   20   60 ... 9023 9031 9034]
[   0    1    2 ... 9036 9037 9038] [   6   25   27 ... 9010 9025 9039]
[   0    1    2 ... 9037 9038 9039] [  15   16   28 ... 9029 9030 9033]
[   0    1    2 ... 9037 9038 9039] [   3   12   40 ... 9015 9017 9028]
[   0    1    2 ... 9037 9038 9039] [   7    8   23 ... 9013 9014 9027]
[   0    1    3 ... 9035 9036 9039] [   2   18   19 ... 9019 9037 9038]
[   0    1    2 ... 9037 9038 9039] [  24   37   39 ... 9012 9016 9026]
[   1    2    3 ... 9037 9038 9039] [   0    9   14 ... 9020 9021 9032]
[0.6504424778761062, 0.6736725663716814, 0.6969026548672567, 0.6692477876106194, 0.6769911504424779, 0.6382743362831859, 0.6692477876106194, 0.6648230088495575, 0.6648230088495575, 0.6814159292035398]
Finally :: 0.6685840707964601

Since your model performances worse on held-out data (validation or test), you have a high variance model.

The most common methods to lower variance are:

Add more data to the training dataset.
Increase regularization. Since you are using LinearSVC, that means increasing C.
Switching to a bagged or boosted algorithms.


Using LSTMs for continous learning and predicting I'm trying to develop a model to predict a commodity price movement direction based on previous observations. The model should learn common technical analysis patterns, e.g. head and shoulders. So, I think I should use a stateful LSTM so that it maintain a long term state to keep track of technical analysis patterns.

On the other side, as the data set is updated daily, i.e. new observations are added, I need the model to keep learning and making predictions every day. So, in order to update the model parameters on each new observation, I think I should use batch_size=1.

If I use the last N observations to predict the next M steps, the input tensor to the model would have the shape of (1, N, num_features):

$$
    X_1, X_2, ..., X_N \rightarrow Y_1, Y_2, ..., Y_M
$$

Questions:

Why should we reset_states after training on train set and prior to predication on test set? I think we shouldn't as a TA pattern may be half in the train set and half in the test set. If we reset the states, the model can't recognize this pattern.
Can I reset_states and feed in data of a different commodity? as I only need the model to learn TA patterns not commodity specific characteristics. If so, how can I restore the states to make prediction on the first commodity?

To answer your questions in order:

The whole point of LSTM (or any time series model) is to predict previously unseen values. If states are not reset - then there is a risk of data leakage - whereby forecasts from the training set will lead into the test set. This would mean that your model would perform quite well on the existing set of data - but would perform badly on new/unseen data. This defeats the purpose of implementing a time series model.
Depending on how similar the pattern is, it might be an option. However, it is just my personal experience that LSTMs are not particularly effective at forecasting commodity prices - the main reason being that these models tend to carry forward too much volatility in the forecasts, which tends to interfere with the overall trend of the data. Here is an example of the use of LSTMs in forecasting oil prices, which you might find of interest: Are LSTMs Always Effective At Time Series Analysis? A Case Study of Oil Prices


Is text recognition by definition a part of image recognition? I'm referring to more advanced text recognition systems that are using neural networks to find and extract text from images like the ones Google and Microsoft are offering on their ML platforms.

If not, what is the technical difference to other branches of image recognition like object or face recognition?

Yes. Please take a look on how to do text recognition in images using opencv and Tesseract opensource libraries.


Should I perform customer segmentation before performing churn prediction? Imagine a company with multiple lines of revenues coming from diferent products, but all customer can access these different products through the same account and the same online platform.
My goal is to predict the churn for each customer.
Should I perform customer segmentation into clusters and build a churn prediction model for each segment?
The straight foward path would be to get all relevant features for all customers and try to predict the churn for all of them. The different bahaviors and personae will theoretically reflect on the features.
What I do expect to get from the clustering are the personae this company intends to serve.
In this case, a churn will be defined as a custumer who stops consuming any product or close its account.
My objective here is to build a churn prediction model according to each pearsonae behavior pattern and I expect from this approach to have better features for each because now we understand them more and then more acurate models.
This would also lead to more personalized business strategies to avoind the churn.
But I don’t know if this is a good practice and if I should try the straight foward approach or a different one.

There is no direct reason to segment your customers before estimating their churn probability, unless you know of business reasons for the segments to behave in a substantially different way with regard to churn, or the business users of the model for the different segments will have differing requirements (e.g. on model interpretability). Otherwise, there is nothing that a well-trained churn model could not do that two models (clustering and churn classification) would.
You can actually see for yourself if the approach brings any benefit - add the segmenting features to the churn model input and check whether they have important influence on its outcomes (they are high in the decision tree, have high feature importance, etc.) If they do, segmenting the customers and training separate churn models might bring benefits, especially if you have strict requirements on model interpretability.


Why is there no end user Application, yet? machine learning is being hyped since the deep neural networks.

It seems to me, that you have to program in order to do machine learning.
But is the process of training data and labeling data is the same of every problem. Why isn't there an Excel like application that enables thousands of non experts to do machine learning ?

Disclaimer : I am not a data scientist .

Listing 2 examples:
IBM Watson Analytics

Amazon ML use case

Preparing the data for supervised learning is require skills. Not all data came labeled and in form to be used as in need to solving problems.

Also many more platforms/Api are in market now but for sure you can't solve a problem only with 1 algorithm, is needed much more ... Hope it help.


Why Matlab neural network classification returns decimal values on prediction dataset? I have an input dataset (matrix 25x2275) which is normalized to values between 0 and 1. I also have a binary formatted output matrix (3x2275) like 0 0 0 0 0 0 0 0 1, 1 0 0 1 1 1 0 0 1 ...

I imported both files in matlab nntool and it automatically created a network with 25 input and 3 output nodes as I wanted.

After I trained this network using feed-forward backProp, I tested the model in its training data and each output nodes returns a decimal value like (0.9999 0.978 1 0 0.99 0.59368 0.38359 0.31435 1.0604).

Why it doesn't return discrete values like 1 0 0 1 1 1 0 0 1? Is there any thing that I must set in nntool to get such values?

Even for classification problem, the output activation can be any slashing function like sigmoid, tanh, using which the error is back-propagated.

I guess, its trying to give you continuous value for a labels more like probabilities, you use a cut off (say 0.5) and convert the values to 0 and 1 labels


Differentiable loss function for ranking problem in regression model In regression problem, we may need a loss function to measure the relative ranking accuracy between targets $y$ and predicted values $y_{pred}$. Abviously, the simple MSE does not consider such ranking relations.

A straight choice is the so-called IC (Information Coefficient)

$$IC\propto corr(\text{rank}(y), \text{rank}(y_{pred}))$$

which use the correlation between two ranks. However, rank function is not differentiable, thus it can't be used in loss function for regression which uses gradient propagation to update the parameters.

Another choice might be a modified pairwise ranking loss designed for regression

$$\text{Loss}=\sum_{i, j} \text{max}(0, -(y_{pred}^{(i)}-y_{pred}^{(j)})\cdot(y^{(i)}-y^{(j)}))$$

which is proposed in this paper [https://www.cc.gatech.edu/~zha/papers/fp086-zheng.pdf]. This function is differentiable but also hard to train.

My questions are: (1) Is there any other differentiable loss function that can be applied in ranking problem with regression model;
(2) Can rank function be approximated by a differentiable function.

One option is to use standard framing as learning to rank the relative positions, thus pairwise loss function.


Working ofLSTM with multiple Units - NER I am trying to understand working of LSTM networks and kind of not clear about how different neurons in a cell interact each other. I had a look at a similar question, but still not clear about few things.

Assume an NER task where each time step get embedding on one word as input and there are 10 time steps. I defined the LSTM cell with 100 units, As per my understanding each units will have 3 separate gates and memory cell as shown in the second figure (total 100 memory cells and 150 gates). From the theory of LSTM, each of these memory states stores different information, like one might store about singularity, other might store information about gender and so on (please correct me if I am wrong). With this set up, I have few doubts about the working of LSTM and the interaction between each units/neurons

1) In the figure, each cell has multiple neuron, lets say 100. Does the first input X1 (first word vector) will be input to the first neuron of the cell along with previous cell's activation function (say h0) and its output will go to the second neuron and so on..? Or same input X1 and h0 will be fed to each 100 neurons parallel (independent) and respective outputs will go to next time step (not sure whether any aggregation happens) ?

2) How different units would be able to store different features (singularity, gender etc..)? is it because they are initialized with random and different weights?

3) If neurons are working parallel, Is there any interaction happens between them to share information? will it be possible that multiple cell state store about same information (say, gender)

4) If LSTM cell 1 has 100 neurons/units and input X1, output h1 will also have 100 values (one for each neuron) ?

Welcome to StackExchange DataScience!

1) The same input $x_1$ and $h_0$ will be fed to each 100 neurons parallel (independent) and respective outputs will go to next time step. Notice that there are two 100-dimensional vector going to the next time step: $c_1$ and $h_1$, which are denoted by the arrows on the right side in your second figure.

2) This is a complicated question. Firstly, there are usually no one-to-one mapping between a neural and its meaning like neural 1 -> gender.

3) Those neurons do share information by the matrix multiplication operations. For example, the cell value at time step $t$ is usually calculated as
$$c_t = f_t c_{t-1} + i_t \tanh(W_{xc}x_t+W_{hc}h_{t-1}+b_c)$$
Notice that the hidden value in the previous time step $h_{t-1}$ is multiplied by a matrix $W_{hc}$, therefore each value in $W_{hc}h_{t-1}$ is a linear combination of all the values in $h_{t-1}$.

4) The number of neurons in the LSTM cell, the dimension of hidden vector $h_t$, the dimension of the cell vector $c_t$, and the dimension of the output vector $o_t$, are all the same (in your case, 100). This is known as the "hidden size", "output size" or "number of units/neurons". This is not the same as the dimension of input vector $x_t$, which is usually known as the "input size".

If you have further questions about the dimension of the LSTM variables, I suggest you to look at the first answer here where the author listed the dimension of all variables and parameters.


How to use hours of the day as a continuous feature? I would like to use the hour of the day (0-23) as a continuous feature, so the model will know that 12pm comes before 13:00, and that 8:00 is further from 21:00 than from 10:00. How do I engineer this feature so it will also understand that 0:00 comes after 23:00?

Great question.

I'd say you should either use fourier transforms or sine / cosine transforms as they should get 0 to be right next to 23. The below post is talking about transforming day features, but should be easily applicable here.

Transforming day features

Transforming hour features


Categorical variables with multiple entries transformed to entity embedding I have structured data with lots (tens of thousads) of categories organized into columns. The goal is to enter the data into gradient boosting machine algorithm for a specific prediction.

Some columns have more than one entry for the same sample, i.e., sample1 for column1 has entry1 for line1 and entry2 for line2. Here's an example:

df = pd.DataFrame({'pat': [1, 2, 3, 3, 3, 3], 'diag_type': ['D', 'OP', 'D', 'D', 'D', 'OP'],
                  'diag': ['D_1', 'OP_1', 'D_1', 'D_3', 'D_4', 'OP_2']})

    pat     diag_type   diag
0   1   D   D_1
1   2   OP  OP_1
2   3   D   D_1
3   3   D   D_3
4   3   D   D_4
5   3   OP  OP_2

Since I have so many categories I would need to do some restructuring, and I decided that entity embedding is the best way to do it. However, I haven't figure out a good way to restructure the dataframe. I thought about doing some sort of dummy coding as exemplified in this stackoverflow post, e.g.:

d.get_dummies(df.set_index('pat')).sum(level=0)
    diag_type_D     diag_type_OP    diag_D_1    diag_D_3    diag_D_4    diag_OP_1   diag_OP_2
pat
1   1   0   1   0   0   0   0
2   0   1   0   0   0   1   0
3   3   1   1   1   1   0   1

But then I still get a lot of sparsity and it takes forever. Is there a better way of doing it?

Why don't you try with a different kind of encoding?

One of the most typical encodings, when you have high cardinality, is Target Encoding. You can find an implementation in this library. You can find an explanation of what is it and when to apply it here.


Run python script using spark-submit on windows 7 I'm trying to submit a python script using spark-submit on windows 7. I'm using spark 2.3.0 built for hadoop 2.7 . The python script is supposed to read data from a csv file and plot graphs. is it possible to run such a script using spark-submit? In the python script I included this block of code for spark context.

import findspark
findspark.init()
import pyspark
from pyspark import SparkContext
APP_NAME = 'kineticPCA'
conf = SparkConf().setAppName(APP_NAME)
conf = conf.setMaster('spark://192.168.xx.xxx:7077')
sc = SparkContext(conf=conf)

This is the how submit the job on command prompt, on the spark bin directory

spark-submit G:/guru orders/pca kinetics/files/kineticPCA.py

This is the error I get:

  Error: Cannot load main class from JAR G:/guru with URI G. Please
  s through --class.
  Run with --help for usage help or --verbose for debug output

What could be the cause of this error?

use this command
.\bin\spark-submit \guruorders\pcakinetics\files\kineticPCA.py


How to represent a set of sets as a vector I'm pretty new to machine learning.

I know I can represent a set of discrete values as a vector of 0/1 values. For instance, in the set of features {a, b, c, d, e}, the subset containing {a, c} can be represented as [1, 0, 1, 0, 0] and the subset containing {c, d, e} can be represented as [0, 0, 1, 1, 1], meaning I have as many dimensions as elements, which is workable when you have a finite (and small) number of elements.

But now, for a clustering task, I want to represent sets of sets, like, for instance, representing the set {{a, c}, {c, d, e}}. How can I do that? Here, the basic 0/1 approach won't work, as I'll have 2^n possible combinations. What is the workaround, if any?

Edit: here is the transcription as a less abstract, more business problem. I want to find clusters of people according to the trips they made. A trip consists in a set of cities visited, and a set of transportation used. For instance, people might have visited cities in the set {Rabat, Alger, Marrakech, Tunis, Hammamet} with transportation such as {car, plane, train}. A trip could be {Rabat, Marrakech, plane} or {Alger, Marrakech, Tunis, car, train}. Note the order in which cities were visited, or the order in which vehicles were used, is not considered.

An example of the items I want to find clusters of could be a person having made those two trips, represented as p1 = {{Rabat, Marrakech, plane}, {Alger, Marrakech, Tunis, car, train}}.

You are describing one-hot encoding. There is a slot for each element. If the element is present, the slot has a one, and if the element if not present, the slot has a zero.

Typically, people will encode orthogonal features in different dimensions. In your case, cities would be one dimension and transportation type would be another dimension. A given data point would be one-hot encoded in a matrix (a 2D collection of vectors). If you want to add people, you would another dimension. That would create a 3D tensor with each person in a row.

Another way to compress your data is to encode not the cities (nodes) but the path between the cities (edges). From that encoding, you can create a Laplacian matrix that sets up spectral clustering. Since you have multiple transportation methods for the routes, you can create clusters with multi-dimensional spectral clustering.


Data science and MapReduce programming model of Hadoop What are the different classes of data science problems that can be solved using mapreduce programming model?

Let's first split it into parts.

Data Science is about making knowledge from raw data. It uses machine learning, statistics and other fields to simplify (or even automate) decision making. Data science techniques may work with any data size, but more data means better predictions and thus more precise decisions.

Hadoop is a common name for a set of tools intended to work with large amounts of data. Two most important components in Hadoop are HDFS and MapReduce.

HDFS, or Hadoop Distributed File System, is a special distributed storage capable of holding really large data amounts. Large files on HDFS are split into blocks, and for each block HDFS API exposes its location.

MapReduce is framework for running computations on nodes with data. MapReduce heavily uses data locality exposed by HDFS: when possible, data is not transferred between nodes, but instead code is copied to the nodes with data.

So basically any problem (including data science tasks) that doesn't break data locality principle may be efficiently implemented using MapReduce (and a number of other problems may be solved not that efficiently, but still simply enough).

Let's take some examples. Very often analyst only needs some simple statistics over his tabular data. In this case Hive, which is basically SQL engine over MapReduce, works pretty well (there are also Impala, Shark and others, but they don't use Hadoop's MapReduce, so more on them later).

In other cases analyst (or developer) may want to work with previously unstructured data. Pure MapReduce is pretty good for transforming and standardizing data.

Some people are used to exploratory statistics and visualization using tools like R. It's possible to apply this approach to big data amounts using RHadoop package.

And when it comes to MapReduce-based machine learning Apache Mahout is the first to mention.

There's, however, one type of algorithms that work pretty slowly on Hadoop even in presence of data locality, namely, iterative algorithms. Iterative algorithms tend to have multiple Map and Reduce stages. Hadoop's MR framework reads and writes data to disk on each stage (and sometimes in between), which makes iterative (as well as any multi-stage) tasks terribly slow.

Fortunately, there are alternative frameworks that can both - use data locality and keep data in memory between stages. Probably, the most notable of them is Apache Spark. Spark is complete replacement for Hadoop's MapReduce that uses its own runtime and exposes pretty rich API for manipulating your distributed dataset. Spark has several sub-projects, closely related to data science:

Shark and Spark SQL  provide alternative SQL-like interfaces to data stored on HDFS
Spark Streaming makes it easy to work with continuous data streams (e.g. Twitter feed)
MLlib implements a number of machine learning algorithms with a pretty simple and flexible API
GraphX enables large-scale graph processing

So there's pretty large set of data science problems that you can solve with Hadoop and related projects.


Which data science framework should I start learning? I have learned Python but am confused about which data science framework should I start off with? Matplotlib or Pandas or Numpy? Also, can you recommend which tutorials I should look for each one of them?

Matplotlib is for visualization. Pandas is for handling data frames. Numpy is more matrix-oriented. I feel like a good starting point is numpy with the various casting/slicing functionalities and some libraries such as script/optimization. Matplotlib would help you “see” what you are doing. For data science specifically, data frames are easier to deal with and more affine to reality (they are tables), so pandas are more indicated for starting. For Machine Learning a good source is the book by Ageron on scikit learn and tensor flow. There you could see some applications of the various libraries. If instead, you want to see some hard theory, if I remember correctly  “statistical machine learning” is great as a book (don’t remember the author). I personally did a little bit of both and it ended up being useful. For basic models, it’s definitely better if you understand the theory first.


Statistics + Computer Science = Data Science? i want to become a data scientist. I studied applied statistics (actuarial science), so i have a great statistical background (regression, stochastic process, time series, just for mention a few). But now,  I am going to do a master degree in Computer Science focus in Intelligent Systems.

Here is my study plan:

Machine learning
Advanced machine learning
Data mining
Fuzzy logic
Recommendation Systems
Distributed Data Systems
Cloud Computing
Knowledge discovery
Business Intelligence
Information retrieval
Text mining

At the end, with all my statistical  and  computer science knowledge, can i call myself a data scientist? , or am i wrong?

Thanks for the answers.

Well it depends on what kind of "Data Science" you wish to get in to.  For basic analytics and reporting statistics will certainly help, but for Machine Learning and Artificial Intelligence then you'll want a few more skills

Probability theory - you must have a solid background in pure probability so that you can decompose any problem, whether seen before or not, into probabilistic principles.  Statistics helps a lot for already solved problems, but new and unsolved problems require a deep understanding of probability so that you can design appropriate techniques.
Information Theory - this (relative to statistics) is quite a new field (though still decades old), the most important work was by Shannon, but even more important and often neglected note in literature is work by Hobson that proved that Kullback-Leibler Divergence is the only mathematical definition that truly captures the notion of a "measure of information". Now fundamental to artificial intellgence is being able to quantify information.  Suggest reading "Concepts in Statistical Mechanics" - Arthur Hobson (very expensive book, only available in academic libraries).
Complexity Theory - A big problem many Data Scientists face that do not have a solid complexity theory background is that their algorithms do not scale, or just take an extremely long time to run on large data.  Take PCA for example, many peoples favourite answer to the interview question "how do you reduce the number of features in our dataset", but even if you tell the candidate "the data set is really really really large" they still propose various forms of PCA that are O(n^3).  If you want to stand out, you want to be able to solve each problem on it's own, NOT throw some text book solution at it designed a long time ago before Big Data was such a hip thing.  For that you need to understand how long things take to run, not only theoretically, but practically - so how to use a cluster of computers to distribute an algorithm, or which data structures take up less memory.
Communication Skills - A huge part of Data Science is understanding business.  Whether it's inventing a product driven by data science, or giving business insight driven by data science, being able to communicate well with both the Project and Product Managers, the tech teams, and your fellow data scientists is very important.  You can have an amazing idea, say an awesome AI solution, but if you cannot effectively (a) communicate WHY that will make the business money, (b) convince your collegues it will work and (c) explain to tech people how you need their help to build it, then it wont get done.


Period Predictive Model I am not sure how to formulate this problem clearly into a machine learning task yet. So hope you guys can chime in and give me some help.

Problem : To predict whether someone will pick up their phone during office hours in week n+2 by looking at customer's behaviour in week  n.

Data    : I have calling records for about 3 months, which is aggregated on customer level. The various attributes include, num of calls, duration of calls, time of calls, amount of data traffic. But of course, these main attributes are further split into at about 20 attributes.

Current Approach (Very Manual) : I look at data at week n+2 and get the group of guys who picked up the phone during office hours (duration of calls > 5s and time of call). This is the target group, T.
I look at data at week n and manually try all possible combinations of the attributes to get as close to T as possible. But trying manually seems tiring after some time. The baseline is of course using the same conditions as at week n+2. But the whole idea will be to increase this number.

Question : Is there any way I can transform this dataset so that I would be able to do accomplish it as a machine learning task ?

You can try to build some kind of "sliding window table". Let's say you have following attributes:

call duration (x1)
time of call (x2)
picked the phone (x3)

Let's further assume that you have data from past 3 weeks, which allows us to set following table. The rows contain individual calls, the columns the attributes. The appendix _1 tells us the time. So for example x1_1 is call duration previous week, x2_2 is time of call two weeks before etc.

client | x1_1 | x2_1 | x3_1 | x1_2 | x2_2 | x3_2 | ... | x3_3

You can train your model using historical data, where x3_3 is last week. Then, you will feed the model with current data (_3 is current week*) and try to predict x3_3 - whether the customer will pick the phone.

*I am assuming that you know who you are going to call, hence you have _3 attributes, but you don' know yet whether they respond or not

The aim is to give the model the opportunity to learn the time dependencies - maybe time of call week before together with call duration strongly correlate with future chance of picking up the phone again.

What can also help is perform feature selection. The assumption is that some attributes are strongly correlated with others, whereas others are not. You can simply use x1_1, x2_1 and see the relationship to x3_1. But I'd suggest recalculate these often as the preferences might change in time.


Implementing the SVHN CNN architecture in Srivastava et al. 2014 Dropout paper I am trying the implement the CNN architecture introduced in Srivastava et al. 2014 Dropout paper (appendix B.2), for the SVHN dataset. I implemented only the convolutional layers part, without dropout or any regularization for the moment, following this description of the mentioned paper:
"The convolutional layers have 96, 128 and 256 filters respectively.
Each convolutional layer has a 5 × 5 receptive field applied with a stride of 1 pixel. Each
max pooling layer pools 3 × 3 regions at strides of 2 pixels."

This is my code, implemented in TensorFlow 2.0 with Keras API

    from tensorflow.keras import layers, Sequential

    model = Sequential(name= "fMap_svhn_DANN")

    model.add(layers.Conv2D(filters= 96, kernel_size= 5, activation= 'relu', input_shape= (32,32,3)))
    model.add(layers.MaxPool2D(pool_size= 3, strides= 2))

    model.add(layers.Conv2D(filters=128, kernel_size= 5, activation= 'relu'))
    model.add(layers.MaxPool2D(pool_size= 3, strides= 2))

    model.add(layers.Conv2D(filters= 256, kernel_size= 5, activation= 'relu'))
    model.add(layers.MaxPool2D(pool_size= 3, strides= 2))

    model.add(layers.Flatten())

And this is the error I get"

Negative dimension size caused by subtracting 5 from 4 for 'conv2d_11/Conv2D' (op: 'Conv2D') with input shapes: [?,4,4,128], [5,5,128,256].

Any idea to help me ?

This means that your input shape was too small by the time it reached the last Conv2D layer.

By removing the last Conv2D and MaxPool2D layer, this is how the model looks like:

You can see that the last output before the Flatten layer has height and width of 4, which is smaller than your kernel_size of 5 in the last Conv2D layer.

Heres what you can do:

Increase the input image height and width OR
Set padding='same' in the Conv2D layers


Calculating distance between data points when there are more than 3 features in KNN algorithm I've been reading about K-nearest neighbors algorithm and want to clarify few things.
If we have 2 features we could simply plot it on 2-d plane and calculate distance by using euclidean distance or Manhattan distance.
When there are more than 3 features, exceeding 3-d going into 4-d and more I've read that we use PCA to reduce dimension to 2-D and then calculate distance on PCA plot.
But my question is, is that only way? so in order to use KNN for more than 3 features we must use PCA?

No, you can definitely search for k-NN with more than 2-dimension data. Here is an example based on sklearn:
X = [[0, 0, 0], [3, 3, 3], [1, 2, 3]]
from sklearn.neighbors import NearestNeighbors
neigh = NearestNeighbors(n_neighbors=2)
neigh.fit(X)
print( neigh.kneighbors([[2,2,2]]) )

PCA is used to reduce the input dimensionality but this is not mandatory before searching k nearest neighbors (it is often used in tutorials so the data could be visualized on a 2-d plot).
One thing to know/understand about k-NN is that if you plan to use it for classification, it will handle features with a lot of information the same way as the features with no information (if you normalize them). PCA could be used to handle this problem (but this is not the only way and would not always work, but I think this is another question :) ).


How to use SKLEARN PIPELINE I have a question on using sklearn pipelines to predict/classify data.
I understand how to build a pipeline, train it with training data, test it with test data; but after that is where I get lost. How do I use the pipeline to predict values on new/unseen data?

I built a pipeline which transforms categorical data with a OneHotEncoder and another that transforms numerical data by scaling the columns with a StandardScaler. I then used FeatureUnion to combine the two pipelines and join a DecisionTreeClassifier() at the end, to classify the data.
I fit the training data sucessfully with pipeline.fit(X_train,y_train) and thereafter predict with pipeline.predict(X_test). This all works fine.

Next I want to use the pipeline to predict classification of new/unseen data; but when I call pipeline.predict(X_unseen) I get a ValueError. The error relates to one of the categorical features of the data, which is the names of a cities. The pipeline does not seem to transform the unseen data.

Reading through the documentation on pipelines and several examples I understand that when .fit() and .predict is called the data is passed through the entire pipeline. If my understanding is correct then the pipeline.predict(X_unseen) should pass the new data through the pipeline, transforming it and then classifying it. However this does not seem to be the case. Can anyone tell me what I'm missing or misunderstanding?
How do I use the pipeline I built to predict on new data?

Could you share the code? It is kind of hard to debug with out the code

When you do pipeline.predict it uses the transform method for all the steps.

Might be that in the One Hot Encoding you have not specified how to handle the unknown.

from sklearn.preprocessing import OneHotEncoder
enc = OneHotEncoder(handle_unknown='ignore')


Machine learning system which can learn from data and human rules I have a data set and few human rules. I want to learn the machine learning system. One of the paper I came across with is:

Harnessing Neural Network

Above paper trained some teacher network to learn the system in order to improve the accuracy but they still require enough data.
But my requirement is different. It may be possible that we have very few data points corresponding to a rule. How can I design a system which can utilize the both learning from data set and human rules.

The area of combining machine learning and human knowledge is called expert-augmented machine learning.
One common method is called RuleFit. RuleFit has two steps:

Fit a tree-based model to make rules.
Fits a linear model with these rules as input.

The result is highly interpretable which can be filtered or applied by humans.


Making predictions from keras with SciKit Ive been thinking about combining some processes between keras and Sci-kit Learn and am looking to the this group to either validate my process or tell Im crazy. Im creating a simple Regression problem using 17 inputs like this:

Creating test/train here:

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.25, random_state=7)

Building the network here:

model = Sequential()
model.add(Dense(34, input_dim=17, kernel_initializer='normal', activation='relu'))
model.add(Dense(17, kernel_initializer='normal', activation='relu'))
model.add(Dense(8, kernel_initializer='normal'))
model.add(Dense(1, kernel_initializer='normal'))

compiling the model with:

model.compile(loss='mean_squared_error', optimizer='adam')

fitting the model here:

model.fit(X_train,y_train,validation_data=(X_test,y_test), epochs=100, batch_size=10)

Now that Ive fit the model is the any reason I can use some of the SciKit functions and do the following?

Make predictions

y_pred = model.predict(X_test)

Assess the model results:

mse = mean_squared_error(y_test, y_pred)
rmse = sqrt(mse)
r2score = r2_score(y_test,y_pred)

Am I way off-base here?

This seems about right.

You can use SciKit learn quite easily, as the predictions and test results you have should all be in NumPy arrays anyway. Take a look at the regression metrics. The metrics you have named are shown in the documentation to take lists, but here is an example showing they work just as well with NumPy arrays:

In [1]: from sklearn.metrics import r2_score

In [2]: y_true = [3, -0.5, 2, 7]

In [3]: y_pred = [2.5, 0.0, 2, 8]

In [4]: r2_score(y_true, y_pred)
Out[4]: 0.94860813704496794

In [6]: import numpy as np

In [13]: y_true = np.array([3, -0.5, 2, 7])

In [14]: y_pred = np.array([2.5, 0.0, 2, 8])

In [15]: r2_score(y_true, y_pred)
Out[15]: 0.94860813704496794            # identical result

One last comment:

17 examples doesn't sound like a lot. Look at model.summary() after compilation to see how many parameters your model has (spoiler - it's 1360). I would expect that you model (with your number of epochs and batch size etc.) will overfit, just memorising the dataset, and probably score 100%.

While this is a good sanity check to make sure your model can indeed learn, it might be a good idea to split a larger dataset (if available) into train, validation and test datasets. The simply use train and val as you have above, but in the prediction line, us the test set, which the model has never seen before. Unless your data is extremely homogenous, I wouldn't expect a accuracy metric near 100%.


Tensorflow (or Keras) vs. Pytorch vs. some other ML library for implementing a CNN I am looking into implementing a convolutional neural network for a research problem. I've heard of deep learning libraries like Pytorch and Tensorflow and was hoping to get some additional information about their suitability for my needs.

I haven't looked much into Pytorch, and have only briefly read about Tensorflow. I don't hear very nice things about Tensorflow in terms of ease of use. I hear Pytorch is easier to use. But there seems to be more tutorials for Tensorflow, and specifically for creating CNNs.

What sort of questions should I be asking myself in determining which library would best suit my needs?

If you are looking for something easy to use and to read, definitely go for Keras.

Example of CNN in Keras :

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
metrics=['accuracy'])

So easy to read !

Source, literally the first link when searching for "keras CNN" on Google.

I really enjoy Keras, because it's easy to read, easy to use, great documentation, and if you want to mess up things at lower level you can do it by touching the back-end of Keras (Tensorflow or Theano)

EDIT (following your comment)

Excellent blog : Keras vs Tensorflow


keras Sequential CNN for image data reshaping data issues I am new at keras and CNN and am working on building at CNN  for sequential analysis of movement in a image. What I am having issues with is the reshaping the data and the labels that go into the fitting and testing the data for the model.
So the original size/shape of the numpy file is (18, 50,50,16) which is saved in a text file from another program.  I know the text file is ok because I can read it in and display it correctly with the debug_potion method. So that looks good. There are 18 images in the folder for the data and otherData variable. I dont really know what the 16 is but the image size are 50*50.

The issue is the reshaping of that data is the problem. Can anyone suggest how to reshape this data in a way that I can train it. I think I need to do onehot encoding but not quite sure how. Any help will be appreciated. Here is what I have so far.

def debug_potion(data):
    for i in range(0, 16):
        # get for each joint
        potion = Potion(data)
        plt.show(potion.display(joints=[i, i], channels=[0, 1, 2]))

# Build Model for sequential building
def create_model(input_shape):
    model = Sequential()
    model.add(Conv2D(32, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=input_shape))
    model.add(Conv2D(64, (3, 3), activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.25))
    model.add(Flatten())
    model.add(Dense(128, activation='relu'))
    model.add(Dropout(0.5))
    model.add(Dense(num_classes, activation='softmax'))

    model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])
    return model

def read_in_data(file):
    with open(file, "rb") as f:
        return pickle.load(f)

batch_size = 1
num_classes = 2
epochs = 12

# input image dimensions
img_rows, img_cols = 50, 50
input_shape = (img_rows, img_cols, 16)

data = read_in_data("heatmap.txt")
otherData = read_in_data("othermove.txt")

data = np.array(data)
otherData = np.array(otherData)
print("Data shape", otherData.shape)

xtrain = []
ytrain = []
xtest  = []
ytest  = []

for x in range(0,len(data)):
    if x < 15:
        xtrain.append(data)
        ytrain.append(("FWAC", num_classes))
    else:
        xtest.append(data)
        ytest.append(("FWAC", num_classes))

for x in range(0,len(otherData)):
    if x < 15:
        xtrain.append(otherData)
        ytrain.append(("Other", num_classes))
    else:
        xtest.append(otherData)
        ytest.append(("Other", num_classes))

#Create x test and train arrays
xtrain = np.array(xtrain)
xtrain = xtrain.reshape(30,img_rows, img_cols,16)
ytrain = np.array(ytrain)

xtest = np.array(xtest)
xtest = xtest.reshape(108, img_rows, img_cols,16)
ytest = np.array(ytest)
print(xtrain.shape)
print(xtest.shape)
print(ytrain.shape)
print(ytest.shape)

# Build Model
model = create_model(input_shape)

model.fit(xtrain, ytrain,
          batch_size = batch_size,
          epochs = epochs,
          verbose=1,
          validation_data=(xtest, ytest))

score = model.evaluate(xtest,ytest, verbose=0)
print(("Test loss", score[0]))
print("Test Accuracy", score[1])

As far as labels are concerned, you can one-hot-encode by using (assuming ytrain is converted to numpy array) the below code. Image reshaping looks fine but if you are having issues with image reshaping then, you might be giving the first argument i.e., the number of images wrong. So try this

    xtrain = xtrain.reshape(xtrain.shape[0],img_rows,img_cols,16)

    ytrain = keras.utils.to_categorical(ytrain, num_classes)

Make sure you import to_categorical from keras.utils


Newbie: What is the difference between hypothesis class and models? I am new to machine learning and I am confused with the terminology. Thus far, I used to view a hypothesis class as different instance of hypothesis function... Example: If we are talking about linear classification then different lines characterized by different weights would together form the hypothesis class.

Is my understanding correct or can a hypothesis class represent anything which could approximate the target function?  For instance, can a linear or quadratic function that approximates the target function together form a single hypothesis class or both are from different hypothesis classes?

Your hypothesis class consists of all possible hypotheses that you are searching over, regardless of their form. For convenience's sake, the hypothesis class is usually constrained to be only one type of function or model at a time, since learning methods typically only work on one type at a time. This doesn't have to be the case, though:

Hypothesis classes don't have to consist of only one type of function. If you're searching over all linear, quadratic, and exponential functions, then those are what your combined hypothesis class contains.
Hypothesis classes also don't have to consist of only simple functions. If you manage to search over all piecewise-$\tanh^2$ functions, then those functions are what your hypothesis class includes.

The big tradeoff is that the larger your hypothesis class, the better the best hypothesis models the underlying true function, but the harder it is to find that best hypothesis. This is related to the bias–variance tradeoff.


Get Label Statistics of Image Dataset I have a labeled image dataset, where the images are in subfolders and there is one Pascal XML per image with the labels.

I would like to compute stats like: how many images have exactly two labels?

Or - what is the average size of the labeling rectangle?

Ideally also statistics on image resolution, file size etc, but mostly labels.

This is probably an easy question (many papers include that info), but did not see that function in labelImg and Sloth. How can I do that?

From this SO answer

https://stackoverflow.com/a/53832130/1727543

import xml.etree.ElementTree as ET
import os

def read_content(xml_file: str):

    tree = ET.parse(xml_file)
    root = tree.getroot()

    list_with_all_boxes = []

    for boxes in root.iter('object'):

        filename = root.find('filename').text

        ymin, xmin, ymax, xmax = None, None, None, None

        for box in boxes.findall("bndbox"):
            ymin = int(box.find("ymin").text)
            xmin = int(box.find("xmin").text)
            ymax = int(box.find("ymax").text)
            xmax = int(box.find("xmax").text)

        list_with_single_boxes = [xmin, ymin, xmax, ymax]
        list_with_all_boxes.append(list_with_single_boxes)

    return {'filename':filename,'box_list': list_with_all_boxes, 'box_len':len(list_with_all_boxes)}

result = []
for file in os.listdir("path_to_Annotation_xml_folder"):
    result.append(read_content(file))

Extract more relevant features using 'EelementTree' to add your analysis.


Modelling population changes with years on a network graph This may be a silly question, apologies if it is. And further apologies if I've said something wrong.

Network graphs, or visualisations look somewhat like this, with each node representing a thing connected to another thing. So for my example, I'd like to model population changes for refugees from their origin country to their destination country.

I've been tasked to create a graph which shows a similar change over a period of years, but I am really not sure how years could be modelled? e.g. how would a graph like the example above, show a change in years?

Edit: preferably using the NodeXL plugin

The Sankey diagram would be nicely suited for your problem statement.

The flow width can be the number of refugees migrated, and the end nodes can be the destinations (from and to) of the refugees, and the flow width would be the magnitude of refugee migration.

If you want to model the graph on a geographical map, it can look something like this:

I am not sure whether it can be done in NodeXL or not.  But, a google search has returned me this link.


Anomaly detection in a database We have a production database. The load on the database varies at different times. I want to identify anomalies; for e.g, the number of database processes responding to user queries at 9 am is 100 for a given day. If the number is 200, then it's an anomaly and as the DBA, we need to check the DB immediately. The goal is to identify a pattern and alert when there's an event outside this pattern.

Day  time  processcount label
Mon  09:00 100          Normal
Mon  09:05 150          Normal
Tue  09:00 200          Abnormal

I'm using pandas to collect the data but I'm not sure how to identify the pattern and report anomalies. The closest i could get is this thread
How do I approach grouped anomaly detection?

There are some packages for anomaly detection such as Linkedin's Luminol (https://github.com/linkedin/luminol) or Microsoft's TagAnomaly (https://github.com/microsoft/TagAnomaly).
Also you can use clustering algorithms for your features and detect outlier clusters. Or you can tag your previous anomalies to train your data (however its not possible most of the cases)

So my suggestion is prepare your data with features such that 5_min_window_processcount_mean, 5_min_window_processcount_std, 5_min_window_processcount_std and 5_min_lag_processcount_std etc.(get lagged and windowed features and then calculate their mean,std,median etc)

After that try clustering and check it if it can find anomalies or not. If not try to label your data with those features and try classification algorithms. Meantime you can use above packages with your features. (time based lag/window features are very important).


What's a good machine learning algorithm for low frequency trading? I'm trying to train an algorithm to copy some of the top traders on various forex social trading sites. The problem is that the traders only trade around say 10 times per month so even if I only look at minute resolution numbers that's .02% of the time [ 10/(60*24*30)*100 ].

I've tried using random forest and it gives an error rate of around 2% which is unacceptable and from what I've read most machine learning algorithms have similar errors rates.

Does anyone know of a better approach?

Random forests, GBM or even the newer and fancier xgboost are not the best candidates for binary classification (predicting ups and down) of stocks predictions or forex trading or at least not as the main algorithm. The reason is that, for this particular problem, they require a huge amount of trees (and tree depth in case of GBM or xgboost) to obtain reasonable accuracy (Breiman suggested using at least 5000 trees and to "not be stingy" and in fact his main ML paper on RF he used 50,000 trees per run).

However, some quants use random forests as feature selectors while others use it to generate new features. It all depends on the characteristics of the data.

I would suggest you read this question and answers on quant.stackexchange where people discuss what methods are the best and when to use them, among them ISOMAP, Laplacian eigenmaps, ANNs, swarm optimization.

Check out the machine-learning tag on the same site, there you might find information related to your particular dataset.


Conv1D specify output chanels in tensorflow 2.1 Hello I'm trying to implement the "Tacotron Towards end to end" paper and in the
Encoder CBHG - a Conv1D bank of K=16, conv-k-128-ReLU
“conv-k-c-ReLU” - denotes 1-D convolution
with width k and c output channels with ReLU activation.
And I wanted to ask how exactly I can implement it in Tensorflow 2.1? because in the documentation I haven't seen how to specify the output channels

Reference to the paper: https://arxiv.org/pdf/1703.10135.pdf

Refer to this: https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv1D

The parameter filter determines the output channel.


Creating a generic mathematical formula using a genetic algorithm Assuming all of the following;

I have 4 known numbers, all within a 0-400 range, like this:

Variable1   Variable2   Variable3   Variable4
0-400       0-400       0-400       0-400

I know that there is a mathematical relationship between the numbers.
I would like to use a genetic algorithm (computer code) to estimate/approximiate Variable2 and Variable3 based on Variable1 and Variable4.
Also, importantly, assume that there are many input samples and that each sample will differ slightly. Thus, a genetic algorithm optimization of "a mathematical formula/algorithm" to estimate/approximate Variable2 and Variable3 in all cases becomes possible.
(In other words, the genetic algorithm will be able optimize the
mathematical formula towards the already-known Variable2 and
Variable3 across many input samples, each with a similar though
slightly different mathematical formula.)

How can I then write the following into a genetic algorithm:

Variable2=?
Variable3=?

Where ? could be any mathematical function (+/-/*/:/√/^2/cos/sin/tan/etc.) involving Variable2 and Variable3

In other words; I would like the genetic algorithm to build a generic mathematical formula.

How can I define Variable2 and Variable3 as the outcome of a mathematical formula so that estimation by a computer algorithm becomes possible?

I am not sure how to approach this. The genetic algorithm software I use can use as many variables as is needed, and they can be in disparate ranges.

So for example, I could write my algorithm like this easily;

Variable2=Variable1(op)Variable4
Variable3=Variable1(op)Variable4

Where Variable1 is the first variable for the genetic algorithm, with a range of 0-400, and Variable4 is the second variable for the genetic algorithm, with a range of 0-400, and finally (op) is the third variable for the genetic algorithm, for example with a range of 1-4 where 1 stands for +, 2 for -, 3 for *, 4 for : etc.

However, the complexity of this algorithm is very limited and crude; it is not optimized towards a nice and complex real estimation algorithm. Also, as soon as a secondary operator is introduced, for example:

Variable2=[Variable1 or Variable4](op)[Variable1 or Variable4](op)[Variable1 or Variable4]
Variable3=[Variable1 or Variable4](op)[Variable1 or Variable4](op)[Variable1 or Variable4]

The coding complexity for this would start to increase quickly, and there may be a need to use ( and ) to prioritize mathematical calculations, etc. The coding complexity for even more complex calculations becomes seemingly unmanageable.

Is there a better and more straightforward way to let the genetic algorithm approximate/estimate Variable2 and Variable3 based on Variable1 and Variable4 into an overall optimized generic mathematical formula/algorithm?

I would suggest to use a sequence of operations of the form
op1(var) op2 op1(var) op2 op1(var) ...
where op1 refers to a unary operation and op2 refers to a binary operation. Making unary operations mandatory gets rid of a distinction of cases. If you don't want a unary operation in front of a variable, just use the identity function.

I wrote two implementations for unary composition of a list/tuple of unary operations. One in Haskell, one in Python:

square:: Num a => a -> a
square x = x * x

compose_multiple:: [a -> a] -> (a -> a)
compose_multiple []     = (\x -> x)                    -- for the empty case, use the identity
compose_multiple (f:fs) = f . compose_multiple fs  -- recursive case

This gives us 256 for

compose_multiple [square, square, square] 2

In Python:

def compose(op1, op2):
    def result(x):
        return op1(op2(x))
    return result

def compose_multiple(operations: tuple):
    if len(operations) == 1:
            return operations[0]
    return compose(compose_multiple(operations[:-1]), operations[-1])

This gives us 256 for

def square(x):
  return x * x
compose_multiple((square, square, square))(2)

An implementation for the composition of binary operations should work on the lines of the unary case.

The complexity of the final code can be reduced by using infix notation (which is easy to do in Haskell using parenthesis () in the function definition).
I have no experience using infix notation in Python. Maybe https://pypi.org/project/infix/ might help.


How to implement gridsearchCV for onevsrestclassifier of LogisticRegression classifier? parameters = [{'C': [10**-2, 10**-1, 10**0,10**1, 10**2, 10**3]}]

model_tunning = GridSearchCV(OneVsRestClassifier(LogisticRegression(penalty='l1')), param_grid=parameters,scoring="f1")
model_tunning.fit(x_train_multilabel, y_train)

ValueError                                Traceback (most recent call last)
<ipython-input-38-5d5850fe8978> in <module>()
  2
  3 model_tunning = GridSearchCV(OneVsRestClassifier(LogisticRegression(penalty='l1')), param_grid=parameters,scoring="f1")
----> 4 model_tunning.fit(x_train_multilabel, y_train)

ValueError: Invalid parameter C for estimator OneVsRestClassifier(estimator=LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
      intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,
      penalty='l1', random_state=None, solver='liblinear', tol=0.0001,
      verbose=0, warm_start=False),
      n_jobs=1). Check the list of available parameters with `estimator.get_params().keys()

When you use nested estimators with grid search you can scope the parameters with __ as a separator. In this case the LogisticRegression model is stored as an attribute named estimator inside the OneVsRestClassifier model:
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn.multiclass import OneVsRestClassifier

tuned_parameters = [{'estimator__C': [100, 10, 1, 0.1, 0.01, 0.001, 0.0001]}]

# Find Optimal C by grid search

log_reg_clf = OneVsRestClassifier(LogisticRegression())

logistic_gs = GridSearchCV(log_reg_clf, tuned_parameters,scoring = 'f1_micro', cv=3)

logistic_gs.fit(x_train_bow, y_train)
print(logistic_gs.best_estimator_)


Cluster analysis as an associative model? I have a set of data with many samples and many features, but where half of the data is missing one variable (call it A), which is composed of four categories. Based on the half of data which has A, I want to know what category the samples without A would most likely be in if they did have A.

I could build a classifier based on the data with A, and predict the data without A (this is the best route IMHO).

But I'm wondering, out of curiosity, if this method could also be a very, very, very rough way of doing something similar:

Cluster the data which has A into the same number of clusters as categories in A (in this case four).
Check for an an association between the clusters and the categories in A (using a frequency table and chi-square test).
If there is an association, run the data without A through the clustering model to figure out what category of A it's most likely associated with (based on what cluster it is in).

Thoughts?

I think the question you're wrestling with is essentially this: Is there a way to use information that may be present in the data without A as part of your strategy for predicting A?  There is actually a name for the set of methods that do exactly that: semi-supervised learning.

While there are multiple techniques, a method analogous to what you suggest in your question would be something like the following.  Here I refer to the set of data with A as labeled and the rest as unlabeled.

Apply an unsupervised learning technique (such as clustering) to the full data set (both labelled and unlabelled).
Transform the labelled data based on the result of your unsupervised learning technique.
Apply a supervised learning technique with the transformed set (possibly combined with the original set) of labelled data.

Generally you would want to apply a cross-validation strategy as well to detect overfitting.


read csv file directly from URL / How to Fix a 403 Forbidden Error The csv file is downloadable. I can download the file and use read_csv, But I want to read the file via direct URL in jupyter, I used the following code, but I get the HTTP 403 Forbidden
 error

from io import StringIO

import pandas as pd
import requests
url="https://fineli.fi/fineli/en/elintarvikkeet/resultset.csv"
s=requests.get(url).text

c=pd.read_csv(StringIO(s))
c

how do I read the csv file via URL directly in python with a delimeter ";"

The problem is that the url you have doesn't accept "non-browser" requests. The default header of Python requests is

'User-Agent': 'python-requests/2.13.0'

You can pass your own headers as an argument like that

from io import StringIO
import pandas as pd
import requests

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.76 Safari/537.36'}

url="https://fineli.fi/fineli/en/elintarvikkeet/resultset.csv"
s=requests.get(url, headers= headers).text

c=pd.read_csv(StringIO(s), sep=";")
c


Classification using texts as features I want to build a classification model to match customers and products. I have a description of each product, and a description of each customer, and the label : customer *i* buy/did not buy product *j*.
Each sample/row is a pair (customer, product), so Feature 1 is customer's description, Feature 2 is product's description, and the target variable y is: "y = 1 : customer buys product", "y = 0 otherwise". The goal is to predict for new arriving products whether each customer is going to buy them or not.
I want to use Tf-Idf Vectorizer. I don't in which specific step I should fit_transform the descriptions, and how to put together Feature 1 with Feature 2.

Should I concatenate the descriptions of each pair (customer, product) and fit_transform only once I have the concatenation?

Should I put together 2 columns using ColumnTransformer? If so, is the classifier going to fit correctly the obtained features?

Should I transform using a unique vocabulary?

I found here a reference of three possible ways of working with two columns, but I don't see which one fits for my case.
Ps. Until now, I only got to build a similarity pairwise coefficient (using this), but there is no classification, and I know using labelled data can help. In particular, similarity measure gives the same weight to any text coincidence, but some coincidences should be more important than others.

One uses a frequency or TFIDF representation in the features when the target directly depends on specific words. For example for spam classification words like "cheap, free, viagra, exclusive..." are direct indicators of the target label.
In your case the target doesn't directly depend on specific words, it depends whether the same words appear in both the customer and product descriptions. This is an indirect relationship and most regular ML algorithms cannot really deal with that. So your design is unlikely to work in my opinion.

Until now, I only got to build a similarity pairwise coefficient (using this), but there is no classification, and I know using labelled data can help. In particular, similarity measure gives the same weight to any text coincidence, but some coincidences should be more important than others.

This makes more sense for your purpose: use only the similarity score as a feature and train a model to predict the label. Technically the model will only learn the optimal threshold to separate the labels, so you can just use linear regression for instance. You could improve this method by calculating different types of similarity measures and provide all of them as features.
Note: if you use TFIDF vectors for measuring similarity, it doesn't give the same weight to every word. However don't expect perfect result: a lot depends on the data itself, are you sure that the customer description gives useful indications about the products they're interested in? For example if a customer description contains the word "computer" it doesn't mean they're interested in every possible type of computer.


Select samples from a dataframe in python I have a data set (pandas dataframe) with a variable that corresponds to the country for each sample. I have to take the samples that corresponds with the countries that appears the most.

thanks

If I understand your question correctly you can break this problem down into two parts:
1. Figuring out which country occurs most frequently and then
2. Subsetting the pandas dataframe to that country

import pandas as pd
from scipy.stats import mode

# 1
mock_df = pd.DataFrame([{'country': 'a'}, {'country': 'b'}, {'country': 'a'}])
print(mock_df)
#   country
# 0       a
# 1       b
# 2       a
mode_country, _ = mode(mock_df.country)

# 2
mock_df[mock_df.country == mode_country[0]]

#   country
# 0       a
# 2       a
```


How Can I Solve it? TypeError: fillna() got an unexpected keyword argument 'implace' I am trying to replace NaN values in a given dataset with this

import pandas as pd
import quandl
import math

df.fillna(-9999, implace=True)

But I keep on getting this error:

**

Traceback (most recent call last):

File "regression.py", line 44, in
    df.fillna(0,implace=True)

File "/home/compname/.local/lib/python3.6/site-packages/pandas/core/frame.py", line 3790, in fillna
    downcast=downcast, **kwargs)

TypeError: fillna() got an unexpected keyword argument 'implace'

**

You have a typo, change implace for inplace (change the m for the n)


Elastic net regression in orange The penalty term for Elastic Net Regression is written as:
$\lambda_1  \|\theta\|_1 + \lambda_2\|\theta\|_2^2$
How are the values of lambda's calculated if the slider is moved from the right to the left?
I have read most of the material available on the internet on Elastic Net Regression. Here, they define $\alpha=\lambda_2/(\lambda_1 + \lambda_2)$. If the value of $\lambda_2 $ and $\alpha$ is given, then $\lambda_1$ is calculated. In Orange the slider is provided.

Finally I found the answer. In elastic net the cost function is written as
$J(\theta)=MSE + r\alpha\sum_{i=1}^n \vert\theta_i \vert + \frac {1-r}{2}\alpha\sum_{1=1}^n \theta_i ^2 $. Here $r$ is the ratio, $\alpha$ is the hyperparameter and $n$ is the number of features. The ratio slider controls $r$ and $\alpha$ slider controls the value of hyperparameter.
If the ratio $r=0$, then Elastic net regression is equal to ridge regression. For $r=1$, the Elastic Net Regression is equal to Lasso regression. I hope this perfectly answers the question I posed. I would like to have comments from followers of this post.


What affects the magnitude of lasso penalty of a feature? Is there a way to intuitively tell if the lasso penalty for a particular feature will be small or large?
Consider the following scenario:

Imagine we use Lasso regression on a dataset of 100 features $ [X_1, X_2, X_3, ..., X_{100}] $. Now, if we take one feature and scale up its values by a factor of 10, so for example $X_1 = X_1 * 10$, and then we fit lasso regression on the new dataset, which of the following might be true?

X1 probably would be excluded from the fitted model
X1 probably would be included in the fitted model

Someone told me that 2 is true. Somehow the operation on $X_1$ would result in less lasso penalty for $X_1$, therefore the feature is likely to be kept.

I cannot understand why.
 Can anyone tell if this whole statement is not correct to begin with or if there is some truth in it?

When you increase the scale of $X_1$ by 10, the scale of the corresponding weight in you linear regression sees its scale divided by 10 (it is exactly 10 for standard linear regression and the same order of magnitude holds for lasso loss). As a consequence, the $L_1$ penalty for this coefficient is also divided by a factor 10 and it is less likely to be set to 0.


roc_auc score GridSearch I am experimenting with xgboost.

I ran GridSearchCV with score='roc_auc' on xgboost. The best classificator scored ~0.935 (this is what I read from GS output). But now when I run best classificator on the same data:

roc_auc_score(Y, clf_best_xgb.predict(X))

it gives me score ~0.878

Could you tell me how the score is evaluated in both cases? Thanks

Try using predict_proba instead of predict as below. It should give you the same number.

roc_auc_score(Y, clf_best_xgb.predict_proba(X)[:,1])

When we compute AUC, most of time people will use the probability instead of the actual classs.


Finding optimal threshold in multi-class classification task In a binary classification problem, it is easy to find the optimal threshold (F1) by setting different thresholds, evaluating them and picking the one with the highest F1. Similarly is there a proper way to find optimal thresholds for all the classes in a multi-class setting.

This will be a grid search problem if we do it brute force way. Any efficient way to do this?
Is there any package that ppl use or I can use for this?
Also is it common practice to find optimal threshold in multi-class settings, I couldn't any examples?

One solution is to explore a One-Vs-Rest classifier which creates separate binary classifiers for each class.


Why does this paper claim to have found a minimal width of $d_{in}+1$? Why does this paper (click the link) claim to have found a minimal width of $d_{in}+1$ in the abstract? I mean, if you read the main result, it seems like they only find a universal approximator with width $d_{in} + d_{out}$. What am I missing or did they make a mistake? I do not completely understand all the math but I do think I get the gist of it, and also $d_{in}+1$ is not even mentioned when showing a bound of the difference between the function approximated by the neural network and the actual function...

From the abstract of the paper:

Specifically, we answer the following question: for a fixed $d_{in} \ge 1$,
what is the minimal width $w$ so that neural nets with ReLU
activations, input dimension $d_{in}$, hidden layer widths at most $w$,
and arbitrary depth can approximate any continuous, real-valued
function of $d_{in}$ in variables arbitrarily well?

"Real-valued function" means that the output dimension $d_{out}=1$. Their Theorem 1 is more general and says that for any $d_{in}, d_{out} \ge 1$ you will need $d_{in}+1 \le w \le d_{in}+d_{out}$.


Large Numpy.Array for Multi-label Image Classification (CelebA Dataset) Task: Build CNN Model (preferably Keras or TensorFlow) to Predict Labels Associated to Each Image in CelebA Dataset (Multi-label Image Classification)

In past, for majority of multiclass/binary image classification problems, I used to feed images efficiently using ImageDataGenerator and .flow_from_directory in Keras after images are properly organized in a separate directory for each class. Therefore, I have never bothered converting images to numpy.arrary prior to feeding to the model, unless I had to and of course datasets were small so that I could do it easily in my local machine.

However, CelebA is a Multi-label Image Classification with each images having 40 labels (attributes like Smiling, Eyeglasses, Young etc.)  meaning that I can not organize them in subclasses as I used to do, so .flow_from_directory is off the table (as far as I know!). Still I've managed to convert the images to numpy.array by the following simple loop:

import numpy as np
import skimage.transform

images_path='../img_align_celeba/'

train_images=[]
from skimage import data

for filename in train['Images'].tolist()):
    tmp=np.array(skimage.transform.resize(io.imread(os.path.join(images_path,filename))/255., (64, 64)))
    train_images.append(tmp)

x_train=np.array(train_images)
del train_images

Well it was not an impossible task. CelebA dataset is large, well not super large compared to many other image datasets (>200K RGB images, totally 1.4GB in size, each image ~ 8 KB). YET surprisingly it takes the hell of the time to convert these images to numpy arrays and even stuck during the run of a small CNN model.

My computer specs: MacBook Pro (2015), Memory: 8GB, Harddisk: 128 GB.

Even with almost more than 4GM free memory, and 20 GB free hard-disk I could not manage this on my local machine.

UPDATE: It seems quite possible and way more efficient via .flow_from_directory method of ImageDataGenerator in Keras. While it is not that option for this multi label classification at hand, I simply made some dummy subclasses and it worked and the model runs much faster!!

To My Questions (finally!!):

Maybe I am not doing the image-to-numpy conversion efficiently? Please advice how I could construct the arrays in a more efficient approach!

UPDATE: I found a very similar question in stackoverflow a year ago, yet answers do not seem to offer any better alternative.

Maybe it is what it is and I only need better hardware to do it locally!?
How then Keras manage to do the conversion efficiently under the hood then?

At the end (as we speak), I sampled only 20% of the images to have at least a model prototype up and running, although accuracy is not impressive!!

I ended up writing a python generator, which actually works very well, for manually feeding desired number of images chunk by chunk into my CNN model like this:

def image_batch_generator(df,images_path, batch_size):

    '''
    A generator that takes a dataframe (for image names) and
    with a given image path goes to conver images to numpy array over
    batch (chunk by chunk).

    "df": is the "Attributes Annotations" text file from CelebA dataset.
    It has a column for image names, and another 40 attributes columns
    (binary) for each image.

    "images_path": it is the path to CelebA dataset image files.

    "batch_size": the batch size by which image will be read chunk by chunk.
    '''

    L = df.shape[0]
    files = df['Images'].tolist()

    #this line is just to make the generator infinite, keras needs that
    while True:

        batch_start = 0
        batch_end = batch_size

        while batch_start < L:
            limit = min(batch_end, L)

            X= np.array([np.array(skimage.transform.resize(io.imread(os.path.join(images_path,fname))/255., (64, 64))) for fname in files[batch_start:limit]])
            y=df.loc[df["Images"].isin(files[batch_start:limit]), :].drop(['Images'],axis=1).values

            yield (X,y) #a tuple with two numpy arrays with batch_size samples

            batch_start += batch_size
            batch_end += batch_size

Hope it will help someone out there.


Detection model - Training with class-instance limit-awareness Is there a way to make a detection model aware of the maximal number of possible objects of a given class within a single image?

For example in a toy case with 2 classes. If I know that in every single image, class A can have no more than 5 instances and class B no more than 1. Is there a way to incorporate it into the training process?

To make it clear, I'm not talking about an additional algorithm which runs on top of the trained model (such as non-maximum suppression which is used to select a single bounding box for an object). I specifically ask about the actual model and its training process.

I can think of the following approach:

Let's say that you have two classes, A and B. Additionally, you now that for class A there is at maximum 5 instances (so 0, 1, 2, 3, 4 or 5) and for B 1 instance (0 or 1).

For this purpose, you can have 6 outputs for class A and 2 outputs for class B.
Between those 6 outputs for class A, only one should be active; same for B - only one of two should be active.

For example, if on some image there are 3 objects of class A and 0 for B, the outputs would be: [0, 0, 0, 1, 0, 0] for A, and [1, 0] for B (or something very close to 0s and 1s, right?)

With these outputs, you can also combine other outputs which are needed for detection.