Distance calculation/vector range significance I'm trying to implement item based collaborative filtering. Do any distance calculations allow for weighting of certain ranges of values within each vector?

For example, I would like to be able to say values 10..22 within each vector are more significant than values within the range 0..10.

I've been experimenting with Pearson, Tanimoto and Euclidean algorithms, but they all seem to assume equal weighting for each value within the vector.

Am I approaching this problem in the right way, and if not, how do others deal with this problem?

I created a scoring system ("Thomas Scoring System") to deal with this problem.  If you treat "distance" as a similarity score, this system should work for you. http://exploringpossibilityspace.blogspot.com/2014/02/thomas-scoring-system.html

  Here's Thomas Scoring System (TSS) in a nutshell: Treat each metric and their values as evidence that weigh for or against specific values in the performance index. This is an inference process and not an arithmetic calculation, as in the Usual Method. The output of TSS is an estimation of the Weight of Evidence for all index values.  The Weight of Evidence can be expressed as a probability distribution, where all the evidence weights sum to one.

To use your example, each range is a condition: A) 10 <= x <= 22; and B) x < 10. If A is true, then it weighs in favor of certain similarity (distance) scores.  If B is true, then it weights in favor of other similarity (distance) scores, but would have less weight compared to other evidence.


Find intersection of data between rows and columns Below is an Excel sheet with crops and countries. "1" means that the crop is grown in that country whiles "0" means it is not grown in that country.

I want to find where there is an intersection (as in I want to find similar crops that are grown in most of the countries or at least most of the countries.) How do I do this ?

Suppose that you want to know the name of countries with more than ten Ginger:

# read your data using pd.read_csv and specify appropriate inputs based on your excel file in a typical variable named csvFile and transpose it
csvFile = csvFile[csvFile['Ginger'] > 10]

Also if you want to see the countries with zero Ginger:

csvFile = csvFile[csvFile['Ginger'] == 0]


How to do k-folds in python whilst splitting into 3 sets? Consider the following data:

   import pandas as pd
    wine = pd.read_csv(r'wine_data.csv', names = ["Cultivator", "Alchol", "Malic_Acid", "Ash", "Alcalinity_of_Ash", "Magnesium", "Total_phenols", "Falvanoids", "Nonflavanoid_phenols", "Proanthocyanins", "Color_intensity", "Hue", "OD280", "Proline"])
    X = wine.drop('Cultivator',axis=1) #input
    y = wine['Cultivator'] #output

y is what i am trying to predict and X is the input and i will be using some sort of mlp classifier. What I want to do is split this data into test, training and validation and then apply K-folds. I'm struggling to see how you do this..

I know that I can obtain validation, test and training by using the following:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.25, random_state=1)

But what I want to do now is apply k folds such that for each fold I have 3 sets: validation, testing , training rather than just 2 sets.

I know I can use the below for Kfolds:

kf = KFold(n_splits = 5, shuffle = True, random_state = 2)
X_np=np.array(X)
y_np=np.array(y)

After converting to a numpy array I can then do this:

for train_index, test_index in kf.split(X_np):
    print("TRAIN:", train_index, "TEST:", test_index)
    X_train, X_test = X_np[train_index], X_np[test_index]
    y_train, y_test = y_np[train_index], y_np[test_index]

But how do I get a 'validation_index'. In general the question is how do I use k-folds when I have 3 sets as opposed to just 2?

Also when do I normalize the data; do I normalize when i've split into X_train, X_test as above... or do I do it before?

Any help appreciated.

I don't think there's a builtin way to do it, but the two methods you've mentioned combine pretty nicely to do the job:

kf = KFold(n_splits = 5, shuffle = True, random_state = 2)

for train_index, test_index in kf.split(X):
    X_tr_va, X_test = X.iloc[train_index], X.iloc[test_index]
    y_tr_va, y_test = y[train_index], y[test_index]
    X_train, X_val, y_train, y_val = train_test_split(X_tr_va, y_tr_va, test_size=0.25, random_state=1)
    print("TRAIN:", list(X_train.index), "VALIDATION:", list(X_val.index), "TEST:", test_index)

(I also opted to keep things as dataframes.  You could even use "nested cross-validation," using another CV instead of the train_test_split inside the loop, depending on your needs and computational budget.)

For the question of normalizing data, you don't want to let information from the testing fold affect the training, so normalize within the loop, using only the training set;
https://stats.stackexchange.com/questions/77350/perform-feature-normalization-before-or-within-model-validation


Choosing best methods for estimating the unknown parameters in a linear regression model Given some dataset for prediction,

for eg say I have different housing price prediction dataset:

dataset  1 : 100 training and 100 testing sample, 50 feature

dataset  2 : 100 training and 100 testing sample, 120 feature

dataset  3 : 1000 training and 1000 testing sample, 50 feature

dataset  4 : 1000 training and 1000 testing sample, 5000 feature

how should I choose the best methods for estimating the unknown parameters ( predict price) in a linear regression model from the following for each of these dataset?

Ordinary least squares
Stepwise regression
Principal component regression
Partial least squares regression

Should I experiment with each of these one by one and compare the results or is there any rule of thump on when to use each of them based on the dataset ?

Please help

After the Data Munging, this is the most difficult task on a prediction model. However, in order to answer it, we need more details. What do you mean by the "best model"? Do you want accuracy and long training time? Do you need something really fast with lower accuracy? Something between the two of them? What are your features? Have you just taken them or have you created new features from them?

In any case, I suggest that you should spend some time and read this perfect tutorial from Microsoft about Machine Learning. A part of the tutorial to understand what I mean:

Regression

Ordinal regression: Data in rank ordered categories
Poisson regression: Predicting event counts
Fast forest quantile regression: Predict a distribution
Linear regression: Fast training, linear model
Bayesian linear regression: Linear model, small data sets
Neural Network regression: Accuracy, long training time
Decision forest regression: Accuracy, fast training
Boosted decision tree regression: Accuracy, fast training

When I have a similar question and I don't know which one to choose, I usually end up to 3-4 different algorithms based on the cheat sheet of Microsoft or the one from the scikit-learn, try them all and choose one or two of them with the best results.


How can I use mean normalization. Should I use it for numerical columns or categorical columns as well? Should we normalize the categorical columns in our dataset? Or just the numerical columns?

Normalization and standardization are transformations that can only be applied on metric variables. You cannot normalize categorical variables.

Either you label-encode your data or you one-hot-encode it, depending on the algorithm you use. That's all you do with it.


Model with two targets(classification, regression)? I have a dataset with features like Cylinders, Transmission, ExteriorColor, OdometerReading, StockType(New/Old), MakeModel, ModelYear, SoldYearMonth, CountSoldByMonth, Price, TopSelling.

Target column is 1 if a particular MakeModel is the top 5 selling vehicle in a month. So, given 60 months data I have to predict the next months top selling vehicles. I also have to predict the price go the vehicle. So I guess, I need to do both Classification and Regression .

MSRP   Cylinders   Transmission   ExteriorColor   OdometerReading   StockType     MakeModel            SoldYearMonth   CountSoldByMonth   Price   TopSelling
42098      8            MYC            Black           70348           Used    Chevrolet Tahoe            5/1/2019           10           39998     0
26390      4            MNK            Black            10             New      Buick Encore              5/1/2019           31           26390     1

and so on..
MSRP and Price are not always same.

How do I approach this problem?

You can also consider a multi-output model using the Keras functional API (see "Deep Learning with R", Ch. 7, p. 224). Here are the respective Keras docs. There are also some tutorials online. I'm not sure if a NN is suitable for your problem, but this would be an option to consider. The advantage is that your model can also take correlation between the features and the two outputs into consideration while learning (which may give better results).


Why is activation needed at all in neural network? I watched the Risto Siilasmaa video on Machine Learning. It's very well explained, but the question emerged that at what stage should we use the activation function and why we need it at all. I know that by definition the activation function transforms the sum of w*x+b to a number between some lower and upper limit.

In the video Risto Siilasmaa explains that in the training process there are the following steps:

Start with random weights.
Calculate the outcome (sum of w*x+b) - we know what it should be because we know what image we gave to the system.
Calculate the error.
Nudge all weights to reduce the error.

But what about the activation? Where to place it in the previous list? Before the error calculation?
And what would happen if we omitted it altogether? Just calculate the outcome and error and nudge the weights? Is it because the error calculation doesn't work well when the value of outcome isn't transformed between some lower and upper limit?

Generally the activation is part of the model and gets applied for each neuron, so definitely before the error calculation. What the activation function is depends on what task you are solving and where the neuron of interest is. In principle the activation function $f$ would go to the calculation of the outcome

$$ y = f(Wx + b)$$

For output neurons, if you are doing classification, then $f$ should map between 0 and 1, since you'll interpret the outcome as a probability. For regression $f$ could be just the identity.

For hidden (i.e. non-output neurons), you definitely want to use a non-linear $f$. The reason is that the neural network would otherwise be equivalent to a regular linear model. So the non-linear activations are needed to harvest the expressive power of neural networks.

For deep learning the most popular $f$ for hidden neurons would probably be the rectified linear unit (relu)

$$ f(x) = \max(0,x)$$


How to compute precision and accuracy of a sequence that is not strictly binary? Given a predicted sequence and actual sequence I want to compute it's precision and accuracy, for example: Note that these sequences will only contain 0, 1 or -1

predicted sequence: -1,0,1,1,-1,0,1,1,0,-1

actual sequence:     -1,1,0,1,-1,1,0,1,0,-1

I know that precision is computed using this tp/tp+fp and accuracy is computed using tp + tn /tp + tn + fp + fn. But because I have -1 in it I am unsure how I would compute true positives? My understanding that a true positive is if I predicted a 1 and it's corresponding actual value is a 1. A walk through of the computation for precision and accuracy would help.

Welcome to the Site!

We know that this problem is Multi-Class Classification Problem.

To get a confusion matrix for the same you can use the following command:

from mlxtend.evaluate import confusion_matrix

#import the required packages
from mlxtend.evaluate import confusion_matrix
from mlxtend.evaluate import plot_confusion_matrix

#Actual Target Values
y_target =    [-1,1,0,1,-1,1,0,1,0,-1]
#Predicted Values
y_predicted = [-1,0,1,1,-1,0,1,1,0,-1]

#creation of confusion matrix
cm = confusion_matrix(y_target=y_target,
                      y_predicted=y_predicted,
                      binary=False)
#to print the calculated values  of Confusion Matrix
cm

Outcome:

array([[3, 0, 0],
   [0, 1, 2],
   [0, 2, 2]])

For visualizing the cm you can use the following command:

fig, ax = plot_confusion_matrix(conf_mat=cm)
plt.show()

You can go through this Link for better understanding of mlextend.

You can get the Precision and Accuracy values by using the following formulas:

$\text{Precision}_{~i} = \cfrac{M_{ii}}{\sum_j M_{kji}}$

$\text{Recall}_{~i} = \cfrac{M_{ii}}{\sum_j M_{ijk}}$

Go through these Link-1,Link-2 for better understanding on how to compute the same, in the Link-3 is GitHub link which explains on how they implemented for a 1-D array, looking at that you can try expanding it for your outcome.


Anybody know what this type of visualisation is called? I think this is a pretty cool way to visualise changes in values but I can’t find any name for this type of visualisation.
I
Source: https://www.economist.com/graphic-detail/2020/07/28/americans-are-getting-more-nervous-about-what-they-say-in-public

This type of visualization is called Arrow and Range plot.
You can refer here


Parameterization regression of rotation angle Let's say I have a top-down picture of an arrow, and I want to predict the angle this arrow makes. This would be between $0$ and $360$ degrees, or between $0$ and $2\pi$. The problem is that this target is circular, $0$ and $360$ degrees are exactly the same which is an invariance I would like to incorporate in my target, which should help generalization significantly (this is my assumption). The problem is that I don't see a clean way of solving this, are there any papers that try to tackle this problem (or similar ones)? I do have some ideas with their potential downsides:

Use a sigmoid or tanh activation, scale it to the ($0, 2\pi)$ range and incorporate the circular property in the loss function. I think this will fail fairly hard, because if it's on the border (worst prediction) only a tiny bit of noise will push the weights to go one way or the other. Also, values closer to the border of $0$ and $2\pi$ will be more difficult to reach because the absolute pre-activation value will need to be close to infinite.
Regress to two values, a $x$ and $y$ value and calculate the loss based on the angle these two values make. I think this one has more potential but the norm of this vector is unbounded, which could lead to numeric instability and could lead to blow ups or going to 0 during training. This could potentially be solved by using some weird regularizer to prevent this norm from going too far away from 1.

Other options would be doing something with sine and cosine functions but I feel like the fact that multiple pre-activations map to the same output will also make optimization and generalizations very difficult.

The second way, predicting $x=cos(\alpha)$ and $y=sin(\alpha)$ is totally okay.

Yes, the norm of the predicted $(x, y)$ vector is not guaranteed to be near $1$. But it is not likely to blow up, especially if you use sigmoid activation functions (which are bounded by they nature) and/or regularize your model well. Why should your model predict a large value, if all the training samples were in $[-1, 1]$?

Another side is vector $(x,y)$ too close to $(0,0)$. This may sometimes happen, and could indeed result in predicting wrong angles. But it may be seen as a benefit of your model - you can consider norm of $(x,y)$ as a measure of confidence of your model. Indeed, a norm close to 0 means that your model is not sure where the right direction is.

Here is a small example in Python which shows that it is better to predict sin and cos, that to predict the angle directly:

# predicting the angle (in radians)
import numpy as np
from sklearn.neural_network import MLPRegressor
from sklearn.model_selection import cross_val_predict
from sklearn.metrics import r2_score
# generate toy data
np.random.seed(1)
X = np.random.normal(size=(100, 2))
y = np.arctan2(np.dot(X, [1,2]), np.dot(X, [3,0.4]))
# simple prediction
model = MLPRegressor(random_state=42, activation='tanh', max_iter=10000)
y_simple_pred = cross_val_predict(model, X, y)
# transformed prediction
joint = cross_val_predict(model, X, np.column_stack([np.sin(y), np.cos(y)]))
y_trig_pred = np.arctan2(joint[:,0], joint[:,1])
# compare
def align(y_true, y_pred):
    """ Add or remove 2*pi to predicted angle to minimize difference from GT"""
    y_pred = y_pred.copy()
    y_pred[y_true-y_pred >  np.pi] += np.pi*2
    y_pred[y_true-y_pred < -np.pi] -= np.pi*2
    return y_pred
print(r2_score(y, align(y, y_simple_pred))) # R^2 about 0.57
print(r2_score(y, align(y, y_trig_pred)))   # R^2 about 0.99

You can go on and plot the predictions, to see that predictions of the sine-cosine model are nearly correct, although may need some further calibration:

import matplotlib.pyplot as plt
plt.figure(figsize=(12, 3))
plt.subplot(1,4,1)
plt.scatter(X[:,0], X[:,1], c=y)
plt.title('Data (y=color)'); plt.xlabel('x1'); plt.ylabel('x2')
plt.subplot(1,4,2)
plt.scatter(y_simple_pred, y)
plt.title('Direct model'); plt.xlabel('prediction'); plt.ylabel('actual')
plt.subplot(1,4,3)
plt.scatter(y_trig_pred, y)
plt.title('Sine-cosine model'); plt.xlabel('prediction'); plt.ylabel('actual')
plt.subplot(1,4,4)
plt.scatter(joint[:,0], joint[:,1], s=5)
plt.title('Predicted sin and cos'); plt.xlabel('cos'); plt.ylabel('sin')
plt.tight_layout();

Update. A navigation engineer noticed that such a model would be most accurate when the angle is close to $\frac{\pi N}{2}$. Indeed, near 0° and 180° the angle $\alpha$ is almost linear in $\cos(\alpha)$, and near 90° and 270° it is almost linear in $\sin(\alpha)$. Thus, it could be beneficial to add two more outputs, like $z=\sin(\alpha+\frac{\pi}{4})$ and $w=\cos(\alpha+\frac{\pi}{4})$, to make model almost-linear near 45° and 135° respectively. In this case, however, restoring the original angle is not so obvious.

The best solution may be to extract coordinates $(x,y)$ from both representations (in the second one, we need to rotate $(z,w)$ to get $(x,y)$), average them, and only then calculate arctan2.


Unbalanced discounted reward in reinforcement learning : is it a problem? Discounted rewards seems unbalanced to me.

If we take as example an episode with 4 actions, where each action receive a reward of +1 :

  +1 -> +1 -> +1 -> +1

The discounted reward for the last action is : 1
The discounted reward for the first action (considering gamma = 1 for simplicity) is : 4

Intuitively both action are as good as the other, because both received same reward.
But their total reward is different, unbalanced.

So when we will backpropagate, first action will be favored over last action ?

Most of the RL problems are sequential decision making processes. That is the action taken at time $t$, can influence the future rewards in a sequential way. From this perspective, the first action lead to a sequence of states/actions which produced positive rewards, which means it deserves more positive feedback than the last action. It only seems natural to prefer the first action in the first state unless we know some other action yields better cumulative rewards. Rewards are gains in short-term, an action might be good in the short term but not in the long run, which is why we use cumulative rewards called returns to estimate the value of an action in a given state.


Why is the memory of common GPUs so constrained? Often we find that the NN training process could be highly constrained by GPU memory size, like in object detection models the training batch size could be limited to 1 or 2, which has inflicted those working in such tasks for long and led to a number of tricks proposed sorely for the work-arounds on such limited memory/batch size scenarios. So why is the GPU memory limited to, e.g. 16GB on common P100/V100, compared to up to 256GB RAM for CPU on high performance workstations? What is the obstacle for Nvidia to release a, say, 128GB V100? And what would be the cost for that?

Basically, the reason is because the memory of the GPU is much more expensive. This is due to the fact that GPUs using a very high memory bandwidth. A normal GTX 480 has 384 bit wide bus, which allows a maximum transfer rate of ~180 GB/sec. However, an Intel i7 CPU using a 192 bit wide bus  has a maximum of about ~30 GB/sec depending of what type of RAM you use. This means that the transfer of data in the GPU is about 5x faster compared to a CPU. Therefore, memory for GPUs is much more expensive but you can get more transfer ratio.

Additionally increasing more chips, the subsequent increase in capacity could end up messing with the timing of transfer of data.


could not broadcast input array from shape (2,3) into shape (3) while using timestamp to build neural network in python Here I want predict value every 60 minutes. So I have data 540 with three inputs. so I wrote an code with time steps and it gave me this error. Can anyone help me to solve this problem?

my code :

y=data['y1'].astype(int)
cols=['x1', 'x2', 'x3']
x=data[cols].astype(int)
n = x.shape[0]
p = x.shape[1]
x  = x.values
y = y.values
train_start = 0
train_end = int(np.floor(0.8*n))
test_start = train_end+1
test_end = n
x_train = x[np.arange(train_start, train_end), :]
x_test = x[np.arange(test_start, test_end), :]
y_train = y[np.arange(train_start, train_end), :]
y_test = y[np.arange(test_start, test_end), :]
x_train=x_train.reshape(x_train.shape +(1,))
x_test=x_test.reshape(x_test.shape + (1,))

num_time_steps = 9
num_features = x.shape[1]

x_train = np.zeros((x_train.shape[0] - num_time_steps + 1, num_time_steps, num_features), dtype="float32")
x_test = np.zeros((x_test.shape[0] - num_time_steps + 1, num_time_steps, num_features), dtype="float32")

for i in range(len(x_train)):
for timestep in range(num_time_steps):
    x_train[i][timestep] = x_train[i + timestep]

for i in range(len(x_test)):
for timestep in range(num_time_steps):
    x_test[i][timestep] = x_test[i + timestep]

 y_train = y_train[num_time_steps - 1:]
 y_test = y_test[num_time_steps - 1:]

change code:

train_end = 80
x_train=x[0: train_end ,]
x_test=x[train_end +1: ,]
y_train=y[0: train_end]
y_test=y[train_end +1:]
x_train=x_train.reshape(x_train.shape +(1,))
x_test=x_test.reshape(x_test.shape + (1,))
num_time_steps = 9
num_features = x.shape[1]
x_train_n = np.zeros((x_train.shape[0] - num_time_steps + 1, num_time_steps, num_features), dtype="float32")
x_test_n = np.zeros((x_test.shape[0] - num_time_steps + 1, num_time_steps, num_features), dtype="float32")
for i in range(len(x_train_n)):
for timestep in range(num_time_steps):
    x_train_n[i][timestep] = x_train[i + timestep]
for i in range(len(x_test_n)):
for timestep in range(num_time_steps):
    x_test_n[i][timestep] = x_test[i+timestep]
y_train_n = y_train[num_time_steps - 1:]
y_test_n = y_test[num_time_steps - 1:]

Error,

First of all, you should use separate variables for the 3D and 2D x_trains:

x_train_n = np.zeros((x_train.shape[0] - num_time_steps + 1, num_time_steps, num_features), dtype="float32")
x_test_n = np.zeros((x_test.shape[0] - num_time_steps + 1, num_time_steps, num_features), dtype="float32")

And, your outer loop should span the new x_train, i.e. x_train_n:

for i in range(len(x_train_n)):
   for timestep in range(num_time_steps):
       x_train_n[i][timestep] = x_train[i + timestep].squeeze()

Change the other loop (the one for the test case) as well.


How do we get the coefficients and intercept in Logistic Regression? I'm using Codecademy to learn about logistic regression and there are some holes in my understanding of this topic.
import numpy as np

hours_studied = np.array([[ 0],[ 1],[ 2],[ 3],[ 4],[ 5],[ 6],[ 7],[ 8],[ 9],[10],[11],[12],[13],[14],[15],[16],[17],[18],[19]])
calculated_coefficients = np.array([[0.20678491]])
intercept = np.array([-1.76125712])

def log_odds(features, coefficients,intercept):
  return np.dot(features,coefficients) + intercept
# z = b0 + b1x1 + b2x2 + ....
# log odds measures how likely it is that the data sample belongs to the positive class

def sigmoid(z):
    denominator = 1 + np.exp(-z)
    return 1/denominator
## In order to map predicted values to probabilities, we use the sigmoid function.
# The function maps any real value into another value between 0 and 1.
# In machine learning, we use sigmoid to map predictions to probabilities

# Create predict_class() function here
# features = matrix
# coefficients = vector
def predict_class(features, coefficients, intercept, threshold):
  calculated_log_odds = log_odds(hours_studied, calculated_coefficients, intercept)
  probabilities = sigmoid(calculated_log_odds)
  return np.where(probabilities >= threshold, 1, 0)
# If a value in array_to_check is above threshold, the output is 1. If a value in array_to_check is below threshold, the output is 0.

# Make final classifications on Codecademy University data here
final_results = predict_class(hours_studied, calculated_coefficients, intercept, .5)
print(final_results)

hours_studied, calculated_coefficients and intercept were given by codecademy.
I'm not sure how to get these inputs myself to plug into the model and codecademy didn't explain either. Hours_studied I understand.
Maybe could someone help explain what are the possible steps before I define log_odds, sigmoid, etc to get coefficients and intercept?

Maybe this part of the code is trying to teach you activities post ML training i.e. prediction
The coefficient and intercept are the parameters of the Model. These are determined by using Training data (Features and Labels) and training process
You follow these steps(Very high level) -
 Get data - X , Y
 Define model i.e. Logistics Regression
 Train Model using the data - Here you get the Coef/Intercept
 Predict using the Model
In your work, you are at the last step.
 You have a model ready with the parameters and you also got the data i.e. Features(X)
 It is as good as calling a function i.e. get_answer(x) and get_answer is your model with it's parameters.
Then you say
if answer > THRESOLD(normally 0.5 but we get the best by trials) it's Class-A else it's Class-B
You may learn Logistics regression from scratch form Blog-machinelearningmastery


SUMMARIST: Automated Text Summarization There is a text summarization project called SUMMARIST. Apparently it is able to perform abstractive text summarization. I want to give it a try but unfortunately the demo links on the website do not work. Does anybody have any information regarding this? How can I test this tool?

http://www.isi.edu/natural-language/projects/SUMMARIST.html

Regards,
PasMod

It dates back to 1998, so most likely has been abandoned, or "acquired" by Microsoft as the creator currently works there and has done since publishing that research.

See https://www.microsoft.com/en-us/research/wp-content/uploads/2016/07/ists97.pdf
and http://research.microsoft.com/en-us/people/cyl for the author. Maybe you could try to contact him.


How do linear learning systems classify datapoints that fall on the hyperplane How do linear learning systems, such as the simple "closest to the class average" algorithm or SVMs, classify datapoints that fall on the hyperplane?

Linear, binary classifiers can choose either class (but consistently) when the datapoint which is to classify is on the hyperplane. It just depends on how you programmed it.

Also, it doesn't really matter. This is very unlikely to happen. In fact, if we had arbitrary precision computing and normal distributed features, there would be a probability of 0 (exactly, not rounded) that this would happen. We have IEEE 754 floats, so the probability is not 0, but still so small that there are much more important factors to worry about.


Choosing between Regression and Classification I want to build a model using a neural network that will be able to extract some features from landscape pictures.

In order to improve the efficiency of my model, I first want to extract the "vertical orientation" of the picture. This "vertical orientation" would be next a feature for my neural network.

Now, to compute this feature I see two solutions.

Build a Regression model that would return a result in degrees (0 to 180)
Build a Classification model that would return the class of orientation (ex: High, Medium High, Medium, Medium Low, Low)

Is there a way to decide which solution to use, or should I test both solutions to find the better one ?

Note :

The dataset is not a problem. I can have pictures labeled in degrees or in class easily

If you are trying to build a neural network you should stick with a the orientation as a numeric value since neural networks only accept numeric data as input.  If you want to use the class values, you will need to one hot encode them.  It will also result in a loss of information.

Of course, it depends what kind of neural network you are trying to implement. In most cases, I don't think this step is necessary and I think it may complicate the architecture of your model.

If you are open to suggestions, instead of building a regression model, I would investigate Principal Component Analysis.  It could alleviate the need to have the photographs labeled before input into the neural network.


What is an example of normalization (making the row unit norm)? I am learning about standardization and normalization concepts for feature engineering.
Standardization is done for example using z-score where based on the mean and std deviation we re-calculate the values so that the mean is 0 and std deviation is 1. This is a column wise operation.
Where as Normalization is performed row wise (to make the entire row unit norm) – thus making it good for calculating things like cosine similarity. I am looking for an example of what it means to make the row unit norm.

A very simple example is the use of relative frequency instead of absolute frequency, for instance in text classification: if the features represent the raw absolute frequency of every word, then large documents have higher values than small documents. In this case the learning algorithm won't be able to distinguish between a word used 10 times in a small 50 words document (so this word is important for the document) vs. the same word used 10 times but in a large 1000 words document (not very important for the document).
Using relative frequency instead (i.e. dividing by the size of the document) solves this problem. It's an example of row-wise normalization, there are many others.


SPARK RDD - Clustering - K-Means Imagine that I've this dataset (just a sample)

A   B   C
1   23  1000
2   52  5000
3   12  500
4   10  450

I'm trying to assign each row to a clustering based on C value. Like this:

A   B   C   CLUSTER
1   23  1000    2
2   52  5000    1
3   12  500     3
4   10  450     3

For that I'm using K-Means algorithm using Spark:

import org.apache.spark.mllib.clustering.{KMeans, KMeansModel}
import org.apache.spark.mllib.linalg.Vectors

val data = sc.textFile("/user/cloudera/TESTE1")
val parsedData = data.map(s => Vectors.dense(s.split(',').map(_.toDouble))).cache()

val numClusters = 4
val numIterations = 20
val clusters = KMeans.train(parsedData, numClusters, numIterations)

val WSSSE = clusters.computeCost(parsedData)
println("Within Set Sum of Squared Errors = " + WSSSE)

clusters.save(sc, "/user/cloudera/KMeansModel")
val sameModel = KMeansModel.load(sc, "/user/cloudera/KMeansModel")

But this script extracts me a .gz.parquet file and when I try to see what type of information this file contains, using:

sqlContext.read.parquet("/user/cloudera/KMeansModel/data/part-r-00000-f551ea29-54db-45be-8cba-d06a97d6d9f2.gz.parquet").show

I'm getting this:

+---+--------------------+
| id|               point|
+---+--------------------+
|  0|[9.39601519885208...|
|  1|[9.80112351958380...|
|  2|[9.63822872186722...|
|  3|[9.44194658832542...|
+---+--------------------+

How can I get the table that I put above? Basically I just want to extract the same fields and add a column with the cluster calculated by K-Mwans to each row...

Many thanks!

This is well answered in this earlier question:

https://stackoverflow.com/q/31447141/1060350

Beware that Spark k-means is slow.

If your data fits into main memory (i.e. a few gigabyte, which means billions of vectors!) then other tools such as ELKI that don't have the cluster overhead will be much faster. Use spark only for preprocessing the data, if you e.g. have several TB of jsons, and you need to first extract the numbers out of the JSONs, then here is where Spark shines.

Once your data is then vectors, use ELKI instead, it's much faster.


Using deconvolution in practice Suppose that I have an input image with 224 * 224 * 3 dimensions. I pass it through a convolution layer with 64 filters and same convolution operation. Suppose that I want to reconstruct the original image using the output of the convolution layer which has the size 224 * 224 * 64. I have the following questions:

Should I use deconvolution? If so, how is the arrangement of deconvolution layer (number of filters and the value of weights. Also when should the activation be applied)?
Are the number of filters and weights in forward pass equal to the backward pass?
Is there any technique other than deconvolution?
Is there any available Keras code for my need?

I've seen here and also here but they represent a high level abstraction and don't contain appropriate detailed answer.

Since what you are describing sounds like some sort of auto-encoder, you can use this blog on how to use convolution layers in Keras.

Most of the logic is straightforward: If you wish to reconstruct your input, you need to your output to have the same spatial dimensions and the same number of channels as your input. As in most neural network architectures, you will gradually decrease the spatial dimensions of your input and increase the number of filters, and if you wish to reconstruct your input you will use the same concept of layers, but in reverse order, using deconvolution layers.

As to whether you need to use deconvolution or what is the optimal number of channels, layers and symmetry is entirely up to what you are trying to accomplish and the specific details of your problem.


Can VAEs be used to generate multivariate data? Most of the tutorials online seem to use VAEs to generate images and use CNNs to generate data. I am working on a game with multivariate data consisting of character position and the character attributes to train a VAE.

I was wondering if it was possible because I have tried training the network with pytorch but I continue to get poor results with KL Divergence term decreasing to 0.

Any help would be really appreciated.

The CTGAN paper (github) use a VAE to generate multivariate synthetic data from tabular data. See section 4.5.


What are the latest and most succesful classification algorithms at the moment? I have googled this for some time with no luck. All i get are tutorials or articles explaining the classic algorithms like linear regression, random forest, etc.

I would like to know which are the most succesful recent algorithms at this moment for classification with an implementation in Python. I'm interested specially on ensemble methods, such like XG Boost, Catboost and Light Gradient Boosting Machines. Are there any other algorithms similar to these 3 mentioned above with proven sucess capabilities? I would appreciate any help from experienced people that have tried any.

I know there is not such a thing like the best algorithm for anything, what i want to know is if there are any other algorithms that perform the same or better than the ones mentioned above in general. The goal is to solve classification problems using tabular data and supervised learning.

Thanks in advance

Deep Learning is your friend. Neural Nets with the growth of publicly available data, and also privately, have beaten every other architecture with the sufficient hardware for it to run properly on.Here's a depiction of why NN's are so popular


Calculating correlation between two time variables I have a dataset that looks like this :

    UserID  App_open_Time(Hour_ofDay)   Email_Open_Time(hour_ofDay)
       1        1                         1
       2        1                         1
       3        1                         1
       4        1                         1
       5        1                         1
       6        2                         1
       7        2                         1
       8        3                         1
       9        3                         1
      10        3                         1

I would like to know if the App_open_time is correlated with Email open time .
How can i do this analysis in python.
I was planning to do a pearson-correlation in Python using Numpy ,Is this the best approach?

I don't think you can use Pearson's correlation because it is used for continuous variables. Your variables are ordinal, so a test like Spearman's would be more appropriate. However I don't think that tests for ordinal variables would be appropriate either, because your variables are also cyclical, in the sense that Hour_ofDay=23 and Hour_ofDay=1 are really just 2 hours apart, but for the Spearman's test they would be considered 22 hours apart.

I think in this case it would be more appropriate to look at the distribution of the distance (measured in hours) between the two variables.
The appropriate distance metric in this case is defined as follows (distance originally defined in the accepted answer to this other question)

import numpy as np
distance = np.sign(a1-a2)*(12 - abs(abs(a1 - a2) - 12))

Where a1 and a2 are your App and Email opening time variables. Note that the variables need to be in the range 0 to 23 for this distance to work.

Compute this distance for each row, add it as a column to your dataframe and plot it with an histogram.
This histogram will tell you quite a lot about the "correlation" between the two variables.

For example

No correlation: the histogram will be uniform between -12 and 12
Instantaneous correlation, i.e. user opens email and app at the same time: the histogram will have a peak at 0
Anticipated delay, i.e. user opens email 1 hour before app: the histogram will have a peak at 1
Delayed correlation, i.e. user opens email 1 hour after app: the histogram will have a peak at -1

This visualization will allow you to draw rich conclusions about the relation between App- and Email opening times.

Note: if your variables also include minutes and seconds in a date format you have to convert the variables to numerical. E.g. 01:30 (hour 1 and 30 minutes) becomes 1.5 . Also pay attention to the date format in case you have time expressed in the 12-hour clock (e.g. 6 PM, 1 AM)


Is there a real life meaning about KMeans error? I am trying to understand the meaning of error in sklearn KMeans.

In the context of house pricing prediction, the error linear regression could be considered as the money difference per square foot.

Is there a real life meaning about KMeans error?

The K-means Error gives you, what is known as, total intra-cluster variance.

Intra-cluster variance is the measure of how much are the points in a given cluster spread.

The following cluster will have high intra-cluster variance

In the image below, even though the number of points are same as that of the image above, the points are densely distributed and hence will have lower intra-cluster variance.

K-means Error is interested in the total of such individual cluster variances.

Suppose for a given data, if clustering 'A' forms clusters like the first image and clustering 'B' forms clusters like the second image, you will in most cases choose the second one.

Although this does not mean that the K-means Error is a perfect objective to optimize on to form clusters. But it pretty much catches the essence behind clustering.

Code used for cluster plot generation -

import numpy as np
from matplotlib import pyplot as plt

sparse_samples = np.random.multivariate_normal([0, 0], [[50000, 0], [0, 50000]], size=(1000))
plt.plot(sparse_samples[:, 0], sparse_samples[:, 1], 'b+')
axes = plt.gca()
axes.set_xlim(-1000, 1000)
axes.set_ylim(-1000, 1000)
plt.show()

dense_samples = np.random.multivariate_normal([0, 0], [[5000, 0], [0, 5000]], size=(1000))
plt.plot(dense_samples[:, 0], dense_samples[:, 1], 'r+')
axes = plt.gca()
axes.set_xlim(-1000, 1000)
axes.set_ylim(-1000, 1000)
plt.show()

In both cases, a 1000 datapoints from a Bivariate Normal Distribution are  sampled and plotted . In the second case, the Covariance Matrix is changed to plot a denser cluster. np.random.multivariate_normal's documentation can be found here. Hope this helps!


Deep Learning with Keras. Classification problem with large data I have looked on the internet and found a lot of discussion about image classification problems, but almost none of just classification problems without images. I am trying to build a model with Keras using Python, I have a dataset of 827 features (43 are integers, and the rest is one-hot encoded) and 24,800 samples. My target is 3 classes one-hot encoded as well ( [1 0 0] [0 1 0] [0 0 1] ).

I have found a lot of complex image recognition networks but only simple deep networks for non-image problems. I need help on how I can make the deep neural network architecture for my problem.

There are actually a lot of examples out there which show how to apply Keras to different types of data. A really good source is F. Chollet's book "Deep Learning with Python". The code is online. You could start with the tutorial on predicting house prices. It is a regression problem, but you can easily change it to a classification problem.

A general remark: Neural nets (such as Keras) usually work well with very complex data (images, sound, text). For "normal" data, other approaches are often better suited (e.g. boosted trees). However, with the large number of features you have, NN is worth a try.


Extract Domain related words I am doing a research regarding on automatic text summarizing. So in order to weighting sentences I need to get words related to a particular field or domain like shown below.

Topic word - Car
Related words - engine, driver, road, break, accelerator

Is there any direct method that I can use like wordnet synsets.

One very obvious way is to use gensim's word2vec most_similar() function to get the most related words to the queried word.

You can check an online version of how it works out here: https://projector.tensorflow.org/

Again it depends on what kind of relations you seek. Other useful methods include:

from nltk.corpus import wordnet as wn

wn.synsets('car')
wn.synsets('car')[0].hyponyms()


Decision Trees Should We Discard Low Importance Features? I just started to work with feature selection. Let's say I have a decision tree model. I get its feature importances by tree.feature_importances_.

In my model out of around 30 features, 20 of them has importance value of zero. Does that mean that I should discard those low importance value features from my model? As I understood the answer is no, but I don't know the reason behind it. Can anyone explain?

As for many questions, the answer is "it depends":

features which have a low individual importance may still add predictive power to your model, because the model benefits from combining their information together with information of other features.
However they may introduce noise in the model and cause overfitting, thus decreasing the performance of the model.

The best way to answer your question is to experiment:

order the features by decreasing importance
loop from say 5 features to 30, each time selecting the top N features by importance, and training/testing a model based on this subset of features.
plot the performance

You're likely to observe that the performance increases quite a lot at the start for each "important" feature added, then slows down as feature importance decreases and probably doesn't increase at all at some point, possibly even decreasing a bit.


Advice on how to get started for someone a bit later in life? I am 33 years old and graduated in 2006 from college with a degree in analytic philosophy. For a long time I thought that I was going to get a phd in phil and teach but that dream is long gone, and I am intent on having a career in data science/analysis.

I do have a history in mathematics; I finished calculus 3 in high school and finished a set theory course in college as well as advanced logic courses (symbolic, modal, predicate, etc), but I have never studied how to code, nor have I taken any statistics courses.

Are there any suggestions on how to best proceed?  I of course want to be as attractive to employers as possible.  I have no debt or family and am open and willing to take multiple years (and take on loans) studying for this profession.

Would getting a math degree (with an emphasis in statistics and taking courses like linear algebra) be the right (or at least, good) way to go?  A local university offers a concentration in data science within the mathematics department, so I could do that as well. And in addition, taking courses in computer science to learn how to code?

Basically I'm looking to push a reset button (well not entirely since I have a background in mathematics, and I believe my background in philosophy will help too) on my career path and will do whatever it takes.  I want a career with a future and with hopefully opportunity to choose where I work (city), and a career that fits with how my mind works (I believe I could thrive in this field).

Jobs that I have had since college will not help in any way towards future jobs in this field, so I will be fully leaning on future classes/preparation.

Thanks in advance and for reading this partial ramble!  I'm motivated but don't know exactly what I should do.  I realize that there are many paths people take to get into this profession but would love advice for someone like me in my situation.

I for myself wouldn't advise you to not take that route. I do agree with most of the points that were mentioned before but my conclusion is a slightly different one:

Trying to get into data science without any preceding field knowledge deserves respect and it is not impossible. Also, there is still a lack of qualified data scientist and I would predict that the demand is growing.

But you should consider the following points:

Data scientist is probably the job in the statistics/IT area that requires the highest level of education. Ask yourself if this is really what your aiming at.
If you choose to go on that route be aware that it will be extremely hard. Data science is so multifaceted and interdisciplinary that there is a lot to learn. Even if you'd have a couple of hours to study each day I'm afraid it would take years. Being able to work as a data scientist needs far more than an academical math knowledge.
You will have to compete with PhDs in their mid-twenties who already have years of experience in publishing and conducting machine-learning/math/programming projects. Employers will probably favor them over you.

If you don't care that it will be painful and want to start learning, I'd do the following:

Get basic knowledge of statistics, programming, and IT-infrastructure. You should be familiar with t-tests, ANOVAs, regressions and cluster-/classification techniques. You should start learning a statistical programming language. For starters I'd recommend R or Python. Basic knowledge of another higher programming language is useful. Knowing basic programming principles such as object-oriented programming doesn't hurt either. I'd recommend to take a look at Java since a lot of big data tools such as Hive, Hadoop or Spark are Java-based frameworks. Look at these frameworks, too. Understand what they are used for. At last you should have at least basic knowledge of databases and know the difference between a relational and a non-relational database.
Don't spend money on courses. The internet is full of free sources. Just take into account who offers them. There are a lot of lecturers and professors from very good universities who provide online courses or share their lectures.
Read books. Again: Don't spend money. Search the internet for free high-quality materials (see here for example).
Check out community sites such as Kaggle and join competitions. This is probably as close as you will get to real-job tasks.
Continue doing that for a couple of month and ask yourself again if this is really where you want to go. If yes, continue.

But realistically, here is what I would try to do:

Don't aim at becoming a data scientist right away.
Consider to start in a statistics-related job (e.g. controller, business analyst, marketing manager, game designer etc.) and work your way up. I strongly believe that such a strategy is more likely to be successful.

Because no matter what you do before starting in the industry, you will have to learn on the job.


For imbalanced classification, should the validation dataset be balanced? I am building a binary classification model for imbalanced data (e.g., 90% Pos class vs 10% Neg Class).

I already balanced my training dataset to reflect a a 50/50 class split, while my holdout (training dataset) was kept similar to the original data distribution (i.e., 90% vs 10%).
My question is regarding the validation data used during the CV hyperparameter process. During each iteration fold should:

1) Both the training and test folds be balanced

or

2) The training fold should be kept balanced while the validation fold should be made imbalanced to reflect the original data distribution and holdout dataset.

I am currently using the 1st option to tune my model; however, is this approach valid given that the holdout and validation datasets have different distributions?

Both test and validation datasets should have the same distribution. In such a case, the performance metrics on the validation dataset are a good approximation of the performance metrics on the test dataset. However, the training dataset can be different. Also, it is fine and sometimes helpful to balance the training dataset. On the other hand, balancing the test dataset could lead to a bias estimation from the performance of the model because the test dataset should reflect the original data imbalance.
As I mentioned at the beginning the test and validation datasets should have the same distribution. Since balancing the test dataset is not allowed, the validation dataset can not be validated too.

Additionally, I should mention that when you balance the test dataset, you will get a better performance in comparison to using an unbalanced dataset for testing. And of course, using a balanced test set does not make sense as explained above. So, the resulted performance is not reliable unless you use an unbalanced dataset with the same distribution of classes as the actual data.


What is the "novel reinforcement learning algorithm" in AlphaGo Zero? For some reason, AlphaGo Zero isn't getting as much publicity as the original AlphaGo, despite its incredible results. Starting from scratch, it's already beaten AlphaGo Master and has passed numerous other benchmarks. Even more incredibly, it's done this in 40 days. Google names it as "arguably the best Go player in the world".

DeepMind claims this is a "novel form of reinforcement learning" - is this technique truly novel? Or have there been other times when this technique was used- and if so, what were their results? I think the requirements I'm talking about are 1) no human intervention and 2) no historical play, but these are flexible.

This appears to be a similar question, but all the answers seem to start from the assumption that AlphaGo Zero is the first of its kind.

The AlphaGo Zero article from Nature, "Mastering the Game of Go without Human Knowledge", claims four major differences from the earlier version:

Self-learning only (not trained on human games)
Using only the board and stones as input (no hand-written features).
Using a single neural network for policies and values
A new tree-search algorithm that uses this combined policy/value network to guide where to search for good moves.

Points (1) and (2) are not new in Reinforcement learning, but improve on the previous AlphaGo software as stated in the comments to your question. It just means they are now using pure Reinforcement Learning starting from randomly initialized weights. This is enabled by better, faster learning algorithms.

Their claim here is "Our primary contribution is to demonstrate that superhuman performance can be achieved without human domain knowledge." (p. 22).

Points (3) and (4) are novel in the sense that their algorithm is simpler and more general than their previous approach. They also mention that is is an improvement on previous work by Guo et al.

Unifying the policy/value network (3) enables them to implement a more efficient variant of Monte-Carlo tree search to search for good moves and simultaneous using the search tree to train the network faster (4). This is very powerful.

Furthermore, they describe a number of interesting implementation details like batching and reusing data-structures to optimize the search for new moves.

The effect is that it needs less computing power, running on 4 TPUs rather than 176 GPUs and 48 TPUs for previous versions of their software.

This definitely makes it "novel" in the context of Go software. I believe that (3) and (4) are also "novel" in a broader context and will be applicable in other Reinforcement Learning domains such as e.g. robotics.


How to see actual numbers in a hist on R? Just as an example I'm using iris data. I have a histogram, which is nice, but the distribution is weird, and I'd like to see what the bars represent:

Here's a histogram:

And I'd like to see the high/low that each bar represents. For instance, the first bar has a frequency of 34, but a width of something in the range of the lowest third of 0.0-0.5.

Obviously there won't be a width of 0.0, so what is the width range? How can I show it on the histogram?

Hope this may help you

hist(iris$Petal.Width,labels=T)


Model comparison with CV using standard error Discovering the ML world with sklearn, I'm testing a large panel of models onto my dataset. This is for learning purpose but also for work so I want the final model to be as accurate as possible, while I can progress in my understanding of ML.

I've separated my dataset (16k rows) into 80% training and 20% testing, and I'm testing at least KNN, Logistic, DecisionTree, RandomForest, NaivesBayes and maybe SVC (if my computer can handle it), with some Bagging and Boosting when I find out how to.

My training sample comes in 4 ways: 2 sets of features (95 and 11), standartised (with a StandardScaler) or not.

My outcome is binary and I'm using a custom scorer "amelioration" which maximalise the number of positives for the 30th percentile (easier to get it with the code at the end of my post), along with specificity and roc_auc.

For each dataset, I cross validate (stratified and shuffled with a random state, 5 folds, with repetition when possible) all hyperparameters I find relevant and repeat this for each training sample.

For each crossvalidation, I refit using my scorer so I can compare the results amongst models and datasets. For models I've tested so far, my mean_test_amelioration range from +42% to +114%.

Finally, I'll measure the performances of the selected model on the testing sample and report results.

I guess this can feel quite cumbersome to a pro (and I'm probably building a tank to kill a fly), but I've already learned so much in this manner.

I'm only comparing all these models on the mean_test_amelioration and do not take into account standard error (for instance). Could this lead to overfitting so that my final model won't generalize well? If yes, how could I take variability into account ?

Any educationnal link is also very welcome.

PS: As this could be relevant, here is my custom scorer code:

def get_amelioration(y_true, y_pred, **kwargs):
    """
       If I select 30% of my sample with this algorithm, I will have
       `amelioration`% more positives in my selection than without

       :use as: make_scorer(get_amelioration, needs_proba=True, N=30)
    """

    N = kwargs.pop('N', False)
    if kwargs: raise TypeError('Unexpected **kwargs: %r' % kwargs)

    decisions = (y_pred > np.percentile(y_pred, 100-N)).astype(int)

    tn, fp, fn, tp = metrics.confusion_matrix(y_true, decisions).ravel()
    v = (fp+tp)/(tn+fp+fn+tp)
    r = tp/(fp+tp)
    r_base = np.mean(y_true) #around 15% in my sample, expected to be stable
    amelioration = 100*(r/r_base-1)
#    print("N=%i, v=%0.3f, amelioration=%0.3f" %(N,v, amelioration))
    if v<0.75*N/100: return 0
    return amelioration

It is possible, however, this is usually not of great concern. If you would like to take into account the standard error, one possibility is the following:

You compute the standard error of the best performing model (using your amelioration metric)
Consider all models that perform within one standard error (or less if you prefer)
Select the model with the simplest decision boundary

This is a heuristic(!) based on Occam's razor that simpler models tend to generalize better. However, keep in mind that for complicated problems simpler models are probably just that: simpler. They don't have to be more accurate.

Alternatively, instead of 3. you can select the model with the smallest standard error, if your main goal is to limit your exposure to this kind of variability.

Also keep in mind that there is No Unbiased Estimator of the Variance of K-Fold Cross-Validation, so this method brings with it its own uncertainty.

In general, I would just keep an eye on the standard error and if it's not too concerning just go with the numerically best-performing model.


What is the 1 Unit in the contraint of SVM: $y_i(wx_i+b) \geq1$ I am following this note on SVM.

The constraint, $y_i(wx_i+b) \geq 1$, basically said all inputs, $x_i$, lie at least 1 unit away from the hyperplane on the correct side.

What does it mean by 1 unit? In 2D space, does it mean moving the hyperplane, $w$, up by 1?

It is a unit of distance, I would usually assume euclidean distance.

In more detail:

The data point $x_i$ is projected onto the vector $w$, which defines the orientation of the discriminating linear hyperplane as it is orthogonal to $w$. Where the discriminating hyperplane is "fixed" along the orientation of $w$ is decided by the bias Term $b$.

So for classifying a positive data point $x_{+}$ we would want $wx_{+} + b$ to a positive number.

For a negative $x_{-}$ datapoint $wx_{-}+b$ should be negative.

We can combine these two disired outcomes into a single constraint by using the labels, so we want $y_{+}(wx_{+}+b) \geq 0$ and $y_{-}(wx_{-}+b) \geq 0$.

By replacing the $0$ with a positive value $d \geq 0$, we can alter the constraint to encode our desire that the points should be $d$ far away from from the discriminating hyperplane, giving the constraint is $y_i(wx_i +b) \geq d$.

What kind of unit of distance depends on the vector space. Usually orthogonal projections in euclidean vector spaces are used, in that case it would be euclidean distance.


Converting feature coefficients to probabilities scores for linear models I have been looking into interpreting feature coefficient in a bounded capacity (0,1) as probability scores. So for example, if one has the following linear model for classification(Logistic regression),
then inverse logit should return one linear probability scores right as it is the inverse logs of odd ratio?

y<- c(1:3, 7, 6)
x1<- 1:5
x2<- c(20:25)
temp<-coef(lm(y ~ x1 + x2))
library('arm')
invlogit(temp[1]+temp[2])

However, in-case of linear regression for continuous value prediction,

coeff1/sum(coeff)

should return one probability scores right ?
Did I miss something ?

No, coeff1/sum(coeff) cannot be a probability due to the fact that it can be bigger than 1. For instance:
coeff1 = 1., coeff2 = -.5, then coeff1/sum(coeff) = 2. The interpretation of coefficients in linear regression is the derivative of the output variable with respect to the given predictor.


Training stateful LSTM with different number of sequences I'm using a stateful LSTM for stock market analysis, and I have varying amounts of data for each stock, ranging from 20 years to just a few weeks (i.e. for newly listed stocks).

I use 3 years of data as a minimum for training as I want to create some state within the network. I set a year as my sequence length, so if I have 12 years of data then I will submit 4 batches with 3 sequences in each. Only after I've submitted all batches do I then reset the network state ready for the next stock.

But is there any issue training with differing number of sequences? E.g. if I train with a company that has 20 years of data then the network will build up much more state than a company that I only have 3 years of data.

In the circumstance of stock market prediction, I think that when the sequence length reaches a certain point, the network will learn to forget the opening price 6 months ago or the volume 3 years back. That data is no longer relevant to the cell state, as recent events are more indicative of stock price changes.In terms of the LSTM method of storing both long and short-term memory, the LSTM will have the functionality to "forget" old and irrelevant data from earlier in the sequence, maintaining a stable and efficient cell state regardless of sequence length (of course, the sequence should be long enough for the LSTM to build up any cell state at all).
Hope this helps.


Layer normalization details in GPT-2 I've read that GPT-2 and other transformers use layer normalization before the self-attention and feedforward blocks, but I am still unsure exactly how the normalization works.

Let's say that our context size is 1024 tokens, the embedding size is 768 (so that each token and its subsequent hidden states are represented by vectors of size 768), and we are using 12 multi-attention heads. So in the diagram above, there are 1024 r's and each r has dimensionality 768.
For a given layer in the transformer, how many normalization statistics (sample mean and stdev) are computed? Do we do one normalization per token, for 12x1024 normalizations so that the feature values within each token have mean 0 and std 1? Or do we normalize the values for each feature across tokens, for 12x768 normalizations? Or do we normalize all the feature values for all the tokens together, for 12 normalizations? Do we compute separate normalizations for each context in the minibatch?
I'm also keen to understand intuitively why this normalization is desirable. Assuming that the scheme is to normalize the feature values within each token: let's say one of our tokens is a bland word like "ok" whereas another token is the word "hatred". I would expect that the representation of "hatred" would be spikier, with higher variance among the different feature values. Why is it useful to throw away this information and force the representation for "ok" to be just as spiky? On the other hand, if the normalization scheme is to normalize across feature values, so that if you take feature 1 from all of the tokens in our context, they will have zero mean and stdev 1, doesn't this throw away information when all of the words in our context are very negative, for example in the context "war violence hate fear"?
Separately, with layer normalization it seems like it is optional to re-scale the normalized values through learned bias and gain parameters. Does GPT-2 do this, or does it keep the values normalized to mean 0 and std 1?

The most standard implementation uses PyTorch's LayerNorm which applies Layer Normalization over a mini-batch of inputs. The mean and standard-deviation are calculated separately over the last certain number dimensions which have to be of the shape specified by normalized_shape argument. Most often normalized_shape is the token embedding size.
The paper "On Layer Normalization in the Transformer Architecture" goes into great detail about the topic. The paper proposes "the layer normalization plays a crucial role in controlling the gradient scales." Better behaved gradients help with training.


does xgb multi-class require one-hot encoding? I was trying an xgboost from python with a multiclass single-label problem and assumed the label can be an integer indicating my class (as opposed to eg one-hot) .

params = {'eta': 0.1,
#          'objective': 'binary:logistic',
          'objective': 'multi:softmax',
          'scale_pos_weight':9,
           'eval_metric': 'auc',
           'nthread':25,
         'num_class':6}

dtrain = xgb.DMatrix(df_train_x,label= df_train_y)
dvalid = xgb.DMatrix(df_val_x,label= df_val_y)
watchlist = [(dtrain, 'train'), (dvalid, 'valid')]
model = xgb.train(params, dtrain, 500,watchlist, maximize=True, verbose_eval=50,early_stopping_rounds=20)

However I hit an error

(1353150 vs. 225525) label size predict size not match

and I note that my sample size is 225525 , number of classes is 6 , and 6*225525 is 1353150 so it appears that xgb is looking for one-hot ... however when i use one hot I get an error hinting that one-hot can't be used -

dtrain = xgb.DMatrix(df_train_x,label= df_train_y)
ValueError: DataFrame for label cannot have multiple columns

!!!

As Majid stated in the comment, using AUC is causing this error as normally ROC curves are calculated for binary classification. Try removing the eval_metric line and your code will run properly. That or removing the watchlist and early stopping options.

The eval metrics you need to use for multiclass are either merror or mlogloss which are the only ones specific for multiclass in the xgboost documentation.


Highly accurate convnets appear to have random-looking visualized weights? I'm building a TensorFlow convoluted neural network that isn't getting the accuracy that I hoped for. So I figured I would visualize the learned weights to see where the network might be stumbling. As a benchmark I started visualizing the weights for a totally different project, Google's MNIST convnet example, which has very high accuracy (99.2%).

I assumed that a very accurate model would have intuitive weights, but in reality I got weights that looked completely random. Other people seem to get similarly random results. See martin-gorner's comment with visuals on Mar 18. His are results are similar to mine.

More Googling shows that other people see essentially random behavior for non-MNIST datasets. Is this common? If so, it would seem that visualizing weights is a fruitless exercise unlikely to bring any helpful insights to the modeler. Is this true? If not, why do so many people use this visual (clear edges in the first layer, and composite images in subsequent layers) when describing how convnets work? Stanford's course material also seems to at least show edges (you have to scroll down about half of the page) when visualizing the weights. I'm not sure what to think.

The weights learned have no straightforward interpretability and i would not interpret those to be "random". What exactly do you mean by "random"? They seem to be learning very basic patterns that might reasonably extract and combine into more complex patterns as you deeper in the network, which is exactly how the networks are supposed to work.

from the Keras blog post you linked,

      A remarkable observation: a lot of these filters are identical, but rotated by some non-random factor (typically 90 degrees). This means that we could potentially compress the number of filters used in a convnet by a large factor by finding a way to make the convolution filters rotation-invariant. I can see a few ways this could be achieved --it's an interesting research direction.

The stanford page shows a great bit of what the network seems to be learning at an early layer -- combinations of edges and color blobs.

Perhaps this is more philosophical in nature, but why would we expect the layers to be meaningful or easy to interpret for humans? The network's huge number of weights are learned by optimizing a loss function via gradient descent. It can be difficult to interpret how far less complicated non-linear models "learn" or weight their coefficients; I'd expect these models would be (and are!) wildly difficult to interpret. The fact that we can see any structure in the various visualizations is cool, but they probably won't help us understand why a model isn't working all that well.

Regarding your problem, how many classes do you have and what type of accuracy are you hoping for? MNIST is an almost trivial dataset since the classes are very well separated. Some of the real-life machine vision tasks I deal with at work with good somewhat clean classes can get 80-90%~ accuracy, but errors are made in photos that humans might call something else (e.g., a photo of a studio apartment's living room in which the kitchen is clearly visible might be labeled as a kitchen and not a living room, but this is mostly okay). Defining what is a "good enough" accuracy is an issue on its own -- that's more than enough for what we're doing at the moment.

If I missed the point of your question, please let me know.


How to normalize data for Neural Network and Decision Forest I have a data set with 20000 samples, each has 12 different features. Each sample is either in category 0 or 1. I want to train a neural network and a decision forest to categorize the samples so that I can compare the results and both techniques.

The first thing I stumbled upon is the proper normalization of the data. One feature is in the range $[0,10^6]$, another one in $[30,40]$ and there is one feature that mostly takes the value 8 and sometimes 7. So as I read in different sources, proper normalization of the input data is crucial for neural networks. As I found out, there are many possible ways to normalize the data, for example:

Min-Max Normalization: The input range is linearly transformed to the interval $[0,1]$ (or alternatively $[-1,1]$, does that matter?)
Z-Score Normalization: The data is transformed to have zero mean and unit variance:
$$y_{new}=\frac{y_{old}-\text{mean}}{\sqrt{\text{Var}}}$$

Which normalization should I choose? Is normalization also needed for decision forests? With Z-Score normalization, the different features of my test data do not lie in the same range. Could this be a problem? Should every feature normalized with the same algorithm, so that I decide either to use Min-Max for all features or Z-Score for all features?

Are there combinations where the data is mapped to $[-1,1]$ and also has zero mean (which would imply a non-linear transformation of the data and hence a change in the variance and other features of the input data).

I feel a bit lost because I can't find references which answer these questions.

I disagree with the other comments.

First of all, I see no need to normalize data for decision trees. Decision trees work by calculating a score (usually entropy) for each different division of the data $(X\leq x_i,X>x_i)$. Applying a transformation to the data that does not change the order of the data makes no difference.

Random forests are just a bunch of decision trees, so it doesn't change this rationale.

Neural networks are a different story. First of all, in terms of prediction, it makes no difference. The neural network can easily counter your normalization since it just scales the weights and changes the bias. The big problem is in the training.

If you use an algorithm like resilient backpropagation to estimate the weights of the neural network, then it makes no difference. The reason is because it uses the sign of the gradient, not its magnitude, when changing the weights in the direction of whatever minimizes your error. This is the default algorithm for the neuralnet package in R, by the way.

When does it make a difference? When you are using traditional backpropagation with sigmoid activation functions, it can saturate the sigmoid derivative.

Consider the sigmoid function (green) and its derivative (blue):

What happens if you do not normalize your data is that your data is multiplied by the random weights and you get things like $s'(9999)=0$. The derivative of the sigmoid is (approximately) zero and the training process does not move along. The neural network that you end up with is just a neural network with random weights (there is no training).

Does this help us to know what the best normalization function is? But of course! First of all, it is crucial to use a normalization that centers your data because most implementation initialize bias at zero. I would normalize between -0.5 and 0.5, $\frac{X-\min{X}}{\max{X}-\min{X}}-0.5$. But standard score is also good.

The actual normalization is not very crucial because it only influences the initial iterations of the optimization process. As long as it is centered and most of your data is below 1, then it might mean you have to use slightly less or more iterations to get the same result. But the result will be the same, as long as you avoid the saturation problem I mentioned.

There is something not here discussed which is regularization. If you use regularization in your objective function, the way you normalize your data will affect the resulting model. I'm assuming your are already familiar with this. If you know that one variable is more prone to cause overfitting, your normalization of the data should take this into account. This is of course completely independent of neural networks being used.


When to One-Hot encode categorical data when following Crisp-DM I have a dataset that contains 15 categorical features (2 and 3 level factors which are non-ordinal) and 3 continuous numeric features. Seeing as most machine learning algorithms require numerical data as input features, and actually automatically One-Hot encodes them on the fly (random forest, glmnet etc.), should you not perform One-Hot encoding during data pre-processing to allow exploration of the relationship of the encoded feature data? Or is it best to rather explore relationships between raw categorical data and then only encode before running algorithms?
Basically my question evolves around data exploration and data understanding, and whether this needs to be performed on the raw or encoded categorical features?

To me it depends, because I would separate some types of Categorical Variables :

Categorical variables with few classes : OneHot as fast as you can
Categorical variable with some highly-represented classes and some low-represented classes : You can pre-process and regroup both low-represented classes in a huge "Other" class, and then OneHot and get a reasonable number of variables
Categorical variables with A LOT of low-represented class : If you OneHot directly, you'll create a lot of variables, so this feels impossible. You can, for example, browse those data so you calculate, for each class, the rate of "1" classes on your X_train. You then transform your class by this number, which is continuous, between 0 and 1, and so have information and is accepted by all models. This is called Target Encoding, and some packages built to be compatible with sklearn exist to do it automatically (like TargetEncoder, LeaveOneOut, WeightOfEvidence or JamesStein).

These are the kind of changes you can do, the choice of OHE directly, or pre-process before, it depends on the variable...
If your question is, for example, to know if you make feature selection before OHE or after, I'd suggest you mainly making it after : Remove useless variable (with no info), then OHE/preprocess remaining ones, and then make feature selection again.
Let's take an example : a variable called Age being classes like [0;10], [10;20], ... it's often significative if the value is >80 or <20, but doesn't care if it's 35 or 45, so the OHE will only select Age_[0;10], Age_[10;20], Age[80_90] and Age_90+


How can I provide an answer to Neural Network skeptics? After given several talks on NN's, I always have a skeptic that wants a real measure of how well the model is. How do you know the model is truly accurate?

I explain the use of test data etc. to evaluate the total error, however, there is always someone who wants to know about the error associated with each weight?

Can anyone enlighten me as to how I can satisfy these types of questions?

It has become a real issue.

Neural networks are essentially a black box, especially big ones. You could know even how it is designed and how it is training, but you really do not know how it is working in the end. In my work lots of people want to understand the model instead of using "black box" models. This is the reason why companies choose to use linear regressions and polynomial models instead of using stronger machine learning algorithms, like LightGBM and Neural Networks.

I never found a true answer to this question. Some engineers are taught that you cannot use models that you cannot understand. Therefore every model that is a "black box" is not usable for them. This means that most of machine learning models are magic and heresy for them. Though take this with a grain of salt, this is my subjective experience. Sometimes as the time passes these people are more willing to use data science methods because it becomes mainstream. They start to trust the methods, because others use them.

The situation is different on the higher level. For high-tier managers it matters less how to interpret the model, but more what results it could give you. They are more willing to try, especially if there is a hype of something, like "artificial intelligence", "data science".

As a result, I could only give you an advice to find some good support higher in the hierarchy of the company. Someone who believes in data science more and who has more power in the company.

In data science community the performance of the model on the test dataset is one of the most important things people look at. Just look at the competitions on kaggle.com. They are extremely focused on test dataset and the performance of these models is really good.

The only problem with performance on the test dataset is that it depends on the data in the test dataset. If in real life you will have completely different data that will be outside of the bounds of the test dataset, then the test dataset will not be able to give a good approximation of the performance of the model in real life.


Is my general understanding of finding weights correct? I started a course in Deep Learning. I'm trying to make an example in order to explain to myself how the weights are found mathematically.
If what I wrote below is nonsense I'll be glad to hear an explanation. Thanks.

So, for a given image we do WX+b. We get some vector Y and then we compare it to a desired label vector L according to . I'm assuming that we calculate D with "Cosine Similarity". For simplicity S(Y)==Y. So what we're trying to do is to calculate  so it will be one.
Let’s say we have image X of the letter “a”  and two labels (“a”, “b”). Then  . We want to calculate W and b for which we will get such vector  that when we’ll insert it into  we’ll get zero. We convert X to a vector. Since we have 2 labels and size of the X is 9, the W and b are the following: .  So, we get: . This gives us the following system of equations: . So, now we need to solve the following .

If what I wrote above is not nonsense, I don't quite understand where finding minimum is applied?

This is interesting !?!
The way you think about it is as solving a linear system equation, which is not exactly the case.
Imagine if you have thousands of data points , and you try to find the set of weights that solves ( satisfies) these points, what if an exact solution is not exist. What if there is no set of weights that exactly solves all these points. It becomes an optimization problem. What is the set of weight that minimizes the error. In this case the error is the difference between the actual output and the output you get by using the current weights.
In fact, in current machine learning technology, it is difficult even to construct such a linear system.
We use optimization techniques to solve this and find the optimal set of weights


Is BERT a language model? Is BERT a language model in the sense of a function that gets a sentence and returns a probability?
I know its main usage is sentence embedding, but can it also provide this functionality?

No, BERT is not a traditional language model. It is a model trained on a masked language model loss, and it cannot be used to compute the probability of a sentence like a normal LM.

A normal LM takes an autoregressive factorization of the probability of the sentence:

$p(s) = \prod_t P(w_t | w_{<t})$

On the other hand, BERT's masked LM loss focuses on the probability of the (masked) token at a specific position given the rest of the unmasked tokens in the sentence.

Therefore, it makes no sense to use the token probabilities generated by BERT and multiply them to obtain a sentence level probability.

A secondary issue is the BERT's tokenization is subword-level so, even if it would make sense to compute a sentence-level probability with BERT, such a probability would not be comparable with a word-level LM, as we would not be taking into account all possible word segmentations into subwords.

UPDATE: there is a new technique called Masked Language Model Scoring (ACL'20) that allows precisely what the OP asked for. From the article:

  To score a sentence, one creates copies with each token masked out.
  The log probability for each missing token is summed over copies to
  give the pseudo-log-likelihood score (PLL).

So the answer is now YES. It is possible to score a sentence using BERT, by means of the described pseudo-log-likelihood score.


Detecting boilerplate in text samples I have a corpus of unstructured text that, due to a concatenation from different sources, has boilerplate metadata that I would like to remove. For example:

DESCRIPTION PROVIDED BY AUTHOR: The goal of my ...
Author provided: The goal of my ...
The goal of my ... END OF TRANSCRIPT
The goal of my ... END, SPONSORED BY COMPANY XYZ
The goal of my ... SPONSORED: COMPANY XYZ, All rights reserved, date: 10/21

This boilerplate can be assumed to occur in beginning or end of each sample. What are some robust methods for wrangling this out of the data?

This might get you started. Phrase length is determined by the range() function. Basically this tokenizes and creates n-grams. Then it counts each token. Tokens with a high mean over all documents (occurs often across documents) is printed out in the last line.

from sklearn.feature_extraction.text import CountVectorizer
import numpy as np
import nltk

text = """DESCRIPTION PROVIDED BY AUTHOR: The goal of my a...
Author provided: The goal of my b...
The goal of my c... END OF TRANSCRIPT
The goal of my d... END SPONSORED BY COMPANY XYZ
The goal of my e... SPONSORED: COMPANY XYZ All rights reserved date: 10/21
"""

def todocuments(lines):
    for line in lines:
        words = line.lower().split(' ')
        doc = ""
        for n in range(3, 6):
            ts = nltk.ngrams(words, n)
            for t in ts: doc = doc + " " + str.join('_', t)
        yield doc

cv = CountVectorizer(min_df=.5)

fit = cv.fit_transform(todocuments(text.splitlines()))
vocab_idx = {b: a for a, b in cv.vocabulary_.items()}

means = fit.mean(axis=0)
arr = np.squeeze(np.asarray(means))
[vocab_idx[idx] for idx in np.where(arr > .95)[0]]
# ['goal_of_my', 'the_goal_of', 'the_goal_of_my']


Random Forest: how efficient is it for features in a range of values (not binary thresholding)? Is it correct that each feature is only used once in one decision tree ? Then, if a feature (the most decisive for the classification) determines a classification for a range of values, can the algorithm detect that in any way or will it fail ?
To be more clear, let's say the classification is to classify people as teenager or not. Then the determinant feature will be the age, and classification will be if this feature is between 12 and 18.
So we don't have only one threshold (age<12) or (age>18), but two.
Can the Random Forest algorithm succeed in this case ?
Thanks for any suggestion

Is it correct that each feature is only used once in one decision tree?

No, if the variable is a continuous value like age, it can use threshold splits at each level. So, same feature can be used multiple times in any given branch of the decision tree.
And since the random forest is nothing but collection of decision tree it follows the same principle.
Reference


Issues related to the code for ROI pooling from the feature map I am trying to do ROI pooling on the feature map obtained from the VGG layers but I don't know how to code this layers. Can anybody help me out?

Here is my VGG layers:

model=Sequential()
    model.add(ZeroPadding2D((1,1),input_shape=(3,112,112)))
    model.add(Convolution2D(64, 3, 3, activation='relu'))
    model.add(BatchNormalization(momentum=0.99))
    model.add(ZeroPadding2D((1,1)))
    model.add(Convolution2D(64, 3, 3, activation='relu'))
    model.add(BatchNormalization(momentum=0.99))
    model.add(MaxPooling2D((2,2), strides=(2,2), dim_ordering="th"))

model.add(ZeroPadding2D((1,1)))
model.add(Convolution2D(128, 3, 3, activation='relu'))
model.add(BatchNormalization(momentum=0.99))
model.add(ZeroPadding2D((1,1)))
model.add(Convolution2D(128, 3, 3, activation='relu'))
model.add(BatchNormalization(momentum=0.99))
model.add(MaxPooling2D((2,2), strides=(2,2), dim_ordering="th"))

model.add(ZeroPadding2D((1,1)))
model.add(Convolution2D(256, 3, 3, activation='relu'))

`
The corresponding model is described in the picture below:

For the implementation of ROI pooling kindly refer to the following article...

https://medium.com/xplore-ai/implementing-attention-in-tensorflow-keras-using-roi-pooling-992508b6592b

https://deepsense.ai/region-of-interest-pooling-explained/


Clustering Weekday Weekend Data and Multicollinearity Hi I have data of weekday and weekend step counts in which I extracted metrics from them such as the wd steps, we steps, standard deviation of wd steps, standard deviation of we steps and so on...

  wd_count  we_count  wd_sd_count  we_sd_count  ... .... ....
1  5000      3000      300          500
2  7000      2000      400          100

If I do clustering on this data, the weekday and weekend variables are going to be highly correlated and I will have to remove them before clustering. Is there any way around this problem for this kind of analysis?

Yes its called correlation clustering.

Even though correlation can cause problems with many clustering algorithms by giving extra weight on these attributes, it would be best to drop highly correlated variables for example with PCA

However, there exist correlation clustering algorithms that are meant to process data containing multiple correlations, and cluster objects based on the correlations they exhibit, using your problem exactly to an advantage.


How to interpretate multiple histograms corresponding to each feature in multiple linear regression for relationship? Used matplotlib to plot the histograms for each feature in  Boston dataset available in scikitlearn library.
How to interpretate the histograms to determine the correlation or significance of that feature in model training?

A histogram shows the distribution of values in a feature/variable. For instance in CHAS you see that there are two values (0,1), which means binary encoding (aka one-hot or dummy).

Histograms are helpful to understand the general structure of data and to detect things such as outlier or censoring. They are also good to get a quick idea of skewness in the data/distribution. Look at „age“ for example. You can see concentration of values at the top of the x-values. This likely is due to houses being older than 100 years are recorded as „100“ in the data. A similar pattern occurs for the price.

However, wrt „significance“ and correlation, histograms do not tell you a lot. For correlation, look at correlation measures. For significance (I guess you mean importance), you need to do sequential testing of features or you employ Lasso/Ridge regression for a more systematic approach.


How to do target encoding when data has repeated rows? How can I do encoding for a category when data has repeated rows? Can I do target encoding? Or Is there another encoding I can use?

I want to figure how to include a categorical variable in a model to predict a numerical variable Y.    Because I am working with some legislative data, my challenge is my category code is over 4000 unique values, those values that cannot be easily grouped(*), and they can have repeats.  In fact, anecdotally I know that both that code is a strong predictor of Y and repeats in a code per row will have a stronger impact on Y.

My attempt at a solution:
In fact, I tried to do this myself thinking that maybe something as simply as mean target times the count of each code (Table B * Table C). However, I am not sure if this would be valid type of encoding.

A = sample legislative data to predict ethics variable Y, the values have been anonymized

+-----+-----------+------------+
| row | Category  | Y variable |
+-----+-----------+------------+
|   2 | 15-11-2   |        370 |
|   2 | 15-11-106 |        370 |
|   2 | 15-5-44   |        370 |
|   2 | 15-11-41  |        370 |
|   3 | 15-7-23   |        363 |
|   3 | 15-7-23   |        363 |
|   3 | 15-11-37  |        363 |
|   3 | 15-11-37  |        363 |
|   3 | 15-5-21   |        363 |
|   6 | 15-11-106 |        287 |
|   6 | 15-5-21   |        287 |
|   7 | 16-5-70   |        931 |
|   7 | 15-5-23   |        931 |
|   7 | 15-5-41   |        931 |
|   9 | 15-11-7   |        336 |
|   9 | 15-2-30   |        336 |
|   9 | 40-11-76  |        336 |
|   9 | 15-2-2    |        336 |
|   9 | 15-11-131 |        336 |
|   9 | 15-11-131 |        336 |
|  10 | 15-2-30   |        350 |
|  10 | 15-11-131 |        350 |
+-----+-----------+------------+

B = The Mean Values for the unique rows

+-----------+-----------------------+
| Category  | Average of Y variable |
+-----------+-----------------------+
| 15-11-106 | 334.714               |
| 15-11-2   | 298.994               |
| 15-11-37  | 251.931               |
| 15-11-41  | 399.210               |
| 15-11-7   | 288.021               |
| 15-2-2    | 234.533               |
| 15-2-30   | 303.127               |
| 15-5-21   | 317.999               |
| 15-5-23   | 252.109               |
| 15-5-41   | 300.882               |
| 15-5-44   | 499.405               |
| 15-5-70   | 277.306               |
| 15-7-23   | 244.976               |
| 40-11-76  | 261.938               |
| 15-11-131 | 319.799               |
+-----------+-----------------------+

Table C The counts of each category

+-----+-----------+-------------------+
| row | Category  | Count of Category |
+-----+-----------+-------------------+
|   2 | 15-11-106 |                 1 |
|   2 | 15-11-2   |                 1 |
|   2 | 15-11-41  |                 1 |
|   2 | 15-5-44   |                 1 |
|   3 | 15-11-37  |                 2 |
|   3 | 15-5-21   |                 1 |
|   4 | 15-7-23   |                 2 |
|   4 | 16-8-18   |                 1 |
|   5 | 16-7-1    |                 1 |
|   6 | 16-8-2    |                 1 |
|   6 | 15-11-106 |                 1 |
|   7 | 15-5-21   |                 1 |
|   7 | 15-5-23   |                 1 |
|   8 | 15-5-41   |                 1 |
|   8 | 16-5-70   |                 1 |
|   8 | 16-11-37  |                 1 |
|   8 | 16-7-21   |                 1 |
|   9 | 15-11-131 |                 2 |
|   9 | 15-11-7   |                 1 |
|   9 | 15-2-2    |                 1 |
|   9 | 15-2-30   |                 1 |
|   9 | 40-11-76  |                 1 |
|  10 | 15-11-131 |                 1 |
|  10 | 15-2-30   |                 1 |
|  10 | 40-11-76  |                 1 |
|     |           |                   |
+-----+-----------+-------------------+

Footnote:
(*) What I mean by "the category code cannot be easily grouped" is that 15-11-2 and 15-11-1 cannot be group under 15-11 or 15.  I have also tried clustering.

Yes, you can do target encoding. watch this video from @2:18 to @3:50. Hope this helps.

Update: Table B is enough to encode, you have mean values of y among each category. If a responce has like this [15-11-106 ,15-11-2], then you can replace those values with them [334.714, 298.994], now you can apply any statistic to get final encoding value, whether it could be mean/mode/median etc... of those values.

The mean target times the count of each code (Table B * Table C) is nothing but the sum of the target variable for each unique category(instead of mean of target variable you are going to choose sum of target variable). I don't think it's a good strategy.


How reproducible should CNN models be? I want to train several CNN architectures with Google Colab (GPU), Keras and Tensorflow.
Since the trained models are not reproducible due to GPU support, I would like to train the models several times and determine the mean and the standard deviation of the results.
I'm totally unsure if I should at least try to make the models minimal reproducible? For example with the following code at the beginning of the program:

import numpy
import tensorflow as tf
import random as rn
import os

os.environ['PYTHONHASHSEED']='0'
np.random.seed(1)
rn.seed(1)
tf.set_random_seed(1)

from keras import backend as K
if 'tensorflow' == K.backend():
   import tensorflow as tf
   tf.set_random_seed(1)
   from keras.backend.tensorflow_backend import set_session
   config = tf.ConfigProto()
   config.gpu_options.allow_growth = True
   config.gpu_options.visible_device_list = "0"
   set_session(tf.Session(config=config)

I don't know if this makes sense. Would it be better if I did not use seeds at all?
What do you think?

Getting slightly different results is natural and should not be a problem. How to minimize the instabilities due to several contributing factors is discussed at length in the linked post below, for Keras using different backends including TensorFlow:

https://machinelearningmastery.com/reproducible-results-neural-networks-keras/


Clustering algorithm for a distance matrix I have a similarity matrix between N objects. For each N objects, I have a measure of how similar they are between each others - 0 being identical (the main diagonal) and increasing values as they get less and less similar. Something like that (actual matrix would be 10 000 x 10 000 if I have say 10 000 elements to cluster together):

[ 0 5 12 7 ]
[ - 0 9  2 ]
[ - - 0  6 ]
[ - - -  0 ]

(Supposing index starting at 0) So here object_0 would be most similar to object_1 (5), but object_1 itself would be more similar to object_3 (2), etc.

I would like to cluster that into k groups, so as to minimize either the overall score or intra-cluster score. I'm not sure if that would practically make that much of a difference. I haven't checked the maths on that, but I feel that there could be cases where minimizing a specific cluster's score wouldn't necessarily minimize the overall score (across all clusters). At the end of the day, even thought rigorously speaking there may be a difference I may not practically speaking care if the results are close enough either way.

A few details:

Clusters don't have to be equal sizes
I don't really mind if the number of clusters is an input or if it is being decided by the algo itself in some fashion (thought I would prefer inputing it)
Typically I would have about 10 000 elements to cluster. Maybe 40 000 if I push it. But then I may need to repeatedly run it for a bunch of matrices that size.

On top of an algo/lib that does that, bonus points for:

Some library that's already implemented it, so I can hopefully just format the data properly for it and just see if it seems to give good results or not before I sink a whole lot of time in it
For such a lib, support for parallel processing
Other nice bells & whistles (like maximum number of iterations, ability to input some stopping criteria, etc.)

Failing that, of course it could also just be some pseudo-code that allows me to cluster them into groups.

There are hundreds of algorithms to choose from.

Hierarchical clustering in it's myriad of variants. Cut the dendrogram as desired, e.g., to get k clusters
PAM, the closest match to k-means on a distance matrix (minimizes the average distance from the cluster center)
Spectral clustering
DBSCAN
OPTICS
HDBSCAN*
Affinity Propagation
...


Python : Approaches to mine for time ordered sequences I have a dataset that looks like this :
timestamp          event

12/1/2020 14:30     A
12/1/2020 14:12     C
12/1/2020 14:10     A
12/1/2020 12:01     B
11/1/2020 21:20     A
11/1/2020 21:00     B
   .....           ...

events are actions by the user on an application. timestamp is when that particular event was raised.
I want to mine for sequences of events in the data to find out what a user's work-flow (market basket analysis with time constraint?) has been inside the application from the day user started using the application.
What would the proper approach to solving this problem be?
Are there any libraries that will allow me to apply sequence mining as a black box??

Tru to make some features from the datetime columns -
attr = ['Year', 'Month', 'Week', 'Day', 'Dayofweek', 'Is_month_end', 'Is_month_start',
'Is_quarter_end', 'Is_quarter_start', 'Is_year_end', 'Is_year_start']

you can google search about making date features.
2) Find the cumulative sum of the events -
cumsum_df_all = df.groupby('column')[other-columns].cumsum()
cumsum_14_day = (df.groupby('column')[other-columns].rolling(14, min_periods=1).sum().reset_index(0, drop=True)

like wise for 7,3,1 day, just look how frequent the data is and fast it changes.

Likewise find the roling std dev. and avg.

rolling_avg_14_day = (df.groupby('column')[other-columns].rolling(14,min_periods=1).mean().reset_index(0, drop=True))
set the rolling period accordingly.

find the difference between the event values for a specific period -
diff_7_day = df.groupby('column')[other-columns].diff(periods=7)

basically, we are trying to convert time ordered sequences to linear regression problem.