Why can decision trees have a high amount of variance I've heard that decision trees can have a high amount of variance, and that for a data set $D$ split into test/train the decision tree could be quite different depending on how the data was split. Apparently, this provides motivation for algorithms such as Random Forest.

Is this correct? Why does a decision tree suffer from high variability?

edit

just to note that I don't really follow the current answer
 and haven't been able to solve that in the comments.

It is relatively simple if you understand what variance refers to in this context. A model has high variance if it is very sensitive to (small) changes in the training data.

A decision tree has high variance because, if you imagine a very large tree, it can basically adjust its predictions to every single input.

Consider you wanted to predict the outcome of a soccer game. A decision tree could make decisions like:

  IF

  player X is on the field AND
  team A has a home game AND
  the weather is sunny AND
  the number of attending fans >= 26000 AND
  it is past 3pm

  THEN team A wins.

If the tree is very deep, it will get very specific and you may only have one such game in your training data. It probably would not be appropriate to base your predictions on just one example.

Now, if you make a small change e.g. set the number of attending fans to 25999, a decision tree might give you a completely different answer (because the game now doesn't meet the 4th condition).

Linear regression, for example, would not be so sensitive to a small change because it is limited ("biased" -> see bias-variance tradeoff) to linear relationships and cannot represent sudden changes from 25999 to 26000 fans.

That's why it is important to not make decision trees arbitrary large/deep. This limits its variance.

(See e.g. here for more on how random forests can help with this further.)


How to identify a field as holding personal identifiable information from the name of the field itself using ML model in python? Is it possible to automatically detect fields holding personal information (name, phone, address, SSN, passport, gov ID...) from its names, using python in order to upload datasets into the cloud after encrypting or anonymizing the PII fields?
I am open to do my own model by training it on a dataset that holds thousands of fields and each one is classified whether personal or not. But apparently I can't find any related datasets.

In such cases, where the data is not availible, it will be better to make the dataset on our own. Just create a google form and sent to a few friends and family and hence you will end up with a dataset. Apart from this it you can check this article on anonymizing information on kaggle


Gradient Descent - how many values are calculated in loss function? I'm a little bit confused how loss function is calculated in neural network training. There's is said that in theory when using Grid Search or Monte Carlo methods we can calculate all the possible loss function values. But obviously this requires too many resources and is not a good way for neural network training. Alternatively, when using Gradient Descent we have a possibility to evaluate a single value to know in which direction we should go in order to test for the next value. We then could climb down the ladder one step by another until we reach the optimal value.

But on the other hand, in the following PyTorch example there is said that the loss function is calculated on the basis of all predicted and real values. And after that the gradient is calculated.
So, what's the point of Gradient Descent in this way when all the loss values in previous step are calculated anyway?

Gradient descent is based on sources: your data and your loss function.

In supervised learning, at each training step the predictions of the Network are compared with the atcual, true results. The value of a loss function is calcualated, which is telling your model "how wrong it is". At this point, the weights of the Network must be updated accordingly.

In order to do that, a formula based on the chain rule of derivatives calculates retrospectively the contribution of each weight to the final loss value. The value of each weight is then changed, based on their impact on the loss function (mathematically, it's a first partial derivative for each weight). This process is called backpropagation, since it logically starts from the bottom of the Network and is computed backwards up to the input layer.

This process has to be done for each of the Network's learnable weights. The higher the number of parameters, the higher the number of partial derivatives that are computed at each training iteration for the weight update. When all the weights are updated, the position of the Gradient will (hopefully) be a small step closer to the global minimum of the loss function.

Coming to your doubts: grid search on the loss function values is something you can do only in theory, not in practice. It could take an impossible amount of time to do that, even for the most powerful computers. At the moment, the Gradient descent algorithm is the main tool to train Neural Networks. As far as I know, other methods such as Monte Carlo and Genetic algorithms are feasible but not practical, therefore not state-of-the-art.

I suggest you two great posts to understand how Gradient descent and backprop work: this one by Andrej Karpathy and this one by Colah.


spark item similarity recommendation I would like to build an recommendation engine using spark's Mlib itemsimilarity as mentioned here LINK

But it seems spark do not have this algorithm any more and some forums suggesting me to use ALS, but please clarify is item similarity removed from spark or any other alter natives?
 I want to index my indicators into search engine and I find Sparks Item similarity best suits for me. Please advice.

For your recommendation engine, if you've chosen to go by item similarity approach, then you can use Spark's RowMatrix datatype to achieve this task.

Item similarity approach is just about creating a square matrix of items in your catalog (i.e. itemID X itemID), where each element    of the matrix is the magnitude  of similarity between  and . This magnitude of similarity can be calculated by using any similarity function, most popular being the Cosine Similarity.

In spark this can be done by:

Create a k X n matrix, where n items are described as k-dimensioned vectors. For representing each item as a k dimension vector, you can use ALS which represents each entity in a latent factor space. The dimension of this space (k) can be chosen by you.

This k X n matrix can be represented as RDD[Vector].

Convert this k X n matrix to RowMatrix.
Use columnSimilarities() function to get a n X n matrix of similarities between n items.

Finally , whenever an itemi is being viewed, you can recommend other m items obtained by sorting the items in row i by the decreasing value of similarities and picking top m.

More details can be found here.


Learnable parameters in DNN I've come across the term "learnable parameters" recently, and googling didn't help much as most search was describing learnable parameters in a CNN instead of a DNN. Is there any difference between the two?
How would I compute the number of learnable parameters in a DNN? Could anyone please explain what those are with an example? I'm new to machine learning so I would appreciate some help on this.

In neural networks in general, and in deep learning algorithms (CNN, DNN, etc.) that are also based on neural networks, learnable parameters are parameters that will be learned by the model during the training procedure such as weights and biases.
You can generate learnable parameters for each layer of your model.
DNN is a deep neural network. What I understand that when a neural network becomes deep, it can be said that it's a deep learning model.
So, to compute a DNN or CNN learnable parameters, you can build your model using Keras. It can automatically generate the models learnable parameters for each layer. When you complete building your model, type model.summary(). It generates a list of learnable parameters when you run it.
This link builds a DNN model using using Keras.


Possible harm in standardizing one-hot encoded features While there may not be any added value in standardizing one-hot encoded features prior to applying linear models, is there is any harm in doing so (i.e., affecting model performance)?
Standardizing definition:  applying (x - mean) / std  to make the feature mean and std 0, 1 respectively)
I prefer applying standardization to my entire training dataset after one-hot encoding, rather than applying it only to the numerical features. I feel it would significantly simplify my pipeline.
For example, if I have a binary feature then the vector that will be provided to the model is [1,1,0,0,0,1,1].
If standardization is applied to this binary feature prior to fitting the model (subtract mean = ~0.57 and divide by std = ~ 0.49), the vector will become
[ 0.8660254 ,  0.8660254 , -1.15470054, -1.15470054, -1.15470054, 0.8660254 ,  0.8660254 ]

With unpenalized linear models, there is no difference.  The coefficients will just scale to counteract the new scale of the variables, and the intercept will shift to compensate for the centering.
With penalized linear models though, there will be a difference.  Since the standard deviation of a binary variable is at most $1/2$, you'll be increasing the overall scale of the variable by standardizing.  That will cause the unpenalized coefficient to decrease in magnitude, which will change the balance of how the penalty applies to different features.  I suspect there is no "better" approach then: sometimes the penalty improves performance when the dummies are scaled, and sometimes degrades performance.


Back-propagation and stochastic gradient descent Is backpropagation a learning method or an optimisation method?

How are backpropagation and stochastic gradient descent related to each other?

Stochastic Gradient Descent (SGD) is an optimization method. As the name suggests, it depends on the gradient of the optimization objective.

Let's say you want to train a neural network. Usually, the loss function $L$ is defined as a mean or a sum over some "error" $l_i$ for each individual data point like this

$$ L(\theta) = \frac{1}{N} \sum_{i=0}^N l_i(\theta)$$

where $N$ is the number of data points and $\theta$ the model parameters.

For SGD you would randomly sample $i$ at each time step $t$ and do

$$ \theta_{t+1} = \theta_{t} - \alpha \nabla_{\theta}l_i(\theta_t)$$

with some learning rate $\alpha$ and the gradient with respect to the model parameters $\nabla_{\theta}$.

Backpropagation is now used to compute the gradient $\nabla_{\theta}l_i(\theta_t)$. As $l_i$ depends on a neural network with parameters $\theta$, this is not necessarily straight-forward, but it can be done quite efficiently using the chain rule in a smart way. This involves recursively computing the gradient of parameters in some layer using the gradients from higher layers, i.e. the gradients are computed starting at the network output and moving backwards. Hence the name backpropagation.

The wikipedia article on backpropagation goes through the math in a detailed manner.


Neural Net for regression task on images only learning mean of training data I'm trying to train neural networks to extract parameters out of input data.
In particular determining the width, height and color of bricks (see attached pictures as example inputs).

I've already tried CNNs operating on the raw images and NNs operating on extracted features ( 2 describing edges and a color histogram ).

Both approaches seem to learn the mean of the training data. The color is mostly gray and the width and height between the two extremes. The output does not depend on the input as the NN always outputs the "same" values (only very small fluctuations).

As I'm operating on generated data I have a perfect correlation between input and desired output and infinite training data. The used loss function is mean squared error (also tried the absolute error).

Can someone explain why this is happening or how to avoid this training behavior?

I'm using the library tiny-cnn.
My current approach (using features) uses 5 fully connected layers, decreasing layer size from 1200 to 17(amount of parameters).

    net << fully_connected_layer<sigmoid>(1200, 600)
    << fully_connected_layer<sigmoid>(600, 300)
    << fully_connected_layer<sigmoid>(300, 150)
    << fully_connected_layer<sigmoid>(150, 60)
    << fully_connected_layer<identity>(60, 17);

I've test several CNN structures, always ending in 2 fully connected layers. I tried different kernel sizes from 3 to 11.

This is one of the last I tried:

net << convolutional_layer<tan_h>(100, 100, 5, 3, 4)
    << average_pooling_layer<tan_h>(96, 96, 4, 2)
    << convolutional_layer<tan_h>(48, 48, 7, 4, 8)
    << average_pooling_layer<tan_h>(42, 42, 8, 2)
    << convolutional_layer<tan_h>(21, 21, 7, 8, 16)
    << average_pooling_layer<tan_h>(15, 15, 16, 3)
    << convolutional_layer<tan_h>(5, 5, 5, 16, 32)
    << fully_connected_layer<relu>(32, 64)
    << fully_connected_layer<identity>(64, size);

5-conv, average, 7-conv, average, 7-conv, average(3x3), 5-conv, full, full

I've mainly used RMSProp and adagrad with standard learning rates but also tried higher and lower rates.

Example inputs:

I took a deeper look at this problem with some example data (that the OP provided) and some simplifications. I just predicted brick height param as opposed to the 17 example params in the question. I used the Keras library in Python to explore a few different architectures.

Initially I replicated the problem, simple CNNs were predicting identical values for all heights, and the best loss (around 0.0053 on the sample data) was when this predicted the mean very precisely for all inputs. For some reason the network could not find the visual structure that is obviously in the data.

After some experimentation, I had a network that functioned as intended. It was surprisingly deep for such a simple problem. My best guess at the reason why is that the combined overlap of all the convolutions needs to be of the same size order as the features being detected.

Some important details:

Check that the order of the dimensions of the images when read with the image library you are using match the expected structure that the CNN is working on. If the dimensions are in a different order, this will not cause any error, but it scrambles the interpretation used by the CNN and prevents it learning anything but the mean.
Avoid using pooling layers, as they obscure pixel locations of features from further layers.
Check that your parameters are not ambiguous. The two "brick width" parameters seem interchangeable for instance. That makes it much harder for the network to predict them, and it is likely to output a rough mean for the example even if you avoid the problems with architecture.
The problem is harder to find a working architecture for than I first thought. I am not entirely sure why that is the case, but am guessing it is to do with the size of the features that need to be isolated in order for the network to "measure" them. Eventual working architectures were quite deep e.g. 10 convolutional layers, despite the pixel-level structures being quite obvious and clear in the training data.

Here is a gist of my solution in Python/Keras. Summary of architecture in the gist: 4 times Conv2d layers with 8 feature maps and 5x5 kernel. Dropout 25%. 4 more times Conv2d layers with 16 feature maps and 5x5 kernel. Dropout 50%. Fully connected layer with 256 outputs. Dropout 50%. Fully connected later with 128 outputs. Dropout 50%. Single output neuron. All layers use ReLU activation, except output which is linear.

The example re-sizes images to 50x50 for speed. However, using 100x100 images works fine. I added a couple of convolutional layers to it for that (one each for the 8 features and 16 features section), and get a reasonable loss of ~0.00025 on a validation set. It looks possible to get a lower loss still with more training examples, more epochs and perhaps more feature maps. Many of the predictions were very good, close to the actual value.

Interestingly, the worst errors were for short bricks predicting close to a 2:1 or 3:2 ratio with the ground truth, implying that the network had found correct structure but was being confused by the periodic nature of the bricks. I think this is something that could be resolved with more training examples (and perhaps more features in higher layers) and it should be possible to get much lower loss metric on brick height.

I think that the corresponding tiny-cnn structure looks like this:

net << convolutional_layer<relu>(100, 100, 5, 3, 8)
    << convolutional_layer<relu>(96, 96, 5, 8, 8)
    << convolutional_layer<relu>(92, 92, 5, 8, 8)
    << convolutional_layer<relu>(88, 88, 5, 8, 8)
    << convolutional_layer<relu>(84, 84, 5, 8, 8)
    << dropout(80 * 80 * 8, 0.25)
    << convolutional_layer<relu>(80, 80, 5, 8, 16)
    << convolutional_layer<relu>(76, 76, 5, 16, 16)
    << convolutional_layer<relu>(72, 72, 5, 16, 16)
    << convolutional_layer<relu>(68, 68, 5, 16, 16)
    << convolutional_layer<relu>(64, 64, 5, 16, 16)
    << dropout(60 * 60 * 16, 0.5)
    << fully_connected_layer<relu>(57600, 256)
    << dropout(256, 0.5)
    << fully_connected_layer<relu>(256, 128)
    << dropout(128, 0.5)
    << fully_connected_layer<identity>(128, 1)


How do I use the data I have to make predictions? I've a set of CPUs (~100). I've the results of each CPU being benchmarked against a suite of benchmarks (~8). There is a specific task that interests me, and I need to find the optimal CPU for that task. A small subset of CPUs (~10-20) has noisy measurements for how well it performs at that specific task.
Two questions:

How do I use the data I have to make predictions about how well a given CPU will perform at that specific task (when its performance wasn't directly measured on that specific task)? This presumes that the large set of benchmark suite results have captured the CPU performance aspects that are relevant to that specific task. An interpretable white-box solution would be prefered, so I can tell which benchmarks are relevant for which task (and how relevant).
How do you formulate this problem in a general way, so that I could research it?

I'm quite pessimistic about this: ideally one would run the actual task (or a proxy for the task, like a simplified version of the task) with the 100 CPUs. Here the only option I can think of is to:

use the noisy measurements of the 10-20 CPUs as the basis of the data for the actual task
try to map these measurements to the 8 benchmarks, so that we can predict the performance of the 100 CPUs based on their results with the 8 benchmarks.

I'm pessimistic because:

the measurements on the tasks are noisy,
there's no guarantee that these measurements can be reliably linked with the 8 benchmarks,
and no guarantee either that the 8 benchmarks would provide a good estimate on the task.

That being said, technically you could imagine a design with supervised learning, the idea being to predict performance on the task (target variable) from the results of the 8 benchmarks (features). The training data would consist of the results of the 10-20 CPUs for which you have both results (8 benchmarks + performance on the task). I would suggest trying this with decision tree regression because it provides you with an interpretable model (white box solution).


PCA, covariance, eigenvector matrix and rotation I am following the Coursera NLP specialization, and in particular the lab "Another explanation about PCA" in Course 1 Week 3.
From the lab, I recovered the following code. It creates 2 random variables, rotates them to make them dependent and correlated, and then run PCA on them:
import numpy as np
from sklearn.decomposition import PCA
import math

std1 = 1     # The desired standard deviation of our first random variable
std2 = 0.333 # The desired standard deviation of our second random variable

x = np.random.normal(0, std1, 1000) # Get 1000 samples from x ~ N(0, std1)
y = np.random.normal(0, std2, 1000)  # Get 1000 samples from y ~ N(0, std2)

# PCA works better if the data is centered
x = x - np.mean(x) # Center x
y = y - np.mean(y) # Center y

#Define a pair of dependent variables with a desired amount of covariance
n = 1 # Magnitude of covariance.
angle = np.arctan(1 / n) # Convert the covariance to and angle
print('angle: ',  angle * 180 / math.pi)

# Create a rotation matrix using the given angle
rotationMatrix = np.array([[np.cos(angle), np.sin(angle)],
                 [-np.sin(angle), np.cos(angle)]])

# Create a matrix with columns x and y
xy = np.concatenate(([x], [y]), axis=0).T

# Get covariance matrix of xy
print("Covariance matrix of xy")
covmat = np.cov(xy, rowvar=False)
print(f"{np.sqrt(covmat[0,0]):.3f} = {std1}")
print(f"{np.sqrt(covmat[1,1]):.3f} = {std2}")

# Transform the data using the rotation matrix. It correlates the two variables
data = np.dot(xy, rotationMatrix)

# Get covariance matrix of data
print("Covariance matrix of data")
covmat = np.cov(data, rowvar=False)
print(f"{np.sqrt(covmat[0,0]):.3f} = {std1}")
print(f"{np.sqrt(covmat[1,1]):.3f} = {std2}")
print(f"{covmat[0,1]:.3f} = {n}")

# Apply PCA.
pcaTr = PCA(n_components=2).fit(data)

# In theory, the Eigenvector matrix must be the
# inverse of the original rotationMatrix.
print("** These two matrices should be equal **")
print("Eigenvector matrix")
print(pcaTr.components_)
print("Inverse of original rotation matrix")
print(np.linalg.inv(rotationMatrix))

I get the following output:
angle:  45.0
Covariance matrix of xy
1.031 = 1
0.325 = 0.333
Covariance matrix of data
0.764 = 1
0.765 = 0.333
0.479 = 1
** This two matrices should be equal **
Eigenvector matrix
[[ 0.70632393  0.70788877]
 [ 0.70788877 -0.70632393]]
Inverse of original rotation matrix
[[ 0.70710678  0.70710678]
 [-0.70710678  0.70710678]]

Why does n=1 define the magnitude of the covariance?
Why we can obtain the angle between the variables as angle = np.arctan(1 / n)?
Why I don't achieve this covariance between the variable when I take element (0,1) of the covariance matrix (the second time that I run np.cov, so the line covmat = np.cov(data, rowvar=False)?
Why does the rotation change the variables variance from the initial std1 and std2?
Why "In theory, the Eigenvector matrix must be the inverse of the original rotationMatrix" ?
Why this is not the case?

Having 2 uncorrelated normal variables $x_1 \sim N(0, 1)$ and $x_2 \sim N(0, 1)$, one can correlate them through a linear transformation:
$$A = \begin{bmatrix}a & b \\ c & d\end{bmatrix}$$
$$X' = AX$$
The covariance matrix of the transformed correlated variables $X'$ is given by:
$$\Sigma' = A A^T$$
For a rotation by an angle $\theta$, and scaling for adding individual standard deviations, the transform is  (sign is same as that given):
$$A =  \begin{bmatrix}cos(\theta) & sin(\theta) \\ -sin(\theta) & cos(\theta)\end{bmatrix} \begin{bmatrix}\sigma_1 & 0 \\ 0 & \sigma_2\end{bmatrix}$$
This produces the covariance matrix $\Sigma'$ as:
$$\Sigma' = \begin{bmatrix}\sigma_1cos(\theta) & \sigma_2sin(\theta) \\ -\sigma_1sin(\theta) & \sigma_2cos(\theta)\end{bmatrix} \begin{bmatrix}\sigma_1cos(\theta) & -\sigma_1sin(\theta) \\ \sigma_2sin(\theta) & \sigma_2cos(\theta)\end{bmatrix}$$
The cross-variance component is given by:
$$\Sigma'_{12} = \sigma_1\sigma_2(cos^2(\theta)-sin^2(\theta))+(\sigma_2^2-\sigma_1^2)cos(\theta)sin(\theta)$$
This is the relation of cross-variance on angle $\theta$. Setting $\theta = 45^\circ$ we get:
$$\Sigma'_{12} = \frac{\sigma_2^2-\sigma_1^2}{2}$$
Setting $\sigma_1 = 0.333$ and $\sigma_2 = 1$ then $\Sigma'_{12} = 0.445$
Note $n = \frac{1}{tan(\theta)} \ne \Sigma'_{12}$ does not represent the covariance component as shown by analysis above.
PCA decomposes the covariance matrix into un-correlated components. In other words it undoes the rotation (the scaling remains), since the rotation correlated the components in the first place.
Explicitly PCA decomposes the covariance matrix into:
$$\Sigma' = U S U^T$$
where $U$ is an orthogonal matrix of eigen-vectors and $S$ is a diagonal matrix which represents individual variances.
Going back to $\Sigma' = A A^T$ and representing the rotation by $R$ we have:
$$\Sigma' = R \begin{bmatrix}\sigma_1^2 & 0 \\ 0 & \sigma_2^2\end{bmatrix} R^T$$
Thus the orthogonal matrix of eigen-vectors $U$ corresponds to the (orthogonal) rotation matrix $R$.
Thus the eigen-vectors should (precisely in theory, approximately in practise) correspond to the rotation matrix (up to some sign factor and permutation which remains arbitrary).
For further info see:

Bivariate normal distribution
Understanding the Covariance Matrix
Interesting Properties of the Covariance Matrix


Including column of indices as predictor for model? I have a small dataset consisting of 1000 observations (rows), 11 predictors + 1 response (12 columns). It is a toy dataset used for learning purposes in a machine learning class at university -- binary classification (heart disease vs. no heart disease).
I have fitted a logistic regression, SVM, and KNN.
However, I am using python and I was deleting some rows corresponding to missing values and outliers I did not want to consider -- doing it with df.drop() and then reset_index(). If you do not set the drop flag of reset_index to true you will get a column in your data frame with the indices you wanted to reset. I accidentally forgot to set the flag to true and when fitting the 3 mentioned models included the column with these indices. This gave me an f1-score of 99-100%. When I realized that I had accidentally included this column, I dropped the column, re-fit and achieved ~ 60-80 percent F1 score with the respective models.
Is there a reason why including this column of indices gets you to 100% F1 score or even accuracy? Is this merely a coincidence that apparently works with the dataset?

Including column of indices as predictor for model?

No, don't include indices. They don't provide any meaningful information about the problem. They are just an enumeration. Take them away.

Is there a reason why including this column of indices gets you to 100% F1 score or even accuracy?

You probably get that when evaluating in train set, in test you wont get that. If you do, then you have a problem with your data (it might be sorted).
What does the underlying algorithm do here? Something like this for a decision tree
if index == 1:
 predict =1
if index == 2:
 predict =0
.
.
.

As you can see there is no meaningful information about the problem here and the prediction in train will still be zero.
Conclusion: don't use indexes as a training feature.


Convert CSV from an api to a dataframe format in python I am new to python, I have extracted some reviews from a website and I used the api  of the webscrapping tool to import my data in python and the format is in csv. I want to convert this csv to a dataframe in python. Can someone guide me on how to perform this please.

Below is the code for importing the api extraction in csv format.

import requests

params = {
  "api_key": "abc",
  "format": "csv"
}
r = requests.get('https://www.parsehub.com/api/v2/runs/ttx8PT-EL6Rf/data', params=params)
print(r.text)

My output for the above codes are as follows:

"selection1_name","selection1_url","selection1_CommentID_name","selection1_CommentID_Date","selection1_CommentID_comment"
"A","https://www..html","137","February 02, 2020","I enjoy the daily package from the start with the welcoming up to the end.
I recommend this hotel."
"A","https://www.e a lot. Relaxing moments with birds chirping, different swings to chill. Overall, I shall visit again. Thanks Azuri & Marideal."
"A","https://www.html","17","June 12, 2019","Had an amazing stay for 2 nights.
The cleanliness of the room is faultless"
"B","https://www.html","133","April 16, 2019","Had a good time. Food is good."

etc...
Can you please help me to convert this into a dataframe in python please please.

Try the following code:

import requests
import pandas as pd
import io

params = {
  "api_key": "abc",
  "format": "csv"
}
r = requests.get('https://www.parsehub.com/api/v2/runs/ttx8PT-EL6Rf/data', params=params)
r = r.content
rawData = pd.read_csv(io.StringIO(r.decode('utf-8')))


Training accuracy decreases I have a program in which I use sequence to sequence approach as a prediction model  with attention. The problem is, while training, the accuracy is always decreasing at each epoch, like shown in the figure attached.

If anyone can tell me the reason, I will be soo thankful.

The screenshot looks like output of Keras, so I assume you're using a deep neural network. The training (epoch) is organized with batches of data, so that optimization function is calculated within subset of whole dataset. The console output shows the accuracy of the full dataset, so the optimization of a single batch can decrease the accuracy of the other part of the dataset and decrease the global result.

The other possible reason is that you provide test set to Keras API and the console output is result of the test set that is not used for training of the model. If it's true you can see that your model is overfitting.


Finding similarity between two datasets I have two datasets. One is actual percentage of white population in counties in an american state and the other is the simulated percentage of white population in counties in an american state.

Bits about my simulation:

  It is a random simulation done on California map with two different
  agents, white and minority. Their total population is based on the
  real white to minority ratio in California. For example if there is
  70% white and 30% minority in California then the agents(say total
  100) would be 70 white and 30 minority. First the map is randomly
  populated with both the agents and then around 100 iteration of the
  simulation is performed. On every iteration agent moves based on
  certain conditions. Data is taken after the 100th iteration and it
  includes what percentage of the white and minority are in a certain
  county in California.

So below is the data from the state of California

california_actual_white = [0.52, 0.72, 0.9, 0.86, 0.91, 0.91, 0.67, 0.79, 0.89, 0.77, 0.89, 0.84, 0.9, 0.81, 0.82, 0.81, 0.87, 0.82, 0.71, 0.86, 0.86, 0.9, 0.86, 0.82, 0.89, 0.91, 0.82, 0.84, 0.93, 0.72, 0.85, 0.91, 0.8, 0.64, 0.88, 0.77, 0.76, 0.54, 0.67, 0.89, 0.61, 0.85, 0.55, 0.87, 0.88, 0.94, 0.87, 0.61, 0.87, 0.83, 0.73, 0.9, 0.88, 0.88, 0.9, 0.84, 0.75, 0.79]

california_simulated_white = [0.48, 0.54, 0.6, 0.62, 0.66, 0.69, 0.71, 0.71, 0.71, 0.72, 0.74, 0.75, 0.77, 0.78, 0.79, 0.79, 0.8, 0.8, 0.8, 0.81, 0.81, 0.82, 0.82, 0.82, 0.83, 0.84, 0.85, 0.85, 0.87, 0.87, 0.87, 0.88, 0.91, 0.92, 0.93, 0.93, 0.94, 0.94, 0.94, 0.94, 0.95, 0.95, 0.97, 0.97, 0.98, 0.98, 0.98, 0.98, 0.99, 0.99, 0.99, 0.99, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]

How would I found a metric of similarity between these two datasets.

I found out a these three option can be used to find similarity and also all of them have a method in Python:

  1) Earth mover's distance

  2) Kullback–Leibler divergence

  3) Cosine Similarity

But I have some doubts using these methods. They are

  1) With Kullback-Leibler divergence and Cosine Similarity, the value
  changes if i reshuffle both the arrays and compute both the metrics
  again but with Earth movers distance its not the case. It will give
  you the same value for the two datasets regarding the
  reshuffling/position of the data points which made me think to use
  this metric.

  2) But the second doubt is that you can use the K-L divergence and
  Earth movers distance only with two probability distribution. So i am
  not sure here that if the two datasets above are the probability
  distribution or not.

So my final two questions are -

1) Are the above datasets considered as probability distribution? If yes why?

2) If 1) is yes then what is the best method to determine the similarity. If 1) is no then what is the best method to determine the similarity ?

Before I was leaning towards Earth Movers Distance as I stated the reason in the doubts section.

I would say that they are probability distributions. You can interpret them as the probability that a randomly drawn person from a given county belongs to the white majority. However, they are not probability vectors, as this would require them to sum up to 1. It's more that each element of the array defines a distribution, e.g a random person from county 1 is white with a probability of 0.52 and a member of a minority with probability 1-0.52=0.48. If you want to compare this distribution with your simulation, you would have to make the comparison element-wise, e.g. compare the 0.52/0.48 actual distribution with the 0.48/0.52 simulated distribution. This is also why reshuffling leads to different results: suddenly you are comparing the actual distribution from county 1 with e.g. the simulated distribution from county 5. This does not make much sense. So if you shuffle, you need to shuffle both arrays in the same way.

As you can interpret the distributions in each of the counties as being independen from each other, you can compute DKL or cosine or Earth's Movers for each county and sum them up probably.

I'm not sure which method is the best. I would probably go with DKL out of habit. You could compute it as such

dkl = 0.52*log(0.52/0.48) + (1-0.52)*log((1-0.52)/(1-0.48)) + ...

I hope this helps!


Linear regression with a fixed intercept and everything is in log I have a set of values for a surface (in pixels) that becomes bigger over time (exponentially). The surface consists of cells that divide over time. After doing some modelling, I came up with the following formula:

$$S(t)=S_{initial}2^{t/a_d},$$

where $a_d$ is the age at which the cell divides. $S_{initial}$ is known. I am trying to estimate $a_d$. I simply tried the $\chi^2$ test:

# Range of ages of division.
a_range = np.linspace(1, 500, 100)

# Set up an empty vector to store the chi squared value
chi_sq = np.zeros(len(a_range))

# Iteration through division ages
for i in range(len(a_range)):
    # Compute the expected value at each time point.
    expect = cell_area[0] * (2**(time_range/a_range[i]))

    # Compute chi squared
    chi_sq[i] = np.sum((cell_area - expect)**2)

# Plot chi squared test
plt.plot(a_range, chi_sq, '.')
plt.yscale('log')

# Labelling
plt.xlabel('division age [min]')
_ = plt.ylabel('$\chi^2$')

but the minimum is always at the upper bound of the age range, depending on what I set this range to be. It doesn't seem right. So I linearized the model:

$$ln(S(t))=ln(S_{initial})+(1/a_d)ln(2)*t,$$

which is now just simple linear regression with a fixed intercept.

Questions:

Why didn't the first method work?
Are there any resources on how to implement the above regression in Python? I'm new to this and everything I found was very simple, but I don't know how to deal with logs and fix the intercept.

Chi squared test does not serve any purpose. The nonlinear equations/functions can be handled by transforming them  in linear functions. The linear model can be used once we transform the nonlinear data/relations into linear format. chi squared test checks for variability. You seem to be interested in sum total of surface (area) i.e. linear model and not a linear regression.


AttributeError: 'numpy.ndarray' object has no attribute 'img_to_array' from keras.models import load_model
from keras.preprocessing import image
import matplotlib.pyplot as plt
import numpy as np
import math
import cv2
import numpy as np
import matplotlib.pyplot as plt
from keras.models import model_from_json
import json

img_width, img_height = 150, 150
with open('model_architecture.json', 'r') as f:
    model = model_from_json(f.read())

model.load_weights('second_try.h5')
model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy'])
image = cv2.imread('/home/hduser/Desktop/IITDPalmprint/training_data/class1/002_5.bmp', 0).astype(np.float32) / 255
plt.title('image')

kernel = cv2.getGaborKernel((121, 121), 4.0, 0, 10.0, 0.5, 0, cv2.CV_32F)
kernel /= math.sqrt((kernel * kernel).sum())
img1 = cv2.filter2D(image, -1, kernel)
x = image.img_to_array(img1)

You are using the variable name image in your code, while you also import a module named image. This conflict results in trying to find img_to_array method in the variable rather than in the module.

Use a different name for you image variable, or directly import the img_to_array function:

from keras.preprocessing.image import img_to_array
[...]
x = img_to_array(img1)


One-Class discriminatory classification with imbalanced, heterogenous Negative background? I'm working on improving an existing supervised classifier, for classifying {protein} sequences as belonging to a specific class (Neuropeptide hormone precursors), or not.

There are about 1,150 known "positives", against a background of about 13 million protein sequences ("Unknown/poorly annotated background"), or about 100,000 reviewed, relevant proteins, annotated with a variety of properties (but very few annotated in an explicitly "negative" way).

My previous implementation looked at this as a binary classification problem:
Positive set = Proteins marked as Neuropeptides.
Negative set: Random sampling of 1,300 samples (total) from among the remaining proteins of a roughly similar length-wise distribution.

That worked, but I want to greatly improve the machines discriminatory abilities (Currently, it's at about 83-86% in terms of accuracy, AUC, F1, measured by CV, on multiple randomly sampled negative sets).

My thoughts were to:
1) Make this a multiclass problem, choosing 2-3 different classes of protein that will definetly be negatives, by their properties/functional class, along with (maybe) another randomly sampled set.
 (Priority here would be negative sets that are similar in their characteristics/features to the positive set, while still having defining characteristics) .
2) One class learning - Would be nice, but as I understand it, it's meant just for anomaly detection, and has poorer performance than discriminatory approaches.

*) I've heard of P-U learning, which sounds neat, but I'm a programming N00b, and I don't know of any existing implementations for it. (In Python/sci-kit learn).

So, does approach 1 make sense in a theoretical POV? Is there a best way to make multiple negative sets? (I could also simply use a massive [50K] pick of the "negative" proteins, but they're all very very different from each other, so I don't know how well the classifier would handle them as one big , unbalanced mix).
Thanks!

One class learning

I wouldn't be too quick to throw out one-class classification methods (option 2) - the key is to model the positive (minority) class with the one-class model.

There has been research demonstrating cases where one-class classification out-performed other approaches like sampling for highly imbalanced data as often seen with protein classification tasks.

I couldn't find the research I recalled, but I did find some other comparisons, showing using one-class classifiers (typically modeling the minority class) achieved as good or better performance than binary classification typically with sampled "negatives" from the large set of proteins not known to be positive.

Additionally this approach also gives the advantage of much improved run-time - since you only need to train the classifier on the smaller, positive set.  A couple papers:

"Prediction of protein-protein interactions using one-class classification methods and integrating diverse biological data"

"A One-Class Classification Approach for Protein Sequences and Structures"

At the very least I would try some one-class methods and compare the performance using validation with your binary/multi-class classification approaches.  There are also open source implementations for many of these so it shouldn't be too costly to try them out, for example LibSVM has a one-class SVM implementation.  Additionally, it might prove valuable for use in an ensemble with binary classifiers, since there may be more disagreement in their predictions.

Higher level representation embedding / clustering

Along the lines of what you were thinking with (1) and the other post suggesting PCA, approaches like clustering, sparse coding, or even topic modeling - treating each protein as a document string and different protein families as different topics - could yield a representation that might make classifying the proteins straightforward.

I.e., you could identify which group/cluster a protein belongs to or classify the cluster memberships / embedded representations.

E.g., such embedding approaches as sparse coding can yield representations that reveal which cluster a protein belongs too - so that some sets of features are only active (non-zero) for proteins in the same cluster - which can make classifying them much easier.

Additionally class labels or known cluster membership can be incorporated in the embedding process for most methods.

Ensemble

Ensembles of multiple classifiers tend to work best - especially when the classifiers are very diverse and can achieve comparable performance individually.

There are at least two ways use ensembles for this problem.

You can build an ensemble of binary classifiers by sampling multiple different same-size negative sets and training a classifier on each.
You can build an ensemble from different approaches, such as binary classifiers with different negative samples, combined with a one-class classification approach, combined with classification models trained on the embedded data.


Is there a problem of over fitting in my dataset? I have applied the sequential forward selection to my dataset having 214 samples and 515 features (2 class problem). The feature selection algorithm has selected 8 features. Now I have applied the svm (MATLAB) on these 8 features. I have also tried to see the performance after adding more features. The table given below gives the correct rate of the algorithm (training data set) along with the feature set used. The result obtained is:
8 features = 0.9392
10 features = 0.9439
12 features = 0.9672
14 features = 0.9672
16 features = 0.9626
18 features = 0.9766
20 features = 0.9672

As visible, the accuracy seems to increase. Is it because of over fitting? Should I use the default feature set as given by the sequentialfs function of Matlab or should I force it to deliver more features to get more accuracy?
I have uploaded the validation training and testing performance (70-15-15). Now can you tell me if my data is being over-fitted or not?

It is not possible to tell whether a machine learning algorithm is overfitting based purely on the training set accuracy.

You could be right, that using more features with a small data set increases sampling error and reduces the generalisation of the SVM model you are building. It is a valid concern, but you cannot say that for sure with only this worry and the training accuracy to look at.

The usual solution to this is to keep some data aside to test your model. When you see a high training accuracy, but a low test accuracy, that is a classic sign of over-fitting.

Often you are searching for the best hyper-parameters to your model. In your case you are trying to discover the best number of features to use. When you start to do that, you will need to make multiple tests in order to pick the best hyper-parameter values. At that point, a single test set becomes weaker measure of true generalisation (because you have had several attempts and picked best value - just by selection process you will tend to over-estimate the generalisation). So it is common practice to split the data three ways - training set, cross-validation set and test set. The cross-validation set is used to check accuracy as you change the parameters of your model, you pick the best results and then finally use the test set to measure accuracy of your best model. A common split ratio for this purpose is 60/20/20.

Taking a pragmatic approach when using the train/cv/test split, it matters less that you are over or under fitting than simply getting the best result you can with your data and model class. You can use the feedback on whether you are over-fitting (high training accuracy, low cv accuracy) in order to change model parameters  - increase regularisation when you are over-fitting for example.

When there are a small number of examples, as in your case, then the cv accuracy measure is going to vary a lot depending on which items are in the cv set. This makes it hard to pick best hyper-params, because it may just be noise in the data that makes one choice better than another. To reduce the impact of this, you can use k-fold cross-validation - splitting your train/cv data multiple times and taking an average measure of the accuracy (or whatever metric you want to maximise).

In your confusion matrices, there is no evidence of over-fitting. A training accuracy of 100%* and testing accuracy of 93.8% are suggestive of some degree of over-fit, but the sample size is too low to read anything into it. You should bear in mind that balance between over- and under- fit is very narrow and most models will do one or the other to some degree.

* A training accuracy of 100% is nearly always suggestive of overfit. Matched with e.g. 99% test accuracy, you may not be too concerned. The question is at worst "could I do better by increasing regularisation a little"? However, matched with ~60% test accuracy it is clear you have actually overfit - even then you might be forced to accept the situation if that's the best you could achieve after trying many different hyperparameter values (including some attempts with increased regularisation).


Spark, optimally splitting a single RDD into two I have a large dataset that I need to split into groups according to specific parameters.  I want the job to process as efficiently as possible.  I can envision two ways of doing so

Option 1 - Create map from original RDD and filter

def customMapper(record):
    if passesSomeTest(record):
        return (1,record)
    else:
        return (0,record)

mappedRdd = rddIn.map(lambda x: customMapper(x))
rdd0 = mappedRdd.filter(lambda x: x[0]==0).cache()
rdd1 = mappedRdd.filter(lambda x: x[1]==1).cache()

Option 2 - Filter original RDD directly

def customFilter(record):
    return passesSomeTest(record)

rdd0 = rddIn.filter(lambda x: customFilter(x)==False).cache()
rdd1 = rddIn.filter(customFilter).cache()

The fist method has to itterate over all the records of the original data set 3 times, where the second only has to do so twice, under normal circumstances, however, spark does some behind the scenes graph building, so I could imagine that they are effectively done in the same way.  My questions are:
a.) Is one method more efficient than the other, or does the spark graph building make them equivalent
b.) Is it possible to do this split in a single pass

First of all let me tell you that I'm not a Spark expert; I've been using it quite a lot in the last few months, and I believe I now understand it, but I may be wrong.

So, answering your questions:

a.) they are equivalent, but not in the way you're seeing it; Spark will not optimize the graph if you are wondering, but the customMapper will still be executed twice in both cases; this is due to the fact that for spark, rdd1 and rdd2 are two completely different RDDs, and it will build the transformation graph bottom-up starting from the leafs; so Option 1 will translate to:

rdd0 = rddIn.map(lambda x: customMapper(x)).filter(lambda x: x[0]==0).cache()
rdd1 = rddIn.map(lambda x: customMapper(x)).filter(lambda x: x[0]==1).cache()

As you said, customMapper is executed twice (moreover, also rddIn will be read twice, which means that if it comes from a database, it may be even slower).

b.) there is a way, you just have to move cache() in the right place:

mappedRdd = rddIn.map(lambda x: customMapper(x)).cache()
rdd0 = mappedRdd.filter(lambda x: x[0]==0)
rdd1 = mappedRdd.filter(lambda x: x[0]==1)

By doing this, we are telling spark that it can store the partial results of mappedRdd; it will then use these partial results both for rdd1 and rdd2. From the spark point of view this is equivalent to:

mappedRdd = rddIn.map(lambda x: customMapper(x)).saveAsObjectFile('..')
# forget about everything
rdd0 = sc.objectFile('..').filter(lambda x: x[0]==0)
rdd1 = sc.objectFile('..').filter(lambda x: x[0]==1)


How to interpret feature importance (XGBoost) in this case? I found two dominant features from plot_importance.  My dependent variable Y is customer retention (whether or not the customer will retain, 1=yes, 0=no). My problem is I know that feature A and B are significant, but I don't know how to interpret and report them in words because I can't tell if they have a position or negative effect on the customer retention. Is there a way to find that out or anything that helps make it clear?

Thanks.

I think you can find the correlation matrix for the feature which could provide you with evidence to justify your hypothesis.


unhashable type: 'numpy.ndarray' classes = {0: 'Other', 1: 'Thunder'}

    x_new = p.read_csv('LeeAnnDataSet.csv', sep=',').values

    y_predict = train_test_split(x_new)

    print(classes[y_predict[0]])

    print(classes[y_predict[1]])

**Every time I try using this command, I receive the error message below. I don't understand what I may be doing wrong. **
    TypeError                                 Traceback (most recent call last)
    <ipython-input-66-5abff76d706f> in <module>

    4 y_predict = train_test_split(x_new)

    5

    ----> 6 print(classes[y_predict[0]])

    7 print(classes[y_predict[1]])

    TypeError: unhashable type: 'numpy.ndarray'

The error is caused by passing a numpy array into a function that expects an integer value. read_csv() will read a file, and create a numpy array from the data inside. You can slice off the column of the numpy array that you want to use, convert it to a list and then pass this one by one into classes[]
# this should work, but change the value of 12 to the column you want to convert to a class label
import pandas as p
import numpy as np
from sklearn.model_selection import train_test_split

classes = {0: 'Other', 1: 'Thunder'}

x_new = p.read_csv('train.csv', sep=',').values # x_new: <class 'numpy.ndarray'>
y_predict = train_test_split(x_new) # y_predict: <class 'list'>

z = np.asarray(y_predict)[0].T  # transpose your y_predict dataset
for i in range(0,len(z)):
    print(z[12][i])
    print(classes[z[12][i]])

Issues with the datatypes in your original code:
import pandas as p
import numpy as np
from sklearn.model_selection import train_test_split

classes = {0: 'Other', 1: 'Thunder'}

x_new = p.read_csv('train.csv', sep=',').values
print('x_new:',type(x_new))
y_predict = train_test_split(x_new)
print('y_predict:',type(y_predict))
print('y_predict[0]',type(y_predict[0]),y_predict[0])
print(classes[y_predict[0]])
print(classes[y_predict[1]])

x_new: <class 'numpy.ndarray'>
y_predict: <class 'list'>
y_predict[0] <class 'numpy.ndarray'>

[[2078 90.59 'y' ... 0 0 0]
[7384 106.47 'x' ... 0 0 0]
[2860 115.75 'ak' ... 0 0 0]
...
[7897 87.76 'f' ... 0 0 0]
[5029 87.16 'f' ... 0 0 0]
[2414 99.16 'aj' ... 0 0 0]]

    TypeError                                 Traceback (most recent call last)
    <ipython-input-57-ea39b293dd61> in <module>
         10 print('y_predict:',type(y_predict))
         11 print('y_predict[0]',type(y_predict[0]),y_predict[0])
    ---> 12 print(classes[y_predict[0]])
         13 print(classes[y_predict[1]])

    TypeError: unhashable type: 'numpy.ndarray'

To give you an idea of what is expected, here is an example that works:
import pandas as p
import numpy as np
from sklearn.model_selection import train_test_split

classes = {0: 'Other', 1: 'Thunder'}

y_predict = [1,1,0,0]
print(type(y_predict))
print(type(y_predict[0]))
print(classes[1])
print(classes[y_predict[0]])

<class 'list'>
<class 'int'>
Thunder
Thunder


Is this a good classified model based confusion matrix and classification report? I build a classification model. The results below when implemented Decision Tree Classifier.

Prediction accuracy

0.785813630042

confusion matrix

[[ 1302  1581]

[ 2577 13953]]

         precision    recall  f1-score   support

    2.0       0.34      0.45      0.39      2883
    3.0       0.90      0.84      0.87     16530

avg / total       0.81      0.79      0.80     19413

For the Neural Network---

Prediction accuracy

0.863132952145

Confusion Matrix

[[  718  2165]

[  492 16038]]

         precision    recall  f1-score   support

    2.0       0.59      0.25      0.35      2883
    3.0       0.88      0.97      0.92     16530

avg / total       0.84      0.86      0.84     19413

For the Gradient Boosting Classifier---

Prediction accuracy

0.870035543193

Confusion Matrix

[[  971  1912]

[  611 15919]]

         precision    recall  f1-score   support

    2.0       0.61      0.34      0.43      2883
    3.0       0.89      0.96      0.93     16530

avg / total       0.85      0.87      0.85     19413

Gradient boosting classifier gives me good accuracy score but it gives bad recall for class 2 compared with Decision Tree classifier.
Train Data for both classes:

Class 2: 2908

Class 3: 16673

Is recall depend on training data? How do I improve recall for class 2 along with accuracy?

The question 'is this a good model?' is essentially a business question.
There is always a precision/recall tradeoff and you decide based on your business goal what model to choose.

Data science can help you with the question on how to compare classifiers.

Let us start, that each (or nearly each) classifier can predict not only the outcome (Class2 or Class3), but also the probability of this outcome.

Your confusion matrix and all your metrics are suitable for making business decision, but, from data science point of view, have two problems

1) they do not take into account the agreement by chance

2) they are based on the threshold probability 0.5, which is not necessarily the optimal probability for deciding about the class.

To deal with (1), have a look at Cohen's kappa

To deal with (2), use metrics based on probability. The simpler one is Area Under the ROC Curve, see also here. Even finer, but not so straightforward to interpret is Logarithmic Loss

  Is recall depend on training data? How do I improve recall for class 2 along with accuracy?

Recall depends on the quality of your model (and thus also on the quality of your training data) and on the chosen probability threshold. By decreasing probability threshold, you improve recall but worsen precision. Which tradeoff to choose is again a business decision.


where can i find the algorithm of these papers? I am reading about clinical NER
I found 2 papers talking about it
Paper 1
and
Paper 2
They are talking about algorithms and ML has been used to approach clinical NER.
I could not find anywhere on how exactly these algorithms are implemented in these papers.
Can anyone help me find it or build it?

You can email the authors to ask them if they could share their code with you, but maybe they can't for IP reasons or don't want to share it.

Papers like these are not unusual in experimental research. In theory you should be able to reproduce their system following the explanations in the paper.

However there are other tools available for biomedical NER: MetaMap, cTakes.


Average of importance gain for a categorical variable Suppose I have a set of M categorical variables, some of them with a different number of categories (for instance, var1 has five categories, var2 has three, etc).
I train an XGBoost model on a numeric target Y after having performed one-hot encoding on the M categorical variables, thus creating a set of dummy inputs.

When looking at the model results, I get a table of importance gain for the categories of each feature, meaning how important they are in the model. A toy result would look like this:

feature | category               gain
       var1 | cat3                 25
       var2 | cat1                 20
       var1 | cat5                 12
       var5 | cat6                 11
       var4 | cat1                  8
           ...                    ...

The main question I'm asking is the following:

In order to get an idea of how important a variable is overall rather than just one of its categories (for instance, how much var1 is important overall rather than just category cat3 of var1), does it make sense to take the average of all the importance gains for each feature as an importance indicator?

Probably the sum of such gains would not be correct as the features may have a different number of categories, but I'm wondering if the average of such gains might serve as an indicator of the importance of a particular feature overall.

I already looked at some questions like this without gaining much insight about this topic.

I think you are looking for information gain.

The way you would compute it for 1 variable is:

Let's say that you label variable is binary.

1) Compute percentages of label per category, for example you have three categories: "US", "UK", "Ger", if there are 5 labels that are marked 1, and 3 of the area associated with "UK", 2 with the "US" and 0 with "Ger", your percentages would be: [3/5, 2/5, 0/5].

2) Do step 1) for every label and aggregate

3) Calculate entropy for every percentage

4) Information Gain = Entropy of aggregated percentage - sum of weighted percentages. Weight is just number of instances in a label / total number. For example if we had labels = [1,1,0,0,1,1,0,0,1], entropy of percentages of label 1 would be weighted by 5/9 and entropy of percentages of label 0 would be weighted by 4/9.

Then you compute IG for every variable, and compare!

As a side note, if you are working with a lot of categorical variables, you might want to look into LightGBM or CatBoost, these algorithms allow you to specify categorical variables without creating one hot encoded vectors, and they provide feature importance on these variables.


How to prepare the training data for SVD-based Recommendation? I am trying to build an SVD-based recommender system. According to my understanding, the training data should only contain the users who buy at least m items and the items which are bought by n unique users. Otherwise, if I use all users and items (including low-frequency users and items), I think the training data will be noisy.

However, here is the problem: To build the training data for SVD, I first retrieve all users who buy at least m items from the database. But there are some items bought by these users are not bought by at least n users. But after further filtering out these low-frequency items from the training data, some of the remaining users will not have item sets which contain m items because some items are deleted.

I feel that I am not on the right track. How am I supposed to build the training data for SVD?

I consider this answer as a general approach for the data preparation for CF algorithms and based on my experience. I do not know the specific case of Your data.

While preparing the data for CF I would take care about those things:

dataset size and sparsity - there are many CF algorithms and they are giving different results depending on dataset size (users and items) and its data density. Simple intuition: memory-based KNN methods will be suffering from data sparsity while on the same setup the MF can perform better. Good comparison is presented in this article: A Comparative Study of Collaborative Filtering Algorithms. Moreover, CF methods can depends on the number of users and items differently. To reduce sparsity, You removing items/users which have lower number of ratings/click than selected threshold. This is iterative process. As it's said in the question, when You remove particular item/user it can happen that different user/item respectively can be below the threshold. For research purposes, I even have prepared the function which calculate items/user threshold in order to receive the given sparsity.
outliers in the meaning of filtering unusual users and items. Those, which can influence Your model greatly, but in a wrong way. In my cases (e-commerce website) it were cases like: really heavy users - other shops, bots (scrapping scripts) that was collecting data, internal traffic (blacklist IPs), bidding automation tools, funny offers which lands on a digg-like webpages. Someone can say that this can be removed by user/item bias. Yes, as soon as Your CF method is using it :-)
correct slicing of the dataset. For some reasons (e.g. computational) You have to divide You data on smaller parts. And this is reasonable. Divide and conquer approach. But in such cases before doing this, check if You are not loosing a lot of the information by such slicing. Here is the example: we are slicing offers by categories, categories are formed into a category tree, for this reason the same leaf can be part of few main categories. Thus, slicing by main categories can result in having only a part of the whole user-item interactions about viewed/bought together items in this leaf category.

Those are only few tips from my experience.
But I hope this would help.

I would appreciate any edits and sharing the results :-)


Loss is bad, but accuracy increases? I have a multicategorial classification problem for images. There are 5 (imbalanced) classes for which i use different class weights. In general there are only a few training images per class: ~56-238

To classify them, I use a neural network with much data augmentation. I have a validation set with the same distribution as the train set (but it only has about 30% of the images per class).

The resulting loss / accuracy graphs look a bit weird (edit: the second graph contains the term "Test Loss", but it is the "Validation Loss"):

I'm not sure how I can interpret these two images: The validation accuracy cleary increases, but the validation loss doesn't change too much. Can anyone help me with the interpretetion of these graphs?

Thank you very much

If you have an imbalanced dataset, you may want to take steps to resample. When evaluating your results of your algorithm, it may be a good idea to choose a metric that is better suited for class imbalance problems.

Accuracy is not a good metric for evaluating the performance of class imbalance problems because accuracy is rewarded for predicting the most commonly occurring label. Accuracy is the measurement of the the number of labels predicted correctly over the number of all the observations. For example, if you had 99% of the data with label 'a' and 1% of the data with label 'b', and you then labeled the entire dataset with label 'a', your accuracy would be 99%. However, your ability to label 'b' correctly was 0%. Log loss is also not particularly good for class imbalance problems. You may want to consider weighted log loss instead.

Here are some other metrics that you can investigate.

Precision and Recall

Precision is the ratio of correctly predicted positive labels to the total predicted positive observations. Recall is the ratio of correctly predicted positive observations to the all observations in the given class. You can read more about these metrics here. For a Binary Classification problem, you can view the predicted class vs the actual class in a confusion matrix.

F1 Score

F1 Score is the weighted average of precision and recall. F1 score takes false negatives and false positives into account. So, it is more informative than accuracy when evaluating class imbalance problems.

F1 Score exists as a Python package in sklearn. It also exists as a function in R.

Area Under Precision Recall Curve

You can also investigate Area under precision recall curve (PR), which the measurement under the area of precision recall curve plot. It can be used to evaluate large class imbalance problems. PR Curve exists as a python package in sklearn and an R package.


Pandas dataframe groupby and then sum multi-columns sperately I have a pandas dataframe as following:

df = pd.DataFrame({
    'Date': ['2017-1-1', '2017-1-1', '2017-1-2', '2017-1-2', '2017-1-3'],
    'Groups': ['one', 'one', 'one', 'two', 'two'],
    'data1': range(1, 6),
    'data2': range(10, 15})

Output:

    Date      Groups     data1    data2
0  2017-1-1    one       1        10
1  2017-1-1    one       2        11
2  2017-1-2    one       3        12
3  2017-1-2    two       4        13
4  2017-1-3    two       5        15

I would like the output to look like this:

    Date      Groups     sum of data1    sum of data2
0  2017-1-1    one       6                33
1  2017-1-2    two       9                28

I can groupby "Group" and agg.(sum) either data columns, but couldn't do 2 simultaneously. How can I do this?

Thanks.

IIUC:

In [5]: (df.groupby("Groups", as_index=False)
           .agg({"Date":"first", "data1":"sum", "data2":"sum"}))
Out[5]:
  Groups      Date  data1  data2
0    one  2017-1-1      6     33
1    two  2017-1-2      9     27


AUC ROC metric on a Kaggle competition I am trying to learn data modeling by working on a dataset from Kaggle competition. As the competition was closed 2 years back, I am asking my question here. The competition uses AUC-ROC as the evaluation metric. This is a classification problem with 5 labels. I am modeling it as 5 independent binary classification problems. Interestingly, the data is highly imbalanced across labels. In one case, there is an imbalance of 333:1. I did some research into interpreting the AUC-ROC metric. During my research, I found this and this. Both these articles basically say that AUC-ROC is not a good metric for an imbalanced data set. So, I am wondering why would they be using this metric to evaluate models in the competition? Is it even a reasonable metric in such a context? If yes, why?

As you would have seen in the research, AUC ROC prioritizes getting the order of the predictions correct, rather than approximating the true frequencies.

Usually, like in the credit card fraud problem you link to, the impact of one or two false negative is more devastating that many false positives. If those classes are imbalanced, like they are in the fraud case, AUC ROC is a bad idea.

It appears that in the competition you are referring to, the hosts are more interested in labeling which comments are more toxic than others rather than rating how toxic they each are. This makes sense since in reality the labels are subjective.


Using neural networks with jumps in stock returns I am using an LSTM network to analyse stock return patterns. A problem is that, there is usually huge jumps in stock returns but if you are only using the trading data, the jumps would seem pretty random. (For example, the jumps from SEC ruling against or in favor of a company.)

Thus, if the neural network learns too much from the jumps, the results would not generalize well. One might cap the returns or use auto encoders. What are some other methods to regularize such jumps and limit the changes the jumps cause to the network?

In standard models that try to characterise jumps, such as the Jump Diffusion model from Mertons (a short introduction), the model consists of two main parts:

Brownian motion; a random walk to account for the random path, perhaps with some drift - when the values head upwards or downwards in a consisten manner. And
An additive Possion process, which with some probability add a jump in a time-step, with a given probability.

Drawing from this approach, you could also consider using two models that work independently (or separately) to model the overall market and trend, along with a model that introduces jumps at certain timesteps and itnervals.

You could try using different input data for the second model, such as signals taken from text, such as news feeds or newpapers that discuss current market dynamics/politics, possible decisions from SEC and the like. The first (stable) part could be modelled by your current neural network.

This is just a high level idea, and I haven't actually seen any research that already tried it, so unfortunately cannot provide any links to literature.


Dataset management: What are some strategies/solutions for efficiently storing datasets with their versions? The problem: I've N classification models (independent), for each of these N models, I've different versions (eg: V0, V1, ..., Vfinal_production,Vexperimental). I'm looking for a way to store my datasets efficiently on the cloud (for redudency).

Note: We're not talking about BigData here.

Current Solution: Created a private GitHub repo. Made N directories and inside, pushed different dataset versions as different files.

Are there better solutions for this (because I feel VCS is an overkill for this problem)?

I used my existing directory structure with Git LFS and it works well.

Here are the advantages I got by using Git LFS other than the alternative solutions.

It saves space: Since Git LFS manages my dataset files (100MB to 1GB) with pointers and keeping only the minimum in the local memory.
It works seamlessly with a git enabled repository: I was planning to use some storage solution (like Amazon S3) specific for my datasets. But I would have to make extra efforts to manage multiple version and keep my dataset files in sync.


Meaningful predictive analytics for small (n=114) dataset with just 1 explanatory variable and 1 response variable? I am given an Excel pivot table that aggregates data from a somewhat sizable data source (a database table with 1.9m records and another of about 490k). The data within the Excel file consists of 3 columns: dates of Mondays which represent their respective weeks, quantity of items, and number of shipments (that it takes for the quantity of items). I am supposed to concoct a model that predicts the number of shipments that would be required for a given quantity of items in the future. What models could I implement for such a small dataset with just 1 explanatory and 1 response variable? I know the run-of-the-mill linear regression with a confidence interval would be a start but the data has a dense cluster and then sparse data with some positive correlation. The color bar represents the date (purple is earlier, yellow is most recent)

Create additional features from the given features. From the Monday "date" feature, the features ("month","year") can be found. Additionally, months of the year contain either 4 or 5 Mondays, so a "week" feature can be created with a value 1...5 that can be used to represent the week of the month that the Monday occurred on. Additional features can be created by calculating trends (Average Shipments for the previous N shipments,...etc.). This gives the algorithm more information to work with and should provide more refined results.


is it bad to have many different measurements for the same target variable? I'm working on a dataset that has repeated measurements for the same target variable.

When I don't change anything and create model, cross validation overfits with 0.99 score but in testset it gives around 0.39.

When I use mean, std, skew, quartiles for each measurement to have only one measurement for each feature, it gives a much better score.

Can anyone explain to me why? and when it is good to use the second method?

the original dataset looks like this (all numbers are fake):

id /measurement1/measurement2/.../target/
0-1/0.18283     /0.12855     /.../  1   /
0-2/0.1141      /0.38484     /.../  1   /
0-3/0.4475      /0.18374     /.../  1   /

and transformed dataset looks like this:

id /meas1_avg/meas1_std/meas1_skew/meas2_avg/meas2_std/.../target/
0  /0.28747  /0.183848/ 0.198384  /0.18484  /0.28474  /.../  1   /

Note that you are solving two different problems here.

In the first problem, you want to predict the target variable given one noisy measurement.

In the second problem, you want to predict the target variable given some statistics from a group of noisy measurements.

Your results show that the second problem is easier to solve which is intuitive, since the amount of noise (variance) for average of multiple measurements is less than only one measurement (closely related to Law of Large Numbers), thus the relation in the second problem is easier to find by the model.

Therefore, if both problems are equivalent to you, go with the second problem which is easier to solve.


Is it ok to train the model only on the interested part of the data? Let's say I have a dataset where one feature is 'Car type' : say 'A', 'B' and 'C'.
The test set consists of samples where 'Car type' is always equal to 'A'.
Therefore, should I train my model only on the subset where 'Car type' is 'A' or on the whole training set?
What are the pros and cons of both approaches?

I think it depends on your understanding of the data set.

How similar are car $A$ with car $B$ and car $C$?

Is car $A$ an electric car and car $B$ and car $C$ running on gas? Is one of them a self driving car and the others are not?

If you are training them together, I think there is an implicit assumption that their behavior is similar and you want to take advantage of that, in particular, perhaps you do not have sufficient data from car $A$ and you are hoping that you can use data from car $B$ and car $C$ to help you.

However, if car $B$ and car $C$ are very distinct and you are trying to predict accident or car failure, adding it to them might not help that much. If the design of cars are very distinct, they might not cause accident or car failure due to the same features.


In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it? While reviewing the Transformer architecture, I realized something I didn't expect, which is that :

the positional encoding is summed to the word embeddings
rather than concatenated to it.

  http://jalammar.github.io/images/t/transformer_positional_encoding_example.png

Based on the graphs I have seen wrt what the encoding looks like, that means that :

the first few bits of the embedding are completely unusable by the network because the position encoding will distort them a lot,
while there is also a large amount of positions in the embedding that are only slightly affected by the positional encoding (when you move further towards the end).

  https://www.tensorflow.org/beta/tutorials/text/transformer_files/output_1kLCla68EloE_1.png

So, why not instead have smaller word embeddings (reduce memory usage) and a smaller positional encoding retaining only the most important bits of the encoding, and instead of summing the positional encoding of words keep it concatenated to word embeddings?

When you concatenate, you have to define a priori the size of each vector to be concatenated. This means that, if we were to concatenate the token embedding and the positional embedding, we would have to define two dimensionalities, $d_t$ for the token and $d_p$ for the position, with the total dimensionality $d = d_t + d_p$, so $d>d_t$ and $d>d_p$. We would be decreasing the total size we devote to tokens in favor of positional information.
However, adding them together is potentially a super case of the concatenation: imagine that there is an ideal split of $d$ into $d_t$ and $d_p$ in terms of minimizing the loss; then, the training could converge to position vectors that only take $d_t$ elements, making the rest zero, and the positions were learned and happened the same, taking the complementary $d_p$ elements and leaving the rest to zero.
Therefore, by adding them, we leave the optimization of the use of the $d$ dimensions to the optimization process, instead of assuming there is an optimal partition of the vector components and setting a new hyperparameter to tune. Also, the use of the vector space is not restricted by a hard split in the vector components, but takes the whole representation space.


Is it accurate to say that "K-means clustering the vectors based on keywords weight similarity"? Long story short, I have 200 vectors as a result of TF-IDF (Term Frequency - Inverse Document Frequency) on thousands of keywords in hundreds of vectors. The total number of unique keywords that I got is 745 keywords, meaning that there are 745 dimensions/axes. Now, I was wondering how does K-means clustering work on those 200 vectors? Is it accurate to say that K-Means is clustering those 200 vectors by the keywords weight similarity?

K-means clusters points (in your case represented as 745 dimensional vectors) by their similarity, that is some distance measure between points (usually the Euclidean distance).
TF-IDF produces a vector from a sentence or document, where each entry (axis) represents the frequency of a word divided by the frequency of it's occurrence in all sentences or documents, hence the name. Other weighting scheme are possible as well, see here.
If two vectors are very close to each other, it means that the content of their documents is very similar. Therefore it's likely they end up in the same cluster. In contrast, if two vectors are far away, the words in each document might be completely different or the frequency of words might be different.
So the distance between vectors can be interpreted as measure of similarity between the documents content.


How to derive false positive and false negative from top-k accuracy? I am working on the following "equality identification" problem and become quite confused on how to reasonably define false positive and false negative in my case.
Problem:
Suppose I have a large set of graphs $G$, where for every graph $g \in G$, there is one and only one graph $g'$ that is identical to $g$. I am training a model $M$ which, given two graphs $g, g' \in G$, can decide how similar these two graphs are. The output of $M$, $o =M(g, g')$, denotes a similarity score of these two graphs. $o$ is a floating number, ranging from $[0, 1]$.
To evaluate the performance of $M$, I am measuring the top-$k$ accuracy of its predications in the following very standard way:
for g \in G:
    for t \in G:  // suppose here t != g and therefore comparison is meaningful
        accuracy[(g, t)] = M(g, t)

correct_prediction = 0
total_prediction = 0
for g \in G:
     total_prediction += 1
     sort_accuracy_array_on_graph (g); // we sort all comparisons on $g$ with other graphs \in G
     if g' in top-$k$ most similarity graphs with g:
        // g' is the ground truth, the "identical" graph with g
        correct_prediction += 1

correct_prediction / total_prediction // the average top-$k$ accuracy

OK, so this seems very standard approach to computing the top-$k$ accuracy of $M$. However, currently I want to take one step further and somewhat measure the false positive and false negative of $M$. I understand that FP and FN are both very standard metrics in data mining, but just somewhat become very confused on how to define it in my problem. For instance, I can define False Negative in the following way:

False Negative of $M$: conceptually, FN denotes that $g'$, the identical graph of $g$, does not appear in top-$k$ most similarity graph of $g$. Then, perhaps FN is interchangeable with top-$k$ accuracy in my case?

False Positive of $M$: but if we accept that FN is interchangeable with top-$k$ accuracy in my case, then how to define FP of $M$? FP denotes that $M$ aggressively treats graph $t$, where $t != g'$, as the "similar" graph of $g$ and appear in its top-$k$ most similar set. Then isn't it indicating that FP is ALSO interchangeable with top-$k$ accuracy in my case? That seems suspicious because in my understanding, FP should not equal to FN? There must be something wrong.

Am I clear on this confusion? Any suggestion would be appreciated. Thank you!

I think your confusion comes from the fact that the false negative of $M$ as you thought it, is not top-$k$ accuracy but perhaps $1$ $-$ top-$k$. Moreover, how do you evaluate the condition if g' in top-$k$ most similarity graphs with g?
In any case, for your case I thought the following.
Let's say that if the model $M$ returns a value $o > o^*$, where $o, o^* \in [0, 1]$ (and $o^*$ is a threshold value decided by a human), then we assume that $g \equiv g'$; $g \neq g'$ otherwise.
Thus, a false negative of $M$ is when $o < o^*$ but in reality, $g \equiv g'$.
A false positive of $M$ instead is when $o > o^*$ but in reality, $g \neq g'$.
Of course your result will depend on what value(s) of $o^*$ you choose, but now it should be more clear what are FN and FP.


predicting number of customers based on long/lat coordinates I have a very simple dataset that has as columns the latitude and longitude of bookstores, and in the output column, has the number of customers / year. I've never explicitly deal with longitude/latitude coordinates as predictor variables before, but I was curious if one could use them as predictors to predict the number of customers/year for an arbitrary set of coordinates?

The answer relies on the question: are sales related to location?

There might also be some linear relation between location and sales in the sense that "the further you go away from city centre, the less books people read", which is definitely worth checking by simply running a correlation analysis between longitude,latitude and sales and see it in a matter of seconds using either Matlab or Python (Pandas).

Another thing that you can do, since you have two inputs, you can plot the sales for each location (longitute,latitude) on a 2D colormap (heatmap), where color intensity shows number of sales. Let us assume that light and dark colors correspond to low and high sales respectively. If you see that similar color intensity is gathered in same regions of the map, it means that location actually affects the sales. On the other hand, if you see randomly colored points in a small area, this means that sales are irrelevant to location.


How to compare Random Forest with other models I am new to Machine Learning and I am trying to undrestand the Out of Bag Error in Random Forests and its use.
Let's say that we have a dataset. First we use the whole dataset (without splitting it) to get a Random Forest and its Out of Bag error. Then we split the dataset, train a Neural Network on the training part and test it on the test part of the dataset.
Can I choose between the two models by comparing the Out of Bag error of the random forest with the total test error of the Neural Network ? Does it make sense ?

We generally rely on sampling to validate the result of our Model.
 We make train/test to test a model on a separate unseen dataset.
 If we are doing a Hyperparameter tuning, we keep another set as a validation set to validate the result of new hyperparameters.
Random forest builds each new tree on a Bagged sample from the original sample(train data).
Bagging means sampling is done with replacement i.e. you pick one data point, put it back and then pick the next.
In this process, many duplicated data is sampled and many data points are not sampled.
~63% of the data points are selected Read here
The other "not selected" 37% data points is called Out of Bag samples.
Hence, the way Bagging, and RF is designed, we got another set of data to do our validation. It an opportunity to do a level of validation with these samples.
What it means,

You are getting 37% data points to validate your model
But OOB is not done with the fully grown Ensemble. It is done using all the trees in the random forest ensemble for which the particular data point was omitted during training Read here
It's not equivalent to K-Fold or Train/test on fully built RF but it gives a decent idea about the validation error about to come.


term for data that compares a day to same day of a week/month/year ago What is the technical term for a report or dataset that compares data of an interval of time with that of the same interval in a previous week, month, or year?

A repeated cross-sectional study. Cross-sectional refers to data collected at one specific point in time or over a short period. Repeated since you compare at least two time frames. If there are several time frames, one could talk about time-series cross-sectional (TSCS) data. Repeated cross-sectional studies may become pseudolongitudinal, or longitudinal if carried over a longer lapse of time.


Item Similarity with Location Feature I'm currently learning about Collaborative Learning and Content-based Recommendation.

One of the main things that is discussed in both methods is about calculating similarity between two users or two items. Commonly, similarity between two entity (each represented by a vector) can be calculated using Cosine similarity or other similarity coefficients.

One question that I haven't found the answer is about calculating similarity between two entities in which each has a feature that explains each entity's location.

Let's say, the entities in my data is about restaurants and each restaurant has several attributes such as

food price average
user rating/review score
location in $(\text{lat}, \text{long})$
etc.

For food price average or user rating, maybe it's still relevant to include them in cosine similarity function, but how about location–which is described in a tuple of real numbers? What's the best way to include distance between entities as a component in the similarity function?

Thanks!

Perhaps you could transform the latitude and longitude into spherical coordinates. In this coordinate system, the cosine of the angle between the vectors has a natural geometric interpretation.


How do GD, Batch GD, SGD, and Mini-Batch SGD differ? How do these four types of gradient descent functions differ from each other?

GD
Batch GD
SGD
Mini-Batch SGD

Gradient Descent is an optimization method used to optimize the parameters of a model using the gradient of an objective function ( loss function in NN ). It optimizes the parameters until the value of the loss function is the minimum ( of we've reached the minima of the loss function ). It is often referred to as back propagation in terms of Neural Networks.
All the below methods are variants of Gradient Descent. You can learn more from this video.
Batch Gradient Descent:
The samples from the whole dataset are used to optimize the parameters i.e to compute the gradients for a single update. For a dataset of 100 samples, updates occur only once.
Stochastic Gradient Descent:
Stochastic GD computes the gradients for each and every sample in the dataset and hence makes an update for every sample in the dataset. For a dataset of 100 samples, updates occur 100 times.
Mini Batch Gradient Descent:
This is meant to capture the good aspects of Batch and Stochastic GD. Instead of a single sample ( Stochastic GD ) or the whole dataset ( Batch GD ), we take small batches or chunks of the dataset and update the parameters accordingly. For a dataset of 100 samples, if the batch size is 5 meaning we have 20 batches. Hence, updates occur 20 times.

All the above methods use gradient descent for optimization. The main
difference is that on how much samples are the gradients calculated.
Gradients are averaged in Mini-Batch and Batch GD.

You can refer to these blogs/posts:
Batch gradient descent versus stochastic gradient descent
Gradient Descent Algorithm and Its Variants


Validation accuracy for neural network When training a neural network, I usually plot the accuracy obtained on the validation data (validation accuracy) as an intermediate measure of the network's performance – the final measure being test accuracy. This validation accuracy is measured one or more times during a single epoch, depending on the size of the training data.

Usually, I stop training when the validation accuracy starts dropping – a sign of overfitting.

However, for large datasets I often don't have enough computational power to wait for the validation accuracy to drop. I merely stop the training after the validation accuracy (or validation loss) stays constant for a while, like so:

Technically, I do not know if the model can improve further, but is it reasonable to assume that it won't? Again, this is only for large datasets with limited computing power available. With sufficient computing power available, I can afford to wait until validation accuracy drops.

You can't assume the loss wouldn't drop with additional cycles, but it's generally safe to assume there are diminishing returns. This is why stopping rules are defined in terms of both number of iterations and change in the loss function. You want to stop when the gains become vanishingly small. At that point, increasing your accuracy from, say, 90.1% to 90.2% probably have little practical benefit, and performance on a hold-out sample probably won't differ much. But, that is an empirical question you can actually test. Run a model to near completion then run another with more stringent stopping criteria and compare on a hold-out.


Implementing a CNN with one convolution layer I am trying to implement a 1 channel CNN by slightly changing this article: this article.
The problem is that I am new to keras and deep learning and I don't know this far why I am getting this error:

ValueError: Negative dimension size caused by subtracting 100 from 1 for 'conv2d_1/convolution' (op: 'Conv2D') with input shapes: [?,1,70,100], [100,100,100,64]

Obviously, it's a mismatch in the dimensions.

I am using this code code:

from keras.layers import Embedding
from keras.layers import Conv2D
from keras.models import Sequential
from keras.layers import MaxPooling2D
from keras.layers import Reshape
import pdb
Vocab_Size=11123
MAX_SEQUENCE_LENGTH=70
EMBED_DIM=100
model = Sequential()
embed1=Embedding(Vocab_Size+1,EMBED_DIM,input_length=MAX_SEQUENCE_LENGTH,input_shape=(MAX_SEQUENCE_LENGTH,EMBED_DIM,1))
nb_labels=6
model = Sequential()
model.add(embed1)
model.add(Reshape((1,MAX_SEQUENCE_LENGTH, EMBED_DIM)))
model.add(Conv2D(64, strides=5, kernel_size=EMBED_DIM, activation="relu", padding='valid'))
model.add(MaxPooling2D((MAX_SEQUENCE_LENGTH-5+1,1)))
model.add(Flatten())
model.add(Dense(256, activation="relu"))
model.add(Dropout(0.3))
model.add(Dense(len(nb_labels), activation="softmax"))

model.compile(loss='categorical_crossentropy',
                optimizer='rmsprop',
                metrics=['acc'])

Edit1:
I updated as mentionned by Media padding to same.
Now I have another problem for the next layer:

ValueError: Negative dimension size caused by subtracting 66 from 1 for 'max_pooling2d_1/MaxPool' (op: 'MaxPool') with input shapes: [?,1,14,64].

I guess you should change the following line to solve the problem:

model.add(Conv2D(64, strides=5, kernel_size=EMBED_DIM, activation="relu", padding='valid'))

instead use this code:

model.add(Conv2D(64, strides=5, kernel_size=EMBED_DIM, activation="relu", padding='same'))

this will keep the dimension of your input. If it does not work, let me know to help you more.


Netflow anomaly detection python packages Is anyone aware of any open source / python packages for Netflow Anomaly detection  ? I found some on github but anyone who has more experience with it. please advise.

I could not find any specific package in the internet that does the same thing as netflow anomaly detection, but I found out how to train some standard algorithms that solve the same problem. My understanding is that the problem is "network intrusion detection" or "anomaly detection in network traffic to detect network intrusion"

So creation of a model can be divided into several parts:

choosing an appropriate algorithm
understand how to transform the data for the algorithm
finding the data set
training the model

The above tasks in more detail:

algorithm: k-NN or Random Forest. According to the paper Machine Learning DDoS Detection for Consumer Internet of Things Devices k-nearest neighbor is a pretty precise algorithm in network anomaly detection.  Nearest neighbor algorithms are present in scikit-learn python package (link). Random forest classifier performed even better. scikit-learn also has a random forest classifier (link).

Here is the precision table from the first paper. KN is k-NN. RF is Random Forest. NN is Neural Network. LSVM is a variation of Support Vector Machine algorithm.

How k-NN for DDoS works is explained in this paper: Proactive Detection of DDoS Attacks Utilizing k-NN Classifier in an Anti-DDos Framework.
According to the above mentioned papers and other sources the following datasets are used for training:

KDCup1999 This is a network intrusion database
MIT Lincoln Lab, 2000, DARPA intrusion detection scenario specific datasets Contains two DoS attack scenarios
Reasearchgate question on available datasets

On this link you will find how to train a k-nearest neighbor in scikit-learn.

Here is an example code from the link:

>>> X = [[0], [1], [2], [3]]
>>> y = [0, 0, 1, 1]
>>> from sklearn.neighbors import KNeighborsClassifier
>>> neigh = KNeighborsClassifier(n_neighbors=3)
>>> neigh.fit(X, y)
KNeighborsClassifier(...)
>>> print(neigh.predict([[1.1]]))
[0]
>>> print(neigh.predict_proba([[0.9]]))
[[0.66666667 0.33333333]]


Multiple GPU in MXNet C++ I am trying to make MXNet (C++ API) learn, with a common sample in C++, on multiple GPU.
According to this MXNet forum post, we need to aggregate manually the gradients that we fetch at the backpropagation time.
Now, if I separate the gradients of each GPU, both networks are training. If I concatenate the weights, it doesn't work. (Like this) :

gradValuesCombined.insert(gradValuesCombined.end(), gradValues1.begin(), gradValues1.end());
gradValuesCombined.insert(gradValuesCombined.end(), gradValues1.begin(), gradValues1.end());
NDArray combined1(gradValuesCombined.data(), combinedShape, ctx1);

However, if the gradients have the same batch_size, then summing the gradients works.

gradArray1[i] + gradArray2[i]

But, summing all the gradients like if it was a one-batch vector, doesn't work:

combinedOneDim[0] = 1;
int sizeOneBatch = combinedOneDim.Size();
gradValuesCombinedOneDim.insert(gradValuesCombinedOneDim.end(), gradValues1.begin(), gradValues1.begin() + sizeOneBatch);
for(int i = 1; i < curGradArrayShape1[0]; i++) {
     std::transform(gradValuesCombinedOneDim.begin(), gradValuesCombinedOneDim.end(), gradValues1.begin() + i * sizeOneBatch, gradValuesCombinedOneDim.begin(), std::plus<mx_float>());
}
for(int i = 0; i < curGradArrayShape2[0]; i++) {
     std::transform(gradValuesCombinedOneDim.begin(), gradValuesCombinedOneDim.end(), gradValues2.begin() + i * sizeOneBatch, gradValuesCombinedOneDim.begin(), std::plus<mx_float>());
}

At that stage, I figured that it was because I had to keep the initial shape of the gradient. But if I try to average the weights on the batch dimension, and copy the values to keep the same gradient shape, it should work. But it doesn't... And if I just sum everything on the first dimension and fill the rest with zeroes, it doesn't work either...

I verified, the synchronization of data is correct and working. So what do I miss?

Okay, I finally found the problem. It was that I was backpropagating the gradients on every layers. Including the input and output layers. And it seems that Mxnet expect strictly to have the same NDarray size than the one he computed (no bigger for instance). Otherwise, the array seems not to be copied at all (not even partially)and thus gives wrong results.

But to do that, I had to make sure the gradients size where equals, which is the case for every layers but the entry and output gradients. I incorrectly assumed that, if I increased the batch size of the input/output, all the layers of the network would have an increased batch size. However, the batch size of the input doesn't matter inside the network. So, what I thought I was doing by summing over the values for every batch size was clearly not doing what I thought - meaning averaging the gradients for every image. What I actually did with that was to level off every 3D kernels.

Thus, the simple solution in my case was just to sum the gradients for every layers but the first and last one.


Machine Learning validation data returns 100% accuracy I'm Testing a Machine Learning model with validation data returns that return 100% correct answers, is it overfitting or the model works extremely well, do I need to continue training on more data?
I'm not sure how to interpret the result, any guidance please?

I'm assuming this is a classification task as you mentioned accuracy:
100% on the validation set is usually a red-flag. Check there is no data leakage in your code (inspect it at each line). Make sure your validation set has a mixture of both classes (for example: is this a very imbalanced set and your validation set has all negative class?).
There is really no way to tell if or what the problem is without some description of the data or presence of code - but 100% accuracy is usually a big indication to dig into the details to find out if there's a bias you are adding to the system.


Why large weights are prohibited in neural networks? Why  weights with large values cause neural networks to be overfitted, and consequently we use approaches like regularization to neutralize weights with large values?

Many strategies used in machine learning are explicitly designed to reduce the test error, possibly at the expense of increased training error. Generally,
regularization is any modification we make to a
learning algorithm that is intended to reduce its generalization error but not its training error.

The L2 parameter norm penalty commonly known as Weight decay is one of the simplest and most common kinds of regularization technique which forces the weights to become smaller, by adding a parameter norm $Ω(θ) = 1/2 || w||^{2}_2$ penalty to the objective function.
For example, in linear regression, this gives us solutions that have a smaller slope, or put weight on fewer of the features. In other words, even though the model is capable of representing functions with much more complicated shape, weight decay has encouraged it to use a simpler function described by smaller coefficients.

Intuitively, in the feature space, only directions along which the parameters contribute significantly to reducing the objective function are preserved relatively intact. In directions that do not contribute to reducing the objective function, movement in this direction will not significantly increase the gradient.So, Components of the weight vector corresponding to such unimportant directions are decayed away through the use of the regularization throughout training.

Another simple explanation is when your weights are large, they are more sensitive to small noises in the input data. So, when a small noise is propagated through your network with large weights, it produces much different value in the output layer of the NN rather than a network with small weights.

Note that weight decay is not the only regularization technique. In the past few years, some other approaches have been introduced such as Dropout, Bagging, Early Stop, and Parameter Sharing which work very well in NNs.

There are other interesting findings in this very rich chapter.