XGBoost Classification Probabilities higher than RF or SVM? I am using Random Forests, XGBoost and SVMs to classify whether the home team wins or the away team wins their bowl game (in college football). I trained the models on all the games during the season.
I've come across something that is a bit weird and can't explain. I calculated a prediction confidence by subtracting the class probabilities. The XGBoost confidence values are consistency higher than both Random Forests and SVM's. I've attached the image below.
I did some hyper-parameter tuning for all of my models and used the best parameters based on testing accuracy.

Random Forest:

700 trees
15 variables randomly sampled (mtries)
minimum split criteria of 5 rows.

XGBoost:

0.5, Learn rate
gbtree as my booster
max depth of 6

SVM:

RBF kernel
C (slack) of 1
0.01, Sigma

I wasn't clear with my question: Why exactly does XGBoost prefer one class greatly to the other? In comparison to these other methods. I'm trying to figure out why my prediction confidences of a class are so high for XGboost.

Rather than answering why XGBoost give very confident predictions, I will answer why random forest and SVM give not-so-confident predictions.

Random forest probability estimates are given by the percentage of the forest that predicted a particular class. For example, if you have $100$ trees in your forest and $81$ of them predict some class for some example, the probability estimate for that example belonging to that class is calculated to be $\frac{81}{100} = 0.81$. Because of the random nature of the ensemble members, it's very unlikely that each individual tree will end up with the correct prediction, even if the majority do. This makes probability estimates from random forests shy away from the extreme ends of the scale.

SVM is a slightly different case, because they are unable to produce probability estimates directly. Typically, Platt scaling (essentially logistic regression) is used to scale the SVM output to a probability estimate. This has the added benefit of calibrating the probability estimates, meaning the predicted probability is quite accurate - in other words, if a probability of $0.8$ is given for a prediction, it actually has approximately an $80\%$ chance of being correct. For a problem like this where there's a lot of noise (underdog teams do win sometimes, and there's a lot of evenly matched games that are hard to predict), these predictions will tend not to be overconfident.

I don't have a good reason as to why XGBoost is possibly overconfident, but it has been observed in the past that additive boosting models tend to provide distorted probability estimates without applying post-training calibration e.g. here, here & here.


Derivation of backpropagation for Softmax So, after a couple dozen tries I finally implemented a standalone nice and flashy softmax layer for my neural network in numpy. All works well, but I have a question regarding the maths part because there's just one tiny point I can't understand, like at all.

Having any kind of activation function in the output layer, back-propagation looks like:

$$ z = x\cdot w+b \\ a = f(z) \\ E = \frac{1}{N} \sum^{N}_{i=1} (a_{i} - y_{i})^{2}
 \\ \frac{\partial E}{\partial w} := \frac{\partial E}{\partial a_{i}} \cdot\frac{\partial a_{i}}{\partial z} \cdot \frac{\partial z}{\partial w}  \\ \frac{\partial E}{\partial w} := (a_{i} - y_{i}) \cdot f^{'}(z) \cdot x $$

So by differentiating $ a_{l} $ with respect to $ z_{l} $, the result is the derivative of the activation function with $ z_{l} $ itself. Now, with Softmax in the final layer, this does not apply. If I use $ Softmax'(z_{l}) $ I get incorrect results, but I rather need $ Softmax'(a_{l}) $. Looking at a couple online materials like LINK in the Backpropagation phase section in the python code, the author also uses the activation itself in the argument for softmax's derivative and I've been differentiating there and back for the last 2 weeks, ... I don't understand why it works like that.

In short, my question is: With specifically Softmax in the output layer, why is my $ \frac{\partial a_{l}}{\partial z_{l}} $ becomes $ Softmax'(a_{l}) $ instead of $ Softmax'(z_{l}) $ like with pretty much any other normal activation function that's out there?

We let $$a=\operatorname{Softmax}(z)$$

that is $$a_i = \frac{e^{z_i}}{\sum_{j=1}^N e^{z_j}}.$$

$a$ is indeed a function of $z$ and we want to differentiate $a$ with respect to $z$. The interesting thing is we are able to express this final outcome as an expression of $a$ in an elegant fashion.

If you look at the section of "Derivative of Softmax Function" in your link, using quotient rule:

We can see that

if $i=m$,
\begin{align}\frac{\partial a_i}{\partial z_m} &=\left(\frac{e^{z_i}}{\sum_{j=1}^N e^{z_j}}\right)\left(\frac{\sum_{j=1}^Ne^{z^j}-e^{z_m}}{\sum_{j=1}^N e^{z_j}} \right)=a_i(1-a_m)\end{align}

if $i\ne m$,
\begin{align}\frac{\partial a_i}{\partial z_m} &=-\left(\frac{e^{z_m}}{\sum_{j=1}^N e^{z_j}}\right)\left(\frac{e^{z_i}}{\sum_{j=1}^N e^{z_j}} \right)=-a_m(a_i)\end{align}

If you want to evaluate things in terms of $z$, you can still use the middle term though using the formula

$$\frac{da}{dz}=ae^T\circ (I-ea^T)$$

is more elegant.


What is the difference? "Adding more LSTM layers" or "Increasing epochs"? To get more accurate results which one is better? Adding more layers or increasing number of epochs? I like to know the difference between effects of these two approaches?

Increasing the number of epochs :
This is the best when you have a large and diverse dataset. A large dataset refers to a number of diverse samples in equal amounts ( no bias ). Also, you make your model generalise better, you might increase the dropout rates and decrease the learning rate. At such a time increasing the number of epochs would help the model generalise better. Also, since it has generalised on a diversed dataset means it can perform well in the real world. For LSTMs, they can make correct predictions based on the sequences.
Increasing the number of layers of LSTM :
If we treat each LSTM as a memory unit then we are increasing the number of memory units and thus the overall memory. The first sequences would be feed to the next layer hence a model could create a hierarchical representation of the data. This is better if you have sequences which have such a context like an image.

On considering both the scenarios, if you have abundant data then the first option is optimum. If you have complex data which has less diversity then the second and the first points need to be considered.


how to make sklearn pipeline using custom model? I want to make a sklearn pipeline using the custom Artificial Neural Network I already have. I want to make pipeline in which input goes to ANN and its output goes to the sklearn.svm.SVC model and final prediction is made.
So, how can I do this using sklearn pipeline?

Implementing a custom transformer is simple. You have to implement the fit and transform methods like below. Since your ANN is already trained (right?) the fit method has to do nothing, just return self. And the transform method has to pass the incoming data to the ANN and return its output.

from sklearn.base import BaseEstimator, TransformerMixin

class MyANNTransformer(BaseEstimator, TransformerMixin):
    def __init__(self, ann):
        self.ann = ann

    def fit(self, X, y):
        return self

    def transform(self, X)
        return self.ann.predict(X)

Now you can include that in pipelines:

from sklearn.pipeline import make_pipeline
from sklearn.svm import SVC

pipe = make_pipeline(MyANNTransformer(ann),
                     SVC())

pipe.fit(Xtrain, ytrain)
pipe.predict(Xtest)


Avoid recycling when merging with R Data Table I got tableA, which contains a numeric variable, CdTaller (e.g. 1948675).

I also got tableB, which contains two variables: Id (e.g. 2513978) and Name (e.g. "JOSE MANUEL PEREZ")

I want to add a new column to tableA, Name_Taller, which contains a name for those CdTaller which correspond to the Id variable of tableB, so I do the following:

tableA[, Name_Taller := tableB[Id==tableA\$CdTaller]$Name]

However, for those CdTaller of tableA that are not in tableB, R recycles the new variable Name_Taller:

Warning message:
In \`[.data.table\`(tableA, , \`:=\`(Name, tableB[Id ==  :
  Supplied 867324 items to be assigned to 947871 items of column 'Name'
  (recycled leaving remainder of 80547 items).

So for the tableA$CdTaller == '1320402', which can't be found in tableB, R still adds a Name_Taller which actually corresponds to CdTaller == '4430219', because of this recycling.

The question is: how can I avoid this recycling, so that R only adds a NA or empty string to those rows containing a CdTaller not found in tableB? Is there any parameter in data table I can use for this purpose?

What you want to do is join-like behaviour that is among others documented here:
joins and others

To make things readable here a version with dplyr

ex1 <- data.frame(id = 1:10, name = replicate(10, paste(sample(letters[1:5], size = 2), collapse = "")))

ex2 <- data.frame(otherinfo = (1:5)+50, othername = replicate(5, paste(sample(letters[1:5], size = 2), collapse = "")))

dplyr::left_join(ex1, ex2, by = c(name = "othername"))

   id name otherinfo
1   1   ac        NA
2   2   ac        NA
3   3   ab        NA
4   4   da        53
5   5   ab        NA
6   6   cb        NA
7   7   eb        NA
8   8   de        NA
9   9   ad        NA
10 10   ec        52


How to penalize for empty fields in a DataFrame? I have to calculate the consistency of racing car drivers during the whole season. My DataFrame consists of 10 columns  (10 circuit names) and for each of those columns I have the standard deviation in lap time the driver posted in that circuit. In other words, how consistent the driver is from lap to lap. In races the driver did not finish the field is blank.

So far I have calculated their average season consistency by averaging all 10 columns. However, not finishing a race should affect a driver's consistency negatively and I do not know how to implement that.

This heavily depends on the domain knowledge. A general approach would be to place

A multiplicative of the worst or average consistency at each circuit $c$, i.e. $(1 + m)\text{max}(\sigma_c)$ or $(1 + m)\text{avg}(\sigma_c)$ respectively, for the null values at that circuit, or
A multiplicative of the worst or average consistency of each driver $d$, i.e. $(1 + m)\text{max}(\sigma_d)$ or  $(1 + m)\text{avg}(\sigma_d)$ respectively, for their unfinished races, or
A multiplicative of average of driver and circuit average consistencies, i.e. $(1 + m)[\text{avg}(\sigma_d) + \text{avg}(\sigma_c)]/2$, for unfinished race of driver $d$ at circuit $c$, or some other combinations.

No matter which approach to choose, the choice of coefficient $m$ affects the final ranking and could be determined either

Subjectively by looking at the rankings from an expert point of view and selecting the one that makes more sense, or
By trying a range of values like $m \in \{-0.2, -0.1, 0, 0.1, 0.2, .., 0.5\}$ and averaging the consistencies $\sigma_d$ or rankings $R_d$ for each driver $d$. An advantage of this approach would be that when rank of a driver has a low variance over different values of $m$, it implies that driver's rank is insensitive to the choice of $m$, i.e. it is less controversial, and when rank changes a lot with different choices of $m$, the average rank is more controversial.


How to test the quality of a word embedding? I have trained a word2vec model using GenSim 4.
The problem is that my corpus is quite small.
How can I test the quality of the word embeddings I have obtained?
Is there some standard measures to do that?

One way to test your embedding is see how often your model agrees with the common consensus of how other embeddings complete word analogies. A collection of established word embedding analogies are here.


Isn't graph embedding a step back from non-euclidean space? As I understand, we use graph embedding to make a euclidean representation of non-euclidean structure - graph. Does it mean that conceptually we just take a step back to, may be, more complex, but still grid processing?

Building embeddings is the first step in graph processing.
We mainly use to apply mathematics on something like plane or hyperplane. Almost all mathematical methods can work with linear space, especially linear algebra which we use in neural networks and computer data processing.
So, in fact, embedding is just one of the first steps we need to do in order to apply some math on graph data. We can embed node features to a lower dimension, we can also embed an entire graph to some space we know how to work with. Then apply functions, transformations, whatever we want.
When you are building an embedding, your task is to represent a graph in linear space so that it will keep all the characteristics it has in non-linear "graph" space.
In this article you can read more about different approaches to graph embedding. I will just cite one phrase from there.

Machine learning algorithms are tuned for continuous data, hence why embedding is always to a continuous vector space.


plot show relationship between independent variable and dependent variable(Binary) My dependent variable is binary. Most of my independent variables are not. I am at the exploratory stage right now.

Y     X1     X2
0     23    0
1     29    1
0     15    1
1     40    0
1     25    1
0     22    1
This is just a portion of my data.

I was thinking scatter plot to find out the relationship between Y X1 and X2.
What other plots I can do to see the relationship more clearly.

There is package named Boruta in R/Python. This is also called as predictor importance test. Used to pick the important variable.

Uses of this package:

Boruta is a feature selection algorithm. Precisely, it works as a wrapper algorithm around Random Forest. This package derive its name from a demon in Slavic mythology who dwelled in pine forests.

We know that feature selection is a crucial step in predictive modeling. This technique achieves supreme importance when a data set comprised of several variables is given for model building.

Boruta can be your algorithm of choice to deal with such data sets. Particularly when one is interested in understanding the mechanisms related to the variable of interest, rather than just building a black box predictive model with good prediction accuracy.

The outcome would be a box whisker plot with their importance with the target variable. You need not worry about categorical and continuous variables. you just need to cast them correctly before passing the data to the algorithm.

You can go through this Link for better understanding on the package and implementation of the same.

This is the sample plot:

Let me know if you need any additional information.


What is the main goal of using activation function in CNN? What is the main goal of using an activation function in CNN?

I know the activation functions types and the purpose of each one. But here I am asking why to use them.

The idea of convolutional layers is that we need same weights to be applied to different regions of the input. It lets you identify same patterns that occur in different regions of the input. You can consider each convolution operation which is carried out by each window as a single neuron which just transforms a local region with a non-linear transformation. The linear and non-linear operations that are done by these so-called local neurons help the convolutional layers learn non-linear and complex features. Another aspect of this operation is that it reduces the number of weights significantly. The final interpretation is that if you don't use non-linear activations, you would have linear local features extracted from the input. Those linear features can easily be found with fewer number of parameters.


Clarification wanted for make_step function of Google's deep dream script From https://github.com/google/deepdream/blob/master/dream.ipynb

def objective_L2(dst):          # Our training objective. Google has since release a way to load
    dst.diff[:] = dst.data      # arbitrary objectives from other images. We'll go into this later.

def make_step(net, step_size=1.5, end='inception_4c/output',
              jitter=32, clip=True, objective=objective_L2):
    '''Basic gradient ascent step.'''

    src = net.blobs['data'] # input image is stored in Net's 'data' blob
    dst = net.blobs[end]

    ox, oy = np.random.randint(-jitter, jitter+1, 2)
    src.data[0] = np.roll(np.roll(src.data[0], ox, -1), oy, -2) # apply jitter shift

    net.forward(end=end)
    objective(dst)  # specify the optimization objective
    net.backward(start=end)
    g = src.diff[0]
    # apply normalized ascent step to the input image
    src.data[:] += step_size/np.abs(g).mean() * g

    src.data[0] = np.roll(np.roll(src.data[0], -ox, -1), -oy, -2) # unshift image

    if clip:
        bias = net.transformer.mean['data']
        src.data[:] = np.clip(src.data, -bias, 255-bias)

If I understand what is going on correctly, the input image in net.blobs['data'] is inserted into the NN until the layer end. Once, the forward pass is complete until end, it calculates how "off" the blob at the end is from "something".

Questions

What is this "something"? Is it dst.data? I stepped through a debugger and found that dst.data was just a matrix of zeros right after the assignment and then filled with values after the backward pass.
Anyways, assuming it finds how "off" the result of the forward pass is, why does it try to do a backwards propagation? I thought the point of deep dream wasn't to further train the model but "morph" the input image into whatever the original model's layer represents.
What exactly does src.data[:] += step_size/np.abs(g).mean() * g do? It seems like applying whatever calculation was done above to the original image. Is this line what actually "morphs" the image?

Links that I have already read through

https://stackoverflow.com/a/31028871/2750819

I would be interested in what the author of the accepted answer meant by

  we take the original layer blob and "enchance" signals in it. What
  does it mean, I don't know. Maybe they just multiply the values by
  coefficient, maybe something else.

http://www.kpkaiser.com/machine-learning/diving-deeper-into-deep-dreams/

In this blog post the author comments next to src.data[:] += step_size/np.abs(g).mean() * g: "get closer to our target data." I'm not too clear what "target data" means here.

Note I'm cross posting this from https://stackoverflow.com/q/40690099/2750819 as I was recommended to in a comment.

What is this "something"? Is it dst.data? I stepped through a debugger and found that dst.data was just a matrix of zeros right after the assignment and then filled with values after the backward pass.

Yes. dst.data is the working contents of the layer inside the CNN that you are trying to maximise. The idea is that you want to generate an image that has a high neuron activation in this layer by making changes to the input. If I understand this correctly though, it should be populated immediately after the forward pass here: net.forward(end=end)

  Anyways, assuming it finds how "off" the result of the forward pass is, why does it try to do a backwards propagation? I thought the point of deep dream wasn't to further train the model but "morph" the input image into whatever the original model's layer represents.

It is not training. However, like with training we cannot directly measure how "off" the source that we want to change is from an ideal value, so instead we calculate how to move toward a better value by taking gradients. Back propagation is the usual method for figuring out gradients to parameters in the CNN. There are some main differences with training:

Instead of trying to minimise a cost, we want to increase a metric which summarises how excited the target layer is - we are not trying to find any stationary point (e.g. the maximum possible value), and instead Deep Dream usually just stops arbitrarily after a fixed number of iterations.
We back propagate further than usual, all the way to the input layer. That's not something you do normally during training.
We don't use any of the gradients to the weights. The weights in the neural network are never changed. We are only interested in the gradients at the input - but to get them we need to calculate all the others first.

  What exactly does src.data[:] += step_size/np.abs(g).mean() * g do? It seems like applying whatever calculation was done above to the original image. Is this line what actually "morphs" the image?

It takes a single step in the image data along the gradients that we have calculated will trigger more activity in the target layer. Yes this alters the input image. It should be repeated a few times, according to how extreme you want the Deep Dream effect to be.


Network structure: k-cliques vs. p-cliques In network structure, what is the difference between k-cliques and p-cliques, can anyone give a brief explaination with examples? Thanks in advanced!

============================
EDIT:
I found an online ppt while I am googling, please take a look on p.37 and p.39, can you comment on them?

In graph theory a clique indicates a fully connected set of nodes: as noted here, a p-clique simply indicates a clique comoprised of p nodes. A k-clique is an undirected graph and a number k, and the output is a clique of size k if one exists.

Clique Problem


Any good Implementations of Bi-LSTM bahdanau attention in Keras? From past few weeks I'm trying to learn sequence to sequence machine translation modelling but I couldn't find any good examples/tutorials with bahdanau attention implemented. I did come across a ton of examples where people have implemented attention but it's mostly on gru (or) luong attention (or) outdated code.

Has anyone come across any good implementations of bahdanau attention model in keras which you have implemented (or) tried? I really want to learn the coding part but no good material regarding implementation is found?

Please help.

Here's the Deeplearning.ai notebook that is going to be helpful to understand it. Neural machine translation with attention


Anomaly detection - relation between thresholds and anomalies I'm developing an anomaly detection program in Python.
Main idea is to create a new LSTM model every day, training it with the previous 7 days and predict the next day.
Then, using thresholds, find anomalies day by day.

I've already implemented that and these thresholds are working well:

upper threshold is equals to trimmed_mean + (K * interquartile_range)
lower threshold is equals to trimmed_mean - (K * interquartile_range)

where trimmed_mean and interquartile_range are calculated on prediction error (real curve - predicted one) and K is set to 5.

What i would like to know is if there is a rule or a method to correctly configure the value of K. Because at the moment, my approach was to adjust thresholds looking at false positive.

I'm looking for any relation between anomalies and how to set up thresholds.

I already tried to calculate for each model the AUC (area under the curve) and looking for relation, but without success. In fact, since i create a new model every day, and for the most of the days I don't have any anomalous value, i'm not able to calculate the true positive rate correctly.

Thank you

EDIT
Based on Ben's comment, I'm going to add details about my problem.
Let's start to explain better what anomalous value is for me.
So, I'm analyzing timeseries created by acquisition of kWh each 5 minutes, so 288 records per day.
An anomalous value is basically a peak, so a value that is very different from the others.
Furthermore, i need to create a model day by day because customer has not too much data and I need to catch also seasonality.
Last but not least, I working on a univariate problem, so I have just the acquisition value and time, that's it.

EDIT REPLYING BEN'S QUESTIONS
First of all yes, kWh means power consumption.
I have another type of anomalies that are blocked values (like 0, 0, 0 ecc.), but i think that i can identify them "adding a simple rule" which detect these situations.
About seasonality:
Each new model is trained using the previous 7 days, and day by day this time window moves forwarding. So, every day model is deleted and trained using just the previous 7 days.
I already managed the issue that you mentioned (train new model with anomalous data) substituting anomalous data with a new value calculated using the difference between next value and previous value.

Here a couple of plots of my data:
image 1
image 2

where green line is the prediction and blue one is the real timeseries

To give a helpful answer more information is necessary. When you want to detect anomalies, I would say, from my feeling and understanding, that it doesn't make sense to train the model every day. You should define your normal system once and try to catch anomalies from there. Trends are hard to catch but this is always hard.

Identifying anomalies means also you have to know your system but here I have to refer to the first sentence: What is your system? Is it a technical system (then the task becomes quite complex) or is it "just" some pattern recognition or fraud detection and so on?

However, when I did that for a large machine with ~60 variables (and 12 months of data), I used the first 3 months to train and defined this state as normal. Then I set the thresholds such that an anomaly is identified when the RMSE reaches $ \sqrt{max (X_i)} $. This worked quite well with the exception of 1-2 variables.
But as I said, it depends on your situation. For example, when you have a steam machine and the normal temperature is at 98 °C and everything bursts at 101 °C, you probably have to use a custom threshold as e.g. a standard deviation wouldn't help here much.

To add some more information: Especially in case of anomaly detection ,it is often the case that one approach is not sufficient. I would implement some basic approaches like using standard deviations for the most trivial cases which definitely have to be detected and the more advanced methods (like LSTM) to track more complex patterns.


Is there a good deployment module using Golang's amazing single binary compilation model? From my limited understanding, it's quite annoying trying to deploy models like PyTorch. So one would need something like ONNX.
But another approach like Golang with a single binary is VERY attractive. I have used Golang before and it's reasonably fast and compiles to one binary easily. For Rust, compiling to a single binary is a nightmare.
So I wonder if anyone has seen work that translates XGBoost trees to Golang and uses arrow as the data exchange format? That way, it makes deployment super easy, wouldn't it?

I code up a Golang package call gotch that can take Pytorch pretrained model and do further training/finetuning or inferring.
Have a look at “example” subfolder as well as Gophernotes notebook links in README.md.


normalize dependent variable with respect to one independent variable I have a data set with 50-60 independent variables and 1 categorical (1-0) dependent variable. Out of all the independent variables say, A is one variable which has majority of the effect on the target variable. For ex

A        target variable
a              1
a              1
a              1
a              1
b              0
b              1
b              1
b              1
c              0
c              0
c              0
c              1
c              1

Clearly whatever analysis I do here, whatever probabilities I calculate for With respect to other variables, I would not be able to get the actual picture with respect to other variables.

Also I do not want to build any model at this point, I just want to see the individual effects of other variables on the dependent variable after removing the effect of variable 'A' i.e. normalizing dependent variable with respect to 'A'. How do I do? If the dependent variable was a continuous variable, I would have simply divided it by the respective averages and removed the effect. How do I do this in this case where the dependent variable is 1-0?

Though continuum regression methods appear to be significantly different from binary classification methods, the crux of the difference is usually tied up in a sigmoid function.

$$S(t)=\frac{1}{1+e^-t}$$

Sigmoid literally means sigma like or "S" like, as becomes obvious in looking at its graphical representation:

The sigmoid provides a tractable means to map from continuum analytics into discrete space while retaining differentiability and integrability needed for some theoretical derivations.

The important part is that in classification methods, one has to choose a threshold value of the sigma function for mapping to a 0 or 1.  This is overwhelmingly chosen as 0.5, but there are cases of unbalanced classes where other values might work better.

That said, you should be able to proceed exactly as you have already suggested: "I would have simply divided it by the respective averages and removed the effect."  Do this, but then you have to turn the results back into 0's and 1's by choosing where your cutoff threshold lies.

Hope this helps!


Make Keras run on multi-machine multi-core cpu system I'm working on Seq2Seq model using LSTM from Keras (using Theano background) and I would like to parallelize the processes, because even few MBs of data need several hours for training.

It is clear that GPUs are far much better in parallelization than CPUs. At the moment, I only have CPUs to work with. I could access 16 CPUs(2 Threads per core X 4 cores per socket X 2 sockets)

From the doc of multi-core support in Theano, I managed to use all the four cores of a single socket. So, basically the CPU is at 400% usage with 4CPUs used and the remaining 12 CPUs remain unused. How do I make use of them too. Tensorflow could also be used instead of Theano background, if it works.

In order to set the number of threads used in Theano (and, therefore, the number of CPU cores), you'll need to set a few parameters in the environment:

import os
os.environ['MKL_NUM_THREADS'] = '16'
os.environ['GOTO_NUM_THREADS'] = '16'
os.environ['OMP_NUM_THREADS'] = '16'
os.eviron['openmp'] = 'True'

This should allow you to use all cores of all CPUs.

This can, of course, also be done in Tensorflow:

import tensorflow as tf
from keras.backend import tensorflow_backend as K

with tf.Session(config=tf.ConfigProto(
                    intra_op_parallelism_threads=16)) as sess:
    K.set_session(sess)
    <Your Keras code>


Suggestion for stacked modelling in machine learning I have built several models on the training dataset and i am not happy with the results and I wish to club them all together and generate a new model, so here is my idea as i already have the results for the existing models i would like to create a new dataset with the existing model results as separate features on top of the original feature dataset applying a clustering to filter some data in the original dataset and would like to train the model across all the same models and get the result, Would that be called as stacked modelling?

Would that be called as stacked modelling?

Yes, this is exactly what stacking models means.

I didn't understand what is the role of clustering in your design?

The standard approach consists in using only the predictions of the $N$ individual learners as feature for the meta-model. The training/testing data split is a bit more complex: the training of the meta-model requires predictions from the individual models, so the data could be split like this:

training data for individual models
training data for the meta-model (on which individual models are applied)
testing data for the full system

Generally it's safer to choose a simple learning algorithm for the meta-model, because there is a higher risk of overfitting (for instance linear regression or majority voting).

[Detailed answer to comments below]

The standard setting would be like this:

Train $N$ individual models $m_1,...,m_N$ using first-level training set $T_1$
Apply these models in order to obtain $N$ features for the meta-classifier. At this stage it's a bad idea to use instances from $T_1$, because the individual classifiers have been trained on $T_1$. so one needs a second set of instances $T_2$: for every instance $x\in T_2$, let $m_1(x),...,m_N(x)$ be the predictions resulting from applying classifiers $m_1,...,m_N$.
Train the meta-classifier model $M$ using $T_2$ as training set, with $m_1(x),...,m_N(x)$ the features for any $x\in T_2$.
Testing: as usual one needs a fresh set of instances, say $T_3$. For any instance $x\in T_3$ calculate the prediction by applying $M$ to the $N$ predictions obtained by the individual models, i.e. $M(m_1(x),...,m_N(x))$


Python: convert variables into correct format for DataFrame I have 3 variables that I would like to use to build my dataset but since they are in a weird shape/format, I had no success so far. I'm quite new to this and really appreciate any help!!
The 3 variables I have are:
print(newspaper)

['Bolero']
['Schweizer Illustrierte Style']
['Bolero']
print(title)

['Schönheit und Tragik']
['magie pur']
['Das sind unsere Favoriten']
print(pubDate)

['2007-01-01']
['2007-01-01']
['2007-01-01']
It seems to like all variables are a list of lists, but I'm not quite sure. However, since the data is scrapped from a private website, I can't post the entire code here, but I hope this is already enough for you to access what the problem is with that variable format.
What I would like to have is a dataset of this format:

Newspaper
Title
PubDate

Bolero
Schönheit und Tragik
2007-01-01

Schweizer Illustrierte Style
magie pur
2007-01-01

Bolero.
Das sind unsere Favoriten
2007-01-01

First, you need to convert list of list into a list.
From link, you can convert a list of lists into a list by declaring the following function.
flatten = lambda t: [item for sublist in t for item in sublist]

Now all you need is to create dataframe using created lists.
data = {"Newspaper":flatten(newspaper), "Title": flatten(title), "PubDate": flatten(pubDate)}
pd.DataFrame.from_dict(data)


Problem of continuous training - Supervised learning I am sure this is a most common problem, but would like to know by experts on how to tackle it. Note that, I mostly deal with textual data (NLP problems).
When a supervised learning model is created, say a text classifier, and it works well on seen data then we deploy the model in production (you can think of a chatbot also).
But in real time, when new type of data comes where the prediction fails, we find that a new word or new pattern is breaking the model. So we go ahead and retrain the model with new encountered data. This is where the continuous learning problem starts.
Can ML/NLP veterans please suggest some alternates to solve this labor work? Following approaches have been tried and the problems also listed:

We simply can't train with new data infinitely. As production systems should be self healing. We cant put the cost of a human admin constantly monitoring the project. Also, it is practically not possible to get huge domain data during model training phase.
Use of advanced embeddings, and SoTA models like BERT.   (Problem: The accuracy of these models is too hard to control)
Synthetic data generation/data augmentatoin.  (Problem : Does not work well in case of NLP problems. Refer: training-with-less-data )
Unsupervised classification  (Problem: Does not work well on closed domain problems, as most unsupervised models are either statistical which give a fair value of accuracy but not decent , or are trained on public domain data)
Reinforcement learning. (Problem: Real world NLP data is not labeled unlike a self driven car where the feedback is instant)

What you are describing is called auto-adaptive learning. This is what most recommendation systems use to adapt to ever changing data and feedback. It is also known as autoML.
This https://towardsdatascience.com/how-to-apply-continual-learning-to-your-machine-learning-models-4754adcd7f7f
Article does a good job of explaining it. Based on what your data looks like, you might have to choose the appropriate retraining strategy and do a staggered deployment.


What would be my features? I was thinking of training a neural network that would be able to classify twitter users according to their followers. For example, I would like to know if a user is "gamer" or not by the people he follows. Have a dataset of people who are gamers or not and train them based on that.

The problem that I find is that the number of followers of each person will vary ... One person could follow thousands of users while another could only follow hundreds, so how can I determine my feature there?

Any suggestion? How could I do this?

This is in the way I am representing the data, so you can understand the problem of the features I was referring to

I did not know very well if asking this on StackOverflow, I have finally come here, it is the first time I post here, I hope it is the right place. Thank you.

Your features could be the other twitter accounts that the users follow with a value of 1 if your user follows that account or a value of 0 if the user does not. Here is a very simplified version, using something like your starting point.

# If you start like this:
user_id  feature_1  feature_2  feature_3  label
user_1   account_1  account_5  NA         1
user_2   account_2  account_3  account_4  0
user_3   account_1  account_2  account_5  1
user_4   account_3  NA         NA         0

# Transform your data to look like this:
user_id  account_1  account_2  account_3  account_ 4  account_5  label
user_1   1          0          0          0           1          1
user_2   0          1          1          1           0          0
user_3   1          1          0          0           1          1
user_4   0          0          1          0           0          0

This is a trivial example with only five accounts to follow. You may be dealing with millions of accounts to follow, which would be at best unwieldy. Since you already have a sense that some followed accounts are correlated with your label, you could simplify the features to only check those that are strongly correlated.

In my simple example above, we may only want to keep the features account_1, account_3, and account_5 since they correlate strongly to the label.


Dealing with long sequence labeling I am dealing with a problem in which I have to label the inputs (in a sequence format) to 5 distinct classes.
The input would be like:

X = {x_1,_x_2,...,X_500}

and the output should be something like:

Y = {Y_1,Y_2,...,Y_500}

But the problem is that too many of the labels are from the first class so a sample output would have many first class labels and only a few (5 or 6 samples) related to other classes.

The classifier tends to learn to classify everything to first class and yet get higher accuracy which is not correct.

What are your suggestions?

Edit:
Loss function being used is cross entropy loss, the architecture itself is a bilstm which is applied after an embedding layer, to be more precise:

InputLayer -> Embedding -> BidirectionalLSTM -> NN -> Softmax

The input is in character format (indexes of characters) and the output for each character is of 5 distinct classes while the problem is:

Most of characters belong to first class.

PS: I hope this would leave enough information.

First of all, don't use classification accuracy as a metric. Use precision, recall or F-score. They are best suited for multiclass unbalanced datasets.

Secondly, if you want to enrich your dataset with synthetic points of the minority class, a common way is to use the SMOTE algorithm.

Another way of creating more synthetic samples of your classes would be to use Generative Adversarial Networks (GANs). You will train a Generator and a Discriminator network based on the samples that you have. See this for further info.


K-means: What are some good ways to choose an efficient set of initial centroids? When a random initialization of centroids is used, different runs of K-means produce different total SSEs. And it is crucial in the performance of the algorithm.
What are some effective approaches toward solving this problem? Recent approaches are appreciated.

An approach that yields more consistent results is K-means++.  This approach acknowledges that there is probably a better choice of initial centroid locations than simple random assignment. Specifically, K-means tends to perform better when centroids are seeded in such a way that doesn't clump them together in space.

In short, the method is as follows:

Choose one of your data points at random as an initial centroid.
Calculate $D(x)$, the distance between your initial centroid and all other data points, $x$.
Choose your next centroid from the remaining datapoints with probability proportional to $D(x)^2$
Repeat until all centroids have been assigned.

Note: $D(x)$ should be updated as more centroids are added.  It should be set to be the distance between a data point and the nearest centroid.

You may also be interested to read this paper that proposes the method and describes its overall expected performance.


Classification of phone numbers belonging to same client I have the following task at hand:
Suppose, that there is a data on clients actions from the base stations of the mobile operator.

While being in the reach of the base station, client can make the following actions: Turn off the phone, change location, turn out of range, and etc. Hence, there are several data points for each phone number in the dataset, with attributes like coordinates of base station, type of the base station, type of device, client action, and so on. Also, there could be several base stations in one location, but pointing in different directions.

Also, there is a training set of phone numbers that belong to the same client. The task is to find all phone numbers that belong to one client (Each client can have 2 or 1 sim-cards).

As of now, my main idea is to somehow embed data points of each phone number into vector, and use some distance threshold small enough, to group these vectors into the pairs. But in this scenario i'm not using the training data at all.

Any suggestions would be appreciated.

P.S. After digging into the data a little bit, i found out that the data on each phone number is hugely varied in length. For instance, there could be 8 data points for one number, and 1000 for another. On this evidence, i'm not considering the idea with embedding viable anymore.

So, finally i managed to come up with a plausible solution.

First of all, i built a dataset by combining each phone number pair, and calculated some mutual features, like proportion of same location per day, wether it is the same device type or not, the group of base station, and etc.

Having done that, i realised that i only have a positive class, hence, can't employ binary classification straight away. To solve this problem i used what's called a p-u (positive-unlabeled) classification, in which i trained 1000 decision trees with my labeled data points as positive class, and random sample from unlabeled data as negative class. Then, i averaged predictions for each entry.
This approach has resulted in decent F-score.

Here is a resource where you can learn more about pu classification: pu-classification


Data Science Conferences w/ Poster Sessions? I am in search of some data science conferences (for 2017) that have a mix of big name speakers and poster sessions.  Ideally, I would like to be able to present a poster that doesn't necessarily have to be a research paper, but simply the development of a tool and its use.

Please provide the name and link.  Thank you!

KDNuggets has a really nice list of conferences on their page:

http://www.kdnuggets.com/meetings/index.html


Identifying Customers who are more likely to purchase a given product category- Which model to use? I'm pretty new to data science. I am working on a model to identify customers who are more likely to purchase a given product category.

I did try market basket using arules package from R.
I also used MatchIt, Matching algorithm and did try Propensity score matching to find similar customers who tend purchase in the given product category. But I'm not quite sure the model is good enough for this kinda problem.

I'm just wondering what kind of model will be useful to find customers who are more likely to purchase a given product group.
Any help is greatly appreciated.

Thanks in advance.

Actually, I'd propose to tackle it from a different angle - assuming you want to predict product categories, and those are usually few, you can see it as a regular classification problem.

Every product group is a category, which you will try to predict. So you can use simple models, which are easy to train (logistic regression, decision trees etc.), easy to build, debug, explain and test. And you can experiment with various features - which ones are the most helpful when predicting product category (this is crucial, since many people will probably ask you why the model predict this category for this person).

Once you master this, then the regular approach to this is any type of recommender system (collaborative filtering, matrix factorization etc.). Plus you can use your simple approach as a benchmark.


Random selection of variables in each run of python sklearn decision tree (regressio ) When I put random_state = None and run Decision tree for regression in python sklearn, it takes different variables to build tree each time?
Shouldn't there be only few top variables which should be used to split and should throw me similar trees everytime?
Also, if I use integer for random_state and run the decision tree, it gives me a different tree for each random_state setting. Which tree should be selected in case of so many trees?

The documentation says:

random_state : int, RandomState instance, default=None
Controls the randomness of the estimator. The features are always randomly permuted at each split, even if splitter is set to "best". When max_features < n_features, the algorithm will select max_features at random at each split before finding the best split among them. But the best found split may vary across different runs, even if max_features=n_features. That is the case, if the improvement of the criterion is identical for several splits and one split has to be selected at random. To obtain a deterministic behaviour during fitting, random_state has to be fixed to an integer.

So:

it takes different variables to build tree each time?

The randomness comes from picking different variables when building the tree, indeed.

Shouldn't there be only few top variables which should be used to split and should throw me similar trees everytime?

Not necessarily, this depends on the parameters and the data:

If an int is given as value for the parameter random_state, a particular random state is used every time so the tree is fixed, even though it's not necessarily the only one possible. Let's assume that the default value None is provided.
If the parameter max_features is anything other than None or auto, then a random subset of the features is selected before every split. Of course, this means that a particular variable might not be selected, so this might cause differences at every run. Let's assume that the default value None is provided.
If all the features can be used at every run, there's no randomness anymore in the selection before the split. But even if the default value best is provided for splitter, it would be wrong to assume that there is a single best variable
at each node: it's possible that two (or more) different variables are tied, i.e. they would provide an equal improvement in the criterion chosen (by default mse).

Conclusion: assuming you used the default None for max_features, the fact that you obtain different trees at every run means that there are frequently ties between variables with your data. If there are only minor differences in the nodes at the bottom of the tree then it's not significant, but if there are many changes including close to the root of the tree then it means that the model is unstable. The latter case is usually a bad sign, possibly none of the features is really helpful to predict the value and the model overfits.

Also, if I use integer for random_state and run the decision tree, it gives me a different tree for each random_state setting. Which tree should be selected in case of so many trees?

The possibility to select a random state is not intended to select a particular tree, it's intended to make the experiment reproducible. So in my opinion you shouldn't select one of the trees like this. Instead of selecting an arbitrary tree, you could try to investigate why the model is unstable and fix it if possible. The first step is to actually evaluate the model(s) on some test data: if there is a problem such as overfitting, it will be visible in the performance.


Identifying “clusters” or “groups” in a matrix I have a matrix that is populated with discrete elements, and I need to cluster them (using R) into intact groups. So, for example, take this matrix:

[A B B C A]
[A A B A A]
[A B B C C]
[A A A A A]

There would be two separate clusters for A, two separate clusters for C, and one cluster for B.

The output I'm looking for would ideally assign a unique ID to each cluster, something like this:

[1 2 2 3 4]
[1 1 2 4 4]
[1 2 2 5 5]
[1 1 1 1 1]

Right now I wrote a code that does this recursively by just iteratively checking nearest neighbor, but it quickly overflows when the matrix gets large (i.e., 100x100).

Is there a built in function in R that can do this? I looked into raster and image processing, but no luck. I'm convinced it must be out there.

What do you think is distance measure in your case?

I assume there are three dimensions here:

RowN (row number)
ColN (column number)
Value (value: A, B or C)

That means data you get from 4x5 matrix looks like:

Sample1 -> (1, 1, A)
Sample2 -> (1, 2, B)
...
Sample5 -> (1, 5, A)
Sample6 -> (2, 1, A)
...
Sample15 -> (3, 5, C)
...
Sample20 -> (4, 5, A)

Is value scaled? In other words, is A < B < C?

If yes, then

you can replace {A, B, C} with {0, 1, 2} (or may be {10, 11, 12}, if you want this difference be less important than RowN and ColN attributes)
normalize your data
use, for example, K-Means clustering algorithm (http://stat.ethz.ch/R-manual/R-patched/library/stats/html/kmeans.html) from stats R package

In that case the distance between two will be:

Sqrt( (RowN1-RowN2)^2 + (ColN1-ColN2)^2 + (Value1-Value2)^2 )

If value is not scaled (regular categorical variable), use some modifications of K-Means that work with categorical data.

So in case of 100x100 matrix you have 10000 observations and three variables, which is pretty trivial sample size.


Transforming target variable Building my machine learning model, I have a target variable that consists of dollar values ranging from $0.01$ to $1000000$+. I want to transform these dollar values in to a scale of 1 to 10 and then try and predict that level of 1 through 10.

What is the best approach to normalize my dollar values in to a scale of 1 to 10?

The answer depends on how your data is distributed, and what kind of error you can accept. If the distribution is close to uniform, and you care about having an absolute error of, say, $1000, data does not necessarily need scaling, or you could use standard scaling (min to max).

In any other case, you may want to apply a non linear transformation to your target variable. For instance, if the observations are evenly distributed among 0.01, 0.1, 1, 10, 100, etc., or most specifically when you care about relative error (a certain percentage of the target value) I would recommend to use a log transformation. This is because an error of 10% will always represent 0.1 unit after being log-transformed, whatever the target variable value is.

Most cases, if not all, will lie in the second case.


Similarity Matching Algorithm I am looking for help on identifying a class of algorithm. If I have a tabular training and test set I want to know the similarity of rows based on some numeric features. The training data would be labelled such that rows would be paired (or even grouped). The output for each row in the test/prediction set would be the most similar row and the probability that it would have been paired with that row.
In theory there could be a matrix with a score for the pair by pair comparison but I only need the highest for my use case.
This feels somewhat like collaborative filtering but I am not quite sure what to call this class of algorithm. Bonus points if you can point me to a python library.

It seems there does exist a python library for supervised ML to solve this
https://pypi.org/project/dedupe/


Why don't means of color channels average to equal flattened array mean I am experimenting with different types of mean subtraction on RGB images for a convolutional neural network. I tried calculating a single grand mean of all the training images. I also tried calculating a mean per color channel. The grand mean I got was 122.84 which sounds like a reasonable number. However, when I tried calculating the means per color channel I got  [4.86049, 4.86049, 4.86049].
First off, I find it strange that they are all the same number. Secondly, they seem ridiculously low. And thirdly and perhaps most confusingly, shouldn't the mean of the three color channels equal the grand mean?

Maybe it is a problem with my code? This is what I'm using. The images are in a numpy array of the shape (17611, 224, 224, 3)

trainImgsArr = np.random.rand(17611, 224, 224, 3)
grandMean = np.mean(trainImgsArr)
meanByChannel = np.mean(trainImgsArr, axis = (0,1,2))

Any ideas?

EDIT: Ah, ha! I think I have found the true channel means with the code

 ch1, ch2, ch3 = np.split(trainImgsArr, 3, axis = 3)
 np.mean(ch1)
 np.mean(ch2)
 np.mean(ch3)

Sorry if this was super basic I am very new to python. Could someone perhaps explain where I went wrong using a single np.mean call as in the first code bit?

You just understood numpy wrong (God bless those who understood it right). The correct code should have been:

trainImgsArr = np.random.rand(17611, 224, 224, 3)
grandMean = np.mean(trainImgsArr)
meanByChannel = []
for i in range(0, 3):
    avg = np.mean(trainImgsArr[:, :, :, i]
    meanByChannel.append(avg)

Basically I am doing splicing of the array,taking all pixels of a colour channel from all images and averaging them. Whereas you did something else entirely (your results are not reproducible). Axes (0, 1, 2) does not mean the colour channels it only means the index you are referring to and performing the required operation.

Ignore the convolution part but check the subscripts of h_n-1 you can access the those positions by varying the last argument in trainImgsArr.shape.


Two definitions of DCG measure I wanted to check the definition of Discounted Cumulative Gain (DCG) measure in the original paper Jarvelin and it seems it differs from the one given in the later literature Wang. Originally, for $n$ documents ranked from $r = 1, \ldots, p$, the $\text{DCG}_p$ is defined as
$$\text{DCG}_p = \sum\limits_{r=1}^{b} G_r + \sum\limits_{r=b}^{p}\frac{G_r}{\log_br},$$
where $G_i$ is the relevance (or gain) of the $i$-th document. So the measure depends on the logarithm base $b$. For ranks below $b$, i.e. $r<b$, gains are not penalized. If $b=2$, then we can write:
$$\text{DCG}_p = G_1 + \sum\limits_{r=2}^{p}\frac{G_r}{\log_2 r}.$$
It does not look the same as the one given on wikipedia, where the argument of the logarithm is shifted by $1$:
$$\text{DCG}_p = G_1 + \sum\limits_{r=2}^{p}\frac{G_r}{\log_2(r+1)}.$$

Where does this change come from? Why others use different metric?

I believe you are correct, that the paper and Wikipedia disagree. The paper's formula suggests you apply no discount at $r <= b$, which means both of the first two elements are not discounted.

The Wikipedia formula discounts the second element onward.

There's an impassioned statement in Talk about why the Wikipedia formula is right: https://en.wikipedia.org/wiki/Talk:Discounted_cumulative_gain

But I can't see why; it offers no reference other than observing that "it seems plainly wrong to not discount from the second element." I'll comment there.


What's the difference between Attention vs Self-Attention? What problems does each other solve that the other can't? As stated in the question above..is there a difference between attention and self attention mechanism ? Also additionally can anybody share with me tips and tricks about how self attention mechanism can be implemented in CNN?

Here's the list of difference that I know about attention (AT) and self-attention (SA).

In neural networks you have inputs before layers, activations (outputs) of the layers and in RNN you have states of the layers. If AT is used at some layer - the attention looks to (i.e. takes input from) the activations or states of some other layer. If SA is applied - the attention looks at the inputs of the same layer where it's applied.
AT is often applied to transfer information from encoder to decoder. I.e. decoder neurons receive addition input (via AT) from the encoder states/activations. So in this case AT connects 2 different components - encoder and decoder. If SA is applied - it doesn't connect 2 different components, it's applied within one component. There may be no decoder at all if you use SA, as for example in BERT architecture.
SA may be applied many times independently within a single model (e.g. 18 times in Transformer, 12 times in BERT BASE) while AT is usually applied once in the model and connects some 2 components (e.g. encoder and decoder).
SA is good at modeling dependencies between different parts of the sequence. For example - understand the syntactic function between words in the sentence. AT on the other hand models only the dependencies between 2 different sequences (for example, the original text and the translation of the text). While still the SA may be very good in translation task (see Transformer)
AT can connect 2 different modalities (i.e. text and image). SA is usually applied within a single modality but you still can join activations from 2 modalities into a single sequence and apply SA on it.
Generally SA mechanism looks like a more general to me as it can do more than AT. You can simulate AT with SA by just replacing/concatenating the input sequence with the target sequence that you want your attention to look at.

Some more notes

The term Multi Head Attention is often used with SA. But theoretically you can apply Multi Head approach to AT also.
The following terms: content-base attention, additive attention, location base attention, general attention, dot-product attention, scaled dot-product attention - are used to describe different mechanisms of how inputs are multiplied/added together to get the attention score. All these mechanisms may be applied both to AT and SA.
Key/Query/Value approach of attention calculation is usually applied to SA. But you can use it for AT also.


What do all the column names mean in the SentimentAnalysis package's analyzeSentiment() function in R? If you used the analyzeSentiment function from SentimentAnalysis in R:

library(tm)
library(SentimentAnalysis)
cor <- VCorpus(VectorSource(c('testing sentiment', 'this is great'))
sent <- analyzeSentiment(cor)
names(sent)

The names of the columns are:

"WordCount", "SentimentGI", "NegativityGI", "PositivityGI", "SentimentHE", "NegativityHE", "PositivityHE", "SentimentLM", "NegativityLM", "PositivityLM", "RatioUncertaintyLM" "SentimentQDAP", "NegativityQDAP", "PositivityQDAP"

Other than wordcount, what do all these labels mean?  I didn't see any definitions in the vignette or docs.

The different dictionaries are in the documentation for the function:

For example DictionaryHE is the "Dictionary with opinionated words from Henry’s Financial dictionary" (see DictionaryHE).

Example:

sentiment <- analyzeSentiment(document)   # Document is a Corpus
sentimentData <- sentiment$SentimentHE   # Generates analysis based on Henry's dictionary
plotSentiment(sentimentData) # Quick line plot of sentiment


Can I deal with a missing not at random column by creating a new column? (Feature engineering) Task: Binary classification
Example problem:
Let's say we have two feature columns A and B. A has no nulls and is a binary column if a user completed an action (=1), 0 if they didn't. For all users that completed the action, B is the resultant score. As a result, B has nulls for those that didn't complete the action (missing not at random as the nulls are dependent in B are due to A).
To deal with this missing not at random problem, is it possible to create a new variable that is equal to 1 if the user completes the action and achieves a certain score, 0 otherwise?
The column B is valuable but I'm trying to find a good way of dealing with the nulls.

A
B

1
94

0

0

1
45

Creating a new variable as you described would be redundant as it is a function of the other two variables. In other words it is not adding any information.
The below suggestion assumes the model cannot deal with missing values, but a lot of the best models (ex. xgboost is typically one of the best for classification) will deal with this in a smart way for you.
Just fill the missing values via imputation, ex. by using the mean score or training a model using all the other features to predict the scores, and then predicting the missing ones. The model will be able to take into account that some users didn't complete the action, so it won't just treat them as if they got the imputed score, since their value in column A is 0 and that is useful information.


df['date'] = pd.to_datetime(df['created_at']).dt.date

to_datetime: Convert argument to datetime. For example, if your column of "created_at" is a string column, it converts it to a datetime column

dt: Access object for datetime like properties of the Series values.


Imbalanced classes (balance of train, validation, and test) 1) I am currently trying to set up a feedforward neural network with highly imbalanced classes (binary classification) in which the number of observations of class 1 is very low (and the class of interest to predict).

The dataset is very large, so in order to make the network better to predict class 1, I have downsampled the class 0 observations in the training set (ending up with a training set of approximately 600,000 observations).

Right now, I am estimating the neural network using 'accuracy' as the metric to optimize, while using early stopping to optimize the accuracy on an unbalanced validation set (monitor='val_acc').

My question is, therefore, whether I also should downsample class 1 observations in the validation set? The final test will naturally be unbalanced with a low number of class 1 observations.

Edit:
My initial thought was to do use a function that takes the unbalanced classes into account, but the problem was that the dataset is too large, so I run into memory errors before I am able to run the neural network.
After studying different approaches, I've read several papers suggesting to balance the classes (in which the problem of memory was also solved).
My thought was that by balancing the classes, the accuracy function would make sense to use, but since I optimize the network directly on the validation within the model, I'm confused whether the validation set should also be balanced

2) Additionally, is there any common order on how to optimize a neural network (etc. first determine the number of hidden layers, number of neurons, and so)? A paper or similar arguing for its order of optimization would be very interesting (haven't managed to find one myself).

Thanks in advance!

If you can change the Loss function of the algorithm, It will be very helpful. There are many useful metrics which were introduced for evaluating the performance of classification methods for imbalanced data-sets. Some of them are Kappa, CEN, MCEN, MCC, and DP.

If you use python, PyCM module can help you to find out these metrics.

Here is a simple code to get the recommended parameters from this module:

>>> from pycm import *

>>> cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})

>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]

After that, each of these parameters you want to use as the loss function can be used as follows:

>>> y_pred = model.predict      #the prediction of the implemented model

>>> y_actu = data.target        #data labels

>>> cm = ConfusionMatrix(y_actu, y_pred)

>>> loss = cm.Kappa             #or any other parameter (Example: cm.SOA1)


Search for keywords within csv/word files which are listed in an excel sheet We have 1000's of csv, word files in different folders. Is there any open source data mining software which will look into the csv files for

1) First has to check for the Project names. If present..

2) Look for the County

3) Then, State -> Country

4)If all the conditions are satisfied, assign Order to the file.

I've brief experience working with Orange data mining software. Tried that, but couldn't even able to check project names within the csv's. Just installed Rapidminer and i can't even comprehend what i can do with it.

If anyone can shed light on what sort of software/process i have to look, will be greatly helpful.

Note: Most of the csv's & word documents are having more than 20 pages

Try something like the following in python:

import pandas as pd

df = pd.read_csv(path_to_file, header=0) # header=0 means the first row contains column names

# locate the records where none of the following columns are null ('~' means not)
mask = ~df[['Project Name', 'County', 'State', 'Country']].isnull()
df[ mask ]

I'm not entirely sure what you want to do with the 'Order' columns, but if it's to get just the "Order" column from the above statement, then just slightly modify it to the following:

df['Order'][mask]

If you want to set the value of "Order" for those columns that aren't null, then just do this:

# we need to adjust our mask because the current version won't work with the .loc function that we'll use below

mask = df[(~df['PROJECTNAME'].isnull())&(~df['County'].isnull())&(~df['State'].isnull())&(~df['Country'].isnull())]

df.loc[ mask, "Order"] = whatever you want

If you want to save all the orders where the specified columns aren't null to a separate csv, do this:

df[ mask ].to_csv(name_your_file)


Good formula for turning star reviews into upvotes I want to turn reviews of up to 5 stars and the number of reviews into upvotes. What's a good algorithm for doing this?

A venue with 10 reviews total with a 5-star average rating should obviously get more upvotes than a venue with 10 reviews total with a 3-star average rating. Also, a venue with 60 ratings and a 4-star rating should probably get more upvotes than the one with 10 reviews and a 5-star rating.

I need this rating to be based off of the total number of reviews and the average star rating, but I would also like the number to stay below a variable number (for example, say upvotes stay below 100, but I can also plug in 200 and it would stay below 200).

My recommendation would be to explore some statistical approach to represent the reviews/rating pair as significance. For example, to translate the [# reviews, rating] into some test statistic type model, such as Student's t for starters.

Considering your example numbers, some approaches can be:

1)

>>> 5/(1/sqrt(10))
15.811388300841896
>>> 3/(1/sqrt(10))
9.4868329805051381
>>>
>>>
>>> 4/(1/sqrt(60))
30.98386676965934
>>> 5/(1/sqrt(10))
15.811388300841896

2) Or, diving deeper into stats,:

>>> (5-3)/(sqrt(1/10+1/10))
4.4721359549995796
>>> (4-5)/(sqrt(1/60+1/10))
-2.9277002188455996

(you will need to do some work on the alpha level to get at the significance in ex 2 above)

You can see how these work out for you; if too basic/too many inaccurate assumptions/some other filtering that you need more resolution in your model/etc., you can explore ways to better represent:

the sample standard deviation (assumed to be 1 in my examples above), or
the distribution (assumed Gaussian in examples above), or
the appropriate statistical test (plow through Wikipedia), or
etc

Point is you can continuously refine your model, depending how much resolution you need. Only you can make that call. Hope this helps!


Feed data into Keras LSTM layer I'm trying to understand how to feed data into LSTM layer of Keras, but I'm in trouble and I don't understand how to do it.
I've have a dataset composed by words and each words is embedded with a vector of 839 elements, so the shape of my dataset is (x, 839).

I want to feed my dataset into the LSTM layer, but I don't properly understand the 3D object wanted by Keras, composed by (batch_size, timesteps, feature). I want to feed one word per time into the LSTM, how can I do?

Thanks in advance.

UPDATE
I still receive an error about input shape:

  ValueError: Error when checking input: expected lstm_11_input to have
  shape (2, 839) but got array with shape (839, 1)

I'm using batch_size_shape(batch_size, timesteps, feature) at the moment.
This is the code:

class KendallTauHistory(Callback):
  def __init__(self, dataset, y_true, groups):
    self.y_true = y_true
    self.dataset = dataset
    self.groups = groups

  def on_epoch_end(self, epoch, logs=None):
    predictions = self.model.predict(self.dataset)
    predictions = predictions.flatten()
    predictions = list(map(lambda element: element + np.random.uniform(0.0, 1.0) * 0.02 - 0.01, predictions))
    # For batch training
    ranked_predictions = np.array([])
    kendalls = np.array([])
    start_range = 0
    for group in self.groups:
        end_range = (start_range + group[1]) # Batch is a group of words with same group id
        batch_predictions = predictions[start_range:end_range]
        batch_labels = self.y_true[start_range:end_range]
        batch_predictions = list(map(lambda element: element + np.random.uniform(0.0, 1.0) * 0.02 - 0.01, batch_predictions))
        ranked_predictions = np.append(ranked_predictions, np.floor(rankdata(batch_predictions)))
        kendalls = np.append(kendalls, kendalltau(batch_labels, batch_predictions))
        start_range = end_range
    #self.y_true = self.y_true[0:len(ranked_predictions)]
    print('\nORIGINAL LABELS: {0}\n'.format(self.y_true))
    print('PREDICTED LABELS: {0}'.format(ranked_predictions))
    print("\nEpoch Kendall's tau: {0}".format(np.mean(kendalls)))

model = tf.keras.Sequential()
model.add(LSTM(units=10, batch_input_shape=(None, 2, 839)))
model.add(Dense(15, activation='sigmoid'))

model.summary()

model.compile(loss=listnet_loss, optimizer=keras.optimizers.Nadam(learning_rate=0.000005, beta_1=0.9, beta_2=0.999))
real_labels = np.array([])
losses = np.array([])

with tf.device('/GPU:0'):
  model.fit(training_dataset, training_dataset_labels, epochs=10, workers=10,
            verbose=1, callbacks=[KendallTauHistory(training_dataset, training_dataset_labels, groups_id_count)])

Take the example of time series: $\mathbf{x}1,\mathbf{x}2,\ldots,\mathbf{x}10$ where each $\mathbf{x}i$ is let 5 dimensional. The 'timestep' here will be the window chosen such that value at time instant is dependent on previous $p$ lags. So data passed to LSTM will be of the form $\mathbf{x}1,\mathbf{x}2$ as input with $p=2$ lags. In your case as well, each word is dependent on previous words. So timestep will be the number of previous words you need to pass to LSTM in context of current words. Batchsize argument is the number of examples after which the backpropagation will happen. It is a free parameter you can control it but it should be completely divisible by number of training examples. Feature will be 839, the length of embedding vector.


Adam Optimiser First Step Plotting the paths on the cost surface from different gradient descent optimisers on a toy example, I found that the Adam algorithm does not initially travel in the direction of steepest gradient (vanilla gradient descent did). Why might this be?
Later steps where affected by momentum etc but I would assume these effects wouldn’t come into play for the first few steps.

These are the equation of Adam [Ref - Dive Into Deep Learning]
\begin{aligned}
       \mathbf{v}_t & \leftarrow \beta_1 \mathbf{v}_{t-1} + (1 - \beta_1) \mathbf{g}_t \\
       \mathbf{s}_t & \leftarrow \beta_2 \mathbf{s}_{t-1} + (1 - \beta_2) \mathbf{g}_t^2
\end{aligned}
\begin{aligned}
\hat{\mathbf{v}}_t = \frac{\mathbf{v}_t}{1 - \beta_1^t} \text{ and } \hat{\mathbf{s}}_t = \frac{\mathbf{s}_t}{1 - \beta_2^t}
\end{aligned}
\begin{aligned}
\mathbf{g}_t' = \frac{\eta \hat{\mathbf{v}}_t}{\sqrt{\hat{\mathbf{s}}_t} + \epsilon}
\end{aligned}
\begin{aligned}
\mathbf{x}_t \leftarrow \mathbf{x}_{t-1} - \mathbf{g}_t'
\end{aligned}

The first two are the accumulation of momentum and the second moment of the gradient
The second set is for correction of initial bias
Last two are parameter update

Initial values are - [Ref - Arxiv Paper]
\begin{aligned}
       \mathbf{v} = \mathbf{s} = 0; \mathbf{t} = 1^{**} ; \beta_1=0.9 ; \beta_2=0.999 ;  \epsilon = 10^{-8}
\end{aligned}
Note - ** - It is initialized with 0 but incremeted in the loop before any other operation
These defaults will make,
\begin{aligned}
\mathbf{g}_0' = \eta \text{  (approximated for  } \epsilon\text{ )}
\end{aligned}
So, the initial movement will not be proportional to the Gradient.


Is there a Hyperopt equivalent for optimization in R? I've used Hyperopt in Python, but I'm looking for a package with similar capabilities in R. Does a package like this exist?

Give a try to DEoptim(). This package might solve your problem. For the documentation and more information visit-

Documentation
CRAN-R : DEoptim()

Hope it helps!


Are CNNs insensitive to rotations and shifts in images? Can CNNs predict well if they are trained on canonical-like images but tested on a version of images that are little bit shifted?

I tried it using mnist dataset and found the contrary. The accuracy of the test set that was shifted was very low as compared to MLPs.

If you use max-pooling layers, they may be insensetive to small shifts but not that much. If you want your network to be able to be invariant to transformations, such as translations and shifts or other types of customary transformations, you have two solutions, at least as far as I know:

Increasing the size of data-set
Using spatial transformers

Take a look at
What is the state-of-the art ANN architecture for MNIST and Why do convolutional neural networks work.

Thanks to one of our friends, another way is to use transfer learning after data-augmentation.


How to best visualize or capture time interval between lab measurements? I have a table like as shown below
subject_id   lab_test_id    lab_test_date       value    difference

"10005606"  "20364112"  "2143-12-06 02:32:00"   "1.3"   "13:10:00"
"10005606"  "20364112"  "2143-12-06 15:42:00"   "1.3"   "02:02:00"
"10005606"  "20364112"  "2143-12-06 17:44:00"   "1.3"   "10:11:00"
"10005606"  "20364112"  "2143-12-07 03:55:00"   "1.3"      NULL
"10005866"  "20364112"  "2149-10-01 19:30:00"   "2.1"   "07:42:00"
"10005866"  "20364112"  "2149-10-02 03:12:00"   "2.2"   "08:51:00"
"10005866"  "20364112"  "2149-10-02 12:03:00"   "2.1"   "04:59:00"
"10005866"  "20364112"  "2149-10-02 17:02:00"   "1.6"     NULL

the difference column indicates the time difference between each of the lab tests
Now, the doctor likes to know how often these measurements are done for each patient?
How can I best convey this information? Should I capture the average difference between lab tests?
For example, for patient = 10005606, it is 8 hours. Meaning adding all different values and divide by total number of records. 25 hours, 30 mins (approx)/3 records = 8 hours approx
Is there any other better way to represent this using median or any other measure etc?
Can guide me with this?

The average or median seem reasonable here. A boxplot might be a good choice of visualistion:

The y-axis could be your time dimension, and boxplot shows the normal distributions of differences, including the median value and outliers. You could then compare certain measurements against this. But it would of course depend on if you have enough data in your sample to represent the total population.
Or simply calculate the standard deviation of differences, and report this with your median (the mean value may be warped by outliers).


How to convert trained data (feature extracted) into a prediction model? Background:

I am analyzing and labeling some log data. (parsed already, sample data below)
I have extracted the major features of data.

For examples, the classification results ("1 - normal" or "0 - anomaly") largely depends on columns "duration", "mean", "std". For example,

records/sec usually <= 10
std usually < 0.5   , etc

Then I want to convert current result into a prediction model, in order to classify the future log data.

Sample data:

| ID |        datetime       | No. of Records | duration(s) |  mean | std  |  labels |
| 1  |  26/7/2019 8:06:00 PM |       5        |     1.0     |  0.33 | 0.47 |  normal |
| 2  |  ...                                                                | anomaly |
...
| 1,000,000 |  ...                                                         |  normal |

Question:

How to convert such parsed & feature extracted & labelled data into a prediction mode (or prediction function)?

I am NOT asking the theory. I have read some articles or projects of machine learning models. But many of them introduces too high-level. I still don't understand how to implement those machine learning models. I need to convert them into a real prediction function running in the computer.

I am asking the detailed / hands-on implementation steps. Maybe it becomes a call function f(x1, x2, x3, ...) or API in the code? The new data may be the input of a function call? The output will be normal or anomaly in this case.
In other words, how to present a prediction model in the form of code?

This question is insanely broad… you should better go for an API documentation/tutorial... Python is usually the preferred porgramming language and two major API are used: scikit-learn for machine learning and Keras/tensorflow for Deep learning… (but there are quite a few more)

I'll give you some links... You should firstly consult these kind of sources of information and coming here later on with more specific and detailed issues.

Scikit-learn examples:

https://scikit-learn.org/0.20/auto_examples/plot_anomaly_comparison.html (for your problem)

https://towardsdatascience.com/5-powerful-scikit-learn-models-e9b12375320d (general)
Really good blog with Machine/Deep Learning examples:

https://machinelearningmastery.com/blog/

https://machinelearningmastery.com/introduction-python-deep-learning-library-keras/


Keras: ValueError: Input 0 is incompatible with layer lstm_1: expected ndim=3, found ndim=2 I am using Keras functional API to write an LSTM model but It throwing an error can somebody please help below is the code for the model the output shape is 65. I am using Keras 2.2.4 and TensorFlow 1.15
event_num = 65
droprate = 0.3
vector_size = 572

def DNN():
    train_input=Input(shape=(vector_size*2,),name='Inputlayer')
    train_in=LSTM(512)(train_input)
    # train_in=BatchNormalization()(train_in)
    train_in=Dropout(droprate)(train_in)
    train_in=LSTM(256)(train_in)
    # train_in=BatchNormalization()(train_in)
    train_in=Dropout(droprate)(train_in)
    train_in=Flatten()(train_in)
    train_in=Dense(event_num)(train_in)
    out=Activation('softmax')(train_in)
    model=Model(input=train_input,output=out)
    model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
    return model

Your post is a duplicate : Here someone asked the same question.
Have a look at the answers, you can also try this link that gives a detailled explanation about how to reshape inputs for LSTM layers.


Git for Deep Learning - what are the best tools for versioning/tracking machine learning experiments? I'm looking for a tool to track the results of several experiments/iterations in machine learning.

Training a model can take days/weeks, so it's essential to track its performance and be able to easily replicate results and roll back to previous versions.
I'm looking for a tool that makes the process easy and streamlined like git did for version control.

Useful features would be:

open source with a license that allows commercial application
self-hosted [all the information should reside on computers we own]
python-friendly (ideally, tensorflow-friendly as well)
can store both hyperparameter values as well as references to specific datasets (the latter would be useful to track the effect of hard negative mining, etc.)
cluster-friendly
include basic plotting (it's useful to visualize learning curves)
automatic testing on multiple datasets
user tracking [like "blame" in git]

One such tool is Polyaxon. I'm using it extensively and it's really helping the experimentation pipeline.

It has features like experiment queuing, hyperparameter tuning, clustering, could be self-hosted or on the cloud, plots the metrics, it is independent of the framework and so on ...

I definitely recommend it.


The truth value of an array with more than one element is ambiguous I was trying to print  the index  of each of the maximum probability of the probability
 Array, While Executing the following code I get the error

" The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()"

for prob_ in prob:
        max_prob=max(prob_)
        index=clf.classes_[prob.index(max(prob_))]

where the prob is an array

I had the same problem in my code:

c = max(
[np.dot(theta[j], X[i]) for j in range(theta.shape[0])]
) / temp_parameter

I ran debugger and figured out that theta[j], X[i] were not the same shape, so dot product gave an ndarray instead of scalar as a result. max() function was a bit confused about it, because it does not work with ndarrays.
So, the conclusion is:

if prob_ is an ndarray, please, use numpy.max(), it should definitely work:

[In]: f = np.array([[1, 1, 1], [2, 2, 2]])
[In]: max(f)
[Out]: Traceback (most recent call last):
  File "<input>", line 1, in <module>
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
[In]: np.max(f)
[Out]: 2
if prob_ is a list, check whether its components are scalars or lists:

[In]: f = [[1, 1, 1], [2, 2, 2]]
[In]: max(f)
[Out]: [2, 2, 2]
[In]: g = [numpy.array([1, 1, 1]), numpy.array([2, 2, 2])]
[In]: max(g)
[Out]: Traceback (most recent call last):
  File "<input>", line 1, in <module>
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()


How to train f(x)=x*x using Artificial neural network? let's take some training data of size 100

x_input = [1,2,3,4,.....,100]
y_label = [1,4,9,16,....,10000]

Now, let's consider that we don't know the function f where f(x_input) = x_input2
How should we train it?

The answer to your question lies here.

Personally, since I can see the high autocorrelation of your input signal x (obvious that x(t+1) = x(t) + 1), you could treat it as timeseries and use an LSTM-RNN to model the nonlinear function y. For simplicity, you could always treat your data as iid and, under this assumption, use an MLP.


Optimizing Expensive Functions I'm trying some different techniques to optimise a Boosted Gradient Regressor by using an evolutionary programming technique to try and find the most efficient set of features. So far I've been having some good results (Been able to remove 65% of the features with an increase in accuracy), but I'm not impressed with how expensive each epoch is to run (in terms of time). Currently, this is a very expensive problem to optimise, 2^79 as there are a maximum of 79 features to choose from, which is 6.0446291e+23 possible feature permutations.

So far, each individual in my population has a binary encoded gene, where 1 = use this feature, and 0 = do not use this feature. Each individual in a population is evaluated by running the boosted regressor with the selected features, and then the RMSLE is calculated. I start to see good results around 3 hours into the optimisation function (I'm not doing any parallel computing).

I've been doing a little bit of research into how I can attempt to "predict" how good my boosted regressor will be without actually needing to evaluate its performance. So far I've found the following techniques:

Problem Approximation. I can see how this would be useful in some specific cases, but as I'm not really able to reduce the complexity of evaluating how accurate my boosted regressor is it seems irrelevant.
Functional Approximation. I think this is quite an interesting technique. They whole premise is to approximate the fitness of an individual without actually needing to evaluate the function
Fitness Inheritance. This also seems rather interesting, and perhaps the most efficient of the methods listed above? Individuals are clustered into specific groups, and then the "represented" individual of each cluster has its fitness evaluated, and then the remaining individual's fitness are approximated by upon a distance measurement.

I could see some issues with these techniques though. If the combination of features used don't have a linear relationship with the output, then surely the function approximation would have a hard time approximating the potential fitness of an individual? I'm a bit stumped on what to further look at.

According to the two papers (Limited Evaluation Evolutionary Algorithm (LEEA) and Limited Evaluation Cooperative Co-evolutionary Differential Evolution Algorithm (LECCDE)) you can approximate the result of the evolutionary algorithms by applying the algorithm not on the whole dataset but on mini-batches like in SGD (Stochastic Gradient Descent). This significantly reduces computation time (about 25 times in LECCDE paper for an ANN on MNIST dataset).

According to Prellberg 2018 Limited Evaluation Evolutionary Optimization the training times of LE algorithms is still 10 times longer than the Adam variation of stochastic gradient descent. In your case it is not possible to calculate gradients.

The following changes to the evolutionary algorithm are proposed in the first paper (LEEA):

Use mini-batching while ensuring the diversity of the expected outputs.

  Ideal mini-batch size in LEEA was determined experimentally to be 2. ...To apply the idea of diversity in minibatches, the output range is divided into two equally sized sections, and examples selected so that each mini-batch contains
  one example for both sections of the range. A new mini-batch is randomly generated at the beginning of each generation.

Use ancestor fitness for evaluation.

  f'=(f_p1 + f_p2)/2*(1-d) + f.

  Where f' is the individual's modified fitness. f is the fitness of the
  individual against the current mini-batch. f_pn is the fitness of
  parent n. d is a constant decay value.

The second paper (LECCDE) points to the following speed-up using Limited Evaluation algorithms:

LECCDE vs CCDE for different datasets:

Dataset    Parameters   Speedup             Accuracy
WBC        1652          1.10 times faster  1.2% worse
ESR        9052          2.05 times faster  0.3% worse
HAR        28406         4.00 times faster  0.2% worse
MNIST      47710        25.3  times faster  unknown


Can I create random forest with RandomForestClassifier which will consist the same trees? Based on answers to this question, I should be able to build a random forest with all the same trees by using bootstrap = False, max_features = None, random_state = 42 parameters.

I wrote quick code to test it, and it seems that different trees are created.

Is it possible to create a random forest using RandomForestClassifier which will produce the same trees?

You want to create a random forest where all decision trees are the same. (I am trusting you are trying to do this as an educational exercise, not as something serious. :))

You tried to do this by setting random_state and cited my answer from

Why `max_features=n_features` does not make the Random Forest independent of number of trees?

Indeed, as you discovered, my answer is not fully correct.

I have edited it to add

  While this removes the stochasticity component of the training, the decision trees would still be different. The thing is that sklearn ensembles generate a new random seed for each child based on the random state they are given. They do not pass along the same random_state.

  You can see this is the case by checking the _set_random_states method from the ensemble base module, in particular this line, which propagates the random_state across the ensembles' children.

As you can see, sklearn uses the random_state you give RandomForest to create new random states for each child,

for key in sorted(estimator.get_params(deep=True)):
    if key == 'random_state' or key.endswith('__random_state'):
        to_set[key] = random_state.randint(MAX_RAND_SEED)

Therefore they will be different. This is done presumably because heterogeneity is what makes an ensemble powerful. Notice this is not specific to RandomForest, but to any sklearn ensemble.

Sorry for my incorrect answer to the question you cite. If you want, you can use the random forest implementation. That one is completely homogeneous.


Neural network for variable length data classification How can I create a network which can predict labels of variable lengths data:

Training data:

label1: abcdeaefafere
label1: afdfdofdjfdjdfdofdj
label1: dffdpodfdajfdjdddfddfd

label2: reorefdfpreperpe
label2: rexcxfrerperuetupterer
label2: erfdfdrpoeregjroeptreereter
...
...

test data:

fdldjffdjfjdfd
xcdjeioreweoforpeeedfdfd
...
...

Please note these are sequences belonging to different classes and this is a classification problem. I am not trying to predict future data for these sequences.

Thanks for your insight.

Each input sequence should be padded to the same length. The most common method is to find the longest sequence and then add zeros to all shorter sequences. Most Deep Learning frameworks will have a built-in function to do this.

Consistently sized-input data allows common neural networks models (e.g., Convolutional Neural Network and Recurrent Neural Network) to be fit in a straight-forward manner.


Feature engineering from date, mean and standard deviation I have a multi class classification problem where I should predict the passengers for flights (0-7 classes). The training set consists of the following features:

Date of the flight
Mean of the weeks that the passengers bought their tickets
Standard Deviation of the above

I extracted from the date, the day of the week, the month and if the flight is in a high season. What other features could I extract from the date? How could I use the mean and standard deviation to create new features?

Date fields are quite interesting data since the limit of what you can "feature engineer" with them is your imagination. However, it is difficult to know a priori if one of them will improve your model before you try it.

Here some ideas:

Year
Month (1-12)
Day of Month (1-31)
Day of week (1-7)
Week of year (1-52)
The quarter of the year (1-4)
Is it weekend or weekday (0,1)
Season (Winter, Spring, Summer, Autumn)

If you have also the time on the date field, then you can try these:

Hour of day (1-24)
Part of the day based on eating habits (breakfast, lunch, dinner etc)

If you have the destination of the flight, then you can use an external source and get the holidays or specific "big events"

Is the flight during a holiday (0,1)
What is the distance in days from the closest holiday (before and after the flight)
Any big event on the destination like Olympic Games, Superbowl, Football finals etc


What to do with stale centroids in K-means When I run Kmeans on my dataset, I notice that some centroids become stale in the they are no longer the closest centroid to any point after some iteration. Right now I am skipping these stale centroids in my next iteration because I think those centroids no longer represent any useful set of the data, however I wanted to know if there are other reasonable ways to deal with these centroids.

k-means finds only a local optima. Thus a wrong number of cluster or simply some random state of equilibrium in the attracting forces could lead to empty clusters. Technically k-means does not provide a procedure for that, but you can enrich the algorithm with no problem.

There are two approaches which I found that are useful:

remove the stale cluster, choose a random instance from your data set and create a new cluster with centroid equal with the chosen random point
remove the stale cluster, choose the farthest distant point from any other centroids, create a new cluster with centroid in that point

Both procedures can lead to indefinite running time, but if the number of this kind of adjustments is finite (and usually it is) that it will converge with no problem. To guard yourself from infinite running time you can set an upper bound for the number of adjustments.

The procedure itself is not practical if you have a huge data set a a large number of clusters. The running time can became prohibitive.

Another procedure to decrease the chances for that to happen is to use a better initialization procedure, like k-means++. In fact the second suggestion is an idea from k-means++. There are no guarantees, however.

Finally a note regarding implementation. If you can't change to code of the algorithm to make those improvements on the fly, your only option which comes to my mind is to start a new clustering procedure where you initialize the centroid positions for non-stale clusters, and follow procedures for stale clusters.


Should I use epochs > 1 when training data is unlimited? If I have virtually endless training data (it's synthesized) is there still purpose in having epochs? I.e. training on the same samples multiple times?

If the data is unlimited, how would you have an epoch to begin with? For example, if you are analyzing tweets, you could never finish an epoch will all the tweets, since there will be an endless supply of new tweets. A much better approach will be to do some online or streaming learning.

Would it make sense to create a subset by ignoring new incoming tweets or data in general?

That really depends on your problem. For example if you have a datastream of tweets, or any other where new data is being produced in real time, you would miss out any trends or patterns that emerged after you sampled from the stream. Are those missed patterns relevant to your problem? They may or may not be.


Understanding Confidence Interval I am trying to understand the concept of  Confidence Intervals. What is the meaning of point estimates and confidence intervals? What I understood is the point estimate in confidence interval is basically the statistics of sampling distribution. Can we say that after using central limit theorem and finding population mean using CLT rather than giving point estimate we will give confidence interval?

When you look at a linear regression $y_i = \beta_0 + \beta_1 x_i + u_i$, you can estimate the (ex ante unknown) coefficients $\beta$ using matrix algebra $(X'X)^{-1} X'y = \hat{\beta}$.
A point estimate would be the "best guess" $\hat{y}=\hat{\beta} X$.
Each $\hat{\beta}$ is associated with uncertainty about the estimate, expressed by the standard error of the coefficient (see this post). The confidence interval intuitively says that you want to find the range in which the true $\hat{\beta}$ is with 95% likelyhood (assuming a normal distribution). In this case, you can calculate a confidence interval (for $\hat{\beta}$) by:
$$ CI_{0.95}^{\beta} = [\hat{\beta_i} - 1.96*SE(\hat{\beta_i}), \hat{\beta_i} + 1.96*SE(\hat{\beta_i})].$$
So you can say (under some assumptions) that the true value of the estimated $\hat{\beta}$ is within the confidence band with 95% probability. Note that in linear regression, when you say a coefficient is "statistically significant", this coincides with "the confidence band is strictly positive or negative" (does not cross zero).
Example in R:
Linear regression:
library("ISLR")
auto = ISLR::Auto
ols = lm(mpg~horsepower,data=auto)
summary(ols)

Result:
Coefficients:
             Estimate Std. Error t value Pr(>|t|)
(Intercept) 39.935861   0.717499   55.66   <2e-16 ***
horsepower  -0.157845   0.006446  -24.49   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 4.906 on 390 degrees of freedom
Multiple R-squared:  0.6059,    Adjusted R-squared:  0.6049
F-statistic: 599.7 on 1 and 390 DF,  p-value: < 2.2e-16

This tells us that when horsepower goes up by one unit, mpg goes down by -0.158 (point estimate). Now when we ask, what is the true effect (under quite some assumtions) with 95% probability, we look at the CI.
# Confodence interval
confint(ols)

Which gives:
                2.5 %     97.5 %
(Intercept) 38.525212 41.3465103
horsepower  -0.170517 -0.1451725

We can do this "manually" using:
# Get standard errors
sqrt(diag(vcov(ols)))

(Intercept)  horsepower
0.717498656 0.006445501

And we can calculate the CIs:
# Lower CI
-0.157845 - 1.96*0.006445501

# Upper CI
-0.157845 + 1.96*0.006445501

Which yields:
[1] -0.1704782
[1] -0.1452118

So we could say that the true effect of horsepower on mpg is between -0.17 and -0.15 (and since the CI does not "cross" zero, the effect is statistically significant, meaning p-value < 0.05).