practical improvements worth trying over plain LSTM in text classification? I have a dataset of about 1 million tweets corresponding to about 30,000 user accounts, labelled with binary data (classifying the tweet as written by a bot).

With that amount of data, I could use a LSTM-based NLP approach to take advantage of the sequence data in the text.

I would like to get the best accuracy possible, but don't know what models are available that could be combined with or build on LSTM-based approaches.

I also wonder, as one part of the data is sequential and the other stationary (account metadata), whether some kind of hybrid classifier could do better than using just an LSTM.

Are there any hybrid models or any other proven models that would be worth investigation further, to implement for my project?

A properly written LSTM network is quite powerful for NLP - why do you think that "combining" this with something else is the answer???

The project you're describing should be pretty easy to implement. The only barrier you have here is how much labeled data you can collect. Personally, I would focus on getting hundreds of thousands of examples, in a balanced dataset, and then feed that (with NLP approaches) into an LSTM network. You should be able to generate very good accuracy rates in a reasonable amount of time.


Is a BiLSTM layer required if we use BERT? I am new to Deep learning based NLP and I have a doubt -
I am trying to build a NER model and I found some journals where people are relying on BERT-BiLSTM-CRF model for it. As far as I know BERT is a language model that scans the contexts in both the directions and embeds words according to the context. Now my question is - if context is captured during word embedding with BERT, why do we need another layer of BiLSTM?

That layer isn't required indeed as it also encodes the sequence, albeit in a different way than BERT.
What I assume is that in a BERT-BiLSTM-CRF, setup, the BERT layer is either frozen or difficult to fine-tune due to its sheer size. Which is likely why the BiLSTM layer has been added there.


Retrieving date ranges with the r platform Using r if I have a data frame that contains dates and temperatures.

I would like to pull out all temperatures for January how would I go about

phrasing this.  I've tried may ways such as

louisville_weather <- filter(louisville_weather$DATE <= 2013-01-31)

If louisville_weather$DATE is formatted as POSIXct, the month can be extracted, thanks @RockScience, with

lvmonth <- format(louisville_weather$DATE, "%m")

It's a character variable.  So, filtering

louisville_weather <- louisville_weather[lvmonth == "01", ]

should extract your weather for January.

Hope you'll find it cold enough to balance the warmth of datascience.SX !


What does this formula in Glorot & Bengio mean? In this paper, on page 5, we find the formula
$$Var(z^i)=Var(x)\prod_{i'=0}^{i-1}n_{i'}Var(W^{i'})$$
I am really struggling to understand what is meant by this formula. I think at least some of the following are true:

We're dealing with a linear neural network, i.e. no activation functions.
$z^i$ is the output of one layer of the network, so for the very first hidden layer (closest to the input) of this network we would have:
$$Var(y)=nVar(x)Var(W)$$
where $y$ is the output vector of the first hidden layer, $x$ is the input vector and $W$ is the matrix of weights connecting that first layer to the input.

However, it's entirely unclear to me what these variances are. I thought maybe $Var(y)$ just meant the empirical variance of the vector $y$, i.e. the sum of the squares of the differences of the elements of $y$ from the mean of $y$, and likewise for $Var(x)$ and $Var(W)$, where the latter is just the variance of all of the entries of $W$. But under this interpretation the formula turns out to be false numerically, so I'm at a bit of a loss to understand what this equation is supposed to mean.

1. Meaning of the formula
$$Var(z^i)=Var(x)\prod_{i'=0}^{i-1}n_{i'}Var(W^{i'})$$
This formula expresses the variance of the values of the activation for each neuron $k$ of the layer $i \rightarrow z^i_k$. This value, under the assumptions that are going to be mentioned along this post, is the same for all the neurons in the layer $i$. This is why the authors express it as $Var(z^i)$ instead of a particular $Var(z^i_k)$. Just to simplify the notation
Note that this value is scalar, as well as the other variances involved in the equation: $Var(x)$ and $Var(W^i)$. So, just as a summary, these variances have the next meanings:

$Var(W^i)\rightarrow$ The variance of the random initialization of the weights.
$Var(x)\rightarrow$ The variance of each feature (which is asummed to be the same for every feature as we are going to see in the next lines).
$Var(z^i)\rightarrow$ As said, this is the variance of the activation of each neuron of the layer $i$.

So now let's see how to reach this formula.
2. Reaching the formula
2.1 Analysing some parts of the paper
Let's analyse some parts of that page of the article that answer the other questions and that are going to be also useful to understand the assumptions that are being made.

For a dense artificial neural network using symmetric activation function $f$ with unit derivative at $0$ (i.e. $f'(0) = 1$), if we write $z^i$ for the activation vector of layer $i$, and $s^i$ the argument vector of the activation function at layer $i$, we have:
$$ s^i = z^iW^i + b^i\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,z^{i+1}=f(s^i)$$

We can draw some conclusions given that paragraph:

The authors are considering an activation function subject to $f'(0) = 1$ in that part of the article. The fact that $f'(0) = 1$ means that locally the function $f(s^i)$ will behave linearly at the neighborhood of $s^i_k=0\rightarrow f'(s^i_k) \approx 1$, where $k$ is the index of the neuron.
As you said $\rightarrow z^i$ represents the activation output of one layer

The variances will be expressed with respect to the input,
outpout and weight initialization randomness

This means that the variances that they are going to use are going to be $Var(x)$, $Var(z^i)$ and $Var(W^i)$ respectively, and during the initialization (the weights are only spread randomly at the initialization).
The variance of the biases is not being considered as they are assumed to be initialized to the same value at the initialization $ \Rightarrow Var(b^i)=0$.

Consider the hypothesis that we are in a linear regime at the initialization, that the weights are initialized independently and that the inputs features variances are the same ($=Var(x))$.

Given this, we know that:

Linear regime $\Rightarrow f'(s^i_k) \approx 1$ as we saw above.
Weights are initialized independently $\Rightarrow Var(W^iW^{i+1})=Var(W^i)Var(W^{i+1})$
Inputs features variances are the same $\Rightarrow Var(x_k) = Var(x_{k+1}) \,\,\forall\,\, k$ where $k$ is the index of the input

2.2 Example
So now, let's compute for example the Variance of the activations of the first hidden layer ($i=1$) i.e. $Var(z^1_k)$:
$$ \begin{align}
Var(z^1_k) &= Var(f(s^0_k)) && (1)\\\\
&\approx (f'(\mathbb{E}(s^0_k))^2\,\,Var(s^0_k) && (2)\\\\
&= (f'(0))^2\,\,Var(s^0_k) && (3)\\\\
&= Var(s^0_k) && (4) \\\\
&= Var(W^0_k x + b_k^0) && (5)\\\\
&= Var(W^0_k x) = Var(w^0_{k1} x_1 + w^0_{k2} x_2 +... )  && (6)\\\\
&= Var(w^0_{k1} x_1) + Var( w^0_{k2} x_2) +...  && (7)\\\\
&= n_0Var(w^0_{kj})Var(x_j)  && (8)\\\\
&= n_0Var(W^0)Var(x) && (9)
\end{align}$$
Note that we would end up with the same expression of $Var(z^i_k)$ for every neuron $k$ of the layer $i \rightarrow $ Now we understand why $Var(z^i)$   represents the variance of the activation for each neuron in the layer $i$ i.e. $Var(z^i) = Var(z^i_k)$.
2.3 Justifications
The justifications for each step are:

$z^{i+1}=f(s^i) \Rightarrow z^{1}=f(s^0)$
The justification to this can be found in the Wikipedia post "Approximating the variance of a function"
$\mathbb{E}(s^0)= \mathbb{E}(W^0x) = \mathbb{E}(W^0)\mathbb{E}(x)=0$. This is because $x$ and $W^i$ are independent and $W^i$ is assumed to be randomly initialized with mean $0$.
Linear regime $\Rightarrow f'(s^i_k) \approx 1$.
$s^i = W^ix + b^i$
Bias values are initialized independently and $Var(b^i)=0$ as we mentioned above.
This is true because $Cov(w^0_{k1} x_1, w^0_{k2} x_2)=0$. Something that is proved at the end of the post.
(and 9) This is because the weights and inputs are independent, and because of the assumptions made by the authors, which are:
$$Var(x_j)=Var(x)\,\,\forall\,k \,\,\,\,\,\,\,\text{ and }\,\,\,\,\,\,\, Var(w_{jk})=Var(W)\,\,\forall\,\,k,j$$

3. Conclusion
Extending this reasoning to all the layers we end up with the equation given by the authors:
$$Var(z^i)=Var(x)\prod_{i'=0}^{i-1}n_{i'}Var(W^{i'})$$

Edit regarding the comments
The reasoning holds even if the features are not independent. This is because, we can proof that the covariance of the $z^{i+1}_k$ terms in the same layer $i$ is $0$. This also serves as an explantion to the step $(7)$ of the example above.
To see this let's compute $Cov(w_k^iz^i,w_j^iz^i)$ where $w_k^i$ and $w_j^i$ represent the vector of weights related to the neurons $k$ and $j$ of the layer $i$, with $k\neq j$:
$$\begin{align}
Cov(w_k^iz^i,\,\,w_j^iz^i) &= \mathbb{E}\left[(w_k^iz^i-\mathbb{E}(w_k^iz^i))(w_j^iz^i-\mathbb{E}(w_j^iz^i))\right]\\
&=\mathbb{E}\left[(w_k^iz^i)(w_j^iz^i)\right]\\
&=\mathbb{E}\left[w_k^iz^i\,w_j^iz^i\right]\\
&=\mathbb{E}\left[w_k^i\right]\mathbb{E}\left[z^i\,w_j^iz^i\right]\\
&=0^T\,\mathbb{E}\left[z^i\,w_j^iz^i\right]=0
\end{align}$$
Note that we can do $\mathbb{E}\left[w_k^iz^i\,w_j^iz^i\right]=\mathbb{E}\left[w_k^i\right]\mathbb{E}\left[z^i\,w_j^iz^i\right]$ because $w_k^i$ is independent of $z^i\,w_j^iz^i$
Then by extending this to the other neurons of the layer, we can confirm that: $Var(W^iz^i) = n_iVar(W^i)Var(z^i)\rightarrow$ We can reach the equation given by the authors.


Do we need the Preprocessing step on both Test and Train data sets? I have seen many people handle the missing or inconsistent data in both their test and train data sets. Sometimes they handle only the train data sets and sometimes they merge the train and test data sets and handle the missing parameter.

So, what is the best approach and What's the difference of these two? Does any approach effect the predictive model or it's just optional or good practice to handle missing data in both data sets.

There are a few things that you need to be careful with here.

You can do certain things when preprocessing data or performing data augmentation that can be applied across an entire dataset (train and validation). The main idea is not to allow the model to gain insight from the test data.

Time-series example

Missing data can be managed in many ways, such as simple imputation (filling the gaps). This is very common in time-series data. In your training data, you can fill the gaps using the previous value, the following value, the average of the data or something like the moving average. Where you must be careful is with violating the information flow through time. For example, in your test data, you should not fill gaps using a method that looks at data points in front of the emtpy time slot. This is because, at that point in time, you will not be able to do the same as you shouldn't know the future values.

Image data example

Looking instead at image data, there are data-preprocessing steps such as normalisation. This means just scaling the image pixel values to a range like $[-1, 1]$. To do this, you must compute the population mean and variance, which you then use to perform the scaling. When computing these two statistics, it is important not to include the test data. The reason is that you would be leaking information the dataset that is then used to train a model. Your technically knows things that it shouldn't; in this case, clues regarding the mean and variance of the target distribution.

People might also consider "missing data" to include imbalanced datasets; i.e. there are cases that you know of, but just don't appear in your dataset very often. There are some tricks to help with this, such as stratified sampling or cross-validation. The optimal solution would, of course, be to gather a dataset that more closely represents the problem at hand.


Using standard ML models for modeling a derivative when the data set only contains function values I want to model a process that is a function of time, $X(t)$. I have a data set which corresponds to a coarse sampling of the function values $X$ at different time points, $t_1, t_2, ...$. I have a analytic-derived model for the derivative $\frac{dX}{dt}$ = $F(X(t),G(t))$ which has a couple tunable parameters in the (known) functions $F$ and $G$. I've been able to tune these models with the following process:

Chose a set of parameters for the functions $F$ and $G$.
Do a forward-Euler method from the earliest time in my data set to the ending point in my data set, generating the $X(t)$ which corresponds to the above parameters in $F$ and $G$.
Use this $X(t)$, along with my measured data points to generate a model error.
Use the above modeled error to go back to (1), iterating and doing an optimization over the parameters in the expression for $\frac{dX}{dt}$ to minimize the error.

Note that in the above I have to deal with the function itself and not the derivative because my measured data is on a coarser time grid than the derivative needs to be computed. For example, my derivative changes meaningfully on a scale of $\Delta t$, but my data set might only have points every $5\Delta t$ to $10 \Delta t$. I cannot take the function values themselves from the data set and create a derivative myself because of this discrepancy in the time scales.

Now this process works pretty well, and I'm able to achieve a pretty good error. However, I'm wondering if there's a better way to model the derivative $\frac{dX}{dt}$ than this analytic model. For example, would a neural network or a regression tree be able to do a better job?

My question is, how would I do this in a computationally efficient way? Let's take a neural network as an example. I'm used to having inputs and their associated outputs when training a NN with standard packages. But the problem is, if I want to model the derivative, I don't have the derivative as an output to train on. I only know the function values and have only been able to compute a error to minimize on through the forward Euler integration. Of course I could treat the NN parameters as I did with the parameters above, and do a brute force minimization on them, but I obviously lose a lot of the efficiency of backprop in doing so (or, if I was doing this with a decision tree, would lose the logic of how regression trees are normally optimized).

Is there a better way for me to do this? Are there methods that exist to handle this problem? If anyone could point me in a fruitful direction it would be much appreciated.

This is actually really easy to implement in any deep learning framework with automatic differentiation capability.

You need two neural networks. One to approximate the $X$, another to approximate the $dX/dt$.

The first step is easy. You have samples of input and target for $X$.
Just follow the normal process to train you network to learn $X$
For the second step is that you don't have the target values (i.e. gradient) directly. The trick is that you already got an approximator for $X$ from the first step, so use that to get your target for the second network.

I am most familiar with pytorch, so I will use it as an example.

import torch
from torch.nn import Sequential, Linear, ReLU
from torch.autograd import Variable
from torch.optim import Adam
import numpy as np

x = np.linspace(0, 10, 101).reshape(-1, 1)
y = np.sin(x) # Using sin as an example

x = Variable(torch.Tensor(x), requires_grad=True)
y = Variable(torch.Tensor(y), requires_grad=False)

f = Sequential(....) # Build f approximator
f_optimizer = Adam(f.parameters())

for epoch in range(1000):
    loss= f(x) - y
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

# Build grad approximator
f_grad = Sequential(....)

# need a new optimizer to train f_grad
f_optimizer = Adam(f_grad.parameters())

for epoch in range(1000):
    # Directly backprop from the output to the gradient of the f
    model(x).backward()
    loss= f_grad(x) - x.grad # x.grad is the grad of f w.r.t x
    f_optimizer.zero_grad()
    loss.backward()
    f_optimizer.step()

f_grad # this is the gradient function you are looking for

(I don't have a machine with pytorch install at the moment, so don't expect the code to run with some debugging)


The effect of the image type and the image conversion on deep learning CNN model Does the type of the image affects (jpg, png, bmp) on the CNN deep learning algorithm?

Dose converting the image type affects on the CNN deep learning algorithm (ex. converting bmp to jpg or ppm to jpg)?

Thanks

I haven't seen this discussed anywhere really, so an interesting thought.

I think the answer will be that it doesn't matter. I think you should stick to one format (not important which, but don't mix them). Once the images are read into memory for processing and training a model, they are going to just be numbers in an array, regardless of the format they were loaded from.

Perhaps the numbers will differ slightly, but we usually normalise the input to something like the range [0, 1], so that wouldn't matter.

If your file formats change the image in a way, such that some features of the image are different (like the hue), then perhaps results will differ slightly, but the contents and spatial positioning of objects remain the same relative to themselves.

EDIT:

I should mention that converting between "image types" is nothing more than using a different compression algorithm to store the data. So when a photo is taken, the camera software compresses the raw sensor data into the target format (jpg, png, ...) and that format will have characteristic artefacts. Perhaps we don't recognise them, but they are there, and there is research done trying to reverse the information loss due to such compression algorithms. Here just a random example.


Does it make sense to train a CNN as an autoencoder? I work with analyzing EEG data, which will eventually need to be classified. However, obtaining labels for the recordings is somewhat expensive, which has led me to consider unsupervised approaches, to better utilize our quite large amounts of unlabeled data.

This naturally leads to considering stacked autoencoders, which may be a good idea. However, it would also make sense to use convolutional neural networks, since some sort of filtering is generally a very useful approach to EEG, and it is likely that the epochs considered should be analyzed locally, and not as a whole.

Is there a good way to combine the two approaches? It seems that when people use CNN's they generally use supervised training, or what? The two main benefits of exploring neural networks for my problem seem to be the unsupervised aspect, and the fine-tuning (it would be interesting to create a network on population data, and then fine tune for an individual, for instance).

So, does anyone know if I could just pretrain a CNN as if it was a "crippled" autoencoder, or would that be pointless?

Should I be considering some other architecture, like a deep belief network, for instance?

Yes, it makes sense to use CNNs with autoencoders or other unsupervised methods. Indeed, different ways of combining CNNs with unsupervised training have been tried for EEG data, including using (convolutional and/or stacked) autoencoders.

Examples:

Deep Feature Learning for EEG Recordings uses convolutional autoencoders with custom constraints to improve generalization across subjects and trials.

EEG-based prediction of driver's cognitive performance by deep convolutional neural network uses convolutional deep belief networks on single electrodes and combines them with fully connected layers.

A novel deep learning approach for classification of EEG motor imagery signals uses fully connected stacked autoencoders on the output of a supervisedly trained (fairly shallow) CNN.

But also purely supervised CNNs have had success on EEG data, see for example:

EEGNet: A Compact Convolutional Network for EEG-based Brain-Computer Interfaces

Deep learning with convolutional neural networks for brain mapping and decoding of movement-related information from the human EEG (disclosure: I am the first author of this work, more related work see p. 44)

Note that the EEGNet paper shows that also with a smaller number of trials, purely supervised training of their CNN can outperform their baselines (see Figure 3). Also in our experience on a dataset with only 288 training trials, purely supervised CNNs work fine, slightly outperforming a traditional filter bank common spatial patterns baseline.


ML algorithm for target minimization? Simplified question:
I have a dataset of how well certain agents perform on certain tasks, and based on this I've trained a model that can make predictions for newly incoming tasks. I'd like to make an algorithm that distributes tasks amongst agents to maximise performance.

One way would be to make a prediction for each scenario - each agent assigned to each new set of tasks (note that one agent can work on more than one task at a time) -, thus iterating through all possible solutions, and summing performance and finding the max.

This, however, would be very complex - to iterate through every possible combination an finding a maximum.

Is there a better way to optimise (maximise performance) without having to iterate through all possible solutions? How could I do that?

Original question:

Let's take a look at the following setup:

There are a number of "agents"
Agents work on "jobs"
Each agent might be better at certain jobs, but the jobs get assigned by random, and not based on performance (good performance = the least amount of time spent working)

I'd like to know if there's any ML algorithm that can learn how to assign jobs to agents as to minimise time spent working (maximise performance).

I know that with methods like regression or classification, or even deep learning you can teach the algorithm to predict a target variable, but in this case I do not want to reach a target variable, but want to increase performance (minimise time spent working). Is there such an algorithm/method that could "learn" from past performance reviews and assign new jobs to agents to maximize performance?

Edit:

A bit more formalised:

I teach an SVM (or a regressor, or a neural net), that will be able to predict how well a certain agent A1 performs on a certain (type of) task T1 (call this performance P1). So when a new task comes in, I'll be able to predict P1 based on A1 and T1. BUT! And here comes the question. Now that I can predict P1 based on A1 and T1, how do I use this knowledge to actually assign ALL the tasks - T1...Tn amongst all the agents A1...Am as to maximise the performance: sum(P1, P2, ..., Pi)?

Some ideas:1. Could you assign features for jobs and assignments? Performance would be the labels. Then you could simply use an SVM for example.
2. Given a user representation matrix X, each row i is an agent, and a job representation matrix Y, each j is a job representation, and a matrix of performance P, where every row stands for a user and every column how well the agent performed on that job, you could learn X and Y through optimization. Check this: https://en.wikipedia.org/wiki/Non-negative_matrix_factorization
I hope this helps.


Kernel PCA and K largest eigenvectors How can one prove that the optimal kPCA solution $a^*=\{a_1...a_K\}$ are the $k$-largest Eigenvectors of the (centered) kernel matrix $K$?

I referred to a lot of resources and couldn't find a proper explanation.

Let us say $K$ is an $m \times n$ matrix.
You want to find a vector $a$ that maximizes $\frac{||Ka||}{||a||}$.
Now maximizing $\frac{||Ka||}{||a||}$ implies maximizing its square:
$$\begin{eqnarray*}
\frac{||Ka||^2}{||a||^2} \\
= \frac{(Ka)^TKa}{a^Ta} \\
= \frac{a^TK^T Ka}{a^Ta} \\
= \frac{a^T(K^TK) a}{a^Ta} \\
= \frac{a^TSa}{a^Ta}
\end{eqnarray*}
$$
where $S = K^TK$ is always an $n \times n$ symmetric positive semidefinite matrix for all $K$
Now symmetric positive semidefinite matrices have several special properties one of which is that they always have REAL eigen values. Also the eigen vectors associated with these eigen values are always orthogonal to each other.
What this means is that if you locate the eigen vector for $S$ with the highest eigen value - you have basically found the vector that maximizes:
$\frac{a^TSa}{a^Ta}$
So in essence the eigen vector with the largest eigen value for $S$ or $K^TK$ maximizes both $\frac{a^TSa}{a^Ta}$ and the original $\frac{||Ka||}{||a||}$
Use the same analogy for the remainder eigen vectors - i.e the second best vector that maximizes $\frac{a^TSa}{a^Ta}$ is the eigen vector corresponding to the second highest eigen value.


Very low probability in naive Bayes classifier I have designed NB classifier from scratch in python for binary classification problem. There are total 220 records out of which 85 records belongs to 'Yes' class and 135 to 'No' class. My classifier is giving 88% accuracy.

So, whenever I calculate the posterior probability of one sample that belongs to 'Yes' class it is very low in terms of numbers. For eg., I'm making a prediction that whether the batsman is rising star (i.e. the probability of sample belongs to 'Yes' class ).

Here The posterior probability of being rising star i.e. P(RS) is very low in numbers something like 2.33E-8. But the posterior probability of being not rising star is also very low similar in the range of E-8 to E-16. Some features I'm using to calculate posterior probability are also small values in the range of 0.1 to 0.01.

My question is how to represent posterior probability i.e. P(RS) in terms of percentage. Like P(RS)=90%.

PS: I googled this problem and tried log method which returns a negative value.

I can't tell you for sure without you describing your calculation more or showing code, but my guess is you're not actually calculating the posterior probability here. I bet this is just the conditional likelihood, or at best the unnormalized posterior. Remember: the posterior calculation has a division component. Does yours? You're probably forgetting to divide by the "evidence".


Autoencoder feature extraction without validation set? I plan to use autoencoder for feature extraction, then use the latent vector for clustering.

My autoencoder performs very very well on my training set (loss small and reconstructed image look very similar to input) but shows lower performance on validation set.

Is it a good idea to NOT use validation set but only a training set? Since I can reach good performance extracted vector should be "good" features

If I use new data I'll need to retrain the model, but I should always be able to reach good performances since new data will be very similar to current data

To the best of my knowledge, if the performance is higher on the training set than on the validation set, you should probably be worrying about over-fitting.

I am assuming that your validation set, for the moment, is just a fraction of the data of the training set.

Notice that if the autoencoder is not performing well on your validation set, it is unlikely that you will reach good performance on new data "since new data will be very similar to current data", as already on current data you don't have good performance.

If you have to retrain the autoencoder every time you want to encode something, it may be not worth using the autoencoder in the first place.


Understanding why ensembling only improves marginally I have two models, A and B, trained on Imagenet. Their accuracies on Imagenet validation set are 35.6% and 28.64% respectively, while the accuracy of their ensemble (averaging their scores) is 35.68%. I am interested in finding out why the ensembling isn't effective here.

Specifically, I was going to inspect the confusion matrices for each model, but Imagenet has more than 1000 classes which makes this intractable. Another thing that was suggested to me is Mutual Information, but I can't figure out how to apply it in this context.

So, I have a two part question:

Why doesn't the accuracy of the ensemble degrade (to the average of
two accuracies) or improve?
Is there a way of visualizing/scoring
the output of the two networks to measure correlation?

Edit 1: Both are AlexNet models, but were trained with two different pre-trained weight initializations. The pre-trained weights themselves come from two different self-supervised tasks. Also, when these models were trained (initialized with respective pre-trained weights) on Pascal, there's is a significant boost in the accuracy. Thus, my quest to figure out how do you measure the correlation between models that are ensembling.

From what you are saying on can make up that you predict scores for each class and that the highest score is your class prediction.

A question you may ask is whether or not the class prediction is affected by the ensembling at all. For the sake of the example consider an example where Model A predicts a class always with a normalised score of 1.0 (i.e. full confidence, recommended reading). Suppose Model A is right 80% of the time. Model B however, is less certain. It places his bets on the top 5 classes that it deems likely with an equal normalised score of 0.2. Averaging the scores results in the same top predicted class as for Model A alone. You could easily check this by calculating the correlations between the predictions of the models with the ensembled predictions.


What are some standard ways of computing the distance between documents? When I say "document", I have in mind web pages like Wikipedia articles and news stories.  I prefer answers giving either vanilla lexical distance metrics or state-of-the-art semantic distance metrics, with stronger preference for the latter.

There's a number of different ways of going about this depending on exactly how much semantic information you want to retain and how easy your documents are to tokenize (html documents would probably be pretty difficult to tokenize, but you could conceivably do something with tags and context.)

Some of them have been mentioned by ffriend, and the paragraph vectors by user1133029 is a really solid one, but I just figured I would go into some more depth about plusses and minuses of different approaches.

Cosine Distance - Tried a true, cosine distance is probably the most common distance metric used generically across multiple domains. With that said, there's very little information in cosine distance that can actually be mapped back to anything semantic, which seems to be non-ideal for this situation.
Levenshtein Distance - Also known as edit distance, this is usually just used on the individual token level (words, bigrams, etc...). In general I wouldn't recommend this metric as it not only discards any semantic information, but also tends to treat very different word alterations very similarly, but it is an extremely common metric for this kind of thing
LSA - Is a part of a large arsenal of techniques when it comes to evaluating document similarity called topic modeling. LSA has gone out of fashion pretty recently, and in my experience, it's not quite the strongest topic modeling approach, but it is relatively straightforward to implement and has a few open source implementations
LDA - Is also a technique used for topic modeling, but it's different from LSA in that it actually learns internal representations that tend to be more smooth and intuitive. In general, the results you get from LDA are better for modeling document similarity than LSA, but not quite as good for learning how to discriminate strongly between topics.
Pachinko Allocation - Is a really neat extension on top of LDA. In general, this is just a significantly improved version of LDA, with the only downside being that it takes a bit longer to train and open-source implementations are a little harder to come by
word2vec - Google has been working on a series of techniques for intelligently reducing words and documents to more reasonable vectors than the sparse vectors yielded by techniques such as Count Vectorizers and TF-IDF. Word2vec is great because it has a number of open source implementations. Once you have the vector, any other similarity metric (like cosine distance) can be used on top of it with significantly more efficacy.
doc2vec - Also known as paragraph vectors, this is the latest and greatest in a series of papers by Google, looking into dense vector representations of documents. The gensim library in python has an implementation of word2vec that is straightforward enough that it can pretty reasonably be leveraged to build doc2vec, but make sure to keep the license in mind if you want to go down this route

Hope that helps, let me know if you've got any questions.


NLP and one-class classifier building I have a big dataset containing almost 0.5 billions of tweets. I'm doing some research about how firms are engaged in activism and so far, I have labelled tweets which can be clustered in an activism category according to the presence of certain hashtags within the tweets.

Now, let's suppose firms are tweeting about an activism topic without inserting any hashtag in the tweet. My code won't categorized it and my idea was to run a SVM classifier with only one class.

This lead to the following question:

Is this solution data-scientifically feasible?
Does exists any other one-class classifier?
(Most important of all) Are there any other ways to find if a tweet is similar to the ensable of tweets containing activism hashtags?

Yes, this is feasible.
One-class classification is a thing, but it is usually used in a context where it is hard or impossible to get negative samples. In your case, I would argue, you can quite easily get tweets that are not about activism, therefore you can render it as a binary classification, because you have data points of two classes or labels: 1 for tweets that are part of your class and another 1 for tweets that are not.
There are many ways to build a classifier, SVM is only one of them. You could also use a Naive Bayes algorithm, or as @Kasra mentioned a neural network model.
No matter what you use, you will have to organise your data such that you have samples of both classes: activism and non-activism within your set. This means that you should randomly pick tweets from your big dataset and manually check if they relate to activism, even if they don't have the hashtags in them that you used for identifying the activism tweets in the beginning.
Further, you have to think about the features that your classifier will use. The simplest might be the bag of words within the tweets, but you might also pre-process the tweet to exclude stop-words.
Depending on which algorithm you use, you might find that your classifier relies a lot on the presence of your particular hashtags as features for predicting the class. In this case it might struggle to identify other tweets without this hashtags as activism, even if they are activism. I would experiment with pre-processing the tweets in your entire dataset to remove those hashtags from the the tweets.


Why is my LSTM, stateful network getting 0 accuracy I'm building a NN to predict stock prices but I'm getting an accuracy of 0, I really don't know what hyper parameters to play with anymore but it seems to always get this accuracy

Build LSTM model

model.add(LSTM(
    50, activation="tanh", recurrent_activation="sigmoid",
    return_sequences=True, batch_input_shape=(batch, timesteps, features)))#stateful=True,
model.add(layers.Dropout(0.2))

model.add(LSTM(100, activation="tanh", recurrent_activation="sigmoid",
               stateful=True, return_sequences=True)) #stateful=True
model.add(layers.Dropout(0.4))

model.add(layers.LSTM(50, activation="tanh", recurrent_activation="sigmoid",
               stateful=True, return_sequences=False)) #stateful=True
model.add(layers.Dropout(0.2))

model.add(Dense(25))
model.add(Dense(1))

# Complie model
model.compile(optimizer="adam", loss="mean_squared_error",
              metrics=["accuracy", "mse"])

# Train the model
for i in trange(1, desc="Training the fuck out of this model"):
    model.fit(x_train, y_train,
              batch_size=batch, epochs=1,
              shuffle=False)#shuffle=False
    model.reset_states()```

#OUTPUT
```Epoch 1/1

   1/2918 [..............................] - ETA: 1:47:06 - loss: 0.0318 - acc: 0.0000e+00 - mean_squared_error: 0.0318
   2/2918 [..............................] - ETA: 56:57 - loss: 0.0246 - acc: 0.0000e+00 - mean_squared_error: 0.0246
   3/2918 [..............................] - ETA: 40:29 - loss: 0.0174 - acc: 0.0000e+00 - mean_squared_error: 0.0174
   4/2918 [..............................] - ETA: 32:15 - loss: 0.0135 - acc: 0.0000e+00 - mean_squared_error: 0.0135
   5/2918 [..............................] - ETA: 27:18 - loss: 0.0141 - acc: 0.0000e+00 - mean_squared_error: 0.0141
   6/2918 [..............................] - ETA: 24:00 - loss: 0.0150 - acc: 0.0000e+00 - mean_squared_error: 0.0150
   7/2918 [..............................] - ETA: 21:40 - loss: 0.0142 - acc: 0.0000e+00 - mean_squared_error: 0.0142
   8/2918 [..............................] - ETA: 19:48 - loss: 0.0125 - acc: 0.0000e+00 - mean_squared_error: 0.0125
   9/2918 [..............................] - ETA: 18:26 - loss: 0.0115 - acc: 0.0000e+00 - mean_squared_error: 0.0115
  10/2918 [..............................] - ETA: 17:20 - loss: 0.0108 - acc: 0.0000e+00 - mean_squared_error: 0.0108
  11/2918 [..............................] - ETA: 16:27 - loss: 0.0100 - acc: 0.0000e+00 - mean_squared_error: 0.0100
  12/2918 [..............................] - ETA: 15:42 - loss: 0.0094 - acc: 0.0000e+00 - mean_squared_error: 0.0094
  13/2918 [..............................] - ETA: 15:08 - loss: 0.0089 - acc: 0.0000e+00 - mean_squared_error: 0.0089
  14/2918 [..............................] - ETA: 14:32 - loss: 0.0083 - acc: 0.0000e+00 - mean_squared_error: 0.0083
  15/2918 [..............................] - ETA: 14:06 - loss: 0.0080 - acc: 0.0000e+00 - mean_squared_error: 0.0080
  16/2918 [..............................] - ETA: 13:39 - loss: 0.0076 - acc: 0.0000e+00 - mean_squared_error: 0.0076
  17/2918 [..............................] - ETA: 13:17 - loss: 0.0072 - acc: 0.0000e+00 - mean_squared_error: 0.0072
  18/2918 [..............................] - ETA: 12:58 - loss: 0.0068 - acc: 0.0000e+00 - mean_squared_error: 0.0068
  19/2918 [..............................] - ETA: 12:40 - loss: 0.0065 - acc: 0.0000e+00 - mean_squared_error: 0.0065```

Your code is absolutely fine. Also, the model is working as you can see the loss is continuously decreasing.

The thing is, this is a regression problem and accuracy is used for classification tasks. It just calculates how often predictions match labels by comparing if the classes are the same. See here for more details on how it works. So, just remove accuracy from your metrics and you are good to go.


Tools for visualisation: making a video of a sliding windows moving on the time series I have a Neural Network (an autoencoder) that gets as the input a time-window of length M from a time series of length N, (M << N), and transforms it to another same dimensional time-window.

In purpose of visualization, I want to make a video that shows input of the NN in the top of the frame and the corresponding output in the bottom of the frame, runnig the NN on consecutive windows.

I was wondering if you could introduce me some tools or give me some hints on this.

Using this, I've written a script for it.

%matplotlib notebook

import numpy as np
import matplotlib.pyplot as plt
import time

plt.ion()
size_x = 128
def pltts(ts1, ts2, ax, colors, lbl):
    x = np.linspace(0,2.5,size_x)
    if ax.lines:
        ax.lines[0].set_xdata(x)
        ax.lines[1].set_xdata(x)

        ax.lines[0].set_ydata(ts1)
        ax.lines[1].set_ydata(ts2)
    else:
        l1, l2 = ax.plot(x, ts1, colors[0], x, ts2, colors[1])
        fig.legend((l1, l2), ("BLUE", "RED"), loc='center', fontsize=fnt_siz-2)
        ax.set_title(lbl, fontsize=fnt_siz-2)
    fig.canvas.draw()

fnt_siz = 16

fig,ax = plt.subplots(2,1, figsize=(10,10))
fig.suptitle('Data', fontsize=fnt_siz)
ax[0].set_xlabel('Time', fontsize=fnt_siz-4)
ax[0].set_ylabel('Magnitude', fontsize=fnt_siz-4)
ax[1].set_xlabel('Time', fontsize=fnt_siz-4)
ax[1].set_ylabel('Magnitude', fontsize=fnt_siz-4)
plt.subplots_adjust(hspace=0.75)
# ax[0].grid(True)
# ax[1].grid(True)
for f in range(50):
    ts1 = np.random.normal(0, 4, size=(size_x,1))
    ts2 = np.random.normal(1, 1, size=(size_x,1))
    pltts(ts1, ts2, ax[0], ['b', 'r'], "ORGINAL")
    ts1 = 2*ts1
    ts2 = -3*ts2
    pltts(ts1, ts2, ax[1], ['b', 'r'], "TRANSFORMED")
    time.sleep(1)

This is a sample output:


Is training one epoch using mini-batch gradient descent slower than using batch gradient descent? I wonder whether one epoch using mini-batch gradient descent is slower than one epoch using just batch gradient descent.

At least I understand that one iteration of mini-batch gradient descent should be faster than one iteration of batch gradient descent.

However, if I understand it correctly, since the mini-batch gradient descent must update the weights by the number of the batch size in one epoch, the training would be slower than the batch gradient descent, which computes and updates the weights only once in one epoch.

Is this correct? In that case, is it worth worrying about the loss of the overall training time?

You are correct, there is more overhead to process the same amount of data because you do more weight updates and maybe preprocessing the batches in your generator will take more time as well. However, since you are doing more updates and if your batch size is big enough, the gradients of your mini-batches will approximate the gradients of your full batch fairly well and you will have gotten much closer to the optimum of your full dataset by splitting your full batch into mini batches than you would have gotten by doing one non-stochastic weight update.


Tree based feature transformation I'm new to data science field and interested in performing prediction using clickstream data. In Practical Lessons from Predicting Clicks on Ads at Facebook paper section 3.1, a method called Decision Tree Feature Transforms confused me a lot. After searching around, there is no website / book / paper describing about this method. Can anyone give me link references describing this method in detail?

What the article mentions is random tree embeddings.
This is a kind of unsupervised feature extraction method based on random trees.
In python you can find the Scikit-learn implementation.
From the documentation:

Transform your features into a higher dimensional, sparse space. Then train a linear model on these features.

First fit an ensemble of trees (totally random trees, a random forest, or gradient boosted trees) on the training set. Then each leaf of each tree in the ensemble is assigned a fixed arbitrary feature index in a new feature space.

These leaf indices are then encoded in a one-hot fashion.
Each sample goes through the decisions of each tree of the ensemble and ends up in one leaf per tree. The sample is encoded by setting feature values for these leaves to 1 and the other feature values to 0.
The resulting transformer has then learned a supervised, sparse, high-dimensional categorical embedding of the data.

You may also find these references very helpful:

https://gdmarmerola.github.io/forest-embeddings/

https://blog.davidvassallo.me/2019/08/06/3-uses-for-random-decision-trees-forests-you-maybe-didnt-know-about/


Binary text classification with TfidfVectorizer gives ValueError: setting an array element with a sequence I am using pandas and scikti-learn to do binary text classification using text features encoded using TfidfVectorizer on a DataFrame. Here is some dummy code that illustrates what I'm doing:

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.svm import LinearSVC
from sklearn.feature_extraction.text import TfidfVectorizer
data_dict = {'tid': [0,1,2,3,4,5,6,7,8,9],
         'text':['This is the first.', 'This is the second.', 'This is the third.', 'This is the fourth.', 'This is the fourth.', 'This is the fourth.', 'This is the nintieth.', 'This is the fourth.', 'This is the fourth.', 'This is the first.'],
         'cat':[0,0,1,1,1,1,1,0,0,0]}
df = pd.DataFrame(data_dict)
tfidf = TfidfVectorizer(analyzer='word')
df['text'] = tfidf.fit_transform(df['text'])
X_train, X_test, y_train, y_test = train_test_split(df[['tid', 'text']], df[['cat']])
clf = LinearSVC()
clf.fit(X_train, y_train)

This gives the following error:

Traceback (most recent call last):

  File "<ipython-input-151-b0953fbb1d6e>", line 1, in <module>
    clf.fit(X, y)

  File "C:\Users\Me\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\svm\classes.py", line 227, in fit
    dtype=np.float64, order="C")

  File "C:\Users\Me\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\utils\validation.py", line 573, in check_X_y
    ensure_min_features, warn_on_dtype, estimator)

  File "C:\Users\Me\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\utils\validation.py", line 433, in check_array
    array = np.array(array, dtype=dtype, order=order, copy=copy)

ValueError: setting an array element with a sequence.

I have found numerous posts (e.g. here, here) mentioning that this error can indicate non-uniformity of the data. This post for the same error suggests it can also be due to a data typing issue. However, I can't see how my very simple example could be due to either of these. There is surely something simple I am missing. Help!

TfidfVectorizer returns a (sparse) 2-D array or a matrix. You can't set the column df['text'] to a matrix without messing up the dimensions. Instead, you need to concatenate the result from TfidfVectorizer with the remaining features in the dataframe.

df_text = pd.DataFrame(tfidf.fit_transform(df['text']).toarray())
X_train, X_test, y_train, y_test = train_test_split(pd.concat([df[['tid']],df_text],axis=1), df[['cat']])


What is the state of the art in the field of NLP? I am new to Natural Language Processing, I think NLP is a challenging field, the syntax and semantic ambiguities could cause a lot of problems. For example I think for these problems machine translation is a hard task.

Therefore there are probably many approaches and methods that have been applied to this field. But what are the latest and most promising approaches and methods in the field of NLP?

Are these techniques highly dependent on the target language?

You are probably aware that deep learning is all the rage these days, and it has touched NLP too. There is a tutorial on it from a recent conference: Deep Learning for Natural Language Processing (without Magic) by Richard Socher and Christopher Manning, who are from Stanford.


Time Series data visualization When I visualize data using matplotlib it displays very well, but when using Plotly, the data display very bad.

Try changing the plot type on Plotly to line:
import plotly.express as px

fig = px.line(df, x="Date", y="Close", title='COMERCIAL INTERNATIONAL BANK')
fig.show()


Extract text from a image - OCR This is the first time I am working with OCR. I have an image and want to extract data from the image. My image looks like this:

I want to extract the parameters and the values against them. Can someone guide me on how to do this? I know we can extract text from an image using tesseract and PIL libraries if the image contains some simple text. What to do in this case where there are several parameters?

I have used tesseract for similar tasks. I can give you few recommendation. You can choose the best that works for you.

Extract the parameter values by finding the exact locations they appear.

If the parameters always appear in the same place you can find those coordinates and extract parameters from there. You can get the location information by getting the TSV output produced by Tesseract while performing OCR.

Looking at the line / block / paragraph numbers produced by tesseract for each text

Since the values and parameter pairs are separated by lines or to separate sections, I think tesseract will do a reasonable job in giving your parameters either different line numbers or block/paragraph numbers. You can get this information if you get the TSV output from Tesseract using the create_tsv=1 configuration I believe.

Using regex to match parameters

It seems that your parameters are pretty consistent (i.e. either categorical ON/OFF or a number). In that case what you can do is, while reading the output from tesseract, match the description in front of the value. For example, as soon as you encounter AHU ON/OFF STATUS text, look for the word either on or off.

And you can use these in combination to make the results even more accurate. Hope these suggestions help.


Handling many missing values I have a dataset I want to perform multivariate linear regression to it. The dimensions of the dataset are 832085 rows and 11 columns. The data are quite messy and given the size and my lack of experience I am confused on how to clean them.

First of all, 6 out of 11 columns have more than 277000 NA values. In that case I know that I cannot remove them because they are a big portion of the dataset so I have to impute them. Usually I would do mean substitution but I've read that this approach might create bias in the data and I wouldn't like to have that. I have tried the packages Amelia and mice in R, mice couldn't run properly and after a while it was giving me an error, Amelia was very fast and completed 5 imputations but it introduced many negative values in the dataset.

Anyone with similar experience on that?

There are a number of different ways to impute data when you have missing values. Far to many people just delete those records, which often contain useful training data.  Further, some people advocate for simple methods like using the mean of the feature when it is missing.  I don't like this method as it seems too simplistic.

Here is a good presentation detailing a couple of methods and particularly drawing attention to imputation through maximum likelihood.

However, my favorite method is collaborative filtering, which is detailed in this paper. The nice thing about collaborative filtering is that it is easy to write from scratch and is also very popular, so is usually contained in most machine learning libraries.

Also think about taking a look at the imputation techniques that might be available in the specific machine learning library that you may be using.

Finally, a word of caution... imputing data adds some linear dependence, so you should expect that you might have to increase your regularization slightly in order to avoid high variance scenarios. I would experiment with (cross validate) removing features, removing records, then imputing data using collaborative filtering and see what produces the best results.

Hope this helps!


How to sample with guaranteed minimum spacing? I have a timeseries stored as a pandas.Series, and have computed the autocorrelation to be negiligable for 12 hours lag. Therefore, I want to test the approach of sampling data points randomly, but with at least 12 hours between them, and then treating them as IID data.
How do I do this algorithmically, in an efficient way?

ps if I want at most one data point per 12 hour window, I could do like below:
import pandas as pd
import sklearn
import numpy as np

ndata = 2*24*60*60
s = pd.Series(data=np.random.random(ndata),index=pd.date_range(start=pd.Timestamp("2000-01-01"), periods=ndata, freq='S'),name='data')
window = pd.Series((s.index - s.index[0]).total_seconds()//(12*60*60),index=s.index,name='window_id')
df = pd.concat([s,window],axis=1)

random_datapoints = sklearn.utils.resample(df,stratify=df.window_id,n_samples=df.window_id.unique().size,replace=False)

However, most probably (this happens almost surely as $n\to\infty$, Ill get two samples that are less than 12 hours apart, even though they will be 12 hours apart on average....

Let's say you have very frequent data across a period of time T and you want to sample N points.
Instead of sampling points directly you could sample the time that separate them. You then just need to add 12 hours to all gaps to enforce your constraints.
To do so you could sample N points uniformly in [0, T - (N-1) x 12h] and then compute the difference between consecutive points.
Sampling N + 1 points in [0, T - N x 12h] allows you to also sample the first timestep.
Afterwards you can match thoses times with the closest samples available in your data.
import numpy as np
T = 500
N = 200
min_diff = 0.5

t_diffs = min_diff + np.diff(np.sort(np.random.uniform(0, T - N * min_diff, N+1)))

times = np.cumsum(t_diffs)

import matplotlib.pyplot as plt
plt.figure()
plt.scatter(np.arange(len(times)), times)
plt.ylabel("Time")
plt.xlabel("Point index")
plt.savefig("fig.png")
plt.show()


Convolution over volume in CNNs I have a simple question about convolution layers in CNNs. Consider that we have 32 features map with size $100\times100$. So, can we set 16 convolution layer with size $9\times9\times16$ after features map? There isn't any limit on depth of the convolution layer? Then if we can, so every $9\times9\times16$ layer produces just one layer? (How does depth=16 work on depth=32)?

When you have 32 feature maps with height and width equal to 100 and the depth of each equal to one it means that you have 32 planes, a common jargon among vision people, with 100 by 100 entries. You can set the height and width of the next layer and they can be arbitrary. You can also set the number of feature maps but the depth of each feature map would be equal to the number of feature maps of the previous layer. So it should be 16 * 9 * 9 * 32 if you set height and width equal to 9 and the number of feature maps to 16. As you can see in 16 * 9 * 9 * 32, 32 is located at the end, this is called channels last. You can not set the depth, because it should be equal to the number of channels, features maps, of the previous layer.

16 * 9 * 9 * 32 means that you have 16 feature maps of dimension 9 * 9 * 32, so the output of each feature map would be the member-wise product of all the outputs of the feature maps of the previous layer and each of 9 * 9 * 32 kernels. Consequently the result would be 16 planes.

I highly recommend you taking a look at here.


Branch of data science that covers event based time series? Let's say I have discrete events in time, e.g. patients getting sick, and I want to predict whether theses events are indicators of some other underlying event, e.g. a disease outbreak. Usually, one would transform the event based time series into a regular time series by for example aggregating the counts of patients over each week. However, I feel that by this aggregation a lot of information is lost and it is also hard use to multiple features of events, e.g. patients' age, patients' sex, etc. My question is very general: is there a branch of statistics / data science that treats time series as composed of events at arbitrary times rather that aggregated values at evenly distributed intervals?

I tried googling it, but it seemed hard to phrase the question in a way that a search engine understands.

"rather than aggregated values at evenly distributed intervals?" I think that part defines your question very well; i.e that is usually what we generally do in usual time-series problems, merging the events in some discrete time intervals.

What you can do is that, you can give the time distance between the events as features, without giving evenly distributed intervals. For example, the exact time distances between events of every disease outbreak or every patient getting sick and etc. Moreover, you can do some feature engineering; squared time differences or percentage changes of time difference between the differences between the sequential doubles (x, x-1) and (x-1, x-2) where x-n,.....,x-2, x-1, x are the sequential events, not the time.

The relevant concept here seems to be "unevenly spaced time-series", the wiki page Wikipedia: Unevenly spaced time series and this researcher's website http://eckner.com/research.html  seems to give enough material to get started with.


variance threshold, returning the names of the selected features I am trying the variance threshold method for the first time and I am following the example in sklearn to work on it.
>>> X = [[0, 2, 0, 3], [0, 1, 4, 3], [0, 1, 1, 3]]
>>> selector = VarianceThreshold()
>>> selector.fit_transform(X)
array([[2, 0],
       [1, 4],
       [1, 1]])

However, at the end, it only returns an array of the values of the selected features. How can I get the name of the features that met the variance threshold limit here? I just need the names of the selected features.

Use selector.get_support (Documentation).
This will give you a mask of the features that were selected and features that were discarded.
>>> selector.get_support()
array([False,  True,  True, False])

And here is how you get your selected features indices
>>> [ i for i, f in enumerate(selector.get_support()) if f ]
[1, 2]


Observing a set of features while training is same as seeing it one time. How? If we use the same training example to train multiple times, it won't bring any change to desired theta values. Can someone provide intuition behind it or some theoretical explanation to back this?

To me, it looks like using the same training example in training could help. Given a training example, weights try to converge but they don't converge to the optimum level because of small learning rate, etc. So, when the model sees the training example again, it may help the weights to converge towards the minima. Thoughts/Intuitions?

This really depends on the type of algorithm you are using. With iterative approaches like stochastic gradient descent this is exactly what does happen. With tree based methods it will not matter except that the interpretation of your hyperparameters surrounding regularization changes by artifically creating a bigger dataset.


What should be the criteria to select features using correlation factors between features? For the titanic dataset, I have done some feature engineering (one-hot encoded the features) and now I have developed a heatmap to view the correlation between different features.

I'm not able to understand what to do with them. Lets say two features are highly correlated, eg, in the image, Name_title_mr and sex_0 (ie. male) are having correlation factor of 0.84. So, does that mean I should drop Name_title_mr, since sex_0 is a very important feature. (This we know with experience, that sex in titanic is very important but is there any way we could view this as well by just observing the heatmap?).
One more doubt I have is: How would I know that I can just add two features, like sibsp and parch ? I have seen many kernels where they just create one feature with no_of_family_members by just adding sibsp and parch.

Will that adding of two features helpful? and should I drop the individual features sibsp and parch in that case?

sibsp_1 and sibsp_0 (sibsp_1 is when sibsp=2 and sibsp_0 is when sibsp=1) seems to be highly negatively correlated. Should I consider some action on this as well?

There are several questions here, but let me try to answer the overall issue.  How do I know which features to drop or add to make the best model?

There is no rule that can be used to say “always drop all but one feature if X correlation exists”.  I usually start with > +/- 0.8 but you should test on a per dataset basis.
feature engineering, creating new features or eliminating features, is the art side of analytics.  Take into account what you know about the dataset, what you can learn from data analysis and experimenting.   Often all of the experiments don’t get captured in data kernels (especially those that don’t have positive impacts) so it just looks like magic to people reviewing the final/posted solution.


Neural Network training time? I built a fairly standard backpropagation algorithm and just the process of forward propagating through a 5 layer x 5 nodes network using a data set of 10,000 observations of 39 variables takes almost 5 minutes for one iteration.

Do neural networks typically take many hours to train using data sets this size? My initial data set was 10x as long, but I couldn't wait an hour just for one forward pass to be completed.

This is quite standard for the training time. It depends on how much optimization you did on your code. The speed of your processing unit, it's often better to use a GPU as opposed to a CPU. GPUs do mathematical operations much faster. Also, you should use parallel computing when you can, in the case of NN you definitely can.

Training a machine learning algorithm only needs to be done once. Let it run all night and then you will be ready to do some pretty good predictions.


Why is my training accuracy decreasing higher degrees of polynomial features? I am new to Machine Learning and started solving the Titanic Survivor problem on Kaggle.

While solving the problem using Logistic Regression I used various models having polynomial features with degree $2,3,4,5,6$ . Theoretically the accuracy on training set should increase with degree however it started decreasing post degree $2$ . The graph is as per below

I disagree with the assertion of, "Theoretically the accuracy on training set should increase with degree". The goal of polynomial regression is not to randomly try new polynomials. The goal is to use a polynomial that better fits your data because the correlation is not linear.

Let's think about the end result of linear regression - it usually something like y = mx + b

If you show that to a data scientist, they're going to tell you it's linear regression.  You show that to a math student and they will tell you its the formula for a straight line. Either way, it's just a formula for a graph. But, note that this is for a straight line and not all data is linear. So, knowing that you're just coming up with a formula, you should think about polynomial regression in the same way - what graph am I trying to draw?

If you use a scatter plot and you are seeing a correlation but that relationship is exponential, then you should use the corresponding polynomial; same goes for all of the other variations. There is no logical explanation to use a polynomial that will not draw a graph that will closely align with your data correlation.


Where to get the Datascience Use cases for practice I just started  learning data science. I have gone through some of the courses in coursera & udemy, now i want to practice what i have learned. What i want to know is from where can i get the Use cases (linear regression & multiple linear regression) so that i could practice

Maybe you can start with the Kaggle datasets: https://www.kaggle.com/datasets
There are more then 22.000 datasets and they are very well documented, enough for hours and hours of practice.


Pre-processing irregular, high frequency time-series data in python ...posted originally in StackOverflow (might be better suited here)

Small Picture:

I am working on pre-processing irregular, high frequency time-series data. In one second, I can have multiple data points, as seen below in the timestamp field:

"timestamp": "2018-06-03T12:27:54.253"
"timestamp": "2018-06-03T12:27:54.409"
"timestamp": "2018-06-03T12:27:54.548"

I'm in the process of developing a sampling scheme for this time-series data so that I can reduce the number of data points, and standardize time steps without losing information or introducing any bias.

So far I've been using Pandas pd.resample() on just a small subset of our data (5 days ~ 2 million records) by using mean as the aggregation function and linear interpolation. I am on downsampling the data by seconds, minutes, and hours for experimental purposes which takes care of the irregular time steps of the original data.

Larger Picture:

I am working with many millions of records (data from April - today) that have been queried from elasticsearch and, ideally, want to pre-process/sample within this large pool of time series data to only obtain statistically significant data points. The purpose of this pre-processing step is for future data exploration/modeling.

Question:

How can I modify my current pre-processing scheme to ensure optimal processing for Millions of records and still maintain their statistical properties (no bias introduced)?

I'm aware that there is possibly a better way to approach the processing this kind of data. Any input is greatly appreciated...thanks!

Keep in mind, downsampling is always lossy. I will give you two hints.

Nyquist-Shannon Theorem. What it says in short is that the max frequency that you may observe in a sampled signal is half of the sampling frequency. This means that if you downsample the data every 1 hour, you can observe dynamic phenomena with period of 2 hours (or more). Another way of looking at the theorem is "you have to sample the signal at least twice as fast as the maximum frequency that you expect to see in it".
Engineering rule of thumb: Sample the signal at least 10 times faster than the expected highest frequency, not just 2 times.

Histogram Representation. Since you have a huge database, it might make sense to convert your variables into histograms! In this way, you will greatly reduce their size. You discretize every real-valued variable into bins and then instead of storing the exact value of the variable you just increase the count of the appropriate bin by 1. There are a lot of Machine Learning algorithms that deal with histogram representations, such as this:
https://pdfs.semanticscholar.org/b8c8/347f9c33935b97703ecd35a67af5c5508487.pdf

Hope it helps :) !


Is there an appropriate sequence of parameters to be considered when building a pipeline in Scikit learn? DISCLAIMER: I understand this could be considered a subjective type of question, but I'm hoping it is specific enough to not be flagged. Please give it a chance.

I am doing a GridSearch in scikit-learn on a corpus of 50 books (all kinda average commercially published texts...each ~200 pages of text).

As I began playing with GridSearch, I got to thinking. Surely there must be some sort of 'best practices' for building a pipeline?

Sure, I can include and remove parameters, but I'm really just sorta stabbing around in the dark for the best score.

Possibly, it could make sense to create a smaller, exploratory pipeline using the larger, more effective parameters first. After running this basic pipeline, more parameters that have a smaller effect (in general) could be run next.

For example, say I'm GridSearching a TfidfTransformer.  I am assuming that flipping the use_idf switch on or off is going to make a real impact on the scoring, whereas perhaps specifying alpha=0.000010 versus alpha=0.000015 won't.

In general, when using a pipeline, is there an alternative to "throw in the kitchen sink" that results in a more cognizant, informative (and possibly faster) experience?

What you are asking for is a gradient based hyperparameter optimization. To enable such a grid search one would have to define a metric on the space of all hyper parameter combinations for the current pipeline.

I don't think this makes sense in your example because of the binary nature of the use_idf flag. Lets say your parameter combinations are (use_idf, alpha). Then you would have to do decide what the distance between (True, 0.00015) and (True, 0.00010) is. In your case it should be smaller than the distance between (True, 0.00015)and (False, 0.00015). So

d((True, 0.00015), (True, 0.00010)) < d((True, 0.00015), (False, 0.00015))

But what about the distance between (True, 0.00015) and (True, 0.05)? Will it be still smaller than d((True, 0.00015), (False, 0.00015))? I think you get the point.

By choosing the metric you will subjectively influence the search for the best hyper parameters. Probably a random search would be more suitable as suggested in the following paper

@article{bergstra2012random,
  title={Random search for hyper-parameter optimization},
  author={Bergstra, James and Bengio, Yoshua},
  journal={The Journal of Machine Learning Research},
  volume={13},
  number={1},
  pages={281--305},
  year={2012},
  publisher={JMLR. org}
}


Randomizing selection in R I am trying to create a comparison group. So far this group contains 45 data points and I need to populate the remaining 55 (for a total of 100 data points).

These remaining 55 need to be a randomized selection supplied from a larger data set. Any recommendations for R code that would create a randomization loop?

This will give you a sample with 55 record from the larger data-set that you have.

 sample <- df[sample(1:nrow(df), 55, replace=FALSE),]

If you want to make a reproducible example of the sample, you need to set seed like this:

 set.seed(57)
 sample <- df[sample(1:nrow(df), 55, replace=FALSE),]

So later you can use the same seed for getting the same result.

p.s. replace=FALSE means that each record after that has been picked from the df to be included in the sample, will be excluded from the df and it will not be chosen multiple times.


XGBoost validation for number of trees I have a simple Question:

I am using XGBoost to classify some data:
1.) With 100 estimators I have following scores(roc_score):
    train_data : 98.5
    validation_data : 97.2

2.) With 500 estimators I have following scores(roc_score):
    train_data : 99.4
    validation_data : 97.7

Acco. to above can we say model with 500 estimators works better. Or should I change the validation data few times and see if similar kinds of increment happens by shifting from 100 to 500 estimators.

At first glance, your conclusion appears correct, but there are some important caveats to keep in mind.

First, what are the sizes of your training and validation sets?  If your validation set is too small, then the observed difference may not be statistically significant.

Second, you should verify that your validation set is a representative sample.  (i.e. it should come from the same distribution as the training set).  If it's not representative, then it may give poor estimates of performance.

Third, when tuning hyperparameters, it's a good idea to split your dataset into three shards - training, validation, and testing.  You can use the training and validation sets to find optimal hyperparameters (as you have done), then use the testing set to generate a performance estimate for the tuned model.  If you trust the validation accuracy obtained during hyperparameter tuning, you are liable to a subtle form of overfitting where hyperparameters are specialized for the validation set.

Finally, if you have the computational resources, then it's always a good idea to evaluate accuracy with cross-validation rather than a train-test split.  This will give you a more robust estimate of accuracy.

If you've checked all these boxes, then you have good reason to believe that 500 estimators is better than 100 estimators!

  [S]hould I change the validation data few times and see if similar kinds of increment happens by shifting from 100 to 500 estimators?

Yes, it's always a good idea to try many different configurations of hyperparameters.  You can use scikit-learn's GridSearchCV or RandomizedSearchCV to easily run a search over the hyperparameter space.


How best to use the resale transaction year in predicting housing prices? I'm looking into the classic problem of predicting apartment prices (resale market) based on the their type, size, location, etc. Pretty straightforward and Linear Regression or Regression Trees give some first decent result -- I'm still more in the exploratory phase.
However, I'm not sure how to best incorporate the year of the resale transaction, since there are clear long-term trends over the years. Right now, I just keep it as a feature, which seems to be a valid approach. I just wonder if there might be alternative approaches. For example, I also have to overall price movement on a quarterly basis. So I assume I could adjust the each resale price based on those trends and ignore the year as feature. Would this make sense?
What are other approaches? (Again, I'm not even sure if this an issue at all.)

I think you have identified the two main options:

Model the price trend, i.e. make your model fit to capture the price trend over time
Clean your [price] data so that prices are expressed in "real terms" (not including the price trend).

Option 2 would imply that you "deflate" a standardised price (e.g. price per square meter). Thus the price per sqm in 2018, 2019, and 2020 would be "adjusted" to the price level of (for instance) 2017 so that all prices are "prices of 2017". You need a standardised price (e.g. per sqm) because you need to control for possible unobserved effects in the composition of your data, e.g. when the average house sold in 2020 is "larger" than the average house in 2017. In essence you need make sure that the "deflated" prices are comparable. This can be a problem, for instance when there are changes in the market over time. You could imagine that the willingness to pay for "large" houses change over time, so that one sqm of a "large" house becomes more expensive over time. It can be hard to capture such effects by simply "deflating" (average) prices per sqm.
Option 1 can partly capture the effect(s) described above. Consider the case of a linear regression. Say you have two years (2019,2020) and you want to "control" inflation over time. Your (simplyfied) base model, with price $p$ and $sqm$ as independent variable would look like:
$$ p = \beta_0 + \beta_1 sqm + u. $$
Now you can add a "year dummy" (=1 if year==2020):
$$ p = \beta_0 + \beta_1 sqm + \beta_2 t_{2020} + u. $$
Coefficient $\beta_2$ captures the average effect on $p$ in 2020 compared to 2019. This is sometimes called a "fixed effect" since the variable simply is a "shift" in prices in 2020 compared to 2019 for all levels of $sqm$.
If you think $sqm$ and "time" are somehow related, you can also add interaction terms, e.g.:
$$ p = \beta_0 + \beta_1 sqm + \beta_2 t_{2020} + \beta_3 sqm * t_{2020} + u. $$
In this model you allow for a different intercept (in 2019 and 2020) and for a different slope of $sqm$ in both years. Instead of an interaction of "time" and $sqm$ you could also add and interaction with "size dummies" (e.g. "small" vs. "large" houses).
In essence, option 1 gives you more flexibility since linear regression allows you to "deflate" prices inside the model. Note that linear regression is a parametric approach, so you need to find a proper parameterisation of the model (just as you would need to find the right strategy to deflate prices when you do this outside the model).
When you use regression trees, you don't need to worry about the functional representation of the model. The advantage of linear regression is that the "time dummy" is forced to be fitted on all data. In regression trees the effect of dummies is less prevalent. So in this case "deflating" data outside the model could be worth a try.
However, when you are up to predictions with low variance, you ultimately need to check what approach works best based on test results.
Edit (20-12-30): Dummies
Suppose you have a vector of IDs:
  id
1  1
2  1
3  2
4  2
5  3
6  3

Dummy encoding will look like:
  id1 id2 id3
1   1   0   0
2   1   0   0
3   0   1   0
4   0   1   0
5   0   0   1
6   0   0   1

In a linear regression, dummies generally work as "contrasts", e.g. the effect of id2 vs. id1 and id3 vs. id1, so that you include n-1 of the dummies.


How to discretize certain features with a feature set? I am working with typing data with timing features(unit: ms) and some of the features are based on the keyboard keyCodes(positive integers, range:[8, 222]). Currently, I use StandardScaler() by scikit-learn to scale all the features, so that my learning models do not overweigh the keyCode based features. I would like dicretize the keyCode based features and run the StandardScaler() for the timing features only. How can I go about this?

Not sure about your question, but maybe something like the following could help:

Create dummy variables for your KeyCodes variable
Normalize only those variables using the StandardScaler.

import pandas as pd
import random
from sklearn.preprocessing import StandardScaler

# Let's assume the following dataframe
data = {
    'KeyCodes': [random.randrange(2, 223, 1) for _ in range(10000)],
    'age': [random.randrange(1, 100) for _ in range(10000)],
    'id': [i for i in range(10000)]
}

# First you need to create the dummy variables based in KeyCodes
df = pd.DataFrame.from_dict(data)
df.head()
dummies = pd.get_dummies(df['KeyCodes']).rename(columns=lambda x: f'KeyCode_{x}')
df = pd.concat([df, dummies], axis=1)
df.drop(['KeyCodes'], inplace=True, axis=1)

# Then you can apply the normalization to the subset of features you wish to normalize
normalized_df = df.copy()
col_names = [f'KeyCode_{i}' for i in range(8, 223)]
features_normalized = df[col_names]
scaler = StandardScaler().fit(features_normalized.values)
features_normalized = scaler.transform(features_normalized.values)
normalized_df[col_names] = features_normalized

# Explore the output
normalized_df['age'][:10]  # you can see it was not normalized
set(normalized_df['KeyCodes_8'])  # normalized version of the feature
set(df['KeyCodes_8'])  # not normalized version of the feature


Calibrating Correlation I am facing a weird problem in my on going project and thought if someone here could help me out with this. Actually I have large data set. I have to perform a regression task on top of that. While doing the initial analysis and feature selection task, I did correlation of all the features with the target variable. When I got the results I saw one of the feature having negative correlation with the target variable but my business unit suggests it must have a positive correlation to proceed further. Now they are asking me to filter out the rows causing negative correlation. I don't know how to proceed in this context. Can anybody help me out or give me heads up in this? Thank you very much in advance

First what comes to my mind: "the only statistics you can trust are the ones you have falsified yourself" - Churchill

However, to answer your question.

The correlation coefficient is defined as:

You can find data points that are negatively correlated by looking at the covariance, that is defined as:

First, you calculate the mean of your dependent and independent variable. Then you subtract the mean from the corresponding data points. After that, you multiply those two series. Negative values are rows that contribute to a negative correlation coefficient.

However, as a statistician, I am not supporting this approach (data deletion on purpose is not scientific working).


Feature engineering for categorical variables I have some categorical variables in my dataset for a regression problem.

1) One of the variable can take 3 values (Girls, Boys, Girls&Boys). Converting it into one-hot encoding or binary encoding will treat all three values as a different class. How can I use it efficiently retaining the information that 'Girls&Boys' include both? Is breaking into two separate columns for girls and boys is the only approach in such cases?

2) Age range: (18-35, 35-50 etc.) I have broken it down to 2 columns of age_min and age_max. Is there any better way to use such kind of features with values in range?

3) Range of percentage: It can take only 5 values (0, 1, 1-5, 5-10, 10). How should I use this variable for training my model? Here I cannot break it down into 2 columns (like with age) because of "1" and "10" fixed values. How to treat variables with fixed as well as range values?

4) Similarity: It can also take 5 values (0, 1, 1-5, 5-10, 10) but it also has an "auto" option. That means if it is "auto" it can take any random value which we do not know about. How should I incorporate that?
Should I create a separate column for "whether it is auto or not"? Then, in the original similarity column, what value should I put for observations with "auto". I cannot put "0" as it is already a value and I am sure that "auto" won't be 0. Does putting it "None" will create a difference?
How to treat unknown values?

Before I begin let me note that there isn't a right and a wrong way to do feature engineering. If you want to be thorough, you need to try things and see if they work or not.

Now let me give you my opinions (i.e. how I might approach things), but they aren't guaranteed to work:

  1) One of the variable can take 3 values (Girls, Boys, Girls&Boys). Converting it into one-hot encoding or binary encoding will treat all three values as a different class. How can I use it efficiently retaining the information that 'Girls&Boys' include both? Is breaking into two separate columns for girls and boys is the only approach in such cases?

I think one-hot encoding works best here. Depending on what information does Girls&Boys store you can select how to encode it. If Girls&Boys == 1 if both Girls and Boys are 1, then I think you can safely omit it.

  2) Age range: (18-35, 35-50 etc.) I have broken it down to 2 columns of age_min and age_max. Is there any better way to use such kind of features with values in range?

What you've done isn't bad. Another option would be to treat this like an ordered variable and simply map each range to a number (while preserving the order). E.g. 18-35 --> 1, 35-50 --> 2, etc.

  3) Range of percentage: It can take only 5 values (0, 1, 1-5, 5-10, 10). How should I use this variable for training my model? Here I cannot break it down into 2 columns (like with age) because of "1" and "10" fixed values. How to treat variables with fixed as well as range values?

Same thing as before. I'd treat this like an ordered variable. You could even leave 0 and 1 as they are. The rest would go like this: 1-5 --> 2, 5-10 --> 3, 10 --> 4. If some of these 5 categories has very few values you could merge it with a neighboring one!

  4) Similarity: It can also take 5 values (0, 1, 1-5, 5-10, 10) but it also has an "auto" option. That means if it is "auto" it can take any random value which we do not know about. How should I incorporate that? Should I create a separate column for "whether it is auto or not"? Then, in the original similarity column, what value should I put for observations with "auto". I cannot put "0" as it is already a value and I am sure that "auto" won't be 0. Does putting it "None" will create a difference? How to treat unknown values?

This is trickier here. One thing I'd do is what you mention: to have an auto column which can be either 1 or 0 if the user chose "auto" or not. However, it's tricky to encode the range now, because how would you encode those that selected "auto"? One way is to make a separate category (e.g. auto --> 5), but this wouldn't correct for an ordinal variable. Another option would be to one-hot encode this. Your choice I think depends on two reasons: a) Given the "auto" option how important is it preserving the range information as is? Could you group categories together? could you disregard their order? b) How many of the samples are "auto"? If they are very few it might not even make a difference. If they are too many then encoding the range might have no meaning.


How and When features are attached to target label I am using Mallet CRF library and having training set sequences like below.

KFC     Bangalore  INDIA        <--- Sequence
PLACE   CITY       COUNTRY      <--- Target Label

During training/validation time, I don't have any features.
While testing time, I give features like below.

KFC                 Bangalore            INDIA                  <--- Sequence
PLACE_LIKE(1.0)     CITY_LIKE(1.0)       COUNTRY_LIKE(1.0)      <--- Features

Since there is no features in the training time, will these runtime features are useful?
I believe, the association of features and target label is happening during the training time itself so any new feature directly comes during runtime will not get associated to target label and useless, is that correct?
Can i attach Bangalore with CITY_LIKE(1.0) while training and pass lower weight feature CITY_LIKE(0.5) during testing time? Will that has impact?

I assume you're training a CRF model, or some other variant of sequence labeling, right?

I'm not especially familiar with Mallet but it's very unlikely that any model would let you add features at testing stage: the trained model and the test data must have exactly the same features, no more no less.

Apparently the features you want to add consist of a similarity score with a few particular categories, I assume this score is calculated independently during pre-processing? If yes this would usually be represented like this in the data:

<token>   <PLACE_LIKE> <CITY_LIKE> <COUNTRY_LIKE> <label>
KFC       1.0          0.0         0.0            PLACE
Bangalore 0.0          1.0         0.0            CITY
INDIA     0.0          0.0         1.0            COUNTRY

Note that this format with token + N features must be followed in both the training and test data. Therefore you must apply the same pre-processing steps for both.


Algorithms for Anomaly Detection of Event Sequence Data [Python/R] I am building an anomaly detection system of event sequence data (transactions). For each timestep, a transaction can be in any of 76 different stages.
My dataset is therefore a 3D array of size(m,t,N).
m is number of samples (~40 000), t = the length of the longest transaction (I am zero padding all other sequences of shorter length), N is the number of possible events at each timestep. $x_{m,t,N} = \{0,1\}$ where $x_{v,u,w} = 1$ if transaction v is in event w at timestep u otherwise it's 0.
So far I have created an LSTM Variational Auto Encoder in Python with Keras and doing outlier detection by applying Local Outlifer Factor on the latent representations of the transactions. It's based on this paper https://arxiv.org/abs/1906.10896 and following various blogs such as https://blog.keras.io/building-autoencoders-in-keras.html.
I would like to compare the results with other techniques. I was wondering if anyone has any other suggestions on algorithms and techniques for anomaly detection in unlabeled event sequence data?

A simple comparison is to use the reconstruction error as the anomaly score.
This can be done instead of, or in addition to, the Local Outlier Factor that you have now.
You can also try other ways of computing the anomaly score from the latent feature vector. Some popular alternatives are:

Mahalanobis distance on the
Fitting a Gaussian Mixture Model, and use the probability density function

A completely different approach would be to use a self-supervised training setup instead of the autoencoder. Such as Skip-Gram or predicting future time-frame.


Accuracy function in R language I would like to find the accuracy of predicted data. Which package in R can I use that has the accuracy function?

You can use confusion matrix present in caret package.


Find closest numbers approximately I have a table like this; all columns are numeric:

TableTest

Col1  Col2  Col3  Col4
  5    6     7     8
 12    6     5     6
  2    3.5   6     1

And I want to find the closest row with these values: 4, 5.75, 7.2, 6. A human can find it (row 1 is answer), but how can I write a query for a computer to find it?

I'm using SQL Server but I think it's relevant to machine learning.

This problem can be implemented as a query that minimizes "loss" from the inputs. If "mean square error" loss is used, loss calculation will look like :

select col1, col2, (col1 - inp1)^2 + (col2 - inp2)^2 as mse_loss,
from table

Now, you can find the row with least value of mse_loss


What is the difference between dcast and recast in R? I am working with a dataframe in R that is formatted like this sample:

Countries <- c('USA','USA','Australia','Australia')
Type <- c('a','b','a','b')
X2014 <- c(10, -20, 30, -40)
X2015 <- c(20, -40, 50, -10)
X2016 <- c(15, -10, 10, -100)
X2017 <- c(5, -5, 5, -10)

df_sample <- data.frame(Countries, Type, X2014, X2015, X2016, X2017)

The dataframe looks like this:

  Countries Type X2014 X2015 X2016 X2017
1       USA    a    10    20    15     5
2       USA    b   -20   -40   -10    -5
3 Australia    a    30    50    10     5
4 Australia    b   -40   -10  -100   -10

I want to be able to create columns of year values for each type by each country, yielding something that looks like this:

        Countries   Year     a     b
1       USA         X2014    10   -20
2       USA         X2015    20   -40
3       USA         X2016    15   -10
4       USA         X2017     5    -5
...

With recast I get this:

recast(df_sample, Countries ~ Type)

  Countries a b
1 Australia 4 4
2       USA 4 4

With dcast I get this:

dcast(df_sample, Countries ~ Type)

  Countries a   b
1 Australia 5 -10
2       USA 5  -5

The dataset I'm working with has 44 years of data, so I'd like to be able to indicate all columns of yearly data without having to enter each column id manually into a cast formula.

What is the difference between dcast and recast (i.e. what situations might they be best suited to), and is it possible to shape my data with them?

See ?reshape2::recast: The function

  conveniently wraps melting and (d)casting a data frame into a single
  step.

library(reshape2)
recast(df_sample, Countries+variable~Type, id.var=1:2)
#   Countries variable  a    b
# 1 Australia    X2014 30  -40
# 2 Australia    X2015 50  -10
# 3 Australia    X2016 10 -100
# 4 Australia    X2017  5  -10
# 5       USA    X2014 10  -20
# 6       USA    X2015 20  -40
# 7       USA    X2016 15  -10
# 8       USA    X2017  5   -5

So, it's just a shortcut for these two steps:

(tmp <- melt(df_sample, id.vars=1:2))
#    Countries Type variable value
# 1        USA    a    X2014    10
# 2        USA    b    X2014   -20
# 3  Australia    a    X2014    30
# 4  Australia    b    X2014   -40
# 5        USA    a    X2015    20
# ...
dcast(tmp, Countries+variable~Type)
#   Countries variable  a    b
# 1 Australia    X2014 30  -40
# 2 Australia    X2015 50  -10
# 3 Australia    X2016 10 -100
# 4 Australia    X2017  5  -10
# 5       USA    X2014 10  -20
# 6       USA    X2015 20  -40
# 7       USA    X2016 15  -10
# 8       USA    X2017  5   -5


How to retrain a K-Modes model based on daily data? I have read that retraining a model depends highly on what you are trying to achieve. I am conscious that maybe I need to retrain my model daily and after a certain time I have to train the model again from scratch. That's no problem. The thing is, if I were to retrain a K-Modes model, I would need the previous centroids saved. I think this is also not a problem. Seeing the implementation of K-Modes here, the parameter init can be Huang or Cao. In the K-Means implementation of sklearn (see here), we can pass a ndarray of shape (n_clusters, n_features). I can't find this feature in the K-Modes implementation. So my specific questions are:

Is saving the previous centroids sufficient to retrain the model?
Is there a way to pass previous centroids in the K-Modes implementation I have mentioned? If not, should I implement my own K-Modes?

There's dedicated pypi package for incremental/online learning. It's called Creme and here's there repo. It contains KMeans implementation. Though under the hood, there's no incremental stuff going on, as you need all data in a pass (read more about Lloyd's algoritm here).


Feature Engineering - correlation with binary outcome - Titanic Dataset - Ticket feature I am currently building my first machine learning model using the titanic dataset.
After the data exploration, I decided to focus my attention on the 'Ticket' feature.
One thing I have noticed about this feature is that it is not unique per each passenger; this had led me to believe that other features can be extracted from this variable:

Is_Group -> to indicate if the ticket represents a group booking
Group_Size -> the number of passengers in each group.

# save the tickets that appear more than once (i.e. group tickets)
_ = (data.Ticket.value_counts()>1).to_dict()
ls = []
for key in _:
    if _[key]==True:
        ls.append(key)
#extract the feature
data['Is_Group'] = data['Ticket'].apply(lambda x: 1 if x in(ls) else 0)

# create another dict containing the number of counts per each ticket
group_size = (data.Ticket.value_counts()).to_dict()
# extract the feature from the mapping
data['Group_Size'] = data['Ticket'].map(group_size).fillna(0)

The reason I am doing this is because I wanted to explore the nature of the relationship between the extracted features from Ticket and the target Survived (and later decide how to deal with outliers in the SibSp and Parch feature):

data.groupby(['Is_Group','Group_Size'])['Survived'].describe()

From the above table I can see that group size 2/3 have almost a 60% and 70% chance of survival. Now, this led me to think that there is a correlation (or at least some sort of relationship between group size and Survived). Therefore, I decided to create a correlation matrix to make sure that is the case.

As I expected there is a correlation between Is_Group and Group_Size (as they have been extracted from the same feature) but there is no correlation between these extracted features and Survived. Hence my, confusion.
I thought given the high mean values of Survived for Group_Size (2,3) there was a relationship but clearly I am getting something wrong here.

Can anyone help clear this doubt of mine?

The thing is that you are looking at correlation with binary outcome. There as some counter intuitive things when doing that (see for example: https://stats.stackexchange.com/questions/103801/is-it-meaningful-to-calculate-pearson-or-spearman-correlation-between-two-boolea).
Outside than that, the features do not appears so much predictive.
More specifically for Is_group: Individual tickets (Is_group = 0), yields à 30% mean, while individuals with group tickets have around 50%. This is not really predictive.
Similarly, the average probability doesn't evolve monotonically of group size (from 0.3, to 0.6 then 0.7, then droping to 0.5 followed by 0 twice then back to 0.23). It doesn't seems really appropriate to use linear correlation to measure the relation between group_size and output here.


What is the role hidden vectors play in RBM? I'm learning about the restricted Boltzmann machine, but I can't decide whether the hidden vectors of RBM are useless or not.

The following are my two understandings for the role hidden vectors play. Have I misunderstood something? If not, then how could the role those vectors play  be so different?

This is the first understanding:

I learned from this video RBM that hidden units are just structural support and we don't care about what those hidden vectors really are. We introduce hidden units into RBM just for it to gain more expressive power for probability distribution.

Let's say I have only one image, and I transform this image to a binary vector and feed this vector into a RBM with random variables (all the weights and biases are chosen randomly). Then, by turning on the machine, the first hidden vector would be constructed. But this hidden vector does not tell us anything, it would be only used to reconstruct a visible vector.

(my understanding for this reconstructed visible vector is this vector is a vector encoded in the defined RBM in the first place, we are not really construct something new, but we just happen to sample this vector from the defined RBM)

And we just run this loop of construction and reconstruction for infinitely many times. Finally, what we will get is just the probability distribution encoded in this RBM with random variables.
My second understanding goes like this:

RBM can be used to perform dimensionality reduction, and those hidden vectors are some abstract representations of the raw inputs:

Given a RBM, each hidden units of the RBM would be a classifier, and what it does is to check the input vector lies in which side of the hyperplane defined by this hidden unit(by the weights and bias). So, if we input an image into this RBM, what the RBM will do is to project this input vector onto those hyperplanes defined by all the hidden units. Thus for an input vector, the corresponding hidden vector is very important, it is some abstract representation. And we can further feed this representation into other models for doing classification.

So, these are my understandings, if you can answer the question by explaining how RBM is used for MNIST that would be extremely helpful for me.

From my point of view, both answers are correct - it depends on what you want to achieve.

One way to use an RBM is as a generative model. If you wanted to do this, you would train the network in order to obtain weights generating a distribution the network which is as close as possible to the real distribution of the input data (not exactly true as CD is known not to converge to the sample distribution, but this just as an aside). You could then, for instance, reconstruct images from partial or sample completely new images. In this context, the main purpose of the hidden units is to increase the power of the network - a Boltzmann machine without hidden units is not able to distinguish target distributions with the same second moments. However, the actual value of the hidden units during and after is less important, as your primary goal is to generate samples.

If you want to use the model to detect features, the hidden units have an additional role. To explain this, suppose you use contrastive divergence as a learning algorithm. In each iteration, you would then sample the hidden units from the visible units and the visible units from the new values of the hidden units. If the result is close to the original value of the visible units, i.e. if your reconstruction error is small, then, apparently, the hidden units and the weights have learned enough about the training data to reconstruct the value of the visible units from the value of the hidden units. As you say, this is in fact similar to a dimensional reduction or an autoencoder. You could then build a second RBM on top the hidden units, i.e. you could start to stack RBMs to build deep networks, where each new layer of hidden units learns more and more abstract features from the training data.