Converting a nominal attributes to numerical ones in data set I'm using the NSL-KDD data set which contains nominal and numerical values, and I want to convert all the nominal values to numerical ones. I tried the get_dummies method in python and the NominalToBinary method in WEKA, but the problem is that some nominal features contain 64 values so the conversion increases the dimensionality of the data a lot, and this can create problems for the classifier.

My question is if I can convert the nominal attributes by establishing a correspondence between each category of a nominal feature and a sequence of integer values, for example protocol_type {tcp=0, udp=1, icmp=2...etc}? Would this alter the credibility of the resulted data set?

By converting a nominal attribute to a single numeric attribute as you described, you are implicitly introducing an ordering over the nominal labels which is a bad representation of the data, and can lead to unwanted effects from a classifier. Does it make sense to say that UDP should be inbetween TCP and ICMP? (no!) Imagine you are training a $k$-NN model on this data. It doesn't make sense to say that ICMP should be "further away" from TCP than UDP, but if you adopted the mapping that you suggested, the representation of the data has this assumption built-in. Alternatively, what if you are training a decision tree-based model? Usually, in decision trees, binary split points are chosen for numeric attributes. There could be some randomness in your training data where splits at certain values of the numeric attribute results in overfitting to noise.

Typically when converting a nominal attribute to numeric, one numeric attribute per nominal label is created. Each attribute is set to one if the corresponding nominal label is set, and zero otherwise. For example, if a nominal attribute called protocol has labels {tcp, udp, icmp}, then this dataset:

$$
\begin{array}{ccl}
\text{inst.} & \text{protocol} & \text{other attributes} \\ \hline
1 & \text{tcp} & \dots \\
2 & \text{icmp}& \dots \\
3 & \text{icmp}& \dots  \\
\vdots & \vdots & \ddots
\end{array}
$$

could be converted as follows:

$$
\begin{array}{ccccl}
\text{inst.} & \text{tcp} & \text{udp} & \text{icmp} & \text{other attributes} \\ \hline
1 & 1 & 0 & 0 & \dots \\
2 & 0 & 0 & 1 & \dots \\
3 & 0 & 0 & 1 & \dots \\
\vdots & \vdots & \vdots & \vdots & \ddots \\
\end{array}
$$

This is what the NominalToBinary filter does in WEKA. As you mention, the downside of this is that a large number of additional attributes can be introduced if the number of distinct nominal values is high.

If the dimensionality is too high after the conversion, you may want to consider using a dimensionality reduction technique such as random projection, PCA, t-SNE, etc. Note that this will reduce the interpretability of your model. You could also use feature selection techniques to remove some of the less useful attributes. It is possible that some of the nominal labels are not useful for your model, and you will improve performance by removing them. Another thing you could try is to use your domain knowledge to reduce the number of categories. For example, TCP and UDP are both transport protocols, maybe for your application the distinction between TCP and UDP is not that important and you can put instances with protocol $\in$ {tcp, udp} into a new category, removing the old ones.


Clustering categorical data I have a dataset with categorical features. I want to segment the data using clustering techniques. What could be the possible choices for this scenariogiven the fact that data has categorical features. Is there any variation of k-means which can be used here.

k-means is not a good choice, because it is designed for continuous variables. It is a least-squares problem definition - a deviation of 2.0 is 4x as bad as a deviation of 1.0.

On binary data (such as one-hot encoded categorical data), this notion of squared deviations is not very appropriate. In particular, the cluster centroids are not binary vectors anymore!

The question you should ask first is: "what is a cluster". Don't just hope an algorithm works. Choose (or build!) and algorithm that solves your problem, not someone else's!

On categorical data, frequent itemsets are usually the much better concept of a cluster than the centroid concept of k-means.


Statistics Before Linear Algebra? I know this is an opinion-based question and will be closed but this is the only place I know that can answer it reasonably and it is a very important matter to me.

I am pursuing a machine intelligence track at my university and as I am moving along from cs fundamentals to track courses I am stuck on what math to take first: Linear Algebra or Statistics. I know both are essential pillars and are interdependent, but I am leaning towards  taking stats before linear, which is the opposite of what most ppl do here (hence the hesitation).

Thank you and sorry for making it opinionated.

Not sure how intense your professor is going to make either course, but assuming it's the hardest possible introductory course, it would be better to take linear algebra before statistics. A lot of statistical operations require fundamental linear algebra concepts. You need to be able to understand abstract notions like vector spaces and fields to know when your data is in the correct range, you need to be able to understand what operations are linear and what are non-linear so you don't apply a wrong operation on something, and you need to most importantly be able to understand linearity and non-linearity because that is the basis of a lot of statistical methods. Statistics involves transforming data in a way that makes sense but you learn the fundamentals of transformation in linear algebra. But if your stats course is just about p-values and superficial hypothesis testings, and your linear algebra course is just direct applications like row-reducing matrices, then it doesn't really make a difference which you take first.


Deep learning - rule generation I wanted to know if there is any methodology in Deep/Machine learning, where given a set of input/output values, it can derive rules for the same.

Lets say I generate training input and output by $y=x^2$

i/p  |  o/p
 0       0
 2       4
 .       .
1000   1000000

It sort of generate rule like, $y=x*x$

One way of stating what you are looking for is to find a simple mathematical model to explain your data.

One thing about neural networks is that (once they have more than 2 layers, and enough neurons total) they can in theory emulate any function, no matter how complex. This is useful for machine learning as often the function we want to predict is complex and cannot be expressed simply with a few operators. However, it is kind of the opposite of what you want - the neural network behaves like a "black box" and you don't get a simple function out, even if there is one driving the data.

You can try to fit a model (any model) to your data using very simple forms of regression, such as linear regression. So if you are reasonably sure that your system is a cubic equation $y= ax^3 + bx^2 +cx +d$ then you could create a table like this:

  bias   |   x  |  x*x  |  x*x*x  |     y
     1       0       0         0        0
     1       2       4         8        4
     1       3       9        27        9
     1       .         .        .       .
     1     100   1000000    10000   10000

and then use a linear regression optimiser (sci-kit learn's SGD optimiser linked). With the above data this should quickly tell you $b=1, a,c,d=0$. But what it won't tell you is whether your model is the best possible or somehow "correct". You can scan for more possible formulae by creating more columns - any function of any combination of inputs (if there is more than one) that could be feasible.

However, the more columns you add in this way, the more likely it is you will find an incorrect overfit solution that matches all your data using a clever combination of parameters, but which is not a good general predictor. To address this, you will need to add regularisation - a simple L1 or L2 regularisation of the parameters will do (in the link I gave to scikit-learn, the penalty argument can control this), which will penalise large parameters and help you home in on a simple formula if there is one.


What are features in computer vision? I'm learning how U-NET network works to do semantic segmentation.

I think I have understood everything but features. What are those image features?

I read that convolutional layers extract features from the images using their filters, but what are they? Are they corners? edges? colours?

I have read this article, "Finding Features", but I think I need more information about them.

CNNs like U-Net extract lower level features like edges on lower layers (i.e. the first convolutional layers) and higher level features on higher layers (i.e. convolutional layers closer to the final linear layers). This principle is losely inspired by how visual perception is implemented in the Visual Cortex among humans (and other animals).

In a CNN the feature maps could for example look like this:

As you can see the lower level feature maps detect simple structures like edges while higher level feature maps recognize more complex structures like eyes or faces.

Colors, however, are processed differently by CNNs than spatial features since color images are usually fed to CNNs using three input channels (one for each color in RGB format). So colors are not detected in the same way as spatial features but instead the first convolutional layer receives a 3-dimensional input image with one dimension for each color-component (one for red, one for green, one for blue).

The article What exactly does CNN see? and the paper Visualizing and Understanding Convolutional Networks (which are also the sources of above images) provide a more detailed explanation.


Non-convex feasible region in a linear programming problem Problem:  I am working on a linear programming problem, i.e. a linear objective function to minimize:

$\mathbf{c}\cdot\mathbf{x}$,
where $\mathbf{c},\mathbf{x}\in\mathbb{R}^{N}$

Subject to constraints:

$\mathbf{x}\geq 0$ and $\mathbf{A}\mathbf{x} \leq \mathbf{b}$
where $\mathbf{A}\in\mathbb{R}^{MxN}$ and $\mathbf{b}\in\mathbb{R}^{M}$

I am using CyLP and the COIN-OR simplex solver to perform the optimization.

Question: A feature I would like to be able to build into
the problem is the ability to handle some atypical constraints:

Suppose $x_{i}$ is a component of $\mathbf{x}$. I want to apply the following constraints:

$x_{i} = 0$ or $\mathrm{lb} \leq x_{i} \leq \mathrm{ub}$.

In geometric terms, I want $x_{i}\in [0]\cup [\mathrm{lb}, \mathrm{ub}]$.
Unlike the original problem, where all $x_{i}$ are bounded within convex sets (i.e. non-piecewise intervals).

Is there a good way to modify an LP for the above atypical constraint?

Sorry, this cannot be done with a continuous LP solver. As you observed, this construct introduces a non-convexity. However, it can be handled by a Mixed Integer Programming (MIP) solver.

$x=0$ or $\ell\le x\le u$ is what is known as a semi-continuous variable. Many advanced MIP solvers support this variable type directly.

If a MIP solver does not support this, we can introduce a binary variable $\delta \in \{0,1\}$ and use the constraint:

$$ \ell \cdot \delta \le x \le u \cdot \delta  $$

to simulate a semi-continuous variable.

COIN-OR has a capable MIP solver called CBC (I believe it supports semi-continuous variables directly).


Impact of sparse features on tree-based models Say you have a highly imbalanced binary classification problem.
Some of the features are binary features, where they're false most of the time, but when they're true they tend to be highly predictive (of the minority class).
What impact, if any, would the sparseness of the feature have on the likelihood of the feature being picked as a split?

There are of course different split algorithms and losses that change the behaviour in these cases a little bit. However if we take a look at the most common choice in modern tree based algorithms we can deduce what the impact would be.

Let's have a scenario where we have two binary features that we have to decide between and their corresponding target distribution.

$$P(f_1=0) = 0.95$$

$$P(y=0|f_1=0)= 0.5$$

$$P(y=0|f_1=1)= 0.05$$

$$P(y=0) = 0.5225$$

This means feature 1 is mostly 0, it is 50/50 whether the target is 0 in this case but if it is 1 then the target is almost always 1 as well.

Now we have a second feature which is a bit different.

$$P(f_2=0) = 0.5$$

$$P(y=0|f_2=0)= 0.4225$$

$$P(y=0|f_2=1)= 0.6225$$

In this case the feature is split evenly and there is predictive impact but it is less extreme.

Gini impurity

By reducing the gini impurity we hope to minimize the number of splits neccessary. The current gini impurity is:

$$G_I(p) = 1 - 0.5225^2 - (1-0.5225)^2 = 0.4989875$$

When splitting on a feature we need to take the weighted average gini impurity and check the reduction.

$$G_{If1=0} = 0.5$$

$$G_{If1=1} = 1 - 0.05^2 - 0.95^2 = 0.095$$

Given that most splits (95%) will go the left we weight that one much more:

$$G_{If1} = 0.95*0.5 + 0.05*0.095 = 0.47975$$

In the second feature we have:

$$G_{If2=0} = 1 - 0.4225^2 - (1-0.4225)^2 = 0.4879875$$

$$G_{If2=1} = 1 - 0.6225^2 - (1-0.6225)^2 = 0.4699875$$

Given that the splits are equal we get a normal mean:

$$G_{If2} = 0.4879875*0.5 + 0.5*0.4699875 = 0.4789875$$

In this case it is a bit close but it would choose the less sparse feature. The reason is that while the minimum impurity is much worse, because the split is completely balanced the impact in reduction on both sides has a bigger effect. In the sparse case the split to the right has a major reduction but only counts for 0.05 of the cases while the impurity on the left is at it's highest. As you can see there is a trade-off between the impurity on either side and the amount of cases where the impurity is reduced.


Best practices for selecting categorical features I'm trying to create a classifier that will predict whether someone will attend an interview or not. Each data point is for a single candidate and contains details such as the location of the interview, the candidate's current location, the job skill requirements, interview type etc. All of the data is categorical.

There are also some features, which you may not think would have an impact on the candidate's attendance e.g. marital status. My initial thought was, based on "common sense" (I use this term loosely), to drop this feature, but I wanted to take a more rigorous approach to determine the importance of this. However, I am unsure of what is best practice. How do I go about determining whether this is a feature that can be removed or not?

Secondly, is there a downside to keeping it even if it has limited predictive power? Aside from creating a more cumbersome model, could it adversely impact the accuracy of a prediction?

Thanks.

For removing categorical variable, generally used techniques is chi-square distribution based selection. The technique falls under general umbrella term univariate feature selection.


Mini Batch Gradient Descent shuffling My data set is of shape (60,784,1000) with mini batches for input and (60,10,1000) for labels, should I shuffle only the 60 mini batches or the training examples themselves?

Normally, you would shuffle up all of the examples and then portion them off into batches of some chosen size. Then, you would do a parameter update based on the gradient of the loss function with respect to each batch. This whole process is one "epoch" of training. Typically, deep neural nets are then trained over many epochs, often with a learning rate that varies as training proceeds.

An important aspect of this process is that when the data is shuffled up at the beginning of an epoch, examples are put into batches with different examples than they were matched with in the previous epoch. This gives us a more complete sampling of batch gradients and improves our collective stochastic estimation of the optimal gradient (the derivative of the cost function with respect to the model parameters and data).

Short answer: your model performance will almost certainly be worse if you choose static batches and shuffle those batches around instead of shuffling the data and then dividing them into batches.

Also, be careful of your shapes. With a dataset shape of (60, 784, 1000), it's highly likely that you're working on MNIST or one of its cousins - MNIST is 60,000 examples of length (784,), if the images have been flattened down from 28 x 28 pixels. The 784 being axis 1 in your shape tuple makes me concerned that you've reshaped your data incorrectly. Make sure that the array entries that represent a single image are where you think they are.


How to create a SAS Library Folder I am trying to create a folder to use in the SAS library. I have tried to do this many different ways. Every time I run the code it always says that the path does not exist.

Below is my error messages from my last attempt:

 72
 73         libname "D:\Users\clkehl01\Documents\myprj1\";
 ERROR: "D:\Users\clkehl01\Documents\myprj1\" is not a valid SAS name.
 ERROR: Error in the LIBNAME statement.
 74
 75         data myprj1.autos;
 76         set sashelp.cars;
 77         run;

 ERROR: Library MYPRJ1 does not exist.
 NOTE: The SAS System stopped processing this step because of errors.

library name should be defined along with libname statement and its path. try below

libname myprj1 'D:\Users\clkehl01\Documents\myprj1\';

data myprj1.autos;
set sashelp.cars;
run;


SVDD vs once Class SVM Can some one please explain me what is the difference between one class SVM and SVDD(support vector data description)

Support vector data description (SVDD) finds the smallest hypersphere that contains all samples, except for some outliers. One-class SVM (OC-SVM) separates the inliers from the outliers by finding a hyperplane of maximal distance from the origin.

If the kernel function has the property that $k(\mathbf{x}, \mathbf{x}) = 1 \quad \forall \mathbf{x} \in \mathbb{R}^d$, SVDD and OC-SVM learn identical decision functions. Many common kernels have this property, such as RBF, Laplacian and $\chi^2$.

SVDD and OC-SVM are also equivalent in the case that all samples lie on a hypersphere centered at the origin, and are are linearly separable from it.

See Lampert, C. H. (2009). Kernel methods in computer vision (Chapter 5) for more detailed descriptions of these models.


Class balancing of the dataset While performing the SMOTE for balancing the class data, what should be the proportion of both class? For instance, if we have 100 instances, what (%) should be the Yes class and what should be the No class?

As I have mentioned in the comment, there is not rule of thumb. One has to do a lot experimentation.

Based on how much data you have, bringing minority class to be be equal (50-50) to the majority class can be misleading the model.

You can also try ADASYN during your experimentation.

ADASYN is a improved version of SMOTE. What it does is same as SMOTE just with a minor improvement. After creating those sample it adds a random small values to the points thus making it more realistic. In other words instead of all the sample being linearly correlated to the parent they have a little more variance in them i.e they are bit scattered.

You can read about the implementation here


One Hot Label Encoding Scikit_learn convert back to Data Frame I have a data frame with 4 features and 1 target. The 4 features are 3 categorical and 1 numerical.

I created X which is a new data frame for the 3 categorical features. I use one hot label encoding but now it is a numpy array. Why?

Should i convert it back to a data frame? why not?

what is the best practice to merge X with my 1 numerical feature now ?

Should i convert it back to a data frame? why not?

  If you have some specific requirements like, saving data in
  a file or want to perform some specific operations which can be run
  better on DataFrame, then its a good choice to convert it back to
  dataframe. Otherwise it should be ok to go with numpy array, even
  Scikit_learn different algo takes numpy array as an input.

what is the best practice to merge X with my 1 numerical feature now ?

  I can share my experience and what exactly I did.

  Save separately and drop the categorical feature and move rest of the features in to numpy array.
  Convert categorical features in to OneHot encoding.
  Concatenate OneHot Encoding numpy array with rest of the features and consume this array for model training.


how Lasso regression helps to shrinks the coefficient to zero and why ridge regression dose not shrink the coefficient to zero? How Lasso regression helps feature selection of model by making the coefficient to zero? ,
I could see few below with below diagram ,can any please explain in simple terms how to corelate below diagram with i.) how lasso shrinks the coefficient to zero ii.) how ridge dose not shrinks the coefficient to zero

These diagrams show the "constrained" version of lasso/ridge, in which you minimize the pure loss function subject to a constraint $\|\beta\|_1\leq t$ or $\|\beta\|_2\leq t$.  (Another common version adds a penalty to the loss, and these are equivalent.)
The bluish solid shapes are the set of points with $\|\beta\|\leq t$, on the left with L1 norm and on the right with L2 norm.  $\hat{\beta}$ represents the unpenalized optimum value of $\beta$, and the ovals around it are the level curves of the loss function.
We know that the optimum constrained solution will occur at a point of tangency of the level curves of the loss function and the constraint boundary, specifically with the "smallest" level curve that touches the constraint region.  Then the point of the picture is that, with the pointy L1 constraint, you are much more likely for that point of tangency to occur at one of the corners, which correspond to one (or more, if we try to generalize to higher dimensions) of the coordinates in $\beta$ being zero.  Compare to the L2 boundary, where the tangency being exactly along one of the axes is much less likely.


Poor performance of SVM after training for rare events I found out that Weighted SVM is a classification approach to handle class imbalance problem. My data set is highly imbalanced with rare event (minority class, labeled as 1) and the majority class (label 0). So I implemented the supervised classification weighted svm technique with stratified cross-validation as these are able to handle class imbalance. I added an additional tuning for the C parameter (boxconstraint). The training is done using 5 folds cross-validation approach. The method works well on the training set. I get good performance after the training. This I can say because by looking at the confusion matrix after the training.

cmMatrix =

        1443          27
           0          30

It is generally recommended to re-train using the optimized hyperparameters. So, I ran the trained model on the entire dataset again (re-trained it) an predicted on the same dataset.

PROBLEM: If I give a highly imbalanced unseen new data set (this set is never used by the model and is the Test set) to the trained SVM model, the prediction on this data is totally biased towards the majority class as shown below

cmMatrix_TestData =

        98     2
         5     0

Where did I go wrong? Please help, I practically have no method working for class imbalance problem whereas several articles and suggestion suggest these two approaches which I am not able to make it work for me.

This I can say because I ran the trained model on the entire dataset
  again (re-trained it) an predicted on the same dataset.

You seem to be making a fundamental mistake here. If you train and test on the same data, your performance will not be representative of how the model can perform on unseen data points.

Make sure that you train and test on different datasets. If the difference in performance persists, make sure that the validation dataset is representative of your training set. If it's not, this might explain why the model is not performing well on the validation set.


Caret and rpart - does caret automatically prune rpart trees Question relating to the caret package 'rpart' method.

Does the method='rpart' automatically prune the tree?  If so, what rules does it follow?  If not, how does one go about directing caret to do this?

To give a proper background for rpart package and  rpart method with caret package:

1. If you use the rpart package directly, it will construct the complete tree by default.
If you want to prune the tree, you need to provide the optional parameter rpart.control which controls the fit of the tree. R documentation below, eg.:

    rpart(formula, data, method, control = prune.control)

    prune.control = rpart.control(minsplit = 20,
                    minbucket =  round(minsplit/3), cp = 0.01,
                    maxcompete = 4, maxsurrogate = 5, usesurrogate = 2,
                    xval = 10, surrogatestyle = 0, maxdepth = 30 )

these are the hyper parameters you can tune to obtain a pruned tree.

One followed way is to not provide the cp i.e complexity parameter and perform cross validation (xval), something like:

    rpart.control(minsplit = 20, minbucket = round(minsplit/3), xval = 10)

complexity parameter (cp) can be thought of as a measure of complexity/ no of splits of your model and you want to increase complexity until your model generalizes to new observations. i.e. regularization

Therefore evaluate the cross validated error vs cp and choose the cp that gives the good value (cp_good).

Finally, add it as your control parameter i.e. rpart.control(cp = cp_good) or use the prune function i.e. prune(fit, cp = cp_good) to get the desired tree.

2. caret package on the other hand already implements the rpart method with cp as the tuning parameter. caret by default will prune your tree based on a default run it makes on a default parameter grid (even if you don't supply any tuneGrid and trControl while training your model:

    model <- train(data,
                   labels,
                   method = "rpart")


What does it mean for a method to be invariant to diagonal rescaling of the gradients? In the paper which describes Adam: a method for stochastic optimization, the author states:

The method is straightforward to implement, is computationally
efficient, has little memory requirements, is invariant to diagonal
rescaling of the gradients, and is well suited for problems that are
large in terms of data and/or parameters.

What does "invariant to diagonal rescaling of the gradients" imply here?

(Copied from my answer on stats.SE. (This meta.SE answer seems to approve of this copy-paste pattern.))

The original Adam paper briefly explains what it means by "invariant to diagonal rescaling of the gradients" at the end of section 2.1.

I would try to explain it in some more detail.

Like stochastic gradient descent (SGD), Adam is an iterative method that uses gradients in order to find a minimum of a function.
(By "gradients" I mean "the values of the gradient in different locations in parameter space". I later use "partial derivatives" in a similar fashion.)

But in contrast to SGD, Adam doesn't really use gradients. Instead, Adam uses the partial derivatives of each parameter independently.
(By "partial derivative of a parameter $x$" I mean "partial derivative of the cost function $C$ with respect to $x$", i.e. $\frac{\partial C}{\partial x}$.)

Let $\Delta^{(t)}$ be the step that Adam takes in parameter space in the $t^{\text{th}}$ iteration. Then the step it takes in the dimension of the $j^{\text{th}}$ parameter (in the $t^{\text{th}}$ iteration) is $\Delta^{(t)}_j$, which is given by:
$$\Delta^{(t)}_j=-\frac{\alpha}{\sqrt{{\hat v}^{(t)}_j}+\epsilon}\cdot {\hat m}^{(t)}_j$$
while:

$\alpha$ is the learning rate hyperparameter.
$\epsilon$ is a small hyperparameter to prevent division by zero.
${\hat m}^{(t)}_j$ is an exponential moving average of the partial derivatives of the $j^{\text{th}}$ parameter that were calculated in iterations $1$ to $t$.
${\hat v}^{(t)}_j$ is an exponential moving average of the squares of the partial derivatives of the $j^{\text{th}}$ parameter that were calculated in iterations $1$ to $t$.

Now, what happens when we scale the partial derivative of the $j^{\text{th}}$ parameter by a positive factor $c$?
(I.e. the partial derivative of the $j^{\text{th}}$ parameter is just a function whose domain is the parameter space, so we can simply multiply its value by $c$.)

${\hat m}^{(t)}_j$ becomes $c\cdot{\hat m}^{(t)}_j$
${\hat v}^{(t)}_j$ becomes $c^2\cdot{\hat v}^{(t)}_j$
Thus (using the fact that $c>0$), we get that $\Delta^{(t)}_j$ becomes:
$$-\frac{\alpha}{\sqrt{c^2\cdot{\hat v}^{(t)}_j}+\epsilon}\cdot c\cdot{\hat m}^{(t)}_j=-\frac{\alpha}{c\cdot\sqrt{{\hat v}^{(t)}_j}+\epsilon}\cdot c\cdot{\hat m}^{(t)}_j$$
And assuming $\epsilon$ is very small, we get:
$$\begin{gathered}
-\frac{\alpha}{c\cdot\sqrt{{\hat v}^{(t)}_j}+\epsilon}\cdot c\cdot{\hat m}^{(t)}_j\approx -\frac{\alpha}{c\cdot\sqrt{{\hat v}^{(t)}_j}}\cdot c\cdot{\hat m}^{(t)}_j=\\
-\frac{\alpha}{\sqrt{{\hat v}^{(t)}_j}}\cdot{\hat m}^{(t)}_j\approx-\frac{\alpha}{\sqrt{{\hat v}^{(t)}_j}+\epsilon}\cdot{\hat m}^{(t)}_j
\end{gathered}$$

I.e. scaling the partial derivative of the $j^{\text{th}}$ parameter by a positive factor $c$ actually doesn't affect $\Delta^{(t)}_j$.

Finally, let $g=\left(\begin{gathered}g_{1}\\
g_{2}\\
\vdots
\end{gathered}
\right)$ be the gradient. Then $g_j$ is the partial derivative of the $j^{\text{th}}$ parameter.

What happens when we multiply the gradient by a diagonal matrix with only positive elements?
$$\left(\begin{matrix}c_{1}\\
 & c_{2}\\
 &  & \ddots
\end{matrix}\right)g=\left(\begin{matrix}c_{1}\\
 & c_{2}\\
 &  & \ddots
\end{matrix}\right)\left(\begin{gathered}g_{1}\\
g_{2}\\
\vdots
\end{gathered}
\right)=\left(\begin{gathered}c_{1}\cdot g_{1}\\
c_{2}\cdot g_{2}\\
\vdots
\end{gathered}
\right)$$
So it would only scale each partial derivative by a positive factor, but as we have seen above, this won't affect the steps that Adam takes.

In other words, Adam is invariant to multiplying the gradient by a diagonal matrix with only positive factors, which is what the paper means by "invariant to diagonal rescaling of the gradients".


Fill missing values AND normalise I have two columns of training data for a neural net which are missing values. (There are many other columns which aren't missing values.)

For example

Height  | Weight
180     | 70
175     | N/A
N/A     | N/A

I want to fill missing values and normalise the columns.

The data is heights and weights so I thought a good fill value would be 0 or -1. This is based on the book Deep Learning in Python:

  In general, with neural networks, it's safe to input missing values as 0, with the condition that 0 isn't already a meaningful value.

EDIT I assumed 0 wasn't meaningful in a dataset with values from 150-200

I was also recommended to normalise the data by subtracting the mean and dividing by the std for each column.

Both of those are fine on their own - I understand how  and why to do them. What I don't get is how to combine them. I can either ...

Fill missing values then normalise, but then a) my zeros will no longer be zeros (will my network still learn they are a special value?), and b) the zeros will affect the mean/std to a degree determine by how many values are missing. I suppose I'm concerned this would give a weird distribution
Normalise then fill missing values. But after I've normalised my data, 0 is now the mean of my column and so isn't a fill value of the same kind. I'd rather let the network know the values are unfilled than assume they all take the mean value

I'm using Keras, Numpy and Pandas with Dense layers for a multiclass classification problem.

I don't understand why you would like to fill values with zeros ! This would basically mean, "this guy, who is 170 cm tall, weights 0 kg" and would fool your network. In my opinion, you have two options:

discard missing values (the entire row): you end up with less but more consistent training data
if you really need these rows, then fill missing values with some heuristics: for example, give them the mean of their column, or apply a simple linear regression. Beware that this will add a bias in the learning process, but it would be definitely better than giving random values.

At very least, if you have a lot of missing values, then maybe you should think of selecting a specialized model for partial training data. You actually don't tell us what network you are using, but you might modify it to handle missing values more intelligently than by filling missing values.

Finally, the need of normalizing once again depend on your model (which you don't describe). But this would definitely come after the processing part.


What are the good parameter ranges for BERT hyperparameters while finetuning it on a very small dataset? I need to finetune BERT model (from the huggingface repository) on a sentence classification task. However, my dataset is really small.I have 12K sentences and only 10% of them are from positive classes. Does anyone here have any experience on finetuning bert in small datasets? Do you have any suggestions for learning rate, batch size, epoch number warmup steps etc.?

How many classes do you have? Bert can handle a high-quality 12k dataset for binary classification. I recommend duplicating your positive test case 4x and sampling a 5k test cases from your negative class. This will give you a balanced dataset. Then implement BERT in google colab using the original GitHub repository from the google Bert team. You can find a functional collab here: https://colab.research.google.com/github/tensorflow/tpu/blob/master/tools/colab/bert_finetuning_with_cloud_tpus.ipynb
On collab, TPUs takes about 10 to 15 minutes to run. All you have to do is use write your data processor.
The following parameters are a good place to start.

TRAIN_BATCH_SIZE = 32
EVAL_BATCH_SIZE = 8
PREDICT_BATCH_SIZE = 8
LEARNING_RATE = 2e-5
NUM_TRAIN_EPOCHS = 3.0
MAX_SEQ_LENGTH = 128
# Warmup is a period of time where hte learning rate
# is small and gradually increases--usually helps training.
WARMUP_PROPORTION = 0.1
# Model configs
SAVE_CHECKPOINTS_STEPS = 1000
SAVE_SUMMARY_STEPS = 500


Is feature selection necessary? I would like to run some machine learning model like random forest, gradient boosting, or SVM on my dataset. There are more than 200 predictor variables in my dataset and my target classes are a binary variable.

Do I need to run feature selection before the model fitting? Does it affect the model performance significantly or is there not much difference if I directly fit the model using all predictor variables?

Feature selection might be consider a stage to avoid.
You have to spend computation time in order to remove features and actually lose data and the methods that you have to do feature selection are not optimal since the problem is NP-Complete.
Using it doesn't sound like an offer that you cannot refuse.

So, what are the benefits of using it?

Many features and low samples/features ratio will introduce noise into your dataset. In such a case your classification algorithm are likely to overfit, and give you a false feeling of good performance.
Reducing the number of features will reduce the running time in the later stages. That in turn will enable you using algorithms of higher complexity, search for more hyper parameters or do more evaluations.
A smaller set of feature is more comprehendible to humans. That will enable you to focus on the main sources of predictability and do more exact feature engineering. If you will have to explain your model to a client, you are better presenting a model with 5 features than a model with 200 features.

Now for your specific case:
I recommend that you'll begin in computing the correlations among the features and the concept. Computing correlations among all features is also informative.
Note that there are many types of useful correlations (e.g., Pearson, Mutual information) and many attributes that might effect them (e.g., sparseness, concept imbalance). Examining them instead of blindly go with a feature selection algorithm might save you plenty of time in the future.

I don't think that you will have a lot of running time problems with your dataset. However, your samples/features ratio isn't too high so you might benefit from feature selection.

Choose a classifier of low complexity(e.g., linear regression, a small decision tree) and use it as a benchmark. Try it on the full data set and on some dataset with a subset of the features. Such a benchmark will guid you in the use of feature selection. You will need such guidance since there are many options (e.g., the number of features to select, the feature selection algorithm) an since the goal is usually the predication and not the feature selection so feedback is at least one step away.


Is XGBoost better with numeric predictors? I have a categorical feature that I one-hot encoded and used in my XGBoost model, but it consistently underperforms as a predictor compared to the other predictors.

Then I created a new variable that contains the same kind of information that the categorical feature has...

  Imagine I'm interested in predicting price of a house and the
  categorical feature is the town that it is in, and the numerical
  feature is a feature I generated ranking the town's relative
  expensiveness, based on some prior knowledge of the towns.

TownA - 100
TownB - 40
TownC - 65
TownD - 15

Now all of a sudden the new numerical variable that was directly derived from the categorical one is outperforming. Is this because XGBoost just works better with numerical variables and maybe some predictive capability is lost when I one-hot encode the variable?

By default, feature importance in xgboost is given by how many times a given feature appears as a split feature across all trees in the ensemble.

When one-hot encoded, each newly created dummy variable can only take the values 0 and 1, and so can only appear once in each (sub)tree. However, when combining the values into one numeric feature by giving each category a different value, the feature can appear many more times on different levels in each tree, which brings up the importance score.


How to compare fast the value with the ones of neighbours in a 3D array Question:

I want to compare every item in a 3D array with its first neighborhoods. It's really slow when I have a 500x500x500 (e.g. with only values of 0, 1, 2) ndarray. I post the principle lines here:

import numpy as np

# Create a list to stock all the neighbours' coordinations of the voxel wanted
def check_neighbor(array, x, y, z):
   #top
   t = array[x, y , z + 1]
   #down
   d = array[x, y , z - 1]
   #left
   l = array[x, y - 1 , z]
   #right
   r = array[x, y + 1 , z]
   #front
   f = array[x - 1, y , z]
   #back
   b = array[x + 1, y , z]
   return [t, d, l, r, f, b]

# Check the voxel with all its neighborhood
def compare_neighbor(array, Value2match, Value2Bmatched):
   for index in np.argwhere(array==Value2match)
      output[index] = 1 if Value2BMatched in checkneighbor(array, index[0],index[1], index[2]) else 0
   return output

# Main
array = np.random.randint(3, shape=(500, 500, 500))
output = compare_neighbor(array, 1, 2)

This code take me hours only for the n=1 neighbour! Is there an efficient way which can also check the 2, 3... nearest neighbours? Can somebody help me?

Solution 1:

Based on jayprich's answer and n1k31t4's comments, I fused the both function into one and replace the argwhere() by where(). The advantage of this code is that we don't iterate voxel by voxel but do it in a vectorized way:

import numpy as np
# Build a helper function to SHIFT(not roll) a 3Darray
def shift_helper(array, neib_value, shift=0, axis=0):
    #Roll the 3D array along axis with certain unity
    _array = np.roll(array == neib_value, shift=shift, axis=axis)

    # Cancel the last/first slice shifted to the first/last slice
    if axis == 0:
        if shift >= 0:
            _array[:1, :, :] = 0
        else:
            _array[-1:, :, :] = 0
        return _array
    elif axis == 1:
        if shift >= 0:
            _array[:, :1, :] = 0
        else:
            _array[:, -1:, :] = 0
        return _array
    elif axis == 2:
        if shift >= 0:
            _array[:, :, :1] = 0
        else:
            _array[:, :, -1:] = 0
        return _array

def compneib(array, that_value, neib_value):
    _array = np.zeros(array.shape)
    _array[np.where((array == that_value)
                    & (shift_helper(array, neib_value, shift=-1, axis=0)
                    | shift_helper(array, neib_value, shift=1, axis=0)
                    | shift_helper(array, neib_value, shift=-1, axis=1)
                    | shift_helper(array, neib_value, shift=1, axis=1)
                    | shift_helper(array, neib_value, shift=-1, axis=2)
                    | shift_helper(array, neib_value, shift=1, axis=2)
                    ))] = 1
    return _array

# Main
array = np.random.randint(3, shape=(500, 500, 500))
output = compneib(array, 1, 2)

Solution 2:

If there would still be a faster way.

NumPy performs operations in a vectorised manner, you should try and operate on the whole array and avoid explicit loops.

e.g. np.argwhere( (test == 1) & np.roll( test==2 , shift=-1 , axis=0 ) )

can be done for different shift and axis values and results combined


MinMaxScaler broadcast shapes I use a neural network with 3 inputs and 1 output with Keras. I'm using MinMaxScaler from sklearn to normalize my inputs in the range [0,1]

my input shape is (XX,3)
my output shape is (XX,1)

I don't have any input while scaling the input and output arrays

    self.scaler = MinMaxScaler(feature_range=(0,1))

    dataX = self.scaler.fit_transform(dataX)
    dataY = self.scaler.fit_transform(dataY)

    self.model.fit(dataX,dataY,nb_epoch=1000)

After the training I have able to make prediction like this :

    data    = numpy.array([[ val1,val2,val3]])
    data = self.scaler.fit_transform(data)
    prediction = self.model.predict(data)

but my output is not scaled correctly, it make sense, as I didn't call inverse transform to apply the inverse scaling

but when I call it, I get a ValueError , I tried to apply the transformation on a single prediction or on an array of prediction but the problem is the same

self.scaler.inverse_transform(predictions) # prediction shape is (18,) :

ValueError: operands could not be broadcast together with shapes (18,) (3,) (18,)

ValueError: non-broadcastable output operand with shape (18,1) doesn't match the broadcast shape (18,3)

I undestand it is a shape issue, but the error message does not help so much

(12,1) was accepted by fit_transform (18,1) seems pretty similar.. so I dont get the error.

You made some mistakes on MinMaxScaler.

MinMaxScaler shouldn't be fitted twice(as internal parameters inside MinMaxScaler will be changed), and dataX & dataY should have their own scaler(as they have different minimum and maximum values.

Try to do something like this:

self.x_scaler = MinMaxScaler()
self.y_scaler = MinMaxScaler()

dataX = self.x_scaler.fit_transform(dataX)
dataY = self.y_scaler.fit_transform(dataY)

self.model.fit(dataX, dataY, nb_epoch=1000)

data = numpy.array([[val1, val2, val3]])
data = self.x_scaler.transform(data)
prediction = self.model.predict(data)
real_prediction = self.y_scaler.inverse_transform(prediction)


Linear Discriminant Analysis + bayesian theorem = LDA classifier?? I am new to machine learning and as I learn about Linear Discriminant Analysis, I can't see how it is used as a classifier.

I can understand the difference between LDA and PCA and I can see how LDA is used as dimension reduction method. I've read some articles about LDA classification but I'm still not exactly sure how LDA is used as classifier.

From what I understand, we consider the features vector x as multivariate gaussian distribution and use the bayesian rule to calculate which category gives the max probability for x, then it belongs to that category.

Is this understanding generally correct?

I found the answer in: https://machinelearningmastery.com/linear-discriminant-analysis-for-machine-learning/

  Making Predictions with LDA

  LDA makes predictions by estimating the probability that a new set of
  inputs belongs to each class. The class that gets the highest
  probability is the output class and a prediction is made.

  The model uses Bayes Theorem to estimate the probabilities. Briefly
  Bayes’ Theorem can be used to estimate the probability of the output
  class (k) given the input (x) using the probability of each class and
  the probability of the data belonging to each class:

  P(Y=x|X=x) = (PIk * fk(x)) / sum(PIl * fl(x))

  Where PIk refers to the base probability of each class (k) observed in
  your training data (e.g. 0.5 for a 50-50 split in a two class
  problem). In Bayes’ Theorem this is called the prior probability.

  PIk = nk/n

  The f(x) above is the estimated probability of x belonging to the
  class. A Gaussian distribution function is used for f(x). Plugging the
  Gaussian into the above equation and simplifying we end up with the
  equation below. This is called a discriminate function and the class
  is calculated as having the largest value will be the output
  classification (y):

  Dk(x) = x * (muk/siga^2) – (muk^2/(2*sigma^2)) + ln(PIk)

  Dk(x) is the discriminate function for class k given input x, the muk,
  sigma^2 and PIk are all estimated from your data.

So yes, it uses Bayes Theorem to estimate the probabilities.


Do you apply confidence threshold before calculating mAP for YOLOv2? I've been using YOLO v2 for object detection and have been trying to use mAP for validating the model.

For training I use a confidence threshold of 0.25, however I am unsure as to whether to apply this threshold to filter out predictions BEFORE calculating mAP in validation? or whether the threshold should be ignored and all proposals should be evaluated?

In the end the solution is not to apply the threshold prior to the mAP calculation. After researching further it turns out by definition that mAP is (roughly speaking) a measure of precision recall trade off calculated using all possible thresholds. By applying the threshold first you are not accounting for the higher recall the model would obtain by choosing a threshold less than or equal to the one applied.


How to visualize segmentation result with CrossEntropyLoss I know this question already has a clear answer about how to visualize segmentation result. My question is if I'm using the CrossEntropyLoss as the loss function, after training. The predicted classification (segmentation) result doesn't represent the class 0,1,2. It represents the unnormalized Log probability as told in cs231n standford class.
So if I want to visualize the segmentaion result, should I use the

- nn.LogSoftmax()

to get the classes and then plot them out?
or should I do a maximum function over the first dimension of my output.

For example, if the predicted output is size 3x1024x1024, how could I visualize the segmenation result?

Thank you in advance.

In either case (with or without using nn.LogSoftMax()) you should do some argmax operation to get the class from probabilities.

The reason for that is that order of probabilities will always stay the same after the nn.LogSoftmax() since that operation squeezes the numbers in the [0, 1] interval, but keeps the ordering (biggest one before this operation will also remain the biggest in the [0, 1] interval, and so on ...).

Hence, for the purpose of determinating the class, it doesn't matter whether you do nn.LogSoftmax() or not.

nn.LogSoftmax() is needed only during the training process in order to calculate the loss and backpropagate that information to update the parameters of the network.


Understanding LSTM text input I am an beginner in text generation and and deep-learing but I like to get in touch with it. Currently I am learing about LSTM networks and VAEs for text generation. I would like to read a sequence at once and output another sequence. What I learned from this post is that the input shape should be 3D. Since my text data is 2D I want to reshape it. But this is now the confusing part. As I have training data with the shape of (nb_sequences, max_seq_lenth) = (274, 95). As the post pointed out, it would be important to reshape the data to a format of input = (nb_sequence, nb_timestep, nb_feature) with nb_sequence being the number of sequences (274), nb_timestep the length of the sequence (95) and nb_feature the number of features describing a sequence at a timestep, right?
I have two questions here:
1). How to reshape properly? My vocab size is 359 which I imagined (obviously wrong) to be the number of the features. --> input = (274, 95, 359). This is not possible since an array of 274*95 = 26030 can't be reshaped into (274, 95, 359).
If I take the number of diffrent words my sequences are containing, wouldn't it lead to some erros since sentences do not consist out of the same words naturally? Meaning that I would have a variable feature number with each sequence?
2). How would one realize the problem I am facing? I read about one-hot encoding the sequences. This might solve the problem I have regarding question 1), but as far as I understand this would lead me to a quite inefficient way of predicting sequences as I want to handle more data. Is there any suggestion how to do it in a effient way?

The input to an LSTM must be a batch of sequences of vectors of real numbers, i.e. 3D tensor). Textual inputs are discrete tokens, so they are a batch of sequences of integers (indexes to the vocabulary table), i.e. a 2D tensor.
Before any reshaping, you must transform each integer value into a vector. For that, the usual approaches are one-hot encoding and embedding tables:

With one-hot encoding, you encode each integer index as a vector with 359 elements, where all are 0s except the position at the integer index, which is a 1.

With an embedding table, you keep a table with 359 vectors of an arbitrary dimensionality (the embedding size).

In either case, you get the "extra dimension" that you were lacking with integer indices.


Tokenize sentence based on a dictionary I have few key words say - RAM, Speaker, Brand, Display etc. I have made a dictionary of all possible values these keys can have. Like - RAM can have 12, 12gb, 12GB; Display can have 12inch, 12", 12 inch.

I am given a title say - "Apple iPhone 5s (Space Grey, 16GB)"

I have to extract tokens from these titles and map to the dictionary values and assign appropriate keyword to each token.

Like for this - Brand - Apple, Color - Grey, Storage - 16GB

How should i tokenize these sentences? Doing it just by space will not be enough- like in this title - "Samsung Guru Music 2 SM-B310E (White)" 2 will be one token and can map to RAM,Display anything.

Will any NLP library help with this? I am using python to code and new to NLP.

Look at the NLTK library for Python, there are functions to facilitate tokenizing sentences.


Linear Regression with vs without polynomial features I have a conceptual question about why (processing power/storage aside) would you ever just use a regular linear regression without adding polynomial features? It seems like adding polynomial features (without overfitting) would always produce better results. I know linear regression can fit more than just a line but that is only once you decide to add polynomial features correct? My experience is with python using sklearn's libraries.

Occam’s razor principle [ref]:

Having two hypotheses, that has the same empirical risk (here, training error), a short explanation (here, a boundary with fewer parameters) tends to be more valid than a long explanation.

Hence, complexity must add additional comparable accuracy. Otherwise, we should go with the simple model
Let's say you have 50-50 chance of Linear and Non-Linear data. With your approach, you will end up using regularization in 50% of the model. First, make a complex model and then regularize to make it simple.
Overall, you will increase the complexity which might impact future activities e.g Maintenance
Interpretability
 Many businesses demand a reason for the decision e.g. Credit card related decision. Linear regression will give you a simple explanation for each feature
Also, read these SE answers


doubt in improvement of performances given by a layer in a deep neural network Today I was discussing about neural network with a friend, and he said that the more layers we add, the less increase in accuracy each layer gives individually. Is this fact true?

I know that it is better to go deep then wide when building a neural network, but it is true that if we have a lot of layers, individually they give a smaller contribution to the performances? Thanks in advance.

Yes, there is a threshold where additional layers are too much.
Look at  residual networks


If categorical variable has some hierarchy, should I just label them or split into dummy variables (One-Hot encode)? I have a column which has 5 unique categories. There's a hierarchy between these categories (Best > good > OK/Not Sure > Bad > Worst)

In this case, should I label them based on hierarchy like:

Best: 4

good: 3

OK/Not Sure: 2

Bad: 1

Worst: 0

Or should I perform one-hot encoding/Dummy encoding? What's the intuition behind how the model perceives the two things?

Note: I should also mention that this is for predictive modeling using Logistic Regression.

One hot encoding should be performed between independent values like flowers type etc.
Values that you have mentioned are relative rank where bad is better than worst and so on.
So it should not be one hot encoding except converting it into numerical values.


When I should use PCA? I have a data set with 60000 rows and 32 columns. I want to use SVM (with some more constraints that make it more complicated)and I think 32 columns are too large. So I decided to use PCA. But when I load PCA, the first 20 component describe 85% of data which 20 variables still is too large I think, but it is better than 32.
I am wondering is that ok if I use PCA? when n>>m? if not what is a better option?

If you can, go for some non-linear dimensionality reduction technique. The most powerful are Autoencoders, but you can also use t-SNE or other manifold techniques.

The problem of PCA is that it can extract only latent factors that are linearly associated with your variables. Using non-linear techniques, less variables can let you capture more of the original variance.


String to Data frame column I have 2 column in data frame, X and Y. And I have some string values stored in text, which I want to put in X, Y as shown in the example.

Example :

text=9 10 13 110 14 16
     12 1 6   1  1  2

X    Y

9    12

10   1

13   6

110  1

14   1

16   2

I will assume your text is in two strings like this:

In [1]: import pandas as pd

In [2]: text1 = "9 10 13 110 14 16"

In [3]: text2 = "12 1 6 1 1 2"

A one-liner solution would be:

In [4] df = pd.DataFrame.from_records(zip(text1.split(" "), text2.split(" ")))

A Pandas Dataframe can be created by passing it one list (or tuple) for each row that you want in the table. This is done by using the from_records() method you see above.

So the steps that make the above line work:

split() each string on the spaces, to get a list of strings - one per value.
create each row that we want in the dataframe, which is each matched pair from the two lists of values. zip does exactly that for us.
Put the result into the from_records() method.

The final result:

In [7]: df
Out[7]:

0   1
0    9  12
1   10   1
2   13   6
3  110   1
4   14   1
5   16   2

Because we just gave the dataframe lists of strings, the values are still strings in teh dataframe. If you want to actually use them as number, you can use the astype() method, like this

df_integers = df.astype(int)    # now contains integers
df_floats = df.astype(float)    # now contains floats, i.e. decimal values


pivoting two column in pandas How can I transfer two column features into pivot table on the following dataset

I have tried the aggfunc function but this fill the value either 0 or 1. I want to transfer the row as cell value.

Here is the dataset

content  Users
22       1196
23       1196
23       1216
16       880
20       880
20       1224
22       1245
23       1122
22       872

I want to transfer this dataset into this

Users   1    2
1196    22  23
1216    23  NaN
880     16  20
1224    20  NaN
1245    22  NaN
1122    23  NaN
872     22  NaN

I have tried it by using

df.pivot_table(index=["city"], columns="cuisine", aggfunc=lambda x: 1, fill_value=0)

  df.pivot_table(index=["users"], columns="content", aggfunc=lambda x: 1, fill_value=0)

But this fills the values either 0 or 1

You can accomplish what you want by doing:

  df.groupby('Users')['content'].unique().apply(pd.Series)

First, what you want is to groupby your values by users, to get all content values. Since you want unique ones, you can apply unique() built-in pandas function.

Then, since you will have a columns with a list of values, you can apply pd.Series to each row, to expand it into columns, by default, they are named from 0 to n, where n is the maximum length of a list, in this case, two.


Improving LSTM Time-series Predictions I have been getting poor results on my time series predictions with a LSTM network. I'm looking for any ideas to improve the model.

The above graph shows the True Data vs. Predictions. The True Data is smooth zig zag shaped, from 0 to 1. However the predictions rarely reach 0 or 1.

The distribution in the prediction data-set rarely reaches 0 or 1 and it's centered around 0.5.

However the distributions in the True Data set is evenly distributed.

Here is the LSTM model built in keras:

model = Sequential()
model.add(Dropout(0.4, input_shape=(train_input_data_NN.shape[1], train_input_data_NN.shape[2])))
model.add(Bidirectional(LSTM(30, dropout=0.4, return_sequences=False, recurrent_dropout=0.4), input_shape=(train_input_data_NN.shape[1], train_input_data_NN.shape[2])))
model.add(Dense(1))
model.compile(loss='mae', optimizer='adam')

How do I get the predictions to be more similar to the true data?

I have just used LSTM to train a model predict time series value and get good result as below:

# reshape input to be [samples, time steps, features]
trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))
testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))

model = Sequential()
model.add(LSTM(40, input_shape=(trainX.shape[1], trainX.shape[2])))
model.add(Dense(1))
model.compile(loss='mean_absolute_error', optimizer='adam')
model.fit(trainX, trainY, epochs=10, batch_size=1, verbose=0)

Here is my result: blue is real value, orange is predict.


What is the best way to propose an item from a set based on previous choices? The goal of this question is to be able to propose a user further choices based on his past experiences: like Amazon's book advices.

From a set of mp3 files, I assume that a set of mp3 tags data is already filled, based on the music he/she has alredy listened to  : what is the easiest way to  implement a machine learning that is able to propose a list of music choices  based on the user's set ?

NB : I'm a Machine learning novice I'd appreciate if the answer could be based on Orange, Weka or these kind of tools.

Update: removed the classification tag as recommanded.

For new comers as me:
- the book Predictive Analytics For Dummies is a nice general introduction about this subject
- a next step would be the the paper Finding Clusters of Similar Artists which is really interesting especially for the K-means approach
- The millions songs dataset which is a gold mine with its huge dataset as well as tutorials with Python codes to use with it
- special thanks sheldonkreger for his answer with neo4J graph usage idea wich is really interesting

There are many great ways to handle this problem. It is a recommendation problem, not a classification problem, as pointed out by others. There are many ways to do recommendation with a data set like this. I'll point out a few methods and you can choose one or try them all.
The first method is called user-based collaborative filtering. The basic idea is to give users recommendations based on the tastes of like-minded users. So, you'd be trying to recommend music based on the listening history of users who have listened to the same songs. Such data can be modeled as a graph or sparse matrix. Then, you choose the exact algorithm depending on how you want to model your data.
The second method is called item-based collaborative filtering. Rather than associating users together, this strategy looks at the set of items a user has 'rated' (the songs a user has listened to) and calculates how similar they are to a specific target item (song), or even to all the songs in your data set. It grabs the set of most-similar items and uses various methods to predict how much a user will like the song.
In this case, you only have binary data (user listened to it or they did not). These calculations tend to work best with actual rating scores (like a 5 star system) because this gives more detailed variation amongst items in the data set.
The third option is to model your data in a graph database like Neo4J and write graph traversal queries in order to find similar items. If you like graph theory, this can be a lot of fun. The sky is the limit in regards to what kinds of traversals will return good results. To get started, think of the users and songs as nodes in the graph, and 'listened' as the edge. $user->listened->$song
Because of ratings and item-based filtering, and because there are probably many songs in your data set, and each user only listens to a very small portion of them, I'd first try a user-based collaborative filtering method which uses sparse matrix operations to calculate recommendations. If your data set is large, these computations scale horizontally so you can leverage parallel processing if you run into performance issues.
You can find more detail about collaborative filtering in this paper.


Do timesteps must have the same temporal distance in training a RNN? I have a recurrent neural network with LSTM units that I want to train with batches of 6 timesteps. Each timestep is a record of a dataset and represents the temporal aggregation over 5 minutes of data taken every 10 seconds. Unfortunately, the dataset has temporal interruptions, so sometimes two successive records in the dataset can be temporally separated by 10 minutes to a few weeks. I was wondering if the timesteps I give to the network in each batch must all be separated by the same temporal distance, that is in my case they must all be within 5 minutes of each other or it is sufficient that they are subsequent and therefore I could also give the network for example a batch with 4 records at a distance of 5 minutes from each other and then 2 taken for example two days later.

UPDATED:
The data comes from an electronic system so the time interruptions are due to the device being turned off. Rarely it is turned off only for a few minutes while most of the time it remains off for longer periods, like hours or sometimes days. Among other elements, the device includes a battery, therefore many of the features present in the dataset records are temperature, current and voltage readings which change value during temporal interruptions (little when turned off for a few minutes and significantly when turned off for hours or days). The goal of the network is to predict the value of some features two steps ahead.

As always in ML modelling problems: it depends.

The critical factor here is that you are predicting based on properties of a sequence. The sequence does not need to be sampled at fixed time steps, or even be time based at all. E.g. in natural language processing the sequence of letters or words is only very loosely associated with the timing of the same items when spoken or read. The sequence does need to be drawn from a self-consistent source where moving from one item to next item in the sequence has the same meaning with respect to what you are trying to predict.

If the interruptions you see would cause large differences in the expected distribution of your input vector X or output Y, then you may not be able to use them as-is. You may still be able to rescue them as input data for a RNN, and make predictions based on them, if you add whether or not there has been an interruption - and if it might be relevant to your purpose, for how long - as additional features of input items in X.

Another option is to follow the time sequence strictly but add placeholder value of X (with a different flag set) for "unkown", maybe along with some rough guess for the missing features (or even a ML prediction from a different model), which you might do if the quantity being aggregated over is still active and important where there are missing records, but the aggregation service you are using as input has failed.

Depending on how much data you have available, and whether you need to make predictions at times when your aggregation service has failed, you could do anything from discarding incomplete data to ignoring the time differences, to interpolating missing records.

The first question you need to answer is "what does it mean for a sequence item to be missing":

If the whole system that provided the aggregation data is switched off, and there is no concept of pending items that will enter the system the moment it is switched on, then you can probably safely ignore the missing records and ignore the precise timing.
If your prediction target is sensitive to elapsed time, and yo uneed to make predictions even when there have been interruptions to the aggregation system, then you will want to augment the sequence data somehow to account for the missing records, and train/predict with that added feature engineering.
If only the aggregation is switched off (or has a chance of failure causing gaps in the record), and you don't need to run any predictions when it is not available, then you may be able to make the rule that the prediction system will only function when there are 6 recent consecutive records available, and discard the data that is causing you to worry. This makes sense if interruptions are rare or the predictive model is not critical.

If you are not sure which of these things to do, then the next approach is to do all of them that seem that they could be intuitively correct, tuning each model separately, then see how each different approach performs on your test set.


How Object oriented programming approach can be utilised in data science domain? I have been asked many times that why you are not using classes, objects, and object oriented concepts while coding?
What I do?

Write python scripts to fetch data from different sources, apply
pre-processing code, create necessary variables, and take these
refined variables to modeling process.

How I do?

Write necessary functions to carry-out specific task, all the
functions will be called one-by-one as top down, procedural approach.
In case of different modules, I keep them into different python files,
and then simply import them.

In the current process, I never found a motivation to use OOPs concepts.
Can you suggest how can I improve my coding practice, and what are the flaws I am putting into my current architecture which can be fulfilled by OOPs?

This is a really interesting question about the motivation behind Object orientated programming.
The main reason that makes OOP particularly attractive is the reusability of functions and objects which you can reuse in different analysis.
For example, for one NLP model, you create a function which cleans text (returns lowercase text, free from stopwords). A few days later you want to generate another NLP model which preprocesses text in the same way as your previous NLP model. Here you can put the clean text method instead a Preprocessing class and reuse the clean text method to clean the text in the same way as before.
Another reason behind OOP is its legibility. I find it is noticeably easier to track where bugs arise in OOP code than functional programming code. This, in turn, makes it easier to test and debug separate components within a class.
This is my take on it, and I am sure everyone has their own way of programming that suits you and the teams you work in.
I look forward to reading other people's views/answers to this query.


Linear Regression in python with multiple outputs I have a time series dataset which represented as following:

x=[
       [12.19047619,  18.28571429,   6.0952381 ] ,

       [ 80.98765432,  14.17283951,  11.13580247 ] ,

       [ 50.82644628,  16.26446281,   9.14876033 ] , .... ]

and to predicted -->
 Y  = [13.9,  18,   14.987]

How I can use LASSO and SVR linear regression models in python to predict Y (which represented as a vector as shown in the above example)

Both Lasso and SVM are available in sklearn library. Lasso: sklearn.linear_model.Lasso. SVM: sklearn.svm.SVT

An example from Lasso page:

>>> from sklearn import linear_model
>>> clf = linear_model.Lasso(alpha=0.1)
>>> clf.fit([[0,0], [1, 1], [2, 2]], [0, 1, 2])
Lasso(alpha=0.1, copy_X=True, fit_intercept=True, max_iter=1000, normalize=False, positive=False, precompute=False, random_state=None, selection='cyclic', tol=0.0001, warm_start=False)
>>> print(clf.coef_)
[0.85 0. ]
>>> print(clf.intercept_)
0.15...

In your case clf.fit looks like this:

clf.fit(X, Y)

X should be the size (nn,n)
Y should be the size nn
Where nn is the number of observations (points) and n is the number of variables. So rows in X are observations and columns are different variables.

If you have more variables than observations then you should read this post about the problems you can have with it and how to solve them.


What is the best way to encode an arbitrary collection of strings into int categorical variables? I have a bunch of categorical labels which I want to transform into int categorical features for an ML algorithm.
The problem is I don't have a prior list of the categories, so that I can't just define a dictionary or mapping function before hand.

Say for example that I am using food labels - my current data has the following labels:
['Steak','Potatoes','Soup'], but it is possible that later, I will gate data with the labels 'Asparagus' or 'Chow mein', and I have no way of knowing the list of all potential labels before hand. Moreover it is possible that some of the incoming labels are proper names or strings that are idiosyncratic and not part of any standard vocabulary, e.g. 'Double Super Mac-Whopper'.

I thought of simply building my own hash map, but then I would have to put a lot of effort into saving and versioning the resulting map to maintain consistency across experiments and later in production.

I tried using the int.from_bytes function in Python 3, but it gives wildly varying int sizes (I think because it is using string length):

> int.from_bytes('steak'.encode('utf-8'),'little')
461195539571
> int.from_bytes('milk'.encode('utf-8'),'little')
1802266989
> int.from_bytes('Bok Choy'.encode('utf-8'),'little')
8750327238520172354

I looked at the sklearn categorical encoders (preprocessing.LabelEncoder() or sklearn.feature_extraction.FeatureHasher), but they all seem to require knowledge of the number of categories before hand (by having to specify a dictionary or fitting an encoder to the available data, etc...)

I thought about using some word embeddings like word2vec, but they return pretty large vectors, and all I need is an int, and I don't really care about semantic similarity etc...(i.e. using a word embedding is overkill).

Is there some sort of preprocessing utility from and ML library, or some publicly available string to int hash map that is stable that I can use?

Let's see. You want to be able to work with unspecified number of categories.
There are multiple ways to work with this.

Create a special category "other" and put all categories which are very rare into this category during processing. When you encounter a new category, put it into this "other" category. And you can use any common preprocessing - label encoding, one hot encoding and so on. This way you'll be able to make predictions for new categories. And when you have enough data for them, you can leave then as they are and refit you preprocessing and the model.
Target encoding. There are multiple ways to convert categories to float numbers. You train target encoding on the data which you have and can apply it to new categories (usually unknown values are assigned a global mean value).


Using Majority Class to Predict Minority Class Suppose I want to train a binary model in order to predict the probability of who will buy a personal loan and in the dataset only 5 percent of the examples are people who marked as bought a personal loan. So, in this scenario maybe I can leverage downsampling or upsampling to balance the dataset, but if my dataset isn't big enough there may left very few examples or may be upsampling isn't appropriate. Then suppose I decided to use whole dataset, I partitioned it to the training and test sets in order to predict the probability of who won't buy a personal loan. Considering it's a binary model does it make sense to subtract this model's output probabilities from 1 and predicting who will buy a personal loan by using this result?

Yes that's correct, but assuming that you follow the exact same methodology you will obtain exactly the same performance at the end, so there's no advantage.

Keep in mind that the problem with class imbalance is not that one class is harder to identify than the other, but rather that it's harder to properly separate the two classes.

[edit] It would be a different story when using one-class classification. I'm not sure if it makes sense in this case but maybe it could be something worth trying.


How do I interpret the given classification report? The given classification report was obtained from running a Random Forest binary classifier on the test data. There is huge class imbalance in the training data.
How do I interpret the given classification report showing very high values for one particular label?

            precision    recall  f1-score   support

      0       0.98      1.00      0.99     35050
      1       0.98      0.72      0.83      1982
  total       0.98      0.98      0.98     37032

Precision is the proportion of predictions of that class that are true. So 98% of the predictions for each of your classes are actually of the predicted class, and 2% are actually of the opposite class. Recall is the proportion of the true positives that are identified as such. This means that your model is correctly identifying 100% of the class 0s, but only 72% of the class 1s.

F1-Score is a kind of average of the two; it's an attempt to provide a unified figure of the model's performance, but personally I consider it less useful than the separate figures. It's calculated via the formula 2 x ((precision x recall) / (precision + recall)).

Wikipedia's pages on these metrics are comprehensive:

https://en.wikipedia.org/wiki/Precision_and_recall
https://en.wikipedia.org/wiki/F1_score


Reticulate vs Python I have gotten very used to coding in R and especially in RStudio. I like that interface. Nonetheless, I have some work that I ought to do in Python. I know that I can run Python code in RStudio if I use the "reticulate" package, and I've written some code that has switched between the two. As far as I can tell, anything I could run in Python from the command line or in Jupyter could be done in RStudio with reticulate.

Should I expect to have longer run times using Python commands in R, than I would if I ran Python regularly (whatever "regularly" means)?

I actually tested this recently on a random forest fitting using two approaches:

Using Jupyter notebooks to fit my model via python with data that had been tidied in the same notebook also using python.

vs:

Using reticulate in a rstudio notebook to fit the same model in python but using a converted dataset I had tidied in r in the same notebook.

I made the following observations:

Native python is faster, in my benchmark it fitted the model ~1 min faster (9 vs 10 mins) on my 16GB Ram laptop.
Additionally you lose some time converting objects between the two languages, especially if we talk about bigger datasets.

Conclusion

In most cases doing r stuff first, saving objects to CSV and then opening jupyter or python command lines to finish a job is not necessary thanks to reticulate.

If you do a repeated or time-critical job try to use python natively but otherwise reticulate is fine especially, if like me you save a lot of time doing prep work in r.


How to deal with position bias in search? In search, position of the search result affects the click-through rate a great deal. How do people usually deal with this ? In practice how to remove such bias to create unbiased training data for training learning to rank model ?

What are your output (prediction) and inputs?

The bias in itself is very important in predicting the click-through rate, but definitely less useful if you want to assess your ad copy/ page title in relation to the click-through rate.

One potential way to do that is to model the impact of the position of the ad / (or page, if it's for SEO), and try to normalize the data either by including the impact factor in your model or adjusting the result by the impact factor.

I would suspect it would be a log function, as the number of clicks tend to drop significantly after the top 1/2 of the first page.


KNN Variance using a high value of K and cross-validation it has come to my understanding, that a value of K=1, gives a high variance because we are only using only one data point, hence we are very likely to model the noise in that training example.

Bias: It will take the value of point 3 as it’s the closest one. It looks much better here (I know not the best example but you can consider another example such as house price and axis to be locality and price). It makes more sense that the point will be similar to the closest points compare to the point far away.

Variance: Let’s say point 11 was at age 32, then the closest point would have been 9 and hence the new predicted value would have been much different than the current one. Hence, it has a high variance.

They used a different data point from the test data, but why is point 11, at age 32?, what makes for the two data points to be related, because surely if I use age 50, I do not expect
a prediction close to the prediction at age 30, do they assume they have the same height, if then, why are we using our dependent variable to predict our dependent variable
Can someone provide me answers

Much appreciated

It's just an example in the article trying to make the point that a small change with regards to the age of $p_{11}$ from $30$ to $32$ has relatively large impact on the prediction since with $k=1$ for

$age_{p_{11}}=30$, the model would predict $heigth_{p_{11}}\leftarrow heigth_{p_{3}}$ and for
$age_{p_{11}}=32$, the model would predict $heigth_{p_{11}}\leftarrow heigth_{p_{9}}$.

Moving $p_{11}$ to $age = 50$ would just not serve as a good example for the case they are making because it's not surprising that a model predicts different height values for $age=30$ and $age=50$.
To phrase it differently, the relation between $p_{11}$ being at $age=30$ or $age=32$ is that it's only a small difference in terms of age.


Actor Network Target Value in A2C Reinforcement Learning In DQN, we use;
$Target = r+\gamma v(s')$  equation to train (fit) our network. It is easy to understand since we use the $Target$ value as the dependent variable like we do in supervised learning. I.e. we can use codes in python to train the model like,
model.fit(state,target, verbose = 0)

where $r$ and $v(s')$ can be found by model prediction.
When it comes to A2C network, things becomes more complicated. Now we have got two networks. Actor and Ctitic. It is said, the Critic network is not different from the what it is done in DQN. Only difference is now we have got only one output neuron in the network. So, similarly, we calculate the $Target = r+\gamma v(s')$ after acting via sampled action from $\pi(a|s)$ distribution. And we train the model with model.fit(state,target, verbose = 0) in python as well.
However, in Actor case it is so confusing. Now, we have another neural network which takes states as input and gives probabilities as the output by using softmax activation function. It is cool. But the point I stuck is,  the dependent variable to adjust the Actor network. So in python,
model2.fit(state,?,verbose=0)

What is the ? value to "supervise" the network to adjust the weights? and WHY?
In several resources I found the Advantage value which is nothing but the $Target - V(s)$.
And also there is something called as the actor loss which is calculated by,

Why?
Thanks in advance!

The target will still be a form of a return estimation ($V(s_t)$, $Q(a_t,s_t)$, Advantage, n-step reward, etc). For example in your case the $Q_w$ that Critic estimated.
You will need to review a bit Policy Gradient methods in this order: PG Theorem, REINFORCE (Actor only method) then AC (Actor-Critic) and then A2C. I will give you a conceptual explanation, abstracted from the math behind. The general form of Policy Gradient is:

likelihood of action given state multiplied by a form of return: $log\pi_{\mathbf{\theta}}(a_t|s_t)\cdot R_t$

What this tells us is: maximize the likelihood of a specific action multiplied by the return. In classification we do know the correct class (action) but here this is not the case. The only learning signal comes from the reward. Therefore multiplying the likelihood of an action with a form of return for selecting that action, that will decrease/increase the probability of selecting that action again given at the current state. The PG theorem states that this is the direction to update the weights $\theta$ in order to maximize return.
In A2C, you can have various implementations: 2 separate networks or 1 network with 2 separate heads. Actor is responsible for learning the distribution of actions that maximize the return given state. Critic is responsible for estimating the return from current state (and action). Thus the loss function in A2C is usually the Policy Gradient loss plus the Mean Squared Error between expected return and observed return (plus entropy for exploration). Please refer to the original A2C paper for the equations. As you can see we have 2 main loss functions (one for each network (head) ).
The return, in any form, affects training in 2 ways:

Learning signal for the actor: Multiplies the likelihood of actions and by doing so increases/decreases the probability for selecting these actions. This will change the parameters of the policy in order for the policy to favor rewarding actions.
Learning signal for the critic: MSE target.

As you can see in a loose sense you are doing again supervised learning for both Actor and Critic but with classification loss for Actor and regression loss for Critic. The why this occurs comes from the Policy Gradient theorem that shows that in order to maximize the return following a parametrized policy we need to update the parameters towards the direction that maximize the PG loss function (for the Actor).
Please note that for various reasons using the return might not be that of a good idea. That's why you will see the PG loss in various forms ( e.g. policy likelihood multiplied by the advantage $A(t)$ or $V(s_t)-R_t$, $Q(a_t,s_t)$ etc)


shape of theano tensor variable out of keras Conv2D Being new to theano, pls bear with me. I thought the shape of the tensor variable is already well defined out of the Conv2D layer since the input is specified, as follow,

from keras.layers import Input, Convolution2D
import theano
input_img = Input(shape=(1, 28, 28))
x = Convolution2D(16, 3, 3, activation='relu', border_mode='same')   (input_img)
print type(x)
print theano.tensor.shape(x)

But the output is,

<class 'theano.tensor.var.TensorVariable'>
Shape.0

Since I'm taking the default stride of 1, and same border mode here means that padding is added so the output is the same as input. Using this information I could calculate by hand what the output shape should be.

Did I miss something here? The question is how to get the shape of the output of a convolution layer?

You can't get the shape of a theano tensor, because it is not fixed. The output of the convolutional layer is just a symbolic variable and its shape depends on whatever you put into the layer as input.

You can get the shape of the output for a specific input by making a theano function for the output of the layer, and feeding a numpy array through the function:

import numpy as np
input = np.ones(28*28).reshape(1, 1, 28, 28).astype('float32')
fn = theano.function([input_img], x)

print fn(input).shape
>>> (1, 16, 28, 28)


How do I write a SQL (or dwSQL) query across these two different datasets? I'm having a little fun and I'd love to know how to write a SQL or dwSQL query across these two datasets of Trumps tweets and Facebook posts. Here are the datasets:

Trump Facebook Posts
https://data.world/dave/trumps-facebook-statuses

Trump Tweets
https://data.world/dave/trump-tweets

Ideally I would like to be able to count the distinct instances that a word or phrase (like 'crooked' or 'massive' or 'the best' appeared across both datasets.

The table name for the Facebook Post dataset is TrumpFacebook and for the Trump Tweets is trump_tweets.

Have you tried something like this in SAS?

PROC SQL;
 CREATE TABLE TRUMP_FB_TWEETS AS
 SELECT *
 FROM TRUMP_FB
 UNION CORRESPONDING ALL
 SELECT *
 FROM TRUMP_TWEETS;
QUIT;

Source:http://www2.sas.com/proceedings/sugi25/25/cc/25p109.pdf


GANs and grayscale imagery colorization I am currently studying colorization of grayscale satellite imagery as part of my Master's internship. After looking for various machine learning techniques, I quickly decided to go for deep learning, as the colorization process can be completely automated and gives good results. I trained a wide variety of models and architectures (CNNs, Pix2Pix, DCGANs and WGANs with UNet or residual architectures, etc.) and settled with convolutional cGANs (condition: grayscal image ; output : colored image), but end up facing the same problem every time. Indeed, when training my network, the output is quite correct but always gives me grayish roofs.

I think it has something to do with the distribution of pixels values, as roofs are usually orange or black in my geographic area. Many articles state the fact that GANs are prone to mode collapse. In this case, there are two modes (orange and black) which probably results in a bad local equilibrium.

I tried many different techniques in order to improve training and/or avoid mode collapse :

L1 loss regularization for the generator
Label smoothing
Noise taken from a normal distribution and concatenated to every layer in the generator/discriminator
Dropout for some layers in the generator
Gradient penalty
Learning procedure inspired by DRAGANs and BEGANs

So far, gradient penalty has given me the best results, despite still having these grayish roofs. I know there are other ways of improving the training procedure (e.g. batch discrimination), but I do not have the pre-requisites in mathematics or computer science nor the time for implementing such techniques.

I was wondering if someone had some ideas for getting better results, or if someone also tried colorizing satellite imagery and ended up having such issues.

Thanks for reading this long message.

Have a nice day !

Colorfulness, specifically managing overly gray output, was touched on in the pix2pix paper, down in Section 4.2. It might be worthwhile to try out their formulation of a conditional GAN, as their results indicate that their objective function was useful in generating a more diverse set of colors.


The memorisation capacity of an LSTM (real numbers) My question is the following:
It is known that a LSTM can remember sequences of one-hot encodings which represent integers (i.e. output $x_1, ... x_n$ after receiving $x_1, ... x_n$ as inputs, $x_k \in \{0,1\}^m$, where $m$ is the number of distinct integers).
Is it theoretically possible for the LSTM to learn to remember sequences of real numbers instead (that can be expressed in a finite number of bits), i.e. if $x_t \in \mathbb{R}$ instead.
The task I'm concerned with is much simpler - I just want to output the first input $x_1$ after reading the entire sequence $x_1, ... x_n$. I have done some small experiments with $x_t \in \mathbb{R}$, using square loss. There seems to be some level of success, however the results aren't very interpretable (when I look at the weights). Can anyone shed some light on this, specifically:

Does such a configuration of weights exist? (the questions following this quesetion suggests that it does exist)
If so, what are they and if not, why not?

The LSTM model is specified by:
The input, forget and outputs gates:
$$f_t = \sigma(W_f [h_{t-1}, x_t] + b_f)$$
$$i_t = \sigma(W_i [h_{t-1}, x_t] + b_i)$$
$$o_t = \sigma(W_o [h_{t-1}, x_t] + b_o)$$
And the internal state $c_t$ and hidden state $h_t$:
$$c_t = f_t * c_{t-1} +  i_t * \text{tanh}(W_c[h_{t-1}, x_t] + b_c) $$
$$h_t = o_t * \text{tanh}(c_t)$$
As requested, this is the assignment question:

Memory Task Description
Consider the following task: Given an input sequence of $n$ numbers, we would like a system which, after reading this sequence will return the first number in the sequence. That is given an input sequence: $(x_1, x_2, \cdots x_n)$, $x_i \in \mathbb{R}$ the system has to return, at time $t=n$ after 'reading' the last input $x_n$, the first input $x_1$.

Given the task above, consider the above recurrent models (RNNs/LSTMs/GRUs). Which of these arhitectures can (theoretically) perfom the task above? In answering this questions, please consider a simple one-layer model of RNNs/GRU/LSTM with a one-dimensional input $x_t$, a $32$-dim hidden and output layer, followed by a transformation to a one-dimensional final output which should predict $x_0$.
Whenever the answer is positive, give the gates' activations and weigths that will produce the desired behaviour. Whenever the answer is no, prove that there exists no such parameters that an arbitrary input sequence can be transformed to produce the first symbol read.

Welcome to the site! If you're referring to a series of numbers, like what you would get during tokenization/NLP then, yes, LSTM can certainly handle that without many issues. If you are talking about a much larger range than that, then you might want to consider a scenario where you scale your inputs instead.


How to add column name I have a dataset in which some columns have header but the other ones don't have. Therefore, I want to add column name only to those who doesn't have column name. I know how to add column names using header but  I would like to know how to do this using index so that I can add column names only to the empty ones. Any suggestions will be appreciated.

Loop through df.columns checking whether there is a column name or if it is empty:

import pandas as pd

# create df with two empty columns
df = pd.DataFrame({'a': [1,2], 'b': [1,2], 'c': [1,2], 'd':[1,2]})
df.columns = ['', 'b', '', 'd']

df with empty columns:

       b     d
 0  1  1  1  1
 1  2  2  2  2

# new of list columns names must be the same length as number of empty columns
empty_col_names = ['x', 'y']
new_col_names = []
count = 0

for item in df.columns:
    if item == '':
        new_col_names.append(empty_col_names[count])
        count += 1
    else:
        new_col_names.append(item)

df.columns = new_col_names

df with filled columns:

    x  b  y  d
 0  1  1  1  1
 1  2  2  2  2


In the Keras Tokenizer class, what exactly does word_index signify? I'm trying to really understand Tokenizing and Vectorizing text in machine learning, and am looking really hard into the Keras Tokenizer class. I get the mechanics of how it's used, but I'd like to really know more about it. So for example, there's this common usage:

tokens = Tokenizer(num_words=SOME_NUMBER)
tokens.fit_on_texts(texts)

tokens returns a word_index, which maps words to some number.
Are the words all words in texts, or are they maxed at SOME_NUMBER?
And are the dict values for word_index the frequency of each word, or just the order of the word?

Are the words all words in texts, or are they maxed at SOME_NUMBER?

Yes, it will be maxed at the most common SOME_NUMBER-1 words.

  And are the dict values for word_index the frequency of each word, or just the order of the word?

It is just the index of the word in the dictionary.

You can read more here in the documentation.


VAR model ValueError: x already contains a constant I'm using VAR model for multivariate time series. The structure is that although each variable is a linear function of past lags of itself and past lags of the other variables, one and/or two of the variables MAY NOT alter within the period under investigation. Out of 10 variables.

Below is a similar dataframe to the one I'm working on. The actual dataset has 190 rows.

x0 = [0,0,0,0,0]
x1 = [0.011866,0.013380,0.015357,0.024451,0.030889]
x2 = [0,2,2,3,3]
x3 = [1,1,2,3,3]
x4 = [0,0,0,0,0]

T = ['2000-01-01', '2000-01-02', '2000-01-03', '2000-01-04', '2000-01-05']
TDT = pd.to_datetime(T)
df = pd.DataFrame({'X0': x0, 'X1': x1, 'X2': x2, 'X3': x3, 'X4': x4})
df.index = TDT
df

model = VAR(df)
result = model.fit(1)

**ValueError: x already contains a constant**

Is there a way to fix this?

Your two columns X0 and X4 are constants, i.e. they contain a single value throughout.

The model will be trying to find a constant during its fit, so probably has a check that you're not including one.

Once you have your entire dataframe (e.g. called df), you can remove constant columns like the two above by using:

df_no_constants = df.loc[:, (df != df.iloc[0]).any()]

Then try putting that into your model as before.


How can I plot/display a dataset or an image distribution? I want to view a specific image or a dataset's distribution, and see if they are different.
Does simply writing something like :

# mydataset.shape = (50k,32,32,3)
plt.hist(mydataset.reshape(-1))

do the trick? or should I be doing something else?
for example doing so on cifar10 gave me this plot :

However, it doesn't look right, there are 50K training images, and  I don't know how to interpret this or even if it is the correct way of doing it in first place!

if I do :

#dataset shape is (50k, 3072)
plt.hist(mydataset.reshape(-1,32*32*3))
#and testset which is (10K,3072)
plt.hist(mytestset.reshape(-1,32*32*3))

This is what I get :

So this is very puzzling to me, and I have no idea how to go about this!
Since there are 10 bins, is it giving me the distribution of each class(cifar10 as 10 classes) ? if it is so why do I get the same shape when I try to get the distribution of a single image?
I mean when I try to do :

#image is of shape (1,3072)
plt.hist(mytestset[0])

this is what I get :

and interestingly, the histogram for the whole test set looks like this :

#dataset shape is (10k,32,32,3)
plt.hist(mytestset.reshape(-1,32*32*3))

Why do I get ten bins for the single image as well?
what does each axis mean then?
What should I be looking for when it comes to image/dataset distribution?
Is it the raw values for the whole dataset only?
or Is it the raw values for each class? or even each image?

I want to view a specific image or a dataset's distribution, and see
  if they are different. Does this do the trick?

It depends what you want to understand or learn about your data.

  what does each axis mean then?

In all of your plots, the x-axis ranges from 0-255, which is because in all your plots, you are creating histograms of the individual pixel values of your images. A single pixel consists of a vector (tuple) of three values: (red, green, blue). Each of those three colours can take a value 0 to 255 (usually an unsigned 8-bit integer)So by creating a histogram over your images, you are essentially counting how many time each of the possible values appear.

These plots can therefore tell you something about the average colour distribution. The peaks seem to be around 100 for each of your plots, so I guess there are a lot of mixed colours - colour where the RGB values are in the range 100 - 150. So less pixels are purely reg, green or blue e.g. (0, 0, 255) would be purely blue.

You can also compute/visualise the histogram of colours using the OpenCV library, which has great functions for doing exactly this kind of thing (and will run way faster than matplotlib's histogram method because OpenCV uses the C++ backend library). Have a look at this great walkthrough.

  Why do I get ten bins for the single image as well?

The reason all your histograms have 10 bins, is because you are not specifying a value for the bins argument to plt.hist, so the default value is taken from the basic configurations of matplotlib, which you can see by running this:

print(plt.rcParams["hist.bins"])    # will print 10 by default

The y-axes on those two plots do make sense; the dataset with 50k images has higher counts for each pixel value compared to the 10k dataset.

What should I be looking for when it comes to image/dataset distribution?

  Is it the raw values for the whole dataset only? or Is it the raw values for each class? or even each image?

Each plot is showing the distribution of raw values only, for whichever set of data you use. You used mtdataset and mytestset as input, so in each case you are only seeing the distribution for those images of course. There is no inclusion of the actual labels anywhere, so you are not breaking down the distributions into the target classes, for example. Just raw pixel values are considered.

  What should I be looking for when it comes to image/dataset
  distribution?

This is problem specific.

You could be looking to see that there is indeed a distribution of RGB values, as a sanity check that you don't have some really skewed set of colour images that e.g. are mostly black or white. You might compare the distributions of the training and test sets to one another, to ensure that they are similar - meaning the training set is indeed representative of the test set. If this were not the case, any model you might train on a specific task could be biased towards the training set and perform badly on the test set (it will not have seen images similar to the test set if the distributions are very different!)

For training neural networks with images, it is common to normalise the distribution of pixel values to the range [-1, +1], which helps smoother learning via smoother gradient updates.