How to do SVD and PCA with big data? I have a large set of data (about 8GB). I would like to use machine learning to analyze it. So, I think that I should use SVD then PCA to reduce the data dimensionality for efficiency. However, MATLAB and Octave cannot load such a large dataset.

What tools I can use to do SVD with such a large amount of data?

First of all, dimensionality reduction is used when you have many covariated dimensions and want to reduce problem size by rotating data points into new orthogonal basis and taking only axes with largest variance. With 8 variables (columns) your space is already low-dimensional, reducing number of variables further is unlikely to solve technical issues with memory size, but may affect dataset quality a lot. In your concrete case it's more promising to take a look at online learning methods. Roughly speaking, instead of working with the whole dataset, these methods take a little part of them (often referred to as "mini-batches") at a time and build a model incrementally. (I personally like to interpret word "online" as a reference to some infinitely long source of data from Internet like a Twitter feed, where you just can't load the whole dataset at once).

But what if you really wanted to apply dimensionality reduction technique like PCA to a dataset that doesn't fit into a memory? Normally a dataset is represented as a data matrix X of size n x m, where n is number of observations (rows) and m is a number of variables (columns). Typically problems with memory come from only one of these two numbers.

Too many observations (n >> m)

When you have too many observations, but the number of variables is from small to moderate, you can build the covariance matrix incrementally. Indeed, typical PCA consists of constructing a covariance matrix of size m x m and applying singular value decomposition to it. With m=1000 variables of type float64, a covariance matrix has size 1000*1000*8 ~ 8Mb, which easily fits into memory and may be used with SVD. So you need only to build the covariance matrix without loading entire dataset into memory - pretty tractable task.

Alternatively, you can select a small representative sample from your dataset and approximate the covariance matrix. This matrix will have all the same properties as normal, just a little bit less accurate.

Too many variables (n << m)

On another hand, sometimes, when you have too many variables, the covariance matrix itself will not fit into memory. E.g. if you work with 640x480 images, every observation has 640*480=307200 variables, which results in a 703Gb covariance matrix! That's definitely not what you would like to keep in memory of your computer, or even in memory of your cluster. So we need to reduce dimensions without building a covariance matrix at all.

My favourite method for doing it is Random Projection. In short, if you have dataset X of size n x m, you can multiply it by some sparse random matrix R of size m x k (with k << m) and obtain new matrix X' of a much smaller size n x k with approximately the same properties as the original one. Why does it work? Well, you should know that PCA aims to find set of orthogonal axes (principal components) and project your data onto first k of them. It turns out that sparse random vectors are nearly orthogonal and thus may also be used as a new basis.

And, of course, you don't have to multiply the whole dataset X by R - you can translate every observation x into the new basis separately or in mini-batches.

There's also somewhat similar algorithm called Random SVD. I don't have any real experience with it, but you can find example code with explanations here.

As a bottom line, here's a short check list for dimensionality reduction of big datasets:

If you have not that many dimensions (variables), simply use online learning algorithms.
If there are many observations, but a moderate number of variables (covariance matrix fits into memory), construct the matrix incrementally and use normal SVD.
If number of variables is too high, use incremental algorithms.


How to interpret a specific feature importance? Apologies for a very case specific question. I have a dataset of genes, with which I am using machine learning to predict if a gene causes a disease. One of the features I have is a beta value (which is the effect size of the gene's impact on the disease), and I'm not sure how best to interpret and use this feature.

I condense the beta values from the variant level to the gene level, so a gene is left with multiple beta values like this:

Gene         Beta
ACE      -0.7, 0.1 ,0.6
NOS      0.2, 0.4, 0.5
BRCA     -0.1 ,0.1, 0.2

Currently I am trying 2 options of selecting a single beta value per gene, one where I select the absolute value per gene (and ignore whether it was a previous negative value) and another where I select the absolute value and return the previous negative numbers back to being negative. I am trying this as for beta values a postive or negative direction indicates the size of the effect a gene has on the disease, so I would think it's important to retain the negative information (as I understand it).

However, I've been advised to use just the absolute values with not retaining negative status, and I'm not sure if there's a way for me to know if one option is better than the other from the machine learning perspective. I am also having a problem in either case where my model values this feature as much more important than any other feature in my dataset. For example gradient boosting gives this an importance of 0.01, the next most important feature is at 0.001.

So my question is, how best can I interpret a highly important feature like this? If it is much more important is it actually a bias and is it likely due to my own handling/preprocessing of the feature or is it acceptable that is it just very important? Would it be possible for me to set my model to re-weight the importance of this particular feature? I have a biology background so not sure what is the normal or least biased approach.

You can use one of 2 approaches:

The 1st is unsupervised:

Use PCA algorithm to extract the feature vectors best representing the dataset variance. The PCA algorithm extract new features which each of them is a linear combination out of the other features (independent from the label) when the 1st feature it extracts is most important feature and last one is the least important. Then you can retrieve the weights of each "Beta" value at the most important feature. Here is an example for that:
https://stackoverflow.com/a/34692511/6677037

Another approach is the supervised:

using the labels, which you should use carefully and not choose the features based on the test set.
With these methods you can see the most important features using "Xi Square" , "mutual information gain" etc. Then you remove the least important features.
here is the easiest way to do that:
https://hub.packtpub.com/4-ways-implement-feature-selection-python-machine-learning/

good luck.


How tensorflow understand accuracy and loss of training data? Code :-
models = keras.Sequential(
    [
        keras.Input(shape=(28*28)),
        layers.Dense(512,activation = 'relu'),
        layers.Dense(256,activation = 'relu'),
        layers.Dense(10),
    ]
)

models.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=keras.optimizers.Adam(lr=0.001),
    metrics = ["accuracy"]
)

models.fit(x_train,y_train,batch_size=32,epochs=5,verbose=2)

Output:-
Epoch 1/5
1875/1875 - 8s - loss: 0.1854 - accuracy: 0.9450
Epoch 2/5
1875/1875 - 11s - loss: 0.0790 - accuracy: 0.9760
Epoch 3/5
1875/1875 - 7s - loss: 0.0561 - accuracy: 0.9822
Epoch 4/5
1875/1875 - 6s - loss: 0.0423 - accuracy: 0.9865
Epoch 5/5
1875/1875 - 6s - loss: 0.0329 - accuracy: 0.9893

<tensorflow.python.keras.callbacks.History at 0x7efda0e109a0>

While learning TensorFlow, My first doubt was what is accuracy and loss? By google, I Learned:-
Loss value implies how poorly or well a model behaves after each iteration of optimization. An accuracy metric is used to measure the algorithm's performance in an interpretable way. The accuracy of a model is usually determined after the model parameters and is calculated in the form of a percentage.
But my doubt is how TF knows about accuracy as we pass only training data, not Test Data. To calculate accuracy I think test data needed. Is TF split train data into another train and test data?

You are still able to calculate metrics such as loss and accuracy on training data (or any data for that matter), however the important thing to keep in mind is that it is by definition training data. Therefore the metrics from the training data are not how you would expect the model to perform on new unseen data as the model has been training on these specific samples (and can possibly remember them).


Are there any projects to work on the Internet to fight against COVID? Are there any open source projects that a novice data analyst and mathematician can do, to fight against covid-19 epidemic? I mean, I know that the best I can do is to stay away from people and now I have a laptop and plenty of time to work on some project. But the thing is that I don't know what kind of computations would be useful to save people.

I have Ubuntu, LibreOffice, R, Python, and Sagemath installed on my computer.

You might want to have a look at this Kaggle initiative and this github repository with South Korea data


Any research on relationship between the dimensions of a (word2Vec) space and how the human mind constructs meaning (or reality) through language? Neuroscience is still trying to "find" how the mind (and language) somehow "works". Is there any theory linking a (low-dimensionality) embedding space (like word2Vec) to a mind (linguistic) model? Any Cognitive Linguistics theory?

Some initial steps where taken here:
Connecting concepts in the brain by mapping cortical representations of semantic relations

To represent words as vectors, we used a pretrained word2vec model21.
Briefly, this model was a shallow neural network trained to predict
the neighboring words of every word in the Google News dataset,
including about 100 billion words
(https://code.google.com/archive/p/word2vec/). After training, the
model was able to convert any English word to a vector embedded in a
300-dimensional semantic space (extracted through software package
Gensim56 in python). Note that the basis functions learned with
word2vec should not be interpreted individually, but collectively as a
space.
Applying the encoding model to the differential vector of a word pair
could effectively generate the cortical representation of the
corresponding word relation. With this notion, we used the encoding
model to predict the cortical representations of semantic relations.
For each class of semantic relation, we calculated the relation vector
of every word pair in that class, projected the relation vector onto
the cortex using the encoding model, and averaged the projected
patterns across word-pair samples in the class.


Correlation threshold for Neural Network features selection I'm trying to do a correlation analysis between inputs and outputs inspecting the data in order to understand which input variables to include. What could be a threshold in the correlation value to consider a variable eligible to be an input for my Neural Network?

Given non-linearity of neural networks, I believe correlation analysis isn't a good way to estimate importance of variables. For example, imagine that you have 2 input variables - x1 and x2 - and following conditions hold:

cor(x2, y) = 1 if x1 = 1
cor(x2, y) = 0 otherwise
x1 = 1 in 10% of cases

That is, x2 is a very good predictor for y, but only given that x1 = 1, which is the case only in 10% of data. Taking into account correlations of x1 and x2 separately won't expose this dependency, and you will most likely drop out both variables.

There are other ways to perform feature selection, however. Simplest one is to train your model with all possible sets of variables and check the best subset. This is pretty inefficient with many variables, though, so many ways to improve it exist. For a good introduction in best subset selection see chapter 6.1 of Introduction to Statistical Learning.


Applying Standardization OLS estimator I have basic understanding of how to perform linear regression with sklearn and statsmodels. There are several questions that I would like to ask regarding Linear Regression (OLS estimator) :

Is standardization always necessary for OLS estimator? I am still learning a bit of theoretical aspect of regression analysis. Standardization is mentioned in my data science class but almost None throughout my study in regression analysis. If no, when do I apply standardization (also do I need to standardize the predicted variable)?
I tried comparing both, in terms of evaluation metrics(mse) the results are relatively similar, but in terms of coefficient it is very different (due to different scales of predictors). If this is the case how to compare the significance of a predictor?
I tried comparing the result with statsmodel OLS on unstandardized data. Coefficient result is different by huge margin and on top of that mean squared error is worse with statsmodel than with sklearn model. However, sklearn model is less informative in terms of statistical information, for example if I want to remove a variable based on p-value from regression result. Is this supposed to happen(different result)?

1) No. You can standardize data, but you don‘t need to. Often you don‘t want this because you like to see the effect of some $x$ on $y$ in „real units“ (not standard deviations).

2) In case you standardize $X$, the coefficients must of course change as the X‘s are now no longer real units but standard deviations. The advantage here is that you can compare coefficients in terms of the magnitude or importance (larger absolute value, more important). This is sometimes called „beta coefficients“ or „standardized coefficients".

The model fit may be a little better with standardization, but the main restriction in OLS is the linear model structure. You can relax this assumption by using "Generalized Additive Models" GAM or introducing polynomials.

3) I don‘t understand your question here and there is no code, but it is likely that you did something wrong. OLS like $y=\beta X + u$ simply requires solving

$$ \hat{\beta}=(X'X)^{-1}X'y.$$

Never drop variables due to high p-value (it summarizes confidence bands you could say). A variable may still be useful and important for the model even with high p-value. If you want to do feature/variable selection, use "shrinkage" via Lasso.


What is the meaning of the sparsity parameter Sparse methods such as LASSO contains a parameter $\lambda$ which is associated with the minimization of the $l_1$ norm. Higher the value of $\lambda$ ($>0$) it means that more coefficients will be shrunk to zero. What is unclear to me is that how does this method decide which coefficients to shrink to zero?

If $\lambda =  0.5$ then does it mean that those coefficients whose values are less than or equal to 0.5 will become zero? So in other words, whatever be the value of $\lambda$, the coefficients whose values fall within $\lambda$ will be turned off/become zero? OR is there some other meaning to the value of $\lambda$?
Can $\lambda$ be negative?

When we implement penalized regression models we are saying that we are going to add a penalty to the sum of the squared errors.
Recall that the sum of squared errors is the following and that we are trying to minimize this value with Least Squares Regression:
$$SSE = \sum_{i=1}^{n}(y_i-\hat{y_i})^2$$
When the model overfits or there is collinearity present, the estimates for the coefficients for our least squares model may be higher than they should be.
How do we fix this? We use regularization. What this means is that we add a penalty to the sum of squared errors thereby limiting how large the parameter estimates can get.
For Ridge Regression this looks like this:
$$SSE_{L2 norm} = \sum_{i=1}^{n}(y_i-\hat{y_i})^2 + \lambda \sum_{j=1}^{P}\beta_j^2$$
Notice what is different with this model. Here, we add a L2 regularization penalty to the end of the SSE. What this does is add the multiplication of $\lambda$ by the square of the parameter estimations as a penalty to the SSE. This limits how large the parameter estimates can get. As you increase the "shrinkage parameter" $\lambda$ the parameter estimates are shrunk more towards zero. What is important to note with Ridge Regression, is that this model shrinks the values towards zero, but not to zero.
You may also use the LASSO Regression technique as shown below:
$$SSE_{L1 norm} = \sum_{i=1}^{n}(y_i-\hat{y_i})^2 + \lambda \sum_{j=1}^{P} \lvert{\beta_j^2}\rvert$$
Notice here that the change by adding the L1 penalty is very similar. However, this difference here is that we are now penalizing the absolute value of the coefficients. This allows shrinkage to zero and can be considered a form of feature selection.
In either case both methods penalize model complexity and are parsimonious!
Two things to note:

In answer to your question, no, $\lambda$ cannot be negative. Why? This would make no sense. $\lambda$ is multiplied by either the L2 or L1 norm to add a penalty to the SSE. If instead, you had a negative $\lambda$ you would actually be rewarding the model's complexity not penalizing it.

When you have a value of $\lambda = 0$ you have no penalty and just have regular least squares regression!


ANN Variable Correlation I am using an ANN for predicting high value customers based on their first 0, 5, 10 days.

I've made 3 variables: PurchasesOnDay0, PurchasesOnDay0to5, PurchasesOnDay0to10.

PurchasesOnDay0to5 will include PurchasesOnDay0 and PurchasesOnDay0to10 will include PurchasesOnDay0to10 and PurchasesOnDay0to5.

My question is, will the ANN pick up the correlation and adjust accordingly or will this be a problem when creating my model?

Generically speaking a neural network can automatically learn the needed feature transformation for a model to perform well.  What tends to matter more is how the features relate to your output.

If you're ever bored, you can run your original matrix, a transformed matrix that has been de-correlated via a principle axis transformation, and the original matrix combined with correlation information.  I suspect performance using the same architecture will be relatively similar.


An exhaustive, representative test database in phrase search algorithm For a phrase searching algorithm, imagine the goal is to search for a name phrase and return matched results based on a pre-defined threshold. For example, searching for "Jon Smith" could return "Jon Smith", "Jonathan Smith", "Jonathan David Smith", "Jonathan Smith-Mikel", "Jonathan 'Smith' Mikel" etc.

The plan is to manually choose N test cases and put them in a benchmark database. I have concern about this plan because the test cases is likely to be not exhaustive. I know there are pretty mature search engines there, so is there a test database which covers all possible cases, such as different name combinations, punctuations, symbols, etc. such that we can use this as our benchmarks instead of guessing?

For example, this test database should contain all cases for "Jon Smith", as well as connectors such as hyphen, apostrophe, and so on.

Based on the example I assume that the target is persons names.

Let's be clear, there's no such thing as an exhaustive dataset containing all possible persons names in the world. Also a crucial part of the question is: in What kind of names? In which language? Persons names in English are pretty different from names in Chinese for instance. And there is also the difficult question of transliteration of proper names.

That being said, a few resources exist. They would usually be found by searching resources for "personal names", "record linkage", "named entities matching/coresolution". The following ones probably don't cover all the requirements but it's a start:

Febrl (see also here)
Found This paper which presents a large resource and this corresponding resource description but couldn't find the data.
The EMM news explorer has an interesting database of named entities including persons names with all the spelling variants/transliterations.


How to create more dimensions for data mining? I want to analyze product sales data with WEKA. My data have attributes: sell quantity and sales price. I want to add more attributes, such as mean and standard deviation.

What formula I should use to create more dimensions?

In general, this is called feature engineering. Expansion of dataset attributes via attribute transformations is called basis function expansion. These are often nonlinear functions of the original attributes. Popular nonlinear functions are polynomials.


Correlation between categorical variables based on the target distribution Let $X$ be a category with very high cardinality and $Y$ be my target. when I look at $X$ distribution to $Y$ I see that some of the levels are very similar to each other . I would like to find a way to combine them (lets say that $X_1$ and $X_3$ are very similar in the frequency of the there $Y$ distribution) where $X$ in ($x_1,...,x_n$) and y in ($y_1,...,y_n$)

What is the best way to find all this $X$ subgroups that have similar $Y$ distribution? the reason I'm doing this is because I know that lots of my $X$s are the same but someone labeled them as different.
I have started with doing spearman corr matrix over the frequency table for each $X$ and its $Y$s distribution , but I'm not sure it is the right way to handle it and it gives me some bad results

Your answer is $\chi^2$ (chi-squared) test of independence.

First we have to calculate the expected value of the two nominal variables.  We can calculate the expected value of the two nominal variables by using this formula:

$E_{i,j} = \frac{\sum_{k=1}^cO_{ik}*\sum_{k=1}^rO_{kj}}{N}$

Where,

$E_{ij}$ = Expected value of the cell $i$, $j$

$\sum_{j=1}^cO_{ij}$ = Sum of the ith column

$\sum_{k=1}^rO_{kj}$ = Sum of the kth row

$N$ = total number

After calculating the expected value, we will apply the following formula to calculate the value of the Chi-Square test of Independence:

$\chi^2 = \sum\sum\frac{(O_{ij}-E_{ij})^2}{E_{ij}} $: Chi-Square test of Independence

$O_{ij}$ = Observed value of two nominal variables

$E_{ij}$ = Expected value of two nominal variables

Degree of freedom is calculated by using the following formula:
$df = (r-1)(c-1)$

Where,
DF = Degrees of freedom
r = number of rows
c = number of columns

Hypothesis:

Null hypothesis: Assumes that there is no association between the two variables.

Alternative hypothesis: Assumes that there is an association between the two variables.

Hypothesis testing: Hypothesis testing for the chi-square test of independence as it is for other tests like ANOVA, where a test statistic is computed and compared to a critical value.  The critical value for the chi-square statistic is determined by the level of significance (typically .05) and the degrees of freedom.  The degrees of freedom for the chi-square are calculated using the following formula: df = (r-1)(c-1) where r is the number of rows and c is the number of columns. If the observed chi-square test statistic is greater than the critical value, the null hypothesis can be rejected.


Keras -- Transfer learning -- changing Input tensor shape This post seems to indicate that what I want to accomplish is not possible. However, I'm not convinced of this -- given what I've already done, I don't see why what I want to do can not be achieved...

I have two image datasets where one has images of shape (480, 720, 3) while the other has images of shape (540, 960, 3).

I initialized a model using the following code:

input = Input(shape=(480, 720, 3), name='image_input')

initial_model = VGG16(weights='imagenet', include_top=False)

for layer in initial_model.layers:
    layer.trainable = False

x = Flatten()(initial_model(input))
x = Dense(1000, activation='relu')(x)
x = BatchNormalization()(x)
x = Dropout(0.5)(x)
x = Dense(1000, activation='relu')(x)
x = BatchNormalization()(x)
x = Dropout(0.5)(x)
x = Dense(14, activation='linear')(x)

model = Model(inputs=input, outputs=x)
model.compile(loss='mse', optimizer='adam', metrics=['mae'])

Now that I've trained this model on the former dataset, I'd like to pop the input tensor layer off and prepend the model with a new input tensor with a shape that matches the image dimensions of the latter dataset.

model = load_model('path/to/my/trained/model.h5')
old_input = model.pop(0)
new_input = Input(shape=(540, 960, 3), name='image_input')
x = model(new_input)
m = Model(inputs=new_input, outputs=x)
m.save('transfer_model.h5')

which yields this error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 2506, in save
    save_model(self, filepath, overwrite, include_optimizer)
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/models.py", line 106, in save_model
    'config': model.get_config()
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 2322, in get_config
    layer_config = layer.get_config()
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 2370, in get_config
    new_node_index = node_conversion_map[node_key]
KeyError: u'image_input_ib-0'

In the post that I linked, maz states that there is a dimension mismatch that prevents changing the input layer of a model -- if this was the case, then how is that I put a (480, 720, 3) input layer in front of the VGG16 model which expects (224, 224, 3) images?

I think a more likely issue is that my former model's output is expecting something different than what I'm giving it based on what fchollet is saying in this post. I'm syntactically confused, but I believe the whole x = Layer()(x) segment is constructing the layer piece by piece from input->output and simply throwing a different input in front is breaking it.

I really have no idea though...

Can somebody please enlighten me how to accomplish what I'm trying to do or, if it's not possible, explain to me why not?

You can do this by creating a new VGG16 model instance with the new input shape new_shape and copying over all the layer weights. The code is roughly

new_model = VGG16(weights=None, input_shape=new_shape, include_top=False)
for new_layer, layer in zip(new_model.layers[1:], model.layers[1:]):
    new_layer.set_weights(layer.get_weights())


How to calculate y_score for ROC AUC? how can I calculate the y_score for a roc_auc_score?

I have a classifier, for classes {0,1}, say RandomForestClassifier.

Then, when I apply it to my test data, I will get a list of {0,1}

But roc_auc_score expects y_true and y_score.

As dummy as it might look, after fitting the model, I was making the following:

def find_prob(df):
    def prob(token):
        # the mean is the probability of label {1}
        return df[df.token == token].predicted.mean()
    df['predicted_score'] = df.token.map(prob)
    return df

X_val['predicted'] = model.predict(X_val).astype(int)
X_val = find_prob(tokenize(X_val))

roc_auc_score(y_val, X_val.predicted_score)

where tokenize is only a function which concatenate all the values in a row to a "unique" token string, so that I would be able of finding all instances of a token and check how many were labeled as {0} and how many were labeled as {1}.

Is this the correct approach? Thank you very much.

y_score can either be probability estimates of the positive
    class, confidence values, or non-thresholded measure of decisions

y_score = model.predict_proba(x)[:,1]

AUC = roc_auc_score(y, y_score) # Above 0.5 is good


MLP Parameter tuning - gridsearchCV cannot fit? I'm making an MLP classifier for binomial classification from 145 features.

I want to get the best parameters on my MLP classifier to get a better prediction so I followed the answer to this question, which is to use gridsearchCV from sklearn. However, when I get to

clf.fit(DEAP_x_train, DEAP_y_train)

I get the ff error:

TypeError: '<=' not supported between instances of 'str' and 'int'

I checked my train data and both X and y have already been changed to become float data type, so I don't understand why I keep getting this response.

For reference, I have 145 features to predict a binary response, and this is my parameter_space:

parameter_space = {
    'hidden_layer_sizes': [(368,), (555,), (100,)],
    'activation': ['identity', 'logistic', 'relu'],
    'solver': ['sgd', 'adam'],
    'alpha': [0.0001, 0.05],
    'learning_rate': ['constant','adaptive'],
    'max_iter': ['200', '1000', '5000', '10000']
}

I've tried to search for other cases online that this error response is shown when trying to fit, but I haven't come across any.

Would appreciate any response. Thanks!


Difference between RMSProp and Momentum? Can someone please tell me the clear difference between the approaches of RMSProp and Gradient Descent with Momentum ? Both try to achieve the same effect . One of the blogs that I read states the difference as : "RMSProp and Momentum take contrasting approaches . While momentum accelerates our search in direction of minima, RMSProp impedes our search in direction of oscillations. "
I don't get this statement . Can someone elaborate on the difference between the two ?

Optimizers evolved with small Fix/Improvement on the previous one. So, if you will read in sequence, you will have a better understanding. In this context, RMSProp was a fix on Adagrad and it was an improvement on Momentum.
Let's see this Loss surface which is like a Valley (Imagine a River)
$\hspace{2cm}$
$\hspace{5cm}$Image source - http://d2l.ai/
Momentum -
Let's start from the red-circled point. We have a very large Gradient in X2 direction and very little in X1 and global minima is towards X1.
In Momentum, we accumulated the resultant Gradient which will obviously point more towards X2.
As a result, we will move very fast towards the other side of the river and very little towards X1. As we cross the river and start moving up, counter Gradient of X2 will start minimizing the Aggregate. Remember, it's leaky aggregation i.e. recent ones have more say. At a point, it will stop and reverse.
In the whole process, we had a little movement in X1 and a lot of oscillations in X2
 This was one of the points of the Author
What AdaGrad did -
 - Manage Gradient for each coordinate separately
 - Added a scaling factor in the denominator which will act as a brake. This scaling is based on the square of past Gradients.
Now X2 will have a large brake, so it will not move so fast with momentum to cross the river. Since X1 is having a very small Gradient, its scaling will be positive(if < 1) Or almost constant(if ~ 1). So, movement in X1 will be same Or even faster.
That's why the author said: "RMSProp impedes our search in direction of oscillations"
Problem with Adagrad was that it aggregates all the past Gradients for the scaling factor which causes the brake to become larger for any case after a good number of iterations even if it has not reached Global optimum.
 Let's say, if Gradient is small 0.5 then also it will start dividing by 2.5 after 10 iterations. if it is large e.g. 10, then it will start dividing by 1000 after 10 iterations. Even this large gradient will become small in the subsequent iteration.
 What RMSProp changed - It made the aggregation leaky i.e. recent one will be considered more (Just like Momentum does for Gradient). With this change, the aggregation will be almost constant or at least not die as fast as in Adagrad.
$\hspace{2cm}$
$\hspace{5cm}$Image source - http://d2l.ai/


Regression and Neural networks I'm trying to restore this function:

$$ F(x) = x*sin(\alpha x)+b; \space\space \alpha,b \in (-20,20) $$

My NN model(with Keras) is:

1 layer: GRU, 9 neurons, selu activation
2 layer: GRU, 3 neurons, selu activation
1 layer: GRU, 7 neurons, selu activation
1 layer: Dense, 1 neuron, linear activation
kernel init he_normal for all layers

For the training dataset I've generated function values $f(x), x \in -5..5$ (e.g. linspace(-5,5,500) and random integer values for $\alpha$ and $b$, 250 times.

Then I've selected $f[i-1]$ and $f[i]$ (previous steps) and for this "xs" the output is $f[i+1]$. So the training dataset is like:

X:
   first row: previous previous $f$, $\alpha$, $b$
   second row: previous $f$, $\alpha$, $b$

Y:
   current $f$ value

After training over 200 epochs, mae on validation data was 0.1851.

Now if I try to predict new data it seems normal, but when I try to predict new values using previously predicted by model points it falls down and it doesn't looks like sine function at all.

What am I doing wrong?

The problem is that your error is accumulating and diverging. In other words, a small error in the first prediction is leading to a larger error in your second prediction, which is leading to an even larger error in your third prediction, and so on.

With that said, this LSTM example seems to do well predicting multiple future values for a sine wave.


How to use Variational Autoencoder's μ and σ with user-generated z? My understanding of VAE is that unlike Autoencoders, it does not directly give you a discrete encoding (latent code vectors n-dim) instead, it gives you both mu and sigma (n- dim mean vectors and n-dim standard deviation vectors). Then you have epsilon which you use to sample from a normal distribution with mu and sigma to create z. When combining mu, sigma and epsilon, you get z which is the one decoded by the VAE's decoder. z is basically the main encoding.

Say my z, mu, sigma are of n-dimension like 10 (10-dim z, mu, sigma). I enable the user to have a free picking/giving me numbers 10 vectors [-a, a], say a = 5. So the user is free to pick 10 vectors between -5, 5.

This becomes my z that is decoded by my decoder to generate a new image.

[Main problem]

My VAE is trained on a dataset of apparel. Now, if I run my VAE's encoder on each of the data in the dataset, I'd get a mu and sigma for each (not sure if this is still correct).

Using the z given by the user, how do I find the most similar from the dataset using VAE's encoding of only mu and sigma?

My thinking is to generate z using mu and sigma generated by VAE'S encoder but in order to generate z, I still need to sample from a distribution using epsilon in which makes it non-discrete w.r.t user-generated z. This adds randomness to it so I am not sure how would I use the encoded z to find the nearest to user-generated z.

If you are training a VAE the encoder is essentially parameterizing the variational posterior distribution of $z$ given $x$, i.e.
$$ q(z | x) = \prod_{i=1}^{N_z} q(z_i | x) = \prod_{i=1}^{N_z} \frac{1}{\sqrt{2\pi}\sigma_i(x)}  \exp \left[ -\frac{(z_i - \mu_i(x))^2}{2\sigma_i(x)^2} \right] $$
where $\mu_i(x)$ and $\sigma_i(x)$ are given by the encoder and $z$ is in the $N_z$-dimensional latent space.
I would think of the problem as if the $x$ are parameters of a probability distribution and the $z$ are some observations you made.
The "nearest" encoding from your training data $x$ would be the encoding with the highest likelihood, i.e. you compute the likelihood for each data point for a given $z$ by evaluating the above expression and take the $x$ with the maxmal value.

The log-likelihood is usually used in these scenarios because it's more convenient, but it is equivalent, as the likelihood is non-negative and the log is a monotonic function.

In the comments you mentioned to use a distance metric. The log-likelihood provides a nice interpretation, because it gives you something similar to the negative euclidean distance between $\mu(x)$ and $z$, but scales and shifts by terms determined by the standard deviation:

$$ \log q(z|x) = \sum_{i=1}^{N_z} \log q(z_i|x) = \sum_{i=1}^{N_z} \left[ -\frac{(z_i - \mu_i(x))^2}{2\sigma_i(x)^2} - \log \left( \sqrt{2\pi}\sigma_i(x) \right) \right] $$

So intutively, by maximizing the (log-)likelihood, you are minimizing the euclidean distance between a given z and the encoding $\mu(x)$ of $x$ from the training data set, but you pay penalties for having large variances.

(Furthermore, if you do it this way, there is no sampling of $\epsilon$ required.)


Output of machine learning classification prediction on out-of-sample data has too few observations Using the mlr package, I have developed a task using word frequency data to categorize tweets into two categories (TRUE and FALSE). Now I am using this task to classify out-of-sample tweets. Sorry, I cannot share my data, but I will show my code here:
task = makeClassifTask(data = train, target = "category")
mod = train("classif.randomForest", task)

newdata.pred = predict(mod, newdata = outofsample)
newdata.pred

My output is shown here:
Prediction: 17981 observations
predict.type: response
threshold:
time: 0.29
... (#rows: 17981, #cols: 1)

And as a dataframe:
response
<fctr>
1   TRUE
2   FALSE
3   FALSE
4   FALSE
5   FALSE
6   FALSE

I now want to use my categorization to remove any tweets falling in the "FALSE" category. But if I have 17981 tweets, why do I only see 6 observations? I cannot find anything wrong with the "train" dataframe or "outofsample" dataframe (they both have the appropriate number of observations and are listed as dataframes in the global environment, but I did notice that the object "task" creates a list of 6. Is this just a coincidence? How do I retrieve the classification for all of my tweets? And how do I link this information to the out of sample dataset to delete tweets by ID number, if this classification method does not include ID number?
Please don't judge me too harshly, I'm very new to machine learning and R in general. Any advice would be a big help.

The actual predictions can be found in newdata.pred$data.
Note that mlr is no longer maintained:

{mlr} is considered retired from the mlr-org team. We won’t add new
features anymore and will only fix severe bugs. We suggest to use the
new mlr3 framework from now on and for future projects.

See the docs: https://mlr.mlr-org.com/
library(mlr)
library(ISLR)

df = ISLR::Auto
df$cylinders = as.factor(df$cylinders)
df$name <- NULL

task = makeClassifTask(data = df, target = "cylinders")
mod = train("classif.randomForest", task)

newdata.pred = predict(mod, newdata = df)
result = data.frame(newdata.pred$data)

    truth response
1       8        8
2       8        8
3       8        8
4       8        8
5       8        8
6       8        8
7       8        8
8       8        8
9       8        8
10      8        8
11      8        8
12      8        8
...


tf.trainable_variables() returns blank list outside model_fn In a downloaded tensorflow code, when below model_fn_builder is called, then model_fn (down below) loads the tf.trainable_variables().

I tried to move tvars = tf.trainable_variables() out of model_fn so that variables do not get loaded everytime the prediction is called (model_fn is called on every prediction generator parsing).

However tf.trainable_variables() returns blank list. Why is this so? Is tf different within model_fn and outside?

model_fn = model_fn_builder(
  bert_config=bert_config,
  num_labels=len(label_list),
  init_checkpoint=FLAGS.init_checkpoint,
  learning_rate=FLAGS.learning_rate,
  num_train_steps=num_train_steps,
  num_warmup_steps=num_warmup_steps,
  use_tpu=FLAGS.use_tpu,
  use_one_hot_embeddings=FLAGS.use_tpu)

In below code #THIS I AM TRYING TO MOVE OUT OF MODEL_FN is the comment which to be focused:

def model_fn_builder(bert_config, num_labels, init_checkpoint, learning_rate,
                     num_train_steps, num_warmup_steps, use_tpu,
                     use_one_hot_embeddings):
  """Returns `model_fn` closure for TPUEstimator."""

  global tvars , initialized_variable_names , scaffold_fn
  tvars = tf.trainable_variables()  #THIS I AM TRYING TO MOVE OUT OF MODEL_FN

  if init_checkpoint:
      (assignment_map, initialized_variable_names
       ) = modeling.get_assignment_map_from_checkpoint(tvars, init_checkpoint)
      if use_tpu:

          def tpu_scaffold():
              tf.train.init_from_checkpoint(init_checkpoint, assignment_map)
              return tf.train.Scaffold()

          scaffold_fn = tpu_scaffold
      else:
          tf.train.init_from_checkpoint(init_checkpoint, assignment_map)

  def model_fn(features, labels, mode, params):  # pylint: disable=unused-argument
    """The `model_fn` for TPUEstimator."""

    tf.logging.info("*** Features ***")
    for name in sorted(features.keys()):
      tf.logging.info("  name = %s, shape = %s" % (name, features[name].shape))

    input_ids = features["input_ids"]
    input_mask = features["input_mask"]
    segment_ids = features["segment_ids"]
    label_ids = features["label_ids"]
    is_real_example = None
    if "is_real_example" in features:
      is_real_example = tf.cast(features["is_real_example"], dtype=tf.float32)
    else:
      is_real_example = tf.ones(tf.shape(label_ids), dtype=tf.float32)

    is_training = (mode == tf.estimator.ModeKeys.TRAIN)

    (total_loss, per_example_loss, logits, probabilities) = create_model(
        bert_config, is_training, input_ids, input_mask, segment_ids, label_ids,
        num_labels, use_one_hot_embeddings)

    # tvars = tf.trainable_variables() #THIS IS ORIGINAL PLACE WHICH WORKS
    # initialized_variable_names = {}
    # scaffold_fn = None
    # if init_checkpoint:
    #   (assignment_map, initialized_variable_names
    #   ) = modeling.get_assignment_map_from_checkpoint(tvars, init_checkpoint)
    #   if use_tpu:
    #
    #     def tpu_scaffold():
    #       tf.train.init_from_checkpoint(init_checkpoint, assignment_map)
    #       return tf.train.Scaffold()
    #
    #     scaffold_fn = tpu_scaffold
    #   else:
    #     tf.train.init_from_checkpoint(init_checkpoint, assignment_map)

    tf.logging.info("**** Trainable Variables ****")

This problem is resolved... tensorflow.python.framework.errors_impl.NotFoundError: Key output_bias not found in checkpoint – Actually there were model.ckpt-10949 files.. and tensorflow was picking up this checkpoint automatically... these files seem to be incomplete or corrupt. After deleting 1model.ckpt-10949* files key NotFoundError went away


How can I compare the grammatical complexity between two texts using their sentences dependency length? This is a continuation to the following thread.
I have two texts, common English texts such as news articles and informative texts versus a technical textbook. I want to compare the grammatical complexity between those texts using their sentences dependency length in order to conclude whether they both have the same level of complexity or not. I was thinking about using the p-value as an evidence against the null hypothesis. Here is how the data would look like:
Text 1
ID  Dependency Length   Sentence Length
0   13                  7
1   5                   3
2   20                  8

Text 2
ID  Dependency Length   Sentence Length
0   8                   5
1   10                  7
2   14                  7

By the way, I am using python.

If your goal is only to determine whether the level of complexity is the same between these two sources based on these particular features, yes you can simply compare the distributions with a Student's t-test if the distributions are normal or a Wilcoxon test if they are not.
Spoiler alert: it's very likely that they are different. A statistical significance test doesn't give you much information, it's usually much more useful to try to quantify the level of complexity, but it's also much harder. Based on your previous linked question, you might be interested to read about text complexity/readability, quite a lot of research has been done on this topic (e.g. here, here, or there, among many others). There are also a few general tutorials apparently (e.g. this or that). I also know that there have been a number of readability/complexity metrics proposed in the literature, but I don't have references.
Note that looking at the features from the text is unlikely to be very useful on its own. Probably you will need either some kind of corpus annotated by complexity, or to use a metric which has been proved to correlate with sentence complexity.


How to deal with categorical feature of very high cardinality? I would like to train a binary classifier on feature vectors.
One of the features is categorical feature with string, it is the zip codes of a country.

Typically, there is thousands of zip codes, and in my case they are strings.
How can convert this feature into numerical?

I do not think that using one-hot-encoding is good as a solution for my case.
Am I right by saying that? If yes, what would be a suitable solution?

This is an old question. I am surprised that I don't see anyone mentioned Mean Encoding (a.k.a Target Encoding). It is very popular in supervised learning problems. Besides, I have seen people use frequency or the cdf of the frequency (to avoid noise generated by heavy-tailed pdf), and they achieved pretty good results with lightGBM. However, i cannot really explain why it works rigorously.


Bayesian network for classification using PyMc or PyMc3 I am seraching for a  while  an example  on how to use  PyMc/PyMc3 to do  classification task, but have not  found  an concludent   example  regarding  on how to do   the  predicton on a  new data point.
I am currious if some   could   give  me some references.

pymc will not provide you pretty sklearn-style .predict method for this case, however you can do it on your own. The idea is simple enough: you should draw coefficients for the classifier using pymc, and after it use them for the classifier itself manually.

You can see a very basic example at this blogpost or more complicated case at pymc3 documentation.


Questions about LSTM cells, units and inputs I'm trying to learn how LSTM networks work, and even if I get the basics, the details of the internal structure is not clear for me.

On this blog link, I found this scheme of a LSTM architecture

Where apparently, every circle should correspond to an individual LSTM unit like this

Is this correct?
Is each unit in the cell independent from the others? Or do they share information?
Imagine I have the following configuration:
Number of samples = 1000
Number of time-steps = 10
Number of features = 5

In this case, each unit in a cell will take as input a vector of size 5 right?

But what will be the size of the output for one unit? 1?

Thanks

Is this correct?

Yes the diagrams both look correct to me. The key thing to understanding both diagrams is that the inputs and outputs of an LSTM cell are vectors.

The circles in the first diagram do represent the concept that the layer contains multiple individual artificial neurons, and that might make you assume that the second diagram is a picture of one of those neurons. Arguably there are multiple "neurons" or sub-layers with different roles inside a cell, because there are multiple places in which calculations of the form $f(W\mathbf{x} +b)$ occur, performing slightly different roles. I think the term "cell" is used to refer to this architecture of neurons. As a short-hand when I say "neuron", I tend to think of the output stage of the hidden layer.

However, in the second diagram, all of the operations shown work with vectors. Most importantly the left-to-right arrows in the second diagram represent vectors of hidden state from timestep to timestep from the whole layer. So each neuron in the cell is recurrently connected to every other neuron in that cell - twice as it happens in an LSTM, because LSTMs have both an internal cell state and a layer output.

  Is each unit in the cell independent from the others? Or do they share information?

To match your description of the diagrams, let's define a "unit" as a collection of one of each type of neuron/gate used to make up the cell, that in theory could be wired together to make a working LSTM cell layer with a single scalar cell state and output value.

These units are independent in that each has its own weight parameters. There are no shared parameters for the connections between input and the units, or for the recursive connections that forward state from one time step to the next. In that sense the units do not share information.

However, the connections do mean that on each time step, input data and hidden state plus output from last output from all other units in the cell are combined are used in calculations. Any cell unit can base its new internal state plus its output on the values of all other outputs and internal states from other units in the cell. In this sense, the units do share information. I guess from your question that it is probably this second issue that you are concerned about, as the second diagram makes you think of a wiring diagram for a single neuron, but as explained above that is not the case.

  Imagine I have the following configuration: Number of samples = 1000 Number of time-steps = 10 Number of features = 5
  In this case, each unit in a cell will take as input a vector of size 5 right?

Almost. Each neuron inside the cell will take an input of 5 from $\mathbf{x}$, plus an input of the hidden layer output, $\mathbf{h}$. So if in your case the LSTM cell size was 10, then each neuron would take a combined vector of 15. In addition, a second cell state vector is maintained, not labelled in your diagram. That is not used directly as an input to any neurons (i.e. components of the form $f(W\mathbf{x}+b)$), but does interact with the other values and can get changed itself, through the various gates. In your second diagram, it is the uppermost arrow going left to right.

  But what will be the size of the output for one unit? 1?

The cell as a whole will have the output of whatever size you have made the layer. That is what the diagram number 2 tries to show. However, using our working definition of "unit", the output of each unit will be two scalar values - the hidden layer output, and the cell state - which will be part of their respective vectors shown in the diagrams.


how to scale a dataset contains a b&w and Grayscale images I have a dataset that contain both black and white images and a grayscale images (some of them are scnned by printer and other by camera and changes into gray)
how can make or scale my dataset so I can use it to feed a model ?

$\hat{p}=2\big(\frac{p}{p_{max}}\big)-1$ so that $\hat{p}\in [-1,1]$ where $p\in[0,p_{max}]$
E.g. with 8-bit channels, $p_{max}=255$, black/white correspond to $-1$ and $1$ respectively, and grayscale is linearly mapped in $[-1,1]$.


Is the gradient descent the same if cost function has interaction? We know how to determine regression parameters using gradient descent. If

and the cost function is C=|Y-Y(X)|^2, we update b as

where  is the learning rate and  is the partial differential of the cost function C with respect to b.

If in multiple regression there exist an interaction and we want to stick on the linear model formulation (not using tree or other non-linear regressors), such that

and the cost function is still the same, do we just do the same way to update b? i.e. the existence of interaction terms doesn't have impact on gradient descent. I didn't see any difference of gradient descent between with/without interaction.

The usual way to use interaction terms in linear regression is to construct new $x_n$, e.g. $x_3 = x_1 x_2$, and treat those identically as any other $x_n$. The learned parameter $b$ does not "know" the difference in how you calculated $x$, and the problem is still considered linear regression even if you create really complex functions of $x_n$ to create an input.

Taking your example, but with slightly different notation:

The model estimate for Y is $\hat{y} = a + bx_1x_2$
The value of Y you want to learn is $y$
Mean squared error for a single example is $L = \frac{1}{2}(y-\hat{y})^2$ The factor of 2 does not change this answer, and is commonly used to simplify the gradient. Typically $C$ is the mean of $L$ over all examples.

In order to learn optimal value of $b$, for one example the gradient you need is $\frac{\partial L}{\partial b}$. We can get that by expanding the loss function:

$L = \frac{1}{2}(y-\hat{y})^2$

$L = \frac{1}{2}(y^2 - 2y\hat{y} + \hat{y}^2)$

We could expand further, but typically now we calculate $\frac{\partial L}{\partial \hat{y}}$ and use the chain rule, because that has a simpler, more intuitive-looking result. Terms without $\hat{y}$ are zero:

$\frac{\partial L}{\partial \hat{y}} = \hat{y} - y$

We want $\frac{\partial L}{\partial b}$ for gradient descent

$\frac{\partial L}{\partial b} = \frac{\partial L}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial b}$ (Chain rule)

$\frac{\partial L}{\partial b} = (\hat{y} - y)(\frac{\partial}{\partial b} a + bx_1x_2)$

Again, terms without $b$ in them are constants:

$\frac{\partial L}{\partial b} = (\hat{y} - y)(x_1x_2)$

Note that the $x_1x_2$ term is unchanged from the input. It could be any function $x_n = f(x_1, x_2, x_3 ....)$


Tagged dataset with photos for race detection Looking for the tagged dataset, because I would like to identify race by photo. I tried using the UTKFace dataset from Kaggle, but it outputs hispanic and Arab people on images as other. My customer would like those ones as separate categories. Do you have any datasets like that? Because UTKFace has only 4 tags - black, white, indian, other.

Alright, I believe I found one:
https://github.com/dchen236/FairFace


The goal of fine tuning I would like to ask what is the goal of fine-tuning a VGGNet on my dataset.
What does fine-tuning mean? Does it mean to change the weights or keep the weight values?

Fine tuning means changing the weights such that the VGGNet can perform the task you want in your dataset. The reason why fine-tuning is not called training (which is what you are doing) is because it implies that you already use a network that has been trained on a dataset. However, the concept is the same as training, but you just happen to it do with a convenient set of initial weights.


What is the meaning of hand crafted features in computer vision problems? Are these the features which are manually labelled by humans? or  Is there any technique for obtaining these features. Is this related to learned features?

"Hand Crafted" features refer to properties derived using various algorithms using the information present in the image itself. For example, two simple features that can be extracted from images are edges and corners. A basic edge detector algorithm works by finding areas where the image intensity "suddenly" changes. To understand that we need to remember that a typical image is nothing but a 2D matrix (or multiple matrices or tensor or n-dimensional array, when you have multiple channels like Red, Green, Blue, etc). In the case of an 8-bit gray-scale image (or a "black and white" image, although this latter definition is not quite accurate) the image is typically a 2D matrix with values ranging from 0 to 255, with 0 being completely black and 255 being completely white. Now imagine an image of a blackboard set against a totally white wall. As we move left-to-right in the image the values in one of the rows of the matrix might look like 255-255-255... since we will be "moving" along the wall. However, when we are about to hit the blackboard in the image it might look like 255-255-0-0-0...As you might have guessed, the blackboard "begins" in the image where the zeros start. In other words, the "intensity" of the image along the "x" axis has dropped rather suddenly (or a very large negative gradient along x), which means a typical edge detector will consider it to be a good candidate for an edge.

The algorithm that we just saw is only the very basic of algorithms, and others like Harris corners and Hogg detectors use slightly more "sophisticated" algorithms. Actually, even the Canny edge detector does a lot more than I what I just described, but that is besides the point. The point is that once you understand that an image is nothing more than a data matrix, or an n-dimensional array, the other algorithms are not that difficult to understand either.

As regards your last question:

  Is this related to learned features?

The "handcrafted features" were commonly used with "traditional" machine learning approaches for object recognition and computer vision like Support Vector Machines, for instance. However, "newer" approaches like convolutional neural networks typically do not have to be supplied with such hand-crafted features, as they are able to "learn" the features from the image data. Or to paraphrase Geoff Hinton, such feature extraction techniques were "what was common in image recognition before the field became silly".


Determine highly correlated segments Given a dataset that has a binary (0/1) dependent variable and a large collection of continuous and categorical independent variables, is there a process and ideally a R package that can find combinations/subsets/segments of the IVs that are highly correlated with the DV?

Simple example:
DV: college education (0/1), and IVs: age (20 to 120), income (0 to 1 million), race (white, black, hispanic etc), gender (0/1), state, etc.

Then finding correlations combining IVs and subsets of IVs (e.g. women between 30 and 50, with incomes over 100k are highly positively correlated with the DV), and then being able to compare the combinations (e.g. to find out women between 30 and 40, with incomes over 100k have a higher correlation than women between 40 and 50, with incomes over 100k)

The idea you have in mind is called "feature selection" or "attribute selection". The fact that you have a categorical dependent variable and continuous independent variables is mostly irrelevant because you're expected to use an algorithm or statistical method that is suitable for your requirements.

As for feature selection methods, there are several options:

Find the subset of features that achieves better performance (usually in cross validation)
Find the subset of features that correlates highly with the target variable and low with each other (although other criteria can be used)
Use an algorithm that includes a built-in feature selection mechanism (e.g. decision trees, hierarchical bayesian methods)

Furthermore, there are several methods aimed at obtaining a good compromise between a thorough search and a reasonable time execution (e.g. best first, steepest ascent search, etc)

This question in particular provides very good suggestions for R packages.


How do you manage expectations at work? With all the hoopla around Data Science, Machine Learning, and all the success stories around, there are a lot of both justified, as well as overinflated, expectations from Data Scientists and their predictive models.

My question to practicing Statisticians, Machine Learning experts, and Data Scientists is - how do you manage expectations from the businesspeople in you company, particularly with regards to predictive accuracy of models? To put it trivially, if your best model can only achieve 90% accuracy, and upper management expects nothing less than 99%, how do you handle situations like these?

Rational business people don't pay for accuracy, they pay to either

save money on a profitable process (thereby making it more profitable), or by
creating new money (creating new profitable processes).

So any project that is undertaken has to be couched in terms that
   reflect this. The first step is always understanding which of the two
   processes you are working on, and you should have a clear idea of how
   that might be achieved, while keeping in mind that as you make
   progress the details of how you do that might change.

If you can improve the accuracy of a process, you can probably make money for the firm and the business people will invest in your progress. The only rational reason a business person could have for insisting on 99 percent accuracy and reject 90 percent is if they already had a way of doing it that was better than the 90 percent.  If that is the case they are of course justified in their position.

Understanding and presenting the business case of the projects you are working on in terms that the Business People understand is part of the maturing process of any Engineer. It is not unique to Data Science at all, though Data Science has some unique aspects (like lower maturity but higher probability of fortuous serendipity discovery - at least in today's environment.

A relevant process that is close to Data Science that makes this step explicit can be found here:https://en.wikipedia.org/wiki/Cross_Industry_Standard_Process_for_Data_Mining

But most Enterprise Architecture Frameworks are similarly applicable.


Can RNN work with just one input? We have a time series data.Is it possible to create a RNN such that there is only dimenson(or feature) in data(as shown in fig1).If yes,will it identify the pattern in time series data correctly? The confusion arises because only neuron (yellow to blue in figure 2) information is passed to the output.Is it sufficient to predict a pattern in time series data?

Yes, in this case train_x and train_y both have just 1 column.  RNN learns patterns in just that variable and tries to predict next instance based on the patterns.

Usually, this is just a theoretical exercise. In real world, you will have multiple features.

Some examples :

https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/

https://machinelearningmastery.com/time-series-forecasting-long-short-term-memory-network-python/


Similarity between an n-dimensional curve and selected subsets of them I'm asking here to get some advice. My goal is to detect similar patterns in an n-dimensional data. For example (extreme simple) you have two axes: X-Axis is Time (in seconds) and Y-Axis is Power (in Watt). So you get time-series with a specific pattern. My goal is to select a subset of the time-series and calculate the similarity of other input data to the selected subset.
In the end, I want to define multiple patterns and the output predicts which pattern comes most closely to the given input data.

Is it suitable to use Machine Learning e.q. pattern recognition for that problem or is it a pure overkill? (I got the advice to use ML; also under the condition that I have an generic n-dimensional data)
If ML is suitable for this problem, can you give me some advice where I could start? I don't ask for a solution in any way, I just want to have some suggestions in which direction I should look, because ML is a very new topic for me and I don't have much experience in practice.

Thank you.

So let's start!

First of all please have a look at the edit I made to your original question. It was not just an edit but implied important conceptual thing so I need to start with them:

Graphs: Graph is a reserved term in mathematics for network-like objects. You better use the term Plot for sake of clarification. So the question is absolutely not about graphs.
Time-series data: The example you made i.e. a temporal sequence of data points is called Time-series. A n-dimensional data is not necessarily a time-series and however both seem to inhibit the same structure, they are conceptually different (but not necessarily irrelevant) types of data and different sets of algorithms and problems are defined for either, so please comment here if the data is time-series or not.
Curve Plots: The last but not the least is that a plot is used to just illustrate the data and all Data Analysis algorithms and approaches use that data for inference. So you never infer anything from the plot itself but from the data that it depicts. If you have any plot-like data which you don't know the data behind you need to use Image Processing which I'm pretty sure is not the topic of this question but wanted to mention as you explicitly asked how to do the analysis with plots themselves.

After these points I get back to your question. Yes, what you are looking for is Pattern Recognition and you can use many Machine Learning approaches to solve your problem. If you already knew what are different classes of patterns, it would be called Classification but now, as I understood, you don't know the classes and you want to find similar patterns. Finding patterns of similarity in data is called Clustering.

As I don't know if your data is a time-series (e.g. amount of raining vs time) or just a structured n-dimensional data (e.g. amount of raining vs humidity) I try both.

Time-Series

When you have different segments of time-series and you need their similarity you can use Correlation Analysis or Dynamic Time Warping. If the time series is high frequency e.g. speech data or EEG, then you better convert data to frequency (or time-frequency) domain and then extract features from those segment and then use those features to determine similarity. Please note that the term similarity is pretty wide and it's defined according to the nature of data and problem. You may search for time-series clustering for more approaches.

Non-temporal Data

Similarity between the parameters of a regression model fitted to those subsets of data might help. Another approach is to apply clustering algorithms to those segments. These algorithms can be applied to the original data subsets or some features extracted from those. Dimensionality Reduction algorithms like PCA can be used for feature extraction and might be helpful for a better clustering.

I wrote my answer in rush so I would appreciate any comment or further questions.

Good Luck!


How can I predict the last element of the fixed length=8 sequence after I get each element? There are fixed length lists [X1, X2, X3, X4, X5, X6, X7, X8] like this.
I have many lists like them from the past. In the future, I will get new element of current list on weekly bases. one new element for each week.
what machine learning model can I use to predict the last element as soon as I get new element ?

This can be modelled as a Markov Chain.
Let us say each element in your list can take N states. You can then represent a first order markov chain by build an N x N matrix - where each cell represents a state transition probability.
You can use this state transition probability to predict the most likely value X9 takes based on the known value from the previous timestep X8.
You also have the option of using a higher order markov chain - for e.g. a second order chain could allow you to predict X9 based on values from the previous 2 timesteps - X7 and X8.


Apache Spark Question I am trying to parse the files using Stanford nlp in Spark in mapper function. How to set the number of mappers in Apache Spark? Please help me.

It automatically determines the amount of mappers by the number of partitions your data is in. You can call getNumberPartitions on your data source (RDD/DataFrame) to see how much it is and use repartition for scaling this up or coalesce to scale this down (you can use repartition for this as well but this is slower). Repartitioning is expensive however and should be avoided when unneccesary.


Jupyter, Python: the kernel appears to have died while training a model on a big amount of data I am training my model on almost 200 000 images, i'm using Jupyter and now after 3 days of training ( i used 800 epochs and batch-size = 600) I have this " the kernel appears to have died. It will restart automaticaly" And this appears after 143 epochs only. Can anyone help me to solve this, and also can anyone advise me something in case of using big amount of data, because i am struggling with this dataset and I can't retrain the model each time the Jupyter blocks. Infact, I'm working on my internship project so I have to use all the data. I will be so grateful for your help.

I have had the same problem while training in huge data sets in Jupyter Notebooks. The only solution I found was to create a scrip .py with my training process (including model persistence) and running it from my terminal (python3 myscript.py)


Keypoint detection from an image using a neural network I am trying to design and train a neural network, which would be able to give me coordinates of certain key points in the image.

Dataset

I've got a dataset containing 1800 images similar to these:

This dataset is generated by me. Each image contains two circles, one smaller and one bigger, generated randomly in the image. My goal is to train the neural network to return 2 sets of coordinates, each of them pointing precisely at the center of the circle. Each image has the shape (320, 320, 1).

Current model

I've been successful to do so to some degree, but it's not good enough. Below, you can see the neural net architecture I've been most successful so far. I use Python, Tensorflow 2 including Keras. I use Adam optimizer, MeanSquaredError loss and RootMeanSquaredError as a metric.

The current result looks as follows. The coordinates that the neural net gave me are drawn in the image.

As you can see from the first image, the result is quite precise that I am almost satisfied with. But the average result looks like as in the second image, which is not good at all.

I have trained this model for 35 epochs in a total of 4 runs and it is not able to learn any further as you can see from the Tensorboard.

I have tried many different variations of the architecture and tuned hyperparameters. I am not satisfied with the result I got so far. I plan to continue on detecting keypoints from images on more complex datasets and that is the reason why I'm trying to make some progress on much simpler datasets first and add complexity gradually.

I would appreciate any advice you can give me on the model architecture that I would have better results with or maybe a different approach. Tell me if you need to know more implementation details.

Thanks

Edit: To complete the details, Conv2D layers use leaky relu activation function and both dense layers use the sigmoid activation function.

You could train YOLO to detect and create bounding box around the circle and then calculate its center. Or even segmentation could help you do this precisely.


Multiple Categorical values for a single feature how to convert them to binary using python I have a data set of movies which has 28 columns. One of them is genres. For each row in this data set, the value for column genres is of the form "Action|Animation|Comedy|Family|Fantasy". I want to encode them using pandas.get_dummies() but since the columns have multiple values, how to deal with such conditions?
      Additinal information on below link(question moved from stackoverflow)
https://stackoverflow.com/q/40331558/4028904

I'm starting with the following dataset:

import pandas as pd
data = pd.DataFrame({'title': ['Avatar', 'Pirates', 'Spectre', 'Batman'],
                 'genres': ['Action|Adventure|Fantasy|Sci-Fi',
                            'Action|Adventure|Fantasy',
                            'Action|Adventure|Thriller',
                            'Action|Thriller']},
                columns=['title', 'genres'])

     title                           genres
0   Avatar  Action|Adventure|Fantasy|Sci-Fi
1  Pirates         Action|Adventure|Fantasy
2  Spectre        Action|Adventure|Thriller
3   Batman                  Action|Thriller

First, you want to have your data in a structure pairing titles with one genre at a time, multiple rows per title. You can get it in a series like this:

cleaned = data.set_index('title').genres.str.split('|', expand=True).stack()

title
Avatar   0       Action
         1    Adventure
         2      Fantasy
         3       Sci-Fi
Pirates  0       Action
         1    Adventure
         2      Fantasy
Spectre  0       Action
         1    Adventure
         2     Thriller
Batman   0       Action
         1     Thriller
dtype: object

(There's an extra index level that we don't want, but we'll get rid of it soon.) get_dummies will now work, but it only works on one row at a time, so we need to re-aggregate the titles:

pd.get_dummies(cleaned, prefix='g').groupby(level=0).sum()

         g_Action  g_Adventure  g_Fantasy  g_Sci-Fi  g_Thriller
title
Avatar        1.0          1.0        1.0       1.0         0.0
Batman        1.0          0.0        0.0       0.0         1.0
Pirates       1.0          1.0        1.0       0.0         0.0
Spectre       1.0          1.0        0.0       0.0         1.0


python sklearn decision tree classifier feature_importances_ with feature names when using continuous values I'm using sklearn Decision Tree Classifier with some continuous features.
When I run export_graphviz I see the same features in more than one nodes and with different values. Example:

I would like to take the top important ones and want to use feature_importances_ for that.
The problem is that feature_importances_ is array without reference to the tree nodes. I have the original features but as each one can be more than one time in the tree I'm not sure how to relate importance to node.

I think you are mixing two different things here.

The feature_importance_ - this is an array which reflects how much each of the model's original features contributes to overall classification quality.
The features positions in the tree - this is a mere representation of the decision rules made in each step in the tree. A feature position(s) in the tree in terms of importance is not so trivial.

There are some potential heuristics for understanding the relation between the two. If a feature dosn't appear in the tree it has 0 importance and generally the higher the feature is in the tree the more important it is (assuming its being compared to another feature on the same branch).


Question about (Python/Orange) Apriori associative algorithm I'm trying to wrap my head around Association rules and frequent itemsets.
So I threw my data in, instead of the samples one and sometime it works, sometimes it doesn't.

rules = Orange.associate.AssociationRulesSparseInducer(data, support = 0.3)

print "%5s   %5s" % ("supp", "conf")
for r in rules:
    print "%5.3f   %5.3f   %s" % (r.support, r.confidence, r)

inducer = Orange.associate.AssociationRulesSparseInducer(support = 0.2, store_examples = True)
itemsets = inducer.get_itemsets(data)
print itemsets
print data.domain
print [data.domain[i].name for i in itemsets[4][0]]

More often than not, itemsets[4][0] shows an IndexError: list index out of range error.
So I start playing around with support = 0.3, support = 0.5, support = 0.2 and itemsets[2][0] or itemsets[3][0].

From the docs:

  support
  Minimal support for the rule. Depending on the data set it should be set to sufficiently high value to avoid running out of working memory (default: 0.3).

True - I tried 0.2 and it quickly blasted my memory on a 800 rows data file.

Any idea what I should do best there or which are viable values for a shopping cart analysis?

800 rows of data (800 orders)
1 to x item(-categories) per order
15 different item-categories in the file, so my data looks like:

ItemCat1
ItemCat2, ItemCat2, ItemCat2, ItemCat2, ItemCat7, ItemCat7, ItemCat7,     ItemCat7, ItemCat7
ItemCat1, ItemCat1, ItemCat1, ItemCat1, ItemCat1, ItemCat1, ItemCat1,     ItemCat2
ItemCat4, ItemCat4
ItemCat1, ItemCat1, ItemCat1, ItemCat1, ItemCat1, ItemCat1, ItemCat1,     ItemCat1, ItemCat2
ItemCat5

If Python 3 is an option, Orange 3 features FP-growth in its Orange3-Associate add-on:

pip3 install Orange3-Associate

It's well documented: http://orange3-associate.readthedocs.org/

Apriori due to its slowness and resource requirements, as you'd noticed, really isn't an algorithm to consider anymore.


GAN for inpainting an image Is it possible to train a GAN model to inpaint an image taken from a specific setting(e.g. an office, woods, beach ...) after we have cropped out people of it?

For example, I used this repo's pretrained GAN model on Places2

I used a MaskRCNN model to segment people, and then cropped them out :

But when I fed the image to the mentioned GAN model, results weren't so great :

If it's possible, what is the best GAN architecture to use?

I would doubt there's a single correct answer for best available architecture, but the current best results come from this paper by NVIDIA and this technical report by Adobe. The latter paper is the second iteration of the repo you linked. Both papers focus on non-rectangular inpainting, which seems to be your targeted interest. Neither has publicly available code, however, so if you're interested in seeing what those models can accomplish you'll have to re-implement them yourself!

Edit: NVIDIA released a PyTorch implementation since the initial answer, here's the repo.


Rough vs Fuzzy vs Granular Computing For my Computational Intelligence class, I'm working on classifying short text. One of the papers that I've found makes a lot of use of granular computing, but I'm struggling to find a decent explanation of what exactly it is.

From what I can gather from the paper, it sounds to me like granular computing is very similar to fuzzy sets. So, what exactly is the difference. I'm asking about rough sets as well, because I'm curious about them and how they relate to fuzzy sets. If at all.

Edit: Here is the paper I'm referencing.

"Granularity" refers to the resolution of the variables under analysis.  If you are analyzing height of people, you could use course-grained variables that have only a few possible values -- e.g. "above-average, average, below-average" -- or a fine-grained variable, with many or an infinite number of values -- e.g. integer values or real number values.

A measure is "fuzzy" if the distinction between alternative values is not crisp.  In the course-grained variable for height, a "crisp" measure would mean that any given individual could only be assigned one value -- e.g. a tall-ish person is either "above-average", or "average".  In contrast, a "fuzzy" measure allows for degrees of membership for each value, with "membership" taking values from 0 to 1.0.  Thus, a tall-ish person could be a value of "0.5 above-average", "0.5 average", "0.0 below-average".

Finally, a measure is "rough" when two values are given: upper and lower bounds as an estimate of the "crisp" measure.  In our example of a tall-ish person, the rough measure would be {UPPER = above-average, LOWER = average}.

Why use granular, fuzzy, or rough measures at all, you might ask?  Why not measure everything in nice, precise real numbers?  Because many real-world phenomena don't have a good, reliable intrinsic measure and measurement procedure that results in a real number.  If you ask married couples to rate the quality of their marriage on a scale from 1 to 10, or 1.00 to 10.00, they might give you a number (or range of numbers), but how reliable are those reports? Using a course-grained measure (e.g. "happy", "neutral/mixed", "unhappy"), or fuzzy measure, or rough measure can be more reliable and more credible in your analysis.  Generally, it's much better to use rough/crude measures well than to use precise/fine-grained measures poorly.


What are the most suitable machine learning algorithms according to type of data? I am beginner to data science. I found that some machine learning algorithms perform better, when given particular kind of data(ie - numerical, categorical, text, graphical).
I searched about this topic on the web, but no luck.

I would like to know what kind of data performs best according to given machine learning algorithm?

It is better to explain briefly why certain kinds of data suitable for certain machine learning algorithms?

Hope answer to this question will help beginners in data science.

Update: It is better if you can explain what are the types of data most suitable for following algorithms.
Naive Bayes, SVM, Regression, K-Means, Deep Neural networks.

Given the list of algorithms you provided, these falls under 3 major classification of ML algorithms.

1) Classification Algorithms - Naive Bayes Classification, Decision Tree, Random Forest, kNN, Support Vector Machine (SVM), Neural Networks, etc.

2) Regression Algorithms - Linear Regression, Logistic Regression, Lasso Regression, etc.

(Note: Although Logistic Regression has Regression in its name, it is essentially a classification algorithm.

3) Clustering Algorithms - K-Means Clustering, Fuzzy C Means, Mixture of Gaussian, etc.

You might also be knowing there are 4 types of ML algorithms:

1) Supervised Learning

2) Unsupervised Learning

3) Reinforcement Learning

4) Semi-supervised Learning

Among these 4. first and second are the most important ones.

Supervised Learning is applied when we have a labelled data set i.e., we already our output variable/dependent variable. For example, a data set which contains the size of the house (independent variable) and corresponding house price (dependent variable). We can predict the house price of new data points with respect to the size of the house. Another example is, determining if a tumor is harmful or not harmful when we already have a list of tumors which are harmful or not.
In supervised learning we know the problem statement and have all the necessary features to get answer.

In Unsupervised Learning, we do not have labelled data. We do not have any output variable. We do not know the problem statement. It is applied when we need to find a structure in the data set and extract meaningful insights out of it. For example, a data set of Walmart containing its customer's buying pattern. Given this, Walmart will ask its data scientists to extract some meaning. The data scientist may choose to apply K-Means Clustering and find how customers are segmented. Group A customers -- buys X,Y,Z products; Group B customers -- buys U,V,X products.

Classification and regression algorithms are used when dealing with a supervised learning problem and clustering algorithms are used when dealing with unsupervised learning.

Now coming back to your original query,
1) Naive Bayes -- best applied to a data set containing multiple features (independent variable) and an output variable which takes two discrete value (Yes/No). Thus, categorical data.

2) SVM -- best applied to a data set containing infinite number of features and you need to reduce these features down to a number so that it can be computed. Since it's a classification algorithm so it best works upon categorical data.

3) Regression -- Linear Regression is applied to a continuous numerical data set in which the dependent and independent variable exhibits linear relationship. For example, size of the house vs house price.  Logistic Regression is a classification algorithm so it is best applied to categorical data.

3) K-Means -- K-Means can applied to many types of data sets. What it does is segmenting data points into clusters. Data points with similar features are clustered together.

4) Neural Networks -- Neural Networks can be shallow neural networks and deep neural networks and both of these could be applied to supervised or unsupervised problem as it has separate algorithms for both the cases. It is the most powerful and popular class of ML algorithms. It can be used in every problem statement. Main intuition behind it learning from its own error. I do not have much knowledge about neural network so I will not write further more.

If you want to learn more about ML and Neural Network you can apply for Machine Learning by Andrew Ng class. It's the best course out there for beginners like us.


Are there any control-flow/conditional statements in AI/ML models? I was recently asked this during an interview.
When we write a C program, it has a control-flow in the form of conditional statements like if, while, for that executes a condition, and that decides how the control flows through the application.
In AI/ML models, e.g. CNNs like ResNet, I don't see any control-flow (either static or dynamic). It's just a series of consecutive convolution layers. So,

where are these control-flow/conditional statements in AI domain? What do they look like?
Which models have them? Any examples?

Yes, there are control flow constructions in neural networks, for instance:

Variable-length sequence classification with recurrent units: in text classification tasks with recurrent units (vanilla RNNs, LSTMs, GRUs), both at training and inference time, you basically iterate over the time dimension of the input batch, passing the next token and the previous steps' state to the recurrent unit. This is a for loop.

Variable-length sequence generation: in text-generation encoder-decoder networks, at inference time the decoder generates one discrete token at a time until a special "end-of-sequence" token is generated. This is pretty much a while construction. The decoder can be based on any type of neural net units: LSTMs, CNNs, self-attention (Transformer), etc.

As you can imagine, this kind of control flow construction is typical of variable-size input/outputs. You won't see this kind of construction in most image processing architectures.
Control-flow stuff is more difficult to implement in declarative deep learning frameworks, like Theano or Tensorflow 1.x. In Tensorflow 1.x, there was actually a specific primitive to implemente while loops: tf.while_loop. With imperative frameworks like Pytorch or Tensorflow 2.x, control-flow constructions are far easier to implement, because you can integrate normal logic between the computations.


How to use Cohen's Kappa as the evaluation metric in GridSearchCV in Scikit Learn? I have class imbalance in the ratio 1:15 i.e. very low event rate. So to select tuning parameters of GBM in scikit learn I want to use Kappa instead of F1 score. My understanding is Kappa is a better metric than F1 score for class imbalance.

But I couldn't find kappa as an evaluation_metric in scikit learn here
sklearn.metrics.

Questions

Is there any workaround for including kappa in gridsearchcv?
Is there any other better metric i can use instead of kappa in scikit learn?

Cohen's kappa was introduced in scikit-learn 0.17.

You can wrap it in make_scorer for use in GridSearchCV.

from sklearn.metrics import cohen_kappa_score, make_scorer
from sklearn.grid_search import GridSearchCV
from sklearn.svm import LinearSVC

kappa_scorer = make_scorer(cohen_kappa_score)
grid = GridSearchCV(LinearSVC(), param_grid={'C': [1, 10]}, scoring=kappa_scorer)


Convnets : do we have separate activation maps for images in a batch I know that if we input an image of shape [6,128] to a convolutional layer with 5 filters each of shape[3,128] with S=1 and P=0 , then there will be 5 activation maps as output from the layer ..each map has a shape of [4,1]

But what about the number of maps outputted when we use batching ?

I mean if we are using a batch with size of 2 images (each of shape[6,128]) as input to a convolutional layer with 5 filters each of shape[3,128] with S=1 and P=0 also , then there will be 10 activation maps as output from the layer instead of 5 (each image in the batch has 5 maps) ?? or we will have also 5 maps but each of shape[2,4,1] ???

I think there will be 10 maps since if we apply then max pooling , we want to choose the max from each image independently , right? so each image must have separate maps from the maps of the other image..

Yes, separate maps. The number of the image in the batch is another dimension.

Max pool is performed on a per-image basis, other layers operate on an entire batch - e.g. batch normalisation.


Purpose of trace-decay parameter in eligibility traces In TD/SARSA-lambda, eligibility traces are decayed after each step by multiplying by the discount rate and the trace-decay parameter.

I understand that:

The discount rate is used to reduce the value of future actions relative to a state.
An eligibility trace should decay over time, as it represents how "responsible" a state-action pair is for an obtained reward.

What I don't understand why the trace-decay parameter is required, given that multiplying by the discount rate after each step will cause it to decay over time anyways. What does the trace-decay parameter add to the algorithm?

The beauty of eligibility trace lies in the fact that it generalizes and unifies TD and Monte-Carlo methods.

When lambda=0, only the present state will get updated, because for the other states eligibility trace will be 0 and they will not be updated. This will be a TD update.

For larger values of 0 < lambda < 1, more of the preceding states are changed, but each state which is more temporally distant is changed less due to decaying property of eligibility traces.

When lambda=1, the preceding states with positive eligibility traces will be updated. This allows us to achieve monte-carlo updation method by backward view. Eligibility trace allows us to look backward and perform updated to the preceding states. Here, the decay parameter is used to achieve the discounted reward setting for the monte-carlo updates. This property of eligibility trace will also allow us to use monte-carlo approaches for continuing tasks.


Product classification in hierarchical categories based on multiple parameters and non-standard descriptions I want to start a machine learning project in my company and a really big pain for spend analysts is to classify the products that buyers order for maintenance, tooling, raw material and such, as the description on the purchase order is free text and people can write just about anything (often the bare minimum for the order to be approved)

Some importan considerations are:

As the company is global, the language of the description is local (german, spanish, english, russian, chinesse, etc)
There is no standard for the structure of the sentences
People can input abbreviations
For certain products, the buyers don't define if the part is a repair part or a brand new product

I have read about sklearn libraries for text analysis but I really new at machine learning algorithms (I have mostly completed basic tutorials). Most of the examples are for analyzing tweets or complains, but I don't know where to start if I want to consider variables other than text, such as quantity, unitary price, provider and other parameters; and additionally, my categories are a hierarchy of four levels based on certain rules (such as, if the unitary price is over X, then it belongs to category A)

An example of the database could be:

DESCRIPTION                             uom    Mgroup   Provider     Category   Unit    Spend     Total Quantity
CATALOG: A6-CJR-45 XRE:    N/A C-RING   FT     A        Prov1        31000000   5.1     $5.10     1
Contactor iec, 9a, 24v dc, 50/60hz (    FT3    B        Prov2        32131000   82      $164.00   2
Ducto ranurado de 2 x 3, color gris,    BAL    C        Prov1        32131000   24.34   $486.80   20
Modulo de 8 salidas aisladas 5-235 vc   ST     B        Prov2        32131000   254.74  $764.22   3
Selector no-il. plastico 2 pos. mant.   ST     B        Prov3        32131000   6.46    $32.30    5
(ELEC-L2GEW4) TERMINAL TIPO LENGÜETA P SER    D        Prov2        39120000   3.77    $56.55    15
2 Position Selector Switch - Plastic,   M      E        Prov2        23161500   9.89    $69.23    7

So my question is, where could I start to investigate? Which algorithm would be best suited to tackle this problem?

Thank you!

I am working on a similar problem where we are classifying 2 million products into about 1000 categories.  I converted the product descriptions using tf-idf vectorization, and then used SVM to run the supervised classification.  There is a lot of optimization you can do with the sklearn package in Python for natural language processing.  Additionally I struggled because I wanted to include other features beyond just text descriptions.  There is a union method in sklearn for accomplishing that.  The final model was about 90% accurate in classifying, but out of a 1000 classes there were quite a few with low accuracy rates from the test set.  This is likely due to small training sample for those classes.  So, just make sure there are enough training samples for each class when you build your model.