Differences of get_dummies and labelbinarizer? Are there any differences between get_dummies and labelbinarizer in terms of what they want to achieve? It seems to be both will somehow do a one-hot encoding.

A couple of things comes to mind.

get_dummies can transform a dataframe with many columns, whereas LabelBinarizer will only do one column.

get_dummies outputs a dataframe (if the input is a dataframe) with a nicely formatted columns, whereas LabelBinarizer outputs a numpy array, so if you want to attach labels to them, you'd need to get them from the fitted instance of the LabelBinarizer.

Inverse transform is more intuitive with LabelBinarizer, it has a method named inverse_transform, whereas with get_dummies you would need to do something like dummies.idxmax(axis=1)

So overall, get_dummies seems to be a better choice


not quite sure about the difference between RNN and feed forward neural net I'm a bit confused after reading this paper: https://arxiv.org/abs/1705.09851

on page 22, the author writes

response:

\begin{equation}
Y = softmax(Z^{L-1})
\end{equation}

and hidden state

\begin{equation}
Z^\ell = max(W^\ell *Z^{\ell-1} + b^\ell, 0)
\end{equation}

which is a relu

But, to me, this looks like a regular feed forward neural net- you multiply your input by a matrix, add a bias unit, then activate. Alternatively, your hidden layer is equal to the activation of the sum of a bias and the previous hidden layer times a weight matrix.

What am I missing?

The authors state that's the formulation for a feed forward deep learner, so you're exactly right. The two equations at the bottom of the page are where they formulate their recurrent neural net

The response is

$\hat{Y} = \text{softmax}(W^2Z_t+b^2) $

and the hidden state is

$Z_{t-j} = \text{tanh}(W^1[Z_{t-j-1},X_{t-j}] + b^1), j \in\{k,...,0\}$

The authors punt to this paper for implementation details, but the recurrent nature here comes from directly articulating your hidden state off of your most recent hidden state $Z_{t-j-1}$ (which involves the input $X_{t-j-1}$ from the last time step).

Hope this helps!


A few questions to understand a random forest blog I'm trying to understand a nice blog on the trade-off between sensitivity versus specificity with the random forest and logistic regression models. I have a few questions:

1) The blog used a 10 fold cross-validation in the ranger package in R (see the model mod_rf) and set the metric as ROC. So, is the final output (confusion matrix) we get there is for the one with the best ROC (AUC value) among the 10 validation sets?

2) When I try to see the variable importance by varImp(mod_rf), it says the importance values are not available. Why is that? How can I get it?

3) The caret package in R allows upsampling to adjust for an imbalance in the data. They tried the logistic regression (see model sim_glm) to do upsampling and specified repeats = 2 to repeat the 10-fold cross-validation 2 times. How does it work? I'm not clear. Does it upsample females to create a 50-50 ratio of males and females "before" each fold of cross-validation? How would the process work for upsampling, repeats = 2 and 10-fold cross-validation in the case of a random forest?

4) If the AUC (from ROC curve) in my training data is about 10 percentage points less than the AUC from the test data, how should I explain that (this happened to my data)? I thought the training data would always show higher AUC than the test data because we used training data to build our model.

I appreciate your responses.

I had a look at your questions and a brief look at the blog you posted. I will not go into detail and answer all your questions since it will take time and there should be one question. Nonetheless, I would like to give you some heads up on this and help you where possible.

When doing a confusion matrix is usually on a test set, you can see also where is used confusionMatrix(data = rparty.test, reference = testing$Sex). What is the usual is to have a training set, cross-validated to pick hyper-parameters and so forth and then test it on the test set. There is no rule just following a logical procedure. If these are new I would recommend diving into what are cross validations vs training set, test set and validation set.
Probably is talking about SMOTE upsampling (aka oversampling). This theory is for creating new artificial variables by creating synthetic ones (by combining features of the unprecedented class). Again I would recommend having a look there are many ways of dealing with an unbalanced dataset. Let's name a few: undersampling, oversampling, ensemble sampling (combing the previous) or you can tune certainly your model in order not to be fair towards a class and more.
Will not go into much detail about the ROC curve but I would recommend checking the AUC graphical plot instead of in this example and try to understand. Hint, the training dataset was balanced hence is reasonable a model would expect a balanced dataset for a test set instead is unbalanced.

All in all, I had a quick overview didn't go into depth of your blog and I didn't the find it very comprehensive plus uses confusing terms (as I remember), for instance, oversampling instead of upsampling or precision instead of specificity and recall for sensitivity. I wrote the answer as I empathise you as new in the area but next time please post only one question and make these sort of questions at the blog if applicable instead of here where the author directly can answer. Hope I helped you little bit, but you should also understand you opened a lot of topics.


DeprecationWarning: The 'categorical_features' keyword is deprecated in version 0.20 I was watching Machine Learning A- Z from SuperDataScience but when I was doing below code sample:

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

dataset = pd.read_csv('Data.csv')
X = dataset.iloc[:, :-1].values

from sklearn.impute import  SimpleImputer
imputer = SimpleImputer(missing_values=np.nan, strategy='mean')
imputer = imputer.fit(X[:, 1:3])
X[:, 1:3]= imputer.transform(X[:,1:3])

from sklearn.preprocessing import LabelEncoder, OneHotEncoder
labelencoder_X = LabelEncoder()
X[:, 0] = labelencoder_X.fit_transform(X[:, 0])
onehotencoder = OneHotEncoder(categorical_features =[0])
X = onehotencoder.fit_transform(X).toarray()

I got this warning message:

  /usr/local/lib/python3.5/dist-packages/sklearn/preprocessing/_encoders.py:363: FutureWarning: The handling of integer data will change in version 0.22. Currently, the categories are determined based on the range [0, max(values)], while in the future they will be determined based on the unique values.
  If you want the future behaviour and silence this warning, you can specify "categories='auto'".
  In case you used a LabelEncoder before this OneHotEncoder to convert the categories to integers, then you can now use the OneHotEncoder directly.
    warnings.warn(msg, FutureWarning)
  /usr/local/lib/python3.5/dist-packages/sklearn/preprocessing/_encoders.py:385: DeprecationWarning: The 'categorical_features' keyword is deprecated in version 0.20 and will be removed in 0.22. You can use the ColumnTransformer instead.
    "use the ColumnTransformer instead.", DeprecationWarning)
  And this below message also

  /usr/local/lib/python3.5/dist-packages/sklearn/preprocessing/_encoders.py:363: FutureWarning: The handling of integer data will change in version 0.22. Currently, the categories are determined based on the range [0, max(values)], while in the future they will be determined based on the unique values.
  If you want the future behaviour and silence this warning, you can specify "categories='auto'".
  In case you used a LabelEncoder before this OneHotEncoder to convert the categories to integers, then you can now use the OneHotEncoder directly.
    warnings.warn(msg, FutureWarning)

  /usr/local/lib/python3.5/dist-packages/sklearn/preprocessing/_encoders.py:385: DeprecationWarning: The 'categorical_features' keyword is deprecated in version 0.20 and will be removed in 0.22. You can use the ColumnTransformer instead.
    "use the ColumnTransformer instead.", DeprecationWarning)
  I was reading ColumnTransfer in sklearn website library I didn't understand how to fix these error messages

SampleFile:Data.csv

You can do this to get rid of the deprecation messages

from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer

ct = ColumnTransformer(
    [('one_hot_encoder', OneHotEncoder(), [0])],    # The column numbers to be transformed (here is [0] but can be [0, 1, 3])
    remainder='passthrough'                         # Leave the rest of the columns untouched
)

x = np.array(ct.fit_transform(x), dtype=np.float)


Best practice on count of manual annotations for building criminal detection from news articles? We have 7 million news articles corpus, which we want to classify into crimes or non-crimes and further identify criminals by using NERs/annotating criminals, crime manually. For creating a model that identifies criminals, what is the number of annotated articles that we must train/build our model on? Is there any industry best practice on this count? Is there any better way to come to this number of training(annotated) dataset, than random guessing? Are there any best practices resources that anyone can point to? Thanks in advance!

It always depends on each specific problem. The amount of data depends on a number of factors including (but not limited to) the complexity of the problem, the number of features, the quality of the training data, the ratio of the training classes (i.e. class imbalance), etc.

If you have no idea where to start, sometimes the best thing is just to experiment. For example, I would create a model with as much data that is both readily available and reasonably easy to handle in a cross-validation framework. From there, you can test the effects of increasing the training set size versus tuning the hyper-parameters. While you're not necessarily trying to find the point of diminishing returns (it's usually best to err on the side of too much than not enough), this type of exercise at least gives a general idea of the complexity of your problem.

A lot of people will have their own rule-of-thumbs or metrics; there are plenty of reading out there. For example, there's the Vapnik–Chervonenkis dimension and the one in ten rule (though I think computer vision people's rule-of-thumb is one in 1000, which again illustrates how variable these rule-of-thumbs are).


Understanding how many biases are there in a neural network? I am trying to understand biases in neural nets, but different websites show very different answers.

For example, how many biases is there in a fully connected neural network with a single input layer with 5 units and a single output layer with 4 units? And what about a fully connected neural network with a single input layer with 5 units, a single hidden layer with 4 units, and a single output layer with 3 units?

For example, if I understand this correctly, https://ai.stackexchange.com/questions/17584/why-does-the-bias-need-to-be-a-vector-in-a-neural-network, the answer of the first should be 5 and for the second 4 + 3. Each neuron except for in the input-layer has a bias.

However, at https://ayearofai.com/rohan-5-what-are-bias-units-828d942b4f52, it is explained such that each layer including the input-layer has one bias. So the answer to the example above is one in the first and two in the second.

What is correct? What am I misunderstanding here?

The sources are both correct, they implement bias in different ways, and are counting slightly different things:

Your first source implements a separate bias vector in each output layer in addition to the weights, and is referring to the dimension of the bias vector when it is counting biases.
Your second source implements bias as both a separate fixed value $1.0$ in each input layer, and a larger weights matrix with an extra column containing the actual learned bias value. It is referring to the extra added value in each layer when counting the "biases" - more accurately it is counting the added bias "signals" and not the learned bias values, because the learned bias values are implemented inside the weights matrix when using this approach.

In both cases, the number of learned values added due to bias are the same, and are the same as this:

  the answer of the first should be 5 and for the second 4 + 3. Each neuron except for in the input-layer has a bias

In the second case the equivalent values appear in weight matrices as extra columns. It's really just an implementation difference. By adding a fixed bias signal to inputs, it simplifies the learning update for each layer to occur to only one matrix, as opposed to a matrix plus a separate bias vector (requiring different update rules for each). This is at the expense of needing to manipulate the inputs each time, and there is no great difference between the approaches in terms of efficiency. Some libraries take one approach, some take the other.


Assumptions/Limitations of Random Forest Models What are the general assumptions of a Random Forest Model? I could not find by searching online. For example, in a linear regression model, limitations/assumptions are:

It may not work well when there are non-linear relationship between dependent and independent variables.
It may not work if the dependent variables considered in the model are linearly related. Therefore one has to remove correlated variable by some other technique.
It assumes that model errors are uncorrelated and uniform (No hetroscedasticity).

Are there any assumptions/limitations on similar lines.

Reproducing the accepted answer from CrossValidated here, as it is the most complete and the best answer to this question.

In order to understand this, remember the "ingredients" of random forest classifier (there are some modifications, but this is the general pipeline):

At each step of building individual tree we find the best split of data
While building a tree we use not the whole dataset, but bootstrap sample
We aggregate the individual tree outputs by averaging (actually 2 and 3 means together more general bagging procedure).

Assume first point. It is not always possible to find the best split. For example in the following dataset each split will give exactly one misclassified object.

And I think that exactly this point can be confusing: indeed, the behaviour of the individual split is somehow similar to the behaviour of Naive Bayes classifier: if the variables are dependent - there is no better split for Decision Trees and Naive Bayes classifier also fails (just to remind: independent variables is the main assumption that we make in Naive Bayes classifier; all other assumptions come from the probabilistic model that we choose).

But here comes the great advantage of decision trees: we take any split and continue splitting further. And for the following splits we will find a perfect separation (in red).

And as we have no probabilistic model, but just binary split, we don't need to make any assumption at all.

That was about Decision Tree, but it also applies for Random Forest. The difference is that for Random Forest we use Bootstrap Aggregation. It has no model underneath, and the only assumption that it relies is that sampling is representative. But this is usually a common assumption. For example, if one class consist of two components and in our dataset one component is represented by 100 samples, and another component is represented by 1 sample - probably most individual decision trees will see only the first component and Random Forest will misclassify the second one.


A multivariate linear regression for explaining impacts of the predictors I am trying to build a multivariate linear regression and the main goal is to understand how the various features impact the response by understanding the coefficients and their confidence intervals.

For this reason, I chose multivariate linear regression because the coefficients are intuitive to interpret and from the standard error and the degrees of freedom, I can get the 95% confidence interval of the coefficients. Therefore, I can tell what the impact of a unit increase of a predictor on the outcome is. I could use more complex models such as tree based models but, even if I can get the variable importance, it is not easy to quantify the coefficient of each variable.

My problem, though, is the data is time series data and hence it shows auto-correlation. I know using regression with ARIMA errors can handle the autocorrelation issue but I found it difficult to interpret the coefficients specially when d is non-zero in ARIMA(p, d, q). The attached image, for example, is model diagnosis for one of the models I have (I have to build 1000s of them).

How can I handle the autocorrelation issue and still get coefficients that are readily interpretable like in a multivariate linear regression? My residuals are not normally distributed and I am planning to use box-cox transform to see if it can solve that problem. But, I am not sure what I should use for the autocorrelation issue.

To handle autocorrelation, you should try to "subtract it" from the series. In other words, using differentiation, de-seasonalization, and transformations you should try to subtract the impact of time on your time series data. Once you end up with a distribution that looks like white noise, then you can run your regression.

You can do it with an ARIMA model, simply add all the AR(), I(), and MA() components in the regression equation, and then ignore them when you evaluate the impact of other coefficients.


Working with large datasets python My data file is of 4 GB (json), I need to apply everything on this dataset (from applying clustering/ML algorithm to wrangling it) like the way people do with pandas/scikit. But it is difficult to work with them on my local machine. I tried using dask and blaze but could not find them efficient for my case because of their limited functionalities. I don't wanna use hadoop because it is not meant for such small datasets.

I wanted to know approaches for working with not so large datasets with python and applying ML algos on them. Just like the way you work on small datasets using pandas (if any exists).

With today's standard amount of RAM for a personal desktop or laptop computer (usually 8Gb or more), there should be little problem in handling 4Gb in memory. If your hardware setup is short in memory and your ML algorithms make your SO start paging, the best hint is to use algorithms and settings that do not copy the data around, but do everything in place.

In pandas, many operations provide a boolean parameter precisely called inplace that controls this behaviour, e.g. DataFrame.reset_index. Numpy also has support for in-place operations, e.g. sort.

Take into account that:

inplace=True does not guarantee that the operation is actually performed in place, it is just a hint that may or may not be honored.
for some operations. inplace=True can worsen performance, like dropna.

All that said, in my experience inplace=True actually makes a difference in memory consumption and can be used to avoid paging and therefore enabling computations to take place completely in RAM for datasets like yours.

Finally, be careful with inplace default values. As noted in this warning in the pandas documentation:

  Warning: For backwards compatability, inplace defaults to True if not specified. This will change in a future version of pandas - if your code depends on an inplace assignment you should update to explicitly set inplace=True


How to Calculate semantic similarity between video captions? I intend to calculate the accuracy of a caption generated by comparing it to a number of reference sentences.

For example, the captions for one video are as follows:
These captions are for the same video only. However, reference sentences have been broken down with respect to different segments of a video.

Reference sentences (R):

A man is walking along while pushing his bicycle.
He tries to balance himself by taking support from a pole.
Then he falls on the sidewalk along with the pole and the bicycle with him.

Candidate Caption generated (C):

A person is trying to use a pole to push off his bike ride but ends up falling down.

I want to calculate a similarity score between each pair.
That is, (R1,C), (R2, C) and (R3, C)

What is the best method?

I tried using TF-IDF and then Cosine similarity. However, that only got the word matching. I want lexical and semantic accuracy between these sentences to estimate how accurately the sentence C has been written.

You can refer the code I have done till now here

I understand I need to tokenize, do word embedding, semantic analysis and then some similarity metric but not sure? In which order and which algorithm is best suited for which?

There's not exactly best solution, mostly depending on the experiment results. So maybe you need to try several different approaches to find out the best way. But one common way (to my knowledge) is using a word embedding (add an embedding layer or use GloVe, BERT etc.) to get the word vector, and then concatenate the word vectors in each sentence into sentence representation. You can calculate the similarity between the sentence representation.

Well actually I don't practically test this :p, but I have read it in the papers. Therefore this approach is for reference only.


How to find the average and median values of the column 'Total Traded Quantity' for each of the types of 'Trend' df_trend = df.groupby(["Trend"])
average_trades = df_trend["Total Traded Quantity"].mean()
print("The average traded quantity is: ", average_trades)

>>The average traded quantity is:  Trend
Slight or No Change    1.635627e+06
Name: Total Traded Quantity, dtype: float64

But I have 9 types of trends in Trend column, although I get only one type here in the output. How shall I proceed?

df.groupby(df.Trend).mean()['Total Traded Quantity']
df.groupby(df.Trend).median()['Total Traded Quantity']

or

df_trend_average = df.groupby(df.Trend).mean()
df_trend_median = df.groupby(df.Trend).median()


With regards to VC-dimension, why can you shatter 3 points with circles but not 4 points? When using VC-dimensions to estimate the capability of a binary classifier, you can find 3 points in R2 that can be shattered, e.g.:

But you can not shatter any 4 points with a circle.

This is stated in these lecture notes. Can anyone give me an intuitive explanation?

Given $4$ points $A,B,C,D$. If they do not lie on the boundary of a convex hull, then it is impossible to shatter the inner point from the boundary.

So assume they lie on the boundary of the hull. So they form a convex quadrilateral. Meaning $\angle A+\angle B +\angle C +\angle D =360^\circ$
Then we can assume w.l.o.g. $\angle A +\angle C \leq 180^\circ$, where $A$ and $C$ are opposite points.

Now the claim is that you cannot have a circle containing $A,C$, but not $B,D$.

Assume that you have such a circle, that contains $A,C$ but not $B,D$. Then we can make the circle smaller if necessary such that $A,C$ lie on the boundary, but $B,D$ is still not contained in the circle.

But now since the the points lie outside the circle $\angle B +\angle D < 180^\circ$. But this is a contradiction thus such a circle does not exist.

This last part has to do with the fact that for a circular quadrilateral opposite angels sum up to $180^\circ$ together with the fact that an angle of a point outside a circle is smaller that on the circle.

https://en.wikipedia.org/wiki/Cyclic_quadrilateral


how to decide categorical variables for prediction I have a dataset that contains weekly sales for stores and categories. It looks like this:

I would like to apply gradient boosting method to predict weekly sales. My question is: Should I create dummy variables for categories(1 to 7 which indicates product type) and stores(1 to 11)?

Short answer : yes.

Gradient boosting relies on decision trees. The leaves of your decision trees are built in a fashion to discriminate optimally your features. For numeric features, this means finding the best separation value to decline your dataset into two subsets. One contains observations with a value above or equal to this separation value, while the other presents values that are below this separation value.

It would not make any sense to have a leaf splitting your data on a criterion such as $Store >= 5$. However, it would make sense to have a separator such as $Store_5 = 1$ (vs $Store_5 = 0$). This is precisely why dummy variables are created for categorical values in ensemble methods, such as gradient boosting.


Overfitting - how to detect it and reduce it? I have a side project where I am doing credit scoring using R (sample size around 16k for train data and 4k for test data, and also another two 20k data batches for out-of-time validation) with unbalanced classes (3.5% of BAD type of clients).

I'm trying to make various models to have enough of them to make ensembles, but for this purpose, let's focus on one of them, particularly XGBoost.

I was reading a lot on how to tackle overfitting, but I haven't found any good source on how to do it in practice, step-by-step. As for now, my two best models have:

0.8 AUC on training data, around 0.799 AUC on holdout set and around 0.7355 and 0.7195 AUC on out-of-time batches.
0.764 AUC on training, 0.7785 AUC on the holdout test set and 0,7285 AUC on both out-of-time batches.

I am worried about is that drop on out-of-time batches, since I think that 0.05-0.08 drop is huge and it might be a sign that models that I did, really are overfitting and don't generalize well. To clarify, while I was tweaking models, I didn't know about those out-of-time scores.

Could anyone share the experience what is best practice to detect overfitting? And does those two models overfit, or I am just panicking, and this drop in performance is normal?

My current pipeline in general looks like this:

Data cleanup
Feature importance using xgboost package to take best 300 features from all 2400 available.
Removing highly-correlated features (0.75 cutoff) - 123 features left
Train/test split - 0.8 vs 0.2 plus two out-of-time batches
Model selection using nested CV(5-fold CV in outer) with hyperparameter tuning in inner loop(5-fold CV in inner) - all done in MLR package.
From 5 models I get from nested CV, I'm picking the best performing one (that has the closest AUC in both train and holdout test set)

And then when I was happy with the model I performed a test on out-of-time models.

How I could improve my pipeline, in a way that I could detect overfitting? Is there any list of steps that would roughly cover what it could be done to reduce it?

Also, in highly unbalanced case, choosing a good validation set means that I only need to take care of the proper distribution of the target variable? Should I take care of something else as well?

When choosing the validation set and the test set, it is important that it reflects the actual "production environment" of your problem. Since you have "out of time" validation sets, I assume you have some time structure in your data that you need to address when making predictions. If you are developing your model, not taking this time aspect into consideration, you are likely to get issues when this model is put into a realistic environment because the models are fit to an unrealistic scenario where time does not matter.

From the scores you show here, the train and test scores are very similar, while it drops off in the "out of time"-set. This might be an indication that you are not putting enough emphasis on the time dimension while developing your model.

Another point: There is also fully possible to overfit to your validation set, when as in your case, you have a lot of variables. Since some combination of these variables might randomly fit your train and validation set well, it might not be the case for your test set. This effect is also magnified by my earlier point, where the data generating process is not really stationary across time.

Here is a nice article about how to choose a good validation set:
https://www.fast.ai/2017/11/13/validation-sets/


What is this called by gathering the meaning from a sentence? What would this process of gathering the meaning of a sentence be called? What would the segments derived from the sentence be called?

"John and Derrek both love cake"
    -> John loves cake
    -> Derrek loves cake

"John was mad that the weather was rainy today"
     -> John was mad
     -> weather was rainy today

It is called relation extraction, it is a subdomain of Natural Language Processing. Often, you first have to apply Named Entity recognition or Concept extraction to extract the entity or concept. You can then extract the relations betweens the entities with several approaches.


Supervised Learning could be biased if we use obsolete data What if the data that we could use for the training is obsolete. For instance, if I train my model with the computer sales report from the 20th century and try to predict the actual trends, a disaster, right?

Another good example is the one that profiles criminals. It will use the historical "mistakes" as fact and could wrongly suspect innocents(based on race or ethnicity).

How could I avoid this kind of situations?

One approach to using knowledge from the example you gave above is to use that information as a prior for your current model. While it is unlikely that the exact trends observed a while ago will predict future trends, some general observations/correlations will likely still be relevant for the present day.

Concerning your example of criminal profiling. Making predictive models in such cases can be highly biased and is controversial. I invite you to read the Outlook section XVIII.D of this ML review on the social implications of machine learning and the references mentioned.


How to get the mu+0.1sigma from a normal distribution? I have a normal distribution and I would like to know what is the confidence level of mu+0.1sigma (where mu is the mean and sigma the standard deviation). How do i get the confidence level?

Using python or R

From your question, I imagine you are interested in:

$\alpha = P[\mu - k \, \sigma \le x \le \mu + k \, \sigma]$

where, in your case, $k = 0.1$. In other words, by "confidence level" you mean alpha.

In R, you can obtain that via function pnorm (empirical cumulative distribution function for the standardized normal distribution):

alpha <- 2 * (pnorm(k) - 0.5)

which, for k = 0.1 produces 8%.


how do tree based methods deal with missing feature columns? all,
i have trained a model using xgboost. Some of the features are one hot encoded e.g. currency where it is either gbp or usd. it seems that when i output the feature importance gbp and usd were in 7'th 8th place respectively.
now i would like to use the model to predict whether defaulter or not on australian countries, however the currency for these is in AUD. Therefore when i apply my feature engineering it will create a column AUD once one hot encoded.
since my model doesn't have AUD as a feature how does it handle features which have been unseen? i am not clear on this

You can use Parameter to handle the unknown classes called handle_unknown in sklearn.preprocessing.OneHotEncoder
Best practice for any type of encoding :
You should train an estimator for  Onehot encoding on the training data only, and when encoding test data, you should use the same estimator used on training data.
Eg : sklearn.preprocessing.OneHotEncoder does this, and it has a parameter called : handle_unknown.

handle_unknown{‘error’, ‘ignore’}, default=’error’

Whether to raise an error or ignore if an unknown categorical feature is present during transform (default is to raise). When this parameter is set to ‘ignore’ and an unknown category is encountered during transform, the resulting one-hot encoded columns for this feature will be all zeros. In the inverse transform, an unknown category will be denoted as None.
Optimal option is : You could use this parameter and set it to ignore, in order to ignore the unknown feature value and avoid an error, until you retrain your model eventually and add the new feature values to your model.
from sklearn.preprocessing import OneHotEncoder

ohe=OneHotEncoder(handle_unknown='ignore')

train=ohe.fit_transform(train)

test=ohe.transform(test)


ML - Service Desk classification I'm trying to explore an use-case in ML but stuck at a point. May i please request your advise please.

Have a service desk web application for logging tickets, which is essentially a form having various fields like -

subject,
content,
username,
emailid,
category - {hardware, application, datafix, mobileapp, etc.},
service group - {AAA, BBB, CCC, DDD, etc.},
domain - {email, walkin, phonecall, etc.},
priority - {high, medium, low}
#(Apologies for the poor quality of sample data provided above.)

Based on this info, the ticket is then manually assigned to respective team owners for resolution.

My intent is to use ML - based on the above fields, predict the Team who will work on this ticket. (Team ex. HR or IT or Desktop support or Pantry or Facilities, etc. )

Can this use-case be categorized as Multi-class classification
problem?
The field values are stored as words in database. How can it be fed
to my ML as numbers?

Please advise.

Welcome to the DataScience Stack exchange.

Since your target variable has many categories your problem is Multi class classification. On how to implement and find a solution for this problem you can find on google. The following 2 links are for reference:

Link-1
Link-2

Here I'm not giving much information on which algorithm to use. If you need any additional help you can raise one more question so that we can help you better as we don't now much about your data and business problem.

So, to answer your question 2, you can convert them into number it is one line code,I've added the link for your reference.

Please let us know if you need any additional information.


Significance of log in DCG metric? I am trying to make logical sense about why DCG is metric is formulated as such but I am not able to understand what's the need of taking the log of (1 + rank) term. Can you help me undestand the significance of log here?

From the wikipedia page:

  Previously there has not been any theoretically sound justification for using a logarithmic reduction factor2 other than the fact that it produces a smooth reduction. But Wang et al. (2013)[3] give theoretical guarantee for using the logarithmic reduction factor in NDCG. The authors show that for every pair of substantially different ranking functions, the NDCG can decide which one is better in a consistent manner.

The work showed that the log function is able to converge to consistent results for different ranking functions. It distinguishes the methods nicely even at high numbers of documents (at thelimit).

For example, here is a graph of the ranking measures using NDCG@k, where only the top k entries in the ranked list are considered:


predicting next observation using HMMLearn.multinomialhmm(discrete hmm) I have implemented a HMM using hmmlearn:

states = ['healthy','sick']
observations = ['sleeping','eating','pooping']
model = HMM(n_components=2)
model.n_features = 3
model.startprob_ = [0.7, 0.3]
model.transmat_ = [
    [0.8, 0.2],
    [0.4, 0.6]
]
model.emissionprob_ = [
    [0.2, 0.6, 0.2],
    [0.4, 0.1, 0.5],
]

Also, I have a sequence of observations:

obs = np.array([0,0,1,0,2,0,1,2,0,1,0,2,0,1,1,2,0])
obs = obs.reshape(-1, 1)

Now, I would like to predict the next observation (at t+1), but don't know how to do this.

(I've read the documentation but haven't found anything)

i found out that there is no function to do this just can use model.predict() to get hidden states probabilities and then find out the next state(and observation) using Viterbi algorithm.


Clustering with k-means for text classification based on similarity I have a column that contains all texts that I would like to cluster in order to find some patterns/similarity among each other.

Text
Word2vec is a two-layer neural net that processes text by “vectorizing” words.

Its input is a text corpus and its output is a set of vectors: feature vectors that represent words in that corpus.

While Word2vec is not a deep neural network, it turns text into a numerical form that deep neural networks can understand.

Documentation and examples for common text utilities to control alignment, wrapping, weight, and more.

NounEdit. text (countable and uncountable, plural texts). A writing consisting of multiple glyphs, characters, symbols or sentences.

Fish are gill-bearing aquatic craniate animals that lack limbs with digits.

I tried to determine the optimal number of clusters with elbow method, but unfortunately with no success.
So I remove the stopwords and vectorised texts in the column:

import re
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans
from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer
import nltk
from nltk.corpus import stopwords
from sklearn.pipeline import Pipeline

stop_words = stopwords.words('english')

def preprocessing(line):
    line = re.sub(r"[^a-zA-Z]", " ", line.lower())
    words = word_tokenize(line)
    words_lemmed = [WordNetLemmatizer().lemmatize(w) for w in words if w not in stop_words]
    return words_lemmed

  vect =TfidfVectorizer(tokenizer=preprocessing)
    vect_text=vect.fit_transform(df['Text'])
    kmeans =KMeans(n_clusters=3).fit(vect_text)

       n_clusters=3, n_init=10, n_jobs=1, precompute_distances='auto',

Is it right to cluster the texts above based on their similarity using this approach?

So, the problem is how to cluster texts.
Firstly, an alternative approach to representing document meaning, you can use Doc2Vec and compare similarities between document embeddings (https://medium.com/wisio/a-gentle-introduction-to-doc2vec-db3e8c0cce5e).
Secondly, if you are unsure about the ideal number of clusters, instead of using k-means, you can use agglomerative clustering, which is essentially bottom-up method which clusters individual document embeddings by a distance metric to eventually merge all of the clusters to a "mega-cluster", containing all documents.
You can then see how the documents cluster together using a dendrogram.
Here is a good resource on agglomerative clustering : https://stackabuse.com/hierarchical-clustering-with-python-and-scikit-learn/


KeyError: 'i' and  Error I am implementing the codes for https://www.kaggle.com/eray1yildiz/using-lstms-with-attention-for-emotion-recognition
which is an emotion analysis from the text. I am having some errors when I am Encoding my samples with corresponding integer values. The following codes are:

X = [[word2id[word] for word in sentence] for sentence in input_sentences]

Error:
Traceback (most recent call last):
  File "<pyshell#32>", line 1, in <module>
    X = [[word2id[word] for word in sentence] for sentence in input_sentences]
  File "<pyshell#32>", line 1, in <listcomp>
    X = [[word2id[word] for word in sentence] for sentence in input_sentences]
  File "<pyshell#32>", line 1, in <listcomp>
    X = [[word2id[word] for word in sentence] for sentence in input_sentences]
KeyError: 'i'

can someone please help me. thanks

The dictionary word2id does not contain the word "i". I can run the Kaggle Kernel and do not get the error.
When I run the code my word2id dictionary looks like this:

{'i': 0,
 'feel': 1,
 'awful': 2,
 'about': 3,
 'it': 4,
 'too': 5,
 'because': 6,...}

The word "i" is definitely in there, I think you did not run the code to create the word2id dictionary properly. Make sure to run all the cells of the kernel.


Keras Sequence Generator leads to huge memory usage when used with fit_generator I want to use a fit_generator to stabilize the memory usage when training with very large datasets.
Q1: From my understanding, the generator puts batches in a queue which is fetched by the fit_generator function of keras to train the model on that batch. After the raining with that batch, it should release that batch from memory?

Q2: In my example, the memory increases with every batch. No memory cleaning is seen. Furthermore the memory does switch into the swap memory. afterwards the memory stays stable but the swap memory increases until the program crashes.

Q3: I am using the cpu only for testing purposes right now. Does the fit_generator can work as intended when everything has to run on the ram of the computer and everything is computed with the cpu?

Here is my code:

class DataGenerator(Sequence):
    def __init__(self, batch_size, id_list, dim, shuffle=True):
        # batch_size: batch_size at each iteration
        self.batch_size = batch_size
        self.id_list = id_list
        self.dim = dim
        self.batch_size = batch_size
        self.shuffle = shuffle
        self.on_epoch_end()

    def __len__(self):
        # Denotes the number of batches per epoch
        return int(np.floor(len(self.id_list)/self.batch_size))

    def __getitem__(self, index):
        '''''
        MAIN function:
        - index tells the number of batches that have already been processed
        :return: X and y with shape [batch_size, window_size, features]
        '''''
        X, y = self.__data_generation(self.id_list[index])
        return X, y

    @profile
    def on_epoch_end(self):
        if self.shuffle:
            random.shuffle(self.id_list)
        gc.collect()

    def __data_generation(self, fname):
        # 1. load data from specified path
        data = self.__load_data(filename=fname)
        data_x = np.squeeze(data[0, 0])
        data_y = np.squeeze(data[1, 0])
        return data_x, data_y

        # --------------------------------------- utility functions data loading

    def __load_data(self, filename):
        data = np.load(filename, allow_pickle=True)
        # load data from the filename
        return data

The generator is than put into the fit_generator function

train_gen = DataGenerator(batch_size=1,
                             id_list=train_files,
                             dim=(256, 6),
                             shuffle=True)

model.fit_generator(generator=train_gen,
                       epochs=2,
                       verbose=1,
                       use_multiprocessing=True,
                       workers=5,
                       max_queue_size=10)

I would like to know if my concept is wrong or if I misunderstood the use of the fit_generator function.

Q1: Yes, in theory memory should be released
Q2: This is not a question
Q3: Yes, fit_generator can work perfectly fine on a CPU.

Your data generator looks fine as far as I can see, based on my own experiences with implementing them, so I can only advise on how to proceed to find the root-cause for now as a potential remedy for the actual problem.

The first thing I would try is to set multi-processing to False and workers to 1. I have had some issues myself a while back when using the multi-processing functionality, and while I do not remember if it was memory related or race-condition related, I would start by checking if things work fine.
The second thing I would look at is alternative ways of loading the files and see if these things still lead to memory leaks.


Predict_proba() probabilities distribution I’m trying to calculate probability of class 1. I’m using gradients boosting (catboost classifier)
Is it normal to have an equal rate of positive classes in every predict_proba() bucket? e.g.:

[Probability] : Positive rate

[0 - 0.25] : 17%

[0.26 - 0.50] : 17%

[0.51 - 0.75] : 17%

[0.76 - 1] : 17%

However when I 'm using logistic regression and convert WOE to the scores, the distribution of class 1 rate decreases with increasing score. e.g.:
[Score] : Positive rate

[100-150] : 17%

[151-200] : 15%

[200 - 250] : 13%

[250-300] : 10%

Based on the context provided this is what I assume you have done:

Cat boost: Predicted the probability on 1 using all your x variables and bucket the probabilities and calculate the percentage of 1s in each bucket.

Logistic Regression: You have used 1 of the x variables which was continuous and divided into the bands mentioned and calculated the incidence rate of 1 in each band.

If both the above statements are true then

Your catboost model most likely is either underfit or did not learn anything from the model.
This inference is because the logistic Regression variables indicates that at least 1 variable has some prediction power on y.

It would be better to provide some additional details regarding the type of data and classes to analyze this further.


Is it suitable to change a feature by itself to generate an another feature? Hi all I would love to hear your answers on this. Lets say I have two variables, voltage and current, in my data set. I could add another feature by squaring current (so as to calculate power).

Is this an example of feature engineering?

Recently I tried to predict on a diameter prediction on asteroids and I took the natural log of some features which worked well.

Can someone provide some insight as to why this may have improved the model's performance?

Sure, that's feature engineering.

If you're fitting a linear model, then you are looking for features that have a linear relationship with the predicted value. If you're predicting, say, the cost per hour of a device consuming current I, then clearly that's directly related to power not current, so $I^2$ is more likely to be useful.

What you want to be careful about is trying a bunch of transformations of the input blindly; it's possible for a small data set that one odd function of an input happens to be predictive in that sample, but doesn't generalize.


Why aren't languages like C, C++ used for data analytics instead of R, Python? I have started learning data science using R, however I have C++ as a subject this semester, and my project is to predict the outcome of a game using C++.
I have not come across many instances (close to none, I did find libraries like Shark though) of implementation in C++.

Is it to do with the fact that C++ isn't as simple to use when it comes to manipulating large amount of data?

Yes, you're correct -- it's that C and C++ are harder to use and are more burdened with boilerplate code that obfuscates your model building logic. When you build models, you have to iterate rapidly and frequently, often throwing away a lot of your code. Having to write boilerplate code each time substantially slows you down over the long run.  Using R's caret package or Python's scikit-learn library, I can train a model in just 5-10 lines of code.

Ecosystem also plays a big role. For example, Ruby is easy to use, but the community has never really seen a need for machine learning libraries to the extent that Python's community has. R is more widely used than Python (for stats and machine learning only) because of the strength of its ecosystem and its long history catering to that need.

It's worth pointing out that most of these R and Python libraries are written in low-level languages like C or Fortran for their speed. For example, I believe Google's TensorFlow is built with C, but to make things easier for end users, its API is in Python.


Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec) I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this:
S1 = ['Walk','Eat','Going school','Eat','Watching movie','Walk'......,'Sleep']
S2 = ['Eat','Eat','Going school','Walk','Walk','Watching movie'.......,'Eat']
.........................................
.........................................
S50 = ['Walk','Going school','Eat','Eat','Watching movie','Sleep',.......,'Walk']

The number of unique actions in the dataset are fixed. That means some sentences may not contain all of the actions.
By using Doc2Vec (Gensim library particularly), I was able to extract embedding for each of the sequences and used that for later task (i.e., clustering or similarity measure)
As transformer is the state-of-the-art method for NLP task. I am thinking if Transformer-based model can be used for similar task. While searching for this technique I came across the sentence-Transformer. But it uses a pretrained BERT model (which is probably for language but my case is not related to language) to encode the sentences. Is there any way I can get embedding from my dataset using Transformer-based model?

Yes you can get the embedding of words. In sentence-transformers you need to see where and how the embedding of words are extracted. Usually the sentence embedding in BERT is simple a max or mean over all word embeddings, so BERT does have word embeddings.
The question is if it helps you at all as, like you mentioned, at the end of the day you have a bunch of words not a language. The point is that if you have enough data you can fine-tune BERT (see sentence-transformers documentation) and probably capture the difference-similarity between each sample.
Another way is to get your embedding trough LSTM which is more suitable for such sequences and you can train it using for example Keras.
I suggest trying SGT as well and compare its embedding capability with other methods so you can choose the best.


GridSearchCV with Random Forest Classifier I'm working with a supervised learning problem and trying to predict a binary label and using a Random Forest to do so.
I'm trying to tune my hyper-parameters to give me a best model based on my data.
I can do this with GridSearchCV(), but is this correct to do with a random forest?
If I'm using GridSearchCV(), the training set and testing set change with each fold. From my understanding we can we set oob_true = True in RandomForestClassifier(), we are already evaluating on the out-of-bag samples (so CV is kind of already built in RF).
What is the convention to hyper-parameter tune with Random Forest to get the best OOB score in sklearn? Can I just loop through a set of parameters and fit on the same training and testing set? Can I use GridSearchCV(), or does that make no sense with RF?

You can definitely use GridSearchCV with Random Forest. In fact you should use GridSearchCV to find the best parameters that will make your oob_score very high.
Some parameters to tune are:
n_estimators: Number of tree your random forest should have. The more n_estimators the less overfitting. You should try from 100 to 5000 range.
max_depth: max_depth of each tree. You should specify certain max_depth so that your model don't memorise train examples.
min_sample_split: the minimum number of samples to have before splitting into new nodes.
and many more...
But these are the main hyperparameters we tune to get a forest that works well.And to get these hyperparams you should use GridSeachCV.


How to split data into 3 parts in Python - training(70%), validation(15%) and test(15%) and each part have similar target rate? I'm working on a company project which I will need to do data partition into 3 parts - Train, Validation, and Test(holdout).

Does anyone know how I can split the data into 3 parts above and each part will have similar response variable(target rate) - (similar accuracy for classification and similar mean of (response) for regression.

I know how to split data into 3 parts by using train_test_split function from SKLEARN

from sklearn.model_selection import train_test_split

x, x_test, y, y_test = train_test_split(xtrain,labels,test_size=0.2,train_size=0.8)
x_train, x_cv, y_train, y_cv = train_test_split(x,y,test_size = 0.25,train_size =0.75)

But this does not give a similar target rate, can someone help me?

Split to a validation set it's not implemented in sklearn.
But you could do it by tricky way:

1) At first step you split X and y to train and test set.

2) At second step you split your train set from previous step into validation and smaller train set.

X_train, X_test, y_train, y_test
    = train_test_split(X, y, test_size=0.7, random_state=123)

X_train, X_val, y_train, y_val
    = train_test_split(X_train, y_train, test_size=0.5, random_state=123)


How to create new feature based on clustering result I'm still a beginner in machine learning and I want to know how to code this situation based on python and machine learning (clustering).

I have data like:

id      Column1    duration(seconde)    column3
1       aaa        20                   bbb
2       ccc        01                   ddd
3       eee        150                  fff
4       ggg        25                   hhh

I want to group my data according to the duration column value and create new column containing a category name based on duration cluster.  I want to get this result:

id      Column1    duration(seconde)    column3      NewColCategorie
1       aaa        20                   bbb          Cat2
2       ccc        01                   ddd          Cat1
3       eee        150                  fff          Cat3
4       ggg        25                   hhh          Cat2
5       iii        175                  jjj          Cat3

To do clustering you can use sklearn's KMeans Clustering function - sklearn.cluster.KMeans with n_clusters=3 and other parameters as default. This will give you 3 clusters. After you have trained your model you can use the .labels_ attribute of the trained model to classify every example. You can do this in the following way:

>>> from sklearn.cluster import KMeans
>>> import numpy as np
>>> X = np.array([[1, 2], [1, 4], [1, 0],
...               [10, 2], [10, 4], [10, 0]])
>>> kmeans = KMeans(n_clusters=2, random_state=0).fit(X)
>>> kmeans.labels_
array([1, 1, 1, 0, 0, 0], dtype=int32)

To create a new column based on category cluster you can simply add the kmeans.labels_ array as a column to your original dataframe:

>>> df['categories'] = kmeans.labels_


Why is the correlation coefficient of a constant function with function input is not defined? Mathematically, in case of a constant function (f(x) = c) the correlation coefficient of a constant function with function input is not defined. Neverthless, from the function plot, we see that there is no correlation between the function input and output, so it must be returning 0.

The correlation coefficient between two random variables is a rigorously defined mathematical parameter. It is undefined when either of the random variables have zero variance. Therefore "NaN" is a very appropriate value to return in this case.

You are welcome to define it to be zero when one of the random variables is constant, if you find this more practical. But this does not agree with my mathematical intuition using limits. For example, if $X$ is $Uniform[-1,1]$, then the correlation coefficient between $X$ and $Y = c + \epsilon X$ is 1 for all $\epsilon > 0$, -1 for all $\epsilon < 0$. You can modify this example to get functions that get closer and closer to a constant function as $\epsilon$ goes to zero, while the correlation between X and its image is any number from the interval [-1,1].

I also like this explanation by David Epstein: the cosine of an angle between a non-zero and a zero vector is fundamentally undefined.


Loss function not working (RNN) I am building an RNN by following Siraj Raval's video implementation. I've adapted it to use my dataset rather than importing one from a file. When the program gets to the loss function, it reports
    loss += -np.log(ps[t][y[t], 0])
    IndexError: index 7 is out of bounds for axis 0 with size 4

What does this mean and how can i fix this? Also, in Siraj's version, he calculates the loss as:

loss += -np.log(ps[t][y[t], 0])

Shouldn't it be this instead

loss += -np.log(ps[t]) * [y[t], 0]

as the cross entropy loss is L = -yln(yhat)?

My code:

import numpy as np

# Data Processing
x = np.array([
    # t/no. of inputs
    [1, 2, 3, 4, 5, 6],
    [7, 8, 9, 10, 11, 12],     # Samples
    [13, 14, 15, 16, 17, 18],
    [19, 20, 21, 22, 23, 24]]).T

# Model Parameters
numInputs = x.shape[1] # Yields 4
numNeurons = 8 # Yields 8
numEntries = x.shape[0] # Yields 6

u = np.random.random((numNeurons, numInputs))
v = np.random.random((numInputs, numNeurons))
w = np.random.random((numNeurons, numNeurons))
bh = np.zeros((numNeurons, 1))
bo = np.zeros((numInputs, 1))
atimeline = [] # Contains 6 timestep's woth of a's
yhattimeline = [] # Contains 6 timestep's worth of yhat's
hprev = np.zeros((8, 1)) # Contains previous a state

# Training
def loss(x, y, hprev):
    xs, hs, ys, ps = {}, {}, {}, {}
    hs[-1] = np.copy(hprev) # Copies hprev so hprev can still be used
                            # Adds key: -1, value: hprev, to the dict
    loss = 0
    xs[0] = np.zeros((numInputs, 1)) # Sets t = 0 to 0's
    for t in xrange(numEntries):
        xs[t + 1] = x[t] # Adds in data from x to the dict
        hs[t] = np.tanh(np.dot(u, xs[t]) + np.dot(w, hs[t - 1]) + bh)
        ys[t] = np.dot(v, hs[t]) + bo
        ps[t] = np.exp(ys[t]) / np.sum(np.exp(ys[t])) # Softmax
        loss += -np.log(ps[t][y[t], 0])
    du, dv, dw = np.zeros_like(u), np.zeros_like(v), np.zeros_like(w)
    dbh, dbo = np.zeros_like(bh), np.zeros_like(bo)
    dhnext = np.zeros_like(hs[0])
    for t in reversed(xrange(numEntries)):
        dy = np.copy(ps[t])
        dy[targets[t]] -= 1 # Derived from dL/dyhat
        dv += np.dot(dy, hs[t].T)
        dbo += dy
        dh = np.dot(v.T, dy) + dhnext
        dhraw = (1 - hs[t] * hs[t]) * dh # tanh
        dbh += dhraw
        du += np.dot(dhraw, xs[t].T)
        dw += np.dot(dhraw, xs[t - 1].T)
        dhnext = np.dot(w.T, dhraw)
        return du, dv, dw, dbo, dbh, hs[numEntries - 1]

u, v, w, dbo, dbh, hprev = loss(x, x, hprev)

What happened was that

xs[0] = np.zeros((numInputs, 1))

Generates an array of ((4, 1)), and

xs[t + 1] = x[t]

Generates an array of ((1, 4)). To solve this, I transposed x[t] so now it is the same shape as above - ((4, 1)):

xs[t + 1] = np.array([x[t]]).T

Annoying bug and hard to solve, but a fairly easy solution.


Is empirical risk the same thing as loss function? I am reading the article Stochastic Gradient Descent Tricks by Léon Bottou (avaible here) and on the very first page they introduce empirical risk
$E_n(f) = \frac{1}{n} \sum_{i=1}^{n} l(f(x_i),y_i),$
where $l(f(x),y)$ is a loss function, that measures the cost of predicting $f(x)$ when the actual answer is $y$.
Then, there is written:
We seek the function $f \in \mathcal{F}$ that minimizes the loss
$Q(z,w) = l(f_w(x),y)$ averaged on the examples.
I fail to see the difference between empirical loss and the $Q$ loss function, can anybody provide explanation? Also I am not native speaker so there might be misunderstanding in "averaged on examples".

I found the accepted answer pretty hard to understand. Here is a simplified version that cleared it up for me:

Loss Function: a loss or risk function, $\mathcal{L}(\hat{y}^i, y^i)$, quantifies how well (more accurately, how badly) $\hat{y}$ approximates y. smaller values of $\mathcal{L}(\hat{y}^i, y^i)$ indicate that $\hat{y}$ is a good approximation of $y$

Empirical Risk: the empirical risk is the average loss over the data points. $$\mathcal{L} = \frac{1}{n}\sum_{i=1}^n \mathcal{L}(\hat{y}^i, y^i)$$

See these Stanford Lecture Slides for more details.


How to evenly distribute data to multiple GPUs using Keras I am using Keras=2.3.1 with Tensorflow-gpu=2.0.0 backend. While I trained model on two RTX 2080 ti 11G GPUs, it allocates all data to '/gpu:0',and nothing changed with '/gpu:1'. Surely, the second GPU not used at all.
However, every GPU could work if I selected only one GPU.
Moreover, the two gpus can be run parallelly in Pytorch.
Follow some instances, I try to run multi-gpus with these codes:

Below is NVIDIA-SMI output when I run a multi-gpus model.

and cuda = 10.1, cudnn = 7.6.5.

Check out the docs on TensorFlow GPU usage
If you wanted data parallelism where you run a copy of your model on multiple GPUs and split the data between them, you could use the tf.distribute.MirroredStrategy.
The tf.distribute.Strategy docs are also a good source to read.
Also, you should also profile your application; adding a second GPU has the potential to reduce performance depending on what your bottlenecks are.


Lambda Architecture - How to implement the Merge Layer / Query Layer I am reading up about lambda architecture.

It makes sense. we have queue based data ingestion. we have an in-memory store for data which is very new and we have HDFS for old data.

So we have our entire data set. in our system. very good.

but the architecture diagram shows that the merge layer is able to query both the batch layer and the speed layer in one shot.

How to do that?

Your batch layer is probably a map reduce job or a HIVE query. The speed layer query is probably a scala program which is execution on the spark.

Now how will you merge these?

Is there any guidance.

What you are asking about is, in my view, the main problem of implementing a lambda architecture.  Here are some suggestions on how to solve it.

The combination of Spark and Spark Streaming largely supersedes the original lambda architecture (which usually involved Hadoop and Storm).  Read here an example of how to use a SparkContext and a separate StreamingContext to produce different RDDs, one for batch processed results and another for real-time results.

Once you have replicated that in your system, you still have to think about how to query both kind of RDDs.  The trivial case would be to just union both of them:

scala> rdd1.union(rdd2).collect

Or maybe you can create a new DStream, similar to stateStream in the linked example, where some keys are kept for real-time results, and others for batch results.


A Neural Network That Learns Bitwise XOR I am trying to build a deep neural network that learns the coordinate-coordinate bitwise XOR of two matrices, but it performs poorly.

For example, in the 2 bits case, its accuracy stays around 0.5. Here is the code snippet:

from keras.layers import Dense, Activation
from keras.layers import Input
import numpy as np
from keras.layers.merge import concatenate
from keras.models import Model

size=1
data1 = np.random.choice([0, 1], size=(50000,size,size))
data2 = np.random.choice([0, 1], size=(50000,size,size))
labels  = np.bitwise_xor(data1, data2)
a = Input(shape=(size,size))
b = Input(shape=(size,size))
a1 = Dense(size, activation='sigmoid')(a)
b1 = Dense(size, activation='sigmoid')(b)
merged = concatenate([a1, b1])
hidden = Dense(1, activation='sigmoid')(merged)
hidden = Dense(3, activation='sigmoid')(hidden)
hidden = Dense(5, activation='relu')(hidden)
hidden = Dense(4, activation='sigmoid')(hidden)
hidden = Dense(3, activation='sigmoid')(hidden)
outputs = Dense(1, activation='relu')(hidden)

model = Model(inputs=[a, b], outputs=outputs)
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit([data1, data2], np.array(labels), epochs=15, batch_size=32)

What's going on here?

Epoch 1/15
50000/50000 [==============================] - 7s 130us/step - loss: 0.7118 - acc: 0.5044
Epoch 2/15
50000/50000 [==============================] - 4s 78us/step - loss: 0.6933 - acc: 0.5023
Epoch 3/15
50000/50000 [==============================] - 4s 74us/step - loss: 0.6934 - acc: 0.5030
Epoch 4/15
50000/50000 [==============================] - 4s 86us/step - loss: 0.6935 - acc: 0.5002
Epoch 5/15
50000/50000 [==============================] - 4s 79us/step - loss: 0.6934 - acc: 0.5015
Epoch 6/15
50000/50000 [==============================] - 5s 96us/step - loss: 0.6935 - acc: 0.5030
Epoch 7/15
50000/50000 [==============================] - 5s 105us/step - loss: 0.6934 - acc: 0.5026

I think there might be a few things going on.

You might have a reason but I don't know why you have shaped your input data into three dimensions: size=(50000,size,size).

Also, you might have a reason but I don't know why you ran each feature separately through a different layer (each with a single hidden unit), and then merged the outputs before running the merged output through another series of layers:

a = Input(shape=(size,size))
b = Input(shape=(size,size))
a1 = Dense(size, activation='sigmoid')(a)
b1 = Dense(size, activation='sigmoid')(b)
merged = concatenate([a1, b1])

Also, I suspect that running the features through a single hidden unit reduces the information sent through the rest of the network, so the network cannot learn the XOR function.

Here is some code that works for me:

from keras import models

from keras.layers import Dense

import numpy as np

Simulate data:

X_1 = np.random.choice([0, 1], size = (50000, 1))
X_2 = np.random.choice([0, 1], size = (50000, 1))

X = np.concatenate((X_1, X_2), axis = 1)

Y = np.bitwise_xor(X[:, 0], X[:, 1])

FNN Model:

# Define model.

network_fnn = models.Sequential()
network_fnn.add(Dense(4, activation = 'relu', input_shape = (X.shape[1],)))
network_fnn.add(Dense(4, activation = 'relu'))
network_fnn.add(Dense(1, activation = 'sigmoid'))

# Compile model.

network_fnn.compile(optimizer = 'rmsprop', loss = 'binary_crossentropy', metrics = ['acc'])

# Fit model.

history_fnn = network_fnn.fit(X, Y, epochs = 5, batch_size = 32, verbose = True)


Ranking ATM based on Utilization and Economic Data (Scoring/Rank Model) I have a sample data of around 10 ATM's Locations along with their Utilization Count (Deposits, Withdrawals and Others) for the past 3 months. I am planning to collect additional data such as nearby places of Commercial Interest and Others where there might be demand of Cash. The data is collected for approximately 300 meters of each ATM, i.e., places of Commercial Interest nearby the ATM.

I would like to build a 'Scoring/Rank Model' which can take all these inputs into consideration and Rank these 10 ATM based on Utilization and places of economic interest.

As much I have searched Google, I could not find any answers which can provide a solution. I am pretty new to Data Science and any inputs/suggestion will be really valuable.

If you want to use a prediction model, then you need a well-defined target. In your case, the Utilization of an ATM is a vague term. You cannot measure it as it is right now. If you solve this problem, then what you are looking at is algorithms called Ranking Methods or Learning-to-Rank

Abstract from Wikipedia

  Learning to rank or machine-learned ranking (MLR) is the
  application of machine learning, typically supervised, semi-supervised
  or reinforcement learning, in the construction of ranking models for
  information retrieval systems. Training data consists of lists of
  items with some partial order specified between items in each list.
  This order is typically induced by giving a numerical or ordinal score
  or a binary judgment (e.g. "relevant" or "not relevant") for each
  item. The ranking model's purpose is to rank, i.e. produce a
  permutation of items in new, unseen lists in a way which is "similar"
  to rankings in the training data in some sense.

Let's move to another working domain to make the example easier to understand. The example was taken from TowardsDataScience tutorial page.

You have an e-commerce shop and you want to rank your products with that way that will be sorted on a search page and maximize your revenue. The features you have are the attributes of the product and the target is if the visitor bought or not on that session.

You train then a Classifier (LogisticRegression for instance) and get the probability of prediction for that class as the ranking.


update model prediction for each day closer to the event I want to predict whether people will renew their yearly subscription.  I want to make this prediction though for each user on every day of their subscription up until the day before the subscription ends (i.e., make predictions for each user on day 1, 2, 3... 364).

A simple approach is to obtain training data by randomly sampling users at some point in their history (e.g., look at information we knew about user 1 when they were 300 days away from renewal; look at what we knew about user 2 when they were 23 days away from renewal, etc) and then build a model with a "days away" variable.

One issue with this approach is that unless you have a ton of data, the model won't perform as well on any given "days away" (e.g., 30 days before) than a model in which all data was trained with same "days_away" (all users data was limited to what we knew about them exactly 30 days before they were up for renewal).

I'm curious know about other people's thoughts about how to best train a model(s) that can update its prediction each day as the prediction-event gets closer in time. Are there better ways to approach it than the approach outline above?

I would rephrase the problem into a business problem. I.e., What we are trying to accomplish here.

From the sound of it, it seems like a Churn prediction problem. The article outlines some ideas and potential solutions.

In simpler terms, if the goal is to prevent churn by executing the right marketing messages, etc.

How can we predict the risk of someone leaving? Since you have historic data, it can be a supervised learning algorithm to help you identify such risk.
What other features do we want to add to the model? Actions / usage data / web visitation / email open, etc. leading to the renewal days. You can normalize the data by the days prior to renewal, etc.
Determine what actions you want to take; it will help you determine if you want to get a risk score, or a classifier with differing level of risk, etc.


Error calculation in Logistic Regression Suppose we have a linear regression model that predicts an item’s price. If the item’s prediction is 8 USD and the actual value is 10 USD, then it is clear that the error is pow(10-8, 2)=4.
But how is the error calculated when there are more than two classes?

For example, the model that predicts the numbers trained on the MNIST dataset. In this case we have 10 labels that mark the actual values - range between 0-9. But if we use the sigmoid function for the activation then the range for possible predicted values is between 0-1. Right? How can we compare these values if they are on a different scale? For example, the sigmoid function outputs the values 0.5 and we have to compare it to 3?

The problem here is that you are comparing a regression problem with a classification problem. Classifying MNIST digits is treated as a classification task because it is assumed that predicting the wrong digit is always bad, regardless of how "close" you are to the right answer. One could even argue that predicting a 9 instead of a 3 is a better guess than predicting 5 for instance.

If you want really want to compare the values, I would simply pick the class with the highest value and use this as the regression prediction.


Showing standard deviation for training curve I am training a neural network and I wanted to plot the evolution of different metrics (MSE…) during training. To get an idea of the variations between between different trainings, I am using several models and plotting average value and standard deviation. My problem is the following: I did not manage to find a good way to plot this curve and also did not find any good explanation on internet.
Let's denote by y the metric to be displayed, and x is the number of epochs. On the plots below, one standard deviation is given by the shaded area.
Here is what I tried and why I think it is not working:

A linear scale is not great because the metric spans different order of magnitude. In particular, the early values are much larger than late values. One possibility could be to start the curve at epochs 5, for example, to remove the early large values, but I would prefer to display everything.

A log scale seems much more natural in view of the previous point, however variations can be large, of similar scale compared to the scale of the mean value. Since the space between two ticks is geometrically distributed, one standard deviation below the curve appears much bigger than above. This tends to produce large spikes on the figure, or even completely fill the area below the curve.

I have tried to display the relative standard deviation instead (rescaled by $\log_{10} \mathrm e$). This is the appropriate standard deviation for the variable $z = \ln y$. However, I dont want to plot z but really y in log scale. Nonetheless, I thought it's fine to plot the relative standard deviation if it displays better. But, I am not sure if it's really intuitive what it shows. A worse problem is that the standard deviation appeard very large for small values (since we divide by a number < 0).

Finally, I tried to use the symlog scale but I am not sure it's a good solution: if the linear threshold is large, then I find that that it makes the figure difficult to interpret. If it is small, we get the same problems as for the log scale. Below is a plot for a threshold of 0.01.

Conclusion: what would be the best way to display the standard deviation a log scale?

One option is to plot fewer epochs. There is no useful information after 20 epochs because training and validation performance are the same after that point.


XGBoost for binary classification: choosing the right threshold I am working on a highly-imbalanced binary-labeled dataset, where number of true labels is just 7% from the whole dataset. But some combination of features could yield higher than average number of ones in a subset.

E.g. we have the following dataset with a single feature (color):

  180 red samples —  0

  20 red samples — 1

  300 green samples — 0

  100 green samples — 1

We can build a simple decision tree:

                      (color)

                red /       \ green

 P(1 | red) = 0.1              P(1 | green) = 0.25

  P(1) = 0.2 for the overall dataset

If I run XGBoost on this dataset it can predict probabilities no larger that 0.25. Which means, that if I make a decision at 0.5 threshold:

0 - P < 0.5
1 - P >= 0.5

Then I will always get all samples labeled as zeroes. Hope that I clearly described the problem.

Now, on the initial dataset I am getting the following plot (threshold at x-axis):

Having maximum of f1_score at threshold = 0.1. Now I have two questions:

should I even use f1_score for a dataset of such a structure?
is it always reasonable to use 0.5 threshold for mapping probabilities to labels when using XGBoost for binary classification?

Update. I see that topic draws some interest. Below is the Python code to reproduce red/green experiment using XGBoost. It actually outputs the expected probabilities:

from xgboost import XGBClassifier
from sklearn.model_selection import train_test_split
import numpy as np

X0_0 = np.zeros(180) # red - 0
Y0_0 = np.zeros(180)

X0_1 = np.zeros(20) # red - 1
Y0_1 = np.ones(20)

X1_0 = np.ones(300) # green - 0
Y1_0 = np.zeros(300)

X1_1 = np.ones(100) # green  - 1
Y1_1 = np.ones(100)

X = np.concatenate((X0_0, X0_1, X1_0, Y1_1))
Y = np.concatenate((Y0_0, Y0_1, Y1_0, Y1_1))

# reshaping into 2-dim array
X = X.reshape(-1, 1)

import xgboost as xgb

xgb_dmat = xgb.DMatrix(X_train, label=y_train)

param = {'max_depth': 1,
         'eta': 0.01,
         'objective': 'binary:logistic',
         'eval_metric': 'error',
         'nthread': 4}

model = xgb.train(param, xg_mat, 400)

X0_sample = np.array([[0]])
X1_sample = np.array([[1]])

print('P(1 | red), predicted: ' + str(model.predict(xgb.DMatrix(X0_sample))))
print('P(1 | green), predicted: ' + str(model.predict(xgb.DMatrix(X1_sample))))

Output:

P(1 | red), predicted: [ 0.1073855]
P(1 | green), predicted: [ 0.24398108]

You have to decide what you want to maximize.

Classifying by comparing the probability to 0.5 is appropriate if you want to maximize accuracy.  It's not appropriate if you want to maximize the f1 metric.

If you want to maximize accuracy, always predicting zero is the optimal classifier.

Alternatively, given a probability score $p$, another option is to randomly flip a biased coin; with probability $p$, output classification 1, otherwise output classification 0.  This doesn't always predict zero.  However it's probably not actually any better in any useful way.

If you want to maximize f1 metric, one approach is to train your classifier to predict a probability, then choose a threshold that maximizes the f1 score.  The threshold probably won't be 0.5.

Another option is to understand the cost of type I errors vs type II errors, and then assign class weights accordingly.


Same probability for all classes I implemented a fully connected MLP of shape [783 (input), 128 (hidden layer) and 10 (output)] the hidden layer had a sigmoid activation function and the output a sofmax.

I tested with the dataset of keras: Classify images of clothing.

At first I got the ouput was 0.1 at all the exits not matter the input. I then read this and because someone asked about the weights initialization I changed my weights initialization from a normal distribution between [0, 1) to [-1, 1). After that my network started working.

Why did this happen? I believe the prection of 0.1 is some kind of local minima because it just says the same probability for all, at least is what makes sense if you knew nothing about the data. But why? I would love to be refered to a paper that talks about this issue and how to prevent it because I am trying with another dataset now and I got the same problem (but this time I could not make it work. I even try Xavier initialization and still no good result).

Assuming that you normalized the pixel-values as it is done in the tutorial, your inputs were vectors of numbers betweeen 0 and 1. Now, if your weight matrices are also randomly drawn numbers between 0 and 1, the input to the hidden layer will be sums of 783 numbers between 0 and 1, i.e. probably something > 100. Now, check out the sigmoid function and its derivative

As you see, it saturates quite quickly for input values > 5. If you choose the initialization as you did, all hidden neurons should be very close to one, while at the same time the derivative should be very close to zero. This would explain how all outputs of the softmax were equally close to 0.1 and since the gradient was close to zero, the network probably didn't learn anything.

Once you changed the weight initialization to something between -1 and 1, the inputs to the hidden layer should be sums of numbers fairly evenly distributed around 0, thus the sigmoid output was about 0.5 and, most importantly, the gradient was non-zero so your network actually got trained.

As you already noticed, choosing the initialization wisely is crucial for getting decent results. Initializations should also take the number of input neurons into account, otherwise you initialize the network in such a way, that the gradients will be close to zero.

A similar problem can occur if you do not normalize the input data properly, e.g. if you feed the pixel-values between 0 and 255 directly to the model.

I'm not sure about papers on this, but maybe start with the original work on Glorot initialization or checkout the initializers Tensorflow has to offer. On that site under "Functions" they list common initializers and also link to the respective papers.

I hope this was helpful.


How to train a Machine Learning model for blocked data I'm concerned with a supervised classification problem for the following type of data. The data consists of $N$ rows (where $N$ is not very large - this is not a big-data problem) and $M$ columns (features) and each row has a certain label I'm interested in. Every row belongs to a certain block and each block consists of 1-50 rows. The size of a block depends on the duration of a block (so the rows in each block are correlated, but the correlation between different blocks can be neglected).

The aim is to learn a classification algorithm on the data that allows to classify a new row or a new block.

Now there are two important things: The labels are constant within each block and there are some features that allow the identification of blocks.

My question is: What might be the best way to learn a model on this specific type of data?

To illustrate this problem a bit further, let me report some problems I found when I worked with Random Forests on the data set.

Suppose I take a part of the data as validation data, which contains whole blocks. If I split the remaining rows randomly in training and test data, the accuracy of the learned Random Forest is very high on training and test data but very low on the validation data. This is due to the fact that the Random Forest learns in this case to identify the individual blocks and as the validation data contains some unknown blocks, the accuracy drops. This can be also seen in the resulting importance for the features: the most important ones are those features that allow an identification of a cycle very easily.

Splitting the remaining data into training and test data and leaving the blocks whole helps a bit but doesn't fix the problem completely.

Another approach would be to remove all features that allow the identification of blocks. But this is difficult a priori and some of these features could have important information for my problem.

A simple way to overcome the problems would be to take the row mean in each block and use the resulting data for the classification problem. However, in this case you loose a lot of information.

So I'm wondering if there are more natural ways to approach this classification problem which respect the block-structure of the data.

I think your approach is more or less valid, but you’re overfitting the model because you don’t have a lot of data. So that’s my main point.

There are many ways in principle to deal with this, but if you have little data, perhaps you simply shouldn’t be expecting great results. If you use the averages, you get better results, but “the problem isn’t solved completely”. A priori I’m not sure you have reason to think you can do better.

Anyway, step 1 should be to simplify the model (fewer and more shallow trees). You should find that your test error deteriorates but your validation error improves, because rather than simply memorizing records, the model is now forced to learn something about the blocks. This information can be reused on records the model hasn’t seen yet.

You can also look at regularization, but it is essentially an automatic way of finding a simpler, but better model. Also, data augmentation. Is it possible to generate more data by varying the existing data a bit? Like the way you can generate more images by changing the images that you have: By cropping, flipping, rotating, etc.

Finally, make sure that your split between train-test-validate sets is random. I guess it depends on your data but it seems you can’t expect the model to learn about a block that doesn’t exist at all in the training set. If that block appears in the validation set only, I guess the model would be random on that data.


Prevent overfitting when decreasing model complexity is not possible I'm fairly new to machine learning and as an exercise for a more complicated task, I'm trying to do the following what I thought was a trivial task. Suppose as an input I have population density maps. These are 2D images with one layer, in which each pixel is the count of persons living in that area.
From that data, I'd like my model to "estimate" (in fact it would be possible to calculate the exact solution) the total number of persons living on that density map. Essentially, the task consists of just taking the sum of the 2D input.
I have tried many architectures and I found that the simpler the better. In fact a model containing no hidden layers performed best:

from keras.layers.core import Dense
from keras.layers import Flatten, Input

inputs = Input(shape=(225, 350, 1))
x = Flatten()(x)
x = Dense(1)(x)

While this performs very well on the training data, it fares very poorly on the validation data. I know that this is a sign of overfitting, but how can I prevent overfitting given that is not possible to further decrease complexity of the model? Or would another approach / architecture be better altogether?
Note that I have performed the usual data pre-processing (normalising inputs and outputs).

Thanks in advance for any hints.

is this about trying to have the model learn the sum function? Because if it is you can always initialize the weights to be 1 and then make the entire model untrainable.
If all weights are one and the model has linear activations it will just compute the sum of your inputs. If its important to you to train the model to do this I would add a Dropout layer (?).

Your model is so simple it can be written down in closed form as it is $\sum_{i=0}^{255*350}w_i x_i$ and therefore I would just set the $w_i$ to be 1.

(In general initializing the weights to 1, the ideal solution, should make training very fast)


What model do I use to predict a regression problem with timeseries data Overall Goal
To predict how much reagent "A" I started with in a reaction.

Data:
To predict this I have timeseries data of reagent "B". For each time step a measurement of reagent "B" is taken (the amount of reagent "B" present at that time point). The overall timeseries is a curve. This curve may change based on how much reagent "A" I start with.

My question is what model should I use to predict reagent A? Will a Recurrent NN work? I have only seen RNN used in predicting the very next time steps or to classify something based on the timeseries. I am looking for the model to use time series data to predict a regression problem.

[Updated]

If you have measured Reagent B for same number of time steps for each experiment, then:

Using these time steps as feature values, you can build a regression model using, Multi Layer Perceptions, XGBoost/LGBM/RandomForests etc.
Even Recurrent Neural Nets could work if enough data is available. Treat your time step values of Reagent B as single dimention embedding values and use BiLSTM architecture with Linear activation function in the last layer. Loss could be mean squared error. BiLSTM could work in your case as it makes sense to read from last time step to first time step also

If reagent B is measured for different time steps, in different experiments:

You can still use Ensemble Trees/MLP/RNN based methods, but you will have to consider adding paddings(imputing) or limiting max time steps to make number of time steps uniform across all experiments.

XGB/LGBM can take missing values as inputs and handle it well, so you dont need to do any imputation.


How to decide if gradients are vanishing? I am trying to debug a neural network. I am seeing gradients close to zero. How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing gradient by looking at the values? I am getting values close to 4 decimal places(0.0001) and in some cases close to 5 decimal places (0.00001). The network seems not to be learning since the histogram of weight is also quite similar in all epochs. I am using RELU activation and Adam optimizer. What could be the reason for the vanishing gradient in case of RELU activation? Is possible please point me to some resources that might be helpful. Thanks in advance.

Vanishing gradients are the updates to the weights (not the weight themselves) over the layers after each training batch.
You should check the update signal between the top-most layer and lowest layer after a single batch.


Eigenfaces VS Deep neural network for classification I just saw a video from Washington University where professor Steve Brunton explain how to use Eigenfaces for classfication e.g image recognition.

Eigenfaces is really simple and does not require any training or parameter tuning etc. He show that it took just few seconds and then it's done. Super fast.

Eigenfaces was solved by using Singular value decomposition.

$$A = USV^T$$
Eigenfaces classification

So what is best if I want to have some kind of simple image classification?
Should I spend time and energy to train a deep neural or should I focus on eigenfaces? Eigenfaces is legacy, but it's still not bad, right?

Is it easy to train a deep neural network with multiple pictures? Or is that a hard task? As I see other people doing deep learning, they just cook up someting and then they have solved world problems. Sounds too good to be true right?

In what situation do you prefer eigenfaces over deep neural network? No?

As it stands, neural networks outperform the Eigenfaces approach. With this I mean that neural networks can solve a large range of problems with more accuracy.

Eigenfaces are nice because they can work already with a small amount of training samples, specially compared to neural networks that are known to be data intensive. So, in a small amount of data setting, you could start with Eigenfaces for feature extraction and pair it up with an SVM for classification. However, now a days, there are pre-trained neural network models (e.g. Facenet), which offer pre-trained embeddings for you to use for feature extraction. Then, all you need to do is train "a last layer" for classification. Bottom line is, you don't need to start from scratch, making it obsolete to use Eigenfaces for feature extraction.


First steps when analyzing a company's data I'm not sure if this question is appropriate for this forum, so excuse me if it's not (if not, any suggestions on where might be a better place would be very much appreciated). I'm currently an undergrad in a quantitative field, and for the summer, I've been given an opportunity to do a data project by the company I am working for. I'm not really sure where to start. I had a conversation with one of the business owners today, in order to get a better handle on how the business works, and what kind of data they have. We talked a little bit about what sorts of questions they have, and what sort of things would be nice to know. I guess that seems to be the main question: What questions to ask? My initial thoughts are to first just look at the data via traditional descriptive stats methods (histograms, scatter plots etc....), and maybe that creates some ideas. If anyone has some tips, or even some good links (yes, I have already Googled it quite a bit), I would be grateful. Thanks.

The thing you need to understand as completely as possible is how they expect a data analysis to enable them to achieve their objective. They are a business, so their overall objective is likely related to maximising profit. However, there will be a more immediate objective underneath that heading. To maximise profit you can either reduce costs or increase sales. In turn, to increase sales you can increase the number of customers or increase the amount of sales to each customer etc.

The question then turns on how you can use data science to perform one those objectives.

For example, questions that can almost be answered with data science could be 'how do I better identify potential customers?' or 'how do increase existing custmers' spend?' These are still very high level questions, but they are the sort of questions that you need to have in mind as you start to do your descriptive stats etc.

Bear in mind that this is an iterative process and it is completely normal to start off in a fuzzy sort of area. At this stage it is almost the case that having a question in mind is a McGuffin - it will kick things off, but it may not be the question you end up answering.

The CRISP-DM process is a process that has been built for data mining that discusses how to iteratively use results from analyses and models to increase your understanding of the customer's situation, and hence drive the development of a better business objective for use in a data science project.


Evaluating a trained Reinforcement Learning Agent? I am new to reinforcement learning agent training. I have read about PPO algorithm and used stable baselines library to train an agent using PPO. So my question here is how do I evaluate a trained RL agent. Consider for a regression or classification problem I have metrics like r2_score or accuracy etc.. Are there any such parameters or how do I test the agent, conclude that the agent is trained well or bad.

Thanks

If your goal is optimal control, then you will want to measure the agent by how well it does at the task. You should use some aggregate measure of reward, such as total reward per episode (aka "return"), or mean reward per time step.

If you are working with a toy problem, or one designed so that it is easy to identify a maximum bound on the reward-based measure, then you can compare your agent against this known value. It is reasonable to expect a good agent to approach the maximum value.

In practice, many interesting problems do not have a known upper bound on reward totals or averages. For those problems, typically the best you can do is compare between agents. You can compare with:

A randomly acting agent. This would normally be just as a baseline, to show that the agent had learned something.
An automated agent using a simple action choice heuristic, which might be something natural or obvious in the given problem.
One or more humans on the same task
Other ML-trained agents including previous instances of the same agent

You will probably want to run multiple tests and average the results, if either the policy or environment are stochastic, in order to assess an agent with expected values as much as possible. It is also important, if you are using any off-policy technique such as DQN, to switch off any exploration during tests, to get a fair measure of how well the trained agent behaves (as opposed to how it behaves during training, which will be different).

If your agent is designed to continually learn and explore, and/or uses an on-policy approach, you can use results during training to assess it. For instance you can take a rolling average of total reward over the last N episodes or something similar. This is not a bad metric to monitor training, even for off-policy approaches, although for off-policy you will likely get an underestimate of performance compared to separate test runs.

There are other approaches and other metrics to assess an agent - e.g. how much experience, or how much computation the agent needs to learn to a certain level is often of interest.

If you want to "conclude that the agent is trained well or bad" for an optimal control task, then this assessment of total reward might be all you need. However, you can also look at loss metrics inside any neural networks - you would not do this in order to rank agents as "better" or "worse" but might do so in order to identify problems. These loss metrics are generally the same as supervised learning equivalents. So for instance in DQN, or for the "critic" part of PPO, you would be interested in whether the predicted value of any state matched the eventual value, and use MSE loss. For stochastic environments there is a caveat to this, that any collected data will be noisy so it is hard to tell the difference between a high loss due to variance in the data and high loss due to poor training or incorrect hyperparameters.


How do I resolve this error: "expected string or bytes-like object" please? I was doing Lemmatisation but I got the above error. I think it's because my data isn't in string form, is that correct? If so, which part of my data isn't in string form?

This is my data source: https://www.kaggle.com/c/fake-news/data

You are applying that function to a column in a panda's dataframe. Double check the datatype of the column:
lemmatised_df['label'].dtype


Why is TF IDF output lognormal? I ran a TF IDF algorithm and the result of predicted similarities using cosine similarity is a log-normal distribution.  Is this a feature of the algorithm (e.g., all logit probabilities are log-normal), or might it be a result in my data?

Thanks.

Let’s look at the formula used for the calculation of tf-idf scores

You can see a log with the idf term.
The following answer explains this pretty well.
log with idf


Using GridSearchCV and a Random Forest Regressor with the same parameters gives different results As the huge title says I'm trying to use GridSearchCV to find the best parameters for a Random Forest Regressor and I'm measuring my results with mse.

Inputs_Treino = dataset.iloc[:253,1:4].values
Outputs_Treino = dataset.iloc[:253,-1].values
Inputs_Teste = dataset.iloc[254:,1:4].values
Outputs_Teste = dataset.iloc[254:,-1].values

estimator = RandomForestRegressor()
para_grids = {
            "n_estimators" : [10,50,100],
            "max_features" : ["auto", "log2", "sqrt"],
            "bootstrap"    : [True, False]
        }

grid = GridSearchCV(estimator, para_grids, scoring = 'mean_squared_error')
grid.fit(Inputs_Treino, Outputs_Treino)
forest = grid.best_estimator_

reg_prediction=forest.predict(Inputs_Teste)

print (grid.best_score_, grid.best_params_)

mse = mean_absolute_error(Outputs_Teste, reg_prediction)

This is the gist of the code (nothing too complex I know, just getting started with it all)

When I print the result of grid.best_estimator_ I get this

RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,
           max_features='auto', max_leaf_nodes=None,
           min_impurity_split=1e-07, min_samples_leaf=1,
           min_samples_split=2, min_weight_fraction_leaf=0.0,
           n_estimators=10, n_jobs=1, oob_score=False, random_state=None,
           verbose=0, warm_start=False)

The problem is if I try to create a regressor with these parameters (without using grid search at all) and train it the same way I get a waaaay bigger MSE on the testing set (5.483837301587303 vs 43.801520165079467)

Inputs_Treino = dataset.iloc[:253,1:4].values
Outputs_Treino = dataset.iloc[:253,-1].values
Inputs_Teste = dataset.iloc[254:,1:4].values
Outputs_Teste = dataset.iloc[254:,-1].values

regressor = RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,
           max_features='auto', max_leaf_nodes=None,
           min_impurity_split=1e-07, min_samples_leaf=1,
           min_samples_split=2, min_weight_fraction_leaf=0.0,
           n_estimators=10, n_jobs=1, oob_score=False, random_state=None,
           verbose=0, warm_start=False)

regressor.fit(Inputs_Treino,Outputs_Treino)

#fazer as predictions
Teste_Prediction = regressor.predict(Inputs_Teste);

mse = mean_squared_error(Outputs_Teste, Teste_Prediction);

Does this have to do with the cross validation GridSearchCV performs ? What am I missing here ?

RandomForest has randomness in the algorithm. First, when it bootstrap samples the data for each tree. Second, when it chooses random subsamples of features for each split.

To reproduce results across runs you should set the random_state parameter. For example:

estimator = RandomForestRegressor(random_state=420)