How to find "regions" with high purity I am applying a ML model (LGBM binary classifier) to data and would now like to identify the part of data where I have a low ratio of false-negatives (false-postives are not such a problem) and as much as possible true-negatives.

Background

The data I am classifying is from a system, that includes many complex rules that decide whether or not for some cases the company pays goodwill/warranty. In many of the cases this rule based system already finalizes the decision and initiates the payments. However there are cases, where this rule based system isn't able to finalize a decision but instead forwards the case along with a proposed decision to a manual decision process. It seems that this often also happens, because the rules (which are highly configurable) are not sharp enough or because the price-depending parts are not adjusted to inflation etc.

Now in this configuration it turns out, that the manual decision process very often just accepts the proposed decision. I am working on training a ML model (the LGBM classifier) to identify the cases, where the manual decision process leads to just accepting the proposed decision without changeing it.

Motivation

I am only interested in identifying cases where the proposed decision made by the rule system can just be taken over, because I'd like to reduce the amount of decisions that have to be checked manually.
My model currently gets a MAE of around 0.8 at the moment.

I'd now like to find a segment of the data, where the ratio of false negatives (the decisions where the model states, they need not change but in fact the descider changed them) is smallest (or below a certain threshold --> say 0.02).

Question

Can anybody suggest, how I could find such a region with a low ratio of false-negatives (false negatives in region by number of samples in the region)?
I thought about a simple decision tree, but would like to know if there are better options.

E.g. I suspect, that in case a leaf contains a high percentage of say 95% class 1 samples and 5% class 0 and some condition would allow to split it in two regions with 98% class 1 and 92% class 1 it wouldn't perform that split, since both would be classified as class 1 anyways, or am I wrong here?

A way to explore your dataset would be to use different techniques and see how they preform. The first approach I would consider is a tree based learner of modest complexity (or at least not boosted), like C4.5. It's possible (or perhaps likely) that won't perform as well as some more powerful approaches, but it has the added benefit of being relatively readable.

If you are familiar with a decision tree visualisation, you'll know that you can identify (leaf)nodes with high(er) impurity, and a large(r) population quite quickly. You can find an example of such a visualisation here. Those nodes can help you understand your domain, and possible how to improve performance (For instance by boosting, something LGBM already does for you).

If you want to optimise the LGBM straight away, another thing that could help here is to cluster your dataset (kMeans could be a good starting point) and then calculate performance metrics per cluster. Possibly you could also consider clustering the scored items, including the classification, or even the evaluation (TP, FP, TN, FN). If used right that could help identify clusters of false negatives for instance, and the commonality they share. Disclaimer: I have never done this myself.

PS: Lot's of times I find it helpful to use KNN (over a subsample of the feature space) and Naive Bayes. They can quickly tell you something about the global correlations. Other golden oldies are feature importance rankers and a correlation plot.


NLP: What are some popular packages for multi-word tokenization? I intend to tokenize a number of job description texts. I have tried the standard tokenization using whitespace as the delimiter. However I noticed that there are some multi-word expressions that are splitted by whitespace, which may well cause accuracy problems in subsequent processing. So I want to get all the most interesting/informative collocations in these texts.

Is there any good packages for doing multi-word tokenization, regardless of specific programming language? For example, "He studies Information Technology" ===> "He" "studies" "Information Technology".

I've noticed NLTK (Python) has some related functionalities.

collocations Module:
http://www.nltk.org/api/nltk.html#module-nltk.collocations
nltk.tokenize.mwe module:
http://www.nltk.org/api/nltk.tokenize.html#module-nltk.tokenize.mwe

What the difference between these two?

The MWETokenizer class in the nltk.tokenize.mwe module seems working towards my objective. However MWETokenizer seems to require me to use its construction method and .add_mwe method to add multi-word expressions. Is there a way to use external multi-word expression lexicon to achieve this? If so, are there any multi-word lexicon?

Thanks!

The multiword tokenizer 'nltk.tokenize.mwe' basically merges a string already divided into tokens, based on a lexicon, from what I understood from the API documentation.

One thing you can do is tokenize and tag all words with it's associated part-of-speech (PoS) tag, and then define regular expressions based on the PoS-tags to extract interesting key-phrases.

For instance, an example adapted from the NLTK Book Chapter 7 and  this blog post:

def extract_phrases(my_tree, phrase):
   my_phrases = []
   if my_tree.label() == phrase:
      my_phrases.append(my_tree.copy(True))

   for child in my_tree:
       if type(child) is nltk.Tree:
            list_of_phrases = extract_phrases(child, phrase)
            if len(list_of_phrases) > 0:
                my_phrases.extend(list_of_phrases)

    return my_phrases

def main():
    sentences = ["The little yellow dog barked at the cat",
                 "He studies Information Technology"]

    grammar = "NP: {<DT>?<JJ>*<NN>|<NNP>*}"
    cp = nltk.RegexpParser(grammar)

    for x in sentences:
        sentence = pos_tag(tokenize.word_tokenize(x))
        tree = cp.parse(sentence)
        print "\nNoun phrases:"
        list_of_noun_phrases = extract_phrases(tree, 'NP')
        for phrase in list_of_noun_phrases:
            print phrase, "_".join([x[0] for x in phrase.leaves()])

You defined a grammar based on regex over PoS-tags:

grammar = "NP: {<DT>?<JJ>*<NN>|<NNP>*}"
cp = nltk.RegexpParser(grammar)

Then you applied it to a tokenized and tagged sentence, generating a Tree:

sentence = pos_tag(tokenize.word_tokenize(x))
tree = cp.parse(sentence)

Then you use extract_phrases(my_tree, phrase) to recursively parse the Tree and extract sub-trees labeled as NP. The example above would extract the following noun-phrases:

Noun phrases:
(NP The/DT little/JJ yellow/JJ dog/NN) The_little_yellow_dog
(NP the/DT cat/NN) the_cat

Noun phrases:
(NP Information/NNP Technology/NNP) Information_Technology

There is a great blog post by Burton DeWilde about many more ways to extract interesting keyphrases: Intro to Automatic Keyphrase Extraction


Resampling with Python SMOTE I am trying to do a simple ML re-sampling approach after the train-test split. However when I do this, it throws the below error. Can you please help me understand what this error is about?

KeyError: 'Only the Series name can be used for the key in Series dtype mappings.'

The code is given below:

# split into training and testing datasets
from sklearn.model_selection import train_test_split
from sklearn.utils import resample
from imblearn.over_sampling import SMOTE
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.30, random_state = 2, shuffle = True, stratify = y)
print("Number transactions X_train dataset: ", X_train.shape)
print("Number transactions y_train dataset: ", y_train.shape)
print("Number transactions X_test dataset: ", X_test.shape)
print("Number transactions y_test dataset: ", y_test.shape)
print("Before OverSampling, counts of label '1': {}".format(sum(y_train==1)))
print("Before OverSampling, counts of label '0': {} \n".format(sum(y_train==0)))

sm = SMOTE(random_state=2)
X_train_res, y_train_res = sm.fit_sample(X_train, y_train.ravel())   # error is thrown here

print('After OverSampling, the shape of train_X: {}'.format(X_train_res.shape))
print('After OverSampling, the shape of train_y: {} \n'.format(y_train_res.shape))

print("After OverSampling, counts of label '1': {}".format(sum(y_train_res==1)))
print("After OverSampling, counts of label '0': {}".format(sum(y_train_res==0)))

Here is the full error message:

KeyError                                  Traceback (most recent call last)
<ipython-input-216-af83b63865ac> in <module>
      3
      4 sm = SMOTE(random_state=2)
----> 5 X_train_res, y_train_res = sm.fit_sample(X_train, y_train.ravel())
      6
      7 print('After OverSampling, the shape of train_X: {}'.format(X_train_res.shape))

~\AppData\Local\Continuum\anaconda3\lib\site-packages\imblearn\base.py in fit_resample(self, X, y)
     86         if self._X_columns is not None:
     87             X_ = pd.DataFrame(output[0], columns=self._X_columns)
---> 88             X_ = X_.astype(self._X_dtypes)
     89         else:
     90             X_ = output[0]

~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\generic.py in astype(self, dtype, copy, errors, **kwargs)
   5863                     results.append(
   5864                         col.astype(
-> 5865                             dtype=dtype[col_name], copy=copy, errors=errors, **kwargs
   5866                         )
   5867                     )

~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\generic.py in astype(self, dtype, copy, errors, **kwargs)
   5846                 if len(dtype) > 1 or self.name not in dtype:
   5847                     raise KeyError(
-> 5848                         "Only the Series name can be used for "
   5849                         "the key in Series dtype mappings."
   5850                     )

KeyError: 'Only the Series name can be used for the key in Series dtype mappings.'

do it without ravel (or reshaping of any kind).

Or if you going to do than transform dataframe X_train into an matrix also. This is the correct format  fit_sample


Ground-truth and feature extraction for predictive modelling I have a dataset of users, each user has has daily information about his activities (numerical values representing some measurements of his physical activities).

In addition, each user in each day has a boolean value that represents if he/she took a particular action.

The dataset looks as follow

+------+----------+----------+----------+-------+
|userID|      date| activity1| activity2| action|
+------+----------+----------+----------+-------+
| user1|2016-06-05|       5.3|         6|  false|
| user1|2016-06-04|       3.1|         8|   true|
| user1|2016-06-03|       2.0|        13|  false|
| user1|2016-06-02|       4.7|         1|  false|
| user1|2016-06-01|       1.3|         9|  false|
| user1|   ...ect.|       ...|       ...|    ...|
| user2|2016-06-05|       0.6|         5|   ture|
| user2|2016-06-04|       3.0|         5|  false|
| user2|2016-06-03|       0.0|         0|  false|
| user2|2016-06-02|       2.1|         3|  false|
| user2|2016-06-01|       6.3|         9|  false|
| user2|   ...ect.|       ...|       ...|    ...|
| user3|2016-06-05|       5.3|         0|  false|
| user3|2016-06-04|       5.3|        11|  false|
| user3|2016-06-03|       6.8|         5|  false|
| user3|2016-06-02|       4.9|         2|  false|
| user3|   ...ect.|       ...|       ...|    ...|
+------+----------+----------+----------+-------+

Note that the dataset is not fixed, so one new row is added for each user on every new day. But the number of columns is fixed.

Goal

Build a model that predicts which user is likely to take the action in the near future (e.g. in any of the next 7 days).

Approach

My approach is to build feature vectors representing the activity values for each users over a period of time, and use the action column as a source of ground-truth. Then I feed the ground-truth and the feature vectors to a binary classification training algorithm (e.g. SVM or Random Forest) in order to generate a model able to predict if a user is likely to take the action or not.

Problem

I started by the positive examples that are the users who took the action. To extract the feature vector of a positive example, I combined the activity values in the X (30 or 7 or 1) days preceding the action (the day of taking the action is included).

When I moved to the negative examples, it gets less obvious, I am not sure how to select negative examples and how to extract features from them. This has led me actually to re-question if my way of selecting positive examples and building my the feature vectors was correct.

Questions

How to build the ground-truth of positive (users who did take the action) and negative (users who didn't take the action) examples?
What is a negative example in this case? is it the user who didn't take the action in a fixed period of time? What if he didn't take the action in this fixed period, but he just took it right after?
What are the possible approaches of selecting the ranges of dates to extract feature vectors from.

Rational Question

Is there more suitable approaches (other than classification) to solve this kind of problems?

Your approach is a good one.
This way to extract features can lead to good results.
But before moving forward I recommend answering these two questions:
Is there is any correlation between the behavior of the user today and yesterday ... Etc " temporal correlation/ temporal dependency"
Is there is any kind of dependency between the users?
Answering these questions can measure the quality of your feature ?
Sometimes there is no need even to do calculations to answer these questions , if you tell us what kind of actions or activities you are looking at we may help in better way .
For example : if user 1 used his car today , there still a high prob to use it tomorrow, but if user 1 went to barbershop today there is a high probability that will not go tomorrow


Problem designing CNN network I seem to have a problem modelling my CNN network.

I want to extract from features vector from different sized images.
Whats consistent with the images is the y-axis, and the color dimension, but the x-axis is not constant.

Depending on the length of the x-axis will the length of the feature vector also be altered. I already know how long they should be,
but not the ratio between the length of the x-axis and length of the feature vector, I guess one does exist.

Is it possible to train a CNN network such that it can alter the feature vector length depending on the length of the input of the x-axis of the image?

It is not really possible to alter input feature array size per example on normal CNNs. Instead this is fixed when the model is built for the first time, before you start training.

Depending on your goal, it might be possible to work around that using some kind of pipeline that worked with image patches (taking multiple slightly randomised patches to augment the training data can improve results and doing the same with prediction inputs can also drive up classifier accuracy). Or a more complex variant using RNN/CNN hybrid to consume an image as a sequence of parts, which might also be used for multi-object recognition.

However, these solutions are complex, and state-of-the-art results in image classification can be achieved by simpler techniques such as taking a centre crop and/or padding. Provided your training data is also treated the same way, and aspect ratios are not extreme this can work adequately.


How to decide the number of trees parameter for Random Forest algorithm in PySpark MLlib? I am working on Random Forest algorithm in PySpark MLlib and have a doubt regarding the number of trees parameter that we pass to the model. The standard format of Random Forest modeling in PySpark MLlib is:

model = RandomForest.trainRegressor(trainingData, categoricalFeaturesInfo={},
                                    numTrees=3, featureSubsetStrategy="auto",
                                    impurity='variance', maxDepth=4, maxBins=32)

The doubt that I have is how to decide the optimum value of trees to pass to numTrees parameter? I assume the more the number of trees better should be the performance but would it keep on improving with increase in number of trees? Is there a point after which it will start to have negative impact in performance? If yes then how can I find the optimum number of trees for my dataset?

Maybe you will start to have negative impact of performance by increasing the number of trees. But definitely, at some point, increasing the number of trees will not add much accuracy in your model.

The usual way is to perform a k-fold cross-validation for different number of trees (and any other combination of model's parameters) and choose the one with the best performance. If you have any question about cross-validation, check this page. Instead of using simple accuracy, try to use other metrics that take into account the complexity of the model, like AIC.

Obs.: the final accuracy of the algorithm should be calculated using a separate dataset, not used in the cross-validation (usually called test set).


What should I use if I have millions of possible values for a feature in a sklearn predictive model? I am trying to create a large model. One of the features is categorical, and it has almost 100 million entries.

I have looked at sklearn LabelEncoder, but I am concerned that it will still create an ordering in my labels which I would like to avoid. If I use one hot encoding I would end up with really large vectors with 100 million dimensions. What are my options?

Unfortunately, AFAIK no sklearn model supports categorical variables.

For instance, sklearn decision trees only support rules such as X<n, not X==n which would be desirable here.

Also, the decision tree algorithm they implement only produces local one-look-ahead optimization. What this means is that, it may not produce rules such as X<n followed by X>n-1, even if such rule would be highly desirable.

In the end you will end up with non-sense things like: Car > 1 and then Car < 6 where 1 is Volkswagen and 6 is Ferrari.

The typical workaround is to use one-hot encoding, which might be a pain in the ass in your case. And then, it might not: sklearn decision tree supports sparse matrices, so the memory penalty would be low. You can use scipy for this. (Sparse matrices stores data differently than regular matrices. Instead of requiring $n\times m$ size, the memory requirement is proportional to the number of non-zeros in your matrix.) In terms of speed, it should not be any different than if the algorithm natively supported categorical variables.

This being said, your data may not allow for use of sparse, if the rest of the features are non-sparse. I don't think scipy has support for a sparse-dense hybrid matrix.

Another workaround I can think of would be to produce an Euclidean distance matrix between observations of the various categories (you may want to normalize first). Then group categories that are close together. Then build a hierarchical model, where you predict the final category for each category. In python, this is easier than it seems. You can create a class for your model using sklearn base classes.

I love Python and sklearn. But I believe in using the right tool for each job. I would use R which has native support for categorial variables (they call them factors) and has a plethora of decision tree packages. (Note: xgboost for R does not support categorial variables, it ignores the factor class-type.) Weka could also be a good tool, which also has very powerful decision tree algorithms.


Scikit-learn: train/test split to include have same representation of two different types of values in a column I have a dataset of online purchase orders that contains two types of customers:

Customers who have an account and thus are known customers with unique customer number.
Customers who do not have an account and do their purchases as a guest; thus, these are unknown customers and are given a default number(out of a list of 50 pre-defined numbers). Meaning it is not a unique number and the only way to distinguish transactions here is by the unique order number.

To avoid distortion and make sure that my train/test dataset will have the same distribution when it comes to known and unknown customers, I want to select a Train/Test set as follows:
The part of the dataset that includes the known customers to be split 80/20.
The part of the dataset that includes the unknown customers to be split 80/20 ( the selection of the sub dataset here will depend on the rule that orders are not linked to a customer (= linked to a default customer number).
In Scikit-learn I need to set up the X(features) and Y(target) in order to do the train_test_split. My question is would it be logical and doable to split my dataframe into:

df1 which contains all known customers.
df2 which contains all unknown customers.

Then set up X, Y and consequently x_train, x_test, y_train , y_test for each of df1 and df2, then merge the result into one and continue to build the rest of the model?
Is this a good approach or is there another solution?

Welcome to Data Science at StackExchange,
One way to accomplish this is to use the stratify option in train_test_split, since you are already using that function (this will also work for ensuring your labels are equally distributed, very useful in modelling an unbalanced dataset):
Train,Test = train_test_split(df, test_size=0.50, stratify=df['B'])
In my example, you can see that there are 2 values in column B, and they are equally distributed between the 2 datasets. In your case, B would be the column indicating those customers with an account and those without.


What do each of the three SVM classes in R represent? Inspired by this post, I took a look at this doc

SVN in R output this:

Statistics by Class:

                     Class: 1 Class: 2 Class: 3
Sensitivity            1.0000   0.9412   1.0000
Specificity            1.0000   1.0000   0.9762
Pos Pred Value         1.0000   1.0000   0.9796
Neg Pred Value         1.0000   0.9865   1.0000
Prevalence             0.2778   0.1889   0.5333
Detection Rate         0.2778   0.1778   0.5333
Detection Prevalence   0.2778   0.1778   0.5444
Balanced Accuracy      1.0000   0.9706   0.9881

What are each of the three classes?

Go to the kaggle site that you shared.
Click on Data on the left. Click on datadict.txt. This is known as data dictionary, a file that help us understand the dataset.

We can see

23: outcome
- what eventually happened to the horse?
- possible values:
1 = lived
2 = died
3 = was euthanized

Those are the meaning of the $3$ classes. You might also like to print out hodata$outcome or do str(hodata$outcome).


How to find pattern in sequential data? I have multiple sequential data that lead up to an event. All I want to do if find the equivalent sequential data or a pattern that leads to the event. The data is also multivariate so I want to consider multiple features rather than just a column. Help will be greatly appreciated.

I'm not entirely sure but it looks like sequence labeling might be what you need:

Sequences of varying length
Supervised: you would need to train a model with a sample of sequences annotated with a label at every step (not sure that this is your use case?)
Can handle any number of features

Conditional Random Fields is the state of the art method, there are many libraries
available.


How to upload a saved file from google colab to a site such as kaggle or github? We are able to load a file from a site such as kaggle and git hub to google collab, we apply the below code to download from other sites into google colab:

!wget nlp.stanford.edu/data/glove.6B.zip

but I am very curious to know if I am able to upload a saved file (such as an output from google colab) to kaggle or github with a command like above?

In your directory /miryam/myproject:

git init .
git add myfile
git commit -m "First commit"
git remote add origin remote https://github.com/yours_project
git push origin master


Is it correct to define the F-measure as the harmonic mean of specificity and sensitivity in such a way? It is common to define the F-measure as a function of precision and recall, as mentioned in [1]:

  $F_{\beta}=\frac{(1+\beta^2)PR}{\beta^2 P+R}$

However I came across some other cases, another definition is used [2] (without weights):

  $F = H(sensitivity, 1- specificity)$

Where H is harmonic mean.

Reference:

F - measure derivation (harmonic mean of precision and recall)
https://link.springer.com/chapter/10.1007/978-3-540-68947-8_133.
https://stackoverflow.com/a/52892413/2243842

Yes, as they are effectively synonyms of one another. See for instance this link

If you pay attention, the first formula is the (weighted) harmonic mean of the recall and precision.


Retrieving follower and followee (friend) counts using Twitter API I want to retrieve Twitter users' follower and followee (friend) count using Twitter API. (I  am reluctant with using a crawler since I need this data for research purposes) However, I was only able to find get_friend_list DOC which returns the whole list of followers.

I just want to get the number of followers and follweees not the while list of follower id (since this will require me to use more POST requests for Twitter API)

Could someone help me how to retrieve follower count with Twitter id (or username)?

I end up using GET users/show from Twitter API.

The details of User object which can be retrieved with this GET request can be founded in here.


Overfitting/Underfitting with Data set size In the below graph,

x-axis => Data set Size
y-axis => Cross validation Score

Red line is for Training Data

Green line is for Testing Data

In a tutorial that I'm referring to, the author says that the point where the red line and the green line overlap means,

Collecting more data is unlikely to increase the generalization
performance and we're in a region that we are likely to underfit the
data. Therefore it makes sense to try out with a model with more capacity

I cannot quite understand meaning of the bold phrase and how it happens.
Appreciate any help.

So, the underfitting means that you still have capacity for improving your learning while overfitting means that you have used a capacity more than needed for learning.

Green area is where testing error is rising i.e. you should continue providing capacity (either data points or model complexity) to gain better results. More green line goes, more flat it becomes i.e. you are reaching the point where the provided capacity (which is data) is enough and better to try providing the other type of capacity which is model complexity.

If it does not improve your test score or even reduce it that means that the combination of Data-Complexity was somehow optimal and you can stop training.


How does bagging help reduce the variance I learned that bagging helps reduce variance by averaging but I couldn't understand this. Can someone explain this intuitively?

High Variance  - Model varies a lot on small changes
High Bias - Model doesn't vary so much but predict quite away from the truth

Let's check a Decision Tree on 5 values -
\begin{array} {|r|r|}
\hline
1 &5 &10 &15 &20\\
\hline
\end{array}
In this tree split,
Value of 9.9 will be 7.5
Value of 10.1 will be 12.5.
Showing a very high variance.

Let's create 4 Random Tree of 3 elements each -
\begin{array} {|r|r|}
\hline
Tree-1 &5 &10 &15\\
\hline
Tree-2 &1 &15 &20\\
\hline
Tree-3 &1 &05 &20\\
\hline
Tree-4 &5 &15 &20\\
\hline
\end{array}
Value of 9.9 = (7.5 + 7.5 + 12.5 + 10)/4 ~ 9.375
Value of 10.1 = (12.5 + 7.5 + 12.5 + 10)/4 ~ 10.625
Variance is reduced a lot.
In bagging, we build multi-hundreds of the Tree(Can build other models too which offers high variance) which results in a large variance reduction


Pandas datetime error when reading from excel file I am trying to read an excel file that has two columns using pandas.

This is how the data looks in excel file:

DT                    Values
2019-11-11 10:00      28.9
2019-11-11 10:01      56.25
2019-11-11 10:02      2.45
2019-11-11 10:03      96.3
2019-11-11 10:04      18.4
2019-11-11 10:05      78.9

This is how it looks when I read using pandas:

DT                         Values
2019-11-11 10:00:00.000    28.9
2019-11-11 10:01:00.000    56.25
2019-11-11 10:01:59:995    2.45
2019-11-11 10:02:59:995    96.3
2019-11-11 10:03:59:995    18.4
2019-11-11 10:04:59:995    78.9

I have tried creating a new DateTime column, putting the data in a new excel file, converting the DT column to DateTime format in both pandas and excel. Nothing has worked yet!

Why does this happen?

EDIT - 1

I already tried the following code but forgot to mention the snippet,

df= pd.read_excel('data.xlsx', parse_dates = ['DT'])
df.head()

df['DT'] = pd.to_datetime(df['DT'])

Using pandas, first make sure you have a datetime column:

df['DT'] = pd.to_datetime(df['DT'])

To remove the milliseconds, a possible solution is to use round to obtain a specified frequency (in this case seconds).

df['DT'] = df['DT'].dt.round(freq='s')

Depending on the wanted final result, ceil (to always round up) or floor (always round down) could be more suitable.


Problem when using Autograd with nn.Embedding in Pytorch I am in trouble with taking derivatives of outputs logits with respect to the inputs input_ids. Here is an example of my input:

# input_ids is a list of token ids got from BERT tokenizer
input_ids = torch.tensor([101., 1996., 2833., 2003., 2779., 1024., 6350., 102.], requires_grad=True)

content_outputs = self.bert(input_ids,
                        position_ids=position_ids, token_type_ids=token_type_ids,
                        attention_mask=attention_mask, head_mask=head_mask)

# content_outputs[1] is sentence embedding
logits = F.linear(content_outputs[1], self.W_s, self.b_s)

# Notes:
# - input_ids.dtype = torch.float32
# - input_ids.is_leaf = True
# - input_ids.requires_grad = True
# - Torch version: 1.0.1

To compute the gradients for input_ids, input_ids.dtype must be float; otherwise, I will get the following error:

  RuntimeError: Only Tensors of floating point dtype can require gradients

However, my model is using an embedding layer which requires the input with dtype=long causing another problem if I use the input_ids initialized with float type above:

  RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.FloatTensor instead (while checking arguments for embedding)

I have also searched on Stack Overflow as well as Stack Exchange but I found nothing related to this problem. Please help me with this. Any comments would be appreciated!

So, this is just half answer, I write it here to be able to format the text clearly. You are facing troubles because you are trying to do something that you shouldn't, which is applying gradient to indices instead of embeddings. When using embeddings (all kinds, not only BERT), before feeding them to a model, sentences must be represented with embedding indices, which are just number associated with specific embedding vectors. Those indices are just values mapping words to vectors, you will never apply any operation on them, especially you will never train them, cause they are not parameters.

['Just an example...'] # text
       |
       | Words are turned into indices
       v
[1., 2., 3., 4., 4., 4.] # indices, tensor type long, no sense in applying gradient here
       |
       | Indices are used to retrieve the real embedding vectors
       v
[0.3242, 0.2354, 0.8763, 0.4325, 0.4325, 0.4325] # embeddings, tensor type float, this is what you want to train

If you want to fine tune BERT for a specific task, I suggest you to take a look to this tutorial BERT-fine-tuning


What is the difference between the value -99 and NaN in a data column? I am new to data science. I was looking into some datasets and I saw some values like -99, which I discovered later that it means that there is a missing value. Does this mean the same thing as NaN? If it is the same thing, why do we use -99 instead of NaN?

No, it is not the same. It may have that meaning in that particular dataframe, but don't take that as a rule.

I'd recommend you replace that magic number with actual NaNs, and then try to find the best possible way of filling up the missing values.


Sklearn feature selection stopping criterion (SelectFromModel) Sklearn has several functions for feature selection that lets the user determine the size of the chosen subset. An example of this is SelectKBest where the user determines the value of "k", which is the number of top performing features.

Does anyone know what stopping criterion SelectFromModel uses when it selects a feature subset? The documentation mentiones a "threshold"-parameter that determines which features are important enough, and that this parameter is set to "median" OR "mean" by default.

Some regression/classification models can also calculate feature importances - for example RandomForestClassifier models have a property feature_importances_ and LogisticRegression models have a coef_ property. There are many more models that can provide feature importances - but all of them either have a coef_ or feature_importances_ property.

What SelectFromModel does is to check if the object you pass as "estimator" has one of these two properties and to return features where this property is higher than the specified threshold (or the mean/median).

For example if you pass a RandomForestClassifier to SelectFromModel, it will return all features where the random forest's feature_importances_ property is higher than the specified threshold. The same happens if you pass a LogisticRegression model, except that it'll compare the coef_ property with the threshold instead.

Selecting the best value for the threshold can be done using a grid- or randomized search.


Decision boundary in a classification task I have 1000 data points from the bivariate normal distribution $\mathcal{N}$ with mean $(0,0)$ and variance $\sigma_1^2=\sigma_2^2=10$ with the covariances being $0$. Also there are 20 more points from another bivariate normal distibution with mean $(15,15)$ with variance $\sigma_1^2=\sigma_2^2=1$ and with the covariances being $0$ again. I used the least squares method to calculate the parameters of the decision bounday $\theta_0 + \theta_1 x_1 + \theta_2 x_2=0$, that is $$\theta = (X^T X)^{-1}(X^Ty)$$ where $y$ is a column matrix with labels $+1$ for points from the first class and $-1$ for points from the second.
The resuliting plot is as follows:

It is obvious that the decision boundary failed to be correct, as it passes right through class $-1$ and therefore it won't classify correctly future points that might stem from the same distribution. Now, there is the issue of why this is happenning. I understand that the main problem here is the imbalance of the data set, as there are $1000$ points from one class but only $20$ from the other. This, intuitively, makes sense.
What I want someone to help me with, if possible, is to understand how this imbalance problem is incorporated into the process of minimizing the least squares cost function $$J(\theta)=\sum_{n=1}^{200}(y_n-\theta^T x_n)^2$$
How does the fact that there are only $20$ points from the second class causes the minimzation task $\frac{\partial J(\theta)}{\partial \theta}=0$ to "fail"? How do the insufficient amount of these points causes this line to pass right through them? If there is some mathematical way to show me this, it would be nice, as I already got the intuition.

For those data points, a threshold just on x1 axis would perfectly separate the two distributions. You could fit a decision stump to calculate the single parameter of the decision boundary.


Bias and Variance definitions Is it correct to say that, in the Bias-Variance trade-off:

the bias error represents the ability of the model to possibly map the trend of the training set ?

and the variance represents how much the model is prone to variations if the dataset changes ?

I would say that both the definitions you gave above refer to variance.

Variance is an index of model complexity. Thus:

high variance -> high complexity -> many degrees of freedom in model -> it overfits training data -> if dataset changes, accuracy degrades a lot

Bias is an index of the systematic error of the model, as it shows the difference between the expected value of various model's predictions for a given target output and the ground truth.


Fitting Gaussian to data: Density-Estimation vs Regression Recently I discussed the following topic with a friend. The setting is that we have a one-dimensional set of data. (In the example it was points of students, we would be grading.) The goal was to make a density estimation, but not to use anything "fancy" like kernel density estimation, but just use a Gaussian as the estimation. (Sure that makes the big assumption that the data is Gaussian, but that is not the point here.) We discussed two ways:

Make a density estimation using an unsupervised learning method, e.g. using EM-algorithm. In this case the claim is that simply calculating the mean of the data and the standard derivation is already giving one the parameters to get the Gaussian parametrized the right way.
Add up the number of occurrences for each value and then use a supervised learning regression with the Gaussian as function, powered by an optimisation algorithm.

Through discussion we found out that the two clearly have different outcomes - though probably similar. For case 2 we optimise the parameters of the Gaussian such, that the sum of the distances from the Gaussian to the occurrences is minimized (along the y-axis if you will). For case 1 we optimise the parameters along the x-axis if you will.

Questions

Preliminary question: Is it correct that the EM-algorithm will have the same result as just calculating mean and standard deviation from the data?

Assuming the answer to the preliminary question is "yes, the results are the same":

What is the intuition interpretation of those two approaches?
None of them can be wrong in itself, but one of them could be wrong in the sense of wrong-usage. Meaning: I want to do something and I use the wrong method for it, because of misunderstood interpretation of what happened. So in that sense: Is one of them wrong in a way?

Example Code

I managed to express myself in R code. As one can see from the plots, the result is definitely not the same for any dataset. Only if the data is Gaussian and large the results get to be similar. But that means little to me ($2^x$ and $exp(x)$ converge both to $infinity$ for $x -> infinity$ and they have little in common otherwise).

library(ggplot2)
library(dplyr)
library(tidyr)

std <- 10

datasets <- list(
  data.frame( x = round(rnorm(1000, sd = std))),
  data.frame( x = round(rnorm(1000, sd = std))) %>% filter(x > 0)
)

for(data1 in datasets){

  data1$densest <- dnorm(data1$x, mean = mean(data1$x), sd = std, log = FALSE)
  data2 <- data1 %>%
    group_by(x) %>%
    summarise(count = n())

  f <- function(x, m, sd, k) {
    k * exp(-0.5 * ((x - m)/sd)^2) # 1/sqrt(2*pi*sd^2) *
  }

  cost <- function(par) {
    rhat <- f(data2$x, par[1], par[2], par[3])
    sum((data2$count - rhat)^2)
  }

  o <- optim(c(0, std, 10), cost, method="BFGS", control=list(reltol=1e-9))
  data1$regr <- f(data1$x, o$par[1], o$par[2], o$par[3])
  data1$regrNormalized <- dnorm(data1$x, mean = o$par[1], sd = abs(o$par[2]), log = FALSE)

  g1 <- ggplot(data=data1, aes(x=x)) +
    geom_point(data=data2, aes(x=x, y=count), alpha=0.2)+
    geom_line(aes(y=densest), color="green") +
    geom_point(data = data.frame(mean = mean(data1$x)), aes(x=mean, y=0), color="green") +
    geom_line(aes(y=regr), color="blue") +
    geom_point(data = data.frame(mean = o$par[1]), aes(x=mean, y=0), color="blue")

  g2 <- ggplot(data=data1, aes(x=x)) +
    geom_line(aes(y=densest), color="green") +
    geom_point(data = data.frame(mean = mean(data1$x)), aes(x=mean, y=0), color="green") +
    geom_line(aes(y=regrNormalized), color="blue") +
    geom_point(data = data.frame(mean = o$par[1]), aes(x=mean, y=0), color="blue")

  plot(g1)
  plot(g2)

}

Plots

The difference comes about because the sample mean and sd estimators are assuming the data is a random sample from the distribution on the full range of the distribution, in this case -Inf to +Inf. In your second case above there's no data below zero. The sample mean estimator can't see that you've only got half a Gaussian there, it will always be a positive number.

On the other hand, the curve-fitting approach of the least-squares fitting procedure (which is presumably possible via EM, or any of a dozen other algorithms) can see the data as part of a curve that has a Gaussian density shape. If the data is just curving upwards, it will fit to the left side of a Gaussian. If its curving down it will fit to the right side. If its flat it will either fit it to the peak with a very large variance, or a tail and might even go degenerate and divide by zero.

So you are doing parametric estimation of a distribution, or curve-fitting, which are categorically different things.


Can we use DecisionTreeClassifier of sklearn for continuous target variable? I have a continuous target variable named "quality" which ranges from 0 to 10. Also I have 11 input variables in my dataset.

When I'm building my model using DecisionTreeClassifier() of sklearn then I'm getting a score of 60% but when I'm building my model using DecisionTreeRegressor() of sklearn then I'm getting accuracy of 3% only and also RMSE as 85%.

Also, when using Linear Regression my R-squared value is 0.376. Is it good?

Dataset Link : https://archive.ics.uci.edu/ml/datasets/Wine+Quality

Am I doing something wrong?

I need help. Thank you.

In the Wine Dataset you linked, the quality column is not a continues variable but a discrete. It takes integer value between 0 and 10.

When you use the DecisionTreeClassifier, you make the assumption that your target variable is a multi-class one with the values 0,1,2,3,4,5,6,7,8,9,10. So, the model tries to predict one of these and only these values.

When you use the DecisionTreeRegressor, the assumption is that any number between 0 and 10 is acceptable. Like the number 4.52356. As a result, the accuracy will be noticeable worst. If you still want to use the Regressor for some reason, you can try to round the outcome and then calculate the accuracy. Keep in mind that RMSA doesn't fit your problem. You have a multi-class and not a regression model.


Faced problem while applying OneHotEncoder For classification, I was trying to convert categorical data into numeric by applying OneHotEncoder. But it shows error could not convert string to float

Here is the sample of my categorical data set and code of One Hot Encoding.

# TODO: create a OneHotEncoder object, and fit it to all of X
from sklearn.preprocessing import OneHotEncoder

# 1. INSTANTIATE
enc = OneHotEncoder()

# 2. FIT
enc.fit(train_obj)

# 3. Transform
train_ = enc.transform(train_obj)
train_.head()

And error message:

I couldn't understand what's the problem & how could I solve it.

If I apply get_dummies() method, is it similar of OneHotEncoder ?

Consider the following demo:

Source DF:

In [53]: df
Out[53]:
   A          B                   C
0  1  Bachelors  Married-civ-spouse
1  2    HS-grad            Divorsed
2  3       11th  Married-civ-spouse

Option 1:

In [54]: for c in df.columns[df.dtypes.eq('object')]:
    ...:     df = df.join(df[c].str.get_dummies())
    ...:

Result:

In [55]: df
Out[55]:
   A          B                   C  11th  Bachelors  HS-grad  Divorsed  Married-civ-spouse
0  1  Bachelors  Married-civ-spouse     0          1        0         0                   1
1  2    HS-grad            Divorsed     0          0        1         1                   0
2  3       11th  Married-civ-spouse     1          0        0         0                   1

Option 2:

In [58]: df = df.join(pd.get_dummies(df.select_dtypes(['object'])))

In [59]: df
Out[59]:
   A          B                   C  B_11th  B_Bachelors  B_HS-grad  C_Divorsed  C_Married-civ-spouse
0  1  Bachelors  Married-civ-spouse       0            1          0           0                     1
1  2    HS-grad            Divorsed       0            0          1           1                     0
2  3       11th  Married-civ-spouse       1            0          0           0                     1


How many vectors does paragraph vector generate for each paragraph? For example,if I have a corpus with two paragraphs, does paragraph vector generate two vectors?Additionally, on Distributed Representations of Sentences and Documents (Q. Le, T. Mikolov) paper I do not understand why paragraph vectors are unique among paragraphs but the word vectors are shared. Why?

It would make no sense of the word embeddings to change within a document. It would be as if the spellings of the words changed; how would that help?

When you use a document embedding, you find its numerical representation in a way that loosely captures its meaning. If you want to capture the meaning of each paragraph separately, find their embeddings separately. If you want to capture meaning of the entire document, feed the entire document.

For example, if your document covers a range of topics and you want to allow users to pinpoint where a particular topic is covered, you can find embeddings of each section (paragraph, page, etc.), then find the section that's nearest to your query's embedding.

One use case for dynamic word embeddings is to identify temporal dynamics; how the meaning of a word changes over time, as in this paper: The Visualization of Change in Word Meaning over Time using Temporal Word Embeddings.


Complete link clustering I'm conjecturing that with Complete-linkage clustering two elements from the same cluster will always be closer to each other some other element from another cluster.

In more formal terms:

Let $C$ be a clustering.
 $\not\exists z \in C_j$ s.t. $\bigtriangleup(x, z) < \bigtriangleup(x, y)$ where $x,y \in C_i$, $C_i \neq C_j$ and $C_i, C_j \in C$.

I haven't been able to prove the conjecture yet, thus I'm wondering whether I'm right or wrong. If this is indeed the case, I would much appreciate a sketch a proof. I'm pretty sure I can work my way from there.

On a side note (not that I think it makes a difference), I'll be applying the clustering algorithm on a one-dimensinal dataset.

Your input is much appreciated.

Complete-linkage clustering tends to find compact clusters with equal diameters. It is define the inter-group distances as the largest distance of objects between two clusters. Beside, it is may produce clusters that has the same number of objects and the distance between objects is contiguous.

So, I can say that your conjecture is correct. I cannot wait for any experts to explain this in detail.


Non-mutually exclusive classification sum of probabilities So I have the following problem: I realized (while writing my master thesis) that I am still not sure/have vague descriptions of some of the machine learning principles. I already asked one question regard definitions that can be found here.

Now I stumbled over another definition Problem.
Here is an excerpt from my thesis (this is in particular about neural-network classification):

  If the classes are mutually exclusive (i.e. if a sample $x^{j} = C_{0}$, $x^{j} \neq C_{i}\setminus~C_{0}$ ), the probabilities of all classes add up to one like \begin{equation}
\sum_{i} P(x^{j}=C_{i}) = 1.
\end{equation}
  In this case the best practice is to use a softmax activation function for the output neurons.
  If the classes are not mutually exclusive it would suffice to use a sigmoid output activation function, as the sigmoid function gets independent probabilities for each class \begin{equation}
\sum_{i} P(x^{j}=C_{i}) \geq 1.
\end{equation}

I already found the following link regarding this topic.
However I know that in practise if you don't use softmax activation function in your output layer, the value can be larger than 1 but can a probability be larger 1? Isn't that against its definition?

Is a non-mutual classification really a common case? Can somebody may be linking some cases (paper?) were they needed non-mutual classification?

You are correct: a probability cannot be larger than 1.

At the final layer, the activations (also known as logits), are passed through the final a softmax function in order to fulfill this constraint. The standard neural network does not have an implicit mechanism by it can ensure that constraint is met during training etc.

By non-mutual classification, we could be talking about something like classifying cat and dog images - in which case the label for each image either cat or dog. So they are mutually exclusive. This is a very common case - almost any form of image classification falls into this category.

You do not use a sigmomid function (or any other non-linearity for that matter) after the final layer, as there are no neurons following them, making a non-linearity somewhat redundant. Using a non-linearity for the purpose of fitting a non-linear model is different to the purpose of a final softmax function. This has exactly the purpose of scaling the final logits/activations into the nice range of [0, 1] that can be interpreted as probabilities. That allows us to make simple rules on how to classify the outputs - e.g. if p = [0.51, 0.49] then that sample was a cat, whereas p =[0.49, 0.51] is a dog.

I used those values in the example to highlight a further point; namely that you cannot interpret them as pure probabilities. Those examples don't mean the model was really unsure in both cases because all four "probabilities" we close to 0.5. The model gives more weight to the option is believes is correct - the relative magnitudes of those values are not directly interpretable.


Can we optimize regression problems that have categorical variables by encoding them if on the other hand we are inserting multicollinearity? Can we optimize regression problems that have categorical variables by encoding them if, on the other hand, we are inserting multicollinearity?

Multicollinearity can be a problem if you choose to optimize linear regression with ordinary least squares (OLS). Because the data matrix $X$ can have less than full rank, therefore the moment matrix $XᵀX$ cannot be inverted.

If you choose to optimize linear regression with gradient descent, multicollinearity is not an issue in finding an optimal solution.


Feature selection: Information leaking if done before CV-split? Currently I'm doing some simple feature selection based on correlation between features and variance within one feature. I'm applying this on the whole dataset used for model building before creating the cross-validation.

My question now is if this is acceptable workflow or can significantly affect the CV stats suggesting a better model than it actually is?

Is it technically better to do the CV-split and only then select features on the training set for that split to not leak information?

If you do not use target for feature selection, then there is no leakage and you can apply this on the whole dataset. The methods that you mention: checking correlation between features and checking variance of each feature belong here.

If you do use target for feature selection, then you should use 'train' subset only. Otherwise there is a leakage. The examples here would be (i) you check correlation between each feature and target (ii) use build Random Forest model and take best features


1st order Taylor Series derivative calculation for autoregressive model I wrote a blog post where I calculated the Taylor Series of an autoregressive function. It is not strictly the Taylor Series, but some variant (I guess). I'm mostly concerned about whether the derivatives look okay. I noticed I made a mistake and fixed the issue. It seemed simple enough,but after finding an error, I started to doubt myself.
$$f(t+1) = w_{t+1} \cdot f(t) $$
$$y^{*}_{t+1} = f(t+1)-{\frac {f'(t+1)}{1!}}(-t-1+t)$$
$$y^{*}_{t+1} = w_{t+1} f(t) + \dfrac{d}{df(t)}w_{t+1}f(t) + \dfrac{d}{dw_{t+1}}w_{t+1}f(t)$$
$$y'_{t+1} = w_{t+1} f(t) + w_{t+1} + f(t)$$
The details can be found in the blog post:
EDIT 7/6/20:
The AR form:
$$y^{*}_{t+1}=c+\sum _{{I=0}}^{L}w _{t+1-i}y_{{t-i}}+\varepsilon _{t}$$
f(t) is a recursive dense layer, y is the predicted output, and w are the weights, and L are the number of lag components. For the simple case where the next value only depends on the previous value, I got the following result.
$$f(t+1) = w_{t+1} \cdot f(t) $$
$$y^{*}_{t+1} = f(t+1)-{\frac {f'(t+1)}{1!}}(-t-1+t)$$
$$y^{*}_{t+1} = w_{t+1} f(t) + \dfrac{d}{df(t)}w_{t+1}f(t)$$
$$y'_{t+1} = w_{t+1} f(t) + w_{t+1}$$
EDIT 7/7/20:
The function f(t) represents y(t) with an error term. The error term might have some random process, but I'm going to assume that the errors are independent.
$$f(t+1) = w_{t+1} \cdot y(t) + \epsilon_t$$
EDIT 7/9/20:
Changed the dimensionality of w_t+1 to w_t.
$$f(t+1) = w_{t} \cdot f(t) $$
$$y^{*}_{t+1} = f(t+1)-{\frac {f'(t+1)}{1!}}(-t-1+t)$$
$$y^{*}_{t+1} = w_{t} f(t) + \dfrac{d}{df(t)}w_{t}f(t)$$
$$y'_{t+1} = w_{t} f(t) + w_{t}$$


Basic DNN with highly imbalanced dataset -- network predicts same labels I will try to explain my issue at a high level, and I hope I'd be able to get some better understanding of the ML behind it all.
I am working with aggregated features extracted from audio files, so each feature vector is of size (1xN). The output would be a single sentiment label, Positive, Neutral, or Negative. I mapped these to 2, 1, 0 respectively (the labels are discrete by design, but maybe I could make it continuous?)

The dataset I am using is 90% neutral, 6% negative, and 4% positive, and I split these into train/dev/test. I wrote up a basic DNN in PyTorch, and have been training using CrossEntropyLoss and SGD (with nesterov momentum). The issue I am running into is that the network, after seeing only ~10% of the data, starts to predict only netural labels. The class weights converge to something like

tensor([[-0.9255],
        [ 1.9352],
        [-1.1473]])

no matter what 1xN feature vectors you feed in. I would appreciate guidance on how to address this issue.
For reference, the architecture is

DNNModel(
  (in_layer): Linear(in_features=89, out_features=1024, bias=True)
  (fcs): Sequential(
    (0): Linear(in_features=1024, out_features=512, bias=True)
    (1): Linear(in_features=512, out_features=256, bias=True)
    (2): Linear(in_features=256, out_features=128, bias=True)
  )
  (out_layer): Sequential(
    (0): SequenceWise (
    Linear(in_features=128, out_features=3, bias=True))
  )
)

def forward(self, x):
    x =  F.relu(self.in_layer(x))
    for fc in self.fcs:
        x = F.relu(fc(x))
    x = self.out_layer(x)
    return x

Not sure if NN actually makes sense -- could it be the relus between each hidden layer or the bias? Or something else?

Reposted from Stack Overflow here, since this forum is more appropriate: link

A neural network is probably a perfectly reasonable approach here. But this is an extremely unbalanced dataset and you're going to have to handle that somehow.

The network is learning that the best way for it to reduce the loss is to always guess neutral - if you were taking a standardized test and you realized that you'd get 90% right if you always guessed 'C', you certainly wouldn't answer 'A' and 'B' in equal measure.

There are a few standard ways to deal with this, and probably depend on how big your dataset is (how many examples 6% and 4% come out to be):

Oversample the minority classes to even things out (so you'd show the net the same example of the positive / negative classes multiple times for every time it saw each neutral example)
Undersample the majority class (only show the net a random subset of the neutral samples during each epoch)
Set a class weight / sample weight - proportionally weight down the contribution to the loss function by examples of the majority class.

I'd probably try 3 first because it's super easy. Check out the documentation of the Cross Entropy Loss. It takes a tensor of weights as a parameter and weights the classes according to the entries in that tensor. So just say the positive (4%) class gets weight 1, negative(6%) gets 2/3, and 90% gets 2/45 (if I did the math right...).

That would even out the total contributions of each class in the dataset. Hopefully you'll see better results that way. You'll probably want to look at precision / recall to evaluate the results (because random guessing gets you 90% if you're using accuracy).


Is it possible to predict sentiment of unlabelled dataset using BERT? I have a large unlabeled dataset and I want to predict sentiment for each document in this dataset. I want to know, is it possible that I can use BERT for sentiment analysis of unlabeled data? I have seen so many tutorials and read the blog posts but I couldn't find one. All shows the use of BERT on datasets that are already labeled such as the IMDB review dataset or Yelp review.

Bert is a pre-trained language model with objectives like masked token prediction and next sentence prediction. So, it doesn't have any setup to do sentiment analysis,but you can use the pre-trained information to do the same.
There are couple of ways to solve your problem, they are

Annotate few of your documents and fine-tune the model for sentiment analysis.
Get open-source sentiment analysis datasets which fit your requirement, train Bert on the data and use the same classifier for your purpose (But this will work only if both data distributions look the same).


Why my model can't recognise my own hand written digit? Currently i am working on digit recognizer[0-9]. My model train accuracy 100% and test accuracy 90%. But when i train to feed my own written digit, it always give me wrong prediction.

I know test and train images should come from same source. But how could i feed different source data?

You have to remember that machine learning model do not understand any concepts as we do, humans. It cannot generalize something it hasn't seen. And yours hasn't seen black digits on white background, so it has no way of predicting a digit right.

The only two things you can do is :

Train your model from scratch with both datas : black digits on white background and white digits on black background. These datas have to be balanced (as much as possible). Maybe the model need to be more complex, maybe you need more data, and maybe your accuracy will decrease. You can then predict both type of datas.
Uses a test dataset only with similar datas of the train set, white digits on black background.

You can also do a data augmentation with your previous train dataset :
as the digits are white and background black, you can reverse images color, then you will have black digits on white background. That way, you have your train set doubled in size, without writing any more digits by yourself !


No gradients provided for any variable: ['Variable:0', 'Variable:0'] I am using Python 3.6 and Tensorflow 2.0 for the following code for linear regression:
tx = tf.constant(x, dtype=tf.float32)
ty = tf.constant(y, dtype=tf.float32)

tw = tf.Variable(initial_value=np.random.randn(), dtype=tf.float32, trainable=True)
tb = tf.Variable(initial_value=0, dtype=tf.float32, trainable=True)
tyhat = tx * tw  + tb

cost = lambda: tf.reduce_mean(tf.square(tyhat-ty))
optimizer = tf.optimizers.SGD(0.01)
train = optimizer.minimize(cost, var_list = [tw,tb])

and got an error: No gradients provided for any variable: ['Variable:0', 'Variable:0']
Any help?

tw = tf.Variable(initial_value=np.random.randn(), dtype=tf.float32, trainable=True)
tb = tf.Variable(initial_value=0, dtype=tf.float32, trainable=True)

cost =  lambda :tf.reduce_mean(tf.square(tx*tw+tb-ty))
optimizer = tf.keras.optimizers.SGD(0.01)
train = optimizer.minimize(cost, var_list = [tw,tb])

Reference to tw and tb directly in cost function rather than through yhat. It will work.


Encoding multiple observations from the same feature space My data contains multiple observations of categorical feature.The feature space is medical symptoms, so the data for this feature is like : ['fever','pain','yellow skin' .... ] .The amount of symptoms observations per sample is not fixed and i have around 50 different symptoms

how can i encode this feature into something that ML model can deal with ?  the order of the symptoms in the array is not important.

i tried one hot encoding but projecting feature space of 50 category levels into 50 indicators means losing information (having a sparse matrix )

any ideas?

What you need is Encoding Categorical variables. This topic is discussed in tons of blogposts, it is definitely worth checking out this recent article that nicely and extensively go through most of the methods, since I do not intent to rewrite again what is out there but rather giving you my personal experience.

I asked you earlier what algorithms you are after, simply that could change your choice of encoding method. Some encoding methods like One-hot-encoding makes your feature space very sparse when your categorical variable is very cardinal (usually not recommended!), and it is best to go with sparse-aware algorithms.

In a nutshell, I suggest to start with the following (classic and simple), explanations borrowed from that article:

  OneHot — one column for each value to compare vs. all other values.
  Binary — convert each integer to binary digits. Each binary digit gets one column.
  Hashing — Like OneHot but fewer dimensions, some info loss due to collisions.
  Backward Difference — the mean of the dependent variable for a level is compared with the mean of the dependent variable for the
  prior level.
  Target — use the mean of the dependant variable.

The above-mentioned methods can be used pretty much for all algorithms. And each single has its pros and cons. Some you have more information loss than the other and so on. Good news is that most of these methods are quite easy to use, e.g. via this Python package.

Another method that I found quite interesting and is suitable for Neural Networks is:

Entity Embeddings - An special encoding method borrowed from NLP, in which an embedding space is learned on the fly for each categorical variable.

See these blogpost 1, 2 ,3. I am writing an up-and-running code in Google Colab to fully demonstrate this, but you still find pieces of code in those blogposts to give it a try. I find this method way better than for example OneHot for Neural Networks.

Hope these help!


DONUT- Anomaly detection Algorithm ignores the relationship between sliding windows? I'm trying to understand the paper : https://netman.aiops.org/wp-content/uploads/2018/05/PID5338621.pdf about Robust and Rapid Clustering of KPIs for Large-Scale Anomaly Detection.

Clustering is done using ROCKA algorithm.

Steps:

1.) Preprocessing is conducted on the raw KPI data to remove amplitude differences and standardize data.

2.) In baseline extraction step, we reduce noises, remove the extreme values (which are likely anomalies), and extract underlying shapes, referred to as baselines, of KPIs. It's done by applying moving average with a small sliding window.

3.) Clustering is then conducted on the baselines of sampled KPIs, with robustness against phase shifts and noises.

4.) Finally, we calculate the centroid of each cluster, then assign the unlabeled KPIs by their distances to these centroids.

I understand ROCKA mechanism.

Now, i'm trying to understand DONUT algorithm which is applied for "Anomaly Detection".

How it works is :

DONUT applies sliding windows over the KPI to get short series x and tries to recognize what normal patterns x follows. The indicator is then calculated by the difference between reconstructed normal patterns and x to show the severity of anomalies.
In practice, a threshold should be selected for each KPI. A data point with an indicator value larger than the threshold is regarded as an anomaly.

Now my question is :

IT seems like DONUT is not robust enough against time information related anomalies. Meaning that it works on a set of sliding windows and it ignores the relationship between windows. So the window becomes a very critical parameter here. So it might generate high false positives. What I'm understanding wrong here?

Please help and make me understand how DONUT will capture the relationship between sliding windows.

The code is available on github and the sliding windows are actually not overlap. Below code from file donut/utils.py, method  minibatch_slices_iterator:

start = 0
stop1 = (length // batch_size) * batch_size
while start < stop1:
    yield slice(start, start + batch_size, 1)
    start += batch_size

I haven't inspect deeper but maybe the windows are not related, as you said.


What to input into machine learning algorithm for image recogniton? I am working on a project that involves classifying images as either that of a cat or that of a dog, without using CNNs. I used SKImage to convert the images to a matrices and changed it to grayscale to reduce dimensions and complexity. I then flattened the matrices to vectors and input them into an SVC, Logistic Regression and Stochastic GD Classifier, but all of them were very innacurate and had less than 60% accuracy. This leads me to believe that I converted the images to the wrong format, which is why the ML algos are unable to fit the data. So what should I convert them to? Should I use feature extraction?

Usually, neural networks are used to do image classification. The reason is that image data is very complex (highly non-linear) if you want to say so.

Have a look at Keras/Tensorflow. You can even use pre-trained models to quickly achieve good results on most problems. Find an example for Python here: https://github.com/Bixi81/Python-ml/blob/master/keras_pretrained_imagerec_binaryclass.py

There is a very instructive book „Deep learning with Python“, which can give you a quick start. Find the book here: https://github.com/fchollet/deep-learning-with-python-notebooks


IID violation in machine learning Imagine I'm collecting some training data. Lets say I collect a 30minute time series from 1000 people so I have 1000 observations (rows) in my feature matrix.  I train some model (lets say a neural net for this example) and I find my AUC is really poor and I believe the problem to be the fact that I only have 1000 observations - so I don't have enough data

However, I am now unable to collect anymore data. One thing I could do, is take that 30minute time series, and slice it up into 30 1-minute sections. Then use these 1 minute series as rows in my data. So I would end up with 30 observations per person, and 1000 people, giving me 30000 rows in my feature matrix. I've now increased the size of my training set by 30x

If I were to do this when doing statistical/inferential tests I would be violating the assumption of independence, and I'd be forced to model this with some multilevel model to correct

Is this the same as the IID assumption in machine learning, where each of your rows must be independent of each other?

For inferential tests, (one of) the reason why this assumption matters is because it affects the Type 1 error of your inference. However, in machine learning we're not doing any inferential tests, so what effect does violating IID actually have on results? In other words, why do rows need to be independent of each other? Especially in a case like above where I can drastically increase the size of my training set by reusing different parts of 1 person's data

Suppose you are investigating if heart rate can predict if a person smokes. You measure bpm for 30x1m consecutive times, and ask if the person smokes in order to build your training model data set.

What would contribute to a better predictor? 30 observations from a person who smokes, or 1 observation from 30 people who smoke? Given that one person's heart rate won't change much over 30m, it seems clear that you'd rather have 1 sample from 30 people than 30 samples from one person. 30 samples from one person are not worth as much as 1 sample from 30 people.

Because the samples from one person are not independent.

I think if you put non-independent samples into a neural net then it won't affect the predictive power too much as long as the non-independence is similar across all your training data. In one extreme, if all smokers and non-smokers have the same heart rate over the 30m period, then all you've done is repeated your input data precisely 30 times and nothing will be changed (except it will take 30x as long to run...).

However, if smokers' heart rates are constant, and non-smokers' vary, then you add 30 measurements of each smoker's heart rate to your model, and a bunch of random measurements correlating those rates with non-smoking. Your NN is very likely to predict anyone with one of those smokers' heart rates is a smoker. This is clearly wrong - those 30 measurements from each smoker are only worth one measurement, and putting them in a NN will train the network wrongly.


Machine learning model bundled with a library vs. an API I am thinking to "deploy" a machine learning model (in pickle it is sized 3 megabytes) and after discussing with my developer colleagues, they said it would be better if the model is packed as a python library instead of a microservice (like a rest API).

I wanted to ask what's your view on this: Pickled model packed in a library specifically meant for it vs. a rest API, pros and cons?

I was thinking that having it as a library could possibly be easier to use and wouldn't require to have to worry about deployment, web address etc.

The advantages of deploying the model as a package is the code will be part of the monolith application and require no remote calls thus:

No additional operational demands
No external dependencies
Faster
Fewer security issues
Same uptime as the rest of the application

In general, most features stay in a single monolith application until there is are specific reasons to create separate services.


Is it valuable to normalize/rescale labels in neural network regression? Have there been any papers, or does anyone have any specific experience to know whether normalizing labels in a regression problem is likely to improve the performance of a neural network? I have labels that are in the range (0,1000) applying square loss in a ConvNet. I want to know if it might be useful to normalize these to a (0,1) range, or whether that's known to not matter.

Yes, you should do this. Given the initialization schemes and normalized inputs, the expected values for the outputs are 0. This means that you will not be too far off from the start, which helps convergence. If your target is 1000, your mean squared error will be huge which means your gradients will also be huge which can lead to numerical instabiliy.


Which is best FaceNet or dlib_face_recognition_resnet_model_v1? I am newbie in face recognition related things... As far i observed dlib's frontal_face_detectoris widely used to find the faces in an image and after that, to extract face_descriptor vectors which is better for real time face authentication system ?

FaceNet by google
dlib_face_recognition_resnet_model_v1 by face_recognition

It looks both working fine.. but in real-time implementation,

Is there some thing important to understand the performance ?

Or any comparison checks done on real time / large data sets ?
Thanks in advance.

In my experience with the natively LR, tinyface dataset, dlib's resnetv1 model failed to extract embeddings for a number of faces from images with high gaussian blur. However, FaceNet was able to vectorize those same poor-quality LR images.


Difference between stacked lstm vs attention mechanism in LSTM What is the difference between stacked lstm vs attention mechanism in LSTM? It seem to me that both produce the same context vector at the end.
EDIT: From suggestion by @shepan6,

the difference in architecture of stacked ltsm and attention mechanism
not too sure i am right here, it seem that both used the notion of being able to select important features to construct the final context vector.
if the answer to above is yes, am i right to say the goal is the same for both, just that the mechanism is different

Thanks

Stacked LSTM is a special version of hierachical recurrent neural networks, where hard-wired memory and gating units help long-term preservation of state information.
Hierarchy and recurrence have been explored in many works.
One early example is the Neural Abstraction Pyramid, which introduced recurrent computation to hierarchical convolutional neural networks (a.k.a. deep learning).
It incorporates partial interpretations from a larger-and-larger context via horizontal and vertical feedback loops in order to iteratively refine an interpretation.
It was trained using Backpropagation Through Time to solve several computer vision tasks, such as image denoising, superresolution, and object detection.
The recurrent computation is also well suited for maintaining hierarchical state information when processing image sequences.
http://www.ais.uni-bonn.de/books/LNCS2766.pdf


SHAP value analysis gives different feature importance on train and test set Should SHAP value analysis be done on the train or test set?

What does it mean if the feature importance based on mean |SHAP value|  is different between the train and test set of my lightgbm model?

I intend to use SHAP analysis to identify how each feature contributes to each individual prediction and possibly identify individual predictions that are anomalous. For instance, if the individual prediction's top (+/-) contributing features are vastly different from that of the model's feature importance, then this prediction is less trustworthy. Does this approach make sense?

Since SHAP gives you an estimation of an individual sample (they are local explainers), your explanations are local(for a certain instance)
You are just comparing two different instances and getting different results. This is normal and can happen in train and test set. This doesn't mean also that your train and test set have bad split, they could be good split.
In the end SHAP is done to help you understand how the model behaves in a particular instance. It should be done where you are interested in understanding. I guess that you can also try to find what is the difference between train and test with shap values, but they are local explainers so you might not find much success.
I wouldn't say anything about the quality of predictions given the feature importance.


Applying Hold-out and CV technique I have a methodology question: are hold-out and CV generalization-optimization techniques mutually exclusive? It gets really confusing to me at times, because in the most recent project I have been doing something as follows:

I have split the dataset into TRAIN and TEST sets (with stratified distribution)
I have applied gridsearchCV on the TRAIN set with cv = 10, so effectively the model was splitting TRAIN set into TRAIN' and VALIDATION sets at each fold
I have used the optimized trained model with the TEST set, and ended up with results 0.97 (TRAIN set) vs 0.68 (TEST set).

Normally, when I see results like that my immediate reaction would be to assume that I am overfitting to the TRAIN set, and that the model does not generalize well enough. However, I have already used CV with the TRAIN set to make sure I was tuning the model to generalize as well as possible. I do not have any immediate tools (nor I should touch the already trained model) to improve this score now, since as I understand Hold-out technique, it just gives me the final score of how my model performs on previously unseen data.

At the same time, it feels kind of strange, since with using both CV and Hold-out, I am guaranteed to get some outliers in the hold-out set (TEST in my text above), which will fail.

Any suggestions or ideas?

You are correct in the sense that when tuning your model via. grid search you are technically not leaking any data. But, recall that tuning your model (via. a specific procedure such as grid search) is one of only many steps you probably took in fitting your model pipeline. In particular, areas such as pre-processing, feature engineering, imputation, model tuning, data aggregations, etc. The point of the test set is to capture the entire model building process and not just the process of model tuning.

Furthermore, it is highly known that validation scores reported during model tuning tend to be optimistically biased (and this bias tends to be worse with smaller datasets). This is because the probability of finding a set of hyper parameters that coincidentally minimizes the error for the validation set but not to the overall population (i.e., overfitting to the validation set) becomes higher the finer your grid is. Imagine theoretically tuning your model to one million different hyper parameter combinations. The probability of selecting a bogus set of hyper parameters (i.e. that are only optimal for the validation set) is now quite large due to the sheer number of possible candidates you have elected to try.

With a test set, this still won't prevent overfitting to a validation set. However, it will allow you to detect the problem and give you the true unbiased measure of model performance.

  At the same time, it feels kind of strange, since with using both CV and Hold-out, I am guaranteed to get some outliers in the hold-out set (TEST in my text above), which will fail.

Indeed, which is a major drawback of cross validation and data splitting especially for smaller datasets (with a lot of outliers/noise). Basically, the performance measure you observe tends to be highly variable with how you split the data in the first place (that is, the seed you choose when splitting your data can lead to large changes in estimated model performance depending on where your outliers fall). The solution is unfortunately, not very glamorous and time consuming. In order to gain more certainty in our estimate, we need more than just a single estimate of model performance. Thus, simply repeat the entire model building process again with a different data partition (a different test set). Repeat this however many times, and average over all repeats. Possibly, form a confidence interval that allows you to see for yourself how variable your model's performance is.

I will also note that there are other ways to get around this problem, such as "optimism adjusted bootstrap", but recent issues have arisen with this method which have potentially shown that for high dimensional data this method does not do well, despite being more efficient than cross validation. Since high dimensional data is the norm these days, I have my doubts but perhaps it may be of use to you.


Bert Fine Tuning with additional features I want to use Bert for an nlp task. But I also have additional features that I would like to include.

From what I have seen, with fine tuning, one only changes the labels and retrains the classification layer.

Is there a way to used pre-trained Bert models and include additional features?

To add additional features using BERT, one way is to use the existing WordPiece vocab and run pre-training for more steps on the additional data, and it should learn the compositionality. The WordPiece vocabulary can be basically used to create additional features that didn't already exist before.

Another approach to include additional features would be to add more vocab while training. Following approaches are possible:

Just replace the "[unusedX]" tokens with your vocabulary. Since these were not used they are effectively randomly initialized.
Append new vocabulary words to the end of the vocab file, and update the vocab_size parameter in bert_config.json. Later, write a script which generates a new checkpoint that is identical to the pre-trained checkpoint, but with a bigger vocab where the new embeddings are randomly initialized (for initialized we used tf.truncated_normal_initializer(stddev=0.02)). This will likely require mucking around with some tf.concat() and tf.assign() calls.

Please note that I haven't tried any of these approaches myself.


Alternatives to Graph theory? I was planning to use a graph theoretic and page-rank approach to finding the most influential person in an organization.

  Influential person is someone who drives a lot of activity in the
  organization. When he assigns a task, most people do it. When he sends
  a mail, most people reply to him. When he assigns some training, most
  people do it.

But there is one drawback to using graph theory here. For example, say Person A is very influential, and Person B has just joined an organization and works under A. B puts in a leave request and sends it A for approval. A approves the leave request. This seems contradictory, since A is very influential but A responded to B. According to graph theory, wouldn't this make Person B also very influential? This approach would give very bad results.

How can i overcome this limitation? Can anyone suggest an alternative approach?

I am using number of interactions to calculate an affinity scores between two users. Based on this, I suggest what content should be shown to a user (like FaceBook does). User A responds to User B most of the time.  so (A->B) has a high affinity score. So, in user B's newsfeed I intend to show content coming from User A.

is there a better way than just counting number of interactions?

If two users A & C have same affinity scores with B, if A is more influential than B, A's content will be shown first to user B.

According to graph theory, wouldn't this make Person B also very influential?

No. You need to think about how to set up your graph first and then go further. In this case your graph is not simple but both weighted and directed thus handling a request of B by A neither reduces A's influence nor improves B's. Because the weights of directed edges are determining the influence.

Is there a better way than just counting number of interactions?

Community Detection or what computer scientists call Graph Clustering. If you do not have to limit yourself to hierarchies, you can extract communities who have the most interaction and share suggestions inside the community.


Feature selection vs Feature extraction. Which to use when? Feature extraction and feature selection essentially reduce the dimensionality of the data, but feature extraction also makes the data more separable, if I am  right.

Which technique would be preferred over the other and when?

I was thinking, since feature selection does not modify the original data and it's properties, I assume that you will use feature selection when it's important that the features you're training on be unchanged. But I can't imagine why you would want something like this..

Adding to The answer given by Toros,

These(see below bullets) three are quite similar but with a subtle differences-:(concise and easy to remember)

feature extraction and feature engineering: transformation of raw data into features suitable for modeling;
feature transformation: transformation of data to improve the accuracy of the algorithm;
feature selection: removing unnecessary features.

Just to add an Example of the same,

  Feature Extraction and Engineering(we can extract something from them)

Texts(ngrams, word2vec, tf-idf etc)
Images(CNN'S, texts, q&a)
Geospatial data(lat, long etc)
Date and time(day, month, week, year, rolling based)
Time series, web, etc
Dimensional Reduction Techniques (PCA, SVD, Eigen-Faces etc)
Maybe we can use Clustering as well (DBSCAN etc)
.....(And Many Others)

  Feature transformations(transforming them to make sense)

Normalization and changing distribution(Scaling)
Interactions
Filling in the missing values(median filling etc)
.....(And Many Others)

  Feature selection(building your model on these selected features)

Statistical approaches
Selection by modeling
Grid search
Cross Validation
.....(And Many Others)

Hope this helps...

Do look at the links shared by others.
They are Quite Nice...


How to bootstrap the AUC on a data-set with 50,000 entries? I am learning classification. I do have AUC-ROC curve of the data-set. How should I proceed to bootstrap the AUC? If I do the sampling of the same size as data-set then my AUC value would not be the same all the time?

Yes, sample the same size of your data set (with replacement), then find your AUC, say 10,000 times.  You will then have 10,000 different AUCs which will give you some idea of how confident you are in the AUC result.

When you sample with replacement, you will only find around 63% of the entries in your new bootstrapped data set will be unique (so many of these will have 2+ identical entries in the bootstrapped data set.  Every time you make a new bootstrapped sample, the 63% that are unique will change, as will how many times a record is duplicated in the bootstrapped sample.  This helps extract some information on the natural variability of your data.

So to recap, if you have 50,000 records, which means 50,000 probabilities/values and 50,000 class labels:

1) samples 1:50,000 with replacement.  You should find about 31,500 unique records in this new sample.

2) calculate the AUC

3) repeat steps 1 and 2, 10,000 times(pending computational times, I would shoot for at least 1000), and save those 10,000 AUCS.

4)  Now you have 10,000 AUCs.  If you take the 5th percentile and 95th percentile, you have the bootstrapped 95% confidence interval.  The big benefit here is now you can take the mean of those 10,000 AUCs, and give the standard error $(sd/sqrt(n))$ to give some idea of the variability.

Rather than just doing one AUC calculation on your full data and saying the AUC is $.77$, you may end up finding your AUC is $.75 +/- .03$, which is much more reliable to make a claim on.  Now in hindsight you can see your single AUC measure was a little optimistically high.


Feature Importance without Random Forest Feature Importances Is their an intuitive way of finding feature importances without just using the random forest feature importances method?
I have a binary logistic regression problem where I have binary features (1 or 0) and a binary target (1 or 0).
I want to see which features are most important towards predicting the target and somehow rank them.
I did an odds ratio for each feature, which gave me some idea of importance.
Are there any other methods?

There are many ways to try to estimate feature importance. Personally I think the random forest measures get overused simply due to the fact that they have “importance” in their name and many people have heard of them. However, what people don’t realize is that those features that the random forest deems important are important for the random forest. They are good at predicting your feature of interest in a random forest setting. Taking this and blindly applying it to non random forest problems is dangerous. What might be predictive for a random forest may not be for some other algorithm. Also these random forest importance measures are not without their faults, for example they are biased towards variables with wide ranges.
There are many other methods available for variable importance such as information gain and relief. I suggest you read this paper by Robnik-Sikonja
https://link.springer.com/content/pdf/10.1007%2F978-3-540-39857-8_30.pdf
It covers many different methods.


Predicted features combined with original ones I am currious  how good  such  a procedure could be :
I  get some predictions  of some 10  learners  trained   on train set  and predicted on train set also .
Then I am column binding   those predictions  to the original train.
Could this  be  a valid procedure  on improving  learning process?

This procedure exists and is called stacked generalization or simply stacking. See stacking section from wikipedia page. Strating from there you can red more by following references from the page. The first paper on subject was published by Wolpert in 1992.

[Later edit]

Do not combine the results with original features, but keep only the predictions combined only with the target.


In clustering, sequence number such as customer ID and dates such as purchase date should be dropped? I am learning K-means clustering and found that in most datasets, there are sequence number such as customer ID and dates such as purchase date.
I don't see any use in them for clustering.
Should I include them for clustering or can simply ignore them?
Let's say other attributes are like purchase amount and number of purchases and etc.

I think you should drop the customer ID since it's a unique identifier and won't give you any additional value.
Instead, I would be rather cautious with the date. The date can be split into year, month and day. Depending on the analysis you are doing, it might be interesting to have K-means running also with those features.
For instance, suppose that you want to check whether you have some blobs of data in a specific period of the year, you should use the month feature.


Sequence of numbers as single feature Is it possible to use a sequence of numbers as one feature?

For example, using libsvm data format:

<label> <index1>:<value1> <index2>:<value2>

+1 1:123.02 2:1.23 3:5.45,2.22,6.76
+1 1:120.12 2:2.23 3:4.98,2.55,4.45
-1 1:199.99 2:2.13 3:4.98,2.22,6.98
...

Is there any special machine learning algorithm for this kind of data?

2 solutions:

You aggregate each sequence of numbers into a single number, which use as a feature. There exist plenty of aggregation functions, such as some derived from descriptive statistics root-mean-square, kurtosis, skewness, max, min, duration, standard deviation, crest factor, mean, or more specific aggregation such as fourier transforms or wavelet transforms.
You use some model that accepts sequences as input. Sequences may be of variable length. Example of such model: recurrent neural networks, Dynamic Bayesian networks.


What's cooking Kaggle - Improve model I'm participating in a Kaggle contest "What's cooking". The goal is to know wich kind of cuisine we have, depending on some ingredients. So it's a multiclass classification problem. I have an existing model, and I have been trying to improve it for 2 weeks without result. I am using scikit learn, and my existing model is

pipelineLR = Pipeline([
('vect', CountVectorizer(max_df=0.75)),
('lr', LogisticRegression(C=0.4,solver='lbfgs',multi_class='ovr',fit_intercept=True,warm_start=True)),
])

It's really basic. But the problem of this dataset is that we have unbalanced data. Also we have a a huge sparse matrix because of the CountVectorizer. I tried a lot of different model, with and without TF-IDF and this model is the best I can get. I have also confusion matrix, classification report and learning curve for it.

What I can notice is that is have high-variance. I think I can do something by preprocessing the data but I am really stuck. I don't know if my model is good enough to be improved, or if I have to find a totally new approach to it. Also what I can get from the classification report is that the model is working well for classes where I have a lot of data, but it's making a lot of mistakes for classes with less data.

I am very new to data science and I am trying to learn by myself. But I think I really need to have another point of view with this problem, because I can't manage to improve my score.

But the problem of this dataset is that we have unbalanced data

I think that the way to fix your problem is to use something like SMOTE or one of its variants. (My favorite is SMOTE-ENN, but go with what works the best). Here is an implementation that you can use to fix the class imbalances. I don't know if this will solve your specific problem, but it is worth a shot.


Is k-means with Mahalanobis a valid option for clustering? I want more info into if k-means with Mahalanobis distance is a mathematically/methodologically correct option for datasets with different variance clusters.
The steps are:

Create aggregate datasets (initially randomly or other way, doesn't matter)
Estimate mu, sigma for each aggregate/cluster dataset
Recompute clusters by calculating Mahalanobis distance of each point to each cluster and updating the clusters.
go to 2 until clusters do not change.

I have seen this implementation used, also have seen it in dissertations. Yet something don't feel comfortable about it.
There is no way to have a 'absolute' best clustering (i.e. silhouette metric) with Mahalanobis.
You can only estimate the error (e.g. Bayesian Information Criterion) of your model (different normal distributions) on your data. And you can definitely over-fit.
Is this still 'correct' in terms of clustering and methodologically accepted?
Is this still termed 'k-means' clustering?

I guess must be valid since I haven't seen any argumentation against it. Yet I feel obliged to ask, just to make sure, before I use it.

Thanks a bunch.

It doesnt feel right cause there are convergence problems. See here

It has tendency to fail. With kmeans and euclidian distance you have some really nice mathematical properties and you can gurantee convergence.


In CNN, why do we increase the number of filters in deeper Convolution layers for complex images? I have been doing this online course Introduction to TensorFlow for AI, ML and DL. Here in one part, they were showing a CNN model for classifying human and horses. In this model, the first Conv2D layer had 16 filters, followed by two more Conv2D layers with 32 and 64 filters respectively. I am not sure how the number of filters is correlated with the deeper convolution layers.

For this you need to understand what filters actually do.
Every layer of filters is there to capture patterns. For example, the first layer of filters captures patterns like edges, corners, dots etc. Subsequent layers combine those patterns to make bigger patterns (like combining edges to make squares, circles, etc.).
Now as we move forward in the layers, the patterns get more complex; hence there are larger combinations of patterns to capture. That's why we increase the filter size in subsequent layers to capture as many combinations as possible.


Generalization Error Definition I was reading about PAC framework and faced the definition of Generalization Error. The book defined it as:

  Given a hypothesis h ∈ H, a target concept c ∈ C, and an underlying distribution
  D, the generalization error or risk of h is defined by

  The generalization error of a hypothesis is not directly accessible to the learner
  since both the distribution D and the target concept c are unknown. However, the
  learner can measure the empirical error of a hypothesis on the labeled sample S.

I can not understand the equation. Can anyone please tell me how it can be interpreted? Also what is x~D?

Edit:
How do I formally write this term? Is something like $$\mathbb{E}_{x \sim D} [1_{h(x)\neq c(x)}] = \int_X 1_{h(\cdot) \neq c(\cdot)} (\omega) dD(\omega)$$
correct or do I need to define some random variable? Also, to show that the empirical error
$$ \hat{R}(h) = \frac{1}{m} \sum_{i =1}^m 1_{h(x_i)\neq c(x_i)} $$
is unbiased, we have
$$\mathbb{E}_{S \sim D^m} [\hat{R}(h)] = \frac{1}{m} \sum_{i =1}^m   \mathbb{E}_{S \sim D^m} ~ \left[ 1_{h(x_i)\neq c(x_i)} \right] = \frac{1}{m} \sum_{i =1}^m   \mathbb{E}_{S \sim D^m} ~ \left[ 1_{h(x)\neq c(x)} \right]$$,
but how do we formally get
$$ \mathbb{E}_{S \sim D^m} ~ \left[ 1_{h(x)\neq c(x)} \right]= \mathbb{E}_{X \sim D} ~ \left[ 1_{h(x)\neq c(x)} \right] = R(h)$$

I think that I understand it intuitionally, but I can't write it down formally. Any help is much appreciated!

There exists somewhere in the world a distribution $D$ from which you can draw some samples $x$. The notation $x \sim D$ simply states that the sample $x$ came from the specific distribution that was noted as $D$ (e.g. Normal or Poisson distributions, but also the possible pixel values of images of beaches).

Say you have some ground truth function, mark it as $c$, that given a sample $x$ gives you its true label (say the value 1). Furthermore, you have some function of your own, $h$ that given some input, it outputs some label.

Now given that, the risk definition is quite intuitive: it simply "counts" the number of times that $c$ and $h$ didn't agree on the label. In order to do that, you (ideally) will

go over every sample $x$ in your distribution (i.e. $x \sim D$).
run it through $c$ (i.e. $c(x)$) and obtain some label $y$.
run it through $h$ (i.e. $h(x)$) and obtain some label $\hat{y}$.
check if $y \neq \hat{y}$. If so, you add 1 to your count (i.e. $1_{h(x) \neq c(x)}$ - that notes the indicator function)

Now last thing to note is that I wrote above "count", but we don't really care if the number is 500 or 100, we care for the relative number of mistakes (like 40% or 5% of the samples that were checked were classified differently). That is why it is noted as expectancy ($\mathbb{E}$).

Let me know if that was clear enough :-)