Small amount of training data set for naive Bayes classifier for binary classification I'm implementing prediction system for young cricketers in ODI format using Naive Bayes classifier. The output of the system is to predict whether the young player is rising star or not. I have collected data from statsguru API of espncricinfo but I'm getting only about 300 records of players from ODI. Is it very small for training dataset?

Actually in machine learning more data equals more accuracy but as you mentioned in the question you had 300 sample dataset.So, the classifier has the little room to decide whom it should select but if you have less number of classes and features you may get better results.When I'm doing my project based on sensor data I used just about 100 sample records from a sensor and it actually predicted very well and got the accuracy of 77 and it has grown once I increased the dataset.but don't train it with the noise like outliers and unwanted features they affect your accuracy a lot


Transformer seq2seq model and loading embeddings from XLM-RoBERTa Is it possible to feed embeddings from XLM- RoBERTa to transformer seq2seq model? I'm working on NMT that translates verbal language sentences to sign language sentences (e.g Input: He sells food. Output (sign language sentence): Food he sells). But I have a very small dataset of sentence pairs - around 1000. And the language is a low resource language.

I am a new researcher on the field of deep learning. Please help with your valuable advice.

It is indeed possible, but the question is if it is a good idea.

FairSeq already contains a pre-trained XLM-R model, you can use by creating a new model: just copy the most suitable existing one and replace the encoder with XLM-R.

Another option would be using Huggingface's Transofrmers that also provides basic support for sequence-to-sequence models as they show in their blog post.

Now, why I don't think it is a good idea. Recent papers show that:

Improving MT with pre-trained Transformers requires a lot of effort and it is still questionable whether it pays off, see Incorporating BERT into Neural Machine Translation
.
The encoder of MT capture different structures: see [What does BERT look at?](What does BERT look at?) and Analyzing Multi-Head Self-Attention:
Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned.
In an end-to-end trained translation system, a large part of the translation process is happening already in the encoder, see Analyzing Word Translation of Transformer Layers.

I assume you only have little training data and therefore you hope that pre-trained representation might help, but I believe you should put more effort in optimizing an end-to-end model to work in low-resource setup (have a look at Revisiting Low-Resource Neural Machine Translation: A Case Study) and data augmentation technique like iterative back-translation that it used in unsupervised machine translation.


R: remove single characters from string I have a huge string in R where I want to remove single characters from. So for example, the string might be

blabla a test this that b á

And the result should be

blabla test this that

How do I do this in R?

This splits the string and filters the substrings by length, no regex:

string <- "blabla a test this that b á"
paste(Filter(function(x) nchar(x) > 1,
             unlist(strsplit(string, " "))),
      collapse=" ")
# [1] "blabla test this that"

(does not work with multiple space characters)


Define a Model for predicting missing values in Data Set I have the following problem: I'm searching for methods to predict randomly missing data in a given dataset.
For example: I have a dataset which contains information about a product. This can be category, brand_name etc. Let's assume there are information missing of brand name for a specific product(4261 rows / 10 000). How can this information be predicted based on the information I have in my dataset?

You could try a classification approach like random forests for categorical data.
For example in your brand name assumption, Take your existing data and treat the brand name as the response variable. So when you give the values of the other features in the entry where the brand name is missing it should try to predict what the brand name is.

The reason I am suggesting random forests is that it uses an ensemble approach. Please explore the working of Random Forests if you don't know the working behind it.


Combining binary features with textual ones I have a binary feature that i want to use it with textual features i.e. unigrams. I use logistic regression and TF/IDF for representing text. So i simply add a unique feature, say ss or oo, to text of each instances. But in practice, i see adding more number of these features to instances, say two oo or ss or more get me a better results. What is the reason? How these weights improve the classification results? Should not logistic regression can get more weights to this features instead of weighting them by hand?

hstak is a good solution. Using this I combined these features.


Geometric weighting of temperature I have a number of weather stations, and I know their positions. I would like to interpolate measurements from them for various other positions as a weighted average of these stations, but of course I need weights for this. I am thinking the most logical choice here from a physics point of view will be weighting by the inverse of the distance to the station, but I haven't quite convinced myself that that is right, and am not sure if I should maybe be using the distance squared, or maybe something in between.

Any comments? Are there any other reasonable alternatives I should be considering?

It sounds like you're describing optimal interpolation (AKA Gauss-Markov analysis, objective analysis, probably others too). This is a solid intro, and this is a powerpoint on the subject. It's a hard process to summarize quickly, but roughly speaking, you're on the right track.

Optimal interpolation is fairly common in meteorology and other environmental sciences, but I'm not convinced it's really the best tool for the job. Any statisticians want to take a crack at it?


Does Karl Pearson correlation indicate linear relationship between two variables ? Or it indicates nonlinear relationship or simply the correlation? Wikipedia and literature do not seem to convey correct interpretation of Karl Pearson correlation. A few authors interpret it as a linear correlation or association. To me it simply tells direction and extent of relation between two variables (one being categories based).

The Pearson correlation coefficient does indeed quantify the linear relationship between two variables.

Have a look at one of the many mathematical formulas to compute it, based on a sample of data from two variables $X$ and $Y$:

I like to read this loosely in terms of variance (or spread across each variable). It is asking, how do $x$ and $y$ scale together? This is the covariance, top part of the fraction. This is scaled by the product of their individual standard deviations, like normalizing the covariance by the magnitude of the individual variations $\rightarrow$ bringing the final value into the range $[-1, +1]$.

The geometric interpretation is also quite intuitive and worth reading.

You could also notice the power of the values. The numerator has $x^1 * y^1$ (so a like a squared term), and the denominator multiplied two terms that are each the square root of a squared term, also meaning $x^1 * y^1$. So the final dimensionality is a linear value, as we divide one squared terms by another. (This naïve argument of course neglects the units of $x$ and $y$.)

Here is a nice comparison of many values to gain an intuition - try imagining the linear line of best fit through each of these sets of points, and then how far away the points are from that line. This further away they are, the larger the value of the Pearson correlation coefficient $\rho$. As with normally regression, the sign of the gradient also gives a sign the $\rho$:

Image source

What value do you think the Pearson Correlation would be for a group of points almost all lying on a vertical line parallel to the y-axis?  :)

Compare correlation to regression here.


Column With Many Missing Values (36%) Hello this is my first machine learning project, I got a dataset with 18.000 rows and I have a column with 4244 values missing.

I don't know why the values are missing since when it's appropriate there's a 0 value in it.
The dtype of the column is int64 I consider this column usable and would like to implement it to the model.

Could you please help me with how to deal with this problem, or lead my to a resource to teach me how to deal with this ?

I don't know why the values are missing since when it's appropriate there's a 0 value in it.

The first step is to check with some SME (Subject Matter Expert) or the Data Custodian. I can't tell you how many times I've built a model/started analysis just to figure out that the data was wrong. Try to figure out the reason behind the Nulls/0.

Besides that there are many ways to handle missing data a few are below:

Remove records with this missing value in your column. If this is an important column to your model it may be best to get rid of that record depending on the shape (rows x cols/features) of your dataset. Don't throw off the results of your model because there's some data that may throw it off (even if you use some of the methods below)
Mean/Median/Mode Impute - A common method of handling missing data is to fill the missing values with the column's mean or median (rarely do you use the Mode).
Fill the values that creates a normal distribution - it depends on your data, but filling the values so you get normally distributed column data can be beneficial
Try all these methods and more - When you start modeling you'll learn to "throw stuff at the wall" and see what sticks. Look at your model results, talk with SMEs, and think about what makes sense. Some ways of handling missing data will work better with different models/datasets. Experiment and have fun!


Bert: fine-tuning the entire pre-trained model end-to-end vs using contextual token vector In the official github page of BERT, it mentions that:

  In certain cases, rather than fine-tuning the entire pre-trained model
  end-to-end, it can be beneficial to obtained pre-trained contextual
      embeddings, which are fixed contextual representations of each input
      token generated from the hidden layers of the pre-trained model. This
      should also mitigate most of the out-of-memory issues.

I am wondering in which cases, using only token vectors, will be more beneficial (other than out of memory issue)?

It goes like this: In the case of fine-tuning, you may want to use huge batch size, which may lead to the Out of Memory (OOM) issues. In such case, you may find the embeddings of BERT useful than fine-tuning on BERT itself. Maybe you can think of using BERT embeddings to train CNN or RNN classifier, in such case you can try to obtain the embeddings of BERT using a small batch size (it can be as low as 1), and then use these embeddings to further train your CNN or RNN classifier.


CNN memory consumption I'd like to be able to estimate whether a proposed model is small enough to be trained on a GPU with a given amount of memory

If I have a simple CNN architecture like this:

Input: 50x50x3
C1: 32 3x3 kernels, with padding (I guess in reality theyre actually 3x3x3 given the input depth?)
P1: 2x2 with stride 2
C2: 64 3x3 kernels, with padding
P2: 2x2 with stride 2
FC: 500 neurons
Output: softmax 10 classes
Mini batch size of 64

Assuming 32bit floating point values, how do you calculate the memory cost of each layer of the network during training? and then the total memory required to train such a model?

I will assume by C1, C2, etc, you mean convolutional layers, and by P1 ,P2 you mean pooling layers, and FC means fully connected layers.
We can calculate the memory required for a forward pass like this:
One image
If you're working with float32 values, then following the link provided above by @Alexandru Burlacu you have:
Input: 50x50x3 = 7,500 = 7.5K
C1: 50x50x32 = 80,000 = 80K
P1: 25x25x32 = 20,000 = 20K
C2: 25x25x64 = 40,000 = 40K
P2: 12x12x64 = 9,216 = 9.2K <- This is a problem (and my approximation is a very hand-wavy guess here). Instead of working with 50, 25, '12.5', it would make more sense to work with multiples of 32. I've heard working with multiples of 32 is also more efficient from a memory standpoint. The reason this is a bad idea is 2x2 pooling doesn't divide the space properly, as far as I can tell. Feel free to correct me if I'm wrong.
FC: 1x500 = 500 = 0.5K
Output: 1 x 10 = 10 = 0.01K (next to nothing)
Total memory: 7.5K + 80K + 20K + 40K + 0.5K = 157.2K * 4 bytes = 628.8 KB
That's for one image.
Minibatch
If you're working with a minibatch size of 64, then you're reading 64 of these into memory at once and performing the operations all together, scaling everything up like this:
Input: 64x50x50x3 = 480,000 = 480K = 0.48M
C1: 64x50x50x32 = 5,120,000 = 5.12M
P1: 64x25x25x32 = 1,280,000 = 1.28M
C2: 64x25x25x64 = 2,560,000 = 2.56M
P2: 64x12x12x64 = 589,824 = 590K = 0.59M
FC: 64x500 = 32,000 = 32K = 0.032M
Output: 1x10x64 = 640 = 0.64K = 0.00064M (we don't care, this is tiny)
Total memory: 10M x 4 bytes ~ 40MB (I'm saying approximate because the website also says an approximate value)
EDIT: I misread the website, sorry.
According to the website, a backward pass requires about triple this, because of the need to store:

the activations and associated gradients for each neuron - these are of equal size;

the gradients of the weights (parameters) which are the same size as the parameters;

the value of the momentum, if you're using it;

some kind of miscellaneous memory (I don't understand this part)


How to know which loss function is suitable for image classification? I am implementing a CNN model for image classification where I am learning about loss functions. There are several types of loss functions to determine error. However, how do we find out which function is suitable for a image multi-classification model?

If you have two classes (i.e. binary classification), you should use a binary crossentropy loss.

If you have more than two you should use a categorical crossentropy loss.


Multivariate Time Series Anomalous Entry Detection I have a multivariate data set of the following structure.

It is a time series sequence of logs with additional string attribute columns id1 and id2. If too many entries come in a sequence that have similar values for either id1 or id2, I want to classify them as anomalies and flag them. I tried using LSTM, but I'm confused in how to use it for textual data, specially when the values that can come in as inputs are not always the same (one time window might have more entries than other).

The similar entries may not always be together but may be very frequent in a specific time window.

I am going to be classifying anomalies at the end of every day in a batch, so I'd like to feed the trained model a bunch of log entries and then find anomalies in it.

I'd appreciate any help w.r.t. to what direction to look in and if LSTM is even required for this.

If too many entries come in a sequence that have similar values for either id1 or id2, I want to classify them as anomalies and flag them (...) I'd appreciate any help w.r.t. to what direction to look in

General Approach

Here's how I would approach this:

Select the time windows, e.g. using pandas.rolling
For each time window, count the number of occurances for each string (per id1, id2)
Compare occurances to usual occurances. Usual here could be in comparision to say the moving average of occurances of the same string in the same window, across many windows, in comparison to the same window/range on the previous day etc.
Identify the anomalies. Anomalies would be those values that occur less or more frequently as "usual" by some threshold (say occurances within 2 standard deviations of the mean is normal, anything else is an anomaly)

A more elaborate and extensive treatment of possible approaches is given in Detecting Anomalies in System log files by Tim Zwietasch.

What algorithm to use?

This can be done with or without a machine learning algorithm:

LSTM might be an option if the process generating these logs can be modelled as some distribution so that an LSTM is able to generate the most likely sequences. In this case an anomaly would be a sequence that has a low probability of being generated by the model.
A simpler ML option would seem to use a "classic" anomaly detection algorithm such as one-class SVM, KNN, K-Means or LOF. A good overview is given in Introduction to Anomaly Detection by datascience.com (I have no affiliation)
Build your own "classifier" using simple statistics.

Personally I would start out with building my own, using a subset of the data, just to understand the data and its patterns better. Then, perhaps, use machine learning for robustness and scalability.

Encoding the log files

Whatever your choice, the key will be in encoding the data. Any algorithm will model the distribution of your data in one way or another (i.e. occurances of values per time window), so you have to present the data in a way suitable to detection of anomalies as per your criteria (anomaly relative to what?).

One encoding might be as follows:

for every time window, build one row of input data
for every value, have one column as the number of occurances of this value
add more columns for any other factor that could be influecing what is "normal" e.g. hour of day, day of week, week of year

Dealing with similar values

Note I did not yet treat "similar values" but assumed that the occurance counts are for exact values. However it should be relatively easy to extend the above to similar values by creating columns of occurance patterns as opposed to specific values. A pattern in this case could be e.g. all strings that contain some substring, say 'aaa' - or you could use a machine learning model to create clusters of value. Then use counts by patterns/clusters as the input instead of counts per value.


Why is the Naive Bayes classifier of sklearn faster than sklearns SVM? I've used scikit-learn in Python to compare results of naive Bayes and SVM. I've found that naive Bayes is quicker than SVM. Could anyone shed some light on reasons for such finding?

The differences in speed between Naive Bayes and SVM simply boils down to the formulation and the assumptions of each model, and has little to do with the particular library or implementation.

Not only is naive bayes a simple probabilistic classifier, it also makes an additional assumption of independence between its features, so that parameter estimates can be calculated independently and thus possibly very quickly.

In comparison, with SVM, every single record in the data base will require a computation of the distance function to determine the optimal decision boundary.

We can observe that for SVM, we have a complexity of $O(\text{#features} \times \text{#observations}^2)$ or more depending on the choice of SVM and kernel. On the other hand, for naive bayes we would have a complexity of $O(\text{#features} \times \text{#observations})$.

Hence in general, we can say that SVM will take longer than Naive Bayes.


Methods for augmenting binary datasets I have a small (~100 samples) dataset with roughly 20 features which are mostly binary, and a few are numeric (~5). I wanted to use methods for augmenting the training set and see if I can get better test accuracy. What methods/code can I use for augmenting binary datasets?

You may try Metropolis–Hastings sampling if multi-dimensional, or else Adaptive rejection, unless you've already tried.


Effect of outliers on Naive Bayes Are Naive Bayes algorithms affected by outliers in the data? Suppose there is a data set, does one need to remove outliers before applying Naive Bayes?

There are different flavors of Naive Bayes, so the answer depends a bit on the use case.

One potential issue with outliers is that unseen observations can lead to 0 probabilities. For example, Bernoulli Naive Bayes applied to word features will always produce 0 probabilities when it encounters a word that wasn't seen in the training data. Outliers in this sense can be a problem. However, all these and similar issues of Naive Bayes have well-known solutions (like Laplace smoothing, i.e. adding an artificial count for every word) and are routinely implemented.

In Gaussian Naive Bayes, outliers will affect the shape of the Gaussian distribution and have the usual effects on the mean etc.

So depending on your use case, it still makes sense to remove outliers.


Installing Orange Package with older Python Version? I am trying to install an Add-on package from GitHub that contains prototype widgets for Orange Data Mining. I am trying to install it from the GitHub page found here.
I am using the following Terminal code to install this:
git clone http://github.com/biolab/orange3-prototypes.git

Everything then appears to install correctly and the download shows 100%. Then, however, it throws an error and says:

Orange requires Python >= 3.4

I am using Mac OS. Clearly, it is suggesting that I need to use a different version of python to install, however, I already updated my pip install. Any insight into how I can fix this?

The error message shows exactly what you need to do:

Orange requires Python>=3.4

You have to specify a Python>=3.4 version (with orange3 itself installed) while installing orange3-prototypes.
I'm not a Mac user, however, in Windows, Orange3 installer will automatically install a Python 3.4 if there's no compatible Python available.


Classifying points exactly on decision boundary For calculating loss arose by classification we do this:

If $y (w \cdot x + b) > 0$: $\text{no loss}$

If $y (w \cdot x + b) < 0$: $\text{loss} = −y (w \cdot x + b)$

So here what about the points exactly on decision boundary? How do we classify and compute their loss? (since it would become 0)

It is up to the user how they want to handle such cases. The loss for the points on the decision boundary will be 0. It's a design decision the user has to take. One example could be to consider such cases in misclassification and update your weights while training.

If $y (w \cdot x + b) > 0$: $\text{no loss}$

If $y (w \cdot x + b) \leq 0$: $\text{loss} = −y (w \cdot x + b)$

During inference, if such cases arise, you can call a random classifier to classify it in one of the classes.


Find effective feature on machine learning classification task with scikit-learn I'm tackling a binary classification task using SVM implemented in python scikit-learn. Datasize is around 10,000 and the number of feature is 34.
After finding nice parameter set (using RandomizedSearchCV class), I evaluate the model by the cross validation. The result seems nice.
criteria_list = ["precision", "recall", "f1", "roc_auc"]
score_df = []
score_df2 = []
clf = svm.SVC(**random_search_clf.best_estimator_.get_params())
for crit in criteria_list:
    scores = cross_validation.cross_val_score(clf, X, y, cv=3, scoring=crit)
    score_df.append(["{} (±{})".format(np.round(np.mean(scores),3), np.round(np.std(scores),4)), scores])
    score_df2.append(["{} (±{})".format(np.round(np.mean(scores),3), np.round(np.std(scores),4))])

pd.DataFrame(np.transpose(score_df2), columns=criteria_list, index=["SVM"])

My question is whether it is possible to find out which feature is effective to classify the test data. I thought it's relating to sensitivity analysis, but good answer cannot be shown by googling "sensitivity analysis + svm" or "sensitivity analysis + scikit learn".

Example code of the nice suggestion from stmax above, with modification to use RandomForest and match the questions sample size and number of features, I hope that helps:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier

X, y = make_classification(n_samples=10000,
                           n_features=34,
                           n_informative=10,
                           n_redundant=0,
                           n_repeated=0,
                           n_classes=2,
                           random_state=0,
                           shuffle=False)

forest = RandomForestClassifier(n_estimators=250,
                              random_state=0)

forest.fit(X, y)
importances = forest.feature_importances_
std = np.std([tree.feature_importances_ for tree in forest.estimators_],
             axis=0)
indices = np.argsort(importances)[::-1]

# Print the feature ranking
print("Feature ranking:")

for f in range(X.shape[1]):
    print("%d. feature %d (%f)" % (f + 1, indices[f], importances[indices[f]]))

# Plot the feature importances of the forest
plt.figure(figsize=(20,10))
plt.title("Feature importances")
plt.bar(range(X.shape[1]), importances[indices],
       color="g", yerr=std[indices], align="center")
plt.xticks(range(X.shape[1]), indices,rotation=60)
plt.xlim([-1, X.shape[1]])
plt.show()


Heterogeneous clustering with text data I have a dataset which consists of multiple user ratings. Each rating looks similarly to:

| Taste | Flavour | Look | Enjoyed | ..... | Tag    |
|-------|---------|------|---------|-------|--------|
| 4     | 2       | 2    | 3       | ..... | Banana |
| 5     | 4       | 1    | 2       | ..... | Apple  |
| 3     | 1       | 4    | 1       | ..... | Pasta  |
| ....  | ....    | .... | ....    | ....  | ....   |

The columns contain ranks for each row. The task is to clusterize rows,
e.g. I would like to find something similar to:

cluster 1: Banana, Apple
cluster 2: Pasta, Spagetty
....

We use HDBSCAN with edit distance metric to find clusters, and it works more or less.
The problem, however, is that there are too few features (12 in total) to have "good" clusters.
Therefore I would like to somehow account for the information from "Tag"
in clustering. The idea is to calculate embeddings for each tag and use them as features.

What I'm not certain about is how to include these new features? I would
like the clustering to be primarily determined by the original features. The dimension
of embeddings is much larger than the dimension of the original features, and
the metric on these features is different (e.g. cosine similarity). Therefore, I would like to answer 2 questions:

What will be a proper method to combine these heterogeneous features?
How to properly select the weight for the "Tag" feature? Ideally, I would not like to just postulate it

TL;DR

Pass the data twice in HDBSCAN.

Cluster based on the tag using word embeddings and cosine distance.
Sub-cluster the cluster from step 1 with your existing method (using the remaining 12 features).

Explanation

I suggest you do it in two steps and not give a weight to the tag feature, because the distance metric used is different. For word-embeddings, you will need to use the cosine distance between embeddings. Where as for the other 12 features you are currently using another distance (Euclidean I assume).

The first step should cluster your data on the semantical characteristic of your tag. This should cluster things like fruits, meats, vegetables, pastas...

Then, the second step can sub-cluster the data with your other 12 features. However, given your example

  cluster 1: Banana, Apple

  cluster 2: Pasta, Spaghetti

I don't see why this second step is necessary. You could instead of clustering a second time, just use the 12 features for ordering the data points for the purpose of your exercise. E.g. getting the top "fruits", as clustered at step 1, that people "enjoy" the most.


p1 = (X.dot(X.T))**-1 needs to be p1 = np.linalg.inv((X.T.dot(X)))

X**-1 doesn't invert a matrix, this is equivalent to 1/X (elementwise) and you need to transpose the first not the second matrix.


How does epochs related with converging the model? I have read on Internet that epochs is used to give the time for the model to converge but I don't know how ? . I was thinking that epochs is used because to train the model sufficient times . How does model convergence relates with epochs . Also tell me that why epochs is useful ?

I don't know if I understood correctly the question, but basically, epochs are the unit of measure of the time spent on training the machine learning models.
An epoch is conventionally considered as a full pass on the dataset, meaning that for each epoch, you feed the data to the model from the beginning to the end of the dataset. Once an epoch is complete, you can start again another epoch training your model from the beginning of the dataset to the end, and so on and so forth.
The convergence of the performance of the models is of course dependant on the epochs, because usually what happens is that the more epochs you train your model, the less noisy (more stable) will be the predictions.
Hope I answered your question.


Dummy coding a column in R with multiple levels I have a dependent variable measuring the net revenue. One of the major predictor affecting this is "product" i.e. the product sold to the customer. My randomly sampled dataset contains 1.4 million entries.

Products are assigned a specific categorical value.
I feel that using dummy variables to represent the products would be apt however, there are 4481 levels of products. I do not know how to code so many levels in R.

model.matrix(~ product, data=salesdata)
 returns an error. (Needs 38.4GB of memory)

Can someone guide me a little on how to code these categorical variables?

Dependent: Net revenue (quantitative)
Independent: Product code (quantitative but treated as qualitative since values are nominal)

You can use either sparse matrices or feature hashing.

Sparse Matrix

I suppose that using a sparse matrix is the only choice. I suspect that this line of code will work. This uses the Matrix package.

sparseProducts <- sparse.model.matrix(~ product, data=salesdata)

Take my example:

sparseDiagonalMatrix <- sparse.model.matrix(~., data.frame(V1 = as.factor(seq(1, 10))))

each column represents a different factor, this will yield:

1  1 . . . . . . . . .
2  1 1 . . . . . . . .
3  1 . 1 . . . . . . .
4  1 . . 1 . . . . . .
5  1 . . . 1 . . . . .
6  1 . . . . 1 . . . .
7  1 . . . . . 1 . . .
8  1 . . . . . . 1 . .
9  1 . . . . . . . 1 .
10 1 . . . . . . . . 1

> class(sparseDiagonalMatrix)
[1] "dgCMatrix"
attr(,"package")
[1] "Matrix"

alternatively you can remove the intercept and have all zeros represent class 1

sparseDiagonalMatrix <- sparse.model.matrix(~., data.frame(V1 = as.factor(seq(1, 10))))[, -1, drop=FALSE]

10 x 9 sparse Matrix of class "dgCMatrix"
V12 V13 V14 V15 V16 V17 V18 V19 V110
1    .   .   .   .   .   .   .   .    .
2    1   .   .   .   .   .   .   .    .
3    .   1   .   .   .   .   .   .    .
4    .   .   1   .   .   .   .   .    .
5    .   .   .   1   .   .   .   .    .
6    .   .   .   .   1   .   .   .    .
7    .   .   .   .   .   1   .   .    .
8    .   .   .   .   .   .   1   .    .
9    .   .   .   .   .   .   .   1    .
10   .   .   .   .   .   .   .   .    1

> class(sparseDiagonalMatrix)
[1] "dgCMatrix"
attr(,"package")
[1] "Matrix"

You will need a package that supports sparse matrices for measuring the net revenue though. Fortunately, most modern mainstream packages support sparse matrices.

Feature Hashing

Here is a great explanation of feature hashing in R (among other techniques) which is also an alternative, specially useful when you have hundreds of thousands or millions of multiple levels.

https://amunategui.github.io/feature-hashing/


Are there any open-source text annotation for multi label classification tools? I have a large texts in each document and I want to know if there are any open source text annotation tools available online for multiple label annotation.

Each sentence takes two labels. If there are any please let me know.

There are many manual text annotation tools available, but you will probably have to search around in order to find the one which suits your precise needs.
Here are a few pointers:

Gate
Text annotation tools
Doccano
a recent review


Keras early stopping to a target I'm really struggling to understand how the parameters of Keras early stopping callbacks play out, especially in the presence of a baseline.  What I want is simply for the training to stop within 2 epochs of the validation accuracy hitting 95%.  So I try:
(trainX, testX, trainY, testY) = train_test_split(Tensor, Labels1Hot, test_size=0.2)
when2stop = EarlyStopping(mode='max',monitor='val_accuracy',verbose=1,patience=2,baseline=0.95)
history = model.fit(trainX, trainY, epochs = 100, batch_size = 500, validation_data = (testX, testY), callbacks=when2stop)

This just stops after just 2 epochs, even if the val accuracy has in fact improved. But the val accuracy is way off 0.95. Is this a bug or am I mis-understanding the baseline and patience settings?
Using: Tensorflow 2.4

Comment above to SO post is spot on. A custom callback, as provided in that post, is the solution and the term 'baseline' is not meant to be interpreted as a threshold.


Which tool is good to collect tweets on 50 keywords over the last 5 years and then analyze them with the LDA algorithm or sentiment analysis? I want to find tweets from the last 5 years to a topic. For this I decide for 50 Keywords (related to the main topic), where I want to find data on Twitter. I want to find out how the trend on the subject is, are there any opposing/negative opinions, are there recognizable patterns, ... (I am also unsure which possibilities I have to get something useful from the data)
I totally new to Data Science and never done it before.
I have found the following tool https://algorithmia.com/pricing and https://monkeylearn.com/ . Especially Algorithmia looks like it could help me. Are there other alternatives that could help me with this task? Since my programming skills are not very good and my data science skills are not existing, I thought it would be the easiest thing to buy a tool.
I am happy about any advice. :)

As for Algorithmia and monkeylearn, I don't know if you will find much help for those here due the fact that Python and R/RStudio are the predominant 'data science' open source (read: free) platforms. Otherwise, if Algorithmia or monkeylearn has tutorials go for it.
The project you are interested in doing is not trivial (with no experience) but completely within the grasp of a novice given enough time. May I suggest starting with R/RStudio. This is simply my opinion based on my learning path. "Your mileage may vary!" Then find some tutorials such as Towards Data Science and see how it goes.


Pandas column count mismatch after insert into dataframe I am trying to insert a column of values from one dataframe to another. The total count of values of both the indiviual dataframes is 91744 but when I use

df1.insert(loc=0, column='x1', value=ts)

to insert the values of ts as the first column in df1, the value count after insertion turns out to be 84175 instead of actual 91744. Please tell me what is wrong with this.

>>> df1.insert(loc=0, column='x1', value=ts)
>>> df1.x1.count()
84175
>>> ts.count()
0    91744
dtype: int64

>>> df1.describe()
                 x1            x2
count  84175.000000  91744.000000
mean       0.249044     11.744387
std        0.191239     13.294185
min        0.000000      1.000000
25%        0.108383      3.000000
50%        0.212760     12.000000
75%        0.311860     15.000000
max        1.000000    551.000000

Sample data is given below.

>>> df1.head()
   x2  x3  x4   x5                   x6  y
0  12   1  13  497  2017-11-07 09:30:38  0
1  25   1  17  259  2017-11-07 13:40:27  0
2  12   1  19  212  2017-11-07 18:05:24  0
3  13   1  13  477  2017-11-07 04:58:08  0
4  12   1   1  178  2017-11-09 09:00:09  0
>>> ts.head()
         x1
0  0.218561
1  0.269357
2  0.257698
3  0.238417
4  0.164645

My objective was to add the x1 column from of ts dataframe as the first column of df1 data frame.

Can you share some of your data frame i would recommend using merge instead of insert if you have a common column let's say ID for example your code will look like this:

mergedStuff = pd.merge(df1, df2, on=['id'], how='outer')
mergedStuff.head()

as simple as that


How to Build Mobile Application for Image Recognition? I want to write an application on (Android) phone for image recognition. The (Keras) model itself is written and trained on a desktop machine and works satisfactorily with standard images. However, I have no experience with app programing so I have no clue how to write an app itself which could utilize the model.

The purpose is to use the mobile camera for seeking the object; have a big button to shot an image; have the image classified and display the output label as text. No need for any sophisticated outlook. I can write programs in Python.

Is there any handy framework that could be used for this purpose?

I would advise you to use kivy for Python. It has an active community and there is also a book on this topic Practical Computer Vision Applications Using Deep Learning with CNNs With Detailed Examples in Python Using TensorFlow and Kivy. Kivy is very easy for newbies and you can develop multiplatform applications (Windows, iOs, Android).


Is it possible to call from Keras unsupported backend function directly from tensorflow? I am working to create a Keras custom layer. this layer needs to use a function that doesn't exist in Keras TF backend, although it exist in TF.

What would be the best way to solve it?

Is there a way to call TF function directly from Keras and not from backend?
Should I work to modify backend to include a new unsupported function?

Any example or pointer how to implement the best solution.

Thank you

There is no possible way to directly call tensorflow functions from keras. Keras is a stand-alone project. Although, tensorflow >= 1.12 has keras built in, so you could play with the source of tensorflow and figure it out.


Is Python a viable language to do statistical analysis in? I originally came from R, but Python seems to be the more common language these days.  Ideally, I would do all my coding in Python as the syntax is easier and I've had more real life experience using it - and switching back and forth is a pain.
Out side of ML type stuff, all of the statistical analysis I've done have been in R - like regressions, time series, ANOVA, logistic regression etc.  I have never really done that type of stuff in Python.  However, I am trying to create a bunch of code templates for myself, and before I start, I would like to know if Python is deep enough to completely replace R as my language of choice.  I eventually do plan on moving more towards ML, and I know Python can do that, and eventually I would imagine I have to go to a more base language like C++.
Anyone know what are the limitations of Python when it comes to statistical analysis or has as link to the pros and cons of using R vs. Python as the main language for statistical analysis?

Python is more "general purpose" while R has a clear(er) focus on statistics. However, most (if not all) things you can do in R can be done in Python as well. The difference is that you need to use additional packages in Python for some things you can do in base R.
Some examples:

Data frames are base R while you need to use Pandas in Python.
Linear models (lm) are base R while you need to use statsmodels or scikit in Python. There are important conceptional differences to be considered.
For some rather basic mathematical operations you would need to use numpy.

Overall this leads to some additional effort (and knowledge) needed to work fluently in Python. I personally often feel more comfortable working with base R since I feel like being "closer to the data" in (base) R.
However, in other cases, e.g. when I use boosting or neural nets, Python seems to have an advantage over R. Many algorithms are developed in C++ (e.g. Keras, LightGBM) and adapted to Python and (often later to) R. At least when you work with Windows, this often works better with Python. You can use things like Tensorflow/Keras, LightGBM, Catboost in R, but it sometimes can be daunting to get the additional package running in R (especially with GPU support).
Many packages (or methods) are available for R and Python, such as GLMnet (for R / for Python). You can also see based on the Labs of "Introduction to Statistical Learning" - which are available for R and for Python as well - that there is not so much of a difference between the two languages in terms of what you can do. The difference is more like how things are done.
Finally, since Python is more "general purpose" than R (at least in my view), there are interesting and funny things you can do with Python (beyond statistics) which you cannot do with R (at least it is harder).


Tools to do Semantic Anotation on 3d CAD Models I am trying to do Segmentation on my custom dataset consisting of 3D CAD Models.
However, I am not able to find the proper tools to generate my dataset for Segmentation.
I am able to find the existing dataset with segmentation information, but am not able to find any information on how to create one for our own CAD Models.
I need to train the model on my custom dataset hence creating one.
I will really appreciate any help that is provided.

I would like to update that I was able to perform annotation on the 3d models by using following tools

Semantic Segmentation Editor
Blender
FreeCAD

Blender and FreeCAD are easy to use, but needs some postprocessing to get the annotations right especially on point cloud data, whereas Semantic Segmentation Editor can be used to directly annotate the point cloud but that stuff is not easy.


Where can I find out about the "Helvetica scenario"? From the paper introducing GANs:

It makes sense that collapsing too many $\vec{z}$-values to a single $\vec{x}$-value will cause problems. However, I was a bit confused as to how training $G$ for a shorter amount of time before running $D$ can fix it. But when I googled "Helvetica scenario", all I could find were references to a parody show called Look Around You.

Since it seems like this is an actual thing, not just a parody—where can I find out more about it, and how it can be prevented? And why is it called "the Helvetica scenario"?

Placing too much probability mass on a single $x$ is indeed a major problem with GANs. Usually it is referred to as mode collapse. Most GAN papers will discuss it.

  I was a bit confused as to how training  for a shorter amount of time before running  can fix it.

The idea is to keep the discriminator $D$ up to date with the generator $G$. By training $G$ less, you are training $D$ more. Usually, an alternating optimization is being done with GANs, i.e. train $D$ for $n$ iterations, then $G$ for $m$ iterations, and so on. The "job" of $D$ is to be able to discern fake $G$ outputs from real data points. But, if you don't train $D$ enough (i.e., you train $G$ too much, so $m>n$), then $G$ will quickly "run off" and exploit local minima in $D$. From $G$'s point of view, if $D$ "likes" some output $x=G(z)$, then the easiest thing to do is just only output that $x$! Training $G$ less (and thus $D$ more) allows this minima to move, and prevents $G$ from exploiting them.

Notice that overtraining $D$ doesn't seem to be a bad thing a priori.
Indeed, with more modern GAN losses (e.g., Wasserstein ones), one wants to $n$ to be as large as possible.
It just means that $D$ converges to being a great discriminator at every step, so it should be informative for $G$. (Indeed, proofs of convergence hinge on doing this!) But in practice, for the original GAN, overtraining $D$ tends to lead to useless gradients (see below): nothing $G$ can do will please the discriminator, so all weight updates are useless, so the GAN just futilely flails around parameter space). This flailing is called unstable training by others. :)

In other words, for the vanilla GAN, undertraining $D$ can lead to mode collapse, while overtraining it can cause instability. People seemingly have focused on mitigating instability so that one simply overtrains $D$ and avoids mode collapse.

  how it can be prevented?

One common way to mitigate it is to use a different loss function, such as a Wasserstein loss. One problem with the original GAN formulation is that (by minimizing the JS divergence) can have very uninformative gradients because the loss does not handle distributions with very little overlap very well (i.e., when the shared support is small). In some sense, it is too harsh when the generator is doing poorly. In contrast, an optimal transport loss smoothly decreases even when the the two distributions are far from each other. See e.g. the Wasserstein GAN paper and its followups.

  And why is it called "the Helvetica scenario"?

It refers to the show you mentioned, where too much Calcium is put in one place and causes a disaster. Here, a disaster occurs when too much generator probability density is placed in a small area of the data space. :)


how to prepare data for cross validation in mnist dataset? How to use k-fold cross validation for MNIST dataset?
I read article documentation on sci-kit learn ,in that example they used the whole iris dataset for cross validation.

from sklearn.model_selection import cross_val_score
clf = svm.SVC(kernel='linear', C=1)
scores = cross_val_score(clf, iris.data, iris.target, cv=5)
scores

for example while importing mnist dataset in keras

from keras.datasets import mnist
(Xtrain,Ytrain),(Xtest,Ytest)=mnist_load()

in this dataset is already divided in test and train , so to apply cross validation on the entire dataset do we need to make Xtrain and Xtest as one entity to exploit the whole data.

from sklearn.model_selection import cross_val_score
clf = svm.SVC(kernel='linear', C=1)
scores = cross_val_score(clf, iris.data, iris.target, cv=5)
scores

They are not using the whole data for cross validation as such ( it's just an illusion)

When the cv argument is an integer, cross_val_score uses the KFold or StratifiedKFold strategies by default, the latter being used if the estimator derives from ClassifierMixin..

So it's kind of automated inside the call..

Check this kaggle kernel link


Do you know a dataset for regression where deep learning outperforms svm and random forests? I am currently trying to learn about deep learning. I asked myself where deep learning outperforms SVM and Random Forest on regression?

Do you have any dataset for regression where deep learning outperforms SVM and Random Forest?

My main objective for asking this question, is to see on which kind of datasets deep learning outperforms SVM and Random Forests. Since I do not know what kind of datasets these are, I do not want to be too restrictive when asking about these datasets.

Update:
From the answer given by tom, it gets clear to me, what kind of dataset I am looking for. So to be more specific: If you know a dataset with spatial/temporal structure at different levels of granularity which is known that for example convolutional deep networks outperform SVM and Random Forest on this dataset, then please share this dataset or a link to it.

Don't know why the question is still marked as on hold for being to broad, as I updated the question.

Thank you for your help.

As a general rule, convolutional deep networks will perform better than SVM and Random Forest models on data that has hierarchical spatial/temporal structure, so for example sound, images, video.  Deep networks are very good at learning this spatial/temporal structure at different levels of granularity.  For classic datasets like Iris uses for teaching ML, however, I seriously doubt deep networks would perform better than SVM or Random Forest.


Measuring similarity for sets with same cardinality The Jaccard coefficient measures similarity between finite sample sets, and is defined as the size of the intersection divided by the size of the union of the sample sets.

I had 100 sets all of same cardinality. By mistake I calculated the similarity measures as the ratio of intersection with total elements in a set (i.e 100).

This gives different similarity values than the original Jaccard formula.

I was wondering if the original formula has considered the union of two sets to handle cases where there might be sets with different cardinalities.

I think though numerically my values are different, they repersent the same idea.

If anybody could verify/disverify what I am trying to do ?

Yes, the Jaccard similarity score is normalized by the union to deal with sets of different cardinality. Without this normalization (if you used just the intersection), very small sets would always have very low scores.

When the cardinalities of all your sets are the same, the union of any two sets will be a straightforward function of the intersection (this is easy to visualize - as the two sets intersect more and more, their unions get smaller and smaller). The formula is:

union = 2 * cardinality - intersection

So the Jaccard score in your case would be:

intersection / (200 - intersection)

If you plot this, you'll see it's monotonically the same as what you did.


tensorflow simple regression nan after >29 observations I have code as below. If the number of data points changed to any number above 30 (example 40) then i get nan for values of final_slope , final_intercept
why?

For 25 examples it runs fine. I am using a cpu version on tensorflow on my windows machine.

The number of datapoints can be changed by changing number on the line 4th line
n= 40

import numpy as np
import tensorflow as tf

n= 40
x_data = np.linspace(0,10,n) + np.random.uniform(-1.5,1.5,n)
y_label = np.linspace(0,10,n) + np.random.uniform(-1.5,1.5,n)

import matplotlib.pyplot as plt
#%matplotlib inline
plt.plot(x_data,y_label,'*')

m = tf.Variable(0.39)
b = tf.Variable(0.2)

error = 0

for x,y in zip(x_data,y_label):

    y_hat = m*x + b  #Our predicted value

    error += (y-y_hat)**2 # The cost we want to minimize (we'll need to use an optimization function for the minimization!)

optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001)
train = optimizer.minimize(error)

init = tf.global_variables_initializer()

with tf.Session() as sess:

    sess.run(init)

    epochs = 1000

    for i in range(epochs):

        sess.run(train)

    # Fetch Back Results
    final_slope , final_intercept = sess.run([m,b])

print (final_slope , final_intercept)

The way you calculate the 'error' function is wrong.
You aggregated error value whenever you train the model. Therefore, the error value increases over time and reaches infinity (you can print the error value at each epoch to check that).
The error function should be computed as follows.

  error = tf.reduce_mean((y_hat - y_label)**2)

By the way, you can check the error value at each epoch.
Choose the hyperparameters of the model to make sure that the error value decreases in the time span.
Here is the error curve over the epoch span.


How to aasign a new data point to a cluster? I have a userdataset which contains fields like ['age','gender', 'computer_literacy', 'vision', 'colour_blind', 'education', 'font_size','colour'].
I clustered these data and assigned the new cluster to the existing data. Now I want to know how can I find a similar cluster for a new user?
As an example, If I am a new user I only submit few details like ['age', 'gender', 'computer_literacy', 'vision'] and I want to know, for which cluster I belong.
What are the possible approaches to solve this problem?

Suppose if  you use kmeans clustering then you can
1.train and  save the model using pickle
2.loa the model using pickle
3.pass your new sample as a vector to the predict function of the loaded model  object
model.predict([[0, 0], [12, 3]])
this will give you only one cluster label
4.if you want to get  top n clusters that the sample might belong to then take the cluster centers of your model as save it in a variable this will be list of list or matrix .
cc=kmeans.cluster_centers_
compute the similarity between your new sample and the cc matrix , rank the distances and yo will get n nearest clusters


Estimating data set size for grammar extraction I have a dependency treebank including 100 sentences, which I divide into a training set and a test set. I extract some rules ((DS,PS) pairs) to convert the treebank to phrase structures. When I extract such rules from the training set, I can measure the percentage of rules (DS patterns) that cover the test set, suppose

(10, 24%), (20, 34%), (30,40%), (40,44%), (50, 55%),(60, 58%), (70, 61%)...

As you see as I increase the size of the training set, the coverage of extracted patterns increases! however its not linear!, I want to see how many data I need to reach 100% coverage?
I guess I can use a regression, but which regression? logarithmic?

Is this related to 'learning curve'? if yes how can I use regression for a learning curve?

You can never get 100% coverage for real-world grammar extraction. Grammar is complex and undefined for real-world data. In addition, it is an "open world" data problem because novel grammar phrases can always be created.

You might get 100% coverage for a small data set through overfitting.

Statistical learning theory provides a framework for predicting the limits of machine learning.


How to determine if dataset is a suitable representation of the context? How can I determine if the data I have collected is a good enough representation of the context? For example, I am working on an object detection system and have been building an image dataset. How can I know if my dataset represents the task? For example, I need to account for instances where the object is close up and far away. But what about contexts/situations I have missed, or not even considered? Is there an art to building datasets?

Before collecting the data, domain knowledge is used to determine plausible variations the task may present. It is not necessary to capture in data all the variations experts may point out (for example, some may be syntheticaly constructed). But domain knowledge indeed dictates the variations a certain task may have to take account for.
That being said, one then gathers the data trying to match plausible variations in the percent that domain experts dictate. Again some may even be syntheticaly constructed a-posteriori. Then we have a representative dataset.
In case the dataset proves inadequate, this means domain knowledge needs to be updated with new information about the task.


sklearn preprocessing MinMaxScaler I am reading, a machinelearningmastery.com post about improving neural network performance, and I am attempting to normalize my own data set with the tips mentioned in the post using sklearn preprocessing MinMaxScaler.

In my code I am reading a CSV file directly into pandas.

#read CSV file
df = pd.read_csv('C:\\Users\\desktop\\EC\\data.csv', index_col='Date', parse_dates=True)

The snip below is what the data looks like with column names:

From the post, I normalize the data with this code below. The process appears to just return a numpy array, but I use Pandas during the machine learning fit process.

from sklearn.preprocessing import MinMaxScaler

# create scaler
scaler = MinMaxScaler()

# fit and transform in one step
df2 = scaler.fit_transform(df)

df2 = pd.DataFrame(df2)

What's happening, is my column names are stripped away and I use column names a lot in dropping & selecting. For example, I fit a lot of models like this process below to differentiate target and input variables.

#Test random Forest

import numpy as np
from sklearn import preprocessing, cross_validation, neighbors
import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.externals import joblib
import math

rmses = []
for i in range(2):

    X = np.array(df2.drop(['Demand'],1))
    y = np.array(df2['Demand'])

    offset = int(X.shape[0] * 0.7)
    X_train, y_train = X[:offset], y[:offset]
    X_test, y_test = X[offset:], y[offset:]

    clf = RandomForestRegressor(n_estimators=120, min_samples_split=20)

    clf.fit(X_train, y_train)
    mse = mean_squared_error(y_test, clf.predict(X_test))
    rmse = math.sqrt(mse)
    print("rmse: %.4f" % rmse)
    rmses.append(rmse)

print(sum(rmses)/len(rmses))

joblib.dump(clf, 'rfrModel.pkl')

But maybe this isn't a bid deal... Sorry not a lot of wisdom here any tips help. Is there another method to madness that I don't need to dependent on column names? If the df2 = scaler.fit_transform(df) leaves columns in place but just removes names, I could just use a column number to .drop - differentiate target & input variables.. Demand is the name of my target variable, and I could just call the second column..., right??

You can just use the previous column names for creating a new dataframe

# Use the previous column names from df and assign it to df2
df2 = pd.DataFrame(df2, columns=df.columns)


Estimating location in a model I have a big dataset with 10 columns and about a 100,000 rows. Each 5 rows represent a person being tracked and the data related to this tracking such as time, velocity, etc. the last two columns are the longitude and latitude for that person.

To test the model, the test set has the fifth row for each person missing in longitude and latitude. What's the best way to approach this problem?

for example the test set looks like:

id   time    feature2  feature3  long    lat
1      x          x        x     number  number
1      x          x        x     number  number
1      x          x        x     number  number
1      x          x        x     number  number
1      x          x        x
2      x          x        x     number  number
2      x          x        x     number  number
2      x          x        x     number  number
2      x          x        x     number  number
2      x          x        x

etc

One option would be to cluster the longitude and latitude. Point estimates based on longitude and latitude would be wrong much of the time. Clustering would lower the precision of the data to increase the chance of the model being approximately correct.
Longitude and latitude can be clustering using spatially-aware indexing such as H3. Spatially-aware indexing allows for different size bins.


Is there an asymmetric version of nominal correlation? I use Cramer's V to calculate correlation of features in a dataset made of only nominal features.

Let's consider the following dataset:

a  |  b
--------
0  |  0
0  |  1
0  |  0
1  |  2
1  |  2
1  |  3

Calculating Cramer's V for features a and b yields 0.707. Since it's symmetric, there's information loss in this case - as we can see, knowing the value of b means we know for sure what is the value of a, but this is no the case if we are given the value of a; in this case, the number possible values of b decreases, but it's still not known for sure.

I'd like to find an asymmetric metric that will provide this information for nominal values - meaning, will give a different value when calculated a -> b and b -> a. Is there anything like this?

I found what I was looking for - it's called Theil's U, or the Uncertainty Coefficient.

I've used it in this Kaggle kernel, you can check it out for an example and code implementation in Python

EDIT: I also have a blogpost about it.


Probability of dropout growth In the DNN literature, is there analysis or a term on a dropout ratio (oppositely-)proportional to the depth of a layer?

By intuition, I'd like to dropout fewer neurons on the layers next to the input and drop more when approaching the end layers. For example, passing from a p_keep = 0.8 to p_keep=0.5, especially when the inputs features are all relevant. Where p_keep is the probability of keeping a node.

For instance:

Layer1: p_keep=0.9
Layer2: p_keep=0.7
Layer3: p_keep=0.6
Layer4: p_keep=0.55
Layer5-n: p_keep=0.5

Hinton advocates tuning dropout in conjunction with tuning the size of your hidden layer. Increase your hidden layer size(s) with dropout turned off until you perfectly fit your data. Then, using the same hidden layer size, train with dropout turned on. This should be a nearly optimal configuration

  A good initial configuration for the hidden layers is 50%. If applying dropout to an input layer, it's best to not exceed 25%.

By intuition I'd like to dropout fewer neurons on the layers next to the input and drop more when approaching the end layers

If I were to try to generalize, I'd say that it's all about balancing an increase in the number of parameters of your network without over-fitting. So if e.g. you start with a reasonable architecture and amount of dropout, and you want to try increasing the number of neurons in some layer, you will likely want to increase the dropout proportionately, so that the same number of neurons are not-dropped. If p=0.5 was optimal for 100 neurons, a good first guess for 200 neurons would be to try p=0.75.

Also, I get the feeling that dropout near the top of the network can be more damaging than dropout near the bottom of the network, because it prevents all downstream layers from accessing that information.

As to why 0.5 is generally used, I think its because tuning things like the dropout parameter is really something to be done when all of the big choices about architecture have been settled. It might turn out that 0.10 is better, but it takes a lot of work to discover that, and if you change the filter size of a conv-net or change the overall number of layers or do things like that, you're going to have to re-optimize that value.
So 0.5 is seen as a sort of placeholder value until we are at the stage where we are chasing down percentage points or fractions of a percentage point.

Original Paper and the opening paragraph is from the original paper...


What are the common properties of all machine learning algorithms? The definitions of machine learning I was able to find are very vague. What are the common properties of all machine learning algorithms? For example, keyword extraction algorithm (that use only statistical formulas) is ML? Language detection algorithm (that use dictionlaries for every language and statistic) is ML?

Machine Learning is enabling the machine to learn by its mistakes. As the machine is not human you need to find a way to tell it what it is doing wrong and a way to reduce its errors. Typically machine learning involves predictions of something (Numbers, Classes or whatever) All the fields you mentioned involve machine learning and of course statistics. Language is transformed into numeric to be processed by learning algorithm which is then able to predict a language from a text or keywords.
Hope been clear.


How to interpret Shapley value plot for a model? I was trying to use Shapley value approach for understanding the model predictions. I am trying this on a Xgboost model. My plot looks like as below

Can someone help me interpret this? Or confirm my understanding is correct?

My interpretation

1) High values of Feature 5 (indicated by rose/purple combination) - leads to prediction 1

2) Low values of Feature 5 (indicated by blue) - leads to prediction 0

3) Step 1 and 2 applies for Feature 1 as well

4) Low values of Feature 6 leads to prediction 1 and high values of Feature 6 leads to Prediction 0

5) Low values of Feature 8 leads to prediction 1 and high values of Feature 8 leads to prediction 1 as well. If it's too the extreme of x-axis (meaning from x(1,2) or x(2,3) - it means the impact of low values (in this case) of this feature, has a huge impact on the prediction 1. Am I right?

6) Why don't I see all my 45 features in the plot irrespective of the importance/influence. Shouldn't I be seeing no color when they have no importance. Why is that I only see around 12-14 features?

7) What role does Feature 43,Feature 55, Feature 14 play in prediction output?

8) Why is the SHAP value range from -2,2?

Can someone help me with this?

1. 2. not always there are some blue points also.

3. 4. 5. yes

6. it depends on the shap plot you are using, on some them default is to surpress less important features and not even plot them.

7. They are discriminatory but not as much, you can reconcile them with some other feature selection technique and decide if you want to keep them.

8. Range of the SHAP values are only bounded by the output magnitude range of the model you are explaining. The SHAP values will sum up to the current output, but when there are canceling effects between features some SHAP values may have a larger magnitude than the model output for a specific instance. If you are explaining a model that outputs a probability then the range of the values will be -1 to 1, because the range of the model output is 0 to 1. If you are explaining a model that outputs a real number or log odds the SHAP values could be larger since the model outputs can be larger.


Gap leaderboard score and model scoring on a Competition I'm working on a Veolia challenge on Ens Data Challenge ens-data (equivalent to Kaggle) the goal is to classify very rare binary events (the failure of a pipeline ) for 2014 and 2015 (y={2014,2015}). In input we have 5 features 3 categorical features (which I turned into dummy variable) and two continuous. The score is average AUC, $0.6*AUC_1 + 0.4*AUC_2$.

My problem is the following, when I compute each AUC (for 2014 and 2015) with a stratified kfold cross validation and I compute the average AUC I get roughly 0.88 and when I submit on the website I end up with 0.67, I guess there is a problem in my code.

Here is my code for choosing the best model for 2014:

Rk: to predict on the test set (2014,2015 unknown), I first predict with all 5 features, 2014. Then I add the prediction of 2014 to my feature to predict 2015

# Spot Check Algorithms
models = []
models.append(('LG', LogisticRegression()))
models.append(('LDA', LinearDiscriminantAnalysis()))
models.append(('RF', RandomForestClassifier()))
models.append(('GBC', GradientBoostingClassifier()))
models.append(('SVM', SVC()))
# evaluate each model in turn
results = []
names = []

# stratifiedkfold is defined by default when there is an integer
scoring = 'roc_auc'
num_folds = 10

for name, model in models:
    cv_results = cross_validation.cross_val_score(model, X, Y, cv=num_folds, scoring=scoring)
    results.append(cv_results)
    names.append(name)
    msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std())
    print(msg)

As you have commented, you are concerning about over-fitting.

In fact, cross validation will help to weaken over-fitting, but it can't eliminate over-fitting. Model scoring on TRAIN dataset sometimes exceeds scoring on TEST dataset. Here are some examples I can find:

Deep Residual Learning for Image Recognition (the famous ResNet paper). Checkout figure.1
This kernel of Two Sigma Financial Modeling Challenge on Kaggle says:

  we are getting a public score of 0.0169 which is slightly better than the previous one.
  Submitting this model to the LB gave me a score of 0.006

In my practice, I am more willing to concern the scoring gap between TRAIN, VALIDATION(DEVELOP) and TEST dataset. Personally speaking, TRAIN>VALIDATION=TEST is better than TRAIN=VALIDATION>TEST.

Edit: For class imbalance problem, there are some resources:

8 Tactics to Combat Imbalanced Classes in Your Machine Learning Dataset. This blog shows a common workflow dealing with imbalanced class issue.
Class Imbalance Problem in Data Mining: Review. This paper compares several algorithms created for solving the class imbalance problem.


How can I randomly sample the space of consistent neural networks for given data? Suppose I have a dataset $X$ and target labels $Y$. For a fixed neural network architecture, how can I randomly and uniformly sample from the space of all possible assignments of weights such that the neural network maps $X$ to $Y$?

I am trying to formalize your question before discussing it.
If I understand correctly, you ask for the following:
For $X \subset \mathbb{R}^{n}$ and $Y \subset \mathbb{R}^m$, let $f:X \rightarrow Y$ be a map.
Let $w \in \mathbb{R}^q$ be weights. We consider a neural network $g: \mathbb{R}^{n}\times \mathbb{R}^q \rightarrow \mathbb{R}^m$, and let $g^{(w)}: \mathbb{R}^{n} \rightarrow \mathbb{R}^{m}, x \mapsto g(x,w)$ be the neural network parametrized by $w$.
Now you want to sample from the set $\underline{W}(f,g,X):= \{w \in \mathbb{R}^{q} \mid f = (g^{(w)})_{\mid X} \}$.
However, I think constructing $\underline{W}(f,g,X)$ is very difficult in general.
The following question arises:
Do you already have some $w \in \underline{W}(f,g,X)$ ?
If not, note that $\underline{W}(f,g,X) = \emptyset$ is possible! (its easy to construct an example for that)
Note also that all known universal approximation theorems have some requirements on $f$, and only state that $f$ can be approximated by some neural network. However, for fixed architecture, it might be that there is no $w \in \mathbb{R}^q$ with $f = (g^{(w)})_{\mid X}$ nor that $f$ can be approximated by $(g^{(w)})_{\mid X}$ (e.g. in terms of the uniform-norm).
If you have some $w \in \underline{W}(f,g,X)$, there are certains trivial permutations (e.g. permuting the nodes of a fully-connected layer, or some channels). Apart from that, I am not aware of a full description of $\underline{W}(f,g,X)$. And without further details or constraints, I think its there is no general answer at the moment.
I hope this helps!


What it Would be easier Building a Deep Net From Scratch or Using an existing Architecture? In Practice with CNN what would be easier: Building a CNN from scratch or using a an existing architecture with some updates?

It depends on your task and the amount of data you have. If you have so much data but you can not find similar tasks to have appropriate architecture you should stack convolution and dense layers yourself. But if you have appropriate amount of data and there exist good architectures then you have to decide what you want and how is your situation. Suppose that you want to have recognition task, there are so many architectures that are applied to ImageNet data-set. You can use transfer learning but there is a point here. Suppose that you want to fine tune GoogleNet. This is a very large network and is capable for recognizing about a thousand distinct classes. If you have a recognition task with 5 classes and you have an agent that should be online, this is not logical to have such a big network. You may have similar performance by stacking a few layers and get better time complexity. If you don't have so much data, freezing the layers and applying transfer learning to the last layer maybe a typical solution.


How to remove rows from a data frame that are identical to other df? I have two data frames.
So how do I remove rows that have identical email addresses in df2 from df1?

>>df1
   First  Last  Email
0 Adam   Smith  email@email.com
1 John   Brown  email2@email.com
2 Joe    Max    email3@email.com
3 Will   Bill   email4@email.com

>>df2
  First  Last   Email
0 Adam   Smith  email@email.com
1 John   Brown  email2@email.com

You can try this:

cond = df1['Email'].isin(df2['Email'])
df1.drop(df1[cond].index, inplace = True)

>>df1
    First   Last    Email
2   Joe     Max     email3@email.com
3   Will    Bill    email4@email.com


Training on skewed dataset I have a problem of multi class classification and I'm using a simple 2-Layer Bi-directional LSTM with keras.

The model in a simple form:

Bidirectional LSTM (64)
Bidirectional LSTM (64)
Dense (128)
Activation Sigmoid
Dense (14)
Activation Softmax

I have a raw and skewed dataset so I'm doing all the pre-processing myself to balance it.

Firstly, I replicated the original data and produced 3*OriginalData
to produce more examples of certain classes that weren't represented.
Then I performed classification and the loss and accuracy history are
depicted below for both training and validation sets

Accuracy built up after 25 epochs, but loss had major variations, so I decided to replicate more of my data so I produced 10*OriginalData, and performed classification again.

Now my loss and accuracy behaved so much worse.

My question is whether the problem lies in my model, or at the replication of data. Is it possible for a model to work well with less data but not so good when it has so many? Or maybe I'm causing it to overfit by providing over-replicated data?

Note: By replicating data, I don't mean duplicating, but because I work with audio data, I play with different pitch shifts.

It could be your model or your data. You need to perform the experiment of changing your model (while holding data constant) to isolate the causal reason.

Yes - models can get worse with more data. One of the primary reasons is models often have a limited capacity to learn. Simple models can only successfully model simple relationships. Again - if you fit a more complex model, you'll start to understand the mechanism that could be causing this observation.


Building a scorecard I would like to build a scorecard from scratch.

Currently I have been working on data cleaning, and I created a dataset for training and another for test.
What I have is a dataset of around 700 rows (maybe not so many as I could expect; it is for a computer science project), 20 columns. The target is if the borrower has defaulted or not (indicator). Most of the factors are categorical or Boolean.
I am assigning a score based on these parameters, specifically:

- if the account has defaulted in the last 90 dats, then assign 60;
- if the account has defaulted in the last 30 days, then assign 50;
- if the account has not defaulted in the last 90 days, then assign 80;
...

I would like to know if this is a good approach and/or if there is an algorithm to assign scores to the accounts or if I should do something different.
I do not know very well the logic behind a scorecard, so any information would be extremely useful.

Thanks

What you describe it is NOT a typical approach for scorecard design.

Typically binary target - "0/1" is used for target variable and logistic regression is typical method to create a scorecard.

Typically to define "default trigger" one fixes two time periods:

1) fix say 1 year from credit start, that means defaults happening after 1 year are not considered as defaults

2) fix say 90 day period - means "default trigger" is 1 if client delays payment for more than 90 days from the payment date, that means you do not consider as "default" if delay is less than 90 days.

There can be some other conditions - like threshold on amount of delinquency - if below - do not consider as "default". And may be some other.

You might consider the book:

Credit Risk Scorecards: Developing and Implementing Intelligent Credit Scoring

by Naeem Siddiqi


Which is the "BEST" deep learning model for "Custom" object detection for images & real time. YOLO v3, v4, v5, EfficientDet? Whenever I look for object detection model, I find YOLO v3 most of the times and that might be due to the fact that it is the last version created by original authors and also more stable. In 2020, a new author released unofficial version called YOLO v4 and just after 5 days, another author launched YOLO v5. I am confused that if I have to chose one of the models, which one should I chose? EfficientDet is better than YOLO v3 in my opinion but there is very less talk about that. Is there any other model which is better than thsese? I want to know which models should I chose if I'm detecting objects in images and videos respectively. When speed is not an issue, which one is the best model and when speed is an issue  like in videos, which one should I use?

There are many subtle differences between yolov4 and v5 other than speed like YOLOv4 exceeds YOLOv5's performance on the COCO benchmark.this link will help you explain.
YOLOv4 VS YOLOv5.
I dont know much about EfficientDet.
Here's YOLOv3 Versus EfficientDet for State-of-the-Art Object Detection
.
It explains quite nicely difference between the two.


Classification: how to handle reviews/long english words in feature set with all other numerical features I am currently working on an use case where feature set contains numeric values such as amount, as well as a review feature which contains long winded english text.
the english text will very well differ between train and test data.
eg 'i have seen and its good' , 'nto ok','timepass',etc

how do i combine the text feature set with numerical data and feed it to a machine learning model?

i will nt be able to use encoding , these text variables are not categorical values . they are varying .

import pandas as panda
from sklearn.feature_extraction.text import TfidfVectorizer

words = ['i hv paid','i dont like','its good','yum yum']

a = panda.DataFrame({'amount':[10,20,30,40],'word':words})

tf = TfidfVectorizer()

csr = tf.fit_transform(words)

#how do i now use my csr to feed both amount and word to my machine learning model

One of the ways to address your use case could be to create 2 separate models, one model using your text data features and another one using your numerical features and combine their results using ensembling.

The other way could be to create numeric features out of your text features (e.g. tf-idf, word2vec) and combine them with your numeric features and feed them to your model.

Sample code-

from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np
import pandas as pd

words = ['i hv paid','i dont like','its good','yum yum']
tf = TfidfVectorizer()
vector = tf.fit_transform(words)
df = pd.DataFrame([10,20,30,40])
df2 = pd.DataFrame(vector.toarray())
pd.concat([df,df2], axis=1)

Sample output-


GAN - why doesn't the generator nullify the noise input? In GAN architecture, during training, what keeps the generator's output dependant on the input noise? Why don't the weights of the noise input become zero (plus a bias)?
I would expect the generator to converge to outputting a single picture which is extremely real and non-distinguishable from a real picture, and ignore the noise all together, since this is a "cheaper" way (in convergence time and in number of parameters used) to decrease the generator's loss.

If the generator always outputs the same image, then it's easy for the discriminator to win the game and tell apart the output of the generator from random images in the training set: if the input to the discriminator is that one image, then it outputs "came from the generator", otherwise outputs "came from the training step".  The game is set up so that the generator is rewarded for fooling the discriminator.  Always outputting the same image isn't going to fool the discriminator.


Deep learning error: All layer names should be unique I am working in deep learning project using vgg16.
I got the following error and I could not solve it.

RuntimeError: ('The name "block1_conv1" is used 2 times in the model. All layer names should be unique.

The code:

vgg16_model = keras.applications.vgg16.VGG16()

input_layer1 = vgg16_model.input
vgg16_model.get_layer(index = 0).name = 'input1'
last_layer1 = vgg16_model.get_layer('flatten').output

# Adding layers
x = Dense(2048, activation='relu', name='fc1')(last_layer1)
x = BatchNormalization(name='bn1')(x)
fc12 = Dense(2048, activation='relu', name='fc2')(x)
bn12 = BatchNormalization(name='bn2')(fc12)
out1 = Dropout(0.4, name='dropout1')(bn12)

#Classification layer
output_layer1 = Dense(no_classes, activation='softmax', name='prediction1')(out1) #106 classes
model1 = Model(input_layer1, output_layer1)

#Train the last block
model1.trainable = True
set_trainable = False
for layer in model1.layers:
    if layer.name == 'block5_conv1':
        set_trainable = True
    if set_trainable:
        layer.trainable = True
    else:
        layer.trainable = False

vgg16_model2 = keras.applications.vgg16.VGG16()

input_layer2 = vgg16_model2.input
vgg16_model2.get_layer(index = 0).name = 'input2'

# Remove layers from block4_pool layer
last_layer2 = vgg16_model2.get_layer('block4_pool').output

# Adding layers
x = Conv2D(512, (3, 3), activation='relu', name='block5_conv1', padding='same')(last_layer2)
x = BatchNormalization(name='bn1')(x)
x = Conv2D(512, (3, 3), activation='relu', name='block5_conv2', padding='same')(x)
x = BatchNormalization(name='bn2')(x)
x = Conv2D(512, (3, 3), activation='relu', name='block5_conv3', padding='same')(x)
x = BatchNormalization(name='bn3')(x)
x = MaxPooling2D(pool_size=(2, 2), strides=(2, 2), name = 'block5_pool')(x)
x = Flatten(name = 'flatten')(x)
x = Dense(2048, activation='relu', name='fc1')(x)
x = BatchNormalization(name='bn4')(x)
fc22 = Dense(2048, activation='relu', name='fc2' )(x)
bn25 = BatchNormalization(name='bn5')(fc22)
out2 = Dropout(0.3, name='dropout1')(bn25)

#Classification layer
output_layer2 = Dense(106, activation='softmax', name='prediction2')(out2)
model2 = Model(input_layer2, output_layer2)

#Train the last block
model2.trainable = True
set_trainable = False
for layer in model2.layers:
    if layer.name == 'block5_conv1':
        set_trainable = True
    if set_trainable:
        layer.trainable = True
    else:
        layer.trainable = False

con = concatenate([out1, out2])  # merge the outputs of the two models
output_layer3 = Dense(no_classes, activation='softmax', name='prediction3')(con)
multimodal_model1 = Model(inputs=[input_layer1, input_layer2], outputs=[output_layer3])

I tested model1 and model2, and I got good results. But when I tried to merge the two models the error appear. The problem initiated here:

multimodal_model1 = Model(inputs=[input_layer1, input_layer2], outputs=[output_layer3])

Please help to solve this problem.

The problem is, you are trying to instantiate two VGG16 models at the same time and its confusing for the kernel to figure out which graph it needs to use. Atleast that is the speculation I have, because of the problems faced when trying to load multiple models in keras with tensorflow backend, its not straight forward.

Try commenting the code from when you are instantiating the second model and this error should go away.


Detect sensitive data from unstructured text documents I know this question is broad, but I need an advice to know if it's possible to achieve what I want to do.

The problem is that I have around 2500 documents with sensitive data being replaced by four dots. I do not have the original documents, so I wonder if there is a way to build a model that can detect sensitive data from any new documents (with sensitive data not being removed) using the previous documents? I want to apply machine learning or deep learning approaches. And what I know is that the original data set with annotated sensitive data should be used for training, which I can't obtain.

I am new at this field so any advice would be very appropriated

Welcome to the site! Assuming that I understand your problem correctly, I think you can achieve a working model.

If I was in your position I would:

Obtain the cleanest data possible from the documents. For example, you don't state if the docs are already in simple text or if you need to do something like OCR or whatnot. Having the cleanest set possible will be key for this.
Make sure you have a consistent marker for the sensitive data. You mention four dots - is that the case for ALL instances? If not, clean that data now
You're going to need to do standard NLP cleansing stuff like removing punctuation but you may or may not want to keep stop words (this will be part of your model testing). Also, this is key, be 100% certain that the four dots are viewed as a single work in your tokenization process -  you should be able to verify this prior to committing to your tokenization file.
I would take all my documents and create 3 word ngrams. I would then separate out ngrams that contain sensitive data and not sensitive data. That, essentially, becomes your labeled dataset and you should label them accordingly.
My base model would use all entries that contain sensitive data in the second position of the ngram (the middle of the three words). I would train a neural network on that and see what kind of results I achieve with that. NOTE that your four dots will not be an input, only the word previous and after will be your inputs. You could almost treat this as a binary classification model - the middle word is either sensitive or it's not.
Future iterations of my model would maybe use a multi-classification approach with something like (1) No sensitive data (2) sensitive data in first position (3) sensitive data in second position and (3) sensitive data in third position and so on and so on.
From there, you can play with variations on the size of the ngram since the immediate words may or may not actually have an effect on the predictions. There's no limit to how crazy you can get with this - you won't know until you start modeling.

Finally, your entire project becomes even more interesting when you go to the prediction phase with new data. You will do the same and break down your document into ngrams and create a prediction for each one and output the result. In other words, you will need to break down your document only to turn around and build it up again - that should be a fun script to write! Good luck with this, let us know how it turns out.


Do we need to do multiple times deep learning and average ROC? Do we need to do multiple times deep learning and average ROC(AUC)?
Since we've might get different ROC(AUC) every round we train and test (by KERAS)
Is it necessary to average ROC(AUC) with multiple times training and testing?
(or just choose the best round?)

No, you don't need to do that. Do probability calibration (temperature scaling) on your trained network. Here a link to do that.


Why are Convolutional Networks not using cross-correlation To my knowledge cross-correlation is used to measure the similarity of certain values, like to images. Same applies to the process of feature extraction in CNNs, where input matrices are multiplied by filters. So it seems odd to me that they are call Convolutional Networks.

The Pytorch documentation for Conv2d even says that it is using the cross-correlation operator.

So why are CNNs called convolutional when they are actually using cross-correlation?

That's a very interesting question.
I started looking into it recently while trying to understand better convolutional neural networks.
In short: yes! They are called convolutional while in actual practical terms using the cross-correlation operator. So this is a case of a misnomer.
I think the important thing to understand is that correlation and convolution differ only because of a flip that is present in the convolution.
So they differ only because of a sign.
Correlation:

Convolution:

The main difference in practical terms is that the convolution is associative.
This means that

This is not true for the correlation operator.
When training a neural network though, the difference in the sign in front of the i, does not matter because you will adjust your weights in order to optimise your objective function.
This is also explained in another post:
Convolution and Cross Correlation in CNN
In this post, it is also explained that what is actually used for CNN is the cross-correlation operator and not the convolution one.
I hope this helps.
PS
Also, see the notes on convolution from the David Jacobs CS course.


XGBoost Huge Dataset ~1TB Can a gradient boosting solution like XGBoost or Lightbgm be used for a huge amount of data ? I have a csv file of 820GB containing 1 Billion observations and each observation has 650 datapoints.

Is this too much data for XGBoost ? I have searched all over the internet for a solution to when the data won't fit into RAM memory to no avail. I read about external memory for xgb but there is no detailed doc. Can someone point me in the right direction please and thank you !

1) Split data in smaller blocks

2) learn clf on first block

3) Dump model into pickle

4) load second block

5) load model from pickle

6)
    learn clf on second block and so on


how to find the best parameters to solve a differential equation? I have a differential equation:
def func(Y, t, r, p, K, alpha):
    return r * (Y ** p) * (1 - (Y / K) ** alpha)

and I want to find the best parameters that fit (r,p,K,alpha). I tried to use curve fit but it was too bad, this is my code
# I chose the value of maxfev randomly
popt, pcov = curve_fit(func, df.index, df.Value,method='lm',maxfev = 8000)

t = np.linspace(0, len(df), len(df))
y0 = popt[0]
params = (popt[1], popt[2], popt[3], popt[4]) # r, p, K, alpha
sol = odeint(func, y0, t, args=params)

and this the plot:

Note: this is how my real data looks like:

what I'm looking for it the best values for (r,p,K and alpha), how to find them?

If nothing else is possible, you could try using a genetic algorithm.


Text Classification misclassifying? I am trying to solve a binary classification problem. My labels are abusive (1) and non-abusive (0).
My dataset was imbalanced (more 1 than 0s) and I used oversampling of the minority label (i.e. 1) to balance my dataset. I have also done pre-processing, feature engineering using TF-IDF and then fed the dataset into a pipeline using 3 classification algorithms namely: Logistic Regression, SVM, and Decision Tree.
My evaluation metrics are:
    Logistic Regression:
[[376  33]
 [ 18  69]]
          precision    recall  f1-score   support

       0       0.95      0.92      0.94       409
       1       0.68      0.79      0.73        87

accuracy                               0.90       496
macro avg          0.82      0.86      0.83       496
weighted avg       0.91      0.90      0.90       496

    SVM:
[[383  26]
 [ 23  64]]
          precision    recall  f1-score   support

       0       0.94      0.94      0.94       409
       1       0.71      0.74      0.72        87

accuracy                               0.90       496
macro avg          0.83      0.84      0.83       496
weighted avg       0.90      0.90      0.90       496

    Decision Tree:
[[383  26]
 [ 28  59]]
          precision    recall  f1-score   support

       0       0.93      0.94      0.93       409
       1       0.69      0.68      0.69        87

accuracy                               0.89       496
macro avg          0.81      0.81      0.81       496
weighted avg       0.89      0.89      0.89       496

The issue I'm facing is that certain new abusive text is being predited as non-abusive. I think that I think my false positive (FP) and false-negative rate (FN) are too high and need to be reduced.
Do you have any suggestions on how to reduce FP and FN or any other suggestions to cater for my issue?
Thanks.

All 3 algorithms are giving very similar results. And looking at evaluation sample size I think training sample is not too large. It indicates me that if there is any opportunity, it's
a) in feature engineering
b) in not predicting for less confident cases
c) getting more data to train more complex algo
a) Feature Engineering - TFIDF or count vectorizer has a real world problem of having test words outside of training vocabulary. If you can use general language vocabulary for embedding both train and test sets, then results should improve. There are open source pretrained embeddings like USE, Glove etc. to do so.
b) Prediction Confidence - Along with class prediction you can also get probability of classification. Then check - below which cutoff of probability your F1 score gets too low. Don't predict for those low probability cases. Most practical systems accept limitation of AI.
c) More data will allow you to learn through more complex algorithms like boosting which may improve results. Hopefully you are already crossvalidating.
Also, based on the cost of error, you can decide whether FP or FN should be prioritized. Accordingly optimize precision/ recall.


How to handle a zero factor in Naive Bayes Classifier calculation? If I have a training data set and I train a Naive Bayes Classifier on it and I have an attribute value which has probability zero. How do I handle this if I later want to predict the classification on new data? The problem is, if there is a zero in the calculation the whole product becomes zero, no matter how many other values I got which maybe would find another solution.

Example:

$P(x|spam=yes) = P(TimeZone = US | spam=yes) \cdot P(GeoLocation  = EU | spam = yes)  \cdot  ~ ... ~  = 0.004 $

$P(x|spam=no) = P(TimeZone = US | spam=no) \cdot P(GeoLocation  = EU | spam = no)  \cdot  ~ ... ~  = 0 $

The whole product becomes $0$ because in the training data the attribute TimeZone US is always Yes in our small training data set. How can I handle this? Should I use a bigger set of training data or is there another possibility to overcome this problem?

An approach to overcome this 'zero frequency problem' in a Bayesian setting is to add one to the count for every attribute value-class combination when an attribute value doesn’t occur with every class value. So, for example, say your training data looked like this:

$$\begin{array}{c|c|c|}
 & \text{Spam} = yes & \text{Spam} = no \\ \hline
\text{TimeZone} = US & 10 & 5 \\ \hline
\text{TimeZone} = EU & 0 & 0 \\ \hline
\end{array}$$

$ P(\text{TimeZone} = US | \text{Spam} = yes) = \frac{10}{10} = 1$

$P(\text{TimeZone} = EU | \text{Spam} = yes) = \frac{0}{10} = 0$

Then you should add one to every value in this table when you're using it to calculate probabilities:

$$\begin{array}{c|c|c|}
 & \text{Spam} = yes & \text{Spam} = no \\ \hline
\text{TimeZone} = US & 11 & 6 \\ \hline
\text{TimeZone} = EU & 1 & 1 \\ \hline
\end{array}$$

$ P(\text{TimeZone} = US | \text{Spam} = yes) = \frac{11}{12}$

$P(\text{TimeZone} = EU | \text{Spam} = yes) = \frac{1}{12}$


Multiclass classification in a balanced dataset with one high-priority label I have a balanced dataset for a multiclass classification problem with one high-priority label (this ought to be classified properly at all costs). How do I go about creating a workflow for this problem? What specific feature engineering/selection methods and classifiers should I be considering for this problem?

To be more specific, the data I'm dealing with (including labels) is completely anonymized, so I don't have a clue as to what it actually stands for.

Some approaches I am considering -

Creating synthetic data points for the priority label through oversampling.
Creating a highly non-linear model for prediction as accuracy is very important.

Any help is much appreciated!

This can be tackled in two places:

Data: as you mentioned, this is done by artificially increasing the number of samples from critical class $cc$. This produces the same effect as data-sets that are naturally imbalanced,
Model: this is generally done by over-penalizing the miss-classification of $cc$ compared to other classes. One place for this modification is the loss function. A frequently used loss function in classification is cross entropy. It can be modified for this purpose as follows. Let $y_{ik} = 1$ if $k$ is the true class of data point $i$, otherwise $y_{ik} = 0$, and $y'_{ik} \in (0, 1]$ be the corresponding model estimation. The original cross-entropy can be written as:
$$H_y(y')=-\sum_{i}\sum_{k=1}^{K}y_{ik}log(y'_{ik})$$
which can be changed to
$$H_y(y')=-\sum_{i}\sum_{k=1}^{K}\color{blue}{w_{k}}y_{ik}log(y'_{ik})$$
For example, by setting $w_{cc} = 10$ and $w_{k \neq cc}=1$, you are essentially telling the model that miss-classifying $1$ member from $cc$ is as punishable as miss-classifying $10$ members from other classes. This is roughly equivalent to increasing the ratio of class $cc$ $10$ times in the training set using method (1).