clustering multivariate time-series datasets I am new to clustering.i have data from quality testing of an automobile manufacturing company.

I have 100000 datasets.each dataset has 4 variables force, voltage, current, distance. each variable is a continuous time-series with 8000 data points each(1 to 17000 milliseconds). the length of time series differs from on dataset to another. all variables in one dataset has to be compared with another dataset

I have to find clusters in the 100000 datasets based on similarities in shape of each variable in a dataset.

which type of measuring best suits, in this case, to find similarity in shape of time-series

For most clustering approaches, first you need to choose a similarity measure. Some common default ones for raw time series are Euclidean distance and Dynamic Time Warping (DTW).

When you have computed the similarity measure for every pair of time series, then you can apply hierarchical clustering, k-medoids or any other clustering algorithm that is appropriate for time series (not k-means!, see this).

Update: if the number of time series (along with their size) makes it computationally not acceptable to compute pairwise distances, then one option can be to extract features from each time series, and then use such features as proxies for the time series in the clustering process. Some examples of such features are maximum value, number of peaks, mean value. There are libraries like tsfresh in Python that are meant to easily extract such kind of features from time series. With these features, then any clustering approach like k-means can be applied.


Using strong predictor in Model training? I am trying to build a Disease predictor based on symptoms. I am using data scraped from Symcat website. After sampling the data we have symptoms to disease mapped for training purpose.Data looks like these: X --> S1,S2,S4... Y --> D1,D2,D5. The problem is some symptoms are strong predictors for a particular disease. Say S1 is a strong predictor for D1. So basically model should predict higher probability score for D1 compared to others due to presence of S1. How to imbibe the information that S1 is strong predictor for D1 so that it can be used in model training

Tried oversampling those strong predictor to disease combination. But dont think that is right as I have a multi label data set. results are not reflecting either
Dataset looks like this:
![Symptoms     D1  D2  D3
S1,S2,S4    0   1   1
S3,S4,S5    1   0   0
S1,S4,S6    1   0   1]1
Symptoms are features. We are using symptoms definition embeddings. Y is 1 hot vector.

I can think of two way inducing priors into supervised machine learning training process.
1) Using a Bayesian framework(PYMC for example) where you can define you random variables with whatever prior you like.
2) Engineering features that describe the needed relation. For example, in your instances I would create a new Boolean feature named 'is_strong_predictor_by_prior', and would assign true if x=s1 and y=d1 and false else wise. This new feature can hold multiple priors(not only the s1>>d1 relation).


Why naive bayes is "naive" Some articles say that naive Bayes is naive because of "independence of attributes". Whereas others say "independence of attributes within a class". Can anybody please clear this confusion?
Thanks

Naive Bayes doesn't assume independence of attributes ... It assumes conditional independence (or what you call independence within a class). This allows us to write the likelihood in the bayes rule P(X | Y) as the product of all P(Xi | Y), where X = (X1, ... Xi, ... Xn) and n is the number of attributes.


Using Clustering in text processing Hi this is my first question in the Data Science stack. I want to create an algorithm for text classification. Suppose i have a large set of text and articles. Lets say around 5000 plain texts. I first use a simple function to determine the frequency of all the four and above character words. I then use this as the feature of each training sample. Now i want my algorithm to be able to cluster the training sets to according to their features, which here is the frequency of each word in the article. (Note that in this example, each article would have its own unique feature since each article has a different feature, for example an article has 10 "water and 23 "pure" and another has 8 "politics" and 14 "leverage"). Can you suggest the best possible clustering algorithm for this example?

If you want to proceed on your existing path I suggest normalizing each term's frequency by its popularity in the entire corpus, so rare and hence predictive words are promoted. Then use random projections to reduce the dimensionality of these very long vectors down to size so your clustering algorithm will work better (you don't want to cluster in high dimensional spaces).

But there are other ways of topic modeling. Read this tutorial to learn more.


Understanding Weighted learning in Ensemble Classifiers I'm currently studying Boosting techniques in Machine Learning and I happened to understand that in Algorithms like Adaboost, each of the training samples is given a weight depending on whether it was misclassified or not by the previous model in sequential boosting.
Although I intuitively understand that by weighting examples, we are letting the model pay more attention to examples that were previously misclassified, I do not understand "how" the weights are taken into account by a machine learning algorithm. Are the weights taken as a separate feature? I want to know "how" a simple machine learning model, lets say a decision tree, understands that a particular example carries more weight and therefore needs to be given more attention/ classified correctly.
Can someone please explain to me in simple words/equations how weighted training samples are handled by machine learning algorithms? How do they force themselves to classify samples with higher weights correctly? Should there be modifications made to existing ML algos to account for weights?

Adaboost is an Ensemble Classifier, which means that it consists of several weak classifiers that are able to consider weights in the training set. These weights can be used to inform the training of each weak classifier. For example, decision trees can be grown in a way that favor splitting sets of samples with higher weights.


Is boosting resistant to overfitting for both number of iterations and number of features? Boosting methods (such as the popular xgboost) do not tend to overfit when we use many iterations - Schapire and Freund. Are they also resistant to overfitting when we feed them with a large number of features (where some of the features are not very useful?) If so, is there a theoretic connection between these two desirable properties?

(related to
https://stats.stackexchange.com/questions/35276/svm-overfitting-curse-of-dimensionality)

I can't speak from a theoretical perspective, but I can say that in practice overfitting with a large number of features is not an issue with gradient boosting. I work at a huge financial institution that has been doing credit risk modeling for decades, and for the past decade or so they have almost exclusively been doing their variable selection using gradient boosting. Gradient boosting (along with any tree-based method) can be used to find relative feature importances (based on how much error is reduced after each split).

Statisticians where I work will often use gradient boosting to narrow down 2,000 or so features into a more manageable 10-15 by taking the top ranked features. These features are then fed into a logistic regression model, which is more interpretable than gradient boosting. Even on the hold-out partition, the gradient boosting algorithm built on the 2,000 features generally outperforms the logistic regression model. The gradient boosting model is thus used for variable selection and predictive performance benchmarking.


Attention to multiple areas of same sentence Lets consider some sentences below:
"Datascience exchange is a wonderful platform to get answers to datascience related queries and it helps to learn various concepts too"
"Can company1 buy company2? What will be their total turnover then?"
"Coronavirus was originated in china. After that it is spreading all over the world. To prevent it everyone has to take care of cleanliness and prefer vegetarians."

In all above sentences you can see there are multiple questions or utternaces. Sometimes separated by and sometimes by question mark and sometimes by just a dot.
A rule based separation of these sentences fail in many cases.  I want to split these sentences in individual intents.

One of the approach I am guessing is by using attention mechanism on different parts of sentences. I cant use gensim etc sentence embeddings as I dont have clear sentence boundaries here.
Can someone suggests if attention approach will work? If yes, any similar code if they can point to, that would be helpful as I haven't coded this before.

If any other better approach can solve this problem then please suggest.

For the sentences you provided the nltk sentence tokeniser would work just fine.

from nltk.tokenize import sent_tokenize

sentences = ['Datascience exchange is a wonderful platform to get answers to datascience related queries and it helps to learn various concepts too','Can company1 buy company2? What will be their total turnover then?','Coronavirus was originated in china. After that it is spreading all over the world. To prevent it everyone has to take care of cleanliness and prefer vegetarians.']

for sent in sentences:
    print(sent_tokenize(sent))

out:
['Datascience exchange is a wonderful platform to get answers to datascience related queries and it helps to learn various concepts too']
['Can company1 buy company2?', 'What will be their total turnover then?']
['Coronavirus was originated in china.', 'After that it is spreading all over the world.', 'To prevent it everyone has to take care of cleanliness and prefer vegetarians.']

It is true that these rule based tokeniser do suffer from several issues (see: https://github.com/nltk/nltk/issues/494 ). If you are specifically working on a sequential model for sentence slitting then an LSTM with attention mechanism would surely be a good choice, you can find many repositories on GitHub which implement the code of the original paper 'attention is all you need' (among others: https://github.com/kaushalshetty/Positional-Encoding )


Converting pandas series object to int in pandas I started my machine learning journey by deciding to explore recommender systems so that I can apply it in some of the projects for my company. Now, I am using Pandas for data analysis. The data set is the imdv movies data set. Here, I am trying to convert a pandas series object to int but it converts the series to float64.

Here is the screenshot:

'clean_ids' is the method that I am using to do this and you can see that 'id' changes to float64. Please help me with this.

Your code is correct. The problem is that you have NaN values in your id column and python interprets NaN as float. In your code any decimal values it encounters will be converted to integer but represented as $x.0$ due to presence on NaN values. See here for more.

As for a solution to your problem you can either drop the NaN values or use IntegerArray from pandas. It is currently experimental but suits yor problem.


How to use Automated Labelling for documents? Let's say I have been given 1000 documents and 6 labels from someone. My job is to label each of these 1000 documents into 1 of the 6 labels which are words not numbers. How can I automate or semi-automate this process using data science??

Semi-supervised learning. You label 1% manually, let the algorithm learn, then it labels unknown data, learns from it and labels again.


Anconda R version - How to upgrade to 4.0 and later I use R through the anaconda navigator, which manages all my package installations. I need to use qgraph for a project, which is dependent on mnormt library, which in turn needs RStudio verion >4.0
I think the solution to my problem would be to upgrade Anaconda R to the latest R version, but the r-base search in conda has no R version greater than 3.6. Any help in upgrading Anaconda R to newer versions would be very useful.
The problem is in installing qgraph, which stops with errors in installing dependencies (pysch)
During startup - Warning messages:
1: Setting LC_CTYPE failed, using "C"
2: Setting LC_TIME failed, using "C"
3: Setting LC_MESSAGES failed, using "C"
4: Setting LC_MONETARY failed, using "C"
Error: .onLoad failed in loadNamespace() for 'mnormt', details:
  call: library.dynam("mnormt", pkg, library)
  error: shared object 'mnormt.dylib' not found
Execution halted
ERROR: lazy loading failed for package 'psych'

You need to create a new environment and then you can install R 4.+ in Anaconda. Follow these steps.
conda create --name r4-base

After activating r4-base run these commands
conda activate r4-base
conda install -c conda-forge r-base
conda install -c conda-forge/label/gcc7 r-base

Finally, you will notice r-basa version 4 will be installed.

Thereafter, you can install any supported packages. But with this only, you won't have the ability to use it in the Jupyter notebook. You need to install install.packages('IRkernel') and Jupyter notebook as well if you want to use it. Otherwise you are good to go with R-Studio.
For Jupyter Installation and RKernel.
conda install jupyter

Then open the R console. Write in R console
install.packages('IRkernel')
IRkernel::installspec()

Congrats! You can use Notebook for Python and R.


Loss function for ReLu, ELU, SELU Question
What is the loss function for each different activation function?
Background
The choice of the loss function of a neural network depends on the activation function. For sigmoid activation, cross entropy log loss results in simple gradient form for weight update z(z - label) * x where z is the output of the neuron.
This simplicity with the log loss is possible because the derivative of sigmoid make it possible, in my understanding. The activation function other than sigmoid which does not have this nature of sigmoid would not be a good combination with the log loss. Then what are the loss function for ReLu, ELU, SELU?

References

Derivative of Binary Cross Entropy - why are my signs not right?

The question requires some preliminary clarification IMHO. The choice of activation and loss function both depend on your task, on the kind of problem you want to solve. Here are some examples:

If you are training a binary classifier you can solve the problem with sigmoid activation + binary crossentropy loss.
If you are training a multi-class classifier with multiple classes, then you need softmax activation + crossentropy loss.
If you are training a regressor you need a proper activation function with MSE or MAE loss, usually. With "proper" I mean linear, in case your output is unbounded, or ReLU in case your output takes only positive values. These are countless examples.

With activation function here I refer to the activation of the output layer. The activation that performs the final prediction scores doesn't have to be (and usually isn't) the same as the one used in hidden layers.
ELU and SELU are typically used for the hidden layers of a Neural Network, I personally never heard of an application of ELU or SELU for final outputs.
Both choices of final activation and loss function depend on the task, this is the only criterion to follow to implement a good Neural Network.


How to decide who to market? Clustering or Decision Tree? I am working with a dataset that has enough observations and ~ 10 variables,

half of the variables are numeric
another half of the variables are categorical with 2-3 levels (demographics)
one ID variable
one last variable that has sales value, 0 for no sale and bill amount for sale

Using this information, I want to understand which segments of my customers to market. I am using R for code but that's not relevant here. :)

I am confused about which statistical technique to use. Since I want to identify what types of customers I want to acquire and build my campaigns, I was initially thinking of using k-means clustering, i.e. go with unsupervised learning.

However, considering I know who purchased and how much they purchased from the sales value figures, I believe it's worth including this information and have decided to go with predictive modeling instead. Here regression will only tell the importance of variables, but I am interested in nodes (for example, I want rules that can support my marketing campaign such as age 45+, from LA region...etc.) so decision tree would be a better fit.

What are your thoughts? Clustering or decision tree? Or actually something else?

Since you have label data (i.e., sale amount), you can apply supervised machine learning.
After a model has been fit, the features that contributed to predicting targets can be found.
Decisions trees would be a relatively simple option because it easily handle the different types of features and can a yield a decision path.


Data Cleaning, refining etc I have data as below:

Carpanter
Carepnter
Carpentor
Labourer
Labor
Labour
Housewife
House Wife
housewife.

I want to clean data and rectify the spelling mistakes but not manually because its a huge data. Due to spelling mistakes these 50/60 occupations have become around 2000.

What you are trying to do is text normalization, which is a part of NLP pipeline.

To successfully preprocess your data, please try this below:

Install ekphrasis

pip install ekphrasis

Apply seg & spell correction:

from ekphrasis.classes.spellcorrect import SpellCorrector
from ekphrasis.classes.segmenter import Segmenter
sp = SpellCorrector(corpus="English")
seg_eng = Segmenter(corpus="English")

words_to_correct = ['Carpanter','Carepnter','Carpentor','Labourer','Labor','Labour','Housewife','House Wife','housewife.']
for word in words_to_correct:
    segmented = seg_eng.segment(word)
    corrected = sp.correct(segmented)
    print(word + " -> " + corrected)

Outputs as:

Carpanter -> carpenter
Carepnter -> carpenter
Carpentor -> carpenter
Labourer -> labourer
Labor -> labor
Labour -> labour
Housewife -> housewife
House Wife -> housewife
housewife. -> housewife

Note: Check this amazing work and paper if you are interested in!


SKNN regression problem I am trying to learn scikit-learn neuralnetwork and am coming up against the same problem in regression where no matter the dataset I getting a horizontal straight line for my fit.

here is an example using the Linear regression example from scikit-learn and then using the SKNN regressor , simple example code from the docs.

# -*- coding: utf-8 -*-

# Code source: Jaques Grobler
# http://scikit-learn.org/stable/auto_examples/linear_model/plot_ols.html
# License: BSD 3 clause

import matplotlib.pyplot as plt
import numpy as np
from sklearn import datasets, linear_model

# Load the diabetes dataset
diabetes = datasets.load_diabetes()

# Use only one feature
diabetes_X = diabetes.data[:, np.newaxis]
diabetes_X_temp = diabetes_X[:, :, 2]

# Split the data into training/testing sets
diabetes_X_train = diabetes_X_temp[:-20]
diabetes_X_test = diabetes_X_temp[-20:]

# Split the targets into training/testing sets
diabetes_y_train = diabetes.target[:-20]
diabetes_y_test = diabetes.target[-20:]

# Create linear regression object
regr = linear_model.LinearRegression()

# Train the model using the training sets
regr.fit(diabetes_X_train, diabetes_y_train)

print "Results of Linear Regression...."
print "================================\n"
# The coefficients
print('Coefficients: ', regr.coef_)
# The mean square error
print("Residual sum of squares: %.2f"
      % np.mean((regr.predict(diabetes_X_test) - diabetes_y_test) ** 2))
# Explained variance score: 1 is perfect prediction
print('Variance score: %.2f' % regr.score(diabetes_X_test, diabetes_y_test))

# Plot outputs
plt.scatter(diabetes_X_test, diabetes_y_test,  color='black')
plt.plot(diabetes_X_test, regr.predict(diabetes_X_test), color='blue',
         linewidth=3)

plt.xticks(())
plt.yticks(())

plt.show()

# Now using the sknn regressor
# http://scikit-neuralnetwork.readthedocs.org/en/latest/guide_beginners.html
#

from sknn.mlp import Regressor, Layer

nn = Regressor(
    layers=[
        Layer("Rectifier", units=200),
        Layer("Linear")],
    learning_rate=0.02,
    n_iter=10)

nn.fit(diabetes_X_train, diabetes_y_train)
print "Results of SKNN Regression...."
print "==============================\n"

# The coefficients
print('Coefficients: ', regr.coef_)
# The mean square error
print("Residual sum of squares: %.2f"
      % np.mean((nn.predict(diabetes_X_test) - diabetes_y_test) ** 2))
# Explained variance score: 1 is perfect prediction
print('Variance score: %.2f' % nn.score(diabetes_X_test, diabetes_y_test))

# Plot outputs
plt.scatter(diabetes_X_test, diabetes_y_test,  color='black')
plt.plot(diabetes_X_test, nn.predict(diabetes_X_test), color='blue',
         linewidth=3)

plt.xticks(())
plt.yticks(())

plt.show()

Results of Linear Regression:

('Coefficients: ', array([ 938.23786125]))
Residual sum of squares: 2548.07
Variance score: 0.47

￼
Results of SKNN Regression:

('Coefficients: ', array([ 938.23786125]))
Residual sum of squares: 5737.52
Variance score: -0.19

Changing the number of iterations to 1000 results in a score of -0.15

My best guess here is that your learning rate is way too high for the problem. You also probably have far more neurons in your hidden network than you need, seeing as you're using just one feature.

Recall that learning rate is controlling the "step size" in gradient descent and that for your dataset, it is likely far too high. I made some minor changes to your code and got better results than linear regression. Notice the use of 2 hidden neurons, a 0.001 learning rate, and 20 iterations.

# Now using the sknn regressor
# http://scikit-neuralnetwork.readthedocs.org/en/latest/guide_beginners.html

from sknn.mlp import Regressor, Layer

nn = Regressor(
    layers=[
        Layer("Rectifier", units=2),
        Layer("Linear")],
    learning_rate=0.001,
    n_iter=20)

nn.fit(diabetes_X_train, diabetes_y_train)
print("Results of SKNN Regression....")

# The coefficients
print('Coefficients: ', regr.coef_)
# The mean square error
print("Residual sum of squares: %.2f"
      % np.mean((nn.predict(diabetes_X_test) - diabetes_y_test) ** 2))
# Explained variance score: 1 is perfect prediction
print('Variance score: %.2f' % nn.score(diabetes_X_test, diabetes_y_test))

# Plot outputs
plt.scatter(diabetes_X_test, diabetes_y_test,  color='black')
plt.plot(diabetes_X_test, nn.predict(diabetes_X_test), color='blue',
         linewidth=3)

plt.xticks(())
plt.yticks(())

plt.show()

SKNN regression:

Results of SKNN Regression....
Coefficients:  [ 938.23786125]
Residual sum of squares: 6123.67
Variance score: 0.50


Issue translating large amounts of tweets using Google Translate I am working on translating large amounts of tweets using this deep-translator which uses the Google Translate API.
Initially everything was fine and tweets were translated with no problems whatsoever but I recently encountered an issue.
The issue with the data currently is that when there are non-English words in a particular tweet as in the attached image, Google Translate marks it as English and does not translate them. In the API I have set the source language to auto-detect the words then translate them to English.

The only workaround I have come up with to solve this is to turn the tweet into chunks and perform batch translations on them.
Example string:
"NEW YOUTUBE VIDEO OUT NOW:TOTTENHAM NEWS TRANSFER WINDOW UPDATE 손흥민 Son Award Link to Premier League Defen... TOTTENHAM NEWS TRANSFER WINDOW UPDATE Carabao Cup Win Final. 손흥민 Son Contract"

Output after chunking the string into batches with a maximum of 5 words:
['NEW YOUTUBE VIDEO OUT NOW:TOTTENHAM', 'NEWS TRANSFER WINDOW UPDATE 손흥민', 'Son Award Link to Premier', 'League Defen... TOTTENHAM NEWS TRANSFER', 'WINDOW UPDATE Carabao Cup Win', 'Final. 손흥민 Son Contract']

The issue is to translate the batch it takes about 13 seconds whereas if I translate the entire string(even though it won’t work) it takes under a second as the batched string performs 6 API request compared to the single API request for the normal string.
The time to translate will be quite high when I batch the strings and translate them.
Assuming it takes 13 seconds per tweet with 5000 tweets in the csv file I am currently working on it will take roughly 18 hours and I have csv files with significantly more tweets than this.
The chunk size of 5 is what I found to still be able to translate the words even if there is a single non-English word in the string. Anything more than that will still not translate the tweets.
Anyone have any workarounds that are faster than this?

So this was my work around:

Convert the list of strings that need to be translated into an indexed tuple with the first value being the index and the second value being the string:

s = [(0, 'NEW'), (1, 'YOUTUBE'), (2, 'VIDEO'), (3, 'OUT'), (4, 'NOW:TOTTENHAM'), (5, 'NEWS'), (6, 'TRANSFER'), (7, 'WINDOW'), (8, 'UPDATE'), (9, '손흥민'), (10, 'Son'), (11, 'Award'), (12, 'Link'), (13, 'to'), (14, 'Premier'), (15, 'League'), (16, 'Defen...'), (17, 'TOTTENHAM'), (18, 'NEWS'), (19, 'TRANSFER'), (20, 'WINDOW'), (21, 'UPDATE'), (22, 'Carabao'), (23, 'Cup'), (24, 'Win'), (25, 'Final.'), (26, '손흥민'), (27, 'Son'), (28, 'Contract')]

Next as the words that were causing issues when translating were ASCII I extracted them using this code:

(index, item) for (index, item) in s if not item.isascii()]
# result
[(9, '손흥민'), (26, '손흥민')]

Then I ran the translations on these words instead of the entire string or the chunks of strings.
After translating I then added them back in the list of tuples
Convert the list of tuples to a list of strings and concatenate them into a single string

This significantly reduced the time to translate the data.


Reproducing randomForest Proximity Matrix from R package in Python I am trying to port this little piece of R code to python:
rf <- randomForest(features, proximity = T, oob.prox = T, ntree = 2000)
dists <- as.dist(1 - rf$proximity)

with parameters
oob.prox: Should proximity be calculated only on “out-of-bag” data?
proximity: if proximity=TRUE when randomForest is called, a matrix of proximity measures among the input (based on the frequency that pairs of data points are in the same terminal nodes).
I am currently trying using sklearn.ensemble.RandomTreesEmbedding for this task, however there is no functionality for the proximity matrix. I found the following developer comment though:

We don't implement proximity matrix in Scikit-Learn (yet).
However, this could be done by relying on the apply function provided
in our implementation of decision trees. That is, for all pairs of
samples in your dataset, iterate over the decision trees in the forest
(through forest.estimators_) and count the number of times they fall
in the same leaf, i.e., the number of times apply give the same node
id for both samples in the pair.

And so I tried, utilizing numpy's pdist() function along with my custom distance (or in this case, proximity) measure. I still have several problems:

The proximity function is outstandingly slow
How to handle the out-of-bag behaviour
How to recreate the exact behaviour of as.dist(1- rf$proximity): I think I need to normalize my count matrix, then subtract it from 1 and then afterwards compute the euclidean distances between its rows!?

My code as of now looks like this:
# grow a random forest from points
rf = ensemble.RandomTreesEmbedding(n_estimators=200,
    random_state=0,
    max_depth=5
)
rfdata = rf.fit_transform(xdata);

# define an affinity measure function to use with numpy's pdist
def treeprox(u, v):
    leafcount = 0
    # needs reshaping for single samples
    u = u.reshape(1,-1)
    v = v.reshape(1,-1)
    a = rf.apply(u)
    b = rf.apply(v)
    # count number of times they fall in the same leaf
    # (use of np forces element-wise)
    c = np.sum(np.array(a)==np.array(b))
    return c

distm = pdist(xdata, proxfun)
distm = squareform(distm)

There must be a better way I guess, since this functionality is readily provided by the R package randomForest.
Any suggestions?
tia

I have written some code for this. It can be found here. In answer to your specific questions:

I have tried to optimize for speed. What I did should be a little faster than the code above.
I do not use out of bag records. In fact the original documentation does not suggest this. I created another post to see if the consequences are understood.
This is handled in my code by normalizing by the total possible leaves that could be matched.


Why does GridSearchCV (sklearn) change the value of n_samples? I thought n_samples is the number of training examples. But when using GridSearchCV, n_samples becomes 32 rather than 50.

Error when using GridSearchCV:

  Expected n_neighbors <= n_samples,  but n_samples = 32, n_neighbors =
  50

Training examples:

print(X_train.shape[0]) => 50

print(len(y_train)) => 50

This works:

neigh = KNeighborsClassifier(n_neighbors=50)
neigh.fit(X_train, y_train)
result = neigh.predict(X_test)

This fails:

from sklearn.model_selection import GridSearchCV

grid_params = {
    "n_neighbors" : [50]
}

g = GridSearchCV(KNeighborsClassifier(), grid_params)
g.fit(X_train, y_train)

I'm confused why n_samples becomes 32 when using GridSearchCV.

The CV stands for CrossValidation, meaning it will split up your training set in a number of folds (in this case 3), train on n-1 of those folds and test on the remaining one. This is why your training is now done on 32 instead of 50 samples. Crossvalidation is useful for estimating how well your model (including specific hyperparameters) does on unseen data.


Why is count encoding effective in improving accuracy? Can someone please explain why/how Count encoding of categorical features improve accuracy in classification  when compared to simply label encoding them ?
I found one explanation in kaggle " Rare values tend to have similar counts (with values like 1 or 2), so you can classify rare values together at prediction time. Common values with large counts are unlikely to have the same exact count as other values. So, the common/important values get their own grouping. " which doesn't seem convincing or I don't understand the reasoning .
Can someone please explain why it performs better than label encoding ? Label encoding can also find frequent patterns to correlate with the target variable , right ?

Its a good question,
I would just like to add my points
Lets assume you have dataset with features (patient: id,execercise_duration: int, fav_products: category) target(diabetes: Binary)
Label encoding will just give numbers to every unique category.
Lets assume Category A is ice-cream and Category B is juice and Category C is chocolates.
Now if Category A is encoded 1 and Category B is encoded 2  and Category C is 3 but you keep the encoded feature as numerical series then it would simply mean Category C > Category B > Category A (since 3> 2 > 1). But is it the right information to send to model ?
I guess not.
Intution says people with fav_products as ice-cream and chocoloate will be diabetic.
Category A and B and C just represent three different things nothing is large or small in them.
But if you send frequency or count then lets say more observation in data are of ice-cream, chocoloates and less are of juice. Becuase usually icecream and chocolate are more desirable food than juice. Frequency or count of ice-cream and chocolate will be more than juice.
So keeping frequency or count encoded feature as numerical can give information to model that when this encoded feature value is high outcome is diabetes and when it is low outcome is non-diabetic.
Note: A more complex model like decison tree may be able to give good accuracy even with label encoding atleast for this simple example.


Evaluation of semantic segmentation network with mAP I am interested in evaluating a semantic segmentation network. I've seen lots of challenges such as PASCAL VOC use the mean average precision metric(mAP). I understand how this would work with an instance detection approach, however I am unsure how it works for networks which just provide a  class label to each pixel. Should I treat each image as a single instance of each class?

Thanks

Here is a great post which explains the evaluation methods for semantic segmentation: https://www.jeremyjordan.me/evaluating-image-segmentation-models/

It also describes the mean average precision at the end.


Left Join with b.key being NULL in R I am trying to replicate the below sql query in R

select a.*, b.key from Table1 a
LEFT OUTER JOIN Table2 b
on a.key = b.key where b.key is null

I have read through this post however I am still struggling to code my specific case. https://stackoverflow.com/questions/1299871/how-to-join-merge-data-frames-inner-outer-left-right

I have tried the below but the result doesnot allow me to filter for b.key IS NULL

LoansToInsert_stg1 <- merge(x = Prior_stg2, y = BankLoans_stg2,
                            by = "Account_ID", all.x = TRUE)

Any insights?

Example:

Key1 <- c("A1","A2","A3","A4","A5")
Key2 <- c("A1","A2","A3","B4","B5")
BV1 <- c(100, 200, 300, 400, 500)
BV2 <- c(150, 250, 350, 450, 550)
df1 <- as.data.frame(cbind(Key1, BV1))
df2 <- as.data.frame(cbind(Key2, BV2))

Expected Output as a new df:

Key1 BV1 Key2 BV2
A1 100 A1 150
A2 200 A2 250
A3 300 A3 350
A4 400 NA NA
A5 500 NA NA

If I understand correctly:

Table1 <- data.frame(key = seq(1,100),a.data = rnorm(100))
Table2 <- data.frame(key = c(seq(1,30),rep(NA,30)), b.data = seq(1,60))

##Assuming this is what you want
library(sqldf)
sql.ans <- sqldf("select a.*, b.key from Table1 a LEFT OUTER JOIN Table2 b on a.key = b.key where b.key is null")

## dplyr version
library(dplyr)

dplyr.ans <- Table1 %>% filter(!key %in% Table2$key)

## Regular R version
R.ans <- Table1[which(!Table1$key %in% Table2$key),]

EDIT after dummy data and expected output

dplyr.ans2 <- left_join(df1,df2, by = c("Key1" = "Key2"))
 Key1 BV1  BV2
 1   A1 100  150
 2   A2 200  250
 3   A3 300  350
 4   A4 400 <NA>
 5   A5 500 <NA>


Temporal Difference Learning Getting Stuck I'm trying to implement a temporal difference algorithm that learns the maximum revenue over a period of time using prices as the actions, inventory as the state, and revenues realized as the reward. The problem I'm having is that I can't seem to get it to converge on an optimal policy. It seems like it gets stuck at a revenue somewhere around 60% of optimization and then won't budge anymore. Are there some common pitfalls that might be causing this? I've tried playing with the rate of exploration a little bit, but that hasn't seemed to help it much.

EDIT:
Ok, so I went back through everything, and it seems like the problem is that, when it stops exploring, it just keeps increasing the Q-values of states that it's already deemed the best in the past. So for example, it visits the price 5 at a certain state and time and gets a reward. Then, in the next several episodes, it continues to visit 5, and continues getting a reward, adding that reward to the Q-value until it's pretty high. At that point, even if it were to explore at the same state and time, the reward it gets isn't enough to overcome the inflated Q-value, so it just goes right back to 5 in the next episode. Here are the steps I'm trying to follow.

Sounds like an interesting application. To debug RL applications, I like to perform rollouts. First identify a state where the current policy appears to be clearly wrong; (maybe sample some states and look at them individually). Then switch off the learning and run a sample from that state for both the current RL policy and the action you think is correct. This should give you the true action values. Hopefully, that will give you a hint as to what's going wrong. If it's an exploration problem, then the current policy won't be trying any actions similar to the action you think is correct. If the form of the value function is wrong, then it won't fit the values coming from your samples.


Does high regression coefficient for Principal components that don't explain much variance imply that my data is not a good predictor? There isn't much to add to the question.
Essentially i had some data that I reduced to 4 principal components, the first two components of which explain 99% of the variance in my data.
Upon building a linear regression model using these 4 components, the coefficients for component 3 and 4 is significantly higher (by an order of 20) in magnitude than the components 1 and 2.

Without having to look at accuracy of the model, is this information sufficient to conclude that my data is a very weak linear predictor of the dependent variable?

The variance of the data is just that, variance. Imagine I'm trying to predict y from x1 and x2, and the true underlying model is y = x1 + e, where e~N(0,.001) and x1~N(0,.01), and x2 is drawn independently from N(0,1). There is significantly more variance in x2, but it is useless in predicting y, where x1 would be an extremely good predictor.

EDIT:
We can implement this in R.

e = rnorm(100,0,.001)
x1 = rnorm(100,0,.01)
x2 = rnorm(100,0,1)
y = x1 + e
x = data.frame(cbind(x1,x2))
pca1 = prcomp(x)
pca1
summary(pca1)
model <- lm(y ~ ., data=data.frame(pca1$x))

Here is what our PCA looks like

PC1           PC2
x1 0.0003702204 -0.9999999315
x2 0.9999999315  0.0003702204
Importance of components:
                          PC1     PC2
Standard deviation     1.0466 0.01098
Proportion of Variance 0.9999 0.00011
Cumulative Proportion  0.9999 1.00000

So component PC1 represents 99.99% of our variance. Wow it must be a good predictor! Let's look at our model.

Call:
lm(formula = y ~ ., data = data.frame(pca1$x))

Coefficients:
             Estimate Std. Error t value Pr(>|t|)
(Intercept) 7.589e-04  1.017e-04   7.463 3.64e-11 ***
PC1         3.561e-05  1.035e-04   0.344    0.732
PC2         1.002e+00  1.039e-02  96.424  < 2e-16 ***

Oh. It turns out it's useless for prediction.


How to deal with a feature that has lot of categorical values? I know this question has been asked before and I have tried a few things but those things are not working as expected for my usecase.

I have a 500 length feature vector. One of these features is a categorical value pincode. For our dataset, the pincode can take more than 20,000 unique values. So we can't used one hot encoding because it will blow up our feature space.

I have also tried Binary Encoding which assigns a unique integer to every unique categorical value and then converts it into Binary. It then treats every bit as a column. This way the dimension of the pincode feature vector had reduced from 20,000 to around 20. But I didn't find any improvement in our evaluation metrics which is AUC (Area under Curve). Intuitively also it didn't felt right thing to do.

I also tried to apply PCA on the pincode to reduce the dimensions. I tried to reduce the dimensions from 20,000 to 100. Upon applying PCA, the model performed worse. Also, I read somewhere that PCA doesn't work well on categorical values. It's better for continous values.

So what do we do with this feature? We don't want to throw it away as we think that it might be an important feature. But we want to reduce it dimension and then use.

Apologies if this is a basis question. I am new to this field and trying various things out.

P.S - We are using xgboost for training our model.

one idea is to craft some features of this feature - when you classify cars you don't have data like strings "Ferrari 991 year 2014 red" "BMW z4 year 1999 2.0L blue" but you would like to have columns like "Manufacturer", "Type(SUV/Cabrio...)", "Year", "Engine" etc.

you could transform it or just discard


How to calculate accuracy, precision and recall, and F1 score for a keras sequential model? I want to calculate accuracy, precision and recall, and F1 score for multi-class classification problem. I am using these lines of code mentioned below.

from keras import backend as K
def precision(y_true, y_pred, average='None'):
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
    precision = true_positives / (predicted_positives + K.epsilon())
    return precision

def recall(y_true, y_pred, average='micro'):
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
    recall = true_positives / (possible_positives + K.epsilon())
    return recall

def f1(y_true, y_pred, average='weighted'):
    def recall(y_true, y_pred):
        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
        possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
        recall = true_positives / (possible_positives + K.epsilon())
        return recall

    def precision(y_true, y_pred):
        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
        predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
        precision = true_positives / (predicted_positives + K.epsilon())
        return precision

    precision = precision(y_true, y_pred)
    recall = recall(y_true, y_pred)
    return 2 * ((precision * recall) / (precision + recall + K.epsilon()))

Don't know why the value of recall is 1 for both testing and training.
Please help me to calculate accuracy, precision and recall, and F1 score for multi-class classification using the Keras model.

if you want to compute this for every class then:

def recall(y_true, y_pred,class_to_analyse):
     pred = K.argmax(y_pred)
     true = K.argmax(y_true)
     p = K.cast(K.equal(pred,class_to_analyse),'int32')
     t = K.cast(K.equal(true,class_to_analyse),'int32')
     # Compute the true positive
     common = K.sum(K.dot(K.reshape(t,(1,-1)),K.reshape(p,(-1,1))))
     # divide by all positives in t
     recall = common/ (K.sum(t) + K.epsilon)
     return recall

def precision(y_true, y_pred,class_to_analyse):
     pred = K.argmax(y_pred)
     true = K.argmax(y_true)
     p = K.cast(K.equal(pred,class_to_analyse),'int32')
     t = K.cast(K.equal(true,class_to_analyse),'int32')
     # Compute the true positive
     common = K.sum(K.dot(K.reshape(t,(1,-1)),K.reshape(p,(-1,1))))
     # divide by all positives in t
     precision = common/ (K.sum(p) + K.epsilon)
     return precision

def fbeta(y_true, y_pred,class_to_analyse):
    beta = 1 # for f1 score
    precision = precision(y_true, y_pred,class_to_analyse)
    recall = recall(y_true, y_pred,class_to_analyse)

    beta_squared = beta ** 2
    return (beta_squared + 1) * (precision * recall) / (beta_squared * precision + recall)

to make it work you need to create a list of function with a fixed class (ex: recall_1(y_true, y_pred) = recall(y_true, y_pred,class_to_analyse = 1)


What is a visual bag of words and how is it implemented? I'm currently working on implementing a bag of visual words in Python. I get the general gist of how it works but I can't seem to find any sources that explain it in more detail to a level where I can implement it. I'm guessing scikit learn and scikit image would come in but I can't seem to point myself in the right direction. Any help?

The necessary information can be found on wikipedia.
"when we use more technical features such as colour histograms"
Judging from this sentence I guess you need to understand the "Codebook" generation.
First step is to extract features of patches in the image. For efficiency you only want to take patches which are interesting and calculate discriminative features on them. SIFT is one Method which performs both steps for you. It takes care of finding good spots and it calculates features on this spot.
Now you can generate your codebook. A codebook will map every possible featurevector ( after all they're just numeric vectors ) to a certain output codeword. One possibility to do this, is to use k-means for codebook generation. After you built your codebook, a vector is mapped to a code by finding the minimal distance to all the entries ( since you used k-means you can use euclidean distance ).
Now you have a complete realization of the bag of words model. You can now dive into using it for classification.
The required algorithms can be implemented using the libraries you mentioned.


How can I fit categorical data types for random forest classification? I need to find the accuracy of a training dataset by applying Random Forest Algorithm. But my the type of my data set are both categorical and numeric. When I tried to fit those data, I get an error.

  'Input contains NaN, infinity or a value too large for dtype('float32')'.

May be the problem is for object data types. How can I fit categorical data without transforming for applying RF?

Here's my code.

You need to convert the categorical features into numeric attributes. A common approach is to use one-hot encoding, but that's definitely not the only option. If you have a variable with a high number of categorical levels, you should consider combining levels or using the hashing trick. Sklearn comes equipped with several approaches (check the "see also" section): One Hot Encoder and Hashing Trick

If you're not committed to sklearn, the h2o random forest implementation handles categorical features directly.


How to use LSTM to make prediction with both feature from the past and the current ones? Suppose I have a data frame with 2 columns, which are sales and promotions. I want to predict the next day sales based on the past sales and promotion info of 3 days, plus the promotion to be applied at the next day? How do I process and reshape the dataframe? I mean if only previous promotions need to be considered, then, after some shifting, the data frame could be reshaped into (sample size, 3, 2), but it becomes a problem if I also need to consider the promotion at the next day. It is a pretty common issue, does anyone has any thought about this?

Have a look at this question. There is a nice discussion about how to implement this. The idea in to create a separated Input to your model and concatenate it AFTER the recurrent layer(s). Also in the Keras documentation, there is an example on how to build such models with a few lines of code.


Is Label Encoding with arbitrary numbers ever useful at all? From what I read online, there seems to be some confusion regarding the taxonomy and the terms used, so to avoid misunderstanding I'm going to define them here:
Label Encoding - encoding a nominal variable with arbitrary numeric labels.
Ordinal Encoding - encoding an ordinal variable with numeric labels arranged in a specific order.
The course on Machine Learning I'm currently taking compares One-Hot Encoding with Ordinal Encoding. However, during my research online I came to realize that "Ordinal Encoding" is actually a misnomer, and what that course actually demonstrates is called "Label Encoding". Ordinal Encoding is supposed to pertain strictly to ordinal variables, and the dataset in question didn't even have any ordinal variables.
Where did that misnomer come from? Turns out that it comes from the scikit-learn library that has LabelEncoder and OrdinalEncoder classes. The thing is, OrdinalEncoder class actually does not perform Ordinal Encoding by default. To make it ordinal, you have to specify the order in the 'categories' parameter (and its usage is extremely not-user-friendly - dictionary mapping by pandas can do the same way easier). If you don't, OrdinalEncoder will assign labels alphabetically, just like LabelEncoder does. So the real difference between these two classes is that one encodes only a single column, while the other encodes two or more columns at a time. Perhaps it would be better and much less confusing if these classes were called "LabelEncoder1D" and "LabelEncoder2D".
So that's where mistakenly calling Label Encoding "Ordinal Encoding" is coming from. But getting back to the question, the course I'm taking advocates the usage of (what I learned to be) Label Encoding for tree-based algorithms, because One-Hot encoding performs much worse for trees (which is ture). However, from what I read online, it appears that other Machine Learning platforms, such as R or H2O, are capable of processing nominal variables for trees without any kind of encoding at all, and the requirement to encode everything into numeric form seems to be exclusively scikit-learn's problem. And there's conflicting information as to whether trees perform better with Label Encoding - my course, as well as some responses online, advocate for its usage, but my intuition, as well as some other responses online, seem to indicate that scikit-learn trees will not be able to distinguish these labels as categories, and will mistakenly assume that they are continuous values on a meaningful scale instead. So they recommend using One-Hot Encoding even for trees as the only option despite it being sub-optimal.
So my questions are 1) is it true that Label Encoding will be misinterpreted as a numeric scale by scikit-learn trees? 2) if so, are there any situations at all where arbitrary Label Encoding can be useful? Or does this technique has no use at all unless the variable is ordinal, and a specific labeling order is given?
P.S: I'm asking because my course has a whole lesson dedicated to teaching students "Ordinal" Encoding. At first I wanted to suggest them to rename it to "Label Encoding", but now I suspect that that whole lesson is best removed altogether to avoid teaching students bad practices.

First, I generally agree that encoding unordered categories as consecutive integers is not a great approach: you are adding a ton of additional relationships that aren't present in the data.
CART
First, let me point out (because I nearly forgot) that there are two main types of decision tree: CART and the Quinlan family.  For the Quinlan family, categorical variables are dealt with by using higher arity splits, so no encoding is needed and this is mostly moot.
Q1, yes, ordinally encoding will be treated by the model as numeric (unless some other parameter controls that, e.g. LightGBM).  But for (most) trees, only the order is actually relevant: the scale is irrelevant, and e.g. the relationship "10 is twice as much as 5" is completely invisible to the tree.
As you point out, one-hot encoding for a CART model can be detrimental, especially when there are many levels in a categorical feature: will the tree ever actually decide to split on one of the dummy variables, if it is only 1 for a small subset of the data?
(Q2)But when you encode ordinally, there will just by chance be some splits that are useful and split many levels in each direction.  (You may even try more than one random ordering of the levels as different features!)
And yes, presumably the best approach is to use an implementation that can take advantage of the raw categoricals, using the average-response trick.  (There's even some debate on how much that helps: some studies have been done, but generally the datasets are synthetic or too small to be representative.)
Other models
In other models, very often one-hot encoding is just fine, and doesn't suffer from the same problem as trees.  If there are too many levels, and especially if some of them are too small, you may consider smoothing techniques to avoid overfitting.  (Q2)I'd be surprised if ordinally encoding is ever worth it for most models, but one would need to consider each model type individually, and probably do some testing.
Names
As for naming, things are a bit muddy, but I don't think this is sklearn's fault.  The "label" in LabelEncoder means it is supposed to be used on the labels, a.k.a. the dependent variable.  And for that usage, there is no debate about whether it's appropriate: sklearn just requires consecutive integer labels for its multiclass classification; it doesn't use the numeric values as though they were mathematically meaningful.
As for OrdinalEncoder, it is meant to be used with input ordering of the categories.  See sklearn Issue#13488 below.  But one could argue that you are encoding the categorical variable in an ordinal way, so even with unordered categories this isn't necessarily a misnomer.
See also
See Issue#13488 for some related discussion.


The actual results and results from pickle files are not matching in pandas for DBSCAN clustering I've built a DBSCAN clustering model. The output result and the result after using the pickle files are not matching.

Based on HD and MC column, I am clustering WT column.

data = HD,MC
Target = WT

Below, for 1st record the cluster is 0.

But after running it from 'pkl' file, it is showing predicted result as [-1]

Dataframe:

      HD         MC             WT         Cluster
      200        Other          4.5        0
      150        Pep            5.6        0
      100        Pla            35         -1
      50         Same           15         0

Code:

 le = preprocessing.LabelEncoder()
 df['MC encoded'] = le.fit_transform(df['MC'])

 col_1 = ['HD','MC encoded']
 data = df[col_1]
 col_2 = ['WT']
 target = df[col_2]
 data = data.fillna(value=0)

 model = DBSCAN(eps=1, min_samples=20).fit(data)
 outliers_df = pd.DataFrame(data)
 print(Counter(model.labels_))

 x = model.fit_predict(target)
 print(Counter(x))

Result:

  Counter({-1: 604, 0: 142, 1: 83, 9: 36, 2: 27, 7: 26, 10: 26, 8: 24, 4: 23, 5: 23, 3: 22, 11: 21, 6: 20, 12: 20, 13: 20})
  Counter({0: 1093, -1: 24})

Code:

  df["Cluster"] = x

  filename1 = '/model.pkl'
  model_df = open(filename1, 'wb')
  pickle.dump(model,model_df)
  model_df.close()

  output = open('/MC.pkl', 'wb')
  pickle.dump(le, output)
  output.close()

  with open('model.pkl', 'rb') as file:
     pickle_model = pickle.load(file)

  pkl_file = open('MC.pkl', 'rb')
  le_mc = pickle.load(pkl_file)
  pkl_file.close()

 def testing(HD,MC,WT):
     test = {'HD':[HD],'MC':[MC], 'WT':[WT]}
     test = pd.DataFrame(test)
     test['MC_encoded'] = le_mc.transform(test['MC'])
     pred_val = pickle_model.fit_predict(test[['HD','MC_encoded']])
     print(pred_val)
     return(pred_val)

      pred_val = testing(200,'Other',4.5)

Result:

    [-1]

It appears your pickle file isn't being loaded as a pandas dataframe. Why not just use df_pickle = pd.read_pickle('/MC.pkl') – the rest should fall into place after.


Distinguish Multi-Task vs Single-incremental Task in Continual Learning I read through the Internet and found this:

  Most of Continuous Learning studies focus on a Multi-Task scenario, where the same model is required to learn incrementally a number of isolated tasks without forgetting the previous ones.

And they explain PermutedMNIST is Multi-Task. I have no idea why?

In other words,

  Single-Incremental-Task considers a single task which is
  incremental in nature. In other words, we still add new classes
  sequentially but the classification problem is unique and when
  computing accuracy we need to distinguish among all the classes
  encountered so far.

However, I still can not clarify two these definitions? Could someone help me out and give me example of Multi-Task scenario and Single-Incremental-Task scenario.

TL;DR/Summary: The classes ($y1$, $y2$, $y3$ below) in multi-task can be anything (it may be that $y1 \cap y2 = \emptyset$, $y1 \cap y2 = \emptyset$, and so on).  In Incremental we take the labels (and data from a common set, (i.e. $y[:2] \subseteq y[:4] \subseteq y$, by the definition of subsetting)

It is just a question of the interpretation of the definition.  The definitions look very similar to each other but the devil's in the details.

Assuming that the model can distinguish the upper bond number of classes: For example, it is an ANN with $N$ neurons in the output layer and the number of classes ($k$) in the task with most classes is $k <= N$ (multi-task) or the total number of classes ($k$) is also $k <= N$ (single-incremental-task); we can say that:

Multi Task

Here we will train the model on different tasks over time, this is often called as reinforcement learning.  In semi-python pseudo code (where .train already includes things like cross validation):

model = Whatever(...)

X1 = [[1, 0],
      [2, 2],
      [3, 0]]
y1 = [0, 1, 0]
model.train(X1, y1)

X2 = [[4, 4],
      [5, 5],
      [6, 0]]
y2 = [1, 2, 0]
model.train(X2, y2)

X3 = [[7, 0],
      [8, 8],
      [9, 9]]
y3 = [0, 1, 1]
model.train(X3, y3)
score = model.score(X3, y3)

Here the tasks may or may not be related.  Or often are slightly related (e.g. identifying different types of objects in each training).

Single Incremental Task

This is also training the model several times, here we have a single task in X but do not feed the entire dataset at once.  In semi-python pseudo code:

model = Whatever(...)

X = [[1, 0],
     [2, 2],
     [3, 0],
     [4, 4],
     [5, 5],
     [6, 0]]
y = [0, 1, 0, 3, 2, 0]

model.train(X[:2, :], y[:2])
score1 = model.score(X[:2, :], y[:2])
model.train(X[:4, :], y[:4])
score2 = model.score(X[:4, :], y[:4])
model.train(X, y)
score3 = model.score(X, y)

Here the task is one but it may be a big one.  One place where this technique is used is to build a learning curve, which is one way of evaluating if we have enough data to understand the variation of the task.

Extra note: in the multitask case we said that $y1 \cap y2 \cap y3 = \emptyset$ could be (and most likely is) possible.  One example would be: $y1$ are different models of cars and $y2$ are different models of ships. And the question is: Do understanding different models of cars help with differentiation different models of ships?

(P.S. ys will always be enumerated from 0 up to the number of classes, i.e. the numeric values of y will always be the same but their class meaning does not need to be).


Regression in Python with many NaN values spread across all columns I want to do a regression to predict "value" based on the other columns from below example table. The data was collected by single indicator and not across all data points, resulting in many NaN/blank values:

value     age     education     gender
32.3                            Male
31.8                            Female
32.8              High school
33.8              Technical school
26.4              College graduate
16.3     18 - 24
35.2     25 - 34
35.5     35 - 44

I converted categorical data by using dummy variables which resulted in below column examples. I guess that the quality of my model will be affected because I have only a single 1 by row and the rest is all 0.

value   18 - 24   25 - 34   35 - 44   College   High school
32.8       0         0         0        0          1
26.4       0         0         0        1          0
16.5       1         0         0        0          0

So my question is, what is the best way to clean and convert the data for given source data structure?

If you have 3 completely different data sources where there are no common columns consider creating 3 separate models instead of trying to force them into one.

If you have overlapping columns as well handling of NaN values can be influenced by your model choice. (i.e. most tree based model can handle missing values) If you use linear regression you need to have numerical values and you can experiment with several options:

adding new variables indicating if a certain variable is missing or not (0/1)
instead of using 0 for the missing values experiment replacing them with either median/mean/random value from the original distribution of the variable (as a result your new data will have the same distribution of values as the original one)
create model for predicting variables containing missing values and use it to replace NaNs
use some dimensionality reduction if your matrix is too big and sparse


Elasticsearch + Clustering I'm currently working on a project that relies on the clustering of documents into an unknown number of clusters, based on a similarity threshold (ideally using cosine distance between tf-idf vectors).
I'm attracted to elasticsearch for the project due to the 'out-of-the-box' similarity metrics provided when querying by string, however, I'm looking for some guidance as I'm very new to this. If anyone could provide any critique of the following approach, I'd be very grateful.
Is this whole approach horrible and inefficient? How feasible is this? Am I asking 'too much' of ES for this task?
Any help would be greatly appreciated, sorry for the long read. Thanks :)

I'm afraid I have zero experience with ElasticSearch, so I can't help with that but here are a few thoughts about the process you propose.
Overall I think the approach makes sense, especially in terms of efficiency but also check the following points:

This is not strictly speaking a clustering task, in the sense that the system doesn't find itself how to separate the clusters since this is based on a predefined threshold.
The threshold is crucial and you need a method to determine it. You might also want to plan for updating the threshold as the collection grows. The process is likely to lead to some very big clusters and many very small ones (unique document in a cluster).
If you assign a document to the best match cluster, it means that a document belongs to a single cluster. Depending on the application this could be problematic, with some documents expected in a cluster but found in another.


Convert json to dataframe in python I have a json array f below format

[{
    'Address': 'xxx',
    'Latitude': 28. xxx,
    'Longitude': 77. xxx,
    'reached': False
}, {
    'Address': 'yyy',
    'Latitude': 18. yyy,
    'Longitude': 73. yyy,
    'reached': False
}]

i want to convert into dataframe. if the column name is same it should have (Address_0, Address_1 etc) and should be side by side, not below. How can i do this?

Have you tried using the pandas.read_json method? (documentation)

And it looks like your json is structured like 'records' so use

pd.read_json(_, orient='records')


Missing Values in Classification I'm working on a classification problem. I'm trying to build a model which can predict if a bank client will get a loan or not. Some of clients have co-borrower and the majority don't.

I also have information on co-borrowers like salary, etc. but as the majority of clients don't have co-borrower, I have missing values.
How can I impute this missing data ?

thank you

As @shadowtalker indicated, a binary features indicating the existence of co-borrower should be helpful anyway. As you certainly know, "imputing missing values" as a procedure for guessing value of missing values is absolutely meaningless as it produces values for something which do not exist. If it's about salary and similar numeric data, just simply put 0 (or the neutral value. Whatever it is.) and if it's about nominal variables, add a new value equivalent to None. at the end, all single customers must have exactly same values for these features.

Also one practical idea could be dividing your data to single borrowers and 2-borrowers at the very beginning of classification so the whole model will become like this:

Above model is somehow equivalent to the feature-selection property of some classifiers such that Decision Trees. In a Bayesian Network, such a feature could be the root node.


Classification problem using features with unequal sizes I am relatively new to Machine Learning/ Deep Learning and currently I am working on a classification problem. I have many 2D images and each of them is a cross section of a specimen showing the deformation after applying a load. The images show the edges of the specimen with the same color and the deformed edges are shown with different colors. I want to use the values of pixels as one of the feature to my classification problem. At the same time I also want to use the load value, the co-ordinates of the position of impact as features. Therefore my problem would look something like this

     Feature 1           Feature 2          Feature 3         Class
Vector of size N*1  Vector of size 3*1   Scalar value m1     Class A
Vector of size N*1  Vector of size 3*1   Scalar value m2     Class B

and so on..

I would like to know if there is a way to deal with this kind of problem or if there are any papers which focus on approaches to deal something similar to this. Any kind of help is really appreciated. If any clarification is needed from my side, do let me know.
Thank you.

I am not sure if what I am gonna say is totally correct but you could try it. So instead of treating each of the pixels as part of one feature, you could append everything to make one array. That array would be of size N+3+1. So this would be the size of one row in data matrix and further you could provide weightage to the elements. Like if pixels are more important then mulitply them by some factor. One problem that I see in dealing with this might be standardization of data matrix.


ValueError: Found array with 0 sample(s) (shape=(0, 13)) while a minimum of 1 is required I am new to this logistic regression and finding this error. Please help.
solver='lbfgs'
def lr_model(X_train, y_train):

    # import the LogisticRegression class
    from sklearn.linear_model import LogisticRegression

    # create the model object
    model = LogisticRegression()

    # fit the model to the training data
    model.fit(X_train, y_train) #error here
    time.sleep(0.1)

    #return the fitted model
    return model

# Test
tmp_model = lr_model(X_train[0:12], y_train[0:12] ) #showing error here
print(tmp_model.predict(X_train[4:5]))
print(tmp_model.predict(X_train[5:6]))

It looks like there are no rows in your X_train array or y_train array. Check the shape and contents of X_train using
np.shape(X_train) if X_train is an array or X_train.shape if X_train is a pandas dataframe.
Also, if you are using a pandas dataframe for X_train, you should change tmp_model = lr_model(X_train[0:12], y_train[0:12] ) to tmp_model = lr_model(X_train.iloc[0:12,:], y_train[0:12] )  as this way you are correctly accessing the rows and columns of the pandas dataframe.


Creating an easy but not trivial dataset I am working on the problem of automatic punctuation: given a stream of words, decide for each word whether there should be a punctuation mark after it (in future work I also want to distinguish between different punctuation marks, but currently it's a binary classification problem).
My classifier uses a bidirectional LSTM whose output is fed into a multi-layer perceptron. The accuracy of the current classifier on the entire dataset is not good enough. In order to understand why, I want to create an "easier" dataset - a dataset on which even a simple (but not trivial) classifier would give good results. Then, I will make the dataset more and more difficult and see what improvements have to be made to the classifier. What is a good technique for creating synthetic datasets that are easier (but not trivial) to classify correctly?

In case it is relevant, here are some details on my system:

Each word is represented by an embedding vector of length 200.
The embedding vector is fed into a bidirectional LSTM with an output length of 50.
The LSTM output is fed into a multi-layer perceptron with two layers, the hidden layer has size 60.
The dataset has about 2000 paragraphs, each of which contains about 200 words.
Since the classes are imbalanced (only 20% of the words are punctuated), I check the system accuracy using Cohen's kappa coefficient - a number between -1 and 1. A random classifier has  kappa around 0. A good classifier has kappa above 0.7. My current classifier has kappa around 0.4 (average of 10-fold cross-validation), that is, better than random but not very good.
I train the classifier in 80 iterations. After each iteration I check the kappa of the model on a validation set. Finally I return the model with the highest kappa.

You might use a Markov chain for generating a simple enough dataset. You can estimate the probabilities of punctuation by scanning a large corpus of text (e.g. wikipedia), then you can use the model to generare fictious sentences with punctuation marks. You can set the order of the Markov chain so as to make the generated dataset increasingly closer to real-world text. The Markov chain should be defined on the set of words, however, so you have to carefully choose the vocabulary.


How to add attention mechanism to my sequence-to-sequence architecture in Keras? Based on this blog entry, I have written a sequence to sequence deep learning model in Keras:

model = Sequential()
model.add(LSTM(hidden_nodes, input_shape=(n_timesteps, n_features)))
model.add(RepeatVector(n_timesteps))
model.add(LSTM(hidden_nodes, return_sequences=True))
model.add(TimeDistributed(Dense(n_features, activation='softmax')))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(X_train, Y_train, epochs=30, batch_size=32)

It works reasonably well, but I intend to improve it by applying attention mechanism. The aforementioned blog post includes a variation of the architecture with it by relying on a custom attention code, but it doesn't work my present TensorFlow/Keras versions, and anyway, to my best knowledge, recently a generic attention has been added to Keras -- I was not able add it to my code, however.

Additionally, I tried to complicate my architecture above by adding 2-2 LSTM layers for the encoder and the decoder respectively instead of 1-1 with this:

model = Sequential()
model.add(LSTM(hidden_nodes, return_sequences=True, input_shape=(n_timesteps, n_features)))
model.add(LSTM(hidden_nodes, return_sequences=True))
model.add(RepeatVector(n_timesteps))
model.add(LSTM(hidden_nodes, return_sequences=True))
model.add(LSTM(hidden_nodes, return_sequences=True))
model.add(TimeDistributed(Dense(n_features, activation='softmax')))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(X_train, Y_train, epochs=100, validation_split=0.15, batch_size=32)

but I get error message (in the 2nd or 3rd row, I assume):

ValueError: Input 0 of layer repeat_vector_17 is incompatible with the layer: expected ndim=2, found ndim=3. Full shape received: [None, 20, 128]

What could be the reason here?

By default an LSTM model returns only the output of last timestep.
model = Sequential()
model.add(LSTM(hidden_nodes, input_shape=(n_timesteps, n_features)))
##output shape is (n_features)

So the below step is needed to repeat the output vector 'n' number of times where 'n' should be the number of time-steps
model.add(RepeatVector(n_timesteps))
##now shape becomes (n_timesteps,n_features)

But when you specify 'return_sequences=True', LSTM returns a hidden state for ALL timesteps. The output shape from the LSTM directly is (n_timesteps,n_features). So you DONT need to to a 'Repeat Vector'
So to eliminate the error, just remove line 4
Edit - I would recommend using the 'return_sequences=true' option and NOT using Repeatvector option even though the latter may compile. This will lead to better results as you are passing far more data across timesteps to the next layer and this is the accepted approach for most situations


Predicting sequences newbie question I have a ranked list of rows of 100 lines of data

1-    8  4  0  5  9  3

2-    0  3  3  5  3  2

3-    0  0  2  4  0  2

..

100-  0  2  3  2  2  0

Is it possible to predict

a) when given a new sequence where it would fit?

b) a way to generate a row that would fit into the top 1, top 5, top 10

The short answer is yes.

What you are looking at would be developing a model that would regress an input of 6 integers to a single integer between 0 and 100.  $f(x) : \mathbb{Z}^6 \rightarrow \mathbb{Z}^1$

This could also be done as as 100 class classification problem.

The regression approach would be better in my opinion.

One big question that you need to know to better determine if machine learning is the correct approach:  are the sequences (e.g., 0 3 3 5 3 2) the only sequences that = the given ranks (e.g.,1).   If so you only need to match sequences, no need for M L.

If no, and you chose to develop an ML approach, an issue you have is that you only have 1 observation for each integer.

Depending on the variability of the input integers and their output rank {1 - 100} you would want multiple observations corresponding to each integer.

One last thing, I would start with one linear and one non-linear algorithm to see which method fits this data best.


The mix of leaky Relu at the first layers of CNN along with conventional Relu for object detection First of all, I know the usage of leaky RELUs and some other relevant leaky activation functions as well. However I have seen in a lot of papers on object detection tasks (e.g YOLO) to use this type of activators only at the first layers of the CNN and afterwards a simple RELU follows at the end. Regarding this, how we end up with a model which uses a leak at the first layers and then a conventional RELU at the end?

Secondly, as far as I'm concerned because of the vanishing gradient problem the neurons at the beginning tend to fall into zero more often than those at the top of the network and then it is very difficult (or even impossible) to activate again; Wouldn't be correct to allow the negative gradient at the whole pipeline of the Neural Network?

how we end up with a model which uses a leak at the first layers and then a conventional RELU at the end?

What matters is to add a non-linearity to the outputs of a neuron. Consequently, by employing each function that adds non-linearity, the network will work due to the derivative that can backpropagate the differentiation of the error term. If you use Relu or leaky Relu, the update terms will change but your network will work fine. The reason that leaky version is used in the mentioned papers is due to avoiding dying relu problem which can happen a lot for regression problems.

About your second question, consider that it usually suffice to use leaky relu in the first layers and due to them, the chance of deep neurons to be stuck at zero is not very much as the results of those papers show. You can use leaky version all over the network but by experience, relu is very fast to be trained!


Detect the time at which deviation occurs in time series data I working on multivariate time series data. I have sensor data generated by a machine every time it is operated. Data set consists of machine_ID(machines of same model), hours_ operated, measurements from various sensors. The machine starts to degrade after operating for certain hours. I would like to find the hours after which there is step change after which the performance starts to degrade.

I want to do this using machine learning approach preferably and would like to plot the graph marking the deviation. Which ML techniques could be used for this approach.

I have performed exploratory data analysis where I could find the point at which there is deviation occurrence. Now, I want to confirm this by running a model to detect the occurrence of step change. In the figure above, the decline starts somewhere at 100 and decline gradually. Now, is there any way I could find this pint through models.

I greatly appreciate any links or suggestion to deal with this.

Thanks in advance.

AnomalyDetection is an open-source R package to detect anomalies
  which is robust, from a statistical standpoint, in the presence of
  seasonality and an underlying trend.

A blogpost that introduces the package can be found here and more formal paper can be found here.


Digits Localization on Streets View House Numbersm I am trying to learn a bit of deep learning playing with the Street View House Numbers data set. I have managed to recognize sequences of digits and I'd like now to train a CNN to localize digits and provide boxes coordinates. The problem is that I have the boxes coordinates related to the original image sizes which are always different.

I have to resize the images in order to have homogenous input for the NN but I don't know hot to transform the boxes coordinates coherently.

Since you are saying that you got box coordinates in the original image, why not reduce coordinates by exactly same scale.

For example a coordinate say $(100,100)$ in the image of say $(1024,1024)$ size when resized to say $(256,256)$ will be $(25,25)$ (assuming you didn't crop anything in the mean while), which is in general

$$(x_{new},y_{new}) = (\frac{x_{old}*l_{new}}{l_{old}},\frac{y_{old}*b_{new}}{b_{old}})$$ where $l$ is the length of the image, $b$ is the width.

In times of fractional answers after reduction make sure your bounding box covers a bigger area. For example, Y-coordinate for the left side of bounding box should be pushed towards the Y-axis while on the right should be pushed towards the ceil value like say you got $25.5$ by reducing, go to 25 when on the left side while $26$ when on right side of bounding box. Similarly up and down for $x's$.

Hope this helps.


Why does my LSTM perform better when randomizing training subset vs. standard batch training? I am training a simple LSTM network using Keras to predict time series values. It is a simple 2-layer LSTM.

I get the best performance when I train on subsets of the training set that start at random points. Each subset has a training size of 100 samples and a validation size of 30 samples. At each sample the model has a batch size of 16, trains for 100 epochs with an early stop after 20 epochs of little improvement. I run this training 10 times.

How is this different from me training me simply training my model as such:

model.compile(loss='mean_squared_error', optimizer='adam')

results = model.fit(X_train, y_train, epochs=100, batch_size=32, shuffle=False,
                validation_split=0.3, verbose = 1, steps_per_epoch=10,
callbacks = [EarlyStopping(monitor='val_loss', min_delta=5e-5, patience=20, verbose=1)])

Doesn't Keras automatically select random subsets of the training data for training and validation through the mini batches? Is it simply because I am training the model so many more times?

Is this effectively a k-fold-esque training implementation?
However, using sk-learn's KFold or Repeated Kfold doesn't get results as good.

NB: I do not want to use shuffle as this is time series data and that would distort the model training.

Couple of points:

1) When you 'run this training ten times' does that mean you completely re-initialize the model and train from scratch?

2) I'm not entirely sure that you're loading the data correctly from the code you've supplied. What are the shapes of X_train and y_train?

3) It may depend on your data and your selection process. Time-series data is typically temporally correlated -- the closing value of the Dow Jones on Thursday is more likely to be close to the values on Wednesday and Friday than on Monday or in two weeks. It may be that if you sample 100 data points in a row and they're temporally correlated, you're giving your model an easier problem, because those values are more like each other than values elsewhere in the series.

Good luck!


How to create a dummy model in Tensorflow I am a newbie in Machine learning.

I found this example using tflearn somewhere. It is the part of the program where we initialize a dummy model before training it. In the example, there were 4 input layers and 2 output layers. I didn't understand any terms but I ran that program and it worked.

def neural_network_model(input_size):
    network = input_data(shape=[None, input_size, 1], name='input')

    network = fully_connected(network, 128, activation='relu')
    network = dropout(network, 0.8)

    network = fully_connected(network, 256, activation='relu')
    network = dropout(network, 0.8)

    network = fully_connected(network, 512, activation='relu')
    network = dropout(network, 0.8)

    network = fully_connected(network, 256, activation='relu')
    network = dropout(network, 0.8)

    network = fully_connected(network, 128, activation='relu')
    network = dropout(network, 0.8)

    network = fully_connected(network, 2, activation='softmax')
    network = regression(network, optimizer='adam', learning_rate=LR, loss='categorical_crossentropy', name='targets')
    model = tflearn.DNN(network, tensorboard_dir='log')

    return model

I wanted to reuse this function to create a model that I will train with a data having 24 input layers and 4 output layers but it gave me the following error:

  Cannot feed value of shape (64, 4) for Tensor 'targets/Y:0', which has
  shape '(?, 2)'

Can someone explain me what does all of the term above mean ? And what should I change to adapt with my 24 by 4 layer?

The reason is that in your model, you have specified the input size to vary, as the parameter of your function specifies. You have to change your code as follows:

def neural_network_model(input_size, output_size):
network = input_data(shape=[None, input_size, 1], name='input')

network = fully_connected(network, 128, activation='relu')
network = dropout(network, 0.8)

network = fully_connected(network, 256, activation='relu')
network = dropout(network, 0.8)

network = fully_connected(network, 512, activation='relu')
network = dropout(network, 0.8)

network = fully_connected(network, 256, activation='relu')
network = dropout(network, 0.8)

network = fully_connected(network, 128, activation='relu')
network = dropout(network, 0.8)

network = fully_connected(network, output_size, activation='softmax')
network = regression(network, optimizer='adam', learning_rate=LR, loss='categorical_crossentropy', name='targets')
model = tflearn.DNN(network, tensorboard_dir='log')

return model

You have hard coded the output size, which is two, you have to change it to be a variable and during calling, you have to specify the size of your new output, which is four.


K-Means clustering - What to do if a cluster has 0 elements? I'm writing code for k-means clustering. I have around 100000 vectors of size 128x1 (SIFT descriptors). I'm trying different initialization methods such as Forgy and Random Partition.
What if suppose, no vectors are classified to a cluster (in one of the iterations)? How to calculate centroid for that cluster?

If you get an empty cluster, it has no center of mass. You can simply ignore this cluster (set k=k-1 for next iteration), or repeat the k-means run from a new initialization. You can also choose to place a random data point into that cluster and carry on with the algorithm if you must have this specific number of K clusters.

If it keeps happening, there is a good chance that your K parameter is too high.
You should iterate over a few different values for k and pick the best.

Here is how an empty cluster might happen:

Consider the following example with 7 data points. The different shapes of the points are just to show that there are actually 2 natural clusters (but we do not know that before running the algorithm). In this case we choose k = 3.

Three random cluster centers are initialized. At the end of first iteration points 3, 1, 2, and 7 will be in one cluster. 4 and 5 will be in another cluster. And 6 will be in the last cluster. Note here that the distance between 3 and 4 is larger than the distance between 4 and 5 and so 4 is assigned to the cluster represented by 5. Before we begin the second iteration we update the cluster centers and the following picture shows the centers and the clusters at the end of first step.

Now, the cluster center for the red cluster moved closer to point 4 due to 1, 2, and 7. And the cluster center for the blue cluster moved away from 5 due to point 4. In the next iteration point 4 will decide that it is closer to the red cluster and point 5 will decide that it is closer to the green cluster. This will cause blue cluster to be empty as shown below.

The images were taken from the following link: K-Means Empty Cluster Example


Why is backpropagation used for finding the loss gradient? I am relatively new to the world of machine learning. After getting a general idea of the concept, I tried creating a program for training a deep learning network from scratch. My goal was to use as little outside resources/help as possible and figure out every step and issue on my own. Now that I'm done, I'm comparing the differences between my approach and the solutions that were created by people who know what they're doing. One of these differences I've seen is the backpropagation algorithm, and I'm in the process of trying understanding it (but am struggling a bit).
While researching the algorithm, most sites describe it as a method of calculating the loss gradient of the network. However, the method that I used for calculating the loss gradient involves considerably less math and complexity - which I don't fully understand - and seems to accomplish the same thing:
$$Loss(w_0,w_1,w_2,...,b_0,b_1,b_2,...)$$
$$\nabla Loss(...)=\begin{bmatrix}{\frac{\partial Loss(...)}{\partial w_0}},{\frac{\partial Loss(...)}{\partial w_1}},...\end{bmatrix}$$
To figure out the gradient of this loss function, you can just tweak each parameter in the loss function and find out how much the loss changes. Dividing by how much you change it, you can get an approximation of the loss gradient at the current network state. Here is a rough outline of what I mean in code:
function LossGradient():
    current_loss = CalculateLoss()
    gradient_vector = []
    i = 0
    delta = 0.00001 # some really small value
    for weight in network:
        weight += delta # tweak weight by delta ( a tiny amount )
        new_loss = CalculateLoss() # calculate the loss after tweaking
        gradient_vector[i] = (new_loss - old_loss) / delta # store ∂Loss/∂W_i in gradient vector
        i += 1

    # do the same for biases

    return gradient_vector

My question is, what's the advantage of the backpropagation over this simple method that I used? I'm in the middle of struggling to understand the algorithm fully, and no resource seems to explain why and how exactly the algorithm is more efficient.
Any insight would be much appreciated.

If you know that $f(x) = wx + b$, then you also know that $\frac{\partial f}{\partial b} = 1$.  The function LossGradient does not use that information, and would estimate it by evaluating the function $f$. Then it computes the loss. In general, it computes $f$ and $Loss$ once for every learnable parameter.
Backpropagation uses knowledge about the function being optimized to be more efficient. This means first evaluating $f$ once, then evaluating $Loss$ once, then updating all of your parameters. This efficiency may not matter to you when you have a small number of parameters, but it will make a huge difference if $f$ has hundreds of millions, or hundreds of billions of parameters.
It's not that an approach like yours is never useful (see derivative-free optimization), it's just that if you have extra information, using it is helpful.


What is representation in optical character recognition? I am learning OCR and reading this book

The authors define 8 processes to implement OCR that follow one by one (2 after 1, 3 after 2 etc):

Optical scanning
Location segmentation
Pre-processing
Segmentation
Representation
Feature extraction
Recognition
Post-processing

This is what they write about representation (#5)

  The fifth OCR component is representation. The image representation
  plays one of the most important roles in any recognition system. In
  the simplest case, gray level or binary images are fed to a
  recognizer. However, in most of the recognition systems in order to
  avoid extra complexity and to increase the accuracy of the algorithms,
  a more compact and characteristic representation is required. For this
  purpose, a set of features is extracted for each class that helps
  distinguish it from other classes while remaining invariant to
  characteristic differences within the class.The character image
  representation methods are generally categorized into three major
  groups: (a) global transformation and series expansion (b) statistical
  representation and (c) geometrical and topological representation.

This is what they write about feature extraction (#6)

  The sixth OCR component is feature extraction. The objective of
  feature extraction is to capture essential characteristics of symbols.
  Feature extraction is accepted as one of the most difficult problems
  of pattern recognition. The most straight forward way of describing
  character is by actual raster image. Another approach is to extract
  certain features that characterize symbols but leaves the unimportant
  attributes. The techniques for extraction of such features are divided
  into three groups’ viz. (a) distribution of points (b) transformations
  and series expansions and (c) structural analysis.

I am totally confused. I don't understand what is representation. As I understand after segmentation we must take from image some features, for example topological structure like Freeman chain code and must match to some saved at the learning stage model - i.e. to do recognition. By other words - segmentation - feature extraction - recognition. I don't understand what must be done on representation stage. Please, explain.

The representation step is before the feature extraction step for exactly the reasons that they state.

If you take the full image representation and go right to feature extraction, you have much more data to extract features from (added complexity) and the features you do extract will be much more noisy. This is why one at least reduces the representation to grey level (as the authors suggest).

Feature extraction is choosing properties of the image which you will then perform recognition on. It is important to already have thrown out unimportant details of the image before doing this. That is why the authors, and people in general, reduce to a proper representation before performing feature extraction.


Can RL learn to mimic a hash function? Theoretically, is this possible: When you have a given input string, there are a set of permutations and bit operations to do.

The problem is, when you choose an approach like reinforcement learning, the model could do these kind of actions again and again , and could only reach its destination by chance.

When there is an upper limit of steps (timesteps), you cannot tell how far the model is from reaching its goal because you just don't know what to do next. The reward is always 0 when the goal has not been reached and 1 when it has been reached. Thus, since the reward is basically always 0, the algorithm cannot learn.

Any ideas as to why it is so?

Has there been put any thought into it?

Probably not at a formal research level, it is basically a dead-end, as you have pointed out there will essentially be one correct solution amongst a large number of incorrect ones. No useful heuristic exists to assess partial solutions, so logically RL will not work, as an agent needs to experience (or simulate accurately) success and failure in order to learn correctly. This is a hard limit to all learning algorithms, they can only learn from available data - any combinatorial problem with one special permutation, and zero clues that disambiguate it from a large number of similar permutations is inherently unlearnable.

Only approaches that would work against a generic hash algorithm are brute-force searches that try every possible combination of operations suspected of being in the algorithm, up to a certain size of program. For any real-world secure hashing algorithm, this would be a huge search space, and intractable. For smaller insecure hashing algorithms, it is more feasible, but there would likely be no value to even demonstrating it was possible.

Related to this (when trying to reverse a hash or find collisions, not figure out the algorithm from examples), cryptographic hashes, such as MD5, SHA-1, SHA-256 etc, are designed such that brute-forcing them is the only way to solve them. In the case of MD5 and SHA-1, that has since been broken mathematically, and they are weaker then brute-forcing, but this was done starting with knowledge of the algorithm. Current RL agents and neural networks cannot match a human in learning and understanding abstract mathematics, so that approach is ruled out at least for now.


TypeError: Cannot cast array data from dtype('float64') to dtype('S32') according to the rule 'safe' I'm getting the above error while I'm extracting the sensor data from serial port and giving it for prediction.Tried reshape and string functions but every trial throws back me same error.i'm using mlp neuralnetwork in my model.Thanks in advance

Spent some time with error and found that serial data consists a newline character"\n" in it and it caused error pitch data is a float and due added character roll value is a string.So, changed code as below a little and it worked


Why $L2$ loss is strictly convex if number of samples $N$ is larger than input dimension $d$? I am using $L2$ loss in my linear regression problem and I have to prove that my $L2$ loss is strictly convex if number of samples $N$ is larger than input dimension $d$.

I think, if I can prove that my hessian matrix, $ X^T X $ for $L2$ loss is positive definite then that will do. Here, $X$ is the feature matrix with  $N$ samples as rows and  $d$ columns.

Also, I understand $v^T X^T X v$ $\ge 0$,

as,  $v^T X^T X v$ = $(Xv)^T. Xv$ = $|| X.v ||^2_2 $ (Which is the euclidean norm of $X.v$) i.e hessian matrix is positive semi-definite.

But I don't understand how to prove $ X^T X $ is positive definite or $L2$ loss is strictly convex.

-please ignore the constants in $L2$ loss.

I think an additional assumption of linear independent inputs is necessary to have a strictly convex loss function.
Linear independence with $N>d$ gives you $Xv \neq 0 ~ \forall v \neq 0$, which implies that $X^\top X$ is positive definite, as $||Xv||_2 > 0 $.


Logistic regression prediction changed after executed couple of time I noticed that after each time I execute the following lines of code, my results are different. Any idea why?
I think  that the main issue here is this line of code
But I dont understand why?

test_data['prediction']=sentiment_model.predict_proba(test_matrix)[:,0]

What is the best way to add new column with .predict_prob()?

test_data['prediction'] = sentiment_model.predict_proba(test_matrix)[:,0]
test_data['prediction_label'] = sentiment_model.predict(test_matrix)
test_data['prediction'] = test_data['prediction'].apply(lambda x: round(x,2))
test_data.sort_values(by='prediction', ascending=False, inplace=True)
test_data[test_data['name'] == 'Britax Decathlon Convertible Car Seat, Tiffany']
test_data.head()

There is some randomness in the results from selecting/shuffling data that is used in the model.

If you don't want that, you could set a fixed random_state (seed) in your model.


How to remove irrelevant text data from a large dataset I am working on a ML project where data were coming from a social media, and the topic about the data should be depression under Covid-19. However, when I read some of the data retrieved, I noticed that even though the text (around 1-5 %) mentioned some covid-related keywords, the context of those texts are not actually about the pandemic, they are telling a life story (from 5-year-old to 27-year-old) instead of how covid affects their lives.
The data I want to use and am looking for is some texts that tell people how covid makes depression worse and what not.
Is there a general way to clean those irrelevant data whose contexts are not covid-related (or outliers)?
Or is it ok to keep them in the dataset since they only count for 1-5% ?

You can use BERT to create vectors that will capture the context of the whole tweet. Once, you do that, try clustering (K-Means or GMM). You can then look at the clusters found and separate out this unwanted data.


Comparing ML models to baselines When comparing ML models with baseline or "dummy" models, are there best practices for building and comparing baselines?
I'm doing a binary classification task where 40% of the samples are class_0 (untreated class), and the other 60% are class_1 (treated/positive class).
I have two baselines: baseline_0 predicts randomly, and baseline_1 predicts class_1 every time.
Because the metrics are calculated relative to class_1, when baseline_1 predicts class_1 on every sample, it ends up with perfect recall (1.0), which inflates the f1 score. Does this mean this baseline model is not good for comparing to my experimental models and I should use baseline_0 instead, or that baseline_1 is good but that the f1 score is not good for making these comparisons?

This depends on what you want to show.
When working with metrics you shouldn't just take the value as is, but see what each metric are telling you. baseline_1 isn't better/worse than baseline_0 because it has a higher/lower value in metric X. Both baselines give an interesting perspective on a given dataset and if unsure I'd suggest keeping both.
A couple of notes:

when saying baseline, I will refer to the two baseline strategies that you said in your post
I will use the accuracy metric for examples but what I'm saying is true for any metric.

Why use baselines?
People usually tend to see accuracy (or other measures) as absolute values. E.g. accuracy=0.9? "very good", accuracy=0.3? "very bad". This isn't true however, as metrics are influenced by the number of classes and the proportion of samples between them.
However an accuracy of 0.3 in a classification task with 1000 classes is arguably much harder to achieve than an accuracy of 0.9 on a binary classification task (assuming class balance in both cases).
Here is were baselines come in. They can show how much better a model is than a dump classification strategy.
How baselines help?
Baselines help by putting a lower bound on your metrics. For example an accuracy of 0.55 on a binary classification task is slightly better than random, but the same accuracy on a 10-class setting is much better. Baselines help quantify that and tell you how much better you are than predicting random  or the most common values.
What effect do baselines have?
Now on to why keep both baselines:

The first baseline (i.e. random) helps show how metrics can be influenced by the number of classes has on the dataset.
The second baseline (i.e. most common) helps show how metrics can be influenced by class imbalance.

How baselines actually help?
Let's you have two models, one with an accuracy of 0.92 and another with an accuracy of 0.93. How much better is the second model to the first? This depends on the value of your baseline. If you have a baseline accuracy of 0.5 then both models are relatively strong and the difference is not that significant. If you have a baseline of 0.9 then the models aren't as strong and an improvement of that magnitude is more significant.