Unable to Use The K-Fold Validation Sklearn Python I have an dataset.
I am unable to use the K-Fold Validation. I am getting the error raised:

ValueError("{0} is not supported".format(y_type))
ValueError: continuous is not supported .

I do not want to do encoding to int, since it may affect the data, and also I want to understand why K-fold is not working.
Below is my python code.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.ensemble import RandomForestRegressor
from sklearn import cross_validation, metrics
from sklearn.cross_validation import train_test_split
from sklearn import model_selection
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.naive_bayes import GaussianNB

from sklearn import svm
from sklearn import preprocessing

 - `List item`

url = "https://archive.ics.uci.edu/ml/machine-learning-databases/00242/ENB2012_data.xlsx"
names=['Relative Compactness','Surface Area','Wall Area','Roof Area','Overall Height','Orientation','Glazing Area','Glazing Area Distribution','Heating Load','Cooling Load']
df = pd.read_excel(url,names=names)

#Feature selection
train=df.sample(frac=0.8,random_state=150)
test=df.drop(train.index)

#save the original values in a dataframe so we can compare later
test_loads=test[["Cooling Load"]]

#Create 2 lists of response values to train our model
Y1=np.array(train['Heating Load'])
Y2=np.array(train['Cooling Load'])

#Select the features
train_corr=train[['Overall Height','Relative Compactness','Roof Area','Surface Area']]
test_corr=test[['Overall Height','Relative Compactness','Roof Area','Surface Area']]
seed = 7
scoring = 'accuracy'
X_train,X_test,y_train,y_test=cross_validation.train_test_split(train_corr,Y1,test_size=0.2)
kfold = model_selection.KFold(n_splits=10, random_state=seed)
cv_results = model_selection.cross_val_score(RandomForestRegressor(), X_train, y_train, cv=kfold, scoring=scoring)
print (cv_results.mean())

Based on the answer here, Since you are doing a classification task, you should be using the metric R-squared (co-effecient of determination) instead of accuracy score (accuracy score is used for classification purposes). You should use something like score for evaluation because your task is regression.


100% accuracy on both train and test after feature engineering The original dataset is of ~17K compound structures almost equally divided with labels indicating yes or no, after heavy use of mol2vec and rdkit I have created ~300 datapoints
Using the boosted trees method on the same shuffled train and test dataset gives 98% train accuracy and 89% test accuracy, but a simple neural network gives 100% train and test accuracy
I have checked the code again to ensure target leakage is not occurring, I have also coded it from the scratch twice to ensure I haven't made any mistake, yet I do not believe I should be getting 100% accuracy on both train and test
Does this mean that the model is actually accurate due to so many data points?

Yes - Getting 100% accuracy is possible for neural networks compared to tree-based models. Neural networks can learn non-linear relationships through the activation function. Tree-based models are restricted to piece-wise linear relationships.


How to plot two columns of single DataFrame on Y axis I have two data frames (Action, Comedy). Action contains two columns (year, rating) ratings columns contains average rating with respect to year. The Comedy data frame contains the same two columns with different mean values.
I merged both data frames into a total_year data frame.
The output of total_year:

Now, I would like to plot total_year on a line graph in which the X axis should contain the year column and the Y axis should contain both the action and the comedy columns.
I can plot only 1 column at a time on Y axis using following code:
total_year[-15:].plot(x='year', y='action' ,figsize=(10,5), grid=True)

How can I plot both columns on the Y axis?
I took this photo from google just to let you know guys I want to draw graph in this way:

Feeding your column names into the y values argument as a list works for me like so:
total_year[-15:].plot(x='year', y=['action', 'comedy'], figsize=(10,5), grid=True)

Using something like the answer at this link is better and gives you way more control over the labels and whatnot:
adding lines with plt.plot()


How to have Multiple labels in a single video? I am building a Tennis stroke classification system using CNN.
I assume each stroke contains 3 steps/classes ('Ready', 'Impact', 'Finish'). I want to train a model which will predict whether the input video contains these steps/classes in it.
I have tried training 3 models for each step as binary classification.
Example of one step model classes:
1 - ready
0 - not-ready(other incorrect steps).

But this method failed since there are more features in 'not-ready' class. I got only 4% accuracy.
Can anyone help me to find a solution for this problem.

Given that you have only 3 classes and that they closely depend on each other, I think it's worth trying a multiclass setting as WBM said. The idea is to label each video using the full combination of actions, since the maximum number of combinations is 2^3 = 8:

R-I-F
R-I
R-F
R
I-F
I
F
none

Probably some combinations of actions are impossible, so the number of classes is likely less than 8. Why this is a reasonable approach:

The setup is exactly the same, i.e. you can use the same labels and the predictions can be used the same way as in your multi-label approach
This is a "joint model", i.e. a model which learns everything together and therefore can exploit fine-grained distinction between classes (e.g. between R-I-F and R-I).

However note that this kind of method may require more data, in particular it needs to have enough instances for each class.


Trying to analyze market data in R, but companies continually show very low average growth over large time intervals I think the title explains it all.
In more detail, I'm using the data set from this Kaggle page.
The data set doesn't come with a daily change in percent, so I copied the dataset and made a new column for the daily percentage change, which I calculate by $$\Delta daily = \frac{Opening Price-Closing Price}{Opening Price}.$$
I went to do a basic sanity check, and make sure that nothing is going wrong with my column for daily percentage change.
I first picked a few dates and companies and calculated the $\Delta daily$ by hand, then compared it to the value my code gives; everything matched up there. However, I then tried to look at the mean $\Delta daily$ for a few stocks that have been wildly successful over the last few years (Facebook, netflix, amazon, etc). When I take the average $\Delta daily$, a lot of these companies are coming up with an average daily percentage change that is very near zero, or even slightly negative. Worth noting, I tried taking the average growth over several different time intervals; I'd even restrict myself to stretches of time where a stock had very few dips, but still the average $\Delta daily$ for the restricted time windows. For example AMZN was selling at roughly 300\$/share in january of 2015 and roughly 700\$/share in january of 2016, but when I look at $\Delta daily$ for this period of time, I get $-.0003913$. More than double the growth over the year, but an average daily change of $-.4$%? That seems way off.
I began wondering if there was something wrong with the dataset, but then I chose a few of the tech giants and plotted their growth against time over the last few years, and the graph is a near perfect resemblance of the growth chart that I see elsewhere, so the data seems to be fine.
I'm wondering if anyone has any thoughts on what's going on here. Is my intuition just wrong? Is it possible to have an average daily return that's near zero or negative, but still very strong growth in the long run? Is there something about the way I'm building my percent.delta vector that's incorrect?
There's really not much code to share, but here's what I have, in case it's useful. Sorry if it's a little sloppy; I was just messing around with some stuff and so didn't make it as tidy as I would for a more formal project.
Thanks in advance for any thoughts.
allstocks<-read.csv(file='all_stocks_5yr.csv')
allstocks2<-read.csv(file='all_stocks_5yr.csv') #copy data to alter
percent_delta <-as.vector(numeric(619040)) #no. observations; vector that will contain daily %change

for (i in 1:619040){
  percent_delta[i]<-(allstocks2$open[i]-allstocks2$close[i])/allstocks2$open[i]
} #fill vector with daily %change

allstocks2$percent.delta<-percent_delta

#just a check that trends in R match trends seen elsewhere. Can use smth other than NFLX
plot(subset(allstocks2, Name=="NFLX")$date, subset(allstocks2, Name=="NFLX")$close)

#avg percent.delta for, e.g., AMZN between, e.g., jan2015-jan2016
mean(subset(subset(allstocks2,Name=="AMZN"),as.Date(date)>=as.Date("2015-01-23")
& as.Date(date)<=as.Date("2016-01-04"))$percent.delta)
#avg percent_delta for, e.g., AMZN between, e.g., jan2015-jan2016

This simply has to do with the fact that returns are compounding, i.e. the daily percentage changes are cumulative. If a stock gets has a daily percentage change of 1% for a single year (trading for 252 days), and therefore also an average change of 1% the total return over this year will not be equal to 1%. The actual return is equal to $(1 + 0.01) ^{252} - 1 \approx 11.274$, meaning around 1127.4%, which is much larger than the average daily change of 1%. To get the correct total return over a specific period you have to multiply the percentage change for each period, which in your case is daily.


Use trained non linear regression model to identify variables that maximize the predicted value I trained a non-linear regression model with 23 features. I tried to make sure the model doesn't overfit with ~0.6 r squared on validation data and with 0.75 correlation coefficient between the actual and model-predicted target values.
I would like to know if doing the following would make sense.

Use the model to identify values of the 23 features that would maximize the predicted value:

For example, consider a model to predict house price using: avg_area_price and sqfeet then identify value of avg_are_price and sqfeet that would result in the maximum house price. Note that we may not have observed this in the data but the regression model captured the input feature to output relationship so it could identify this.
I tried to look into it and found that linear programming could help doing this for linear regression. https://stats.stackexchange.com/questions/475639/can-we-use-linear-regression-to-define-the-objective-function-in-linear-programm
I did not find anything for finding the feature values that leads to the maximum predicted value for non-linear regression model.
Can anyone help with how to do this when we have non-linear regression model?

For non linear regression are you using SVM Kernel approach or deep learning?  One of the articles I have referred to in the past is this research paper from computational biology where in they have defined a measure for feature importance. I had come across this in the past when preparing for an interview so didnt actually try it out and see if it works.The link to the article :- [https://arxiv.org/pdf/1611.07567.pdf] . Let me know incase you do practically try this out


Getting lower performance metrics when using GridSearchCV I have defined an XGBoost model and would like to tune some of its hyperparameters.

I am using GridSearchCV to find the best params. However, I also tried to fit the model on the entire training dataset, and I have noticed that the 'roc_auc' performance metric is higher than when I used the Grid Search. I was surprised, because I was expecting Grid Search to perform better. I think I am missing the intuition here.

My understanding was that for grid search cross-validation, for say k folds,  given a parameter value from the param_grid, gridsearchcv fits the model on the folds separately and calculates the desired performance metric. Later, for that particular parameter, it takes the 'average' of all the folds' calculated 'roc_auc'. The gridsearch repeats this process for all the other given parameters in the params_grid.
Finally, the '.best_params_' is the one for which the calculated metric is higher.

This is what I tried:

param_test = {'max_depth':[3,5,6,7,9]}
model = XGBClassifier(learning_rate=0.3,
                     n_estimators=16,
                     max_depth=6,
                     min_child_weight=1,
                     gamma=0,
                     subsample=1,
                     colsample_bytree=1,
                     objective='binary:logistic',
                     nthread=4,
                     scale_pos_weight=1,
                     random_state=27)
gsearch = GridSearchCV(estimator=model, param_grid = param_test, scoring='roc_auc', cv=5)
gsearch.fit(X_train, y_train)
print('Best found params: {}'.format(gsearch.best_params_))
print('Best (Train) AUC Score: {:.4f}%'.format(gsearch.best_score_*100))

This prints:

Best found params: {'max_depth': 6}
Best (Train) AUC Score: 87.2186%

Now, when I use the same model and fit it on the entire training dataset, this is what I get:

model = XGBClassifier(learning_rate=0.3,
                     n_estimators=16,
                     max_depth=6,
                     min_child_weight=1,
                     gamma=0,
                     subsample=1,
                     colsample_bytree=1,
                     objective='binary:logistic',
                     nthread=4,
                     scale_pos_weight=1,
                     random_state=27)

model.fit(X_train, y_train, eval_metric='auc')

# Predict training set:
y_pred_train = model.predict(X_train)
y_pred_proba_train = model.predict_proba(X_train)[:,1]

# Print model report:
auc_score_train = roc_auc_score(y_train, y_pred_proba_train)

print("AUC Score (Train): {:.4f}%".format(auc_score_train*100))

which prints:

AUC Score (Train): 97.0311%

Why is there such a discrepancy? What am I missing here?

Your confusion seems to stem from this line:

print('Best (Train) AUC Score: {:.4f}%'.format(gsearch.best_score_*100))

The best_score_ is not exactly a training score (nor is it an unbiased estimate of future performance*): as you say, it's the averaged score across different fold splits, but each of the scores that get averaged are the performance of the models on their test fold.  So, this score reflects performance of models on unseen data.

But, when you compute the score of the model retrained on the entire training set, that model has seen the training data, and the score there is inflated (quite a lot) as you'd expect.

* This is discussed at length elsewhere, but in short, while the scores are based on performance on data unseen by the models, you have looked at that data when selecting the "best" model, so to use that score now would be optimistically biased.


R data frame create a new variable which corresponds to one of the existing one Currently this is what I get. "community_area_clean" is the new variable that I added by extracting a variable that lists all the communities from a data frame. "CommArea_Name" is the original "unclean" variable. But I find that the names in "community_area_clean" do not correspond to the names in "CommArea_Name". How can I fix it? Thanks.

If my assumptions about CommArea_Num are correct, then:

data$community_area_clean = data$community_area_clean[data$CommAreaNum]

should remap that column. Note that the AREA_KM fields will remain in the order currently shown.


What is the best way to match street addresses in a dataset in Python? I have a dataset of land parcels owned by the government and I am attempting to match street addresses to an existing list of government agencies. I've used fuzzy matching and used a regex that ignores casing and distinctions between direction (for example North and N are treated the same).

However, the program ends up having a very poor matching rate, as a lot of the addresses are not getting matched. What are some other ways I should try to improve the matching rate?

You can try using this to help:
https://github.com/openvenues/libpostal

libpostal looks like it can normalize across various geographic styles with the expand addresses functions.


How to select a model based on statistical analyses? I've been statistically validating the performance of different Deep Learning models in classifying parasitized and normal cells. In the process, I could find that there is no statistically significant difference in performance between the pre-trained CNN models. In the absence of such significance, which model am I supposed to choose?

This is a very general question, since you are not providing any statistics, that one can comment on. But in general: yes, this can be the case. However, there is an interesting paper on comparing classification models with statistical tests written by Demšar (2006). He shows a way to vizualize the results of different models that do not significantly differ in quality.


What's the best way to use binned data in a tree-based model? I have some numeric data that has come 'binned', but the bins are not of equal sizes in terms of scale or quantile

For example, an age variable that is [0-16), [16-21), [21-30), [30-45), [45-65), [65, ]

If I leave it as a categorical variable, a tree will treat each category separately and discount the ordered relationship between the factors.

If I change it to an ordinal variable, e.g. [0, 1, 2, 3, 4, 5] and keep an array of labels for later reporting, the tree might split on, e.g. <2.5, which seems more natural to me, but then again the distance between 1 and 2 is not the same as the distance between 2 and 3 by any measure.

I'm leaning towards the second solution, but I would love some input!

For a complete decision tree, either of your proposed models would be able to represent the same set of concepts since a decision tree can be decomposed into a disjunction of conjunctions. I don't think the fact that the bins have different sizes is really an issue because placing any significance on the exact location of splits occurring "inside" a bin is probably not a good idea anyway. From a performance perspective, what probably matters most is how your data are distributed (i.e., does your concept split the data between a single pair of adjacent bins or is it more complicated).

Since you have the "random-forest" tag associated with your question, I'm inclined to agree with you that keeping them ordered is probably better because if you are generating many trees, the ordered representation will likely result in more terse/smaller trees.


Which metrics for evaluating a recommender system with implicit data? I am currently in the process of creating a recommender system. This recommender system works with a neural network and then searches for the closest neighbors and thus gives recommendations for a user.
The data is implicit. I only have in the data which products a user has bought.On the basis of this data, I create the recommendations.

What are the best metrics to evaluate this recommender system with implicit data?

Can I evaluate the model and then the search algorithm? If so, are the metrics different? Which metrics do I have to use for what?

There are two major types of evaluation - online and offline.
Online evaluation means showing the model's predictions to actual users. Since the goal of a recommender system to sell more products, the best overall best metric for a recommender system is increasing sales to actual users. This is best done by putting the model in production and A/B test if the model increase sales. This approach is not always possible given the limited resources (time or access to a production system).
Offline evaluation means simulating online evaluation by holding out existing data to evaluate the model.
If possible, split the data based on time. Train the model on the earlier data. Test the model on the later data. For a given pairing of product and user, the model will predict buy or not buy (binary classifier). The model can be evaluated as any binary classifier. For given domain, precision or recall may be more important.
However, this is not often times possible because time data many not be tracked or product-user pairs maybe sparse. If time is not tracked, the data may be split randomly to simulate time. If product-user pairs are sparse, product-user pairs are clustered along latent factors.


Does it makes sense to apply feature scaling on timestamp I was wondering if it makes sense to apply normal standardization on a feature like timestamp ?
The data that I process are network packets.

Thank you

For time series analysis.  Yes

But, turning data into a computable object for using in the ML computation?  No

Using the data as a feature.  Then, Yes

I would give you a general time series example:

Consider the number of days in months. The irregularity causes friction while analyzing the model.

Consider this:

So, such type of transformations would be helpful which analyzing time series', which can help reduce friction in the models sensibly.
Link of the explanation


How to build parse tree with BNF I need to build parse tree for some source code (on Python or any program language that describe by CFG).

So, I have source code on some programming language and BNF this language.

Can anybody give some advice how can I build parse tree in this case?
Preferably, with tools for Python.

I suggest you use ANTLR, which is a very powerful parser generator. It has a good GUI for entering your BNF. It has a Python target capability.


Extract the "path" of a data point through a decision tree in sklearn I'm working with decision trees in python's scikit learn. Unlike many use cases for this, I'm not so much interested in the accuracy of the classifier at this point so much as I am extracting the specific path a data point takes through the tree when I call .predict() on it. Has anyone done this before? I'd like to build a data frame containing ($X_{i}$, path$_{i}$) pairs for use in a down-stream analysis.

Looks like this is easier to do in R, using the rpart library in combination with the partykit library. I'd ideally like to find a way to do this in python, but here's the code, for anyone who is interested (taken from here):

pathpred <- function(object, ...){
    ## coerce to "party" object if necessary
    if(!inherits(object, "party")) object <- as.party(object)

    ## get standard predictions (response/prob) and collect in data frame
    rval <- data.frame(response = predict(object, type = "response", ...))
    rval$prob <- predict(object, type = "prob", ...)

    ## get rules for each node
    rls <- partykit:::.list.rules.party(object)

    ## get predicted node and select corresponding rule
    rval$rule <- rls[as.character(predict(object, type = "node", ...))]

    return(rval)
}

Illustration using the iris data and rpart():

library("rpart")
library("partykit")
rp <- rpart(Species ~ ., data = iris)
rp_pred <- pathpred(rp)
rp_pred[c(1, 51, 101), ]

Yielding,

       response prob.setosa prob.versicolor prob.virginica
 1       setosa  1.00000000      0.00000000     0.00000000
 51  versicolor  0.00000000      0.90740741     0.09259259
 101  virginica  0.00000000      0.02173913     0.97826087
                                           rule
 1                          Petal.Length < 2.45
 51   Petal.Length >= 2.45 & Petal.Width < 1.75
 101 Petal.Length >= 2.45 & Petal.Width >= 1.75

Which looks to be something I could at least use to derive shared parent node information.


How to automatically disable register_hook when training model is in eval() phase in PyTorch? I require to update grads of an intermediate tensor variable using the register_hook method. Since the variable isn't a leaf-variable, I require to add the retain_grad() method to it after which, I can use the register_hook method to alter the grads.
score.retain_grad()
h = score.register_hook(lambda grad: grad * torch.FloatTensor(...))

This works perfectly fine during the training (model.train()) phase. However, it gives an error during the evaluation phase (model.eval()).
The error:
File "/home/envs/darthvader/lib/python3.6/site-packages/torch/tensor.py", line 198, in register_hook
    raise RuntimeError("cannot register a hook on a tensor that "
RuntimeError: cannot register a hook on a tensor that doesn't require gradient

How could the model automatically disable the register_hook method when it in eval() phase?

Removing score.retain_grad() and guarding register_hook with if condition (if score.requires_grad) does the trick.
if score.requires_grad:
    h = score.register_hook(lambda grad: grad * torch.FloatTensor(...))

Originally answered by Alban D here.


Best neural network architecture for object detection without location I want to build a model which is able to detect the presence of a specific object in the image without caring about the location (i.e. no bounding boxes, just 0/1 output). More specifically, the object is always the same and it is characterized by a specific pattern (e.g. a national flag or a bar code). The algorithm should be robust to scale, rotation and distortion.
What is the best architecture in this case? Is there something better than a generic CNN binary classifier?

It mostly depends on the amount of data you have available for training.

However, unless you have an extremely limited dataset and you are unable to generate synthetic data, CNNs are currently the best approach for solving your task.  With sufficient data, they are very robust.  To improve their performance, you can augment your dataset by randomly rotating, flipping, brightening, contrasting, scaling, cropping, blurring, and adding noise to the images throughout training.

It also depends on your speed requirements and computational resources.  Traditional methods have the potential to be quite fast, but if you only care about accuracy, a deep network would probably perform better.  Note that some CNNs are also quite fast, such as MobileNets.

There are a few reasons you might go "old school" and use template matching and more traditional approaches as suggested by @TheDjentleman:

You do not have enough data to make a CNN robust AND you cannot generate synthetic data
CNNs do not meet your speed requirements AND the traditional methods do
You do not have sufficient resources to train a deep network, such as GPU(s).  FYI Google developed a cloud computing tool for researchers and developers at a pretty reasonable cost.

If none of those criteria fit your situation, go with a CNN.

Lastly, you do not necessarily need a generic binary classification CNN.  You can use an object detection network or even a semantic segmentation network and just ignore the location data if you do not need it.


Differences and similarities between nonnegative PCA and nonnegative matrix factorization I have seen references in the literature to nonnegative principal component analysis (nPCA) and nonnegative matrix factorization (NMF). I have tried reading the papers on both of them but it is not clear to me what the differences and similarities between them are. By similarity, I mean I am also interested in knowing when the nPCA and NMF method will give the same solution. Can someone clarify this?

I came across this paper by Asteris et. al. entitled "Orthogonal NMF through Subspace Exploration". You can google it as links posted here might break in the future.

The basic idea boils down to if we can construct non-negative principal components of the matrix of interest, we can use these principal components to do an orthogonal NMF. Since the matrix of interest is non-negative (this is what NMF assumes) and the non-negative principal components are non-negative, then by construction the transformed matrix is also non-negative. The paper details how exactly the construction can be achieved and guarantees convergence.

To answer my own question above: There are many ways to do NMF. In the case where the transforming vectors are orthogonal, i.e. Orthogonal NMF, it is identical to Non-negative PCA. Details can be found in the paper cited above.


How to use bigrams for a text of sentences? I have a text which has many sentences. How can I use nltk.ngrams to process it?

This is my code:

   sequence = nltk.tokenize.word_tokenize(raw)
   bigram = ngrams(sequence,2)
   freq_dist = nltk.FreqDist(bigram)
   prob_dist = nltk.MLEProbDist(freq_dist)
   number_of_bigrams = freq_dist.N()

However, the above code supposes that all sentences are one sequence. But, sentences are separated, and I guess the last word of one sentence is unrelated to the start word of another sentence. How can I create a bigram for such a text? I need also prob_dist and number_of_bigrams which are based on the `freq_dist.

There are similar questions like this https://stackoverflow.com/questions/12821201/what-are-ngram-counts-and-how-to-implement-using-nltk?rq=1 but they are mostly about a sequence of words.

One way is to loop through a list of sentences. Process each one
  sentence separately and collect the results

Brian is totally right about the solution but I think that's actually the part that is missing from his answer :) Brian's code assumes that the sentences have already been segmented, which was Ahmad's original question as far as I can tell.

With that in mind, here's an updated proposal for adding the sentence splitting to the script:

import itertools, nltk

def ngrams_wrapper(sent):
    return list(nltk.ngrams(sent, 2))

raw = "To Sherlock Holmes she is always the woman." + \
      "I have seldom heard him mention her under any other name."

# Given a string as input (`raw`), split it into sentences.
# Returns an object of type 'list: str'
sentences = nltk.sent_tokenize(raw)
# ['To Sherlock Holmes she is always the woman.',
# 'I have seldom heard him mention her under any other name.']

# Next, tokenize every sentence (string) in the list of sentences. The tokenizer takes
# strings as input so we need to apply it on each element of `sentences` (we can't apply
# it on the list itself). For that, we can use the function `map`, which applies any
# callable Python object to every element of a list.
# The output of this step will be an object of type
#   'list: list: str'
# such that
#   'sentences: sentence: word'
tokenized = map(nltk.tokenize.word_tokenize, sentences)
# [['To', 'Sherlock', 'Holmes', 'she', 'is', 'always', 'the', 'woman', '.'],
# ['I', 'have', 'seldom', 'heard', 'him', 'mention', 'her', 'under', 'any', 'other',
# 'name', '.']]

# We now generate the ngrams for each tokenized sentence in the list of sentences. Since
# we want to generate the ngrams for each sentence specifically, we again must apply the
# function to each element of the list separately, not the whole list. So, we use the
# `map` function again.
# The output of this step will be an object of type:
#   'list: list: tuple: <str, str>'
# such that
#   'sentences: sentence: ngram: <word_1, word_2>
bigrams = map(ngrams_wrapper, tokenized)
# [[('To', 'Sherlock'), ('Sherlock', 'Holmes'), ('Holmes', 'she'), ('she', 'is'), ('is',
# 'always'), ('always', 'the'), ('the', 'woman'), ('woman', '.')], [('I', 'have'),
# ('have', 'seldom'), ('seldom', 'heard'), ('heard', 'him'), ('him', 'mention'),
# ('mention', 'her'), ('her', 'under'), ('under', 'any'), ('any', 'other'), ('other',
# 'name'), ('name', '.')]]

# Finally, since we want all the bigrams in a single list to work with them more easily,
# we flatten the list, that is, we concatenate all the elements at the first level of
# depth in the list. So, if the list was an object of type
#   'list: list: tuple: <str, str>'  (sentences: sentence: ngram)
# we will transform it into:
#   'list: tuple: <str, str>'        (sentences: ngram)
bigram = list(itertools.chain.from_iterable(bigrams))
# [('To', 'Sherlock'), ('Sherlock', 'Holmes'), ('Holmes', 'she'), ('she', 'is'), ('is',
# 'always'), ('always', 'the'), ('the', 'woman'), ('woman', '.'), ('I', 'have'), ('have',
# 'seldom'), ('seldom', 'heard'), ('heard', 'him'), ('him', 'mention'), ('mention',
# 'her'), ('her', 'under'), ('under', 'any'), ('any', 'other'), ('other', 'name'),
# ('name', '.')]

freq_dist = nltk.FreqDist(bigram)
# [(('always', 'the'), 1), (('woman', '.'), 1), (('Holmes', 'she'), 1), (('seldom',
# 'heard'), 1), (('Sherlock', 'Holmes'), 1), (('him', 'mention'), 1), (('I', 'have'), 1),
# (('any', 'other'), 1), (('under', 'any'), 1), (('the', 'woman'), 1), (('her', 'under'),
# 1), (('other', 'name'), 1), (('To', 'Sherlock'), 1), (('name', '.'), 1), (('she', 'is'),
# 1), (('heard', 'him'), 1), (('mention', 'her'), 1), (('have', 'seldom'), 1), (('is',
# 'always'), 1)]

prob_dist = nltk.MLEProbDist(freq_dist)
number_of_bigrams = freq_dist.N()


Bioinformatics widget Why is there are no Volcano-plot or MA-plot in bioinformatics widget?

Volcano-plot and MA-plot are graphical methods. MA-plot of what? Everything in your screenshot is bioinformatics analysis. The plots are part of the analysis but not the name of the analysis itself.


How to verify a CNN encoder works as expected? I am using CNN as a part of kernel warping. The purpose here is to reduce input dimension (from N*M to K *1). The input data is not image data. I suspected that the CNN network might not work as I expected. How can I verify my assumption? I am using pytorch, by the way.

Thank you in advance.

In order to verify the encoder efficiency than it is better to design some down stream task like classification, similarity , etc. Based on that u can validate your encoder. U may refer this link for downstream Task.


Google OR-Tools - Routing - Penalties - Adding Different Penalty to Different Location (Python) I am using Google's OR-Tools for route optimisation. References can be found here.

I am performing an optimisation where certain pick-up locations are dropped based on a penalty at each location. The documentation gives an example and states: "In this context, a disjunction is simply a variable that the solver uses to decide whether to include a given location in the solution. In this example, the method adds the same penalty to each location, but in general you can add different penalties to different locations.", and provides the following chunk of code as an example of adding a constant penalty to each location:

penalty = 1000
for node in range(1, len(data['distance_matrix'])):
    routing.AddDisjunction([manager.NodeToIndex(node)], penalty)

where data is a dict containing a n x n distance matrix (distance_matrix). (This code runs and returns a sensible outcome).

Now, I would like to vary the penalty at each location, and I've used the following code to do so:

for node in range(1, len(data['distance_matrix'])):
    routing.AddDisjunction([manager.NodeToIndex(node)], data['penalty'][manager.NodeToIndex(node)])

where penalty is an array of length n containing penalties for each location. (In these examples, the depot is assumed to be at the first location and is penalty-free; range starts at 1).

When I run my code, I get the following error message:

TypeError: Wrong number or type of arguments for overloaded function 'RoutingModel_AddDisjunction'.
Possible C/C++ prototypes are:
operations_research::RoutingModel::AddDisjunction(std::vector< int64 > const &,int64,int64)
operations_research::RoutingModel::AddDisjunction(std::vector< int64 > const &,int64)
operations_research::RoutingModel::AddDisjunction(std::vector< int64 > const &)

Question: What am I doing wrong? How do I go about adding different penalties to different locations? Could someone please translate the C/C++ into Python?

Given your code, it looks like data['penalty'][manager.NodeToIndex(node)] is not an integer, probably a float, and therefore it doesn't match the function signature, as the error message suggests. Try converting it to an integer.


is there a way to check if i got a "good price" on something? I'm looking at some data. Actually, the Boston Housing dataset is probably a good proxy for it: https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html

I'm wondering if there's a way to predict if I got a "good price" given certain conditions. So something like, if I'm given a tuple such as CRIM, ZN, INDUS = (0.006320,18, 2.31), then is a house price of 50 significantly higher or lower than expected?

This isn't quite vanilla anomaly detection, because the combination of a particular CRIM, ZN, INDUS may be rare and I don't want to flag it unless price is out-of-spec for that. I guess the question is given a combination of (CRIM, ZN, INDUS) (no matter how anomalous that is), is the combination of (CRIM, ZN, INDUS, Price) rare? So, it's like a naiive bayes' problem, but that only works with categorical outputs

Yes, fit a regression equation ( or deep NN)  to predict price based on all other parameters. then calculate error ( actual- predicted) , positive error indicates higher price than expected( what is should be) and vice versa. get a basic idea of types of anomalies-

https://machinelearningstories.blogspot.com/2018/07/anomaly-detection-anomaly-detection-by.html


Recognizing numerical entities I'm trying to perform classification on a large dataset with mixed numerical and categorical features. The dataset is all semi-structured text, so everything is a String. Does anyone know of a library I could use to automatically identify numeric features (e.g. "width (in.)", which corresponds to the width in inches)? This is important because numeric and categorical features are encoded/handled differently in my analytics pipeline. I think this is basically a named entity recognition problem wherein the entity is a numeric variable or feature that exists in the data set as a String/text.

There are many good libraries for identifying number-like values, but identifying corresponding fieldnames is trickier and likely very problem-specific.

A purely data driven approach might look for co-occurrences with numbers, for example:

if [*number-like* capacity] or [capacity: *number-like*] occurs in > x% of

the instances where "capacity" occurs,
    then "capacity" can be guessed to be a label of number-like things.

Even the nature of these relationships (eg, whether it should be before, after, capitalised) could be learnt from existing known labels.

This can be done with Google BigQuery using the public tri-gram sample data.  But I assume that's out of scope and impractical here.

A practical approach, might aim to classify these strings as numerical_units, categorical_units or unsure or even more classes, and then do human review of the latter.  (There are some very tricky cases, for example, "capacity" is numerical, but "capacity type" would not be.)

As a starting list you can use:
wiktionary:Category:en:Units_of_measure
wiktionary:Category:Symbols_for_SI_units
wiktionary:Category:en:Mathematics
Note that they are in the singular.  For your domain considering finding other such lists.

If a string is a full match with one of those labels, you can consider it a numerical_unit.
Your next concern is tolerant, fuzzy matching.  You could treat '.' as a wildcard (so "in*" matches "inch"), or find the actual abbreviations of units like "inch".  These you can label as unsure and then review.  Likewise if the word is simply contained in the string, eg "arch length" contains "length".

I think once you have done this you can make some refinements and add some special cases.  Without seeing even a sample of your data, it is difficult to say more.  If most of your strings are numerical units, then it may be easier to identify categorical units instead.


Python Convert indexes starting from 1 to 365 into months names and display them into the x axis without changing the plot structure? I need to display months on x axis of a plot instead of the indexes number of a data frame that goes from 1 to 365, which they represent the number of day of the year. So instead of an x-axis which goes from 1 to 365, I want to display it as "Jan", "Feb" and so on, without losing the structure of the plot.
Here is the main structure of my data frame:
    Month   Day Max_Data    Min_Data    MonthDay
1   1       1   1.1        -13.3        1-1
2   1       2   3.9        -12.2        1-2
3   1       3   3.9        -6.7         1-3
4   1       4   4.4        -8.8         1-4
5   1      5    2.8        -15.5        1-5

I am currently plotting using:
plt.scatter(data_2015.index, data_2015['Max_Data'], marker='^', color='green',s=40, alpha=1.0)

And if I changed data_2015.index to Month the graph will plot a different a completely wrong values, as they 28, 30 or 31 rows for each month.
So what is the way to convert indexes into month and display them into the x axis of a plot?

You will have to tweak the xaxis locator
e.g. For your sample data -

Month,Day,Max_Data,Min_Data,MonthDay
1,1,1.1,-13.3,1-1
1,2,3.9,-12.2,1-2
1,3,3.9,-6.7,1-3
1,4,4.4,-8.8,1-4
1,52.8,-15.5,1-5

import pandas as pd, numpy as np, seaborn as sns, matplotlib.pyplot as plt

data_2015 = pd.read_csv("/content/tst.csv")
plt.scatter(data_2015.index, data_2015['Max_Data'], marker='^', color='green',s=40, alpha=1.0,)

import matplotlib.ticker as mticker
plt.gca().xaxis.set_major_locator(mticker.FixedLocator(np.arange(len(data_2015.index))))
plt.gca().xaxis.set_major_formatter(mticker.FixedFormatter(np.array(['Jan','Feb','Mar','Jan','Feb'])))

Output


Removing duplicates and keeping the last entry in pandas I have a table and I'm trying to remove all the duplicate and keep the
the rows that has the latest datestamp.

Here is the table:

email address       orgin_date   new_opt_in_date   datestamp
123@ax.tu            1/1/1900     1/1/1900          3/15/2016
123@ax.tu            1/1/1900     1/1/1900          3/15/2016
iron_man@metrix.com  2/15/2015    3/5/2015          6/6/2017
iron_man@metrix.com  2/15/2015    3/5/2015          7/6/2018
sleep@dort.st        2/15/2015    3/5/201           7/6/2018
sleep@dort.st        2/15/2015    3/5/201           5/6/2018

I'm trying to keep only the data that has the recent datestamp and the
output will like this:

 email address       orgin_date   new_opt_in_date   datestamp
 123@ax.tu            1/1/1900     1/1/1900          3/15/2016
 iron_man@metrix.com  2/15/2015    3/5/2015          6/6/2017
 sleep@dort.st        2/15/2015    3/5/201           7/6/2018

I use this formula:

df.drop_duplicates(keep = False)

or this one:

df1 = df.drop_duplicates(subset
['emailaddress', 'orgin_date', 'new_opt_in_date','datestamp'],keep='first')
print(df1)

but nothing works

You can see from the documentation of the method that you can change the keep argument to be "last".

In your case, as you only want to consider the values in one of your columns (datestamp), you must specify this in the subset argument. You had tried passing all column names, which is actually the default behaviour. Now we can use this (along with the correct value for the keep argument) to get this:

For example, a dataframe with duplicates:

In [1]: import pandas as pd

In [2]: df = pd.DataFrame({'datestamp': ['A0', 'A0', 'A2', 'A2'],
                           'B': ['B0', 'B1', 'B2', 'B3'],
                           'C': ['B0', 'B1', 'B2', 'B3'],
                           'D': ['D0', 'D1', 'D2', 'D3']},
                           index=[0, 1, 2, 3]).T

In [3]: df
Out[3]:
  datestamp   B   C   D
0        A0  B0  B0  D0
1        A0  B1  B1  D1
2        A2  B2  B2  D2
3        A2  B3  B3  D3

Now we drop duplicates, passing the correct arguments:

In [4]: df.drop_duplicates(subset="datestamp", keep="last")
Out[4]:
  datestamp   B   C   D
1        A0  B1  B1  D1
3        A2  B3  B3  D3

By comparing the values across rows 0-to-1 as well as 2-to-3, you can see that only the last values within the datestamp column were kept.


How is training data generated Very new to Machine learning process.

I am curious as to how researchers, companies, academics get training data for their research.

Do they pay some users to train data. If yes, how do they guarantee accuracy of it.

Do graduate students (on PhD track) do it for their professor. (This is meant to be a joke)

'Training' data is really just splitting data you have already collected into test or training sets. For example, if you want to build a classifier for handwritten numbers, you collect thousands of samples of handwritten numbers like the MNIST database. When you think you have enough data to build a model, you then split it into train and test sets (usually by randomly assigning individual samples to one group or another at a specific ratio).

I think where your confusion lies is in the idea of collecting a 'training' set first as if it's truly independent from the test set. When collecting handwritten numbers, the researchers did not say, well, we have 10,000 samples, let's build a model with 10,000 samples and then have it running for our future data sets that we have not collected yet - in fact, that strategy is particularly bad and can lead to overfitting.

What you would do is take those 10,000 samples and split them - say 7,000 for training to build a model, 3,000 for testing said model - and maybe you would randomly build up many 7,000/3,000 models and take the average of the parameters for those models that you built. Then you can say our model predicts our test set with an accuracy of 97%, we think it will work well on data we have not yet collected.

How you collect that initial data set is specific to the process you are trying to understand. Maybe it's clicks on a website, images from a satellite, or electrical recordings from an ensemble of neurons. Sometimes you pay money to collect data - like a census or survey or maybe even buying another company that collected a bunch of user data that you want - typically, data collection is an inherent process to what you are doing and you are using statistical methods to make models and inferences about your population of interest.


Attribute selection from dataset I work on a dataset with numeric values.The class labels has also numeric values.I made the 6 numeric class labels into one witch contain values like e.g happy_pleased. I want to insert the new .arff file into weka but i have a problem with the class @attribute as i declare it as nominal. Witch is the right type to declare the class label? I tried nominal but nothing happened

You need to list the set of nominal values for the class in the arff file, comma-separated and enclosed in curly brackets {}. Here's an example:

@RELATION iris

@ATTRIBUTE sepallength  REAL
@ATTRIBUTE sepalwidth   REAL
@ATTRIBUTE petallength  REAL
@ATTRIBUTE petalwidth   REAL
@ATTRIBUTE class    {Iris-setosa,Iris-versicolor,Iris-virginica}

In your case, Iris-setosa,Iris-versicolor,Iris-virginica would be replaced with happy_pleased,...

If this still doesn't work, you can tell WEKA which attribute is the class in the explorer by choosing the correct attribute in the drop-down list above the histogram in the bottom right corner.


Replacing words by numbers in multiple columns of a data frame in R I want to replace the values in a data set (sample in the picture) using numbers instead of words, e.g., 1 instead of D, -1 instead of R, 0 for all other values. How can I do it with a loop? I know it can be done doing this instead:

(suppose d is index name)

d[d$Response == "R",]$Response = -1

d[d$Response == "D",]$Response = 1

... (other values code it and assign value of) = 0

I guess this is off-topic but here's a reproducible code:

#Preparing mocking data
>set.seed(123) #for reproducibility
>Party <- sample(c('D', 'N','R'), 10, replace = TRUE) #sample random values
>d <- data.frame(Party) #put values into a dataframe

# Here's how you substitute the values in just one line. This operation is vectorised.
>d$Response <- ifelse(d$Party=='R', -1, ifelse(d$Party=='D', 1, 0))

Basically in new Response column, you check if R and assign -1, if not you check if D and assign 1; otherwise you assign 0.

> d
   Party Response
1      D        1
2      R       -1
3      N        0
4      R       -1
5      R       -1
6      D        1
7      N        0
8      R       -1
9      N        0
10     N        0


Search the Number of occurrences of the particular words in data using Pandas. I wanna Search how many times "bad"  and "good" words are repeated in the data frame and visualize  this with histogram.

Jurgy's answer should work. However, based on comments above, you can simply do like this,

df['yourcolumn_name_good_bad'].value_counts().plot(kind='bar')

Otherwise, if you want to use seaborn library and create a plot follow this.


Why is scale parameter on batch normalization not needed on ReLU? I now read a book titled "Hands-On Machine Learning with Scikit-Learn and TensorFlow" and on the chapter 11, the author writes the following explanation on batch normalization:

  Note that by default batch_norm() only centers, normalizes, and shifts the inputs; it does not scale them (i.e., γ is fixed to 1). This makes sense for layers with no activation function or with the ReLU activation function, since the next layer’s weights can take care of scaling, but for any other activation function, you should add "scale": True to bn_params.

The batch_norm function is from TensorFlow, FYI. The author explains that the γ parameter should not be set on ReLU activation function.

However, I don't understand why on ReLU, the next layer's weights can take care of scaling... I understand that the next layer takes input from the ReLU output, which is max(0, x). Why does it take care of scaling and thus no need to set γ parameter on ReLU?

Scaling in batch normalization is said not to be required because of the nature of ReLU. Although it is a non-linear operation, a multiplication with a positive scalar $\gamma$ over the values of a vector $x$ which are already normalized and centered to the trained bias $\beta$ makes no influence on its non-linear outcome: for $\gamma > 0$, $max(0, \gamma \times x) = \gamma \times max(0, x)$. In fact, the extra parameter adds no representative power in the model, for the same reason that we must include non-linearities between other neuron layers.

And so, the weights of the following layer would scale the values to fit accordingly. We could have $z = Wgx$, but that would be the same thing as $z = Wx$ if $W$ is scaled by $g$. The same reasoning can't be made for other non-linearities, such as the sigmoid or tanh, which have a saturating effect that is clearly influenced by a scalar.

Curiously, the official documentation on TensorFlow's tf.contrib.layers.batch_norm makes the same recommendation:

  scale: If True, multiply by gamma. If False, gamma is not used. When the next layer is linear (also e.g. nn.relu), this can be disabled since the scaling can be done by the next layer.


Can you use two different datasets as train and test sets with countVectorizer and test_train_split? So I managed to run my code on a combination of train data and validation data, but now I need to create a text file that contains the predictions for the test data and I just don't understand how. Is there any way to make the X_train work with train_data and X_test with test_data? I think that would solve my problem but I can't find how or if it is even possible.

train_data = np.genfromtxt('train_samples.txt', delimiter = '\t', dtype = None, encoding = 'utf-8', names = ('id', 'text'),
                               comments = None)

    train_labels = np.genfromtxt('train_labels.txt', delimiter='\t', dtype = None, names = ('id', 'label'))

    test_data = np.genfromtxt('test_samples.txt', delimiter = '\t', dtype = None, encoding = 'utf-8', names = ('id', 'text'),
                              comments = None)

    validation_data = np.genfromtxt('validation_samples.txt', delimiter='\t', dtype = None, encoding='utf-8',
                                    names = ('id', 'text'), comments = None)
    validation_labels = np.genfromtxt('validation_labels.txt', delimiter = '\t', dtype = None, names = ('id', 'label'))

    for x in range(len(train_data)):
        train_data[x][0] = train_labels[x][1]

    for x in range(len(validation_data)):
        validation_data[x][0] = validation_labels[x][1]

    train_data_text = np.append(train_data['text'], validation_data['text'])
    train_data_labels = np.append(train_data['id'], validation_data['id'])

    # show shape of training data
    cv = CountVectorizer()
    word_count_vector = cv.fit_transform(train_data_text)
    print(word_count_vector.shape)

    # train_data = np.concatenate((train_data, validation_data))
    X = cv.fit_transform(train_data_text).toarray()
    y = pd.get_dummies(train_data_labels)
    y = y.iloc[:, 1].values

    # Train Test Split
    from sklearn.model_selection import train_test_split

    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=0)

    # Training model using Naive bayes classifier
    from sklearn.naive_bayes import MultinomialNB

    results = MultinomialNB().fit(X_train, y_train)

    y_pred = results.predict(X_test)
    print(y_pred)

    from sklearn.metrics import accuracy_score

    # Evaluate accuracy
    print(accuracy_score(y_test, y_pred))

    classifier.fit(X_train, y_train)
    y_pred = classifier.predict(X_test)

You're doing a big mistake in your code, which is applying the vectoriser before the train/test splitting. The vectoriser should be fit only on the training dataset, then the learned counts should be applied to the test set. Instead you applied the vectoriser to the whole data which you then splitter into train and test.

# THIS IS OK
# train_data = np.concatenate((train_data, validation_data))
X = cv.fit_transform(train_data_text).toarray()
y = pd.get_dummies(train_data_labels)
y = y.iloc[:, 1].values

# NOT OK
# Train Test Split
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=0)

# CORRECT WAY
X_train = X  # already good as it is
y_train = y  # also good

# I suggest to use pandas to read the txt files
X_test = cv.transform(test_data['text']) # <-- APPLY VECTORIZER TO TEST DATA USING TRANSFORM ONLY

Then you can procede as you did in the rest of your code.


Understanding why in deep reinforcement learning correlations in the data reduce the effectiveness From the paper

  Human-level control through deep reinforcement
  learning, Mnih et al. Nature 2015

It says

Reinforcement learning is known to be unstable or even to diverge
when a nonlinear function approximator such as a neural network is
used to represent the action-value (also known as Q) function 20 .
This instability has several causes: the correlations present in the
sequence of observations

I am unsure how to understand this and cannot create any hypothetical examples where this can happen. What are some hypothetical scenarios, or real examples where the correlations present in the sequence disrupt the use of a 'deep learning' approximator is used?

One reason can be the high correlation between training data can cause bias the deep learner. In this way, if the learner gets a new data which is not correlated with the training data, it will have large error.


Time dependent Classification problem I am trying to solve a decision making problem. In it, information evolves and increases with time for each event observed, and the history of the event may be useful.

The problem is as follows: in an electric network at time $t_0$ there is signal that automatically triggers the possible need of a technical intervention in a network to correct the problem. The system at $t_0$ has a large set of general indicators describing the state of the network based on which I'd like to determine whether to proceed with an intervention at $t_{0}$.

It's possible to carry specific measurements at the point where the problem was detected (this takes some time $\Delta t$). At this stage we can again ask whether the intervention is necessary or not? We can iterate this procedure a few times. Then, my question is, which is the machine learning method best suited for this classification problem? Can it be formulated as a Survival problem?

From the description of the problem, you can just choose whatever classification algorithm you like to classify the need of intervention at  $t_{0}$.

The general rule for choosing a classifier for this is to start simple--e.g., by using nearest neighbor and iterate to more powerful classifiers until you get enough accuracy.

If additional measurements at $t_{0}$ can be made a classifier can be used to determine the necessity of intervention without this additional data, then based on the probability output of this classifier and a threshold the additional measurements can be triggered. Based on this measurements another probability can be output which hopefully uses more informative features.

If you care about $\Delta t$, you should think about wether there is a real time dependency. Does the need of intervention depend upon the series of classifications? Is the order of determining interventions as necessary important for you?

In my opinion, you do not have to care about ordering for this application.
Rather, I would suggest pooling classifications for one instance according to the most frequent label or something like that.


Is it acceptable not to transform() test data after train data is being fit_transform()-ed We know that the best practice in data preprocessing (such as standardization, Normalization, ... etc) is that while we perform fit_trasform() on the training data, we apply transform() testing data so that the learned parameters from scaling the train data are applied on testing data. Similar to this:

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform (X_test)

The question is: Does it also make sense to perform fit_transform() on the training data but NOT transform() testing data at all so that we get to test the model performance on actual real-world data that are not transformed at all?

Thank you

No, it does not make sense to do this.

You model has learned how to map one input space to another, that is to say it is itself function approximation, and will likely not know what to for the unseen data.

By not performing the same scaling on the test data, you are introducing systematic errors in the model. This was pointed out in the comments by nanoman - see that comment for a simple transformation example.

To exaggerate the point, imagine you have a language model, translating text from English to French. You apply a single transformation on your English data: you first translate it to Spanish. Now the model is trained to translate Spanish to French, with some accuracy. Now you move on to the  test data - still in English - and you do not apply the same transformation as you did to your training data. You are asking the model to translate directly from English to French, instead of Spanish to French, and it is obvious that the results won't be good.

In principal, this idea is the same as with any other model and transformations, just that the impact might not always be so visible i.e. you might get really lucky and not notice a large impact.

The language model might have learnt some elementary linguistics common to all three languages (e.g. overlapping vocabulary or sentence structuring), but we cannot expect the model to perform well, translating English to French.

Practical Note

You should only compute the transformation statistics (e.g. mean and variance for normalisation) only on training data and use these values to then transform the training data itself, and then the same values to transform the test data.

Including the test dataset in the transform computation will allow information to flow from the test data to the train data and therefore to the model that learns from it, thus allowing the model to cheat (introducing a bias).

Also, it is important not to confuse transformations with augmentations. Some "transformations" might be used to synthetically create more training data, but don't have to be used at test time. For example, in computer vision, deleting regions of an image. Test time augmentation is something you could read about.

Extra discussion

More complicated models (ones with many many more parameters) might be able to perform some kind of interpolation, especially if your dataset if N-dimensions with a large N (e.g. > 10).

This has recently been seen with extremely large models, such as Open AI's GPT-3, which has 175 BILLION parameters, and is therefore even able to perform quite well on completely different tasks, let alone the given problem in the training set range.


Creating a feature by combining 2 features with different units? If I want to create a new feature by combining 2 features with the same unit, I can just subtract them (e.g. buy volume vs sell volume). If the units are different, it doesn't make sense to subtract them (e.g. trade volume vs orderbook size). I can divide them if one of the features is guaranteed to never be 0.

If either features can be 0, what's the best way to compare them? I don't want to use an arbitrary high number to represent infinity if the denominator is 0.

You can combine multiple features only if they are on same scale.

Combining feature A with range of possible values (100-1000) with feature B with range of possible values (1-10) will result in feature biased towards A.

You will need to standardize them to the same scale.

For every value X of A = (X - smallest(A))/(biggest(A) - smallest(A))
For every value Y of B = (Y - smallest(B))/(biggest(B) - smallest(B))

Now all values of A and B resides in the range of (0,1). You can add based on the weight of every attributes. If both are equally important then C = (0.5 * A) + (0.5 * B) and if A should have 70% of weight then C = (0.7 * A) + (0.3 * B)


What knoweldge framework supports this type of query? I have knowledge that is organized as in the example below, where items or nodes can belong to multiple hierarchies and can have arbitrary numbers of children and parents:

sports
    golf
    baseball
    tennis
equipment
    golf
        clubs
            club1
    baseball
        bats
            bat1
    tennis
        racket
            racket1

I need to be able to run queries like "Show me all sports that use clubs or rackets, and their associated equipment."  The queries do not need to be posed in natural language, but the results must be displayed in a hierarchy:

sports
    golf
        club1
    tennis
        racket1

I'm going crazy trying to figure out what I need.  It doesn't seem to quite be a relational database, an ontology, or a taxonomy.  Or maybe I don't quite understand how to use one or more of those to do what I need.  What standard approach is most similar to what I'm trying to do?

As always with stackexchange questions, I'm utterly terrified of asking a question that doesn't fit the exact requirements of a particular forum.  I tried to make it as specific and answerable as possible.  I BEG YOU PLEASE, do not delete or lock this question without at least explaining why it is unacceptable or offering a suggestion for making it acceptable.  Thank you!

Well, to answer my own question, it turns out the issues I was struggling with are almost exactly the issues that motivated the development of and that are addressed by graph databases.  I would have been out of luck if this was 1999, but it turns out 2016 is a good time to be alive for databases that go beyond the traditional relational model!  I guess all this NoSQL stuff that the kids are always talking about is more than just hype.  :)


Different definitions of Macro F1 score, which one is used in Scikit-learn? In this article Macro F1 and Macro F1 two different definitions of the F1 used in the literature are demonstrated.
The first F1 score is computed such as:

  F1 scores are computed for each class and then averaged via
  arithmetic mean

The second such as:

  The harmonic mean is computed over the arithmetic means of precision
  and recall

I was wondering which definition is actually implemented in Scikit-learn.
From the docs I cannot derive which definition is used:

  Calculate metrics for each label, and find their unweighted mean. This
  does not take label imbalance into account.

The first variant is implemented:
$$F1_{macro}= \ \sum_{classes} \frac{F1\text{ }of \text{ }class}{number\text{ }of\text{ }classes}$$

You can find an example calculation in this answer.

Sometimes the scikit learn documentation does not include all the details. In these cases it is often helpful to look into the source code which is linked on all help-sites. Here you can find some more details on the f1 score calculation.


How to generalize comments using NLP I have list of log comments in CSV file. I want to cluster those log comments using K-Means and after that I want convert each cluster comments into general form.
for eg. I have bunch of comments in one cluster which starts from "Reservation number failed......" and I want to convert those comments into particular comment like "Reservation failure".

I can achieve this by giving specific name to each cluster after seeing each cluster. But I don't want like this. I want to create intelligent model which automatically create generalized comments for me.

I would not like to assign name to each cluster. Basically I am done with clustering part. that is, I have lets say 3 clusters as below

cluster 0 : list of comments like "Reservation number failed......",
total comments: 15
cluster 1 : list of comments like "Request timeout failed due to
......", total comments:9
cluster 2 : list of comments like "Dinning reservation successfully
completed...", total comments: 5

I want to build model that intelligently assign name to each cluster by its contents. for eg .

cluster 0 will get name as "Reservation failure"
cluster 1 will get name as "Request timeout failure"
cluster 2 will get name as "Dinning reservation successful"

if after training more data with some different comments. it should create another cluster and assign the name as per content.

Basically, we want to cluster similar comments and assign a name/entity to it. I suggest you to use Doc2Vec to convert the comments to fixed-sized vectors. Each of your comments will then be a n-dimensional vector. Comments with similar words/phrases will lie in the close proximity of one another.

  Now, using K-Means Clustering, we can form clusters of vectors that
  represent comments having a similar meaning. Once the clusters are
  formed, assign a name to each of them.

For a given sample ( comment ), the model will first transform the given comment into a vector. The model will then check for a cluster that is nearest to the given sample's vector. The output will be the name of the nearest cluster.


RNN time-series predictions with multiple features containing non-numeric features and numeric features? The question RNN's with multiple features is ambiguous and not explicitly in differentiating different features. I want to understand how to use RNN to predict time-series with multiple features containing non-numeric data as well. As a deep learning model, I assume I don't need to quantify the non-numeric elements.

Suppose sales data

Sales | Weather | Holiday | Temperature
100   | Windy   | Yes     | 3
2000  | Sunny   | Yes     | 20
200   | Sunny   | No      | 30
-5    | Stormy  | No      | 3
-50   | Cold    | No      | -50
500   | Cold    | Yes     | -20

where I want to predict the Sales column with the other columns. I have found demos such as here about using RNN with numeric data, without enriching the data with non-numeric data.

So

How can I predict time series with multiple features contain numeric features and non-numeric features?

Other interesting questions related to RNN

Multivariate time-series RNN -question in SO with numeric data
Time series prediction using ARIMA vs LSTM

Possible ways to expose these categorical variables as part of a time-step to be fed to an RNN include producing One_Hot Tensors, Hash Table Representations, or Embedding Layer Outputs from the categorical fields and concatenating all of the feature tensors at each time-step to feed to your RNN.

If, instead of discrete categories, you're looking to interpret the strings that make up the fields as sequences themselves, you could encode them numerically at a word- or character level and pass them through a subsidiary RNN to produce the internal representation at each timestep. This would be appropriate only for cases where you have a very large variety among strings in those fields.

Apologies for the Tensorflow links if you're not using that framework.


Python - Converting 3D numpy array to 2D I have a 3D matrix like this:

array([[[ 0,  1],
      [ 2,  3]],

      [[ 4,  5],
      [ 6,  7]],

      [[ 8,  9],
      [10, 11]],

      [[12, 13],
      [14, 15]]])

and would like to stack them in a grid format, ending up with:

 array([[ 0,  1],
        [ 2,  3],
        [ 4,  5],
        [ 6,  7],
        [ 8,  9],
        [10, 11],
        [12, 13],
        [14, 15]])

Currently, I'm using numpy as a library.

I suggest you to visit this link
also for this case would work np.reshape((-1,2))


Orange 3 "Find Informative Projections" and target variables? Having issues with the "find informative projections" feature in Orange 3. In order to be able to use this feature in the scatterplots, you have to select a target variable in the "select columns". So I choose the value that I'm most interested in finding correlations to. However, whatever I choose as a target variable doesn't show up in the informative projections list as being correlated to other features in my data set. Why not?

I have a lot of features in my data and was under the impression that the "find informative projections" option finds pairs of features that seem to have a linear correlation and sorts them by which pairs are most correlated.

If that's the case, why do you need a target variable? How do I choose an appropriate target variable?

Am I misunderstanding the purpose of the "find informative projections" feature?

I'm attaching a pic of my work flow.

Slightly related with the question (issues with Find Informative Projections), I found this post trying to solve this issue: the tool Find Informative Projections was disabled and Orange did not provide any hint about why.

In the end the issue was on my side: I forgot to set any variable as Target when loading the data so it was useless to select it in the Scatter Plot points-colors section.

Double click on the File to load the data where you can choose if a variable is categorical or numerical, then double click on the variable I wanted to use as Target (that was marked as Feature) in order to select it as Target and the Find Informative Projections was enabled again.

PS: I have also noted that when making predictions, the file loaded should have all variables marked as Features. If you mark as before one of the variables as target, Orange shows a red circle with an exclamation mark on the Predictions widget, and a message: "Data does not have the same target as predictors" (even though it is). I guess that the target in the prediction file is automatically set by checking which info is missing in the file.


How does one deploy a model, after building it in Python or Matlab? I have been playing around with a lot of different machine learning models (clustering, neural nets, etc...), but I am sort of stuck on understanding what happens after you finish building the model in Python or Matlab.

For example, let's say that I trained a basic neural network model for a binary classification problem.  How does one deploy that to colleagues, for example, so that they can load in the dataset to spit out a prediction?  I have trained a model, but what happens to that model, now?  How do I "save" that model that I just trained in Python?

I see a lot of tutorials on how to pre-process data, train the model, spit out the statistics and predictions; but, what comes next?

Obviously Facebook, Google, and anyone else heavily involved in machine learning / AI applications are creating a framework to use their models.  But, is there software that allows you to pull in data and then apply it to your Python code?  Is this what Weka, TensorFlow, and these other packages do?

what comes after training a model is persisting (in layman terms, saving) a model, if you want to save the parameters for later deployment.

All companies that do serious machine learning spend many hours (sometimes even days or weeks) training the models. Obviously, when you need rapid predictions in a deployment mode, you use a pre-trained model to generate said prediction.

In python, for example, there are a number of way to persist models. The most common libraries used are pickle and cpickle (i/o much faster I believe as the core is in C).

Here is a link to model persistence in sklearn:

http://scikit-learn.org/stable/modules/model_persistence.html


How to detect anomalous points As is clear from the figure, the blue points, which don't follow the trend, are anomalous points.

I'm wondering about the best non-parametric method to detect those points. I have tested some outlier detection methods such as standard deviation, etc. but they don't provide good results while it is clear from the figure.

In your example what differentiates the clusters is not the raw value but rapid departure from previous points.  I might look into change-point detection. Nonparametic, but still requires some fiddling with tuning parameters.


Workaround for word embeddings that do not "see" antonyms Most word embeddings do not "see" antonyms. For instance, among many words they will place vectors for "dependent" and "independent" (as an example) quite close, - actually as close as with synonyms such as "independent" and "autonomous". So it is easy to identify synonyms as close vectors, but how to identify antonyms, or generally work with antonyms? There are some few rare papers that try to develop embedding algorithms "aware" of antonyms (just web-search word-embedding antonyms). But I am working with standard very powerful and already trained on massive data embedding libraries. Is there a workaround to somehow work with STANDARD embeddings but to make them not "blind" to antonyms? Thanks!

You could probably use the technique to combine vectors with antonyms with some kind of weights. Below thread suggests adding POS tagger with word2vec vectors, instead of POS you can concatenate antonyms:

https://stats.stackexchange.com/questions/238016/deep-learning-word-embedding-with-parts-of-speech

https://stackoverflow.com/questions/51537441/how-to-combine-both-word-embeddings-and-pos-embedding-together-to-build-the-clas

You can identify Antonyms using below:

https://www.geeksforgeeks.org/get-synonymsantonyms-nltk-wordnet-python/


How to Precondition A Fully Convolutional Neural Network Lets say I am creating a fully convolutional classifier for identifying different types of animals. Now lets assume that I know what part of the animal kingdom the animal belongs to beforehand (mammal, reptile, amphibian .. etc). How would I present this extra information to the fully convolutional network?

Any ideas or research would be very helpful.

Edit*
More specifically I am concerned with domain adaptation as shown here, In this example I am using a fully convolutional network without any pooling or striding, where the output is the same size as the input (i.e. an image transformed from one domain to another). In this respect I want to be able run the domain adaptation on images of any size. Adding extra channels to the input seems to make the most sense, however if I am one-hot encoding a bunch of categorical variables, this will consume a lot of memory.

I've been thinking about this problem before and while I have not researched this or even implemented this, I have a few ideas, from easy/obvious to more elaborate:

Don't make it fully convolutional but add some dense layers where you concat this information
If it's just one category, one-hot encode it and broadcast it over your picture as extra channels at the input or maybe halfway in the network
Similar to the previous suggestion but learn an embedding for your categories that you broadcast either at the beginning or the middle
This time, use one-hot encoding as input and learn an embedding that is for example 8x8 and upsample this to your image resolution
Learn an embedding again, but this time use some upconvolutions to scale it to the input (or middle) size

Let me know how much it helps!


Gym action space for board game with reward function Im trying to design an openai gym environment that plays a quite simple board game where each player has 16 pieces that are exactly the same in regard to how they can move.

The board is 10x10 and each piece can go UP, DOWN, LEFT, RIGHT, UP_LEFT, UP_RIGHT, DOWN_LEFT, DOWN_RIGHT. They can move in that direction as many fields as pieces are in that line, including the piece that moves. So if I want to go LEFT I count all other pieces to my left AND my right add 1 for myself and then go that many fields to the left. The field may be obstructed though in which case the move is not possible.

So my question is:
How could I implement an action space for this? Would be discrete with the (sice of the board) * (how many actions[Up, DOWN...]) suitable? And how can I teach the rl AI (PPO2) that a move is not possible? Should I just give a negative reward and give the same state as before?

I would greatly appreciate help :)

You can use each position on the board as an action (100). Then use a mask that will give you only the available positions (from your current position) as possible actions.


Tensorflow - How to set gradient of an external process (py_func)? I'm computing and optimization of some Variables that are used on an external process, but I get the error "No Gradient".

A heavily simplified (not tested) version of the code, but you can get the idea:

def external_process (myvar):
    subprocess.call("process.sh", myvar)
    with open('result.json', 'r') as f:
        result = json.load(data, f)
    return np.array(result["result"])

myvar = tf.Variable(1.0, dtype = 'float32', trainable = True)

loss = tf.reduce_sum( tf.py_func(external_process, [myvar], [tf.float32])[0] )

optimizer = tf.train.AdamOptimizer(0.05)
train_step = optimizer.minimize(loss)
sess.run(train_step)

I saw this discussion but I don't fully understand it: https://github.com/tensorflow/tensorflow/issues/1095

Thanks!

Here https://www.tensorflow.org/versions/r0.9/api_docs/python/framework.html (search gradient_override_map) is an example on gradient_override_map:

@tf.RegisterGradient("CustomSquare")
def _custom_square_grad(op, grad):
  # ...

with tf.Graph().as_default() as g:
  c = tf.constant(5.0)
  s_1 = tf.square(c)  # Uses the default gradient for tf.square.
  with g.gradient_override_map({"Square": "CustomSquare"}):
    s_2 = tf.square(s_2)  # Uses _custom_square_grad to compute the
                          # gradient of s_2.

So, a possible solution could be:

@tf.RegisterGradient("ExternalGradient")
def _custom_external_grad(unused_op, grad):
    # I don't know yet how to compute a gradient
    # From Tensorflow documentation:
    return grad, tf.neg(grad)

def external_process (myvar):
    subprocess.call("process.sh", myvar)
    with open('result.json', 'r') as f:
        result = json.load(data, f)
    return np.array(result["result"])

myvar = tf.Variable(1.0, dtype = 'float32', trainable = True)

g = tf.get_default_graph()
with g.gradient_override_map({"PyFunc": "ExternalGradient"}):
    external_data =  tf.py_func(external_process, [myvar], [tf.float32])[0]

loss =  tf.reduce_sum(external_data)

optimizer = tf.train.AdamOptimizer(0.05)
train_step = optimizer.minimize(loss)
sess.run(train_step)


How to fix TypeError in spyder? How to fix TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''?

from sklearn.preprocessing import Imputer
imputer=Imputer(missing_values="np.nan",strategy="mean",axis=0)
imputer=imputer.fit(x[:,1:3])
x[:,1:3]=imputer.transform(x[:,1:3])

  Traceback (most recent call last):

  File "", line 3, in
      imputer=imputer.fit(x[:,1:3])

  File
  "C:\Users\Nibir\Anaconda3\lib\site-packages\sklearn\preprocessing\imputation.py",
  line 167, in fit
      self.axis)

  File
  "C:\Users\Nibir\Anaconda3\lib\site-packages\sklearn\preprocessing\imputation.py",
  line 254, in _dense_fit
      mask = _get_mask(X, missing_values)

  File
  "C:\Users\Nibir\Anaconda3\lib\site-packages\sklearn\preprocessing\imputation.py",
  line 30, in _get_mask
      if value_to_mask == "NaN" or np.isnan(value_to_mask):

  TypeError: ufunc 'isnan' not supported for the input types, and the
  inputs could not be safely coerced to any supported types according to
  the casting rule ''safe''

You might try it without the double quotes around the np.NaN and do not forget to import numpy (import numpy as np)

As a sidenote, you should use SimpleImputer instead of Imputer:

Imputer was deprecated in version 0.20 and will be removed in 0.22. Import impute.SimpleImputer from sklearn instead.


NLP methods specific to a language? What NLP methods / algorithms depend on the features existing only in some languages? For example, does French has any NLP algorithms that English NLP and Spanish NLP do not have?

This question is quite open, but nonetheless, here are some:

lemmatization/stemming only makes sense in languages where there is a lemma/stem in the word. Some languages like Chinese have no morphological variations (apart from some arguable cases like the explicit plural 们), and therefore lemmatization and stemming are not applied in Chinese.

Word-based vocabularies are used to represent text in many NLP systems. However, in agglutinative and polysynthetic languages, using word-level vocabularies is crazy, because you can put together a lot of affixes and form a new word, therefore, a prior segmentation of the words is needed.

In some languages like Chinese and Japanese, there are no spaces between words. Therefore, in order to apply almost any NLP, you need a preprocessing step to segment text into words.


Predicting return renters I am trying to figure out an approach for calculating the probability of a renter making > 1 booking. I would prefer to use Python for this project. I have the following columns ready:
State, Category (fun, adventure, etc.), returning_renter? (TRUE if >= 1 booking).

What would be the most efficient approach for this project? Are there any particular Python libraries you would recommend?

You could trivially prototype your solution in Orange (it's a Python library that uses scikit-learn under the hood). It also has a neat GUI.

But with only two features (state and renting_reason), I'm afraid your models might not be as precise as you'd want them to be, unless you have lots of examples to learn from.


News clustering on unlabeled datasets I currently have a bunch of extracted news articles to perform news classification. However, the articles are unlabeled. There are about 160k articles therefore manually labeling them is impossible. I'm thinking of clustering the similar news articles together for easier labeling. Is this approach possible?

UPDATE: Now I only want to separate business related news and non-business related news apart.

Simply clustering and then labeling all the texts in cluster will yield very noisy results (and you might learn your clustering similarity function rather that the "real" one). while there are some techniques that handle noisy labels (e.g- this NIPS paper: http://papers.nips.cc/paper/5073-learning-with-noisy-labels.pdf), there are several other options to consider:

labeling a small amount of data (be sure to have label data from each category, assuming you know the categories), training a classifier, classifying new examples, adding the most certain one to the training set, than retraining the model and so on. For example, see this article: http://link.springer.com/article/10.1023/A:1007692713085
Active Learning can be useful in this scenario. E.g: http://www.kamalnigam.com/papers/emactive-icml98.pdf
Semi-supervised learning techniques. E.g: http://mlg.eng.cam.ac.uk/zoubin/papers/zglactive.pdf

It is a common practice to combine more than one method. Also, LDA might be useful here.


How to approach this multi label classification problem and what will be its accuracy metric? I have a dataset for people doing trade in various segments (classes) .I am trying to build a multi-label classifier to predict people trading in various segments (classes).

My dataset :

Client_id [Demographic data] orders_perday Traded_days Segment
 A123      ..............     3             4           equity
 A123      ..............     2             2           commodity
 A123      ..............     1             9           currency
 B789      ..............     7             8           equity
 B789      ..............     3             2           futures
 C456      ..............     2             7           currency
 D987      ..............    10             1           equity
 C183      ..............     2             9           currency

Demographic data includes age, sex, city, income ,etc.

I need to build a model which will predict new clients belong to which class(segment) can be 1 or more.
Please give some advice of how to approach this problem

As with any data science problem: Approach the problem by first exploring the data and then trying the simplest solution that you think might just work, to some extent.

To explore the data, look at the distributions of all the features. Are they at all helpful to identifying the segments? How many trades do you have in the segments, are the classes balanced (more or less similar counts in all classes)? If not, you need to be careful with a metric like accuracy and you may be better off using F1 score, for example.

For an algorithm, how about starting with a decision tree and taking it from there? You can use it as a baseline model to compare to the results of other experiments. For example more complex algorithms using trees like boosted trees or random forest, or neural networks.

The performance metric is often averaged across all classes, using a technique called “1 versus all”. For that you start with the first segment, like “equity”. You can use the model to identify “equity” or “not equity”, and calculate the F1 score for this task. Repeat for all other classes and average.


What is the use of using width/height shift in data augmentation? I'm not sure to understand the use of augmentation data using width shift and height shift.
Say I have limited image data, and I want to create new data using Keras' ImageDataGenerator. To classify between images, I use CNNs. Since CNNs are translation invariant, aren't the translational shifts from keras useless as those shifts will not result in new images per say? (I know new images will be created from the generator, but the CNNs will not learn new features from the pictures, and might instead cause overfitting?)

Since CNNs are translation invariant, aren't the translational shifts
from keras useless as those shifts will not result in new images per
say?

Invariance to translation means that if we translate the inputs the CNN will still be able to detect the class to which the input belongs. Translational Invariance is a result of the pooling operation. In pooling operation, we replace the output of the convnet at a certain location with a summary statistic of the nearby outputs such a maximum in case of Max Pooling. As in the case of max-pooling, we replace the output with the max, so even though we adjust the input slightly, it will not impact the values of most pooled outputs.
The major issue with max pooling is that the network fails to learn the spatial relation between different features, and thus will give a false positive if all features are present in the wrong position with respect to one another. This happens for Cases like Image Segmentation where we require position.
So, Data Augmentation comes in handy to handle heavy distortion in images and make the model more robust to these distortion when we have less images.
Reference:
Translation Invariance


Kmean clustering on text data I have a large raw dataset on crime and I want to cluster the data using k-means, However, I get an Error when I enter this code:

Rawdata.3means <- kmeans(Rawdata, centers = 3).

Error:

Error in kmeans(Rawdata, centers = 3) :
  more cluster centers than distinct data points.
In addition: Warning message:
In storage.mode(x) <- "double" : NAs introduced by coercion

How can I resolve this error?

Kmeans utilize the mean of your data points for clustering . If your dataset is made of plain text or other type of factors (i.e not numbers) then it wont work for you . You need to do another step of preprocessing your data before you can apply Kmean or most of the ML algorithms .

Categorical dataset : i.e your data is in the form of multiple categories like column of fruits  with values of Apple , orange ,banana etc. Then you can use "one hot encoding" method that will transform your category column into multiple columns that each indicate if the sample is belong to the relevant category (i.e for column with 3 fruit types you will get 3 new binary (1 or 0) columns - is apple ? is orange? is banana ?  read more about how to do it in R here : One hot encoding in R

Update: like some suggested in the comments , K means wont be the best approach for clustering categorical data and in some cases you can get much better results when using more suitable approaches .Here is a link to another (more advanced) method for clustering categorical data in R - ROCK algorithem (kaggle notebook) . Also ,you can read about "Kmode" which is similar to kmeans for categories and implemented in R

If your dataset is plain text (like tweets or stackexchange posts) :
One common method is using td-idf (but there are many more) , you can read more here:
Text clustering using R: an introduction for data scientists
and here in a nice kaggle R notebook:
R : cleaning data, and using TF-IDF


Call a SQL procedure, for each group by I am trying to call a procedure for each group of data. For example, I have a list of customers and movie's features that they like. Is a relation one to many, I have many records for one specific customer in dataset. Therefore, what I am trying to do is for each specific customer to apply a clustering algorithm. Need some help with what statement should I apply this on SQL.

Thank you in advance!

I'm not sure I get your question, but not enough reputation to comment. Thus, I'll make some assumptions and answer to the best of my understanding with them.

For example with SQLite (this is one of the simplest, variations possible in different dialects), let's assume your data is created with:

CREATE TABLE "Customer" (
  id INTEGER NOT NULL,
  name VARCHAR(64) NOT NULL,
  PRIMARY KEY (id),
  UNIQUE (id)
);

CREATE TABLE "Movie" (
  id INTEGER NOT NULL,
  customer INTEGER NOT NULL,
  movie VARCHAR(64) NOT NULL,
  PRIMARY KEY (id),
  UNIQUE (id),
  FOREIGN KEY(customer) REFERENCES "Customer" (id)
);

Let's populate with some data:

INSERT INTO Movie (id, customer, movie) VALUES(0, 0, 'Rambo');
INSERT INTO Movie (id, customer, movie) VALUES(1, 0, 'Rocky');
INSERT INTO Movie (id, customer, movie) VALUES(2, 1, 'Mamma Mia');
INSERT INTO Movie (id, customer, movie) VALUES(3, 1, 'Grease');
INSERT INTO Movie (id, customer, movie) VALUES(4, 0, 'Terminator');

As long as you are happy with applying some aggregate functions, you can just use a single SELECT like this:

SELECT
  customer, MAX(id),
  COUNT(DISTINCT id), AVG(LENGTH(movie)), SUM(LENGTH(movie))
FROM Movie GROUP BY customer;

If you need to get data from the Customer table, you might need to make a join like this:

SELECT
  Customer.name, Customer.id, MAX(Movie.id),
  COUNT(DISTINCT Movie.id), AVG(LENGTH(Movie.movie)), SUM(LENGTH(Movie.movie))
FROM Movie
JOIN Customer on Movie.customer=Customer.id
GROUP BY customer;

If you want to do k-means, SQLite is quite limited, but with other dialects you can still do it. See this paper for an example. On the internet one can easily find examples in PostgreSQL, MySQL, SQL Server,... you could google yours.

Most other SQL dialects would have a richer set of predefined functions. For more complicated analysis you might need to define your own functions, and probably work with the results out of your SQL.


exporting the model parameters after training and testing I want to construct a python micro service to generate predictions based on inputs from web via an API.
I have trained a logistic regression model, but need to create a python script to run in AWS lambda or a container with minimum set of libraries.
How do I see and possibly export the model parameters?

I'm not sure if that's what you are looking for - but you should check out the package 'pickle'. You can use it to easily dump your model into a file and load it again.


Determining whether a sentence is "cliche" using NLP I have a collection of essays from students. Each essay is about the same topic and of the same word length. My goal is to develop a machine learning algorithm that pinpoints "cliche" sentences (i.e., sentences that are "unoriginal" and similar to what other students have written).
Let document X be the essay we're trying to analyze. Let S be the set of all essays. Here is the rudimentary approach I've thought of:

Split all essays (from set S - X) into individual sentences. Call this collection of sentences Y. Use this solution to compare each sentence of X to each sentence in Y. This will yield an array of scores Z for each sentence in X.
For each Z: weight each score in Z based on position. If a sentence in X and a sentence in Y are in similar locations in their respective essays (e.g., both at the intro of the essay) the weight on the respective score in Z will be higher than if these two sentences were farther apart.
Average all of the Z arrays for all of the sentences. The sentence with the highest average is the "most cliche."

Is there a better approach to this problem?

IMO. Firstly, you make the hypothesis that cliche sentences are in the same part of the document, which makes sense. But how do you determine the weight of the position and what if the students use different structures (e.g., some write the conclusion and the discussion sections separate, others write them together)?
Then you could get more up to date sentence embeddings algorithms than the one you suggested to provide you with more accurate similarity scores.
Finally, how I would approach it. I would use a clustering technique such as DBScan or HDBScan (with a cosine distance). The reason I would use these is that they are distance-based clustering techniques. Hence, initially, you can start with a small aperture, distance, and view which sentences are very similar to each other, almost identical. Then you can increase your distance, open your aperture, and view which others are quite similar to them. You could of course try other clustering techniques and see how they would work since those two are usually not the best. Then I would take into consideration where each sentence position is and see if it would change the results, like that, I believe, you have more of an insight into what is happening.


Small project ideas for Machine Learning I need some serious help. I am supposed to implement a project (Non-Existing as of now) for my Machine Learning course. I have no basics in AI or Data mining or Machine learning. I have been searching for a while and unable to find something that i can finish implementing in 3-4 weeks time. It carries a huge chunk of my final marks and no matter how much i try i am unable to understand how it works!
Can the machine learning masters please help me out with this. I need a project suggestion to start with. And i want to know how to proceed after gathering the data set. I am totally blank and running out of time for my graduation :(

Appreciate your suggestions! Thanks in advance.

Gather million twitter user accounts. Then try to guess their gender based on their avatar, tweets and who they follow.


When to use Random Forest over SVM and vice versa? When would one use Random Forest over SVM and vice versa?

I understand that cross-validation and model comparison is an important aspect of choosing a model, but here I would like to learn more about rules of thumb and heuristics of the two methods.

Can someone please explain the subtleties, strengths, and weaknesses of the classifiers as well as problems, which are best suited to each of them?

I would say, the choice depends very much on what data you have and what is your purpose. A few "rules of thumb".

Random Forest is intrinsically suited for multiclass problems, while SVM is intrinsically two-class. For multiclass problem you will need to reduce it into multiple binary classification problems.

Random Forest works well with a mixture of numerical and categorical features. When features are on the various scales, it is also fine. Roughly speaking, with Random Forest you can use data as they are. SVM maximizes the "margin" and thus relies on the concept of "distance" between different points. It is up to you to decide if "distance" is meaningful. As a consequence, one-hot encoding for categorical features is a must-do. Further, min-max or other scaling is highly recommended at preprocessing step.

If you have data with $n$ points and $m$ features, an intermediate step in SVM is constructing an $n\times n$ matrix (think about memory requirements for storage) by calculating $n^2$ dot products (computational complexity).
Therefore, as a rule of thumb, SVM is hardly scalable beyond 10^5 points.
Large number of features (homogeneous features with meaningful distance, pixel of image would be a perfect example) is generally not a problem.

For a classification problem Random Forest gives you probability of belonging to class. SVM gives you distance to the boundary, you still need to convert it to probability somehow if you need probability.

For those problems, where SVM applies, it generally performs better than Random Forest.

SVM gives you "support vectors", that is points in each class closest to the boundary between classes. They may be of interest by themselves for interpretation.