Add extra term weight when grouping strings by similarity? I'd like to convert a set ducoments to term weight matrix(features) by tf-idf.
Then calculate the similarity of two document by their features.

similarity is compute by result = matrix*matrix.T > 0.9 detail here, and group them by the result(loop result, if result[1,0] >0.9, then index 1 is similar with index 2)

Now I have a lot of resource to group.

For example, I have some books with different names(more complex in fact)
I can roughly group these books name by similarity, like below:

step1:

group 1:
    1.The Three Body Problem vol 1
    2.[Chinese]The Three Body Problem no 1
    3.The Three Body Problem 2
    4.The Three Body Problem vol 3[Japanese]
    5.Problem of Three Body vol 3
    6.(xx)The Three Body Problem 2
    7.The Three Body Problem 1[English]
group 2:
    1.Another book 1
    ....

But xxx 2 and xxx vol3 is needless when I want to find xxx vol 1 ,
So have to do

step2:
   tokenize each book name again, use some patterns/rules to extract the book number to distinguish them.

Is there any way to add some term(such as Arabic numerals:0-1, English number:one- twenty) with high dissimilarity weight, to make step1 result

group 1:
    1.The Three Body Problem vol 1
    2.[Chinese]The Three Body Problem no 1
    7.The Three Body Problem 1[English]
group 2:
    3.The Three Body Problem 2
    6.(xx)The Three Body Problem 2
group 3:
    4.The Three Body Problem vol 3[Japanese]
    5.Problem of Three Body vol 3
group 4:
    another book 1
    [xx]another book vol.1

update

If there are some titles like below:

1. There are 2 man vol.1
2. (xx)There man 2 boy 2

I need add a lot detection(number position or something else), that's why I want a way to add a extra weight to somewhere(to avoid step 2, the redundant tokennizing and custom extraction rule).

I think the similarity weight may work like:

tfidf weight two title, plus each number weight, then calculate the similarity matrix.
But now I am using tfidf matrix power to get similarity matrix,
I don't know how to add a extra weight to  tfidf weight result, the weight meaning is different between tfidf weight and what mentioned extra weight.

I want to know where and how to add the proper extra weight, how calculate it value?

When you have a similarity measure fixed already, you can use correlation clustering to turn your similarity matrix into groups as you wanted. Here's a good resource about correlation clustering http://www.francescobonchi.com/CCtuto_kdd14.pdf

For the second step to attempt to make sure volume 1 and volume 2 are not the same, I would add a number of rules that try to find the difference. Look for numerals, if they don't match they have 0 similarity. Play around a bit with this. After you have your new similarity matrix you can use correlation clustering which should seperate volumes 1 and 2 into different groups.


Machine learning with constraints on features I am working on a learning to rank problem. I have queries and documents related to every query which I have to rank. I used lightgbm ranker to fit the model. Some of features are very important and if they are changed the fitted model predicts a better score for that document and thus a better rank.
Lets say, for a single query id, I have a group of documents d1....d5 each having features f1...fn.
I change the features f1,f2,f3 for document d4 and it pushes the rank of d4 from 4 to 3.

What should be the values of these features f1,f2,f3 that can push the rank for document d4 from 4 to 1 subject to the condition that f1+f2+f3=45 is always satisfied ? (i.e. the sum of new modifications for f1,f2,f3 should be always equal to 45 )

As far as I understand this problem is simply a constrained optimization problem. In such a problem we have an objective function and constraints. In my case I do not have a mathematical equation that relates rank to my features r=f(f1....fn) but I do have a learned model which I can use as an objective function.
Follow up question
All this made me think of another question. In the case of OLS, which is a constrained optimization problem, the constraints present in this kind of optimization are always on the coefficients.

If I would like to have constraints on the predictors does it really make sense ? How does it relate to my problem ?

I think the objective function of a regression problem and the objective function of constrained optimization problem i mentioned above are different.
I would really appreciate some thoughts on this, thanks !

To be honest I didn't understand the problem very well, but it looks like genetic learning could be a good direction to study.
Essentially a genetic algorithm works by iteratively trying many combinations of values for some features, evaluating "something" based on the features, selecting the best combinations and starting again with a combination of the best features so far. The goal is to eventually converge to an optimal combination of values.


Linear Regression not working due to wrong kind of array I try to deal with my homework. The Job is to take this Data and perform a linear regression on it.
The code is published here.
I am quite new to programming in Python and in data science. So I tried transforming as the interpreter suggests, but it didn't work.
My first error was that there was a 2d array expected but 1d given. Then I took the pure array and put it into an empty one suggested by a StackOverflow answer now the error is that a scalar array is given but a 2d array is given.
import pandas as pd
from sklearn.preprocessing import StandardScaler

#Import
data = pd.read_csv('uscrime.txt', sep="\t")
crime = pd.concat([data], axis = 1)
print(crime)

from sklearn.linear_model import LinearRegression
regression = LinearRegression()
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(crime.get("M"), crime.get("Crime"), test_size=0.2, random_state=0)

X_train_new = []
X_train_new.append(X_train.values)

y_train_new = []
y_train_new.append(y_train.values)

regression.fit(X_train_new, y_train_new)

import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

data = pd.read_csv("http://www.statsci.org/data/general/uscrime.txt", sep="\t")
x = data.loc[:, data.columns != 'Crime'].to_numpy()
y = np.squeeze(data.loc[:,'Crime'].to_numpy())

regression = LinearRegression()

regression.fit(x, y)

scikit learn expects numpy arrays and not pandas dataframes. You need to convert from one to the other on top, you need to make sure that the array for y only has one dimension which I achieved via np.squeeze.
Bonus: see above how you can directly load the csv from the website.


algorithm to predict numeric values with sklearn I'm new to ML and trying to learn it. I scraped information (using python) from a website where people try to sell their cars. I put data in a pandas data frame. Now I'm wondering how should I predict car's price according to it's manufacturing year and its mileage! I drew some plots and there is a correlation between price-year and price-mileage!

You should try linear regression with sci-kit learn. You can use both year and mileage as predictors for price.


How to model this "un predicatability" problem? Imagine modeling the "input(plaintext) - output(ciphertext)" pairs of an encryption algorithm as a data science problem. Very informally, the strength of an encryption scheme is measured by the randomness (unpredictability, or entropy) of the output. This is counter-intuitive to classic regression problems, which are frequently used for prediction. Say, informally, the strength of an encryption scheme is determined by the number of such input-output pairs needed beyond which it becomes predictable.

How do we model this as data science problem? Given all pairs of two different encryption schemes, can we determine which is stronger just by using the input-output pairs of both the schemes? Is there any other way apart from regression to solve this problem?

tl;dr: This is not a great candidate for machine learning solutions.

It is not exactly true that "the strength of an encryption scheme is measured by the randomness (unpredictability, or entropy) of the output".  Instead, strength of an encryption scheme is how resistant it is to cryptanalysis. Yes, input $\rightarrow$ output maps that show patterns are relatively easy to break. But the input $\rightarrow$ output map might be very random (high entropy) from many perspectives but still might be defeated through feasible cryptanalysis.

You offer another definition: "informally, the strength of an encryption scheme is determined by the number of such input-output pairs needed beyond which it becomes predictable."  This is usefully different from your first definition because it focuses on the information that might be gleaned about the encryption scheme from a stream of input $\rightarrow$ output pairs.  This is different and easier than the usual cryptanalysis problem where all you have is a data set of (encrypted) outputs.

Framed this way, I think you are heading toward a model selection problem.  While this may be relatively straight forward with weak encryption schemes, it becomes very hard, very quickly with strong schemes.  In fact, I believe you walk right in to the No Free Lunch theorem:

  "In formal terms, there is no free lunch when the probability distribution on problem instances is such that all problem solvers have identically distributed results. In the case of search, a problem instance is an objective function, and a result is a sequence of values obtained in evaluation of candidate solutions in the domain of the function. For typical interpretations of results, search is an optimization process. There is no free lunch in search if and only if the distribution on objective functions is invariant under permutation of the space of candidate solutions."

Translated: if you find yourself needing to search over all possible cryptanalysis methods, there exist input $\rightarrow$ output streams where no method is probabilistically better than any other. This will defeat any model selection method based on optimization of some objective function.


Interpreting fraction of zero weights in TensorFlow I am using the TensorFlow to do a simple linear classification using logistic regression. The graph included from the TensorBoard displays what they call the fraction of zero weights. How do I interpret this in terms of model evaluation? I am assuming this is good since I got the good results in terms of loss, precision, recall, etc but not sure.

Thank you.

If you are performing linear (logistic) regression your weights are simply your $\beta_i$. If none of them are $0$ that simply means all features are 'important' to some degree.


How to normalize data from multiple sources? I am trying to model an individuals' purchasing behavior using different data sources (ex: Zalando, Otto, etc.,). When I combine data sources, I see that the data across these channels is very different.

For example, 5% heavily shop using a particular channel, but do not utilize other channels. When I try to model anything from this information, it performs poorly because overall, its a sparse dataset, but a % of each column should be a very good predictor of a small subset of population.

My question is: How do I combine/normalize such a dataset wherein the data is super sparse?

Make sure that when you merge your datasets, the column information is as generic as possible, which means: When you merge A with B, make sure you name "Purchases at (any of the) online channel" not "Purchases at A's online channel" and "Purchases at B's online channel.

Merging datasets from completely different datasources is very difficult because they were not thought with the same basic ideas.


LDA as a dimensionality reducer I know how to use LDA as a classifier.
But how to use Linear Discriminant Analysis as a dimensionality reducer to reduce the number of features and apply logistic regression on top of it.
I am using R language.

We can do LDA via the lda function from the MASS package. The reduced dimension data can be computed as follows:

# Model Discriminant Analysis
library( MASS )
model = lda( class ~ ., data = X_train )
# Ploting LDA Model
projected_data = as.matrix( X_train[, 1:18] ) %*% model$scaling

You can then feed the projected_data into another supervised learning method.

Credit: The code is obtained from here.


CART algorithm (Classification and regression trees) question So this is taken from an exam I just did. I'd like to know if there are any instances same as in the image where the CART algorithm could use a negative alpha and thus encourage a larger tree? Or does the algorithm state that alpha must be a non negative integer at all times?

Making an additional split will always decrease $Err$ (until pure leaves are reached), so any negative $\alpha$ will have the same minimizing $T$ as $\alpha=0$.  So we might as well take $\alpha\geq0$.  (But there is no requirement that $\alpha$ be an integer.)

Erm, unless you implement things so that even a pure node can be split, in which case a negative $\alpha$ will make the tree split even these, until every leaf consists of a single sample (and duplicates thereof) or the maximum depth is reached.


I have 32k black and white images. Want to do clustering on them As the title says I'm trying to do clustering on a set of black and white images. These images are all 200x200 with black dots on a white canvas Example pics here (These are not actual photos from the data set. Just a general idea of what they look like). The idea is to hopefully find an underlying pattern when it comes to general shape of these images and hopefully cluster by that.

What I've done so far is turned each of my 200 x 200 image in a numpy array of size 40,000. Then put all the images together into a singly numpy array of 32k x 40k.

I'm kind of not sure where to go from there. What I did next was use scikit learn's TruncateSVD on my data set and set the paramater 'n_componenets=100' and fitted and transformed it, so now my data set of images are 32k x 100.

From here on out I don't know what's the best plan of action. Should I just start using k_means algos on my data set? and how would I visualize it?

Sorry if this is a lot. Any help/tips would be much appreciated it.

As the dimension of your data is very high (40000), you can try using Spherical K-Means on your raw data. It uses the cosine similarity to compute the distance between 2 points. Indeed, euclidian distance is not so much informative in high dimensional vector spaces, because of the curse of dimensionality. see https://stats.stackexchange.com/questions/99171/why-is-euclidean-distance-not-a-good-metric-in-high-dimensions

Or you can apply a dimension reduction strategy like truncated SVD or ACP like you did, and then apply the classical k-means algorithm. So yeah you should try to apply k-means on your 100-dimensions data. Maybe try with k = 30 or something similar as 100 may already be too high.


Data scaling for large dynamic range in neural networks The usual strategy in neural networks today is to use min-max scaling to scale the input feature vector from 0 to 1. I want to know if the same principle holds true if our inputs have a large dynamic range (for example, there may be some very large values and some very small values). Isn't it better to use logarithmic scaling in such cases?

If it is a classification problem,  then you will use sigmoid or softmax to make the output value in (0,1) and all the value must sum to 1 as per the rule of probability.


Visualization using D3 I am new to D3 programming (any programming, for that matter). I have protein-protein interaction data in JSON format and csv format. I would like to use that data for network visualization.

Data attributes: Protein Name, Protein Group, Protein type, Protein Source Node, Protein Target Node

Can anyone suggest good network visualizations for such data. How does it work with hive plots?

Why dont you have a look at the following example?
http://bl.ocks.org/mbostock/2066421

You can also find a fiddle here: http://jsfiddle.net/boatrokr/rk2s5/

Pay attention to the part where the links are defined.


Is this correlation between distance matrices? I have a set of objects. I have calculated two distance matrices: $X$ defining distance between each objects pair using metric $f1$, and $Y$ -- using metric $f2$. Now, I would like to understand if two objects are similar according to metric $f1$, then they are also similar according to metric $f2$. How can I do it?

For instance, $f1$ could say whether two objects have similar color, and $f2$ --- whether two objects have similar size. But metrics can be anything. For instance, we could talk about articles, $f1$ could be Jaccard distance measuring how many tags both articles share, and $f2$ could be euclidian distance measuring distance between word vectors of two articles. Now I would like to understand if two objects of blueish objects tend to be big, or whether articles tagged with "racism" have similar content.

Am I asking about correlation? How can I calculate it between $X$ and $Y$?

You are basically right. You want to check the degree of dependence of one variable with respect to another one. No matter how you generate each variable, if you want to know how dependent it is from the other one, you normally use correlation to run that evaluation.

Very useful options you should consider to analyze the potential dependence between X and Y are:

Correlation: to measure both the strength and direction of the linear relationship between two variables.
Covariance: to assess the direction of the linear relationship between variables (not the strength).
Pearson's correlation: to obtain a single metric representing the degree (strength) of dependence between the two variables
Spearman's correlation: to evaluate a potential non-linear dependence between the two variables.

For more information, please, check this post here.


LinearRegression with fixed slope parameter I have some data $(x_{1},y_{1}), (x_{2},y_{2}), ..., (x_{n},y_{n})$, where both $x$ and $y$ represent real numbers (float). I want use Scikit-learns LinearRegression model to fit a model of the form:
$y_{i} = b_{0} + b_{1}x_i + e_{i} $
Typically, I know that OLS is used to compute the parameters $b_{0}, b_{1}$. However, in my case, I happen to know that  $b_{1}=c$ so I only want to fit $b_{0}$. Is there a way to force scikit-learn to use $b_{1}=c$ as the slope ratio and only estimate the intecept $b_0$, or is a custom class necessary?

You can just compute:  $\hat{b}_0 = \operatorname{mean}(y-cx)$


Multi-class classification with only one feature I am studying the efficacy of using a single feature for predicting a set of events (which is a multi-class classification problem). I was wondering if it makes any sense to use only one feature for multi-class classification. Thanks.

Yes! You can use only one feature for multi-class classification. For example, you have one feature height and you want to classify short , medium or tall.


Using a random forest, would a RandomForest performance be less if I drop the first or the last tree? Suppose I've trained a RandomForest model with 100 trees. I then have two cases:

I drop the first tree in the model.
I drop the last tree in the model.

Would the model performance be less in the first or the second case?
As the last tree should be the best trained one, I would say that the first scenario should be less performant than the last one.
And what if I was using another model like a Gradient Boosting Decision tree?  I guess it should be the same.
I am okay with some math to prove it, or any other way that might prove it.
Update
I tried with two different learning rates 0.1 and 8. With 0.1 I get:
# For convenience we will use sklearn's GBM, the situation will be similar with XGBoost and others
clf = GradientBoostingClassifier(n_estimators=5000, learning_rate=0.01, max_depth=3, random_state=0)
clf.fit(X_train, y_train)

y_pred = clf.predict_proba(X_test)[:, 1]
# "Test logloss: {}".format(log_loss(y_test, y_pred)) returns  0.003545821535500366

def compute_loss(y_true, scores_pred):
    '''
        Since we use raw scores we will wrap log_loss
        and apply sigmoid to our predictions before computing log_loss itself
    '''
    return log_loss(y_true, sigmoid(scores_pred))

'''
    Get cummulative sum of *decision function* for trees. i-th element is a sum of trees 0...i-1.
    We cannot use staged_predict_proba, since we want to manipulate raw scores
    (not probabilities). And only in the end convert the scores to probabilities using sigmoid
'''
cum_preds = np.array([x for x in clf.staged_decision_function(X_test)])[:, :, 0]

print ("Logloss using all trees:           {}".format(compute_loss(y_test, cum_preds[-1, :])))
print ("Logloss using all trees but last:  {}".format(compute_loss(y_test, cum_preds[-2, :])))
print ("Logloss using all trees but first: {}".format(compute_loss(y_test, cum_preds[-1, :] - cum_preds[0, :])))

which gives:
Logloss using all trees:           0.003545821535500366
Logloss using all trees but last:  0.003545821535500366
Logloss using all trees but first: 0.0035335315747614293

Whereas with 8 I obtain:
clf = GradientBoostingClassifier(n_estimators=5000, learning_rate=8, max_depth=3, random_state=0)
clf.fit(X_train, y_train)

y_pred = clf.predict_proba(X_test)[:, 1]
# "Test logloss: {}".format(log_loss(y_test, y_pred)) returns 3.03310165292726e-06

cum_preds = np.array([x for x in clf.staged_decision_function(X_test)])[:, :, 0]

print ("Logloss using all trees:           {}".format(compute_loss(y_test, cum_preds[-1, :])))
print ("Logloss using all trees but last:  {}".format(compute_loss(y_test, cum_preds[-2, :])))
print ("Logloss using all trees but first: {}".format(compute_loss(y_test, cum_preds[-1, :] - cum_preds[0, :])))

gives:
Logloss using all trees:           3.03310165292726e-06
Logloss using all trees but last:  2.846209929270204e-06
Logloss using all trees but first: 2.3463091271266125

The two slightly-smaller models will perform exactly the same, on average.  There is no difference baked in to the different trees: "the last tree will be the best trained" is not true.  The only difference among the trees is the random subsample they work with and random effects while building the tree (feature subsetting, e.g.).
Gradient boosted trees are a different story.  If you drop the first tree after you finish training, the resulting model will be mostly garbage.  Every subsequent tree was trained to improve upon the fit of the previous trees, and removing any single tree will put all future trees out of context.  (To give an extreme example, suppose the first tree actually captures "the correct" model.  All future trees will just fit on the remaining noise.)  On the other hand, removing the final tree is equivalent to having just trained one fewer tree, which may be good or bad depending on your bias-variance tradeoff at that point.


How to deal with Missing Not at Random Data for k-means clustering? I am running k-means clustering on a customer dataset. One of the available demographic fields is inferred homevalue, represented as an integer.
This field has value 0 when it's inferred that the customer is not a homeowner at all (they are more likely a renter, live with relatives, etc).
I'm struggling to think of a good way to treat this value.

Does it make sense to keep this value as 0? Then my understanding is that the algorithm will interpret this as someone who doesn't own property is closely related to someone who owns an extremely low value property which doesn't seem intuitively right.

Is there a better way of dealing with this?

I think there are two approaches you can take:

1) Square the 'homevalue'

Assuming that homevalue is a numeric you can square this value.

Then, you will increase the difference between $0$-valued homes and low-valued homes. Further, it will make the difference between high-valued homes to low-valued homes also larger.

2) Split into two features

You can also split this information into two different features:

Home value $v$, where $v \in \mathbb{R}^+$
Owns a home $h$, where $h \in \{true, false\}$ and $h = \substack{false, \quad if \, v \, equals \, 0\\true, \quad \ otherwise}$

This way, your model has a way to explicitly distinguish between people that own or not a home.

Then hopefully you have another feature indicating the lifestyle or earnings of a person that doesn't own a home.


ValueError: X has 3 features per sample; expecting 1500 import numpy as np
from flask import Flask, request, jsonify, render_template
import pickle
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.linear_model import LogisticRegression
from flask_bootstrap import Bootstrap
app = Flask(__name__)
Bootstrap(app)
model = pickle.load(open('model.pkl', 'rb'))

@app.route('/')
def home():
    return render_template('index.html')

@app.route('/predict',methods=['POST'])
def predict():
    '''
    For rendering results on HTML GUI
    '''
    vectorizer = TfidfVectorizer()
    namequery = request.form['namequery']
    data = [namequery]
    vect = vectorizer.fit_transform(data)
    my_prediction = model.predict(vect)
    return render_template('result.html',prediction = my_prediction)

if __name__ == "__main__":
    app.run(debug=True)

here is preprocessing of dataset code

import numpy as np
import pandas as pd
import re
import string
import nltk
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn import metrics
import matplotlib.pyplot as plt
import seaborn as sns
import pickle

data = pd.read_csv('train.csv')
#removing id column
data=data.drop('id',axis=1)
string.punctuation
nltk.download('stopwords')
stopword=nltk.corpus.stopwords.words('english')
def remove_punct(text):
  text_nopunct=''.join([char for char in text if char not in string.punctuation])
  return [word for word in text_nopunct.split() if word.lower() not in stopword]
x=data['text'].apply(remove_punct)
ps=nltk.PorterStemmer()
def stemming(text):
  text=[ps.stem(word)for word in text]
  return ' '.join(text)
x=x.apply(stemming)
vectorizer = TfidfVectorizer(max_features=1500)
X = vectorizer.fit_transform(x)
print(vectorizer.get_feature_names())
y=data['target']
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.25,random_state=0)
# instantiate the model (using the default parameters)
logreg = LogisticRegression()

# fit the model with data
logreg.fit(X_train,y_train)

#
y_pred=logreg.predict(X_test)
from sklearn import metrics
cnf_matrix = metrics.confusion_matrix(y_test, y_pred)
cnf_matrix
class_names=[0,1] # name  of classes
fig, ax = plt.subplots()
tick_marks = np.arange(len(class_names))
plt.xticks(tick_marks, class_names)
plt.yticks(tick_marks, class_names)
# create heatmap
sns.heatmap(pd.DataFrame(cnf_matrix), annot=True, cmap="YlGnBu" ,fmt='g')
ax.xaxis.set_label_position("top")
plt.tight_layout()
plt.title('Confusion matrix', y=1.1)
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
# Saving model to disk
pickle.dump(logreg, open('model.pkl','wb'))

# Loading model to compare the results
model = pickle.load(open('model.pkl','rb'))

please help me someone to reslove this error. Iam new to machine learning am not understanding where i made mistake.
Thanks in advance!!

You should not so a fit transform while prediction. You should save the fit model in your preprocessing code for the tfidfvectorizer and use the same vectorizer while predicting.

Otherwise your vectorizer is learning new word representations and whose dimensions might not be 1500 what you have specified in your preprocessing code.


Is supervised machine learning by definition predictive? I am trying to organize a cheat sheet of sorts for data science, and I am working with the basic distinction between description, inference, and prediction. As examples of the first I see unsupervised methods described, and for the last I see supervised methods. So my question is simply, do these two sets of categories align? Is unsupervised to supervised as description is to prediction?

A description as any statistic drawn from your sample data, say the sample mean, quantiles, etc.. Inference is a conclusion drawn from your sample data about the population, e.g., rejecting or accepting some hypothesis or stating that a model is suitable or not for describing your data. Prediction is simply a guess about future observations, which hopefully uses your data and some function/model of the data in a way to formulate that guess.

Both unsupervised and supervised learning methods aim to learn a function of the data that predicts another variable (typically called y) so both are drawing an inference (i.e., a model is well suited to describe your data, see the first sentence here). However, these two methods differ in what data is available. In supervised learning you are able to use an observed sample of y for training your model and in unsupervised learning, y is unobserved.

Hope that helps!


Getting the positive impacting features using SHAP I'm attempting to use SHAP to automatically extract feature names that have a positive impact on my regression models. On inspection of the code I see that the bar plot, for example, determines these by taking the mean absolute SHAP values for a feature. Being an absolute value, it obviously takes the absolute impact but I want to only consider positive impacting values.

Is my intuition that I can just take the mean instead of the mean of the absolute values correct?
(highly) Negative SHAP values should give a negative mean value.

Is this a good approach or am I missing some better way to do this?

EDIT:
I am specifically interested in features that raise the predicted value. ie. if feature_1 lifts the predicted value by 100 and feature_2 by 1000, I want this information to be extracted as feature_2 has and higher impact on the output value.

Depending on your model there may be some better model-specific approaches than SHAP. It is also important to note that SHAP is an approximation of Shapley value, with the main assumption of not having too much correlation between your features.

That being said, taking the mean instead of the mean absolute values seems to be the most efficient approach in the continuity of what you are doing. Just keep in mind that :

SHAP don't have a "physical" interpretation, in terms of direct impact on the output. It may lack some meaning for real life users.
Taking the mean can "hide" skewed SHAP profile : a variable with high impact on a small subgroup of instances and no impact on the rest may get the same average than a variable with small impact on every instance.


classification algorithms for anomaly detection when there is class imbalance My target variable is boolean and has 0.001% of NO records and remaining YES records.
Can anyone suggest algorithms which are best for anomaly detection in R when there is severe class imbalance.

Here is a pretty detailed summary on Handling Imbalanced Classes. Since you mentioned R, you might want to take a look at unbalanced package.


How do I right feature selection for DBSCAN? I want to use DBSCAN to recognize any clusters within all text elements from the DOM tree of any webpage. For example all menu items shall be clustered separatey to all main content or footer elements.

My features for now are:
- DOM path to the text element
- visual rendered CSS properties (used PhantomJS) of  the text element
- the middle x and y position of the text element
- the width of  the  text element

Before using DBSCAN I use StandardScaler for my feature vector.

But I have no good results even after trying channging eps.

Now the question:
Can DBSCAN handle att all such different kind of features like OneHotEncoding for DOM path combined (!) with features like positions or widths in pixel? Should I weight some features before using StandardScaler that all have a similar value?

Automatic weighting will likely not be enough.

For examples standard scaler will assign twice as much weight to the one-hot encoded parts than to the other attributes. Plus, it is based on the assumption that all variables should have the same (assuming they are all normal distributed) weight. But should they all have the same weight? Does it even make sense to use distances of one-hot encoded data?

You are not ready to use DBSCAN (or any other clustering) if you don't have a reliable distance yet that discriminates similar from dissimilar instances.


Efficient algorithm to compute the ROC curve for a classifier consisting of an ensemble of disjoint classifiers Suppose I have classifiers C_1 ... C_n that are disjoint in the sense that no two will return true on the same input (e.g. the nodes in a decision tree).  I want to build a new classifier that is the union of some subset of these (e.g. I want to decide on which leaves of a decision tree to give a positive classification).  Of course, in doing so there will be a trade off between sensitivity and positive predictive value.  So I would like to see a ROC curve.  In principle I could do this by enumerating all subsets of the classifiers and computing the resulting sensitivity and PPV.  However, this is prohibitively expensive if n is more than 30 or so.  On the other hand, there are almost certainly some combinations that are not Pareto optimal, so there might be some branch and bound strategy, or something, that avoids most of the computation in many cases.

I would like advice about whether this approach is likely to be fruitful and whether there is any work or if you have any ideas about efficiently computing the ROC curve in the situation above.

If I understood the question correctly, you have trained an algorithm that splits your data into $N$ disjoint clusters. Now you want to assign prediction $1$ to some subset of the clusters, and $0$ to the rest of them. And amont those subsets, you want to find the pareto-optimal ones, i.e. those who maximize true positive rate given fixed number of positive predictions (this is equivalent to fixing PPV). Is it correct?

This sounds very much like knapsack problem! Cluster sizes are "weights" and number of positive samples in a cluster are "values", and you want to fill your knapsack of fixed capacity with as much value as possible.

The knapsack problem has several algorihms for finding exact solutions (e.g. by dynamic programming). But a useful greedy solution is to sort your clusters in decreasing order of $\frac{value}{weight}$ (that is, share of positive samples), and take the first $k$. If you take $k$ from $0$ to $N$, you can very cheaply sketch your ROC curve.

And if you assign $1$ to the first $k-1$ clusters and to the random fraction $p\in[0,1]$ of samples in the $k$th cluster, you get the upper bound to the knapsack problem. With this, you can draw the upper bound for your ROC curve.

Here goes a python example:

import numpy as np
from itertools import combinations, chain
import matplotlib.pyplot as plt
np.random.seed(1)
n_obs = 1000
n = 10

# generate clusters as indices of tree leaves
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_predict
X, target = make_classification(n_samples=n_obs)
raw_clusters = DecisionTreeClassifier(max_leaf_nodes=n).fit(X, target).apply(X)
recoding = {x:i for i, x in enumerate(np.unique(raw_clusters))}
clusters = np.array([recoding[x] for x in raw_clusters])

def powerset(xs):
    """ Get set of all subsets """
    return chain.from_iterable(combinations(xs,n) for n in range(len(xs)+1))

def subset_to_metrics(subset, clusters, target):
    """ Calculate TPR and FPR for a subset of clusters """
    prediction = np.zeros(n_obs)
    prediction[np.isin(clusters, subset)] = 1
    tpr = sum(target*prediction) / sum(target) if sum(target) > 0 else 1
    fpr = sum((1-target)*prediction) / sum(1-target) if sum(1-target) > 0 else 1
    return fpr, tpr

# evaluate all subsets
all_tpr = []
all_fpr = []
for subset in powerset(range(n)):
    tpr, fpr = subset_to_metrics(subset, clusters, target)
    all_tpr.append(tpr)
    all_fpr.append(fpr)

# evaluate only the upper bound, using knapsack greedy solution
ratios = [target[clusters==i].mean() for i in range(n)]
order = np.argsort(ratios)[::-1]
new_tpr = []
new_fpr = []
for i in range(n):
    subset = order[0:(i+1)]
    tpr, fpr = subset_to_metrics(subset, clusters, target)
    new_tpr.append(tpr)
    new_fpr.append(fpr)

plt.figure(figsize=(5,5))
plt.scatter(all_tpr, all_fpr, s=3)
plt.plot(new_tpr, new_fpr, c='red', lw=1)
plt.xlabel('TPR')
plt.ylabel('FPR')
plt.title('All and Pareto-optimal subsets')
plt.show();

This code will draw a nice picture for you:

The blue dots are (FPR, TPR) tuples for all $2^{10}$ subsets, and the red line connects (FPR, TPR) for the pareto-optimal subsets.

And now the bit of salt: you did not have to bother about subsets at all! What I did is sorted tree leaves by the fraction of positive samples in each. But what I got is exactly the ROC curve for the probabilistic prediction of the tree. This means, you cannot outperform the tree by hand-picking its leaves based on the target frequencies in the training set.

You can relax and keep using ordinary probabilistic prediction :)


which application domains are not well suited to deep learning? Deep learning seems to be the new cool thing in AI/machine learning and it works well in many domains, but I want to know- what are the specific application areas where deep learning is not the best approach and what is the reason for that?

Has some evaluation been made?

Is there a particular class of problems?

If so- what solutions are superior in solving that task?

Deep learning is generally not well suited for domains that have only very small training datasets. This includes fields like medicine where, for example, patient data can only be obtained through expensive and time consuming clinical trials. Clinical trials will often contain only several hundreds records.

By design, deep learning models have many parameters (e.g., millions), and like with any machine learning algorithm, you're likely to see overfitting when the number of parameters greatly exceeds the number of training records.


Number of features vs. number of samples : if small sample size is sufficient, why take large number of samples? As a newbie, I am a little confused. I have a dataset for binary classification with 11 features and 102 sample data. I have seen in most places (e.g., kaggle competions), the dataset may have hundreds of thousands of data samples for tens of features. On the other hand, this paper says (at least for LDA classifier) optimal number of features is n-1 for a sample size n. My question is, if small no. of samples is enough (or even optimal), why care for larger samples? What am I missing here?

Bounds on the needed amount of samples are very common in PAC learning.
When you define a concept class you can compute a minimal size set of sample that will enable learning.
However,

More samples will allow improving accuracy
More samples will enable learning more complex concepts, that might fit your data better.
As @Emre wrote, real life data sets usually are not clean as in PAC learning. The concept class is not given to you, the data has noise and a given distribution is not guaranteed.

Showing that a classifier can be learnt with a small amount of data is great.
It is a big advantage of the learner.
However, more data usually helps and if it helps more than expected for such a classifier, it is possible that the classifier requirements doesn't hold.


Exploratory Data Analysis and selecting good predictor variables ? In what way would exploratory data analysis aid in feature selection, other than to preprocess the data ? Say, if a bivariate analysis was conducted for each predictor variable w.r.t. the target variable, in what way would this help with feature selection, if possible ?

This is an interesting but broad question.

Imagine PCA. You yse it for exploring the data embedded in lower dimensional space but the first $n$ principale components are also used as the features (after projection of data on them).

Or you use correlation analysis and remove (deselect) features with high correlation with an existing feature.

You calculate the variance of each feature abd low variances tell you that there is no infirmation in this feature.

You inspect feature distributions according to target to determine how much they contribute to the prediction.

And of course much more ...


Choosing One vs All and One vs One for Multiclass SVM What consideration do I need to take when choosing between One vs All or One vs One algorithm for Multiclass SVM?

The main consideration is the number of classes, assume you have $N$ different classes: "one vs all" will train one classifier per class, so $N$ classifiers in total. For a given class $c_i$ the classifier assumes samples with $c_i$ as positives and the rest as negatives. Obviously, this leads to imbalanced datasets, moreover, in "one vs all" the outputs must be calibrated to produce comparable results.

In "one vs one" you train a separate classifier for each different pair of classes, so $\frac{N(N - 1)}{2}$, obviously this is computationally expensive.

For more details see this: 1 and 2.


Convnet training error does not decrease I'm training a convoluted neural net to drive a toy car, and no matter what I do the training accuracy does not increase beyond 30-35%, which is where it starts when the convnet is randomly initialized. What's strange is that a much simpler model, a neural network with a single hidden layer and no convolution, does substantially better, consistently getting accuracy of 65-75%. I've been working on this project for over a year and feel like I've tried everything to make the convnet better. What am I doing wrong?

Notes:

Dataset contains 150,000 records, but since my video feed generates 20 frames per second and since frames don't change much from second to second, it's more like 150,000 / 20 = 7,500 unique records.
Three equally proportioned classes: turn left, go straight, turn right
Both the convnet and simple NN use Tensorflow's AdamOptimizer. The simple net does well with 1e-5 and 1e-4, but the convnet doesn't do well with any value, I've tried 1e-2 all the way through 1e-6
Simple NN uses the sigmoid activation function, the convnet is all relu activations
Both models read the same data. I have a single sampling and data augmentation class that's shared among all my models, so my data isn't bad because the simple net works fine on the same inputs
My convnet can't even overfit on the training data, so it's not data size that's the issue, in my opinion
Overfitting doesn't seem to be a problem with either model: training and validation sets tend to have similar performance, give or take 5%
I'm using initial = tf.truncated_normal(shape, stddev=0.1) to initialize all weights
I'm using initial = tf.constant(0.1, shape=shape) to initialize all biases
It could be specifically the convolution that's the problem, since a shallow single-hidden-layer convnet did poorly but a two-hidden-layer fully-connected neural net with no convolution gets around 65% accuracy

Convnet Notes (all yield same poor results):

Batch normalization
Max pooling
50% drop-out probability
Various stride sizes
Various depths of layers: I've tried 2-5 convent layers, multiple 2-4 fully connected layers. I've had as few as 3 layers and as many as 9. Convnet layers have between 32 and 64 neurons, fully connected layers have between 32 and 512 neurons.
3D convolution (took too much memory and caused my GPU to crash)
1x1 convolutions
The one thing I haven't tried is transfer learning, and I hope to only use that as a last resort since the simple net works fine

Code:

Simple neural net
Deep Convnet

In your convnet code, you compute the cross entropy manually:

cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y_conv), reduction_indices=[1]))
train_step = tf.train.AdamOptimizer(1e-5).minimize(cross_entropy)

If you do that, you may run into numerical stability issues. Instead, you should use

tf.nn.softmax_cross_entropy_with_logits()

If using tensorflow version 1.8 or later, use

tf.nn.softmax_cross_entropy_with_logits_v2()

and weep at the state of tensorflow's API.

See this issue for more information: https://github.com/tensorflow/tensorflow/issues/2462


How to calculate precision at K and NDCG for ranking algorithms I am ranking a filtered item list as per user's metadata and historical behaviour.

Now how to calculate metrices like precision at K?

One approach could be -
Divide historical data in training and test dataset(e.g. 90% and 10%). Now pick all the items for a user from test data and randomize the order and pass it to the ranking algorithm.

Now compare actual items for the user from test data to the ranked output.

Precision at k (precision@k) requires labeled data. Each item needs to be labeled as relevant or not relevant for the given user.

$$ precision@k = \frac{Number\ of\ relevant\ @k}{Number\ of\ recommended\ items\ @k} $$

Labeling data is independent of train/test split. Train/test split is useful to avoid overfitting and for estimating a model's ablity to generalize.


What is the difference between SGD classifier and the Logisitc regression? To my understanding, the SGD classifier, and Logistic regression seems similar.
An SGD classifier with loss = 'log' implements Logistic regression and loss = 'hinge' implements Linear SVM. I also understand that logistic regression uses gradient descent as the optimization function and SGD uses Stochastic gradient descent which converges much faster. But which of the two algorithms to use in which scenarios? Also, how are SGD and Logistic regression similar and how are they different?

Welcome to SE:Data Science.

SGD is a optimization method, while Logistic Regression (LR) is a machine learning algorithm/model. You can think of that a machine learning model defines a loss function, and the optimization method minimizes/maximizes it.

Some machine learning libraries could make users confused about the two concepts. For instance, in scikit-learn there is a model called SGDClassifier which might mislead some user to think that SGD is a classifier. But no, that's a linear classifier optimized by the SGD.

In general, SGD can be used for a wide range of machine learning algorithms, not only LR or linear models. And LR can use other optimizers like L-BFGS, conjugate gradient or Newton-like methods.


Is it possible for a DCGAN to do regression? What are some examples of this? I'm currently a student doing some machine learning projects, and I want to use generative adversarial networks to train some data to discern for example, how old someone is. The intended output is a regression continuous output in term of months.

I see the DCGANs and GANs are targeting mainly classification problems. Can they be used to solve regression problems?

Thank you.

The answer is yes.  There is a paper Intra-class Variation Isolation in Conditional GANs (https://arxiv.org/pdf/1811.11296.pdf) discussing it.

They propose a "C-GAN that is able to learn realistic models with continuous, semantically meaningful input parameters".  They actually cover generating images of people at different ages as well.


Choosing a model for dataset with categorical variables I have a question about the type of model which I should use for a dataset I have.

The dataset has a total of 7 independent variables and 1 dependent variable which I need to predict.

Out of the 7 input variables, 6 of them are categorical and 1 is a date column. Now I have encoded the categorical columns using label encoding and converted them into numerical values. Now I’ve used a simple linear regression model on this dataset and achieved a normalized RMSE value of 0.11.

If I want to improve this accuracy, how do I go about doing it? How can I derive upon the kind of models that I can use considering the data set I have?

The data is mostly about forecast revenues for each product group. That's why I have those categorical columns which represent the hierarchy of the product groups.

I know you ask about the model choice here, but it is worth to discuss about your input data first. Data with many categorical features is still an active research; so it is not that straightforward. I suggest you first look at this very similar post, where I discuss some techniques to convert categorical variables to numerical values.

Since you provide little information about your categorical variables, for example how many levels each categorical variable have or how you do label encoding (just out-of-the-box method?) it is hard to give better guidelines. Generally speaking though, label encoding is mostly used for encoding  target variables (dependent variable) if it is categorical, while you mention you use label encoding for other 6 independent variables (inputs). Here your target is rather numerical values (regression), so I would go beyond label encoding to convert your independent categorical variables.

Image the following very simplified scenario (just for educational purposes) to understand why label encoding is not a good idea where there is more than 1 independent categorical variables:

   Color   Year
0    Red   2010
1   Blue   2011
2  Green   2012

Now label encoding this would converting this to (naively speaking):

  Color  Year
0   1    1
1   2    2
2   3    3

There ways like constructing your own dictionary for each categorical variable and values associated to its levels while label encoding to prevent this obvious mistake. Apart from that, since you hardly go beyond RMSE value of 0.11, one could think of your input values (maybe not encoded properly!) and choice of your model.

Maybe try using this code for handling your independent categorical variables. Please note here, as a general rule of thumb, avoid using one-hot encoding although it is widely used in XGBoost (explained in the post). Maybe try "Target Encoding"!

Now speaking of the choice of model:

Have you tried regularization (L1, L2)?
Have confident you are about the independency of your independent variables? In simple regression you are assuming this holds, but may it is not which is easily the case that your independent variables are correlated. Maybe you should analyze this first using confusion matrix (see here for a brief tutorial)
Have you tried gradient boosting decision trees (GBDT) for regression?
Note1: If you want to go with GBDTs, do not leave that other 1 independent variable alone. You said you have 6 independent variables out of 7. I assume the other one is continuous numerical values? If that so, you have to handle that properly in GBDTs like binarizing (put in bins).
Note2: You may try either XGBoost, or Catboost, and a nice tutorial using Catboost. Each has its pros and cons. For the former hyperparameter tuning could be challenging and there is no ways to automatically account for categorical variables but have larger community. While the latter is recently released (mid last year), it is much easier in terms of hyperparameter tuning (often defaults works great), and there are ways to automatically account for categorical features without explicitly encode them, but less support and community is smaller.
Note3: The good thing about GBDT is that you do not need to worry about the correlation of features per se. Algorithm automatically ignore the coupling of strongly correlated features.
Note4: You may easily overfit, although there are built-in regularization. Catboost for example have a interactive learning curve for training and validation to control overfitting (see the last tutorial).

I strongly believe taking care of your inputs together with a proper choice of algorithm would boost your RMSE error.


Tattoo Image Recognition - Should I Crop Training Data Background I am trying to train a neural network to detect objects within a tattoo. I couldn't find any existing labeled dataset so I need to manually create and label my own. I only understand the basics of machine learning and was going to start by using TensorFlow's Object Detection (https://github.com/tensorflow/models/tree/master/research/object_detection)

For my labeling I am stuck between two options. Using https://scontent-ort2-1.cdninstagram.com/vp/85794872ffd8b8166b1dd559fcf718b0/5C90FA77/t51.2885-15/e15/11326797_987161747982140_1156040219_n.jpg as an example:

Approach 1:
Create a simple bounding box around each object in the tattoo: Multiple boxes: 1 around the rose, the clock and the words. The box for the clock however may bleed into the roses, if the tattoo was more complicated and compact it would bleed a lot (Bleed as in parts of the rose are in the clock's bounding box).

Approach 2:
Cut out each object exactly so there is no bleeding and every object is exactly what it is and fill the negative space with black.

I understand that background noise is important. When the NN is training against many images it detects that the background isn't significant and can pick up the features of just object being detected, however, since all my images are tattoos which are on people's bodies the background for everything is just skin tone. I am afraid if I don't have an exact depiction of the tattoo object it will not train accurately especially since every tattoo is different.

I am not sure which approach to take or if I should go a different directions. Thanks

It is really hard to give an answer on what would work best, as Deep Learning is often based on experimenting with different things. But I would highly recommend you start quick and dirty. This is an advice given at all universities or by the famous Andrew Ng. Just start with the easiest model and then iterate. For example if your simple model doesn't work try to create a more complex pipeline by preprocessing the images as you suggested.

In Deep Learning, time is most often the biggest factor. Don't waste it by trying to create a highly complex model to then just observe that it didn't help to improve your model.

I would like to add, if you decide to add more parts to your pipeline, for example 1) Find the human in the image 2) Find the tattoo 3) Classify tattoo etc. you should take a look at this video as it explains how to improve your pipeline in the most efficient way.


How to determine Nonnegativity in Matrix Factorization? We have information about what the user likes in our app and we want to recommend content to similar users even those who may not have explicitly like a particular content but are similar to those who have explicitly liked that content.

I was planning to use the implicit feedback variation of Alternating Least Squares using Apache Spark to implement a recommender system which will find me the additional users that I can target.

One of the tuning parameters is a boolean flag nonnegative for non-negative matrix factorization.

I will be counting the number of likes per user and tag so I won't have negative values. Can I say nonnegative=true or does it mean something completely different. My understanding is that the nonnegative constraint is for the values inside the two matrices which the algorithm will decompose my original matrix to,  but I don't have the knowledge to know if those values will be non-negative for my scenario.

Ref:
Spark ALS Algorithm. Not exactly since there is another API but good enough for this question.

Yes, the parameter nonnegative is for constraining values for both matrices: user features and item features to not have values below zeros while optimizing using ALS algorithm. It drive which solver to use: CholeskySolver or NNLS(conjugated gradient). Both are implemented in Spark mllib.

The flag is of course connected with the input. If You have ratings below zeros and You directly pump all of Your data to model, the prediction of negatives values while You have nonnegative constrain check will be somehow hard :-)

In Your case, what's important is that You have implicit feedback. So, You should set appropriate flag or use trainImplicit method. The easiest way to find out what works for Your case best with the nonnegative flag is to threat it as another hyper-parameter which needs cross-validation check. And for me, the measurement of the model while validating is a more crucial aspect. If You are trying to do top-N recommendations, go for ranking measures like Recall@N, MRR@N or AUC.


Check If Answer for a Question is Correct by Similarity Let's say in an NLP problem, I have a question and some correct answers to that question (say, 10 correct answers).

Is there a way to get a new answer as input, and "calculate" whether it is correct?

This problem can be solved better if you also include the background text from where the questions have been picked up. Then, firstly train word embeddings on the background text. Further, generate sentence compositionality of the questions as well as their answers using a Recursive Neural Network or some variant. After all these steps you can compute the similarity between compositions of the new answer and given answers to know if it is correct or not.


Can machine learning algorithms predict sports scores or plays? I have a variety of NFL datasets that I think might make a good side-project, but I haven't done anything with them just yet.

Coming to this site made me think of machine learning algorithms and I wondering how good they might be at either predicting the outcome of football games or even the next play.

It seems to me that there would be some trends that could be identified - on 3rd down and 1, a team with a strong running back theoretically should have a tendency to run the ball in that situation.

Scoring might be more difficult to predict, but the winning team might be.

My question is whether these are good questions to throw at a machine learning algorithm.  It could be that a thousand people have tried it before, but the nature of sports makes it an unreliable topic.

There are a lot of good questions about Football (and sports, in general) that would be awesome to throw to an algorithm and see what comes out. The tricky part is to know what to throw to the algorithm.

A team with a good RB could just pass on 3rd-and-short just because the opponents would probably expect run, for instance. So, in order to actually produce some worthy results, I'd break the problem in smaller pieces and analyse them statistically while throwing them to the machines.

There are a few (good) websites that try to do the same, you should check'em out and use whatever they found to help you out:

Football Outsiders
Advanced Football Analytics

And if you truly want to explore Sports Data Analysis, you should definitely check the Sloan Sports Conference videos. There's a lot of them spread on Youtube.


Good explanation for why regularisation works I am trying to understand regularisation for logistic regression currently, but I am not sure I get it. I understand the issue of overfitting when there are relatively too many features, and I get that you would like to limit the impact of some of these superfluous features, but you are doing in regularisation is impacting all features.

And this brings me to my next point, which is that by putting a limit on weights, you can still construct the same hyperplane. As long as you have the same ratio between the weights $w_i$ for $i \in \{1,2,..,n\}$, the "angle" of the hyperplane would still be the same, and then you can simply "adjust" it's position by the intercept $w_0$.

Many lecturers and authors use the complex polynomial example with additional terms $f(x_i)$ that make the decision boundary non-linear to illustrate that using regularisation will make this boundary more linear, and thus less prone to overfitting, but then again, you could plot $f(x_i)$ on a separate axis, making decision boundary a hyperplane, and then you could fit the exact same decision boundary with smaller weights, as long as the ratios are conserved.

Thus, if you can make the exact same hypothesis with smaller sum of squares of weights $\sum w_i^2$ (that is with smaller regularisation term), what is the point of regularisation? Or in other words, as there obviously exists empirical evidence showing that regularisation works, how does it work? Is there a good proof for it, or some good (and less vague) intuition?

The "same angle hyperplane" does not have the same cost. It is the same decision boundary as you describe it, but perpendicular distances to it are larger wrt the norm of the weights. In effect with higher weights in the same ratio (i.e. without any regularisation effect), the classifier will be more confident in all of its decisions. That means the classifier will be more sensitive to getting as many observations as possible in the training set on the "right" side of the boundary. In turn this makes it sensitive to noise in the observations.

Your estimated probability for being in the positive class is:

$$p(y=1|X) = \frac{1}{1+e^{-W^TX}}$$

This includes $w_0$ and fixed value 1 for $x_0$. If you take the midpoint, decision line where $W^TX$ is zero (and the output is at the threshold value 0.5), that defines your decision hyperplane in $X$ space.

When $W$ has the same factors but a larger norm with $w_0$ compensating to make the same hyperplane, then $X$ values that were on the decision hyperplane still give threshold value of 0.5. However, $X$ values away from the hyperplane will deviate more strongly. If instead of 0 you had $W^TX=1.0$ and doubled the weights keeping the same hyperplane, you would get $W^TX=2.0$ for that example. Which changes your confidence from 0.73 to 0.88.

The usual cost function without regularisation for logistic regression with example vectors $X_j$ and targets $y_j$ is:

$$J = - \sum_{\forall j} y_jlog(\frac{1}{1+e^{-W^TX_j}}) + (1 -y_j)(1  - log(\frac{1}{1+e^{-W^TX_j}}))$$

The cost is more sensitive to distances from the hyperplane for larger weight values. Looking your example for the imaginary item (with 0.73 or 0.88 confidence), when the categorisation is correct (i.e. y=1) the score would improve by 0.19 for that example if the weights doubled. When the categorisation is wrong (y=0) then the score would worsen by 0.81. In other words for higher weights, with same weight ratio, the same miscategorisations are punished more than correct categorisations are rewarded.

When training, weights will converge to specific balanced weight vector for the minimum cost, not to a specific ratio that forms a "best decision hyperplane". That's because the hyperplane does not correspond to a single value of the cost function.

You can demonstrate this effect. Train a logistic regression classifier - without any regularisation to show it has nothing to do with that. Take the weight vector and multiply by some factor e.g. 0.5.Then re-train starting with those weights. You will end up with the same weights as before. The cost function minimum clearly defines specific weight values, not a ratio.

When you add regularisation, that changes the cost and how the weights will converge. Higher regularisation in effect makes the classifier prefer a boundary with lower confidence in all its predictions, it penalises "near misses" less badly because the weights are forced down where possible. When viewed as a hyperplane, the boundary will likely be different.


Dataframe has no column names. How to add a header? I am using a dataset to practice for building a decision tree classifier.

Here is my code:

import pandas as pd
tdf = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/breast-cancer-wisconsin.data', sep = ',', header=0)
tdf.info()

The column has no name, and i have problem to add the column name, already tried reindex, pd.melt, rename, etc.

The column names Ι want to assign are:

Sample code number: id number
Clump Thickness: 1 - 10
Uniformity of Cell Size: 1 - 10
Uniformity of Cell Shape: 1 - 10
Marginal Adhesion: 1 - 10
Single Epithelial Cell Size: 1 - 10
Bare Nuclei: 1 - 10
Bland Chromatin: 1 - 10
Normal Nucleoli: 1 - 10
Mitoses: 1 - 10
Class: (2 for benign, 4 for malignant)

Thanks,

For any dataframe , say df , you can add/modify column names by passing the column names in a list to the df.columns method:
For example, if you want the column names to be 'A', 'B', 'C', 'D'],use this
df.columns = ['A', 'B', 'C', 'D’]
In your code , can you remove header=0? This basically tells pandas to take the first row as the column headers . Once you remove that , use the above to assign the column names


How to prepare the varied size input in CNN prediction I want to make a CNN model in Keras which can be fed images of different sizes. According to other questions, I could understand how to set a model, like Input =(None,None,3). However, I'm not sure how to prepare the input/output datasets.
Concretely, now I want to combine the datasets with (100,100) and (240,360).
However, I don't know how to combine these datasets.

Conventionally, when dealing with images of different sizes in CNN(which happens very often in real world problems), we resize the images to the size of the smallest images with the help of any image manipulation library (OpenCV, PIL etc) or some times, pad the images of unequal size to desired size. Resizing the image is simpler and is used most often.

As mentioned by Media in the above answer, it is not possible to directly use images of different sizes. It is because when you define a CNN architecture, you plan as to how many layers you should have depending on the input size. Without having a fixed input shape, you cannot define architecture of your model. It is therefore necessary to convert all your images to same size.


Feature Selection and Outlier Detection How does feature selection impact outlier detection and also, removing outliers impact feature selection?
It could be a basic question. However, just to know the boundaries, I asked. Thanks in advance.
I have gone through the following:Feature selection and outlier order

Outlier detection is part of data preprocessing and used to remove some of the rare events but it could happen that rare events are important to us like fraud detection in that case it becomes important and so we can't do outlier detection beforehand.
In that case we do various approaches like undersampling of majority events or oversampling of minority class. For various approaches refer this
You could also look into KPCA. Having said that there is no particular solution it all depends upon your dataset.


Recurrent Neural Net (LSTM) batch size and input I am working in Keras to build LSTM models. I understand that setting STATEFUL=FALSE means that the different batches are treated as independent when training the model.

Suppose I want to build a model that uses 6 weeks of hourly temperature observations in order to predict tomorrow's 24 hourly temperatures. I have been using a batch size of 24, meaning each day of training data gets used independently by the LSTM when training. But this means that I am not allowing the RNN to pick up longer correlations across days, and only within days, correct? So should I instead be using a batch size of 6*7*24=168? Would this larger batch size allow for the LSTM to pick up longer range correlations across the 6 week training period?

Secondly, I have been using the previous day's observations $y_{(t-24)}$ as predictors for the current day's temperature (at the same time period) $y_t$. Is this a valid approach, or is this redundant with the long term correlations that the LSTM will be looking for?

Batch sizes are different to series length. Batch size refers to the number of observations, and are typically defined when one is performing mini batch stochastic gradient descent.

If you are doing N mini batch SGD, you need to ensure the inputs to your network are a N x 168 tensor. Batch sizes are normally 32-128 observations.

WRT your second question, the approach is fine, it's just different. See which works better.


does xgboost's eval_metric changes the loss function being optimized? I'm using xgboost with the reg:logistic objective. As far as I understand, that means that I'm trying to optimize the log-likelihood loss function of my model, and that each boosting step would be in the gradient direction of this function (w.r.t all the examples I've given).

Does changing the eval_parameter to auc affect any of the above? Or is it just affecting what's measured on my validation set examples every 10 steps of boosting?

Chaging eval_metric does not affect the fitted model in any way. You have to interact with objective to do this.


How do I deal with additional input information other than images in a convolutional neural network? I try to convert a game state of a board game into the input for a convolutional neural network. A convolutional neural network is useful because the players have to place items on the board, and the convolutional neural network can take advantage of that spatial structure. Therefore I can describe the board well with a binary feature plane for each player (1 if there is an item on the board and 0 if not).
The players can not only place objects but also collect cards. A player can own a maximum of 19 of five different card types. How many cards a player has and what type they are, is important information for the neural network, but I cannot describe this with another feature plane. This has nothing to do with spatial structure. So how do I give the convolutional neural network such additional information, for example that the player has 6 cards of type A, as input?
There are also "places" where a player can place his items. These places have a number (so to speak how good the place is) from 2 - 12. I can describe these places again with a feature plane. But I wonder if the network distinguishes well enough between two numbers like 6 and 7? I could imagine that it can distinguish much better between ones and zeros.

I think there are three questions here:
How to incorporate non-spatial information into the network?
When combining different information modalities, a typical approach is to do it at the internal representation level, that is: the point where you lose the spatial information (normally with a flatten operation) after the convolutions. You can have your extra information be processed by an MLP and the result be combined with  the representations obtained by the convolutional layers by concatenating both.
How to represent the cards as input to the network?
In order to represent the card a user has, you can represent them as discrete elements (i.e. tokens), just like text is usually handled in neural networks. This way, you can use an embedding layer, which would receive as input the index of the card. As the user can have any number of cards, you could use an LSTM. In order to represent the "end of the card collection", you can have a special token, and yet another one to represent "padding", which would be useful to create minibatches with different number of cards.
How to represent the places?
You should decide if these are better represented as discrete or continuous values. Or maybe just try both options and choose the best performing one. For continuous features, you could add, as you suggested, another feature place. For discrete features, you would just have an embedding layer, and then concatenate the output to the other channels.
Update: Some clarifications:

A "token" is a term used in NLP to refer to a value which is discrete, that is, the number of values it can take is finite, normally small. In your case, the different values that a card token can take is 19. Usually, we refer to tokens by the index they occupy in the list of all possible values.

In order to represent discrete values in neural networks, we normally represent each different value as a fixed-size vector.

An embedding table is just a table with the fixed-size vectors used to represent your discrete elements. The embedding layer is normally the first in the network architecture. It receives as inputs token indexes and outputs their associated vectors. The entries of the embedding table are updated during the backpropagation process.

You don't "concatenate an MLP with a convolutional network", you concatenate their outputs. Specifically, once the output of the last convolutional layer is computed, you normally "flatten" it, meaning you remove the spatial information and just place the output tensor elements in a single-dimension vector. That vector is what you concatenate with the output of the MLP, which is also a single-dimension vector (apart from the minibatch dimension).


GBM vs XGBOOST? Key differences? I am trying to understand the key differences between GBM and XGBOOST. I tried to google it, but could not find any good answers explaining the differences between the two algorithms and why xgboost almost always performs better than GBM. What makes XGBOOST so fast?

Quote from the author of xgboost:

Both xgboost and gbm follows the principle of gradient boosting.  There are however, the difference in modeling details. Specifically,  xgboost used a more regularized model formalization to control over-fitting, which gives it better performance.
We have updated a comprehensive tutorial on introduction to the model, which you might want to take a look at. Introduction to Boosted Trees
The name xgboost, though, actually refers to the engineering goal to push the limit of computations resources for boosted tree algorithms. Which is the reason why many people use xgboost. For model, it might be more suitable to be called as regularized gradient boosting.

Edit: There's a detailed guide of xgboost which shows more differences.
References
https://www.quora.com/What-is-the-difference-between-the-R-gbm-gradient-boosting-machine-and-xgboost-extreme-gradient-boosting
https://xgboost.readthedocs.io/en/latest/tutorials/model.html


Are machine learning models good at autoregressive forecasting of time series? AR, MA, GARCH and VAR models are standard for autoregressive prediction, which is the forecasting of a variable using its own historical lags for features/regressors.

Support vector regression (SVR), random forests, boosting and ANNs are typical machine learning algorithms.

How good are these at time series forecasting? Are any of them able to forecast farther into the future beyond 1-day forecast horizons by using historical lags for input features? Do any sources give a theoretical basis for machine learners' applicability to autoregression? And to assess their out-of-sample performance, given that the task is autoregressive prediction of time series, is bootstrap aggregation or cross-validation more appropriate?

Raw regression is usually not as good performance-wise but there are variations of said algorithms to accommodate time series more effectively like Long Short Term Memory (LSTM) neural networks, Gated Recurrent Units Neural nets among others (GRU) as well as other non neural net approaches and closed to graph theory like Dynamic Bayesian Networks.


Classify Customers based on 2 features AND a Time series of events I need help on what should be my next step in an algorithm I am designing.

Due to NDAs, I can't disclose much, but I'll try to be generic and understandable.

Basically, after several steps in the algorithms, I have this:

For each customer that I have, and events that they do during a month, during first steps I have clustered the events into several categories (each customer will have the events separated into categories that go from 1 to x being x between 1 to 25, generally the first categories have more density of events than the others).

For each category and customer I have created a time series aggregating the events of the month per hour (getting patterns of when these events are being done). Also I am using a couple of normalizing variables based on the number of days over a month (30 days) that the guy performs at least one event, and the number of days with at least one event over the total of days with at least one event (aggregating all clusters). The first one gives me a ratio of how active the customer is during the month, and the second one weights the category against the others.

The final table looks like this

|*Identifier*|  *firstCat* | *feature1* | *feature2*  |   {      *(TIME SERIES)*   }

CustomerID  |  ClusterID |  DaysOver30 | DaysOverTotal | Events9AM Events10AM ...

 xx | 1 | 0,69 |  0,72 |  0,2   0,13   ...

 xx | 2 | 0,11 |  0,28 |  0,1   0,45   ...

 xy | 1 | 0,23 |  0,88 |  0,00  0,60   ...

 xy | 2 | 0,11 |  0,08 |  1,00  0,00   ...

 xy | 3 | 0,10 |  0,04 |  0,40  0,60   ...

The time series variables are the percentage over the total of events per day on each specific category (this means that per each row adding up all variables should be 1). The reason of doing it like that is because for example a time series with events 0 0 0 1 0 and 1 1 1 2 1 are completely different, and standardizing to normal would give similar results. And due to high skew between different categories, I check the values on the time series independently with the others.

What I need to do now is to identify these categories (remember, they can be from 1 to x being x any number from 1 to 25) into 3 tags: tag A, tag B and None of Them. Looking at these variables I can manually identify which tag they belong to, and the idea is to identify manually as much as I can and use any classifier algorithm to learn from that and identify all of them.

My idea was to use multiple logistic regressions on the table, but all the variables of the time series are correlated (since they are a linear combination of each other), so I thought I better use a clustering algorithm only over the time series using euclidean distance to categorize the different patterns and use the result and the other two normalizing variables in the logistic regression.

The other concern that I have is that this approach takes each row independently from the others, and in theory, for each customer there should be only 0 or 1 tag A, 0 or 1 tag B and the remaining of them should be None (another tip is that normally Tag A and B are between first categories, because is highly dependent on the normalizing features (if days over total is High, there is a high possibility that the row is either A or B, depending on the Time Series Pattern).

Edit: This is no longer a concern, I will just perform two different logistic regressions, one for Tag A or Other and another for Tag B or another, with the result probabilities I can select only the best of each.

The dataset is enormous and the final algorithm needs to be applied using SQL (on Teradata), but for getting the coefficients of the logistic regression, or the centers of the clustering I get a sample and use R.

I know it has been a while since the question has been posted but for future readers I propose here another solution for the problem of classifying time-series of proportional data i.e., vectors of proportions.

Hidden Markov Models (HMMs) are broadly used for time-series classification. The original implementations were developed for discrete and Gaussian data but since then, they have been extended to more probability distributions.

Among them, the Dirichlet, the generalized Dirichlet, and the Beta-Liouville distributions which are perfectly suited to work with proportional data.

Usually, a model is trained for each category ("Tag" in this question) from time-series for which the category is known. When a new time-series has to be classified, its likelihood with respect to each model is computed. The new time-series is usually assigned to the category leading to the highest likelihood.

Here are the reference papers for these models presenting them more deeply and providing all the equations needed for implementation:

Dynamical Dirichlet Mixture Model, Research Report from IDIAP, L. Chen, D. Barber, J.-M. Odobez, April 2007
Proportional data modeling with hidden Markov models based on generalized Dirichlet and Beta-Liouville mixtures applied to anomaly detection in public areas, E. Epaillard, N. Bouguila, Pattern Recognition 55, pp 125-136, 2016

For the additional two features mentioned in the question, HMMs for mixed data have also been developed in the following research paper:
Hybrid hidden Markov model for mixed continuous/continuous and discrete/continuous data modeling, E. Epaillard, N. Bouguila, MMSP, pp. 1-6, 2015.

These models are heavier to deploys than regression models but have the advantage of being discriminative and generative models and to be able to generalize well when some variability is present in the processed data.


Gradient flow through concatenation operation I need help in understanding the gradient flow through a concatenation operation.

I'm implementing a network (mostly a CNN) which has a concatenation operation (in pytorch). The network is defined such that the responses of passing two different images through a CNN are concatenated and passed through another CNN and the training is done end to end.

Since the first CNN is shared between both of the inputs to the concatenation, I was wondering how the gradients should be distributed through the concatenation operation during backprop? I'm not an expert on backprop and this is the first time I'm tinkering with a custom backward implementation so any pointers would be helpful.

I can provide more details if you guys need it.

For concatenation, the gradient values during back propagation split to their respective source layers. There is no direct interaction between gradients in either of the source layers.

The layer immediately after the concatenated layer does interact with both networks, and it will have some weight parameters that multiply outputs from network A and some that multiply outputs from network B. There will not be any parameters that multiply outputs from both layers (unless you are forcing them to be the same through weight sharing, but that won't be the case if for example you are stacking features from both starting networks).

The only issue you might have is clearly identifying which parameters link to each original network. That is an implementation detail, so you would need to share your code so far in order to debug that if it goes wrong.


How to run tensorflow model twice before computing the loss I want to compute a loss function which uses output of the network twice on different inputs. For example as hypothetically,

first_output = model(first_input)
second_output = model(second_input)
loss = mean_absolute_error(first_output, second_output)

How to achieve this in tensorflow or keras?

So, as in comment:

Put as many inputs in the batch as you want, and the run prediction on that batch. That way, you could use all outputs to calculate the loss function.

Keras

In Keras framework, if you want to calculate the mean_absolute_error as you state in your question, you could use predefined mean_absolute_error loss. Also, to have two inputs as you state in your question, specify the batch size of the model to 2.


What is the intuition behind larger number of samples are better for statistics? It may be well-known that when we take statistics, we essentially need a large number of samples. Because I am taught this fact before studying the math, I have been here without exploring the reason. What is the intuition behind larger number of samples are better for statistics?

Intuitively we can imagine statistics as trying to see an image using only finitely many pixels/samples. If you have more pixels/samples and they're well-distributed, then the image is more clear.


Find columns with numeric values, but stored as string I need to find the columns in data frame, which has numeric values and are stored as string.

data_set = pd.DataFrame({"Number":["1","2","3","4","5"], "Char":["A","B","C","D","E"]})
data_set.dtypes

In above code, column "Number" has numeric values, but stored as string. I have to iterate through columns and convert it to int or float.

I can do it by taking column name and convert it.

data_set["Number"] = data_set["Number"].astype(int)

What i need is to do it dynamically. Dynamically means, iterating through columns and changing it.

I think this does what you want.

for column in data_set:
    If isinstace(data_set[column].dtypes,str)
        data_set[column]=data_set[column].astype(int)


Looking for smallest set of rows that form a natural key in a data set I have several sets of text files on hdfs that are exports from relations.  Unfortunately I do not know the structure of the table is, but I do know that each has a multi-part key that defines a row uniquely.  I know through domain knowledge that the key is multi-part (e.g. reporting-date, and item number) and I can identify some columns that are clearly not in the key (e.g. revenue from sale).  What is an effective way to identify potential sets of columns that are natural keys, because they are not duplicated in the observed data?  I can get several days of logs in a few Gig, so python or sql could work.  This seems like an great application for a dictionary, but I am not sure how to approach this.

You could do this a few ways...

write a script that does the following.

pick a table,

get number of rows in the table, can do this as you go. (+=1 in loop)

select several fields(columns) in table that you think may form the key,

create empty Set()

burn through the file row by row, for each row grab the target fields and construct a string key str(field1)+'_'+str(field2)..etc.  Add this key to your set.

when you're done going through all rows of a table, check len(set).  If the key you picked is good then len(set) will equal # rows.

Another way, and this will depend on what you want to do with your data and how often you will access it...is to import it into a sql database.  I usually use mysql. Once your data is in:

select count(*) from table_name;  will give you rows in table

select count(distinct field1, field2, field3) from table;  gives number of distinct combos of field1+field2+field3

if the two selects give the same number then you have a key that works. It may not guarantee a join-key but it will be unique to that table and will help the linking process.

many people highly advocate the use of pandas. If you get the data imported into arrays you can create a dataframe and do some queries similar to SQL.  I'm not super familiar with pandas so I can't give an example.

If it's a one-off, I'd make a python script.  But if you think you'll be exploring the data further, spending the time to put the data in a database could be useful.


Why we take $\alpha\sum B_j^2$ as penalty in Ridge Regression? $$RSS_{RIDGE}=\sum_{i=1}^n(\hat{y_i}-y_i)^2+\alpha\sum_{i=1}^nB_j^2$$
Why we are taking $\alpha\sum B_j^2$ as a penalty here? We are adding this term for minimizing variance in Machine Learning Model. But how this term minimizing variance. If I add suppose $e^x$ or any increasing function then it also minimizing the variance. I want to know how this term minimizing the error

Expanding my comments into an answer.
Ridge regression is a by definition an augmentation of Least Squares method, especialy for problems where the data may be highly correlated (there is what is called multi-colinearity).
Lets assume the dependent variable is $y$ and $x_i$ are the independent variables.
Then assume the true mapping between $X$ and $y$ is:
$$y = \sum_i \beta_i x_i$$.
Ordinary Least Squares (OLS) assumes the mapping is given by $\hat{\beta_i}$ coefficients which solve the least squares problem:
$$\hat{B} = (X^TX)^{-1}X^TY$$
The above formula indeed minimises the square error $RSS = \sum_i(y_i-\hat{y_i})^2$
However, if some columns in $X$ are correlated then ordinary least squares may not provide the best solution.

Least Square Method finds the Best and Unbiased Coefficients
You may know that least square method finds the coefficients that best
fit the data. One more condition to be added is that it also finds the
unbiased coefficients. Here unbiased means that OLS doesn’t consider
which independent variable is more important than others. It simply
finds the coefficients for a given data set. In short, there is only
one set of betas to be found, resulting in the lowest ‘Residual Sum of
Squares (RSS)’. The question then becomes “Is a model with the lowest
RSS truly the best model?”.
Bias vs. Variance
The answer for the question above is “Not really”. As hinted in the
word ‘Unbiased’, we need to consider ‘Bias’ too. Bias means how
equally a model cares about its predictors. Let’s say there are two
models to predict an apple price with two predictor ‘sweetness’ and
‘shine’; one model is unbiased and the other is biased.
First, the unbiased model tries to find the relationship between the
two features and the prices, just as the OLS method does. This model
will fit the observations as perfectly as possible to minimize the
RSS. However, this could easily lead to overfitting issues. In other
words, the model will not perform as well with new data because it is
built for the given data so specifically that it may not fit new data.
The biased model accepts its variables unequally to treat each
predictor differently. Going back to the example, we would want to
only care about ‘sweetness’ to build a model and this should perform
better with new data. The reason will be explained after understanding
Bias vs. Variance. If you’re not familiar with the bias vs. variance
topic, I strongly recommend you to watch this video that will give you
insight.

In order to affect bias-variance tradeoff ridge regression adds a small offset in the estimation process, ie:
The magnitude of the $B$ coefficients must satisfy a criterion (the "ridge") eg:
$$|B|_2^2 \le C^2$$
Then this constraint can be added in the Least Squares formula (eg via a Lagrange Multiplier).  This constraint affects the bias-variance tradeoff and can be helpful in cases such as highly correlated data or data with unequal significance. Then the new $RSS$ is similar to what you ask in the question.
Again the square error is measured along with constraints on the coefficients in order to affect bias-variance tradeoff.
Reference:

Ridge Regression for Better Usage
Bias-Variance Tradeoff


What are the online platforms that cater to freelance data scientists looking for work? For example, something along the lines of the professionalism of:

https://www.kaggle.com/

Stackoverflow has a jobs page. This link is to a search with keyword data and option Remote.
freelandr offers you to `Land your next Freelance Job'.
Here are two specialised job agencies, although I don't know if they relay freelance opportunities.

  We are well aware of the data scientist being the sexiest job of the 21st century. JamieAi harnessing this fact connects technical talent with data-oriented jobs organizations of all types and sizes. The start-up firm has combined AI insights and human oversight to reduce hiring costs and eliminate bias.  Also, third party recruitment agencies are removed from the process to boost transparency and efficiency in the path to employment. Another example is Woo.io, a marketplace for matching tech professionals and companies.
  https://hub.packtpub.com/25-startups-machine-learning-differently-2018/

R-users specialises in the language R.


Handling rows with 2 lines of data My dataframe looks like this :

there are some rows ( example : 297) where the "Price" column has two values ( Plugs and Quarts) , I have filled the Nans with the previous row since it belongs to the same Latin Name. However I was thinking of splitting the Price Column further into two columns with Names "Quarts" and "Plugs" and fill the amount, 0 if there is no Plugs found and same with Quarts.

Example :

Plugs | Quarts
0 | 2
2 | 3
4 | 0

Can someone help me with this?

The code below should achieve the desired results:

import pandas as pd
import numpy as np
import re

df2 = pd.DataFrame([[1, 'plugs: $3.00'],
                    [4, np.NaN],
                    [7, 'quarts: $3.00']],
                    columns=['name', 'price'])

df2

    name          price
 0     1   plugs: $3.00
 1     4            NaN
 2     7  quarts: $3.00

def price(x):
    rprice = re.search('(plugs:|quarts:)\s*\$([\d\.]*)', x)
    if rprice == None:
        return ('','0')
    else:
        return rprice.groups()

df2.fillna("", inplace=True)
df2['price'].map(lambda x: price(x))
df2['Plugs'] = df2['price'].map(lambda  x: float(price(x)[1]) if price(x)[0] == 'plugs:' else 0)
df2['Quart'] = df2['price'].map(lambda  x: float(price(x)[1]) if price(x)[0] == 'quarts:' else 0)

df2 (with new columns below)

   name          price  Plugs  Quart
0     1   plugs: $3.00    3.0    0.0
1     4                   0.0    0.0
2     7  quarts: $3.00    0.0    3.0

Some Notes:
I used regular expressions to extract the type and cost, and I replaced the NA with blank text.


Improve results of a clustering I'm a beginner and I'm trying to do a clustering of a multi-sentence text, but my results are horrible. Any tips for me to improve my result?

import pandas
import pprint
import numpy as np
import pandas as pd
from sklearn.cluster import KMeans
from sklearn.metrics import adjusted_rand_score
from sklearn.feature_extraction.text import TfidfVectorizer

dataset = pandas.read_csv('text.csv', encoding = 'utf-8')

comments = dataset['comments']

comments_list = comments.values.tolist()

vetorize = TfidfVectorizer()
X = vetorize.fit_transform(comments_list)

clusters_number = 6
model = KMeans(n_clusters = clusters_number, init = 'k-means++', max_iter = 300, n_init = 1)

model.fit(X)

centers = model.cluster_centers_
labels = model.labels_

clusters = {}
for verbatim, label in zip(verbatim_list, labels):
    try:
        clusters[str(label)].append(verbatim)
    except:
        clusters[str(label)] = [verbatim]
pprint.pprint(clusters)

#Top terms for cluster
print("Top termos par cluster:")
ordem_centroides = model.cluster_centers_.argsort()[:, ::-1]
termos = vetorizar.get_feature_names()
for i in range(clusters_number):
    print ("Cluster %d:" % i,)
    for ind in ordem_centroides[i, :10]:
        print (' %s' % termos[ind],)
        print()

I have many different themes present in various clusters. I pre-processed my data (stopwords, lowercase, I removed the punct...). But still I have 'like to cancel order' in one cluster and 'love cancel order' in another. When, in fact, the ideal is to join all 'cancel order' in a single cluster.

The very obvious tip I would say is that K-means is not the algorithm you use for clustering text data in general. The nature of text data is a bit more complicated than structured data on which K-means is one of basic but still working algorithms (of course it also depends on the way you model the text i.e. how you convert a text dataset to a of numbers). Let me propose two things with one hint:

Hint

Document Clustering is also referred to as Topic Modeling. So you really need to have a look at this as I assume you didn't yet (according to using k-means for this problem). Now we see two standard algorithms for topic modeling but I strongly recommend you to see other algorithms as well.

NMF

Non-negative Matrix Factorization is a well-known decomposition method for non-negative matrices like TF-IDF or other variants of Bag-of-Words. You can simply apply it to your problem using SKLearn.

LDA

or Latent Dirichlet Allocation (don't get confused by Linear Discriminant Analysis). This is a pretty standard algorithm for topic modeling and implementations can be found in SKLearn, Gensim, NLTK, Spacy and other NLP/ML libraries.

Hope it helped! Good Luck :)


is it possible to use the train result as another feature and retrain? is it possible to use the train result as another feature and retrain?

for example I make prediction with classification and
add this result to the table and train xgboost?

Yes but there are two issues:

Is it really useful? If the prediction is just added to the features on which it's based, it's unlikely to improve performance. However there are cases where this is useful, e.g. stacked learners.
You need to split the training set into two parts t1 and t2:

t1 is the regular training set used to produce the first model
t2 is the data used to predict the new "feature" and train the second model with this new feature.

It would be a bad idea to use the same training set for the two models because the predictions used in the second model would be obtained on the training set, so they would be unrealistically good.


How to check two images(one is original image and other one captured by mobile) similarity using deep learning? My problem statement is - In my project, original image of product is stored in database. Now whenever any person uploading that product's camera pic(for internal audit process) then I need to verify whether this uploaded image is similar of original image or not. So can anyone help me on this, how can I achieve this verification automated using deep learning or tensor-flow ?

There are a few approaches to this off the top of my head:

1. Trivial case

You can first check if the images are identical by simply using numpy's functions, either numpy.array_equal(A, B) or numpy.allclose(A, B). Where A and B are the two images stored as numpy arrays.

To check if images are semantically similar, you need to compute a similarity measure. First let me back up a bit...

2. Latent representations

In deep learning, we can represent data such as images, voice recordings, text etc. in a latent space. This is an unobservable ("not real") space, where we attempt to capture the essence of the data. For words, we might create a 300-dimensional vector that should encode the information that a given word represents. For images, we could use an Auto Encoder.

The idea is to distill the images into a matrix that really contains all the core information in an image. This would be your latent representation of an image,  and this is where you compare images. Here is a schematic of such a model, where the "bottleneck" in the middle is where you have achieved a dense representation of an input image:

The model is trained end to end, by trying to expand that condensed middle section into an array the same as the input array (input image).

You create the latent, encoded representation of two images to be compared by running them just through the first half of your trained model (i.e. just the encoder), and calculate a distance or similarity metric between the resulting arrays. You could perhaps use a simple cosine-similarity (of flattened arrays) or even just the $L_2$ norm (sum of squared differences).

Hopefully that all provides enough guidance and keywords for you to find relevant tutorials :-)