Classification vs Regression Algorithms - Should exists algorithms only for Classification and/or Regression Dummy question:

There exists algorithms that should only be used for Classification or Regression problems?

For example, should Random Forest should only be apply on Classification problems and Neural Networks for Regression problems?

Thank you for your time :)

Well you can think of a regression as being a classification with a very large number of ordinal classes, therefore algorithms can be used for both.
Indeed you can build a regression with a random forest, and a neural network can be used both as a regressor or a classifier depending on the activation function of its output layer.


pandas.isna() vs pandas.DataFrame.isna() I've seen the two documentation pages for pandas.isna() and pandas.DataFrame.isna() but the difference is still unclear to me.  Could someone explain the difference to me using examples?

They call the same underlying method, so there is no functional difference.

Calling the dataframe member function is preferred for OOP patterns, but there are many redundancies/aliases in pandas and python in general.

In case you are curious, here is how the source code breaks down (it is a mess).

The DataFrame (pandas/core/frame.py) method is simply:

def isna(self):
    return super().isna()

Where DataFrame extends NDFrame (implemented in pandas/core/generic.py). NDFrame subsequently invokes:

def isna(self):
    return isna(self).__finalize__(self)

Which was imported here:

from pandas.core.dtypes.missing import isna, notna

In pandas/core/dtypes/missing.py:

def isna(obj):
    return _isna(obj)

The _isna function is later aliased as _isna = _isna_new because there is a deprecated method _isna_old(obj).

The _isna_new(obj) function then performs the logic operations:

def _isna_new(obj)
    if is_scalar(obj):
        return libmissing.checknull(obj)
    # hack (for now) because MI registers as ndarray
    elif isinstance(obj, ABCMultiIndex):
        raise NotImplementedError("isna is not defined for MultiIndex")
    elif isinstance(obj, type):
        return False
    elif isinstance(
        obj,
        (
            ABCSeries,
            np.ndarray,
            ABCIndexClass,
            ABCExtensionArray,
            ABCDatetimeArray,
            ABCTimedeltaArray,
        ),
    ):
        return _isna_ndarraylike(obj)
    elif isinstance(obj, ABCGeneric):
        return obj._constructor(obj._data.isna(func=isna))
    elif isinstance(obj, list):
        return _isna_ndarraylike(np.asarray(obj, dtype=object))
    elif hasattr(obj, "__array__"):
        return _isna_ndarraylike(np.asarray(obj))
    else:
        return obj is None

Ultimately, the DataFrame method passes itself as a parameter to the same function that you call with pandas.isna().


Will the features in the image (edge, color, etc.. ) impacts on the performance of the spherical k-means? I am very new in Machine learning, I recently implemented the spherical k-means, but finally I found a interesting point from the result. I used four datasets, they are minst, cifar-10, fashion-minst, and svhn. I was following the paper Learning Feature Representations with K-means (Coates & Ng, 2012) https://www-cs.stanford.edu/~acoates/papers/coatesng_nntot2012.pdf. I did not finish the deep learning yet.

See the following image, I tried the max pooling and average pooling to see if there any difference, but I found an interesting point, the dataset with the clear feature (edge, color, etc. of the object in the image ) has the high performance, the dataset without clear feature has low performance.

See the four pictures on the bottom, I extracted one image from each dataset. I ordered them according the performance. I tried search online, but I did not get any useful information about my question.

I am not sure whether the point is correct or not. Could any one please explain this to me. Thanks!

Yes it is true that in these images follow the trend you pointed, but that is not the only reason for the drop in accuracy:

Image 1 is pretty simple, and binary so it is easier samples, not because of clarity but of simplicity.
Image 2 is a bit more complex, but it is still binary and can be learned easily.
Image 3 is simple, but it is noisy and (as you pointed out) does not have strong features, since you're not dealing with CNNs you should pre-process this kind of dataset by binarization or image sharpening after noise removal (if you do it before you will amplify noise). This probably was taken with bad light conditions.
Image 4 this is really complex image, it is on color and until now I couldn't figure out what it is lol

Well, you are not wrong in your idea, but taking conclusions that way may mislead you on the future. There are other reasons (as pointed above) for your findings. So:

Create a thesis (like you did)
Test your thesis by trying to change the observed phenomena (e.g. pre-process images to find stronger features)
Check if that improved your results, create a new thesis and repeat until satisfied with results


Multidimensional Dynamic Time Warping Implementation in Python - confirm? I believe that I implemented MDTW in python here but I don't know if I did it correctly. The results seem intuitive. Can someone look at this code and tell me if you see anything wrong?

A lot of the papers I read on the subject kinda go over my head, but this just takes the mse of all column vectors in matrix s1 and s2 at each time point inside a given window w instead of single dimensional dynamic time warping which just takes the mse error of each value between two vectors and finds the smallest one.

I use panda's Panel, which is essentially a 3D version of a DataFrame.

Also, after I have this, I want to do clustering with some multidimensional time series. Thoughts on which clustering algorithm to run? Kmeans? Hierarchical? I'll start by building a dendrogram at least.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

def MDTWDistance(s1, s2, window=10, num_columns=1):
    DTW={}

    w = max(window, abs(len(s1)-len(s2)))

    for i in range(-1,len(s1)):
        for j in range(-1,len(s2)):
            DTW[(i, j)] = float('inf')
    DTW[(-1, -1)] = 0

    for i in range(len(s1)):
        for j in range(max(0, i-w), min(len(s2), i+w)):
            #print "Finding Distance of", s1.loc[i], s2.loc[j]
            dist= mdist(s1.loc[i], s2.loc[j], num_columns)
            #print "Dist", dist
            #print i, j, dist
            DTW[(i, j)] = dist + min(DTW[(i-1, j)],DTW[(i, j-1)], DTW[(i-1, j-1)])

    return np.sqrt(DTW[len(s1)-1, len(s2)-1])

def mdist(a, b, num_col):
    dist = 0
    for col in range(num_col):
        #print "Finding Distance of", a[col], b[col]
        dist = dist + (a[col]-b[col])**2
    return dist

x=np.linspace(0,50,100)
ts1=pd.Series(3.1*np.sin(x/1.5)+3.5)
ts2=pd.Series(2.2*np.sin(x/3.5+2.4)+3.2)
ts3=pd.Series(0.04*x+8.0)
ts4=pd.Series(0.048*x+8.6)
ts5=pd.Series(-0.17*x+4.1)
ts6=pd.Series(-0.14*x+4.5)

ts1.plot()
ts2.plot()
ts3.plot()
ts4.plot()
ts5.plot()
ts6.plot()

plt.ylim(-4,12)
plt.legend(['ts1','ts2','ts3','ts4','ts5','ts6'])
plt.show()

timeSeries = pd.Panel({0:pd.DataFrame(np.transpose([ts1, ts2])),
                 1:pd.DataFrame(np.transpose([ts3, ts4])),
                   2:pd.DataFrame(np.transpose([ts5, ts6]))
                  })

print "0 and 1:",MDTWDistance(timeSeries[0], timeSeries[1],window=10, num_columns=2)
print "0 and 2:",MDTWDistance(timeSeries[0], timeSeries[2],window=10, num_columns=2)
print "1 and 2:",MDTWDistance(timeSeries[1], timeSeries[2],window=10, num_columns=2)

The output is as follow:

0 and 1: 89.354619036
0 and 2: 58.8268328591
1 and 2: 133.434513377

With the graph:

EDIT: I found the distance using an MDTW package in R with the following code:

import rpy2.robjects.numpy2ri
from rpy2.robjects.packages import importr
rpy2.robjects.numpy2ri.activate()

# Set up our R namespaces
R = rpy2.robjects.r
DTW = importr('dtw')

# Generate our data
idx = np.linspace(0, 2*np.pi, 100)
template = np.cos(idx)
query = np.sin(idx) + np.array(R.runif(100))/10

for i,j in ([0,1], [0,2], [1,2]):
    print "From", i, "to", j, R.dtw(np.transpose(np.array(timeSeries[i])), np.transpose(np.array(timeSeries[j])), keep=True).rx('distance')[0][0]

which output:

From 0 to 1 186.623310713
From 0 to 2 119.769089068
From 1 to 2 272.849560995

So these numbers are about double the distances I have... so I'm not far off, but there's something wrong with what I'm doing. Halp.

Could the difference be caused by the step pattern you are using? Dtw in R defaults to the symmetric2 step pattern. Explanation of the step pattern from Toni Giorgino's paper:

  symmetric2 is normalizable, symmetric, with no local slope constraints. Since one diagonal step costs as much as the two equivalent steps along the sides, it can be normalized dividing by $N+M$ (query+reference lengths).

I thought this might be the problem after reading Comparing Dynamic Time Warping in R and Python.

If your problem is the same try adding step=symmetric1 to your R.dtw arguements.


Conditionally replace values in column with value from another column I have a 5 column dataframe where one column has some NaN's. My goal is to replace these NaN's with the corresponding value in another column.

So if my DF looks like this:

YearBuilt     LastModified

1993          2004

1994          NaN

1995          2004

I would want to replace the NaN with the corresponding YearBuilt, which is 1994 in this case.

for i in range(df.shape[0]):
    if(df['LastModified'][i].isnull()):
        df.loc[i,'LastModified']==df.loc[i,'YearBuilt']


Hypertuning XGBoost parameters XGBoost have been doing a great job, when it comes to dealing with both categorical and continuous dependant variables. But, how do I select the optimized parameters for an XGBoost problem?

This is how I applied the parameters for a recent Kaggle problem:

param <- list(  objective           = "reg:linear",
                booster = "gbtree",
                eta                 = 0.02, # 0.06, #0.01,
                max_depth           = 10, #changed from default of 8
                subsample           = 0.5, # 0.7
                colsample_bytree    = 0.7, # 0.7
                num_parallel_tree   = 5
                # alpha = 0.0001,
                # lambda = 1
)

clf <- xgb.train(   params              = param,
                    data                = dtrain,
                    nrounds             = 3000, #300, #280, #125, #250, # changed from 300
                    verbose             = 0,
                    early.stop.round    = 100,
                    watchlist           = watchlist,
                    maximize            = FALSE,
                    feval=RMPSE
)

All I do to experiment is randomly select (with intuition) another set of parameters for improving on the result.

Is there anyway I automate the selection of optimized(best) set of parameters?

(Answers can be in any language. I'm just looking for the technique)

Whenever I work with xgboost I often make my own homebrew parameter search but you can do it with the caret package as well like KrisP just mentioned.

Caret

See this answer on Cross Validated for a thorough explanation on how to use the caret package for hyperparameter search on xgboost.
How to tune hyperparameters of xgboost trees?

Custom Grid Search

I often begin with a few assumptions based on Owen Zhang's slides on tips for data science P. 14

Here you can see that you'll mostly need to tune row sampling, column sampling and maybe maximum tree depth. This is how I do a custom row sampling and column sampling search for a problem I am working on at the moment:

searchGridSubCol <- expand.grid(subsample = c(0.5, 0.75, 1),
                                colsample_bytree = c(0.6, 0.8, 1))
ntrees <- 100

#Build a xgb.DMatrix object
DMMatrixTrain <- xgb.DMatrix(data = yourMatrix, label = yourTarget)

rmseErrorsHyperparameters <- apply(searchGridSubCol, 1, function(parameterList){

    #Extract Parameters to test
    currentSubsampleRate <- parameterList[["subsample"]]
    currentColsampleRate <- parameterList[["colsample_bytree"]]

    xgboostModelCV <- xgb.cv(data =  DMMatrixTrain, nrounds = ntrees, nfold = 5, showsd = TRUE,
                           metrics = "rmse", verbose = TRUE, "eval_metric" = "rmse",
                           "objective" = "reg:linear", "max.depth" = 15, "eta" = 2/ntrees,
                           "subsample" = currentSubsampleRate, "colsample_bytree" = currentColsampleRate)

    xvalidationScores <- as.data.frame(xgboostModelCV)
    #Save rmse of the last iteration
    rmse <- tail(xvalidationScores$test.rmse.mean, 1)

    return(c(rmse, currentSubsampleRate, currentColsampleRate))

})

And combined with some ggplot2 magic using the results of that apply function you can plot a graphical representation of the search.

In this plot lighter colors represent lower error and each block represents a unique combination of column sampling and row sampling. So if you want to perform an additional search of say eta (or tree depth) you will end up with one of these plots for each eta parameters tested.

I see you have a different evaluation metric (RMPSE), just plug that in the cross validation function and you'll get the desired result. Besides that I wouldn't worry too much about fine tuning the other parameters because doing so won't improve performance too much, at least not so much compared to spending more time engineering features or cleaning the data.

Others

Random search and Bayesian parameter selection are also possible but I haven't made/found an implementation of them yet.

Here is a good primer on bayesian Optimization of hyperparameters by Max Kuhn creator of caret.

http://blog.revolutionanalytics.com/2016/06/bayesian-optimization-of-machine-learning-models.html


Finding the most important questions from a questionary-> results Lets assume that I have 5000 articles and I create the TF-IDF of these articles.

Now I ask som people to answer 30 questions and I create the TF-IDF of these answers from the IDF of the articles and use the cosinus similarity to find the 20 closest articles per person with a weight of 1 for each answer. People can mark these 20 articles as relevant to her/him or not relevant(0 or 1).

So, which approach would you follow to see which questions are the most important in their choices? In my mind it seems like feature selection problem. Or not? Any recommendation?

Generally, what I want to do is reduce the number of questions from 30 -> 5 and I have to find which one are the most important questions.

Edit

As @null_9 mentioned in a comment, I am not sure which is the best way of providing the most relevant articles: 1) Summing up the similarity scores from all the answers and give the highest, or 2) find N closest articles per each question and sample 20 of them and provide to the user.

If selecting the most important and relative questions is the question and you choose the option:

option 1.) find the closest document to each answer(30 docs) and after the documents are validated relative or not. You will have the results of each person's evaluation (Xpersons x 30 scores). Now order the questions in descending order from most scored question to less scored one.

option 2.) Summing up all the answers and finding 20 docs. After the validation of relativeness by the persons, it is not possible to find the important features(questions) because they are summed up and the information is already lost. From this result you could only find which person validates closer to your model among all, which is not the aim of the work.


Is there a clustering algorithm that works with only pairwise distances as input? My data are places for which I know all pairwise travel times (='distances'), and I want to cluster those places minimising the total pairwise travel time inside a cluster.

K-means can't be used because it's centroid-based and the 'distance' is a duration that is provided, not computed thanks to coordinates
DBSCAN can't be used because it excludes outliners and I want to include each places in a cluster (not 100% sure for this one)

Bonus: a Java library would be much appreciated

Yes, this is a classic Graph-Based Clustering problem in which each location is a node and you have the distances between them. Forgetting about concept of graphs and graph-based algorithm which might be complicating, I directly jump to your answer.
The most well-known algorithm is Spectral Clustering. There are tones of tutorial out there and it is well implemented in all programming languages including Java.
I briefly explain but do not Panic if you are not familiar with Math terms. They are pretty intuitive and easy. You just need to follow a good tutorial.

Calculate the Similarity Matrix (often referred to as Affinity Matrix in literature). As you have distance matrix you have several ways to do it. Simplest starts from saying $sim = \frac{1}{dist}$ till more sophisticated ones which use Gaussian Kernels for calculating similarity.
Calculate the Laplacian Matrix out of similarity matrix.
Calculate eigenvectors and eigenvalues in order to embedd your data points into their eigen-space (like exactly what we do in PCA)
Use a simple clustering algorithm like K-Means to cluster points in that space


How to classify a new email as spam/not spam? I am working on a small exercise for determining if an email is spam or not. My dataset is the following:
                       Email                                                   Spam
0   Hi, I am Andrew and I want too buy VIAGRA                                   1
1   Dear subscriber, your account will be closed                                1
2   Please click below to verify and access email restore                       1
3   Hi Anne, I miss you so much! Can’t wait to see you                          0
4   Dear Professor Johnson, I was unable to attend class today                  0
5   I am pleased to inform you that you have won our grand prize.               1
6   I can’t help you with that cuz it’s too hard.                               0
7   I’m sorry to tell you but im sick and will not be able to come to class.    0
8   Can I see an example before all are shipped or will that cost extra?        0
9   I appreciate your assistance and look forward to hearing back from you.     0

where 1 means spam, 0 not spam. What I have tried is the following:
#Tokenization

def fun(t):

# Removing Punctuations
remove_punc = [c for c in text if c not in string.punctuation]
remove_punc = ''.join(remove_punc)

# Removing StopWords
cleaned = [w for w in remove_punc.split() if w.lower() not in stopwords.words('english')]

return cleaned

So I applied the function: df['Email'].apply(fun). Then I converted the text into a matrix as follows:
from sklearn.feature_extraction.text import CountVectorizer
mex = CountVectorizer(analyzer= fun).fit_transform(df['Email'])

and split the dataset into train and test:
X_train, X_test, y_train, y_test = train_test_split(mex, df['Email'], test_size = 0.25, random_state = 0)

I applied a classifier (I would apply the logistic regression to determine if an email is spam or not, but I have currently only used Naive Bayes:
from sklearn.naive_bayes import MultinomialNB

classifier = MultinomialNB()
classifier.fit(X_train, y_train)

Finally I applied the classifier first to the train set, then to the test set:
from sklearn.metrics import classification_report,confusion_matrix, accuracy_score
from sklearn.metrics import classification_report,confusion_matrix, accuracy_score

pred = classifier.predict(X_test)

print(classification_report(y_test ,pred ))
print('Confusion Matrix: \n', confusion_matrix(y_test,pred))
print()
print('Accuracy: ', accuracy_score(y_test,pred))

The code works but I would like to know how to visually see with an example of new email if this has label 1 or 0. For example: if I have a new email 'Hi, my name is Christopher and I like VIAGRA', how could I determine the label/class?
I feel I am missing something or probably I am following a wrong way to demonstrate this.
My question is the following:
Given this new email: Hi, my name is Christopher and I like VIAGRA, how can I see if this is spam or not? I have thought about classification but probably my approach is wrong. I would like to have something like:
Email                                        Spam
...
Hi, my name is Christopher and I like VIAGRA 1

as this is very similar to the email  'Hi, I am Andrew and I want too buy VIAGRA' (if included in the train set or correctly predicted in the test set).
Maybe what I want to do requires only tf-idf algorithm or a different approach.
Any advice will be appreciated.

I modified your code so the code runs as a block and is setup to predict new data:
import string

from nltk.corpus import stopwords
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report,confusion_matrix, accuracy_score
from sklearn.naive_bayes import MultinomialNB

#  Define training data
df = pd.DataFrame(data={'Email': [
"Hi, I am Andrew and I want too buy VIAGRA",
"Dear subscriber, your account will be closed",
"Please click below to verify and access email restore",
"Hi Anne, I miss you so much! Can’t wait to see you",
"Dear Professor Johnson, I was unable to attend class today",
"I am pleased to inform you that you have won our grand prize.",
"I can’t help you with that cuz it’s too hard.",
"I’m sorry to tell you but im sick and will not be able to come to class.",
"Can I see an example before all are shipped or will that cost extra?",
"I appreciate your assistance and look forward to hearing back from you.",],
'Spam': [1, 1, 1, 0, 0, 1, 0, 0, 0, 0]})

def fun(text):
    # Removing Punctuations
    remove_punc = [c for c in text if c not in string.punctuation]
    remove_punc = ''.join(remove_punc)

    # Removing StopWords
    cleaned = [w for w in remove_punc.split() if w.lower() not in stopwords.words('english')]

    return cleaned

# Create a vectorizer object to enable both fit_transform and just transform
vectorizer = CountVectorizer(analyzer=fun)
X = vectorizer.fit_transform(df['Email'])

X_train, X_test, y_train, y_test = train_test_split(X, df['Spam'], test_size = 0.25, random_state = 0)

classifier = MultinomialNB()
classifier.fit(X_train, y_train)

pred = classifier.predict(X_test)

print(classification_report(y_test ,pred ))
print('Confusion Matrix: \n', confusion_matrix(y_test,pred))
print()
print('Accuracy: ', accuracy_score(y_test,pred))

Here is how to predict on new data:
# Given a new email
new_email = "Hi, my name is Christopher and I like VIAGRA"

# Apply the same preprocessing steps and transformation
X_new = vectorizer.transform([fun(new_email)])

# Predict new email with already trained classifier
classifier.predict(X_new)


Perform bootstrapping of an ordinary linear regression model, using B=100 bootstrap resamples of my dataset, and getting RMSE So Im studying machine learning through R, and Im working with the boston data set from the library MASS. I am practicing bootsrapping. I already carried out analysis to determine how ,many  distinct data points on average are drawn from the sample to make up a bootsratp resample, using B=100 resamples of the dataset. Next I would like to do two things- perform boostrapping of an ordinary linear regression model using B=100 resamples of the data set again and use the OOB samples to predict RMSE. And aslo use bootstrapping of a ridge regression model with 100 Bootstrap samples to predict RMSE and then compare my answers. I am having difficulty adapting the code from what I have already done into this. Does anyone have any idea?

The following code will run Ordinary linear regression and ridge regression on B=100 different samples of Boston data set and calculate the RMSE on the test set for all 100 different test sets and store the RMSE for ordinary linear regression in olr_rmse_all  and for ridge in ridge_rmse_all.

You can do any type of analysis on RMSE vectors.

I calculated mean and standard deviation of the RMSE of B=100 for Ordinary Linear regression and Ridge regression

library(MASS)
library (ridge)

data1 = Boston

RMSE = function(actual, predicted){
  sqrt(mean((actual - predicted)^2))
}

test_perc = 0.3
B = 100

olr_rmse_all = c()
ridge_rmse_all = c()

for (i in c(1:B)){

  cat("running for sample = ", i, '\n')
  train_rows = sample(nrow(data), (1-test_perc)*nrow(data))
  test_rows = setdiff(c(1:nrow(data)) , train_row)

  Boston.train <- Boston[train_rows, ]
  Boston.test <- Boston[test_rows, ]

  olr_model  <- lm(medv ~ ., data = Boston.train)
  ridge_model <- linearRidge(medv ~ ., data = Boston.train)

  test_predicted_olr = predict(olr_model, newdata = Boston.test)
  test_predicted_ridge = predict(ridge_model, newdata = Boston.test)

  test_actual = Boston.test$medv

  rmse_test_olr = RMSE(test_actual, test_predicted_olr)
  rmse_test_ridge = RMSE(test_actual, test_predicted_ridge)

  cat(paste("olr_rmse: " , rmse_test_olr,  "ridge_rmse: ", rmse_test_ridge, sep = " "), '\n')

  olr_rmse_all = c(olr_rmse_all, rmse_test_olr)
  ridge_rmse_all = c(ridge_rmse_all, rmse_test_ridge)

}

#-- Mean of RMSE----

olr_rmse_all_mean = mean(olr_rmse_all)
ridge_rmse_all_mean = mean(ridge_rmse_all)

#-- Standard Deviation of RMSE----

olr_rmse_all_sd = sd(olr_rmse_all)
ridge_rmse_all_sd = sd(ridge_rmse_all)

Hope, this answer your question.

There is package for bootstrapping in R, name "boot", you can also use that. I do it without package for more understanding.

reference: https://www.statmethods.net/advstats/bootstrapping.html


For binary classification, which is best Random Forest or Neural networks? I had to perform a binary classification, and from the beginning I started thinking about using the Random Forest classifier. But now I'm thinking, if using a neural network would've not been better.

So my question is, which model you would've chosen.? and if not the Neural Networks, what are the tasks the most suited for the Neural Network and why.?

There is no best model for binary classification. Instead it is better to focus on what kind of data you have and let that guide your choice of model.

Decision tree based models usually perform very well when dealing with regular tabular data of categorical and numerical features. Mainly random forest and gradient boosting (LightGBM and XGBoost). If you have very few samples a SVM can also be a good option.

Neural networks shine on more unstructured data like images and text. Time series are a bit more nuanced and often multiple models can work well.


What type of neural network could emulate a binary to HTML conversion tool? I've got a problem, which I thought could be solved by using a neural network:

I've got a binary file and a tool that converts that file into a readable html file (probably a text file as well). How the tool is converting that file is a secret and it is too complex to reverse engineer (>100000 lines of assembler for this function) and the binary files are too big and complex to interpret for a human in a feasible time. My idea was to teach a neural network with those binary files and the known output to train the network, how to convert these files. The binary files differ extremely in size(from a few kb up to serval mb).

I've got some basic knowledge of neural networks but I'm not sure where to start to search for the right approach. A friend of mine (with a bit more basic knowledge) said this is definitely doable and suggested a seq2seq model in tensorflow, but after some research I'm not sure if this is the right one for my problem.

In short your friend is correct, seq2seq is a reasonable match to the problem.

However, from your numbers this could be too complex to use current machine learning libraries on.

Despite you calling it "not feasible", you are far better off with the reverse engineering effort in my opinion.

If you really want to have a try using ML, you could start with some shorter files with the generated output and attempt to train a LSTM-based seq2seq model to match them. It may help, if the HTML is strongly templated and driven by data in the binary file, to reverse-engineer just the templating, and use seq2seq model to derive a more raw data sequence as opposed to HTML.

You should expect to spend several weeks learning how seq2seq works, followed by an unknown amount of time in the attempt to apply it to your problem. You will need a large number of sample files to train your model - hundreds of thousands, maybe millions. There might be exceptions if the generated HTML is very repetitive/formulaic, or if short segments of binary data can also generate short segments of HTML (so you could train on smaller segments of the larger files).

If any of this would break the terms and conditions for use of the software, I advise you not to attempt either reverse engineering or machine learning.


Find regularization loss component When Keras prints the validation loss, it prints the total loss value (regularization loss + fitting loss).

How can I get the two separately? I'm specifically looking to obtain value for regularization loss.

By default, most loss functions only return one scalar value. Some will also give you a 'per-sample loss in the batch' value. If you want other loss categories, you can use the add_loss() to add, compute and keep track of different types of losses in different layers.

Refer to https://keras.io/api/losses/
for more details.


Can I run Orange widgets from normal Python scripts? I'd like to do normal machine learning with Python and Sklearn, but occasionally use the nice graphical widgets or Orange for presentation.

Is this possible?

I don't think Orange was ever intended to be used that way, but if you can convert your data into Orange.data.Table, you might be able to instantiate the widgets you need with it.

See the if __name__ == '__main__' block at the bottom of most widgets for some example.


Is there a benefit to using hadoop with only one node? I just started learning about Hadoop. From what I understand its primary strength is its ability to distribute a task across many nodes. Is there any benefit to using Hadoop with only a single node, other then the potential of expanding to more nodes in the future? What if you have no plans of ever having more then one node? Or is there an expectation that if you are going to use Hadoop you are also going to be setting up a cluster of many nodes?

If you are testing or building something that will eventually be run on a distributed cluster, yes it make sense. If you're playing around to learn, yes. Otherwise no I don't see much value.


samples for different objects with unique labels Lets say i have a samples taken from veterinary hospital , one of my feature will be the type of  the animal and some other features such as fever , size,symptom etc.. , my labels are the medicine given to that animal.

if each medicine is unique to that type of animal (Medicine A should be given to animal A , and there are no Animal B that taken Medicine A) . What will be the cons and pros to building one classifier for the whole data set vs split classifier for each animal (since there wont be valid generalization between animals)

Have run into this problem, when we wanted to run with patient-based deep learning models or individual observations (the same patient could have come several times) based deep learning models.

In your case, they could be analogous to Animal-based model vs general-one-for-all model which has all the animals.

Pros of animal-based:

If you are going to build an animal-level classifier, it certainly is going to generalize better when compared to one-model for every animal.

Cons of animal-based:

If you don't have enough data for one particular animal, you can't help out much, for that case. This is slightly advantageous as well, I can't say for veterinarian examples, but if you are going to use a general-one-for-all model it might misclassify the prediction as well (because of lack of data for that particular example).

I would suggest you try both. There is nothing certain in this world. Only hunches and guesses from past experiences.

Hope this helps.


Face Recognition using Eigenfaces and SVM I am new to machine learning. I want to develop a face recognition system using scikit-learn. This is the example given in the tutorials of scikit-learn.
I am not getting how the input is being provided to the program. How should I load a particular image and make my program run to predict the label for that?

Take a look at the code that you linked to:

# Download the data, if not already on disk and load it as numpy arrays

lfw_people = fetch_lfw_people(min_faces_per_person=70, resize=0.4)

fetch_lfw-poeple is a routine that loads the data and is detailed here.

Hope this helps!


Is this attribute numeric or categorical (ordinal)? Help! So I have this dataset I need to perform several techniques on as part of a data mining/machine learning project of some sort in PYTHON. There are a couple of features however, that have me very worried as I don't know whether I whould handle them as Categorical (ordinal to be precise) or as numeric.

The first feature takes integer values from 0 to 9 and it's supposed to represent population percentage of a certain religion (so 0 is 0%, 1 is 1-10%, 9 is 100%)
The second feature represents contibution (money) and also takes values from 0-9 depending on how big the contribution is (so 0 is.. well, 0 , 1 is 1-49 , 9 is 20000+).

How should I handle these features? Are they interval data? Should I one-hot-encode them into dummies or should I leave them as they are now?
I am going to have to perform some clustering and classification techniques on the dataset.
Thank you in advance!

They are not categorical as they have a meaningful ordering that you likely want to use. The first is usable as is as it is roughly fraction religious times 10. Yes it's ordinal but happens to be just a rescaled continuous feature.
The second is ordinal and so you don't quite want to use it as a continuous feature as that doesn't capture the difference in scale that they indicate. It might work OK, but you could also translate each number to the midpoint of corresponding range.


Getting different precisions for same neural network with same dataset and hyperparameters in sklearn mlp classifier I get WAY DIFFERENT results in each run despite using random state for making sure that network outputs same result for same hyper parameters, here is some sample outputs(I've printed the hyper parameters manually to show that they are the same) and I think I may missing something...

{'hidden_layer_sizes': [20, 20, 30], 'activation': 'tanh', 'solver': 'adam', 'alpha': 0.02, 'learning_rate': 'adaptive', 'iteration': 400}
0.8888888888888888

{'hidden_layer_sizes': [20, 20, 30], 'activation': 'tanh', 'solver': 'adam', 'alpha': 0.02, 'learning_rate': 'adaptive', 'iteration': 400}
0.3333333333333333

{'hidden_layer_sizes': [20, 20, 30], 'activation': 'tanh', 'solver': 'adam', 'alpha': 0.02, 'learning_rate': 'adaptive', 'iteration': 400}
0.4444444444444444

{'hidden_layer_sizes': [20, 20, 30], 'activation': 'tanh', 'solver': 'adam', 'alpha': 0.02, 'learning_rate': 'adaptive', 'iteration': 400}
0.7777777777777778

here is some of the code:

mlp = MLPClassifier(random_state=2,
                    hidden_layer_sizes=(20, 20, 30),
                    max_iter=400, alpha=0.02,
                    activation='tanh',
                    solver='adam',
                    learning_rate='adaptive')
mlp.fit(X_train, y_train.values.ravel())

If the initialization of your network is random, and the order in which you feed it the training examples is not fixed, then your training can lead to a different model every time, with different performance. That’s normal, although if you train long enough, you would hope that you often reach a solution of similar quality. So perhaps you are not training long enough. Have you plotted the loss as a function of the training time (in terms of max_iter)? Is your model training until reaching the maximum number of iterations, or is it converging before?

Another possibility is the amount of training data. So repeat the above exercise where you figure out how long you need to train with smaller fractions of your training data. For example, what’s the best you can do with 20% of your training data, with 40%, 60%, and so on? If your best result is still improving significantly when you reach 100%, then maybe you need to get more training data.


Why is my U-matrix visually not separating the classes? I am visualizing the U-matrix generated using a Self Organizing Map codebook to (visually) identify regions of similarity in the data.

Although I would like to use SOM to identify clusters in unlabeled data, I am running it first with some labeled benchmark data sets to make sure that my implementation is bug free.

For Iris data set, I can see a clear demarcation between the classes suggesting that the implementation is correct. The classes have been re-labeled as 1,2,3.

However, when I run the same code on another data set, I can see that SOM was able to map members of different classes together (all the 0s are together and 1s are together and there are two distinct regions), but there is no clear demarcation.

How can I improve the U-matrix or SOM to produce a clear demarcation between the classes?

Is the number of predictor variables in your new dataset large? If so, Euclidean distance can be unreliable due to the curse of dimensionality. You might want to try reducing the dimension of your dataset using Principal Component Analysis or feature selection. Alternatively, you might want to try using a different distance metric, like the cosine similarity.


How to use machine learning to extract product info from the titles of eBay listings Reposting here because someone correctly pointed out it is better suited for here.

So I have a bunch of titles of eBay listings. I want to extract product information from each title, so I can categorize the listings by product since these titles are coming from wide-ranging searches.

The are the following Titles:

5 PCS 3 Digits 0.56" NUMERIC DISPLAY 7 seg segment Common cathode
5 PCS LD-3361AS 3 Digits 0.36" NUMERIC DISPLAY 7 SEG COMMON CATHODE digit
6 x MAN71A Common Anode Red Led 7-Segment Display (6 pcs)
Experiment Board Breadboard Circuit Board ZYJ - 60 White NEW

The format in which I want to extract the data is as follows:

For the 1st listing:

There are 5 pieces
It has 3 digits
It is of size .56"
It is 7 segments
It is numeric
It is cathode
It is a display

For the 2nd listing:

There are 5 pieces
It is of size .36"
It is model LD-3361AS
It has 3 Digits
It is Numeric
It has 7 segments
It is cathode
It is a display

For the 3rd listing:

There are 6 pieces
It is brand MAN71A
it has 7 segments
It is Anode
It is a display

For the 4th listing:

It is New
It is a Breadboard
It is White
It is model ZYJ - 60

The reason I want to learn machine learning to do this is that it would make it very easy to just take those specifics and store it for later use and because it can adapt to any search and not just in a specific category. It would also be able to adapt to different items or formats in the title. I tried this with scikitlearn's kmeans, but it gave me a lot of overlap and just clustered them, rather than extract the details from it. What I want to do is have the program look at the title, identify specifics as what they refer to (i.e., size, color, brand, etc.) and extract the data. Ideally, this would be unsupervised learning, but as I write this, I start to realize that may be impossible to do with this. The part I'm stuck on is what to use to achieve this. Should I use NLTK? One of the classifiers from scikitlearn? Clustering?

Like Toros91 explained, it is a raw data format. You want to convert 5 PCS into There are 5 pieces. How does you know PCS means pieces? You have a link somewhere in your brain that tells this. You have to do the same in programming. Create a dictionary which tells the machine that wherever it encounters PCS in line, PCS means pieces (it can be a model name also, you have to take care of that). The number just before PCS is the value. All this you have to code manually. There is no shortcut here. And of course you have to use NLP to achieve your task.

Pre-Processing your raw data is the hard task, using ML algorithm on a formatted data is easy one.


How to distinguish between different values of a hyperparameter in communication? From Wikipedia:

In machine learning, a hyperparameter is a parameter whose value is
used to control the learning process.

If we go by the definition of parameter in What's the difference between an argument and a parameter?, then a hyperparameter wouldn't be considered a value itself. Rather, it would be the spec for this value (for example, this value is a learning rate, or this value is a dropout). This also seems to agree with the way people use "hyperparameter" in communication.
There are a lot of times where I want to distinguish between particular values of a hyperparameter. Going by the question I linked, I could say something like hyperargument. For example, the hyperargument 0.001 is different than the hyperargument 0.01, even though they are under the same hyperparameter (let's say learning rate). This is what I do in my own codebase for variable names. For example, if I had this dict:
hyperarguments = {
    "learning_rate": 0.001,
    "dropout": 0.25,
}

I would iterate over it like this:
for hyperparameter, hyperargument in hyperarguments.items():
    ...

Is there a more common terminology to emphasize this distinction of values?

TLDR: Basically, you have answered your question with your quote from Wikipedia: the common terminology is "values".

The question on SE SO which you have quoted discusses the terminology from a software development perspective. But there is a difference between the use of the word "parameter" in software development and machine learning. This is how Wikipedia defines parameter from a software development perspective:

In computer programming, a parameter or a formal argument, is a special kind of variable, used in a subroutine to refer to one of the pieces of data provided as input to the subroutine. These pieces of data are the values of the arguments (often called actual arguments or actual parameters) with which the subroutine is going to be called/invoked. An ordered list of parameters is usually included in the definition of a subroutine, so that, each time the subroutine is called, its arguments for that call are evaluated, and the resulting values can be assigned to the corresponding parameters.

Let's make an example:
n_estimators = 100
model = RandomForestRegressor(n_estimators=n_estimators)

From a software development perspective n_estimators is a parameter, 100 is the parameter's value and n_estimators=n_estimators is the argument.
Howevever, in machine learning terminology you'd consider n_estimators a hyperparameter and 100 its value. Going back to your examples I would just do something like this:
hyperparameter_dict = {
    "learning_rate": 0.001,
    "dropout": 0.25,
}

and
for hyperparameter, value in hyperparameter_dict.items():
    ...

Introducing a new terminology which is not common in machine learning (like "hyperarguments") would only cause confusion and not make your code or communication any clearer.


Boruta Python No feature Selected I run Boruta with RandomForestClassifier the previous day on my data (nb features = 36) and got 17/36 confirmed. Now I run it again and there is 0/36 and stop at the 9th iteration. Any idea why this is happening?

%%time
rfc = RandomForestClassifier(n_estimators=200, n_jobs=-1, class_weight= 'balanced_subsample' , max_depth=10)

boruta_selector = BorutaPy(rfc, n_estimators='auto', verbose=2, random_state=1)

boruta_selector_before.fit(X, y)

the current output look  is the follow

Iteration:  1 / 100
Confirmed:  0
Tentative:  36
Rejected:   0
Iteration:  2 / 100
Confirmed:  0
Tentative:  36
Rejected:   0
Iteration:  3 / 100
Confirmed:  0
Tentative:  36
Rejected:   0
Iteration:  4 / 100
Confirmed:  0
Tentative:  36
Rejected:   0
Iteration:  5 / 100
Confirmed:  0
Tentative:  36
Rejected:   0
Iteration:  6 / 100
Confirmed:  0
Tentative:  36
Rejected:   0
Iteration:  7 / 100
Confirmed:  0
Tentative:  36
Rejected:   0
Iteration:  8 / 100
Confirmed:  0
Tentative:  0
Rejected:   36

BorutaPy finished running.

Iteration:  9 / 100
Confirmed:  0
Tentative:  0
Rejected:   36
CPU times: user 1.95 s, sys: 377 ms, total: 2.33 s
Wall time: 2.79 s

My mistake came from the fact that my target variable sample only had one class. To reduce computation time, I took a sample from the main data.


which neural network topology to learn correlations between time series? I have two (or more in principle) 1xN time series, and I would like to train a NN to predict the next value of both. I can arrange them as a 2xN matrix and feed a window from this matrix as input to the NN, but I'm not sure how to structure the NN itself.

I have made a NN with convolutions that can do a pretty decent job with a single series, but I'd like to exploit cross-series correlations. What topology works to let the NN notice correlations between the time series?

It depends a little on what kind of correlations you're looking for. Are you expecting a correlation that is present at each time step/window, or a different level of correlation per time step/window? Are you doing a classification or a regression task? Sometimes predicting the next value involves classification, but I'll assume you're looking for regression for now.

As a starting point, try feeding each of those sequences separately as input to a recurrent neural network (start with a basic LSTM and pare it down if it's overkill). I like your suggestion of overlapping windows.

For example (pseudocode-ish):

series_1 = 1, 2, 3, ..., 100

series_2 = 5, 6, 7, ..., 200

input_1 = Input(series_1,       window_size)

input_2 = Input(series_2, window_size)

layer_1 = LSTM(input_1, input_2)

final_layer = fully_connected(layer_1)


Stacking and Ensembling methods in Data Science I understand that using stacking and ensembling has become popular, and these methods can give better results than using a single algorithm.
My question is: What are the reasons, statistical or otherwise, behind the improvement in results?
I also understand that at a high level that combining these methods will combine the predictions from different algorithms. Each algorithm has its own strengths and weaknesses, but not sure how combining them will actually improve the results.
Here is a simple balanced example (50% of the labels are actually T and 50% are actually F), where I think stacking or ensembling algorithms will still give the same result as the original model. In this case, for the ties (1 F and 1 T), I decided to select T. The same problem would occur if I selected F instead. This is a little more complicated if I used the predict_proba, but think the same problem would occur.

It sometimes is called "ensemble learning" where several "weak learners" make a prediction. These predictions are "combined" by some meta-model. A simplistic approach would be that you just use the majority vote. You can also use logistic regression. You can (and should!) of course check the performance of the stacked model(s) by predicting on test data. Related ensemble techniques are boosting and bagging.
Simplified Example: Assume you want to predict a binary outcome. Say you have two different models, which do not perform extremely well, but perform better than random guessing. Also, assume the models are independent. Suppose each of your models makes a correct prediction with $0.8\%$ probability. Now when you have two models, the possible outcomes are:
Correct, Correct:   $0.8*0.8=0.64$
Wrong, Wrong: $0.2*0.2=0.04$
Correct, Wrong: $0.8*0.2=0.16$
Wrong, Correct: $0.2*0.8=0.16$
You see that the probability that both models are wrong is "only" $0.04$. So if you manage to identify the remaining cases as "correct predictions" by some intermediate/meta (stacking) model, you end up with quite a good result. Essentially you would "boost" the expected accuracy of the model(s) from $0.8$ (single model) to $0.64+0.16+0.16=0.96$ (ensemble) here.


How to calculate precision and recall in an unsupervised ranking method in order to be compared with a supervised one? I am working on an ML supervised learning project for Key-phrase extraction. The evaluation dataset (different from training-validation datasets) contains around 200 phrases labeled as key or not key.

The problem is that I want to compare the results obtained (in terms of precision, recall and f1) via different classifier's algorithms with existing unsupervised methods. These methods are based on ranking and extract the top-k key phrases.

Using the dataset that I mention, what is the proper way to calculate precision and recall for these unsupervised ranking methods?

You first need to decide what your "label" is, and this label will apply to both your supervised and unsupervised models.  Obviously, in a supervised world, knowing your label make training the model and evaluating it easy, so I won't go into that because it seems like you have a good handle.

In the unsupervised world, you'll have to think really hard about the mapping of the unsupervised outputs to the label you've decided to use for evaluation.  A lot of this is going to come down to how you structure that unsupervised problem.  Like, if you're going to be using unsupervised clustering, then maybe you run the model, see what the clusters look like, and see if there's a way to map those clusters the label you're evaluating.

It sounds like you're using unsupervised ranking and extracting top key phrases after the unsupervised ranking.  In this case, you'd probably want to map the possible top key phrases to the labels you want to evaluate.  Then after ranking and key phrase extraction, you can get a mapped "prediction label" based on the key phrases for each prediction.  Of course, you'll likely have a messy collection of "prediction labels" because it's unlikely that each test prediction will output N-key phrases that perfectly match 1 label which means you'll have to come up with a way to round those maps (maybe you take the mode, maybe you round the average, it all depends on what you value most) to a singular prediction label.  From there, doing recall and precision and f1 is straight forward.

I will say though, that trying to bend an unsupervised model to evaluation like a supervised model is only useful if recall, precision, and f1 are the best way to evaluate your model's usefulness in its problem space.  This is very different from evaluating its effectiveness.

This question is: Does this models output fit the use case of the problem best?  When I implement this output to an end user, would they appreciate this model more than another model?  I'd encourage you to, on top of using classic evaluation scores to measure model effectiveness, also find a way to measure or understand model usefulness in the application that you have.


What would be a good loss function to penalize big differences and reward small ones, but not in a linear way? I have an image with the differences between 2 other images. Concentrations of black pixels mean similar regions between the images, whereas, white values highlight differences.

Thus I want a function to provide rewards when the pixels are the same (or relatively similar - a term to weight this "similarity threshold" would be nice) and penalize when the differences are bigger (penalizing more as the differences grow).

A differentiable function is much appreciated.

So in the context of machine learning and this loss function being a way to help train a generator, what kind of function do you recommend or can come up with?

Remember, the ideia is to reward similarities and penalize differences (such that "really different" equates to a bigger loss than "slightly off" or "different").

Thanks in advance to you all!

Square loss (MSE or SSE) does this. Let $y_i$ be an actual value and $\hat{y}_i$ be its estimated value (prediction).

$$SSE = \sum (y_i -\hat{y}_i)^2$$

$$MSE=\dfrac{SSE}{n}$$

Except for numerical issues of doing math on a computer, these are optimized at the same parameter values of your neural network.

The squaring is critical. If a prediction is off by 1 unit, it incurs one unit of loss. If the prediction is off by 2, instead of incurring 2 units of loss, there are 4 units of loss—much worse than being off by 1. If the prediction is off by 3, wow—9 units of loss!

(If you look at statistics literature or some Cross Validated posts, you may see $n-p$ in the denominator of MSE, where $p$ is the number of parameters in the regression equation. This does not change the optimal value but does have some advantages in linear regression, chiefly that it is an unbiased estimate of the error variance under common assumptions for linear regression that you are unlikely to make in a neural network problem.)


Gradient of a function in Python I've defined a function in this way:

def qfun(par):
    return(par[0]+atan(par[3])*par[1]+atan(par[4])*par[2])

How can I obtain the gradient of this function for only some of the elements (par [0:2]) in a specific point? I only find functions with only one "x", so for those cases it is simple, but when your function has more parameters what should I do?

Several options:

You can use the defintion of the derivative to have an approximation....

def f(x):
    return x[0]**2 + 3*x[1]**3

def der(f, x, der_index=[]):
    # der_index: variable w.r.t. get gradient

    epsilon = 2.34E-10
    grads = []

   for idx in der_index:
       x_ = x.copy()
       x_[idx]+=epsilon

       grads.append((f(x_) - f(x))/epsilon)

   return grads

print(der(f, np.array([1.,1.]), der_index=[0, 1]))

If you can solve it analytically, it is better you write the derivative function by yourself
You can also use symbolic programming, like in Matlab, with the library sympy https://towardsdatascience.com/taking-derivatives-in-python-d6229ba72c64
Another way to do it is going for the "differentiable programming" paradigm or "software 2.0"


confusion matrix results in CNN keras I am writing a code to classify images from two classes, dogs and cats. I wrote the below code, but always all the dogs images are classified as cats as shown in the confusion matrix.

Am I missing something in the code?

Note: I tried epochs up to 70, but got same results.

from keras.models import Sequential
from keras.layers import Conv2D,Activation,MaxPooling2D,Dense,Flatten,Dropout
import numpy as np
from keras.preprocessing.image import ImageDataGenerator
from IPython.display import display
import matplotlib.pyplot as plt
from PIL import Image
from sklearn.metrics import classification_report, confusion_matrix

classifier = Sequential()
classifier.add(Conv2D(32,(3,3),input_shape=(64,64,3)))
classifier.add(Activation('relu'))
classifier.add(MaxPooling2D(pool_size =(2,2)))
classifier.add(Conv2D(32,(3,3)))
classifier.add(Activation('relu'))
classifier.add(MaxPooling2D(pool_size =(2,2)))
classifier.add(Conv2D(64,(3,3)))
classifier.add(Activation('relu'))
classifier.add(MaxPooling2D(pool_size =(2,2)))
classifier.add(Flatten())
classifier.add(Dense(64))
classifier.add(Activation('relu'))
classifier.add(Dropout(0.5))
classifier.add(Dense(1))
classifier.add(Activation('sigmoid'))
classifier.summary()
classifier.compile(optimizer ='rmsprop',
                   loss ='binary_crossentropy',
                   metrics =['accuracy'])
train_datagen = ImageDataGenerator(rescale =1./255,
                                   shear_range =0.2,
                                   zoom_range = 0.2,
                                   horizontal_flip =True)
test_datagen = ImageDataGenerator(rescale = 1./255)

batchsize=32
training_set = train_datagen.flow_from_directory('/home/osboxes/Downloads/Downloads/dogs-vs-cats/train/',
                                                target_size=(64,64),
                                                batch_size= batchsize,
                                                class_mode='binary')

test_set = test_datagen.flow_from_directory('/home/osboxes/Downloads/Downloads/dogs-vs-cats/test/',
                                           target_size = (64,64),
                                           batch_size = batchsize,
                       shuffle=False,
                                           class_mode ='binary')
history=classifier.fit_generator(training_set,
                        steps_per_epoch =9000 // batchsize,
                        epochs = 30,
                        validation_data =test_set,
                        validation_steps = 4500 // batchsize)

Y_pred = classifier.predict_generator(test_set, steps=4500 // batchsize+1)
y_pred = np.argmax(Y_pred, axis=1)
print('Confusion Matrix')
print(confusion_matrix(test_set.classes, y_pred))
print('Classification Report')
target_names = test_set.classes
class_labels = list(test_set.class_indices.keys())
report = classification_report(target_names, y_pred, target_names=class_labels)
print(report)

# summarize history for accuracy
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()
# summarize history for loss
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()

I think you should replace this:
classifier.add(Dense(2))

because you have 2 class: cat & dog.


printing theano.tensor without having any idea about the shape and type of the tensor I am debugging a code which I haven't written. I want to print out state_below_ variable which is calculated as following:

state_below_ = tensor.dot(state_below*emb_dropout[1], tparams[pp(prefix, 'W')]) +\
        tparams[pp(prefix, 'b')]

When I use state_below_.eval(), I get

  MissingInputError: ("An input of the graph, used to compute
  Reshape{1}(y_sampler, TensorConstant{(1,) of -1}), was not provided
  and not given a value.Use the Theano flag
  exception_verbosity='high',for more information on this error.",
  y_sampler)

error. How can I print this damn "state_below_" value?

Thanks,

For evaluating and tensor, you have to pass the value of the input tensors as shown in the below code snippet.

import numpy
import theano.tensor as T
x = T.dscalar('x')
y = T.dscalar('y')
z = x + y
print(z.eval({x : 16.3, y : 12.1}))


Can Anyone Explain this code piece by piece? Function that creates a DataFrame with a column for Cluster Number

def pd_centers(featuresUsed, centers):
    colNames = list(featuresUsed)
    colNames.append('prediction')

    # Zip with a column called 'prediction' (index)
    Z = [np.append(A, index) for index, A in enumerate(centers)]

    # Convert to pandas data frame for plotting
    P = pd.DataFrame(Z, columns=colNames)
    P['prediction'] = P['prediction'].astype(int)
    return P
```

def pd_centers(**featuresUsed**, centers):

inputs of the function are featuresUsed that are the features(column names) you already have and centers.

colNames = list(**featuresUsed**)

names of columns (featuresUsed) are saved here as a list named colNames

colNames.append('prediction')

Here a new column name rather than those already existing is added to column names (features) named prediction

# Zip with a column called 'prediction' (index)
Z = [np.append(A, index) for index, A in enumerate(centers)]

A list of elements of centers with their indices are saved as Z
For instance, if we have centers equal to ['CA', 'TEX', 'FLO', 'ORE']
 then Z would become and finally these indices would become the values of prediction column.

[array(['CA', '0'], dtype='<U21'),
 array(['TEX', '1'], dtype='<U21'),
 array(['FLO', '2'], dtype='<U21'),
 array(['ORE', '3'], dtype='<U21')]

# Convert to a pandas data frame for plotting
P = pd.DataFrame(Z, columns=colNames) #save Z as a dataframe with columns colNames
P['prediction'] = P['prediction'].astype(int) #converting the values in Prediction column to int type
return P

here Z is converted to dataframe named P and the elements in the prediction column is converted to int type and P as the ultimate dataframe is returned. this prediction column is the cluster number.

if centers is a list of tuples, and it should be I assume, then instead of increasing indices, then the corresponding elements in each tuple are saved as A and index.


methodologies for ML algorithms audit Is there any good reference for methodologies on quality assessment of machine learning algorithms design. The scope is wide and embraces each steps from data preparation to generalisability testing.

Further, are there any automated "stress-test tools" for ensuring that the algorithms is not biased, that it fits the alledged performances?

Here's a few fragments of information relevant to your question.

For generalisability testing, it is most common to split the dataset on which you aim to apply a Machine Learning model into a training, validation and testing set. K-Fold cross validation is also common: splitting the dataset into training and validation sets multiple times, randomly, to ensure a good performance on the validation set isn't just pure luck.
Likewise, to ensure your algorithm is not biased and fits alleged performances, there is no better way than to benchmark it on test datasets that the model has not seen during training time. Some deep learning frameworks come with tools that generate reports regarding your model's performance, such as Tensorflow's Tensorboard.
To find out what attributes influence an algorithm, there are various approaches you can take. Interpretability tools such as LIME will help you see what characteristics of the data you trained your model on affect your model's predictions most.

This article on interpretability continues the discussion, goes more in-depth, presents a few frameworks as well as a tutorial you can follow.


How does the time taken for backpropagation change with the increase in neurons? This is a question I come across a lot in interviews but I don't understand how and what the answer is:-
If we have a neural net, and we double the neurons in it, how does the time taken for backpropagation change? does it increase by

2 times
4 times
8 times
16 times

The general idea is that time complexity is related to the number of operations needed to perform all the steps of backpropagation. You can find an example workthrough here:
https://medium.com/swlh/computational-complexity-of-neural-networks-38c01e7e566a
For a 4 layer NN it seems the complexity is O(n⁵), where n is the number of nodes of each layer (also n is the same for every layer in that example).


SMOTE for multi-instance learning i.e num_rows(x_train) > num_rows(y_train) I have an imbalanced dataset and I wish to predict classes(0 or 1).

Sample x_train:

id      date    c1   c2 . . . . . .  c20
101  13-02-2015  2    7 . . . . . .   14
101  14-02-2015 24    7 . . . . . .    8
  .
  .
  .
105  13-02-2015 12    5 . . . . . . .  4
  .
  .

Sample y_train

id   class
101    1
105    1
107    0
 .
 .
 .

Now I wish to over sample class 0 in the dataset but the problem is for each id I have just one row in y_train whereas I have 50 rows for the same id in x_train.

What you have here is called Multi-Instance Learning. From Wikipedia

  In machine learning, multiple-instance learning (MIL) is a type of supervised learning. Instead of receiving a set of instances which are individually labeled, the learner receives a set of labeled bags, each containing many instances.

Source: https://en.wikipedia.org/wiki/Multiple_instance_learning

The approach you take in this case is different. You need to bring the Multi-Instance Learning problem into a Single-Instance Learning one. One way you can do this is:

Perform for each bag of instances a K-Means clustering
Calculate the Hausdorff distance of each instance from each of the cluster
Use those distances as features and keep the label from the y_train set

Then apply SMOTE on the new dataset (where you have one row for features and label) and any kind of model of Single-Instance Learning.

You can find details in this Review of Multi-Instance Learning and Its applications


Vectorizing Skipgrams in sklearn I want to try the skipgrams approach on my dataset. But I do not know how to vectorize it. For example, I have my cleaned document for which I got it's skipgrams. Now, how do I know vectorize it so that I can use it further for classification? I use sklearn for all the above purposes.

The answer to your question can be found at: https://stackoverflow.com/a/45997893/5312422

  To vectorize text with skip-grams in scikit-learn simply passing the
  skip gram tokens as the vocabulary to CountVectorizer will not work.
  You need to modify the way tokens are processed which can be done with
  a custom analyzer. Below is an example vectorizer that produces
  1-skip-2-grams,

  from toolz import itertoolz, compose from toolz.curried import map as
  cmap, sliding_window, pluck from sklearn.feature_extraction.text
  import CountVectorizer

class SkipGramVectorizer(CountVectorizer):
    def build_analyzer(self):
        preprocess = self.build_preprocessor()
        stop_words = self.get_stop_words()
        tokenize = self.build_tokenizer()
        return lambda doc: self._word_skip_grams(
                compose(tokenize, preprocess, self.decode)(doc),
                stop_words)

    def _word_skip_grams(self, tokens, stop_words=None):
        # handle stop words
        if stop_words is not None:
            tokens = [w for w in tokens if w not in stop_words]

        return compose(cmap(' '.join), pluck([0, 2]), sliding_window(3))(tokens)

  For instance, on this Wikipedia example,

text = ['the rain in Spain falls mainly on the plain']

vect = SkipGramVectorizer()
vect.fit(text)
vect.get_feature_names()

  this would vectorizer would yield the following tokens,

['falls on',  'in falls',  'mainly the',  'on plain',  'rain spain',
'spain mainly',  'the in']


How to properly mean-center my data My task is to estimate a person's age based on a rgb image of the face of that person. I'm using ResNet-50 to that end. At first stage I trained my net on a dataset which is called WIKI-IMDB (after filtering and expanding my dataset by horizontal flipping it contains ~300k images) and I initialized the weights of the network with the weights of a pre-trained ResNet-50 network which trained on ImageNet. When training on the WIKI-IMDB dataset I calculated the mean of each channel (RGB) of the train set input and subtracted it from each example in the batch (I also used the same mean to subtract it from the validation set which I'm using for early stopping). after I finished training my network on WIKI-IMDB I want to test it on another dataset set which is a benchmark for this task. my new dataset which I'm testing on is called CACD and contains ~160k images which I'm dividing to train set and test set. I need to fine-tune my network on the CACD train set and finally report the metric I'm using on the CACD test set.

My question is: when learning on the new CACD train set, should I subtract the mean of the CACD train set from each example in the batch or should I keep subtracting the mean of the WIKI-IMDB train set?
and if I need to subtract the new mean of the CACD train set, should I use that mean also when evaluating on the CACD test set?

Each channel of rgb colors has values between 0 and 255.
There are many normalization methods. For this case I would just divide the images over 255 without subtracting the mean so that I make sure the output is between 0 and 1

Z = X/255

if you are interested in centering the images around zero, you can subtract 127 (as a mean) and then divide over 255

Z = (X-127)/255


How to identify potential customers who are ready to convert in to paid? I have data that has actions that perform on my tool and I would like to predict the customers who are ready to convert from free/trail to the paid category.

My data looks like the following:

dummy<-data.frame(license=sample(c("Free","Trail","Paid"),10000,replace = T,prob = c(0.6,0.35,0.05)),
           plan_type=sample(1:5,10000,replace=T),
           action1=sample(0:100,10000,replace = T),
           action2=sample(0:1000,10000,replace = T),
           action3=sample(0:10,10000,replace = T),
           num_days_in_product=sample(0:500,10000,replace = T))

head(dummy)
  license plan_type action1 action2 action3 num_days_in_product
1    Paid         1     100      71       5                 285
2    Free         5      75     438       1                   2
3    Free         1       5     555       7                 389
4    Free         3       4     105       0                 150
5    Free         1      16     348       7                 423
6    Free         5      15     866       8                 270

> table(dummy$license)

 Free  Paid Trail
 6016   516  3468
> prop.table(table(dummy$license))

  Free   Paid  Trail
0.6016 0.0516 0.3468

Let me know if any extra information needed.

Two items come to mind:

You will gain the most benefit from this site when you propose a model and let other people comment on and modify the model. As it stands, you are merely proposing what your data looks like. Do you have a model that you've tried? What data science steps have you already taken with your project? If you post that, this site will be of much better help to you
You should be aware that what you posted makes for a good initial dataset but that's all it is, your initial set. If I was in your position, I would be working towards developing a meta dataset that contains a much larger number of factors. When did the person join? What was the time in between actions? Which country is their IP address from? How many times have they logged into the tool? Etc, etc, etc. When you start to think of your data in this manner, and give yourself a much larger number of factors to start with, then you are taking steps that are much more likely to result in a reliable predictive model.


What's the best way to train a NER model? I am trying to do a project using NLP. My goal is to process Cyber Threat Intelligence articles like this to extract information such as actor’s name, malwares and tools used…

To do that I want to use NER. However, there isn’t training data available on the web. So I was wondering if I should process manually 10-20 articles to make my training data or if I could do something like taking only interesting lines such as “Rancor conducted at least two rounds of attacks intending to install Derusbi or KHRat malware on victim systems” in multiples articles and replacing the group name by another actor. This way I could deduplicate my training data by the number of known actors. But doing that, only the actor name is changing. So, the context is always the same.

I am wondering what’s the best way to train my model considering the quantity of training data available?

I would start by training some very strong Named Entity classifier on available datasets for NER. One is the Annotated Corpus for Named Entity Recognition available on Kaggle.

Additionally, you can find a good list of datasets here. I know they have nothing to do with cybersecurity, but I think it's important to incorporate very different sources in a big, final dataset, in order to make a model that is good at generalizing on texts it has never seen before.

Another source of data for NER tasks is the annotated corpora available from nltk library, such as the free part of the Penn Treebank dataset, and Brown corpus.

Please beware that different datasets might use different categories for classification (i.e. the set of Named Entities can be different from dataset to dataset). Make sure you make all your data compatible to your classifier before training

After that, I suggest you to go with seq2seq models. Every state-of-the-art RNN is some form of seq2seq. Once you trained a classifier, you could try to annotate few articles manually, and check the performance of your model on those. It's time consuming, but I personally like these "qualitative" checks, I think they can tell you a lot.


how to remove unwanted characters from data I need to write a function which identifies and removes the "*" character after some numeric values in a vector. I also need  that the resultant vector is a numeric vector.

c("21,34,99*", "56,90*", "45*")

I need to remove "*" which is unwanted.

I want to write function so whenever such data cleaning requirement I can use function and pass certain parameters.

Should have been posted in SO as others rightly pointed out.

Simple solution would be to:

Split on '*' and select the first part of the split.
Apply (1) over list items and return a vector.

x <- c("21,34,99*", "56,90*", "45*")               #test case

data_clean <- function(x) sapply (strsplit(x , '[*]' ), `[` , 1)
data_clean(x)


Bar plot with varying length Hello folks,
            I am trying to plot a grouped bar plot of two variables with varying lengths, which means the x and y length of both the variables are different. The format of the data is given below:
This is for NRI.

This is for RI.

I want these two datasets to be grouped together. When I try to plot it both the datasets are overlapping each other. If anyone can help me in this regard it will be much appreciated.

Here is the code I used:

import numpy as np
from mpl_toolkits.basemap import Basemap
import matplotlib.pyplot as plt
import matplotlib.path as mpath
from PIL import *
import os
import sys
import csv
from matplotlib import rc, rcParams
import pandas as pd
from matplotlib.ticker import (MultipleLocator, FormatStrFormatter,
                               AutoMinorLocator)
#df = pd.read_csv('E:/v.csv')
df = pd.read_excel('IC12_freq.xlsx',sheet_name='NRI')
df_1 = pd.read_excel('IC12_freq.xlsx',sheet_name='RI')

x = df.IC12.values
x1 = df_1.IC12.values
y = df.FRQ.values
y1 = df_1.FRQ.values
fig, ax = plt.subplots(figsize=(10,10))
ax.bar( x+1.3,y,color = 'w', width = 1.3,hatch='***',edgecolor='k',label='NRI',align='center')
#ax.twinx()
ax.bar( x1,y1,color = 'w', width = 1.3,hatch='/////',edgecolor='k',label='RI',align='center')
#ax.plot(x, y,'ro',color = 'k')
#ax.plot(x1, y1,'ro',color = 'r')
ax.xaxis.set_major_locator(MultipleLocator(10))
ax.xaxis.set_major_formatter(FormatStrFormatter('%d'))
ax.xaxis.set_minor_locator(MultipleLocator(5))
ax.xaxis.set_minor_formatter(FormatStrFormatter('%d'))
# For the minor ticks, use no labels; default NullFormatter.
ax.yaxis.set_major_locator(MultipleLocator(10))
ax.yaxis.set_major_formatter(FormatStrFormatter('%d'))
ax.yaxis.set_minor_locator(MultipleLocator(5))
ax.yaxis.set_minor_formatter(FormatStrFormatter('%d'))

plt.rcParams["font.weight"] = "bold"
axis_font = {'fontname':'Arial', 'size':'14'}
tick_font = {'fontname':'Arial', 'size':'5'}

ax.tick_params(which='both', width=2)
ax.tick_params(which='major', length=7)
ax.tick_params(which='minor', length=7)
plt.xlabel('IC12(kt)', fontweight='bold',**axis_font)
#plt.xticklabel(**tick_font)
plt.ylabel('Frequency(%)', fontweight='bold',**axis_font)
ax.set_facecolor("#f1f1f1")
ax.spines['top'].set_linewidth(1.5)
ax.spines['right'].set_linewidth(1.5)
ax.spines['bottom'].set_linewidth(1.5)
ax.spines['left'].set_linewidth(1.5)
leg = ax.legend()

#plt.grid(True)
#plt.style.use('ggplot')
##plt.xlabel("x axis", **axis_font)
#plt.ylabel("y axis", **axis_font)
#plt.bar(y,x)
#plt.savefig('IC12_frq.tif', bbox_inches='tight', dpi=300)
plt.show

I would first decide to bin the x-axis such that it can be plotted in groups. Thus if we want to for example group them into bins of width 5 then plot them next to each other we would do something like this

width_box = 5
low = -40
high = 40
width_graph = 1.5 # should be at most width_box/2 to not overlap

def sum_in_range(df, min_, max_):
    return df[(df['IC12']<max_) & (df['IC12']>=min_)].FRQ.sum()

indices = np.array(range(low, high, width_box))
sum1 = [sum_in_range(df, i, i+width_box) for i in indices]
sum2 = [sum_in_range(df_1, i, i+width_box) for i in indices]

fig = plt.figure()
ax = fig.add_subplot(111)
ax.bar(indices,sum1,width_graph,color='b',label='-Ymin')
ax.bar(indices+width_graph,sum2,width_graph,color='r',label='Ymax')
ax.set_xlabel('Test histogram')
plt.show()

Which gives us

You can play around with the width_graph parameter to change how wide the bars look, and you can change width_box to group up the x-axis in different ranges


How interpret keras training loss without compare with validation loss? I have several implementation of the same neural network, but each one with different starting parameter.

This is one of my plot comparing the training loss of the base experiment with the training loss of another experiment.

I have also other exaples:

May anyone point me to some instruction on how understand these output from the keras fit()? Note that I don't have any validation set.

Thanks

In Keras model it defines its own bias and weight..and eventually they converge closer to Y as loss decreases..the output graphs would change based on learning rate, batch size and epochs


Why is sklearn.metrics.roc_auc_score() seemingly able to accept scores on any scale? I had input some prediction scores from a learner into the roc_auc_score() function in sklearn. I wasn't sure if I had applied a sigmoid to turn the predictions into probabilities, so I looked at the AUC score before and after applying the sigmoid function to the output of my learner. Regardless of sigmoid or not, the AUC was exactly the same. I was curious about this so I tried other things like multiplication by arbitrary numbers and applying arbitrary log or exp functions and the score was still the same.

Assuming I haven't made some other error somewhere, why is the sklearn function for ROC AUC able to work on any scale of scores?

The documentation says

  Target scores, can either be probability estimates of the positive
  class, confidence values, or non-thresholded measure of decisions
  (as returned by "decision_function" on some classifiers).

(My emphasis)

This is possible because the implementation only requires that the y_true can be sorted according to the y_score.

The false positive rate that is returned is given as follows:

  A count of false positives, at index i being the number of negative
  samples assigned a score >= thresholds[i]. The total number of
  negative samples is equal to fps[-1] (thus true negatives are given by
  fps[-1] - fps).

The false positives can be calculated with only taking into account the order of the predicted scores/thresholds.

Multiplying the scores by a scalar has no effect on the order of the values, thus the resulting score remains the same!

See the sklearn GitHub page, the relevant code is under the definition of _binary_clf_curve.


AUC-ROC of a random classifier Why the area under the ROC Curve for a random classifier is equal to 0.5 and has diagonal shape?
For me a random classifier would have 25% of TP,TN,FP,FN and therefore it would only be a single point on the ROC Curve.

Premise:
The confusion matrix that you mention above is only correct if

there is as much Positive and Negative cases in the data
we use a random classifier that assigns Positive and Negative class both with probability 0.5

So you derive the confusion matrix for a very particular random classifier for a very particular dataset. This very particular random classifier is indeed, as you point out, only a point in the ROC plot. To obtain the whole ROC curve, we have to vary the probability with which we assign the positive class, from 0 to 1.
So in effects, the ROC curve is a graphical evaluation of the performance of infinitely many classifiers!
Each one of these random classifiers with a different probability will have a different expected confusion matrix.

Derivation: Here I derive the AUC of a random classifier on a dataset with an arbitrary class imbalance.
Assuming that there is a fraction $x$ of positive cases and a fraction $1-x$ of negative cases, and that our classifier consists of randomly assigning the positive class with probability $\rho$ and the negative class with probability $1-\rho$. The confusion matrix of a random classifier will have the following expected proportions
$$TP = \rho x \\ FP = \rho (1-x) \\ FN = (1-\rho)x \\ TN = (1-\rho)(1-x)$$
Then we calculate the True Positive Rate (sensitivity) and False Positive Rate (1-specificity) of our random classifier
$$TPR=\frac{TP}{TP+FN}=\frac{\rho x}{\rho x + (1-\rho)x}=\rho\\ FPR=\frac{FP}{TN+FP}=\frac{\rho(1-x)}{\rho(1-x)+(1-\rho)(1-x)}=\rho$$
So as you can see TPR and TFR do not depend on the class proportion $x$, which means that the ROC-AUC will also be independent of $x$. Moreover TPR=FPR, meaning that the ROC-curve will look like a straight line from (0,0) to (1,1). Now we calculate the AUC as the integral between 0 and 1 of the area under TPR as a function of FPR as we vary the threshold $\rho$.
$$AUC=\int_0^1 \rho d\rho' = \frac{\rho'^2}{2}|_0^1 = 1/2$$
So the area under the ROC curve for a random classifier is 0.5 regardless of the class proportion.


Changing order of LabelEncoder() result Assume I have a multi-class classification task. The labels are:

Class 1
Class 2
Class 3

After LabelEncoder(), the labels are transformed into 0-1-2.
My questions are:

Do the labels have to start from 0?
Do the labels have to be sequential?
What happens if I replace all label 0s with 3 so that my labels are 1-2-3 instead of 0-1-2 (This is done before training)
If the labels were numeric such as 10-100-1000, will I still have to use LabelEncoder() to encode them into 0-1-2?

Do the labels have to start from 0?

No it doesn't matter where they start as long as they have distinct values.

Do the labels have to be sequential?

Well it depends from the feature. For example if you have features that are showing order of magnitude, like small<big<vast, then yes the order matters and they are called ordinal features, but if the feature's values represent for example countries then there is no such thing as order, so probably one should use OneHotEncoder, in order to be equally distanced in space. (see here)

What happens if I replace all label 0s with 3 so that my labels are 1-2-3 instead of 0-1-2 (This is done before training)

Except the previous bullet, one should consider the type of model that will use. For example tree based model like RandomForest work very well with categorical data, and the numerical value of a category could be arbitrary. But this is not the case for the linear models.

Closing if you want to convert a categorical feature to numerical values, you should consider two things the features values (ordinal?) and the type of model.
P.S. To improve the performance of the model there are many way to convert categorical to numerical features, like Target encoding techniques, that have been showing to improve also tree based classifiers, but perhaps this is a conversation for another time :)


applying word2vec on small text files I'm totally new to word2vec so pls bear it with me. I have a set of text files each containing a set of tweets, between 1000-3000. I have chosen a common keyword ("kw1") and wants to find semantically relevant terms for "kw1" using word2vec. For example if the keyword is "apple" I would expect to see related terms such as "ipad" "os" "mac"... based on the input file. So this set of related terms for "kw1" would be different for each input file as word2vec would be trained on individual files (eg., 5 input files, run word2vec 5 times on each file).

My goal is to find sets of related terms for each input file given the common keyword ("kw1"), which would be used for some other purposes.

My questions/doubts are:

Does it make sense to use word2vec for a task like this? is it technically right to use considering the small size of an input file?

I have downloaded the code from code.google.com: https://code.google.com/p/word2vec/ and have just given it a dry run as follows:

 time ./word2vec -train $file -output vectors.bin -cbow 1 -size 200 -window 10 -negative 25 -hs 1 -sample 1e-3 -threads 12 -binary 1 -iter 50

./distance vectors.bin

From my results I saw I'm getting many noisy terms (stopwords) when I'm using the 'distance' tool to get related terms to "kw1". So I did remove stopwords and other noisy terms such as user mentions. But I haven't seen anywhere that word2vec requires cleaned input data...?
How do you choose right parameters? I see the results (from running the distance tool) varies greatly when i change parameters such as '-window', '-iter'. Which technique should I use to find the correct values for the parameters. (manual trial and error is not possible for me as I'll be scaling up the dataset).

Word2Vec isn't a good choice for a dataset of such size. From researches I have seen, it will unleash its power if you feed at least couple of million of words, 3k tweets wouldn't be enough for a concise word similarity.


Am I doing a log transformation of data correctly? I'm doing some exploratory data analysis on some data and I get these histograms:

That looks like a candidate for a log transformation on the data, so I run the following Python code to transform the data:

df["abv"].apply(np.log).hist()
df["ibu"].apply(np.log).hist()
plt.show()

And I get this new plot of the transformed histograms:

Am I correct that a log transform was ok to do in this case, and if so, what's the best way to interpret the results?

Yes, log transform seems a good solution for better interpretation. Overlap between these two datasets is really small, so, only by looking at the plot, you can say with high certainty, that they are significantly different from each-other.


How to count categorization instances in a NN? Say one-hot encoding is the perfect way to represent a series of objects such as clothing items. Ie: A hat is [1 0 0], a tie is [0 1 0].

I want to predict what a customer buys if they spend a certain amount of money, which will serve as an input to the NN. Eg: For one example, the customer spends $22. The thing is, the customer may buy multiples of the same clothing item.

What would be the best way to make the neural network output something like [2 1 0] to show its prediction as 2 hats and 1 tie, given the input of $22?

As far as I understand binary crossentropy is good for multilabel problems, but nowhere could I find an example of how I could apply this when the same label occurs more than once.

The only solution I could think of was turning this into a regression problem, but doing so would seem bad since this is a categorical problem, and the network should output fixed natural numbers depending upon the number of times an object is predicted.

But your intuition about regression is right. Basically, you have multi target regression problem. Each category is a separate target. If your get float output as prediction, you can round it. Float output also makes sense, because it shows you how confident is your model about prediction and the expectation of the number of bought items.
Number of bought items is structured as ordinals, so it makes sense to predict it as multi target regression. Moreover, you may not know the upper limit of the number of bought items, so it makes no sense to predict number of items as categories.
Look at this for reference:

Neural Network for Multiple Output Regression
Resources for learning about multiple-target techniques?


Is there a way to add new feature to existing machine learning model and retrain it? Suppose a machine learning model is trained on two features (f1, f2). If there comes a new feature(f3) is there any way to pass this new feature to existing model and retrain instead of training a model again on the three features?

So the question asks if you can retrain the model after adding another feature.
Answer is yes but you need to instantiate a new model which can accept a 3-feature/dimensional input.
This is due to the mechanics of neural networks whereby in the 1st layer, the features in your original model would be multiplied by a ($2 x n$) matrix where $n$ represents the number of hidden neurons in the second layer of the model.
Therefore by multiplying a ($3 x 1$) input with the weight matrix will throw an error.


Estimating the value of $\pi$ with a Monte Carlo dartboard: $<$ or $\leq$? I'm trying to figure out which is the proper way to estimate $\pi$ using the Monte Carlo method randomly distributing points in a square that also contains an inscribed circle.

Some sources say to use the comparison of $\sqrt{x^2+y^2}\le 1$, while others use $\sqrt{x^2+y^2}<1$.

Here's some example code from a wikipedia article:

def monte_carlo_pi(nsamples):
    acc = 0
    for i in range(nsamples):
        x = random.random()
        y = random.random()
        if (x**2 + y**2) < 1.0:
            acc += 1
    return 4.0 * acc / nsamples

Instead of posting a long list of websites that use  $\le 1$  or  $< 1$, I've made the list and stored it on the following websites:
See either: socrates.io or markdown.press or markdownshare for examples using less than and less than or equal to.

Short answer: Both formulations lead to the same answer.

Mathematical explanation:

In order to understand that let us look at two similar problems. Imagine we want to integrate a function $f(x)=x^2$ on two intervals $I_1=[0,1]$ (including both bounds) and $I_2=(0,1)$ (excluding both bounds).

For $I_1$ we have

$$\int_0^1 x^2~dx=1/3.$$

For the second interval, we need to introduce a positive dummy parameter $\varepsilon$ then we can calculate the integral as

$$\lim_{\varepsilon\to 0}\int_{0+\varepsilon}^{1-\varepsilon}x^2~dx=1/3.$$

So the line of separation between both intervals does not contribute to the integral (area) because it has an infinitesimal width. The same argument can be applied to the circle area. But including the line of the circle will make the calculations easier because we do not need to introduce a dummy variable.

Numerical explanation:

Because of the numerical precision of your computer, it will be very unlikely that the generated random numbers will lead to points that are really on the line of the circle. Numerically it will be impossible to obtain any value in which at least one of the coordinates is irrational. Only if both values are rational and if $x^2+y^2=1$ they can lie on the circle. But this case is also very unlikely.


How can I fill NaN values in a Pandas DataFrame in Python? I am trying to learn data analysis and machine learning by trying out some problems.

I found a competition "House prices" which is actually a playground competition. Since I am very new to this field, I got confused after exploring the data. The data has 81 columns out of which 1 is the target column which is the house value. This data contains multiple columns where majority of values are "NaN". When I ran:

nulls = data.isnull().sum()
nulls[nulls > 0]

This shows the columns with missing values:

LotFrontage     259
Alley           1369
MasVnrType      8
MasVnrArea      8
BsmtQual        37
BsmtCond        37
BsmtExposure    38
BsmtFinType1    37
BsmtFinType2    38
Electrical      1
FireplaceQu     690
GarageType      81
GarageYrBlt     81
GarageFinish    81
GarageQual      81
GarageCond      81
PoolQC          1453
Fence           1179
MiscFeature     1406

At this point I am totally lost and I don't know how to get rid of these "NaN" values.
Any help would be appreciated.

You can use the DataFrame.fillna function to fill the NaN values in your data. For example, assuming your data is in a DataFrame called df,

df.fillna(0, inplace=True)

will replace the missing values with the constant value 0. You can also do more clever things, such as replacing the missing values with the mean of that column:

df.fillna(df.mean(), inplace=True)

or take the last value seen for a column:

df.fillna(method='ffill', inplace=True)

Filling the NaN values is called imputation. Try a range of different imputation methods and see which ones work best for your data.


How do I use a model after it's fitted to predict the class of a single string? After a model is built, how can I use it to predict the class of a single string?

model.predict() is returning something like [[0.41100174 0.5889983 ]] instead of it's predicted class (0 or 1).

Say I just built model like so:

hist = model.fit(data.x_train,
                 data.y_train,
                 validation_data=(data.x_test, data.y_test),
                 epochs=500,
                 batch_size=50,
                 shuffle=False,
                 verbose=2,
                 callbacks=[checkpoint, estopping, tensorboard])

I'm looking to predict a string's class using model.predict(), but it returns something like [[0.41100174 0.5889983 ]] instead of it's predicted class (0 or 1).

The shape of data.x_test (used for validation data) is the same shape as data.x_data (reformatted string to predict): (1, 250, 70) (except the number of rows, obviously)

Here's how I'm trying to use the model to predict the class of a string.

def predict_string(model,s):
    df = pd.DataFrame([s], columns=['text'])
    df = df.reset_index(drop=True)
    df['label'] = [0]

    df.label = pd.to_numeric(df.label, errors='coerce') # Convert to integer
    df = df.dropna()
    df = df[df.label.apply(lambda x: x !="")]
    df = df[df.text.apply(lambda x: x !="")]

    vocab_len = 70
    data = char_preproc(df.text, df.label, vocab_len, True, None)
    y_pred = model.predict(data.x_data)
    return y_pred

s = "Best movie ever" # Out: [[0.41100174 0.5889983 ]]

# s = "Worst movie ever" # Out: [[0.5436389  0.45636114]]

y_pred = predict_string(model, s)
print("Review: {}\"\nPredict: {}".format(s, y_pred))

I'm not sure it matters, but for testing, I'm classifying movie reviews as good (1) or bad (0) using a Character-level CNN trained on the Rotten Tomatoes Movie Review dataset, running on GPU via Google Colab.

[[0.41100174 0.5889983 ]] what this means is the probability of class 0 is 0.411 and probability of class 1 is 0.588. Since probability of class 1 is greater than probability of class 0, it belongs to class 1.

a = [[0.41100174 0.5889983 ]]

np.argmax(a)

Output : 1

np.argmax will get you the class.


One way is to normalize your quantitative  values (play, eat, drink, sleep rates) so they all have the same range (say, 0 -> 1), then assign each game to its own "dimension", that takes value 0 or 1. Turn each row into a vector and normalize the length to 1. Now, you can compare the inner product of any two people's normalized vectors as a measure of similarity. Something like this is used in text mining quite often

R Code for Similarity Matrix

Assumes you've saved your dataframe to the variable "D"

#Get normalization factors for quantitative measures
maxvect<-apply(D[,1:4],MARGIN=2,FUN=max)
minvect<-apply(D[,1:4],MARGIN=2,FUN=min)
rangevect<-maxvect-minvect
#Normalize quantative factors
D_matrix <- as.matrix(D[,1:4])
NormDMatrix<-matrix(nrow=10,ncol=4)
colnames(NormDMatrix)<-colnames(D_matrix)
for (i in 1:4) NormDMatrix[,i]<-(D_matrix[,i]-minvect[i]*rep(1,10))/rangevect[i]
gamenames<-unique(D[,"game"])
#Create dimension matrix for games
Ngames<-length(gamenames)
GameMatrix<-matrix(nrow=10,ncol=Ngames)
for (i in 1:Ngames) GameMatrix[,i]<-as.numeric(D[,"game"]==gamenames[i])
colnames(GameMatrix)<-gamenames
#combine game matrix with normalized quantative matrix
People<-D[,"name"]
RowVectors<-cbind(GameMatrix,NormDMatrix)
#normalize each row vector to length of 1 and then store as a data frame with person names
NormRowVectors<-t(apply(RowVectors,MARGIN=1,FUN=function(x) x/sqrt(sum(x*x))))
dfNorm<-data.frame(People,NormRowVectors)

#create person vectors via addition of appropriate row vectors
PersonMatrix<-array(dim=c(length(unique(People)),ncol(RowVectors)))
rownames(PersonMatrix)<-unique(People)
for (p in unique(People)){
  print(p)
  MatchIndex<-(dfNorm[,1]==p)*seq(1,nrow(NormRowVectors))
  MatchIndex<-MatchIndex[MatchIndex>0]
  nclm<-length(MatchIndex)
  SubMatrix<-matrix(NormRowVectors[MatchIndex,],nrow=length(MatchIndex),ncol=dim(NormRowVectors)[2])
  CSUMS<-colSums(SubMatrix)
  NormSum<-sqrt(sum(CSUMS*CSUMS))
  PersonMatrix[p,]<-CSUMS/NormSum
}
colnames(PersonMatrix)<-colnames(NormRowVectors)
#Calculate matrix of dot products
Similarity<-(PersonMatrix)%*%t(PersonMatrix)


Multilabel Classification - increasing the confidence I have a basic multilabel topic classifier(Tfidf vectorizer with OneVsRest Classifier) built on some customer reviews. I observed that there are some classes with right features but it still predicts with very low confidence.

What is the reason for such a classification in general?
How do I boost the confidence?

It's hard to say without knowing the data.

Often times the classes are imbalanced. This could cause your problems.
So let's say that the "features are right" for 5% of your data and they belong to class $A$. You would expect the 5% to be classified correctly as $A$ almost all the time. But what about the remaining 95% in class $B$? The classifier learns that it would be often correct if the sample is classified as $B$. This leads to a low classification rate for class $A$.

You can tackle this by setting different miss-classification costs for the classes. Often you multiply the error rate by these costs to scale them accordingly. Some classifiers let you set this during training.

The standard case would be as follows:

$$C = \left( \begin{matrix}
0 & 1  \\
1 & 0  \\
\end{matrix} \right)$$

Meaning that correct classification (diagonal) is not punished and miss-classifying $A$ as $B$ and vice versa is punished in the same way. However, this assumes uniformly distributed classes. To account for the case explained above, you may set it the following way:

$$C = \left( \begin{matrix}
0 & 0.05  \\
0.95 & 0  \\
\end{matrix} \right)$$

Here i set it as the inverse of the class distribution. However, you may try different settings.