Rules by which RStudio sets Headings RStudio automatically recognizes headers in an R script that are set via comments:

I would like to exploit that feature, but I don't quite understand what the rules are for RStudio to recognize them as headers. Can someone explain?

Check out Code Folding and Sections:

  Code sections allow you to break a larger source file into a set of
  discrete regions for easy navigation between them. Code sections are
  automatically foldable—for example, the following source file has
  three sections (one expanded and the other two folded):

  To insert a new code section you can use the Code -> Insert Section
  command. Alternatively, any comment line which includes at least four
  trailing dashes (-), equal signs (=), or pound signs (#) automatically
  creates a code section. For example, all of the following lines create
  code sections:

  # Section One ---------------------------------
   # Section Two =================================
   ### Section Three #############################

  Note that as illustrated above the line can start with any number of
  pound signs (#) so long as it ends with four or more -, =, or #
  characters.

(highlights by myself)


Difference between various linear regression implementations Aim: To find the coefficients for the regression line (hyperplane in case of multiple variables?) that models the data best. Let's call this w

What is the difference between:

1) Estimating using MAP: $w=(XX^T+\lambda I )^{-1}Xy^T$ where $X$ is the input training data and $y$ is the training outputs

and

2) Using a neural network to perform regression (I don't know how this is implemented)

(and any other method used for linear regression)

In the first, you need a linear model and the cost function is RMSE. with this model and cost function, w is generated.

on the other hand, for the second equation, you use a non-linear NN equation as estimation and the cost function may RMSE, log or ... .( NN without activation-function is linear)
NN without activation function is a linear model with bias. if you use RMSE it is like the first example.

any linear regression with the same cost function will generate, the same value for coefficient after convergence (In some condition the equation has many solutions, In this case, the cost function may converge to a new value)

Note: many solutions for an equation meaning, for example, if one point is the solution of the equations, then any hyperplane that crosses that point is the solution.


Confusion matrix doesn't display properly I am trying to plot a confusion matrix using the Logistic Regression for a multi-class dataset.

But the problem is when I plot the confusion matrix it only plot a confusion matrix for binary classification.

Here is where I am plotting it.

%matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd

dataframe = pd.read_csv("WA_Fn-UseC_-HR-Employee-Attrition.csv")

from sklearn.linear_model import LogisticRegression

LRModel = LogisticRegression(C=100, max_iter=5500)

LRModel.fit(X_train, y_train)

predicted_values_ = LRModel.predict(X_test)
from sklearn.metrics import confusion_matrix

cm = confusion_matrix(y_test, predicted_values_)

misclassified = (y_test != predicted_values_).sum()
misclassified

import seaborn as sn

# plt.figure()
sn.heatmap(cm, annot=True)
plt.xlabel("Predicted")
plt.ylabel("Actual")

And I get this matrix as shown below.

Can someone tell me where I am doing wrong?

This is where I am using Logistic Regression for multi-class scenario

Please check how many classes y_test has.

if Y is array-

np.unique(y_test)

If y is DataFrame column,

y_test.unique()

Looks like y_test has only 2 classes.

As far as I know HR-Employee-Attrition.csv dataset has only 2 classes.


Should Principal components be normalized before applying K means on them? I want to get the Principal components of a dataset and apply K mean clustering on them. Do I need to Normalized the PCA output before applying Kmeans on them ?

No - There is no need to normalize after Principal Component Analysis (PCA) because each dimension is on the same scale.


Jaccard similarity calculate similarity It is not clear to me how to calculate similarity between two products from the example.

How do they calculate that?

Note the instruction is to view the utility matrix as boolean. That is if it is positive, then view it as $1$ and $0$ otherwise.

Example, to compute the Jaccard similairty between $a$ and $b$.

Consider the component of $a$ to be $(1,0,1,1,0,1)$ and component of $b$ to be $(0,1,1,0,1,1)$.

To compute $|a \cap b|$, look at how many position where both of the values are $1$. This is satisfied at the third and the sixth coordinate.  Hence the value is $2$.

To compute $|a \cup b|$, look at how many position where at least one of the value is $1$. This is satisfied at every position.  Hence the value is $6$.


Going from report to feature matrix I am starting to play around in datamining / machine learning and I am stuck on a problem that's probably easy.

So I have a report that lists the url and the number of visits a person did. So a combination of ip and url result in an amount of visits.

Now I want to run the k-means clustering algorithm on this so I thought I could approach it like this:

This is my data:

url      ip    visits

abc.be   123   5
abc.be/a 123   2
abc.be/b 123   2
abc.be/b 321   4

And I would turn in into a feature vector/matrix like so:

abc.be  abc.be/a   abc.be/b   impressions
   1       0          0          5
   0       1          0          2
   0       0          1          2
   0       0          1          4

But I am stuck on how to transform my data set to a feature matrix. Any help would be appreciated.

I don't understand what you mean by

  So I have a report that lists the url and the number of visits a person did. So a combination of ip and url result in an amount of visits.

Assuming that you equate an IP with a user, and you wish to cluster users by their URL visitation frequencies, your matrix, M, would have

One row per IP (user)
One column for each URL that you are tracking (your features)
and the entries in M would be "visits" of a given URL by a particular IP

Given these assumptions, and your report, M would be:

    abc.be  abc.be/a  abc.be/b
123   5        2         2
321   0        0         4


Why do classes with fewer words have higher probability? Given that I have a word that does not occur in any of my documents: newword, and given that I have two classes: class1 and class2. For instance, the total number of words in class1 is 3, the total number of words in class2 is 6, and the number of unique words in all documents is 8.

Also, given the Naive Bayes formula, the word newword will have a higher probability of belonging to class1 (because the denominator will be lower while the numerator stays the same). Is there any statistical/logical theoretical explanation (that drives the algorithm) about this behavior?

In a nutshell, I just want to know what is the motivation of giving higher probability to classes with fewer words.

Ps.: I am using Laplace Smoothing. Therefore:
 P(newword|class1) = 0 + 1 / 3 + 8 = 0.09 >
P(newword|class2) = 0 + 1 / 6 + 8 = 0.07

The Naive Bayes formula doesn't do what you say. For classes $C_k$ and words $x_1, \ldots, x_n$, the formula is

$P(C_k | x_1, \ldots, x_n) \propto P(C_k) \prod_{i=1}^N P(x_i | C_k)$

If the word newword doesn't occur, we have no prior information about $P(x_i | C_k)$, so all these conditional probabilities should be equal. So the right-hand side is just proportional to $P(C_k)$, which is independent of the number of words in $C_k$.


Attribute discretization I have a dataset where I need to explore using unsupervised technics (clustering and association rules). What are the best strategies to discretize the numeric attributes? Also, does this (attribute discretization) makes sense for clustering (using K-Means)?

Since I'm using weka, I know that I can just use the Discretize filter, with findNumBins option or useEqualFrequency. But are there other strategies?

Also, does it makes sense to use discretization on the attributes, and then look for clusters (using K-Means) or not? (I suspect that it doesn't make that much sense, but I just want to confirm)

Thanks!

Most clustering algorithms work best with continuous values (in particular k-means). Many cannot use categorial attributes and won't work very well with one-hot encodings either.


Drop NA values in an imported xlxs document with R I have the attached document.  I've imported an xlxs document into R.  I would like to get rid of all rows containing NA.  I've tried the following but it isn't working.

# install packages Note# tidyverse installs packages for dplyr and ggplot2
install.packages("tidyverse")
install.packages("readxl")

#open the library
library(tidyverse)
library(readxl)

setwd("~/Documents/UofL/Data_Science/CECS_635/week_3/")

un <- read_excel("UnitedNations.xlsx")

un1 %>% drop_na(un,)

its should be working

data_set<-na.omit(un)

If want to handle NA's to mean or median:

na.mean(un, option = "mean")

Source:
https://www.rdocumentation.org/packages/imputeTS/versions/2.7/topics/na.mean


Interpreting random seed effect on correlation in regression with small sample I am working on a project with a sample size of 30. I have 7 features predicting a continuous variable where I am aiming to optimize the r-value. If I change the random seed of my train_test_split() 100 times, I get correlation coefficients that range between .6 and .9 with an occasional negative r-value.

How should I interpret this? Is this variability normal for such a small data set and a random shuffling of my training data? Or is something else going on? How should I decided which seed to use?

This means that your model is not consistent. Your problem is hard because of the small sample size. I would recommend reducing the number of predictor variables.

7 predictors is quite a few for linear regression, even with a large sample size. Try and get the best results you can with 2 or 3 predictors.

If you can predict with 30% accurately all of the time it is better than 70% accuracy 10% of the time.


How max_features parameter works in DecisionTreeClassifier? What is the parameter max_features in DecisionTreeClassifier responsible for?

I thought it defines the number of features the tree uses to generate its nodes. But in spite of the different values of this parameter (n = 1 and 2), my tree employs both features that I have. What changes so?

max_features = 2

max_features = 1

You can see x1 and x2 are used in both cases

Max_feature is the number of features to consider each time to make the split decision. Let us say the dimension of your data is 50 and the max_feature is 10, each time you need to find the split, you randomly select 10 features and use them to decide which one of the 10 is the best feature to use.
When you go to the next node you will select randomly another 10 and so on.
This mechanism is used to control overfitting. In fact, it is similar to the technique used in random forest, except in random forest we start with sampling also from the data and we generate multiple trees.
So even if you set the number to 10, if you go deep you will end up using all the features, but each time you limit the set to 10.
If you compare the definition of the max feature in the decision tree and random forest, you will see that they are the same.
https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html
https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html


Is it safe to use labels created from unsupervised model to train a supervised model using the same data? I have a dataset where I have to detect anomalies. Now, I use a subset of the data(let's call that subset A) and apply the DBSCAN algorithm to detect anomalies on set A.Once the anomalies are detected, using the dbscan labels I create a label variable (anomaly:1, non-anomaly:0) in the dataset A. Now, I train a supervised algorithm on dataset A to predict the anomalies using the label as the dependent/target variable and finally use the trained supervised model to predict the anomalies on the rest of the data (A compliment).

While, this seems to be a fair approach to me, I am just wondering if there is any data leakage happening at any stage. Please note that I am using the same set of variables/features at both stage(unsupervised and supervised). Reason for posting is when I train the supervised model, I get very high roc-auc score, which is around 0.99XX and that is suspicious.

Note that, I can not use the DBSCAN algorithm for the entire data set because of computational constraints.
I can not use supervised model as I do not have labels.

Please take care of stratification while sampling data for training. You should mention stratify = y in train_test_split .

High ROC-AUC may be pointing to imbalance dataset - when-is-an-auc-score-misleadingly-high

Also check this out - k-fold-cross-validation-auc-score-vs-test-auc-score


Two different GPUs for Keras (Python)? One question guys, someone knows if it should be ok to get one more GPU of type Nvidia Geforce GTX 1070 (gaming version), given that now I have GTX 1070 Titanium? They don't have another Titanium card available here, so I have to get a different one, but closely similar, and I wonder if for using Keras (with TensorFlow backend), will it work fine? They are not exactly the same cards, but similar enough maybe. I want 2 GPUs for Keras.

I tried training with a 1080 and a 2080 ti and I found that I didn't get any speed up from multi-gpu training because the 1080 acted as a bottleneck. So I while I think this would work fine, you would be better served to run a different model on each GPU rather than trying to train across GPUs.


How to use vectors produced by TF-IDF as an input for fuzzy c-means? I have done text processing with TF-IDF method and as an output got a list of normalized vectors [0, 1] for each document. Such as below:

Document 1
word1:1.0, word2:0.9, ..., word_n:0

Document 2
word2:1.0, word1:0.4, ..., word_n:0
...
etc

The above is basically a list of key-values where key is a term and values  are TF-IDF values, where value 1 means that the term matches the document the most compared to other terms in the set.

My question is, to what form should I transform these vectors in order to properly use fuzzy c-means clustering on them? I feel like it should be 2D matrix of something, but can't figure it out.

At the very end I would like to have a trained model which on a given input could say to what documents (based on the membership values) it belongs with the highest chance.

Assign a unique column to each word.

All remaining values are zero, obviously.


NLP: Information extraction I need to extract product names from a text column in a dataset. Currently I'm using regular expressions to extract the product names from the middle of the text, but sometimes the product name is misspelled, incomplete or even amended in another word, which means that I am unable to identify and extract the product name.
We currently have around 1500-2000 products on that list and I have a data set with those products already identified from approximately 30,000 lines. Is there an approach that I can use this historical data to improve the identification of products that have not yet been identified?
Just an example:
The product X produced by the Company Y is used to treat skin diseases

Note: The product names doesn't appear in a fixed position.

What you need is perhaps Named Entity Recognition with custom entity dictionary. See this example:

Many packages like NLTK or Spacy have a large dictionary of such entities that enable models to identify them without using regular expressions. However, those pre-defined entities often is not applicable to one's application, thus won't recognize what you are looking for. That is why you need to spend time to update/add a list of such entities (products in your case). Also you mentioned that there are misspelling, variations etc. that make it harder. Here is where you need to expand your dictionary for such products with a fuzzy-matching algorithms to account for such edge cases programmatically.
You may start reading this article to the idea. More about Named Entity Recognition: a Blogpost, or the official Named Entity Recognition 101 by Spacy. Spacy supports adding products name in the Entity list, see page. Check out this question in stackoverflow too. But it is not easily available as you may have thought!!


Comparing models using P-value I have 2 different ML architectures for a translation task, I evaluate them using BLEU score (higher is better)

I've run them 9 times each, yielding the following scores

Architecture 1 | 36.52 |
36.27 |
35.9 |
35.22 |
37.13 |
35.53 |
35.3 |
34.14 |
35

Architecture 2 | 36.85 |
35.64 |
36.37 |
36.82 |
36.74 |
36.46 |
35.77 |
37.31 |
36.68

Means are 35.67 for the first and 36.52 for the second

I want to calculate a P-value that confirms (or not) the superiority of the second model, with alpha = 0.05

Thanks in advance

I think you can just apply a t-test or Wilcoxon test if you're don't believe in the normality of your observations.

In R:

A <- c(36.52 , 36.27 , 35.9 , 35.22 ,37.13 , 35.53 , 35.3 , 34.14 , 35)
B <- c(36.85 , 35.64 , 36.37 , 36.82 , 36.74,36.46 , 35.77 , 37.31 , 36.68)
wilcox.test(A,B,alternative="less",paired=T)
t.test(A,B,alternative="less",paired=T)

I've done paired tests under the assumption that you did the same folds for each architecture. If that is false, then do unpaired tests.

Edit: You may be interested in a question I posted a few months ago. Cross-validation for model comparison: use the same folds?


How to handle date data for Knn? I'm working on a project about predicting kickstarter project success(classification) and my dataset has many columns that could be used as features such as :
state_changed_at,
launched_at,
created_at.

Now the dataset has these features on unix timestamps.

Do I need to convert dates to some other format?

Can date data be used as a feature ?

If so how do I handle them ?

Do I keep them as a unix timestamp and try to scale/normalize them ?

Can date data be used as a feature?

Yes.

  If so how do I handle them ?

Think about your problem. Why should the date be a reasonable indicator for the success of a startup? Answering this question tells you also in which way you need to transform it.

Most often, when I use some date information for models, I do the following:

Day of the week: integer/one-hot encoding for Monday, Tuesday, ..., Sunday
Month: integer/one-hot encoding for January, February, ..., December
Hour of the day: integer/one-hot encoding for 0, ..., 23
Seconds/minutes/hours/days since XY: Usually normalized or at least scaled in some way


Visualizing a large graph (10'000 nodes) I am currently trying to find a way to visualize a graph which has 10'000 nodes, and about 20'000 edges.
I was trying this javascript library, but quickly realised that this is too computationally intensive.
What are some viable options to visualize a graph with about 10'000 nodes, and up to 100'000 edges?

If there is no library that allows the computational effort, what are some techniques which will still allow me to gain valuable insight / or represent the graph without changing too much on it's structure?

Gephi may be able to handle it. Graphistry definitely can but you may need to pay. Ultimately though, unless it fits some specific graph structures, most if not all major algorithms will produce an unintelligible mess.


How can I ensure about my R^2 score? I have a dataset with 10 columns and 158 rows. I try to predict my test dataset which is 1 columns with 158 rows.

I made cross-validations, grid-search and use ElasticNet algorithm.

Also before the evaluate the model I check the pearson correlation between 10 columns which I used for train with other 1 column which I try to predict. The correlation is not good but when I evaluate model the R^2 score is near 0.98 .

How can I be ensure that this score is confidental ? Because I didn't expect a R^2 like this. This is too high that I expect.

Thanks in advance.

$ R^2 $ shows what variation of your purpose variable is described by independent variables. So their synergetic effect could give you good better answer than their correlation. Better use $ R^2 adjusted $. Look at p-values of your variables and think about their real correlation. Are they important in real life. If they have any adequate relationship, so your regression is right.

Hope it will help.


Convert a csv file so that all attributes will become columns with corresponding element so I have this dataset in a csv file. I would like to convert it into a matrix form.

The data currently looks like this:

University, adelphi
State,  newyork
Control,  private
No-of-Students, 5-10
Male:Female Ratio:, 30:70
Student:Faculty Ratio:, 15:1
Sat Verbal,  500
....
Academic-Emphasis,  business-administration
Academic-Emphasis,  biology

I would like to convert it so that the Attributes become columns and the elements on each corresponding row. Like in a excel sheet.

Example:

University | State   | Control | No-of-Students| ect.
adelphi    | newyork | private | 5-10          |

Thank you very mcuh for your help!

Here's two different ways to do it. One reading through the file one line at a time to generate a list of dictionaries and one using pure pandas;

import pandas as pd

with open('datafile.csv', 'r') as f:
    data = []
    current_university = {}
    for line in f:
        line = [x.lstrip().strip() for x in line.split(',')]
        if (line[0] == 'University'):
            if current_university:
                data.append(current_university)
            current_university = {line[0]: line[1]}
        else:
            current_university[line[0]] = line[1]
    data.append(current_university)
df = pd.DataFrame(data)
df.set_index('University', inplace=True)

And then pandas;

df = pd.read_csv('datafile.csv', header=None)
df = df.set_index(0).T
df = df.groupby(df.columns.values, axis=1).agg(lambda x: x.values.tolist()).sum().apply(pd.Series).T.sort_values('University')
df.set_index('University', inplace=True)

Personally I think it's much easier to see what's going on in the first option but just wanted to show that it can be done using only pandas as well.


t - table and degrees of freedom I am interested if someone can review this process and give me some tips. (I don't have any data science coworkers or friends to collaborate with...) When the script (.py file) below runs, it will output/print:

The total savings are 9793 kWh + - 192 kWh
The total savings are 9793 kWh + - 2.0 %

Ultimately I am attempting calculate "precision" with my results (+ - 192 kWh AND + - 2.0 %) ..

For starters, I cant find this info online so here's a snip from a book I am referencing on calculating data set standard error, then absolute precision, relative precision.

For retrieving my critical t value, I am following these steps from machinelearningmastery.com. This is where the scipy.stats import t package is required.

I cant remember where degrees of freedom comes into play. The machinelearningmastery example, the author uses df = 10 So until I can figure out this, I left that as a default value in my code below. If someone wanted to copy and paste the code below it should run...

I cant remember from college where degrees of freedom comes into play and how to utilize it. The length of the data when its converted into df2 Pandas dataframe is 31, which represents one months data 31 days. Any tips greatly appreciated..

import os
import numpy as np
import pandas as pd
import math
from scipy.stats import t

#actual electrical energy in kWh
actual = 40000

#calculated savings in kWh, output from Keras regression
data = [np.array([[1083.8748]], dtype='float32'), np.array([[998.98773]], dtype='float32'), np.array([[1137.0487]], dtype='float32'), np.array([[1077.2798]], dtype='float32'), np.array([[926.41284]], dtype='float32'),
np.array([[1030.7125]], dtype='float32'), np.array([[1028.0048]], dtype='float32'), np.array([[523.9799]], dtype='float32'), np.array([[1125.092]], dtype='float32'), np.array([[1119.7738]], dtype='float32'),
np.array([[918.6966]], dtype='float32'), np.array([[1112.5186]], dtype='float32'), np.array([[555.6942]], dtype='float32'), np.array([[1096.5643]], dtype='float32'), np.array([[826.35657]], dtype='float32'),
np.array([[1014.35406]], dtype='float32'), np.array([[1027.6962]], dtype='float32'), np.array([[924.20087]], dtype='float32'), np.array([[1035.217]], dtype='float32'), np.array([[1008.9658]], dtype='float32'),
np.array([[970.54047]], dtype='float32'), np.array([[847.0671]], dtype='float32'), np.array([[913.5519]], dtype='float32'), np.array([[1047.0747]], dtype='float32'), np.array([[920.0606]], dtype='float32'),
np.array([[994.2266]], dtype='float32'), np.array([[991.4501]], dtype='float32'), np.array([[972.43256]], dtype='float32'), np.array([[934.8802]], dtype='float32'), np.array([[912.04004]], dtype='float32'), np.array([[1131.297]], dtype='float32')]

#convert data to pandas series then DataFrame
df = pd.Series(data)
df2 = pd.DataFrame(df)

#define sum, standard deviation, mean of calculated savings data
total = df2.sum()
totalStd = np.std(df2.values)
totalMean = df2.mean()

#calculate saings
diff = int(actual - total)

#compute precision of results, starting with standard error
stdErr = totalStd / math.sqrt(len(total.index))

# define probability & degrees of freedom
p = 0.90
degf = 10

# retrieve t value <= probability
t = t.ppf(p, degf)

#calculate absolute precision
absPrec = t * stdErr

#calculate relative precision
relPrec = (100 * (absPrec / diff)).round(decimals=1)

#print results
absSavings = f'The total savings are {int(diff)} kWh + - {int(absPrec)} kWh'
print(absSavings)

relSavings = f'The total savings are {int(diff)} kWh + - {float(relPrec)} %'
print(relSavings)

See this post on machinelearningmastery to use scipy to find critical t value. This code below seemed to work.

from scipy.stats import t

#savings calc
diff = actualKwh - total

#compute precision of results, starting with standard error
stdErr = totalStd / math.sqrt(len(total.index))

# define probability & degrees of freedom
p = 0.95
degf = len(data.index) - 1

#print("dfLen=", len(data.index))

#print("degf=", degf)

# retrieve t value
t = t.ppf(p, degf)
#print("t =", t)

absPrec = t * stdErr
#print("absPrec =", int(absPrec))

relPrec = (100 * (absPrec / diff)).round(decimals=1)
#print("relPrec =", relPrec)


Randomstate and kmeans issues I try to cluster a dataframe of 227 rows in 5 clusters using kmeans algorithm. Each time I run my code I got different labels and different clusters which make my analysis afterwards a bit tricky.

Someone told me to use the parameter: randomstate to have a reproductility in my results. I did.  I have the same clusters but still not the same label. Is it normal? Is there a way to get the same labels ?

below my code:

Test sur 5 clusters

# Data
X = df.iloc[:,1:]
myseed = 10

# Modèle kmeans à 5 clusters
km = KMeans(n_clusters=5, random_state=myseed, n_init=30)

# Fitting du modèle aux points
km = km.fit(X)
y_km = km.predict(X)

Unfortunately, there isn't a built-in option to do it. Each time you run K-Means, the labels are assigned randomly. Even if you state the same random seed. However, based on this answer in StackOverFlow, you can create a lookup table and run it after your K-Means.
from sklearn.cluster import KMeans
k = 5
kmeans = KMeans(n_clusters=k, random_state=0).fit(X)

idx = np.argsort(kmeans.cluster_centers_.sum(axis=1))
lut = np.zeros_like(idx)
lut[idx] = np.arange

With this, you will always have the same output as
In [73]: kmeans.labels_
Out[73]: array([1, 4, 1, ..., 0, 1, 0])

In [74]: lut[kmeans.labels_]
Out[74]: array([3, 0, 3, ..., 2, 4, 2], dtype=int64

When lut[kmeans.labels_] = 0, then you always have the smallest cluster and with lut[kmeans.labels_] = 4 you have the largest cluster.


Does Naive Bayes Classifier require assuming a specific distribution for training? Since NB is a generative classifier, we assume that the data points are all generated from a distribution, right?

But since we can compute the MLE of p(x_i|y) by counting (MLE of p(x_i|y) = # of (x_i,y) / # of y), do we really need a specific distribution, if at all, to model the likelihood(p(x_i|y))?

The most harrowing problem would be "interpolation". That is, computing $p(x_i | y)$ for $x_i$ and $y$ you have not yet observed, but fall within the convex hull of your observations.

You have to bin events together, or come up with a wacky interpolation scheme. This does work. However, If you assume a distribution, $p(x_i | y)$ becomes continuous, and you end up with a more statisically sound means of interpolation.


Cropping Images for Dataset Problem
I want to train HyperGAN with a set of 400+ images of people, but they aren't the specified size (32x32 pixels) for training.
Question
Is there any way/program to help cropping/resizing them so as to not do it 100% manually?

Cropping and/or resizing is very trivial using OpenCV. You could write a 10 line script to iterate over all your images and apply the transformation you require.


What's an appropriate datastore for variable length sequence data for PyTorch consumption? I have a large number of sequences - potentially hundreds of thousands - each consisting of between 100 and 10,000 items, which each consist of about 5 floats.

I need a datastore that can rapidly serve these up in batches for PyTorch training. I also need to be able to rapidly write new sequences to the store. It's like an experience replay buffer for reinforcement learning, but I want to store every single run.

These sequences should each have some attached structured metadata in a queryable format so that I can select subsets of sequences.

The best solution looks like HDF5 - either through h5py or PyTables - except that I don't know how to make it efficiently handle the variable sequence lengths. Padding isn't appropriate because of the wildly varying lengths, and storing each sequence as its own HDF5 dataset seems like a poor idea as HDF5 doesn't seem to be optimised for massive numbers of small datasets.

Ideas on my radar include Pandas multi-indexing, HDF5 region references, and building a custom metadata index system from scratch. I'm not really sure where to go from here.

Storage compactness matters - I need to be reasonably efficient with my storage space.

What I've opted for at the moment is packing all of the samples into a single HDF5 table buffer, and keeping a separate table with metadata that tracks each individual sequence's buffer position and length. This works, but I won't be marking this answer as correct because I'm not satisfied with it. This storage method is very poorly suited to editing, and it's vulnerable to loss if a bug were to cause the tables to become out of sync.


Slice rows in R based on column value I have multi-touch attribution data like:

    medium  conversion
1   organic 0
2   (none) > referral > referral > (none) > (none) > referral   0,0,0,0,0,0
3   (none)  0
4   organic > referral > referral   0,1,0
5   referral > referral > referral > referral   0,0,1,0
6   organic > referral > referral > (none) > referral   0,1,0,1,0

I'd like to remove rows with no conversions (like rows 1, 2 and 3) and I tried grepl but couldn't make it work.

For the remaining rows how to split rows so it ends with a conversion, e.g. row 4 will be organic > referral 0,1
and row 6 will split into organic > referral 0,1 and organic > referral > referral > (none) 0,1,0,1

This can be done with packages from the dplyr which is part of the tidyverse made by Hadley Wickham. The stringr package (also made by Hadley) is really helpfull in working with vectors of strings. Another package purrr is helpfull for applying functions t lists.

First, lets import the libraries and create data:

library(tidyverse)
library(stringr)

df = tibble(
  medium = c("organic", "(none) > referral > referral > (none) > (none) > referral", "(none)", "organic > referral > referral", "referral > referral > referral > referral", "organic > referral > referral > (none) > referral"),
  conversion = c("0", "0,0,0,0,0,0", "0", "0,1,0", "0,0,1,0", "0,1,0,1,0")
)

You probably have the data as strings while a list represention would be much easier to work with. The following code converts the strings to lists by splitting on ">" or "," (resp. for medium and conversion). Also read upon the %>% which is really handy for working with data frames. After converting the strings to lists the conversion column can be made boolean by mapping the == operator on each row.

df <- df %>%
  mutate(medium = str_split(medium, ">"),
         conversion = str_split(conversion, ","),
         conversion = map(conversion, `==`, "1"))

Removing the items after the first conversion is now a matter of simple indexing. A little magic is done with dplyr::lag and cumsum to get a boolean list that indicates all the steps before and during conversion. Then map is used to get (using the [ operator) all the mediums by boolean indexing with no_conversion_yet.

df <- df %>%
  mutate(no_conversion_yet = map(conversion, function(x) dplyr::lag(cumsum(x) < 1, default=TRUE))) %>%
  mutate(medium = map(no_conversion_yet, medium, `[`))

Filtering the rows that do not have a conversion is now easy. Simple remove all the rows that do not have any TRUE value in the conversion column.

df <- df %>%
  mutate(any_conversion = map_lgl(conversion, any)) %>%
  filter(any_conversion)

Voila, life made easy by the superb packages of Hadley Wickham! (Also check his paper.)


Content based recommendation on Mahout Is it possible to get  recommendation on similar product using Mahout ?

eg :

I have data set of movies with following attributes

Movie_name, Actor_1, Actor_2, Actress_1, Actress_2, Director, Theme, Language

Now given a Movie_name the system should  recommend top 3 similar movies based on the attributes .

Can this be done using Mahout. If yes how ?

Generally, this is done using spark-rowsimilarity algorithm - it is a class of content based recommendation. However, the actual process of doing this is quite simple. Here are the steps:

For each movie, convert your categorical variables into columns. For lets say that actor_1 has Brad Pitt, Daniel Craig, and Vin Diesel for different movies. This will become three columns with a 1 denoting which movies have each actor. Your movie matrix will look something like:

Movie Name, Has_Brad_Pitt, Has_Daniel_Craig, Has_Vin_Diesel, ...
MI-6      ,     1        ,       0         ,     0         , ...
Fast&Furios,    0        ,       0         ,     1         , ...
Casino Royale,  0        ,       1         ,     0         , ...

Now, to find similarity score of movies, you can just compute the cross product of the two vectors. Higher the value, more they are similar.

This can be done by the spark-rowsimilarity algorithm in one shot. You may have to do some work in encoding categorical variables.


Does the test set has to be in [0,1] range? I have standardized training set using
mean = XTrain.mean()
XTrain-=mean

std = XTrain.std()
XTrain/=std

And then used mean and std to standardize validation and test sets. The training and validation sets have values that are greater than 1 and less than zero is that okay?

Standardization centers the values around a mean of $0$ with standard deviation $1$. Therefore, having values smaller than $0$ or greater than $1$ is to be expected. If you want to make sure values are between $0$ and $1$ you need to normalize the data instead.
Here is an example of the two procedures taken from the book "Python Machine Learning" by Raschka:

Be aware though to apply the procedure to your test data with parameters obtained from the training data (in case of standardization: mean and std. dev. of train data).
Sklearn has methods for standardization and normalization which you might want to have a look at.


Classification loss function: how to implement individual weights for each observation and class The problem I have to solve is a classification problem. The costs of a misclassification are very different (but known) for the various observations, so I plan to include them by assigning weights to each observation accordingly. My issue is that additionally, the costs of misclassification are different for different classes (and these differences depend on the observations). So in theory, I would need to incorporate into the loss function weights $w_{ij}$ for each observation $i$ and class $j$. But I have no idea how to do this for example for a neural network in keras (or with any other classifier I didn't build from scratch myself).

Any ideas on how to do this would be greatly appreciated!

What you are referring to is called a weighted loss function.
In Keras:

Define a dictionary with your labels and their associated weights or just a list of the weights (by class order):

loss_weights = {0: 0.5,
                 1: 0.2,
                 2: 1.5}...
Feed the it to the compile method:

model.compile(optimizer=opt,
               loss='categorical_crossentropy',
               metrics='acc',
               loss_weights=loss_weights)

The loss value that will be minimized by the model will then be the weighted sum of all individual losses, weighted by the loss_weights coefficients.


Alternative approach for Q-Learning I have a question related to an alterative Q-Learning approach. I'd like to know if this already exists and I am not aware of it, or it doesn't exist because there are theoretical problems behind it.

Traditional Q-Learning
In traditional Q-learning, the update of the Q-value happens at every iteration. The agent is in state s, performs action a, reaches state s' and obtains reward r. The Q-value for that pair state-action is updated according to the Bellman equation. As an example, let's consider an agent exploring a grid-world. One specific cell in the world gives a positive reward. Until the agent finds the cell leading to a positive reward, the agent gets small or no rewards. Then, after reaching the cell with the high reward, all the previous cells gets updated iteratively with positive Q-values when they are explored. This means that we need many iterations, and especially if the path leading to the positive reward is very long or requires a specific set of actions, it can take a significant amount of episodes (or never be fully discovered, even if it was explored previously).

Alternative approach
My question is: why can't we update all the Q-values related to the agent at the end of the episode? That means, information is cached during iterations and when the agent terminates the episode, the Q-values gets updated for all the cells that led to that reward (or did not lead to a reward). In this case, the agent doesn't need tons of episodes to update the Q-values related to that path.
Is this approach legit? Which problem do we face following this?
The only drawback I can think of is lack of exploration, that can be solved choosing random actions according to a decreasing epsilon value.

EDIT

My problem is not computing efficiency, but it's mainly that I am using a descending learning rate, equal to $$\alpha = \frac{1}{1+number\_visits}$$
For this reason, my intuition says that updating the Q-values at the end of the episode should work better. Indeed, since the learning rate depends on the number of visits, the contribution of the early visits is way higher. Updating as soon as possible should reduce the negative impact of this aspect.

There are many variations of reinforcement learning methods. Some are tweaks to existing algorithms, some are radically different. The problems that you are concerned about with your analysis and suggestion appear to be:

Credit assignment. Whether or not a particular state or state action pair earlier in a trajectory is important to receiving a reward later on, and thus should have any value update due to new experience. This is a core problem to reinforcement learning (RL). All RL algorithms solve it, the differences are in how they deal with levels of difficulty.
Computational efficiency. Whether the algorithm makes best use of CPU cycles and doesn't waste its time calculating many updates of zero in an environment with sparse rewards.
Sample efficiency. Whether the algorithm makes best use of available data to make the most robust predictions of values it can.

Single-step Q learning does address all of these issues to at least some degree:

For credit assignment, the single step bootstrap process in Q learning will backup estimates through connected time steps. It takes repetition so that the chains of events leading to rewards are updated only after multiple passes through similar trajectories.
For computational efficiency, it is not always clear what the best approach would be. Single step Q learning updates are efficient to calculate, so even though each one only updates a small segment of a trajectory, this can be made up, in some environments, by running more updates in the same CPU time.
For sample efficiency, single-step updates on their own are not very good - they can leave many values untouched even though logically they could be updated based on new information. However, there are other approaches to address this - for instance using an experience replay table or Dyna-Q planning can take advantage of previous states and actions to re-calculate Q values without needing to see more experience.

As usual with ML problems, it is not always clear what adjustments and variations of an algorithm may work best. So you need to experiment with your environment.

You also mention this:

  Until the agent finds the cell leading to a positive reward, the agent gets small or no rewards.

This is somewhere that single-step updates can work well. In your grid world example, if you are training to minimise the number of time steps to complete an episode, then typically you will have a negative reward associated with each time step that does not complete an episode. In that case, updating your Q estimates immediately on each action can be a large benefit - it encourages exploration by progressively scoring repetition worse (as the initial agent blunders around randomly) - meaning that the agent tries to find new states and actions that are still scored at original default levels. You won't get that searching behaviour if you wait until the end of the first episode before updating the value tables.

Whether or not this is an issue will depend on the environment, but such time related goals are relatively common, so it is worth bearing in mind.

  Alternative approach
  My question is: why can't we update all the Q-values related to the agent at the end of the episode?

This is a well known common approach in RL. The simplest variant is called Monte Carlo Control, and it uses trajectories from whole episodes to update values of all state/action pairs seen in those trajectories.

More sophisticated approaches blend between single-step Temporal Difference learning (of which Q learning is one example) and whole episode Monte Carlo methods. These are n-step methods (n-step Q-learning is one example) and TD($\lambda$) methods (Q($\lambda$) is one example). These are relatively popular as they combine best features of, and avoid compromises of, the two extremes of whole episode vs single step.

  Is this approach legit?

Yes, in fact as described above, variations of it are used routinely.

  Which problem do we face following this?

When using whole episodes, you face the following problems:

High variance. If either your policy or your environment has randomness, then the final trajectory and reward in each episode will vary a lot. Which means to get reliable estimates of value you may need more samples, not less as you hoped.
Off-policy learning is more complex. Q-learning in particular is an off-policy method, meaning it learns values of its best guess at an optimal policy (called the target policy) whilst still exploring using a non-optimal policy (called the behaviour policy). With single-step updates this difference can be ignored as you always calculate the updated value estimate for an action you've taken (you don't care what policy generated this) using the maximising action for the next step (so you don't use the action actually taken). When you extend to calculating value updates over multiple steps, you need to adjust for differences between behaviour and target policies. If you look at Q($\lambda$) algorithm you will see it copes with this by resetting something called the eligibility trace (which you can think of as a vector that explicitly tracks credit assignment) whenever the agent takes a non-optimal action according to the current value estimates.

There is also Off-policy Monte Carlo Control, which uses importance sampling to decide whether and how much to use particular updates - for learning optimal control typically this will only update from the ends of episodes after the last exploratory action that the agent took.

  The only drawback I can think of is lack of exploration, that can be solved choosing random actions according to a decreasing epsilon value.

Usually you will want some initial high exploration that settles down closer to the predicted optimal policy later on. This or something similar should be standard practice in many environments. However, it would not solve the search problem you mention - for that you need an exploration system that is aware of when states and values are repeated themselves over multiple steps within a single episode. There's more than one way to solve that, too, but the single-step Q learning approach is quite nice when it works because it is a natural consequence of the algorithm.


Why does the MAE still remain, at all? This may seem to be a silly question. But I just wonder why the MAE doesn't reduce to values close to 0.

It's the result of an MLP with 2 hidden layers and 6 neurons per hidden layer, trying to estimate one outputvalue depending on three input values.
Why is the NN (simple feedforward and backprop, nothing special) not able to maybe even overfit and meet the desired training values?
Costfunction = $0.5 (Target - Modeloutput)^2$
EDIT:
Indeed I found an inconsistency in the inputdata.

Already cheering, I was hoping to see a better result, after fixing the issue with the input data. But what I got is this:

I'm using a Minibatch-SGD and now I think it might get trapped in a local minimum. I read about Levenberg-Marquardt algorithm, which is said to be stable and fast. Is it a better algorithm for global minimum detection?

There can be other reasons related to the model but the most simple explanation is that the data contains contradicting patterns: if the same features correspond to different target values, there is no way for any model to achieve perfect performance.
Let me illustrate with a toy example:
x   y
0   4
0   4
1   3
1   4
1   3
1   3
2   8
2   7
2   7
2   6

The best a model can do with this data is to associate x=0 with y=4, x=1 with y=3 and x=2 with y=7. Applying this best model to the training set would lead to 3 errors and the MAE would be 0.3.


Assuming you have access to dplyr package, you can do the following.

data %>% group_by(session_id) %>% summarise(start_time = min(timestamp), end_time = max(timestamp), unique_items=length(unique(item_id)))


XGBoost increase the error when changing evaluation function I have changed the eval function of XGBoost to rmsle and the optimisation increase the error after the iteration [2] instead of decreasing it. If I change to the default eval function, RMSE, this does not happen.

This is the code of RMSLE used:

def evalerror(preds, dtrain): # this is compatible with DMatrix
    labels = dtrain.get_label()
    assert len(preds) == len(labels)
    labels = labels.tolist()
    preds = preds.tolist()
    terms_to_sum = [(math.log(labels[i] + 1) - math.log(max(0,preds[i]) + 1)) ** 2.0 for i,pred in enumerate(labels)]
    return 'error', (sum(terms_to_sum) * (1.0/len(preds))) ** 0.5

This is the parameters of XGBoost used:

param = {'bst:max_depth':1, 'bst:eta':0.025, 'silent':False, 'objective':'reg:linear','eval_metric':'rmse' }
bst = xgb.train( param, d_train, num_rounds,early_stopping_rounds=20, evals=eval_list, verbose_eval=True, feval=evalerror)

and this is the evaluation:

[0] eval-error:0.836219 train-error:0.835095
Multiple eval metrics have been passed: 'train-error' will be used for early stopping.

Will train until train-error hasn't improved in 20 rounds.
[1] eval-error:0.809301 train-error:0.806747
[2] eval-error:0.792647 train-error:0.78908
[3] eval-error:0.803355 train-error:0.798805
[4] eval-error:0.803261 train-error:0.79835
[5] eval-error:0.809352 train-error:0.804283
[6] eval-error:0.810453 train-error:0.805126
[7] eval-error:0.811059 train-error:0.805646
[8] eval-error:0.815261 train-error:0.809722
[9] eval-error:0.820237 train-error:0.814521
[10]    eval-error:0.823378 train-error:0.817408
[11]    eval-error:0.824981 train-error:0.81868
[12]    eval-error:0.826607 train-error:0.820176
[13]    eval-error:0.827813 train-error:0.821358
[14]    eval-error:0.827625 train-error:0.821007
[15]    eval-error:0.823347 train-error:0.816547
[16]    eval-error:0.824362 train-error:0.81752
[17]    eval-error:0.82529  train-error:0.818321
[18]    eval-error:0.824621 train-error:0.817463
[19]    eval-error:0.824103 train-error:0.816766
[20]    eval-error:0.814759 train-error:0.807234
[21]    eval-error:0.807961 train-error:0.800186
[22]    eval-error:0.808398 train-error:0.800246
Stopping. Best iteration:
[2] eval-error:0.792647 train-error:0.78908

It may be the case that I need to adjust my objective function to this evaluation metric?

If your goal is to minimize the RMSLE, the easier way is to transform the labels directly into log scale and use reg:linear as objective (which is the default) and rmse as evaluation metric. This way XGBoost will be minimizing the RMSLE direclty. You can achieve this by setting:

dtrain = DMatrix(X_train, label=np.log1p(y_train))

where np.log1p(x) is equal to np.log(x+1).

When you want to make your prediction in the original space, you will need to compute the inverse transform of np.log1p, that is to say np.expm1:

predictions = np.expm1(bst.predict(dtest))

If you are just interested into monitoring the RMSLE through the training of your XGBoost which actually is minimizing the RMSE, then you should expect to see the RMSLE behave a little strangely as it is not what you are minimizing.


Is there a difference between np.matrix(np.array([0,0])) and np.matrix([0,0])? I was reading this code, for implemnting linear regression from scratch:

# convert from data frames to numpy matrices
X = np.matrix(X.values)
y = np.matrix(y.values)
theta = np.matrix(np.array([0,0]))

When I came accross this line :  np.matrix(np.array([0,0]))

I was wondering why didn't the person just write np.matrix([0,0]).

I ran both in jupyter notebook and got the same output:

theta = np.matrix([0,0])
theta2 = np.matrix(np.array([0,0]))
print(theta,theta2,type(theta),type(theta2))

Output:[[0 0]] [[0 0]] <class 'numpy.matrix'> <class 'numpy.matrix'>

Is there a difference between the two? Does the extra np.array somehow part add to the functionality of theta? Will the final code function properly if I replace the former with the latter?

Thanks.

Edit: Is this the right place to ask this question? I am new here...

No, they're absolutely the same.

In this case there is absolutely no difference apart from perhaps a trivial amount of processing time.  This is all open source code so we can just read it:

The relevant part of numpy for us here is the matrix constructor (yes, np.matrix is a python class below the hood).  In a summary from the NumPy code we see:

class matrix(N.ndarray):
# ...
    def __new__(subtype, data, dtype=None, copy=True):
    # ...
        if isinstance(data, N.ndarray):
            if dtype is None:
                intype = data.dtype
            else:
                intype = N.dtype(dtype)
            new = data.view(subtype)
            if intype != data.dtype:
                return new.astype(intype)
            if copy: return new.copy()
            else: return new
        # ...
        arr = N.array(data, dtype=dtype, copy=copy)
        ndim = arr.ndim
        shape = arr.shape
        # some extra checks
        ret = N.ndarray.__new__(subtype, shape, arr.dtype,
                                buffer=arr,
                                order=order)
        return ret

What we give as the data argument is literally what we give to to np.matrix().  Therefore we can draw for the two cases:

np.matrix([0, 0])

The python interpreter builds two integers: 0 and 0.
The python interpreter builds a list from the pointers to the two integers.
The python interpreter evaluates the matrix constructor with the list as data.
The if in the constructor is not executed, instead an np.array is build from the list.
Inside the array constructor data types are checked.
The final array is returned (the second array constructor perform much less work because it is passed buffer=)

np.matrix(np.array([0, 0]))

The python interpreter builds two integers: 0 and 0.
The python interpreter builds a list from the pointers to the two integers.
The python interpreter evaluates the array constructor.
The resulting array is passed as data to the matrix constructor, and the if is executed.
Within the if the data type is taken from the existing array.
The array is copied an returned.

Both ways execute pretty much the same number of constructors and lines of code.  One could argue that copying the array (the copy= argument) may be a slow operation.  Yet, given the fact that to have enough data for array.copy() to be slow one would first need to construct a full python list of that size, the copy() time is negligible compared to the list construction.  In other words, both methods need to construct the list - because python will always evaluate arguments before passing them - which is the slowest part of the execution of this code.

As for the return value, they're absolutely and completely the same.  Most of the code within the constructor summarized (and linked) above is to make sure that you get the same return if you give equivalent input.

P.S. (Unrelated Note) If one starts reading data from a file (or any other external source) the picture changes.  If one reads directly into an array without going through the python list phase, that method is bound to be much faster.  The processing bottleneck is the python list, if one can avoid that things will go faster.


Training a model sample by sample I'm training a Scikit model but it seems that in all examples, they call the fit method on the entire training set. What I want to do however is call it per sample (i.e. looping through all samples). This has multiple reasons but most importantly are

MemoryError with my huge trainingset
Training with new data instead of recompiling entire model

Yet when I loop and call the fit model self.gnb = self.gnb.fit(sample.data,labels) and then debug, the gnb model only has one class (namely the last one). So how should I approach this?

In the general case this is impossible because standard learning algorithms require all the instances at once in order to compute the model. They usually work by making calculations based on the set of instances, then only the results of these calculations are stored as a model (for instance a linear regression classifier stores only the final weights which are needed to make a prediction).

It is however possible to train a model incrementally:

Very often this is done by simply adding the new instances to the original training set, then re-train a new model on the full set of instances (new and old ones together).
There are also actual incremental learning algorithms, including adaptations of some common learning methods. However this is not very common, I'm not aware of these being implemented in the most standard ML libraries that I know of.


Train, test split of unbalanced dataset classification I have a model that does binary classification.

My dataset is highly unbalanced, so I thought that I should balance it by undersampling before I train the model. So balance the dataset and then split it randomly. Is this the right way ?  or should I balance also the test and train dataset ?

I tried balancing only the whole dataset and I get train accuracy of 80% but then on the test set I have 30% accuracy. This doesn't seem right ?

But also I don't think that I should balance the test set because it could be considered as bias.

What is the right way to do this?

Thanks

UPDATE: I have 400 000  samples, 10% are 1s and 90% 0s. I cannot get more data. I tried to keep the whole dataset but I don't know how to split it into train and test set. Do I need the same distribution in the train and test dataset ?

Best way is to collect more data, if you can.

Sampling should always be done on train dataset. If you are using python, scikit-learn has some really cool packages to help you with this. Random sampling is a very bad option for splitting. Try stratified sampling. This splits your class proportionally between training and test set.

Run oversampling, undersampling or hybrid techniques on training set. Again, if you are using scikit-learn and logistic regression, there's a parameter called class-weight. Set this to balanced.

Selection of evaluation metric also plays a very important role in model selection. Accuracy never helps in imbalanced dataset. Try, Area under ROC or precision and recall depending on your need. Do you want to give more weightage to false positive rate or false negative rate?


Keras model giving error when fields of unseen test data and train data are not same I have created a simple Keras deep learning model in python. Total no of variables in training are 195 while in unseen test data are 181.All input fields are categorical(converted by one hot encoding). Since unseen test data has some different categories thats why after one hot encoding fields are not matching with train.

So during predict step on unseen test data, model gives the following error. Is there any way out?

ValueError: Error when checking input: expected dense_30_input to have shape (195,)
but got array with shape (181,)

As others before me pointed out you should have exactly the same variables in your test data as in your training data.

In case of one-hot encoding if you have unseen categories in your test data your model doesn't know how to handle them it was not trained on those variables. In that case during data preparation you shall create all the variables that you had during training with the value of 0 and you don't create new variable for the unseen category.

I think your confusion and the differing number of variables come from the function that you use to do the one-hot encoding for you. Probably you run them on the two datasets separately and it will only create the variables that it founds in the specific datasets.
You can overcome on it by using label encoder or onehotencoder transformer from scikit-learn that will save inside its obeject the original state and in every transformation it will recreate exactly the same structure.

UPDATE to use sklearn onehotencoder:

from sklearn.preprocessing import OneHotEncoder
encoder = OneHotEncoder(handle_unknown='ignore')
encoder.fit(train_categorical_data)
encoded_train=encoder.transform(train_categorical_data)
encoded_test=encoder.transform(test_categorical_data)

You can save the encoder to use it later. See more about it in the official documentation.


Mean encoding in times series Considering a TS from 1-5 blocks and using 1-4 block as train data.
Is it invalid to build mean encode on whole train data / or I should mean encode block 1 / block 1-2 / ... / block 1-2-3-4 ?
Edit 1 : By mean encoding, I mean to encode my categorial features by taking target's mean on associate values.

The main rule when doing mean encoding is not using the test data. As long as you don't use the test data, you'll be mostly alright.
That being said, if you encode with all your training data you're probably going to see a drop of performance in the test set, which will not happen if you encode in a cross-validated fashion. So, the second option has some advantages with respect to the first one, but the first one is simpler. Still, I'd go for the second one.
In addition, you can add regularisation to the mean as done in the python package category-encoders, and it will be more robust to overfitting. This package allows for other options like leave one out encoding, that ensures that a sample's target is not used to compute the encoding for that sample.


ffill missing data based on the sort order of another column I have a column with missing data that I need to imput.  Column is called 'Bandwidth'.  There is a relationship between the Bandwidth column, and another column called 'Age'.  As Age increases, so does Bandwidth.
I want to ffill Bandwidth, but I have to do it based on the value in Age.  I have over 10k records, so there is too much data to create a dictionary. I have scoured the internet, and so far am not able to find a solution.
In the simplest form of what I want to do is this:
df.sort_values(['Age'])
df['Bandwidth'].fillna(method='ffill',inplace=True)
It did fill the bandwidth, based on the'sort' of bandwidth,  but not based on the Age sort.
The yellow ones were filled with the data from the green ones, but as you can see, the 'Age' is random.

But this is what I am needing it to do... (sort by age, then fill bandwidth from the above value.)

Since you mentioned that there is a linear relationship between Age and Bandwidth, you could train a simple linear model i.e linear regression on your dataset to estimate the bandwidth based on Age. Once you have figured out the relationship you can use it to impute the missing bandwidths.


Metric/loss for bin classification I have a model that has to classify inputs into one of 45 categories but those categories actually represent bins (e.g. bins 1, 2 and 3 are between 1 and 10, 11 and 20, 21 and 30 respectively). What I would like is my model to classify properly values into bins but I am not too upset if it puts 19 into bin #3 even in bin #1. What is the loss function that would measure distance from the correct category and will score the classification in neighbouring bins with weight of say 1/n where n is the distance to the correct bin.

mind you that what I am looking for is different from the top_k metric that keras has.

It looks like you should redefine your task as a regression problem instead of a classification problem, because your target variable is numerical.

The performance will be much better and it will avoid the need for a questionable pseudo-regression-like measure. Mean absolute error and mean squared error are standard regression evaluation measures.


Genetic Optimization, Heuristic regarding choosing the number of generations and population size I have a simple model with some fitness function that I'm trying to max out.
This model have ~20 variables, each about ~15 options.
Is there a heuristic formula or a study of some sort that can guide me regarding the best:

Population size
Generations

There are other options of course such as Mutation rate / Crossover rate but I mainly interested in the 2 above.

Is there a heuristic formula or a study of some sort that can guide me regarding the best

Maybe, but I don't know any. My approach is to run a few experiments and observe the speed of convergence with different values for the size of the population, typically in the range 50 to 500. My experience is that usually this parameter doesn't have a big impact (especially compared to rate of mutation/crossover) so I tend to stay on the low side for efficiency reasons.

It's not technically required to specify the number of generations provided there is a criterion to check convergence, either manually or programmatically. I think it's more important to have such a criterion, because there's a always a risk that the predefined number of generations won't be sufficient.


How should I create a single score with two values as input? I have two series of values, a and b as inputs and I want to create a score, c, which reflects both of them equally.  The distribution of a and b are below

In both cases, the x-axis is just an index.

How should I go about creating an equation c = f(a,b) such that a and b are (on average) represented equally in c?

Edit: c = (a+b)/2 or c = ab will not work because c will be too heavily weighted by a or b.

I need a function, f, where c = f(a,b) and c' = f(a + stdev(a),b) = f(a, b + stdev(b))

If you're looking for something where A and B are equally represented, consider trying something like Z score normalization (or standard score):

c = (a-u_a)/sigma_a + (b-u_b)/sigma_b

That score equally represents the two, but would be on a smaller scale.  It really shouldn't matter since the numbers are arbitrary, however, if you need to scale it up, you could do something like:

c2 = (sigma_a+sigma_b)*(c) + u_a + u_b


CycleGAN: Generator losses don't decrease, discriminators get perfect So I´m training a CycleGAN for image-to-image transfer.

The problem is: while the discriminator losses decrease, and are very small now, the generator losses don't decrease at all.

The generator loss is:

1 * discriminator-loss + 5 * identity-loss +
10 * forward-cycle-consistency + 10 * backward-cycle-consistency

Somehow the discriminator models, although not being that big, become very strong quite fast, while the generators don't seem to learn much at all.

What do I have to change to resolve this problem and make the generators more effective?

This usually happens when your discriminator starts to just understand that the difference between your generated(fake) and real samples only differ by some n/255 factor. This is easy to learn, and if your generator cannot properly generate "good" samples then within a few 10-100 epochs your Discriminator will learn this highly trivial transformation.
Now when we reach a point where the generator cannot converge from pure random output it will get stuck generating the same random output with a constant high loss. Since I don't know the exact architectures of your network there are some solutions out of the box that has helped me.
Try adding some gaussian noise at the very beginning of your Discriminator network, this will make sure your Discriminator will not learn trivial transformations right away. I cannot comment on this channel yet, so if you can post your network architecture then I could possibly give you a better answer.


What features most contribute for label y? On a neural network, how can I find out what features most contribute for some label (the dependent variable y)?

I have good news and bad news.

The bad news is that due to the nonlinear nature of neural networks, the same feature will contribute differently depending on the other features.

Here's a simple nonlinear function: f(a,b,c) = y = ba^3 + bc + c.  One could argue that 'a' contributes most to this function, but when b is negative, a more positive a might make things worse.  When b is zero, c contributes most.  When b is negative, a very negative a will contribute most.

The good news is that you may be able to visualize some simple single-layer networks.  There's a fun trick for this used in deep learning.

Assuming you've got x for your examples, W for your weights, and y for your outcomes, set the feature you want to show in y to 1 and the rest to zero, then multiply by W-transpose.  You should get some output values for x which show you how connected everything is.  It's not a perfect solution, but if you get some huge values coming out after the multiplication, you can say those values are highly-connected for y.  (Usually.)


How to approach this data set for linear regression? I am new to data science and am currently working on a data science project and have to answer a few questions about the following data set with 18k data points:
https://www.kaggle.com/karangadiya/fifa19

The question I have to answer is as follows:
To which extent can you determine the value of a player using the players most important features?

Features in this case being the columns from 'Crossing' to 'GKReflexes'. I grouped up all the players as follows:
Att = ['RF', 'LF', 'RS', 'LS', 'RS', 'RW', 'LW', 'CF', 'ST'] # attack positions
Mid = ['LAM', 'CAM', 'RAM', 'LM', 'LCM', 'CM', 'RCM', 'RM', 'LDM', 'CDM', 'RDM'] # midfield positions
Def = ['LCB', 'CB', 'LB', 'RCB', 'RB', 'LWB', 'RWB'] # defense positions
GK

I already looked at the data and was able to determine that certain player features are much higher for certain positions.

I plan on using multiple linear regression to answer the research question, but I'm not sure on how to split the data into a train and test set.

I'm thinking of one of the following approaches:

First split the large data set into smaller data sets for each position (GK, DEF, ATT and MID) and then make a train and test set for each position.

or

Leave the large dataset as it is with all of the positions combined and split that into a training and test set.

I am currently leaning more towards the first approach because I think I'll be able to better determine the price of the player using the features that are important to the players position, but I am not sure if that's the correct way to go at it.

So in the first approach you want to manually choose some features that seem more important?

If so, I think you might not be able to accurately diagnose the most important features, I think it is better to first choose the most impactful features using an algorithm like backward-elimination, etc to figure out what are the most important features, then ignore the less important columns and split the dataset to training and test error.


Wilcoxon W value different from python I use data from https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test, W value is 9.
But, for the following code W value is 18,
what is the reason?

# Wilcoxon signed-rank test
from numpy.random import seed
from numpy.random import randn
from scipy.stats import wilcoxon
# seed the random number generator
seed(1)
# generate two independent samples

a=np.array([125,115,130,140,140,115,140,125,140,135])
b=np.array([110,122,125,120,140,124,123,137,135,145])

# compare samples
stat, p = wilcoxon(a,b)
print('Statistics=%.3f, p=%.3f' % (stat, p))
# interpret
alpha = 0.05
if p > alpha:
    print('Same distribution (fail to reject H0)')
else:
    print('Different distribution (reject H0)')

If you look at the wikipedia page of Wilcoxon signed-rank test, under the section of the original test, it's mentioned that

  The original Wilcoxon's proposal used a different statistic. Denoted by Siegel as the $T$ statistic, it is the smaller of the two sums of ranks of given sign; in the example given below, therefore, $T$ would equal $3+4+5+6=18$


A Loss of 55.2164 with a sparse_categorical_crossentropy in a sequential neural network? I'm following Aurélion Géron's book on Machine Learning.
The following code tries to evaluate a neural network with  a sparse categorical cross entropy loss function, on the Fashion Mnist data set. How come I get such a strange value for loss?

import tensorflow as tf
from tensorflow import keras
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

fashion_mnist=keras.datasets.fashion_mnist

(X_train_full,y_train_full),(X_test,y_test)=fashion_mnist.load_data()

X_valid , X_train = X_train_full[:5000]/255.0 , X_train_full[5000:]/255.0
y_valid , y_train = y_train_full[:5000] , y_train_full[5000:]

model=keras.models.Sequential()
model.add(keras.layers.Flatten(input_shape=[28,28]))
model.add(keras.layers.Dense(300,activation="relu"))
model.add(keras.layers.Dense(100,activation="relu"))
model.add(keras.layers.Dense(10,activation="softmax"))

model.compile(loss="sparse_categorical_crossentropy",optimizer="sgd",metrics=["accuracy"])
history=model.fit(X_train,y_train,epochs=30,validation_data=(X_valid,y_valid))

#some instructions with *history* for plotting a graph

model.evaluate(X_test,y_test)

and the output I get is [55.21640347443819, 0.8577]

How come I get a loss over 1?

The reason you are getting a very high loss has to do with the normalization you are performing. You are normalizing both the X_train and X_valid, however you do not normalize X_valid. Adding normalization of X_valid to your code the final loss for X_valid is much lower.

fashion_mnist = keras.datasets.fashion_mnist

(X_train_full,y_train_full), (X_test,y_test) = fashion_mnist.load_data()

X_valid , X_train = X_train_full[:5000]/255.0 , X_train_full[5000:]/255.0
y_valid , y_train = y_train_full[:5000] , y_train_full[5000:]

# Normalizing X_test as well
X_test = X_test/255.0

model = keras.models.Sequential()
model.add(keras.layers.Flatten(input_shape=[28,28]))
model.add(keras.layers.Dense(300,activation="relu"))
model.add(keras.layers.Dense(100,activation="relu"))
model.add(keras.layers.Dense(10,activation="softmax"))

model.compile(loss="sparse_categorical_crossentropy",optimizer="sgd",metrics=["accuracy"])
history = model.fit(X_train,y_train,epochs=30,validation_data=(X_valid,y_valid))

model.evaluate(X_test,y_test)
# [0.328696608543396, 0.8866999745368958]


How to approach deep learning CV/Resume parser using Convolutions? I'm currently looking to invest some time on how to make a resume parser using deep learning.

I need some initial ideas (or) approaches on how to put together things at first, where to start. If you have any good approaches on handling resume parsing for multiple type resume structures please let me know.

Also to start here's my approach I was thinking, do let me know if you think this works!

1) First I will take a 100 resume as a image(i.e., jpg) file.
2) I will create bounding box around projects/education area and create my training data.
3) After training I'll try to predict it on test data.

Is this an ideal approach? Are there any good approaches you can suggest which works based on any paper?

Much appreciated. Thanks.

Typically, resumes are not images. Almost all resumes are Microsoft Word or pdf. Given those document formats, a deep learning parser should be sequence-based (e.g., Recurrent Neural Network (RNN) or Long Short Term Memory (LSTM)).

To apply Deep Learning, you'll need many thousands of examples with each section labeled. There is HR-XML (Human Resources - Extensible Markup Language) which are the industry standards for labels of resume sections.


Don't understand why I get an inverse ROC curve for SVM (Python) I build an SVM classifier but get an inverse ROC curve. The AUC is only 0.08. I've used the same datasets to build a Logistic Regression classifier and a Decision Tree classifier, and the ROC curves for them look good.
Here are my codes for SVM:
from sklearn.svm import SVC
svm = SVC(max_iter = 12, probability = True)
svm.fit(train_x_sm, train_y_sm)
svm_test_y = svm.predict(X = test_x)
svm_roc = plot_roc_curve(svm, test_x, test_y)
plt.show()

Could anyone tell me what is wrong in my codes?

For any classification problem if AUC<0.5, you are not performing better than random(0.5).
Reason could be:

Your classifier is over-fitted on the training set and performs very poorly on the test set.
Your test sample might be very small.
Your classifier is giving you the probability that the class is -1. Thus, you get a prediction (close to) 0 for a class 1, and 1 for a class -1 prediction. If your ROC method expects positive (+1) predictions to be higher than negative (-1) ones, you get a reversed curve.

A valid strategy is to simply invert the predictions as:
invert_prob=1-prob

Reference: ROC


How to add Earlystopper in Classifier Model I have designed the following Binary Classifier Neural Network Model for a task. I want to add an early stopper to the model so that the model stops at an epoch where it has stopped learning significantly. How can I do that?
Model
X = badge1_data[['1','2','3','Score']]
y = badge1_data['APR']

#Standardizing the Input Features
scaler = StandardScaler()
X = scaler.fit_transform(X)

#Train Test split
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.3)

#Create Model
model = Sequential()
model.add(Dense(4,
                  input_dim=4,
                  kernel_initializer='normal',
                  activation='relu'))

model.add(Dense(2,
                  kernel_initializer='normal',
                  activation='relu'))

model.add(Dense(1,
                  activation='sigmoid',
                  kernel_initializer='normal'))

#Compile Model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

#Fit Model
model.fit(X_train, y_train, epochs = 5000, validation_split = 0.3, verbose=0, batch_size=256)

#Make predictions and convert to binary value
pred_train = model.predict(X_train)
pred_test = model.predict(X_test)

#ROC AUC Score
print('Train AUC = {:}'.format(roc_auc_score(y_train,pred_train)))
print('Test AUC = {:}'.format(roc_auc_score(y_test,pred_test)))

#Accuracy
print('Train Accuracy = ',accuracy_score(y_train,pred_train.round()))
print('Test Accuracy = ',accuracy_score(y_test,pred_test.round()))

Early stopping can be achieved with Keras callbacks.
early_stop= tf.keras.callbacks.EarlyStopping(monitor='loss', patience=3)

history = model.fit(X, y, callbacks=[early_stop])

patience: Number of epochs with no improvement after which training will be stopped.
To define the meaning of "significant learning stopping", you will have to try with the parameters of Callback. e.g.
monitor="val_loss",  min_delta=0, patience=0, mode="auto", baseline=None, etc.
Keras_API


Distributed PCA or an equivalent We normally have fairly large datasets to model on, just to give you an idea:

over 1M features (sparse, average population of features is around
12%);
over 60M rows.

A lot of modeling algorithms and tools don't scale to such wide datasets.

So we're looking for a dimensionality reduction implementation that runs distributely (i.e. in Spark/Hadoop/ etc). We think to bring number of features down to several thousand.

Since PCA operate on matrix multiplication, which don't distribute very well over a cluster of servers, we're looking at other algorithms or probably, at other implementations of distributed dimensionality reduction.

Anyone ran into similar issues? What do you do to solve this?

There is a Cornell/Stanford Abstract on "Generalized Low-Rank Models" http://web.stanford.edu/~boyd/papers/pdf/glrm.pdf that talks specifically into this:

page 8 "Parallelizing alternating minimization" tells how it can be
distributed;
also page 9 "Missing data and matrix completion" talks
how sparse/ missing data can be handled.

GLRM although seems to be what we are looking for, but we can't find good actual implementations of those ideas.

Somebody wrote a version for Spark (e.g.
https://github.com/rezazadeh/spark/tree/glrm/examples/src/main/scala/org/apache/spark/examples/glrm
but it looks more of a proof of a concept rather than a version that
we could use in production);
H2O has a version of GLRM
http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/glrm.html
but it actually doesn't scale to our dataset size (see above).

Update 7/15/2018: Another Abstract is Fast Randomized SVD from Facebook (read here http://tygert.com/spark.pdf ) and also idea to do low-rank matrix approximation using ALS - http://tygert.com/als.pdf . Although there is no clear way how to use them now - see discussion at https://github.com/facebook/fbpca/issues/6

Any other ideas how to tackle this? Other available GLRM or other distributed dimensionalaity reduction implementations?

There is principal component analysis (PCA) in Spark's
Machine Learning Library (MLlib).


Determine statistically whether new product cannibalise old product by using data Assume that $A$ is a cab company which offers online cab booking through their standard account.

Recently, the company launched a pre-paid premium account with features such as discounted rides,special offers, guaranteed pick up and low waiting time.

A survey is conducted to public and it shows that the response to the new premium account is encouraging.

  Question: Does the encouraging response stem from new customers or existing customers?

We are given a data consisting of time (booking is placed), pick-up and drop-off location, journey fees, ride type (premium or standard), customer demographic and pricing information.

I am totally lost here. What can we do to answer the question above? Any hint is appreciated.

If I were asked to determine whether customers have any fixed budget to transport, how can I solve it using data above?

My plan is to split the spending into monthly sales, then calculate their means. Let's say we have $12$ sample means and sample variances.
Since the sample variances might not be equal, I employ Welch ANOVA test.
Is this sufficient?

You do not have the data to directly answer the question if a ride for a given service is new customer or old customer. You need to have a customer id to properly attribute if there is new growth or just service switching.

Any statistics you run will, at best, show a correlation between new service and increased revenue. There can be no causation attributed. For example, the changes could be due to seasonal differences or other factors unrelated to the new service.


How does a Recommender System recommend movies to a New User? Consider a New user which has never rated any movie on the Website or the System has never seen the user.

How does the System recommend Movies to the User and based on what ?
How will we evaluate the Recommendation , is the recommendation accurate or appropriate and how confident the system is about the Recommendation .

In general, a safe default option for recommender systems is to recommend the most popular product, so two options for the recommendation:

simply recommend the most popular movie in a time window around the current moment (maybe taking into account momentum),
do the same but marginalizing over account info, e.g. age, gender, country, language; also, IP location is an option to get such type of info.

Confidence assessment can be done based on recommendation success rate over the results of the same recommendation to previous users with the same profile characteristics, if such data is available.


Reading file from Object Storage in IBM DSX I'm just getting started with DSX. I have two files in my Object Storage that I want to read in my Jupyter notebook:

Warehousing-data.json
Warehousing-sales_data- nominal_scenario.json

I have the credentials. Now I need the calling sequence.

Here is what have on my local copy of the notebook.

resource_path= "C:\\...\\warehousing\\"
Warehousing_data_json = "Warehousing-data.json"
Warehousing_sales_data_nominal_scenario_json = "Warehousing-sales_data- nominal_scenario.json"

resp = client.execute(input= [
    {'name': Warehousing_data_json,
     'filename': resource_path + Warehousing_data_json},
    {'name': Warehousing_sales_data_nominal_scenario_json,
     'filename': resource_path + Warehousing_sales_data_nominal_scenario_json}])
    ...

What is the equivalent when the files are located in Object Storage?

Found a sample "Load data from different sources" at https://apsportal.ibm.com/exchange/public/entry/view/07db16e78e1722931e27c074ebe343ae
It works.


EasyEnsemble explaination Could someone please explain how the EasyEnsemble algorithm works? Im using it for a prediction model for imbalanced minority class.
Please don't refer me to this paper, as it makes no sense to me.
EasyEnsemble and Feature Selection for Imbalance Data Sets
Im using the algorithm in Pandas, with the UnbalancedDataset library which is on GitHub  UnbalancedDataset
I get an array of matrices as O/P, I don't know how to use this in the end, to train with random forests.
Thanks

The toolbox only manage the sampling so this is slightly different from the algorithm from the paper.

What it does is the following: it creates several subset of data which are balanced. These subsets are created by randomly under-sampling the majority class. That is what you are getting from the toolbox.

To obtain what in the paper, you need to train an AdaBoost classifier for each subset. Thus, what you get is an ensembles of ensembles.

Hope that's help.


Data-preprocessing for Machine Learning model I am confused about how to preprocess range based category such as age, tumor-size & inv-nodes. Should I take an average of the limits, as in - 14.5, 24.5 and so on or do one hot encoding of the co-domain range. Taking one hot encoding would increase the no. of features significantly. Because, each attribute has co-domain as following:

age: 10-19, 20-29, 30-39, 40-49, 50-59, 60-69, 70-79, 80-89, 90-99.
tumor-size: 0-4, 5-9, 10-14, 15-19, 20-24, 25-29, 30-34, 35-39, 40-44, 45-49, 50-54, 55-59.
inv-nodes: 0-2, 3-5, 6-8, 9-11, 12-14, 15-17, 18-20, 21-23, 24-26, 27-29, 30-32, 33-35, 36-39.

You should definitely not use one hot encoding with values which represent numbers, as this removes the natural order between your intervals.

So these values should be represented as numbers:

Either with the average of the limits indeed
Or a simple integer encoding of the intervals, e.g. tumor sizes 0-4, 5-9, 10-14,... would be represented as 0,1,2,...


How does xgboost work with linear booster? I know how to implement linear objective function and linear boosts in XGBoost. My concrete question is: when the algorithm it fits the residual (or the negative gradient) is it using one feature at each step (i.e. univariate model) or all features (multivariate model)?

Any reference to documentation about the linear boosts in XGBoost will be appreciated.

Thanks!

EDIT: Linear boosts can be implemented in XGBoost by setting the 'booster' parameter to 'gblinear'. See:
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3885826/
for useful information on linear boosting. Note that I am not speaking about the objective function (which can be also linear) but about the boosts themselve.

Refer to the authors' paper on xgboost here - XGBoost: A Scalable Tree Boosting System.

Equation 3 on page 2 mentions that in each step all the predictors are used to greedily fit the next additive tree.
However, since column sub-sampling is also employed to prevent over-fitting, not all features may actually be used for each step, see section 2.3 of the same paper.

Hence, the specific answer to your query is that the algorithm uses all features to fit the residual.


How does batch normalization work for convolutional neural networks I am trying to understand how batch normalization (BN) works in CNNs. Suppose I have a feature map tensor $T$ of shape $(N, C, H, W)$
where $N$ is the mini-batch size,
$C$ is the number of channels, and
$H,W$ is the spatial dimension of the tensor.
Then it seems there could a few ways of going about this (where $\gamma, \beta$ are learned parameters for each BN layer)
Method 1: $T_{n,c,x,y} :=  \gamma*\frac {T_{c,x,y} - \mu_{x,y}} {\sqrt{\sigma^2_{x,y} + \epsilon}} + \beta$ where $\mu_{x,y} = \frac{1}{NC}\sum_{n, c} T_{n,c,x,y}$ is the mean for all channels $c$ for each batch element $n$ at spatial location $x,y$ over the minibatch, and
$\sigma^2_{x,y} = \frac{1}{NC} \sum_{n, c} (T_{n, c,x,y}-\mu_{c})^2$ is the variance of the minibatch for all channels $c$ at spatial location $x,y$.
Method 2: $T_{n,c,x,y} :=  \gamma*\frac {T_{c,x,y} - \mu_{c,x,y}} {\sqrt{\sigma^2_{c,x,y} + \epsilon}} + \beta$ where $\mu_{c,x,y} = \frac{1}{N}\sum_{n} T_{n,c,x,y}$ is the mean for a specific channel $c$ for each batch element $n$ at spatial location $x,y$ over the minibatch, and
$\sigma^2_{c,x,y} = \frac{1}{N} \sum_{n} (T_{n, c,x,y}-\mu_{c})^2$ is the variance of the minibatch for a channel $c$ at spatial location $x,y$.
Method 3: For each channel $c$ we compute the mean/variance over the entire spatial values for $x,y$ and apply the formula as
$T_{n, c,x,y} := \gamma*\frac {T_{n, c,x,y} - \mu_{c}} {\sqrt{\sigma^2_{c} + \epsilon}} + \beta$, where now $\mu_c = \frac{1}{NHW} \sum_{n,x,y} T_{n,c,x,y}$ and $\sigma^2{_c} = \frac{1}{NHW} \sum_{n,x,y} (T_{n,c,x,y}-\mu_c)^2 $
In practice which of these methods is used (if any) are correct for?
The original paper on batch normalization , https://arxiv.org/pdf/1502.03167.pdf , states on page 5 section 3.2, last paragraph, left side of the page:

For convolutional layers, we additionally want the normalization to
obey the convolutional property – so that different elements of the
same feature map, at different locations, are normalized in the same
way. To achieve this, we jointly normalize all the activations in a
minibatch, over all locations. In Alg. 1, we let $\mathcal{B}$ be the set of all
values in a feature map across both the elements of a mini-batch and
spatial locations – so for a mini-batch of size $m$ and feature maps of
size $p \times q$, we use the effective mini-batch of size $m^\prime = \vert \mathcal{B} \vert = m \cdot pq$. We learn a pair of parameters $\gamma^{(k)}$ and $\beta^{(k)}$ per feature map,
rather than per activation. Alg. 2 is modified similarly, so that
during inference the BN transform applies the same linear
transformation to each activation in a given feature map.

I'm not sure what the authors mean by "per feature map", does this mean per channel?

Method 2:
 This is original batch Normalization as suggested in the paper [Ioffe & Szegedy, 2015].
It is the most common approach. It is very well explained here [d2l.ai]

Similarly, with convolutional layers, we can apply batch normalization after the convolution and before the nonlinear activation function. When the convolution has multiple output channels, we need to carry out batch normalization for each of the outputs of these channels, and each channel has its own scale and shift parameters, both of which are scalars. Assume that our minibatches contain  m  examples and that for each channel, the output of the convolution has height  p  and width  q . For convolutional layers, we carry out each batch normalization over the  m⋅p⋅q  elements per output channel simultaneously. Thus, we collect the values over all spatial locations when computing the mean and variance and consequently apply the same mean and variance within a given channel to normalize the value at each spatial location

I'm not sure what the authors mean by "per feature map", does this mean per channel?

Yes, two trainable parameters per Channel/Feature map.
Method 3:
 This was the idea suggested as "Layer Normalization". [Paper]
 It fixed the issue of BN i.e. dependence on a Batch of data and also it worked for sequence data. But the paper didn't claim anything great for CNN.

We have also experimented with convolutional neural networks. In our preliminary experiments, we observed that layer normalization offers a speedup over the baseline model without normalization, but batch normalization outperforms the other methods. With fully connected layers, all the hidden units in a layer tend to make similar contributions to the final prediction and re-centering and rescaling the summed inputs to a layer works well. However, the assumption of similar contributions
is no longer true for convolutional neural networks. The large number of the hidden units whose receptive fields lie near the boundary of the image are rarely turned on and thus have very different statistics from the rest of the hidden units within the same layer. We think further research is needed to make layer normalization work well in ConvNets

Method 1:
 This is averaging across the Feature Maps on every pixel. I am not sure of its aplication.


Can you reduce many Naive Bayes training samples to a few big ones? Excuse me if this is a simple question (I'm a newbie to data science).
I am trying to implement a Naive Bayes classifier in Python.

I have implemented a standard Naive Bayes classifier (Bernoulli) and this went very well.
Suppose I have 50,000 samples where the first half is labelled '0' and the second half '1'.
Then the shape of the X_train matrix will be (#number_of_unique_words/features, 50000) and shape of y_train (50000,).

From how I understand Naive Bayes, I can now also join the first 25000 documents to represent one big document with label '0' and one big document with label '1'. However, this gives me very different results when I try to predict the test set.
Is my understanding of Naive Bayes correct or is my implementation just wrong?

Your assumption seems to be correct that both approaches are actually equivalent. I can not see any difference here in the propabilities and conditional propabilities for either one way or the other.

That's why the predictions should also be the same for both ways. It is more likely that there is something wrong with your implementation.


How to go about training a NER model to extract book citations in free-form? I'm doing a project where I wish to create a graph visualization of free-form citations (not academic style citations) across all my e-books. E.g. David Foster Wallace's essays cite a lot of other books by different authors. For that I should be able to detect and extract book and authors names from my own e-books.
I've selected some examples from my e-books that I wish my NER model would tag as "books" (in bold font):

(...) or even the parodistic version of Pater to be found in W. H. Mallock’s The New Republic (...)

Plato words the same conception beautifully in the Republic: (...)

I also wish to tag authors, but I suppose this could be done out-of-the-box with Spacy or other NLP library, with some pre trained PERSON tag.
So, my question is about the best approach to go about creating this NER model.

I could create lots and lots of training samples from my books and create a new NER model. (very time consuming)

Or if there is a dataset or public model with the BOOK or something like WORK_OF_ART tag I could bootstrap my own dataset.

What do you think about this approaches?

Interesting task :)
I think even with a good amount of training data it will be difficult for a regular NER model to perform well with new books titles and authors:

The book may contain persons names which are not authors.
The book titles are difficult to identify as such in general. For example "the Republic" might or might not be about the book, and if the only indication the model can use is the capitalization it's probably going to make some errors.

To be clear, I think it could work to some extent but it would probably make quite a lot of errors.
On the other hand you could obtain a database of books, for instance from Wikipedia (there might be better resources), and you could use this in two ways:

Directly identify the books/authors in the documents by simple string matching. I would imagine that even if the coverage of the resource is not perfect, this method would easily catch a majority of occurrences.
In case the above method is not sufficient, it provides you with some good training data from which you could train a NER model in order collect titles which don't exist in the database. Note that there might be issues due to the unknown books being labelled as negative in the training data, so ideally you would have to go manually through the training data and annotate the remaining cases.


Convert nominal to numeric variables? I am trying to develeop an algorithm with sklearn and Tensorflow to predict which car can be offer to each customer.

To do that I have a database with the answers of one survey to 1000 customers.

An example of questions/[Answers] are:

Color/[Green,Red,Blue]
NumberOfPax/[2,4,5,6,7]
HorsePower/[Integer]
InsuranceIncluded[yes/no/Don't know]

As you can see all questions are answer previously tipified, and in case the answer can be open I validate the value to be an integer or a radio button.

The purpose of that beahivour is that despite the categorical variables I can easily use sklearn to clustering the data.

Will be a good approach to translate this categories to numerical value as an intern procedure an then cluster with this references?

For example: yes --> 0; No --> 1; Don't know --> 2

Then sklearn will cluster with all variables as numerical values.

I have thought this possibility beacuse I believe that sklearn can not cluster nominal data.

What do you think about this approach?

Clustering on categories is not something sklearn can do by default. And assigning sequential values to categories like that certainly won't help - clustering tends to work based on distance, by assigning 0, 1, 2 to Yes, No, Don't Know like that, you are suggesting Yes is 'closer' to No than it is to Don't Know.

I highly recommend having a look at k-modes, a clustering algorithm for categorical data. Essentially it optimises according to how common the set of categories are within the cluster (modes, rather than the means of values). E.g. may find green people carriers (5/6/7 pax) in a cluster, and red sports cars (2/4 pax).

There is a Python library here, which also has links to some papers describing the algorithm. There is also k-prototypes, which clusters with combined numerical and catagorical data.


Imbalanced dataset with 3 classes xgboost scale_pos_weight parameter The xgboost classifier states the use of parameter scale_pos_weight for 2-class problems.

I have a highly imbalanced dataset with 3 classes. Classes '1' and '-1' are very rare (~1% of dataset) and class '0' is very common.

How do I set this scale_pos_weight parameter in the xgboost classifier correctly for my classification problem?

For my multiclass classification problem with similar unbalanced data I used the output from sklearn compute_class_weight function:

https://scikit-learn.org/stable/modules/generated/sklearn.utils.class_weight.compute_class_weight.html

sklearn.utils.class_weight.compute_class_weight(class_weight, classes, y)


One-Hot-Encoding for features! I have a question about One-Hot-Encoding, something confusing me.:\
I have this sample dataset. My dataset is categorical:

F1
F2
F3
F4
Target

1
Blue
3
Car
Yes

4
Red
6
Ship
No

3
Pink
3
Cow
Yes

9
Black
6
Fish
Yes

question:
Should I use the One-Hot-Encoding for F2 and F4 too?
This is clear for me, that must be used one hot encoding for column Target. But what about the Features?
If the answer is yes why and if the answer is no why?
Thanks for your support.

Correct,  all categorical features should be encoded into binary digits so ML algo have more predictive power as categorised features cannot have order or magnitude ( be careful also about
multicollinearity if you use regression)
Some ML framework such as Catboost automatically encode features for you if you specify the feature index. I also like using python patsy to create input matrices