How to use one hot encoding of string categorical features in keras? I am dealing with a binary classification problem. The output column of my dataset is already encoded in 0/1. The problem is that I have many categorical features (columns), which are strings and I would like to one-hot encode them.

I have 18 features (few features are integers and others are strings, the categorical ones) and 1 output column.

I tried this:

dataframe = pd.read_csv('basic_df_export.csv', sep=';', encoding = 'ISO-8859-1', header=None)

dataset = dataframe.values
# split into input (X) and output (Y) variables
X = dataset[:,0:17]
Y = dataset[:,17]

# define example
encoded = to_categorical(X)
print(encoded)

but it doesn't work, throwing me this error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-41-318da09d7033> in <module>()
      9
     10 # define example
---> 11 encoded = to_categorical(X)
     12 print(encoded)

~/anaconda/lib/python3.6/site-packages/keras/utils/np_utils.py in to_categorical(y, num_classes)
     21         is placed last.
     22     """
---> 23     y = np.array(y, dtype='int')
     24     input_shape = y.shape
     25     if input_shape and input_shape[-1] == 1 and len(input_shape) > 1:

ValueError: invalid literal for int() with base 10: 'photo'

For string data, use get_dummies() (from Pandas). to_categorical() takes integers as inputs.

There are two important differences between
Keras: to_categorical() and
Pandas: get_dummies().

Keras: to_categorical()

to_categorical() takes integers as input (no strings allowed).
to_categorical() generates dummies starting at 0 by default!

Looking at the help function:

print(help(to_categorical))

Says:

to_categorical(y, num_classes=None, dtype='float32')
    Converts a class vector (integers) to binary class matrix.

    E.g. for use with categorical_crossentropy.

    # Arguments
        y: class vector to be converted into a matrix
            (integers from 0 to num_classes).
        num_classes: total number of classes.
        dtype: The data type expected by the input, as a string
            (`float32`, `float64`, `int32`...)
...

So if your data is numeric (int), you can use to_categorical(). You can check if your data is an np.array by looking at .dtype and/or type().

import numpy as np
npa = np.array([2,2,3,3,4,4])
print(npa.dtype, type(npa))
print(npa)

Result:

int32 <class 'numpy.ndarray'>
[2 2 3 3 4 4]

Now you can use to_categorical():

from keras.utils import to_categorical
cat1 = to_categorical(npa)
print(cat1.dtype, type(cat1))
print(cat1)

Which yields a matrix:

float32 <class 'numpy.ndarray'>
[[0. 0. 1. 0. 0.]
 [0. 0. 1. 0. 0.]
 [0. 0. 0. 1. 0.]
 [0. 0. 0. 1. 0.]
 [0. 0. 0. 0. 1.]
 [0. 0. 0. 0. 1.]]

Note that the matrix contains five columns (starting at zero up to four, which is my max. value in the np.array). The first two columns (representing 0 and 1 in the original data) are 0 in the whole matrix, because none of these values are found in the original data.

to_categorical() also takes input which is not explicitly defined as np.array. For instance the statements below would also be legal.

alt1 = to_categorical([0,0,1,1,2,2])
print(alt1.dtype, type(alt1))
print(alt1)

alt2 = to_categorical((0,0,1,1,2,2))
print(alt2.dtype, type(alt2))
print(alt2)

Because the range of values now is between 0 and 2, the result would look like:

[[1. 0. 0.]
 [1. 0. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 0. 1.]
 [0. 0. 1.]]

Pandas: get_dummies()

When you have a Pandas df, you can convert some column to dummies using get_dummies(), regardless of the data type in the column. So it is also possible to convert a column of strings to dummies.

import pandas as pd
df = pd.DataFrame(data={'col1':["A", "A", "B", "B", "C", "C"]})
alt3 = pd.get_dummies(df['col1'])
print(type(alt3))

This gives:

<class 'pandas.core.frame.DataFrame'>
   A  B  C
0  1  0  0
1  1  0  0
2  0  1  0
3  0  1  0
4  0  0  1
5  0  0  1

Note that the result is (again) a Pandas df. So we need to convert it to a np.array.

alt3 = alt3.to_numpy()
print(alt3.dtype, type(alt3))
print(alt3)

This yields:

uint8 <class 'numpy.ndarray'>
[[1 0 0]
 [1 0 0]
 [0 1 0]
 [0 1 0]
 [0 0 1]
 [0 0 1]]

So that it is ready to be used with Keras.

Note that the matrix generated here does not (!) start at zero. Instead each distinct value in the chosen Pandas column gets it's own column in the dummy matrix.


Is GridSearchCV in combination with ImageDataGenerator possible and recommendable? I want to optimize some hyperparameters for a CNN architecture by using GridSearchCV (Scikit-Learn) in combination with Data Augmentation (ImageDataGenerator from Keras).
However, GridSearchCV only offers the fit function and not the fit_generator function.
Is it even recommended to use data augmentation with GridSearchCV?
The parameters for the ImageDataGenerator are already fixed and should not be changed. Would it be better to first determine the hyperparameters via grid search without data augmentation and only to use data augmentation for the final model?

What do you think about this topic? What are your experiences?

As promised, here you can find an example of how you could apply kfold cross validation for a defined convolutional neural network model, applied to an augmented dataset. You can find the code as a simple gist here
It is done as follows:

for a subset of the CIFAR10 images dataset, generate 3 augmented images (by applying horizontal_flip) per original image, so we should finally have as the number of final images in the augmented dataset: 'number of images in the original dataset' * 3.

check that indeed the built augmented dataset has the new expected number of images. We have just created the augmented dataset, not the fit step yet

apply kfold cross validation on the augmented dataset for several hiperparameters combinations; in this example, 3 pairs of 'learning rate-momentum' have been tried. It is made via the usual 'fit' method:

display the results in a dataframe

This way, we have applied hyperparametrization via kfold cross validation; not a full grid search but only with 3 pairs of hiperparams, but the idea would be the same, not depending on the fit_generator method but making yourself your k folds cross validation on the generated augmented dataset. We could also include other data augmentation strategies in this cross validation.


Need help in improving accuracy of text classification using Naive Bayes in nltk for movie reviews I am referring http://www.nltk.org/book/ch06.html for generating a movie review classifier. It considers all words (Nouns, adjectives, verbs..) as part of feature set.
I am trying to build a classifier that considers only the verbs and assesses if the movie review is positive or negative.

Please explain if this approach is better, if yes how can it be improved else what other parts of speech tags need to be included to improve the feature set.
Please refer to the following code
1)Import nltk and corpus
2)Understand the categories
3)Create a list of "documents" where each document consists of words that are not in stopwords.words() and filtered from list of "punctuations" as well.
4)Shuffle the documents and generate a list called "all_words" that contains all the words appearing in movie_reviews minus stopwords and punctuations.
5)Create a Freq Dist of "all_words"
6)Create a list of "verbs" by looking at the pos_tag of each word appearing in "all_words"
7)Create a feature dictionary for each document in the "documents" list created in step 3, such that the dictionary will contain keys corresponding to words in "verb" list and a boolean to show if that verb appeared in the document under consideration. This is handled by documentFeature().
8)Create a naive bayes classifier instance and train and compute test accuracy.

from nltk.corpus import movie_reviews
from nltk.corpus import stopwords
from nltk.tokenize import RegexpTokenizer
import nltk
movie_reviews.categories()
#['pos','neg']
#the regextokenier is used to tokenize the words
tokenizer=RegexpTokenizer(r'\w+')
#creating documents based on filtration of stopwords for each review and running a tokenizer on each document
documents=[(tokenizer.tokenize(' '.join(set(i for i in movie_reviews.words(fileid))-set(stopwords.words()))),category)
           for category in movie_reviews.categories()
           for fileid in movie_reviews.fileids()
          ]
import random
random.shuffle(documents)
#each document contains words that are not in stopwords and punctuations.
for i in documents[:5]:
    temp=nltk.FreqDist([j.lower() for j in i[0]])
    print(temp.most_common(5),i[1])
#output of 5 documents
#[('vampires', 1), ('clever', 1), ('interesting', 1), ('sunlight', 1), ('partners', 1)] neg
#[('family', 1), ('nino', 1), ('friends', 1), ('acting', 1), ('higher', 1)] pos
#[('inconsistent', 1), ('eye', 1), ('yes', 1), ('interesting', 1), ('praise', 1)] neg
#[('acting', 1), ('science', 1), ('bucks', 1), ('huge', 1), ('terrific', 1)] pos
#[('acting', 1), ('shielded', 1), ('somewhere', 1), ('think', 1), ('touched', 1)] neg

#generate a list called 'all_words' that contains all the set of words that have appeared so far
all_words=tokenizer.tokenize(' '.join(set(i for i in movie_reviews.words())-set(stopwords.words())))
freqdist=nltk.FreqDist(all_words)

#create a list of all verbs for each word appearing in 'all_words'
verb=[]
pos_=nltk.pos_tag(all_words)
#print([i[1] for i in pos_])
for i in pos_:
    if i[1] in ['VB','VBG','VBN','VBZ','VBD','VBP']:
        verb.append(i[0])

#document - feature set, build a dictionary of verbs for each document
def documentFeature(document):
    feature={}
    for i in verb:
        feature['contains({0})'.format(i)]=(i in document)
    return feature
#build a naive bayes classifier
featureSet=[(documentFeature(d),c) for d,c in documents]
trainSet,testSet=featureSet[100:], featureSet[:100]
classifier=nltk.NaiveBayesClassifier.train(trainSet)

print(nltk.classify.accuracy(classifier, testSet))
#0.03 a very poor accuracy on the testset

Currently I am getting 0.03 accuracy, please help me with improving the accuracy.

Without reading your code (sorry!) I can suggest that you drop the verbs-only approach and use a word embedding to account for word similarity. Your feature dictionary is bound to find very few "hits" in the documents, unless you have a humongous dataset, and even then YMMV because you are just using verbs. Also: what's the justifications for using verbs? "I liked" and "I didn't like" both contain the verb "to like" and they mean opposite things. Same goes for hate (because you only have binary classification: no nuances) and so on.

A popular python implementation of word2vec is gensim, but you could use that of tensorflow or some other embedding like the (allegedly superior) conceptnet numberbatch. If you want to summarize whole documents into numbers you can try doc2vec (aka paragraph2vec, paper here), also available in gensim, tensorflow, etc.

You will need a big dataset. Once you have this up and running you can try other classifiers.


Super basic logistic regression example I am new to ML and I created a super basic logistic regression example with 4 points on the $x$ line that belong to two classes:

points = [[1, 1]]
points = points + [[2, 0]]
points = points + [[1.5, 1]]
points = points + [[2.5, 0]]

data   = np.array([x[:-1] for x in points])
target = [int(x[-1]) for x in points]

alg = LogisticRegression()
alg.fit(data, target)
print (alg.coef_)
print (alg.intercept_)

print (alg.predict_proba(1))
print (alg.predict_proba(1.5))
print (alg.predict_proba(2))
print (alg.predict_proba(2.5))

I expected the model to understand that anything greater than 1.5 belongs to class 0, and anything smaller than 2 belongs to class 0. However, everything is predicted to be in class 0. Also, in this case , what is the meaning for the coef and intercept? More precisely, is there a way to deduce from the coef and intercept where the model thinks the points switch from class 0 to class 1?

What is the meaning for the coef and intercept

A (binary) logistic regression algorithm tries to determine whether the data $x$ belongs to class 0 or class 1 by the value
$f(x)=\omega x+b$. If the value $f(x)>0$, the algorithm believe $x$ is more likely to be in class 1; while if $f(x)<0$, then $x$ is more likely to be in class 0.

In your code alg.coef_ is the $\omega$ above, and alg.intercept_ is the $b$ above.

Why everything is predicted to be in class 0

If you run your code, you should see the result alg.coef_=-0.354, alg.intercept_=0.307. Therefore your alg is calculating $f(x)=-0.354x+0.307$. Now if you plug in your data, i.e. $x=$1, 2, 1.5 and 2.5 respectively, you should get the value $f(x)=$-0.047,-0.401,-0.224 and -0.579 respectively.

As you see, all the four $f(x)$ values are smaller than 0, therefore your alg determines that they all belong be class 0.

But still why? Why the result is not what I expected?

Here is were regularization comes in. In the context of logistic regression, the learning algorithm regularizes that the learned parameter, $w$, should not be too large. To be specific, by default it regularizes the $l_2$ norm norm of $w$. In your case $w$ is a scalar, then $l_2$ norm is its absolute value.

In general, regularization is used to prevent overfitting. But that's another broad topic.

If you want your algorithm to behave as you expected (classify 1 and 1.5 to class 1, and classify 2 and 2.5 to class 0) , there are 2 ways.

You tell you program to use extremely weak regularization.
That can be achieved by passing a large value to the C parameter in alg = LogisticRegression() (default value is 1.0, see detail), e.g.

alg = LogisticRegression(C=1000)

In this case your algorithm should return alg.coef_=-9.894, alg.intercept_=17.156, and it can classify your four data points correctly. Notice that the absotely value of alg.coef_ is now much larger than before.

Provide your algorithm with more data.

For example, simply repeat your four data points 100 times:

data   = [x[:-1] for x in points]*100
target = [int(x[-1]) for x in points]*100

Then you can get alg.coef_=-4.438, alg.intercept_=7.594 without setting C. In this case the same default regularization strength is still applied, but the additional data provide stronger evidence for your algorithm to believe that the alg.coef_ should be allowed larger (in absolutely value).


What does online learning mean in Topic modeling (LDA) - Gensim I came across this line in the Gensim Documentation- Gensim LDA  - "The model can also be updated with new documents for online training."

So my assumption on what it means is - 'Once we have a model trained on one corpus, we can add new data and continue to train the model with new data thereby adding more vocabulary and enriching results. Is this correct?

Is this the same approach discussed in the paper - Online Learning for LDA ? Help me understand this technique.

Yes, your intuition about the definition of online learning in Topic modeling(LDA) is correct :

  "The model can also be updated with new documents for online training."

However, I would quote the standard definition of online learning in machine learning :

  It is a method in machine learning in which data becomes available in sequential order and is used to update our best predictor for future data at each step, as opposed to batch learning techniques which generates the best predictor by learning on the entire training dataset at once. It is a very useful technique in areas of machine learning where it is computationally infeasible to train over the entire dataset, requiring the need of out-of-core algorithms.

You can find more about it here.

In your case (nlp), online learning would -

Add vocabulary as the data comes.
Train the model on newly added corpus.
update the results

All of these would keep happening as the new data comes.


Why do you need to learn statistics for machine learning? This maybe a very broad question. Please do not delete.

However, I am curious on why do we need to learn statistics for Machine Learning?

This is given that the software libraries used already have the ‘knowledge’ encoded.
Of course, a general understanding of how things work under the hood, but other than that is there a need for a more in-depth understanding?

Same question applies regarding inner workings of various models. How will a good understanding (including the maths) help?

You need to learn statistics to be able to make sensible decisions about the ML models you plan to use, for example:

which model to use?
is the training-set large enough?
how to measure the model performance (which error measure)?
how to judge if the result is good enough?
how to (correctly!) compare different models and correct for multiple-testing?
how to avoid/identify selection bias?
how to deal with correlated features?
how to interpret the coefficients of the multiple linear regression model you just trained?

And most importantly: you need to learn statistics to understand that these questions (and many other ones) are important to ask.

If you want to be a software developer you may not need to learn statistics (although it will still be super useful!). But if you want to be a data-scientist, machine learning engineer or AI expert you cannot do your work properly without it.


Keras Multiple “Softmax” in last layer possible? Is it possible to implement mutiple softmaxes in the last layer in Keras? So the sum of Nodes 1-4 = 1; 5-8 = 1; etc.

Should I go for a different network design?

I would use the functional interface.

Something like this:

from keras.layers import Activation, Input, Dense
from keras.models import Model
from keras.layers.merge import Concatenate

input_ = Input(shape=input_shape)

x = input_
x1 = Dense(4, x)
x2 = Dense(4, x)
x3 = Dense(4, x)
x1 = Activation('softmax')(x1)
x2 = Activation('softmax')(x2)
x3 = Activation('softmax')(x3)
x = Concatenate([x1, x2, x3])

model = Model(inputs=input_, outputs=x)


What is the class of this optimization problem? I have the following optimization problem:
Find $\mathbf{w}$ such that the following error measure is minimised:

$E_u = \dfrac{1}{N_u}\sum_{i=0}^{N_u-1}\lVert \mathbf{w}^Tx(t_{i+1})-\mathbf{F}(\{\mathbf{w}^Tx(t_j)_{j=0,i}\})\rVert$,
$t_i \text{ being the i-th timestamp and } \lVert \cdot \rVert \text{ the } L_2 \text{ norm}$
$\mathbf{F}$ is something of the form $\sum_{j=0}^{i}\alpha_j\mathbf{w}^Tx(t_j)$ with $\sum_{j=0}^i \alpha_j = 1$.

It's important to note the $\alpha$'s are fixed (because they are from a subsystem).

With the constraints that: $\mathbf{w}>\mathbf{0}$ and $\mathbf{w}<\mathbf{L}$. Both $\mathbf{0}$ and $\mathbf{L}$ are vectors in $\mathbb{R}^6$, $\mathbf{L}$ being a vector of positive arbitrary limits I set.

Unfortunately, this doesn't look like the standard least-squares problem, due to that pesky $\mathbf{w}$ term that pops in both places (this is fixed in a certain epoch). Essentially, is like least squares but the target $\mathbf{y}$ is the value of the series at the next timestamp.

Is this another class of problems? Unfortunately, I don't have enough background on this area, but I've read something about Recursive Least Squares and Kalman filters - is this something that could be solved with this?

After some research on this problem I've realised the model I've developed was incorrect. This is because I introduced the weight vectors in a wrong manner.

Essentially, in my first model(the one this question was based on), the weight vectors were applied on the target vectors and on the input vectors that went into the model. Suppose this is right...because the model converges to the target => on the long term this will behave like a linear transformation: $F(w\mathbf{x})=wF(\mathbf{x})$
Introducing this into $E_u$ we get that $w$ doesn't even matter. Diving more deeper I realized this was a property of space. I was computing an $L_2$ norm...by defining an error as the distance of the model vector from the target. No matter how you strech you transform all the points (target and model) in the same manner <=> the relative positions between the points don't change (the errors might be bigger or smaller, but the order relationship between the errors will be preserved).

The solution was to update my model to use instead the weighed euclidean distance. This means:
$$E_u = \dfrac{1}{N_u}\sum_{i=0}^{N_u-1}\left\lVert x(t_{i+1}) - F_D(\{x(t_j)_{j=0,i}\})\right\rVert_D$$

I've written $F_D$, because at some point, my model does a query using a Ball Tree, which depends on D. I don't know theoretically why this works, but I can say that experimentally, it works (in the sense that the model outputs things which I expect and are reasonable).


What is the complexity of a recurrent neural network? In particular, how much memory does a recurrent NN require as a function of the dataset size,number of nodes, etc., and how expensive is it to evaluate at runtime given a new test point?

I've found some time ago two interesting papers about recurrent neural networks and their complexity. I guess you can use those as a reference points at least:

Architectural Complexity Measures of Recurrent Neural Networks
Bounds on the complexity of recurrent neural network implementations of finite state machines


What's Joint Training in Neural Networks? I'm having a hard time trying to find a good explanation of the process of Joint Training in Neural Networks. I already understand the concepts of Fine Tuning and Feature Extraction, and i know it has to do with the practice of taking a network model that has already been trained for a given task, and make it perform a second similar task.

You are incorrect that is called Transfer learning, which focuses on storing knowledge gained while solving one problem and applying it to a different but related problem.
Joint Training is basically when the neural network trains on several different task simultaneously so optimizing more than one loss function rather than one loss function as the case with Transfer learning.
It is also called Multi task learning.
Refer this article to know more.


How do Bayesian methods do automatic feature selection? Someone asked me this question and I do not know I answered it correctly.
I answered the question in the following way: One type of Bayesian method is Bayesian inference and feature selection has to do with ${L}^{1}$ regularization because it is used extensively for this purpose. So, for ${L}^{1}$ regularization, the penalty $\alpha \Omega (\boldsymbol{w}) = \alpha \sum_{i} |w_{i}|$ used to regularize a cost function is equivalent to the log-prior term that is maximized by MAP Bayesian inference when the prior is an isotropic Laplace distribution.

But my question is this an automatic feature selection? ${L}^{1}$ regularization finds the specific subset of the available features to be used. Also is my answer correct to this question? I am just curious to know if my line of thinking makes sense or if it does not. If it does not, please let me know why. Thanks

Your interpretation of L1 regularization sounds right. Is it used to perform feature selection? Yes, in the broad sense that this 'encourages' coefficients in the linear model to be 0, and those features with 0 coefficients are not used and can be removed.

Of course, this assumption about the prior distribution of coefficients is just an assumption you're adding. You are asserting that you need a lot of evidence to believe the coefficients are nonzero. I'm not sure it's the L1 regularization that's doing the feature selection, but your assumption, implemented by L1 regularization, that's leading you to conclude that some features do not contribute to the model.


Model selection for a "set of linear models"? I would appreciate your input on which predictive ML model(s) could fit our dataset the best.

The primary features of the dataset are x,y (continous) and c (factor with around L=500 unordered levels). The secondary features (~20) are both continous and factors.
We know that the lowest-order correlation is x~y.
We also know that modeling x~y for each level in c improves the fit significantly.

We could of course create L independent linear models, but would like to explore a single model where these sub-linear relationships are partially correlated. The reason being that some levels of c have relatively low statistics and would benefit from the global x~y relationship. Also, we would like a single model when extending to secondary features.

Any suggestions are welcome. (In case of specific libraries, R is preferred.)

A generalized linear model (glm) appeared to be work reasonable.

glm(y ~ x * c, data=df) and extensions hereof.


Limits of Javascript on the implementation of AI algorithms I have a basic understanding of javascript, and know hardly any other programming language. Am I bound to face some issues in the field of neural networks and machine learning because of this? Should I learn something else for the sake of avoiding some inherent weaknesses of the language? I am most worried about the capacity of javascript to handle data, rather than its possibilities regarding the textual implementation of the algorithms per se...

Thank you

First of all, learning other programming languages is always a good thing.  There are many reasons for this:

Learning new languages makes it easier for you to learn other new languages in the future - and sooner or later in your career you will almost certainly want to or need to.
Learning new languages helps you understand the language you're already working in.  If I told you JavaScript is a dynamically-typed language, would that mean anything to you?  Probably not so much if you've never used a statically-typed language.  If I told you it has first-class functions, would that mean anything to you?  It's like telling someone "night is dark" when they've never seen daylight.
Learning new languages teaches you about ideas from other languages that may eventually make it into your language. ECMAScript 6 added "arrow functions" to JavaScript, these are also known as lambdas and would already be familiar to you if you've used C# (from version 3 onwards), Java (from version 8 onwards), Scala, Haskell, or many other languages.
Learning new languages teaches you different ways to think about code. And this can only make you a better coder.
Learning new languages increases your employment opportunities.
Learning new languages is fun!

In addition to these, and other benefits I've surely omitted, there's this which is directly relevant to your question about machine learning in particular.

Different languages have different libraries and different communities of users that use them for different purposes.  Most people doing machine learning aren't working primarily in JavaScript.  Some are working in Python, some are working in R, some are working with Spark using Scala or Python, etc.

If you already know JavaScript, then learning Python shouldn't be too hard for you, and it has some great machine learning libraries like scikit-learn, with lots of great resources online to help you learn.  So I'd recommend starting there.  But I also recommend that you eventually branch out and learn some languages that are further outside your comfort zone.


How are per-layer-detected-patterns in a trained CNN plotted? In the case my question is not clear, I am talking about the patterns that are detected in each of the layers of an image-trained Convolutional Neural Network (CNN). Take the following image as an example (by Honglak Lee). I think I understand the concept: different layers start coding for different features with increasing complexity. The first layers code for edges, the middle layers code for simple features (e.g. nose, eyes), and the later layers code for whole faces.
However, I do not see the equivalence betweence each of the patterns in the picture and the network elements. Do each pattern inside a pattern-class correspond to one neuron of that layer? How are these cool pattern detection pictures even plotted?

The image you posted is derived from figures which depict the basis functions of a Convolutional Restricted Boltzmann Machine.

I found them in this ICML paper (behind a paywall, sorry) and this PhD thesis, both are by Honglak Lee, who was advised by Andrew Ng.

Strictly speaking they are not visualizations of a feed-forward Convolutional Neural Network, which might be why you are having a hard time interpreting what they depict. I don't really know much about RBM's myself so I'm afraid I can't help with their interpretation.

As far as visualizing CNN's, there are several common methods. You can visualize filter weights directly, find input images which maximally activate a filter, plot activation as a function of image occlusion, etc.
This page has a good summary of techniques, also this paper may be useful.


Hierarchical Clustering and Variable Selection I am using  "Single linkage" hierarchical algorithm to cluster my data points with Gower Distance as my data have both qualitative and quantitative variables.

After applying this for the full model (all variables) I would like to start excluding those variables which are actually the not so important for my data. I was thinking of using principal component analysis (PCA) but I can't because my variables are a mixture of both categorical and continuous. Can someone suggest what is best method to select variables?

Finally I would like to use the Elbow Method to check exactly what is the optimal number of clusters?

Can someone help me with this logic?

I am using R-Studio for my analysis.

PCA is only suitable for continuous variables, and sensitive to scaling. So don't use it here.

Instead, you meat want to look at classic information measures whether some attribute is "correlated" with the clusters. For example Gini, mutual information etc.

Similarly, the Elbow method is a bad idea. It never was a good idea in the first place because of scaling... But it's really only suitable for choosing the k in k-means if you do many random restarts and look at the best partitionings found. But even then: there is no mathematical definition of the "elbow". Most of the time, there is no elbow, and people just pick some random k this way because they don't consider this option...nit's probably widely misused. But it certainly must not be used with hierarchical clustering!


Finding clusters in multidimensional data I have a set of data from 3,000 records. There are 5 attributes per individual (labelled A - E). I can use Kendall's W (coefficient of concordance) to determine the concordance between any two records.

What I require is a way to discern any clusters which exist across the entire data, and simply which of the attributes are within each cluster. e.g. for the data below I would see 2 clusters:

Cluster 1: C, D / Cluster 2: A, B

ID  | A   | B   | C   | D   | E   |
001 | 55  | 80  | 125 | 114 | 75  |
002 | 75  | 78  | 110 | 105 | 95  |
003 | 135 | 105 | 95  | 93  | 92  |
004 | 120 | 115 | 101 | 55  | 44  |

What techniques or approaches could I use to help with this?

In general it does not make much sense to cluster features. In an ideal world for your features to be the best they can be they should actually be independent, thus there should be no relationship between them. Typically when we talk about clustering it is clustering the instances. To attribute some associative labels to a subset of the instances based on the similarity of their feature values.

Many clustering algorithms exist, I would say that the most popular is K-means however spectral clustering and Gaussian mixtures are also frequently used. As always, each algorithm is best suited for a specific type of dataset, it is up to you to choose which is best suited, or you can just try all of them and see which is best.

Here you can find a list of clustering algorithms with their respective usecases. Always use the libraries when you want to implement standard algorithms, they are highly optimized. But for education sake it is good to look at what is happening.

I will describe a homebrew version of the K-means algorithm such that you can understand what is happening under the hood and perhaps you will see why we cluster instances and not features.

The K-means algorithm

import numpy as np
import matplotlib.pyplot as plt

First we will make some artificial data. These will consist of $n$ 2D Gaussian clusters with a given mean and variance. Here $n=5$ and we will have 300 instances per Gaussian distribution.

params = [[[ 0,1],  [ 0,1]],
          [[ 5,1],  [ 5,1]],
          [[-2,5],  [ 2,5]],
          [[ 2,1],  [ 2,1]],
          [[-5,1],  [-5,1]]]

n = 300
dims = len(params[0])

data = []
y = []
for ix, i in enumerate(params):
    inst = np.random.randn(n, dims)
    for dim in range(dims):
        inst[:,dim] = params[ix][dim][0]+params[ix][dim][1]*inst[:,dim]
        label = ix + np.zeros(n)

    if len(data) == 0: data = inst
    else: data = np.append( data, inst, axis= 0)
    if len(y) == 0: y = label
    else: y = np.append(y, label)

num_clusters = len(params)

print(y.shape)
print(data.shape)

  (1500,) (1500, 2)

So now we have 1500 instances (records) in 2D space. This can be extended to any number dimensions. 2 is easiest to plot.

plt.scatter(data[:,0], data[:,1])
plt.show()

I will paste the entire algorithm at the bottom of the answer for a quick copy and paste but here I will go through its different parts so you can see how it works. The algorithm goes as follows: first we initialize some centroids within the range of our data. These are the red dots in the image below

def train(self, data, verbose=1):

        shape = data.shape

        ranges = np.zeros((shape[1], 2))
        centroids = np.zeros((shape[1], 2))

        for dim in range(shape[1]):
            ranges[dim, 0] = np.min(data[:,dim])
            ranges[dim, 1] = np.max(data[:,dim])

        if verbose == 1:
            print('Ranges: ')
            print(ranges)

        centroids = np.zeros((self.k, shape[1]))
        for i in range(self.k):
            for dim in range(shape[1]):
                centroids[i, dim] = np.random.uniform(ranges[dim, 0], ranges[dim, 1], 1)

        if verbose == 1:
            print('Centroids: ')
            print(centroids)

            plt.scatter(data[:,0], data[:,1])
            plt.scatter(centroids[:,0], centroids[:,1], c = 'r')
            plt.show()

Then we will calculate the distance from each instance (record) to each centroid.

distances = np.zeros((shape[0],self.k))
for ix, i in enumerate(data):
    for ic, c in enumerate(centroids):
        distances[ix, ic] = np.sqrt(np.sum((i-c)**2))

Then we will attribute each instance as belonging to the centroid it is closest to.

labels = np.argmin(distances, axis = 1)

Now we will update the position of the new centroids by finding the mean position of all instances which are closest to the given centroid in each dimension.

new_centroids = np.zeros((self.k, shape[1]))
for centroid in range(self.k):
    temp = data[labels == centroid]
    if len(temp) == 0:
        return 0
    for dim in range(shape[1]):
        new_centroids[centroid, dim] = np.mean(temp[:,dim])

We then repeat this process until the centroids no longer move significantly. Usually if the difference in position for all the centroids is less than machine epsilon we consider the algorithm to have converged.

if np.linalg.norm(new_centroids - centroids) < np.finfo(float).eps:
    print("DONE!")
    break

For this dataset it took 16 iterations for convergence, this is the final result.

What we see here is that each instance is grouped into a cluster with similar attributes as itself. For example if the dimensions of our data represented height (x-axis) and weight (y-axis), then we can group people into 5 different BMI indices.

Purple: short and skinny
Green: short and fat
Blue: mid height and mid weight
Yellow: Mid height and fat
Turquoise: Tall and mid weight.

Each member in our population belongs to a specific BMI index which is clustered based on two measurable attributes he possesses (features), his height and his weight.

The full K-means algorithm

This is the algorithm

class Kmeans(object):

    def __init__(self, k=1):
        self.k = k

    def train(self, data, verbose=1):

        shape = data.shape

        ranges = np.zeros((shape[1], 2))
        centroids = np.zeros((shape[1], 2))

        for dim in range(shape[1]):
            ranges[dim, 0] = np.min(data[:,dim])
            ranges[dim, 1] = np.max(data[:,dim])

        if verbose == 1:
            print('Ranges: ')
            print(ranges)

        centroids = np.zeros((self.k, shape[1]))
        for i in range(self.k):
            for dim in range(shape[1]):
                centroids[i, dim] = np.random.uniform(ranges[dim, 0], ranges[dim, 1], 1)

        if verbose == 1:
            print('Centroids: ')
            print(centroids)

            plt.scatter(data[:,0], data[:,1])
            plt.scatter(centroids[:,0], centroids[:,1], c = 'r')
            plt.show()

        count = 0
        while count < 100:
            count += 1
            if verbose == 1:
                print('-----------------------------------------------')
                print('Iteration: ', count)

            distances = np.zeros((shape[0],self.k))
            for ix, i in enumerate(data):
                for ic, c in enumerate(centroids):
                    distances[ix, ic] = np.sqrt(np.sum((i-c)**2))

            labels = np.argmin(distances, axis = 1)

            new_centroids = np.zeros((self.k, shape[1]))
            for centroid in range(self.k):
                temp = data[labels == centroid]
                if len(temp) == 0:
                    return 0
                for dim in range(shape[1]):
                    new_centroids[centroid, dim] = np.mean(temp[:,dim])

            if verbose == 1:
                plt.scatter(data[:,0], data[:,1], c = labels)
                plt.scatter(new_centroids[:,0], new_centroids[:,1], c = 'r')
                plt.show()

            if np.linalg.norm(new_centroids - centroids) < np.finfo(float).eps:
                print("DONE!")
                break

            centroids = new_centroids
        self.centroids = centroids
        self.labels = labels
        if verbose == 1:
            print(labels)
            print(centroids)
        return 1

    def getAverageDistance(self, data):

        dists = np.zeros((len(self.centroids),))
        for ix, centroid in enumerate(self.centroids):
            temp = data[self.labels == ix]
            dist = 0
            for i in temp:
                dist += np.linalg.norm(i - centroid)
            dists[ix] = dist/len(temp)
        return dists

    def getLabels(self):
        return self.labels

To use the algorithm use the following, where data is artificial data we made above but can also be any numpy matrix where the records are the rows and the features are the columns.

kmeans = Kmeans(5)
kmeans.train(data)


What is the stochastic part in stochastic gradient descent? From my understanding, a stochastic process whose value at a particular instant is dependent on the previous values taken and every time the process is run, the path chosen can be different. We can know only the limits and confines the process stays after certain intial seeding value.

The weight values for a neural networks are updated via the stochastic gradient descent method. What is the stochastic part about it? After some initialization of the variable, error function accumulated will be same if the same set of input data is provided after initialization in each test. Why is the relevance of the term stochastic? Where is the scope for the randomness and what is the source of this randomness?

You sample a random mini-batch as opposed to the full dataset. This means that you get a stochastic approximation of the true gradient of your loss function with regards to your dataset and weights.


Linear Regression in Python Below is the dataset for which I am trying to implement Linear regression in
python.

   age     sex     bmi  children smoker     region      charges
0   19  female  27.900         0    yes  southwest  16884.92400
1   18    male  33.770         1     no  southeast   1725.55230
2   28    male  33.000         3     no  southeast   4449.46200
3   33    male  22.705         0     no  northwest  21984.47061
4   32    male  28.880         0     no  northwest   3866.85520

I am confused what to do with the columns children, smoker, sex as they are of type "object".

Data columns (total 7 columns):
age         1338 non-null int64
sex         1338 non-null object
bmi         1338 non-null float64
children    1338 non-null int64
smoker      1338 non-null object
region      1338 non-null object
charges     1338 non-null float64

Do I have to convert this to numeric before building my model ?
Please provide your suggestions.

Thank you.

Yes, you will have to convert everything to numeric. That requires thinking about what these attributes represent accordingly you can use either the below 3 options.

There are three options:

One-Hot encoding for categorical data
Arbitrary numbers for ordinal data
Use something like group means for categorical data (e. g. mean
prices for city districts).

You have to be careful to not infuse information you do not have in the application case.

I'm expanding on option 1 and 3, if you want to know about option 2 you can go through the links attached at last.

One hot encoding

If you have categorical data, you can create dummy variables with 0/1 values for each possible value. Similarly you could implement for children, smoker.

E. g.

id  Sex
0   Male
1   Feamle

to

id   Male   Female
0      1       0
1      0       1

This can easily be done with pandas:

import pandas as pd

data = pd.DataFrame({'Sex': ['Male', 'Female']})
print(pd.get_dummies(data))

will result in:

        Sex_Male  Sex_Female
0           1            0
1           0            1

Using categorical data for groupby operations

This is an additional usecase but in your case it is not necessary to use this but if you feel so, you can try implementing this as well

You could use the mean for each category over past (known events).

Say you have a DataFrame with the last known mean prices for cities:

prices = pd.DataFrame({
    'city': ['A', 'A', 'A', 'B', 'B', 'C'],
    'price': [1, 1, 1, 2, 2, 3],
})
mean_price = prices.groupby('city').mean()
data = pd.DataFrame({'city': ['A', 'B', 'C', 'A', 'B', 'A']})

print(data.merge(mean_price, on='city', how='left'))

Result:

  city  price
0    A      1
1    B      2
2    C      3
3    A      1
4    B      2
5    A      1

For better understanding you can go through this Link-1, Link-2


How to select features based on feature importance using SelectFromModel? I would appreciate if you could let me know how to select features based on feature importance using SelectFromModel.

I wrote:

# data
X = np.array(pd.read_csv('who_X_1.csv',header=None).values)
y = np.array(pd.read_csv('who_Y_1.csv',header=None).values.ravel())
indices = np.arange(y.shape[0])

# # Divide Data into Train and Test
X_train, X_test, y_train, y_test,idx_train,idx_test = train_test_split(X, yy,indices,stratify=yy,test_size=0.3, random_state=42)

scaler = StandardScaler()

# # Compute Cohen's Kappa or Auc as scoring criterion due to imbalanced data set
kappa_scorer = make_scorer(cohen_kappa_score)
auc_scorer=make_scorer(roc_auc_score)
F_measure_scorer = make_scorer(f1_score)

##hyperparameter

param_grid = {
    'clf__colsample_bytree': [i/10.0 for i in range(7,10)],
    #"clf__subsample"  : [i/10.0 for i in range(5,10)],
    #'clf__max_depth':range(5,15,1),
    #'clf__gamma':[i/10.0 for i in range(0,5)],
    #'clf__reg_alpha':[1e-5, 1e-2, 0.1, 1, 100]

            }

##Classifier
xg=XGBClassifier(max_depth=3,
                 learning_rate=0.05,
                 n_estimators=350,
                 objective="binary:logistic",
                 booster="gbtree",
                 gamma=0,
                 min_child_weight=0.8,
                 subsample=1,
                 colsample_bylevel=1,
                 colsample_bytree=0.6,
                 reg_alpha=0.001,
                 reg_lambda=1,
                 scale_pos_weight=22,
                 random_state=4,n_jobs=-1)
pipe=Pipeline(steps=[('pre',scaler),
                    ('clf',xg)])

rg_cv = GridSearchCV(pipe, param_grid, cv=5, scoring = 'f1')
rg_cv.fit(X_train, y_train)
print("Tuned rf best params: {}".format(rg_cv.best_params_))

# Use SelectFromModel
thresholds = np.sort(rg_cv.best_estimator_.named_steps["clf"].feature_importances_)
for thresh in thresholds:
    # select features using threshold
    selection = SelectFromModel(rg_cv, threshold=thresh, prefit=True)
    select_X_train = selection.transform(X_train)

    # train model
    selection_model = rg_cv
    selection_model.fit(select_X_train, y_train)

    # eval model
    select_X_test = selection.transform(X_test)
    y_pred = selection_model.predict(select_X_test)
    predictions = [round(value) for value in y_pred]
    accuracy = accuracy_score(y_test, predictions)
    print("Thresh=%.3f, n=%d, Accuracy: %.2f%%" % (thresh, select_X_train.shape[1], accuracy * 100.0))
    print(confusion_matrix(y_test, predictions))
    print(classification_report(y_test, predictions))

However, the following error occurred:

ValueError: The underlying estimator GridSearchCV has no `coef_` or `feature_importances_` attribute. Either pass a fitted estimator to SelectFromModel or call fit before calling transform.

The error is related to this line:

 selection = SelectFromModel(rg_cv, threshold=thresh, prefit=True)

It should be modified as follows:

print("Tuned rf best params: {}".format(rg_cv.best_params_))
be=rg_cv.best_estimator_
best=be.fit(X_train,y_train)
thresholds = np.sort(rg_cv.best_estimator_.named_steps["clf"].feature_importances_)
for thresh in thresholds:
    # select features using threshold
    selection = SelectFromModel(be.steps [1][1], threshold=thresh, prefit=True)


Does label encoding an entire dataset cause data leakage? I have a dataset on which one of the features has a lot of different categorical values. Trying to use a LabelEncoder, OrdinalEncoder or a OneHotEncoder results in an error, since when splitting the data, the test set ends up having some values that are not present in the train set.
My question is: if I choose to encode my variables before splitting the data, does this cause data leakage?
I'm aware that I shouldn't perform any normalization or educated transformations on the data before splitting the dataset, but I couldn't find a solution for this problem inside scikit-learn.
Thanks in advance for any responses.
Edit: This particular features has very high cardinality, with around 60k possible values. So using scikit-learn's OneHotEncoder with handle_unknown set to ignore would introduce too many new columns to the dataset.

The cleanest solution would be to apply scikit's OneHotEncoder with the handle_unknown parameter set to "ignore":

handle_unknown{‘error’, ‘ignore’}, default=’error’
Whether to raise an error or ignore if an unknown categorical feature is present during transform (default is to raise). When this parameter is set to ‘ignore’ and an unknown category is encountered during transform, the resulting one-hot encoded columns for this feature will be all zeros. In the inverse transform, an unknown category will be denoted as None.

Other manual solution are described in this and this question on Stackoverflow, for example.


predict() function in R While predicting, what happens if we pass the newdata along with the target variable? Do we need to isolate the target variable before feeding into predict function?

data = read.csv("diabetes.csv")  #768 observations of 9 variables

#I split the data into train and test
set.seed(123)
IDS = 1:nrow(data)
train_data = sample(x = IDS,size = nrow(data)*0.7)
test_data = sample(x = setdiff(IDS,train_data),size = nrow(data)*0.3)
train_data = data[train_data,]
test_data = data[test_data,]

#Build logistic regression model
log_model1 = glm(formula = Outcome ~ .,data = train_data,family = 'binomial')

#Predict it on test or train
train_predict = predict(object = log_model1,newdata = train_data,type = "response")

As is common in R the answer depends and you should always double check to make sure including the target (or other) doesn’t affect the answer. But generally it won’t. Superfluous features are ignored as long as newdata includes all expected predictors present during training.


Use the output of 2 hidden neurons in the last hidden layer of a NN to visualize the result of a 4-class classification task I am working on a 4-class classification task using an Artificial Neural Network. My aim is to visualize how well those 4 classes can be separated, moreover how consistent the data coming from each class is and how far they are "away" from one another.

My approach is to use 2 neurons in the last hidden layer and plot the output of the neurons in a scatterplot. However my current output usually looks like this where the different colors stand for the actual classes the data is coming from:

Obviously this is not very useful. First of all I was hoping that the points would not all be all along the same line and second of all I am not sure if this can even be interpreted in the way that I want.

I am using the neuralnet package in R to train the network using linear output. I am guessing that if this approach can work the choice of activation function would be crucial.

I am new to Neural Networks and I am not sure whether or not something like this can be done or not and I am hoping to get some comments.

I'm somewhat new to the topic as well, but I think what you are looking for is an autoencoder.  I have only used it in the h2o deeplearning package, but it seems to work well.

The idea behind an auto-encoder is to begin with your inputs, encode down to less nodes (two is nice for visual representation), then decode back to your original outputs (ass accurately as possible).

If the auto-encoder can decode back to an accurate representation of your inputs, then there was sufficient important information withheld in those two nodes.  You can then plot the features that those two nodes held in a two-dimensional plot that would be analogous to a bounded PCA plot.

Also - the auto-encoder is not constrained to linearity like the PCA is.  Below is a quick snippet of some code, hope it helps.

library(h2o)
localH2O = h2o.init(ip = "localhost", port = 54321, startH2O = TRUE, min_mem_size = "3g", max_mem_size = "4g", nthreads = -1)
dat_h2o <- as.h2o(train.df)

unsupervised <-
  h2o.deeplearning(x = 24:356,  # column numbers to use
                   training_frame = dat_h2o, # data in H2O format
                   autoencoder = TRUE, ## unsupervised autoencoding
                   activation = "Tanh", # or 'Tanh' 'Rectifier' 'WithDropout' node activation function, Tanh seems to work best for autoencoding
                   hidden = c(5,2,5), # three layers of nodes, with 5/2/5 nodes, respectively
                   epochs = 1) # max. no. of epochs

## layer 1 corresponds to hidden[1], so it will reduce to 5 variables
## below is roughly similar to using predict(pca.object, newdata)
training_data <- h2o.deepfeatures(unsupervised, dat_h2o, layer = 1)
testing_data <- h2o.deepfeatures(unsupervised, test_h2o, layer = 1)

## explore the second layer with 2 nodes, similar to exploring PC1 vs PC2
train_supervised_features2 = h2o.deepfeatures(unsupervised, dat_h2o, layer=2)

plotdata2 = as.data.frame(train_supervised_features2)
plotdata2$label = as.character(as.vector(dat_h2o[,364]))

## L2 corresponds to layer 2, so use L2
qplot(DF.L2.C1, DF.L2.C2, data = plotdata2, color = label, main = 'Neural network: 5-2-5')


Function Callers Vs Data Scientists In our organization, there are many people who are into analytics and data science who are OK in their work with in the sense that they know what packages in R/Python etc to use, what algorithms to call for a particular type of problem etc. The problem is that they have very little knowledge of underlying mathematics and are not able to think beyond calling ready made functions i.e. they can never build a custom solution to specific problem. Let us call them as Function Callers as opposed a genuine Data Scientist who has a fair knowledge of the underlying mathematics.

We want to conduction a training course to teach the mathematical foundations of machine learning to the Function Callers and help them become Data Scientists. For this we have shortlisted some ten topics namely.

• Probability
• Various Probability distributions and data
• Maximum likelihood methods
• Linear Algebra (advanced)
• Single variable calculus, Multi variable calculus
• Vector calculus
• Graphical models
• Bayesian networks
• Optimization techniques
• Statistical models

This is going to be an 80 hour course so it is not possible to cover everything in detail like a college/university course.

Question: Assume that the course will have enough time to teach and implement only 3 or 4 things in each of the topics mentioned above. So given this background, what are the best 3 or 4 things that can be covered under each topic?

Note: The target audience have either science or college level mathematics background but they do not have deep knowledge about the mathematics that is used in machine learning.

I've done my Masters in Analytics from National University of Singapore.

The answer which I'm going to give is based on my experience.

Ofcourse all the topics are important in the above mentioned list but if you need to prioritize from the topics which you have mentioned above I think the following are very important for any person who is working in a Data Science field. :

Various Probability and Data Distribution: Here you can give basic intro on Probability. You don't need a separate Probability class. A bit more its applications.
Single variable calculus/Multi variable calculus: There are very important, when you try to forecast something this plays an important role.
Graphical Models: Yes, these are helpful in implementing all kinds of community detection algorithms, social networking and many more.
Bayesian Network: This is one of the basic techniques which is important and should be known by all statisticians.
Statistical Models: They need to be exposed to different statistical models as you know there are many models which can perform better in different scenarios, It comes only by practice.

Rest are not so important at the start, you can conduct another session if necessary based on the outcome of the 1st session. Where you can cover all the topics like Linear Algebra (advanced),  Vector calculus, Optimization techniques, Maximum likelihood methods. As you know these techniques would come into play after getting the basic models ready and you would use these if you want to improve the accuracy of the model or tweaking the model WRT to your Business Problem.

I hope this answer may help you.


Iterative Machine Learning I was wondering what percentage of Machine Learning Algorithms are iterative in nature; i.e. I can run them sequentially, on different set of data and the model refines itself. Are there any algorithms which support this mathematically, moreover, is there some tradeoff in runtime or accuracy of these models.

Iteration here not to be confused with : What does iteration in Machine Learning mean?

what percentage of Machine Learning Algorithms are iterative in nature; i.e. I can run them sequentially

from deep learning book ch5, SGD

A recurring problem in machine learning is that large training sets are necessary for good generalization, but large training sets are also more computationally expensive.

Prior to the advent of deep learning, the main way to learn nonlinear models was to use the kernel trick in combination with a linear model. Many kernel learning algorithms require constructing an m×m matrix. Constructing this matrix has computational cost O(m^2), which is clearly undesirable for datasets with billions of examples.

Gradient descent, in general, has often been regarded as slow or unreliable. In the past, the application of gradient descent to non-convex optimization problems was regarded as foolhardy or unprincipled, The optimization algorithm may not be guaranteed to arrive at even a local minimum in a reasonable amount of time, but it often finds a very low value of the cost function quickly enough to be useful

Gradient Descent-based methods, the basic premise is that you estimate the gradient based on a random small set of the training data (batch) and update the weights accordingly, the algorithm will iteratively keep alternating between the two steps until some objective has been satisfied (find a good local minima in the parameter space), the procedure is called stochastic gradient descent (The insight of SGD is that the gradient is an expectation. The expectation may be approximately estimated using a small set of samples).
In the extreme case, the gradient is estimated based on one single example, (online learning)
see Practical Recommendations for Gradient-Based Training of Deep
Architectures

  is there some tradeoff in runtime or accuracy of these models

For a fixed model size, the cost per SGD update does not depend on the
training set size m. In practice, we often use a larger model as the training set size
increases, but we are not forced to do so. The number of updates required to reach convergence usually increases with training set size. However, as m approaches infinity, the model will eventually converge to its best possible test error before SGD has sampled every example in the training set. Increasing m further will not extend the amount of training time needed to reach the model’s best possible test error. From this point of view, one can argue that the asymptotic cost of training a model with SGD is O(1) as a function of m.


Reinforcement Learning on real time data over a web server Question: is it possible to implement a reinforcement learning model over a NodeJS server?

This server would be receiving binary forms of data (open /close; yes/no) in real time.

The objective for the model is to learn from the incoming data for a few days and then predict values for next day (similar to time series, but not exactly). The learned parameters need to be flushed every week.

Note: the training data I have doesn't allow for supervised learning - i.e. there is no data for the target class.

Yes - You need to define an Agent, Environment, State, and Rewards:

Agent - The current week process.
Environment - Server.
State - Current data.
Rewards - Predicted future values are the same observed values.

Then implement in JavaScript. One JavaScript framework is REINFORCEjs


Applications and differences for Jaccard similarity and Cosine Similarity Jaccard similarity and cosine similarity are two very common measurements while comparing item similarities. However, I am not very clear in what situation which one should be preferable than another.

Can somebody help clarify the differences of these two measurements (the difference in concept or principle, not the definition or computation) and their preferable applications?

The answer from saq7 is wrong, as well as not answering the question.

∥A∥ means the $L2$ norm of $A$, i.e. the length of the vector in Euclidean
space, not the dimensionality of the vector $A$.  In other words, you
don't count the 0 bits, you add up only the 1 bits and take the square root.

Sorry I don't have a real answer as to when you should use which metric,
but I can't let the incorrect answer go unchallenged.


When to use BayesianSearchCV and how it works? Can somebody highlight when to use BayesianSearchCV and how it works? I have seen the implementation of same on kaggle and wanted to explore it further.

Below is the link where the implementation has been performed.

Kaggle link to Bayesian Optimization

As mentioned in that Kaggle notebook, you can use it pretty much as just a drop-in replacement for other search methods (grid or random).  Bayesian searches still are random searches over a predefined search space/distribution, but now the algorithm pays attention to how well hyperparameter combinations perform, and will put more emphasis on high-performing areas.

There are several ways to do this.  In BayesianSearchCV, a surrogate function keeps track of how the algorithm thinks hyperparameters will perform, with some uncertainty.  Yet another function, an acquisition function, balances the desire to exploit high-performing areas with that to decrease uncertainty about unexplored areas (to avoid falling into a local optimum).  A couple of nice descriptions:
http://neupy.com/2016/12/17/hyperparameter_optimization_for_neural_networks.html#bayesian-optimization
http://krasserm.github.io/2018/03/21/bayesian-optimization/

SciKit-Opt is specifically tailored for optimizing very expensive functions; it spends a fair amount of time updating its surrogate and acquisition functions and choosing the next hyperparameter combination to test.  By doing so, it generally requires fewer iterations to get good results, but spends longer per iteration.  This is probably a good tradeoff for, say, neural networks, but less clearly for XGBoost.  But if you really need to eke out the last ounce of performance, it might be worth doing Bayesian over a purely random search.
https://github.com/scikit-optimize/scikit-optimize/issues/334#issuecomment-290358126
http://krasserm.github.io/2018/03/21/bayesian-optimization/
https://scikit-optimize.github.io/notebooks/bayesian-optimization.html

(The search begins with a number of randomly selected hyperparameter combinations, defaulting to 10 in skopt.  So in that version of the Kaggle notebook, actually just a random search is being performed.)


What happens if we use Z-Score(Mean Normalisation) in Neural Networks to standardize values I'm currently doing the DeepLearning.ai Specialization where they divide pixel intensity with the maximum value it has (255) to standardize the data when working on a classification NN for cats.
What effect does Z_Score have on this?
Will my code fail becuase pixels cannot have negative values which will be assigned by Z-Score?

By normalizing the data the code is not likely to get negatively affected. This is because the network doesn't know a priori that images are being used as input data, it only receives a set of numeric values (which can represent any type of data) and finds the appropiate values of weights and biases that decrease our cost function.
Thereby even if we get negative values as the new input data, that doesn't mean that this preprocessing is going to cause a bad effect.
But, why normalization may be useful?
The useful thing about Z-score normalization here is that by doing it, all the "new pixels" have a value with zero-mean and the same variance. This is a good thing as explains Yann LeCun in his paper "Efficient Backprop" (page 8):

Why use data with features having a mean close to zero:

Convergence is usually faster if the average of each input variable over the training set is close to zero...  ....When all of the components
of an input vector are positive, all of the updates of weights that feed into a
node will be the same sign. As a result, these weights can only
all decrease or all increase together for a given input pattern. Thus, if a weight
vector must change direction it can only do so by zigzagging which is inefficient
and thus very slow

The expression that he is refering to is the one that is used to update the value of the weights of the first layer, which is given by:
$$ \frac{\partial C}{\partial w_{jk}^{l=1}} = \delta_j^{l=1}x_k$$
Where $x_k$ represents an input pixel value and $w_{jk}^l$ a weight that connects that input $k$ to the neuron $j$ of the first layer ($l=1$). So, given the scalar term $\delta_j^{l=1}$ it's clear that all the weights that connect the input layer to the first layer of neurons will be updated in the same direction if all $x_k$ are positive, as the quote explains.

Why use data with features having the same variance?:

Scaling speeds
learning because it helps to balance out the rate at which the weights connected
to the input nodes learn.

As we saw earlier, the expression that gives the updates on the weights $w_{jk}^{l=1}$ are proportional to the input $x_k$. So if e.g. $x_1$ samples have bigger values than $x_2$ samples, then the updates on $w_{j1}$ on $w_{j2}$ may not be balanced and therefore the updates are made with different rythms on both parameters.
Just a side note $\rightarrow$ The method of dividing each feature (each pixel intensity) by their maximum value is not a Z-score normalization but is another way of normalizing the data (in order to use the Z-score, the quantity that we would have to use to divide the features is the standard deviation of each pixel throughout all the samples).


Is FPGrowth still considered "state of the art" in frequent pattern mining? As far as I know the development of algorithms to solve the Frequent Pattern Mining (FPM) problem, the road of improvements have some main checkpoints. Firstly, the Apriori algorithm was proposed in 1993, by Agrawal et al., along with the formalization of the problem. The algorithm was able to strip-off some sets from the 2^n - 1 sets (powerset) by using a lattice to maintain the data. A drawback of the approach was the need to re-read the database to compute the frequency of each set expanded.

Later, on year 1997, Zaki et al. proposed the algorithm Eclat, which inserted the resulting frequency of each set inside the lattice. This was done by adding, at each node of the lattice, the set of transaction-ids that had the items from root to the referred node. The main contribution is that one does not have to re-read the entire dataset to know the frequency of each set, but the memory required to keep such data structure built may exceed the size of the dataset itself.

In 2000, Han et al. proposed an algorithm named FPGrowth, along with a prefix-tree data structure named FPTree. The algorithm was able to provide significant data compression, while also granting that only frequent itemsets would be yielded (without candidate itemset generation). This was done mainly by sorting the items of each transaction in decreasing order, so that the most frequent items are the ones with the least repetitions in the tree data structure. Since the frequency only descends while traversing the tree in-depth, the algorithm is able to strip-off non-frequent itemsets.

Edit:

As far as I know, this may be considered a state-of-the-art algorithm, but I'd like to know about other proposed solutions. What other algorithms for FPM are considered "state-of-the-art"? What is the intuition/main-contribution of such algorithms?

Is the FPGrowth algorithm still considered "state of the art" in frequent pattern mining? If not, what algorithm(s) may extract frequent itemsets from large datasets more efficiently?

State of the art as in: used in practise or worked on in theory?

APRIORI is used everywhere, except in developing new frequent itemset algorithms. It's easy to implement, and easy to reuse in very different domains. You'll find hundreds of APRIORI implementations of varying quality. And it's easy to get APRIORI wrong, actually.

FPgrowth is much harder to implement, but also much more interesting. So from an academic point of view, everybody tries to improve FPgrowth - getting work based on APRIORI accepted will be very hard by now.

If you have a good implementation, every algorithm has it's good and it's bad situations in my opinion. A good APRIORI implementation will only need to scan the database k times to find all frequent itemsets of length k. In particular if your data fits into main memory this is cheap. What can kill APRIORI is too many frequent 2-itemsets (in particular when you don't use a Trie and similar acceleration techniques etc.). It works best on large data with a low number of frequent itemsets.

Eclat works on columns; but it needs to read each column much more often. There is some work on diffsets to reduce this work. If your data does not fit into main memory, Eclat suffers probably more than Apriori. By going depth first, it will also be able to return a first interesting result much earlier than Apriori, and you can use these results to adjust parameters; so you need less iterations to find good parameters. But by design, it cannot exploit pruning as neatly as Apriori did.

FPGrowth compresses the data set into the tree. This works best when you have lots of duplicate records. You could probably reap of quite some gains for Apriori and Eclat too if you can presort your data and merge duplicates into weighted vectors. FPGrowth does this at an extreme level. The drawback is that the implementation is much harder; and once this tree does not fit into memory anymore it gets a mess to implement.

As for performance results and benchmarks - don't trust them. There are so many things to implement incorrectly. Try 10 different implementations, and you get 10 very different performance results. In particular for APRIORI, I have the impression that most implementations are broken in the sense of missing some of the main contributions of APRIORI... and of those that have these parts right, the quality of optimizations varies a lot.

There are actually even papers on how to implement these algorithms efficiently:

  Efficient Implementations of Apriori and Eclat. Christian BorgeltWorkshop of Frequent Item Set Mining Implementations (FIMI 2003, Melbourne, FL, USA).

You may also want to read these surveys on this domain:

  Goethals, Bart. "Survey on frequent pattern mining." Univ. of Helsinki (2003).

  Ferenc Bodon, A Survey on Frequent Itemset Mining, Technical Report, Budapest University of Technology and Economic, 2006,

  Frequent Item Set MiningChristian BorgeltWiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 2(6):437-456. 2012


Scikitlearn grid search random forest using oob as metric? Have looked at data on oob but would like to use it as a metric in a grid search on a Random Forest classifier (multiclass) but doesn't seem to be a recognised scorer for the scoring parameter. I do have OoB set to True in the classifier.
Currently using scoring ='accuracy' but would like to change to oob score.
Ideas or comments welcome

Check out the make_scorer function in sklearn: http://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html

You may need to code the OoB component yourself, I haven't ever used that metric for scoring before with Sklearn, so I don't know for sure.

Also, consider using random search rather than grid search, it is more practical when dealing with a large hyper parameter space, since it does not need to search the entire dimension of a parameters that have no impact.


Getting to other parameter sets from GridSearchCV in sklearn I am using GridSearchCV to train a dataset with several parameters.  Using the methods on the object I am able to get to the "best" parameter set (based on the scoring function).  I would like to be able to get to one (or more) of the other models.  Is there a way to do that?  Is that saved?  If so what is the way to get to those?

import numpy as np
import pandas as pd
from sklearn import datasets
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import GridSearchCV

bc = datasets.load_breast_cancer()
X,Y = bc["data"],bc["target"]

sgd = SGDClassifier(loss='log', penalty='l1', max_iter=1E6, tol=1E-6, shuffle=True, random_state=123456)
param_grid = [{'alpha':np.append(10.**-np.arange(3,6), 10.**-np.arange(3,6)/2)}]

sgd_gs = GridSearchCV(sgd, param_grid, scoring='roc_auc', n_jobs=4, cv=4)
sgd_gs.fit(X, Y)
results_pd = pd.DataFrame(sgd_gs.cv_results_).iloc[:,2:7].sort_values(by='rank_test_score').reset_index(drop=True)
results_pd

scores_train = sgd_gs.best_estimator_.predict_proba(X)[:,1]
print sum(scores_train)

#**code below does not seem to work**
params = results_pd.loc[2, 'params']
scores_train = sgd_gs.best_estimator_.set_params(**params).predict_proba(X)[:,1]
print sum(scores_train)

If the parameter refit is set to True, the GridSearchCV object will have the attributes best_estimator_, best_score_ etc. ("Best" measured in terms of the metric provided through the scoring parameter.)

For other models you can only access the cross-validation scores from cv_results_. The models are not stored. If you want to use them to make predictions, you have the re-train the model where you explicitly set the parameters.

If you want to use all the models to make predictions, it would be better to write your own code and simply loop over the parameter constellations without using GridSearchCV.


Create new column with rounded values I am trying to create a new column in this dataset. I would like to have rating to be 1, 2, 3, 4 or 5 instead of 1.25, 2.00, 3.75, ..., 5.00.
Can you help me? The only solution I found is this

chocolate$Rat = gsub("1.25", "1", chocolate$Rating)

but I know it is not a good way to work. Is there something faster which can to the work efficiently?
Thanks

The R function is “round”. The default is to round to the nearest integer, which is what you want. The function takes vector inputs, so you can apply it to your column in one line instead of using a “for” loop.

Then you’ll tack in on to your original data in any number of ways, such as “expanded_data <- data.frame(original_data, round(column))”.


Val Accuracy not increasing at all even through training loss is decreasing I am training a model for image classification, my training accuracy is increasing and training loss is also decreasing but validation accuracy remains constant.

Here is my code:

from keras.applications.vgg19 import VGG19 model= VGG19(include_top=False, weights='imagenet',
        input_tensor=None, input_shape=(224,224,3), pooling=None, classes=1000) x=model.output

x=Conv2D(filters=1024,kernel_size=2)(x)
x=MaxPooling2D()(x)
x=Flatten()(x)
x=Dense(1024,activation='relu')(x)
x=BatchNormalization(axis=1)(x)
x=Dropout(0.8)(x)
x=Dense(64,activation='relu')(x)
x=Dense(4,activation='softmax')(x)

model = Model(inputs=model.input,outputs=x)

for layer in model.layers[:12]:
   layer.trainable = False

for layer in model.layers[12:]:
   layer.trainable=True

opt = Adam(lr=0.0001, decay=1e-6)
model.compile(loss='sparse_categorical_crossentropy',
           optimizer=opt, metrics=['accuracy'])

checkpoint_path="/content/drive/My Drive/Model/model_vgg19_6.h5"

checkpoint = ModelCheckpoint(checkpoint_path, monitor="val_acc", mode="max",
   save_best_only = True,verbose=1)

reduce_lr = ReduceLROnPlateau(monitor = 'val_acc', mode="max", factor = 0.7,
     patience = 5, verbose = 1, min_delta =0.00001)

earlystop = EarlyStopping(monitor = 'val_acc', mode="max", min_delta = 0, patience = 30, verbose = 1,
            restore_best_weights = True)

callbacks = [reduce_lr,checkpoint]

model.fit_generator(aug_train, steps_per_epoch=int((len(data_x)/128)+1), validation_data=
                (val_x,val_y), validation_steps=int((len(val_x)/128)+1), workers=-1,
                use_multiprocessing=True, shuffle=True, epochs=300, callbacks=callbacks )

I am getting a constant val_acc of 0.24541

You start with a VGG net that is pre-trained on ImageNet - this likely means the weights are not going to change a lot (without further modifications or drastically increasing the learning rate, for example).

If you are expecting the performance to increase on a pre-trained network, you are performing fine-tuning. There is a section on fine-tuning the Keras implementation of the InceptionV3 network, but the principals are the same: you should freeze some of the earlier feature-extraction layers, leaving only some of the final layers marked as trainable. This is the example given in the docs, where they add new layers to the base model, train only those layers for a while, then additionally unfreeze some of the base model.

# create the base pre-trained model
base_model = InceptionV3(weights='imagenet', include_top=False)   # <--- leave out final layers!
# first: train only the top layers (which were randomly initialized)
# i.e. freeze all convolutional InceptionV3 layers
for layer in base_model.layers:
    layer.trainable = False

-> run training for a few epochs

The some time later, unfreeze the part of the base model

# we chose to train the top 2 inception blocks, i.e. we will freeze
# the first 249 layers and unfreeze the rest:
for layer in model.layers[:249]:
   layer.trainable = False
for layer in model.layers[249:]:
   layer.trainable = True

Please looked at the full documentation for more details.


Best approach for a simple self driving car I'm planning to build a small car with autonomous driving (maybe modifying my current rc car or using a robot car kit, using arduino and raspberry).

I'll use a CNN, and I'm thinking how to collect data (I want to try a similar approach to the Udacity simulator).
My doubt is if is better to aim at supervised learning or reinforcement learning.

I'm more inclined to the supervised learning, but I don't know the best way to record data. If I'll go with the robot car kit, the front wheel doesn't steer (it use different wheels speed to turn left or right). So I could create 3 function for each direction, like "soft left", "medium left" and "hard left" to control the car when I train it, and then predict from 7 possibility:
- soft left, medium left, hard left, center, soft right, medium right, hard right
So 7 possible output.

Otherwise if I use a car with the front wheel that are able to steer, I have 3 possible class, like -1 for left button, 0 for center (any button pressed) and 1 for right button. So 3 possible output, without any gradient for the steering angle.

With the reinforcement learning honestly I don't know where to start. How can I tell to the car when it does something wrong and give her a reward? With a simulator is simple, but with a real car how can I assing a reward score for each frame that I collect? If you have any paper about this I appreciate it.

What's the best approach in your opinion?

~ If we consider the problem as an image classification task,

Basically, we can use a CNN trained on images belonging to 7 classes ( which represent the steering levels ). We give an image to the CNN and it outputs a steering level. This is a classification problem which will take each frame ( image ) of the video and output the steering level. So as the car steers the probability for the centre steering level will increase assuming that the road is straight after taking the turn.

See multiclass image classification.

~ else, if we train an agent for the problem ( reinforcement learning ),

Reinforcement learning requires an agent ( in our case, the car ) which produces some action ( steering levels ) in an environment ( street, road etc. ) and gets an reward accordingly. The rewards need to be maximized.

We will have a CNN which encodes an image to a vector representation.

  So, consider two cases. We have an image which needs to the car to steer hard left. In the first case, the car turns soft left and loses its path. Here, the reward for this action becomes -100. In the other case, the car turns hard left, stays on the road and hence gets a reward of 100.

Basically, the agent gets a positive reward if it stays on the road doing some action ( steering level ) else it gets a negative reward showing that this action should not be made on such an image ( road ).

See here for learning RL.

Tip:

You can try combining both the above methods. Generate class probabilities ( steering levels ) from CNN. If the car does not steer well, increase the loss by a certain factor.

  Try thinking from a human perspective. Imagine driving a car and the cautions you take to avoid going out the road.


Creating an Object Detection model with images and coordinates of bounding boxes I am trying to build a custom Object Detection model which can detect guns from a given image. The dataset is from here.

The dataset has a good number of images and each image has 4 coordinates of bounding boxes with it. I recently read about YOLO and the structure of its labels is as follows:

I need to get this label for every grid cell of the input image. The dataset which I have contains coordinates for the object and not a grid cell in the image.

  Also, I can manage to get bx, by, bh, bw and c1 , c2 , c3, but how
  do I extract pc for training from the dataset?

I will like to build a YOLO model from scratch but I don't know how to transform the dataset so that I can train a YOLO on it.

So, I found the answer myself. Basically, if the grid cell contains the centre of the bounding box then the grid cell will have a $p_c$ of 1 otherwise 0 if there is no object.


Algorithm for multiple extended string matching I need to implement an algorithm for multiple extended string matching in text. Algorithms to match regular expression would be perhaps too slow.

Extended means the presence of wildcards (any number of characters instead of a star), for example:

abc*def //matches abcdef, abcpppppdef etc.

Multiple means that the search is going on simultaneously for multiple string patterns (not a separate search for each pattern), for example:

abc*def
abc
whatever
some*string

QUESTION:

What is the fast algorithm that can do multiple extended string matching?

Preferably, optimized for SIMD instructions and multicore implementation. Open source implementation (C/C++/Python) would be great as well. I'm interested in 10 Gbps+ performance on a single core of a modern CPU.

Thank you

Do you change the patterns often? If not, then you can use Aho-Corasick method, whose idea is, first, to build a finite automaton based on your patterns and, then, to make a single pass over the text with this automaton to find matches (if the automaton visits a "matching" state, then there is a match). The complexity of the automata building should be linear in the length of the patterns (# of patters * max pattern length), and the matching phase should be linear in the size of the text you search in.


How to define a custom performance metric in Keras? I tried to define a custom metric fuction (F1-Score) in Keras (Tensorflow backend) according to the following:

def f1_score(tags, predicted):

    tags = set(tags)
    predicted = set(predicted)

    tp = len(tags & predicted)
    fp = len(predicted) - tp
    fn = len(tags) - tp

    if tp>0:
        precision=float(tp)/(tp+fp)
        recall=float(tp)/(tp+fn)
        return 2*((precision*recall)/(precision+recall))
    else:
        return 0

So far, so good, but when I try to apply it in model compilation:

model1.compile(loss="binary_crossentropy", optimizer=Adam(), metrics=[f1_score])

it gives error:

TypeError                                 Traceback (most recent call last)
<ipython-input-85-4eca4def003f> in <module>()
      5 model1.add(Dense(output_dim=10, activation="sigmoid"))
      6
----> 7 model1.compile(loss="binary_crossentropy", optimizer=Adam(), metrics=[f1_score])
      8
      9 h=model1.fit(X_train, Y_train, batch_size=500, nb_epoch=5, verbose=True, validation_split=0.1)

/home/buda/anaconda2/lib/python2.7/site-packages/keras/models.pyc in compile(self, optimizer, loss, metrics, sample_weight_mode, **kwargs)
    522                            metrics=metrics,
    523                            sample_weight_mode=sample_weight_mode,
--> 524                            **kwargs)
    525         self.optimizer = self.model.optimizer
    526         self.loss = self.model.loss

/home/buda/anaconda2/lib/python2.7/site-packages/keras/engine/training.pyc in compile(self, optimizer, loss, metrics, loss_weights, sample_weight_mode, **kwargs)
    664                 else:
    665                     metric_fn = metrics_module.get(metric)
--> 666                     self.metrics_tensors.append(metric_fn(y_true, y_pred))
    667                     if len(self.output_names) == 1:
    668                         self.metrics_names.append(metric_fn.__name__)

<ipython-input-84-b8a5752b6d55> in f1_score(tags, predicted)
      4     #tf.convert_to_tensor(img.eval())
      5
----> 6     tags = set(tags)
      7     predicted = set(predicted)
      8

/home/buda/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.pyc in __iter__(self)
    493       TypeError: when invoked.
    494     """
--> 495     raise TypeError("'Tensor' object is not iterable.")
    496
    497   def __bool__(self):

TypeError: 'Tensor' object is not iterable.

What is the problem here? The fact that my f1_score function inputs are not Tensorflow arrays? If so, where/how can I convert them correctly?

You have to use Keras backend functions. Unfortunately they do not support the &-operator, so that you have to build a workaround: We generate matrices of the dimension batch_size x 3, where (e.g. for true positive) the first column is the ground truth vector, the second the actual prediction and the third is kind of a label-helper column, that contains in the case of true positive only ones. Then we check which instances are positive instances, are predicted as positive and the label-helper is also positive. Those are the true positives.
We can make this analog with false positives, false negatives and true negatives with some reverse-calculations of the labels.
Your f1-metric may look as follows:
def f1_score(y_true, y_pred):
    """
    f1 score

    :param y_true:
    :param y_pred:
    :return:
    """
    tp_3d = K.concatenate(
        [
            K.cast(y_true, 'bool'),
            K.cast(K.round(y_pred), 'bool'),
            K.cast(K.ones_like(y_pred), 'bool')
        ], axis=1
    )

    fp_3d = K.concatenate(
        [
            K.cast(K.abs(y_true - K.ones_like(y_true)), 'bool'),
            K.cast(K.round(y_pred), 'bool'),
            K.cast(K.ones_like(y_pred), 'bool')
        ], axis=1
    )

    fn_3d = K.concatenate(
        [
            K.cast(y_true, 'bool'),
            K.cast(K.abs(K.round(y_pred) - K.ones_like(y_pred)), 'bool'),
            K.cast(K.ones_like(y_pred), 'bool')
        ], axis=1
    )

    tp = K.sum(K.cast(K.all(tp_3d, axis=1), 'int32'))
    fp = K.sum(K.cast(K.all(fp_3d, axis=1), 'int32'))
    fn = K.sum(K.cast(K.all(fn_3d, axis=1), 'int32'))

    precision = tp / (tp + fp)
    recall = tp / (tp + fn)
    return 2 * ((precision * recall) / (precision + recall))

Since the Keras-backend calculator returns nan for division by zero, we do not need the if-else-statement for the return statement.
Edit:
I have found a pretty good idea for a exact implementation. The problem with our first approach is, that it is only "approximated", since it is computed batchwise and subsequently averaged. One could also calculate this after each epoch with the keras.callbacks. Please find the idea here.
An example implementation would be:
import keras
import numpy as np
import sklearn.metrics as sklm

class Metrics(keras.callbacks.Callback):
    def on_train_begin(self, logs={}):
        self.confusion = []
        self.precision = []
        self.recall = []
        self.f1s = []
        self.kappa = []
        self.auc = []

    def on_epoch_end(self, epoch, logs={}):
        score = np.asarray(self.model.predict(self.validation_data[0]))
        predict = np.round(np.asarray(self.model.predict(self.validation_data[0])))
        targ = self.validation_data[1]

        self.auc.append(sklm.roc_auc_score(targ, score))
        self.confusion.append(sklm.confusion_matrix(targ, predict))
        self.precision.append(sklm.precision_score(targ, predict))
        self.recall.append(sklm.recall_score(targ, predict))
        self.f1s.append(sklm.f1_score(targ, predict))
        self.kappa.append(sklm.cohen_kappa_score(targ, predict))

        return

To make the network to call this function you simply add it to you callbacks like
metrics = Metrics()
model.fit(
    train_instances.x,
    train_instances.y,
    batch_size,
    epochs,
    verbose=2,
    callbacks=[metrics],
    validation_data=(valid_instances.x, valid_instances.y),
)

Then you can simply access the members of the metrics variable.


Scikit Learn Missing Data - Categorical values I have a dataset containing categorical features, which has 4 labels, and 4 features. (It is a meta classifier, so outputs from base classifier serve as input into this classifier)
Label  Feat1 Feat2 Feat3 Feat4
 1      1     1      2     2
 2      3     1      2     2
 3      4     3      3     1
 4      4     1      2     4

I'm using scikit learn, and am considering using Naive Bayes or a Decision Tree. The classifier needs to be able to deal with missing features, and I read on scikit learn's page that Decision Tree does not support missing values.
What I am looking for is advice as to how to approach missing categorical values when using scikit learn. Also, any links to academic papers addressing this would be appreciated.

We would need more information on the prediction problem and the features to be able to give something more precise.

Anyhow, I am surprised no answer so far included all possible options since they aren't that many:

get rid of incomplete observations or features --- obviously, only viable if there are few incomplete cases since you lose too much information otherwise
replace NAs with some value like -1 --- this depends on the classifier you use; if your classifier supports categorical variables, you can create a new category for those NAs for example. In some continuous variables, sometimes there are some values that make sense (for instance, in text mining classification, if you have a title-length feature but you have no title, it might make sense to replace with title-length=0)
fill up the missing data

This last point encompasses too many things:

replace NAs with the median (this is the usual lazy approach; sklearn has a class for this)
if time series, replace with an average of the previous and following values -- in pandas, this can be done using DataFrame.resample().
use the $k$ closest neighbors. build a KNN model using the other variables and then do the average of those neighbors (if you use euclidean distance, you probably should normalize first). I never seen this done, but you probably could try predicting the missing NAs using another model as well.

But all this depends very much on what you are doing. For instance, if you have performed clustering analysis and you know your data is made up of clusters, you could use the median within each cluster.

Possibly other solutions could include things like multimodal or multiview models. These are recent techniques that can cope with missing modalities, and you can see a feature, or subset of features, as a modality. For instance, you could build a different classifier for various subsets of your features (using the complete cases in each of those subsets) and then build another classifier on top of that to merge those probabilities. I would only try these techniques if most of your data is missing. There are more advanced deep learning versions of this using autoencoders.


Sklearn Pipelines - How to carry over PCA? I'm developing a pipeline to fit parameters for a gradient boosting classifier while also fitting the optimum number of features in a PCA model. This is the current setup:

pipe = Pipeline([
    ('reduce_dim', PCA()),
    ('classify', GradientBoostingClassifier())
])
score = {'f1': 'f1', 'accuracy': 'accuracy'}
N_FEATURES_OPTIONS = [12,13,14,15]
max_dep = [3,4,5,6]
n_est = [50,80,100, 120, 150]
min_samp = [4,5,6,10]
param_grid = [
    {
        'reduce_dim': [PCA()],
        'reduce_dim__n_components': N_FEATURES_OPTIONS,
        'classify__n_estimators': n_est,
        'classify__max_depth': max_dep,
        'classify__min_samples_split':min_samp
    }]
reducer_labels = ['PCA']

grid_adc = GridSearchCV(pipe, cv=5, n_jobs=-1, param_grid=param_grid, scoring=score, refit='accuracy')
grid_adc.fit(X_train, y_train)
grid_adc.best_params_

Which outputs:

{'classify__max_depth': 3,
 'classify__min_samples_split': 4,
 'classify__n_estimators': 50,
 'reduce_dim': PCA(copy=True, iterated_power='auto', n_components=12, random_state=None,
   svd_solver='auto', tol=0.0, whiten=False),
 'reduce_dim__n_components': 12}

Now I'd like to validate and score the model with cross validation. If I run the following:

cross_val_score(grid, X_train, y_train, cv=5, n_jobs=-1)

Does the PCA I fit from the pipeline carry over to the cross_val_score function? If so, does the cross_val_score function transform the data with the PCA every time it generates a new train/test split?

Or do I need to create a new PCA after the pipeline to fit into the cross_val_score function?

When doing GridSearchCv, the best model is already scored. You can access it with the attribute best_score_ and get the model with best_estimator_. You do not need to re-score it in a cross validation.

Also, yes, the pipeline is entirely fitted when doing each split during the cv.


Understanding notation of Goodfellow's GAN objective function What is the meaning of $V(D,G)$? How do we get these expectation parts?
I was trying to understand it following this article: Understanding Generative Adversarial Networks (D.Seita), but, after many tries, I still can't understand how he got from $\sum_{n=1}^{N} \log D(x)$ to $\mathbb{E}(\log(D(x))$.

You are thinking of it backwards :) i.e. don't ask "How do we get these expectation parts?" or "how he got from" sum to expectation, ask where we got the sums!

The expectations are what we actually care about. The sums are just what we use to approximate them.

Generally in ML we define the data space $\mathcal{X}$ (e.g., the set of all possible images) and a probability distribution $\mathcal{P}$ on that space (so that sampling from it gives e.g., the set of natural images). The goal is then to find a function $f$ such that the expected loss $ \mathbb{E}_{x\sim\mathcal{P}}[\mathcal{L}(f(x))] $ is minimal. (This is a weighted average over all possible images, weighted such that only the images we care about are considered).
This is the basic tenet of empirical risk minimization, upon which most ML theory is based (often, implicitly). Hence, most papers that are even remotely formal will state their loss functions in terms of expectations or integrals, not sums (since one cannot usually sum over a continuous space in a meaningful way).

In practice, we cannot access $\mathcal{P}$ (indeed, the whole point of GANs is to implicitly model it!). Instead, we have only a dataset $X\subset\mathcal{X}$ sampled as $X\sim \mathcal{P}$. For some loss function $\mathcal{L}$, we can Monte Carlo estimate the real expected $\mathcal{L}$ via
$$
\mathbb{E}_{x\sim\mathcal{P}}\left[\mathcal{L}(f(x))\right]
= \int \mathcal{L}(f(x)) p_\mathcal{P}(x)\, dx
\approx \frac{1}{|X|}\sum_{x\in X} \mathcal{L}(f(x))
$$
but even this sum is too hard (expensive) to compute, so we take a subsample (minibatch) $M\sim\mathcal{U}(X)$ uniformly from $X$ instead:
$$
\mathbb{E}_{x\sim\mathcal{P}}\left[\mathcal{L}(f(x))\right]
\approx \frac{1}{|M|}\sum_{x\in M} \mathcal{L}(f(x))
$$

So for

  but why is there expectation instead summation in the first place? Is it same to minimize expectation or to minimize function?

The answer is that we cannot compute the expectation, but we can approximate it with the sum or mean over a minibatch instead, so we use that. Ultimately we define our losses via expectations, and derive the sums to compute from those.

Now for the GAN specific stuff.

  What is the meaning of $V(D,G)$?

The point of GANs is learn two functions: $G$, which outputs $x=G(z)$ for $z\sim Q$, and $D$, which outputs a high value for real data ($x\sim P_R$) and low value for fake data ($x\sim P_G$, which we sample via $z\sim Q$ then $x=G(z)$).

What is a good loss function for $D$? Well, we want high values for real data, so let's try to maximize $\log D(x)$ where $x\sim P_R$ is any real data point. So we want the maximum expected value over all of the true dataset,
i.e. $\mathbb{E}_{x\sim P_R}\log D(x)$

Next we want low values for fake data, so let's try to maximize the expected $\log(1-D(x_g))$ over all generated fake data $x_g\sim P_G$. But we can equivalently say $x_g=G(z)$, for some $z\sim Q$. So we can instead maximize $\mathbb{E}_{x_g\sim P_G}[\log(1-D(x_g))]=\mathbb{E}_{z\sim Q}[\log(1-D(G(z)))]$.

Putting the two together, we see that we want $D$ to maximize both these terms, so we'd like
$$ \max_D \mathbb{E}_{x\sim P_R}[\log D(x)] + \mathbb{E}_{z\sim Q}[\log(1-D(G(z)))] =: \max_D V(G,D)$$
to get a good $D$.

What about $G$? Well, we want $G$ to make $D$ do poorly. But which $D$? The best one. In other words, we want $G$ to make the best $D$ (for that $G$) score as poorly as possible, i.e.,
$$
\min_G \max_D V(G,D)
$$
is our objective. That's where $V(G,D)$ comes from.

Now, how to get the sums from the expectations in $V(G,D)$? It's just a Monte Carlo sample approximation (i.e., a sum or mean over a minibatch) of the expectation, as before. :)


What is the meaning of "The number of units in the LSTM cell"? From Tensorflow code: Tensorflow. RnnCell.

num_units: int, The number of units in the LSTM cell.

I can't understand what this means. What are the units of LSTM cell? Input, Output and Forget gates? Does this mean "the number of units in the recurrent projection layer for Deep LSTM"? Then why is it called "number of units in the LSTM cell"?   What is an LSTM cell and how is it different from an LSTM block, what is the minimal LSTM unit if not a cell?

Most LSTM/RNN diagrams just show the hidden cells but never the units of those cells. Hence, the confusion.
Each hidden layer has hidden cells, as many as the number of time steps.
And further, each hidden cell is made up of multiple hidden units, like in the diagram below. Therefore, the dimensionality of a hidden layer matrix in RNN is (number of time steps, number of hidden units).


GPU memory requirement calculation from network architecture I would like to get a rough idea of the GPU Ram requirement for a network based on the number of layers and the number of nodes at each layer.  Is it
number of nodes * 2 * datatype size (64 for double) + number of connections * 2 * datatype size

?
(Using PyTorch)

There is no good way of estimating GPU RAM requirement based the number of layers and the number of nodes at each layer. The operations at each layer and connections between layers matter. For example, a multilayer perceptron (MLP) will use less compute than a Convolutional Neural Network (CNN).
A better idea would be monitor GPU usage during training. If the GPU is a NVIDIA, NVIDIA System Management Interface (nvidia-smi) is built for monitoring and management.