Rate of convergence - comparison of supervised ML methods I am working on a project with sparse labelled datasets, and am looking for references regarding the rate of convergence of different supervised ML techniques with respect to dataset size.
I know that in general boosting algorithms, and other models that can be found in Scikit-learn like SVM's, converge faster than neural networks. However, I cannot find any academic papers that explore, empirically or theoretically, the difference in how much data different methods need before they reach n% accuracy. I only know this from experience and various blog posts.
For this question I am ignoring semi-supervised and weakly supervised methods. I am also ignoring transfer learning.

This is a pretty involved question since this is an active area of research. The first statement is that often, the architecture is important (or number of parameters) before we can say something to the effect of we require $O(n^{k} log(\frac{1}{\delta^i}))$ for $i, k \ge 1$ samples to converge to a local optima. Guaranteeing accuracy is also depending on the data, so it is likely specific to what the data is itself. So you can break your question down into the analysis of stochastic gradient descent and the analysis of it in the context of neural networks. Unfortunately, neither of these deal with specific datasets, so your question about wanting accuracy $\ge n$ is still not possible with such a claim. To the best of my knowledge I am not familiar with claims made along that direction, however, with the former (analysis of SGD/specific neural architectures), there are some claims and papers that I can link below.

On the convergence rate of training recurrent neural networks. While I haven't read this paper -- the results appear to be specific to recurrent neural networks architectures and they give an analysis of a regression type error for certain parameterized forms of RNNs. Their results are also specific to the training error. (https://arxiv.org/abs/1810.12065)
On the rate of convergence of fully connected very deep neural network regression estimates -- This paper deals with a specific loss function not often found in classification ($\ell_2$ norm between the predictions and targets) but might serve as a useful reference. (https://arxiv.org/abs/1908.11133)
Optimization Methods for Large Scale ML -- a pretty comprehensive document on techniques and guarantees for optimization methods, but probably lacking specific results related to neural nets. Also a great reference point for earlier work on the subject. (https://arxiv.org/abs/1606.04838)

Note that this isn't supposed to be a comprehensive list by any means, in fact, this probably just scratches the surface of it.
There are also papers of a different flavour that use Langevin dynamics as a way to analyze the descent trajectory of SGD and provide bounds for it. Here's one https://arxiv.org/abs/1707.06618 but there are obviously several more in the reference of the paper.


Parameters in GridSearchCV in scikit-learn I am trying to build a model in scikit-learn. I used RandomForestClassifier as my method for classification. In order to improve the score and efficiency of my model, I thought about using GridSearchCV.

Here is the code:

import pandas as pd
import numpy as np
from sklearn.cross_validation import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score,roc_auc_score
from sklearn.grid_search import GridSearchCV

..................................... ## code for cleaning data

X_train, X_test, y_train, y_test = train_test_split(train,output, test_size=0.2, random_state =7)

# In[18]:

clf = RandomForestClassifier(n_estimators =100)
param_grid = {'max_depth' : [None, 10,20],
              'max_features' : ['auto',None],
              'n_estimators' :[100,200,300],
              'random_state': 7}
## This line is throwing the error shown below
validator = GridSearchCV(clf, param_grid= param_grid)
vaildiator.fit(X_train,y_train)

The error being thrown by my code is:

ValueError     Traceback (most recent call

last)
<ipython-input-22-3711af477b0c> in <module>()
      3          "max_depth" : [5,10,50],
      4          "random_state" : 7}
----> 5 grid = GridSearchCV(clf, param_grid=param, n_jobs=1)
      6 grid.fit(X_train,y_train)

C:\Anaconda3\envs\DeepLearning\lib\site-packages\sklearn\grid_search.py in __init__(self, estimator, param_grid, scoring, fit_params, n_jobs, iid, refit, cv, verbose, pre_dispatch, error_score)
    785             refit, cv, verbose, pre_dispatch, error_score)
    786         self.param_grid = param_grid
--> 787         _check_param_grid(param_grid)
    788
    789     def fit(self, X, y=None):

C:\Anaconda3\envs\DeepLearning\lib\site-packages\sklearn\grid_search.py in _check_param_grid(param_grid)
    326             check = [isinstance(v, k) for k in (list, tuple, np.ndarray)]
    327             if True not in check:
--> 328                 raise ValueError("Parameter values should be a list.")
    329
    330             if len(v) == 0:

ValueError: Parameter values should be a list.

Please help me in figuring out the above error and why is this happening?

Well the error message is quite clear. GridSearchCV accepts only lists. Therefore 'random_state': [7]} will solve the issue.

However when you have only one value with this parameter, it makes more sense put it directly into the classifier as you did with n_estimators.


How can I use multiple features in basic sentiment analysis in scikit-learn? I've tried to reduce the problem to it's absolute basics. Assume I have data (csv) as such:
label,text-column,gender-column,day-column
1,"Sample positive text", female, 1
0,"Sample negative text", female, 3
1,"Another positive comment", male, 2
0,"Angry text sample", male, 7

And I have this code that trains on label by using BoW (in this case tf-idf) on the text-column. I do a 70/30 train test split and all works well.
vec = TfidfVectorizer()
clf = MultinomialNB()
training_data = pd.read_csv('trainset.csv', delimiter=',')
text_tfidf = vec.fit_transform(training_data['text-column'])
# gen_tfidf = vec.fit_transform(training_data['gender-column'])
X_train, X_test, y_train, y_test = train_test_split(text_tfidf, training_data['label'], test_size = 0.3)
clf.fit(X_train, y_train)

However, for the life of me I simply cannot figure out how to use more than one feature. E.g. I want to use, say, both text-column and gender-column to train the model and see how that impacts accuracy, but I don't understand how to do that!
Am I missing something conceptually important here? Thank you.

You can one-hot encode the gender column and append it to your tfidf table.
gender = pd.get_dummies(training_data['gender-column'])
X = text_tfidf.join(gender)


How to apply Geomertic mean in python I am trying to built a function to calculate the Geometric mean for Growth column and Avg_growth.

Year
Company Name
FH_PBIDT
Growth
Avg_growth

2009-10
Aayush Food & Herbs Ltd.
0.044881
0.000000
-0.773403

2010-11
Aayush Food & Herbs Ltd.
0.063545
0.415839
-0.773403

For that, I am using the below code:

from scipy.stats.mstats import gmean

I created a column name:

df['Geometric Mean'] = gmean(df.iloc[:,3])
df['Avg_Growth Mean'] = gmean(df.iloc[:,4])

I am getting an error that gmean is not defined. Could someone help me here?

Can also write a geometric mean function by a little help  from Numpy.
import numpy as np

def geometric_mean(i):
    x = np.array(i)
    return x.prod()**(1.0/len(x))


Initialisation methods and variance I'm trying to understand some weights initialisation methods by reading the article http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf .

But I don't understand their notation on variance. Right in equation (5) they refer to a variable $z^i$ and I don't know what they mean by it: if it's a collective index for all examples or not?

Welcome to the community.

Definition: Let's define the feature/examples (examples are also called data points). Usually in tabular dataframes when folks doing data analytics, data science etc., features are in columns, and data points of each particular feature are populating the the rows. Here is a simple dataset snapshot showing this definition:

Variance: We are interested in measuring the statistics for each feature. Because in Machine Learning paradigm, those features that have very narrow variance often are not that informative for predicting the target. In this definition it would be along the rows. For example, income feature, we would like to know what is the mean of income, or variance of income for all data points (along rows).

Please NOTE this is just a convention that is very common among Python users and many standard libraries/tutorials out there. As long as you know what you are doing, you can swap these axes. I have seen that some old teaching materials having an opposite definition, meaning that features are in rows, and data points are spread in columns like your definition. If you follow this convention, obviously you have to calculate the variance along column axis, not rows anymore.


how to provide alternate for a word in corpus for word2vec model I need to specify semantically close words in a corpus. For instance newspaper/magazine in the following:

I read [newspaper/magazine] yesterday.

In this example both words are close in meaning. I need to find these kind of words.

How can I find those words?

Add a regularization term for synonymity to the objective function (to be maximized):

$\log P(w_i|h) \rightarrow \log P(w_i|h) + \lambda \frac{1}{|S|} \sum_{(i,j) \in S} sim(w_i, w_j)$ where $S$ is the set of synonyms, and $sim$ is your word similarity function (e.g., cosine similarity).


Understanding Contrastive Divergence I’m trying to understand, and eventually build a Restricted Boltzmann Machine. I understand that the update rule - that is the algorithm used to change the weights - is something called “contrastive divergence”. I looked this up on Wikipedia and found these steps:

Take a training sample v, compute the probabilities of the hidden units and sample a hidden activation vector h from this probability distribution.
Compute the outer product of v and h and call this the positive gradient.
From h, sample a reconstruction v' of the visible units, then resample the hidden activations h' from this. (Gibbs sampling step)
Compute the outer product of v' and h' and call this the negative gradient.
...

I don’t understand step 3 and I’m struggling to grasp the concept of Gibbs sampling. Would someone explain this simply to me? I have covered neural networks if that helps you.

Gibbs sampling is an example for the more general Markov chain Monte Carlo methods to sample from distribution in a high-dimensional space.

To explain this, I will first have to introduce the term state space. Recall that a Boltzmann machine is built out of binary units, i.e. every unit can be in one of two states - say 0 and 1. The overall state of the network is then specified by the state for every unit, i.e. the states of the network can be described as points in the space $\{0,1\}^N$, where N is the number of units in the network. This point is called the state space.

Now, on that state space, we can define a probability distribution. The details are not so important, but what you essentially do is that you define energy for every state and turn that into a probability distribution using a Boltzmann distribution. Thus there will be states that are likely and other states that are less likely.

A Gibbs sampler is now a procedure to produce a sample, i.e. a sequence $X_n$ of states such that, roughly speaking, the distribution of these states across the state space reflects the probability distribution. Thus you want most of the $X_n$ to be in regions of the state space with high probability (and low energy), and few of them to be in regions with low probability (and high energy).

To do this, a naive Gibbs sampling approach would proceed as follows. You start with some state $X_0$. To find the state $X_1$, you would pick some unit and calculate the conditional probability for that unit to be in state 1 ("on") conditional on the current value of all other units. Call this number p. You would then set the unit to 1 with probability p and pick the next unit to repeat this to get from $X_1$ to $X_2$ and so forth.

In the special case of a restricted Boltzmann machine, this can be greatly simplified. Instead of going through, say, first all hidden units and then all visible units and update them like this one by one, you can, in fact, update all hidden units in one step and all visible units in one step, because any two hidden units and any two visible units are independent. Thus, for a full cycle through all units of the state space, you would:

calculate the probability for all hidden units to be 1, given the value of the visible units
set the hidden units to 1 with this probability
calculate the probability for the visible units to be 1, again conditional on the value of the hidden units, and
set the visible units to 1 with this probability

This constitutes one full Gibbs sampling step and is your step 1 + the first part of 3 (the second part is then needed for the further calculation and not part of the sampling). The reason why we do this in the CD algorithm is that we actually want to approximate an expectation value and use a sampler for this.

This is a complex topic and hard to summarize in a few sentences. If you want to learn more about the mathematics behind this (Markov chains) and on the application to RBMs (contrastive divergence and persistent contrastive divergence), you might find this and this document helpful - these are some notes that I put together while learning about this.


How to use the $\chi^2$ test to select the features, that can be String or categorical? I want statistics to select the characteristics that have the greatest relationship to the output variable.

Thanks to this article, I learned that the scikit-learn library proposes the SelectKBest class that can be used with a set of different statistical tests to select a specific number of characteristics.

Here is my dataframe:

    Do you agree    Gender  Age     City     Urban/Rural  Output
0   Yes             Female  25-34   Madrid   Urban        Will buy
1   No              Male    18-25   Valencia Rural        Won't
2   ...             ...     ...     ...      ...          Undecided
....

The output is 'Will buy', 'won't' and 'undecided'.

I then tried the chi-square statistical test for non-negative characteristics to select 10 of the best characteristics:

import pandas as pd
import numpy as np
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2

data = pd.read_csv("D://Blogs//train.csv")
X = data.iloc[:,0:20]  #independent columns
y = data.iloc[:,-1]    #target column i.e price range#apply SelectKBest class to extract top 10 best features
bestfeatures = SelectKBest(score_func=chi2, k=10)
fit = bestfeatures.fit(X,y)
dfscores = pd.DataFrame(fit.scores_)
dfcolumns = pd.DataFrame(X.columns)
#concat two dataframes for better visualization
featureScores = pd.concat([dfcolumns,dfscores],axis=1)
featureScores.columns = ['Specs','Score']  #naming the dataframe columns
print(featureScores.nlargest(10,'Score'))  #print 10 best features

But certain columns are 'String'. So, I get the terminal back:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-59-e64d61febefd> in <module>
      1 bestfeatures = SelectKBest(score_func=chi2,k=10)
----> 2 fit = bestfeatures.fit(X,y)
      3 dfscores = pd.Dataframes(X.columns)
      4 #concat two dataframes for better visualization
      5 featuresScores = pd.concat([dfcolumns,dfscores], axis = 1)

C:\ProgramData\Anaconda3\lib\site-packages\sklearn\feature_selection\univariate_selection.py in fit(self, X, y)
    339         self : object
    340         """
--> 341         X, y = check_X_y(X, y, ['csr', 'csc'], multi_output=True)
    342
    343         if not callable(self.score_func):

C:\ProgramData\Anaconda3\lib\site-packages\sklearn\utils\validation.py in check_X_y(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, warn_on_dtype, estimator)
    754                     ensure_min_features=ensure_min_features,
    755                     warn_on_dtype=warn_on_dtype,
--> 756                     estimator=estimator)
    757     if multi_output:
    758         y = check_array(y, 'csr', force_all_finite=True, ensure_2d=False,

C:\ProgramData\Anaconda3\lib\site-packages\sklearn\utils\validation.py in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, warn_on_dtype, estimator)
    565         # make sure we actually converted to numeric:
    566         if dtype_numeric and array.dtype.kind == "O":
--> 567             array = array.astype(np.float64)
    568         if not allow_nd and array.ndim >= 3:
    569             raise ValueError("Found array with dim %d. %s expected <= 2."

ValueError: could not convert string to float: 'Yes'

You can only compute chi2 between two numerical arrays. You are getting that error because  you are comparing a string. Also I am not sure if it works for multiclassification also.

df = df.apply(LabelEncoder().fit_transform)

This will solve the problem for you. But there are a thousand ways to encode features and for sure other will work better for you.


Deciding the number of clusters in K-means clustering of descriptors I am new to the Machine Learning area and I have a question to ask. But let me first post the problem.

Problem:
The problem is very simple. I want to classify images as Category-1("Images containing garbage") or Category-2("Images not containing garbage"). Garbage is used in every literal sense of the word.

Solution I opted for:
The solution also happens to be pretty straightforward for the most part. Extract points of interest using an Algorithm like SIFT,SURF etc etc. Then obtain the descriptors of these key points and cluster them using the K-means algorithm. Then using this clustered data generate the bag of words and proceed from there.

The thing which I am unable to comprehend is the number of clusters which I might need. Any help with the above example would be much appreciated.

Measure, don't guess

It appears to be common to try k=1024, 2048, 4096, ... and use what works best for classification.

This is possivle because the clustering is actually a quantization task, and you do classification afterwards.


In industry, what type of new data science algorithms does one develop? I've seen several job descriptions for data science which include developing a novel algorithm to be a part of production environments.
Can you give some input of what could be meant here exactly? Would they mean an algorithm that behaves somewhat of an ETL: getting the data, cleaning it, storing it, and running a known model on it? Or something more complex like building variations of known prediction algorithms? Some examples will really be nice, as I'm studying to go into this field.

I am no data scientist, only an aspiring one for two years, moving from my background in software engineering and mathematics. So I took some courses, had some interviews, read a lot on the subject online. My opinion:

New algorithms are developed in research centers and at universities. And even then, most algorithms used in companies are already developed, and just optimized even more.
Don't get your hopes up or be afraid that you have to reinvent gradient descent backpropagation.

With developing algorithms they most likely mean data extraction, data cleaning, data preparation for reporting statistics and presenting graphs. Maybe programmatically, maybe just using tooling.

The presented data may give more insights in simple relations in the domain, and maybe insights in more complex questions that can be asked.

You may get to define data flows, get to compare and select machine learning algorithms and tune its parameters. And continuously evaluate model performance in practice.


Model accuracy when training on GPU and then inferencing on CPU When we are concerned about speed, GPU is way better than CPU.
But if I train a model on a GPU and then deploy the same trained model (no quantization techniques used) on a CPU, will this affect the accuracy of my model? Can the accuracy of the same model degrade on a CPU?
My intuition says, GPU vs CPU should not make any difference if accuracy is concerned.
But I have one doubt that, the GPUs and CPUs have their own different ways to process the information internally. Both of them have different architecture. When a model is trained on a GPU, does the exact same way of processing happens when trained on a CPU but in much slower way? I am not concerned about the accuracy while training, but if a model was trained on a GPU, will it perform exactly with same accuracy on a CPU?

The results you get will be identical* for identical inputs. So for any practical purposes, the accuracy you get will not depend on whether you use a cpu or gpu.
*up to floating point precision. From this post on the pytorch forum:

Both are implementing the floating point number computation standard.
So they are both correct (even though [they may be] different)


Next sentence prediction in RoBERTa I'm trying to wrap my head around the way next sentence prediction works in RoBERTa. Based on their paper, in section 4.2, I understand that in the original BERT they used a pair of text segments which may contain multiple sentences and the task is to predict whether the second segment is the direct successor of the first one. RoBERTa's authors proceed to examine 3 more types of predictions - the first one is basically the same as BERT, only using two sentences insted of two segments, and you still predict whether the second sentence is the direct successor of the first one. But I can't understand what the goal is in the other 2. I will cite their explanation below:
• FULL-SENTENCES: Each input is packed with full sentences sampled contiguously from one or more documents, such that the total length is at most 512 tokens. Inputs may cross document boundaries. When we reach the end of one document, we begin sampling sentences from the next document and add an extra separator token between documents. We remove the NSP loss.
• DOC-SENTENCES: Inputs are constructed similarly to FULL-SENTENCES, except that they
may not cross document boundaries. Inputs sampled near the end of a document may be
shorter than 512 tokens, so we dynamically increase the batch size in these cases to achieve a similar number of total tokens as FULL-SENTENCES. We remove the NSP loss.
So from what I understand in these two training strategies they already sample consecutive sentences, or at least consecutive sentences from neighbouring documents,  and I can't see what they are trying to predict - it can't be whether they're consecutive text blocks, because to me it seems that all of their training examples have already been sampled contiguously, thus making such a task redundant. It would be of enormous help if someone were to shed some light on the issue, thanks in advance!

Similarly to BERT, they sample negative (i.e., non-adjacent) examples and train a classifier telling whether the sentence are consecutive or not.


Reward function to avoid illegal actions, minimize legal action and learn to win - Reinforcement Learning I'm currently implementing PPO for a game with the following characteristics:

Observation space: 9x9x(>150)
Action space: 144
In a given state, only a handful of actions (~1-10) are legal
The state in time-step t can vary a lot from state t+1
The environment is episodic (~25 steps, depending on level) and ends with a win or a loose
In some levels, a random policy (if only legal actions are made) can result in a win, in some levels strategy is needed.

I want the algorithm to learn:

Avoid illegal actions as much as possible, preferably only legal actions
Learn to play the best combination of legal action to win as fast as possible

I've tried different reward functions and the best one so far is:

Illegal action: -0.10
Legal action: +0.10
Win: +1.0

I tested it on a simple level that a random policy will beat. During the time I trained, my policy learned to make around 80% legal actions, but it never won. It will probably go up even more if I run for longer and maybe start winning.

The problem with the above reward function is that it doesn't encourage to win with as few actions as possible, as each legal action get a positive reward.

I've also tried the following reward function:

Illegal action: -0.10
Legal action: -0.02
Win: +1.0

But it converges to around 20% legal moves.

I have two questions:

Anyone familiar with a similar problem that have an idea of how I might design the reward function?
Does anyone know of any papers where they discuss the problem of learning three different objectives: To avoid illegal actions, to win and to win by doing the minimum number of actions?

Do you want your agent to perform illegal actions at any given time or not at all? One way to avoid illegal actions is to use a mask over your 144 action vector (so the action indices are preserved) when you calculate the softmax probabilities. Then you sample actions according to the masked probabilities. First you need to determine which actions are illegal at a given step which should depend on the dynamics of the game. One example are the algorithms used to learn to play the mini-missions in StarCraft II.

In order to enforce your agent to act with the minimum amount of steps you should introduce a small penalty (lets say for example -0.01) so your agent will try to optimize also this part of the reward.

For winning the game question, there is no answer to that. This depends on many factors, I name few here: how appropriate is the agent's architecture given the game, reward sparsity, types of observations (images, non-vector data), data p reprocessing, inputs to your agent, amount of exploration etc. There is no guarantee that if you use the X type of method will solve your task (except if your task has already been solved with a specific type of learner). You could possibly name the game/task you want to solve.


Is it bad to have a large gap between training loss and validation loss? Say my training loss is 0.5 and my validation loss is 2.5 (both have stopped decreasing, validation loss never increased). I am clearly overfitting. If I add regularization, my training loss becomes 1 and validation loss 3.5.

The first model clearly has better validation loss, and the second model overfits less.

Which model should be selected? Is it even possible that a model that overfits more performs better on unseen data or does this mean that there is data leakage of some sort?

The problem here is that I'm not doing simple things like image classification but trying something more complicated and I'm not finding a lot of resources on overfitting on problems that have not yet been solved. Mostly it's disussions of binary image classification where it's possible to find models that get a very good validation error, so the explanations don't apply to my problems.

A model is said to be overfitting if there exist another model or hypothesis that has a higher error on the training set but smaller error overall. A model that overfits fails to generalize on out of sample examples.

In your case, having a much higher error loss on test set could not be a good sign. If you were to chose between the two models I think you should go for the second one, because the first one will probably fail to generalize.

Make sure your training and testing data are picked randomly and represent as accurately as possible the same distribution and the real distribution.

You could do more things other than regularization if want to reduce overfitting, you could try cross validation, or may be reducing the complexity of your model if possible (it depends on the model you training and the data you have)


Understanding the generality of the NER problem Named-entity recognition (NER) is a well-known problem in the NLP literature.
It typically addresses the problem to locate and classify named entities in text, e.g. Organizations and Products.

I'm trying to solve a similar problem but, in my view, a bit more general. Given an input text, I want to be able to comprehensively annotate the whole text; not only specific entities like Actors and Organizations but also higher-level concepts like Conditions of Applicability and Temporal Conditions, e.g.:

The added difficulty is that we have nested "entities", e.g. (from above):
<denotic> must, <temporal> within the specified period </temporal>, notify ... </deontic>

Can this still be formulated as a NER problem? If so, what would be the best type of model to solve this task assuming a dataset of ~ 50 K examples?

The problem described is not a more general version of Named Entity Recognition, it is a different problem called parsing. Parsing consists in extracting the syntactic structure of a text, usually in the goal to better capture its semantics. There are various approaches:

Shallow parsing only identifies the constituents of the sentences (based on your example this could be sufficient in your case)
Statistical parsing and in particular Dependency parsing represent the full structure of the sentence, including the links between its constituents.

There are various libraries and datasets for parsing: one of the most famous is probably the Stanford parser, but there are many others often included in NLP toolkits such as OpenNLP. The Universal Dependencies project is a vast multilingual collection of annotated text which can be used to train parsers.
Semantic Role Labeling (SRL) is a closely related task which consists in identifying the semantic relations between a predicate (verb) and its related constituents (e.g. subject, object).


How to numerically estimate MLE estimators in python when gradients are very small far from the optimal solution? I am exploring how to model a data set using normal distributions with both mean and variance defined as linear functions of independent variables.

Something like N ~ (f(x), g(x)).

I generate a random sample like this:

def draw(x):
    return norm(5 * x + 2, 3 *x + 4).rvs(1)[0]

So I want to retrieve 5, 2 and 4 as the parameters for my distribution.

I generate my sample:

smp = np.zeros((100,2))

for i in range(0, len(smp)):
    smp[i][0] = i
    smp[i][1] = draw(i)

The likelihood function is:

def lh(p):
    p_loc_b0 = p[0]
    p_loc_b1 = p[1]
    p_scl_b0 = p[2]
    p_scl_b1 = p[3]

    l = 1
    for i in range(0, len(smp)):
        x = smp[i][0]
        y = smp[i][1]
        l = l * norm(p_loc_b0 + p_loc_b1 * x, p_scl_b0 + p_scl_b1 * x).pdf(y)

    return -l

So the parameters for the linear functions used in the model are given in the p 4-variable vector.

Using scipy.optimize, I can solve for the MLE parameters using an extremely low xtol, and already giving the solution as the starting point:

fmin(lh, x0=[2,5,3,4], xtol=1e-35)

Which does not work to well:

Warning: Maximum number of function evaluations has been exceeded.
array([ 3.27491346,  4.69237042,  5.70317719,  3.30395462])

Raising the xtol to higher values does no good.

So i try using a starting solution far from the real solution:

>>> fmin(lh, x0=[1,1,1,1], xtol=1e-8)
Optimization terminated successfully.
         Current function value: -0.000000
         Iterations: 24
         Function evaluations: 143
array([ 1.,  1.,  1.,  1.])

Which makes me think:

PDF are largely clustered around the mean, and have very low gradients only a few standard deviations away from the mean, which must be not too good for numerical methods.

So how does one go about doing these kind of numerical estimation in functions where gradient is very near to zero away from the solution?

There are several reasons why you are getting erroneous results.
First, you should consider using log likelihood instead of likelihood. There are numerical issues with multiplying many small numbers(imagine if you had millions of samples you had to multiply millions of small numbers for the lhd). Also taking gradients for optimization methods that require gradients is often easier when you are dealing with the log likelihood. In general, it is good to have an objective which is a sum rather than a product of variables when dealing with optimization problems.
Second, fmin is using Nelder-Mead simplex algorithm which has no convergence guarantees according to scipy documentation. This means the convergence is totally random and you should not expect to find parameters close to the originals. To get around this, I would suggest you to use a gradient based method like stochastic gradient descent or BFGS. Since you know the generative model (rvs are Gaussian distributed) you can write the likelihood and log likelihood as:

Where a,b,c and d are your model parameters 5,2,3 and 4 respectively.
Then take the gradient with respect to [a,b,c,d] and feed that into prime input of fmin_bfgs. Note that due to varying variance what could be solved by just linear regression is now a nastier problem.
Finally, you may also want to check Generalized least squares here and here, which talk about your problem and offer several available solutions.
Good luck!


Should bag of words in training set include test set data when doing text classification? I'm doing text classification to identify 'attacks' from wikipedia comments using a simple bag of words model and a linear SVM classifier. Because of class imbalance I'm using F1 score as my error measure. I'm wondering if the tokens I have in the training data should also include words that exist only in the test data, or does it not matter? I was under the impression that it shouldn't matter, since the counts for these features would be zero anyway in the training set. That should make them irrelevant to the model when training. Apparently that's what some people on SO were saying as well (it didn't find any definitive answer though).
In order to test this I decided to train my model both ways and see the difference: once with only features in the training data, and another with features that included test data. N folds for CV was set to 10. I got a very similar CV error for both of them, but when I generated predictions for my test data my F1 score was 0.06 higher for the model which included features from test data - 0.64 vs 0.58. Because this is a Kaggle assignment I cannot see the true labels for the test set. I'm inclined to believe that such a big difference can't simply be random. It seems like including all the features from test data did improve my model, but how could this be? Can anyone give me an explanation?

This is the problem of out of vocabulary (OOV) words.
As a rule the training should not use anything from the test set for several reasons:

The risk of data leakage, which would cause an overestimated performance on the test set.
During training the model cannot use these words to distinguish between classes anyway, since they are not present. So it would be pointless to include them.
In principle the model is meant to be used with any new input text, not only the documents in the current test set. Using the test set words as features in the model actually restricts the model, since it cannot be applied to any other text than the test set.

The correct way to deal with OOV words is either:

To simply ignore them completely, i.e. filter them out before applying the model.
To account for this possibility in the model from the start. Typically this can be done by having a special token UNKNOWN. This option is often combined with filtering out rare words from the training set: every occurrence of a rare words can be replaced by the UNKNOWN token.

The cause of the higher performance when including these features is unclear. I suspect that the inclusion of these features causes the model to be different in a way which happens by chance to have a positive effect on the test set. It's quite unlikely in general, I'm not sure.


Efficient clustering of sparse binary vectors I'm trying to perform clustering of the data to improve the efficiency of brute-force kNN . The dataset consists of objects described by a large set of binary features, each identified by a 32-bit hash code. Data point can be understood as a 2^32 elements long very sparse binary vector, with bits set to 1 on the positions denoted by hash code of feature. To simplify, each of the data points is being represented as an array of hashes - if we know which of the bits are set to 1, we know which of them are set to 0.

I have a working distance function (mentioned in here) but have difficulties to cluster the dataset in reasonable time.  Because of the binary nature of the data it is impossible to create any kind of mean value based on a collection of datapoints, so k-Centroids is not an option. I tried k-Clustroids but it doesn't converge, hierarchical approaches are too inefficient. Do you happen to know any clustering algorithm that would efficiently handle fixed size dataset, with custom metric calculation method without the need to create any temporary, centroid datapoints?

Many thanks.

You can efficiently implement e.g. Euclidean distance or Cosine on sparse data (iterate over nonzero values only!). Then you can use e.g. Hierarchical clustering.

But also consider frequent itemset mining. Usually, on sparse binary data, frequent itemsets are better than clusters.


Interpreting high precision and very low recall score I was training model on a very imbalanced dataset with 80:20 ratio of two classes. The dataset has thousands of rows and I trained the model using

DeccisionTreeClassifier(class_weight='balanced')

The precision and recall I get on the test set were very strange

Test set precision : 0.987767
Test set recall : 0.01432

I'm unable to interpret the results. What does this tell me about my classifier?

All problems due to "a very imbalanced dataset". Please look at the definition of recall and precision. Based on your score I could say that you a very small set of values labeled as positive, which are classified correctly($precision=\frac{TP}{TP+FP}$). But you have a very big set of values labeled as negative, which have influence on $recall=\frac{TP}{TP+FN}$, in that way that $TP$ stays the same like in precision, but have you a lot of $FN$ values which leads to small value of recall.

At least you should resample of the dataset to make better results.


Using a loop to predict longer vectors with a LSTM I have a model based on LSTMs that can predict a vector output based on a vector input. I can't increase the size of the output because :

I would need a larger network to obtain good results
It would take more time to train
The behaviour of my timeseries can be captured in the size of the vector I'm already predicting

So I tried to predict an output, and then use that prediction as a new input in my model and predict a new vector (without re-training). I iterate 5 times and I get this result :

As you can see, the first prediction is pretty good. Then, my model is lost and loses its accuracy. Do you know how to fix it ?

Thanks

I'm going to answer to myself since I managed to fix that issue. The problem is that my model was overfitting waaaay too much and I had to add a dropout between each layer.

Now the result is better :

I will now re-tune my model and improve it a bit.

note to myself : don't be lazy Julien, plot the validation loss next time

EDIT :

I trained it during 36 mn with more neurons per layer. Without overfitting, it's way way better.

Now I can predict 8 days of datapoints with a better accuracy.


Keras Conv1D for simple data target prediction I am trying to use conv1D layer from Keras for predicting Species in iris dataset (which has 4 numeric features and one categorical target). Following is my code:

import numpy as np
import pandas as pd

irisdf = pd.read_csv('iris.csv')

Xall = irisdf.drop('Species', axis=1)
print(Xall.shape)
Xall = np.expand_dims(Xall.values, axis=2)
print(Xall.shape)

Yall = irisdf['Species']
nb_classes =  3

import keras
from keras.models import Sequential
from keras.layers import Dense, InputLayer, Dropout, Flatten, BatchNormalization, Conv1D
input_shape = (Xall.shape[1:],)
print(input_shape)
model = Sequential([
    InputLayer(input_shape=input_shape),
    Conv1D(32, 2),
    Dense(nb_classes, activation='softmax')
])
model.compile(loss=keras.losses.mean_squared_error,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])
model.summary()
model.fit(Xall, Yall, epochs=25, verbose=True)

However, it is giving following error:

Traceback (most recent call last):
  File "/home/abcde/.local/lib/python3.5/site-packages/tensorflow/python/eager/execute.py", line 141, in make_shape
    shape = tensor_shape.as_shape(v)
  File "/home/abcde/.local/lib/python3.5/site-packages/tensorflow/python/framework/tensor_shape.py", line 946, in as_shape
    return TensorShape(shape)
  File "/home/abcde/.local/lib/python3.5/site-packages/tensorflow/python/framework/tensor_shape.py", line 541, in __init__
    self._dims = [as_dimension(d) for d in dims_iter]
  File "/home/abcde/.local/lib/python3.5/site-packages/tensorflow/python/framework/tensor_shape.py", line 541, in <listcomp>
    self._dims = [as_dimension(d) for d in dims_iter]
  File "/home/abcde/.local/lib/python3.5/site-packages/tensorflow/python/framework/tensor_shape.py", line 482, in as_dimension
    return Dimension(value)
  File "/home/abcde/.local/lib/python3.5/site-packages/tensorflow/python/framework/tensor_shape.py", line 37, in __init__
    self._value = int(value)
TypeError: int() argument must be a string, a bytes-like object or a number, not 'tuple'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "rnkeras_conv1d_iris.py", line 40, in <module>
    InputLayer(input_shape=input_shape),
  File "/home/abcde/.local/lib/python3.5/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/home/abcde/.local/lib/python3.5/site-packages/keras/engine/input_layer.py", line 86, in __init__
    name=self.name)
  File "/home/abcde/.local/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 515, in placeholder
    x = tf.placeholder(dtype, shape=shape, name=name)
  File "/home/abcde/.local/lib/python3.5/site-packages/tensorflow/python/ops/array_ops.py", line 1735, in placeholder
    return gen_array_ops.placeholder(dtype=dtype, shape=shape, name=name)
  File "/home/abcde/.local/lib/python3.5/site-packages/tensorflow/python/ops/gen_array_ops.py", line 4923, in placeholder
    shape = _execute.make_shape(shape, "shape")
  File "/home/abcde/.local/lib/python3.5/site-packages/tensorflow/python/eager/execute.py", line 143, in make_shape
    raise TypeError("Error converting %s to a TensorShape: %s." % (arg_name, e))
TypeError: Error converting shape to a TensorShape: int() argument must be a string, a bytes-like object or a number, not 'tuple'.

Where is the problem and how can it be solved?

(PS: If you find this question to be interesting/important, please upvote it;)

Your error is coming from the Keras framework not working with strings as the output labels. You will want to transform these to 1-hot encoded vectors to train your model. Here is some code to do this.
Getting the data
import pandas as pd
df = pd.read_csv('iris.csv', header=None, names=['sepal_length', 'sepal_width', 'petal_length', 'petal_width', 'species'])

This will assign a class label, we will one-hot encode them later
df['labels'] =df['species'].astype('category').cat.codes

Splitting the data and reshaping the data
First we will split the data into a training and testing set. Then we will one-hot encode the labels. And finally we will structure the inputs to match what is expected from Keras. To use a 1D convolution we need to add a spatial dimension.
from sklearn.model_selection import train_test_split
import keras

X = df[['sepal_length', 'sepal_width', 'petal_length', 'petal_width']]
Y = df['labels']
x_train, x_test, y_train, y_test = train_test_split(np.asarray(X), np.asarray(Y), test_size=0.33, shuffle= True)

# The known number of output classes.
num_classes = 3

# Input image dimensions
input_shape = (4,)

# Convert class vectors to binary class matrices. This uses 1 hot encoding.
y_train_binary = keras.utils.to_categorical(y_train, num_classes)
y_test_binary = keras.utils.to_categorical(y_test, num_classes)

x_train = x_train.reshape(100, 4,1)
x_test = x_test.reshape(50, 4,1)

The model
Your model was insufficient to get good results so I added an additional hidden layer into the mix to get acceptable results.
from __future__ import print_function
from keras.models import Sequential
import keras
from keras.models import Sequential
from keras.layers import Dense, Flatten, Conv1D
from keras.callbacks import ModelCheckpoint
from keras.models import model_from_json
from keras import backend as K

model = Sequential()
model.add(Conv1D(32, (3), input_shape=(1,4), activation='relu'))
model.add(Flatten())
model.add(Dense(64, activation='relu'))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

model.summary()

Now let's train the model
batch_size = 128
epochs = 10
model = model.fit(x_train, y_train_binary,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test, y_test_binary))

100/100 [==============================] - 0s 50us/step - loss: 1.0906 - acc: 0.6400 - val_loss: 1.0893 - val_acc: 0.7000

We get 70% accuracy, that's not so bad. But it can be improved by changing the model to better suit the data source.
Plot the convergence
plt.plot(model.history['loss'])
plt.plot(model.history['val_loss'])
plt.title('model train vs validation loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper right')
plt.show()


Data analytics in road transportation What kind of insights can we give from point A to point B transactions. A and B are two places, All I can think of is about descriptive statistics like how many transactions from A and what is count of transactions between places and all.

You could use the tools from Network Science (1) to model (A,B) as edges in a network. Since you have timing information on the transactions you can reconstruct paths on that network.

Using statistical network analysis you can find for example which points A from your dataset (or nodes in the network) are of importance, as in how many routes lead to that point. For this you'd study a centrality measure (degree centrality, closeness centrality, betweenness centrality, etc.) (2)

Or you can find which points are close together in terms of transactions between them, for this you'd look for community detection or clustering. (2)

You can also look at the timing data, to calculate the intra- and inter- transaction time distributions.

(1): Barabási, A. L. (2016). Network science. Cambridge university press.

(2): Salter‐Townshend, M., White, A., Gollini, I., & Murphy, T. B. (2012). Review of statistical network analysis: models, algorithms, and software. Statistical Analysis and Data Mining: The ASA Data Science Journal, 5(4), 243-264.


Class imbalance problem? While working with 3 classes,my dataset contains different proportions of all classes. For example,

Class "0"  has  11098 cases.....Normal
class  "1" has  2369 cases....."Abnormal"
class  "2" has  3396 cases....."May be Normal or Abnormal"

I trained the model with similar proportions. I have two questions:

Is it necessary to balance all three classes?
What is the impact of class imbalance on classification performance?

As mentioned in the comments, your data doesn't have that much of an imbalance. A good way to check how the imbalance is affecting your model, is by calculating precision and recall. These values will be quite low while your accuracy will shoot high because it will predict the majority class for every sample. However the 'maybe normal' case is dangerous. It would be better to set a different threshold , probably a lower one so that even the marginally abnormal cases will be classified as abnormal. A precision recall curve or a ROC curve is a good way to decide on the cutoff.


In multiple linear regression why is it best to use an $F$-statistic when evaluating predictors? I am currently going through Hastie and Tibshirani's 'Introduction to Statistical Learning' textbook and I have come across something I don't understand on page 77. I have two questions.

The author states that if we had 100 variable predictors (with respective coefficients $\beta_i$) and supposed that the null hypothesis, $$H_0 : \beta_1 = ... = \beta_{100} = 0$$ were true, then roughly $5\%$ of the $p$-values would fall below $0.05$ by chance, and therefore we might wrongly conclude that certain predictors are related to the response. Why would this happen by chance? Is this simply a mathematical truth?

In addition, the author then goes on to state that the $F$-statistic is a better measure because "if $H_0$ were true, then there is only a $5\%$ chance that the $F$-statistic would result in a $p$-value below $0.05$". I don't understand the difference - could somebody explain a bit more clearly?

The example given in your textbook proposes a multiple linear regression with 100 predictors, all of which have a "true" regression coefficient of 0. In other words, your independent variables have no statistical association with your dependent variable.

When you calculate the $p$-value of an individual coefficient, you're looking at the magnitude of the coefficient, the standard error of the coefficient, making some distributional assumptions about it, and asking the following question: "what is the probability of seeing a coefficient value this extreme if the true value is actually zero?"

If our distributional assumptions are correct, any given coefficient with a true value of 0 will report a <0.05 $p$-value approximately 5% of the time. That's not so much of a problem if we only have one predictor, but by the law of large numbers, if we have lots of predictors, we'd expect 5% of them to report a $p$-value this low. This makes the $p$-values in high-dimensional regressions hard to interpret.

The $F$-test is different. Instead of evaluating every single coefficient for statistical significance, it applies a single test to the entire regression. So instead of having 100 chances to throw up an erroneous $p$-value, it only has one chance. This makes the $F$-test useful for evaluating whether or not there is a regression effect for high-dimensional regressions.


The best way to calculate variations between 2 datasets? I am trying to write an application which identifies SQL query patterns. The program will trace all the queries executed in a database. When it identifies a considerable pattern change in the database, it will ask the DBA to tune the database. My tracing data looks like this.

The images show the query execution count from 2 different months. According to the images there is a considerable pattern chnage in query execution. Query2 is executed most of the times in the first dataset and Query3 is executed most of the time in second dataset. I want to mathamatically calcuate this pattern difference. One solution I can think of is calculating the percentage differene of execution count of each query and sum it all. If the value is greater than the predefined thresold I can send a notification to the DBA. Is there a better way to do that? Any data analysis technique that can be applied? Any libraries to do that?

What you are attempting to do is calculate the distance between two discrete probability density functions. The standard way to calculate this distance is via the Kullback-Leibler Divergence. It is defined as such:

$$D(P||Q) = \sum_iP(i)*log(\frac{P(i)}{Q(i)})$$

To be technically correct, this is not a metric in a mathematical sense because it is not symmetric and does not abide by the triangle inequality. But never the less it is a "distance" that is used a lot to compare distributions. From an information theoretical perspective, the KL divergence (given that you're using base 2 for the log) represents the number of bits it would take to encode distribution P as Q.

In your case, you would have to normalize each distribution to ensure that the sum is 1, and then calculate distance via the KL divergence. If you want a real metric you can use the Jensen-Shannon Divergence which is defined as:
$${{\rm {JSD}}}(P\parallel Q)={\frac  {1}{2}}D(P\parallel M)+{\frac  {1}{2}}D(Q\parallel M)$$$$
{\displaystyle M={\frac {1}{2}}(P+Q)} M={\frac  {1}{2}}(P+Q)$$

Hope this helps.


Can one use non converged results from Logistic Regression? I'm running Logistic Regression on a dataset for a classification problem.

I used the model on the dataset when it was normalized and I had no problem with it converging.

Now, I wanted to see the results without feature normalization and I am getting the warning:

ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.
  "of iterations.", ConvergenceWarning)

I've tried increasing the number of iterations and I have upped it to 8000 so far and am still getting the error.

I wanted to ask, is this error is critical? Because, I'm still getting my cross validated results even after the error. What exactly is convergence?

An iterative algorithm is said to converge when, as the iterations proceed, the output gets closer and closer to a specific value. More precisely, no matter how small an error value you choose, if you continue long enough the function will eventually stay closer than that error value from some final value.

In some circumstances, an algorithm will not converge; it could even diverge, where its output will undergo larger and larger oscillations, never approaching a useful result. More precisely, no matter how long you continue, the function value will never settle down within a range of any "final" value.

You could use the results which did not converge, but it's not recommended.


How to club the orders in such a way that maximum number of items are common amongst them? Consider the following data set:
The above table shows the quantity of each item used in the orders SO1 SO2 etc. I need to club the orders in such a way that maximum number of items are common amongst them.
For example:

SO1 SO3 SO5  can be clubbed for 8 items ( Item 1,2,3,5,6,8,9,11) and SO2 and SO4 can clubbed together because 9 items are common ( Item 1,3,4,5,6,8,9,10,11).

The approach I followed was:
 I found out the number of combinations possible as in (SO1 SO2), (SO1 SO3), (SO1 SO4)……(SO1 SO2 SO3), (SO1 SO2 SO4),….(SO1 SO2 SO3 SO4),…(SO1 SO2 SO3 SO4 SO5). For n number of SOs I would be getting around 2^n -n -1 combinations.
Later I compared the data in each combination for equality.

Based on the number of matches, I thought I would be able to select the combinations. But this process would become cumbersome for 300 shop orders with around 2000 items. And it is taking a lot of time to compute as well.

Have you tried using frequent itemset mining?

It finds the most frequent item combinations.

The key idea is that when item A is not frequent, you don't need to further explore any combinations including A.


How much of a disadvantage is a small sample size? I am examining a petition involving all UK constituencies. In this dataset 2 of the 632 constituencies have not participated in the petition - in terms of data quality how does this affect my examination?

I am examing which parts of the UK tend to vote for left/right ideals.

When I read this, I guess you are up to a causal model?

When you use methods like linear regression or logistic regression, $n=630$ is a healthy sample size. So no problem with that. What could be a problem is that you have censored data (you don't know about the remining two constituencies).

IMO the best way to deal with this problem is to cleary state that you have missing data for this constituencies, so that you only can make claims for the remaining ones. It is a common problem that there is missing data. If it is really only about two constituencies and if you look at "left/right" tendence, the missing data is not a big problem as long as the missing observations are not highly important (e.g. super large) and there is no reason to believe that the missing observations come from an entirely different data generating process (e.g. extremely different to all other observations).


How can we use the current rewards as a system input in the RUN time when working with Deep Q learning? When tuning the Deep Q net parameters we use the immediate rewards . Specially in action replay and regressing .

But in the run time we don't care about the rewards at all. Because our neural network will take states as the input . I am explaining this relevant to the deep mind Atari game play paper . Any one thing it is wastage of input ? Is there a mechanism we can use something like Score even in the run time .

p.s - I understand how they update the Neural Net parameters with regression . Sometimes with TD(λ) .

My friend - Thushan Gave me this answer . So I will post it . I think it will describe it . Sometimes in the run time it's not desirable to work with immediate rewards . It can mess up the agent. I will quote it here .

  Your argument makes sense, but not always. And this is a design decision
  you make in your problem formulation. However to highlight why this
  won't work in some situations, I'll give an example.

  Say you're playing a shooting game with a RL agent. And at some
  specific location you get killed by getting shot from an enemy.
  However if you include the reward with the state, this might suggest
  to the algorithm, that this location is always bad and will try to
  avoid it (which is wrong). We should't be avoiding locations, but
  enemies.


Effecient way to decompose multiple time series in a data frame and compare the fit of additive and multiplicative models? I have a data frame in R that contains time series data of 7 variables that were taken on several hundred different individuals. I want to know if it would be more appropriate to use an additive model or a multiplicative model for each variable.

To give an example, the data is structured something like this:

set.seed(123)
ID = factor(letters[seq(15)])
Time = c(1000,1200,1234,980,1300,1020,1180,1908,1303,
        1045,1373,1111,1097,1167,1423)
df <- data.frame(ID = rep(ID, Time), Time = sequence(Time))
df[paste0('Var', c(1:7))] <- rnorm(sum(Time))

What is an effective way to decompose the data for each variable/ID combination, fit each with an additive model and a multiplicative model, and compare the fits?

One way to do this would be to fit the decompositions with the same numbers of degrees of freedom and see which fits the best. It is convenient to do this using the tsibble and feasts packages as they allow for modelling many time series at once.

I've modified your example data so that it is possible to do a multiplicative decomposition -- having negative values in the data makes multiplicative decompositions problematic.

The multiplicative decomposition uses STL on the log data, and then exponentiates the trend and seasonal terms to put them back on the original scale.

Your example has no obvious seasonality so I have arbitrarily set the seasonal period to 12 for illustration purposes. Change it to whatever it should be.

I have set the trend window to be 99 and the seasonal component to be periodic. Again, change these to suit your actual data but the two fits should have the same values.

set.seed(123)
ID = factor(letters[seq(15)])
Time = c(1000,1200,1234,980,1300,1020,1180,1908,1303,
         1045,1373,1111,1097,1167,1423)
df <- data.frame(ID = rep(ID, Time), Time = sequence(Time))
df[paste0('Var', c(1:7))] <- abs(rnorm(sum(Time)))

library(tidyverse)
library(tsibble)
library(feasts)

# Create tsibble in long form
df <- df %>%
  pivot_longer(starts_with("Var"), names_to="Series", values_to="value") %>%
  as_tsibble(index=Time, key=c(ID,Series))
# Additive decompositions
additive <- df %>%
  model(add = STL(value ~ trend(window=99) + season("periodic", period=12))) %>%
  components()
# Multiplicative decompositions
multiplicative <- df %>%
  model(mult = STL(log(value) ~ trend(window=99) + season("periodic", period=12))) %>%
  components() %>%
  mutate(remainder = df$value - exp(trend+season_12))
# Find variance of remainders
rva <- additive %>%
  as_tibble() %>%
  group_by(ID, Series) %>%
  summarise(rv = var(remainder, na.rm=TRUE)) %>%
  ungroup()
rvm <- multiplicative %>%
  as_tibble() %>%
  group_by(ID, Series) %>%
  summarise(rv = var(remainder, na.rm=TRUE)) %>%
  ungroup()
# Which remainder has lowest variance?
left_join(rva, rvm, by = c("ID","Series")) %>%
  mutate(best = if_else(rv.x < rv.y, "additive", "multiplicative"))
#> # A tibble: 105 x 5
#>    ID    Series  rv.x  rv.y best
#>    <fct> <chr>  <dbl> <dbl> <chr>
#>  1 a     Var1   0.357 0.361 additive
#>  2 a     Var2   0.357 0.361 additive
#>  3 a     Var3   0.357 0.361 additive
#>  4 a     Var4   0.357 0.361 additive
#>  5 a     Var5   0.357 0.361 additive
#>  6 a     Var6   0.357 0.361 additive
#>  7 a     Var7   0.357 0.361 additive
#>  8 b     Var1   0.338 0.341 additive
#>  9 b     Var2   0.338 0.341 additive
#> 10 b     Var3   0.338 0.341 additive
#> # … with 95 more rows

Created on 2020-04-22 by the reprex package (v0.3.0)


How to implement clustering on categorical feature vectors in R? I have a dataset like:

 id color  body  eyes
 1  A  blue  slim green
 2  B black   fat  blue
 3  A black  slim black
 4  C green  slim  blue
 5  D black medim black

whereas each id represents an individual with his individual physical characteristics.

Reproducible:

 structure(list(id = structure(c(1L, 2L, 1L, 3L, 4L), .Label = c("A",
"B", "C", "D"), class = "factor"), color = structure(c(2L, 1L,
1L, 3L, 1L), .Label = c("black", "blue", "green"), class = "factor"),
body = structure(c(3L, 1L, 3L, 3L, 2L), .Label = c("fat",
"medim", "slim"), class = "factor"), eyes = structure(c(3L,
2L, 1L, 2L, 1L), .Label = c("black", "blue", "green"), class = "factor")),
 .Names = c("id",
"color", "body", "eyes"), class = "data.frame", row.names = c(NA,
-5L))

Then number of the characteristics is fixed (color: blue/black/green, body: slim/fat/medium, eyes: green/blue/black).

My aim is to cluster those individuals.

My conceptual question regards the approach:

A simple correlation could be a first step. A question could be: how the combination of these characteristic is likely to appear in groups of individuals?

A more complicated approach. Maybe k-means clustering. How can address this given that these are categorical variables? should I convert them into dummies?

I'm new to this kind of analysis and any hint/reference to the implementation in R is highly appreciated!
Thanks

You should use dummy variables and then you can toss it directly into K-means. If you have a lot of categories the efficient way to do this is through a one-hot-encoding (sparse-encoding).

Here's a little demo using clustering and then using the clusters in a regression model. In general, you should avoid doing that but it's illuminating in this case.

library(glmnet)
library(Matrix)
n <- 1e5
nclusters <- 5
set.seed(420)
ls <- data.frame(sample(letters, n, replace=TRUE))
xs <- sparse.model.matrix(~.,data=ls)
print(head(xs))
# Now let's run k-means
out <- kmeans(xs, centers=nclusters)
bs <- rep(1, dim(xs)[2])
# Let's run k-means on the different categories
clusterpred <- data.frame(out[[1]])
ys <- xs %*% bs + rnorm(n)
print(table(clusterpred))
# Now let's use a clustered data set to predict some outcome
cxs <- sparse.model.matrix(~.,data=clusterpred)
model <- glmnet(y=ys, x=xs, alpha=0)
cmodel <- glmnet(y=ys, x=cxs, alpha=0)

# Predictions
yhat <- predict(model, xs)
yhatc <- predict(cmodel, cxs)
# Looking at the difference RMSEs
print(sqrt( sum( (ys-yhat)**2 )))
print(sqrt( sum( (ys-yhatc)**2 )))


I'm planning to record video now and then analyze it in the future once AI programs are more accurate, what resolution and FPS should I record at? I'm going to record videos and want to future proof them for AI analysis as much as possible.
I have a decent amount of storage to work with, 2.7TB. I'm hoping to make it last for at least two years. I would be recording for about 10 - 15 minutes every day.
I'm trying to decide between 30FPS vs 60FP and 1080p vs 4K.
I'm leaning towards 1080p at 60FPS.
I would think recording at a higher resolution and FPS would be better because it would give me more data to work with and I could analyze the smaller changes. Is my thinking correct? I don't know if higher-resolution or higher framerate would be more useful in analysis. Thanks.

This question is definitely opinion-based and so is my answer, but let me try to clarify a few things:

Don't count blindly on AI/ML to become more accurate in general, and even less for a particular project. Imho for your project to be future-proof, you need to test the current technology with a sample of your data (or some similar data) and make sure that the results are at least "encouraging", i.e. decent enough. If the result are terrible or cannot be obtained at all, then there's probably a problem with the whole design and it's unlikely that future technology will solve this.
It's not always by providing more detailed information to ML that it gives better results, it strongly depends on the task. This is why I recommend testing first (see point 1).
Technically it's always possible to downsize resolution but not the opposite, so from this point of view it's clear that the higher the resolution the better. A simple estimation says that with your characteristics a single daily video would require around 2GB, so 2 years would require around 1.5TB.


Convolutional layers without pooling I am studying the CNN architecture of the AlexNet, and I have seen that it has convolutional layers without pooling in between:

but I don' understand why this is done. Wouldn't be better to have something like CONV - POOLING - CONV - POOLING , and so on, instead of CONV - POOLING - CONV - CONV -CONV - POOLING?

Why is this done? Thanks in advance.

The idea behind consecutive convolutional layers with no pooling is actually not to skip any pooling but to replace a single layer with a bigger receptive field. So think of it this way:

Two consecutive 3x3 layers actually lead to a receptive field of 5x5.
Three consecutive 3x3 layers (like in AlexNet) lead to an actual receptive field of 7x7.

Intuitively I like to think of it this way: For two 3x3 layers the actual receptive field gets extended by 1 in all directions as the information is "pulled in" from outside of the 3x3 field by applying two layers consecutively.

In the first layer the 3x3 receptive field is applied (here using A and B as examples):

Now, that means when the second 3x3 layer is being applied cell B contains information from beyond the dark grey area. And this information in B will be considered for A when you apply the second layer. So basically the receptive field of A has grown as it includes information from beyond its actual 3x3 field:

And since this happens in all directions as shown in the second picture you now end of with a sort of 5x5 receptive field. Finally, if you apply this a third time (like AlexNet does with its 3 consecutive 3x3 layers) then the receptive field will be extended a second time by 1 in all directions, i.e. you get a 7x7 receptive field.

Question is why would you actually do this and not just use a single layer with a larger receptive field?

The paper Very Deep Convolutional Networks for Large-Scale Image Recognition
 gives two reasons:

  First, we incorporate three non-linear
  rectification layers instead of a single one, which makes the decision function more discriminative.

  [...]

  Second, we decrease the number of parameters: assuming that both the input and the output of a three-layer 3 × 3 convolution stack has C channels, the stack is parametrised by $3(3^2C^2)=27C^2$ weights; at the same time, a single 7 × 7 conv. layer would require $7^2 C^2=49C^2$ parameters, i.e. 81% more. This can be seen as imposing a regularisation on the 7 × 7 conv. filters, forcing them to have a decomposition through the 3 × 3 filters (with non-linearity injected in between).


Policy Gradient with continuous action space How to apply reinforce/policy-gradient algorithms for continuous action space. I have learnt that one of the advantages of policy gradients is , it is applicable for continuous action space. One way I can think of is discretizing the action space same as the way we do it for dqn. Should we follow the same method for policy -gradient algorithms also ? Or is there any other way this is done?

Thanks

Yes, that is possible. It can be done in the following way:

We assume that the action distribution is guassian, i.e. that we need to learn the parameters $\theta$ of $\mathcal{N}(a|\mu_\theta,\sigma_\theta)$. Let's say that $\theta$ is given by the weights of a neural network, which we find by optimizing the objective
$$\max_\theta \mathbb{E}_{p_{\theta}}\left[ R(s,a)p_\theta(a|s)\right],$$
where $p_\theta(s,a) = \mathcal{N}(a|\mu_\theta, \sigma_\theta)$ and $R(s,a)$ is the cumulative discounted reward. The gradient is then per policy gradient theorem simply $\mathbb{E}_{p_{\theta}}\left[\nabla_\theta R(s,a) \log p_\theta(a|s) \right]$.

In practice we design a neural network to output one $\mu$ per action dimension and $\sigma$ can either be learned or kept fixed. If learned, interpret the output as $\log \sigma$, so it can take en value. To sample the action we use this outputs learned by our network.

See e.g. T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. “Continuous control with deep reinforcement learning,” International Conference on Learning Representations, 2016.


Interpreting MLP output I just wrote an MLP in Python. After having trained it, I pass in some test data to see the result, and I get an array of decimal numbers at the output, rather than the desired binary output.

For example, I'll get [0.43, 0.56, 0.1, 0.8], rather than [0, 1, 0,1].

What can I do to achieve this?

Thanks

Liam

This looks like a case of the model outputting the probability of being in category 1. It then is up to you to decide on the cutoff.

You give an example of an output of $(0.43, 0.56, 0.1, 0.8)$. If your cutoff is $0.5$, you’d get classifications of $(0, 1, 0, 1)$. If you set your cutoff at $0.2$, which you’re allowed to do, you get classifications of $(1,1,1,0)$. There should be a way to do this in whatever software you’re using. If all else fails, you can loop through your vector of probability values and produce a new vector with some conditional statements about if the value exceeds the threshold and if the value does not. I’ll give some pseudo-code.

for value...
  if value>threshold
    new value =1
  else
    new value =0

This leads to ROC curves and area under the curve, where the model has its accuracy assessed at all cutoffs. Perhaps even better is to use a strictly proper scoring rule. You can read more about this on the Cross Validated Stack, stats.stackexchange.com. Look out for posts on this topic by Frank Harrell. Shamelessly, I will mention a post of mine where I somewhat challenge this idea: https://stats.stackexchange.com/questions/464636/proper-scoring-rule-when-there-is-a-decision-to-make-e-g-spam-vs-ham-email.

Loosely speaking, a strictly proper scoring rule penalizes a model for lacking confidence in its prediction. Outputting a probability of being in class 1 of $0.56$ is not as good of a prediction as $0.99$. Think of the first as, “Sure, I guess it’s category 1,” while the second is, “Oh heck yes it’s category 1!”


How can positional encodings including a sine operation be linearly transformable for any offset? In the paper "Attention is all you need" the authors add a positional encoding to each token in the sequence (section 3.5). The following encoding is chosen:

$ PE(pos, 2dim) = sin(pos / 10000 ^ {2dim/d_{model}} ) $

$ PE(pos, 2dim+1) = cos(pos / 10000 ^ {2dim/d_{model}} ) $

The text states that "for any fixed offset $k$, $PE(pos+k)$ can be represented as a linear function of $PE(pos)$". This did not seem obvious due to me due to the nonlinearity of the sine function. Other resources like Attention is all you need Explained mention this property but do not go deeper into it.

I decided to experiment with this by attempting to map a number of outputs from the $PE(pos)$ function to outputs with a given offset $k$.

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression

def PE_even(pos, dim):
    """ This corresponds to the 2dim state. """
    size = 10000
    d_Model = 512
    return np.sin(pos / (size ** (2 * dim / d_Model)))

positions = pd.Series(np.arange(0, 1000))

k = 10
a = positions.apply(lambda p: PE_even(p, 64))
b = positions.apply(lambda p: PE_even(p + k, 64))

X = a.values.reshape(-1, 1)
y = b.values
r = LinearRegression()
r.fit(X, y)
r.score(X, y)

However, no offset nor any range of numbers in these dimensions yields a suitable fit. As $k$ increases and the sine waves resulting from the $PE(pos)$ function get out of sync, the correlation of the transformation and the truth decreases. Even using simple neural networks with and without linearities does not yield a good fit.

Did I misapprehend the statement in the paper, or is my code or understanding of the underlying math here faulty?

I elected to ask this question on the Mathematics Stack Exchange and I thought it prudent to add the answer here:

https://math.stackexchange.com/q/3119882

From what I have learned from @Servaes, who was kind enough to answer the question, there is a function

$\operatorname{PE}(\text{pos}+k,2d)=\operatorname{PE}(\text{pos},2d)\cos(k/c^d)+\operatorname{PE}(\text{pos},2d+1)\sin(k/c^d)$

that allows for the property of transforming any positional encoding into one with a given offset. However, due to the required use of the sin() and cos() functions this is not a linear transformation.


Different strategies for dealing with features with multiple values per sample in python machine learning models I have a dataset which contains pregnancy, maternal, foetal and children data and I am developing a predictive machine learning model to predict adverse pregnancy outcomes.
The dataset contains mostly features with a single value per pregnancy, e.g. maternalObesity = ["Yes", "No]. However, I have some features that have multiple values per pregnancy, such as the foetal abdominal circumference and estimated foetal weight which have been recorded multiple times at different times during gestation (so each pregnancy will have between 1 and 26 observations for these features each), like so:
PregnancyID     gestationWeek    abdomcirc     maternalObesity
1               13               200           Yes
1               18               240           Yes
1               30               294           Yes
2               11               156           No
2               20               248           No

So in pregnancy 1, we can see that the abdominal circumference was recorded 3 times at weeks 13, 18 and 30.
All questions I have seen here which have addresses the issue of multiple values per sample have been about categorical features, like this and this. Here the suggested solution was to OneHotEncode the features. However, like I said, this does not apply in my case as I have continuous (float) variables.
I have spent the last few months attempting different methods to best handle these features such that I don't lose any valuable information. Simply adding these features into in my dataset will result in almost duplicate rows as the vast majority of my samples have single values (like in the table above.
Here are some of the different approaches I have considered to handle these features:

Derive statistical values from the features, like here. So I compute mean the maximum, minimum, variance, range, etc. of all the observations per pregnancy. However, the downfall with this approach is that the time at which the values are recorded is neglected. The time of the measurement may be significant as a higher abdominal circumference earlier in pregnancy may be more correlated with the adverse outcome I am trying to predict.

Summarise the measurements into a fixed number of features by grouping them into 3 trimester, like here. So I can group all measurements by 3 trimesters, and each feature would hold the maximum measurement recorded during that trimester.

So my dataset will look like this:
PregnancyID     MotherID    abdomCirc1st  abdomCirc2nd   abdomCirc3rd
1               1           200           315            350
2               2           156           248            NaN

This approach takes into account the time range of the measurement, but will result in a lot of NaNs in the new features, as many pregnancies do not have a measurement for each trimester. Also, the maximum may result in some statistical information being lost, unlike approach 1.

I initially thought about using a python list for these features. However, I do not know if a machine learning model can handle this data type, and again, the time each measurement was taken is neglected in this approach.

So my data will look something like this:
PregnancyID     maternalObesity    abdomcirc
1               Yes                [200, 240, 294]
2               No                 [156, 248]

In conclusion, I need some guidance as I have found a lack of examples and resources out there about this issue. So please advise what the best approach is in this case and if there are any detailed examples out there that address this issue I would appreciate it.

You should make the "abdomCirc" as Multiple separate features (e.g. one for each month).
 Then handle NaN as it should be i.e.

Remove columns that are having more NaN than a Threshold
Try to fill NaN for cases where only a few are missing.

If we can't accommodate the above solution because of NaN counts, then we should accept that we don't have enough data and ignore the feature-set (i.e. abdomCirc) altogether.
Otherwise you may be at risk of finding a pattern that doesn't exist because of very small amount of available data points for abdomCirc.

I initially thought about using a python list for these features. However, I do not know if a machine learning model can handle this data type

None of the Model will accept this data i.e. a List


Deep fully connected NN with vanishing gradients I am making NN for choosing best bets possible for football matches. And I tried to make network quite deep (12 hidden layers with BN between them and ReLu as activation function) but it resulted in vanishing gradients problem.
Then I made it shallow (2-3 layers in same manner) and it resulted in great performance in train set but poor in validation set. My hypothesis is that in this configuration it could memorize all the possibilities (I have about 14 k of examples in train set).
Finally I made it in between this two - 8 hidden layers. And it resulted in quite nice performance both in train and val set. But i'm still a little worried about vanishing gradients and that it might be just a coincidence.
So I have few question in reference to what I've just written:

When should you be worried about vanishing gradients? I mean I can
inspect some gradients distribution in tensorboard but this doesn't
give me 100% sure answer.
Could using skip connections help?
Is there any rule how much layers is too much?

Vanishing gradients can occur because of adding so many layers to network. You can think neural networks as composite functions. During learning process gradients of loss function with respect to weights calculated according to chain rule, and because of large number of layers gradients can be so small by multiplications. It causes insufficient updates of weights towards optimum points of loss function. If your loss function barely decreasing, or there is no change at all then cause of this might be vanishing gradients.
Activation functions like tanh, sigmoid also can cause this problem, because their gradients become zero in edge points. Skip connections can ease vanishing gradient problem, because when connections are skipped gradient calculation follow a shorter path, and less multiplications can ease vanishing gradient problem. Resnets use skip connections to ease vanishing gradient problem: https://en.wikipedia.org/wiki/Residual_neural_network.
If you observe no decrease in loss function, then you can decrease number of layers gradually. It's a hyperparameter, and you can experiment with different numbers of layers to find optimum number which gives best results.


If my model is overfitting the training dataset, does adding noise to training dataset help regularizing the machine learning model I would like to know if this is a best practice or not. Can we add noise to the training data to help the model "fit less the training data"; as a result, hoping to generalize better on new unseen data?

No, adding noise will not help to regularise the model.
This won't help your model generalise better to unseen inputs. It will probably just make your model perform generally worse. Any modifications you make to the training instances should be learnable.
Take an image classification problem as an example (images of a cat or not cat): randomly converting pixels to white in every image will not help your model learn generalisable features of cats; random white noise is not a 'learnable' feature. However by rotating, flipping, cropping, or adjusting contrast of the images, and adding these as new training instances, this would force the model to be more tolerant of differences in each image of cats, and therefore generalise better.
There are many methods for regularising models, such as l1 or l2 regularisation in linear regression or neural nets, and also plenty of other methods specific to other model types.
Making sure your training data is as varied as possible, and using model specific regularisation techniques, are probably closer to 'best practise' (though I don't really like the term since 'best practises' often change, such as new regularisation techniques, optimisations etc etc).
Update:
I've since experimented with adding Gaussian noise to input features, and especially in cases with overfit neural nets this can help the model generalise. It's also possible to add GaussianNoise layers directly in a neural net for this purpose, as in the TensorFlow documentation here.


Balanced vs total dataset rows, which one is better? I work on a dataset concerning games playing results. i.e every child play an indefinite number of games and it has as output (y) two possible values "success" or "Failure".    It's about 800 000 samples, with the total number of positive output (1) is ~~ 640 000  and the total number of negative output (0) is ~~ 160 000.

id     player     game    y
1      P1         G1      1
2      P1         G2      0
3      P1         G2      1
4      P1         G3      0
5      P2         G1      0
6      P2         G1      1
7      P2         G2      0

For making a prediction I sort my data by "players" first then by "games".

I want to make some tests for improving the prediction accuracy. As I am a beginner in machine learning and data Science I don't know what is better to choose:

working with this database as it is,  even it unbalanced? ??
or selecting just games that have (approximatively) balanced output? i.e to except any game result which has a total number of positive too begger than a total number of negative  ??

You could work with your database as it is, because 20% positives is not really that unbalanced. Most of the algorithms work perfectly on this kind of dataset.

But if you want to work it differently, I suggest you use Synthetic Minority Over Sampling (SMOTE), which is a technique which repeats the minority dataset until is balanced.


How to determine the number of forward and backward passes in deep learning (CNN)? Is there a way to determine the number of forward and backward passes in the training of a neural network using python?

Forward and backward passes of the whole dataset are called epochs. The number of epochs is a parameter of the training procedure that cannot be estimated a priori, and it depends on how low you want your training/validation errors. One approach is to train until your validation error is small enough, and as long as the number of epochs is smaller than a threshold. Another approach is early stopping: when the validation error achieves a minimum, stop training. For a broader explanation on early stopping, see this blog.

To sum up: the number of epochs is particular of every network and dataset, as well as optimization technique, and you should set them in order to have a low enough loss, but not overfit (early stopping is done in order to not overfit).


How to match features in new records for NLP BOW I have a dataset that has 100,000 records
data in this dataset are 2 columns
1- Text
2- Class
When I apply BOW of my model I get big list of features
That is fine, I managed to work with them
my problem is after building the model and deploying.
now if a new text came with new words then the model wont work as it wokds in same feature structure
Example
"This is a test, test is important" , Red
"Adam pass a test", Green
so my final dataset is
This is a test important Adam pass class
 1    2 1 2    1          0    0   Red
 0    0 1 1    0          1    1   Green

once model created and got this text
"test and exam are similar", Yellow
in this case the set of features has new features which are
and exam are similar
the model will break coz these features never included in the training model
I wonder how to resolve this issue?

If there is new data with labels, you have to retrain the existing model to learn the new relationships.


Does Sklean's SGDClassifier automatically standardize the training data when regularization is turned on? Generally speaking--it is best to apply standarizaton (z-scoring the training data) prior to regularization. Does sklearn.linear_model.SGDClassifier automatically standardize the training data or not when the 'penalty' argument is set to a value other than none (i.e. 'l2', 'l2', or 'elasticnet')?

No, sklearn generally doesn't apply scaling inside of any of its models, instead relying on the user to do that.  This seems like the right way to do it, since you might want to try different scaling techniques depending on your data.

From the User Guide:

  Stochastic Gradient Descent is sensitive to feature scaling, so it is highly recommended to scale your data. For example, scale each attribute on the input vector X to [0,1] or [-1,+1], or standardize it to have mean 0 and variance 1...


Sklearn: How to adjust data set proportion during training, but not testing I'm using sklearn/pandas/numpy.

I have a labeled data set, where the potential outcomes are either True or False. However, the data set has a much higher proportion of True entries. When running through classifiers with k-fold (n=5) cross validation, this appears to bias the classifier towards just saying True.

Using weights, I was able to adjust the sample data set I'm using to have a proportion closer to 1:1, like so (using a pandas csv):

results = csv[['result']]
weights = np.where(results.as_matrix() == True,0.25,1).ravel()
csv_sample = csv.sample(n=60000, weights=weights)

And the results are much more promising! However, I'm wondering if there's a way for me to do cross validation where the TRAINING set is adjusted in this manner, but the TEST set is closer to the actual proportion of data.

Try to use predictor option class_weight='balanced' or auto. It worked really well for me for SGDClassifier in a similar situation.


YOLO: How many bounding boxes? In the research paper, for S=7,B=2, the model predicts 2 bounding boxes for every 7x7 grid cell hence 7x7x2=98 images are predicted per image. Yet the demo output image only has 3 boxes. Why is that?

My theory is that since thickness of the lines is proportionate to the confidence scores of the bounding box, after the model is trained, the "lousy" bounding boxes are so thin that they don't even appear.

The paper also says "Often it is clear which grid cell an object falls in to and the network only predicts one box for each object".

I'm confused.

From 98 boxes to 3 boxes, it involve many other things as well.

  x*y*2 = 98, where 2 are the anchor boxes i.e. each grid will predict two bounding box.
  Non Max Suppression: As correctly said by you, discard those boxes which have lesser probability. You can set some threshold value.
  IOU (Intersection over Union): Step used to identify and discard the overlapping boxes.

Once you done all these activities you will get 3 final boxes.

  More about the complete process:

  Discard bounding box which has low confidence score. Say less than 0.6.

  Now pic the grid having highest confidence score. As mentioned below:
  Here 0.9 confidence score has been picked up.
  Now identify all those girds which has IoU score greater than some threshold, say 0.5. Here highlighted in dark blue and discard such
  grids.

  Still left some grids which predicts the object say car, but IoU is low that is not yet discarded. Repeat such process from step 2 again,
  until we left no such grid.

Note: Suggest you to read more about last two process as it involve the core concept of identifying the bounding boxes.


Can't save save/export and load a keras model that uses eager execution I'm following the RNN text-generation tutorial with eager execution pretty much line for line. I've trained the model with my own data set and have saved a low loss checkpoint. I'm able to load the weights and generate text but I want to export/save the model so that I can learn how to deploy one using flask. However I can't figure out how. The version I'm using is '1.14.0-rc1'.

The tutorial: https://www.tensorflow.org/tutorials/sequences/text_generation

I have been able to save the model as an HDF5 file but I cannot load it. I've also disabled eager execution but that causes problems with running the code later on. I have tried the following and a few more snippets but those led to nothing as well:

new_model = keras.models.load_model("/content/gdrive/My Drive/ColabNotebooks/ckpt4/my_model.h5")

How ever I get

RuntimeError: tf.placeholder() is not compatible with eager execution.

Lastly I found this in another post and tried it as well but was met with another error:

tf.saved_model.save(model, "/content/gdrive/My Drive/Colab Notebooks/ckpt4/my_model.h5")

error:

AssertionError: Tried to export a function which references untracked object Tensor("StatefulPartitionedCall/args_2:0", shape=(), dtype=resource).TensorFlow objects (e.g. tf.Variable) captured by functions must be tracked by assigning them to an attribute of a tracked object or assigned to an attribute of the main object directly.

I ran into the same problem and solved it by running the keras that comes with tensorflow:

from tensorflow.python.keras.models import Model, load_model

instead of:

from keras.models import Model, load_model

I suspect there's a version mismatch at the core of this problem. Loading the model worked with the Keras included with the current Tensorflow 2.0.0-beta1.


How does SelectKBest() perform feature selection? SelectKBest(f_classif, k), where k is the number of features to select, is often used for feature selection, however, I am having trouble finding descriptive documentation on how it works.  A sample of how this works is below:

model = SelectKBest(f_classif, k)
model.fit_transform(X_train, Target_train)

The ANOVA F-value, as I understand it, does not require a categorical response.  (see scipy.stats.f_oneway)  It is computing the value between the features. Why does f_classif require the response?

How does SelectKBest actually achieve a ranking of features based on the F-value when there should only be one F-value for a set of data?

Your question is really more about f_classif than SelectKBest. It's to drop duplicate labels; note the np.unique(y):

X, y = check_X_y(X, y, ['csr', 'csc', 'coo'])
args = [X[safe_mask(X, y == k)] for k in np.unique(y)]
return f_oneway(*args)

f_oneway still only gets passed the feature matrix, but a subset of it.


Text classification: how to assign 1 or more classes to each document? I'm very new to text mining. I am trying to build a classifier which assigns zero, one or more topics to each document of the Reuters 21578 dataset. However, I'm not sure how to make it work using standard R packages. I've looked at a lot of simple online examples using packages such as RTextTools and RWeka, etc. which creates classifier to assign one topic to each document and I'm stuck as I can't directly follow the examples to achieve what I desire. Thanks in advance!

I think what you are looking for is multilabel classification.I used mlr package in R ; I followed this link to achieve it.


How to compare the topic coherence between models of different number of topics? If I'm not mistaken, in this paper here http://svn.aksw.org/papers/2015/WSDM_Topic_Evaluation/public.pdf it appears that topics with larger number of topics will inherently have larger coherence measures because of the summation over all the topics. Is there any flaw in dividing this coherence calculation by the number of topics to get an average for a model of 5 topics and then for a model of 10 topics and comparing the two?

In gensim specifically, I'm using c_v coherence calculation.

After doing some more digging -- it appears that gensim's functions output difference calculations

This below actually does what I am describing above:

cm = CoherenceModel(model=model, texts=docs, dictionary=dictionary, coherence='c_v')

While this below is actually only the summation across the topics of the model not divided by the total number of topics in the model.

top_topics = model.top_topics(texts=docs,dictionary=dictionary,coherence='c_v')


Learning with dirichlet prior - probabilistic graphical models exercise I have the following problem:
Suppose we are interested in estimating the distribution over the English letters. We assume an alphabet that consists of 26 letters and the space symbol, and we ignore all other punctuation and the upper/lower case disctinction. We model the distribution over the 27 symbols as a multinomial parametrized by $\theta =(\theta_1,...,\theta_{27})$ where $\sum_i \theta_i = 1$ and all $\theta_i \geq 0$/

Now we go to Stanford's Green Library and repeat the following experiment: randomly pick up a book, open a page, pick a spot on the page, and write down the nearest symbol that is in our alphabet. We use $X[m]$ to denote the letter we obtain in the $mth$ experiment.

In the end we have collected a databse $D=\{x[1],...,x[2000] \}$ consisting of 2000 symbols, among which "a" appears 100times and "p" 87 times. We use a Dirichlet prior over $\theta$, i.e. $P(\theta)=Dirichlet(\alpha_1,...,\alpha_{27})$, where each $\alpha_i=10$.

Suppose we draw two more samples, X[2001] and X[2002]. If we use $\alpha_i=10$ for all $i$, what is the probability of $P(X[2001]="p",X[2002]="a"|D)$?

I thought we could compute: $P(x[2001]=p|D) \times P(x[2002]=a|D)=\frac{10+87}{270+2001} \times \frac{10+100}{270+2002}$ but it's wrong.

Recall formula : $P(x|u,D)=\frac{\alpha_{x,u}+M[x,u]}{\alpha_u+M[u]}$

Your formula is correct, but the final computing is wrong.
It should be: $\frac{10+87}{270+2000} \times \frac{10+100}{270+2001}$


What is the best way to visualize 10 Fold Cross Validation Scores? I have trained a CNN model and I have applied 10 Fold Cross Validation because I don't have much data to train the classifier. Now I am unsure about how to visulize fold wise results. Please suggest some visualization charts or techniques to display fold wise results.

Do you only have one single model? If you were only mixing the data up (and not trying different parameter values) then any plot might not be very really useful. If performance is on the y-axis, what would be on the x-axis? The fold number itself doesn't really give any insight regarding the results (assuming random batch selection).

Are there some statistics you could compute for each of the 10 folds? If there is any metric you can compute for the x-axis, then using a simple line chart would help see how performance varies with that metric. For example, if you have images, you could compute the average pixel value, or the number of true positives/negatives in that batch (if that idea applies to your problem?). An example from Scikit Learn is this, where the gamma parameter of an SVM is altered on the x-axis:

Additionally, if you have several models (e.g. CNNs with varying numbers of layers / depth), you could use a nice box-whisker plot. Seaborn offers some nice variations. Here is one example that would show four different models:

You could make a box plot for a single model too, which wouldn't be so appealing, but might look like this:

Again, I'm not really sure what you would be putting on the axes.


How to apply my deep learning model to a new dataset? I am doing semantic segmentation (multi-class classification of image pixels) using convolutional neural networks (CNN) in Keras.

In particular, I am applying this to aerial images of crops (vegetation). In Keras, I successfully developed a workflow to segment/classify different crops for one specific dataset (let's call this dataset rural area #1).

Can I apply my Keras weights trained on rural area #1 for initializing the training of another dataset rural area #2? Such as:

model = load_model("weights_ruralarea1.hdf5")

Then I will proceed to model.fit.

The rural area #2 dataset has little training images for training the CNN. So would using my weights for rural area #1 be a form of transfer learning? or will it be a form of fine-tuning?

There is a difference between using pre trained weights and transfer learning.

When you use pre trained weights and re train all the layers in the entire model for the dataset, then you are just working towards converging faster in the direction of better results.
If you were to only train the upper layers and lock all the other layers from training, then you are implementing transfer learning.

For your problem at hand, I would use approach 1.

Would be more than happy to elaborate in case you want me to.

Here is a link that explains how to do transfer learning with Keras : https://towardsdatascience.com/transfer-learning-using-keras-d804b2e04ef8