How to include validation set in the pipeline to tune parameters for an SVM? I have a dataset already divided into train, test and validation set. How can I insert the validation in my pipeline?
Code:
class SVMSentiment(Base):
"""Predict sentiment scores using a linear Support Vector Machine (SVM).
Uses a sklearn pipeline.
"""
def __init__(self, model_file: str=None) -> None:
    super().__init__()
    # pip install sklearn
    from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer, TfidfVectorizer
    from sklearn.linear_model import SGDClassifier
    from sklearn.svm import LinearSVC
    from sklearn.pipeline import Pipeline
    self.pipeline = Pipeline(
        [
            #('vect', CountVectorizer()),
            #('tfidf', TfidfTransformer()),
            ('tfidf', TfidfVectorizer()),
            #('clf', LinearSVC(
                #loss='hinge',

            ('clf', SGDClassifier(
                loss='hinge',
                penalty='l2',
                alpha=1e-4,
                random_state=42,
                max_iter=100,
                learning_rate='optimal',
                tol=None,

            )),
        ]
    )

def predict(self, train_file: str, test_file: str, lower_case: bool) -> pd.DataFrame:
    "Train model using sklearn pipeline"
    train_df = self.read_data(train_file, lower_case)
    dev_df = self.read_data(dev_file, lower_case)
    learner = self.pipeline.fit(train_df['text'], train_df['truth'])
    # Fit the learner to the test data
    test_df = self.read_data(test_file, lower_case)
    test_df['pred'] = learner.predict(test_df['text'])
    return test_df

I don't understand where I should include it since the data is already splitted.
Edit: more on the predict method, I see dev is not contemplated here:
def run_classifier(files: Tuple[str, str, str],
               method: str,
               method_class: Base,
               model_file: str,
               lower_case: bool) -> None:
"Inherit classes from classifiers.py and apply the predict/accuracy methods"
train, dev, test = files  # Unpack train, dev and test filenames
result = method_class.predict(train, test, lower_case)
method_class.accuracy(result)
# Plot confusion matrix
make_dirs("Plots")
print(result)
fig, ax = plot_confusion_matrix(result['truth'], result['pred'], normalize=True)
ax.set_title("Normalized Confusion Matrix: {}".format(method.title()))
fig.tight_layout()
fig.savefig("Plots/{}.png".format(method))

Editing my answer -
def predict(self, train_file: str, test_file: str, lower_case: bool) -> pd.DataFrame:
    "Train model using sklearn pipeline"
    train_df = self.read_data(train_file, lower_case)
    dev_df = self.read_data(dev_file, lower_case)
    learner = self.pipeline.fit(train_df['text'], train_df['truth'])
    # Fit the learner to the test data
    test_df = self.read_data(test_file, lower_case)
    # assuming dev_df is validset
    dev_df['pred'] = learner.predict(dev_df['text'])
    test_df['pred'] = learner.predict(test_df['text'])
    return test_df, dev_df

use two variables when you are calling the predict function to store tes and dev results


Is there any standard pattern recognition algorithm in predicting an item which a user will be buying next, given I have the history of the purchases I am having a list of 10 different items a user has bought in the past. Each item has been bought multiple times. I would like to find a pattern in which the user buys a particular item and predict what he will be purchasing next.

For example I am running a cloth factory and sell clothes to different retailers. I would like to identify what clothes a particular retailer might be asking me next, based on the history of how the retailer has purchased different clothes.

We cannot use time series as the date of purchases has no equal intervals. Also I feel simply passing the sequence of purchased products ordered by date to a neural network will not be of much help.

Is there any standard algorithm to identity a pattern?

I am having two columns in my data: 1. item_id  2. date_of_purchase

I would like to create a model which will predict top 3 items which the user will be buying again next. Input to my model will be present date. Output should be top 3 items which the user might be interested in buying.

Take a look at association rule learning (https://en.wikipedia.org/wiki/Association_rule_learning). A really common algorithm is the Apriori agorithm. You could use the package apyori, it works great: https://pypi.python.org/pypi/apyori/1.1.1

Generate association rules over all items bought by a user, not in a single transaction/purchase.


What is sample and feature I'm reading Scikit-learn and I can't understand sample and feature. (n_samples, n_features)
Can anybody describe those by example?

[x[1,2,3,4], x2[1,2,2,3], x[2,3,2,1]]
The data above has 4 features.  We can gives those features labels with a header.  We'll just call them feature 1, feature 2, feature 3.  For the first entry, feature 1 has a value of 1 and feature 2 has a value of 2 and so on.

A sample, is a subset of data taken from your dataset.  x[1,2,3,4] is a single sample of the dataset.

Whatever you are trying to do with Scikit-learn wants to know how many features you have, my example has 4 features (or columns).


Why is the exploding/vanishing gradient problem not solved by line search? The problem of vanishing gradients is basically that since our step size is proportional to the gradient, if the gradient is very small, it might take a long time to reach a local minimum. So why don't we just not take our step size to be proportional to the gradient, and instead do a line search?
The intuition for why gradient descent ought to work is just that if we move in the direction of steepest descent, it should tend to make the function smaller. However, it's not obvious why the step-size should have to be proportional to the magnitude of the gradient. In an old paper on the subject, Haskell shows that gradient descent converges so long as we do a line search, he doesn't consider the case of proportional step size. The more I think about it, the less good reason I see at all why taking the step size to be proportional to the gradient should be considered a good or natural way to do gradient descent.

Line search may help with exploding/vanishing gradient problem. However, line search does not work well with mini-batches and most training uses mini-batches. One of the main advantages of line search is that it tells you whether you have stepped too far. That advantage disappears when you subsample the data.


Reversing Naive Bayes to find extreme points of data sets I'd like to know if this is a sensible idea and if there exist any already formed methods to do this (I'm new to the data science area).

Essentially, I have used Naive Bayes to accurately classify three types of food, based on their nutritious value (fat, salt, sugar, protein, and carbohydrates as my features).

Now that I can accurately classify these foods, Is there a method which uses the Naive Bayes to reverse this approach, and find the extreme values these features can be to be still classified as a type of food?

E.g: The max fat food1 can be, to still be considered food1.

I realize that these values will change, as other nutrient variables are changed, but I wondered if an optimized set of equations could be obtained in 5 dimensions?

What you are looking for is called "decision boundary", which is the set of (extreme) points laying on the boundary between classes.

Decision boundary of NB is a set of points $\boldsymbol{x}$ that satisfy at least one of these  $K(K-1)/2$ conditions $$i\neq j \in [1, K]:{\Bbb P}(\boldsymbol{x}, C_i)={\Bbb P}(\boldsymbol{x}, C_j) \overset{\forall k}{\geq} {\Bbb P}(\boldsymbol{x}, C_k)$$
Generally, Naive Bayes does not learn an explicit decision boundary (specially when categorical features are involved), however, for example here is the derivation of NB boundary for continuous variables when ${\Bbb P}(\boldsymbol{x}|C_i)$ is chosen from exponential family (e.g. a Gaussian). Nonetheless, you could still select a set of random points, then assign a class to each, and finally plot them using something like t-SNE to get a sense of decision boundaries. For example, a tool like this interactive t-SNE plot would be helpful:

Or a library like mlxtend.plotting that carries out a similar procedure:

On the contrary, models like multi-class logistic regression and SVM both have explicit decision boundaries, which are equations of learned parameters $\boldsymbol{w}$ such as $$w_0+w_1\text{fat}+w_2\text{salt}+w_3\text{sugar}=0$$ that you can access via common libraries.


Can you choose a binary feature matrix for a binary classification model This may be a stupid, but, I am new to deep learning (and machine learning for that matter) and I can't seem to find any literature to help with my question. All I can see when Googling many different questions (trying to change keywords to try get a hit on my question) is about binary classification. And also, binary classification where the feature matrix consists of real numbers.
I would like to know, is it possible to build a binary classifier with a binary feature matrix? And please can you point me to some literature.

I'm not aware of any literature specific to the case of classification based on binary features since it's just a subset of the general case, but it's definitely possible.
A very common example is traditional text classification, where the document is represented as a bag of words: there are different options but each word in the vocabulary can be represented as a boolean variable, representing whether it belongs to the document or not. For example (among many others), a Bernoulli Naive Bayes classifier can be trained on such data.


How to approach speech analysis? I would like to analyze speech samples looking for speech-language pathologies.
Most of the resources I can find are about speech recognition which is a completely different problem.

I will always know what is the text that is being spoken on the given sample.
I can assume that audio samples will always be of good quality with no background noises.

I would like to construct a neural net(or maybe some different model) that will detect certain anomalies, but I have few questions.

How should I pass the input to the net? The input will have two parts: speech and text being spoken (+ label). Should I make I somehow divide these parts or just concatenate them as one sample?
What can I read to better understand working with audio/speech in ML environment?

I think you are too quick to throw AI at this problem. (As you already noted, keep alternatives to neural networks open: maybe factor analysis or principal component analysis are more suitable, or support vector machines)

First, you have to do more work in the problem domain, to find out that different pathologies may result in different models.

My recommendation would be to start small with one pathology. Use domain knowledge to determine if (and to what extent) the pathology is present in some samples, and how that could be observed from the audio samples, in a limited number of parameters.

For some pathologies, speech recognition would help. You could count differences in the text spoken and the text recognized. The count (and maybe a pattern in those counts) could be an input for the model (not necessary a neural network as you already noted).

Some other pathologies would need audio spectral or temporal observations in the audio samples (unnatural pitch, unnatural pauses).

To give an example: stuttering.

I would look in the audio samples and spectra for differences in the patterns. E.g. a spectral plot of stuttering vs. regular speech could look like:

So typically you'd want to detect/measure relative power in some frequency regions as a measure for stuttering and learn your model with such data.

An alternative that might also work is to run a stuttering sample (and a natural sample) through speech recognition, to see if differences can be observed in a countable/measurable way.


Effect of Stop-Word Removal on Transformers for Text Classification The domain here is essentially topic classification, so not necessarily a problem where stop-words have an impact on the analysis (as opposed to, say, sentiment analysis where structure can affect meaning).
With respect to the positional encoding mechanism in transformer language models, when using a pretrained LM is stop-word removal as a preprocessing step actively harmful if the LM was trained on a corpus where they were left in? I'm still working on fully understanding the mechanism but I feel like removing stop-words would affect which wavelength is used to construct the context between any given pair of words with stop words between them, which in turn would impact the encoding.
Or, would this not matter because the regression when trained figures it out from consistently processed input? I feel like it should matter but haven't been able to find anything on the topic.

Very interesting question.

Easy, but probably lazy answer

When using pre-trained models, it is always advised to feed it data similar to what it was trained with. Basically, if it matters, don't remove them, and if it doesn't matter, it doesn't hurt to keep them in. Obviously, if you can, try with or without stopwords, and see what works best for your problem.

Longer answer

You actually have two ways to "remove" your stopwords. Either you remove them from the input sequence altogether. Or, you could replace them with a mask token (i.e. <#UNKNOWN> or <#MASK>).
In the latter case, the transformer will implicitly guess what these masks are and you will have achieved the original goal of stopwords removal: make sure they don't affect the predictive outcome. Indeed, take the sentences:
"I like basketball with an audience" & "I like basketball without an audience"
These two sentences are both about basketball and you wouldn't want with/without to make your model think these sentences are about different topics. By masking with and without, you both "remove" the stopwords, and you don't potentially mess with the fact that the pre-trained model didn't use data without stopwords.
Now, what would happen if you fed an "incomplete" sentence to a transformer. The positional encodings would retain the notion that certain words are before or after each other, which is what you want. But if you remove stopwords, some words may appear "too close" to each other. But does that matter? I do not think so.
If the vector your transformer outputs for "word1 <#Mask> word2" is widely different than the one for "word1 word2", it must mean that the masked token is crucial to the overall meaning of the sentence, which would also suggest that it shouldn't be a stopword to begin with.

Final answer

I would suggest masking the stopwords instead of removing them. But, if performance is so critical that you need to feed smaller sequences, I think that you will be alright.


Neural Network outputing the same value / normalization I had a simple neural network that was outputting the same value regardless of the input. During training, it was behaving normally, with training and validation loss diminishing to a floor value.

The data range was in [-1000, +1000].
The model is a 1D convolutional network. If it is useful :

    train_loss = lcategorical_crossentropy(...).mean()

    net = InputLayer((None, net_z, net_x), input_var=input_var)

    net = ConvLayer(net,  16, 9, pad='same' )#, flip_filters=False)
    net = ConvLayer(net,  16, 9, pad='same')#, flip_filters=False)

    net = DenseLayer(net,   num_units=32)
    net = DropoutLayer(net, p=0.5)

    net = DenseLayer(net, num_units=2, nonlinearity=None)
    net = NonlinearityLayer(net, softmax)

This was corrected by normalizing the data in [-1, 1]

What is the reason behind that ? Backpropagation being stuck I guess, but where can I learn about that in depth ?

If it works for your data, I would suggest normalization by:
input = (input-median(input)) / std(input)

Also try to use a smaller batch size for initial training. And triple check how your data actually looks like. The same goes for the labels.

Depending on the framework, you should also check the data type of the labels. Some unfortunate computation might cast your float probabilities to int, thereby killing any meaningful output.

I can recommend Andrew Ng's famous Coursera lectures. They give good insight into what happens within the neurons.


Use of Nash-Equilibrium in big data environments Has there been a successful implementation of Nash-Equilibrium in big data problems like suggesting a best buy in a stock market, in traffic monitoring systems or crowd control systems.
All the above mentioned scenarios have competitive environments and one needs to get the best possible solution in them, which should suit well for Nash-Equilibrium cases.

This question isn't terribly clear.  Data analysis and strategic modeling (game theory) are different tasks.  Nash equilibrium is a way of understanding the incentives they have by assuming a set of players with assumed utility function and making deductive inferences about what they ought to do to maximize those utility functions given their interaction.  Data analysis is an inductive process.

There are a number of ways game theory and data analysis might interact, here are the easy top two:

Someone might use data to infer players' utility functions (I'm sure this exists in econometrics-land somewhere; also, political scientists have a technique called "ideal point estimation," to infer political preferences from voting behavior---which you can easily google to learn more);
Someone might use game theory to generate behavioral predictions which are testable by data.

Thinking about the specific kinds of cases you mention, the obvious application would be in the stock market one.  Suppose you have a ML model that can reliably predict the market behavior of other people at time T from a given feature set.  Then the consumer of the ML model might have an optimal purchase at T-1, and finding that optimal purchase is going to be strategic.

But combining the two approaches might just break the ML. This is really interesting to think about... musing out loud...

Consider the simple case of a two-player market in one stock. Player 1 wants to buy at T-1 if player 2 will be buying at T (because the price will go up); player 1 wants to sell at T-1 if player 2 will sell at T (because the price will go down).  The naive approach for player 1 is "use my ML model to predict what player 2 will do, then do it first at T-1." But, of course, P1's behavior at T-1 is itself observable by P2, and changes P2's behavior (the price has gone up); moreover, by definition P1's behavior at T-1 can't be a feature of the ML model used to predict P2's behavior at T, because it's behavior that is chosen on the basis of the ML prediction.  All sorts of fun puzzles begin here, but none of them look real good...


How to interpert ResNet50 Layer Types I am trying to recreate the ResNet50 from scratch, but I don't quite understand how to interpret the matrices for the layers.

For instance:
[[1x1,64]
[3x3, 64]
[1x1, 4]] x 3

I know it's supposed to be a convolution layer but what do each of the numbers represent?

In order to make the explanation clear I will use the example of 34-layers:

First you have a convolutional layer with 64 filters and kernel size of 7x7 (conv1 in your table) followed by a max pooling layer. Note that the stride is specified to be stride = 2 in both cases.
Next, in conv2_x you have the mentioned pooling layer and the following convolution layers. Here the layers are normally grouped in pairs (trios in bigger architectures) because is how the residuals are connected (the arrows jumping each two layers). The first matrix:

\begin{equation}\begin{bmatrix}
  3x3, & 64 \\
  3x3, & 64
\end{bmatrix}*3\end{equation}

means that you have 2 layers of kernel_size = 3x3, num_filters = 64 and these are repeated x3. These correspond to the layers between pool,/2 and the filter 128 ones, 6 layers in total (one pair times 3).

Following, we have conv3_x:

\begin{equation}\begin{bmatrix}
  3x3, & 128 \\
  3x3, & 128
\end{bmatrix}*4\end{equation}

2 layers of kernel_size = 3x3, num_filters = 128 and these are also repeated but on this occasion times 4. These are the following 8 green layers in the figure.

This continues until the avg_pooling and the softmax.

Be aware that the stride is always 1 except when the filter size increases. This is discusssed in the paper:

  Plain Network: Our plain baselines are
  mainly inspired by the philosophy of VGG nets. The convolutional layers mostly have 3×3 filters and
  follow two simple design rules: (i) for the same output
  feature map size, the layers have the same number of filters;
  and (ii) if the feature map size is halved, the number
  of filters is doubled so as to preserve the time complexity
  per layer. We perform downsampling directly by
  convolutional layers that have a stride of 2.

  Residual Networks: The baseline architectures
  are the same as the above plain nets, expect that a shortcut
  connection is added to each pair of 3×3 filters.

That is why, each time the number of filters is doubled you will see that the first layer of a different colour specifies num_filters/2.


What does it mean when an Actual vs Predicted plot is like this assuming you are using the best model? I'm trying to predict the monetary value in a fixed time-frame for a project. I wanted to start with a baseline model before doing any feature engineering or advanced pre-processing.

I'm using a feed-forward neural network for regression (Sklearn's MLPRegressor) and I'm exploring a normal, wide and deep neural network.

Usually what I do is feed a dictionary of parameters to my grid-search and get my baseline model from there. Surprisingly enough, the Actual vs Predicted plot I'm getting this time from my best model is confusing.

This is the plot I'm getting here, red is actual and blue is predicted :

This is the loss curve for that model, spoiler alert, it's like I've never seen before :

And these are the parameters from the best model :

{'activation': 'relu', 'alpha': 0.05, 'learning_rate': 'adaptive', 'learning_rate_init': 0.001, 'hidden_layer_sizes': (24, 12, 6), 'solver': 'adam'}

What am I doing wrong ?

It's not said taht you are doing something wrong, some datasets don't contain highly predictable data. However in this case it seems there is a pattern in your data that your models is not capturing. I have a couple of questions about it:

Is there a big difference between train and test performances? If so, you must use some regularization techniques, such as dropout, in order to reduce overfitting.
What is the architecture of your Neural Network? And did you try different ones?


How to handle correlated variables in a dataset? I am wondering how to handle correlated fields in a dataset.

Some people suggest to drop this correlated fields because they don't provide any inside, other claim that a lot of information is getting lost. I am not sure if I should keep this correlated fields or not.

How to handle correlated data. Drop it, keep it, or does it depend on additional parameters? If yes, which? Are there rules I can follow?
What if the fields are sparse? (because the dataset is incomplete)
Does 1) depend on what kind of model is used? Regression, classification, NN?

To give a concrete example, the correlated values from the Zillow competition on Kaggle.

If your model has dependent variables, then it's hard to interpret the coefficients down the road.

For example, say that your model has 3 variables: profits, costs and revenue.
You trained your linear regression model to predict revenue (silly, I know).

You could get the following coefficients: $c_{cost}=-1,c_{profits}=1,c_{revenue}=0$

Someone looking at these results, could conclude that revenue is not predictive of revenue.

This is kind of a trivial example because the linear dependency is very clear (revenue=profits-costs).

But consider it a motivation for not adding dependent variables to your model.

Correlation is a measurement of pairwise linear dependency.

It's more of a question of what type are your variables.

If your variables are continuous, Dimensionality reduction algorithms like PCA should account for the correlation.


How to use r2-score as a loss function in LightGBM? I am trying to implement a custom loss function in LightGBM for a regression problem. The intrinsic metrics do not help me much, because they penalise for outliers... Is there any way to use r2_score from sklearn as a loss function for LightGBM?

$R^2$ is just a rescaling of mean squared error, the default loss function for LightGBM; so just run as usual.  (You could use another builtin loss (MAE or Huber loss?) instead in order to penalize outliers less.)


When is a neural network better "traditional" models like decisions trees and lassos? There's a whole theory of statistical inference based off calculus studying consistency, efficiency, robustness, BLUE, unbiasedness of linear models (Gaussian,Exponential, Chi-square, F-distribution, etc)... that make up regression models.

When is a neural network better than these traditional models based off calculus that are used regression-analysis? Is there a whole mathematical theory including robustness, Rao-blackwell stuff, consistency, sufficency dedicated to them that someone like Wackerly/HOFF writes about in a book?

Are neural networks not explainable by calculus as logistic regression is explainable as the maximum likelihood estimator?

What's the mathematics behind neural networks or is that more CompSci theory?

Are neural networks unbiased estimators that are consistent, efficient, etc that are on average the right answer? Can they estimate the true parameters of the population?

Neural network are "just" a way of estimating a numerical function. Their power is that this comes out of simple matrix multiplications with non linear functions that can be efficiently parallelized. As such, they are perfectly described by calculus.

The question on whether they are biased or not is then related to how they are trained and their black-box behavior. You can put stats at the end of a neural network as well.

The problem IMHO is that generalization for them is a complex topic. Presented with new data, they can have unexpected behavior compared to more traditional techniques for which we understand the behavior at the edge.

If you consider simple regression, like least squares, then neural networks are not actually in play. We use the frameworks to build equations, but we don't build neural networks. As such, we just benefit from the capability of these to process huge amount of data.

We are still going to estimate $y=ax+b$ if we want to estimate the coefficients. For logistic regression, it's still the same boundaries as if we were doing a logistic regression without neural networks.

Neural networks are only in play when the function is unknown and needs to be estimated.


Convert a repetitive list into Pandas Dataframe How do you convert something like this:

A: 1
B: 2
C: 3
###
A: 5
B: 5
C: 6
###
A: 2
B: 5
C: 7

into a dataset where the first row would be the first section with

  A as column-1 B as column-2 and C as column-3

so we get this:

 A B C
 1 2 3
 5 5 6
 2 5 7

If I understand this correctly your sequence is always 3 elements. Then you can do this:

a = ['A:1','B:2','C:3','A:5','B:5','C:6','A:2','B:5','C:7']
b = []
rep_len = 3

# Looping with step size equal to repetition length
for i in range(0,len(a),rep_len):

    # Selecting a repetition length
    c = a[i:i+rep_len]

    # Extracting everything in after letter and colon and casting to integer
    c = [int(x[2:]) for x in c]

    # Append to a list of lists
    b.append(c)

df = pd.DataFrame(b, columns=['A', 'B', 'C'])

Resulting in:

    A   B   C
0   1   2   3
1   5   5   6
2   2   5   7


question about `sklearn.ensemble.BaggingClassifier` I am experimenting with BagginClassifier, but I fail to get the expected functionality.

Basically, the BagginClassifier should draw (bootstrapping) a new data set with replacement. For example: the following code should generate a new bootstrapped sample of the same size as the original data set:

import sklearn.datasets as ds
import numpy as np

X, y = ds.load_iris().data, ds.load_iris().target

bag = BaggingClassifier(base_estimator=LogisticRegression(),
                        n_estimators=100,
                        max_samples=1.0,
                        bootstrap=True,
                        n_jobs=1)
bag.fit(X, y)

print X[bag.estimators_samples_[0]].shape
>> 95

(or any other number close to 95).

Naively, I would expect to get the bootstrapped sample of the same size as the original one (150), but with some random repetition of rows. However, I get a smaller sample size with unique rows. That's strange.

What's wrong here?

Found the answer hiding in lines 93-100 in the bagging.py file.
Here is what I understand - the bootstrapping process works in three steps:

Calculate number of samples to train each estimator on (the max_samples variable in the bagging.py code). In your case its $1.0 * x.shape[0] = 150$.
Select with repetition the needed max samples(as calculated in the previous step).
The selection is done by using randint function, and it generates an array of the x series indices. A given index can appear in this array more than once.
In order to account for indices(samples) that were selected more than one, a weights vector is passed into the base estimator fit function. So, for example, a sample that was selected twice will have $weight * 2$ and it will have the desired impact on the fitting algorithm.

As to my understanding, one can use the estimators_samples_ only to find out which samples were included and not how many times each of them did.


Confidence rating for regression tasks In classification tasks, we can interpret the output vector as how "confident" the model is that the input has a certain label. For example,
y = [0.01  0.20  0.99  0.10]

would mean the model is 99% certain the input has label with index 2 and 1% certain it has label with index 0.
My question is: Is there an equivalent for regression tasks?
If my output looks like
y = 0.33

is there a way to measure how "certain" the model is that the output really is 0.33?

There are many approaches to evaluate the uncertainty of your regression algorithms.
There are model dependent approaches as which the bayesian approach allows you to have an predictive distribution instead of an output which allows you to evaluate your confidence interval. You can apply this approach on linear models as well as on DNN.
There are also model free approach as bootstrapping or quantile regression that allows you to evaluate your uncertainty while considering the prediction algorithm as a black-box model


adding logic combinations of boolean features in classification I want to build a classifier from a dataset of vectors that include exclusively boolean values. Is there any chances that my classifier might perform better if, previously to the learning, I add features that consist in combinations (with logic operators « OR » and « AND ») of the original ones or would that be pointless ?

For example I have the following dataset :

            feature A   feature B   feature C
vector 1    True        False       True
vector 2    True        True        False
vector 3    False       True        True

Imagine I have the feeling that the fact that a data has True for both feature C and feature D would typically make it go in one particular class. Before learning the datas to build the classifier, should I add to each vector en extra feature computed by the logical operation « feature B AND feature C » :

            feature A   feature B   feature C   feature D
vector 1    True        False       True        False
vector 2    True        True        False       False
vector 3    False       True        True        True

Or would the relevance of « feature B AND feature C » be taken into account by the classifier anyway ? Does it depend on the algorithm (svm, Knn, etc. ) ?

Try it and see what happens. Neural networks don't have enough representational power to learn an XOR operation without at least one hidden layer, so there are definitely some interesting features you can construct with logical operations. The AND operation is equivalent to multiplication, which corresponds to interaction terms linear models.

But yeah, it does depend on the model. For example, a decision tree can learn these kinds of features on its own (although it won't necessarily). For exame, an AND operation would correspond to two tests on the same branch.


What distance should I use for edges weights in textrank algorithm I found this python implementation on github with 400+ stars which use levenshtein distance between each nodes.

But original paper (page 4) said:

  Next, all lexical units that pass the syntactic filter
  are added to the graph, and an edge is added between
  those lexical units that co-occur within a window of N word

So the question is: is levenshtein distance legit for this algorithm or better to rewrite with windowed edges?

Intuitively levenshtein distance must not work because it's not denote importance of word...

As lib author said using lev-dist was just en experiment and might not work (and it really doesn't), but for some reason people use broken lib w/o questions ¯\_(ツ)_/¯


Data Extraction from images using NLP and ML Hi i'm trying to extract data like name, planType, phone#, .. from images like insurance cards or licence cards with GoogleVision / Textract using some conditions  but it does not extract the correct content .

is there a way to use ML and NLP together to solve this problem

Thank you

You need to perform OCR (Optical Character Recognition).
However, this can lead to bad results if your images are of very bad quality, or they are skewed and distorted).
One tool you can use is tesseract tesseract from Google. You can train it for your purpose to recognize the text in your images.


Find errors if any in tensorflow computation graph Given a computational graph in tensorflow find errors if any in the underlying code.

import tensorflow as tf
import numpy as np
input = tf.placeholder(shape=(1, 224, 224, 3), dtype=tf.float32)
output = input + 5
with tf.Session() as sess:
  network_input = np.random.randint(5,  size=(700, 350, 3))
  out = sess.run(output, feed_dict = {input : network_input})

The code is throwing error as the shape of the placeholder and the input given is not matching.
So for fixing it I am changing the shape of the place holder.
Is it the correct approach or the input given should be reshaped to the placeholder shape?

According to me, reshaping of the input in perfectly ok.
If you are changing the shape of the parameter just to fulfill the requirements of the method parameter or for further processing of the logic, it is perfectly ok, as long as you know what you are sending and how it will be processed by further code.

But in feed_dict, input and placeholder dimensions should be same otherwise you will get the compilation error.

  Note: In case you are not sure about the exact dimensions of your
  placeholder, you can use [None] as a dimension value while declaring a
  placeholder


Vertical concatenation in a df based on column value_python Is there any way to concatenate vertically a specific column from my df (concat1), considering/filtering values from another column (col_value)?
My df looks like this:
col_value         concat1
data1             x;y;z
data1             d;f;h
data1             p;c;j
data2             s;k;a
data3             a;w;q
data2             o;i;s
data3             e;q;j
data4             d;f;n
data4             q;f;k

Expected output:
col_value         vertical_concat

data1             x;y;z;d;f;h;p;c;j
data1             x;y;z;d;f;h;p;c;j
data1             x;y;z;d;f;h;p;c;j
data2             s;k;a;o;i;s
data3             a;w;q;e;q;j
data2             s;k;a;o;i;s
data3             a;w;q;e;q;j
data4             d;f;n;q;f;k
data4             d;f;n;q;f;k

Many thanks in advance

Try:
a = ["data1"
,"data1"
,"data1"
,"data2"
,"data3"
,"data2"
,"data3"
,"data4"
,"data4"]

b = ["x;y;z"
    ,"d;f;h"
    ,"p;c;j"
    ,"s;k;a"
    ,"a;w;q"
    ,"o;i;s"
    ,"e;q;j"
    ,"d;f;n"
    ,"q;f;k"]

e = pd.DataFrame({"col_value":a,"concat1":b})

e["vertical_concat"] = e.groupby("col_value").transform(lambda x: ";".join(x.unique()))

Output:


How do reshape an image to fit my Mnist Convolutional model? I have done research but cannot seem to find what's wrong here
I have created this model for Mnist digit clasification :
import numpy as np
import matplotlib.pyplot as plt
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Conv2D , MaxPool2D,Dropout,Flatten
from tensorflow.keras.datasets import mnist

model = Sequential()
model.add(Conv2D(filters = 32 , kernel_size = (3,3),activation ='relu' , input_shape = input_shape ))
model.add(Conv2D(64,(3,3),activation = 'relu'))
model.add(MaxPool2D(2,2))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128,activation = 'relu'))
model.add(Dropout(0.5))
model.add(Dense(10, activation ='softmax'))

and I have this image to feed into the model:

So i reshaped the image like this:
from skimage import data
from skimage.transform import resize
image = image_gray
image = resize(image, (28, 28,1)).shape
image

But when i try to predict using this image i get this error:
ValueError: Error when checking input: expected conv2d_1_input to have 4 dimensions, but got array with shape (3, 1)

I have tried multiple ways using skimage but I can't seem to get the model to fit. What can I do?

model.predict(img) requires a batch axis, i.e. img needs to have a shape of (BS, H, W, C). Try model.predict(np.expand_dims(img,0)).


How does SVM classify images? I have read about SVM and understood that for complex divisions, the SVM theoretically plots the data into a higher dimensional plane such that the in the new dimension the data is linearly separable and to achieve this in a practical way, it uses kernel functions which in place of actually transforming the data into a higher plane gives the boundary distance between the boundary and that data point.
But how does this work in the case of an image classifier?
Let's say we need to classify pictures as either dog or cat. In this scenario, a CNN model would learn features like ear size, face shape, nose shape, and other visual characteristics of the training set to classify between a dog and a cat. But what does SVM learn while training and how does it work in this case?

A quick look (this post, for example) suggests that to use SVM for image classification, you need to extract features beforehand, and not run it directly on the image data.


Oversampling/Undersampling only train set only or both train and validation set I am working on a dataset with class imbalance problem. Now, I know one needs to oversample or undersample only the train set and not the test set. But my issue is: whether to oversample the train set and then split it to train and validate set or first split into train and val set and then perform sampling(over/under) on the train set only

Oversample the train data and NOT the validation data since if train data is unbalanced, your test data will most likely show the same trait and be unbalanced.
If you don't know if test data will be balanced or not, oversample only train data.
But there is one recommendation which I will personally give you since I suffered from same problem not so long ago.
Oversample the data (train)
Test accuracy on validation data (which is not oversampled)
Test this accuracy with accuracy obtained from not doing oversampling (or undersampling whichever you performed)
If the results vary only marginally, train the model on non oversampled data. The reason being since the oversampling technique will introduce data points near current data points belonging to same class which may not accurately depict your test data.
In such a case there are two things you can do

Tweak the model parameters if available

For example : if using random forest it has a parameter in model called "class_weight" which if kept at "balanced" will give equal weightage to every output variable which would be inversely proportional to class frequencies in input
Similar parameter is available on other models as well (logistic regression etc.)

When splitting data using train_test_split set parameter stratify

Example : train_test_split(train_data, df['target_column'], stratify = df['target_column'])
Stratify will make sure your train and validation data are split based on output label frequencies based on train data. Like if the data was like 90 to class 'A' and 10 to class 'B'. After split both train and validation will have 90:10 ratio of classes


How do I train a model with pre-saved weights? I need to train a model but the data after preprocessing is too large to fit in the RAM. Is it possible to preprocess the data in batches and train a model for the small data set and then save the weights and then train it again for the next batches with the saved weights?

the workflow would be this way:

preprocess -> train -> save weights -> preprocess next batch -> train with saved weights ...

If yes, how can I do it?

As you said:

  Is it possible to preprocess the data in batches and train a model for the small data set

Yes! In practice, training of Neural Networks always happens with batches. You never fit the whole dataset at once in the model, whatever medium sized dataset could crash any machine.

This is how it works: You extract a slice of your dataframe (the number is usually in the range of 32-256 observations, but it's very task-specific), you train the model on that, and you move to the next until all dataset is completed. This is what an epoch is, in the Deep Learnig jargon.

  ... and then save the weights and then train it again for the next batches with the saved weights?

There's no need to save and reload the model at each iteration, just save it once it was trained for the number of epochs of your choice. If the following day you want to keep training, just reload it and keep iterating one batch at a time.


Compressing profiles with a large number of dimensions I 'think' this is a related question, but not sure how to apply it.

I'm trying to build out a very crude recommendation system using Amazon ML, Facebook likes, and historical actions.

So lets say we have a number of users within a system that promotes products within several categories. To better predict which categories of items to present to the user, we will consider their past interactions with specific items, and the past interactions of other users who share a similar profile. The profile consisting of basically the users Facebook likes data and some demographic info.

I'm unsure of how to distill the Facebook likes data in a way that lets me make meaningful comparisons between users.

I'm sure its obvious, but I'm completely new to machine learning, and data science in general. I'm currently limited to the capabilities of Amazon ML. Let me know if the question needs more clarification, constructive criticism is appreciated.

Edit:
As @liangjy pointed out, the solution to the recommendation system in general will be to use the collaborative filtering technique. This is most useful when their is sufficient data to link users based on their individual actions. Because we do not have enough/any data on new users, we are trying to use additional data (Facebook likes in this case) to help create that initial link. Where I'm stuck at is the vast number of profile/sites any one user may have liked. We could be comparing the likes of one user against tens of thousands of possibilities the rest of the users have presented. What is the best way to make this disparate data into something manageable?

I've considered taking a subset of users and pulling the top n sites (1,000?) from each demographic groups (age/gender). Then compare all other users to this base set, and create sub-groups based on their relationships to those sites. However, something about this approach feels like it would be skewed. I'm not sure what, but I'm pretty sure I won't get the results I'm looking for.

This seems similar to the collaborative filtering problem of assigning ratings to movies based on similar users and past reviews: https://en.wikipedia.org/wiki/Collaborative_filtering. The essence of this approach is to find a low-rank factorization of the matrix of ratings, where each user has one rating for each movie/product, and some of the ratings are unseen.


Getting TypeError: expected bytes, Descriptor found while importing tensorflow I am trying to use tensorflow and keras for one of the tasks but while importing the tensorflow I am getting the below error. Till now I have removed the virtual which I have created previously and then created the new virtual environment again, but still I am getting the same error. Any leads would be appreciated. Here is the command below which I am running.

import tensorflow as tf

Here is the error I am getting

runfile('C:/Users/Lenovo/.spyder-py3/temp.py', wdir='C:/Users/Lenovo/.spyder-py3')
Traceback (most recent call last):

  File "C:\Users\Lenovo\.spyder-py3\temp.py", line 1, in <module>
    import tensorflow as tf

File "C:\Users\Lenovo\Anaconda3\envs\myenv\lib\site-packages\tensorflow__init__.py", line 28, in
    from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import

  File "C:\Users\Lenovo\Anaconda3\envs\myenv\lib\site-packages\tensorflow\python\__init__.py", line 52, in <module>
    from tensorflow.core.framework.graph_pb2 import *

  File "C:\Users\Lenovo\Anaconda3\envs\myenv\lib\site-packages\tensorflow\core\framework\graph_pb2.py", line 16, in <module>
    from tensorflow.core.framework import node_def_pb2 as tensorflow_dot_core_dot_framework_dot_node__def__pb2

      File "C:\Users\Lenovo\Anaconda3\envs\myenv\lib\site-packages\tensorflow\core\framework\node_def_pb2.py", line 16, in <module>
    from tensorflow.core.framework import attr_value_pb2 as tensorflow_dot_core_dot_framework_dot_attr__value__pb2

  File "C:\Users\Lenovo\Anaconda3\envs\myenv\lib\site-packages\tensorflow\core\framework\attr_value_pb2.py", line 16, in <module>
   from tensorflow.core.framework import tensor_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__pb2

  File "C:\Users\Lenovo\Anaconda3\envs\myenv\lib\site-packages\tensorflow\core\framework\tensor_pb2.py", line 16, in <module>
   from tensorflow.core.framework import resource_handle_pb2 as tensorflow_dot_core_dot_framework_dot_resource__handle__pb2

  File "C:\Users\Lenovo\Anaconda3\envs\myenv\lib\site-packages\tensorflow\core\framework\resource_handle_pb2.py", line 92, in <module>
__module__ = 'tensorflow.core.framework.resource_handle_pb2'

TypeError: expected bytes, Descriptor found

Below is the configuration of my system.

(myenv) C:\Users\Lenovo>conda list tensorflow
# packages in environment at C:\Users\Lenovo\Anaconda3\envs\myenv:
#
# Name                    Version                   Build  Channel
tensorflow                1.14.0                   pypi_0    pypi
tensorflow-estimator      1.14.0                   pypi_0    pypi

(myenv) C:\Users\Lenovo>conda list keras
# packages in environment at C:\Users\Lenovo\Anaconda3\envs\myenv:
#
# Name                    Version                   Build  Channel
keras-applications        1.0.8                    pypi_0    pypi
keras-preprocessing       1.1.0                    pypi_0    pypi

(myenv) C:\Users\Lenovo>

I resolved the issue by creating new environment with tensorflow by the below two commands:

conda create -n tensorflow_env tensorflow
conda activate tensorflow_env


Querying DBpedia from Python How can I get information about an entity from DBpedia using Python?

Eg: I need to get all DBpedia information about USA.  So I need to write the query from python (SPARQL) and need to get all attributes on USA as result.

I tried :

PREFIX db: <http://dbpedia.org/resource/>
SELECT ?p ?o
WHERE { db:United_States ?p ?o }

But here all DBpedia information is not displaying.

How can I do this and which all are the possible plugins/api available for python to connect with DBpedia ?

Also what will be the SPARQL query for generating the above problem result?

You do not need a wrapper for DBPedia, you need a library that can issue a SPARQL query to its SPARQL endpoint.  Here is an option for the library and here is the URL to point it to: http://dbpedia.org/sparql

You need to issue a DESCRIBE query on the United_States resource page:

PREFIX dbres: <http://dbpedia.org/resource/>

DESCRIBE dbres:United_States

Please note this is a huge download of resulting triplets.

Here is how you would issue the query:

from SPARQLWrapper import SPARQLWrapper, JSON

def get_country_description():
    sparql = SPARQLWrapper("http://dbpedia.org/sparql")
    sparql.setReturnFormat(JSON)

    sparql.setQuery(query)  # the previous query as a literal string

    return sparql.query().convert()


Can we apply community detection algorithms for word vector space? As I understand we can apply community detection algorithms such as Louvain to detect communities in a social network (i.e. involves people).

But I am quite interested in knowing if we can use the same community detection algorithms such as Louvain to identify communities in word vector space (e.g., word2vec), instead of clustering?

"Improving Community Detection in in Wikipedia Articles using Semantic Features"
This paper talks about various methods of community detection and might be helpful.


Is it better to have higher train accuracy with lower test accuracy or higher test accuracy with lower train accuracy? The results from my RandomForest model with 5 max features are as follows:

84% train accuracy
76% test accuracy

The results with 10 max features:

79% train accuracy
77% test accuracy

Which result should I be favoring? Would I be incorrect in saying that the second result is better because the test accuracy is higher even though the training accuracy is lower? Ultimately, you want the model to perform best on the test (unseen) data?

Test accuracy better reflects generalization error, so you want the one with higher test accuracy. In your first setup, the higher train accuracy indicates overfitting, as it's significantly higher than train accuracy. This is also kind of why it generalizes less well than the second one.


Is there a way to drop duplicated rows based on an unhashable column? i have a pandas dataframe df with one column z filled with set values

i want to drop duplicated rows where 2 rows are considered duplicated version of one another when they have same column z values ( which are sets ).

import pandas as pd

lnks = [ ( 'a' , 'b' , { 'a' , 'b' } ) , ( 'b' , 'c' , { 'b' , 'c' } ) , ( 'b' , 'a' , { 'a' , 'b' } ) ]
lbls = [ 'x' , 'y' , 'z' ]
df = pd.DataFrame.from_records( lnks , columns = lbls )

Trying to drop duplicated rows based on column z values :

df.drop_duplicates( subset = 'z' , keep='first')

And i get the error message :

TypeError: unhashable type: 'set'

Is there a way to drop duplicated rows based on a unhashable typed column ?

It is true that a set is not hashable (it cannot be used as a key in a hashmap a.k.a a dictionary). So what you can do is to just convert the column to a type that is hashable - I would go for a tuple.

I made a new column that is just the "z" column you had, converted to tuples. Then you can use the same method you tried to, on the new column:

In [1] : import pandas as pd
    ...:
    ...: lnks = [ ( 'a' , 'b' , { 'a' , 'b' } ) , ( 'b' , 'c' , { 'b' , 'c' } )
    ...: , ( 'b' , 'a' , { 'a' , 'b' } ) ]
    ...: lbls = [ 'x' , 'y' , 'z' ]
    ...: df = pd.DataFrame.from_records( lnks , columns = lbls)

In [2]: df["z_tuple"] = df.z.apply(lambda x: tuple(x))

In [3]: df.drop_duplicates(subset="z_tuple", keep="first")
Out[3]:
   x  y       z z_tuple
0  a  b  {b, a}  (b, a)
1  b  c  {c, b}  (c, b)

The apply method lets you apply a function to each item in a column, and then returns the values as a new column (a Pandas Series object). This lets you assign it back to the original DataFrame as a new column, as I did.

You can also remove the "z_tuple" column then if you no longer want it:

In [4] : df.drop("z_tuple", axis=1, inplace=True)

In [5] : df
Out[5] :
   x  y       z
0  a  b  {b, a}
1  b  c  {c, b}
2  b  a  {b, a}


Encoding Time Values I am using Python/Scikit to do data encoding before I go ahead and train my Neural Network. I have a few columns that look like this

07:05:00
08:41:00
17:25:00
12:58:00
08:56:00
11:59:00
17:25:00
15:24:00

Any suggestions on how to encode this? Or is leaving it like this fine?

A common step in "feature engineering" is parsing these out into multiple values that might give you additional information during ML. For example:

Hour of the day
Minute of the hour
Hours after sunrise
Hours before sunset

If you also had day, that would open a lot more to detect seasonality like:

Month of the year
Month of the quarter
Day of the week
Holiday Y/N


Using GRU with FeedForward layers in Python I'm trying to reproduce the codes in this paper here for the multi-labeling problem (11 classes), which is using

1- Embedding layer
2- GRU
3- two Feed forward Layers with the ReLU activation function
4- sigmoid unit.

I've tried to run the codes, but it is showing the following error:

  ValueError: Error when checking target: expected dense_5 to have 3
  dimensions, but got array with shape (6838, 11)

Edit: The error is fixed. I changed the "return_sequences" to False, and removed flatten() to fix the error.

My code: i'm not sure if 2 Feedforward layers are correct.
in the paper it stated FF1:1024 units, and FF2: 512 units. with mini-batch size of 32. How can I state it in the code?

target_input=Input(shape=(max_length, ))

target_embedding=Embedding(input_dim=vocabulary_size, output_dim=embedding_dims, #embedding_matrix]
                           input_length=max_length, weights=[embedding_matrix] , trainable=False)(target_input)

#target_embedding=Dropout(0.3)(target_embedding)

target_gru1=Bidirectional(GRU(units=200, return_sequences=True, dropout=0.3, recurrent_dropout=0.3))(target_embedding)
target_gru=Bidirectional(GRU(units=200, return_sequences=False, dropout=0.3, recurrent_dropout=0.3))(target_gru1)

# target_gru=Dropout(0.3)(target_gru)

#2 feedforward layers
# target_output1=Activation("relu")(target_gru)
# target_output2=Activation("relu")(target_output1)

FF1 = Dense(1024)(target_gru)
target_output1=Activation("relu")(FF1)
FF2 = Dense(512)(target_output1)

target_output=Dense(units=11, activation="sigmoid")(FF2)#target_output2)
target_model=Model(target_input, target_output)
## configuring model for training:
opt = Adam(lr=0.0001)#lr=0.001,decay=0.5
target_model.compile(optimizer=opt,loss="binary_crossentropy", metrics=["categorical_accuracy"])

and here is the layers

The error is caused by return_states = True. You set it to True only if you feed the output of a recurrent layer to another. The "states" are the hidden states of recurrent cells, that could be fed to Dense() layers only though Flatten().

Moreover, I suggest you to delete the Dropout() layer. Don't put it after an Embedding(), it contains information (i.e. the learned representation of words/chars) that is not safe to be distorted by dropout. This is just a suggestion of course.

EDIT:

You need return_sequences = True only when a recurrent layer outputs to another recurrent layer. If the following layer is Dense(), then you can drop return_sequences = True and also Flatten().


How can data science teams inside businesses measure costs and efficiency of their technical work? How can data science teams measure and improve costs of their technical work, when they often don't know the monetary value of the datasets and insights they are producing? Are they using industry based benchmarks for technical development, and some subjective measurement for business insight creation?

I don't know what you mean by "measure and improve costs of their technical work" - but I'll answer in response to the question "how do you know how much value your insights are bringing the business":
To structure a successful project it is important to understand the baseline you are competing with. For example: let's say I have a timeseries forecasting problem where I'm forecasting how much cash will be in some corporate account. One baseline might be: "let's assume that tomorrow's net cash flows will be identical to yesterday's". Now that you have this benchmark you can evaluate how performant your solution is relative to this.
This can give you insight: "my solution outperforms the benchmark by some % and increases the detection of future overdraft fees by 90%". Convert this 90% improvement into the total # of new overdraft fees detected by your system relative to the baseline. This can be translated to some dollar amount on an annual basis - and you can say "my solution saves the business $###,### annually"
Understanding the system you are replacing allows you something to compare to and can track results. This is also an effective strategy for communicating successes to management (something that is vital in larger enterprises).


How can you determine the growth rate you need to achieve a certain customer base? Say you have 12 K active customers every month in your platform in August 2016, how can you determine at what rate do you need to grow every month to achieve a certain total (say, 1.5 million) by next year, August 2017. Further, how can you generate different growth scenarios? Say, grow aggressively during the first quarter and then plateau. Are there any time series or optimization tools related to this kind of problem?

A growth rate is an exponential growth, you multiply your userbase by a certain fraction, which is above 1 in case of growth, else it would be decline. This effect compounds because the amount of users is bigger while being multiplied by the same multiplier. Let's call this multiplier a. We apply this 12 times, once for every month. We are interested in:

$$12000 \cdot a^{12} = 1500000$$

This equals:

$$a^{12} = 125$$

We can take the 12th root on both sides:

$$a = 1.495$$

This means you would need to grow approximately 50% every month.

For growing faster first and then plateauing you can do similar equations, although a bit more difficult. You have to express the growth at the first level in terms of the other level because else there are two unknowns which make for infinite solutions. Let's say the first 3 months the growth b is five times as high as the growth a last 9 months. This means $b = 5(a-1)+1$. This leads to the following equation.

First we have three months of growth rate b, which means after three months our userbase is $12000\cdot b^3$ and after that we have nine months of growth a which leads to the final user base being $12000\cdot b^3\cdot a^9$. Since $b=5(a-1)+1$ this comes down to $12000\cdot (5(a-1)+1)^3\cdot a^9$, which according to Wolfram Alpha comes down to a being around 1.2785 which means b is around 2.3925. This means the first three months a growth of 139% and then nine months of 27.85% growth.


Face Recognition (Scalability Issue) Background

I would like to build a face recognition model for registration and login for some kind of service. For example, using this approach (CNN + SVM).

When a new user wants to register a service, the image of his/her face is recorded and the machine learning model is trained using these images. Then, when a person requests for the service, the model recognises if this person is a member or not.

Question

But when there is new user comes in for registration, the machine learning model has to be retrained and scans through all the previous images (or feature vectors). It seems that this approach has a scalability issue when the the number of users is large.

I have read through this post but my situation is not quite the same as the suggested answer, because my machine learning model aims to distinguish member vs non-member. Does anyone know how to tackle this scalability issue? Thanks.

The problem you describe may be tackled with online machine learning where you continuously update your model as new data arrive,  avoiding the computationally intensive part of retraining.

For deep neural networks, there is some work in this direction.

scikit-learn and Vowpal Wabbit also provide some online algorithms.


Pickled machine learning models Is there a website where people store their pickled models for others to try? E.g. different people might try different ML approaches on the iris dataset, is there a place where I can find/download models others have constructed?

On Kaggle.com, there is a datasets section. In this section, you can post or find datasets. For each dataset, you have the data, possibility to discuss about it and, of course, a list of kernels posts by the user. For instance, for the iris dataset, you have 592 kernels available.


Explanation of why Neural Networks are non convex Why having a symmetry of values for the hidden state imply that the neural network is non convex? I could not find an intuitive answer for this yet.
Also, if we consider a Fully Connected network wtih ReLU non linearity(which is convex) then it is a composition of convex functions and ReLU is also non decreasing. Still the overall optimization problem becomes non convex.

From the CV thread Cost function of neural network is non-convex?:

How does this relate to our neural network? A cost function $J(W,b)$
has also a number of local maxima and minima, as you can see in this
picture,
for example.
The fact that $J$ has multiple minima can also be interpreted in a
nice way. In each layer, you use multiple nodes which are assigned
different parameters to make the cost function small. Except for the
values of the parameters, these nodes are the same. So you could
exchange the parameters of the first node in one layer with those of
the second node in the same layer, and accounting for this change in
the subsequent layers. You'd end up with a different set of
parameters, but the value of the cost function can't be distinguished
by (basically you just moved a node, to another place, but kept all
the inputs/outputs the same).

From NonConvexity of neural networks:

I don’t remember where I heard this arguement but it was a
pretty interesting one and I couldn’t remember the entire
reasoning and so as I began to think about it, I figured some of it
out and I believe it is pretty reasonable. We have all argued that
neural networks are difficult to optimize  because they are
non-convex(or non-concave) for that matter. What we don’t talk
about so much is why is it that they are non-convex. It probably has
simpler answers where you will argue something about the
non-linearities involved. This post has to do with a much more
interesting answer.
To understand the reasoning, we have to argue that if the function has
atleast two local minimas such that their mid-point is not a local
minima, the function is non-convex. In formal terms, $$\text{If
> }\exists x,y \in Dom(f) \hspace{5pt }s.t.\hspace{5pt} g(x) = g(y) = 0
> $$ and $$\forall z\in [x,y] \text{ s.t. } g(z) \neq 0  $$ where $g(x)$
is a subgradient at $x$ the function is non-convex.
This is pretty simple to follow from the Mid-point value theorem and
the definition of convexity.
Consider a neural network  $F(\cdot)$ with a few layers and name three
layers $L_1, L_2$ and $ L_3$ Consider nodes $a$ and $b$ in $L_1$ and
$c$ and $d$ in $L_2$.Let the parameters connecting node $i$ to $j$ be
called $w_{ij}$.
Thee arguement to the function can be thought of as $A=[\cdots w_{ac},
> w_{ad}, \cdots , w_{bc}, w_{bd}\cdots]$ and the function evaluation as
$f(A)$. Consider $B=[\cdots w_{ad}, w_{ac},\cdots, w_{bd}, w_{bc},
> \cdots]$. I can claim that there exists a $B$ i.e. where $ w_{ad},
> w_{ac}$ and $ w_{bd},  w_{bc} $  are swapped respectively, there
exists an ordering of the other weights(permute the other edges) such
that the function evaluation remains same. It is pretty easy to see
how it can be true.
Look at the following picture where the reg edges come into one node
and the green into another.

For reference consider

The first red edge coming out of the top left node as $w_{ac}$
The gree edge out of the same node as $w_{ad}$.
The red edge out of the second node in the same layer as  $w_{bc}$
The green edge out of the second node as   $ w_{bd} $.

Connect all edges coming in to $L_2$ from $L_1$ into node $d$ to node
$c$ and vice versa and then shift the origin of  all outgoing edges
from node $c$ to $L_3$ to node $d$ and vice versa. What you have
acheived is the network that you would have achieved by holding the
two nodes by your hand and manually exchanging their positions while
keeping the connecting edges intact. And Voila! now you have $A, B \in
> dom(F)$ where $F(A) = F(B) $.
Now look at them after the edges are shifted.

[..]What this effectively means is that if the neural network was
indeed convex, there would be a value which could be given to all
nodes in a layer and yet the network would have represented a local
optima. This is higly absurd and is very apparent if you consider a
classification network where the final outputs should not have the
same value in all nodes given some value in the input layer(which is
also a part of the function arguement).
To generalize this to any neural network, you will have to look at the
activation functions and their properties. The claim you will be
trying to verify is that the average of two permutations of an optimal
point need not be optimal.
I guess this was an interesting read. A very interesting thing this
says is that
A neural network cannot have a unique optima if it has atleast two
distinct node values in the same layer. Amazing , isn’t it ?


Can I use euclidean distance for Latent Dirichlet Allocation document similarity? I have a Latent Dirichlet Allocation (LDA) model with $K$ topics trained on a corpus with $M$ documents. Due to my hyper parameter configurations, the output topic distributions for each document is heavily distributed on only 3-6 topics and all the rest are close to zero ($K$~$\mathcal{O}(100)$). What I mean by this, is that the 3-6 highest contributing topics for all documents is orders of magnitude (about 6 orders) greater than the rest of the topic contributions.

If I use the Jensen-Shannon distance to compute the similarity between documents, I need to store all values of the topic distribution as non-zero, even the very small values of the non contributing topics, because Jensen-Shannon divides by each discrete value in the distribution. This requires a lot of storage and is inefficient.

If, however, I store the topic distributions of each document as a sparse matrix (the 3-6 highest contributing topics are non-zero and the rest are zero) where each row is a unique document and each column is a topic, then this uses far less space. But I can no longer use the Jensen-Shannon metric, because we would be dividing by 0. In this case:

Can I use the euclidean distance between documents topic distributions to compare similarity between documents?

Using the euclidean distance would require far less storage and is extremely fast to compute.

I appreciate that Jensen-Shannon is one of the "correct" metrics to compare discrete probability distributions, as well as the Bhattacharyya distance and Hellinger distance. But ultimately, the output of LDA is a discrete topic distribution for each doucment - each document is a vector (or point) in a $K$ dimensional space. By this argument, is it valid to use the euclidean distance to calcualte documents similarities? Is there something blatantly wrong with this method?

I have tested the euclidean distance to compare documents, and yielded good results, which works well for my industrial application. But I want to know the academics behind such a method. Thanks in advance!

Euclidean distance -by which in this application, I assume you mean the euclidean distance in an $n$-dimensional space defined by the distribution of document contents among $n$ topics considered, is a valid measure to use in comparing the topics represented within two documents.

What you're doing by applying this method is quantifying a topic frequency difference within this newly defined space, and so interpretation of these quanta will require analysis of the space. For example, what euclidean distance indicates that documents are relatively similar?

In distiction, the normalized result of something like the hellinger distance provides an easily interperable framework by which to evaluate the results- a score of 0 indicates no overlap in the distribution over the topics in question of the two documents, and a 1, perfect overlap.

For the efficiency concerns, it's not clear to me why you couldn't truncate your topics considered to the crucial topics and then calculate any of the metrics on the distributions over ony those topics, rather than the entire universe of considered topics.


If i no longer have access to feature in a data set should i retrain my model? based on naive Bayes theorem if I no longer have access to the address book to tell whether the email author is known. Should I re-train the model, and if so, how?

Naive Bayes, as far as I know, does not have any internal method of dealing with missing data (which is what you have). Thus, there are really two options:

1) Retrain the model, discarding the variable that contains the is author known variable. This method will probably make your model worse if the is author known variable is important, but at least you won't have any missing data to deal with.

2) Impute the missing values using some sort of algorithm, like MICE, mode imputation, knn, random forest, etc. For this problem, just from a first glance there might be correlations between x1 and your other variables that you can exploit to make a good guess for what x1 would be, given the other known values. This introduces more uncertainty in your model and might even make it worse then simply dropping x1 outright (option 1), but it is worth a shot if x1 is an important predictor.


How to do backward features elimination when considering interactions between them I have a multi linear regression problem,

$Y$ is my target and $X_1, X_2, X_3$ are my features.

In my regression, I consider the interaction between $X_1, X_2, X_3$ and I add a bias.

So my problem is given by :
$Y  \sim X_1 + X_2 + X_3 + X_1X_2 + X_1X_3+ X_2X_3+ bias$

Now, I fit my model with statsmodels.api.sm and I want to eliminate the feature the highest p value recursively.

My first question is : for example, if the highest p value is for the $X_1X_2$ feature, is it okay to eliminate this feature even when $X_1$ and $X_2$ can be statistically significant ?
My second question : in the case when all the interaction of some feature have a p value greater than 0.05 in the first iteration, Could I eliminate this feature and all the interactions ?

Thank you for your help

My first question is : for example, if the highest p value is for the X1X2 feature, is it okay to eliminate this feature even when X1 and X2 can be statistically significant ?

  Of course, the interaction can have no information about the target. Per example if the problem is perfectly defined by X1 and X2. The interaction $X_1 \cdot X_2$ won't add nothing to the model.

My second question : in the case when all the interaction of some feature have a p value greater than 0.05 in the first iteration, Could I eliminate this feature and all the interactions ?

  I would try a more experimental approach of removing them only if they don't improve the model accuracy rather than having a low P-Value.

As a further reccomendation I would reccomend sklearn.


approach to classify text with natural language processing methods I have a problem with regards to text classification/categorization. The task is bugging me for days already and as I am pretty new to AI and the field of natural language processing (NLP) I am just overwhelmed by the content online and available tools/libraries (e.g. NLTK, Keras, spaCy, etc.). It would be awesome if you could give me some guidance or clues on how you would approach the problem.

Issue: basicially I try to set up a tool for classifying text. I already have an extensive labeled dataset to work with. The input will always be a list of some sort (think of an Excel file with 500 rows). Each row contains a single word or a combination of words, i.e. no sentences.

A simplified example of my labeled dataset - input on the left, classification on the right:

"dog" -> "animal"
"dog owner" -> "person"
"dog owner house" -> "building"
"owner" -> "person"
"dog food" -> "food"
"food court" -> "building"

My existing labeled dataset has around 2,000 of these classifications with in total 50 unique categories. How can I set up an algorithm that scans the input for example for the word "dog" - if it is only "dog" then it is the category "animal", if it is "dog" and "owner" it is the category "person", if it is "dog", "owner" and "house" it is the category "building" and so on.

If I set up a ton of if-else-statements as a decision tree it is just cumbersome and intransparent. Is there a way with NLP to solve such an issue?

Thank you very much in advance! Looking very much forward to your ideas and please let me know if I have to be more specific in any way.

Best regards, pythoneer

This problem seems like a multi-class multi-label problem. The questioner seems to be comfortable in building a detailed ontology. These lead the author to propose the following approach. Please note that a detailed explanation of this can be found in an article here.

Steps to solve the problem:

Build a taxonomy file as a csv file as given below. Please note, the column headings should be identical to whats given below.

Put all your content in another csv file that looks like below. Please note, the column headings should be identical to whats given below.

In the following python code, please enter the path to content in the path to df and path to taxonomy in the path to df_tx. These steps are present near the comment import data for mapping. Add another path value for the output towards the end of the code.

Run the python code below. Please note that this code runs fine on Python 2.7 in Windows 10 machine. Please iron out any technical issues yourself as the author may not be of much help for such issues.

#Invoke Libraries
import pandas as pd
import numpy as np
import re

#import data for mapping
df = pd.read_csv("path to content csv");
df_tx = pd.read_csv("path to taxonomy csv");

#Build functions
#function that identifies taxonomy words ending with (*) and treats it as a wild character
def asterix_handler(asterixw, lookupw):
    mtch = "F"
    for word in asterixw:
        for lword in lookupw:
            if(word[-1:]=="*"):
                if(bool(re.search("^"+ word[:-1],lword))==True):
                    mtch = "T"
                    break
    return(mtch)

#function that removes all punctuations. helps in creation of set of words
def remov_punct(withpunct):
    punctuations = '''!()-[]{};:'"\,<>./?@#$%^&*_~'''
    without_punct = ""
    char = 'nan'
    for char in withpunct:
        if char not in punctuations:
            without_punct = without_punct + char
    return(without_punct)

#function to remove just the quotes(""). This is for the taxonomy
def remov_quote(withquote):
    quote = '"'
    without_quote = ""
    char = 'nan'
    for char in withquote:
        if char not in quote:
            without_quote = without_quote + char
    return(without_quote)

#split each document by sentences and append one below the other for sentence level categorization and sentiment mapping
sentence_data = pd.DataFrame(columns=['slno','text'])
for d in range(len(df)):
    doc = (df.iloc[d,1].split('.'))
    for s in ((doc)):
        temp = {'slno': [df['slno'][d]], 'text': [s]}
        sentence_data =  pd.concat([sentence_data,pd.DataFrame(temp)])
        temp = ""

#drop empty text rows and export data
sentence_data['text'].replace('',np.nan,inplace=True);
sentence_data.dropna(subset=['text'], inplace=True);

data = sentence_data
cat2list = list(set(df_tx['Category2']))
data['Category'] = 0
mapped_data = pd.DataFrame(columns = ['slno','text','Category']);
temp=pd.DataFrame()

for k in range(len(data)):
    comment = remov_punct(data.iloc[k,1])
    data_words = [str(x.strip()).lower() for x in str(comment).split()]
    data_words = filter(None, data_words)
    output = []

    for l in range(len(df_tx)):
        key_flag = False
        and_flag = False
        not_flag = False
        if (str(df_tx['Keywords'][l])!='nan'):
            kw_clean = (remov_quote(df_tx['Keywords'][l]))
        if (str(df_tx['AndWords'][l])!='nan'):
            aw_clean = (remov_quote(df_tx['AndWords'][l]))
        else:
            aw_clean = df_tx['AndWords'][l]
        if (str(df_tx['NotWords'][l])!='nan'):
            nw_clean = remov_quote(df_tx['NotWords'][l])
        else:
            nw_clean = df_tx['NotWords'][l]
        Key_words = 'nan'
        and_words = 'nan'
        and_words2 = 'nan'
        not_words = 'nan'
        not_words2 = 'nan'

        if(str(kw_clean)!='nan'):
            key_words = [str(x.strip()).lower() for x in kw_clean.split(',')]
            key_words2 = set(w.lower() for w in key_words)

        if(str(aw_clean)!='nan'):
            and_words = [str(x.strip()).lower() for x in aw_clean.split(',')]
            and_words2 = set(w.lower() for w in and_words)

        if(str(nw_clean)!= 'nan'):
            not_words = [str(x.strip()).lower() for x in nw_clean.split(',')]
            not_words2 = set(w.lower() for w in not_words)

        if(str(kw_clean) == 'nan'):
            key_flag = False
        else:
            if set(data_words) & key_words2:
                key_flag = True
            elif(bool(re.search('"',df_tx['Keywords'][l]))==True and quote_handler(key_words, comment) == 'T'):
                key_flag = True
            elif(asterix_handler(key_words2, data_words)=='T'):
                    key_flag = True

        if(str(aw_clean)=='nan'):
            and_flag = True
        else:
            if set(data_words) & and_words2:
                and_flag = True
            elif(bool(re.search('"',df_tx['AndWords'][l]))==True and quote_handler(and_words, comment) == 'T'):
                and_flag = True
            elif(asterix_handler(and_words2, data_words)=='T'):
                and_flag = True

        if(str(nw_clean) == 'nan'):
            not_flag = False
        else:
            if set(data_words) & not_words2:
                not_flag = True
            elif(bool(re.search('"',df_tx['NotWords'][l]))==True and quote_handler(not_words, comment) == 'T'):
                not_flag = True
            elif(asterix_handler(not_words2, data_words)=='T'):
                not_flag = True

        if(key_flag == True and and_flag == True and not_flag == False):
            output.append(str(df_tx['Category2'][l]))
            temp = {'slno': [data.iloc[k,0]], 'text': [data.iloc[k,1].strip()], 'Category': [df_tx['Category2'][l]]}
            mapped_data = pd.concat([mapped_data,pd.DataFrame(temp)], sort = False)

#output mapped data
mapped_data = mapped_data[['slno', 'text', 'Category']]

mapped_data.to_csv("Path here/mapped_data.csv",index = False)

Final output looks like this:


Counting Number of Parameters in Neural Networks Note: This is an academics based problem.

So in a recent in-class quiz, we were asked that if we have an input layer consisting of 20 nodes along with 2 hidden layers (one of size 10 and the other of size 5), what will the total number of parameters in this network? How can we compute this?

Additionally, how do we know what shapes are they weights of? How can we determine which activation functions are suitable for such a neural network?

My idea was that (20*10) + (10*5) + (biases = 10+5) = 265. So 265 should be the number of parameters. For shapes/activation functions, from what I understand, it just depends on the data, no? Couldn't think of any way to directly predict it from this limited information

Actiavation function isnt a parameter.

But here is general formula for counting weghts:

Suppose for neural network with two hidden layers, inputs dimension is "I", Hidden number of neurons in Layer 1 is "H1", Hidden number of neurons in Layer 2 is "H2" And number of outputs is "O"

weights = (I+1)*H1 +(H1+1)*H2 +(H2+1)*O


How to give rewards to actions in RL? I'm working on below reinforcement learning problem:
I have bottle of fix capacity (say 5 liters). At the bottom of bottle there is cock to remove
water. The distribution of removal of water is not fixed. we can remove any amount of water from bottle, i.e. any continuous value between [0, 5].

At the top of the bottle one tap is mounted to fill water in the bottle.  RL agent can fill [0, 1, 2, 3, 4] liters in the bottle. Initial bottle level is any value between [0, 5].

I want to train the agent in this environment to get optimal sequence of
actions such that bottle will not get empty and overflow which implies continuous supply of water demand.

Action space = [0, 1, 2, 3, 4] Discrete Space

Observation Space = [0, Capacity of Bottle] i.e. [0, 5] Continuous Space

Reward logic = if bottle empty due to action give negative rewards; if bottle overflow due to action give negative rewards

I have decided to use python to create an environment.

I'm new to RL. I'm not aware about for which condition I have to give the rewards. Is my reward logic is correct or I have to change it?

I'm not aware about for which condition I have to give the rewards. Is my reward logic is correct or I have to change it?

The reward structure in reinforcement learning is a little bit flexible, provided it expresses the goal of your agent.

In your case, you want to avoid two bad scenarios in a continuous problem. There is also a good scenario - providing enough water - which overlaps with the negative consequences of not providing water on demand, but may be subtly different.

Your suggestion of granting a negative reward for spilled water, and a negative reward for unavailable water seems quite reasonable. You have a few choices within that that could make differences to the behaviour of an optimal agent. Given your problem description so far, I think these are a free choice for you:

You can decide on a fixed cost for an error e.g. $-1$ reward, or base the cost on the amount of water spilled or missing when required.
You can weight rewards for water spilled differently to water missing when required. This is the sort of thing that might be a business decision, based on different costs for the different kind of errors. If spillage costs are low, demand is high and impact of failed distribution is high, then the optimal policy may involve over-filling the tank and spilling water deliberately.
You can have a positive reward for the amount of water supplied. This is similar to having a negative reward for failed supply, but again might be weighted differently in a business scenario. If you are supplying water to paying clients, then you would care about profits here and have 3 types of reward covering the profits from selling water, the loss from wasted water and clean-up costs, and (the maybe more nebulous, but still estimated) loss from failing to meet customer demand when supply is not available.

You should bear in mind, that combined with the behaviour of the environment, these decisions may change what the optimal policy is. Until you define these values, there is no single optimal policy. As your problem is simple, there may be a lot of overlap between optimal policies, but choices you make in reward functions and how the environment behaves will result in differences perhaps in how willing the agent is to risk over-filling in some cases.

As the problem is continuous, the long-term reward that you want to optimise could become infinite. To avoid this, and make the problem tractable, you need to make one of the following choices:

Use a discount factor, $\gamma$, which can be from $0$ to $1$ (but not actually 1 for a continuous problem)
Formulate your goal as maximising average reward.

In your case I suggest using a highish discount factor, e.g. $\gamma = 0.99$, as most of the RL literature for value-based agents uses discount factors.

Finally, you may have an issue here:

  The distribution of removal of water is not fixed. we can remove any amount of water from bottle, i.e. any continuous value between [0, 5]

RL depends on the environment behaving as a Markov Decision Process. Which means the theory works best if there is a fixed distribution here. That could be a flat distribution in $[0,5]$, or it could be drawn from a complex real-world scenario, but the assumption from the theory is that the distribution of changes to state is a function of most recent state and action plus a stochastic element.

When you simulate your problem in order to train the agent, you will have to use an actual distribution. The agent will then optimise against the scenarios it experiences. Depending on what you simulate, it may do poorly in other situations. However, provided you don't take this to extremes, the agent should learn to optimise over a broad range of states, and many reasonable distributions of the removal of water.


Need for LIME explainer Is it possible to train a LIME explainer for a binary classfier on a dataset without labels?
I need to understand what is the value of storing a  LIME explainer object trained on the same data used to train the model.
In general, does it make sense to keep a trained LIME explainer around to generate explanations during production or is it better to train the LIME explainer on production data whenever is needed?
Another question. If I train a LIME explainer on training data and I use it with test data, does the LIME explainer suffers from data shift?

Answering the first question:
Is it possible to train a LIME explainer for a binary classfier on a dataset without labels?
From https://christophm.github.io/interpretable-ml-book/lime.html
Local surrogate models are interpretable models that are used to explain individual predictions of black box machine learning models. - So, no, you cannot use Lime on binary dataset without labels, because you cannot train classification model in the first place.
So, you need labels - one approach would be to cluster your data and if the structure in your data will be found, assign clusters to the dataset, train model, use Lime => interpret results.
Second part:
In general, does it make sense to keep a trained LIME explainer around to generate explanations during production or is it better to train the LIME explainer on production data whenever is needed?
As far as I know, there is no clear prove on if you should use LIME trained on train model and then use it on validation dataset or if you can use LIME on the trained row themselves. I would train the LIME model on train dataset and corresponding model and would then use the same LIME in production to explain particular data.
Generally, this book covers all topics (or most of them) regarding interpretable ml. (Im pretty sure that this book has a paragpaph talking about use of trained explainer on test/train dataset, but I dont remember where exactly)
https://christophm.github.io/interpretable-ml-book/


What do you use to generate a dashboard in R? I need to generate periodic (daily, monthly) web analytics dashboard reports. They will be static and don't require interaction, so imagine a PDF file as the target output. The reports will mix tables and charts (mainly sparkline and bullet graphs created with ggplot2). Think Stephen Few/Perceptual Edge style dashboards, such as:

but applied to web analytics.

Any suggestions on what packages to use creating these dashboard reports?

My first intuition is to use R markdown and knitr, but perhaps you've found a better solution. I can't seem to find rich examples of dashboards generated from R.

I think that Shiny is an overkill in this situation and doesn't match your requirement of dashboard reports to be static. I guess, that your use of the term "dashboard" is a bit confusing, as some people might consider that it has more emphasis of interactivity (real-time dashboards), rather than information layout, as is my understanding (confirmed by the "static" requirement).

My recommendation to you is to use R Markdown and knitr, especially since these packages have much lower learning curve than Shiny. Moreover, I have recently run across an R package, which, in my view, ideally suits your requirement of embedding small charts/plots in a report, as presented on your picture above. This package generates static or dynamic graphical tables and is called sparkTable (http://cran.r-project.org/web/packages/sparkTable). Its vignette is available here (there is no link to it on the package's home page): http://publik.tuwien.ac.at/files/PubDat_228663.pdf. Should you ever need some interactivity, sparkTable provides some via its simple interface to Shiny.


Predicting Player Position from pervious positions and LSTMs I am trying to use LSTMs to predict player positions in a field game. I try to overfit 8 slightly different time series. For this overfitting task I just use the positions of the players.
A data sample looks like this:

[Blue are the player trajectories corresponding to the green targets I want to predict, while yellow are the enemies and red is the ball]
When trying to overfit 8 future positions for a single player ( one single player is a single blue track with one green target), this is happening:

[Where red are the targets and blue are the predictions. So all predictions are basically a single point]
So the model is basically collapsing to the mean of the targets. Actually I dont know why this is happening.
My model looks like this:
model = Sequential()
model.add(LSTM(128 ,return_sequences=True))
model.add(LSTM(128 , return_sequences=False))
model.add(Dense(64,activation='relu'))
model.add(Dense(2, activation = 'linear') )
model.compile(loss='mae', optimizer='adam')

while my training data takes the form of input: [BatchSize, SequenceLength, FeatureLen] and output: [BatchSize, Target] which result in [8,20, 28] and [8,2] respectively. I also tried for the output the input shape of [8,1,2], yet this is not changing anything.
then I'll train the model with
loss = model.fit(inp, out,epochs =150, verbose=False)

Do you guys have an idea, why the LSTM is collapsing to the mean?

As I tried to predict the position without normalizing... the error was in the data. After normalizing the positions everything worked as expected.


Gradient Boosting RandomSearchCV or GridSearchCV In your algorithms, when you use Gradient Boosting, do you prefer RandomSearchCV or GridSearchCV in order to optimize your hyperparameters ?

Thanks for sharing your experience.

I think it depends on the size of your multi dimensional grid. If it is small, then you can afford to be exhaustive and do a grid search. But if it is very large, and your computation time for a grid search extends too much, then definitely go to a random search. In fact, with random search one can explore larger regions than with grid search, and that is an advantage.

In any case, for hyperparameters search there are two keys:

Monitor it while it is running, so that you can stop it when you are happy with the results. This is specially important for random search.
Split your data in three, train, cross validation and test. Evaluate the hyperparameter search in the cv set. Once finished, rank them by their performance there, and then take the best point and re-evaluate in test. It may happen that you do what is called "overfitting to cv set", in which case the performance in test set will deteriorate significantly vs the performance in cv set. If that happens, try with the second best, third best etc until the performance in test set is good. But take into account that the more points you take from the ranking, the higher the chance of "overfitting to test set".

Once you find the best regions, you can do either perform a more constrained there with either grid search or random search again.

Another option, which tends to work very well, is bayesian optimisation. Here the library that you use is important. In Python, after trying several which gave different problems, the best I found was skopt: https://scikit-optimize.github.io/


how often should one sample a dataset? I have some archive data that show the variation of a certain quantity X measured over 1-year intervals.

I'm now facing the problem of understanding how X correlates with another variable Y, that I can measure much more frequently, even daily.

So my question is, is it really necessary to collect data for Y so often? This seems such a waste of resources.

So what I would do is to collect data for Y only for 10% of the days of each year and I would select those days randomly. This way I'm sure that the observations are independent because the sample size is not too large and days have been selected randomly.

What do you think of that? Is that the approach you would follow as well?

In order to show a correlation between X and Y they should be aggregated over the same time period. Otherwise, there there may be seasonality or other features in Y that may have been averaged out in X. Is it possible re-index X to get monthly aggregates? I would not sub-sample Y. Find a common time period for both X and Y to make your comparisons.


Strange behavior with Adam optimizer when training for too long I'm trying to train a single perceptron (1000 input units, 1 output, no hidden layers) on 64 randomly generated data points. I'm using Pytorch using the Adam optimizer:

import torch
from torch.autograd import Variable

torch.manual_seed(545345)
N, D_in, D_out = 64, 1000, 1

x = Variable(torch.randn(N, D_in))
y = Variable(torch.randn(N, D_out))

model = torch.nn.Linear(D_in, D_out)
loss_fn = torch.nn.MSELoss(size_average=False)

optimizer = torch.optim.Adam(model.parameters())
for t in xrange(5000):
  y_pred = model(x)
  loss = loss_fn(y_pred, y)

  print(t, loss.data[0])

  optimizer.zero_grad()
  loss.backward()
  optimizer.step()

Initially, the loss quickly decreases, as expected:

(0, 91.74887084960938)
(1, 76.85824584960938)
(2, 63.434078216552734)
(3, 51.46927261352539)
(4, 40.942893981933594)
(5, 31.819372177124023)

Around 300 iterations, the error reaches near zero:

(300, 2.1734419819452455e-12)
(301, 1.90354676465887e-12)
(302, 2.3347573874232808e-12)

This goes on for a few thousand iterations. However, after training for too long, the error starts to increase again:

(4997, 0.002102422062307596)
(4998, 0.0020302983466535807)
(4999, 0.0017039275262504816)

Why is this happening?

This small instability at the end of convergence is a feature of Adam (and RMSProp) due to how it estimates mean gradient magnitudes over recent steps and divides by them.

One thing Adam does is maintain a rolling geometric mean of recent gradients and squares of the gradients. The squares of the gradients are used to divide (another rolling mean of) the current gradient to decide the current step. However, when your gradient becomes and stays very close to zero, this will make the squares of the gradient become so low that they either have large rounding errors or are effectively zero, which can introduce instability (for instance a long-term stable gradient in one dimension makes a relatively small step from $10^{-10}$ to $10^{-5}$ due to changes in other params), and the step size will start to jump around, before settling again.

This actually makes Adam less stable and worse for your problem than more basic gradient descent, assuming you want to get as numerically close to zero loss as calculations allow for your problem.

In practice on deep learning problems, you don't get this close to convergence (and for some regularisation techniques such as early stopping, you don't want to anyway), so it is usually not a practical concern on the types of problem that Adam was designed for.

You can actually see this occurring for RMSProp in a comparison of different optimisers (RMSProp is the black line - watch the very last steps just as it reaches the target):

You can make Adam more stable and able to get closer to true convergence by reducing the learning rate. E.g.

optimizer = torch.optim.Adam(model.parameters(), lr=1e-5)

It will take longer to optimise. Using lr=1e-5 you need to train for 20,000+ iterations before you see the instability and the instability is less dramatic, values hover around $10^{-7}$.


Predicting a numerical value based on past values and categorical attributes I have training data consisting of a time series of numerical values (e.g. a user activity metric on a website over a period of 100 days).
I also have some categorical attributes of the user (demographics, type of browser, location, etc)

What would be a good approach to predict for a new user (say on day 50) the future value of the activity metric, either as a numeric value or as a class (low, medium, high)?

I could think of the following approaches:

A. Perform time series regression or classification separately for most (or all) combinations of the initial categorical features.

or

B. Extract 4 to 10 different numerical and/or categorical features from time series (e.g. high, low, average, weekend levels, weekend to weekday ratio, etc), then performing a random forrest classification or regression on the entire set of features.

or

C. Something else?

Thank you!

Without having seen the data I would say that activity sensors may have a strong seasonal behavior. If that is the case time series analysis could give you some good results. Some basic before you head of in that direction, make sure to make your time series stationary before you use it in ARIMA. See for a good tutorial here.

If you want to forecast a categorical variable you will have to make sure to create sufficient covariates based on historical values. You could then proceed with a Random Forest.

In the end I would recommend you to consider how well you should be able to explain the outcome of the model. ARIMAX models are more tractable than Random Forest since the latter use bagging and averaging. So if you need to explain why your model gives a certain forecast, ARIMAX will probably be easier to explain.

Hope this helps.