Effects of slipperiness in OpenAI FrozenLake Environment I am trying to wrap my head around the effects of is_slippery in the open.ai FrozenLake-v0  environment.
From my results when is_slippery=True which is the default value it is much more difficult to solve the environment compared to when is_slippery=False. It takes roughly 10K iterations to solve when is_slippery=True compared to roughly 150 iterations when is_slippery=False.
I used the same cross-entropy method for both of them.
Now my issue is trying to understand the implementation from the repository and how they were able to model steps in the environment in such a way to mimic slipperiness.
This is the implementation from the repository with the different ways steps are taken based on is_slippery.
for row in range(nrow):
            for col in range(ncol):
                s = to_s(row, col)
                for a in range(4):
                    li = P[s][a]
                    letter = desc[row, col]
                    if letter in b'GH':
                        li.append((1.0, s, 0, True))
                    else:
                        if is_slippery:
                            for b in [(a - 1) % 4, a, (a + 1) % 4]:
                                li.append((
                                    1. / 3.,
                                    *update_probability_matrix(row, col, b)
                                ))
                        else:
                            li.append((
                                1., *update_probability_matrix(row, col, a)
                            ))

According the documentation in that repo:

However, the ice is slippery, so you won't always move in the direction you intend.

That implies is_slippery increases the noise. Specifically, the transition matrix will be effected. It makes sense that your model takes longer to learn when there is an increase in noise between intended and actual action.
There are appears to be many hard-coded numbers in that example. It appears a person with deep knowledge of that specific environment added constants to account for the increase in noise.


I have a 20.2 GB dataset which is entirely consisting of 1 file. Need help on how to open the file Whenever I am dealing with datasets that have weird extensions that I have not encountered before, I typically open them with notepad to see how the data looks like before using pandas to analyze them. But this dataset (link), that has an extension of .tr (stands for trace files I suppose), is not supported by notepad due to its huge size (that's what the error message says).

Which software should I use to open the trace file? Also is there a safe way to segment the file into smaller parts, as I typically upload my data to kaggle and work on their kernel.

I am not sure if this is the right place to post, if not please point me towards the correct place before removing the post.

This file should be a plain text file with n rows and m columns separated by a tabulator. Here is another much smaller file which they offer for download and claim that it has the same data format.

Since the file is too big to fit in memory, you can use the data format HDF5. This will allow you to load slices of your dataset without having to load the whole file into memory. However, first, you need to convert the file to a .h5 file. This can be done with h5py.

I put the following code together right now. It reads the text-file line by line and writes each line to the .h5-file. It works fine on the smaller file which they offer for download. I haven't tested it on a larger file, however. If you try it out, let me know how it goes.

import h5py
import numpy as np

N_COLS = 3  # adjust to your number of columns

n_rows_dataset = 1000

file = h5py.File('/path/to/your/output.h5', 'w')
dataset = file.create_dataset(
    'my_dataset',
    (n_rows_dataset, N_COLS),
    chunks=(5, N_COLS),
    maxshape=(None, N_COLS))

current_row = 0
# this does not load the whole file but creates an iterator instead
for line in open("/path/to/the/large/file"):
    if current_row == n_rows_dataset-1:
        n_rows_dataset = n_rows_dataset + 1000
        dataset.resize((n_rows_dataset, N_COLS))
    dset[current_row] = np.fromstring(line, sep=" ", dtype=np.float64)
    current_row += 1

Note: You can find out the number of colums of your dataset by printing the first lines of your file:

i = 0
for line in open("/path/to/the/large/file"):
    print(line)
    i += 1
    if i >= 5:
        break


How to deal with item belonging to more than one category So I'm training a classification task which takes as input some description of the state of a 2x2x2 rubix cube and outputs the optimal move to take. And a potential problem I noticed is that in many states more than one move is optimal. In particular, only about 46% of states have only one optimal move, and 24% have 2, 12% have three, etc. So I have a few choices.

The options I thought of were

have each data point choose an optimal move at random
Do cross-entropy minimization with the data point containing the same probability for each optimal move. i.e (0.5,0.5,0,0,0,0,0,0,0) if the first two are optimal
Discard states with more than one optimal move (this seems really bad)

What is the standard practice? Also, is there a difference between 1 and 2?

If necessary you may assume that I'm using a neural network model

There are multiple ways you can do it. The first two options you propose would be valid but I'm not sure if the log-loss allows you to use probabilities as target as opposed to just binary labels, but if that is the case they will work for what you want. I have another suggestion. Let's say there are at any point n possible moves available (In case of a 2x2x2 there are three dimensions and in every dimension 4 moves I think, so that would be 12 options) you could train 12 classifiers or in the case of a neural networks just twelve sigmoids in the output layer. Then you can represent every move as a probability, represent the training targets as a vector of 1s and 0s with 1 as being an optimal move and then during classification pick the one with the highest probability.

An extension of your first option would be to add all the optimal moves as state. Say move 2 and 4 are optimal you could just add the same state twice with different labels.

EDIT: That said, I think your second option is the best one if the math works out (I'm uncertain about this and don't have time to figure it out right now) but the others will work as well.

EDIT2: According to Neil Slater the second option will work if it's implemented properly and I think that is the cleanest solution so I would go for that if possible, otherwise I would go for either go for the multiple samples for multiple optimal moves or the 12-way binary classification.


The sum of probabilities is more than 1 I have CNN architecture for object detection ( one object in image ) in KERAS.
It has 22 Convolutional layers ( layer includes max pool , LeakyRelu and Batchnorm )

The last layes are following

model.add(Flatten())
model.add(Dense(6))

Basicly i want to output vector that reresents ( probability that is is ObjECT , probability that it isnt , x , y , xmax, ymax )

I am feeding it data that has one object in it and labels

( 1, 0, xmin , ymin , xmax , ymax )

This has around 0.60% accuracy. However the sum of (isOBject + isntObject ) is greater than 1.

[[   1.02664185    0.34075582  231.69602966  241.59547424  280.6262207
   266.35855103]]

Hos is this possible.

Also when i feed network another data that does NOT have object in it and labl them

( 0, 1 , 0, 0, 0, 0 ).

I get 10% accuracy. How can it be? It has more data to learn from and added false data on top of that.

I appreciate all tips and tricks / explanation, basicly all help. Thanks

You need to use appropriate activations. If you were using softmax for those two components, they'd be constrained to sum to one. LeakyReLU not only doesn't impose this constraint: it can output any real value, so it isn't even constrained to [0,1] for those components individually.


Performing machine learning on small datasets As a beginner at machine learning, I wanted to work on a small project in which the dataset has only 80 rows and 5 columns. The dataset I am working with is related to a medical condition, with 4 columns as biomarkers, and the 5th column indicates whether the row (a patient) has the condition or not.
 So far, I have fitted the following 5 models (with accuracy and MCC scores):

KNN (Accuracy: 43.5%, MCC:-0.164)
Logistic Regression (Accuracy: 65.2%, MCC: .312)
SVM (Accuracy: 60.9%, MCC: .214
Random Forest (Accuracy: 86.95%, MCC: .769)
Decision trees (Accuracy: 65.2%, MCC: .312)

I have used 5-fold cross validation to prevent overfitting, and yet most of my models are underperforming. I was also considering ensembling and bootstrapping, but with these lacking results, I am not sure how effective they would be. Do you have any tips concerning either:

Better algorithms for small datasets
Improvements I could make on the algorithms I have so far
Another method (e.g. Regularization)

With 5-fold cross validation, in each fold you're reducing your training dataset to 64 observations and evaluating against 16 observations. Assuming your data is balanced and you're stratifying folds, you're only giving your models 32 observations from each class to learn from, and misclassification of a single test set observation results in a 6.25% point change in accuracy for that fold. Even if it's classified correctly in the other folds, that single misclassification will still have a 1.25% point change on the across-folds average. Yikes. It should be no surprise your models aren't performing well: not only do they not have much data to train on, but your evaluation methodology is extremely unforgiving.

So yeah, your data is pretty darn small. Off the top of my head, I can think of a handful of general strategies you can potentially use to address this:

Use more of your data for training. Instead of 5-fold CV, try leave-one-out or bootstrap.
Generate fake data. This probably won't get you very far, but it's at least an option. Check out the SMOTE algorithm.
Transfer learning. Depending on what you're doing, it might be possible to leverage a pre-trained model and then tweak it slightly to suit your needs. If this is an option it can be extremely powerful, but chances are this won't be something you can reasonably pursue. Anyway, here's an article demonstrating this on another small data medical example: a pre-trained general purpose image classifier was trained to detect cancer from just 600 images! https://arxiv.org/abs/1711.10752
Go Bayesian. Bayesian methods allow you to incorporate outside information as a "prior belief". If you have subject matter expertise (or better yet, citations) that you can use to set your expectations for values of the model parameters or hyperparameters, bayesian methods will allow you to incorporate that information explicitly, and this can in turn help your model find good parameters faster since it doesn't need to learn everything about your problem directly from the little data available to it. If you're not careful, this can result in you giving yourself the model you want to have, in which case your evaluation metrics may not be as informative as you think they are. Here there be dragons: this approach is powerful, but it can be hard to do correctly, and playing with the prior is basically an invitation for people to be skeptical of your methods even if what you're doing is sound.
Get more data. I'm guessing this isn't an option or you would have done so already. But if there's a chance it's out there: go find it. Try poking around the literature associated with this condition, maybe you'll get lucky and find a public dataset. If you're feeling bold, you could try emailing other researchers and just ask them politely if you can use their data.


Multilabel image classification: is it necessary to have traning data for each combination of labels? I want to train a CNN for a multilabel image classification task using keras. However I am not sure how to prepare my tranining data. More specifically, I am wondering if I need training images that show a combination of two or more labels or if it is sufficient to train the network on single labels and it will then be able to detect multiple labels within an image.

I think my question becomes clearer with an example:

Say I am using the dog vs cat classification dataset and I want to build a model that is able to classify images as either being a dog or a cat or seeing both animals in one image. In this case, do I need to train the model with images showing cats, dogs AND images that show both in one image or is it sufficient to only have training images that only display cats and dogs?

Both cases will work. The point that you have to consider is that you shouldn't use Softmax activation in the last layer because your classes have intersection and they are not mutually exclusive. You should employ Sigmoid activation function and there should be two of them which each of them can be one or zero as the output vector. Consider that in this case you should not compare each activation to the other. Each one shows the probability of existence of the corresponding object, cat or dog.

For your case I suggest you to provide images which don't have either and in some cases have both. The former is more important. Because if you don't provide negative labels, not existing, the net always try to make something out of nothing. Means the net always try to flag similar patters as cats or dogs.


Machine Learning Analysis for Redaction Purposes of Personally Identifying Information from Open Text Fields Let's say that I wanted to use machine learning to find and redact personally identifying information (PII) from millions of records with open text fields.

Let's also say the PII could include a wide array of information categories like full names, dates of birth, places of residence, driver's license numbers, passports, family member names, etc.

Let's not forget to mention that thousands upon thousands of collectors were used to input this data too, which introduces an additional layer of complication (e.g. inconsistent reporting, nuanced terminology, abbreviations, spelling errors).

And finally, let's say that this data is meant to be released to the public so the risks of re-identification for nefarious purposes are very high.

Given that background, how feasible would it be to apply machine learning here? What important limitations/considerations should be taken during this process? Can machine learning address the problem of PII in this case? Why or why not?

NER

Basically the problem you have is Named Entity Recognition (NER).

This is a classic NLP problem and there are many machine learning approaches to it. Some recurring neural networks can be used, or the go-to method conditional random fields.

There are different techniques, usually in training, to overcome the issues (inconsistent reporting, nuanced terminology, abbreviations, spelling errors) that you mentioned.

Consideration

One important thing you need to consider is accuracy vs recall. You can choose to optimize one or the other, or both.

Accuracy = you try to redact something only if it really is some personal information.

Recall = you try to not miss any personal information.

Let say you have the sentence "Hello, my name is John, and my social security number is 123456789."

If you redact the sentence to: "Hello, -- ---- -- ----, --- -- ----- ------ ------ -- --------." You get really high recall (all personal information is gone)! But your accuracy is low, you redacted more than you should.

All this can be leveraged during prediction time. The probability score from your NER can make you optimize or the other.

Of course, this begs the questions: Why not redact EVERYTHING and so that 100% of personal information is not leaked?


Working with two dates fields in table I am working with dates.

My fields are in order table:

Start date
close date
order no.

I have to show open orders in each month.

I have created relationship with date table by start date.
I need to relations like:

If start date is in this month and before this month and end date is this month means the order is closed in this month itself
it should count 0
If start date is in this month or before this month and close in month after this month
it should count 1

Now i have created measure like

if(month(startdate)<=month(date),
  if(month(closedate)>month(date),1,0),0)

It assigns 1 for the condition and then I use sum of all ones in visual for every month. Now this works fine with current start date but
for order with startdate before this month its not showing the data
as start date connected to date field in calendar table, when i put this in visual. It not showing the start date for previous months.
For example: If an order having start date in 7th month and end date in 9th month it should be in, 7th month and as well as in 8th month.
Right now its showing only in 7th month just because of relationship, I guess.

Also, its not evaluating the year changes. like its not capable for comparing data for December '19 and January '20

I used this post to learn how to model something similar in PowerBI/DAX. It summarizes how to use generate to create a calculated table that has one row per date and order that shows the status.  When the users creates relationships to other tables (dates and statuses in this example) then it’s relatively straightforward to get information about number of open orders or some other information you included in the calculated table (I used contract price in my set up).


Isn't one-hot encoding a waste of information? I was just playing around with one-hot representations of features and thought of the following:

Say that we're having 4 categories for a given feature (e.g. fruit) {Apple, Orange, Pear, Melon}. In this case the one-hot encoding would yield:

Apple:  [1 0 0 0]
Orange: [0 1 0 0]
Pear:   [0 0 1 0]
Melon:  [0 0 0 1]

The above means that we quadruple the feature space as we go from having one feature to having four.

This looks like it's wasting a few bits, as we can represent 4 values with $\log_{2}4=2$ bits/features:

Apple:  [0 0]
Orange: [0 1]
Pear:   [1 0]
Melon:  [1 1]

Would there be a problem with this representation in any of the most common machine learning models?

Good idea but...

You encode not just to transform from categorical to numerical features but to give that information to your model.

Let's say that you do that and feed it through a linear model to try to predict the price. Let's say Pear is really expensive(500€) and Melon cheap (1€).

Your model coefficients with one hot encoding will be:

$price = 500 * Pear[0,1] + 1 * Melon[0,1]$

If you do your encoding the linear combination won't work. What will be the coefficients?

One could argue that with decision trees this won't happen since it can make splits... but it would have to make two splits before determining if it is a melon (a greedy decision tree won´t do it) so again you will lose computational power here too.

You could try to run the experiments and see if this is your result. In the end, this should be science and one can do experiments to prove a hypothesis.

On the other hand, Yes,  One Hot Encoding increases the computational time and memory easily since you are creating a lot of features from one and if it has high cardinality you can end up with a lot of features.


What is whole word masking in the recent BERT model? I was checking BERT GitHub page and noticed that there are new models built from a new training technique called "whole word masking". Here is a snippet describing it:

  In the original pre-processing code, we randomly select WordPiece tokens to mask. For example:

Input Text: the man jumped up , put his basket on phil ##am ##mon ' s head

Original Masked Input: [MASK] man [MASK] up , put his [MASK] on phil [MASK] ##mon ' s head

  The new technique is called Whole Word Masking. In this case, we always mask all of the the tokens corresponding to a word at once. The overall masking rate remains the same.

Whole Word Masked Input: the man [MASK] up , put his basket on [MASK] [MASK] [MASK] ' s head

I can't understand "we always mask all of the the tokens corresponding to a word at once". "jumped", "phil", "##am", and "##mon" are masked and I am not sure how these tokens are related.

phil ##am #mon is a subword encoding of the single word “philammon” into 3 tokens. The comment just means that they mask words as opposed to tokens by taking into account subword encoding.

For more on subword encodings take a look at the slides from cs224, especially Byte Pair Encoding, from the Feb 14 subwords lecture at http://web.stanford.edu/class/cs224n/index.html#schedule.


Predicting time series, with few historical samples based on similar series I'm trying to build a model with Keras to predict the time series of a sensor, based on its type and historic data of sensors of the same type.

The figure below shows 3 time series, generated from 3 sensors of the same type, the green dashed line is the new sensor data and the vertical line is where the data for the new sensor end.

I've tried writing an LSTM network, that returns the hidden state output for each input time step, while the target was the values for each timestamp. Then trying to predict the new time series giving the model a few points of the sensor history data. With no luck :(

So I'm guessing I'm walking on the wrong path. What are the options of predicting a time series with just a few historical samples based on the history of other time series of the same type?

Any help / reference / video would be appericiated

I'd suggest statistical forecasting techniques such as ARIMA or ES, but those models usually cannot generalize well across timeseries, so you'd need one for each timeseries

A good starter for using LSTM for forecasting is here - https://www.tensorflow.org/tutorials/structured_data/time_series. But if you don't have enough data, NNs would like result in poor test results.

For your case, I'd suggest you trying a regression approach- structuring your timeseries into a regression features format. After that you can use regression models from sklearn, starting with linear and moving to more complex ones. Because you have less data, you might want to explore less complex models first to prevent overfitting. For the features, for example, you could create lag features (what was the value of your signal two timestep back, what was the mean/std of the past 2 timesteps, what was the maximum in the last 6 timesteps and so on). Look up any kaggle competition on timeseries forecasting if you want to refer code or get specific ideas on feature extraction


Machine Learning & Partial Differential Equations Are there any algorithms which were developed using partial differential equations for tackling some of the machine learning problems? Most works I see online are in the field of computer vision and a few bizarre ones in topic modelling. But just curious if someone has used or seen it being used for some decision making process or classification problems?

Neil is correct. There are partial derivatives evwrywhere in gradient computation for machine learning models training.

For instance you can look at the gradient descent method used in the backpropagation method for a neural network. The course from AndrewNg on coursera describes it very well.


Example of 1D ConvNet filter I understand Conv2D filters. I think I understand Conv1D filters as well but have not seen any examples of the filters like what we have in 2D filters, e.g. Sobel as shown in the following image. Are there any standard one dimensional filters?

Sobel 2D Filter example

Images are two dimensional signals. The use of Conv1D is for one dimensional signals like voice and sound. CNNs are good for these signals too because of taking care of local input patterns. Definitely there are standard one dimensional filters which are most used in signal processing like high pass filters and low pass filters which are so much popular. In order to show you an example take a look at the following figure which illustrates the convolution of two signals:


Using PCA to cluster multidimensional data (RFM variables) So i am performing k-means clustering on RFM variables (Recency, Frequency, Monetary). The RFM variables are in the form of quantiles (1-4). I used PCA and found the PCA components. I then used the elbow method to find the optimal number of clusters and then I use it in the k-means algorithm.
Could anyone guide me if this is a correct method? Further, the clusters I get range on the graph, their axis ranges from -3 to 3 and I am not entirely sure why it ranges from that way.

Judging from the plot, there are no clusters.

K-means requires continuous variables to work well. The data you have has discrete steps (which causes the grid pattern in your plot).

There is no benefit of using PCA here. Use it only for visualization. The scale -3:3 that you don't understand is from PCA. So you probably have not understood PCA enough either.


Explanation for Why Logistic Regression can be so Accurate in Sentiment Classification? My question is about how a logistic regression model performs so accurately. In some exploratory experimentation, I compared a logistic regression model against a long short term memory recurrent neural network for binary classification of consumer reviews (Positive or Negative)(25,000 reviews). The Logistic Regression model performed better with an accuracy of 88.0% and a recall, precision, and f-score of 0.88. The Recurrent Neural Net performed with an accuracy of 81.1%, a recall of 0.83, a precision of 0.80, and an f-score of 0.81. Perhaps my building of the network was flawed but I was still surprised to see that the Logistic Regression had such a high accuracy.

Some people have this preconceived notion that logistic regression shouldn't perform well because it is too simple. While that might be true for certain types of complex problems, the reality is that logistic regression performs relatively well in many different scenarios.
It's difficult to give a definitive answer as to why a model performs well without actually seeing the data and the model you've constructed.
That being said, it could be that the problem you're trying to solve is not very difficult so even a simple logistic regression model yields good results. Another thing to consider is that while 88% accuracy sounds good, it might actually only be OK for this type of problem (and dataset) - it would be informative to try other non-neural-network based classifiers and see how well they perform.


Sci-kit Pipeline and GridsearchCV returns indexError: too many indices for array I'm trying to get to grips with sci-kit learn for some simple machine learning projects but I'm coming unstuck with Pipelines and wonder what I've done wrong...

I'm trying to work through a tutorial on Kaggle

Here's my code:

import pandas as pd

train = pd.read_csv(local path to training data) train_labels =
pd.read_csv(local path to labels)

from sklearn.decomposition import PCA
from sklearn.svm import LinearSVC
from sklearn.grid_search import GridSearchCV

pca = PCA()
clf = LinearSVC()

n_components = arange(1, 39)
loss = ['l1','l2']
penalty = ['l1','l2']
C = arange(0, 1, .1)
whiten = [True, False]

from sklearn.pipeline import Pipeline

#set up pipeline
pipe = Pipeline(steps=[('pca', pca), ('clf', clf)])

#set up GridsearchCV
estimator = GridSearchCV(pipe, dict(pca__n_components = n_components, pca__whiten = whiten, clf__loss = loss, clf__penalty = penalty, clf__C = C))

> estimator

Returns:

GridSearchCV(cv=None,
       estimator=Pipeline(steps=[('pca', PCA(copy=True, n_components=None, whiten=False)), ('clf', LinearSVC(C=1.0, class_weight=None, dual=True, fit_intercept=True,
     intercept_scaling=1, loss='l2', multi_class='ovr', penalty='l2',
     random_state=None, tol=0.0001, verbose=0))]),
       fit_params={}, iid=True, loss_func=None, n_jobs=1,
       param_grid={'clf__penalty': ['l1', 'l2'], 'clf__loss': ['l1', 'l2'], 'clf__C': array([ 0. ,  0.1,  0.2,  0.3,  0.4,  0.5,  0.6,  0.7,  0.8,  0.9]), 'pca__n_components': array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
       18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34,
       35, 36, 37, 38]), 'pca__whiten': [True, False]},
       pre_dispatch='2*n_jobs', refit=True, score_func=None, scoring=None,
       verbose=0)

But when I try to train data:

estimator.fit(train, train_labels)
The error is:

    428         for test_fold_idx, per_label_splits in enumerate(zip(*per_label_cvs)):
    429             for label, (_, test_split) in zip(unique_labels, per_label_splits):
--> 430                 label_test_folds = test_folds[y == label]
    431                 # the test split can be too big because we used
    432                 # KFold(max(c, self.n_folds), self.n_folds) instead of

IndexError: too many indices for array

Can anyone point me in the right direction?

It turns out that the Pandas dataframe is the wrong shape.

estimator.fit(train.values, train_labels[0].values)

works, although I also had to drop the penalty term.


How to set weights in multi-class classification in xgboost for imbalanced data? From this post, I know you can set scale_pos_weight for an imbalanced dataset.  However, for the multi-classification problem in the imbalanced dataset, I don't quite understand how to set the weight parameter in dmatrix.

How can I use XGBoost for an imbalanced dataset in a multi-classification problem?

As you say, scale_pos_weight works for two classes (binary classification). weight can be used for three or more classes. The parameter goes into the xgb.DMatrix function and must contain one value for each observation.
Example:
library(xgboost)
data(iris)

# We'll predict Species
label = as.integer(iris$Species)-1
iris$Species = NULL

# Split the data for training and testing (75/25 split)
n = nrow(iris)
train.index = sample(n,floor(0.75*n))

# For example, pick a weight of 1.5 for label "0", 1.0 for the other Species
weights = sapply(label[train.index], function(x) {ifelse(x == 0, 1.5, 1.0)})

# Train the data using weights
xgb.train = xgb.DMatrix(data=as.matrix(iris[train.index,]), label=label[train.index], weight = weights)

A similar question can be found here.


How to select 'cutoff' of classifier probability I have recently used xgboost to conduct binary classification in an nlp problem. The idea was to identify if a particular article belonged to an author or not, pretty standard exercise.
The results are outputted as a probability between 0 and 1, and there is the ocasional article that is completely misclassified.
I would like to know if there is a statistical approach that gives me a confidence interval for the probability outputs (for example if I consider all articles with prediction of 0.4 I will get 95% of the articles that belong to the author), or something that helps me make decisions regarding the cut-offs.

What you're looking for is something along the line of an ROC curve:
Using the threshold as a decision parameter, you can observe the trade-off between FPR (False Positive Rate: how many of the articles not belonging to the author will be correctly classified) and TPR (True Positive Rate, aka recall: how many of the articles which are really by the author will be classified as such).
When the parameter is at one end, you'll classify all documents as belonging to the author (100% recall, but pretty bad precision), and at the other hand, you'll have 100% precision but pretty bad recall.
The plot will allow you to decide on a value that satisfies your requirements (i.e. how much will your precision suffer when you want 95% recall). You can select it based on your desired value in one metric (e.g. 95% recall), but really I'd just plot it and have a look. You can do it in SKLearn with plot_roc_curve.


When to remove correlated variables Can somebody please suggest what is the correct stage to remove correlated variables before feature engineering or after feature engineering ?

You do not want to remove all correlated variables. It is only when the correlation is so strong that they do not convey extra information. This is both a function of the strength of correlation, how much data you have and whether any small difference between correlated variables tell you something about the outcome, after all.

The first two you can tell before you do any model, the final one not. So, it may be very reasonable to remove variables based on the combination of the first two considerations (i.e. even if the extra variables may in principle contain some useful information, you would not be able to tell given the strength of correlation and how much data you have) before you do any modelling/feature engineering. The final point can really only be assessed after doing some modelling.


How to build a symmetric similarity model on top of embeddings? I have two equal length vectors that come out of two identical embedding layers.

I want to calculate their similarity, and I don't trust the embedding layer enough to just use dot product (e.g. it's plausible that different coordinates are dependent wrt overall similarity). I want to learn this using examples of good and bad pairs, without actually learning the initial embedding.

What I'd like to do is to somehow combine the two vectors using another layer, and then connect this layer to an output layer to get the final decision (similar/not similar).

The trivial way is to add another layer, and fully connect the two concatenated embedding vectors to this new layer. The downside is that the model is not symmetric, which makes the search space bigger than it should be.

Is there a better way?

Ideas I have so far:

Cross product the two vectors, then do the learning on the $n \times n$ output (easy enough, but might have too many weights to learn).
Create two symmetrical neural nets, where the weights are symmetrical (as if mirror image of each other). Intuitively this can allow arbitrary combinations of coordinates from both sides to feed the next layer (how can equalizing the weights be done in keras?).

Is there a better way?

Bonus points: Beyond symmetry, how does one instigate transitive and reflexive properties? By just adding a ton of trivial $(x, x)$ examples?

That's an interesting question ! Surprisingly, I never read specific scientific papers on this particular problem. Here you want to learn a symmetric function that transforms two input vectors into a scalar. In a more general setting, we can find in the literature some needs for permutation invariant functions (see the Set Transformer and the Aggregation Schemes for Graph Neural Network ). Basically, you have two options:

As you pointed out, you can take whatever non symmetric function, such as concatenation + feed forward network and train it with (x,y) and (y,x) pairs. But there might be some smarter way...
Design a permutation invariant architecture. Here your chance is that you only have 2 embeddings $e_i$ and $e_j$. Sadly, you say they can't be trusted for their mutual dot product. Then, I might suggest to apply some linear/nonlinear transformation to your embeddings i.e. define a trainable matrix $Q$, and compute the dot product in the new space $v_i = Qe_i$ and $v_j = Qe_j$. By training $Q$ with example pairs $(e_i,e_j)$, you'll find a suitable transformation of the embeddings such that the dot product $v_i \cdot v_j$ fit the true similarity. You can of course use a more complex transformation (multiple non linear transformations for example). The key points is that using the same parameters for transforming both embeddings ensure the symmetry.

Bonus points: I'm not sure of your question, but my proposition in (2) might naturally deal with transitivity and reflexivity.


Shaping data for ConvLSTM for many-to-one image model Ultimately, I am trying to obtain a binary segmentation mask for an image sequence.

I have n number of image sequences, each with 500 greyscale images of size 256px by 400px. Each of these sequences is matched to a single segmentation mask, or image of shape 256px by 400px. This gives:

input shape : (n, 500, 256, 400, 1)
output shape: (n,   1, 256, 400, 1)

Understandably, and as explained in this blog post it is necessary to reshape my input array into sequence fragments (time steps) because LSTMs prefer sequences smaller than sequence_length = 400 (my GPU cant handle anything larger than 100 anyway). So, with this in mind, I have chosen a sequence "fragment" length of 50, or 50 time steps. Without overlap this brings the shape of a single input sample:

from:      (500, 256, 400, 1)
to  : ( 10,  50, 256, 400, 1)

In my mind it made sense to feed a single sample through the ConvLSTM2D at a time with a data generator, to have:

input shape : (10, 50, 256, 400, 1)
output shape: ( 1,  1, 256, 400, 1)

I thought this meant that the network would associate the 500 images with a single output image ie. remembering the state across the 10 fragment sequences of 50 images each in order to result in a single image output and computing the loss function only when the network has seen all 500 images for example. It did not make complete sense to me when I did it and now I have proven it to make even less sense because I have gotten the following error:

 ValueError: Input arrays should have the same number of samples as target arrays. Found 10 input samples and 1 target samples.

So I am thus a bit stuck with a flawed understanding of Keras's LSTMS and how to get this ConvLSTM2D to manage these image sequences in the way that I want ie. how I might be able to get a many-images (of a fairly long sequence) to one-image model to work. I can simply duplicate the output image for each of the sequence fragments so that there is a dimension match but this is not the problem I am trying to solve because the segmented image is a summary across the entire sequence, and not across sequence fragments.

The problem was for each ConvLSTM layer I was using keep_dims = True which means that the number of dimensions in the input is reflected in the ouput. To obtain a many to one model I just needed to set keep_dims = False. I was then able to manipulate the size of the output using dense layers and convolutions. In this way I could have a video sequence as an input and a single 2D image as the output and I was able to obtain the relationship:

input shape : (10, 50, 256, 400, 1)
output shape:     (10, 256, 400, 1)

Where I was able to get a 2D segmentation for each fragment in the video sequence.

It is not exactlly what I needed, but was much better than getting a segmentation result for every frame in the video sequence.


Implementing sklearn.linear_model.SGDClassifier using python I have an excel file that contains details related to determining the quality of a wine and I want to implement the linear model concept using the function sklearn.linear_model.SGDClassifier(SVM => Hinge loss) and (Logarithmic regression =>log loss) using python. I learned the basics about these function through the scikit learn website and I am not able to implement the model using excel file. I am very new to python and machine learning and I finding it hard to implement the model. I opened the excel file in python and tried to take two columns [randomly] from the file and use that as an input to call the fit function available in the model. But, I got an error stating Unknown label type: array. I tried a couple of other methods too, but, nothing worked. Can someone guide me with the implementation process?

from xlrd import open_workbook
from sklearn import linear_model
i = 0
fa = []
ph = []

book = open_workbook('F:/BIG DATA/winequality.xlsx')
sheet = book.sheet_by_name('Sheet1')
num_rows = sheet.nrows - 1
num_cols = sheet.ncols - 1
curr_row = 0
while curr_row <num_rows:
    curr_row += 1
    cell_val = sheet.cell_value(curr_row,0)
    cell_val1 = sheet.cell_value(curr_row,10)

    fa.append([float(cell_val),float(cell_val1)])
    cell_val2 = sheet.cell_value(curr_row,8)
    ph.append(float(cell_val2))

model = linear_model.SGDClassifier()
print(model.fit(fa,ph))

The error message screenshot:

I think that this is the same issue as in this question.

The shape of $X$ must be (n_samples, n_features) as explained in the
SVC.fit docstring. A 1-d array is interpreted as a single sample (for
convenience when doing predictions on single samples). Reshape your $X$
to (n_samples, 1).

That means you should use numpy.reshape to reshape the $X$ column. If the data frame has n rows, you should use
X_new = X.reshape(n, 1)

Then use the fit method with $X_{new}$. Note: you probably don't need to do this if you use two or more $X$ columns for your model fitting.


Keras- LSTM answers different size Imagine situation, input is some text file and information are spread according to rows. I want to use rows as input of model.

Model is planned as LSTM with softmax as output layer. My problem is the output; I want to classify text file to some categories. So I imagine this like, I open text file, do necessary preprocessing and row by row feed into the model and after I have applied all the rows to the file, I tell softmax which category is right for this document. But I don't know how to make softmax layer "wait" until last row of file is processed by LSTM layers.

For example, I have file with 1000 rows and every row has 5 features. So my dataset is [1000x5] size but I don't have 1000 y I have 1 y which told me category of this document. Documents can be different size so I can't use whole document as input. I also don't see as option make y size 1000 where everything is category index.

When I think about this seems to me, when person implement LSTM from scratch, is possible to make LSTM run(save every state) and after several iterations perform softmax and than apply BTT or more suitable algorithm and perform weights update.

Every LSTM keras tutorial which I saw uses y which is the same shape as number of samples. Maybe I didn't see right ones.

Your dataset is not [1000x5], one of your documents is that size. LSTMs can deal with variable sized input directly or by padding (this is the case with Keras). In Keras you can use the parameter return_sequences as False in your LSTM, which means only after the last part of your sequence has been parsed it will transform it with a Softmax and run the classification. What I would do is pad your documents to a certain size, for example 1000 (though this is high for LSTMs) so that you will have a [n x 1000 x 5] feature set and a [n] target set. You can feed this to your model now.


Overfitting Naive Bayes My question is what are potential reasons for Naive Bayes to perform well on a train set but poorly on a test set?

I am working with a variation of the 20news dataset. The dataset has documents, which are represented as "bag of words" with no metadata. My goal is to classify each document into 1 of 20 labels. My error rate on training data is 20%, but my error rate on test data is 90% (for comparison, guessing yields 95% error rate). For some reason, my classifier is predicting class 16 for almost all documents in test set. In train set this problem does not occur. Furthermore, this issue persists with different train/test splits. I'm trying to figure out what I'm doing wrong.

Here are some things I've considered:

Is Naive Bayes overfitting to the training set? If Naive Bayes is implemented correctly, I don't think it should be overfitting like this on a task that it's considered appropriate for (text classification).
Is my train/test split bad? I've tried splitting the data in different ways, but it does not seem to make a difference. Right now I'm splitting the data by placing a random 90% sample of documents into the train set and the rest into the test set - separately for each label.
Problems with numerical accuracy? I've implemented the calculations in log probabilities, but in any case, I would expect that problem to manifest in train set as well.
Problems with unseen words? The fraction of unseen words relative to a particular newsgroup is the same, 20%, for both train and test sets.
Problems with Laplace smoothing? I wasn't sure what was the appropriate way to implement Laplace smoothing for this task, so I tried a variety of ways. To my surprise they all yielded very similar results.
Imbalanced dataset? Doesn't look like it. There are approximately as many unique words inside documents of label 16 as there are in other classes. Also the number of documents per each label is roughly even.

Edit: Turns out I had an implementation bug. I won't detail it here as it will be unlikely to help anyone with similar issues.

Let me try to answer your questions point by point. Perhaps you already solved your problem, but your questions are interesting and so perhaps other people can benefit from this discussion.

Is Naive Bayes overfitting to the training set? If Naive Bayes is implemented correctly, I don't think it should be overfitting like this on a task that it's considered appropriate for (text classification).

  Naive Bayes has shown to perform well on document classification, but that doesn't mean that it cannot overfit data. There is a difference between the task, document classification, and the data. Overfitting can happen even if Naive Bayes is implemented properly.

Is my train/test split bad? I've tried splitting the data in different ways, but it does not seem to make a difference. Right now I'm splitting the data by placing a random 90% sample of documents into the train set and the rest into the test set - separately for each label.

  It's good that you're keeping the class distribution the same in your training and test set. Are you using cross-validation? Perhaps try it because, even though it's rare, it might happen that you just get unlucky with your splitting due to some seed.

Problems with numerical accuracy? I've implemented the calculations in log probabilities, but in any case, I would expect that problem to manifest in train set as well.

  You're correct, if it was an issue, it would show in the training set as well.

Problems with unseen words? The fraction of unseen words relative to a particular newsgroup is the same, 20%, for both train and test sets.

  This doesn't seem to be the issue then. You could perhaps reduce that percentage by using stemming or lemmatization.

Problems with Laplace smoothing? I wasn't sure what was the appropriate way to implement Laplace smoothing for this task, so I tried a variety of ways. To my surprise they all yielded very similar results.

  Laplace smoothing is useful especially when you do not have a lot of data and you need to account for some uncertainty. For this dataset, this doesn't seem to be an issue, as shown by the similar results you've experienced.

Imbalanced dataset? Doesn't look like it. There are approximately as many unique words inside documents of label 16 as there are in other classes. Also the number of documents per each label is roughly even.

  Are the documents of the same lengths? Because perhaps label 16 simply contains documents that are larger and therefore have bigger word frequencies. They might also contain very common words. It would be interesting to look at a histogram of the words in each class. This could be very informative in understanding whether label 16 is very different from the others.


Representing different sets of data I am out of ideas in regards to plotting different sets of data ....

I have 10 sets of data ... Each set contains Actual Speed of vehicle and predicted Speed of Vehicle..

How can i combine all sets of data and show in a meaningful manner ?  I don't think I can simply average as they are absolute values (In one run , i was going 25mph and second run , i might be going 30 mph)//...

Is there any idea on how i can represent 10 sets of data in a meaningful manner like variance  ?

Any ideas are appreciated

There are two solutions to this problem, though I'm only going explain the second one because I think is the best option.
First option:
10 Scatterplots of speed of vehicle on the x-axis and predicted speed of vehicle on the y-axis. The only scenario I see this plot been useful is for clustering different types of vehicles with their speeds (if there actually are many vehicle models). Naturally you could also aggregate all data points in a single scatterplot and color-code the data set (ex. dataset1 = blue, dataset 2 = red, etc.)
Second option:
Calculate the difference of vehicle speed and predicted speed for each data point and then bind all the results into a single data table so that each column contains the difference between actual speed and predicted speed for each data set.
Then plot each column as a bar in a boxplot, here you will be able to describe all differences including signs for outliers and mean. As you might already know boxes represent standard deviation (variance) and the central line represents the mean. You can find more information about boxplots here:
Wikipedia
Sample code in R and ggplot2
dt1 <- data.frame(data = c(runif(20, min = -1, max = 1),
                       runif(20, min = -1, max = 1),
                       runif(20, min = -2, max = 1)),
              datasetName = c(rep("dataset1", 20),
                              rep("dataset2", 20),
                              rep("dataset3", 20)))

ggplot(dt1, aes(x = datasetName, y = data)) +
geom_boxplot(fill = "white", colour = "#3366FF") +
labs(x = "Dataset Name", y = "Difference Between Speed and Predicted Speed",  title = "Datasets distribution")

The resulting image should look something like this:


What is a better approach for cross-validation with time-related predictors I was a given a data set with different predictors about a store and the idea is to forecast the number of daily shoppers. The predictors are the weekday, time of the day (morning, afternoon, evening), week number, month, weather (humidity, dew point, temperature), holidays. The outcome variable is the number of visitors.

I want to build a regression model to predict the number of visitors using traditional machine learning algorithms such as random forests, SVM, and the like.

My main concern is how to validate this model using CV since some of the predictors are time-related. Plain vanilla CV cannot be performed here. In this question, they suggest a way to perform this but my problem is that I only have data from June 2015 to present.

My initial idea was the following:

train with data from June 2015-December 2015. Test with January
train with June 2015-January 2016. Test with February 2016.

Each time one month of data is added to the training data after having asses the error for that month. Then compute average performance.

My questions:

Is this approach reasonable or not?
If so, should I get rid of the month variable? Note that in a., for instance, I am testing with some data that belongs to different months that the one used for training. I mean, for the training I used data from June to December 2015, but I am testing for January 2016. Seasonality can be something I am missing.
How to validate such models in general?

Your approach 1) is correct, use n ordered data to predict n+1.
You will have to identify the correct window for prediction and a model not too flexible if you feel the amount of data you have is little.

Do not forget about feature engineering and data preparation. The seasonality you mention can be removed if you identity it correctly.

Validation of the model is done as usual, with a square loss function


What companies would be great for entry level data science/ machine learning programmers to help fight for a good cause? I am just starting to learn about machine learning, and I have taken some intro classes in MATLAB and Python. I want to know about fulfilling careers and fields programmers can pursue to help me determine my career direction. Thank you for your time!

If you want to join companies for a good cause then my opinion would be look for startups. Startups have broader definition of work and more freedom with data. You can search through their goals and find what you need.


Pound notation in Summation I was going through a paper comparing glove and word2vec. I came across the pound notation shown below. What does it mean when used like this?

The link for paper is here

In the third paragraph of the first section (page 1), they define $\#(w)$, $\#(w; c)$, and $\#(c)$. Quoting such paragraph:

  Word-context pairs are denoted as $(w; c)$, and $\#(w; c)$ counts all observations of $(w; c)$ from the corpus. We use $\#( w ) =  \sum_c \#(w; c)$ and $\#(c) = \sum_w \#(w; c)$ to refer to the count of occurrences of a word (context) in all word-context pairs. Either $\sum_c \#(c)$ or  $\sum_w \#(w)$ may represent the count of all word-context pairs.

So, considering the equation you cited:

$\#(w_i, c_j)$ counts the occurrences of a specific word $w_i$ over the context $c_j$;
and $\frac{\#(c_)}{\sum_w \#(w)}$ accounts for the the probability of having context $c_j$ in the dataset.


What is the difference between 8 filters twice and one 16 filters in convolutional layers at CNN? Why would we use two convolution layers in a row with same spec? E.g. in VGG16 first two layers, they used 3 x 3 filter with 64 depth twice. What is the difference if we use 3 x 3 filter with 128 depth once.

Each layer has a limited amount that it can transform the layer below it. There is one linear component (weighted sum of output of layer beneath it), and one non-linear component (typically ReLU).

It is in theory possible to approximate any function with a large enough single layer in a fully-connected network. However, a stack of similar smaller layers is more expressive using less resources. That means for the same number of parameters you have access to a more flexible function approximator. At some level of complexity for your target function, the cost (in terms of CPU time, data required and effort in training) of making a single layer wider is higher than the cost of stacking more, similar layers.

In addition, for a CNN, you have to worry about receptive field. Any feature map can only express values that the filter can "see" due to width of the kernel. As you add more layers, each kernel applied extends the width and height of the base image that the features in the last layer effectively calculate over. If you also have a fully-connected layer after the convolutional layer, then you can in theory compensate for a poor receptive field with a very large fully-connected layer - but then you are back to the first problem of wide network with more parameters than strictly necessary to learn the function.


Technique/Algorithm for product categorization Machine Learning I am going in a very abstract layer for the problem, but I think this problem might be a common one so posting it.

So, what I am looking for is any ML algorithm or Data Science technique is there to create a relationship factor between different sets of goods.

I guess, its used heavily in social media, but here I am talking w.r.t. product goods context. Like suppose if we sell a bunch of products together in a store, so in collection there will be many products that are always bought together.

So, accordingly there will be a relationship factor between different products i.e. some product that are always bought together will have a higher factor than the one that are bought together less frequently.

So, is there any approach where we can tackle this problem to get the factor between two products using any technique?

Edit : Narrowing it down to a universe where all items are having same factors and their relationship factor to each other totally depends on their occurance together.

As Sean Owen said, it's a wide problem so I can't be generic on this one. I'll just give you an example on how you could tackle this:

Let say you sell some products on a website. If you have 100 different products, you can represent each sale by a 100 dimensional vector, where each component is the count of ordered product. Example: sale1 = (2,0,1,0,...,0) means 2 item1 and 1 item3 sold. We use the same technique with words, and it is called bag of words. If you stop here, you only can perform statistical counts on individual product sales. So let's use matrix factorization.

If you sold products to 1000 people, then you can concat all your sales in a matrix of dimension 1000 X 100 where each row represent one sale and each column represent one product. Using decomposition algorithm such as SVD, you can find out singular values in your data that will show you latent factors in your sales (basically trends or connections between your products).


What would be the best way to map similar ngrams I'm trying to map similar ngrams using Wordnet and synsets. For example: elder brother and older sibling should map to the same entity.

What would be the best way to implement this? I've been thinking and so far I've only come up with a brute-force approach checking for each synset of each word and trying to find a similar word or else add them as a new entity.

I am wondering whether there are any better methods of implementing this?

for 1-gram:

from nltk.corpus import wordnet as wn
from nltk.stem import WordNetLemmatizer
l = WordNetLemmatizer()

older = 'older'
elder = 'elder'
older_lemma = l.lemmatize(older, pos=wn.ADJ)
elder_lemma = l.lemmatize(elder, pos=wn.ADJ)

for syn in wn.synsets(older_lemma):
    if elder_lemma in syn.lemma_names():
        print(syn)

Ideally, I would want to extend it to n-grams and I'm searching for a better way to do this.

Edit : I'm not looking for vector based solutions.

What I'm thinking of is some sort of crude but fast similarity algorithm that can give me a crude representation of how close can two words/synsets possibly be. That way I could eliminate most of the absolutely dissimilar words saving time. I'm not sure if it exists

You may want to check word2vec algorithm which can map words to a semantic vector space such that words used in similar context will be closer to each other. You can then use cosine similarity on word vectors for calculating similarity.

For extending the similarity comparison to multiple words or sentences, you can look at word mover distance after training word2vec model.

The downside of the above approach is that you need a large text corpus for training word2vec model. You can either use something specific to your domain or use open source datasets (news articles, Project Gutenberg books, wikipedia dumps).


How to approach for predicting semantic similarity between two phrases I need pointers on the latest research, tools, and techniques for predicting semantic similarity between two phrases.
Problem Statement: Given two propositions A and B with A know to be true, predict if B agrees, contradicts, or is neutral.
Examples
A = X is greater than Y | B = Y is greater than X | Result = contradict
A = X is greater than Y | B = Y is less than X | Result = agree
A = X is greater than Y | B = P is less than X | Result = neutral
A = X increases with Y | B = X is directly proportional to Y | Result = agree
A = X is joining Y and Z | B = Y and Z are connected with X | Result = agree
What I have tried
I am looking into Google's Universal Sentence Encoder and alternatives for evaluating semantic similarities with mixed results. Most of the solutions available will give a result of exactly similar
for the first example. I am also going through some research papers on grammar-based similarity techniques in natural language.
Any help or direction towards research papers, tools, or libraries is greatly appreciated.

What you are describing is one of the "standard" NLP problems faced in NLP and it usually referred to as "natural language inference" (NLI), or sometimes also as "textual entailment".
There is plenty of research in this kind of task, and its variants, like cross-lingual NLI (XNLI). I suggest you have a look at nlpprogress (link) or paperswithcode (link) to understand the landscape of benchmark datasets and SoTA models in NLI.


How to deal with changing video frame sizes in a CNN? How to deal with videos where the frame sizes are not the same frame to frame?

For example this video moves up and down and when it does, the video part of the screen has a different amount of pixels vertically.

How to deal with different frame sizes in a CNN?

In your example, you are seeing the effect of a stabilisation algorithm applied to otherwise jerky movement - and the effect is a moving "window" inside the video frame.

The actual video file has a fixed frame size. As far as I know there are no video formats that switch size frame to frame. You may have different areas of focus to worry about, but should never need to worry about frame size switching on you mid video. You can rely on your video processing library to present a consistent frame size for the whole of a video.

If you find that the border of the view inside the video frame interferes with the performance of your CNN once trained, then you could deal with it in a few ways. A couple of simple approaches might be:

1) Crop out a centre portion that always reliably has content in, and use just that portion for your input features.

2) Detect and flood-fill the black border with the mean pixel value, using an image/video library (like OpenCV) so that edges around it cause less activity in your CNN.


Low number of inputs compared to outputs (per row) in neural network I have data that one row has input of size 1x2 (two values) and output is matrix of size 15x3 (fifteen rows with three values) like that

            |-----------------|
            |y01,2 y01,2 y01,3|
            |y02,1 y02,2 y02,3|
            |.                |
            |.                |
            |.                |
            |.                |
            |.                |
-------     |.                |
|x1,x2|     |.                |
-------     |.                |
            |.                |
            |.                |
            |.                |
            |.                |
            |y15,1 y15,2 y15,3|
            |-----------------|

This is one row and I have around 3000 inputs like that.
Most values are real numbers except one output column that is binary (So output y$_{n,3}$ values are 0 or 1).

How should I preprocess this kind of data?
Is it of any sense to train two models, one for real numbers only and the second one for binary output?
What neural network architecture would work best for this task?

1.) Perform standard preprocessing for you input data, i.e. remove mean and scale to unit variance.

2.) No, you should train everything together. Although it is at the moment not clear, there can be relevant relationships between your outputs.

3.) You can actually use any architecture. Simply start with Dense-NN. More important is how you arange your outputs. You should have different activation and loss functions for your different outputs. For your binary outputs use (binary)-Crossentropy and for your real values use Mean Squared Error (or sth. similar). Your final output consists of course of the sum of your single losses.


Why is performance worse when my time-series data is not shuffled prior to a train/test split vs. when it is shuffled prior to the split? We are running RandomForest model on a time-series data. The model is run in real time and is refit every time a new row is added. Since it is a timeseries data, we set shuffle to false while splitting into train and test dataset.

We observed that there is a drastic change in scores when shuffle is True and when shuffle is false

The code being used is as follows

# Set shuffle = 'True' or 'False'
df = pandas.read_csv('data.csv', index_col=0)
X = df.drop(columns=['label'])
y = df['label']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.05, shuffle=True)

count = 0
predictions = []

for idx in X_test.index.values:
    # Train the model on training data
    # print(count, X_train.shape, y_train.shape)
    rf = RandomForestRegressor(n_estimators = 600, max_depth = 7, random_state = 12345)
    rf.fit(X_train, y_train)

    predictions.append(rf.predict(X_test.loc[X_test.index == idx]))
    # print(len(predictions))

    X_train.loc[len(X_train)] = X_test.loc[idx]
    y_train.loc[len(y_train)] = y_test.loc[idx]
    count+=1

Initially, we thought the difference is due to covariance shift in the data. But that shouldn't affect this much for continuous fit

Here are screenshots for the prediction plot

With shuffle = False

With shuffle = True

Without looking further into the data myself, I can surmise that something has changed recently with your data such that if you split without shuffling, some aspect of the data in your test set (which is what you most recently collected) is underrepresented in your training set. By shuffling the data, you allow those more recent samples to also be present in your training set, and thus your test set performance improves.


XGBoost Feature importance - Gain and Cover are high but Frequency is low I have read this question: How do i interpret the output of XGBoost importance? about the three different types of feature importances: frequency (called "weight" in Python XGBoost), gain, and cover.

In my case, I have a feature, Gender, that has a very low importance based on the frequency metric, but is the most important feature by far based on both the gain, and cover metrics.

I know gender should be important for what I'm predicting. If I plot only gender vs. the target, there is a clear correlation. I am not surprised that it's the most important feature. I just want to know why it's not considered an important feature based on the frequency metric.

I just want to know why it's not considered an important feature based on the frequency metric.

Most likely, the variable gender has much smaller number of possible values (often only two: male/female or 0/1, depending on the representation) compared to other predictors in your dataset.

If gender is just binary in your case, it means that it can be used at most once in each tree, while, let say, age might appear much more often on different levels of the trees.


Detect related sentences This question is related to "How to grow a list of related words based on initial keywords?"

In the previous question they attempt to get similar words to a given word. However, I am interested in knowing the possibility of doing it to sentences.

As I'm not familiar with this area, my questions are:

Is there a way to do this with sentences (without just considering words)?
What tools are available for it?

Word Mover’s Distance (WMD) is an algorithm for finding the distance between pairs of strings. It is based on word embeddings (e.g., word2vec) which encode the semantic meaning of words into dense vectors.

  The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to "travel" to reach the embedded words of another document.

For example:

Source: "From Word Embeddings To Document Distances" Paper

The gensim package has a WMD implementation.


Difference between Global Outlier and Contextual Outlier? I am studying "Data Mining: Concepts and Techniques" by Han, Kamber & Pei. In Chapter 12 "Outlier Detection", they have stated that there are 3 types of outliers:

Global Outlier - deviates significantly from the rest of the data set
Contextual Outlier - deviates significantly with respect to a specific context of the object
Collective Outlier - the individual data objects may not be outliers, but the objects as a whole deviate significantly from the entire data set.

According to the authors:

  “The temperature today is 28 C. Is it exceptional (i.e., an outlier)?” It depends, for example, on the time and location! If it is in winter in Toronto, yes, it is an outlier. If it is a summer day in Toronto, then it is normal. Unlike global outlier detection, in this case, whether or not today’s temperature value is an outlier depends on the context—the date, the location, and possibly some other factors. In a given data set, a data object is a contextual outlier if it deviates significantly with respect to a specific context of the object. Contextual outliers are also known as conditional outliers because they are conditional on the selected context. Therefore, in contextual outlier detection, the context has to be specified as part of the problem definition.

My question : Is not Contextual Outliers the same as Global Outliers where the original feature vector describing each data object may have to be extended to incorporate some other attributes (information regarding the "context") ?
Obviously, if the data objects do not contain enough attributes, they can never be classified as outlier ! This holds true for global outliers as well. So it makes no sense to me to form a new category of outliers as Contextual Outliers.

Let's say that your dataset is a bunch of temperatures with a bunch of features related to when/where the measurements were made.
A measurement is a global outlier if it diverges from the distribution of temperatures regardless of the features (when and where) because that measurement is far off the global distribution (100°C for example).
The example makes a good case of how looking at conditional distributions (temperature|place, time of measurement) can provide additional insights (here detecting that 28°C is a contextual outlier). Contextual outliers are within the global distribution but are outliers when looking at conditional distributions.
Now of course your dataset has some context, so in a sense you're right, everything is contextual, but the global/contextual border resides in what you have at hand : if all you have is the temperatures from one day, comparing a temperature from that day to all the measurements of that day will allow you do detect gloabl outliers ; now if you had access to temperatures all year long and that your still looked at that day precisely, that would become a conditional distribution.


Does batch_size in Keras have any effects in results' quality? I am about to train a big LSTM network with 2-3 million articles and am struggling with Memory Errors (I use AWS EC2 g2x2large).

I found out that one solution is to reduce the batch_size. However, I am not sure if this parameter is only related to memory efficiency issues or if it will effect my results. As a matter of fact, I also noticed that batch_size used in examples is usually as a power of two, which I don't understand either.

I don't mind if my network takes longer to train, but I would like to know if reducing the batch_size will decrease the quality of my predictions.

Thanks.

After one and a half years, I come back to my answer because my previous answer was wrong.

Batch size impacts learning significantly. What happens when you put a batch through your network is that you average the gradients. The concept is that if your batch size is big enough, this will provide a stable enough estimate of what the gradient of the full dataset would be. By taking samples from your dataset, you estimate the gradient while reducing computational cost significantly. The lower you go, the less accurate your esttimate will be, however in some cases these noisy gradients can actually help escape local minima. When it is too low, your network weights can just jump around if your data is noisy and it might be unable to learn or it converges very slowly, thus negatively impacting total computation time.

Another advantage of batching is for GPU computation, GPUs are very good at parallelizing the calculations that happen in neural networks if part of the computation is the same (for example, repeated matrix multiplication over the same weight matrix of your network). This means that a batch size of 16 will take less than twice the amount of a batch size of 8.

In the case that you do need bigger batch sizes but it will not fit on your GPU, you can feed a small batch, save the gradient estimates and feed one or more batches, and then do a weight update. This way you get a more stable gradient because you increased your virtual batch size.


Predicting earthquakes using disturbances in DTH TV transmission It is said that before an earthquake happens, a viewer experiences disturbances in DTH TV transmission in the form of distorted images on the screen which automatically correct after a few seconds.

Is it possible to identify patterns of such disturbances by continuously monitoring TV images so that earthquakes can potentially be predicted at least few minutes in advance and many lives could be saved?

I think that if there are disturbances caused by something preceding earthquake, they might be used.

The problem is that you need to find which cause produces that effect and measure it. For example you can team up with geologists, geophysicists or someone similar and try to build as complete hypothesis as possible and then design experiment to gather data. If your experiment would bring some non noise data then you can start to thinking about machine learning algorithms which will work in this situation.

But you might also try to do it on "brute force" way by actually recording DTH TV images in seismically unstable regions and try to correlate those videos with seismic data. Then after some manual examination and categorization you could define (or not if this cause and effect hypothesis is wrong) some possible glitches that are observed and try to develop software that tries to detect them (OpenCV might be useful for example).

IMHO in either way, some domain knowledge related to earthquakes would be more useful on beginning stage, and experience related to machine learning would be more useful on latest stages of such ambitious project.


Retrain Tensorflow CNN Model on additional Images So I've created a model to classify images of traffic signs. I'm currently using this model to predict classes of images I've gotten from the internet. However, I would like to be able to re-classify or correctly label images in which the model failed to correctly predict their classes.
One way would be to place the images in their correct classes and retrain the model from the last saved checkpoint. I don't think this is a good approach as the model should overfit over time. Hence, I'd like to know if there's a better way to achieve this.

There are two basic approaches to this problem:

The performance of the old model was not good enough, you are able to gather much more data and train a new better model

You setup an "online model" which continuously learns and improves but is setup very differently.

Do you have a specific performance level in mind that you need / want to hit? In that case simply gathering more data and retraining may be fine. However if you simply want to continuously improve you need to approach it with an "online model".


Removing sensitive information from a dataset while retaining its usefulness for machine learning tasks? I have a dataset that would be useful for training future machine learning models.

However, it contains sensitive information (personal identifiers like names and addresses) that make it difficult to store or retain longterm.

I'm wondering if there are transformations that could be applied that would remove the sensitive information, while retaining it's usefulness?

The dataset is specifically useful for training NLP related models, but I'm also interested in a general solution to this for general machine learning tasks if one exists.

There are lots of solutions to this problem and both of them would allow you to release the dataset. There are in general, two factors that are of importance, the first, whether the method is error prone or not and secondly, whether it is time consuming or not.

The first, being a simpler and dependent solution. If you have a dictionary of names that you maintain, in your database, what you can do is search for whether the given word is a substring of a name (probably through a trie search) and then, check to see if it's a name in your database. If it is, replace it with some string, something even as easy as "Name101" would be easy to accomplish. The same can be done for geographical locations and other possibly sensitive data. This is a non-error prone and time saving method.

If you haven't done the former, there is a solution, however, it's a natural language processing technique called Named Entity Recognition (https://en.wikipedia.org/wiki/Named-entity_recognition), where you can detect if a given word is a name, location or anything that could be worth hiding. However, please understand that this method is not guaranteed to succeed, it may hide most of your data, but it is not guaranteed to hide all of it. This is time saving but certainly error prone.

The last possible alternative is to actually comb through the data yourself and make sure that you've changed all of the names. This is an error prone and time consuming technique.

Ultimately, you can use any of these to accomplish the tasks, but some of them are possible given your method of data collection, or your data pipeline.


Any usable libs to build and visualise SOM in python? I tried SOMpy, though it is very crude now and works only with oldest versions of matplotlib.
Is there any fancy lib that can build SOM based on array and visualize it in Python?

You could have a try on this package.
There is a working example on this page.
If what you are interested is the Manifold learning,
you could also apply many packages from sklearn.


Do models learn a test set after multiple predictions upon it? In machine learning, when we train a model, such as

vect = CountVectorizer(min_df=5, ngram_range=(1,2)).fit(X_train)

X_train_vectorized = vect.transform(X_train)
model = LogisticRegression()
model.fit(X_train_vectorized, y_train)

predictions = model.predict(vect.transform(X_test))

and make predictions with a test set, does our model learn the test set after multiple predictions are made?

TL/DR: No.

The concept of prediction, here applied by your predict method, is to just try out the model on the test set. It does not imply any kind of adaptation to such set, thus your model doesn't directly learn anything from it.

You can generalize that to any other framework or discussion in data science as a whole - training and fitting are tasks performed on the train set, and they imply adapting to it, and actually learning its patterns. Predicting does not.


BERT - How Question answering is different than classification Basically I am trying to understand how question answering works in case of BERT. Code for both classes QuestionAnswering and Classification is pasted below for reference. My understanding is:

class BertForSequenceClassification(PreTrainedBertModel):
    def __init__(self, config, num_labels=2):
        super(BertForSequenceClassification, self).__init__(config)
        self.num_labels = num_labels
        self.bert = BertModel(config)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)
        self.classifier = nn.Linear(config.hidden_size, num_labels)
        self.apply(self.init_bert_weights)

    def forward(self, input_ids, token_type_ids=None, attention_mask=None, labels=None):
        _, pooled_output = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)
        pooled_output = self.dropout(pooled_output)
        logits = self.classifier(pooled_output)

        if labels is not None:
            loss_fct = CrossEntropyLoss()
            loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
            return loss
        else:
            return logits

In Above code pooled_output is considered useful in line  _, pooled_output = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)

And in below QnA code encoder layer output (i.e., sequence_output) is considered useful in line: sequence_output, _ = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)

class BertForQuestionAnswering(PreTrainedBertModel):
    def __init__(self, config):
        super(BertForQuestionAnswering, self).__init__(config)
        self.bert = BertModel(config)
        # TODO check with Google if it's normal there is no dropout on the token classifier of SQuAD in the TF version
        # self.dropout = nn.Dropout(config.hidden_dropout_prob)
        self.qa_outputs = nn.Linear(config.hidden_size, 2)
        self.apply(self.init_bert_weights)

    def forward(self, input_ids, token_type_ids=None, attention_mask=None, start_positions=None, end_positions=None):
        sequence_output, _ = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)
        logits = self.qa_outputs(sequence_output)
        start_logits, end_logits = logits.split(1, dim=-1)
        start_logits = start_logits.squeeze(-1)
        end_logits = end_logits.squeeze(-1)

Now my questions are:

Why there are 2 logits being returned in sequence_output for Question Answering case
What is different in encoder layer and pooled layer
Why is encoder layer (sequence_output) is considered in QnA case and Pooled layer in classification case

For Question Answering, you need 2 logits : one for the start position, one for the end position. Based on these 2 logits, you have an answer span (denoted by the start/end position).
In the source code, you have : pooled_output = self.pooler(sequence_output)
If you take a look at the pooler, there is a comment :

# We "pool" the model by simply taking the hidden state corresponding
# to the first token.

So the sequence output is all the token representations, while the pooled_output is just a linear layer applied to the first token of the sequence.

In classification case, you just need a global representation of your input, and predict the class from this representation. No need to access every token of the sequence, you want a single representation of the sequence.
But in the Question Answering case, in order to predict the answer span, you need to decide which token will be the start and which token will be the end of the span. For this, you need to consider every token independently.


How to asses employees performance using their data? I have some employees' data consisting of three variables - the name of the employee, a value denoting their positive behavior and a value denoting their negative behavior.

How to score employees using this data with a 70% weight on negative behavior and a 30% weight on positive behavior?

Assuming that the values denoting positive and negative behaviors are both positive, you can set score as -

score = -0.7*negative_behavior_value + 0.3*positive_behaviour_value


Dealing with skewed mean I have a data set of districts, farmland area and fertilizer subsidies issued for those areas. I.e, using made up numbers,

district | area | subsidy | subsidy per area (computed)
abc      |   20 |   500   |         25
cde      |   30 |   750   |         25
fgh      | 0.02 |    15   |        750    <--- looks off

I'm trying to visualise the subsidy per area but in districts that have very small amounts of farming the subsidy per area seems abnormal. The nationwide average is pretty much around 25. So, I can safely say that the subsidy amount is directly related to the area being subsidised, which is to be expected as fertiliser usage is dependent on the area being farmed. My theory is that the exception on small areas is due to there being a minimum subsidy amount irrespective of the land area.

Are there any techniques to deal with the above scenario when visualising data?

As per business statement it is a exceptional scenario when minimum subsidy will be provided. So for displaying common behavior of data you can drop these exceptional values from table. You can use box-plot to visualize spread of data and then remove else, if you are aware with max range in normal scenario you can drop rows having value more than that.


decision rules for each feature (binary classification) I have a collection of 10 features (all numerical) and a single binary outcome variable. I need to train a binary classification model, find the best features and compute thresholds for each feature.

To find the best subset of features is easy, I use logistic regression with L1 penalisation and it works nicely. However, the next step is to find thresholds for each subselected feature: if the values of a feature i, j,...k are above/below than some numerical values, than chances to be in the class A (rather than in B) are higher.

FOR EXAMPLE

Consider, we found that out of 10 original features, there are L1 leaves only 3: F1, F4,F7. And if you take three unseen (new) data with some values {F1_i, F4_i, F7_i}, where

F1 > 1.23
F4 < 7.15
F7 > 2.74

Then the new data point {F1_i, F4_i, F7_i} belongs to class A. Here {1.23, 7.15, 2.74} are thresholds.

I tried to explain the problem as clear as possible, but if it isn't, please let me know.

QUESTION What will be the best approach to solve this problem? How to compute thresholds?

You are describing every binary classifier.  However, you are missing a key point. If your classes are separable by the value of just ONE feature, you can do what you're saying and find e.g. F1 > 1.23 as a threshold.  If classification involves a combination of features, you will need to describe some combination of thresholds for each feature, or (equivalently) some relationship between the features that tells you about the class label.  It's the job of every binary classifier to do exactly this - they just do it in different ways.  See for example this post.  Your desire to have a combination of fixed sets of thresholds will only work if you can have a set of thresholds that will encompass/describe/classify every combination of feature values.

If you want a set of easy-to-read thresholds like you mention, you should read about decision tree classifiers.  They'll do something like what you want - but will also ensure that you provide a class label for every possible combination fo features values. The nice about about decision trees is that they'll let you leave out your current feature selection step - they just do it for you by (1) picking the feature that best discriminates the two classes overall, (2) picking the threshold value of that feature that gives the most information about the class label (usually), and (3) repeating (1-2) several times.


How do I interpret the length-scale parameter of the RBF kernel? According to the Scikit-Learn documentation for the RBF kernel:
The length scale of the kernel. If a float, an isotropic kernel is used. If an array, an anisotropic kernel is used where each dimension of l defines the length-scale of the respective feature dimension.

I am currently working on a problem where I am setting the length-scale of each individual feature (which I assume is synonymous with dimension here). My understanding is that a smaller length scale implies a more complex function.

My question is, can I use this parameter to explain how well a certain feature will help a model generalize to new data?

For example, if I have a data set which, after optimizing the length-scale value looks like this:
[Feature_1: length-scale = 20]
[Feature_2: length-scale = 1]
[Feature_3: length-scale = 5]

Does this mean that, if I had to pick one feature which would help a model generalize to new data, it would be Feature_1? Is Feature_2 potentially causing my model to overfit? Are these fair assumptions to make?

Note: I am using support vector regression with this kernel.

Even though I am more familiar with the use of RBF kernel with Gaussian Processes, I think your intuition is correct since, generally speaking, a larger lengthscale means that the learnt function varies less in that direction, which is another way of saying that that feature is irrelevant for the learnt function.

So if you have to choose which feature is the most important you could say that the one with the lowest lengthscale parameter is the most relevant.

All this only applies if the characteristic scale of the features (the standard deviation for example) is already comparable (i.e. you have to standardize the features before optimizing the lengthscales if you want to retain this intuition).

That said, making assumptions about what is making you overfit is difficult from the lengthscale viewpoint. I think we could say that shorter lengthscales across many independent features are going to increase model complexity, and thus the risk of overfitting. As always there's an optimal complexity for improving out-sample performance.

Just to complete the answer, I like to track the complexity of a kernel model by evaluating the kernel matrix k(x1, x2) and computing its effective rank as in https://infoscience.epfl.ch/record/110188/files/RoyV07.pdf . A lower rank means a less complex model (because all the points look more similar in your implicit kernel space), whereas a higher effective rank allows learning very complex functions but you lose statistical power since in your kernel space all your points are "outliers", in the sense that they are apart from each other.

I hope that helps.


Datasets in NLP research papers I am looking for a dataset containing a large number of NLP research papers and abstracts. Are there any open access datasets like that?

If so, can you please share the details?

I have come across such a dataset in CS 20SI GitHub repo: it's a collection of abstracts from 7200 research papers.

If you need even more, you can always write a simple crawler of arXiv web-site. The abstracts are outlined right on the webpage in a <blockquote> tag, no need to download and parse a pdf.


California housing prices dataset on Kaggle This is the dataset I'm referring to: https://www.kaggle.com/camnugent/california-housing-prices
I am a bit confused regarding the dataset. Specifically, why is the number of houses not provided? We are given the number of households, but that's not equivalent to the number of houses. There's probably a near monotonic relation between the two, but I feel like the dataset should include the number of houses per block.

It looks like the dataset is aggregated from another raw dataset, e.g. the mean house price has been calculated from the raw data listing every house individually.
The creator of the dataset perhaps made a mistake, or intentionally left out the number of houses, perhaps not thinking this feature was relevant for the use-case in mind.
Either way, the dataset is made from a census taken in California in 1990, so perhaps you need to search for the raw census data instead and calculate it yourself.


Create most "average" cosine similarity observation For a recommendation system I'm using cosine similarity to compute similarities between items. However, for items with small amounts of data I'd like to bin them under a general "average" category (in the general not mathematical sense). To accomplish this I'm currently trying to create a synthetic observation to represent that middle of the road point.

So for example if these were my observations (rows are observations, cols are features):

[[0, 0, 0, 1, 1, 1, 0, 1, 0],
 [1, 0, 1, 0, 0, 0, 1, 0, 0],
 [1, 1, 1, 1, 0, 1, 0, 1, 1],
 [0, 0, 1, 0, 0, 1, 0, 1, 0]]

A strategy where I'd simply take the actual average of all features across observations would generate a synthetic datapoint such as follows, which I'd then append to the matrix before doing the similarity calculation.

[ 0.5 ,  0.25,  0.75,  0.5 ,  0.25,  0.75,  0.25,  0.75,  0.25]

While this might work well with certain similarity metrics (e.g. L1 distance) I'm sure there are much better ways for cosine similarity. Though, at the moment, I'm having trouble reasoning my way through angles between lines in high dimensional space.

Any ideas?

You are doing the correct thing. Technically, this averaging leads to computing the centroid in the Euclidean space of a set of N points. The centroid works pretty well with cosine similarities (cosine of the angles between normalized vectors), e.g. the Rocchio algorithm.


sklearn.cross_validation.cross_val_score "cv" parameter question I was working through a tutorial on the titanic disaster from Kaggle and I'm getting different results depending on the details of how I use cross_validation.cross_val_score.

If I call it like:

scores = cross_validation.cross_val_score(alg, titanic[predictors], titanic["Survived"], cv=3)

print(scores.mean())

0.801346801347

I get a different set of scores than if I call it like:

kf = KFold(titanic.shape[0], n_folds=3, random_state=1)

scores = cross_validation.cross_val_score(alg, titanic[predictors], titanic["Survived"], cv=kf)

print(scores.mean())

0.785634118967

These numbers are close, but different enough to be significant. As far as I understand, both code snippets are asking for a 3-fold cross validation strategy. Can anyone explain what is going on under the hood of the second example which is leading to the slightly lower score?

From the sklearn docs for cross_val_score's cv argument :

For int/None inputs, if the estimator is a classifier and y is either
binary or multiclass, StratifiedKFold is used. In all other cases,
KFold is used.

I believe that in the first case, StratifiedKFold is being used as the default. In the second case, you are explicitly passing a KFold generator.
The difference between the two is also documented in the docs.

KFold divides all the samples in $k$ groups of samples, called folds (if
$k = n$, this is equivalent to the Leave One Out strategy), of equal
sizes (if possible).
[...]
StratifiedKFold is a variation of k-fold which returns
stratified folds: each set contains approximately the same
percentage of samples of each target class as the complete set.

This difference in folds is what is causing the difference in scores.
As a side note, I noticed that you are passing a random_state argument to the KFold object. However, you should note that this seed is only used if you also set KFold's shuffle parameter to True, which by default is False.