How does Sigmoid activation work in multi-class classification problems I know that for a problem with multiple classes we usually use softmax, but can we also use sigmoid? I have tried to implement digit classification with sigmoid at the output layer, it works. What I don't understand is how does it work?

If your task is a kind of classification that the labels are mutually exclusive, each input just has one label, you have to use Softmax. If the inputs of your classification task have multiple labels for an input, your classes are not mutually exclusive and you can use Sigmoid for each output. For the former case, you should choose the output entry with the maximum value as the output. For the latter case, for each class, you have an activation value which belongs to the last sigmoid. If each activation is more than 0.5 you can say that entry exists in the input.


How can you build a model that reads out receipts and invoices? The objective is to build a model that is capable of identifying information on receipts and invoices that can look completely different.

I've had a discussion with my brother about the right approach. I have attached an example, here the original and below is the important information in boxes:

The green boxes are the must-have information. The one in purple and green indicates that we need either or. The orange information would be a nice-to-have, but not necessarily required.
Some of the boxes have context and inter linkage.

From a data set point of view, we have a sample size of 1,000 receipts and all of them have the necessary information extracted. We could increase the sample size further if that was required.

The approach that I would have chosen:
Treat everyone of the images of the receipts like a game and let the model figure out itself how to arrive at the right conclusion. This will most likely be very computing intensive but I feel like it will be more robust when dealing with new image types.

The approach my brother has suggested:
Basically using the boxes that I've provided and let me model learn from that. The model would then learn to identify the important areas on a receipt or invoice and would go from there. He compared the model to one that would identify license plates.

Edit:
Just to reiterate why I think OCR is only part of the solution but not the solution itself. Here is the Adobe Acrobat OCR result:

Perfect if you ask me. It just doesn't help me figure out what values to use and which to ignore. I don't want to do this manually. I want the model(s) to return for me:

Total amount
Sales Tax & Amount
Creditor (i.e. the company and ideally the tax identifier CHE-xxxxx MWST
Date and ideally time
Payment method

Does this make more sense now? I just don't see how OCR gets me there. It will only be the method to extract the values.

You should start by trying OCR.  I am not sure why you are rejecting it without trying it. Start with Tesseract (free but not very good) and then try a commercial OCR as well (Abbyy is well regarded, but you could also try Adobe Acrobat).  Also, research document structure extraction.

That idea that you are going to re-invent a machine learning model that does OCR better than existing OCR solutions seems ... not very realistic.  Current OCR tools have years or decades of engineering put into them.  There is no way you are going to be able to afford to put that much effort into your own custom tool.

Image quality on those example images is not wonderful, and you might have to accept that accuracy will be less than 100%.


How to predict unknown unknowns in machine learning I am dealing with a problem about classifying bird species through analysing MFCCs. I already built a dataset with 13 MFCCs for two kinds of birds. And I trained the data with Naive Bayes & KNN model. However, when I tried the model with prediction of third bird species, it is classified as the one of the two species. I am wondering how can I achieve to predict unknown species as unknowns? And I know my existing classification model may not work. So, what kind of model might be helpful? Does SSL useful in my case? Or treat these unknowns as outliers? But how can that be applied in MFCC?

If you want to predict wether a bird is a bird of one of your two classes or unknown you need three classes: $[bird A, bird B, unknown]$.
For the unknown class you need data from birds which are neither $bird A$ nor $bird B$. You should make sure the number of rows for each of the three classes is roughly the same.
If you don't have data of birds which are neither $bird A$ nor $bird B$ you can use anomaly detection to detect wether a bird is $unknown$ before predicting if it is $class A$ or $class B$.


To get the unique elements you can convert the tuples to a set with a couple of comprehensions like:

Code:

[tuple({t for y in x for t in y}) for x in data]

How:

Inside of a list comprehension, this code creates a set via a set comprehension {}. This will gather up the unique tuples. Two loops are needed inside of the set comprehension:

for y in x for t in y

because the tuples of interest are themselves inside of a tuple.

Test Code:

data = [
    [
        ((4, 2), (2, 1)),
        ((4, 2), (6, 3)),
        ((2, 1), (4, 2)),
        ((2, 1), (6, 3)),
        ((6, 3), (4, 2)),
        ((6, 3), (2, 1))
    ], [
        ((4, 2), (-3, 4)),
        ((-3, 4), (4, 2))
    ], [
        ((2, 1), (-3, 4)),
        ((-3, 4), (2, 1))
    ], [
        ((-3, 4), (6, 3)),
        ((6, 3), (-3, 4))
    ]
]

expected = [
    ((6, 3), (4, 2), (2, 1)),
    ((4, 2), (-3, 4)),
    ((-3, 4), (2, 1)),
    ((6, 3), (-3, 4))
 ]

assert expected == [tuple({t for y in x for t in y}) for x in data]


Transfer learning VGG16 does not work as expected. (Detect tacos as hamburgers) I am new in this field of machine learning, to test I wanted to do a simple project. Create a cnn capable of recognizing hamburger images. As I do not have the ability to collect more than 10,000 images of hamburgers I have used an existing model to train what I need ... I have used VGG16.

This is the notebook I am using, to do transfer learning. I am training the model in detecting cats and hamburgers.
https://github.com/EsteveSegura/detecting-food/blob/master/src/BurgerCNN.ipynb

If the notebook does not load, the entire code is also here
https://gist.github.com/EsteveSegura/2be156ba5431fc42fb8ac13eb0506c82

Train

Cat: 1024 imgs

Burger: 1097 imgs

Validation

Cat: 416 imgs

Burger: 326 imgs

When I try to predict photos of hamburgers, it usually gives a good result ... But it also detects as burger photos of tacos (99% acuraccy). Is it an overfitting problem? Do I need more images? Am I doing the transfer learning in a correct way?

My goal will be to detect if there is a hamburger in the photo or not.

One of the recommendations they have given me is to train a third class with completely random objects that do not repeat ... Can this help? In case I can help? How?

This is the log of my training

You are almost there as you mentioned the addition of third class as random photos. Generally what happens when you train a neural network, it assumes the entire domain of images existing as the training data. For  your case each image can be either cat or burger. Now you have not shown random images to network and hence when a image which is neither cat nor burger, network will try it's best to predict it as either cat or burger. Instead I would suggest you to train network as burger vs not burger. In not burger class have all possible kind of images that are not burger. You can use CIFAR 1000 data to get images of different classes. Now your model will learn to differentiate between burger and not burger properly. Obviously you can still find some cases where miss classification happening and the reason will be because you have not covered such images in training.

Also play a bit with softmax threshold to get high presicion and recall.


What is a TensorSpec? (Tensorflow 2.0) what is a tf.TensorSpec() and what is it used for?

TensorSpec is mostly used by tf.function to specify input signature. tf.function will create a graph for different input shapes and datatypes, but it is possible that your function graph is compatible with different shapes. As a performance optimization, You can optionally provide a signature so that no unnecessary graphs are created.


How to most effectively utilize historical data to train churn model Suppose we have some historical data of users activity on a website and we want to build a churn prediction model (let's say we want to predict churn in a 2 month window). The usual approach, as I understand it, is to take a slice of that historical data at time $t$ and see which users churn in the time interval $(t, t + 2 \text{ month})$, so we take some features at time $t$ and train our model.

However, that way we only use a small part of our data, feeding our model only users who were active at time $t$. But what if we want to use all historical data? One way that comes to mind is take a lot of slices of our data at times $t_1, t_2, t_3, \dots$ and just merge them in one dataset, however different slices could have the a lot of same users even if we take these slices very far apart from each other. So our model could potentially learn that if a particular set of features occurs many times in our data set, then the user with these features is less likely to churn, (e.g. if we take two time slices at $t_1$ and $t_2>1$ then, if a user is presented in both of these slices, he can't churn at least at time $t_1$). So it doesn't seem to be the right way to do it...

How can I extract as much information as possible from a historical data over large period of time without spoiling the model?

That's correct. You will have the same user multiple times. This can introduce a small bias. I would make a few points why this is better than working with your first approach where you have less data:

First, if the same user shows up in multiple slices that means he doesn't churn or churns later than others. So you bias the model towards figuring out what high retention users look like. As you say:

  So our model could potentially learn that if a particular set of features occurs many times in our data set, then the user with these features is less likely to churn

and this is actually a good thing if you have a good mix of user retention periods.

Second, you can include the time $t$ as a feature (or their signup date etc.). This will teach your model that users that haven't churned in the past are less likely to churn in the future and reduce the importance the model puts on these other patterns you mention.

More generally, you should not only have a set of static features that don't change for the same user from $t$ to $t+1$. You should also have set of features that is dependent on $t$, e.g. "the number of logins in the last 7 days". That way the model will see differences between the same user at time $t$ and $t+1$.


How does an encoder-decoder network work? Let's say I trained an encoder-decoder network on a cat dataset using reconstruction error as loss function. The network is fully trained and the decoder is able to reconstruct good cat images.

Now what if I use the same network and input a dog image. Will the network be able to reconstruct dog image or not?

It probably won't. The whole point of the training was to encode cat images and thus the network has tried to learn what information is the most necessary to keep to ensure a low reconstruction error (i.e. what separates one cat from another) and what information can it throw away (i.e. what characteristics appear in all cat images and can be discarded).

That being said, a dog image would produce a fairly decent reconstruction because most features are shared between both animals. If you try, however, to reconstruct something completely different (e.g. a car) then it would probably fail.


Different Values in Training and Testing set If I am trying to build a classification model where Values for columns have different ranges: For Example: Column A in training set ranges from 0 - 30 but for testing set it ranges from 0 - 80.

Would a classification model still work?

On principle, you should try to achieve a fairly balanced distribution between your training and testing sets. Otherwise your model might not make appropriate generalizations.  Additionally, you should use some form of normalization such as feature scaling.

It is difficult (or even impossible) to understand your data distribution by simply looking at the range.  It would be better to look at the mean/median or even a histogram/normal curve.

Let's say, for example, that our ordered column A in your training/testing sets look as such (before normalization):

\begin{array}{|c|}
\hline
&column \ A \ split\\
\hline
training && testing \\
0 && 0\\
4 && 2\\
5 && 3\\
7 && 3\\
9 && 8\\
15 && 11\\
23 && 15\\
30 && 80\\
\hline
\end{array}

\begin{array}{|c|}
\hline
&set\ metrics\\
\hline
& training & testing\\
range & [0,30] & [0,80]\\
mean & 11.625 & 15.250\\
median & 8.0 & 5.5\\
stddev & 10.309 & 26.650\\
\hline
\end{array}

\begin{align}
\large\frac{1}{\sqrt{2\pi}\cdot\sigma}e^{\frac{-(x-\mu)^2}{2\sigma^2}}
\end{align}

(red=training, blue=testing)

Here we can see that despite the large range discrepancy, our sets are relatively balanced.

We can observe a problematic training/testing split in this example where we shift the training mean from 11.625 to 100:

Moreover, we could generate another hypothetical dataset with a bad training/testing split where our set ranges are identical.

TL;DR:

Do not evaluate the set distribution by range; instead use mean/median or histogram/normal curve.  Also normalize your data.


state-action-reward-new state: confusion of terms My question may sound like a duplicate of, for example, How is that possible that a reward function depends both on the next state and an action from current state?
but I still feel confused.

In neural network approximation of the Q function, I follow the experience replay routine. Papers and manuals suggest storing state, action, received reward, and next state information in the experience replay. However when one is about to calculate a maximum of Q value based on the next action, which state do they use: state or next state? If the next state is used to calculate the max Q, then what is the whole purpose of storing the previous state and action information?

An example (2D world):

An agent is at cell A1 (state). His goal is to get to C3 to get a positive reward. He then moves to B2 (action); a received reward is, let's say, 0. Next state is B2 (after the action was done).

Questions:

Should one rely on the B2 state to iterate over possible actions from this state (next state) to get an approximation of highest reward (max Q)?

Then, why do we store the A1 and move-to-B2 information at all in the replay buffer?

Or I am wrong and we just use the A1 and iterate over possible actions (including that to B2) to get the max Q?

Edit: I think I have found an answer ). We need to store previous state (A1) and action (move to B2) in order to create the state-action distribution, which will be met with the expected long-term reward distribution, that we get after the next state routine. Right?

The TD Target (for learning update) for using $\hat{q}(s,a)$ neural network in Q-learning is:

$$r + \text{max}_{a'} \hat{q}(s',a')$$

In order to calculate this, you need a starting state $s$, the action taken form that state $a$, and the resulting reward $r$ and state $s'$.

You need the $s, a$ to generate the input to the neural network (what you might call train_X for supervised learning). You need $r, s'$ to generate the TD Target shown above as an output to learn for regression (in supervised learning, that would be train_y). And you need to work through all possible $a'$ based on $s'$ in order to find the maximum value for the TD Target equation used in Q learning - other RL algorithms may use variations of this for calculating the TD Target.

This means your suggestions are all close but not quite right.

  Should one rely on the B2 state to iterate over possible actions from this state (next state) to get an approximation of highest reward (max Q)?

Sort of. The B2 state is $s'$ from the equation, so is responsible for calculating the TD target for state-action value $q(s,a)$.

  Then, why do we store the A1 and move-to-B2 information at all in the replay buffer?

You still need to know $s$ is A1, because the representation of A1 (and whichever action is taken) will be the input to your network.

  Or I am wrong and we just use the A1 and iterate over possible actions (including that to B2) to get the max Q?

Iterate over actions from B2 using your neural network to pick the highest estimate.

  I think I have found an answer ). We need to store previous state (A1) and action (move to B2) in order to create the state-action distribution, which will be met with the expected long-term reward distribution, that we get after the next state routine. Right?

I'm not sure I fully understand this, but it does not seem quite right. One thing that might be confusing you is having your actions as "move to state". Whilst this is quite normal in many deterministic environments, especially board games, most RL formula and tutorials are written using separate state/action pairs, so whilst getting this straight in your mind, best to find another way to represent actions - e.g. move piece from X to Y, or place new piece at (I,J) . . . you can return to "change state from A1 to B2" as the representation later. This is actually more efficient and is called the after-state representation, but most of the literature will show state-action value functions.


Minimum Possible Test MSE I have a little confusion.

What follows is from Introduction to Statistical Learning (2013) by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani.

My understanding of what is going on is the following. The black curve is a function, let us say $y=f(x)$.

We have a random variable $g$ which I write as $g=f+\varepsilon$, $f$ plus noise.

The data points are a subset of the plane, $n$ trials of $g$, the (possibly multi)-set $T:=\{(x_i,g(x_i)):i=1,\dots,n\}$.

I imagine (incorrectly as I know that splines are not polynomials) that the yellow curve is a degree one polynomial (flexibility two) that is the best LS fit to the the trials (the training), the blue is the best degree five polynomial (flexibility six), and the green is the best degree $(20+\alpha)$ polynomial (flexibility $20+\alpha+1$).

In my head, the training should be the data points $T$, while the test should be $f$ (as in $f$ is the expectation of the test data).

I understand that the grey line is telling me that increasing the flexibility (degree of the polynomials in my head), allows me to approximate better the set $T$. However, if I have duplicates of $x$ in $T$, say $x_j=x_k=x^*$, with different $g$ values (e.g. something like $(20,3),\,(20,5)$ both in $T$), then I cannot have a polynomial (or indeed spline or any function) $p$ that has $p(x_j)$ and $p(x_k)$ different: $p(x_j)=p(x_k)=p(x^*)$, single-valued.

Therefore, if I have such duplications in the $x$ variable, I cannot reduce the MSE to zero.

In turn, the red line shows, that when we overfit the data with too much flexibility, the fitted curve is (my usage) biased to the training and so will not model well $f$, and so we have this increasing.

The problem I can't square (excuse the pun), is the dashed line.

It says minimum possible test MSE over all models. Whether 'test' refers to $f$ or $T$ this does not make sense to me.

If 'test' here means $f$, well surely this is zero? We can approximate $f$ arbitrarily well with a polynomial of large enough degree.

If 'test' here means the data $T$, we must conclude that $T$ contains $x$ duplicates: otherwise we could fit a polynomial of degree $n+1$ through all the test points and get this to zero. Therefore there must be duplicates, and so, perhaps, this theoretically best fit goes through all the points which are not duplicated, and goes through the average $g(x_i)$ of the duplicated points... and the answer turns out to be one... but then the grey line should not go below this...

Therefore I conclude that the dashed line is the best possible fit to $f$... but why isn't this zero?

Questions:

  Am I right to be confused by this? Is the black $f$ the test or the
  training?
  Am I misunderstanding something else? Perhaps these (smoothing) splines cannot
  well-approximate as well as polynomials?

Important disclaimer: I'm not a statistician and I'm not sure about my interpretation!

I also thought at first about the duplicates, but I think the problem might be with this assumption:

  In my head, the training should be the data points , while the test should be  (as in  is the expectation of the test data).

Specifically the last part: in principle the test set is made of points from the same distribution as the training data, with the same risk of noise. In other words, the test set $t$ is similar to $T$: $t=\{(x_i,g(x_i)):i=1,\dots,m\}$ (and not  $t=\{(x_i,f(x_i)):i=1,\dots,m\}$).

  If 'test' here means the data , we must conclude that  contains  duplicates: otherwise we could fit a polynomial of degree +1 through all the test points and get this to zero.

Importantly, the test set $t$ is different from the training set $T$, and the estimated function $\hat{f}$ is based only on the points in $T$. So this way it makes sense that even a perfect estimate $\hat{f}=f$ might not be able to predict the true (noisy) value for every point $x\in t$. That could explain the non-zero minimum test MSE.


Item based recommender using SVD I have an item-item similarity matrix. e.g. (the matrix is symmetric, and much bigger):

1.00 0.88 0.96 0.99
0.88 1.00 0.99 0.96
0.96 0.99 1.00 0.86
0.99 0.96 0.86 1.00

I need to implement recommender which, for a set of items, recommends a new set of items.

I was thinking about using SVD to reduce the items to n-dimensional space, let's say 50-dimensional space, so each item is represented with a vector 50 numbers, and similarity between two items is calculated by cosine similarity between two 50-dimensional vectors.

For a base set of items (which can get quite big), I hope I could calculate an average of their vectors, and use it for search.

Is this a good idea? What is this procedure called? And can it be done in Mahout?

EDIT:

This is my code so far:

ItemSimilarity similarity = new LogLikelihoodSimilarity(model);
Matrix m = new DenseMatrix(NUM_ITEMS, NUM_ITEMS);
// copy similarities to a matrix
for (int i = 0; i < NUM_ITEMS; i++) {
        double[] similar = similarity.itemSimilarities(i, range(NUM_ITEMS));
        for (int j = 0; j < NUM_ITEMS; j++) {
            m.setQuick(i, j, similar[j]);
    }
}
Matrix v = new SingularValueDecomposition(m).getV();
Matrix reduced = v.viewPart(0, NUM_ITEMS, 0, 50);

The problem is, SVD is taking forever for NUM_ITEMS > 30. I don't know if there is an issue with data, or with SVD implementation I'm using. The matrix m is symmetrical, could that be an issue? I tried googling "demean matrix mahout" with no results. How should I preprocess it for SVD to work faster? I will need NUM_ITEMS to be about 20.000 - 40.000 in the future. Is this reasonable size for SVD?

EDIT 2:

The problem was the matrix contained a few NaN values, that's why SVD was taking infinite time. After replacing these with 0.0 it works fine for 1000 x 1000 matrix. And my recommendations are working like a charm. I'll still need compute SVD of 20x more rows and columns. If anyone knows what's the easiest way to compute (approximate) SVD of 20.000 x 20.000 dense matrix, probably through some cloud parallel service (?), please let me know.

PS. Thanks for help!

I need to implement recommender which, for a set of items, recommends a new set of items.
Is this a good idea?

Have you looked into Association Rules Mining? If you're open to other procedures this one is first one that came to my mind for recommenders based on sets of items. For those not familiar, this is simple method for retail shops to determine "75% of customers who bought A, B also bought C". Within these algos, the Apriori algo is straight-forward, easy to implement, and may get you what you need.


What is Continuous Ranked Probability Score (CRPS)? I came across some evolution metric at Kaggle:

Continuous Ranked Probability Score (CRPS):

Mathematically,

$C = \frac{1}{199N} \sum_{m=1}^{N} \sum_{n=-99}^{99} (P(y \le n) -H(n - Y_m))^2,$

where P is the predicted distribution, N is the number of plays in the test set, Y is the actual yardage and H(x) is the Heaviside step function $(H(x)=1$ for $x≥0$ and zero otherwise).

Can anyone be kind enough to explain this evolution metric and implementation with Python?

CRPS is in a sense just the mean square error (MSE) of your predicted cumulative density function (CDF) and the true CDF.

The CRPS generalizes the MAE (Mean Absolute Error) to the case of probabilistic forecasts. The CPRS is one of the most widely used accuracy metrics where probabilistic forecasts are involved.

The CRPS is frequently used in order to assess the respective accuracy of two probabilistic forecasting models. In particular, this metric can be combined with a backtesting process in order to stabilize the accuracy assessment by leveraging multiple measurements over the same dataset.

This metric notably differs from simpler metrics such as MAE because of its asymmetric expression: while the forecasts are probabilistic, the observations are deterministic. Unlike the pinball loss function, the CPRS does not focus on any specific point of the probability distribution but considers the distribution of the forecasts as a whole.

Find More Info

Paper

Five ways to calculate CRPS

Using python library properscoring 0.1

pip install properscoring

This library focuses on the closely related Continuous Ranked Probability Score (CRPS) and Brier Score. We like these scores because they are both interpretable (e.g., CRPS is a generalization of mean absolute error) and easily calculated from a finite number of samples of a probability distribution.

We will illustrate how to calculate CRPS against a forecast given by a Gaussian random variable. To begin, import properscoring:

import numpy as np
import properscoring as ps
from scipy.stats import norm

Exact calculation using crps_gaussian (this is the fastest method):

ps.crps_gaussian(0, mu=0, sig=1)

Numerical integration with crps_quadrature:

ps.crps_quadrature(0, norm)

From a finite sample with crps_ensemble:

ensemble = np.random.RandomState(0).randn(1000)
ps.crps_ensemble(0, ensemble)

Weighted by PDF values with crps_ensemble:

x = np.linspace(-5, 5, num=1000)
ps.crps_ensemble(0, x, weights=norm.pdf(x))

Based on the threshold decomposition of CRPS with threshold_brier_score:

threshold_scores = ps.threshold_brier_score(0, ensemble, threshold=x)
(x[1] - x[0]) * threshold_scores.sum(axis=-1)

More at


Multi-Step Forecast for Multivariate Time Series (LSTM) Keras I have been trying to understand how to build LSTM model for multivariate time series forecast using Keras but I am still unsure how to present the data in the correct shape.

My Dataset:
•   5 cities.
•   Each with 2 features. Temperature and humidity.
•   Daily records of the last 10 weeks(Monday, Tuesday, …, Sunday)

What I want to do:

Given Monday’s record of the new week of a city, I'd like to forecast the Temperature and humidity  for the remaining 6 days of that city. I.e. Multi-Step Forecast for Multivariate Time Series. Output shape(6,2)

How I have started off:

For each city, the input shape [(num_samples, num_time_steps, num_features) ] would be  (10, 7, 2).

For 10 weeks, I will have five unique samples(5-cities) with the same shape  (70, 2). So if I stack all vertically I will have (350, 2) or 3D shape (50,7,2).
Then create a supervised series with lag 1, I will have a shape(244, 4)

# Split train/test data.
train on 7-weeks. So input_shape= 5*(7,7,2) = (35,7,2)
test on 3-weeks. . So input_shape= 5*(3,7,2) = (15,7, 2)

What I am confused about:

The above layout seems to disregard the unique nature of each sample. I looked at this but still, a bit confused on how to transform it to a regression model.

I want the network to train each city's data separately as in this pic
I would appreciate any suggestion.
Thanks

You should use Seq2Seq models. Seq2seq models represent, in the RNN family, the best for multistep predictions. More classical RNNs, on the other side, are not that good for predicting long sequences.

If you need to implement a seq2seq model in TensorFlow 2.0 / Keras, each model follows the following structure:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, RepeatVector, Dense, TimeDistributed
from tensorflow.keras.activations import elu

# these are just made up hyperparameters, change them as you wish
hidden_size = 50

seq2seq = Sequential([

    LSTM(hidden_size, input_shape = (input_sequence_length, no_vars)),

    RepeatVector(prediction_length),

    LSTM(hidden_size, return_sequences = True),

    Dense(hidden_size, activation = elu)

    TimeDistributed(Dense(1, activation = elu))

])

and then train it as usual with seq2seq.compile() and seq2seq.fit().

If you want to stack more LSTM() layers on top of each other, simply add them to the model I depicted above. Please keep in mind that LSTM models are very computationally expensice; without a good GPU even this "basic" model could be painfully long to train.

One modification I'd suggest, looking at your image, is to make the LSTM-encoder and -decoder parts of equal size and depth.

Alternatively, you can implement a more classical "Autoencoder-like" architecture, with LSTM() layers for encoding and decoding, and Dense() layers in the middle. However, seq2seq models are the most powerful at the moment.

To my knowledge, the only models more state-of-the-art than this are attention models. The problem is that they are so much state-of-the-art that TensorFlow/Keras doesn't have built-in layers for them, and you'd have to create your own custom layers (it's a pain). The only extensive implementation of attention models I found is from this blog post, but things are going to be very very complicated here. I didn't try to implement one yet.


Why use Variational Autoencoders VAE instead of Autoencoders AE in Anomaly Detection? I have read many papers that recommend using Variational Autoencoders over Autoencoders since they have a more probabilistic approach and the ability to use KL divergence on the latent dimension. But when trying to test both networks I find that the variability of the output in  Variational Autoencoders is reducing the accuracy of the network and I am getting better results when using Autoencoders. I am still working on very simple data and training my network on normal images that do not have any augmentation or changing background.

Does the performance of Variational Autoencoders increase with harder data or is there any other reason to choose it over Autoencoders?
Or do Autoencoders perform better in anomaly detection?

Variational autoencoders encourage the model to generalize features and reconstruct images as an aggregation of those features.  This is what the latent space encodes, a compressed feature vector.

Vanilla autoencoders memorize the input and map to the output without the generalization.  If you want to extrapolate from your dataset, variational is the way to go.


Sales Forecasting - Random Forest - Which features should I use for out of sample forecasting? Sorry for the bad title, I can't find a good one. So I will try to explain what I'm looking for.

I'm doing sales forecasting with a Regression Forest. (Spark - Scala for the technology)
I've worked on some test data and I did my forecast using training data.
But some of the features which I have used can't be employed to forecast the future as they would not be known to me at any given time. For example the numbers of customers of a day, their categories, what kind of advantage they have etc.

Do I have to find others features that will be as useful as these ones or Do I need to perform prediction on these features before my sales forecasting and use the predictions?
Are there any another solutions? Also, what kind of algorithms should I use for the "features forecasting"?

The question is poorly phrased, I've tried to edit it to the best of my abilities. However here are the problems you've stated,

Some of the features which are being used now can't be used later because they might not be known. Will it affect the model?
If they can be used, what type of algorithm can be chosen?

The answer to the first problem, you have to check the accuracy first before making choosing any new features if the rest of your features give a good enough accuracy then there is no need to choose new features.

The second problem, to predict those values of the features you are using now if they are discrete in nature try classification algorithms whichever fits the model best, else try something along the lines of regression if the input values are continuous. And then use these predicted values along with your existing model and check how the accuracy varies.


Why do we use +1 and -1 for marginal decision boundaries in SVM While using support vector machines (SVM), we encounter 3 types of lines (for a 2D case). One is the decision boundary and the other 2 are margins:

Why do we use $+1$ and $-1$ as the values after the $=$ sign while writing the equations for the SVM margins? What's so special about $1$ in this case?

For example, if $x$ and $y$ are two features then the decision boundary is: $ax+by+c=0$.
Why are the two marginal boundaries represented as $ax+by+c=+1$ and $ax+by+c=-1$?

It's important for the optimization formulation of the SVM that $y_i=\{-1,1\}$ which is why it makes sense to also output $y=\{-1,1\}$. If we look at the soft-margin linear SVM we want to minimize:

$\left[\frac{1}{n}\sum_{i=1}^n\max{(0,1-y_i(w\cdot x_i+b))}\right]+\lambda\| w\| ^2$

The $y_i$ is either +1 or -1 which flips the hyperplane in the soft-margin definition of the problem.


What is the best Keras model for multi-class classification? I am working on research, where need to classify one of three event WINNER=(win, draw, lose)

WINNER  LEAGUE  HOME    AWAY    MATCH_HOME  MATCH_DRAW  MATCH_AWAY  MATCH_U2_50 MATCH_O2_50
3         13    550      571          1.86        3.34        4.23       1.66     2.11
3         7     322     334           7.55         4.1         1.4       2.17     1.61

My current model is:

def build_model(input_dim, output_classes):
    model = Sequential()
    model.add(Dense(input_dim=input_dim, output_dim=12, activation=relu))
    model.add(Dropout(0.5))
    model.add(Dense(output_dim=output_classes, activation='softmax'))
    model.compile(loss='categorical_crossentropy', optimizer='adadelta')
    return model

I am not sure that is the correct one for multi-class classification
What is the best setup for binary classification?

EDIT: #2 - Like that?

model.add(Dense(input_dim=input_dim, output_dim=12, activation='sigmoid'))
model.add(Dropout(0.5))
model.add(Dense(output_dim=output_classes, activation='softmax'))
model.compile(loss='binary_crossentropy', optimizer='adadelta')

Your choices of activation='softmax' in the last layer and compile choice of loss='categorical_crossentropy' are good for a model to predict multiple mutually-exclusive classes.

Regarding more general choices, there is rarely a "right" way to construct the architecture. Instead that should be something you test with different meta-params (such as layer sizes, number of layers, amount of drop-out), and should be results-driven (including any limits you might have on resource use for training time/memory use etc).

Use a cross-validation set to help choose a suitable architecture. Once done, to get a more accurate measure of your model's general performance, you should use a separate test set. Data held out from your training set separate to the CV set should be used for this. A reasonable split might be 60/20/20 train/cv/test, depending on how much data you have, and how much you need to report an accurate final figure.

For Question #2, you can either just have two outputs with a softmax final similar to now, or you can have final layer with one output, activation='sigmoid' and loss='binary_crossentropy'.

Purely from a gut feel from what might work with this data, I would suggest trying with 'tanh' or 'sigmoid' activations in the hidden layer, instead of 'relu', and I would also suggest increasing the number of hidden neurons (e.g. 100) and reducing the amount of dropout (e.g. 0.2). Caveat: Gut feeling on neural network architecture is not scientific. Try it, and test it.


(RL Curiosity) - "Exploration by Random Network Distillation" - what's the benefit? Curiosity-Driven learning motivates the agent to explore unseen states. It does it by rewarding the agent more when its expectation differs from the actual next state. (orange color on the diagram). This extra reward is called "intrinsic reward" $r^i$ which is added to the usual reward (we call it with a new name "extrinsic reward" $r^e$).  Here is a video, explaining this architecture

But this causes issues in states that change unpredictably ("noisy TV screens" or tree leaves that move because of the wind). The agent will then get stuck, contemplating them forever, because the ExpectedState - observedState is very high, resulting in high reward.

As I understod, the authors tackle the problem by intruducing the Inverse-Model (blue on the diagram).
It tries to predict an action that was taken to get from currentState to the observedState.

This teaches the agent to only consider features that are relevant to the transition from currentState to nextState, and ignore stuff like noisy TV-screens.

Question:

Why is there a paper called Exploration by Random Network Distillation that builds on top of the original paper? To me, it seems to solve this same issue.

  Namely we
  predict the output of a fixed randomly initialized neural network on the current observation

Wasn't the problem already tackled by the Inverse Model (blue on the diagram) in the original paper?

When using Random network distillation, the agent tries to predict the embedding
of a current state, not the next state as I thought initially.

If the agent sees a random frame on TV, initially it will be receiving high “intrinsic” rewards, because it doesn’t match the output of the fixed random target network. An embedding produced by the target network is always unpredictable (although repeatable), meaning the online network will need to overfit, which is what we want in this case. This approximates the state-visit count. So RND is an interesting form of curiosity. Is more useful than vanilla?

Both papers are attempting to incentivize exploration for reinforcement learning agents.

Curiosity-driven Exploration by Self-supervised Prediction does that by adding an inverse model, increases the complexity of the loss function of the overall model.

Exploration by Random Network Distillation does that having two separate models. One of the model tracks and learns the value of exploration.

There are pros and cons to both approaches. It is easier to train a single loss function/model. However, two models that learn to interact are more flexible.


Algorithm for generating rules for classifying documents Im looking for an algorithm that can deduct a set of rules based on a dataset of "training documents" that can be applied to classify a new unseen document. The problem is that I need these rules to be viewable by the user in the form of some string representation. For example, the algorithm found that documents have a minimum word count of 1000 and that there are 4 citations in each document. The key is that these rules must be deducted by a algorithm. An example of this in practice would be:

Document 1 contains 890 words and only 2 citations

I need it to return something like:

- You should add more words to make it better
- Add more citations to prove your point

I think a simplest way is like this: convert the document to a vector like "bag of words" or "n-grams", then apply algorithm which can be used to derive rules Decision tree or LEM2.


Feature engineering suggestion required I am having a problem during feature engineering. Looking for some suggestions. Problem statement: I have usage data of multiple customers for 3 days. Some have just 1 day usage some 2 and some 3. Data is related to number of emails sent / contacts added on each day etc.

I am converting this time series data to column-wise ie., number of emails sent by a customer on day1 as one feature, number of emails sent by a customer on day2 as one feature and so on. But problem is that, the usage can be of either increasing order or decreasing order for different customers.

ie., example 1: customer 'A' --> 'number of emails sent on 1st . day' = 100 . ' number of emails sent on 2nd day'=0

example 2: customer 'B' --> 'number of emails sent on 1st . day' = 0 . ' number of emails sent on 2nd day'=100

example 3: customer 'C' --> 'number of emails sent on 1st . day' = 0 . ' number of emails sent on 2nd day'=0

example 4: customer 'D' --> 'number of emails sent on 1st . day' = 100 . ' number of emails sent on 2nd day'=100

In the first two cases => My new feature will have "-100" and "100" as values. Which I guess is good for differentiating. But the problem arises for 3rd and 4th columns when the new feature value will be "0" in both scenarios Can anyone suggest a way to handle this.

One way to handle this:

I can add "No change" in those scenarios, but I am confused about one thing. If I do that, I will have to make the new feature as categorical, which is not ideal as the other values will be continuous.

Instead, I can have absolute values in the new feature and indicate the trend as "+1" or increasing "-1" for decreasing "no change" for no change and "0" if both the values have been "0". Would that be a good approach though?

The end goal is to predict if a user would continue using the application or not. So it basically would be a two-class model. And I would want to capture even the scale of usage i.e., "A user sending 100 emails every day" should be different from "B user sending 10000 emails every day"

Well, you want to identify change in usage you could try something like:

$$ f(day_1,day2) = \frac{day_2-day_1 + \delta}{||day_2-day_1+\delta||} \times \Biggr|\Biggr|\frac{day_2+day_1}{(day_2+day_1+1)(day_2-day_1+1)}\Biggl|\Biggl| $$

where $\delta$ is the eps of your machine (minimum value needed to be summed to differ it from other floats)

that will give you
$$f(100,0) \approx -98.02$$
$$f(0,100) = 100$$
$$f(100,100) \approx 0.995$$
$$f(0,0) = 0$$

You can look at my experiment here

This will map all non-changes from $[0,1]$ where $f(0,0)$ maps to $0$ and $f(\infty,\infty)$ maps to $1$

Where is it from? Just tuned the function manually. But I think this might suffice for your application

Explaining the idea

You want to have a feature that packs a lot of information:
 - Is the usage greater than zero?
 - Is it increasing or decreasing?
 - If it is stalled, how much is the usage?

Well, your usage vary in integer values so you can map the entire non-changing but above 0 case to a previously non-used interval.

The function above will map in $[0,1]$ all non-changing possibilities, in a exponential kind of way ($a^{(-\frac{1}{usage})}$) also you can extract the actual value from positive changes and the approximate value for negative change (been a better approximation when the drop is high)

This is not the perfect scenario but it is the maximum information I could compress into 1 variable with little loss.


Orange3 Image Classification Is the attached workflow correct in case of training and testing with different data sets? Is the attached workflow considered "transfer learning" since my images are not related to the images that the network was pre-trained?

Your workflow is correct and you are doing a type of transfer learning where only the last layer is fine-tuned. Image embedding widget calculates features that are the activations of a penultimate layer of Inception-v3, VGG or Painters network. All of those networks are trained on the different set of images. VGG and Inception-v3 are trained on ImageNet dataset and Painters network is trained on the set of artworks. In the transfer learning you are implementing with the workflow you fine-tune all of the connected classifiers for your images.


Classification - get some label value to check how close to another class (Python) I am doing text classification in python with 3 alghoritms: kNN, Naive Bayes and SVM. I have 3 classes - easy, medium and hard. The accuracy is quite fine. Is there a way to check for new text its exact value? After label encoding 0 is easy, 1 is medium and 2 is hard. So base on new text for example it is classified as medium but I want to know how close it was to easy/hard. Some of my code snippets:

Train_X, Test_X, Train_Y, Test_Y = model_selection.train_test_split(
    df['tokens'], df['class'], test_size=0.3, random_state=42)

Encoder = LabelEncoder()
Train_Y = Encoder.fit_transform(Train_Y)
Test_Y = Encoder.fit_transform(Test_Y)

Tfidf_vect = TfidfVectorizer(max_features=35)
Tfidf_vect.fit([' '.join(arr) for arr in df['tokens']])
Train_X_Tfidf = Tfidf_vect.transform([' '.join(arr) for arr in Train_X])
Test_X_Tfidf = Tfidf_vect.transform([' '.join(arr) for arr in Test_X])
Naive = naive_bayes.MultinomialNB()
Naive.fit(Train_X_Tfidf, Train_Y)
predictions_NB = Naive.predict(Test_X_Tfidf)
print(round(accuracy_score(predictions_NB, Test_Y)*100, 2))

Now when I use Naive.predict() I get 0, 1 or 2. Is there a way to get the EXACT value for example 0,5897237489 which is 1 but I see that it is closer to 0 than 2

If it is not possible with NB, I used SVM and kNN as well:

SVM = svm.SVC(C=1.0, kernel='linear', degree=3, gamma='auto')
SVM.fit(Train_X_Tfidf, Train_Y)
predictions_SVM = SVM.predict(Test_X_Tfidf)
print(round(accuracy_score(predictions_SVM, Test_Y)*100, 2))

KNN = KNeighborsClassifier(n_neighbors=3)
KNN.fit(Train_X_Tfidf, Train_Y)
predictions_KNN = KNN.predict(Test_X_Tfidf)
print(round(accuracy_score(predictions_KNN, Test_Y)*100, 2))

The problem is that Naive Bayes is a classification algorithm that outputs the probability of an input belonging to a class. So you cannot use Naive Bayes to achieve what you want. To get 'in-between' values you need to use a regression algorithm instead. I'm not sure what library you're using, but try to use regression versions Knn and SVM models instead.


Why don't tree ensembles require one-hot-encoding? I know that models such as random forest and boosted trees don't require one-hot encoding for predictor levels, but I don't really get why. If the tree is making a split in the feature space, then isn't there an inherent ordering involved? There must be something I'm missing here.

To add to my confusion I took a problem I was working on and tried using one-hot encoding on a categorical feature versus converting to an integer using xgboost in R. The generalization error using one-hot encoding was marginally better.

Then I took another variable and did the same test, and saw the opposite result.

Can anyone help explain this?

The encoding leads to a question of representation and the way that the algorithms cope with the representation.

Let's consider 3 methods of representing n categorial values of a feature:

A single feature with n numeric values.
one hot encoding (n Boolean features, exactly one of them must be on)
Log n Boolean features,representing the n values.

Note that we can represent the same values in the same methods. The one hot encoding is less efficient, requiring n bits instead of log n bits.
More than that, if we are not aware that the n features in the on hot encoding are exclusive, our vc dimension and our hypothesis set are larger.

So, one might wonder why use one hot encoding in the first place?

The problem is that in the single feature representation and the log representation we might use wrong deductions.

In a single feature representation the algorithm might assume order. Usually the encoding is arbitrary and the value 3 is as far for 3 as from 8. However, the algorithm might treat the feature as a numeric feature and come up with rules like "f < 4". Here you might claim that if the algorithm found such a rule, it might be beneficial, even if not intended. While that might be true, small data set, noise and other reason to have a data set that mis represent the underlying distribution might lead to false rules.

Same can happen in logarithmic representation (e.g., having rules like "third bit is on). Here we are likely to get more complex rules, all unintended and sometimes misleading.

So, we should had identical representations, leading to identical results in ideal world. However, in some cases the less efficient representation can lead to worse results while on other cases the badly deduce rules can lead to worse results.

In general, if the values are indeed very distinct in behaviour, the algorithm will probably won't deduce such rule and you will benefit from the more efficient representation. Many times it is hard to analyze it beforehand so what you did, trying both representations, is a good way to choose the proper one.


Why could my SVR prediction be offset, but still follow the actual pattern? I am running an SVR prediction on some time series data, and I am receiving this weird offset between my actual and predicted values.

I found this SVM Regression lag post, that mentions adding a lag of 2 data points behind, instead of one.  However, I am not sure how to incorporate that into my code (which I've included below).

Does anyone have any ideas on why my predicted vs. actual is offset in this manner?

My code is as follows:

#! /usr/bin/python

import math
import statistics
import visualizer
import numpy as np
from datagen import constructData
from sklearn import svm

# Applies Support Vector Regression to the electricity dataset,
# prints out the accuracy rate to the terminal and plots
# predictions against actual values
def suppVectorRegress():

    kernelList = ["linear","rbf",polyKernel]
    names = ["linear","radial basis","poly"]
    preds = []

    # Retrieve time series data & apply preprocessing
    data = constructData()

    cutoff = len(data)-30
    xTrain = data[0][0:cutoff]
    yTrain = data[1][0:cutoff]
    xTest = data[0][cutoff:]
    yTest = data[1][cutoff:]

    # Fill in missing values denoted by zeroes as an average of
    # both neighbors
    statistics.estimateMissing(xTrain,0.0)
    statistics.estimateMissing(xTest,0.0)

    # Logarithmically scale the data
    xTrain = [[math.log(y) for y in x] for x in xTrain]
    xTest = [[math.log(y) for y in x] for x in xTest]
    yTrain = [math.log(x) for x in yTrain]

    # Detrend the time series
    indices = np.arange(len(data[1]))
    trainIndices = indices[0:cutoff]
    testIndices = indices[cutoff:]
    detrended,slope,intercept = statistics.detrend(trainIndices,yTrain)
    yTrain = detrended

    for gen in range(len(kernelList)):

        # Use SVR to predict test observations based upon training observations
        pred = svrPredictions(xTrain,yTrain,xTest,kernelList[gen])
        # Add the trend back into the predictions
        trendedPred = statistics.reapplyTrend(testIndices,pred,slope,intercept)
        # Reverse the normalization
        trendedPred = [np.exp(x) for x in trendedPred]
        # Compute the NRMSE
        err = statistics.normRmse(yTest,trendedPred)

        print ("The Normalized Root-Mean Square Error is " + str(err) + " using kernel " + names[gen] + "...")

        preds.append(trendedPred)

    names.append("actual")
    preds.append(yTest)

    # Change the parameters 2017,2,1 based on the month you want to predict.
    visualizer.comparisonPlot(2017,2,1,preds,names,plotName="Support Vector Regression Load Predictions vs. Actual",
        yAxisName="Predicted Kilowatts")

# Construct a support vector machine and get predictions
# for the test set
# Returns a 1-d vector of predictions
def svrPredictions(xTrain,yTrain,xTest,k):
    clf = svm.SVR(C=2.0,kernel=k)
    clf.fit(xTrain,yTrain)
    return clf.predict(xTest)

# A scale invariant kernel (note only conditionally semi-definite)
def polyKernel(x,y):
    return (np.dot(x,y.T)+1.0)**0.95

if __name__=="__main__":
    suppVectorRegress()

Two things that come to my mind:

A) - have you compared the distributions with a Kolgomorov Smirnov Test or similar - (aka: are they really the same. Sure there is the big spike but besides its not that similar (by eye))

B) - Are training and test in the same magnitudes for the input data? If they are not this might confuse your SVR.


Orange 3 - Same prediction for all of my data when using impute widget When I use a linear or random forest regression to train from data, I can make predictions with test data. I have a problem when I introduce the "impute" data widget in my pipeline to fill in the missing values from the training data. My predictions then are all the same for all rows, even though the data table is correct. I tried to swap the impute widget for the pre-processor and I have the same behaviour. Below is my graph:

This works correctly:

Adding an impute widget and all my predictions become the same values:

When imputing, certain characteristics of the data have to be recorded. E.g. when you impute missing data by mean value parameter $\mu$, you have to record that $\mu$ from training data in order to be able to reapply the exact same transformation on the test data.

Thus you only need to connect your training data to the Impute widget, while the testing data will be imputed with the same parameters when the two domains are matched (in Predictions).

If you connect testing data to its own Impute widget, it gets imputed with its own, testing-data-specific parameters, which is, as you have discovered, incorrect.

See also: Difference between fit and fit_transform in scikit_learn models?


How important is the balance between the components of a loss value? I'm training a neural network for predicting the location of a single object, so the prediction consists of 4 values: x, y, width and height. Without weight decay, the loss starts at around 0.3. If I add weight decay with a factor of 0.004, the loss jumps to around 22. Does this mean the network will give more importance to minimizing the weights over finding good coordinates? How important is the balance between the components of a loss function?

The cost function controls the algorithm completely - the new regularization from weight decay is likely responsible for the jump in loss. If you only added a small amount of regularization and the loss ballooned, you were likely very overfit.

The solution is cross validation. The only way to know the optimal amount if regularization is to see what does the best on held-out data.


Does a Keras checkpointer save the best weights when using chekpoints with restore_best_weights? I m training a sequence model in Keras using the tensorflow backend.  I've also included some callbacks to save checkpoints and revert to best weights if the model starts to overfit (which it will).

My question - when fitting using this set of callbacks, does the final checkpoint contain the version of the model with the best weight?  I know that the weights in classif_model will revert but I'm not sure if that also applies to the final saved state.

from keras import callbacks as kc

classif_model = my_model(input_shape)

# Set up callbacks
checkpointer = kc.ModelCheckpoint(filepath='results/'+name+'.h5', verbose=0)
earlystopping = kc.EarlyStopping(monitor='val_loss', patience=patience, restore_best_weights = True)
callbacks = [checkpointer, earlystopping]

# train the model
hist = classif_model.fit(x = X_tr, y = Y_tr, epochs = epochs, batch_size = batch_size,
                         callbacks = callbacks, validation_data = (X_val, Y_val),
                         verbose = 0)

It will if you set the save_best_only flag in your checkpoint callback definition:

ModelCheckpoint(filepath, monitor='val_loss', save_best_only=True)

From the docs:

  save_best_only: if save_best_only=True, the latest best model
  according to the quantity monitored will not be overwritten.


Overfitting with text classification using Transformers I am trying to make a binary text classification model by using the encoder part of the transformer and then using its output to feed into an LSTM network. However, I am not able to achieve good accuracy on both the training set (92%) and the validation set (72%). Is my approach correct? Please tell me a better way to design the model and improve accuracy.

Your model is overfitting. You should try standard methods people use to prevent overfitting:

Larger dropout (up to 0.5), in low-resource setups word dropout (i.e., randomly masking input tokens) also sometimes help (0.1-0.3 might be reasonable values).
If you have many input classes, label smoothing can help.
You can try a smaller model dimension.

If you use a pre-trained Transformer (such as BERT), you, of course, cannot change the model dimension. In that case, you can try to set a much smaller learning rate for fine-tuning BERT than you use for training the actual classifier.


How is the min_rank used with the flights database I'm exploring the https://r4ds.had.co.nz/transform.html#add-new-variables-with-mutate r for data science handbook and don't really understand the min_rank() operator.  Doing the exercises it asks to Find the 10 most delayed flights using a ranking function. How do you want to handle ties? Carefully read the documentation for min_rank().

here is the code that I performed that doesn't work. Please explain:

min_rank(flights, dep_delay)

You can put the results of min_rank in to a new column, e.g.:

f2 <- flights %>%
  mutate(rank = min_rank(dep_delay))

Where showing just the selected rows:

f2 %>%
  select(dep_delay, rank)

You get this:

# A tibble: 336,776 x 2
   dep_delay   rank
       <dbl>  <int>
 1         2 208140
 2         4 219823
 3         2 208140
 4        -1 164763
 5        -6  48888
 6        -4  94410
 7        -5  69589
 8        -3 119029
 9        -3 119029
10        -2 143247
# ... with 336,766 more rows

Here is another example where you can see all the rows.

aa <- tibble(y = c(9, 8, 3, 4, 5, 7, 6),
             x = c(1, 2, 3, 4, 4, 4, 5))

aa %>%
  mutate(rank = min_rank(y))

Which gives us:

# A tibble: 7 x 3
      y     x  rank
  <dbl> <dbl> <int>
1     9     1     7
2     8     2     6
3     3     3     1
4     4     4     2
5     5     4     3
6     7     4     5
7     6     5     4


Can training examples with almost the same features but different output cause machine learning classification algorithms to perform poorly? We usually filter out features (columns) that have low correlation or no significant impact on target variable. How would an algorithm, being trained with high dimensional data set (let’s say, more than thousands of features) contain rows with very high correlation but have different target variable, perform? Wouldn’t it make the ML algorithm confused in classification task?

Let me give a simple example to explain what I mean. Assume, we are given the price of a car and the task is to classify it as either of ‘Cheap Car’, ‘Budget Car’, ‘Luxury Car’, and ‘Elite Car’. Further assume, the distance between two rows is generally expected to be greater than 1000. For example, if a row describes a Car with price 1000, the next higher level car in our classification expected to be at least 2000. What if there is some anomaly in data set like a car with price 1000 classified as ‘Cheap’ whereas a car with price 1050 classified as ‘Elite’. That is grossly wrong. We eliminate irrelevant features. Shouldn’t there be something to eliminate confusing training examples?

The answer is yes, highly similar instances in your dataset that have different target classes will cause your model to perform poorly.

The reason for this is at the core of how all classification machine learning algorithms work. The goal of a classifier is find a function which can separate the two classes. Thus, if these two classes are very mixed then the probability of making a classification error increases and thus you will lose precision in your resulting classification.

One method to correct this problem is to add more features to your dataset. You should try to find features which will distance the distributions of these two classes. For example if classifying cats and dogs, it would not be a good idea to use features such as: number of legs, number of eyes, etc. This will cause the classes to be indistinguishable. Try adding features such as: weight, frequency of cry, etc. This can be difficult to do often as collecting additional data is expensive. You can also try to transform your data to a new feature space. A transformation mapping your features to a different space can cause their distributions to distance themselves.


Yelp Dataset Archives? The Yelp Dataset Challenge (https://www.yelp.com/dataset_challenge) releases data for a handful of cities each year. I'd like to analyze some cities from past years. Is this data archived anywhere?

It is my understanding that the dataset gets updated / extended every year. It started with Phoenix, then they added Las Vegas, Madison, Waterloo and Edinburgh, and so on, for a total of 11 cities by 2017.

Basically, this year's dataset should also include all cities from previous years.


I'm getting same MAE value in decision tree model for maxdepth value from 1 to 10 R code :

library(tidyverse) # utility functions
library(rpart) # for regression trees
library(randomForest) # for random forests
library(modelr)

split_data = resample_partition(melb_data,c(test=.3,train=.7))

get_mae <- function(maxdepth, target, predictors, training_data, testing_data){

predictors <- paste(predictors, collapse="+")
formula <- as.formula(paste(target,"~",predictors,sep = ""))

model <- rpart(formula, data = training_data,
             control = rpart.control(maxdepth = maxdepth))

mae <- mae(model, testing_data)
return(mae)
}

target <- "Price"
predictors <-  c("Rooms","Bathroom","Landsize","BuildingArea",
             "YearBuilt","Lattitude","Longtitude")

for(i in 1:10){
mae <- get_mae(maxdepth = 3, target = target, predictors = predictors, training_data = split_data$train, testing_data = split_data$test)
print(glue::glue("Maxdepth: ",i,"\t MAE: ",mae))
}

OUTPUT :

Maxdepth: 1  MAE: 356628.697268696
Maxdepth: 2  MAE: 356628.697268696
Maxdepth: 3  MAE: 356628.697268696
Maxdepth: 4  MAE: 356628.697268696
Maxdepth: 5  MAE: 356628.697268696
Maxdepth: 6  MAE: 356628.697268696
Maxdepth: 7  MAE: 356628.697268696
Maxdepth: 8  MAE: 356628.697268696
Maxdepth: 9  MAE: 356628.697268696
Maxdepth: 10 MAE: 356628.697268696

In this line:

for(i in 1:10){mae <- get_mae(maxdepth = 3, target = target, predictors = predictors, training_data = split_data$train, testing_data = split_data$test)}

You are setting maxdepth to 3 for every i. You have to change it for:

for(i in 1:10){mae <- get_mae(maxdepth = i, target = target, predictors = predictors, training_data = split_data$train, testing_data = split_data$test)}


How to import large datasets into Google Colab? Does anyone know a (free) method for importing large datasets into Google Colab, of multiple GB? Github is severely limited, uploading a folder to google drive takes a long time.

One option is that you can download the dataset into your system and save it in an easily accessible directory. Then, run the following codes:
from google.colab import files

data = files.upload()

After running the above line, you will get a Choose File button where you can directly browse your system and choose your file.
Added the screenshot for your reference:


How to use sklearn train_test_split to stratify data for multi-label classification? I am attempting to mirror a machine learning program by Ahmed Besbes, but scaled up for multi-label classification. It seems that any attempt to stratify the data returns the following error: The least populated class in y has only 1 member, which is too few. The minimum number of labels for any class cannot be less than 2.

In my data set, I have 1 column which contains clean, tokenized text. The other 8 columns are for the classifications based on the content of that text. Just to note, column 1 - 4 have significantly more samples than 5 - 8 (more obscure classifications derived from the text).

Here is a generic sample from my code:

x = data['cleaned_text']
y = data[['car','truck','ford','chevy','black','white','parked', 'driving']]

x_train, x_test, y_train, y_test = train_test_split(x,
                                                    y,
                                                    test_size=0.1,
                                                    random_state=42)

print(x_train.shape, x_test.shape, y_train.shape, y_test.shape)

Output: (6293,) (700,) (6293, 8) (700, 8)

Adding stratify=y to train_test_split returns the error previously mentioned. Even when I limit y to just one column, I still get the error.

How can I stratify the data so that I give the program a fair look in the training set?

The error you're getting indicates it cannot do a stratified split because one of your classes has only one sample. You need at least two samples of each class in order to put one in the training split and one in the test split. You should examine what your class breakdown is to find the culprit.


What's the meaning of precomputed features? When i learn about deep learning, I found dataset with precomputed features form. Link (http://cs.stanford.edu/people/karpathy/deepimagesent/coco.zip). What's the different with usual dataset?

Pre-trained - You save your trained network with all its weight on a big and reliable dataset. Then use again for a similar requirement with further training on some additional data. This is a typical transfer-learning.

Precomputed - Pre-compute is the process of storing the output of all your layers before the last one re-use for some other purpose on the same dataset.

It makes the subsequent (beyond the first) training faster. Layer other than the last one will not change with new data.

Since only the last (classifier) layers change during training but not the earlier layers, we can convert the images to feature vectors just once and then train the classifier on these feature vectors instead of on the images.

Reference - 1    Reference - 2


Modelling on one Population and Evaluating on another Population I am currently on a project that will build a model (train and test) on Client-side Web data, but evaluate this model on Sever-side Web data.  Unfortunately building the model on Server-side data is not an option, nor is it an option to evaluate this model on Client-side data.

This model will be based on metrics collected on specific visitors.  This is a real time system that will be calculating a likelihood based on metrics collected while visitors browse the website.

I am looking for approaches to ensure the highest possible accuracy on the model evaluation.

So far I have the following ideas,

Clean the Server-side data by removing webpages that are never seen Client-side.
Collect additional data Server-side data to make the Server-side data more closely resemble Client-side data.
Collect data on the Client and send this data to the Server.  This is possible and may be the best solution, but is currently undesirable.
Build one or more models that estimate Client-side Visitor metrics from Server-side Visitor metrics and use these estimates in the Likelihood model.

Any other thoughts on evaluating over one Population while training (and testing) on another Population?

If the users who you are getting client-side data from are from the same population of users who you would get server-side data from. If that is true, then you aren't really training on one population and applying to another. The main difference is that the client side data happened in the past (by necessity unless you are constantly refitting your model) and the server side data will come in the future.

Let's reformulate the question in terms of models rather than web clients and servers.

You are fitting a model on one dataset and applying it to another. That is the classic use of predictive modeling/machine learning. Models use features from the data to make estimates of some parameter or parameters. Once you have a fitted (and tested) model, all that you need is the same set of features to feed into the model to get your estimates.

Just make sure to model on a set of features (aka variables) that are available on the client-side and server-side. If that isn't possible, ask that question separately.


Regression and Classification, which is better in financial market price prediction? I want to use a model to trade in finanical market.
which i have several features, like macd, rsi, or other common features.
and my target is to make a tradeable predict in every time point.
so my target can be:

yield in a fixed time laster, like, 30 min. yt = close(t+ws) - close(t)
futures price direction, which only can be 1(price up in the future) -1 (price down in the future)

these are difference between regression and classification.
which one you think is better, and, any suggestions about this problem?
Thanks

From my personal experience, I think what matters the most in terms of return is how good your risk management is. You can use both regression or classification, but all approaches have some errors associated with it, so in the long run, it comes down to how you can manage incorrect predictions.
So you can start with any model, say you go for classification and you achieve 70% accuracy. then you use your model to predict on some historical data (that for sure you have not used during trading) and you build some trading strategy using your forecasts.
Then, you can use something like Pythons Backtrader and check how are you doing in terms of return on historical data. Sometimes its not enough to correctly predict price movement in 9 out of 10 cases, because your one loss can be bigger than all the profitable bets and you also need to take into account fees that will burn your profit. Thats a lot of issues you need to think about before putting this model into production.
And even using this approach you should be aware that historical success of your model+strategy does not guarantee that it will be the same in the future.
To sum it up: I think it comes down to the strategy and not the model itself.


Fitting probabilities in scikit-learn RandomForestClassifier This is a newbie questions, so please bear with me. Given this random forest model:
from sklearn.ensemble import RandomForestClassifier

X = [ [2,1,1,1], [2,0,2,1], [3,1,1,1] ]
y = [ 0, 1, 2 ]

regr = RandomForestClassifier(n_estimators=200, max_depth=5)
regr.fit(X, y)

X_test = [ [3, 1, 1, 1] ]
y_result = regr.predict_proba(X_test)
print('y_result:' , y_result )

The result is:
y_result: [[0.26 0.05 0.69]]

I understand that these are the probabilities of the first, second and third value, or 0 = 26%, 1 = 5% and 2 = 69%.
Question: if the test set is [3, 1, 1, 1] and it fits to the value 2, why do I get 69% probability of 2 instead of 100%?

Well, based on how this algorithm works, here is why :
You want to build 200 trees, with a max depth of 5 levels. Each tree is a decision tree, train with a subset of your data. To answer your question, it doesn't fit to the value 2 because that's not how it works. What's happening is that 52 trees predicted the value 0, 10 predicted the value 1, and 138 predicted the value 2. So most of the trees predicted the value 2 You asked for the probability (using predict_proba(X_test)). If you want only one value, don't use predict_proba but predict(X_test). But it will only be the reflect of the probabilities you got.


Is autocorrelation of residuals a problem in machine learning? Let's assume I have a random forest model and the residuals of the model are autocorrelated. Is this a problem?
As an example, let's assume I have two different random forest models, A and B, with a similar predictive performance. The residuals of model A are less autocorrelated than the residuals of model B. Should I prefer model A?

Yes, autocorrelation in residuals is a problem, but this is essentially because it is a clear illustration that there was more learnable information in the process you are modelling but your model missed it.
In the unlikely event that you have two equally performant models but one shows significant autocorrelation (you can test for this using the Durbin-Watson test as suggested in Noah Weber’s answer), this suggests neither model is working as well as we might hope (the autocorrelated model has failed to predict some predictable patterns and the other model is failing in some other way as its predictive power isn’t any better).
If you have two models that have different residuals but both are beating a naïve baseline, you’ve probably got models that will ensemble well.


Getting sentence embeddings with sentence_transformers I have a text column in my data frame which contains paragraph(s) having multiple and variable sentences in each instance/example/row of the dataframe. Then, I created the sentence tokens of that paragraph using sent_tokenizer of nltk and put it into another column.
So my data frame looks like this:
index       text                                              class

0           ["Hello i live in berlin", 'I'm xxx']                                                          1
1           ["My name is xx", "I have a cat", "Love is life"]                                              0

now when I'm using:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('distilbert-base-nli-stsb-mean-tokens')
sentences = df['text']
sentences = sentences.tolist()
embeddings = model.encode(sentences)

I'm getting:
TypeError: expected string or bytes-like object

The encode  method is not taking a list of list of sentences as an argument.

The encode method only works with single sentences as strings, i.e., you need to call it for each sentence independently:
embeddings = [model.encode(s) for s in sentences]


GRU and LSTM does not "take risk" predicting I tested LSTM and GRU models to predict the exchange rate between currencies. I do not take the raw price but a the delta with the previous day, so the data is stationnary around zero.
My problem is that my model always predict really close-to-zero values, like if it minimize the risk and does not want to guess wrong. It may be because it underfit but I wanted to be sure that it is not a common issue that I just totally don't know about.
I have really simple architecture, 1 layer of GRU or LSTM (tried both), data from the past 20 years, and using the 20 previous days to predict the next one. Tried to play with LR, dropout and number of epochs yet but this behaviour seems weird to me. Maybe I miss something?

Thanks!

This is probably because the deltas you are trying to predict are less than 1, so your loss function (I’m assuming MSE) isn’t working as expected.
Squaring an error less than 1 will make it even smaller, so your model is not currently motivated to leave the cosy local minimum of the naïve strategy of always predicting delta as zero.
I recommend rescaling your deltas by multiplying them by 100 and then using MSE as a loss function.


What methods to create singular content classification from inconsistent inbound info? I am attempting to aggregate professional profile info from multiple sources, imposing a consistent taxonomy. Specifically, the current problem is how to impose a preferred taxonomy on profiles with inconsistent or absent in-bound taxonomy terms.

Primary source of profile info is biography pages on people's employer websites. Some of those sites choose to state employees' multiple specialist topics, some make only narrative biographies available, some both. I have collected all available info, using Python's Scrapy, in to CSV files - one per company, people are rows  - where available, topics at my end now themselves reside in a comma-separated field/string.

Example: in one sheet, cell S7 is: "Analytics Applications,Big Data,Cognitive Computing,Competitive Intelligence,eDiscovery,Enterprise Content Management (ECM),Information Architecture,Market Research,Product Information Management (PIM)"

The problem is severalfold:

Taxonomy terms across companies are inconsistent (eg. "Cognitive Computing" in the above example may, to another company, be "AI").
Some companies use far too many terms in total (eg. one company alone uses approx 450 tags in total).
Often, none are available at all.
As biography narratives describe more than just employees' specialist topics (eg. education and upbringing background), their usefulness in automation may be questionable.

My goal is to create a taxonomy that categorises all the collected person bios in a much more harmonious, consistent and briefer fashion.

System setup is PHP/MySQL/WordPress. Profile CSVs are imported in to WordPress, and the system has the ability to perform PHP functions on imported content (not just on the info in WordPress after import, but during import via PHP).

Total profiles count is approx 4,500, so manual taxonomisation is unappealing. So I have examined AI/machine learning techniques. I am not strictly a developer and certainly not a data scientist or mathematician.

So far, I have found text classification tests carried out using Aylien and Monkey Learn to yield poor results. In each case, output results are not granular enough, ie. turning in-bound terms of biogs about granular topics like cloud computing infrastructure and data centres in to overly basic terms like "Computers & Internet". Aylien uses the off-the-shelf IPTC NewsCodes taxonomy, and I understand I can use Monkey Learn to train. I like the idea of using a standardised off-the-shelf taxonomy like NewsCodes, but a) the results are questionable, and b) it may not be granular enough for my needs.

At this point, I have decided to draw up my preferred hierarchy of taxonomy terms, approx 230, which should each speak roughly to the swathe of inconsistent in-bound terms and profiles (in other words, correlate to the people's topics). That seemed like an important step, assuming I need to steer this manually. But I'm struggling to grasp how to actually implement that correlation.

So, I am looking for some guidance on best methods.

One idea I am toying with is to put my own preferred taxonomy in to WordPress as taxonomy terms, and, alongside each, put a cluster of terms from the actual source material so that, if one of the related terms is found in a user's inbound data, the term from my preferred taxonomy should be assigned. But I'm not sure whether this is particularly efficient, or even wise.

This is my first time on the Data Science group at StackExchange. I apologise if I have shot wide of the mark here at all.

You could import each companies data into a specific  table and then develop regular expression Scripts two change specific Expressions into your own taxonomy.
https://en.m.wikipedia.org/wiki/Regular_expression


Appropriate statistical test I am working on a project where I have twitter user profiles and their tweets. The users are divided based on their number of followers in two groups (g1 and g2). Then with each user in g1, one user from g2 were matched based on their profile and activity using nearest neighbor (not propensity score). Now I want to do some statistical tests, for example, how differently the sentiment of the tweets changes for these two groups before and after some events. So I have lets say tweets posted within 7 days before and after some date and estimated the mean sentiment scores of all tweets posted by each user in each group. For two groups sample sizes are different (even though they were matched) since not every one posted any tweets within the date range. Now if I want to do a t-test to see if people in g1 has larger positive change in sentiment than g2 after the reference date. I have the following questions:

I am doing an independent sample t-test, by treating each user in each group independent. For each person, take the difference (before and after) in mean sentiment scores for both groups, then testing to see if there is significant different across the groups in terms of change in sentiment scores. Is this appropriate or I should do the matched pairs test? I have gone through other posts here but did not find and definitive answer.
The users who did not have and tweet for the time range, is this okay to assign difference in mean zero, or I should exclude them from the samples?

Thanks in advance. Cheers!

Regarding your second question:

  The users who did not have and tweet for the time range, is this okay to assign difference in mean zero, or I should exclude them from the samples?

You essentially have missing data in this case. How you can deal with this will depend on the model you are using, and if it is robust to missing data. If the model can ignore $\mu = 0, \sigma = 0$ values, then try it out. Otherwise, you might want to leave them out as you suggest, or perhaps even impute them with their previous known values.
If you are e.g. using something like an ARIMA model, then it keeps track of a moving average. In this case, using zero values will have an undesired impact (assuming zeros are not common in general).

I'm not sure I understand what you are asking in your first question. What have you tried already? Have you got some results?


In Reinforcement Learning can I randomly assign next_states from the state space to my agent while creating transition set? In Reinforcement Learning, while creating transition samples (state, action, next_state, reward), where:

Agent: The learning agent
Environment: The trainer

The environment gives two feedback to the agent: reward and next state.
Can I as the environment, randomly assign next_states from the total state space to my agent. ?
How can I decide what are the allowed next_state(s) from a given state?

It depends on your environment. For example, If you environment is a straight line. Let's say that the states are 1, 2, 3 ... 10. The agent here can either move left or right. Now, if the agent is at state 4, and it moves right, according to the logic of the environment, next_state has to be 5.

Now, if you include some sort of a condition that, if the agent reaches state 5, the agent will be randomly moved to any other position, then you can return the next_state to be any random number.

Long story short, the next_state is dependent on what the environment is.


Using Decision Tree methodology to identify Independent Variables for Multiple Regression Have access to a dataset with hundreds of variables and millions of cases (American Community Survey).

Need to identify a small, manageable set of Independent Variables (IVs) to use for Multiple Regression.

One way to do this, of course, would be to use applicable theories to identify the IVs.

Was wondering how I could use a data-driven (data-mining?) approach as follows:

Use a Decision Tree to identify impactful (candidate? relevant?) IVs?
And then use these as the IVs in the Multiple Regression?

(Seem to remember reading once, in passing, that this approach to variable reduction is permitted.)

Tried searching on Google for articles that clarify the above, but the search terms are such that I keep getting hits to articles that compare Decision Trees and Multiple Regression.

So, if you know of articles and research papers that describe how to do the above, please leave links below. Also, I would welcome your own original suggestions on how to proceed.

Decision trees are useful for determining nested/interactive relationships between combinations of IVs and a DV.

The model you specified, a multiple regression, presupposes a relationship between the IVs and the DV (e.g. linear).

As you are aware, these models are different. So using a decision tree coupled with some importance measure to find predictive variables won't necessarily provide you with an optimal set of IVs in a regression model.

That being said, it can be a helpful exercise to inform you of non-linear relationships or interaction terms that could be predictive, and which may not be captured by specifying a model such as a multiple regression.

If I were you, I wouldn't solely rely on using decision trees to determine a  set of IVs for a regression model. I would investigate penalized regression methods such as LASSO or ridge regression to help take you from a reduced candidate set of IVs to your final IVs. In addition, you might want to explore associative metrics related to your model specification that might be useful in exploring the relationships in your data, such as information values, chi-square tests, correlations, etc.

This may be helpful:
https://stats.stackexchange.com/questions/47367/decision-tree-as-variable-selection-for-logistic-regression


Using an autoencoder for anomaly detection on categorical data Say a dataset has 0.5% of its features continuous and 99.5% categorical (binary) with ~2400 features in total. In this dataset, each observation is 1 of 2 classes - Fraud (1) or Not Fraud (0). Furthermore, there is a large class imbalance with only 2.6% of examples being Fraud, and the other ~97% of examples being Not Fraud.

Say we want to to predict whether a given example is Fraud or Not Fraud, and we take an anomaly detection approach using autoencoders.

Given the mixed data types in the dataset, in general, will an autoencoder, trained on only the Non Fraud examples, perform well in predicting Fraud examples? Is there any literature to suggest what architectures work best / if some preprocessing should be performed beforehand (scaling and PCA)? I ask because I feel an autoencoder may be hard to train with binary features.

In general an autoencoder should perform well, when it comes to detect fraud examples. Fraud examples should have in theory a much higher reconstruction error.
When it comes to train the autoencoder on binary data, I agree with you that it can be quite challenging. I suggest to take a look at this blog:
https://blog.evjang.com/2016/11/tutorial-categorical-variational.html


Multivariate time series classification I have around 72,000 Multivariate time series (MTS) with four Dimensions and of Length around 3000 milliseconds(not constant). It is (4*3000*72000) DLN. This MTS has tow possible outcomes either Pass or Fail.

How to select a better classifier that suits above criteria?

I have read some implementations of MTS classification. I cannot apply distance measure techniques for classification as the data is too lengthy and is not constant when compare to other MTS.

I am considering to apply below techniques for classification

1) Extract global statistical features from data and apply any classifier(KNN or Random Forest or neural networks)

what would be the best approach. Please guide me better technique and I am also open for other techniques to classify MTS

Following up on the comment about deep learning, with high dimensional time series data you would be much better served with a recurrent-type of deep model. For example, an LSTM is a very good starting point with high-dimensional data. This may be a good place to start: Sequence Classification with LSTM Recurrent Neural Networks in Python with Keras

Although CNNs are very useful for high-dimensional data, when you have a time series, it's best to start with a model that is designed for a time series. A CNN may do well, and you should compare your results to a CNN, but it is not a time series model.


Adoptive(Machine wise) ML packages What kind of Machine Learning/Data Mining packages are available that can easily be scaled on a cluster. For instance H2O is one of them because it's running in Java so it can be easily extended to other machines providing parallelism. What other are popular? or even for general Data munging purposes.

Language wise preferably Python or R. (if other please don't be discouraged to include)

SPARKR, http://spark.apache.org/docs/latest/sparkr.html, is designed to support parallel processing.

Additional resources are summarized here:  https://cran.r-project.org/web/views/HighPerformanceComputing.html


Is it possible to forecast the evolution of cars? Let's say for example that I have a dataset about the cars that a company (e.g. Toyota) produced, over the course of the years 1990 - 2016.
Considering that I have already completed the feature engineering step, is there an algorithm that can output predictions on what attributes future Toyota cars will have?
What I mean is, can we produce output that will have the same features/attributes as the input data-points?
simple example:
For input X0-Xn = [EngineType, #OfWheels, #OfDoors, hasRadio, hasGPS, ProductionYear] --> Produce output
Y = [EngineType, #OfWheels, #OfDoors, hasRadio, hasGPS, ProductionYear]

This can be treated as Multivariate Time Series Forecasting, for this you can look into Vector AutoRegression(VAR). Because there might be some association between attributes which we need to take care of hence we can't treat them as separate time series entities.
If you have found some hierarchical relationship between variable you can also look into hierarchical time series forecasting.


Logistic regression with unbalanced data, scoring based only on rare class I have a dataset off app. 600.000 data points in which 0.2% (1.200 samples) is labelled as signifying a rare event. I want to use logistic regression to help me predict this rare event, but even when I apply weighting, the classification accuracy is poor.
I know that I can rebalance the dataset, but the problem with that is that lots of other weird stuff is going on that signifies various events but which is not indicative of the exact type of event I’m trying to predict. Therefore, rebalancing would itself be a large and complicated task in trying to get representative weirdness into the cropped dataset.
I have found that I get great predictive performance by using a home-made logistic regression classifier that scores based on the error measure below, and then using scipy.optimimize.minimize with method='Powell' to tune the parameters to minimize this measure:
e = 1 - tp/(tp+fp+fn)

for the rare class only, ignoring true negatives.
This lets me include the full dataset (and ignore weights and rebalancing considerations), and it makes it clear in the scoring when the classifier starts erroneously including other types of events that shouldn’t be caught by this classifier (i.e. increasing number of false positives).
The problem is that my home-grown classifier/trainer is naturally much slower to train than the sklearn version, and I’d like to build on mature optimized packages instead. So my question is: Is there a Python implementation of a logistic regression classifier that lets me perform fitting based only on the rare class as described above? It seems that setting the rare class weight arbitrarily high does not make it perform like my home-grown method that explicitly ignores the common class.

Firstly, when you have an imbalanced dataset accuracy is not a good metric to be using (see https://en.wikipedia.org/wiki/Precision_and_recall#Imbalanced_data). You should consider what the ultimate use-case of this model is and what metric is properly capturing the performance of the model considering that use case. For example, when classifying the presence of cancer, false negatives are much more undesirable than false positives so you would want to ensure you are using a metric that captures that appropriately.
In sklearn there is a class_weight parameter of the LogisticRegression model which allows you to essentially weigh misclassifications of different classes differently. Setting this to 'balanced' will automatically adjust this weight to be inversely proportional to the amount of samples of that class in your data which might be beneficial. You may want to adjust this in a custom manner also.
Changing the metric you are evaluating on doesn't change the actual training of the model, so I am guessing that your custom implementation of logistic regression should not function significantly differently to the sklearn version in terms of performance (if it does their may be other issues), it seems you are just using a different metric. There are also a number of other metrics besides accuracy that you can use in sklearn (https://scikit-learn.org/stable/modules/model_evaluation.html#model-evaluation) perhaps consider balanced-accuracy to begin with. It is also not to hard to apply your own custom metric to the results from the sklearn logistic regression model.
Tools such as the classification_report (https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html) and/or confusion matrix (https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html) can also be enlightening when dealing with imbalanced data.


Seaborn distplot and KDE data confusion I'm running through a tutorial to understand the histogram plotting. Given the seaborn tips dataset, by running the sns.distplot(tips.tip); function the following plot is rendered.

Looking at the plot, I don't understand the sense of the KDE (or density curve). The middle column (the one with the lower value) between 2 and 4 doesn't seem to support the shape of the curve.

I have to say that I have little if no understanding on the principle used to plot it, so I would love to hear from somebody more experienced on

What's the added value of the KDE?
What's the process behind the calculation

Also, why using the same dataset with the standard matplotlib I get a slightly different representation (in which the density line above probably fit better)?

The difference is caused by the fact that seaborn.distplot and matplotlib.pyplot.hist use different defaults for the number of bins. The bins are ranges of values for which the number of observations are counted before being plotted. For more information on what bins are check the Wikipedia page for histograms.

In your example, the standard matplotlib plot has bigger bins than the seaborn plot since it uses bins=10, whereas seaborn seems uses the Freedman-Diaconis rule to determine the number of bins, which in this case would give a bin width of about 0.5 and bins=18.

Setting the number of bins used equal for both the seaborn and matplotlib.pyplot plots gives the following histograms:

As you can see, using the same value for the number of bins gives the exact same plots. I used the following code to produce this plot, in which you can also change the number of bins used by both plots to compare them.

import seaborn as sns
import matplotlib.pyplot as plt

# Set number of bins
nbins = 10

# Load dataset
x = sns.load_dataset("tips")

# Set up subplots
fig, axs = plt.subplots(1, 2, figsize=(12, 5))

# Seaborn plot
sns.distplot(x.tip, ax=axs[0], bins=nbins, kde=False)
axs[0].set_title("Seaborn plot")

# Matplotlib.pyplot plot
axs[1].hist(x.tip, bins=nbins)
axs[1].set_title("Matplotlib.pyplot plot")

# Set title
fig.suptitle(f"Histograms using $bins=${nbins}")

fig.show()


Pandas vs Linux Datascience I joined a data science learning community in my college and we are using linux terminal commands and awk commands to practice gathering some information from big datasets stored in csv files. About 7140596 columns by 29 rows in a single file.

A sample question would be: "What was the average arrival delay (in minutes) for flights in 2005?" where we have to sum the value of the delay for every row and divide by the total number or rows.

I know that similar data manipulation can be done in Pandas in a Jupyter notebook and was wondering what are the advantages and disadvantages of each method.

Thank you!

Pandas dataframes have many many more high level functions integrated right into the base classes that store the data for you.

Some of the commandline tools can be pretty powerful for manipulating text efficiently (Perl in particular), but I would argue that the learning curve is quite steep and the interactive experience is not as friendly. For one thing, it isn't easy to simply get a glimpse of your data or create an attractive plot.

While I admit that I am not a pro awk/sed or Perl user, I am pretty sure it will be a little less intuitive in those tools/languages to do something like this hypothetical computation, which involves numerical data and text:

In [1]: import pandas as pd
In [2]: import numpy as np

# Create a DataFrame holding some data over a time range

In [3]: df = pd.DataFrame({'A' : ['foo', 'bar', 'foo', 'bar',
                                  'foo', 'bar', 'foo', 'foo']*4,
                           'B' : ['one', 'one', 'two', 'three',
                                  'two', 'two', 'one', 'three']*4,
                           'C' : np.random.randn(32)},
                          index=pd.date_range('01.01.2018', periods=32))

In [4]: df.head()
Out[4]:
              A      B         C
2018-01-01  foo    one  0.965554
2018-01-02  bar    one  0.053814
2018-01-03  foo    two  1.075539
2018-01-04  bar  three -0.999941
2018-01-05  foo    two -1.940361

Now imagine we want to group the rows so we have just rows with column A contains foo in one table, and another with just the rows containing foo.

From those two tables, we only care about column C. We want to compute the moving average over a 5 day time-frame. The moving average will leave some NaN values at the beginning, so we want to drop those time-steps.

Oh, and we want to visualise that!

In[5]: df.groupby('A')['C'].rolling(5).mean().dropna().plot(grid=True, legend=True)

From that one line of code, we get this:

The above also highlights the abundance of other powerful and specialised packages avavilable within the Python environment - here I used numpy in conjunction with Pandas.

For manipulating text files, perhaps cleaning up scraped text and parsing large amounts of text using regular expressions, it might be faster to use one of the commandline options, but as soon as you want to do any data science, I would really recommend using some specialised tools, like Pandas.