Pivot DataFrame while calculating new values I have reduced the data set to only the columns I need:

| yearID | POS | PO | A | E |
|--------|:---:|:--:|:-:|:-:|
|  1871  | SS  | 0.0|3.0|1.0|
|  1871  | 2B  |30.0|1.0|0.0|
|  ...   |  .. | ...|...|...|

source: Sean Lahmans 2015 Baseball Data set Using the Fielding.csv file.

I am trying to calculate the Fielding Percentage:

values = df['E'] / (df['PO'] + df['A'] + df['E'])

Where there are multiple records for each 'yearID'. I am not sure if I need to transpose, apply a function or map one. Additionaly, in what order should I be moving the pieces around in.

data.loc[:,('C')] = middle_infielders.PO + middle_infielders.A + data.E
data.loc[:,('FP')] = 1 - (data.E / data.C)

| yearID | POS | PO | A | E |  C  |  FP  |
|--------|:---:|:--:|:-:|:-:|     |      |
|  1871  | SS  | 0.0|3.0|1.0| 123 | .960 |
|  1871  | 2B  |30.0|1.0|0.0|  12 | .452 |
|  ...   |  .. | ...|...|...| ... |  ... |

I would like it in this form to plot a line graph:

| yearID |  SS  |  2B  |
|--------|:----:|:----:|
|  1871  |0.3745|0.1245|
|  1872  |0.8940|0.3366|
|  ...   |  ... | ...  |

End result: One mean for each 'POS' (SS and 2B) each year.

UPDATE

Expecting pivoting to aggregate the values='FP' from the statement:
data.pivot(index='yearID', columns='POS', values='FP')
But, I get the error:

"ValueError: Index contains duplicate entries, cannot reshape."

Should I apply a Lambda to calculate the Fielding Percentage ('FP') I want in the values instead of pre-calculating it?

The magic word is "pivoting":

records = [(1871,"SS",0.0,3.0,1.0), (1871,"2B",30.0,1.0,0.0)]
pandas.DataFrame.from_records(records, columns=("yearID", "POS", "PO", "A", "E")
     ).assign(result = df.apply(lambda x: x['E']/(x['PO']+x['A']+x['E']), 1)
     ).pivot(index='yearID', columns='POS', values='result')

Result:

| POS    | 2B | SS   |
|--------|----|------|
| yearID |    |      |
| 1871   | 0  | 0.25 |

I went to the trouble of looking at your file now that you linked to it, and the problem is that you have removed to much information, so your index is not unique. Indices always have to be unique, so either you add that information back, or you preprocess the data before pivoting such that the duplication is resolved. I chose to retain the extra columns; I don't know if this is what you want, but it should help you understand what it takes to make pivoting work:

from pandas import read_csv

field_cols = ("playerID", "yearID", "teamID", "POS", "PO", "A", "E")
df = read_csv('Fielding.csv', usecols=field_cols).dropna(
             ).query('not PO == A == E == 0')

df.assign(ID = df[['playerID', 'yearID', 'teamID']].apply(tuple, 1),
          FP = df.apply(lambda x: 1-x['E']/(x['PO']+x['A']+x['E']), 1)
          ).drop(set(df.columns) - {'POS'}, 1).drop_duplicates('ID'
          ).pivot(index='ID', values='FP', columns='POS')

Result:

| POS                    | 1B  | 2B  | 3B  | C   | CF  | LF  | OF  | P        | RF  | SS  |
|------------------------|-----|-----|-----|-----|-----|-----|-----|----------|-----|-----|
| ID                     |     |     |     |     |     |     |     |          |     |     |
| (aardsda01, 2006, CHN) | NaN | NaN | NaN | NaN | NaN | NaN | NaN | 1.000000 | NaN | NaN |
| (aardsda01, 2007, CHA) | NaN | NaN | NaN | NaN | NaN | NaN | NaN | 0.857143 | NaN | NaN |
| (aardsda01, 2008, BOS) | NaN | NaN | NaN | NaN | NaN | NaN | NaN | 1.000000 | NaN | NaN |
| (aardsda01, 2009, SEA) | NaN | NaN | NaN | NaN | NaN | NaN | NaN | 1.000000 | NaN | NaN |
| (aardsda01, 2010, SEA) | NaN | NaN | NaN | NaN | NaN | NaN | NaN | 0.833333 | NaN | NaN |
  ...


When to use a Sigmoid function and when to use ReLu in a Convolutional Neural Network Can someone please refer a good article explaining why we use a Sigmoid activation in the final layer of a neural network and a ReLu activation in the middle layers and input layer while building a Convolutional Neural Network? I am not getting how that results in the right output.

Relu solves the gradient vanishing problem and stops the inactive neurons. Using Sigmoid in output layer because its range is (0, 1) and it can represent the probability of binary class. When you are doing multi-classification, it is more appropriate to use Softmax function.

Maybe you need to read more materials, you can refer to CS231n. Thanks.


In what data science applications has the stack exchange dump been used? Anonymized dumps of the stack exchange data are available here. Do you know projects or article that have been using these data (for social network analysis or information retrieval) ? My little research on the subject on Google Scholar & co seems to indicate that this dataset has been very seldom used.

This is a Community Wiki answer, everyone is welcome to add references.

Kaggle ran a competition to predict whether a question would be closed on Stack Overflow: https://www.kaggle.com/c/predict-closed-questions-on-stack-overflow


Predicting car failures with machine learning I want to start with machine learning with a small prediction problem but I'm not sure I chose the right approach. I want to make a program that gets data of mechanical failures on cars (manufactured time, failure time, reason, and different characteristics of the car). Then I would give the data of new cars that will be released to market and I would try to predict when would they fail.

I was reading that the best approach is using survival analysis with R but since I'm not really familiar with this algorithm, I was wondering If there's any other approach.

I think you should first clearly specify what the covariates are, what is the target variable is and what is your goal.

Therefore, if you have attributes about the car as covariates and target variable is failure time (car failed in 1y, 2y...), then the best approach is indeed Survival analysis, because you try to model time to failure.

On the other side, if you target variable is just a failure - yes or no, then its a classification problem. For that, simple models such as Decision trees or Logistic regression are very well suited.

Dont use algorithm just because its fancy or dont dislike other approaches just because they are not "Machine Learning".

You said you want to start with Machine learning, so go ahead. Dont blindly fit whatever blackbox models, start with simple ones and look inside, how they work.

That being said, pick something more simple. Because Survival analysis requires knowledge or regression and a bit more of statistics.

Best of luck.


Validation loss diverging away from the training loss I used the XLNET for a sentiment classifier in determining whether a comment is positive or negative. I was able to get good results

But when I plotted the validation and training losses I saw this

I think this means that the model is overfitting? But I am not exactly sure. If there is any suggestions I would really appreciate it.

This kind of overfitting is typical when finetuning large LMs.
The usual approaches to "avoid" it are:

Early stopping: select the checkpoint with the best validation loss.
Random restarts: train multiple times from scratch, and select the model with the best validation performance.


Time duration for ML models A/B testing I am going to perform A/B tests for ML models. However I am not sure how long should I run it online in order to see significant differnce. What would be the right time frame ? and what will be the reason behind the time frame ?
The A/B test will run againts the None ML systems. Usally we run for none ML features for 2 weeks max.
Thank you

First it's not a matter of duration, it's a matter of number of data points that can be collected: if there are only a few users every day, it's going to take much longer to collect enough data points than if there are millions.
Now how many data points do you need? There's no simple answer since it depends what exactly you are going to test and how big the difference turns out to be between the two cases.
A good workaround is to start with some hypothetical scenarios: imagine for instance 100, 1000 or 10000 users and an outcome with a large, a medium or a small difference. Run the test in all these different scenarios and check when a significant difference is obtained. Based on this, choose a target number of users which would be sufficient to obtain a clear result in most of the scenarios.
Finally the duration can be approximated based on the average number of daily users.


How to implement global contrast normalization in python? I am trying to implement global contrast normalization in python from Yoshua Bengio's Deep Learning book (section 12.2.1.1 pg. 442). From the book, to get a normalized image using global contrast normalization we use this equation:
$$\mathsf{X}^{\prime}_{i,j,k}=s\frac{\mathsf{X}_{i,j,k}-\overline{\mathsf{X}}}{max\left\lbrace \epsilon, \sqrt{\lambda+\frac{1}{3rc}\sum_{i=1}^{r}\sum_{j=1}^{c}\sum_{k=1}^{3}(\mathsf{X}_{i,j,k}-\overline{\mathsf{X}})^2}\right\rbrace }$$ where $\mathsf{X}_{i,j,k}$ is a tensor of the image and $\mathsf{X}^{\prime}_{i,j,k}$ is a tensor of the normalized image, and $\overline{\mathsf{X}} = \frac{1}{3rc}\sum_{i=1}^{r}\sum_{j=1}^{c}\sum_{k=1}^{3} \mathsf{X}_{i,j,k}$ is the average value of the pixels of the original image. $\epsilon$ and $\lambda$ are some constants, usually with $\lambda=10$ and $\epsilon$ set to be a very small number, and here is my implementation:

import Image
import numpy as np
import math
def global_contrast_normalization(filename, s, lmda, epsilon):
    X = np.array(Image.open(filename))

    X_prime=X
    r,c,u=X.shape
    contrast =0
    su=0
    sum_x=0

    for i in range(r):
        for j in range(c):
            for k in range(u):

                sum_x=sum_x+X[i][j][k]
    X_average=float(sum_x)/(r*c*u)

    for i in range(r):
        for j in range(c):
            for k in range(u):

                su=su+((X[i][j][k])-X_average)**2
    contrast=np.sqrt(lmda+(float(su)/(r*c*u)))

    for i in range(r):
        for j in range(c):
            for k in range(u):

                X_prime[i][j][k] = s * (X[i][j][k] - X_average) / max(epsilon, contrast)
    Image.fromarray(X_prime).save("result.jpg")
global_contrast_normalization("cat.jpg", 1, 10, 0.000000001)

original image:

result image:

I got an unexpected result. What is wrong with my implementation?

there are multiple issues with the code:

You force the values in the image to be uint8 (8-bit integer). Since the values are floats they will be casted/rounded to either 0 or 1.
This will later be interpreted as image in black and the darkest form of gray (1 out of 255).
Once you have proper floats as values PIL or pillow can't handle the array (they only do images with values in [0, 255])

The first problem happened because you/numpy wants the array to be a uint8.
The normalize version will have floats.

You should have used:

X_prime = X.astype(float)

Here is a working version of the code:

import numpy
import scipy
import scipy.misc
from PIL import Image

def global_contrast_normalization(filename, s, lmda, epsilon):
    X = numpy.array(Image.open(filename))

    # replacement for the loop
    X_average = numpy.mean(X)
    print('Mean: ', X_average)
    X = X - X_average

    # `su` is here the mean, instead of the sum
    contrast = numpy.sqrt(lmda + numpy.mean(X**2))

    X = s * X / max(contrast, epsilon)

    # scipy can handle it
    scipy.misc.imsave('result.jpg', X)

global_contrast_normalization("cat.jpg", 1, 10, 0.000000001)

PS: X_prime = X will make X_prime reference X. So changing X_prime will also change X.


Using GPS signal, determine is this person driving a cab New York City provides tens of gigs of data of taxi routes all over the city. What I'd like to do, is use this data (or some other method), to come up with an algorithm that can take a persons GPS data over a short span of time (say an hour), and answer the question: Is this person driving a cab?

The algorithm should work in any location, not just NYC. The idea is that I'd like to be able to determine patterns, that signal that a route being driven by a person is the type of route a person driving a cab would take.

Ideally, I'd like to write this in Ruby. But I am open to other suggestions, approaches, and implementations. Links to projects I should research, suggestions on languages to use, approaches to take, etc are all appreciated.

Take the taxi routes and combine them with civilian car routes to form a data set for classification. Using a map (say, from Google)
 break down each route into a sequence of roads segments, from intersection to intersection. If you only have GPS traces this will involve spatio-temporal segmentation. (Intersections/terminuses are places where cars go but stop at, and the road segments are the places cars move through to get to from one intersection to another). Model these as categorical variables (road segment 1, 2, 3, etc); abstract out the physical location. Then train a classifier than accepts a sequence as input (e.g., a recurrent neural network). Use the time of departure as another feature, modeled as two real variables, $\cos(2*\pi*t/24), \sin(2*\pi*t/24)$, where $0<t<24$. If you have really accurate GPS information, I would also try to estimate the rate of lane crossing; taxi drivers are well known for their aggressive driving, and the rate of lane crossing would capture this well. Finally, use a pair of boolean variables to record whether the points of departure and destination are parking areas, if you can obtain this information. I imagine taxis will be more likely to start and stop at prohibited places.

The rest is your usual hyper-parameter optimization black magic.

If you want to be able to do anything besides classify a given route as taxi-driven or not, please say so.


Log-loss - but only if too far away from target? I'm training a neural net on a "needle in haystack" problem - there is a huge amount of data, also with a huge class imbalance (~1 positive per 10k negatives).  If I train the net on examples drawn so as to balance the classes, it works, but either overfits on the positives (since it sees the few positive examples repeated so often), or doesn't see most of the negatives.  The result of not seeing most of the negatives is that there is a number of false positives that crop up, even on examples which were in the training set but which were never used in training.  Presumably if the network was trained on those that would fix the problem.
The partial solution: every once in a while during training, run inference on all training data, and pick the hard examples (either false positive or false negative); use a balanced sample of just those for the next stage.  This does work, but is quite slow, and it is unclear how many to pick, with what thresholds, etc.
The idea: I would like to train so that the log loss only works if too far from the target.  In other words, true=0 and predicted=0.01 should not create a gradient at all; that way the vast majority of true negative examples would just run through a mostly-trained network but basically not do anything; the few false positives would generate a gradient, counterbalanced by the negatives.
Has this been done before?  Does it work?  Is there a better approach in the same situation?

Have you looked into focal loss?

The idea seems to be similar to what you are describing - If predictions (~0.8) is close to GT Label of 1, it does not add to the loss value.


Is the result of feature extraction a feature representation? If a use a feature extraction method on images, do I then get a feature representation or is there a different meaning behind feature representation?
To my understanding, when I use a CNN on an image the convolutional layers perform feature extraction. Can I then also say that I get a new feature representation after each of these layers?
Thanks in advance! I am relatively new to machine learning...

Yes, you are right. As a result of the extraction of features, new feature representations are created. For example, SIFT and HARR are algorithms to extract the crucial features from the images and create feature representation. CNN does it automatically with the help of convolution layers. In an image, smooth area, edge, corners are considered features that may represent it best. The importance of the smooth area, edge, and corners (there are others too) are normally accepted in ascending order.
In a simple CNN, if you can visualize the layers you can see that what it extracts from the image are mainly, corners and edge. It is also able to learn more complex features than we know (or defined). So, as you said, the convolution layers try to create a representation of features by extracting them. Then, this features representations (important information from the image) are used in the let's say prediction process.


Tensor Mean of values greater than a threshold I have a tensor of shape = [a,b,c]

The tensor mean along dim=1 will give me an output of shape = [a,c]

My goal is to compute the mean of values along dim=1 greater than a threshold.
How is this possible in TensorFlow?

I am aware of tf.where() and tf.boolean_mask() functions but not sure exactly how to use them for this task.


I tried loading my saved .h5 model and predicting with that model, i'm getting error list index out of range I saved my keras model into .h5 format. Again I've loaded that .h5 file into my colab and tried to predict with that model.
model.save("/content/drive/My Drive/Datasets/sentiment_analysis.h5")
from keras.models import load_model
loaded_model = load_model("/content/drive/MyDrive/Datasets/sentiment_analysis.h5")
loaded_model.predict("i love machine learning and google")

It's giving error list index out of range.
IndexError: list index out of range

Try these:

import everything from tensorflow keras API.
tf.keras.models.load_model("./saved_models/our_model.h5", compile=False) (make sure you have model and weights both in the file)


ARIMA training super slow I am fitting ARIMA model (from statsmodels) on 20 000 elements dataset on a 24 CPU 200+GB RAM cloud server for over 24 hours now. It loads all the CPU's. But It takes so long... Is it how it works or is here something wrong?
p,d,q are just numbers
from statsmodels.tsa.arima_model import ARIMA

for t in range(len(test)):
    model = ARIMA(history, order=(p, d, q))
    model_fit = model.fit(disp=1)
    output = model_fit.forecast()
    yhat = output[0]
    predictions.append(yhat)obs = test[t]
    history.append(obs)

You can find below a worked out example (on a univariate version for the temperatures attribute) which, using statsmodels ARIMA, does not even finish the excution after hours trying for it.
An option you have regarding the performance issues is to actually use another efficient library like Keras with Tensorflow modelling an LSTM useful for time series data, which is capable of dealing with so many samples without performance issues.
This dataset contains 420551 samples, which is very costly with the ARIMA statsmodels approach via walk-forward validation...
You can extract the temperature values for carrying out the univariate time series forecast.
See below an example (based on this tutorial) where leaving hyperparametrization aside for now, you can train a nearly half a millon time series samples without performance issues (you can try on google colab for faster checks for free).
From the example in the tutorial, you should pay special attention at:

the way the data is splitted:

def univariate_data(dataset, start_index, end_index, history_size, target_size):
data = []
labels = []

start_index = start_index + history_size
if end_index is None:
  end_index = len(dataset) - target_size

for i in range(start_index, end_index):
  indices = range(i-history_size, i)
  #### Reshape data from (history_size,) to (history_size, 1)
  data.append(np.reshape(dataset[indices], (history_size, 1)))
  labels.append(dataset[i+target_size])

return np.array(data), np.array(labels)

where, using 20 samples as recent history to predict the next value you can use:
univariate_past_history = 20
univariate_future_target = 0

x_train_uni, y_train_uni = univariate_data(uni_data, 0, TRAIN_SPLIT,
                                       univariate_past_history,
                                       univariate_future_target)
x_val_uni, y_val_uni = univariate_data(uni_data, TRAIN_SPLIT, None,
                                   univariate_past_history,
                                   univariate_future_target)

having something like:

the way the model is defined:
simple_lstm_model = tf.keras.models.Sequential([
tf.keras.layers.LSTM(8, input_shape=x_train_uni.shape[-2:]),
tf.keras.layers.Dense(1)
])

which comparing to a baseline model by predicting simply the mean of the history values, you have:

baseline model predicting based on simply the mean of past values:

VS

predicting wih the trained LSTM model:


Clustering with Only Categorical Features I am trying to do clustering with a bunch (24) of categorical features. I have done some research and found a lot of people recommending something such as K-Modes. I tried running K-Modes on my data and the best run had a cost of 27069.0, which seems pretty high.

Some of my features have only a few values, such as P, O, C, T, so I thought I could encode them. But others have many different values. Any tips on a clustering algorithm or some other approach? I would like to use Python.

EDIT: What about using Gower distance on the data and then using K-Means on that?

You can one-hot encode all your features, first. Then, you will face with a sparse feature space. To resolve this issue, you can use an auto-encoder to encode all these values to a low-dimensional and more dense space. Then run one of your clustering methods such as k-means.


Extremely stochastic validation loss/accuracy I am working on training DNNs on satellite data. The class distribution in the data is extremely imbalanced, so I train the neural networks using random majority undersampling to artificially balance the number of training examples from each class. During validation, I don't resample the data in any way.

In the graphs above, orange is training performance and blue is validation.
The validation accuracy and loss values are much much noisier than the training accuracy and loss. Validation accuracy even hit 0.2% at one point even though the training accuracy was around 90%. Why are the validation metrics fluctuating like crazy while the training metrics stay fairly constant?
Some additional information on the model/data:
I'm segmenting satellite images into three land cover classes using a U-Net model. The U-Net model has half the number of filters at each layer compared to the original paper, and I apply batch normalization after each convolution and before the activation. The model is trained from scratch with the Adam optimizer with an initial lr=0.001 and a step decay schedule, optimizing (unweighted) categorical crossentropy. The U-Net has l2 weight regularization applied on each convolutional layer with constant 0.01.
The training and validation data are all floating point images with values in the range 0-2. The images for the two datasets were sampled from the same spatial region. I don't do any preprocessing or data augmentation because of the computational overhead.
The validation set has 317k pixels in class 1, 26.5M pixels in class 2, and 27.8M pixels in class 3 (this is the "true" data distribution). The number of training instances which contain pixels in class 1 is ~200 out of 3500. The training set (in which class 1 is spatially oversampled to artificially balance the training set), consists of 58M class 1 pixels, 166M class 2 pixels, and 127M class 3 pixels. Out of 20k training instances, approximately 8200 contain class 1 pixels.
When feeding data to the model, I sample from the three datasets with probability 0.4, 0.3, and 0.3 for classes 1, 2, and 3 respectively.
EDIT: Problem solved! When working with neural networks and geospatial data it's very important to maintain the same map projection between the test/train/validation sets. I thought the software I was using to extract the test/train/validation data was ensuring the same projection between sets, but I was wrong. Making the projections of the test/train/validation data the same fixed my problem. Essentially I was validating on a different dataset than I was training on, which is not how machine learning models are supposed to be validated.

There can be a few reasons for this behavior as already pointed out in "Why is the validation accuracy fluctuating?":

Size of train / validation stes: Fluctuations may become stronger the smaller your validation set is, especially during the early stage of training where predictions are closer to random predictions.

Overfitting: Train loss seems to be low and accuracy over 90%. If you're overfitting, the predictions on the validation set become almost random and you observe such fluctuations.

Random validation set: If you're validation set is not constant across all epochs, the validation loss will be different and losses from different epochs cannot be compared anymore. Especially if you have imbalanced classes.

Weighted loss function: Using a weighted loss function for imbalanced data sets makes your loss appear more stochastically, especially in the early stage of training and if the distributions of the training and validation sets mismatch.

Maybe you can share additional information to your data, preprocessing, training, model etc. so it is easier to spot the cause of this more precisely.


On what principle did Google's DeepMind learn to walk? I just saw this video on Youtube.
On what principle did Google's DeepMind learn to walk?
Was it Q-Learning or a Genetic Algorithm or Policy Gradient?

The full method is explained in the paper Emergence of Locomotion Behaviours in Rich Environments by the DeepMind team.
Quoting from that paper:

Using a novel scalable variant of policy gradient reinforcement learning, our agents learn to run, jump, crouch and turn as required
by the environment . . .

So to answer your question, the researchers used a policy gradient method.


Proper evaluation method for recommendation system with implicit feedback? I am trying to implement a recommendation system for a live-streaming website. Here "users" are simply the website users and "items" are streamers that they should watch. I collected data on which users follow which streamers -- however, following a streamer does not necessarily correspond to liking the streamer. Thus the setting is: I have binary implicit feedback (0 = no follow, 1 = follow) for 10k users, 2k items, and 200k interactions. I also have follow timestamps, but I have not tried to use this data yet. The models I'm using are primarily collaborative filtering, matrix factorization/latent factor models which are implemented in Implicit & LightFM (Python). My questions are: (i) how do I make a train/validation/test split, (ii) what metric is appropriate?
My understanding is that to make a train-test split we "mask" some of the observations in the interaction matrix $R$, call these observations $\{r_{ui}^t\}$ and then train our model on $R\setminus\{r_{ui}^t\}$. So we just treat the test observations as $0$ in training phase (?). Then for each user we score all of their items in the test set (or do we score all of their items in either the train or test set?), rank the items by their scores, and then calculate an evaluation metric. Please correct/validate me :) Here are my observations regarding my two main questions...
(i). In the paper [1], the authors make a train-test split based on time period. On the other hand, in [2], the authors use a leave-one-out strategy where they drop one positive user-item pair for each user. Finally, [3] generates a test set by using a random 10% sample of entries (rather than selecting random positive entries), although it isn't clear to me how this is favorable to just selecting random positive entries. I'm leaning towards the method in [2] for its simplicity and because there does not seem to be a consensus. Furthermore, none of these works mentioned using, or offered a method of making, an additional validation set to tune hyperparameters. Is it just a matter of repeating the split method twice?
(ii). For implicit feedback I understand that recall-based metrics are more important than precision. In [1,3] they use mean percentage ranking
$$\overline{rank} = \frac{\sum_{u,i} r_{ui}^t rank_{ui}}{\sum_{u,i} r_{ui}^t}$$
where $rank_{ui}$ is the percentile ranking for item $i$ in user $u$'s full ranking list and $r_{ui}^t$ is the implicit feedback rating for a test observation. This seems useful when $r_{ui}$ may take on many values and one would like to respect these ratings; however, for me $r_{ui} \in \{0,1\}$. Also, this uses the entire ranking list, while perhaps a cutoff-sensitive metric such as $MAR@K$ is better for my application (?). In [2] they only report $AUC$, probably because the BPR model optimizes for $AUC$. With their leave-one-out strategy they denote $S_\text{train}, S_\text{test}$ to be the sets of positive interactions and then compute
$$AUC = \frac{1}{|U|} \sum_u \frac{1}{|E(u)|} \sum_{i,j \in E(u)} \delta(\hat{r}_{ui} > \hat{r}_{uj})$$
where $|U|$ is the number of users and $E(u) = \{(i,j) | (u,i) \in S_{\text{test}} \wedge (u,j) \notin (S_\text{test} \cup S_\text{train})\}$ are the evaluation pairs for user $u$. That is, they calculate how often a positive interaction is rated higher than an unobserved interaction. This seems like it might be a favorable metric considering our interaction data is binary.
References
[1] Hu, Y., Koren, Y., & Volinsky, C. (2008, December). Collaborative filtering for implicit feedback datasets. In 2008 Eighth IEEE International Conference on Data Mining (pp. 263-272). IEEE.
[2] Rendle, S., Freudenthaler, C., Gantner, Z., & Schmidt-Thieme, L. (2012). BPR: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:1205.2618.
[3] Johnson, C. C. (2014). Logistic matrix factorization for implicit feedback data. Advances in Neural Information Processing Systems, 27, 78.

First off, in terms of how to make a train/validation/test split, I would only include users with 2 or more interactions, and for each user, remove the latest interaction. Use this latest interaction as a target and recommend, for each user, their top streamers. You then compare your recommendations with the target, and compute a performance metric. Depending on your performance metric of choice (next paragraph), the number of recommendations you make might differ. This gives you a training/validation split. If you want more training data, and you believe a user's interactions are not sequential in nature (eg. liking StreamerA before StreamerB displays a different preference than vice-versa), you can permute each user into more users by leaving out a different target each time. So for each streamer u, who has n_u interactions, leave out one interaction (target), and keep the rest, and do this n_u times.
Secondly, the appropriate performance metric depends on your use case. If you are going to recommend, say k, streamers for each user, then the performance metric of choice should consider only these k. In other words, for any model performance, rank k+10 is no better than rank k+20. Moreover, maybe it is only of importance that the target is part of the top-k recommendations, then some XYZ@K-metric is most appropriate. Personally, I would probably go with top-k accuracy and average this across all users. This metric then specifies, for all my users, how often, on average, is the target in their top k recommendations.


Questions regarding deep learning? I have questions regarding this Normalized weights and Initial inputs video on Udacity course Deep Learning

In this video the lecturer talks about variables that go into Big-loss function should have zero mean and equal variances. I can’t figure about which variables the lecturer is talking about. Are they weights and biases or are they soft-max function and labels in the Big-loss function? And how  would zero mean and equal variance help in optimization?

The other question I have is, in the video the lecturer talks about weight initialization randomly using Gaussian distribution, I cannot understand how can we initialize weights using Gaussian distribution with zero mean and standard deviation sigma? And where optimizer will move the point (initialized weight), up or down to find the local minima?

So can you recommend me a book that I should study before taking this course, so that problems like these don't occur?

I can’t figure about which variables the lecturer is talking about. Are they weights and biases or are they soft-max function and labels in the Big-loss function?

In the case of neural networks, the loss function depends on the weights and the biases (which are often not mentioned extra; they are also weights).

The loss function itself is parametrized heavily. Its parameters is the data (input and labels). Its variables (by which the derivatives are calculated) are weights.

  And how would zero mean and equal variance help in optimization?

Have a look at the derivative of the sigmoid function. It is biggest at 0. This means the gradient there can be big. This helps by learning, because the basic learning rule is

$$w \gets w_i + \Delta w_i\;\;\; \text{ with } \Delta w_i = - \eta \frac{\partial E}{\partial w_i}$$

So you can get bigger adjustments if you normalize it to have mean of 0.

The part about the variance ... hm. That's harder to explain. I'm not entirely sure about it. One thought is that you want the data to be in a very restricted, similar domain (independent of the application) so that you can treat results independently from your application. Also, it might help with mini-batches not varying too much.

  in the video the lecturer talks about weight initialization randomly using Gaussian distribution, I cannot understand how can we initialize weights using Gaussian distribution with zero mean and standard deviation sigma?

I'm not sure what exactly the question is.

Practically: Just use a function. For example, numpy.random.normal if you're using Python.
Mathematically: This is called sampling. You have a process which has a random distribution and you take examples from that.
Technically: If you really want to implement this yourself, you will most likely end up using a random number generator which generates samples from a uniform distribution in [0, 1].

Once you have those sample numbers, you simply assign those numbers to the weights.

  And where optimizer will move the point (initialized weight), up or down to find the local minima?

The optimizer calculates the gradient of the error function. I recommend having a look at the early chapters of the Udacity course. I'm pretty sure this was covered there, too. Another resource would be neuralnetworksanddeeplearning.com

  So can you recommend me a book that I should study before taking this course, so that problems like these don't occur?

Tom Mitchells book "Machine Learning" covers similar topics as the course.


How to restructure my dataset for interpretability without losing performance? What I am doing:
I am predicting product ratings using boosted trees (XGBoost) with a dataset in this format:

What I want to do:
I want to use SHAP TreeExplainer to interpret each prediction my model gives in terms of product attributes and user ids.
What I am getting:
My model is drawing all the conclusions based on product names and user ids, instead of product attributes and user ids.
What I tried:
I discovered that each product name has a unique combination of product attributes, i.e. by knowing the product attributes you can find its name. So my idea was to remove the product_name column, leaving only the attributes.
My reasoning was that restructuring the dataset in this way would lead to the interpretability that I wanted without any performance loss (since the product name doesn't add any new information).
What I got:
The model performance decreased a lot. Even with a great deal of hyperparameter tuning, I couldn't get near the performance I had when also using the product name.
What I think maybe going on:

My dataset is too small for the model to learn with the product attributes (10k samples, 60 attributes).

or

Maybe there are some attributes adding bias and screwing with my model ability to generalize, leading to an overfit.

I am a little skeptical about the number 2, seeing that my training loss also went up when I removed the product name.
My question:
So, how can I restructure my dataset? Does anybody have a clue why my model can't reach the same performance without using the product name? Any light or ideas on what I can try?

What may be happening is that your attribute predictors are weak predictors, they are noisy. Meaningful decision trees can't be made out of product attribute features by xgb.

When you are adding name as a predictor, xgb finds some signal wrt your target variable - rating and thus you get a better score. So your name plus attributes model may be performing better than attributes only model for this reason.

So if you from domain experience know product attributes are very weakly related to rating then you can conclude that this feature set of attributes is not going to help you make accurate predictions. Or instead of relying  on d omain expertise, you can use correlation or relevant statistical tests to understand attributes relation to rating and if found that relationship is non existent or very weak you can conclude model isn't possible.

So may be add more relevant features if possible if you want to make a reasonably good model.

Regards
Vik


Stacked time series plot in python In pandas I can set the date as index, and then run df.plot() to see a line chart. How do I make that line chart stacked as in the picture below?

You can simply use
df.plot.area()

Found here after a quick google search.


Positive or negative impact of features in prediction with Random Forest In classification, when we want to get the importance of each variable in the random forest algorithm we usually use Mean Decrease in Gini or Mean Decrease in Accuracy metrics. Now is there a metric which computes the positive or negative effects of each variable not on the predictive accuracy of the model but rather on the dependent variable itself? Something like the beta coefficients in the standard linear regression model but in the context of classification with random forests.

With decision trees you cannot directly get the positive or negative effects of each variable as you would with say a linear regression through the coefficients. Its just not the way decision trees work. As you point out, the training process involves finding optimal features and splits at each node by looking at the gini index or the mutual information with the target variable. But no parameters are learnt during the process which we could use for such analysis.

A common tool that is used for this purpose is SHAP. In fact, there is a specific explainer for decision trees based models which is the TreeExplainer. With SHAP you can get both the contribution of the features in pushing towards one or another value of the label, and also an overall view of the contribution of all features.


Shuffle the data before splitting into folds I am running a 4-folds cross validation hyperparameter tuning using sklearn's 'cross_validate' and 'KFold' functions. Assuming that my training dataset is already shuffled, then should I for each iteration of hyperpatameter tuning re-shuffle the data before splitting into batches/folds (i.e., the shuffle argument in the KFold function)? I noticed that the outcome of the hyperparameter tuning process will be different depending on if I am shuffling the data prior to splitting it into folds.

I assume that if the outcome depends on shuffling then the model is not robust. Is this correct?
However, this also may not be 'fair' to the model because the result is not reproducible since the data for each fold is different every time I run cross validation (i.e., each hyperparametr combination is evaluated on totally different folds. For example, the training/validation dataset in fold #1 of the 1st tuning iteration is different than fold #1 dataset of the 2nd tuning iteration.)

I am running a 4-folds cross validation hyperparameter tuning using sklearn's 'cross_validate' and 'KFold' functions. Assuming that my training dataset is already shuffled, then should I for each iteration of hyperpatameter tuning re-shuffle the data before splitting into batches/folds (i.e., the shuffle argument in the KFold function)?

No, its no needed, shuffling is needed before split.

  I assume that if the outcome depends on shuffling then the model is not robust. Is this correct?

You are right, good model has great performing on every combination of data.

  However, this also may not be 'fair' to the model because the result is not reproducible since the data for each fold is different every time I run cross-validation (i.e., each hyperparameter combination is evaluated on totally different folds. For example, the training/validation dataset in fold #1 of the 1st tuning iteration is different than fold #1 dataset of the 2nd tuning iteration.)

You make cross-validation to check performance on every chunk of data. Your task is achieved most generous model.


multiclass classification I want to build an ml model, which can when given a text input, can predict predefined tags or labels for the text. I already built one such model, but the problem with that is that it only predicts correct labels when a particular keyword is present in the text. For example - TEXT - "How to crack an interview?". LABELS PREDICTED - "Interview, Fresher". BUT, If TEXT - "How to clear hr round of xyz company, what questions the can ask?". Either it will predict some irrelevant label or no label at all.  My model is very similar to this model

Seems like you overfitted.
I would advise you to get more data if you can and more importantly use a pre-trained model to encode the words to vectors. See for example the answer to this question.


What is the difference between a regular Linear Regression model and xgboost with objective set to "reg:linear"? As I understand it, a regular linear regression model already minimizes for squared error, which means that it is the theoretical best prediction for this metric.  Does xgboost's "reg:linear" objective do something other than minimize squared error?

Linear regression is a parametric model: it assumes the target variable can be expressed as a linear combination of the independent variables (plus error). Gradient boosted trees are nonparametric: they will approximate any* function.

Xgboost deprecated the objective reg:linear precisely because of this confusion. It has been replaced by reg:squarederror, and has always meant minimizing the squared error, just as in linear regression.

So xgboost will generally fit training data much better than linear regression, but that also means it is prone to overfitting, and it is less easily interpreted. Either one may end up being better, depending on your data and your needs.


How does sklearn random forest decide feature threshold at node splitting exactly? Thinking of the RandomForestClassifier function in sklearn.ensemble, I understand that at each non-terminal node the algorithm:

Randomly selects a subset of size max_features from the set of all features
Searches for the feature x and the threshold value x_0 such that when the node is split according to {x<=x_0} and {x>x_0}, the Gini impurity is minimized (I'm using the Gini setting)

My question is about how exactly this search is carried out and a value for x_0 chosen. The obvious method would be to just test all possibilities, taking into account that only the order of x_0 relative to the training set matters. That is, look at the x-values of all training points. If a is the largest of these values <= x_0 and b is the smallest of these values > x_0, then any other choice of x_0 within the half-open interval [a,b) would have produced exactly the same splitting. So it suffices to test one value of x_0 from each half-open interval and choose the one with the lowest Gini impurity.
To be specific then, my question is:

Does sklearn check all possibilities like above?
The value of x_0 makes no difference in the training step as long its nearest neighbors in the training set don't change. But it may make a difference in the testing step, when the tree is applied to new data points. So how does sklearn decide a specific value for x_0 in the training step?

After some searching, I found this source code:
https://github.com/scikit-learn/scikit-learn/blob/7813f7efb5b2012412888b69e73d76f2df2b50b6/sklearn/tree/_splitter.pyx#L467
I think, but am not 100% sure as my code comprehension is imperfect, that:

Yes
x_0 = (a+b)/2, where a is the nearest neighbor to the left and b is the nearest neighbor to the right


How to implement hierarchical labeling classification? I am currently working on the task of eCommerce product name classification, so I have categories and subcategories in product data. I noticed that using subcategories as labels delivers worse results (84% acc) than categories (94% acc). But subcategories are more precise as labels, what's important for the whole task. And then I got an idea to first do category classification and then based on the results continue with subcategories within the predicted category.
The problem here is that I do not know how to approach this problem/define network architecture. Any hints on the neural networks, how to deal with it?
Currently I defined network like this:
model = Sequential()
model.add(Dense(400, input_shape=(FEATURE_NUM,)))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes))
model.add(Activation('softmax'))

For this problem, you need to develop three models:
Model 1- for two main categories
Model 2- for sub-category A
Model 3- For sub-category B

So when you want to predict the result for an unseen data, first you use the Model 1, to find the main category. Based on the prediction and by using an if-else statement, you decide to perform another prediction using Model 2 or Model 3.
Consequently, the Model 1 is a binary classification task, but Model 2&3 are mutli-class classification task. Your networks may look like this:
Model 1:
model1 = Sequential()
model1.add(Dense(60, input_dim=X.shape[1], activation='relu'))
...
model1.add(Dense(1, activation='sigmoid'))
model1.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

Model 2 & 3:
model2 = Sequential()
model2.add(Dense(8, input_dim=X.shape[1], activation='relu'))
...
model2.add(Dense(n_subcats, activation='softmax'))
model2.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

Just don't forget to use correct y (label) for main and sub-categories for each model.


AlphaGo (and other game programs using reinforcement-learning) without human database I am not a specialist of the subject, and my question is probably very naive. It stems from an essay to understand the powers and limitation of reinforcement learning as used in the AlphaGo program.

The program AlphaGo has been built using, among other things (Monte-Carlo exploration of trees, etc.), neural networks which are trained from a huge database of human-played go games, and which are then reinforced by letting play versions of the program against itself many times.

Now I wonder what would happen is we tried to build such a program without human database, i.e. starting with a basic program of Go just knowing rules and some method to explore trees, and letting play against itself to improve its neural network. Will we, after many games against itself, arrive at a program able to compete with or beat the best human players? And if so, how many games (in order of magnitude) would be needed for that? Or on the contrary, will such a program converge toward a much weaker player?

I assume that the experiment has not been made, since AlphaGo is so recent. But the answer may nevertheless be obvious to a specialist. Otherwise any educated guess will interest me.

One can also ask the same question for "simpler" games. If we use roughly the same reinforcement-learning technics used for AlphaGo, but with
no use of human database, for a Chess program, would we eventually get
a program able to beat the best human? And if so, how fast? Has this been tried? Or if not for Chess, what about Checkers, or even simpler games?

Thanks a lot.

I'm no expert but it looks like AlphaGo Zero answers your question.
https://deepmind.com/blog/alphago-zero-learning-scratch/

  Previous versions of AlphaGo initially trained on thousands of human
  amateur and professional games to learn how to play Go. AlphaGo Zero
  skips this step and learns to play simply by playing games against
  itself, starting from completely random play. In doing so, it quickly
  surpassed human level of play and defeated the previously published
  champion-defeating version of AlphaGo by 100 games to 0.


Algorithm for classification of words into given categories I'm working with textual data from medical field. I have a list of words and I want to build an algorithm that can classify each word into one or more given categories, like

Medicine_Name
Disease_Name
Body_Part

etc.

Please give me some ideas to implement it, specifically how to prepare the training example and implement the training phase.

The task you are asking about is Named entity recognition

A good way to identify a word category is by using patterns for the context of the word.

In your case you can use patterns like:

"I took X" for a medicine name
"I had pain in my X" for a body part
"I suffered from X" for a disease

The easiest and fastest way is to discuss the pattern with a domain expert and do a sanity check on the data you have.

A more systematic approach is to deduce the pattern from your data.
I guess you can get a list of samples of the items.
The technical details depend on the patterns you use but for this example let consider tokenization.
Take the context of k-words next to the items on the list. Compare it to the context of other words. Contexts that are related to your items are suitable as patterns.

Bare in mind that these pattern will have very different precision and recall and they will lead to errors. It is important that you will have a labeled dataset so you will be able to estimate the patterns performance.

It will be fairly easy to get pretty good classification. It might be very hard to get a very good classification.


How to design n-dimensional feature descriptor similar as the input image? I am re-writing the H-Net code in Keras for cross-domain image similarity. The network architecture is described in the attached paper. I wrote the encoder and decoder parts but unable to get similar dimenstions from decoder. Following is the code. Please, point me out where I am making mistake.

import tensorflow as tf
from keras.models import Model
from keras.layers import Input, Conv2D, Conv2DTranspose, BatchNormalization, Activation, MaxPool2D

# pe: Photo Encoder, PFM: Photo Feature Map, pd: Photo Decoder

# Input shape for encoder
input_img = Input(shape=(265,256,1))

pe = Conv2D(96, kernel_size=11, strides=(4,4), padding = 'SAME',name='conv1')(input_img)
pe = BatchNormalization()(pe)
pe = Activation('selu')(pe)
pe = MaxPool2D((3, 3), strides=(2, 2), padding = 'VALID', name = 'pool1')(pe)

pe = Conv2D(256, kernel_size=5, strides=(1,1), padding = 'SAME', name='conv2')(pe)
pe = BatchNormalization()(pe)
pe = Activation('selu')(pe)
pe = MaxPool2D((3, 3), strides=(2, 2), padding = 'VALID', name = 'pool2')(pe)

pe = Conv2D(384, kernel_size=3, strides=(1,1), padding = 'SAME', name='conv3')(pe)
pe = BatchNormalization()(pe)
pe = Activation('selu')(pe)

pe = Conv2D(384, kernel_size=3, strides=(1,1), padding = 'SAME', name='conv4')(pe)
pe = BatchNormalization()(pe)
pe = Activation('selu')(pe)

pe = Conv2D(256, kernel_size=3, strides=(1,1), padding = 'SAME', name='conv5')(pe)
pe = BatchNormalization()(pe)
pe = Activation('selu')(pe)
pe = MaxPool2D((3, 3), strides=(2, 2), padding = 'VALID', name = 'pool3')(pe)

pe = Conv2D(1024, kernel_size=7, strides=(1, 1), name = 'con2vector', padding = 'VALID')(pe)
pe = BatchNormalization()(pe)
p_encoder = Activation('selu')(pe)

pfm = tf.reshape(p_encoder,[-1, 1024])

'''
Output of each conv2d and pooling layers for Photo-Encoder

Block 1 convolution:  (?, 64, 64, 96)
Block1 maxpooling:  (?, 31, 31, 96)
Block 2 convolution:  (?, 31, 31, 256)
Block 2 convolution:  (?, 15, 15, 256)
Block 3 convolution:  (?, 15, 15, 384)
Block 4 convolution:  (?, 15, 15, 384)
Block 5 convolution:  (?, 15, 15, 256)
Block 5 maxpooling:  (?, 7, 7, 256)
Block 6 convolution:  (?, 1, 1, 1024)
Shape of photo feature map:  (?, 1024)
'''

'''
Following code is the implementation of photo decoder
'''
pd_input = tf.reshape(p_encoder, [-1, 2, 2, 256])

pd = Conv2DTranspose(128, kernel_size=5, strides=(2, 2), padding='SAME', name="pdcnv1")(pd_input)
pd = Activation("selu")(pd)

pd = Conv2DTranspose(64, kernel_size=5, strides=(2, 2), padding='SAME',name="pdcnv2")(pd)
pd = Activation("selu")(pd)

pd = Conv2DTranspose(32, kernel_size=5, strides=(2, 2), padding='SAME',name="pdcnv3")(pd)
pd = Activation("selu")(pd)

pd = Conv2DTranspose(16, kernel_size=5, strides=(2, 2), padding='SAME',name="pdcnv4")(pd)
pd = Activation("selu")(pd)

pd = Conv2DTranspose(8, kernel_size=5, strides=(2, 2), padding='SAME',name="pdcnv5")(pd)
pd = Activation("selu")(pd)

pd = Conv2DTranspose(4, kernel_size=5, strides=(2, 2), padding='SAME',name="pdcnv6")(pd)
pd = Activation("selu")(pd)

p_decoder = Conv2DTranspose(3, kernel_size=5, strides=(2, 2), padding='SAME', activation='sigmoid', name="pdcnv7")(pd) # (?, ?, ?, 3)

"""
Output of each Conv2DTranspose layer
Photo decoder input shape:  (?, 2, 2, 256)
Photo deconv block 1:  (?, ?, ?, 128)
Photo deconv block 2:  (?, ?, ?, 64)
Photo deconv block 3:  (?, ?, ?, 32)
Photo deconv block 4:  (?, ?, ?, 16)
Photo deconv block 5:  (?, ?, ?, 8)
Photo deconv block 6:  (?, ?, ?, 4)
Photo deconv block 7:  (?, ?, ?, 3)
"""

autoencoder = Model(input_img, p_decoder)

The problem was with the tf.reshape. Instead I replaced it with pd_input = Reshape((2, 2, 256))(p_encoder). Finally I got the desired output which is same as input (265, 256, 3).


What statistical model should I use to analyze the likelihood that a single event influenced longitudinal data I am trying to find a formula, method, or model to use to analyze the likelihood that a specific event influenced some longitudinal data. I am having difficultly figuring out what to search for on Google.

Here is an example scenario:

Image you own a business that has an average of 100 walk-in customers every day. One day, you decide you want to increase the number of walk-in customers arriving at your store each day, so you pull a crazy stunt outside your store to get attention. Over the next week, you see on average 125 customers a day.

Over the next few months, you again decide that you want to get some more business, and perhaps sustain it a bit longer, so you try some other random things to get more customers in your store. Unfortunately, you are not the best marketer, and some of your tactics have little or no effect, and others even have a negative impact.

What methodology could I use to determine the probability that any one individual event positively or negatively impacted the number of walk-in customers? I am fully aware that correlation does not necessarily equal causation, but what methods could I use to determine the likely increase or decrease in your business's daily walk in client's following a specific event?

I am not interested in analyzing whether or not there is a correlation between your attempts to increase the number of walk-in customers, but rather whether or not any one single event, independent of all others, was impactful.

I realize that this example is rather contrived and simplistic, so I will also give you a brief description of the actual data that I am using:

I am attempting to determine the impact that a particular marketing agency has on their client's website when they publish new content, perform social media campaigns, etc. For any one specific agency, they may have anywhere from 1 to 500 clients. Each client has websites ranging in size from 5 pages to well over 1 million. Over the course of the past 5 year, each agency has annotated all of their work for each client, including the type of work that was done, the number of webpages on a website that were influenced, the number of hours spent, etc.

Using the above data, which I have assembled into a data warehouse (placed into a bunch of star/snowflake schemas), I need to determine how likely it was that any one piece of work (any one event in time) had an impact on the traffic hitting any/all pages influenced by a specific piece of work. I have created models for 40 different types of content that are found on a website that describes the typical traffic pattern a page with said content type might experience from launch date until present. Normalized relative to the appropriate model, I need to determine the highest and lowest number of increased or decreased visitors a specific page received as the result of a specific piece of work.

While I have experience with basic data analysis (linear and multiple regression, correlation, etc), I am at a loss for how to approach solving this problem. Whereas in the past I have typically analyzed data with multiple measurements for a given axis (for example temperature vs thirst vs animal and determined the impact on thirst that increased temperate has across animals), I feel that above, I am attempting to analyze the impact of a single event at some point in time for a non-linear, but predictable (or at least model-able), longitudinal dataset. I am stumped :(

Any help, tips, pointers, recommendations, or directions would be extremely helpful and I would be eternally grateful!

For the record, I think this is the type of question that's perfect for the data science Stack Exchange. I hope we get a bunch of real world examples of data problems and several perspectives on how best to solve them.

I would encourage you not to use p-values as they can be pretty misleading (1, 2). My approach hinges on you being able to summarize traffic on a given page before and after some intervention. What you care about is the difference in the rate before and after the intervention. That is, how does the number of hits per day change? Below, I explain a first stab approach with some simulated example data. I will then explain one potential pitfall (and what I would do about it).

First, let's think about one page before and after an intervention. Pretend the intervention increases hits per day by roughly 15%:

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

def simulate_data(true_diff=0):
    #First choose a number of days between [1, 1000] before the intervention
    num_before = np.random.randint(1, 1001)

    #Next choose a number of days between [1, 1000] after the intervention
    num_after = np.random.randint(1, 1001)

    #Next choose a rate for before the intervention. How many views per day on average?
    rate_before = np.random.randint(50, 151)

    #The intervention causes a `true_diff` increase on average (but is also random)
    rate_after = np.random.normal(1 + true_diff, .1) * rate_before

    #Simulate viewers per day:
    vpd_before = np.random.poisson(rate_before, size=num_before)
    vpd_after = np.random.poisson(rate_after, size=num_after)

    return vpd_before, vpd_after

vpd_before, vpd_after = simulate_data(.15)

plt.hist(vpd_before, histtype="step", bins=20, normed=True, lw=2)
plt.hist(vpd_after, histtype="step", bins=20, normed=True, lw=2)
plt.legend(("before", "after"))
plt.title("Views per day before and after intervention")
plt.xlabel("Views per day")
plt.ylabel("Frequency")
plt.show()

We can clearly see that the intervention increased the number of hits per day, on average. But in order to quantify the difference in rates, we should use one company's intervention for multiple pages. Since the underlying rate will be different for each page, we should compute the percent change in rate (again, the rate here is hits per day).

Now, let's pretend we have data for n = 100 pages, each of which received an intervention from the same company. To get the percent difference we take (mean(hits per day before) - mean(hits per day after)) / mean(hits per day before):

n = 100

pct_diff = np.zeros(n)

for i in xrange(n):
    vpd_before, vpd_after = simulate_data(.15)
    # % difference. Note: this is the thing we want to infer
    pct_diff[i] = (vpd_after.mean() - vpd_before.mean()) / vpd_before.mean()

plt.hist(pct_diff)
plt.title("Distribution of percent change")
plt.xlabel("Percent change")
plt.ylabel("Frequency")
plt.show()

Now we have the distribution of our parameter of interest! We can query this result in different ways. For example, we might want to know the mode, or (approximation of) the most likely value for this percent change:

def mode_continuous(x, num_bins=None):
    if num_bins is None:
        counts, bins = np.histogram(x)
    else:
        counts, bins = np.histogram(x, bins=num_bins)

    ndx = np.argmax(counts)
    return bins[ndx:(ndx+1)].mean()

mode_continuous(pct_diff, 20)

When I ran this I got 0.126, which is not bad, considering our true percent change is 15. We can also see the number of positive changes, which approximates the probability that a given company's intervention improves hits per day:

(pct_diff > 0).mean()

Here, my result is 0.93, so we could say there's a pretty good chance that this company is effective.

Finally, a potential pitfall: Each page probably has some underlying trend that you should probably account for. That is, even without the intervention, hits per day may increase. To account for this, I would estimate a simple linear regression where the outcome variable is hits per day and the independent variable is day (start at day=0 and simply increment for all the days in your sample). Then subtract the estimate, y_hat, from each number of hits per day to de-trend your data. Then you can do the above procedure and be confident that a positive percent difference is not due to the underlying trend. Of course, the trend may not be linear, so use discretion! Good luck!


packages installed after activating conda environment I want to know after creating and activating a conda environment in terminal. Say venv:

conda create -n venv
source activate venv

Then the prompt will come with the enviroment name (venv).
would the packages installed   (say conda install tensorflow without --name venv) after environment activation only effect inside the environment venv? Or would it affect outside venv?
I don't want to mess up my environments.

It will only affect the environment that you activated.


LSTM model prediction is almost constant I am new to RNN and LSTM and currently experimenting with different settings.
When trying to model time series data in absolute terms (predicted close price), I am faced with the following problems:

Validation loss falling while training loss raising
Prediction on test set is almost constant

Any ideas why is it the case?
#df is the sample with features and target values (close+1)

# MAKE LARGE SAMPLE FOR TESTING
df_train = df[:40000]
df_val = df[40000:50000]
df_test = df[50000:]
df_train.reset_index(drop=True,inplace=True)
df_val.reset_index(drop=True,inplace=True)
df_test.reset_index(drop=True,inplace=True)

# Make x_train, x_val sets by dropping target variable
x_train = df_train.drop(['close+1'], axis=1)
x_val = df_val.drop(['close+1'], axis=1)

# Scale the training data first then fit the transform to the test set
scaler = StandardScaler()
x_train = scaler.fit_transform(x_train)
x_test = scaler.transform(x_val)

# Create y_train, y_test, simply target variable for regression
y_train = df_train['close+1']
y_test = df_val['close+1']

# Define Lookback window for LSTM input
sliding_window = 30

# Convert x_train, x_test, y_train, y_test into 3d array (samples, timesteps, features) for LSTM input
dataXtrain = []
for i in range(len(x_train)-sliding_window-1):
        a = x_train[i:(i+sliding_window), 0:(x_train.shape[1])]
        dataXtrain.append(a)

dataXtest = []
for i in range(len(x_test)-sliding_window-1):
        a = x_test[i:(i+sliding_window), 0:(x_test.shape[1])]
        dataXtest.append(a)

dataYtrain = []
for i in range(len(y_train)-sliding_window-1):
        dataYtrain.append(y_train[i + sliding_window])

dataYtest = []
for i in range(len(y_test)-sliding_window-1):
        dataYtest.append(y_test[i + sliding_window])

# Make data the divisible by a variety of batch_sizes for training
dataXtrain = np.array(dataXtrain[:39680])
dataYtrain = np.array(dataYtrain[:39680])
dataXtest = np.array(dataXtest[:9728])
dataYtest = np.array(dataYtest[:9728])

# Checking input shapes
print('dataXtrain size is: {}'.format((dataXtrain).shape))
print('dataXtest size is: {}'.format((dataXtest).shape))
print('dataYtrain size is: {}'.format((dataYtrain).shape))
print('dataYtest size is: {}'.format((dataYtest).shape))

### ACTUAL LSTM MODEL

batch_size = 256
timesteps = dataXtrain.shape[1]
features = dataXtrain.shape[2]

# Model set-up, stacked 4 layer stateful LSTM
model = Sequential()
model.add(LSTM(512, return_sequences=True, stateful=True,
               batch_input_shape=(batch_size, timesteps, features)))
model.add(LSTM(256,stateful=True, return_sequences=True))
model.add(LSTM(256,stateful=True, return_sequences=True))
model.add(LSTM(128,stateful=True))
model.add(Dense(1, activation='linear'))

model.summary()

reduce_lr = tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.9, patience=5, min_lr=0.000001, verbose=1)

def coeff_determination(y_true, y_pred):
    from keras import backend as K
    SS_res =  K.sum(K.square( y_true-y_pred ))
    SS_tot = K.sum(K.square( y_true - K.mean(y_true) ) )
    return ( 1 - SS_res/(SS_tot + K.epsilon()) )

model.compile(loss='mse',
              optimizer='nadam',
              metrics=[coeff_determination,'mse','mae','mape'])

history = model.fit(dataXtrain, dataYtrain,validation_data=(dataXtest, dataYtest),
          epochs=100,batch_size=batch_size, shuffle=False, verbose=1, callbacks=[reduce_lr])

predictions = model.predict(dataXtest, batch_size=batch_size)
print(predictions)

plt.plot(history.history["loss"][5:])
plt.plot(history.history["val_loss"][5:])
plt.title("model loss")
plt.ylabel("loss")
plt.xlabel("epoch")
plt.legend(["train", "val"], loc="best")
plt.show()

plt.figure(figsize=(20,8))
plt.plot(dataYtest)
plt.plot(predictions)
plt.title("Prediction")
plt.ylabel("Price")
plt.xlabel("Time")
plt.legend(["Truth", "Prediction"], loc="best")
plt.show()

It could be a number of different reasons but when I had that problem in the past it was usually due to too high of a learning rate or the optimizer. I recommended either dropping the initial learning rate or going with vanilla SGD. Occasionally I saw problems with Adam particularly if you have no warmup. You might want to try more general hyperparameter search techniques as well and iterate over things like batch size, forecast length, and lr etc


Algorithm or formula to measure happiness? I am working on a project that takes in comments in a given Facebook page and determines the average happiness rating for it based on users of the page. My question is where would I find a formula, rating, or any other literature that would help me find a way to measure happiness using words and collections of words?

I'm looking for something similar to cosine similarity I suppose, but not to find similar words, but to find the average positivity or happiness related to a word or collection of words.

I'm not entirely sure this is the correct place for this question, but it has to do with data, and big data within Facebook so I am hoping to either get a question or be directed to somewhere that I may be able to find my answer. Thanks in advance for your help.

You should be looking towards Natural Language Processing, specifically at Sentiment Analysis.

The link I provided is a good starting point for learning about sentiment analysis.  If this is what you are looking for, it is available as part of Stanford's Core NLP.


How to select topology for neural network? I was given a target function to design neural network and train: (y = (x1 ∧ x2) ∨ (x3 ∧ x4))

The number of input and number of output seems obvious (4 and 1). And the training data can use truth table.

However, in order to train as a multilayer artificial neural network, I need to choose number of hidden units. May I know where can I find some general guideline for this?

Thank you!

You problem is linearly separable so you can use a single layer perceptron.
Hidden units are necessary only for non linear problems (xor is a classic example).

As for general guidelines, I don't think anyone has ever needed more than one hidden layer. Also the number of neurons in the hidden layer doesn't need to exceed the number of inputs.


Adding the input layer - units with a decimal I took the course Machine Learning A-Z from Udemy and am trying to apply what I learned in the tutorials.  Theye taught us in the "Adding the input layer" portion of an ANN that the units is based off of the input_dim.  Normally the "units = (input_dim + 1) / 2".  In the dataset that I am working with my input_dim=754.  (754 + 1) / 2 = 377.5.  Should I use 377.5 or should I round up or down to a whole number?

#Adding the input layer and the first hidden layer
classifier.add(Dense(units=377.5, kernel_initializer='uniform', activation='relu', input_dim=754))

Round up or down to a whole number. Keras documentation specifies that units should be a positive integer, and I'm not sure what a fractional unit would even mean. Does this work when you try to run it?


Frequency of occurrence - dummy variables I am thinking about it not the first time, namely if I have a variable that I want to convert later to the variable dummy (cities in this case), should I delete lines that occur less often than N times?

For example, the value of new york has occurred 400+ times but there are cities that only appeared once or twice.

What should I do with values that have appeared only once or twice?

print(df[cities].value_counts())

Output:

city1         424
city2         107
city3          35
city4          33
city5          28
city6          24
city7          15
city8           7
city9           4
city10          3
city11          2
city12          1
city13          1
city14          1
city15          1
city16          1
city17          1

There's no general rule that can apply to all cases, and there's a lot of context missing in your post to say anything conclusive.

Having said that, I think that a good approach is to treat the cities with a lot of occurrences each on its own, and then group all others under a 'other' category.

Going further, you could have multiple 'other' groups, grouped by various criteria, for example, geographical criteria, or anything that makes sense in your context.

Hope this helps.


Count occurences of rows based one cell content with pandas I am currently struggling with the pandas framework.
Let's imagine the following Data Frame:
customer | order_number | product
------------------------------------
A        | O1           | Product_1
A        | O1           | Product_2
A        | O2           | Product_X
B        | O5           | Product_1
C        | O6           | Product_2
C        | 06           | Product_X

What I am trying to do is to count the occurences of "products" based on "order_number". This means that I would like to know how how often e.g. Product_1 and Product_2 were sold within one order. Of course I am interested in every available combination of products. I tried to use df.groupby['order_number'].count() but this is certainly not the data I want. At this point I am not interested in order_numbers nor the customers! Just the available sets of products and how often they appear within the data frame.
Thanks in advance!
Update: Based on my example this explains what I would like to get
Product_1 && Product_2 | One Time
Product_X | One Time
Product_1 | One Time
Product_2 && Product_X | One Time

When asking code related questions, its always a good practice to at least provide a sample of the data. Since, you've not given one, I'll create it based on your question.
import pandas as pd

df = pd.DataFrame([('A','01','prod_1'),
                   ('A','01', 'prod_2'),
                   ('A','02','prod_X'),
                   ('B','05','prod_1'),
                   ('C','06','prod_2'),
                   ('C','06','prod_X')],
                 columns=['customer', 'ordernum', 'product'])
print (df)
        customer ordernum product
0        A       01  prod_1
1        A       01  prod_2
2        A       02  prod_X
3        B       05  prod_1
4        C       06  prod_2
5        C       06  prod_X

Now, you can use size() or Counter like,
val_count = df.groupby(['product','ordernum']).size()
print(val_count)
product  ordernum
prod_1   01          1
         05          1
prod_2   01          1
         06          1
prod_X   02          1
         06          1
dtype: int64

prod_order_count = df.groupby(['product', 'ordernum']).agg(Counter)
print(prod_order_count)
                  customer
product ordernum
prod_1  01        {'A': 1}
        05        {'B': 1}
prod_2  01        {'A': 1}
        06        {'C': 1}
prod_X  02        {'A': 1}
        06        {'C': 1}


Better way of writing following Pandas Python code Following code is given, we need to sum up the medals.

import pandas as pd

# Defining the three dataframes indicating the gold, silver, and bronze medal counts
# of different countries
gold = pd.DataFrame({'Country': ['USA', 'France', 'Russia'],
                         'Medals': [15, 13, 9]}
                    )
silver = pd.DataFrame({'Country': ['USA', 'Germany', 'Russia'],
                        'Medals': [29, 20, 16]}
                    )
bronze = pd.DataFrame({'Country': ['France', 'USA', 'UK'],
                        'Medals': [40, 28, 27]}
                    )

I wrote this working solution. However, it feels very un-pythonic:
And I feel there is a better way of approaching this.

df = gold.set_index('Country')["Medals"].add(silver.set_index('Country')["Medals"], fill_value=0).add(bronze.set_index('Country')["Medals"], fill_value=0)
df = pd.DataFrame(df.sort_values(ascending=False))
print(df)```

I was provided with this answer by someone:

# Set the index of the dataframes to 'Country' so that you can get the countrywise
# medal count
gold.set_index('Country', inplace = True)
silver.set_index('Country', inplace = True)
bronze.set_index('Country', inplace = True)

# Add the three dataframes and set the fill_value argument to zero to avoid getting
# NaN values
total = gold.add(silver, fill_value = 0).add(bronze, fill_value = 0)

# Sort the resultant dataframe in a descending order
total = total.sort_values(by = 'Medals', ascending = False)

# Print the sorted dataframe
print(total)


Pandas Dataframe to DMatrix I am trying to run xgboost in scikit learn.  And I am only using Pandas to load the data into a dataframe. How am I supposed to use pandas df with xgboost?  I am confused by the DMatrix routine required to run the xgboost algorithm.

You can use the dataframe's .values method to access raw data once you have manipulated the columns as you need them.

E.g.

train = pd.read_csv("train.csv")
target = train['target']
train = train.drop(['ID','target'],axis=1)
test = pd.read_csv("test.csv")
test = test.drop(['ID'],axis=1)

xgtrain = xgb.DMatrix(train.values, target.values)
xgtest = xgb.DMatrix(test.values)

Obviously you may need to change which columns you drop or use as the training target. The above was for a Kaggle competition, so there was no target data for xgtest (it is held back by the organisers).


How to Work with Imbalanced Data I am building a binary classifier from a set of feature vectors some of which are categorical like Yes or No (two options). I am replacing them with 1 and 0 and since there is strong imbalance between 1 and 0 my model learns basically on those features. I am afraid that this particular feature might be an artifact - samples that have 0 can also sometimes be classified as 1. What to do in this situation? Should I drop the feature completely?

Update:

Let me explain in detail the problem since the answers mostly focus on imbalanced classes. I have a dataset that comprises of around 30 features and binary classes {0,1}. Features are mostly numerical (continuous), but there are also binary categorical features like YES/NO, MALE/FEMALE etc.

One aspect of this dataset is imbalanced classes (there are more ones than zeros) and the other aspect is that one categorical feature, lets say $x$ (YES/NO) is not very balanced also. In fact if you make predictions solely based on $x$ like: $x = 1 \rightarrow 1, x=0 \rightarrow 0$ you will perform better than a naive model, which predicts only $0$ (remember imbalanced classes).

Now, my dilemma is what to do in that case? Should I remove that variable completely from modelling or maybe use some techniques of bias removal?

Generally, there are three main solutions for imbalance data classification.

Over-sampling the thin class by generating artificial data or some other ways. This solution can be helpful in situations in which generating accurate artificial data is possible.
Under-sampling the populous class. This solution also can be suitable for situations in which the ratio of classes population is not very low (for example 30% to 70% rather than 1% to 99%)
Using a proper loss function which can handle imbalanced data. There are many metrics that can evaluate the performance of the classifiers at the presence of imbalanced data.

Disclaimer:
If you use python, PyCM module can help you to find out these metrics.

Here is a simple code to get the recommended parameters from this module:

>>> from pycm import *

>>> cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})

>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]

After that, each of these parameters you want to use as the loss function can be used as follows:

>>> y_pred = model.predict      #the prediction of the implemented model

>>> y_actu = data.target        #data labels

>>> cm = ConfusionMatrix(y_actu, y_pred)

>>> loss = cm.Kappa             #or any other parameter (Example: cm.SOA1)


How to handle non-stationary data in online neural network based one-class classifier for anomaly detection? I have developed online neural network based one-class classifier and also enabled it for forgetting mechanism. So, now online learning with forgetting mechanism is possible. But how to handle if data is non-stationary. As it is one-class classification so, suppose we have two class data i.e. normal and outlier class then trained by only normal class data in online fashion. But training data might have different distribution compared to distribution of normal and outlier data in testing as their distribution are changing. So, how algorithm will handle this drift/shift.

One more doubt regarding non-stationary data handling: whether we will pass trained labelled data once and further labelled data for training are collected from unlabelled testing data?

Usually, non-stationary series can be dealt by differencing the time series once or twice. You can add verification to your tool - checking auto-correlation plot (or partial correlation plot).


How Do Machine Learning Models Work and Remember? I am new to machine learning. I am confused about how a machine learning model remembers what it learns. And how it learns.

I know the basic workflow of machine learning: first is data gathering, then data preprocessing, then creating training and test data sets, then training the ML learning algorithm. After training, we test the accuracy of our model with test data. But I am confused about how the algorithm is trained and how it remembers what it learns.

ML algorithms all learn a set of numbers associated with whatever model you're training. For a neural network, it's the weights on the network links. For regression, it's the coefficients. And so on. These numbers you can store and use later (most ML libraries have utilities for doing this). You use the coefficients to make predictions on data on which you have NOT trained the model.

The ML algorithm uses some sort of optimization for learning the coefficients. It might be gradient descent, or stochastic gradient descent, or xgboost, or any of a number of optimization algorithms. These are typically used to minimize a cost function. The definition of that cost function is model-dependent.


Why convolutions always use odd-numbers as filter_size If we have a look to 90-99% of the papers published using a CNN (ConvNet).
The vast majority of them use filter size of odd numbers:{1, 3, 5, 7} for the most used.

This situation can lead to some problem: With these filter sizes, usually the convolution operation is not perfect with a padding of 2 (common padding) and some edges of the input_field get lost in the process...

Question1: Why using only odd_numbers for convolutions filter sizes ?

Question2: Is it actually a problem to omit a small part of the input_field during the convolution ? Why so/not ?

The convolution operation, simply put, is combination of element-wise product of two matrices. So long as these two matrices agree in dimensions, there shouldn't be a problem, and so I can understand the motivation behind your query.

A.1. However, the intent of convolution is to encode source data matrix (entire image) in terms of a filter or kernel. More specifically, we are trying to encode the pixels in the neighborhood of anchor/source pixels. Have a look at the figure below:

Typically, we consider every pixel of the source image as anchor/source pixel, but we are not constrained to do this. In fact, it is not uncommon to include a stride, where in we anchor/source pixels are separated by a specific number of pixels.

Okay, so what is the source pixel? It is the anchor point at which the kernel is centered and we are encoding all the neighboring pixels, including the anchor/source pixel. Since, the kernel is symmetrically shaped (not symmetric in kernel values), there are equal number (n) of pixel on all sides (4- connectivity) of the anchor pixel. Therefore, whatever this number of pixels maybe, the length of each side of our symmetrically shaped kernel is 2*n+1 (each side of the anchor + the anchor pixel), and therefore filter/kernels are always odd sized.

What if we decided to break with 'tradition' and used asymmetric kernels? You'd suffer aliasing errors, and so we don't do it. We consider the pixel to be the smallest entity, i.e. there is no sub-pixel concept here.

A.2
The boundary problem is dealt with using different approaches: some ignore it, some zero pad it, some mirror reflect it. If you are not going to compute an inverse operation, i.e. deconvolution, and are not interested in perfect reconstruction of original image, then you don't care about either loss of information or injection of noise due to the boundary problem. Typically, the pooling operation (average pooling or max pooling) will remove your boundary artifacts anyway. So, feel free to ignore part of your 'input field', your pooling operation will do so for you.

--

Zen of convolution:

In the old-school signal processing domain, when an input signal was convolved or passed through a filter, there was no way of judging a-prior which components of the convolved/filtered response were relevant/informative and which were not. Consequently, the aim was to preserve signal components (all of it) in these transformations.

These signal components are information. Some components are more informative than others. The only reason for this is that we are interested in extracting higher-level information; Information pertinent towards some semantic classes. Accordingly, those signal components that do not provide the information we are specifically interested in can be pruned out. Therefore, unlike old-school dogmas about convolution/filtering, we are free to pool/prune the convolution response as we feel like. The way we feel like doing so is to rigorously remove all data components that are not contributing towards improving our statistical model.


Keras model.predict giving different shape from training label array I'm using the following code to try and learn tensorflow. I've clearly specified the shapes of the training and validation X and y arrays.
import numpy as np
import tensorflow as tf

f = lambda x: 2*x
Xtrain = np.random.rand(400,1)
ytrain = f(Xtrain)
Xval = np.random.rand(200,1)
yval = f(Xval)

model = tf.keras.models.Sequential([
    tf.keras.layers.Dense(10, activation='relu'),
    tf.keras.layers.Dense(10, activation='relu'),
    tf.keras.layers.Dense(10, activation='relu'),
    tf.keras.layers.Dense(10, activation='relu')
])

model.compile(optimizer='adam',
              loss=tf.keras.losses.MeanSquaredError()
             )

model.fit(Xtrain, ytrain, epochs=50, verbose=0)

When I run yval.shape, model.predict(Xval).shape, I get the output ((200, 1), (200, 10)). I'm not able to understand where these extra 9 dimensions are coming from. Even the Xval prediction should have the shape (200, 1).

The 10 outputs came from the fact that you have 10 neurons in the final layer of your network.
If you change your model to
model = tf.keras.models.Sequential([
    tf.keras.layers.Dense(10, activation='relu'),
    tf.keras.layers.Dense(10, activation='relu'),
    tf.keras.layers.Dense(10, activation='relu'),
    tf.keras.layers.Dense(1, activation='relu')
])

its output will have a shape of (200, 1).


algorithm to auto-download articles from the internet I have an issued in my homeworks and I thinked if there is an rxisted algorithm or if can i create new one that takes key words like "germany" and "polution" and parses in google scholar. It parses fpr example the 10 first reults and each time it finds the key words in a specific part from the article ( just in the introduction) it downloads it.

Any one can help me ith any infrmation that can help me in this idea? Is it possible? what kind of alforithms should it be? ...

what you are asking for is known as web scraping
, if you don't what it is, you can read about it.
can you please clarify your question more, so you can get the right answer.

what programming languages you'll be using, if you are planning on using python, there's a great framework for web scraping called Scrapy.
I found this Parser which does what you are asking for.
finally, you can check this related question.


LSTM train on varying length sequences So I have around 600 classes and 17k sentences for text classification.The feature vectors are trained using word2vec.

Vector_size = 100 ;
LSTM hidden units= 100

I am using class weight for imbalance problem.I wanted to know if LSTM is affected by sparsity in training data when max length is kept 18 but minimum length of sentences is 1.

And, how can we use keras train_on_batch to create batches of 5,10,25 for variable length instead of padding.

The solutions for sequences with different length are:

1- padding, but you do not want to.

2-truncate the sequences, so if you have for instance 3 sequences of different lengths you truncate the sequences so that they have all the length of the shortest one.

3- using timesteps of 1, but in this case your model will not learn the dependencies between different timesteps

4- using online learning with batch size = 1.

For the fourth solution this is an example in keras to create a random time-length batches of training data: Training an RNN with examples of different lengths in Keras. So you can have different batches with different lengths, and since you are using online learning with batch_size = 1 this can be a good solution. The fourth solution can also have advantages and disadvantages. For online learning the training is slower, but usually you need less epochs compared to sgd with batch size > 1 to have a good model. If you use stateful lstm, this could lead you to an unstable model, not always, but can happen.


How can we convert time series data to supervised learning problem? I am preparing a data for machine learning model. I want to deal with time series data as normal supervised learning prediction. Let's say I have a data for car speed and I have several cars models such as

+-----+---------+-------------+
| day |  Model  |   Speed     |
+-----+---------+-------------+
|   1 | Bentley | 20.47 km/h  |
|   2 | Bentley | 32.22 km/h  |
|   3 | Bentley | 23.11 km/h  |
|   1 | BMW     | 37.60 km/h  |
|   2 | BMW     | 27.90 km/h  |
|   3 | BMW     | 40.47 km/h  |

so I want to deal with several model in training so that predict the speed for Bentley and BMW.

I have converted the data for training like this :

+---------+------------+------------+-------------------+
|  Model  |   day_1    |     day_2  |    label == day_3 |
+---------+------------+------------+-------------------+
| Bentley | 20.47 km/h | 32.22 km/h | 23.11 km/h        |
| BMW     | 37.60 km/h | 27.90 km/h | 40.47 km/h        |
+---------+------------+------------+-------------------+

Is it a correct approach?

Since you always have a fixed number of days, I think your approach is good. In order to help the learning algorithm you might consider adding some statistics as features for every instance, for example:

mean of the last N days
difference dayN-day(N-1) (evolution)
...

Of course this can work only if there is actually a dependency between the features and the predicted speed.


Voting classifier using grid search for Time Series I have three models:

Arima
Auto ARIMA
Double Exponential Smoothing

I would like to apply an ensemble method - a voting method and allow the classifier to learn weights for these three models.

I have checked the votingclassifier present in scikit learn. It requires:
fit(x,y) to run.

Time series object that is present in series object don't have y.

How do you apply a voting classifier and learn weights through grid search?

In a time series problem, your input (x) and output (y) are, in the most basic case, the same variable, recorded at different points in time.

If you have T time points: 1, 2, 3, ... T then we think of x as an array of data points, with an index t to access each time point.

Typically y will simply be your x array shifted forward in time (in the below example, by 1 time unit, so $y_t = x_{t-1}$ in vector notation or y[t] == x[t-1] in array notation):

  x  |   y
-----|-----
 0.1 |  NaN
 0.2 |  0.1
 0.3 |  0.2
 0.3 |  0.3
 0.4 |  0.3
 0.3 |  0.4
 0.5 |  0.3
 1.0 |  0.5

Pandas has a shift() method for a time series, which allows you to shift your x column at different time intervals, and create a new y column using that shifted series. See https://stackoverflow.com/questions/10982089/how-to-shift-a-column-in-pandas-dataframe

You can add levels of complexity to this by including multiple time lags, as well as other variables, but this explains the basic principle of how to convert time series forecasting into a supervised learning problem.


print a specific column with a condition using pandas I have a data set which contains 5 columns, I want to print the content of a column called 'CONTENT' only when the column 'CLASS' equals one.
I know that using .query  allows me to select a condition, but it prints the whole data set. I tried to drop the unwanted columns, but I finished up with unaligned and not completed data: -

CONTENT CLASS
0 Huh, anyway check out this you[tube] channel: ... 1
1 Hey guys check out my new channel and our firs... 1
2 just for test I have to say murdev.com 1
3 me shaking my sexy ass on my channel enjoy ^^ \xef\xbb\xbf 1
4 watch?v=vtaRGgvGtWQ Check this out .\xef\xbb\xbf 1
5 Hey, check out my new website!! This site is a... 1
6 Subscribe to my channel \xef\xbb\xbf 1
8 You should check my channel for Funny VIDEOS!!\xef\xbb\xbf 1
9 and u should.d check my channel and tell me wh... 1
10 Hey subscribe to me\xef\xbb\xbf 1
11 Once you have started reading do not stop. If... 1
12 https://twitter.com/GBphotographyGB\xef\xbb\xbf 1
13 subscribe like comment\xef\xbb\xbf 1
14 please like :D https://premium.easypromosapp.c... 1
15 Hello! Do you like gaming, art videos, scienti... 1
17 http://www.ebay.com/itm/171183229277?ssPageNam... 1
18 http://ubuntuone.com/40beUutVu2ZKxK4uTgPZ8K\xef\xbb\xbf 1
19 We are an EDM apparel company dedicated to bri... 1
21 subscribe to my channel people :D\xef\xbb\xbf 1
22 Show your AUBURN PRIDE HERE: http://www.teespr... 1
24 CHECK OUT MY CHANNEL 1
25 marketglory . com/strategygame/andrijamatf ear... 1
26 Hey guys! Im a 12 yr old music producer. I mak... 1
27 Check me out! I\'m kyle. I rap so yeah \xef\xbb\xbf 1
29 Subscribe to me for free Android games, apps.. \xef\xbb\xbf 1
30 everyone please come check our newest song in ... 1
32 sub my channel for no reason --\xef\xbb\xbf 1
33 Check out my dubstep song "Fireball", made wit... 1
36 Check my channel please! And listen to the bes... 1
37 SUB 4 SUB PLEASE LIKE THIS COMMENT I WANT A SU... 1
.. ... ...
297 Please help me go to college guys! Thanks from... 1
298 https://www.facebook.com/SchoolGeniusNITS/phot... 1
299 I am so awesome and smart!!! Sucscribe to me!\xef\xbb\xbf 1
300 Follow 4 Follow @ Va... 1
301 http://hackfbaccountlive.com/?ref=4436607 psy... 1
302 https://www.facebook.com/nicushorbboy add mee ... 1
303 im sorry for the spam but My name is Jenny. I ... 1
305 please throw a sub on my channel\xef\xbb\xbf 1
307 Go to my channel if u want to see a fly gettin... 1
309 COME AND CHECK OUT MY NEW YOUTUBE CHHANEL, GOI... 1
311 Please check out my vidios guys\xef\xbb\xbf 1
313 Hey guys can you check my channel out plz. I d... 1
315 PLEASE SUBSCRIBE ME!!!!!!!!!!!!!!!!!!!!!!!!!!!... 1
317 Check out my Music Videos! Fuego - U LA LA Rem... 1
318 Check out my Music Videos! and PLEASE SUBSCRIB... 1
319 www.marketglory.com/strategygame/lordviperas\xef\xbb\xbf 1
323 Limit sun exposure while driving. Eliminate th... 1
325 http://hackfbaccountlive.com/?ref=5242575\xef\xbb\xbf 1
327 https://www.facebook.com/FUDAIRYQUEEN?pnref=st... 1
329 FOLLOW MY COMPANY ON TWITTER thanks. https:/... 1
331 Hey come check us out were new on youtube let ... 1
333 Look at the pictures, if not difficult http://... 1
335 Hey guys can you check my YouTube channel I kn... 1
337 https://www.facebook.com/tofikmiedzynB/photos/... 1
338 https://www.facebook.com/eeccon/posts/73394924... 1
339 http://www.bing.com/explore/rewards?PUBL=REFER... 1
340 Please do buy these new Christmas shirts! You ... 1
341 Free my apps get 1m crdits ! Just click on the... 1
347 subscribe to me for call of duty vids and give... 1
348 hi guys please my android photo editor downloa...

In Pandas slice notation one must first indicate the condition to filter on and only eventually the column to select: in particular for the example at hand we have:

df[df['CLASS']==1]['CONTENT']