Time-series multi-step generalization from single step model I have built a generic stacked lstm model of the form:
self.model = tf.keras.Sequential([
            lstm_layer(n_neurons, return_sequences=True),
            lstm_layer(n_neurons, return_sequences=True),
            lstm_layer(n_neurons, return_sequences=True),
            lstm_layer(n_neurons),
            tf.keras.layers.Dense(n_output)
        ])

with the data shaped as x_train = (total_samples,timestep, features) and y_test = (total_samples,n_output,1) and then I fit the model with a validation split of 30%.
My question is:
Is there a good theoretical reason for why the model perfectly predicts one step ahead (n_output = 1) but when I try to set n_output>1 the model becomes very, very imprecise? (loss<val_loss from epoch 1)
Is it a model/data dependant thing or am I botching the data preparation?

Welcome to the community Fra, below you can find a worked out example implementing a multivariate several input features (as I think is your case) time series forecasting, predicting multiple future steps (multi-step forecast), applying bayesian hyperparametrization.
It is based on a from simpler to more complex approach, so you can see there are few layers in the architecture to begin with, and what is being hyperparametrized is the units in such layers. This is a whole worked out example, in case you can almost directly apply it and try out modelling first with a single output node for a single-step prediction, and then add more future step predictions to check if you still run on the same issue as you described (beggining with n_output = 1 and then incrementing this, e.g. in my case it is 72 future values to predict).
For a dataset of this type:

and trying to predict several future time steps (the deep blue dots) for the temperature values:

we have some code like:

train-validation dataset splitter functions:
  def multivariate_data(dataset, target, start_index, end_index, history_size, target_size, step, single_step=False):
    data = []
    labels = []

    start_index = start_index + history_size
    if end_index is None:
      end_index = len(dataset) - target_size

    for i in range(start_index, end_index):
      indices = range(i-history_size, i, step)
      data.append(dataset[indices])

      if single_step:
        labels.append(target[i+target_size])
      else:
        labels.append(target[i:i+target_size])

    return np.array(data), np.array(labels)

This is what we get by applying that function (in your case, you do not need to apply the sampling interval of 6 timesteps, you can set it to 1; this is in case you want to downsample a maybe too long history):

function implementation to build the train-eval sets:
  past_history = 720 # historic values to consider for each prediction
  future_target = 72 # number of future values to predict
  STEP = 6 # sampling frequency (it makes taking 720/6 historic points for each sample)

  x_train_multi, y_train_multi = multivariate_data(dataset_standardized, dataset_standardized[:, 1], 0,TRAIN_SPLIT, past_history, future_target, STEP)
  x_val_multi, y_val_multi = multivariate_data(dataset_standardized, dataset_standardized[:, 1], TRAIN_SPLIT, None, past_history, future_target, STEP)

This way you have:

Single window of past history : (120, 3)
Target temperature to predict : (72,)

For the training phase, we define the batch size and other values and build our training and validation sets:
BATCH_SIZE = 256
BUFFER_SIZE = 10000

train_data_multi = tf.data.Dataset.from_tensor_slices((x_train_multi, y_train_multi))
train_data_multi = train_data_multi.cache().shuffle(BUFFER_SIZE).batch(BATCH_SIZE).repeat()

val_data_multi = tf.data.Dataset.from_tensor_slices((x_val_multi, y_val_multi))
val_data_multi = val_data_multi.batch(BATCH_SIZE).repeat()

and defining a simple architecture to begin with, we have:
multi_step_model = tf.keras.models.Sequential()
multi_step_model.add(tf.keras.layers.LSTM(32,
                                          return_sequences=True,  # "Boolean. Whether to return the last output. in the output sequence, or the full sequence"
                                          input_shape=x_train_multi.shape[-2:]))
multi_step_model.add(tf.keras.layers.LSTM(16, activation='relu'))
multi_step_model.add(tf.keras.layers.Dense(72))

multi_step_model.compile(optimizer=tf.keras.optimizers.RMSprop(clipvalue=1.0), loss='mae')

for which we can check the performance on the validation set as:
EVALUATION_INTERVAL = 200
EPOCHS = 10

multi_step_history = multi_step_model.fit(train_data_multi, epochs=EPOCHS,
                                          steps_per_epoch=EVALUATION_INTERVAL,
                                          validation_data=val_data_multi,
                                          validation_steps=50)

having the predictions in red below, after de-scaling the values to its original scale:
for x, y in val_data_multi.take(3):
    x_denormalized = (x[0])*temps_std+temps_mean
    y_denormalized = (y[0])*temps_std+temps_mean

    multi_step_plot(x_denormalized, y_denormalized, (multi_step_model.predict(x)[0])*temps_std+temps_mean)

and maybe the most interesting part for your use case, you can try implementing hyperparametrization of some params as hidden layers units as follows with the keras bayesian tuner:
from kerastuner import BayesianOptimization

def build_model_2(hp):
    model = keras.Sequential()
    model.add(keras.layers.LSTM(units=hp.Int('units',min_value=16,
                                        max_value=32,
                                        step=16),
                                        #activation='relu',
                                        return_sequences=True,
                                        input_shape=x_train_multi.shape[-2:]))
    model.add(keras.layers.LSTM(units=hp.Int('units',min_value=16,
                                        max_value=32,
                                        step=16),
                                        activation='relu'))
    model.add(keras.layers.Dense(72))

    model.compile(loss='mae', optimizer=tf.keras.optimizers.RMSprop(clipvalue=1.0),

                   metrics=['mae'])
    return model

# define model
bayesian_opt_tuner_2 = BayesianOptimization(
    build_model_2,
    objective='mae',
    max_trials=3,
    executions_per_trial=1,
    directory=os.path.normpath('C:/keras_tuning'),
    project_name='timeseries_temp_ts_test_from_TF_ex_multivar_multistep_2',
    overwrite=True)

EVALUATION_INTERVAL = 200
EPOCHS = 10
bayesian_opt_tuner_2.search(train_data_multi,
             epochs=EPOCHS,
             validation_data=val_data_multi,
             validation_steps=50,
             steps_per_epoch=EVALUATION_INTERVAL)

# and selecting the best model:

    best_MULTIVAR_MULTISTEP_LSTM_model_2 = bayesian_opt_tuner_2.get_best_models(num_models=1)[0]

we have our new model predictions:

which gave an improved MAE on the validation set.


Sobel Operator || Pixel Constraint I am trying out the Sobel vertical operator to identify vertical edges in a picture. In each image there is one rectangle but the difference is that in 1 image the 2 vertical lines in the rectangle is 1 pixel apart whereas in the other it is more than 1 pixel apart as shown below.

Now when i run the sobel vertical filter my expectation is that in both cases i should get 2 edges. But to my surprise it returns 2 edges for the bigger rectangle but only one edge for the smaller rectangle as shown below.

My Python code is as follows.

from PIL import Image
import numpy as np
im = Image.open('Test.png').convert('L')
im.save('greyscale.png')
pix_val = np.array(list(im.getdata())).reshape(81,91)
for i in range(12,14):
    for j in range(12,78):
        pix_val[i,j]=255
for i in range(63,65):
    for j in range(12,78):
        pix_val[i,j]=255
for i in range(14,63):
    for j in range(12,14):
        pix_val[i,j]=255
for i in range(14,63):
    for j in range(76,78):
        pix_val[i,j]=255
Oim1=Image.fromarray(pix_val.astype(np.uint8))
Oim1.save('out1.bmp')
pix_val2=np.delete(pix_val,range(16,75), 1)
Oim2=Image.fromarray(pix_val2.astype(np.uint8))
Oim2.save('out2.bmp')
from scipy import ndimage
dy1 = ndimage.sobel(Oim1, 1)
Oim3=Image.fromarray(dy1.astype(np.uint8))
Oim3.save('Sobel1.bmp')
dy2 = ndimage.sobel(Oim2, 1)
Oim4=Image.fromarray(dy2.astype(np.uint8))
Oim4.save('Sobel2.bmp')

Is this a limitation of sobel operator that it cant detect edges that are very near to each other?

This is not a limitation of the Sobel operator, if the boundaries are clear, even a single pixel wide rectangle should be detectable.

I ran your code, but just used an empty image as the "im" image and it worked fine for me.

Here is my code (similar to yours, just without 'Test.png' and without saving the images):

from PIL import Image
import numpy as np
from scipy import ndimage

im = Image.fromarray(np.zeros((81,91)).astype(np.uint8))
pix_val = np.array(list(im.getdata())).reshape(81,91)
for i in range(12,14):
    for j in range(12,78):
        pix_val[i,j]=255
for i in range(63,65):
    for j in range(12,78):
        pix_val[i,j]=255
for i in range(14,63):
    for j in range(12,14):
        pix_val[i,j]=255
for i in range(14,63):
    for j in range(76,78):
        pix_val[i,j]=255

Oim1 = Image.fromarray(pix_val.astype(np.uint8))
pix_val2 = np.delete(pix_val,range(16,75), 1)
Oim1.show()

Oim2 = Image.fromarray(pix_val2.astype(np.uint8))
Oim2.show()

dy1 = ndimage.sobel(Oim1, 1)
Oim3 = Image.fromarray(dy1.astype(np.uint8))
Oim3.show()

dy2 = ndimage.sobel(Oim2, 1)
Oim4 = Image.fromarray(dy2.astype(np.uint8))
Oim4.show()

This is what I get:


Scaling features separately? I have some features which are in the thousands, which I scale to the max values of these. This solves the general scaling problems, as well as preserves an important absolute value relationship between these features that would otherwise be lost.

But, I also have some other features which are naturally in [0,1] (since they're derived from ratios) right from the start. Does it make sense to simply skip scaling of these, since dividing them by the same max value as above (e.g. 10,000) kind of ruins them?

Obviously I've tested it out, and found that it works in practice, but are there any sound mathematical arguments against this? I haven't seen or heard any thoughts on this matter.

Yes, these features should not be scaled together. Normally ALL features are scaled individually, so grouped scaling is the exception not the norm. From the Wikipedia on feature scaling:

  Since the range of values of raw data varies widely, in some machine
  learning algorithms, objective functions will not work properly
  without normalization. For example, the majority of classifiers
  calculate the distance between two points by the Euclidean distance.
  If one of the features has a broad range of values, the distance will
  be governed by this particular feature. Therefore, the range of all
  features should be normalized so that each feature contributes
  approximately proportionately to the final distance.

You can scale some features together if you know that they exist on the same scale, i.e. that "Sales January" is scaled together with "Sales February". But you should not scale things together that exist on different scales, i.e. "Sales January" and "Discount Ratio" should be scaled separately.


Selecting most relevant word from lists of candidate words Let's suppose I have 1000's of training examples where each consists of a bucket e.g. 'engineering' or 'management' and a list of tags e.g. ['software', 'python', 'product'] where a human has selected the most relevant tag for the use case e.g.'software'.

So our data is like:

bucket         tags                        best_tag
engineering    [fullstack, software]       software
engineering    [java, python, software]    software
management     [technical, product]        product

What kind of model or approach would suit taking a list of tags and predicting the best tag based on some kind of underlying latent hierarchy?

There are many ways you could approach this problem

Word embeddings

If you have word embeddings at hand, you can look at the distance between the tags and the bucket and pick the one with the smallest distance.

Frequentist approach

You could simply look at the frequency of a bucket/tag pair and choose this. Likely not the best model, but might already go a long way.

Recommender system

Given a bucket, your goal is to recommend the best tag. You can use collaborative filtering or neural approaches to train a recommender. I feel this could work well especially if the data is sparse (i.e. lots of different tags, lots of buckets).

The caveat I would see with this approach is that you would technically always compare all tags, which only works if tag A is always better than tag B regardless of which tags are proposed to the user.

Ranking problem

You could look at it as a ranking problem, I recommend reading this blog to have a better idea of how you can train such model.

Classification problem

This becomes a classification problem if you turn your problem into the following: given a bucket, and two tags (A & B), return 0 if tag A is preferred, 1 if tag B is preferred. You can create your training data as every combination of two tags from your data, times 2 (swap A and B).

The caveat is that given N tags, you might need to do a round-robin or tournament approach to know which tag is the winner, due to the pairwise nature.

Recurrent/Convolutional network

If you want to implicitly deal with the variable-length nature of the problem, you could pass your tags as a sequence. Since your tags have no particular order, this creates a different input for each permutation of the tags. During training, this provides more data points, and during inference, this could be used to create an ensemble (i.e. predict a tag for each permutation and do majority voting).

If you believe that it matters in which order the tags are presented to the user, then deal with the sequence in the order it is in your data.

Your LSTM/CNN would essentially learn to output a single score for each item, such that the item with the highest score is the desired one.


How to decide power of independent variables in case of non-linear polynomial regression? Consider one dependent variable 'Y' and 10 independent variables or features- X1, X2, X3, ... X10.

I want to create a non-linear polynomial regression model such that-

Y ~ a1.X1^b1 + a2.X2^b2 + .... + a10.X10^b10

I was wondering is there any algorithm that will determine best possible values for powers of independent variables that is values of b1, b2, ... b10 from data.

If all you care about is the quality of predictions (as opposed to explanatory power), skip linear models altogether and use gradient boosted trees instead. Gradient boosting can generally learn polynomial splines with ease, and you don't have to manually make a bunch of polynomial predictors yourself.

By the way, gradient boosting is implemented in Python's scikit-learn library, R's caret library, and Java/Scala's Weka library.


scipy fit for t distribution seems broken for bi-modal data I am using the scipy.stats.t.fit function, and I am surprised by the results. If I fit on some bimodal data, say

data=[1,1,1,1,5,5]

I get

df=0.39126249808550329
loc=1.0
scale=5.7172845190830792e-21

That is, the scale is effectively zero, and I will never be able to sample anything near 5, just the more frequent data point 1.

I guess you really can't fit on data that is too different from a t-distribution - but is scipy really giving the best t dist fit to the data? I would think that if I compute a sample mean and variance myself, i.e,

df = 5
loc = 2.33
scale = 1.88

That I'd have a better fit, although I haven't computed the likelihood of sampling [1,1,1,5,5] from these two t distributinos.

The t-test has many assumptions. That dataset violates several of them:

Data should be sufficiently large (>30 independent points)
Data should be approximately normally distributed

Given that the assumptions are violated, you can not expect to valid results.


How Can I Compute Information-Gain for Continuous- Valued Attributes Decision trees handle only discrete values, but the continuous values we need to transform to discrete.  My question is HOW?

I know the steps which are:

Sort the value A in increasing order.
Find the midpoint between the values of $a_i$ and $a_{i+1}$.
Find entropy for each value.

I have this example can someone explain how we calculated the entropy values?

In decision trees, the (Shannon) entropy is not calculated on the actual attributes, but on the class label. If you wanted to find the entropy of a continuous variable, you could use Differential entropy metrics such as KL divergence, but that's not the point about decision trees.

When finding the entropy for a splitting decision in a decision tree, you find a threshold (such as midpoint or anything you come up with), and count the amount of each class label on each size of the threshold. For example:

Var1 | Class
0.75 | 1
0.87 | 0
0.89 | 1
0.96 | 0
1.02 | 1
1.05 | 1
1.14 | 1
1.25 | 1

Let's assume a threshold of 0.99 because I want two equal-sized bins and that's the midpoint between 0.96 and 1.02. The classes on the left are [1, 0, 1, 0] (since it's half/half, entropy is 1.0), and on the right [1, 1, 1, 1] (since it's all equal, entropy is 0.0).

To make a table like the one you show here, you need to define N threshold points (2.0, 3.5, ..., 7.5, in your case), which will produce N+1 bins of data, you take the class of each bin and calculate the entropy. A more correct visualization to your table should be changing the "Split point" values to "2.0 or lower", "Between 2.0 and 3.5", "Between 3.5 and 4.5", ..., "7.5 or higher".

Note this all applies to classification (decision) trees, and not regression trees, where the "class" is actually a continuous variable.


convert keywords in one column into several dummy columns My dataframe has some free text fields named: {'title', 'description', 'location'}

I prepared this text column by: concatenating all into a new column, dropping numbers, dropping words less than 3 char, etc...

As final step in preparing, I removed Stop Words. So, probably at this moment the column only contains relevant keywords.

How can I convert these keywords sentences (several words per row, as space separated words, in just one string column) into dummy columns and get the 0-1 values when, per row, this column contains the keyword?

I was checking the CountVectorizer object, but I couldn't find any use case like this one. So, it is probably not the proper tool to use...

Let's say that your dataframe is called df and the column with your preprocessed text as text.

What you want to do, as you already thought, is to apply a CountVectorizer.

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
import pandas as pd

count = CountVectorizer(binary = True)
bow = count.fit_transform(df['text'].values)

pd.DataFrame(bow.toarray(), columns = count.get_feature_names())

If you remove the binary parameter from the CountVectorizer you will get the actual frequency of the keyword and not just the appearance with 0-1.


Methods for standardizing / normalizing different rank scales I know there is the normal subtract the mean and divide by the standard deviation for standardizing your data, but I'm interested to know if there are more appropriate methods for this kind of discrete data. Consider the following case.

I have 5 items that have been ranked by customers. First 2 items were ranked on a 1-10 scale. Others are 1-100 and 1-5. To transform everything to a 1 to 10 scale, is there another method better suited for this case?

If the data has a central tendency, then the standard would work fine, but what about when you have more of a halo effect, or some more exponential distribution?

For item-ratings type of data with the restriction that an item's rating should be between 1 and 10 after transformation, I would suggest using a simple re-scaling, such that the item's transformed rating $x_t$ is given by:

$$x_t = 9\left(\frac{x_i - x_{min}}{x_{max} - x_{min}}\right) + 1$$

where $x_{min}$ and $x_{max}$ are the minimum and maximum possible rating in the specific scale for the item, and $x_i$ is the item rating.

In the case of the above scaling, the transformation applied is independent of the data. However, in the normalization, the transformation applied is dependent on the data (through mean and standard deviation), and might change as more data becomes available.

Section 4.3 on page 30 of this document shows other ways of normalizing in which your restriction (transforming to the same absolute scale) might not be preserved.


When unsupervised learning is more beneficial in comparison with supervised learning even the labelings are existed? When unsupervised learning is more beneficial in comparison with supervised learning even the labeling are existed?
If there is no labeling the unsupervised learning is better than supervised learning but in some cases even the labeling targets are available, the supervised learning approach works betters? What about conditions of these cases? Can we say that if there is no clear dependency between variables unsupervised learning works better?

When unsupervised learning is more beneficial in comparison with supervised learning even the labeling are existed?

I would say that there are two main cases:

The task is semantically more meaningful as an unsupervised task. For instance let's consider a collection of books which have been annotated with topics: if the goal is to classify new books into the same pre-existing categories, then it makes sense to use a supervised setting. However if the goal is to discover new patterns of similarity that might not be intuitively easy to notice for a human annotator, then unsupervised topic modeling makes more sense.
The annotations are available for some subset of data (e.g. for evaluation purposes) but will not be available later in production (to some extent it's also a case of what the goal is, but here for technical reasons).

  If there is no labeling the unsupervised learning is better than supervised learning but in some cases even the labeling targets are available, the supervised learning approach works betters? What about conditions of these cases?

The problem here is what "better" means, i.e. how the task is evaluated. If the task is evaluated against the pre-existing labels, in theory the unsupervised version cannot work better than the supervised one since the supervised one has access to more information. It's possible that a particularly unsuitable supervised method would perform worse than a well chosen unsupervised one, but that's not a fair comparison and it's very unlikely in practice.

In general the two are not comparable because the tasks are fundamentally different: in a supervised setting one wants to find patterns related to some information which is known beforehand (the labels), whereas in an unsupervised setting one wants to discover unknown patterns.

  Can we say that if there is no clear dependency between variables unsupervised learning works better?

I don't think so because:

Some supervised algorithms are very good at optimizing whatever little amount of information is available in the data. Even with "no clear dependency" visible according to standard measures, some algorithms can combine the features optimally to minimize errors.
In the case of strictly no dependency at all between the features and the label, the unsupervised method might perform better in the sense that it might find meaningful patterns, but these won't be related to the labels at all (since there's no dependency). So we go back to the question of what "better" means with respect to the task: if the task was to discover unknown patterns then sure unsupervised is "better", but this has nothing to do with the labels and a supervised method would make no sense here. If the task was about predicting the labels, the unsupervised approach is just as bad as the supervised one (and there's a serious flaw in the design of the task!).


Do DBMS decrease Memory requirements? I finished my Economics thesis using RStudio, but my script was very slow due to massive RAM consumption during the process.

My Case

I had a massive dataset (stock prices in daily frequency for 10 years, ~700 stocks i.e. $3500\times700$) and I was picking each stock as a vector to decompose it into wavelets and CF filter (2 datasets $28000\times700$) and apply benford's law (two datasets $9\times700$).

The Problem

RStudio was storing my datasets in memory and they were consuming a significant proportion just by touching them.

Question

I started learning basic SQL commands and I found out that I can call specific columns from a certain table. Would my script be more efficient if I was calling my stocks one by one as vectors from there instead of picking them directly from RStudio? In other words, do queries call the whole dataset and then retrieve the requested values or do they follow a kind of shortcut to be memory efficient? If not, what's the purpose of using databases for domestic use?

Welcome to the site!

  Would my script be more efficient if I was calling my stocks one by
  one as vectors from there instead of picking them directly from
  RStudio?

Yes as you told, calling the specific columns from Database is better than extracting everything. I generally use Dataframes rather than vectors. Dataframes are very efficient and the transformation using Dataframes is much easier and better. You can go through this link, for better understanding on Vector Vs Dataframe.

Currently, I use 1 year data and consists of 100,000 records. Query takes like 8 minutes to extract the data in Rstudio using SQL DB and store it in an R-dataframe. After the extraction is over I don't hit the database meaning no read and write on the database by which the DB Server is not effected. After doing all the modeling and finally I commit the data into SQL DB to store the committed data(results).

  In other words, do queries call the whole dataset and then retrieve
  the requested values or do they follow a kind of shortcut to be memory
  efficient?

It depends on how you write the query, in my experience when I tried using temp-tables and is much efficient but when tried running that query in R but it wasn't supporting so had to stick to nested queries. If you don't write the nested queries in an optimized manner then it takes more time to extract the data(memory lekage).


Do zero weights receive zero gradient in ReLU neural networks? Suppose I have a deep neural network using the ReLU activation function, that is $\sigma(x) = max(x, 0)$. Suppose some weight $w_i$ becomes exactly $0$ at some point. Am I getting something wrong here, or is it the case that the gradient w.r.t. $w_i$ will be zero at all times and hence $w_i$ won't get any further updates? I feel like I am missing something here.

The derivative (of a loss function) with respect to $w_i$ can be any value, independent of the value of $w_i$.  (E.g. $f(x)=x$ has derivative 1 everywhere, even at $x=0$.)


Cross validation for anomaly detection using autoencoder I am using autoencoder for anomaly detection in warranty data. I don't have any ground truth labels to confirm whether the anomalies detected by the model is really an anomaly or not. Since I don't have the labels, I would like to know if there is a way to do cross validation in that case?

Any help is much appreciated

Autoencoders are used to reduce the dimensionality of the feature space. They can capture nonlinearities that other dimensionality reduction teqniques like PCA can not. Autoencoders are build by training the model to reproduce the input. In this case you can split the data set into three:

training
cross validation
testing

Train you model using the training set, check the performance by looking at your loss for the cross validation set, make some changes, and repeat. After you have a model you are confident in, then using the test data set as the final word in its performance.

I think the main point is that your loss should have something to do with the difference between an element of the data set, and the encoded-then-decoded version of that element. Therefore you cross validate against a separate chunk of data as you tune the hyper parameters of the encoder.


Activity recognition in smart homes with different sources I have a dataset collected in a smart home and I would like to do Activity Recognition.
I am used to classification but in this case data comes from 4 different sources:

Accelerometer from wrist
Binary sensors around the house
proximity beacons (to understand which room the user is at any moment)
ground pressure from floor

Data is labelled and has 25 activities.

At the moment I am working with a CNN on the activity images coming from the sensors and I can reach 45% accuracy, but I need to add the other sources to achieve better results.

My questions:

how to use all (or a group of) different sources for example in a NN?
Can I train 4 different classifier that works together to make a prediction?
Maybe some classifiers try to understand simpler activities and other try to find more complex activities?

Data-fusion is the most probable way to "fight" with all of the data sources. However, additionally, I would strongly suggest imposing an ontology and some set of rules.

Basically what you can do is the following:

-proximity beacons (to understand which room the user is at any moment) based on this sensor data you can introduce localization, decision trees may the most suitable way to go here while imposing some clear manually added rules(if the person is in this room, then goes into this room, etc). Basically, before doing the overall data fusion the localization may be really helpful in increasing the accuracy because you can introduce the ontology with specific activities that can be done in some of the rooms. (The person is not expected to be running in the kitchen or the bathroom for example).

ground pressure from the floor - this type of sensor can be really useful for localization as well.

Also, before data-fusion, the Accelerometer from wrist data readings should be carefully pre-processed. I would strongly suggest finding a nice way how to remove the bias(left handed/right-handed people) and how to remove the DC component(that is the overall gravity of the Earth gravitational field, remove this bias as well)


How deep should my neural network be? Are there any heuristics for deciding on where to start with the number of layers for a neural network? I built one with with 7 layers that has 40 input features and this took around 8 hours to train completely, but now I want to add a bunch of polynomial features which will increase the input size of my network to a few thousand features. Since there are so many features will my network benefit from having more layers? If so, what is a good starting point for the number of features?

Here is some information about my problem: suppose I have a set of usage data from a website. So if a user logs into the website one day their usage statistics during that day is calculated. I am using a few statistical tools to summarize the data and I want to make a prediction about whether or not the user will use the website in the next 60 days. My plan is to try and engineer all the possible monomial quadratic features from this summary and feed this information into a DNN. Without the feature engineering I have roughly 60 different features, but with it I will have at least an addition 3600 features.

Answer

  Are there any heuristics for deciding on where to start with the number of layers for a neural network?

No there isn't one in general. The architecture of the network (including depth) is considered to the new "feature engineering". It's more of an art than science at the moment. Anything above 3 layers worth considering.

  I want to add a bunch of polynomial features

You don't need to do this kind of feature engineering like you would in traditional machine learning. If you have a network that is sufficiently wide and deep, your network should be able to "engineer" those features itself. Just think about how the first hidden layer is a linear combination of input features. And the second hidden layers is a linear combination of the first hidden layer. It should be obvious to see how a neural network is capable of structuring any polynomial features itself.

Suggestion

Try using the original 60 features directly and start with 3 layers. Given the size of the network, the training should take less than 10 minutes on a GPU-enabled machine. Compare your result with your benchmark model (you should always have a benchmark model). Play around with the hidden size and layers, rinse and repeat.


Understanding ROCs in imbalanced data-sets A response variable (label) $B$ can either be $0$ or $1$.

In the training set, $B_i = 1$ is an extremely rare event at only $0.26\%$ occurrences. Which makes the prediction of this label on a test data-set a difficult problem.

I used SMOTE to sample from the training-set of some $1.55 \times 10^5$ rows to obtain a completely balanced set of $620$ rows.

balanced.df <- SMOTE(B ~ ., df, perc.over = 100, perc.under = 200)
randomForest with $1000$ trees was used to fit the model, as shown:

randomForest.Fit <- randomForest(B ~ ., data = balanced.df, ntree = 1000)
For making a validation set, $2000$ rows were sampled at random without repetition from the data-set.

The actual frequencies of $B_i$ in the validation set are:

   0    1
 1998   2

And those in the predicted set are:

   0    1
 1836  164

The results seem promising, but perhaps a little too much. Also, it is essential that the percentage of False Positives are reduced.

My questions are:

How severely do you think the skew in data is affecting the validation results?
Is there a point in validating again by creating an arbitrary data-set with bias, for example by selecting more $B_i = 1$ in the validation set?
What other metrics/validation techniques which reflect the "accuracy*" of prediction?

*The term accuracy is used in a generic sense.

So your data-set of 155000 records has 403 records where B=1, and B=0 for the remaining 154597 records.

You could try splitting your data-set into 2:1 training/test sets sampled by each class of B. After you've done this, then only for the training set use SMOTE to over-sample the records with B=1 along with under-sampling the B=0 training records to bring the class ratio to something like 4:1.

Over/under sampling for the test set is not required as it is supposed to mimic real world uncertainty to test your model's performance.

Your model's AUC will definitely get reduced since (as rightly pointed out by stmax), you've leaked test records into the training set by over sampling B=1 cases before splitting the train-test sets.

The answers to each of your questions are:

Yes, class imbalance does effect a random forest model's accuracy. How severely depends on the severity of the imbalance as well as the nature of the data-set itself.
Yes, you are biasing the training by over sampling the minority class, but if these are large enough samples in the original set, then hopefully they are close representations to the entire population.
I would recommend two metrics/techniques you could use here: Kappa Statistic (refer to this article) and the precision-recall curves to compare different models.


Can anyone verify my NN diagram if it is properly drawn? I am working on a Neural Network that can estimate building's carbon footprint based on the set of features and an image of urban surroundings (via CNN).

I have used Netron to visualize the network (up) but this image is not readable in a publication so I drawn one myself (down).

Can anyone comment if this is proper representation - especially the last part after the concatenation - I am not sure if there should be 2 or 3 dense layers?

I do not think there is a special kind of format that needs to be followed as long as the image is clear and readable, which (imho) it is for your case. Regarding the last 2/3 layers, the final layer is the output with 1 unit, so you pictured it correctly, along as the article mentions the output shape (that is not a multi-output situation).
Good luck with publication!


Algorithm for finding best juice combinations I am making a fun experiment in which a machine will mix different percentages of three juices – orange, apple and grape. After each mix is dispensed, a participant will taste the juice and rate it on a numeric scale, a score from 1 to 7.

Using the data collected, I would like to try and find the optimal juice mix programmatically.

For this I wish to implement a machine learning algorithm, that will both generate new ratios of juices to try, and using the response will try and find to the optimal juice mixture percentages.

What algorithm would you recommend me to use?

Notes:

I am planning on displaying this at an event, and estimate around 200 people to taste, at least.
I am aware that due to the fact that different people have different taste preferences, I aim for the best tasting mixture that will fit most people.

As described, you have no data describing individual people (such as age, sex, shoe size), but are searching for an  optimum value of the mix for the whole population. So what you want is a mix with the maximum expected rating, if you chose a random person to rate it from the population. In principle, this expected rating is a function taking two parameters e.g. $f(n_{apple}, n_{orange})$ - the amount of the third juice type is a not a free choice, so you only have two dimensions.

You can break down your problem into two distinct parts:

Taking samples from your population in order to find approximation to the function $f(n_{apple}, n_{orange})$
Using the approximation as it evolves to guide the search for an optimum value.

For a simple approach, you could ignore the second bullet point and just randomly sample different mixes throughout the event. Then train a regression ML on the ratings (any algorithm would do, although you'll probably want something nonlinear, otherwise you'll just predict one of the pure juices as favourite) - finally graph its predictions and find the maximum rating at the end. This would probably be fine when pitched as a fun experiment.

However, there is a more sophisticated approach that is well-studied and used to make decisions when you want to optimise an expected value of an action whilst exploring options - it is usually called multi-armed bandit. In your case, you would need variants of it that consider an "arm space" or parametric choice, as opposed to a finite number of choices that represent selecting between actions. This is important to you, since splitting your mix parameters up into e.g. in 5% steps, will give you too many options to explore given the number of samples you need to make. Instead, you will need to make an assumption that the expected rating function is relatively smooth - the expected rating for 35% Apple, 10% Orange, 55% Grape is correlated with the rating for  37% Apple, 9% Orange, 54% Grape . . . that seems at least reasonable to me, but you should make clear in any write-up that this is an assumption and/or find something published that supports it. If you make this assumption, you can then use a function approximator such as a neural network, a program like xgboost or maybe some Guassian kernels to predict expected rating from mix percentages.

In brief for a multi-armed bandit problem, you will use data collected as your experiment progresses to estimate the expected value for each choice, and on each step will make a new choice of mix. The choice itself will be guided by your current best approximation. However, you don't always sample the current top-rated value, you need to explore other mixes in order to refine your estimated function. You have choices here too - you could use $\epsilon$-greedy where e.g. 10% of the time you choose completely randomly to get other sample points. However, you might need something more sophisticated that explores more to start with and still converges quickly, such as Gibbs sampling.

One thing you don't say is at what level you are pitching this experiment. Studying the multi-armed bandit problem by yourself referring to blogs, tutorials and papers could be a bit too much work if this is for school science fair. If this all seems a bit too vague and a lot of work to study, then you can probably stick with a simple regression model from the data of a random experiment.

I suggest whichever approach you take, that you run some simulations of input data and see whether your approach works. Obviously there is a lot of guess work here. But the principle is:

Create a "true" model function - e.g. pick an imaginary favourite mix and make it score higher. Make it a simple and probably quite subtle function - e.g. score 5 for best result, and take away euclidean distance in "juice space" times a small factor (maybe 1.5) from it.
Create a noisy sampler that imitates someone in your experiment giving a rating to a specific mix. Ensure that the mean value from this matches the "true" function.
Try out your sampling and learning strategies, see how well they find the favourite mix.

I highly recommend this kind of dry run before putting your system to real use, otherwise you will have no confidence that your ML/approximator is working.

One more piece of advice about your estimator: You are expecting a large amount of variance in your data, and will not have a lot of samples. So to avoid over-fitting you will want to have a relatively simple ML model. For a neural network for example, you will probably want only one hidden layer with very few neurons in it (e.g. 4 or 5 might be enough). Finding a model sophisticated enough to predict a curve, but simple enough that it doesn't overfit given very noisy target outputs might take a few tries - this is the main reason why I suggest performing trial runs with simulated data.


Information Visualization: which parameters should a data scientist plot to best explain the work behind a Neural Network? I'm a beginner in the field and I wrote and trained my first Convolutional Neural Network from scratch. I would like to know what types of graphics and parameters a data scientist uses to plot to explain a neural network in the best possibile and mathematical way. As far as I know I use to plot loss and accuracy graphics using matplotlib.

There are different ways to explain your findings but associating the target with the inputs provided is tough and only a few architectures like Decision Trees do it well.

If you are trying to answer the question "why is my neural network missing out on a few samples from my validation set?" then its going to be a tough but possible road ahead. There are many good resources to interpretable machine learning

If you are trying to compare your model's (CNN) performance to other
architectures on the same data set, then looking at Receiver Operator Characteristics is a good point to start.

If you're having an unbalanced data set, then the F1 Score is a good thing to have look at. Being a classification problem (which is my assumption from your question), you can indicate the performance by a variety of derived quantities.


Eigen Decomposition of Data Matrix for PCA In PCA we Eigen decompose the covariance matrix, not data matrix, Is it because most data matrices are non-square. If they were, isn't is correct to eigen decompose data matrix than the covariance matrix?

Some Intro

Eigen-Decompostion does lots of things among them one thing is interesting. Eigen-Decomposition usually captures the characteristic of the matrix it is applied to. For instance is the matrix is the matrix of similarities, the result of eigen decomposition will be a clustering (see Spectral Clustering).

PCA

The idea behind PCA is to find coordinates on which the intrinsic variance of data is maximized. Well ... according to Intro, if I have a matrix encoding joint variance (co-variance) of all pairs of variables, then I can use eigen decomposition to simply capture this property for me! This is what PCA does.


How to correctly infer vectors in Gensim doc2vec? I would like to know which is the correct procedure for inferring vectors in Gensim doc2vec.

I have a dataframe df with a feature, called name, and composed of two subsets train and test.

df = train + test

My aim is to find the most similar name in train given a name in test.
For doing this I have to train the doc2vec model, and I have two possible choices:

train the model on the entire df and then infer the most similar name by model.infer_vector() on test.
train the model on train, letting out test, and then use model.infer_vector() on test.

I suppose that the correct procedure is first one, but I am not sure.

Also, so doing, there is the possibility that the most similar name  given test will be again in test and not in train.

I would use the first approach, given that both train and test are known, there is no need of generalization i.e. you don't expect unseen vectors.

In order to avoid the problem you mentioned, you have to find the most similar vector to a vector in test considering  only vectors in train. For example:

train = [v1, v2, v3]
test = [v4, v5]
most_similar = {}
for vector in test:
most_similar[vector] = v1
  for vector2 in train:
   if similarity(vector2, vector) > similarity(most_similar[vector], vector):
     most_similar[vector] = vector2

In the end in most_similar you have the most similar vector in train for each vector in test.


Approaches to choosing number of bins in histogram Choosing the number of bins in a histogram has always been something that gets me thinking a lot. Based on the number of bins chosen, the graph at time looks a lot different and also could lead to different interpretations.

Below is the square-root rule, which I use as the thumb-rule for selecting the number of bins in most occasions.

Posting this question here to hear other opinions.

data_pts = len(np.array(data))
bin_cnt = int(np.sqrt(data_pts))

plt.hist(data, bins=bin_cnt)

Another method is Bayesian Blocks from Studies in Astronomical Time Series Analysis. VI. Bayesian Block Representations by Scargle et al.

Bayesian Blocks is a dynamic histogramming method which optimizes one
of several possible fitness functions to determine an optimal binning
for data, where the bins are not necessarily uniform width.

Bayesian Blocks for Histograms


Suspiciously good accuracy using neural network I have a dataset from EEG data that is 24 features (24 electrodes) and 88000 samples with 3 classes, it is normalised and everything and had some noise filtered out via bandpassing.
When I classify with anything but a neural network the accuracy is pretty bad and I am using a 60/40 for training/test set just to make sure I can trust the result.
For example:

Gaussian naive bayes: 42%
Logistic Regression: 52%
Linear Discriminant Analysis: 51%

However I played around with a neural network achieving 95%+ averaged with:

3 hidden layers: 100, 200, 100
Activation: relu
Learning rate: adaptive

I think this is super fishy so I did a PCA analysis

And plotted it with dimension reduction to two dimensions

As you can see, there is nothing significantly separable, which really confuses me.
I am definitely using the test set to run the cross validation which is 40% of the sample data.
Can someone please advise on what's happening and whether I can trust this result? And what further steps I can do to make this result more concrete?
I don't want to celebrate too early!!!

As you can see, there is nothing significantly separable, which really confuses me. I am definitely using the test set to run the cross validation which is 40% of the sample data.

Well, your interpretation is wrong: there is nothing  linearly separable on the two first directions.
The other components in PCA may be more relevant. Remember that PCA gives you the direction where the data has most variance, which doesn't necessarily mean it is the direction in which it is most easily separated/classified.
I see you tryed LDA, which follows on that assumption that the most variable direction is not the most significant for classification, but since it is linear we can conclude that your data is not Linearly separable.
Your model seem pretty large, it is considerably more powerful than LDA, LR, etc.
If you are not confident with the results, try using other validation methods such as K-FOLD or Leave-one-out.


Statistical test to establish that event B is caused by event A I want to have a statistical test to establish that event B is caused by event A
Here
event B is customers that are supposed to leaved leave for competitor stay in our company
event A is an attractive offer for some customers (some may leave, and some may stay)
If we know that some of our customers is going to go for the competitor, and we come up with an attractive offer for them to stay.
If in the end, we notice that there were fewer customers that are leaving for competitors (event B), is there a statistical test that helps to establish a relationship that it is the attractive offer (event A) that retains those customers that will otherwise leave (event B)?

This would be done with a controlled experiment, similar to clinical tests:

Randomly assign customers into to groups 1 and 2. Group 1 is given the attractive offer, group 2 is not (control group).
It's important to avoid any external bias, so for example ideally nobody in the company should know whether a customer is in group 1 or group 2. The goal is to make sure that the only difference between the two groups is the offer.

After a pre-determined duration (e.g. 3 months), collect the results (how many customers in group X have left/stayed). Then a simple Chi-square test can be performed on the 2x2 contingency table in order to assess whether there's a significantly different rate of leaving between the two groups. Since the offer is the only possible explanation, this proves that the offer causes the difference.


NLP for appartement ads I want to analyse the Price situation for flat/appartements in my City.

How can I process the Text from flat ads (like in newspapers or online with its typical structure!) to get Features like size or number of rooms?

Is it simple to take a Package in R or Python (what is suitable?) and to Train the typical structure?

Look for entity recognition. The entities in your case would be price, number of bedrooms, size etcetera. What I would do is label a number of training examples with:

From character 12 to character 22 indicates the number of bedrooms
From character 39 to character 41 indicates the price

Etcetera.

After this you can train a model to classify entities. Sometimes the number of bedrooms will be in text instead of numbers so then you would need to write something that translates it to numerical values but you could likely just use some dictionary to do this for you.

Entity recognition models are plenty, but I think a character based LSTM approach could work well if your texts are not too long.


How to predict orders with a range of items? And total orders which sum up to the total? So I do have data like this:

With the help of distinct order IDs, I can figure out how many orders are there and from units shipped, I can get the number of items in the order. Now I want to predict future orders having x items for different ranges (e.g. 1-10 items, 10-20 items, 20+ items). Like tomorrow there might be 6 orders one of them will have 1-10 items, 4 will have 10-20 items and one will have 20+ items. The problem is I can predict them (using LSTM for now) separately using data for only those kind of orders but they obviously do not sum up to the amount predicted by model for totals. So how can I link them up so that total sums up to the values from the individual models?
Approach I am currently using:
I just predict orders for each item range and then manually sum them up. This way they do sum up but error accumulates

You should munge the data until you have a target value that aligns with the goal of the project. Then fit a model.
It sounds like you want the target to be a binned number of items {(1-10), (10-20), (20+)}. Each row should be unique Order_ID with one of those three target values.


Testing for gender composition between groups I did kmeans with k=4 and I would like to find out if there are any differences in the composition of gender between the 4 clusters.

Do I use Fisher's Exact Test of Independence? Is there any recommended library in r that can handle this as well as post-hoc tests between groups?

For analysis of your clusters you can use the silhouette coefficient or silhouette width. These are available in cluster and factoextra package in R.

I will explain what basically in silhouette analysis:

The silhouette coefficient is calculated as follows:

1) For each observation i, it calculates the average dissimilarity between i and all the other points within the same cluster which i belongs. Let’s call this average dissimilarity “Di”.

2) Now we do the same dissimilarity calculation between i and all the other clusters and get the lowest value among them. That is, we find the dissimilarity between i and the cluster that is closest to i right after its own cluster. Let’s call that value “Ci”

3) The silhouette (Si) width is the difference between Ci and Di (Ci — Di) divided by the greatest of those two values (max(Di, Ci)).

Si = (Ci — Di) / max(Di, Ci)

So, the interpretation of the silhouette width is the following:

Si > 0 means that the observation is well clustered. The closest it is to 1, the best it is clustered.
Si < 0 means that the observation was placed in the wrong cluster.
Si = 0 means that the observation is between two clusters.

Basic code to calculate the above and visualise:

library(cluster)
library(factoextra)

sil <- silhouette(clustering$cluster, dist(input))
fviz_silhouette(sil)


Machine learning algorithms for Geometrical objects shapes identification Are there Machine learning algorithms which will take input dataset of all geometrical objects shapes as images in gif,jpg,tiff formats & output the geometrical shapes names?
i.e. Geometrical Objects pattern identification.
Examples : Square, Rectangle, Kite, Circle, Sphere, Cone, Rhombus, Ellipse, Cylinder, Pentagon etc
These are examples of potential inputs with corresponding labels

This is quite doable.
Just make sure of the following:

Any test data should be one of the shapes you have trained on.
Make sure you have balanced data. (You can introduce bias if required)
To make it more challenging you can create or find data like google street view images for numbers.
Segmentation and Classification, both are possible.

VGG should do just fine, you could also use resnet. The overall model can be fit into an inventory pipeline where you sort the inventory through shapes!


Is there wights of voice or audio for VGG or Inception? I want to use VGG16 (or VGG19) for voice clustering task.
I read some articles which suggest to use VGG (16 or 19) in order to build the embedding vector for the clustering algorithm.
The process is to convert the wav file into mfcc or plot (Amp vs Time) and use this as input to VGG model.
I tried it out with VGG19 (and weights='imagenet').
I got bad results, and I assumed it because I'm using VGG with wrong weights (weights of images (imagenet))

So:

Are there any audio/voice per-trained weights for VGG ?
If not, are there other per-trained audio /voice models ?

As far as I know, VVGish is the VGG adapted to audio processing. I can remember using it with mfcc, not Amp-Time input tho.


Neural Networks - Loss and Accuracy correlation I'm a bit confused by the coexistence of Loss and Accuracy metrics in Neural Networks. Both are supposed to render the "exactness" of the comparison of $y$ and $\hat{y}$, aren't they? So isn't the application of the two redundant in the training epochs? Moreover, why aren't they correlate?

Log loss has the nice property that it is a differentiable function. Accuracy might be more important and is definitely more interpretable but is not directly usable in the training of the network due to the backpropagation algorithm that requires the loss function to be differentiable. When your preferred loss is not directly optimizable (like the accuracy) you use a loss function that behaves similarly to proxy the true metric. In case of binary classification you would use a sigmoid at the end and a log loss to approximate accuracy. They are highly correlated.


Difference between Convolution and Pooling? We uses filters in convolution that make feature map with reduced size(maybe), so does pooling. Then how are they different?

We use filters mostly to get different features(characteristics) about the object(e.g. image). And pooling we're using to reduce the size and at the same time to save the most significant information of each feature map.


GridSearchCV with custom tune grid What is the best way to perform custom parameter search CV with the Scikit-learn API? I really like GridSearchCV. However for my case the  param_grid parameter is inflexible because it will search over the entire span of parameter combinations. Ideally, I would like to provide my own parameter space in a dataframe, one column for each parameter.
Thus, is there an appropriate class within Scikit-learn to help me achieve this result?

GridSearchCV does allow param_grid to be a list of grid-dicts, which sometimes is sufficient.  In this case, separate grids are generated and their union is searched.
There isn't quite a convenient implementation by which you provide your own list of hyperparameter points.  But looking at the source code for GridSearchCV, you'll notice that it's amazingly sparse.  Most everything is handled by the inherited class BaseSearchCV.  The grid-specific stuff is also factored out:
    def _run_search(self, evaluate_candidates):
        """Search all candidates in param_grid"""
        evaluate_candidates(ParameterGrid(self.param_grid))

The ParameterGrid class is what's responsible for turning the (possibly list of) dict(s) into a grid(s).
So you could probably easily write a custom class, inheriting from BaseSearchCV, where you pass the set of hyperparameter points you want and define _run_search to just evaluate each element from your set.  In fact, already evaluate_candidates is defined assuming that the hyperparameter points live in an iterable candidate_params; the ParameterGrid does all the hard work there, but in your case you can just feed your list directly.


How to find feature importance with multiple XGBoost models My problem statement  : Time Series forecasting(Month wise data), training on 96 months of data and predicting next 12 months with a 3 months empty window in between.

Example :

Batch 1

***Training Data index*** <2010-01-01 -----------------------------2017-12-01>

***Unused Month Window*** <2018-01-01---2018-03-01>
***Test Month*** <2018-04-01> [Trained model with Batch 1 training data
                               can ONLY be used for predicting this month,
                               not any other]

Batch 2

***Training Data index*** <2010-01-01 -----------------------------2018-01-01>

***Unused Month Window*** <2018-02-01---2018-04-01>
***Test Month*** <2018-05-01> [Trained model with Batch 1 training data
                               can ONLY be used for predicting this month,
                               not any other]

and so on till Batch 12...

I am training 12 XGBoost models to get predictions for each of the 12 months of FY 18, hence getting  12 different feature importances against each model for the predictors used. But i want to report the feature importance of entire FY 18, instead of giving 12 different set of feature importances against each month. How would I approach that??

Evaluating a single model on the entire test dataset is not an option.

Any help is appreciated. Thanks.

I suspect you may be confusing the terms "test set" and "training set". Normally, the model is trained on the training set, and evaluated on the test set. The feature importance is independent of the test set, it's a property of the model that you trained.

The obvious answer is to combine all months into a single training dataset, and train a model on that, but you say it’s not an option.

Then I think your only option is to combine the results from individual months, for example by summing or averaging them and presenting the final ranking. Whether this is valid depends on the criteria used to train the trees (how the splits are determined). If there is a valid way to combine them, I still recommend against it, because I suspect it will obscure reality and discussion over the methods may distract from the result.

So I would sidestep the problem by presenting a graph like is sometimes shown for sports leagues or polls before elections. On the horizontal axis are the months, on the vertical axis the rankings. This way you visualize all info easy to grasp and you also show any trends and seasonal effects.


K-means vs. online K-means K-means is a well known algorithm for clustering, but there is also an online variation of such algorithm (online K-means). What are the pros and cons of these approaches, and when should each be preferred?

Online k-means (more commonly known as sequential k-means) and traditional k-means are very similar.  The difference is that online k-means allows you to update the model as new data is received.

Online k-means should be used when you expect the data to be received one by one (or maybe in chunks).  This allows you to update your model as you get more information about it.  The drawback of this method is that it is dependent on the order in which the data is received (ref).


When visualizing data that has <1 or <5 ppm how do you display this? I have some data (parts per million) where some of it is 1 or greater than one (but has an actual number.) However, some of the data simply lists "<1" ppm.

What is a good way to visualize this in a graph?

Should I pick an arbitrary decimal less than one, so that it can be shown on the graph? I feel setting it to zero would not be right either, as they went to the extent of listing it, so it must be significant.

Is it a univariate plot?  If so, just bin the data and toss it into the lowest bin.


Testable hypotheses construction; minimum predictive strength vs. significance Is this null hypothesis TESTABLE?

Research Question: "Can a predictive model utilizing logistic regression be built which predicts at least one customer will churn in 90 days, and this individual prediction will be at a minimum of 70% confidence, using a chosen set of independent variables?"
Null Hypothesis: "A predictive model utilizing logistic regression cannot  predicts at least one customer will churn in 90 days, with this individual prediction being at a minimum of 70% confidence, using a chosen set of independent variables."

If so, how do you test the "70% confidence" of the particular churn event that allows you to reject the null hypothesis, at the p < 0.05 level?  I'm not sure the confidence of an individual prediction can be done, but that may depend on the modeling technique.  It may be that you drop the p < 0.05, and just use the validity of the full model (R-square and Pearson chi-square with logistic regression).

Or, is this the only way to write the null hypothesis, and the strength of the model must be left to the discussion section?

"There is no significant relationship between the full model using the chosen independent variables and customer churn."  (With r-square and chi-square showing the significance of the full model.)

Good question.  By convention, the null hypothesis is always framed to say there is no statistical relationship between one or more independent variables and a dependent variable.  This is typically tested at the p < 0.05 level.  The level of the relationship beyond it being significant is not addressed in a null hypotheses.  (See, for example, Encyclopedia of Research Design, edited by N.J. Salkind, 2010).  As for testing the "70% confidence" result, there is no way to do that.  That is the statistical result.  The overall validity of a logistic regression model can be assessed using Pearson's chi-squared, R-square, and classification accuracy.


inputs and outputs of a fully connected layer for a note classifier i'm trying to make an object classifier of different musical note types and I'm having a problem understanding fully connected layers. for example I have a 3d array of binarized images (WxLxNo.OfImages) and i pass that array to the conv layer, relu and pooling. a lot of the videos and other tutorials said that I make a vector using the images but is the input per image, category or the whole training data? My second question is how many neuron should be on the last layer? should it be the number of categories in my training dataset?

Starting with your last question: The final fully-connected layer shoud output the number of target classes you have. This output can then be passed to a softmax, which normalises the values between the range [0, 1] - allowing them to be interpreted as probabilities.

Remember, you train using the following dimensions:

[batch_size, height, width, channels]        # channels = 1 (in your case) so is removed

You always need to maintain the batch_size through one loop forwards and backwards through your neuiral network. Normally it set to None which allows the model to accept any batch size you decide on.

What people mean by "make a vector" is that the output of your convolutional layers will be a 3D (including channels) and so 2D in your case. You need to flatten these into a single vector. The conv layer output, which then flows into the fully-connected layers will then be:

[batch_size, height * width]         # height and width are those from the final conv layer

Depending on your framework, there will be a layer called Flatten or view etc., which gives you the vector for each image, keeping the batch_size. This is performed as part of the model structure and so is applied to all data as it passes through the model (training, validation and test data).


Is my understanding of On-Policy and Off-Policy TD algorithms correct? After reading several questions here and browsing some pages on the topic, here is my understanding of the key difference between Q-learning (as an example of off-policy) and SARSA (as an example of on-policy) methods. Please correct me if I am misled.

1) With an on-policy algorithm we use the current policy (a regression model with weights W, and ε-greedy selection) to generate the next state's Q.

2) With an off-policy algorithm we use a greedy version of the current policy to generate the next state's Q.

3) If an exploration constant ε is set to 0, then the off-policy method becomes on-policy, since Q is derived using the same greedy policy.

4) However, on-policy method uses one sample to update the policy, and this sample comes from on-line world exploration since we need to know exactly which actions the policy generates in current and next states. While off-policy method may use experience replay of past trajectories (generated by different policies) to use a distribution of inputs and outputs to the policy model.

Source: https://courses.engr.illinois.edu/cs440/fa2009/lectures/Lect24.pdf

One more reading: http://mi.eng.cam.ac.uk/~mg436/LectureSlides/MLSALT7/L3.pdf

1) With an on-policy algorithm we use the current policy (a regression model with weights W, and ε-greedy selection) to generate the next state's Q.

Yes. To avoid confusion, it may be better to use the terms "behaviour policy" for the the policy that controls current actions and "target policy" for the policy being evaluated and/or learned.

  2) With an off-policy algorithm we use a greedy version of the current policy to generate the next state's Q.

Sort of. The only requirement for an algorithm to be off-policy is that the target policy is different to the behaviour policy. The usual target policy in Q-learning is not necessarily a greedy version of the behaviour policy, but is the maximising policy over Q. However, if the behaviour policy is ε-greedy over Q, and adapting to updates in Q, then yes your statement holds.

  3) If an exploration constant ε is set to 0, then the off-policy method becomes on-policy, since Q is derived using the same greedy policy.

This is true when comparing SARSA with Q learning, but may not hold when looking at other algorithms. This greedy-only action selection would not be a very efficient learner in all environments.

  4) However, on-policy method uses one sample to update the policy, and this sample comes from on-line world exploration since we need to know exactly which actions the policy generates in current and next states. While off-policy method may use experience replay of past trajectories (generated by different policies) to use a distribution of inputs and outputs to the policy model.

Experience replay is not directly related to on-policy vs off-policy learning. Technically though, yes when the experience is stored and used later, that makes it off-policy for SARSA if Q values have changed enough between the sample and current parameters of the learning agent.

However, you will see experience replay used more often with off-policy methods, since off-policy learners that boot-strap (i.e. use Q value of next state/action to help estimate current Q value) are less stable when used with function approximators. Experience replay helps to address that problem.


How to detect when the "bibliography" of a paper has began? I'm working on project that involves reading papers from ArXiV to look for particular patterns (It gets complicated but it basically has to do with common phrases and expressions) in their text.

My process can be abstractly described as:

$$ PDF \rightarrow "\text{PDF as a string}" \rightarrow \text{Pattern_Finding_ Method}("\text{PDF as a string}") $$

Now unfortunately, the "bibliography" or references section of many papers cause the Pattern_Finding_Method to break, because they just happen to carry a lot of the behavior we are looking for (but they are not of interest).

As a human it's quite easy for me just manually read and see where the "references" begin and then I can trim that part of the pdf string but at scale this is not practical. Moreover paper writers don't have consistent ways of declaring when their "references" "bibliography" or "acknowledgements" begin.

So it seems natural to view this as an ML/AI problem, where I have a string, I have a loose notion of what constitute the "references" of the string which I can provide training data for (I have the pdf as a string and I can list a character index on the string the references begin)

Now given training data I need to come up with some kind of model that can effectively learn how to detect references on its own.

This is where I get stuck. The data problem i'm dealing with is a highly semantic problem (it's particular organizations of words and their underlying meaning and patterns to how those words are displayed that is giving me cues as to when the reference have began), but my knowledge of learning algorithms is limited to mostly geometric data (SVMs), or at least highly continuous data (Neural Network Models), and then in the case of NLP, my understanding is at best a recipe book of very goal specific algorithms: (ex: TF-IDF for document classification).

I don't know how to bridge the gap from my understanding to creating a specialized model for the problem at hand; that I have intuitive reason to believe will work.

Formal Problem Statement:

Given a collection of large strings, each attached with an integer $i$ indicating where the references begin, determine a model that can reliably detect when references begin on new text.

Here are a few heuristics that could help solve the problem without ML:

If all the papers are from the Arxiv, you can download the source tex files and regenerate the paper without the bibliography
You can use Arxiv-Vanity or their code to render the document into HTML (or some intermediate representation), then use that hierarchical structure to easily remove the part with a bibliography
Look for the word "references" followed by high word density of the words "in", "pages" and years in the 1900s and early 2000s

I guess what I am saying is "the input already has a fairly rigid structure, why don't you use that instead of statistics?"


Feature selection or Dimension reduction in unsupervised learning I'm trying to do Embedded clustering using kmeans.

This is customer data, so it involves a lot of sentences, so I'm using the universal sentence encoder before clustering.

But I should be doing a feature selection or dimensionality reduction before embedding the features.

I want to know if there is a method to do feature selection or dimensionality reduction in unsupervised learning. This could be very helpful as the clustering is giving a mixed result as of now and I have a strong feeling that this could be because of the unwanted attribute in the data.

I have read all the resources which only gave options to be done on the supervised learning.

Any help is appreciated!

Thanks
Arav

I found Auto-encoders to be the best solution for this.
Performing auto encoder before clustering reduces the dimensionality of the high dimensional data and then the encoder results can be extracted and used for whatever method we want to implement like,

supervised classification
unsupervised clustering etc.

Autoencoders using CNN (encoder and decoder) for image:
https://www.datacamp.com/community/tutorials/autoencoder-keras-tutorial#comment-6125

Autoencoders - its types and usage - LSTM:
https://machinelearningmastery.com/lstm-autoencoders/

for my usecase I should be performing an embedding before feeding it into the autoencoder (CNN/LSTM) with only the encoder so that the reduced dimensions can be used for kmeans clustering.

Hope this helps people who have same question, I wonder how none has come across this situation or haven't replied for this.

Thanks
Arav


How to handle out of scope value? I have a data set in which the score column has to be between 0 to 100 and the subject column has to be one of ['Math','Science','English']. However, my data set has different values for some rows.

How should I handle those rows?

  subject score ...
1 Math    90    ...
2 Science 85    ...
3 English 105   ...
4 Comp    95    ...
5 Math    80    ...
6 Science 70    ...

If you need to clean your data, you can either drop the rows which contain invalid values, or try to correct them.

Here two examples:

# If you want to change the score, so values below 0
# are changed to 0 and values above 100 are changed
# to 100 you can do that like this:

df['score']= df['score'].clip(0, 100)

# Or alternatively (in case you have more complicated
# operations, you can also use where. For the
# correction of the scores, this would look like

df['score']= df['score'].where(df['score']<=100, 100)
df['score']= df['score'].where(df['score']>0, 0)

# If you want to drop the rows that contain undefined
# subjects, you can do that as follows:

valid_subjects= ['Math','Science','English']
# define an indexer that contains True for all rows which are invalid
invalid_subj_indexer= ~df['subject'].isin(valid_subjects)
# now drop them
df.drop(invalid_subj_indexer.index[invalid_subj_indexer], inplace=True)

The result of this looks like:

   subject  score  ...
1     Math     90  ...
2  Science     85  ...
3  English    100  ...
5     Math     80  ...
6  Science     70  ...

You can test the lines above by executing the following lines first to create the test dataframe:

import io
import pandas as pd

raw=\
"""  subject score ...
1 Math    90    ...
2 Science 85    ...
3 English 105   ...
4 Comp    95    ...
5 Math    80    ...
6 Science 70    ..."""

df= pd.read_csv(io.StringIO(raw), sep='\s+')


Is my model overfitting? I'm currently building my first model with sklearn to predict whether a customer will renew a subscription.  I'm using a random forest because I've heard that they are robust to overfitting.  The confusion matrix generated on the results of my test set (25%) seem to imply that the model is working very well, but the tree I'm generating is massive.  The size of the tree alone, especially considering I'm only using 12 variables, makes it seem like the model is creating too many complex relationships and thus must be overfitting.  If the model is overfitting how can I adjust my code to fix this?  Here is the code I used to tune hyper parameters and fit the model.
pipe = Pipeline([('classifier', RandomForestClassifier())])

param_grid = [
{'classifier' : [RandomForestClassifier()],
'classifier__n_estimators' : list(range(200,500,10)),
'classifier__max_features' : list(range(6,32,5))
}
]

clf = GridSearchCV(pipe, param_grid = param_grid, cv = 5, verbose=True, n_jobs=-1)

predictions = clf.best_estimator_.predict(x_test)
cm = metrics.confusion_matrix(y_test, predictions)
plt.figure(figsize=(9,9))

sns.heatmap(cm, annot=True, fmt=".3f", linewidths=.5, square = True, cmap = 'Blues_r');
plt.ylabel('Actual Values')
plt.xlabel('Predicted Values')
plt.title('Renewal Model Confusion Matrix')

Here is the confusion matrix generated, as well as an image of the tree I'm generating and the code used to generate it.
from sklearn.tree import export_graphviz
import graphviz
import os
os.environ["PATH"] += os.pathsep + 'C:/Users/rich/Anaconda3/envs/32bitpy/Library/bin/graphviz/'

d = export_graphviz(clf.best_estimator_[0][1], out_file='rf.dot',
            rounded = True, proportion = False,
            precision = 2, filled = True)

os.environ["PATH"] += os.pathsep + 'C:/GraphViz/bin'

graph = graphviz.Source(d)
graph

If your model is overfitting then you can use Dropout, L1 & L2 Regularization. It helps to improve your model's accuracy.


What are true positive, false positive, true negative, false negative in the given example? Assume there are 100 images, 30 of them depict a cat, the rest do not.
A machine learning model predicts the occurrence of a cat in 25 of 30 cat images. It also predicts absence of a cat in 50 of the 70 no cat images.

In this case, what are the true positive, false positive, true negative and false negative?

Assuming cat as a positive class,

Confusion Matrix:

  TN | FP
     FN | TP

True Positive(TP): Images which are cat and actually predicted cat
i.e. 25
True Negative(TN): Images which are not-cat and actually predicted not-cat
i.e. 50
False Positive(FP): Images which are not-cat and actually predicted as cat
i.e. 20
False Negative(FN): Images which are cat and actually predicted as not-cat
i.e. 5

  Precision:  TP/(TP+FP)
  Recall:  TP/(TP+FN)

Precision: 25/(25+20) = 0.55,
Recall: 25/(25+5) = 0.833


TensorFlow stopped working after installing opencv I am working on projects that use TensorFlow and Keras and I did the hello-world MNIST project successfully and later I tried to run some other project and TF suddenly stopped working. Now it doesn't even work on the old project. I am working in conda envs and my editor is VSC. Here is the error that I am getting:
Traceback (most recent call last):
  File "c:/Users/miliv/MLRadionica/TensProba.py", line 3, in <module>
    from tensorflow import nn
  File "C:\Users\miliv\Miniconda3\envs\tf\lib\site-packages\tensorflow\__init__.py", line 101, in <module>
    from tensorflow_core import *
  File "C:\Users\miliv\Miniconda3\envs\tf\lib\site-packages\tensorflow_core\__init__.py", line 40, in <module>
    from tensorflow.python.tools import module_util as _module_util
ModuleNotFoundError: No module named 'tensorflow.python.tools'; 'tensorflow.python' is not a package

I tried reinstalling TensorFlow a couple of times but it didn't help. The only thing that comes to my mind is that it has to do something with installing OpenCV maybe because of some packages, but I haven't changed anything else.
I have tried creating a new conda env with TF and without OpenCV but it still throws me the same error.

The first and foremost thing we needs to aware is , from installation perspective is there any micro level relationship between tensorflow and openCV , so that it could help us out in troubleshooting the above issue.
But yes    this link will help you out in accomplishing your tensorflow/openCV setup installation


Disadvantages of hyperparameter tuning on a random sample of dataset I often work with very large datasets where it would be impractical to check all relevant combinations of hyperparameters when constructing a machine learning model. I'm considering randomly sampling my dataset and then performing hyperparameter tuning using the sample. Then, I would train/test the model using the full dataset with the chosen hyperparameters.

What are the disadvantages of this approach?

One of the good practices is to create a split in the dataset for each tuning/ training step of your pipeline. Since you have large datasets, you should have enough data to split the original dataset into multiple subsets and still have a relevant number of rows for each step. As such, as an example, you can divide your dataset in 60% training, 20% hyperparameter tuning and 20% for the test.

It is important to avoid optimizing the hyperparameters with the same data you train on because this can lead to overfitting both tuning steps of your model to the same source of data.

Also, be careful on how you sample the original datasource. When dealing with highly skewed categorical features, random sampling can lead to categories in the test set which are not observed during training, which can cause some models to break. Also, numerical features should have a similar distribution between the training and the test set.


Can machine learning learn a function like finding maximum from a list? I have an input which is a list and the output is the maximum of the elements of the input-list.

Can machine learning learn such a function which always selects the maximum of the input-elements present in the input?

This might seem as a pretty basic question but it might give me an understanding of what machine learning can do in general. Thanks!

Maybe, but note that this is one of those cases where machine learning is not the answer. There is a tendency to try and shoehorn machine learning into cases where really, bog standard rules-based solutions are faster, simpler and just generally the right choice :P

  Just because you can, doesn't mean you should

Edit: I originally wrote this as "Yes, but note that..." but then started to doubt myself, having never seen it done. I tried it out this afternoon and it's certainly doable:

import numpy as np
from keras.models import Model
from keras.layers import Input, Dense, Dropout
from keras.utils import to_categorical
from sklearn.model_selection import train_test_split
from keras.callbacks import EarlyStopping

# Create an input array of 50,000 samples of 20 random numbers each
x = np.random.randint(0, 100, size=(50000, 20))

# And a one-hot encoded target denoting the index of the maximum of the inputs
y = to_categorical(np.argmax(x, axis=1), num_classes=20)

# Split into training and testing datasets
x_train, x_test, y_train, y_test = train_test_split(x, y)

# Build a network, probaly needlessly complicated since it needs a lot of dropout to
# perform even reasonably well.

i = Input(shape=(20, ))
a = Dense(1024, activation='relu')(i)
b = Dense(512, activation='relu')(a)
ba = Dropout(0.3)(b)
c = Dense(256, activation='relu')(ba)
d = Dense(128, activation='relu')(c)
o = Dense(20, activation='softmax')(d)

model = Model(inputs=i, outputs=o)

es = EarlyStopping(monitor='val_loss', patience=3)

model.compile(optimizer='adam', loss='categorical_crossentropy')

model.fit(x_train, y_train, epochs=15, batch_size=8, validation_data=[x_test, y_test], callbacks=[es])

print(np.where(np.argmax(model.predict(x_test), axis=1) == np.argmax(y_test, axis=1), 1, 0).mean())

Output is 0.74576, so it's correctly finding the max 74.5% of the time. I have no doubt that that could be improved, but as I say this is not a usecase I would recommend for ML.

EDIT 2: Actually I re-ran this this morning using sklearn's RandomForestClassifier and it performed significantly better:

# instantiation of the arrays is identical

rfc = RandomForestClassifier(n_estimators=1000, verbose=1)
rfc.fit(x_train, y_train)

yhat_proba = rfc.predict_proba(x_test)

# We have some annoying transformations to do because this .predict_proba() call returns the data in a weird format of shape (20, 12500, 2).

for i in range(len(yhat_proba)):
    yhat_proba[i] = yhat_proba[i][:, 1]

pyhat = np.reshape(np.ravel(yhat_proba), (12500,20), order='F')

print(np.where(np.argmax(pyhat, axis=1) == np.argmax(y_test, axis=1), 1, 0).mean())

And the score here is 94.4% of samples with the max correctly identified, which is pretty good indeed.


Random Forests Feature Selection on Time Series Data I have a dataset with N features, each one with 500 instances in time.

For example, let's say that I have the following:
Features x, y, v_x, v_y, a_x, a_y, j_x, j_y,
A sample with 500 examples (rows in a table) for each feature,
A sample with 500 other instances, and a class.

I'd like to select a subset of the features automatically with the Random Forests algorithm. The problem is that the algorithm (I'm using ScikitLearn, RandomForestClassifier), accepts a matrix (2D array) as X input, of size [N_samples, N_features]. If I give the array as it is, that is a vector (len 500) for the feature x, another (len 500) for the feature y, etc., I get a N_samples x N_features x 500 array, which is incompatible with the requirements of RandomForestClassifier.

I tried to unroll the matrix in a vector, like having so 500 x N_features array, but in that way, in the reduction, it considers all the elements independent feature, and breaks my structure.

How can I select/reduce the features keeping the time instances consistent?
(I can use this algorithm, but i'm also open to other libraries and/or algorithms)

My goal is to do classification, so forecasting resources are limitedly useful to me. Also I have the requirement that each sample has those occurrences, and I don't have them as separate samples unfortunately.

Some EDA might be needed to create new features for each time-series item. You might want to mine for patterns and have random forest reduce the overfitting. Exactly how mining is done depends on the nature of the problem, which might indicate for things like:

interesting time periods,
events that happen at a time,
time lag between different series,
dynamical systems,
latent variables,
scedasticity

Breiman's landmark paper on random forest gives some theoretical guarantees that random forest works well when individual classifiers are good and the correlation between these individuals are low. This can also be a heuristic to prune features.


Calibrate the predicted class probability to make it represent a true probability? Let's say that we have a simple binary classification model (a neural network -- NN) for classifying input images as "dog" ($y=1$) or "not dog" ($y=0$). Let's assume that the NN has one "sigmoid output" $\hat{y}$. After training, we feed an image of a dog and the output of the NN is e.g. $\hat{y} = 0.8$. As I understand it, this value of "0.8" is the probability ("confidence") that the particular image is a "dog", according to the model, and the prediction is that "it's a dog" (because 0.8 > 0.5).

But is the $\hat{y}$ really a (true) probability? What I mean is: if we feed the NN with many different images, and say for 100 of them the model gives the same $\hat{y} = 0.8$. For this $\hat{y}$ to be a true probability, I would expect 80 of these images to actually be dogs and 20 not dogs. To check this we would look at the $y$ (assuming that they exist). In the case when all the 100 images are actually dogs, our model made correct prediction (since it correctly labeled all of them as dogs, because 0.8 > 0.5 for all of the 100 images), but the $\hat{y}$ is not a true probability prediction, because it should be 1. This is actually the accuracy, $100/100 = 1$. On the other hand, if out of 100 images 80 are actually dogs and 20 are not dogs, then the model prediction has accuracy $80/100 = 0.8$, because it incorrectly classified all 100 images as dogs, again, but only 80 are actually dogs. However now the $\hat{y} = 0.8$ represents the true probability. In other cases, the accuracy can be e.g. 0.9, meaning that the predicted $\hat{y} = 0.8$ is not a correct probability, but is closer to it than in the first case.

So, first of all, is what I am saying correct? Second, does the "calibration" fix this issue, i.e. making $\hat{y}$ = accuracy allows the $\hat{y}$ to be interpreted as true probability?

"Not all classification models are naturally probabilistic" is the first line of this chapter [1] .
This means that some models do output probabilities, and some do not. This is highly dependant on the model structure as well as of the objective function that it uses.

In particular, for neural network, a commonly used objective function is Cross Entropy. And to my understanding, Cross Entropy does try to approach the true distribution of the target variable, conditionned on the input variable. This would mean that for a hundred similar cases $X_{similar}$, that are similar in terms of the information that the model is capable to understand, if 80% of those cases are dogs, the predicted score will be a descent approach of the probability you want to estimate: $ \hat{y} \approx p(dog|X_{similar}) = 80/100$.

For other models with less "statistical" behaviour, such as Decision Tree, SVM, of models taken into a Boosting or Bagging process, the output distribution ($\hat{y}$) is skewed. In this binary classification setting, these models would try to give a score as close as possible of the target value. In the case at hand, they would consider the $X_{similar}$ as a group, and the best decision for that group in terms of accuracy beeing to classify them all as dogs, $\hat{y}$ would be skewed toward 1 value.

So, my take on this :

If your goal is to have accurate probabilities, use a model that directly outputs probabilities. If the probabilities are "perfect", a cutoff value of 0.5 would provide the best accuracy (i.e. $\hat{y}\geq{0.5}$ as a decision rule)
If your goal is to have a good classifier in terms of accuracy, use the model that... has the best accuracy. Note that in this case, the best cutoff value may not be 0.5 (it can even be far from it).
If you have both those goals, well, I do not have the answer as this is data and model dependant. If your scores are not probabilities, calibration should make them look more like it indeed. See Calibration in classification for a list of calibration methods.

[1] Wikipedia - Probabilistic Classification  - Probability calibration

[2] Wikipedia - Calibration (statistics) - In Classification


Difference between RFE and SelectFromModel in Scikit-Learn What is the difference between Recursive Feature Elimination (RFE) function and SelectFromModel in Scikit-Learn? Both seems exactly similar.

They effectively try to achieve the same result but the methodology used by each technique varies a little.

RFE removes least significant features over iterations. So basically it first removes a few features which are not important and then fits and removes again and fits. It repeats this iteration until it reaches a suitable number of features.

SelectFromModel is a little less robust as it just removes less important features based on a threshold given as a parameter. There is no iteration involved.


How to feed tuples of multiple features to the input layer of a Neural Network I have a dataset for which every object has multiple features, and some of those features exist in tuples, like as in list of $(x, y)$ coordinates for each object. I know I can create an input layer of number of neurons equal to number of features of each object, but how do I pass these 'tuple' sets to the input. Do I break them and append to create a longer list of features, but will that not lead to loss in some correlation between these 2 sets of points from tuples?

This sounds like an ideal situation for a complex valued neural net. A readable introduction is https://makeyourownneuralnetwork.blogspot.com/2016/05/complex-valued-neural-networks.html


Why logistic regression example code does not port to linear regression example? I am looking at this tutorial: https://www.dataquest.io/mission/74/getting-started-with-kaggle

Following is the code for linear regression to predict, based on some variables, the survival of the titanic passengers.

# Import the linear regression class
from sklearn.linear_model import LinearRegression
# Sklearn also has a helper that makes it easy to do cross validation
from sklearn.cross_validation import KFold

# The columns we'll use to predict the target
predictors = ["Pclass", "Sex", "Age", "SibSp", "Parch", "Fare", "Embarked"]

# Initialize our algorithm class
alg = LinearRegression()
# Generate cross validation folds for the titanic dataset.  It return the row indices corresponding to train and test.
# We set random_state to ensure we get the same splits every time we run this.
kf = KFold(titanic.shape[0], n_folds=3, random_state=1)

predictions = []
for train, test in kf:
    # The predictors we're using the train the algorithm.  Note how we only take the rows in the train folds.
    train_predictors = (titanic[predictors].iloc[train])
    # The target we're using to train the algorithm.
    train_target = titanic["Survived"].iloc[train]
    # Training the algorithm using the predictors and target.
    alg.fit(train_predictors, train_target)
    # We can now make predictions on the test fold
    test_predictions = alg.predict(titanic[predictors].iloc[test])
    predictions.append(test_predictions)

import numpy as np

# The predictions are in three separate numpy arrays.  Concatenate them into one.
# We concatenate them on axis 0, as they only have one axis.
predictions = np.concatenate(predictions, axis=0)

# Map predictions to outcomes (only possible outcomes are 1 and 0)
predictions[predictions > .5] = 1
predictions[predictions <=.5] = 0

accuracy = sum(titanic["Survived"] == predictions)/len(predictions)

print (accuracy)

This is clear. For logistic regression is much easier:

from sklearn import cross_validation

# Initialize our algorithm
alg = LogisticRegression()
# Compute the accuracy score for all the cross validation folds.  (much simpler than what we did before!)
scores = cross_validation.cross_val_score(alg, titanic[predictors], titanic["Survived"], cv=3)
# Take the mean of the scores (because we have one for each fold)
print(scores.mean())

Now, why can't I do just, for LinearRegression:

from sklearn import cross_validation

# Initialize our algorithm
alg = LinearRegression()
# Compute the accuracy score for all the cross validation folds.  (much simpler than what we did before!)
scores = cross_validation.cross_val_score(alg, titanic[predictors], titanic["Survived"], cv=3)
# Take the mean of the scores (because we have one for each fold)
print(scores.mean())

This gives me a totally wrong result. Why is that? What is wrong in the last snippet of code?

I suspect that you are missing this mapping:

# Map predictions to outcomes (only possible outcomes are 1 and 0)
predictions[predictions > .5] = 1
predictions[predictions <=.5] = 0

Basically, what linear regression produces after a call to predict is a set of numbers (scores), then we need to manually map it back to class label. cross_validation.cross_val_score doesn't do the mapping for us, so what it ends up doing is comparing scores with class labels, which turns out to be bad.

On the other hand, predict of logistic regression gives us the class label (Reference)


Can we use a predictive model of a device to another device of similar specification? Let's say, we have a model M1 that is trained on a particular device D1 (let's say the device is a Car), with historical data over 6 months of D1.
M1's purpose is to predict the behavior of the car. Now we have another device D2 (another car), with same specification but new and yet to be run.

Can we use M1 to predict the behavior of D2, since we do not have any data on D2?

The answer to your question is yes, essentially while building a model what we are doing in reality is devising a mathematical function of a physical object (here car). So essentially what we are building is a mathematical model of a car.

What you should be aware of is ?
1. Say for e.g. if you are building a model for a car A and Car B, and if car A is a gasoline(internal combustion car) and car B is an electric car, then things become tricky if you want to build a mathematical function of internals of a car.
2. This does not affect if the model does not consider the internals of the car meaning the outcome (prediction) is not dependent on the internals of the car, which may not be the case always.

but in general you can use the model, it is always preferable to do a validation on the new dataset.


Standard method to integrate tools coded in multiple languages in an analysis workflow I am trying to stitch together multiple packages and tools from multiple languages (R, python, C etc.) in a single analysis workflow.

Is there any standard way to do it? Preferably (but not necessarily) in python.

Luigi is an open source python package by Spotify that does exactly what you described:

  Luigi is a Python (2.7, 3.3, 3.4, 3.5) package that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more.

Its philosophy is similar to GNU Make, letting you define tasks and their dependencies.

There is also Apache Airflow (originally from Airbnb), another python solution:

  Airflow is a platform to programmatically author, schedule and monitor workflows.
  Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies.

You may found a complete comparative table here.