Missing outputs in multiple-output neural net I am looking at a task, where I want to predict multiple things from an image (an animal's breed [categorical], age [continuous number] and gender [categorical]). Unsurprisingly, my first thought was to use a neural network (e.g. adding multiple outputs to a pre-trained covnet) and I would try keras first (having used it before). I assume it would be useful to try to do this all in a single covnet (especially based on a TWiML talk on this paper, where the author suggested that in general one prediction target will help to improve features for another and given that e.g. how one would recognise gender will differ by breed etc.).

My question arises, because for some training/validation/test images some (but not all) of the multiple prediction targets may be missing. Sometimes we will just know the breed, for some images we will not know the gender and the exact age may oftne be unavailable (but only that the animal is adult versus younger than about 4 months, which a human can easily label by just looking at the animal).

If I had to guess, I would expect the missingess to be missing at random (but likely not missing completely at random). For example, for some breeds and ages it is harder to tell the gender by just looking at the animal (perhaps the neural network will manage to do it). So, someone who just takes a photo of an animal may not know the gender and humans are no good at labelling gender from photos. Another example: age might be unavailable, because someone just took a photo of an animal they do not have detailed information on or because someone bought an animal secondhand and never found out.

Is there some standard way of handling this? My intuition is that even records with one or two missing prediction target still should provide useful information for the other ones. Thus, listwise deletion of all records with any missing target information would be extremely inefficient (and possibly even otherwise problematic?!).

Since we are talking about multiple different types of targets (classes versus numerical for example) we already need a composite loss function. I will consider how to balance the different composite parts of the loss function outside of the scope of this answer but if you look into multi-task learning there are solutions to this. What you could do (both in training and evaluation) is to mask the parts of the loss function that are unknown.

You can do this fairly easily (but hacky) by passing weights as input during training or prediction for each output in your sample whether it was available or not as 1 or 0 and multiply that component with the weight. For analysis of the training and prediction you need to make sure to take the mean of only the available labels because otherwise your loss values are too optimistic.

I do think this approach is very valuable in a lot of cases, this can also help with semi-supervised learning where you do have labels available for a similar task.


ROC AUC score is better if test data is imbalanced I have an imbalanced dataset and I'm using XGBoost to do binary classification. I used down sampling together with target and one hot encoding for train data. For test data I once used just the encodings and left it unbalanced and once tried with a balanced test dataset.

The ROC AUC score was quite higher for the imbalanced test data than the balanced one. How is this possible? I thought for the ROC AUC score there should not be any difference?

ROC normally reports higher scores for imbalanced data sets because it does not take into account false discoveries. For this reason, it is recommended to only use ROC when the positive and negative sets are roughly equal in size. Otherwise, the suggestion is to compute the area under the Precision-Recall Curve instead.

You can read a bit about the reasons here.

https://machinelearningmastery.com/roc-curves-and-precision-recall-curves-for-classification-in-python/

  However, ROC curves can present an overly optimistic view of an algorithm’s performance if there is a large skew in the class distribution. […] Precision-Recall (PR) curves, often used in Information Retrieval , have been cited as an alternative to ROC curves for tasks with a large skew in the class distribution.

  — The Relationship Between Precision-Recall and ROC Curves, 2006.


Why is it good news that the expected gradient of the loss for a mini batch is equal to the gradient for the whole set? I recently proved that

$\mathbb{E}_I[\nabla L_I(x,y,w)] = \nabla(L_I(x,y,w))$

Where $I$ represents a randomized batch of size $m$.

However, I cannot really understand why this is good news. I know that gradient is important for optimizing a model, especially with gradient descent. I also understand that in mini-batch algorithms we average over a whole batch instead of doing it for all the data, saving a lot of computations and reducing the variance.

It is usually good news when an approximating function you use to take an action to optimise something is unbiased compared to the thing being approximated. It means your actions in aggregate will head towards the same optimisation point.

If the gradient was biased in the long term, then (depending on nature of the bias) an optimisation routine will converge incorrectly by following them in gradient descent, and give an incorrectly optimised answer.

It is sometimes OK if a value is initially biased, but the bias reduces over time towards zero. Not that this is the case here, but you will see that in some machine learning algorithms. For instance, in a neural network using momentum, typically the initial momentum values are biased, but the bias decays exponentially (it is actually possible to make an unbiased momentum estimate, but most implementations of SGD with momentum or Nesterov momentum do not do that - Adam implementations typically do).


An Artificial Neural Network (ANN) with an arbitrary number of inputs and outputs I would like to use ANNs for my problem, but the issue is my inputs and outputs node numbers are not fixed.
I did some google searches before asking my question and found that the RNN may help me with my problem. But, all examples which I've found somehow have a defined number of input and output nodes.
So, I'm looking for a strategy, how to make it real or at least some examples, preferable in Keras or PyTorch.
More details about my issue:
I have two inputs lists, where the length of the first one is fixed and equals two, e.g.:
in_1 = [2,2]

but the length of the second list is flexible, the length can be from three to inf, e.g.:
in_2 = [1,1,2,2]

or
in_2 = [1,1,1,2,2,2,3,3,3]

Also, input lists depend on each other. The first list shows the dimension of the output list. So if in_1 = [2,2], means the output must have a possibility to be reshaped to [2,2] form.
Currently, I'm thinking to combine two input list into one:
in = in_1 + in_2 = [2, 2, 1, 1, 2, 2]

Moreover, the output has the same length as the in_2 list, e.g.:
if input lists are:
in_1 = [2, 2]
in_2 = [1, 1, 2, 2]

Output should be:
out = [1, 2, 1, 2]

Any ideas are welcome!

The answer may depend on the significance of the length of the input vector or how it originates.
However, the simplest solution is usually to know the largest size input and use that as number of vectors. If the given input has lesser length, you can do the padding with zeros or appropriate symbols. So instead of having a vector $[1, 2, 3]$ and $[1, 2, 2, 3]$ you can have vectors $[1, 2, 3, 0]$ and $[1, 2, 2, 3]$.
The same can apply for output. If output expected is $[1, 2, 1]$ and $[1, 3, 4, 1]$ you can treat the first output as $[1, 2, 1, 0]$
Isn't this just a hack?
Typically, Neural networks does function approximation. Ideally, it represents vectors (matrices) as input and vectors (matrices) as output. This is why it is always desirable that size of your input vector be fixed.


How to get feature importance from RandomForest using scikit-multilearn library? I am working on multi-label classification problem, binary case. As a target variable there are five columns with 0-1 values.

For a model training I use scikit-multilearn library. Below is my code for training a model with Binary Relevance and RandomForest.

parameters = [
{
    'classifier': [RandomForestClassifier(random_state=42)],

    'classifier__n_estimators': [50, 100, 200],
    'classifier__max_features': ['auto', 'sqrt', 'log2'],
    'classifier__max_depth' : [4, 6, 8, 10, 12],
    'classifier__criterion' :['gini', 'entropy']}]

clf = RandomizedSearchCV(BinaryRelevance(), parameters, scoring='f1_weighted', cv=5, \
n_jobs=-1, verbose=10, random_state=10)
clf.fit(X_train, y_train)

How could I extract feature importance from the model?
Should I take the best parameters from this model and then apply to data with each label separately using RandomForest from sklearn?

First, to directly answer your question, the easiest way to get Feature Importance using scikit learn is this, where model is the variable holding your classifier.

print(model.feature_importances_)

However, this method only exists on some of the Ensemble models, namely:

AdaBoostClassifier
AdaBoostRegressor
ExtraTreesClassifier
ExtraTreesRegressor
GradientBoostingClassifer
GradientBoostingRegressor
RandomForestClassifier
RandomForestRegressor
RandomTreesEmbedding

If you're wondering why, there's a fantastic free book online all about Interpretable Machine Learning. Here's an excerpt:

  The easiest way to achieve interpretability is to use only a subset of
  algorithms that create interpretable models. Linear regression,
  logistic regression and the decision tree are commonly used
  interpretable models.

Once you've chosen the right model, beware of using feature importance! See here, it ranks random data very highly. The article proposes using Permutation Importance instead, as well as Drop-Column Importance.

They created a library called rfpimp for doing this, but here's a tutorial from scikit themselves on how to do both of those with just scikit-learn. I've pasted the permutation importance example from that tutorial below:

from sklearn.inspection import permutation_importance

result = permutation_importance(rf, X_test, y_test, n_repeats=10,
                                random_state=42, n_jobs=2)
sorted_idx = result.importances_mean.argsort()

fig, ax = plt.subplots()
ax.boxplot(result.importances[sorted_idx].T,
           vert=False, labels=X_test.columns[sorted_idx])
ax.set_title("Permutation Importances (test set)")
fig.tight_layout()
plt.show()


Please suggest some good ways to test my RNN I have developed an RNN in matlab and now its time to test it. You can set your desired number of layers and nodes and it trains data in random chunks. I also included an annealing function for learning rate that decays gradually from 1 to 1e-5 to avoid overtraining of data.

I would like you to suggest me some simple ways to test my network in a way that i can get some useful feedback about its behavior from the results. I am a little concerned about convergence. I' ve set to do some testing of my own but would like to hear some suggestions from more experienced users.

I try to simulate some 1st, 2nd, 3rd, ... grade function between (0-1) and see how it goes. I will also try to predict the next value of a time series based on previous values, as another testing method. What would be a good result from these tests? An accuracy of ??% and an mse of ?? ?

When I built my first vanilla RNN from scratch, I was very curious about it's correctness. So, I trained it on a portion of a sine curve and tried to generate the part of the curve that comes next. And consequently, I plotted mine and the actual to visualise.

(P.S.: here is my code!)


Why I am having odd results with this neural network model I am kind of new to neural networks, so I was doing some practice with iris data set. But the results are weird. Because I am running the network with the same data (not randomly chosen) and huge number of epochs (500 times), but results are sometimes 0,25 sometimes 1,0 or 0,95. The only thing that random is biases and weights. I think training the network with 500 epochs should remove this randomness at the end. But it seems like it doesn't or there is some another problem that I am not noticed yet. If you help me about this I will be thankful.
(PS: If you give me an advice about making the question better I will edit it, I am new at forum)
import tensorflow as tf
import pandas as pd
import numpy as np
from tqdm import tqdm

def to_one_hot(label_data):
    one_hot = []
    for item in label_data:
        temp = []
        for i in range(3):
            if(i == item):
                temp.append(1)
            else:
                temp.append(0)
        one_hot.append(temp)
    return one_hot

def read_data(n):
    df_data = pd.read_csv('iris.csv')
    x = []
    y = []
    species_dict = {'Iris-setosa' : 0,
            'Iris-versicolor' : 1,
            'Iris-virginica' : 2}
    for i in range(len(df_data['Id'])):
        temp_item = []
        temp_item.append(df_data['SepalLengthCm'][i])
        temp_item.append(df_data['SepalWidthCm'][i])
        temp_item.append(df_data['PetalLengthCm'][i])
        temp_item.append(df_data['PetalWidthCm'][i])
        x.append(temp_item)
        y.append(species_dict[df_data['Species'][i]])
    y = to_one_hot(y)
    x_train = []
    y_train = []
    x_test = []
    y_test = []
    for i in range(50):
        if (i < n):
            for t in range(3):
                x_train.append(x[i+50*t])
                y_train.append(y[i+50*t])
        else:
            for t in range(3):
                x_test.append(x[i+50*t])
                y_test.append(y[i+50*t])
    return x_train,x_test,y_train,y_test

x_train,x_test,y_train,y_test = read_data(40)

n_layers = 30
n_nodes = 7
n_classes = 3

x = tf.placeholder('float', [None,4])
y = tf.placeholder('float')

def neural_network_model(data):
    layers = []
    layers.append({'weights':tf.Variable(tf.random_normal([4,n_nodes])),
        'biases':tf.Variable(tf.random_normal([n_nodes]))})
    for i in range(n_layers-1):
        layers.append({'weights':tf.Variable(tf.random_normal([n_nodes,n_nodes])),
            'biases':tf.Variable(tf.random_normal([n_nodes]))})

    output_layer = {'weights':tf.Variable(tf.random_normal([n_nodes,n_classes])),
            'biases':tf.Variable(tf.random_normal([n_classes]))}

    for i in range(n_layers):
        if (i == 0):
            nl = tf.add(tf.matmul(data,layers[i]['weights']),layers[i]['biases'])
            nl = tf.nn.relu(nl)
        else:
            nl = tf.add(tf.matmul(nl,layers[i]['weights']),layers[i]['biases'])
            nl = tf.nn.relu(nl)

    output = tf.add(tf.matmul(nl,output_layer['weights']),output_layer['biases'])

    return output

def train_neural_network(x):
    prediction = neural_network_model(x)
    cost = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits(logits = prediction, labels = y) )
    optimizer = tf.train.AdamOptimizer().minimize(cost)

    hm_epochs = 500

    with tf.Session() as sess:
        sess.run(tf.initialize_all_variables())
        for epoch in tqdm(range(hm_epochs)):
            _ ,c = sess.run([optimizer,cost], feed_dict = {x:x_train, y:y_train})
        correct = tf.equal(tf.argmax(prediction,1),tf.argmax(y,1))
        accuracy = tf.reduce_mean(tf.cast(correct, 'float'))
        print('Accuracy : ',accuracy.eval({x:x_test,y:y_test}))

train_neural_network(x)

Neural network algorithms are stochastic.
This means they make use of randomness, such as initializing to random weights, and in turn the same network trained on the same data can produce different results.The random initialization allows the network to learn a good approximation for the function being learned.
The most common form of randomness used in neural networks is the random initialization of the network weights. Although randomness can be used in other areas, here is just a short list:

Randomness in Initialization, such as weights.
Randomness in Regularization, such as dropout
Randomness in Layers, such as word embedding.
Randomness in Optimization, such as stochastic optimization.

The Solutions.
The are two main solutions.
Solution #1: Repeat Your Experiment
The traditional and practical way to address this problem is to run your network many times (30+) and use statistics to summarize the performance of your model, and compare your model to other models.
I strongly recommend this approach, but it is not always possible due to the very long training times of some models.
Solution #2: Seed the Random Number Generator
Alternately, another solution is to use a fixed seed for the random number generator.
from numpy.random import seed
seed(1)


Understanding math notation in infoGAN paper I'm reading this paper about mutual information in infoGAN infoGAN_paper_link and already have the code to run it. I pretty much found code for it which is fine and dandy except for the fact that I kinda don't understand some of the code in the cost function. So, I looked at the paper to dissect it for better understanding and came across some math notation that I don't understand (pic below). The usage of the notations I'm trying to figure out are how the "~" in the Expectation subscripts, "'", and "||" symbols are used.
This is what I think the notation means.

"~" in the expectation subscript: the variable on the left of "~" can be any continuous value coming from whatever is to the right of the "~"
"'" next to the "c" in P(c'|x): I have no clue. I thought those were symbols for derivates but that makes no sense for this equation so it's def not that.
"||": I'm not sure. I only know of these symbols being used in Norms but that's obviously not the case here.

Screenshot of a formula from the paper

The actual code I was trying to figure out was this in PyTorch. It's the variational lower bound (mutual information term) in the cost (It's not the formulas in the screenshot above). However, I ran into the formulas in the screenshot first before getting to the formula calculations for the variational lower bound formula.
mutual_information_lower_bound = lambda c_true, mean, logvar: Normal(mean,logvar.exp()).log_prob(c_true).mean()

Thanks for the help!

The notation $D_\mathrm{KL}(P \| Q)$ is pretty much standard for the Kullback-Leibler divergence. There is an interesting discussion on Mathematics SE on the reasons for the fairly unusual notation used for divergences.

In general, $x \sim G(z, c)$ means "$x$ is a random variable distributed according to $G(z, c)$". In the subscript the author means to take the expectation with respect to the random variable $x$, supposing that $x$ has that distribution.

If you have a variable, such as $a$, it is not unusual to call a related variable $a'$ (usually pronounced "a prime"). In this case $c'$ is just a random variable distributed according to $P(c\mid x)$, with the name $c'$ probably meant to remind you that the variable is in some way connected to $c$. In cases where derivatives are obviously not the right thing, this is probably what a prime symbol means.


Regression or Correlation for this RQ? Our little group at uni is investigating if there is a relationship between 3 measures of social well-being (social anxiety, social connectedness and self esteem) and usage time (on-screen time in hours) on several social media platforms. Would you recommend that we use correlation or regression in this case?

Thanks in advance :)

You need to provide us with more information on your variables, are they categorical or numerical? If usage time is numerical then you can build a linear regression model and the get the coefficients and associated p-values out of it.

If all variables are numerical you could calculate the correlation between them, but the linear model gives more information / is superior.


Hyperparameter tuning of neural networks using Bayesian Optimization One of the assumptions for finding good hyperparameters using Bayesian optimization (GP) is that the unknown function is smooth. Is this assumption valid for neural networks or at least for most of the neural networks? Can we find any reference?

Neural networks are optimized by gradient descent which assumes the loss function for parameters is a differentiable function, in other words smooth. Given the nature of the differentiable loss function, Bayesian Optimization could be used for neural networks hyperparameter optimization. In fact, gradient descent can be used to learn the hyperparameters themselves as evidenced in the paper - "Gradient Descent: The Ultimate Optimizer".


Time Posting Data Analysis I work in a professional services company and would like to get some analytics on how discipline the fee earners post their times to the system may have an impact to the revenue.

One area that I am thinking of is to see if there is a correlation between how late the time entry is posted after the work was done and whether that time entry is being written off (i.e. does not make it to the bill to the client).

Is this a reasonable thing to analyse?
If so, what technique should I be looking at to analyse this, e.g. correlation, regression, clustering, etc?

If there are other things that I can analyse as well, please let me know.

I am quite new to this data analytics/mining/science so please guide me. I have some understanding of R language which will be my primary language in analysing the data (and SQL to get the data).

Thank you.

If you visualize the lateness of time posting over the revenue, you will see the clear picture and understand the correlation.

What you need is to create the corresponding columns in your tables.


What kind of data analysis problem is this? This might be a dumb question, but I am an absolute beginner.
What kind of data analysis problem is the following?
Is it an Exploratory Data Analysis problem/Data Mining problem/data analytics problem?
Secondly, what steps can I follow to solve the following problem? Is there any algorithm?

Records in this data set describe decks of cards used in a popular
collectible card video game called "Clash Royale". These decks were
obtained using the RoyaleAPI.com service, from games that took place
in January 2019. Each record consists of five values:

timestamp of the game (column timestamp),
arena ID (column arena_id – higher the arena, more skilled/experienced a player is)
outcome of a game (column has_won, 1 – the player won, 0 the player lost)
a player ID (column tag)
list of exactly eight cards in the player’s deck separated by “_” signs (column player_deck)

Your task is to analyze this data and search for interesting card
usage patterns, and interactions/dependencies between cards.
For example:

find card combos that were particularly popular in January 2019 (e.g., top 100 card sets with regard to their support, top 100 card
sets of size 2, size 3, etc.),
identify those card combos which have high win-rates (e.g., top 100 card sets with regard to win-rate and with support > 1%),
does the card usage/popularity/effectiveness changes in time?
does the arena level have any influence on card usage/popularity/effectiveness?
find interesting associations between cards,
can you cluster players according to their play patterns and card preferences?     Additionally, design and construct a card recommender
system that allows players to indicate four cards which they want to
have in a deck, and recommends the remaining four to create a
reasonable deck. How can you evaluate the effectiveness of the
recommendations?    You have to report the discoveries in the form of
an R notebook (with code and all computation outcomes). You have to
remember about visualizations. I.e. you have to make this report as
visually interesting for a reader as you can.

You should start with exploratory data analysis to understand the data in as much detail as possible. Once you have a good understanding of the obvious patterns in the dataset,  you should then turn your focus towards answering the less obvious questions using modelling techniques.
The final item on the list that hints at building a recommender is an example of data modelling whereas all other items appear to be exploratory data analysis.
It appears this is an open ended problem. It is for you to look at the data, gain an understanding out of it and then figure out answers to interesting and less obvious questions that might interest the Business group.


Best Programming Language for Data Science I'm learning JS, HTML and CSS, but I doubt JS is very good at Data Analysis. So, what would you guys recommend me learning to start my "career" in Data Science? What's the best programming language for processing data?
P.S. I love statistics and programming so I think this will be fun.

This is no doubt a duplicate, but here's how I'd weigh in on the major languages:

R:

Fantastic support for packages and specialised stats analysis
community, you can find a package to do just about anything you need
and it will be relatively easy to use.
Is good for throwing together prototypes and performing exploratory analysis.
Is Free and Open Source.
Slower than Python. Basically don't loop over anything. It's an odd language for a programmer to use (coming from a software dev
background). Clearly designed by mathematicians.
Relatively little choice of good IDEs

Python:

Fast.
Also very good as a general purpose language so has 'broader' package support. Free and Open Source.
Easy to use for Big Data applications.
Not as streamlined for analysis as R.
Syntax can be difficult to read (no surrounding braces to make it obvious where functions/ if statements end).
Can be particularly tedious working with Dataframes compared to R.

MATLAB:

Generally slower.
Has very impressive packages for signal processing/image recognition and all the cool stuff.
Is very readable and easy to comprehend generally.
Is NOT free. Student licenses are available. Was quite complicated for me to get my hands on one though...
Has very good support for mathematical analysis similar to R, but much better matrix functions.

Personal recommendation: Python. Kill two birds with one stone, learn good general to advanced programming concepts and data science at the same time.

Good article: https://www.linkedin.com/pulse/r-vs-python-matlab-octave-julia-who-winner-siva-prasad-katru


Estimating forward velocity for a swimmer With a modern IMU with 9 angles of freedom collecting accelerometer, magnetometer and gyroscope data on 3 axis, what would be the best approach on filtering the data and handling it to accurately estimate the forward velocity of the swimmer?

My approach was to:
1. Use a 3-point moving average to get rid of any vibrations caused by unneeded movements
2. Use a median average to get rid of repetitve movements such as shakes or water  resistance
3. Perform integration of accelerometer data.

At this point, I'm unsure of my data filtering approach. Is there a better approach? What methods should I use for acc. integration?

Your most important question is not the algorithm. You must check the precision of the sensor you can buy. I've tried several sensors, after 50 seconds the cumulated error can be a magnitude higher then the velocity itself. If you can buy a 1000 USD sensor, then it's precision would be sufficient, but you still have the problem that there will be no 0 position and 0 velocity, ie. the sensor will be moving even if the swimmer has not yet started to swim. This movement of the sensor will be greater then the difference between the velocities of two swimmers.


What are x variable and y variable in word2vec model if it is supervised learning What are x variable and y variable in word2vec model if it is supervised learning.
In both the flavours- CBOW and skip-gram model.

Though some blogs have explained it as unsupervised learning.

Thanks

First I would call it self-supervised in the sense that you dont need human labeling, but still need labels for NN to learn.

Both CBOW and skip-gram are "combined" into word2vec but they are shallow-neural-network architectures (albeit different ones).

Since they are different architectures, they do different things:

CBOW learn to predict the word by the context. Or maximize the probability of the target word by looking at the context.

skip-gram model is designed to predict the context so given the word xyz it must understand it and tell us that there is a huge probability that the context abc__dfg is what "surrounds" xyz.

Given that, you could try to deduce yourself what should input and output data look like to model this desired input output of CBOW, skip-gram (matrix representation of words and than probabilities and a vector representing a word and multiclass output).

Check also this answer


How to find unknown number of clusters in circular data? I have some 1 dimensional data. Each record in the data is a specific time of the day. In order to cluster it I projected the data onto a circle of radius 1 unit. Now I need to find clusters in this data. The number of clusters are unknown and it is preferred to find clusters with high density of records in them. By density I mean that a large volume of records should be packed in a small space.

How should I go about finding clusters in the above mentioned data?

Instead of projecting into the circle and thus making your problem 2d, why don't you just use a cyclic distance measure?

This problem should be straightforward by doing kernel density estimation on the (cyclic) time of day. Then find the peaks, which are your clusters.


Feature engineering while using neural networks When doing Data Science, we concentrate on feature engineering first - check on correlations, imputations, tranformations etc.

Do we have to follow the same steps while feeding the inputs to a neural net? Or what should be the steps taken?

If you are talking about tabular data (not images, video, sounds etc) then yes. All of preparations you mentioned are applicable to Artificial neural networks.
For instance, data normalization is very important since neurons activations are calculated applying some activation function on a weighted sum(linear combination) of it's inputs. A value of 0.01 and a value of 10000000 in a weighted sum will always have a large value if the weights are at the same scale. Theoretically the network could learn how to deal with such discrepancies on the input values but in a practical sense normalization can make learning faster.
There is a classical paper on efficient backprop, with many tips and tricks, by Yann LeCun.


Is there a loss function that measures the cross similarity between two 2D tensors? Given two input tensors x1 and x2 with the shape [batch_size, hidden_size], let S be the matrix of similarity between all pairs (predict, target), where predict and target are dense vectors with the shape [hidden_size] and predict belongs to x1 and target belongs to x2.

Is there any loss function that is minimized as the values in the diagonal of S are close to 1 while the other values are close to -1?

That is, the similarity between the vectors from x1 and x2 of the same index must be greater than between vectors of different indexes.

Currently, I've implemented the following solution inpired by N-Pair Loss published from NIPS 2016:

import torch
from torch import nn
from matplotlib import pyplot as plt
import seaborn as sn

class NPairsLoss(nn.Module):
    """
    The N-Pairs Loss.
    It measures the loss given predicted tensors x1, x2 both with shape [batch_size, hidden_size],
    and target tensor y which is the identity matrix with shape  [batch_size, batch_size].
    """

    def __init__(self):
        super(NPairsLoss, self).__init__()
        self.ce = nn.CrossEntropyLoss()

    def show(self, similarity_scores):
        sn.heatmap(similarity_scores.detach().numpy(), annot=True, annot_kws={'size': 7}, vmin=-1.0, vmax=1.0)
        plt.show()

    def similarities(self, x1, x2):
        """
        Calculates the cosine similarity matrix for every pair (i, j),
        where i is an embedding from x1 and j is another embedding from x2.

        :param x1: a tensors with shape [batch_size, hidden_size].
        :param x2: a tensors with shape [batch_size, hidden_size].
        :return: the cosine similarity matrix with shape [batch_size, batch_size].
        """
        x1 = x1 / torch.norm(x1, dim=1, keepdim=True)
        x2 = x2 / torch.norm(x2, p=2, dim=1, keepdim=True)
        return torch.matmul(x1, x2.t())

    def forward(self, predict, target):
        """
        Computes the N-Pairs Loss between the target and predictions.
        :param predict: the prediction of the model,
        Contains the batches x1 (image embeddings) and x2 (description embeddings).
        :param target: the identity matrix with shape  [batch_size, batch_size].
        :return: N-Pairs Loss value.
        """
        x1, x2 = predict
        predict = self.similarities(x1, x2)
        self.show(predict)
        # by construction the probability distribution must be concentrated on the diagonal of the similarities matrix.
        # so, Cross Entropy can be used to measure the loss.
        return self.ce(predict, target)

However, with this loss, the model ends up converging to a scenario where all dense vectors are equal to each other. Which can be seen by executing the following code snippet:

batch_size=7
hidden_size=768
def m_model(scenario=0):
    if scenario == 0: # all equal all
        p1 = torch.ones((batch_size, hidden_size))
        p2 = p1
    elif scenario == 1: # all different all
        p1 = torch.ones((batch_size, hidden_size))
        p2 = -1*p1
    else: # desired case
        p1 = torch.rand((batch_size, hidden_size))
        p2=p1

    return p1, p2

predict = m_model(scenario=0)
target = torch.arange(batch_size)
loss = NPairsLoss(1)

print("Loss:", loss(predict, target))
# Loss: tensor(1.9459), using scenario=0
# Loss: tensor(1.9459), using scenario=1
# Loss: tensor(1.7364), using scenario=2

Any suggestions on how to penalize these scenarios where the similarity matrix has all the same values?


Clustering 1-gram Strings I have a big list of mail addresses (around one million) and I want to use/adapt/create an algorithm to find similarities between them (basically to cluster them).

All the algorithms I've checked so far e.g. n-grams variants, bag of words etc. are only used for clustering strings that are part of a context but here is not the case.

So far I've tried Levensthein, LCS. Both of them are pretty accurate but I'm interested in an algorithm that uses machine learning techniques, if such an algorithm exists for this task. Also they are pretty slow for my dataset.

You said:

  All the algorithms I've checked so far e.g. n-grams variants, bag of
  words etc. are only used for clustering strings that are part of a
  context but here is not the case.

However, those algorithms can still be used if you don't treat tokens as full words, but as combinations of characters instead.

Here is an example of such algorithm in Python using Sklearn.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans

data = [
    'alex@examplemail.com',
    'bernard@examplemail.com',
    'chris@newmail.com',
    'dorothy@nemail.com',
    'emily@examplemail.com'
]

X = TfidfVectorizer(analyzer='char').fit_transform(data)

kmeans = KMeans(n_clusters=2).fit(X)

for cluster in set(kmeans.labels_):
    print('\nCluster: {}'.format(cluster))
    for i, label in enumerate(kmeans.labels_):
        if label == cluster:
            print(data[i])

> Cluster: 0
> alex@examplemail.com
> bernard@examplemail.com
> emily@examplemail.com
>
> Cluster: 1
> chris@newmail.com
> dorothy@nemail.com

The major trick is TfidfVectorizer(analyzer='char'), where you don't treat n-grams of words, but of characters instead.

P.S. If you at some point want to use word n-grams (if you have generally well written sentences) you can use TfidfVectorizer(analyzer='word').


unsupervised anomaly detection on sparse data Given that I have a very sparse data matrix with continuous features, like this dataframe for example
 Feature_A  Feature_B  Feature_C....Feature_Z
 0.3            0       0.1            0
 0.5            0.5     0              0
 0              0       1.0            0
 1.0            0       0              0
 0.7            0       0              0
 1.0            0       0              0
 0.1            0       0.22          0.43

what is the best way to perform unsupervised anomaly detection on this kind of data? my initial idea was to perform some kind of dimensionality reduction first (e.g SVD or NMF) then do a simple anomaly detection technique on the resultant dense matrix (e.g Isolation Forest) but I'm not sure this is the best way to go.

1- You better start with Isolation forest Isolation Forest This is a very simple algorithm where you can control the contamination rate of your data.
2- For visualization you can plot the anomalous points in red, and you can distinguish them using the Isolation Forest predict(X) function that returns -1 for outliers and 1 for the rest.
3- You can use LSTM by comparing the predicted values of your model, with the real values of your dataset, then use a KPI ( if/else depending on the max value of the difference between the predicted values and the real ones), in simple words you will need to define your own contamination rate.
4- For visualization with LSTM you can use the same function described above with Isolation Forest, but instead of the predict function that return [-1,1], you will use the function described in step 4.


I received this error message: Found input variables with inconsistent numbers of samples: [15573, 15600]. How do I resolve it please? Data source

I received that error and when running my code, I realised that the issue lies with my X_train and y_train.

The link below is the source of my data (test.csv and train.csv)

My X_train.shape gave me (15573,) and my y_train gave me (15600,)

How do I reshape this?

You likely get problems because you remove rows containing nulls in X_train and y_train independent of each other. y_train probably has few, or no nulls and X_train probably has some. So when you remove a row in X_train and the same row is not removed in y_train it will cause your data to be unsynced and have different lenghts. Instead you should remove nulls before you separate X and y.

Before this:

X = df['text']
y = df['label']

Do this:

df.dropna(inplace=True)

And remove this:

X_train.dropna(inplace=True)
X_test.dropna(inplace=True)
y_train.dropna(inplace=True)
y_test.dropna(inplace=True)


How to search for an optimal dithering pattern? I'm trying to find an optimal dithering pattern which can be used as a threshold on a greyscale image to generate a 1 bit black and white image. Ideally it would be optimal in the sense that a human would judge it perceptually most close to the source image.

For instance, dithering with white noise looks pretty bad:

While dithering with blue noise looks a lot better, despite having the same amount of error:

(images from https://blog.demofox.org/2017/10/31/animating-noise-for-integration-over-time/)

While white noise doesn't look good, it can be easily generated in real time, whereas blue noise cannot.

Because of this, I'm trying to find a dithering pattern which is:

NxN in size
Has an even distribution of values: 0/N, 1/N ... (N-1)/N
Perceptually the best bang for the buck

After I have the best pattern for a given N (or a reasonably good pattern), step 2 would be to try and figure out how to calculate it mathematically, but the first step is to find the pattern.

Does anyone have any recommendations on a good way to do this?

What I'm currently trying is using simulated annealing to find an optimal solution.  I'm using SSIM to calculate the score of how good a solution is, for use in in the simulated annealing logic.

For the simulated annealing, I initialize the pattern by doing this:

Initialize the NxN pattern such that row M is set to M/N.
Shuffle the pattern

When calculating a possible move to make for the simulated annealing, I pick two values at random in the NxN pattern and swap them.

This isn't working very well for me, but I am unsure if this is not a good way to go about it, or if maybe I'm choosing bad parameters for the simulating annealing and/or the SSIM.

The simulated annealing has a starting temperature, and a specified number of iterations to take it from there down to temperature 0.

SSIM on the other hand has c1, c2, and c3 which I'm setting to zero.  I am also doing SSIM of every 8x8 block in the dithered image (ALL of them, even though they overlap) and averaging them together to get the SSIM score.

Here is an example of a 5x5 dither pattern this algorithm came up with, which really doesn't look good at all.

I manually derived a 5x5 by hand which looks much better in fact:

Does anyone have any recommendations on how I might search / solve this problem better?

I think you're on the right track. If you can produce an objective scoring algorithm for the output which more closely resembles your subjective perception than 1:1 error you could definitely use simulated annealing to derive from that an optimal matrix at a given size N.

Your current scoring calculation for error seems to focus too much on high-frequency information (details), and not enough on low frequency information (larger areas) - notice the results you deem subjectively superior, if blurred (low-pass-filter) more closely resemble the input image (if also blurred) compared to the results you have deemed inferior.

If it were me, my next step might be to try something hierarchical in order to avoid bias toward any given frequencies, maybe a quadtree where at each level 1x1, 2x2, 4x4, etc. I average the values of the node's children, then scoring uses the error between trees.


Keras/Tensorflow: model.predict() returns a list. How do I match the output with my class names? I have a CNN built in Keras. I have saved it and am now using the model.predict() function to make predictions from it. Whenever I run the following code,
def prediction(path):
    import keras
    from keras.preprocessing.image import load_img, img_to_array
    from keras.models import load_model
    import PIL
    import numpy as np

    img = load_img(path)
    img = img.resize((224, 224))
    img = img_to_array(img)

    img = img.reshape( -1,224, 224,3)

    model = load_model('model1.h5')
    pred = model.predict(img)

    return pred

print(prediction('/path/to/image/')

I get an output like this:
[[7.578206e-37 1.000000e+00 0.000000e+00 0.000000e+00]]
I am doing transfer learning using resnet50 with imagenet weights and here is the model.summary().
I have 4 classes. How do I find out where each prediction belongs?
I have looked here as well but it doesn't seem to help me.
Thanks

Model prediction output is a bunch of probabilities. In order to get category name you need use following snippet. It calculates the argmax of predicions and give it to CLASSES list:
print(CLASSES[np.argmax(predictions)])


Is the direction of edges in a Bayes Network irrelevant? Today, in a lecture it was claimed that the direction of edges in a Bayes network doesn't really matter. They don't have to represent causality.

It is obvious that you cannot switch any single edge in a Bayes network. For example, let $G = (V, E)$ with $V = \{v_1, v_2, v_3\}$ and $E=\{(v_1, v_2), (v_1, v_3), (v_2, v_3)\}$. If you would switch $(v_1, v_3)$ to $(v_3, v_1)$, then $G$ would no longer be acyclical and hence not a Bayes network. This seems to be mainly a practical problem how to estimate the probabilities then. This case seems to be much more difficult to answer, so I will skip it.

This made me ask the following questions for which I hope to get answers here:

Is it possible for any directed acyclical graph (DAG) to reverse all edges and still have a DAG?
Assume a DAG $G$ and data is given. Now we construct the inverse DAG $G_\text{inv}$. For both DAGs, we fit the data to the corresponding Bayes networks. Now we have a set of data for which we want to use the Bayes network to predict the missing attributes. Could there be different results for both DAGs? (Bonus if you come up with an example)
Similar to 2, but simpler: Assume a DAG $G$ and data is given. You may create a new graph $G'$ by inverting any set of edges, as long as $G'$ remains acyclical. Are the Bayes networks equivalent when it comes to their predictions?
Do we get something if we have edges which do represent causality?

TL;DR: sometimes you can make an equivalent Bayesian network by reversing arrows, and sometimes you can't.

Simply reversing the direction of the arrows yields another directed graph, but that graph is not necessarily the graph of an equivalent Bayesian network, because the dependence relations represented by the reversed-arrow graph can be different from those represented by the original graph. If the reversed-arrow graph represents different dependence relations than the original, in some cases it's possible to create an equivalent Bayesian network by adding some more arrows to capture dependence relations which are missing in the reversed-arrow graph. But in some cases there is not an exactly equivalent Bayesian network. If you have to add some arrows in order to capture dependencies, you might end up with a graph which represents fewer independence relations and therefore fewer opportunities for simplifying the computations of posterior probabilities.

For example, a -> b -> c represents the same dependencies and independencies as a <- b <- c, and the same as a <- b -> c, but not the same as a -> b <- c. This last graph says that a and c are independent if b is not observed, but a <- b -> c says a and c are dependent in that case. We can add an edge directly from a to c to capture that, but then a and c being independent when b is observed is not represented. That means that there is at least one factorization which we cannot exploit when computing posterior probabilities.

All this stuff about dependence/independence, arrows and their reversals, etc., is covered in standard texts on Bayesian networks. I can dig out some references if you want.

Bayesian networks don't express causality. Judea Pearl, who did a lot of work on Bayesian networks, has also worked on what he calls causal networks (essentially Bayesian networks annotated with causal relations).


Predict a tree structure out of nodes with different features The Problem

Suppose we have a representation of a document's text layout as in the image below:

Here, each rectangle represents a chunk of text (one word, an expression or even part of a sentence).

Each rectangle has a set of features and connections to adjacent rectangles (the image doesn't show all connections, but they are present):

top/bot/lft/rgh sibling
top/bot/lft/rgh distance to sibling (in case of sibling presence)
margins alignment (for rectangles aligned in columns)
font size (continuous variable type)
font type (categorical)
all_caps (binary)
and some other

What I would like to achieve is to train an algorithm (ML? DNN? Graphs? something other?) that would structure these rectangles, would create a tree structure out of these individual nodes. For instance the above example would be structured as follows:

          ROOT
 __________|___
|          |   |
2          10  12
|_         |   |_________
| |        |   |         |
3 9        11  14        20...
|________      |__       |__
| | | | |      |  |      |  |
4 5 6 7 8      13 15...  19 21...
                            |  \
                            22 23...

My results so far

I spend some time (~8h already) searching for a solution, but couldn't find anything applicable to this case. I'm relatively new to ML and don't have much experience, however I think it can be solved using a supervised learning algorithm by feeding a training set consisting of documents with all these nodes with their features and annotated trees that should be generated as a result out of these nodes.

Some ideas I found while searching that might help (probably one of these is the solution, but I need some guidance on how to adjust in properly):

Using some sort of DNN. It would figure out the relationship between distances and other features of the nodes from the training set so that it would be able to predict a structure given only the nodes and their features. I didn't work with DNN and I don't know how to make it generate the structure instead of categorize which digit is shown in the image (I did some basic tutorials)
Creating/Training a PCFG (Probabilistic Context Free Grammar). This approach seems to be valid, as it used in generating parsed trees in NLP (those that structure a sentence in form of a tree) and I found some resources on it, however I don't understand how to use all the features that are available on my nodes in learning such a grammar? Also, I don't see how this approach could make use of the positioning data in 2D space and the distances between siblings, as it is one of the key features in structuring the nodes.
Probably some Graph or Tree related algorithms would solve my problem, but I don't know how to formulate the query that would help me find the solution.

A very similar question was asked on Computer Science SE site, but there are no answers and the OP doesn't respond.

I would really appreciate if you would provide a solution, or even a hint in what direction to move in order to solve this problem.

Don't use machine-learning for this - you probably don't have millions of training examples.

Instead, what you are looking for is a simple depth-first search.

Begin at the root, then define a priority list, e.g. below-left, below, below-right, in which order to proceed. Don't visit nodes twice, but when first visiting them mark them as seen, and add them to your tree instead.

Alternatively, define a priority pattern of how to define the parent node.  Construct a tree based on this single "best parent" relationship,, but make sure you don't create cycles.


Machine Learning: Classify Array of Numbers based on Patterns I have experimented with various regression/classifier libraries, they accept training input like "5 is bad" and "10 is good" so they can tell you 7 is bad and 8 is good.

Now imagine a more complicated example: I have heart rate data and there are certain heart rate patterns that are good and other patterns that are bad. In this case it's nonsense to just say "heart rate 120" is good or bad and so forth.
How do you tackle a problem like this? Are there any known ML/AI algorithms that can intelligently recognize patterns?

A good start could be a classifier that I can at least train with a whole array of numbers instead of a single number?

Actually you want to classify time series. Going from here you have basically two options:

Build features from this time series, i.e. RMS, Peak Values, etc.. and classify them with "classical" predictors like SVM or Random Forests.
Use neural networks, or more specifically use LSTM in order to feed directly your "array of numbers", which is a time series.


Light GBM Regressor, L1 & L2 Regularization and Feature Importances I want to know how L1 & L2 regularization works in Light GBM and how to interpret the feature importances.

Scenario is: I used LGBM Regressor with RandomizedSearchCV (cv=3, iterations=50) on a dataset of 400000 observations & 160 variables. In order to avoid overfitting/reguralize I provided below ranges for alpha/L1 & lambda/L2 parameters and the best values as per Randomized search are 1 & 0.5 respectively.

'reg_lambda': [0.5, 1, 3, 5, 10]
 'reg_alpha': [0.5, 1, 3, 5, 10]

Now my question is about: Feature importance values with optimized values of reg_lambda=1 & reg_alpha=0.5 are very different from that without providing any input for reg_lambda & alpha. The regularized model considers only top 5-6 features important and makes importance values of other features as good as zero (Refer images). Is that a normal behaviour of L1/L2 regularization in LGBM?

Further explaining the LGBM output with L1/L2: The top 5 important features are same in both the cases (with/without regularization), however importance values after top 2 features has been shrunk significantly by the L1/L2 regularized model and after top 5 features the regularized model makes importance values as good as zero  (Refer images of feature importance values in both cases).

Another related question I have is: How to interpret the importance values and when I run the LGBM model with Randomized search cv best parameters do I need to remove the features with low importance values & then run the model? OR should I run with all the features & the LGBM algorithm (with L1 & L2 regularization) will take care of low importance features and won't give them any weight or may be give minute weight when it makes predictions.

Any help will be highly appreciated.

Regards
Vikrant

With regularization, LightGBM "shrinks" features which are not "helpful". So it is in fact normal, that feature importance is quite different with/without regularization. You don't need to exclude any features since the purpose of shrinking is to use features according to their importance (this happens automatically).

In your case the top two features seem to have good explanatory power, so that they are used as "most important" features. Other features are less important and are therefore "shrunken" by the model.

You may also find that different features pop up as top of the list (the list may look different in general) when you run the model multiple times. This is because (if you don't fix a seed), the model will take different pathes to obtain a best fit (so the whole thing is not deterministic).

Overall you should get a better fit with regularization (otherwise there is little need for it).

I wonder if it makes sense to use both (l1 and l2)!? L1 (aka reg_alpha) can shrink features to zero while l2 (aka reg_lambda) does not. I usually only use one of the parameters. Unfortunately, the documentation does not provide too much details here.


RNN's with multiple features I have a bit of self taught knowledge working with Machine Learning algorithms (the basic Random Forest and Linear Regression type stuff). I decided to branch out and begin learning RNN's with Keras. When looking at most of the examples, which usually involve stock predictions, I haven't been able to find any basic examples of multiple features being implemented other than 1 column being the feature date and the other being the output. Is there a key fundamental thing I'm missing or something?

If anyone has an example I would greatly appreciate it.

Thanks!

Recurrent neural networks (RNNs) are designed to learn sequence data. As you guess, they can definitely take multiple features as input! Keras' RNNs take 2D inputs (T, F) of timesteps T and features F (I'm ignoring the batch dimension here).

However, you don't always need or want the intermediate timesteps, t = 1, 2 ... (T - 1). Therefore, Keras flexibly supports both modes. To have it output all T timesteps, pass return_sequences=True to your RNN (e.g., LSTM or GRU) at construction. If you only want the last timestep t = T, then use return_sequences=False (this is the default if you don't pass return_sequences to the constructor).

Below are examples of both of these modes.

Example 1: Learning the sequence

Here's a quick example of training a LSTM (type of RNN) which keeps the entire sequence around. In this example, each input data point has 2 timesteps, each with 3 features; the output data has 2 timesteps (because return_sequences=True), each with 4 data points (because that is the size I pass to LSTM).

import keras.layers as L
import keras.models as M

import numpy

# The inputs to the model.
# We will create two data points, just for the example.
data_x = numpy.array([
    # Datapoint 1
    [
        # Input features at timestep 1
        [1, 2, 3],
        # Input features at timestep 2
        [4, 5, 6]
    ],
    # Datapoint 2
    [
        # Features at timestep 1
        [7, 8, 9],
        # Features at timestep 2
        [10, 11, 12]
    ]
])

# The desired model outputs.
# We will create two data points, just for the example.
data_y = numpy.array([
    # Datapoint 1
    [
        # Target features at timestep 1
        [101, 102, 103, 104],
        # Target features at timestep 2
        [105, 106, 107, 108]
    ],
    # Datapoint 2
    [
        # Target features at timestep 1
        [201, 202, 203, 204],
        # Target features at timestep 2
        [205, 206, 207, 208]
    ]
])

# Each input data point has 2 timesteps, each with 3 features.
# So the input shape (excluding batch_size) is (2, 3), which
# matches the shape of each data point in data_x above.
model_input = L.Input(shape=(2, 3))

# This RNN will return timesteps with 4 features each.
# Because return_sequences=True, it will output 2 timesteps, each
# with 4 features. So the output shape (excluding batch size) is
# (2, 4), which matches the shape of each data point in data_y above.
model_output = L.LSTM(4, return_sequences=True)(model_input)

# Create the model.
model = M.Model(input=model_input, output=model_output)

# You need to pick appropriate loss/optimizers for your problem.
# I'm just using these to make the example compile.
model.compile('sgd', 'mean_squared_error')

# Train
model.fit(data_x, data_y)

Example 2: Learning the last timestep

If, on the other hand, you want to train an LSTM which only outputs the last timestep in the sequence, then you need to set return_sequences=False (or just remove it from the constructor entirely, since False is the default). And then your output data (data_y in the example above) needs to be rearranged, since you only need to supply the last timestep. So in this second example, each input data point still has 2 timesteps, each with 3 features. The output data, however, is just a single vector for each data point, because we have flattened everything down to a single timestep. Each of these output vectors still has 4 features, though (because that is the size I pass to LSTM).

import keras.layers as L
import keras.models as M

import numpy

# The inputs to the model.
# We will create two data points, just for the example.
data_x = numpy.array([
    # Datapoint 1
    [
        # Input features at timestep 1
        [1, 2, 3],
        # Input features at timestep 2
        [4, 5, 6]
    ],
    # Datapoint 2
    [
        # Features at timestep 1
        [7, 8, 9],
        # Features at timestep 2
        [10, 11, 12]
    ]
])

# The desired model outputs.
# We will create two data points, just for the example.
data_y = numpy.array([
    # Datapoint 1
    # Target features at timestep 2
    [105, 106, 107, 108],
    # Datapoint 2
    # Target features at timestep 2
    [205, 206, 207, 208]
])

# Each input data point has 2 timesteps, each with 3 features.
# So the input shape (excluding batch_size) is (2, 3), which
# matches the shape of each data point in data_x above.
model_input = L.Input(shape=(2, 3))

# This RNN will return timesteps with 4 features each.
# Because return_sequences=False, it will output 2 timesteps, each
# with 4 features. So the output shape (excluding batch size) is
# (2, 4), which matches the shape of each data point in data_y above.
model_output = L.LSTM(4, return_sequences=False)(model_input)

# Create the model.
model = M.Model(input=model_input, output=model_output)

# You need to pick appropriate loss/optimizers for your problem.
# I'm just using these to make the example compile.
model.compile('sgd', 'mean_squared_error')

# Train
model.fit(data_x, data_y)


Which variables matter most for prediction of another variable? I have a dataset and need to predict, out of 9 variables, which ones matter most to predict number 10.

I first tried using the selectKBestmethod from sklearn.feature_selection and it looks like 4,5,6 and 8 are best.

Then I tried to look at linear correlation using pearsonr between each of the variables from 1 to 9 and number 10, and I get 2,4,5,8 are most correlated.

However, when I used PCA from sklearn.decomposition and calculate the pca.explained_variance_ I get these values
[  2.13128046e+01   3.44315766e-01   3.26052258e-01   2.67148345e-01
   1.85871921e-01   1.55241540e-01   1.31994073e-01   9.34982387e-02
   1.03297667e-02]

Isn't it a problem that the first variable is so much higher than all the other ones? What does it mean?

Let's use the following notation : $(x_1, ...x_9)$ are the first 9 variables out of which you try to predict $y$, your tenth variable.

I will try to address what looks like a PCA misunderstanding, then give you some ways to predict which variables $ x_i $ matter most to predict $y$

Principal Component Analysis

The returned features you get from the PCA are not the original one!

The PCA transforms $(x_1, ..., x_9)$ into $(x'_1, ..., x'_9)$ where each $x'_i$ is a particular combination of the $(x_1, ..., x_9)$ such that for any $i, i'$, $x_i$ and $x_i'$ are linearly uncorrelated (ie orthogonal). Moreover, it is done such that each $x_i$ explains a certain part of the training dataset variance. In fact, most of the implementation sort the $x'_1, ..., x'_9$ such that the first one is the one explaining the variance the most, then the second, ...

Thus you end up having your first variable $x'_1$ explaining a lot your prediction $y$. But this $x'_i$ is a particular combinaison of $(x_1, ..., x_9)$.

Getting the feature importance

As to assert which variable $x_i$ explains $y$ the best, one might use different ways. Just to mention that the feature importance is not absolute ; it relies on the technique (or estimator) you use to address this question. Here is a non exhaustive list of possibilities :

The linear correlation is a good start
With the RandomForestClassifier (or RandomForestRegressor depending on $y$) of sklearn.ensemble, you can use feature_importances_ method to get which one is used the most at tree nodes. Note : it works with any decision tree estimator.
With standard linear or logistic regression (same with Lasso, Ridge, ...), you can check which variable has the higher coefficient (do not forget to normalize your input variables $(x_1, ..., x_9)$ )

I don't know how it is done using neural networks but would be glad if anyone has an hint.


How to generate and visualize results of all possible parameter values in Python? I have two parameters: demand, ticket_qty.
demand is an integer and can have a possible range of [100 - 200]
ticket_qty is also an integer and can have a possible range of [0 - 200]
I want to visualize how values for the above parameters affect the functions below:
price = demand - tickets_qty
revenue = price * ticket_qty
In Excel I created two matrixes (price and revenue) with demand as row indexes and ticket_qty as column indexes and simply applied the formula for each cell based on the rules above. However, seeing as this is a Kaggle microchallenge, I'd like to perform this using Python so as to then be able to plot and show optimal levels of price/revenue and do all my work inside the Jupyter notebook.
Questions:

How would I go about doing this in Python? pandas? numpy?
In statistics/data science, what is this "process" of mapping out possible values called? So as to facilitate further research

It's probably fastest to do this using numpy, in which you can define the possible ranges of your values using numpy.arange as follows:
import numpy as np

demand = np.arange(100, 201, 1) # integers from 100-200 inclusive
ticket_qty = np.arange(0, 201, 1) # integers from 0-200 inclusive

If you then want to calculate the possibilities using the different values you can use the .outer method of the different universal numpy functions (i.e. numpy.subtract to subtract values) to get easily get the results without looping over the arrays.


Using dhmm_em to form the hmm of mfccs' from song clips I was working on a project on music genre classification and decided to use an hmm to model my data. After extracting the mfccs' from multiple song clips I get multiple matrices of size 13 * x (depending on song length).

From what I understand hmm_train cannot be used for sequences of vectors and murphyk's toolbox might work. I tried modifying the demo file as follows to try a single song but I'm getting the error 'Attempted to access obsmat(:,61.6357); index must be a positive integer or logical..'

O = 5;
Q = 10;

v=wavread('We are the champions');
coeff=melfcc(v);
data=coeff';

% initial guess of parameters
prior1 = normalise(rand(Q,1));
transmat1 = mk_stochastic(rand(Q,Q));
obsmat1 = mk_stochastic(rand(Q,O));

% improve guess of parameters using EM
[LL, prior2, transmat2, obsmat2] = dhmm_em(data, prior1, transmat1, obsmat1, 'max_iter', 5);
LL

I haven't worked on this stuff before so I apologize in advance if my question comes across as really basic. It would be great if somebody could tell me where I'm going wrong. Also for multiple clips, should I simply append the matrices generated to the one I'm supplying above? Thanks.

Edit: It turns out that the above hmm can only be applied to discrete values which is possibly why I was getting errors. It seems I have to use another function for 'gaussian output' instead, mhmm_em. I'll post another question to figure out its working.

Edit: It turns out that the above hmm can only be applied to discrete
  values which is possibly why I was getting errors. It seems I have to
  use another function for 'gaussian output' instead, mhmm_em. I'll post
  another question to figure out its working.

In order to clarify this a bit.
The functions you were trying to use are indeed defined for discrete data represented by probability mass functions.

When working with continuous data and modeling them with an HMM, one needs to decide what probability distribution will best represent the data. When the data has an unbounded support, the Gaussian is often a good first choice, and most machine leaning toolboxes have built-in functions for the training of these HMMs. However, an HMM can be trained with other probability distributions such as the Dirichlet distribution (if your data is proportional), or the Student-t, Weibull distributions,...

Keep in mind that the distribution you choose is a modeling hypothesis that you do with respect to the data you have. If the hypothesis does not make sense, the results might not be accurate. Unfortunately, the use of these other distributions mostly rely on you to either, derive and code the needed equations (however, in some cases the equations are available in some scientific papers).


Machine learning or NLP approach to convert string about month ,year into dates I'm currently in the process of developing a program with the capability of converting human style of representing year into actual dates.
Example : last year last month into December 2018
string may be complete sentence like : what were you doing 5 years ago

it will gives 2014

The purpose is to evalute human style of represting year or date into actual date, i have created collection of this type of strings and matching them with regex.

I have read some machine learning but I'm not sure which algorithm suits this problem the best or if I should consider using NLP.

Does anyone have a suggestion of what algorithm to use or where I can find the necessary literature to solve my problem?

Thanks for any contribution!

I got my answer , NLTK is good to go for this problem.
You may use sutime with python wrapper :

Python wrapper for Stanford CoreNLP's SUTime

The usual approach in NLP is to collect a dataset required for training. Process that dataset so that the words in the dataset are converted into numbers.

One simple example of converting it into numbers is to make a large dictionary of words from the dataset and use the index of each word in the dictionary as the representing number


KNN Regression: Distance function and/or vector representation for datetime features Context: Trying to forecast some sort of consumption value (e.g. water) using datetime features and exogenous variables (like temperature).
Take some datetime features like week days (mon=1, tue=2, ..., sun=7) and months (jan=1, ..., dec=12).
A naive KNN regressor will judge that the distance between Sunday and Monday is 6, between December and January is 11, though it is in fact 1 in both cases.
Domains
hours = np.arange(1, 25)
days = np.arange(1, 8)
months = np.arange(1, 13)

days
>>> array([1, 2, 3, 4, 5, 6, 7])
type(days)
>>> numpy.ndarray

Function
A custom distance function is possible:
def distance(x, y, domain):
   direct = abs(x - y)
   round_trip = domain - direct
   return min(direct, round_trip)

Resulting in:
# weeks
distance(x=1, y=7, domain=7)
>>> 1
distance(x=4, y=2, domain=7)
>>> 2

# months
distance(x=1, y=11, domain=12)
>>> 2
distance(x=1, y=3, domain=12)
>>> 2

However, custom distance functions with Sci-Kit's KNeighborsRegressor make it slow, and I don't want to use it on other features, per se.
Coordinates
An alternative I was thinking of is using a tuple to represent coordinates in vector space, much like we represent the hours of the day on a round clock.
def to_coordinates(domain):
    """ Projects a linear range on the unit circle,
        by dividing the circumference (c) by the domain size,
        thus giving every point equal spacing.
    """
    # circumference
    c = np.pi * 2

    # equal spacing
    a = c / max(domain)

    # array of x and y
    return np.sin(a*domain), np.cos(a*domain)

Resulting in:
x, y = to_coordinates(days)

# figure
plt.figure(figsize=(8, 8), dpi=80)

# draw unit circle
t = np.linspace(0, np.pi*2, 100)
plt.plot(np.cos(t), np.sin(t), linewidth=1)

# add coordinates
plt.scatter(x, y);

Clearly, this gets me the symmetry I am looking for when computing the distance.
Question
Now what I cannot figure out is: What data type can I use to represent these vectors best, so that the knn regressor automatically calculates the distance? Perhaps an array of tuples; a 2d numpy array?
Attempt
It becomes problematic as soon as I try to mix coordinates with other variables. Currently, the most intuitive attempt raises an exception:
data = df.values

Where df is:

The target variable, for simple demonstration purposes, is the categorical domain variable days.
TypeError                                 Traceback (most recent call last)
TypeError: only size-1 arrays can be converted to Python scalars

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
<ipython-input-112-a34d184ab644> in <module>
      1 neigh = KNeighborsClassifier(n_neighbors=3)
----> 2 neigh.fit(data, days)

ValueError: setting an array element with a sequence.

I just want the algorithm to be able to process a new observation (a coordinate representing the day of the week and temperature) and find the closest matches. I am aware the coordinate is, of course, a direct representation of the target variable, and thus leaks the answer, but it's about enabling the math of the algorithm.
Thank you in advance.

I like your idea of converting to 2d (the unit circle), 2d numpy array would be the way to go here Specifically, try puting the days_x and days_y into separate columns if you take the unit circle approach.
An alternative idea - it looks like there's a 'precomputed' option for distance, which will let you use the distance you "really" want, and should not be slow, since there's no computation to be done.


Why don't all feature selection methods in sklearn allow specifying desired variance explained? Why don't all feature selection methods in sklearn allow specifying desired variance explained?

sklearn.decomposition.PCA does allow inputting a percentage of variance that one wants to be explained in place of n_components. However other methods such as SparsePCA expect the user to "know" how many n_components he/she wants to take.

Because PCA is just one of the feature selection methods, don't expect others to have a tractable computable formula to specify ahead of time the optimal number of new features.
I suggest to start reading on Representation learning, and what we mean by latent variables.


How to recognize product based on image using neural network? Company has many products in their offer (some about 100,000), some of these are very similar to each other. In database there is available only one image per product.

Company want to make possible to recognize product based on video camera and display its specification. Is it possible to train new model with this kind of data or build it using existing model?

Is it possible to train new model with this kind of data

Yes, you need Convolutional Neural Networks (CNN) for image classification. If you only have one image per product in your dataset, I suggest you to use a lot of image augmentation, a technique that is meant to artificially increase the size of an image dataset by applying combinations of distortions to image data.

  or build it using existing model?

You can re-train existing models. On the TensorFlow website you can find a huge list of pretrained CNN architectures that you can download ad use for your needs. It's a powerful alternative to training a new model from scratch.


r - How to determine the correlation between unordered categorical variables and individuals? I have a matrix with several unordered categorical variables. Each row represents a type of individual. Each column represents the number of times each type of individual was found to be in that particular condition.

Type    coal    cobalt  concrete    copper  gold
A       12      0       0           19      5
B       5       0       0           11      0
C       4       2       0           14      1
D       1       3       15          0       1
E       0       20      2           1       9

My question is very simple: I want to know if there is a correlation between the type of the individual (A, B or C) with a particular condition (copper, gold, etc).

Which test should I use? If possible, I would like to get the answer by using R.

Thanks!

If you have data sets $X_1,\cdots,X_n$ and $Y_1,\cdots,Y_n$, then you can compute their correlation with the following formula:

$$Cor(X,Y) = \frac{\sum (X_i-\bar{X})(Y_i-\bar{Y})}{\sqrt{\sum (X_i-\bar{X})^2\sum(Y_i-\bar{Y})^2}}$$

(where $\bar{X}$ denotes the average value of the $X_i$'s).  This is accomplished in $R$ with the following command:

cor(x,y)

That being said, it is unclear what two data sets you are trying to find the correlation for.  Finding the correlation between a type (A,B,C) and a condition (copper, gold, etc.) would not make any sense.  You could, however, find the correlation between two different types (A and B, for example), or between conditions (copper and gold).

Edit: I think you might want to do a test for independence between categorical variables...if this is the case then this is what you are looking for.


How can I plot a line for time series data with categorical intervals in R I am working with single time-series measurements that I want to plot for the time window of about 1 week.
This is the data I am working with.
This is my R script:
library(tidyverse)
library(ggplot2)

filesource <- "C:/ ... /testData.csv"
df <-read.csv(filesource, header = TRUE)

ggplot() +
  geom_line(data = df, aes(x = date, y = value, group = 1), color = "red") +
  ggtitle("Some Measure over Time") +
  xlab("Time") +
  ylab("Some Measure in %")

This produces this plot.

What I want is to have the individual unique weekdays show on the x-axis like this, as if I would plot the days as individual categories but only show the first one of each day. I cannot really hardcode this, because I am working with different participants, days and value amounts per day.
Expected Outcome:

So I created a new variable with the weekdays:
df$day <- weekdays(as.Date(df$date, '%d-%m-%Y'))

However, when I want to use this column as the x-axis variable, The days are not in the right order, and all values of one day obviously get plotted on top of each day:
geom_line(data = df, aes(x = day, y = value, group = 1), color = "red")

I have seen this being somewhat solved in python: Visualizing Time Series Data
However, I really want to use R and Markdown to create automated participant reports.
If this is easier to accomplish with another plotting function in R, I am open ears. I just like the customizability of ggplot.
I hope my example is clear. I guess this can be solved with the right ggplot() parameters and settings.
Does anyone have a solution for ending up with something more like the expected outcome montage?

Would something like this work? I simply add an extra column that indicates the row number (which is later used as the x-axis) to make sure all values are displayed as a new point instead of plotting on top of each other for the same day. I then specifiy the custom x ticks and labels by selecting the first row for each day and get the row number (which specifies where the ticks and labels have to be drawn) and the day name (which specifies what the labels should display).
library(readr)
library(ggplot2)

df <- read_csv("testData.csv") %>%
  mutate(
    date = as.Date(date, "%d-%m-%Y"),
    day = weekdays(date),
    row = row_number()
)
ticks <- df %>% group_by(day) %>% filter(row_number() == 1) %>% select(row)

ggplot() +
  geom_line(data = df, aes(x = row, y = value, group = 1), color = "red") +
  ggtitle("Some Measure over Time") +
  xlab("Time") +
  ylab("Some Measure in %") +
  scale_x_continuous(breaks=pull(ticks, row), labels=pull(ticks, day))


suggestion for good online source according to Syllabus Hi, are there some online courses e.g. some classes in Coursera?

I have difficulty in following the professor's teaching because I have a weak statistics background. I want to catch up by reading some online complementary resource!

Thank you!

A good deal of the topics, excluding the more advanced ones like autoencoders or manifold learning, can be learned in the excellent course of andrew ng https://www.coursera.org/learn/machine-learning
There's also https://www.coursera.org/specializations/deep-learning?utm_source=deeplearningai&utm_medium=institutions&utm_campaign=WebsiteCoursesDLSBottomButton that goes deeper into aspects of deep learning
For generative models I don't have specific recommendations...


How do GBM algorithms handle missing data? How do algorithms GBM algorithms, such as XGBoost or LightGBM handle NaN values?
I know that they learn how to replace NaN values with other values but my question is: How do they do it exactly?

LIGHTGBM will ignore missing values during a split, then allocate them to whichever side reduces the loss the most. Section 3.2 of this reference explains it.

There are some options you can set such as usemissing=false, which disables handling for missing values. You can also use the zeroas_missing option to change behavior. GitHub


How do I plot the predictions made by a LinearRegression model? I have made a linear regression using sklearn. I am wondering if there is a convenient way where I can plot the prediction versus a specified variable?

Alternatively, is there any other good way of visualising a linear regression model considering it involves n-dimensions?

Thanks for any help.

You cannot visualise the dimensions of an n-dimensional model all at once if n > 3. However, what you can do is plot the pairwise relationships between dimensions. You might want to look into the Seaborn library for visualisation tools.


How to implement custom loss function that has more parameters with XGBClassifier in scikit-learn? I have following problem with implementing custom loss function with scikit-learn:

I would like to implement Focal Loss as my objective function in XGBClassifier. However, I dont know how to pass additional arguments as a parameter(objective parameter):

 def focal_loss(y_pred, y_true, alpha=0.25, gamma=1):
  a,g = alpha, gamma
  def fl(x,t):
    p = 1/(1+np.exp(-x))
    return -( a*t + (1-a)*(1-t) ) * (( 1 - ( t*p + (1-t)*(1-p)) )**g) * ( t*np.log(p)+(1-t)*np.log(1-p) )
  partial_fl = lambda x: fl(x, y_true)
  grad = derivative(partial_fl, y_pred, n=1, dx=1e-6)
  hess = derivative(partial_fl, y_pred, n=2, dx=1e-6)
  return grad, hess

xgb = xgb.XGBClassifier(objective=focal_loss)

What should I do in following situation? Is there maybe ready version of Focal Loss ready to use? Thanks in advance.

def focal_loss(alpha, gamma):
    def custom_loss(y_pred, y_true):
        a,g = alpha, gamma
        def fl(x,t):
            p = 1/(1+np.exp(-x))
            return -( a*t + (1-a)*(1-t) ) * (( 1 - ( t*p + (1-t)*(1-p)) )**g) * ( t*np.log(p)+(1-t)*np.log(1-p) )
        partial_fl = lambda x: fl(x, y_true)
        grad = derivative(partial_fl, y_pred, n=1, dx=1e-6)
        hess = derivative(partial_fl, y_pred, n=2, dx=1e-6)
        return grad, hess
    return custom_loss

xgb = xgb.XGBClassifier(objective=focal_loss(alpha=0.25, gamma=1))

Using Python Closures!!


Comparing English word pronunciation complexity I'm trying to figure out a way to compute a score for the pronunciation of a given english word, so I can use that score to compare the pronunciation complexity between english words.

Eg: Given words apple and banana, determine which one is harder to pronounce based on that score.

Please share your thoughts.
Thanks.

I think first you need to define "pronunciation complexity." Is it a measure of the amount of time it takes the average L2 learner to correctly pronounce the word? Or, a measure of the diversity of phonemes in the word. Operationalizing your variable will help immensely in clearing up your goal.

Suppose you want to measure complexity by the diversity of phonemes in the word. You could then find a dataset of phonemes for each English (or other language) word. Then you could do something like, calculate the entropy of each phoneme set per word. Or maybe the Shannon diversity (although this might not perform well for sets with very low specie occurrences). You could also vectorize each word and perform some measure of similarity between words.

Of course, this is all hypothetical. Once you update your question with a clear goal in mind it will be easier to help you more concretely!


How to start building a statistical regression analysis model with multiple categorical/discrete input variables of high dimension in Python I'm fairly new to data science and ML. I have data of an item going through a release process. I have collected data on various variables such as "product category", "product line", "design country", "hour of day started" and I also have data on "total time" which is the time it took the item going through the entire process. In total I have 18 different input variables where each variable is either a categorical or a discrete number such as "hour of day started".

Design_cntry      Prod_category    prod_line   ...   time_minutes
   A                  A1             A11       ...     43.2
   B                  B1             A11       ...     20.1
   C                  E1             B11       ...     15.0
  ...                ...             ...       ...     ....

I want to build a statistical regression analysis model in python which outputs the probability of a statement. Say for instance P(time > 1000 min | product category = A, product line = B, ... ) and am wondering how to tackle this problem? Are there any general ways of doing this? Or good articles/literature on this topic anyone could recommend?

I only have non negative data, so maybe there are any good regression forms based on exponential distributions?

First, you have to pre-process your data. it includes encoding your categorical variables. You can either do it using pandas.get_dummies, or sklearn.preprocessing.OneHotEncoder in your pipeline. Based on the algorithm you want to use, you usually have to standardize your numerical variables. this can be done using any of sklearn.preprocessing methods such as StandardScaler.

If you want to do logistic regression, you have to categorize your output variable - time - to classes like ['less than 1000 min', 'more than 1000 min']. Otherwise, if you want to keep time as it is, you are doing a multiple regression.


Checking for skewness in data I have a data frame consisting of some continuous data features. I did a kde plot of the features using seaborn kdeplot functionality which gave me a plot as shown below :

How do I interpret this visualization in order to check for things like skew in the data points, etc.?


Evaluating machine learning model with missing features I am working on a credit risk binary classification problem. The classes are GoodPayers and BadPayers.  The training set has variables/features that contains:

DemoGraphics Data  such as - Age, Education, Loan Amount, Interest Rate
Behavioral Data such as - Payment in Month1, Payment in Month2, Payment in Month3, Payment delay in Month1, Payment delay in Month2.

The 10-fold cross-validation has 0.82 AUC on this set.

However, the unseen data just contains the 'Demographics Data' and does not have 'Behavioral Data' of Payment.  How do we deploy/test the model based on DemoGraphics Dataset only?

If less than 20-25% of behavioral data is missing, maybe you could try to impute missing data  using one of the following solutions :

Impute missing behavioral data using some business rule or by training a machine learning model with demographics data as input and behavioral data as output variable.
Impute missing data with feature mean/median.
Impute missing data by picking up random value in the feature distribution (hot-deck).

In case you have more than 20-25% missing data, it will be really hard to impute values. I think in this situation you should consider creating a new model such as :

The new model doesn't use behavioral data anymore.
The new model is based on a different train-val-test split in order to have behavioral data in each dataset.

If you can't create a new model neither impute missing data, I guess hot-deck would be the best option you have to avoid bad performance on unseen data.


External cluster evaluation for a varying number of cluster There are many external clustering indices like (Adjusted) mutual information, (Adjusted) Rand index, and many more. However, they are not very good at comparing clusterings where the number of clusters is not the same.

For example, if I take one cluster and split it into two new ones, a comparison would yield a low score.

To evaluate the clustering in my case, this operation should not affect the score a lot. Also according to [1], none of the compared cluster evaluation indices utilize optimal pairing, only greedy methods, to find the cluster matches (solve the linear assignment problem).

So my question is: Are there any methods to calculate a matching score between two clusterings where the number of clusters differ and clusters are not only paired but also optimally matched in a 1:n (or even n:m) way? The computation complexity doesn't matter to me very much.

Using an example with NMI:

>>> normalized_mutual_info_score([0, 0, 1, 1, 2, 2, 2, 2], [1, 1, 2, 2, 0, 0, 3, 3], average_method='arithmetic')
0.8571428571428573
>>> normalized_mutual_info_score([0, 0, 1, 1, 2, 2, 2, 2, 2, 2], [1, 1, 2, 2, 0, 0, 0, 3, 3, 3], average_method='arithmetic')
0.8204614780379502
>>> normalized_mutual_info_score([0, 0, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2], [1, 1, 2, 2, 0, 0, 0, 0, 3, 3, 3, 3], average_method='arithmetic')
0.7896900821428475

The scores get progressively worse for an increasingly larger split cluster.

[1]  Mohammad Rezaei, Pasi Fränti. Set Matching Measures for External Cluster Validity.

Most measures such as (Adjusted,Normalized) Rand index / normalized mutual information do perform an optimal n:m-like comparison by design.

They avoid having to make the assumption that cluster A in one clustering is cluster X in the other; instead they look at the resulting object pairs instead. Methods that rely on "set matching" are very uncommon because of this reason. It appears to mostly be  classification people that think you should be using the precision and recall of a 1:1 matching, because that is the measure they always use.

In the Rand index, Mutual Information, etc. moving a single element from a cluster into a small second cluster reduces the quality roughly by 1/n initially; and once the resulting cluster starts becoming larger even less than that (until eventually when most elements have moved, the index starts I proving again). That is not what I would call inappropriately worse, nor "not very good at comparing clusters of different sizes". I think you should provide examples for your claim.

I am not convinced by the Rezaei paper. On one hand, they seem to assume that behavior must be linear in their synthetic data change (Rand index would often be linear; it's the desirable adjustment for chance that produces the non-linearity, and his can we'll be used as an argument that the measure should not be linear). Second, they seem to require that every cluster must have equal weight, so increasing the size of a "perfectly found" cluster must not affect the outcome. Here, again, I beg to disagree. Cluster influence should depend on the number of objects in the cluster. From the "cluster size imbalance" plot my take is that once you add a lot of 1-element clusters, the proposed PSI measure starts looking much worse, because it cannot deal with such noise.


Random forest model gives same result for all test data, Next step? I am teaching myself some data science and have started a Kaggle project.
I have fitted a random forest classification model (using sci-kit learn) with a few millions rows of data. There are two possible outcomes for each row (0 or 1). When I run it against the test data, I get 0 for every row. This is a practical impossibility, but I am at a loss as to how to diagnose my model and how to move forward.
Is this an example of extreme overfitting ? Is it more likely a problem with my training data (or test data) that i am not seeing ?

Should I simply increase the # of estimators? Is it possible that I have misformatted the input files in a subtle way that doesn't cause an error ? I am at a loss as to how to move forward.

Some of the possibilities include the following:

1) The training data has class imbalance.

Solution:

Train the model using CV = 5 or 10;
Do a log transformation to make the target distribution more normal in nature;
Check if you can add a weight variable that can fix the distribution.

2) Testing data is a small sub sample. Since it's Kaggle, I believe the testing data will be standard in nature, so the problem might be in the process that is generating the submission script.

As a beginner, you should use any benchmark script available in the forum to start with, which shouldn't have such issues.


Interpretation of Autocorrelation plot I am trying to understand better how to read the autocorrelation plot here for a timeseries data.
I ran the following code and got the output as a chart show below.
from pandas.plotting import autocorrelation_plot
autocorrelation_plot(df("y"))

Here y is the dependent variable

Should I derive the following conclusions

There are no significant autocorrelations.
The data is random.
Most of the correlations (except for 2 lags) fall within 95% confidence limits
This timeseries is not worth forecasting

Please help me if my understanding is right ?

To address your points:

There are no significant autocorrelations
The correlation is low (~0.25), but there are significant autocorrelations.

The data is random & most of the correlations (except for 2 lags) fall within 95% confidence limits
The confidence intervals are used to show which autocorrelations are significant. As you rightly observed, a couple peaks jump out of this region and this tells us that these few correlations are statistically significant, the rest is random. This post may be helpful here.

This timeseries is not worth forecasting
As per the previous point, there are a couple of statistically significant weak correlations in this dataset. But they are not strong, so a periodicity based forecasting model probably wouldn't be very accurate.


When do we say that the dataset is not classifiable? I have many times analysed a dataset on which I could not really do any sort of classification. To see whether I can get a classifier I have usually used the following steps:

Generate box plots of label against numerical values.
Reduce the dimensionality to 2 or 3 to see if classes are separable, also tried LDA sometimes.
Forcefully try to fit SVMs and Random Forests and look at feature-importance to see if the features make any sense or not.
Try to change the balance of classes and techniques like under-sampling and over-sampling to check if class imbalance might be an issue.

There are many other approaches I can think of, but have not tried. Sometimes I know that these features are not good and not at all related to the label we are trying to predict. I then use that business intuition to end the exercise, concluding that we need better features or totally different labels.

My question is how does a Data Scientist report that the classification can not be done with these features. Is there any statistical way to report this or fitting the data in different algorithms first and looking at validation metric is the best option?

Take a sample element from one class and a sample element from the other class. Is it possible for these two elements to have the exact same feature vector? If that can ever happen then the two classes are not completely seperable using your current feature vectors (since the classification decision is based entirely in the feature vector for a given element).

On the other hand, if *every" element in one class has a corresponding element in the other class such that the two elements have the same feature vectors, then the two classes are indistinguishable using your current feature vectors.

Furthermore, if that condition holds for only some of your elements and not others, then you are somewhere in between, and you can use that as a basis to measure how well you can hope a classifier to perform using your current feature set.

All of these evaluations can be used to argue to varying degrees that you need to extract more features.


What to consider before learning a new language for data analysis I'm currently in the very early stages of preparing a new research-project (still at the funding-application stage), and expect that data-analysis and especially visualisation tools will play a role in this project.

In view of this I face the following dilemma: Should I learn Python to be able to use its extensive scientific libraries (Pandas, Numpy, Scipy, ...), or should I just dive into similar packages of a language I'm already acquainted with (Racket, or to a lesser extent Scala)?

(Ideally I would learn Python in parallel with using statistical libraries in Racket, but I'm not sure I'll have time for both)

I'm not looking for an answer to this dilemma, but rather for feedback on my different considerations:

My current position is as follows:

In favour of Python:

Extensively used libraries
Widely used (may be decisive in case of collaboration with others)
A lot of online material to start learning it
Conferences that are specifically dedicated to Scientific Computing with Python
Learning Python won't be a waste of time anyway

In favour of a language I already know:

It's a way to deepen my knowledge of one language rather than getting superficial knowledge of one more language (under the motto: you should at least know one language really well)
It is feasible. Both Racket and Scala have good mathematics and statistics libraries
I can start right away with learning what I need to know rather than first having to learn the basics

Two concrete questions:

What am I forgetting?
How big of a nuisance could the Python 2 vs 3 issue be?

Personally going to make a strong argument in favor of Python here. There are a large number of reasons for this, but I'm going to build on some of the points that other people have mentioned here:

Picking a single language: It's definitely possible to mix and match languages, picking d3 for your visualization needs, FORTRAN for your fast matrix multiplies, and python for all of your networking and scripting. You can do this down the line, but keeping your stack as simple as possible is a good move, especially early on.
Picking something bigger than you: You never want to be pushing up against the barriers of the language you want to use. This is a huge issue when it comes to languages like Julia and FORTRAN, which simply don't offer the full functionality of languages like python or R.
Pick Community: The one most difficult thing to find in any language is community. Python is the clear winner here. If you get stuck, you ask something on SO, and someone will answer in a matter of minutes, which is simply not the case for most other languages. If you're learning something in a vacuum you will simply learn much slower.

In terms of the minus points, I might actually push back on them.

Deepening your knowledge of one language is a decent idea, but knowing only one language, without having practice generalizing that knowledge to other languages is a good way to shoot yourself in the foot. I have changed my entire favored development stack three time over as many years, moving from MATLAB to Java to haskell to python. Learning to transfer your knowledge to another language is far more valuable than just knowing one.

As far as feasibility, this is something you're going to see again and again in any programming career. Turing completeness means you could technically do everything with HTML4 and CSS3, but you want to pick the right tool for the job. If you see the ideal tool and decide to leave it by the roadside you're going to find yourself slowed down wishing you had some of the tools you left behind.

A great example of that last point is trying to deploy R code. 'R''s networking capabilities are hugely lacking compared to python, and if you want to deploy a service, or use slightly off-the-beaten path packages, the fact that pip has an order of magnitude more packages than CRAN is a huge help.