What kind of plot will be helpful for total count group data ( data grouped by month and year )? I have a group data frame like this :

group_by_region_year_month=train_data.groupby(['Region Name (GO)','Year','Month'],as_index=False)

    region_sum=group_by_region_year_month.agg({"AMV": "sum"})

Here , AMV is my target variable in regression ( this is count of total vehicles).

I have data for each day from 2000-3-17 to 2017-10-11.

I want to show the variation of the count for each year and each month for each Region.

How best can I plot it?

I tried using seaborn.FacetGrid in this way:

# g=sns.FacetGrid(row="Region Name (GO)", col="AMV", data=region_sum)

But it is taking too much time .

Is there any better alternative(s) ?

Edit 1

I tried this :

sns.set(rc={'figure.figsize':(50,30)})

ax=sns.barplot(x='Region Name (GO)', y='AMV', hue='Year', data=train_data)
ax.set_xticklabels(ax.get_xticklabels(), rotation=90)

which gave me a very clumsy plot :

I did not use countplot as countplot would give me count of rows. But , I want sum of all values in a group.

Can I represent it in any better way ?

If you like using Seaborn to construct your plots, sns.countplot is probably the best for plotting data where you want to show the differences in value counts.

You can find the link to the documentation page here which might be helpful for you to construct it in the  particular way you want.

As an example, here is a version of a "multiple" count plot like you suggest. (directly from the Seaborn Documentation page)


User profiling based on multiple posts I currently have collected a dataset of different social media posts for each user with labels assigned to each user. I tried to use LSTM, and BERT for the text classification problem, So for each post, I try to predict the label(for example age). This is not sufficient because you need all of the information contained in the sum of posts to determine the user's age for example.
My first thought was to concatenate all of the posts for a single user but since I am currently using BERT which has a maximum sequence length of 512 it wouldn't work. My second idea was to use a text summary and combine them in one vector and hope it doesn't pass the maximum length limit.

Do you have any suggestions for a possible solution? I would assume this problem has been dealt with in the scientific literature and I would be thankful if anyone could point me in the right direction.

You might want to have a look in XLNet.
In XLNet you can input a sentence, then take the state and inject it into a new run if the XLNet with a second input sentence, inject it into a third one with a third sentence and so on.
This, in essence, allows you to process text of virtually unlimited length.
Please see also this.


How do I train Xgboost classifier for ECG Signal data? I am testing https://www.physionet.org/challenge/2017/sources/ submission.

I like one of the submission code, which use Xgboost to train the classifier. Training data is in .mat file which can be converted to csv format.

In below code, I have pretrained model xgb.bin, using which I can test any input signal. But I Want to train the model using different data and create my own training model.

Here is the code which predicts class name for given input ecg file

def predict(data):

    #data = io.loadmat(path)['val'][0]

    from numpy import genfromtxt
    data = genfromtxt('testdata/val.csv', delimiter=',')

    features_noise = np.zeros((5, ))

    snr, rr_num, var, fr, fr2 = find_noise_features(data)
    features_noise[0] = snr
    features_noise[1] = rr_num
    features_noise[2] = var
    features_noise[3] = fr
    features_noise[4] = fr2
    features = extract_basic_features(data, 30000)
    features = np.hstack((features, features_noise.reshape(1, -1)))

    mean_ = np.array([15.96300284066109753667, 0.00412371298595770857, 38811.34497233365254942328,
                      0.48050717744965593115, 0.14397582347542958736])
    scale_ = np.array([4.22917401559752281770, 0.00093664880988427878, 62350.76443798459513345733,
                       0.15396567666240373873, 0.07085474966801086349])
    features_noise -= mean_
    features_noise /= scale_

    prediction = 0
    if features_noise[0] < -2.9:
        prediction = 3
    if features_noise[2] > 6.0:
        prediction = 3
    if features_noise[3] > 3.0:
        prediction = 3
    if features_noise[4] < -2.0:
        prediction = 3

    bst = xgb.Booster({'nthread': 4})
    bst.load_model("xgb.bin")

    dfeatures = xgb.DMatrix(features)
    prediction_prob = bst.predict(dfeatures)
    prediction = np.argmax(prediction_prob)

    return prediction

def run(data):
    prediction = predict(data)
    print(prediction)

I want to train this classifer using my own dataset. Above model is trained on https://www.physionet.org/challenge/2017/training2017.zip dataset.

Looking at above code, do you have any clue how can I train the model using Xgboost.

Here is my thought steps

Calculate features for all training signal in zip file
Calculate feature_noise as it is in above code for all signals
Calculate dfeatures features for each signals (Should I calculate it for each signals or for all signal together)
Train dfeatures for all signals using Xgboost (How?)
Store the xgboost model

Is this correct steps to do?

I might be a little bit late to the party, but I did exactly what you need in the follow-up challenge of CinC2017, where our algorithm gained the 2nd best score on the hidden test set.
Our code is available at https://github.com/martinkropf/ecg-classification
Paper is available at https://iopscience.iop.org/article/10.1088/1361-6579/aae13e


What if there is huge difference between train data and test data? I trained a model which does well on unseen data,but after deployed on production the data I got is very different,like the highest values in  train & test data is ~23,but the data I got from production have values which is ~80,and there is just 1 feature and it is regression problem.
Can anyone please recommend what can I do to tackle this problems.

To me it seems like you are training on the wrong data. The results of your algorithms will only be as good as the data you train it on. First question you should ask yourself is, if it is possible to transform or update your dataset to make it more realistic. Without deeper insight into the nature of the problem, it is, unfortunately, very hard to help in this regard. Still, in my honest opinion, this will do for the cleanest solution.

If you can not update your training data to represent the real-world distribution, your problem is quite similar to trying to transfer simulation data to real-world applications, like often seen in robotics. Usually, the authors try to learn the transformation between the simulation and real-world data explicitly like shown in this nice paper about optimization. Briefly summarized, the authors model the data by two Gaussian Processes (GPs), one resembling simulation data, whereas the other is the transformation I mentioned above. Thus, your real world data would result as the sum of the two kernels from the GPs. I am sure a similar approach is applicable for your problem, even without using GPs as you (most likely) are not interested in Bayesian Optimization.

Still, if you want go for this method, you will also need to gather some more data from your real distribution, just not a whole new dataset.


Is it possible to change the sequence of convolution layer and pooling layer? Many CNN architectures sequentially consist of convolution layers and pooling layers. I just wonder if it is possible to change the sequence of convolution layers and pooling layers. If not, could you explain the reasons why a convolution layer precedes to a pooling layer?

In crude terms, a single convolution layer tries to capture to what degree certain feature is present in given region of figure. Max pooling layer is a downsampling strategy: instead of capturing presence of certain feature to a very fine-grained degree, the layer downsamples to certain classes like almost present, almost not present. (In crude terms because what exactly each layer learns is highly specific to dataset and network architecture used).

If you think of convolution as signal extraction and max pooling as a downsampling/smoothing strategy, it makes sense to use smoothing after signal extraction and not before.


scikit-learn n_jobs parameter on CPU usage & memory In most estimators on scikit-learn, there is an n_jobs parameter in fit/predict methods for creating parallel jobs using joblib.
I noticed that setting it to -1 creates just 1 Python process and maxes out the cores, causing CPU usage to hit 2500 % on top.
This is quite different from setting it to some positive integer >1, which creates multiple Python processes at ~100 % usage.

How does setting it affects CPU & core usage on a multi-CPU Linux server?
(e.g. if n_jobs=8 then are 8 CPUs fully locked up or do the CPUs still reserve some cores for other tasks/processes?)

Additionally, I do get MemoryError occasionally when setting n_jobs=-1 for large datasets.
However, the memory usage usually hovers at around 30-40 % for the single Python process.
How is the data & memory being managed/copied depending on the value of n_jobs?

I can imagine a value of -1 consumes all available resources as and when they become available. Depending on which function you are talking about, it seems data is copied for each of the jobs, which can lead to memory problems if the dataset is large enough. Here is a snippet of information from the docstring of GridSearchCV:

If `n_jobs` was set to a value higher than one, the data is copied for each
point in the grid (and not `n_jobs` times). This is done for efficiency
reasons if individual jobs take very little time, but may raise errors if
the dataset is large and not enough memory is available.  A workaround in
this case is to set `pre_dispatch`. Then, the memory is copied only
`pre_dispatch` many times. A reasonable value for `pre_dispatch` is `2 *
n_jobs`.

So it might be a good idea to use pre_dispatch to put an upper limit on your memory consumption.

Otherwise, why are you setting it to -1? You should just set it to the number of physical cores on your machine, or maybe 2 times that number, if the task can be multi-threaded.

EDIT:

It seems setting n_jobs=-1 does indeed just select all physical cores and maximises their usage. Have a look at the comments in this answer on StackOverflow.

If you have not set pre_dispatch, it will of course try to copy a lot. This is why you run out of memory. If you have 4 cores there will be, by default, 8 copies of the dataset made (as described above in the quote).

Here is another thread, which looks more at the performance side


Data science / machine learning books for mathematicians I have found other requests for references here. In particular in:
Where to start, which books
and
Books about the "Science" in Data Science?

I have given a glance to:

Artificial Intelligence: A Modern Approach (Russel & Norvig)
Machine Learning: The Art and Science of Algorithms that Make Sense of Data (Flach)
Learning From Data (Abu-Mostafa et al.)
Introduction to Statistical Learning (James et al.)
Elements of Statistical Learning (Hastie et al.)
Pattern Recognition and Machine Learning (Bishop)

Now it is difficult to evaluate if they would fit my needs because only a few pages are generally available online. However my first impression is that they do not. In the appendices of Artificial Intelligence: A Modern Approach I can read:

  Mathematicians define a vector as a member of a vector space, but we will use a more concrete definition: a vector is an ordered sequence of values.

This is exactly the kind of approach I am not looking for.

I'm looking for a book which assumes the reader has a good understanding in set theory, abstract algebra, measure and probability theory, statistics, topology, graph theory, complexity theory, etc and a preference for formal and axiomatic explanations rather than lenghty and so-called "intuitive" approaches based on basic mathematical objects and examples. Furthermore I don't want something that looks like a recipe book from the very beginning. I want a book that formalizes the abstract and common shape of all data science methods as well as their common aim first. Only after that it can start to explain the different categories by explicitely stating which further hypotheses each category is assuming and which cases/problems/domains they are known to handle efficiently or not.

At last, to be clear, I have no problem with being shown concrete examples and their treatment via a specific programming language for example. I just want this to come second as an illustration for the conceptual explanation, not as a substitute.

First of all, data only comes in so many forms that it might make sense to stick to a more "concrete definition". Data Science is necessarily practical. But here are a few other books with a more theoretical grounding. Others will certainly know many more...

A probabilistic theory of pattern recognition by Devroye, Györfi, Lugosi
An Introduction to Computational Learning Theory by Kearns
Information theory, inference and learning algorithms by MacKay
Machine learning: a probabilistic perspective by Murphy
Understanding machine learning by Shalev-Shwartz, Ben-David

However, research in machine learning is mostly found in journals and papers. It will be hard to find one or a few books that cover everything you want to know.


Is (nearly) all data separable? Suppose I have some data set with two classes. I could draw a decision boundary around each data point belonging to one of these classes, and hence, separate the data, like so:

Where the red lines are the decision boundaries around the data points belonging to the star class.

Obviously this model overfits really badly, but nevertheless, have I not shown that this data set is separable?

I ask because in an exercise book, a question asks "Is the above data set separable? If it is, is it linearly separable or non-linearly separable?"

I would say "Yes it is separable, but non-linearly separable."

No answers are provided, so I'm not sure, but I think my logic seems reasonable.

The only exception I see is when two data points belong to different classes, but have identical features. For instance, if one of the stars in the figure above was perfectly overlapping one of the circles. I suppose this is quite rare in practise. Hence, I ask, is nearly all data separable?

TL;DR

Yes, with overfitting all data becomes (non-linearly) separable (as long as the points don't precisely overlap).

Explanation

The problem with your argument is that you are using circles on a 2D plane, which is very difficult to learn. However, I think your argument can be made stronger with a decision-tree.

(0.2, 3.1)? --> yes -> star
            \-> no  -> (1.2, 4.5)? --> yes -> circle
                                   \-> no  -> (x1, x2)? --> yes ...
                                                        \-> no  ...

Decision trees are well accepted models, but note that they are non-linear models. With that, it is easy to argue that all data becomes separable.

However, the issue lies with overfitting. Because models like that become unstable with data points that were unseen before. So, just because the training data is separable, it doesn't mean that the models generated from it become any useful.


Model-agnostic variable importance metric I use genetic / evolutionary algorithms in python's TPOT package to find the overall best model (GBM, RF, SVM, elastic net, etc) and its tuning parameters. Now I need a way to measure each variable's contribution to the chosen model's predictive performance. How can I do this in a model-agonistic way?

My current approach is to retrain the best model architecture after holding out each of the variables. For example, if my variables are [a,b,c] I'll retrain on [a,b], [a,c],  and [b,c]. I define the removed variable associated with the worst performing model as the most important variable and I define the variable's predictive contribution as the decrease in predictive performance. I measure all variable's predictive performance this way. Is there anything obviously wrong with this approach? Is there a better approach? I'm familiar with variable importance in decision trees, and p-values in linear models, but I need a model agnostic approach.

Have you looked at the permutation importance approach in the eli5 package?

The idea is that instead of retraining the model without the feature, which is computationally expensive, they replace each feature in turn with random noise in the test set. To get random noise that is drawn from the same distribution as the original feature, they just randomly shuffle that feature.

Note that, as with feature importance in decision trees, this measure is biased against categorical variables with low cardinality.


Predicting results of tennis matches based on historical data I am writing my Bachelor thesis in Python about predicting results of tennis matches based on historical data. I have started from Logistic Regression but my model isn't efficient. If you could look at it and tell me, what should I change. I am just getting started and I am not really sure if I understand these methods. I am using data from this repository https://github.com/JeffSackmann/tennis_atp, but i changed it a bit for my needs https://drive.google.com/file/d/1w8weQxhsMBvYwrDPVTuiM3diFxtsXaky/view?usp=sharing.
Performance of this model isn't really good and I would like to improve it.
I will deal with missing data, by changing it to mean, median or mode and check which one is best.
import numpy as np
import pandas as pd
import random
from sklearn import preprocessing
from sklearn.linear_model import LogisticRegression

def df_mean(frame):
   index = np.where(frame.columns.values == 'minutes')[0][0]
   columns = frame.columns[index::]
   for i in columns:
      frame[i].replace(np.NaN, frame[i].mean(), inplace=True)
   return frame

def df_median(frame):
   index = np.where(frame.columns.values == 'minutes')[0][0]
   columns = frame.columns[index::]
   for i in columns:
      frame[i].replace(np.NaN, frame[i].median(), inplace=True)
   return frame

def df_mode(frame):
   index = np.where(frame.columns.values == 'minutes')[0][0]
   columns = frame.columns[index::]
   for i in columns:
      frame[i].replace(np.NaN, frame[i].mode(), inplace=True)
   return frame

For the first time i chose mean
df = df_mean(pd.read_excel('matches 00-19.xlsx')).dropna() df.index = range(61475)

frame = df[['winner_name', 'loser_name', 'winner_name']]
frame.columns = ["Player1", "Player2", "Winner"]
X = frame[["Player1", "Player2"]]
y = frame[['Winner']]

In the data I had, in the dataframe the winner was always on the left and the loser on the right so I had to mix it.
for i in range(X.shape[0]):
   rand = random.getrandbits(1)
   if rand == True:
      X.iloc[i][0], X.iloc[i][1] = X.iloc[i][1], X.iloc[i][0]

y1 = pd.DataFrame(data=np.zeros(shape= (len(y),1)), columns= ['Winner'], dtype= np.int64)
for i in y.index:
   if y['Winner'][i] == X['Player2'][i]:
      y1['Winner'][i] = 1

y = y1

I had to change names of the players to numbers so I made a set of all the players and than made a dictionary of it. I know I could use 'winner_id' but those numbers are too big.
s1 = set(frame['Player1'])
s2 = set(frame['Player2'])
set_merged = sorted(s1.union(s2))

d1 = {}
for i in range(len(set_merged)):
   d1[set_merged[i]] = i

keys = list(d1.keys()) X2 = pd.DataFrame(data=np.zeros(shape= (len(X),2)),
columns= ['Player1','Player2'], dtype= np.int64)
for i in X['Player1'].index:
   if X['Player1'][i] in keys:
      X2['Player1'][i] = d1[X['Player1'][i]]

for i in X['Player2'].index:
   if X['Player2'][i] in keys:
      X2['Player2'][i] = d1[X['Player2'][i]]
X = X2

X_train = X[:49180]
y_train = np.ravel(y[:49180])

X_test = X[49180:]
y_test = np.ravel(y[49180:])

model = LogisticRegression()
model.fit(X_train, y_train)
model.score(X_test,y_test)

I think your current under-performance is a data problem. The dataset in your google sheet seems like it contains far too few features that you could conceivably use to predict whether someone would win a tennis match.
You should consider some creative feature-engineering options. Stuff like (but not limited to):

how many wins or loses in a row has this tennis player had
what is the weather during the game
how many years of professional tennis have they played
how many tournaments have they won (somehow encode their previous accolades)

Additionally - you might want to re-structure your data. The schema of having one player on the left and the other on the right of the same row will always make the machine learn that there's a distinction between right and left, when in fact this is just a bias you are introducing via featurizing.
You could consider a system whereby you double-up your data and put each match where player 1 is on the left and then the right. This could conceivably force your estimator to consider the match from both sides.
As for the estimator consider looking into some ensemble methods like gradient boosting classifier, or random forest classifier instead of logistic regression.


How do I predict survival curves using xgboost? The xgboost package enables survival modeling using parameter arguments: objective = "survival:cox" and eval_metric = "cox-nloglik".

The predict method for the resulting model only outputs risk scores (same as type = "risk" in the survival::coxph function in r).

How do I use xgboost to predict entire survival curves?

The proportional hazard model assumes hazard rates of the form: $h(t|X) = h_0(t) \cdot risk(X)$ where usually $risk(X) = exp(X\beta)$. The xgboost predict method returns $risk(X)$ only. What we can do is use the survival::basehaz function to find $h_0(t)$.

Problem is it's not "calibrated" to the actual baseline hazard rate computed in xgboost. What we can do is find some constant $C$ that minimizes the ibrier score between the sample observed death/censorship times and $h_0(t) \cdot risk(X) \cdot C$.

I've implemented this approach in a tiny R package I've written.


Choosing a model for input: categorised, weighted sequence, output: binary variable What would be an appropriate model for predicting a binary target variable, given a weighted sequence?
Sequences will be reasonably short, typically between ~ 1 and 5 elements.
Illustrated example
Say I have the following categories: A, B, C, D.
Each category can have a weight between zero and one.
Example sequences:

A (0.33), B (0.71), C (0.0), D (0.95)   => 1
C (0.21), A (0.67) => 0
B (0), D (1) => 1

One option is a model comparison approach. Start with simpler models then progressively trying more complex models. Along the way, checking if the additional complexity is resulting in increased predictive ability.

If you take a simplifying representation with a bag-of-words model, you can fit a Naive Bayes classifier.

If you make the Markov assumption, you'll only need to look at the last element in the sequence. Then it can be modeled as a simple Probabilistic Graphical Model (PGM).

Then start relaxing the Markov assumption by looking back at progressively more time steps.

Recurrent Neural Network (RNN) might work if you have thousands of labeled data points and the relationship is very complex.

Long Short Term Memory (LSTM) does not make much sense because the sequences are so short. LSTMs work better RNNs when the relevant information could possible be many time steps back.

My hypothesis is that a Naive Bayes classifier will be a good-enough model because there are only 4 features and the sequences are so short they can be ignored.


How to use deep learning to add local (e.g. repairing) transformations to images? I want to train a neural network that removes scratches from pictures. I chose a GAN architecture with a generator (G) and a discriminator (D) and two sets of images scratchy and non-scratchy, with similar motives. G in my setting uses mainly convolutional and deconvolutional layers with ReLU. As input it uses the scratchy images. The D discriminates between the output of G and the non-scratchy images.

For example the generator would perform the following transformations:

(128, 128, 3) > (128, 128, 128) > (128, 128, 3)

Where the tuples contain (width, height, channels). Input and output need to have the same format.

But in order to get output that have same global structure as the inputs (i.e. streets, houses etc) it seem I have to use filter-sizes and strides of 1 and basically pass the complete pictures through the network.

However, the features I am looking at are rather small and local. It should be sufficient to use filters of up to 20 pixels in size for the convolutions. And then then apply the network to all parts of the input picture.

What would be a good generator architecture for such a task? Would you agree with the structure of the generator or would you expect different designs to perform better on local changes?

Here is the code for the generator that I use implemented in tensorflow.

def generator(x, batch_size, reuse=False):
    with tf.variable_scope('generator') as scope:
        if (reuse):
            tf.get_variable_scope().reuse_variables()

        s = 1
        f = 1

        assert ((WIDTH + f - 1) / s) % 1 == 0

        keep_prob = 0.5

        n_ch1 = 32
        w = init_weights('g_wc1', [f, f, CHANNELS, n_ch1])
        b = init_bias('g_bc1', [n_ch1])
        h = conv2d(x, w, s, b)
        h = bn(h, 'g_bn1')
        h = tf.nn.relu(h)
        h = tf.nn.dropout(h, keep_prob)
        h1 = h

        n_ch2 = 128
        w = init_weights('g_wc2', [f, f, n_ch1, n_ch2])
        b = init_bias('g_bc2', [n_ch2])
        h = conv2d(h, w, s, b)
        h = bn(h, "g_bn2")
        h = tf.nn.relu(h)
        h = tf.nn.dropout(h, keep_prob)
        h2 = h

        n_ch3 = 256
        w = init_weights('g_wc3', [f, f, n_ch2, n_ch3])
        b = init_bias('g_bc3', [n_ch3])
        h = conv2d(h, w, s, b)
        h = bn(h, "g_bn3")
        h = tf.nn.relu(h)
        h = tf.nn.dropout(h, keep_prob)

        output_shape = [batch_size, HEIGHT//s//s, WIDTH//s//s, n_ch2]
        w = init_weights('g_wdc3', [f, f, n_ch2, n_ch3])
        b = init_bias('g_bdc3', [n_ch2])
        h = deconv2d(h, w, s, b, output_shape)
        h = bn(h, "g_bnd3")
        h = tf.nn.relu(h)
        h = h + h2

        output_shape = [batch_size, HEIGHT//s, WIDTH//s, n_ch1]
        w = init_weights('g_wdc2', [f, f, n_ch1, n_ch2])
        b = init_bias('g_bdc2', [n_ch1])
        h = deconv2d(h, w, s, b, output_shape)
        h = bn(h, "g_bnd2")
        h = tf.nn.relu(h)
        h = h + h1

        output_shape = [batch_size, HEIGHT, WIDTH, CHANNELS]
        w = init_weights('g_wdc1', [f, f, CHANNELS, n_ch1])
        b = init_bias('g_bdc1', [CHANNELS])
        h = deconv2d(h, w, s, b, output_shape)

    return tf.nn.sigmoid(h+x)

When you use a stride s>1 you will get an hourglass, where the layers get smaller but deeper. The depth is independently controlled by the n_ch variables.

BTW, I am running this on a Google colab notebook. Its amazing to have such an engine for free and be able to experiment with deep learning! Fantastic!

There is a paper called Spatial Transformer Networks written by Max Jaderberg et al. What it does is trying to find the canonical shape of its input by reducing transformations, like translation and rotation, or even diminishing the distortion of the inputs. It introduces a module which helps convolutional network to be spatial invariant. One of the significant achievements of this module is that it tries to enhance distorted inputs. Take a look at here.

I edit the answer because of the request of one of our friends. First I quote something from the popular book written by Pr. Gonzalez, I hope nothing goes wrong with copyright. Then I suggest my recommendation.

Figure 2.40, the one that I've attached, shows an example of the steps in Fig. 2.39. In this case, the transform used was the Fourier transform, which we mention briefly later in this section and discuss in detail in Chapter 4. Figure 2.40(a) is an image corrupted by sinusoidal interference and Fig. 2.40(b) is the magnitude of its Fourier transform, which is the output of the first stage in Fig. 2.39. As you will learn in Chapter 4, sinusoidal interference in the spatial domain appears as bright bursts of intensity in the transform domain. In this case, the bursts are in a circular pattern that can be seen in Fig. 2.40(b). Figure 2.40(c) shows a mask
image (called a filter) with white and black representing 1 and 0, respectively.
For this example, the operation in the second box of Fig. 2.39 is to multiply the mask by the transform, thus eliminating the bursts responsible for the interference. Figure 2.40(d) shows the final result, obtained by computing the inverse of the modified transform. The interference is no longer visible, and important detail is quite clear. In fact, you can even see the fiducial marks (faint crosses) that are used for image alignment.

Okay! After those quotes, I refer to my suggestion. In the ST network, the authors have claimed that their differentiable module can learn different kinds of transformations other than affine transformation. The point about that is that we usually apply two kinds of transformations. One transformation is applied to the intensity of the image, which is popular by the means of filters. The other one is called image warping where we change the position of intensities and the intensity values do not change unless they locate between the discrete grids of image entries also called pixels, picture element. Spatial transformers are good for the second task but they also can be used for the first task. There are studies about using CNNs for evaluating the images in their frequency domain and not in the spatial domain. You can employ these differentiable modules in those nets.

Finally about your specific task, what I'm seeing in your pictures, your noise has a same behaviour. I guess if that is true for all cases, your task is not learning and can be solved using usual image processing techniques.


Stock Price Data Manipulation for LSTM I am trying to construct a machine learning model that predicts the difference in price from tomorrow to the day after tomorrow, using yesterday's OHLCV (open, high, low, close, volume).

My models (LSTM and DNNs), however, give me very poor outputs. I think it is because the data needs to be reformatted in some way, shape, or form (i.e. normalized, log-pct change, etc.)

What is the best way to manipulate the input? Should I use a sci-kit 'scaler', or any other method to normalize the data? If so, what range should the scaler be within?

What about the output? If I want the output to be linear (continuous, non-softmax value), should I manipulate the output data as well to be scaled, and then unscale the predicted output to match the original labels? Does it matter?

I used a MinMax scaler in the range between (0, 1) applied to the closing price of S&P500. The RNN consisted of a single LSTM layer with a lookback window of 10 days to predict the next day's closing price. The following figure shows RNN prediction of the next day's closing price (in red).

For code used to generate the figure, have a look at the following ipython notebook.


Newbie in ML - how to We have a data set of n variables (profile attributes) and want to feed through a model, and classify into M buckets (functionally signifying some action to be performed) .  Which MLmodel/ algorithm is best suited for this. Can someone point me some code.

I'm writing an example code in python:

import sklearn
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

X, y = whatever your data (X are the input variables, y are the output)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

lr = LinearRegression().fit(X_train, y_train)

print("Training set accuracy: {:.2f}".format(lr.score(X_train, y_train))
print("Test set accuracy: {:.2f}".format(lr.score(X_test, y_test))

Imported necessary packages.
Defined your input (X) and output data (y).
Shuffled the data into the training set (75%) and the test set (25%).
Defined the linear regression model and fitted the training data.
Printed training and test set accuracies.

Hope that helps!


How to deal with highly skewed (on counts) dependent variables? I am working on a binary classification problem and the dataset consists of several variables which are count variables. For example, how many times a customer defaulted on a broadband bill payment in the last 3 months.

The problem is, these features are highly skewed. This is how the distribution for the above variable looks like:

0.0     98.175855
1.0      1.275902
2.0      0.348707
3.0      0.199535

This is due to the nature of the event being evaluated during the construction of the feature. For example, the majority of the population may not have defaulted hence the value is 0 for 98% of them.

There are several such variables and they are measuring important events. Therefore I cannot remove these variables. However, I am afraid the model would not be learning anything from these features as there is very less information in them.

Questions:

Am I right in assuming these features will not be useful to the model in the current state?
How can these features be handled?

What models will you employ?

If you are working with deep learning, you can train it using mini batch gradient descent, and "artificially" build each mini batch so that the classes you want to predict appear with a more balanced frequency.


Dealing with missing data in SVD I am a newbie to machine learning and I am trying to apply the SVD on the movielens dataset for movie recommendation. I have a movie-user matrix where the row is the user id, the column is the movie id and the value is the rating.

Now, I would like to perform normalization on the movie-user matrix (subtract the data by users ratings mean). Then pass the normalized matrix to Scipy.sparse svds as follow:

from scipy.sparse.linalg import svds
U, sigma, Vt = svds(R_demeaned, k = 50)

Now, I have the 2 method to do it:

Method 1.) Fill all the missing rating with 0 first, then calculate the user rating mean for normalization.

The predicted rating dataframe for method 1 by using svd is:

Method 2.) Calculate the user rating mean first and do the normalization, then replace the missing rating with 0.

The predicted rating dataframe for method 2 by using svd is:

I would like to know which method is better or there are other methods to do it. So far as I can observe from method 2, the predicted ratings for a user are quite similar. For example, user A may get 4.XX ratings for all movies. Meanwhile in method 1, there are more variation. I would like to know if there are something wrong.

I would recommend trying both methods to see which works better.

However, in my opinion, you should fill first and then normalise it. Imagine if you normalise it first, you could end up with some values very close to 0. So when you fill in the missing values with 0, you are inherently saying that these missing values had the same initial value as those that were normalised to 0, which would be incorrect.


Why should the data be shuffled for machine learning tasks In machine learning tasks it is common to shuffle data and normalize it. The purpose of normalization is clear (for having same range of feature values). But, after struggling a lot, I did not find any valuable reason for shuffling data.

I have read this post here discussing when we need to shuffle data, but it is not obvious why we should shuffle the data. Furthermore, I have frequently seen in algorithms such as Adam or SGD where we need batch gradient descent (data should be separated to mini-batches and batch size has to be specified). It is vital according to this post to shuffle data for each epoch to have different data for each batch. So, perhaps the data is shuffled and more importantly changed.

Why do we do this?

Shuffling data serves the purpose of reducing variance and making sure that models remain general and overfit less.

The obvious case where you'd shuffle your data is if your data is sorted by their class/target. Here, you will want to shuffle to make sure that your training/test/validation sets are representative of the overall distribution of the data.

For batch gradient descent, the same logic applies. The idea behind batch gradient descent is that by calculating the gradient on a single batch, you will usually get a fairly good estimate of the "true" gradient. That way, you save computation time by not having to calculate the "true" gradient over the entire dataset every time.

You want to shuffle your data after each epoch because you will always have the risk to create batches that are not representative of the overall dataset, and therefore, your estimate of the gradient will be off. Shuffling your data after each epoch ensures that you will not be "stuck" with too many bad batches.

In regular stochastic gradient descent, when each batch has size 1, you still want to shuffle your data after each epoch to keep your learning general. Indeed, if data point 17 is always used after data point 16, its own gradient will be biased with whatever updates data point 16 is making on the model. By shuffling your data, you ensure that each data point creates an "independent" change on the model, without being biased by the same points before them.


Deep Learning approach for Signature extraction from emails Deep Learning (DL) is now days are used in various fields. I may like to know any idea/approach on usin DL for Signature extraction from emails.

There are some papers which uses ML like http://keg.cs.tsinghua.edu.cn/jietang/publications/f142-tang.pdf
http://www.cs.cmu.edu/~vitor/papers/sigFilePaper_finalversion.pdf     there are few more.

There are some open source libraries in Python like talon which uses NLP (Regular expressions and other techniques) and uses ML algorithms ( SVM ) for the same.

I have done a little research but I am not getting any resource or any idea on how to proceed with DL for this task.
I thought of using seq2seq models but I am not sure it can be of much help for this task. I am not sure whether summarization techniques can be useful.
Any idea or resources for the same can be helpful.

Detecting a signature in an email is more like a detection problem in an image rather than a natural language processing (NLP) problem. It does not particularly matter what the signature says, you simply want to be able to detect it.

In such a case, detection in images is currently dominated by convolutional neural networks (CNN). This is a very powerful technique which has proved to be the best for a multitude of image-based tasks. However, it requires a very large dataset. So you will need to collect a lot of varied examples and annotate them yourself.

The output of the CNN can provide either a binary output 0/1 determining if a signature exists. Or it can provide an output which is the coordinates of the center of the signature (X, Y coordinates) such that you can know where it is placed in the document.


Is there a difference between on-line learning, incremental learning and sequential learning? What I mean is the following: Instead of processing all the training data at once and calculating a model, we process one data point at a time and update the model directly afterwards.

I have seen the terms "on-line (or online) learning" and "incremental learning" for this. Is there a subtle difference? Is one term used more frequently? Or does it depend on the research community?

Edit: The Bishop book (Pattern Recognition and Machine Learning) uses the terms on-line learning and sequential learning as synonyms but does not mention incremental learning.

So, the following answer is just based on different opinions of collegues and professors from the field. I want to try to summarize it briefly:

Sequential and online learning is mostly associated with Bayesian updating. The sequential learning is used widely for an order in time of the data, meaning that $x_1$ is coming always first, then $x_2$, then $x_3$ and so on. The Dataset each has a certain order in that sense.
In contrast to that incremental may be a whole block of data at time x and another block of data at time y. While the block internally may be randomly ordered.

Concerning online learning, the people mostly referred to a data stream, hence a online learning is always incremental learning but incremental learning does not have to be online.

These are quite fuzzy definitions, and in my opinion there is not clear definition though. I still hope that helps.


How does GlobalMaxPooling work on the output of Conv1D? In the field of text classification, it is common to use Conv1D filters running over word embeddings and then getting a single value on the output for each filter using GlobalMaxPooling1D.

As I understand the process, the convolutional filter is a matrix of the same size as the $$\text{size of filter matrix} = \text{embedding dim}\cdot\text{width of the filter}$$ The filter matrix is then applied to the input embeddings (multiplied element by element) which produces a matrix of the same size for each filter position. Not a single number.

So how does the global max pooling get a single number on the output? Does it simply take a maximum over all the values in all the output matrices, or is there any other processing?

Please correct me if I'm wrong.

Apparently I forgot how the convolution works. The input is multiplied element wise with the filter weights and the products are then summed. That's how a single value is obtained on the output.


Assign more importance to recent records during training My goal is to build a classification model in order to predict if a customer will buy a product or not (binary classification).
Since in the last months (let's say 3-4) I know that the advertising of the company is changed a bit, I want to put more emphasis on the newer records.
I know that it is possible to specify the sample_weights parameter in most of the classification algorithms, but I don't know how to properly build these weights.
Basically, I have the date in which the customers have (or have not) bought the product, but I want to understand if there is a specific way to build the weights using this information.
In addition, I would also know if someone knows some references or applications for this application (time-related sampling weights).
Thank you in advance!

One option is incremental training. Update the model weights as more recent data is available. This automatically assigns more importance to recent records.
Incremental training works well within the Bayesian framework where historical data is the prior and more recent data updates the prior.


Discrete Ordinal Classification with Probabilities If I have classes 1, 2, 3 and 4. But, I also need the probability for each of the other classes. I'm currently using XGBoost for one-vs-rest classification, but that means we're losing information since the classes are ordinal.

Probit and logit ordinal regression model the cumulative probabilities $P(y \leq i | x)$. So you should be able to get class probabilities by taking the difference $P(y=i |x) = P(y \leq i|x) - P(y\leq i-1 |x)$. Then it just depends on the implementation whether or not it makes the probabilities available to you.

In R, for example, you can get these probabilities from the polr package.

[Even though you lose some information when doing one-vs-rest classification, ordinal regression models make some proportionality assumptions that (if not appropriate for your data) may not necessarily give you better results.]


How much data are sufficient to train my machine learning model? I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining.

My colleague (who is a machine learning expert) said that, in his opinion, the arguably most important practical aspect of machine learning is how to understand whether you have collected enough data to train your machine learning model.

This statement surprised me, because I had never given that much importance to this aspect...

I then looked for more information on the internet, and I found this post on FastML.com reporting as rule of thumb that you need roughly 10 times as many data instances as there are features.

Two questions:

1 - Is this issue really particularly relevant in machine learning?

2 - Is the 10 times rule working? Are there any other relevant sources for this theme?

The ten times rule seems like a rule of thumb to me, but it is true that the performance of your machine learning algorithm may decrease if you do not feed it with enough training data.

A practical and data-driven way of determining whether you have enough training data is by plotting a learning curve, like the one in the example below:

The learning curve represents the evolution of the training and test errors as you increase the size of your training set.

The training error increases as you increase the size of your dataset, because it becomes harder to fit a model that accounts for the increasing complexity/variability of your training set.
The test error decreases as you increase the size of your dataset, because the model is able to generalise better from a higher amount of information.

As you can see on the rightmost part of the plot, the two lines in the plot tend to reach and asymptote. Therefore, you eventually will reach a point in which increasing the size of your dataset will not have an impact on your trained model.

The distance between the test error and training error asymptotes is a representation of your model's overfitting. But more importantly, this plot is saying whether you need more data. Basically, if you represent test and training error for increasing larger subsets of your training data, and the lines do not seem to be reaching an asymptote, you should keep collecting more data.


Image classification: Predict all objects' classes without bounding boxes I'm trying to identify the problem here: I want to input an image and output the classes associated with all the objects in that image, not necessarily predict bounding boxes.

I have been looking for the solution but it seems like image classification won't work with images that contain more than one object. Otherwise it is the object detection problem, which is unnecessary because the output does not require bounding boxes.

Could you let me know what type of problem it is?

Object detection doesn't mean bounding boxes. It is as the name suggests to detect objects in an image being single or multiple which is what you actually need in your problem.

The display of bounding boxes is your choice if you need the class along with its position or not. As you already get bounding boxes in your solution, you might be getting their classes as well. Simply, don't draw the bounding boxes using OpenCV or any other. Just comment out those lines.


How to test the influence of a feature on conversion? I have a user journey where I have data of the format:

  userID, did_interact_with_feature(0/1), did_convert(0/1)

I want to verify the hypothesis that if a user is engaging with the feature, he's more likely to get converted.

Now I can get the % of user who engaged in feature and then got converted. But I seem to have hit a mental block here.

I was leaning toward a z-test, but could not formulate the problem.

Any help really appreciated.

Okay, thanks to this post, I was able to test this.
The test we need is a Chi-square test for approx results and Fischer Test for exact variables.

Given, the variables I had were categoric (engaged:yes/no and converted:yes no) in nature, the base assumption was true.

Now, First I created contingency table for each of those engaged and converted with reference from here.

There after, calculated the expected value for each of the cells in table.
Injected the formula and voila, I had the chi-square vale. Further compared with chi-square value table to get the p-value.

This resulted in testing whether the variables are independent or not.

This is what the end results look like:

Reference: https://www.socscistatistics.com/tests/chisquare2/Default2.aspx


What is a suitable Tensorflow model to classify images into foggy/not foggy? I want to classify photos taken by multiple webcams that are operating in mountainous regions into foggy / not foggy. The photos are in various sizes and were taken under very different light conditions and in different areas.

I read about Tensorflow and its ready-to-use image recognition models (which of course would have to be re-trained for the foggy/non-foggy categories).

However, these models are trained to classify images into categories according to objects within these images. As I want to classify my images based on their overall appearance (blurry, greyish, far away objects hardly detectable, ...) I was wondering if these models are really suitable or if there is a better approach for this task. Any help is highly appreciated!

From the sounds of the problem you could probably do some thing with extracting some features from the images such as how many edges they have, brightness (to get day/night), average color values. Then using a more simpler classification algorithm such as SVM, KNN, Decision Tree, Random Forest.

The Tensorflow ready-to-use models look to be very complicated models with a large number of layers so it will take a long time to train and run. It will also be very hard to train them from scratch (Retraining a pretrainid network could help with that though). So I think they might be a bit overkill. Also note that those models were probably made with the imageNet dataset in mind which has 1000 classes where you have just 2.

Its very hard to know what will work without seeing the images or being able to trying it first.

I would start with simpler faster methods before trying slower more complicated methods.
So in order try the feature extraction plus classifier if that is unable to learn a good relationship then move on to a basic CNN if that doesnt work move on to the more complicated CNN models.

With respect to the objects within rather than the overall appearance. For a CNN using different pooling layers can effect this change this. eg. Max Pooling will get the max value of a filter so can be largely effected by a small part of the image where as Average Pooling will be better at looking at the whole image.


Pandas v. SFrame in learning data science I'm taking a machine learning course that introduces the related assignments as such that a student may use either Pandas or SFrame in solving them. As a beginner, it's hard for me to assess which approach would be more beneficial for me in the long run. Hence the question; should I prefer Pandas or SFrame for learning purposes?

Initially I might've preferred Pandas to make sure I learn the "proper foundations" first, so to speak. However, after reading about this online, SFrame appears to be very popular after having been made open source. Can it be considered a Pandas replacement from a (future) data scientist's point of view?

Thanks a bunch! Protips always welcome in comments ;)

SFrame is not used much in industry, so I'd stick to pandas or Spark DataFrames. But they're all rather similar, and you should not spend much time thinking about it: it is easy to pick these things up, and employers understand this. Concentrate on the algorithms; that's the real "foundation", not the tools.


Any regression algorithm can address this problem, to a greater or lesser degree of success- the important thing is in how you form your data.

One reasonable form to use would be a structure with one row for each day and features describing the arrival of trucks over an appropriate timeframe- say, over the past 14 days. Each row would then also have the demand for the day associated with it as the regression target.

You would then be able to train a model that predicts the demand for a given day provided the appropriate truck arrival history.


How are Hyperplane Heatmaps created and how should they be interpreted? For nonlinear data, when we are using Support Vector Machines, we can use kernels such as Gaussian RBF, Polynomial, etc to achieve linearity in a different (potentially unknown to us) feature space and the algorithm learns a maximal separating hyperplane in that feature space.

My question is how do we create heatmaps such as the one seen in the image below to show this max. separating hyperplane in our original space and how should it be interpreted?

I think I can answer that, since I implement such a thing in my own library, even if I really don't know how it's implemented in other libraries. Although I am confident that if there are other ways, they don't differ too much.

It took my a few weeks to understand how such a graph can be drawn.

Let's start with a general function $f:\mathbb{R} \times \mathbb{R} \to \mathbb{R}$. What you want is to draw points with a colour which signifies the value of the function. One way would be to simplify the whole problem and draw one point for each pixel. This would work, but will draw only shaded surfaces and it's impossible to draw lines with various formats (dotted lines with some colours and line widths.

The real solution which I found makes two simplifications. The first one would the that instead of colouring with a gradient you can colour bands. Suppose that your function $f$ takes values in $[-1, 1]$. You can split your co-domain into many subintervals like: $[-1, -0.9]$, $[-0.9, -0.8]$ and so on. Now what you have to do would be to paint a polygon filled with appropriate color, for each interval. So your original problem is simplified to draw multiple instances of a simpler problem. Note that when your intervals are small enough it will look like a gradient and even a trained eye would not notice.

The second simplification would be to split the space which needs to be drawn into a grid of small rectangles. So instead of drawing a polygon on the whole surface, you need to fill each small rectangle with appropriate much simpler polygon.

If it's not obvious, the problem is much much simplified. Take for example a one of the rectangle. This rectangle has four corners and you can take an additional point in the center of that rectangle (you might need that point in some certain situations).

The question is how to fill with appropriate colour the proper region?
You need to evaluate function in all four corners and in the center.

There are some specific cases:

function evaluated in all corners are smaller than the beginning of the interval => you need to do nothing
functions evaluated in all corners are greater than the end of the interval => you need to do nothing
functions evaluated in all corners are within interval => fill the whole rectangle with an appropriate color

You can stop here if you want, but your figures would looks non-smooth. You can go further:

left-up, left-down, right-down in interval, right-up bigger => there are two points, one on up side and one on right side which contains the function evaluated at max value from interval => those two points together with top-right corner forms a triangle which can be filled
many other cases which requires only common judgement to decide which polygons to be formed and should be filled.

Using this algorithm you can fill polygons or draw lines. In the specific case of SVM you need to know that the line which corresponds with $f(x,y)=0$ is the line which classifies points into positive and negative samples. Also, lines which evaluates the function at $-1$ or $1$ corresponds with the maximal margins of SVM.

After some time I found that this kind of approach is named iso lines, or iso curves. Perhaps are more similar algorithms like that.

My implementation is named mesh contour (I did not found a proper name at that time in the beginning) and you can find source here.

Some examples:


Is training in Biostats sufficient to get into Data Science? I'm considering going back to graduate school after a few years working as a market analyst.  I prefer to stay in my current city, where there are two statistics programs I'm interested in.  The first is a more traditional stats program at a private university and the other is a biostats program at a public university.  The biostats program is significantly cheaper, which isn't the sole factor but it is a big one.  I would like to do consulting after grad school (a company like Accenture, for example).  The core curriculum for both is similar - regression analysis, applied modeling, etc - but obviously the Biostats program has classes like epidemiology and survival analysis whereas the traditional program has electives in data mining and visualization.  Would the biostats program be enough to get a solid data science job or should I stick to a traditional program?

a question, and then some comments.  First, what kind of market analyst do you want to be?  This field is rapidly becoming specialized, and the answer will affect the decision.  If you want to specialize in the pharma or especially the medical field, the biostat background could be an advantage.

Generally speaking, you need to think outside the box of what you will learn purely as defined by the program, either stats or biostats.  You will need to branch out into areas not defined by either program.  This could include (1) software language, (2) data science problems (Kaggle or external projects run through a campus department), or (3) courses not expressly given in either department, whether it's in "hard" skills like data mining or "soft" skills like marketing.  Ask for exceptions to do things that don't fit the program, but fit your career goals, because being a market analyst doesn't fit squarely into either field.


clustering 2-dimensional euclidean vectors - appropriate dissimilarity measure I've got a set of approx. 50 000 2-dimensional euclidean vectors which are connected with 20 groups, i.e. each group has approx. 2500 2-dimensional euclidean vectors. My data includes endpoints coordinates, i.e. $x_0, y_0, x_1, y_1$. Now I would like to cluster the vectors within these groups, probably using k-means/k-medoids clutering (or other clustering algorithm with pre-defined no. of clusters). What is also important, my main focus is on vector's direction, length is the minor concern (but at best, still should be taken into conideration). What I'm struggling with is a choice of dissimilarity measure that would be suited to my problem. So here are my question:

Does it matter how the data is specified? Alternatively, I could calculate an angle and length of vector and specify the data as $x_0, y_0, angle, length$. My intuition is that if angle is explicitely present, the euclidean distance should do a better job capturing the vector's direction. What is more I could maybe use some weighting, modify a euclidean distance and calculate distance between two observations as for example:

$\sqrt{(x^1_0 - x^2_0)^2 + (y^1_0 - y^2_0)^2 + (angle^1-angle^2)^2 + \frac{1}{n}(length^1-length^2)^2}$

where $n$ is some constant.

I also considered angular distance as a dissimilarity measure. From what I know this is equivalent to clustering the standarised data points and therefore doesn't capture size (lengths in my case). But I'm not sure if k-means clustering can be done with cosine distance. If so, is there any package in R that allows that?
Is is a good and statistically valid idea to perform clustering twice: firstly, to cluster starting points and secondly, within those clusters perform clustering for angles and lengths?
Do you guys know any papers regarding similar problem, i.e. clustering the 2-dimensional data points? Any example would be very handy.

I hope I followed your question correctly. If you have those data points on as 2D vectors, that would mean, that you have, say $N=50,000$ data points each represented as $\left(x_i,y_i\right)$ , right? If angle and length are what you are concerned about, then your formulation seems right. You can convert each data point $\left(x_i,y_i\right)$ into $\left(\theta_i, d_i\right)$, which are the angle and length of the vector. These two parameters $\left(\theta_i, d_i\right)$ act as two features in each of your samples and you can run k-means on this. You have to be careful about using a consistent measure on your angle (always anti clockwise or clockwise). Your dissimilarity measure seems quite correct as far as I can tell. I'm sure you are aware of this python package but just for the sake of completion you can use  this or in MATLAB you can use this. I'm not sure about R, so pardon me if that was the essential part of your question.


XGBoost: what to do when Kfold is not enough? I have a dataset made of roughly 100 time-series and my final goal is to obtain a classification of each point (detection problem). To do so I have labels so I decided to use an XGB model to perform the detection over some features that I have created. The time-series are not sampled uniformly and the time-order looks not so important for this specific problem so far.

The problem is that when I perform the StratifiedKFold (as per Sklearn) the results looks promising and the standard deviation of the relevant metrics among the kfold is really small. Nevertheless, if I remove one time-series entirely from the training set and fitting the model over the other ones I am not able to replicate the same results.

This gap between Kfold performances and "real test" looks to me like the training is not really generalising the problem, despite the good results during the Kfold validation.

Do you have any idea to fix this problem? or any advice?

Using normal KFold cross-validation for time-series data will yield a highly optimistic error estimate since you are using data from the future to predict the past. The model just has to learn to interpolate, not to predict.
Therefore you have to use time-wise CV:

Furthermore, if your goal is to predict the future for a time-series you do not know the past of, you have to use leave-one-group-out, time-wise-CV to get a realistic performance estimate: Train on all time-series except for a test-set of time-series, but only using data up to a certain point in time. Validate on the test-time-series but only using data after the timepoint.

I doubt you will have success with this approach since this is often a very difficult problem.

For choosing the right cross-validation method you have to be clear about the goal of your method.

Do you want to interpolate missing data for a couple of time-series?
Use K-Fold CV
Do you want to predict the future of some time-series, for which you
know the past? Use time-wise CV
Do you want to predict the future of some time-series, but you only
know the past of some other time series? Use Leave-one-Group-out,
time-wise CV

I assume you are after the second point since this is the most common problem.


Tradeoffs between Storm and Hadoop (MapReduce) Can someone kindly tell me about the trade-offs involved when choosing between Storm and MapReduce in Hadoop Cluster for data processing? Of course, aside from the obvious one, that Hadoop (processing via MapReduce in a Hadoop Cluster) is a batch processing system, and Storm is a real-time processing system.

I have worked a bit with Hadoop Eco System, but I haven't worked with Storm. After looking through a lot of presentations and articles, I still haven't been able to find a satisfactory and comprehensive answer.

Note: The term tradeoff here is not meant to compare to similar things. It is meant to represent the consequences of getting results real-time that are absent from a batch processing system.

MapReduce: A fault tolerant distributed computational framework. MapReduce allows you to operate over huge amounts of data- with a lot of work put in to prevent failure due to hardware. MapReduce is a poor choice for computing results on the fly because it is slow. (A typical MapReduce job takes on the order of minutes or hours, not microseconds)

A MapReduce job takes a file (or some data store) as an input and writes a file of results. If you want these results available to an application, it is your responsibility to put this data in a place that is accessible. This is likely slow, and there will be a lag between the values you can display, and the values that represent your system in its current state.

An important distinction to make when considering using MapReduce in building realtime systems is that of training your model, and applying your model. If you think your model parameters do not change quickly, you can fit them with MapReduce, and then have a mechanism for accessing these pre-fit parameters when you want to apply your model.

Storm: A real-time, streaming computational system. Storm is online framework, meaning, in this sense, a service that interacts with a running application. In contrast to MapReduce, it receives small pieces of data (not a whole file) as they are processed in your application. You define a DAG of operations to perform on the data. A common and simple use case for Storm is tracking counters, and using that information to populate a real-time dashboard.

Storm doesn't have anything (necessarily) to do with persisting your data. Here, streaming is another way to say keeping the information you care about and throwing the rest away. In reality, you probably have a persistence layer in your application that has already recorded the data, and so this a good and justified separation of concerns.

If you want to know more...
If you would like to learn more about real-time systems that that fit parameters with MR and apply the models a different way here are slides for a talk I gave on building real-time recommendation engines on HBase.

An excellent paper that marries real-time counting and persistence in an interesting way is Google News Personalization: Scalable Online Collaborative Filtering

Another interesting marriage of MR and Storm is SummingBird. Summingbird allows you to define data analysis operations that can be applied via Storm or MR.


Dealing with time series data which has multiple observations for the same timestamp I have a time series data in Python 3 as follows:

Date                `Weekly_Sales`
2010-05-02              3400
2010-05-02              5600
2010-05-02`             4590
2010-05-02              5800
2010-05-12              2380
2010-05-12              6700
2010-05-12              3700

The time series  is not continuous as there are multiple observations of the same date.I'm trying to forecast sales in python using ARIMA but my ACF and PACF plot shows that there is no corelation between the lags.Also if i run the dickly fuller test to test stationarity,my system freezes.

How can I fix this?

It looks like you have lost a bit of information in that dataset. You shouldn't have 4  measurement for one timestep for one variable - how do you know which of the first four rows to use for 2010-05-02?

I would suggest checking your data source, or then working out a way to explain the meaning of the four values... are they different somehow (using other information)?

How are you even creating lags on that Date index? Take the average over each day?
Depending on the package you use for your Dickey-Fuller test (and other methods), they might not be made to deal with identical timesteps as input... so could explain why the session crashes.


Test accuracy of neural net is going up and down I am using a CNN to classify medical images. I am using a four convolutional layers with ReLU activation followed by a softmax layer. I am using rmsprop as the optimizer. The problem I am facing is the network's test accuracy increases and then goes down. This continues till the maximum number of epochs is reached. I read somewhere that this could be due to the fact that the network shifts its weights in favor of one class, and then has to do so in favor of the other class. I shuffled the test data, to see if this would help, but no such luck. Is there any remedy?

Based on the image you are sharing, the training accuracy continues to increase, the validation accuracy is changing around the 50%. I think either you do not have enough data to use neural network or the network is small to capture all the information, in both cases I feel there either under fitting or over fitting problem. Can you give us some information about the database so we can help more


Is there a real C4.5 implementation in Python ? (handling missing value) To my understanding, C4.5 comes with 4 improvements compared to ID3:

Handling missing values in both training data and "test" data,
Handling continuous data
Handling costs on attributes.
The pruning

Source

However, not one of all decision tree python modules that I found, even the so-called C4.5, handles missing values.

Do you know of a library that handles them ? And it would be great if it can work with Pandas DataFrames "out-of-the-box"...

I don't think there is a C4.5 implementation in a popular python library. Your options are :

Try github implementations such as : https://github.com/geerk/C45algorithm
Try getting R implementations with rpy package : see over there for an exemple of how that would work (C5.0) https://stackoverflow.com/questions/41070087/calling-c5-0-in-python


Backpropagation - simplest explanation Could you please explain, in simplest way, the algorithm (mathematical equation) of back-prop?
I have read lot of articles on it so I know what it is and understand the intuition behind it, but I still do not understand the equation of "upgrading/changing" neurons' attributes in layers.

You can refer to this answer here for a very detailed explanation and derivation of the backpropagation algorithm.

In general, backpropagation is based on the idea that it is possible to attribute an amount of blame to each parameter in your model for the resulting error. So this is what we do, first we will set all the parameters in our neural network (weights and biases) randomly. Then we will pass examples through the model, and at the output we will compute some loss function. Then using backpropagation we will tune our model parameters proportionally to their contribution to the incurred error.

Here are some other related answers that explain the complexities of backpropagation:

Derivative of the sigmoid
Quick run through of the equations dictating backpropagation
Gradient descent, backpropagation and the partial derivative
Computing the backpropagation
Backpropagation in CNN
Backpropagation with different activation functions


Importance of Variance in Machine Learning Selecting a column from a Dataframe, plotting its Histogram using matplotlib and then finding the variance is the steps that I have to take for this part of a project.

The final goal of the project is detecting anomalies in data.

I understand the part of plotting the data in form of a graph to understand it better.
On the other hand I’m not sure why finding the variance of this column is so important.

Would really appreciate if I get any information regarding this.

I know the definition of variance but just can’t seem to understand the usage and importance of it in this project.

Variance in a feature (defined as the average of the squared differences from the mean) is important in machine learning because variance impacts the capacity of the model to use that feature.

For example, if a feature has no variance (e.g., is not a random variable), the feature has no ability to contribute to task performance. A zero variance feature will be constant for different levels of the target.

Additionally, some machine learning models make strong assumptions about the distribution of features. Variance is one way to check for distributional assumptions.


Multivariate outlier detection with isolation forest..How to detect most effective features? I am trying to detect outliers in my data-set with 5000 observations and 800 features. I have followed the simple steps told in http://scikit-learn.org/stable/auto_examples/ensemble/plot_isolation_forest.html

There are some examples in stackoverflow and other sources as well, however, I could not find a decent explanation about the interpretation of outliers returned by isolationforest. First, what I did is:

from sklearn.ensemble import IsolationForest

X_train = trbb[check_cols]

clf = IsolationForest(n_jobs=6,n_estimators=500, max_samples=256, random_state=23)
clf.fit(X_train)
y_pred_train = clf.predict(X_train)
y_pred_train

This returns array([1, 1, 1, ..., 1, 1, 1]) where -1's are the outliers.

The shape of y_pred_train is 5000, which is identical with X_train[0]. So the index of -1 corresponds to the index of X_train. For exp. one of the outlier indices returned by IF is 532. So this means that the point (in 800 dimensional space) in this index is detected as an outlier. After detecting this, how can I approach and use the results to dig more? For example, can I reach the most important features causing the outliers?

To this end, I applied PCA and used the newly created (50) components as new variables. Then, by plotting component pairs with -1 & 1s returned by IF, I tried to get some insight of possible outliers. Indeed, I observed some points as shown below:

Here, the black dots are -1s (outliers) and the yellow ones are 1s (inliers). I interpret this that if both components have very high values, it might cause an outlier. However, how can I decide which original values are the most effective ones for those outliers? For example, can I use the original variables with the highest weights (let's pick top 10) in the related components? Or could you suggest any other method?

Thanks in advance!

One important difference between isolation forest and other types of decision trees is that it selects features at random and splits the data at random, so it won't produce a nice feature importance list; and the outliers are those that end up isolated with fewer splits or who end up in terminal nodes with few observations.

I guess if you dig into the code you might be able to rank the features according to levels of the splits that use them and the number of observations in the nodes, but AFAIK neither the scikit-learn implementation, nor Zelazny's R implementation, have such a thing built-in (although the R one has some functions for individual nodes: https://github.com/Zelazny7/isofor/blob/master/R/interpret.R).

Nevertheless, after you identify the outliers from your sample, you can use that as labeled data for a classification problem, and from there fit a model such as xgboost or random forest that could give you the feature importances.


How can I use GloVe for text spinning? Hello I am building a python program for text spinning. I have read about GloVe, word2vec, doc2vec or text2vec. I understand that what they do is to represent each word as a semantic vector.

So I guess that if I train a GloVe I could use it just for finding synonyms, or something more?

EDIT: I have come up with one strategy:

First I train a doc2vec model. The goal would be to build semantic vectors of phrases
Then I use my original paragraph input. Word by word I change it randomly with a synonym (provided by nltk or other synonym db). Before finally spinning one word I compute the doc2vec for the changed paragraph and compute the distance with the original paragraph vectorized. I only change the synonym if the distance is small.

How does that sound to you?

So I guess that if I train a GloVe I could use it just for finding synonyms, or something more?

GloVe (vectors) are not the perfect word representations if you were to find synonyms.
Since GloVe vectors are generated on the basis of the contexts in which they appear (context implies words before and after the word in study within a sentence), you may find antonyms will also have similar GloVe vectors since they too will appear in the same context. For eg.

Consider the two words "good" and "bad" used in the following sentences:

John was a good boy who was injured in the war.
John was a bad boy who was injured in the war.

Here, both the words "good" and "bad" appear in the same context. The context before is "John was a" and the context after is " who was injured in the war."

Hence, their GloVe vectors won't have much of a distinction and you might change the meaning of your sentence completely when spinning for a new word.

To address the above, a very good alternative to GloVe vectors is "retro-fit vectors". Here is the link of the original article. and here is the GitHub link of the actual vectors. since they are learned for semantic similarity. I suggest using these retro-fit vectors since you seem to be interested in lexical substitution of synonyms.


How do I create a data set that has a set of features for multiple options, with one option being the expected outcome? Most datasets I see are:

feature 1, feature 2, feature 3, outcome

Where outcome is binary e.g. if they are cancer positive outcome will be 1 and 0 if they don't have cancer.

How do I create a dataset where there are multiple outcomes and each possible outcome has a set of features for it?

e.g. I have a question with 3 possible answers:

"What organ pumps blood around the human body?"
A. Heart
B. Liver
C. Church Organ

And each answer has a set of features with one answer being correct. How would I display this in a csv file? I want to read it into an xgboost algorithm for training.

question, option1 and features, option2 and features, option3 and features, correct option

Many thanks for your help!

The final feature vector would be a concatenation like (for multi-class prediction):

  Question google count | option A google count | option B google count | option C google count | option C no. words | option A no. words | other features | label
  (1, 2, 3)

There is no need to put features related to option A close to each other (or in any particular order), they just need to be on the same column for all rows regardless of the label.

XGBoost parameters for multi-class classification are:

'objective': 'multi:softprob',
'num_class': 3


Inverse_Transform and multistep data - how to scale and when I tried to ask this question before, but I think it was difficult without revealing the full context.  Right now I'm tinkering with what I find online to informally learn about tensorflow before I formally learn it in the spring.
The model I have created takes normal open, high low, close, volume data and calculates a bunch of other features.  My intent is to use the prior 15 days of OHLCV+Technicals to predict the future 5 Close.
My issue is at the very end when I need to invert the model.predict.  Currently I am using the following workaround:
sc = MinMaxScaler()
test_set_tomorrow = dataset_train.iloc[-time_step:,:].values
scaled_test_set = sc.fit_transform(test_set_tomorrow)
scaled_test_set = test_set_tomorrow
X_test = []
for i in range (0,len(test_set_tomorrow)):
  X_test.append(scaled_test_set[i,:])
X_test = np.array(X_test)
X_test=np.reshape(X_test,(1,time_step,-1))
predicted_price = model.predict(X_test)
predicted_price_fix = np.zeros(shape=(len(predicted_price), X_test.shape[2]))
predicted_price_fix[:,0:5]=predicted_price[:,0:5]
print(predicted_price_fix)
predicted_price=sc.inverse_transform(predicted_price_fix)
print(predicted_price)

Which outputs the following:
[[-0.04685311 -0.04455814 -0.04243669 -0.04118608 -0.03964855  0.
   0.          0.          0.          0.          0.          0.
   0.          0.          0.          0.          0.          0.
   0.          0.          0.          0.          0.          0.
   0.        ]]
[[ 1.56266435e+00  1.59385461e+00  1.49811773e+00  1.56682045e+00
   8.50910050e+05  1.65916667e+00  2.13096538e+00  1.60200000e+00
   1.63600000e+00  1.81628246e+00  2.08335540e+00 -3.99787353e-01
   6.99000000e-02  1.37914286e-01 -3.75699907e-01  1.00000000e+00
   2.61479727e+08  0.00000000e+00  2.01494297e+07  1.58000000e+00
   1.90910833e+00  1.76660253e-01 -4.35308886e+02 -2.06430382e-16
  -4.73584558e-18]]

There are two things which I find out.  The first is the values quite a bit off.  The second, is the fifth value.  It appears to be wildly out of whack.

On the original code:
sc = MinMaxScaler()
test_set_tomorrow = dataset_train.iloc[-time_step:,:].values

Hm, you missed the shuffling, but it's OK -- maybe your data was shuffled earlier.

Original code:
scaled_test_set = sc.fit_transform(test_set_tomorrow)

The model should be trained on the train set not the test set - otherwise, data leakage occurs. It should be either:
sc.fit(dataset_train.iloc[:time_step])
scaled_test_set = sc.transform(test_set_tomorrow)

or
scaled_train_set = sc.fit_transform(train_set_tomorrow)

Original code:
scaled_test_set = test_set_tomorrow

You just erased all the pre-processing. If you check the data, scaled_test_set is exactly dataset_train.iloc[-time_step:, :]

Original code:
X_test = []
for i in range (0,len(test_set_tomorrow)):
  X_test.append(scaled_test_set[i,:])
X_test = np.array(X_test)
X_test = np.reshape(X_test,(1,time_step,-1))

can be done quicker. Note that the MinMaxScaler returns the numpy array already.
X_test = np.expand_dims(scaled_test_set, axis=0)

Finally, you apply the inverse transform to the labels while training on the features. For example, consider the model where given height, we predict weight. You create a map that normalizes inches (or centimeters). In case of centimeters, say the smallest height is 150, but it is not true that the smallest weight is 150 kgs or pounds. Different scalers should be used.


Does running the script of train_test_split several times with varying test size in sklearn cause data leakage? I ran a script of ridge and lasso regression twice with and without pca. Both times i got an okay R^2. but when i changed the train_test_split test size from 20 to 30%. My model started to over fit. So i was wondering whether this is because of changing the test size or something else.

Thank you in advance,

It's understandable that when you get less training data your model can overfit (i.e. less data means less requirements to satisfy for your objective function which results in easier fitting for the problem) but it is also interesting that your models are sensitive to that much change. I advise you to try cross-validation with 30 % test split size using RidgeCV and LassoCV classes.


Elbow method for cosine distance I have clustered vectors by cosine distance using nltk clusterer. If I understand correctly, Y axis for elbow method in euclidian distance would be the sum of every distance (squared) between centroid of the cluster with vectors that belongs to that cluster.

My question is: Would it be the same for clusters using cosine distance?

EDIT: ok, so i tried sum of squares with cosine distance and it seems, that it's returning the same values... heres my code:

EDIT2: My bad,is is working

from nltk.cluster import KMeansClusterer, cosine_distance
import numpy as np

#Load dataset obtained from http://cs.joensuu.fi/sipu/datasets/a1.txt
testing_vectors = np.loadtxt("a1.txt")

for k in range(1,10):
    kclusterer = KMeansClusterer(k, distance=cosine_distance)
    assigned_clusters = kclusterer.cluster(testing_vectors, assign_clusters=True)

    sum_of_squares = 0
    current_cluster = 0
    for centroid in kclusterer.means():
        current_page = 0
        for index_of_cluster_of_page in assigned_clusters:
            if index_of_cluster_of_page == current_cluster:
                y = testing_vectors[current_page]
                #sum_of_squares += np.sum((centroid - y) ** 2)
                sum_of_squares += (np.dot(centroid,y)**2)/(np.dot(centroid,centroid) * np.dot(y,y))
            current_page += 1
        current_cluster += 1

    print("for k=%s the sum of squares is:%s" %(k,sum_of_squares))
```

Ok. So what I understood is, that for cosine metrics, I can use both: Sum of squared distances from centroids to vectors that belong to clusters, where the distance can be calculated as euclidian or as cosine (cosine would be probably more precise, but more complicated(thanks to dot product)). The squared distance is only used as optimization, so we don't have to calculate the square root in both euclidian and cosine distance formula.


my xgboost model accuracy decreases after grid search with I tried grid search for hyperparameter tuning in XGBoost classifier but the best accuracy is less than the accuracy without any tuning

// this is the code before the grid search
xg_cl = xgb.XGBClassifier(objective='binary:logistic', seed = 22)
xg_cl.fit(x_train, y_train)
y_pred = xg_cl.predict(x_test)
print(metrics.confusion_matrix(y_test, y_pred))
print(metrics.accuracy_score(y_test, y_pred))
[[11  1]
[ 1 26]]
0.9487179487179487

also, worth mentioning that the dataset shape is (195, 11) after PCA, and i am trying to classify whether a patient has a parkinsons disease or not.

// this is the grid search code
clf_xgb = xgb.XGBClassifier(objective = 'binary:logistic')
params__grid = {
    'n_estimators' : range(50,150,10),
    'max_depth': range(2, 12),
    'colsample_bytree': np.arange(0.5,1,0.1),
    'reg_alpha' : np.arange(0,0.6,0.1),
    'reg_lambda' : np.arange(0,0.8,0.1)

}
search = GridSearchCV(estimator=clf_xgb, param_grid=params__grid, scoring = 'accuracy',
                            cv = 4 )
search.fit(x_train,y_train)

print('best score:/n',search.best_score_)
print('bestparams:/n' ,search.best_params_)

best score:
0.9038461538461539

best params:
{'colsample_bytree': 0.5,
     'max_depth': 7,
     'n_estimators': 50,
     'reg_alpha': 0.2,
     'reg_lambda': 0.1}

then I used these parameters to build and train a new classifier

clf_xgb_1 = xgb.XGBClassifier(objective = 'binary:logistic', max_depth = 7, n_estimators = 50, reg_alpha = 0.2 ,
                              reg_lambda = 0.1, colsample_bytree = 0.5 )
clf_xgb_1.fit(x_train,y_train)
y_pred_2 = clf_xgb_1.predict(x_test)
print('accuracy:/n', metrics.accuracy_score(y_test,y_pred_2))
print('confusion matrix:/n', metrics.confusion_matrix(y_test,y_pred_2))

accuracy:
0.8974358974358975

confusion matrix:
array([[ 9,  3],
       [ 1, 26]], dtype=int64)

How come my results are worse? I would expect that GridSearch would improve on the results.

I think there are several points to take into account:

First, it is possible that, in this case, the default XGBoost hyperparameters are a better combination that the ones your are passing through your params__grid combinations, you could check for it
Although it does not explain your case, keep in mind that the best_score given by the GridSearchCV object is the Mean cross-validated score of the best_estimator (source), it is, the mean test score across the k trainings over the k-folds, so it can maybe give you a more reliable score value (including also the standard deviation of the k scores)

Consider also that you do not need to retrain your model with the best params fron the grid search CV, since you can access the best model via search.best_model

Lastly, I would also recommend other hyperparameter tuning methods like the bayesian tuning, below a sample code gist using hyperopt (info about bayesian optimization here):

from hyperopt import hp
from sklearn.model_selection import StratifiedKFold

from hyperopt import fmin, tpe, hp, STATUS_OK, Trials
from sklearn.model_selection import cross_val_score

def return_model_scores(params, dataX, dataY, n_folds=5):
    import numpy as np

    defined_model, edp_cost_scores, histories =
    evaluate_model_cost(dataX = dataX, dataY = dataY)

return np.array(edp_cost_scores).mean()

param_space = {'max_depth' : hp.choice('max_depth', range(5, 30, 1)),
  'learning_rate' : hp.quniform('learning_rate', 0.01, 0.5, 0.01),
  'n_estimators' : hp.choice('n_estimators', range(20, 205, 5)),
  'gamma' : hp.quniform('gamma', 0, 0.50, 0.01),
  'min_child_weight' : hp.quniform('min_child_weight', 1, 10, 1),
  'subsample' : hp.quniform('subsample', 0.1, 1, 0.01),
  'colsample_bytree' : hp.quniform('colsample_bytree', 0.1, 1.0, 0.01),
  'scale_pos_weight': hp.uniform('scale_pos_weight', 0.2, 0.8)}

# Some variable
dataX = X_train
dataY = y_train
assert len(X_train)==len(y_train)
n_folds=5

global best # global variable defined for convenience of this use case
best = 0
i = 0
def f(params):
    cost = return_model_scores(params, dataX, dataY, n_folds=5)
    if i == 0:
        best = cost

    if cost < best:
        best = cost
    print('new best:', best, params)

    return {'loss': cost, 'status': STATUS_OK}

trials = Trials()
best = fmin(f, param_space, algo=tpe.suggest, max_evals=10, trials=trials)

print ('best:')
print (best)

where evaluate_model_cost is the function you build for getting the cost values after evaluation:

def evaluate_model_cost(dataX, dataY):
    from tqdm import tqdm
    from sklearn.model_selection import KFold

    scores, histories = list(), list()
    # prepare cross validation
    kfold = KFold(10, shuffle=True, random_state=1)
    # enumerate splits
    k = 0
    for train_ix, test_ix in tqdm(kfold.split(dataX)):
        print('kfold {}'.format(k))

        # select rows for train and test
        trainX, trainY, testX, testY = dataX.iloc[train_ix],
          dataY.iloc[train_ix], dataX.iloc[test_ix], dataY.iloc[test_ix]

        # fit model
        history = defined_model.fit(trainX, trainY,
                                eval_metric= 'auc',
                                eval_set=[(testX, testY)])
    # evaluate model
    y_preds = defined_model.predict(testX[features_to_train_on])

    y_true_values = testY
    true_predicted_tuples = pd.DataFrame({'y_true': y_true_values, 'y_predicted': y_preds, 'days_till_slag': days_until_slag})
    true_predicted_tuples = true_predicted_tuples.reset_index(drop=True)
    #true positives, false positives and false negatives number
    tp_savings = 0

    first_tp_detected = False

    while first_tp_detected==False:
        for index in true_predicted_tuples.index:
            # TRUE POSITIVE condition:
            if ((true_predicted_tuples.iloc[index]['y_true']==1)&(true_predicted_tuples.iloc[index]['y_predicted']==1)):
                tp_savings += return_true_positive_savings(true_predicted_tuples.iloc[index]['days_till_slag'])
            break
        first_tp_detected = True

    fp_number = len(true_predicted_tuples[(true_predicted_tuples.y_true==0)&(true_predicted_tuples.y_predicted==1)])
    fn_number = len(true_predicted_tuples[(true_predicted_tuples.y_true==1)&(true_predicted_tuples.y_predicted==0)])

    final_cost = ((costs_dict['fp_cost'])*fp_number) +
        ((costs_dict['fn_cost'])*fn_number) - tp_savings

    score = final_cost_custom_function

    print('score en evaluate_model_with_slag_days', score)

    # append scores
    scores.append(score)
    histories.append(history)

    k = k + 1

return defined_model, scores, histories

where final_cost_custom_function is your custom function of costs to minimize.
This might be a bit complicated case, if you want a very simple example of bayesian optimization, start with:

import pickle
import time
from hyperopt import fmin, tpe, hp, STATUS_OK, Trials

def objective(x):
    return {
     'loss': x ** 2,
     'status': STATUS_OK,
     # -- store other results like this
     'eval_time': time.time(),
     'other_stuff': {'type': None, 'value': [0, 1, 2]},
     # -- attachments are handled differently
     'attachments':
        {'time_module': pickle.dumps(time.time)}
     }
trials = Trials()
best = fmin(objective,
 space=hp.uniform('x', -3, 3),
 algo=tpe.suggest,
 max_evals=100,
 trials=trials)

print('with 100 trials: ', best)

trials = Trials()
best = fmin(objective, space=hp.uniform('x', -3, 3), algo=tpe.suggest,
 max_evals=1000, trials=trials)

print('with 1000 trials: ', best)


Open-source interactive dashboard in Python I am trying to find a package to construct a dashboard with interactive graphs (including widgets such as sliders) in python (mainly IPython notebook). I know there is plotly but I would like a fully open-source solution without constraints (i.e. having a public repository as with plotly without the subscription fee).

I have looked at IPython Dashboard package but it is not compatible with python 3 (because of MySQL-python).

Has anyone had any luck with any other package?

Have a look at IPyWidgets.
I've used it to create interactive dashboards in IPython/Jupyter. It's very concise and powerful.
You define a function:
def f(x):
return x

And you connect it to a widget using the Interact() function.
interact(f, x=10);

This generates a widget (a slider in this case) automatically and updates your function with new input when you interact with the widget. The type of widget is determined from the default value you specify (i.e. a checkbox for a boolean, a textbox for a string, etc.)


How to measure the distance (in generalized sense) between geographical regions? I need to construct a distance matrix for a few U.S. counties that are adjacent to one or another, and choosing the definition of distance is very tricky. The shortest path (i.e the minimum number of times one has to cross county borders to walk from A to B) and the Vincenty distance between the most populous cities are not working well already. What are other options?

Also, I read somewhere and some time ago that distance between two geographical regions can be measured by the strength of "interaction". Volume of telephone calls, trajectories of trucks, and frequencies of flights were mentioned. Yet I can't any details about such modeling. Can somebody refer me to a paper or something? Are there open datasets of those souces available?

Anybody help me out here?

In terms of interaction data, you might be thinking of this paper and this project

I love the idea of using flight data. You could start with this set hosted at Kaggle.

In terms of estimating distance between states, the simplest thing you could do that might work is to measure the distance between state capitals.


Xgboost (classification problem) feature importance per input not for the model I have trained a xgboost model for a classification problem. I'm able to get the feature importance for the model as below.

http://machinelearningmastery.com/feature-importance-and-feature-selection-with-xgboost-in-python/

But I would like to get feature importance per input. Basically, one input may give me a output probability 0.9, another one input may give me a output probability 0.1. I want to know because which features (with values) give me a probability 0.9? which features (with values) give me a probability 0.1?

How can I approach that? Is there a package for this?

A partial dependence plot might be what you are after: these plots show the relationship between the probability and the input variable. The mlr package in R takes care of this.

Discussion of Partial Dependence on XGBoost Git: https://github.com/dmlc/xgboost/issues/486

General Tutorial using mlr: https://mlr-org.github.io/mlr-tutorial/devel/html/partial_dependence/index.html


How to use Autoencoders for outlier detection on images I have a bunch of images taken from a camera showing a pipe and would like to detect if the pipe is leaking or not. There are very few examples of leaking pipe in the data set. So considering this problem as a supervised learning problem, I think that it may not give us good result due to imbalanced data. I'm thinking of using autoencoder and considering it as an outlier detection problem.

I'm new to deep learning so I'd like to know how the architecture of my neural network should look like. Should I have some convolutional layers first and then an autoencoder or I should only have autoencoder? What would be the best DL library for such a use case? I also thinking of using only the photos which do not have any leak for the training phase, is that OK?

If you use supervised learning you need to weighted you labels (here).

Otherwise I strongly recommend to exclude every leaking image for a training dataset for an CAE. If you have the anomaly in your trainign dataset, your CAE will learn it as normal. This is actually an active research field, and not trivial for real datsets, but if you have a static image of the pipe, it should be easy. I would suggest that you use traditional CV for it and not an DL model, because you can interpret and fine tune the result.


Fuzzy Cognitive Maps I am working on a project for prediction, using Fuzzy Cognitive Maps. I am new to data mining and want to learn about Fuzzy Cognitive Maps and its implementation. I also want to know what tools are required for this. I am beginner in this domain, so I need a little help to get started. I would be very grateful for help with some good resources and advice.

You could use competitive fuzzy cognitive maps