feature importance after classification I have time series data and more or less 200 features for each sample, I used a recurrent neural network for the binary classification task.
After the classification I would like to know which features contribute most to one of the target(let's say target=1). Any suggested method? Thank you

You may use Permutation importance
 - Get your base-line score
 - Permutate a feature values. May replace with Random values
- Calculate the score again
- The dip is the feature importance for that Feature
- Repeat for all the Features

....Breiman and Cutler also described permutation importance, which measures the importance of a feature as follows. Record a baseline accuracy (classifier) or R2 score (regressor) by passing a validation set or the out-of-bag (OOB) samples through the Random Forest. Permute the column values of a single predictor feature and then pass all test samples back through the Random Forest and recompute the accuracy or R

To check the importance for the individual Class i.e. 0/1
Extrapolate the same to check if the increase is more for False-Positive or False-Negative.
Read Beware Default Random Forest Importances for more explanation.
Few other quotes from the page-

Any machine learning model can use the strategy of permuting columns to compute feature importances. This fact is under-appreciated in academia and industry.

The permutation mechanism is much more computationally expensive than the mean decrease in impurity mechanism, but the results are more reliable. The permutation importance strategy does not require retraining the model after permuting each column; we just have to re-run the perturbed test samples through the already-trained model.


Applying an algorithm on it's own training data for descriptive purposes? Good Day, I am newer to data science so I am not confident in this. To set up the question I will describe my data and approach.
Data
I don't want to share specific data examples as I want to try and keep some anonymity.  I have data for events between 2016 and 2019.  Each event has ~14 features (categorical and numerical) as well as a binary label of success or failure.  Each of these is run through different transformers (normalizing and one-hot encoding).
Approach
What I am interested in doing is knowing how likely events are to success (not necessarily predict if they will succeed).
I played around with train/test splits to find an algorithm that worked best. I ran a stratified k-fold cross-validation on the data in a grid search to tune the hyperparameter using logloss as my measurement of choice.  I am implementing this in Python so Scikit-Learn as my tool of choice.  The algorithm I settled on was GradientBoostingClassifier.
Problem
What I am interested in doing now is as I look at events in 2020, I am curious on how likely they are to succeed.  When I look at my 2020 data I have the same 14 features (transform them the same way) and as well I instantly know if they succeeded or failed.  So predicting Success/Failure is not interesting nor what I want to do.  I can easily generate probabilities on this 2020 data, using my trained model, with predict_proba in sklearn.
Question
Now my partner wants me to apply this model on the same training data from 2016-2019.  Initially this feels like a big no-go. You never predict your own training data as it will be heavily biased.  But I am not predicting.  These feels more like a traditional statistics descriptive problem where I look at the nature of known data and see how it behaves.
So again, I am not interesting in knowing/predicting if something WILL succeed or fail (we know this instantaneously).  I am more interested in knowing if it succeeded (or failed) vs how likely it was to do so.  A success with a 5% chance vs succeeding with 40% chance is way more interesting in this problem.
So my question, per the title is, can you apply a trained algorithm / model on its own training data if my interest is not in predicting forward but evaluating success vs likelihood of success (and still yield useful information)?

can you apply a trained algorithm / model on its own training data if my interest is not in predicting forward but evaluating success vs likelihood of success (and still yield useful information)?

Yes, absolutely. There are quite a few cases where applying a model on the training data is useful. The most common is probably to detect overfitting: a high difference in performance between the training and test set is a sign of overfitting. In general it can also be useful to know how the model performs on the training set in order to obtain an upper baseline for the performance.
Everybody says "don't predict on the training set" simply because it's a simple rule to remember and such an easy mistake to make for beginners. But as long as one understands what they are doing and knows that obviously the predictions obtained on the training set are biased, there's no problem.
The task described in the question makes sense to me, you have my permission to predict on the training set ;)
Just one more remark: there might be a bias in the training data itself, in the sense that if success for an event is very unlikely then the event will fail more often than suceed even when the condition of its success are satisfied. If the data contains this kind of cases, it means that the model is trained to predict "fail" for potentially successful features. Whether or not this impacts the model depends on whether there are enough similar but successful events in the data, I think.


How to fill in missing value of the mean of the other columns? I had a movie dataset including 'budget' and 'genres' attributes.

I'd like to fill in the missing value of budget with the mean budget of each genre.

I first create two dataframes with or without budget.

BudgetNull = data[data['budget'].isnull()]
BudgetNotNull = data[data['budget'].notnull()]

Then, calculate the mean budget of each genre based on the BudgetNotNull dataset.

budget_of_genre = BudgetNotNull.groupby('genres')['budget'].mean()

Finally, I'd like to fill in the budget of BudgetNull based on its genre.

For instance, 'Marine Boy' is a action movie, therefore, fill in budget_of_genre['Action'].

How do I do this via for loop? Or there's other ways?

Using a transform as follows could work:

df["budget"] = df.groupby("genres")["budget"].transform(lambda x: x.fillna(x.mean()))

The mean calculation uses only the non-null values in its calculation. So the mean of each group's non-null values are imputed to that same group's null values.

See also this question on Stack Overflow.


Plotting Weibull distribution on Wind Speed I have a 720 hourly set of wind speed and wind direction data and I want to fit the Weibull Distribution on it. After searching for some time, I wrote the following code in Python to get my distribution, I will also share my image for clarification.

ws.hist(bins=np.arange(0, ws.max()), alpha=0.5, normed=True)
weibull_params = (sc.stats.exponweib.fit(ws,floc=0,f0=1))
x= ws
def weib(x,lamb,k):
   return (k / lamb) * (x / lamb)**(k-1) * np.exp(-(x/lamb)**k)
k_shape, lamb_scale = weibull_params[1], weibull_params[3]

plt.plot(x, weib(x, lamb_scale, k_shape), label='self-defined Weibull')

Does this seem alright? How do I make sense of this? I also found a package here but I don't understand how to implement it to my code. https://github.com/cqcn1991/Wind-Speed-Analysis

Thank you.

Tried a few things in the OP comments that didn't work.  However, you may apply Wolfram Language to your project. There is a free Wolfram Engine for developers and with the Wolfram Client Library for Python you can use these functions in Python.

import datetime

from wolframclient.evaluation import WolframLanguageSession
from wolframclient.language import wl, wlexpr

Start the Wolfram session

wolfSession = WolframLanguageSession()

I will collect my own WindSpeedData from a WeatherData weather station closest to the "Country" Entity of Bermuda during October 2016.

weatherStation = wolfSession.evaluate(
    wl.WeatherData(wl.Entity('Country','Bermuda'))
);
print(weatherStation)

Entity['WeatherStation', 'TXKF']

windData = wolfSession.evaluate(
    wl.WindSpeedData(
        weatherStation,
        [
            datetime.datetime(2016,10,1),
            datetime.datetime(2016,10,31)
        ]
    )
)

windData is a TimeSeries object with observation count comparable to your problem.

print(wolfSession.evaluate(windData('PathLength')))

1058

The result can be visualised with DateListPlot by Exporting in one of the supported Raster Image Formats or Vector Graphics Formats.

wolfSession.evaluate(
    wl.Export(
        '<path with image filename>',
        wl.DateListPlot(windData,
            PlotTheme='Detailed',
            PlotRange=wl.All,
            FrameLabel=wl.Automatic
        )
    )
)

A hurricane passed over in that month with is the cause of that spike in wind speed.

Now that there is data Histogram on the PDF scale gives.

hist=wolfSession.evaluate(
    wl.Histogram(windData, wl.Automatic, 'PDF',
        PlotTheme='Detailed',
        ChartStyle=wl.ColorData('Crayola','Silver'),
        PlotRange=wl.All
    )
);

wolfSession.evaluate(
    wl.Export(
        '<path with image filename>',
        hist
    )
)

Use EstimatedDistribution for the WeibullDistribution.

windDistrbution=wolfSession.evaluate(
    wl.EstimatedDistribution(
        wl.QuantityMagnitude(windData('Values')),
        wl.WeibullDistribution(wl.Global.alpha, wl.Global.beta, wl.Global.mu)
    )
);
print(windDistrbution)

WeibullDistribution[1.883495945177254, 28.34295076324276, -0.7675654467340361]

Or use FindDistribution restricted to the continuous distribution functions to auto find and fit a distribution.

autoDistribution=wolfSession.evaluate(
    wl.FindDistribution(
        wl.QuantityMagnitude(windData('Values')),
        TargetFunctions='Continuous'
    )
);

print(autoDistribution)

ExtremeValueDistribution[18.436141153779957, 10.204118744677338]

Check for goodness of with DistributionFitTest.

print(
    wolfSession.evaluate(
        wl.Map(
            wl.Function(wl.DistributionFitTest(windData,wl.Slot(1),'PValue')),
            [windDistrbution,autoDistribution]
        )
    )
)

[0.0002862317707539308, 0.1802410209978217]

The Weibull fit is not very good and Extreme Value not that strong.  In any case the Plot the PDF functions.

pdfs=wolfSession.evaluate(
    wl.Map(
        wl.Function(wl.PDF(wl.Slot(1), wl.Global.x)),
        [windDistrbution,autoDistribution]
    )
);

pdfPlot=wolfSession.evaluate(
    wl.Plot(
        pdfs,
        [wl.Global.x, 0, wl.QuantityMagnitude(wl.Max(windData))],
        PlotTheme='Detailed',
        PlotLegends=wl.Placed(
            wl.LineLegend(wl.Automatic, [windDistrbution,autoDistribution]),
            wl.Below)
    )
)

wolfSession.evaluate(
    wl.Export(
        '<path with image filename>',
        pdfPlot
    )
)

Combine the plots with Show.

wolfSession.evaluate(
    wl.Export(
        '<path with image filename>',
        wl.Show(hist, pdfPlot)
    )
)

Terminate the session

wolfSession.terminate()

Hope this helps.


Keras multi GPU in vast.ai I am trying to run a keras model on vast.ai using multiple GPUs. For that I am using keras.utils.multi_gpu_model , however I keep having this error:

if multi_GPU and n_GPUs > 1:
    model = multi_gpu_model(model)

  AttributeError: module 'tensorflow_core._api.v2.config' has no
  attribute 'experimental_list_devices')

I am using this default docker :

Official docker images for deep learning framework TensorFlow

  Successfully loaded tensorflow/tensorflow:nightly-gpu-py3

I have also checked the available GPUs and all the GPUs are detected correctly:

Any ideas?

cheers

Solution:

Finally I found the solution myself.  I just used another docker image with an older version of tensorflow (2.0.0), and the error disappeared.


No statistical significance but observable trends I have a general inference question regarding scenarios when results from data are not statistically significant but there appears to be an observable trend.

For example, treatment A and treatment B are applied to 2 independent populations.  Using a ttest to analyze the resulting data (lets say the data is total revenue), the p value == .2, so the effect of treatment on revenue was not statistically significant. However, the total revenue from treatment A was observably higher in treatment B. What can I say in this regard?

I've had academic advisors recommend saying 'While the effect of treatment was not significant, a trend was observed', and then one would go on describing the trend. Is this an adequate viewpoint, or a statistical folly? What conclusions would a data scientist in industry draw and present to stakeholders from a scenario like this?

There is high variance within each group. Even though there is a mean difference between the groups, there is a high amount of spread within just treatment A or just treatment B.

From a statistical point of view, the difference between the groups could be due to chance because of the large spread relative to the small mean difference.

Due to the amount of randomness observed - if we did ran the experiment again, treatment B mean could be higher than treatment A mean.


Why did Tufte call this a "superbly produced duck"? I think I understand Tufte's concept of a "Duck" -- A graphic that is taken over by decorative forms.

But I couldn't understand why he called this a duck (a "superbly produced" one at that). It seemed to me more functional than decorative.

Thoughts?

If you assume duck to mean "irrelevant decorative elements" there are a few things that strike me as likely: (1) the "squares" only roughly indicate location of the target area; (2) squares represent volume, which is incongruent when overlaid on 2d geography; (3) shading of mountains/geographic features doesn't add detail.

Also problematic: comparison of h20 volume applied to crop types is difficult to discern using grids. Quick, which crop color uses more water??

If you want to compare volume to crop type then you should use a plain column chart, where it is easy to distinguish relative heights. This is analogous to why bar/column charts are preferable to pie charts - humans are better at comparing heights/lengths vs area/volume. If, on the other hand, the point of the graph is to compare crop type irrigation by area, you would want to display a column graph with regions next to each other, but grouped by crop type.

As far as a superbly produced duck, my guess is that the Applied Irrigation Water  is a duck, but a superb duck, at least when compared to the monstrous duck on the preceding page of the Visual Display of Quantitative Information 2nd Ed.


When to use One Hot Encoding vs LabelEncoder vs DictVectorizor? I have been building models with categorical data for a while now and when in this situation I basically default to using scikit-learn's LabelEncoder function to transform this data prior to building a model.

I understand the difference between OHE, LabelEncoder and DictVectorizor in terms of what they are doing to the data, but what is not clear to me is when you might choose to employ one technique over another.

Are there certain algorithms or situations in which one has advantages/disadvantages with respect to the others?

There are some cases where LabelEncoder or DictVectorizor are useful, but these are quite limited in my opinion due to ordinality.

LabelEncoder can turn [dog,cat,dog,mouse,cat] into [1,2,1,3,2], but then the imposed ordinality means that the average of dog and mouse is cat. Still there are algorithms like decision trees and random forests that can work with categorical variables just fine and LabelEncoder can be used to store values using less disk space.

One-Hot-Encoding has the advantage that the result is binary rather than ordinal and that everything sits in an orthogonal vector space.  The disadvantage is that for high cardinality, the feature space can really blow up quickly and you start fighting with the curse of dimensionality. In these cases, I typically employ one-hot-encoding followed by PCA for dimensionality reduction.  I find that the judicious combination of one-hot plus PCA can seldom be beat by other encoding schemes.  PCA finds the linear overlap, so will naturally tend to group similar features into the same feature.


An optimization problem involving perceptron What I do not understand is that is there only one input for each weights or there is a vector of inputs from 1 to n?

I also dont get the notation for inputs and outputs? Is that a notation or a power operation?

The perceptron receives a $n$ number of $m$-dimensional vectors, so, $x^{i} \in \ R^{m}\  \forall\ i \in [1,n]$.

To iterate, the dataset has $n$ different examples and each of which has $m$ dimensions. It's not a power notation, it just denotes the $i$th example.


How to deal with millions or rows of data for analysis/visualization purpose I have data in 2 tables in Sql server.
First table has around 10 million rows and 8 columns.
Second table has 6 million rows and 60 columns.

I want to import those tables into a Python notebook using pandas ( I am importing in "chunksize")  and then merge them and then run analysis on resulting table.

I am unable to import the data due to possible hardware constraints.
System hardware configuration is as follows:

RAM: 4GB
Storage: 160 GB,
CPU : Dual core CPU.

Even if the import goes through (which seems difficult), my resulting table after the merge will have 5 million rows and 40 columns.

Is it feasible to perform analysis/visualization in python notebook using pandas, seaborn on columns on the resulting table?

Would love to understand:

What should be my next steps to solve the problem?

Is installation of Anaconda on Windows 2012 server feasible? (This server has more memory)

Pandas load everything into memory before it starts working and that is why your code is failing as you are running out of memory. One way to deal with this issue is to scale your system i.e. have more RAM but this is not a good solution as this method will eventually fail to scale.

Other option will be to use big data libraries like spark or flink for this on a distributed system. You can try the installation on windows sever but that also wont scale well for you.


PCA on MNIST dataset results in very poor performance I am trying to build a model for classifying MNIST dataset using SVM. With raw features I am getting accuracy of around 94% (using linear kernel).

When I tried it with PCA, with different number of components (35,50,250,500) I am getting accuracy around 11%.

What can possibly the reason for this?

There are a few thing that you may be doing, but worth mentioning.

Did you "whiten" your data---mean center and normalize by the standard deviation? This step may be implementation-dependent.
http://deeplearning.stanford.edu/wiki/index.php/PCA

Did you check to see how much variablility was explained by the number of chosen parameters? For images +80% is common.

Make sure you understand the implementation of the PCA you are using. It always confuses me by the terminology different implementations uses, e.g. score, rotation, etc. I suggest following a tutorial and matching their output with your implementation to get a deeper understanding.
http://sebastianraschka.com/Articles/2015_pca_in_3_steps.html

Your model is now built on transformed data. To predict make sure you are using these same transformations on the new input data.


RNN unable to classify time series I have 400 time series of length 50.
200 of them have values between 1-10 and are considered of type A.
The rest 200 have values 1-10 with the exception that 3 from the total of 50 data points have value 20, and are considered of type B.

I am trying to use a RNN network to make it learn these differences using tensorflow.

I use GRU cell with input size is 50 (the whole time series), internal state is 100, 3 layers, feature length is 1 and dropout 0.8.
Batch size is 1 (1 whole time series of 50 values)
80% of the series are used for training and 20% for test, evenly distributed between type A and type B.

I use softmax as activation function and for gradient descent optimization i use RMSPropOptimizer. Also i am not changing the data at all before feeding it into the neural network, i tried normalizing them using (x-min()/max()-min()) but it didn't work.

When i am trying to train it, i always get 50% error. Maximum epochs i tried are 10 and it still was 50%.

The same network can classify correctly if i use it on data that have way more differences than just on these 3 points, even on small dataset of about 50 runs of each type.

Is it impossible for a RNN to learn these small differences on just those 3 data points out of the 50?
What would you suggest i can change in my configuration?
I also tried with 2000 time series and still the error is the same.

Also note that these 3 data points exist at the beginning of the time series. (in case this has something to do with my issue).

I let it run for more epochs, as suggested, and i noticed that after 23 epochs, rnn did get 0% error.

I then tried changing activation function of GRUCell from the default tanh to relu and it greatly speed up training. It took 1-4 epochs to get 0% error.


How to access substrings in pandas column and store it into new columns? I'm working on a dataset for building permits. In the dataset there is a column that gives the location (lattitude and longitude) for the building permit. The data in the location column look like this:

0    (37.785719256680785, -122.40852313194863)
1     (37.78733980600732, -122.41063199757738)
2      (37.7946573324287, -122.42232562979227)
3     (37.79595867909168, -122.41557405519474)
4     (37.78315261897309, -122.40950883997789)
Name: location, dtype: object

As you can see, the data is stored as strings. I wanted to store the lattitude and longitude in two separate columns, so I wrote the following code to accomplish this:

df.location = df.location.str.replace('(','')
df.location = df.location.str.replace(')','')

for i in range(len(df)):
    if df.location[i] == np.nan:
        df['lattitude'] = np.nan
    else:
        df['lattitude'] = df.location[i][0:df['location'][i].index(',')]

for i in range(len(df)):
    if df.location[i] == np.nan:
        df['longitude'] = np.nan
    else:
        df['longitude'] = df.location[i][0:df['location'][i].index(',')]

There is some missing data in the column, 1700 entries to be exact. So in order to avoid a key error, I wrote the if-else statement to fill in the new columns with np.nan anytime the loop would iterate to a missing entry.

When I ran the code, I got the following error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-72-81826748e81c> in <module>()
      7         df['lattitude'] = np.nan
      8     else:
----> 9         df['lattitude'] = df.location[i][0:df['location'][i].index(',')]
     10
     11 for i in range(len(df)):

AttributeError: 'float' object has no attribute 'index'

Can anybody spot the error I'm making?

This is perhaps more suited for StackOverflow. I would also use a better/more descriptive title for the question itself; that way others that are facing a similar problem are able to find it.

The reason you are seeing that error is because of the nan values, which are of type float. So while most of the rows in df['location'] contain strings, every row instance of an nan in the column is a float, and str.index() is not available for floats.

Your check of if df.location[i] == np.nan: is pointless, because np.nan == np.nan is always False due to the very definition of nan. Refer to this question on the topic. Because your check fails, the loop enters the else block and encounters a float object attempting to invoke a string method.

In my opinion you are using a very complicated approach to get what you want.

Replace your code with this. It should give you what you are looking for. Any nan values encountered will be handled by python.

df['location']=df['location'].str.replace(" ","").str.strip('(').str.strip(')')
df['latitude']=df['location'].str.split(',').str[0]
df['longitude']=df['location'].str.split(',').str[1]

I tested this using the following code segment:

df=pd.DataFrame()

df['location']=(
"(37.785719256680785, -122.40852313194863)",
"(37.78733980600732, -122.41063199757738)",
"(37.7946573324287, -122.42232562979227)",
"(37.79595867909168, -122.41557405519474)",
"(37.78315261897309, -122.40950883997789)",
np.nan,
"(37.78615261897309, -122.405550883997789)")

df['location']=df['location'].str.replace(" ", "").str.strip('(').str.strip(')')
df['latitude']=df['location'].str.split(',').str[0]
df['longitude']=df['location'].str.split(',').str[1]

print(df[['latitude','longitude']])

This produces the output:

             latitude             longitude
0  37.785719256680785   -122.40852313194863
1   37.78733980600732   -122.41063199757738
2    37.7946573324287   -122.42232562979227
3   37.79595867909168   -122.41557405519474
4   37.78315261897309   -122.40950883997789
5                 NaN                   NaN
6   37.78615261897309  -122.405550883997789


Is it possible to have stratified train-test split of a set based on two columns? Consider a dataframe that contains two columns, text and label. I can very easily create a stratified train-test split using sklearn.model_selection.train_test_split. The only thing I have to do is to set the column I want to use for the stratification (in this case label).
Now, consider a dataframe that contains three columns, text, subreddit, and label. I would like to make a stratified train-test split using the label column, but I also want to make sure that there is no bias in terms of the subreddit column. E.g., it's possible that the test set has way more comments coming from subreddit X while the train set does not.
How can I do this in Python?

One option would be to feed an array of both variables to the stratify parameter which accepts multidimensional arrays too. Here's the description from the scikit documentation:

stratify array-like, default=None
If not None, data is split in a stratified fashion, using this as the class labels.

Here is an example:
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

# create dummy data with unbalanced feature value distribution
X = pd.DataFrame(np.concatenate((np.random.randint(0, 3, 500), np.random.randint(0, 10, 500)),axis=0).reshape((500, 2)), columns=["text", "subreddit"])
y = pd.DataFrame(np.random.randint(0,2, 500).reshape((500, 1)), columns=["label"])

# split stratified to target variable and subreddit col
X_train, X_test, y_train, y_test = train_test_split(
    X, pd.concat([X["subreddit"], y], axis=1), stratify=pd.concat([X["subreddit"], y], axis=1))

# remove subreddit cols from target variable arrays
y_train = y_train.drop(["subreddit"], axis=1)
y_test = y_test.drop(["subreddit"], axis=1)

As you can see the split is stratified to subreddit too:
Train data shares for subreddits
X_train.groupby("subreddit").count()/len(X_train)

gives
text
subreddit
0   0.232000
1   0.232000
2   0.213333
3   0.034667
4   0.037333
5   0.045333
6   0.056000
7   0.056000
8   0.048000
9   0.045333

Test data shares for subreddits
X_test.groupby("subreddit").count()/len(X_test)

gives
text
subreddit
0   0.232
1   0.240
2   0.208
3   0.032
4   0.032
5   0.048
6   0.056
7   0.056
8   0.048
9   0.048

Naturally, this only works if you have sufficient data to stratify to subreddit and the target variable at the same time. Otherwise scikit learn will throw an exception.


How to make a CNN predict a continuous value? I understand the dimensionality of convolutions, max pooling and dense as function of stride and kernel size. But I'm having trouble wrapping my head around how to use these layers to end up with my final prediction, which should be one continuous variable.

Here is what I'm working with:

X training: n 2-channel 3D grids of size 46x46x46, shape: (n, 46, 46, 46, 2)
Y training: n-element vector of continuous values, shape: (n,)

I would imagine there will be some resizing and some concatenation involved. But there's no point doing that if I don't actually understand what it's doing.

A common way to do this is flatten your output after your last convolution layer, and pass it through a fully connected layer.


how to calculate similarity between users based on movie ratings Hi I am working on a movie recommendation system and I have to find alikeness between the main user and other users. For example, the main user watched 3 specific movies and rated them as 8,5,7. A user who happened to watch the same movies rated them as 8,2,3 and an another user of the same kind rated those movies as 7,6,6 and some other user only watched first two movies and he rated them as 8,5. Now the question is to which user the main user is close. I tried to come up with functions, but they are prone to fail. Can you help me?

Use euclidean distance
1
$d(p,q) = \sqrt{\sum_{i=1}^{n}(q_i - p_i)^2}$
and check main user vs all other users.


Problem with a feature (normal distribution + peak around 0) I have a feature that shows a characteristic of the instances. That characteristic can be present or not. If present it shows an almost normal distribution of values (actually a bit skewed to the right, but with a log transformation it becomes normalized). When the characteristic is not present in the instance, the value of the feature is just 0.
So at the end, I have a distribution with a lot of instances with value 0 and a bit far right from it the almost-normal distribution. I would like to split it in two different features: one that shows the absence/presence of the characteristic (easy), and a second that shows only a normal distribution without the annoying peak around zero.

Aren't you providing the answer? You can split the feature in two, namely, if feature_to_split is the feature you're talking about, you can create feature_to_split_ispresent which will take either 1 or 0 depending on the presence or absence of that specific characteristic, and feature_to_split_value which will take the actual value of that characteristic.


What are the best tools for manually annotating textual topic segmentation? I checked this old question on tools for text corpus annotation and there are many good tools there, however the tools there seem not to cover my problem.

I have a linear text segmentation problem, in which a text needs to be segmented in different sections (topics). I'm building a model for it, but I have no training data. Therefore, the team decided to manually label some text, by annotating blocks in the text that represent each section. I tried some NER or POS labelling tools, but they are not very convenient for selecting several lines and paragraphs to annotate a label.

Is there a good tool for human annotation of text segmentation?

Since I couldn't find a suitable tool, so I made a tool for textual topic segmentation.

To anyone interested in this question, I forked YEDDA's entity recognition project and changed it, accordingly. You can check it here in GitHub: https://github.com/ThiagoSousa/YEEDASeg. My project is called YEDDASeg.

Edit: People mentioned to use BRAT, but it is good to select a word and tag with with a label. My case is a bit more complicated, because it can be many paragraphs. In YEDDASeg, you basic just click on the text and press a key to label a topic in the text. It is simpler for this purpose.


LSTM vs ARIMA for demand prediction I'm new to the field of time series prediction.
I'm looking for a demand prediction model to predict when the product will be sold out from the online supermarket (when the supply is known in advance).

I know that LSTM and ARIMA are the "best" model for time series prediction. Is there any other model for demand prediction which could be better?

and what are the con and pros for those model for demand prediction?

Also, can someone give me some time frames to developing this kind of model? due I need to consider anything else than things in the regular model (like hyper-parameters tuning and etc.)

ARIMA models are basically linear models, so they can only work if the relation is linear (or linear integrated). They are easy to estimate.

LSTM can model more or less any relationship, but at the cost of additional computation and require more data to be trained.

So start with ARIMA if you don't know anything about our data, if it doesn't work, use a simple LSTM model and then complexity it to match what you need with the test set.


BERT embedding layer I am trying to figure how the embedding layer works for the pretrained BERT-base model. I am using pytorch and trying to dissect the following model:
import torch
model = torch.hub.load('huggingface/pytorch-transformers', 'model', 'bert-base-uncased')
model.embeddings

This BERT model has 199 different named parameters, of which the first 5 belong to the embedding layer (the first layer)
==== Embedding Layer ====

embeddings.word_embeddings.weight                       (30522, 768)
embeddings.position_embeddings.weight                     (512, 768)
embeddings.token_type_embeddings.weight                     (2, 768)
embeddings.LayerNorm.weight                                   (768,)
embeddings.LayerNorm.bias                                     (768,)

As I understand, the model accepts input in the shape of [Batch, Indices] where Batch is of arbitrary size (usually 32, 64 or whatever) and Indices are the corresponding indices for each word in the tokenized input sentence. Indices has a max length of 512. One input sample might look like this:
[[101, 1996, 4248, 2829, 4419, 14523, 2058, 1996, 13971, 3899, 102]]

This contains only 1 batch and is the tokenized form of the sentence "The quick brown fox jumps over the lazy dog".
The first word_embeddings weight will translate each number in Indices to a vector spanned in 768 dimensions (the embedding dimension).
Now, the position_embeddings weight is used to encode the position of each word in the input sentence. Here I am confused about why this parameter is being learnt? Looking at an alternative implementation of the BERT model, the positional embedding is a static transformation. This also seems to be the conventional way of doing the positional encoding in a transformer model. Looking at the alternative implementation it uses the sine and cosine function to encode interleaved pairs in the input. I tried comparing model.embeddings.position_embeddings.weight and pe, but I cannot see any similarity. The in the last sentence under under A.2 Pre-training Procedure (page 13) the paper states

Then, we train the rest
10% of the steps of sequence of 512 to learn the
positional embeddings.

Why is the positional embedding weight being learnt and not predefined?
The next layer after the positional embedding is the token_type_embeddings. Here I am confused about how the segment label is inferred by the model. If I understand this correctly each input sentence is delimited by the [SEP] token. In the example above there is only 1 [SEP] token and the segment label must be 0 for that sentence. But there could be a maximum of 2 segment labels. If so, will the 2 segments be handled separately or are they processed in parallel all the same as one "array"? How does the model handle multiple sentence segments?
finally the output from theese 3 embeddings are added togheter and passed through layernorm which I understand. But, are the weights in these embedding layers adjusted when fine-tuning the model to a downstream task?

Why are positional embeddings learned?
This was asked in the repo of the original implementation without an answer. It didn't get an answer either in the HuggingFace Transformers repo and in cross-validated, also without answer, or without much evidence.
Given that in the original Transformer paper the sinusoidal embedding were the default ones, I understand that during preliminary hyperparameter tuning, the authors of BERT decided to go with learned embeddings, deciding not to duplicate all experiments with both types of embeddings.
We can, nevertheless, see some comparisons between learned and sinusoidal positional embedding in the ICLR'21 article On Position Embeddings in BERT, where the authors observe that:

The fully-learnable absolute PE performs better in classification,
while relative PEs perform better in span prediction.

How does the model handle multiple sentence segments?
This is best understood with the figure of the original BERT paper:

The two sentences are encoded into three sequences of the same length:

Sequence of subword tokens: the sentence tokens are concatenated into a single sequence, separating them with a [SEP] token. This sequence is embedded with the subword token embedding table; you can see the tokens here.
Sequence of positional embedding: sequentially increasing positions form the initial position of the [CLS] token to the position of the second [SEP] token. This sequence is embedded with the positional embedding table, which has 512 elements.
Sequence of segment embeddings:  as many EA tokens as the token length of the first sentence (with [CLS] and [SEP]) followed by as many EB tokens as the token length of the second sentence (with the [SEP]). This sequence is embedded with the segment embedding table, with has 2 elements.

After embedding the three sequences with their respective embedding tables, we have 3 vector sequences, which are added together and used as input to the self-attention layers.
Are the weights in these embedding layers adjusted when fine-tuning the model to a downstream task?
Yes, they are. Normally, all parameters are fine-tuned when fine-tunine a BERT-based model.
Nevertheless, it is also possible to simply use BERT's representations as input to a classification model, without fine-tuning BERT at all.
In this article you can see how these two approaches compare. In general, for BERT, you obtain better results by fine-tuning the whole model.


Are Deep Neural Networks limited to grayscale images depending on whether you use Seq. or Func. API? When I say DNN, I mean the simple usage of densely connected neurons (not CNN).
Say we are using Keras (with Tensorflow backend), the input_dim, using the Sequential API, can only take an integer or None value which restricts it to taking grayscale images right?
However since the functional API can accept dimensions that aren't just restricted to integers I would assume that it can take RGB images as opposed to only grayscale images?
If anyone can confirm or support, that would be great?

The answer is no, they are not limited.
However, your statements seem to contain multiple misunderstandings, so let's first clarify them:

The sequential and functional APIs in Keras are different approaches for structuring the layers of a neural network. Both can have dense layers and convolutional layers.

Convolutional layers exploit information locality and therefore normally perform better on images, where information locality is key.

Dense layers, on the other hand, can only handle vectors as input, therefore, anything we want to feed as input to a dense layer must be a vector; we can feed image data as input to a dense layer, but we must first remove the spatial organization (i.e. flattening), that is, we must lose the information about which pixel is next to another, and which channel it is in.

With this information in mind, we can conclude that, while we can use images, either grayscale or color, as input to dense layers, we need to flatten the image as a vector, which makes dense layers in general not very appropriate to receive images as input, because the locality information is not used by them.
Update: regarding the ability for dense layers to accept inputs of more than one dimension, the multiplication takes place along the last dimension, as explained in the documentation:

Note: If the input to the layer has a rank greater than 2, then Dense computes the dot product between the inputs and the kernel along the last axis of the inputs and axis 1 of the kernel (using tf.tensordot). For example, if input has dimensions (batch_size, d0, d1), then we create a kernel with shape (d1, units), and the kernel operates along axis 2 of the input, on every sub-tensor of shape (1, 1, d1) (there are batch_size * d0 such sub-tensors). The output in this case will have shape (batch_size, d0, units).

But this is applicable to both the functional and sequential APIs.


remove seasonality from weekly time series data I need to decompose a series to remove seasonality. The series has 2 columns date and volume.

This is what my time series object looks like:

salestsDec <- ts(salests, frequency=52, start=c(2010, 1), end=c(2014,12))

I ran the decompose() function on a 'ts' object.

salests = sales[, c(1,6)]
View(salests)
salestsDec <- ts(salests, frequency=52, start=c(2010, 1), end=c(2014,12))
salestsDec <- decompose(salestsDec, type=c("additive"))
plot(salestsDec)

Upon, running the decompose() function, I get a list of 6 components, observed, trend, seasonal, random for both date and volume. I should only be seeing, observed, trend, seasonal and random component for Volume in my plot.

I've attached an image of what the plot looks like.

Moreover, when I try to remove seasonal component from the series, I am getting an error. It appears that it's the same underlying issue.

Error:

Error in salests - salestsDec$seasonal : non-numeric argument to binary operator In addition: Warning message: Incompatible methods ("Ops.data.frame", "Ops.ts") for "-"

I can't replicate (without having the data), but from what I see, you are trying to apply the decompose function to the entire object - it's probably expecting a single vector, but receives a data-frame-like object and attempts to apply the decomposition to both columns. either use a different time series package (like zoo) which does not store timestamps as a column, or apply decompose explicitly to salestsDec[,2].


Simple Q-Table Learning: Understanding Example Code I'm trying to follow a tutorial for Q-Table learning from this source, and am having difficulty understanding a small piece of the code. Here's the entire block:

import gym
import numpy as np

env = gym.make('FrozenLake-v0')

#Initialize table with all zeros
Q = np.zeros([env.observation_space.n,env.action_space.n])
# Set learning parameters
lr = .8
y = .95
num_episodes = 2000
#create lists to contain total rewards and steps per episode
#jList = []
rList = []
for i in range(num_episodes):
    #Reset environment and get first new observation
    s = env.reset()
    rAll = 0
    d = False
    j = 0
    #The Q-Table learning algorithm
    while j < 99:
        j+=1
        #Choose an action by greedily (with noise) picking from Q table
        a = np.argmax(Q[s,:] + np.random.randn(1,env.action_space.n)*(1./(i+1)))
        #Get new state and reward from environment
        s1,r,d,_ = env.step(a)
        #Update Q-Table with new knowledge
        Q[s,a] = Q[s,a] + lr*(r + y*np.max(Q[s1,:]) - Q[s,a])
        rAll += r
        s = s1
        if d == True:
            break
    #jList.append(j)
    rList.append(rAll)

print "Score over time: " +  str(sum(rList)/num_episodes)

print "Final Q-Table Values"
print Q

The code runs well and I'm able to print my results, but here is where I'm having difficulties:

a = np.argmax(Q[s,:] + np.random.randn(1,env.action_space.n)*(1./(i+1)))

My question is, why are we multiplying by 1/(i+1)? Is this supposed to be an implementation of epsilon annealing? Any help is appreciated.

My question is, why are we multiplying by 1/(i+1)? Is this supposed to be an implementation of epsilon annealing?

The code looks like a relatively ad-hoc* adjustment to ensure early exploration, and an alternative to $\epsilon$-greedy action choice. The 1/(i+1) factor is similar to decaying $\epsilon$, but not identical.

$\epsilon$-greedy with the same decay factor might look like this:

a = np.argmax(Q[s,:])
if epsilon/(1+math.sqrt(i)) > random.random():
    a = random.randrange(0, env.action_space.n)

The math.sqrt(i) is just a suggestion, but I feel that epsilon/(1+i) is probably too aggressive and would cut off exploration too quickly.

It is not something I have seen before when studying Q-Learning (e.g. in David Silver's lectures or Sutton & Barto's book). However, Q-Learning is not predicated on using any specific action choice, it just needs enough exploration in the behaviour policy. For the given problem adding some noise to a greedy selection obviously works well enough. Technically for guaranteed convergence tabular Q-Learning needs infinite exploration over infinite time steps. The code as supplied does indeed do that because the noise is unbound from the Normal distribution. So there is always some small finite chance of selecting an action with a relatively low action-value estimate and refining that estimate later.

However, the fast decay (1/episode number) and initial scaling factor for the noise are both hyperparameters that need tuning to the problem. You might prefer something more standard from the literature such as $\epsilon$-greedy, Gibbs sampling or upper-confidence-bound action selection (the example is quite similar to UCB, in that it adds to the Q-values before taking the max).

* Perhaps the approach used in the example has a name (some variation of "Noisy Action Selection") but I don't know it, and could not find it on a quick search.


What is the best solution to replace NaN values? I'm thinking about using the normal distribution of a specific column that has missing values and replace them by random values generated using the normal distribution function of numpy on that specific column ? Replacing by zeros or the mode doesn't really make sense sometimes... When is it relevant to do so ?

You are right in saying that replacing with a simple mean, mode... is a common but unreliable imputation strategy in many cases.
You have in scikit learn some utilities for imputation of missing values (have a look at https://scikit-learn.org/stable/modules/classes.html#module-sklearn.impute) using for instance the knn imputer as an additional strategy.
Take into account you cannot assume your feature of interest follows a normal distribution, so instead you can actually apply a kernel density estimator to model such distribution, see here: http://scikit-learn.org/stable/modules/density.html


I don't want to offend you in any way but the problem you mentioned above is not really a problem but an opinion that you presented on ROC vs F1 score, and I guess you want other people's opinion on the same.
I would highly suggest you to ask to the point questions next time since if the questions get downvoted you can get banned (I am not downvoting so don't worry), just trying to be informative.
Now from what I was able to grasp from your post, I think that you are having difficulty in choosing the right metric for evaluating the three neural net models you have shown results for.
AUC vs F1 :
In general, the ROC is for many different levels of thresholds and thus it has many F score values. F1 score is applicable for any particular point on the ROC curve.
You may think of it as a measure of precision and recall at a particular threshold value whereas AUC is the area under the ROC curve. For F score to be high, both precision and recall should be high.
Consequently, when you have a data imbalance between positive and negative samples, you should always use F1-score because ROC averages over all possible thresholds.
Conclusion :
So in your case particularly since I can clearly see through precision and recall values that your classes are highly imbalanced, use F1 score to choose the model instead of AUC-ROC score.


shapes (127,1) and (13,) not aligned: 1 (dim 1) != 13 (dim 0) i am try to find score of linear regression
it gives me this type error my code is below

from sklearn import datasets
bostan=datasets.load_boston()

x=bostan.data
y=bostan.target

from sklearn import preprocessing

x_scale=preprocessing.scale(x)

yfrom sklearn import model_selection

x_train,x_test,y_train,y_test=model_selection.train_test_split(x_scale,)

from sklearn.linear_model import LinearRegression

clf=LinearRegression()

clf.fit(x_train,y_train)

y_pred=clf.predict(x_test)

y_pred=y_pred.reshape(-1,1)

y_test=y_test.reshape(-1,1)

y=clf.score(y_pred,y_test)   # problem is in this line
```

The score method of the classifier object does not work the way you are trying it to. You need to directly give x_test as input and that it will calculate y_pred on its own and give you the result with y_test. So, you do not need to reshape and the correct syntax would be:

y = clf.score(x_test, y_test)

If for any reason you need to calculate the score using y_pred then take a look at sklearn.metrics module.


Clustering not working as expected I have clusters as shown in the picture below.

The data is 2d : the two parameters are error and time.
I tried using the following clustering algorithms:
1) kmeans:clusters are spherical.
This algo clustered as follows:.
Kmeans did form clusters as needed.
2)dbscan: problem- I have clusters of varying density. It assigned every data point to noisy cluster I.e. it assigned a label of -1 to every point.
3) GMM: this also did not cluster the data points as needed. It performed clustering similar to kmeans.
4)affinity propagation is throwing a memory error.

Can someone please suggest a way of assigning different labels to each of the visually separable cluster.

Scaling matters. The y values will be ignored because of the x axis.
Parameters matter. You probably used DBSCAN with the sklearn default parameter values which are utter nonsense for anything but 2d normalized toy data.
Asking the right question matters. What are clusters here supposed to be anyway? I don't see any clusters. You show a time series with plenty of missing data, but there are no clusters. You seem to want to split your time series where you have missing values, but that is not clustering. You likely can abuse clustering such as DBSCAN here (just try minpts=1, epsilon=100 or similar) but that is an incredibly inefficient way to solve this, because you abuse a multivariate approach for a trivial one-dimensional problem.

Sort by time. If the next timestamp is much larger than the previous, this is the end of a subintervals (I wouldn't call these clusters). That is easily done with a for loop.


Is there any domain where Bayesian Networks outperform neural networks? Neural networks get top results in Computer Vision tasks (see MNIST, ILSVRC, Kaggle Galaxy Challenge).  They seem to outperform every other approach in Computer Vision. But there are also other tasks:

Kaggle Molecular Activity Challenge
Regression: Kaggle Rain prediction, also the 2nd place
Grasp and Lift 2nd also third place - Identify hand motions from EEG recordings

I'm not too sure about ASR (automatic speech recognition) and machine translation, but I think I've also heard that (recurrent) neural networks (start to) outperform other approaches.

I am currently learning about Bayesian Networks and I wonder in which cases those models are usually applied. So my question is:

Is there any challenge / (Kaggle) competition, where the state of the art are Bayesian Networks or at least very similar models?

(Side note: I've also seen decision trees, 2, 3, 4, 5, 6, 7 win in several recent Kaggle challenges)

One of the areas where Bayesian approaches are often used, is where one needs interpretability of the prediction system. You don't want to give doctors a Neural net and say that it's 95% accurate. You rather want to explain the assumptions your method makes, as well as the decision process the method uses.

Similar area is when you have a strong prior domain knowledge and want to use it in the system.


Keras - Computing cosine similarity matrix of two 3D tensors Using TF backend, I need to construct a similarity matrices of two 3D vectors, both with shape (batch_size, N, M), being N and M natural numbers.

The function tf.losses.cosine_distance is only between 1D tensors. I need to build a Tensor matrix batch_sizexNxM such that matrix[k][i][j] will be the cosine similarity of the Tensor1[k][i] and Tensor2[k][j].

How can I do this?

I know of no pairwise distance operations in Keras or tensorflow. But the matrix math can be implemented in TF/Keras backend code and then placed in a Keras layer.

Here's the matrix representation of the cosine similarity of two vectors:
$$
cos(\theta) = \frac{\mathbf{A}\cdot\mathbf{B}}{\|\mathbf{A}\|_2 \|\mathbf{B}\|_2}
$$

I'll show the code and a test that confirms that it works. First, generate non-trivial test data.

import numpy as np
import keras
import keras.backend as K

# set up test data
n_batch = 100
n = 400 # number of points in the first set
m = 500 # number of points in the second set
d = 200 # number of dimensions

A = np.random.rand(n_batch, n, d)
B = np.random.rand(n_batch, m, d)

Define pairwise cosine similarity function.

# convenience l2_norm function
def l2_norm(x, axis=None):
    """
    takes an input tensor and returns the l2 norm along specified axis
    """

    square_sum = K.sum(K.square(x), axis=axis, keepdims=True)
    norm = K.sqrt(K.maximum(square_sum, K.epsilon()))

    return norm

def pairwise_cosine_sim(A_B):
    """
    A [batch x n x d] tensor of n rows with d dimensions
    B [batch x m x d] tensor of n rows with d dimensions

    returns:
    D [batch x n x m] tensor of cosine similarity scores between each point i<n, j<m
    """

    A, B = A_B
    A_mag = l2_norm(A, axis=2)
    B_mag = l2_norm(B, axis=2)
    num = K.batch_dot(A_tensor, K.permute_dimensions(B_tensor, (0,2,1)))
    den = (A_mag * K.permute_dimensions(B_mag, (0,2,1)))
    dist_mat =  num / den

    return dist_mat

Build a shallow Keras model around the function.

# build dummy model
A_tensor = K.constant(A)
B_tensor = K.constant(B)
A_input = keras.Input(tensor=A_tensor)
B_input = keras.Input(tensor=B_tensor)
dist_output = keras.layers.Lambda(pairwise_cosine_sim)([A_input, B_input])
dist_model = keras.Model(inputs=[A_input, B_input], outputs=dist_output)
dist_model.compile("sgd", "mse")

Compare to sklearn implementation

sk_dist = np.zeros( (n_batch, n, m) )
for i in range(n_batch):
    sk_dist[i,...] = cosine_similarity(A[i,...], B[i,...])

keras_dist = dist_model.predict(None, steps=1)
np.allclose(sk_dist, keras_dist)
> True


Normalize the output of a dense layer with linear activation I have the following architecture of my network:

def net_one(message):
    weight1 = np.random.normal(loc=0.0, scale=0.01, size=[16, 16])
    init1 = tf.constant_initializer(weight1)
    out1 = tf.layers.dense(inputs=message, units=16, activation=tf.nn.relu, kernel_initializer=init1)
    weight2 = np.random.normal(loc=0.0, scale=0.01, size=[16, 7])
    init2 = tf.constant_initializer(weight2)
    out2 = tf.layers.dense(inputs=out1, units=7, activation=None, kernel_initializer=init2)
    return out2

Now as the output of the network is linear (None in tensorflow corresponds to a linear activation function), the output is unbounded. I need the square of the 2-norm of the output to be a constant, n (for energy constraint purposes). I do not want to use sigmoid or tanh as they hamper the performance. I tried the following:

code = net_one(input_bits)
code = code * tf.sqrt(n) / tf.linalg.norm(code)

I have two questions:

Does it achieve what I expect it to achieve?
Is there any better way (if this is indeed right) or any alternate way to achieve this?

Your way seems to be correct. I would suggest one other way that you might want to try (maybe it won't make your life better, but still): Since you want your data points to basically live on a sphere, you could train the angles in spherical coordinates. Fix the radius to $\sqrt{n}$ and this way your network has to learn one dimension fewer. I'm not sure how this will affect performance though.
One way to implement this in tensorflow could be
def spherical(input):
    layer = []

    for i in range(input.get_shape()[1]+1):
        elem = 1
        for j in range(i):
            elem = elem*tf.sin(input[:,j])
        if i < input.get_shape()[1]:
            elem = elem*tf.cos(input[:,i])
        layer.append(elem)

    return tf.transpose(tf.convert_to_tensor(layer))

This will map any input (a tensor with shape=(ndatapoints, ndims)) to the unit sphere in ndims+1. By multiplication and shifting you could map it to any sphere you want. It is probably not the most elegant way, but it gets the job done. The mapping is not injective, if you would need it to be, you would have to make sure the elements of input stick to the respective intervals.
If you end up trying this, let me know how it went please ;)


Spatial clustering based on response to inputs and building a reduced model Stats version: I have a few measurements of a function that takes three inputs and produces a few 2D fields of outputs: $f_i(a,b,c;x,y)$, with $f$ being a vector of several quantities. I would like to cluster the points in $(x,y)$-space into regions that have a similar response to the input variables $(a,b,c)$. If I treat the points independently, applying a clustering algorithm is fairly easy. (I can elaborate on which one I'm using if you think it's relevant, but I'm open to suggestions on your favorites.) But there's no guarantee that the clusters are near each other spatially. Having contiguous clusters of roughly equal size is definitely a plus. What are some ways to modify the approach to add in that constraint?

Science version: I'm running an atmospheric chemistry model and varying emissions over the US. In that setup, $(a,b,c)$ are the national-total emissions of NOx, SO2, and NH3; $(x,y)$ are grid points over the US; and $f_i$ is a vector of the various chemical species I'm tracking. I've varied $(a,b,c)$ in a number of simulations. Now I want to divide the country into regions that have broadly similar responses to the emissions changes, and create simplified models for a manageable number of regions. I'm particularly interested in nonlinearities/mixed-effects/second-order derivatives in the chemistry response. It's possible that different regions of the country do respond similarly to emissions even if they're not near each other, but the different regions should have some sort of spatial agglomeration. Thoughts?

Side question: I originally played around with this a few years ago with Matlab code but have since migrated to Python. Suggestions on how to efficiently use Python stats packages for this sort of problem?

I'm not sure I'm following your question completely, but for spatial type clustering maybe a Self-Organizing Map would meet your needs. Basically it maps out the space based on nearby activations.

Side answer: I haven't used python


Grid Search Pool Classifiers for Dynamic Classifier Selection Ensembles I would like to grid search pool classifiers hyper parameter of OLA() ( Overall Local Accuracy ) model from deslib python package.
from sklearn.datasets import make_classification

from sklearn.model_selection import RepeatedStratifiedKFold

from sklearn.model_selection import cross_val_score

from deslib.dcs.ola import OLA

from sklearn.model_selection import GridSearchCV

from sklearn.linear_model import LogisticRegression

from sklearn.tree import DecisionTreeClassifier

from sklearn.naive_bayes import GaussianNB

Then :
X , y = make_classification( n_samples = 10000 , n_features = 20 , n_informative = 15 , n_redundant = 5 , random_state = 999 )

model = OLA()

cv = RepeatedStratifiedKFold( n_splits = 10 , n_repeats = 3 , random_state = 999 )

grid = dict()

grid[ 'pool_classifiers' ] = [ [ LogisticRegression() , DecisionTreeClassifier() , GaussianNB() ] ,
                               [ LogisticRegression() , DecisionTreeClassifier() ] ]

search = GridSearchCV( model , grid , scoring = 'accuracy' , cv = cv )

search_results = search.fit( X , y )

But the following error message is raised :
NotFittedError: This LogisticRegression instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator.

It means model in the pool must be fitted prior to grid searching but I thought fitting took place on each train fold of the cv step.
Does it mean I must fit myself each model on training folds ?
Thanks for helping on this topic.

Here is my solution to this problem :
Let's say I have the classifiers pools I want to cross validate :
grid = dict()

grid[ 'pool_classifiers' ] = [ ( 'pool_01' , [ LogisticRegression() , DecisionTreeClassifier() , GaussianNB() ] ) ,
                               ( 'pool_02' , [ LogisticRegression() , DecisionTreeClassifier() ] ) ]

Then I can use the following function :
def ola_cv( X , y , cv ) :

  scores = dict()

  for pool_classifiers in grid[ 'pool_classifiers' ] :

      scores[ pool_classifiers[ 0 ] ] = list()

  for train_ix , test_ix in cv.split( X , y ) :

    X_train = X[ train_ix , : ]
    y_train = y[ train_ix ]

    X_test = X[ test_ix , : ]
    y_test = y[ test_ix ]

    for pool_classifiers in grid[ 'pool_classifiers' ] :

      for model in pool_classifiers[ 1 ] :

        model.fit( X_train, y_train )

      ola = OLA( pool_classifiers = pool_classifiers[ 1 ] )

      ola.fit( X_train , y_train )

      y_pred = ola.predict( X_test )

      score = accuracy_score( y_test , y_pred )

      scores[ pool_classifiers[ 0 ] ].append( score )

  for k in scores.keys():

    print( f'pool_classifiers : {k} | accuracy : {np.mean(scores[ k ])} ({np.std(scores[ k ])})')

  return scores

to obtain for each classifiers pool mean accuracy ( and standard deviation ) on 10 * 3 = 30 folds.
NB : Mayne grid should be an argument of the ola_cv function.


how to create sklearn pipeline object using predtrained standardscalar object I am having pretrained Sklearn model and pre-trained Standard scalar object saved as pickle . And now I want to create Sklearn pipeline using both of it.
I need sklearn pipeline to convert it into ONNX format.
I couldnt do it as pipeline takes standard scalar class and then we need fit pipeline using data but in my  case models and scalar both are already fitted.

This works the way you would want out of the box.

pipeline takes standard scaler class

No, pipelines get initialized with estimator instances, not the classes.  (This is why you need the parentheses in the steps, e.g. StandardScaler().)
That is, the following works:
from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

X, y = load_breast_cancer(return_X_y=True)

scaler = StandardScaler()
lr = LogisticRegression()

X_sc = scaler.fit_transform(X)
lr.fit(X_sc, y)

pipe = Pipeline(steps=[('scale', scaler),
                       ('lr', lr)])

# Predicting would fail if the pipeline had unfitted estimators:
pipe.predict_proba(X)


Python code not displaying graph I am using some python code from kaggle that plots a bar graph. However, when I run it, it does not display the graph. When I run it in the IDE I get the following output:

<matplotlib.axes._subplots.AxesSubplot object at 0x1200b9ba8>
<matplotlib.axes._subplots.AxesSubplot object at 0x1200b9ba8>
<matplotlib.axes._subplots.AxesSubplot object at 0x1200b9ba8>

I have used the .show() method but nothing seems to work.

Here is the link to the code that I am using:

Code

I also have it directly in the question:

import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
import matplotlib
from sklearn.linear_model import LinearRegression

# Input data files are available in the "../input/" directory.
# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory

#from subprocess import check_output
#print(check_output(["ls", "../input"]).decode("utf8"))

# Any results you write to the current directory are saved as output.

frame=pd.read_csv('Project_Airplane_Crashes_and_Fatalities_Since_1908.csv',sep=',')
matplotlib.rcParams['figure.figsize'] = (12.0, 8.0)

operator = frame[['Operator','Fatalities']].groupby('Operator').agg(['sum','count'])

fig_ops,((ax1, ax2), (ax3, ax4))=plt.subplots(2,2,sharex=True)
accidents = operator['Fatalities','count'].sort_values(ascending=False)
totalacc = accidents.sum()
accprop = accidents/totalacc
accidents.head(10).plot(kind='bar',title='Accidents by Operator',ax=ax1,grid=True,rot=90)
accprop.head(10).plot(kind='bar',title='Proportion of Total Accidents',ax=ax2,grid=True,rot=90)

fatalities = operator['Fatalities','sum'].sort_values(ascending=False)
fatalities.head(10).plot(kind='bar',title='Fatalities by Operator',ax=ax3,grid=True,rot=90)
totalfatal = fatalities.sum()
fatalprop = fatalities/totalfatal
#print(fatalprop)
fatalprop.head(10).plot(kind='bar',title='Proportion of total Fatalities',ax=ax4,grid=True,rot=90)

Any help would be greatly appreciated.

I have gotten the answer to my question since I keep searching for solution.
I got the solution here enter link description here

By StefanM

So all I needed and should have done is to simply do
plt.show()
and I was typing fatalprop.head(10).show()
Thank you


Sentiment Analysis of Movie Reviews using Python I am currently doing sentiment analysis using Python. Here I am taking all the reviews from movie dataset and using Naive Bayes algorithm to predict whether the review is positive or negative. From the input dataset, I am using a logic to remove stopwords and after that training my dataset to predict the result. As of now, my test results are predicted, but few results are predicted wrongly.

This was a great film. Jack Halley is comical as usual and Bela Lugos is  hilarious even in his tiny role. I love this flick.

Example, the above sentence is positive, but it is predicting as negative. Similar instance occurs to many other scenarios. If it has some sarcasm, then there are chances that it may predict the other way. But for few direct reviews, it is predicting the other way.
Could you suggest other ways to better my algorithm, so that it increases the accuracy of prediction. Any other efficient methods apart from removal of stopwords? I have been looking for various suggestions through internet but I am new to this topic and hence couldn't find an effective way to proceed. Any good suggestions would be greatly appreciated.

Given that exact case, I would assume that you are getting negative decision due to names, mentioned in the review (in your training dataset actors were more often met in negative reviews). You should, probably, remove all non-relevant words from reviews, and that includes not only stop words, but all person names (since they are less of a sentiment marker except, maybe, justin bieber, who is a really negative marker to anything :) ).


PCA and K means in the context of machine learning I have a question regarding machine learning. I mostly do exploratory data analysis (EDA) of large 'omics datasets. I recently submitted a metabolomics based manuscript, and one of the reviewers was not happy with my analysis of the data. I think he as misunderstood the concept of machine learning, but I could be mistaken. I have added one of many examples where he refer to machine learning.
Reviewer: "PCA, K-means, and any other machine learning multivariate method of analysis must be performed on sex-specific groups"
I have always believed that machine learning is used for classifying and prediction purposes, but I dont seem to find any clear definition. Anyhow, I have three questions:

Can somebody define machine learning?
Is PCA machine learning? I have always thought its a method of dimension reduction that seeks to explain observed variance? It does not care about labels or groupings, nor does it predict or classify anything. Can somebody explain me where I got this wrong?
Is K means clustering per definition machine learning? This algorithm has been around since the late 60s, long before machine learning was even a thing. I understand that this algorithm is applied in machine learning, but so is linear regression and logistic regression - but we dont call that machine learning.

I would appreciate your input on this?

It is always quite tricky to define Intelligence and Learning since they are very abstract concepts.
in my opinion, both algorithms are machine learning, and I will try to argue why.
Learning is (to me and Wikipedia) the action of improving at performing a task without being programmed explicitly. In our case, both algorithm improve at doing their task the more data we feed into them. And we do not code them explicitly, but instead we give them a goal : minimizing a loss function.
This is quite similar to the way humans learn. When we learn how to read, we get faster and faster and better at imagining what is written the more we read. Yet we do not know exactly how to read faster, we just know the objective function : minimize the time required to understand. And so our brain does the learning from its experiences (which are similar to the datas we feed to ML algorithms).
Another hint that leads to learning to me is the fact that both these algorithms won't give the same results twice if we initialize with random values. To compare to humans again, each person have his way of improving at the task : we all get better, yet each person has its own tricks for the task.
Anyway this is just my opinion on the subject and I do not pretend it does make sense to everyone.
Here is another interesting thread about PCA on Stack if you wanna read some more.
Hope my answer helps you in your philosophical reflection about this complicated topic.


Machine Learning using NVIDIA DIGITS - Can't Classify Left or Right Direction of a Ball Throw I'm using NVIDIA DIGITS 3.0 to do training for detecting direction of a ball throw.

My Dataset contains 400+ binary images each for left and right throw with the following specs in DIGITS:

Image Type: Grayscale, JPG
Image Size: 256 x 256
Resize Transformation: Fill

My Classification Model specs:

Solver Type: NAG
Networks: GoogLeNet
The rest are default values

Problem:

I separated out 8 images (4 for left, 4 for right) from my 400+ training dataset to do testing on my model. The results returned from my classification model always classifies all 8 are left throws at 90%+ accuracy even though 4 of which are clearly right throws.

2 of the test images are shown below. My subject always stands at the same position to throw the ball, which is more or less constant. I've tried to change various parameters in the classification model like no. of epochs, subtract image mean, etc. to no avail.

In a previous dataset, my model classifies all throws as right throw, similarly only one direction is detected. I'm unsure whether the problem lies in my dataset images or the training model. I would greatly appreciate any advice.

Left Throw:

Right Throw:

a few things,
these images look huge. Are they resized or cropped before going into the net. Try resizing instead of cropping. Also, if you use flip/rotation augmentation that will mess up your network.

Second, in the training data what % are right and what % are left throws? ideally, you have about 50:50 in the training set. If images in training are mostly left, it might learn to guess left preferentially.

BTW, there might be an easier way to solve this problem. First, extract small patches and classify each patch as: 1 (contains a ball) or 0 (doesn't contain a ball) - this is segmentation.

After segmentation, a simple algorithm that tells you if the ball is left of center or right of center will give you the correct answer.


How to select optimal threshold which separate different classes? I have trained a network to find the similarity between two images. The test dataset contain equal number of similar and dissimilar samples. Each class has approx. 13822 samples.

I tried different threshold, for example, in the last layer of the network I used sigmoid activation function therefore the output is between [0,1].

I tested threshold such as from 0.5 to 0.9 (for example, if the prediction score is greater then 0.5/0.6/,....,/0.9, consider it as positive sample else negative sample).

Based on this strategy I got the following results:

Threshold   TP  TN  FP  FN  Accuracy    Precision   Recall  F1
0.5 --> 12570   13804   18  1252    0.95    1.00    0.91    0.95
0.6 --> 12115   13813   9   1707    0.94    1.00    0.88    0.93
0.7 --> 11451   13819   3   2371    0.91    1.00    0.83    0.91
0.8 --> 10124   13822   0   3698    0.87    1.00    0.73    0.85
0.9 --> 5132    13822   0   8690    0.69    1.00    0.37    0.54

What I am interested in is to find the optimal threshold based on some method which separates positive and negative classes.

Rank all your instances by their output value. Then for each instance (or for each different output value, if many instances have the same value), calculate the performance (precision, recall, f-score) considering that the threshold is the output value for this instance. When this is done for all the instances, you can plot the graph of f-score as a function of the threshold for a nice visual result, or simply pick the threshold corresponding to the maximum performance.

For the record, this is also how a ROC curve can be built manually.

Btw your optimal threshold is certainly lower than 0.5, since your precision is 1.0 at 0.5 but you can probably still increase recall.


is there an adequate number of levels of categorical variables? I have a project that I'm working on. The dataset contains many categorical variables and some of them have too many levels (+100). My question is : is there any advice to know the "adequate" number of levels of a variable ? is it based on the number of levels of other variables ? (for example most variables have between 10 and 30 levels and one or two variables have 80 100 levels).
For the variables that contain too many levels, I want to take 80% of most frequent levels and put the 20% into a new level "others" but I don't know at which number of levels I should stop (for example :
var 1 : 70 levels, var 2 : 100 level, var 3 : 13, var 4 : 30, var 5 : 60, should I apply the 80-20 method starting from 60? 70? 100?)
I don't know if I'm being clear but I hope you understand

No, there's no "adequate" number of levels for a categorical variable.
The choice to simplify the data by discarding some levels (for example by using a default category, as you propose) depends on what the goal is (and also number of instances, etc.). Very often this choice is made experimentally, that is by trying different methods (e.g. different thresholds) and observing which one gives the best performance: here you could do a program which tries the different proportions as threshold, then train and test the resulting model for every value, and finally plot the performance for every value.


Multivariate Time-Series forecasting using LSTM I have a dataset of hourly measures of pollution('Sample_Measurement) and weather condition. If I want to predict the pollution level of the current hour using the weather and pollution data of the previous n hours i have no problem. In fact, let's say that n = 24 and the number of features is 5. The 3D vectors will have the following shape (number of elements, 24,5).

Until this point I have no problem, but what i want to do is using also the weather condition of the current hour(together with the previous n = 24 hours), since in the previous case I was using only the data for the previous n hours. The problem is that i don't know how to do it since the 3D vector has as third dimension 5 (4 weather features plus 1 one that represents the pollution level), so i cannot simply add the current hour weather features because the number of feature(4) is different(since for the current hour we don't have the current pollution that is what we want to predict) from 5.
I hope I have been clear but it was not easy to explain.

What you could try to do is the following (pseudocode ahead):

input_past_24 = Input(shape=(24,5))
input_today = Input(shape(1,4))
output_from_LSTM = LSTM(some_args, ...)(input_past_24) # this has some size of your choice, let's say N
# if you want to add other layers, you'd do it here.
prediction = Dense(some_args, ...)(concatenate(input_today,output_from_LSTM))
model = Model(inputs=[input_past_24,input_today], output=prediction)
model.compile(some_args)
model.fit([data_from_past_days,data_today],expected_pollution, some_other_args...)

Above, the crucial point is to concatenate the output of the LSTM layer with the input of the day, before sending this to a final Dense layer for the final prediction.

For extra details, you may want to check Keras' Guide to the functional API.


Compare ratings of players over different leagues I want to compare ratings of players from different leagues and predict rating of player in a league he/she didnt participate in. Rating of a player is estimated within a league where he was playing.
There are some cross-observations which are players that have ratings estimated for more then one league.

For example there is a player P1 who has a rating 40 in league L1 and rating 55 in league L2. As you can see at the picture. There are distributions of rating in L1 and L2. Yellow line indicates the mean rating of a league and the green lines are cross-observations .

My question is if I know that player Px has a rating R in L1, what is his rating in L2?

Any ideas are appreciated!

Thank you

An existing way to do this is to use modified Elo ratings.

Video game servers sometimes use a similar scoring system called Glicko or Glicko-2 which might be better for your purposes.

My understanding is that you want to understand the rating of each player relative to every other player and that each in-league rating is essentially the rating of each player in that league relative to each other player in that league. If this is true then there are two ways I can think of to help you work out the hypothetical rating of player X in league L2 based on their performance in league L1:

Generate global ratings based on one of the methods mentioned above and use them to insert the new player in the right place in the ranking
Calculate am in group rating based on the methodologies described above and then calculate the new players ranking by stimulating the score he/she would accumulate if they played each of the players in L2 that they could be compared to in L1 and achieved the expected outcome (i.e. beat all the players they were ranked better than and were beaten by all the players they were ranked worse than)

You might also be interested to check whether the rank correlation between leagues is strong based on players that are ranked in both leagues.


Keras stateful LSTM returns NaN for validation loss I'm having some trouble interpreting what's going on in the training and validation loss, sensitivity, and specificity for my model. My validation sensitivity and specificity and loss are NaN, and I'm trying to diagnose why.

My training set has 50 examples of time series with 24 time steps each, and 500 binary labels (shape: (50, 24, 500)). My validation set has shape (12, 24, 500). Of course, I expect a neural network to overfit massively.

Because I was interested in a stateful LSTM, I followed philipperemy's advice and used model.train_on_batch with batch_size = 1. I had multiple inputs: one called seq_model_in that is a time series, and one called feat_in that is not a time series (and so is concatenated in to the model after the LSTM but before the classification step).

As my classes are highly imbalanced, I also used Keras's class_weights function. To make this function work in a multi-label setting, I concatenated two columns to the front of my responses (one of all 0s and one of all 1s), such that the final shape of the response is (50, 502).

 feat_in = Input(shape=(1,), batch_shape=(1, 500), name='feat_in')
 feat_dense = Dense(out_dim, name='feat_dense')(feat_in)
 seq_model_in = Input(shape=(1,), batch_shape=(1, 1, 500), name='seq_model_in')
 lstm_layer = LSTM(10, batch_input_shape=(1, 1, 500), stateful=stateful)(seq_model_in)
 merged_after_lstm = Concatenate(axis=-1)([lstm_layer, feat_dense])
 dense_merged = Dense(502, activation="sigmoid")(merged_after_lstm)

I have coded this model in Keras for time series prediction (multi-label prediction at the next time step):

The training and validation metrics and loss do not change per epoch, which is worrisome (and, I think, a symptom of overfitting), but I'm also concerned about understanding the graphs themselves.

Here are the TensorBoard graphs:

The training loss should (roughly) be decreasing per epoch, as should the validation loss. The training sensitivity and specificity are 92% and 97.5%, respectively (another possible sign of overfitting).

My questions are:

Am I right in thinking the sensitivity and specificity graphs should all have the same general shape as the train_spec graph?
Why does the training sensitivity graph look like this?
Why does the validation loss return NaN?

No idea about the tensor board stuff, but the NAN for val loss could be being caused by an unexpected very large or very small number. As your training dataset is very small there is a high chance that the validation data has a very different underlying distribution.
If the latent space of your NN isn't continuous and validation samples fall too far away from the distribution of the training samples, the NN will not deal with them well. In a classification problem this would result in misclassification (most likely). However, as you are generating a prediction, this could mean that the NN creates a really large number or a really small one. Then, in your loss function might be getting something divided by zero or by infinity.

I'd suggest looking at the the loss function you are using, if it has a divide, can the denominator be zero? One solution to this would be to use the clip function to make sure that your predicted tensor is between 0+epsilon and 1.

def clipped_squared_error_loss(y_true, y_pred):
    y_true = K.clip(y_true, K.epsilon(), 1)
    y_pred = K.clip(y_pred, K.epsilon(), 1)
    return K.sum(K.square(y_pred - y_true))

You may also want to look into Kullback Leibler Divergence as a means to make the latent space of your NN continuous.


To compute performance metrics like precision, recall and F1 score you need to compare two things with each other:

the predictions of your model for your evaluation set (in what follows, I'll call them y_pred)
the true classes of your evaluation set (in what follows, y_true).

From what you write, you have obtained just the predictions of your model, and that's what you have in y_pred. You have constructed y_pred so that each of its components is equal to the class that is assigned the maximum probability by your model. All fine here!

The key ingredient that you are missing is the array of true classes (in your question, you called them "tags") associated to your evaluation examples. You totally need this information to understand if the predictions of your model are correct (or not). You should be able to construct an array y_true containing the true classes/tags of your examples from...knowing the actual ones.
For example, if your 1st text belongs to class 3, your 2nd text belongs to class 1, your third text belongs to class 2, your y_true will be an array like

y_true = np.array([3, 1, 2, # ... the rest of components ])

Now, to compute accuracy, precision, and recall, you need to compare y_true and y_pred. If they coincide, congratulations: that means that your algorithm works perfectly on your evaluation set!
In general though not all the components of y_pred will coincide with y_true. To quantify agreement/discrepancies you can use metrics like accuracy, precision, etc. You can code them yourself, but the scikit-learn library comes with functions for the purpose.

For instance you can easily compute accuracy, precision, recall and F1 score, even the confusion matrix for your problem with the code

from sklearn.metrics import accuracy_score, classification_report, confusion_matrix

# print accuracy
print("Accuracy: ", accuracy_score(y_true, y_pred))

# print precision, recall, F1-score per each class/tag
print(classification_report(y_true, y_pred))

# print confusion matrix, check documentation for sorting rows/columns
print(confusion_matrix(y_true, y_pred))

P.S. Note that in what I did I didn't use your y_pred_one_hot (which is not a confusion matrix!), and that the precision is not zero (it may be, but you have to compute it using y_true)!

P.S. Beware using predict_proba with Naive Bayes as output probabilities may not be calibrated.


Recommendation engine with mahout I have a list user data: user name, age, sex, address, location etc., and

a set of product data: Product name, Cost, description etc.

Now I would like to build a recommendation engine that will be able to:

1 Figure out similar products

eg :

name  :   category   :  cost    :   ingredients

x     :     x1   :        15  :       xx1, xx2, xx3

y     :    y1   :        14   :     yy1, yy2, yy3

z     :    x1  :          12   :     xx1, xy1

here x and z are similar.

2 Recommend relevant products from the product list to a user

How can I implement this using mahout?

I recommend you to take a look to Oryx (https://github.com/OryxProject/oryx). Oryx is based on Apache Mahout (actually one of the creators of Mahout Sean Owen built it) and provides recommendation using collaborative filtering. Oryx is a very practical tool for implementing recommendation. I have used it in several projects: recommending products in retail stores (small businesses), building an e-commerce recommender and user similarity from mobile app interaction.

You just have to represent data in the form:
UserId ItemId Value

Where value is a measure (subjective) of the importance or influence of the interaction between that user and the item. User and item can be anything actually, and the same procedure can be used for tagging. For example, for recommending songs, finding similar songs and bands, and finding similar users according to their music tastes you can represent data as

UserId SongId NumberOfPlays

Where NumberOfPlays is the amount of times a song has been played by user (in an online music service for example). This exampl was given in Myrrix the predecessor of Oryx. They also show how to recommend tags to questions in StackOverflow.

The github site is not that well documented but it will be enough to get it running (and working :))


When to consider a target Variable unbalanced? i'm performing a binary classification task , and after cheking the target variable , i saw that i had 69% of 0's and 31% of 1's , so , my question is , do i have in this case a unbalanced target variable ?
 and can somebody tell me , if there is a threshold , or when to consider a Target Variable unbalanced ?

Thanks to everyone

There's not a strict threshold about what ratio is considered as unbalanced. But in general, 30 percent is not usually a sign of unbalanced classification.

You can although try different methods for checking if your classification method is accurate and predicts correctly or not, like resampling, over-sampling, under-sampling

increasing the number of records (samples) in the class with less frequency.
randomly removing the samples from the class with higher frequency.

...

from sklearn.utils import resample


Best way to visualize huge amount of data I have a data set of around 3M row. I has only 2 category (category- 2:1 ratio). Now i want to visualize(scatter plot) it's distribution to understand can the data linearly separable or not(In order to choose model type).I already try this and the plot is not understandable. What will be the best way to visualize this data set?

I have three suggestions that may help.

Reduce the point size
Make the points highly transparent
Downsample the points

Since you do not provide any sample data, I will use some random data to illustrate.

## The purpose of S1 is to intermix the two classes at random
S1 = sample(3000000)
x = c(rnorm(2000000, 0, 1), rnorm(1000000, 3,1))[S1]
y = c(rnorm(2000000, 0, 1), rnorm(1000000, 3,1))[S1]
z = c(rep(1,2000000), rep(2,1000000))[S1]

plot(x,y, pch=20, col=rainbow(3)[z])

The base plot without any adjustments is not very nice. Let's apply suggestions 1 and 2.

plot(x,y, pch=20, cex=0.4, col=rainbow(3, alpha=0.01)[z])

Reducing the point size and making the points highly transparent helps some. This gives a better idea of the overlap between the two distributions.

If we downsample, we don't need quite as much transparency.

## The purpose of S2 is to downsample the data
S2 = sample(3000000, 100000)
plot(x[S2],y[S2], pch=20, cex=0.4, col=rainbow(3, alpha=0.1)[z[S2]])

This gives a different view that provides a similar, but not identical understanding of the two distributions.

These are not magic, but I think that they are helpful.


Comapring hidden markov models Given a set of sequence transitions, there are different orders of hidden markov models that can be fitted to a dataset.

Is there any test to determine which is the best model for a given sequence set?
How to compare two HMM fitted on to a sequence set?

Generally models can be compared by evaluating them. It depends on the specific application, for instance if the goal is to predict the next element(s) in a sequence then one can take a subset of the data as test set and see how accurately the model predicts the next element.


Updating One-Hot Encoding to account for new categories My question is focused around how to appropriately update an encoded feature set when a new category is introduced by the test data. I use the data in logistic regression and I know it is not a 'live' model (i.e. gradient descent is performed whenever new data is introduced) but do I have to retrain the model to account for added features or do I just add it to subsequent test set values.

To exemplify the problem consider a TV Show training set where each show has a 'networks' feature set that includes one or more of the following:

["abc","cbs","nbc"]

Then, in the testing set there is a TV Show with the feature set:

["abc", "hulu"]

Would I have to add the new feature retroactively to the training data and retrain the model eventhough it will never occur? Wouldn't this introduce 'look-ahead-bias'?

How do I account for the added feature in the encoder going forward?

I think you have two options:

Automate your train/test pipeline so that one-hot encoding is part of
it. If new categorical variables are introduced, they can be featured
in the training dataset even if not very prevalent. This would
introduce some bias if the nature of the TV show distribution has
changed over time (e.g. 20 years ago there weren't as many options)
but I don't necessarily think it is a show stopper.
If new possibilities are introduced over time but for whatever reason you can't  retrain, then you should omit using that new value. This has its own disadvantages because in your example, it would be a TV show with no network.


Can training label confidence be used to improve prediction accuracy? I have training data that is labelled with binary values. I also have collected the confidence of each of these labels i.e. 0.8 confidence would mean that 80% of the human labellers agree on that label.

Is it possible to use this confidence data to improve the accuracy of my classifier?

Would the following work?

1a) If the label is 0 and the confidence data for that label is 0.8
then give the training data a new label of 0.2
1b) If the label is 1 and the confidence data for that label is 0.8
then give the training data a new label of 0.8
2) Calculate the new label using this method for every entry in the
training set
3) Treat the problem as a regression problem (where the labels range
between 0 and 1)
4) Classify unlabelled data based on whether the new labels are above
or below a specific value. i.e Give all predicted labels class 0 if below X and class 1 if above X.

I am currently using a SVM with an RBF kernel for my model.

Thanks in advance!

Yes, it's possible to use this confidence data.  However, I wouldn't recommend the approach you mention.  Instead, let me suggest a different approach.  Actually, I'll suggest two.  The first one is conceptually clean; the second is probably easier to implement; and they'll probably be approximately equivalent in practice.

Adjust the loss function

You can adjust the loss function to reflect the confidence scores you have on the training data.

In particular, if you are using the cross-entropy loss, there's a particularly clean way to do this.  Let me explain some background on the cross-entropy loss, then explain how.  We think of the label on the training instance as a probability distribution on labels.  In binary classification, such a distribution can be represented as a vector $(p_0,p_1)$ where $p_0$ represents the probability that the label is 0 and $p_1$ the probability that the label is 1.  Normally, we are given "hard labels": if we know that the correct label on instance $x$ is 0, then that corresponds to the probability distribution $(1,0)$; if the correct label is 1, that's the distribution $(0,1)$.  The cross-entropy loss then compares the prediction from the classifier to this distribution.

The nice thing about the cross-entropy loss is that it generates readily to comparing any two distributions.  Thus, if you have a confidence $0.8$ that the correct label for instance $x$ is 0, then that corresponds to a probability distribution $(0.8,0.2)$.  Now, you can compute the cross-entropy of the classifier's prediction with respect to the distribution $(0.8,0.2)$, and that is the contribution to the loss from training instance $x$.  Sum this over all instances in the training set, and you get an adjusted loss function.

Now you can train a classifier by minimizing this adjusted loss function, and that will directly incorporate all of the information in your confidence scores.

Use weights

Alternatively, you can use weights to reflect the confidence information.

Some classifiers allow you to specify a weight for each instance in the training set.  The idea is that a misprediction for a particular instance is penalized proportionality to its weight, so instances with a high weight are more important to get right and instances with a low weight are less important.  Or, equivalently, the training procedure tries harder to avoid errors on instances with a high weight.

You can use weights to reflect confidence information.  Suppose you have an instanced $x$ in the training set that you think should have label 0, with confidence $0.8$.  You would add one copy of $(x,0)$ to the training set with weight $0.8$ (i.e., instance is $x$ and label is 0), and add one copy of $(x,1)$ to the training set with weight $0.2$ (i.e., instance is $x$ and label is 1).  Build up the training set in this way.  This doubles the size of your training set.  Now train a classifier, using these weights.

For classifiers that support weights, this should be easy to implement.

One can also show that it is effective and reasonable.  For instance, when using the cross-entropy loss to train a classifier, this approach using weights is equivalent to adjusting the loss function as highlighted above.  So, in that particular context, the two approaches are actually equivalent.


Adjusting weights in an convolutional neural network I'm trying to implement a convolutional neural network at the moment. A simple feedforward network is not the problem but I'm having some trouble with the weight adjustment in the conv layer.

Lets assume I have four layers. Input, convolution, hidden and output.

src:http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/

In the picture above we just see the input and the convolution layer.
The deltas of the convolution layer are calculated as in a normal feedforward network.
But how do I update the weights/filtermatrix between input and convolutionlayer?

For learning kernel/filter matrix in convolution layer, we find partial derivative of loss w.r.t. filter matrix and use gradient descent method to update filters.
$$ W = W - \alpha\frac{\partial L}{\partial W} $$

Convolutional Neural Networks also use back-propagation algorithm to find partial derivatives of loss w.r.t. filter matrix.


Can Sequence to sequence models be used to convert code from one programming language to another? Is it possible to convert a code from one language to another using sequence to sequence programming. if not sequence to sequence, which algorithm can i use to do this?

Yes, it is possible to convert code from one programming language to another using sequence-to-sequence neural networks. Sequence-to-sequence can learn to translate anything from anything if there are consistent patterns and enough paired examples.

However, it would not be efficient because the models would only indirectly learn the semantics (i.e., the meaning of the code) through the observed string literals (i.e., the characters that appear). It would be more efficient to map one observed sequence of string literals to semantics, then use the latent semantics representation to generate a new observed sequence of string literals.

There are a variety of ways of doing that depending on much you handicap yourself. If you give the model access to abstract syntax tree (AST), it becomes much easier. Then a tree-based Long Short-Term Memory (LSTM) could learn the semantic space.

Overall, this is a field of study that tries to build a Neural Network Transpiler.