Over/Under Sampling for Multi-classification I'm trying to apply xgboost and random forest for over and under sampling

For imbalanced data:

train shape -> (199991, 23)

However, reverse my expectation. The accuracy went down for both cases.

Questions:

Is under and over sampling always not good?
What else should I consider when applying over and under sampling?

You are applying  xgb and random forest to a multiclassification task and you are doing under sampling to some how try to improve your class

If i understand your histogram, that is the distribution of labels.

When you say accuracy went down for both classes you are refering to a multiclassficitaion problem right?

Answers directly to your questions:

There are same cases when over or undersampling improves your model, but there are not many. Selecting the right loss is more important.
Consider using another can also help you to improve your model.


Pandas Conditional Fill NaN Forward/Backward Updated 22 Oct. 2018:
I have the following dataset:
data = [('D',1,10,8),
           ('D',2,12,12),
           ('X',1,28,np.NaN),
           ('D',3,np.NaN,np.NaN),
           ('X',2,np.NaN,25),
           ('X',3,32,25),
           ('T',1,220,np.NaN),
           ('X',4,30,np.NaN),
           ('T',2,240,np.NaN),
           ('X',2,38,np.NaN),
           ('T',3,np.NaN,np.NaN),
           ('T',4,200,150)]

labels = ['item', 'month','normal_price','final_price']

df = pd.DataFrame.from_records(data, columns=labels)

    item    month   normal_price    final_price
0   D       1       10.0            8.0
1   D       2       12.0            12.0
2   X       1       28.0            NaN
3   D       3       NaN             NaN
4   X       2       NaN             25.0
5   X       3       32.0            25.0
6   T       1       220.0           NaN
7   X       4       30.0            NaN
8   T       2       240.0           NaN
9   X       2       38.0            NaN
10  T       3       NaN             NaN
11  T       4       200.0           150.0

I want to fill NaN in the 'normal_price', 'final_price' columns for each item with the 'normal_price', 'final_price' of its preceding month (if not available by its succeeding month). I have tried using this:
df[['normal_price','final_price']]=df[['normal_price','final_price']].fillna(method='ffill')

but it gives me this:
    item    month   normal_price    final_price
0   D       1   10.0                8.0
1   D       2   12.0                12.0
2   X       1   28.0                12.0*
3   D       3   28.0*               12.0
4   X       2   28.0                25.0
5   X       3   32.0                25.0
6   T       1   220.0               25.0*
7   X       4   30.0                25.0
8   T       2   240.0               25.0*
9   X       2   38.0                25.0
10  T       3   38.0*               25.0*
11  T       4   200.0               150.0

The problem is with cases with an asterisk (I've also tried 'bfill'). These values should be filled with based on their correct items. Ideally, I should get:
    item    month   normal_price    final_price
0   D       1   10.0                8.0
1   D       2   12.0                12.0
2   X       1   28.0                25.0
3   D       3   12.0                12.0
4   X       2   28.0                25.0
5   X       3   32.0                25.0
6   T       1   220.0               150.0
7   X       4   30.0                25.0
8   T       2   240.0               150.0
9   X       2   38.0                25.0
10  T       3   220.0               150.0
11  T       4   200.0               150.0

I have also tried the followings (from answers offered by date):
df[['normal_price','final_price']].ffill(limit=1).bfill(limit=1)

or
df[['normal_price','final_price']]=df[['normal_price','final_price']].interpolate(method='nearest')

But none of them are giving me reasonable fillna corresponding to each item. I have found this method:
df[['normal_price','final_price']]=df[['normal_price','final_price']].fillna(df.groupby(['item'])[['normal_price','final_price']].transform('mean'))

It works better, BUT it introduces unpredictable values (in this case the 'mean') for NaN values, not with the preceding or following values as I originally wanted. I am trying to combined the df.groupby(['item']) concept with '.ffill' or '.bfill', but so far no success.

Simply using the fillna method and provide a limit on how many NA values should be filled. You only want the first value to be filled, soset that it to 1:

df.ffill(limit=1)

  item  month  normal_price  final_price
0    1      1          10.0          8.0
1    1      2          12.0         12.0
2    1      3          12.0         12.0
3    2      1           NaN         25.0
4    2      2          30.0         25.0
5    3      3          30.0          NaN
6    3      4         200.0        150.0

You can chain together the above with a bfill to then fill the remaining NaN values:

df.ffill(limit=1).bfill(limit=1)

  item  month  normal_price  final_price
0    1      1          10.0          8.0
1    1      2          12.0         12.0
2    1      3          12.0         12.0
3    2      1          30.0         25.0
4    2      2          30.0         25.0
5    3      3          30.0        150.0
6    3      4         200.0        150.0

This would only not be optimal if there are column in your dataframe which you would like to leave unaffected.

In that case you can do them one column at a time - i use the in_place flag so that we do not need to do any of the ugly re-assignments:

df.final_price.ffill(inplace=True, limit=1)

df

  item  month  normal_price  final_price
0    1      1          10.0          8.0
1    1      2          12.0         12.0
2    1      3           NaN         12.0
3    2      1           NaN         25.0
4    2      2          30.0         25.0
5    3      3           NaN          NaN
6    3      4         200.0        150.0

The same idea will work the backward-filling the values, using the bfill method instead of the ffill, as I have done above.


Tool for annotation of images for semantic segmentation I have been searching around for a software tool, that I can use for annotating images. More specifically I want to do annotation to be used for semantic segmentation, meaning I want to create masks. I want to be able to create training data for applying a segmentation CNN (like for instance U-net).
However I have been digging around the internet, and I have tried out some options. But I have not really found anything that seems to do the trick.
I do not expect to have many classes, but I probably want to be able to be able to annotate more than two classes (something, background). And of course every pixel has to belong to a class.
So what I am looking for is a relatively simple tool that either lets you mark areas of a class with a brush or that lets one mark polygons and then fill those, and then the outside the polygons as background. So can anyone point me to anything like this? And share their experience?

Diffgram is really great for this! I used it for a construction monitoring project. It's Open Source. From their site:
Semantic Segmentation Tools:

Auto Bordering: Automatically detects edges to create 100% coverage
masks. Simple select the intersecting shape.
Combo Shapes: Create shapes that are partially curves and partially straight lines.
Points to Full Polygon: Draw a box (or a few points), get a polygon.
Correct Pre-Label Segmentation Map: Load an existing segmentation prediction (pixels.) Get an editable polygon for correction.

Semantic Segmentation Data Annotation


Histogram indicating 20~25% in a certain range This is a histogram of speeds of certain ships drawn to the density scale:

I was told that the percent of speeds in the [17, 18) range is between 20 and 25, but I believe it's between 30 and 50. Can anyone convince me wrong?

Histograms are not reliable methods to communicate summary statistics because it depends on bin size choice.
The best option to answer numerical questions to find the raw numerical data.


How to mathematically define the architecture of neural network model? And the function space associated with it? My goal is to properly define a search space for neural architecture search (NAS).

I think a proper definition must handle the following issues.

how to mathematically quantify the topology?
how to define the number in each layer? activation functions?

You can use directed graphs. Each neuron is a node in the graph. Each neuron has list of its neighbours and activation function. You can add remove nodes to this graph.

If you just want to search for simple architectures you can use two lists. One for number of neurons in layer and one for activation function of neurons in that layer.

So this is a neural network with 3 layers. First layer has 5 neurons and echo of them have Relu activation function.

  [5,3,2] ['Relu', 'sigmoid', 'Relu']

Hope that helps.


Learning Python for Data Science Is it necessary to learn Python from square one if your ultimate goal is to use it for data science? I have used R a lot for data science. But could I learn Python sufficiently well just by looking at other data science problems and their Pythonic approaches?

Both Python and R are extensively used in the field of data science.Though both have their advantages and disadvantages, python has many libraries that are awesome to use for data science purpose. For example, scikit-learn, Keras, Theano, XGBoost, Matplotlib,etc. These make lives so easy for data science.
Yes, analyzing and understanding the code of others can give you a lit of information but you cannot learn until you don't get your hands dirty with some python code. In order to learn how to use python in data science, this link provides an excellent tutorial.


What if my validation set is worse than my training? I am running a CNN, on the 1st epoch my training set accuracy is 15% and validation set is 12%, by the 51st epoch my training accuracy is 87% and validation set is 13%. What is happening? What does it mean if my validation set is lower or greater than my training set ?? (my test set gives 12%)

This is a common high-variance problem due to overfitting.

Simply put: Good training accuracy together with low dev set (or in your terminology the validation set) accuracy it means you are expressing your training data very well, whereas your model fails to perform well on unseen data (High-variance Problem).

More general: there are few other combinations of this high-/low-bias together with high-/low-variance that are explained in length in the context of Bias-Variance Trade-off by Andrew Ng in his first Machine Learning course and even more relevant to Convolutional Neural Networks in his last DeepLearning course, see this video. You also easily find many blog posts discussing about these stuffs like this blog post, or this one.

What you see here is quite alright. If fact it is highly recommended practice that you start by a model capable of describing your training set very well first (even overfit). Then if overfit meaning there is a large difference in accuracy between train/dev sets (similar to your scenario), you start penalizing the model (aka regularization). Once again you can find tons of materials on how to regularize CNN models. On the top of your search you will find Dropout, Max-Pooling. See here for example.

I also suggest to pay attention to your data split distributions. It is a very fundamental concept that often paid less attention to. Violating this assumption (aka data split mismatch) could lead to problems that could appear in any forms like large accuracy difference of model on data splits. Maybe check Ng's short video.


How to decide to use K-fold validation or not? When K-fold validation should be used and how to decide the value of the K.
I see most of the text books suggesting implementation of algorithms using 2/3 of data as train and 1/3 of data as test

The main criterion is that you need enough data in your training set to get a good model fit - which is both a function of the data quantity and the model complexity.

When you don't have much training data, you want as much data in your training set as possible - in this case, you can do "leave one out" cross-validation.  This is like k-fold, except it's N-fold, where N is the number of observations you have. This means you hold one observation out, build a model using everything else, validate using that one sample, and repeat N times.

When you have more data with a simple model, you can use a much more loose validation strategy. For example, if you have a million records and are doing linear regression with one predictor, you can get away with using a few thousand samples to build your model, since you'll get really good parameter estimates from thousands of samples (or fewer).

Those are the extremes. The suggestions you've seen live somewhere in-between.  When you pick high K, you have to build lots of models (this takes time), but each model is built with more data (good for complex models).  When you pick low K, you have to build fewer models, but each model is built with less data.

Personally, I pick a low value for K when I'm doing model exploration, but a higher value when I'm ready for validation of a good candidate model.  Actually, in this case you should have three sets - a training set, a validation set (that you use to find a good set of parameters), and a second validation set that you've held out the ENTIRE time (since, in a way, the other validation sets are helping you 'train' the model parameters).


Keras - Autoencoder different from Encoder + Decoder I build a CNN 1d Autoencoder in Keras, following the advice in this SO question, where Encoder and Decoder are separated. My goal is to re-use the decoder, once the Autoencoder has been trained. The central layer of my Autoencoder is a Dense layer, because I would like to learn it afterwards.

My problem is that if I compile and fit the whole Autoencoder, written as Decoder()Encoder()(x) where x is the input, I get a different prediction when I do

autoencoder.predict(training_set)

w.r.t. if I first encode the training set in a set of central features, and then let the decoder decode them. These two approaches should give identical answers, once the Autoencoder has been trained.

from tensorflow.keras.layers import Input, Dense, BatchNormalization, Flatten, Lambda, Activation, Conv1D, MaxPooling1D, UpSampling1D, Reshape
from tensorflow.keras.models import Model
from tensorflow.keras import optimizers
from tensorflow.keras.layers import GaussianNoise
import keras.backend as K
from tensorflow.keras.layers import Add

import tensorflow as tf

import scipy.io
import sys
import matplotlib.pyplot as plt
import numpy as np
import copy

training = # some training set, 1500 samples of 501 point each
testing = # some testing set, 500 samples of 501 point each

# reshaping for CNN
training = np.reshape(training, [1500, 501, 1])
testing = np.reshape(testing, [500, 501, 1])

# normalize input
X_mean = training.mean()
oscillations -= X_mean
X_std = training.std()
training /= X_std

copy_of_test = copy.copy(testing)
testing -= X_mean
testing /= X_std

### MODEL ###

def Encoder():
    encoder_input = Input(batch_shape=(None, 501, 1))
    e1 = Conv1D(256,3, activation='tanh', padding='valid')(encoder_input)
    e2 = MaxPooling1D(2)(e1)
    e3 = Conv1D(32,3, activation='tanh', padding='valid')(e2)
    e4 = MaxPooling1D(2)(e3)
    e5 = Flatten()(e4)
    encoded = Dense(32,activation = 'tanh')(e5)
    return Model(encoder_input, encoded)

def Decoder():
    encoded_input = Input(shape=(32,))
    encoded_reshaped = Reshape((32,1))(encoded_input)
    d1 = Conv1D(32, 3, activation='tanh', padding='valid', name='decod_conv1d_1')(encoded_reshaped)
    d2 = UpSampling1D(2, name='decod_upsampling1d_1')(d1)
    d3 = Conv1D(256, 3, activation='tanh', padding='valid', name='decod_conv1d_2')(d2)
    d4 = UpSampling1D(2, name='decod_upsampling1d_2')(d3)
    d5 = Flatten(name='decod_flatten')(d4)
    d6 = Dense(501, name='decod_dense1')(d5)
    decoded = Reshape((501,1), name='decod_reshape')(d6)
    return Model(encoded_input, decoded)

# define input to the model:
x = Input(batch_shape=(None, 501, 1))
y = Input(shape=(32,))

# make the model:
autoencoder = Model(x, Decoder()(Encoder()(x)))

# compile the model:
autoencoder.compile(optimizer='adam', loss='mse')
for layer in autoencoder.layers: print(K.int_shape(layer.output))

epochs = 100
batch_size = 100
validation_split = 0.2
# train the model
history = autoencoder.fit(x = training, y = training,
                    epochs=epochs,
                    batch_size=batch_size,
                    validation_split=validation_split)

# Encoder
encoder = Model(inputs=x, outputs=Encoder()(x), name='encoder')
print('enc:')
for layer in encoder.layers: print(K.int_shape(layer.output))
features = encoder.predict(training) # features

# Decoder
decoder = Model(inputs=y, outputs=Decoder()(y), name='decoder')
print('dec:')
for layer in decoder.layers: print(K.int_shape(layer.output))
score = decoder.predict(features) #
score = np.squeeze(score)

predictions = autoencoder.predict(training)
predictions = np.squeeze(predictions)

# plotting one random case
# score should be equal to predictions!
# because score is obtained from the trained decoder acting on the encoded features, while predictions are obtained form the Autoencoder acting on the training set
plt.plot(score[100], label='eD')
plt.plot(predictions[100], label='AE')
plt.legend()
plt.show()
plt.close()

When you do autoencoder.fit, your model is learning something so your weights are changing for the encoder model and decoder model.
After that while creating your encoder/Decoder like "encoder = Model(inputs=x, outputs=Encoder()(x), name='encoder')", you are creating another encoder model with differnt weights, you have to use the same weight vectors.
Check "autoencoder.layers[1].get_weights()" is equal to "encoder.get_weights()" or not, then you will get to know what is happening in the model.( similar thing for decoder as well). You can check that using below code,

for i in range(len(autoencoder.layers[1].weights)):
    print((autoencoder.layers[1].get_weights()[i]==encoder.get_weights()[i]).all())


How to generate a output like this? can u please give an explanation for this ?

Using the example dataframe provided by@JahKnows:

import pandas as pd

df = pd.DataFrame(data={'shopify_customer_id': [1,1,2,3,4,5],
                        'financial_status': ['paid', 'refunded', 'paid', 'refunded','paid', 'paid'],
                        'count': [8, 1, 13, 1, 1, 1],
                        'order_amt': [19, 19, 0, 19, 0, 24]})

You could also do the following:

df=df.groupby(['shopify_customer_id','financial_status']).sum().unstack()
df.columns = ['_'.join(col).strip() for col in df.columns.values]
df.reset_index(inplace=True)
df.fillna(0,inplace=True)

That gives you this (if this is what you intend to have!):

   shopify_customer_id  count_paid  count_refunded  order_amt_paid  order_amt_refunded
0                    1         8.0             1.0            19.0                19.0
1                    2        13.0             0.0             0.0                 0.0
2                    3         0.0             1.0             0.0                19.0
3                    4         1.0             0.0             0.0                 0.0
4                    5         1.0             0.0            24.0                 0.0


How to deal with disconnected components in isomap? While creating a nearest neighbor graph for isomap, there is a possibility that the graph is disconnected. In this case finding graph distances between all pairs of points will not be possible. Are there any simple methods other than iteratively changing the nearest neighbor search parameters  till we get a connected graph?

I believe here is a solution: https://ieeexplore.ieee.org/document/8282172.
Short summary: Build the network and get disconnected graphs. Calculate the center of each small graph and connect the centers with each other. Then scale the edges connecting different small graphs by some number greater than 1.


Finding the equation for a multiple and nonlinear regression model? Regarding nonlinear and multivariable regression, I am using R or Matlab. In the case where I have a regression with just two variables, I simply draw the graph Y with respect to X and look for the equation of the model that corresponds the most to the distribution of the point cloud.

However, in the case where Y is connected with several independent variables (in my case 12 variables $X_1, X_2,...,X_{12}$). How will I be able to define the equation of the regression model?

This question clearly is the basics of linear regression.

If you are looking for a way to implement linear regression in R, use the lm method.

fit <- lm(y~X1+X2, data = datasetname)

summary(fit)

This should give the coefficients and the p value for each variable.


Text analysis - classification, parsing Excuse if this has been answered before.

I need to extract features and parse from a piece of text and run some analysis.
For e.g. "Plot the past 5-year sales of Apple" should give me the following

Information:

Company - Apple
Item - Sales
Period - past 5 years
Action - Plot

What deep learning techniques / algorithms should I be looking to use?

Any pointers are highly appreciated.

Named-Entity Recognition (NER) is one of the techniques you could look at. Different techniques exist, you could first look at pre-trained models of Spacy in Python here. Otherwise, you could train a model on your own training data if you have.

Also, more systematic approaches can be efficient. For exemple, looking at POS tags values, do regex identification...

More generally, this is the field of Information Extraction (IE) in text mining. Some good resources can be found here and here. Sorry, everything is in Python as I work mainly with it.

Good luck!


What is the difference between topic modeling and clustering? I know that topic modeling and clustering are related, but not similar techniques. Can anyone suggest what are the main differences?

The purpose of topic modeling methods is to discover the latent themes (topics) assumed to have generated the documents of a corpus. Topic modeling methods are built on the distributional hypothesis, suggesting that similar words occur in similar contexts. To this end, they assume a generative process (a sequence of steps), which is  a set of assumptions that describe how the documents are generated. Given the assumptions of the generative process, inference is done, which results in learning the latent variables of the model. For instance, for Latent Dirichlet Allocation, this is the per-topic document distributions and the per-word document distributions. In this sense, a document can be represented by its per-topic distribution (doc$_1$ = 0.3$\times$Sports + 0.7$\times$Cinema). This later can be seen as a soft clustering approach, i.e., doc$_1$ belongs 30% in cluster Sports and 70% in Cinema. But topic models are not solely clustering methods, as can also been used for understanding, exploring, visualizing a collection.

On the other hand, clustering methods aim at partitioning data into coherent groups. Of course, what is coherent and how the partitioning is performed differs between the various clustering algorithms. The distance between the data instances is central for clustering methods and for this the instances can be represented in various ways: for documents this can be term frequencies (tf), tf-idf, and even with the per-document topic distributions learned by topic models.


How to perform Permutation Feature importance? I am trying to perform feature selection. Currently with Tree based classifiers, even randomly generated column is ranking above some of my real columns. So I was reading about PFI. Can someone help me understand the steps to be followed?

1) Have my data in csv file

2) Fill in missing values/clean/prepare data

3) Split the data into train and test

4) Standardize train and test data

5) Select a model (let's say logistic regression)

6) Fit train data to logreg model

7) Predict using input test data

I have given the code below for my model

logreg=LogisticRegression() # step 5
logreg_cv.fit(X_train_std,y_train) # step 6
y_pred = logreg_cv.predict(X_test_std) # step 7

Would really be helpful if you could kind of explain in a way a noob can understand. I am new to ML, so it would really be helpful

Where does PFI fit in here? I see that we have eli5 package but couldn't find an example yet.

update - my code after suggested solution

logreg=LogisticRegression()
pi=PermutationImportance(logreg, cv=2).fit(X_train_std,y_train) # error is in this line.
rfe=RFECV(pi, step=1, n_jobs=-1, cv=5, scoring="auc").fit(X_train_std,y)
y_pred = rfe.predict(X_test_std)

Ok lets answer this with some basic steps:

Yo have done what you said until 5-th step after that you could do:

from sklearn.feature_selection import RFECV
from eli5.sklearn import PermutationImportance
pi=PermutationImportance(logreg, cv=2).fit(X_train,y)
selector=RFECV(pi, step=10, n_jobs=-1, cv=2, scoring="f1_macro").fit(X_train,y)

now make predictions

selector.predict(X_test)

or plot the most important features

import matplotlib.pyplot as plt
plt.plot(selector.grid_scores_)


Data anonymization in Python I am working on an industrial project which consists of real data. Now, the data contains sensitive information about company operations which could not be disclosed publically. As a result, I need to anonymize the original data first before implementing the machine learning algorithms. `The data anonymization includes:

changing the names of persons,
places,
geographical locations, etc.

I would like to know what are the best practices for anonymizing datasets? Ideally, I should be able to get the original data back after performing analysis on the anonymized dataset.

I went through the literature and looked over some answered questions. They all are based on cybersecurity aspects like encryption and decryption algorithms. I am not familiar with cybersecurity algorithms. Is there any way to slightly change the data without digging into cybersecurity algorithms?

As far as I know text anonymization is mostly considered a manual pre-processing step, I'm not aware of any reliable fully automated method. The reliability of the process is usually crucial for legal and ethical reasons, that's why there must be some amount of manual work.

That being said, the process can be made semi-automatically, especially if the scope of the information to be obfuscated is not too large. In your case a NE tagger could probably be applied to capture a large part of the entities.

Once all the entities have been annotated in the original data, it's straightforward to replace them automatically with a placeholder. This can be done while keeping the original and anonymized version aligned (typically using a unique id for every entity).


Impose similar metric on segments to model I am training a binary classifier in a dataset using AUC as a score. The dataset has two main groups (we will refer to them as good and bad population). A property that this dataset has is having a higher proportion of target = 1 in the bad population.

For this reason, a relatively dummy classifier would give higher scores to the bad population and lower scores to the good population. In fact, the AUC of the classifier could be pretty high globally, and, when looking at the AUC inside both populations separately, the AUC might be really low in both of them.

I want to avoid this behavior. In fact, I am willing to sacrifice some AUC in the global population such that the AUC in each group is not very low. An idea that I had was using the harmonic mean of the AUC of both groups as a metric instead of the general AUC. However, this might not really help a classifier in a natural way.

Are there any papers/techniques/software that can help me in solving this problem in a more natural way?

Given that in your data there is correlation between population type (good vs. bad) and target, your model may learn undesirable associations between both. Therefore, the population type is a confounding factor.

A natural tool to cope with scenarios with confounders it causal inference. You can find an overview of causal inference in Judea Pearl's work, either this article or his book. A terser introduction to causal inference can be found in Ferenc Huszár's blog including an entry for controlling confounders.

There are a few python packages providing causal inference functionality, such a Microsoft's dowhy or Causalinference.


Getting Scikit-Learn RandomForestClassifier to output Top N results I'd like to see the top N results for a RandomForestClassifier prediction, ordered by descending probability.

The answer may be predict_proba, but I have no idea how to interpret the results.

Help appreciated!

As there isn't too much code/context you're giving us I assume that you're working with a binary classification problem:

import numpy as np
X = ... # input for classification, shape (n_samples, n_features)
y_pred = rf.predict_proba(X)[:, 1] # index slicing to retrieve a 1d-array of probabilities
y_pred
# array([  0.18, 0.21, 0.1, 0.2, 0.3])

# Now lets see what the 2 biggest probabilities are
np.argsort(y_pred)
# [2, 0, 3, 1, 4] => lowest probability is at index 2 (the 3rd probability as classified by the RandomForestClassifier)
# Now lets see what the top 5 samples look like (remember, argsort sorts in ascending order, so we take the last 5 indices of argsort):
X[np.argsort(y_pred)[-5:]]

Now you should see the rows in X with the 5 most confident predictions.


XGBoost equations (for dummies) I am having a hard time trying to understand the MSE loss function given in the Introduction to Boosted Trees (beware! My maths skills are the equivalent of a very sparse matrix):

$
\begin{split}\text{obj}^{(t)} & = \sum_{i=1}^n (y_i - (\hat{y}_i^{(t-1)} + f_t(x_i)))^2 + \sum_{i=1}^t\Omega(f_i) \\
          & = \sum_{i=1}^n [2(\hat{y}_i^{(t-1)} - y_i)f_t(x_i) + f_t(x_i)^2] + \Omega(f_t) + constant
\end{split}
$

The second equality sign implies that one could easily derive the second equation from the first one, but I cannot see how.
My first naïve attempt was to:

express $y_i$ as $a$
express $(\hat{y}_i^{(t-1)} + f_t(x_i))$ as $b$
and then expand $(a-b)^2$

But I wasn't successful. Any help is really appreciated.

I recall I was struggling for some time deriving the second equation. That constant keeps many of your missing elements. Let's break it down using your $(a-b)^{2}$ notation. We will have $a^{2}$, $b^{2}$, and $2ab$:

$a^{2}$: $y_{i}$ is constant since it is your true labels/values, thus $a^{2}$ i.e. $y_{i}^{2}$  goes to the constant.
$b^{2}$: $y\hat{}_{i}^{t-1}$+$f_{t}^{2}$+$2y\hat{}_{i}^{t-1}f_{t}$

$y\hat{}_{i}^{t-1}$ is constant as it is prediction in a step before $(t-1)$ that we know already, thus goes to the constant term. The other two terms remain as it is.

$2ab$: $2y_{i}(y\hat{}_{i}^{t-1}+f_{t}) = -2y_{i}y\hat{}_{i}^{t-1}-2y_{i}f_{t}$. Here also the first term is constant. Only the second term remains.

The rest should be pretty straightforward, just add things that are left and clean it up and you see the second equation comes out beautifully.


Which Policy Gradient Method was used by Google's Deep Mind to teach AI to walk I just saw this video on Youtube.
Which Policy Gradient method was used to train the AI to walk?
Was it DDPG or D4PG or what?

They used Distributional Proximal Policy Optimization (DPPO). In the article that video is associated to, they provide a brief overview of it:

In order to learn effectively in these rich and challenging domains, it is necessary to have a reliable and scalable reinforcement learning algorithm. We leverage components from several recent approaches to deep reinforcement learning. First, we build upon robust policy gradient algorithms, such as trust region policy optimization (TRPO) and proximal policy optimization (PPO) [7, 8], which bound parameter updates to a trust region to ensure stability. Second, like the widely used A3C algorithm [2] and related approaches [3] we distribute the computation over many parallel instances of agent and environment. Our distributed implementation of PPO improves over TRPO in terms of wall clock time with little difference in robustness, and also improves over our existing implementation of A3C with continuous actions when the same number of workers is used.

Here are some resources:

The deepming blog post describing the method
The original article: Emergence of locomotion behaviours in rich environments


derivation for expected value for variance Hi Im taking a course about probability distribution in datascience and below  is  derivation of the expected value for the variance

Variance = expected value of the squared difference from mean for any value. But generally, variance is just the difference between the value and its mean.

Why are we squaring and adding the expected value symbol?
$$\sigma^2 = E((Y - \mu)^2) = E(Y^2) - \mu^2$$

For the first step in derivation, why do we multiply the summation of $p(x)$  with  $(x - \mu)^2$?

How is this substitution valid? I cannot understand it. I know that  $E(X)=p(X).X$

$E(X^2) = \sum P(X)*X^2$

The expected value of a random variable is defined as (without entering into probability/measure theory):

For discrete distribution $F$, we have $E(X)=\sum_x P_F(X=x) x$
For (absolutely) continuous distribution $F=\int f(x) dx$, we have $E(X) = \int f(x) dx$

Therefore, the first equality is just the definition of expected value for the discrete case.
And finally, the law of the unconscious statistician (LOTUS) states that
$$E(g(X)) = \int g(x) f(x) dx$$ (for continuous F), or
$$E(g(X)) = \sum g(x) P(X=x)$$ (for discrete F).


Reinforcement Learning - How are these state values in MRP calculated? This is a question from the book an Introduction to RL, page 125, example 6.2.

The example compares the prediction abilities of TD(0) and constant $ \alpha $ MC when applied to the below Markov reward process (the image is copied form the book):

In the above MRP, all episodes start in the state C then can go either left
or right by one state in each step (with equal probability).

  Episodes terminate either on the extreme left or the extreme right. When an episode terminates on the right, a reward of +1 occurs; all other rewards are zero. For example, a typical episode might consist of the following state-and-reward sequence: C, 0,B, 0, C, 0,D, 0, E, 1.
  Because this task is undiscounted, the true value of each state is the probability of terminating on the right if starting from that state.

The book mentions V(A)=1/6, V(B)=2/6, V(C)=3/6 , V(D)=4/6 and V(E)=5/6; could you please help me understand how these values were calculated? Thanks.

As it is such a small MRP, it is possible to solve it quickly and analytically using simultaneous equations based on the Bellman equation:

$$v(s) = \sum_{r,s'} p(r,s'|s)(r + v(s'))$$

and substituting in each state in turn. Using the variables $a,b,c,d,e$ to represent $v(A), v(B), v(C), v(D), v(E)$ makes this easier to read:

$a = \frac{1}{2}(0 + b)$
$b = \frac{1}{2}(a+c)$
$c = \frac{1}{2}(b+d)$
$d = \frac{1}{2}(c+e)$
$e = \frac{1}{2}(d+1)$

It takes multiple substitutions to resolve any individual value, at least to start with. But it is possible to resolve these terms by repeated substitution of how many times $a$ they are, and then proceeding to the next equation getting each variable:

$a = \frac{1}{2}(0 + b) \therefore b = 2a$

$b = \frac{1}{2}(a+c) \rightarrow 2a = \frac{1}{2}(a + c) \therefore c = 3a$

$c = \frac{1}{2}(b+d) \rightarrow 3a = \frac{1}{2}(2a + d) \therefore d = 4a$

$d = \frac{1}{2}(c+e) \rightarrow 4a = \frac{1}{2}(3a + e) \therefore e = 5a$

$e = \frac{1}{2}(d+1) \rightarrow 5a = \frac{1}{2}(4a+1) \therefore a = \frac{1}{6}$

That last one resolves everything, and gives the expected answer. You can also see a very strong pattern here, which should intuitively hold for any length of one dimensional random walk with equal $p=0.5$ - there is a proof for that more general case that would void the need to work the substitutions in detail for each length, but that is more complex.

One other simple intuition: Each state value in turn is the mean of the two state values either side of it, without exception. That means taking any three values next to each other, they should be co-linear; the two higher and lower values with the mean between them. As this holds on all values with overlap, plotting the values in order should place all points all on the same line. That in turn means that there is a simple linear relationship between position in the random walk and the state value.


TS-SS and Cosine similarity among text documents using TF-IDF in Python A common way of calculating the cosine similarity between text based documents is to calculate tf-idf and then calculating the linear kernel of the tf-idf matrix.

TF-IDF matrix is calculated using TfidfVectorizer().

from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer(stop_words='english')
tfidf_matrix_content = tfidf.fit_transform(article_master['stemmed_content'])

Here article_master is a dataframe containing the text content of all the documents.
As explained by  Chris Clark here, TfidfVectorizer produces normalised vectors; hence the linear_kernel results can be used as cosine similarity.

cosine_sim_content = linear_kernel(tfidf_matrix_content, tfidf_matrix_content)

This is where my confusion lies.

Effectively the cosine similarity between 2 vectors is:

InnerProduct(vec1,vec2) / (VectorSize(vec1) * VectorSize(vec2))

Linear kernel calculates the InnerProduct as stated here

So the questions are:

Why am I not divding the inner product with the product of the magnitude of the vectors ?
Why does the normalisation exempt me of this requirement ?
Now if I wanted to calculate ts-ss similarity, could I still use
the normalised tf-idf matrix and the cosine values (calculated by
linear kernel only) ?

Normalised vectors have magnitude 1, so it doesn't matter if you explicitly divide by the magnitudes or not. It's mathematically equivalent either way.

I see no reason that you couldn't use normalised vectors in TS-SS, but it seems that the main motivation for using TS-SS in the first place is that it makes more sense for vectors that may have different magnitudes. I would try both cosine similarity and TS-SS for your problem, and see if there is a noticeable performance difference.


Backpropagation: In second-order methods, would ReLU derivative be 0? and what its effect on training? ReLU is an activation function defined as $h = \max(0, a)$ where $a = Wx + b$.

Normally, we train neural networks with first-order methods such as SGD, Adam, RMSprop, Adadelta, or Adagrad. Backpropagation in first-order methods requires first-order derivative. Hence $x$ is derived to $1$.

But if we use second-order methods, would ReLU's derivative be $0$? Because $x$ is derived to $1$ and is derived again to $0$. Would it be an error? For example, with Newton's method, you'll be dividing by $0$. (I don't really understand Hessian-free optimization, yet. IIRC, it's a matter of using an approximate Hessian instead of the real one).

What is the effect of this $h''=0$? Can we still train the neural network with ReLU with second-order methods? Or would it be non-trainable/error (nan/infinity)?

For clarity, this is ReLU as $f(x)$:

$f(x) =$ \begin{array}{rcl}
 0 & \mbox{for} & x < 0\\
 x & \mbox{for} & x \ge 0\end{array}

$f'(x) =$ \begin{array}{rcl}
 0 & \mbox{for} & x < 0\\
 1 & \mbox{for} & x \ge 0\end{array}

$f''(x) = 0$

Yes the ReLU second order derivative is 0. Technically, neither $\frac{dy}{dx}$ nor $\frac{d^2y}{dx^2}$ are defined at $x=0$, but we ignore that - in practice an exact $x=0$ is rare and not especially meaningful, so this is not a problem. Newton's method does not work on the ReLU transfer function because it has no stationary points. It also doesn't work meaningfully on most other common transfer functions though - they cannot be minimised or maximised for finite inputs.

When you combine multiple ReLU functions with layers of matrix multiplications in a structure such as a neural network, and wish to minimise an objective function, the picture is more complicated. This combination does have stationary points. Even a single ReLU neuron and a mean square error objective will have different enough behaviour such that the second-order derivative of the single weight will vary and is not guaranteed to be 0.

Nonlinearities when multiple layers combine is what creates a more interesting optimisation surface. This also means that it is harder to calculate useful second-order partial derivatives (or Hessian matrix), it is not just a matter of taking second order derivatives of the transfer functions.

The fact that $\frac{d^2y}{dx^2} = 0$ for the transfer function will make some terms zero in the matrix (for the second order effect from same neuron activation), but the majority of terms in the Hessian are of the form $\frac{\partial^2E}{\partial x_i\partial x_j}$ where E is the objective and $x_i$, $x_j$ are different parameters of the neural network. A fully-realised Hessian matrix will have $N^2$ terms where $N$ is number of parameters - with large neural networks having upwards of 1 million parameters, even with a simple calculation process and many terms being 0 (e.g. w.r.t. 2 weights in same layer) this may not be feasible to compute.

There are techniques to estimate effects of second-order derivatives used in some neural network optimisers. RMSProp can be viewed as roughly estimating second-order effects, for example. The "Hessian-free" optimisers more explicitly calculate the impact of this matrix.


What information does output of [SEP] token captures in BERT? After reading around on the web I came to understand that the output representation of the special token [CLS] captures the representation of a sentence (am I correct?).

My primary question is what information does the output embedding of [SEP] token (T_SEP) captures?

My other doubt is if I input a bunch of sentences into BERT separated by [SEP] does the output embedding of [CLS] contain information about all the sentences?

You are right, CLS token tries to capture the sentence representation, because during pretraining this token is used to decide if 2 sentences are contiguous or not.

But the author mentioned that CLS token was not supposed to be a sentence representation, and should be used carefully.

The SEP token is a token used simply to separate sentence, in order to make is easier for BERT to know that the input is made of several sentences. Since the SEP token is not used at pretraining time, the SEP token does not represent anything.

About your last question : We don't know.

In pretraining, the model was not train with several sentence, so the model will not know how to handle several sentence.

But if you finetune it, the model may be able to learn new representation and can learn so that the CLS token contain information about all the sentences.

For example, this code finetuned BERT with a new pattern :

[CLS] Sen 1 [SEP] [CLS] Sen 2 [SEP] [CLS] Sen 2 [SEP] ...

And the CLS token is used to represent each sentence. Because the model is finetuned, BERT is learning to represent each sentence in the corresponding CLS token.


Sort Pandas Dataframe per column I have a dataset with age + another 14 variables. I have created 13 bins representing different age groups like so:
data["age_bins"] = pd.cut(data["age"], [16,20,25,30,35,40,45,50,55,60,65,70,75,80])

Then calculated the mean value of those 14 other variables per age group like so
data_age_bins_means = data.groupby(["age_bins"]).mean()

resulting in a 14 by 13 DataFrame called data_age_bins_means
Finally, I want to output a data structure with the 5 variables with the greatest mean value in descending order per each age group i.e. first sort each age group column separately and then choose those five variables with the greatest mean values for each age group. I was thinking about a MultiIndex solution but would badly need some help on a neat solution here. Many thanks!
P.s. I finally want to save that data structure to .json for easy loading to JavaScript

I think first converting the data from a wide to long format should allow you to sort the variables and get the result you want. This would look something like this:
import pandas as pd

# create example df
df = pd.DataFrame({
    "age_bin": ["0-10", "10-20", "20-30"],
    "col1": [5, 4, 2],
    "col2": [15, 23, 34],
    "col3": [32, 12, 24]
})

age_bin
col1
col2
col3

0
0-10
5
15
32

1
10-20
4
23
12

2
20-30
2
34
24

(
    df
    # convert from wide to long format
    .melt(id_vars="age_bin")
    # sort dataframe to make sure highest means are first
    .sort_values(["age_bin", "value"], ascending=[True, False])
    # create groups based on "age_bin" column
    .groupby("age_bin")
    # select top N rows within the groups created, change to 5 to get the top 5 columns
    .head(1)
)

Result:

age_bin
variable
value

6
0-10
col3
32

4
10-20
col2
23

5
20-30
col2
34


Actor-critic architecture: How is the policy updated? I am going through the ddpg baseline code to try and gain an intuitive understanding of how the actor and critic networks function.

DDPG has two components: the actor which is the deterministic policy \pi and the critic which is the state-value function Q(s, a). The way you update the actor \pi is by computing the gradient of Q(s, \pi(s)). The idea is that the policy can be seen as a continuous equivalent of argmax and so you try to update it such as it takes the action that maximizes the Q-function in a given state.

This can be depicted as shown below.

The code shows three different neural networks created.

actor_tf = create_neural_net(observations) # Maps states to desired actions

critic_tf = create_neural_net(observations, actions) # Updates value function
critic_with_actor_tf = create_neural_net(observations, actor_tf) # Used for policy updating

My question is with how the policy is updated, and more specifically with critic_with_actor_tf.

As explained here,

So critic_with_actor_tf represents Q(s,\pi(s)) the action-state value in a state s (here observation = state) following the policy pi (the actor) (a = \pi(s)). This is what is used to compute the gradient for the actor:

 self.actor_loss = -tf.reduce_mean(self.critic_with_actor_tf)

So, it seems like the actor updated by reducing the mean of critic_with_actor_tf.

This raises the question, what does the TD Error shown in the diagram above represent and how is that related to updating the policy?

You basically have two problems on your hand:

learning the Q-function
learning the policy

The Q-functions tells you how rewarding an action $a$ would be in state $s$. Once you take the action $a$ and transition from state $s$ to $s'$, you are rewarded $r$ right away. How rewarding it was to take the action $a$ in $s$ is the sum of the received reward $r$ and the Q-value of the state that you end up in, hence you want to minimize

$$r + Q(s', a', w) - Q(s, a, w)$$

where $w$ are the parameters of the function approximator and $a'$ the action you would take in the following state $s'$. This is the TD Error and mimimizing it will learn the Q-function.

So, the TD Error is used to learn the Q-function, basically the same way as it is done in plain-vanilla (Deep) Q-Learning. But while in plain-vanilla (Deep) Q-learning you would base the actor's policy on the Q-function entirely (plus some randomness to not get stuck), in the actor-critic context the Q-function is just a means to the end of training the policy itself. For this reason, the learned Q-function then serves as an optimization objective for the policy by learning the policy parameters $u$ such that it outputs the action $a$ which maximizes $Q(s,a,w)$ (see your other question). The chain rule pops up in that context, because you are not trying to do regression by minimizing some error, but rather you are trying to maximize $Q$ with respect to one of its arguments.


Problem with important feature having a lot of missing value I am facing a dilemma with a project of mine. One of the variables (numerical) doesn't have enough data i,e almost 99% data are missing. However, upon talking to the domain experts, it appears that the particular variable is important to the problem we are trying to solve (model). Initially, I thought of converting it to a binary variable such that 1 will represent that the variable has a value at that position and 0 will represent the missing value. However, it seems that we are missing information by doing it.

Can anybody suggest any way go forward?

One thought came to me is to discretize the variables using quantiles, but then what to do with the missing values?

Another one is to include both the binary variable along with the original variable in the model with missing values replaced by some imputed values. But I cannot come to any logical reasoning as to why or why not this will work.

Any light on this matter would be greatly helpful (other than of course drop it altogether).

Thanks.

What is the best way to deal with this kind of missing value problem can only be answered empirically? It will vary depending on your dataset and algorithm of choice. But here is a few things you can try.

Impute the missing value

Impute missing value of the mean
Impute missing value with special values. For example, If the variable takes only positive values, then you can encode the missing value as 0.

Try to predict the missing value

use the other variables to predict the missing value.
(However, if you can actually make a good prediction of the missing value out of the rest, then this might suggest you can drop this variable altogether)

  Another one is to include both the binary variable along with the
  original variable in the model with missing values replaced by some
  imputed values.

There is nothing inherently wrong with this method. You should certainly try. One possible downside of this method when you have 99% of value missing, this original variable is going to highly correlated with the derived is_missing variable. This can be problematic depends on the particular classification algorithm you are using. For example:

It is widely known that multicollinearity is a huge problem for any variant of linear regression
Support Vector Machine suffers a similar problem [Ref]
Naive Bayes assumes independence among features. This is even stronger.
The is_missing variable is a categorical variable, this makes it tricky to define a distance metrics in K-Nearest-Neighbour algorithm


TDIDT Decision Trees algorithm What is the Difference between TDIDT, ID3, CART, and C4.5?

My main concern is about TDIDT, Is it first ever algorithm that came with Decision trees?

Is it predecessor or successor of ID3, CART, and C4.5?

What are the differences to others(ID3, CART, and C4.5?)?

TDIDT stands for "top-down induction of decision trees"; I haven't found evidence that it refers to a specific algorithm, rather just to the greedy top-down construction method.  Therefore (seemingly) all the other algorithms you mention are implementations of TDIDT.  The first iteration is due to Hunt, the "Concept Learning System" in 1966.

ID3 is due to Quinlan in 1979, improving upon the CLS.  (Fun fact: it was originally designed to tackle the problem of deciding winnability of King-rook vs king-knight chess endgames.)  This was further improved to C4.5, then to C5.0.  This branch only works for classification.

CART ("classification and regression trees") was developed roughly in parallel with ID3, by Breiman, Friedman, Stone and Olshen in 1984.  As the name suggests, this branch allows prediction of continuous variables.

The main differences between branches is how splits are determined, using different metrics.  Early on it seems there were differences in handling missing data and such, but these appear to have mostly been included in all the more recent iterations.

Of course, there have been numerous other improvements or variations.  Some (historical?) tree building algorithms generate nonbinary trees, some make splits on linear combinations of features instead of just one at a time, some try to reduce the greedy nature of the algorithm by looking ahead, some produce regression models in each leaf instead of constant functions, ...

https://en.wikipedia.org/wiki/Decision_tree_learning#Decision_tree_types
W-Y Loh, "A Brief History of Classification and Regression Trees" (slides)
Lecture slides (Aida Nordman?)
Quinlan's 1986 paper
similar question on Quora
similar SO question
C5.0 introduction


Axis parameter in the Keras backend sum In Keras custom loss function, I often see axis parameter is set to -1.

def custom_loss(y_true, y_pred):
    intersection = K.sum(K.abs(y_true * y_pred), axis=-1)

I always thought that it should be 0, as 0 axis represents the batch. So why is set to -1?

Because y_true and y_pred have shapes (N,). Where N is the number of examples. By setting the axis to -1 you are calculating the sum over the last dimension, which happens to be over N (which is the batch)


Xgboost : A variable specific Feature importance I have a data set something like this:
data = [['Alex',10,13,1,0],['Bob',11,14,12,0],['Clarke',13,15,13,1],['bob',12,15,1,1]]
df = pd.DataFrame(data,columns = ["dealer","x","y","z","loss"])

I am trying to predict binary column loss, I have done this xgboost model. I got Overall feature importance. Now I need top 5 most important features dealer wise.
How to do that?
I have tried to use lime package but it is only working for Random forest.
If I get Feature importance for each observation(row) then also I can compute the feature importance dealer wise.
kindly help
Desired Output :

If you use a per-observation explanation, you could just average (or aggregate in some other way) the importances of features across the samples for each Dealer.

For example, using shap to generate the per-observation explanation:
import pandas as pd
from sklearn.linear_model import LogisticRegression
import shap

data = [['Alex',10,13,1,0],['Bob',11,14,12,0],['Clarke',13,15,13,1],['Bob',12,15,1,1]]
df = pd.DataFrame(data, columns=["dealer","x","y","z","loss"])

lr = LogisticRegression()
lr.fit(df[['x', 'y', 'z']], df['loss'])

# Whatever explainer you prefer:
explainer = shap.explainers.Permutation(lr.predict_proba, df[['x', 'y', 'z']])
shap_values = explainer(df[['x', 'y', 'z']])

# get just the explanations for the positive class
shap_values = shap_values[...,1]

shap_df = pd.DataFrame(abs(shap_values.values))
shap_df.columns = ['x_shap', 'y_shap', 'z_shap']
shap_df['dealer'] = df['dealer']
shap_df.groupby('dealer').mean()

produces

dealer
x_shap
y_shap
z_shap

Alex
0.260427
0.140054
0.075176

Bob
0.106593
0.069035
0.091146

Clarke
0.268328
0.083706
0.085807


Grouping similar rows to detect duplicates I have a dataset containing all the real estate ads for sale in the process of publishing a city:

    ID  URL CRAWL_SOURCE    PROPERTY_TYPE   NEW_BUILD   DESCRIPTION IMAGES  SURFACE LAND_SURFACE    BALCONY_SURFACE ... DEALER_NAME DEALER_TYPE CITY_ID CITY    ZIP_CODE    DEPT_CODE   PUBLICATION_START_DATE  PUBLICATION_END_DATE    LAST_CRAWL_DATE LAST_PRICE_DECREASE_DATE
0   22c05930-0eb5-11e7-b53d-bbead8ba43fe    http://www.avendrealouer.fr/location/levallois...   A_VENDRE_A_LOUER    APARTMENT   False   Au rez de chaussÃ©e d'un bel immeuble rÃ©cent,...   ["https://cf-medias.avendrealouer.fr/image/_87...   72.0    NaN NaN ... Lamirand Et Associes    AGENCY  54178039    Levallois-Perret    92300.0 92  2017-03-22T04:07:56.095 NaN 2017-04-21T18:52:35.733 NaN
1   8d092fa0-bb99-11e8-a7c9-852783b5a69d    https://www.bienici.com/annonce/ag440414-16547...   BIEN_ICI    APARTMENT   False   Je vous propose un appartement dans la rue Col...   ["http://photos.ubiflow.net/440414/165474561/p...   48.0    NaN NaN ... Proprietes Privees  MANDATARY   54178039    Levallois-Perret    92300.0 92  2018-09-18T11:04:44.461 NaN 2019-06-06T10:08:10.89  2018-09-25

I would like to group real estate ads for the same property. Indeed, several professionals (see individual) publish ads on several real estate portals for the same property.

ID  URL CRAWL_SOURCE    PROPERTY_TYPE   NEW_BUILD   DESCRIPTION IMAGES  SURFACE LAND_SURFACE    BALCONY_SURFACE ... DEALER_NAME DEALER_TYPE CITY_ID CITY    ZIP_CODE    DEPT_CODE   PUBLICATION_START_DATE  PUBLICATION_END_DATE    LAST_CRAWL_DATE LAST_PRICE_DECREASE_DATE

What methodology could I use to detect duplicates are the rows aren't ? So far I thought eitehr of checking if the description was exactly the same (but I would need to remove special characters before) or if the images on the url are the same (but I am not a star in image processing)

To my mind, as far as people that post the same houses does it on different website these won't be duplicates on the same website. Probably people post the same DESCRIPTION or the same IMAGES. The houses surely have the same surface however we may have several dealer for a same house. So I did:

# Let's add a new boolean column to our DataFrame that will identify a duplicated order line item (False=Not a duplicate; True=Duplicate)
df['is_duplicated'] = df.duplicated(['DESCRIPTION'])

And did the sum:

# We can sum on a boolean column to get a count of duplicate order line items
df['is_duplicated'].sum()

Which returned 249. I don't know how to compare the images yet.

Is there a better strategy?

I did this when i had duplicate values and multiple other variable which are unique.

adding unique variable to remove duplicates

"i have added variables as per above data please change as per your requirement"

Df <- DF%>%
  mutate(Path = paste(id,property_type,NEW_BUILD,SURFACE, sep = ">"))

retaining Unique values in the data

Df <-
  distinct(DF,Path, .keep_all= TRUE)


Putting a predictive model into production Even after all these years of data science from 2010 to 2018, why is there no general framework for putting a predictive model into production?

Depending on what exactly you mean by framework, I would argue that there is. Using a REST interface to serve a production model at inference time is pretty close to a general serving framework at this point.

However, there are a potentially infinite number of use cases that would diverge from this standard even if one categorically existed. Some mobile apps using image recognition models on personal data that the user doesn't want transmitted to the cloud are pushing the envelope of ways to compress formerly huge neural network architectures into ones that easily fit into mobile-sized memory, for example.

If you're talking software framework, it shouldn't be surprising that a field of programming practice (or, rather, plethora of fields) that find needs for a variety of languages and even ecosystems within languages would have difficulty forming a consensus around a single aporoach.

It's likely that deployment frameworks have yet to build in enough benefit to justify a reduction in general flexibility to justify their use.


Use a GPU to speed up neural net training in R I'm currently training a neural net model in R and am wanting to use a GPU to speed up this process.  I've looked into this but it appears that this is unavailable to Mac users as Apple no longer uses NVIDIA GPUs.
Can anyone tell me if this is the case, and if not how I can go about utilizing a GPU?

If you're able to convert the code into python, then you could use the Google colab environment or Kaggle kernels. These online platforms provide free GPU's that you can utilize.
Kaggle kernels also support R directly.


What is the difference between sklearn's transform() and predict() methods? What I have understood thus far is that transform() gives you values that are qualitatively similar to X (your features) and predict() gives you values that are qualitatively similar to y (your labels). But what I'm seeking some clarity over is why are there only a few classes that have BOTH of these methods, e.g KMeans, PLSRegression, etc.

Why doesn't it make sense to either put both methods in every class, or never let these two occur in the same class together? For example, if KMeans needed to have a method that returns the Euclidean points, why not let it have a separate method? Implementing transform() to achieve this functionality, in my opinion, takes away the clear distinction between the two methods. Similarly, in PLSRegression, I haven't been able to understand the difference between the two methods.

Partial Least Squares Regression is a supervised learning technique that also does dimensionality reduction. The PLSRegression class needs both transform and predict interfaces to properly implement the behavior of Partial Least Squares Regression. The transform method is the interface for dimensionality reduction. The predict method is the interface for generating targets from a trained regression model.
Not all Estimators in scikit-learn do both feature manipulation and prediction (typically target), thus not all classes are going to have both transform and predict interfaces.


What is the positional encoding in the transformer model? I'm trying to read and understand the paper Attention is all you need and in it, there is a picture:

I don't know what positional encoding is. by listening to some youtube videos I've found out that it is an embedding having both meaning and position of a word in it and has something to do with $sin(x)$ or $cos(x)$

but I couldn't understand what exactly it is and how exactly it is doing that. so I'm here for some help. thanks in advance.

For example, for word $w$ at position $pos \in [0, L-1]$ in the input sequence $\boldsymbol{w}=(w_0,\cdots, w_{L-1})$, with 4-dimensional embedding $e_{w}$, and $d_{model}=4$, the operation would be
$$\begin{align*}e_{w}' &= e_{w} + \left[sin\left(\frac{pos}{10000^{0}}\right), cos\left(\frac{pos}{10000^{0}}\right),sin\left(\frac{pos}{10000^{2/4}}\right),cos\left(\frac{pos}{10000^{2/4}}\right)\right]\\
&=e_{w} + \left[sin\left(pos\right), cos\left(pos\right),sin\left(\frac{pos}{100}\right),cos\left(\frac{pos}{100}\right)\right]\\
\end{align*}$$
where the formula for positional encoding is as follows
$$\text{PE}(pos,2i)=sin\left(\frac{pos}{10000^{2i/d_{model}}}\right),$$
$$\text{PE}(pos,2i+1)=cos\left(\frac{pos}{10000^{2i/d_{model}}}\right).$$
with $d_{model}=512$ (thus $i \in [0, 255]$) in the original paper.
This technique is used because there is no notion of word order (1st word, 2nd word, ..) in the proposed architecture. All words of input sequence are fed to the network with no special order or position; in contrast, in RNN architecture, $n$-th word is fed at step $n$, and in ConvNet, it is fed to specific input indices. Therefore, proposed model has no idea how the words are ordered. Consequently, a position-dependent signal is added to each word-embedding to help the model incorporate the order of words. Based on experiments, this addition not only avoids destroying the embedding information but also adds the vital position information.
This blog by Kazemnejad explains that the specific choice of ($sin$, $cos$) pair helps the model in learning patterns that rely on relative positions. As an example, consider a pattern like
if 'are' comes after 'they', then 'playing' is more likely than 'play'

which relies on relative position "$pos(\text{are}) - pos(\text{they})$" being 1, independent of absolute positions $pos(\text{are})$ and $pos(\text{they})$. To learn this pattern, any positional encoding should make it easy for the model to arrive at an encoding for "they are" that (a) is different from "are they" (considers relative position), and (b) is independent of where "they are" occurs in a given sequence (ignores absolute positions), which is what $\text{PE}$ manages to achieve.
This article by Jay Alammar explains the paper with excellent visualizations. The example on positional encoding calculates $\text{PE}(.)$ the same, with the only difference that it puts $sin$ in the first half of embedding dimensions (as opposed to even indices) and $cos$ in the second half (as opposed to odd indices). As pointed out by ShaohuaLi, this difference does not matter since vector operations would be invariant to the permutation of dimensions.


How to utilize user feedback due to miss-classification when correct class label is unknown? Suppose we are developing an app which is supposed to predict a dog's breed by it's picture. We trained a classifier (in my case an MLP) using some dataset and shipped the app to users. Now suppose some user comes and takes a picture of a friend's dog and the app tells her there is 90% chance that this dog is an X. The user knows that this is not true, but she doesn't know what is the dog's breed (if she knew, why would she use our app in the first place?). So we get a feedback which tells us "this is a picture of a dog which is not an X". This sample could be a sample of some other class or a new class or not a dog at all.

I'm looking for a way to use this feedback, to improve the precision of my MLP in class X without touching other classes.

This can be accomplished by a modification to multi-class cross-entropy.

We are faced with two types of supervision. First type is "data $i$ belongs to class $k$" denoted by $y_{ik}=1$, and second type is "data $i$ does not belong to class $k$" denoted by $\bar{y}_{ik}=1$. For example, for 3 classes, $y_i=(1, 0, 0)$ denotes that point $i$ belongs to class $1$, and $\bar{y}_{i}=(0, 0, 1)$ denotes that point $i$ does not belong to class $3$. Let $y'_{ik} \in [0, 1]$ denote the model prediction. The original cross-entropy for $K$ classes is:

$$H_y(y')=-\sum_{i}\sum_{k=1}^{K}y_{ik}log(y'_{ik})$$.

This objective assigns loss $-log(y'_{ik})$ to $y_{ik} = 1$ to encourage the model to output $y'_{ik} \rightarrow 1$ leading to $-log(y'_{ik})\rightarrow 0$.

On the other hand, for the second supervision $\bar{y}_{ik}=1$, we want to encourage the model to output $y'_{ik} \rightarrow 0$. Therefore, loss $-log(1- y'_{ik})$ can be used to have $-log(1- y'_{ik})\rightarrow 0$.

Accordingly, second supervision can be combined with first one as follows:

$$H_{(y,\bar{y})}(y')=-\sum_{i}\sum_{k=1}^{K}y_{ik}log(y'_{ik})+\bar{y}_{ik}log(1-y'_{ik})$$

Note that supervision "data $i$ does not belong to classes $1$ and $2$" is also supported. For example, $\bar{y}_{i}=(1, 1, 0,...)$ activates both $-log(1 - y'_{i1})$ and $-log(1 - y'_{i2})$ to encourage the model to output less probabilities for classes $1$ and $2$, i.e. $y'_{i1} \rightarrow 0$, and $y'_{i2} \rightarrow 0$.


How to prevent a neural network from choosing the 'easiest' solution I have a neural network that takes in roughly twelve values, and outputs a singly probability. The issue is that the network appears to be smart enough to realize there is a very significant correlation between three of the values and the output, and doesn't seem to be recognizing any correlation with the remainder of the inputs. In other words, the model has figured out that if the and of three specific inputs are non-zero, there is an ~80% chance the output is 1.

This seems like expected behavior, but I am looking to have the network enhance it's 80% prediction with data from the other nine inputs. Is this possible?

My proposed solution requires a bit more information regarding the nature of the data. The twelve inputs can be categorized into four groups of three variables each. For instance, we have three metrics: purchase cost, warranty length and average maintenance costs. These three metrics are then applied to four different appliances in a household: the boiler, washing machine, dishwasher and refrigerator. The network is then supposed to output the probability of the owner of the household moving out in a few months. In this example, there happens to be a 80% correlation between refrigerators and the homeowner moving: if they own a refrigerator, there is an 80% chance they will move out within the next few months.

As mentioned before, the network is smart enough to recognize the correlation between the refrigerator and the moving out, but I am hoping to get it to augment this prediction with the inclusion of the other inputs.

My first strategy for addressing this would be to split the single network into four separate networks, each handling one appliance. These will output four probabilities total, which I can then either weight and compute by hand, or feed into a fifth network for final processing. My worry is that this would result in the same issue occurring - the final network would determine that ignoring the first three inputs and only returning the fridge-based probability is the best way to minimize error.

My second possible solution is to simply up the amount of iterations over the dataset, in the hope that the network happens to locate a slightly better solution by chance, then works to optimize it. This seems unlikely, and overly dependent on the problem.

My third and final possible solution is to initially only train the network on input where the homeowner doesn't own a fridge (all fridge values are zero) such that the network optimizes it's solution without having to worry about the fridges, and then gradually introduce inputs from homeowners with fridges.

Will any of these work? Is there a better solution?

I am using the Tensorflow Estimator API.

Thanks!

This is fairly common with scenarios like predicting fraud when overall cases of fraud are uncommon. The model will frequently just say "no fraud!" and be right a large percent of the time. You need to assign a penalty, cost, or weight so that output 1 isn't so easy.

This article describes that further in a lot of detail:

https://blog.fineighbor.com/tensorflow-dealing-with-imbalanced-data-eb0108b10701


Extracting useful features for k-means clustering So say suppose I have a data-set with features being either present or not i.e. 0or 1. Now I want to identify the features which really help in  the clustering.

Like say I have 4 training examples. Now say I have a feature which is  present i.e = 1 in all the training example, thus I can remove the feature as it dies not help me. Now let's talk about 2 more features, if the number of training examples they are present in common is high, they also do not help much in clustering (think of 2 highly overlapping circles in a Venn Diagram). So in this way I want to find features which has a significant impact on the clustering i.e mostly non-overlapping features.

Is there any good way to do this? (my features are all represented in binary, either it is there or it is not).

There are many ways for features selection:

One possible way (Wrapper methods), is to start by clustering with all related features and save the clusters, then cluster again after removing 1 feature and compare the obtained clusters (without the feature) to the clusters (with the feature). If the difference between the results are negligible then you can eliminate that feature.

Filter methods include what you mentioned in your question, where you remove the feature that changes rarely or doesn't change at all which can be expressed mathematically by "entropy", "information gain", "Chi square test"...etc. these concepts measure the impurity of a feature.

To sum up, "Wrapper" methods usually provide better results but they are computationally more expensive.

For more please see this


How to get Keras accuracy for each step in an epoch like in Tensorflow? Like in tensorflow I get accuracy for each step -

Step 1, Minibatch Loss= 68458.3359, Training Accuracy= 0.800
Step 10, Minibatch Loss= 451470.3125, Training Accuracy= 0.200
Step 20, Minibatch Loss= 582661.1875, Training Accuracy= 0.200
Step 30, Minibatch Loss= 186046.3125, Training Accuracy= 0.400
Step 1, Minibatch Loss= 161546.6250, Training Accuracy= 0.600
Step 10, Minibatch Loss= 286965.3125, Training Accuracy= 0.400
Step 20, Minibatch Loss= 205545.7500, Training Accuracy= 0.600
Step 30, Minibatch Loss= 202164.6562, Training Accuracy= 0.800
Step 1, Minibatch Loss= 214717.7969, Training Accuracy= 0.600
Step 10, Minibatch Loss= 108088.7344, Training Accuracy= 0.800
Step 20, Minibatch Loss= 80130.6016, Training Accuracy= 0.800
Step 30, Minibatch Loss= 28674.1875, Training Accuracy= 0.800
Step 1, Minibatch Loss= 78675.6641, Training Accuracy= 0.400
Step 10, Minibatch Loss= 168231.2812, Training Accuracy= 0.600
Step 20, Minibatch Loss= 77828.1406, Training Accuracy= 0.600
Step 30, Minibatch Loss= 56584.9609, Training Accuracy= 0.800
Step 1, Minibatch Loss= 29474.0898, Training Accuracy= 0.600
Step 10, Minibatch Loss= 79742.9531, Training Accuracy= 0.800
Step 20, Minibatch Loss= 0.0000, Training Accuracy= 1.000
Step 30, Minibatch Loss= 6736.4688, Training Accuracy= 0.800

But in keras I get accuracy for each epoch -

156/156 [==============================] - 6s 39ms/step - loss: 13.0185 - acc: 0.1923
Epoch 2/10
156/156 [==============================] - 3s 18ms/step - loss: 12.9151 - acc: 0.1987
Epoch 3/10
156/156 [==============================] - 3s 18ms/step - loss: 13.1218 - acc: 0.1859
Epoch 4/10
156/156 [==============================] - 3s 18ms/step - loss: 12.9151 - acc: 0.1987
Epoch 5/10
156/156 [==============================] - 3s 18ms/step - loss: 13.1218 - acc: 0.1859
Epoch 6/10
156/156 [==============================] - 3s 18ms/step - loss: 12.9151 - acc: 0.1987
Epoch 7/10
156/156 [==============================] - 3s 18ms/step - loss: 12.8118 - acc: 0.2051
Epoch 8/10
156/156 [==============================] - 3s 18ms/step - loss: 12.8118 - acc: 0.2051
Epoch 9/10
156/156 [==============================] - 3s 18ms/step - loss: 12.8118 - acc: 0.2051
Epoch 10/10
156/156 [==============================] - 3s 18ms/step - loss: 12.9151 - acc: 0.1987

Create a custom CallBack:

from tensorflow.keras.callbacks import Callback

class NBatchLogger(Callback):
    "A Logger that log average performance per `display` steps."

    def __init__(self, display):
        self.step = 0
        self.display = display
        self.metric_cache = {}

    def on_batch_end(self, batch, logs={}):
        self.step += 1
        for k in self.params['metrics']:
            if k in logs:
                self.metric_cache[k] = self.metric_cache.get(k, 0) + logs[k]
        if self.step % self.display == 0:
            metrics_log = ''
            for (k, v) in self.metric_cache.items():
                val = v / self.display
                if abs(val) > 1e-3:
                    metrics_log += ' - %s: %.4f' % (k, val)
                else:
                    metrics_log += ' - %s: %.4e' % (k, val)
            print(f'step: {self.step}/{self.params['steps']} ... {metrics_log}')
            self.metric_cache.clear()

The inspiration for the code is here.


CNN, which layer to choose for a similarity measure I built a model (InceptionResnet v2) to classify images and I would like to use it to measure similarity between objects.

One way to measure that similarity is to catch an intermediate layer's results from two vector and look their distance.

I don't know how to choose the layer, my last layers are :

Conv -> Flatten(98304,) -> Dense(256,) -> Dense (softmax)

I hesitate between Flatten and first Dense, how to choose ?

For similarity computations, you should generally prefer the last (before softmax) dense layer. This layer will generally capture the most higher level representation of the images. It is also preferable to have the size of vector as small as possible when similarities are to be computed for efficiency reasons.


Learning a threshold from a matrix I have a 2D matrix, and I want to learn a threshold from this matrix. All items in the matrix bigger than the threshold will become 1s, and all smaller than the threshold will become 0s. Then, I have a differentiable loss function to evaluate the resulting 0-1 matrix. What methods are typically used to learn the threshold? Do we put the matrix across a network, or are there any non-neural-network methods?

The hierarchy of methods for thresholds are:

Hand-coded thresholds
Learned thresholds with a decision tree
Learned thresholds with an ensemble of trees (e.g., Random Forest or XGBoost)
Learned thresholds with neural networks


converting array to a true/false matrices I have a data set where each record is a json document with a label, and an array of signals. The signals will vary for each record:
{
    "label":"bad",
    "id": "0009",
    "signals":["high_debt_ratio", "no_job"]
},

{
    "label":"good",
     "id": "0002",
    "signals":["high_debt_ratio", "great_credit", "no_id_match"]
},

{
    "label":"good",
    "id": "0003",
    "signals":["low_debt_ratio", "great_credit"]
},

{
    "label":"bad",
    "id": "0001",
    "signals":["high_risk_loc", high_debt_ratio", "no_job", "no_id_match"]
}

I want to convert this to a matrices that looks like this:

id
label
high_risk_loc
high_debt_ratio
no_job
great_credit
no_id_match
low_debt_ratio

0009
bad
false
true
true
false
false
false

0002
good
false
true
false
true
true
false

0003
good
false
false
false
true
false
true

0001
bad
true
true
true
false
true
false

I created a function but it seems like this would be a common thing to do. Is there a python lib (pandas, scikit, etc.) that does this for you? I'd rather use something from a package but i'm not sure what to search for.

You can try this one line solution:
d = pd.json_normalize([{
    "label":"bad",
    "id": "0009",
    "signals":["high_debt_ratio", "no_job"]
},

{
    "label":"good",
     "id": "0002",
    "signals":["high_debt_ratio", "great_credit", "no_id_match"]
},

{
    "label":"good",
    "id": "0003",
    "signals":["low_debt_ratio", "great_credit"]
},

{
    "label":"bad",
    "id": "0001",
    "signals":["high_risk_loc", "high_debt_ratio", "no_job", "no_id_match"]
}])

d.merge(d.signals.apply(lambda x: "|".join(x)).str.get_dummies(sep = "|"), left_index = True, right_index= True)

Outputs:


How can you build a model based on non-independent imbalance data? I am trying to predict customer churn based on the data that I have. I am defining churn as an activity that is not followed by another activity within a week. The customer might come back in two months and become active again and those activities are not considered churn.

Therefore, the same user might be considered churned or not churned depending on future behavior. Also, only 0.5% of these observations result in churn.

Now for this data set, I have both dependant and imbalance data. How can I perform modeling, or even simple t-test (as its assumption is independence)?
Any idea or direction is highly appreciated.

So in essence (if I understand correctly), you want to predict a customer "not being active" within a week versus "being active"?! So you have a binary problem. The choice of method depends on your data. But a good start might be, for instance, catboost, a gradient boosting tool. I have good experiance with it.

https://catboost.ai/docs/concepts/python-usages-examples.html

With the CB classifier, you can specify class_weights, which might help to cope with unbalanced classes.

https://catboost.ai/docs/concepts/python-reference_catboostclassifier.html#python-reference_catboostclassifier

See also: https://stackoverflow.com/a/45708903/9524424

Here is a Python tutorial: https://github.com/catboost/tutorials/blob/master/python_tutorial.ipynb

Regarding the "non-independent" aspect: Do you mean that the SAME customer can fall in both classes at different points in time? Maybe this is not too much of a problem as long as something changes in the X variables (the ones used for prediction). Time certainly changes, right? So you might include time, e.g. as a factor (or dummy) variable. Often this works well with gradient boosting. Maybe (just an idea) you could also add a customer ID in addition to time, in order to try to control for each individual customer or (similar) groups of customers.

Alternative methods/tools would be lightgbm, logistic regression (with regulation), or even neuronal networks (e.g. Keras/tensorflow based). Time can also be included as count or dummy in logistic regression. In neuronal networks, dealing with time often is a bit complicated. To say more about it, one ultimately need to see the data in detail (number of X variables/columns, number of observations etc).


Difference between parameters and weights in ANN I think in writing we often refer to the part of the neural networks that tell us about the data are the parameters when we really mean the weights, is this correct? Would the sentence...

  Better understanding parameters of the neural network after training on bird migration data can allow us to comprehend the behavior of these animals.

...be better phrased as...

  Better understanding the weights of the neural network after training on bird migration data can allow us to comprehend the behavior of these animals.

That's partially right.

Apart from the weights (and bias) you also have the so called hyper-parameters. Depending on your architecture, you may have different ones. For a simple feedforward neural network, the hyperparameters are:

number of neurons
number of layers
learning rate eta (η)
regularization penalty lambda (λ)
momentum
number of epochs
batch size
dropout

etc

These hyperparameters are (usually) user-tuned. The weights (and bias) are self-trained by the network, based on optimization. Keep in mind that hyperparameters like the number of neurons or layers are immediately connected with the underlying model of your data, therefore they are an index of your dataset's complexity.


How to paste string and int from map to an array in hive? I am trying to paste a string and int from map in Hive to an array.
For now, record looks like this:

{"string1":1,"string2":1,"string3":15}

Is there a way to convert it to an array like this:

["string1:1","string2:1","string3:15"]

Assuming your map is called "M" and you want your array field to be called "A"

SELECT
...
array(concat_ws(":","string1",M[1]),
      concat_ws(":","string2",M[2]),
      concat_ws(":","string3",M[3]) as A
....
FROM table;


Are there any machine learning algorithms that focus on comparing items, rather than classification or regression? This is more of a hypothetical than something I'm actively trying to solve. It just struck me that a machine learning algorithm that specifically looked at two pieces of data and had to label one as greater than the other or something of that nature might be inherently different than classifying each separately and comparing the strength of the classifications.

There are ranking algorithms based on machine learning that are aimed to build ranking models. Training data for these models is given in the form of partial ordering between each pair of elements in a sample. A brief description, together with a list of useful references, is given in the corresponding Wikipedia page.


Text annotating process, quality vs quantity? I have a question regarding annotating text data for classification.

Assume we have ten volunteers who are about to annotate a large number of texts into label A or B. They probably won't have time to go through all the text samples, but at least a significant portion of them.

Should we focus on generating new samples for each annotator? (They never see the same text samples as any other annotator) (quantity approach).
Or should all annotators see the same samples and the annotator agreement is taken to account? (quality approach).

Thoughts,

will generate more unique samples than 2. (more training samples for a classifier) - and hoping that in the feature extraction part, the useful features will appear by themselves.
will generate less unique samples, but with the annotator agreement taken into account. (less training samples for a classifier, but with higher quality)

Both metods are against the nature of machine/statistical learning. Lack of data will not let you generalize well in any case. It will have different effect though. Generally speaking:

You will be able to classify more test samples, but sometimes will do
it wrong due to errors in data.
You will have precise classification of some test samples, but will have no idea how to classify the rest of them.

So, it's up to your's classification score/quality estimation function which case is better for you.


How to collect data which exists centre or border of the cluster? After applying clustering algorithm I need to extract those data which exists centre of the cluster and which exists border of the cluster. How could I do this by using python.

I use k-means clustering algorithm and divide those data into 19 cluster. I am also using scikit learn library.

Edited:

Here's my plot after clustering:

You can use numpy.where documentation can be fouind here :
https://docs.scipy.org/doc/numpy-1.10.1/reference/generated/numpy.where.html

You can define a function to extract your data ID somthing like this

def ClusterIndicesNumpy(clustNum, labels_array):
    return np.where(labels == cluster)[0]

To get samples from cluster 3 for example:

ClusterIndicesNumpy(3, km.labels_)


Word2Vec - CBOW and Skip-Grams I'm wondering how Word2Vec is being constructed.

I've read tutorials simply stating that we can train a skip grams neural network model and use the weights that are trained as word vectors.

However, I've also seen this picture:

If reading this diagram correctly:

1) The CBOW and Skip Grams model are both trained with some inputs

2) The output of CBOW is used as an input to the middle neural network

3) The output of skip grams is used as an output to the middle neural network.

The output of CBOW is the a prediction of the center word given a context, and the output of skip grams is the prediction of the surrounding center word.

These outputs are then used to train another set of neural network.

Hence we first train the CBOW, and Skip-gram neural network, then train the middle neural network afterwards? And the input to the middle neural network is one hot encoded.

Is the above interpretation correct?

There is only one neural network to train in word2vec. CBOW, continuous-bag-of-words', and skip-gram are two different methods of constructing the training data for the neural network. You have to pick one of those two training methods.


How to interpret the AUC score in this case? I just run a random forest model on a imbalance dataset. I got the set of AUC and the confusion matrix. The AUC seemed not bad but actually the model predict every instance as positive. So how it happened and how to use AUC properly?

The ROC Curve as below:

I plot out the predicted probability of positive class in test set. The probability was within a tight range (0-0.4).

If you want to figure it out how this ROC happens , you would better LIST the tuples including your "predicted" values and the "truth" values, and SORT with the "predicted" value , then PLOT the ROC .
In your case , the tuples and points should be like this :
  predicted truth (x,y)
    0.53 0  (6/6,14/14)
    0.55 0  (5/6,14/14)
    0.57 1  (4/6,14/14)
    0.59 0  (4/6,13/14)
    0.60 1  (3/6,13/14)
    0.62 1  (3/6,12/14)
    0.63 0  (3/6,11/14)
    0.64 0  (2/6,11/14)
    0.66 1  (1/6,11/14)
    0.68 1  (1/6,10/14)
    0.71 1  (1/6,9/14)
    0.73 1  (1/6,8/14)
    0.77 1  (1/6,7/14)
    0.78 0  (1/6,6/14)
    0.82 1  (0/6,6/14)
    0.86 1  (0/6,5/14)
    0.89 1  (0/6,4/14)
    0.92 1  (0/6,3/14)
    0.94 1  (0/6,2/14)
    0.96 1  (0/6,1/14)
            (0/6,0/14)

Put these points (x,y) on ROC picture , it should be like this :

Just much like that of yours !
By the way , if you want to know how these points are figured out , you can check the code written in scala in the below :
    def computeAuc(predict: BDV[Double],groundTruth: BDV[Double]): Double
    = {

        // retrieve number of positive and negative samples in ground truth
        val nPos = groundTruth.toArray.filter(_>0).length
        val nNeg = groundTruth.toArray.filter(_<=0).length

        // tuple predict with ground truth , and sort with predict
        val pair = predict.toArray.zip(groundTruth.toArray)
        val sortedPair = pair.sortBy(_._1)
        var auc = 0.0.toDouble
        val x = BDV.zeros[Double](predict.length + 1)
        val y = BDV.zeros[Double](predict.length + 1)
        x(0) = 1.0
        y(0) = 1.0

        // calculate auc incrementally
        var i = 1.toInt
        while(i < sortedPair.length) {
          y(i) = (1.0 * sortedPair.slice(i,pair.length).filter(_._2 > 0).length) / nPos
          x(i) = (1.0 * sortedPair.slice(i,pair.length).filter(_._2 <= 0).length) / nNeg
          auc = auc + (((y(i) + y(i - 1))*(x(i - 1) - x(i)))/2.0)
          i += 1
        }
        auc = auc + ((y(i - 1) * x(i - 1))/2.0)

        auc
   }

Finally , your imbalance problem is severe , and you would better do down-sampling or up-sampling before training .
Hopes this contributes you , good luck -)


What algorithm should I use to find which sub-component is associated with a attribute? I have a n x m one hot encoded matrix of a number of units, where n corresponds to the number of units, and m is the number of total sub-components. This corresponds to a n x 5  matrix which has the 5 attribute values for each of these units.

If I want to create clusters of these sub-components corresponding to these 5 attributes (basically see which sub-component is associated with which attribute), which technique should I use? Would something like PCA or t-SNE work or does this problem require a different approach and why should I use what I use?

I am quite new to this, I would appreciate any help on this matter.

The type of clustering depends on the measures scale of the data. You stated you have 5 attribute values. Are those values nominal, ordinal, interval, or ratio?
If they are nominal, correspondence analysis is a common option.
If they are ordinal, k-medoids clustering could work.
If they are interval or ratio, k-means clustering is a very popular choice.


One starting approach is doing the cross-correlation as you've said. The correlation will fail to capture the time shifted behavior. But you should calculate it for every pair of Milk/Drug you have.

The ccf() function in R does it for you.

If you're not successful i suggest you to take a look at the multiple time-series plotted side by side (like here) and check if really there exists this time-shifted pattern in your customer behavior.


Calculate image width In this code below, a picture can be loaded into openCV and then the region of interest RIO can be created by just selecting a box around something with the mouse, then press enter.

What I am trying to do is measure the apparent width in pixels of the ROI. I am going thru some of the theory in this pyimagesearch blog post for calculating distances in images.

I have a folder called images and a .png file named 1 within the folder.

import cv2

image = cv2.imread("images/1.png")
cv2.imshow("image", image)
marker = cv2.selectROI("image", image, fromCenter=False, showCrosshair=True)

print(marker)

print(marker[1]-marker[3])

The output is below but I am not really sure how/if this can be used. I think its stating the ROI x,y height & width but I am not sure:

(317, 181, 269, 359)
-178
>>>

Any tips help... Thanks

cv2.selectROI() returns a 2D rectangle with (x, y, width, height) (see the Rect constructor - that object is created from selectROI()).

So, if you want to measure the apparent width in pixels, you need marker[2].


General strategy for imbalanced, semi-supervised, sparse problem I am looking for some general advice on where to start with this problem.  There are 350 sparse (low positive integer) features.  I have 2000 positives, 1000 negatives, and infinite unlabeled data, where the estimated true positive rate in the unlabeled data is ~1% (but this isn't known for certain).  The goal is to predict the probability of an example being positive.

How would you approach the problem if you had roughly 40 hours to attain the best possible model?

What about if you had more time?

It's a binary semi-supervised classification problem. First, establish a base-line for the supervised case. Then try if the unlabeled data helps

Supervised

From your labeled data: create a training, validation and test set. Don't touch the test set until the very end.
Try something simple, e.g. a multilayer Perceptron (MLP) with 350 input nodes and 1 output node (giving the probability of "true").
Try more stuff (e.g. https://github.com/MartinThoma/algorithms/blob/master/ML/mnist/many-classifiers/python.py#L92).
Try to combine classifiers in ensembles (see examples).
A (naive) bayes classifier might be worth being investigated.

Semi-supervised

You could train an auto-encoder with a bottleneck on the unlabeled data. Then remove everything after the bottleneck. Use this network as a preprocessing step. The idea is that this network finds a more meaningful abstraction of the relevant data. However, similar as PCA can have a very low projection error and still destroy the possibility to distinguish the classes.
SVMs can also be used in a semi-supervised setting.
You could try to find clusters in the unlabeled data + labeled data. Then you can get an a priori probability of "true" for each cluster (ignoring the unlabeled data). You could train very small models (overfitting!) on the data in the clusters.

Usually, you can interpret features. This might help to develop new features.


perform cluster on a multiple dimensional data in R I have a data set which has 2488 samples and each sample has 13 features.Now I want to perform cluster on this data set in R but I found k-means method usually for two dimensions data.So can any one help me? Many thanks!

Short answer
You can use any number of dimensions for k-means. Therefore, you can use the standard k-means library for R as long as:

your data contains only metric variables and
every column is scaled to the same range.

See this related question.
If your data is not scaled:
Scale it.
If your data contains only categorical variables:
Use k-modes. [R]
If your data contains categorical and metric variables:
Use k-prototypes. [R]

Long answer
How does k-means work?

The algorithm randomly initializes k points in your data. These are your cluster centers.
It assigns every data point to the nearest cluster center. Usually, this is done by calculating the Euclidian distance of the data point to each of the k cluster centers. Now you have k clusters.
It chooses the centroid of each of the k clusters as the new cluster center.
The steps 2 and 3 are repeated until the solution converges or the maximum number of steps is reached.

What this means
The Euclidian distance can be calculated in n-dimensional space. Therefore, you can use any number of dimensions you want.
However, if your data is not scaled, the distance calculations will be different for every feature. Therefore, you need to scale it.
Since Euclidian distance won't make a lot of sense with categorical or with mixed data, you have to use an algorithm that uses Hamming distance instead for these use cases. k-modes uses only Hamming distance. k-prototypes uses Hamming distance for categorical and Euclidian distance for metric variables.


Pre trained dataset for Car damage detection I'm making a Car Damage Detection model which would have 2 classes to detect upon. My dataset has a total of 300 images (out of which I'd be using some for testing), which are totally insufficient to train the model from scratch.

Can I use a pre-trained model to train my dataset and detect images based on the 2 classes, and if yes, then which one should be the best on my problem set?

P.S. - I would prefer to use Faster-RCNN on my dataset

For this task you should use any model pre-trained on the COCO's dataset.
We managed to get quite good results with only 80 images with a pre-trained MaskRCNN.


sklearn SVM really slow I'm working with sklearn SVM and I have a problem.

When I run the method sklearn.SVM.SVC.fit() using a database with only a few features (< 10) it takes a very long time. This is weird because, when I run the same method with the same database using all of the features (> 100) it takes just a few seconds.

I'm using a polynomial kernel and this problem only appears when the degree is >= 3. And it is also weird that if I use only a few features, but I add 50 features with 0s in all the samples, it works fine!

I have tried the following experiments:

fit() with all the features and any kernel degree. Works fine.
fit() with a few features and kernel degree <= 2. Works fine.
fit() with a few features and kernel degree >= 3. Very slow.
fit() with kernel degree = 3, a few features but adding 50 features with 0s in all the samples. Works fine!!

My question is, how can I solve this? Does my trick of adding features with 0s affect the classification?

Support Vector Machines attempt to find a hyerplane that divides two classes with the largest margin.  Therefore this is an optimization problem.  Support vectors are the points that lie along the supporting hyerplane.  One key factor that plays into the complexity of the runtime for a support vector is the slack parameter C.   The slack parameter allows for a soft-margin and better generalization.  The number of support vectors varies depending on how much slack we allow and how the data is distributed.  The less slack we give the SVM the fewer support vectors we get and converserly the more slack we give it the more support vectors we receive.

There also exist a relationship between the complexity of the model. The more complex the models tend to require more support vectors.

In short, it doesn't really matter how many features you feed it, the SVM computational complexity is linear in the number of support vectors.


Image resizing and padding for CNN I want to train a CNN for image recognition. Images for training have not fixed size. I want the input size for the CNN to be 50x100 (height x width), for example. When I resize some small sized images (for example 32x32) to input size, the content of the image is stretched horizontally too much, but for some medium size images it looks okay.

What is the proper method for resizing images while avoiding the content being destroyed?

(I am thinking about padding images with 0s to complete size after resizing them to some degree keeping ratio of width and height. Would it be okay with this method?)

This question on stackoverflow might help you. To sum up, some deep learning researchers think that padding a big part of the image is not a good practice, since the neural network has to learn that the padded area is not relevant for classification, and it does not have to learn that if you use interpolation, for instance.