Powers of 2 for batch_size in model fit in deep learning I am currently reading Deep Learning with Python by Francois Chollet, the author of Keras, and in one of his definitions for Mini-batch, he explains that the power of 2 for the batch_size is due to memory allocations in gpu/ Could anyone elaborate on this?

  Mini-batch or batch—A small set of samples (typically between 8 and
  128) that are processed simultaneously by the model. The number of
  samples is often a power of 2, to facilitate memory allocation on GPU.
  When training, a mini-batch is used to compute a single
  gradient-descent update applied to the weights of the model.

It's not just on GPU, you want to have a small power of two as submultiple for CPU as well.

The reason is to allocate the processors on the GPU (or the SIMD registers) as best as possibles. This is directly linked to the warp sizes and what the GPU vendors offer for maximum block sizes,

See also this thread on SO which what the warp size is and its relation to best occupancy.


Test data significantly different from training data What are the risks if the test data is significantly different from the training data?

Is the most significant problem related to both?

The main risk is underfitting, a model trained on a significantly different dataset will poorly predict the test set


Error using twitter R package's userTimlien I am using twitteR package to retrievie timeline data. My request looks as follows:

tweets <- try(userTimeline(user , n=50),silent=TRUE)

and this worked quite well for a time, but now I receive this error message:

Error in function (type, msg, asError = TRUE)  : easy handle already used in multi handle

In a related question on Stackoverflow one answer is to use Rcurl directly but this does not seem to work with twitteR package. Anybody got an idea on this?

It seems to be working well on my configuration:

Ubuntu Vivid and R:

> sessionInfo()
R version 3.1.2 (2014-10-31)
Platform: x86_64-pc-linux-gnu (64-bit)

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] twitteR_1.1.8

loaded via a namespace (and not attached):
 [1] bit_1.1-12     bit64_0.9-4    bitops_1.0-6   DBI_0.3.1      httr_0.6.1
 [6] magrittr_1.5   RCurl_1.95-4.6 rjson_0.2.15   stringi_0.4-1  stringr_1.0.0
[11] tools_3.1.2

Maybe you should update packages versions?


Reordering feature and its impact How does reordering the features impact model training and its performance?

Per my understanding, it should not impact the model performance as weights get tuned according to feature value and not per the feature order.

Interested to know from others as well.

It depends on the type of dataset you have. For instance, if you are trying to classify different types of flowers, the order in which you train the model on the features is irrelevant.

However, that being said, if you are dealing with non-stationary data such as time series values i.e. stock market prediction, the order of the features relative to time, is very important.


Does self driving technology gain more from the data or the state-of-the-art algorithm? When people start to figure the self driving cars will replace some vehicles on the road in the very near future, it somehow means the learner in the robot software could reach a very low empirical error to ensue the safety of passengers. The great deal of driving data collected and the deep reinforcement learning algorithms contribute mostly to the success of the self driving practice as we see it.

When it comes to many cases of our own practice, we yet cannot build up a predictive model to such a high accuracy to bring the huge difference to our own business.

My question is, when I saw the machine learning techniques can make revolution on the car driving due to its successful algorithms to implement behavioral cloning tasks well, what is the foremost reason for this success, the big training data set or the deep reinforcement learning techniques or any particular reason in self driving problems?

Furthermore, can this success be copied in most of the practical problems we solve in machine learning? In other words, if today's machine learning techniques can help a car drive by itself, how can I reach the same success when developing my own predictive models to boost my business? If we cannot, what are the constraints, not enough data in the particular case or not a smart model yet? The reason the self driving thrives can help answer this question.

I'll give a high level answer.

Our Bias on Effort

Although self-driving technology is now on the verge of becoming mainstream, incredible effort has gone into its development. The DARPA Grand Challenge, which helped in this technology's advancement, started in 2004. Imagine the total effort that went into this technology over 13-14 years. Many labs were setup with sole aim of competing in this challenge.

So when you say that we can't replicate the success in our businesses directly, it is better to ask if the comparison is fair. In your business, have you worked on a single problem to such an extent that many bright minds have worked on it for many years? If not, the comparison is not fair (to your business!).

Self-driving cars are not overnight success due to a single algorithm.

No Single Secret Sauce

Media reporting makes us believe that there's one algorithm that rules the technology, reality is more complex. Check this review paper on the complexity of tasks. There are many modules involved (sensors, perception, planner and control module, feedback loops). Yes, probability and machine learning plays a large part in all modules, but there's no single oracle algorithm. Old workhorse algorithms like Kalman Filters also shoulder heavy workload, but wouldn't get mentioned in the deep learning hype.

Yes large data helps, reinforcement learning helps. But so do rules inferred from manual interventions and control theory and older algorithms.

Summary

Many areas from engineering (classic control theory, reinforcement learning, machine learning) form the basis of self driving cars. We can't copy over the success to other domain easily: there's no single secret sauce but tons of hard work and solid engineering over long time.


How to properly predict date using Orange 3 First of all - I'm new to all of this.

I'm trying to create a model to predict the release of ios 12 based on previous years. I've got an excel that has a format like this:
ios version | release name | date

Each version consists of about 5 betas + general release.
I've set up the import like this:

Secondly I've set up flow like this:

First step selects all the rows that have [date] param - those go to either kNN or Linear Regression, everything else (1 row with ios 12 release date) goes to prediction and then to a table.

Cool, now depending on the modeling I get different results (as expected). It's either 2015-09-17 (kNN) or 2018-09-10 (LR). I've used the Test & Score, which gives me:

If I'm reading this properly - Linear Regression is more accurate (R2 column), meaning that iOS 12 will release on 10th of Sept 2018  (DUH! not 2015).

But if I look at release dates up to now, this might be wrong, because no other release happened so soon:

So I also did an exercise where I've adjusted the dates, so that they are all in 2018 (so that the year is less important, and I'd care more about relation within each year, given the features) - here are predicted results:

and Test & Score for the offsets:

Now... I would welcome all comments.
Am I using the wrong modeling? Something other than kNN & LR? Should I be using offsets? Am I using the tool completely incorrectly?

So you have two independent variables. iOS version and release name, which is actually a type. Both are categorical and only one has any predictive power, because iOS version you are trying to predict does not appear in your training set. Thus, effectively, you are predicting based on the release type only and only the "release" category is actually meaningful because this is what you are trying to predict.

I can hardly think of a model that would perform ok with such limited information. Anyway, regression models will always give you some answer, even if your input is unsound.

I guess the better features would be:

month
day of month
days since last release
months since last release
years since last release
days since beta
months since beta
years since beta

This type of thinking. Then just by looking at the data you would clearly see that prod releases of iOS happen always in mid Sept each year. No need for model to figure it out :) Any method would suffice, especially kNN or least squares.


How does the Nearest Centroid method work? I have read this Wikipedia article.

But, the idea is still very fuzzy to me.

Suppose, k=5.

Then, we have,

$X_5 = \{A, B, C, D, E\}$

$Y_2 = \{Triangle, Square\}$

$R_5 = \{9, 8, 5, 1, 4 \}$ (just assumed)

Now, $\mu_{Triangle} = \frac{5}{2} = 2.50$

and, $\mu_{Square} = \frac{22}{3} = 7.33$

Since, $\mu_{Triangle} < \mu_{Square}$, $class(?) == Triangle$.

Am I correct?

You are making a mistake regarding what is given: during training, you don't have a radius $R$. You have the coordinates $\vec{x}$ and the label $y$ for each point:
$$ \vec{x}_1 = [-2;1] \quad \vec{x}_2 = [1;2] \quad \vec{x}_3 = [0;-1] \quad \vec{x}_4 = [1;0] \quad \vec{x}_5 = [1;1] $$
and
$$ y_1 = T \quad y_2=T \quad y_3=T \quad y_4=S \quad y_5=S $$
with that, your "trained" centroids are
$$ \mu_T = \frac 13 [-2 + 1 + 0; 1 + 2 - 1] = \left[-\frac 13, \frac 23\right] $$
$$ \mu_S = \frac 12 [1 + 1; 0 + 1] =\left[1, \frac 12\right]$$
You calculate these centroids before you get any test values.

Then, for testing, for your observed point $\vec{x} = [1,1]$, you calculate the Euclidean distance between the point $\vec{x}$ and the centroids $\mu_T$ and $\mu_S$:
$$ \| \vec{x} - \mu_T \| = \left\| [1;1] - \left[-\frac 13; \frac 23\right] \right\| = \left\| \left[\frac 43; \frac 13\right] \right\| = 1.37 $$
$$ \| \vec{x} - \mu_S \| = \left\| [1;1] - \left[1; \frac 12\right] \right\| = \left\| \left[0; \frac 12 \right] \right\| = 0.5$$
Finally, the term $\hat{y} = \arg\min_{l \in \mathbf{Y}} \|\vec{x}-\mu_l\|$ is used to find the estimated class $\hat{y}$ (the hat symbol is to denote that this is an estimated $y$, not one we knew before.). $\arg\min$, means that you find the minimum value - which is 0.5 in our case, and chose which "argument", i.e. which class, leads to that minimum value. In our case, the class which leads to the minimal distance is $S$, so the result is $\hat{y} = S$, and our test point is a square.


Amazon SageMaker input data? I am exploring Amazon SageMaker as a scalable machine learning solution.

My question is; is it required that the training data first be uploaded in Amazon S3?

Yes, the built-in models pull from S3 and write the computed model back to S3.


Decision Tree used for Calculating Precision, Accuracy, and Recall, class breakdown question I am creating decision trees modeling data that looks like this.

pelvic_radius   degree_spondylolisthesis    class
82.45603817    41.6854736      Abnormal
114.365845     -0.421010392    Normal

When finished, I run my test data through my tree and compare the outputs from the run to the ones I am given. This will allow me to check my accuracy, precision, and recall values.

TP = 0; % True Positives
TN = 0; % True Negatives
FP = 0; % False Positives
FN = 0; % False Negatives

And then once those values are calculated, I can calculate the following.

precision = TP/(TP+FP);
accuracy = (TP+TN)/(TP+TN+FP+FN);
recall = TP/(TP+FN);

However, this can be done in two ways. One considering the 'Normal' class as Positive and one Considering the 'Abnormal' class as positive.

Here is the sudo code to further explain what I mean.

for k=1:length(resultsOfTestSet)
    if(strcmp(resultsOfTestSet{k},'Normal'))
        if (strcmp(testSet{k}, 'Normal'))
            % TRUE POSITIVE
            TP = TP + 1;
        else
            % FALSE POSITIVE
            FP = FP + 1;
        end
    elseif(strcmp(resultsOfTestSet{k},'Abnormal'))
        if(strcmp(testSet{k},'Abnormal'))
            % TRUE NEGATIVE
            TN = TN + 1;
        else
            % FALSE NEGATIVE
            FN = FN + 1;
        end
    end
end

The above case assumes Normal as the 'Positive' resultant class. However by just flipping the compare statements, I can get alternate values.

for k=1:length(resultsOfTestSet)
    if(strcmp(resultsOfTestSet{k},'Abnormal'))
        if (strcmp(testSet{k}, 'Abnormal'))
            % TRUE POSITIVE
            TP = TP + 1;
        else
            % FALSE POSITIVE
            FP = FP + 1;
        end
    elseif(strcmp(resultsOfTestSet{k},'Normal'))
        if(strcmp(testSet{k},'Normal'))
            % TRUE NEGATIVE
            TN = TN + 1;
        else
            % FALSE NEGATIVE
            FN = FN + 1;
        end
    end
end

So after running it for both cases, I get the following values.

For Abnormal being my Positive case

precision = 96.5517
accur = 95
recall = 87.5000

For Normal = Positive case

precision =  94.3662
accur = 95
recall = 98.5294

So how do I calculate the combined result, prec, accu, and recall? Or, am I just missing the point and you just calculate it for one class at a time, like the one you are focusing on.

The reason I am asking is because now lets say I have a set with multiple class outcomes in my decision tree. This is where I realized I have to pick a class to determine as my positive, or just look at classes individually.

Here is a similar set with 3 class possibilities. Again, how do I calculate for the whole data set? Or is an individual class thing? Or do you calculate the individuals and then come together with a total for the whole decision tree.

pelvic_radius   degree_spondylolisthesis    class
82.45603817    41.6854736      Abnormal
114.365845     -0.421010392    Normal
95             25              Perfect

The metrics you calculate are of two types, metrics that depict the entire prediction model you have built like accuracy which will be same in both the cases of your pseudo code. While the others like precision says how precise are you in explaining particular class of interest (accuracy can also be expressed this way in multi-class classification, see the diagram). This score depends on which class you had selected as a positive one. If you put positive class as face of your model, then it is called ppv or precision and npv, if vice versa.

Coming to the multi-class classification, the core definition holds the same. Now the matrix will be n x n(n being number of classes). The sample matrix looks likes this.

The diagonal elements explains the number of 1 class's predicted as 1. Now there are n precision values for each class. Precision for class 1 is how many values were truly predicted as 1 divided by how many are predicted as 1 (FP's included), which is the sum of the first column.

Not but not the least if you adamantly want a precision like metrics for the entire model, you got micro & macro averaging methods, which are helpful in giving a combined metric. This blog post explains it pretty well. Hope this clears something.


Emotion detection from text I am doing a project where I will have to detect emotions from reviews gained by users (through comments). My task is to classify the comments in at least 5 types of emotions that is: happiness, satisfy/unsatisfied, disappointment, sadness, joy. Can I find an already existing model trained for emotion analysis where I can analyze the emotions in each comment. My output needs to give whether the comment is happy or sad/angry etc. I am willing to use the RNN technique. Can someone please help me with my project? Thank you.

There are certain pre-trained models available regarding sentiment analysis or emotion analysis like textblob , flair and deepmoji. Here the link to the in the medium ->
https://medium.com/@b.terryjack/nlp-pre-trained-sentiment-analysis-1eb52a9d742c

But if you have your own set of emotions for which you want to classify, then the best way would be to gather data with labels (the emotions) and train a model from scratch.


Cannot impute 1D array I'm trying to impute 1D array with shape (14599,) with simple imputer with most_frequent strategy but it said it expected 2D array, i already tried reshaping it (-1,1) and (1,-1) but its error ValueError: could not broadcast input array from shape (14599,1) into shape (14599) how can i impute this since reshaping wont solve the problem? i dont understand why it throws error

##libraries
import pandas as pd
import seaborn as sns
import numpy as np
import matplotlib.pyplot as plt
from sklearn.impute import SimpleImputer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder

##codes
plt.close('all')
avo_sales = pd.read_csv('avocados.csv')
avo_sales.rename(columns = {'4046':'small PLU sold',
                            '4225':'large PLU sold',
                            '4770':'xlarge PLU sold'},
                 inplace= True)

avo_sales.columns = avo_sales.columns.str.replace(' ','')

plt.scatter(avo_sales.Date,avo_sales.TotalBags)

x = np.array(avo_sales.drop(['TotalBags','Unnamed:0','year','region','Date'],1))
y = np.array(avo_sales.TotalBags)

X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)

impC = SimpleImputer(strategy='most_frequent')
X_train[:,8] = impC.fit_transform(X_train[:,8].reshape(-1,1)) <-- error here

imp = SimpleImputer(strategy='median')
X_train[:,1:8] = imp.fit_transform(X_train[:,1:8])

le = LabelEncoder()
X_train[:,8] = le.fit_transform(X_train[:,8])

Probably it's problem with Pandas.Series, just change type to numpy array.

X_train[:,8] = impC.fit_transform(np.array(X_train[:,8]).reshape(-1,1))


How to add previous predictions for new predictions in LSTM? I am trying to train a model on a big data sequence like this [0.2 0.1 0.1 ..... 0.4 0.8] . I create X vectors with length 60 for inputs and Y scaler numbers as labels(It means the LSTM reads first 60 numbers as input(one row in X_train) and the 61'th number as the output label(rows in y_train)).

model = Sequential()
model.add(LSTM(units = 50, return_sequences = True , input_shape = (X_train.shape[1], 1)))
model.add(Dense(units = 1))
model.compile(optimizer = 'adam', loss = 'mean_squared_error')
model.fit(X_train, y_train, epochs = 100, batch_size = 32)

And:

X_train.shape = (1000,60,1) ,       y_train.shape = (1000,)

It's OK till here, but the problem is in the prediction part. What I am trying to do is create a (60*1) input vector and use it to predict the next number in my sequence. Then adding the new predicted number to my sequence to predict the next number(second predicted  number) and so on.
For that goal, I created a new model, retrieved the weights from previous model, then fed new_model only with one (60*1) vector to predict the next number. then added the predicted number to the sequence and shift input vectors one number to the right(to use new predicted number for the next number prediction).

new_model = Sequential()
new_model.add(LSTM(units = 50, return_sequences = True , input_shape = (1 , 60 , 1 )))
new_model.add(Dense(1))
old_weights = model.get_weights()
new_model.set_weights(old_weights)
new_model.compile(optimizer = 'adam', loss = 'mean_squared_error')

inputs = []
for i in range(10):
    inputs = dataset_total[len(dataset_total) - 60:].values
    inputs = np.reshape(inputs, (1 , 60, 1))
    predicted = new_model.predict(inputs)
    inputs.append(predicted)

But what do I get is such errors:

ValueError: Input 0 is incompatible with layer lstm_61: expected
ndim=3, found ndim=4

I don't know how to solve this problem!

There is also a similar but without related answer here(for clarification):
How to Predict the future values of time horizon with Keras?

For your problem, Keras is telling you that the input vectorhas 4 dims, which is invalid, because for a LSTM model you need a input vector with (batch_size,num_time_steps,feature_space_dimension). Check the shape of the input vector, you may be adding an extra 1 dimension with the reshape operator

Other thing is the approach your are taking, which may be valid, but usually this problem is tackle with a specific type of models: Sequence-to-Sequence models, using LSTM encoder-decoder architecture.

This models try to predict a sequence

$Y = [x_{(t=1)}, x_{(t=2)},...,x_{(t=t+N)}]$

from a sequence of previous values

$X = [x_{(t=0)}, x_{(t=-1)},...,x_{(t=t-M)}]$

Note that the number of samples in each sequence can be different

In these models, the input sequence $X$  is fed into a LSTM encoder, which captures its dynamics creating a new context feature vector $F$. This new feature vector are then fed to the LSTM decoder as initial state, and then, the prediction $\hat y(t)$ of cell for time $t$ is used as input for the cell for time $t+1$ to generate the predicted value $\hat y(t+1)$

One key difference is, that Seq2seq models should be more efficient than your approach, since it performs the prediction of the N future values with a single run of the model. It could be interesting if you try this approach out and to compare it with yours

You can visit a Keras tutorial for these type of models here


One sided label smoothing in GANs How does one-sided label smoothing make the discriminator more robust by reducing the confidence in correct class?

Well, let's start with what label smoothing is. We replace the 0s and 1s in one hot encoding for example for the MNIST dataset with 10 classes by smoothed values e.g.: 0.1 for 1, 0.9 for 9 etc.

If we replace positive classification targets with $\alpha$ and negative targets with $\beta$ the optimal discriminator $D$ becomes:

$$D(x)=\frac{\alpha p_{data}(x) + \beta p_{model}(x)}{p_{data}(x) + p_{model}(x)}$$.

Where the $p_{data}(x)$ corresponds to the original distribution; at least we think. And the $p_{model}(x)$ corresponds to the distribution that we want to target and $x$ are the input features to the discriminator $D$.

Now, when $\beta \neq 0$, erroneous samples from $p_{model}$ have no incentive to move nearer to the data i.e. it reinforces current generator behaviour. Hence, we smooth only the positive labels to $\alpha$, leaving negative labels set to 0. This behaviour prevents discriminator from giving very large gradient signal to the generator.


Stacking LSTM layers Can someone please tell me the difference between those stacked LSTM layers?

First image is given in this question and second image is given in this article. So far what I learned about stacking LSTM layers was based on the second image. When you build layers of LSTM where output of one layer (which is $h^{1}_{l}, l=..., t-1, t, t+1...$) becomes input of others, it is called stacking. In stacked LSTMs, each LSTM layer outputs a sequence of vectors which will be used as an input to a subsequent LSTM layer. However, in the first image, the input variables are fed again into second layer. Can someone tell me whether there is something wrong about stacking LSTM layers like given in the first image?

You are correct that "stacking LSTMs" means to put layers on top of one-another as in your second image.

The first picture is a "bi-directional LSTM" (BiLSTM), whereby we can analyse a point in a series (e.g. a word in a sentence) from both sides. We care about the context of that point.

The most common example I know of is within NLP. Here we want to know the representation of a word in a gap, how it is found between other words. If we have the entire sentence, we can look at the words before and the words after our word. In this case, we could use a bi-drectional LSTM to process the sequence in the opposite direction, which your first diagram shows.

Let's play a game, and say you need to guess the missing word in this text snippet:

  i need to review an __________ ...

What could it be? "article", "iPad", "aerial image" ?

Here is the solution:

  i need to review an article, ...

It was incredibly hard to get that right - perhaps impossible! Well, maybe not if you have some context with it. How about I give you both sides of that snippet:

  i need to review an ________, for tomorrow's newspaper.

A BiLSTM would be fed the sentence from both sides, thus letting it see some more context to understand each word.

Have a look at this article, which eventually get the bi-directional networks.
Here is a similar question to yours with a few nice answers.

In time-series data, such as device readings from IoT devices or the stock market, using such a bi-directional model wouldn't make sense, as we would be violating the temporaneous flow of information i.e. we cannot use information from the future to help predict the present. That isn't a problem in text analysis, voice-recordings or network analysis on sub-network traffic flow.


A way to Identify tuning parameters and their possible range I am a novice in Machine Learning. But when I started learning, I figure out that all the methods have some tuning parameters and those parameters take a range of possible values. By grid searching, we identify a set of these parameters that optimize some function. But is there any way to find the possible domain of the tuning parameters? This would definitely save my time and the computer's job. In addition, some methods such as xgboost have loads of tuning parameters. Is there any way to know which one to tune and which one to leave as it is. I have been using sklearn python library.

Not a complete answer, but was too long for a comment.

I always first try to see how the default parameters perform. Then from the documentation or some reading, you can see what is each parameter global influence (by influence I mean maybe increasing parameter X means complexifying the model, or parameter Y means increasing the convergence speed towards a solution). Depending on the first result you get, pick up one parameter, the one that seem to have the most influence on the model, and make it vary a bit in the way that make sense from your first results. If things improve on the validation set, keep moving the value this way, if not do the opposite. Often times you get good results without tuning every single parameter.

This is a method by hand, it is not optimal. But as you precise that you are a beginner in machine learning, I believe it is the best way to learn to "feel" what usually impact the performance of an algorithm as Xgboost and what impacts less and that therefore can be overlooked for a primary coarse tuning.

https://xgboost.readthedocs.io/en/latest/parameter.html has some nice pieces of information about what parameter impacts what. Don't hesitate to ask more precise questions about some specific parameters if you need :)


Data pre processing Libraries I'm new to Deep Learning. I only know about scikit learn when I have to pre process data for a neural network. are there any other good libraries for that ?

Pre-processing is 50% of the entire pipeline. With better data you build better machine learning/deep learning models. But unfortunately, cleaning data is something that needs time and experience. You need to visualise it and make your hands dirty. The aim is to remove noise, garbage and outliers, in short.

There are some really good data visualisation libraries in Python such as :

MatplotLib
Seaborn
GGPlot
Bokeh
PyGal
Plotly
and more.

Since, you are comfortable with Scikit learn, as you mentioned in the question, I would suggest you to look up the preprocessing modules in scikit learn, it contains several APIs such as feature extraction, normalization, feature scaling, mean removal, variance scaling, standardization, etc.

  But, it is always better to understand the data, visualise it like a
  story and clean it manually,
  slowly, instead of passing it through predefined frameworks or
  pipelines.

These videos :

Why You Need Data-Preprocessing
Data-preprocessing tasks

can be helpful. There are many more available easily. Good luck !

Also, do upvote my answer, if it has helped you. It encourages the community to help each other.


What are practical differences between kernel k-means and spectral clustering? I've been lately wondering about kernel k-means and spectral clustering algorithms and their differences.

I know that spectral clustering is a more broad term and different settings can affect the way it works, but one popular variant is using K-means clustering on the spectral embedding of affinity matrix.
On the other hand kernel K-means applies K-means clustering directly to the affinity matrix. Therefore one immediate, theoretical difference is it omits spectral embedding step, i.e. it doesn't look for the lower-dimensional representation of data with eigenvectors.

I think it could be beneficial in high-dimensional settings (with many observations to cluster), but does it provide any boost with small sample size, for example from 10 to 20 observations?

What are other practical implications of using either of these algorithms versus another (e.g. which one is more sensitive to any changes in affinity etc.)?

The differences are indeed not too large. There is a paper called
Kernel k-means, Spectral Clustering and Normalized Cuts by
Inderjit S. Dhillon, Yuqiang Guan, Brian Kulis from KDD 2004
that is discussing that relationship.
The authors show that spectral clustering of normalized cuts is a special case of weighted kernel k-means. The reason is that the "graph cut problem and weighted kernel k-means problem can both be written as trace maximization problems".
They write that

this has important implications: a) eigenvector-based algorithms, which can be computationally prohibitive, are not essential for minimizing normalized cuts, b) various techniques, such as local search and acceleration schemes, maybe used to improve the quality as well as speed of kernel k-means.

Further they combine ideas from both to improve the total results and

show that using eigenvectors to initialize kernel k-means gives better initial and final objective function values and better clustering results.

Regarding robustness in spectral clustering you might want to look at Robust Spectral Clustering for Noisy Data by Aleksandar Bojchevski, Yves Matkovic, Stephan Günnemann, published at the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2017: https://www.kdd.org/kdd2017/papers/view/robust-spectral-clustering-for-noisy-data

Inderjit S. Dhillon, Yuqiang Guan, Brian Kulis, Kernel k-means: spectral clustering and normalized cuts, KDD '04: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, August 2004, Pages 551–556, https://doi.org/10.1145/1014052.1014118


Orange3: Is it possible to get the total number of rows for a given column from inside 'Feature Constructor'? I'm using the Orange3 gui and have been asked to generate basic plots where values relative to a "mean" or  "Percentile Rank" are colorized.

I looked in Feature Constructor, but as far as I could tell it lacks a way to calculate the total number of rows, or instances of incoming data.

I've looked elsewhere for widgets that might help and stumbled upon Box Plot, however, if one attaches a Data Table to the output of Box Plot, we can see that no new columns contain the values of the mean or standard deviation are generated.

Is there some widget that has a function that can output these values, or the total number of rows selected / in the data set so that I can write my own function?

Python Script widget looks like it might fit your needs, where, according to the docs, a code like this should give you the number of rows in a data set:

n_rows = len(in_data)


Predicting three related scores with one joint model I have multiple features and I want to predict three outcome scores.

Features:

Length in cm smallest is 40cm biggest is 209cm
Kilo: 39 till 302
Age: 19 till 111
Gender: Male, female, transgender
Diagnoses: numbers of different diseases
Medicines: numbers of different medicines
Urine level: 0 till 5

Scores:

Scale of Happy 1 till 7
Scale of depression 1 till 7
Scale of health 1 till 7

I know I can use supervised learning and create models where I predict the scores individually. I already pre-trained three different models.

Do you have some feedback?

Happiness, depression and health are  related. Are there algorithms that can deal with this? Based on historical data (the features and predictions) and the trained model I want to predict the scores for new patients based on their features and historical scores.

Do you have a labeled set? Any algorithm can do it in a supervised manner, as long as you have enough labeled data to train it on.

A tree-based model (Random Forests Classifier for instance) could do really well on this task, especially since you have categorical data. All you need to do is encode the data as numbers (for instance 0: male, 1: female, 2: transgender) and feed it into your model (applying random splits and cross-validation and all that).

In case you don't, then the best you can do without collecting new data is to find clusters of related patients in your dataset. The same rules for encoding variables apply, although you may have to pay some attention to how you scale your variables since clustering algorithms are sensitive to relative scaling of features. This happens because most clustering algorithms cluster data using distance metrics which lose meaning when you are dealing with categories. See this for more info on clustering categorical data:


numpy array reshape question Let's say that I have image data with shape $(32, 32, 3)$ and $50000$

If I would like to reshape it to $(50000, 3, 32, 32)$ what should I do?

I tried np.transpose(0, 3, 1, 2) but it failed.

If I would like to print the number $3$ from $(50000, 3, 32, 32)$ what should I do?

If image is a numpy object:

image = image.reshape((50000,3,32,32))

and then print:

print(image[:,3,:,:])


GlobalAveragePooling2D in inception v3 example I'm a complete begginer at Keras. In the Inception v3 example at https://keras.io/applications/

# create the base pre-trained model
base_model = InceptionV3(weights='imagenet', include_top=False)

# add a global spatial average pooling layer
x = base_model.output
x = GlobalAveragePooling2D()(x)
# let's add a fully-connected layer
x = Dense(1024, activation='relu')(x)
# and a logistic layer -- let's say we have 200 classes
predictions = Dense(200, activation='softmax')(x)

As at first step it adds a GlobalAveragePooling2D layer, which is described as:

  Global average pooling operation for spatial data.

What GlobalAveragePooling2D does and why the example uses it instead of something like Flatten? Which information is averaged?

Prior to GAP, one would flatten your tensor and then add a few fully connected layers in your model.  The problem is that a bunch of parameters in your model end up being attributed to the dense layers and could potentially lead to overfitting.  A natural solution was to add dropout to help regulate that.

However a few years ago, the idea of having a Global Average Pooling came into play.  GAP can be viewed as alternative to the whole flatten FC Dropout paradigm.  GAP helps prevent overfitting by doing an extreme form of reduction.  Given a H X W X D tensor, GAP will average the H X W features into a single number and reduce the tensor into a 1 X 1 X D tensor.

The original paper simply applied GAP and then a softmax.  However, it's now common to have GAP followed by a FC layer.


Clustering a variable based on another variable or set of variables df11[['COMPONENT_ID','FIRMWARE','SERIAL','CRP0_VDDN']].head()

Consider I have these four columns to analyse. I want to form say 3-5 clusters of COMPONENT_IDs with similar characters. I want this to happen based on the remaining features or just CRPO_VDNN in relation with COMPONENT_IDs. How can I do this ?

First of all, clustering is used only for numeric values, especially for continuous values. What you are trying to do here is to cluster a categorical variable, that too an ID column. I'm unsure about the goal but this is not a good technique to cluster values.

That being said, I'm not sure how many unique entries you have in the ID column. You have to convert it into categorical numbers before clustering. Then take only the ID column and the CRPO_VDNN column and use KNN to cluster. You can set the number of clusters in the KNN too.


SelectFromModel vs RFE - huge difference in model performance Note: I have already looked at Difference between RFE and SelectFromModel in Scikit-Learn post and my query is differnt from that post

Expectation:
SelectFromModel and RFE have similar/comparable performance in the model built using their recommendations.
Doubt: Is there any known usecase where RFE will fare better? As a student of data science (just starting learning) its a weird observation for me
Code:
# RecursiveFeatureElimination_ExtraTreesClassifier

from sklearn.feature_selection import RFE
from sklearn.ensemble import ExtraTreesClassifier

rfe_selector = RFE(estimator=ExtraTreesClassifier(), n_features_to_select=20, step=10)
rfe_selector.fit(x_raw, y_raw)
[x[0] for x in pandas.Series(rfe_selector.support_, index=x_raw.columns.values).items() if x[1]]

# returns
['loan_amnt','funded_amnt','funded_amnt_inv','term','int_rate','installment','grade','sub_grade','dti','initial_list_status','out_prncp','out_prncp_inv','total_pymnt','total_pymnt_inv','total_rec_prncp','total_rec_int','recoveries','collection_recovery_fee','last_pymnt_amnt','next_pymnt_d']

# SelectFromModel_ExtraTreesClassifier
from sklearn.ensemble import ExtraTreesClassifier
from sklearn.feature_selection import SelectFromModel

selector = SelectFromModel(ExtraTreesClassifier(n_estimators=100), max_features=20)
selector.fit(x_raw, y_raw)

[x[0] for x in pandas.Series(selector.get_support(), index=x_raw.columns.values).items() if x[1]]

# prints
['loan_amnt','funded_amnt','funded_amnt_inv','term','installment','out_prncp','out_prncp_inv','total_pymnt','total_pymnt_inv','total_rec_prncp','total_rec_int','recoveries','collection_recovery_fee','last_pymnt_d','last_pymnt_amnt','next_pymnt_d']

Code for Model train and test
# internal code to select what variables I want
x_train, y_train, x_test, y_test = get_train_test(var_set_type=4)

model = ExtraTreesClassifier()
model.fit(x_train, y_train)

# then just print the confusion matrix

ExtraTreesClassifier Model from SelectFromModel variables

ExtraTreesClassifier Model from RFE variables

My confusion matrix is powered by this Open Source project: DTrimarchi10
/
confusion_matrix

To start with, let me repeat here what I have already answered to another OP wondering if Recursive feature selection may not yield higher performance?:

There is simply no guarantee that any kind of feature selection
(backward, forward, recursive - you name it) will actually lead to
better performance in general. None at all. Such tools are there for
convenience only - they may work, or they may not. Best guide and
ultimate judge is always the experiment.
Apart from some very specific cases in linear or logistic regression,
most notably the Lasso (which, no coincidence, actually comes from
statistics), or somewhat extreme cases with too many features (aka
The curse of dimensionality), even when it works (or doesn't), there is not necessarily much to explain as to why (or why not).

Having clarified that, let's see your case in more detail.

None of the feature selection procedures here takes into account the model performance; in classification settings, the sole criterion by which features are deemed as "important" or not is the mean decrease in the Gini impurity achieved by splitting in the respective feature; for some background, see the following threads (although they are about Random Forests, the rationale is identical):

How is the 'feature_importance_' value calculated in sklearn random forest regressor?
Relative importance of a set of predictors in a random forests classification in R
How are feature_importances in RandomForestClassifier determined?

Although it is often implicitly assumed that a reduction of the features using this importance as a criterion may lead to gains in the performance metric, this is by no means certain and far from straightforward (I am actually repeating my intro here).
Given that, it would seem that the actual question here should be why the two methods end up selecting different features, for which the thread you have linked yourself, Difference between RFE and SelectFromModel in Scikit-Learn, is arguably relevant. In any case, the expectation that they should offer similar results is arguably not well-founded; the relative feature importance changes when features are removed from the model (RFE case), hence it cannot be directly compared with the SelectFromModel approach (use all features, remove those with importance below a threshold). Similarly, the question "Is there any known use case where RFE will fare better?" is ill-posed - you have not shown that RFE is consistently inferior, and the results of a single experiment with a single dataset and a single parameter setting (such as the no. of required features and the threshold involved in SelectFromModel) should not be generalized light-heartedly.
In any case, selecting features based on their Gini importance (Mean Decrease in Impurity - MDI) has started falling out of fashion, mainly because the calculated importance is spurious in cases of categorical features with high cardinality (see the academic paper The revival of the Gini importance?); in fact, there is already a relevant warning in the scikit-learn classifiers inorporating this "classical" feature_importances attribute:

Warning: impurity-based feature importances can be misleading for high cardinality features (many unique values). See sklearn.inspection.permutation_importance as an alternative.

See the scikit-learn vignette Permutation Importance vs Random Forest Feature Importance (MDI) for a concrete example.

Irrelevant to the exact question, and if the classes in your dataset are imbalanced (as they seem to be), you could (and should) inform your models about this, using the class_weight argument of ExtraTreesClassifier (docs), i.e. change the model in both cases to
ExtraTreesClassifier(class_weight='balanced')

(h/t to Ben Reiniger for a constructive comment that helped to improve the answer)


Feature importance over a subset of instance space instead of an entire instance space I'm really curious if anyone has faced this problem before, or is it even widely studied at all.

Imagine we have a feature that isn't important (based on many widely available and textbook feature selection methods) over the entire instance space of training data. However, if we try to  divide the instance space into certain subsets, can the feature become important over that subset, and are there any methods available for arbitrary datasets?

When I say 'dividing the instance space', I mean for example if I have 3 features $X_0, X_1, X_2$ with a binary label $Y$, a subset of instance space will be a region where, for example, when $X_0 >5, X_1>5$. In particular, could it be possible that $X_2$ isn't significant over the entire instance space, but is significant over the restricted space mentioned above. If it's possible (my intuition says so), are there any methods in current literature that allow me to find such a space (if it exists)?

If anyone has any links/papers/resources which can direct me to a similar study, I would gladly accept it as well.

This is a common occurrence in practice. For example, in the bioinformatics space, a protein can change a phenotype significantly over a very limited range, but outside that range has little to no effect.

If you want to where the 'range of highest effect' falls I recommend partial dependence plots which are means of ICE plots. It offers an advantage over decision trees in that you won't overfit as easily to your training dataset and can be applied to high dimensional data. Here's an example of the plot. Read more here. The following packages have some implementation of the above:

mlinsights - https://ml-insights.readthedocs.io/en/latest/
https://media.readthedocs.org/pdf/pdpbox/latest/pdpbox.pdf

Here's a decent example from the above:

If you only want to know if something is important, not where it is important. You can use something called permutation importance or Mean Decrease in Accuracy outlined very briefly in a seminal random forest paper, page 20-21: https://www.stat.berkeley.edu/~breiman/randomforest2001.pdf

Look into the eli5 package for easy calculation.


Model inclines because of imbalanced data In life, some events are rare and most cases are normal. So I am wondering, to detect the rare cases, shall we use an imbalanced dataset with more historical rare cases?
Taking the German Credit Data, as an example. It contains data on 20 variables and the classification whether an applicant is considered a Good or a Bad credit risk for 1000 loan applicants. 70% are Good. 30% Bad.
With this original data set, I assume the model would incline to better recognize the normal cases (because more normal cases in the data)
If a balanced dataset is used, i.e. the number of good credits equals to that of bad credits, a final model will be good at predicting both ‘Good’ and ‘Bad’.
But if we want to use machine learning to recognize the rare events, e.g. in this case, the bad credit customers. Shall we use an imbalanced dataset (for example 70% Bad credits, 30% Good in total 1000 records), that contains much more bad credit customers, than good ones, so the final model is good at recognizing the bad customers? Or a balanced dataset is always necessary (also the only right way).
Can anybody please shed some light on this?
Thank you.
Link to data: https://online.stat.psu.edu/stat857/sites/onlinecourses.science.psu.edu.stat857/files/german_credit/index.csv

Impact of imbalanced datasets
First I would say that imbalanced dataset impact depends on the type of model you are using.
For instance:

Gaussian Naive Bayes should not be that much impacted if you have a certain amount of data for each class that is enough to approximate your gaussian distributions. (and that your data are normally distributed)
Neural Networks learning is using the error of your predictions to update its model, so having an imbalanced dataset would lead to imbalanced learning (for example 70% of the weights/biais have been updated according to class 'Good'). You don't want this happening a priori.

How to deal with imbalanced dataset ?
There might be other approaches, but you can do at least:

Use an algorithm that is not much impacted by imbalanced datasets

Some algorithms have a class_weight parameter. You can use it to penalize more the minority class during the learning process so the model is forced to pay more attention to the minority class observations. See this post for more details: How does the class_weight parameter in scikit-learn work?

Resample your dataset so it becomes balanced. That could be through undersampling the majority class or oversampling the minority one.

Further reading
How to Deal with Imbalanced Data
8 Tactics to Combat Imbalanced Classes in Your Machine Learning Dataset
Hope it helps.


When is centering and scaling needed before doing hierarchical clustering? I am working on a clustering project where we have collected protein data from over 100 patients samples. This data is normalized and log transformed. The goal is to cluster samples based upon their similarities, I am using hierarchal clustering and trying out combinations of distance metrics and clustering algorithms. (We haven't made a decision on distance method or clustering algorithms) My question is related to the centering and scaling, Is it absolutely necessary to both scale and center the data?, even in scenarios where all the data is coming from the same platform and with same units of measurement.

Appreciate your input on this one.

Thanks

My question is related to the centering and scaling, Is it absolutely necessary to both scale and center the data?, even in scenarios where all the data is coming from the same platform and with same units of measurement.

It depends on the type of data you have. For some types of well defined data, there may be no need to scale and center. A good example is geolocation data (longitudes and latitudes). If you were seeking to cluster towns, you wouldn't need to scale and center their locations.

For data that is of different physical measurements or units, its probably a good idea to scale and center. For example, when clustering vehicles, the data may contain attributes such as number of wheels, number of doors, miles per gallon, horsepower etc. In this case it may be a better idea to scale and center since you are unsure of the relationship between each attribute.

The intuition behind that is that since many clustering algorithms require some definition of distance, if you do not scale and center your data, you may give attributes which have larger magnitudes more importance.

In the context of your problem, I would scale and center the data if it contains attributes like patient height, weight, age etc.

This answer on a similar question has more.


Number of parameters keras dense layer with a 2D input I am using 2D data in a classification problem using keras.

So I am defining a keras model as following:

in_ = Input((5, 10))
out = Dense(100, activation='relu', name = 'dense_1')(in_)

model = Model(in_, out)
model.compile(loss='categorical_crossentropy', optimizer='adam')
model.summary()

which returns a compiled model with the following parameters:

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_9 (InputLayer)         (None, 5, 10)             0
_________________________________________________________________
dense_1 (Dense)              (None, 5, 100)            1100
=================================================================
Total params: 1,100
Trainable params: 1,100
Non-trainable params: 0
_________________________________________________________________

What I don't understand is why the dense_1 layer has only 1100 parameters and not 5100 parameters.
What I was expecting is that the Dense Layer is going to connect to all the inputs 50 (5*10=50 inputs) giving a number of parameters of 5100 (100*50+100=5100, weights + biases). So apparently the Dense Layer only connects to the last dimension of the input? What happens in the other dimension?

If I flatten the input layer I obtain my expected number of parameters:

in_ = Input((5,10))
x = Flatten()(in_)
out = Dense(100, activation='relu', name = 'dense_1')(x)

model = Model(in_, out)
model.compile(loss='categorical_crossentropy', optimizer='adam')
model.summary()

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_13 (InputLayer)        (None, 5, 10)             0
_________________________________________________________________
flatten_6 (Flatten)          (None, 50)                0
_________________________________________________________________
dense_1 (Dense)              (None, 100)               5100
=================================================================
Total params: 5,100
Trainable params: 5,100
Non-trainable params: 0
_________________________________________________________________

So what is going on with a Dense Layer when the previous layer has more than one dimension? What happens with the dimensions and the dot products and biases? Why does the number of parameters changes?

Yes, it takes only to the last dimension, accordingly to the source code (comments are mine):
https://github.com/keras-team/keras/blob/88af7d0c97497b5c3a198ee9416b2accfbc72c36/keras/layers/core.py#L880

def build(self, input_shape):
    assert len(input_shape) >= 2
    input_dim = input_shape[-1]  # uses last dimension


How to predict when and why of hospitalization? I have an EHR data source which has info on
a) Patient visit records (Inpatient, outpatient, Emergency etc) and why did he visit hospital (diagnosis codes attached to each visit)
b) Patients drugs data
c) Patients lab test data
d) Patient diagnosis history
e) Patient demographics data
If my objective had just been to predict when will we visit hospital next, I can use Time to event analysis or survival analysis etc.
But, I would also like to know why will he visit hospital next or what might be the reason of his next hospital visit?
While predicting "when" seems to be doable, can experts here guide me on how can I find why/what will be the reason for his next hospital visit?

Why is potentially hidden in your feature explanations.
Given that your Features give some indication on why, like Patient demographics data, or some other Features that you can include here, you can use them to answer the why.
Like this using shapley and or eli5 you can integrate it with your Standard predictor classes and for the explanations with user friendly API.
BUT i have to prefice that even tough it might be hidden it must not be at the same time, since the predictiveness of time might have Nothing to do real why you are Looking for.


why always predict same class for all input import numpy as np
from sklearn import datasets as ds

iris = ds.load_iris()
x = iris.data
y1 = iris.target

x = x / x.max()
y1 = np.matrix(y1)
np.random.seed(1)
y = np.zeros((y1.size, y1.max() + 1))
y[np.arange(y1.size), y1] = 1

class NeuralNetwork(object):

    def __init__(self):
        self.inputSize = 4
        self.outputSize = 3
        self.hiddenSize = 5

        self.W1 = np.random.randn(self.inputSize, self.hiddenSize) * 0.01
        self.W2 = np.random.randn(self.hiddenSize, self.outputSize) * 0.01
        self.b1 = np.random.randn(1, self.hiddenSize)
        self.b2 = np.random.randn(1, self.outputSize)

    def forward(self, x):
        self.z = np.dot(x, self.W1) + self.b1
        self.z2 = self.sigmoid(self.z)
        self.z3 = np.dot(self.z2, self.W2) + self.b2
        o = self.sigmoid(self.z3)
        return o

    def sigmoid(self, s):

        return 1 / (1 + np.exp(-s))

    def sigmoidPrime(self, s):

        return s * (1 - s)

    def backward(self, x, y, o):
        l = 0.2
        self.dz2 = y - o
        self.dw2 = (1 / 150 * self.dz2.T.dot(self.z)).T

        self.db2 = 1 / 150 * np.sum(self.dz2, axis=0).reshape(1, 3)

        self.dz1 = 1 / 150 * self.W2.dot(self.dz2.T).T * self.sigmoidPrime(self.z)

        self.dw1 = (1 / 150 * self.dz1.T.dot(x)).T

        self.db1 = 1 / 150 * np.sum(self.dz1, axis=0).reshape(1, 5)

        self.W1 = self.W1 - l * self.dw1

        self.W2 = self.W2 - l * self.dw2

        self.b1 = self.b1 - l * self.db1

        self.b2 = self.b2 - l * self.db2

    def train(self, x, y):
        o = self.forward(x)
        self.backward(x, y, o)

when I run this code it predicts always same class
but when I change backward function to this one:

def backward(self, x, y, o):
    self.o_error = y - o
    self.o_delta = self.o_error * self.sigmoidPrime(o)
    self.z2_error = self.o_delta.dot(self.W2.T)
    self.z2_delta = self.z2_error * self.sigmoidPrime(self.z2)
    self.W1 += x.T.dot(self.z2_delta)
    self.W2 += self.z2.T.dot(self.o_delta)

It predicts correctly. Why first function doesn't work correctly (backward)?

I solve this problem with adjusting learning rate and adding regularization and some optimization


Random Forest VS LightGBM Random Forest VS LightGBM

Can somebody explain in-detailed differences between Random Forest and LightGBM? And how the algorithms work under the hood?

As per my understanding from the documentation:

LightGBM and RF differ in the way the trees are built: the order and the way the results are combined. It has been shown that GBM performs better than RF if parameters tuned carefully.

Random Forest: RFs train each tree independently, using a random sample of the data. This randomness helps to make the model more robust than a single decision tree, and less likely to overfit on the training data

My questions are

When would one use Random Forests over Gradient Boosted Machines?
What are the advantages/disadvantages of using Gradient Boosting over Random Forests?

RandomForest advantage compared to newer GBM models is that it is easy to tune and robust to parameter changes. It is robust for most use cases although the peak performance might not be as good as a properly-tuned GBM. Another advantage is that you do not need to care a lot about parameter. You can compare the number of parameter for randomforest model and lightgbm from its documentation. In sklearn documentation the number of parameter might seem a lot, but actually the only parameter you need to care about(ordered by importance) are max_depth, n_estimators, and class_weight, and the other parameters are better to be left as is. So for me, I would most likely use random forest to make baseline model.

GBM is often shown to perform better especially when you comparing with random forest.
Especially when comparing it with LightGBM. A properly-tuned LightGBM will most likely win in terms of performance and speed compared with random forest.

GBM advantages :

More developed. A lot of new features are developed for modern GBM model (xgboost, lightgbm, catboost) which affect its performance, speed, and scalability.

GBM disadvantages :

Number of parameters to tune
Tendency to overfit easily

Please bear in mind that increasing the number of estimators for random forest and gbm implies different behaviour. High value of n_estimators for random forest will affect it robustness, where as for GBM model will improve the model fit with your training data (which if too high will cause your model to overfit).


sklearn predict: IndexingError: ('Too many indexers', 'occurred at index ') The goal of what I'm trying to accomplish here is to have the output contain all of the use_cols but the model only be built to calculate on categorical_features. The output will then be used to predict and compare the prediction 'REVIEW_ACTION' to the actual 'REVIEWER_ACTION'. Ignoring for the moment why this is A BAD THING TO DO, can we focus on how to achieve this?

use_cols = ['FIRST_NAME', 'LAST_NAME', 'PERSON_STATUS', 'DIVISION_NAME',
            'PERSON_TYPE', 'JOB_CHANGE', 'JOB_TRANSFER', 'IDENTIFY_DATE',
            'SSO', 'USER_ID', 'ASSET_ID', 'ROLE', 'HPA',
            'LAST_LOGIN_DATE', 'ASSET_NAME', 'AUDIT_ID',
            'REVIEWER_ACTION', 'USER ID', 'LABELED_ROLE', 'REVIEW_ACTION']
prediction_col = ['REVIEW_ACTION']

categorical_features = ['DIVISION_NAME', 'PERSON_TYPE', 'JOB_CHANGE', 'JOB_TRANSFER',
                       'SSO', 'USER_ID', 'ASSET_ID', 'ROLE', 'HPA', 'LAST_LOGIN_DATE',
                       'USER ID', 'LABELED_ROLE']

categorical_transformer = Pipeline(steps=[
    ('si', SimpleImputer(strategy='constant', fill_value='missing')),
    ('ohe', OneHotEncoder(handle_unknown='ignore'))],remainder='passthrough')

preprocessor = ColumnTransformer(
    transformers=[
        ('cat', categorical_transformer, categorical_features)])

rf = Pipeline(steps=[('preprocessor', preprocessor),
                      ('rfc', RandomForestClassifier(n_estimators=100))
                     ])

I then fit the data and try to run the prediction

df['rf_prediction'] = df[categorical_features].apply(rf.predict)

and get the error:

IndexingError: ('Too many indexers', 'occurred at index DIVISION_NAME')

This has something to do with the columns being 'passthrough''d but I'm not sure how to resolve it. I don't want to process some of these columns but want them in the results when I write the file so that I can validate the results.

Somewhere in my debugging I took a step back and am seeing different errors when I call fit. If X = data I see:

ValueError: could not convert string to float: 'Herve'

Note that "Herve" is a field that is passed through (remainder='passthrough') so the model shouldn't be seeing it -- or so I've been led to believe.

Removing remainder='passthrough' resolved the problem. I just tacked the prediction onto 'X' and got what I needed since sklearn v20 can deal with pandas dataframes.


How to plot data points and not centroids using sklearn k-means? I have issues finding a way to plot data points colored by cluster with k-means.

I have a very long list of strings.I managed to plot the centroids but not the data points;

import matplotlib.pyplot as plt
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans

vectorizer = TfidfVectorizer(stop_words='english')
X = vectorizer.fit_transform(cleans)

true_k = 5
model = KMeans(n_clusters=true_k, init='k-means++', max_iter=100, n_init=1)
model.fit(X)
print("Top terms per cluster:")
order_centroids = model.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(true_k):
    print("Cluster %d:" % i),
    for ind in order_centroids[i, :10]:
        print(' %s' % terms[ind]),
    print()

print("\n")

plt.scatter(order_centroids[:, 0],order_centroids[:, 1], marker="x", s=150, linewidths=5, zorder=10)
plt.show()

I expect the output to be the data points colored by cluster, and not the centroids

Scatter plots work well for 2 dimensional, continuous data. Like the toy examples you see everywhere.

But the data produced by tfidfvectorizer is high-dimensional and sparse.

It is to be expected that almost all points will have a 0 value in any two dimensions you plot. One does not simply plot test as 2d coordinates in a meaningful way. And regular k-means doesn't work that well either. The means just move close together, and all correspond to average stopword vectors - not very useful.


Feature scaling data with missing values am interested in doing some feature scaling to try and tease out something from my data (box plots by outcome show that the 25/50/75 quantiles are very similar; certain variables have more "outliers" than other by class. The issue I have however is that there is a lot of missing data. I would like to replace these NAs (they are numerical) with something like -9999. Should I first scale my data and then assign -9999 to NAs or first assign then scale?

Well -9999 can matter more or less depending on the variability within your data (is -9999 an extreme outlier value, or a moderate value somewhat close to the mean/median>)

Depends a lot on your data.  A value like that can skew it a lot, and can affect modeling quite a bit depending on the technique you use.  Are the NA's biased for a class?  Consider decision trees that classify on optimal splits.  They would detect a bias (e.g. if 90% of your NA's a certain class) and split at -9999 value to manifest that bias.

also check out Pareto scaling, which emphasizes small to medium changes in your data.  Might be along the lines you are looking for.


How to work with input which is a combination of metadata+ vectorized text data + image pixel data to build a Regression Model (predict views)? There are 4 datasets (all in csv format), each has a uniqueID column by which each record can be identified. Image and text datasets are dense datasets.(need to be converted to ndarray).
Can someone suggest how to use all these 4 datasets for building a regression model?
This is how the datasets look,
Metadata having some input features and target variable(views)
uniqueID    ad_blocked embed  duration language hour views
     1        True     True    68        3      10   244
     2        False    True    90        1      15   63
     3        True     False   195       3      7    350

Vectorized title data - one entire row represents a title
uniqueID  title_1   title_2    title_3
     1   -0.977637  -0.543310  0.079403
     2    0.041873   0.644655  -0.406487
     3    0.503560  -0.085412  0.841144

Vectorized descriptions data - one entire row represents a description
 uniqueID  title_1   title_2    title_3
     1   -0.052256  -0.016036  0.079403
     2    0.000106  0.356706  -0.025788
     3    0.015774 -0.085412   0.712229

Thumbnail pixel data - one entire row represents an image
uniqueID  image_1    image_2   image_3
     1   -0.484456  -0.543310  0.032915
     2    0.666147  0.644655  -0.005733
     3    0.035018  -0.011111  0.841144

If we avoid a Neural Network,

You should reduce the dimensionality of images e.g. PCA [ Assuming no translational variance among images]
Can try the same for the other two datasets with 50 features
Then try the ML models

If that doesn't work, we may try CNN for the image data.
In that scenario, we will need a mixed architecture of simple Neural Network and CNN.

Check this Paper
Blog on the paper


Scikit learn train test split without mixing participants in trails I have dataset of trails. There are 90 participants that appear in 90 trails each. So 450 trails in total.

I'm looking to split my dataset so participants doesn't get mixed over training and test set. I want all trails of 25 participants in my training set and trails of the remaining 25 in the test set.

The reason is I want to test on totally unseen data to access overfitting. Is there a way of achieving this within scikit-learn?

You can use one of scikit-learn's options for grouped data. In particular, GroupKFold should do the trick: something like

from sklearn.model_selection import GroupKFold
group_kfold = GroupKFold(n_splits=2)
group_kfold.get_n_splits(X, y, groups)

where groups is an array of group indices.


Using R to organize/rearrange CSV - group by multiple columns? I have a CSV that I need to clean up / organize in a usable way using R. I need to group by the property ID and then want to take all the unique years for the defor year column and make each year into a sperate column with the amount of deforestation for that year. My data frame / CSV looks like this:
Prop_ID deforYear   deforHA
1           2010         15
1           2011         0
1           2012         10
2           2010         35
2           2011         45
2           2012         0

and I want the output to look like this:
Prop_ID defor_2010  defor_2011 defor_2012
1               15           0         10
2               35          45          0

I'm assuming I use the group_by function in dplyr but I can't seem to figure it out.

This very simple using the pivot_wider function from tidyr:
library(tidyr)

df <- data.frame(
  Prop_ID = c(1, 1, 1, 2, 2, 2),
  deforYear = c(2010, 2011, 2012, 2010, 2011, 2012),
  deforHA = c(15, 0, 10, 35, 45, 0)
)

df %>%
    pivot_wider(names_from=deforYear, values_from=deforHA, names_prefix="deforYear_")

# result
  Prop_ID deforYear_2010 deforYear_2011 deforYear_2012
1       1             15              0             10
2       2             35             45              0


Would Topic Modelling be classified as NLP or NLU? I recently started my journey into the world of NLP, it's been one heck of a ride. I'm currently trying to understand whether topic modelling would be considered as NLP or NLU.

Initially I would assume that topic modelling would be classified as NLP. However, if we use word embeddings for topic modelling wouldn't it then be classified as NLU, as we have deeper understanding of how the words relate to each other in vector space?

Maybe I'm having trouble formulating the inherent difference between NLP and NLU, when do we draw the line between the two?

Your insight regarding this matter would be highly appreciated.

Maybe I'm having trouble formulating the inherent difference between NLP and NLU, when do we draw the line between the two?

There is a confusion here: NLP is the whole domain of AI which deals with natural language. It includes virtually any task related to processing language data (usually mostly written data, but that's not the point). Topic modeling is one of these tasks.

NLU is the problem of Natural Language Understanding, which is usually considered as one of the main goals of NLP. If anything, NLU is a problem that NLP tries to solve, i.e. a sub-topic in the large area of NLP.

Also notice that using words embeddings can improve things, but it doesn't solve all the difficulties related to semantics, far from it.

[edit] The scope of NLU is not strictly defined: in the broadest possible definition, it would include anything vaguely related to extracting meaning from text, and in this very generous sense topic modeling would have a connection to it with or without embeddings (and so would a lot of other NLP tasks). Wikipedia says:

  The umbrella term "natural-language understanding" can be applied to a diverse set of computer applications, ranging from small, relatively simple tasks such as short commands issued to robots, to highly complex endeavors such as the full comprehension of newspaper articles or poetry passages. Many real world applications fall between the two extremes, for instance text classification for the automatic analysis of emails and their routing to a suitable department in a corporation does not require in depth understanding of the text.

But the most commonly accepted definition of NLU is stricter, it would only consider tasks which directly involve the interpretation of text in a quite complex setting. The typical example is the "virtual assistant" such as Amazon Alexa, OK Google, Apple's Siri. In this sense topic modeling is simply a completely different task, no matter the "degree of understanding".


Understanding word2vec vectors representation I'm trying to obtain the word2vec representation of few words using gensim.

At present, this is the model that I have:

model = Word2Vec(desc, size=1, seed=7, window=1, iter=1000)

And the current output is

Ansible automate 0.37919244170188904
Docker container -0.05218552052974701
Ansible automation 0.39010459184646606
Automate Ansible 0.1009570062160492
Contain Docker 0.41532352566719055
Dock contain -0.38013115525245667
Dock container -0.17561538517475128

I'm wondering why Contain Docker is closer to Ansible automate and Ansible automation than it is from the other terms which contain the word Docker, Container etc.

How can I train the word2vec model to get this sorted out?

First of all, consider this: Word2Vec generates this representation by looking at the context (neighboring words, here with a window of 1) of each word or n-gram (e.g. pair of words such as Contain Docker).

Your are using a uni-dimensional representation of those pairs of words. Even in a particular training text such as yours which seems to be revolving around a unique subject (i.e. docker), a unique value as representation means a unique dimension in the "semantic space". This yields really poor representation power, and Word2Vec models are usually trained with more than 300 hundreds dimension to fully capture the semantic of words. And usually, more is better (but not always).

I hence suggest that you increase the size of your vectors (try different values, such as 100 and 200).

To see if the new representation makes sense, I suggest to use Multidimensional Scaling, which enables to project N-dimensional data on a plan. See for instance the answers to this question

What if you still want to use a uni-dimensional encoding ?

For a training text about emotions, it could mean a positive vs. negative dimension (example representations: happy = 0.85, sad = -0.94). In your case, I have no idea and suggest to plot a selection of words along this axis, to see if there are any obvious pattern for having high value or low value.


Which is faster: PostgreSQL vs MongoDB on large JSON datasets? I have a large dataset with 9m JSON objects at ~300 bytes each. They are posts from a link aggregator: basically links (a URL, title and author id) and comments (text and author ID) + metadata.

They could very well be relational records in a table, except for the fact that they have one array field with IDs pointing to child records.

What implementation looks more solid?

JSON objects on a PostgreSQL database (just one large table with one column, namely the JSON object)
JSON objects on a MongoDB
Explode the JSON objects into columns and use arrays on PostgreSQL

I want to maximize performance in joins, so I can massage the data and explore it until I find interesting analyses, at which point I think it will be better to transform the data into a form specific to each analysis.

For data load, Postgre outperforms MongoDB.
MongoDB is almost always faster when returning query counts.
PostgreSQL is almost always faster for queries using indexes.

Check out this website  and this one too for more info. They have very detailed explanations.


Combining 2 Neural Networks 2 images as input, x1 and x2  and try to use convolution as a similarity measure. The idea is that the learned weights substitute more traditional measure of similarity (cross correlation, NN, ...). Defining my forward function as follows:

def forward(self,x1,x2):
    out_conv1a = self.conv1(x1)
    out_conv2a = self.conv2(out_conv1a)
    out_conv3a = self.conv3(out_conv2a)

    out_conv1b = self.conv1(x2)
    out_conv2b = self.conv2(out_conv1b)
    out_conv3b = self.conv3(out_conv2b)

Now for the similarity measure:

out_cat = torch.cat([out_conv3a, out_conv3b],dim=1)
futher_conv = nn.Conv2d(out_cat)

Question is as follows:

Would Depthwise/Separable Convolutions as in the google paper yield any advantage over 2d convolution of the concatenated input. For that matter can convolution be a similarity measure, cross correlation and convolution are very similar.
It is my understanding that the groups=2 option in conv2d would provide 2 separate inputs to train weights with, in this case each of the previous networks weights. How are these combined afterwards?

Depthwise separable convolution followed by 1x1 convolution is an alternative to the convenient 2d convolution. They were introduced in https://arxiv.org/abs/1704.04861 to aiming on reducing the computational effort. The combination of depthwise and pointwise convolutions often learn pretty much the same representation as the normal conv.
Depthwise separable conv is an extreme case of grouped conv, where num_groups=num_input_channels. Setting number of groups to 2 means slicing the input into two inputs across the channel dim and learning two separate filters for each sliced input. Here is nice tutorial on that. The output is combined by stacking each group's output. Often this is followed by 1x1 conv to mix the data flow between the groups.


Detecting Different Distributions in data Supposing I have a dataset that I assume that have instances generated by two different distributions, is it possible to separate these instances based on the underlying generating distribution?

Also, if I relax this hypothesis, and suppose I know one of the distributions beforehand, is this task possible? The second distribution could he chaotic, and also it is impossible to know it.

Finally, is there any result concerning the distance between the two distributions? Is there any theoretical proof of correctness, if there are algorithms for realizing that?

Thank you!

It's not really feasible to label points as coming from distribution A or B if there is be overlap amongst the distributions. You can't know whether the point fell near the mean of A or the tail of B. In the simplest case, consider two normally distributed datasets with overlap - how can you know which generated it?

You could estimate the probability of each point falling under A or B, but that doesn't really solve the problem.

A different approach, but one I'm not sure will help, would be to fit a new distribution defined as the convolution of A and B. The ex-gaussian distribution is one such example. But you'd have to do the maths to figure out the PDF equation, and that still won't necessarily label your points.


How to prevent/tell if Decision Tree is overfitting? In SKLearn's documentation on Decision Trees, they say we should pay special attention not to overfit the tree. How can we do this? I am aware that using random forests may prevent it, but how do I generally tell if it's overfitting? Can you tell by an accuracy score?

Is, for example, a 0.99 accuracy score an indicator of overfitting? Would in this case 0.95 mean not overfitting? What are some best practices in addition to a balanced input to avoid overfitting (especially with using SKLearn)?

Overfitting meaning your model is learning the noise from the data and its ability to generalize the results is very low. In this case you have a small training error but very large validation error. If you inspect (e.g. by plotting) the evolution of training and validation errors, you see that training error is always going down but validation error is goes up at some point. That is the point you need to stop training to avoid overfitting. I strongly recommend you to read this.

So, the 0.98 and 0.95 accuracy that you mentioned could be overfitting and could not! The point is that you also need to check the validation accuracy beside them. If validation accuracy is falling down then you are on overfitting zone!

  What are some best practices in addition to a balanced input to avoid
  overfitting?

It is called Prunning. Beside general ML strategies to avoid overfitting, for decision trees you can follow pruning idea which is described (more theoretically) here and (more practically) here. In SciKit-Learn, you need to take care of parameters like depth of the tree or maximum number of leafs.


Find the number of homes with 3000 sq ft , 3 number of bedrooms and 40 years of age? there is a dataset like this
area   bedroom  age  price

2600        3    20  550000
3000        4    15  565000
...

Now the questions are

find the price of the house with 2500 sft and 4 bedrooms and 5 years old?

find the home with 3000sft and 3 bedrooms and 40 years old?

I know 1st question can be done with Linear Regression. How should I do second question ?

If it's "find the house" then that implies that you need to find the house with those specific attributes in the dataset. I'm not sure what programming language you're using but it should be pretty simple to do such a thing anyways.
If it's "find the price of a house with..." that implies there probably isn't such a house in the dataset and you need to train a predictive model to predict the price of such a house. You said to use linear regression, which is a fine choice, if your data is linear... But I would instead suggest an SVR (Support Vector Machine Regressor), and fine-tune some things like the kernel and regularization.


Image Classification using Single Class Dataset using Transfer Learning I only have around 1000 images of vehicle. I need to train a model that can identify if the image is vehicle or not-vehicle. I do not have a dataset for not-vehicle, as it could be anything besides vehicle.

I guess the best method for this would be to apply transfer learning. I am trying to train data on a pre-trained VGG19 Model. But still, I am unaware on how to train a model with just vehicle images without any non-vehicle images. I am not being able to classify it.

I am new to ML Overall, Any solution based on practical implementation will be highly appreciated.

You can use Tensorflow Object Detection API.

In the API, you dont have to have 2 or more labels. It can be used with only one label. By transforming images to tfrecords, you can train and predict other images. The prediction will you give you bounding boxes and probabilities that how much an object identified as a vehicle.

The challenge in here is labeling object locations on the images. LabelImg is a great tool for that.

Hope it helps!


How can I normalize my output data for neural network? I have a dataset consisting of 5 numerical variables, like the sample in the following image:

In this dataset the first four variables are the input and the GDP is the output. I am trying to build a neural network for regression of the GDP variable. For the input variables, I have used PCA to normalize the data, while for the GDP variable I have used a MinMax normalization algorithm. However, the accuracy of the results of the neural network is very low and I guess that the problem is on the normalization of my output variable. Any suggestions on how I can improve the accuracy of my neural network?

Normally, you shouldn't make normalization on your target data. If you want to increase your accuracy of model, you can try different neural network architectures or activation functions and monitor your train/val/test scores.

If you insist to normalize target data, then you may want to convert your predictions and then calculate the accuracy.

Steps may be like below:

normalize data => fit model => take predictions => denormalize predictions => calculate accuracy


What should be the ratio of True vs False cases in a binary classifier dataset? I am using a CNN for sentiment analysis of news articles. It is a binary classification with outputs: Interesting & Uninteresting.
In my dataset, there are around 50,000 Uninteresting articles and only about 200 Interesting articles. I know the ratio is badly skewed.

My question is what should be the ratio in such a scenario.
One approach that I want to try is to cluster the Uninteresting news
articles and take a sample from each cluster for training. Is there
a better approach?

Ideal true vs false ratios don't exist and they should reflect the the reality the best they can, you can always remove negatives if the ratio is too skewed to improve training speed though. Let me explain it with an example. Ads CTR is as old as the internet and it's skewed to less than 1% positives vs. plus 99% negatives. Yet, data scientists prefer to train it on the entire dataset because many negatives will include information that models couldn't find otherwise. They might not provide a lot of information as a positive one but they are still somewhat important. There are approaches where CTR ratios get artificially rebalanced by sampling in case you want a swifter training and it will still work. In your case, positives are 0.4% which resemble CTR on ads so you can: gather more data to increase the number of positives in order to better understand what makes an article interesting. In case that is not possible trying ensembles which often improve prediction performance.

Clustering is an unsupervised approach so you would be losing information by doing so (training labels) besides, sentence embeddings (representations)  of one big cluster of negatives and a tiny cluster of positives do not convey information as well as word embeddings which have already been trained on billions of documents.

In addition, running k-means on categorical variables will yield anomalous clusters because it's meant to be used with continuous variables. You can find more information about the topic on the following links:

Kmeans: Whether to standardise? Can you use categorical variables? Is Cluster 3.0 suitable?

My data set contains a number of numeric attributes and one categorical

Kaggle

Why does K means clustering perform poorly on categorical data The weakness of the K means method is that it is applicable only when the mean is defined one needs to specify K in advance and it is unable to handle noisy data and outliers

Therefore, you should use high dimensional embeddings or representations to cluster meanings together, this has been explored in word meanings but for sentences or articles, a vector representation becomes more complicated to implement. One possible approach is the Word Movers’ Distance but there are many more possible approaches, you should google them. In addition a non-linear clustering algorithm such as t-sne will  probably yield better results than k-means using the embeddings approach.
A better approach is:

to use multiple models and compare their performance on this dataset. I have the impression that there will be certain keywords that make articles interesting, so a bag of words will still be helpful, even as a starter model.

Use feature engineering. Your model might be overloooking important features, such as article length, reading time, number of paragraphs, ratio of complex words  (measured by length), etc. Feature engineering is always important in case you haven't used it yet.

Use pretrained embeddings. CNN and RNN models can use pretrained embeddings such as GloVe, Word2Vec or FastText so you use better representations plus other complex layers later on in the architecture. This is extremely important to increase accuracy.

Use metrics to measure improvement and ranks to check for the best predicted interesting articles.


interpret cnn structure I am trying to interpret the CNN model from the below settings. AS I am new to deep learning and I am not able to fully comprehend the layer structure . Could someone please tell me is these two settings are similar because I read in one research paper they used first setting and wrote that is similar to the second setting

A CNN for MNIST with two 5x5 convolution layers (the first with 32 channels, the second with 64, each followed with 2x2 max pooling), a fully connected layer with 512 units and ReLu activation, and a final softmax output layer (1,663,370 total parameters).

The CNN for MNIST has 8 layers with the following structure: 3×3×32 Convolutional → 3×3×64 Convolutional → 2×2 MaxPool → Dropout → Flatten → 1 × 128 Full connected → Dropout → 128 × 10 Fully connected → Softmax.

The CNN for MNIST: 5x5x32 Convolutional -> 2x2 MaxPool -> 5x5x64 Convolutional -> 2x2 MaxPool -> Flatten -> 1x512 Full connected -> ReLu -> 512 x 10 Full connected -> Softmax.
The CNN for MNIST has 8 layers with the following structure: 3×3×32 Convolutional → 3×3×64 Convolutional → 2×2 MaxPool → Dropout → Flatten → 1 × 128 Full connected → Dropout → 128 × 10 Fully connected → Softmax.

Converted to both to same formats. They are not the same.


How to bypass ID column without being used in the training model but have it as output - Azure ML The input data in the model includes column ControlNo.

But I don't want this column being part of learning process so I'm using Select Columns in Dataset to exclude ControlNo column.
But as a output I want those columns:
ControlNo, Score Label, Score Probability

So basically I need NOT to include column ControlNo into learning process,
but have it as output along with Score Label column.
How can I do that?

You have two options to do this, one less elegant than the other:

You can drop whatever column you don't want to use and add them together after getting the scores. Though you might need to reorder columns etc.

Or...

There is a block called Edit Metadata which can be used to do what you need. Select whatever columns from the column selector you don't want to use and use the option Fields > Clear Feature from the block properties. These columns won't be used in the calculations but will be present when you visualise the Score Model output together with the rest of the columns and the scores. Moreover, columns in that output statistics will be labelled as Feature type if they were used in the calculations. So you straightaway know which columns were actually used in the calculations or not. Block help is quite useful here.


how to handle values that only appear once in a column? Counting the values of a column using pandas I got the following result:

Human                 195
Mutant                 62
God / Eternal          14
Cyborg                 11
Human / Radiation      11
Android                 9
Symbiote                8
Kryptonian              7
Alien                   7
Demon                   6
Atlantean               5
Alpha                   5
Asgardian               5
Cosmic Entity           4
Inhuman                 4
Human / Altered         3
New God                 3
Animal                  3
Saiyan                  2
Eternal                 2
Frost Giant             2
Human-Kree              2
Demi-God                2
Human / Cosmic          2
Vampire                 2
Metahuman               2
Amazon                  2
Icthyo Sapien           1
Czarnian                1
Rodian                  1
Martian                 1
Clone                   1
Zombie                  1
Maiar                   1
Yoda's species          1
Human-Vulcan            1
Zen-Whoberian           1
Mutant / Clone          1
Korugaran               1
Dathomirian Zabrak      1
Parademon               1
Kaiju                   1
Flora Colossus          1
Human-Spartoi           1
Yautja                  1
Ungaran                 1
Human-Vuldarian         1
Neyaphem                1
Xenomorph XX121         1
Bizarro                 1
Human / Clone           1
Gungan                  1
Bolovaxian              1
Talokite                1
Luphomoid               1
Tamaranean              1
Kakarantharaian         1
Spartoi                 1
Strontian               1
Gorilla                 1
Name: Race, dtype: int64

I am new to data science, but I think that all those value appearing only once in the dataset is not going to help the classifier, so is there a good way to handle those values? I was thinking about grouping all those values that appear less than 5 times, or maybe I should remove the lines. By the way, I do not know if it is important to know, but I want to apply the gaussian naive bayes, knn and logistic regression to this dataset. This column is a feature to predict a binary value.

Assuming those are target categories, the most common strategies are:

Drop them. Do not try to predict categories with single observations because the model will overfit.
Bundle them to together into a "dustbin" category. Then the model can learn to predict "dustbin" category membership.


Named entity recognition (NER) features I'm new to Named Entity Recognition and I'm having some trouble understanding what/how features are used for this task.

Some papers I've read so far mention features used, but don't really explain them, for example in
Introduction to the CoNLL-2003 Shared Task:Language-Independent Named Entity Recognition, the following features are mentioned:

  Main features used by the the sixteen systems that participated in the
  CoNLL-2003 shared task sorted by performance on the English test data.
  Aff: affix information (n-grams); bag: bag of words; cas: global case
  information; chu: chunk tags; doc: global document information; gaz:
  gazetteers; lex: lexical features; ort: orthographic information; pat:
  orthographic patterns (like Aa0); pos: part-of-speech tags; pre:
  previously predicted NE tags; quo: flag signing that the word is
  between quotes; tri: trigger words.

I'm a bit confused by some of these, however. For example:

isn't bag of words supposed to be a method to generate features (one for each word)? How can BOW itself be a feature? Or does this simply mean we have a feature for each word as in BOW, besides all the other features mentioned?
how can a gazetteer be a feature?
how can POS tags exactly be used as features ? Don't we have a POS tag for each word? Isn't each object/instance a "text"?
what is global document information?
what is the feature trigger words?

I think all I need here is to just to look at an example table with each of these features as columns and see their values to understand how they really work, but so far I've failed to find an easy to read dataset.

Could someone please clarify or point me to some explanation or example of these features being used?

The features for a token in a NER algorithm are usually binary. i.e The feature exists or it does not. For example, a token (say the word 'hello'), is all lower case. Therefore, that is a feature for that word.

You could name the feature 'IS_ALL_LOWERCASE'.

Now, for POS tags, lets take the word 'make'. It is a verb and hence the feature "IS_VERB" is a feature for that word.

A gazetter can be used to generate features. The presence (or absence) of a word in the gazatter is a valid feature. Example: the word 'John' is present in the gazetter of Person names. so "IS_PERSON_NAME" can be a feature.


Computation of kernel matrix using radial basis kernel in svm I want to compute a kernel matrix using RBF on my own. The training data is multidimensional. My query is whether we will apply
$$e^{-\gamma(x-y)^2}$$
for each dimension and then sum the values across all dimension?

The kernel matrix $K$ or Gram Matrix is a positive-semidefinite symmetric matrix that has the form:
$$ K(x_1,x_2,\dots,x_n) =
\begin{vmatrix}
\langle x_1,x_1 \rangle & \langle x_1,x_2 \rangle & \dots & \langle x_1,x_n \rangle \\
\langle x_2,x_1 \rangle & \langle x_2,x_2 \rangle & \dots & \langle x_2,x_n \rangle \\
\vdots & \vdots & \ddots &\vdots \\
\langle x_n,x_1 \rangle & \langle x_n,x_2 \rangle & \dots & \langle x_n,x_n \rangle
\end{vmatrix}
$$

Where $\langle a,b \rangle$ detones the inner product/dot product of vectors $a$ and $b$ and $\{x_1,x_2,\dots,x_n\}$ is a set of vectors (in this case, training samples).

In order to calculate the Gramian Matrix you will have to calculate the Inner Product using the Kernel Function. For a RBF kernel function $\kappa_{RBF}$ this can be done by

$$K_{ij} = \kappa_{RBF}(x_i,x_j) = e^{\gamma D_{ist}(x_i,x_j)^2} $$

where $\gamma$ is a function hyperparameter, $K_{ij}$ is the element in row $i$ and column $j$ of the matrix $K$ and $ D_{ist}(x_i,x_j)$ is some distance between two vector measured in some vector space.

Usually, the distance measure used is the $L_2$ norm or euclidean distance. Remember that when computing the Kernel function you will use the square $L_2$ norm, so there is no need to take the square root.

There is a open library named pykernels that has implementations to dozens of kernel functions. This here is the implementation for the RBF (or Gaussian) Kernel:

class RBF(Kernel):
    """
    Radial Basis Function kernel, defined as unnormalized Gaussian PDF
        K(x, y) = e^(-g||x - y||^2)
    where:
        g = gamma
    """

    def __init__(self, gamma=None):
        self._gamma = gamma

    def _compute(self, data_1, data_2):
        if self._gamma is None:
            # libSVM heuristics
            self._gamma = 1./data_1.shape[1]

        dists_sq = euclidean_dist_matrix(data_1, data_2)
        return np.exp(-self._gamma * dists_sq)

    def dim(self):
        return np.inf

it uses the square distance define as:

def euclidean_dist_matrix(data_1, data_2):
    """
    Returns matrix of pairwise, squared Euclidean distances
    """
    norms_1 = (data_1 ** 2).sum(axis=1)
    norms_2 = (data_2 ** 2).sum(axis=1)
    return np.abs(norms_1.reshape(-1, 1) + norms_2 - 2 * np.dot(data_1, data_2.T))

The class is defined as a subclass of the Kernel class and has some methods to deal with kernel combinations.

Note that this is not the fastest way to implement euclidean distance, you can do better but this is a simple form to do it. also that "np.abs" should not be needed

Note 2 that this SO Question has a lot of optimized version of RBF Kernel