Sampling n data points from high dimensional data I have some face images(of a single person), which I ran through an embedding generator to get 128-dimensional embedding.
I have 1000 such embedding (shape of the dataset (1000, 128)). I have a restriction on the number of embeddings that can be used to train the model (100 embeddings). I want to pick 100 embeddings from all 1000  which will represent all 1000 embeddings.
My question is, how can I choose the best 100 embeddings which will represent the entire 1000 embeddings.
Some things I have tried out.

Picking the 100 farthest points in the cluster. (all the images are edge cases like blurred image, improper pose e.t.c)
Random sampling ( works ok but sometimes pick embeddings which were calculated from a face which was having some problem, like blurred e.t.c.)

There is one more thing, I have thought of but not tested.
Sampling with probability. Probability = 1/(euclidean_distance of point from cluster centroid).
I wanted to know if there are any alternatives, that I can look into which can provide better results.

One option is to use something like seeding technique of k-means++ to pick representative sample of data points. K-means++ picks each subsequent cluster center from the remaining data points with probability proportional to its squared distance from the point's closest existing cluster center.


Can a recommendation system be used as a binary classifier? I have a computer-generated music project, and I'd like to classify short passages of music as "good" or "bad" via machine learning.

I won't have a large training set. I'll start by generating 500 examples each of good and bad music, manually. These examples can be transposed and mirror-imaged to produce 12,000 examples of each good and bad.

I have a way of extracting features from the music in an intelligent way that mimics the way a perceptive listener would do. The trouble is that it involves thousands of features on a single passage. Each feature is a number from 0 to 1.

I'm pretty new to data science, and my understanding is that binary classifiers like decision trees wouldn't work well with that many features. But note that I'll have a lot of clumping and dependence of features. I just don't know how to predict that structure. Also, good examples will probably clump around a few categories, while bad examples will tend to be all over the place and "far" from good examples.

It occurred to me that maybe a recommendation system could work.

Taking the example of movie recommenders, the score of each of my features  would be like the rating a viewer gives a movie. Each example will rate thousands of features, i.e. a viewer rating thousands of movies.

Then we have one movie, called the "Good Movie." Each training example gives that either 5 stars or 0 stars depending on whether it's good or bad.

We then take a test example and predict whether it likes the "Good Movie."

Can this sort of thing work? Is there another approach to working with thousands of features?

EDIT: adding a comment on dependence or "clumping" of features: I intuitively think that "good" music will probably clump around several categories and within those clumps there will be strong correlation between features. Let's say inside clump #1, features A and B are correlated. But inside clump #2, A and B might be uncorrelated with each other, although each correlated to other features. I mention this in case it's relevant to doing feature reduction with PCA or other techniques.

Would this approach work?

I don't know, you should try, it very well could.

Can recommender systems be used as classifiers?

Absolutely! But first, let's clarify a couple of things. Recommender systems are systems. They are the output you are trying to achieve. This can theoretically be achieved without machine learning but in general, building recommenders happens via a learning algorithm (i.e. collaborative filtering, neural nets, etc.).

What that learning algorithm does is learning to provide a "score" to each item given a specific context (i.e. user X wanted to watch a movie on a Tuesday night) such that the highest scores are given to items that are worth recommending.

This can directly be applied to a classification problem where your learning algorithm learns to give high scores to the correct class.

So in summary, you would still be building a classifier system (what it does) by using a recommendation learning algorithm (how it does it).


How would you deal with inf. or NA for rate or ratio as a feature variable I'm trying to create a feature for a churn model (binary classifier).
The feature is mean of sales growth rates for several months. But if I just take the mean of sales for several months, I often get NAN or inf. since sales are often zeros. I could impute some numbers like 0 or mean as the missing sales but I feel I'm modifying the pattern/underling distribution. How would you go by creating such a feature for a classification model?

Thanks!

It depends on how would you define sales growth from a business point of view. For example if you trying to detect sales peaks with considering zero sales as normal,  you can fill zeroes, nans or infs with its most recent previous that has an ok number.


Keras' Evaluate function training model on test set I've been using Keras to do some timeseries predictive neural nets.
One thing that's had me staggered is how I was getting an almost 99% accuracy rate on very very noisy and uncorrelated data (I was looking for accuracy of around 50% to be good!)...

What I've now found, is that after I trained my model on the training set, I was casually running model.evaluate(x_test, y_test) before then running a manual backtest on the data. The manual backtest on x_test and comparing it to the error in y_test then showed almost perfect accuracy...

Am I then correct in assuming that Keras when using the evaluate() function in Keras the original model is actually updated with the testing data as well? This seems a bit strange and counterintuitive to me?

Keras doesn't update your model with testing data.
It might be that your labels have been provided wrong in the test data, check the model.predict_classes() to get the output of your classes and crosscheck them with your actual output manually by picking random subset.


How many dimensions to reduce to when doing PCA? How to choose K for PCA? K is the number of dimensions to project down to. The only requirement is to not lose too much information. I understand it depends on the data, but I'm looking more for a simple general overview about what characteristics to consider when choosing K.

After performing the PCA algorithm you get the principal components, sorted by the amount of information they hold. If you keep the whole set there is no information lost. Removing them one by one and projecting them back onto the original space you can calculate the information loss. You can plot this information loss against number of principal components removed and see if it makes an 'elbow' where it makes sense. A lot of this depends on your use case though.


PCA-like analysis for dataset that has both categorical and continuous variables I have a dataset containing a categorical variable and multiple continuous variables. The categorical variables are coded as discrete integers, whereas the continuous variables are just a range of floats. I believe that the variance in my dataset can be almost entirely described by the single categorical variable and one of the many continuous variables. To justify this, I would be interested in using PCA, but I'm not sure the best approach to use when I am considering categorical data. Any suggestions?

I'm not aware of any the dimensionality reduction algorithms (like PCA) that can work with categorical values.

However, an approach that could help you with that is to make a one-hot encoding of your categorical variables (if the number of possible values is manageable. Otherwise, try to pick only the most frequent values and assign the rest to a single variable).

If you are using Pandas DataFrames, get_dummies can be helpful.


When and where dummy.data.frame would be used? I am learning this post, in which, the author gives the piece of code:

outcome<-dataf1$outcome
dataf1$outcome<-NULL
datafd<-dummy.data.frame(dataf1)
hdata<-cbind(data,datafd,outcome)
head(str(hdata))
data<-hdata

Other parts are easy to understand though, what does this line: dummy.data.frame(dataf1) do?

I constructed a tiny dataset to figure this out:

title <- c('engineer','manager','sales','engineer','sales')
salary <- c(51000, 33700, 26800, 53700, 36800)
df = data.frame(title, salary)
df
dummy.data.frame(df)

Which produces:

  titleengineer titlemanager titlesales salary
1             1            0          0  51000
2             0            1          0  33700
3             0            0          1  26800
4             1            0          0  53700
5             0            0          1  36800

I cannot imagine what this is used for.

The original df looks like:
     title salary
1 engineer  51000
2  manager  33700
3    sales  26800
4 engineer  53700
5    sales  36800

Using dummy.data.frame(df), the character column "title" is recoded as "one-hot" (aka as dummy variable). Each column =1 if title=TRUE (or zero otherwise).
  titleengineer titlemanager titlesales salary
1             1            0          0  51000
2             0            1          0  33700
3             0            0          1  26800
4             1            0          0  53700
5             0            0          1  36800

An alternative encoding would be to use "title" as a factor with three levels
df$title = as.factor(df$title).
However, in many cases "one hot" can be digested better by ML algorithms.
Simple example:
title <- c('engineer','manager','sales','engineer','sales')
salary <- c(51000, 33700, 26800, 53700, 36800)
df = data.frame(title, salary)
df

library(dummies)
df2 = dummy.data.frame(df)
df2

OLS model with "factor":
df$title = as.factor(df$title)
ols = lm(salary~., data=df)
summary(ols)

Output:
Call:
lm(formula = salary ~ ., data = df)

Residuals:
         1          2          3          4          5
-1.350e+03  5.684e-14 -5.000e+03  1.350e+03  5.000e+03

Coefficients:
             Estimate Std. Error t value Pr(>|t|)
(Intercept)     52350       3662  14.295  0.00486 **
titlemanager   -18650       6343  -2.940  0.09882 .
titlesales     -20550       5179  -3.968  0.05804 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 5179 on 2 degrees of freedom
Multiple R-squared:  0.8992,    Adjusted R-squared:  0.7983
F-statistic: 8.918 on 2 and 2 DF,  p-value: 0.1008

OLS model with "dummies":
ols2 = lm(salary~titlemanager+titlesales, data=df2)
summary(ols2)

Output:
Call:
lm(formula = salary ~ titlemanager + titlesales, data = df2)

Residuals:
         1          2          3          4          5
-1.350e+03  5.684e-14 -5.000e+03  1.350e+03  5.000e+03

Coefficients:
             Estimate Std. Error t value Pr(>|t|)
(Intercept)     52350       3662  14.295  0.00486 **
titlemanager   -18650       6343  -2.940  0.09882 .
titlesales     -20550       5179  -3.968  0.05804 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 5179 on 2 degrees of freedom
Multiple R-squared:  0.8992,    Adjusted R-squared:  0.7983
F-statistic: 8.918 on 2 and 2 DF,  p-value: 0.1008

Summary:
Results are the same. Dummies are a different representation of R's factors. Sometimes you need to pass content explicity as dummies. In this case model.matrix() often is useful. (docs)


How to measure/rate the effect of a exogenous covariate in a ARIMAX Model? I have an ARIMA model, I'm trying to figure out how much an external variable (exogenous covariate) could improve the forecast, so I need to "synthesize" a rate that tell me the usefulness (or impact) of a new external regressor. Does anyone know a good way to do it?

Do this: create a series X, which is a multiple of your target variable say 3 or 5 times or a fraction of your target variable say 10% or 15%. Multiply your target variable values with the selected multiple or fraction but don't multiply all your target values by say a single multiple or fraction value, multiply target values by say numbers in a close range say between 10-13. example below

so Step 1: if my Y (target)= [10,20,30,40..], then I create X as [10*10=100,20*13=160,30*11=330,40*11.5=460...] (created random numbers between 10-13 and multiplied them by  target values) Same multiple will create perfect relation between X & Y, and different multiple values in a close range will overcome this to a good extent.

Step 2: To further ensure that  X is not a perfect predictor of Y but only reasonably good one and there are other predictors at play like TIME (since you are doing ARIMA), add random noise to X by creating some random numbers and adding them to series X creating a new series say Xnew.

random noise= [25,37,31..]

Xnew= [100+25, 160+37, 330+31..so on]

thus you can now add Xnew to your ARIMAX model.

Xnew is definitely a significant predictor, that you will see in your ARIMAX results.

Best Vik


What brings the performance difference in Deep Learning with different data augmentation strategies? I am studying the performance of deep learning models toward abnormality detection in chest X-rays.

Due to sparsity of data, I augment the data using different augmentation strategies including:

Traditional augmentation methods (Gaussian smoothing, unsharp masking, and minimum filtering)
Generative Adversarial Networks

Contrary to the existing literature, I find that the models showed promising results with traditional augmentation methods (that i have mentioned herewith) than with GAN-generated synthetic images.

What brings this performance difference?

In general, you have to be careful when using data augmentation.

For example, doing rotation for this kind of image makes sense, we expect to see any of these images as potential 'real-life' example :

However, doing rotation for this kind of image is less meaningful. We don't expect to see this in 'real-life' example :

And GAN potentially makes generated image meaningless.
If your GAN produce 'thrash' augmented-data, then your network will train and learn 'thrash', which you don't want.

When you are training your model on GAN generated images, you're actually training your model to recognize GAN-generated image, not real-life example.

Sources:
 - towardsdatascience
 - quora question


What is the difference between float64 and double in TensorFlow? In storing floating point values both overflow and underflow problems cause loss of data. In machine learning tasks underflow is a common problem. I wanted to know if double is better than float64 in TensorFlow or not and if there is any difference between them?

Taking a look at tensorflow's dtypes.py, there's this line:

double = float64

So double is exactly the same at float64.


Could I add a one hot encoding to each feature representing "has data" versus "has no data" I have a data set that has some holes in it. I was wondering if I could add two columns for each feature representing this feature has data and this feature doesn't have data for each of my features. Would a CNN be able to take advantage of this or should I just use another method to approximate the result.

Note The reason why I'm looking for an alternative is that I have a very small dataset and I assume extrapolating would insert to great of a bias on whatever method I use determine the missing value.

Sample Data

0.0,0.3,0.2
0.4,Nan,0.2
0.2,0.6,Nan
Nan,0.1,0.5
0.0,0.2,0.0
0.9,Nan,Nan

Would be converted to

0.0,1,0, 0.3,1,0, 0.2,1,0
0.4,1,0, 0.0,0,1, 0.2,1,0
0.2,1,0, 0.6,1,0, 0.0,0,1
0.0,0,1, 0.1,1,0, 0.5,1,0
0.0,1,0, 0.2,1,0, 0.0,1,0
0.9,1,0, 0.0,0,1, 0.0,0,1

Since my dataset is already relatively small I don't particularly want to remove entries. This would reduce my data by roughly a third.

If this is a 'foul ball out of left field' (makes no sense and would cause more errors) how would you deal with a small data set with a decent amount of holes.

EDIT: As per @zachdj suggestion in comments

Would be instead converted to this data set which is much smaller.

0.0,1, 0.3,1, 0.2,1
0.4,1, 0.0,0, 0.2,1
0.2,1, 0.6,1, 0.0,0
0.0,0, 0.1,1, 0.5,1
0.0,1, 0.2,1, 0.0,1
0.9,1, 0.0,0, 0.0,0

If you are using Python, the Imputer class could be a way to fill in the holes. You can use it to define what counts as a missing value and define a method for replacing those (mean, median, mode, constant value). Without knowing what the data is, it's hard to say whether this is an ideal solution, but it is a very quick one.

The Imputer class also includes a MissingIndicator method which would ouput a Boolean array based on the data. This could be another option, and then you could stack the Boolean array onto the original data.


Anomaly detection with time series I have to study the behavoiur of a machine during its works: I have available time series of pressure, temperature and other physical measures for each work it is performing,
I would like to predict errors or failures but till now they have not been recorded yet. So I thought to using these time series to understand if there are abnormal behaviours (which could lead to a faillure), maybe clustering time series. Beacuse I suppose that the machine, in normal condition, has the same parameters for each work.
Is there a procedure/documentation/ best practises in order to do anomaly detecion with time series?
Thanks a lot!

You can extract features from timeseries sliding window, eg. mean, std, etc etc, and then use these as features into your model for anomaly detection


Free Service for Alpha Zero training I'm an AI student I need to train a deep neural network using the Alpha Zero (Silver et al) for a simple game using this implementation: http://web.stanford.edu/~surag/posts/alphazero.html. I was wondering if any cloud provider like Google or Amazon offers a free trial which suffice to train the model and supports the implementation mentioned above. The game is an Android app called Soccer Stars which is a fairly simple game with simple strategies.

Thank you very much for your time.

You might be able to fit your training into a Google Colab session. You can search for tutorials using different libraries, here is one for PyTorch. Google Colab is essentially a free online Jupyter notebook, and makes K80 GPU available for acceleration.

Sessions on Colab are limited to ~12 hours maximum, so you would need to start with some simple variants and build up to max allowed training time (this is good practice anyway). There is no guarantee that this will be enough for your game, but you should learn a lot even if the resulting agent is not perfect.

It is not clear whether you have already figured this out, but you most likely will not be able to host the Android App natively to run as the environment, and will need to code some kind of simulation or re-implement the game. I would also assume it is a card, board or turn-based strategy game with perfect information, if you are hoping to use an AlphaZero-based learning agent with it (AlphaZero could probably be adapted to video games, but I would not expect it be as efficient as simpler algorithms without a planning phase, such as A3C in that case).


suggest ingredients based on recipe title I would like to construct system that would suggest user ingredients once he/she inputs title of the recipe.

I think that this is the task of machine learning or AI, but on the other hand I am pretty new to ML and generally in AI development and I feel kinda lost, since I don't know where to start and with what to start.

So far, I've managed to crawl data from web page with recipe title and ingredients which are used for this recipes.

My thoughts are that, this problem may be solved with suggester system, but on the other hand I think classification / clustering algorithms may be used to divide recipes in clusters / categories and once input title is associated to cluster ingredients may be generated from that cluster, but I don't know which is best solution and I wonder if there may be other ones.

Thanks in advance!

I'm posting my temporary solution which may help others solve similar problems and I think this is pretty basic, but works nice:

import all data to database (postgres in my case), where Recipe table has only name and Ingredient table has name and ForeignKey to recipe
once user inputs recipe name (rname), I run query using postgres' Trigram Similarity module to detect similarity between rname and recipe table names. Then I filter out recipes which similarity is greater than .1 (for example)
After that, I get all ingredients which are linked to filtered recipes, annotate ingredient usage for individual ingredient (calculate ingredient usage ratio). And order by usage ratio DESC
Finally, I return ingredients (10 mostly used ones)

I think, looking through proposed solutions and also posting them.

If the user inputs a title, then you could construct a system which finds the most similar titles in the corpus, and outputs the ingredients in the retrieved recipes. Some ideas below:

1) Represent the titles in a common vector space using a vocabulary-based vectorization and use Jaccard or Cosine similarity to find the most similar titles.  examples of similarity measures

2) Convert the words in the title to their respective word-embeddings using FastText, GloVe, or Word2Vec (publicly available), and then use cosine distance or  word-movers distance to find the most similar titles. example of publicly available GloVe word embeddings

3) Create sentence embeddings for the title by taking the average of the word embedding vectors or using InferSent embeddings, and then use a distance function to find the most similar titles. InferSent repository link

spaCy is a good Python package for performing these type of NLP tasks link


Can I create a layer with multiple rnn cell ? [question about a paper] I am trying to implement https://dl.acm.org/doi/pdf/10.1145/3269206.3271794  .
Structure:

As it said:

In particular, we integrate the embedding vectors learned from each
individual recurrent encoder into a new conclusive embedding vector to
jointly consider various time series patterns with different ⟨α, β⟩
configurations

For my understanding, it use multiple individual  rnn cell to process different timeseries, then concat all hidden states together to form a 3D input which can use 2d conv extract features .
But I didn't see there is a way to create multiple rnn cells in same layer , do I misunderstand??
If not , could you please give me a guide or an example ?

Not exactly familiar with this model architecture but, I don't think it works the way you described it.
From a quick skim, a simplified overview of the model may be

data input: $X$ multiple time series
RNN: pass each time series $x$ through a recurrent neural network and grab its hidden state
Joint embedding: Concat hidden states from each $x$ series into an embedding with $X$ channels
Convnet

But I didn't see there is a way to create multiple rnn cells in same layer , do I misunderstand?? If not , could you please give me a guide or an example ?

Essentially, any rnn will have a number of cells equal to the number of time steps. As an example, in NLP that number of cells would be equal to the number of words in a sentence, where each sentence in your trainset must have the same length.
I suggest you familiarise yourself with sequence models to better understand the concept of this paper.


Class label prediction in keras sequential model showing different results in confusion matrix With Keras Sequential Model Prediction
To get Class Labels
we can do
yhat_classes1 = Keras_model.predict_classes(predictors)[:, 0] #this shows deprecated warning in tf==2.3.0

WARNING:tensorflow:From <ipython-input-54-226ad21ffae4>:1: Sequential.predict_classes (from tensorflow.python.keras.engine.sequential) is deprecated and will be removed after 2021-01-01.
Instructions for updating:
Please use instead:* `np.argmax(model.predict(x), axis=-1)`,   if your model does multi-class classification   (e.g. if it uses a `softmax` last-layer activation).* `(model.predict(x) > 0.5).astype("int32")`,   if your model does binary classification   (e.g. if it uses a `sigmoid` last-layer activation).

or
yhat_classes2 = np.argmax(Keras_model.predict(predictors), axis=1)

With the first class labels if i create confusion matrix, i get
matrix = confusion_matrix(actual_y, yhat_classes1)
 [[108579   8674]
 [  1205  24086]]

But with the second class labels with the confusion matrix, i get 0 for True Positive and False Positive
matrix = confusion_matrix(actual_y, yhat_classes2)
 [[117253      0]
 [ 25291      0]]

May I know whats the issue here?

Model.predict_classes gives you the most likely class only (highest probability value), therefore it is of dimension (samples,), for every input in sample there is one output - the class.
To be more precise  Model.predict_classes calls argmax on the output of predict. See the code (of predict_classes below)
def predict_classes(self, X, batch_size=128, verbose=1):
  proba = self.predict(X, batch_size=batch_size, verbose=verbose)
        if self.class_mode == 'categorical':
            return proba.argmax(axis=-1)
        else:
            return (proba > 0.5).astype('int32')

As the warning suggest, Please use instead:

np.argmax(model.predict(x), axis=-1),   if your model does multi-class classification   (e.g. if it uses a softmax last-layer activation).
(model.predict(x) > 0.5).astype("int32"),   if your model does binary classification   (e.g. if it uses a sigmoid last-layer activation).


Naive Bayes Classifier - Discriminant Function To classify my samples, I decided to use Naive Bayes classifier, but I coded it, not used built-in library functions.

If I use this equality, I obtain nice classification accuracy: p1(x) > p2(x) => x belongs to C1

However, I could not understand why discriminant functions produce negative values. If they are probability functions, I think they must generate a value between 0 and 1.

Is there anyone who can explain the reason ?

In Naive Bayes, for the case of two classes, a discriminant function could be $$D(\boldsymbol{x}) = \frac{P(\boldsymbol{x}, c=1)}{P(\boldsymbol{x}, c=0)}$$ which can be anywhere in $[0, +\infty)$, and decides $c=1$ if $D(\boldsymbol{x})>1$, $c=0$ otherwise, or it could be the logarithm of that value

$$d(\boldsymbol{x}) = \text{log}\frac{P(\boldsymbol{x}, c=1)}{P(\boldsymbol{x}, c=0)}=\text{log}P(\boldsymbol{x}, c=1)-\text{log}P(\boldsymbol{x}, c=0)$$
which can be anywhere in $(-\infty, +\infty)$ (handling zero probability as a special case), and decides $c=1$ if $d(\boldsymbol{x})>0$, $c=0$ otherwise.

As a side note, $P(\boldsymbol{x}, c=k)$ in Naive Bayes is calculated as
$$P(\boldsymbol{x}, c=k)=P(c=k)\prod_{i=1}^{d}P(x_i|c=k)$$
or equivalently for log probabilities as
$$\text{log}P(\boldsymbol{x}, c=k)=\text{log}P(c=k) + \sum_{i=1}^{d}\text{log}P(x_i|c=k)$$


Is there any way how to make samples balanced? I have a dataset which consists of attributes on breakdown of machines.The target variable is machine status which are populated with ones and zeros. The distribution of ones and zeros are given below

 0 - 19628
 1 - 225

0 - signifies the machine is running good and 1 signifies there was a breakdown.

Now, should I go by splitting the dataset using scikit train_test_split method ?. or introduce artificial rows to mitigate the tradeoff between ones and zeros and then split the dataset.

Well, What do I mean by artificial rows ?
Populate  some random data with having target variable as 1's
But that would ultimately mislead the system. I don't see any other options or alternatives.

Is there any way how to make samples balanced?

SMOTE is a python library popularly used with unbalanced datasets like yours - it applies resampling to create a new balanced dataset. You can find some example implementation here.

Maybe you will also find this question and answers relevant, as it contains advice on limiting bias during modeling and validation of unbalanced data classifier.


LabelEncoder with a Multi-Layer Perceptron? So we're working on a machine learning project at work and it's the first time I'm working with an actual team on this. I got pretty good results with a model that uses the following SKLearn pipeline:

Data -> LabelEncoder -> MinMaxScaler (between 0-1) -> PCA (I go from 130 columns to 50 prime components that cover the variance) -> MLPRegressor

One of my colleagues mentioned that I shouldn't normally use LabelEncoder to encode training data, as it's meant for encoding the target variable. I did some research and now and I understand why

  LabelEncoder only is not a good choice, since it brings in a natural
  ordering for different classes.

Then, however, my colleague mentioned that in this case it shouldn't make much of a difference as I'm using a neural net (~MLPRegressor). My question is - (if he's right - is he?) why? He basically commended me saying this usually would be a bad idea but in this case it should work.

I will try to move to one-hot encoding (currently I'm only stuck with it because I run out of memory while doing PCA on that many columns, but that's another question and I'll do some research on that separately), but for now I'd like to know if using this kind of encoding can result in inaccurately good results (I'm having an r^2 score of around 0.9 and my boss literally won't believe I achieved a result that good haha).

Your colleague was correct that LabelEncoder is meant for the target variable (a.k.a. labels); more-or-less equivalent OrdinalEncoder is meant for the independent variables.

IMO, your colleague is partially correct about an ordinal encoding "not making much of a difference" for a neural network.  A neural net can contort its way into a correct understanding of your labels anyway, although I think an ordinal encoding will make its job substantially more difficult, so I still wouldn't advocate for it.  The intervening steps, scaling and PCA, make it less clear exactly what the neural network will actually see, and I don't know if that alleviate or worsen the issue of an ordinal encoding.  (Given your good score, perhaps alleviate.)


Ordinal Attributes in a Decision Tree I'm reading the book Introduction to Data Mining by Tan, Steinbeck, and Kumar.
In the chapter on Decision Trees, when talking about the "Methods for Expressing Attribute Test Conditions" the book says :

  "Ordinal attributes can also produce binary or multiway splits. Ordinal
  attribute values can be grouped as long as the grouping does not
  violate the order property of the attribute values. Figure 4.10
  illustrates various ways of splitting training records based on the
  Shirt Size attribute. The groupings shown in Figures 4.10(a) and (b)
  preserve the order among the attribute values, whereas the grouping
  shown in Figure a.10(c) violates this property because it combines the
  attribute values Small and Large into the same partition while Medium
  and Extra Large are combined into another partition."

Why ordinal attribute values can be grouped as long as the grouping does
not violate the order property of the attribute values?

I'd say the distinct handling of the ordered and unordered factor in decision trees is more convention and
implementation detail than a necessity.

But it is also an important optimization feature. See the documentation of the  rpart here

  We have said that for a categorical predictor with $m$ levels, all $2^{(m-1)}$   different possible splits  are  tested..

and

  Luckily,  for any ordered outcome there is a computational shortcut that allows the program to find the best split using only $m-1$ comparisons.

As you see, the ordered factor may be processed much effectively.

My advice therefore - as a part of the feature ingeneering decide whether to use a factor ordered or unordered:

Use ordered factor only if it is highly correlated with the output variable, otherwise fall back to an unordered factor

Bellow is a simple example, how can a scattered output variable with an ordered factor as a feature fool the decision treee to be very deep and ineffective.

> df
  X Y
1 1 0
2 2 1
3 3 0
4 4 1
> str(df)
'data.frame':   4 obs. of  2 variables:
 $ X: Ord.factor w/ 4 levels "1"<"2"<"3"<"4": 1 2 3 4
 $ Y: num  0 1 0 1

Notice that the output variable $Y$ is highly uncorrelated with the ordered factor.

> fit <- rpart(Y ~ X, method="class", data = df, control=rpart.control(minsplit = 1))
> print(fit)
n= 4

node), split, n, loss, yval, (yprob)
      * denotes terminal node

 1) root 4 2 0 (0.50000000 0.50000000)
   2) X=1 1 0 0 (1.00000000 0.00000000) *
   3) X=2,3,4 3 1 1 (0.33333333 0.66666667)
     6) X=3,4 2 1 0 (0.50000000 0.50000000)
      12) X=1,2,3 1 0 0 (1.00000000 0.00000000) *
      13) X=4 1 0 1 (0.00000000 1.00000000) *
     7) X=1,2 1 0 1 (0.00000000 1.00000000) *

Which leads to a deep (and unscalable) decision tree.

Making the factor unordered results in the optimal  decision tree.

> df
  X Y
1 1 0
2 2 1
3 3 0
4 4 1
> str(df)
'data.frame':   4 obs. of  2 variables:
 $ X: Factor w/ 4 levels "1","2","3","4": 1 2 3 4
 $ Y: num  0 1 0 1
>

> fit <- rpart(Y ~ X, method="class", data = df, control=rpart.control(minsplit = 1))
> print(fit)
n= 4

node), split, n, loss, yval, (yprob)
      * denotes terminal node

1) root 4 2 0 (0.50000000 0.50000000)
  2) X=1,3 2 0 0 (1.00000000 0.00000000) *
  3) X=2,4 2 0 1 (0.00000000 1.00000000) *


Does RandomForest convergence imply I can solve a problem with a NN too? I'm trying to perform a regression on a dataset, and I've been testing a few models, mostly for practice.
I was able to get good results with a RandomForestRegression model, as you can see in the scatter plot:

So, I tried to solve the same problem with a NN (again, I'm doing it mostly to practice), but the results are definitely bad.
As you can see in the top part of the following picture, the loss on the training and test set (red and green lines, respectively) seem to somewhat converge, but their value is still higher than the one obtained with the RF (the horizontal blue line). Also, the scatter plots are horrible, since the model is basically always predicting the mean of the data.

The model is a simple MLP, with one hidden layer. I'm using a tanh activation function for the hidden neurons. Since the losses seem to have converged, I don't know if using a smaller learning rate or training the model for more epochs would improve the results.
However, my question is more general, independently of this specific result.
I have the general understanding that NNs are a more flexible than a RandomForest, so I am wondering if the fact that I obtained a good result with a RandomForest, automatically implies that there should exist some NN configuration which could provide a result at least as good as the one obtained with the RF.
Is this true, and I just have to fix the NN model until I find the right configuration?
Or is it possible that this problem can be properly modelled by a RF, but not by a NN?

To me those are separate things since both models have a different cost function to be optimized.
On the other hand you could combine those models by constructing embeddings based on random forest splits and then using those embeddings as inputs for a neural network.
Toy example shows that there is a non-trivial configuration of a neural net that can get as good results as those obtained a by a random forest:
from sklearn.datasets import load_iris
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import RandomTreesEmbedding

X, y = load_iris(return_X_y = True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 42, test_size = .2)

params = dict(n_estimators=100,
    max_depth=None,
    min_samples_split=2,
    min_samples_leaf=1,
    min_weight_fraction_leaf=0.0,
    max_leaf_nodes=None,
    min_impurity_decrease=0.0,
    min_impurity_split=None,
    random_state=42,
    verbose=0)

mlp = Pipeline([("embeddings", RandomTreesEmbedding(**params)),
                ("model", MLPClassifier(activation = "identity",hidden_layer_sizes=(1000,), max_iter = 10000, random_state = 42))]).fit(X_train, y_train)

rf = Pipeline([("model", RandomForestClassifier(**params))]).fit(X_train, y_train)

mlp.score(X_test, y_test)
rf.score(X_test, y_test)


Mutate with custom function in R does not work I have a data frame, containing a column called: "Frequency". Frequency has values like "Year", "Week", "Month" etc. Now I want to create a new column based on the Frequency column where Year's new corresponding value will be 1, Month's corresponding value will be 12 and Week corresponding value will be 48. I tried to make a function for this as "getValue" and tried to make a new column applying a mutation (dplyr) on that funcion. But unfortunately I am getting the following warning and all values being transformed by the value of "1".

getValue <- function(input) {

          if (input == 'Year')
          {
            result <- 1
          }
          else if(input == 'Month')
          {
            result <- 12
          }
          else if(input == 'Week')
          {
            result <- 48
          }

          return(result)
        }

      Data =  mutate(gateway, YearlyHit = getValue(gateway$Frequency))
      Data

Here is the warning message I am getting -

Warning message:
In if (input == "Year") { :
  the condition has length > 1 and only the first element will be used

How can I achieve the required result in R?

Why don't you solve it in this way?

mutate(gateway, YearlyHit = case_when(Frequency == 'Year' ~ 1,
                                      Frequency == 'Month' ~ 12,
                                      Frequency == 'Week' ~ 48)
)


Which are the strategies to counter the 80/20 dilema in Data Science projects? Most of the time in Data Science projects is not spent in (performing) actual analytics but rather in other tasks, such as organizing data sources, collecting samples and preparing datasets, compiling and validating business rules in data, etc.This fact has been studied as the 80/20 dilemma in Data Science projects

In order to tackle this dilema, I would like to ask which are the strategies used to decrease the 80% of time spent in the other stages (organizing data sources, collecting samples and preparing datasets, compiling and validating business rules in data)

Some of these pain points are unavoidable. We are data scientists, we need lots of clean, relevant data and outside sources don't generally just hand that over. Someone on your team will have to do collection, organizing, cleaning, etc. The question then is, how do we make it as painless as possible?

Know your people. You presumably have a team, and I'm sure some people on it enjoy the analysis side of things while others prefer data engineering. Identify who is a best fit from both an enthusiasm and skill perspective and delegate accordingly. If you miss on this step, the work may get done, but it will be very slow compared to an environment where people are passionate about the problem they are solving.

Know your tools. How much data are you working with? If it's small enough to fit into the working memory on your machine and just run some Python/R scripts, then just do that. No need to be fancy when a simple solution will do. Just make sure to project into the future and verify your solution won't need to significantly scale up. If you're working with very large datasets or streaming data that won't play nicely with your local machine, then look into tech like Scala/Java running on a Spark cluster.

Know your process. Identify redundant tasks and cut them out as much as possible. Data science is an iterative process and the less time you spend repeating yourself the more you can focus on the actual analysis. That means no re-cleaning of data, updating existing models with more data rather than re-training, etc.


Why is my Python Altair chart printing a blank line? Wondering if anyone can help me understand why python altair chart is not printing...

I’m only seeing this when I run the code:

It literally isn’t outputting anything.

This is what my chart looks like converted to_dict; recommended for troubleshooting to ensure you have data.

Any ideas about why it’s not displaying a graph/chart/etc, and only printing a blank line?

Adding the head of the csv in pandas form.

The issue was enabling the vega module/extension in the notebook. Installing it is only half the battle, then you must enable it. :)

sudo jupyter-nbextension enable vega --py --user


Interpreting Categorical Crossentropy Loss I would like to ask for clarification about the loss values outputted during training using Categorical Crossentropy as the loss function. If I have 11 categories, and my loss is (for the sake of the argument) 2, does this mean that my model is on average 2 categories off the correct category, or is the loss used purely for comparative purposes and cannot be interpreted like I am suggesting ?

Cross-entropy is an information-theoretic measure about probability distributions, and it's measured in units that are determined by the base of the logarithm used in its computation (nats for the natural logarithm or bits for $log_2$). There are already several posts about intuitive understanding of cross entropy and its relationship to KL divergence (which are good to understand).
I think the more interesting part of your question is what "2 categories off" means with respect to your problem. Generally in classification optimized using categorical cross entropy, the classes are pairwise orthogonal: the content in the image is of a dog or a cat or a ship or whatever, so the ground-truth vector is always one-hot (1 in the index of the correct class, 0 elsewhere). Your question implicitly places an ordering over the classes: for something to be a number of "categories off", there must be some sense of similarity that makes categories $i$ and $j$ more alike than $i$ and $k$, say. If that's the case, you could look into alternative loss functions (but probably don't have to, depending on what you're doing).


Transforming negative correlated non linear variable to linear positive correlated variable At my office, I am stuck in a weird situation. I am asked to perform a regression algorithm on the data, in which the target variable is continuous having values range between 0.6 to 0.9 with 8 digits of precision after the decimal. Although I know and have applied many linear and non-linear regression algorithms in the past the case here is something different. There is one variable, which, according to my BU, should have a positive and linear correlation with the target variable. But when I ran Pearson's correlation, the variable is negatively correlated and by plotting a scatter plot I can see that the relationship is not linear at all.
What transformations can I perform on the variable so that it can show a positive correlation? I am fairly new to this problem so hoping to get it solved here. Thanks much everyone in advance.

What's a BU? Business Unit?

Can you tell anything about the variable and target variable? Can you include the scatter plot?

It is still not clear if the problem is not in the domain knowledge/assumptions.

A business example:
Profit depends on sales, which is generally assumed to have a positive correlation. But after a threshold marginal sales cost exceeds profits (very expensive to acquire new customers), we can still have a (local) negative correlation. If you only have data in the 'marginal' segment, you can draw incorrect conclusions.


How to use External Data Sets in test set I have a doubt regarding usage of external datasets like gdp rate, unemployment rate... etc., in test set for time series prediction. These datasets are historical and can be used along with train set, but how can it be used along with test set ? Do we need to predict it first and then use it along test set or are there any better ways to do it ?

The general idea is that you should use the same data on your train, test and prediction sets. For economical data this can be tricky :

In your train set, you need to get what data was available at a given time. This can be tricky because most of the time such economical data can be revised a long time after being first published. For example in my country, I think GDP can be revised up to 3 years after initial publication. Sor for exemple, if you have an instance in 2015 in your train / test set, you need to get what data was available in 2015, not revised numbers that could happen up to 2018.
In your test set, and for general prediction, the actual value can be an estimation of what is currently happening (nowcasting), so you have to make sure that you data source are consistent. Nowcasting techniques are sometimes used to get the current values (for exemple to get an idea of the activity of the hotel sector you can look at booking.com # of reservation ... etc.). Those techniques are entirely different from what is generaly used to get final numbers (agglomeration of fiscal declarations). Ideally you have to make sure the methods are consistent between, train, test and prediction sets. Using other methods to get data for prediction should not be intirely excluded, but should come with very strong warnings.


Can I specify the root node splitting feature in XGBoost? Just what the title says. Suppose I know the feature that I want to be used for splitting for the root node in a tree model in XGBoost; is there a way for me to tell XGBoost to split on this feature at the root node?

I do not have the whole answer, but I believe that xgboost allows warm-starting, which implies starting from a provided point.

xgb.train(…,xgb_model=model_to_start_from)


Standard Scaler drops accuracy significantly in Scala Spark I am working on Scala with Spark for a prediction model. I tried both Normalization and Standard Scaling and both of them drops my accuracy significantly.

Without the accuracy is ~90% (on training set), 77% (on testing set)

With Normalizer or Standard Scaler is 19% (on training), 0% (on testing)

I have the feeling that it's not related to the normalization per se, but more like a code bug.

Here is part of the code:

val labelIndexer = new StringIndexer()
  .setInputCol("label")
  .setOutputCol("indexedLabel")
  .fit(assemblerDF)

val featureIndexer = new VectorIndexer()
  .setInputCol("features")
  .setOutputCol("indexedFeatures")
  .setMaxCategories(4) //if column has more than 4 categories, then use it as continious instead
  .fit(assemblerDF)

val labelConverter = new IndexToString()
  .setInputCol("prediction")
  .setOutputCol("predictedLabel")
  .setLabels(labelIndexer.labels)

val scaler = new StandardScaler()
  .setInputCol("indexedFeatures")
  .setOutputCol("scaledFeatures")
  .setWithStd(true)
  .setWithMean(false)

val ml = new LogisticRegression()
  .setMaxIter(15)
  .setFeaturesCol("scaledFeatures")
  .setLabelCol("indexedLabel")

val pipeline = new Pipeline().setStages(Array(labelIndexer, featureIndexer, scaler, ml, labelConverter))

Where then I pass the Pipeline into a CrossValidator. Any idea what is going wrong here?

As @n1k31t4 said in the comments, it's the .setWithMean(false) call (should be true).

Moreover, you don't really need the StandardScaler at all, since LogisticRegression will perform standardization by default (see the standardization param).


What is required in Affinity Propagation I am hoping to use affinity propagation to cluster my data using sklearn. But I came across a question whether to use a distance matrix or similarity matrix in the fit method.

Please let me know what is suitable to use?

According to sklearn.cluster.AP (in case in AP chosen affinity = "precomputed") the method "fit" requires X  where X is matrix of similarities / affinities, that is, if distance matrix is estimated, use as:

X $ = matrix\ of\ similarities = -distance\ matrix$.


Difference between CBOW and SKIP Gram word vectors I have gone through several links but was not able to understand how CBOW and Skip Gram is trained from scratch?
Any good link/blogs or books would be very helpful.
Word2Vec - CBOW and Skip-Grams
What's the difference between Skipgram word2vec and CBOW word2vec during training and when to use CBOW Skip-gram. ?
Example or Application where CBOW would be preferable choice but not Skip-gram and vice versa.

Here is a example for what @Christos Karatsalos is describing:
If you take a statement like "The cat jumped over the puddle",
In CBOW's case, the neural network's word embeddings are trained by passing a input set of words to the neural network and making the network to predict an output word as shown below:
$\{ cat, jumped, over, the, puddle\} \rightarrow  The $
$\{ The, jumped, over, the, puddle\} \rightarrow  cat $
$\{ The, cat, over, the, puddle\} \rightarrow  jumped $
$\{ cat, jumped, the, puddle\} \rightarrow  over $
$\{ The, cat, jumped, over, puddle\} \rightarrow  the $
$\{ The, cat, jumped, over, the\} \rightarrow  puddle $
In skip-gram's case, the neural network's word embeddings are trained by passing an input word to the neural network and teaching the network to predict a set of words as shown below:
$The \rightarrow  \{ cat, jumped, over, the, puddle\} $
$cat \rightarrow \{ The, jumped, over, the, puddle\}$
$jumped \rightarrow  \{ The, cat, over, the, puddle\}$
$over \rightarrow  \{ cat, jumped, the, puddle\}$
$the \rightarrow  \{ The, cat, jumped, over, puddle\}$
$puddle \rightarrow \{ The, cat, jumped, over, the\}$
For usage, if you are thinking of a fill-in-the-blanks kind of problem CBOW Word2Vec would be a suitable vector to use, on the other hand if you have a word and you are trying to come up with a new sentence with it then   skip-gram Word2Vec will be useful.
Take a look this lecture notes for more information.


How flexible is the link between objective function and output layer activation function? It seems standard in many neural network packages to pair up the objective  function to be minimised with the activation function in the output layer.

For instance, for a linear output layer used for regression it is standard (and often only choice) to have a squared error objective function. Another usual pairing is logistic output and log loss (or cross-entropy). And yet another is softmax and multi log loss.

Using notation, $z$ for pre-activation value (sum of weights times activations from previous layer), $a$ for activation, $y$ for ground truth used for training, $i$ for index of output neuron.

Linear activation $a_i=z_i$ goes with squared error $\frac{1}{2} \sum\limits_{\forall i} (y_i-a_i)^2$
Sigmoid activation $a_i = \frac{1}{1+e^{-z_i}}$ goes with logloss/cross-entropy objective $-\sum\limits_{\forall i} (y_i*log(a_i) + (1-y_i)*log(1-a_i))$
Softmax activation $a_i = \frac{e^{z_i}}{\sum_{\forall j} e^{z_j}}$ goes with multiclass logloss objective $-\sum\limits_{\forall i} (y_i*log(a_i))$

Those are the ones I know, and I expect there are many that I still haven't heard of.

It seems that log loss would only work and be numerically stable when the output and targets are in range [0,1]. So it may not make sense to try linear output layer with a logloss objective function. Unless there is a more general logloss function that can cope with values of $y$ that are outside of the range?

However, it doesn't seem quite so bad to try sigmoid output with a squared error objective. It should be stable and converge at least.

I understand that some of the design behind these pairings is that it makes the formula for $\frac{\delta E}{\delta z}$ - where $E$ is the value of the objective function - easy for back propagation. But it should still be possible to find that derivative using other pairings. Also, there are many other activation functions that are not commonly seen in output layers, but feasibly could be, such as tanh, and where it is not clear what objective function could be applied.

Are there any situations when designing the architecture of a neural network, that you would or should use "non-standard" pairings of output activation and objective functions?

It is not so much which activation function which you use which determines which loss funtion you should use, but rather what the interpretation you have of the output is.

If the output is supposed to be a probability, then log-loss is the way to go.

If the output is a generic value then mean squared error is default way to go. So for example, if your output was a grey scale pixel with grey-scale labelled by a number from 0 to 1, it might make sense to use a sigmoid activation function with a mean squared error objective function.


Build knowledge bot using deep learning I'm going through this chatbot example, which uses the Cornell movie dialog corpus. Expanding this example, is it possible to build a "knowledge bot" (ie) a bot that can chat and be knowledgeable in a specific domain? So for example, I use the Bible to train the system. Then it should be able to answer questions on Bible. Or I use the wikia content of Star Wars and be able to answer all the questions related to Star Wars. Is such a system even possible, or is it still in the realm of scifi?

It is definitely possible, but obviously to a certain extent. What you need, is a sequence to sequence model trained on questions and answers data of a domain. Denny Britz has this amazing blog post on impelementing a  retreival based chatbot trained on ubuntu dialog corpus using tensorflow. Go through the post to understand the difference between retreival based and generative model chatbot. The efficiency and efficacy of such a bot, is another question all together and rigorous research is going on to improve such bots. IBM watson does it pretty well as mentioned in the comments.


Is there any market for data science consulting Is there any market for data science as service which provides service without selling software, just consulting, modelling and developing algorithms?

Yes. If there wasn't a market, the largest consulting ("Big 3") and audit ("Big 4") firms wouldn't offer and continue to be expanding their advisory services in data & analytics. The same could be said for the thousands of independent consultants who offer such services (such as those who can be found on www.upwork.com).


Regression model for a count proces In R I have data where head(data) gives

day   count   promotion
1        33        20.8
2        23        17.1
3        19         1.6
4        37        20.8

Now day is simply the day (and is in order). promotion is the promotion-value for the day. It is simply the number of times an advertisement has been on television. count is the number of new users we got that day.

I want to investigate the impact the promotion-value has on new users (count). Since we have a count process I thought it would be best to make a poisson regression model.

model=glm(formula= data$count ~ data$promotion, data=data)

When we type summary(model) we get

Coefficients:
           (Intercept)              good_users$promotion
              13.40216                   0.24342

Degrees of Freedom: 793 Total (i.e. Null);  792 Residual
Null Deviance:      9484
Residual Deviance: 9325     AIC: 12680

Here is a plot of the data.

But when I plot the fitted values for the model

points(model$promotion, model$fitted, col="blue")

we get this

Here is another plot that shows the same but where days with 0 promotion are removed.

How should I chose my regression model (should I use lm instead of glm) or is the another better approach to solve this? Because the data is not pretty but more random like this what should one do ?

Updated

Finding the sweet spot

I have done the following for finding a sweet spot.
I divide data into 10 groups. group1 is simply a subset where the promotion-value is within 1:10. group2 is data where the promotion-value is between 11:20, and so on for the other groups. So in R we have

group1 <- subset(data, data$promotion %in% 1:10)
group2 <- subset(data, data$promotion %in% 11:20)
group3 <- subset(data, data$promotion %in% 21:30)
...
group10 <- subset(data, data$promotion %in% 91:100)

Now I can use wilcox.test to test if there is a significantly difference between the groups by typing

wilcox.test(group2, group1, alternative="greater")

which gives a low p-value, ie group2 has significant higher new_good_users than group1. The same goes for

wilcox.test(group3, group2, alternative="greater")

but for     wilcox.test(group4, group3, alternative="greater")
I get a p-value at 0.20, ie there is no significant difference in new_good_users between group4 and group3. And the same goes for the rest of the group-pairs up to 10.

So this must mean that if we increase promotion in the first groups we have an increase in new_good_users but in the last groups we do not have that increase. This means that we have a sweet spot at group3 where the promotion-value is 21:30. Is this not correct ?

I have to quote Tukey, perhaps the grandfather of data science:

  The combination of some data and an aching desire for an answer does
  not ensure that a reasonable answer can be extracted from a given body
  of data.

I see nothing wrong with your Poisson model. In fact its a pretty good fit to the data. The data is noisy. There is nothing you can do about it. Perhaps the noise if due to whatever else is on TV at the time, or the weather, or the phase of the moon. Whatever it is, its not in your data.

If you reasonably think the weather might be affecting your data, get the weather data and add it. If it decreases the log-likelihood enough for each degree of freedom then it's doing a good job and you leave it in. This is regression modelling 101.

Of course there's a zillion other things you can do. Scale the data by any old transformation you want. Fit a quadratic. A quartic. A quintic. A spline.  You could include the date and possible temporal correlation effects. But always bear in mind what Tukey was saying - if your data is noisy, you won't get anything much out of it. So it goes.


How to predict specific user from session logs? Let's say I have a dataset with 800 rows (40 entries for each of 20 users). The entries are user session logs (columns are - browser, os, time, date etc for a specific session).

Now each user has unique id (1-20). Let's say user_id=1 is special one and I need to detect it whenever new data comes.

So for new data I need to predict whether that session is of user_id=1 or not.

My question is: how can I do that?

One way I thought of is to make a feature representation for each of the 20 users and whenever new data comes in, take the distance of the data from each of feature and see the minimum distance.

However, the problem is that when I make a unique feature representation for each user, how do I deal with the columns like browser, os - because a user can have used multiple browsers in all those 40 sessions?

There are certainly different ways to approach the problem, I am giving you what I think.

The main key is to create as many as features as you can, well it depends on what algorithm you may end up using, but still more features won't hurt. After all you want to differentiate each user, so your feature space should be rich enough to be able to distinct them from each other. I think you are on the right track. For example, there are many features can be extracted from date (weekday, weekend, month etc.), or the browser you mentioned, if it is multiple browser you can create a new feature num_broswers (count number of distinct browser that user uses). You better know your data. Are sessions equally spaced on time or not? For instance, could be that user_one exits a session faster than user_two? Then total time spent for each user on each session could be another feature (engagement factor). You mentioned time in your columns, but I would not know what it is. You can even take time_of_day as another feature (morning, mid-day, afternoon, evening etc.).

You need also to clarify in your question whether number of users are fixed for a specific period of time, or constantly changing?

After addressing key points about data and features, I am thinking of two algorithmic approaches:

Similarity-based Method:
This is exactly what you mentioned, and it is a valid approach and perhaps the fastest and most straightforward one. As mentioned above, you can make new features like num_browsers for multiple browsers entries for a user in those 40 sessions, which was your major concern. I am more concerned that naturally you may have a mix of numerical and categorical features, and I would not be very sure if normal KNN (Euclidean metric) would work! Doesn't hurt trying though after encoding those categorical features or normalizing features are not continuous, but check this post, or another one, or at least be aware of pitfalls!!
Classification-based Method:
You could even turn this into a Multiclass Classification problem, i.e. classifying which user the new incoming user is based on features. The challenge is that the quality of your classifier strongly depends on your feature space. The choice of the algorithms does not matter (even kNN again can be used, but perhaps Decision Trees i.e. RandomForest would be better). If you won't have a good set of distinctive features, classifier can not do magic! This could be also true about kNN, but since kNN doesn't need training, you may get more reasonable results than a classifier.

P.S. This method would be more suitable "if number of users are fixed". Still it would be possible to train a classifier if number of users are NOT fixed. It is ONLY then you need to re-train whenever you have a new user or user_ids are changing, otherwise your algorithm have not seen such user and likely it is going to miss it!

I would personally go with the Similarity-based Method and try to enhance my features as much as I can.


How to cluster a link traversal dataset I'm using Google Analytics on my mobile app to see how different users use the app. I draw a path based on the pages they move to. Given a list of paths for say a 100 users, how do I go about clustering the users. Which algorithm to use? By the way, I'm thinking of using sckit learn package for the implementation.

My dataset (in csv) would look like this :

DeviceID,Pageid,Time_spent_on_Page,Transition.

ABC,Page1, 3s, 1->2.

ABC,Page2, 2s, 2->4.

ABC,Page4,1s,4->1.

So the path, here is 1->2->4->1, where 1,2,4 are Pageids.

@Shagun's answer is right actually. I just expand it!

There are 2 different approaches to your problem:

Graph Approach

As stated in @Shagun's answer you have a weighted directed graph and you want to cluster the paths. I mention again because it's important to know that your problem is not a Graph Clustering or Community Detection problem where vertices are clustered!
Cunstructing a Graph in networkx using the last two column of the data, you can add time spent as weight and users who passed that link as an edge attribute. After all you'll have different features for clustering: the set of all vertices an individual ever met in the graph, total, mean and std of time spent, shortest path distribution parameters, ... which can be used for clustering the user behaviors.

Standard Data

All above can be done by reading data efficiently in a matrix. If you consider each edge for a specified user as a single row (i.e. you'll have MxN rows where M is the number of users and N the number of edges in case you stick with 100 case!) and add properties as columns you'll probably able to cluster behaviors. if a user passed an edge n times, in the row corresponding to that user and that edge add a count column with value n and same for time spend, etc. Starting and ending edges are also informative. Be careful that node names are categorical variables.

Regarding clustering algorithms you can find enough if you have a quick look at SKlearn.

Hope it helped.
Good Luck :)


Calculate feature weight vector for one-hot-encoded data frame in R I have the following data frame with one categorical and two numerical columns:

    V1  V2  V3
1   A   1   3
2   A   3   5
3   B   3   3
4   C   2   3

I have turned this into the following dummy variables:

    V1.1  V1.2  V1.3 V2  V3
1   1     0     0    1   3
2   1     0     0    3   5
3   0     1     0    3   3
4   0     0     1    2   3

Now, I want to apply clustering to this latter set. I guess that I could get better results if I downweight the dummy variable columns (proportionally to their number) because with equal weights the distance based clusters will be distorted.

My question is, how could I get the following weight vector from the new data set:

0.33, 0.33, 0.33, 1, 1

The information you need is in the assign attribute on the matrix returned by model.matrix. Something like this seems to work for me:

data <- data.frame(V1 = factor(c("A", "A", "B", "C")), V2 = c(1, 3, 3, 2), V3 = c(3, 5, 3, 3))
model <- model.matrix(~ . + 0, data = data)
assignments <- attr(model, "assign")
counts <- table(assignments)
weights <- sapply(assignments, function (x) { 1 / counts[[x]] })

weights is now (0.3333333, 0.3333333, 0.3333333, 1.0000000, 1.0000000).


Machine-learning algorithm / library for outcome prediction I am extremely new to this data science world so bear with me if my question is not very clear, I'd be glad to clarify. What I am looking for is simple: train a program with a set of values (5 ordered integer inputs, 1 boolean output). Then I would give it 5 inputs where the outcome is not known, and it has to tell me the outcome.

It is a basic classification model. Our focus is to train the model using training dataset and evaluate its performance with the test set.

In your case, assume you have 10 numbers as input and a binary output.
Consider random 8 entries as your training data and train it with any classification algorithm using R/Python. Then test it on the remaining entries which acts as a test data.

Since it is associated with classification we need to consider performance metrics like accuracy, precision and recall based on the labeled feature.

For more details on classification try here


Why can't I specify the correct NumPy size? In the network (model of Keras, Sequential), the input layer must have 4 neurons. The input must be 1 list, the length of which is 4, each element is a number.
print("SHAPE:", np.array([1, 1, 1, 1]).shape)

self.model.fit(np.array([1, 1, 1, 1]),
self.rightAnswer,
epochs = 1,
batch_size = 1)

Here is the conclusion:
SHAPE: (4,)

ValueError: Error when checking input: expected dense_1_input to have shape (4,) but got array with shape (1,)

Why is this happening, and how can I fix it?

Keras needs you to pass one more dimension than it says in the error message: the batch dimension.
I.e.
If you have a model that requires each sample to have an input shape of (4,) and you have 1000 training samples you need to feed it with an array of (1000, 4).
In your case since you want to feed it with just one sample you need to pass a shape of (1, 4). Try self.model.fit(np.array([[1, 1, 1, 1]])) (notice the one extra bracket with makes the array's shape be (1, 4)) .


Standardization, Oversampling and Evaluation Please help me to understand which of the following is correct and why.

X_train, X_test : Split the training set into training and test sets
X_train_std : Standardize X_train
X_train_res : Oversample X_train_std
model.fit : Train the model using X_train_res
X_test_std : Standardize X_test using the parameters calculated in 2)
model.evaluate : Evaluate the model using X_test_std

Or:

X_train, X_test : Split the training set into training and test sets
X_train_std : Standardize X_train
X_train_res : Oversample X_train_std
X_train_res_std : Standardize X_train_res
model.fit : Train the model using X_train_res_std
X_test_std : Standardize X_test using the parameters calculated in 4)
model.evaluate : Evaluate the model using X_test_std

Let's discuss the second variant:

If a standardization is considered as a function f(data) and the second standardization after oversampling is considered as a function g(f(data)), then you should use the same transformation for your test data. However, in the second option you just implement the function g(data) without transformation the data using function f().

Therefor, it seems to me, that the first variant is better. Moreover, it is enough to standardize data just once, so the first option is ok.


optical chemical structure recognition from images I want to recognize name of the chemical structure from the image of chemical structure, like in the given image it is benzene structure I want to recognize that it is benzene from the image(I should be able to recognize all these structures as benzene) What way I could follow to achieve this? (like this given any structure I want to recognize the name of it)  .

What way I could follow to achieve this?

Create a dataset with image-label pairs.
Create a classification model.

I don't think you will find any algorithm "out-of-the-box" for you to use.
You might have a look at some paper to see the kind of approaches they have.

I would recommend you look at these:

https://arxiv.org/ftp/arxiv/papers/1802/1802.04903.pdf which uses convolution neural nets and a heatmap as an attention mechanism. It even deals with noisy images.
https://web.stanford.edu/class/cs231a/prev_projects_2016/optical_recognition_of_chemical_structures.pdf which tries to identify hand-drawn structures. It uses a sort of edge detection algorithm for recreating the hand-drawn structures.


Error opening Iris.tab file in Orange load data widget I am attempting to load in the Iris.tab data set into the Orange load data widget. However, when I do this, I get the following error message (shown above).
This keeps popping up when I start up Orange Data Mining and try to load the iris.tab data file. It used to work before, but now it is showing this error.
Does anyone have any idea what why this might be happening and what might be causing this error?
Thanks!

This was an error in Orange 3.3.11, and fixed in release 3.3.12. Update to the newest release to get rid of the error.


Role of Image Resolution in Deep Learning I have multiple image datasets about the same topic that I want to use for a classification task utilizing Deep Learning. The datasets differ in the resolution of images (i.e. some pictures are 128x128px, some 512x512, others 2048x2048).
If I used the dataset with the highest resolution for training my intuition would be that it'll be harder to classify images with a lower resolution as the network learns patterns that it may not recognize in lower resolution pictures. On the other hand, if I use the low resolution dataset for training, the patterns learnt are more crude and it performs better on any new data as higher resolution images can be scaled down easily. Is my intuition right or am I missing something? What would be the best approach in my case for selecting proper training data?

The answer may depend on what kind of information you want to extract from the images. However, the general approach to the problem is to find a perfect balance so that your image is not too small which is hard to extract too much information or it is not high-resolution input which will unnecessarily complicate your model. The latter will also be hard to train in terms of space complexity and time complexity.
Thus, if your objective is not something like identifying and classifying minuscule objects in the image, or a similar detailed and complex task, then you can use the small size of images.
Having good architecture and a well-trained model can ensure your powerful outcome. Consider that most of the famous and powerful NN models in Computer Vision and Image Processing fields are using input sizes like 96x96, 128x128, 224x224, 256x256. Maybe I am going too further by saying this, but, the good challenge would be building a powerful model with a small size of pixels such as 224x224 or close to this because in that case, the model's usability and usefulness will be scaled up. As you also mentioned, high-quality images can be downscaled to low scales but it is not possible to enlarge the small-sized image (at least, without the help of AI). As your model reasonably small, it will address that many of the all available images (also, detecting or classifying cropped images, small objects in the images, etc. can be considered another reason for having a small input size). So if you build a model using a large pixel size, will also require a high-quality image to be able to work and would not accept on low-quality images. Consider that google vision API works superb even with the 64x64 images.
Briefly, if your aim is not unusual and it does not necessitate high-resolution images to work with small-sized objects in the images, then use small-sized images. With the help of good architectured convolution layers, you can extract lots of information from those small images and processes in the next layers. Also, consider that even small changes in the input image size may affect the training time drastically.


Can we use difference-in-differences with a biased A/B test? We noticed we had a biased sample in our A/B test and was wondering if difference-in-differences would help us make valid conclusions about the data, or if there was another way to proceed.

We ran an new experiment on our site, where we offered 50% of our users a new feature. We assigned users with odd ids into the experiment group and users with even ids into the control group and then ran the experiment. However, we saw that even prior to running the experiment, there was a statistically difference between the two groups. We think this is because we ran many experiments where we segmented based on odd/even of the id, so people in the experiment group have seen many treatments.

So lesson learned, we'll flip a coin next time instead of using the id. However, we'd like to see if we can still make inferences from the current experiment. I've heard of something called difference-in-differences. Would this work in this case, or is there a different approach that would work better? Ideally, we don't want to scrap the test and start over since 50% of our users have already seen the new feature.

Why are you hesitant to start over? A/B testing is double-edged sword because it is a relatively straightforward exercise (as these things go) but it has the power to completely pivot your business in a very short amount of time. Plus keep in mind that the thresholds you are seeking to surpass can be "tempermental" (for the lack of a better word) where a z-value of (for example) 1.92 can't be, "oh, that's close enough . . . ".

As a data scientist, I would never want to sign my name to an experiment that I didn't have complete confidence in both the design and execution. If there is anything that can affect your final scores, you should absolutely consider scrapping it and starting over.


Find the furthest pages of a website I would like to find the two furthest pages of a big websites (+5millions pages).

(It's the degree of separation, the most links between the two pages)

This is a classic "graph database" question. I would do the following steps:-

Traverse all the pages using a webcrawler, or if you can interrogate the website server directly that may be qukcker.
Ensure you keep all links between pages, for later use.
Put the web pages into a neo4j database with connections between the pages

This will allow you to be able to do the query of pages with the maximum number of steps, or pages most frequently linked to, etc.


What is the better to pre-process data, MS-Excel or Python/R While handling data, many anomalies occur that might change the results. For example, outliers, missing values, etc. So in this context, what is the better tool to pre process data? A visual tool like MS-Excel or programming tool like R or Python

I mean most of the time it's very much about preference really.

If you want any indication on when to use what:

MS-Excel can only handle so much data. Good for quick analysis of small data but neither good for production or middle to large amounts of data
R great tool for calculating stuff, running models, etc. Pre-Processing is good, lots of libraries to use. Some people don't like using R for production pipelines but I know companies where this is working just fine
Python is a great overall language for pre-processing and creating production worthy data pipelines. So if you want to have a scalable, reusable pre prcessing pipeline this is certainly one of the ways to go

Of course Excel has the lowest learning curve compared to R and Python. The latter ones are similar in that.


Prerequisites for Data Science I'm a Java developer and I want to pursue career in Data Science and machine learning.

Please advise me where and how to begin? What subjects and mathematical/statistical skills are required and so on?

I would say that trying out the resources online such as the Explore Data Science course and Andrew Ng's ML Course on Coursera (mentioned in the notes above) are great. However, nothing beats true data that you understand inside and out. After picking up some skills from Explore Data Science, I propose you collect your fuel usage data continuously. Collect the basics: city, state, gallons, total cost, and your vehicles mileage. When you forget to enter it, then you have to learn how to work with missing data. As you feel more comfortable, enter in kaggle contests. The data science community is growing so quickly that the online resources are vast and will help you identify any areas where a text book is what you need. Good Luck!! Have fun!


In regression, is a higher adjusted R-Squared ALWAYS better? Bit of a strange question...I'm trying to put together a multiple regression model.  I have daily temperature as one of my control variables.  I really want to control for temperature, but interestingly, removing it seems to very slightly increase my adjusted-R squared (from 0.23 to 0.24 or similar).

I know theoretically, I should just remove it right, as it adds nothing to my model?  But given that the increase is so small and I quite like controlling for it in my model, would it be totally wrong to keep it in?  If I take it out, am I still "controlling" for temperature in my model?

Generally, a higher adj. R-square is better. In your case, you might be better off working on the representation of temperature in the model. It depends on the data, really, but you could try polynomials for temperature (squared term or so) or you could make „classes“. Plot temperature against y, to see if there is a non-linear trend in the data. If so, you need to try to represent this structure in your model, because OLS regression is parametric by definition.

Adjusted R-square balances the advantage of adding a new variable and the disadvantage of losing degrees of freedom. So in your case, the explanatory power of temperature seems to add little to your model. This may change if you change the representation of the variable, so that it better fits the data generating process. You need to try.


Problem with converting string to dummy variables I'm new in data science, I have data which want to work on it, I omitted extra columns and convert it to 4 columns ( Product, Date, Market, Demand ) . in this data Product and Market are string, I know for working on this data must convert them. I want to convert the string to dummy variables but this isn't logical because I have 64 fruits in the product column.
I am confused and I don't know what can I do whit this strings.

You can use label encoders for your product and market. however one hot encoding will be the best one to suit for converting string to numbers to feed into an algorithm  . I suggest looking at this link for further details :- https://towardsdatascience.com/choosing-the-right-encoding-method-label-vs-onehot-encoder-a4434493149b


Custom loss function with additional parameter in Keras I'm looking for a way to create a loss function that looks like this:
The function should then maximize for the reward. Is this possible to achieve in Keras?
Any suggestions how this can be achieved are highly appreciated.

def special_loss_function(y_true, y_pred, reward_if_correct, punishment_if_false):
    loss = if binary classification is correct apply reward for that training item in accordance with the weight
           if binary classification is wrong, apply punishment for that training item in accordance with the weight
                    )
    return K.mean(loss, axis=-1)

The approach I've been looking at for my example is to pass in the weights along with y_true and then cut the tensor into two, separating out the weights and the y_true as seen below. Would an approach like this be possible at all or would this interfere with the normalization process etc?

def decompose_y_true(y_true_and_weights):
    y_true = y_true_and_weights[:,1]
    weights= y_true_and_weights[:,1:]
    return y_true, weights

def custom_loss(y_true_and_weights, y_pred):
    y_true, y_weights = decompose_y_true(y_true_and_weights)
     loss = # some loss operation
    return K.mean(loss, axis=-1)

Much more elegant would be if I could pass in my weights over the sample_weights parameter in the fit function, but it seems there are some limits what shape those weights can have, and also there's no way to retrieve them within the loss function as far as I can tell. Or is there any way to pass this into the loss function somehow so I can operate on them from there?

You can write a function that returns another function, as is done here on GitHub

def penalized_loss(noise):
    def loss(y_true, y_pred):
        return K.mean(K.square(y_pred - y_true) - K.square(y_true - noise), axis=-1)
    return loss

input1 = Input(batch_shape=(batch_size, timesteps, features))
lstm =  LSTM(features, stateful=True, return_sequences=True)(input1)
output1 = TimeDistributed(Dense(features, activation='sigmoid'))(lstm)
output2 = TimeDistributed(Dense(features, activation='sigmoid'))(lstm)
model = Model(input=[input1], output=[output1, output2])
model.compile(loss=[penalized_loss(noise=output2),
penalized_loss(noise=output1)], optimizer='rmsprop')

In the wrapper function you can pass scalars or keras tensors like additional inputs. In your case you would add the weights in the top wrapper function and reference them in your inward function.


Finding similar data columns with machine learning I'm trying to find similar data columns (text and numerical) in other tables by using classification in machine learning. What are some good features that can represent a column for my feature space ?

If I'm not mistaken, you are looking for some metrics to investigate if two columns of values represent similar information. Right?

When the order of numbers is fixed for both columns (such that the ith elements of both columns relate to the same record) and hence both columns have the same length, then you can use a wide range of techniques to measure the similarity/distance between the two vectors such as cosine similarity, Pearson correlation coefficient, mutual information, etc.

If the order of numbers in the columns are different or they basically refer to different concepts, then the above techniques are meaningless. In this case, you can look at these columns as two populations of samples. In this case, you may compare their statistical distributions using their distribution parameters (eg mean and std for a normal distribution) or statistical tests (eg ANOVA). In some cases, the number of items (population size) can be different.


Clustering based on features of varied importance Suppose I have a dataset that includes the following features {HairColor, EyeColor, EducationLevel, Income}. I would like to perform clustering to separate the dataset into smaller datasets that you would expect to behave similarly.  The difficulty that arises is that it is clear that EducationLevel and Income are much more important than HairColor and EyeColor but I do not know how to measure that importance for the sake of clustering.
In the example below, I would want it to be clear that Row 1 is more similar to Row 3, than to Row 2.

ID
EyeColor
HairColor
EducationLevel
Income

1
1
1
1
1

2
1
1
2
2

3
2
2
1
1

If education level and income are more important than other features, you can multiply those features by a factor (greater than 1). That will allow the  clustering algo to focus more on those features than the rest.
For larger datasets, with features where the differences are not so obvious, you will need to rely on your judgement based on the end objective of the clustering. If you have a target variable in mind, you want to choose or boost those features that are highly correlated with the target


Handling underflow in a Gaussian Naive Bayes classifier I am implementing a Gaussian Naive Bayes classifier (so each feature is continuous and assumed to be coming from a Gaussian distribution).  When evaluating the probability of a feature value in the test set, if the value is sufficiently far away from the mean (e.g. the mean and s.d. on the training data is say 0 and 1 but the test value is 10^10) then there is underflow.  This is an issue because then the probability will be calculated as 0.0 so the log probability is undefined.  Is there a standard way of handling underflow in this case?

The standard answer is to work in log space, and manipulate the log of probabilities instead of probabilities, for exactly this reason. This classifier involves products of probabilities which just become sums of log probabilities.

You allude to that already, but the problem you suggest isn't a problem. Internally you don't calculate a probability and then take the log again. It stays in log space. So for very small P, log P is a large negative number while P itself may underflow to 0.0. But you may never need to evaluate P internally.


Which correlation test can be used on low count contigency table? I need to find the correlation between the ratings given by two parties.

Let's say we have 2 coaches, namely A and B.

Coach A will rate the skills of the 12 players, either "Strong", "Moderate" or "Weak".
Coach B will rate the same 12 players.

If Coach A rated all the players as "Strong", and Coach B rated 10 players as "Strong" and 2 players as "Weak".

I have tried putting the results into a contingency table form and ran Somers D test. However the Somers D shows 0% which does not truly representing the correlation of the ratings. What test could we carry out to find the correlation between the ratings of the coaches?

I don't think it's worth doing any kind of significance test for such a small sample.

option 1: pure categorical values

You could just count the number of identical ratings between the two ratings and divide by the total number of players

option 2: take the order into account: weak < moderate < strong

Define a similarity measure such as:

sim(X, X) = 1
sim(weak,moderate) = 0.5
sim(moderate,strong) = 0.5
sim(weak,strong) = 0

Then use the same process as option 1 except that for every player you add sim(ratingA, ratingB), and divide by the total. This will also give you a normalized score between 0 and 1 which is a bit more precise (the value will be higher or equal than in option 1).


InvalidArgumentError for placeholder I'm copying an example directly out of a book I am working through, and I currently getting this error:

InvalidArgumentError: You must feed a value for placeholder tensor 'Placeholder_10' with dtype float and shape [?,2]
 [[{{node Placeholder_10}} = Placeholder[dtype=DT_FLOAT, shape=[?,2], _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

My tensors and placeholders are declared like so:

XOR_X = [[0.0, 0.0], [0.0, 1.0], [1.0, 0.0], [1.0, 1.0]]
XOR_Y = [[0.0], [1.0], [1.0], [0.0]]

num_input = 2
num_classes = 1

x_ = tf.placeholder("float", shape=[None, num_input], name='X')
y_ = tf.placeholder("float", shape=[None, num_classes], name='Y')

And the line producing the error is here:

_, summary = sess.run([train_step, merged_summary_op], feed_dict={x_ : XOR_X, y_: XOR_Y})

What exactly is the issue here?

for context, I attach the entire implementation here:

import tensorflow as tf

XOR_X = [[0.0, 0.0], [0.0, 1.0], [1.0, 0.0], [1.0, 1.0]]
XOR_Y = [[0], [1], [1], [0]]
# XOR_Y = [0.0, 1.0, 1.0, 0.0]

num_input = 2
num_classes = 1

x_ = tf.placeholder("float", shape=[None, num_input], name='X')
y_ = tf.placeholder("float", shape=[None, num_classes], name='Y')

#Model structure
H1 = tf.layers.dense(inputs=x_, units=4, activation=tf.nn.sigmoid)
H2 = tf.layers.dense(inputs=H1, units=8, activation=tf.nn.sigmoid)
H_OUT = tf.layers.dense(inputs=H2, units=num_classes, activation=tf.nn.sigmoid)

#Define cost function
with tf.name_scope("cost") as scope:
    cost = tf.losses.log_loss( labels=y_, predictions=H_OUT)
    # Add loss to tensorboard
    tf.summary.scalar("log_loss", cost)

with tf.name_scope("train") as scope:
    train_step = tf.train.GradientDescentOptimizer(0.1).minimize(cost)

merged_summary_op = tf.summary.merge_all()

# Initialize variables(weights) and session
init = tf.global_variables_initializer()
sess = tf.Session()

# Configure summary to output at given directory
writer = tf.summary.FileWriter("./logs/xor_logs", sess.graph)
sess.run(init)

# Train loop
for step in range(10000):
    # Run train_step and merge summary op
    _, summary = sess.run([train_step, merged_summary_op], feed_dict={x_ : XOR_X, y_: XOR_Y})
    if step % 1000 == 0:
        print("Step/Epoch: {}, Loss: {}".format(step, sess.run(cost, feed_dict={x_ : XOR_X, y_: XOR_Y})))
        writer.add_summary(summary, step)

Try using the number of sample instead of None in the x_ placeholder. Also, try making these changes.

n_samples = len( XOR_X )
x_ = tf.placeholder(dtype=tf.float32 , shape=[n_samples, num_input], name='X')


Why is the number of samples smaller than the number of values in my decision tree? I'm using scikit-learn RandomForestClassifier for a classification problem. When taking a closer look at one of the trees I noticed that the number of samples at the root was 662, but there were 507 instances of the first class and 545 of the second. What's going on or did I understand something wrong? Is the number of samples actually the number of unique samples and since I used bootstrap aggregation there are many samples that were chosen multiple times?

Yes, it seems to display unique samples, the others have been duplicated by the bootstrap sampling.

There's the 0.632 rule - when you have N items and you take a random sample of size N with replacement (as bootstrap does), you only get 63.2% of the samples from N, the rest are duplicates.

That roughly matches what you've seen: 0.632 * (507+545) = 665 unique samples.

You can also try it with some Python code:

samples = np.arange(507 + 545)

bootstrap_sample = np.random.choice(samples, size=len(samples), replace=True)

print(len(np.unique(bootstrap_sample)))

This always prints values closely around 665.


Why does increasing the training set size not improve the results? I have trained a model on a training set, which is not that big (overall around 120 true positives, and of course lots of negative examples). What I am trying to do is to improve the results by increasing the data size. I tried two approaches:

I added data from a different domain and concatenated the data with the existing one. It increased the F-score from 0.13 to 0.14.
I added the same extra data instances, but this time with a domain adaptation technique (feature augmentation). This time, I got an improvement of around 6 percent which was significant.

My question is why does the first approach help a little bit?

The additional data set was around 10 times bigger than the main data set. That is, adding around 1000 instances from a different domain increased the results 1 percent. If it didn't help at all, I think, that would be easier to explain. But now, I don't understand. If the data is biased and test set and training set are so close to each other such that the additional data set can not help, why do I get 1 percent improvement?

On the other hand, how do I get almost the same result when the main data set in this situation is just around 1/10 of the data? I don't understand the behaviour of the model. I am using a Naive Bayse classifier.

The model that you are generating is most probably under-fitting the model. Imagine, you are fitting a straight line to a non-linear data. Certainly even if you are increasing your sample size, the model is still not getting any better.

Study about learning curves and how they can be used to solve the under-fitting!


Predicting positive/negative experience with very few labels and labels from only one class I have video viewing data (length of session, nb of videos, etc), as well as if the user clicked on the like button. We can use the like button as a confirmation that the user had a positive viewing experience, however, only 0.1% of users click on this button.
I would like to find a way to find users that have a similar data to those who liked the video without having them explicitly click the like button.

I thought about having the like variable be the response variable in a binary classification problem, however, not liking the video does not mean negative experience.

I also thought of maybe treating it as an unsupervised task, where I look if the liked sessions fall naturally inside a specific cluster.

Edit:  I did not make it clear, but the service is similar to Youtube, where we are trying to figure out if a user had a positive viewing experience after clicking on a video. Right now, there is no recommendation engine and this is the first part in building one.

Edit: After the answers, I am leaning more towards approaching this task as an unsupervised learning task, rather than supervised.

Any thoughts how to approach this problem?
Thanks

This is not a data science solution, but wouldn't the fact that the user watched the video in the first place (took the time to find and locate the video then started the video) imply that the user have some interest in that particular video over users that never interacted with the video? I would consider the fact that the entry exists as a positive response, the lack of a "like" as a neutral response, and (maybe?) the lack of watching it as a negative response. It, of course, depends heavily on the source and assumptions of your dataset.

Nonetheless, I agree that clustering the videos with likes is a good starting point in this situation, and that would definitely be my first gut instinct thing to do.


This is not necessary related to xts. You are comparing to floating point values. As pointer out in this excellent resource these type of comparisons are dangerous. Best way to code it if you do not have Inf, -Inf or NaN is by doing,

(abs(rts.fut$Ret - rts.fut$Ret2) / pmin(abs(rts.fut$Ret) + abs(rts.fut$Ret2),
  .Machine$double.eps)) < .Machine$double.eps

Hope this helps.


Find ID for the name in the list from csv file in python The CSV file contains:
ID,Name
1,AS
2,er
3,rtf
4,addfs

The list contains the name for example (er,rtf)
I want to find the ID's corresponding to these names mentioned above
How to find the ID's using the Python code.

Thanks in advance

This question should be posted in StackOverflow.

Store your CSV into a Pandas DataFrame and operate on that.

import pandas as pd

df=pd.read_csv('nameofyourcsv.csv')
names=('er','rtf')

for each in names:
    print(df.loc[df['Name'] == each, 'ID'].iloc[0])


Multilayer Perceptron: What is the value used to update the weights in the hidden layers? As i understand for the output layer the error rate is used with the mean squared error function to update the weights.
For the hidden layers as well? Does that make sense?

A Multilayer Perceptron changes your weights by an algorithm called "backpropagation". This algorithm uses gradient descent combined with a learning rate to change every weight in your MLP.

Basically the backpropagation functions by chaining together all functions that are called when calculating the output of one particular node (so chaining together all possible ways). Now using the chain rule a gradient is calculated which directs into the direction of minimal error and changes all weights accordingly.


Piece-wise regression by clustering I was wondering about the possibilities of clustering numerical data (more than 3 dimensions) into different clusters and doing curve fitting on each cluster to get much higher accuracy than using a single model.

Since linear regression is preferred, is there any way to cluster the data points based on their linear fitting?

This is because I need a result matching the input data and do not care about unseen data. I cannot hard code the data and use a lookup mechanism. Instead an approximate math function would be preferable.

Is there an existing implementation ? (Preferably in Python)

If you use these 2 packages:

http://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html

https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html

you will get what you want :)


Newbie question on restricted boltzmann machine I’m quite a newbie to RBMs so I’m trying to understand how do you feed real valued data to it given that all the visible and hidden units are binary?

Here is an answer which might help: https://www.quora.com/What-are-best-approaches-to-modeling-real-value-continuous-input-variables-in-RBM-pretraining