How to plan a model analysis that avoids overfitting? Coming from statistics, I'm freshly trying to learn machine learning. I've read a lot of tutorials about ML, but have no real training.

I'm working on a little project where my dataset have 6k lines and around 300 features.

As I've read in my tutorials, I split my dataset into a training sample (80%) and a testing sample (20%), and then train my algorithm on the training sample with cross-validation (5 folds).

As I re-ran my program twice (I've only tested KNN which I now know is quite not appropriate), I got really different results, with different sensitivity, specificity and precision.

I guess that if I re-run the program until metrics are good, my algorithm will be overfitted, and I also guess it would be because of the resample of test/training samples, but please correct me if I'm wrong.

If I'm going to try a lot of algorithms to see what I can get, should I fix my samples somewhere? Is it even OK to do try several algorithms? (it would not always be in statistics)

In case it matters, I'm working with python's scikit-learn module.

*PS: my outcome is binary and my features are mostly binary, with few categorial and few numeric. I'm thinking about logistic, but which algorithm would be the best one ?

I guess that if I re-run the program until metrics are good, my
  algorithm will be overfitted

Re-running an algorithm does not contribute to over-fitting. Re-run and select the best model no problem. However, when we try to compare two algorithms we must average over large enough number of re-runs. However, in practice, we want the best model not the best algorithm.

  I also guess it would be because of the resample of test/training
  samples, but please correct me if I'm wrong.

Re-sampling the training set only contributes to what is known as model variance, which denotes the fact that different training samples yield different models.

  As I re-ran my program twice (I've only tested KNN which I now know is
  quite not appropriate), I got really different results

Your observation is natural. A general approach to decreasing the variance of KNN is to increase the parameter $K$, higher $K$ means KNN looks at more points around a query point (see these plots).

  If I'm going to try a lot of algorithms to see what I can get, should
  I fix my samples somewhere?

Random sampling is OK.

In python, sklearn.model_selection.cross_validate would build and validate K models for a given algorithm and returns K results; assuming we are feeding 80% of data to K-fold CV, it splits the 80% to 80(K-1)/K% training set and 80/K% validation set for K times.

In summary, first split the data 20%-80%, do cross validation on 80% of data for each algorithm (algorithms would be 1-NN, 2-NN, SVM, etc.), then select a model with the best [validation] accuracy (set return_estimator = True to get K models per algorithm from K-fold CV, so for 3 algorithms we are selecting among 3K models), and finally test the best model on the held-out 20% to get the test accuracy; cross-validation has no meaning here since the best model is already built. The final result is the test accuracy.

Also, take a look at this answer on train, validation, test sets.

  Is it even OK to do try several algorithms? (it would not always be in statistics)

Yes. Always try various algorithms.

A side note on parameter finding

We can apply the above procedure on (1-NN, 2-NN, 3-NN), and if the winner model is always selected from 3-NN algorithm, then we can limit our experiments to only (3-NN, SVM) instead of all the four. Otherwise, if the winner is not consistently from 3-NN, we can experiment with all the four.


Machine Learning Startup Ideas I just started using machine learning and was wondering if anybody have cool ideas for a Startup project, I've seen this website Treato  and was amazed by it.
Thanks.

First of all, I would get acquainted with some core concepts of Machine Learning before building websites:

Common algorithms, differences between them (Naive Bayes, KMeans, Regression, Support Vector Machines, Decision Trees, you name it...)
Supervised vs. unsupervised
Features and feature selection
Classification evaluation (precision, recall, accuracy)

Machine learning is a huge field which is applied in many different areas. I don't know how proficient you are at this stage, but I would certainly play around with these things first in order to get a solid understanding of what you're dealing with. After that, the options are virtually limitless in a world of Big Data (forgive me for using the annoying buzz-word). What is truly valuable is someone knows how to do machine-learning rather than what (because there's enough to go about).


Cross validation vs leave one out I have found the following definitions, but I don't really see the difference.

cross validation
Method for testing classification and prediction models. The data are randomly split into N partitions (typically N=10) and then N times a model is created from N-1 partitions and tested on the "holdout" data.

Leave one out
Every data point gets to be in a test set exactly once, and gets to be in a training set k-1 times.

Assuming your dataset includes $k$ samples:

In cross-validation, there are $N$ partitions, and the test split for each partition will have size $\frac{k}{N}$.

Leave-one-out validation is a special type of cross-validation where $N = k$.  You can think of this as taking cross-validation to its extreme, where we set the number of partitions to its maximum possible value.  In leave-one-out validation, the test split will have size $\frac{k}{k} = 1$

It's easy to visualize the difference. Here's two figures which contrast cross-validation and leave-one-out.  In these figures, each sample in the dataset is represented by a colored circle. The training set is represented by the green circles, and the testing set is represented by the yellow circles.

5-fold cross validation:

Leave-one-out cross validation:


Smith-Waterman-Gotoh Algorithm - how to determine an overall similarity percentage Using the Smith-Waterman-Gotoh algorithm I want to get an overall similarity percentage between two sequences. What would be the best way to do this?

eg. comparing strings COELACANTH and PELICAN in this example gives a score of 4 with alignment:

ELACAN
ELICAN

How would I then go an determine the overall similarity percentage between COELACANTH and PELICAN based on this?

I don't know how to write math algebra like on the Wikipedia page for Smith-Waterman, so i'll use pseudo code. I found the logic at SimMetrics in the SmithWatermanGotoh java code.

str1 = PELICAN
str2 = COELACANTH
matchValue = 1 #in the comparisons below, when characters are equal, assign this value
mismatchValue = -2 #in the comparisons below, when characters are not equal, assign this value
gapValue = -0.5 #the gap penalty used in smith-waterman

# get the maxDistance which is the smallest number of characters between str1 and str2 multiplied by
# the largest of matchValue and gapValue
maxDistance = min(length(str1), length(str2)) x max(matchValue, gapValue);

# function to compare character at index aIndex of string a with character at index bIndex of string b
function compareCharacters(a, aIndex, b, bIndex, matchValue, mismatchValue) {
  if a[aIndex] === b[bIndex]
    return matchValue
  else
    return mismatchValue
}

v0 = an array
v1 = an array

lengthOfStr1 = number of characters in str1
lengthOfStr2 = number of characters in str2

# do the smith waterman similarity measure (currentMax)
currentMax = v0[0] = max(0, gapValue, compareCharacters(str1, 0, str2, 0, matchValue, mismatchValue))

for (j = 1; j < lengthOfStr2; j++) {
  v0[j] = max(0, v0[j - 1] + gapValue,
            compareCharacters(str1, 0, str2, j, matchValue, mismatchValue))

  currentMax = max(currentMax, v0[j])
}

for (i = 1; i < lengthOfStr1; i++) {
  v1[0] = max(0, v0[0] + gapValue, compareCharacters(str1, i, str2, 0, matchValue, mismatchValue))

  currentMax = max(currentMax, v1[0])

  for (j = 1; j < lengthOfStr2; j++) {
    v1[j] = max(0, v0[j] + gapValue, v1[j - 1] + gapValue,
            v0[j - 1] + compareCharacters(str1, i, str2, j, matchValue, mismatchValue))

    currentMax = max(currentMax, v1[j])
  }

  for (j = 0; j < lengthOfStr2; j++) {
    v0[j] = v1[j]
  }
}

# calculate the overallSimilarity between the strings
overallSimilarity = currentMax / maxDistance #<- 0.4767 for COELACANTH vs PELICAN


How to Split And Resample Imbalanced Dataset Into Train, Validation and Test I want to understand how to split the imbalanced data set with a binary target variable where 87% of the samples are negative and 13% of the samples are positive. Now, I know that you should always split the data into train and test set before doing any kind of resampling to avoid info leakage but how what strategy gives better results - random train_test_split or StratifiedShuffleSplit? The general sentiment is that the latter seems to be outperforming the former in case of an imbalanced data set. Btw, how does StratifiedShuffleSplit work?

Also, how to factor in the cross-validation set and how to process it?

Should it also contain the same number of positive and negative samples as per the sampling was done on the training set?

Grzegorz's answer gets to the root of the problem - if the future data this model will be used to make predictions on will have the same distribution, then stratification by class % makes sense.

One thing I wanted to add is I typically use the normal train_test_split function and just pass the class labels to its stratify parameter like so:

train_test_split(X, y, random_state=0, stratify=y, shuffle=True)

This will both shuffle the dataset and match the %s of classes in the result of train_test_split.


Re: Logistic Regression I am working on a dataset that has a dependent variable that is binary, but it contains 98% of 0's and 2% of 1's. I am trying to use Logistic regression to predict purchase of a product. But because of the huge number of 0's, the model is not predicting well and getting a large number of false positive result.

Kindly suggest how can I approach this.

This is a class imbalance problem. Your data has more number of 0's, thats why the model is also predicting 0's only. There are many solutions to this problem, like over/under sampling, SMOTE etc. Here are the links which show how to tackle the problem-

Binary classification with strongly unbalanced classes

Imbalanced classification problem


Assessing significance / confidence of a crossvalidated performance measure I have a prediction model $P$ and I use some performance measure $I$ to measure $P$'s accuracy.
The distribution of $I$ is unknown (it's a custom metric, which is somehow similar to the precision metric).

My validation prediction is as following:

Randomly split the data to $k$ stratified folds
Fit $k$ models
Estimate each model according to $I$ (which results $k$-crossvalidated values of $I$)
The final model prediction performance is calculated by the average of $k$-crossvalidated $I$ measures.

I would like to perform some significance testing - to be able to say the confidence for the model prediction performance - to what extent I am “sure” in this $I$?

You may use bootstrapping to estimate a confidence interval for the prediction error. Further help can be found in some Stanford online course slides, but I haven't done this.

Besides that, it should be no problem to compare the estimated performance obtained with cross-validation (mean and standard deviation) with a reference point (e.g., AUC = 0.5) or with the results of another benchmark model such as logistic regression or nearest-neighbor classifier using a simple statistical test using a given level of confidence.

A similar question is discussed here.


KMeans for crime patterns I'm trying to find the patterns in the crime records I have in a database.
I thought clustering would be a way to do it.

This is my (cooked up) dataset:

age,nationality,country_of_birth,place_of_birth,no_of_checkedinbaggage,noofcabinbaggage,no_of_co_passengers,watchlist
34,GBR,GBR,London,2,1,0,Drug Trafficker
32,IND,IND,Delhi,2,1,0,Human Trafficker
31,USA,USA,Tampa,2,1,0,Arms Dealer
.....

Basically, I'd want to identify the clusters of watchlists and see if they are having a pattern. Based on the cluster, I'd want to predict future data as well.

Is clustering (K-Means) the correct choice? And also, do all the variables have to be numeric? If so, I'm not sure how I can encode them to numerics. Thoughts?

Welcome to the site!

As you know KMeans is an Unsupervised learning and it helps you to find out if there are any patterns in the data. Yes, the procedure which you are following to find some commonality/patterns in the data. But this is not generally used for prediction. FYI you can use K-means for predicting too, recently I came across that but I don't know whether it would yield the desired results.

If the data is categorical you need to apply One-Hot Encoding, which converts the categorical data into numeric, you can go through the link for better understanding. If not so you cannot apply K-Means algorithm.

If you cannot convert the categorical data into numeric data then you can use this package ClustMixType this is in R and you can use KMODES in Python.

If you have your target variable ready, as you haven't mentioned the data target variable, if the target variable is Numeric you can use

Neural Network
Regression
SVM
Random Forest and many more

if the target variable is Binary(is he a criminal yes/no)

Neural Network
SVM
Logistic Regression
Random Forest
Naive Bayes Classifier
KNN and many more.

Please go through this Link, for better understanding on Mixed Data type Clustering

Let me know if you need any help.


Memory allocating error with coxph model I'm working with cox models and as I am adding more variables, I am facing a memory issue. I tried to subset my dataframe by selecting columns which have variables of interest but even then I am facing this problem. How can I overcome that?

My data has around 451,557 observations with 270 variables. 70 Percent of variables are categorical (strings).

coxdf_2<- coxph(surv_df ~ NS_group + age_at_recruitment + Gender + Alcohol_drinking +
                              Smoking + Diabetes + BMI +  +Hypertention,
                method="breslow", data = DFMODEL)

  Error: cannot allocate vector of size 16.1 Gb

coxdf_3<- coxph(surv_df ~ NS_group + SOC+ towndep_I + Education+age_at_recruitment +
                          Gender + Alcohol_drinking  + Smoking + Diabetes + BMI +
                          +Hypertention,
                method="breslow", data = DFMODEL)

  Error: cannot allocate vector of size 24.4 Gb

If you're using a 64-bit version of R on Windows (I'm not sure about other operating systems), then you can allocate additional memory to R with memory.limit function.

memory.limit(size = 25000) ## limit is set based on Mb

32-bit version (depending on the system, whether it is 32-bit or 64-bit) is limited to
2 to 4Gb.

You can further read about memory limits in R.

All that said, I would take a step back beforehand to make sure I actually need to deal with all that data and would investigate other alternatives.


Machine Learning Validation Set I have read that validation set is used for Hyper-parameter tuning and comparing models. But, what if my algorithm/model does not have any hyperparameter? Should I use validation set at all? Because comparing models can be done using Test set also.

The validation set is there to stop you from using the test set until you are done tuning your model. When you are done tuning, you would like to have a realistic view of how the model will perform on unseen data, which is where the test set comes into play.

But tuning the model is not only hyperparameters. It involves things like feature selection, feature engineering and aslo the choice of algorithm. Even though it seems like you are already decided on a model, you should consider alternatives as it might mot be the optimal choice.


How does scikit-learn decision function method work? The scikit-learn docs say it is the signed distance of that sample to the hyperplane.

I've taken the sum of the weights and their corresponding coefficient and added the intercept to that sum but this does not return the value given by the decision_function method.

When doing classification tasks how is the signed distance different from taking the linear summation of the weights, feature values, and intercept?

For example, If I have the following feature values, intercept, and weights I would calculate the distance from the point to the hyperplane as...

$x_1 = 3, x_2 = 2$

$w_1 = -0.28, w_2 = -0.20$

$b = -0.68$

$$\theta^T \cdot \textbf{X} + b$$

When sklearn calculates the decision_function it returns

return safe_sparse_dot(X, self.coef_.T,
                               dense_output=True) + self.intercept_

Where self is our fitted classifier and X is the example for which we are calculating the decision function.

How can I replicate what decision_function is doing?

Both approaches seem to do the same:

safe_sparse_dot(X, self.coef_.T, dense_output=True) + self.intercept_

seems to compute a dot product of the coefficients (you called these $w_1, w_2$, and combined them to $\theta$), and the point ($x_1, x_2$, combined to $\mathbf{X}$). The intercept $b$ is added. So the formulas should do exactly the same, once in mathematical, once in Python notation: they compute the distance to the separating hyperplane, called the margin.

You also wanted to know

  how it works in the case of SGD Classifier

As that also fits a linear classifier, the computation should be the same (even though the hyperplane might have different parameters).


How to extract assignment from natural language text? I'm a bit new to NLP/IE.
I'm looking for the task within NLP/IE that would be concerned with extracting a value that has been assigned.

For instance, given the text "The value is 45.1hz" or "The color is blue", I would like to be able to extract the fact that 45.1hz has been assigned to the value, or that the color has been set as blue.

I have tried to find the right name for this task - It seems somewhat related to relation extraction, however it doesn't seem to have two or more entities, and the relation unclear to me.

It may also be related to triplet extraction, but once again I'm a bit loss on what the values would be, i.e. (color, be, blue) ?

I'm looking for the name for this type of task, where you extract the value of an assignment, for a specific pre-defined value.

It's indeed a very specific type of relation extraction. Generally relation extraction is much more complex because it's not only about simple sentences Subject Verb Object and not only with the verb is.

It's not clear to me if your examples are representative of the real cases you're dealing with: if yes, you probably don't need full blown relation extraction, basic pattern matching rules will do the trick.


Scikit learn: which regressors natively support multi-target regression? The docs on sklearn.multioutput.MultiOutputRegressor state that it implements a strategy for extending regressors that do not natively support multi-target regression.

I'm interested to know: which ones do natively support multi-target regression ? (and/or how can I find out?)

based on this article, I assume:

ExtraTreesRegressor
KNeighborsRegressor
LinearRegression
RidgeCV

All support this? (unless I'm getting confused and a 'multi-output estimator'  is not the same thing as 'a regressor that natively supports multi-target regression'; I'm new to this!)

In scikit-learn, there is a MultiOutputMixin class. So, you could check if the estimator inherits from that one.
Alternatively, you could use estimator tags and check if {'multioutput': True} is included in the tags of the estimator.


droplevels not dropping anything I am trying to use droplevels.
First I create a subsect of my dataset where the value of attribute Rat must be different from 1 or 5. That value must be a factor. Then, I can see levels are "1", "2", "3", "4" and "5". As I use droplevels() nothing changes. How can I do properly? I wrote this code:

chocolate234 <- subset(chocolate, Rat!="1")
chocolate234 <- subset(chocolate234, Rat!="5")
droplevels(chocolate234$Rat)

Like most function in r, droplevels operates on your data but does not change it. The droplevels function returns a factor vector with the levels dropped, but it does not change the original vector. To do that, you need to assign the result of droplevels back to the original original vector. Here is an example:

# Building an example data.frame
set.seed(1234)
rating <- factor(as.character(sample(1:5, 100, replace = T)))
label <- sample(LETTERS, 100, replace = T)
example <- data.frame(label = label, Rat = rating)
summary(example$Rat)
  1  2  3  4  5
 25 26 17 17 15

# Subsetting the data
example234 <- subset(example, Rat != "1")
example234 <- subset(example234, Rat != "5")
summary(example234$Rat)
  1  2  3  4  5
  0 26 17 17  0

# Dropping unused levels
droplevels(example234$Rat)
  [1] 4 4 4 4 2 4 3 4 3 2 2 2 2 2 2 2 2 3 3 2 2 3 4 2 2 3 4 2 4 2 3 4 3 2 4 2 4
  [38] 3 3 3 4 2 2 4 2 3 3 4 3 2 2 4 3 3 2 2 3 2 2 4
  Levels: 2 3 4
summary(example234$Rat)
  1  2  3  4  5
  0 26 17 17  0

# Reassigning the data with dropped levels
example234$Rat <- droplevels(example234$Rat)
summary(example234$Rat)
  2  3  4
 26 17 17

Notice in my example, like your attempt, if you just run droplevels on the data, it does not change the underlying data. However, if you assign the results of droplevels(example234$Rat) to example234$Rat, your underlying data has been properly changed.


Keras: Misunderstanding the Conv2D's param "filters"? From https://keras.io/layers/convolutional/ (Conv2D):

keras.layers.Conv2D(filters, kernel_size, strides=(1, 1), padding='valid',
                    data_format=None, dilation_rate=(1, 1), activation=None,
                    use_bias=True, kernel_initializer='glorot_uniform',
                    bias_initializer='zeros', kernel_regularizer=None,
                    bias_regularizer=None, activity_regularizer=None,
                    kernel_constraint=None, bias_constraint=None)

  filters: Integer, the dimensionality of the output space (i.e. the number of output filters in the convolution).

Why is it possible to indicate the number of features maps (results of the scalar product between the kernel and the input), since we specify the kernel's size and the input?

Indeed, if we know the kernel's size (width and height) and the input's size (width and height), thus we know the number of features maps.

filters are the numbers of kernels or feature detectors that we choose for the convolutional layer to learn.
in the end, the number of feature maps that we get equals to this number of filters that we have declared here.
Now, what are these feature detectors? these are matrixes usually smaller (much smaller) than the input images. It's occasionally $3by3$, $5 by 5$ or like the one in Alexnet $7by7$.
In the convolution layer, a convolution operation is done using each of these filters, each time we take a filter and our input image, then we put the filter on the left top corner of the input image and multiply each matching cells, then we would sum up the results and put that in the very first cell of our output (feature map) matrix.
Then we move this filter on the image and do the same,... till we fill the whole feature map (stride defines how many cells we move in the image). Once we have our feature map number one that is a result of applying filter number one on the input image, we take another filter and do the same and so on and so forth.
Obviously, the outputted feature maps are smaller than the input, the amount by with the input size is reduced depends on the stride that we choose, moreover, we have kinda aggregated features in our feature map (also called convolution/convolved map)

Why is it possible to indicate the number of features maps (results of the scalar product between the kernel and the input), since we specify the kernel's size and the input?
Indeed, if we know the kernel's size (width and height) and the input's size (width and height), thus we know the number of features maps.

The thing is if we just know the input size and the size of the kernel, we don't know the stride, then setting a stride of 2 or 1 or anything else would give us different results. If you assume your stride to be a specific number, then you would know how many filters should be used.


model.predict keeps predicting the same, wrong, class I have a simple oxford102 (102 UK flowers) tensorflow model and Android app from here.
I'd like to see if I can add recognition (model.predict) to the python code in the subproject FlowerML which is python code that generates the model in the first place.
Added the following to FlowerML2.py to recognize a single flower:
# sequence of calls adapting image then calling predict:
# https://datascience.stackexchange.com/questions/31167/how-to-predict-an-image-using-saved-model
test_image = image.load_img('d:/ML/images/CallaLily-1.jpg', target_size=(224, 224))
print("test_image PIL size " + str(test_image.size))
test_image_arr = image.img_to_array(test_image)
print(test_image_arr.shape)
# double-check visually
image.save_img('d:/ml/images/WhatIsIt-1.jpg', test_image_arr)
test_image = np.expand_dims(test_image_arr, axis=0)
print(test_image.shape)
result2 = model.predict(tf.convert_to_tensor(test_image), batch_size=1, verbose=2)
print("\nnp.where result2: " + str(np.where(result2 == result2.max())) + " max = " + str(result2.max()) + "\n")

The model correctly identifies Calla Lily in the Android app however when I query the model as above it keeps telling me the wrong class.  'Keeps' refers to many modifications not above - and even with a different flower.  It's like the model, in this context (but not in the Android app), gets stuck with an output to its head layer of n=1.
Since it works in Android, my assumption is that I'm not calling predict properly - although exactly what goes wrong I can't say and I have little experience, so far, digging into how models work which is maybe what you need to do when they don't function correctly.

Figured it out.  Need to normalize the image I'm going to predict to $[-1,1]$ instead of what-I-assume-is-the-standard $[0,1]$.  It seems this is model-dependent.  The project I'm working off of uses transfer learning and imports this model.
It's not immediately clear to me though just how this model needs inputs normalized to $[-1,1]$.


Dimension of output in Dense layer Keras I have the sample following model
from tensorflow.keras import models
from tensorflow.keras import layers
sample_model = models.Sequential()
sample_model.add(layers.Dense(32, input_shape=(4,)))
sample_model.add(layers.Dense(16, input_shape = (44,)))

sample_model.compile(loss="binary_crossentropy", optimizer="adam", metrics = ["accuracy"])

IP for the model:
sam_x = np.random.rand(10,4)
sam_y = np.array([0,1,1,0,1,0,0,1,0,1,])
sample_model.fit(sam_x,sam_y)

The confusion is the fit should have thrown an error of shape mismatch as the input_shape for the 2nd Dense Layer is given as (None,44) but the output for the 1st Dense Layer (which is the input of the 2nd Dense Layer) will be of shape (None,32). But it ran successfully.
I dont understand why there was no error. Any clarifications will be helpful


What's the difference between fit and fit_transform in scikit-learn models? I'm a newbie to data science, and I do not understand the difference between the fit and fit_transform methods in scikit-learn. Can anybody explain simply why we might need to transform data?
What does it mean, fitting a model on training data and transforming to test data? Does it mean, for example, converting categorical variables into numbers in training and transforming the new feature set onto test data?

To center the data (make it have zero mean and unit standard error), you subtract the mean and then divide the result by the standard deviation:

$$x' = \frac{x-\mu}{\sigma}$$

You do that on the training set of data. But then you have to apply the same transformation to your testing set (e.g. in cross-validation), or to newly obtained examples before forecast. But you have to use the exact same two parameters $\mu$ and $\sigma$ (values) that you used for centering the training set.

Hence, every sklearn's transform's fit() just calculates the parameters (e.g. $\mu$ and $\sigma$ in case of StandardScaler) and saves them as an internal object's state. Afterwards, you can call its transform() method to apply the transformation to any particular set of examples.

fit_transform() joins these two steps and is used for the initial fitting of parameters on the training set $x$, while also returning the transformed $x'$. Internally, the transformer object just calls first fit() and then transform() on the same data.


What are your thoughts on SKLearn's dismissal of GPUs for machine learning? SKLearn has this broad claim in its FAQs:

  Outside of neural networks, GPUs don’t play a large role in machine learning today, and much
  larger gains in speed can often be achieved by a careful choice of
  algorithms.

Anyone care to add color for or against this claim?

GPU doesn't inherently fit naturally into all machine learning algorithms.  A natural contender is one that inherently takes a myriad of matrix multiplication.  This makes sense since graphic processors inherently were design for this.  However, for an algorithm like a Random Forest this may not be so important.  Also there exist a cost to transfer information to a GPU.  Any algorithm that is O(n) should not be computed on a GPU because it takes O(n) to communicate the data.  There's a few other issues that GPU present dealing with RAM and Threading, each of which often render making a GPU variant of a project more of a hassle than it's worth.

Furthermore, adding GPU to the sklearn framework inherently adds a hardware dependency and complexity that seems needless for shallow algorithms.  Odds are, if you're needing access to your GPU, you are dealing with a neural network, which has it's own unique architecture challenges.  I think it makes far more sense to separate deep learning into it's own module (look at how huge tensorflow/pytorch/etc) project are) than force Sklearn to add hardware dependencies for marginal computational gains.


Activation in convolution layer On a CNN, what is use of using Activation function in convolution layer?

Does single weight is used for full matrix or for every pixel or box it may vary?

The use is the same as always, without the non-linear activation, no matter how many layers you use, the result will still be a linear model.

The non-linear activation is performed on every single output "pixel" indevidualy:


How to create a df that gets sum of columns based on a groupby column? country   year    gender    measure               value0 ... value12
 A         2000    1         vaccinated_at_month   2      ... 1
 B         2000    1         vaccinated_at_month   13     ... 12
 A         2000    0         vaccinated_at_month   4      ... 3
 A         2000    9         vaccinated_at_month   5      ... 4
 B         2000    0         walked_at_month       3      ... 13
 C         2001    1         vaccinated_at_month   6      ... 5
 C         2001    0         vaccinated_at_month   3      ... 2

I want to be able to remove the gender column and collapse all values into sums based on the previous categorical columns.

I have tried

df_new = df.groupby(['country', 'year', 'gender', 'measure'])['value0', ... 'value12'].apply(lambda x : x.astype(float).sum())

However, df_new still preserves the gender column. I am trying to get this outcome:

country   year       measure               value0      ... value12
 A         2000       vaccinated_at_month   11 (=2+4+5) ... 8 (=1+3+4)
 B         2000       vaccinated_at_month   13          ... 12
 B         2000       walked_at_month       3           ... 13
 C         2001       vaccinated_at_month   9 (=6+3)    ... 7 (=5+2)
 C         2001       vaccinated_at_month   3           ... 2

This question is more appropriate at Stackoverflow since it is more of a programming question.

However, I think you almost got it working. Just remove gender from your groupby:

df_new = df.groupby(['country', 'year', 'measure'])['value0', ... 'value12'].apply(lambda x : x.astype(float).sum())


Bubbleplot with seaborn Is there someway to create a bubble plot with seaborn? I already know how to do it with Matplot lib, as in this tutorial.
# libraries
import matplotlib.pyplot as plt
import numpy as np

# create data
x = np.random.rand(40)
y = np.random.rand(40)
z = np.random.rand(40)

# use the scatter function
plt.scatter(x, y, s=z*1000, alpha=0.5)
plt.show()

I would like to replicate this with Seaborn. I am able to plot, but cannot control the size of marker.
import seaborn as sns
import numpy as np
%matplotlib inline
data=pd.DataFrame({"x":x, "y":y,"z":z})
sns.lmplot(x="x", y="y",data=data,  fit_reg=False)

A duplicate of this question in stackoverflow. Anyhow just to recap quickly, you can do it with:

scatter_kws={"s": 10}

And in your case it is simply:

sns.lmplot(x="x", y="y",data=data,  fit_reg=False,scatter_kws={"s": z*1000})


'tuple' object is not callable while reshaping training data using python I have data csv file with three inputs names temperature, humidity, wind. Here I want to predict temperature value in every 60 minute using LSTM model.

Here I write the code to reshape the train . But I got an error tuple' object is not callable

My code:

data = pd.read_csv('data6.csv' )
data['date'] = pd.to_datetime(data['date'] + " " + data['time'], format='%m/%d/%Y %H:%M:%S')
data.set_index('date', inplace=True)
data = data.values

scaler = MinMaxScaler(feature_range=(0, 1))
data = scaler.fit_transform(data)

train_size = int(len(data) * 0.67)
test_size = len(data) - train_size
train, test = data[0:train_size,:], data[train_size:len(data),:]

X = 1
n_out = 1

x,y=[],[]
start =0
data = train.reshape(train.shape(train.shape[0] ,3, train.shape[1]))
for _ in range(len(data)):
  in_end = start+X
  out_end= in_end + n_out
  if out_end < len(data):
    x_input = data[start:in_end]
    x.append(x_input)
    y.append(data[in_end:out_end,0])
start +=1

x = np.asanyarray(x)
y = np.asanyarray(y)

My csv file

Error:

  data = train.reshape(train.shape(train.shape[0] ,3, train.shape1))

Error image:

Can anyone help me to solve this problem?

Error:

The problem is that train.shape is simply a tuple. So train.shape(arguments) doesn't make sense because train.shape is not callable (hence the error). Try to just replace that line by

data = train.reshape((train.shape[0], 3, train.shape[1]))


New to ML and NLP. Is topic/subject extraction a good place to start? I'm new to AI and specifically NLP, I always love to study things as part of a random project I decide to build, right now I'm working on a chat bot and I was looking for an easy way(in terms of code) to extract the main topic/subject from a sentence(in English), the problem is that wherever I search, even if it's a simple code, it combines a full sentence "meaning"(device parts of speech), I don't like to implement something I don't understand so a small step for how to extract only the subject will be really good and can help my bot with some directions, Is there a simple code(don't care what language it written with) to extract, only main topic/s from a sentence(in English)?.

You could learn about word embeddings. These will provide you a natural path to topic models, and many other NLP tasks. Look up word2vec.

I think it is better not to learn two new subjects (NLP and ML) at once. I would start with ML on its own; take this class perhaps. A good place to start is linear regression, or binary classification.


Neural Network accuracy and loss guarantees? This question is part of a sample exam that I'm working on to prepare for the real one. I've been stuck on this one for quite a while and can't really motivate my answers since it keeps referring to guarantees, can someone explain what guarantees we can have in a neural network such as the one described?

Network description:

Assume a deep neural network (DNN) that takes as input images images of handwritten digits (e.g., MNIST). The input network consists of 28x28 = 784 input units, a hidden layer of logistic units, and a softmax group of 10 units as the output layer. The loss function is the cross-entropy. We have many training cases and we always compute our weight update based on the entire training set, using the error backpropagation algorithm. We use a learning rate thats small enough for all practical purposes, but not so small that the network doesnt learn. We stop when the weight update becomes zero. For each of the following questions, answer yes or no, and explain very briefly.

Questions:

Will this DNN configuration enable the the weights to minimize the loss value (there may be multiple global optima)?
Does this DNN configuration guarantee that the loss reduces on every step? We use our network to classify images by simply seeing which of the 10 output units gets the largest probability when the network is presented with the image, and declaring the number of that output unit to be the networks guessed label.
Does this DNN configuration guarantee that the number of mistakes that the network makes on training data (by following this classification strategy) never increases?

For my answers, I assume you are talking about batch (not mini-batch or stochastic) gradient descent.

No. Assume you initialize all weights with the same value. Then all gradients (in the same layer) will be the same. Always. Hence the network effectively only learns one parameter per layer. It is possible (and likely) that this is neither a gloabl nor a local minimum of the network (which has more parameters).
Yes, as the learning rate is "small enough for all practical purposes". (No, if you use SGD or mini-batch gradient descent)
Usure. I think the correct answer is "No, the network can make more mistakes in between with cross entropy.". It is certainly sure to improve CE loss while at the same time getting worse at accuracy (see proof below). However, I'm not sure if the gradient would ever lead to such a result.

Example for 3

#!/usr/bin/env python

from math import log

def ce(vec):
    """index 0 is the true class."""
    return -(log(vec[0]) + sum([log(1-el) for el in vec[1:]]))

a = [0.1001, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.0999]
print(ce(a))
b = [0.49, 0.51, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
print(ce(b))

gives:

ce(a) = 3.24971912864
ce(b) = 1.42669977575

Hence the cross entropy loss dropped (as expected), the probability for the correct class increased (as expecte) but it makes a mistake (if you simply take the argmax).


Use heading in Neural Network model I am working on a prediction model where I must find out the destination of a boat based on its actual coordinates and heading (compass direction) :
In[8]: X.head()
Out[8]:
   latitude  longitude     heading
0  0.094700   0.094700  332.398791
1  0.090828   0.090828  197.320172
2  0.085800   0.085800  140.537550
3  0.081676   0.081676  128.891893
4  0.077804   0.077804  129.881418

Latitude and longitude are in degrees (normalised here) and heading is in degrees.
The output variable is an integer between 1 and n (there are n possible destinations).
My problem here is to make the neural network understand how heading difference works : 5 degrees and 15 degrees are as close as 5 degrees and 355 degrees because we have to use a modulo 360 after making the difference :
In[11]: diff_heading(355, 5)
Out[11]: 10
In[12]: diff_heading(15, 5)
Out[12]: 10

I've got some acceptable results when I put the coordinates and heading in a basic multilayer perceptron neural network without any preprocessing on the heading. But I'm pretty sure it could be way more better if the network could take into account this heading problem. For now, the network must attribute weights to the heading which indicate that 0 and 360 degrees are opposite, which is false.
Do you have any idea on how to process the data or how to change the network structure to achieve this ?

You could break the heading ($\theta$) into components: $sin(\theta)$ and $cos(\theta)$, which would remove the modulo problem.


Decision Tree gives 100% accuracy - what am I doing wrong? My assumption is that my training set includes the test set, but I don't know how to change this.

from sklearn.model_selection import train_test_split
import sklearn.metrics as metrics

# dataframe to store model performances
scores=pd.DataFrame([],columns=['model', 'recall', 'f1', 'accuracy'])
from sklearn.tree import DecisionTreeClassifier

cv_scores_ac=[]
cv_scores_f1=[]
cv_scores_re=[]
for cv in range(1, 6):
    print ("Decision Tree - Iteration %i" % cv)
    X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=cv)

    tree=DecisionTreeClassifier(min_samples_split=20)
    tree.fit(X_train, y_train)
    y_pred=tree.predict(X_test)
    cv_scores_ac.append(metrics.accuracy_score(y_test, y_pred))
    cv_scores_f1.append(metrics.f1_score(y_test, y_pred))
    cv_scores_re.append(metrics.recall_score(y_test, y_pred))

Machine Learning is one of the few things where 99% is excellent and
  100% is terrible.

Well, I cannot prove this because I don't have your data, but probably:

the test data is included in the training data.

To check this possibility, here's a hint:

print(X_test[X_test.isin(X_train)])

will print all the rows in X_test that appear in X_train. Can you think what to do with that information? :)

Or

you forgot to remove the target variable y from X.

To check this possibility, simply type:

print(X_train.columns)

and check whether one of the columns matches the name of the target variable.


Image Recognition Model with Adaptive Learning Problem Statement: To be able to detect faces of specific people with good accuracy and tag those images with the names of the people it contains.

To do this we have about 40 images person, and 10 classes.

I have been using Facenet model ( https://github.com/davidsandberg/facenet ) and I am getting good results after training the model with 10 classes & 40 images per class. I have used 100 images for testing.

The current hurdle im facing is : Suppose during testing it suggested with accuracy 40% for an image containing person of say class A. I would like to add this image back to the models training dataset and train the model adaptively, so that it improves the accuracy for this person. That would mean transferring an image from the test set to the training set.

Is it possible to have an adaptive learning added to my current solution?
Not sure if I made my problem clear , I could explain more in detail if needed.

Suppose during testing it suggested with accuracy 40% for an image containing person of say class A. I would like to add this image back to the models training dataset and train the model adaptively, so that it improves the accuracy for this person. That would mean transferring an image from the test set to the training set.

Whilst adding data that appears important to your task to the training set will often improve performance, you have a major problem here. You must base your measure of performance on the test set. Altering the test set can invalidate your test results so far. And of course if you take a problem item out of the test set, your performance on the test set will go up even if you don't use the it to train with, because it was a result that was bringing down your average. So, you might improve your classifier, but at the expense of not knowing whether you have improved the classifier.

A good default position here is that the test set should not be altered. Once set aside, that data is for measuring performance of your tuned model, and should not be used for anything else, to avoid this measurement problem.

However, there are a couple of things you could do:

Error analysis

Look at the failing cases in the test set and try to figure out why they are being misclassified. For an image classification task, you might be able to figure out by viewing the image directly. Is there unusual lighting, pose, any different foreground or background objects? In the case of face recognition, has the subject got different hairstyle, makeup, hat, glasses or clothing?

If you spot something that's a one-off difference in your problem image, then that may be a sign that you need some more variety in your training data and should collect more. You should not collect it from the test data though, because then you will not be able to tell how well your classifier is doing at the task any more.

It is possible that investigating a failed item will show up something that means you want to alter or remove the test data item. This would happen for example if you decide it is not representative of how you want your model to be used. For instance, all the other images in the data set are from CCTV - which is how you intend it to be used - and the bad example is from a studio photo shoot. In that case, it may be worth the effort to clean up your test data and re-evaluate your best models so far.

K-fold cross validation

You can use k-fold cross validation to gain more accurate assessments of perfomance when tuning hyperparamaters (number of neurons, layer architecture). This doesn't technically replace a test set for measuring final performance, but it's not too bad a fallback position. An extreme version of this would be leave-onme-out cross validation.

In theory you could select your best hyperparameters using k-fold cross validation, have a good idea that this would be the best you could get, and a rough idea of what the generalisation performance was. Then train a final model using the same hyperparameters and all of your data.

You may be able to get away with this if you are happy to not know (or not need to report) an unbiased performance metric, but be reasonably certain you have close to the top performance out of the models that you have investigated.

Keep the test set as-is, do both the above, maybe collect more training data

I would recommend instead this as the best approach if you can afford the time and effort. For better or worse, you have set your goals for the first attempt at your project, when you selected the test set.

You can search harder for hyperparameters that generalise the best, using a small amount of data, by using k-fold cross validation. You can try to figure out why your accuracy is low by looking at the errors from the model and finding ways to fix them without using the test data in the training set.

Eventually you will have your best model (according to k-fold CV on the training data), and a measure of how it performs on the test set. The project is then done. Any further work to get even better accuracy would require some kind of re-thinking or start of a new project.


Leaky ReLU inside of a Simple Python Neural Net To build a simple 1-layer neural network, many tutorials use a sigmoid function as the activation function. According to scholarly articles and other online sources, a leaky ReLU is a better alternative; however, I cannot find a way to alter my code snippet to allow a leaky ReLU.

I tried logic like if x > 0 then x else x/100 as the activation function, then the same for the derivative.

Is it failing because the output layer cannot have a ReLU? Should I change the first layer to ReLU then add a softmax output layer?

import numpy as np
np.random.seed(1)

X = np.array([[0,0,1],[0,1,1],[1,0,1],[1,1,1]])
y = np.array([[0,1,0,1]]).T

class NN:
  def __init__(self, X, y):
    self.X = X
    self.y = y
    self.W = np.random.uniform(-1, 1, (X.shape[1], 1))
    self.b = np.random.uniform(-1, 1, (X.shape[1], 1))

  def nonlin(self, x, deriv=False):
    if deriv:
      return x*(1-x)
    return 1/(1+np.exp(-x))

  def forward(self):
    self.l1 = self.nonlin(np.dot(self.X, self.W + self.b))
    self.errors = self.y - self.l1
    print(abs(sum(self.errors)[0]))

  def backward(self):
    self.l1_delta = self.errors * nonlin(self.l1, True)
    self.W += np.dot(self.X.T, self.l1_delta)
    self.b += np.dot(self.X.T, self.l1_delta)

  def train(self, epochs=20):
    for _ in range(epochs):
      self.forward()
      self.backward()

nn = NN(X, y)
nn.train()

I assume the task you're working on is a binary classification task since y = np.array([[0,1,0,1]]).T. And as an error function you use self.errors = self.y - self.l1. Now compare this to the curve of a leaky ReLU function:

The leaky ReLU is an unbounded function. How is your network supposed to model a binary classification task where output values are elements of $\{0,1\}$ using this function? And what is the result of applying the absolute difference as an error function to your labels $y \in \{0,1\}$ and your outputs $\hat{y} \in (-\infty,\infty)$?

What you would need is a translation of the lReLU outputs to your classes, e.g. something like prediction = 1.0 if activation >= 0.0 else 0.0.

Which is one reason why it is more common to apply last layer activation functions to classification problems which can be interpreted as class probabilities, e.g. Softmax or Sigmoid:

You still need a threshold here to move from real valued outputs (which are interpreted as probabilities) to binary class labels but that is fairly straightforward since often you just set it to a probability of $0.5$.

I suggest to go follow a tutorial for your from scratch implementation. If you are willing to take a step back here is one for a simple percepton (i.e. just a single neuron).

Images taken from this article


Metric MAP@k for what What is the MAP@K metric for?
What are you measuring? And where does it make sense to use it?
Unfortunately, I can't find much about this on the Internet. Could someone help me with this? Thanks in advance.

MAP@k is normally used in recommendation systems, but also in other kinds of systems. Quoting from here:

If you have an algorithm that is returning a ranked ordering of items, each item is either hit or miss (like relevant vs. irrelevant search results) and items further down in the list are less likely to be used (like search results at the bottom of the page), then maybe MAP is the metric for you!

Some application examples are these Kaggle competitions:

Expedia Hotel Recommendations
Santander Product Recommendation
National Data Science Challenge 2019

And these are some resources with more information about it:

Mean Average Precision (MAP) For Recommender Systems

How mean Average Precision at k (mAP@k) can be more useful than other evaluation metrics


Why does an SVM model store the support vectors, and not just the separating hyperplane? In every explanation of SVMs, we're shown how training finds a hyperplane that best separates the data.  Presumably then for inference, you just check which side of the plane a point is on.

However, all the "disadvantages of SVMs" posts [1, 2] lament that SVM models are large and slow because they end up storing most of the data as support vectors.

Why would SVMs store any of the data, rather than just the (coefficients of the) separating hyperplane?  (And what is a "support vector" in the soft-margin case, when points of both classes are scattered on both sides of the hyperplane, anyway?)

The hyperplane is a linear combination of the support vectors. In the soft margin case, there is only a limited amount of slack; every input does not get to be support vector. In the nonlinear case, the separating hypersurface may be embedded in an infinite-dimensional space, making it impossible to store. To borrow from the Wikipedia article, the normal vector $w$ is given by

$$w = \sum_i c_i y_i \phi(x_i)$$

where $\phi$ is the feature embedding function, and $c_i$ is a Lagrangian dual variable that is zero for points on the correct side of the margin. Instead, test points are classified through a kernel function $k(x_i,x_j) = \left< \phi(x_i), \phi(x_j) \right>$ like so:

$$x \to \mathrm{sgn}(\left<w , \phi(x)\right> + b) \equiv \mathrm{sgn} \left( b+\sum_i c_i y_i k(x_i, x)\right)$$

Notice how we avoided explicitly calculating $w$.


Simple linear regression in PyTorch I am performing simple linear regression using PyTorch but my model is not able to properly fit over the training data. please look at the code to find the mistake.
Dataset is here
import torch
from torch import nn
from torch.optim import SGD, Adam
from torch.autograd import Variable

class Linear_Reg(nn.Module):
    def __init__(self):
        super(Linear_Reg, self).__init__()
        self.linear = nn.Linear(1,1)

    def forward(self, x):
        y_pred = self.linear(x)
        return y_pred

net = Linear_Reg()

Xt = Variable(torch.Tensor(X[:,0]))
yt = Variable(torch.Tensor(y))
Xt = Xt.view(-1,1)

criterion = nn.MSELoss()
optimizer = Adam(net.parameters(), lr=0.001)

EPOCHS = 500
for epoch in range(EPOCHS):
    pred_y = net(Xt)
    loss = criterion(pred_y, yt)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    print('Eopch: {}, \t\t loss: {}'.format(epoch, loss.data.item()))

The loss decreases from ~68.88 to ~30.26
and the resulting fitting is this:
xxt = torch.arange(5,23)
with torch.no_grad():
    a = net(xxt.reshape(-1,1).float())
plt.scatter(X[:,1], y, s=30, c='r', marker='x', linewidths=1)
plt.plot(xxt.data.numpy(),a.data.numpy(), label='Linear regression (Gradient descent)')

What is the problem here?

I know this doesn't answer you 100%, but maybe it helps. Feel free to disregard.

I just implemented this in keras and didn't have any issues

model = Sequential([
    Dense(1,activation='linear',input_shape=(1,))
])
sgd = optimizers.Adam(lr=0.001)
model.compile(optimizer='adam',
              loss='mean_squared_error',
              metrics=['mae'])
model.fit(test_df.x, test_df.y, validation_split = .2,epochs=500, batch_size=1)

plt.scatter(test_df.x,test_df.y,s=30,c='r',marker='x')
x = np.linspace(4,25,100)
y = model.get_weights()[0][0][0]*x+model.get_weights()[1][0]
plt.plot(x, y)
plt.ylim([-5,50])
plt.xlim([4.5,23])

Is it possible you read your data in incorrectly maybe? Or maybe you are referencing it incorrectly when you set your Xt and yt?


demand forecast for B2B I am attempting to create a demand forecasting model in python to predict future sales of a particular category of product, using historical sales data.

We are a B2B company, which means that we often get large orders at random times in a year, and there are other periods of no orders.

When using data for the past 5 years, the pattern looks a bit odd. from 2014 to 2016 there is an upward trend, followed by a downward trend.

When I attempt a linear forecasting method, the training set seems to predict the test set fairly well based on the graph. However, the metrics show that the projected sales count is off significantly.

What is the reason for such a huge error rate?

I realize that perhaps the best model for time series such as this would be something like SARIMA. I tried attempting ARIMA models, but because of the nature of the business (B2B) and very few data points available, SARIMA spits out too many errors. (for example, while trying to smooth out data for trend and seasonality, there are lots of logarithmic transformations. This removes resulting Nans and -infs, which leaves the data with almost zero data points.)

For a B2B company with very limited data points on purchase of a particular product (out of 5 years/1825days, only about 400 days when sales occur, for example), is it possible to build a reliable demand forecast model?

Is it possible to use linear regression to forecast future sales without the model projections being significantly off?

In summary, I would like to know if:

it is realistic to attempt reliable demand forecasting for a B2B sales with limited sales info.
What would be the best approach to forecasting B2B sales of particular products?

I would suggest to look at intermittent demand models, I think for such type data ARIMA type model is an inappropriate model.

Intermittent Demand model first proposed by J. D. Croston and I would suggest to look at that paper too. Here, you can find a short note in intermittent demand forecasting.


How can I estimate user-item purchase probabilities of a e-commerce website? I am writing my Master thesis, where the goal is to estimate user-item purchase probabilities. In other words, for a given user, what is the probability he/she will buy a certain item. I have session logs (click sessions on items of users) and buys from an e-commerce website.

I found the evaluation to be very hard. Therefore, my approach was the following:

1 Predict the probability that a user will buy something in the next session, given the current session (and earlier sessions).
Using this, I can filter out people that have a reasonable probability to buy something at all.

2 I want to take the top 50 items (recommended/ranked by the recommender system currently in place) and from these items, I want to estimate the probability. (As estimating this from all items would be impossible to evaluate at all).

Something else I could do is only looking at the items that a user have clicked on, and estimate those probabilities.

The biggest problem is how to evaluate these probabilities?

Can someone help me define the problem, or give me some hints/tips on how to proceed? If this would not be possible, I have to find another (related) topic to my thesis.

If there are users' session available I'd recommend to go for a session-based recommendation - when we make recommendation of next click/buy item based on the current user's session. That seems the most appropriate for your data and also very attractive way of doing recommendation systems in the e-commerce, when still most of users while searching for purchase or window-shopping is not logged in.

While predicting the next action, few measures can be used. But from practical perspective and current publications, two are quite interesting:

REC@N - Recall at top-N best recommended items. This measure gives the information about the overall engagement of the users and many times it's said that is correlates with CTR on live(production) use.
MRR@N - Mean Reciprocal Rank at top-N ranked recommendation, which gives you the information how well is your personalized ranking for the current session prediction.

I'd recommend reading two recent papers about session-based recommendation - one is about making the next item recommendation based on sequences of items [1]. The other one [2] is about making recommendations based on the current user's session when you have rich context for recommended items and session events. Which is mostly true for the e-commerce system, where you have items described by: title, attributes, text description, images, etc. And also in users session there are additional events like interaction which search engines, which gives lots of users preferences in data for the query, sorting by which attribute, etc. This can be used as a current session context. In both works you can find references to related works, which can be helpful for you.

[1]: Hidasi, B., Karatzoglou, A., Baltrunas, L., & Tikk, D. (2015). Session-based Recommendations with Recurrent Neural Networks, 1–9. Retrieved from http://arxiv.org/abs/1511.06939

[2]: Twardowski, B. (2016). Modelling Contextual Information in Session-Aware Recommender Systems with Neural Networks, 0–3. http://doi.org/10.1145/2959100.2959162


How to show value of a classification model even though it doesn't get the desired performance? I developed a classification model for a telecom client. Where we classify between Dual-sim and non-Dual-Sim clients. After many iteration the best precision we can get is 60%. The contract says that the acceptance criteria is 75% precision. The client measures precision based on campaign results. In other words, they call sample data and ask them explicitly are a dual sim or not.

Facts we know:

Learning curve shows that more training data won't do the trick.
We have got all variables that were used in the market before(and everything we could think of, but nothing improved the model).
Training data don't even get 1% of the whole population I am generalizing on(and we can't get more than that).
The random baseline is 35%, so we got a lift of around 1.9 (but they won't approve of it).
We have tried many iterations from the most simple models to the most complex, and this is the best we can get.
The dual sim and non-dual sim distribution across the variables, isn't that different.
The language used were SQL, R.

So the question is, what else can I do to prove that the model has a value but without getting the 75% precision?

Often customers don't understand the value behind the performance of machine learning models. 60% precision might sound not so good, since 50% is often related to nearly random for binary problems. However, it might well be a very difficult problem, and 60% might well be very good performance.

One way you can show your customer that your model is indeed performing well, is to compare its precision against a human classification.

You can:

Define a random baseline, and explain why it is 35% and not 50% for your problem.
Get a couple of humans from your client side to try the task manually.
Let your model perform the task and show the improvement.

More often than not, humans perform tasks more precisely than models (if in small scale). So, if your model performs better, hopefully, they will see the value your model is adding to their company.


What is the value of AIC criterion if RSS is 0? The AIC formula is :
$AIC = 2k + n Log(RSS/n)$
So if RSS is equal to 0, it is undefined. How do I deal with this? What value should it take?

RSS = 0 implies that the model is a perfect fit since there was no residual. The limit of the log of 0 is $-\infty$, and since lower AICs are better, and this model is perfect,it makes perfect sense that the AIC should be a negative number such that no number can be lower.


What does n means in neural network neuron output? I've found this equation that explains the output of a neuron in a MLP network:
$y(n) = f(\mathbf{w}^T \mathbf{x}(n) + b)$
I can understand the general context, but since i have no background with mathematical notation, i don't understand what the $(n)$ parameter means (e.g. $y(n)$, $x(n)$). Is it sort of a temporal or sample index? I've seen this notation in other machine learning subjects, but didn't find an explanation.

n is the dimension of the vector x and also y, as you can see wT is a transpose of w with dimension (n,n), is the image z is y and a is x. and dont bother about l it indicates the index of layer.


How the combination of cross entropy loss and gradient descent penalizes and rewards For a simple problem of classification (C classes) using the softmax classifier, most people use the cross-entropy loss function to quantify the objective. The cross-entropy loss is:

$$L = -\sum_{i=1}^C y_ilog(p_i)$$

where $p_i$ is the classifier prediction score and $y_i$ is the ground-truth labels. To update the parameters, gradient descent is used to make the distribution of $p_i$ as similar as possible to the distribution of $y_i$.
Let's say for an arbitrary point in space ($x_i$) we have the ground truth  label $y_i = [1, 0, 0]$ which means for a $C=3$ class problem, $x_i$ belongs to the class 0. And let's say the softmax classifier has a prediction of $p_i = [0.3, 0.6, 0.1]$ in the first iteration. When optimizing $L$ with gradient descent, the prediction changes like this:

[iteration 0] ....... $p_i = [0.3, 0.6, 0.1]$
[iteration 1] ....... $p_i = [0.4, 0.5, 0.08]$
[iteration 2] ....... $p_i = [0.55, 0.3, 0.02]$
...........................................................
[iteration n] ....... $p_i = [0.98, 0.001, 0.0001]$

So it seems like gradient descent is trying to increase the value for index 0 and decrease value for the other indices.
Based on my understanding, cross-entropy is only evaluated on the correct class, so the other terms will be zeroed out because of the ground truth array $y_i = [1, 0, 0]$:

$L = -\sum_{i=1}^C y_ilog(p_i) = - 1 * log(0.3) = 1.2$

I don't understand the mechanism here. Is it the gradient descent doing these modifications or is it the cross-entropy loss we define or is it the combination of both? How does the prediction score for the correct class increases and the prediction score for the incorrect classes decrease while optimizing?

The value of the loss function depends upon the prediction (which is a function of the input data and the model parameters) and the ground truth. Gradient descent works like this:

Initialize the model parameters in some manner.
Using the input data and current model parameters, figure out the loss value of the current network weights and biases.
Figure out how to update the weights and biases such that the loss value improves.
Update the weights and biases a certain amount based on the current learning rate.

This process is repeated until convergence (as measured in a few different ways, say improvement against a validation set).

If you want to dig in to how this all works, start with 3Blue1Brown's Neural Network series on YouTube, it's not even that long. It's a good start to get a feel for the concepts before you dig in to the math.


Why fitting/training a model can be considered as learning? I was looking around and I can't find a good answer, I just want to know why it can be considered as learning and is not just "calibration" or "parametrization".
I feel the word "learning" is overqualified for the things the models do.
Thanks in advance.

When a baby tunes the connections between neurons in his brain until he can recognize a dog and say "this is a dog!" we say that he learned.
Why wouldn't we say the same about neural networks (or other models) that happened to be inside a computer and not in a human brain?


Does dropout require multiple passes of the same data set, as a sort of ensemble method? I'm a bit confused about dropout -- on one tutorial, it was described as basically an 'ensemble method' of sorts. This implies that you might need to create an ensemble of networks. Is this the case, where you would need to get a consensus from the ensemble at the end? Or is dropout run over one single network?

Dropout is applied over one network. Sometimes (like with non-dropout networks) you will run your data through it multiple times before it converged and this number will be a bit higher on average with dropout but it is one network. Per layer you have a dropout probability during training and during testing/prediction you use the full network without dropout by multiplying the weights by (1-p) where p is the dropout probability. This is because during training (1-p) of the nodes are actually used.

Why this is related to ensembles is that every training instance is basically trained on a different network, by randomly dropping out nodes, forcing it to learn specific things using different nodes because it will not have all the nodes available at all times. It is not a traditional ensemble in that you combine multiple networks, just during training it acts a bit like it.


How can I evaluate data mining model? I will evaluate classification models I made. That's logistic regression and decision tress.
1. What I use standards for comparison?
2. Suppose model selection's standard is ASE. One is high ASE of training data, low ASE of test data, and the other is ASE of training data is low and ASE of test data is high.. If you select a model, which models do you choose?

Accuracy (for classification problems)
Precision
Recall
F1 Score
AUC-ROC, particularly for imbalanced datasets

Good performance on the training set and bad performance on the test set is due to overfitting. So you should try to find ways to tackle overfitting, such as regularization of parameters, parameter tuning using cross-validation etc...


LSTM/RNN model fails on new test data - TFLearn i'm fairly new to ML and at the moment i'm trying to develop a model that can classify spoken digits (0-9) by extracting mfcc features from audio files.

I trained the model on a data set that consists of 15 speakers and 2400 training examples (240 audio examples for each digit). After 3000 epochs the model has achieved 97% accuracy.

The problem is, when i record my own digit wavs, the model fails to classify them correctly. Why is this happening and what could i do to fix this issue? Is this an example of overfitting?

Extracting mfcc features

  wave, sr = librosa.load(wav_file, mono=True)
  mfcc = librosa.feature.mfcc(wave, sr)
  mfcc = np.pad(mfcc,((0,0),(0,80-len(mfcc[0]))), mode='constant', constant_values=0)

Parameters

learning_rate = 0.0001
training_iters = 150
batch_size = 64

width = 20  # mfcc features
height = 80  # (max) length of utterance
classes = 10  # digits

Network building

net = tflearn.input_data([None, width, height])
net = tflearn.lstm(net, 128, dropout=0.8)
net = tflearn.fully_connected(net, classes, activation='softmax')
net = tflearn.regression(net, optimizer='adam', learning_rate=learning_rate,
      loss='categorical_crossentropy')

Training

model = tflearn.DNN(net, tensorboard_verbose=0)

EPOCHS = 20
for i in range(training_iters):
    model.fit(X_train, y_train, n_epoch=EPOCHS, validation_set=(X_val, y_val),
          show_metric=True, batch_size=batch_size)

It looks like you are using the same training and test data for each iteration. This is causing your model to be overfit.

You need to use cross-validation. I would recommend using fewer training iterations and doing the following:

Split your training data into k chunks where k is your number of training iterations
At each training iteration make your training set by combining k-1 chunks and your validation data the remaining chunk. Note that this should be done so that your validation and training sets are never the same across two training iterations. Additionally the same chunk should never be in both your train and validation sets.

You can keep the the accuracy metrics on each training iteration if you want to see how your model's accuracy is converging, but the true accuracy of your model can only be measured on a holdout set which was not used in this validation scheme.

If you have enough data I would recommend doing an 80/20 train/validation initial split on your data, doing steps 1 and 2 on the 80% split and measuring the accuracy on the 20% split.

If you really want to be careful and have a lot of data you could do the following:

Perform a 60/20/20 train/validation/test split
Split both the train and validation sets by the number of iterations you are using.
At each iteration, use k-1 chunks of training data and validate against one chunk from the separate validation set.


Why is the optimal C chosen by GridSearchCV so small? I'm trying to use GridSearchCV to select the optimal C value in this simple SVM problem with non-separable samples. The issue I'm having is that when I run the code the optimal C is chosen to be ridiculously small (~e-18) so that the margin is expanded to contain all samples. Even when I alter the samples so that they are easily separable, the optimal C is still on the scale of e-18. GridSearchCV selects a very small C however I try to alter the samples. Does anyone know why this is happening?
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets.samples_generator import make_blobs
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import GridSearchCV

X, y = make_blobs(n_samples = 500, centers = 2, random_state = 6,
                  cluster_std = 1.2)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.scatter(X[:,0], X[:,1], c = y, cmap = 'rainbow', s = 30,
           edgecolors = 'white')
ax.set_xlabel(r'$x_1$', fontsize = 20)
ax.set_ylabel(r'$x_2$', fontsize = 20)

svc = SVC(kernel = 'linear')
c_space = np.logspace(-20, 1, 50)
param_grid = {'C': c_space}
svc_cv = GridSearchCV(svc, param_grid, cv = 5)
svc_cv.fit(X, y)
c = svc_cv.best_params_['C']
svc.C = c
svc.fit(X, y)

support_vecs = svc.support_vectors_

x1_min = min(X[:,0])
x1_max = max(X[:,0])
x2_min = min(X[:,1])
x2_max = max(X[:,1])
x1 = np.linspace(x1_min, x1_max, 100)
x2 = np.linspace(x2_min, x2_max, 100)
X1, X2 = np.meshgrid(x1, x2)
points = np.vstack([X1.ravel(), X2.ravel()]).T
boundary = svc.decision_function(points).reshape(X1.shape)
ax.contour(X1, X2, boundary, colors = 'k', levels = [-1, 0, 1],
           linestyles = ['--', '-', '--'])
ax.scatter(support_vecs[:,0], support_vecs[:,1], s = 250, linewidth = 1,
           facecolors = 'none', edgecolors = 'k')

```

Have a look at svc_cv.cv_results_: there are many values of C that tied for best, with accuracy 99.6%, and the chosen C is the smallest of those.  The point is that the width of the margin doesn't affect the actual hyperplane very much, and so the accuracy score doesn't change very much.
A few suggestions:

With larger and less-separable data sets, this might get minimized, since small changes in the hyperplane would be more likely to classify points differently.

For this case, where many scores are exactly equal, you might prefer to tie-break for larger values of C.  This can be easily accomplished by just reversing the order of the list c_space, or more robustly by defining a custom scorer for the search, that takes the mean test score plus some function of C.

Something more refined than accuracy as the scorer for the search could help separate the values for different C, but it's not clear what would be best.  Something like log-loss would require calibrating probabilities; maybe AUROC?


How to deploy machine learning models as a chrome extension? I have trained a stance detection model using SVMs. Wanted to know how can I deploy this as a chrome extensions. I do understand that the question is a bit broad but any links, suggestions etc. will be appreciated:)

Since what you want to do is to apply a pre-trained model and use the predictions on the client side, you have two options:

Client-side application of the model: the extension would have to download the model file and store it, then it would need to compute the predictions from this model. This would probably require the SVM library used for the training to be available on the client side, this might be an issue.
Server-side application of the model: the extension would have to upload the data to the server, then the server computes the predictions using the model, and sends the predictions back to the client.

Usually option 1 is preferable since it doesn't require transfering the input data to the server and then sending back the predictions. It's also safer with respect to the user privacy, their data is not uploaded anywhere. However it requires the client to be able to compute the predictions, so probably requiring the SVM library to be installed and the extension must be able to communicate with it, so it might not be feasible.


in TensorFlow 2.0, what is the different between LSTM and LSTMCell objects? I am trying to implement an RNN in TensorFlow 2.0 (beta1). Looking at the layer functions (inherited from Keras) I found:

tf.keras.layers.LSTM

and

tf.keras.layers.LSTMCell

What is the different between the two? If you take a look at their arguments, they seem the same.

You are right - the difference is minimal. The base LSTMCell class implements the main functionality required, such as the build method, whereas the LSTM class only container an entry point: the call method, as well as a bunch of getters to retrieve attribute values. LSTMCell is the base class, which is used as a cell that is used inside the LSTM class.

All links point to relevant parts of tensorflow.keras source code.

My advice would be to use the standard LSTM class in your model as a normal layer. If you have a GPU at your disposal, you might want to use the version of the layer that is optimised with CUDA for execution on a GPU. As per the documentation:

  Note that this cell is not optimized for performance on GPU. Please use
    tf.keras.layers.CuDNNLSTM for better performance on GPU.

There is also a GRU layer as well as a CuDNNGRU layer.

If you want to tweak how things work under the hood, you might create a class and inherit from the LSTMCell, or even the base class:

from tensorflow.python.keras.engine.base_layer import Layer

class MyLSTM(Layer):
    pass

But you would have to implement many things for yourself.


Can clustering my data first help me learn better classifiers? I was thinking about this lately. Let's say that we have a very complex space, which makes it hard to learn a classifier that can efficiently split it. But what if this very complex space is actually made up of a bunch of "simple" subspaces. By simple, I mean that it would be easier to learn a classifier for that subspace.

In this situation, would clustering my data first, in other words finding these subspaces, help me learn a better classifier? This classifier would essentially be an ensemble of each subspace's classifier.

To clarify, I don't want to use the clusters as additional features and feed it to a big classifier, I want to train on each cluster individually.

Is this something that's already been done/proven to work/proven to not work? Are there any papers on it? I've been trying to search for things like this but couldn't find anything relevant so I thought I'd ask here.

It is absolutely a way to improve your classifier's accuracy. Actually a "strong" enough classifier such as a neural network could be able to learn by itself these clusters. However, you would need a substancially deeper network.

The "smartest" way to do this, if you know there are many groups/clusters in your data is to actually perform a 2-steps process:

Cluster your data
Train X models, one for each of your clusters

A nice way to visualise this is the following problem, you want to build a recommendation engine for a Netflix-like application, you don't want to build one model per person, how would you do this ?

First find clusters of similar users (geeks, SF fans, teenagers, etc.)
Fit one model for each of these clusters


What are the 2 classes of categories to help define a problem While studying about machine learning, I've learnt the importance of defining your problem before getting started trying to model it.

I can see 2 types of problem categorification:

Supervised / unsupervised / reinforcement algorithms
Classification / clustering / regression / ranking

Example definitions found on the net:

First type:

Supervised Algorithms: The training data set has inputs as well as the desired output. During the training session, the model will adjust its variables to map inputs to the corresponding output.
Unsupervised Algorithms: In this category, there is not a target outcome. The algorithms will cluster the data set for different groups.
Reinforcement Algorithms: These algorithms are trained on taking decisions. Therefore based on those decisions, the algorithm will train itself based on the success/error of output. Eventually by experience algorithm will able to give good predictions.

Second type:

Classification: You want an algorithm to answer binary yes-or-no questions (cats or dogs, good or bad, sheep or goats, you get the idea) or you want to make a multiclass classification (grass, trees, or bushes; cats, dogs, or birds etc.) You also need the right answers labeled, so an algorithm can learn from them.
Clustering: You want an algorithm to find the rules of classification and the number of classes. The main difference from classification tasks is that you don’t actually know what the groups and the principles of their division are. For instance, this usually happens when you need to segment your customers and tailor a specific approach to each segment depending on its qualities.
Regression: You want an algorithm to yield some numeric value. For example, if you spend too much time coming up with the right price for your product since it depends on many factors, regression algorithms can aid in estimating this value.
Ranking: Some machine learning algorithms just rank objects by a number of features. Ranking is actively used to recommend movies in video streaming services or show the products that a customer might purchase with a high probability based on his or her previous search and purchase activities.

Do each type of categories have a name ?
And are these types correlated or independent ?

I never saw a name for these categorifications. However, it seems to categorize by type how target variable exists or what is the type of target variable (in supervised and semi-supervised) or input variable (in unsupervised)

[wrong] 2. I think in data mining we call "Reinforcement Algorithms" semi-supervised.

I think Ranking will be a regression problem. just after defined number you can change them to ranks based on their order.
There is some correlation between the first and second type of categorifications you named, you can see that in this picture:

https://towardsdatascience.com/supervised-vs-unsupervised-learning-14f68e32ea8d


Is splitting the data set into train and validation applicable in unsupervised learning? I am having a tough time implementing all the steps of setting up support vector machine (SVM) for unsupervised learning. My data set is labelled but for educational purposes I am learning unsupervised methods especially how to use SVM in an unsupervised way.

Basically, in supervised learning we have a data set X that is split into Xtrain Xval and an independent Xtest which is never seen by the SVM during training. During training, the hyperparameters are learned using cross-validation (CV approach). My question is that for unsupervised learning should I need to have Xtrain, Xval and CV for learning the hyperparameters? The reason I ask is that theoretically in unsupervised learning we don't have any labelled dataset. Does this mean that we don't have Xtrain and Xval?

In the Matlab link, cross validation is done but it is unclear to me if that is done on part of the training data Xval

CVSVMModel = crossval(SVMModel);
[~,scorePred] = kfoldPredict(CVSVMModel);

QUESTIONS:

1) Do we estimate the hyperparameters in SVM for unsupervised learning?

2) If yes, then this implies that we have access to labels when theoretically unsupervised learning is without knowing the labelled data set. This is contradictory to me.

How do I estimate the hyperparameters in unsupervised SVM without Xval and splitting the dataset using c = cvpartition(n,'KFold',k)

Can sombebody please suggest how to go about unsupervised learning. Thank you.

For an unsupervised technique, if you have some metric of "goodness of fit", it makes sense to have a train-test split. In your case, it seems as though you'd want to split the data so that you can see if your SVM classifies the test data as outliers.

However, your goal is unclear. It might help if you clarify what you mean.


How to extract audio features for each video frame using pyAudioAnalysis I'm trying something like extracting audio features for each video frame. I know there are 30 video frames and 16000 audio frames per second in the video file. I'm using pyAudioAnalysis python lib to achieve the goal but no success.
Here is my code.
from __future__ import print_function
from pyAudioAnalysis import audioBasicIO
from pyAudioAnalysis import ShortTermFeatures,MidTermFeatures
import matplotlib.pyplot as plt
import os,shlex, subprocess
import pandas as pd
import numpy as np

# -acodec pcm_s16le  -vn -ar 16000
command_line = "ffmpeg -i test.mp4 -ac 1 -ar 16000  -vn test_mono.wav"
#command_line = "ffmpeg -i test.mp4 -ab 160k -ac 2 -ar 44100 -vn test_stereo.wav"
args = shlex.split(command_line)
print(args)
processResult = subprocess.call(args) # Success!
print(processResult)
[SamplingRate, signals] = audioBasicIO.read_audio_file("test_mono.wav")
print(SamplingRate)

Output

['ffmpeg', '-i', 'test.mp4', '-ac', '1', '-ar', '16000', '-vn', 'test_mono.wav']
1
16000

#mid_feature_extraction(signal, sampling_rate, mid_window, mid_step,short_window, short_step):
MidFeatures,ShortFeatures,MidFeatureLabels=MidTermFeatures.mid_feature_extraction(signals, SamplingRate, 0.043*SamplingRate,
                                                                                0.043*SamplingRate,0.00016*SamplingRate,
                                                                                0.00016*SamplingRate)
print('Mid Features Extr Success')
MidFeatures_Dataframe = pd.DataFrame(data=MidFeatures.transpose(), columns=MidFeatureLabels)
print(type(MidFeatures))
#print(MidFeatures)
print(type(ShortFeatures))
#print(ShortFeatures)
print(type(MidFeatureLabels))
#print(MidFeatureLabels)
MidFeatures_Dataframe.to_csv('Audio_MidFeatures.csv')
print('Mid Features File Success')

Output

Mid Features Extr Success
<class 'numpy.ndarray'>
<class 'numpy.ndarray'>
<class 'list'>
Mid Features File Success

As per my calculations I should get 338 rows of audio features, and after a long time of struggle I'm getting 326 with the above parameters but still don't know how. If anyone can help me how, window and steps are working here. I know the basic concepts of window and step as work in CNN but not getting in this context.

I don't know all your calculations, but looking at the doc in the code here, short_window and short_step should be in samples (probably as well as mid_window and mid_step).
However in your code:
#mid_feature_extraction(
    signal, sampling_rate,
    mid_window, mid_step,
    short_window, short_step
)
midFeat,shortFeat,midFeatLabels=MidTermFeatures.mid_feature_extraction(
    signals, SamplingRate,
    0.043*SamplingRate, 0.043*SamplingRate,
    0.00016*SamplingRate, 0.00016*SamplingRate
)

it seems that short_step=short_window=0.00016*16000=2.56 which is not in samples. So it will be cast to integer and will be equal to 2 instead of 2.56.
Hope it helps.


How can I calculate mean and variance incrementally? Say I have a set S of values, and want to store in a database some summary information about that set, so that later when I acquire a new value v I can make a reasonable estimate of what the summary information would be about the set S ∪ {v} --- although by now I no longer have access to the original members of S. I'd like the summary information to include the mean and variance of these sets, and as minimal additional information as needed. A natural idea for the additional information would be S's cardinality. But I'm willing to save more complicated information about S if needed. My main constraint is to minimize the size of the retained information.

If I only cared about the mean of the sets, then storing the mean plus cardinality of S would obviously be enough. I could update with a new value by just taking a weighted average of the old mean (times the old cardinality) and the new value. But I'd like to be able to keep track of the variance of the sets too. A good estimate is enough; I don't need to be able to reconstruct what the exact mean and variance of S ∪ {v} would be.

I expect that even asking this displays how naive I am about statistics, but I'd appreciate any help. I don't know where to look for answers.

This problem was discussed, with proof and some alternate methods over on math.stackexchange.


Reducing features to 2 by PCA confusion Matlab I am trying to reduce $500$ features to $2$ as an assignment. I wrote the following code and I am deeply concerned if it is true as when I plot it on the graph it does not look good. It should look good as I have to apply cluster algorithm to is as a next step. Can you help me pls?
X = data;

X = bsxfun(@minus,X,mean(X));  %%standardize

CM = transpose(X)*X;  %%coveriance matrix is calculated

[V,D] = eig(CM);
eigenvalues = zeros(1,306);

for k = 1:306
    eigenvalues(1,k) = D(k,k);
end

sortedEigenValues = sort(eigenvalues,'descend');  %%eigenvalues are sorted
sortedEigenVectors = zeros(306,306);

for k=1:306  %%eigenvectors are sorted according to eigenvalues
    for l=1:306
        if sortedEigenValues(1,k) == D(l,l)
            sortedEigenVectors(:,k) = V(:,l);
        end
    end
end
projectionMatrix = sortedEigenVectors(:,1:2); %% first to columns of sortedEigenVectors matrix are taken to create projection matrix
projectedMatrix = CM*projectionMatrix; %%projectedMatrix is what I am intending to use.

figure
stem(projectedMatrix(:,1),projectedMatrix(:,2)); %%plotting altogether

I have found eigenvalues and sort them and also sort eigenvectors and I take first $2$ eigenvectors and multiply it with the coveriance matrix and hope that this new matrix is what I am supposed to cluster, but I am very confused. Can you give some feedback?

After obtaining the principal component, you should compute $XW$ rather than $X^TXW$ where $W$ is the principal component.
Remark about computation efficiency: As you sort the eigenvalues, you could have obtained the corresponding permutation indices.


bert-as-service maximum sequence length I installed bert-as-service (bert-as-service github repo) and tried encoding some sentences in Japanese on the multi_cased_L-12_H-768_A-12 model.  It seems to work as I am getting vectors of length 768 per word but np.shape() shows this for each sentence:

np.shape(vec_j[0]): (25, 768)
np.shape(vec_j[1]): (25, 768)
np.shape(vec_j[2]): (25, 768)
np.shape(vec_j[3]): (25, 768)
np.shape(vec_j[4]): (25, 768)
type: <class 'numpy.ndarray'>

My sentences are short so there is quite a bit of padding with 0's.  Still, I am unsure why this model seems to have a maximum sequence length of 25 rather than the 512 mentioned here:  Bert documentation section on tokenization

  "Truncate to the maximum sequence length. (You can use up to 512, but
  you probably want to use shorter if possible for memory and speed
  reasons.)"

The default setting for max_seq_len is 25 as seen here under heading Server API:  bert-as-service readme

There is an open issue regarding this on the Github repo here and the creator seems to be implementing a feature:  bert-as-service issues


Will Keras fit( ) function automatically shuffles the input dataset by default? I am asking this model fit( ) function.

fit(x=array_x, y=array_y, batch_size=32, epochs=10)

The question is straightforward:

Whether fit( ) will automatically shuffle the input dataset? (array_x, array_y in this case)

I ask this question because I find my fit( ) and fit_generator( ) has different outputs, with same input. My generator feeds inputs to fit_generator( ) in order. I guess the reason is about shuffle.

Yes, by default it does shuffle.
Here is the documentation.
The default call signature:
fit(x=None, y=None, batch_size=None, epochs=1, verbose=1, callbacks=None,
    validation_split=0.0, validation_data=None, shuffle=True,
    lass_weight=None, sample_weight=None, initial_epoch=0,
    steps_per_epoch=None, validation_steps=None, validation_freq=1,
    ax_queue_size=10, workers=1, use_multiprocessing=False)

where the description for shuffle is:

shuffle: Boolean (whether to shuffle the training data before each epoch) or str (for 'batch'). This argument is ignored when x is a generator. 'batch' is a special option for dealing with the limitations of HDF5 data; it shuffles in batch-sized chunks. Has no effect when steps_per_epoch is not None.

This means it will not shuffle if you have set a value for steps_per_epoch.


Can we Create Neural network(Simple one such as Multi Layer perceptron) that only contains positive weights only? I was wondering if there is a specific method to create a well performing neural network with only positive weights (I already tried clipping the weight before training or so and initializing the weights with only positive value ) but still doesn't give a good results so is there any other method to do so ?

Thank You

One way to do that is to customize the formula to update weights.

Normally, the formula is something like this:

weight = weight - learning_rate * error

You can change it to:

weight = max(0, weight - learning_rate * error)

This will perform well or not? I have no idea. We can't say much about the guaranteed performance of neural network models in general.

However, if by good results, you mean a model with all positive weights, I believe this one will qualify.


How to build the following the string in R? I am trying to build a long string of the form.....\'name1\',\'name2\',\'name3\',..etc. How do I do it ? I am trying to do it using R ? By hand is impossible as there are currently over 200 such names and in future I may have more. So I need to automate the process. However, every time I tried to use R's paste0 function I am getting a different result. Here is what I did:

t <- c()
for( nms in name){ t <- paste0(t, paste0("\\'",nms,"\\'", sep = ",") ) }

However, the result turns out to include double backslashes instead of one, like this "\\'name1\\',\\'name2\\',\\'name3\\' etc. I couldn't get a single backslash as I want.

Any help would be appreciated.

Thanks.

The way R displays the string (using print command) is different from the actual value of this string. Even though you see 2 backslashes for your output, the actual string contains a single backslash. This is because when you type the name of your variable it shows the value within the quotes and within quotes it should use 2 backslashes. When you display the value of the sting using message() or other similar function, when it just prints the value without quotes, then you will see a single back slash only:

#create a vector of names (words)
words <- c("word1","word2","word3")

#add prefix and suffix part to each word
words_vec <- paste0("\\'",words,"\\'")
words_vec
"\\'word1\\'" "\\'word2\\'" "\\'word3\\'"

#Combine the vector into the string
line <-paste(words_vec,collapse='')
line
"\\'word1\\'\\'word2\\'\\'word3\\'"

#Check how the string looks:
message(line)
\'word1\'\'word2\'\'word3\'

By the way, the above (vectorized) method is more efficient than the for-loop method you are using to put all names together.


random forest and log_loss metric? Light gbm has the metric with log_loss for binary or multi classification.

is Random Forest also has the loss function with log_loss?

Yes, there are decision tree algorithms using this criterion, e.g. see C4.5 algorithm, and it is also used in random forest classifiers. See, for example, the random forest classifier scikit learn documentation:

  criterion: string, optional (default=”gini”)
  The function to measure the quality of a split. Supported criteria are “gini” for the Gini impurity and “entropy” for the information gain. Note: this parameter is tree-specific.

Please note that usually it is referred to as cross-entropy or information gain rather than log loss.

For some background information you can give this and the links provided in the answers a read (although they may refer rather to decision trees):
When should I use Gini Impurity as opposed to Information Gain?


should we include or exclude a variable in a logistic regression based on the description below? should we include or exclude a variable in a logit regr. model which will only obtain values if a certain event takes place otherwise will show N/A? this variable tells whether or not a product will be bought based on calls made by the company. the variable tells: number of days that passed by after the client was last contacted from a previous campaign. the values (-1) are for users who

You may want to add an indicator variable, 0 if the event has happened and 1 if it has not.  In a linear model, the coefficient on the original can fit to the "real" slope, while the coefficient on the indicator "fixes" for the missing values (and whatever imputation you use).

https://stats.stackexchange.com/a/105258/232706
https://datascience.stackexchange.com/a/64709/55122
https://datascience.stackexchange.com/a/69841/55122


How to detect phrases from an English sentence. The question is not about detecting keyphrases. It is about detecting a combination of words makes a valid phrase or not. For example,

"John reads New York Times in New York."

Here, the phrases are

New York Times
New York

Detecting keyword phrases is a Text Summarization problem, however, here it is classifying whether a combination of words make a valid phrase or not.
There have been a few algorithms we have gone through including but not limited to models.phrases – Phrase (collocation) detection of Gensim, however, we are looking for better results.

The name for what you want to do is Named Entity Recognition (NER).  There are several great software packages for this, namely:

Stanford NER
MIT's Information Extractor
spaCy