Appropriate model metric for a truncated response variable? Here's a straightforward question I can't seem to find a good answer to. Let's say you're using some variables to predict age. I'm assuming a regression model is the right approach. In this case, what would be a suitable metric to evaluate the model performance? My sense is that MAE and RMSE aren't appropriate because they assume your predictions can be equally wrong in both directions, but that's not true: predictions can be larger in the positive direction than in the negative direction. For example, if the true age is 5, the prediction can be no less than 0, but could be infinitely high.

Is RMSE/MAE an appropriate metric to evaluate the size of the error, and if not, what would be? Or if there isn't an appropriate "size" metric, should I use R^2 or some other "fit" metric?

Either RMSE or MAE will give you a sense of accuracy in prediction.  The exact value of any error metric is up to your specific use case and tolerance.

In other words, if you are predicting ages of 5-10 year olds and see a RMSE of 5 then you might think that you have a significant problem.  But if you are predicting ages expecting them to be 40-80 year olds, then a RMSE of 5 wouldn't be a big concern right?


where does kmeans store its trained model parameters in scikit learn? Like in linear regression, there is model.intercept_ and model.coef_ but in kmeans i found  kmeans.inertia_ , I dont know what is that.

K-means stores 4 model parameters namely cluster_centers_, labels_, inertia_, n_iter_. For details on what they store you can look at the official documentation. Also, Inertia, or the within-cluster sum of squares criterion, can be recognized as a measure of how internally coherent clusters are. You can see a full description here under the section: 2.3.2. K-means.


Hyperparameter tuning for stacked models I'm reading the following kaggle post for learning how to incorporate model stacking

http://blog.kaggle.com/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice/
in ML models. The structure behind constructing the 5 folds and creating out of sample predictions on the training data makes sense for the purpose of building the meta model or the model on top of the base models. However i'm not sure how it uses hyper parameter tuning especially for the base models.

So the concept of getting out of sample predictions makes sense to me. We essentially for each of the 5 folds use the other 4 folds to train and then predict on the fifth. So how do we actually hyper parameter tune the base models on this same dataset without adding bias, it's seems to me that this is not possible?

Note i'm making the assumption that there is no more data available to use. I'd appreciate any help!

Do not mix dividing the data into k-fold with cross validation.

You can use the 4 folds ( training data) to optimize the base classifiers. You can also find the best hyper parameters by applying cross validation on your training data, re-train using all training data ( the 4 folds) and then test using the last fold to generate the meta data.

After you finish generating the meta data you can now use all the data the 5 folds to train the base learners. The higher the convergence between the final set of base classifiers and the ones you used to generate the meta data the better. This is why the less the data you have the larger the k you need to go with and vice versa.

Finally, you use the meta data to train the meta classifier

Here is one last note to think about: using cross validation to optimize base learners my not be very beneficial, but why?
If you have access to N different training algorithms and you can use cross validation to optimize them and choose the best one ( the one that leads to a very low bias error), then there is a high probability that you don't need to use stacking. But stacking is very beneficial when all these N algorithms can not lead to 0 bias error (if you have variance error use bagging), it is a case that may confuse the cross validation algorithm. The other thing is using cross validation to optimize algorithms may lead to different algorithms that have similar biases, which reduces the benefits from the stacking technique.

you need to know that stacking, is one of the most tricky ensembles and this is why it is not well studied in literature in compare to bagging and boosting. In the other hand it is proven that it is very important, especially for practical purposes.


what is the meaning of independent and identically distribution of samples in a dataset for neuralnetworks random variables like heads or tail that generated by flipping a coin is independent because each time we tossing the result isn't depend on previous toss( in other words the function that generates random variable doesn't have memory). the example also is identically distributed because each toss result follows same distribution (binomial distribution).
but what is the meaning of IID in the case of images classification( like hand written digits dataset). each sample is independent from previous samples but how we ensure they generated with the same probability distribution(or what is the meaning of that in this specific case)?
or in the case of recurrent neural network after that we create over sample (according to the constrains of the problem) from the series. we can do shuffling to ensure the samples in dataset are independent but how about identically distribution of samples?

Once the data is with us, the best we can do is shuffling the samples to make them Identically distributed assuming a hypothetical process behind it that generates all the samples.
Let's take the example of MNIST digits, the hypothetical process is our similar knowledge of digits, so most of us will write a specific digit in a similar way.
Now let's test the MNIST model on Japanese MNIST, it will fail as the assumption is broken.
Again, mix both the MNIST, Shuffle it, Split it and Train it. It will learn the pattern and can predict the digits.
Once again, both the train/test have a similar probability distribution of pixels.
While training the model,
If the process is incremental e.g. Gradient Descent, we try to assure that each batch represents the parent distribution. Again this is achieved by Shuffling esp. Stratified shuffling.


CNN Multi-class vs Binary Class Image Classification Suppose we have a training set of 3 classes of image: 1.Cats, 2.Dogs, 3.Neither cats nor dogs.
We're only really bothered about detecting whether an image is a cat/dog, or neither, but we don't care whether it's a cat or a dog, just that it's one of the two.

Is it inherently "better" to train a binary class CNN by merging the Dog and Cat classes into a single "positive" class rather than train a 3-class CNN?

If it is, does it then follow that if we did care about whether the image was a dog or cat, it would also be better to first run the binary classifier to decide if it was a dog/cat or not, then run a second binary classifier trained on just cats and dogs to decide which of the two it actually is?

I would train one classifier with 3 classes:

cat, dog, neither

Use cross entropy as your cost function.

There are many examples of this online. If you Google MNIST neural network you'll find many simple examples. Or you can take one of Google's trained networks and retrain the last layer.

Here is a convolutional being trained on the Cifar set of images.

Using a pre-trained network to tell two house cats apart.


shifting the last column in the dataframe to the first place While this question had relative answers, I couldn't find an efficient and robust way of doing it. I have to shift the last column in my data frame to the first without disturbing any of the other columns. How should I do this ? Thanks in advance.

cols = list(df.columns)
cols = [cols[-1]] + cols[:-1]
df = df[cols]


Visualization methods in R to examine correlation of labels against response Question
What are some good plotting methods in R for examining the relationship between a target variable and various explanatory variables? In particular, I'm looking for visualization techniques that scale to more variables than the traditional scatterplot matrix.
More details
The scatterplot matrix is a great tool for visualizing pairwise relationships between variables. For example, with the swiss dataset in R, we can easily plot a matrix of scatterplots.
library(datasets)
data(swiss)
plot(swiss[1:3])

which yields

I am interested in the case where I want to predict some response, say Fertility using some combination of explanatory variables. I want to closely examine how each explanatory variable correlates with Fertility. If I have many columns in my dataframe, using plot(swiss) becomes unwieldy.
For example, the following plot (generated following instructions here) shows pairwise correlations for all columns in a dataframe. If I could plot something like this but only showing correlations between Fertility and other columns, that would be useful.
library(datasets)
data(swiss)
plot(swiss[1:3])

library(devtools)
library(inspectdf)
library(tidyverse)
library(readr)

show_plot(inspect_cor(swiss))

which yields

Below are two functions using my favorite packages:

The first one shows a scatterplot of every column against the target column
The second one shows the correlation of every column with the target column, with confidence intervals (I found how to do that with ggplot here).

Code:
library(ggplot2)
library(reshape2)
library(plyr)

scatterplot <- function(data, targetColumn='Fertility') {
  d<-melt(data,id.vars = targetColumn)
#  ggplot(d, aes_string('value',targetColumn))+geom_point()+facet_grid(variable~.)
  ggplot(d, aes_string('value',targetColumn))+geom_point()+facet_wrap(variable~.)
}

corplotCI <- function(data, targetColumn='Fertility', method='pearson') {
  d<-ldply(colnames(data), function(col) {
    if (col != targetColumn) {
      r <- cor.test(data[,col], data[,targetColumn],method=method)
      data.frame(variable=col,cor=r$estimate, lowerCI=r$conf.int[1],upperCI=r$conf.int[2])
    }
  })
  ggplot(d,aes(cor,variable))+geom_point(size=3)+geom_errorbarh(aes(xmin = lowerCI,xmax = upperCI),height=.5)+coord_cartesian(xlim=c(-1,1))
}

Usage:
scatterplot(swiss)

corplotCI(swiss)


Cleaning the univariate dataset with high noise At this time, I am having a dataset containing the operating duration for some sensors. This could be considered as a univariate dataset because it has only 1 dimension.

For example:

[1]: [10, 12, 13, 15, 16] indicates that the sensor [1] will have some operating duration like [10, 12, 13, 15, 16].

I want to see the range of operating duration for each sensor, by measuring the mean and standard deviation for each sensor. But my problem is in my dataset, each sensor has many noises. For example:

[1]: [1, 1, 1, 1, 1, 2, 2, 2, 10, 12, 13, 15, 16, 200, 400, 500].

You could see that sensor [1] has many noises like 1, 2, 200, 400 and 500. In my dataset, there are many cases like this.

Without removing the noise, the standard deviation is always smaller than the mean. This makes my duration analysis not meaningful.

So my question is: I want to ask if there is any method for removing the noises like that in my dataset.

Thank you.

There are multiple algorithms for this. On one of the kaggle competitions I saw applying several algorithms and then throwing the data away by the number of votes from all algorithms. The algorithms that I remember:

Remove all points above or below 2 standard deviations. X>mean+2*std dev, X < mean-2*std dev. 2 is actually a parameter you need to optimize.
Remove the 5% of the highest and lowest values. X>95% quantile, X<5% quantile. 5% is a parameter you need to optimize.
X - median > 2*1.4826*MAD (median absolute deviation. 1.4826*MAD is an estimator for standard deviation under some assumptions). And the same for lower bound. https://eurekastatistics.com/using-the-median-absolute-deviation-to-find-outliers/

Repeat several times until no points are thrown away.

Example of this method with voting from some other kaggle winners: https://nycdatascience.com/blog/student-works/kaggle-predict-consumer-credit-default/


Handling binned feature I am new to the field of data science and trying to figure out ways to handle data quality issues before performing any modeling.

I am working on a house rental price data set. In this data there is feature called Total Squarefeet. The issue I am facing here is that out of 12000 records 200 have a range, for e.g., 1200 - 1800, OR 850 - 855, OR even and these ranges also have a random difference between them. Rest other are simple numbers. Is there a way to correctly handle this kind of data. Can anyone help me or guide me to a place from where i can learn and use techniques to handle such data. Thanks in advance.

The column you are using must be of string data type. First filter the dataset where it contain "-" and create another dataframe. On this new dataframe perform split string operation and create two different column and convert them to integer. Now create another column which has average or whatever you prefer with two column. Rename column and drop unnecessary columns.  Merge it back to dataframe.


can we learn a model to pre-process text? I'm in a very dire situation where I have to preprocess the text but the text in the documents is very random. It is in the form of numerical points.

I want to remove a certain class of points  (like bullet points) from those documents but simple preprocessing is not helping as the points are appearing randomly anywhere in the document.

I was thinking of learning a model where the model identifies the points which I want to remove.

Is it viable? And If so what approaches should I look for.

[Natural language processing]
Thanks!

Based on the details given in the comments, it looks like what you need is a binary classifier which predicts whether or not to remove a "numerical point". This means that each instance is an individual "numerical point", and you would need to annotate a sample of instances to use as a training set.

A simple option is to use a bag of words representation, i.e. represent every instance as a vector over the vocabulary where each cell contains the frequency of the corresponding word (or other variants, e.g. binary or TF-IDF weights). Then many classification algorithms can be used: decision trees, SVM, Naive Bayes, etc.


Perform Person Re-Identification on custom image set I've been recently using the YOLO system to detect people on images, which turned out really well.

My next step is to try to find images of the same person across the whole set of images I've retrieved earlier. I came across the AlignedReID: Surpassing Human-Level Performance in Person Re-Identification paper, read it through and found an implementation of it that offers pre-trained weights.

However, I'm kind of lost when trying to apply it to my (unlabelled) custom dataset. To me, the input of the system is an image and its outputs are k candidate images, ranked accordingly to the system's confidences. In other words, the system will try to match an image with the other images he works with.

Since the images I retrieved are unlabelled, I'd like to avoid re-training anything. But the more I think of it, the less it seems possible to me.

Can I load the pre-trained weights, define the images I retrieved as the test set and run it? Or should I re-train the last layers to adapt it to my dataset?

In other words, how generalisable are person re-identification systems?

I haven't tried testing images from a custom dataset on AlignedReID but I have used another implementation laid out in the paper In Defense of the Triplet Loss for Person Re-identification.

The model is a simple feed-forward CNN which returns a 128 dimensional feature vector for each image. You can find the feature vectors for test images from your custom dataset and associate images with identities based on Euclidean distance for instance.

You can find the Github repository here.


Program to fine-tune pre-trained word embeddings on my data set I am looking for a program that would allow me to fine-tune pre-trained word embeddings on my data set. Ideally, open source and working on Linux or Windows.

While I am not aware of software specifically for tuning trained word embeddings, perhaps the following open source software might be helpful, if you can figure out what parts can be modified for the fine-tuning part (just an idea off the top of my head - I'm not too familiar with the details):

GloVe: Global Vectors for Word Representation (part of Stanford NLP Group software);
SENNA (its use for word embeddings is mentioned in this blog post);
Code on GitHub for Deep Learning-based word embeddings training;
Neural Probabilistic Language Model Toolkit, mentioned above (also Deep Learning-based).


LSTM with return_sequences - "Training a model on multiple timesteps simultaneously" So I'm following Tensorflow's LSTM/time series tutorial and there's something I don't understand. I do understand what happens with return_sequences on/off, however, it is stated, that with return_sequences on you allow

Training a model on multiple timesteps simultaneously

I don't quite understand what this means. If you keep it off, the input will go through all the previous timesteps, and make a prediction. Your loss will be the error of this prediction vs. the actual. You're still training on all the timesteps, right (although not simultaneously I guess?).
Now if it's on, it's making a prediction on each timestep - what's the loss in this case? Is it the average error for each prediction in the sequence? Why is it better to switch return_sequences on, as it's done in the tutorial? Does the model learn more or faster in this case? How does it impact the learning?
It is also stated that

With return_sequences=True the model can be trained on 24h of data at a time.

Isn't that still the case with return_sequences=False? In the example they're still teaching the model to predict 1 timestep based on the previous 24.

Setting return_sequences to True or False depends entirely on which one is appropriate with respect to how you are going to make predictions at inference time:

If you want your model to make a prediction only when it has consumed a whole sequence as input, then you use return_sequences=False, so that your loss is only based on the output after ingesting the whole sequence.
If you want your model to be able to make predictions after any number of timesteps, you set return_sequences=True and set the loss over all the time step outputs. As we are dealing with time series, the appropriate loss could be the average squared error over all time step outputs.

About the statement "With return_sequences=True the model can be trained on 24h of data at a time", I think it means that you are training the model for each of the 24h time steps, instead of the single prediction trained with return_sequences=false.


When to use Linear Regression and When to use Logistic regression - use cases Linear Regression is used to find the relation between dependent variable and independent variable.

Logistic Regression is used when dependent variable is categorical.

But this doesn't exactly suffice, what use cases you can use either of these, or can we use a combination of them?

Logistic Regression is used when you know that the data is lineraly seperable/classifiable and the outcome is Binary or Dichotomous but it can extended when the dependent has more than 2 categories.

Linear Regression is used to find the relation and based on the relation between them you can predict the outcome, the dependent variable should be numeric.

What kind of usecases are you expecting? give an example so that we can extend it further.


Averaging two Word2vec vectors to obtain a unified representation for single word I have been working on a trained data for Word2vec algorithm. Since we need words to stay as original we don't make them lowercase at the preprocessing phase. Thus there are words with different variations (e.g "Earth" and "earth").

The only way I can think of is to take average of vectors for "Earth" and "earth" to create a single vector to represent the word. (Since the feature vector's dimensions are similar)

Is this an "okay" method? If it is not, what might be a good way to handle this issue?

Note: Lowering all words in preprocessing is not an option for now.

Edit: The info about whether or not feature dimensions are truly linear would also be helpful.

Edit 2: Combining both answers from patapouf_ai and yazhi gave the best results. How are these combined? Weighted average improved the results but putting word frequencies through a scaled sigmoid function gave the best results, because using word frequencies in a linear manner gives them more importance than they bear.

Just averaging them might not be good because that would assume that they they have the same weight, and that is probably not the case if the capitalized and uncapitilized version appear with very different frequencies in your training data.

An incremental improvement would be to average them proportionally to their frequency in the corpus. So say Earth appears 159 times and earth 1239 times do something like:

v(Earth & earth ) = 159/(159+1239) * v(Earth) + 1239/(159+1239) *v(earth).

The vectors are supposed to encode semantics linearly, so this should give you a resonable approximation.


Replace values that starts with "<" in a data frame in r I am new to R and would like some assistance on replacing multiple values in a large dataset with many variables to NA at once in r.
I want all the cells with "<0.01, <0.02, and many others with < sign at the beginning" to be changed to NA.

How do I do it at once? Here is a screenshot of the data.


Predict ratings for Item Based Collaborative Filtering Given the (cosine) similarity score of top 100 neighbors of every item, how do I predict ratings for unrated items? Please explain in simple terms.

Item
1       260 0.577305 780 0.5655413 1210 0.5529503 3114 0.5425038 1270 .....

2       367 0.5202925 364 0.5093084 500 0.5082204 586 0.4978301 480 ......

.
.
.

You should at first calculate the similarity between co-rated items (items which both active user and another user rated them). then you can predict the rate of active user to the unrated item.
one of the methods for this work is Slope One


When should you remove outliers? Let's say I've found some outliers in a column in my dataset and have decided to remove them.

Should I do this before or after I split the dataset into train/test sets?

If you decided to remove outliers. Please remove them before the split(even not only before a split, it's better to do the entire analysis(stat-testing, visualization) again after removing them, you may find interesting things by doing this).

If you remove outliers in only any one of train/test set it will create more problems.
(EX: An outlier in train set may not be an outlier in combined/full set, also the model will have high variance if you do so)


Using pandas, check a column for matching text and update new column if TRUE My objective: Using pandas, check a column for matching text [not exact] and update new column if TRUE.

From a csv file, a data frame was created and values of a particular column - COLUMN_to_Check, are checked for a matching text pattern - 'PEA'. Based on whether pattern matches, a new column on the data frame is created with YES or NO.

I have the following data in file DATA2.csv

ASSIGNMENT,Open date,Resolved date,COLUMN_to_Check,NUMBER,Open Time,RESOLVED_GROUP,RESOLVED_TIME,SUBCATEGORY
GBL_IS_GRC_PROCESSCONTROL,3/1/2017 13:39,11/1/2017 13:09,APAC_LT-ERP-FICO-BOKADABISH_PRD,IM-17-001200,3/1/2017 13:39,GBL_GSO_MQG,11/1/2017 13:09,Security (breach or weakness)
RSP_SERVICEDESK,12/1/2017 0:08,12/1/2017 0:27,APAC_LT-ERP-SALES-PEA_PRD,IM-17-006462,12/1/2017 0:08,RSP_SERVICEDESK,12/1/2017 0:27,failure
RSP_SERVICEDESK,10/1/2017 5:27,12/1/2017 0:52,APAC_LT-ERP-SUPPLY-PEA_PRD,IM-17-004667,10/1/2017 5:27,RSP_PCS_INCIDENTS,12/1/2017 0:52,failure
RSP_SERVICEDESK,12/1/2017 2:35,12/1/2017 3:03,APAC_LT-ERP-SALES-PEA_PRD,IM-17-006483,12/1/2017 2:35,RSP_SERVICEDESK,12/1/2017 3:03,access
RSP_SAP_BI,10/1/2017 21:04,12/1/2017 6:01,APAC_LT-ERP-SALES-PEA_PRD,IM-17-005498,10/1/2017 21:04,RSP_SAP_SALES,12/1/2017 6:01,SAP Sales

And using this code....

import pandas as pd

df=pd.read_csv('DATA2.csv')

Search_for_These_values = ['PEA', 'DEF', 'XYZ'] #creating list

pattern = '|'.join(Search_for_These_values)     # joining list for comparision

IScritical=df['COLUMN_to_Check'].str.contains(pattern)
for CHECK in IScritical:
    if not CHECK:
        print CHECK
        df['NEWcolumn']='NO'
    else:
        print CHECK
        df['NEWcolumn']='YES'

df.to_csv('OUPUT.csv')

Printing the value of 'CHECK' returns correct values, i.e., first row returns false.

C:\Users\ME\Documents\SandBox (master)
λ python numpytest_pub.py
False
True
True
True
True

But the output csv file shows all values of 'NEWColumn' as 'YES', where on 'NEWcolumn', row[0], value should be 'NO' as the 'COLUMN_to_Check' here should not match the pattern.

,ASSIGNMENT,Open date,Resolved date,COLUMN_to_Check,NUMBER,Open Time,RESOLVED_GROUP,RESOLVED_TIME,SUBCATEGORY,NEWcolumn
0,GBL_IS_GRC_PROCESSCONTROL,3/1/2017 13:39,11/1/2017 13:09,APAC_LT-ERP-FICO-BOKADABISH_PRD,IM-17-001200,3/1/2017 13:39,GBL_GSO_MQG,11/1/2017 13:09,Security (breach or weakness),YES
1,RSP_SERVICEDESK,12/1/2017 0:08,12/1/2017 0:27,APAC_LT-ERP-SALES-PEA_PRD,IM-17-006462,12/1/2017 0:08,RSP_SERVICEDESK,12/1/2017 0:27,failure,YES
2,RSP_SERVICEDESK,10/1/2017 5:27,12/1/2017 0:52,APAC_LT-ERP-SUPPLY-PEA_PRD,IM-17-004667,10/1/2017 5:27,RSP_PCS_INCIDENTS,12/1/2017 0:52,failure,YES
3,RSP_SERVICEDESK,12/1/2017 2:35,12/1/2017 3:03,APAC_LT-ERP-SALES-PEA_PRD,IM-17-006483,12/1/2017 2:35,RSP_SERVICEDESK,12/1/2017 3:03,access,YES
4,RSP_SAP_BI,10/1/2017 21:04,12/1/2017 6:01,APAC_LT-ERP-SALES-PEA_PRD,IM-17-005498,10/1/2017 21:04,RSP_SAP_SALES,12/1/2017 6:01,SAP Sales,YES

I can sense that something is missing in the CHECK part, but not able to figure out what. Can anyone help ?

Let me know if the question needs rephrasing for better understanding or future community use.

You simply need to do:

df['NEWcolumn'] = df['COLUMN_to_Check'].str.contains(pattern)
df['NEWcolumn'] = df['NEWcolumn'].map({True: 'Yes', False: 'No'})


How to predict unknown(hidden) value by incomplete value or partly recorded value Let me make it clear by make an example:

Suppose I knew a person's cost each month for 3 years like:

2016Jan : $2500

2016Feb : $4000

2016Mar : $3500
...

Just according to this, can I predict how much he earned each month in these years? If I could, what model should use? If I couldn't, what info I need to make this possible. If I couldn't guess in 100 percent but I could guess by some probability, what model should I use?

Another example:
Need to estimate how much each store sells in each city, however I could only got the sales when they record it in electric system, if they didn't record it in electric system(may use paper or something else); then I don't know, how can I get all sales, not just the sales record in electric system?

Any suggestion is welcomed

Actually you can use almost any regression model. When you do not want to go too much into theory simply use a gaussian process provided by sklearn. This model gives you predictions and uncertainties.


Difference between isna() and isnull() in pandas I have been using pandas for quite some time. But, I don't understand what's the difference between isna() and isnull(). And, more importantly, which one to use when identifying missing values in a dataframe.

What is the basic underlying difference of how a value is detected as either na or null?

Pandas isna() vs isnull().

I'm assuming you are referring to pandas.DataFrame.isna() vs pandas.DataFrame.isnull(). Not to confuse with pandas.isnull(), which in contrast to the two above isn't a method of the DataFrame class.

These two DataFrame methods do exactly the same thing! Even their docs are identical. You can even confirm this in pandas' code.

But why have two methods with different names do the same thing?

This is because pandas' DataFrames are based on R's DataFrames. In R na and null are two separate things. Read this post for more information.

However, in python, pandas is built on top of numpy, which has neither na nor null values. Instead numpy has NaN values (which stands for "Not a Number"). Consequently, pandas also uses NaN values.

In short

To detect NaN values numpy uses np.isnan().
To detect NaN values pandas uses either .isna() or .isnull().
The NaN values are inherited from the fact that pandas is built on top of numpy, while the two functions' names originate from R's DataFrames, whose structure and functionality pandas tried to mimic.


Extract key phrases from a single document I need to extract relevant key phrases from a single document. Since I don't have a lot of documents, TF-IDF doesn't really work.

Currently I'm using TextRank. It produces okay-ish result - some really good phrases along with a lot of garbage.

Is there a better algorithm to use for this? Can anyone give me a rundown of available options?

Real-world use case: I'm developing a help desk app that comes with Knowledge Base (a bunch of articles, think of it as FAQ). When a user writes a new support ticket I want to extract key phrases and find the most relevant KB articles. Overall there is not enough data to train a model. I need to compare sets of key phrases I think.

A related keyword to your case can be Single Document Keyword Extraction. A good paper about this is:

  We present a new keyword extraction algorithm that
  applies to a single document without using a corpus.
  Frequent terms are extracted first, then a set of cooccurrence
  between each term and the frequent terms,
  i.e., occurrences in the same sentences, is generated.
  Co-occurrence distribution shows importance of a term
  in the document as follows. If probability distribution of
  co-occurrence between term a and the frequent terms is
  biased to a particular subset of frequent terms, then term
  a is likely to be a keyword. The degree of biases of distribution
  is measured by the $\chi^2$-measure. Our algorithm
  shows comparable performance to tfidf without using a
  corpus.

You can find the paper here.

In sum, this paper gives a rank on keywords based on the defined $\chi^2$-measure.


Kaiser-Guttman Criterion in R I've been using some packages in R for MCA/PCA-like analysis, yet I can't seem to find a package (or a function) within a package which allows me to implement the Kaiser-Guttman Criterion for specifying all the eigenvalues greater than 1.
Would anybody know a command in R for doing so, and the relevant packages?
There is a command at the very bottom of this page, but the authors do not specify the package used in R.
https://www.geo.fu-berlin.de/en/v/soga/Geodata-analysis/Principal-Component-Analysis/principal-components-basics/Choose-principal-components/index.html
Would be good to hear everyone's thoughts.

If you have the eigenvalues in a single vector you can simply use base R commands to only select values greater than one. The linked page uses the following commmand:
food.pca.eigen$values[food.pca.eigen$values >=1]

which simply uses base R syntax. food.pca.eigen$values >=1 returns a true/false vector depending on if the value is greater than or equal to 1 or not, which is then used as a mask for the food.pca.eigen vector. The only values selected are the ones where the mask has the value of true, which are then ones with a value equal to or greater than 1.


Selecting the right time series model Using Python, I am trying to predict the future sales count of a product, using historical sales data. I am also trying to predict these counts for various groups of products.

For example, my columns looks like this:

Date Sales_count Department Item Color

8/1/2018, 50, Homegoods, Hats, Red_hat

If I want to build a model that predicts the sales_count for each Department/Item/Color combo using historical data (time), what is the best model to use?

If I do Linear regression on time against sales, how do I account for various categories? Can I group them?

Would I instead use multilinear regression, treating the various categories as independent variables?

You are saying "For various group of products" and this is your answer.

Forecast each group of product independently, then split the group total on individual products. The reason for this strategy are that it is easier and more accurate to forecast big numbers than rare cases. An other reason is that by grouping similar product, you have chances to capture niceties like new products cannibalizing old products.


Graphs demonstrating the structure of neural networks are very unclear This image looks pretty familiar to anyone getting acquainted with neural networks, and on first glance it makes a lot of intuitive sense.

But on second, third, fourth, etc glance, some questions pop up! I hope I'm not the only one with these concerns.

There are 3 circles representing Input nodes. Does this mean that the data sent into the NN has to be separated into multiple batches? I don't hear much talk of that, or any kind of emphasis / reasoning. Why don't we send the data through the network all at once? Can't the video card handle it? (Or is that the reason why right there, that the card can't handle it?).

There are 4 circles representing Hidden nodes. Each previous node points to
a Hidden one. The way this is show, it is confusing:

Does this mean that data coming in is somehow merged together? I see 3 arrows all coalescing together into the next layer. That's what it looks like visually.

Is there some order, or sequence here? There's no indication of this process happening in some order. It just looks like a mess of arrows pointing from Input to Hidden. Shouldn't be be concerned with the order in which these activations flow? After all, we are looking at a network graph.

There are 2 circles representing Output nodes. Is this showing that a NN can output 2 different sets of data?? Why do we want multiple outputs? We started with 3 blocks of data, now we have 2. How are 2 separate blocks of data going to help us make a binary choice on, say, image classification?

The more I look at this image, the more I dislike it. Yet I see it everywhere, in many introductory / intermediate discussions of NN. I'm hoping there is a reason why such an unclear image is so popular.

You are making one mistake which cascades on towards other mistakes. The multiple inputs are different features of the same sample. Let's say we want to regress house prices based on some features of a house. These three nodes could represent the rating of the neighborhood, the surface area and the number of bedrooms for example. Then you take a weighted sum to every of the hidden nodes. Every hidden node takes different weights over these features. The order you mention is per layer, since we need the output of the previous layer as input for the next one, these dependencies force this order.

With regards to output, you can have a combined loss. In case of binary classification you would probably just have a sigmoid activation at the end, however if you would have multiclass classification you would have a softmax layer at the end, one output for every potential class. There are a lot of other things you could want to output, for example a bounding box so you would need coordinates of where the network thinks the box is, or even per pixel if it's part of the object or not, then you would have a m x n grid as output.

EDIT: All the things you mentioned about the GPU etcetera, that is way too far for what this image represents. It's one network for one sample. I do agree it would have been better to just have one output node because most more basic examples in supervised learning have just 1 target.


Transformer: where is the output of the last FF sub-layer of the encoder used? In the "Attention Is All You Need" paper, the decoder consists of two attention sub-layers in each layer followed by a FF sub-layer.
The first is a masked self attention which gets as an input the output of the decoder in the previous step (and the first input is a special start token).
The second, 'encoder-decoder', attention sub-layer gets as an input queries from the lower self-attention sub-layer and keys & values from the encoder.
I do not see the use of the output of the FF sub-layer in the encoder; can someone explain where is it used?
Thanks

We can see this in the original Transformer diagram:

The output of the last encoder FF layer is added to the original input of the same layer, then layer normalization is applied and that is the output of the whole encoder, which is used as keys and values for the encoder-decoder attention blocks in the decoder.


Explain forward filling and backward filling (data filling) Can I understand in this way? Let me know if any statement is wrong or not accurate.

Reason of data filling:

Assume I have a consecutive data (e.g., daily log data), and partial data are missing. In order to make some calculation (e.g., mean value), we first need to assign values to the missing parts (e.g., equal to existing data)
Forward filling and backward filling are two data filling methods.

The difference is the filling direction? E.g., Tuesday data (missing) equals to Monday data (existing) is forward filling. The opposite is backward filling.

Yes you are correct. Forward filling and backward filling are two approaches to fill missing values.
Forward filling means fill missing values with previous data.
Backward filling means fill missing values with next data point.

You can refer the below url for more on missing values:

https://www.kaggle.com/juejuewang/handle-missing-values-in-time-series-for-beginners

https://www.kaggle.com/dansbecker/handling-missing-values

https://www.kaggle.com/meikegw/filling-up-missing-values


How to plot large web-based heatmaps? I want to plot large heatmaps (say a matrix $500 \times 500$). I can do it in Python/matplotlib.pyplot with pcolor, but it is not interactive (and I need an interactive heatmap). I have tried with D3.js but what I found is aiming at displaying small heatmaps: http://bl.ocks.org/tjdecke/5558084
Naively extending this example with a bigger matrix (e.g. $500 \times 500$) can crash the web-browser.

So, can anyone point me toward a good way of displaying and interacting with large heatmaps with a web-based technology: I want to be able to interact on a web-page or a ipython notebook.

Plotly and Lightning are [supposedly] able to visualize extremely large data sets.


K.gradients gives type error where both arguments are tensors On lines 142 and 143 of: https://github.com/nyck33/openai_spinup_my_implements/blob/master/continuous/mountaincar/my_ddpg_ac.py

I have:

self.get_action_gradients = K.function(inputs=[self.model.input[0], self.model.input[1], \
            K.learning_phase()], outputs=[action_gradients])

Which tells me:
line 143, in build_model
K.learning_phase()], outputs=[action_gradients])

TypeError: Can not convert a list into a Tensor or Operation.

action_gradients are calculated on line 140 via:

action_gradients = K.gradients(Q_value, actions)

so I did not think that is a problem but when I take the brackets off of the output argument for K.function like so:

self.get_action_gradients = K.function(inputs=[*self.model.input, \
            K.learning_phase()], outputs=action_gradients)

Now I get a slightly different error mentioning Nonetype rather than list:

rning-copied/_1my_imps/continuous/mountaincar/my_ddpg_ac.py", line 143, in build_model

TypeError: Can not convert a NoneType into a Tensor or Operation.

Printing out the Q_values and actions show they are tensors:

Q_values Tensor("q_values/BiasAdd:0", shape=(?, 1), dtype=float32) actions Tensor("actions:0", shape=(?, 1), dtype=float32)

But printing out the action_gradients and type(action_gradients) just confuses me more:

action_gradients [None]
action_gradients type <class 'list'>

calling:  K.gradients() on two tensors should work shouldn't it?

This DDPG code is originally from here:  https://github.com/nyck33/autonomous_quadcopter

and I am trying to adapt it for MountainCarContinuous-v0

I am very sorry but this was a careless error.  In the critic network I had different pathways for state and action, joining them right before the Q-value output layer but...

I actually had state as the input for both pathways rather than action for the action pathway.

So of course, if the action is not present in a network, there is no way to use the chain rule to get the dQ(s,a)/da gradient of output w.r.t. actions.

Here is the code for clarification:

def build_model(self):
    #lrelu = LeakyReLU(alpha=0.1)
    #Define input layers
    states = layers.Input(shape=(self.state_size,), name="states")
    actions = layers.Input(shape=(self.action_size,), name="actions")

    #Add hidden layers for state pathway
    net_states = layers.Dense(units=32, use_bias=False)(states)
    net_states = layers.BatchNormalization()(net_states)
    net_states = layers.LeakyReLU(alpha=0.1)(net_states)

    net_states = layers.Dense(units=64)(net_states)
    net_states = layers.BatchNormalization()(net_states)
    net_states = layers.LeakyReLU(alpha=0.1)(net_states)

    #hidden layers for action
    net_actions = layers.Dense(units=32)(actions) #had (states) here instead
    net_actions = layers.BatchNormalization()(net_actions)
    net_actions = layers.LeakyReLU(alpha=0.1)(net_actions)

    net_actions = layers.Dense(units=64)(net_actions)
    net_actions = layers.BatchNormalization()(net_actions)
    net_actions = layers.LeakyReLU(alpha=0.1)(net_actions)

and the output to console shows that it is no longer a None type:

action_gradients [<tf.Tensor 'gradients_3/dense_13/MatMul_grad/MatMul:0' shape=(?, 1) dtype=float32>]

Edit:  The code works for MtnCarContinuous-v0 environment of openai gym with a modified reward function (incremental based on car position because the final reward is so sparse), over 90 seems to be achieved in 11 episodes.  I rendered so actually saw the car make it to the top on the attempts where score is over 100 below.
Here are the results:

episode: 0 score: -41.59034754863541 mean: -41.59 std: 0.0
episode: 1 score: -75.87242108451743 mean: -58.73 std: 17.14
episode: 2 score: 32.01844640263133 mean: -28.48 std: 45.01
episode: 3 score: 132.904319261567 mean: 11.86 std: 80.02
episode: 4 score: 125.81198290946529 mean: 34.65 std: 84.85
episode: 5 score: 84.26163480413017 mean: 42.92 std: 79.63
episode: 6 score: 126.89684490110164 mean: 54.92 std: 79.37
episode: 7 score: 139.18190524840517 mean: 65.45 std: 79.3
episode: 8 score: 100.24481691450521 mean: 69.32 std: 75.56
episode: 9 score: 165.80286734425076 mean: 78.97 std: 77.31
episode: 10 score: 109.29507292352991 mean: 94.05 std: 66.24
episode: 11 score: 209.07900825070152 mean: 122.55 std: 44.84

Source code here:  https://github.com/nyck33/openai_my_implements/tree/master/continuous/mountaincar


Multi-class classification with extremely small dataset I am working on a text classification task that contains 216 labeled paragraphs. The distribution of tags is as follows:
{0: 17, 1: 15, 2: 16, 3: 9, 4: 10, 5: 18, 6: 24, 7: 9, 8: 33, 9: 38, 10: 27}.
The keys are representing the classes, and the value is the number of samples.
Here's my questions:
Q1. Is there any way to train a classifier?
Q2. If we want an 11-class text classifier how many samples do we need?

Yes, you can employ different methods. You can employ deep learning models, but you should not train them from scratch. You should employ transfer learning. Due to the fact that your dataset is small, you should utilize a deep learning model that is already trained. Next, you should replace the last layer with another layer that has the same number of neurons as your classes. The connections of this newly added layer should have random weight at first. Finally, you will freeze all the weights except for the newly added layer. In this case, your model will have a nice capacity to learn your data, and it will not overfit it. You may want to see the following links:

Pr. Ng's video
Another helpful video

You can also utilize SVM with soft-margin to have good generalization.
About the number of samples, it cannot be said in advance. Moreover, for different tasks, it may be different. By the way, someone who looks at your data can easily figure out you have small dataset.


Dropout Decreases Test and Train Accuracy in one layer LSTM in Pytorch I have a one layer lstm with pytorch on Mnist data. I know that for one layer lstm dropout option for lstm in pytorch does not operate. So, I have added a drop out at the beginning of second layer which is a fully connected layer. However, I observed that without dropout I get 97.75% accuracy on the test data and with dropout of 0.5 I get 95.36%. I want to ask whether I am doing something wrong or what is the reason for this phenomena?
I change it into eval mode in test but I reach to 96.44% accuracy. Still it is less than without dropout.
Thanks a lot

# RNN Model (Many-to-One)
class RNN(nn.Module):
   def __init__(self, input_size, hidden_size, num_layers, num_classes):
    super(RNN, self).__init__()
    self.hidden_size = hidden_size
    self.num_layers = num_layers
    self.lstm = nn.LSTM(input_size, hidden_size, num_layers,
                        batch_first=True,bidirectional=True)

    self.fc = nn.Sequential(
        nn.Dropout(0.1),
       nn.Linear(hidden_size*2, num_classes),
        nn.Softmax(dim=1)
    )
def init_hidden(self,x):

     return(Variable(torch.zeros(self.num_layers*2, x.size(0), self.hidden_size)).cuda(),
     Variable(torch.zeros(self.num_layers*2, x.size(0), self.hidden_size).cuda()))

def forward(self, x):
    # Set initial states

    # Forward propagate RNN

    hidden = self.init_hidden(x)
    #print(len(hidden))
    out, _ = self.lstm(x, hidden)

    # Decode hidden state of last time step
    out = self.fc(out[:, -1, :])
    return out

It might be because of learning rate. You need to reduce your learning rate and let it run for more epochs.


Is random forest better suited than neural network in my scenario? I am learning Neural network and facing this scenario.

Say in my input, X has 20-30 features, and Y is a classification (e.g. 1,2,3,4, 5).

What I need is to find the features that contribute most to the output Y (i.e. the most important features).

Base on my limited understanding about neural network and machine learning, random forest is the model that can give out this information. For neural network, I am not really sure it can achieve what I want.

Is it possible that someone give more ideas?

I'd use Random Forests for your scenario because RF is well known for determining the importance of different features; the methods are implemented in many Machine Learning frameworks, and it's very simple to understand.

While neural network are powerful, the methods for extracting the importance of individual features are less clear. Neural networks are more appropriate if you are more focused on just results.


Visualizing deep neural network training I'm trying to find an equivalent of Hinton Diagrams for multilayer networks to plot the weights during training.

The trained network is somewhat similar to a Deep SRN, i.e. it has a high number of multiple weight matrices which would make the simultaneous plot of several Hinton Diagrams visually confusing.

Does anyone know of a good way to visualize the weight update process for recurrent networks with multiple layers?

I haven't found much papers on the topic. I was thinking to display time-related information on the weights per layer instead if I can't come up with something. E.g. the weight-delta over time for each layer (omitting the use of every single connection). PCA is another possibility, though I'd like to not produce much additional computations, since the visualization is done online during training.

The closes thing I know is ConvNetJS:

  ConvNetJS is a Javascript library for training Deep Learning models (mainly Neural Networks) entirely in your browser. Open a tab and you're training. No software requirements, no compilers, no installations, no GPUs, no sweat.

Demos on this site plot weighs and how do they change with time (bear in mind, its many parameters, as practical networks do have a lot of neurons). Moreover, if you are not satisfied with their plotting, there is access to networks parameters and you can plot as you wish (since it is JavaScript).


Confused about the different aspects in Machine Learning After reading different articles about ML and algorithms, scientist tends to use different words when describing the different aspects in ML.

So now I'm a bit confused myself and I hope you can correct me if I'm wrong.

1) So to my understanding supervised/unsupervised learning are different categories of machine learning algorithms. Each category contains different algorithms such as Neural Networks and Bayesian?

2) Regression, Classification and Clustering are types of models?

3) A model is the result of a trained algorithm?

I hope that I'm not completely wrong, thanks! :)

Good question and welcome to Datascience

Imagine you have the tree as follows.

                 Machine Learning Models
                          |
           ----------------------------------------------------
           |                                                  |
       Supervised                                         Unsupervised
           |                                                  |
   - --------------------                                  Clustering
   |                    |
Regression          Classification

A model is indeed a trained version of the algorithm you are chosing,
regardless of whether its Regression, Classification or Clustering. The above
tree is ofcourse a simplified version of all ML algorithms.


Can GAN generate very happy facial expression I have some dataset ${(x1,y1), (x2,y2)...(xn,yn)}$, where, $x$ is the picture of a facial expression，while $y$ is the fraction corresponding to their degree of the happiness (happy laugh: $y$ close to $100$; sad cry : $y$ close to $0$). The range of the $y$ is $0$~$100$.

In my dataset, I didn’t have the sample with $y$ value greater $80$，but I want to get a very happy facial expression. In other words，I want to get a facial expression with very high y value $(y>95)$.

Can GAN (Generative Adversarial Networks) implement this? If not，is there any model that can implement this goal?

Here's a paper by Tom White and its repository studying the specifics of sampling from generative models. Section 3 in the paper, Attribute Vectors, might be of interest for your problem. I think you'll likely have to rely on another dataset to learn the latent code for a smile to use on your own images, but given the results of the author's own work it should be fairly good.

Hope this helps!


How does resnet model restores the skipped layers as it learns the feature space? From the definition of resent from wikipedia: it is mentioned that resent model uses fewer layers in the initial training stages. This speeds learning by reducing the impact of vanishing gradients, as there are fewer layers to propagate through. The network then gradually restores the skipped layers as it learns the feature space.

I am new to this domain, and have tried using resent50 and resnet18 for toy examples, but I am not sure about these theoretically parts. Can someone help to answer or give directions to study material for below queries:

How does resnet model restores the skipped layers as it learns the feature space?
it seems like the certain layers are skipped due to problem of vanishing gradients, but why does not the adjacent layers (who's activation is used while skipping certain layers) face the same issue?

In the architecture of resnet there are skip connections between certain blocks. Now, the significance of these skip connections is that during the initial training weights are not that significant and due to multiple hidden layers we face the problem of vanishing gradients. To deal with this researchers introduced residual connection which connects the output of the previous block directly to the output of the next block and thus information keeps on transmitting. Like, just see the architecture of the residual network:

  Ref: https://missinglink.ai/guides/pytorch/pytorch-resnet-building-training-scaling-residual-networks-pytorch/

In VGG-19 the information is directly being transferred from one hidden layer and in ResNet the output of one block is being transferred to next block's input as well as to the ouput of the next block, so that if we face the vanishing gradients problem due to the hidden layers of this block network can deal with it.

To your query of why does the activation is not skipped, the output of both previous and next block is concatenated and then passed through an activation layer, that's why it doesn't skips the activation layer.


Last cell in recurrent network always the most accurate I am using a recurrent network for time series forecasting.

The prediction from the last cell in the network always seems to be the most accurate. For example if I have 20 cells (so my input samples are of length 20), the prediction from the latter cells (e.g. cells 11-20) seem better than the predictions from the early cells (e.g. cells 1-10).

Is this normal?

Lets assume my network only has 3 cells. At present I would input data x(t=0, t+1, t+2) to obtain predictions for t=0, t+1, t+2.Is this the wrong approach? If I want good predictions at all timesteps, do I need to using a "moving window" i.e. should I use a rolling sequence of inputs:

For prediction at t=0, input x(t-2, t-1, t=0)
For prediction at t=1, input x(t-1, t=0, t=1)
For prediction at t=2, input x(t=0, t+1, t+2)

I am using a recurrent network for time series forecasting.

Interesting would be especially the type of the network you chose. How many layers? How many neurons per layer?

  The prediction from the last cell in the network always seems to be the most accurate.

I assume by cells you refer to your samples. Always, if you learn in datastreams (online), the network in later iterations has seen more data than the network in early iterations. Thus, with having seen more samples, your predictions become more accurate. Of course, this assumes that you do online learning on your datastream.

  At present I would input data x(t=0, t+1, t+2) to obtain predictions for t=0, t+1, t+2.Is this the wrong approach?

This depends on how your RNN is set up. A Jordan Network for example has context neurons, which should do exactly what you propose. The reason for recurrency in the network is typically that you want to simulate short term memory - meaning, in a mathematical view, you want to feed information from previous iterations into the front of the network. Thus, since this feature should be taken care of in your RNN, I do not think it is good to extend your input, since you introduce additional, potentially useless weights that have to be learned by the network.


NLP - Is Gazetteer a cheat? In NLP, there is the concept of Gazetteer which can be quite useful for creating annotations. As far as I understand:

  A gazetteer consists of a set of lists containing names of entities
  such as cities, organisations, days of the week, etc. These lists are
  used to ﬁnd occurrences of these names in text, e.g. for the task of
  named entity recognition.

So it is essentially a lookup. Isn't this kind of a cheat? If we use a Gazetteer for detecting named entities, then there is not much Natural Language Processing going on. Ideally, I would want to detect named entities using NLP techniques. Otherwise, how is it any better than a regex pattern matcher?

Gazetteer or any other option of intentionally fixed size feature seems a very popular approach in academic papers, when you have a problem of finite size, for example NER in a fixed corpora, or POS tagging or anything else. I would not consider it cheating unless the only feature you will be using is Gazetteer matching.

However, when you train any kind of NLP model, which does rely on dictionary while training, you may get real world performance way lower than your initial testing would report, unless you can include all objects of interest into the gazetteer (and why then you need that model?) because  your trained model will rely on the feature at some point and, in a case when other features will be too weak or not descriptive, new objects of interest would not be recognized.

If you do use a Gazetteer in your models, you should make sure, that that feature has a counter feature to let model balance itself, so that simple dictionary match won't be the only feature of positive class (and more importantly, gazetteer should match not only positive examples, but also negative ones).

For example, assume you do have a full set of infinite variations of all person names, which makes general person NER irrelevant, but now you try to decide whether the object mentioned in text is capable of singing. You will rely on features of inclusion into your Person gazetteer, which will give you a lot of false positives; then, you will add a verb-centric feature of "Is Subject of verb sing", and that would probably give you false positives from all kind of objects like birds, your tummy when you're hungry and a drunk fellow who thinks he can sing (but let's be honest, he can not) -- but that verb-centric feature will balance with your person gazetteer to assign positive class of 'Singer' to persons and not animals or any other objects. Though, it doesn't solve the case of drunk performer.


Tuning C hyper parameter in Soft Margin SVM in Matlab How to tune the C 'BoxConstraint'  hyperparameter in soft margin SVM to get the best optimal value?

The easiest way to tune a single hyperparameter is to use what is called the elbow method. Do the following:

Define a range of C you want to try, i.e C = [1.0, 1.5, 2.0, ...]
Loop over all values of C in your range

Train a new model with the current value of C
Evaluate each model on the validation set and store the results

Plot your metric over over your range of C's

If you didn't choose a too narrow range you should visually see your optimal value of C by finding the value that either minimize or maximize your metric.

If you need to tune C alongside other hyperparameters you can use strategies such as grid search, randomized search or bayesian search.


Common Techniques to Generate from a Regression Neural Network Model I am used to train neural networks that are designed for generation, such as GANs or VAEs.

I am wondering what are the common techniques to generate data that would minimize the target/energy learned by a regression model, following the idea of Deep Dream.

I can think of two ways :

1) Use the trained regression neural network as a loss function (with its gradients) for another neural network that is trained to produce structures that produce a given energy / target, as given by the first neural network.

2) Use an standard optimization algorithm (not a neural network) to find which inputs minimize the output of the regression model.

Are there any other common methods to do this ? What are the most know / effective methods ?

Any idea / reference would be great !

Let $E_\phi : D\rightarrow \mathbb{R}$ be your trained differentiable regression model, where $D$ is the data space, e.g. images. Let $G : \mathbb{R}^d\rightarrow D$ be some generative model or decoder from a latent space (e.g., GAN or decoder from a VAE).

Suppose we want to find $x\in D$ such that $x = \min_y E_\phi(y)$. Then there are two obvious ways:

Gradient descent in the data space: i.e., solve $x = \min_y E_\phi(y)$ via iterating $ x_t = x_{t-1} - \eta\nabla E_\phi(x_{t-1}) $.
Gradient descent in the latent space: i.e., solve $z = \min_u E_\phi(G(u))$ via iterating $ z_t = z_{t-1} - \eta\nabla (E_\phi(G(z_{t-1}))) $.

Both are very common.
The main benefit of (2) is that (a) you are more likely to get a "reasonable" $x=G(z)$ because the generator $G$ was trained to give you one (i.e., $x$ should look like it was from $D$, which is not guaranteed in (1)), (b) $d$ is often lower dimensional than $|D|$, making the optimization easier, and (c) it may be impossible to even do (1) (for instance, in molecule generation, we can do (2) but not (1) due to the discreteness problem). Nevertheless, there is a large drawback: the need to learn $G$, which is itself non-trivial to obtain. (Hence why for e.g. generating adversarial examples, (1) is more popular.)

One might also want to avoid gradient-based optimization in some cases, in which case I often see Bayesian optimization approaches being used.


Categories with the same mean in target encoding While doing target encoding it can happen that two categories have the same target mean.

This is bad because there will be no difference in the new feature in it and we will lose some information.

Also, this is potentially harmful to the model, choosing this split in the feature can produce some incongruences.

Is there any way to fix this problem?

I assume tree-based models in this answer
Bayensian Mean Encoding can help
The main problem is that the model will be unable to split between the merged categories. So you implicitly accept that there is no interaction between the two merged categories and other variables. (succession of splits will be the same for cat1 and cat2)
If you want to enable the tree to split anywhere (because you suspect interactions), you have to create a space between the values of cat1 and cat2 for the tree to split. Bayesian Mean Encoding can help you. It takes the frequency of the categories into accounts in the calculation of the target means.
The formula is the following :
\begin{equation}
\mu = \frac{n * \bar{x} + m*w}{n+m}
\end{equation}
where :

$\mu$ is the category mean
$n$ is the frequency of this category
$\bar{x}$ is the target mean for this category
$m$ is the weight you want to give to the overall mean
$w$ is the overall mean

nb: with small $m$ you will have a result very similar to simple target encoding while moving slightly the means (enough to enable the split). The main goal of Bayesian Mean is to limit overfitting, which is also a problem when doing target encoding.
There is no point of doing that
The idea behind target encoding is to make the assumption that the category has no interaction with other variables. So you convert the categorical space (high dimensional) into a simple continuous space, where only the value counts. If you think that the algorithm might confuse two categories and that this is bad, it means target encoding is not suited. If different categories should follow a different decision path, then it means there are interactions between the categorical variable and the others.
If you want every category to have different decisional paths why don't you keep categories? Because implicitly, the tree will compute mean of the splits (explained in this video: https://www.youtube.com/watch?v=g9c66TUylZ4)


Should inputs be shuffled for training a model with expected time dependency? Consider a prediction model with numerical inputs and outputs.

Suppose data is inserted tick by tick, i.e., when new data is available it is inserted asynchronously. Current output depends on current inputs and past inputs and outputs.

Normally for training, inputs/outputs pairs are shuffled since no time dependency exists.

With such time dependency, should inputs still be shuffled?

You don't shuffle the actual inputs and outputs in your prediction model. Nevertheless, depending on your prediction model you might have to model the sample distribution including mean and variance. To model sample statistics you can use methods like bootstraping. However, this method assumes your model and inputs to be stationary (none changing distribution over time) and therefore neglects timely-variations, like structural breaks. But there have been modifications, e.g. bloc bootstrapping, to overcome this. You can read more about sampling in time-series here


How do autoencoders reconstruct images/color? I am abit confused by how autoencoders are able to reconstruct colors in images. According to me a CNN has feature detectors that convert the image into a sequence of feature or activation maps. These activation map corrospond to the degree of presense of a structural feature. Then how is the color information preserved in autoencoders if it is not a structural feature? To me the color information gets lost very quickly but most visuals of autoencoders show they preserve color very well.

Autoencoders ( AE ) are obviously unsupervised learning algorithms as they try to reconstruct the input or something similar to their input. To get a better understanding, we may use autoencoder to colourizing grayscale images. Given an input grayscale image, the AE will predict coloured images.

I have tried one such AE here.

  Autoencoders learn to represent the given grayscale image into a latent ( compressed ) form. The encoder does this job. Now when the image has been brought down to its latent form, the decoder reconstructs the coloured image using that latent representation.

Due to this ability of autoencoders, they are used in image colorization, neural implanting and even generating images ( VAEs ). See this video.

Image colourization is not pixel-level operation. We need a smarter system which can literally observe the surroundings and assign a colour to the pixel. If we are training one such model on images of various landscapes ( sky and green croplands, for instance ) the most probable colour for green ( grass ) will be blue ( sky ).

Meaning, image colourization AEs extract spatial features which may be responsible for a colour change in the target image. A model trained on face images knows that there is a dark-coloured region above the eyes ( hair ). Many such features are learnt by even more complex models which give excellent results.

  Autoencoders store the knowledge required to colourize an image by learning features from grayscale images. After enough they tend to know which region has to colourized with which colours.


One Hot Encoding of Age My task is to predict how many years a person has left to live using an MLP. There is one specific feature I'd like to discuss: current age.

Statistically, it's a conditional probability. Example: When 0 years old, I'm expected to live until 70 years old (70 years left). However when I'm 70 years old, I'm expected to live until 83 (13 years left).

In my dataset, I have the true age of when someone died, and it follows a distribution. I have thus augmented my data, if someone died at the age of N, then there will be N datapoints of current age(feature) from 0 -> N and years left to live (target) from N -> 0 accordingly.

Through this augumentation, I'm hoping to model the distribution of current age vs years left to live through the data. I would never want my network to output (after denormalization) that if the current age was 70, the person would have 50 years left.

Now, the best practice of an MLP states that 1-2 hidden layers is enough, and amount of nodes in the hidden layers should be between the amount of input and output layers. However if I just normalized age through feature scaling, I would have one input node, one hidden node and one output node. This is realistically not enough, I'd expect that I need a lot of hidden nodes.

If I used one hot encoding that represents current age from 0 to 100. I'd be able to use 50 hidden nodes through good practice. If I was 70 years today, I would probably never activate the node that will tell me I'd have another 50 years to live.

My question then, is that should I skip the one hot encoding, and just scale the age and use for example 50 hidden nodes?

Theoretically, because age is continuous as not as distinct nor discrete as say "cat" and "robot" then one-hot is not necessary, but is there any drawbacks if I choose to one-hot encode?

The task of predicting how many years a person has left to live is called survival analysis. Survival analysis is a type of time to event analysis. Thus, survival analysis needs a special type of loss function. An appropriate loss function would avoid predictions like 50 years left when the current age is 70. A common survival analysis model is Cox regression model which can be adapted to neural networks.

If survival analysis is used, the current age can be inputted in the model directly in a single input node.


How to find unique occurance of a value based an another value using pandas? Below is my dataset.

Student_code   Subject
___________    _______
123            Maths
123            Chemistry
124            Maths
125            Chemistry
126            Physics
126            Maths
127            Physics
127            Chemistry

I would like to find number of students who have just enrolled for one course. In this example, given "Maths", i should get 1 as answer as student id 124 enrolled just for Maths.

Assuming that you want to return students who only took one class, You can use groupby and filter:

df.groupby('Student_code', as_index=False).filter(lambda x: x['Subject'].count() ==1 )

  Student_code    Subject
3          124      Maths
4          125  Chemistry


Is a "curve" considered "linear"? In linear regression, we are fitting a polynomial to a set of data points. In Bishop's book of Pattern Recognition & Machine Learning, there are a few examples where the fit is a curve or a straight line. I am a bit confused if a curve is linear or not. The term linear means that the fit should be a linear function or a polynomial of degree 1 i.e., a straight line. But in many resources, examples are shown where the fit can be a polynomial of degree 3,9 etc. So, are these higher order polynomials linear?

Polynomial regression (for nth degree polynomial) in statistics is a special case of linear regression. Lets give an example for square function:

1. y = w*x

  This is linear in terms of both weight (w) and data (x).

2. y = w*(x^2)    OR        y = w*z ; where z = x^2

  This is still linear in terms of weight (w) and still treated as a linear regression for the transformed data (z). While the modeled relationship between y and x is certainly non-linear.

As you can notice above: The commonality in (1) and (2) is the linearity with the weight/ coefficient of linear regression.


k-fold cross-validation: model selection or variation in models when using k-fold cross validation I am new to machine learning, though I have a background in statistics. But I had a question about $k$-fold cross-validation. So I understand the basic idea that we divide the dataset into $k$ partitions and then train a model on $k-1$ partitions while testing on the $k$th partition that was left out. So we don't want to train the model over the entire dataset, but instead over just our $k-1$ partitions.

My question was how do we handle changes in the model parameters with each fold in the process. Another way of asking this question is how do we choose the model to test? So when you train on $k-1$ folds, in each iteration there will be a subtle change in the parameter estimates--namely the $\beta_{0...p}$s will change. So when we test the model against the testing partition, we are not testing the same model each time--because the parameter coefficients are different.

Given that the $\beta$s change for each fold, how do we choose which model to use? And once we choose a model, do we need to test only this model for the $k-fold$ cross-validation, for otherwise we are testing different models and then averaging their prediction error.

Any clarification would be appreciated.

There are two main (set of) things your model needs to learn from training: Parameters and Hyperparameters.

Parameters are those that can be inferred (learnt) from data. These are internal to the model and will be calculated in the training phase during. The hyperparameters however, cannot be learnt from data and need to be decided (make assumptions) before training. As you come from statistics, an example could be modelling a distribution from a set of samples. Parameters could be mean and std, and hyperparameter could be selecting a Normal distribution or a Beta distribution.

If you don't have hyperparameters (e.g. simple regression model), you only need to train the model and let the algorithm find the optimal weights (parameters) using the data. However, if you want to use let's say Support Vector Machines, then you need to provide some hyperparameters such as the cost, kernel, parameters of the kernel, and so on... before starting the training.

Cross Validation can be used to find the optimal hyperparameters. It is done by creating a grid of potential hyperparameter values and training each combination of them against the whole data, performing k trainings of k-1 portions (or folds) of data. For each specific combination of hyperparameters the model is then trained and evaluated k times, getting a number of predictions equal to the number of folds (k) for each hyperparameter combination. The combination that yields better results based on the specified metric, is considered the best option (final model).

Have a look at this nice blog for further clarification.

Cross validation procedure:


How to equalize the pairwise affinity perplexities when implementing t-SNE? I'm trying to implement the t-SNE algorithm:

I found that to compute the pairwise affinities, I have to follow this:

My problem is computing $\sigma_i$. In the Wikipedia I found:

  The bandwidth of the Gaussian kernels $\sigma_{i}$,
  is set in such a way that the perplexity of the
  conditional distribution equals a predefined perplexity using a binary
  search. As a result, the bandwidth is adapted to the density of the
  data: smaller values of $\sigma_{i}$ are
  used in denser parts of the data space.

I don't understand what this really means. How can I calculate $\sigma_i$?

It simply means that you should set the bandwidths through binary search. The way it works is that you start with a preset target perplexity (Mark's link suggests values from 5 to 50 as reasonable values), and bounds for the bandwidth. If the target perplexity is inside the interval defined by the boundary perplexities, you iteratively halve the search space until you converge to the target:

$$2^{H(p; \sigma_L)} < PP_\mathrm{target} < 2^{H(p; \sigma_U)}$$

If the target was not in the initial interval, you expand the interval and try again.


What ML architecture fits fixed length signal regression? My problem is of  regression type -
How to estimate a fish weight using fixed length signal (80 data points) of the change in resistance when the fish swim through a gate with electrodes (basically 4 seconds of the fish passing at 20Hz logger)?
It is a spike shaped-signal, height and width depends on the size of the fish, its speed and proximity to the gates edges and probably other things like the water salinity and temperature.
I have a data set of 15 different weights, each with 20-110 samples, each with 2 spikes for the 2 sets of electrodes I use for measurement (using 2 sets can help determine where the fish is heading).
Here is an example of resistance readout of 340 gram fish experiment:

And here is an example of the extracted spikes from the same 340 gram fish experiment:

As you can see, there is a significant variance, which led me to look for a Neural Network approach that can get such signal as an input and estimate the fish weight.
Do you know of such "State of the Art" network that does that?
What would you try?
Maybe different ML technique?
Thanks!
Edit:
The data presented is post-processing, I extract the spikes using this python code (attached) so some of the noise is cleaned.
I'm not sure as to how to clean it any better since the experimenter didn't record when a fish goes through the gate - all we have is the electrodes signal to deduce that a fish passed through.
# extracting the spikes
def get_spikes(data_series_elc1, data_series_elc2, signal_meta):
    window_size = int(signal_meta['freq'])*4
    half_window = int(window_size/2)

    std = np.std(data_series_elc1)
    p10 = np.quantile(data_series_elc1, 0.9)
    spikes = []
    i = 0
    while i < len(data_series_elc1)-half_window:
        if data_series_elc1[i] > p10:
            #find next max to fix as the center
            max_indx = np.argmax(data_series_elc1[i:i+window_size]) half_window:i+max_indx+half_window])
            spike_list = [[data_series_elc1[i+max_indx-half_window:i+max_indx+half_window]],[data_series_elc2[i+max_indx-half_window:i+max_indx+half_window]]]
            if len(spike_list[0][0])==window_size:
                spikes.append(spike_list)

            i = i+max_indx+half_window
        else:
            i = i+1
    print('Number of Spikes: ',len(spikes))
    return spikes

Also, I extract features like max, width, integral and Gaussian fit but a linear regression model only gets me ~R^2=0.6 => a mean error of ~103 gram over all fish
[100., 144., 200., 275., 339., 340., 370., 390., 400., 404., 480., 500., 526., 700., 740., 800., 840.], which is quite a large error.
A vanilla fully connected neural network gets about the same.
model = keras.Sequential()
model.add(keras.Input(shape=(80,)))
model.add(layers.Dense(40, activation="relu"))
model.add(layers.Dense(10, activation="relu"))
model.add(layers.Dense(1))

So I'm looking to improve these results, any ideas?

One common approach for this type of data is take the integral and learn either a translation function to fish weight. Taking the integral simplifies the problem to a single number.
You probably do not need a state of the art model. A general linear model would probably pick out a signal.


How does word2vec handle the input word being in the context? If word2vec encounters the same word multiple times in the same window, what occurs?  Obviously it is meaningless to decrease the distance between the vectors for the input word and the target word.  But will the repetition strengthen the relationship between the repeated word and the context words?

We can look at the source for guidance.

  How does word2vec handle the input word being in the context?

It is skipped; for both the skip-gram and CBOW models.

  If word2vec encounters the same word multiple times in the same window, what occurs?

The relationship is strengthened.


Labelled dataset for NLP regression I am looking for a dataset, where I could use NLP techniques to estimate target value using regression. For example, I could be give a few sentences that describe an accident, and the target value would be the cost of an accident. Kaggle has quite a few datasets for NLP, but they  all, as far as I can see, for classification.

I know of a good few NLP regression tasks, but these are usually not about a predicting an extrinsic value from the semantic description, but rather predict some numerical property of the text itself:

MT quality estimation: predict the quality of a translated sentence, usually as a score. Data available at the WMT QE Shared Task series. There are probably other similar datasets about predicting the quality of a sentence in general.
Author profiling: predicting the age of the author of a text. Some datasets exist in particular through the PAN workshop series.
Predicting a rating based on a comment - there's data everywhere on social media, Amazon etc. You could even use StackExchange posts and predict votes :)

I'm not aware of any data similar to your example (btw it would be a very challenging task). I know that there are some works on detecting fraud from the text of an insurance claim, so maybe there are datasets that you could use in this area.


How to check for overfitting with SVM and Iris Data? I am using machine learning predictions for the sample iris dataset. For instance, I am using the support vector machines (SVMs) from scikit-learn in order to predict the accuracy. However, it returns an accuracy of 1.0. Here is the code I am using:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=51)
svm_model = svm.SVC(kernel='linear', C=1, gamma='auto')
svm_model.fit(X_train,y_train)
predictions = svm_model.predict(X_test)
accuracy_score(predictions, y_test)

How to find out or to measure if this over-fitting or if the model is so good? I assume that its not over-fitting but what are the best ways to validate this?

You check for hints of overfitting by using a training set and a test set (or a training, validation and test set). As others have mentioned, you can either split the data into training and test sets, or use cross-fold validation to get a more accurate assessment of your classifier's performance.

Since your dataset is small, splitting your data into training and test sets isn't recommended. Use cross validation.

This can be done using either the cross_validate or cross_val_score function; the latter providing multiple metrics for evaluation. In addition to test scores the latter also provides fit times and score times.

Using your example;

from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris
from sklearn import svm
from sklearn.metrics import accuracy_score

iris = load_iris()
X = iris.data[:, :5]  # we only take the first two features.
y = iris.target

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=51)
svm_model = svm.SVC(kernel='linear', C=1, gamma='auto')
svm_model.fit(X_train,y_train)
predictions = svm_model.predict(X_test)
accuracy_score(predictions, y_test)

raw accuracy: 0.96666666666666667

Using the cross_val_score function, and printing the mean score and 95% confidence interval of the score estimate:

from sklearn.model_selection import cross_val_score

scores = cross_val_score(svm_model, iris.data, iris.target, cv=5)
print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))

Accuracy: 0.98 (+/- 0.03)

Of course the iris dataset is a toy example. On larger real-world datasets you are likely to see your test error be higher than your training error, with cross-validation providing a lower accuracy than the raw number.

So I wouldn't use the iris dataset to showcase overfitting. Choose a larger, messier dataset, and then you can start working towards reducing the bias and variance of the model (the "causes" of overfitting).

Then you can start exploring tell-tale signs of whether it's a bias problem or a variance problem. See here:

https://www.quora.com/How-many-training-samples-are-needed-to-get-a-reliable-model-in-ML/answer/Sean-McClure-3?srid=zGgv


Find recurrent dates in a small set (and get rid of non recurrent ones) I need help in the analyse of a categorization problem.

Given a set of dates (small set: 20 elements maximum), I would like to group dates which are equally distributed (with a tolerance).
It can be, for instance, monthly or weekly separated dates.

Here is an example. Given this repartition :

I would like to categorize into these two groups:

The problem is that I am a developer, not a data scientist. I have an intuition that it should be possible to do a kind of regression.

I have no clue how to analyse this problem. Can you help me with that, please ?

Cheers

PS: I have already seen this thread (Recurring events - finding in a time series) but I have not helped me.

If it is a categorization problem then you should look for a classification algorithm, not a regression technique. The simplest classification algorithm is Logistic Regression.

But by the looks of it, seems like you do not have a labelled data-set and if that's the case you should look for Clustering techniques. Clustering is a part of Unsupervised Learning Technique in ML which create clusters or groups of similar data points.


How does combining neurons create non-linear boundaries? I have been working with NNs for a while, but haven't dug too deep into this unfortunately.

By looking at the three neurons below, in each of their boxes we can see that they are really just making linear separations in the x1, x2 plane (of course not taking into account the third and upward Y dimension that make the sigmoid) and combining these into the decision boundary we see at the right.

I understand we need non-linear activation functions, but why?

How can combining non-linear perceptrons can make this three walled decision boundary?
Mathematically, can it be illustrated how we are allowed to go from a simple linear function, to a more complex non-linear function using multiple neurons?

I have looked around and articles just says that they can, but not how?

I think what you are referring to is how neural nets work as universal function approximators. Check out this link for an intuitive explanation.


For text classification that has innumerable features, how do I choose the number of neurons and layers for MLPClassifier? In my use case of text classification (identify the author from a subset of 10 authors), I find that post all processing with trigrams, there are a 100 thousand and odd features with nearly 50k observations/samples to be trained.

It leaves me baffled to decide on the  number of neurons and layers to configure for my MLPClassifier of scikit-learn. If I try a number like 250 neurons with just a single layer, my laptop hangs in about 5 minutes :(

Your advice based on your experience is highly appreciated.

As you know, you can basically reduce the number of features by choosing best discriminant ones.
If you used scikit-learn vectorizers like TF-IDF, they have the parameter max_features which chooses $n$ best features for you. But my point is something else:

All BoW models which I guess you used are unsupervised. I strongly recommend that you use mutual information between features and targets (you may also search "Supervised TF-IDF" to get more insight) and choose your best features more effectively. At the end you need to choose $n$ neurons as input layer but with this way you may be able to get the most out of less number of features.

If you are familiar with autoencoders, they are also a pretty strong way to reduce the number of features effectively.

Hope it helps!


Correcting ALL CAPS for human and algorithmic consumption United States federal tax returns tend to be written in ALL CAPS to facilitate OCR. This practice has persisted even when returns are filed electronically. Thus, much of the text in the IRS 990 dataset is in all caps. This makes it hard to read, and limits the ability of algorithms such as Treebank to accurately tag part of speech.

I understand that the approach of the Stanford POS tagger may be more amenable to correction of capitalization, but in practice, I have not had much luck in using it to correct the text in the IRS 990 corpus, in which nearly every sentence contains one or more proper nouns.

Are there any "tricks of the trade" for improving the performance of an off-the-shelf POS tagger when using ALL CAPS text, and/or an algorithm that may do better at identifying the proper nouns therein?

If sensitivity to case is breaking your models you have two options:

Train or find a new model that's case-insensitive. This is probably the easiest thing to do. The Stanford parser has one.
Train a model to correct the case of your input, this is sometimes called truecasing. The Stanford Parser has this functionality too.


Any Neural Network implementations that allow for a cost function of more than just the network output? I have an application of a straightforward MLP, for which the cost function is a function of both the network output, in addition to another value calculated from the network weights (actually the partial derivative of the output with respect to one of the inputs).

Are there any flexible NN packages where I can code this? The only ones I've seen (Keras, nnet in R) require that the cost function is just parameterised by the network output and the target value.

You should be able to do that in Keras, actually you should be able to do that in almost any flexible package. For example regularization does exactly that, it adds a penalty term to your loss function that punishes high weights.


How can I deal with circular features like hours? Assume I want to predict if I'm fit in the morning. One feature is the last time I was online. Now this feature is tricky: If I take the hour, then a classifier might have a difficult time with it because 23 is numerically closer to 20 than to 0, but actually the time 23 o'clock is closer to 0 o'clock.

Is there a transformation to make this more linear? Probably into multiple features? (Well, hopefully not 60 features if I do the same for minutes)

The question was already posted, you can find the answer there :

What is a good way to transform Cyclic Ordinal attributes?

The idea is to transform your time feature into two feature : it's like if you represent the hour as the angle of the hand on the clock, and use the sin/cos of the angle as your features


How do I read a dat file, for which I don't know its structure? Is there any way to atleast read the text from the dat file. I have its corresponding mdf file hence I know what all data and columns are there in it.
How do I figure out the contents in my dat file. Cuz all that I am getting currently is some gibberish even if I am opening in binary mode.

from asammdf import MDF
dat_file = r"C:\Users\HPO2KOR\Desktop\Work\data1.dat"
mdf_file = r"C:\Users\HPO2KOR\Desktop\Work\data1.mdf"
df = mdf.to_dataframe()
mdf = MDF(mdf_file)
df.head()

which gives me

How do I read the same data from the dat file, is there any library or code for the same?

And when I am opening the same in binary mode I am getting

data = open(dat_file, "rb")
data.readlines()

which is giving me

b'|CF,2,1,1;|CK,1,3,1,1;\r\n',
 b'|NO,1,7,1,0,,0,;\r\n',
 b'|NL,1,10,1252,0x407;\r\n',
 b'|CT,1,41,0,6,Bench#,24,Korrosionstest 15A046-01,0,;\r\n',
 b'|CT,1,30,0,11,StartOfTest,8,06/30/17,0,;\r\n',
 b'|CT,1,58,0,10,ResultPath,36,c:\\korrosionstest\\daten\\#170161-OR02,0,;\r\n',
 b'|CT,1,59,0,11,GraphicPath,36,c:\\korrosionstest\\daten\\#170161-OR02,0,;\r\n',
 b'|CT,1,31,0,15,GraphicBaseName,5,736_2,0,;\r\n',
 b'|CT,1,26,0,10,PartNumber,5,736_2,0,;\r\n',
 b'|CT,1,31,0,9,VA-Nr. GS,11,170161-OR02,0,;\r\n',
 b'|CT,1,62,0,9,VA-Nr. CC,42,TO_ENV_2017_G2_C1_Platform_CC-122164-03-08,0,;\r\n',
 b'|CT,1,24,0,6,Tester,8,Behrendt,0,;\r\n',
 b'|CT,1,32,0,15,Test Department,6,GS/ETR,0,;\r\n',
 b'|CG,1,5,1,1,1;\r\n',
 b'|CD,1,16,1E-2,1,1,s,0,0,0;\r\n',
 b'|NT,1,27,30, 6,2017,14,25,15.8050001;\r\n',
 b'|CC,1,3,1,1;\r\n',
 b'|CP,1,16,1,2,4,16,0,0,1,0;\r\n',
 b'|Cb,1,33,1,0,1,1,0,11718,0,11718,1,5E-3,0,;\r\n',
 b'|CR,1,30,1,6.103888176768602E-3,0,1,1,A;\r\n',
 b'|CN,1,28,0,0,0,16,ai_iB1_Strom_ECU,0,;\r\n',
 b'|CG,1,5,1,1,1;\r\n',
 b'|CD,1,16,1E-2,1,1,s,0,0,0;\r\n',
 b'|NT,1,27,30, 6,2017,14,25,15.8050001;\r\n',
 b'|CC,1,3,1,1;\r\n',
 b'|CP,1,16,2,2,4,16,0,0,1,0;\r\n',
 b'|Cb,1,37,1,0,2,1,11718,11718,0,11718,1,5E-3,0,;\r\n',
 b'|CR,1,30,1,3.662332906061161E-3,0,1,1,V;\r\n',
 b'|CN,1,31,0,0,0,19,ai_iB1_Spannung_UBB,0,;\r\n',
 b'|CG,1,5,1,1,1;\r\n',
 b'|CD,1,16,1E-2,1,1,s,0,0,0;\r\n',
 b'|NT,1,27,30, 6,2017,14,25,15.8050001;\r\n',
 b'|CC,1,3,1,1;\r\n',

You can try:
np.fromfile(yourfilepath, dtype="byte")

to read content from .dat and then load as a Numpy array.


Why Uniformity of subpixel in image analysis ensures correctness I don't know if its right place to ask this question.
I was working on particle tracking of series of pictures (.tiff) of colloids. And I was Using trackpy in python.
The code gives me the position of particles to subpixel accuracy.

There is another step involved is of checking the subpixel accuracy. In this method we check the uniformity in the decimal part of the position

a quick way to check for subpixel accuracy is to check that the decimal part of the x and/or y positions are evenly distributed. Trackpy provides a convenience plotting function for this called subpx_bias

Mask size is more or less the diameter of the particle.
I am not getting how does evenly distribution of decimal part ensures that we are on the right track?
And how a dip shows that we are wrong somewhere?
You can also refer to Eric weeks website. on this website, it is briefly mentioned that

One failure mode is if the length scale in the feature is made too small, then all the x and y coords get 'rounded off' to the nearest pixel value. The above command plots a histogram of the fractional part of the x-coords of the image. The physically distributed positions should be random -- giving a flat histogram. If the histogram has two peaks (near 0 and 1), set the size parameter in 'feature' a little bigger, determine a new masscut, and repeat until everything is happy.

If the subpixels are not roughly evenly distributed then this means (among others) that certain decimal values are more prevalent than others and this can lead to large gaps in tracking the particle (gaps which can be larger than the particle's size).
So this amounts to saying that the outcome is not very precise.
On the other hand, basic continuity of motion requires that decimal values should be evenly distributed on average.
For example imagine the subpixels have a prevalence of 0.1 and 0.9 decimal values. So for a particle of size 1 its positions may vary by more than 2 pixels (eg 1.1-2.9) so this creates inaccurate trackings on average.


Time series binary classificaiton with labelling issues My situation is quite complicated so I will give a similar example from a simpler domain. Suppose we want to try to predict WHEN a mobile game users will make a purchase if given a sale. Almost every user is always instantaneously a non-purchaser because everybody is constantly not buying anything. Some people buy something to instantaneously become a purchaser but go back to the standard state of not purchasing. We have a time-stamp for that purchase. If we could understand the user behavior well enough, we could predict when a user will purchase before they do so. If this is the case then users with actions and game states very similar to the purchasers but who do not actually purchase would do so if given a sale. So the question is how do we turn this into a machine learning question. My current plan is to use binary classification by labeling the YES cases as the purchasers since those who purchase at regular price would have purchased if given a sale. I build my features based on actions in recent look back windows, like how many actions of a type in the last day. For all users who never purchase I choose random time-stamps, build my features and use them as the NO case. Since I can choose as many NO time-stamps as I want I have been doing 100 times my YES case. Then I can use a classifier, I like tree ensemble methods but I do not think the classifier will really matter here.

The problem is that this is not working. There are two things worth noting which I think are at the core of the issue I am having. First, we are trying to predict the "when" not just the "who". Users who play a lot or who have purchased before are much more likely to purchase in general so it is easy to make a time independent classifier and predict who is likely to purchase. This means that the features which are useful for the easy time-independant problem can sort of "contaminate" the feature set. I have taken out a number of these features and seen some improvement. The second issue has to do with labelling. We have many YES cases for the purchasers but how does one define a NO case. I described what I am doing above but I am not sure that it is really a best method. Many of the users I defined as a NO might have effectively been a YES since if they were given the sale they would have bought.

Also, I should point out two situations which this is not. This is not one class binary classification which generally uses the large class in the imbalanced case reducing the problem to unsupervised anomaly/outlier detection. This is also not the semi-supervised situation where I have both classes labelled for a subset of the data. I really only have some of the YES cases for sure and none of the NO case. Any thoughts welcome.

From your description of the data, this is not a time-series problem. Time is not a factor here, for each user you have a set of variables after you choose a time threshold. Although you seek the "when", that's probably out of your reach, based on your explanation.

If I understand correctly you're doing classification but you're unsure about your "data collection process", at least for the "NO" labeled obs.

Why not fix a time stamp, and register all the variables you think are correlated with the outcome, and keep the outcome as you defined (never buys = "NO", buys at least once = "YES).

I'd treat this as a classification problem but instead of looking at the classic metrics (from confusion matrix), I'd be more interested at the probabilities you get from each obs, compared to the truth (that you know up to that point in time).

Imagine a user, with an estimated probability of 0.90 (90%), but the labels says "NO". He's someone who the model thinks that should have bought but didn't, he's highly likely to be interested in a sale.

Maybe you could cluster on this kind of information, creating groups of users where your model is wrong, but it's really confident in it's estimation. Basically you're more interested in false positive obs.

But also false negatives can give you insight, again looking at the probabilities you get.

I know this won't probabily be possible but if you could collect more data, I'd suggest a variable (or more), that measure the relationship of the user with sales.

Also, I agree about the decision of removing the fact that the user plays for long, that's probably "too much correlated" with your outcome.

Without knowing more about the data, that's my two cents.


What is the complexity of Mixture Hidden Markov Models (MHMM)? I wonder what is the theoretical computational complexity of MHMM.
Is it related to the number of alphabets in sequence mining?

Mixture hidden markov model is not one algorithm. It's made of forward algo, viterbi algo and forward-backward algo.

The time complexity of computing the forward variable used in hidden markov models is O(N2T) In which N is the number of states and T is the length of the observations sequence.