Labelling a Time series dataset I was asked this during an interview that I did not crack.
You are given data of total (aggregated) power consumption, and the TV power consumption of three households. Each household has data for one day. Task 1: Using TV instantaneous power consumption data, identify the times when the TV is “ON”. Notice that some TVs may have standby modes. Task 2: Using all the data given, design a classifier to identify times when the TV is “ON”. You may also want to train and test your designed classifier. The trained classifier should not take TV instantaneous power as input. Please provide a script which attempts task 1 and 2. Please do not provide a script without explanations.
Additional information: Data format in the attached .csv file: “House” column indicates which house it is, ranging from 1 to 3. “Time” column indicates time stamps. “TV” column indicates TV instantaneous power consumption. “Agg” column indicates total (aggregated) instantaneous power consumption. Sampling rate is 1/60 Hz.
My submission is here.
I did not receive any feedback from the employer and I don't know what went wrong. Please let me know your thoughts on what could have been done better.

Some feedback/tips/tricks/opinions here:
Problem setup
Including requirement analysis. Gotta decide how the system/solution should work, how to know ho how well we are doing, and then how to get there.
Model evaluation.
It is very desirable to have a quantitative way to evaluate our model performance.
For that we want some labeled data. It is very quick to count the number of ON events (given assumptions) for each house. House 1, 4 times. House 2, 3 times. House 3, 2 times.
Then at least we can use that as an evaluation metric.
Could also mark the times in the dataset to compute a percentage of ON time. That would be a more granular/precise metric to evaluate by. Strongest evaluation would be to check that each event time was also correct (within some tolerance).
Generalization
A model trained on N TVs/houses will also work on unseen houses. Very desirable, as allows the same model to be used for all houses, including future ones.
For this reason one should do possibly do Cross Validation grouped by house.
Might also want to use Time-Series splitting, but a bit hard on such a tiny dataset.
System behaviour assumptions
We assume that that Power(ON) >> Power(Standby) > Power(Off).
However each TV might have different Power levels, as well as differences/ratios between the three levels. Different houses may also have different behavioral patterns in turning on/off.
Minimum on time.
It is reasonable to assume/set some minimum time for TV on/off. Can be informed from data, or set based on what you know about TVs. Maybe 10 seconds is OK?
Technique feedback
EDA
Really missing a histogram over power level!
Denoisning
KNearestNeighbours is too complex/opinionated for this. A running mean or running median would be more suitable.
Classifier
GradientBoostingClassifier wayy to complex. You have a single feature and tiny amounts of data.
GaussianMixtureModel is quite relevant, as this is a classic case of Hidden States and transitions. However it is still not clear if the complexity is justified - ie if the problem cannot be solved with just state-independent thresholding
Report feedback
Your notebook indicates heavy work-in-progress, it does not have the qualities of a report. Of course if you did this live, on-the-spot or with a very tight deadline, that is natural. But if you had several hours on this task, I would expect a much cleaner notebook.

The intro should define/re-state the problem(s). So that the report is self-container
An Exploratory Data Analysis section should be before models. Question -> Data Exploration -> Findings.
All non-interesting findings or explorations should be removed!
Low use of global variables, proper use of functions. Models should use pipelines where appropriate
Use pandas time series functionality. All time-series plots should have time on axis!
After the EDA and models, Results, and then a Discussion/Conclusion
You may want to also have the most important findings/conclusions summarized on top


Graph isn't an attribute in TensorFlow? Very basic question Hi am am just starting to learn how to use TensorFlow and this is one of my first projects. I am simply trying to create a graph.

This is my code:

    import tensorflow as tf
    graph1 = tf.Graph()

I cannot seem to create a Graph. The compiler threw an AttributeError (not exactly sure what that is)...

    AttributeError                            Traceback (most recent call last)

The compiler said the issue was with line two in creating the graph.

    ----> 2 graph1 = tf.Graph()

    AttributeError: module 'tensorflow' has no attribute 'Graph'

If it is of any use I am using a Jupyter Notebook in JetBrain's PyCharm. I believe I imported everything correctly but I am not sure if using PyCharm will mess anything up. Any help you be very appreciated!! :)

2 options.

reinstall and try again (dont ask)

option b) downgrade to versions 1.13.1 and lower (again dont ask)

ofcourse you can ask I am just lazy


Why k-fold cross validation (CV) overfits? Or why discrepancy occurs between CV and test set? Recently, I was working on a project and found my cross-validation error rate very low, but the testing set error rate very high. This might indicate that my model is overfitting. Why does my cross-validation not overfit while my test set overfits?

More specifically, I have about 2 million observations with 100 variables (n>>p). I randomly split the data set into 80/20 train and test. Then, I fit a model (i.e. XGboost) using a 5-fold cross validation on the training set and the estimated error rate is pretty low. Then, I used the same parameter setting and used the entire training set to fit the model. Surprisingly, when I used the test set to evaluate the performance of the model, the error rate is significantly higher than the CV error rate. Why?

Edit:

(About the error rate)

The error rate is actually multinomial logloss. I achieved a CV error rate of 1.320044 (+/- 0.002126) and a testing error rate of 1.437881. They might seem close by staring at these two numbers, but actually, they are not. I don't know how to justify this, but I am sure that they are different within the scale of performance of this project, which is from ~1.55 to ~1.30.

The way of 5-fold cross validation is like following,

divide the train set into 5 sets.
iteratively fit a model on 4 sets and test the performance on the rest set.
average the performance of all the five iterations.

I mean, if my parameter settings make model overfit, then I should see it at this cross-validation procedure, right? But I don't see it until I use the test set. Under what circumstances on the earth this could happen?

Thanks!

Added:

The only reason I could think of that why CV error rate diffs from test set error rate is

Cross-Validation will not perform well to outside data if the data you do have is not representative of the data you'll be trying to predict! -- here

But I randomly 8/2 split the 2-million-sample data set and I believe that the train set and test set should have a same distribution of variables.

(Pardon me that I post the same question here also.)

Edit:

(About the data leakage)

I got one interesting commend from @darXider in cross validated. He says,

  Sometimes when feature engineering, you have to be careful to avoid any data leak between training and test sets. For example, if you do a
  PCA on your original, untouched data, use PC1 and PC2 as "new"
  features, and then split your dataset into train and test, you are
  leaking information from training set into test set. That will boost
  your score up. You mentioned that after some feature engineering, your
  CV score and test score started to disagree. That could suggest some
  sort of information leak between training set and test set.

I wonder what is exactly the "data leakage" and why feature engineering BEFORE splitting can still cause "data leakage"??

I agree with the comment you received from Cross Validated – data leakage is something that fits this problem setting as it's known to cause too optimistic CV score when compared to test score. We could confirm that it's actually a data leakage problem if you provided information about the data pre-processing steps that you've taken.

Data leakage means that information is leaking from the test data to the training data, causing the trained model to over-estimate expected performance and to generalize poorly to unseen data, because the leaked information allows the model to learn something that it otherwise wouldn't learn.

As an example, let's say that you had only a small amount of training examples and you end up synthesizing more training examples that are somehow similar to the other training examples. If you implement the synthesis before CV loop and not inside it (within each fold), your CV accuracy will skyrocket but test accuracy will be low. In addition to data synthesis, data leakage can also happen with other pre-processing steps, such as scaling the data, inputting missing values or performing feature selection.


Classification of potential customers I have a table with many financial information of thousands of companies around the world (revenue, income, number of employees, total assets etc) and an extra information indicating whether that company owns a specific high value machine ("1" if the company has and "0" if the company hasn't).

My goal is to identify which companies don't have this machine, but are potential customers. My first approach was to use a classification algorithm, such as SVM, random Forest etc and use those financial characteristics as predictors of the "has machine"/"hasn't machine" column. The "potential customers" would be, actually, the individuals the model classified as "1" (has machine), but they are actually "0" (hasn't machine). In other words, the false positives of this model.

I am not comfortable with this approach (after all, a perfect model wouldn't indicate a single potential customer!), but I don't know any other way I could approach this problem. I was wondering if someone could give me some direction!

Thanks

What you described is the more-or-less standard approach, with a couple of caveats.

Leakers

Remember that we live in a stochastic world, so a perfect model is immediately suspect of using leakers, i.e., variables which have a causal connection with the target. In your case, e.g., it would be the presence of a service contract for the machine (which would indicated that the company actually does own one). This, building an interpretable model and examining the most powerful variable for such causal connections (and eliminating them) would be a good idea.

Better Approach

However, one could improve on this approach in a very powerful way by replacing the current state of the company with its historical state at the time of the machine purchase. This way one eliminates the leakers automatically and also gets a better model because the predictor values which are more relevant: after all, you are trying to predict which company is ready to buy the machine now rather that which company bought it in the past when its conditions were very different.

In fact, your approach works precisely because it is an approximation of this approach (approximating the company state at the time it bought the machine with its current state) and the absurdity/uselessness of the perfect model that you mention is the artifact of your approach being an approximation of The Right Thing.


Machine Learning Based Algorithm for Image Inpainting? I have set of images which they look like the below sample.

You can think of it as a person's 2-week schedule represented by different color. Inside the images. However, there are some small missing areas shown in black. I'm interested in inpaint/fill the missing areas using ML methods.

Note that I have already tried classical methods like some those mentioned in the paper: "Image Inpainting, Christine Guillemot and Olivier Le Meur", but the result was not satisfactory.

Since almost all images have the same pattern, I'd like to know if anyone can suggest the ML-based approaches tackle this problem.

If I were you I would use deep learning. You can use an autoencoder format for this. Essentially you would feed the image in, then each layer yields a smaller output. Then you would feed the output in reverse, using transposed weight matrices, which would yield progressively larger outputs. The final output from the backward pass would be the filled in image. Since your image has such a small color range I would recommend representing pixel colors using one-hot encoding.


How to Present All Categories in All Samples I have a data contains many categorical columns. When I sampled this data randomly a few times and applied one-hot encoding to categorical columns I noticed that it ended up with datasets with different column counts. Because not all categories in columns preserved in samples and different samples includes different subset of categories for each column. Is there a way to ensure all categorical columns in all samples contains all possible categories?

The first thing we must accept that the sampling is probably doing the right job.
What I mean is that if only 10% is being sampled then some unique value which is less than 5 can be easily missed.
 Ideally, you should club these values into some generic value i.e. OTHER_COL_1
But, if you want to get away with this natural result, you should apply some tweaking.
We may do the following -

Get the sample as you are doing now
Match the unique element of each column to the unique from the main data
Iterate on each col and missed unique value
Let's assume UNIQUE_4 is missed for COL_2
Sample all the records for UNIQUE_4 from COL_2 of main data and
Pick one random data out of it


Count the number of false positives with respect to the first class Can anybody tell me the formula how to find the number of false positives with respect to the first class?

where $y$ is the truth / target and $a(x)$ is the prediction

I always find the notion of false positive and negative confusing, especially when it comes to multi-class problems.

A good rule of thumb that I came up with is the following

True positive: "I predicted that you were a certain class, and I was right"

False positive: "I predicted that you were a certain class but I was wrong"

True negative: "I predicted that you weren't a certain class, and I was right"

False negative: "I predicted that you weren't a certain class, but I was wrong"

So in your case, there are 31 true positives with regard to the first class since you predicted 31 times that something was class 1 but you were wrong.


Is there a rule of thumb for the initial value of loss function in a CNN? I notice that most of my successfully trained CNNs initiate (i.e. first epoch) loss is in the 1-2 range. Typically, these are image classifiers, either for discrete classes (cat, dog, ship, etc) or semantic (pixelwise) segmentation. Occasionally, either because of poor design or architecture misunderstand on my part, I see an initial loss in the 60s; the other day one started at 200 (and didn't converge).

So my question is: is there a heuristic that would allow me to determine a ball park value for my initial loss value? If so, on what aspects of my trained model is it contingent?

The initial loss is purely dependent on your weight initialization and your data normalization. Random weights = random initial loss. If you're interested in the variance of the loss, this will roughly relate to the number of layers you have and number of weights in each layer. For a good example, check out this paper:

Understanding the difficulty of training deep feedforward neural networks -- Glorot, Bengio


Image classification in python I have a set of images that are considered as good quality image and other set that are considered as bad quality image. I have to train a classification model so that any new image can be said good/bad.
SVM seems to be the best approach to do it. I know how to do it in MATLAB.

But,can anyone suggest how to do it in python? What are the libraries?
For SVM scikit is there, what about feature extraction of image and PCA?

As this question highly overlaps with a similar question I have already answered, I would include that answer here (linked in the comments underneath the question):

In images, some frequently used techniques for feature extraction are binarizing and blurring

Binarizing: converts the image array into 1s and 0s. This is done while converting the image to a 2D image. Even gray-scaling can also be used. It gives you a numerical matrix of the image. Grayscale takes much lesser space when stored on Disc.

This is how you do it in Python:

from PIL import Image

%matplotlib inline

#Import an image
image = Image.open("xyz.jpg")

image

Example Image:

Now, convert into gray-scale:

im = image.convert('L')

im

will return you this image:

And the matrix can be seen by running this:

array(im)

The array would look something like this:

array([[213, 213, 213, ..., 176, 176, 176],
       [213, 213, 213, ..., 176, 176, 176],
       [213, 213, 213, ..., 175, 175, 175],
       ...,
       [173, 173, 173, ..., 204, 204, 204],
       [173, 173, 173, ..., 205, 205, 204],
       [173, 173, 173, ..., 205, 205, 205]], dtype=uint8)

Now, use a histogram plot and/or a contour plot to have a look at the image features:

from pylab import *

# create a new figure
figure()
gray()
# show contours with origin upper left corner
contour(im, origin='image')
axis('equal')
axis('off')

figure()

hist(im_array.flatten(), 128)

show()

This would return you a plot, which looks something like this:

Blurring: Blurring algorithm takes weighted average of neighbouring pixels to incorporate surroundings color into every pixel. It enhances the contours better and helps in understanding the features and their importance better.

And this is how you do it in Python:

from PIL import *

figure()
p = image.convert("L").filter(ImageFilter.GaussianBlur(radius = 2))
p.show()

And the blurred image is:

So, these are some ways in which you can do feature engineering. And for advanced methods, you have to understand the basics of Computer Vision and neural networks, and also the different types of filters and their significance and the math behind them.

The entire analytics is done with the PIL package. I wouldn't claim that it's a one-stop shop for Image analytics, but for a starter to novice level, it is pretty much it.


LSTM regression bias increases when targets go close to 0 I've build a LSTM model for time series forecasting. Results are not bad, with a mean normalized error of 7%. However, this normalized bias shows a clear pattern: The closer to 0 the value to predict, the higher the bias, as depicted in the figure below:

NOTE 1: True and Pred values are scaled in the figure for clarity

NOTE 2: The bias is not constant, the un-normalized bias (or bias) shows the same pattern

The model:

* 1 LSTM layer with 256 hidden units and no peep-holes
* Dropout in LSTM with keep_prob = 0.8
* One Dense Layer after the LSTM with 128 units with relu activation
* One Dense Layer after the first dense with 1 unit (predictor)
* Adam optimizer, with learning rate = 0.001
* Loss: Mean Squared Error

Any hint?

I found a solution for the problem: scaling the features and the labels.
By performing min-max scaling over the features and labels, the curves become:

The error is still showing a pattern related to the target value, but the normalized error is now really low...
I think this is a nice example of the benefits of normalizing features...


What does it mean that an hypotesis is consistent? I am studying concept learning, and I am focusing on the concept of consistency for an hypotesis.
Consider an Hypotesis $h$, I have understood that it is consistent with a training set $D$ iff $h(x)=c(x)$ where $c(x)$ is the concept and this has to be verified for every sample $x$ in $D$.
For example consider the following training set:

and the following hypotesis:
$h_1=<?,?,?,Strong,?,?>$
I have that this is not consistent with D because for the example $3$ in $D$ we have $h(x)!=c(x)$.
I don't understand why this hypotesis is not consistent.
Infact, consider the following hypotesis:
$h=<Sunny,Warm,?,Strong,?,?>$
this is consistent with $D$ because for each example in $D$ we have $h(x)=c(x)$.
But why the first hypotesis $h_1$ is not consistent while the second,$h$, is consistent?
Can somebody please explain this to me?

I'm not especially familiar with this but from the example provided we can deduce that:

An hypothesis is a partial assignment of values to the features. That is, by "applying the hypothesis" we obtain a subset of instances for which the features satisfy the hypothesis.
An hypothesis is consistent with the data if the target variable (called "concept" apparently, here EnjoySport in the example) has the same value for any instance in the subset obtained by applying it.

First case: $h_1=<?,?,?,Strong,?,?>$. All 4 instances in the data satisfy $h_1$, so the subset satisfying $h_1$ is the whole data. However the concept EnjoySport can have two values for this subset, so $h_1$ is not consistent.
Second case: $h_2=<Sunny,Warm,?,Strong,?,?>$. This hypothesis is more precise than $h_1$: the subset of instances which satisfy $h_2$ is $\{1,2,4\}$. The concept EnjoySport always have value Yes for every instance in this subset, so $h_2$ is consistent with the data.
Intuitively, the idea is that an hypothesis is consistent with the data if knowing the values specified by the hypothesis gives a 100% certainty about the value of the target variable.


Frequency bin of the positive and negative frequency I am using this MATLAB tutorial for Frequency-Domain Linear Regression. There is one part of code they provide, where it is necessary to determine the "frequency bin of the positive and negative frequency", for fft and ifft:

freqbin = 72/12;
freqbins = [freqbin 72-freqbin]+1;
tsfit = zeros(72,1);
tsfit(freqbins) = tsdft(freqbins);
tsfit = ifft(tsfit);
mu = mean(ts);
tsfit = mu+tsfit;

Length of time series is 72, and 12 months is considered as one cycle. How there can be only one frequency bin for positive frequency? How do we know that it is exactly 72/12 + 1 (+1 is because first bin is for zero frequency)? Is this some formula or what?

I don't understand what is exacty frequency bin, and why there is only one bin for positive frequency? I thought that we have many bins in time series.

The tutorial you are looking at is attempting to model only the seasonality of a time-series.  They first look at power of different frequencies, in the second figure in the tutorial.  They find a lot of power for the 1/12 frequency, as well as a few other harmonics, and also the 1/72 frequency.  But they decide that since the 1/12 frequency has the most power, and most of the others are harmonics of this, they will only model the 1/12 frequency.  You can see this from this quote:

  Based on the spectral analysis of the data, fit a simple linear
  regression model using a cosine and sine term with a frequency of the
  most signficant component: 1 cycle/year (1 cycle/12 months).

So you therefore only need one frequency bin, and its negative, since you are only modeling one frequency.  In other words, in this part, they are just using essentially the equivalent of a single sine wave (at the 1/12 frequency) as their model to fit the data.  All the frequency bin variable does is to act as an index and select the component of the fft() that contains the 1/12 frequency.

They then analyze their result (the autocorrelation plots) and find that they have indeed missed some of the cyclical patterns at other frequencies they did not model.  So then they add three more frequencies to the model.  Rather than using frequency bins this time to do the indexing, they simply ask for the 6 highest in power (6 because each of the 3 needs both positive and negative frequencies) components of the fft() with these lines:

[Y,I] = sort(abs(tsdft),'descend');
indices = I(1:6);
tsfit2dft(indices) = tsdft(indices);

You can see the resulting plot that follows now better models the time series since it includes more of the frequencies that showed increased power.


Cost of greater than 1, is there an error? I'm computing cost in the following way:

cross_entropy = tf.nn.softmax_cross_entropy_with_logits(y, y_)
cost = tf.reduce_mean(cross_entropy);

For the first cost, I am getting 0.693147, which is to be expected on a binary classification when parameters/weights are initialized to 0.

I am using one_hot labels.

However, after completing a training epoch using stochastic gradient descent I am finding a cost of greater than 1.

Is this to be expected?

The following piece of code does essentially what TF's softmax_cross_entropy_with_logits functions does (crossentropy on softmaxed y_ and y):

import scipy as sp
import numpy as np

def softmax(x):
    e_x = np.exp(x - np.max(x))
    return e_x / e_x.sum(axis=0)

def crossentropy(true, pred):
    epsilon = 1e-15

    pred = sp.maximum(epsilon, pred)
    pred = sp.minimum(1-epsilon, pred)

    ll = -sum(
        true * sp.log(pred) + \
            sp.subtract(1,true) * \
            sp.log(sp.subtract(1, pred))
    ) / len(true)

    return ll

==

true = [1., 0.]
pred = [5.0, 0.5]

true = softmax(true)
pred = softmax(pred)

print true
print pred

print crossentropy(true, pred)

==

[ 0.73105858  0.26894142]
[ 0.98901306  0.01098694]
1.22128414101

As you can see there is no reason why crossentropy on binary classification cannot be > 1 and it's not hard to come up with such example.

** Crossentropy above is calculated as in https://www.kaggle.com/wiki/LogarithmicLoss, softmax as in https://en.wikipedia.org/wiki/Softmax_function

UPD: there is a great explanation of what it means when logloss is > 1 at SO: https://stackoverflow.com/a/35015188/1166478


How to model product popularity by their prices and sales volume? I have a bunch of sales data of similar products and I want to know which product might be the best popular one.

However, for the same category of items, there are cheap ones and expensive ones, and the sales volume of the cheap ones are always higher than the expensive ones.

For instance, we have some ceiling lamps which cost 45\$, and some are 200\$. If the sales volumes for the 45\$ ones and 200$ are 200 and 180 respectively, even the cheap one is sold more than the expensive one. We know that the expensive one is actually more popular then the cheap one.

How can I model the popularity by this kind of information?

We know that the expensive one is actually more popular then the cheap one

What makes you think that? Apparently you have a reason for that, so start from there, find those attributes - is it high customer satisfaction? positive reviews? repurchase? low retention?

Gather those attributes together, evaluate them on all products to see what are the popular ones. It'd help to have a proper definition of popularity.

Sales volume can help too, give it a weight to balance the price difference.

Once you have all this in one table, you can start playing with prediction, clustering, trends etc.


How to approximate constant parameters of any generalized function? I am new to Data Science and Machine Learning.
I am trying to approximate constant parameters of any generalized not necessarily polynomial function given the input, desired output and the form of the function using neural network.
Is there any other way to do it using neural network or should I use some other technique?
For example this is one of the functions whose parameters par6 and par7 I am trying to approximate:

I have the values for x and f2(x).
Any help will be appreciated.
Thank You.

There are general techniques around.  E.g., https://en.wikipedia.org/wiki/Non-linear_least_squares, https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html

For your specific example, there's a linearization (depending on the values that $y$ can take on):

$$
\begin{align*}
y &= \left[1+\left(\frac{x_1-x_2}{p_6}\right)^{p_7}\right]^{-1} \\
\frac1y-1 &= \left(\frac{x_1-x_2}{p_6}\right)^{p_7} \\
\frac1y-1 &= \left( \frac1{p_6}\right)^{p_7} (x_1-x_2)^{p_7} \\
\underbrace{\log\left(\frac1y-1\right)}_{y'} &= \underbrace{-p_7\log p_6}_{a} + \underbrace{p_7}_b\underbrace{\log(x_1-x_2)}_{x'}
\end{align*}$$
At this point, your original problem is expressed as a linear model.  Of course, the error terms are now skewed from what they originally might have been, and taking the logarithm won't have worked if $y\notin(0,1)$.


How to define the adequate cash prize sizing for hosting a Kaggle or similar compeition? If you take Kaggle as a well known example of data science competition size, how do you know what is an adequate budget for the cash prize size? At least to determine an order of magnitude, given I am not able to study all previous competitions, cluster them by this factor and assess driving factors? irony - could be a competition itself?

I've also looked up the Q&A page at Kaggle and have found more or less same question but without answer yet:

We would like to understand:

  What is the minimum prize pool that starts capturing your attention?

  ..

  Open-ended question - could you give us idea on what are the most important things you take into account before deciding to participate
  are not?

This is a generic question that may need more than just cash prize.

1) Organizational Reputation
2) Dataset Size
3) Defining Problem statements with Domain knowledge
4) Time period
5) Computation required for the contest.
6) How curated/clean is the dataset
and much more.

If the problem statement is huge and the reward is in single digits, its not worth anyones time and at the same time, if the problem statement is mediocre and huge money is rewarded, it does not serve the purpose of the Kaggle competition at all.

even though Kaggle provides GPU and 16GB ram with CPU's, sometime the Kernel crashes and there is no work around unless you spend some money on AWS/Azure instances.

I hope this gives few pointers


Classification based on a Clustering Result Here is the Sample data:

Values    Attribute1    Attribute2    Attribute3    Attribute4
  2.5       1980            A            1.5           C
  1.8       2000            A            0.4           B
  12.4      2017            S            18.5          D
  0.4       1987            R            10            P
......

In my real data, I have more than 60 attributes. What I need to do is:

Category Values into different groups
classify these groups using Attribute1 - Attribute4.

Some of the difficulties are:

Values mean something in my application. For example, Values>10 will be put into one group. So, it might be not ideal to use a clustering algorithm such as density based to divide Values into groups.
When I use the groups based on Values and all the attributes to run a classification, I might need to choose some important attributes.

What I want is to give Values(meaningful data) to a clustering algorithm. So, that I can get best results/Insights from that Clustering.

So, here are my Questions:

How to cluster?
How to choose attributes?

I think you need to do some Feature Engineering, i.e., as you explained in the question, those values mean something to your application.

For example : 1-3 : Bad, 4-6 : Average, 7-10 : Good

V1 new_V1
3   Bad
5   Average
7   Good

Something like this, so that clustering algorithm can make sense out of it.

Assuming that you don't have any constraint in using R.
If you have both Numeric and Categorical variable(as above) then you can usethis package : ClustMixType, it can understand and do the clustering based on the data fed. As traditional K Means algorithm is not applicable here as it works only for Numeric Data.

If there is any discrepancy in data you can convert them explicitly by
new_V1 <- as.factor(new_V1);
if it is a numeric variable V2 <- as.numeric(V2) before feeding the data to Clustering algorithm.

Once you get it, then you can use this package: Bruota, use this package to get Predictor Importance(what all variables are important). This can be done only when you know the Target Variable, Most likely your target variable would be the Clustering outcome.

Implementation is over here, please follow this Link

Most likely you would end up getting your desired result, Do let me know if you have any additional questions.


Date transformation for KNN I have data set with date features like 01/01/2019 and I would like to use KNN. However, I cannot find a good transformation for dates that has a meaningful distance result for the last feature.

For example:

f1 | 1 | 2 | 3 | 4 | 01/01/2019

f2 | 10 | 3 | 12 | 1 | 14/01/2019

Does anyone have any recommendations?

Find the time difference in an appropriate granularity(/ies):

daysDiffFeature = diffInDays(dateCol)
hoursDiffFeature = diffInHours(dateCol)

Then you can use whatever distance based metrics/calculations are required in your algorithsm


neural network with large numbers of inputs I learn about neural networks with a heavy excercise. I have products and want to forecast the revenue. I have 10 features. But 4 features  have large numbers of expressions. So my "group" features has over 500 different groups (of products). If I do binarization with the features I end up with vectors of over 1000 dimensions as input.
All the product classes are names with letters and numbers such as "100XA9". Now there are 500 of those. For a neural network I have to do binarization with those names. Is that right?
Then my input layer is 1000, right?

My question is this "normal"? And is this problematic for computation?

I mean this is a typical question. A store with many products (like over 500) will forecast its revenue based on the prices and other features.

I hope you can help me to untestand how to use neural networks for classification.

One-hot encoding is the normal approach, and yes you would end up with 500 features for your group alone. Depending on how much training data you have this does not need to be a problem. If you have a lot of other features without direct interactions between the individual products you could use an Embedding layer before adding them to the rest, which maps your sparse categorical one-hot encoded features into a dense space via backpropagation. This will reduce the amount of parameters significantly.

If you don't have enough training data to do this, you could look at gathering statistics about your product, that say something about other features for the group, or about the target for other rows (be sure not to use the current row, which would introduce target leakage) which would allow you to drop the category all together.

Turning them into IDs and then using this as a numerical feature is a bad idea, since there is no inherent structure in the numbers, which means there is no signal in that and it would need a very high complexity to learn how to distinguish 1 and 5 being very similar but 2 and 4 very different.


How to set class weights for imbalanced classes in Tensorflow? Do we have an equivalence for the following question, but with tensorflow : How to set class weights for imbalanced classes in Keras? ??

There's a function that does it automatically: tf.contrib.losses.softmax_cross_entropy(logits, onehot_labels, weight=weight)


How to divide a dataset for training and testing when the features and targets are in two different files? I am trying to divide a dataset into training dataset and testing dataset for multi-label classification. The datset I am working on is this one. It is divided into a file which contains the features and another file which contains the targets. They look like this below.

This is the image about the features.

This is the image about the targets.

I intend to use this dataset for multilabel classification. I am following this tutorial . Here the dataset looks like this.

The dataset that I am working on has 17203824 samples and 58255 different and unique labels in the target file. So to follow the tutorial what I intend to create is a new numpy 2d array with 17203824 rows and 58255 columns where appropriate indices will be marked with 1. I am able to create it. But when I try to populate with 1s in the appropriate indices, I am getting an error. Its says that I don't have enough memory. My code is given below.

questions = pd.read_csv("/kaggle/input/stacklite/questions.csv")
question_tags = pd.read_csv("/kaggle/input/stacklite/question_tags.csv")
d = {v: i[0] for i, v in np.ndenumerate(question_tags["Tag"].unique())}

y = np.zeros([questions.shape[0], len(question_tags["Tag"].unique())], dtype = int)

for k in question_tags["Tag"]:
    j = d[k]
    for i, l in enumerate(y):
        y[i][j] = 1

Can anyone please help in telling me how I should proceed?

You can load the features in a variable (X) and the 2nd column of targets in another variable (y). Then use train_test_split available in sklearn library.

X_train,X_test,y_train,y_test = train_test_split(X,y, test_size=__)

BTW you need to preprocess the data first. There are a lot of NAN values, id and date probably has no value in predicting the target.


Purpose of weights in neural networks I'm beginner at Neural Networks. After reading multiple articles on wikipedia, i've seen the term "weight" being used a lot, although it is a little confusing.

I know, that before the inputs are summed and passed to activation functions, they are separately weighted, after some research, i found out that the purpose of the weight function was to:

ensure orthonormality
avoid data loss

I know that two inputs are only orthonormal if they are orthogonal unit vectors, if they are orthogonal then their dot product is always 0.
Dot product:

where theta is angle between two unit vectors and absolute value of a is norm.

For example, if two unit vectors are perpendicular to each other, then they are orthonormal and their dot product is 0.

But what does this have to do with data loss? I've also heard that sometimes the value of input might be zero, and we know that multiplication by zero outputs zero.

So what is the real purpose of the weight in neural networks? and what does it have to do with orthonormality? For example, what would be the purpose of weights in linear regression?

The reason for weights in machine learning is actually a lot easier than it seems. It's the way by which our model learns some underlining function and performs the classification or regression. We tune these weights in order to model some underlining function which can map our input to a desired output. Either a class in classification, or a range of values in a regression.

Let's look at an easier machine learning model so that we can understand why weights are needed.

Linear Regression

This is just a straight line which splits data into two sections. Let us apply this model in 2D, where we have two features $x = [x_1, x_2]$, for example weight and height. And labels $y$ which will split our data into $y \in \{men, women\}$.

A random line in this space is defined as

$0 = -2x_2 + x_1 + 1$.

Assume this is our boundary line. Let's just assume that women fall above this line and men fall below it. Like this

Now if we get a new data point $x_{new} = [2, 8]$ we will label this as being a woman. So the entire decision is based on the numbers $1, -2, 1$ from our linear equation. We need to tune these values using the training data. We usually call these trainable parameters the weights $w$ associated with the features $x$ and we also add a bias $b$. In general the linear separator in 2D is

$0 = w_1x_1 + w_2x_2 + b$.

Our predicted label is $\hat{y} = w_1x_1 + w_2x_2 + b$. If $\hat{y} > 0$ then woman, else man.

In $n$ dimensions this can be written as a matrix multiplication as

$\hat{y} = w^Tx + b$

Obviously, a linear separator is not sufficient for most classification tasks. Things are not always linearly separable. So we use more complex models.

Neural networks

In neural networks each node is associated with a function much like the linear separator. However, we will use the sigmoid function

$\sigma(w^Tx) = \frac{1}{1 + exp^{-(w^Tx + b)}}$.

The weights here have the same effect. They will modulate the input values $x$ such that we are able to learn some classification or regression.

How do the weights affect the decision boundary. We want the two different classes, circles and x's, to be on opposite sides of this boundary in order for us to be able to correctly classify them.

The weights are trained iteratively using gradient descent. We can see that the decision boundary starts off terribly and then gets progressively better.

$0 = -1.0 - 1.0x_1 - 1.0x_2$

$0 = -16.0 - -39.0x_1 - 71.0x_2$

$0 = -36.0 - -94.0x_1 - 61.0x_2$

$0 = -83.5 - -134.0x_1 - 76.0x_2$

$0 = -88.5 - -114.0x_1 - 98.5x_2$

As you can see we changed the weights until we were able to find this ideal boundary between the two classes. If you want to find our more about how the gradient descent algorithm works to tune these weights you can look here.


Can this be a case of multi-class skewness? I have been working on an email data set, and trying to predict the owner team for it. But my prediction accuracy is just 58%. I have implemented data cleansing, null value removals, duplicate removal, excluding stopwords, and then calculating tf-idf value to get my final document term matrix.

Tried various classifiers (but with default settings) but max I achieved is 58% accuracy. Might give a try to hyperparameter tuning.

Could you please give me some pointers, based on your previous experience what possibly could be going wrong? I'm simultaneously doing my own research and going through various blogs and articles.

One thought - Could it be a case of data skewness?

My output classes are distributed like this -

Your thoughts please?

Just telling the accuracy does not mean anything in classification problems. The first thing you must do is to calculate your baseline, that is, what is the percentage of your majority class? For the bar plot above it is difficult to measure, can you tell us in percentage, instead of counts? In this way we can assess better your results.

Also, have you plot a confusion matrix? In this way you can see where your model are getting more wrong and try to infer why this is happening.

And yes, since you have too many classes to predict and most of them are with low representativeness this will be difficult to overcome. Maybe you can try things such as Oversampling, Undersampling techniques considering one-vs-all approach. This is just an idea, I haven't yet encountering a problem with so many classes to predict.


Estimating Propensity Score via Regression Trees (in R Using rpart) I am trying to estimate propensity scores in R. By this I mean that I am trying to estimate probability that an individual selects into treatment, where the selection into treatment is a binary variable. Typically people use logit or probit to estimate these. I was thinking of using regression trees.

Is it okay to use rpart "anova" method to estimate propensities? I think I cannot use classification tree because that gives a binary outcome whereas I want the continuous propensity score function i.e. how does probability of treatment changes as the continuous explanatory variables changes.

I am using the following command where x and z are continuous explanatory variables and d is a binary treatment variable and all 3 are observed for each individual and I want to estimate P(D=1|X,Z).

  cart = rpart(d~x+z, data=data, method="anova")

  data[,pz_c:=predict(cart, type = "vector")]

where the first command estimates the propensity and the second adds the predicted propensities to the data set for each individual.

I originally posted this on stackoverflow and was told to post thee question here instead.

You can use logistic regression itself, since your target variable is binary.

While predicting, use type = "prob"
This will provide the propensities which is basically just the probability of being a 1 or 0.


measuring flip-flop behaviour across several topics I'm trying to analyze a behavior called "sentiment flipping" of users in a dataset, but I'm not able to step on.

Let's suppose that I have two groups of users, say them good and bad users.

My dataset contains N tweets that classified into 6 topics. The tweets were created by the bad and good users.

The 6 topics are about general issues, but 3 of these topics are about organization/individuals supported (A) by the "bad" users and the other 3 are against (B) their ideologies.

The difference between the bad and good users in their tweeting behavior is:

The good user posted tweets in some of the topics (and maybe all of
them) without forcing "positive" or "negative" sentiment in the
topics.
The bad user posted tweets contain negative sentiment
on the topics against her/his ideologies and positive sentiment
on the topics she/he supports.
The clear difference between both users also is that the bad user posts negative sentiment profusely on B topics and positive sentiment on A topics.

How can I measure/show this flipping behavior in a score/value; given that each tweet is represented by a vector like: <# of Pos words, # of Neg words>.

I think a good solution will consider how dense and ideologically clear the bad user behavior.

This image summarizes the previous description:

So I have a theory for this. We want to be able to distinguish users who are emotional and biased from those that are unemotional and unbiased.
$$M = emotionality\\
B = Bias\\
t_i = tweet\ number\ i\\
b_i = bad\ words\ in\ t_i\\
p_i = good\ words\ in\ t_i\\
w_i = words\ in\ t_i\\
M = \frac{(b_i+p_i)}{w_i}\\
B = \frac{(b_i-p_i)^2}{w_i}\\
$$

M is the emotionality because it calculates the frequency that word is going to be an emotional one (instead of a descriptive one).

B is the bias because biased tweets will have more good (or bad) words than ones that hold both sides equally.

I don't know if this prediction will hold true, but i reckon if you plot each tweets x as bias and y as emotionality with a hue of good or bad, there will be two clusters (i am eager to be shown to be mistaken this certainly sounds like an interesting problem)!


using unsupervised learning algorithms on images I am working on a project to classify images of types of cloth (shirt, tshirt, pant etc). While this is a standard supervised classification problem, the accuracy of the neural network is not good. This is because of the close similarity of the types of cloth that I am trying to classify.

I am working with 9 classes with around 10,000 images per class. For the classification problem I tried using CNN to classify the images. But over fitting took place with a good training accuracy (around 95%), but not so great validation accuracy (around 77%).

I wanted to know if there was any way I could create clusters based on the type of cloth using some unsupervised learning algorithm like K Means or DBScan.

Have you included dropout in your model? It can help avoid overfitting issue.

For your problem, yes, you can use auto-encoders, GAN, etc. for feature learning.
However, I'm not sure if unsupervised learning can help, since it's more like a training issue. Your have label with your data so supervised learning is ideal, plus supervised learning generally shows better performance than unsupervised in image classification. You might want to check the false classification examples in your dataset, and try to alter the CNN structure based on that, which would be a more direct way.


Cross Entropy vs Entropy (Decision Tree) Several papers/books I have read say that cross-entropy is used when looking for the best split in a classification tree, e.g. The Elements of Statistical Learning (Hastie, Tibshirani, Friedman) without even mentioning entropy in the context of classification trees.

Yet, other sources mention entropy and not cross-entropy as a measure of finding the best splits. Are both measures usable? Is only cross-entropy used? Since the two concepts significantly differ from each other as far as my understanding goes.

Are both measures usable? Is only cross-entropy used?

They both could  be used for this special case. However, I personally prefer "entropy" because it requires less mental gymnastics.

Let's first review the definitions. The most agreed upon and consistent use of entropy and cross-entropy is that entropy is a function of only one distribution, i.e. $-\sum_x P(x)\mbox{log}P(x)$, and cross-entropy is a function of two distributions, i.e. $-\sum_x P(x)\mbox{log}Q(x)$ (integral for continuous $x$).

The formula used in The Elements of Statistical Learning [Page 308, 9.2.3 Classification Trees] can be written as
$$-\sum_k P_{m}(k) \text{log}P_{m}(k)$$
where $P_{m}(k)$ is the ratio of class $k$ in node $m$. This could be interpreted as a function of only one (data) distribution, i.e. an entropy, that measures the impurity of node $m$. Nonetheless, it can also be interpreted as a cross-entropy between data distribution and model estimation (based on @DrewN nice explanation), i.e.
$$-\sum_k P^{\text{data}}_{m}(k) \text{log}P^{\text{model}}_{m}(k)$$
where we, hypothetically, set the model estimation to match the data distribution in node $m$, i.e.
$$P^{\text{model}}_{m}(k)=P^{\text{data}}_{m}(k) = P_{m}(k)$$
to minimize the cross-entropy.
Accordingly, cross-entropy would be the same as both data entropy and model entropy in value, i.e.
$$\begin{align*}
\overbrace{-\sum_k P^{\text{data}}_{m}(k) \text{log}P^{\text{model}}_{m}(k)}^{\text{data-model cross entropy } H(P^{\text{data}}_{m}, P^{\text{model}}_{m})}&=\overbrace{-\sum_k P^{\text{data}}_{m}(k) \text{log}P^{\text{data}}_{m}(k)}^{\text{data entropy } H(P^{\text{data}}_{m})}\\
&=\overbrace{-\sum_k P^{\text{model}}_{m}(k) \text{log}P^{\text{model}}_{m}(k)}^{\text{model entropy } H(P^{\text{model}}_{m})}
\end{align*}$$

but it is different in meaning and rightfully has a different name.  I say "hypothetically" because in practice, classifier only chooses the class with maximum probability, i.e.
$$P^{\text{classifier}}_{m}(k)=\left\{\begin{matrix}
 1 & k=\underset{k'}{\text{argmax }}P_m(k')\\
 0 & \text{o.w.}
\end{matrix}\right.$$

From another perspective, when cross-entropy is equal to entropy, it means KL divergence is zero
$$\text{KL}(P^{\text{data}}_{m} \parallel P^{\text{model}}_{m}) = H(P^{\text{data}}_{m},P^{\text{model}}_{m}) - H(P^{\text{data}}_{m})=0$$

All in all, we can still confidently use "entropy" for decision trees when we talk about node splitting and node impurity. For example, a split occurs when entropy of class distribution in parent node is higher than the weighted-average of class entropies in left and right children (i.e. positive information gain).

As an extra note, cross-entropy is mostly used as a loss function to bring one distribution (e.g. model estimation) closer to another one (e.g. true distribution). A well-known example is classification cross-entropy (my answer). Also, KL-divergence (cross-entropy minus entropy) is basically used for the same reason.


SGDClassifier: Online Learning/partial_fit with a previously unknown label My training set contains about 50k entries with which I do an initial learning. On a weekly basis, ~ 5k entries are added; but the same amount "disappears" (as it is user data which has to be deleted after some time).

Therefore I use online learning because I do not have access to the full dataset at a later time. Currently I'm using an SGDClassifier which works, but my big problem: new categories are appearing and now I can't use my model any more as they were not in the initial fit.

Is there a way with SGDClassifier or some other model? Deep learning?

It doesn't matter if I have to start from scratch NOW (i.e. use something other than SGDClassifier), but I need something which enables online learning with new labels.

It sounds like you don't want to start retraining the model every time a new label category appears. The easiest way to retain maximal information of past data would be train one classifier per category.

This way you can continue to train each classifier incrementally ("online") with something like SGDClassifier without having to retrain them.
Whenever a new category appears you add a new binary classifier for just that category. You then select the class with the highest probability/score among the set of classifiers.

This is also not much different from what you are doing today, because scikit's SDGClassifier already handles the multiclass-scenario by fitting multiple "One vs All" classifiers under the hood.

If a lot of new categories keep coming up, of course, this approach might become a little tricky to manage.


Generate tags for live chat transcripts I'm wondering if there's a way to automatically generate a list of tags for live chat transcripts without domain knowledge.  I've tried applying NLP chunking to the chat transcripts and keep only the noun phrases as tag candidates.  However, this approach would generate too many useless noun phrases.  I could use some rules to prune out some of them, but it would be hard to generalize the rules.

You can try RAKE(Rapid Automatic Keyword Extraction) and there is a python implementation here. RAKE is an document-oriented keyword extraction algorithm and also language-independent(theoretically, since RAKE use a generated stop word list to partition candidate keywords, and considering different languages, we need to find a better way to generated stop word list.). However, about English documents, RAKE can extract keywords(or tags) in a acceptable precision and recall. RAKE is also efficient, because to use it we don't have to training a whole corpus, RAKE can generate a keyword list by calculating the word's degree and frequency then comes up a score for every candidate keyword then pick the top N words.
Hope this answer helps you or lead you a direction for your next step investigation.


How to leverage description data in multi-class classification (dimensionality reduction) I'm currently working with a dataset of 55k records and seven columns (one target variable), three of which are nominal categorical. The other three are 'description' fields with high cardinality, as would be expected in many cases of description data:

in>>
df[['size description', 'weight Description', 'height Description']].nunique()

out>>
 size Description       4066
 weight Description      736
 height Description     3173
 dtype: int64

some examples of these values could be:

   Product                  Product Description
  ---------               ------------------------
   Ball                 Round bouncy toy for kids
   Bat                  Stick that kids use to hit a ball
   Go-Kart red/black    Small motorized vehicle for kids
   Go-Kart blue/green   Small motorized vehicle for kids
   Wrench               Tool for tightening or loosening bolts
   Ratchet              Tool for tightening or loosening bolts
   Reclining arm-chair  Cushioned seat for lounging

I think that the descriptions are standardized if they fall within a particular category but at this time I cannot confirm if the number of unique descriptions are finite. At this time, my assumption would be to treat these as nominal-categorical, as these are literally descriptive and not qualitative.

To that end, my question is what are some best practices for handling categorical features such as these?

Things I have considered:

Label encoding is obviously not viable in this situation, as the descriptions have no hierarchy.
One-hot encoding seems an unlikely solution as it balloons the shape of the dataset from (55300 , 6) to (55300 , 65223) due to the high cardinality of the other variables. However, I tried it anyway and generated 98% accuracy on my test set but very poor results on an out-of-sample validation set (5k records, 0-5% accuracy). Seems pretty clear it's over-fitting and thus, not viable.
Hashing, for whatever reason, will not apply to one of the columns, but I suppose it could still be viable. I just need to figure out why it's not hashing all of my features (I suppose this would be suited best for a separate question?)
PCA - could be viable, but if I'm understanding correctly the cardinality after one-hot encoding is too great and PCA will throw an error. In fairness, I have not tried this yet.
Binning doesn't seem feasible since I could have a value of 3.5 or three and 1/2. Each one would be considered a separate bin and thus not a solution to my problem.

Thanks to all that can share their insight/opinion.

You have textual descriptions, i.e. unstructured data. So you should probably use one of the standard representation methods for text. There are many options including sentence embeddings and this kind of advanced methods, but I'm going to describe the simple traditional option:

Each description value can be represented as a vector of feature, one for each word in the full vocabulary (i.e. over all the values for this field).
The value of each feature can be boolean (i.e. does this word appear in this description) or better a TF-IDF weight for the word.
Obviously this would lead to too many features, so one needs to apply select the most relevant ones. This part is very experimental, you might have to try various options to find the right one:

Get rid of the stop words since they provide no semantic information.
Discard all the words which appear only once, and probably also all the words which appear less than some minimum frequency $N$ (try with $N=2,3,4...). The rationale is that rare words are more likely to cause overfitting than to really help any kind of classification.
Beyond that, you could use general feature selection (e.g. information gain) or feature clustering.

[ obsolete answer to the first version of the question ]

I would definitely try to normalize these values, because semantically they are numerical and in their original form they are almost useless, no matter the method to categorize them. Making them categorical loses a lot of information, especially the ones which actually provide a number. Since some of the strings used are very vague I would probably try using intervals, i.e. two numeric values for every original input value:

three and a half inches, three and 1/2 inches -> min 3.4 - max 3.6
27.6234 inches -> min 27.6234 - max 27.6234
tall -> large range high values
short, kinda short -> large range low values

Normally there are not that many ways to give numbers as words, only a few patterns should be enough to capture all the variants. For the other non-standard cases such as "kinda short" I would start by looking in the data their distribution: if a value is frequent enough (e.g. probably "short", "tall") then manually predefined range. Values which are not frequent can be ignored, e.g. replaced with NA (since they are not frequent that shouldn't affect the data too much).


tensorflow in production I am using tensorflow-serving to write a server to consume models in production.
I have a question about consuming the service by clients: does tensorflow-serving support a REST API? Is there is anyway to modify it?

I have checked several github projects: here, here, and here.

I have an answer now for my question.
I will share briefly the main steps / technologies I used to deploy the model in production.

I am using Python programming language.
After training and generating valid models I wrote a restful api using Python programming language and flask.

Using flask you can write a restful api. Three important points:

1- It is very important to give attention to where you will define the model architecture/initialize the parameters/ define the session.
Avoid doing this each time you call the restful api. this will be very expensive.

2- Flask provide a very good mechanism to run servers in production environment.
Read about flask + wcgi
Avoid runing the server code (the resful api) directly, in this case you will not have direct and full control.

3- Watch the memory and the cpu usage, make sure to limit the maximum number of instances that can run in parallel. Remember these models can take a lot from the memory.

unfortunately, I can not share codes with public. But hopefully my answer can give an idea about how to do it in production.


Can you provide examples of business application of vector autoregressive model? Vector Autoregressive models are exploited at Economics faculties all around the world. They are just another statistical model that solves problem of forecasting, although in a deeply complexity-uncovering manner.

Yet to my surprise, there is no evidence it has been used outside pure economics domain, namely, to solve business problems like we all - Data Scientists - do.

Can you share either your experience with application of VAR to solve business problem, a scenario in which it could hypothetically be used or state why it cannot be easily applied in business environment (if this is an objective, mathematically/empirically grounded reason, I am not looking for opinions on why this method has slow adoption)?

The question can apply as well to VECM models of course.

Caveat: I have a doctorate in economics and that is why I knew how, and where, and when to apply this type of model.

Sure, I used a vecm model last year to figure out how many credit cards get compromised per month given how much activity we see. (I work for a financial institution).  We were looking for a long-run relationship and a short-run relationship. From this model, a survival analysis, and a couple of other techniques we were able to determine the optimal time to set the expiration date on a card. I applied this technique because I was afraid I was getting the wrong answer, namely because the number of transactions and number of fraud transactions were cointegrated.(All of the models I tried gave similar results which was comforting). Given that we are talking about fraud strategies I don't feel comfortable giving more details than this, hopefully, that gives you a flavor of what I was trying to accomplish. Basically, I was trying to do a robustness check.

It isn't easily applied in a business context because this model isn't intuitive, it isn't easily explained, and it takes great skill to interpret it. The "mathematical" answer probably has something to go like this, ask an average data scientist what the impulse-response function for their model looks like and you will get a blank stare. Ask them what the FEVD looks like for their model, and you will get a blank stare. Data scientists tend to be trained in non-linear models (read machine learning), not cointegrated time-series models. It isn't that they aren't capable, they've just never been trained to think in those terms. I know technically not math, and technically that is an opinion but it is probably the truth.

Also doing the math, I can probably get a better fit out of a non-linear model like an LSTM network(?). So if I only care about pure prediction, it wouldn't be worth my time. Where VAR and VECM shine is that although complicated, they are essentially linear, and therefore, fairly interpretable. So perhaps what you are looking for is a business setting that requires causally valid, interpretable, multivariate time-series models. If you look hard enough, I'm sure you can come up with a couple of interelated KPIs being tracked in your business where it would make sense.

But in my experience, if you need a fast dirty answer, forgive me econometrics teachers everywhere, you can just plug the differenced series into a linear regression. (Warning not best practices, but it does alright).


Categorical Clustering of Users Reading Habits I have a data set with a set of users and a history of documents they have read, all the documents have metadata attributes (think topic, country, author) associated with them.

I want to cluster the users based on their reading history per one of the metadata attributes associated with the documents they have clicked on. This attribute has 7 possible categorical values and I want to prove a hypothesis that there is a pattern to the users' reading habits and they can be divided into seven clusters. In other words, that users will often read documents based on one of the 7 possible values in the particular metadata category.

Anyone have any advice on how to do this especially in R, like specific packages? I realize that the standard k-means algorithm won't work well in this case since the data is categorical and not numeric.

Sounds like you are set to do collaborative filtering - you have readers (users) and documents (items) creating a unary-response matrix with 1 cells indicating the reader (row) read the given document (col). There are various types of recommender systems (e.g., user-based and item-based collaborative filters) and several of them can account for the meta-data associated with your documents (e.g., content-based systems). You should look into R package {recommenderlab}


Pyspark: Filter dataframe based on separate specific conditions How can I select only certain entries that match my condition and from those entries, filter again using regex?

For instance, I have this dataframe (df)

+----+----+----+----+
|col1|col2|col3|col4|
+----+----+----+----+
|   A|   f|   5|   g|
|   D|  er|  2e|  sd|
|   F|   g|23sd|   a|
|   F| fgf|  45|   d|
|   E|   r|   3|   e|
|   A|  sd|  8f|  dw|
|   F|  sd| 3h1|   d|
+----+----+----+----+

I want to select those entries with 'F' value in col1, and filter again with regex ([a-zA-Z0-9]+) to get only entries with numbers and letters.

+----+----+----+----+         +----+----+----+----+
|col1|col2|col3|col4|         |col1|col2|col3|col4|
+----+----+----+----+         +----+----+----+----+
|   F|   g|23sd|   a|   -->   |   F|   g|23sd|   a|
|   F| fgf|  45|   d|         |   F|  sd| 3h1|   d|
|   F|  sd| 3h1|   d|         +----+----+----+----+
+----+----+----+----+

You can use the filter method on Spark's DataFrame API:
df_filtered = df.filter("df.col1 = F").collect()
which also supports regex
pattern = r"[a-zA-Z0-9]+"
df_filtered_regex = df.filter([df_filtered.c.rlike(pattern) for c in df.columns]).collect()`


cosine similarity between items (purchase data) and normalisation I'm using IndexedRowMatrix which represents the products's user purchase behaviours and in order to build product recommendations, I use cosine similarity to calculate similarities between products. PySpark provides a function called columnSimilarities() to do that.

My question is, do I need to normalize each product's vector before using columnSimilarities()? I read about normalisation and cosine similarity and understood that cosine similarity normalises vectors already, as if we normalize the vectors, then the cosine similarity will be just the dot product of the 2 vectors. Reference

Also, one of the answers in this question Cosine similarity versus dot product as distance metrics suggests that Sometimes it is desirable to ignore the magnitude, hence cosine similarity is nice, but if magnitude plays a role, dot product would be better as a similarity measure. which means cosine similarity and dot product are not the same..

I'm confused about the difference and when it's good to use normalisation before calculating cosine similarity and when it's not? and what are the different ways to normalize?

Any help?

My question is, do I need to normalize each product's vector before
  using columnSimilarities()?

No, you do not need to normalize each product's vector before using columnSimilarities() since it is performed within the operation already. I think your confusion comes from the fact that your considering dot product and cosine similarity to be the same. They are not.

Dot Product is only a component of the Cosine Similarity Function Denoted as "A (dot) B"

With regards to the answer you referenced, the solution suggests using the dot product as an alternative to cosine similarity because a dot product calculation is not affected by the magnitude. The dot product doesn't use the mean as part of its calculation.

  when it's good to use normalization before calculating cosine
  similarity and when it's not?

I would advise against using normalization before cosine similarity. However, there are other methods you can consider if cosine similarity is not returning desired results. You can use adjusted cosine similarity or dot product(as referenced in the answer you linked). Both of these measures take into account differences in magnitude. The adjusted cosine similarity subtracts the mean before calculating cosine similarity. Dot product doesn't use the mean in its calculation.  Which is important in your context. For example, Users who always rate products 5 stars and users who always rate products 1 star are not equivalent.

  what are the different ways to normalize?

There are a lot of ways to normalize a vector. You can use Z-score standardization, the standard min-max scaling, l1 or l2-normalization,etc etc

Further Information: Is feature normalization needed prior to computing Cosine distance


How to evaluate sequence to sequence models? I wonder how to evaluate variable long sequence-to-sequence predictions?
Let us say I have the following $Y$ and $\hat{Y}$

$Y = [["1", "2", "2"], ["3", "2", "2"], ["1", "3", "2", "2"]]$

$\hat{Y} = [["1", "3", "2"], ["3", "3"], ["1", "3", "2"]]$

Shall I use a binary comparison where any mismatch counts as zero and any full match as one? Or shall calculate conventional accuracy by character-wise comparison?

My concern here is that on one hand, if this is a numeric predcition then any digit mismatch spoils the whole number so doesn't really matter where is the mismatch; on the other hand it would be nice which digits tend to miscalculated in order to find ways to improve the training sets.

Addition: the task is a numeric OCR so -- in contrast with a machine translation job where minor mistranslations are tolerable -- any digit mismatch could result significant business problem (different invoice sums for example). Moreover I'd like to know which individual digits tend to be misread more often so I need a way to get a statistic this way also.

Regarding your concern, there is no reason for you to choose only one evaluation metric. If there are several values that give you different views of the performance of the system, then compute all of these values. The evaluation should depend on your specific use case, so the important thing is that the values correlate with a good or bad performance of the system in the real world problem.

Even though I don't know exactly what your system is expected to do [written before question edit, see below for addition], maybe you could take as reference how performance is measured in speech and text recognition tasks. There you have a reference and a predicted sequence composed of characters and these characters form words. The length of the reference and the prediction is not necessarily the same. The performance is measured both at character and at word level (character error rate (CER) and word error rate (WER)). When measured at word level, even if only one character is wrong, the whole word is wrong. In both cases the main idea is to compute the levenshtein/edit distance (either at word or character level) between the reference and the prediction and then divide by the length of the reference.

In your example

Y=[["1","2","2"],["3","2","2"],["1","3","2","2"]]

Ŷ=[["1","3","2"],["3","3"],["1","3","2"]]

At inner bracket level the edit distances for the three inner brackets would be 1+2+1=4, since in the first a "3" is substituted by "2" (one edit) in the second a "3" is substituted by a "2" and a "2" is inserted (two edits) and in the last case again a "2" is inserted (one edit). The total length of the reference is 3+2+3=8. So the final performance measure is 4/8.

I am not sure if each of the inner brackets in your case are different samples (i.e., Ŷ is the whole dataset) or Ŷ is a single sample and you have something similar to characters and words. But anyway, if you do the edit distance between Ŷ and Y and divide by the length of Ŷ then you have 3/3, since all three brackets are different.

Additional response to question edit: If the task is number OCR, then you would compute character error rate. Computing an error value for each of the 10 digits is also possible with the edit distance. It would be by counting how many edit operations correspond to errors in each of the digits. For this in python you can use the editops function from the python-Levenshtein package (https://rawgit.com/ztane/python-Levenshtein/master/docs/Levenshtein.html#Levenshtein-editops).


how to explain the behaviour: linear svm does better than non-linear RBF I am working on a binary class classification problem.
Each sample is a vector 1x101, I have a lot of data samples more than 150k
I tried training a linear svm and a non-linear svm (RBF) "zscore normalization is used in both cases". surprisingly, the linear does better than the svm (RBF).
I am trying to explain this by considering the following points:

I beleive that the quality of my feature is not very good.
I think the nonlinear case experinces a kind of overfitting.

my question is how to explain this behaviour?!! does what I am thinking in make sense?!! I am thinking in using Adaboost to perform  the training, is it a good idea or not?

The quality of your features might actually be better than you think. If they provide linear separability, nonlinear kernels will overfit more readily than a linear kernel, leading to your result.


Similarity between two time series with different sampling frequency, different amplitude, and different lengths but taken from the same source? I have two files with accelerator readings and I want to get some metric/ measurement to get the similarity between these two files. I have tried Pearson’s R coefficient, dtw distance, dtw score. Pearson’s r gives returns a value 1 if the files are identical, the dtw score and path are 0 if the files are identical.
But I need a solution if the files are as the ones in the figures, similar, with a little time lag. They are readings from two different accelerators who were attached to the same source. The sampling frequency and amplitude is not same. Even the number of readings are not same. Time stamps could be different.
How do I measure the similarity between such files? Is there some metric or measurement I can get using Python? Because dtw score and dtw distance do give some output, but there is no way I can say the files are similar using those values.

What you could do is

downsample both series to match sampling rate between them
crop the longest series to match the shorter one

Then, there is a variety of methods to estimate similarity between the two series. Some of those are:

cross correlation: this will be affected by the amplitude and will not be able to estimate lagged correlations, prone to noise.
coherence: normalised frequency based correlation (cross-spectrum), not prone to amplitude or noise.
wavelet coherence: similar to above but based on wavelet transformations instead of STFFT.
dynamic time warping: measuring similarity between two temporal sequences, which may vary in speed.


What is meaning of zip(kfold.split(X, Y) in sklearn What is meaning of zip(kfold.split(X, Y) in sklearn?

for (train, test)in zip(kfold.split(X, Y)):


How to append new numerical features to an embedding from word2vec, such that KNN on embeddings is not biased for one feature? I am working on similarity calculation between entities of similar type. For each entity I am able to make a vector that comprises of multiple vectors itself.

A = 50*1 vector
B = 100*1 vector
C = 50*1 vector
D = [age, gender, x-feature, y-feature, z-feature]

Entity = [A B C D], basically concatenating all of those.

My problem is that: since they come from different spaces, each would have different orders of magnitude.
If I run KNN on these vectors, I doubt that results are going to be governed by features of one space only. What should be done to get best KNN results? What "normalisation" would be best here?

Since you have mixed data, use one-hot encoding for the categorical data so that they become binary data. For the numerical data, normalize them so that their ranges are within [0,1].

Next, you'll have to choose an appropriate distance measure. See the answer here:
https://stats.stackexchange.com/questions/218092/how-to-calculate-the-distance-in-knn-for-mixed-data-types


How to calculate TPR and FPR for different threshold values for classification model? I have built a classification model to predict binary class. I can calculate precision, recall, and F1-Score. Now, I want to generate ROC for better understanding the classification performance of my classification model. I do not know how to calculate TPR and FPR for different threshold values.

To calculate TPR and FPR for different threshold values, you can follow the following steps:

First calculate prediction probability for each class instead of class prediction.

Sorting the testing cases based on the probability values of positive class (Assume binary classes are positive and negative class).

Then set the different cutoff/threshold values on probability scores and calculate $TPR= {TP \over (TP \ + \ FP)}$ and $FPR = {FP \over (FP \ + \ TN)}$ for each threshold value.


Visualizing community composition using network of pie charts Given a social network, I want to perform community detection and compare the result to known node metadata, such as gender, age, etc. to see if certain communities are largely composed of "similar" people. I have seen this done before in visualizations like this:

(image from https://arxiv.org/pdf/0809.0690.pdf)

where each circle represents a community and the coloring of the circle shows the breakdown of some attribute (e.g. nationality) within that community.

Does anyone know what tool can be used to create such a visualization? I have seen it in multiple papers, so there must be something...

Layout algorithms in tools like networkx, igraph, and gephi will associate coordinates with your nodes which you should be able to access fairly easily. Once you have those coordinates, you just need to plot your pie-charts on top of the relevant node location. Alternative, these tools also support using external images as node markers, so instead of building the plots in the same script you could build the pie charts separately, save them to disk, and then associate them with nodes when you draw the graph.

I've never seen an "out-of-the-box" solution for this specific kind of graphic, but it shouldn't be too hard to do this yourself. You just need to figure out how to access the layout coordinates. If you clarify what your preferred analytic environment is and/or graph analysis tool, I can give you more specific advice.

EDIT: I managed to find the code that was used to build the chart in that paper. I searched the paper for "we used" and found this in the acknowledgements:

  We are especially indebted to Aaron Clauset and James Fowler for thorough readings of a draft of this manuscript and to Christina Frost for developing some of the graph visualizations we used.

Searching "Christina Frost UNC" led me to this page which contains a collection of graph visualization tools for matlab. The one you are looking for is at the bottom: drawForceCPie.m. The site is super slow, but it eventually shared the code with me. Here it is for posterity in case the site crashes:

function drawForceCPie(A,XY,scores,gn)

gnu=unique(gn);
CAM=commAdjMatrixSparse(gn,A);
map=colormap;
if min(scores)==0
    map=[.7 .7 .7; map];
end
colorsu=unique(scores);
% colorsu(2:end)=colorsu(randperm(length(colorsu)-1)+1);
% scores2=zeros(size(scores));
% for i=1:length(colorsu);
%      scores2(find(scores==colorsu(i)))=i;
% end
scores2=scores;
nodes=length(scores2);
C=length(map);
colorsu=unique(scores2);
R=colorsu-min(min(colorsu))+1e-10;
Rcolor=C*R/max(max(R));
idcolors = map(ceil(Rcolor),:);
edges=find(CAM);
We=[CAM(edges),edges];
sortWe=sortrows(We);

hold on

alpha=2;
x=XY(:,1);
y=XY(:,2);
str=(CAM/max(max(CAM))).^alpha;
N=length(CAM);

% for i=1:length(colorsu),
%     if colorsu(i)==1
%         h=plot(XY(1,1),XY(1,2),'o','markersize',10);
%         set(h,'Color','k');
%     else
%         h=plot(XY(1,1),XY(1,2),'.','markersize',25);
%         set(h,'Color',idcolors(i,:));
%     end
% end

for ie=sortWe(:,2)',
    i=mod(ie-1,N)+1;
    j=floor((ie-1)/N)+1;
    if (j>i)
        h=plot(x([i,j]),y([i,j]),'k-');
%        set(h,'linewidth',str(i,j))
        set(h,'color',[.5 .5 .5]*(1-str(i,j)));
    end
end
for i=1:length(gnu)
     nodes_percom = length(find(gnu(i)==gn));
     idx=find(gnu(i)==gn);
     radius=15*(((nodes_percom)^(.25))*(pi/sqrt(nodes*length(gnu))));
    comcolors=scores2(idx);
    comcolorsu=unique(comcolors);
     for j=1:length(colorsu)
         percents(j)=length(find(comcolors==colorsu(j)))/length(idx);
     end
     drawpie(percents,XY(i,:),radius,idcolors);
end

hold off

end
function drawpie(percents,pos,radius,colors)

points = 40;
x = pos(1);
y = pos(2);
last_t = 0;
if (length(find(percents))>1)
    for i = 1:length(percents)
        end_t = last_t + percents(i)*points;
        tlist = [last_t ceil(last_t):floor(end_t) end_t];
        xlist = [0 (radius*cos(tlist*2*pi/points)) 0] + x;
        ylist = [0 (radius*sin(tlist*2*pi/points)) 0] + y;
        patch(xlist,ylist,colors(i,:))
        last_t = end_t;
    end
else
    i=find(percents);
    tlist = [0:points];
    xlist = x+radius*cos(tlist*2*pi/points);
    ylist = y+radius*sin(tlist*2*pi/points);
    patch(xlist,ylist,colors(i,:))
end
end

function mat = commAdjMatrixSparse(groups, A)
% Creates a community adjacency matrix using
% groups from the output for reccurrcommsNew2Sparse, A is the adjacency matrix
% 0's on the diagonal, other elements consist of the total number of
% connections between the two communities
%
% Last Modified by ALT 20 June 2007

h=sort(groups);
g=unique(h);
d=diff(g);
f=sort(d);
z=unique(f);
cuts=size(z,2);

[communities cut]=findcommunitiesatcut(groups,cuts);
rows = max(communities);
mat=spalloc(rows,rows,2*rows);

for i = 1:rows
    for j = 1:rows
        if(i ~= j)
            comm1 = find(communities==i);
            comm2 = find(communities==j);
            %comm1=comm1(find(comm1));
            %comm2=comm2(find(comm2));
            mat(j, i) = sum(sum(A(comm1, comm2)));
        end
    end
end
end

function [communities cut] = findcommunitiesatcut(groups,cut)
%[communities cut]=findcommunitiesatcut(groups,cut)
%
% Gives the community numbers at a requested cut or level in the groups vector,
% if the cut number is not valid the program changes it to a valid one.
% Uses a groups vector and a scalar cut number, gives communities and the cut number,
% which is needed when cut is changed.
%
%
%Last modified by ALT, 20 June 2007

%Error checking
n=unique(groups);
f=diff(n);
z=unique(f);
cutmax=length(z);
if(cut>cutmax)
    disp(['That is too many cuts! I have changed the cut number.']);
    cut=cutmax;
elseif(cut<0)
    cut=cutmax;
    disp(['Negative numbers dont work, I have changed the cut number to the max'])
end

%Identify distinct group values and number of cut levels in dendrogram:
groupnumbers=unique(groups);
differences=diff(groupnumbers);
diffnumbers=unique(differences);
cuts=length(diffnumbers);

if cut==0,
    communities=ones(size(groups));
else
    cutdiff=diffnumbers(cuts+1-cut); %NOTE THERE IS NO ERROR CHECKING HERE, ASSUMED VALID CUT NUMBER

    commnumbers=cumsum([1,diff(groupnumbers)>=cutdiff]);

    %Define communities by replacing the groupnumbers values in groups with the
    %corresponding commnumbers values, component by component.
    %Is there a more efficient way to specify this in MATLAB?
    communities=groups;
    for ig=1:length(groupnumbers),
        indx=find(groups==groupnumbers(ig));
        communities(indx)=commnumbers(ig);
    end
end
end

If you use this code for research, I believe this is the citation you should reference (in addition to the UNC webpage that hosted the code):

  "Visualization of communities in networks," Amanda L. Traud, Christina Frost, Peter J. Mucha, and Mason A. Porter, Chaos 19, 041104 (2009).


Catastrophic forgetting in linear semi-gradient RL agent? I've been working through the Sutton + Barto RL text, implementing a number of the algos + running them in the OpenAI gym. One phenomenon that I seem to come across quite regularly is that agents who, at certain points during their training appear to be making good progress towards learning a plausible state-value/state-action functions, "catastrophically forget" the insights they glean and subsequently never recover.

To make this more concrete, here's the (smoothed) reward history of my implementation of a semi-gradient expected SARSA agent with linear function approximation and binary features running on the MountainCar environment.

Problem Details

The problem has a bounded, continuous state space and a discrete, 3-action action space.
The learner receives a reward of $-1$ at each timestep. The episode ends when the car makes its way all the way up the hill OR it reaches 200 timesteps without success.
Prior to learning, I tile-coded the state space using an 8x8 grid with 8 overlapping tilings, generated using Sutton's own tiles.py script.

Learner Details

The learner is an implementation of the semi-gradient SARSA algorithm with linear function approximation for the Q value.
The agent begins with all Q weights initialized to 0.
The learning rate for the agent was set at $\frac{1}{10} \times (\text{# tiles})^{-1}$, where $\text{# tiles}$ in this case was $8 \times 8 \times 8 = 512$
During learning, the agent selected its actions using an $\epsilon$-soft policy, where $\epsilon$ is set to 0.10
The agent's temporal discount factor is set to $\gamma = 0.95$ (though I realize it could have been 1 given the episodic nature of the task)

My Question

One possibility I want to rule out is that my implementation of the learning agent is incorrect. To that end I am curious to know whether others experience this kind of "forgetting" behavior (even on simple problems like this), and if so, how it might be reduced.

The same thing was happening to me with a deep Q Network on the cart-pole  problem. Having a "memory" with past (S,A,R,S) sequences and sampling it to form mini batches with the new observations helped a lot to reduce catastrophic forgetting. Reducing the step size once the agent has improved a certain amount also helped.


How similar is Adam optimization and Gradient clipping? According to the Adam optimization update rule:
$$m \leftarrow \beta_1 m + (1 - \beta_1)\nabla J(\theta)$$
$$v \leftarrow \beta_2 v + (1 - \beta_2)(\nabla J(\theta) \odot \nabla J(\theta))$$
$$\theta \leftarrow \theta - \alpha \frac{m}{\sqrt{v}}$$
From the equations, it is clear that $m$ is accumulated gradient for each $\theta$ based on an exponential decay function and $v$ is doing a similar thing (kind of) but with the magnitude of the gradient. Then, when we update the parameters $\theta$, we divide the accumulated gradient with square root of the accumulated magnitude to only update those parameters largely which haven't been updated much and vice-versa.
In gradient clipping, we do a kind of similar thing by scaling the gradient vector with respect to a threshold. My question is, why do we need gradient clipping to solve the problem of gradient explosion when we can use Adam optimizer to do a controlled search of the space for the minima.

Both have a different role and approach. So, I will say both are not comparable.
Gradient Clipping has a simple task to clip the Gradient to a certain threshold. Its job is done as soon as this is achieved, i.e. Gradient is brought to a decent level to avoid the Explosion.
It has no responsibility to see if learning will converge to best possible minima of Loss.
Adam, on the other hand, is an optimizer. It came as an improvement over RMSprop. The improvement was to have the goodness of both, i.e. Momentum and RMSProp (Read this answer)
Adam is expected to help the learning converge towards the minimum loss function when it is going in Valley or Plateau by managing both the Momentum and Coordinate specific Gradient.
Coming back to your question

why do we need gradient clipping to solve the problem of gradient explosion when we can use Adam optimizer to do a controlled search of the space for the minima.

Let's say, in the first iteration, the Model accumulate very large Gradient because of its depth. So, it will take a considerable jump when updating the weight ($\theta$) and can land to any random coordinates in the Loss function space.
Adam can definitely provide a brake on the Gradient, but it has few challenges -

It needs a few iterations to accumulate the $v$ to apply the brake.
But it is very much possible that during these iterations, the point will take any random coordinates in the space. This randomness will also cause all the coordinates to have random Gradients i.e. no relation to the last Gradient.
This will not let Adam do any accumulation and it will have no clue to act in a corrective manner. Everything will happen haphazardly.

Also, keep in mind, exploding Gradient has bidirectional causation, i.e. explosion causes more Gradient, and then this significant Gradient adds more to the Explosion. So, the learning reaches very quickly to NaN state.


Studying social media platforms - key data analysis methods? I am seeking a basic list of key data analysis methods used for studying social media platforms online. Are there such key methods, or does this process generally vary according to topic? And is there a standard order in which these methods are applied?(The particular context I'm interested in is how the news is impacting on social media)

The kinds of tools you will use will vary based on the problem you are trying to solve. Social media data is rich and therefore many questions can be asked - and many tools can be used.

However, there is a general pattern you might keep in mind. Typically, you will have to use the platform's API to gather data. You will then have to normalized and store the data in a data warehouse. Finally, you will access and analyze the data with the tools you desire. Keeping the end goal in mind, you will have to strategically choose the best technologies for the job you are doing.

For example, let's say you wanted to study the relationships between users on a social network - a question like "Who are the mutual friends of the most popular individuals in this social network?" In this case, you would gather data using the social media platform's API, normalize it into a CSV, import it into a Neo4j database, and then use Cypher to make queries.


What is an expert system I know that

  In artificial intelligence, an expert system is a computer system that
  emulates the decision-making ability of a human expert.

Does the architecture of an expert system same as that of knowledge based system or is there any difference?

What are the stages in development of expert system?

As far as I know, experts systems were popular around the 80s-90s, before the big trend of Machine Learning. They were mostly based on symbolic logic reasoning, as opposed to statistics in ML.

They can probably be considered as one the first stages of ML-based systems: experts systems automate the reasoning steps based on some data, which is a bit similar to ML. However the data was strongly structured (and possibly quite complex) and usually had to be entered manually by an expert. This is the main reason why ML approaches made them obsolete for the most part: ML systems can learn from a vast amount of possibly noisy features with little structure, whereas the cost of preparing the data for experts systems was prohibitive.

Experts systems were indeed a kind of knowledge-based system (probably the first generation), but I think the latter are much more general: knowledge-based system include many different kinds of data base and different methods for processing it.

Note: this answer is based on my general understanding and knowledge of these terms, which is probably correct but not sure ;)


is there any rule to apply pca to the imbalance data? Is there any rule to apply PCA to imbalanced data? (randomforest, xgboost)

I used multiclass imbalance data to pca
but the log-loss accuracy getting decrease
any theoritical background of this?

PCA is method for unsupervised learning so you shouldn't add the label.

Imbalanced data refers to the label.

So for me it does not make too much sense to talk about PCA and imbalanced data.


How can I do the correlation between two estimators? I'm working with several estimators of all kind. Then, I want to stack these estimators, and the best is if they have low correlation between them.

I suppose that the correlation method depends on the type of dependent variable, if it's categorical or numerical. In my case, it's categorical, and the estimators are classifiers.

How can I do the correlation between two estimators?

There are probably many options, I'd look at inter-annotator agreement.

An alternative option would be to to evaluate one (e.g. by accuracy or f1-score) using the other as the gold standard.


How to classify movement data (time series) in real time I have some movement data sampled over a time series. I am trying to classify the movements in real time as either smooth or shaky. For example, as the movement is smooth it is classified as smooth until it suddenly becomes shaky and is then classified as so.

I am having some serious trouble doing this. Aside from using an RNN and feeding it subsets of the time series data to classify (which I do not have enough data to train), does anybody know of any time series classification algorithms? I plan on feeding subsets of the time series into the algorithm with a single label for training and prediction.

Any help is greatly appreciated.

Supervised learning for classification in machine learning trains a model in order to determine which distribution a certain novel instance belongs to. Model complexity is directly correlated with the performance of a specific algorithm given a set number of instances.

This means you should always start with understanding your data before selecting a model.

Your dataset

Your dataset is comprised of instances $x_i \in \mathbb{R}^{300}, i \in {1, ..., n}$ and $10 < n < 50$.

This is a very small number of instances $n$. Usually I recommend to have ten times more instances than features for a two class classification problem. Furthermore, less than $50,000$ instances is not sufficient for training any deep learning model. You will need to consider using traditional machine learning algorithms (SVM, KNN, Random Forests, etc.). Lastly, the class imbalance in your dataset will introduce some bias, this will lead to poorer results.

All hope is not lost, we can still do something to solve this problem!

Feature selection

The first step which I would suggest is to consider performing some feature reduction. You want to map your $X \in \mathbb{R}^{300}$ to a lower dimensional set $\mathbb{R}^{p}$ where $p<300$. However, in removing features you will evidently be losing information, so you want to be able to learn what are the relevant features to your decision smooth/shaky. For example, when classifying cat/dog, the weight may be a highly informative feature thus its contribution in the decision should be highly considered, whereas the number of feet would not provide any useful information.

Some techniques to do this include PCA, LDA, and theres others as well. You can try them, its one line in scikit-learn.

Choosing an algorithm

Due to the heavy class imbalance, I would suggest that classification based algorithms will perform poorly. In contrast, I would attempt to use anomaly detection algorithms. These techniques will learn the distribution of your nominal set (smooth) and attempt to detect when signals differ significantly from this distribution and will then classify them as anomalous (shaky).

Many of these techniques exist, please refer to the following answers:

Using time series data from a sensor for ML

How to train model to predict events 30 minutes prior, from multi-dimensionnal timeseries


How to choose best model for Regression? I'm building a model to predict the flight delay. My dataset contains the following columns:

FL_DATE (contains months(1-12)), OP_CARRIER (One hot encoded data of Carrier names), ORIGIN(One hot encoded data of Origin Airport), Dest(one-hot encoded data of Dest Airport), CRS_DEP_TIME(Intended time of departure ex: 1015), DEP_TIME(Actual time of departure ex: 1017),DEP_DELAY(the difference between crs-dep ex: -2), ARR_DELAY(arrival delay ex: -2)

My target variable is ARR_DELAY. After checking my data, I have decided it is a regression problem. However, I'm not sure what method do I need to use for selecting the appropriate columns. On the other hand, I was plotting each column with ARR_DELAY to check their relation and got something like this: FL_TIME vs ARR_DELAY

In such scenarios, if I have to build a model for such data which regression technique should I use?

PS: I'm new to Machine Learning. Please correct me If I'm heading in the wrong direction

I think a better option will be that you try different regression techniques (it will be good for your learning curve) and see advantages and disadvantage.

For subset selection purposes few options are:

1- Best subset selection : in which you start with a null model and fill all possible combination of models and select best model on the basis of AIC, BIC or adjusted R2.

2-Forward selection: You start with a null model as well and add feature one by one to see which one gives lowest RSS or highest R2. Then add another feature again until you have the best model.

3-Backward selection: you start with a model with all p predictors remove one predictor evaluate adjusted R2 or AIC, BIC. do it again and again until you have a satisfactory model.

You don't have to do it manually both R and python have library that will help you in this.