What are available methods for modeling startup survival rates? I am interested in modeling startup companies failure and success rates to describe what is the representative startup.

I have 40 companies in a dataset. Each company is represented as a list of all the investment financing rounds it has gone through. For example:

Company 1: Seed Round, Series A Round
Company 2: Seed Round, Series A Round, Series B Round
Company 3: Seed Round, Series A Round, Series B Round
Company 4: Seed Round, Series A Round, Series B Round, Series C Round
Company 5: Seed Round
Company 6: Series A Round, Series B Round
Company 6: Series A Round

And so on.

I can see that each round can be represented as a state in a Markov Chain, and transitions are only allowed from earlier stages to later stages. I can go from Seed to Series A, and from Series A to Series B, but not from Series B to Series A.

So we have N-order markov chains (production data has N <= 4).

The output I'm looking for is a binary tree chart showing each stage as a node and each node can either transition to the next node or to a final state meaning the company has failed.

This problem can also be seen as a real options model...

Any ideas on how to implement this model?

I can code in Python or Ruby (but I am no expert).

If the data you show are the only data that you have, then the Markov Chain is really boring: it is a linear chain, going from Round A to Round B to Round C, and all those states are connected to a base state (which is Death, or something).

You can directly calculate the transition probabilities from the data you have, since the number of companies that reached round N are all companies that could have reached round N (there is no alternative path).  The death probability at the previous stage is (1 - $P_{reaching N}$)

In [1]: raw_data = """
   ...: Company 1: Seed Round, Series A Round
   ...: Company 2: Seed Round, Series A Round, Series B Round
   ...: Company 3: Seed Round, Series A Round, Series B Round
   ...: Company 4: Seed Round, Series A Round, Series B Round, Series C Round
   ...: Company 5: Seed Round
   ...: Company 6: Series A Round, Series B Round
   ...: Company 6: Series A Round
   ...: """
In [2]: data_lines = raw_data.splitlines()[1:]
In [6]: key_vals = {}
In [12]: for line in data_lines:
             key, val = line.split(':')
             key = key.strip()
             vals = [v.strip() for v in val.split(',')]
             key_vals[key] = vals

In [13]: key_vals
Out[13]:
{'Company 1': ['Seed Round', 'Series A Round'],
 'Company 2': ['Seed Round', 'Series A Round', 'Series B Round'],
 'Company 3': ['Seed Round', 'Series A Round', 'Series B Round'],
 'Company 4': ['Seed Round',
  'Series A Round',
  'Series B Round',
  'Series C Round'],
 'Company 5': ['Seed Round'],
 'Company 6': ['Series A Round']}

In [14]: transitions = ['Seed Round', 'Series A Round', 'Series B Round', 'Series C Round']

In [19]: for transition in transitions:
    summed = 0
    for company, rounds in key_vals.iteritems():
        if transition in rounds:
            summed += 1
    prob = float(summed) / float(len(key_vals.keys()))
    death_prob = 1 - prob
    print "From previous to %s: probability %s" % (transition, prob)
    print "Death rate at %s: probability %s" % (transition, death_prob)

From previous to Seed Round: probability 0.833333333333
Death rate at Seed Round: probability 0.166666666667
From previous to Series A Round: probability 0.833333333333
Death rate at Series A Round: probability 0.166666666667
From previous to Series B Round: probability 0.5
Death rate at Series B Round: probability 0.5
From previous to Series C Round: probability 0.166666666667
Death rate at Series C Round: probability 0.833333333333

However, if you had some more features of each company, like the amount of money they received at each stage, or the profit they were making, then you could train a decision tree, e.g. with this implementation in sklearn, that told you, in simple words, "if a company arrived at round X with at least Y dollars raised and at least Z dollars of profit, then they are passing to the next round with 0.XX probability".  Which is, I think, what you are aiming at.


Putting dirty data back into a model Suppose I am processing 3rd party vendor error log files that I am unable to change the schema from the source. And I am trying to predict a label.

Once logs are collected and I have all I need, in varying log formats, with each missing random information compared to the others.

-------------------------------------------------------
| vendor | variable_one | variable_two |variable_three|
-------------------------------------------------------
|    1   |       -      |"some string" |     NULL     |
-------------------------------------------------------
|    3   |      GET     |"some string" |     1270     |
-------------------------------------------------------
|  ...   |     POST     |    NULL      |      760     |
-------------------------------------------------------

Even after all the preprocessing and data transformations to change labels to Binary, and Categorical values and such. After cleaning, feature selection and training. Having a model trained, evaluated, and tested. (label not depicted)

-------------------------------------------------------
| vendor | variable_one | variable_two |variable_three|
-------------------------------------------------------
|    1   |      GET     |"some string" |        0     |
-------------------------------------------------------
|    3   |      GET     |"some string" |     1270     |
-------------------------------------------------------
|  ...   |     POST     |     ""       |      760     |
-------------------------------------------------------

The questions are:

When I go back to send my vendors logs to through the model, I am still sending it through with, potentially, missing values from what was altered, or selected as a feature for the model.

Is the model then useless?
Do the transformations made in data cleansing need to be applied to each log entry that comes through, somewhere upstream?
Assuming the model only knows "nice pretty" cleaned data, and I send log entries with missing values. Is the model making assumptions or less accurate prediction?

This question has no general answer; it depends on the specifics of both the statistical method in question and the implementation of that method in whatever language you used.

For example if you used a logistic GLM, then a missing value is an error since data records must be complete. Presumably you would have included a default value, and you will have to manually apply that if you want those records to be scored.

Alternatively, if you were using gradient boosted trees (gbm), then they can cope with missing values, but do so in an implementation-defined way. The answers would be different in the gbm and xgboost R packages, for example.


Neural networks with not-fixed dimension for input and output I would like to know if it exists a model/method which can deal with input and output of different dimension.
For example, let us say that the maximum number of info we could have is 6 features and 5 output.
Then I could have examples with 4 features and 3 output.
Less input features always relates to less output. And relations stays the same.
with only 4 features I have only 4 outputs, and so on.
Most important, it is not that I do not have them for missing knowledge, but because in the same problem dominion I could have all 6 of the features, or less.
It is possibile to create a model which deal with this kind of things ?
The other solution I thought was to just use a simple deep network, with the maximum number of features and output as dimension, and use a value = 0 when I have a missing feature or a missing target. But that destroyed completely the training performances

If youre searching for neural network architecture that have varying number of inputs and outputs, Recurrent Neural Networks, LSTM's .. etc are examples. They are used in Natural Language Processing where the main goal is to examine patterns in sentences.  But I highly doubt that they will work for your use case since no information about it is provided.
Another way would be to create multiple neural networks with a different input/ output sizes, such that the input/output sizes are averages of the groups of similar input/output sizes.


Confusion regarding epoch and accuracy I have been learning keras and TensorFlow for some weeks now, and get confused with epoch.

I trained my network for 50 epochs, the test data and training data are randomly split (80% train, 20% test). The training data's accuracy grows nicely, but the test data's accuracy goes up and down. I am not sure why it happens like that.

Say in epoch 10, the test data's accuracy is 92%, in next epoch, how can accuracy drop? The thing I can think of is that maybe for each epoch, the test data and training data are randomly selected again, so the system sees new data which doesn't fit previous parameters?

Is it the case?

Variation in test data accuracy is perfectly normal. Your network is fitting for your train data. If you see (big) variation in train accuracy, then you should be worried about learning rate tuning, regularization etc.
But here, everything seems to be going fine. Remember your network only fits on your train data, so it modifies its weights according to that data. Since your test data -which the network never uses for weight update- is not exactly your train data (even though it should come from the same distribution ... so test accuracy should globally improve over iterations if the network is learning), test accuracy is bound not to improve exactly at each iteration.


Comparison of performance of autoencoder with PCA I am running PCA and autoencoder (2 hidden layer with relu) on a data. Both PCA and autoencoder give similar accuracy of the order 50%. I have tried different variations of autoencoder: changing layers, neurons,activations. But it did not perform any better. What can be the possible reason behind this. Can anybody suggest any analysis of data to identify the reason for autoencoder to not perform any better.

The features in your data are probably linear as an Autoencoder should give the same results then (see here and here). The surplus of an AE is that it is efficient to detect nonlinear features which a PCA cannot.


Is it necessary to take log transformation on the data values to get the minimum mean squared error? I have tried the house price prediction problem using simple linear regression by using the square feet as independent variable and price as dependent variable.while trying to check the MSE of the model,I get the output as 61940787124.62475. But after taking the log transformation to both the variables,the model shows the MSE value as 0.14824951630518876. which method is correct?And since the first model gives the larger MSE value,does it means the first model is error prone?

If you applied the log transformation to all your labels as well as the inputs, then it is normal that the MSE decreases. It is the case because when you apply log to big numbers they all become smaller and less spread. The standard deviation of the log price is a lot smaller than the one of the price. The MSE of any model trying to predict the log price (for instance the model that predicts the mean log price every time) will be smaller than the MSE of a model trying to predict the price.
Using the log square footage to predict the price could be useful if you are using a linear regression for instance and if the relation between square footage and price were logarithmic. It is probably not the case but you can try it.
However changing the target (applying log to it) changes the loss (you are computing the MSE between log price and prediction) and thus changes the problem you are trying to solve. You cannot compare the performances of the two models directly using the outputed MSE.


how to extract high frequency phrase This is the corpus

abcdef
abcd
xabcd
b
bcef
ef
aef

what I want to extract is abcd, ef, then I try to count all n-grams:

abcdef 1
abcd 3
abc 3
...

My question is how to distinguish abcd with abcdef and abc, since abcdef just appear once, and abc is the sub-string of 'abcd'. Of course, I can filter with this two conditions, but just want to know any existing algorithm target on solving this problem.

This uses python and NLTK

import nltk

text = """abcdef
abcd
xabcd
b
bcef
ef
aef
"""

def tokens(lines):
    for line in lines:
        for n in range(1, len(line)):
            ts = nltk.ngrams(line, n)
            for t in ts: yield str.join('', t)
        yield line

ts = tokens(text.splitlines())
freq = nltk.FreqDist(ts)
print(freq.most_common())
# [('b', 5), ('c', 4), ('ef', 4), ('bc', 4), ('e', 4)...

There are other solutions, of course, but I think this meets your criteria. You may need to optimize if your lines are huge as this will create every possible n-gram.


How to feed training set labels into Keras LSTM I’m implementing an LSTM with Keras and I know that I have to reshape the training dataset in a 3D object. Basically I have a dataset of shape (300000, 839)and I reshape it as (100000, 3, 839) because I want a timestep of 3. The question is: how could I treat the training set labels? Have I to reshape also them? if I reshape the labels to 100000, I don't truncate 200000 labels, since the starting number of labels is 300000?

Thanks in advance.

Your approch so far is not exactly right. Assuming you want to use the many to one approcah, the way you do it is as follows:

You have a dataset like this:

And you prepare your data in a way like this:

Source

For a timestep of 3, you discard the first 2 occurrences of your labels and use $y_{3}$ for the inputs $\{x_1, x_2,x_3\}$.

Next you don't start in $x_4$. Instead you use the next three-item slice $\{x_2, x_3, x_4\}$ with the training label $y_4$.

You continue this way until the last imput $\{x_{n-2}, x_{n-1}, x_{n}\}$ to predict $y_n$. If you have $n=30000$ samples, you can generate up to $n-2$ (29998) training samples for your model following this logic.


requesting password while git push in jupyter notebook I am trying to push the commited changes from kaggle kernels using jupyter notebook but while pushing the changes git asking me for username and I really don't know how to give username as it seems there is no textbox or anything where I can type my username.

Use one of the following command:

git push https://username:password@myrepository.biz/file.git --all

git push https://username:password@myrepository.biz/file.git master

Found in:
https://stackoverflow.com/a/29776651/10929995


Linear Model to generate probability of each possible output This is related to Sports prediction (Cricket). I am new to Machine Learning and learning it through TensorFlow.

I am focusing only on one topic which is "How many runs Player A will score in today's match?"

My raw data includes all cricket player's score in all matches. It also includes more finer details like the ground the match happened, day/night match, opposition team etc., I need to use this data and predict four possible ranges for a player's score with all 4 range having equal probability.

I am planning to keep 4 simple FeatureColumn to start with:

Average of runs of the given player in the past 10 matches (relative to the match that is being trained)
Batting order (1 to 10) (in the match being trained)
Average of runs since debut (relative to the match that is being trained)
Average no. of balls faced in the past 10 matches (relative to the match that is being trained)

My actual output will be the runs scored by that player in the match being trained.

I can consider 100 recent matches of a player for this training.

This tutorial https://www.tensorflow.org/tutorials/wide from TensorFlow seems to match with what I am approaching. But the final output is a binary value (whether salary is >50K or not). But I need to generate output with probability of a given player getting a run. For eg.

Run     | Probability of getting this run
-----------------------------------------
0 runs  :  0.01
1 run   :  0.01
2 runs  :  0.02
....
13 runs :  0.5
14 runs :  0.04
....
56 runs :  0.08
57 runs :  0.03
....

Sum of Probability should add up to 1.
I'll then group all these values into 4 ranges with each range having an equal probability (0.25 for each range) like this:

Which approach can I use for solving this problem? I read about Linear regression and that seems to be the best fit for me, but I am not clear on how to generate probability of all possible outcomes.

From what I understood -

Train a classifier with the features as you mentioned and the output probabilities like you have shown -

So, for Player A, you can use all the features (average runs scored overall, average score in past 10 innings etc.)
Treat the runs scored as a categorical variable and try to learn a classifier
Put it through the network with a softmax layer at the end - that will give you Sum of Probability should add up to 1

My recommendation

Each player will have scored different runs so you will not get enough data points in a particular category (so no. of people who score 83 will be really less for example)
Pre-group the ranges in your training data and then do the classifier.
Having a softmax layer at then end won't give you the "linear model" you want, but would definitely solve the problem with a lot more clarity.


Decision Tree on Unbalanced Data While running decision tree, I have unbalanced data. The data balance is 93%(Class 0) to 7% (Class 1).
Now when I am plotting decision tree to understand the factors contributing to class 1, then I find that most boxes are for 0 class (as it is 97% of the data).
Also, after pruning, very few contributors are there which are contributing to class 1. How can I get the factors contributing to class 1?

Which function are you using at Loss? Using the right one is important when dealing with imbalanced datasets. 7% is imbalanced, but not that bad.

Have you tried any eXplainable Artificial Intelligence (XAI) method? Normally I use Shap.
It is really good to see which feature contributes in which direction. You can see an example here.

You can check the github here
You can check the paper here


Language modelling for Spell Checker I am working on spell checkers, I want to create a spell checker, I am confused about which model to use

Word-Level modelling
Character-Level modelling

plus I am preferring neural networks over Peter Norvig or N-gram/K-gram or any other "Vanilla" algorithm so that networks capture context

Example:
Input: I have aplied fo r credit cart
Output: I have applied for credit card

I have gone through models which use character level BLSTM with noise makers/error models and attention mechanism but not satisfied by result. https://towardsdatascience.com/creating-a-spell-checker-with-tensorflow-d35b23939f60

One of the article I have gone through is to use OpenNMT, but results were disastrous. https://medium.com/scribd-data-science-engineering/neural-spelling-corrections-and-the-importance-of-accuracy-977c0063d20f

A simple way to start with word-level:

Given a phrase credit cart, you'd like to correct this to credit card. Take all the correctly spelled words (assume for now that anything found in your dictionary is correctly spelled even if grammatically incorrect) and embed them into vectors using your favorite embedding, or better yet, fine-tuning that embedding on a corpus related to yours.

Next, for all the misspelled words, generate a small number of candidates based on edit distance. Then choose the winner from this set based on the distance from the mean of the embeddings from the correctly spelled set.

This certainly isn't optimal, but would be a framework to get started.

A way to start with word-level (sequence modeling):

Get your favorite word-level sequence model, fine tune it on corpus of your choosing.

Then, you process the document to be checked one word at time, and if word is misspelled, predict the next word based on the context before it. You should peel back the last layer to see the probabilities, then select the top N of those words most likely.

Then score the predicted next words based on edit distance to the actual next word, and choose that.

Final notes

For both these schemes you'd certainly have to do some engineering work and may need to weight by relevance or likelihood of the word, and so on.

Finally, I suspect the truly optimal way to do this is just to have a giant corpus of spelling mistakes & subsequent corrections (something a MS or a Google would have from their document software usage databases). With that much data, you might just be able to look up the most common misspelling and replace it, no ML needed. Or at very least narrow the candidate set of "correct" replacements for another simple scoring technique.

Otherwise, you could try with a character-level model to do something similar to my second example and move forward one character at a time, and for those misspelled words, generating a distribution of probabilities per character. Based on this you could cleverly try to figure out the best "path" that moves through your likely character distributions and again score the resultant candidates with some type of embedding distance to the words in the vicinity. Thus getting you the most likely character combination to replace the misspelled word with. It's not a great solution since you're not taking into account the words after, but perhaps with a bidirectional model you might be able to do this.

Anyway, hope this long ramble was somewhat helpful! Remember, a lot of deep learning for real use cases is just engineering!


Creating dummy variables to match fitted model at inference I have built a machine learning classifier using Sklearn and pandas as my main tools. Now, one of the input features to the model is country (to letter country code such as US). I have fit a model using the pd.get_dummies function.

Now I want to run inference on the data but a few of the countries haven't appeared in my dataset over the past months, so the pd.get_dummies function is misaligned with the fitted model.

How can this dealt with?

I think the answer to this question will solve your problem.

import pandas as pd

train = pd.DataFrame(data = [['a', 123, 'ab'], ['b', 234, 'bc']],
                     columns=['col1', 'col2', 'col3'])
test = pd.DataFrame(data = [['c', 345, 'ab'], ['b', 456, 'ab']],
                     columns=['col1', 'col2', 'col3'])

train_objs_num = len(train)
dataset = pd.concat(objs=[train, test], axis=0)
dataset_preprocessed = pd.get_dummies(dataset)
train_preprocessed = dataset_preprocessed[:train_objs_num]
test_preprocessed = dataset_preprocessed[train_objs_num:]

If that doesn't help, a simple solution would be to add columns of all zeros for each country that the new dataset it missing. This will give you the correct shape of data that your classifier is expecting.


how to limit datetime in x axis Hello folks,
            I am trying to plot three variables in a single XY plot. I am having a problem setting the x-axis limit which is for time. I tried setting limits using x_limit but getting error. I will provide the code I am using and also the fig I got. If anyone can help me in this regard it will be much appreciated.

   import matplotlib.pyplot as plt
   from   netCDF4 import *
   import netCDF4
   import numpy as np
   from datetime import datetime
   import pandas as pd
   import matplotlib.dates as mdates
   myFmt = mdates.DateFormatter('%b %d')
   df = pd.read_excel('SUCS_1982_2020.xlsx',sheet_name='TEST')
   time = df.Date
   temp = df.air_tmp
   prs = df.Pressure
   fig = plt.figure(figsize=(8,4))
   ax = fig.add_subplot(111)
   ax.set_facecolor("#f1f1f1")
   #ax = plt.axes([1, 1, 0.7, 0.7])
   lns1 =ax.plot(time, sst,'-' "r",label='SST')
   lns2 =ax.plot(time, temp,'-' "g",label='AIR')
   ax2 = ax.twinx()
   lns3 =ax2.plot(time, prs,'-' "b",label='PRESSURE')
   ax.set_ylim(26,29)
   ax2.set_ylim(920,1010)
   legend=fig.legend(loc ="upper right",bbox_to_anchor=
                       (.21,.8),bbox_transform=ax.transAxes,frameon=False)
   legend.get_frame().set_facecolor('none')
   ##########################################
   import matplotlib.dates as mdates
   myFmt = mdates.DateFormatter('%b %d')
   ax.xaxis.set_major_locator(mdates.DayLocator(interval=1))
   ax.xaxis.set_major_formatter(myFmt)
   #plt.grid(True)
   ax.grid()
   ax.set_xlabel("DAY")
   ax.set_ylabel(r"SST/AIR TEMPERATURE($^\circ$C)")
   ax2.set_ylabel("PRESSURE(hpa)")
   #ax.set_xlim(0,37)

I've had a similar problem to putting in x limits to date time data. To combat this, I normally extract the subset of the data which fall in the limits you want to review in further detail.
To do this, you can do something like this:
df_subset = df[(df["Date"] < end_date) & (df["Date"] > start_end)]

And then plot this subset.


How to check if the dataset contains sufficient information to predict a continuous variable Is there a way to check if a dataset generally contains enough information to predict a target variable?  In other words, can you calculate something like a correlation-coefficient between all features X and the target y, instead of training models?

Just to get an estimate of how good you could probably get, or if there is a constellation of features in X that correlate with y?

You are describing something that you can create a series of scatterplots for. Hold your dependent variable constant and then just run scatterplots against all possible features and see what comes out. You can visually determine what makes sense (or at least what makes sense as a starting point) and then go from there.

Also, don't forget that this is where you will also need to apply some of your own knowledge around transformations. Just because a scatterplot isn't obvious doesn't mean that it can't be used


What do you call a feature that always has the same value? Is there a standard term for a feature that always has the same value, i.e. that can be discarded without loss of information?

For example I am trying to classify cats vs dogs, and every example in my training set has has_two_eyes=true.

I am thinking something like "useless," "redundant," "constant," or "degenerate," but I don't know what the standard terminology is here.

I think there are a few terms, but the one I have seen most often is "Zero Variance Predictor" or "Zero Variance Feature"


how to identify the unknown class in machine learning? In the problem of multi-label classification how to identify the unknown class which is not in training labels or classes. In the prediction phase, the classifier puts the data in any of the class even though it's not related to any of the classes or labels?
For binary, the prediction probability of classification is 50/50 so how to interpret these as the confusion or just anomaly but when the data appears in both of the class or the similar to both of the class we can understand this scenario but what about the data which should be in any of the classes.
So, how to handle the same situation for multi-class or a general solution which can provide the proper classification for unknown and non-relevant data.

I have come across a very similar problem lately. It is a case of novelty/anomaly detection as you have mentioned.

There are broadly 2 "working" solutions to these type of problems:

You can check the softmax-posterior probability value of the output layer for all classes and threshold it(say 0.5). If the probability is < 0.5 for all other classes, then definitely this test case is an outlier(our unknown class in this case).
Adding a Garbage class and training it. But, collecting/randomly generating the features for this unknown class may sometimes become tedious and difficult. It is dependent on the application and on the number of classes. If you have smaller number of classes(<10), then you could try this too.

Best would be to try both and see which one best fits the purpose of your application and which increases your prediction accuracy positively.


Incremental modelling of kmeans in pyspark I have a large dataset and trained the model with kmeans for the first time. I  saved the model and pipeline used . Now again I started collecting data.  After sufficient data is collected using old model and pipeline I want to retrain the model on new collected data in Pyspark

Is it possible to use transfer learning in unsupervised learning algorithms (or clustering) like Kmeans in Pyspark? If so, how it can be done?

There's dedicated lib for incremental / online learning. It's called Creme. You can find it here. It has kmeans implemented, though I don't know how well it's suited for spark data sets.


best way to regularize gradient boosting regressor? i am testing gradient boosting regressor from sklearn for time series prediction on noisy data (currency markets).

https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html

surprisingly, the the gradient boosting regressor achieves very high accuracy on the training data - surprising because the data is so noisy. however, it performs poorly on the test set.

this is clearly a case of overfitting, so i'm wondering what parameters i can change to regularize the gradient boosting regressor.

so far i've tried max_depth, reducing it to 1 (from the default of 3). this seems to work pretty well in increasing accuracy on the validation set.

does anyone know what other parameters i could tweak, to improve performance on the validation/test set? thanks

The hyper parameters that you could tune in any boosting technique are:

Depth of each tree: As you rightly pointed out this is very important because each tree in boosting technique learns from the errors of the previous trees. Hence underfitting the initial trees ensure that the later trees learn actual patterns and not noise.
Number of trees: this is kind of intuitive from previous point as the number of trees increase the learnable signal decreases and hence the ideal number of trees is more than underfitting trees and less than overfitted trees.
Learning rate: this parameter gives weights to previous trees according to a value between 0 and 1. Lower learning rates give lesser importance to previous trees. Higher weights lead to faster steps towards optimization.  Lower weights typically lead to global optimum. But lower learning rates need more trees to learn the function.

4.Sub sample: if the value is less than 1 a subset of variables is used to build the tree making it robust and learn signal from more variables. This variable reduces overfitting by not fitting only 1 variable but a group of variables.

These  variables if tuned correctly are sufficient to reduce overfitting.


Is there a safe and simple way to estimate a standard deviation for a next subset? In case I receive only standard deviation from a sensor of a value $v$ (that is btw normally distributed) each 4th minute but need to provide a standard deviation $\sigma$ for each 15 minutes is there a safe way to do it.

There are two things that came into my mind:

1) One and safe way is to get the mean, generate possible values using standard deviation of the 4 minute interval for the 15 minutes period (15*60 values). Calculate the $\sigma$ for this period

2) Alternatively one can naively estimate the value of $\sigma$ of the next time interval based on two previous values. For example, use  and  standard deviations to estimate

In case the standard deviation is increasing/descreasing in previous cases  and   it will increase/descreasing in the next time interval on absolute value  -

The first method can be time-consuming/computationally-consuming comparing to the second method. Though the second method may suffer on precision.

Edit 16.04:  Since I'm limited in the amount of data i preferably would use  only the last standard deviation and no mean data

Edit 23.04: There is one more way that bring me to the result very close to 1st way of problem solving.

Let say $\sigma_i$ is based on $n$ observations while $\sigma_{i+1}$ is based on $k$ observations and $k > n$. Then
$\sigma^2_{i+1} = \frac{(n-1) * \sigma^2_i * \frac{k}{n}}{k-1} $

The benefit in this case that you are not dealing with a mean value. I suppose that this solution works well only with normally distributed values.

Basically, the two methods you are proposing are the same.

The first one is computationally more consuming, but they are the same.

In the first method you are calculating $\sigma$ generating possible values of random variable with already the same $\sigma$ you have had historically. This is the same as calculating $\sigma$ with all the historical data you have.

In the second method you are doing a stimation with limited data, this is the correct way unless you have sufficient amount of data to estimate a GARCH model.

A GARCH model is a statistical model for time series data that describes the variance of the current error term or innovation as a function of the actual sizes of the previous time periods' error terms.

Meaning:
$\sigma_t^2 = w+\alpha_1\epsilon_{t-1}^2+...+\alpha_q\epsilon_{t-q}^2+\beta_1\sigma_{t-1}^2+...+\beta_p\epsilon_{t-p}^2$

This model requires a sufficient amount of data and knowledge on time series analysis, of the two options you posted, I would use the second with as much data as possible.


What does the notation mAP@[.5:.95] mean? For detection, a common way to determine if one object proposal was right is Intersection over Union (IoU, IU). This takes the set $A$ of proposed object pixels and the set of true object pixels $B$ and calculates:

$$IoU(A, B) = \frac{A \cap B}{A \cup B}$$

Commonly, IoU > 0.5 means that it was a hit, otherwise it was a fail. For each class, one can calculate the

True Positive ($TP(c)$): a proposal was made for class $c$ and there actually was an object of class $c$
False Positive ($FP(c)$): a proposal was made for class $c$, but there is no object of class $c$
Average Precision for class $c$: $\frac{\#TP(c)}{\#TP(c) + \#FP(c)}$

The mAP (mean average precision) = $\frac{1}{|classes|}\sum_{c \in classes} \frac{\#TP(c)}{\#TP(c) + \#FP(c)}$

If one wants better proposals, one does increase the IoU from 0.5 to a higher value (up to 1.0 which would be perfect). One can denote this with mAP@p, where $p \in (0, 1)$ is the IoU.

But what does mAP@[.5:.95] (as found in this paper) mean?

mAP@[.5:.95](someone denoted mAP@[.5,.95]) means average mAP over different IoU thresholds, from 0.5 to 0.95, step 0.05 (0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95).

There is
an associated MS COCO challenge with a new evaluation
metric, that averages mAP over different IoU thresholds,
from 0.5 to 0.95 (written as “0.5:0.95”). [Ref]

We evaluate the
mAP averaged for IoU ∈ [0.5 : 0.05 : 0.95] (COCO’s
standard metric, simply denoted as mAP@[.5, .95])
and mAP@0.5 (PASCAL VOC’s metric). [Ref]

To evaluate our final detections, we use the official
COCO API [20], which measures mAP averaged over IOU
thresholds in [0.5 : 0.05 : 0.95], amongst other metrics. [Ref]

BTW, the source code of coco shows exactly what mAP@[.5:.95] is doing:
self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True)
References

cocoapi

Inside-Outside Net: Detecting Objects in Context with Skip Pooling and
Recurrent Neural Networks

Faster R-CNN: Towards Real-Time Object
Detection with Region Proposal Networks

Speed/accuracy trade-offs for modern convolutional object detectors


customer segmentation with categorical variables I was adviced to write in this group regarding my question about modeling categorical database.
I have a customer dataset, which is a survey result. I have 1595 obs. and about 200 columns(200 because most of the cases the questions were multiple choice and we had to split it into columns). Majority of variables are categorical or binary. I do not have continous variables at all. My task is to do customer segmentation, clustering. There is no initial assumptions although as I have also the questionnaire so can logically seperate the important questions.

I face several issues regarding the modeling

I need to validate the choice of variables i use
I am trying to find associations, pairwise associations and trends, as I do not have initial assumtions who can be my segments
Clustering models are not working good for categorical variables and the ones I tried for example kmods, ignore the associations, correlations and return me not clear picture.

Can you please suggest how to approach, or from where to start. I am new in data analytics and I need some hints to go on with the analysis and I will be grateful to have some guidance at least high level what can be done.

Thanks in advance!

You can use mixture model with variable selection. In this framework, the challenge of variable selection consists in model selection. Thus, detection of the relevant features and estimation of the number of clusters can be done according to information criteria (like BIC or ICL).

To perform this analysis, you can use the R package VarSelLCM. Because you consider categorical variables, your dataset must be a data.frame and each column must be a factor. Here is an example of the script. Your dataset is denoted by "my.data".

## Clustering by considering all the variables as discriminative
# Number of clusters is between 1 and 6
res.all <- VarSelCluster(my.data, 1:6, vbleSelec = FALSE)

# partition
res.all@partitions@zMAP

# shiny application
VarSelShiny(res.all)

## Clustering with variable selection
# Number of clusters is between 1 and 6
res.selec <- VarSelCluster(my.data, 1:6, vbleSelec = TRUE)

# partition
res.selec@partitions@zMAP

# shiny application
VarSelShiny(res.selec)


How to understand the performance of different machine learning models? I have a dataset, which contains the processing conditions (i.e., 42 features) and the property (i.e., 1 target) of a class of material. To know the performance of different machine learning models, I tested five different machine learning models by considering different numbers of features in training. These models are linear regression (LR), Bayesian ridge (BR) , k-nearest neighbor (NN), random forest (RF), and support vector machines (SVM) regression. The coefficient of determination (R2) for the test dataset is used to represent the performance of trained machine learning models.

As we can see that the max. accuracy of these models are in order: RF>BR~LR~SVM>NN. The top 8 features are required to obtain good accuracy for RF and afterward the accuracy is almost independent of the number of top-ranking features. The performance of BR, LR and SVM continuously improve with an increasing number of top-ranking features until reaching the maximum with top 26 features. NN exhibits different trends from others. It already reaches the best performance with top 8-12 features but gets worse and worse with more features.
I am wondering what are the possible reasons for this results? Some directions or some hints? for example:

why the accuracy is in the order of RF>BR~LR~SVM>NN.
why RF model reaches a good accuracy with a few features then keeps almost constant with more features?
why linear-based models BR and LR have very similar performance as the SVM models.
why NN model reaches a good accuracy with a few features then the accuracy reduces with increasing numbers of features?

I understand the reason is from case to case, but what is the general explanation or directions for finding the answers?

First, it would be beneficial if you could mention whether that $R2$ presented in the figure is for the test or training dataset.
Let's assume that it is for the test dataset.
The answer to all of your questions is subjective and depends on the type of data you used for this purpose. But let me briefly answer them.
Because the Random Forest model (RF) is an ensembling method (a combination of weak learners), it is expected to have a very good performance. This method is robust to overfitting. We can see that when the number of features increases, there is no drop in the performance of the RF model; while using a large number of features could lead to a drop in the performance of the model on the test dataset. RF model is not prone to overfitting because of using many weak learners (decision trees here).
Linear Regression and SVM probably perform the same because you might use a linear kernel for SVM. If that is not the case, it means that transforming the data to a new space (SVM transforms the data into new space to easily separate the classes) is not useful in your dataset because of the nature of your data (it means that your data might be linearly separable).
It seems that you did not use regularization for linear regression. If that is true, your model is overfitted especially when you have a large number of features. If you add regularization to your model, the performance of the linear regression model does not decrease as the number of features increases.
Naive Bayes is good when features are independent and when dependencies of features from each other are similar between features. So, if they are not true for your dataset that might be the reason that Naive Bayes does not work well.
Also, Naive Bayes classifiers can quickly learn to use high dimensional features with limited training data because of the independence assumption. So, it is expected that Naive Bayes works better when the number of features >> sample size compared to more sophisticated ML algorithms. But we can see that is not the case for your study. Probably, it is because of the violation of the feature's independence.


Questions When Advancing from Vanilla Neural Network to Recurrent Neural Network I've recently learned how a vanilla neural network would work, with given number of inputs, hidden nodes, and the same number of outputs as inputs.

I've been looking at various posts now related to recurrent neural network, and I understand the concept behind it, but I fail to understand certain parts of the logic behind the RNN.

Here are four main questions I have:

How does back-propagation work in recurrent neural network?
Are the weights that lead from the input to the hidden nodes the same for every other step? What about the weights from the hidden nodes to the outputs?
How do biases exactly work in a recurrent neural network?
Why is tanh function usually used instead of sigmoid function as the activtion function?

I realize some of these questions are very basic, but I guess the basics are exactly what I need right now.

Even links to related videos or posts will be extremely helpful, and so will google keywords that show the right results. These four questions are hindering me from understanding sample Python codes, so I really need some help.

This question gets at some very important qualities of RNNs and DNNs in general. I'll answer each of your sub-questions, though not in the same order (I'll try to highlight where I do)

Parameter Sharing

First, the most important quality of RNNs is called parameter sharing. Sequential data is typically input into separate layers. For a length 20 input, an RNN network would have 20 layers. The same internal parameters are used for each layer, so all 20 layers use the same weights $W$ and bias $b$. Compare this to a multilayer perceptron which would have 20 separate weights and biases.

Parameter sharing has several benefits:

We now have many fewer parameters. There is 1 repeated block instead of 20 separate layers. A 20x reduction!
This effectively multiplies the training data. The recurrent layer gets to learn from every word in a single example sentence whereas each layer in an MLP learns from a single word per sentence.
Our network is now much more flexible. We can train on sentences up to 20 words and then generalize to a 25 word sentence by adding more steps or by using a dynamic RNN

Network architecture

You ask about tanh and sigmoid activations. To answer this we have to talk about specific RNN architectures. The simple RNN discussed above has a single activation. Simple RNNs tend to create the problem of vanishing gradients (or exploding!) due to the repeated application of the same weights and activation function.

Gated RNN blocks (like GRUs and LSTMs) use gating mechanisms to pass activations into and out of memory states and to combine memory states with input to generate the output of each block. Because of this, gates can stop the gradient from propagating backward. Sigmoid is a common activation function for gates because it squashes activations to (0,1)---0 completely stops the activation and 1 lets it pass through. Any decent activation function with a similar squashing profile works, though. Anecdotally, hard sigmoid is quite common these days.

In addition to the gates, gated RNN blocks have an internal state for which the activation varies quite a bit. Because gating limits gradient backprop, we have a lot of flexibility on this activation. It need not be squashing for instance, and this is where the rectifying activations (relu, elu, islu, etc.) are often seen. Tanh is a perfectly sensible choice as well.

Regarding biases and weights, each activation in an RNN cell typically has its own weight and bias. So a GRU has 3 activations (hidden, update, and reset) and each has its own weight and bias. Though, recall that as an RNN, each of these are re-used for every timestep.

Backward Pass

That covers the forward pass pretty well but you also ask an important question about how error propagates backward. There are two methods for approaching this.

Teacher Forcing

For RNNs that output a prediction at each time step (such as predicting the outcome of following steps, translation, or phoneme recognition), teacher forcing is a method for isolating each step of the RNN. By removing these dependencies, Teacher Forcing allows RNN to use conventional backprop with the chain rule.

But how does it work? Teacher forcing networks have separate train and test architectures. For training, at each timestep $t$, the input $x_t$ is concatenated with the previous target, $y_{t-1}$. Imagine this for a network tasked with predicting the following character. The network has just tried to predict the character for the previous timestep. But we instead use the observed character at that timestep (we know this because we're in the training phase). Thus the error at timestep $t$ depends only on the observed value at $t-1$ and the input at $t$. We've thus removed any connections through time from the network.

At test, we don't know the true value at each timestep so we replace $y_{t-1}$ with the output of the previous layer $o_{t-1}$. In this case, temporal connections have returned but only for the test phase.

Back Propagation Through Time

But we don't have to resort to Teacher Forcing. Back propogation through time allows us to apply the backprop algorithm to RNNs. Consider the network with $n$ timesteps and an input $x$, hidden state $h$, output $o$, and observed value $y$ for each timestep.

BPTT works in the following steps.

Compute the gradient $\nabla_{o_t}$ for each $o_t, y_t$ pair. (This may be done all at once.)
Compute the gradient $\nabla_{h_t}$ for each timestep, beginning with the last timestep and iteratively working backward. (This must be done one at a time.)
This gives us $n$ edges for each internal parameter of our RNN. The trick to updating the parameters is finding the gradient contribution for each timestep (eg $\nabla_{W_t}$ even though we only have one $W$) and then summing those gradients to update the internal parameters.

Further reading

I highly recommend chapter 10 of Goodfellow, Bengio, and Courville's Deep Learning for more information on RNNs. Additionally Graves' RNN book is fantstic for higher-level detail


Checking linearity for a linear regression model? I've read that there are various assumptions associated with a multiple linear regression model which you should check/validate before getting too excited about your model results.
One of these is the assumption of linearity. I get that you would plot the dependent variable against the independent variable and visually check for linearity, but is there a more scientific way to do this?
I have the two plots below. Looking at the first, I can see some linearity by removing the outliers. The second however, is much harder. I can *maybe* see something, but I'm not sure if this is my eyes playing tricks on me.

If I determine the second plot does not satisfy linearity, what do I do? Exclude the feature from the model?

I like GAM (Generalised additive model) with regression splines:
# Load data
library(ISLR)
a = Auto

# Run GAM with splines
library(gam)
g = gam(mpg~s(displacement,5)+s(horsepower,5),data=a)
plot(g, se=T)

The result (plot) shows you that displacement can be well approximated by a linear function for lower values (< 250 or so). However, there is a "kink" at about 250, so that overall, a linear approximation would not be very good here.
See ISL, Chapter 7 for more details. There are also Labs for Python and R where you can see code details.
Also see this example in R with simulated data for more details.
Alternatively, look at a Q–Q plot after regression, e.g. in R:
l = lm(mpg~s(displacement,5)+s(horsepower,5),data=a)
plot(l)


What is coadaptation of neurons in neural networks? Looking for a bare minimum example (3 hidden units only maybe?) for what weights of a neural network with heavily coadapted weights would look like and showcase why they are bad. Also, how is coadaptation a sign of overfitting?

Co-adaptions in simple English term would mean co-operation. If you think nodes of a NN as workers it would mean missing even a few workers would result in failure of the NN to do something substantial. This happens mainly due to few nodes of NN outputting values which get cancelled by other nodes (so we can remove those nodes altogether -  essentially redundant units), thus resulting in inefficient use of the NN. Thus instead of say 2 nodes, 8 nodes are performing the same job with 6 nodes cancelling each other out. Here is the best illustration of the same.

In case of XOR approximation you need minimum (I'm not sure) 2 nodes to approximate the function correctly, but you can use 100 nodes and most of them will be redundant.

Also this will clearly cause over-fitting as the nodes instead of generalising on the training data are micro-managing the training data.

Now we do not want our NN's to behave in such a manner, we  want the other nodes to take over jobs of missing nodes (if any are missing) just like our brain takes over function when one centre (Say speech, hearing is damaged) and the state of art mechanism currently is dropout in layers.


How Google Trip Bundle works? Google Trip is now performing automated clustering of travel items into trip bundles.

https://support.google.com/inbox/answer/6228360?hl=en-GB

So, if in your mail you have a flight confirmation mail and a hotel confirmation mail related to the same trip, Google will cluster them automatically into a trip bundle.

This means that that:

They scan all your e mail
They find the ones related to trips
They cluster those for the same trip into a bundle.

My question regards point 3. Which kind of algorithm is needed to perform such task?

Consider that the task is unsupervised, because the user can not interact with the system while creating the bundles..

Company secrets - Google does not disclose what they actually use. Any answer to this is pure speculation.

It probably is not even just "one" algorithm, but many. But it may be as simple as finding overlapping dates and locations, assuming that people cannot be in two places at once. You could test this: book a flight, and a hotel on a different destination during the same time interval. I wouldn't be surprised if they nevertheless show up as one "trip" (and it doesn't really degrade user experience much).

It will very likely be a supervised approach. Which does not mean "interactive". On the contrary, unsupervised approaches will likely need interactions to choose parameters. Using some training data (e.g., from employees, or user interactions) to improve matching quality looks promising, though.


Convert exponential to normal distribution For the distribution shown below, I want to convert the exponential distribution to a normal distribution. I want to do this is as part of data pre-processing so that the classifier can better interpret the feature (named ipc here).

The regular log transformation does not work here because of the (x-axis) spread.

How can I transform this data to a normal distribution?

A related answer has been pointed out in the comment but I am looking for some Python code excerpt as well.

Thanks

You can use sklearn.preprocessing.QuantileTransformer (or sklearn.preprocessing.PowerTransformer) which does exactly what you want:

from sklearn.preprocessing import QuantileTransformer
import numpy as np

ey = np.random.exponential(size=100)
qt = QuantileTransformer(output_distribution='normal')
no = qt.fit_transform(ey.reshape(-1, 1))

You can plot histograms to compare "before" vs "after":

# Plot histograms to see before vs after.
import matplotlib.pyplot as plt
%matplotlib inline
plt.subplot(2, 2, 1)
plt.hist(ey, bins='auto')
plt.subplot(2, 2, 2)
plt.hist(no, bins='auto')
plt.show()

The advantage of this approach is that it will also work for other input distributions, not only exponential.


How evaluate text clustering? What metrics can be used for evaluating text clustering models? I used tf-idf + k-means, tf-idf + hierarchical clustering, doc2vec + k-means (metric is cosine similarity), doc2vec + hierarchical clustering (metric is cosine similarity).
How to decide which model is the best?

Check out this paper.  It also addresses question of how many clusters to use.  The R package mclust has a routine which will try different cluster models/number of clusters and plot the Bayesian inference criterion (BIC). (great vignette here).  It's a general method, meaning, something you can do without being domain/data specific.  (It's always good to be domain-specific if you have the time and data.)

The chart is from the vignette by Lucca Scrucca.  MClust tries 14 different clustering algorithms (represented by the different symbols), increasing the number of clusters from 1 to some default value.  It's finds the BIC each time.  Highest BIC is usually the best choice.  You could apply this methodology to your own stable of clustering algorithms.


How to efficiently iterate a supervised model over the Cartesian product of very large number of records? The problem:

Two large databases, with ~1M records each, "old customer data" and "new customer data". The data came from different sources and was ingested at different times, so there are many duplicates, but the duplicates might not be exact matches. For example In the old data, the customer was listed as "Michael Smith" but in the new data, they were listed as "Mike Smith" or "M. Smith", or the names match, but the address field is different: How do we now if it is different people with the same name or the same person who changed address?

Approach:

It seems possible to use a supervised learning approach to classify a pair of records as either being duplicates or not.

Question:

Assuming such a model is possible and we have trained it, how would we iterate over the entire data set to produce our predictions?

To apply a supervised approach (or any ML/Probabilistic approach), I can only think of naively examining each pair of records one-by-one, but this means our models would have to iterate through $10^{12}$ records, which doesn't seem feasible even with advanced compute capabilities?

How can one efficiently iterate/search through the combined data set in a situation like this?

This problem is called record linkage and there are methods to avoid iterating the whole cartesian product. The main method I know was called "blocking" and consists in doing a first "rough" pass to create groups of matching candidates (the "blocks"). For example you can create groups which contain at least X n-grams in common. This can be done through one linear iteration through all the entities, storing them in every applicable bin based on their n-grams (an entity can be stored in several bins). I assume that some kind of clustering could also be used to generate the groups of similar entities. Then you end up with multiple groups of smaller size, and you run the cartesian product comparison on every group individually. This can greatly reduce the complexity.

note: I was working on this maybe 10 years ago so there might be more recent approaches.


Loss function returns nan on time series dataset using tensorflow This was the follow up question of Prediction on timeseries data using tensorflow.
I have an input and output of below format.
(X) = [[ 0  1  2]
       [ 1  2  3]]

  y   = [ 3  4 ]

Its a timeseries data. The task is to predict the next number. Basically input was crafted by the below snippet
 def split_sequence(arr,timesteps):
     arr_len = len(arr)
     X,y = [],[]
     for i in range(arr_len):
       end_idx = i + timesteps
       if end_idx > arr_len-1:
          break
       input_component,output_component =  arr[i:end_idx],arr[end_idx]
       X.append(input_component)
       y.append(output_component)

     return np.array(X), np.array(y)

Now i want to train the model on the input and predict the next number. For instance x = [81,82,83] and the predicted output would be y = 84. In the previous problem, i had confronted the shape issue. Fortunately, i got a quick fill.
Now, when i am training the model,I observe my mse values are nan.
My implementation in keras is working but not in tensorflow. Most of the solutions in stackoverflow was pointing out to learning rate. Irrespective of giving different learning rate, my mse values are still nan. I have gone through many times reading my implementation but nothing turned up. Below is my tensorflow implementation.
 import tensorflow as tf
 import numpy as np

   # Create a dataset
  def split_sequence(arr,timesteps):
    arr_len = len(arr)
    X,y = [],[]
    for i in range(arr_len):
       end_idx = i + timesteps
       if end_idx > arr_len-1:
          break
       input_component,output_component = arr[i:end_idx],arr[end_idx]
       X.append(input_component)
       y.append(output_component)

    return np.array(X), np.array(y)

 # data generator
def generate_batch(X,y,batch_size):
    m = X.shape[0]
    indexes = range(m)
    n_batches = m // batch_size
    for batch_index in np.array_split(indexes,n_batches):
       yield X[batch_index],y[batch_index]

  # parameters
   n_inputs = 3
   n_epochs = 400
   batch_size = 500
   learning_rate = 0.0005
   n_steps = 3
   input, output = split_sequence(range(10000),n_steps)

   # input and output and theta variable
   X = tf.placeholder(tf.float32,shape=(None,n_inputs),name='X')
   y = tf.placeholder(tf.float32,shape=(None),name='y')
   theta = tf.Variable(tf.random_uniform([n_steps,1],-1.0,1.0),name='theta')

   # Predictions and the residual
   y_predictions = tf.matmul(X,theta,name='predictions')
   error = y_predictions - y

   mse = tf.reduce_mean(tf.square(error),name='mse')

   # Training the algorithm
   optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
    training_op = optimizer.minimize(mse)

   # initialize the variables
   init = tf.global_variables_initializer()

  with tf.Session() as session:
  # create  a session

     session.run(init)
     for epoch in range(n_epochs):
        for X_batch,y_batch in generate_batch(input,output,batch_size):
          #print(X_batch,y_batch)
          mse_val,_ =  session.run([mse,training_op],feed_dict={X:X_batch,y:y_batch})

          if epoch % 10 == 0:
             print('epoch',epoch,'MSE=',mse_val)

And the output i got was

epoch 0 MSE= nan
epoch 10 MSE= nan
epoch 20 MSE= nan

Any help is greatly appreciated. Thanks

This is because the the input values are not normalised to a standard scale. X consists of values ranging from 0 to 9999. This would make the training unstable. If the idea is to train the net to predict the sequence that you described, an easy fix is to feed it smaller values in the training data. For instance, feed it the input,

output = split_sequence(range(10),n_steps)

Along with a batch size of say, 5. It will learn to generalize to higher numbers as well.

With the changes made and the following for loop, the output is as expected.

for epoch in range(n_epochs):
    for X_batch, y_batch in generate_batch(input, output, batch_size):
        # print(X_batch,y_batch)
        mse_val, _ = session.run([mse, training_op], feed_dict={
                                 X: X_batch, y: y_batch})

    if epoch % 100 == 0:
        print('epoch', epoch, 'MSE=', mse_val)
        error_list.append(mse_val)

X_test = np.array([[45, 46, 47], [57, 58, 59], [1000, 1001, 1002]])
y_test = session.run(y_predictions, feed_dict={X:X_test})

print(y_test) # Should be 48, 60, 1003

plt.plot(error_list)
plt.show()

Final y_test is

[[  47.997036]
 [  59.996166]
 [1002.9279  ]]

If not this change, another change that you can make is to use the Adam Optimiser instead of SGD. It will make the learning more stable and it works even with bigger values.


Difference Between Feature Engineering and Feature Learning I am playing with features (input data) to improve my model's accuracy.

If I have a raw time-series dataframe, does feature engineering mean extracting properties or characteristics of my raw data and feed it as input? Or will the algorithm learn these from the time-series itself?

In other words, should I create a column that is comprised of the moving average, or will the algorithm pick up on moving average from the raw data?

Is feature engineering just the munging of independent variables? Or is it extracting features that are dependent on other raw data?

EDIT:

Here's another question: If I have a categorical feature, would it be better to have it as a one-hot vector (say, 5 binary inputs), or to have it as one input with range [0,4]?

How does one intuitively know the answer to these questions??

Feature engineering refers to creating new information that was not there previously, often by using domain specific knowledge or by creating new features that are transformations of others you already have, such as adding interaction terms or as you state, moving averages.
A model generally cannot 'pick up' on information it doesn't have, and that is where  finesse and creativity comes into play.

Whether you should one-hot or leave a feature as categorical depends on the modeling approach. Some, like randomForest will do fine with categorical predictors; others prefer recoding.

Intuition on these questions comes with practice and experience.  There's no substitute for trying out and comparing toy examples to see how your choices affect outcomes. You should take the time to do that, and intuition will follow.


I don't think you need to worry, instead I would ask myself if the accuracy I'm getting is good enough for the task that the NN is supposed to do.
Having higher training loss than validation loss can mean different things:

Your validation data is easier to assess than training data. If the train/validation split is done randomly and there is enough data in both subsets, this shouldn't be the case.
You're using dropout in training but not in validation. This is the default of some deep learning libraries, and it makes sense. If this is the case and you want to see less of a gap, try to reduce the amount of dropout rate and you'll see less of a gap.

To sum up, I don't think it's an issue but you might be able to improve your validation performance by reducing the amount of regularization or increasing the complexity of the NN. However, this is just a hypothesis and the only way to know is to re-train and check the new performance.

Edit
By default, keras doesn't do dropout in prediction so this is likely your case since you have high dropout rates.


Feature selection - QR code localization I just started learning about machine learning recently and have a project where I have to develop a program for QR code localization so that a QR code can be detected and read at any angle of rotation. Development will be done in Python.

The plan is to gather various images of the QR codes at different angles with different backgrounds. From this I would like to create a dataset for training with neural networks and then testing.

The issue that I'm having is that I can't seem to figure out a correct feature design for the dataset and how to identify the QR code from the images for feature processing.
Would I use ground-truth images or edge magnitude maps?

Any help with this would be amazing?
Thanks for your time.

The idea would be to find

1) Rotational independent and
2) Shift invariant features.

Simply speaking the Fourier transform is a rotational and translational (shift) independent feature.
There are SIFT features used in computer vision that can extract such features. This is something you can try.

Since you are using QR codes,I think edge magnitude maps would be better than images because you could use image processing techniques to suppress the  noise in the background and highlight the edges on the QR code. You could even look at autoencoder and CNN related feature extraction.


Clustering for Categorical Data? How exactly does k-means clustering for categorical data work?
I have a dataset which has several categorical features that can have 2,3,4,..,n values. I could one hot encode them, but I'm not sure if it even makes sense because k-means uses a distance metric.
I looked into k-prototype clustering, but tried it on my dataset, but plotting it does not make sense when the y-axis and y-axis are just integer values that are small.

You can use distance metrics that are better suited to boolean values such as Jaccard, or Manhattan. Usually K-means uses mean as the metric to re-adjust the centroid of the current cluster, but I think you could change this to majority vote.
I think if you one-hot encoded the categorical features, you could use Jaccard for similarity and then majority vote for updating the centroid. Jaccard would be good if your data is quite sparse. Can you reform your data so that it is one-hot encoded? What does the distribution look like of the count per row?


How does scaling affect Logistic Regression? I have searched a lot on the web for this question, but I never seem to find a consistent yet straight forward answer. Simply put, the question is: How exactly does scaling affect logistic regression? What should you expect from a logistic regression classifier when the data is scaled and when it is not? Is there any difference between min-max scaling and standard scaling in terms of logistic regression?

It affects anything optimized by a form of gradient descent, because it affects the relative scale of the dimensions of the input. If A is generally 1000x larger than B, then changing B's coefficient by some amount is in a sense a 1000x bigger move. In theory this won't matter but in practice it can cause the gradient descent to have trouble landing in the right place in B's dimension.

I think the more significant effect may be regularization. All terms are penalized equally in a simple formulation of regularization. Again if A is generally 1000x larger than B, its coefficient will tend be 1000x smaller (all else equal) than B's, so will be far less penalized, when there's no particular reason to treat them differently.

Finally it affects interpretation, if you want to view the coefficients' magnitudes as correlating with importance, for the same reason.


Subsampling the “right” amout of data to train an ML model I am training a machine learning model (i.e., a classifier) on a large dataset. I know that I can get the same results using less data (about 30%) but I would like to avoid the trial and error process to find the 'right' amount of data to retain from the dataset.
Of course I can create a script which automatically tried different thresholds but I was wondering if there is any principled way of doing this. It seems strange that nobody ever tried to create a proper solution since this seems to be a very common problem.
Some additional criteria:

I am using sub sampling on a stream of data so it would be better to find something that works in this setting
I would prefer to avoid training the classifier more than once since it takes some time
I appreciate theoretically justified approaches

Any suggestion or reference?

Every Machine Learning Problem is different, so there is no standard answer to your question. For the problem you're working on maybe a 70-30 train-test split would result in an optimal model which performs equally well on the test dataset, whereas for another problem may be that ratio just won't do any justice to the model. It's all about experimentation.
Basically, While training your model, you are basically trying to teach the model all relations, dependencies among attributes which can help the model create a clearer decision boundary between data points associated with your response variable.
Less training may not achieve the task as the model may not learn the underlying structure of the data and hence how the attributes are linked to the response variable, too much training data can have adverse affects too as then you may overfit. I would recommend that start from a 50-50 split record the model performance based on the metric you chose and then repeat the exercise for 60-40, 70-30 etc.


How to drop points from a data series for presentation? We monitor long running industrial engines and we have data series that we want to present on a line chart on a web page. For instance, we have sensors that monitor the oil temperature and pressure on the engine.

There are several other similar data series on the components of the equipment.

The objective is to have a human operator identify deviations in engines, for post-analysis. Our chart will display 24 or 48 hours of engine operation and the operator may identify peaks in temperature or pressure, or on the other measurements.

As such, it is a large amount of data to present on the chart on the web page, and we're starting to hit limitations in several places.

At 24 hours * 3600seconds/hour * 1 data point/second = 86400 data points on the chart.

This amount of points is slowing down the rendering of the web page, and is a lot of data to transfer.

We want to reduce the count of data points presented on the chart, without losing much context. So I ask:

How can I drop data points without losing much precision?
What techniques are usually applied in this scenario?

A first (naive) thought was to group them in 5-second windows and only return one data point to represent the 5-second window on the chart;

should I average the points in the window?
should I consider the maximum in the window?

Are there other techniques than grouping data points in window, to reduce the loss of meaning for the monitoring?

Indeed this problem isn't a very simple one to deal with, despite looking very easy to conceptualise. There exist a certain number of techniques for "reducing" the number of points of timeseries, one being called "downsampling".

A little litterature : https://skemman.is/bitstream/1946/15343/3/SS_MSthesis.pdf

hope this helps,

Cheers


Segmenting data in a Dataframe and assigning order numbers (Python using Pandas) I have a large Excel data set which I load as Pandas. In one column I have repeating strings/numbers (the information can be a text and or integers), in another column I have a unique identifier of that row, there is also a column with a due date. Using sort_values I can group the repeating values and sort according to the due dates. But now I would like to add a column containing numbers to get a ranking of the due dates of the first column.
example data:

[[Item, Kind, ID, Client, Duedate],[A, 3, 689, XY, 01.04.2019], [A, 3, 689, XY, 02.04.2019],[B, 3, 689, XY, 01.04.2019],[C, 3, 689, XY, 01.04.2019],[A, 3, 689, XY, 01.04.2018],[C, 3, 689, XY, 01.04.2019]

I sort them so that the output is all the A's first, then all the B's, and within that group they are sorted according to priority. But now I want to extend my data set into looking like this:

[Item, Kind, ID, Client, Duedate, **priority**],[A, 3, 689, XY, 01.04.2018, **0**],[A, 3, 689, XY, 01.04.2019, **1**], [A, 3, 689, XY, 02.04.2019**2**]

if tried looping over the values, but it doesn't seem to work, and I could not find anything on Google to help me. Any hint would be much appreciated, I am stuck with this for 10 days now..

I think you are asking how to add a column "priority" to the end of a dataFrame that has been sorted.

Assuming the dataFrame is in the order you like, I would first get the length of the frame:

length = len(df)

Then apply that to a new column in the dataFrame:

df['**priority**'] = ["**"+str(x)+"**" for x in range(length)

I'm still new to posting on here so I don't know how to get the input/output examples working, but that should get you a new column on the end of your frame, in string format, like you pictured above.


How to use SMOTE in Java Weka API? I am trying to build classification model using Java Weka API. My training dataset have class imbalance problems. For this reason, I want to use SMOTE to reduce class  imbalance problem. But, I do not know how to use it in Java Weka API.

Welcome to the community.

You can use the following code:

import weka.filters.supervised.instance.SMOTE;
import weka.core.Instances;
import weka.core.converters.ConverterUtils.DataSource;
import weka.filters.Filter;

Instances data = DataSource.read(".../file.arff"); //Dataset

SMOTE smote=new SMOTE();  //create object of SMOTE
smote.setInputFormat(data);
Instances data_smote = Filter.useFilter(data, smote); //Apply SMOTE on Dataset


How to handle negative words in word2vec? I am training a big corpus using word2vec and averaging the word vectors to get sentence vectors. What is the best way to address negative words so that negative and positive sentences are away from each other? For e.g.: "After the fix code worked" and "After the fix code did not work" should ideally give sentence vectors which are far from each other. I heard one approach is to look for negative words like "not" and negate the next word vector. Can someone please clarify if that's a good approach or can suggest a better approach?

When you look at the vectors that word2vec generates - negative words may have unique features but can be treated just like positive words. That is to say, as far as the NN is concerned - these are just similar words. You may have to construct "concept vectors" on top of the word vectors to do what you would like to do.

Your parts of speech tagging should automatically mark negating words as ADV. You can then train on these adverbs in conjunction to your verbs as a positive or negative output. Here's an example using spacy:-

import spacy

nlp = spacy.load('en')        # this can take a while
sample_text = u'Do not go.'
parsed_text = nlp(sample_text)
token_text = [token.orth_ for token in parsed_text]
token_pos = [token.pos_ for token in parsed_text]

At this point token_text will be a list of your words and token_pos will be the POS tagging:-

Do - VERB
not - ADV
go - VERB
. - PUNCT

As you can see, "not" is tagged as ADV here. You can now feed this tagged output (or a better parse tree) into a second network to train for a negative or positive output.

Hope this helps.


Is it valid to include your validation data in your vocabulary for NLP? At the moment, I am following best practices and creating a "bag of words" vector with a vocabulary from the training data. My cross validation (and test) datasets are transformed using this model, using the same vocabulary created by the training set. They don't contribute any vocabulary, or affect the document frequency (for "term-frequency inverse document frequency" calculation).

However, this is restrictive in a few ways. Firstly, calculating the bag of words model is expensive, and so this prohibits me carrying out k-folds cross validation (since it would require constant re-calculation of the bag of words). My dataset is around 10 million words, and I'm calculating bag of words and bag of bi-grams, which takes around 5 minutes each time.

This also means I currently have holdout data for both my cross validation and test sets, which is data I can't use for training.

Would I be biasing my results significantly if I fit the bag of words on both the training set, and the cross validation set? In other words, if I use the vocabulary in the validation set to calculate the vocabulary for the bag of words? The way I figure, even though they might contribute to the vocabulary, there's no risk of overfitting since the frequency for those specific samples won't be seen at training. This allows me to slice the validation set later however I like, and I still have a "test" set for an accurate predictor of generalisation error (the test set won't be seen at all until test time).

I wonder if there's any precedent for something like this, and what your experiences are doing anything similar.

Yes.  That should be fine.  What you suggest makes sense and should not bias the results.  The reasons you give are good ones.  For any reasonable classifier, if the value of an attribute is always zero in the training set, that should cause the attribute to be essentially ignored.

There is a simple test to let you confirm this.  You can try, for each document in the validation set, zeroing out the entries in the feature vector that correspond to words that were not present in any document in the training set, and see if that changes the classification.  If it doesn't, then you know that your method has had no effect and hasn't introduced any bias.

Here is another alternative, if you prefer:

Conceptually, in an ideal world the vocabulary for each fold would only include words in the training set.

As a matter of implementation, it's certainly possible to implement that in a more efficient way than re-generating the vocabulary and re-generating the feature vectors for each fold.  As long as your implementation generates the same vectors, it doesn't matter how it obtains them.

Thus, you could have an implementation that works like the following:

Generate a "superset vocabulary" that includes all of the words found anywhere in the dataset (not just the training set).
Generate a sparse feature vector for each document, based on the "superset vocabulary".  I suggest that you represent these in an efficient way, e.g., as a Python dictionary that maps from word to count.
For each fold:

Split into training and validation sets.
Generate a subset vocabulary, containing only the words in the training set.  Probably there is a small set $S$ of words that are in the superset vocabulary but not the subset vocabulary, so I suggest storing that set $S$.
For each document in the training set and validation set, generate a derived feature vector for the document, using only the subset vocabulary.  I suggest generating this by starting from the feature vector for the superset vocabulary, then removing the words in $S$.  This should be more efficient than regenerating the feature vector from scratch.
Now train and validate these derived feature vectors.

This is equivalent to the "conceptually ideal" approach, but will run faster.

That said, I think your proposal is the pragmatic one, and in practice should yield equivalent results, and be even faster still.


How would one use a network trained on CIFAR-10 to find things in a photo? There are a million and one examples and tutorials on how to train up a neural network on the sample sets like the MNIST data and the CIFAR-10 data, but how does one go from the toy examples of recognising 200x200 clips each containing a single centred object to a real problem like finding CIFAR-10 category objects (the dog and the cat below) within a picture, like I presume Google does for their photo annotation.

Can someone describe how one might approach this leap from the classroom to the real world?

This is a well defined problem called text spotting. There are numerous avenues to tackle this problem but most of the good ones are based on deep learning. Naively you would use a network like the one you trained on MNIST to slide over your input and see where it fires strongly to build up a string. This approach works reasonably well but to convolute this over your whole input image is extremely computationally expensive. A way that is actually used in practice is a two step process, first there is a network that is trained in localizing areas of interest, which are bounding boxes of parts where there might be texts and then use a more advanced network to grab the text. To my understanding this is also done in one network pass nowadays as opposed to querying on single patches. If you search for text spotting you will find a lot of nice papers and a thesis, mostly from the same guy.

http://www.mathstat.dal.ca/~hgu/Neural%20Comput%20&%20Applic.pdf


Outputs of an LSTM Cell from each cell of lstm, what are the output's and what does they signify? i understand that there will be three outputs. A long term memory, short term memory and a output. But, i am little confused from colah blog which can be found here. Here he shows that there will be three outputs, one is long term state and other two outputs are exactly same. What is the use of two outputs being same?

The outputs are the "cell state" which is transferred only to the next LSTM cell and the "hidden state" which goes to the next cell and also as an output of the layer (in the case LSTM should return sequences).


NLTK pos-tag fails to recognize names I'm trying to identify all the names in a novel (fed as a text file) using NLTK. On a smaller scale, the POS tagging works perfectly. However, when I feed a large body of text, by which I mean three or four paragraphs, the system fails miserably. How can I fix this?

I read the file, split it into lines, sentences and then words. What I get is a list of lists where each internal list contains words of a sentence. Then, I use NLTK to tag each sentence.

def process_file(_file, tagger, stemmer, stopwords, filename, printinfo):
    sentences = []
    _nnp = set()
    words = dict()
    for line in _file:
        for sentence in nltk.tokenize.sent_tokenize(line):
            sentences.append(nltk.tokenize.word_tokenize(sentence))
    sent_count = 0
    for sentence in sentences:
        sent_count+=1
        tags = tagger.tag(sentence)
        for tag in tags:
            if tag[1] == "NNP" or tag[1] == "NNPS":
                _nnp.add(tag[0])
            else:
                if tag[0] not in stopwords:
                    stemmed_word = stemmer.stem(tag[0])
                    if stemmed_word not in words.keys():
                        words[stemmed_word] = 1
                    else:
                        words[stemmed_word] += 1
        print("\r[{0}] Reading file '{1}'[{2:>3.1%}] ".format(printinfo, filename, sent_count/len(sentences)), end='')
return _nnp, words

Main code:

dictionary = dict()
nnp = set()

# Initialize tagger and stemmer
pos_tagger = nltk.tag.PerceptronTagger()
ps = nltk.PorterStemmer()
stopwords = nltk.corpus.stopwords.words('english')
file_count = 0

# For each file, extract words and named entities
for file in files:
    file_count += 1
    _nnp, _dictionary = process_file(open(file, 'r', encoding='utf-8'), pos_tagger, ps, stopwords, file, str(file_count)+"/"+str(len(files)))

    # Extend dictionary
    for word in _dictionary.keys():
        if word in dictionary.keys():
            dictionary[word] += _dictionary[word]
        else:
            dictionary[word] = _dictionary[word]

    # Join sets
    nnp = nnp.union(_nnp)
end = time.time()
print("COMPLETED: Step 2 completed in {0:.3f}s".format(end-start))

Here's a sample of what the text might look like:

In slow motion, afraid of what he was about to witness, Langdon rotated the fax 180 degrees. He looked at the word upside down.

Instantly, the breath went out of him. It was like he had been hit by a truck. Barely able to believe his eyes, he rotated the fax again, reading the brand right-side up and then upside down.

"Illuminati," he whispered.
```

I just processed a bunch of files from a random corpus and the results seem to make sense generally speaking, the entities list (at the bottom) does not contain any suspicious words -there may be an issue with false negatives, but there seems to be no obvious problem with false positives.

Any chance you mistook the keys of the variable dictionary for the contents of nnp? Check this out:

# print(dictionary.keys())
dict_keys(['thi', 'excel', 'start', 'film', 'career',
'.', 'hi', 'talent', 'show', 'long', 'ahead', 'the', 'car',
 'truck', 'chase', 'excit', '1937', 'era', 'seri', 'american',
 'treasur', 'book', 'spring', 'perform', 'usual', 'owner',
'right', ',', 'take', 'chanc', 'get', 'produc', 'I', 'think',
'would', 'winner', 'recent', 'air', 'episod', "'s", 'It',
#
# ... (skipping many items) ...
#
'bang-up', 'horror/act', 'hybrid'])

# print(nnp)
{'Better', 'Rackham', 'Unicorn', 'Rooney', 'Alucarda',
'Mejia', 'Alejandra', 'Red', 'Soviets', 'Marisol', 'Remi',
'Captain', 'Santacruz', 'Digest', 'Mickey', 'de', 'Infernal',
'PBS', 'Andy', 'Recently', 'England', 'Director/co-writer',
'yummy', 'Anda', 'DVD', 'Mauri', "I'ts", 'Hergé.', 'Zero',
'Alberto', 'Herge.', 'Baron', 'Byington', 'Sadoul', 'Galindo',
 'Mauricio', 'Tono', 'Free', 'Mexican', 'P.O.V', 'Boys',
'Author', 'Swedish', 'Snowy', 'Breaking', 'Vietnam', 'TinTin',
 'Georges', 'Adriana', 'Antonio', 'Anyway', 'Remy', 'Charly',
'Daniels', 'Mr', 'J.', 'Chan', 'A', 'Mr.', 'Plascencia',
'Brilliantly', 'Congo', 'Pedro', '>', '<', 'Hardy', 'Hergé',
'Awesome', 'Valentino', 'Blue', 'Please', 'Jesse', 'Gonzalez',
 'Fanny', 'Children', 'Tintin', 'Numa', 'Sea', 'Vega', 'Panic',
 '/', 'Herge', 'Edith', 'III', 'Catholic', 'Nacho', 'Lotus',
 'Sharks', 'CD', 'Fernandez'}


Kmeans large dataset we are currently performing a K-MEANS under scikit-learn on a data set containing 236027 observations with 6 variables in double format (64 bits).

According to our calculations, the complexity of the algorithm is O(n * k * v * i), with n the number of observations, k the number of clusters, v the number of variables and i the number of iterations which is 300 maximum.

Let (236027 * 3 * 6 * 300) * 64 (size of the double in bits): we have 81570931200 bits which is ~10.1 GB of memory.

However, by running the algorithm on a 600 GB VM, it crashed due to lack of memory.

Are we wrong in our calculation of the space required for KMEANS?

Is KMEANS suitable for this volume?

What solutions are available to us?

Thanks you

You should consider hierarchical K-Means Clustering.

Link to show an idea:

https://www.datanovia.com/en/lessons/hierarchical-k-means-clustering-optimize-clusters/


What ML/DL techniques power Youtube/Netflix search systems? Video platforms like YouTube, Netflix, Amazon prime have an excellent search system - given a search string, find most relevant videos. Which Machine Learning /Deep Learning techniques used for this?

Any pointers will be of great help

At Netflix, the machine learning algorithms used are more complex since the data is really vast and increases day to day. Take a look at these:

Netflix Recommendations: Beyond the 5 stars (Part 1)

Machine Learning Is Everywhere: Netflix, Personalized Medicine, and Fraud Prevention | Udacity


Why would one crossvalidate the random state number? Still learning about machine learning, I've stumbled across a kaggle (link), which I cannot understand.

Here are lines 72 and 73:

parameters = {'solver': ['lbfgs'],
              'max_iter': [1000,1100,1200,1300,1400,1500,1600,1700,1800,1900,2000 ],
              'alpha': 10.0 ** -np.arange(1, 10),
              'hidden_layer_sizes':np.arange(10, 15),
              'random_state':[0,1,2,3,4,5,6,7,8,9]}
clf = GridSearchCV(MLPClassifier(), parameters, n_jobs=-1)

As you can see, the random_state parameter is been tested across 10 values.

What is the point of doing this?

If one model perform better with some random_state, does it make any sense to use this particular parameter on other models?

I personally think that the general idea of optimising your model with different random seeds is not a good idea. There are many other, more important, aspects of the modelling process that you can worry about, tweak and compare before spending time on the effects of random initialisation.

That being said, if you just want to test the effect of random initialisation of model weights on a final validation metric, this could be an approach to do so. Kind of the reverse argument to my point above. If you can show for different random seeds (ceteris paribus: with all other parameters equal) that the final model performs differently, it shows maybe that their is either inconsistency in the model, or a bug in the code even. I would not expect a well-validated model to give hugely differing results if being run with a different random seed, so if it does, it tells me something weird is going on!


How to cluster Houses on the basis of similarity of features+location? I have a dataset of houses like this:

HouseID  Latitude Longitude PriceIndex
  1          1.4     103.120    1.21
  2          1.42    103.112    2.01

I want to find houses which are similar to each other both on the basis of their position as well as their price index.[Also would need to Rank in order of similarities, given one house] I tried using hclust package in R  and was able to extract 9 classes. However the groups don't seem to have any interpretable similarities (for example the points are spread all across the city etc). I haven't done clustering based projects before so any help in the right direction will be helpful. Thanks!

Edit: Removing the price index column from the clustering data-set actually clusters spatially. But adding the price shows only price-based clustering

Check the ranges of your dimensions and consider scaling if you see a large difference.

I would interpret your described  behaviour due to much larger range if the index that the two other dimensions.
See also the question.


Can The linearly non-separable data be learned using polynomial features with logistic regression? I know that Polynomial Logistic Regression can easily learn a typical data like the following image:

I was wondering whether the following two data also can be learned using Polynomial Logistic Regression or not.

I guess I have to add more explanation. Assume the first shape. If we add extra polynomial features for this 2-D input (like x1^2 ...) we can make a decision boundary which can separate the data. Suppose I choose X1^2 + X2^2 = b. This can separate the data. If I add extra features I will get a wavy shape (maybe a wavy circle or wavy ellipsis) but it still can not separate the data of the second graph, can it?

Yes in theory the polynomial extension to logistic regression can approximate any arbitrary classification boundary. That is because a polynomial can approximate any function (at least of the types useful to classification problems), and this is proven by the Stone-Weierstrass theorem.

Whether this approximation is practical for all boundary shapes is another matter. You may be better looking for other basis functions (e.g. Fourier series, or radial distance from example points), or other approaches entirely (e.g. SVM) when you suspect a complex boundary shape in feature space. The problem with using high order polynomials is that the number of polynomial features you need to use grows exponentially with degree of the polynomial and number of original features.

You could make a polynomial to classify XOR. $5 - 10 xy$ might be a start if you use $-1$ and $1$ as the binary inputs, this maps input $(x,y)$ to output as follows:

$$(-1,-1): -5 \qquad (-1,1): 5 \qquad (1,-1): 5 \qquad(1, 1): -5$$

Passing that into the logistic function should give you values close enough to 0 and 1.

Similar to your two circular areas is a simple figure-of-eight curve:

$$a(x^2 - y^2 - bx^4 + c)$$

where $a, b$ and $c$ are constants. You can get two disjoint closed areas defined in your classifier - on opposite sides of the $y$ axis, by choosing $a, b$ and $c$ appropriately. For example try $a=1,b=0.05,c=-1$ to get a function that clearly separates into two peaks around $x=-3$ and $x=3$:

The plot shown is from an online tool at academo.org, and is for $x^2 - y^2 - 0.05x^4 -1>0$ - the positive class shown as value 1 in the plot above, and is typically where $\frac{1}{1+e^{-z}} > 0.5$ in logistic regression or just $z>0$

An optimiser will find best values, you would just need to use $1, x^2, y^2, x^4$ as your expansion terms (although note these specific terms are limited to matching the same basic shape reflected around the $y$ axis - in practice you would want to have multiple terms up to fourth degree polynomial to find more arbitrary disjoint groups in a classifier).

In fact any problem you can solve with a deep neural network - of any depth - you can solve with a flat structure using linear regression (for regression problems) or logistic regression (for classification problems). It is "just" a matter of finding the right feature expansion. The difference is that neural networks will attempt to discover a working feature expansion directly, whilst feature engineering using polynomials or any other scheme is hard work and not always obvious how to even start: Consider for example how you might create polynomial approximations to what convolutional neural networks do for images? It seems impossible. It is likely to be extremely impractical, too. But it does exist.


If the set of all possible states changes each time, how can Q-learning "learn" anything? I found this resource that explains q-learning with a very simple example. Make it a 2D problem, a rectangle instead of a line, and it's still simple. The only difference is that now there are 2 more possible actions (up and down).

My question is: if the length and height of the rectangle are random, as well as the starting position and the location of the Treasure, how can the bot apply the knowledge acquired to the new problem? Is there an evolved version of q-learning for problems with dynamical-states?

if the length and height of the rectangle are random, as well as the starting position and the location of the Treasure, how can the bot apply the knowledge acquired to the new problem?

You have two possible approaches here, depending on how the problems are being presented to you:

If the agent has time to learn/plan on each environment separately, then you need an agent that has capability to learn each environment. A simple tabular Q-learning agent already has this capability, up to a certain size of problem (where the number of states and actions would fit in memory and can be iterated over in simulation enough times). Beyond that size, provided you can come up with a fixed feature set capable of representing any shape and size of problem that the agent could be presented with, and used e.g. DQN or other approximation technique, then you still have a generalised learning bot.
A bot that is generalised during training to attempt to solve new instances with variation needs to be trained with many variations and more state data. If shape, size and layout of the maze can change between episodes, then this data must become part of the state. This can expand the state space a lot, and requires different representations. A simple representation of grid spaces would be actual map of the grid as a rectangular "image", assuming the agent start, walls and goal position could be almost anywhere within the space. If the maze has lots of objects, then you can put each object type into a separate "channel" and use a Convolutional Neural Network as part of the Q function approximation. If the area is more sparse with just a few objects (e.g. just the agent, a single blocker and the treasure) then it would be easier to use a direct vector representation of positions of the objects and use a fully connected neural network.

For a toy problem where key positions can change between episodes, yet this still can be solved easily by a tabular agent, see Open AI's Taxi-V2 which is an implementation of a classic control problem where the locations of a sub-goal and goal are randomised on each episode.


How does $\chi^2$ feature selection work? I can't find the information how $\chi^2$ are used to select numerical features for a model.

Fro instance, If I employ the sklearn library:

from sklearn.datasets import load_iris
from sklearn.feature_selection import chi2

iris = load_iris()
X, y = iris.data, iris.target # X contains 4 features, y does 3 classes

I can build the following table for the dataset

pivot_table = np.zeros((X.shape[1], len(np.unique(y))))
pivot_table[:, 0] = X[y==0].sum(axis=0)
pivot_table[:, 1] = X[y==1].sum(axis=0)
pivot_table[:, 2] = X[y==2].sum(axis=0)

print(pivot_table.T)
array([[250.3, 171.4,  73.1,  12.3],
       [296.8, 138.5, 213. ,  66.3],
       [329.4, 148.7, 277.6, 101.3]])

Then applying  the contingency table to it:

from scipy.stats import chi2_contingency
chi2_contingency(pivot_table, correction=False)

Output:

125.58397740261773
1.0924329599765022e-24
6
[[213.82265358 301.31664021 361.36070621]
 [111.8757204  157.6540915  189.0701881 ]
 [137.51492279 193.78458652 232.40049069]
 [ 43.88670323  61.84468177  74.168615  ]]

But this only provides the information about dependence of variables. And they are dependent because p-value is quite low.

Then I use the other function and I get what I want:

chi2(X,y)
(array([ 10.81782088,   3.7107283 , 116.31261309,  67.0483602 ]),
 array([4.47651499e-03, 1.56395980e-01, 5.53397228e-26, 2.75824965e-15]))

But how does this work? What math lies at the heart of it?

Just go inside the chi2 function

It calculates chi2 values for Y's based on the contingency table then computes p-values for each feature


Time series modelling I have daily data for 2.5 years , but with more data points as 0, so when i excluded them in the cases which seems to be invalid. Can i use any other model than models used in time series or should i consider time series with missing values and proceed ?
since they are quite a lot of missing values and am new to time series i dont know how to proceed with this. Any input is highly appreciated.

First idea is to use, instead of an invalid data point, the last valid value. This is done in finance where some stocks are not quoted everyday (and when you want to publish the value of your fund you use the last available value for the stocks).

However, this can be an issue when you have too many missing values (especially for prediction).

You can try Bayesian data augmentation (works somewhat well with time series) but if you say that you have a lot of missing data you have to accept that anything that you do on rather poor quality data will be of limited quality.