Why are RNN/LSTM preferred in time series analysis and not other NN? I had recently a great discussion about the advantages of RNN/LSTM in time series analysis in comparison to other Neural Networks like MLP or CNN.

The other side said, that:

The NN just have to be deep enough to model the time connections
RNNs are just used, because they are faster
With RNN the older (e.g. t-20) timepoints are not that relevant in comparison to the newer timepoints (e.g. t-1), because the data from older timepoints have to go through all neurons until the weight is updated in this neuron.
It is easier to understand the weight matrix of normal NN in comparison to RNN to understand partly the behavior of the NN

My answer to that was:

Everyone suggests using the LSTMs for times series analysis
It can consider the time better in comparison to a sliding window approach especially if the length of the sequence is quite long
To 3. That happens just if you use vanilla RNN instead of LSTM, because of the gradient explosion

So I know that I did not answer him completely (and maybe not correctly) and I am now really interested in the real reasons why LSTMS are suggested the most of the times. Maybe he is right? What do you think?

Can a normal NN model the time connections the same way like a
RNN/LSTM does when it is just deep enough?
Does an RNN need more or less data in comparison to a NN to get the
same/ better results?
Are there time series where normal NN or RNN/LSTM perform better?
Is it time data depended which Model will perform the best or are there some guidelines?
Can the NN behavior be understood better than the RNN behavior?

I'll try to provide some insight which will hopefully help.

Can a normal NN model the time connections the same way like a RNN/LSTM does when it is just deep enough?

Every neural net gets better in theory if it gets deeper. For a regular NN to model time connections properly, you could use the last n time steps as your input and the n+1th time  step as your target. This will generate your training set and depending on your data, you could be able to model your time series fairly efficiently.

All of the most obvious pitfalls of this approach are actually addressed by RNNs/LSTM.

Does an RNN need more or less data in comparison to a NN to get the
same/ better results?

Difficult question, which would probably require some empirical results to check that theory. Also, with neural nets, sometimes, it's more about how fast it trains rather than how much training data it has that will make the biggest difference in performance.

To me, the main difference is that your regular NN will need a fixed-size input, whereas your RNN will be able to learn with input "up to" a certain size, which can be a big advantage to model the entire time series well.

Are there time series where normal NN or RNN/LSTM perform better?

Again, this is a difficult question as it will depend on the data, the architecture of the networks, the training time etc.

I haven't done empirical research on this, but I'd say that your best guess would be time series that are rather short (less than 100 time steps for instance). If the time series is short, you might not need to model such an intricate relationship through time, which a regular NN could perhaps do as well as an RNN.

Is it time data depended which Model will perform the best or are there some guidelines?

As explained above, every learning exercise will highly depend on the architecture and hyperparameter used, even the initialization of your weights, whether you use pre-training or not, and of course, your data. Again, I think that the shortest the time series, the more competitive regular NNs could be. But again, this is merely intuition-based and hasn't been checked thoroughly.

- Can the NN behavior be understood better than the RNN behavior?

It all depends on what you mean by understanding. RNN have more weights than NN so ultimately, there will be more things to analyze and eventually understand. But, with more data also comes more information, so perhaps there is more information to be gained from an RNN than a simple NN, even if it's a deep one.

Plus, sometimes, based on the initialization of the weights and other parameters, the interpretation of the model could vary, even if it's trained on the same data.


Approaches to pre-processing the huge but organised text data, with & without the generators I've a huge text file, hence I'm reading it line-by-line, applying some basic cleaning, and separately writing the X & Y to 2 different csv files. Further I'm preparing 3 directories for each csv - train, val & test and writing each line as a separate csv to appropriate directories - This aids in using the fit_generator() method conveniently, by reading these files 1-at-a-time and train the model.

The concern is, before training, I've pre-processing steps and performing those on these many files, 1 file at a time, doesn't seem to be a practical approach(it won't be time-efficient as the operations wouldn't be vectorized, besides there would be lot of read/write on disk since storing each processed file is also inevitable), are there any other approaches in dealing with such scenarios? What are the best practices? Are custom generator functions the only way? Appreciate any help.

Update: Also, What if my processed data-set is a coo-matrix? Is there a viable way other than converting it to dense before writing? Moreover, my concern is neither about optimal resource utilization nor about time efficiency, it is more about what are the different ways of handling such scenarios, an example might help.

Could you reduce data types, for example, int32 to int16, but you should be careful, you have to be sure that you don't lose important a piece of information by reducing of memory.
Iteratively read CSV and dump lines into the SQLite table. Working on database is faster than on CSV file.
Use the library for parallel computing in Python like Dask or Pandarallel.


Sensitivity to scaling of features in a multivariate gaussians I'm using the HMMLearn python package for hidden markov models.  That implementation is build on multivariate gaussian distributions.

So I have a string of features.  How sensitive are gaussians to vastly different feature scales?  Will it be really skewed if one feature is scaled between 0 and 1, and another is scaled between 0 and 1e8?

There are many pitfalls to not scaling your data and it is generally very advisable to scale it. It is so easy to do, it is reversible, and it useful in other operations like removing outliers.

Upon further analysis of the specifics of Hidden Markov Models using Multivariate Gaussians the theoretical accuracy should not suffer as a result of drastic differences in the scales of your features.  But, an important operation involving multivariate Gaussian distribution is matrix inversion.

Though the theoretical invertibility won't change, the practical numerical solution to your problem will likely suffer inaccuracy due to the difference in scales.  Iterative methods will have issues with convergence and direct solve methods will suffer from stiffness.  This is especially true when common complications like linear dependence are present.

Here is a set of 3 lectures on HMM with some specifics on multivariate Gaussians (1-2-3)

I know we've already discussed this in the comments, but I wanted to close out this question, so have added it as an answer.

I hope this helps!


lookup and fill some value from one dataframe to another I have 2 dataframes, df1,and df2 as below.
df1

and df2

I would like to lookup "result" from df1 and fill into df2 by "Mode" as below format. Note "Mode" has become my column names and the results have been filled into corresponding columns.
also note that "ID" from df2 may not necessary equal to "ID" from df1.For example, I am only interested in 4 IDs (A01,A03,A04 and A05, no A02)
while df1 may contain more IDs

I tried to use below code but it doesn't give me a good result.
merged_df = pd.merge(df2, df1,left_on = 'ID', right_on = 'ID', how='outer')
How can I do this?

I would recommend "pivoting" the first dataframe, then filtering for the IDs you actually care about.
Something like this:
useful_ids = [
    'A01',
    'A03',
    'A04',
    'A05',
]
df2 = df1.pivot(index='ID', columns='Mode')
df2 = df2.filter(items=useful_ids, axis='index')


How can we use Neural Networks for Decision Making intead of Bayesian networks or Desicion Trees? I am working on Decision Making in Self driving cars and I am wondering how I can use Neural networks (is there any type) ? that can repleace or mimic the bayesian networks or Decision Tree for Decision Making process ?

One of the central challenges to using neural networks for decision processes is the transparency of the decision mechanism. We tend to grant more trust and credence to algorithms that we can see some of the intermediate reasoning steps for.

Neural networks are, notoriously, black-boxes algorithms in which the intermediate steps are largely obscured from the developer and/or user.

To that end, using a method to explain the decision mechanism of the neural network such as lime or some proxy model approach, to express the decisions made by the network via an alternative model type such as decision trees or linear models, which lend themselves better to interpretation allows us to address this property of neural networks.

That said, there's no direct barrier to using neural networks as the basis of a decision system. Any function which takes data and outputs a prescriptive decision could meet this criterion.

The real trick is ensuring to yourself (as the developer of the algorithm) that the decisions the network is making are in line with the decisions you want it to make and, separately, building confidence through evidence to the stakeholders of the algorithm that:

The engine is making the decisions they need it to
That they have a sufficient understanding of how that decision is being made to trust it to make the decision correctly going forward
That the mechanism the algorithm is using is "fair"-- in that it's not going to adversely affect one group or another in ways that could have serious ethical, regulatory, or legal consequences-- and other risks posed by the implementation of the alorithm for its intended purpose are expressed. This instance comes to mind as as a specific risk for self-driving car decisions. Self-driving cars have potentially huge benefits, but it would be irresponsible (at least) to ignore the ways the AI that powers them could go wrong.

It's worth saying that the last two points above are often not well-understood by stakeholders and therefore fall to the developer, as the person who should have the knowledge of risks related to the approach they are using, to proactively address.


LSTM / GRU weights during test time I am working on a historic time series dataset and using RNN, LSTM, GRU models, and I didn't find an answer if in test time, the h (or h, c) weights should be zeors for each batch?
If the weights should be zero, what they should be? the last updated weights from the training?
Thanks

The initial weights of h for GRU and h,c for LSTM are are often set to zeros, setting random weights is also an option. Also people have tried to learn the initial hidden states.
Since the hidden states are updated with every cell, if your sequences are long enough, it would not make a big difference how you initialize the hidden states.


Should boolean features be normalized and should false be -1 or 0 I am attempting to train an SVM from a set of features which are both numeric and categorical, for example:

Distance X (Numeric)
Distance Y (Numeric)
Font Size Difference (Numeric)
Word 1 Bold (Boolean)
Word 2 Bold (Boolean)
Word 1 Font Size (Numeric)
...

For mapping the features to a feature array I am treating true as 1 and false as 0 and then normalizing the entire feature array using Z-Scores.

Should I instead encode false as -1 so it has a numerical impact on the generation of the support vectors, with a value of 0 it does not modify the chosen vector (I assume)?

Should boolean features be normalized in the same way as numerics or should they be left with their encoded values?

Based on the comment by Ricardo Cruz I tried switching the value of false from 0 to -1 and also normalization on and off.

Neither switching the value from 0 to - 1 or normalization of features had any impact on the values predicted by the SVM. In this case I was using a Gaussian kernel and Sequential Minimal Optimization for my SVM.


Classification of goods by category i have labeled data with goods like:
"Double chamber refrigerator Hitachi R-WB 482 PU2 GBW".

I need to predict category like: laptops, household appliances etc.

How can i do this?

Random forests should be able to handle categorical values natively so look for a different implementation so you don't have to encode all of those features and use up all your memory..
XGBoost
Dimensionality Reduction
CatBoost
Hierarchical Clustering Using Scipy
Target Encoding


PCA before Affinity Propagation (AP) I have a large-ish dataset (100k samples, ~100 features), that I am trying to cluster, to an unknown number of clusters.
I thought of using PCA first, to reduce dimensionality, since I understand that distances (euclidean, cosine) in high-dimensionality might not be actually accurate.

However, I found very little on using PCA before AP, unlike K-Means, which made me wonder - is there any reason to not use PCA and then affinity propagation?

Independent of the number of features, you will obviously need much more than

3 * 100.000 * 100.000 * 8 bytes

of ram (with double precision floats). That is 240.000.000.000 or about   240 GB. Not only is this a lot of RAM to have, but AP will have to do many passes over this RAM, so this will take forever, even after computing the distance matrix.

So clearly PCA won't help. You just cannot afford an algorithm with O(n²) memory and O(n²) runtime. PCA reduces d, but the runtime of AP itself does not depend on d.


PicklingError in pyspark (PicklingError: Can't pickle : attribute lookup Person on __main__ failed) I am unable to pickle the below class. I am using data bricks 6.5 ML (includes Apache Spark 2.4.5, Scala 2.11)

import pickle
class Person:
  def __init__(self, name, age):
    self.name = name
    self.age = age
p1 = Person("John", 36)
pickle.dump(p1,open('d.pkl','wb'))```

PicklingError: Can't pickle <class '__main__.Person'>: attribute lookup Person on __main__ failed

Possible answer from here

The problem is that you're trying to pickle an object from the module where it's defined. If you move the class into a separate file and import it into your script, then it should work.

That solution isn't viable for me in an iPython notebook though. So here I some additional information from here

Python's pickle actually does not serializes classes: it does
serialize instances, and put in the serialization a reference to each
instance's class - and that reference is based on the class being
bound to a name in a well defined module. So, instances of classes
that don't have a module name, but rather live as attribute in other
classes, or data inside lists and dictionaries, typically will not
work.
One straight forward thing one can try to do is try to use dill
instead of pickle. It is a third party package that works like
"pickle" but has extensions to actually serialize arbitrary dynamic
classes.
While using dill may help other people reaching here, it is not your
case, because in order to use dill, you'd have to monkey patch the
underlying RPC mechanism PySpark is using to make use of dill instead
of pickle, and that might not be trivial nor consistent enough for
production use.
If the problem is really about dynamically created classes being
unpickable, what you can do is to create extra meta-classes, for the
dynamic classes themselves, instead of using "type", and on these
metaclasses, create proper getstate and setstate (or other
helper methods as it is on pickle documentation) - that might enable
these classes to be pickled by ordinary Pickle. That is, a separate
metaclass with Pickler helper methods to be used instead of type(...,
(object, ), ...) in your code.
However, "unpickable object" is not the error you are getting - it is
an attribute lookup error, which suggests the structure you are
building is not good enough for Pickle to introspect into it and get
all the members from one of your instances - it is not related (yet)
to the unpickleability of the class object. Since your dynamic classes
live as attributes on the class (which is not itself pickled) and not
of the instance, it is very well possible that pickle does not care
about it. Check the docs on pickle above, and maybe all you need there
is proper helper-method to pickle on you class, nothing different on
the the metaclass for all that you have there to work properly.


Attempt to reuse RNNCell with a different variable scope I'm using Tensorflow 1.1.0 with GPU support and I have this function:

def get_init_cell(batch_size, rnn_size, keep_prob=0.75, layers=2):
    """
    Create an RNN Cell and initialize it.
    :param batch_size: Size of batches
    :param rnn_size: Size of RNNs
    :return: Tuple (cell, initialize state)
    """
    # Basic LSTM cell
    # lstm = tf.contrib.rnn.BasicLSTMCell(rnn_size)
    lstm = tf.contrib.rnn.BasicLSTMCell(rnn_size, forget_bias=0.0, state_is_tuple=True, reuse=tf.get_variable_scope().reuse)

    # Add drop to the cell
    drop = tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)

    # Stack multiple LSTM layers
    rnn_cell = tf.contrib.rnn.MultiRNNCell([drop for _ in range(layers)], state_is_tuple=True)

    # Getting an initial state of zeros
    initial_state = rnn_cell.zero_state(batch_size, tf.float32)

    # Set the name
    initial_state = tf.identity(initial_state, 'initial_state')

    return rnn_cell, initial_state

The previous code works perfectly with Tensorflow 1.0.0 if I use lstm = tf.contrib.rnn.BasicLSTMCell(rnn_size) but now I have changed the way I get the lstm cell. And I get this error:

  ValueError: Attempt to reuse RNNCell
   with a different variable scope than its
  first use.  First use of cell was with scope
  'rnn/multi_rnn_cell/cell_0/basic_lstm_cell', this attempt is with
  scope 'rnn/multi_rnn_cell/cell_1/basic_lstm_cell'.  Please create a
  new instance of the cell if you would like it to use a different set
  of weights.  If before you were using:
  MultiRNNCell([BasicLSTMCell(...)] * num_layers), change to:
  MultiRNNCell([BasicLSTMCell(...) for _ in range(num_layers)]).  If
  before you were using the same cell instance as both the forward and
  reverse cell of a bidirectional RNN, simply create two instances (one
  for forward, one for reverse).  In May 2017, we will start
  transitioning this cell's behavior to use existing stored weights, if
  any, when it is called with scope=None (which can lead to silent model
  degradation, so this error will remain until then.)

I have changed get_init_cell with this one:

def lstm_cell(rnn_size, keep_prob=0.75):
    # Basic LSTM cell
    lstm = tf.contrib.rnn.BasicLSTMCell(rnn_size, reuse=tf.get_variable_scope().reuse)

    # Add drop to the cell
    return tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)

def get_init_cell(batch_size, rnn_size, keep_prob=0.75, layers=2):
    """
    Create an RNN Cell and initialize it.
    :param batch_size: Size of batches
    :param rnn_size: Size of RNNs
    :return: Tuple (cell, initialize state)
    """
    # Stack multiple LSTM layers
    rnn_cell = tf.contrib.rnn.MultiRNNCell([lstm_cell(rnn_size, keep_prob) for _ in range(layers)])

    # Getting an initial state of zeros
    initial_state = rnn_cell.zero_state(batch_size, tf.float32)

    # Set the name
    initial_state = tf.identity(initial_state, 'initial_state')

    return rnn_cell, initial_state

And I still getting the same error.

I have also try it with the same error:

def get_init_cell(batch_size, rnn_size, keep_prob=0.75, layers=2):
    """
    Create an RNN Cell and initialize it.
    :param batch_size: Size of batches
    :param rnn_size: Size of RNNs
    :return: Tuple (cell, initialize state)
    """

    def lstm_cell():
        # Basic LSTM cell
        return tf.contrib.rnn.BasicLSTMCell(rnn_size, reuse=tf.get_variable_scope().reuse)

    def attn_cell():
        return tf.contrib.rnn.DropoutWrapper(lstm_cell(), output_keep_prob=keep_prob)

    # Stack multiple LSTM layers
    rnn_cell = tf.contrib.rnn.MultiRNNCell([attn_cell() for _ in range(layers)], state_is_tuple=True)

    # Getting an initial state of zeros
    initial_state = rnn_cell.zero_state(batch_size, tf.float32)

    # Set the name
    initial_state = tf.identity(initial_state, 'initial_state')

    return rnn_cell, initial_state

Any idea?

It looks like you might be trying to use MultiRNNCell as a recurrent neural network. This isn't really its purpose; that is what dynamic_rnn, static_rnn, and raw_rnn are for. MultiRNNCell is intended for creating a feedforward, multilayer network which is the "cell" to be iterated to produce a recurrent neural network.

The old version of your code, if I understand it correctly, creates a network consisting of a sequence of LSTM layers, each with its own set of weights. If your goal is to make the weights shared between layers (so that your network is recurrent), you want to instantiate a dynamic_rnn and pass it a single instance of one of the RNNCell subclasses. That instance generates all of the layers of the network by having its __call__ method invoked repeatedly, once for each layer. The dynamic_rnn instance handles the scoping and variable reuse issues for you and connects the layers together.

You have at least three options for the cell to pass to the dynamic_rnn constructor:

1) A DropoutWrapper which was created using a BasicLSTMCell (essentially, the output of one call to attn_cell, but get rid of that reuse option). Do this option.

2) A MultiRNNCell consisting of an LSTM layer, followed by a dropout layer (unneeded here, but this is what you would do in some more general case).

3) Subclass rnn_cell and create your own type of layer.


Estimating expected revenue generation I have a number of features X and a target which is the revenue generated by this interaction. More often than not no revenue is generated, but if there is it can differ considerably, it has a big right tail. I've made a model to estimate the expected revenue generated by a new x. Due to the weird distribution of the revenues I have split the model up in two parts, first estimating the probability of revenue generation and then conditionally on the fact that somebody will book, predict the amount of $log($revenue$)$. The first model is trained on a dummy variable which indicates revenue and the second one is trained on a subset of the data which contains all interactions that generated revenue.

After I have these two models P (probability of generation) and R (expected revenue generated conditioned on > 0) we can calculate the expected revenue for a new x using:

$$f(x) = P(x)e^{R(x)}$$

I've done a reasonable amount of cross validation on both the models individually and they work decent, however when they are combined they severly underestimate the amount of revenue generated. I have a good enough number of rows for the dimensionality of my data and I'm using Logistic Regression for the probability (linearity assumption is somewhat reasonable, however unbiased probabilities are very important) and Gradient Boosting for revenue estimation (got the best results for the second portion).

What should I do about the underestimation of the revenue? What are alternative unbiased probablity estimators, should I train a neural network with a hidden layer for the log-loss function? Why does the combined model underestimate? Is there a good alternative to do it in one model? Which would not increase small errors into huge errors due to the multiplication and exponent?

Why not just use quantile regressions?

Usually, in regressions, the coefficients are estimated for the average case: $\min_\beta \sum_i||X_i\cdot\beta -y_i||^2$.

If you want, you can estimate your coefficients so that they tell you things like $P(R=y)>p$. You give a probability $p$, and the model gives you $y$. You can even build probability density function out of these estimation techniques.

In quantile regressions, you solve this optimization problem for every quantile $\tau$ that you want, $\min_\beta \sum_{i|y_i\geq X_i\cdot\beta}\tau|X_i\cdot\beta-y_i| + \sum_{i|y_i< X_i\cdot\beta}(1-\tau)|X_i\cdot\beta-y_i|$.

This is a more robust approach to modeling the revenue distribution. Usually people use quantile regressions because software is readily available, but they can be implemented for other models as well. I work in an institute that does exactly that.


good algorithm for outliers detection I have 2 independent data sets (1. 300 rows and 2.3000 rows) with 6 months trades observations for 50 traders. In both datasets I have:
trader id,
stock title,
buy/sell volume,
date of trade,
sector of stock

My goal is to detect possible outliers (suspicious trades) in this two datasets.

What algorithm you would recommend for this 2 tasks and why?
What can we use for 1 task when we have average only 5-6 trades per trader?

I'd go out on a limb to say this is a difficult problem without incorporating other data. Models just try to explain future action with past information. Traders act differently in different scenarios.

Without data on their analysis or financial data on the stocks they are analyzing, a model on the previous transaction history of traders would not be effective. Unless, if the "traders" you describe are more like etf fund managers that operate on a predefined set of rules.


How to understand ANOVA-F for feature selection in Python. Sklearn SelectKBest with f_classif I am trying to understand what it really means to calculate an ANOVA F value for feature selection for a binary classification problem.

As I understand from the calculation of ANOVA from basic statistics, we should have at least 2 samples for which we can calculate the ANOVA value. So does this mean in the Sklearn implementation that these samples are taken from within each feature? What exactly do these samples represent in the case of feature selection for this problem?

I tried to setup a very simple example which I have listed below, but I am still struggling to understand what the ANOVA value really means here? I'm also struggling to understand how to calculate this by hand, which usually helps me see what is happening on the inside.

In this example, repayment status 0 means the loan is repaid and 1 means it defaulted. I have only supplied 5 rows of data to keep it simple.

The code and results are as follows:

Intuition
We have two classes and we want to find a score for each feature saying "how well this feature discriminates between two classes". Now look at the figure bellow. There are two classes red and blue and two features on $x$ and $y$ axes.

$x$ feature is a better separator than $y$ because if we project data on $x$ axis we get two completely separated classes but if we project data onto $y$, two classes have overlap in the middle of axis (comment if we need more clarification).
What makes $x$ better than $y$? As you see in the figure above:

According to $x$, two classes are far from each other.

Math Translation: The distance between means of class distributions on $x$ is more than $y$.

According to $x$, the scatter of classes do not fall on each other but according to $y$ they do. It means that according to $x$, classes are more compact so more probable to not have an overlap with another class.

Math Translation: The variance of each single class according to $x$ is less than those of $y$.

Now we can easily say $\frac{distance\_between\_classes}{compactness\_of\_classes}$ is a good score! Higher this score is, better the feature discriminates between classes.
Now we know, according to this definition, what $good$ and $bad$ features mean. Let's find a math formulation to quantize it.
Mathematics (to do on the paper)
Let's formulate our two criteria:

Distance between means of class distributions is the numerator. Population is taken into account, I assume for statistical significance (needs a reference from a statistician!).

A concept similar to sample variances of classes is the denominator. Here instead of dividing sum of squares by $(sample\_population -1)$, we sum up all $(sample\_population -1)$s and divide the final value by them.

Now Back To Your Data
To calculate the above you calculate sum of between-class distances and sum of within-class variations for each feature according to different classes. I do it for only one feature. Let's choose Loan.
Class 1: [5000, 18000]
Class 2: [47500, 45600, 49500]

Mean of all points: (47500 + 45600 + 49500 + 5000 + 18000) / 5 = 33120
Mean 1: (5000 + 18000) / 2 = 11500
Mean 2: (47500 + 45600 + 49500) / 3 = 47533
Numerator: 2 x (11500 - 33120)^2 + 3 x (47533 - 33120)^2 = 1,558,052,507

For denominator we go with Sum of Squares Within class (it is simply the numerator in formulation of sample variance):
SSW 1: (5000 - 11500)^2 + (18000 - 11500)^2 = 84,500,000
SSW 2: (47500 - 47533)^2 + (45600 - 47533)^2 + (49500 - 47533)^2 = 7,606,667
Na = 2, Nb = 3 --> (Na - 1) + (Nb - 1) = 1 + 2 = 3
Denominator: (84,500,000 + 7,606,667)/3 = 30,702,222

Now the F-Score for feature Loan is:
F-Score: 1,558,052,507 / 30,702,222 = 50.74

as you see with your calculation in Python.
Note

I tried to explain in a simple way. For example the denominator of sample variance is called degree of freedom but I skipped those terms for simplicity.
Just understand the main idea. Further the means and smaller the within variances, better the feature is. You can formulate it yourself as well (however you will not have p-values anymore ;) )
Finding P-values and understanding what it means, is another story which I skipped.

Hope it helped.
Good Luck!


How to classify images Neural Network didn't trained to Understand Let's say I trained a Convolution neural network to Identify Cats , Dogs and wolves . But suddenly I feed it pictures of rabbits and  Lions. so  how can I classify those as pictures as "Other"

I tried to do this by Adding "Sigmoid" activation functions and getting probabilities for each selection. I thought probabilities for each selection might be different. I thought If I feed a Rabbit picture to NN it will out 20% cat ,44% dog and 34% wolf. That way  I can clearly figure NN is confused. But Unfortunately it gives me results like 70% cat , 10% Dog , 20% wolf.

Can You suggest me a Way to fix this problem ?

You need to modify the dataset by adding 1 label called 'others'. Now, there are 4 output labels: cat, dog, wolf, and others. Then, you train the model again.


Are all classifiers linear in some high dimensional space? Of all possible classifiers (including SVMs, locally weighted regression, softmax regression, lots others I'm sure I don't know about, etc.), are they all linear in some high dimensional space?

E.g. An SVM with a Gaussian kernel is linear in infinite-dimensional space.

The answer is yes!

One can use Cover's theorem to prove it.

Remark 1: by "linear in some high dimensional space" we assume here, for any classifier, an existence of an equivalent linear classifier in higher dimensions (i.e., a decision hyperplane that yeilds identical results to the original classifier for any possible input).

Remark 2: it is definitely true for any classifier on a countable domain (e.g., all possible 8bit RGB 256x256 images is quite large but still a countable domain).

For classifiers on uncountable domains (like fields of real numbers), the answer would depend if Cover's theorem holds for infinitely large number of points.
However, I'd rather not be worried about that, since all real-world classifiers implemented on classical computers are discrete :-)

Remark 3: Of course, classifier must be deterministic...


Does the add_regressor method on Facebook Prophet also work with categorical variables? I went through the documentation of Facebook Prophet and was able to build a similar model for my time series dataset. The additional regressors I used were numeric. I achieved a reasonable MAPE score. However, I also want to analyse this time series on a zip code level, which adds a categorical variable to the mix. Would one-hot encoding do the trick? I'm doubtful about it's efficiency as I would need to individually add each category (zip code) as a separate additional regressor.

My dataset is in the following format:
Zip code | Date | Numeric Attr 1 | Sales (predictor)

I am also open to switching to a different model, should there be a good one which does both time series and regression. Any pointers would be appreciated.

From my understanding Prophet is just a linear regression library that helps to analyze time series with some nice features (like holidays, Fourier transformation, etc.). So from the mathematical standpoint, the regressor must be an ordinal scaled value. The docstring also implicitly says something about it. See the keywords 'additive' and 'multiplicative'. Categorical data is neither of the two. A category is a nominal scale, which can only be counted and if it's ranked, it can be sorted.
Have a nice day.


Clustering stores based on weekly data I have 1 year transaction level data aggregated at a weekly level for 1000 different stores. I want to cluster similar stores based on 8 variables such as sales, customer count etc. The concern is that as the data is at a store week level I cannot directly run the clustering algorithm as it may compare and classify the data points from same store into one cluster, which is not what is required.

So the second option is to transpose the data i.e. 1 data point for each store having 8(variable) * 52(week) columns and then run clustering on it. I am not sure if k-means or the converntional clustering method will work well on such high dimensional data.

So I am looking for suggestions as to how can we handle this situation.

You basically have data with a panel structure ($i=$shops, $t=$ weeks). So for a simple clustering problem, you need to "tell" the algorithm that observations for each $i$ (shops) are not independent.

I think you are on the right way to treat each week $t$ as an own variable/feature here. In this case you would have 416 features and 1000 observations. This should be "okay", but especially with non-supervised problems, it is up to a try.

In any case, I would also try to do a little aggregation on the data (over time). Maybe there are good reasons to do this by month or quarter (or some other sort of "season"), meaning to sum up the weekly values over some period of time. Say you do this for quarters, you would have "only" 32 features, which should be absolutely fine with 1000 observations.

See also this post: https://stats.stackexchange.com/questions/88320/cluster-analysis-on-panel-data


How to change random number of random elements in a column by group in R? I have the following data frame:

>df
   Name
 1 Ed
 2 Ed
 3 Bill
 4 Tom
 5 Tom
 6 Tom
 7 Ed
 8 Bill
 9 Bill
10 Bill

My goal is that from each group by "Name" change the "Name" values 25-75% of random rows to "Name"+"_X" (the remaining rows don't change). So the expected output is similar to:

   Name
 1 Ed
 2 Ed_X
 3 Bill_X
 4 Tom
 5 Tom_X
 6 Tom
 7 Ed_X
 8 Bill
 9 Bill
10 Bill_X

I have tried with for cycle like this (a the moment, for 50% random rows:

for (n in unique(df$Name)){
  df[sample(which(df$Name==n), nrow(df[df$Name==id,])/2), 1] <- paste(df$Name, "_X", sep="")
}

Unsuccessfully, however.

The idea is correct, even if you have a couple of mistakes in your code (like id is not defined, and inside paste you want to use just n, not df$Name). This is not super compressed code but it does the job:

Name = c('Ed','Ed','Bill','Tom','Tom','Tom','Ed','Bill','Bill','Bill')
df = data.frame(Name)

for (n in unique(df$Name)){

    # get indices
    indices = which(df$Name==n)

    # sample size
    samp_size = round(length(df[df$Name==n,])/2)

    # get indices to replace
    samp = sample(indices, samp_size)

    # need to set column as character
    df$Name = as.character(df$Name)

    # set new values
    df[samp,] =  paste(n,'_X',sep='')

    # set again column as factor
    df$Name = as.factor(df$Name)
}

Out:

     Name
1      Ed
2    Ed_X
3    Bill
4   Tom_X
5   Tom_X
6     Tom
7    Ed_X
8    Bill
9  Bill_X
10 Bill_X


How to impute Missing values not the usual way? I have a dataset of 4712 records working on binary classification. Label 1 is 33% and Label 0 is 67%. I can't drop records because my sample is already small. Because there are few columns which has around 250-350 missing records.

How do I know whether this is missing at random, missing completely at random or missing not at random. For ex: 4400 patients have the readings and 330 patients don't have the readings. But we expect these 330 to have the readings because it is a very usual measurement. So what is this called?

In addition, for my dataset it doesn't make sense to use mean or median straight away to fill missing values. I have been reading about algorithms like Multiple Imputation and Maximum Likelihood etc.

Is there any other algorithms that is good in filling the missing values in a robust way?

Is there any python packages for this?

Can someone help me with this?

To decide which strategy is appropriate, it is important to investigate the mechanism that led to the missing values to find out whether the missing data is missing completely at random (MCAR), missing at random (MAR), or missing not at random (MNAR).

MCAR means that there is no relationship between the missingness of the data and any of the values.
MAR means that that there is a systematic relationship between the propensity of missing values and the observed data, but not the missing data.
MNAR means that there is a systematic relationship between the propensity of a value to be missing and its values.

Given what you have told its likely that its MCAR. (assumption is that you already tried to find this propensity yourself (domain knowledge) or build a model between the missing columns and other features and failed in doing so)

Some other techniques to impute the data, I would suggest looking at KNN imputation (from experience always solid results) but you should try different methods

fancy impute supports such kind of imputation, using the following API:

from fancyimpute import KNN

# Use 10 nearest rows which have a feature to fill in each row's missing features
X_fill_knn = KNN(k=10).fit_transform(X)

Here are different methods also supported by this package:

  •SimpleFill: Replaces missing entries with the mean or median of each
  column.

  •KNN: Nearest neighbor imputations which weights samples using the
  mean squared difference on features for which two rows both have
  observed data.

  •SoftImpute: Matrix completion by iterative soft thresholding of SVD
  decompositions. Inspired by the softImpute package for R, which is
  based on Spectral Regularization Algorithms for Learning Large
  Incomplete Matrices by Mazumder et. al.

  •IterativeSVD: Matrix completion by iterative low-rank SVD
  decomposition. Should be similar to SVDimpute from Missing value
  estimation methods for DNA microarrays by Troyanskaya et. al.

  •MICE: Reimplementation of Multiple Imputation by Chained Equations.

  •MatrixFactorization: Direct factorization of the incomplete matrix
  into low-rank U and V, with an L1 sparsity penalty on the elements of
  U and an L2 penalty on the elements of V. Solved by gradient descent.

  •NuclearNormMinimization: Simple implementation of Exact Matrix
  Completion via Convex Optimization by Emmanuel Candes and Benjamin
  Recht using cvxpy. Too slow for large matrices.

  •BiScaler: Iterative estimation of row/column means and standard
  deviations to get doubly normalized matrix. Not guaranteed to converge
  but works well in practice. Taken from Matrix Completion and Low-Rank
  SVD via Fast Alternating Least Squares.

EDIT: MICE was deprecated and they moved it to sklearn under iterative imputer


Selecting a boundary on a binary classifier to optimal precision and recall I have a logistic regression classifier that shows differing levels of performance for precision and recall at different probability boundaries as follows:

The default threshold for the classifier to decide which class something belongs to is 0.5. However, am I right in understanding that in order to get the best performance trade-off I should set the decision boundary to be about 0.82 below? That can be done in Scikit-Learn, but I want to make sure that I am drawing the correct conclusions. Any advice would be appreciated.

The intersection of the precision and recall curves is certainly a good choice, but it's not the only one possible.
The choice depends primarily on the application: in some applications having very high recall is crucial (e.g. a fire alarm system), whereas in some other applications precision is more important (e.g. deciding if somebody needs a risky medical treatment). Of course if your application needs high recall you'd choose a threshold before 0.6, if it needs high precision you'd choose a threshold around 0.85-0.9.
If none of these cases apply, people usually choose an evaluation metric to optimize: F1-score would be a common one, sometimes accuracy (but don't use accuracy if there is strong class imbalance). It's likely that the F1-score would be optimal around the point where the two curves intersect, but it's not sure: for example it might be a bit before 0.8, when the recall decreases slowly and the precision increases fast (this is just an example, I'm not sure of course).
My point is that even if it's a perfectly reasonable choice in this case, in general there's no particular reason to automatically choose the point where precision and recall are equal.


What are the most popular data science application use cases for consumer web companies The most popular use case seem to be recommender systems of different kinds (such as recommending shopping items, users in social networks etc.).

But what are other typical data science applications, which may be used in a different verticals?

For example: customer churn prediction with machine learning, evaluating customer lifetime value, sales forecasting.

Satisfaction is a huge one that I run into a lot.  Huge referring to importance/difficulty/complexity.

The bottom line is that for very large services (search engines, facebook, linkedin, etc...) your users are simply a collection of log lines.  You have little ability to solicit feed back from them (not a hard and fast rule necessarily).  So you have to infer their positive or negative feedback most of the time.

This means finding ways, even outside of predictive modelling, to truly tell, from a collection of log lines, whether or not someone actually liked something they experienced.  This simple act is even more fundamental (in my biased opinion) than a/b testing since you're talking about metrics you will eventually track on a test scorecard.

Once you have a handle on good SAT metrics then you can start making predictive models and experimenting.  But even deciding what piece of log instrumentation can tell you about SAT is non-trivial (and often changes).


Is there a way of performing stratified cross validation using xgboost module in python? I am training and predicting on the same data-set, but I want to perform 10-fold cross-validation and predict on the left out fold and thus predict on the whole data set. How can I do this?

The libraries which I am using are:

from sklearn import cross_validation
import xgboost as xgb

What you are doing is a typical example of k-fold cross validation.

XGBoost is just used for boosting the performance and signifies "distributed gradient boosting".

First, run the cross-validation step:

kfld = sklearn.cross_validation.KFold(labels.size, n_folds=10)

Then, use the train and test indices in kfld for constructing the XGBoost matrix and re-scaling weights by looping over them(the indices).

A very neat implementation has been given as a Kaggle example here.

So, cross validation is not done with the xgboost package, it is done with the cross_validation module of sklearn, and then the gradient boosting is done on the indices of the k-fold validation variable results.


What’s the most suitable programming language for AI development? For the past couple of years I’ve been learning how to use Python to script.  But I would like to start getting into scripting more things like computers and AI.  So, with that said, and please no hate, what in your opinion would be best to script things like that?  For example - JavaScript or c#/c+/c++

Disclaimer: this is massively subjective topic, and below is just my opinion.

I would say it depends on your focus. You could do work in these areas with any major programming language, but some have better suited capabilities and/or better support from a community.
My biased and very simple answer would be to use Python for basically everything, and learn C++ for times when performance is important. And then, if you get really serious, you'll need to learn CUDA (extensions to C++ to programme directly for Nvidia GPUs). If you are proficient at Python and C++, there isn't really anything you couldn't do.
In summary

If you want to do research, you could probably do everything with just Python. There are libraries such a NumPy/PyTorch/Tensorflow which do all the heavy lifting for you.

If you want to go the way of robotics and embedded hardware/software, you'll likely need C/C++.

If you want to make models to deploy in a browser, perhaps Javascript would be useful.


How to use random forest model to new data? I am new to this Data Science field. I have a question to apply Random forest to new data.

I have this table.

Y prop_A prop_B
A   0.8    0.2
A   0.7    0.3
B   0.5    0.5
B   0.4    0.6
B   0.1    0.9

I assumed that if the proportion of the group is high, chances are high that it is in the group. I built a model using random forest and test it with validation set (8/2 splits).

I thought the above model can be used for new data. This is an example of the data. The data structure and variable meaning is same, but the number of variable is different.

Y prop_C prop_D prop_E prop_F
-   0.8    0.1   0.05   0.05
-   0.6    0.3   0.05   0.05
-   0.5    0.4   0.05   0.05
-   0.4    0.2   0.4     0
-   0.1    0.5   0.4    0.4

The new data is unlabeled so I would like to make a label using the Random forest I used with previous data. Is it right approach to label the new data?

In the model, it doesn't works (due to different independent variables).

How should I do to label the new data based on a model using labelled data, which is  different?

Yes, you can do that. However, how accurate your new labels are depends on the ability of your model to generalize to new data and the similarity of the new data to your training data. Therefore, it is something you need to test for in the first place by using a separate test dataset to assess model performance. In contrast to that, a validation dataset is used for model selection and hyperparameter tuning (e.g. max depth of your trees).

An in-depth coverage of the topic you can find in chapter 7 of "The Elements of statistical Learning" by Hastie et al.

With regards to your second question: If your training data does not contain the variables prop_C and prop_D then your model cannot make use of them. So either include these in the training data or ignore them in the new data.


How to avoid overfitting in Reinforcement Learning I have implemented a RL model based on Deep Q-Learning for learning how to play a 2D game, like the ones in the OpenAI Gym. For testing the model, unlike most people, I have chosen to evaluate its performance on different levels from the ones used for training. This way, I can assess if the knowledge learnt by the model generalizes well to previously unseen levels.

After doing some testing and plotting the results, I have noticed an issue. Although the model generalizes well, it ends up overfitting the training set (levels used for training) if the training process continues for too long. This makes sense since early stopping is a common technique used to prevent overfitting. The problem is that the longer the training lasts, the more samples the agent is trained on, which should improve its test performance. This doesn't happen, however, due to overfitting, as I have explained.

I would like to run the model and plot the performance (reward obtained) obtained on the test set as the number of samples in the experience replay increases. The plot I want to obtain should show how the performance increases as the number of samples grows, but overfitting causes that, after a point, the performance decreases, even if more samples are used for training. I want to avoid this. These are the solutions I can think of:

Use techniques to prevent overfitting. Right now I'm using Dropout but it still occurs.
Use an exponential decaying learning rate. Right now I'm using a constant one, since I don't want the newer samples to impact the training less than the older ones.
Give up and change the number of training steps depending on how many samples I want to use for training. Then, to obtain the plots, I will have to repeat the training once per each point (number of samples) of the plot, in order to obtain the best performance for each different size of the experience replay. Of course, this is my last resort.

What do you think I should do?

Edit: I have just tried L2 regularization. It works more or less the same as Dropout.

In RL this is known as the exploration-exploitation trade-off, so i don't think you can avoid this regularizing the neural network, but rather the action policy it learns. The agent you are training is learning to solve a specific task and its policy is "over-fitting". You could tackle this in several ways, one of which is to encourage exploration by using a distribution over the Q Values such as Softmax

$$p(a|s) = \frac{\exp(\beta Q(a,s)}{\sum_{a'} \exp(\beta Q(a',s))},$$

or by applying a $\epsilon$-greedy action selection, where you choose a random action with probability $\epsilon$ and otherwise the max Q Value. There are a lot of other ways to achieve this, just to quick "exploration strategies for dqn" search on google scholar.


How to input different sized images into transfer learning network I have been looking online for a solution but have a difficult time finding a clear enough solution. I want to know how to use transfer learning (VGG16 for example) on images that have different sizes than the images the network originally trained on (so instead of inputting images of size (224,224,3) I want to input images of size (32,32,3)).

I initially thought about just padding those images but the network may look into the black pixels and think that they mean something, and I realize that might hard the accuracy and also when I tried to do that my colab notebook collapsed.

This is my VGG-16 code:
def vgg16_model(img_rows, img_cols, channel=1, num_classes=None):

model = VGG16(weights='imagenet', include_top=True)

model.layers.pop()

model.outputs = [model.layers[-1].output]

model.layers[-1].outbound_nodes = []

x=Dense(num_classes, activation='relu')(model.output)

model=Model(model.input,x)

#To set the first 8 layers to non-trainable (weights will not be updated)

for layer in model.layers[:15]:

   layer.trainable = False
for layer in model.layers[16:]:
  layer.trainable=True
model_new = Sequential()
for layer in model.layers[:-1]: # just exclude last layer from copying
    model_new.add(layer)
model=model_new
model.add(Dense(256,activation='relu',input_shape=(1000,)))
model.add(Dropout(0.3))
model.add(Dense(256,activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(256,activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(256,activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(10,activation='softmax'))
#model.add(Dense(1,activation='softmax'))

# Learning rate is changed to 0.001
#sgd = SGD(lr=1e-2, decay=1e-6, momentum=0.9, nesterov=True)
sgd = SGD(lr=lr,decay=decay,momentum=0.95, nesterov=True)
adam=Adam(lr=1e-5, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0001, amsgrad=True)
#model.compile(optimizer=adam, loss='binary_crossentropy',metrics=['accuracy'])
model.compile(optimizer=adam, loss='categorical_crossentropy', metrics=['accuracy'])

# checkpoint
filepath="weights-improvement-{epoch:02d}-{val_acc:.2f}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list = [checkpoint]

return model

I would greatly appreciate if someone could write the corrected version of this code in order to enable me to insert pictures of size (32,32,3).

Thanks a lot in advance!!

Resizing is the best option, if they are bigger downscale them, else upscale them.


How to determine whether a semantic concept is present in a string I need to find a way to detect if a large string contains a specific substring.
Imagine that I have a full contract page converted to string in my Python program. What I want to do is to say if a specific term (a smaller string than the whole page string) exists in the page converted to string. The matching must be done semantically.
As an example, in the following case, I expect to have a good score for Query 1 and a bad score for Query 2. Notice that it's possible that my query has more than one sentence.
Query 1
"The supplier will accomplish the fulfill the delivery of the products in one week"
Query 2
"I like Phil Colins"
Page text
"The supplier will be paid annually in jully.
The delivery of products will be made in one week.
.........
"
How would you do this task?

This is essentially information retrieval: usually there is a collection of documents and the goal is to find the document which is the most similar to a given query (what you call the "semantic concept").
The traditional way to do that is to convert the collection of documents as vectors, typically with TFIDF weight but there are many options (I assume that recent approaches favor using word (or document) embeddings). Then a similarity measure (for example cosine) is used to measure the similarity between every document and the query, and the most similar document is selected. Since in this case there would be a single document, you could use a threshold on the similarity level in order to answer yes or no (concept found or not).


Do CNNs benefit from HDR images? I have images with 12 bits per color channel which I use for several detection networks (YOLO, RetinaNet, etc.).
Can I expect any precision difference between 12 bpp and 8 bpp as network input? Or is it better to just scale the images down to 8 bpp?

I would say it depends on your images themselves and the task. So my short answer would be: for your task at hand, would this help a human being to perform the task? A similar answer may apply to your algorithm.

To be more specific: as far as I understand HDR compared to traditional images, the aim is to increase the dynamic range of the pictures to make it more similar to what the human eye sees. This means that for pictures with complex lighting conditions (strong backlighting for instance), there will be a real difference between classic / HDR pictures, so that if you aim at detecting objects in theses conditions, HDR will help.

On the other hand, if you have standard lighting conditions, I believe @mikalai answer fully applies and there will be not much difference.

I know this answer relates more to the "photography" side of the question rather than the machine learning side, but I think it is important to consider both in your case.

Does this help?


How to control false positives in sequential A/B testing while keeping a low sample size? I am working on a planned sequence of n independent A/B tests(=run a maximum of n tests or stop earlier if a good improvement is found) and in order to keep the significance level within an acceptable level(=0.05) I'm considering ways of controlling the false positives rate while keeping the sample size as low as I can(it's for a low traffic website)

I'm aware of the Bonferroni, Benjiamimi-Hochberg and related methods designed to control the proportion of false positives in the multiple comparisons situation. However the problem of computing the sample size still remains, the simplest approach seems the one of computing the required sample size starting from alpha = 0.05/n as dictated by the Bonferroni correction and then perhaps use Benjiamimi-Hochberg for the actual testing. Is there a better way of computing the sample size when Benjiamimi-Hochberg is used?

Further as the A/B tests are designed to test possible improvements I was thinking that if the sequential tests are based on independent samples then then probability of two consecutive false positives would be alpha^2 = 0.025. My understanding is that the sequential tests would be independent if the respective samples do not refer to the same users. These could for example be users that joined the website after the previous test.

If the above thinking makes some sense, I could use a "weak" Bonferroni (i.e. alpha' = alpha/(n*0.2) )correction in order to run tests with a reasonable power and lower sample size and once a positive is found(the null hypothesis is rejected), I could repeat the test and if it's positive again then the result is positive with p < 0.025 which means that I would accept the result. This approach would allow for some control over the sample size but... is it sound?

Any opinion would be greatly appreciated. Thanks in advance. AMAR

=run a maximum of n tests or stop earlier if a good improvement is found

Don't do that. Don't stop before n tests have been run if you like what you've seen so far....

http://www.evanmiller.org/how-not-to-run-an-ab-test.html

In general, however, figuring out how big an N you need given an expected effect size and alpha level is the domain of power analysis. Read more here:

http://www.uta.edu/faculty/sawasthi/Statistics/stpowan.html


Logistic regression with high cardinality categorical variable I have a logistic regression model where I care about predictive power solely over comprehensibility. I'm interested in predicting win rates in a video game.

There are 133 characters. Each team picks 5 of them (no repeats). Each of these characters is assigned to one of five positions (again no repeats).

Currently I have each of these characters as a dummy variable. In addition I have an interaction variable between each of these variables. The position of a character is not included in the model at present.

I know I can trim down the size of the model by excluding low-playrate characters, however my concern is that the required sample size is still far too small for the complexity of the model. Any advice would be appreciated.

Sample Size: Aprox. Two million

So I believe you're building a model on the binary outcome {lose, win}:= {0, 1}, correct?

I'd recommend just using a one-hot-encoding or a sparse matrix to store these inputs, then the model should run just fine. This is very straightforward in R (sparse.model.matrix) or Python (pd.get_dummies(sparse=True)).

Here's a quick demo of how to build a sparse matrix in R out of sampled categories and select a subset of them with at least 5 observations.

library(MASS)
require(glmnet)
n <- 1000
x1 <- sample(paste(letters,1), n, replace=T)
x2 <- sample(paste(letters,2), n, replace=T)
x3 <- paste(x1,x2,sep='-')
xdf <- data.frame(x1,x2,x3)
xs <- sparse.model.matrix(~.-1, data=xdf)
vars <- colnames(xs)
colsmry <- colSums(xs)
colsubset <- colsmry > 4
xs_ss <- xs[,vars[colsubset]]
dim(xs)
dim(xs_ss)


What is a good objective function for allowing close to 0 predictions? Let's say we want to predict the probability of rain. So just the binary case: rain or no rain.

In many cases it makes sense to have this in the [5%, 95%] interval. And for many applications this will be enough. And it is actually desired to make the classifier not too confident. Hence cross entropy (CE) is chosen:

$$H_{y'} (y) := - \sum_{i} y_{i}' \log (y_i)$$

But cross entropy practically makes it very hard for the classifier to learn to predict 0. Is there another objective function that does not behave that extreme around 0?

Why it matters

There might be cases where it is possible to give the prediction 0% (or at least something much closer to 0 like $10^{-6}$). Like in a desert. And there might be applications where one needs this (close to) zero predictions. For example, when you want to predict the probability that something happens at least once. If the classifier always predicts at least a 1% chance, then having rain at least once in 15 days is

$$1 - (1-0.05)^{15} \approx 54\%$$

but if the classifier can practically output 0.1% as well, then this is only

$$1 - (1-0.001)^{15} \approx 1.5\%$$

I could also imagine this to be important for medical tests or for videos.

Please be aware that neural networks are normally badly calibrated. Essentially this means that for a binary classification, neural networks are good at keeping the prediction score of a sample in the right area (above 50% or below 50%, depending on the class), but the actual values do not have to represent plausible probabilites of the real world.
There is a lot of research going on about how to calibrate neural networks, e.g. this approach.


Plotting Precision Recall Curve I was wondering mathematically how Precision Recall curve is plotted. How is this curve useful?

In short, the precision-recall curve shows the trade-off between the two values as you change the strictness of the classifier.

There is a great explanation here, using the classification of images of airplanes and geese as an example.

  A good way to characterize the performance of a classifier is to look at how precision and recall change as you change the threshold. A good classifier will be good at ranking actual airplane images near the top of the list, and be able to retrieve a lot of airplane images before retrieving any geese: its precision will stay high as recall increases. A poor classifier will have to take a large hit in precision to get higher recall. Usually, a publication will present a precision-recall curve to show how this tradeoff looks for their classifier.


How to select features from text data? I have a data set of questions belonging to ten different categories namely (definitions, factoids, abbreviations, fill in the blanks, verbs, numerals, dates, puzzle, etymology and category relation).

The categories are briefly described as follows:

Definition – A question that contains a definition of the answer.
Category Relation – The answer has a semantic relation to the question where the relation is specified in the category.
FITB – These are generic fill in the blank questions – some of them ask for the completion of a phrase.
Abbreviation – The answer is an expansion of an abbreviation in the question.
Puzzle – These require derivation or synthesis for the answer.
Etymology – The answer is an English word derived from a foreign word.
Verb – The answer is a verb.
Number – The answer is a numeral.
Date – The question asks for a date or a year.
Factoid – A question is a factoid if its answer can be found on Wikipedia.

I used the Stanford core NLP package called shiftreducer to find out the Part-Of-Speech (POS) values for each question in a category. I thought of using this POS pattern as a discriminant among the classes but it turned out to be generalized since:

All the classes follow a similar pattern
Nouns top the POS count followed by Determinants, Prepositions, Adjectives, Plural nouns and finally verbs.

What could be the other ways in which I could differentiate among the question categories? Or as my question was in its first place, "What kind of features do I select for efficient categorization?"

If I understand you correctly, you're looking to take the text of these questions and train classifiers to identify which of 10 categories they belong to. And you'd like to come up with a decent feature representation in order to do this.

I think your finding about part-of-speech is intuitive. It makes sense that in grammatical English (assuming your question data is written in English), most questions would follow similar part-of-speech sequences since grammatically correct questions follow a particular syntactic form (at least when posed interrogatively as in the case of "When was George Washington born?")

So, you've ruled something out - which you should actually view as progress. If you haven't tried it already, one simple thing you might do is use the actual words within the questions as features. You could use any order n-gram you like, but unigrams stick out as an immediate linguistic feature to try. It seems likely to me that while the POS-tags are similar across classes, making them difficult to distinguish between, the actual words being used in the questions may vary from class to class, giving your model a better shot of differentiating between classes.

That is, maybe words like "time", "year", and "when" co-occur more highly with the Date class while words like "numerical" and "quantity" co-occur more with the Number class (obviously, this is speculation - I haven't seen your data). You might also look at bigrams, trigrams, or any other number n-gram for this feature set as well.

Finally, there may be other features you could generate using NLP methods that may be useful. I'm not familiar with the Shiftreducer software, but Named-Entity Recognition could be helpful in generating features for the Factoid class if there are many questions about proper nouns. Other really simple features such as length of the question (counted in number of tokens). A final thought would be to use only the tagged verbs from your POS-tagger, tab these up and to see whether they differ between classes. This may be a useful feature for identifying questions present in your Verb class. Hopefully, those are some ideas to get you started.


Functions in scikit that detect outliers automatically? I know a way to visualize outliers is to make a box plot, but wanted to know if scikit had any quick ways to detect outliers for each variable?

The scikit-learn implementations of Isolation Forest and Local Outlier Factor should help you out here. Details on both at this link:

https://scikit-learn.org/stable/modules/outlier_detection.html


Relating changes of a value in time to known events I work with two datasets. The first dataset contains fluor values measured every minute. The second dataset contains certain events and their time. We know that these events cause peaks in fluor values shortly before and shortly after the event time.

A simplified reproducible example in R:

Here I provide a simplified version of the R code I use to relate the fluor values to events. I have a series of fluor values measured every minute. Next I have a second dataset with three events A, B and C that occur at three different times. It seems there are three large peaks in fluor value around each event.

The objective is to classify each value as corresponding to an event whenever the time of that value falls within 15 minutes of that time of an event. Fluor values that fall outside of any events are categorized as NA.

Set seed

set.seed(10)

Generate fluor values dataset and events dataset

values <- data.table(date.time.value=seq(ymd_hms("2018-01-01 00:00:00"), by= "min", length.out = 200),fluor=5*sin(seq(0,20,length.out = 200)))+(abs(rnorm(200,6,1))+2)
events <- data.table(event=c("A","B","C"),date.time.event=c(ymd_hms("2018-01-01 00:20:00"),ymd_hms("2018-01-01 01:20:00"),ymd_hms("2018-01-01 02:20:00")))

Using the data.table package: I add an "event" variable to the values dataset. That variable takes on the name of an event (in this case A, B or C) whenever the date.time.value is within 15 minutes of a date.time.event.

values[, event:=events[.SD[, .(d_dn=date.time.value-15*60, d_up=date.time.value+15*60)], on=.(date.time.event>=d_dn, date.time.event<=d_up), event]]

Here's what the result looks like. We see that we have one fluor value peak for each event A, B and C and that the other fluor values have NA as event:

ggplot(values, aes(date.time.value, fluor,col=event)) + geom_point()

A real example:

I managed to relate the two datasets by coloring the fluor values that occur several minutes before or several minutes after an event. This is what the result looks like:

What I'm missing:

I would like to define "peaks" so that in my simplified example all the green-colored fluor values of event B belong to a specific "peak number x". Currently it's impossible to distinguish between fluor values of the same event but from different peaks because I only have observations of fluor with a corresponding variable "event".
In the simplified example you can see (exagerated) that quite a lot of high fluor values are considered to be NA even though their high value is most likely are caused by an event. Hence I'm looking for a better way to relate fluor values to events than simply saying "15 minutes before or 15 minutes after".

I'm almost certain that there must be some kind of R-package that's specifically built for my problem here but I just can't figure out which one...

You seem to be looking for a clustering mechanism for your event sequence data based on a dissimilarity/ distance measure.
I would recommend you to explore TraMineR package.


Segmentation Network produces noisy output I've implemented a SegNet and SegNet ReLU variant in PyTorch. I'm using it as a proof-of-concept for now, but what really bothers me is the noise produced by the network. With ADAM I seem to get slightly less noise, whereas with SGD the noise increases. I can see the loss going down and the cross-evaluation accuracy rising to 98%-99% and yet the noise is still there.

On the left is the actual image, then you can see the mask, and finally the actual output from the network. There's 1024 samples per class, and two classes, which are very consistent as the documents are very structured. I'm using the vanilla SegNet (same kernel, striding and padding) on 224x224.

What could explain this noise, and how could I potentially address the issue?

I'm going to try and attempt to answer my question, but won't accept it as an answer, simply because I'm sure there is more than one reasons as to why this is happening.
I've solved the issue by increasing the areas by adding more "features", e.g., I've made sure there is more text, table boxes and other visual features for the convolutions to "pick up". To a certain degree this seems to have helped a lot.
What also helped, is using a more modern model, I've tried Unet with a resnet34 encoder and I also tried a DeepLabV3 which outperformed all others. So I suspect that the "noise" (for lack of a better word) is a by-product of the network being uncertain where exactly the boundaries of the segmentation are, due to absence of features. I suspect that the more modern models are better suited to deal with that problem.
EDIT
So far what seems to have worked for me is DeepLabV3 with Resnet encoders, but another thing that really seems to make a difference is how "large" the segmented area is. Obviously in some domains this is not possible to adjust (e.g., robotics, autonomous vehicles and suchs) but in document or text processing it most likely is. What I've noticed is that after the downsizing to 224x224 smaller and thinner areas become very hard if not impossible to learn, whereas larger and thicker areas are easier. I suspect that averaging CELoss might be a culprit here, and that there may be a loss mechanism which emphasises errors in smaller regions/areas where errors still exist. The hypothesis is that the smaller regions will tolerate errors due to the averaging operation whereas larger ones won't tolerate the errors as much.


EM-ELM Cross validation I know that cross validation is used to find the best hyperparameters that minimize the average error. For example, the number of neurons that minimize the average error of cross-validation is estimated in ELM.

But I would like to know how I can apply cross-validation (eg K-Fold) in EM-ELM networks:

Feng, G., Huang, G. B., Lin, Q., & Gay, R. (2009). Error minimized extreme learning machine with growth of hidden nodes and incremental learning. IEEE Transactions on Neural Networks, 20 (8), 1352-1357.

Where the architecture is estimated automatically using a training set. What is the best and most standard way to apply cross-validation in EM-ELM? and is it possible to incorporate the cross-validation process into the growth of the network?

Cross-validation can be used to tune any hyperparameter.
One way tune Extreme learning machines (ELM) with cross-validation is to:

Fit a model to a subset of the training dataset. Measure performance on a validation dataset with a evaluation metrics.
Change the number of neurons hyperparameter.
Fit the updated model to a subset of the training dataset. Measure performance on a validation dataset with a evaluation metrics.
Compare the two models and see which model is better on a evaluation metric on their respective validation dataset.
Repeat steps 3-4 until satisfied.


How to install pytorch in windows? I was wondering, if there is any way to install pytorch in windows like the way we can install tensorflow. My machine is not supporting docker.

As of August 14, 2017, you can install Pytorch from peterjc123's fork as follows. Currently, python 3.5 and 3.6 are supported.

# If your main Python version is not 3.5 or 3.6
conda create -n test python=3.6 numpy pyyaml mkl

# for CPU only packages
conda install -c peterjc123 pytorch

# for Windows 10 and Windows Server 2016, CUDA 8
conda install -c peterjc123 pytorch cuda80

# for Windows 10 and Windows Server 2016, CUDA 9
conda install -c peterjc123 pytorch cuda90

# for Windows 7/8/8.1 and Windows Server 2008/2012, CUDA 8
conda install -c peterjc123 pytorch_legacy cuda80

pytorch support for windows Link


How to make custom callback in keras to generate sample image in VAE training? I'm training a simple VAE model on 64*64 images and I would like to see the images generated after every epoch or every couple batches to see the progress.

when I train the model I wait until the training is done and then I look at the results.

I tried to make a custom callback function in Keras that generates an image and saves it but couldn't do it. is it even possible? I couldn't find anything like it.

it would be awesome if you refer me to a source that explains how to do so or show me an example

I think the two following links could help you 1, 2.

The first one is a tutorial, which introduces you how to display images in TensorBoard.
If you look at the part on the confusion matrix, you should find a way to make your desired callback, i.e. a callback which involves displaying an image.

The second link is also a tutorial that shows you how to write your very own custom callback in TensorFlow with Keras.

I think by combining both tutorial you should have a pretty good idea how to do what you want.


hyperparameter tuning with validation set For what I know, and correct me if I am wrong, the use of cross-validation for hyperparameter tuning is not advisable when I have a huge dataset. So, in this case it is better to split the data in training, validation and test set; and then perform the hyperparameter tuning with the validation set.

In the case that I am programming I would like to use scikit, the yeast dataset available at: http://archive.ics.uci.edu/ml/datasets/yeast; and for example to tune the number of epochs.

First, I have separated my training, validation and test set by using the train_test_split twice according to one answer that I saw here. The loss plot that I got is the following for 1500 max iterations:

Then I wanted to use my validation set with a list of different values for the hypeparameter of max iterations. The graph I obtained is the following (with some warning messages of non-convergence for max_iter values less than 1500):

So, I have the first question here. It seems that for a value of max_iter of 3000 the accuracy is 64% approximately, so I should choose that value for the max_iter hyperparameter; is that correct? I can see from the graph that also the red line of 3000 has a less value of loss than the other compared options.

My program so far is the following:

import numpy as np
import pandas as pd
from sklearn import model_selection, linear_model
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
from sklearn.neural_network import MLPClassifier
import matplotlib.pyplot as plt
from sklearn.model_selection import GridSearchCV

def readFile(file):
    head=["seq_n","mcg","gvh","alm","mit","erl","pox","vac","nuc","site"]
    f=pd.read_csv(file,delimiter=r"\s+")
    f.columns=head
    return f

def NeuralClass(X,y):
    X_train,X_test,y_train,y_test=model_selection.train_test_split(X,y,test_size=0.2)
    print (len(X)," ",len(X_train))
    X_tr,X_val,y_tr,y_val=model_selection.train_test_split(X_train,y_train,test_size=0.2)
    mlp=MLPClassifier(activation="relu",max_iter=1500)
    mlp.fit(X_train,y_train)
    print (mlp.score(X_train,y_train))
    plt.plot(mlp.loss_curve_)

    max_iter_c=[500,1000,2000,3000]
    for item in max_iter_c:
        mlp=MLPClassifier(activation="relu",max_iter=item)
        mlp.fit(X_val,y_val)
        print (mlp.score(X_val,y_val))
        plt.plot(mlp.loss_curve_)
    plt.legend(max_iter_c)

def main():
    f=readFile("yeast.data")
    list=["seq_n","site"]
    X=f.drop(list,1)
    y=f["site"]
    NeuralClass(X,y)

Second question, is my approach valid? I have seen a lot of information over the web and all point to cross validation for hyperparameter tuning, but I want to perform it with a validation set.

Any help?

PD. I have tried early stopping and the results are poor compared to the ones obtained with the method I programmed.

Thanks

It seems to me that you're manually iterating through the hyper-parameters.

scikit-learn has a number of helper functions that make it easy to iterate through all of the parameters using various strategies:
https://scikit-learn.org/stable/modules/grid_search.html#grid-search

Secondly, if I was 'manually' tuning hyper-parameters I'd split my data into 3: train, test and validation (the names aren't important)

I'd change my hyper-parameters, train the model using the training data, test it using the test data.  I'd repeat this process until I had the 'best' parameters and then finally run it with the validation data as a sanity check (should have similar scores).

With scikit-learn's helper functions, I just split the data into two parts.  Use GridSearchCV with one part and then at the end using the best parameters (stored in the attribute best_estimator_) run a sanity check with the second part.

# define parameter sweep
param_grid = { 'max_iter' : [100, 1000, 10000] }

# define grid search
clf = GridSearchCV(mlp, param_grid, cv=5)

# perform search
clf.fit(X, y)

# best estimator
clf._best_estimator

For info, every estimator and their respective scores are also available as attributes (see https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV) which means you can check the result of every CV if you wanted to make sure you got the best parameters.

As you can see you can avoid thinking too hard about whether CV was the right approach by using a final validation set as a sanity check.


Is Minimax Linkage a Lance-Williams hierarchical clustering? I found the following article on "Hierarchical Clustering With Prototypes
via Minimax Linkage".

It is stated in Property 6 that

  Minimax linkage cannot be written using Lance–Williams updates.

A succinct proof using a counter-example is given:

  Proof.
  Figure 9 shows a simple one-dimensional example that could not
  arise if minimax linkage followed Lance– Williams updates. The upper
  and lower panels show two configurations of points for which the
  right side of (4) is identical but the left side differs; in
  particular, $d(G_1 \cup G_2,H) = 9$ for the upper panel, whereas
  $d(G_1 \cup G_2,H) = 8$ for the lower panel.

But I do not understand their proof. For both cases (upper and lower panels), $d(G_1,H) = 16$, $d(G_2,H) = 7$, $d(G_1,G_2) = 5$.

I cannot see any reason that $\alpha(G_2)$ in the first case must equal $\alpha(G_2)$ in the second case. For instance, $G_2$ has not the same cardinal.

My take after a quick read of the references.
First of all, Lance and Williams's original paper mentions that their linear scheme works (and offers computational advantage) only for combinatorial strategies. Is minimax linkage such a combinatorial strategy? In other words, does it depend (linearly) on pair-wise distances? By the defintion of minimax distance it is clear that this is implausible.
It is like the difference between mean and median computation in statistics. Mean is linear while Median is non-linear. There is no linear combination that can compute median from mean (although in certain limited cases they can coincide).
Second, the authors do not mention the form (or values) of the $\alpha, \beta, \gamma$ parameters in a hypothetical Lance-Williams method for minimax linkage. But in any case they are constants and $\alpha(\cdot)$ can be constant or rational functions of the respective cluster sizes (per the original Lance-Williams reference).
$G_2$ might not be of the same cardinality in the two panels, but minimax linkage distance depends on radius of cluster not cardinality (unlike average or centroid linkage) and the two examples have same radius thus $\alpha(G_2)$ is same in both cases.
Another way to see that this is the case is a plausible variation of the proof where $G_2$ in both cases has both same radius and same cardinality but different configuration.

Maybe such a proof would make it more clear. But I will leave it at that, at this point.


What are "downstream models"? In the ResNeSt paper they say on page 4:
"despite their great success in image classification, the meta network structures are distinct from each other, which makes it hard for downstream models to build upon."
What are downstream models in this context?
Paper can be found at https://arxiv.org/abs/2004.08955

Downstream models are simply models that come after the model in question, in this case ResNet variants. Models for various topics within the computer vision domain often use a backbone to extract features from images, after which a downstream model is used to help to fit the model better to the task at hand. Tables 5, 6, and 7 in the linked paper give a good overview of the different ways backbones are often combined for topics such as object detection and segmentation.


Url string processing: what is the best way? I have ~1000 different news websites and I scraped and saved all the internal url links for each website. For instance, the website dcgazette.com has a 2MB text file with associated urls:

1) https://dcgazette.com/writers-wanted/

2) https://dcgazette.com/2019/fifteen-things-that-caught-my-eye-today-june-20-2019/

3) https://dcgazette.com/page/494/

4) https://dcgazette.com/page/3/

5) https://dcgazette.com/2019/trump-has-reason-to-be-concerned-about-2020/

etc

Notice that urls 2 and 5 are associated with actual articles. Across websites, each has a different way of representing their article urls. But within each website, all article urls have the same format, i.e. "domain/date/title" (for dcgazette.com).

Is there any unsupervised learning algorithm to process these url strings such that I can cluster all article urls together for each domain? The result I am aiming for is extracting all article urls for each domain with as little noise as possibe from non-article urls for all these domains.

An idea I had was separating each url into pieces by splitting on "/" and generating features such as length or type of letters used for each piece and using a k-means clustering algorithm. I'm not sure how this will actually perform however and am open to advice on any methodology that can make this as accurate as possible

I don't think you need any ML, in the best case it's going to be very slow compared to direct processing and it's very likely to cause errors.

For each address you can parse the url string and extract the domain substring (in your example they are all prefixes, if it's always the case it's very easy). Then you just group them by domain in a map... done.


Large Graphs: NetworkX distributed alternative I have built some implementations using NetworkX(graph Python module) native algorithms in which I output some attributes which I use them for classification purposes.

I want to scale it to a distributed environment. I have seen many approaches like neo4j, Graphx, GraphLab. However, I am quite new to this, thus I want to ask, which of them would be easy to locally apply graph algorithms (ex. node centrality measures), preferably using Python. To be more specific, which available option is closer related to NetworkX (easy installation, premade functions/algorithms, ML wise)?

Good , old and unsolved question! Distributed processing of large graphs as far as I know (speaking as a graph guy) has 2 different approaches, with the knowledge of Big Data frameworks or without it.

SNAP library from Jure Leskovec group at  Stanford which is originally in C++ but also has a Python API (please check if you need to use C++ API or Python does the job you want to do). Using snap you can do many things on massive networks without any special knowledge of Big Data technologies. So I would say the easiest one.

Using Apache Graphx is wonderful only if you have experience in Scala because there is no Python thing for that. It comes with a large stack of built in algorithms including centrality measures. So the second easiest in case you know Scala.

Long time ago when I looked at GraphLab it was commercial. Now I see it goes open source so maybe you know better than me but from my out-dated knowledge I remember that it does not support a wide range of algorithms and if you need an algorithm which is not there it might get complicated to implement. On the other hand it uses Python which is cool. After all please check it again as my knowledge is for 3 years ago.

If you are familiar with Big Data frameworks and working with them, Giraph and Gradoop are 2 great options. Both do fantastic jobs but you need to know some Big Data architecture e.g. working with a hadoop platform.

PS

1) I have used simple NetworkX and multiprocessing to distributedly process DBLP network with 400,000 nodes and it worked well, so you need to know HOW BIG your graph is.

2) After all, I think SNAP library is a handy thing.


Confusion matrix in multilabel classification of an object in more than one class simultaneously Regarding a classification problem where for example given an image which depicts a human and we are trying to predict their stance and their behavior. For example Human 1: 'Sitting' and 'Eating' in the first image whilst Human 2: 'Standing up' and 'laughing' in the second. What is the appropriate way of applying the confusion matrix on the predictions. Do I have to unify the predictions?

E.g we have 5 different stances and 5 different behaviors, as a result the confusion matrix is of size 25x25 because we have 25 different classes. Or is there any other way of dealing with such problems?

Is it possible to do the same with multiple objects on an image and how?

Confusion matrix is generally not considered as a useful tool to evaluvate our model for multiclass classification and we rather use what is known as AUC-ROC curve.

AUC stands for Area Under Curve whereas ROC stands for Reciever Operating Curve. AUC - ROC curve is a performance measurement for classification problem at various thresholds settings. ROC is a probability curve and AUC represents degree or measure of separability. It tells how much model is capable of distinguishing between classes.

Here is a detailed explaination about the AUC-ROC curve.


Propensity Modeling, still use Test/Train Split? I'm using Sklearn to build a classifier in which my client wants a predicted probability for each row of data.  The default of let's say Random Forest is if > 50%, then classifies as TRUE, but using the predict_proba function I'm able to get the probability.

The data I'm given has 10k rows, which all are labeled TRUE or FALSE.  If it's my job to provide a predicted probability by row, should I still use a 70/30 Train/Test split.  In which I create my best model on the Train 70% using a 10-fold CV.  But then when I need to output for all 10k rows, I would actually be make a prediction on the training data and the test data, which doesn't seem correct to predict on training data.  Any thoughts on how to approach this?

You are right; you shouldn't predict on the same data you have used for training your model. If your goal is to only output probabilities and you don't mind achieving this by having not a single classifier but a number of them, you can potentially use nested cross-validation to achieve that.

In the outer cross-validation steps, you break your data in $N_{out}$ folds, where one fold is kept for testing the model and the remaining $N_{out} - 1$ folds are used to train. You then feed this training dataset (which is similar, in nature, to that 70% you have mentioned in OP) to the inner cross-validation step with $N_{in}$ folds. Here, you will compare different models and choose the best performing one. This chosen model is then passed to the outer fold (see above) and can be used to obtain probabilities for the 1 fold that was left out. You then repeat the same procedure for other folds in the outer fold. At the end of this nested cross-validation process, you will have probabilities for all your rows (but they will have come from $N_{out}$ different classifiers, each corresponding to one of the outer folds).

Note that the ultimate purpose of nested cross-validation is not to do this but it will give you what you want as a by-product.


Nested cross-validation and selecting the best regression model - is this the right SKLearn process? If I understand correctly, nested-CV can help me evaluate what model and hyperparameter tuning process is best. The inner loop (GridSearchCV) finds the best hyperparameters, and the outter loop (cross_val_score) evaluates the hyperparameter tuning algorithm. I then choose which tuning/model combo from the outer loop that minimizes mse (I'm looking at regression classifier) for my final model test.

I've read the questions/answers on nested-cross-validation, but haven't seen an example of a full pipeline that utilizes this. So, does my code below (please ignore the actual hyperparameter ranges - this is just for example) and thought process make sense?

from sklearn.cross_validation import cross_val_score, train_test_split
from sklearn.grid_search import GridSearchCV
from sklearn.metrics import mean_squared_error
from sklearn.ensemble import RandomForestRegressor
from sklearn.svm import SVR
from sklearn.datasets import make_regression

# create some regression data
X, y = make_regression(n_samples=1000, n_features=10)
params = [{'C':[0.01,0.05,0.1,1]},{'n_estimators':[10,100,1000]}]

# setup models, variables
mean_score = []
models = [SVR(), RandomForestRegressor()]

# split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.3)

# estimate performance of hyperparameter tuning and model algorithm pipeline
for idx, model in enumerate(models):
    clf = GridSearchCV(model, params[idx], scoring='mean_squared_error')

    # this performs a nested CV in SKLearn
    score = cross_val_score(clf, X_train, y_train, scoring='mean_squared_error')

    # get the mean MSE across each fold
    mean_score.append(np.mean(score))
    print('Model:', model, 'MSE:', mean_score[-1])

# estimate generalization performance of the best model selection technique
best_idx = mean_score.index(max(mean_score)) # because SKLearn flips MSE signs, max works OK here
best_model = models[best_idx]

clf_final = GridSearchCV(best_model, params[best_idx])
clf_final.fit(X_train, y_train)

y_pred = clf_final.predict(X_test)
rmse = np.sqrt(mean_squared_error(y_test, y_pred))

print('Final Model': best_model, 'Final model RMSE:', rmse)

Yours is not an example of nested cross-validation.

Nested cross-validation is useful to figure out whether, say, a random forest or a SVM is better suited for your problem. Nested CV only outputs a score, it does not output a model like in your code.

This would be an example of nested cross validation:

from sklearn.datasets import load_boston
from sklearn.cross_validation import KFold
from sklearn.metrics import mean_squared_error
from sklearn.grid_search import GridSearchCV
from sklearn.ensemble import RandomForestRegressor
from sklearn.svm import SVR
import numpy as np

params = [{'C': [0.01, 0.05, 0.1, 1]}, {'n_estimators': [10, 100, 1000]}]
models = [SVR(), RandomForestRegressor()]

df = load_boston()
X = df['data']
y = df['target']

cv = [[] for _ in range(len(models))]
for tr, ts in KFold(len(X)):
    for i, (model, param) in enumerate(zip(models, params)):
        best_m = GridSearchCV(model, param)
        best_m.fit(X[tr], y[tr])
        s = mean_squared_error(y[ts], best_m.predict(X[ts]))
        cv[i].append(s)
print(np.mean(cv, 1))

By the way, a couple of thoughts:

I see no purpose to grid search for n_estimators for your random forest. Obviously, the more, the merrier. Things like max_depth is the kind of regularization that you want to optimize. The error for the nested CV of RandomForest was much higher because you did not optimize for the right hyperparameters, not necessarily because it is a worse model.
You might also want to try gradient boosting trees.


Sourcing (discounted) products customers want Goal: Generate a list of 100 products per vertical (e.g. fashion, electronics) that the teams should source, discount, and list on the website over a specific period. You may assume all customers are online only.

My thinking so far:

Predict the customers that will come to the website during the
specific period (time series). Only 30-40% customers return YOY.
Understand what they want (use search data, add to basket but didn't checkout etc). Potentially segment it further by looking at those customers who generate the highest revenue in general vs one off purchasers.
Further filter those products that these customers add (or take from 'viewed or saved' state to the 'checkout' state) once the product is on deal.
Potentially use clustering to recommend products similar to those from step 3 but that have never been on a deal?

I cannot influence the amount of discount. I can only influence what products we source. Therefore I want to source the ones with the highest potential of purchasers from the customer.

Any thoughts on the approach above?

Clustering is not well suited for prediction.

Use a recommender system instead.


How is computed the train data set score and why? In a lot of machine learning blogs or review, the training dataset accuracy (or other metric) is given alongside the test dataset score. Is this score calculated through the training, or is the training data given at the final model to see if it gives a good score ?

Moreover, what is the point of computing such a score, as the model has been trained with this data ?

Overfitting

Computing the training score and the test score is usefull to detect overfitting

for exemple here we can see that the training score (in blue) keep improving but at some point the test score (in red) stop improving, that's the sign of overfitting.

But for a final model the training score is not usefull and only the test score should be used.


Plotting Stacked Histogram for Time-series data Given the dataset:

timestamp                     item       itemcount
2019-03-18 07:40:08.759        A          10
2019-03-18 08:40:08.759        B          5
..................................................
2019-05-20 07:40:08.759        D          4
2019-05-21 07:40:08.759        E          8

I want to plot stacked histogram like:

where the x-axis should be the date and y axis the itemcount and stack will be each item. I want the graph with subplots for every month.

I am new here so will be happy to get any feedback on my mistakes. Thank you.

here's one sample code i found online which plots the same graph in the figure above.

  # Import Data
df = pd.read_csv("https://github.com/selva86/datasets/raw/master/mpg_ggplot2.csv")

# Prepare data
x_var = 'manufacturer'
groupby_var = 'class'
df_agg = df.loc[:, [x_var, groupby_var]].groupby(groupby_var)
vals = [df[x_var].values.tolist() for i, df in df_agg]

# Draw
plt.figure(figsize=(16,9), dpi= 80)
colors = [plt.cm.Spectral(i/float(len(vals)-1)) for i in range(len(vals))]
n, bins, patches = plt.hist(vals, df[x_var].unique().__len__(), stacked=True, density=False, color=colors[:len(vals)])

# Decoration
plt.legend({group:col for group, col in zip(np.unique(df[groupby_var]).tolist(), colors[:len(vals)])})
plt.title(f"Stacked Histogram of ${x_var}$ colored by ${groupby_var}$", fontsize=22)
plt.xlabel(x_var)
plt.ylabel("Frequency")
plt.ylim(0, 40)
plt.xticks(ticks=bins, labels=np.unique(df[x_var]).tolist(), rotation=90, horizontalalignment='left')
plt.show()

This solution uses plotnine which is based on R ggplot and the grammar of graphics.

Generally you have to transform the timestamp to months first and then understand that:

X = Months
Y = itemcount
Group = item

This should help also for your matplotlib solution!

from datetime import datetime
import pandas as pd
from plotnine import *

df['month'] = pd.DatetimeIndex(df['timestamp']).month

p = (ggplot(pdf, aes(x = 'month', y = 'itemcount',fill='item')) + geom_bar( Star = 'identity',position='fill')

# fill here acts the same as group in matplotlib
```