Does gradient descent always converge to an optimum? I am wondering whether there is any scenario in which gradient descent does not converge to a minimum.

I am aware that gradient descent is not always guaranteed to converge to a global optimum. I am also aware that it might diverge from an optimum if, say, the step size is too big. However, it seems to me that, if it diverges from some optimum, then it will eventually go to another optimum.

Hence, gradient descent would be guaranteed to converge to a local or global optimum. Is that right? If not, could you please provide a rough counterexample?

Gradient Descent is an algorithm which is designed to find the optimal points, but these optimal points are not necessarily global. And yes if it happens that it diverges from a local location it may converge to another optimal point but its probability is not too much. The reason is that the step size might be too large that prompts it recede one optimal point and the probability that it oscillates is much more than convergence.

About gradient descent there are two main perspectives, machine learning era and deep learning era. During machine learning era it was considered that gradient descent will find the local/global optimum but in deep learning era where the dimension of input features are too much it is shown in practice that the probability that all of the features be located in there optimal value at a single point is not too much and rather seeing to have optimal locations in cost functions, most of the time saddle points are observed. This is one of the reasons that training with lots of data and training epochs cause the deep learning models outperform other algorithms. So if you train your model, it will find a detour or will find its way to go downhill and do not stuck in saddle points, but you have to have appropriate step sizes.

For more intuitions I suggest you referring here and here.


How to do stepwise regression using sklearn? I could not find a way to stepwise regression in scikit learn. I have checked all other posts on Stack Exchange on this topic. Answers to all of them suggests using f_regression.

But f_regression does not do stepwise regression but only give F-score and pvalues corresponding to each of the regressors, which is only the first step in stepwise regression.

What to do after 1st regressors with the best f-score is chosen?

Scikit-learn indeed does not support stepwise regression. That's because what is commonly known as 'stepwise regression' is an algorithm based on p-values of coefficients of linear regression, and scikit-learn deliberately avoids inferential approach to model learning (significance testing etc). Moreover, pure OLS is only one of numerous regression algorithms, and from the scikit-learn point of view it is neither very important, nor one of the best.

There are, however, some pieces of advice for those who still need a good way for feature selection with linear models:

Use inherently sparse models like ElasticNet or Lasso.
Normalize your features with StandardScaler, and then order your features just by model.coef_. For perfectly independent covariates it is equivalent to sorting by p-values. The class sklearn.feature_selection.RFE will do it for you, and RFECV will even evaluate the optimal number of features.
Use an implementation of forward selection by adjusted $R^2$ that works with statsmodels.
Do brute-force forward or backward selection to maximize your favorite metric on cross-validation (it could take approximately quadratic time in number of covariates). A scikit-learn compatible mlxtend package supports this approach for any estimator and any metric.
If you still want vanilla stepwise regression, it is easier to base it on statsmodels, since this package calculates p-values for you. A basic forward-backward selection could look like this:

```

from sklearn.datasets import load_boston
import pandas as pd
import numpy as np
import statsmodels.api as sm

data = load_boston()
X = pd.DataFrame(data.data, columns=data.feature_names)
y = data.target

def stepwise_selection(X, y,
                       initial_list=[],
                       threshold_in=0.01,
                       threshold_out = 0.05,
                       verbose=True):
    """ Perform a forward-backward feature selection
    based on p-value from statsmodels.api.OLS
    Arguments:
        X - pandas.DataFrame with candidate features
        y - list-like with the target
        initial_list - list of features to start with (column names of X)
        threshold_in - include a feature if its p-value < threshold_in
        threshold_out - exclude a feature if its p-value > threshold_out
        verbose - whether to print the sequence of inclusions and exclusions
    Returns: list of selected features
    Always set threshold_in < threshold_out to avoid infinite looping.
    See https://en.wikipedia.org/wiki/Stepwise_regression for the details
    """
    included = list(initial_list)
    while True:
        changed=False
        # forward step
        excluded = list(set(X.columns)-set(included))
        new_pval = pd.Series(index=excluded)
        for new_column in excluded:
            model = sm.OLS(y, sm.add_constant(pd.DataFrame(X[included+[new_column]]))).fit()
            new_pval[new_column] = model.pvalues[new_column]
        best_pval = new_pval.min()
        if best_pval < threshold_in:
            best_feature = new_pval.argmin()
            included.append(best_feature)
            changed=True
            if verbose:
                print('Add  {:30} with p-value {:.6}'.format(best_feature, best_pval))

        # backward step
        model = sm.OLS(y, sm.add_constant(pd.DataFrame(X[included]))).fit()
        # use all coefs except intercept
        pvalues = model.pvalues.iloc[1:]
        worst_pval = pvalues.max() # null if pvalues is empty
        if worst_pval > threshold_out:
            changed=True
            worst_feature = pvalues.argmax()
            included.remove(worst_feature)
            if verbose:
                print('Drop {:30} with p-value {:.6}'.format(worst_feature, worst_pval))
        if not changed:
            break
    return included

result = stepwise_selection(X, y)

print('resulting features:')
print(result)

This example would print the following output:

Add  LSTAT                          with p-value 5.0811e-88
Add  RM                             with p-value 3.47226e-27
Add  PTRATIO                        with p-value 1.64466e-14
Add  DIS                            with p-value 1.66847e-05
Add  NOX                            with p-value 5.48815e-08
Add  CHAS                           with p-value 0.000265473
Add  B                              with p-value 0.000771946
Add  ZN                             with p-value 0.00465162
resulting features:
['LSTAT', 'RM', 'PTRATIO', 'DIS', 'NOX', 'CHAS', 'B', 'ZN']


Predict compatibility of 2 people as boolean classification problem How can I predict the compatibility of 2 people as a boolean classification problem?

I want to know if below is an appropriate approach to modelling compatibility, or if I should be using "market basket analysis" or some other approach instead?

I'm less interested in the specific result below, and more interested in if this is a realistic way to frame this data science problem.

Background:

Assume people only have 3 attributes: compassion, extroversion and humor. These are also boolean and can be modelled as 1s and 0s in a list ([compassion, extroversion, humor]).

So someone with all 3 characteristics would be [1,1,1] and someone with only humor would be [0,0,1].

We have pairs of people who match and do not match, specified by 1 or 0, where 1=match and 0=no_match.

How to solve this?

I don't consider this a simple distance problem (ie: euclidean distance) because its very possible that generally an introvert and extrovert get along, but 2 extroverts don't.

Data:

person1   person2     match?
--------  --------    ------
([1,1,0], [1,0,1]) => 1
([0,0,0], [1,1,1]) => 1
([1,0,1], [1,0,0]) => 0
([1,1,1], [0,1,0]) => 0
([0,0,0], [0,1,1]) => 1
([1,1,0], [1,1,1]) => 0
([1,0,0], [1,0,1]) => 0
([0,0,1], [0,0,0]) => 0
([0,0,0], [0,0,1]) => 0
([0,0,0], [0,1,1]) => 1
([0,1,0], [0,1,1]) => 0
([0,1,0], [0,1,1]) => 0
([0,1,0], [1,0,0]) => 1

What I've tried:

My first thought was to concatenate both individuals' data for each example. Then use that to fit the model.

Data structured as python code:

X_train = [
    [1,1,0,1,0,1],
    [0,0,0,1,1,1],
    [1,0,1,1,0,0],
    [1,1,1,0,1,0],
    [0,0,0,0,1,1],
    [1,1,0,1,1,1],
    [1,0,0,1,0,1],
    [0,0,1,0,0,0],
    [0,0,0,0,0,1],
    [0,0,0,0,1,1],
    [0,1,0,0,1,1],
    [0,1,0,0,1,1],
    [0,1,0,1,0,0],
]

y_train = [1,1,0,0,1,0,0,0,0,1,0,0,1]

X_test = [
    [0,1,1,0,0,0],
    [1,1,0,1,0,1],
    [1,0,0,1,0,0],
    [0,0,0,1,0,0],
    [0,1,0,0,0,0],
    [0,0,0,0,0,0],
]

y_test = [1,1,0,0,1,0]

Computing the match:

from sklearn.metrics import classification_report

from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier()
rf.fit(X_train, y_train)
y_pred = rf.predict(X_test)

print(classification_report(y_test,y_pred))

             precision    recall  f1-score   support

          0       0.60      1.00      0.75         3
          1       1.00      0.33      0.50         3

avg / total       0.80      0.67      0.62         6

I'm less concerned about the results here and more interested in if this is a proper way to frame this problem.

Can you offer a suggestion?

Your approach to modeling compatibility seems sound, and definitely makes more sense than market basket analysis. From what I could gather, your goal is to predict compatibility, and you have access to a certain signal that you can use as your compatibility label for the training set (for example, you know for a fact that persons X and Y are compatible but Y and Z are not). In these circumstances, supervised learning definitely sounds appropriate.

Based on the example you gave, there are a few things about this problem that you may consider (if you haven't already):

Should the features (e.g. compassion, extroversion, humor) be binary, or on a continuous spectrum? This depends on your data, of course, but perhaps you may get a better measure of compatibility if you can get the finer grain of detail that continuous features provide.
Same for the class: is it exclusively binary (classification) or could this be treated as a regression problem (predict the degree of compatibility)? Even if treating it as classification, certain models output class posterior probabilities, which could be interpreted as degrees of compatibility.
Compatibility is most likely a symmetrical relationship: if a is compatible with b, then b is compatible with a. If your dataset is formed of the two individuals' data concatenated, should you create two examples per pair (one in each order)? Is there another way to enforce symmetry in your model? Also, can the signal you use for your compatibility label really be interpreted in this symmetrical sense, or does it only tell you that, for instance person A likes person B, but not necessarily the other way around.
How will you evaluate your model? If compatibility is more continuous than discrete, for example, then perhaps recall, precision and accuracy aren't the best metrics, and you may want to use something like RMSE?


plot the histogram of purchases i have a dataframe with 34154695 obs. in a dataset a Class variable with value 0 indicate "not purchased" and 1 indicate "purchase".

> str(data)
'data.frame':   34154695 obs. of  5 variables:
 $ SessionID: int  1 1 1 2 2 2 2 2 2 3 ...
 $ Timestamp: Factor w/ 34069144 levels "2014-04-01T03:00:00.124Z",..: 1452469 1452684 1453402 1501801 1501943 1502207 1502429 1502569 1502932 295601 ...
 $ ItemID   : int  214536500 214536506 214577561 214662742 214662742 214825110 214757390 214757407 214551617 214716935 ...
 $ Category : Factor w/ 339 levels "0","1","10","11",..: 1 1 1 1 1 1 1 1 1 1 ...
 $ Class    : Factor w/ 2 levels "0","1": 1 1 1 1 1 1 1 1 1 1 ...

I am facing difficulties finding a solution to plot a histogram of the number of purchase per week, per day and time wise purchase based on class value = 1 and wanna output like this.

Could someone please inform how I should proceed??
Really, thank you for any help and suggestings.

Kind regards

I think first of all, if you want to get aggregated data, you need to group it by day/week, like this:
library(dplyr)
library(lubridate)

x <- strptime(data$Timestamp, format = "%Y-%m-%d")#assume you need only days/month , assign to a variable, because dplyr has problems with with date type.

data$month <- month(x) #get month from date obj.

month_summ <- data %>% group_by(month) %>%  #group by month and calculated sold items per month
  summarise(
    total_sales = n()
  )

library(ggplot2)

ggplot(data=month_summ, aes(x=month, y=total_sales)) +   geom_bar(stat="identity") #plot the histogram

This should do the work for you or act like a starting point. Here is a good reference for ggplot2 bar plots
Hope this helps!


In a directed graph, how to measure whether a node is more "upstream" or "downstream"? Let's say that I have a directed graph, and for each node I want to have some measure of whether it is more of an "upstream" node (that lies at the beginning of many paths), or a "downstream" node (to which many paths converge).

Say, in a sequence A->B->C I want A to have the highest value, and C the lowest value, while B would be "in between". In a perfect cycle A->B->C->A I want all nodes to have the same "in between" value. The metrics should tolerate a mix of cycles and acyclic elements.

Ideally, it would also give more extreme values to nodes from longer acyclic paths: higher values to nodes with larger estuaries (the 'mother of all' node), and lower values to nodes with larger watersheds (the 'all roads lead to Rome' node that integrates all flows).

And also ideally it should generalize to weighted graphs. And, ideally, in a way that does not depend on the scaling factor for weights (so if you multiply all weights in the graph by 2, the metrics for each node shouldn't change, as the topology clearly didn't change with this scaling).

For a non-weighted graph I can come with a metrics like that: find all predecessors of a node, find all children of a node, then divide the number of children to the sum of children and ancestors; this value would be 1 for any point of origin; would give 0 for dead-ends, and would give 0.5 for cycles. So it's mostly OK. What I don't like about this metrics is that 1) it does not care about the length of the path, 2) it will be computationally slow, 3) I don't know how to generalize it to weighted graphs in a scale-invariant way.

So I was wondering whether there's a known metrics with approximately these properties that was described and studied before. It feels like a logical thing to calculate that many people would use when analyzing social networks, for example; so it feels like it should have a name and published algorithms. Thank you!

Edit: I think it's fair to say that the pagerank metrics has many of the properties I described (with the values reversed): sinks are high, sinks with larger watersheds are higher, nodes of origin are low, cycles tend to have "in-between" values, and the algorithm clearly supports weighted graphs. The part it does not care about is whether a node of origin has a large estuary or no estuary at all. Now I'm wondering whether I actually need 2 metrics: one page-rank, for watersheds, and a different one for estuaries. Like a weighted share of nodes visited by random walks initiated in the node of interest, or something like that. Or are there simpler metrics?

Please clarify what you are looking for in the presence of cycles.  I'd assume the cycles aren't standalone- any node in a cycle could also have inputs from further upstream, and could also send output further downstream. Even in the unweighted case, it seems to me that some nodes in a cycle can be more "upstream" than others.  For instance, most of the nodes in the cycle might be downstream of a set of source nodes.  One node in that cycle might be the only path out of the cycle to a sink.  Surely that node is more "downstream" than the others in the cycle?

Sorry the above isn't just a comment - I'm too new to have high enough reputation in this group for that.

So, to give an answer as well.  Take the "flow network" analogy to an extreme.  Imagine each source upstream of the target node puts out a water flow, with dye added at the first instant only. Treat the weights as time delays.  For each node, calculate the first time the dye reaches a given node.  Among all the sinks downstream of your target node, pick either the longest time or the mean time (or itself, if the node is a sink).  The target node's position is then just its early arrival time divided by the sinks' mean or longest arrival time.  Computationally, this earliest dye arrival time is done by some modification of breadth first traversal of the directed graph.  Start at the sources with arrival times of 0, and then each node arrival time is just the minimum over all its predecessors.  (As in any graph traversal, visited and completed nodes have to be remembered so cycles aren't followed more than once. This would also be called shortest distance from a set of nodes to a given set of nodes.

But note that cycles aren't given any particular special treatment.   If you really want cycles to matter, you could assume that output flows are split equally, and do mixing calculations, so that the dye concentration would decay gradually over time.  But that would seem ridiculously complex, and take time, for probably no benefit.  But in either case, the nodes in cycles won't all have the same value - some will be more "upstream" than others.

If you really want all the nodes in a cycle to have the same value, you might need a separate pre-processing step to find the directed cycles, and aggregate each cycle into a single node (one for each cycle).  That adds complexity, although it simplifies the shortest distance calculations since then the graph will be acyclic and a trivial breadth first traversal will work.

You probably don't really want to require a global analysis of a graph to get your feature value, although that's what you had to do in your initial analysis too.  If this is something you would run on a regular basis, you could probably just do local updating when changes occur, with limited propagation of earliest arrival times, assuming you could tolerate some "error" in the somewhat arbitrarily-chosen metric anyway.


How to train Ner Model having 1 entity? I am Creating a Custom NER (named entity recognition ) Model using bi directional LSTM and CRF.

During Study on Ner i see all example includes Multiple entities per sentence.
For eample this sentence includes 2 entities

  (jhon lives in Us)  jhon = S-Per , US=S-Country

Question 1:

  Can we Create a model using (bi lstm crf) where we only want to predict 1 entity.?

Question 2:

  In CRF States of the neighbors affect the current prediction so
  predicting 1 entity per sentence seems difficult specially  with CRF?

Question 3 :

  if i Cannot achieve this with CRF can I use Bert to train a model
  having 1 entity per model?

Thanks In advance.

Can we Create a model using (bi lstm crf) where we only want to predict 1 entity.?

[edited] Yes, the trick I would use is to train a regular model and predict the top 10 most likely predictions for each sentence. This will give you the different possible labelings ordered by probability. The idea is to select the first labeling which contains a single entity, i.e. the most likely option after eliminating the predictions with several entities.

  In CRF States of the neighbors affect the current prediction so predicting 1 entity per sentence seems difficult specially with CRF?

There's a confusion here: this is true but the "neighbors" are not other entities, they are the other words of the sentence. This means that if a word belongs to an entity then the next word is more likely to belong to an entity as well (i.e. to be part of the same entity). So using CRF doesn't increase the probability to find several distinct entities in a sentence. However if all the target entities are single words then CRFs might not be needed (I'm not sure whether there would any better alternative though).

  if i Cannot achieve this with CRF can I use Bert to train a model having 1 entity per model?

I don't know the answer to this one but I doubt it's needed.


Word Embedding for Item Names(integer, one-hot encoding) I am looking for the way to get the similarity between two item names using integer encoding or one-hot encoding.
For example, "lane connector" vs. "a truck crane".
I have 100,000 item names consisting of 2~3 words as above.
also, items have its size(36mm, 12M, 2400*1200...) and unit(ea, m2, m3, hr...)
I wanna make (item name, size, unit) as a vector. To do this, I need to change texts to numbers using some way. All I found is only word2vec things, but my case has no context corpus. So I don't think it is possible to learn some context from my data.

I'm not sure, if it's possible with this data set. Word2Vec is used to generate word embedding, which works on the principle of "words association" in a sentence.

So I dont think you can apply Word2Vec on this dataset which looks like doesn't have any association, except on some places where you can match(perform clustering) some parameters like:

Units
Size/dimension of the item-name

Interested to know some solution for such types of problems.


What is the effect of KL divergence between two Gaussian distributions as a loss function in neural networks? In many deep neural networks, especially those based on VAE architecture, a KL divergence term is added to the loss function. The divergence is computed between the estimated Gaussian distribution and prior. Since Gaussian distribution is completely specified by mean and co-variance, only those two parameters are estimated by the neural network. For Gaussian distributions, KL divergence has a closed form solution. By minimizing KL divergence, we bring the estimated distribution closer to the prior.

My question is, since Gaussian distribution is completely specified by mean and co-variance, why don't we just take MSE between estimated parameters and prior parameters? Minimizing MSE between mean and co-variance also brings the two distributions closer. Does taking KL divergence have any significance?

It's too strong of an assumption (I am answering generally, I am sure you know. Coming to VAE later in post), that they are Gaussian.

You can not claim that distribution is X if Moments are certain values. I can bring them all to the same values using this.

Hence if you can not make this assumption it is cheaper to estimate KL metric

BUT with VAE you do have information about distributions, encoders distribution is $q(z|x)=\mathcal{N}(z|\mu(x),\Sigma(x))$ where $\Sigma=\text{diag}(\sigma_1,\ldots,\sigma_n)$, while the latent prior is given by $p(z)=\mathcal{N}(0,I)$. Both are multivariate Gaussians of dimension $n$, for which in general the KL divergence is:
$$
\mathfrak{D}_\text{KL}[p_1\mid\mid p_2] =
\frac{1}{2}\left[\log\frac{|\Sigma_2|}{|\Sigma_1|} - n + \text{tr} \{ \Sigma_2^{-1}\Sigma_1 \} + (\mu_2 - \mu_1)^T \Sigma_2^{-1}(\mu_2 - \mu_1)\right]
$$
where $p_1 = \mathcal{N}(\mu_1,\Sigma_1)$ and $p_2 = \mathcal{N}(\mu_2,\Sigma_2)$.

In the VAE case, $p_1 = q(z|x)$ and $p_2=p(z)$, so $\mu_1=\mu$, $\Sigma_1 = \Sigma$, $\mu_2=\vec{0}$, $\Sigma_2=I$. Thus:
\begin{align}
\mathfrak{D}_\text{KL}[q(z|x)\mid\mid p(z)]
&=
\frac{1}{2}\left[\log\frac{|\Sigma_2|}{|\Sigma_1|} - n + \text{tr} \{ \Sigma_2^{-1}\Sigma_1 \} + (\mu_2 - \mu_1)^T \Sigma_2^{-1}(\mu_2 - \mu_1)\right]\\
&= \frac{1}{2}\left[\log\frac{|I|}{|\Sigma|} - n + \text{tr} \{ I^{-1}\Sigma \} + (\vec{0} - \mu)^T I^{-1}(\vec{0} - \mu)\right]\\
&= \frac{1}{2}\left[-\log{|\Sigma|} - n + \text{tr} \{ \Sigma \} + \mu^T \mu\right]\\
&= \frac{1}{2}\left[-\log\prod_i\sigma_i^2 - n + \sum_i\sigma_i^2 + \sum_i\mu^2_i\right]\\
&= \frac{1}{2}\left[-\sum_i\log\sigma_i^2 - n + \sum_i\sigma_i^2 + \sum_i\mu^2_i\right]\\
&= \frac{1}{2}\left[-\sum_i\left(\log\sigma_i^2 + 1\right) + \sum_i\sigma_i^2 + \sum_i\mu^2_i\right]\\
\end{align}

You see that the mean minimisation is the same so the only cost factor is between these stds and the covariance matrix values. We can see that its more expensive to evaluate these integrals in covariance matrix and then minimise them, then just to minimise these std.

TL;DR cheaper, but you are right it could be done. Good question!


Should augmentation also be performed on the validation set when the dataset is imbalanced? I am training a CNN on images (2 classes) and I have an imbalanced dataset (1:7 ratio). I am trying to tackle this by performing offline image augmentation. Should I perform augmentation also on the validation set or is it ok if the validation set remains imbalanced?

You have the idea of augmentation wrong I suppose. Image augmentation is used to introduce variations in your existing image dataset by using different operations like rotation, slicing, mirroring etc to make the model more robust. But using image augmentation on unbalanced data would keep the resultant data unbalanced as long as all the operations are performed on the entire dataset. So the question of performing augmentation on just train or both train and validation doesn't make much sense if you look at it the above way.
I would recommend you to opt for image augmentation only if you feel you have less examples to train from and also you want to make your model more robust but do it on your entire dataset.
You can oversample your minority class as well if you feel it's suffering during training or recall is less.
To oversample your minority class, you can perform image augmentation on your minority class only.


Time Series Model for multiple trends I am a newbie in the ML field. So please, neglect or better correct, if I am wrong somewhere.
Currently working on a model training for time series data.
My problem is a little more specific to bike-sharing. I have a count of bike-sharing for each area and for each bike type(gear, without gear...) for each day.

For example, Data:

Date    BikeType    Area    BikeCount
1/1/19  Gear        1        10
1/1/19  WithoutGear 1        15
1/1/19  Gear        2        8
1/1/19  WithoutGear 2        12
2/1/19  Gear        1        11
2/1/19  WithoutGear 1        17
2/1/19  Gear        2        9
2/1/19  WithoutGear 2        16

So, I will have a trend for each type of bike for each area.
How to use time series for this data.
I have to predict the bike required for each Type and each Area.
For example, for the given data I have to predict the count of Geared bike required and WithoutGeared bike required for both 1 and 2 areas on 3/1/19. (Considering two dates' data is enough for prediction, I have 2 years data for each specific area and type, they have a good trend)

And the second question is...
Currently, I have to only two dimension bike types and area, they may increase later(like their color and their condition) how to handle this.
Any contribution will be helpful.

I get a hint from the question:
https://stackoverflow.com/questions/55545501/how-to-perform-time-series-analysis-that-contains-multiple-groups-in-python-usin

But, is this the best any only way...

Thank you
(I need suggestion for the question title also)

Edit:

I get a similar problem in the following references:

Multi-dimentional and multivariate Time-Series forecast (RNN/LSTM) Keras

Multivariate and multi-series LSTM

Now, I have two more doubts:

Is the LSTM is the only way?
Are my data columns (data types and area) dimensions or features.

I think you should look into multivariate regression. You could use these variables (type, area etc) along with other factors such as seasonality. Create dummy variables such as date of the week, week number , month etc to capture seasonality. For example rainy month may have less bike demand. These are inherent features from the data.


How to approach a Data Science case study question? I recently had a phone interview with a consumer tech company for a quant position. The question was basically, "imagine a facebook style social network site. Six months ago a new feature called 'mentions' was added which allows you to tag your friends with an @ sign. How would determine whether this feature was a success?"

I was a bit taken aback by how broad the question was. I first asked if the feature was given to everyone in the network or a sample, to which the interviewer responded "you decide" - meaning I could approach the analysis either way. I talked in general terms about calculating week over week usage of the feature as well as month over month. I also discussed computing a baseline metric for product interaction and then comparing the usage of the new mentions feature relative the baseline statistic. Overall I left the interview feeling quite dumb, as I have a pretty solid command of stats, but came away looking like an idiot.

Are there specific statistical procedures to test for something like this? al la A/B testing, or some kind of hypothesis test? And secondly, is there a good framework for approaching these types of open ended case study style questions in general?

This question (something I've asked variants of several times in interviews) has absolutely nothing to do with statistical or other quantitative procedures. What is being asked here is for an understanding of the overall data mining process itself.  The first thing to determine is what the definition of success. So you have to ask. The stakeholder usually will not volunteer this unless asked anyway. Then, depending on the answer describe the overall process for data mining based on this end goal.


Add new factor across multiple groups I am trying to add multiple new rows for 3 new factor level in an existing data frame. Please refer to sample data for an example. My starting data frame has 18 levels for col1 and all 12 months for column mon and past 20 years for year. I then impute values and add new columns, however I need new factors to be added for further analysis.

For each mon and year combination, a new level should exist.

Sample df:

col1 <- c(rep("a",4),rep("b",4))
col2 <- c(1:4)
mon <- c(rep(c("Jan","Feb", "Mar","Apr"), 4))
year <- c(rep("2016",8), rep("2015",8))
df <- as.data.frame(cbind(col1,col2,mon,year))

head(df,8)  # edited to make it readable

  col1 col2   mon year
1 a    1      Jan 2016
2 a    2      Feb 2016
3 a    3      Mar 2016
4 a    4      Apr 2016
5 b    1      Jan 2016
6 b    2      Feb 2016
7 b    3      Mar 2016
8 b    4      Apr 2016

Expected Output

   col1 col2   mon year
1  a    1      Jan 2016
2  a    2      Feb 2016
3  a    3      Mar 2016
4  a    4      Apr 2016
5  b    1      Jan 2016
6  b    2      Feb 2016
7  b    3      Mar 2016
8  b    4      Apr 2016
9  c    NA     Jan 2016  # New level c for each mon and year
10 c    NA     Feb 2016  # New level c for each mon and year
11 c    NA     Mar 2016  # New level c for each mon and year
12 c    NA     Apr 2016  # New level c for each mon and year

How do I go about reaching the expected df?

Several possibilities. For example, to add c for existing mon-year combinations in your data frame:

rbind(df, transform(df[!duplicated(df[, 3:4]), ], col1="c", col2=NA))
#     col1 col2 mon year
# 1      a    1 Jan 2016
# 2      a    2 Feb 2016
# 3      a    3 Mar 2016
# 4      a    4 Apr 2016
# 5      b    1 Jan 2016
# 6      b    2 Feb 2016
# 7      b    3 Mar 2016
# 8      b    4 Apr 2016
# 9      a    1 Jan 2015
# 10     a    2 Feb 2015
# 11     a    3 Mar 2015
# 12     a    4 Apr 2015
# 13     b    1 Jan 2015
# 14     b    2 Feb 2015
# 15     b    3 Mar 2015
# 16     b    4 Apr 2015
# 17     c <NA> Jan 2016
# 21     c <NA> Feb 2016
# 31     c <NA> Mar 2016
# 41     c <NA> Apr 2016
# 91     c <NA> Jan 2015
# 101    c <NA> Feb 2015
# 111    c <NA> Mar 2015
# 121    c <NA> Apr 2015

To add c for all possible combinations of existing mon values and existing year values:

rbind(df, data.frame(col1="c", col2=NA, expand.grid(mon=levels(df$mon), year=levels(df$year))))

To add c for all possible combinations of all months names and existing year values:

rbind(df, data.frame(col1="c", col2=NA, expand.grid(mon=month.abb, year=levels(df$year))))

and so on.


How to specify version for dependencies so that each one is compatible and stays within a size limit? I am trying to deploy a web app to Heroku. The free tier is limited to 500 MB.
I am using my resnet34 model as a .pkl file.
I create model with it using the fastai library.
This project requires torch and torchvision as dependencies.
But not specifying the dependency will download the latest version of torch which alone is 750 MB and exceeds the memory limit.
So, I specify torchvision version as 0.2.2 and specify the wheel for torch for v1.1.0 in the requirements.txt file. Now, this gives rise to other problems.
For example, I got this error message from Heroku build log-

ERROR: torchvision 0.2.2 has requirement tqdm==4.19.9, but you'll have tqdm 4.48.0 which is incompatible.

But if I specify tqdm's version to 4.19.9, some library called spacey becomes incompatible and the app fails. If I install more recent version of torchvision, the app size will exceed permitted limits.
How do I get around this problem? How do I find out which dependency versions are compatible with each other, and the total size doesn't exceed 500 MB? Is there an easy way to do that?

Till now, what I have learned is that it is a trial and error process.
And the best solution is to look for projects that have already done this using a working combination of libraries.
I used this requirements.txt file, which works fine:
gunicorn
flask
numpy
https://download.pytorch.org/whl/cpu/torch-1.1.0-cp36-cp36m-linux_x86_64.whl
torchvision==0.2.2
fastai==1.0.52
jupyter

The web app works perfectly and the permitted size is not exceeded.
Although Heroku's size limit is 500 MB, they have a soft limit of 300 MB. These libraries, their dependencies, and the model file together exceed that resulting in a slightly longer response time.


Explainability ML Methods I am currently working on a Machine-Learning Model. In order to explain how it works, I have looked at Partial Dependence Plots, Feature Importance and all kinds of methods, but one thing still concerns me:
Should I evaluate these on the dataset I trained the model with, or the current dataset on which it performs predictions? This of course leads to different results depending on the distribution and the performance of the model.
How do you handle this?
Any opinions or input would be highly appreciated!

You can look at the different evaluations as evaluation of the different stages of the model.

by evaluating the model's performance on the training dataset, you could assess how and what the model has learned from the data structure. This evaluation is mainly relevant for your research stage and the reporting of your results.
by evaluating the model's performance on the current/test dataset, you would see how the model is behaving and performing on current data the real predictions. This may help you to asses that

These two may differ in cases where the current data is behaving differently than the data that the model been trained on, and to help you to understand that the model needs to be updated and retrained on the data that it is performing poorly on.
If you're not monitoring your current data feed (distribution, range of values, the ratio of missings), I would suggest you do both for assessing if your ongoing observed performance is not what you are expecting.


Compare two topic modelling sets I have two sets of newspaper articles where I train the first newspaper dataset separately to get the topics per each newspaper article.

E.g., first newspaper dataset
article_1 = {'politics': 0.1, 'nature': 0.8, ..., 'sports':0, 'wild-life':1}

Again, I train my second newspaper dataset (from a different distributor) to get the topics per each newspaper article.

E.g., second newspaper dataset (from a different distributor)
article_2 = {'people': 0.3, 'animals': 0.7, ...., 'business':0.7, 'sports':0.2}

As shown in the examples, the topics I get from the two datasets are different, thus I manually matched similar topics based on their frequent words.

I want to identify whether the two newspaper distributors publish the same news in every week.

Hence, I am interested in knowing  if there is a systematic way of comparing the topics across two corpora and measuring their similarity. Please help me.

Considering you extracted the news using a TF-IDF approach what you have as a result is just one feature (frequency of terms). I think you would need to add more features to your corpus to be able to match two news as the same (or similar).

One new feature would be the temporal, where you would add a timestamp for the news. It will allow you check if two news (from different publishers) were published on the same period. (a week, 2 weeks, etc).

The second one could be a spatial, if you have geolocation of the news for example, you can add it to your train dataset. Something similar was done by Chi-Chun Pan. It would allow you be more confident if the news happened on the same place.


Technical term to describe/differentiate the structure of an instance by the number of samples per instance ('tuple' vs 'time series') Several terms are used to describe the structure of instances of a dataset. For example, an instance can be 'univariate' if it depends on only one variable or 'multivariate' if it depends on 2 or more variables. I wonder if there are technical terms to distinguish the structure of a, let's call it a 'tuple', and a 'time series' or 'sequence'.
Here are two examples that I am trying to describe or distinguish structurally:

The left example is a time series, where all 11 samples belong to one instance. The right example is a collection of 'tuples' (if that is the correct term), where each row represents a separate instance.
I always refer to the term 'tuple' here, because the concept seems similar to me with the record in a database or the representation of points from mathematics. I am not sure if 'time series' and 'tuple' are the correct terms to describe how many samples an instance consists of. The terms should be analogous to uni/multivariate to describe the number of features in an instance.
Hence my question:
If univariate/multivariate describes an instance in the horizontal, what terms are used to describe an instance vertically?

We have a Dataset
 Dataset has Instances
Instances can be a Vector Or a Sequence.

If it is a vector, It has Features
Examples - Image, Tabular data
If it is a Sequence, It has Time-steps**
Each Time-step can have Features i.e. Uni/Multi-Variate
Examples - Video, Audio, Time-series data

**When we use text-data as a sequence, we call the words as "Tokens"


Was this dataset analysed correctly? There's a Twitter thread going around that claims there are signs of voter fraud due to anomalies in the election vote count data set. You can download the dataset here and find the script to generate the analysis here.
Was the dataset analysed correctly? If yes, are these 'anomalies' significant enough to indicate fraud? If no, in what way is the analysis lacking?
PS: I'm not looking to start a political discussion, or to make any claims of fraud myself.

TL;DR
While the data comes from the NYTimes and seems legit, the presentation is intentionally misleading and the subsequent assertions are baseless. I say "intentionally" because an unbiased and reputable analysis would not propagate such major allegations from the data they have presented. The data does not prove nor disprove voter fraud, so the unfounded assertions are simply disinformation.

Before diving into the data
We are going to see many more such threads/posts in the coming months and years, so I need to address this bit before delving into the question.
Consider the following whenever you view anything online:

Fake news is much more viral than true news! Always air on the side of caution.
Is the source credible? In this case, you have no reason to believe the random twitter user "CulturalHusbandry @Aphilosophae".
Do other credible sources corroborate the findings? Your confidence in the findings should increase with every independent credible source that come to the same conclusions. In this case, we do not have a credible source to begin with.
Is the language charged and the rhetoric conspiratorial? This is unfortunately becoming a prominent red flag for disinformation campaigns that are developed to sow doubt and distrust in democratic institutions.

How this Twitter thread is disinformation (step-by-step)
1. False premise
He/she first establishes the fact that, in total, batches of votes do not conspicuously favor one candidate or another (they do so via the tweets below). The distribution of the D/R ratio in vote batches over time is not reflected in this aggregate plot. However, this could mislead viewers to believe that they should expect 50/50 Democrat/Republican consistency throughout the vote counting process, which cannot be assumed from this information. The "anonymous data scientist" seems to encourage this logical fallacy by using this as his/her premise and posting his/her follow-up tweet to the plot (image below).

2. Misleading visualizations
He/she then establishes that the initial counts are "random" and noisy as one might expect from a 50/50 split. However, these batches do not have associated sizes with them! A "batch" may be 100K votes or it could fewer than 10. An actually informative portrayal of the data would have made the size of the circle/point reflect the size of the batch (bubble chart) rather than a scatter plot. Although this plot may technically be accurate, this can still be quite misleading, particularly because the "anonymous data scientist" made no note of this deficiency (lack of batch size information). But for the sake of argument, let's go with it.

3. Unsubstantiated claims and ignoring legitimate possibilities
Now they start talking about an "anomaly" and introducing conspiratorial allegations. They note the jump toward D, given the D/R ratio per batch in Milwaukee at some point in time. There are many legitimate explanations for this.
For example, a high/dense population such as a city (likely Democrat-leaning) would of course take more time to collect/count ballots, so they may not have started their reporting until after low/sparse populations (likely Republican-leaning). This can cause a sudden jump in favor of Democrats.
Also, the majority of the early ballots are in favor of Republican. This is to be expected; of course the candidate who consistently attempted to delegitimize mail-in ballots would receive fewer mail-in ballots from his followers. By contrast, Democrats were encouraged to vote by mail (and they did), making the later mail-in vote counts lean Democrat. This is another factor that could explain why the initial vote counts are skewed toward Republican in most states when compared to the final counts.


Goodhart's law applied to data science I recently learned about Goodhart's Law. Simply put,

  When a measure becomes a target, it ceases to be a good measure.

However, in Data Science, we really do aim at improving our performance by increasing or decreasing a metric, and improve our models based on that. For instance, in Kaggle competitions. Is Goodhart's Law applicable to Data Science? Why or why not?

The function problem which Goodhart's Law described is the changes of the underlying model from which the data is generated.

One way to look at this is that the data generating process in the past is bias. Say we our try model is a product of two variables $z = xy$. In the past, $y$ has always been constant at $a$, so you come to the conclusion $z$ must equal to $ax$. But this is only true when $y=a$.

In Goodhart's setting (economic policy and social science in general), it is usually the prediction value of $z$ that causing the changes in $y$. It is a dynamic system of which we only have a bias static slice of data.


Multi-label text classification with minimum confidence threshold Building out a system that tries to apply zero or more predefined labels to text.

For each label, we've:

built out a reasonably good vocabulary of high-value words/features
developed a corpus containing thousands of labeled entries
trained a NaiveBayesClassifier for each topic that does a good job of classifying valid vs noisy content

The problem seems to be that the individual classifier is great at differentiating between valid & noisy content WITHIN a topic:

"the green energy bill will revolutionize..." (green = "green energy")
"the green bay packers went on to lose their..." (green != "green energy")

...but when classifying content that shouldn't match ANY topic it has a very high rate of false positives. There's no "everything else" label!

tl;dr it's good at subtle, in-topic differentiation, but terrible at broad topic labeling

Are there any algorithms that help you classify into N categories, but allow for "everything else" which might not fit into ANY of the categories?

Looks like the common approaches to multi-class classification actually solve this challenge.

Building individual Naive Bayes Classifiers with only the training data for a single label is insufficient - we must also the include data from other labels as "everything else".

See Text-Classification-Problem, what is the right approach?


Deciding which samples the model will (probably) classify incorrectly Problem:
Given a neural network for image classification, the objective is to develop an algorithm which decides which images are 'problematic' and the model is probably going to classify them incorrectly.
Discussion:
So far, I've thought of two possible approaches:

Feed the given image to the model and then analyse its softmax ouput with various metrics (difference between first and second class confidence, entropy, gini index etc).
Perform some kind of image processing (feature extraction) on the given image, to obtain some features that indicate whether the image is not going to be correctly classified.

Questions:
Can you provide me with more suggestions about the second approach? What type of feature extraction would you think will help distinguish those images?
Any other ideas that are not mentioned here are welcome.

You might want to frame your problem as an uncertainty estimation problem.
The idea is that you want to evaluate how comfortable your model will be when making a prediction. If your model is not very comfortable with a prediction (even if gets classified as predic_proba = 0.99), then the uncertainty prediction should be high.
What @BrianSpiering is proposing is a way to calculate uncertainty with neural networks. This method is known as Monte Carlo Drop Out as a Bayesian estimation.
The idea is to apply drop out regularization when doing predictions and doing several times. This way your model predicts a probability distribution in which you can calculate several statistics as standard deviation. This will let you know how comfortable is your model doing such prediction.
This is just a method but there are several ways to estimate uncertainty.
There are some blogs and papers that might help you:

Uncertainty in Deep Learning. How To Measure?.Deep Learning Model Says: “sorry, I don’t know the answer”. That can be absolutely OK
Simple and Scalable Predictive Uncertainty
Estimation using Deep Ensembles: From Google DeepMind at NIPS


how to do regression and classification in a same time? I was thinking of doing a job including regression firstly, then doing the classification. I read lots of sources that are saying it is Semi-Supervised learning while I think it is not.
Imagine after doing regression we will reach to the graph below:

we know that there are 2 classes and now the goal is for prediction of the possibility of locating in class 1 or 2. It means that, for the next data that we have, we need to make a classification.

I believe you are just describing standard classification.
Take the image below, the model of the classification is the decision boundary found between the two classes. After we have found this model, using a classification algorithm (clustering or otherwise), for new data we can use it to predict which class the new data belongs to.


scikit learn target variable reversed (DecisionTreeClassifier) I created a Decision Tree Classifier using sklearn, defined the target variable:
#extract features and target variables
x = df.drop(columns="target_column",)
y = df["target_column"]
#save the feature name and target variables
feature_names = x.columns
labels = y.unique()
#split the dataset
from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(x,y, test_size = 0.3, random_state = 42)

Additionally I checked the count of each of the two classes (Success, Failure) within y which confirmed to me that each has the correct count.
Then I fitted my DTClassifier:
clf = DecisionTreeClassifier(
    criterion='gini',
    splitter='best',
    max_depth=None,
    min_samples_split=2,
    min_samples_leaf=1,
    min_weight_fraction_leaf=0.0,
    max_features=None,
    random_state=42,
    max_leaf_nodes=None,
    min_impurity_decrease=0.0,
    min_impurity_split=None,
    #class_weight="balanced",
    presort='deprecated',
    ccp_alpha=0.0,
)
clf.fit(x_train, y_train)

The problem becomes apparent at the visualization step when I plotted the tree, each node shows me class = Failure when Failure is the minority and vice versa. Further down the line plotting the confusion matrix and calculating all the performance metrics it also becomes apparent, that the labels were reversed and I cannot figure out as to why.
Any ideas where I might need to look for the answer? If more code is necessary to give a feedback I can provide.
Much appreciated.

It might be because of the conflict of order between model.classes_ and series.unique()
For a binary labels,
model.classes_ = array([0, 1])
series.unique() = array([1, 0])
Try creating a constant value i.e. np.array[0,1] for labels and see.


Vectorized method to find matching values between two columns I'm trying to locate the most recent rows within my Dataframe that contain the same values in two separate columns.

Presently, I am doing this slowly with looping, but I suspect there's a way to cleverly use the apply method or some other vectorized function to do this faster.
My present code:

def enumerate_matching(df):
    a = list(df['A'])
    b = list(df['B'])
    matching = []

    for i in range(0, len(a)-1):
        for j in range(i+1, len(b)):
            if a[i] == b[j]:
                matching.append(i)
                matching.append(i+j)
                break
    return matching

Is there a faster method to do this?

you could use set to get the intersection (it has a complexity logarithmic in the size of the sets a and b)

 a = set(df['A'])
 b = set(df['B'])
 a.intersection(b)


Warning message in randomForest I want to applicate the randomForest to my data for predicting target variable, but I have got a warnings message saying:

Warning message:
In randomForest.default(m, y, ...) :
  The response has five or fewer unique values.  Are you sure you want to do regression?

I didn't know what was going on?

this is my code:

library(randomForest)
M=data.frame(Type_peau,PEAU_CORPS,SENSIBILITE,IMPERFECTIONS,BRILLANCE ,GRAIN_PEAU,RIDES_VISAGE,ALLERGIES,MAINS,
             INTERET_ALIM_NATURELLE,INTERET_ORIGINE_GEO,INTERET_VACANCES,INTERET_COMPOSITION,DataQuest1,Priorite2,
             Priorite1,DataQuest4,Age,Nbre_gift,w,Achat_client)

factor_vars <- c("Type_peau","PEAU_CORPS","SENSIBILITE","IMPERFECTIONS","BRILLANCE","GRAIN_PEAU",
                 "RIDES_VISAGE","ALLERGIES","MAINS","INTERET_ALIM_NATURELLE","INTERET_ORIGINE_GEO",
                 "INTERET_VACANCES","INTERET_COMPOSITION","DataQuest1","Priorite2","Priorite1","DataQuest4")

head(M)
str(M)

sample.ind <- sample(2, nrow(M), replace = T,prob = c(0.6,0.4))
cross.sell.dev <- M[sample.ind==1,]
cross.sell.val <- M[sample.ind==2,]

table(cross.sell.dev$Achat_client)/nrow(cross.sell.dev)

table(cross.sell.val$Achat_client)/nrow(cross.sell.val)

varNames <- names(M)
# Exclude ID or Response variable
varNames <- varNames[!varNames %in% c("Achat_client")]

# add + sign between exploratory variables
varNames1 <- paste(varNames, collapse = "+")

# Add response variable and convert to a formula object
rf.form <- as.formula(paste("Achat_client", varNames1, sep = " ~ "))

test$Variable1 <- factor(test$Variable1,levels=levels(train$Varialbe1))

# Buiding
cross.sell.rf <- randomForest(rf.form,cross.sell.dev,ntree=500,importance=T)
plot(cross.sell.rf)

Thanks for your help!

In fact, the problem was in the type of target variable.

It should be assumed as factor and not numeric !!

So the solution is just to did like that after reading table:

Achat_client=as.factor(Achat_client)

Good Luck!


Training Restricted Boltzmann Machines (RBMs) using gradient descent Hey I am a little new to the whole RBM entropy/energy training thing, having some trouble understanding and trying to figure whether it is worth the effort needed to understand.

Can't RBMs quite easily be trained with a standard gradient descent algorithm? Is the current way just much faster?

Here is some sample Theano code of what I mean:

def rbm(inputs, shape, cost=0, last_cost=1):
    X = T.vector()                     # input
    W = theano.shared(np.zeros(shape)) # weights

    b = theano.shared(np.zeros( shape[1] )) # bias to hidden
    B = theano.shared(np.zeros( shape[0] )) # bias to reconstruction

    H = sigmoid(T.dot(X, W) + b) # hidden activation
    Y = T.dot(H, W.T) + B        # reconstruction

    L = ((X - Y) ** 2).mean()                    # loss function
    G = T.grad(L, W), T.grad(L, b), T.grad(L, B) # gradients

    a = .1                                                  # learning rate
    U = [(W, W - a*G[0]), (b, b - a*G[1]), (B, B - a*G[2])] # shared updates

    train = theano.function([X], L, updates=U) # training function

    while abs(last_cost-cost) > 0.01:
        last_cost, cost = cost, 0
        for input in inputs: cost += train(input)

    return W.get_value(), b.get_value()

And yes this code is very crude.

What you're trying to do is to minimize the difference between input and its reconstruction. They may be completely different values and RBM may be working fine! It's due to the fact that Restricted Boltzmann Machine models some probabilistic distribution. Learning rule is to find values of weights to model the distribution of the data. You may clamp a data (visible units) and see a reconstruction, but since hidden units are stochastic units, they would have many values. Even when you set your network free and let the units have some values according to stochastic units rule, after some time you can get a sample from RBM, which should be from your data distribution.


Which algorithm or tool to use to classify as good or bad? I have a feature vector with different data types, Considering all the data in that feature vector. I have to classify as Good or Bad.

Which algorithm should be used to just get the output Good or bad based on different data types in a feature vector?

The feature vectors are as follows:

[Application_Name(string) , Uptime (Integer) , Criticality factor (0-1 float value) and few integer type ]

Based on the comments I'll try to answer. I guess you don't have the corresponding labeles. What you can do as a solution is that you can use k-means algorithm as the easiest start point to and specify the hyper parameter, k, to two. Then you can find two clusters and you yourself can evaluate the results. As another approach, you can increase the size of k and again evaluate your answers. You can also use Gaussian Mixture Models for finding better non-convex clusters which have better results. The point is that you have to evaluate the results as an expert and label them manually. This task accelerates the labeling process. After that you can employ a simple MLP for finding a descriminative model.


Encoding for classifiers I have some doubts regarding encoding (i am not familiar with tasks like these) categorical variables in order to use them as parameters in a model like logistic regression or SVM.
My dataset looks like the following
Text                                  Symbol    Note    Account    Age   Label
There is a red car                      !        red      John    24   1
My bag was very expensive               ?       orange    Luke    36  0
Where are my keys?                      @        red      Red     58  1
I promise: I will never let you go!    ...       green    Aoife   28  0

In text there are stored comments from users in a community; symbol includes the most used symbol by an user; Note represents its level: green is more experienced, red is a new joiner; Account is the user name. Label gives me information about user’s trustworthy: if 0 the user is not fake; if one the user might be a possible bot.
I would like to classify new users based on the current information (see columns above). My dataset includes more than 1000 rows and 400 users.
Since to use classifiers I need to encode categorical fields, I have tried to do as follows, by using MultiColumnLabelEncoder in sklearn:
MultiColumnLabelEncoder(columns = ['Text', 'Symbol', 'Note', 'Account']).fit_transform(df)

where df is my dataframe. However, I understood that also onehotencoder should be preferable. I also included Account as there might be more comments from the same account, so if I classified an account as fake and I receive a new comment from the same account, then this account could be easily detected as fake.
The aim, as earlier i mentioned, would be to classify, with a certain accuracy, new elements from a test set, based on the information given(symbol, note , age, texts), i.e. looking for a possible correlations among these variables which can allow me to say that a new account is fake (1) or not (0).
The problem, as you can see, is related to classifiers where parameters are not only numerical, but also categorical.
For data preprocessing (removing stopwords and cleaning data), I have used  Python packages of nltk; regarding features extraction ( this should be a key point as it is linked to the next step, i.e. using a classifier to predict class - 1 or 0), I have found difficulties in understanding what output I should expect from the encoding in order to be able to use information above as inputs in my model (where target is called label and it is a binary value).
I am using as classifier logistic regression, but also
SVM.
My expected output, in case of user X with age 16, symbol #, note Wonderful, and note red (new joiner) would be a classification as fake with a certain percentage.
I would appreciate if someone could explain me, step by step, the way to transform my dataset in a dataset whose variables I can use  within a logistic regression in order to determine the label (fake or not fake) of new users.

You will have to use a mix of text processing and one hot encoding. Text column should not be treated as one-hot encoded since it will try to create one new variable for every unique sentence in the dataset, which will be a lot (and not very helpful from learning). Text vectorizer will summarize text column based on type of words/tokens that appear in it.
So you should use a text vectorizer for processing only the text column first. This will give you one data-frame (say A). This data-frame will have columns corresponding to tokens/words in the dataset. So if text vectorizer picks up 100 unique words then you will have a 1000x100 size data-frame. Note that these 100 columns have been generated only by text vectorizer.
For symbols and notes, you can use one-hot encoding, which will get you another data-frame (say B). Then you should join A & B on common key to get your final data-frame if input. The common key here will be row ID (though read the following comment on aggregating data at user level).
It is not clear if the user name (Account) column unique in the data? If there are 1000 rows but only 400 users, that means there can be more than 1 row per user. In that case, you can consider aggregating the data at the user level (for text column, you can simply concat all strings for same user).


What's the most effective way to measure the accuracy of my multi-class classification NN? I'm relatively new to data science, and am confused about how to measure the accuracy of a multi-class neural network. The model I'm building is attempting to predict the outcome of a given event with 20 different outcomes - there is some imbalance in the labels, the most frequency label is around 15% of the total, and the least frequent is around 1%. The event I'm attempting to classify is difficult to predict, and I'll be happy with around 20% accuracy over all events - however I'm worried that when I achieve that, then my model will simply be predicting the most frequent class all the time, and just getting lucky on enough of the others to bring overall accuracy up to 20%. I'm using keras' 'categorical cross-entropy' loss function and 'accuracy' metric - are these the most appropriate for the problem? Which other metrics are best suited to a problem like this? Very happy to be referred to literature to help build my understanding of this problem.
I understand the confusion matrix for binary classification, however for multi-class classification, however I'm at a loss (no pun intended) as to how to apply these principles to a multi-class problem, whilst ensuring my model isn't just predicting the most popular class and getting lucky on enough others to meet my target.
Thanks in advance for any assistance!

Keras has some accuracy metrics for multiclass neural networks.

CategoricalAccuracy
Top K  Categorical Accuracy
Sparse Top K Categorical Accuracy

There are some others if you want to implemented from scratch. Taken from: https://www.sciencedirect.com/science/article/abs/pii/S0306457309000259

These work for most problems rather nicely combined with a good cross validation approach and leak prevention.
Having an imbalanced dataset always makes it challenging but picking a metric that doesn't punish results too much like Top K Categorical Accuracy combined with n-fold cross validation like this article suggests, is often enough to have a solid evaluation procedure that you can work on.


Why autoencoders work well for outlier detection? Apart from the fact that they are neural networks, which usually is a reason for outperforming other algorithms, is there other reason helping auto-encoders perform well in outlier detection?
I know that autoencoders work by encoding a sample into lower-dimension representation, then decoding the representation to reconstruct the sample. As outliers usually have higher reconstruction, they can be detected. However, this does not convince me why auto-encoders can beat other methods. Is it because the outliers errors are very high, so that they can be spotted easily? If so, then what makes the reconstruction errors so high?
Both intuitive and/or theoretical explanations are welcomed.

It helps to first understand why outlier detection is generally a difficult problem and why other methods struggle with it.
By their very nature outliers are rare and most data we have is heavily imbalanced. Quite likely you might have not enough "positive cases" / outliers to train a model at all.
Autoencoders solve this problem because they do not try to identify outliers per se. As you have described they basically learn to down- and upsample input with a high resolution performance. However when an input is very different from usual inputs the upsampling might result in more errors than usual which then helps us in identifying outliers.
The reconstruction errors are higher because the autoencoder has been trained mostly / almost exclusively on non-outlier data so when they encounter an outlier they cannot deal with it as well.
Imagine an autoencoder trained to downsample pictures of an orange and then upsample it. If we feed the picture of an apple into this autoencoder it will not produce a very accurate result and that helps us identify that the input is actually an outlier.


Low Kappa score but high accuracy I have a dataset with 600 examples of target class yes and 300 of no.
After classification I get high accuracy(75%) but low kappa (0.4).
What can it mean? It seems that kappa is really low, am I right?
Other metrics like precission or AUR are about 0.7-0.8 what seems to be good.

I'm not sure that anything is wrong here. With a binary target, chance is 50%, and 75% is half way to perfect accuracy of 100%.  Kappa, which ranges from 0 at chance to 1 at perfect separation, is 0.4, which is again a little less than half way to perfect prediction.  So it seems to be corroborating the performance of your model.

Kappa is defined as:
(observed accuracy - expected accuracy)/(1 - expected accuracy)

Since your classes appear balanced, you have (0.75 - 0.5) / (1 - 0.5) = 0.5


Should I transform my feature into normal distrubition before Isolation Forest I have a anomaly detection problem and my features are following exponential distrubition. Should I first transform my features into normal distrubition before feed into isolation forest?

Its better to use data as is for anomaly detection as the underlying data is not normal. Apart from what you asked, Isolation Forest has a problem that it has the chances of creating ghost clusters where anomaly scores would be incorrect. Hence the recent one is EIF works better.


Predict buying behavior under the condition that a customer is advertised or not I'm trying to predict response of customers on a marketing campaign.
As of now, I have data from one Marketing campaign and rfm data of my customers.

Some proportion of customers say 60%, received adverts. Roughly 10% responded

From the response data, i.e. whether purchased or not, on this marketing campaign I built a random forest using scikit-learn.

The model performs on a hold out set really well. but the most influential variable is the boolean: CustomerHasBeenAdvertised

I want to use this model, to select customers for a future marketing campaign.
To obtain the "purchasing probabilities" of the customers under the condition of similar advertisement, I set the variable CustomerHasBeenAdvertised to 1.

However, on a data set with this side condition, all the forecasts are above 0.5.

Is this extra ordinary high value due to the variable importance? Or are there other explanations?

Is setting the variable CustomerHasBeenAdvertised to 1 the false approach?

If so, how could one handle the case: Customer bought without beeing  advertised?

Should one simply neglect the information whether advertisement took place or not?

Thanks in advance

It depends what you want to do. If you want a model that will predict who to target/not-target for a marketing campaign, then you want to train a model using only the people who were marketed to, which defines your two classes "responders" and "nonresponders". Given an unseen customer, your classifier will then determine whether they are likely to respond if you send them a directed marketing advertisement. Take a look at uplift marketing.


Are there any frameworks available that allow for automated large scale supervised machine learning? The typical steps for solving a machine learning/pattern recognition problem:

Data Analysis and splitting the data into test and train sets.
Choosing a model.
Training the model, and testing the model against the test set.
If the accuracy of the model is not acceptable, start over with a new model.

Step (2) can be automated to some extent using a grid search, but are there ways of automating the whole process?

I'm thinking in particular of large scale data applications (for example when dealing with retail data, or customer analytics on a site like Netflix, ) where there are millions of instances of similar but distinct machine learning problems that each need to be trained and validated separately.

In such a situation it is impossible for a team of analysts or data scientists to perform the above steps and some sort of automated model development framework must be used.

What are the frameworks that allow for this?

I think, Rapidminer can be used for that. However, the free version can only load 10,000 rows of data.


How to combine and separate test and train data for data cleaning? I am working on an ML model in which I have been provided the data in 2 files test.csv and train.csv. I want to perform data cleaning on both files together be concatenating them and then separating them.
I know how to concatenate 2 dataframes, but after data cleaning how will I separate the two files? Please help me complete the code.
CODE
test = pd.read_csv('test.csv')
train = pd.read_csv('train.csv')

df = pd.concat([test, train])

//Data Cleaning steps

//Separating them back to train and test set for providing input to model

There are several methods to choose from. If you insist on concatenating the two dataframes, then first add a new column to each DataFrame called source. Make the value for test.csv 'test' and likewise for the training set.
When you have finished cleaning the combined df, then use the source column to split the data again.
An alternative method is to record all the operations you perform on the training set and simply repeat for the test set. This won't work it you normalise values based on the population.


Machine learning for Point Clouds Lidar data Our main use case is object detection in 3d lidar point clouds i.e. data is not in RGB-D format. We are planning to use CNN for this purpose using theano. Hardware limitations are CPU: 32 GB RAM Intel 47XX 4th Gen core i7 and GPU: Nvidia quadro k1100M 2GB. Kindly help me with recommendation for architecture.

I am thinking in the lines of 27000 input neurons on basis of 30x30x30 voxel grid but can't tell in advance if this is a good option.

Additional Note: Dataset has 4500 points on average per view per point cloud

First, CNNs are great for image recognition, where you usually take sub sampled windows of about 80 by 80 pixels, 27,000 input neurons is too large and it will take you forever to train a CNN on that.

Furthermore, why did you choose CNN? Why don't you try some more down to earth algorithms fisrst? Like SVMs, or Logistic regressions.

4500 Data points and 27000 features seems unrealistic to me, and very prone to over fitting.

Check this first.

http://scikit-learn.org/stable/tutorial/machine_learning_map/


XGBoost Predictions all the same When I evaluate the model I seem to be getting a decent RMSE score but when I try to actually see the predictions when I call the model all my values are the same.

 xdata = xgboost.DMatrix(X_train, y_train, feature_names=all_vars)
 xdata_val = xgboost.DMatrix(X_valid, y_valid, feature_names=all_vars)
 xgb_parms['seed'] = random.randint(0,1e9)
 model = xgboost.train(xgb_parms, xdata)
 model.eval(xdata_val)
 ypred = model.predict(xdata_val)

I believe the error is on my last step, what am I doing wrong?

Make sure to pass the model an "objective" parameter and also use "rmse" for the "eval_metric" parameter.
https://github.com/dmlc/xgboost/blob/master/doc/parameter.rst


Similarity Measure of Simulated Time Series vs Observed time Series In my work I have an observed Time Series and Simulated ones. I want to compare the Light Curves and check for similarityto find out which simulated curve fits best respectivley which parameters simulate the Light Curve the best.

At the moment I do it with the Cross-Correlation function from numpy. But I am not sure if that is the best option, due to the fact that the Light Curve with the highest Cross-Correlation Coefficient not always looks like the best fit/simulation compared to other simulations with a lower CC-Coefficient. Is there a another way to measure for similarity? I read something about the Chi-Square Statistics, but I am not sure how that works and how this could be applied to my problem.

The observation data I use is not evenly binned, so I used the interpolation function of Scipy. Should I also smooth the observation data or would I lose true features of my data. I thought about using the savitzky golay smoothing.

At the moment I am using a brute force method to try out all possible parameters and simulated the corresponding light curve. The problem is this takes a lot of time with 20 parameters. The parameters are more or less dependent on each other. So I cant use a least-square fit method, because there are multiple possible minimas. Is there a simple method that I overlooked. Or is a restricted brute force fit my best option?

In the picture below you'll see one plot with the simulation and the observaton data.

Thanks for all suggestions.

in my opinion you can do the following to compare your simulated with your actual data.

Use usual measurements like  those you would use in predictions vs actual data:

RMSE
MAE (Mean Absolut Error)
MSE (Mean Squared Error) --> Gives greater weight to larger errors/gaps

Another statistical test that comes to my mind to test "prediction" quality between different models is the Diebold-Mariano test. It is fully implemented in R. As far as I know, not in any Python Library so far.

As a statistical test you could have a look at the distribution (but as you dealing with time-series you may face "memory", meaning that observations t and t+1 face autocorrelation (against most of statistical tests, as they assume independent observations). Autocorrelation seems to be the case for your data series. Therefore you should have a look on the % change of your sample data (from t to t1) and look if both series are drawn from the same distirbution - You can perform the kolmogorov smirnov test.

Hope this helps.

EDIT

Regarding your question concerning smoothing and interpolation. Empiricis (not statistics) is more a trial and error concept. It really depends on your data behavior.


python xgboost DMatrix - get feature values or convert to np.array I'm trying to create a custom evaluation metric (feval) function for xgboost.cv. It should process some of the training features, however I can't find a way to extract features from xgboost.DMatrix() object (only labels)..

As I was suggested it can be done with first making a DMatrix slice then saving it to a svmlight file and finally reading it with scikit-learn, but is there a more elegant way if doing it?

Currently we cannot direct extract data matrix from DMatrix, mainly due to DMatrix is a internal data structure that may or maynot sit in memory.

--by tqchen commented on 3 Aug 2015

(Source)


Understanding convolutional pooling sizes (deep learning) I'm dumb but still trying to understand the code provided from this e-book on deep learning, but it doesn't explain where the n_in=40*4*4 comes from. 40 is from the 40 previous feature maps, but what about the 4*4?

>>> net = Network([
        ConvPoolLayer(image_shape=(mini_batch_size, 1, 28, 28),
                      filter_shape=(20, 1, 5, 5),
                      poolsize=(2, 2),
                      activation_fn=ReLU),
        ConvPoolLayer(image_shape=(mini_batch_size, 20, 12, 12),
                      filter_shape=(40, 20, 5, 5),
                      poolsize=(2, 2),
                      activation_fn=ReLU),
        FullyConnectedLayer(
            n_in=40*4*4, n_out=1000, activation_fn=ReLU, p_dropout=0.5),
        FullyConnectedLayer(
            n_in=1000, n_out=1000, activation_fn=ReLU, p_dropout=0.5),
        SoftmaxLayer(n_in=1000, n_out=10, p_dropout=0.5)],
        mini_batch_size)
>>> net.SGD(expanded_training_data, 40, mini_batch_size, 0.03,
            validation_data, test_data)

For instance, what if I do a similar analysis in 1D as shown below, which should that n_in term be?

>>> net = Network([
        ConvPoolLayer(image_shape=(mini_batch_size, 1, 81, 1),
                      filter_shape=(20, 1, 5, 1),
                      poolsize=(2, 1),
                      activation_fn=ReLU),
        ConvPoolLayer(image_shape=(mini_batch_size, 20, 12, 1),
                      filter_shape=(40, 20, 5, 1),
                      poolsize=(2, 1),
                      activation_fn=ReLU),
        FullyConnectedLayer(
            n_in=40*???, n_out=1000, activation_fn=ReLU, p_dropout=0.5),
        FullyConnectedLayer(
            n_in=1000, n_out=1000, activation_fn=ReLU, p_dropout=0.5),
        SoftmaxLayer(n_in=1000, n_out=10, p_dropout=0.5)],
        mini_batch_size)
>>> net.SGD(expanded_training_data, 40, mini_batch_size, 0.03,
            validation_data, test_data)

Thanks!

In the given example from the e-book, the number $4$ comes from $(12-5+1) \over 2$, where $12$ is the input image size $(12*12)$ of the second constitutional layer; 5 is the filter size (5*5) used in that layer; and $2$ is the poolsize.
This is similar to how you get the number $12$ from the first constitutional layer: $12= {(28-5+1) \over 2}$. It's well explained in your linked chapter.
Regarding your "For instance" code, your 6th line is not correct:

ConvPoolLayer(image_shape=(mini_batch_size, 20, 12, 1),

The number 12 should be replaced by $(81-5+1)\over2$ which unfortunately is not an integer. You may want to change the filter_shape in the first layer to (6,1) to make it work. In that case, your 6th line should be:

ConvPoolLayer(image_shape=(mini_batch_size, 20, 38, 1),

and your 11th line should be:

n_in=40171, n_out=1000, activation_fn=ReLU, p_dropout=0.5),


Linear SVM with slack variables: Will it find a perfect decision boundary if it exists? Suppose I use a linear Support Vector Machine with slack variables on a dataset that is linearly separable.
Could it happen that the Support Vector Machine reports a solution that does not perfectly separate the classes?

As an illustration: Is the situation in the picture possible for a Support Vector Machine with slack variables? Although there is a "better" boundary that allows perfect classification, the Support Vector Machine goes for a sub-optimal solution that misclassifies two samples.

Having linear separable data means that an optimal solution exists. The support vector machine is an approach to identify the optimal solution with multiple steps through Lagrange multiplication. This method is guaranteed to give you the optimal solution if there is one.


How to maximize recall? I'm a little bit new to machine learning.
I am using a neural network to classify images. There are two possible classes. I am using a Sigmoid activation at the last layer so the scores of images are between 0 to 1.
I expected the scores to be sometimes close to 0.5 when the neural net is not sure about the class of the image, but all scores are either 1.0000000e+00 (due to rounding I guess) or very close to zero (for exemple 2.68440009e-15). In general, is that a good or bad thing ? How can this behaviour be avoided?
In my use case I wanted to optimize for recall by setting a lower threshold but this has no impact because of what I described above.
More generally, how can I minimize the number of false negatives when in training the neural net only cares about my not ad-hoc loss ? I am ok with decreasing accuracy a little bit to increase recall.

To answer the last question, suppose that you have a binary classification problem. It is customary to label the class as positive if the output of the Sigmoid is more than 0.5 and negative if it's less than 0.5. For increasing recall rate you can change this threshold to a value less than 0.5, e.g. 0.2. For tasks which you may want a better precision you can increase the threshold to bigger value than 0.5.

About the first part of your question, it highly depends on your data and its feature space. There are problems which the data is linearly separable in higher dimensions which means you can easily employ just a single neuron for classifying the data by a single hyper-plane. If it has happened that you have such a good accuracy you can not say anything unless you try to find the value of cross validation error. By interpreting the difference between the value of training data, and cross-validation or maybe test data, you can figure out whether your classifier performs well or not.


logistic regression - why exponent (log ratio) is linear I am new to ds and stats in general. I read numerous article to understand logistic regression. I got some idea why it works and how it fits to scattered plot of 1s and 0s when target variable is binary. however one piece of puzzle I still don't understand is how come someone derived that

ln(p/1-p) = B0 + B1X1 + ..

I see all articles assume that this is the link function and then go on talking about how it solves our regression problem for binary variable. but how this link function came about.

p is a probability so it is strictly between 0 and 1. So ln(p/(1-p)) is:

for p = 0: ln(0/1) = -Inf

for p = 1: ln(1/0) = +Inf

So now you've rescaled the probability to +/- Inf. In the GLM framework you can use practically any function that scales probability to +/-Inf (see any GLM textbook, or https://stats.stackexchange.com/questions/20523/difference-between-logit-and-probit-models).

How did it come about? Well, mathematicians and statisticians realised they needed a function with the above properties, had a think, came up with a few, decided on the ones that had tractable asymptotic properties, explored how they worked with real data and decided the sensible ones were logit (as above), probit (see references) and a few others. Of course you should always test your model assumptions against your data, and choice of link is just another one of those assumptions, just like assuming linearity in a covariate.


OpenNLP Coreference Resolution (German) I need to do coreference resolution for German texts and I plan to use OpenNLP to perform this task.

As far as I know OpenNLP coreference resolution does not support the German language.

Which components/data do I need to adapt the code such that it is possible to perform coreference resolution for German texts?

From an older version of the OpenNLP README:

  Training the Tools

  There are training tools for all components expect the coref component. Please consult the help message of the tool and the javadoc to figure out how to train the tools.

  The tutorials in our wiki might also be helpful.

  The following modules currently support training via the WordFreak opennlp.plugin v1.4 (http://wordfreak.sourceforge.net/plugins.html).

  coreference: org.annotation.opennlp.OpenNlpCoreferenceAnnotator (use opennlp 1.4.3 for training, models are compatible)

  Note: In order to train a model you need all the training data. There is not currently a mechanism to update the models distributed with the project with additional data.

As you can see, OpenNLP does not provide training tools for the coreference component. However, it seems at one point it was possible to train new models for OpenNLP's coref component using the third-party WordFreak plugin... however, it hasn't been updated in over a decade, so your mileage may vary.


RL agent, how to forbid actions? In Q-learning, how to tell the agent that action $a_7$  is unavailable from within state $s_{t}$?

Is supplying a very large negative reward good, or might throw it off-track?
From what I read (link), it's best to work with small rewards (between -1.0 and 1.0)

In Q-learning, how to tell the agent that action $a_7$  is unavailable from within state $s_{t}$?

Partially, it depends on what you mean by "unavailable". If the environment is such that it is possible to take the action, but that the consequences are very bad, then a negative reward due to the consequences is probably the best option. You might do this for a real-time system where the action can actually be taken, cannot be automatically blocked in any way, and has unwanted consequences.

Perhaps an action is unavailable, because it is not possible to even attempt it in the supplied environment. This could occur in a board game where certain moves are not allowed by the rules of the game. There you have a few options, depending on how you have constructed the agent.

A simple approach is to only present the agent with actions that it is allowed to take in the first place. The code for the environment should already know this list, so if you were considering penalising the choice, then you can also choose not to present it. This works well for value-based methods such as Q-learning, when you have implemented $Q(s,a)$. If the set of allowed actions is $\mathcal{A}(s)$, then the greedy action choice from state $s_t$ is $a_t = \text{argmax}_{a \in \mathcal{A}(s_t)} Q(s_t, a)$. All you need to do is implement $\mathcal{A}(s)$ in code and loop over different calls to Q(s,a) to find the best value.

If you have implemented the Q function in parallel over the whole action space (a good choice for efficiency with neural networks), then the network takes only the state $s$ as input, and outputs a vector e.g. $[Q(s, a_0), Q(s, a_1), Q(s, a_2), Q(s, a_4), Q(s, a_5) ... Q(s, a_N)]$. In that case you would use your implementation of $\mathcal{A}(s)$ to mask out unwanted actions before choosing the best action. It is a little bit wasteful to calculate the non-required actions, but if it was a rare enough exception, then probably still more efficient overall.

Beyond Q-learning, you could also use a similar action mask when you had a policy network that output softmax probabilities of actions. In that case you would also want to re-normalise the probabilities before generating the selected action.

Unless you have a specific goal or benefit in mind for the agent to learn about actions that are blocked by the environment, then it is simplest to query the environment to support the blocking, as described above. This is definitely the case for game-playing bots covering moves which are illegal according to the game rules. The justification is that the agent is learning to play the game, and the goal of it learning what the rules are for the game is a less interesting distraction.

Alternatively, it should work to allow the incorrect action selection, then give a negative reward (can be a small one, in same scale as other rewards in the system). Note this should also increment the time step and return the unaltered state. An agent will eventually learn not to take such a pointless, barred action. Maybe sometimes you are interested in whether the agent can learn the rules.


Printing all files' names from a folder (Kaggle kernel) train_dir= '../input/train'
for file in os.listdir(train_dir):
    print(file)

I am trying to get all the file names in a directory called "train", from a Kaggle dataset.

However, using the above code I just get a return of the folder "train", instead of all the images in the folder "train".

If I repeat the code above using:

data_dir = '../input'
for file in os.listdir(data_dir):
    print(file)

Then, it works correctly, it outputs the folder "train", the file "train.csv", the file "sample_submission.csv" and the folder "test" correctly.

Hence, I am quite puzzled why it doesn't work in the first case.

Thanks a lot. I am working directly on the Kaggle kernel by the way.

Usually a simple

import os
print(os.listdir("../input"))

works pretty well. Modify as required


Which Loss cross-entropy do I've to use? I'm working with this dataset https://www.kaggle.com/c/sf-crime to predict the crime incident using keras. I've encoded the category with pd.get_dummies and then use it as the validation data. At first I try to use categorical_crossentropy and adam for the loss function but result returns with very bad acc which nearly 0.3 and about 2.7 of loss. So I try to experiment with different entropy by using binary_crossentropy. The result seem to be exaggerate the acc is higher than 0.97 and the loss is less than 0.1 at first epoch. I also provide 0.3 of drop out for each layer. So which crossentropy is accurate or better use for this scenario and which kind of this classification is multi-label or multi-class?

which kind of this classification is multi-label or multi-class?

If we take a look at the data panel at the competition's page in kaggle:

  Category - category of the crime incident (only in train.csv). This is the target variable you are going to predict.

The target we are trying to predict is the category of the crime (which can take multiple values) so this answers your question: the problem is multi-class.

  So which crossentropy is accurate or better use for this scenario?

There are three main types of cross-entopy in keras:

categorical cross-entropy is used for multi-class classification, where the target variable is one-hot encoded.
sparse categorical cross-entropy is used for multi-class classification, where the target variable is not one-hot encoded.
binary cross-entropy is used for binary classification problems.

From your question I assume that you think the different types of crossentropy are interchangeable. That is not the case. You should select the cross-entropy based on how your problem and how you encoded your data.

In your case, since you have a multi-class problem and you state that you have used pandas.get_dummies(), which ont-hot encodes your data you should use categorical cross-entropy and nothing else!


How to train a xgboost model on data that is too big for the memory? What are the best practices to train xgboost (eXtreme gradient boosting) models on data that is to big to hold it in memory at once? Splitting the data and train multiple models? Are there more elegant solutions?

You can train xgboost, calculate the output (margin) and then continue the training, see example in boost from prediction.

I‘ve not tried it myself, but maybe you could train on the first subset of your data (say 10%) and then continue on another subset, etc.

Update

Step by step procedure

Split the data into N manageable subsets, set n=1
Train xgboost on n-th subset
Calculate the prediction (margin) for n+1 subset using the model obtained from previous
Add the margin into the n+1 subset via setinfo
Increment n

Steps 2-5 to be repeated N times.


How to tackle too many outliers in dataset I boxplot all of my columns with seaborn boxplot in order to know how many outliers that i have, surprisingly there're too many outliers and so i can remove the outliers because i'm afraid with too many outliers it will have bad impact to my model especially impacting the mean,median, variance which will further impact the performance of my model. Then i found this QnA about too many outliers the answer said that the case is not really an outliers. The answer makes sense but i'm still afraid it will give bad impact to the performance of my model. What is the best thing should i do?

These are two of examples of what i mean, 9 out of 10 columns of my data are exactly like this and i'm really worried about this because you see, it's not only there're too many outliers, it also happens in almost every (9 of 10 columns in total) columns.

It looks like most of your data is at 0, right? Either:

Those values are not really 0 but 'missing' and should be treated as missing, rather than as 0. If a lot is missing, this will indeed be problematic for models that do not handle these, because you have to infer some value for them. You can't really drop all these if there are a lot. You could impute a mean or pick a model that handles missing values. Or:

The distribution of these values isn't normal/Gaussian, in which case there's no strong reason to believe that values outside the IQR are 'outliers'. In particular looks like it could be exponential, or a mixture of two distributions in the second case. Just use the values as is. For linear models, some transformation of the input might be better, like its log, but depends a lot on what you're doing.


When creating a classification model, should predictors with little correlation to the response variable be included in the model? I am building a predictive model designed to predict attrition within my organization.  I am trying to decide whether to add certain predictors to my model.  I used a Kruskal-Wallis rank sum test to check the correlation between a few of my predictor  variables and my response variable and found that the predictors are independent from my response.  Should I still include these variables in my model? I am leaning towards no due to the lack of correlation with my response variable but don't want to discard a potential "split" in my decision tree.  My data set contains 8,225 observations and 173 columns which I have been using as predictors.

In general, a Kruskal-Wallis test (or any other univariate test) cannot guarantee that there are no useful interactions with other features that a decision tree might still pick up. That means you won't know for sure until you try it.

You can test the performance with and without these "weak" features and compare results. Make sure you control for overfitting when including all features. Alternatively, after fitting the decision tree, you can look at the target response when varying the feature values ("partial dependence").


Which open-source sgdb for kind of large data I have a 7 giga confidential dataset which I want to use for a machine learning application.

I tried :

Every package recommanded for efficient dataset management in R like :

data.table,
ff
and sqldf with no success.

Data.table needs to load all the data in the memory from what I read, so it's obvious that it will not work since my computer has only 4g RAM. Ff leads to a memory error too.

So I decided to turn to sgdb and I tried :

Mysql which managed to load my dataset in 2 hours and 21'. Then I began my requests (I have a few requests to do to prepare my data before I export a smaller set in R for machine learning application), and then I had to wait for hours before I got the following message "The total number of locks exceeds the lock table size" (my request was just an update to extract the month from a date for each tuple).
I read that postgre was similar to mysql in performance so I didn't try
I read that redis was really performant but not at all adapted to massive importation like I want to do here so I didn't try
I tried mongoDb, the nosql upraising solution that I heard everywhere about. Not only I find rather disturbing that mongoimport is so limited in options (I had to change all semi-colon in commas using sed before I can import the data), but It seems to be less performant that mysql since I launched the loading yesterday and it is still running.

What I can't try : data are confidential so I don't really want to rent some space on Azure or Amazon clouding solution. I am not sure that it is that big that I have to turn to Hadoop solution but maybe I am wrong about that.

Is there an open-source performant solution that I didn't try that you would recommend to perform some sql-like requests on a biggish dataset ?

Edit :
Some more details about what I want to do with these data for you to visualize. These are events with a timestamp and a geolocalisation. I have 8 billions of lines. One example of what I want to do :

standardize series identified by geolocalisation (I need to compute mean grouping by geolocalisation for example),
compute average count of events by type of season, day... (usual group by sql request)...

Edit

As a beginning of answer for those who have limited hardware like me, rSQLite seems to be a possibility. I am still interested in other people experiences.

analyzing 8 billion lines on a 4gb computer is pretty silly, but you can try

http://www.asdfree.com/2013/03/column-store-r-or-how-i-learned-to-stop.html


In softmax classifier, why use exp function to do normalization? Why use softmax as opposed to standard normalization? In the comment area of the top answer of this question, @Kilian Batzner raised 2 questions which also confuse me a lot. It seems no one gives an explanation except numerical benefits.

I get the reasons for using Cross-Entropy Loss, but how does that relate to the softmax? You said "the softmax function can be seen as trying to minimize the cross-entropy between the predictions and the truth". Suppose, I would use standard / linear normalization, but still use the Cross-Entropy Loss. Then I would also try to minimize the Cross-Entropy. So how is the softmax linked to the Cross-Entropy except for the numerical benefits?
As for the probabilistic view: what is the motivation for looking at log probabilities? The reasoning seems to be a bit like "We use e^x in the softmax, because we interpret x as log-probabilties". With the same reasoning we could say, we use e^e^e^x in the softmax, because we interpret x as log-log-log-probabilities (Exaggerating here, of course). I get the numerical benefits of softmax, but what is the theoretical motivation for using it?

It is more than just numerical. A quick reminder of the softmax:
$$
P(y=j | x) = \frac{e^{x_j}}{\sum_{k=1}^K e^{x_k}}
$$

Where $x$ is an input vector with length equal to the number of classes $K$. The softmax function has 3 very nice properties: 1. it normalizes your data (outputs a proper probability distribution), 2. is differentiable, and 3. it uses the exp you mentioned. A few important points:

The loss function is not directly related to softmax. You can use standard normalization and still use cross-entropy.
A "hardmax" function (i.e. argmax) is not differentiable. The softmax gives at least a minimal amount of probability to all elements in the output vector, and so is nicely differentiable, hence the term "soft" in softmax.
Now I get to your question. The $e$ in softmax is the natural exponential function. Before we normalize, we transform $x$ as in the graph of $e^x$:

If $x$ is 0 then $y=1$, if $x$ is 1, then $y=2.7$, and if $x$ is 2, now $y=7$! A huge step! This is what's called a non-linear transformation of our unnormalized log scores. The interesting property of the exponential function combined with the normalization in the softmax is that high scores in $x$ become much more probable than low scores.

An example. Say $K=4$, and your log score $x$ is vector $[2, 4, 2, 1]$. The simple argmax function outputs:

$$
[0, 1, 0, 0]
$$

The argmax is the goal, but it's not differentiable and we can't train our model with it :( A simple normalization, which is differentiable, outputs  the following probabilities:

$$
[0.2222, 0.4444, 0.2222, 0.1111]
$$

That's really far from the argmax! :( Whereas the softmax outputs:
$$
[0.1025, 0.7573, 0.1025, 0.0377]
$$

That's much closer to the argmax! Because we use the natural exponential, we hugely increase the probability of the biggest score and decrease the probability of the lower scores when compared with standard normalization. Hence the "max" in softmax.