Log loss vs accuracy for deciding between different learning rates? While  model tuning using cross validation and grid search I was plotting the graph of different learning rate against log loss and accuracy separately.

Log loss

When I used log loss as score in grid search to identify the best learning rate out of the given range I got the result as follows:

Best: -0.474619 using learning rate: 0.01

-0.674328 (0.000482) with: learning rate: 0.0001
-0.583335 (0.003236) with: learning rate: 0.001
-0.474619 (0.004336) with: learning rate: 0.01
-0.494540 (0.008705) with: learning rate: 0.1

Accuracy

When I used accuracy as score in grid search to identify the best learning rate out of the given range I got the result as follows:

Best: 0.781958 using learning rate: 0.1

0.656220 (0.085705) with: learning rate: 0.0001
0.715279 (0.010021) with: learning rate: 0.001
0.740141 (0.007927) with: learning rate: 0.01
0.781958 (0.003770) with: learning rate: 0.1

In both cases I got different learning rates that I should use to tune my model. When the score is log loss, I got optimum setting for learning rate as 0.01. When score is accuracy, I got optimum setting for learning rate as 0.1.

In such cases, what score should I use for my model?

According to me, it is not correct to co-relate loss with accuracy.

  Loss is used to optimize the hypothesis such that we can get best
  weights whereas accuracy is used to identify how well model is doing in term
  of correctly predicting the values.

Model internally takes the reference of predict_proba() and returns 1 if probability is > .5 otherwise 0. For example if returned predict_proba() is (.49, .51), model will return 1 as an classification output.

Now consider a use case where some trained model is used for test-data prediction. Assumed such model has 100% accuracy but predict_proba() value close to (.49,.51) or (.51,.49) i.e. having very low confidence level.

In such case logloss is quite high, even though it's having 100% accuracy.

If our criteria for model selection is "accuracy value" then we are selecting the bad model.


Question Regarding Multi-label probability predictions I have been doing a problem in which I have to predict probabilities for each of the labels in a multi-label (four to be precise) classification problem.

Example of a solution:

Id, North,          East,            West,           South
1,  0.71663940211,  0.037567315693,  0.03525987339,  0.0021068944991
...

The training data is of the form where each y(i) is labelled either 0,1,2 or 3 (encoded for N,E,W & S respectively)

I will be grateful if you can just tell me how to approach this problem.
Links giving direct insight to the problem will also sufficient.

A question is a bit broaden as you do not specify if you do not know how to do it in theory or how to tackle it with a ML method.

Some of ML methods:

LogisticRegression in sklearn handles multiple class

lr = LogisticRegression()
lr.fit(X, y)
class_probabilities = lr.predict_proba(X)  # outputs the probabilities

You might also want to consider Support Vector Machines.

Theory:

You can do a "one vs rest" when you train a single classifier per class taking the sample of all other classes as negative example. (see wiki article for that)


Stratify on regression I have worked in classification problems, and stratified cross-validation is one of the most useful and simple techniques I've found. In that case, what it means is to build a training and validation set that have the same prorportions of classes of the target variable.

I am wondering if such an strategy exists in regression. A simple approach would be to split the data in quartiles or deciles and make sure that the proportions of training and validation instances in the respective quartiles and deciles are the same.

The question is: is there a standard way to do this? if so, is there an implementation in sklearn?

I have done this before and didn't find a default implementation - the StratifiedKFold and RepeatedStratifiedKFold are only documented to work with classes.

The way I ended up doing it was not quite as you are thinking with quartiles/deciles, but rather using a histogram (it matched my needs).

NumPy has a nice method for doing this, with many different formulas for computing the bin sizes that you can play around with to be match your data i.e. if it is normally distributed or not. I can't post the entire method code, but here is the gist:

samples_per_bin, bins, = np.histogram(data, bins='doane')    # Doane's method worked best for me

min_bin_size = samples_per_bin.min()

# compute the maximum batch size possible, using all samples from the bin with lowest population
n_bins = len(samples_per_bin)
max_batch = min_samples_single_bin * n_bins

I then put the data into a Pandas DataFrame and added a column indicating which bin each data point was in - finally doing something like this to perform the sampling from each of the bins:

df.groupby('bin_name', group_keys=False).apply(
            lambda x: x.sample(n_per_group, replace=True))

Obviously you can allow duplicates in a batch or not by changing the replace argument. It might be necessary if you want a larger batch size while forcing stratification. It is definitely a limitation of my approach, which you might be able to overcome by using quartiles etc as you suggest.


BERT minimal batch size Is there a minimum batch size for training/re-fining a BERT model on custom data?
Could you name any cases where a mini batch size between 1-8 would make sense?
Would a batch size of 1 make sense at all?

Small mini-batch size leads to a big variance in the gradients. In theory, with a sufficiently small learning rate, you can learn anything even with very small batches.
In practice, Transformers are known to work best with very large batches. You can simulate large batches by accumulating gradients from the mini-batches and only do the update once in several steps.
Also, when finetuning BERT, you might also think of fine-tuning only the last layer (or several last layers), so you save some memory on the parameter gradients and can have bigger batches.


Why is orthogonal weights initialization so important for PPO? I have implemented PPO to solve Atari environments. For the longest time I couldn't figure out why my model would not converge as fast as other open source solutions. Eventually it boiled down to this single weights initialization (in PyTorch) for the CNN:

init_ = lambda m: init(m, nn.init.orthogonal_, lambda x: nn.init.constant_(x, 0), nn.init.calculate_gain("relu"))

which is then applied to each of the layers. In other words, the neural network HAS to be initialized orthogonally. Once I added this, my model converges as expected.

Why is it that orthogonal initialization is so much superior to xavier initialization for PPO?

See this paper's (Exact solutions to the nonlinear dynamics of learning in
deep linear neural networks) result:

Moreover, we introduce a mathematical condition for faithful backpropagation of error signals, namely dynamical isometry, and show, surprisingly that random scaled Gaussian
initializations cannot achieve this condition despite their norm-preserving nature, while greedy pre-training
and random orthogonal initialization can, thereby achieving depth independent learning times. Finally, we
show that the property of dynamical isometry survives to good approximation even in extremely deep nonlinear random orthogonal networks operating just beyond the edge of chaos.

I think this is an answer to your question.


Is it common to add noise to Time Series data before training a model I once read about somebody who added noise to their time series before training a model. They didn't write why they did it though.

Is this common practice?

If it is, why do people do it ie. to prevent over-fitting?

I'll go through your questions one by one:

  I once read about somebody who added noise to their time series before training a model. They didn't write why they did it though.

No, noise is not adding any value to your dataset. The reason is that noise, by definition, cannot be learned by any statistical/ML model.

  Is this common practice?

No it's not.

  why do people do it ie. to prevent over-fitting?

It's not a technique that helps you in preventing overfitting. There are several techniques to do it (ensemble modeling, cross validation), many of them depend on the kind of model you want to implement (such as recurrent dropout for RNNs).


Linear Regression. Interpret weights as an effect on a prediction? In a basic linear regression, can I use the weights of each explanatory variable to describe their relative effects on the predicted value? If parameter A has a weight of 100, and parameter B has a weight of 10, can I say that parameter A has 10 times more effect on the outcome than parameter B?

An example: From a weather data set, I used two parameters, humidity and pressure, to make a prediction on temperature.

When I plot humidity vs. temperature, there's a very clear inverse relationship between the two. That is, the scatter plot tends down and to the right.

When I plot pressure vs. temperature, there's no relationship at all. The pressure in this report has very little variance, and the scatter plot is nearly vertical.

Based on those two plots, I intuit that changes in humidity have an effect on the high temperature, while changes in pressure have almost no effect.

I built a linear regression model (gradient descent, by hand, in Octave). Humidity had a weight of -.32386. Pressure had a weight of -.02219. Humidity's weight was 14.6 times larger than pressure's weight.

Based on that, can I state that parameter Humidity has nearly 15 times more effect on Temperature than parameter Pressure?

Forgive what must seem like a very amateur question. If you can give a relatively simple answer and point me toward resources where I can learn more, I'd really appreciate it. Thanks in advance.

In a basic linear regression, can I use the weights of each
  explanatory variable to describe their relative effects on the
  predicted value? If parameter A has a weight of 100, and parameter B
  has a weight of 10, can I say that parameter A has 10 times more
  effect on the outcome than parameter B?
  Blockquote

Somewhat yes. Precisely it means if A changes by one unit the outcome will change 100 times, on average (while B is frozen because it is a multiple linear regression -> you have more than one independent variable).

There are tons of examples and blogs explaining it, see for example here .
Also you can analyze your regression model more by doing some statistical tests like identifying how variables are correlated using Variance Inflation Factors and estimate the statistical significance (p-value, F-test), here is a very nice intro.

  An example: From a weather data set, I used two parameters, humidity
  and pressure, to make a prediction on temperature.

I do not see any problem with your analysis. As you said you see an inverse correlation between humidity vs. temperature in your plot as well as via your -.32386 weight. While the weight of -.02219 for pressure is nearly zero and negligible and obviously you won't see such relationship between pressure-temperature.


Calculation of VC dimension of simple neural network Suppose I have a perceptron with one-hidden layer, with the input - one real number $x \in \mathbb{R}$, and the activation function of the output layers - threshold functions:
$$
\theta(x) =
\begin{cases}
0, x \leq 0 \\
1, x > 0
\end{cases}
$$
The hidden layer may contain $k$ units and I would like to calculate the VC dimension of this feed-forward neural network. The VC dimension is defined as a cardinality of the maximal set, that can be shattered by properly adjusting the weights of the neural network.

The threshold functions have a VC dimension of $n+1$, where $n$ is a number of input neurons, because by a plane $n-1$ plane one may split $n$ points in any way. So when considering the results in the first layer, we have a VC dimension of $2$ for each gate, and the total number of points, that can be separated by the activation is $2 k$. Then we have a vector $\in \mathbb{R}^k$ to be processed to output, and the output unit has a dimension $k + 1$.
Do I understand correctly, that the resulting VC dimension of this simple neural network is :
$$
2 k + k + 1 = 3k + 1
$$

I do not believe this is correct.  The entire network will represent a piecewise-constant function with at most $k+1$ pieces, and has VC dimension $k+1$.
Each hidden neuron is a step function, and together there are at most $k$ jump points among them.  Taking a linear combination of those, we still cannot create any new jump points, so at the output neuron before activation, we have a piecewise-constant function with at most $k$ jump points, and arbitrary constant values on the $k+1$ intervals between them.  After the activation, it's just piecewise constant with values 0 and 1 on at most $k+1$ intervals.


Need help in understanding a small example Pardon me, I agree the title of the question is not clear. I would like to know the understanding of below steps which are picked from the textbook "Hands on machine learning".

 >>> housing['income_cat'].value_counts()
 >>> 3.0    7236
     2.0    6581
     4.0    3639
     5.0    2362
     1.0     822

  If I am not wrong, the above step is to get the counts for each class. For example, for class '3' there are 7236 instances. Likewise, for class '2' there are 6581 instances.

>>>  housing['income_cat'].value_counts / len(housing)
>>>  3.0    0.350581
     2.0    0.318847
     4.0    0.176308
     5.0    0.114438
     1.0    0.039826

  Next, I was not clear, what was the intention behind the above step. By doing the above step, what am I suppose to learn ?. and

 >>> from sklearn.model_selection import StratifiedShuffleSplit

 >>> split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
 >>> for train_index, test_index in split.split(housing, housing["income_cat"]):
         strat_train_set = housing.loc[train_index]
         strat_test_set = housing.loc[test_index]

 >>>  strat_test_set['income_cat'].value_counts() / len(strat_test_set)
 >>>  3.0    0.350533
      2.0    0.318798
      4.0    0.176357
      5.0    0.114583
      1.0    0.039729
      Name: income_cat, dtype: float64

  How come strat_test_set['income_cat'].value_counts() / len(strat_test_set) results are almost same to the results of housing['income_cat'].value_counts / len(housing) ?

Regarding your first question, by dividing by the length, you get the percentage of each category, and you can see that one category has a very low percentage.

If you use the regular train_test_split, the proportion of the categories will be different in each set, as the split is made randomly, and this can introduce a bias.

Imagine categories with a very low number of observations, you could even have categories missing entirely in either of the sets, which will cause trouble for your model.

The use of stratified sampling allows you to have the same proportion for the categories. That is why you see the same results: it is the goal of this sampling.


In R's randomForest, is predict() non-deterministic? Training a random forest model is inherently non-deterministic (absent control over the random number generator), but is predict() also non-deterministic? That is, if I construct randomForest (with an odd number for ntree per the caveat in the doc) and save an .rda, will loading that .rda give me identical results given identical inputs?

I tried answering this myself by looking through https://github.com/cran/randomForest , and it seems that nobody along the predict() path is calling for a random number, but I'm very new to R and rather rusty with C and may be missing something. Pointers into the code or the docs will be appreciated.

The model will not change unless you re-train it.
The same input sample should always have the same output value for a given model.
changing the seed, saving and reloading, etc should have no impact on the results.

The training itself is indeed non-deterministic, predict is not.

library(randomForest)
# sample 80% of data to train
split = sample(1:nrow(iris), floor(nrow(iris) * 0.8))
df_train = iris[split,]
df_test = iris[-split,]
# rf model
mod = randomForest(Species ~ ., df_train)
# predictions
set.seed(123)
res1 = predict(mod, df_test)
set.seed(999)
res2 = predict(mod, df_test)
identical(res1, res2)


Classifcation of non-linear regressions based on their shapes I have a data set of thousands of individual y ~ x relationships that can have varying shapes in their relationships. For example, they can follow an exponential, asymptotic, logistic or hump-shaped (with different skewness) pattern. I actually don't know all the shapes the relationships can take, these are just a few patterns that I hypothesise to emerge.

I need a way to classify each of these relationships into one of the non-linear patterns without looking at each individual relationship. I was wondering whether there is way to do this in machine learning. I am willing to sit down and classify maybe some of the patterns myself but would like to avoid classifying all of them by hand.

Ultimately, I'm not interested into prediction. The regression should only serve as a way to describe different non-linear patterns and classify each individual into a category based on the shape of the regression.

What I am doing so far is fitting non-linear regressions with different equations to each relationship and decide which of the non-linear regressions (exponential, asymptotic etc) is fitting best by evaluating the residual variance and the p-value. However, I was wondering whether there is a way to do this with machine learning instead.

As a quick brainstorming idea, you could create new attributes based on your original ones.
Let's say you have a 2D feature set x and y; so, you could test if adding x2, y2 , x•y, x3... as new attributes, and applying regularization, gives you as important features the ones you expect for each type of regression (test it with hand made examples for exponential regression, logarithmic, etc)


Simulating datasets of species abundances with varying degrees of turnover in R I am working with a data set containing the abundance of 100+ species at 50+ locations over 20+ time-points. I want to simulate equivalent data sets with varying degrees of community turnover i.e. abundant species staying abundant/rare staying rare --> abundant becoming rare and vice versa while maintaining the species abundance distribution in the original data set. I have seen mentions of resampling simulation techniques using bootstrapping where the desired difference between "template" and simulated data sets can be specified in terms of the proportion of all species that were present in both assemblages.

I am wondering if that is the best way forward to get the desired simulated data sets and if anyone could point me towards a tutorial on how to do this in R/give me some tips themselves.

Thanks!

Without any example of your data or distributions it is a hard question to answer. Without further info I think there are two approaches:

resample your data using stratified sampling. The R sample package can be of good help here.
generate a multivariate distribution, as far as I know there are no packages that will directly do this for you. Depending on the distributions of your data, creating the right multivariate distribution which includes the right correlations may be very hard. In some cases you may even need to use copulas (this is especially necessary if you are interested in tail events, correlation will not be sufficient to create dependence of these events). Note that this topic can be a separate PhD thesis on itself.

I hope this helps.


Dataset where svm performance is significantly different from random forest Is there a specific dataset where svm performs significantly better or worse than random forest?

I know that the performance could depend on the dataset but is there a specific dataset?

Geographical datasets (e.g. when predicting population density from the built environment characteristics in my case) are, at least from my experience, one of the cases where SVM performs consistently worse than random forests. Although I have not investigated the issue in detail, it seems that random forests are more resistant to high amounts of noise and can learn to apply different rulesets in different regions or environments, something which SVM (SVR) fails to do.


When to question output of model I'm unsure of how to ask a question without making it seem like a code review question. At what point does one question whether they've actually implemented the algorithm and-or model correctly? Getting spot-on results is great and all, but seems highly suspect. Also, what checks can be done to ensure that the algorithm and-or model is being implemented correctly? The reason I'm asking is because I'm getting perfect classification and subsequently accuracy, precision, etc. w/ the implementation of SVM.

I am including the code, but feel free to ignore.

# Make a copy of the df
iris_df_copy = iris_df.copy()

# Create a new column, labeled 'T/F', whose value will be based on the value in the 'Class' column. If the value in the
# 'Class' column is 'Iris-setosa', then set the value of the 'T/F' column to 1. If the value in the 'Class' column is
# not 'Iris-setosa', then set the value of the 'T/F' column to 0.
iris_df_copy.loc[iris_df_copy.Class == 'Iris-setosa', 'T/F'] = 1
iris_df_copy.loc[iris_df_copy.Class != 'Iris-setosa', 'T/F'] = 0

X_svm = np.array(iris_df_copy[['Sepal_Length', 'Sepal_Width', 'Petal_Length', 'Petal_Width']])
y_svm = np.ravel(iris_df_copy[['T/F']])

# Split the samples into two subsets, use one for training and the other for testing
X_train_svm, X_test_svm, y_train_svm, y_test_svm = train_test_split(X_svm, y_svm, test_size=0.25, random_state=4)

# Instantiate the learning model - Linear SVM
linear_svm = svm.SVC(kernel='linear')

# Fit the model - Linear SVM
linear_svm.fit(X_train_svm, y_train_svm)

# Predict the response - Linear SVM
linear_svm_pred = linear_svm.predict(X_test_svm)

# Confusion matrix and quantitative metrics - Linear SVM
print("The confusion matrix is: " + np.str(confusion_matrix(y_test_svm, linear_svm_pred)))
print("The accuracy score is: " + np.str(accuracy_score(y_test_svm, linear_svm_pred)))
print("The precision is: " + np.str(precision_score(y_test_svm, linear_svm_pred, average="macro")))
print("The recall is: " + np.str(recall_score(y_test_svm, linear_svm_pred, average="macro")))

You need to know what the outcome should be of a given test on a dataset before you try to test a new method on them. Ask yourself, 'What do I expect from this?'

Linear SVM finds a plane to cut through the data to best represent the difference between two sets.

If you have a look at what you are separating (Iris_setosa from Iris_virginica and iris_versicolor), you'll find that the clumps themselves are perfectly separated. You can draw a line easily on each graph you care to use, and that is what I have done in the picture below. If the clumps are perfectly separated, then the SVM will return a perfectly separated result.

By Nicoguaro - Own work, CC BY 4.0, https://commons.wikimedia.org/w/index.php?curid=46257808

Test the SVM on separating virginica and versicolor to see how it does in a more difficult context. Or alternatively, just generate a dataset of your own from randomly placed gaussian points.


Compute specificity and sensitivity at certain thresholds I have the following table with predictive probabilities and true class labels:

\begin{array}{|c|c|c|c|}
\hline
P(T=1) &0.54& 0.23 & 0.78 & 0.88 & 0.26 & 0.41 & 0.90 & 0.45&0.19&0.36 \\ \hline
 T&1&0 &0 &1 &0 &0& 1& 1& 0& 0\\ \hline
\end{array}

The question is to compute the specificity & sensitivity at the threshold of 0.5.

My attempt at answering this question:

Sensitivity = true positive rate[P(T=1) > 0.5]

= (0.54 + 0.88 + 0.9)/4 = 0.58

Specificity = 1-false positive rate[P(T=1) > 0.5]

= 1- [(0.78)/6]
 = 0.87

Not sure if my working above is correct. I would appreciate if someone can guide me to the correct solution. Thanks.

For threshold = $0.5$ we have:

Sensitivity = True Positive Rate

= (number of points with label $1$ and $P(T = 1)\geq 0.5$) divided by (number of points with label $1$)

= $\left|\{(1, 0.54), (1, 0.88), (1, 0.90)\}\right| / 4$ = $3/4$ = $0.75$

Specificity = 1 - False Positive Rate

= 1 - (number of points with label $0$ and $P(T = 1)\geq 0.5$) divided by (number of points with label $0$)

= $1 - \left|\{(0, 0.78)\}\right|/6$ =  $1 - 1/6$ = $0.833$


User actions sequence classification I have a training set where each row is a series of user actions on a website (logged in, sent an invoice, etc.) and times deltas in ms between these actions. Each row has a label — a corresponding user class (10 classes):

action132, 2340, action21, 300, ...  3
action238, 1240, action22, 350, ...  6
...
action763, 1240, action42, 750, ...  2

Say, I have 500 unique action types and each row has a variable length. Max sequence length is 100k and there are 10k rows in the training set (10k users).

I need to predict labels for the test set.

If I had only the action sequence, it would be a more understandable task similar to DNA sequence classification of text classification which are normally solved with LSTM/CNN models or transformers. But in my problem these time deltas are good indicators of user behavior and I'd like to make use of these features. Also, the action order is important as well.

What would be an efficient model for this problem?

NN is automatic feature extractor, it will find columns that you suspect are valuable, if you encode it right

Dont worry about indicating specifically these columns. Worry about optimising your DNN architecture. Since you say you have variable input lengths, one optimisation method would be utilising dynamic sequence bucketing

There is no ultimate theory (without some serious assumptions being made) that says if only X is satisfied than you can be sure Y will beat all other methods, or that it will converge. Try a couple of different approaches in all stages (models, preprocessing etc) and see what works. Just make sure you make minimum viable experiment enviroment, i.e. representative train/test set where you can quickly iterate at test different approaches


Feature selection for time series prediction I'm working on an LSTM-based stock market forecasting problem and trying to figure out a way to select input variables.

When calculating correlation between variables (e.g. Close price of Tesla vs Close price of Microsoft), would differentiating the curves give a more accurate (or correct) correlation index ? I'm finding values in the range 0.7-0.9 for non-differentiated variables, and lower values after differentiation.
Once I have a correlation matrix of all my variables, is there a way to figure out which ones would add information to the neural net and which ones would just add noise ?

You don’t need to select variables for feeding to network, deep neural networks (DNN) will do this automatically. Actually DNN gives more importance to relevant variables by setting its weights. After setting the weights, some of the hidden nodes take 0 and some of them take 1 (because of sigmoid function). You can think of this 1 and 0’s as choosing relevant variables, too.

By the way, correlation matrix can not be used to select relevant variables directly. If you want to reduce the number of variables that are fed to DNN, you can use PCA. Actually PCA components are calculated by getting the Eigen-vectors of correlation matrix.


Can the attention mask hold values between 0 and 1? I am new to attention-based models and wanted to understand more about the attention mask in NLP models.

attention_mask: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices
selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max
input sequence length in the current batch. It's the mask that we typically use for attention when
a batch has varying length sentences.

So a normal attention mask is supposed to look like this, for a particular sequence of length 5 (with last 2 tokens padded) --> [1,1,1,0,0].
But can we have attention mask like this --> [1, 0.8, 0.6, 0, 0] where values would be between (0 and 1) to indicate that we want to pay attention to those tokens, but it's result wouldn't be completely effective on the model's result due to it's lower attention weights (kinda of like dealing with class imbalance where we weight out certain classes to deal with imbalance).
Is this approach possible? is there some other way to have the model not use the information presented by some tokens completely?

In theory maybe yes, but you would probably need to reimplement the model yourself.
In practice, with the current implementations, probably no. (Judging from the documentation snippet, you use Huggingface Transformers.) The documentation says it expects a LongTensor, i.e., a tensor with integer values. Internally, the attention mask is used to compute sequence lengths, but summing the mask along dimension 1. This would need to be fixed and there might many other places in the code just assume the mask values are zeros and ones.


Heuristic argument for Weight decay and regularization I have been reading this book and I have no idea how to make an argument for this problem. http://neuralnetworksanddeeplearning.com/chap3.html

Connecting regularization and the improved method of weight initialization L2 regularization sometimes automatically gives us something similar to the new approach to weight initialization.

Suppose we are using the old approach to weight initialization. Sketch a heuristic argument that:

(1) supposing $λ$ is not too small, the first epochs of training will be dominated almost entirely by weight decay;

(2) provided $ηλ≪n$ the weights will decay by a factor of $e^{\frac{−ηλ}{m}}$ per epoch; and

(3) supposing λ is not too large, the weight decay will tail off when the weights are down to a size around $\frac{1}{\sqrt{n}}$, where n is the total number of weights in the network. Argue that these conditions are all satisfied in the examples graphed in this section.

Can anyone explain a little bit more and guide my direction? Thank you so much for reading it.

My argument is that:

(1) I am thinking about this formula $w' = w(1-\frac{ηλ}{n}) - \frac{n}{m} \sum \frac{dCx}{dw}$ is related. Since the weight is larger without using new weight initialization (the standard deviation of $w = \frac{1}{\sqrt{total number of weights in the network}}$, $\frac{ηλw}{n}$ will be significantly larger than sum of $\frac{dCx}{dw}$ （the average rate of change of $C$ in respect of weight for a minibatch).

(2) By substituting $ηλ+b = n$ into $w'$??

(3)

Thank you again.

Perhaps I can answer the first and second one:

(1)According to (93) on his book:
$w \rightarrow w=\left(1-\frac{\eta \lambda}{n}\right) w -\frac{\eta}{m}
  \sum_x \frac{\partial C_x}{\partial w}$,
there is an item $(1-\frac{\eta \lambda}{n})$on the weight, so weights will shrink every epoch. As weights were random chose at the beginning, it seems reasonable that the changes(the item $\frac{\eta}{m}\sum_x\frac{\partial C_x}{\partial w}$) will be chaotic on first epoch. Maybe that is why the weight decay will dominate the first epoch.
And there is another explanation here, which may be helpful.

(2)At each epoch, the item $(1-\frac{\eta \lambda}{n})$ will be multiplied $\frac{n}{m} $times, where $n$ is the number of training data, $m$ is the size of minibatch. So after each epoch, the weight will decay by a factor of$(1-\frac{\eta \lambda}{n})^\frac{n}{m}$. According to the definition of $e$:$e=\lim_{n\to \infty}(1+\frac{1}{n})^n$, lets change the format a little bit:

$(1-\frac{\eta \lambda}{n})^\frac{n}{m}=[(1-\frac{\eta \lambda}{n})^{-\frac{n}{\eta \lambda}}]^{-\frac{\eta \lambda}{m}}$

As you can see, when $\eta \lambda \ll n$,  $(1-\frac{\eta \lambda}{n})^{-\frac{n}{\eta \lambda}} \approx e$.Then we can come to the conclusion that provided $ηλ≪n$ the weights will decay by a factor of $exp(−ηλ/m)$ per epoch.

(3)I have been through this for two days and have no idea about it,but I think it should be related to the formula(85):

$\begin{eqnarray} C = -\frac{1}{n} \sum_{xj} \left[ y_j \ln a^L_j+(1-y_j) \ln
(1-a^L_j)\right] + \frac{\lambda}{2n} \sum_w w^2.
\end{eqnarray}$

where the weights were summed on last item.


How does the naive Bayes classifier handle missing data in training? Naive Bayes apparently handles missing data differently, depending on whether they exist in training or testing/classification instances.
When classifying instances, the attribute with the missing value is simply not included in the probability calculation (reference)
In training, the instance [with the missing data] is not included in frequency count for attribute value-class combination. (reference)
Does that mean that particular training record simply isn't included in the training phase? Or does it mean something else?

In general, you have a choice when handling missing values hen training a naive Bayes classifier. You can either choose to either

Omit records with any missing values,
Omit only the missing attributes.

I'll use the example linked to above to demonstrate these two approaches.  Suppose we add one more training record to that example.

Outlook  Temperature  Humidity   Windy   Play
-------  -----------  --------   -----   ----
rainy    cool        normal    TRUE    no
rainy    mild        high      TRUE    no
sunny    hot         high      FALSE   no
sunny    hot         high      TRUE    no
sunny    mild        high      FALSE   no
overcast cool        normal    TRUE    yes
overcast hot         high      FALSE   yes
overcast hot         normal    FALSE   yes
overcast mild        high      TRUE    yes
rainy    cool        normal    FALSE   yes
rainy    mild        high      FALSE   yes
rainy    mild        normal    FALSE   yes
sunny    cool        normal    FALSE   yes
sunny    mild        normal    TRUE    yes
NA       hot         normal    FALSE   yes

If we decide to omit the last record due to the missing outlook value, we would have the exact same trained model as discussed in the link.
We could also choose to use all of the information available from this record.  We could choose to simply omit the attribute outlook from this record. This would yield the following updated table.

           Outlook            Temperature           Humidity
====================   =================   =================
          Yes    No            Yes   No            Yes    No
Sunny       2     3     Hot     3     2    High      3     4
Overcast    4     0     Mild    4     2    Normal    7     1
Rainy       3     2     Cool    3     1
          -----------         ---------            ----------
Sunny     2/9   3/5     Hot   3/10   2/5    High    3/10   4/5
Overcast  4/9   0/5     Mild  4/10   2/5    Normal  7/10   1/5
Rainy     3/9   2/5     Cool  3/10   1/5

            Windy        Play
=================    ========
      Yes     No     Yes   No
False 7      2       10     5
True  3      3
      ----------   ----------
False  7/10    2/5   10/15  5/15
True   3/10    3/5

Notice there are 15 observations for each attribute except Outlook, which has only 14. This is since that value was unavailable for the last record. All further development would continue as discussed in the linked article.

For example in the R package e1071 naiveBayes implementation has the option na.action which can be set to na.omit or na.pass.


Flair Embeddings - Significance of Backwards vs Forwards? I'm working on a project that makes use of Flair for stacked embeddings. I'm looking at the built in embeddings on this page. I noticed that the table shows news-X as being "Trained with 1 billion word corpus". However when actually making use of the embeddings it seems you either use news-forward or news-backward.

I'm assuming this means both of these embeddings are trained on the same dataset and related. However one of the projects I'm looking at actually stacks both of these embeddings:

    embedding_types: List[TokenEmbeddings] = [
        FlairEmbeddings('news-forward'),
        FlairEmbeddings('news-backward'),
        # ... additional embeddings
    ]

I'm having trouble understanding what the forward and backwards means in this situation. Furthermore if they're both based on the same data what would be the benefit of stacking them?

The page linked above also says: "We recommend combining both forward and backward Flair embeddings" but doesn't explain exactly how the two different embeddings are generated from the training data.

Upon further research I found that during training forward language models try to predict the next word in a sequence. Backwards language models on the other hand start at the end of a sequence and attempt to predict the proceeding word. It seems that by stacking both forward and backwards models produced from the same data-set you get better results than using a forward or backwards model alone.


fastai - using 'untar_data' function in kaggle kernel I have recently started with fastai lesson 1 and I am using kaggle to run the course notebooks. While going through the ‘lesson1-pets’ notebook we use untar_data(URLs.PETS) to get the data.

What I want to understand is where does this data get downloaded to?
As I can observe, after running the untar_data(URLs.PETS) function, it says downloading… ,the data gets downloaded, but nothing gets added to the data section of kaggle kernel.

Note: I am able to run the whole notebook in kaggle and just want to understand the working of untar_data function.

It goes to PosixPath('/root/.fastai/data/oxford-iiit-pet') so you won't see it on the data pane.


What is the difference between majority vote, and greedy action in ensembling? I read some stuff about majority vote and greedy action in ensembling, however, they kind of sound similar, but also different. What is the real difference between those two?

Thanks for your help!

Ensemble learning basically means combining several base models to produce one optimal solution.

And from my understanding,
usually in Majority vote, we train the models with all the data and wait for all the models to result in a prediction and at the end we choose the major result (result that appeared more than any other) as the final output.

But in Greedy way, we usually iterate on the data, run prediction on a subset of the data, pick the best result of this subset and move to the next, and at the end we only consider the best results of each iteration. So we don't look at the big picture here but only a collection of subsets. Not the Global best result but Local best results.

Hope this helps.


What does pandas describe() percentiles values tell about our data? Let say this is my dataframe

x=[0.09, 0.95, 0.93, 0.93, 0.34, 0.29, 0.14, 0.23, 0.91, 0.31, 0.62,
       0.29, 0.71, 0.26, 0.79, 0.3 , 0.1 , 0.73, 0.63, 0.61]

x=pd.DataFrame(x)

When we x.describe() this dataframe we get result as this

>>> x.describe()
               0
count  20.000000
mean    0.50800
std     0.30277
min     0.09000
25%     0.28250
50%     0.47500
75%     0.74500
max     0.95000

What is meant by 25,50, and 75 percentile values? Is it saying 25% of values in x is less than 0.28250?

It describes the distribution of your data: 50 should be a value that describes „the middle“ of the data, also known as median. 25, 75 is the border of the upper/lower quarter of the data. You can get an idea of how skew your data is. Note that the mean is higher than the median, which means your data is right skewed.
Try:
import pandas as pd
x=[1,2,3,4,5]
x=pd.DataFrame(x)
x.describe()


Find inputs that give highest variance in output space I have a relatively simple problem that I think should have a known solution, but that I can't figure out. Any help would be greatly appreciated.
Basically, I have a function $f : \mathbb{R}^d \rightarrow \mathbb{R}^p$ which has vector input of features and vector output of images. Given a training set of combinations of the input variables and their corresponding outputs $\{x_i, y_i\}_{i=1}^N$, I would like to find the input feature (i.e. element of the inputs) that leads to the highest variability in the outputs, i.e. highest pixel-wise variance. Broadly speaking, I'm hoping to quantify the relationship between each feature and its effect on the variance of the output, to get an idea of how "sensitive" that feature is.
Methods like PCA wouldn't work because I am working in a supervised setting. Whereas PCA tries to find the directions of maximum variance of an unsupervised dataset, I am trying, in a sense, to find the directions of maximum variance in the label (output) space as it relates to the input space. Does an answer to this exist? Thanks in advance.

Very interesting question! I didn't get if the out put of one input vector is just one image or a set of images but this does not change the solution much.
I propose you directly learn the relation between input and the variance of output images. For simplicity, I assume you target the pixle-wise variance within an output image. Then your training data would become $\{x_i, Var(y_i)\}_{i=1}^N$ instead of $\{x_i, y_i\}_{i=1}^N$!
Then your problem can be solved as:

Classification: You threshold the "high variance" and discritize your output into high variance and low variance classes
Regression: Even better! you directly learn the value of variance and have more flexibility on interpreting the results at the end (e.g. the thresholding mentioned above can be applied afterwards).

I hope I understood your question right, I was clear and it could help!
Good Luck!


assignment of observations to known size classes Assume I have some observations and I want to predict their classes. For each observation my classification algorithm spits out probabilites for each class. Now assume I know exactly what distribution my prediction has to have. For example I know that 20 of my observations have to be in class 1, 15 in class 2 and so on. What is the best way to assign my observation based on their probabilities of belonging to a class. Does this problem have a name?

You can potentially model this Information in form of a Prior, depending how you proceede with modelling. Check out bayesian statistics


Isolation forest: how to deal with identical values? I am trying to develop my own implementation of isolation forest algorithm. However I don't know how to deal with points that have the same value for a given feature. To better understand the problem, consider this example:
in my dataset I have the following data: (1, 2), (3,5), (3,4)

We may suppose that after one iteration, we get the following split:

      root
     /     \
    /       \
   /         \
(1,2)    (3,4), (3,5)

Now, how am I supposed to deal with the right branch?
If (due to randomness) we decide to split according to the second feature, we should not have any problem because they are different (4 and 5). However what about if we decide to split according to the first feature (which is 3 in both cases)? Should I repeat the random feature selection until I can split the remaining data?

No need to use all the features. That is the cool part of having an ensemble. If one tree who is using a small set of data with some set of features is unable to use the feature, some other tree will use it. If the point is a true outlier, the score that is calculated later will offer the due reflection.


Is the R language suitable for Big Data R has many libraries which are aimed at Data Analysis (e.g. JAGS, BUGS, ARULES etc..), and is mentioned in popular textbooks such as: J.Krusche, Doing Bayesian Data Analysis; B.Lantz, "Machine Learning with R".

I've seen a guideline of 5TB for a dataset to be considered as Big Data.

My question is: Is R suitable for the amount of Data typically seen in Big Data problems?
Are there strategies to be employed when using R with this size of dataset?

Actually this is coming around. In the book R in a Nutshell there is even a section on using R with Hadoop for big data processing. There are some work arounds that need to be done because R does all it's work in memory, so you are basically limited to the amount of RAM you have available to you.

A mature project for R and Hadoop is RHadoop

RHadoop has been divided into several sub-projects, rhdfs, rhbase, rmr2, plyrmr, and quickcheck (wiki).


Similar output from random forest and neural network I used a training data set to train both a random forest and a neural network (one hidden layer). Then I compared how both systems perform on a test data set. Interestingly, both turned out to have about the same prediction probabilities. The forest classified 86% of the data correctly into category 0 or 1, the network achieved 85%. As there is 75% of the data in category 0, I am not too happy with the result and hoped for something better.

I then analyzed, which data sets were classified incorrectly. It turned out that there is a large overlap: 86% percent of the data, which was classified incorrectly by the forest, was also classified wrongly by the network. I then compared the probabilities which were attributed to those test samples by the forest and the network. Also the probabilities were almost always comparable so that when the forest was sure that sample A belongs to category 0, then also the network was sure that 0 is the right category. Also, if the probabilities of the forest were around 50% for both categories, also the network was "unsure" which category was correct.

Is such an outcome somehow obvious? Is there an explanation?

I think that your neural network is probably learning the same features as your Random Forest. I do not think it is obvious because it is hard to say what the neural network is learning, but in this case based on the your model it might be working similar.


What is an 'old name' of data scientist? Terms like 'data science' and 'data scientist' are increasingly used these days.
Many companies are hiring 'data scientist'. But I don't think it's a completely new job.
Data have existed from the past and someone had to deal with data.
I guess the term 'data scientist' becomes more popular because it sounds more fancy and 'sexy'
How were data scientists called in the past?

In reverse chronological order: data miner, statistician, (applied) mathematician.


How to approach semi-supervised binary classification problem with few labels only from one class? I confront with a binary classification problem where I do have a few instances with labels (so far this is "semi-supervised" learning as far as I know), but only from the positive class. So I cannot take any negative examples as basis for learning the other class. What is the best practice here? I assume that I should find some examples farthest from the explicit positives and treat these like negatives; but if so, what is a handy way for this in Python (preferably in sklearn)?
Furthermore, following the approach above, I'm a bit confused when to switch to supervised mode (if any time at all) if instances could be separated only with clustering?

I see two approaches:

either you do all your process using only the positive class, so based on one class classification approaches such as isolation forest, one class SVM, recontruction error of autoencoder (trained only by your positive class), and so on... All those classifiers learn from one class.

or you can do it in 2 steps. First try unsupervised approaches (such as clustering) to extract negative (and other positive) class samples. Then use a binary classifier on positive vs negative class. In this approach, you want to be careful with the unsupervised split that give you the input dataset of your second step. You want to avoid mislabeling.


How to tell my neural network that I care much more about precision than recall? I am training a neural network for a multilabel classification problem, so my last layer consist of n_classes sigmoid neurons.

Now, I know that it is impossible to nail the learning task both because of noise and lack of predictable power in my features. However, I am assuming that the predictable power should be enough to make good predictions at least for some data points. Even if these cases are just a few, I would be happy if the neural network gets at least those with certainty (high precision) and misses the grand majority because of their difficulty (low recall).

Of course, I could train the model normally and then set that threshold that matches my goals, but this doesn't convince me. I would like to introduce this goal in the training process. If some points are hard to classify I want my neural network to discover that during training, forget about them and focus on the other more doable cases.

So far my only idea is to make training robust towards outliers via label smoothing.

Any other ideas to influence the model during training like this?

I think you have two options that don't exclude each other:

Check precision/recall metrics constantly during training, on Validation data. In this way, during experimentation and hyperparameter tuning, you will choose models that are more favourable for precision rather than recall.
Customize your loss function to correct the model in the right way. The problem is that precision, recall, F1 score, they are all non differentiable functions, i.e. they cannot enter a Loss function as they are. For that reason, some tried to approximate these metrics in a differentiable way. One example is the soft-F1 score, as explained in this article. If you read the function code, you will find differentiable approximations of TP, TN, FP, FN's. You can copy those lines to customize your personalized pseudo-precision Loss function.


Standardizing Vegas odds for a randomForest I'm sorry I don't have reproducible code, but I have a pretty specific question that I can't find an answer to.

I'm using randomForest to project NBA statistics. Vegas-odds are incredibly useful because it's provides the wisdom of the crowd. Intuitively I feel like they need to be standardized for analysis, but maybe randomForest is good enough.

The reason why I feel it needs to be standardized is because it's disjoint. If a team has a moneyline of -125, that means that you must pay \$125 to win \$100 (payout of \$225). If a team has a betting line of +110, that means you need to bet \$100 to win \$110 (payout of \$210). Therefore, it's disjoint in that there would never be scores in (-100, 100) since +100 or -100 are both even odds.

With that said, would you recommend reshaping the vector in some way so that the random forest can learn "better"? E.g. -125 is a (125/(100 + 125) 55.6% chance of winning and a +110 is a (100/110+100) 47.6% chance of winning. Would changing the moneylines to percentages help performance? I know the only surefire way to check would be to run models, but I really don't have time for it at the moment, and this question will help me to determine in general if/when standardizing is necessary.

This is directly related to the idea of calibrating probability values produced by a random forest. Aside from the link, there is substantial literature on how to do this. The simplest approach basically amounts to fitting a logistic regression on the outputs of the random forest to change the response surface into a logistic form.

Once the predictions have this form, you've demonstrated that you have the knowledge to turn the probability estimate into Vegas odds, so that should be a straightforward process for you.


What is dimensionality reduction? What is the difference between feature selection and extraction? From wikipedia:

dimensionality reduction or dimension reduction is the process of
reducing the number of random variables under consideration, and
can be divided into feature selection and feature extraction.

What is the difference between feature selection and feature extraction?
What is an example of dimensionality reduction in a Natural Language Processing task?

Simply put:

feature selection: you select a subset of the original feature set; while
feature extraction: you build a new set of features from the original feature set.

Examples of feature extraction: extraction of contours in images, extraction of digrams from a text, extraction of phonemes from recording of spoken text, etc.

Feature extraction involves a transformation of the features, which often is not reversible because some information is lost in the process of dimensionality reduction.


Can a novelty detection model overfit? Can a novelty detection model overfit? In novelty detection, the model is trained on normal data instances (not polluted by outliers) where no labels are used in the training process, while validated and tested on a data instances that contain outliers in them. An example of algorithms that can be used for novelty detection are one-class SVM (OCSVM) and Local Outlier Factor (LOF).

Answering your question: yes, depending on the hyperparameters you choose, you could overfit the considered normal data, if you fit your separating hyperplane between normal and novel points being too much "shaped" on your input data.
There are, for instance in case of one-class support vector machines, some important hyperparams like nu or gamma:

nu: with this one, you tell the oc-SVM the fraction of novel points (i.e. anomalies) you want to consider in your input data; this way, you don't overfit your model by not considering normal all the input data (it also depends on your use case, how sure you want to be about normality of the input data points...)
You can test it with the scikit-learn package this way:

with nu=0.01

VS with nu=0.1, where you tell the model to consider a higher fraction of points as being abnormal:

so, the lower nu value is, the more you are "overfitting" your novelties detector (which could be better or worse depending on how well you know your input data).

gamma: now, look at the effect of the value of gamma, which is crucial for overfitting your model:

with gamma=0.1 (and rbf kernel), you have the following decision surface:

VS with gamma = 10

where, with this last option, your are overfitting so much.


Training Error decreasing with each epoch I am trying to train a VGG-19 neural network on STL-10 dataset containing 5000 images(500 for each class). And the number of output classes is 10.

I have made no changes to the architecture except that I reduced the size of fully-connected layer from 4096 to 2048, keeping the dropout(0.5) same

[The reason behind doing this is that, since the number of training images is less, so to avoid over-fitting, I decreased the size of fully-connected layer, but I don't know if this is the right thing to do].

I have also used Adam optimiser(learning rate = 0.001) instead of SGD as mentioned in the paper.

I have only run the code for only 4 epochs. And I have observed that although the cost is decreasing by very small amount, but the training accuracy is decreasing.

After the 1st epoch, cost:: 2.304091 and training accuracy:: 11.99%

After the 2nd epoch, cost:: 2.303365 and training accuracy:: 11.249%

After the 3rd epoch, cost:: 2.301936 and training accuracy:: 10.5625%

After the 4th epoch, cost:: 2.30415 and training accuracy:: 8.1249%

I want to know, that is this behavior natural or is it due to some fault in the changes I made in the architecture?
Or using less number of layers would have been better? (For example VGG-16)

(This is my first hands-on neural network experience)

I am not very sure about your case, usually I have seen the same problem but in the other direction, you can take a look here:

Quora loss drecrease, accuaracy constant
Cross validated loss drecrease, accuaracy constant

I would guess that your DNN is getting more answers right but without impoving its confidence  a lot.
Regarding your other questions, I would advise you to start with simplest possible model, for example, in this case it would be using the vgg16 (which is widely more used than the vgg19) and the default parameters ( number of neurons, optimization function...) Once you have something working you can start tuning and modifying things.


difference between XGBRegressor and XGBClassifier I'm trying to understand the difference between xgboost.XGBRegressor and  xgboost.sklearn.XGBClassifier.

Can someone explain the difference in a concise manner?

Because when I fit both classifiers with the exact same data, I get pretty different performance.

This is how I fit the data.

clf = xgboost.XGBRegressor(alpha=c)
#clf = xgboost.sklearn.XGBClassifier(alpha=c)

clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
print('Model roc auc score: %0.3f' % roc_auc_score(y_test, y_pred, average='macro', sample_weight=None))
print('Model pr auc score: %0.3f' % average_precision_score(y_test, y_pred))

when  clf = xgboost.sklearn.XGBClassifier(alpha=c)

Model roc auc score: 0.544

Model pr auc score: 0.303

when clf = xgboost.XGBRegressor(alpha=c)

Model roc auc score: 0.703

Model pr auc score: 0.453

What would cause this performance difference?

XGBRegressor is for continuous target/outcome variables. These are often called "regression problems."

XGBClassifier is for categorical target/outcome variables. These are often called "classification problems."


Data preprocessing, relative scale problems in features of same type I am using Keras NN with theanos backend in Python. In my data i have multiple features of the same type but in different columns (on purpose). Here is an example.

   F1   F2
0. 8673 7490
1. 5602 5602
2. 4352 2365

When i go to process this, say MinMaxScaler 0-1, even though '1.' in both features should be the same, because both columns have different min-max values won't the normalized value be different and therefor cause problems in pattern finding? If so, what method can i use to make sure that they are of the same relative scale? Thanks for any help.

If want you want is to scale both columns using the maximum of both and the minimum of both. You can do it like this:

df = pd.DataFrame(data=data, columns=['F1', 'F2'])
columns = ['F1', 'F2']

ma = float(np.amax(df[columns].values))
mi = float(np.amin(df[columns].values))

for column in columns:
    df[column] = df[column].apply(lambda e: (e - mi) / (ma - mi))

According to the documentation of sklearn that is what MaxMinScaler does, if you want to use predefined values for max and min, you can just change ma and mi. If you have more columns you just have to change the list of columns. For rescaling to an arbitrary range (max, min) read the sklearn documentation on MaxMinScaler.


Is SuperLearning actually different to stacking,or are they essentially the same thing? Articles which use the terms 'stacking' and 'Super Learner' often seem to use the terms interchangeably. Is the Super Learner algorithm a specific form of the more generic stacking concept, or is Super Learner essentially the same thing, and at some stage one of these terms is likely to become redundant?

So Ensemble Learning is essentially using multiple learning algorithms and providing the best predictive performance considering all of them. This gives a better detailed description. Now, Ensemble Learning can be broadly divided into multiple types like

Boosting
Bagging
Stacking / Super Learning

Stacking contains a bunch of algorithms along with a learner to ensemble a group of base learners. Going deeper, the term Stacking was used way before Super Learning. Later, when the algorithm was actually developed theoretically and made popular in 2007, it was given the name 'Super Learner'.


How to rank a feature importance? If I trained a network using Neural Network classifier, how can I know which feature was most important for predicting the target variable? I mean how to create a "feature ranking" among the features (from high importance value to low). I have seen some literature about decision trees/AdaBoost but I am typically interested in Neural networks especially for classification purpose. To make it more clear, an example is shown in the figure.

There are multiple possibilities.

The obvious one is to sum weights of all connections from input layer to the first hidden layer per input neuron. The neuron with highest sum of connection weights should be the most important. This has several problems.

Weights could be negative so you should use absolute value of them.
The weight depend on the scale of the input variable. You should normalize the input for example by subtracting mean and dividing by standard deviation of the input (z-normalization).

Another possibility is to selectively remove features, retrain the network and observe removal of which features caused the biggest decrease in model accuracy. These are potentially the most important features.

The problem of this approach is, that it changes the architecture of the network, retrains it and does not report the relevance of features entering the original network.

You can find an elaborate discussion on the problem here ftp://ftp.sas.com/pub/neural/importance.html

There are multiple other approaches, such as Garson's, using complex functions to correct for some problems. You can find several of them implemented in NeuralNetTools package in R.


Asynchronous Hyperparameter Optimization - Dependency between iterations When using Asynchronous Hyperparameter Optimization packages such as scikit optimize or hyperopt with cross validation (e.g., cv = 2 or 4) and setting the number of iteration to N (e.g., N=100), should I expect:

Dependency between sequential iterations where the loss value improves sequentially (e.g., the optimized hyperparameters in iteration number 10 are better than the optimized hyperparameters generated in iteration number 9, etc.). In this case I should always select with the hyperparameters generated in the last iteration.

or

Expect independency between iterations where after all 100 iterations are completed I should select the iteration with the smallest loss value.

If option a) is the right answer, that what does it mean if the best Hyperparameter are associated with iteration 50, does it mean that the data is not stable, or the loss function is ill-specified, and therefore, the hyperparameter optimization process outcome should not be trusted?

hyperopt proceeds sequentially, unless you let it use a parallel backend. Is N the max_evals parameter? Yes, you always want to select the hyperparams with the best validation loss. That's what it returns in the end. It may be that it found the best hyperparameters well before the final trial. It does not mean anything is wrong. It learns as it goes the distribution of the loss conditional on hyperparameters, and on purpose explores less-certain parts of the space that are most likely to yield improvement, but may not, especially towards the end of the search. This happens with entirely well-defined loss functions.


Is autocorrelation the same as multicollinearity? I have a time series dataset that has 63 features with traffic_volume as the target.
print(main_data.columns)
Index(['air_pollution_index', 'clouds_all', 'humidity', 'temperature',
       'wind_direction', 'wind_speed', 'is_holiday_0', 'is_holiday_1',
       'is_holiday_2', 'is_holiday_3', 'is_holiday_4', 'is_holiday_5',
       'is_holiday_6', 'is_holiday_7', 'is_holiday_8', 'is_holiday_9',
       'is_holiday_10', 'is_holiday_11', 'weather_descr_0', 'weather_descr_1',
       'weather_descr_2', 'weather_descr_3', 'weather_descr_4',
       'weather_descr_5', 'weather_descr_6', 'weather_descr_7',
       'weather_descr_8', 'weather_descr_9', 'weather_descr_10',
       'weather_descr_11', 'weather_descr_12', 'weather_descr_13',
       'weather_descr_14', 'weather_descr_15', 'weather_descr_16',
       'weather_descr_17', 'weather_descr_18', 'weather_descr_19',
       'weather_descr_20', 'weather_descr_21', 'weather_descr_22',
       'weather_descr_23', 'weather_type_0', 'weather_type_1',
       'weather_type_2', 'weather_type_3', 'weather_type_4', 'weather_type_5',
       'weather_type_6', 'weather_type_7', 'weather_type_8', 'weather_type_9',
       'weather_type_10', 'dew_point_1', 'dew_point_2', 'dew_point_3',
       'dew_point_4', 'dew_point_5', 'dew_point_6', 'dew_point_7',
       'dew_point_8', 'dew_point_9', 'is_weekend', 'traffic_volume'],
      dtype='object')

All variables are univariate including the target. However, I would like to have differences between autocorrelation and collinearity to be clarified since this has been my first major time series project and some terms are certainly different than statistics. From what I've analysed, there are collinear variables present and they seem to be hindering the accuracy of predictions but visualising the auto and partial correlation doesn't seem to be suggest that. Can someone help me out on pointing out anything wrong with this data and does it need some removal of variables to make it more robust? I've ued XGBoost algorithms with a MAE of 1635 which is not a good score. The first plot is the correlation matrix while the rest are the auto and partial correlation plots.

Please note the partial and auto correlation plots relate to response variable only.

Autocorrelation is a measure of a correlation of a signal with itself, as a function of delay.

Correlation is usually between two different variables (without respect to time). You can take it with the same variable, but that will of course equal 1.

Either way, XGB is robust to multi-colinearity (correlations between features, or explanatory variables as you'd call them in statistics), please have a look at this post for more information.

Lots of things could be wrong with the model, I would start with really understanding the data and making sure there's nothing wrong with it(missing values, unexpected distributions etc...).


Need help on Deep learning model on sentiment analysis Before I present my problem, please note that I am a newbie in deep learning and I am trying things for the first time. Most of my code/logic were adopted from various references in the internet.
Goal : Build a LSTM/CNN model to classify the IMDB reviews available in tensorflow datasets
Approach 1 : 1) LSTM based - train data - 45000 (10% validation split),test data - 5000 , accuracy > 95% , validation_accuracy > 85% , glove embeddings of size 100 was used
Approach 2 : 1) CNN model - a) train data - 45000 , test_data - 5000
b) train data - 50% , test_data - 50%
accuracy > 95% , validation_accuracy > 85%
Code :  https://github.com/shankartmv/Deep-Learning-Work/blob/main/IMDB_Sentiment_reviews_using_tensorflow_dataset.ipynb
Problem : Test_data accuracy doesn't go beyond 52% with both the approaches.Most of the code/references available out there use test_data during training.  test_data wasn't part of my training.
Methodologies tried to increase test accuracy :

Movie reviews length (padded) and max number of words in the vocabulary
dropout
epochs
train test split ratio
embeddings trainable=true/false
with and without GLove word embeddings

My guess is there isn't enough training data. I need help on how to increase the test data accuracy.

Most of the code/references available out there use test_data during training. test_data wasn't part of my training.

While this is the way we should do it but stuff like Encoding must be done holistically.
 In your case, you have called the pre_process separately for Test and Train.
 So, the words are converted to Numbers independently. This should not happen.
tokenizer.texts_to_sequences(test)
Above Tokenizer should be the one that was fit on train data.
If I randomly print token with key 101 for train, test. This is the result

print(train_tokn.index_word[101])
print(test_tokn.index_word[101])

think
characters

I think you should use the train_tokn for the test data and it should improve. I believe a very simple LSTM can achieve 85% on this dataset
Or, manually embed both Train, Test using the GloVe embedding.
 A simple example for the issue
from keras.preprocessing.text import Tokenizer

train = ['I am sorry']
test = ['I am very sorry']
max_words = 10

# Train
tokenizer = Tokenizer(num_words=max_words)
tokenizer.fit_on_texts(train)
tokenizer.index_word # {1: 'i', 2: 'am', 3: 'sorry'}
# Test
tokenizer = Tokenizer(num_words=max_words)
tokenizer.fit_on_texts(test)
tokenizer.index_word  # {1: 'i', 2: 'am', 3: 'very', 4: 'sorry'}


what could this mean if your "elbow curve" looks like this? This is from running kmeans clustering with k on the x-axis (ranging from 2 to 10) and the silhouette distance on the y-axis.

Clearly there's peaks at k=3, k=4 and it seems to decline from there. It doesn't resemble an elbow and thought it should rise as k gets larger (due to over fitting on he training set). Do I just lack data?

I'm computing the silhouette distance using a 80-20 train test split.

First of all, you do have two elbows: one at $k=4$ and a large one at $k=8$. The second isn't very apparent because you haven't drawn out the plot for larger values of $k$. If you do you might get a figure like this:

Secondly, you aren't meant to look for an elbow when computing the silhouette score! The silhouette score accounts for both inter- and intra-cluster distance, as such it can be used for selecting $k$ on its own (i.e. select the $k$ that produces the best silhouette score).

Note: I'm not familiar with the "silhouette distance", I assume it is somewhat related to the silhouette score (maybe its inverse).

The "elbow" criterion should be used when dealing with metrics that tend to improve as $k$ increases (e.g. inertia).


How to do Feature Scaling for these ranges [0,1] and [-1,1]? I want to rescale the features of my data to be between [0,1] and [-1,1]? Is their a clear cut way that works every time for these ranges? I think the below equation works for [0,1] but when it is describe people say generally it works, so I am not certain it works every single time.

What you said is right, the above equation is for normalizing the data with-in the range of [0,1]

Now, we can generalize using the below equation

To normalize in $[-1,1]$ you can use:

$$
x'' = 2\frac{x - \min{x}}{\max{x} - \min{x}} - 1
$$

In general, you can always get a new variable $x'''$ in $[a,b]$:

$$
x''' = (b-a)\frac{x - \min{x}}{\max{x} - \min{x}} + a
$$


Can we remove features that have zero-correlation with the target/label? So I draw a pairplot/heatmap from the feature correlations of a dataset and see a set of features that bears Zero-correlations both with:

every other feature and
also with the target/label

.Reference code snippet in python is below:

corr = df.corr()
sns.heatmap(corr) # Visually see how each feature  is correlate with other (incl. the target)

Can I drop these features to improve the accuracy of my classification problem?
Can I drop these features to improve the accuracy of my classification problem, if it is explicitly given that these features are derived features?

Can I drop these features to improve the accuracy of my classification problem?

If you are using a simple linear classifier, such as logistic regression then yes. That is because your plots are giving you a direct visualisation of how the model could make use of the data.

As soon as you start to use a non-linear classifier, that can combine features inside the learning model, then it is not so straightforward. Your plots cannot exclude a complex relationship that such a model might be able to exploit. Generally the only way to proceed is to train and test the model (using some form of cross-validation) with and without the feature.

A plot might visually show a strong non-linear relationship with zero linear correlation - e.g. a complete bell curve of feature versus target would have close to zero linear correlation, but suggest that something interesting is going on that would be useful in a predictive model. If you see plots like this, you can either try to turn them into linear relationships with some feature engineering, or you can treat it as evidence that you should use a non-linear model.

In general, this advice applies whether or not the features are derived features. For a linear model, a derived feature which is completely uncorrelated with the target is still not useful. A derived feature may or may not be easier for a non-linear model to learn from, you cannot easily tell from a plot designed to help you find linear relationships.


How can we create an label, value detector? I am trying to implement an text detector using MaskRCNN such that the model detects the label and value as shown in the image below.

Detecting the same is easier for fields like page date and order no. but fails for invoice number and salesman shown in image below. Is there any pretrained models for the same? or some other technique that might help.

Here is a paper doing exactly what you want with detailed explanation of architecture.

There is not much sense of me doing a TL;DR here because they have some special labeling of data etc... but the gist of it is LSTM if you do some pre-processing steps and get your data in right format.


Method for solving problem with variable number of predictors I've been toying with this idea for a while. I think there is probably some method in the text mining literature, but I haven't come across anything just right...

What is/are some methods for tackling a problem where the number of variables it its self a variable.  This is not a missing data problem, but one where the nature of the problem fundamentally changes. Consider the following example:

Suppose I want to predict who will win a race, a simple multinomial classification problem. I have lots of past data on races, plenty to train on.  Lets further suppose I have observed each contestant run multiple races. The problem however is that the number or racers is variable. Sometimes there are only 2 racers, sometimes there are as many as 100 racers.

One solution might be to train a separate model for each number or racers, resulting in 99 models in this case, using any method I choose.  E.g. I could have 100 random forests.

Another solution might be to include an additional variable called 'number_of_contestants' and have input field for 100 racers and simply leave them blank when no racer is present.  Intuitively, it seems that this method would have difficulties predicting the outcome of a 100 contestant race if the number of racers follows a Poisson distribution (which I didn't originally specify in the problem, but I am saying it here).

Thoughts?

I don't see the problem. All you need is a learner to map a bit string as long as the total number of contestants, representing the subset who are taking part, to another bit string (with only one bit set) representing the winner, or a ranked list, if you want them all (assuming you have the whole list in your training data). In the latter case you would have a learning-to-rank problem.

If the contestant landscape can change it would help to find a vector space embedding for them so you can use the previous embeddings as an initial guess and rank anyone, even hypothetical, given their vector representation. As the number of users increases the embedding should stabilize and retraining should become less costly. The question is how to find the embedding, of course. If you have a lot of training data, you could probably find a randomized one along with the ranking function. If you don't, you would have to generate the embedding by some algorithm and estimate only the ranking function. I have not faced your problem before so I can't direct you to a particular paper, but the recent NLP literature should give you some inspiration, e.g. this. I still think it is feasible.


What do Python's pandas/matplotlib/seaborn bring to the table that Tableau does not? I spent the past year learning Python. As a person who thought coding was impossible to learn for those outside of the CS/IT sphere, I was obviously gobsmacked by the power of a few lines of Python code!

Having arrived at an intermediate level overall, I was pretty proud of myself as it greatly expands my possibilities in data analysis and visualization compared to Excel (aside from the millions of other uses there are for Python).

Purely in terms of data analysis and visualization:

what does approaching the same data set with pandas/matplotlib/seaborn/numpy bring to the table as opposed to using Tableau?

(sidenote: I was greatly disappointed to see all my hard-earned Python data wrangling skills were available in such a user-friendly GUI... :'( )

Don't worry - your hard-earned Python skills are still important ;)

Tableau is not a replacement - it is essentially a means of sharing your insights/findings. It is a wrapper around your normal toolkit (Pandas, Scikit-Learn, Keras, etc.). It can do some basic analysis (just using basic models from sklearn), but the powerful thing is it can deploy your models to allow people to run inference on stored data/new data, and then play around with it in an interactive dashboard.

Watch this video for a good overview of everything it can do, and how it connects to Python (and R/MatLab). There is just a bit of boiler plate code around your normal Python code.

Tableau also offer TabPy to set up a server, allowing nice deployments of your work, but in the end you need their desktop application to view the results (i.e. your customers need it to look at the results). This is not free: https://www.tableau.com/pricing/individual

In summary, I'd say Tableau is more of a business intelligence tool, allowing e.g. your non-data-scientist boss or other stakeholders to interactively explore the data and the results of your modelling. Similar to Microsoft's PowerBI.


Retrieving column names in R I am trying to retrieve the column names of the data set model$data using the following formula:

sample(colnames(model$data),1)

When I run it I receive the following error message:

Error in sample.int(length(x), size, replace, prob) :
  invalid first argument

Appreciate any help!

str(model) looks like this:

> str(model)
    List of 13
     $ data            :List of 1
      ..$ : num [1:1000, 1:56] 1 1 1 1 0 1 1 0 1 1 ...
      .. ..- attr(*, "dimnames")=List of 2
      .. .. ..$ : chr [1:1000] "7530" "5975" "552" "815" ...
      .. .. ..$ : chr [1:56] "Agriculture_and_Hunting" "Baking" "Biochemistry" "Braiding" ...
     $ unit.classif    : num [1:1000] 3 5 5 5 16 3 5 1 3 3 ...
     $ distances       : num [1:1000] 0.000806 0.000239 0.000239 0.000239 0.001953 ...
     $ grid            :List of 6
      ..$ pts              : num [1:25, 1:2] 1.5 2.5 3.5 4.5 5.5 1 2 3 4 5 ...
      .. ..- attr(*, "dimnames")=List of 2
      .. .. ..$ : NULL
      .. .. ..$ : chr [1:2] "x" "y"
      ..$ xdim             : num 5
      ..$ ydim             : num 5
      ..$ topo             : chr "hexagonal"
      ..$ neighbourhood.fct: Factor w/ 2 levels "bubble","gaussian": 1
      ..$ toroidal         : logi FALSE
      ..- attr(*, "class")= chr "somgrid"
     $ codes           :List of 1
      ..$ : num [1:25, 1:56] 0.000388 0.99996 1 1 1 ...
      .. ..- attr(*, "dimnames")=List of 2
      .. .. ..$ : chr [1:25] "V1" "V2" "V3" "V4" ...
      .. .. ..$ : chr [1:56] "Agriculture_and_Hunting" "Baking" "Biochemistry" "Braiding" ...
     $ changes         : num [1:100, 1] 0.00261 0.00263 0.00262 0.00254 0.00254 ...
     $ alpha           : num [1:2] 0.05 0.01
     $ radius          : Named num [1:2] 3 0
      ..- attr(*, "names")= chr [1:2] "67%" ""
     $ user.weights    : num 1
     $ distance.weights: num 1
     $ whatmap         : int 1
     $ maxNA.fraction  : int 0
     $ dist.fcts       : chr "sumofsquares"
     - attr(*, "class")= chr "kohonen"

Your data boils down to something like this structure:

> str(model)
List of 2
 $ data:List of 1
  ..$ : int [1:3, 1:4] 1 2 3 4 5 6 7 8 9 10 ...
  .. ..- attr(*, "dimnames")=List of 2
  .. .. ..$ : NULL
  .. .. ..$ : chr [1:4] "a" "b" "c" "d"
 $ foo : num 1

but you have some other components that we don't need to bother with, and my data doesn't have row names and is a lot smaller.

model is a list of 2 (for me) and 13 (for you) parts.

The $data component is also a "List of 1" component.

So colnames(model$data) is trying to get the colnames of a list, and failing:

> colnames(model$data)
NULL

Which you would have spotted if you'd tried running colnames(model$data) yourself.

You want the colnames of the first element of the list model$data:

> colnames(model$data[[1]])
[1] "a" "b" "c" "d"

and hence:

> sample(colnames(model$data[[1]]),1)
[1] "b"

Its possible that because this is a "kohonen" class object that there are functions that get these data matrices for you. You'll need to read the documentation to figure this out. What I've shown above is digging in the structure to find the data you want.


Exponential runtime on distance matrix I have an exploratory script running a Databricks notebook that performs a simple arithmetic function (Pythagorean theorem) on all possible pairwise combinations of a list of pairs of floats (akin to coordinates).

The values are generated randomly, like so:

vals = np.random.rand(num_samples, 2)

The list is then converted to 2 RDDs of Rows, like so:

rdd = sc.parallelize(vals)
rows_1 = rdd.map(lambda v: Row(x=float(v[0]), y=float(v[1]), join_val=1))
rows_2 = rdd.map(lambda v: Row(x_r=float(v[0]), y_r=float(v[1]), join_val=1))

Which are then registered as tables:

sqlContext.createDataFrame(rows_1).registerTempTable('sdf_1')
sqlContext.createDataFrame(rows_2).registerTempTable('sdf_2')

sdf_1 = sqlContext.table('sdf_1')
sdf_2 = sqlContext.table('sdf_2')

Each table contains the same content, just different columns names. The two are then joined:

sdf_1.join(sdf_2, sdf_1.join_val==sdf_2.join_val).registerTempTable('sdf_join')
sdf_join = sqlContext.table('sdf_join')

With the tables joined, the following UDF is defined:

def calc_dist(x1, y1, x2, y2):
  return math.sqrt((x1-x2)**2 + (y1-y2)**2)

calc_dist_udf = udf(calc_dist, FloatType())

Finally, the operation is performed on all rows:

sdf_join\
  .select(calc_dist_udf('x', 'y', 'x_r', 'y_r').alias('dist'))\
  .filter('dist<0.05')\
  .count()

This operation completes successfully, but I have noticed that, as num_samples increases, the execution time increases exponentially. I believe I am failing to correctly parallelize the row-wise operation.

Is this assumption correct?
How can I achieve parallelization on such an operation?

For 2 tables of size $n$ and $m$, join operation complexity is in $O(n.m)$ in worst case. If $n=m$ we are facing a quadratic operation which is why you observe slowness with the increase of tables size. Parallelize a quadratic problem requier at least to increase quadratically the number of slaves, without taking account shuffling, to keep constant execution time with increase of problem size which is not really a feasable option...


use ANN to predict several properties Let’s say I have tens of thousands of datasets (rows) consisting each of 300 columns of integer, double and strings (no image, no audio). Five of these target columns represent interesting resulting properties (integer) of the product that is represented by of each dataset (row) that I want to predict.

My current idea is to create classes for each target column and combine these classes: column1 has classes 1-5, column2 has also class 1-5… so I get combined target classes in the form “11111”, “15312”, “43151”, “55555”. Now I have only one target-“label” for each dataset (row) that could be the output of an ANN.
This approach has at least two disadvantages:

some combined classes have only a small number of datasets and
the binning of each column in only five classes decreases the precision.

If I increase the number classes of each column, I get even a smaller number of datasets in some combined classes.

So my question is if I should go on with the approach or are there better ideas? It is possible to use ANN to predict several properties (i.e. classes) at the same time?

It's quite a weird problem, at least for me... Following your approach you'll end up with $5^5$ classes, which is an intractable problem with typical Neural Networks. You can visist this paper, which shows a solution with mathematical rigour:

https://arxiv.org/pdf/1806.02507.pdf

I'll also expose three possible naive solutions, awating for people who have faced a similar problem:

Convert your 5 columns into binary columns (3 bits for 5 levels) and treat the problem as a multi-label classification with 15 classes https://en.wikipedia.org/wiki/Multi-label_classification.
You can fit 5 multi-class models for each column (divide and conquer navie approach, it seems to be the state-of-the art phylosohpy)
I've never tried beta-regression, but it would make sense: you can convert your target labels to frequencies of occurrence, think about pruning targets with really low probability. Then, you can perform a preliminary beta regression to predict the occurrence-target. Further on, you may have other classification models depending on the probability given by the beta regressor, let's say 5 models for targets between 0-10%, 10-20% and so on...
Treat your problem as a regression problem, but some questions arise: is '11111' truly close to '11112' in your dataset? i.e. are your labels a ordered set?

  So my question is if I should go on with the approach or are there better ideas? It is possible to use ANN to predict several properties (i.e. classes) at the same time?

Yes, an ANN can output more than one class for a sample (but will not work in such a large scale):

https://www.researchgate.net/publication/3297607_Multilabel_Neural_Networks_with_Applications_to_Functional_Genomics_and_Text_Categorization
https://arxiv.org/pdf/1312.5419.pdf


Gridsearch XGBoost for ensemble. Do I include first-level prediction matrix of base learners in train set? I'm not quite sure how I should go about tuning xgboost before I use it as a meta-learner in ensemble learning.

Should I include the prediction matrix (ie. df containing columns of prediction results from the various base learners) or should I just include the original features?

I have tried both methods with just the 'n_estimators' tuned with F1 score as the metric for cross-validation. (learning rate =0.1)

Method 1: With pred matrix + original features:

n_estimators = 1 (this means only one tree is included in the model, is this abnormal? )
F1 Score (Train): 0.907975 (suggest overfitting)

Method 2: With original features only:

n_estimators = 1
F1 Score (Train): 0.39

I am getting rather different results for both methods, which makes sense as the feature importance plot for Method 1 shows that one of the first-level predictions is the most important.

I think that the first-level predictions by the base-learners should be included in the gridsearch. Any thoughts?

You should tune the meta-estimator using whatever data you want it to eventually predict with.  This should definitely include the base model predictions (else you aren't actually ensembling), and may or may not include (some of) the original features.

One important note though: you should not be training the meta-estimator using "predictions" of the base models on their own training data; those are more accurately called estimations rather than predictions, because the base models already had access to the truth.  A common approach is to train the meta-estimator on out-of-fold predictions from a cross-validation training of the base models.

If the base models are quite good, then it's reasonable that the xgboost model might only use one tree; it just has to tweak the already-good predictions from the base models.  But, consider dropping the learning rate or otherwise increasing regularization, to see if more trees can perform better.


A good reference for the back propagation algorithm? I'm trying to learn more about the fundamentals of neural networks. I feel like I understand the basics of back propagation, but I want to solidify the details in my mind.

I was working through Ian Goodfellow's famous Deep Learning text. However, I found their exposition of back propagation unsatisfactory. They model the computational graph the wrong way, making the weights into the vertices and the operators into the edges. But this makes no sense, as it is the operations (not the variables) which must have an in-degree and out-degree to them.

I was wondering if there is a better reference, either in a textbook, a paper, or a blog that would rigorously outline the details of back propagation in its full generality.

Perhaps we can forget the graph idea to start and just go through the mathematics of it and then we will map our computations to a type of graph just so that it makes it easier to scale.

One neuron network is called the perceptron

The first neural networks only had a single neuron which took in some inputs $x$ and then provide an output. A common function used is the sigmoid function

$\sigma(z) = \frac{1}{1+exp(z)}$

$\sigma(w^Tx) = \frac{1}{1+exp(w^Tx + b)}$

where $w$ is the associated weight for each input $x$ and we have a bias $b$.

How do we train these weights

We will use gradient descent to train the weights based on the output of the sigmoid function and we will use some cost function $C$ and train on batches of data of size $N$.

$C = \frac{1}{2N} \sum_i^N (\hat{y} - y)^2$

$\hat{y}$ is the predicted class obtained from the sigmoid function and $y$ is the ground truth label. We will use gradient descent to minimize the cost function with respect to the weights $w$. To make life easier we will split the derivative as follows

$\frac{\partial C}{\partial w} = \frac{\partial C}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial w}$.

$\frac{\partial C}{\partial \hat{y}} = \hat{y} - y$

and we have that $\hat{y} = \sigma(w^Tx)$ and the derivative of the sigmoid function is $\frac{\partial \sigma(z)}{\partial z} = \sigma(z)(1-\sigma(z))$ thus we have,

$\frac{\partial \hat{y}}{\partial w} = \frac{1}{1+exp(w^Tx + b)} (1 - \frac{1}{1+exp(w^Tx + b)})$.

So we can then update the weights through gradient descent as

$w^{new} = w^{old} - \eta \frac{\partial C}{\partial w}$

where $\eta$ is the learning rate.

What if we have 2 layers?

Take a look at the following answer: Compute backpropagation

Why model neural networks as a graph

In graph theory nodes are connected by vertices. Each neuron takes in some inputs and produces an output. The inputs to this node are not the same as the outputs of the previous layer. They are affected by some weight. Thus we can consider these weights to be the vertices of the graph that link nodes together.


Cross validation schema for imbalanced dataset Based on a previous post, I understand the need to ensure that the validation folds during the CV process have the same imbalanced distribution as the original dataset when training a binary classification model with imbalance dataset. My question is regarding the best training schema.
Let’s assume that I have an imbalanced dataset with 5M samples where 90% are pos class vs 10% neg class, and I am going to use 5-folds CV for model tuning. Also, let’s assume I will hold out a random 100K samples for test (90K samples w/ pos class vs 10K samples w/ neg class). Now I have two options:
Option 1)

Step 1: Pull a randomly selected 200K imbalanced data for training (180K samples pos class vs 20K samples neg class)
Step 2: During each CV iteration:

The training fold will have 160K samples (144K pos vs 16K neg)
and the validation fold will have 40K samples (36K pos vs 4K neg)

Step 3: Apply data balancing for the training fold (e.g., Downsampling, Upsampling, SMOTE, etc.) and fit a model
Step 4: Validate the model on the imbalanced training fold

However, given that I have enough data, I want to avoid using any data balancing algorithm for the training folds.
Option 2)

Step 1: Pull a randomly selected 200K balanced data for training (100K samples pos class vs 100K samples neg class)
Step 2: During each CV iteration:

The training fold will have 160K samples (80K pos vs 80K neg)
and the validation fold will have 40K samples (20K pos vs 20K neg)

Step 3: Fit a model for the already balanced training fold
Step 4: Can I apply down sampling to the balanced validation dataset to restore it to its imbalanced state? If so, how can I do that in sklearn?

I am also clear that I have a 3rd option, which is based on the 1st option above, where the model could be trained on an imbalanced dataset. Therefore, a data balancing algorithm can be avoided.
My questions are:

Is option 2 better than option 1?
How to apply a downsampling to a balanced validation dataset (Option 2-step 4

I'm not sure if there's a question here, but I'll add some comments.
Firstly, if you can get it in the wild, always work with balanced data. However, if you are going to manually create a "balanced" data set yourself, make sure that the selection criteria that you use to create that data is appropriate. As an example, choosing the 100k most recent positive and negative outcomes may not be appropriate because the time frame of the positive outcomes may extend well beyond that of the more common negative outcomes. So in this 200k data set, your negative 100k outcomes may relate to data from the last year while the data relating to your 100k positive outcomes may relate to the last ten years.
Secondly, if you are going to balance your data be aware of how the balancing technique works and try to understand its weaknesses / limitations. Be mindful that rebalancing a data set will result in a new data set, and remember that you will have to check that the new data set is still appropriate to use. As an example, you will need to check that the distribution of input variables is still roughly the same as before. Applying this thinking to your options above, can you be certain that the data in each fold will be roughly similar?
Lastly, if you are going to use a modelling framework which can handle imbalanced data then make sure you understand why it can handle the imbalanced data. In particular, if the framework applies some weighting / balancing technique in the background you should be aware of this and be able to explain it.


Ploting eigenvectors I've generated two clouds of 3d points from multivariate_normal

data = np.random.multivariate_normal([2,2,2],[[1,0,0],[0,5,0],[0,0,10]],
size=500)
data = np.vstack((data, np.random.multivariate_normal([-2,-2,-2], [[1,0,0],[0,5,0],[0,0,10]], size=500)))
data = data - data.mean(axis=0)

And try to do PCA like this

covmat = np.cov(data.T)
v, W = np.linalg.eig(covmat)

And draw:

def get_vec(eig_v, eig_vec):
    t = np.linspace(0, eig_v)
    return np.array([np.array(v * eig_vec) for v in t])

def ang(v1, v2):
    return np.rad2deg(np.arccos(np.dot(v1,v2)/np.linalg.norm(v1)/np.linalg.norm(v2)))

l1 = get_vec(v[0], W[:,0])
l2 = get_vec(v[1], W[:,1])
l3 = get_vec(v[2], W[:,2])
x = data[:,0]
y = data[:,1]
z = data[:,2]
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot(l1[:,0],l1[:,1],l1[:,2], c='r')
ax.plot(l2[:,0],l2[:,1],l2[:,2], c='b')
ax.plot(l3[:,0],l3[:,1],l3[:,2], c='y')
ax.scatter(x,y,z,c='g')
plt.show()

This is what I get:

It's clearly visible that axes are not orthogonal. I've checked it and they seem to be orthogonal with regard to numbers:

print(ang(W[:,0], W[:,1]))
print(ang(W[:,0], W[:,2]))
print(ang(W[:,1], W[:,2]))

90.00000000000003
89.99999999999999
90.0
Could it be that such a tiny error makes that much visual difference?

The PCA projections do not look not orthogonal because your figure axes are not equal.

Set all axis equal with something like this:

ax.axis('equal')

or

ax.xlim(-5, 5)
ax.ylim(-5, 5)
ax.zlim(-5, 5)
ax.gca().set_aspect('equal')


Deep Learning to estimate what is beyond the edge I have an image of some data which is approximately 4,000 x 8,000 pixels.  I am interested in finding out if anyone has used a deep learning algorithm to predict what would be on the image if it extended 100 more pixels in each direction.  I would imagine that the data could be trained on smaller rectangles, and then the rules developed would be used to extend beyond the image given.  Has anyone seen a problem like this (and is there a reference)?   Even if not, what deep learning scheme would be best for this?

I think the closest problem that has been addressed with deep learning is image inpainting, that is, filling a blacked out region in the image:

For instance, this paper: Semantic Image Inpainting with Perceptual and Contextual Losses.

So it is certainly possible to fill missing information from an image with deep learning.


Outlier detection by unsupervised algorithm: Fraud Detection I have set of 300,000 set of rows with credit card transactions and my job is to find outliers (suspicious transactions) in those dataset.

I have created around 5 features (All continuous data, with 1 column as transaction id)

I need to return list of all transaction id,which looks suspicious

What have I tried

I have tried using K means algorithm, but it does does not fit my laptop's memory (8 GB) and it crashes.

I wanted to try 1 class SVM, but I do not see any good tutorial to get me started. I tried scikitlearn official tutorial, but it already has outliers added to it and they are just plotting it.

http://scikit-learn.org/stable/auto_examples/svm/plot_oneclass.html#sphx-glr-auto-examples-svm-plot-oneclass-py

http://scikit-learn.org/stable/auto_examples/covariance/plot_outlier_detection.html#sphx-glr-auto-examples-covariance-plot-outlier-detection-py

How do I automatically detect the outliers and return those observations ?

First of all, start with a subset until you know what you are doing.
There is no use in waiting for hours for a result that doesn't work, or to run out of memory, or to optimize, just to find out it does not work.

Secondly, makes sure your preprocessing is very very well done. Bad preprocessing will hurt your algorithms.

From my experience, one-class SVM does not work well. It assumes all your training data is 'normal' class (no outliers), and this a representative sample of all normal data. Secondly,.you need to tune kernel parameters, but how.can you do so without labeled anomalies?

Instead, I'd try knn outlier detection, LOF and LoOP.

But for these you need to make sure your distance is a very good measure of similarity. If you don't preprocess well, distance does not work, and then nearest-neighbor methods don't work either.