Batch Normalization vs Other Normalization Techniques In the context of neural networks, I understand that batch normalization ensures that activation at each layer of the neural net does not 'blow-up' and cause a bias in the network. However, I don't understand why it would be used as opposed to other normalization techniques such as Cosine or Weight normalization, these achieve the same goal and don't seem to be any more computationally complex.

Could someone please explain to me the advantages and disadvantages of using batch normalization vs other normalization techniques, and which contexts batch norm would be most beneficial?

Cosine normalisation is result of the fact that we bound dot
product and hence decrease the variance, when we use cosine similarity or centered cosine similarity instead of
dot product in neural networks (which is quasi ground-stone in NN)

Main benefit of cosine normalisation is Cosine
normalization bounds the pre-activation of neuron within
a narrower range, thus makes lower variance of neurons.

Also it does not depend on any statistics on batch or mini-batch
examples, and performs the same computation in forward
propagation at training and inference times. In convolutional networks, it normalizes the neurons from the receptive fields rather than the same layer or batch size.

Have a look at this paper showing emipirically comparison between normalisations you mentioned. C.N. comes on top.


Why do we have multiple neurons in the output layer of a neural network? Why do we have two neurons in the output layer?

What does each neuron mean?

If our classifier is a binary classifier, will we have only one neuron in the output layer?

Here is a picture of the neural network:

If you want multiple things out of your network you need multiple output nodes. In the case of multiclass classification you want multiple outputs, one for each class. These represent the probability distribution over the different classes. For binary you can get away with only one output because the other class has 1-P as the probability. You could say the same for multiclass that you need one less, however this is commonly parameterized with a softmax which needs all values for the denominator.

Other cases might be for example one output per pixel, for example for image segmentation.


What to infer from a given scatter plot? I am quite new to data science and I have been given the task to find relationship between two entities for a given data.This is the scatter plot graph I made

It is a time series data.Here x axis is the mean and y axis is the mse (mean square error).

What I can infer from this graph is the mean and mse don't really go along with each other.In other words when mean and mse doesn't share any strong relationship?Is my understanding of the graph correct?Is there anything else that can be found?

It is not easy to say based on this graph. Maybe if you could apply a transformation, it was easier to see a trend. I would have at least plotted them on log-log, log-linear, etc. to get better idea.


is multicollinearity a problem also in classification? This might be a dumb question, but from what I'm grasping, multicollinearity seems to be uniquely a problem in regression problems, and therefore models.

I was thinking about this very issue for some time. It seems like in machine learning, the multicollinearity is usually not such a big deal because it should not mess up the prediction power as such. It is problematic for estimation of the effects, for coefficients of the individual variables - hence, the problem with regression.
This post summarizes it quite well https://stats.stackexchange.com/a/168631/197070.


Explanation for MLP classification probability I showed some results of one implemented NN MLP model.

In the result, for classification of two categories, if I sum up the probabilities of both cats, them some sum would be greater than 1.

When I was asked why the sum is greater than 1, I gave the guess that the probability stands for confidence concept in mathematics.

Could anyone tell me am I right?

It depends a little on the specific model.

If your model is using a softmax output layer, then the values are usually interpreted as mutually-exclusive class probabilities and should sum to 1.

If your model is using a sigmoid output layer, then the values for a classification problem can be interpreted as the individual, non-exclusive probabilities. In that case, it is possible for an example to not be in either of your classes (an output close to [0,0]), or to definitely be in both classes (an output close to [1,1]) - if that is not the case in the problem that you have trained the NN to model, then you might want to consider altering the model.

So, if your classification should be mutually-exclusive, then the person asking you the question has a valid point. The probabilities should sum to 1, and you may need to fix the model. In that case, you should use a softmax output layer, and would not have this problem because the probabilities would always sum to 1.

Whether it is worth fixing the model, depends on your understanding of mutually-exclusive vs non-exclusive classes in the problem, whether the current model meets its goals (it is not a big issue in practice, if the model is not a strict theoretical match to the problem, but has a perfectly good accuracy) and how much effort it would take to alter your code and re-train the model.

Your statement:

  the probability stands for confidence concept in mathematics.

is technically correct, but not useful as an answer to your colleague's query. For classification, the terms confidence and probability are almost interchangeable, so you cannot really use one word to explain the other. If someone asked why one of the numbers was larger than another in your model, you would not answer "that is because x is greater than y" - essentially this is what you have done here. So your statement is not wrong as such, but neither does it explain the result.


When does decision tree perform better than the neural network? I was experimenting with different modelling methods including KNN, Decision Trees, Neural Networks and SVN and trying to fit my data to see which works the best. To my surprise, the decision tree works the best with training accuracy of 1.0 and test accuracy of 0.5. The neural networks, which I believed would always perform the best no matter what has a training accuracy of 0.92 and test accuracy of 0.42 which is 8% less than the decision tree classifier.

Could someone please explain the circumstances/cases where neural networks could have low accuracy when compared to a modelling technique like the decision tree. I had tried my neural network with different configurations like:

1 hidden layer and 1 neuron : Train Accuracy 34% Test Accuracy 42%
7 hidden layers and 5 neurons in each layer: Train Accuracy 79% Test Accuracy 42%
1 hidden layer and 100 neurons: Train Accuracy 34% and Test Accuracy 35%

but not in a single case, I found the neural network to beat the decision tree test accuracy of 50%.

Neural Networks, in my experience have several hyper-parameters (number of layers, neurons per layer, activation functions, optimizers, regularizers, etc.) and are very hard in finding the best configuration for each task. In fact in most cases it's not even worth it trying to find the optimal configuration as other classifiers can outperform Neural Networks with default hyper-parameters. Furthermore, NNs require caution as they are prone to overfitting.

For most tasks where you deal with structured data, I've found tree-based algorithms (especially boosted ones) to outperform NNs.

Some NN architectures are state-of-the-art tasks where we have a lot of unstructured data (e.g. CNNs for image-related tasks).

Finally, I'd like to say that there are no absolutes (e.g. SVMs will alawys outperform DTs). There is also a theorem along these lines: No Free Lunch Theorem.


Unbalanced training data for different classes What precautions do I need to take while trying to develop a CNN for classification of images if there is much more training data for one label. For example:

label1 : 1000 images
label2 : 100 images
label3 : 100 images
label4 : 100 images

Numbers will become larger later but proportion is likely to stay the same.

Thanks for your insight.

The problem you face is commonly called the class imbalance and has been the subject of quite a bit of research. Here's a literature review, if you're interested: He, H., & Garcia, E. A. (2008). Learning from imbalanced data

In particular, you might encounter two forms of imbalance:

Absolute imbalance/rarity occurs when, while you have plenty of data from some classes, you have only a few examples of some other classes (or subconcept of a class). In this case, the issue is that there may not be enough data for the learning algorithm to learn the minority class. In the example you give, with 100 examples of the minority class, depending on the nature of your data, you might have this problem. If you expect to have more data in the future, however, absolute rarity should eventually no longer be an issue.

Relative imbalance, on the other hand, does not go away with more data. You have a relative imbalance when the prior probability of certain classes is much larger than that of some other class or classes. For example, you will always have 10 times as many examples of class 1 than examples of class 2, because class 1 occurs 10 times as often.

Most learning algorithms for classification optimize for accuracy, or something similar like RMSE. This means that, when solving the real classification problem is hard enough, and the data is strongly imbalanced towards one class, the model may resort to predicting the majority class whenever there's a doubt. Recall for the majority class may be great, but not so much for the minority class.

This becomes an issue in many domains where detection of the minority class is particularly important. For example, in medical diagnoses, we may be willing to sacrifice overall accuracy (because of more false positives) in order to have a better true positive rate.

In short, it depends on your domain. Are you OK with optimizing for overall accuracy, or is it more important to have comparable performances across the classes? If you choose the latter, then there are a few things you may try:

Use cost-sensitive learning: Certain learning algorithms and implementations allow you to assign a cost to each class, essentially describing how bad it is if an example of that class is misclassified. If I recall correctly, this is usually considered the best approach when you have a good idea of these different costs.
Rebalance the classes: You can oversample the minority class (which carries a risk of overfitting), undersample the majority class (which is dangerous if you don't have a lot of data), use a mixture of both or perhaps something a bit more advanced like synthetic sampling (attempt to generate new examples of the minority class using something like SMOTE)

Overall, you should also be careful to pick the right evaluation metric. Evaluating your model using accuracy may lead you to believe that your model is performing very well when in fact it is classifying everything into the majority class. There are many metrics you may use, each of which have their pros and cons. The area under the ROC curve (AUC) is a common metric which gives you a general idea of your model's performance averaged over different class misclassification costs. If you can plot the ROC curve for multiple models and notice that one curve dominates the others across the entire width of the plot, then that is the clearest sign that you have a winner. There's a whole chapter on the subject, if you're interested, in the book Imbalanced Learning: Foundations, Algorithms, and Applications.


Extract and normalize date from string for the proper text mining I am trying to perform string delivery classification (X: delivery_string, y: delivery_string_relevance (values 0 or 1)). I am using DTM (document term matrix) for the feature extraction, and experimenting with text-mining algorithms. However, I still believe that I can preprocess my raw strings better (before creating the DTM) by extracting delivery dates and obtaining one common format representation for all of them. Any suggestions on good date parsers? datefinder.find_dates() in Python is doing quite a decent job, however still fails for some cases, such like when string contains "delivery time 10 - 12 days" it extracts it as 10-12-2017.

Sample data (language used Dutch):
beschikbaar vanaf 04 okt // beschikbaar vanaf: 05-08-2017 // 8 - 9 weken // 8 - 9 werkdagen // 8 dag(en)

The best option is dateparser, whose goal is to parse natural language dates with support for Dutch. There is another package recurrent which does natural language parsing of dates focusing on recurring events.


80-20 better than full dataset for LightGBM Recently I have been using LightGBM as regressor in order to predict, on a dataset of 20 thousand observations and 40 variables.

I have two modes, 1) Production and 2) Testing. The first one just trains a model with the whole dataset. The second does the same with an 80% of the dataset and tests over the remaining 20% (80-20 done with train_test_split, from sklearn.model_selection, no seed used).

In both cases I show the neg_mean_squared_error at the end of the execution. In the first case for the whole dataset, in the second I get two values (training and testing)

I am shocked to see that while in the first case my error is around -10, in the second the values are -5 (training) and -5.3 (testing). An average value of my objective variable can be 80. I would expect to lose accuracy since I train with less data and then I test over a different sample.

Question: There is any theoretical reason that explains that a 80-20 test gets a "better" neg_mean_squared_error than in the full data case? Or it has to be a (sneaky) bug in my code?

I cannot think about any theoretical reason. However:

It may be the case that the neg_mean_squared_error is not correctly implemented. Try another metric, like Median or Mean Percentage Error and see if that continues to happen.
It may be related to how the data is distributed, but still that's not common.

Lastly, I wouldn't put a model in production (if by that you mean deploying into a product, for example) without testing its performance in a test set, even if in another instance of the model, trained using cross validation, has good performance. One reason: it may be the case that a small part of the training data breaks your model, and does not generalise.


Binary classification: how to transform features in real numbers? I want to train a binary classification algorithm for spam detection using labeled data set. The dataset has the following features:
Email address, text message (split into subject and corpus), date

An example of data is:
Email | Subject | Corpus | Date
Gg12_school@gmail.com | Example | this is just an example of my dataset  | 2020/08/20

What I would like is to transform data features in real numbers and binarize email addresses.
As algorithm I was thinking of
SVM and/or Naïve Bayes.
My difficulties are, however, in how transform data features in real numbers in order to get more parameters in my classifier.
I am using Python.
Could you please give me an example of how to do it?

The term you are looking for is text classification. There exists a huge number of tutorials and papers out there, for example this tutorial  and this survey.


Why TREC set two task: document ranking and passage ranking TREC is https://microsoft.github.io/TREC-2019-Deep-Learning/

I am new to text retrieval. Still can not understand why set the two similar task.

Thank you very much.

The idea behind the two tasks is to explore how document length affects the effectiveness of the different retrieval models.

Ellen Voorhees
TREC project manager
NIST


Forward pass vs backward pass vs backpropagation As mentioned in the question, i have some issues understanding what are the differences between those terms.

From what i have understood:

1) Forward pass: compute the output of the network given the input data

2) Backward pass: compute the output error with respect to the expected output and then go backward into the network and update the weights using gradient descent ecc...

What is backpropagation then? Is it the combination of the previous 2 steps? Or is it che particular method we use to compute dE/dw? (chain rule ecc...)

In brief, backpropagation references the idea of using the difference between prediction and actual values to fit the hyperparameters of the method used. But, for applying it, previous forward proagation is always required. So, we could say that backpropagation method applies forward and backward passes, sequentially and repeteadly.
Your machine learning model starts with random hyperparameter values and makes a prediction with them (forward propagation). Then it compares with real values while adjusting those random initial values (backpropagation), trying to minimize the error (depending of your objective function and optimization method applied). And then, it starts over again, until you reach the stopping criteria.
You may find a better explanation in this question.
Furthermore, you will find more topic explanation here.


Need sugestion to accuire a job in data mining I interested in data mining specialization and have been enrolled in data mining specialization course at Coursera by University of Illinois. This specialization course contains four parts. The very first course is "Data Visualization" which contains more conceptual knowledge rather than technical skills that not as expected.

Anybody can give me suggestion what can I expect after completed that course in the case to find a job in data mining specialization?

If I am intended to acquire engineering job, is this course right for me?

Welcome to the site! The course you are taking sounds like a good start; but I am not sure if it will be enough to land you a job as a Data Engineer (depending on your other qualifications, of course).
If you are willing to invest more time, here is a great text on how to learn Data Science in one hundred hours.


which is the most effective(accurate) face detection method in python i try haar_cascade for face detection and LBPH for face recognition , but the result wasn't good enough, please suggest good ways to detect and recognize faces.

my aim is to create an app which take a photograph of students and by scanning this one photograph, it will predict which student is present and which is absent

You can use the Facial landmarks library, it's very easy to use
you will get 68 point and compare them from your dataSet very dirty and easy way of doing things note that

The users must take the pictures at the same position to keep the same depth


Which order is correct Feature Selection then Outlier Detection or vice versa? which of these orders is correct?

First (Feature Selection) Second (Outlier Detection)

or

First (Outlier Detection) Second (Feature Selection)

In majority of the cases feature selection should be done after outlier detection. Outlier detection should be done at the initial stage of data pre-processing while feature extraction / selection can be don in the last part of  data pre-processing.

Outlier must be detected beforehand so that actual behavior o that particular predictor is known. Then depending upon the behavior after removal of outlier it can be decided whether to keep that as feature or not.

Many feature selection algorithms like PCA, regression, etc are sensitive to outliers and so if such algorithms are used to extract the features then it would be better to remove the outliers beforehand.


How to add time as a feature into clustering algorithm? I've wrote here about my problem. And now I have a new one.
How to use time as a feature? I have a pandas DataFrame with date-time column('date') and I don't know how to use it as a feature. For now I'm using  TfidfVectorizer.

from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer(min_df=5, max_features=10000, ngram_range=(1, 2))
vz = vectorizer.fit_transform(df['word'])

Where 'word' column is tokens on my text.
I'm using MiniBatchKMeans. Here is code example:

from sklearn.cluster import MiniBatchKMeans

num_clusters = 32
kmeans_model = MiniBatchKMeans(n_clusters=num_clusters, init='k-means++',     n_init=1,  init_size=1000, batch_size=1000, verbose=False, max_iter=1000)
kmeans = kmeans_model.fit(vz)
kmeans_clusters = kmeans.predict(vz)
kmeans_distances = kmeans.transform(vz)

I'm interested at how to add my time feature(my time column) to clusterizator and is there any possibility to make some kind of "weight" factor for time features and for different tokens.

I've decided to try LDA:

cvectorizer = CountVectorizer(min_df=4, max_features=10000, ngram_range(1,2))
cvz = cvectorizer.fit_transform(df['word'])
n_topics = 32
n_iter = 2000
lda_model = lda.LDA(n_topics=n_topics, n_iter=n_iter)
X_topics = lda_model.fit_transform(cvz)

Then I've concatenated time feature to the LDA vector:

 X_all  = np.hstack((X_topics, X_time))

Where X_time was my time feature column.


Advantages of Recurrent Neural Networks over basic Artificial Neural Networks I have started reading Deep Learning Book, and I am having trouble understanding the advantages of RNN. This part of confuses me:

  The unfodling process thus introduces two major advantages:

  Regardless of the sequence length, the learned model always has the same input size, because it is specified in terms of transition
  from one state to another state, rather then specified in terms of
  variable-length history of states.
  It is possible to use same transition function f with same parameters at every time step.

  These two factors make it possible to learn a single model f that
  operates on all time steps and all sequence lengths, rather then
  needing to learn a separate model for all possible time steps.
  Learning a single shared model allows generalization to sequence
  lengths that did not appear in the training set, and enables the model
  to be estimated with far fewer training examples than would be
  required without parameter sharing.

I understand the 2nd advantage. Because the computations are recurrent, the input besides the current element in the sequence is the output of the previous hidden state which has the same structure as the current hidden state, thus the shared parameters.

But I do not understand the first advantage. I cannot visualize it or mathematically prove it, or at least I dont know how. Can anyone help me with this one? And if anyone has anything else to add on the difference ( advantages ) of RNN over ANN I would really appreciate it. Thanks in advance!

okey, I will try to explain them as easy as possible.

  Regardless of the sequence length, the learned model always has the same input size, because it is specified in terms of transition from one state to another state, rather then specified in terms of variable-length history of states.

I will use a simple example for simplification but it does not lead to losing generalisation. Suppose that your task is to add to binary numbers. Numbers are stored bitwise in memory and usually for applying any arithmetic operations, they should have a same size. What you need for adding two numbers is to learn how to add zeros and ones and when to output carray to the next step if you use sequential learning approaches, like RNNs. In this case, it is not really important that the two numbers do not have the same shape due to the fact that they both can be resized to the biggest size by adding zeros to their most significant bits. After resizing the inputs to have the same shape you have two options, to use MLPs or RNNs. If you use MLPs what the network learns is entirely different from what an RNN learns. The former does not learn the transition of carray at least as the way RNN learns. Another difference is that your MLP always will be restricted to the size which it was trained while the RNN model will be able to add two numbers with even more bits. To explain it why, for MLPs, all inputs which are connected to the hidden layers or maybe output layers, have weight. Consequently, increasing the number of inputs will lead to more weights which are not trained yet. You are not allowed to input a signal which its size is not equal to the input size of the MLPs. On the contrary, RNNs are exploited in a different way.

First, you should know RNNs better. Try to think of hidden layers of anRNN. They are like usual MLPs. Their difference is that for each node, the inputs come from the previous step's outputs and the current time's inputs. Bear in mind that the outcomes of the previous time step are not coming from previous neurons. The reason and I guess the main answer to your question is that each RNN is repeated for each input in time $t$. It means you have just an MLP which is used for all time step. Suppose you are at the middle of the calculations. Two inputs are $1$ and $1$. You RNN takes them and the carray value and outputs $1$ as the result of time step $t$ and outputs $1$ to the next step as carray. For the next time step the same RNN is again used and takes the inputs alongside the carry which is coming from the previous step's outputs and outputs the corresponding outputs and carries.

Due to the nature of RNNs which just take the inputs of time step $t$, they are capable of dealing with signals with different lengths. The reason is that The RNN is used for each time step. This behaviour is usually called unfolding the network.


training neural network I was given the task as follows,

  Scrape articles appearing in Times of India since 2010 on HIV and AIDS. Classify them using training a neural network of your choice. Find patterns in those articles, analyze the data.

I have done the scraping part, But I have no clue what to do now with the data I have extracted. I searched for hours on internet but could not find any help.
I know theory part about the neural network but lack knowledge for implementing it on a text. It would be of great help if someone can suggest me any method or provide me some helpful link.

There are many solutions for this task. I suggest one of them. As you know, words have relation and if you choose to give each word a special code, you can't have this relation. Consequently, first try to use an embedding network in order to assign each word a code. Then assign each article a label. Next, for each article, you have a sequence of words, codes, which are now embedded. You can employ LSTM networks for classification.

If you are not very familiar with the concepts I referred to, you may want to look for Word2Vec.


Validation Loss is not decreasing - Regression model I'm trying to train a regression model with 6 input features.
No. of tuples - 7287.
Train set - 5465
Test set - 1822

I've tried changing no. of hidden layers and hidden neurons, early stopping, shuffling the data, changing learning and decay rates and my inputs are standardized (Python Standard Scaler).
Validation loss doesn't decrease.

NN_model = Sequential()

NN_model.add(Dense(128, kernel_initializer='normal',input_dim = x_train.shape[1], activation='relu'))

NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))

NN_model.add(Dense(1, kernel_initializer='normal',activation='linear'))
#adam=optimizers.Adam(lr=0.01,decay=0.0001)

NN_model.compile(loss='mse', optimizer='adam', metrics ['mean_absolute_error','mse'])
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=200)
NN_model.summary()
history=NN_model.fit(x_train,y_train,epochs=1000,batch_size=64,validation_data=[x_test,y_test],callbacks=[es])

Loss graph looks like:

Mean square error is very high and r2 score is 0.5276 for the train set and 0.3383 for the test set.

I've tried other machine learning models like Gradient Boosting Regressor, Random forest regressor, decision tree regressor but they all have high mean square error.

I'm new to keras and deep learning. How do I solve the issue?

Is this amount of training data enough for the neural network? I can't get more data.

the network architecture above is a very strange choice. When you have only 6 input features, it is weird to have so much Dense layers stacked.
if network is overfitting, WHERE IS DROPOUT? Why not trying some regularizers, if the latter does not help?
+1 for David Waterworth - correlation/causal analysis is not everything yet. Does linear regression provide better R-square values?
what is output(target) variable range? Maybe it should be mapped/scaled to something reasonable? (I judge from loss values).
activation function and initializers are important too. Try using different values, rather than relu/linear and 'normal' initializer.

EDIT: yes, this should be enough data, if your data has only 6 inputs. However, you can try augmenting data too, if it makes sense and you can make reasonable assumptions in your case - sometimes it gives difference in the long run, even if in the beginning you think it does not work.

EDIT2: with specific datasets, neural network can get into local plateau (not minima however), where it does not escape. To test this hypothesis,  you can set learning rate to small value and all initializers to generate small values too - then network may not go to this plateau suddenly, but goes to global minima instead.

EDIT3: increasing batch size leads to faster but poorer convergence on certain datasets. On a smaller network, batch size = 1 sometimes makes wonders.


Adam optimizer for projected gradient descent The Adam optimizer is often used for training neural networks; it typically avoids the need for hyperparameter search over parameters like the learning rate, etc.  The Adam optimizer is an improvement on gradient descent.

I have a situation where I want to use projected gradient descent (see also here).  Basically, instead of trying to minimize a function $f(x)$, I want to minimize $f(x)$ subject to the requirement that $x \ge 0$.  Projected gradient descent works by clipping the value of $x$ after each iteration of gradient descent: each negative entry is replaced with 0, after each step.

Unfortunately, projected gradient descent seems to interact poorly with the Adam optimizer.  I'm guessing that Adam's exponential moving average of the gradients gets messed up by the clipping.  And plain projected gradient descent has hyperparameters that can be tuned.

Is there a version of Adam that can be used with projected gradient descent?  I'm looking for a method that is an improvement on projected gradient descent, in the same way that Adam is an improvement on ordinary gradient descent (e.g., doesn't require hyperparameter tuning).  Is there any such algorithm?

tl;dr: All gradient descent-based methods have an update rule similar to Adam's, so I think they'd all give you some grief. I don't know of any optimizers tailored to your application (but maybe you could invent one!)

Long answer:

According to this, Adam has four hyperparameters that typically don't require much tuning (although of course model hyperparameters will still need to be tuned, such as number of hidden units per layer, etc.)

alpha. Also referred to as the learning rate or step size. The proportion that weights are updated (e.g. 0.001). Larger values (e.g. 0.3) results in faster initial learning before the rate is updated. Smaller values (e.g. 1.0E-5) slow learning right down during training
beta1. The exponential decay rate for the first moment estimates (e.g. 0.9).
beta2. The exponential decay rate for the second-moment estimates (e.g. 0.999). This value should be set close to 1.0 on problems with a sparse gradient (e.g. NLP and computer vision problems).
epsilon. Is a very small number to prevent any division by zero in the implementation (e.g. 10E-8).

There are several other optimizers out there, such as AdaGrad and RMSprop. The idea from the 2015 paper on Adam was that by doing a combination of AdaGrad and RMSprop, you get a better optimizer (Adam).

The algorithm for Adam is explained in the 2015 paper (Algorithm 1). The exponential smoothing portion applies specifically to the (biased) estimates of the first and second moment. The only time your actual parameter value plays a role is in the calculation of the gradient, and then in the update.

In the update step (last part of the while loop in Algorithm 1), you perform this calculation:

x_t = x_{t-1} - alpha*(unbiased estimate of the first moment)/(sqrt(unbiased estimate of the second moment) + epsilon)

If you restrict x to be greater than or equal to 0, then if x_{t-1} ever becomes zero, you can no longer update (the subtraction of alpha*(stuff) would result in something less than 0, which you then clip to be 0, which means you'd get 0 as the result in each subsequent update step).

Maybe that's the source of the issue? If so, I would recommend restricting your entire parameter space to be above zero. That is, in your initializer, make sure the initial values are all positive (i.e. don't use Glorot/Xavier initialization, because you could get initial values less than 0, as this initializer is based on the Normal distribution).


How does the forward method get called in this pyTorch conv net? In this example network from pyTorch tutorial

import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 3x3 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 3)
        self.conv2 = nn.Conv2d(6, 16, 3)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 6 * 6, 120)  # 6*6 from image dimension
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square you can only specify a single number
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

net = Net()
print(net)

net = Net()
input = torch.randn(1, 1, 32, 32)
out = net(input)
print(out)

Why is the method forward() not explicitely called? I mean how does just calling net(output) calls forward() ? (which is what happens as far as I understand)
By the way I dont understand what this line means:

super(Net, self).__init__()

I can imagine super() is calling the constructor of a parent class but …?

If you look at the Module implementation of pyTorch, you'll see that forward is a method called in the special method __call__ :

class Module(object):
   ...
   def __call__(self, *input, **kwargs):
      ...
      result = self.forward(*input, **kwargs)

As you construct a Net class by inheriting from the Module class and you override the default behavior of the __init__ constructor, you also need to explicitly call the parent's one with super(Net, self).__init__().


Why transpose of independent feature matrix is necessary in case of linear regression? I can follow classical linear regression steps:

$Xw=y$

$X^{-1}Xw=X^{-1}y$

$Iw=X^{-1}y$

$w=X^{-1}y$

However, on implementing in Python, I see that instead of simply using

w = inv(X).dot(y)

they apply

w = inv(X.T.dot(X)).dot(X.T).dot(y)

What is the explanation of the transpositions and the two times multiplication here? I'm confused...

That answer comes from the set of weights $w$ (or $\theta$) that analytically solves the cost function which is defined to be

$J(\theta) = (X\theta - y)^T (X\theta - y)$

(See here for more info)

Expanding the cost function we get

$J(\theta) = \theta^TX^TX\theta - 2 y^TX\theta + y^Ty$

(Note that all three terms come out to be scalers)

Before we take the next step, we need to brush up on derivatives of matrices

Some common matrix derivative formulas for reference:

$\frac{\partial (AX)}{\partial X} = A^T \ ;\  \frac{\partial (X^TA)}{\partial X} = A \ ;\ \frac{\partial (X^TX)}{\partial X} = 2X \ ;\ \frac{\partial (X^TAX)}{\partial X} = AX+A^TX$

Using those rules we can take the derivative of the cost function with respect to $\theta$

$\frac{\partial J(\theta)}{\partial \theta} = 2 X^TX \theta - 2 X^Ty$

Setting this to 0 we get

$2 X^TX\theta - 2 X^T y = 0$

solving for $\theta$ we get

$\theta = (X^TX)^{-1} X^T y$

Written in code that's

w = inv(X.T.dot(X)).dot(X.T).dot(y)

Hope this helps.


why transpose convolution is called "transpose" https://d2l.ai/chapter_computer-vision/transposed-conv.html
https://en.wikipedia.org/wiki/Transpose
I understand what transpose convolution does, but I am confused about the name of 'transpose'.
In linear algebra, transpose is the action which flips a matrix over its diagonal;, which is absolutely what transpose convolution does in this case.
So why named as transpose convolution？

Transpose in mathematics means to change the order of matrix in an opposite way, the same notion carries here but not the exact sense, you are talking about.
The same problem exists with the word 'convolution', it means something else in mathematics. What is done in deep learning in name of convolution is cross-correlation in mathematics.


Cost function in linear regression Can anyone help me about cost function in linear regression. As from the below plot
we have actual values and predicted values and I assumed the answer as zero but it actually is 14/6?

Can anyone help out please?

$h_\theta$ implies that you're trying to model the relation between $h$ and $x$ with an straight line coming from the origin $(0,0)$. The parementer $\theta_1$ is the slope of this line.

Evaluating $J(\theta_1=0) = J(0)$, implies that $h_\theta(x_i)=0$ whatever the value of $x_i$ is. Since $m=3$, and the labels are $y_1=1, y_2=2, y_3=3$

$J(0) = \frac{1}{6}[(0 -1)^2 + (0 - 2)^2 + (0-3)^2] = \frac{14}{6}$

The value of $\theta_1$ for which $J(\theta_1) = 0$ is obviously 1


How will Occam's Razor principle work in Machine learning The following question displayed in the image was asked during one of the exams recently. I am not sure if I have correctly understood the Occam's Razor principle or not. According to the distributions and decision boundaries given in the question and following the Occam's Razor the decision boundary B in both the cases should be the answer. Because as per Occam's Razor, choose the simpler classifier which does a decent job rather than the complex one.

Can someone please testify if my understanding is correct and the answer chosen is appropriate or not?
Please help as I am just a beginner in machine learning

Occam’s razor principle:

  Having two hypotheses (here, decision boundaries) that has the same empirical risk (here, training error), a short explanation (here, a boundary with fewer parameters) tends to be more valid than a long explanation.

In your example, both A and B have zero training error, thus B (shorter explanation) is preferred.

What if training error is not the same?

If boundary A had a smaller training error than B, selecting becomes tricky. We need to quantify "explanation size" the same as "empirical risk" and combine the two in one scoring function, then proceed to compare A and B. An example would be Akaike Information Criterion (AIC) that combines empirical risk (measured with negative log-likelihood) and explanation size (measured with the number of parameters) in one score.

As a side note, AIC cannot be used for all models, there are many alternatives to AIC too.

Relation to validation set

In many practical cases, when model progresses toward more complexity (larger explanation) to reach a lower training error, AIC and the like can be replaced with a validation set (a set on which the model is not trained). We stop the progress when validation error (error of model on validation set) starts to increase. This way, we strike a balance between low training error and short explanation.


xgboost: Is there a way to perform regression on rates/percentages data? I have a dependent variable, $Y$, that is made up of rates/percentages data, so each value is between $0$ and $1$. I was attracted to the xgboost library because it allows focusing in on specific subsets of the data in training itself, but I am stumped as how to perform regression on the data I have.

Normal OLS regression will produce outputs that will be over and under the [0,1] range if you do not change the likelihood to be the Beta distribution, or something else bounded in the same range, but will xgboost suffer the same mistake?

Any advice on trying to get this to work would be greatly appreciated by me.

In principle: yes, you will have the same problem as with OLS. However, since xgboost is tree-based (and by that non-parametric), you may get relatively accurate estimates, meaning that values which are below zero or above one would be rare (at least the problem should be less severe than with OLS). In this case you could simply restrict results to $\hat{y} \in [0,1]$. An alternative would be to do a multiclass classification task where you have 100 classes $y=[1\%,2\%,...,100\%]$. Boosting usually performs well on classification tasks.

Just a hint: xgboost can be somewhat "heavy" in data handling. Boosting tools such as lightgbm are a good alternative. I prefer lightgbm over xgboost.


Why CNN doesn't give higher accuracy over simple MLP network? [From Keras examples] I'm still new to machine learning and just came across powerful deep learning library, Keras.

I've read Keras document and tried few Keras examples on Github here. I've also studied some basic knowledge and concepts of deep learning from several sources but still haven't really had solid understanding in CNN and RNN which look to be very powerful networks.

So, to prove my assumption, I downloaded reuters_mlp.py example from Keras Github which originally uses simple MLP networks as a model. I combined the idea of CNN which I got from imdb_cnn.py example to reuters_mlp.py example and then observed the result.

Surprisingly, the result didn't come out like I expected. CNN performed worst than simple MLP networks. Can someone please explain why the accuracy of CNN is lower than the simple MLP networks?

Here are the outputs (Tensorflow as backend)

8982 train sequences, 2246 test sequences, 46 classes, num_words=1000

MLP (sequences_to_matrix, mode=bianry):

Epoch 1/5
8982/8982 [==============================] - 3s - loss: 1.3236 - acc: 0.6984
Epoch 2/5
8982/8982 [==============================] - 2s - loss: 0.7182 - acc: 0.8250
Epoch 3/5
8982/8982 [==============================] - 2s - loss: 0.4544 - acc: 0.8864
Epoch 4/5
8982/8982 [==============================] - 2s - loss: 0.3197 - acc: 0.9192
Epoch 5/5
8982/8982 [==============================] - 2s - loss: 0.2511 - acc: 0.9356
1920/2246 [========================>.....] - ETA: 0s

Test loss: 1.05213204963 Test accuracy: 0.785396260071

CNN (pad_sequences):

Epoch 1/5
8982/8982 [==============================] - 81s - loss: 1.9794 - acc: 0.5181
Epoch 2/5
8982/8982 [==============================] - 78s - loss: 1.4289 - acc: 0.6591
Epoch 3/5
8982/8982 [==============================] - 79s - loss: 1.1546 - acc: 0.7175
Epoch 4/5
8982/8982 [==============================] - 78s - loss: 0.9639 - acc: 0.7663
Epoch 5/5
8982/8982 [==============================] - 77s - loss: 0.8378 - acc: 0.7935
2240/2246 [============================>.] - ETA: 0s

Test loss: 0.960687935512, Test accuracy: 0.764470169243

CNN (and RNN) models are not general improvements to the MLP design. They are specific choices that match certain types of problem. The CNN design works best when there is some local pattern in the data (which may repeat in other locations), and this is often the case when the inputs are images, audio or other similar signals.

The reuters example looks like a "bag of words" input. There is no local pattern or repeating relationships in that data that a CNN can take advantage of.

Your results with a CNN on this data set look reasonable to me. You have not made a mistake, but learned how a CNN really works on this data.


How to know if the deep learning model is overfitting or not? I created a deep learning (CNN) model, I used data augmentation and two dropout layers (0.5).

How to know if the deep learning model is overfitting or not?

Please consider going through other questions before posting a question. This will not only help you learn more by showing you multiple perspectives to a problem, but will also help you frame a clearer question explaining why the answers to other questions don't apply to your problem.

Refer this link for your answer.


What happened to the accuracy before and after 75th epoch. Why its unstable at first, then stepped up after the 75th epoch? What happened to the accuracy before and after 75th? Why before 75th, its unstable and suddenly stepped up after that? I am using RESNET and CIFAR-10 dataset.

I think that it is a convergence of the gradient descent happened. While training,  we are looking for correct weights and that is why we see these fluctuations on the picture. When we are close to a global minimum (found good weights), the model starting to converge. That is what happened after the 75th epoch in this case.

By the way, you have an overfitting


Detect a bot in a audit log I need to develop a report that will show automated queries in an audit log of queries on a system of the company. The logs have this fashion:

query_id id          query_time
1        1           2018-02-01 00:09:02
2        1           2018-02-01 00:24:55
3        1           2018-02-01 00:58:55
4        1           2018-02-01 01:01:49
5        1           2018-02-01 01:05:42
6        1           2018-02-01 01:18:56

Where query_id is an index of the query, the id is who has queried and query_time is the time of the query.

I tried to make the difference between the time of the queries and count how much was less than 60 seconds, but don't make the job. One other way is see the periodicity of the queries. I was thinking if there was a better way to do this. Maybe something using time series. There is some technique that could identify a query bot?

It sounds like you have good initial data but you might need to develop a meta data set. I would start by taking all this info and building behavior profiles. How often do they query per session? When do they query? How many sessions per hour/day/week/month? And so on, I think you get the idea. It just doesn't strike me like the raw file is the way to go, you need a little bit more to work with but you can get there with what you have now. Get down to various behaviors, one ID per row, and you'll be in a much better position to model this out.


Attitude to text mining and preparing tokens, irrelevant words, low accuracy For purpose of quite big project I am doing a text mining on some documents. My steps are quite common:

All to lower case
Tokenization
Stop list and stop words
Lemmatizaton
Stemming
Some other steps like removing symbols.

Then I prepare bag of words, make DTF and classify to 3 classes with SVM and Naive Bayes.

But the accuracy I get is not too high (50-60%). I think that may be because in array of words after all the steps are still many words very irrelevant like first and last name from document. What is the attitude in such case? What can be done during preprocessing to make classifiers work better with higher accuracy?

I was thinking about preparing some dictionary consists of all relevant words to my area but it could be too hard and for sure some important words will be missed.

Any advice what could be done here?

The question is very broad, so the general answer is: it depends on the specifics of your data and the problem you're trying to solve. The general idea would be to analyze what's happening:

Is it actually possible to solve the problem given the indications in the data? Would a human expert do better than 50-60% accuracy given only the information in the data? If yes which clues would they use, and are these potential clues directly available as features to the learning algorithm?
Ratio between size of data and size of features: if there are not enough instances, the algorithm doesn't have enough to generalize from. If there are too many features, the algorithm is likely to overfit (i.e. consider things which happen by chance in the data as significant patterns). And of course if there are not enough features the algorithm doesn't have the indications necessary to make accurate predictions.

Common things to try:

remove features (here words) which appear very rarely, because they can't help and can cause overfitting.
if the problem seems to be that words are not informative enough, try with bigrams or even trigrams... but be careful to avoid overfitting, the number of features is likely to increase a lot.
use weights (e.g. TFIDF) to help the learning algorithm focus on the relevant features.
depending on the task, more sophisticated methods could be relevant: topic modeling, disambiguation techniques, syntax...


Confused about description of YearPrediction Dataset https://archive.ics.uci.edu/ml/datasets/YearPredictionMSD

According to the description given in the above link,
the Attribute information specifies "average and covariance over all 'segments', each segment being described by a 12-dimensional timbre vector". So the covariance matrix should have 12*12 = 144 elements. But why is the number of timbre covariance features only 78 ?

You are right, the covariance matrix should have n^2 elements. However, since cov_{i,j} = cov_{j,i}, there is no need to have a repeated feature cov_{j,i} if cov_{i,j} is already accounted for. Hence there will be only n*(n+1)/2 = 12*13/2 = 78 unique covariances and thus only 78 unique covariance based features (n of those will be variances).


replicability / reproducibility in topic modeling (LDA) If I'm not wrong, topic modeling (LDA) is not replicable, i.e. it gives different results in different runs. Where does this come from (where does this randomness come from and why is it necessary?) and what can be done to solve this issue or gain more stability across runs?
Thanks for the help.

LDA is Bayesian model. This means the desired result is a posterior probability distribution over the random vectors of interest (probability of topics etc. having seen some data).

Inference for many Bayesian models is done by Markov Chain Monte Carlo. Indeed the wiki on LDA suggests that Gibbs sampling is a popular inference technique for LDA. MCMC draws random samples to provide an approximation of the posterior distribution.

Variational inference methods should typically be deterministic, but I'm not too familiar with the VB inference for this particular model.

Also one can typically replicate runs of random algorithms by setting of random number generation seeds (if your purpose is scientific).

In either case if the results from a Bayesian model show huge variability in the the parameters of interest, it may be telling you that the model is not a good fit for the dataset, or the dataset is not big enough for the model you are fitting.

EDIT: I don't know which inference method (Gibbs, VB etc.) the backend of your software is using, so it's not possible to determine what type (if any) of randomisation is going on.

For scientific purposes, you'll probably want to read up some more on Bayesian inference. Standard software (e.g. the LDA in scikits.learn) will give you a summary of the outputs of the inference (e.g. most coders just want the best assignment of docs to topics). There's more information hanging around behind the scenes which you might be able to get access to, and could be useful.

E.g. (roughly) for scientific applications of Gibbs sampling methods we'd typically run multiple chains, drop the first N samples generated by each, and check if the resulting samples look like they came from the same distribution. If you're concerned about dependence on seeds etc. and your backend is the Gibbs sampler you will want to check out MCMC convergence diagnostics for this model.


Cost tends to infinity when relu activation is used I have implemented a neural network with 1 hidden layer using sigmoid activation unit but after watching a video on how relu activation function can be much faster a tried implementing it but the cost function is either Nan or inf in python. I have found that many people have the same problem but I couldn't find any solution on the internet.
I have added '#changed' comment to the lines that I have changed from the previous version in which I used sigmoid activation function.
Related python code snippets:

def relu(arg):  #I have tried both relu and leaky relu
    return 1*(arg<0)*0.0001*arg + (arg>=0)*arg

def reluGrad(arg):
    for i in range(arg.shape[0]):
        for j in range(arg.shape[1]):
            if arg[i][j]>0:
                arg[i][j]=1
            else:
                arg[i][j]=0
    return arg
def softmax(x):
    x = x.transpose()
    e_x = np.exp(x - np.max(x))
    return (e_x / e_x.sum(axis=0)).transpose()

#forward prop:
a1 = np.insert(data,0,np.ones(len(data)),1).astype(np.float64)
    z2 = a1.dot(theta1)
    a2 = relu(z2) #changed
    a2 = np.insert(a2,0,np.ones(len(a2)),1)
    z3 = a2.dot(theta2)
    a3 = softmax(z3) #changed
#compute the cost:
    cost = -(output*(np.log(a3))+(1-output)*(np.log(1-a3))).sum()
    cost = (1/len(data))*cost + (lamb/(2*len(data)))*((np.delete(theta1,0,0)**2).sum() + (np.delete(theta2,0,0)**2).sum())

backProp:
    sigma3 = a3-output
    sigma2 = (sigma3.dot(np.transpose(theta2)))* reluGrad(np.insert(z2,0,np.ones(len(z2)),1)) #changed
    sigma2 = np.delete(sigma2,0,1)
    delta2 = (np.transpose(a2)).dot(sigma3)
    delta1 = (np.transpose(a1)).dot(sigma2)

    grad1 = delta1/len(data) + (lamb/len(data))*np.insert(np.delete(theta1,0,0),0,np.zeros(len(theta1[0])),0)
    grad2 = delta2/len(data) + (lamb/len(data))*np.insert(np.delete(theta2,0,0),0,np.zeros(len(theta2[0])),0)

#update theta
    theta1 = theta1 - alpha*grad1
    theta2 = theta2 - alpha*grad2

What is causing this problem ? How can it be fixed ?

If a3 is 0 or 1, np.log(a3) or np.log(1-a3) is going to give an error as $\log(0)$ is undefined, and $\lim_{x -> 0} \log(x) = -\infty$


Support Vector Machine analysis - Python, how do I determing whether to use linear, square, or other types of SVM models? how do I determine whether to use linear, square, or other types of SVM models?

under which criteria should an SVM model be applied in the first place?

By Default, SVM in Sklearn uses RBF Kernel. You have to try out all the 3 kernels, with different Gamma and C.

SVM treats outliers better and add a penalty on every outlier it detects.

You should understand your data well or vizualize well to know what kernel fits well.
SVM literally adds a new dimension to the data to create a hyperplane.


How do a real brain actually learn? In the biology class we've learned, that the neurons are connected. And if two or more neurons interact with each other often, then the connection gets stronger and stronger, and new connections may form. But if a connection is unused for a while, then it gets weaker and weaker, and it may lost forever. That is how the human brain/memory learns and forgets something.

I am a bit confused now. Because I've started to learn about artificial neurons, and the method of learning is very different. We analyze, how efficient the network was, and before restarting it, we update the weights a bit, based on the relative difference between the returned and the expected value.

So my question is: How is a real-world brain capable of learning? And does it have a default configuration?

Artificial Neural Networks are algorithms loosely based in how a brain functions so they shouldn't be treated as the equivalent of brain learning.

The science that studies all the details of how neurons and brains learn is Computational Neuroscience and the question you ask is pretty much an open question but it has many great hypothesis.

The method you described about how artificial neural networks update their "weights" is called backpropagation and some authors included O'Reilly, Randall; Munakata, Yuko (2000) in Computational Explorations in Cognitive Neuroscience: Understanding the Mind by Simulating the Brain argue that backpropagation is actually the algorithm that drives learning in the brain. In later publications they propose architectures that include the Hippocampus region to re-create short term memory.

However in recent years, with the advent of mainstream deep learning new ideas have arrived along with old ideas with new approaches. The ones that look to replicate the brain and its function or at least approximate it as much as we currently can are:

Spiking neural networks
Hierarchical temporal memory / Numenta (They also have resources and solutions for data science problems like anomaly detection and classification)
Other theories in Computational Intelligence

Spiking neural networks are computationally expensive and hard to tune, their performance isn't close to what deep learning can do, however natural brains use some form of Spiking neural networks which tells you about the potential of these kind of models.

Another interesting mention is Neural Turing Machines. It's an interesting approach on how certain brain inspired processes can be used to compute diverse problems.


Slight confusion on the learning process Hi guys I have a slight confusion on the learning process of neural networks.
When the input layer receives inputs, goes through the hidden layers and then into the output layer. How does the neural network know that the outputs at the output layer are incorrect?
When the error is calculated at the output layer, it needs the predicted output which is fine but how is the actual output found?
Thanks

Well, you make a very good point, this is the most painful part of machine learning, we have to label data to make them usable to train models.By labelling the data, we mean to go through the dataset and say what the output of each input is. It means that to each input we attribute a true output.
The input of the learning function in sk-learn for example are always (almost) model.train(X, Y). with X the inputs and Y the corresponding expected outputs.
Labelling is a tedious and long task that must be hand made most of the time. Some website even pay people to label data (especially for image classification or segmentation), such as amazon mechanical turk.


what is the difference between "fully developed decision trees" and "shallow decision trees"? As reading Ensemble methods on scikit-learn docs, it says that

  bagging methods work best with strong and complex models (e.g., fully
  developed decision trees), in contrast with boosting methods which
  usually work best with weak models (e.g., shallow decision trees).

But search on google it always return information about Decision Tree.

I'd like to know the detail of the two trees mentioned in the doc, what's the fully developed and shallow meanings.

Update:

About why bagging work best with fully developed and why boosting work best with shallow.

First, I think complex models (e.g., fully developed decision trees) means such a data set has a complex format be called as fully developed decision trees.

After I read above quote over 20 times and rapaio's answer, I think my poor English lead me to the wrong road(misunderstand). I also mistake shallow as shadow , which make me confusing a long time....Now I understand the meanings of fully developed and shallow .

I think the quote is saying bagging work best with a models(already trained) which algorithm is complex. And boosting only need simple model.
Both bagging and boosting need  many estimators, as n_estimators=100 in scikit-learn examples.

If n_estimators=100 :

bagging need 100 fully developed decision trees estimators(models)
boosting need 100 shallow decision trees estimators(models)

Does my thoughts is right?
Hope my update can help non-native speakers.
e.g. means for example, so there are other models can use for bagging and boosting. How about changing the model to svm or something else? Or both of they need a tree base model?

[Later edit - Rephrase everything]

Types of trees

A shallow tree is a small tree (most of the cases it has a small depth). A full grown tree is a big tree (most of the cases it has a large depth).

Suppose you have a training set of data which looks like a non-linear structure.

Bias variance decomposition as a way to see the learning error

Considering bias variance decomposition we know that the learning error has 3 components:

$$Err = \text{Bias}^2 + \text{Var} + \epsilon$$

Bias is the error produced by the fit model when it is not capable to represent the true function; it is in general associated with underfitting.

Var is the error produced by the fit model due to sampled data, it describes how unstable a model is if the training data changes; it is in general associated with overfitting.

$\epsilon$ is the irreducible error which envelops the true function; this can't be learned

Considering our shallow tree we can say that the model has low variance since changing the sample does not change too much the model. It needs too many changed data points to be considered unstable. At the same time we can say that has a high bias, since it really can't represent the sine function which is the true model. We can say also that it has a low complexity. It can be described by 3 constants and 3 regions.

Consequently, the full grown tree has low bias. It is very complex since it can be described only using many regions and many constants on those regions. This is why it has low bias. The complexity of the model impacts also variance which is high. In some regions at least, a single point sampled differently can change the shape of the fitted tree.

As a general rule of thumb when a model has low bias it has also high variance and when it has low variance it has high bias. This is not true always, but it happens very often. And intuitively is a correct idea. The reason is that when you are getting close to the points in the sample you learn the patterns but also learn the errors from sample, when you are far away from sample you are instead very stable since you do not incorporate errors from sample.

How can we build ensembles using those kind of trees?

Bagging

The statistical model behind bagging is based on bootstrapping, which is a statistical procedure to evaluate the error of a statistic. Assuming you have a sample and you want to evaluate the error of a statistical estimation, bootstrapping procedure allows you to approximate the distribution of the estimator. But trees are only a simple function of sample "split the space into regions and predict with the average, a statistic". Thus if one builds multiple trees from bootstrap samples and averages the trees can be considered i.i.d. and the same principle works to reduce variance. Because of that bagging allows one to reduce the variance without affecting too much the bias. Why it needs full depth trees? Well it is a simple reason, perhaps not so obvious at first sight. It need classifiers with high variance in order to reduce it. This procedure does not affect the bias. If the underlying models have low variance high bias, the bagging will reduce slightly the already small variance and produce nothing for bias.

Boosting

How does boosting? Many compare boosting with model averaging, but the comparison is flawed. The idea of boosting is that the ensemble is an iterative procedure. It is true that the final rule for classification looks like a weighted average of some weak models, but the point is that those models were built iteratively. Has nothing to do with bagging and how it works. Any $k$ tree is build using information learned from all previous $k-1$ trees. So we have initially a weak classifier which we fit to data, where all the points have the same importance. This importance can be changed by weights like in adaboost or by residuals like in gradient boosting, it really does not matter. The next weak classifier will not treat in the same way all the points, but those previously classified correctly has smaller importance than those classifier incorrectly. The consequence is that the model enriches it's complexity, it's ability to reproduce more complex surfaces. This is translated in the fact that it reduces the bias, since it can go closer to data. A similar intuition is behind: if the classifier has already a low bias, what will happen when I boost it? Probably a much unbearable overfit, that's all.

Which one is better?

There is no clear winner. It depends too much on data set and on other parameters. For example bagging can't hurt. It is possible to be useless ut usually does not hurt performance. Boosting can lead to overfit. That is because you can go eventually too close to data.

The is a lot of literature which says that when the irreducible error is high, the bagging is much better and boosting does not progress too much.

Can we decrease both variance and bias?

Pure bootstrap and bagging approaches serves a single purpose. Either reduce variance either reduce bias. However modern implementations changed various things in how those approaches works. Sampling can be used in boosting and it seems to work towards reducing also the variance. There are bagging procedures which takes some ideas prom boosting, for example iterative bagging (or adaptive bagging) published by Breiman. So, the answer is yes, is possible.

Can we use other learners other than trees?

Of course. Often you will see boosting to use this approach. I read some papers on bagging svm or another learners also. I tried myself to bag some svms but without much success. The trees are the preferred way, however, for a very simple reason, they are simple to build, simple to adapt and simple to control their effect. My personal opinion is that it is not everything said regarding ensembles of trees.

PS: a last note on the number of weak classifiers: this depends entirely on the complexity of the data set mixed with the complexity of the learner. There is no recipe. Often 20 of them are enough to get most of the information and additional ones are only for tiny tuning.


Finding roots of a non linear expression when multiplied with a linear expression Here is a simple polynomial equation:

b^2 + 2b + 1 = 0

I could easily solve this as:

import numpy as np
from scipy.optimize import fsolve

eq = lambda b : np.power(b,2) + 2*b + 1
fsolve(eq, np.linspace(0,1,2))

Similarly I could solve any equation, that has finite number of terms. But how do I solve an equation with infinite number of terms which is given as :

The above equation could be written as :

(1 - l) * (5.5 + 4.0*l + 4*l^2 + 6*l^3 + 5*l^4 + 5*l^5 + 5*l^6 + 5*l^7 + 5*l^8 + 5*l^9 + 5*l^10   ) = 5

when n goes from 1 to 10. But I want to solve this for sufficiently large value of n such that LHS ~= RHS.

I know the values of LHS and G1 -> Ginf but cannot understand how could I compute the value of lambda here.

I tried looking at numpy polynomial functions but could not find a function that is relevant here.

I don't think this equation can be solved in general. Since it is an infinitely long polynomial, it cannot be evaluated. It can only be done if the constants G_t:t+n followed a very specific pattern, so that the series becomes equivalent to a known function that we can evaluate. For example:

If $G_{t:t+n}$ were equal to [ 1, 1/2!, 1/3!, 1/4!, ...] (where 4! is the factorial of 4), then the sum would be equal to $e^\lambda$
If $G_{t:t+n}$ were equal to [1, 0, -1/3!, 0, 1/5!, ...] then the sum would be equal to $\sin(\lambda)$

But in general, if G were any random set of constants, the function cannot be evaluated.


Recombining rare values I have a column called item_colour which describes the colour of products in my dataset.

There are 85 colours and some of the colours are unique or they represent a small part of the data ( I have 100,000 observations). For example, there is only one "almond" colour or only two "sky blue". I want to recombine rare values and put them together into a group called "other colour".

How can I find the threshold to recombine them? For example, combine the colours together which have values less 50 or so.

P.S.I am working with R

This is going to be a situation where there will be no fixed rule. One important factor is how meaningful colour differences are to the other parts of your problem. If colour has low correlation/impact in a supervised learning/prediction scenario for example, and the dataset is noisy, then you will want to merge more colours (at a higher fraction of total number) to reduce sampling bias effects that might otherwise assign importance to the colour and increase error rates in test and production.

The safest approach is to treat the colour combination threshold as a hyper-parameter to the model building process, and test to see what differences it makes. If there is little or no impact to model effectiveness, then a higher threshold could be useful purely to reduce number of parameters in the model - decreasing resources used to train and run it.

If that seems time-consuming, then picking something by feel (e.g. your idea of picking count less than 50 in the dataset) is not usually too bad, at least to start with. You can go back and re-evaluate your choice if you have problems with the model.

One other possibility for feature engineering is to use the rareness of the specific colour as an additional feature. So in addition to categories for the popular colours and an "other colour" category, add a real value "colour frequency" = the observed ratio of that colour in the training set. Whether or not this is useful will depend on the problem, but it may help address some of the lost information when merging categories with a wider range of rareness values, assuming that unusual colours indicate anything at all (they may not).


P-value mining on large number of combinations of variables I really don't know any machine learning, but have a problem that seems like one where I should use some ML algorithm.

I am analyzing a medical study with one age-related condition, age, a treatment, gender, and the abundance of two particular gut bacteria. Many researchers in the field also like to look at the ratio of these two bacteria.

Playing around with some regressions with one, two, or three explanatory variables, I found some unexpected combinations with very good p-values. For instance, controlling for age, bacteria-A seems to be strongly associated with the condition regardless of treatment. The other bacteria seems to be strongly associated with the treatment regardless of age. I would have had no way to expect this to be the case.

I feel there might be value in searching for more unexpected associations. I can make a list of all one, two, and three combination explanatory variables and perform regression of my six variables against these combinations and basically sort by p-value. But, 1) this sort of p-value mining is generally frowned upon, and 2) there are a bazillion possible regressions.

Seems like there is probably some sort of ML algorithm that would hunt down the unexpected associations in an objective and systematic way.

What would that be?

You need to investigate multiple hypothesis correction methods, like Bonferroni correction or Benjamini-Hochberg false discovery rate. The problem with this sort of analysis is that your associations are unexpected, so you don't have any a priori hypothesis. All you can do is test every combination, and then see what is statistically significant after accounting for all the tests.

As you do more tests, it becomes increasingly likely that you'll find an association just by chance alone. To combat this, multiple hypothesis correction methods make the p-value threshold for significance more conservative. If you do only 1 test, a p-value of 0.05 may indicate a significant result. If you do a million tests, however, many of them will have a p-value of 0.05 by chance, so you need to be more conservative. This practice is not generally frowned upon, since it appropriately takes into account the analysis methods. You can easily be dishonest with this approach, however, by doing many tests and only reporting the significant ones without correction.

Obligatory xkcd: https://xkcd.com/882/


While it is not very clear to me what specific relationships between your data you want to display, I can give some general advice. I think that this time series visualization calls for so-called diverging color scheme (as opposed to sequential, categorical and other ones). ColorBrewer is nice online tool for selecting an appropriate color scheme and other parameters (note that ColorBrewer's color schemes have built-in support in major data analysis and visualization software, such as R, Python, d3.js, plot.ly and others).

If you would use R environment, it would be quite easy to produce the time series trend line chart that you want by using ggplot2 package and its scale_color_gradient2() function:

... + scale_color_gradient2(midpoint=midValue, low="blue", mid="white", high="orange" )

A quick glance at your profile suggested me that you would prefer a JavaScript solution. In that case, I would advise to use d3.js library or one of multiple other JavaScript visualization libraries or packages. For more details, check this relevant answer of mine.


Forecasting sales of next year using sales of past years? I have a dataset of sales of a company for different products throughout 3 years. I have to forecast the sales for each of these products for next year. A sample of the dataset is:

Here I have to forecast sales for year 4 from the sales record of each product over 3 years.

N.A is present where the particular product was not sold in that year.

Questions

1.Which algorithm should I use to forecast the sales for year 4? I have heard about ARIMA and xgboost being used for time series data. Can you please help me with that?

2.Now I am being given a new product which was not sold before. I am being asked to predict its sales for year 4, using the data for sales of other products:

What should be my approach for this problem?

Please help me out here. Thank you for your time and support in advance. I am sorry if my documentation or question is poor.

If you have daily data you could create a dummy time calendar, i.e. you create a dummy variable for each day of the week and include your company's promotions for each product, Christmas, Easter, public holidays.
Then use autoarima() or nnetar() (or combine them) to forecast the time horizon you want. This link is a good example.

Another possible way to do that is to use the method of following paper: "Hsiang-Fu Yu, Nikhil Rao, and Inderjit S. Dhillon. 2015. Temporal Regularized
Matrix Factorization. CoRR abs/1509.08333 (2015)" This paper can help you with with forecasting the new products too.

Or for forecasting new products you could find a comparable products from your historical data then use those comparables' sales data to forecast performance of the new ones.

If your data really looks like above I think you could use Hsiang-fu's paper. this is a good approach.

For autoregressive forecasting you could read from Rob Hyndman's blog.


Remove columns with a certain number of consecutive zeros I'm using pandas and I am dealing with time series of sales.

What I would like to do is to remove the columns where a certain number of consecutive zeros appear, since forecasting for sparse series or repeated zero values tend to be unreliable. My dataframe has a bit more than 4400 time series for a time span of 5 years.

I have tried applying this strategy from a previous SO question but I can't drop the columns fullfilling the condition.

Below is an example where the threshold would be 12. Any column having 12 or more consecutive zeros would be discarded.

input :

      A   B   C

0     1   1   8
1     2   0  14
2     3   0  20
3     0   0  15
4     0   0  23
5     0   0  25
6     0   0  22
7     0   0  18
8     0   0  16
9     0   0  14
10    0   0   8
11    1   0  10
12   18   0   7
13   34   0   4
14  110   1  14
15   11  30   2
16    0  24   8
17    0  11   7
18    1  22  11
19    0  33   3
20    0  90  12
21   12  32  19
22   11  90  17
23   77  13   2

desired output:

      A   C

0     1   8
1     2  14
2     3  20
3     0  15
4     0  23
5     0  25
6     0  22
7     0  18
8     0  16
9     0  14
10    0   8
11    1  10
12   18   7
13   34   4
14  110  14
15   11   2
16    0   8
17    0   7
18    1  11
19    0   3
20    0  12
21   12  19
22   11  17
23   77   2

Thanks.

Short answer:

# Max number of zeros in a row
threshold = 12

# 1. transform the column to boolean is_zero
# 2. calculate the cumulative sum to get the number of cumulative 0
# 3. Check how much of each count you get and remove 0 counts
# 4. Get the maximum number of cumulative zeros
# 6. Drop the columns with more than your threshold

to_remove = []
for col in df.columns:
    count_zeros = (df[col] == 0).cumsum().value_counts().drop(0)
    if len(count_zeros) and count_zeros.idxmax() > threshold:
        to_remove.append(col)

df = df.drop(to_remove, axis=1)

You're welcome


How to test for Overall Significance? I have two models: one a simple linear regression and the other a multi linear regression.

The strongest relationship is between the two variables in the simple linear regression. The multi linear regression includes two extra variables that can be connected with the dependent, however, statistically they are insignificant (P > 0).

How do I interpret whether the best model to use is the simple linear regression or the multi-linear regression?

Here are some values from the regression:

Simple Linear : F(1, 77) = 21.07 , Prob > F = 0.0000 , Rsquared = 0.2148, Adj Rsquared = 0.2046

Multi Linear: F(3, 75) = 7.29 , Prob > F = 0.0002, Rsquared = 0.2258, Adj Rsquared = 0.1948

Assuming that the "best" model is the model that makes the most accurate out-of-training sample predictions, there are two types of measures that you can use. R squared will only increase as you add more variables hence it is not a suitable measure.

There are two common approaches:

(1) Indirectly estimate test error by making an adjustment to the training error (e.g. Adjusted Rsquared / BIC/ AIC). In your case, the simple linear regression performed slightly better in terms of Adj Rsquared.

(2) Directly estimate the test error using either a validation set or cross-validation.


Different number of images in classes I am working on a deep learning CNN project. The dataset contains more than 500 classes and the classes have different numbers of items (images). For example, some of the classes have 5 images and some of the classes have 10 and some of the classes have 20  images and some of the classes have more then 20 images.
Can I use this dataset to create the CNN model?
Should the number of the images in each class be the same number?
Note: I will use VGG to train the model.

Frankly, even 50 images will not be sufficient if you are going to create and use a CNN model. If you think you want more images for you model training, then go for data augmentation. It is a process of transforming an image by a small amount (be it height, width, rotation etc or any combination of these). In this way, an image and its augmented image will differ slightly. You can find relevant article here-

https://medium.com/nanonets/how-to-use-deep-learning-when-you-have-limited-data-part-2-data-augmentation-c26971dc8ced

To answer the part that should there be same number of images in each class, there should be approximately same number. This problem is a general problem while working on classification task and there are several ways to deal with it, including simulating the data (augmentation).

I would suggest that first create a separate test set, then on the remaining train set, use data augmentation and finally create the model.

EDIT

Using a pretrained convnet is also an option, as stated in a deep learning book-

  A common and highly effective approach to deep learning on small image datasets is to use a pretrained network. A pretrained network is a saved network that was previously trained on a large dataset, typically on a large-scale image-classification task. If this original dataset is large enough and general enough, then the spatial hierarchy of features learned by the pretrained network can effectively act as a generic model of the visual world, and hence its features can prove useful for many different computer vision problems, even though these new problems may involve completely different classes than those of the original task. For instance, you might train a network on ImageNet (where classes are mostly animals and everyday objects) and then repurpose this trained network for something as remote as identifying furniture items in images.


Dataset with features as SNOWMED codes and outputs as names of medications prescribed.What are some ML algorithms that suit this case? So, i am constructing a dataset (from doctor prescription notes and diagnosis) where

X (input features) represent the various snowmed codes for illnesses/surgeries performed/chronic conditions etc.

Y is a set of pills prescribed for X.

Presently I do not have this data in a matrix format. For ex. i have the snowmed codes for an illness that reads as follows: "Diabetes, Hypertension, Cardiac arrest, Left MCA stroke". Y is the set of pills that the doctor noted down for the above.

As one can note, the above data is not in a matrix format.
There are many snowmed codes and various different pills overall. I'd like to apply ML algorithms to predict the pills for a given set of snowmed codes about the illness.

How to organize this data in a format convenient for ML? I am open to other analytics approaches as well. Any discussion in this matter will be helpful.

In that case, Multi Class Classification will be the best approach. You can use Neural Network approach or K-Nearest Neighbours for this problem. Your DataFrame is going to be really sparse if you take a numeric approach.

From my point of view, the best approach is to build a vector model.

After you arrange your data this way, it's easier to build a topic model using NLP techniques. It can also tell you which illnesses are most similar and which codes are similar.

This is what I could think of. These sort of questions are very open ended and can often lead to creative approaches. I am sure there'll be something better if you think hard enough.


How to learn certain Maths to understand machine Learning papers? I have done the deeplearning.ai course on deep learning. But I cannot Understand equations like

minGmaxDV(D,G)=Ex∼pdata(x)[logD(x)]+Ez∼pz(z)[log(1−D(G(z)))]

What kind of Maths am I am supposed to learn? I know Calculus basic Multivariable Calculus and Linear Algebra. So should I learn Probability (Advance probability maybe)?

As Elias mentioned, the expectations are related to random variables and you would be good to go if you know about conditional probability, multivariate probability, joint and marginal distributions. I would suggest you take a course that has a syllabus on the lines of https://secure.oregonstate.edu/ap/cps/documents/view/134169.


Accuracy mismatch in tensorflow model I am working on image classification project. I have trained model on car dataset. So it gives good accuracy but when I predict BMW car with my model it gives below results.
BMW :- 98%
Audi :- 91%
Why Audi labels shows 91% it should be low. How to solve this issue? I am using resnet 50 pre trained model.

I found a solution that we should use the softmax activation function in the last layer. previously I used the sigmoid activation function


What does the "dual" parameter in sklearn.svm.LinearSVC and sklearn.svm.LinearSVR do? While I am more or less familiar with the idea of the SVM, I do not understand the meaning of the dual parameter, which is described in the documentation as:

dual : bool, (default=True)

    Select the algorithm to either solve the dual or primal optimization problem.
    Prefer dual=False when n_samples > n_features.

Can someone explain it, preferably in ELI5 style?

ELI5-style, here we go.

Kidwarts is a world-famous magical kindergarten for talented 5-year old wizards. Most of the day the kids spend playing in a large hall, which is full of fluffy puppies, magical seesaws, jumping chairs, tiny flying carpets (remote-controlled using kid-approved magic-wands), and all the other cool toys one could even imagine. Each kid has their favourite spot in the hall, where they would typically spend most of the day, petting their favourite puppy or jumping on their favourite jumping chair.

One sunny Tuesday morning miss Fluffypuff (the kindergarten's supervisor) brought a huge magic mirror to the center of the playhall. The mirror has two faces. If one looks into the mirror from one side, they would see meadows full of beautiful rainbow ponies hopping around. From the other face the mirror shows heroic fables of brave young wizards riding dragons and fighting evil witches. Obviously, it would be best if the mirror could be oriented in the room in such a way, that the "pony" face would be turned towards where most of the girls play, while the "dragon" one could be observed mostly by the boys. Everyone knows that 5 year old girls love ponies while boys adore dragons!

It is not an easy task, however. Although many boys indeed mostly hang around the corner of the hall with the flying carpets while a lot of girls prefer the corner with puppies, a bunch of kids are spread all around and could become unhappy and start crying using magical tears if the mirror is positioned unfavourably to them. Miss Fluffypuff has been strugging to find the best spot for the mirror the whole day.

Here is how she did it. First she hovered the mirror right in the middle of the room, turned in a somewhat random direction. She then waited and observed who of the kids started crying because they did not like what the mirror showed them. Based on that information, she nudged the mirror slightly towards the most unhappy kids, so that they would get to see more of their preferred side (or less of the "bad" side). After the whole day of such nudging the mirror seems to have converged to a more or less optimal orientation, with just a few kids left unhappy. To them miss Fluffypuff gave pieces delicious magical candy, so they were still quite happy after all!

In the evening Miss Fluffypuff was excited to tell the mirror story to her friend, Mister Puppypaw. "Ah, I would have done things differently" said Mister Puppypaw, twirling his gorgeous mustache with his fingers.
My method would be the following: I would first pick two kids in the room, and ask them to decide on their favourite orientation for the mirror. Then I would see who of the other kids complains about the mirror, add them to the team and have the new team decide together again. If at this point there were still unhappy kids, I would continue adding them to the "council" and let them find a compromise which suited everyone. We would thus end up with the situation, where the configuration of the mirror is decided to satisfy precisely the group of kids who care about it, while those without opinion would not have to be bothered. This would be super fair, wouldn't it?

Miss Fluffypuff agreed, but noted that her method also seemed totally fair in the end. They kissed and lived happily ever after. The End.

In this story, Miss Fluffypuff was solving the primal problem. That is, she was searching for the orientation $\mathbf{w}$ of the mirror directly. Mister Puppypaw, instead, proceeded in a roundabout, dual manner. He assumed that we can always compute the orientation $\mathbf{w}$ by having each kid vote on their preference, and aggregating these votes:

$$\mathbf{w} = \sum_{i} \alpha_i \mathbf{x}_i, $$

where $\mathbf{x}_i$ is the favourite position of kid $i$ and $\alpha_i$ is the vote (which will be $0$ if the kid does not care, positive if the kid prefers one face of the mirror, and negative otherwise). Consequently, the problem of Mister Puppypaw is not in finding the best orientation $\mathbf{w}$ directly, but rather in figuring out the vector of "votes" $\mathbf{\alpha}$ (from which the orientation $\mathbf{w}$ can be derived in a straightforward manner). YOu can also say that his main problem would be to find a subset of kids whose vote would be nonzero - these would be the "support vectors".

When $\mathbf{w}$ is just a two-dimensional vector and the number of kids is large, as is in the original story, the method of miss Fluffypuff is probably the best. Keeping track of a two-dimensional orientation of the mirror is simpler than managing a list of hundred or so "votes" or figuring out a proper subset of "kids who care".

If there were only one or two kids, however, it would instead be faster to just ask them rather than play this "mirror-nudging" game. Similarly, if the mirror had a million different knobs which needed to be configured, it would be easier to let even a hundred kids simply vote on their preferences (assuming each kid has a clear idea what million-knob configuration fits him best), rather than carefully tune these knobs in tiny increments, checking who of the kids started crying after every change.

In the end, either of the two methods would produce the same result, though.


I used Stargazer in Python. How to convert output to LaTex? Hello I typically use R but is there a way how to use stargazer in python

For example when doing an arima model

Model = SARIMAX(data, order = (1, 0, 0), trend = “c”
model_results = Model.fit()
Print(model_results.summary())

I get output table - is there a way how to convert it to latex

In order to convert the statsmodels (single model) summary to latex, you will use summary.as_latex(), as in model_results.summary().as_latex().

If instead you want stargazed-type tables (regrouping multiple models), you will want to use the corresponding python package: https://pypi.org/project/stargazer


Philosophical question on redundancy Suppose I implement a supervised learning version of LSTM similar to this. Namely, I have these univariate time series data:

t Y
1 101
2 105
3 108
4 104
5 110
6 112
7 119
8 111
9 113
10 115

So Y at time 1 is 101, Y at time 2 is 105, etc.

The goal is to predict Y at time t+1 given Y at time t and t-1. That is, I can rearrange this data set to look like supervised learning:

Y_(t-1) Y_t Y_(t+1)
101 105 108
105 108 104
108 104 110
104 110 112
110 112 119
112 119 111
119 111 113
111 113 115

Notice here that each Y value from t=3 to t=10 (max) gets predicted only once.

Now say I partitioned my data set into overlapping minibatches of size 5. We'll consider only the first two minibatches for simplicity:

t Y
1 101
2 105
3 108
4 104
5 110

2 105
3 108
4 104
5 110
6 112

I do the same trick of converting this to a supervised learning problem for each minibatch:

Y_(t-1) Y_t Y_(t+1)
101 105 108
105 108 104
108 104 110

Y_(t-1) Y_t Y_(t+1)
105 108 104
108 104 110
104 110 112

Notice: now two of the predictions from the first minibatch are repeated in the second minibatch.

I know that if I had wanted to have only one prediction per Y value, then I should have converted the entire dataset to supervised first and then created minibatches.

Questions:

Is this alternative approach incorrect? If so, why? If not, should I average the predictions, or is there another accepted way of combining the predictions from several time steps like this?

EDIT: I should mention that the learned parameters of the model are different for each position, even though the data points are the same. For example, the parameters that would be used to predict Y_(t+1) = 104 in minibatch 1 are different from the parameters that would be used to predict Y_(t+1) = 104 in minibatch 2.

EDIT 2: I'm thinking this approach isn't incorrect, just perhaps a little odd. It would be similar, in a way, to a bidirectional LSTM (reading the input backward and forward to increase the exposure of the network to the end points - i.e. boost the signal). This also boosts the signal, but perhaps increases the bias by quite a bit. Thoughts?

Repeating samples will probably only slow down the learning process. As long as the mini batches are small this will probably not have a big effect.
However, if batches and the number of repetitions become large gradient based learning can run into difficulties.

If you don't shuffle the batches, the repeated samples will prevent the model from taking in new information and may nudge it to "overfit" on the current samples. Since at every batch you only get one new sample (while all the others are repeated), the learning will either slow down or become stale and stuck at an unfavorable location.

Since your approach only has a two-dimensional input, this is probably not a major concern since the loss function will probably not be as complex as in other neural network applications.

Overall, in your case, you probably will not see a big impact on performance except for a longer learning time.

You do not need to be concerned with combining predictions for repeated samples, since you can make predictions using the final parameters after the learning is completed.


Rescaling each image Individually with keras I am a beginner working on a simple CNN to classify X-ray detector images. Due to source intensity, all images have different max values. I want to use ImageDataGenerator to rescale those images to be in range [0,1], but cant't find a way to do that for each individual image. As far as I understand it, people usually just divide by 255 because that is the RGB max, but in my case the max could be anything between 1 and serveral million. Does anyone have an idea how to do this within ImageDataGenerator? Thank you for your time!

Edit:
I ran over my images and normalized all of them prior to feeding them to the generator. Inspecting them then showed that the generator had scaled them back to rgb values (I assume due to color_mode). I then used rescale with the standard 1./255 and got the expectd max value of 1. However, after adding back samplewise_std_normalization, I again am getting values larger than 1. I understand why this is happening, but I'm not sure which "rule" is more important to follow: normalize to [0,1] or use samplewise_std_normalization

Normalize to $[0, 1]$ is what I'd do to ensure that the model gets expected (sanitized) inputs.

Using the samplewise_std_normalization is something that I'd do inside the model to highlight features.
E.g. a white pixel is more important in a mostly black image than in a noisy image.