The difference between `Dense` and `TimeDistributedDense` of `Keras` I am still confused about the difference between Dense and TimeDistributedDense of Keras even though there are already some similar questions asked here and here. People are discussing a lot but no common-agreed conclusions.

And even though, here, @fchollet stated that:

  TimeDistributedDense applies a same Dense (fully-connected) operation to every timestep of a 3D tensor.

I still need detailed illustration about what exactly the difference between them.

Let's say you have time-series data with $N$ rows and $700$ columns which you want to feed to a SimpleRNN(200, return_sequence=True) layer in Keras. Before you feed that to the RNN, you need to reshape the previous data to a 3D tensor. So it becomes a $N \times 700 \times 1$.

$ $

The image is taken from https://colah.github.io/posts/2015-08-Understanding-LSTMs

$ $

In RNN, your columns (the "700 columns") is the timesteps of RNN. Your data is processed from $t=1 \  to \ 700$. After feeding the data to the RNN, now it have 700 outputs which are $h_1$ to $h_{700}$, not $h_1$ to $h_{200}$. Remember that now the shape of your data is $N \times 700 \times 200$ which is samples (the rows) x timesteps (the columns) x channels.

And then, when you apply a TimeDistributedDense, you're applying a Dense layer on each timestep, which means you're applying a Dense layer on each $h_1$, $h_2$,...,$h_t$ respectively. Which means: actually you're  applying the fully-connected operation on each of its channels (the "200" one) respectively, from $h_1$ to $h_{700}$. The 1st "$1 \times 1 \times 200$" until the 700th "$1 \times 1 \times 200$".

Why are we doing this? Because you don't want to flatten the RNN output.

Why not flattening the RNN output? Because you want to keep each timestep values separate.

Why keep each timestep values separate? Because:

you're only want to interacting the values between its own timestep
you don't want to have a random interaction between different timesteps and channels.


Smoothen the Classification output I am working on an image classification tasks in which I have 4 classes (For example A, B, C, D). I used CNN model (transfer learning) to train the model and predict the video frames.

Ideally, there would not be any sudden transition from one class to another. Ideal Transition would be as given below:
A A A A A ..... A A A A B B B B B B ..... B B B B B B B D D D D D ..... D D D C C C C C C C ............. C C C

However when predicting using the trained model, I could see some frames being mis-classified. For example I given an video input of Class A which has around 30 frames. In that 30 frames 5 would be predicted as Class C or D.

How would I make the smooth transition from one class to another. There should be sufficient number of evidences so that I can make transition from one class to another.

As of now I found moving average technique which does similar kind of smoothening. I would like to know is there any other method which is more related to probability.

Kindly let me know if you need more details.

Thank you,
KK

An option could be using the output of the i-th image as an input for classifying the (i+1)-th image.


Do batch GD and stochastic GD give the same results? If a neural network is trained on a dataset of M samples for N epochs, do batch GD and SGD give the same result? Is SGD is faster because utilize the hardware better?
I am asking because I figured out that both (batch GD & SGD) give mathematically the same result, but I read SGD avoid local minima, how can this can be true if SGD & batch GD give the same result!?

They are not the same; the batch gradient descent averages the gradients of samples 1 through M. This is seen in the first equation, denoted by the summation over M elements and divided by M total elements (hence the average).
Stochastic gradient descent, as illustrated in the second equation, updates via a single, randomly selected instance.
The confusion I believe stems from thinking about loops over 1 through M. The batch gradient descent summation (loop from 1 to M) is performed per iteration, whereas the stochastic gradient descent loop is performed over all epochs.
Thus the intermediate weight updates will be different, and these results compound. So even if both methods process every instance over the entire training course, the act of updating only after a single instance will inherently cause a different trajectory than if one were to average all the gradients per iteration.
SGD can be faster simply because it only processes a single instance per weight update. In fact, purely from a hardware efficiency perspective, SGD is suboptimal due to its intrinsic serialized nature over the course of training. Batch gradient descent can easily be parallelized and take advantage of GPUs by processing multiple instances simultaneously.
Because SGD is, well, stochastic, the perturbations this causes in the surrogate model development can help alleviate bias in the training set. This is why SGD can sometimes help avoid local minima, albeit using a rather crude optimization strategy.
Edit to answer the comment asking about the relationship between training set bias and local minima:
Picture an example where a model has 2 parameters that define the xy plane and the corresponding error is plotted as the z component. This defines a function mapping R² to R. This function is only an approximation of the real-world function because you are not testing on every possible training instance. A biased training set poorly approximates the real-world function. Thus its local minima are different from the real-world local minima. When using batch GD, you will likely target the nearest local minima, without really any change of target over the course of training. This is good if your training set is relatively unbiased, but otherwise not ideal as you would be over fitting to a poor approximation. SGD can improve generalization by forcing a redirect of target in each iteration. Every iteration, if you sample a different function out of the M training samples, then you will likely change target minima multiple times. If your training set is relatively biased, then this can help not overfit to a poor approximation. If your training was a perfect approximation, then SGD would be pointless. Some may argue, given a perfect approximation, that the random sampling helps to avoid getting trapped in the starting basin, but at that point it's just luck.


What activation function should I use for a specific regression problem? Which is better for regression problems create a neural net with tanh/sigmoid and exp(like) activations or ReLU and linear? Standard is to use ReLU but it's brute force solution that requires certain net size and I would like to avoid creating a very big net, also sigmoid is much more prefered but in my case regression will output values from range (0, 1e7)... maybe also sigmoid net with linear head will work? I am curious about your take on the subject.

The issue with sigmoid and tanh activations is that their gradients saturate for extreme values of their arguments. This may occur if you do not normalize your inputs. If the learned weights of the unit are such that the gradient of its activation is close to zero, it will take longer for any updates to be reflected in the unit's weights. A final layer with no non-linearity will help you scale up your hidden layers' outputs.

In the end, the performance is application specific. You should try out both kinds of activations on a subset of your data and see which performs better.


How to take advantage of variables whose values are available in the past but not in the future? Example: weather data. You know the location data, but you don't know the previous days'/weeks' temperatures and other weather conditions. How can you exploit these variables in your past data when you build a predictive model that attempts to forecast pretty far out into the future?

My generic answer to the title is to use the extra data for regularization in representation learning; a transformation of your features into a space conducive to your main task: regression (prediction, forecasting). Here's a survey [PDF].

For your example, you could build a model that takes the delay of the target time from the present as an input, so you can predict arbitrarily far into the future, though it probably would not predict as well as a simple regressor that has a fixed horizon since it is trying to learn a more complex function.


Nominal categorical variable with two levels: Label Encoding or One Hot encoding? For a nominal categorical variable that has two levels, e.g. Gender (levels = Male,Female), is it feasible to use label encoding instead of One Hot encoding ? If it is, are there any implications of using one encoding method over the other, for such a categorical variable ?

If you use one-hot here, you're just adding an unnecessary variable that is perfectly correlated with another variable in your model. Rather than thinking of it as a label encoding of gender where "0=male, 1=female", think of it as a binary flag for is_female, where "0=false, 1=true".


Ordinal Integer variable vs Continuous Integer variable I am working on titanic dataset. I have one feature Pclass which I understand is an ordinal variable having values 1,2 and 3. I have created a new feature cabin_int from feature Cabin, which is essentially the number of cabins alloted to a passenger. So, it has values like 0,1,2,3 and 4. Now this new feature is not ordinal, it is just a continuous variable taking only integer values.

My question is how does Machine Learning algorithm understands the difference between the two, if I pass these two columns (Pclass and cabin_int) as they are during training of my model?

If some more feature engineering needs to be done, please tell.

There is a rule called there is no free launch. It means that there isn't a learning algorithm that solves all the problems. You as a machine learning practitioner should decide when and how to use which algorithm. Suppose that you want to recognize faces. This problem is a learning problem which if you increase the number of training data, you will get better results. In these cases neural nets and deep nets are highly recommended. In this case it is not logical to use non-linear SVM because it will be so costly and you may not even get good answers. the reason is that deep nets cares about local patterns but SVM considers all the input pattern simultaneously. Actually in your case, I guess your data is categorical. For categorical data, people often use decision trees.

To illustrate an example, once I decided to train a simple MLP to distinguish whether an input pattern is in correct position, to solve 8-queen problem. I solve the game using Genetic algorithm and made data for training the net. The data I brought to net was categorical in some extant. I used it and the net was so good for the trained data, but input features similar to training data which were a bit different had bad recall rate. I trained a decision tree, I get so much better result.

Which algorithm depends on your task and your input features.


Why is Reward Engineering considered "bad practice" in RL? Reward engineering is an important part of supervised learning:

  Coming up with features is difficult, time-consuming, requires expert knowledge. "Applied machine learning" is basically feature engineering. — Andrew Ng

However tweaking the reward function by hand is generally frowned upon in RL. I'm not sure I understand why.

One argument is that we generally don't know, a priori, what will be the best solution to an RL problem. So by tweaking the reward function, we may bias the agent towards what we think is the best approach, while it is actually sub-optimal to solve the original problem. This is different from supervised learning, where we know from the start what we want to optimize.

Another argument would be that it's conceptually better to consider the problem as a black box, as the goal is to develop a solution as general as possible. However this argument could also be made for supervised learning!

Am I missing anything?

One argument is that we generally don't know, a priori, what will be the best solution to an RL problem. So by tweaking the reward function, we may bias the agent towards what we think is the best approach, while it is actually sub-optimal to solve the original problem.

This is the main issue.

  Am I missing anything?

I think you are missing the main point of setting a reward function. It should be a value that is maximised by an agent achieving the goals it has been set. Each time you change the reward function you may be explicitly setting new and different goals.

Changing a reward function should not be compared to feature engineering in supervised learning. Instead a change to the reward function is more similar to changing the objective function (e.g. from cross-entropy to least squares, or perhaps by weighting records differently) or selection metric (e.g. from accuracy to f1 score). Those kinds of changes may be valid, but have different motivations to feature engineering.

Tweaking a reward function is also called reward shaping, and sometimes it can have good results. It does come with the same risks as above, but if done carefully it can improve learning rate. For example if to achieve the main goal A, it is absolutely necessary to achieve B and C as interim steps, then it should be OK to reward achieving B and C - typically the things you have to worry about is whether it is possible for the agent to repeatedly achieve B or C via some loop through states - so you may need to add to the state vector whether it has achieved B or C enough times and only grant reward for first visit to B and C.

Feature engineering does still exist in RL, and is about how you represent state and action spaces. For instance, in a chase scenario where an agent needs to get close to a moving target, you should find that representing the state as polar co-ordinates of the target from the current agent's position is far easier for RL to learn optimal policy than if you represent state as cartesian co-ordinates of agent and target.


Feeding uint8 type to NN I’m facing some (RAM) memory issues to train a neural network.

I have an input array consisting of grayscale images encoded as the type numpy.uint8 (therefore the whole range 0-255 can be covered). When feeding the data to the network, I’m supposed to normalize the values into the 0-1 range, making them now floating points (numpy.float64 or 32). However, this conversion makes my data 8x(or 4x) bigger, which my RAM memory can’t handle.

Would there be a way I could overcome this issue?

Thanks !

After normalization, cast the numbers to float16 or bfloat16. Those are the smallest floats available in Keras.


How to perform data scaling/standardization on dataset containing grouped values? So I have a dataset containing the results of executing problem instances with different given solver strategies. Simplified example:

| Problem_instance | Problem_Size | Used_Solver | Cost |
| P1               |           50 | A           |   75 |
| P1               |           50 | B           |  125 |
| P1               |           50 | C           |  225 |
| P1               |           50 | D           |  100 |
| P2               |          150 | A           |  165 |
| P2               |          150 | B           |  360 |
| P2               |          150 | C           |  275 |
| P2               |          150 | D           |   45 |
| P3               |           25 | A           |   35 |
| P3               |           25 | B           |   65 |
| ...              |          ... | ...         |  ... |

I'm trying to use machine learning to predict the best performing Solver for a given problem instance. In data processing stage, I need to standardize or scale my data, but I'm not sure how to this best.

Firstly, I'm not sure which sklearn's Scaler to use (StandardScalar/ MinMaxScaler/..).

Secondly, I'm confused how to handle the different records for each instance. When I group the data first based on problem_instance and then use a MinMaxScaler, the record with Cost = 0 would be the best solution for this problem and Cost=1 the worst. But if I use the same strategy to scale the Problem_Size this would be equal to 0 everywhere. On the other hand if I use a global scaling, the information about which Solver is the best for each instance is lost.

Can someone help me how to handle the data preprocessing for this problem?

There's not one right answer to this question, because what scaler works best really depends on the data and the algorithm you use to make the prediction. You should try different scalers combined with different algorithms to decide which preprocessing is best by comparing the cross validation results of each pipeline.
Of course, you don't have unlimited time. I would:

First look at the distribution of cost globally and see if you think you need to transform the target to make it more normal
Consider a tree-based algorithm for a first-pass which won't require you to scale your Problem_Size
Scale globally first, and then if you have time experiment with scaling based on problem type

You'll eventually run out of time or patience, but I think going in this order will make the most efficient use of your time.
Remember, there's not 1 best way to do it for every problem.


Model prediction on meshgrid in python Suppose I have data with two independent variable $X_1$, $X_2$ and one dependent variable say $y$, as follows:

$X_1$: $x_{1,1}$, $x_{1,2}$ , $x_{1,3}$

$X_2$: $x_{2,1}$, $x_{2,2}$, $x_{2,3}$

$y$: $y_1$, $y_2$, $y_3$

I built some Machine learning model which is good .

Now I want to generate predictions not just for test data but for all possible combinations of test data for example, if our test data looks like

$X_1$: $a$, $b$, $c$

$X_2$: $p$, $q$, $r$

then I want predictions for pairs $(a,p)$$(a,q)$,$(a,r)$,$(b,p)$,$(b,q)$....etc

I have tried np.ravel, Meshgrid kind of commands but find it difficult.

It is possible.

It is called Multi-output regression. Allows you to predict more that one variable:

Multioutput regression


What's the difference between Error, Risk and Loss? When we talk about 'Minimizing Loss', we often talk about loss functions such as Mean Squared Error (MSE); the term 'Empirical Risk Minimization' is often used interchangeably. So what's the difference between Error, Loss and Risk?

Error

In this context, error is the difference between the actual / true value ($\theta$) and the predicted / estimated value ($\hat\theta$)

$$Error = \theta - \hat\theta$$

Loss

Loss and Risk are both measurements of how well a model fits the 'data'. The difference is what 'data' means.

Loss ($L$) is a measurement of how well our model perform against your training data. You calculate loss using a loss function, many of which uses error in its equation; for example, simple loss functions may include:

Mean Absolute Error (MAE) - average of the errors

$$L = \frac{1}{n}\sum_{i=0}^n|\theta - \hat\theta|$$
Mean Squared Error (MSE) - average of the squares of the errors

$$L = \frac{1}{n}\sum_{i=0}^n(\theta - \hat\theta)^2$$
Lp - MAE and MSE apply a power of 1 and 2 to the errors, respectively, and averages them. Because of this, they are also called L1 and L2 loss functions. However, you can apply higher (or negative) powers, or $p$, to the errors.

$$L = \frac{1}{n}\sum_{i=0}^n|\theta - \hat\theta|^p$$

Whilst error is simply difference between $\theta$ and $\hat\theta$, loss can differ depending on which loss function you pick. The loss function you pick depends on your data and your objectives. For instance, MAE, or L1, are sensitive to outliers; so if you want your model to also be sensitive to outliers, then you can use MAE.

Your loss depends on the loss function used, which depends on your use case; on the other hand, the error is always the same.

Risk

To re-iterate, loss measures how well your model fits against your training data. However, our end goal is not to fit our model to our training data, which can lead to overfitting. Instead, it is to fit against our validation and test data, or simply any new unseen data.

This is where (true) risk comes in. Risk is the average measure of loss, or expected loss, across your whole data distribution.

$$R(\theta,\hat\theta) = \mathbb{E}[L(\theta,\hat\theta)]$$

To illustrate, let's imagine that you have an overfitted model. Both the errors and loss of your model would be very low (because they are measured against your training data). But because it is overfitted, you'd expect the loss for any new data to be high. Thus, you have a model which has low error and loss, but very high risk.

Empirical Risk

To bring this back full circle - when we train our model, we do not have the full distribution of the data. This may be because some of our data is used for validation and testing, or that new data points are produced in real-time. The best we can do is to pick our training data in a random way and assume that our training data is representative of the real data.

Therefore, because we don't have all the data, the best we can do is to minimize the empirical risk, from data that we do have (our training data), and use regularization techniques to generalize (i.e. avoid overfitting). This is why minimizing loss and minimizing empirical risk are roughly the same thing.

Summary

When we are training our model, our focus should not be on minimizing errors or loss, but to minimize true risk. But most of the time, we can't, so we minimize the empirical risk and regularize.


What stopping metric to chose to optimize 'sensitivity' for a GBM in H2O? I am predicting a disease and want to get the highest possible sensitivity score for the predicted values on my validation and test set.

What stopping metric can be used to optimize the sensitivity score on validation set?

I have about 400 observations. The response variable is binary (0/1) and I have 40 predictor variables.

My current set-up uses AUC as stopping metric.

df <- as.h2o(df)
split <- h2o.splitFrame(data=df, ratios=c(0.6, 0.2)) # split 60, 20, 20%
train <- h2o.assign(split[[1]], "train.hex") # 60%
valid <- h2o.assign(split[[2]], "valid.hex") # 20%
test <- h2o.assign(split[[3]], "test.hex") # 20%

x <- setdiff(names(df), "disease")
y <- "disease"

gbm <- h2o.gbm(
  x = x,
  y = y,
  training_frame = train,
  validation_frame = valid,
  ntrees = 10000,
  learn_rate=0.01,
  # Stopping parameters
  stopping_rounds = 5, stopping_tolerance = 1e-4, stopping_metric = "AUC",
  sample_rate = 0.8,
  col_sample_rate = 0.8,
  seed = 1234,
  nfolds = 50,
  score_tree_interval = 10
)
h2o.auc(h2o.performance(gbm, valid = TRUE))

One solution to make the recall more important whitout getting to the problem mentionned by @Jurgy is to use $F_\beta$, a modified version of $F_1$ where recall is considered $\beta$ time more important. As seen here : https://en.wikipedia.org/wiki/F1_score, $F_\beta$ can be formulated both in terms of recall/precision, and in term of type I /type II error. You then need to select a $\beta$ based on how much recall is more important to you... this can be done by considerations on the costs of type I vs the cost of type II errors.


Impute missing value: transpose or not? I'm building a model that fills the missing values from a Dataframe that contains the number of visitors for different stores, each day:

day
store_a
store_b
store_c

2021-01-01
100
200
300

2021-01-02
110
220
290

2021-01-03
50
110
170

2021-01-04
NAN
220
290

2021-01-05
7
16
NAN

2021-01-06
90
NAN
NAN

I'm using the IterativeImputer class from scikit-learn, this method of imputation puts aside one column at each step and train an estimator on the other columns to predict the column that was put aside.
My question is: should I transpose my dataframe or not?
If I keep my dataframe (1 line = 1 day, 1 column = 1 store), this means that we can completely predict the number of visitors in a store in a particular day just by looking at the other stores.
But if I transpose my dataframe (1 line = 1 store, 1 line = 1 day), this means that we can predict the number of visitors just by looking at the history of one store.
I guess one simple way to check if I should transpose is comparing the RMSE of the two methods but I wanted some explanation instead of "method A works better, move along."

Just by eyeballing the data i would think the number of visitors for store A on jan 4th is around 110. Basically it is always approximately half of Store B on any given day.
It seems like the number of visitors across stores are correlated, so you could potentially use simple linear regression to get a reasonable estimate for any given day.


What's the difference of calculating features directly vs having a CNN interpret it? What's the difference of calculating features directly vs having a CNN interpret it?

If I train a CNN, then it will predict, whether a given input is of the recognized type.

However, what if my data is such that I can interpret, whether it's of a certain type by doing direct measurements on the data? That I don't need to "compare" it to something. Is CNN pointless in this case?

Or perhaps it's that CNN becomes viable when there are so many measurable parameters that trying to infer all of them by manual algorithm becomes laborious?

what if my data is such that I can interpret, whether it's of a certain type by doing direct measurements on the data?

I love machine learning as much as the next guy, but if there is a known way to map your input to your desired output, you should use that, period.

Or perhaps it's that CNN becomes viable when there are so many measurable parameters that trying to infer all of them by manual algorithm becomes laborious?

If speed becomes an issue, then indeed, you may need to consider alternatives, including training a model. There are indeed situations where a learned function may be a bit more simplistic, but good enough and faster than an exact algorithm.

However, these scenarios are quite rare. Inference with neural networks may require non-negligible computational power, so I would guess that in most situations, the exact algorithm will be faster regardless.


A practical reason to use Cross-entropy as a error-function in Neural networks? Cross-entropy tends to allow errors to change weights even when nodes
  saturate (which means that their derivatives are asymptotically close
  to 0.)
  Link

Why is the above statement true? Figures and examples if possible.

During back-propagation training, you want to drive output node values to either 1.0 or 0.0 depending on the target values.
If you use MSE, the weight adjustment factor (the gradient) contains a term of (output) * (1 – output). As the computed output gets closer and closer to either 0.0 or 1.0 the value of (output) * (1 – output) gets smaller and smaller.
For example, if output = 0.6 then (output) * (1 – output) = 0.24 but if output is 0.95 then (output) * (1 – output) = 0.0475.
As the adjustment factor gets smaller and smaller, the change in weights gets smaller and smaller and training can stall out, so to speak.
But if you use cross-entropy error, the (output) * (1 – output) term goes away (the math is very cool). So, the weight changes don’t get smaller and smaller and so training isn’t s likely to stall out.

Note that this argument assumes you’re doing neural network classification, with either softmax output node activation plus multiclass logloss, or sigmoid output node activation plus binary  logloss.

Reference: https://jamesmccaffrey.wordpress.com/2013/11/05/why-you-should-use-cross-entropy-error-instead-of-classification-error-or-mean-squared-error-for-neural-network-classifier-training/


Is feature scaling necessary in reinforcement learning for the agent to learn successfully? I often have trouble deciding how to tweak my input data for the agent. I am changing my data so that values in the hundreds get changed to a value between -1 and 1, and values that are originally small get changed the same ratio. However, my agents often aren't successful.

Could the numeric range of the input data be a cause for the inability to learn?

I assume that when you say

  values that are originally small get changed the same ratio

you are standardizing your features so that every dimension is on the same scale from [-1,1]

This is common practice as it helps many models converge faster.  It would not prevent your models from generalizing the data, so here are some other possible causes for your problem:

The ML algorithm you are using is not appropriate for your task
The data is either insufficient in quantity or quality (e.g. noisy data)
The preprocessing stage is lacking some steps such as dimensionality reduction


Isolation Forest Prediction Mechanics: Does it compare value with every tree (and the original training subset)? So I understand the general idea of how isolation forests works, but I'm having trouble understanding how the model makes predictions on new data.

Does it pass every new point (separately) through every tree in the trained model, and then runs the exact same splits with the exact same original subset data (plus this single new point), to determine the number of steps (anomaly score) until this new point gets isolated?
Then this score gets compared back to the anomaly score threshold that was set when the model was trained?

You can find the sklearn implementation of Isolation Forest in Python at https://github.com/scikit-learn/scikit-learn/blob/a24c8b46/sklearn/ensemble/iforest.py#L229

It calculates the mean path depth needed to classify new samples. They are scored relatively to a theoretical average path length. Original training data is not used, only the learned trees.


How to create a social network like IBM's Watson News Explorer? I was much impressed by the news-explorer view of IBM.It shows a network view with on searching a keyword and it explores the relational edges with a graph.

Here it is

I think it uses the k-means clustering in background.I too want to do the same for twitter data.I have tweets and users and mentions i want show same a network relation between them.can any one suggest me any graphical data tool which shows the network just like IBM by taking data.

using clustering algorithms like k-means or any other make my work done?
or do i need to go for any other approach ?

They've created a graph from the news articles, topics, and named entities (locations, persons, companies, organizations). There are a lot of things going on here, but k-means is not one of them. If I had to do this I would use a named entity recognition (NER) and document embeddings (doc2vec, etc.). Once you have the embeddings, and the edges (thanks to NER), use a graph layout algorithm like force direction. If the graph is too dense, thin out the weaker edges. If all that sounds Chinese to you, start by reading about "named entity recognition", and "word embeddings". The idea is to attach a number (or rather, a vector) to everything from a word to a document.

Twitter is a different beast. The textual content, such as it is, won't play nice with these embedding algorithms, but you have hashtags and strong social signals; mentions, retweets, and follows. This is pretty complicated, too, so I'll leave you with a relevant paper: Twitter-Network Topic Model: A Full Bayesian Treatment for Social Network and Text Modeling.


Weights initialization in Neural Network I was viewing code for custom neural network for sentiment analysis. It had 3 layers (1 hidden layer). I am more concerned with weight initialization for the layers

 self.weights_0_1 = np.zeros((self.input_nodes,self.hidden_nodes))
 self.weights_1_2 = np.random.normal(0.0, self.output_nodes**-0.5,(self.hidden_nodes,self.output_nodes))

What is the idea behind initializing zero's weight matrix. I have learned that initializing weights to zero might lead to linearity.

This might be a very vague question, I will be happy to provide any specifics you want.
https://github.com/udacity/deep-learning/blob/master/sentiment-network/Sentiment_Classification_Solutions.ipynb

Three layers with one hidden layer? This sounds wrong. You have 2 layers, input -> hides -> output, that's only 2 layers, with two sets of weights.

A single layer that gets initialized with only 0 will not converge, because the derivatives will be strictly identical (this rule also applies for any such last layer in a network, the layer gets basically useless). Here, there is a second layer after that saves the day, but it's still a bad practice in general.


What is non-decomposable and/or non-differentiable loss function? I have been reading some deep learning literature and came up with these concepts of non-decomposable and non-differentiable loss functions. My question is are these same thing? if not how are they different? How do we know if a loss function is non-decomposable/non-differentiable?

Non-differentiable (loss) functions can be labelled according to the stardard definitions of differentiation: that the function has a derivative at all points in its domain. For example, the ReLU loss function is technically non-differentiable, because its gradient is not defined at zero (or wherever the two lines meet, if you're using a variant of the standard ReLU).

Non-decomposable functions must be looked at slightly differently. A loss function that is not decomposable is usually one that is composed of several statistics across the training metrics. Take the F1 score, for example:

$$
F1 = 2 \cdot \frac{precision \cdot recall}{precision + recall}
$$

Using this as a loss function means you would have to expand the terms for precision and recall in terms of your predictions over a batch, then work out the gradients of each function, finally combining them. It can be done (but it gets complicated), but I would imagine there are trade-offs when optimising at the level of such a metric, which really summarises your model's performance as an average over samples. Using a traditional loss function focusses on improving the model at the more granular sample level. This means the gradients are concentrated at the purest level and I would imagine better optimisation is possible. It could be that optimising at the more abstract level of the F1 score leads to less overfitting or even better generalisation. This is all just an idea though; I haven't seen any work making such comparisons directly.

Here is a short reddit thread with some interesting points and examples about when you might even consider trying to optimise such a non-decompasable function directly (i.e. using it as a loss function).

Have a look at this paper, which speak about non-decomposable loss functions and offers a way to deal with them. Reading through that and comparing the functions that the authors talk about should help you gain a better understanding of the key differences.


Word2Vec, softmax function I was going term by term through the softmax function for the word2vec (SKIP-GRAM) model.  I found  most definition of these functions to be not 'clear'  so I modified the notation to make sure I understand it.

Is the following  formulation correct?

$$P(w_{-t} | w_{t} ; \theta) = softmax(score(w_{-t}, w_t))$$

$$P(w_{-t} | w_{t} ; \theta) = \frac{exp(score(w_{-t}, w_t))}{\sum_{w' \in \theta} exp(score(w', w_{t}))}$$

where:

$w_{-t} =$ context

$w_{t} =$ target word

$score(A,B)$ a measure of similarity between vectors A and B.

$\theta = $ vector representation for all words in vocabulary

In the simplest case:

$$score(A ,B) = A \cdot B$$

Your definition is correct. For the reference you can compare it with the probabilistic model from Tensorflow "Vector Representations of Words" tutorial:

  $$
\begin{align}
P(w_t | h) &= \text{softmax} (\text{score} (w_t, h)) \\
\\
           &= \frac{\exp \{ \text{score} (w_t, h) \} }
             {\sum_\text{Word w' in Vocab} \exp \{ \text{score} (w', h) \}}
\end{align}
$$

It's the same as yours, but they generalize the condition $h$ as a history.

In case of skip-gram, the target word $w_t$ is any context word ($w_{-t}$ in your notation) and the history is the center word ($w_{t}$ in your notation). You specify $\theta$ explicitly in the condition, but that is usually omitted, because there is only one vocabulary in a given problem.


Keeping part of a string in R I have a dataframe with the following column

city <- c("Sydney NSW", "Newcastle NSW", "Liverpool NSW", "Broken Hill NSW")

I want to maintain everything prior to NSW (space included). What Regex expression can be used in R for that?

Use strsplit().

If you want to remove the space between the two words (along with 'NSW'):

city_clean <- unlist(c(strsplit(city, " NSW")))

Output: [1] "Sydney"      "Newcastle"   "Liverpool"   "Broken Hill"

It wasn't clear to me whether you wanted to keep the space or not. If you want to keep the trailing space after the city name

city_clean <- unlist(c(strsplit(city, "NSW")))

Output: [1] "Sydney "      "Newcastle "   "Liverpool "   "Broken Hill "

If you prefer to use regex, here's another solution:

city_clean <- gsub(" NSW", "", city)

Output: [1] "Sydney"      "Newcastle"   "Liverpool"   "Broken Hill"


Why doesn't overfitting devastate neural networks for MNIST classification? I have a simple neural network(NN) for MNIST classification.  It includes 2 hidden layers, each with 500 neurons.  Hence the dimensions of the NN are: 784-500-500-10.  ReLU is used in all neurons, softmax is used at the output, and cross-entropy is the loss function.

What puzzles me is why overfitting doesn't appear to devastate the NN?

Consider the number of parameters (weights) of the NN.  It's approximately
$$784\times500+500\times 500+500\times 10=647000.$$
However, in my experiment, I used only $6000$ examples (a tenth of the MNIST training set) to train the NN. (This is just to keep the run time short.  The training & test error would both go down considerably if I used more training examples.)  I repeated the experiment 10 times.  Plain stochastic gradient descent is used (no RMS prop or momentum); no regularization/drop-out/early-stopping was used.  The training error and test error reported were:

$$\begin{array}{|l|c|c|c|c|c|c|c|c|c|c|}
\hline
\textrm{No.} & 1 & 2 & 3 &4 &5&6&7&8&9&10\\ \hline
E_{train}(\%) & 7.8 & 10.3 &  9.1 & 11.0 & 8.7 & 9.2 & 9.3 & 8.3 &10.3& 8.6\\ \hline
E_{test}(\%) & 11.7 & 13.9 & 13.2 & 14.1 &12.1 &13.2 &13.3 &11.9 &13.4&12.7\\ \hline
\end{array}$$

Note that in all 10 experiments (each with independent random parameter initialization), the test error differed from the training error only by approx. 4%, even though I used 6K examples to training 647K parameters.  The VC dimension of the neural network is on the order of $O(|E|log(|E|))$ at least, where $|E|$ is the number of edges (weights).  So why wasn't the test error miserably higher (e.g. 30% or 50%) than the training error?  I'd greatly appreciate it if someone can point out where I missed.  Thanks a lot!

[ EDITS 2017/6/30]

To clarify the effects of early stopping, I did the 10 experiments again, each now with 20 epochs of training.  The error rates are shown in the figure below:

$\qquad\qquad\qquad$

The gap between test and training error did increase as more epochs are used in the training.  However, the tail of the test error stayed nearly flat after the training error is driven to zero.  Moreover, I saw similar trends for other sizes of the training set.  The average error rate at the end of 20 epochs of training is plotted against the size of the training set below:

$\qquad\qquad\qquad$

So overfitting does occur, but it doesn't appear to devastate the NN.  Considering the number of parameters (647K) we need to the train and the number of training examples we have (<60K), the question remains: why doesn't overfitting easily render the NN useless?  Moreover, is this true for the ReLU NN for all classification tasks with softmax output and cross-entropy objective function?  Have someone seen a counter-example?

I have replicated your results using Keras, and got very similar numbers so I don't think you are doing anything wrong.

Out of interest, I ran for many more epochs to see what would happen. The accuracy of test and train results remained pretty stable. However, the loss values drifted further apart over time. After 10 epochs or so, I was getting 100% train accuracy, 94.3% test accuracy - with loss values around 0.01 and 0.22 respectively. After 20,000 epochs, the accuracies had barely changed, but I had training loss 0.000005 and test loss 0.36. The losses were also still diverging, albeit very slowly. In my opinion, the network is clearly over-fitting.

So the question could be re-phrased: Why, despite over-fitting, does a neural network trained to the MNIST data set still generalise apparently reasonably well in terms of accuracy?

It is worth comparing this 94.3% accuracy with what is possible using more naive approaches.

For instance, a simple linear softmax regression (essentially the same neural network without the hidden layers), gives a quick stable accuracy of 95.1% train, and 90.7% test. This shows that a lot of the data separates linearly - you can draw hyperplanes in the 784 dimensions and 90% of the digit images will sit inside the correct "box" with no further refinement required. From this, you might expect an overfit non-linear solution to get a worse result than 90%, but maybe no worse than 80% because intuitively forming an over-complex boundary around e.g. a "5" found inside the box for "3" will only incorrectly assign a small amount of this naive 3 manifold. But we're better than this 80% lower bound guesstimate from the linear model.

Another possible naive model is template matching, or nearest-neighbour. This is a reasonable analogy to what the over-fitting is doing - it creates a local area close to each training example where it will predict the same class. Problems with over-fitting occur in the space in-between where the values of activation will follow whatever the network "naturally" does. Note the worst case, and what you often see in explanatory diagrams, would be some highly curved almost-chaotic surface which travels through other classifications. But actually it may be more natural for the neural network to more smoothly interpolate between points - what it actually does depends on the nature of the higher order curves that the network combines into approximations, and how well those already fit to the data.

I borrowed the code for a KNN solution from this blog on MNIST with K Nearest Neighbours. Using k=1 - i.e. choosing the label of the nearest from the 6000 training examples just by matching pixel values, gives an accuracy of 91%. The 3% extra that the over-trained neural network achieves does not seem quite so impressive given the simplicity of pixel-match counting that KNN with k=1 is doing.

I tried a few variations of network architecture, different activation functions, different number and sizes of layers - none using regularisation. However, with 6000 training examples, I could not get any of them to overfit in a way where test accuracy dropped dramatically. Even reducing to just 600 training examples just made the plateau lower, at ~86% accuracy.

My basic conclusion is that MNIST examples have relatively smooth transitions between classes in feature space, and that neural networks can fit to these and interpolate between the classes in a "natural" manner given NN building blocks for function approximation - without adding high frequency components to the approximation that could cause issues in an overfit scenario.

It might be an interesting experiment to try with a "noisy MNIST" set where an amount of random noise or distortion is added to both training and test examples. Regularized models would be expected to perform OK on this dataset, but perhaps in that scenario the over-fitting would cause more obvious problems with accuracy.

This is from before the update with further tests by OP.

From your comments, you say that your test results are all taken after running a single epoch. You have essentially used early stopping, despite writing that you have not, because you have stopped the training at the earliest possible point given your training data.

I would suggest running for many more epochs if you want to see how the network is truly converging. Start with 10 epochs, consider going up to 100. One epoch is not many for this problem, especially on 6000 samples.

Although increasing number of iterations is not guaranteed to make your network overfit worse than it already has, you haven't really given it much of a chance, and your experimental results so far are not conclusive.

In fact I would half expect your test data results to improve following a 2nd, 3rd epoch, before starting to fall away from the training metrics as the epoch numbers increase. I would also expect your training error to approach 0% as the network approached convergence.


Can hadoop with Spark be configured with 1GB RAM I'm trying to set up a cluster (1 namenode, 1 datanode) on AWS.
I'm using free one year trial period of AWS, but the challenge is, instance is created with 1GB of RAM.

As I'm a student, I cannot afford much. Can anyone please suggest me some solution?

Also, it would be great if you could provide any links for setting up multi cluster hadoop with spark on AWS.

Note: I cannot try in GCE as my trial period is exhausted.

So if 4GB of RAM isn't sufficient, 1GB isn't going to be. That is really too little to run an HDFS namenode, a datanode, YARN, Spark driver alone, let alone leaving room for your workers.

Much more reasonable is to simply run Spark locally on that instance without Hadoop at all.

But I would question whether Spark is the right choice if you are definitely limited to such a small machine.


Training Dataset required for Classifier I am currently trying to develop a classifier in python using Naive Bayes technique. I need a dataset so that I can train it. My classifier would classify a new document given to it into one of the four categories : Science and technology, Sports, politics, Entertainment. Can anybody please help me find a dataset for this. I've been stuck on this problem for quite some time now. Any help would be greatly appreciated.

You have at least three options:

Use some of many available datasets (for example: BBC documents); if you need more, simply go to Google Scholar or any similar service and search for classification news politics sports. In the articles you may find many references to available datasets;
Crawl any news service, and use clustering techniques to group articles into clusters (this often separates the articles along their domain, such as politics, sports etc.), and label the articles by their affiliation to a cluster;
Crawl politics-specific, sports-specific news services and use them as labeled datasets.


how to do topic modeling on very huge data? When i come to know that gensim is useful library for topic modeling, I tried it on my huge amount of document. It works well only if the dictionary size is to be fix. In my case i have each and every token is important so i would have huge amount of token at that time gensim algo like(LDA,LSA,etc) fails and get out of memory execution issue. Actually they written that there is no limit for amount of document but internally they keep the dictionary size fixed that's why if new word will come and exceed the limit than it starts to truncate it. I dont want to truncate my token as possible. So is there any solution for that can resolve my issue of topic modeling on big data. I have resource limitation.

Use filter_extremes to Filter out tokens that appear in less than no_below documents (absolute number) or more than no_above documents (fraction of total corpus size, not absolute number).

keep only the first keep_n most frequent tokens (or keep all if None).

After the pruning, shrink resulting gaps in word ids.


What does this symbol means, what operator is it? I am confused about the $E_{x\sim P_{data}(x)}$, what does $E$ means here. I cannot find an appropriate answer on the internet, and hence I am trying data science stack exchange. Please help.

$\mathbb{E}$ means expected value.

The subscript is there to clarify which random variable is the expected value taken.


LSTM with auxiliary data achieves abismal results I have a fairly simple LSTM models that achieves ok results.

I wanted to add auxiliary data as appeared in the keras docs. I added only one binary variable. I expected this model to be at least as good, assuming that if it has no information its weights will go to zero. In reality, the model fails and just predicts zeros for my entire test set.
The second model:

More specifically, I'm predicting a time series of human-behaviour, and my binary feature tags whether the time point is in a weekend. I'd expect it capture some of the weekend (black background) reduction, and if not, at least not to ruin the model completely.

Is there a reason why this new information has to be a part of your model? There's no rule that says that all data must be fed into your model. Maybe you have just found a feature that shouldn't be included in your predictive phases? And if so, that's perfectly fine, there is nothing wrong with that.


Should I keep common stop-words when preprocessing for word embedding? If I want to construct a word embedding by predicting a target word given context words, is it better to remove stop words or keep them?

the quick brown fox jumped over the lazy dog

or

quick brown fox jumped lazy dog

As a human, I feel like keeping the stop words makes it easier to understand even though they are superfluous.
So what about for a Neural Network?

In general stop-words can be omitted since they do not contain any useful information about the content of your sentence or document.
The intuition behind that is that stop-words are the most common words in a language and occur in every document independent of the context. Therefore they contain no valuable information which could hint to the content of the document.


Is the external memory of Differtiable Neural Computer limited? Is there a CONSTANT limit in DNC external memory usage?  Like a human brain, 7 chunks of information? Or is it some sort of a hyperparameter? How much data it can associate using that Short-Term memory?

If the reference to 7 chunks of information for humans is a reference to the 'Magical Number Seven' Paper, which is the human working memory, then:

Consider the high level operation of DNC at a timestep. An input comes in, and is processed by an LSTM to generate a 'Interface vector' which interacts with memory with Write, Erase and Read operations. The operations are then executed, and a final layer produces an output based on the retrieved information and the input.

Where is the analogy to human 'working memory'?

1). If you consider the working memory is what is retrieved on read, then there is a hyperparameter for number of memory reads. This applies generally to Write and Erase as well as those are all hyperparameters which determine the size of the Interface Vector. In the paper they used a write size of 1 in all cases for purposes of controlling the experiments. Note that the 'size' of what is read/written in a single operation is a separate parameter, which just depends on how the problem is encoded (see the methods section of the paper).

2). If you consider working memory as external memory, then this is another hyperparameter. I think it is a conceptual mistake to view the external memory as short term memory, as the addition of this and all the gating was specifically developed to store information long term and then recall when needed.

3). There is a part of the DNC's operation which queries a link matrix (storing what was written to long term memory and when) during the read phase. This can be seen as being related to human working memory, as numerous behavioral economics studies showed that what we recall and how we process it is effected by what we have just seen. This is the only part in the paper where the computation is O(N^2) w.r.t to the size of the memory, (though they approximate with an O(NlogN) method). This portion of retrieval/bias of what is retrieved to most recently accessed information can then be said to have a limit, since for large memories it will not scale with rest of network, but this is still not constant.

4). Probably the best candidate for the working memory analogy IMO is the hidden state of the controller LSTM, since this part of the algorithm is responsible for storing what is needed over time for retrieval, and on its own is used for storing a state over time. The reason that this is short term memory in this analogy is just that the external memory is an explicit external memory, and this is the only other place where a state is maintained.

So in short no, there is no Constant, as just about every conceivable analogy to 'working memory' and the entire architecture is controlled by parameters. It is worth checking out the code and paper for more details on exactly how the parameters are set for different experiments.

Let me know if this makes sense.


How to determine if my GBM model is overfitting? Below is a simplified example of a h2o gradient boosting machine model using R's iris dataset. The model is trained to predict sepal length.

The example yields an r2 value of 0.93, which seems unrealistic. How can I assess if these are indeed realistic results or simply model overfitting?

library(datasets)
library(h2o)

# Get the iris dataset
df <- iris

# Convert to h2o
df.hex <- as.h2o(df)

# Initiate h2o
h2o.init()

# Train GBM model
gbm_model <- h2o.gbm(x = 2:5, y = 1, df.hex,
                     ntrees=100, max_depth=4, learn_rate=0.1)

# Check Accuracy
perf_gbm <- h2o.performance(gbm_model)
rsq_gbm <- h2o.r2(perf_gbm)

---------->

> rsq_gbm
[1] 0.9312635

The term overfitting means the model is learning relationships between attributes that only exist in this specific dataset and do not generalize to new, unseen data.
Just by looking at the model accuracy on the data that was used to train the model, you won't be able to detect if your model is or isn't overfitting.

To see if you are overfitting, split your dataset into two separate sets:

a train set (used to train the model)
a test set (used to test the model accuracy)

A 90% train, 10% test split is very common. Train your model on the train test and evaluate its performance both on the test and the train set. If the accuracy on the test set is much lower than the models accuracy on the train set, the model is overfitting.

You can also use cross-validation (e.g. splitting the data into 10 sets of equal size, for each iteration use one as test and the others as train) to get a result that is less influenced by irregularities in your splits.


Validation Curve Interpretation I have been reading about the validation_curve function from scikit learn. When I run this it takes too long. Therefore, I am plotting the results from grid search instead, which seems to be far quicker. However, when I plot using the code from: https://matthewbilyeu.com/blog/2019-02-05/validation-curve-plot-from-gridsearchcv-results.
I get:

Why are the lines so close together? Shouldn't I get a curve instead? More generally, how can I tell if my model is under-fitting or overfitting? How  can I plot the overfit/under-fit areas on my plot.
edit: i am trying to predict if a client will  default on a loan or not so 1 for yes 0 for no. the data are financials. also the data i have used is about 100k samples

You can understand whether your model is overfitting or underfitting  by the difference in the graph of your train and validation score. If your train score(performance not loss) is low and so is val score then your model is underfitting. On the other hand if your model is overfiiting you will have high training accuracy but your validation score will be low and the train and val graph will be far from each other. A perfect model just has high training score with the validation  curve as close as possible and the two graphs will be very close like the graph you provided.


Obtaining a confidence interval for the prediction of a linear regression The data I am working with is being used to predict the duration of a trip between two points. There are about 100 different trips in the data and ~90k observations.

I am using the standard pattern:

feature_cols = df_features.columns.drop( [ 'log_duration' ] )
X            = df_features[ feature_cols ]
y            = df_features.log_duration

X_train, X_test, y_train, y_test = train_test_split( X, y, random_state = 42 )
linreg = LinearRegression()
linreg.fit( X_train, y_train )
linreg.score( X_test, y_test )

To perform the regression and obtain my score (~.74).

However, let's say, that it predicts that it will take 40 minutes to make the trip between two points. Obviously, it will not take exactly 40 minutes. What I am looking for is a way to report that it will take 40 minutes +/- C number of minutes for the trip.

Using Pandas and SciKit, how can I obtain C?

You can estimate the standard deviation of your prediction:

stdev = np.sqrt(sum((linreg.predict(X_train) - y_train)**2) / (len(y_train) - 2))

Then, for any significance level you want, you should check correspondent Gaussian critical value (for example, for significance level 95% it is 1.96).

Finally, confidence intervals are (prediction - 1.96*stdev, prediction + 1.96*stdev) (or similarly for any other confidence level).

Another approach is to use statsmodels package.


Is a dense layer required for implementing Bahdanau attention? I saw that everyone adds Dense( ) layer in their custom Bahdanau attention layer, which I think isn't needed.

This is an image from a tutorial here. Here, we are just multiplying 2 vectors and then doing several operations on these vectors only. So what is the need of Dense( ) layer. Is the tutorial on 'how does attention work' wrong?

A Dense layer inside the Attention layer logic is a must. This is typically a single unit, single layer MLP (though in case of Additive Attention, there is a provision to use more units). This layer helps us derive the attention weights during training
This Dense layer transforms the input (each of the encoder LSTM layer output) keeping the dimensions the same. This is then multiplied individually with the last decoder hidden state & softmaxed to generate the attention weights.
Multiplying the attention weights with the encoder hidden states help to magnify or diminish the relevant input information and this context can then be used for Predictions. (In Additive layer addition is used instead of multiplication)
Without the Dense layer in the Attention layer, we don't get the Attention weights. Without Attention weights, there is no context. Without context there is no Attention.
So in order to incorporate Attention, a single layer Dense MLP is mandatory. The model learns the weights of this layer during training. These layer weights are then used to derive the attention weights.


Why do we have to divide by 2 in the ML squared error cost function? I'm not sure why you need to multiply by $\frac1{2m}$ in the beginning. I understand that you would have to divide the whole sum by $\frac1{m}$, but why do we have to multiply $m$ by two?

Is it because we have two $\theta$ here in the example?

It is simple. It is because when you take the derivative of the cost function, that is used in updating the parameters during gradient descent, that $2$ in the power get cancelled with the $\frac{1}{2}$ multiplier, thus the derivation is cleaner. These techniques are or somewhat similar are widely used in math in order "To make the derivations mathematically more convenient". You can simply remove the multiplier, see here for example, and expect the same result.


Genetic algorithm only using selection Suppose you have a population of N individuals with fitness 1, 2, . . . , N (i.e., all individuals have a
unique fitness value). Suppose you repeatedly apply tournament selection without replacement with
tournament size s = 2 to this population, without doing crossover, mutation, and replacement.

In other words, you run a genetic algorithm with selection alone.
After a certain number of generations you will end up with a population consisting of N copies of
the same individual.
Can you give an estimate of the number of generations needed to achieve that?

Is the answer to this just log(n) ?

The complexity of one generation will be O(N/2). In order to make the whole population the same, you will need N generations. My answer will be O(N**2).

You can also evaluate the time complexity by actually measuring the run time and averaging.

If you don't do crossover, mutation or replacement you are just selecting the best of your individuals.


Clamping Q function to it's theoretical maximum, yes or no? I'm implementing DQN algorithm from scratch on MountainCar simulation.
I'm using a setup of $reward = 1.0$ when car hits the flag, and $0$ otherwise.
Reward decay factor is set to $\gamma=0.99$. Algorithm starts with exploration factor of $\epsilon = 1.0$ and decreases that over time to $\epsilon = 0.1$.

If I understood correctly, $Q$ function for some state and action pair is defined as:

$Q(s_t, a_t) = r_t + \gamma \times \operatorname{arg\,max}_a Q(s_{t+1}, a_{t+1})$

So, $Q_{max}$ would satisfy the condition:

$Q_{max} = r_{max} + \gamma \times Q_{max}$

Which means:

$Q_{max} = \frac{r_{max}}{1 - \gamma}$

However, since my network only approximates the $Q$ function it is possible for it to sometimes produce value greater than $Q_{max}$. When that happens, further training causes the values to start growing exponentially, and the entire thing blows up.

When I clamp the error of expected value vs. current predicted value to some small number, it still causes the blowup just a bit slower.

The only solution I can think of is to clamp the predicted value $Q(s_{t+1}, a{t+1})$ to $Q_{max}$ and forcing it to never go above that.
I have done that, and got OK results with it.

Does this make sense? Is this a situation that happens in DQN? Or maybe I missed something and my implementation is a bit buggy?

As MountainCar is often solved with $\gamma = 1$ and a negative reward per timestep, you would immediately hit a problem with your ability to calculate a maximum action value in that case. However, I don't think that is your problem here, the discounted return with positive reward at the end should still encourage desired behaviour for the problem.

It is likely that you are experiencing known problems with RL, and the "deadly triad":

Function approximation (neural network)
on a bootstrap method (Q-Learning or any TD-learning approach)
off policy (learning optimal policy from non-optimal behaviour*, which is a feature of Q-learning)

This combination is often unstable and difficult to train. Your Q value clamping is one way to help stabilise values. Some features of DQN approach are also designed to deal with this issue:

Experience replay. Agent does not learn online, but instead puts each sample (S, A, R, S') into a memory table and trains the neural network on mini-batches sampled from this memory. Commonly this minibatch update is run on every step (after enough experience collected) and might be e.g. size 32. So learning updates happen faster than experience is collected.
Frozen bootstrap target network. The network used to calculate $\operatorname{max}_{a'} Q(s', a')$ when setting the target values to learn (in td target $R + \gamma \operatorname{max}_{a'} Q(s', a')$) is kept stable, and refreshed to use a copy of the current weights after a certain number of steps (e.g. 100 or 1000, or once every 10 episodes).

* Technically off-policy learning is learning the cost function for any policy $\pi$ from a different behaviour policy $b$, but the common case is for control systems attempting to find an optimal policy from an exploratory policy.


Appraise the statement: “For the model  = 0 + 1 + , 1 reflects the causal effect of  on .” Ask not sure if this was the right place to ask my question, but I saw some questions regarding linear regression so I'd thought I would try to get some answers here. I just started learning about linear regression so this is the homework posed to me.

I assume that the statement is true since 1 is the coefficient for . And its the coefficient that would determine if the slope (i.e. the relationship) is positive or negative.

Am I missing out anything or what should I expound on?

Thanks for reading and for the guides and opinions.

Hi and welcome to the forum. Homework is not so well received here in the forum. But still a fair question in my view.

You have two aspects here. In principle you are right that $\beta_1$ is the slope of $x_1$ (you can say the marginal effect of $x_1$ on $y$) and $\beta_0$ is the intercept. This is simply a linear function of form $f(x)=\beta_0 + \beta_1 x$.

However, to claim "causality", a few more things are required. First, you need to make the assumption that there is a causal relation between $x$ and $y$ and $x$ must be exogenous.

Another important aspect is, that if there are additional variables with a causal influence on $y$, say $x_2$, you cannot really claim that $\beta_1$ is the causal effect on $...$, because you omitted $x_2$, so that your model suffers from the omitted variable bias. To claim for causality you need to make sure that your model reflects the data generating process in a proper way.


heatmap color and d3.js This follows a recent post where I got my answer concerning possible web-based techno for plotting big heatmaps ($600 \times 600$ dissimilarity matrix).

Now, that I have achieved that, I'm struggling with setting the color scale.
I would like some fine continuous color scale. For now I have plugged a discretization into 17 colors... not enough. With Python, pretty much is done with pyplot.pcolor.

How can I obtain such a 'continuous' color scale?

Here the difference between the two heatmaps parameterized by the same matrix:

Using matplotlib.pyplot in Python

Using D3.js and the code below

Here is the code:

<script src="http://d3js.org/d3.v3.js"></script>
<canvas id="heatmap"></canvas>
<script>
var tsvName = 'wasserstein_dists.tsv';
var canvas = document.getElementById('heatmap');
var ctx = canvas.getContext("2d");

var scale = 1;
var colors = ["#6363FF", "#6373FF", "#63A3FF", "#63E3FF", "#63FFFB", "#63FFCB",
           "#63FF9B", "#63FF6B", "#7BFF63", "#BBFF63", "#DBFF63", "#FBFF63",
           "#FFD363", "#FFB363", "#FF8363", "#FF7363", "#FF6364"];
var heatmapColor = d3.scale.linear()
    .domain(d3.range(0, 1, 1.0 / (colors.length - 1)))
    .range(colors);

// On charge le tsv :
d3.tsv(tsvName, function (d) {
  return {
    x: +d.x,
    y: +d.y,
    v: +d.v,
  };
}, function (err, data) {
  if (err != null) { return alert('No tsv file'); }

  var size = Math.sqrt(data.length);
  canvas.width = size * scale;
  canvas.height = size * scale;

  var c = d3.scale.linear()
    .domain([d3.min(data, function (d) { return d.v; }), d3.max(data, function (d) { return d.v; })])
    .range([0,1]);

  data.forEach(function (d) {
    ctx.fillStyle = heatmapColor(c(d.v));
    ctx.fillRect(d.x * scale, d.y * scale, (d.x + 1) * scale, (d.y + 1) * scale);
  });
});

Actually you are almost there. Just use the min/max function directly in the  color scale instead of using the c var. Use the max/min function in the domain and the start/stop color in the range. The d3.scale is already a gradient function.

Something like:

<script src="http://d3js.org/d3.v3.js"></script>
<canvas id="heatmap"></canvas>
<script>
var tsvName = 'wasserstein_dists.tsv';
var canvas = document.getElementById('heatmap');
var ctx = canvas.getContext("2d");

var scale = 1;

// On charge le tsv :
d3.tsv(tsvName, function (d) {
  return {
    x: +d.x,
    y: +d.y,
    v: +d.v,
  };
}, function (err, data) {
  if (err != null) { return alert('No tsv file'); }

  var size = Math.sqrt(data.length);
  var heatmapColor = d3.scale.linear()
    .domain([d3.min(data, function(d) { return d.v; }), d3.max(data, function(d) { return d.v; })])
    .range(["#6363FF",  "#FF6364"]);
  canvas.width = size * scale;
  canvas.height = size * scale;

  data.forEach(function (d) {
    ctx.fillStyle = heatmapColor(d.v);
    ctx.fillRect(d.x * scale, d.y * scale, (d.x + 1) * scale, (d.y + 1) * scale);
  });
});

Of cause you have to first define data and then the color gradient with the min/max function.


How to handle large number of features in machine learning? I try to do normal classification on high dimensional traditional columnar data (several hundred columns). The features are of different type. In this case, it's clearly out of question to examine each features one by one to figure out what are they exactly and what optimization or feature engineering could be done with them.

Still, I have to do all the necessary preprocessing steps like imputation, standardization etc. But even such basic steps like categorical feature encoding or imputation are problematic because R/Python-pandas are sometimes wrongly recognized the numeric/categorical nature of some variables (and as a consequence, wrongly try to encode or mean-impute the NAs), not to mention other very problematic issues that could be handled if one could oversee the features one by one.

Of course, I could turn to models which are capable of handling non-standardized features with NAs but this limits the number of possible models on one hand and seems me very unprofessional on the other hand. What is the way to get over this issue?

There are 4 ways I know in Python. In the following I copied the code I wrote for regression purposes. Classification would be very similar :

First: SelectKBest:

from sklearn.feature_selection import SelectKBest, f_regression
train_data = train_data.apply(pd.to_numeric).astype('float32')

kb = SelectKBest(score_func=f_regression, k=70)
kb.fit(train_data.loc[:, train_data.columns != 'SalePriceLog'], train_data.SalePriceLog)
indices = np.argsort(kb.scores_)[::-1]
selected_features = []
for i in range(5):
  selected_features.append(train_data.columns[indices[i]])
plt.figure()
plt.bar(selected_features, kb.scores_[indices[range(5)]], color='r', align='center')
plt.xticks(rotation=45)

results:

Second: RFE

 from sklearn.linear_model import LogisticRegression, LinearRegression
    from sklearn.feature_selection import RFE
    model = LinearRegression()
    rfe = RFE(model, 10)
    fit_rfe = rfe.fit(train_data.loc[:, train_data.columns != 'SalePriceLog'], train_data.SalePriceLog)
indices_rfe = np.argsort(fit_rfe.ranking_)
selected_features_rfe = []
for i in range(10):
    selected_features_rfe.append(train_data.columns[indices_rfe[i]])
selected_features_rfe
plt.figure()
plt.bar(selected_features_rfe, fit_rfe.ranking_[indices[range(10)]], color='r', align='center')
plt.xticks(rotation=45)

results:

Third: PCA

from sklearn.decomposition import PCA
# pca = PCA(n_components=5)
pca = PCA(0.999)
fit = pca.fit(train_data.loc[:, train_data.columns != 'SalePriceLog'])

Fourth: ExtraTrees

from sklearn.ensemble import ExtraTreesRegressor

model_extra_tree = ExtraTreesRegressor()
model_extra_tree.fit(train_data.loc[:, train_data.columns != 'SalePriceLog'], train_data.SalePriceLog)
indices_extra_tree = np.argsort(model_extra_tree.feature_importances_)[::-1]
selected_feature_extra_tree = []
for i in range(10):
    selected_feature_extra_tree.append(train_data.columns[indices_extra_tree[i]])
plt.figure
plt.bar(selected_feature_extra_tree, model_extra_tree.feature_importances_[indices_extra_tree[range(10)]])
plt.xticks(rotation=45)

results:


What is the common practice for NLP or text mining for non-English? A lot of natural language processing tools are pre-trained with corpus in English. What if ones need to analyze, say, Dutch text? The blogs I find online are mostly saying traslating text into English as pro-processing. Is this the common practice? If not, then what? Also, does how similar a language is to English have an impact on the model performance?
For some also widely speaking languages (e.g French, Spanish), do people construct corpus in their own language and train models on it? Forgive my ignorance because I'm not able to read papers in many languages.

Translation as a pre-processing step is usually sufficient for many tasks (e.g. sentiment classification), but naturally undesirable for other tasks e.g. grading someone in written Dutch fluency.
Hence, for these tasks, the objective is:

Be able to train a language model for your specific language
However, you want to be able to do this with minimal resources

Thus, people in research view this as few-shot learning. One of the most common approaches to this is to exploit meta-learning, i.e. use models that have been pre-trained using many tasks (languages in this case) and they have been trained so that only a few gradient steps and little data is required for them to be applicable to a new task (Dutch in your example). An example of one such model you could finetune (transfer learning) is called multiliingual BERT... this was back in 2019, so I expect since many other multilingual language models have emerged - these can be good initialisation points for any bespoke model you want to train for some new language...


Statistical distances for time series of distributions I am interested in clustering $N$ time series of $T$ 'values' each. These values are distributions (which can be represented by their cumulative distribution functions (cdf), or their probability density functions (pdf), or more convenient forms such as square-root pdfs yielding a simple spheric geometry).

For comparing given distributions, there is an extensive literature on statistical distances (KL, Hellinger, Wasserstein, and so on), but for comparing given time series of distributions, I am not sure whether there is any literature at all?

Such distances should somehow take into account dynamics information besides the distribution proximity at time t. Ideally, I wish I could have a kind of information factorization similar to this result.

I am wondering if such distances already exist and whether this kind of problem has already been formulated in the literature?

--
edit for further precisions and answer to comments:

Thanks for your answer, but dynamic time warping does not suit to my need. This dp technique only captures a rough similarity of shapes by allowing non-linear time distortion. But, it does not amount for the whole information in these time series, e.g. what about the distribution of distortions? Do the distributions of a given time series vary smoothly through time or violently? DTW is not always the solution, for instance, when working with random walks, it does not make sense to use a DTW since there are no time patterns! In this case, the only information is "correlation" and "distribution" (cf. Sklar's theorem in Copula Theory), and the paper cited above.

-- edit 2 Here are the papers that are somehow related to my question:

Predicting the Future Behavior of a Time-Varying Probability Distribution
Clustering on the unit hypersphere using von Mises-Fisher distributions
Unsupervised clustering of multidimensional distributions using earth mover distance
Hilbert space embeddings of conditional distributions with applications to dynamical systems

This is similar to the fundamental information theory problem that Shannon explored.  In that domain, it is framed this way: given two rvs, X and Y, what information does X convey about Y?

An example would be that I create sequence of numbers/bit/letters from a known PDF (X) and you receive a distorted version of those values for which you also know the PDF (Y).  The mutual information is the number of bits that Y communicates about X which can be thought of as a type of correlation.

In the case of time varying PDFs (ie stochastic processes), information theory would just treat the ensemble of rvs as one joint rv and then calculate the mutual information of the joint PDFs.  If the PDFs are iid, then significant simplifications are possible.  Joint Gaussian PDFs also make things easier.

Another information theory concept that might be useful is the entropy rate of a process.  ER is a quantification of the amount of information contained in a process.  Depending on your problem, you might be able to compute the ER for each process and then use those values as realizations of a feature that can be grouped with a clustering algorithm.


Contextual Spell Correction I want to create a spell checker that corrects the spelling mistakes contextually.

For example,

Erroneous sentence: I want to apply for credit cart

Corrected sentence: I want to apply for credit card

Here, the respective spellings of cart and card are correct but the cart is contextually incorrect.

So what methods we can apply for contextual errors like this?

Here's an approach which we can try and probably work for less complex problems.

A large corpus of data: We will require a very large corpus of textual data. It should contain as much as words possible. Natural language phrases and collocations are also required.

Now, we encode all the sentences in the corpus using Word2Vec.

After that, we will train a text generation model which given a word predicts the next word.

If we take in consideration the above example, the word "card" is colloquial with the word "credit". This relation will be learnt my the model.

  The model will behave like, Which is the most frequent word after "credit"? It isn't "cart".


How to add data points to a trained KNN I have a trained KNN, I created with
https://github.com/kevinzakka/blog-code/blob/master/knn/knn.py

I want to add more data points to the KNN but I am on a raspberry pi so limited by RAM and therefore the number of data points I can add at a time to the model.

I have 20k images, I can add 2k at a time. Because of the way scikitlearn works, I cant use the model trained on a Windows PC on the pi (different architectures).

Can I create a model, and then add more data points to it somehow?

Re-estimating parameters with additional data is called online machine learning.

Scikit-learn is not designed for online learning. Scikit-multiflow is designed for online learning.


How do I visualize data for a natural language processing project? I am using a question-and-answer dataset. My neural network takes a question and an article content, and outputs where an answer starts (as an integer). To visualize my data, how should I process it and what plot(s) should I use?

I'm considering:

Word/N-gram frequency histogram for the questions. Another one for the answers.

Plots mapping word/n-gram frequency to output features

Plots mapping word/n-gram frequencies to Shannon entropy values.

On that note, maybe using a smaller machine learning model - such as a decision tree - qnd graphing the resulting probabilities.

What is the best plot for a project like mine?

I'm not an expert but let me try to think with you. What's your vocabulary size?

I think certainly starting with a small machine learning model is a good idea, but I think that a decision tree would quickly suffer with even a medium-sized vocabulary. You would need a huge tree to do anything. So I think I would start with pretrained word embeddings, and use a small neural net to predict the starting point. This helps, because words that are close in meaning have similar vectors, and the decision tree wouldn't be able to use that kind of information.

Your suggestions for histograms don't seem bad, but you would have a histogram that is as wide as your vocabulary, which seems like it defeats the purpose of visualizing it... If you went with word embeddings, how about using a technique like UMAP to plot the questions and articles in 2D?


Difference between Time series clustering and Time series Segmentation In the context of time series data mining, I have read about time series segmentation and time series clustering, but I couldn't differentiate between both. In case they are different, how these methods are correlated with each other?

Well from my understanding (please correct me if I am wrong), the segmentation is considered as a preprocessing step for the clustering phase. I mean that the segmentation step is used mainly to partition your time series data into segments, let's say into states. After that, a conventional clustering algorithm can be applied to group these segments into clusters (similar segments belong to the same cluster).

As an example, let's say that the segmentation process represents a given time series into the following segments: (S1, S2, S3, S4, S5, S6). Then after applying the segmentation process, a conventional clustering method is applied to cluster the extracted segments. So we might end up with something like this: If k = 3: then K1 {S1, S5}, K2 {S3, S6}, K3 {S2, S4}

Please correct me if I am mistaken, and provide links for more clarification if you have any.

Actually there is no fixed terminology and these two terms sometimes used in the same meaning and sometimes different. I would suggest following the terminology bellow for yourself, then you can differentiate methods according to this:

Time-Series Segmentation means partitioning an individual time series to similar segments i.e. clustering within an individual time-series (e.g. i have a video in which someone is reading a book for a while, then starts walking and then starts cycling. now I want to segment these three actions).

Suggestion: State-Space reconstruction, moving Autocorrelation, moving DTW, Fourier Analysis, Visibility Graphs or any other method which can measure the similarity of a time-series with itself.

Time-Series Clustering (or this) means finding similar time-series within a dataset of time-series (e.g. i have 10 brain signals, 5 from healthy subjects 5 from patients without knowing who is patient and who is healthy. Now I want to cluster this dataset into two clusters)

Suggestion: Build a similarity matrix between time-series using e.g. DTW and then apply Spectral Clustering (just improvised. If you search literature there should be more mature solutions)

Hope it helped :)


Isn't computing the "tractable error" in Restricted Boltzmann Machines (RBM) intractable? Let $v \in \{0,1\}^M$ be the visible layer, $h \in \{0,1\}^N$ be the hidden layer, where $M$ and $N$ are natural numbers. Given the biases $b \in \Re^M$, $c \in \Re^N$ and weights $W \in \Re^{M \times N}$, the energy and probability of an RBM is given by:

Energy, $E(v,h; b,c,W) = -b^T v - c^T h - h^T W^T v$

Probability, $p(v, h; b, c, W) = \frac{e^{-E(v,h; b,c,W)}}{Z}$

where $Z = \sum_{v,h} e^{-E(v,h; b,c,W)}$

The negative log likelihood error for a Restricted Boltzmann Machine (RBM) is given by:

$\mathcal{L}(b,c,W) = \frac{1}{T} \sum_{t=1}^{T} \left( -\log \sum_{h} e^{-E(v^t, h; b,c,W)} \right) + \log Z$

where:

$T$ is size of training dataset; and

$v^t$ represents $t^{th}$ data point in the training dataset

It is clear that computing $Z$ (and hence $\log Z$) is intractable because we have to sum over $2^{M+N}$ configurations of $v$ and $h$ - exponential time algorithm.

However, shouldn't computing $\sum_{h} e^{-E(v^t, h; b,c,W)}$ be intractable as well? Aren't we summing over all the $2^N$ configurations of $h$ here? If say $N = 64$, then we are already reaching exascale computations ($2^{64} = 1.84 \times 10^{19}$)!

Yes, calculating $\sum_{h} e^{-E(v, h)}$ does include summing over $2^N$ possible configurations of the hidden vector $h$. However you don't need to actually calculate it:

If we have the negative log likelihood function
$$ \mathcal{L}(W, c, b) = - \frac 1T \sum_{t=1}^T \log\left(p(v^{(t)})\right) $$
where $p(v^{(t)}) = \frac 1Z \sum_h e^{-E(v^{(t)},h)}$,
we don't need to calculate $\mathcal{L}(W, c, b)$ directly, instead we want to minimize it using gradient descent. Using $\theta$ as the full parameter vector instead of $W, c, b$ for simplicity,  we get:
$$\begin{array}{ll}
    \nabla\mathcal{L}(\theta)
    &\displaystyle = \frac{\partial}{\partial\theta} \left(- \frac 1T \sum_{t=1}^T \log\left(p(v^{(t)})\right)\right) \\
    &\displaystyle= \frac 1T \sum_{t=1}^T \left(
        \sum_h p(h \mid v^{(t)}) \frac{\partial E(v^{(t)},h)}{\partial\theta} - \sum_h\sum_vp(v,h)\frac{\partial E(v,h)}{\partial\theta}
    \right)
\end{array}$$
The first term of the difference above is the expected value of $\frac{\partial E(v^{(t)},h)}{\partial\theta}$, given a specific visible vector (i.e. a training example) $v^{(t)}$. This can easily be calculated for each parameter $W, c, b$.

The second term is the expected value of the energy, calculated over the full distribution of the RBM. Unfortunately, this is again intractable. Here, the contrastive divergence algorithm by Hinton comes in useful: we run Gibbs sampling for a number of steps and estimate the second term using the resulting states $\tilde{v}$.

tl;dr: yes, the problem is intractable, which is why we use the contrastive divergence algorithm by Hinton, which is only an approximation but works well in practice.


Why is Spark's LinearRegressionWithSGD very slow locally? I have been trying to run linear regression with SGD that is found in Spark mllib for some time and I am experiencing huge performance problems. All examples that I was looking have number of iterations set to 100 and say that Spark mllib can run very fast for big data.

This the code I am having problems with:

    def train(data: RDD[TrainFeature], scalerModel: StandardScalerModel): LinearRegressionModel = {
    val labeledData = data map { trainFeature =>
      LabeledPoint(trainFeature.relevance.value, toVector(trainFeature.feature, scalerModel))
    }
    labeledData.cache()

    val algorithm = new LinearRegressionWithSGD()
    algorithm.optimizer
      .setNumIterations(10)
      .setRegParam(0.01)
      .setStepSize(0.1)

    algorithm run labeledData
  }

private def toVector(feature: Feature, scalerModel: StandardScalerModel): Vector = scalerModel transform toVector(feature)

private def toVector(feature: Feature): Vector = Vectors dense feature.coordinates.toArray

I have scaled the data first and then run the algorithm to train the model. Even when I use 10 iterations it takes around 10 minutes to train model for 70,000 entries with feature vector of size 2. And the results I get are not good at all. I start getting decent results after numberOfIterations = 1000, but that would take ages.

Is it normal for linear regression with SGD to be this slow for 70,000 vectors of size 2?

My JVM min and initial memory is set to 4g.
I have tried setting the following as well (devastated try): System.setProperty("spark.executor.memory", "3g")

I am running this locally and since normal LinearRegression written in MatLab would finish the job very fast, I am wondering what am I doing wrong?

Edit: When I look at the spark UI in jobs section, I see that it is creating way too many jobs for gradient descent. Is there a way that I could tell Spark to create very little jobs - i.e. don't split data, run everything in a single thread? Maybe that can help me debug the problem further.

Spark is designed to be distributed across a cluster and use stochastic gradient descent (SGD) to optimize linear regression.
There is overhead for cluster infrastructure (even when the "cluster" is a single local node). Also, SGD is an iterative method that uses many batches to find a solution.
Given that your problem is 70k rows, it would be better to use a single node framework (e.g., scikit-learn) and ordinary least squares (OLS), a closed form solution, to optimize linear regression. Those two changes will greatly speed up training.


How does u-net work? I have read this paper about U-Net.

This kind of network is quite similar to an autoencoder, in addition it has concatenations between the encoder and the decoder parts.

I would like to know the meaning of this concatenations, and why the feature maps are cropped before this concatenation.

Excerpts from the very same source tell the answers:

In order to localize, high resolution features from the contracting path are combined with the upsampled output.

--

A successive convolution layer can then learn to assemble a more precise output based on this information.

--

The cropping is necessary due to the loss of border pixels in every convolution.

So, the combined data adds precision of U-net (reduces error) and cropping filters blurry borders out of that combined data.
Edit:

For the pixels on the border of image matrix, some elements of the kernel might stands out of the image matrix and therefore does not have any corresponding element from the image matrix.

(kernel is the matrix with which the convolution of image is done to produce the output)
Source of edit


How can I know the name of the features selected by a Deep Belief Network? I want to use DBN to reduce the 41 features of nslkdd dataset after transforming nominal data to numeric the number of features increases from 41 to 121 . I used 3 RBMs (121-50-10) now I want to know the 10 selected features i.e know their names to put them as an input to the classifier. how can I do it?

In general you are extracting/creating 10 features that scan theoretically recreate the input i.e. the 41 features. The features on their own may not necessarily make a lot of sense and (depending on the dataset) may not be easily interpretable. One could draw a comparison to Principal Component Analysis and the result components as features.

One benefit of using Deep Belief Nets to pre-train your model and encode features is that the data must not be labelled. This leads to the final point, which is that your 10 features must not necessarily have names. You can just call them e.g. feature1, feature2, ..., feature10.

If you are having problems with the actual code and the way to push the 10 features further into the classifier, I would suggest you provide the code that you have so far and add more detail regarding your exact problem.


How to perform convolution with kernel bigger than image? In this question I've seen an example of convolution by the kernel with the shape bigger than initial image's one:
import numpy as np
from scipy import signal

x = np.array([(0.51, 0.9, 0.88, 0.84, 0.05),
              (0.4, 0.62, 0.22, 0.59, 0.1),
              (0.11, 0.2, 0.74, 0.33, 0.14),
              (0.47, 0.01, 0.85, 0.7, 0.09),
              (0.76, 0.19, 0.72, 0.17, 0.57)])

y = np.array([(0, 0, 0.0686, 0),
              (0, 0.0364, 0, 0),
              (0, 0.0467, 0, 0),
              (0, 0, 0, -0.0681)])

gradient = signal.convolve2d(np.rot90(np.rot90(y)), x, 'valid')

So, we get this:

array([[ 0.044606, 0.094061], [ 0.011262, 0.068288]])

I understand that y is flipped by 180 degress. But how does "valid" convolution work here? When we can get (2x2) shape from (4x4) convolved by (5x5)?

Is your question how can we perform convolution operation with kernel bigger than input image?

If yes then answer is padding. TL;DR you increase the size of the original image with boundary pixels so that kernel can "fit"


Running an LSTM with Music Data I'm working on a project for a class where I'm trying to create an algorithm that learns music and creates its own music.

I'm having trouble on how to set up the data for it to be inputted into the LSTM.

A single training example consists of a chord that is a vector of binary values based on what keys are pressed in MIDI form (indices 0-127), a value that denotes duration of the note, beat strength, numerator of the time signature, and denominator of the time signature, and the key signature represented by the number of flats

So one example might look like

$$\left[ \begin{array}{c} {0} \\ {1} \\{0} \\{1} \\{\vdots} \\ {1} \\ {0} \\ {4} \\ {3} \\ {4} \\ {4} \\ {-2}   \end{array} \right]$$

The result is a 132x1 vector

I was having trouble conceptualizing how to input this data type into an LSTM. Doing a linear output would not make that much sense, but I don't think I can directly one-hot this vector either.

You should ask yourself - are you teaching an algorithm to play chords or to play music? In addition, what are you trying to predict here?

It seems to me that you need to create input data that is a series of chords and your label is the next chord in the tune. So you should design a neural network that takes in a series of chords and can tell you the next chord in the sequence, add that back to the input sequence and pick the next chord, add that back to the input sequence, etc, etc. Next thing you know, you have a neural network that can play music.


Calculate weighted mean for two columns and hundreds of rows? I'm trying to teach myself basics of R and I couldn't find the answer:

Say, I have a csv file and I want to calculate weighted mean for each subject such that I have a mean mu = 0.015*1030+0.16*26930+0.24*0+0.87*250+0.29*310+0.77*6240+0.98*3730+0.98*0+0.08*1400 for Subject A, for example.

How would I accomplish it in R or Excel?

Also, if I need to multiply each pair by corresponding value from the same row from the column salary:

0.015*1030*111+0.16*26930*222+0.24*0*333+0.87*250*444+0.29*310*555+0.77*6240*666+0.98*3730*777+0.98*0*888+0.08*1400*999

Thanks

UPDATE:

Given your data is in DataFrame called df, you can simply do this:

mu = sum(df$Probability * df$SubjectA * df$salary)


Regression error increase after shuffing data I'm trying to do multivariate regression using a 3-dimension data set. I noticed a strange problem that my fitting error increase dramatically after I pre-shuffled the data matrix comparing using original data. Is that possible? I've checked my code and confirmed that it's the shuffled process that increased the error. Why is that? Shouldn't shuffle process only change the order of the data?

You need to shuffle the whole dataset together before separating the features (X) and target variable (y). This is the only reason I can think of for getting this error.


Why do we normalize the discounted rewards when doing policy gradient reinforcement learning? I'm trying to understand the policy gradient approach for solving the cartpole problem.
In this approach, we're expressing the gradient of the loss w.r.t each parameter of our policy as an expectation of the sum of gradients of our policy gradient for all actions in a sequence, weighted by the sum of discounted rewards in that sequence:

$$\nabla_\theta L(\theta) = E[ G(S_{0:T}, A_{0:T})\sum_{t=0}^{T}\nabla_\theta log\pi_\theta (A_t|S_t) ]$$

and we estimate it using an empirical average across all samples in an episode - which makes sense intuitively.

BUT the less intuitive part is that I saw a common practice to normalize advantages in between episodes in several implementations (and indeed it works better). So after they calculate the they wouldn't directly use the advantage, but rather would normalize it, e.g. here they do after every episode:

discounted_epr = discount_rewards(epr)
discounted_epr -= np.mean(discounted_epr)
discounted_epr /= np.std(discounted_epr)

what's the justification for that - both in theory and in intuition? It seems to me that if an episode is long and as such has large advantages, it's worth learning more from that episode than from a 3 moves episode. What am I missing?

In general we prefer to normalize the returns for stability purposes. If you work out the backpropagation equations you will see that the return affects the gradients. Thus, we would like to keep its values in a specific convenient range. We don't follow this practice for theoretical guarantees but for practical reasons. The same goes with clipping $Q$ value functions in Q-learning combined with NNs. Of course, there are some drawbacks with these approaches but in general the algorithm behaves better as the backpropagation does not lead your network weights to extreme values. Please take a look at this excellent post by Andrej Karpathy (I attach the part related to your question as a blockquote) which gives additional insights:

  More general advantage functions. I also promised a bit more discussion of the returns. So far we have judged the goodness of every individual action based on whether or not we win the game. In a more general RL setting we would receive some reward $r_t$ at every time step. One common choice is to use a discounted reward, so the “eventual reward” in the diagram above would become $R_t=∑^∞_{k=0}γ^kr_{t+k}$, where $\gamma$ is a number between 0 and 1 called a discount factor (e.g. 0.99). The expression states that the strength with which we encourage a sampled action is the weighted sum of all rewards afterwards, but later rewards are exponentially less important. In practice it can can also be important to normalize these. For example, suppose we compute $R_t$ for all of the 20,000 actions in the batch of 100 Pong game rollouts above. One good idea is to “standardize” these returns (e.g. subtract mean, divide by standard deviation) before we plug them into backprop. This way we’re always encouraging and discouraging roughly half of the performed actions. Mathematically you can also interpret these tricks as a way of controlling the variance of the policy gradient estimator. A more in-depth exploration can be found here.


Calculate number of parameters for ConvLSTM2D layer time_distributed_24 (TimeDis (None, 16, 64, 64, 512)   0

conv_lst_m2d_2 (ConvLSTM2D)  (None, 16, 64, 64, 128)   2949632

time_distributed_25 (TimeDis (None, 16, 64, 64, 128)   512

time_distributed_26 (TimeDis (None, 16, 128, 128, 128) 0

For example, why does this ConvLSTM2D layer have 2949632 parameters?
A standard LSTM layer has 4(n*m + n^2 + n) parameters where m = input dim, n = output dim.

A standard 3x3 conv layer with 128 kernels over a 64x64x512 tensor would have 3x3x512x128 parameters

What would be the n and m for the LSTM part?

i=512, input dimension or input channel

h=128, output dimension or output channel

k=3, kernel size

number of parameters = 4 * h * ( k**2 * (i+h) + 1 )  = 2949632

if k == 1, ConvLSTM is just a LSTM.

You can check the original paper for detail.
https://arxiv.org/abs/1506.04214