text:
Why is the state-action value function used more than the state value function?
In reinforcement learning, the state-action value function seems to be used more than the state value function. Why is it so?

What do you mean by "more"? Btw, if I understand your question correctly, it is because in a non-deterministic setting, you won't know which action to select simply from the state values.

@HaiNguyen thanks. What happens in the case of deterministic setting

If you know what state you will end up after taking an action (for example if you walk in a maze you always know your next position after you take a step), you can use the value function of possible next states and pick the action that leads to the highest value.

We are ultimately interested in getting an optimal policy, that is the optimal sequence of actions to reach the final goal. State values on its own don't provide that, they tell you expected return from specific state onward but they don't tell you which action to take. In order to derive an optimal action in a specific state you would have to simulate all possible actions one step ahead and then pick the action that leads you to the state with highest state value. That is often inconvenient or impossible. State action values connect the expected return with actions, not states, so you don't need to simulate all actions one step ahead and see where you end up, you only need to pick an action that has the highest value because you know that is the best action to take in that state.

It may be also worth emphasizing that, if you're using a table, the state-action value function will be larger than the state value function, so it will require more memory to be stored.

what did you mean by simulate all possible actions one step ahead?


text:
Why does the KL divergence not satisfy the triangle inequality?
The KL divergence is defined as
$$D_{KL}=\sum_i p(x_i)log\left(\frac{p(x_i)}{q(x_i)}\right)$$
Why does $D_{KL}$ not satisfy the triangle inequality?
Also, can't you make it satisfy the triangle inequality by taking the absolute value of the information at every point?

To prove that the KL divergence does not satisfy the triangle inequality, you just need a counterexample.
Definitions
KL divergence
Let's first recapitulate the definition of KL divergence for discrete probability distributions $p$ and $q$ (for simplicity).
$$
D_{\text{KL}}(p\parallel q)
=
\sum_{x\in {\mathcal {X}}}
p(x)\log
\left(
\frac {p(x)}{q(x)}
\right)
$$
Triangle inequality
Let's also recap the definition of the triangle inequality for distance $d$, which can be expressed as
$$
d(p,q) \leq d(p, r) + d(r, q),
$$
where $p, q$ and $r$ are probability distributions.
Proof
For simplicity, let the sample space be $\mathcal{X} = \{0, 1 \}$. Consider now the following three discrete probability distributions.
$$
p(x)=
\begin{cases}
\frac{1}{2}& \text{if } x = 0\\
\frac{1}{2}& \text{if } x = 1
\end{cases}
$$
$$
r(x)=
\begin{cases}
\frac{1}{4}& \text{if } x = 0\\
\frac{3}{4}& \text{if } x = 1
\end{cases}
$$
$$
q(x)=
\begin{cases}
\frac{1}{10}& \text{if } x = 0\\
\frac{9}{10}& \text{if } x = 1
\end{cases}
$$
Let's now apply the definition of the triangle inequality and the KL divergence for these discrete probability distributions. So, we want to show the following inequality does not hold.
$$
D_{\text{KL}}(p\parallel q)
\leq
D_{\text{KL}}(p\parallel r)
+
D_{\text{KL}}(r\parallel q)
$$
which can be expanded to
\begin{align}
\sum_{x\in \{0, 1 \}}
p(x)\log
\left(
\frac {p(x)}{q(x)}
\right)
&\leq
\sum_{x\in \{0, 1 \} }
p(x)\log
\left(
\frac {p(x)}{r(x)}
\right)
+
\sum_{x\in \{0, 1 \}}
r(x)\log
\left(
\frac {r(x)}{q(x)}
\right)
\end{align}
which roughly corresponds to
\begin{align}
0.51 &\leq 0.14 + 0.09 \\
0.51 &\leq 0.24
\end{align}
which is clearly false.
Given that the triangle inequality does not hold in one case, it doesn't hold in all cases, so the triangle inequality does not hold for the KL divergence.
$$\tag*{$\blacksquare$}$$
For reproducibility, I have used the following Python (3.7) program to compute the KL divergences.
from math import log

one = (1 / 2 * log((1 / 2) / (1 / 10))) + (1 / 2 * log((1 / 2) / (9 / 10)))
two = (1 / 2 * log((1 / 2) / (1 / 4))) + (1 / 2 * log((1 / 2) / (3 / 4)))
three = (1 / 4 * log((1 / 4) / (1 / 10))) + (3 / 4 * log((3 / 4) / (9 / 10)))

print(one)
print(two)
print(three)
print(two + three)
print(one <= two + three)

There may be cases where it holds, but a single counter example is a sufficient disproof on its own.

Is there any variant of the triangle inequality that does hold?

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What is the purpose of the arrow $\leftarrow$ in this formula?
What is the purpose of the arrow $\leftarrow$ in the formula below?
$$V(S_t) \leftarrow V(S_t) + \alpha \left[ G_t - V(S_t) \right]$$
I presume it's not the same as 'equals'.

I assume this is an iterative function. It means the current $V(S_t)$ is the sum of the previous plus some adjustment. The arrow is like an assignment.
In code, you would do
vst = vst + alpha * (gt - vst)

So vst will be overwritten.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How should I penalize the model proportionally to the error?
I am making an MNIST classifier. I am using categorical cross-entropy as my loss function. I want to make it so that if the correct label is 3, then it will penalize the model less heavily if it classifies a 4 than a 7 because 4 is closer numerically to 3 than 7 is. How do I do this?

I want to make it so that if the correct label is 3, then it will penalize the model less heavily if it classifies a 4 than a 7 because 4 is closer numerically to 3 than 7 is. How do I do this?

Really you should not, because the symbols used (Arabic numerals) do not have direct relation to quantity in the same way e.g. tally counts or dots do. They are good candidates for classification, and despite the conventional mapping to quantity when you read them, the symbols themselves are poor candidates for regression, because for instance the symbols $3$ and $4$ do not differ in a way that captures quantity in any intuitive manner.
However, if you are keen to do this, it is relatively simple to construct a suitable loss function in most auto-differentiating frameworks. You will need to read up on how to do so. For instance, here is a Stack Overflow answer explaining where to start with writing custom loss function in Keras.
In order for your loss function to work, it will need to be differentiable and  smoothly changing as predictions get better. That rules out using any form of argmax for the current prediction. If you want to stick with softmax for the final layer, then I suggest using a mean squared error against the expected prediction, e.g. if $d_i$ is the numerical digit for example $i$ and $y_{i,j}$ is the ground truth expressed as a one-hot vector, where $i$ is the example and $j$ is the digit class, then $\hat{y}_{i,j}$ is the probability predicted by your model. You could use $\hat{d}_i = \sum_{j=0}^9 j\hat{y}_{i,j}$ for the expected value and MSE loss of $\mathcal{L}(d_i,\hat{d}_i) = \frac{1}{2}(\hat{d}_i - d_i)^2$
You can also use a weighted sum of the MSE loss and cross entropy loss as your final loss, with the balance between the two losses being a new hyperparameter of your model.
Note this solution makes $0$ close to $1$ but far away from $9$. If you want the digits to be considered close on a cycle (e.g. $8$ is closer to $1$ than it is to $4$) the you will need something more creative.
Whilst I don't think this will help you discover any improvements to MNIST classification, combining two or more loss functions to achieve a more complex goal can be really useful sometimes, so it is a skill worth practicing.


text:
What is an end-to-end AI project?
I often read about the so-called end-to-end AI (or analytics) projects, but I couldn't find a definition of it. What is an end-to-end AI project?  Can someone explain what is meant/expected when someone asks you "Have you already implemented an end-to-end AI project"?

Here's a good source for what it means in deep learning: https://www.youtube.com/watch?v=ImUoubi_t7s

Hi and welcome to AI SE! Can you please provide links to the resources where you read this expression? The context will definitely help to answer this question.

It's a pretty grey area.
I think they are usually talking about the full spectrum of the effort in taking an "AI project" all the way from the drawing board to production. This could easily be anything from a Stock Market Price Prediction model, to deploying a customer service chatbot/Virtual Assistant. I find that people who ask such questions only care about the fact that "AI" is being used to solve a problem and actually is working in the real world.
The effort in the Model would be the standard DataScience/Machine learning process of obtaining data, feature engineering, training and refining your model and eventually deploying it somewhere where it works. It could be in a web browser, it could be in an app, an API endpoint, etc.
For the customer service chatbot it could easily be largely a pure programming project utilising a cloud provider's resource for to conversational capabilities, but as long as it falls under the bracket of "AI" and has been completed and deployed to production/an environment where it works, its end to end.
Disclaimer: Depending on the technical knowledge of who is asking, they might care that you have done some real machine learning work in the development process, and so the example of the bot wouldn't fly.
Hope this helps somehow.


text:
Why is the space-complexity of greedy best-first search is $\mathcal{O}(b^m)$?
I am reading through Artificial Intelligence: Modern Approach and it states that the space complexity of the GBFS (tree version) is $\mathcal{O}(b^m)$.
While I am reading, at some points, I found GBFS similar to DFS. It expands the whole branches and goes after one according to the heuristic function. It doesn't expand the rest like BFS. Perceiving this as similar to what depth-first search does, I understand that the worst time complexity is $\mathcal{O}(b^m)$. But I don't understand the space complexity.
Shouldn't it be the same as DFS, $\mathcal{O}(bm)$, since it only will be expanding $b*m$ nodes during the search in one path?

After spending some time on the problem, I concluded that it is due to the fact that we need to store the heuristic function evaluations for all nodes during the traversal. So, one might claim that it is the space complexity of the whole nodes which is simply $\mathcal{O}(b^m)$. I hope this is correct.
Also the one with having the space complexity of $\mathcal{O}(bm)$ is called recursive best-first search which is the one that is most similar to DFS implementation I descried in the question.

Have you any reference for this?

I was struggling with the same question. This is what I came up with after thinking it through.
With depth-first-search, you backtrack to a node that is a non-expanded child of your parent (or the parent of the parent when your parent has no more non-expanded children (and so on going up the tree)). So the space complexity is limited by your ancestors and the children of these ancestors. Which translates in m*b where m is the max path length (so max number of ancestors) and b is the branching factor (number of children per ancestor).
With greedy search when you backtrack you can jump to any evaluated but unexpanded node, you passed going down on paths earlier. So the algorithm, when backtracking, can make pretty random jumps throughout the tree leaving lots of sibling nodes unexpanded. You will have to remember the value of the evaluation function for all these unexpanded nodes though because possibly they are next up when backtracking occurs. So in a theoretical very worst-case scenario that could mean that almost the whole tree needs to be remembered. Hence O(b^m).
I know there are still gaps in the reasoning but intuitively this makes me understand it best.


text:
How many ways are there to perform image segmentation?
I'm new in Artificial Intelligence and I want to do image segmentation.
Searching I have found these ways

Digital image processing (I have read it in this book: Digital Image Processing, 4th edition)
Convolutional neural networks

Is there something else that I can use?

Apart from the multitudes of traditional image segmentation techniques (Watershed, Clustering or Variational methods), newer Segmentation schemes using Deep Learning are actively being used, which provide better results and are better for real-time applications, owing to minimum computation overheads involved.
The following blog provides a detailed review of recent advancements in this field: Review of Deep Learning Algorithms for Image Semantic Segmentation
For the traditional methods, this Wikipedia article provides a nice summary:
Image Segmentation


text:
Why is exp used in encoder of VAE instead of using the value of standard deviation alone?
There's one VAE example here:
https://towardsdatascience.com/teaching-a-variational-autoencoder-vae-to-draw-mnist-characters-978675c95776.
And the source code of encoder can be found at the following URL: https://gist.github.com/FelixMohr/29e1d5b1f3fd1b6374dfd3b68c2cdbac#file-vae-py.
The author is using $e$ (natural exponential) for calculating values of the embedding vector:
$$z = \mu + \epsilon \times e^{\sigma}$$
where $\mu$ is the mean, $\epsilon$ a small random number and $\sigma$ the standard deviation.
Or in code
z  = mn + tf.multiply(epsilon, tf.exp(sd))

It's not related to the code (practical programming), but why using natural exponential instead of:
$$z = \mu + \epsilon \times \sigma$$

In the source code, the author defines sd by
sd       = 0.5 * tf.layers.dense(x, units=n_latent)

which means that $\operatorname{sd}\in \mathbb{R}^n$.
In particular, the support over sd includes negative numbers, which is something we want to avoid.
Since standard deviations are always nonnegative, we can exponentiate to get us in the correct domain.
This is a case where the variable is inappropriately named.
Here, sd is not the standard deviation itself but rather the logarithm of the standard deviation. This allows it to be predicted as the output of a layer in a neural network, so extracting the predicted value of the standard deviation would require exponentiation.

hm yes, tks, that sd is ln of sd, not sd


text:
Why does the error of my LSTM not decrease after 10 epochs?
Despite the problem being very simple, I was wondering why an LSTM network was not able to converge to a decent solution.
import numpy as np
import keras

X_train = np.random.rand(1000)
y_train = X_train
X_train = X_train.reshape((len(X_train), 1, 1))

model= keras.models.Sequential()
model.add(keras.layers.wrappers.Bidirectional(keras.layers.LSTM(1, dropout=0., recurrent_dropout=0.)))
model.add(keras.layers.Dense(1))

optimzer = keras.optimizers.SGD(lr=1e-1)

model.build(input_shape=(None, 1, 1))
model.compile(loss=keras.losses.mean_squared_error, optimizer=optimzer, metrics=['mae'])
history = model.fit(X_train, y_train, batch_size=16, epochs=100)

After 10 epochs, the algorithm seems to have reached its optimal solution (around 1e-4 RMSE), and is not able to improve further the results.
A simple Flatten + Dense network with similar parameters is however able to achieve 1e-13 RMSE.
I'm surprised the LSTM cell does not simply let the value through, is there something I'm missing with my parameters? Is LSTM only good for classification problems?

I think there are some problems with your approach.
Firstly, looking at the Keras documentation, LSTM expects an input of shape (batch_size, timesteps, input_dim). You're passing an input of shape (1000, 1, 1), which means that you're having "sequences" of 1 timestep.
RNNs have been proposed to capture temporal dependencies, but it's impossible to capture such dependencies when the length of each series is 1, and the numbers are randomly generated. If you want to create a more realistic scenario, I would suggest you generate a sine wave, since it has a smooth periodic oscillation. Afterward, increase the timesteps from 1, and you can test on the following timestamps (to make predictions).
For the second part, if you think about a normal RNN (I will explain for a simple RNN but you can imagine a similar flow for LSTM) and a Dense layer when applied to 1 timestamp, there are not so many many differences. The dense layer will have $Y=f(XW + b)$, where $X$ is the input, $W$ is the weight matrix, $b$ is the bias and $f$ is the activation function. Whereas RNN will have $Y=f(XW_1 + W_2h_0 + b)$, since is the first timestamp $h_0$ is $0$, so we can reduce it to $Y=f(XW_1 +b)$, which is identical with the Dense layer. I suspect that the results differences are caused by the activation functions, by default Dense layer has no activation function, and LSTM has tanh and sigmoid.


text:
Why do CNN's sometimes make highly confident mistakes, and how can one combat this problem?
I trained a simple CNN on the MNIST database of handwritten digits to 99% accuracy. I'm feeding in a bunch of handwritten digits, and non-digits from a document.
I want the CNN to report errors, so I set a threshold of 90% certainty below which my algorithm assumes that what it's looking at is not a digit.
My problem is that the CNN is 100% certain of many incorrect guesses. In the example below, the CNN reports 100% certainty that it's a 0. How do I make it report failure?

My thoughts on this:
Maybe the CNN is not really 100% certain that this is a zero. Maybe it just thinks that it can't be anything else, and it's being forced to choose (because of normalisation on the output vector). Is there any way I can get insight into what the CNN "thought" before I forced it to choose?
PS: I'm using Keras on Tensorflow with Python.
Edit
Because someone asked. Here is the context of my problem:
This came from me applying a heuristic algorithm for segmentation of sequences of connected digits. In the image above, the left part is actually a 4, and the right is the curve bit of a 2 without the base. The algorithm is supposed to step through segment cuts, and when it finds a confident match, remove that cut and continue moving along the sequence. It works really well for some cases, but of course it's totally reliant on being able to tell if what it's looking at is not a good match for a digit. Here's an example of where it kind of did okay.

My next best option is to do inference on all permutations and maximise combined score. That's more expensive.

@nbro Just so you know what I've tried. I removed the softmax from the last dense layer so I can see the activations before normalization. It's defintely giving me richer information that I can use for error reporting. I think that maybe if I then apply softmax and use a condition which combines the two metrics, I can improve my error report. In any case, let me know what your answer is when you have time :)

I need to think about your solution and workaround, but I would have given an answer similar to https://ai.stackexchange.com/a/17722/2444. There's an issue with traditional neural networks, as they are programmed to learn a point estimate so they do not usually model uncertainty.

This is very obviously a 2... It's just that the actual digit is a 3D object, and it's being viewed at an unusual angle.

Going from left to right and also the other way round might help, sometimes. Sometimes, there may be a clearly visible digit on one end but not the other, so some meet-in-the-middle might help. This obviously doesn't solve the problem, but it's not too expensive and might be an idea to try.

Can you share your code for this? I'm looking to obtain confidence metrics for my CNN classifier as well as make that subplot that you have that displays all the images with their predicted and actual labels. Greatly appreciate it!

The concept you are looking for is called epistemic uncertainty, also known as model uncertainty. You want the model to produce meaningful calibrated probabilities that quantify the real confidence of the model.
This is generally not possible with simple neural networks as they simply do not have this property, for this you need a Bayesian Neural Network (BNN). This kind of network learns a distribution of weights instead of scalar or point-wise weights, which then allow to encode model uncertainty, as then the distribution of the output is calibrated and has the properties you want.
This problem is also called out of distribution (OOD) detection, and again it can be done with BNNs, but unfortunately training a full BNN is untractable, so we use approximations.
As a reference, one of these approximations is Deep Ensembles, which train several instances of a model in the same dataset and then average the softmax probabilities, and has good out of distribution detection properties. Check the paper here, in particular section 3.5 which shows results for OOD based on entropy of the ensemble probabilities.

BNNs help, but it should be noted that humans have this problem too (and there's a good chance our brains use BNNs). There will always be things that are obviously true for two different systems, even though both "100% certain" (sigh) results are different - the 2015 blue-black/gold-white dress phenomenon should serve as a great example.

You can also get an ensemble from a single model if you leave dropout on at inference time: http://mlg.eng.cam.ac.uk/yarin/blog_3d801aa532c1ce.html

@Davidmh Actually thats not really an ensemble, and I did not mention it because MC Dropout does not have good out of distribution detection properties, you can see that in the Deep Ensembles paper I linked.

MC dropout is just an approximation of a BNN (or deep Gaussian process). I think it's worth mentioning it because it can easily be applied (e.g. Keras already provides an implementation of dropout that can be enabled at inference time), so people can easily use it.

@AlexanderSoare I don't think that works, because specifying the "other" class is pretty much any class, its not bounded (open set), so you will always have trouble finding enough data to match a Bayesian model (which models the opposite proble, which is bounded).

Decent related paper: On Calibration of Modern Neural Networks

I think this is not correct. Since the model has to choose, it is actually correct to say it is 100% more likely to be a 0 than anything else. I don't believe that property changes when using BNN. If the choose would against another valid number, I believe confidence would help.

@PascalVKooten I am not sure what you mean by "this is not correct", a BNN effective makes out of distribution predictions with a more flat distribution (which has higher entropy) indicating that it does not know the answer, or even hinting that there are many possible answers. A non-BNN model makes overconfident predictions that are generally wrong.

Your classifier is specifically learning the ways in which 0s are different from other digits,
not what it really means for a digit to be a zero.
Philosophically, you could say the model appears to have some powerful understanding when restricted
to a tightly controlled domain, but that facade is lifted as soon as you throw any sort of wrench in
the works.
Mathematically, you could say that the model is simply optimizing a classification metric for data
drawn from a specific distribution, and when you give it data from a different distribution, all
bets are off.
The go-to answer is to collect or generate data like the data you expect the model to deal with
(in practice, the effort required to do so can vary dramatically depending upon the application). In
this case, that could involve drawing a bunch of random scribbles and adding them to your training
data set. At this point you must ask, now how do I label them? You will want a new "other" or
"non-digit" class so that your model can learn to categorize these scribbles separately from digits.
After retraining, your model should now better deal with these cases.
However, you may then ask, but what if I gave it color images of digits?  Or color images of farm
animals?  Maybe pigs will be classified as zeros because they are round.  This problem is a
fundamental property of the way deep learning is orchestrated.  Your model is not capable of higher
order logic, which means it can seem to go from being very intelligent to very dumb by just throwing
the slightest curve ball at it.  For now, all deep learning does is recognize patterns in data that
allow it to minimize some loss function.
Deep learning is a fantastic tool, but not an all-powerful omnitool.  Bear in mind its limitations
and use it where appropriate, and it will serve you well.

@Fnguyen yes totally agree there. Why wouldn't it at least think that maybe it's looking at something else. Although I'm not really sure how overfitting would cause this issue. Unless there were many zeros in the training set that looked like this image. For reference I used MNIST, then post trained on a set of 3000 digits with slightly different handwriting styles (turns out South Koreans write digits a bit different than westeners).

Broken assumptions
Generalization relies on making strong assumptions (no free lunch, etc). If you break your assumptions, then you're not going to have a good time. A key assumption of a standard digit-recognition classifier like MNIST is that you're classifying pictures that actually contain a single digit. If your real data contains pictures that have non-digits, then that means that your real data is not similar to training data but is conceptually very, very different.
If that's a problem (as in this case) then one way to treat that is to explicitly break that assumption and train a model that not only recognizes digits 0-9 but also recognizes whether there's a digit at all, and is able to provide an answer "that's not a digit", so a 11-class classifier instead of a 10-class one. MNIST training data is not sufficient for that, but you can use some kind of 'distractor' data to provide the not-a-digit examples. For example, you could use some dataset of letters (perhaps omitting I, l, O and B) transformed to look similar to MNIST data.

Apollys,
That's a very well thought out response. Particularly, the philosophical  discussion of the essence of "0-ness."
I haven't actually performed this experiment, so caveat emptor... I wonder how well an "other" class would actually work. The ways in which "other" differs from "digit" has infinite variability (or at least its only limitation is the cardinality of the input layer).
The NN decides whether something is more of one class or more of a different class. If there isn't an essence in common among other "non-digits", I don't believe it will do well at identifying "other" as the catch-all for everything that has low confidence level of classification.
This approach still doesn't identify what it is to be "not-digit". It identifies how all the things that are "other" differ from the other labeled inputs -- probably poorly, depending on the variability of the "non-digit" labeled data. (i.e. is it numerically exhaustive, many times over, of all random scribbles?) Thoughts?

This answer looks like it should be a comment, but it looks like you don't have enough reputation to comment. You brought up some great concerns about the effectiveness of adding an "other" class. It can be really hard to predict what will or won't work well with neural networks.

Totally agree here. I was thinking something along the lines when I saw the suggestion. @MatiasValdenegro (don't know if tagging you here works), I was wondering what your thoughts on such an approach might be.

Yes, this is an important point.  Once again, the "other" class will have to come from some specific distribution and have enough samples to cover that distribution.  That means if you really want your model to learn what it means to be a digit rather than anything else that is possible, you will need a (probably intractably) lot of data.  In this way, we see that deep learning is largely about understanding and confining the problem domain very well.

Point from practical experience: having one other class may work, but in some cases it can help to have multiple. I'm not aware of a method to determine up front what the best number of "other" classes is.

An approach may be to use/train some form of binary classifier to discard non-digit examples prior to the recognition step (i.e., 0-9 classification). You'll have to maximise recall of the digit inputs, so there may still be a few non-digits slip through, but such an approach may reduce the detriment to the performance of classifying the real digits.

I'm an amateur with neural networks, but I will illustrate my understanding of how this problem comes to be.
First, lets see how trivial neural network classifies 2D input into two classes :

But in case of complex neural network, the input space is much bigger and the sample data points are much more clustered with big chunks of empty space between them:

The neural network then doesn't know how to classify the data in the empty space, so something like this is possible :

When using the traditional ways of measuring quality of neural networks, both of these will be considered good. As they do classify the classes themselves correctly.
Then, what happens if we try to classify these data points?

Really, neural network has no data it could fall back on, so it just outputs what seems to us as random nonsense.

In your particular case, you could add a eleventh category to your training data: "not a digit".
Then train your model with a bunch of images of incorrectly segmented digits, in addition to the normal digit examples. This way the model will learn to tell apart real digits from incorrectly segmented ones.
However even after doing that, there will be an infinite number of random looking images that will be classified as digits. They're just far away from the examples of "not a digit" you provided.


text:
Is there a reason to choose regular momentum over Nesterov momentum for neural networks?
I've been reading about Nesterov momentum from here and it seems like a nice improvement over regular momentum with no extra cost whatsoever.
However, is this really the case? Are there instances where regular momentum performs better than Nesterov momentum or is Nesterov momentum performs at least as good as the regular momentum all the time?

The book Deep Learning by Goodfellow, Bengio, and Courville says (Sec 8.3.3, p 292 in my copy) states that

Unfortunately, in the stochastic gradient case, Nesterov momentum does not improve the rate of convergence.

I'm not sure why this is, but the theoretical advantage depends on a convex problem, and from this, it sounds like the practical advantage does too - or at least, that it isn't applicable to typical neural network landscapes.
Perhaps it can be implemented more efficiently, but it seems to me that you need to do parameter updates twice (in order to calculate the gradient in a place you aren't moving two) and thus Nesterov requires more computation and memory than plain ole momentum.

It depends what you mean by "improving" convergence - finding a better optimum or getting there faster.  I think you mean the former (and are mostly correct, but there may be exceptions which are probably hard to quantify), while Goodfellow et al mean the latter (the following discussion discusses rate of convergence).

It's frustrating that the book makes this claim without any citation.

Let $y_t = x_t + \beta_t (x_t - x_{t-1})$.
Nesterov momentum is $x_{t+1} = y_t - \alpha_t f'(y_t)$.
Classical momentum is $x_{t+1} = y_t - \alpha_t f'(x_t)$.
Isn't that the same number of additions/subtractions?


text:
In reinforcement learning, is the value of terminal/goal state always zero?
Let's assume we are in a $3 \times 3$ grid world with states numbered as $0,1, \dots, 8$. Suppose that the goal state is $8$, the reward of landing in the goal state is $10$, and the reward of just wandering around in the grid world is $0$. Is the state-value of state $8$ always $0$?

In reinforcement learning, is the value of terminal/goal state always zero?

Yes, always for episodic problems, the value of a terminal state is always zero, from the definition.
The value of a state $v(s)$ is the expected sum (perhaps discounted) of rewards from all future time steps. There are no future time steps when in a terminal state, so this sum must be zero.
For the sake of consistent maths notation, you can consider a terminal state to be "absorbing", i.e. any transition out of it results in zero reward and returning to the same terminal state. Then you can use the definition of value function to show the same thing:
$$v_{\pi}(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_{t} = s]$$
If $s = s_T$, the terminal state, then all the "future rewards" from $k=0$ onwards starting with reward $R_{t+1}$ must be zero. This is consistent with the reward $R_{t}$, i.e. the reward when transitioning to the terminal state, being any value.
You can show similar using action value functions, if you accept a "null" action in the terminal state.

To be clearer, the value of the terminal state is zero by definition of the terminal state. In theory, if you keep on receiving reward once you're in the terminal state, then the value won't be zero.

@nbro: If you keep on receiving reward once you're in a "terminal state" then it wasn't a terminal state. A terminal state is where the MDP stops. I think that is the more fundamental definition, the rest is consequences and book-keeping around that. The use of "absorbing" states is an example of that book keeping, and it is used so that some of the theory for continuous and episodic MDPs can use the same notation and derivations.


text:
Why is the evidence equal to the KL divergence plus the loss?
Why is the equation $$\log p_{\theta}(x^1,...,x^N)=D_{KL}(q_{\theta}(z|x^i)||p_{\phi}(z|x^i))+\mathbb{L}(\phi,\theta;x^i)$$ true, where $x^i$ are data points and $z$ are latent variables?
I was reading the original variation autoencoder paper and I don't understand how the marginal is equal to the RHS equation. How does the marginal equal the KL divergence of $p$ with its approximate distribution plus the variational lower bound?

In variational inference, the original objective is to minimize the Kullback-Leibler divergence between the variational distribution, $q(z \mid x)$, and the posterior, $p(z \mid x) = \frac{p(x, z)}{\int_z p(x, z)}$, given that the posterior can be difficult to directly infer with the Bayes rule, due to the denominator term, which can contain an intractable integral.
Therefore, more formally, the optimization objective can be written as
\begin{align}
q^*(z \mid x) = \operatorname{argmin}_{q(z \mid x)} D_{\text{KL}}(q(z \mid x) \| p(z \mid x))\tag{1} \label{1}
\end{align}
However, solving this optimization problem can be as difficult as the original inference one of computing the posterior $p(z \mid x)$ using the Bayes rule, given that it still involves the possibly intractable term $p(z \mid x)$.
If you use the definition of the KL divergence, you can derive the following equation
\begin{align}
D_{\text{KL}}(q(z \mid x) \| p(z \mid x)) = \mathbb{E}_{q(z \mid x)} \left[ \log q(z \mid x) \right] - \mathbb{E}_{q(z \mid x)} \left[ \log q(z, x) \right] + \log p(x) \tag{2} \label{2}
\end{align}
First, note that the expectations are with respect to the variational distribution, which means that, if you want to approximate these expectations with Monte Carlo estimates, you can do it with respect to the variational distribution, and, given that it is assumed that one can easily sample from the variational distribution (which can e.g. be a Gaussian), this is a nice feature.
Second, the KL divergence contains the term $p(x) = \int_z p(x, z)$, the denominator term in the Bayes rule to compute the posterior $p(z \mid x)$, which (as I said) can be intractable. $p(x)$ is often called the evidence.
The solution is then to optimize an objective that does not contain this annoying intractable term $p(x)$. The objective that is optimized is the so-called ELBO objective
\begin{align}
\text{ELBO}(q) = \mathbb{E}_{q(z \mid x)} \left[ \log q(z, x) \right] - \mathbb{E}_{q(z \mid x)} \left[ \log q(z \mid x) \right]\tag{3} \label{3}
\end{align}
The KL divergence \ref{2} and the ELBO objective \ref{3} are similar. In fact, ELBO is an abbreviation for Evidence Lower BOund, because the ELBO is a lower bound on the evidence $p(x)$, i.e. it is a number that is smaller than $p(x)$ or, more formally, $\text{ELBO}(q) \leq \log p(x)$. Therefore, if we maximize $\text{ELBO}(q)$, we also maximize the evidence $p(x)$ of the data (where $x$ is the data in your dataset).
So, the objective in variational inference is
\begin{align}
q^*(z \mid x)
&= \operatorname{argmax}_{q(z \mid x)} \operatorname{ELBO}({q}) \\
&= \operatorname{argmax}_{q(z \mid x)}  \mathbb{E}_{q(z \mid x)} \left[ \log q(z, x) \right] - \mathbb{E}_{q(z \mid x)} \left[ \log q(z \mid x) \right]
\tag{4} \label{4}
\end{align}
First, note that \ref{4} only contains terms that depend on the variational distribution, so we got rid of intractable terms, which was our goal.
Second, note that, as opposed to \ref{1}, we are maximizing (or finding the parameters that maximize the objective).
The ELBO objective is actually the negative of \ref{2} plus the logarithm of the evidence term, $\log p(x)$ (and you can easily verify it), that is
\begin{align}
\text{ELBO}(q)  = -D_{\text{KL}}(q(z \mid x) \| p(z \mid x)) + \log p(x)
\end{align}
which can also be rearranged as
\begin{align}
\log p(x) = D_{\text{KL}}(q(z \mid x) \| p(z \mid x)) + \text{ELBO}(q) \tag{5}\label{5}
\end{align}
which is your equation (where $\text{ELBO}(q)$ is your $\mathcal{L}$). Therefore, your equation is true by definition, i.e. we define the ELBO such that \ref{5} is true. However, note that we haven't defined the ELBO in the way we dit only for the sake of it, but because it is a lower bound on the log evidence (and this follows from the fact that the KL divergence is never negative).


text:
Which is a better form of regularization: lasso (L1) or ridge (L2)?
Given a ridge and a lasso regularizer, which one should be chosen for better performance?
An intuitive graphical explanation (intersection of the elliptical contours of the loss function with the region of constraints) would be helpful.

L1 and L2 have slightly different effects. Have you read anything about them?

Yes, I do have an understanding in terms of the math involved. I am interested to know about the effect that it has on the parameters, that help in achieving the regularization effect.

The following graph shows the constraint region (green), along with contours for Residual sum of squares (red ellipse). These are iso-lines signifying that points on an ellipse have the same RSS.

Figure: Lasso (left) and Ridge (right) Constraints  [Source: Elements of Statistical Learning]
As Ridge regression has a circular constraint ($\beta_1^2 + \beta_2^2 <= d$) with no edges, the intersection will not occur on an axis, signifying that the ridge regression parameters will usually be  non-zero.
On the contrary, the Lasso constraint ($|\beta_1| + |\beta_2| <= d$) has corners at each of the axes, and so the ellipse will often intersect the constraint region at an axis. In 2D, such a scenario would result in one of the parameters to become zero whereas in higher dimensions, more of the parameter estimates may simultaneously reach zero.
This is a disadvantage of ridge regression, wherein the least important predictors never get eliminated, resulting in the final model to contain all predictor variables. For Lasso, the L1 penalty forces some parameters to be exactly equal to zero when $\lambda$ is large. This has a dimensionality reduction effect resulting in sparse models.
In cases where the number of predictors are small, L2 could be chosen over L1 as it constraints the coefficient norm retaining all predictor variables.

Your answer describes L1 and L2 regularization and their effect, but how does this relate to the concept of regularization? You just clarify what a person means by "regularization" and in which cases is L1 preferred over L2 (and vice versa), which is the actual question (I think).

Given that the author of the question asked for a graphical intuition behind regularization (with the constraint curve), I am given to understand he has knowledge of what "Regularization" signifies. The answer simply does not warrant a definition of the term.


text:
In this VAE formula, why do $p$ and $q$ have the same parameters?
In $$\log p_{\theta}(x^1,...,x^N)=D_{KL}(q_{\theta}(z|x^i)||p_{\phi}(z|x^i))+\mathbb{L}(\phi,\theta;x^i),$$ why does $p(x^1,...,x^N)$ and $q(z|x^i)$ have the same parameter $\theta?$
Given that $p$ is just the probability of the observed data and $q$ is the approximation of the posterior, shouldn't they be different distributions and thus their parameters different?

I will try to answer your questions directly (but I guess I won't be able to), otherwise, this can become quite confusing, given the inconsistencies that can be found across different sources.

In $logp_{\theta}(x^1,...,x^N)=D_{KL}(q_{\theta}(z|x^i)||p_{\phi}(z|x^i))+\mathbb{L}(\phi,\theta;x^i)$ why is $\theta$ and param for $p$ and $q$?

In a few words, your equation is wrong because it uses the letters $\phi$ and $\theta$ inconsistently.
If you look more carefully at the right-hand side of your equation, you will notice that $q_{\theta}$ has different parameters, i.e. $\theta$, than $p_{\phi}$, which has parameters $\phi$, so $p$ and $q$ have different parameters, and this should be the case, because they are represented by different neural networks in the case of the VAE. However, the left-hand side uses $\theta$ as the parameters of $p$ (while the right-hand side uses $\phi$ to index $p$), so this should already suggest that the equation is not correct (as you correctly thought).
In the case of the VAE, $\phi$ usually represents the parameters (or weights) of the encoder neural network (NN), while $\theta$ usually represents the parameters of the decoder NN (or vice-versa, but you should just be consistent, which is often not the case in your equation). In fact, in the VAE paper, in equation 3, the authors use $\phi$ to represent the parameters of the encoder $q$, while $\theta$ is used to denote the parameters of the decoder $p$.
So, if you follow the notation in the VAE paper, the ELBO can actually be written something like
\begin{align}
\mathcal{L}(\phi,\theta; \mathbf{x})
&=
\mathbb{E}_{\tilde{z} \sim q_{\phi}(\mathbf{z} \mid \mathbf{x})} \left[ \log p_{\theta} (\mathbf{x} \mid \mathbf{z}) \right] - \operatorname{KL} \left(q_{\phi}(\mathbf{z} \mid \mathbf{x}) \| p_{\theta}(\mathbf{z}) \right) \tag{1} \label{1}
\end{align}
The ELBO loss $\mathcal{L}(\phi,\theta; \mathbf{x})$ has both parameters (of the encoder and decoder), which will be optimized jointly. Note that I have ignored the indices in the observations $\mathbf{x}$ (for simplicity), while, in the VAE paper, they are present. Furthermore, note that, both in \ref{1} and in the VAE paper, we use bold letters (because these objects are usually vectors), i.e. $\mathbf{x}$ and $\mathbf{z}$, rather than $x$ and $z$ (like in your equation).
Note also that, even though $p_{\theta}(\mathbf{z})$ is indexed by $\theta$, in reality, this may be an un-parametrized distribution (e.g. a Gaussian with mean $0$ and variance $1$), i.e. not a family of distributions. The use of the index $\theta$ in $p_{\theta}(\mathbf{z})$ comes from the (implicit) assumption that both $p_{\theta}(\mathbf{z})$ and $p_{\theta} (\mathbf{x} \mid \mathbf{z})$ come from the same family of distributions (e.g. a family of Gaussians). In fact, if you consider the family of all Gaussian distributions, then $p_{\theta}(\mathbf{z}) = \mathcal{N}(\mathbf{0}, \mathbf{I})$ also belongs to that family. But $\theta$ and $\phi$ are also used to denote the parameters (or weights) of the networks, so this becomes understanbly confusing. (To understand equation 10 of the VAE paper, see this answer).

Why does $p(x^1,...,x^N)$ and $q(z|x^i)$ have the same parameter $\theta?$

This is wrong, in fact. If you look at equation 1 of the VAE paper, they use $\theta$ to denote the parameters of $p(\mathbf{x})$, i.e. $p_{\theta}(\mathbf{x})$, while the parameters of the encoder are $\phi$, i.e. $q_{\phi}(\mathbf{z} \mid \mathbf{x}$).

Cause $p$ is just the probability of the observed data and $q$ is the approximation of the posterior so shouldn't they be different distributions and their parameters different?

Yes.


text:
Should I choose the model with highest validation accuracy or the model with highest mean of training and validation accuracy?
I'm training a deep network in Keras on some images for a binary classification (I have around 12K images). Once in a while, I collect some false positives and add them to my training sets and re-train for higher accuracy.
I split my training into 20/80 percent for training/validation sets.
Now, my question is: which resulting model should I use? Always the one with higher validation accuracy, or maybe the higher mean of training and validation accuracy? Which one of the two would you prefer?
Epoch #38: training acc: 0.924, validation acc: 0.944
Epoch #90: training acc: 0.952, validation acc: 0.932

Neither of the above mentioned methods could be a potent indicator of the performance of a model.
A simple way to train the model just enough so that it generalizes well on unknown datasets would be to monitor the validation loss. Training should be stopped once the validation loss progressively starts increasing over multiple epochs. Beyond this point, the model learns the statistical noise within the data and starts overfitting.

This technique of Early stopping could be implemented in Keras with the help of a callback function:
class EarlyStop(tf.keras.callbacks.Callback):
    def on_epoch_end(self, epoch, logs={}):
        if(logs.get('val_loss') < LOSS_THRESHOLD and logs.get('val_categorical_accuracy') > ACCURACY_THRESHOLD):
            self.model.stop_training = True

callbacks= EarlyStop()
model.fit(...,callbacks=[callbacks])

The Loss and Accuracy thresholds can be estimated after a trial run of the model by monitoring the validation/training error graph.

So I should track (and log) the val_loss metric instead of val_acc in the name of output model. Is that right? And choose the lowest one.

@TinaJ Yes, that would be better indicator of the model performance. Validation accuracy may fluctuate throughout the training procedure (a high validation accuracy reached in the initial epochs could be just a fluke, signifying little about the predictive power of the model).

umm I re-train my model once in a while with added data. I see val_acc increases but when I test the model expecting better accuracy, it actually behaves poorlier than before. Is it because the val_loss was probably higher than before?

The training accuracy tells you nothing about how good it is on other data than the ones it learned on, it could be better on this data because it memorized this examples.
On the other hand the validation set is here to indicate you how good the model is to generalize what it learned to new data (hopefully the testing dataset accurately represents the diversity of the data).
As you are looking for a model which is good on every dataset you don't want to use training accuracy to choose your model and so you should choose the first one.


text:
What are the state-of-the-art meta-reinforcement learning methods?
This question can seem a little bit too broad, but I am wondering what are the current state-of-the-art works on meta reinforcement learning. Can you provide me with the current state-of-the-art in this field?

One of the most recent papers on meta-RL is meta-Q-learning. This paper introduces Meta-Q-Learning (MQL), a new off-policy algorithm for meta-reinforcement learning (meta-RL). MQL builds upon three simple ideas.

Q-learning is competitive with state-of-the-art meta-RL algorithms if given access to a context variable that is a representation of the past trajectory.

Using a multi-task objective to maximize the average reward across the training tasks is an effective method to meta-train RL policies.

past data from the meta-training replay buffer can be recycled to adapt the policy on a new task using off-policy updates

Experiments on standard continuous-control benchmarks suggest that MQL compares favorably with state-of-the-art meta-RL algorithms.
I think that other references to other work on meta-RL are present in the experiments part of the MQL paper.

Meta-Reinforcement Learning can refer to a broad range of ideas. Also, different algorithms are SOTA under different evaluation metrics (sample efficiency, agent performance, adaptation speed on a new task, etc)
Assuming that you are referring to the problem of quickly learning/adapting to a new task by training an agent on a distribution of related tasks, the following are some popular algorithms

PEARL [Rakelly et al., 2019]
VariBAD [Zintgraf et al., 2020]
Meta-Q-Learning [Fakoor et al., 2020]

References:

K Rakelly, A Xhou, D Quillen, C Finn, S Levine - Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables, ICML 2019.
L Zintgraf et al., - VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning, ICLR 2020.
R Fakoor, P Chaudhari, S Soatto, A J Smola - Meta-Q-Learning, ICLR 2020.


text:
Is a non-linear activation function needed if we perform max-pooling after the convolution layer?
Is there any need to use a non-linear activation function (ReLU, LeakyReLU, Sigmoid, etc.) if the result of the convolution layer is passed through the sliding window max function, like max-pooling, which is non-linear itself? What about the average pooling?

Let's first recapitulate why the function that calculates the maximum between two or more numbers, $z=\operatorname{max}(x_1, x_2)$, is not a linear function.
A linear function is defined as $y=f(x) = ax + b$, so $y$ linearly increases with $x$. Visually, $f$ corresponds to a straight line (or hyperplane, in the case of 2 or more input variables).
If $z$ does not correspond to such a straight line (or hyperplane), then it cannot be a linear function (by definition).
Let $x_1 = 1$ and let $x_2 \in [0, 2]$. Then $z=\operatorname{max}(x_1, x_2) = x_1$ for all $x_2 \in [0, 1]$. In other words, for the sub-range $x_2 \in [0, 1]$, the maximum between $x_1$ and $x_2$ is a constant function (a horizontal line at $x_1=1$). However, for the sub-range $x_2 \in [1, 2]$, $z$ correspond to $x_2$, that is, $z$ linearly increases with $x_2$. Given that max is not a linear function in a special case, it can't also be a linear function in general.
Here's a plot (computed with Wolfram Alpha) of the maximum between two numbers (so it is clearly a function of two variables, hence the plot is 3D).

Note that, in this plot, both variables, $x$ and $y$, can linearly increase, as opposed to having one of the variables fixed (which I used only to give you a simple and hopefully intuitive example that the maximum is not a linear function).
In the case of convolution networks, although max-pooling is a non-linear operation, it is primarily used to reduce the dimensionality of the input, so that to reduce overfitting and computation. In any case, max-pooling doesn't non-linearly transform the input element-wise.
The average function is a linear function because it linearly increases with the inputs. Here's a plot of the average between two numbers, which is clearly a hyperplane.

In the case of convolution networks, the average pooling is also used to reduce the dimensionality.
To answer your question more directly, the non-linearity is usually applied element-wise, but neither max-pooling nor average pooling can do that (even if you downsample with a $1 \times 1$ window, i.e. you do not downsample at all).
Nevertheless, you don't necessarily need a non-linear activation function after the convolution operation (if you use max-pooling), but the performance will be worse than if you use a non-linear activation, as reported in the paper Systematic evaluation of CNN advances on the
ImageNet (figure 2).

Is the same not true for ReLU? And even more so for LeakyReLU?

@Kasia What exactly are you referring to?

ReLU is simply a max function. Couldn't it basically be considered a max-pool with additional constant dimension?

@Kasia In max-pooling, conceptually, you slide a window. When you apply ReLU you do not slide any window. It's true that ReLU is a max between 0 and the input, but I fail to understand how you want to make it a pooling operation.

@DuttA So I spent a day thinking about it. In an intuitive approach, ReLU is something like an on/off for gradient flow. Either lets it go through freely, or turns it off completely. In the same way max-pool allows the gradient to flow freely through the maximum value, and blocks it for all the other values inside the sliding window. That' s the resemblance I see.

@Kasia Yes, the derivative of the ReLU is consistent with your description.


text:
Are there names for neural networks with a well-defined layer or neuron characteristics?
Are there names for neural networks with a well-defined layer or neuron characteristics?
For example, a matrix that has the same number of rows and columns is called a square matrix.
Is there an equivalent for classifying different neural network structures. Specifically, I am interested if there is a name for a neural network with x number of layers, but each layer has the same number of neurons?

Neural networks (NNs) are usually classified into feed-forward (i.e. NNs with feedforward connections), recurrent (i.e. NNs with recurrent connections) and convolutional (i.e. NNs that perform a convolution or cross-correlation operation). The term multi-layer NN may also be used to refer to feed-forward neural networks or, in general, neural networks with multiple (hidden) layers. There are also the perceptrons, which do not have hidden layers (i.e. the inputs are directly connected to the outputs).
You may still divide neural networks into classifiers (i.e. the outputs and labels are discrete) and regressors (the outputs and labels are numerical). Furthermore, you may classify them into generative models (e.g. the VAE) or discriminative models.
By analogy with linear algebra concepts, each layer of a feedforward neural network (FFNN) can be seen as a linear operation followed by an element-wise application of a linear or non-linear activation function.
$$
\mathbf{o}^{l} = \sigma \left(\mathbf{a}^{l} \mathbf{W}^{l} + \mathbf{b}^{l}\right)
$$
where $\sigma$ is the activation function, $\mathbf{a}^{l}$ the inputs to the layer $l$ and $\mathbf{W}^{l}$ the parameters of the layer $l$ (similar to the parameters or coefficients in linear regression) and $\mathbf{b}^{l}$ is the bias vector (a scalar bias for each neuron) of layer $l$. $\mathbf{o}^{l}$ will then be $\mathbf{a}^{l+1}$ (i.e. the input to the next layer).
A recurrent neural network (RNN) performs a slightly more complex operation.
$$
\mathbf{o}^{l}_t = \sigma \left(\mathbf{a}^{l}_t \mathbf{W}^{l} + \mathbf{o}^{l}_{t-1} \mathbf{R}^{l} + \mathbf{b}^{l}\right)
$$
where, in addition to the matrix $\mathbf{W}^{l}$ associated with the feedforward connections, it also uses another matrix $\mathbf{R}^{l}$ associated with the recurrent connections (i.e. cyclic or loopy connections of the neurons), which is multiplied by the output of the same layer but at the previous time step. $\mathbf{o}^{l}_t$ may actually just be the state of the recurrent layer, which is then used to compute the actual output of the layer, but, for simplicity, you can ignore this. Furthermore, note that there are more complex recurrent architectures, but this is the basic idea.
A convolutional neural network (CNN) performs the convolution (or cross-correlation) operation. If you are familiar with signal processing, e.g. kernels, convolution, etc., then you can view a CNN as performing a convolution (or cross-correlation) operation. It's actually possible to view the convolution as a matrix multiplication operation, but the details can easily become cumbersome and tedious to explain in an answer. A CNN may also perform other types of operations (such as downsampling) and it can also be composed of a feedforward part (usually, the last layers of a CNN are feedforward layers), but a CNN is a CNN because it performs the convolution (or cross-correlation).
In all cases, the matrices do not necessarily take any particular form (e.g. they are not necessarily square matrices). The dimensionality of the matrices depends on the number of layers and connections in the network, which can vary depending on many factors (e.g. the need for recurrent connections because they are useful for sequence modeling). These matrices (along with the biases) are the learnable parameters of the networks, but the learned values of these matrices highly depend on your data, the way you initialize them before learning, the architecture of the network, the learning algorithm, etc.
In the case of the FFNN, if the previous layer $l-1$ has the same number of neurons as the current layer $l$, then $\mathbf{W}^{l}$ is a square matrix. AFAIK, there is no name for a neural network with the same number of neurons for each layer. It could be called a rectangular neural network (but this is a name I've just come up with).
To conclude, there are many different neural networks and taxonomies for neural networks, so it is impossible to list or discuss them all in an answer, but, nowadays, the subdivision into feedforward, recurrent and convolution is the most common and general. See e.g. the paper A Taxonomy for Neural Memory Networks if you are interested in a more detailed taxonomy for memory-based neural networks (e.g. recurrent neural networks).


text:
How to detect vanishing gradients?
Can vanishing gradients be detected by the change in distribution (or lack thereof) of my convolution's kernel weights throughout the training epochs? And if so how?
For example, if only 25% of my kernel's weights ever change throughout the epochs, does that imply an issue with vanishing gradients?
Here are my histograms and distributions, is it possible to tell whether my model suffers from vanishing gradients from these images? (some middle hidden layers omitted for brevity)

What did you do to visualize weights during training, please?

@Avv I used Tensorboard, and went to the histograms tab. Please google a good Tensorboard tutorial.

Vanishing Gradients can be detected from the kernel weights distribution. All you have to look for is whether the weights are dying down to 0.
If only 25% of your kernel weights are changing that does not imply a vanishing gradient, it might be a factor, but there can be a variety of reasons, such as poor data, loss function used to the optimizer, etc. Kernel's weight not changing only points out that the model is not learning well.
From the histograms, only the conv_2d_2 layer shows any form of vanishing gradients since the numbers are pretty small. But even that seems to be picking after 600 epochs. Usually, a good indicator is to have the mean of the weights in a layer to be closer to 0 and the standard deviation to be closer to 1. So, if this is maintained to a good extent during training, you're good to go.

how I can visualize weights during training, please as @Elegant Code did?


text:
Should I apply image processing techniques to the inputs of convolution networks?
After working for some time with feature-based pattern recognition, I am switching to CNN to see if I can get a higher recognition rate.
In my feature-based algorithm, I do some image processing on the picture before extracting the features, such as some convolution filters to reduce noise and segmentation into the foreground and background, and finally identifying and binarization of objects.
Should I do the same image processing before feeding data into my CNN, or is it possible to feed raw data to a CNN and expect that the CNN will adapt automatically without per-image-processing steps?

The whole interest of using deep learning-based solutions is that you don't have to do all those pre-processings, i.e. binarization, segmentation of background. CNNs, such as YOLO or FasterRCNN, can learn how to retrieve that information by themselves.

This answer can be misleading. Input preprocessing (e.g. normalization) can actually enhance a lot the performance of a CNN. Maybe you should clarify this in this answer.

I believe that in this question, he means by pre-processing doing some feature extraction: Segmentation into the foreground and background. But of course I agree that normalizing, adding noise, rotating can improve the quality of CNN based methods. And can also constitue a data augmentation

The CNN should work without trying to do special feature extraction. As pointed out some pre-processing can aid in enhancing the CNN's classification results. The Keras
ImageDataGenerator provides optional parameters you can set to provide pre-processing as well as provide data augmentation.
One thing I know that works for sure but can be painful is cropping the images in such a way that the subject of interest occupies a high percentage of the pixels in the resultant cropped image. The cropped image can than be resized as needed. The logic here is simple.
You want your CNN to train on the subject of interest (for example a bird sitting in a tree where the bird is the subject of interest). The part of the image that is not of the bird is essentially just noise making the classifier's job harder. For example say you have a 500 X 500 initial image in which the subject of interest (the bird) only takes up 10% of the pixels (25,000 pixels). Now say as  input to your CNN you reduce the image size to 100 X 100. Now the (pixels that the CNN 'learns' from is down to 1000 pixels.
However lets say you crop the image so that the features of the bird are preserved but the  pixels of the bird in the cropped image take up 50% of the pixels. Now if you resize the cropped image to 100 X 100 , 5000 pixels of relevance are available for the network to learn from. I have done this on several data sets. In particular images of people where the subject of interest is the face. There are many programs that are effective at cropping these images so that  mostly just the face appears in the cropped result. I have trained a deep CNN in one case using uncropped images  and in the other with cropped images. The results are significantly better using the cropped images.

So cropping training data to object of interest improves the recognition rate, is that valid regardless of the size of the Object of interest in validation/test data?

I have not conducted experiments to determine what I would call a threshold point. That is at what point you should elect to crop images based on what percentage of pixels in the image the subject of interest occupies.Clearly at 50% you will probably not gain much by cropping the images. What I have found is cropping the images does improve accuracy on data sets I have developed. where in many cases the subject occupied less than 10% of the image.

Gerry P, doesn't that make it harder for the detector for different sized images though?

No after cropping all images you use as input to the classifier must be the same size so just resize the cropped image. Keras flow_from_directory will do that for you just specify the height and width.


text:
What are the conditions for the convergence of SARSA to the optimal value function?
Is it correct that for SARSA to converge to the optimal value function (and policy)

The learning rate parameter  $\alpha$ must satisfy the conditions:
$$\sum \alpha_{n^k(s,a)} =\infty \quad \text{and}\quad \sum \alpha_{n^k(s,a)}^{2} <\infty \quad \forall s \in \mathcal{S}$$
where $n_k(s,a)$ denotes the $k^\text{th}$ time $(s,a)$ is visited
$\epsilon$ (of the $\epsilon$-greedy policy) must be decayed so that the policy converges to a greedy policy.
Every state-action pair is visited infinitely many times.

Are any of these conditions redundant?

There is probably some relation between the fact that the TD(0) function converge to the true Value function with probability 1 when the learning rate parameter is decayed like above. There also exists some interval of alphas $\alpha \in (0,p)$ such that the TD estimate converges in expected value to the true value function

The paper Convergence Results for Single-Step On-Policy Reinforcement-Learning Algorithms by Satinder Singh et al. proves that SARSA(0), in the case of a tabular representation of the value functions, converges to the optimal value function, provided certain assumptions are met

Infinite visits to every state-action pair
The learning policy becomes greedy in the limit

The properties are more formally stated in lemma 1 (page 7 of the pdf) and theorem 1 (page 8). The Robbins–Monro conditions should ensure that each state-action pair is visited infinitely often.

I was already writing this answer before the other answer was published, but these answers are equivalent. I am only citing the paper that originally proved this.

Thanks for the answer. Sorry about my poor formatting of posts here. Just learning about markdown.

@KaneM No problem! I have a look at source code of my and your answer (now after my edits) to understand better how markdown works :) Anyway, I am not completely sure that the Robbins-Monro conditions ensure that each state is visited infinitely often. Maybe it's the other way around: if each state-action pair is visited infinitely often, then the Robbins-Monro conditions are satisfied.

I believe they are separate conditions. From my understanding, the RM conditions ensure that the Q function eventually converges to a value while the condition that every state-action pair is visited infinitely often ensure that it converges to the correct value.

@KaneM These conditions are very related. I'm linking to a paper that seems to relate the two in a more precise way, but I haven't yet fully read this paper.

I wonder if the Robbins-Monro conditions are not present, does there exist some range of $\alpha$ such that the policy converges in some notion of expectation to the optimal policy.

@KaneM This is an interesting question. Maybe ask on the site! I wouldn't currently be able to answer it, but maybe there's already some research work that goes in this direction. In all convergence results I've seen so far, the RM conditions are assumed (essentially, because these RL theorems and proofs follow from the original results by Robbins and Monro, AFAIK).

I have the conditions for convergence in these notes SARSA convergence by Nahum Shimkin.

The Robbins-Monro conditions above hold for $α_t$.
Every state-action pair is visited infinitely often
The policy is greedy with respect to the policy derived from $Q$ in the limit
The controlled Markov chain is communicating: every state can be reached from any other with positive probability (under some policy).
$\operatorname{Var}{R(s, a)} < \infty$, where $R$ is the reward function


text:
Is there an advantage in decaying $\epsilon$ during Q-Learning?
If the agent is following an $\epsilon$-greedy policy derived from Q, is there any advantage to decaying $\epsilon$ even though $\epsilon$ decay is not required for convergence?

Yes Q-learning benefits from decaying epsilon in at least two ways:

Early exploration. It makes little sense to follow whatever policy is implied by the initialised network closely, and more will be learned about variation in the environment by starting with a random policy. It is fairly common in DQN to initially fill the experience replay table whilst using $\epsilon = 1.0$ or otherwise an effectively random policy.

Later refinement. Q-learning can only learn from experiences it has. A behaviour policy that is too random may not experience enough states close to optimal in order to gain enough statistics on them to overcome variance. In more difficult cases, it may never experience the whole of an optimal trajectory even when combining all the different transitions that it ever observes.

In addition, when using a function approximator such as a neural network, predictions of the target policy will be influenced by the distribution of states and actions in the experience replay memory. If that is is biased towards the distribution of states of a very different behaviour policy, then basic Q-learning has no good way to adjust for that - it adjusts for differences in expected return between behaviour and target policies, but not in distribution of observed states. In fact this is still a somewhat open problem, you want the agent to learn from mistakes and imperfect behaviour so as to avoid it, but you don't want those mistakes to skew predictions of what should happen under an optimal policy. Function approximation such as neural networks is influenced by the distribution of input data, so it typically performs better in Q learning when the behaviour policy and target policy are close e.g. $\epsilon$ should be relatively low if using $\epsilon$-greedy.
A typical implementation might start with $\epsilon=1.0$, set a decay factor per time step or per episode e.g. $0.999$ per episode, and a minimum $\epsilon = 0.01$. These are all hyper-parameters that you can adjust depending on the problem.


text:
How is the expected value in the loss function of DQN approximated?
In Deep Q Learning the parametrized Q-functions $Q_i$ are optimised by performing gradient descent on the series of loss functions
$L_i(\theta_i)= E_{(s,a)\sim p}[(y_i-Q(s,a;\theta_i))^2]$ , where
$y_i = E_{s' \sim \mathcal{E}}[r+\gamma \max_{a'}Q(s',a';\theta_{i+1})\mid s,a]$.
In the actual algorithm, however, the expected value is never computed. Also, I think it cannot be computed since the transition probabilities of the underlying MDP remain hidden from the agent. Instead of the expected value, we compute $y_i = r_i + \gamma \max_a Q(\phi_{i+1},a;\theta)$. I assume some sort of stochastic approximation is taking place here. Can someone explain the details?

Just as the paper says
$$L_i(\theta_i)= E_{(s,a)\sim p}[(y_i-Q(s,a;\theta_i))^2]$$
where
$$y_i = E_{s' \sim \mathcal{E}}[r+\gamma \max_{a'}Q(s',a';\theta_{i+1})\mid s,a]$$
Then in the Background section of the paper, it says

Differentiating the loss function with respect to the weights we arrive at the following gradient:

$$\nabla_{\theta_i} L_i(\theta_i)\\= E_{(s,a)\sim p,s'\sim\mathcal{E}}\left[\left(r+\gamma \max_{a'}Q(s',a';\theta_{i+1})-Q(s,a;\theta_i)\right)\nabla_{\theta_i}Q(s,a;\theta_i)\right]\tag{1}$$

Rather than computing the full expectations in the above gradient, it is often computationally expedient to optimize the loss function by stochastic gradient descent.
...
and the expectations are replaced by single samples from the behavior distribution $ρ$ and the emulator $\mathcal{E}$ respectively.

If you're familiar with SGD and Stochastic Optimization then you know what happens here:
The expression inside the expectation of (1), i.e. $\left(r+\gamma \max_{a'}Q(s',a';\theta_{i+1})-Q(s,a;\theta_i)\right)\nabla_{\theta_i}Q(s,a;\theta_i)$ , is an unbiased estimation of the real gradient $\nabla_{\theta_i}L_i(\theta_i)$ - its expectation is the real gradient. In other words,
$$\widehat{\nabla L}=\left(r+\gamma \max_{a'}Q(s',a';\theta_{i+1})-Q(s,a;\theta_i)\right)\nabla_{\theta_i}Q(s,a;\theta_i).$$
Then by theory of Stochastic Optimization we can optimize $L$ by $\theta\leftarrow \theta - \alpha\widehat{\nabla L}$ , which is how SGD works.
The unbiased estimation $\left(r+\gamma \max_{a'}Q(s',a';\theta_{i+1})-Q(s,a;\theta_i)\right)\nabla_{\theta_i}Q(s,a;\theta_i)$ can be sampled and calculated directly - you can run the emulator $\mathcal{E}$ and current behavior policy to collect $r, s', a'$ and calculate the gradient of your Q network $\nabla Q$ using TensorFlow. So the gradient to $L$ can be approximated.
The $y_j$ in the pseudocode  of the paper is also an estimation. It'll be more proper to denote it as $\hat{y_j}$.
(I'm not an native English speaker so forgive my poor expression.)

Welcome to AI SE! Thanks for contributing! This answer seems consistent with knowledge. Maybe you could spend more words on why that is an "unbiased estimator of the gradient" (maybe just link to a paper that shows that this an unbiased estimator of the gradient) and how this estimator is actually called. Furthermore, what do you mean by "which can be sampled and calculated directly", i.e. how can an estimator be sampled? Also, which pseudocode are you talking about where $\hat{y_j}$ is present?

Thank you for your feedback. I have tried to edit my post to make it more clear. My English writing skill is not so good.

Thanks for addressing my questions. Now your answer is clearer! And thanks again for contributing! I've already upvoted your answer. Feel free to stick around and contribute more :)

Thanks for your contribution. I must admit that I haven't watched enough into stochastic gradient. As I understand it we try to find a zero of the gradient via the Robbins-Monro-Algorithm. In general, does that mean you have to check the prerequisites of Robbins-Monro before using stochastic gradient ? I think here they are satisfied.

I don't understand how it is an unbiased estimator, can someone elucidate?

Why is expectation in Loss function is with respect to (s,a)?  (s,a) is given.

authors: Мухаммедали Калым


text:
Should I grey-scale the coloured frames/channels to build the approximation of the state?
I'm doing reinforcement learning, and I have a visual observation that I will use to build an input state for my agent. In the DeepMind's Atari paper, they greyscale the input image before they input it into the CNN to reduce the input space's size, which makes sense to me.
In my environment, I have, for each pixel, 5 possible channels, which are represented in black, white, blue, red, and green. This also makes intuitive sense to me since it's like a bit-encoding.
Any thoughts on what could be better? Greyscaling into 2 shades of grey and black and white also maintains the information, but feels somehow less direct, since my environment's visual space is categorical, which makes more sense in a categorical encoding.

Greyscaling can only make sense, when the meaning of blue, red and green is somewhere between white and black. Otherwise, you'd force the network to decode it first. And even when the meaning is like above, providing one bit for each possible state sounds better. That's how networks for chess work: one bit per piece kind and field.

I've edited this post in order to clarify it further and to improve the title. Please, make sure I didn't change the meaning of it.

We can't say for sure which approach would work best in the general case. If you have domain knowledge, you can make a better guess. You'll basically want to answer the question: which information is important for learning an optimal policy?

In my environment, I have, for each pixel, 5 possible channels, which are represented in black, white, blue, red, and green. This also makes intuitive sense to me since it's like a bit-encoding.

Generally, if you have an environment like this, I would (without any other information) guess that each of the 5 colors have some meaning that may be relevant for your agent. That's just what I would guess though. In theory, it might be possible that white means one thing (e.g. "empty"), and every other colour means the same other thing (e.g. "not empty"). If you had domain knowledge like that, and knew that it is only important whether or not any given pixel is white, you could of course binarise your input.
But in general, if the colours might be important, I'd recommend including them. If you really only have a few distinct colours like that though, I would not recommend encoding them in some format like RGB where values can range from 0 to 1 or 0 to 255. I would recommend having 4 (or 5?) binary channels in your input:

Binary channel containing 1s for pixels that are black, and 0s for all other pixels.
Binary channel containing 1s for pixels that are white, and 0s for all other pixels.
Binary channel containing 1s for pixels that are blue, and 0s for all other pixels.
...
etc.

The reason for this is that deep neural networks often tend to have an easier time learning with binary inputs, and here you can completely binarise your inputs without requiring an excessively high number of channels. If you had hundreds or thousands of different possible colours, this would probably no longer be a good idea.


text:
Why not more TD() in actor-critic algorithms?
Is there either an empirical or theoretical reason that actor-critic algorithms with eligibility traces have not been more fully explored? I was hoping to find a paper or implementation or both for continuous tasks (not episodic) in continuous state-action spaces.
This has been the only related question on SE-AI that I have been able to find Why are lambda returns so rarely used in policy gradients?.
Although I appreciated the dialog and found it useful, I was wondering if there was any further detail or reasoning that might help explain the void.

Theoretically, nothing precludes the use of $\lambda$-returns in actor-critic methods. The $\lambda$-return is an unbiased estimator of the Monte Carlo (MC) return, which means they are essentially interchangeable. In fact, as discussed in High-Dimensional Continuous Control Using Generalized Advantage Estimation, using the $\lambda$-return instead of the MC return can actually help reduce the variance of gradient updates.
The above is similar to my answer in the other question you linked, so let me try to answer your question more specifically. Even though we can use $\lambda$-returns, why are they not too common in practice? I suspect there might be a few reasons:

Empirically, faster credit assignment might be more desirable than lower variance. Sometimes the learning speed of your algorithm is constrained simply by how quickly you can learn about the consequences of certain actions. In this case, it is faster to use the MC return, even if it theoretically has higher variance than the $\lambda$-return.
When proposing a new algorithm, adding $\lambda$-returns to it might give it an "unfair" advantage if the other baseline methods do not use them (the reviewers would not like this), so researchers tend to favor simpler 1-step or MC returns for the sake of consistency. I would guess this is why you don't typically see $\lambda$-returns in papers that propose new actor-critic methods. In some sense, it is generally assumed that you could always add $\lambda$-returns to them later and probably get better performance.$^*$
A decent number of deep RL researchers don't know what $\lambda$-returns are. This is especially true if they come from a pure deep learning background; they may have never read Reinforcement Learning: An Introduction which is where most people are introduced to TD($\lambda$) and $\lambda$-returns.

$^*$Exceptions to this are papers like mine (Reconciling $\lambda$-Returns with Experience Replay) where the contribution is the use of $\lambda$-returns in methods that previously could not use them. But actor-critic methods are pretty straightforward to combine with $\lambda$-returns.

Why are $\lambda$ returns unbiased estimators of MC returns, given that they are an average of returns that go from TD(0) to TD(1)? Maybe this could be a question on the main site. Anyway, you say that $\lambda$ returns are unbiased estimators of MC returns, but then you say "In fact, they help to reduce the variance", while the use of MC returns usually leads to high variance. In other words, the "In fact" isn't probably the best way of starting that phrase. Also, I think you should better distinguish between "faster credit assignment" and "faster implementation".

The $\lambda$-return is a statistically unbiased estimator of the MC return because it's a weighted average of $n$-step returns, each of which are themselves unbiased estimators of the MC return on average. This should not be confused with the "bias" due to bootstrapping; the terminology is a bit confusing because there are two notions of bias here. I am referring to the former notion: each $n$-step return, in expectation, is equal to the MC return and therefore so is the $\lambda$-return.

This also explains why the $\lambda$-return should reduce variance. Each $n$-step return has some variance, but they all have the same expectation. The $\lambda$-return is a convex combination of the $n$-step returns, where the weights are positive and sum to 1. Therefore, the $\lambda$-return must also have the same expectation. But from the variance formula $Var(aX+bY)=a^2 Var(X)+b^2 Var(Y)$ (assuming independence), you can see how this should reduce the variance because $a^2 + b^2 < 1$. This can be viewed as an ensemble method of sorts.

Finally, I did intend to say "faster credit assignment". The MC return propagates reward information back in time to all states in the episode. The $\lambda$-return does too, technically/mathematically, but the exponential decay means that it practically stops updating states after some finite number of timesteps.


text:
Why does the machine learning algorithm need to learn a set of functions in the case of missing data?
I am currently studying the textbook Deep Learning by Goodfellow, Bengio, and Courville. Chapter 5.1 Learning Algorithms says the following:

Classification with missing inputs: Classification becomes more challenging if the computer program is not guaranteed that every measurement in its input vector will always be provided. To solve the classification task, the learning algorithm only has to define a single function mapping from a vector input to a categorical output. When some of the inputs may be missing, rather than providing a single classification function, the learning algorithm must learn a set of functions. Each function corresponds to classifying $\mathbf{x}$ with a different subset of its inputs missing. This kind of situation arises frequently in medical diagnosis, because many kinds of medical tests are expensive or invasive. One way to efficiently define such a large set of functions is to learn a probability distribution over all the relevant variables, then solve the classification task by marginalizing out the missing variables. With $n$ input variables, we can now obtain all $2^n$ different classification functions needed for each possible set of missing inputs, but the computer program needs to learn only a single function describing the joint probability distribution. See Goodfellow et al. (2013b) for an example of a deep probabilistic model applied to such a task in this way. Many of the other tasks described in this section can also be generalized to work with missing inputs; classification with missing inputs is just one example of what machine learning can do.

I was wondering if people would please help me better understand this explanation. Why is it that, when some of the inputs are missing, rather than providing a single classification function, the learning algorithm must learn a set of functions? And what is meant by "each function corresponds to classifying $\mathbf{x}$ with a different subset of its inputs missing."?
I would greatly appreciate it if people would please take the time to clarify this.

Intuitively, this is similar to the case when you are making predictions but you don't have all the necessary information to make the most accurate prediction or maybe there isn't a single accurate prediction, so you have a set of possible predictions (rather than a single prediction).
For example, if you hadn't seen the last Liverpool game (in the Champions League) against Atlético Madrid, you would have probably said that Liverpool was the most likely team to win the CL this year (2020) too. However, after having seen their last game, you noticed that they are not unbeatable and they are not perfect, so, although they have shown you (during this and the previous season) that they are a very good team, they may also no be the best until the end of the season. So, at this point, you may have a set of two possible hypotheses: Liverpool will win the CL or Liverpool will not win the CL.
In general, if you had a dataset that is representative of your whole population, then the dataset alone should be sufficient to make accurate predictions (i.e. it contains all the information sufficient to make accurate predictions). If that's not the case (which is often true), then you will have to account for all possible values of the missing data or you will have to make assumptions (or introduce an inductive bias).
The authors also mention the concept of marginalization, which is used in probability theory to calculate marginal probabilities, e.g. $p(X=x)$ (or for short $p(x)$), when there's another random variable $Y$, by accounting for all possible values of $Y$. In other words, you're interested only in $p(x)$ and you may have the joint probability distribution $p(x, y)$, then marginalization allows you to compute $p(x)$ using e.g. $p(x, y)$ and all possible values that the random variable $Y$ can take.
In any case, I think their description is a little bit vague and using the concept of marginalization to convey the idea behind the "multiple hypotheses" isn't the most appropriate approach, IMHO. If you are interested in these concepts in the context of neural networks, I suggest you read something about Bayesian machine learning or Bayesian neural networks.

Comments are not for extended discussion; this conversation has been moved to chat.

@The Pointer the $2^n$ came from the question: How many function do we need to have if each of the $n$ inputs can be missing?
example: $f_1(\text{missing}, x_2, x_3, \dots, x_n)$ for $x_1$ missing
$f_2(x_1, x_2, \text{missing}, x_4, \text{missing}, \dots, x_n)$ for $x_3$ and $x_5$ missing.
So this problem is a combinatorial one and the event for each $x_i$ is Missing or Not.
Each function corresponds 1-by-1 with a possible set $(x_1, \text{missing}, x_2, \dots, x_n)$. So how many sets can you form $2^n$. Why?
This formula comes back to like tree developing. First variable $x_1$ missing or not (2 possbile events). Now after that, for EACH of these events 2 possbile events for $x_2$ so in total 2 \times 2 (2 for $x_1$ , 2 for $x_2$ for each $x_1$'s event) and etc., $2 \times 2 \times 2 \times \dots$ $n$ times = $2^n$.

Thanks for the clarification. Your post is helpful, but, since it doesn't address the main question, I think it would have been more suitable as a comment.


text:
How do I generate a feature representation of a saliency map (or mask)?
Generally, CNNs are used to extract feature representations of an image. I'm right now dealing with the class of CNN that produces saliency maps, which are generally in the format of a mask. I'm trying to generate a feature representation of that specific Mask. What could be the best way to approach this problem?

I found a method to do it in the paper Cross-Modality Personalization for Retrieval (2020, accessed: 20-Feb-2020).

Representation. For images, we extract Inception-v4 CNN features [36]. We then mask the image convolution feature with the BubbleView saliency map, by resizing the saliency map to the convolution feature size and multiplying them together. Finally, average pooling is performed to obtain a 1536-dimensional feature vector. We represent textual descriptions as their average 200-dimensional Glove embedding [31]. For personality, we use a 10-dimensional feature vector containing the scores for the personality questions in [32]. Below, we describe how we learn projections of these representations that place them in the same feature space.


text:
Why is there more than one way of calculating the accuracy?
Some sources consider the true negatives (TN) when computing the accuracy, while some don't.
Source 1:
https://medium.com/greyatom/performance-metrics-for-classification-problems-in-machine-learning-part-i-b085d432082b

Source 2:https://www.researchgate.net/profile/Mohammad_Sorower/publication/266888594_A_Literature_Survey_on_Algorithms_for_Multi-label_Learning/links/58d1864392851cf4f8f4b72a/A-Literature-Survey-on-Algorithms-for-Multi-label-Learning.pdf

which can be translated as

which one of these must be considered for my multi-label model.

There are many different metrics for all sorts of features; generally there is no right or wrong, and it depends on your application.  If you have multiple labels, you probably don't have TN, so the second one looks to be a better fit.

In machine learning, the accuracy is usually defined as the number of correct predictions divided by the total number of predictions. The correct predictions are the true positives ($\mathrm {TP}$) and true negatives ($\mathrm {TN}$), so the usual formula to calculate the accuracy is the following one (your first one).
\begin{align}
\text{Accuracy}=\frac {\mathrm {TP} + \mathrm {TN}}{\mathrm {TP} + \mathrm {TN} + \mathrm {FP} + \mathrm {FN}} \label{0}\tag{0}
\end{align}
The following formula corresponds to the threat score ($\mathrm{TS}$) or critical success index ($\mathrm {CSI}$).
\begin{align}
{\displaystyle \mathrm {TS} ={\frac {\mathrm {TP} }{\mathrm {TP} +\mathrm {FN} +\mathrm {FP} }}} \label{1} \tag{1}
\end{align}
In section 7 of the paper A Literature Survey on Algorithms for Multi-label Learning, it is written

In traditional classification such as multi-class problems, accuracy is the most common evaluation criteria. Additionally, there exists a set of standard evaluation metrics that includes precision, recall, F-measure, and ROC area defined for single-label multi-class classification problems. However, in multi-label classification, predictions for an instance is a set of labels and, therefore, the prediction can be fully correct, partially correct (with different levels of correctness) or fully incorrect. None of these existing evaluation metrics capture such notion in their original form.

Then they say

Depending on the target problem, evaluation measures for multi-label data can be grouped into at least three groups: evaluating partitions, evaluating ranking and using label hierarchy. The first one evaluates the quality of the classification into classes, the second one evaluates if the classes
  are ranked in order relevance and the third one evaluates how effectively the learning system is able to take into account an existing hierarchical structure of the labels.

In section 7.1, the authors state

To capture the notion of partially correct, one strategy is to evaluate the average difference between the predicted labels and the actual labels for each test example, and then average over all examples in the test set. This approach is called example-based evaluations.

To capture the notion of partially incorrect in multi-label classification problems, they then use the following definition of accuracy (proposed in section 5.2 of the paper Discriminative Methods for Multi-labeled Classification)
\begin{align}
\frac{1}{n} \sum_{i=1}^{n} \frac{|Y_i \cap Z_i|}{|Y_i \cup Z_i|} \label{2} \tag{2}
\end{align}
where $Y_i$ is the true set of labels and $Z_i$ the predicted set of labels for the single instance (or observation) $i$, so $\frac{|Y_i \cap Z_i|}{|Y_i \cup Z_i|}$ is the accuracy for the instance $i$. If $|Y_i|=1$ and $|Z_i|=1$, then we have a single-label classification problem.
The metric \ref{2} does not correspond to the metric \ref{1}. In \ref{2}, the accuracy is calculated for each example (or instance) and then we average all these accuracies, because, for each example, you may have that one predicted label is correct, two predicted labels are correct, and so on (also depending on the number of labels you need to predict for each of the examples). In \ref{1}, there is no notion of multiple labels.

In equation.1, though there is no notion of multiple labels, I can consider confusion matrix for vector of predicted  and vector of actual  to calculate TS(Theta Score) and if I average over all the examples or instances isn't it same as equation.2?

@StephenPhilip I don't think so. $Y_i$ and $Z_i$ should be sets (rather than vectors).

Yes,  and   are sets, just to disambiguate I was using programming language terminology(C++ uses generally vectors are used to store as arrays, in python lists are used), but my doubt still persists, isn't eq.1 is representation of eq.2 with average over all the observations.

@StephenPhilip I don't understand where you see the equivalence between equation 1 and 2, but maybe I am blind now. Does $|Y_i \cap Z_i|$ represent only the number of true positives (i.e. the numerator of equation 1)? $Y_i \cap Z_i$ is the intersection between two sets. It doesn't matter the order of the elements, but only whether the elements are or not in the sets. When you're counting the true positives, the order matters (i.e. you should compare the labels of instance $i$ with the corresponding predictions $i$).

Btw, you also have sets in Python and C++, you don't have to use lists or vectors, which represent ordered sequences.

Thanks for making a comment on Python and C++ I'll make a note of it. Coming back.
if ∩ =  (intersection)  is right way to understand, in a set of [0,0,1,1] ∩ [0,1,0,1] the intersection(∩) here is acting as XNOR operation but according to most of the observed explanations I found that it is an AND operation and this is inconsistent with eq.2


text:
In the policy gradient equation, is $\pi(a_{t} | s_{t}, \theta)$ a distribution or a function?
I am learning about policy gradient methods from the Deep RL Bootcamp by Peter Abbeel and I am a bit stumbled by the math presented. In the lecture, he derives the gradient logarithm likelihood of a trajectory to be
$$\nabla log P(\tau^{i};\theta) = \Sigma_{t=0}\nabla_{\theta}log\pi(a_{t}|s_t, \theta).$$
Is $\pi(a_{t} | s_{t}, \theta)$ a distribution or a function? Because a derivative can only be taken wrt a function. My understanding is that $\pi(a_{t},s_{t}, \theta)$ is usually represented as a distribution of actions over states, since input of a neural network for policy gradient would be the $s_t$ and output would be $\pi(a_t, s_t)$, using model weights $\theta$.

First, the derivative is usually taken with respect to a variable (input) of the function. Hence the notation $\frac{df}{dx}$ for some function $f(x)$.
If you look at your equation more carefully
$$\nabla log P(\tau^{i};\theta) = \Sigma_{t=0}\nabla_{\theta}log\pi(a_{t}|s_t, \theta).$$
You will see that the gradient is taken with respect to $\theta$, which are the parameters (i.e. a vector) e.g. of your neural network, that is, $\nabla_{\theta}$.
In this case, it doesn't really matter whether $\pi$ represents a distribution or not (for some specific value of $\theta$), but you're right that $\pi$ often represents a probability distribution over the possible actions (given a specific state). In any case, $\pi$ is a function of the parameters $\theta$ (i.e. in the case of a distribution, $\pi_{\theta}$ is a family of distributions for all possible values of $\theta$), i.e. if you change $\theta$ the outputs of $\pi$ also change, so you can take the derivative of it with respect to $\theta$.


text:
How do I optimize the number of filters in a convolution layer?
I’m trying to figure out how to write an optimal convolutional neural network with respect to maximizing and minimizing filters in a convolution 2D layer. This is my thinking and I’m not sure if it's correct.
If we have a dataset of 32x32 images, we could start with a Conv2D layer, filter of 3x3 and stride of 1x1. Therefore the maximum times this filter would be able to fit into the 32 x 32 images would be 30 times e.g. newImageX * newImageY
newImageX = (imageX – filterX + 1)
newImageY = (imageY – filterY + 1)

Am I right in thinking that because there are only newImageX * newImageY patterns in the 32 x 32 image, that the maximum amount of filters should be newImageX * newImageY, and any more would be redundant?
So, the following code is the maximum possible filters given that we have 3x3 filter, 1x1 stride and 32x32 images?
Conv2D((30*30), kernel=(3,3), stride=(1,1), input_shape=(32,32,1))

Is there any reason to go above 30*30 filters, and are there any reasons to go below this number, assuming that kernel, stride and input_shape remain the same?
If you knew you were looking for one specific filter, would you have to use the maximum amount of filters to ensure that the one you were looking for was present, or can you include it another way?

Am I right in thinking that because there are only newImageX * newImageY patterns in the 32 x 32 image, that the maximum amount of filters should be newImageX * newImageY, and any more would be redundant?

Your assumption is wrong. If you have a $32 \times 32$ images (so consider only grayscale images), then you have $256^{32 \times 32}$ possible patterns (i.e. a huge number of possible patterns!), given that, for each pixel, you can have $256$ values, and there are $32\times 32$ pixels. Of course, for a human, many of these patterns may be very similar to each other (or even indistinguishable), but the way you measure similarity, in this case, is another hard problem. Hence also the difficulty in choosing the number of filters.

Later, I will eventually add more details about the possible "optimal" number of filters.

authors: Станислав Сычик