text:
Understanding how to calculate $P(x|c_k)$ for the Bernoulli naïve Bayes classifier
I'm looking at the Bernoulli naïve Bayes classifier on Wikipedia and I understand Bayes theorem along with Gaussian naïve Bayes. However, when looking at how $P(x|c_k)$ is calculated, I don't understand it. The Wikipedia page says its calculated as follows
$$P(x|c_k) = \prod^{n}_{i=1} p^{x_i}_{ki} (1-p_{ki})^{(1-x_i)}. $$
They mention that $p_{ki}$ is the probability of class $c_k$ generating the term $x_i$, does that mean $P(x|c_k)$? Because if so then that doesn't make sense since to calculate that we need to have calculated it already. So what is $p_{ki}$?
And in the first part, after the product symbol, are they raising this probability to the power pf $x_i$ or does that again just mean 'probability of class $c_k$ generating the term $x_i$'?
I also don't understand the intuition behind why or how this calculates $P(x|c_i)$.

Isn't $x$ a vector $[x_1, \dots, x_n]$?

@nbro Yes it's a feature vector although it can also be a matrix for larger data sets where the amount of rows is the 'number of samples', and the amount of columns is the 'number of features'.

Therefore, that probability you're analysing can also be written as $P(x_1, \dots, x_n \mid c_k)$. Does this make things clearer? Now, what happens if you assume the observations are independent of each other?

@nbro I understand that. The problem I was facing is understanding what $p_{ki}$ is. And when in the formula they write $p_{ki}^{x_i}$ are they raising it to the power of $x_i$ or is it just saying that we're calculating $p_{ki}$ given $x_i$?

It's been a while since I had to deal with naive Bayes. so I cannot help you more unless I read about it again (maybe I'll do that later), but have a look at the binomial distribution. The formulas look very similar. Anyway, I am pretty sure that this is an exponentiation operation, otherwise, what would the exponent $1 - x_i$ mean? In any case, $x_i$ must be encoded as a number. Maybe they mean $1 - i$ (rather than $1 - x_i$). Similarly, maybe they meant $p_{ki}^i$.

@nbro That exponent thing makes sense, just wanted to check that as well. And thanks for the help you're able to give at the moment. And I like I said I understand the rest and how to implement it except for what $p_{ki}$ is and how to calculate it.

Bernoulli naïve Bayes
$P(x \mid c_k) = \prod^{n}_{i=1} p^{x_i}_{ki} (1-p_{ki})^{(1-x_i)}$
Let's examine the example of document classification.
Let K different text classes and n different terms that our vocabulary contains. $x_i$  are boolean variables (0, 1) expressing if the $i^{th}$ term exists in document $\mathbf{x}$. $\mathbf{x}$ is a vector of dimension $n$.
$P(x \mid c_k)$ is the probability that given the class $k$, document $\mathbf{x}$ to be generated. The equation uses a common trick to represent a multivariate Bernoulli event model, taking into account that when $x_i = 1$, then $1 - x_i = 0$ and inversely. In other words, for each term, it takes the probability that the document does contain this term or it does not.
$p_{ki}$ is the probability of class $c_k$ generating the term $x_i$, that is it could be the prior probability a document that belongs to $k$ class contains this term of the vocabulary.


text:
How can blackjack be formulated as a Markov decision process?
I am reading sutton barton's reinforcement learning textbook and have come across the finite Markov decision process (MDP) example of the blackjack game (Example 5.1).
Isn't the environment constantly changing in this game? How would the transition probabilities be fixed in such an environment, when both you and the dealer draw cards?

Isn't the environment constantly changing in this game?

The current state of the agent and the environment is constantly changing as you play, but not necessarily the transition probabilities. For simplicity, you may assume that the transition probabilities do not change (e.g. if the dealer and the deck are the same every time you play).

How would the transition probabilities be fixed in such an environment, when both you and the dealer draw cards?

The actions of the dealer will be incorporated into the transition probabilities of the environment. Whenever the player (or RL agent) takes an action, then it will receive a reward, according to the reward function of the environment (the rules of the game), and the agent and the environment will be moved to the next state, according to the transition probabilities, which do not have to change for stochasticity to occur in the environment. In fact, these transition probabilities already incorporate this stochasticity.
Also, if even the environment was changing, you could still model your problem as an MDP, but the MDP would change accordingly.
The example 5.1 of the book (that you mention) actually explains in detail how to formulate this game as a finite MDP.

In fact the book ignores drawn cards and (unrealistically) assumes drawing with replacement, because it is simpler to formulate. Drawing without replacemant still results in a valid MDP as you suggest, but the state model becomes far more complex for little gain when used as a teaching example

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How do you find the homography matrix given 4 points in both images?
I want to understand the process of finding a homography matrix given 4 points in both images. I am able to do that in python OpenCV, but I wonder how it works behind the scenes.
Suppose I have points $p_1, p_2, p_3, p_4$ in the first image and $p'_1, p'_2, p'_3, p'_4$ in the second. How am I going to generate the homography matix given these points.

To understand homographies and how to find them, you will need a good dose of projective geometry. I will briefly describe some preliminary concepts that you need to know before trying to find the homography, but don't expect to understand all these concepts with one reading iteration and only by reading this answer, if you are not familiar with them, especially, if you don't even know what homogenous coordinates are. For more details, I suggest you read the book Multiple view geometry in computer vision (2004) by Richard Hartley and Andrew Zisserman, in particular, chapter 4.
The projective space $\mathbb{P}^2$
$\mathbb{P}^2$ is the projective space of $\mathbb{R}^2$, so it is $\mathbb{R}^2$ augmented with lines and points at infinity. All points and lines of $\mathbb{P}^2$ actually belong to $\mathbb{R}^3$, i.e. they are vectors of three components, because they are homogenous representations of the counterparts in $\mathbb{R}^2$. To emphasize, in $\mathbb{P}^2$, both points and lines can be represented by a vector in $\mathbb{R}^3$, which is the homogenous representation of the counterpart vector in $\mathbb{R}^2$ (if it exists, e.g. points at infinity do not exist in $\mathbb{R}^2$).
What is a homography?
A homography (aka projectivity, collineation or projective transformation) is an invertible map $h$ from the projective space $\mathbb{P}^2$ to itself, $$h: \mathbb{P}^2 \rightarrow \mathbb{P}^2,$$ such that $\mathbf{x}_1, \mathbf{x}_2, \mathbf{x}_3$ lie on the same straight-line if and only if $h(\mathbf{x}_1), h(\mathbf{x}_2), h(\mathbf{x}_3)$ do.
This property is called collinearity. In practice, this property means that projective transformations can map straight-lines in one image to straight-line to another image, but it cannot map e.g. straight-lines to parabolas (and vice-versa). So, if you have a distorted image, a homography cannot convert it to a non-distorted image.
All and only linear maps in $\mathbb{P}^2$ are homographies
All linear maps in $\mathbb{P}^2$ are homographies and only linear maps can be homographies. Hence, when you want to find a homography, you are looking for a linear map in $\mathbb{P}^2$.
A homography can be represented as $3 \times 3$ matrix
Given that homographies are linear maps they can be represented as an invertible matrix
$$\mathbf{H}
=
\begin{bmatrix}
h_{11} & h_{12} & h_{13} \\
h_{21} & h_{22} & h_{23} \\
h_{31} & h_{32} & h_{33}
\end{bmatrix}
\in \mathbb{R}^{3 \times 3},
$$
such that, $\forall \mathbf{x} \in \mathbb{P}^2$, the following equation holds
$$
h(\mathbf{x}) = \mathbf{H}\mathbf{x}
$$
Homographies map points in $\mathbb{P}^2$ to points in $\mathbb{P}^2$
Given that homographies map points in $\mathbb{P}^2$ to other points in $\mathbb{P}^2$ and there's a matrix $\mathbf{H} \in \mathbb{R}^{3 \times 3}$ for each homography $h$, then it follows that, $\forall \mathbf{x} \in \mathbb{P}^2$,
\begin{align}
\mathbf{x}'
&=
\mathbf{H}\mathbf{x} \\
\begin{bmatrix}
\mathbf{x}'_1 \\ \mathbf{x}'_2 \\ \mathbf{x}'_3
\end{bmatrix}
&=
\begin{bmatrix}
h_{11} & h_{12} & h_{13} \\
h_{21} & h_{22} & h_{23} \\
h_{31} & h_{32} & h_{33}
\end{bmatrix}
\begin{bmatrix}
\mathbf{x}_1 \\ \mathbf{x}_2 \\ \mathbf{x}_3
\end{bmatrix}
\tag{1}\label{1}
\end{align}
where $\mathbf{x}' \in \mathbb{P}^2$, for some homography $\mathbf{H} \in \mathbb{R}^{3 \times 3}$.
In practice, this means that you can transform points in an image $I_1$ to points in image $I_2$ by matrix multiplication.
A homography has $8$ degrees of freedom
Although the matrix $\mathbf{H}$ has $9$ entries, it has only $8$ degrees of freedom, which means that, in practice, there are only $8$ variables that you need to find. This property comes from the fact that only the ratio between the elements of $\mathbf{H}$ actually counts. This means that equation \ref{1} can actually be written as
\begin{align}
\mathbf{x}' = \lambda \mathbf{H}\mathbf{x} \tag{2}\label{2}
\end{align}
for all $\lambda \in \mathbb{R} \setminus \{0 \}$.
How to estimate a homography from point correspondences?
We want to estimate a homography $\mathbf{H} \in \mathbb{R}^{3 \times 3}$ from point-to-point correspondences $\{(\mathbf{x}^i, \mathbf{x}'^i) \}_{i=1}^N$, where $N \geq 4$, such that $\mathbf{x}'^i = \mathbf{H}\mathbf{x}^i, \forall i$.
First, recall that the matrix $\mathbf{H}$ has $8$ degrees of freedom (i.e. variables we want to find).
Each point-to-point correspondence represents $2$ constraints
Each point-to-point correspondence $(\mathbf{x}^i, \mathbf{x}'^i)$ accounts for $2$ constraints, i.e. $\mathbf{H}\mathbf{x}^i$ maps to the point $\mathbf{x}'^i$, which has $2$ degrees of freedom (even if it's defined by 3 components) because it's represented in homogenous coordinates and so, as for equation \ref{2}, it's defined "up to scale". In other words, these $2$ degrees of freedom represent $2$ constraints.
$4$ point-to-point correspondences are necessary to estimate a homography
Given that $1$ point-to-point correspondence represents $2$ constraints, then $4$ point-to-point correspondences corresponds to $8$ constraints. Given this and given that homographies have $8$ degrees of freedom, at least $4$ point-to-point correspondences are necessary to estimate a homography.
The equation \ref{2} can actually be written as
$$
\mathbf{x}' \times \lambda \mathbf{H}\mathbf{x} = \mathbf{0} \label{3} \tag{3}
$$
where $\times$ is the cross-product.
Direct linear transformation
If you manipulate equation \ref{3}, then you will end up with another equation (if you ignore the scaling factor $\lambda$, which we can ignore because the homography converts points independently of their magnitude)
$$
\mathbf{A}_i \mathbf{h} = \mathbf{0}
$$
where $\mathbf{A}_i \in \mathbb{R}^{2 \times 9}$ is the design matrix (i.e. the matrix that contains the input data to estimate the homography) that contains the elements of the the point-to-point correspondence $(\mathbf{x}^i, \mathbf{x}'^i)$ and $\mathbf{h} \in \mathbb{R}^{9}$ is a vector that contains the unknown elements of $\mathbf{H}$. The details of this manipulation can be found in section 4.1 of the book Multiple view geometry in computer vision.
Now, the idea is that you can vertically stack $N \geq 4$ equations of the form $\mathbf{A}_i \mathbf{h} = \mathbf{0}$ to build the final linear system that you would like to solve
$$\mathbf{A} \mathbf{h} = \mathbf{0} \tag{4}\label{4}$$
where $\mathbf{A} \in \mathbb{R}^{2N \times 9}$.
The solution to this system is the vector $\mathbf{h} \in \mathbb{R}^{9}$, that is, your homography!
If you know something about linear algebra, you know that the solutions to $\mathbf{A} \mathbf{h} = \mathbf{0}$ are elements of the null space of $\mathbf{A}$.
Then, to find $\mathbf{h}$, you will typically use singular value decomposition (SVD). See e.g. How is the null space related to singular value decomposition? for a possible explanation of why you can use SVD to find an element of the null space.
This algorithm (which is described on page 91 of the cited book, i.e. Algorithm 4.1) is called direct linear transformation.

A wonderful explanation .

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What is the difference between training and testing in reinforcement learning?
In reinforcement learning (RL), what is the difference between training and testing an algorithm/agent? If I understood correctly, testing is also referred to as evaluation.
As I see it, both imply the same procedure: select an action, apply to the environment, get a reward, and next state, and so on. But I've seen that, e.g., the Tensorforce RL framework allows running with or without evaluation.

As RL is not a supervised algorithm (it's a third type of ML algorithms), you can't the same expectation from testing and training algorithm here, as well as, supervised algorithms.

@OmG OK. So, as I understand from you, this concept does not apply to RL?

Not the same as the supervised learning.

What is reinforcement learning?
In reinforcement learning (RL), you typically imagine that there's an agent that interacts, in time steps, with an environment by taking actions. On each time step $t$, the agent takes the action $a_t \in \mathcal{A}$ in the state $s_t \in \mathcal{S}$, receives a reward (or reinforcement) signal $r_t \in \mathbb{R}$ from the environment and the agent and the environment move to another state $s_{t+1} \in \mathcal{S}$, where $\mathcal{A}$ is the action space and $\mathcal{S}$ is the state space of the environment, which is typically assumed to be a Markov decision process (MDP).
What is the goal in RL?
The goal is to find a policy that maximizes the expected return
(i.e. a sum of rewards starting from the current time step). The policy that maximizes the expected return is called the optimal policy.
Policies
A policy is a function that maps states to actions. Intuitively, the policy is the strategy that implements the behavior of the RL agent while interacting with the environment.
A policy can be deterministic or stochastic. A deterministic policy can be denoted as $\pi : \mathcal{S} \rightarrow \mathcal{A}$. So, a deterministic policy maps a state $s$ to an action $a$ with probability $1$. A stochastic policy maps states to a probability distribution over actions. A stochastic policy can thus be denoted as $\pi(a \mid s)$ to indicate that it is a conditional probability distribution of an action $a$ given that the agent is in the state $s$.
Expected return
The expected return can be formally written as
$$\mathbb{E}\left[ G_t \right] = \mathbb{E}\left[ \sum_{i=t+1}^\infty R_i \right]$$
where $t$ is the current time step (so we don't care about the past), $R_i$ is a random variable that represents the probable reward at time step $i$, and $G_t = \sum_{i=t+1}^\infty R_i $ is the so-called return (i.e. a sum of future rewards, in this case, starting from time step $t$), which is also a random variable.
Reward function
In this context, the most important job of the human programmer is to define a function $\mathcal{R}: \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$, the reward function, which provides the reinforcement (or reward) signal to the RL agent while interacting with the environment. $\mathcal{R}$ will deterministically or stochastically determine the reward that the agent receives every time it takes action $a$ in the state $s$. The reward function $R$ is also part of the environment (i.e. the MDP).
Note that $\mathcal{R}$, the reward function, is different from $R_i$, which is a random variable that represents the reward at time step $i$. However, clearly, the two are very related. In fact, the reward function will determine the actual realizations of the random variables $R_i$ and thus of the return $G_i$.
How to estimate the optimal policy?
To estimate the optimal policy, you typically design optimization algorithms.
Q-learning
The most famous RL algorithm is probably Q-learning, which is also a numerical and iterative algorithm. Q-learning implements the interaction between an RL agent and the environment (described above). More concretely, it attempts to estimate a function that is closely related to the policy and from which the policy can be derived. This function is called the value function, and, in the case of Q-learning, it's a function of the form $Q : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$. The name $Q$-learning derives from this function, which is often denoted as $Q$.
Q-learning doesn't necessarily find the optimal policy, but there are cases where it is guaranteed to find the optimal policy (but I won't dive into the details).
Of course, I cannot describe all the details of Q-learning in this answer. Just keep in mind that, to estimate a policy, in RL, you will typically use a numerical and iterative optimization algorithm (e.g. Q-learning).
What is training in RL?
In RL, training (also known as learning) generally refers to the use of RL algorithms, such as Q-learning, to estimate the optimal policy (or a value function)
Of course, as in any other machine learning problem (such as supervised learning), there are many practical considerations related to the implementation of these RL algorithms, such as

Which RL algorithm to use?
Which programming language, library, or framework to use?

These and other details (which, of course, I cannot list exhaustively) can actually affect the policy that you obtain. However, the basic goal during the learning or training phase in RL is to find a policy (possibly, optimal, but this is almost never the case).
What is evaluation (or testing) in RL?
During learning (or training), you may not be able to find the optimal policy, so how can you be sure that the learned policy to solve the actual real-world problem is good enough? This question needs to be answered, ideally before deploying your RL algorithm.
The evaluation phase of an RL algorithm is the assessment of the quality of the learned policy and how much reward the agent obtains if it follows that policy. So, a typical metric that can be used to assess the quality of the policy is to plot the sum of all rewards received so far (i.e. cumulative reward or return) as a function of the number of steps. One RL algorithm dominates another if its plot is consistently above the other. You should note that the evaluation phase can actually occur during the training phase too. Moreover, you could also assess the generalization of your learned policy by evaluating it (as just described) in different (but similar) environments to the training environment [1].
The section 12.6 Evaluating Reinforcement Learning Algorithms of the book Artificial Intelligence: Foundations of Computational Agents (2017) by Poole and Mackworth provides more details about the evaluation phase in reinforcement learning, so you should probably read it.
Apart from evaluating the learned policy, you can also evaluate your RL algorithm, in terms of

resources used (such as CPU and memory), and/or
experience/data/samples needed to converge to a certain level of performance (i.e. you can evaluate the data/sample efficiency of your RL algorithm)
robustness/sensitivity (i.e., how the RL algorithm behaves if you change certain hyper-parameters); this is also important because RL algorithms can be very sensitive (from my experience)

What is the difference between training and evaluation?
During training, you want to find the policy. During the evaluation, you want to assess the quality of the learned policy (or RL algorithm). You can perform the evaluation even during training.

Here is another answer that is worth reading too.

Reinforcement Learning Workflow
The general workflow for using and applying reinforcement learning to solve a task is the following.

Create the Environment
Define the Reward
Create the Agent
Train and Validate the Agent
Deploy the Policy

Training

Training in Reinforcement learning employs a system of rewards and penalties to compel the computer to solve a problem by itself.
Human involvement is limited to changing the environment and tweaking the system of rewards and penalties.
As the computer maximizes the reward, it is prone to seeking unexpected ways of doing it.
Human involvement is focused on preventing it from exploiting the system and motivating the machine to perform the task in the way expected.
Reinforcement learning is useful when there is no “proper way” to perform a task, yet there are rules the model has to follow to perform its duties correctly.
Example: By tweaking and seeking the optimal policy for deep reinforcement learning, we built an agent that in just 20 minutes reached a superhuman level in playing Atari games.
Similar algorithms, in principle, can be used to build AI for an autonomous car.

Testing

Debugging RL algorithms is very hard. Everything runs and you are not sure where the problem is.
To test if it worked well, if the trained agent is good at what it was trained for, you take your trained model and apply it to the situation it is trained for.
If it’s something like chess or Go, you could benchmark it against other engines (say stockfish for chess) or human players.
You can also define metrics for performance, ways of measuring the quality of the agent’s decisions.
In some settings (e.g a Reinforcement Learning Pacman player), the game score literally defines the target outcome, so you can just evaluate your model’s performance based on that metric.

The goal of the reinforcement learning (RL) is to use data obtained via interaction with the environment to solve the underlying Markov Decision Process (MDP). "Solving the MDP" is tantamount to finding the optimal policy (with respect to the MDP's underlying dynamics which are usually assumed to be stationary).
Training is the process of using data in order to find the optimal policy.
Testing is the process of evaluating the (final) policy obtained by training.
Note that, since we're generally testing the policy on the same MDP we used for training, the distinction between the training dataset and the testing set is no longer as important as it is the case with say supervised learning. Consequently, classical notions of overfitting and generalization should be approached from a different angle as well.

If you want, you can do training and testing in RL.
Exactly the same usage, training for building up a policy, and testing for evaluation.
In supervised learning, if you use test data in training, it is like cheating. You cannot trust the evaluation. That's why we separate train and test data.
The Objective of RL is a little different.
RL trying to find the optimal policy. Since RL collects the information by doing, while the agent explores the environment (for more information), there might be a loss in the objective function. But, it might be inevitable for a better future gain.
Multi-arm bandit example,  If there are 10 slot machines. They will return random amounts of money. They have different expected returns. I want to find the best way to maximize my gain. easy, I have to find the machine with the greatest expected return and use only the machine. How to find the best machine?
If we have a training and testing (periods),
For example, I will give you an hour of the training period, so it doesn't matter if you lose or how much you earn. And in the testing period, I will evaluate your performance.
What would you do? In the training period, you will try as much as possible, without considering the performance/gain. And in the testing period, you will use only the best machine you found.
This is not a typical RL situation. RL is trying to find the best way, Learning by doing. All the results while doing are considered.
Suppose that I tried all 10 machines once each. And, the No.3 machine gave me the most money. But I am not sure that it is the best machine, because all the machines provide a RANDOM amount. If I keep using the No.3 machine, it might be a good idea, because according to the information so far, it is the best machine. However, You might miss the better machine if you don't try other machines due to randomness. But if you try other machines, you might lose an opportunity to earn more money.
What should I do?  This is a well-known Exploration and Exploitation trade-off in RL.
RL trying to maximize the gain including the gains right now and the gains in the future.
In other words, the performance during training is also considered as its performance.
That's why RL is not unsupervised nor supervised learning.
However, in some situations, you might want to separate training and testing. RL is designed for an agent who interacts with the environment. However,  in some cases, (for example), rather than having an interactive playground, you have data of interactions. The formulation would be a little different in this case.


text:
Is there any research on models that provide uncertainty estimation?
Is there any research on machine learning models that provide uncertainty estimation?
If I train a denoising autoencoder on words and put through a noised word, I'd like it to return a certainty that it is correct given the distribution of data it has been trained on.
Answering these questions or metrics for uncertainty are both things I am curious about. Just general ways for models to just say "I'm not sure" when it receives something far outside the inputs it's been trained to approximate.

Yes, there is some research on this topic. It's often called Bayesian machine learning or Bayesian deep learning (but I don't think this is a good name because there are models that aren't really based on a direct application of Bayesian statistics). Some ML/DL models that provide some kind of uncertainty estimation are, for example, Monte Carlo Dropout (MC dropout) or Bayesian neural networks. In theory, these techniques look promising. In practice, I don't think they are the ultimate solution to the problem of uncertainty estimation in deep learning. In fact, e.g. in the case of Bayesian neural networks, they have some disadvantages, such as more parameters to tweak and save.

Do you know of any practical examples where the disadvantages you mention really get clear?

@Mathy Off the top of my head, no. I only worked with relatively small BNNs (1 million parameters? I don't remember, honestly) in the past (it was already several months ago when I worked with them - I don't know the SOTA now - But probably nothing changed much). But let's say that in the future models are of size like GPT-3's size, then this could be a problem, as you may need to double the number of parameters, so I assume that this will cause issues, in addition to the very likely longer training times to fit all those parameters.


text:
Are these two definitions of the state-action value function equivalent?
I have been reading the Sutton and Barto textbook and going through David Silvers UCL lecture videos on YouTube and have a question on the equivalence of two forms of the state-action value function written in terms of the value function.
From Question 3.13 of the textbook I am able to write the state-action value function as
$$q_{\pi}(s,a) = \sum_{s',r}p(s',r|s,a)(r + \gamma v_\pi(s')) = \mathbb{E}[r + \gamma v_\pi(s')|s,a]\;.$$
Note that the expectation is not taken with respect to $\pi$ as $\pi$ is the conditional probability of taking action $a$ in state $s$. Now, in David Silver's slides for the Actor-Critic methods of the Policy Gradient lectures, he says that
$$\mathbb{E}_{\pi_\theta}[r + \gamma v_{\pi_\theta}(s')|s,a] = q_{\pi_\theta}(s,a)\;.$$
Are these two definitions equivalent (in expectation)?

The definition of the state-action value function is always the same.
Your definition is correct, as $q_{\pi}(s,a)$ is conditioned on $a$, so you don't need to write $q_{\pi}(s,a)$ as an conditional expectation that depends on $\pi$. In fact, the conditional expectation is taken wrt the probability distribution $p(s',r|s,a)$. However, you need the subscript $\pi$ in $v_\pi(s')$ because $v_\pi(s')$ is defined as the expected return by following $\pi$ starting in $s'$.
If you didn't write $q_{\pi}(s,a)$ in terms of $v_\pi(s')$, then you could write $q_{\pi}(s,a)$ as an expectation that depends on $\pi$, because, in that case, $q_{\pi}(s,a)$, is defined as an expectation of the return (after having taken $a$ in $s$), which depends on $\pi$ (see equation 3.3 of Sutton & Barto book, p. 58). Of course, this way of writing $q_{\pi}(s,a)$ is equivalent to writing it in terms of $v_\pi(s')$.
I think David Silver's notation might be an abuse of notation. In his equation, the policy is parametrized by $\theta$, so I think he wants to emphasize that you will estimate the state-action value function (critic) based on $\pi_\theta$ (actor). Alternatively, he uses $\pi_\theta$ as a subscript of $\mathbb{E}_{\pi_\theta}$ to emphasize that the future return starting in $s'$, after having taken action $a$ in $s$, still depends on $\pi_\theta$.


text:
How does the AlphaGo Zero policy decide what move to execute?
I was going through the AlphaGo Zero paper and I was trying to understand everything, but I just can't figure out this one formula:
$$
\pi(a \mid s_0) = \frac{N(s_0, a)^{\frac{1}{\tau}}}{\sum_b N(s_0, b)^{\frac{1}{\tau}}}
$$
Could someone decode how the policy makes decisions following this formula? I pretty much understood all of the other parts of the paper, and also the temperature parameter is clear to me.
It might be a simple question, but can't figure it out.

The formula in question uses a function N(state, action) that defines a visit count of a state-action pair (introduced on page 3). To describe how it is used, lets first describe the steps of AlphaGo Zero as a whole.
There are 4 "phases" to the Monte-Carlo tree search in AlphaGo Zero as depicted in Figure 2. The first 3 expand and update the tree and together are the "search" in Monte-Carlo tree "search" in AlphaGo Zero.

Select an edge (action) in the tree with maximum action-value Q (plus upper confidence bound U)
Expand and evaluate the leaf node using the network
Backup Action-values Q are updated to track the evaluations of the Value in the subtree.
Play - After the "search" is complete, the search probabilities are returned proportional to the visitation count of the nodes of the tree.*

* This is where the formula in question comes into play (pun intended). During the "search", nodes that looked good were expanded on and thus their visitation counts were updated. So the formula is essentially describing this logic:
Good nodes have higher counts, so choose the nodes with higher counts often
But what if there is a really good node but it hasn't been visited a lot so its not chosen?
This is where the temperature parameter comes in:

If the temperature is 1, this selects moves proportionally to their visit counts.
If the temperature is 0 (not actually but rather in infinitesimal that approaches 0), and with some added noise, it "ensures that all moves may be tried, but the search may still overrule bad moves."

So altogether, the formula is saying : Pick good things most of the time
The mathematical evaluation of the formula is described below:
AlphaGo Zero defines the probability of each action (aka the policy) by that formula. If there are 3 nodes, A, B and C, and they have each been visited 10, 70, and 20 times (100 times in total), respectively, then the probability of taking those actions is:

P(A) = 10/100 = .10
P(B) = 70/100 = .7
P(C) = 20/100 = .2

While I get that "good nodes have higher counts", it feels to me that the better way to pick the best move should have been argmax Q(s,a) which we already have. What do you think could be the reasoning for choosing count over the best Q?

Intuitively:

Larger Q means larger probability that the node (s'|s,a) would be chosen. When we selected most visited node, we selected a node with good Q.
More visited count means more accurate estimation. And the chosen node proved itself as a good choice even after more trials than other nodes.
Less computation in some cases (integer/long vs float/double)


text:
Are activation functions applied to feature maps?
If I have a convolutional neural network, and I convolve my input tensor with a kernel, the output is a feature map. Is an activation function then applied to this feature map?
If its an image that is a 2D tensor, would the activation function change every single value on this image?

Yes, it is applied element-wise on every single value of the feature map.
Assuming ReLU as your nonlinearity function, all negative values of the image feature map are set to zero, and the rest of the elements stay unchanged.


text:
How can reinforcement learning be unsupervised learning if it uses deep learning?
I was watching a video in my online course where I'm learning about A.I. I am a very beginner in it.
At one point in the course, the instructor says that reinforcement learning (RL) needs a deep learning model (NN) to perform an action. But for that, we need expected results in our model for the NN to learn how to predict the Q-values.
Nevertheless, at the beginning of the course, they said to me that RL is an unsupervised learning approach because the agent performs the action, receives the response from the environment, and finally takes the more likely action, that is, with the highest Q value.
But if I'm using deep learning in RL, for me, RL looks like a supervised learning approach. I'm a little confused about these things, could someone give me clarifications about them?

Supervised learning
The supervised learning (SL) problem is formulated as follows.
You are given a dataset $\mathcal{D} = \{(x_i, y_i)_{i=1}^N$, which is assumed to be drawn i.i.d. from an unknown joint probability distribution $p(x, y)$, where $x_i$ represents the $i$th input and $y_i$ is the corresponding label. You choose a loss function $\mathcal{L}: V \times U \rightarrow \mathbb{R}$. Then your goal is to minimize the so-called empirical risk
$$R_{\mathcal{D}}[f]=\frac{1}{N} \sum_{i=1}^N \mathcal{L}(x_i, f(x_i)) \tag{0}\label{0}$$
with respect to $f$. In other words, you want to find the $f$ that minimizes the average above, which can also be formally written as
$$
f^* = \operatorname{argmin}_f R[f] \tag{1}\label{1}
$$
The problem \ref{1} is called the empirical risk minimization because it is a proxy problem for the expected risk minimization (but you can ignore this for now).
Reinforcement learning
In reinforcement learning, you typically imagine that there's an agent that interacts, in time steps, with an environment by taking actions. At each time step $t$, the agent takes $a_t$ in the state $s_t$, receives a reward $r_t$ from the environment and the agent and the environment move to another state $s_{t+1}$.
The goal of the agent is to maximize the expected return
$$\mathbb{E}\left[ G_t \right] = \mathbb{E}\left[ \sum_{i=t+1}^\infty R_i \right]$$
where $t$ is the current time step (so we don't care about the past), $R_i$ is a random variable that represents the probable reward at time step $i$, and $G_t = \sum_{i=t+1}^\infty R_i $ is the so-called return (i.e. a sum of future rewards, in this case, starting from time step $t$), which is also a random variable.
In this context, the most important job of the programmer is to define a function $\mathcal{R}(s, a)$, the reward function, which provides the reinforcement (or reward) signal to the RL agent. $\mathcal{R}(s, a)$ will deterministically or stochastically determine the reward that the agent receives every time it takes action $a$ in the state $s$. (Note that $\mathcal{R}$ is different from $R_i$, which is a random variable that represents the reward at time step $i$).
What is the difference between SL and RL?
In RL, you (the programmer) need to define the reward function $\mathcal{R}$ and you want to maximize the expected return. On the other hand, in SL you are given (or you collect) a dataset $\mathcal{D}$, you choose $\mathcal{L}$ in \ref{0}, and the goal is to find the function $f^*$ that minimizes the empirical risk.  So, these have different settings and goals, so they are different!
However, every SL problem can be cast as an RL problem. See this answer. Similarly, in certain cases, you can formulate an RL as an SL problem. So, although the approaches are different, they are related.
Is RL an unsupervised learning approach?
In RL, you do not tell the agent what action it needs to take. You only say that the action that was taken was "bad", "good" or "so so". The agent needs to figure out which actions to take based on your feedback. In SL, you explicitly say that, for this input $x_i$, the output should be $y_i$.
Some people may consider RL is an unsupervised learning approach, but I think this is wrong, because, in RL, the programmer still needs to define the reward function, so RL isn't totally unsupervised and it's also not totally supervised. For this reason, many people consider RL an approach that sits between UL and SL.
What is deep learning?
The term/expression deep learning (DL) refers to the use of deep neural networks (i.e. neural networks with many layers, where "many" can refer to more than 1 or 1000, i.e. it depends on the context) in machine learning, either supervised, unsupervised, or reinforcement learning. So, you can apply deep learning to SL, RL and UL. So, DL is not only restricted to SL.

Nice answer. Since the OP asked about Deep Q-Learning, it might be worth clarifying the role of DL in that algorithm, along the following lines: a popular method for solving RL problems is Q-Learning, where one learns the Q-value function. In some RL problems with large state or action spaces, this function is difficult to compute directly, so we approximate it with a DNN—this is Deep Q-Learning. So there is no essential connection between RL and DL—it's just that some value functions are too complex to learn any other way (as far as we know).

@ptal Your explanation of DQN is correct, but I'd not go that far to say that there's no connection between RL and DL. They are both try to come up with solutions to the general learning problem and they can be combined, so there's a connection in this sense.

Yes, you're right—it would be misleading to say that there is no connection between DL and RL (though I did say "no essential connection," by which I meant that it is possible to do RL without DL, contrary to what OP's instructor seemed to claim).

Reinforcement learning can be confusing in relation to supervised and unsupervised learning, especially for deep learning models. Here is the simple explanation make this clearer:
Supervised Learning
In the case of supervised learning, we are given a function that maps inputs to their desired outputs based on example input-output pairs. The basic setup is like this:

Dataset: There is a dataset with inputs $(x_i)$ and corresponding labels $(y_i)$.
Objective: The objective is to learn a function $(f(x))$ which makes the most accurate possible predictions about the labels $(y_i)$.
Loss Function: You minimize a loss function $(L(f(x), y))$ over the dataset to find the best $(f)$.

Reinforcement Learning
In reinforcement learning, rather than being shown an example of what to do and being told this is the correct action for a given input, the agent has to learn from a scalar reward signal.

Actions and States: At every time step $(t)$ the agent is in state $(s_t)$, takes action $(a_t)$, and moves to a new state $(s_{t+1})$.
Rewards: The agent receives a reward $(r_t)$ after taking an action.
Goal: The goal is to maximize the cumulative reward, called the return $(G_t = \sum_{i=t+1}^\infty R_i)$.

The Role of Deep Learning in RL
Deep learning is commonly used in RL to approximate functions such as the policy (what action to take for a given state) and the value function (expected return from a state). But the presence of deep learning does not fundamentally change RL.
RL vs. Supervised Learning

Feedback: In supervised learning, you give explicit labels to each input. You provide rewards in RL: feedback signals that show the agent how to learn the best actions over time.
Learning Process: RL learns through exploration and exploitation, which is way more dynamic than static datasets used in supervised learning.

Is RL Unsupervised Learning?
Probably, some will argue that RL is unsupervised since it does not depend on labeled data like supervised learning. However, RL is not purely unsupervised because of the following two factors:

Guidance: The reward function is a form of guidance, indicating in some sense what the right thing to do is.
Goal-oriented: The agent learns to achieve an explicit goal—maximizing reward—a kind of supervision.

Reinforcement learning is squarely situated within this learning paradigm:

It is not strictly supervised in that it does not use labeled input-output pairs in a straightforward way.
It is not purely unsupervised because it relies on rewards, a form of feedback that orients learning.

Therefore, RL is often considered a separate machine learning category, which can leverage deep learning techniques to accomplish its goals.
Additional Resources
To gain deeper insights, consider watching the Reinforcement Learning Course by David Silver, which takes a comprehensive approach to understanding the principles and implementations of RL.


text:
Shouldn't the utility function of two-player zero-sum games be in the range $[-1, 1]$?
In Appendix B of MuZero, they say

In two-player zero-sum games the value functions are assumed to be bounded within the $[0, 1]$ interval.

I'm confused about the boundary: Shouldn't the value/utility function be in the range of [-1,1] for two-player zero-sum games?

it can be either. If you consider the lack of reward as "penalty" then getting 0 reward is bad.
if you use a value estimator through a neural network, the range of rewards will dictate the squashing function you use for the output layer

Hi, @nikos if we use [0,1] as the bound for the value functions, then how can the game be zero-sum?

The utility function is valued from the perspective of ONE of the 2 players, so it definitely shouldn't sum to zero! If you train it properly, it should give a larger percentage (probability) to the AI player winning. You should regard "zero sum" in the sense one wins and the other loses

Hi @nikos. The game will return either $+1$ or $-1$ to the agent to indicate whether it wins or not. I get that the agent is unlikely to take actions that cause it to lose, but I cannot see any guarantee that restricts the value function to positive. For example, if the agent is in a state where there is no way to win if its opponent acts optimally. Shouldn't the value function of that state be negative in that case?

yes, if a board state is close to the opponent winning, then its value will be negative. If using [0,1] valuation, the value will be under 0.5. So either way the message will be transmitted

If that's the case, shouldn't the game return 0 to indicate the agent for its loss?

well at the final state, when the game is lost, the value function is 0. At previous states further back, the state value can be interpreted as the probability of winning from that point onwards

In section 3 MuZero Algorithm from the paper, they say "Final outcomes ${ lose, draw, win }$ in board games are treated as rewards $u_t\in { −1, 0, +1 }$ occuring at the ﬁnal step of the episode." If I take it correctly, value function should be $-1$ when the game is lost. Where did I misunderstand?

I haven't read the particular paper that you refer to, but what I've been trying to tell you all along is that the actual range of rewards doesn't matter. If you want [-1,1], that's fine, and [0,1] is also fine, as well as [-1000,1000]. You just have to be consistent

I got that, but this range is not an arbitrary choice; it affects the value of pUCT, which is important to MCTS.

@Maybe Yes, you're right that the range selected can be very important in terms of hyperparameters. But nikos is also correct that the selected range is a fairly arbitrary choice -- in a mathematical sense. Whether we train an agent to optimise values in $[0, 1]$ or in $[-1, 1]$ doesn't really matter mathematically; exactly the same states will be winning or losing, exactly the same policy will be optimal, etc. It's an implementation detail, which mathematically does not really matter. Implementation details can be important for empirical performance though, and indeed for hyperparameters!

Hi @DennisSoemers. I know that these does not matter in a mathematical sense. But I'm afraid that I might understand the idea in a wrong way as I think value function should be in $[-1,1]$. Moreover, for Atari games, the Q value normalization also aims to restrict the Q value in this range($[0,1]$ from the paper).

@Maybe Technically you're completely right that the true, formal definition of "zero sum" would imply that losses are $-1$, draws are $0$, and wins are $+1$. Or at least that it would have to be a range centered on $0$. But the authors may have found the $[0, 1]$ range more convenient to work with for other reasons. I know that range is quite common in MCTS / bandit algorithms literature, and historically all kinds of theoretical analyses and proofs are based on that range. So also many people implemented it that way. And if they did implement it like that, its important to report in the paper

Hi, @DennisSoemers. Did you say that MuZero actually uses reward [0, 1] to indicate the win/loss, even for Go and chess?

@Maybe That's what they write in the paper, right? So yes. At least for the search. Maybe for learning they use a different range, not sure, would have to check. They also actually explain why they chose that $[0, 1]$ range in the sentence after the one you quoted; it ensures that their $Q(s, a)$ values are in the same range as the $P(s, a)$ values, and both of those are combined in the pUCT rule. I suppose that keeping them both in the same range can make hyperparameters slightly more easy to tune (or interpret / understand).

@DennisSoemers Nope, They said the rewards in board games are {-1, 0, +1} in the end of Section 3. But, at the Appendix, they also said the value functions are assumed to be in the range of [0, 1]. That's why I'm confused

@Maybe Ah right, I see. Yeah, so in Section 3 they describe their Neural Network Reinforcement Learning approach. For the purposes of this training algorithm, they use rewards of $-1$, $0$, and $1$ for losses, draws, and wins, respectively. In Appendix B they describe the tree search approach. That's a different component. And for the purposes of that part, they instead use (different) values in $[0, 1]$.

@DennisSoemers Shouldn't the value estimate be in the same range as the reward in board games? I failed to see the difference.

@Maybe It's about the same board games in both cases. It's just, one thing is the learning algorithm (learning values and policies in any board game), and the other thing is the tree search (again in all the same board games). They're mostly separate, and you can use different ranges of values in each of them. I suppose in the parts where these two separate components start "communicating" to each other, they'll make the translation between the two ranges. i.e. if the search uses a value in $[-1, 1]$ from the NN, it can just map that to the $[0, 1]$ range before making use of it.

Let us continue this discussion in chat.


text:
Why does it make sense to study MDPs with finite state and action spaces?

In the standard Markov Decision Process (MDP) formalization of the reinforcement-learning (RL) problem (Sutton & Barto, 1998), a decision maker interacts with an environment consisting of finite state and action spaces.

This is an extract from this paper, although it has nothing to do with the paper's content per se (just a small part of the introduction).
Could someone please explain why it makes sense to study finite state and action spaces?
In the real world, we might not be able to restrict ourselves to a finite number of states and actions! Thinking of humans as RL agents, this really doesn't make sense.

You are dealing with a model -- the real world is too complex for many problems, but if you restrict the scope/domain, they become tractable.

In addition to the reason outlined in the comment, also note that if the state-space and action-space are both finite and of feasible size, tabular methods can be used, and there are some advantages to them (like the existence of convergence guarantees and generally a smaller number of hyperparameters to tune).

Note: I assume you mean, countable Action and State Sets by 'Finite'.
MDP(s) are not exclusive to finite spaces only. They can be used in Continuous/uncountable sets of Action and States too.
Markov Decision Process (MDP) is a tuple $(\mathcal S, \mathcal A, \mathcal P^a_s, \mathcal R^a_{ss'}, \gamma, \mathcal S_o)$ where $\mathcal S$ is a set of States, $\mathcal A$ is the set of actions, $\mathcal P_{s}^a: \mathcal A \times \mathcal S \rightarrow [0, 1]$ is a function that denotes Probability distribution over the states if action $a$ is execuited at state $s$. [1][2]
Where, Q-function is defined as:
$$ Q^\pi (s,a) = \mathbb E_\pi \left [ \sum \limits_{t=0}^{+\infty} \gamma(t)r_t | s_o = s, a_o = a \right] \tag{*}$$
Note that $r_t$ is just special case of Reward function $\mathcal R^a_{ss'}$.
Now, if states and actions are discrete, then, the Q-Table Method[3] which is a state-action matrix helps us to evaluate $Q$ function and optimize efficiency.
Whereas, in cases where the state/action sets are infinite or continuous, Deep Networks are preferred to Approximate $Q$ function. [4].
Q-Learning is Off-Policy method, doesn't require $\pi$ policy function

References:

R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press,
1998.
Alborz Geramifard, Thomas J. Walsh, Stefanie Tellex, Girish Chowdhary, Nicholas Roy and Jonathan P. How. A Tutorial on Linear Function Approximators for Dynamic Programming and Reinforcement Learning. Foundations and Trends (R) in Machine Learning
Vol. 6, No. 4 (2013) 375–454
Andre Violante. Simple Reinforcement Learning: Q-learning, Create a q-table, https://towardsdatascience.com, 2019.
Alind Gupta. Deep Q-Learning, Deep Q-Learning, https://www.geeksforgeeks.org/deep-q-learning/, 2020.

Edit: I'd like to thank @nbro for editing suggestions.

To my knowledge you can't compute or solve an uncountably large MDP numerically. It will need to be discretized in some capacity. The same applies for classic control: you can't optimize over the true functional so you use a discrete approximation to the system and solve that.

I should point out as a matter of practicality, everything is perhaps discrete in the limit. Think about an incredibly fine grained MDP to the point quantum effects start to dominate and the idea of "position" no longer exists.


text:
What are some resources with exercises related to neural networks?
I am asking for a book (or any other online resource) where we can solve exercises related to neural networks, similar to the books or online resources dedicated to mathematics where we can solve mathematical exercises.

The book Grokking Deep Learning, by Andrew Trask (a PhD student at Oxford University and a research scientist at DeepMind), a wonderful, clean, and plain-English discussion of the basic mechanics that go on under the hood of neural networks - from data flow to updating of weights.  It is written without a slant on normally-wonky math, the concepts are presented and then advanced at a digestible pace for anyone.
Here are a few more possibly useful resources.

Neural Networks: Playground Exercises
Getting Started With Deep Learning: Convolutional Neural Networks
Deep learning focuses on practical aspects of deep learning
First lab assignment in Deep learning
Deeplearning stanford

There are actually quite a few. Personally I would say these courses have high quality and strong  focus on practice:

Standford computer vision cs231. Check the assignments materials on this page. This course has good explanation/exercises of how generally neural nets and backprop works.
Fastai course notebooks. You can listen to the lectures as well, but notebooks are quite self-containing
Practical reinforcement learning course, if you interested in NN application for RL

One of the most famous books dedicated to neural networks is Neural Networks - A Systematic Introduction (1996) by Raul Rojas. Most chapters end with a series of exercises that test your understanding of the material. For example, in chapter 14 Stochastic Networks, one of the exercises is

Solve the eight queens problem using a Boltzmann machine. Define the
  network's weights by hand.

This should give you a sense of the type of exercise that you will find in this book.


text:
Which reward function works for recommendation systems using knowledge graphs?
I've been reading this paper on recommendation systems using reinforcement learning (RL) and knowledge graphs (KGs).
To give some background, the graph has several (finitely many) entities, of which some are user entities and others are item entities. The goal is to recommend items to users, i.e. to find a recommendation set of items for every user such that the user and the corresponding items are connected by one reasoning path.
I'm attaching an example of such a graph for more clarity (from the paper itself) -

In the paper above, they say

First, we do not have pre-defined targeted items for any user, so it is not applicable to use a binary reward indicating whether the user interacts with the item or not. A better design of the reward function is to incorporate the uncertainty of how an item is relevant to a user based on the rich heterogeneous information given by the knowledge graph.

I'm not able to understand the above extract, which talks about the reward function to use - binary, or something else. A detailed explanation of what the author is trying to convey in the above extract would really help.

Are you familiar with the basic concept of RL, including reward functions?

Yes, I am familiar with it.

I didn't read this paper, but the authors probably mean that, given that you don't know a priori (or beforehand) if a user will buy or not an item, it probably doesn't make sense to give a reward of 1 whenever the system recommends an item that the user eventually buys. It probably makes more sense to associate a stochastic reward to recommendations of the system based on your knowledge encoded in the KG. By "stochastic reward" I mean something that isn't deterministic. Maybe later I will investigate that paper and provide a more solid answer, but I can't assure you.

You're absolutely right, @nbro
I've added an answer that I found convincing using the details I uncovered from the latter half of the paper.

I found the answer further into the paper (section 3.2 Formulation as Markov Decision Process)! I'll post it here for everyone.

Given any user, there is no pre-known targeted item in the KGRE-Rec problem, so it is unfeasible to consider binary rewards indicating whether the agent has reached a target or not. Instead, the agent is encouraged to explore as many “good” paths as possible. Intuitively, in the context of recommendations, a “good” path is one that leads to an item that a user will interact with, with high probability. To this end, we consider to give a soft reward only for the terminal state $s_{T}=\left(u, e_{T}, h_{T}\right)$ based on another scoring function $f(u, i)$. The terminal reward $R_{T}$ is defined as
$$
R_{T}= \begin{cases}\max \left(0, \frac{f\left(u, e_{T}\right)}{\max _{i \in I} f(u, i)}\right), & \text { if } e_{T} \in \mathcal{I} \\ 0, & \text { otherwise }\end{cases}
$$
where the value of $R_{T}$ is normalized to the range of $[0,1] . f(u, i)$ is also introduced in the next section.

Details about $f(u,i)$, the scoring function, can be found in the paper.

For clarification, a state is defined as a tuple $(u,e_t,h_t)$, where $u$,$e_t$,$h_t$ represent the starting user identity, entity the agent has reached at step $t$, and history prior to step $t$.

to incorporate the uncertainty... I'm not able to understand the above extract,

usually, uncertainty is difined by variance... therefore you can add some noise - see here

Moreover, the concept of natural noise is an emerging trend that is
related to inconsistent behaviour of users

-> read Conclusion

the results obtained from experimental section indicate that it is
possible to improvethe accuracy of recommendation methods through the
detection and correction of noisy ratings

for details - read the whole...
though algorithms can differ... another possible implementation is described here... it's up to data-analyst what tools for analysis to choose


text:
Shouldn't expected return be calculated for some faraway time in the future $t+n$ instead of current time $t$?
I am learning RL for the first time. It may be naive, but it is a bit odd to grasp this idea that, if the goal of RL is to maximize the expected return, then shouldn't the expected return be calculated for some faraway time in the future ($t+n$) instead of current time $t$? It is because we are building our system for the future using current information. ( I am coming from machine learning background and this makes more sense to me).
Generally, the expected return is:
$$\mathbb{E}[G_t] = \mathbb{E}[ R_{t+1} + R_{t+2} + R_{t+3}+... R_{t+n}]$$
However, shouldn't the expected return be:
$$\mathbb{E}[G_{t+n}] = \mathbb{E}[R_{t+1} + R_{t+2} + R_{t+3}+... R_{t+n-1}]$$

shouldn't the expected return be calculated for some faraway time in the future (+) instead of current time $t$?

This is partly a notation issue, but $G_t$ is already the future sum of rewards as seen by the first (and correct) equation in your question. You don't actually know the value of any individual return $g_t$* until after $t+n$. However, you can predict the expected value $\mathbb{E}[G_t]$ provided the environment and policy remain consistent.
Your second equation is pretty similar to the first one, which is why I say it is partly a notation issue. However, the point of calculating the forward-looking expectation for the return is to allow for assessment at time $t$. This is to either predict likely outcome for future rewards knowing the system is in state $s_t$, or in control scenarios to choose an action $a_t$.
In addition, in both prediction and control scenarios, the past (from time steps $0$ to $t-1$) has already happened. Measurements of how well a system did in terms of gathering reward, looking backwards, might be useful metrics for e.g. "how good is this agent?". However, they are not in general a guide to the future. In many sparse environments (e.g. a board game scoring +1 for a win), this data is basically useless for predicting the future and all you want to know is summarised by the current state and the acting policy.
Regardless of when you get to finally calculate a return, the start time step - the part of the trajectory that the return is calculated from, is a key parameter. The end time step is a practical concern for implementation, but can often be considered to be at infinity for theoretical purposes (i.e. we are interested in measuring or optimising all future rewards). So if you are only going to display/use one parameter in the notation, the start time $t$ is the one to use.
There are variants of notation, to show how the return is calculated, where the calculation horizon is made explicit, e.g. $G_{t:t+n}$ for a truncated return or $G_{t:t+1}$ when calculating a one step temporal difference target. All the ones I have seen still maintain the forward view that defines a value associated with a current timestep $t$, for the same reason as explained above - it is at timestep $t$ where this value is of most interest as a prediction.
In practice during training you often wait until $t+n$ before you know the correct value of $g_t$* to apply as a training value - which is then used to update value estimates for $\hat{v}(s_t)$ or $\hat{q}(s_t, a_t)$. It is possible to make partial updates before that ending time step using techniques such as eligibility traces.

* Using notation of uppercase $G$ for random variable, and lowercase $g$ for a measured value.


text:
What is the KWIK framework?

...for learning transition dynamics...in the KWIK framework.

The above is part of a paper's conclusion - and I don't really seem to understand what the KWIK framework is. In the details of the paper, is a brief highlight of the KWIK conditions for a learning algorithm, which go as follows (I paraphrase):

All predictions must be accurate (assuming a valid hypotheses class)
However the learning algorithm may also return $\perp$, which indicates that it cannot yet predict the output for this input.

A quick Google search brought me to this paper from ICML 2008, but it is a little difficult to comprehend without a detailed read.
Could someone please help me understand what the KWIK framework is, and what implication does it have for a learning algorithm to satisfy KWIK conditions? An explanation that starts at simple and goes to fairly advanced discussions is appreciated.

What about this presentation about the very same paper from ICML 2008? It has less pages and is more simply put than the paper.

Based on the presentation by Praveen Venkateswaran on the same paper, KWIK framework is any learning algorithm that makes the following:

if it knows (by its learning) the answer already, it does tell that
if it is uncertain, it tells the "I don't know" mark
"I don't know" marks should have an upper limit, which is the accuracy/error term of the algorithm.

The idea is that algorithm has a trainer who knows the correct answers, like in any machine learning case, but the difference is that the algorithm agent should know when it has enough knowledge to give the result right away, and when is time to ask and learn. In learning phase the trainer does not calculate errors, only he gives right answers if asked.
There I see two implications: algorithm should know when it has not all the necessary data to answer (when to use "I don't know" answer) and a mechanism to be sure you're correct when the answer is e.g. 1 or 0 in binary case.
How to put that in mathematics, it varies by your chosen algorithm. Also there is a formal equation set for this, but I leave here only this literal definition to give the basic idea.

So, according to that presentation, KWIK stands for "Know What It Knows".


text:
What is meant by a multi-dimensional continuous action space?
In the context of Reinforcement Learning, what does it mean to have a multi-dimensional continuous action space?
I came across the following in the COBRA Paper

A method for learning a distribution over a multi-dimensional continuous action space. This learned distribution can be sampled efficiently.

and

During the initial exploration phase it explores its environment, in which it can move objects freely with a continuous action space but is not rewarded for its actions.

So, what do the multi-dimensionality and the continuity of the action space refer to? It'd be great if someone could provide an explanation with examples!

It means $a \in R^n$, where $a$ is action

Ohh, okay! I don't see why researchers resort to complicated terms in papers.

I wouldn't know how to write it simpler. It is just the translation of $a \in \mathbb{R}^n$ into text. Continuous values means (most likely) $\mathbb{R}$ and multidimensional $^n$. If you understand the mathematical expression you are most likely familiar with math and if that is the case then you most likely also know about these two terms. Maybe action space is a term that is quite unfamiliar for people who are learning about reinforcement learning? That just means the set of all possible actions.

Let me rephrase it a little - it's a multidimensional continuous space of actions. So, you assign each action some vector from $R^{n}$. For intuition -- imagine you have a robot arm with four joints. For every joint you could applied a rotation force from [-1, 1] and thus you get a 4-D vector with float numbers for each possible action.

The question has already been answered by Kirill, but I thought I'll add a good example of a multi-dimensional continuous action space too, namely the one I just encountered in the COBRA paper itself.

In all of our experiments we use a 2-dimensional virtual "touch-screen" environment that contains objects with configurable shape, position, and color. The agent can move any visible object by clicking on the object and clicking in a direction for the object to move. Hence the action space is continuous and 4-dimensional, namely a pair of clicks.


text:
Does the "lowest layer" refer to the first or last layer of the neural network?
People sometimes use 1st layer, 2nd layer to refer to a specific layer in a neural net. Is the layer immediately follows the input layer called 1st layer?
How about the lowest layer and highest layer?

"Deep learning" may suggest vertical terminology, but everyone seems to draw left-to-right (horizontal) networks. Perhaps we should rename it to wide learning ;)

People sometimes use 1st layer, 2nd layer to refer to a specific layer in a neural net. Is the layer immediately follows the input layer called 1st layer?

The 1st layer should typically refer to the layer that comes after the input layer. Similarly, the 2nd layer should refer to the layer that comes after the 1st layer, and so on.
However, note that this convention and terminology may not be applicable in all cases. You should always take into account your context!

How about lowest layer and highest layer?

To be honest, I also dislike this ambiguous terminology. From my experience, I don't think there's an agreement on the actual meaning of "lowest" or "highest". It depends on how you depict the neural network, but it's possible that "lowest" refers to the layers closer to the inputs, because, if you think of a neural network as a hierarchy that starts from the inputs and builds more complex representations of it, the "lowest" may refer to the "lowest in the hierarchy" (but who knows!).

Lowest layer generally refers to the layer closest to the input. This comes from the idea that layers closer to the input represent low-level features such as gradients and edges, while layers closer to the output represent high-level features such as parts and objects.

You will also see the term "TOP" layer. This usually refers to the output layer of the network.

What quantifies a feature as "low" or "high" level? I never understood the origin of this terminology.


text:
Why do we set offset (0.5) in single shot detector?
In the paper SSD: Single Shot MultiBox Detector, under section 2.2 - (4), why do we add an offset of 0.5 to x, y in generating the anchor boxes across feature maps?

The integer coordinates refer to the top-left corner of a pixel. Adding 0.5 to both x and y gives you the center location of a pixel instead.


text:
What is the difference between the state transition of an MDP and an action-value?
Let's say we have MDP where we have a state transition matrix.
How is this state transition different from action value in reinforcement learning? Is the state transition in MDP stochastic transition, meaning transition to some other state without taking any action?

Transition Probabilities: Consider that you are at state $s$ and from that state take an action $a$.Then there are some probability you will land up at state $s_{1}'$ or $s_{2}'$ ($s'$ indicate the next states). Those probabilities are called transition probabilities. In this example, the transition matrix is just a 3D array since it depends on your state and action($p(s, a)$).
Action value function $Q_{\pi}(s, a)$: It is the expected total reward you get from state $s$, taking action $a$ and thereafter following the policy, $\pi$.

Is the state transition in MDP stochastic transition, meaning transition to some other state without taking any action?

The environment can be stochastic or deterministic. If the environment is stochastic then those transitions are stochastic. If the environment is deterministic then those transitions are deterministic.

Yes I am aware of transition probabilities, but what I am asking is how does that transition occur? Is the transition occur by taking "action" or is a stochastic phenomena? In other words how is the transition matrix populated? In other words the transition probability matrix arises if we record action over large period of time or its just that if you in state 1 there is simply stocastic chance that you will end up in state say 7. How do you end up in state 7 from state 1? by taking an action?

The transition occurs by taking action. For example, if you take action 'left' then there is some chance you land up in the left state and there is also some probability that you will land up in other states than the left state. This thing will happen if the environment is stochastic.  In reinforcement learning, if you use dynamic programming, then you have to know the dynamics of the system(means the transition matrix and rewards). Or another thing you can do is go to the environment collect a lot of samples and calculate transition matrix from your collected samples..

ok so in MDF we build the transition matrix by taking actions and recording their frequency?

yes, we can find the transition matrix by using those samples(this is called model-based reinforcement learning).

ok makes sense.


text:
What happens to the channels after the convolution layer?
I wonder what happens to the 'channels' dimension (usually 3 for RGB images) after the first convolution layer in CNNs?
In books and other sources, it is always said that the depth of the output from convolutional layers is the number of kernels (filters) in that layer.
But, if the input image has 3 channels and we convolve each of them with $K$ kernels, shouldn't the depth of the output be $K * 3$? Are they somehow 'averaged' or in other way combined with each other?

This question has been asked multiple times in the past. Your filters will typically have a depth equal to the depth of the input. Does this https://ai.stackexchange.com/q/17783/2444 or https://ai.stackexchange.com/a/18669/2444 answer your question? Maybe have a look at https://ai.stackexchange.com/q/3287/2444 or https://ai.stackexchange.com/q/5769/2444 too. Tell me which one answers your question, then I will mark this as a duplicate of that one.

I would say that none of these question is excactly what I meant BUT answer to my question is somewhere among answers to this one: https://ai.stackexchange.com/q/5769/22659. The direct answer to my question is that values obtained from convolutions among different channels sum up together, therefore 3 channels after convolution with one filter give one output. The best explanation delivered by Andrew: https://www.coursera.org/lecture/convolutional-neural-networks/convolutions-over-volume-ctQZz

I think you shouldn't think of kernels (or filters) as 2d matrices, but as (3d) tensors. Anyway, feel free to provide an answer to your own question (if this isn't really a duplicate)! Also, don't forget next time to tag me with @nbro, otherwise, I don't see your comments.

Answer to my question is that values obtained from convolutions among different channels sum up together, therefore 3 channels after convolution with one filter give one output.
Best explanation delivered by Andrew: https://www.coursera.org/lecture/convolutional-neural-networks/convolutions-over-volume-ctQZz

I think this answer is a little bit misleading. What happens is that the kernel actually has the same depth as the depth of the input. So, if the input has 3 channels, then a $3 \times 3$ kernel is actually a $3 \times 3 \times 3$ kernel. From your explanation, it seems like you're suggesting that you perform different convolutions (with the same or different kernel?) for each channel, then sum up the results. That's somehow equivalent, but, as I said, in my opinion, conceptually more complicated than thinking that the kernel has the same depth as the input.


text:
Why is probability that at least one hypothesis out of $k$ being consistent with $m$ training examples $k(1- \epsilon)^m$?
My question is actually related to the addition of probabilities. I am reading on computational learning theory from Tom Mitchell's machine learning book.
In chapter 7, when proving the upper bound of probabilities for $\epsilon$ exhausted version space (theorem 7.1), it says that the probability that at least one hypothesis out of the $k$ hypotheses in the hypotheses space $|H|$ being consistent with m training examples is at most $k(1- \epsilon)^m$.
I understand that the probability of a hypothesis, $h$, consistent with m training examples is $(1-\epsilon)^m$. However, why is it possible to add the probabilities for $k$ hypotheses? And might the probability be greater than 1 in this case?

Let $A$ and $B$ be two events. In general, the probability that either $A$ or $B$ occurs is defined as
$$
P(A \text{ or } B) = P(A) + P(B) - P(A \text{ and } B)
$$
If $A$ and $B$ are disjoint, i.e. they cannot happen at the same time, then $P(A \text{ and } B) = 0$, so the above formula becomes
$$
P(A \text{ or } B) = P(A) + P(B)
$$
If the probability of one arbitrary hypothesis being consistent with $m$ training examples is $(1-\epsilon)^m$, then, given the rule above and assuming that only one hypothesis is consistent with $m$ training examples, the probability of one or more (i.e. at least one) of the hypotheses being consistent with training examples is the sum of the probabilities, i.e. $k (1-\epsilon)^m$.
This probability can be bigger than one if more than one hypothesis is consistent with $m$ training examples. In that case, you have to take into account the probability that both hypotheses are consistent.
See e.g. notes General Probability, I: Rules of probability for more details about the union rule and other rules of probability.

is $k(1 - \epsilon)^m$ a lower bound on the probability that exactly one hypothesis out of k is consistent ?

@calveeen It cannot be a lower bound. The probability that one hypothesis out of $k$ is consistent (according to you and your book) is $(1 - \epsilon)^m$, which is smaller than $k(1 - \epsilon)^m$, because $k \geq 1$.


text:
How does the crossover operator work when my output contains only 2 states?
I'm currently working on a project where I am using a basic cellular automata and a genetic algorithm to create dungeon-like maps. Currently, I'm having an incredibly hard time understanding how exactly crossover works when my output can only be two states: DEAD or ALIVE (1 or 0).
I understand crossover conceptually - you find two fit members of the population and they exchange genetic material, hopefully producing a fitter offspring. I also understand this is usually done by performing k-point crossover on bit strings (but can also be done with real numbers).
However, even if I encode my DEAD/ALIVE cells into bits and cross them over, what do I end up with? The cell can only be DEAD or ALIVE. Will the crossover give me some random value that is outside this range?
And even if I were to work on floating-point numbers, wouldn't I just end up with a 1 or 0 anyway? In that case, it seems like it would be better to just randomly mutate DEAD cells into ALIVE cells, or vice versa.
I've read several papers on the topic, but none seem to explain this particular issue (in a language I can understand, anyway). Intuitively, I thought maybe I can perform crossover on a neighbourhood of cells - so I find 2 fit neighbourhoods, and then they exchange members (for example, neighbourhood A gives 4 of its neighbours to neighbourhood B). However, I have not seen this idea anywhere, which leads me to believe it must be fundamentally wrong.
Any help would be greatly appreciated, I'm really stuck on this one.

You mention both "fitness of cells" and "fitness of individuals" - you should be using the latter to select individuals for crossover, not the former. A cell can only be dead or alive, but the whole individual has more than 2 states - its fitness could be defined as the sum of "alive" cells. If your fitness function of individuals truly only has 2 states, then you have a problem that's not well suited to genetic algorithms.

What is the difference between "fitness of cells" and "fitness of individuals" in my context? I am using a 100x100 grid, which is full of cells that are either dead or alive. What is a 'whole individual'? A neighbourhood around a single cell? 3x3 for example?

Your goal is to find a 100x100 grid that meets some criterion. How close the whole grid is to meeting that criterion is your fitness function of individuals. If your goal was to turn the whole grid to 1's, your fitness function would be simply the count of how many 1's you have. If your goal was to have no 1's adjacent, your fitness function might be some large number minus the count of adjacent 1's. Once you have scored each 100x100 grid (each individual), you choose the best ones to "mate" and crossover.

Once you pick what individuals to mate based on their fitness, you do crossover by selecting parts of each - maybe you take the left half of one parent and the right half of the other, or maybe one quadrant from each of 4 parents. The crossover just copies different parts of the parent individuals, it doesn't combine them in any way by "averaging" the values. Crossover should leave you with identifiable parts from each parent, but nothing that wasn't in either parent.

This is considered to be 'Generational' evolution, right? I was originally planning to do a 'Steady' evolution where I only generate one map and mutate the cells within it many times. Would you say it's better to generate many maps? If I were aiming to create, for example, a 100x100 grid that has several 'room like' sections.

I'm not terribly familiar with the "Steady" methodology. One of the things I like about generational GAs is that you can explore many parts of the solution space at once in the beginning, and hone in the promising ones, which is achieved by generating many maps and iterating over them. It's not clear to me how crossover would even work in the Steady method, since you only have a single map - you can mutate the cells, and you can copy information from one part of the map to another, but your next iteration will have only the information that came from the previous one, plus random differences.

I like the idea of a generational GA, but I imagine storage would be an issue. With a 100x100 grid, I'd be storing arrays/lists of 10,000 cells - wouldn't that be very awkward to deal with/slow?

Depends on what you're running on, but that doesn't seem like an outrageous size - even a population of 100 individuals would give you some variation. I often work with data that has 1000 samples and 10000 features, which would be the same as a 100x100 grid with 1000 individuals. If storage is a big issue, it looks like you can encode your grid variables as boolean, which will give you significant space savings in most languages.

OK, sounds good. This has been very useful - thank you for the responses.

How do you calculate fitness for your organism if there are only two states? I would think it would be a solution to work on combinations of cells:
If you have blocks of, say, 4x4 cells, then each organism would be encoded using 16 bits. A crossover then is like overlaying two 4x4 blocks, picking the new block from either the first or the second parent. Effectively you are doing this:
0011                   1111                 xx11
0001  combined with    0000  would become   000x
1001                   0000                 x00x
1111                   0000                 xxxx

(this would be a bottom-right corner combined with a top wall segment)
or, as a single bit-string:
0011000110011111 +
1111000000000000 =
xx11000xx00xxxxx

where 'x' can be either 0 or 1, as this is where the two organisms are different.

This is why I was considering the idea of rating neighbourhoods as a whole for fitness, though I can't find any examples of people doing this so it makes me very unsure. So, let's say I have a 3x3 grid and I rate cells with alive cells in the corner highly. Would that be reasonable?

Yes, that would be an option. It really depends on what you want to achieve!


text:
What kinds of techniques do autopilots of autonomous cars use?
What kinds of techniques do autopilots of autonomous cars (e.g. the ones of Tesla) use? Do they use reinforcement learning? Which types of neural network architecture do they use?

I will focus on Tesla's autopilots in this answer (because that's the only specific autopilot you mention).
In their website, they mention the basic technologies underlying the current autopilots, which includes deep neural networks.

To make use of a camera suite this powerful, the new hardware introduces an entirely new and powerful set of vision processing tools developed by Tesla. Built on a deep neural network, Tesla Vision deconstructs the car's environment at greater levels of reliability than those achievable with classical vision processing techniques.

So, they are probably using convolutional neural networks for the computer vision tasks.
There's also a Wikipedia article completely dedicated to Tesla's autopilot.

So i read that wikipedia page.they don't mansion the particular software, just for interest do you have any idea how car moves?i mean i think there is an RL agent who manages gas and stop pedals and drives car.like some people said they don't use RL, so what else they should be using?

They are probably using a combination of different ideas and tools. Maybe there's some technical paper that talks about it. RL isn't probably ready for real-world applications like autonomous vehicles.

They definitely don't use RL.
My guess is a mix of NN for segmentation, lidar, radar and how for perception.
Once object detection has been performed they use a trajectory generator and MPC.
There is an autonomous car class on Coursera that goes over this.

Your guess? This is not a question that should lead to speculative answers. It's a question looking for facts. So, an answer to this question requires citations. I will remove my downvote once you provide those citations.

why not? so then what is controlling the rudder? then what decides when to accelerate and when to stop?

i mean recognition is obvious but source of it's engine is not.can you provide more additional detailes?


text:
Should I compute the gradients with respect to the flatten layer in a convolutional neural network?
I'm trying to create a convolutional neural network without frameworks (such as PyTorch, TensorFlow, Keras, and so on) with Python.
Here's a description of CNN taken from the Wikipedia article

In deep learning, a convolutional neural network (CNN, or ConvNet) is a class of deep neural networks, most commonly applied to analyzing visual imagery. They are also known as shift invariant or space invariant artificial neural networks (SIANN), based on their shared-weights architecture and translation invariance characteristics. They have applications in image and video recognition, recommender systems, image classification, medical image analysis, natural language processing, and financial time series.

A CNN has different types of layers, such as convolution, pooling (max or average), flatten and dense (or fully-connected) layers.
I have a few questions.

Should we compute gradients (such as $\frac{\partial L}{\partial A_i}$,$\frac{\partial L}{\partial Z_i}$,$\frac{\partial L}{\partial A_{i-1}}$ and so on) in flatten layer or not?
If no, then how should I compute $\frac{\partial L}{\partial A_i}$ and $\frac{\partial L}{\partial Z_i}$ of first layer of convolutional layer? With $\frac{\partial L}{[\frac{\partial g(A_i)}{\partial x}]}$ or with
$\frac{\partial L}{\partial dA_{i+2}}$(P.S. as you know iteration of BackPropagation is reverse, so I used i+n for denote the previous layer)?
Or can I compute derivatives in Flatten layer with
$$\frac{\partial J}{\partial A} = W_{i+1}^T Z_{i+1}$$(i+1 denotes prev.layer in BackProp)
$$\frac{\partial L}{\partial Z} = \frac{\partial L}{\partial A} *\frac{\partial g(A_i)}{\partial x} $$
and then reshape of Conv2D shape?

P.S. I found questions like mine (names are same), but there're not answer to my question as I asking about formula.

What are $A$ and $Z$ here?

I have found that you should compute derivatives $\frac{\partial L}{\partial A}, \frac{\partial L}{\partial Z}$ in Flatten layer and then reshape Conv2D input shape.

Would you share an example of neural network model having Flatten --> Convolution?


text:
Is it theoretically possible (or impossible) that principal component analysis worsens the performance of the model?
In case I had a prediction model and decided to add a PCA step prior to the model, is it theoretically possible/impossible that the number of output dimensions that is better for all tests may perform worse than the model without PCA?
My question comes from the fact that I want to add a PCA step prior to a model and hyperparameterize the PCA output dimension from 1 to N (N being the number of dimensions in the original dataset) and I wanted to know if there is any theoretical basis that there is no case in which performing this previous step could have a worse performance than the previous model.
Especially, my doubt comes if the best PCA case from a selection of dimensions from 1-N is always better than the best case without PCA.

What do you mean by "number of output dimensions that is better". How is an "output dimension" better than another? Please, edit your post to clarify this!

PCA works well where data sample space is linear. If data sample space is not linear or it is manifold data then model without PCA may perform better than model using PCA.

In the given image you can see, data is manifold. In this type of data, PCA, which is based on projection technique does not work well. That's why we use manifold learning technique to handle these cases.

PCA can make models worse,  imagine data points scattered along two elongated parallel rectangles. The axis with the greatest variation will be parallel to the rectangles but doesn't provide any benefit in classifying the points.


text:
If vanishing gradients are NOT the problem that ResNets solve, then what is the explanation behind ResNet success?
I often see blog posts or questions on here starting with the premise that ResNets solve the vanishing gradient problem.
The original 2015 paper contains the following passage in section 4.1:

We argue that this optimization difficulty is unlikely to
be caused by vanishing gradients. These plain networks are
trained with BN, which ensures forward propagated
signals to have non-zero variances. We also verify that the
backward propagated gradients exhibit healthy norms with
BN. So neither forward nor backward signals vanish. In
fact, the 34-layer plain net is still able to achieve competitive accuracy, suggesting that the solver works
to some extent.

So what's happened since then? I feel like either it became a misconception that ResNets solve the vanishing gradient problem (because it does indeed feel like a sensible explanation that one would readily accept and continue to propagate), or some paper has since proven that this is indeed the case.
I'm starting with the initial knowledge that it's "easier" to learn the residual mapping for a convolutional block than it is to learn the whole mapping. So my question is on the level of: why is it "easier"? And why does the "plain network" do such a good job but then struggle to close the gap to the performance of ResNet. Supposedly if the plain network has already learned reasonably good mappings, then all it has left to learn to close the gap is "residual". But it just isn't able to.

They explained in the paper why they introduce residual blocks. They argue that it's easier to learn residual functions $F(x) = H(x) - x$ and then add them to the original representation $x$ to get hidden representation $H(x) = F(x) + x$ than it is to learn hidden representation $H(x)$ directly from original representation. That's the main reason and empirical results show that they might be right. Better gradient propagation might be an additional bonus but that's not why they originally introduced the idea.
"Normal" networks work too but, at some point, they become too deep and start working worse than shallower versions (they empirically showed that in the paper). Again, they argue that the reason for that might be that at deeper layers hidden representations become approximately similar $H_n \approx H_{n+1}$ because representation is already well learned and you only need some slight adjustments.  That would mean that transformation for deeper layers is similar to identity transformation and that ordinary layers might have trouble learning that, while for residual blocks it would be easy to learn slight modification and add that to the already existing representation from the previous layer.

Thanks! I'm almost there. It's just that I wonder why the non-residual version gets close to good mapping but doesn't quite get it. I feel like maybe it's because it settles on a minimum in hp space which is not the correct one, and actually quite far from the correct one. Whereas maybe when mapping a residual, you get dropped in at least the same town as the global minimum, so any local minimum you find is just a few blocks away.
And this explanation doesn't rely on vanishing gradients at all.

Well I don't know that's a lot of speculation. You shouldn't take all of this to literally its just an interpretation. The way I looked at it, if you look at the identity operator, that is an identity matrix with ones on diagonal and 0 elsewhere. Now if that matrix represents weights, then diagonal weight should be really large (around 1) and others should be very small (around 0). Those are not very balanced weight relations so NN might have trouble learning that. With residuals you wouldn't have that problem because you're adding to original representation so all weights would be small.

I like the idea of identity propagator! In quantum mechanics the time evolution operator for very short time can be expressed as the sum of an identity operator and the Hamiltonian. So in the very short time intervals after the time evolution starts, the system's evolution can be expressed by an identity operator and Hamiltonian operator.
In neural networks, I can imagine that as we go from layer to another layer, the representations or weights of layers change gradually which hints the next representation can be obtained by an identity operator and some nonlinear operator.


text:
What is the relationship between the Q and V functions?
Suppose we have a policy $\pi$ and we use SARSA to evaluate $Q^\pi(s, a)$, where $a$ is the policy $\pi$.
Can we say that $Q^\pi(s, a) = V^\pi(s)$?
The reason why I think this can be the case is because $Q^\pi(s, a)$ is defined as the value obtained from taking action $a$ and then following policy $\pi$ thereafter. However, the action $a$ taken is the policy according to $\pi$ for all $s \in S $. This seems to corresponds to the value function equation of $V^\pi(s_t) = r(s_t) + \gamma V^\pi(s_{t+1})$.

Can we say that $Q^\pi(s, a) = V^\pi(s)$

No.
The correct relationship is this:
$$V^\pi(s) = \sum_a \pi(a|s) Q^\pi(s, a)$$
or, if you have a deterministic policy $a = \pi(s)$ you can instead write:
$$V^\pi(s) = Q^\pi(s, \pi(s))$$
Intuitively, this is because the $V^\pi(s)$ is the expected future return when following the policy $\pi$ from state $s$, whilst $Q^\pi(s, a)$ is the expected future return where it ignores the policy for only the next action $a$ (which will decide immediate reward $r$ and next state $s'$ independently of the policy), and thereafter follows $\pi$.
The above equations essentially show what happens when you apply the policy in state $s$ to decide which Q value(s) to use, they remove the independent choice of $a$ in $Q^\pi(s,a)$.
One possible misunderstanding that you have is that "on-policy" means the same thing as the equations show when considering action values (Q values) - it does not. When any algorithm learns action values, it learns the same thing conceptually, i.e. the expected (and maybe discounted) sum of future rewards given state $s$ when making a free choice of $a$ for the next step, and thereafter strictly following the policy being evaluated.
What is different between on-policy and off-policy is which policy gets evaluated - for on-policy methods like SARSA you evaluate action values for the same policy that you use to generate actions. For off-policy methods like Q learning you evaluate a different target policy. Both approaches have the same interpretation of what $Q(s,a)$ means otherwise, and have the same relationship between Q and V for their repsective policies.


text:
How to perform insect classification given two images of the same insect?
I'm relatively new to image classification. Currently, I am trying to classify insect images, using a convolutional neural network (CNN).
When I ask a human expert to identify an insect, I usually provide 2 photos: back and face. It seems that sometimes one feature stands out and allows identification with high certainty ("spots on the back - definitely a ladybug"), while other times you need to cross-reference both angles ("grey back could mean a few things, but after cross-referencing with the eyes - it's a moth").
How is it customary to implement this? Naively I was considering:

Two separate networks, one for backs and one for faces? If so, what formula is best for weighing in their outputs?
Single network, but separate dual classifications - e.g. "moth face", "moth back", "ladybug face", "ladybug back"?
A single network, feed everything naively (e.g. moths from different angles, all with the same classification "moth") and rely on the NN to sort it out itself?

There are several ways you can do this.
One is to input both images in input, so it can be a 2 input system or an input with 6 channels.
As you suggested in 1st point, you can make 2 networks, connect them at the end and add another layer for final classification or use outputs from both and train another classifier (like Gradient bosting). You can look up ensemble techniques and ensemble of neural networks for more.
For dual classification and naive approach, you will again have a problem in case there are some insects that can solely be identified from features at a specific angle, if so you can then discard other angles (e.g. ignore moth face if moth back is the distinguishing feature)