text:
Can an optimal policy have a value function that has a smaller value for a state than a non-optimal policy?
I'm starting to learn about the Bellman Equation and a question came to my mind.
A policy $\pi$ is optimal if the value $v_\pi(s)$ is greater or equal than the value $v_{\pi'}(s)$ for all states $s \in S$.
Why does this work?
Can't it be that the optimal policy thinks a state isn't that good and gives him a low value but perform best in comparison with other policies which have higher values for this state?

Can't it be that the optimal policy thinks a state isn't that good and gives him a low value but perform best in comparison with other policies which have higher values for this state?

No, this is not possible, and this is part of the definition of an optimal policy.
You are asking if it is possible to construct a policy $\pi^?$ where for some state $s_z$, $v^?(s_z) \gt v^*(s_z)$, yet for some other state $s_y$, $v^?(s_y) \lt v^*(s_y)$
In general, comparing two arbitrary policies, this situation is possible. However, there is no way to construct a policy that it does better than the optimal policy from any specific state.
The only thing you can change between policies is the action choice. Looking at the Bellman equation:
$$v_{\pi}(s) = \sum_a \pi(a|s) \sum_{r,s'} p(r,s'|s,a)(r +v_{\pi}(s'))$$
you can see, substituting in $s_z$, that its value depends on the immediate decision made by the policy then a weighted sum of values of next states, depending on how likely they are to be found depending on the policy. An optimal policy maximises this value in all states.
If there were a policy that made a different decision to the optimal policy on state $s_z$ and that was the only difference (all other decisions the same, and all other $v_{\pi}(s)$ values the same in the sum, then it would clearly be as good or better than the optimal policy in all states, and that would mean that it was the optimal policy, and the policy that you originally labelled as optimal, $\pi^*$, was not. That contradicts your starting statement.
If, in order to get a higher value, you said that one of the $r + v^?(s')$ values were higher (in expectation), then you could set $s_z = s'$ and repeat the argument - at some point the alternative "non-optimal" policy has to make a decision that is better than the optimal policy in order to get a higher value out of values that are otherwise the same (or even worse). The optimal policy would not be restricted from choosing that action, and in fact must do so in order to be optimal, so you would have shown that the optimal policy $\pi^*$ is not in fact optimal, which is a contradiction to your starting statement.
If the other $r + v^?(s')$ values were lower in expectation - a necessary condition at some point in the allowed trajectories in order to make $\pi^?$ non-optimal, then you will have shown that $v^?(s_z) \lt v^*(s_z)$ which contradicts your original statement.
In any case where you construct a locally unexplained better $v^?(s_z)$ (i.e. where somehow the expected values at the next state are better) then you have kicked a can down the road, it depends on other values in the Bellman equation that you can follow through. Eventually you have to either accept that your declared optimal policy is not optimal, or that $\pi^?$ cannot be better than it, because it involves contradicting the statement that it is not optimal.
This kind of thinking, handled more formally, leads to the Policy Improvement Theorem, where you can unroll the dependency between values for states indefinitely and show that it is possible to reach the optimal policy by improving the decisions made by the policy in each state until it can no longer be improved. Your concern that somehow there could be  locally-better-than-optimal policies different to the optimal policy is the flip side of that.

Thank you for taking the time to answer my question in such detail, I really appreciate it. Now I understand!

I guess it is possible. If you take actions greedily with respect to the desired state, it is possible. The definition in the Sutton textbook, there is no better value, holds for all states. I mean the definition should hold for all states $v_\pi_*(s) \ge v_pi(s)$. At least I guess this may happen. You can simply imagine it.


text:
To assess the quality of the reconstructed images, which metric is more reliable: PSNR or LPIPS?
I am training a model for image reconstruction. I used several metrics to assess the quality of the reconstructed images. LPIPS is decreasing, which is good. PSNR goes up and down, but the L1 loss and SSIM loss are increasing.
So, which metric should I care more about?
My datasets are Paris Street View and CelebA.
I'm not sure if the VGG that extracts features for LPIPS is reliable here or not.

Hello. I've made an edit to your post in order to clarify what your question was. Make sure I didn't change the meaning (in particular, please, check the new title, which is supposedly your question).

After working 1 year on a project on super resolution, I've learnt the following things about image quality metrics in general:

There's no such thing as a perfect metric. Every metric suffers from specific downsides and has specific benefits. So you ultimately want to rely on more than one, and check that they all have a consistent (improving) trend.
Every metric is sensitive to different types of noise, and I would argue (no reference to point to unfortunately) even to different image contents. The first point is easily provable with a toy experiment, check images (and code to reproduce them) below. This is a fact you can leverage. For example the LPIPS metric seems to lead to much worse values when gaussian or salt and pepper noise are present compared to speckle noise. So bad lpips value might be a hint that your model is producing those kind of artifacts.
The PSNR is indeed really unstable. In the images below the image with 3 iterations of speckle noise (bottom right) has a higher psnr than the image with one iteration of gaussian noise (top left), despite looking worse.

My suggestion is to play with the images you're using, add noise, compute metrics and see what happens. It's the only way to gather knowledge about your data, and it makes much more easier to trouble shoot what might be going wrong in your training regime.

Code:
import matplotlib.pyplot as plt
import numpy as np
import skimage.util as sku
from skimage.data import astronaut

import torch
import piq

img = astronaut()
# Normalize image
img = (img - img.min()) / (img.max() - img.min())

modes = ["gaussian", "s&p", "speckle"]

for mode in modes:
    img_noise1 = sku.random_noise(img, mode=mode)
    img_noise2 = sku.random_noise(img_noise1, mode=mode)
    img_noise3 = sku.random_noise(img_noise2, mode=mode)

    tensor = torch.tensor(img).permute(2,0,1).unsqueeze(0)
    tensor_noise1 = torch.tensor(img_noise1).permute(2,0,1).unsqueeze(0)
    tensor_noise2 = torch.tensor(img_noise2).permute(2,0,1).unsqueeze(0)
    tensor_noise3 = torch.tensor(img_noise3).permute(2,0,1).unsqueeze(0)

    psnr1 = piq.psnr(tensor_noise1, tensor).item()
    psnr2 = piq.psnr(tensor_noise2, tensor).item()
    psnr3 = piq.psnr(tensor_noise3, tensor).item()

    ssim1 = piq.ssim(tensor_noise1, tensor).item()
    ssim2 = piq.ssim(tensor_noise2, tensor).item()
    ssim3 = piq.ssim(tensor_noise3, tensor).item()

    lpips = piq.LPIPS()
    lpips1 = lpips(tensor_noise1, tensor).item()
    lpips2 = lpips(tensor_noise2, tensor).item()
    lpips3 = lpips(tensor_noise3, tensor).item()

    fig, (ax1, ax2, ax3) = plt.subplots(1, 3)

    ax1.imshow(img_noise1)
    ax1.set_xlabel(f"PSNR: {psnr1:.2f} \n SSIM: {ssim1:.2f} \n LPIPS: {lpips1:.2f}")
    ax2.imshow(img_noise2)
    ax2.set_xlabel(f"PSNR: {psnr2:.2f} \n SSIM: {ssim2:.2f} \n LPIPS: {lpips2:.2f}")
    ax2.set_title(f"{mode}")
    ax3.imshow(img_noise3)
    ax3.set_xlabel(f"PSNR: {psnr3:.2f} \n SSIM: {ssim3:.2f} \n LPIPS: {lpips3:.2f}")
    plt.show()

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Does Seq2Seq decoder take a special vector or the weights of the last encoder cell as an output?
I'm reading Sequence to Sequence Learning with Neural Networks and there's a thing that I couldn't quite grasp.
Paper says the encoder outputs a vector to be fed to the decoder. More precisely

Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector

However, when I look at the diagram:

there's no such vector here. What I understand from this diagram is decoder RNN takes the weights of the last encoder cell as an input.
Which one is correct? can you explain?
Stanford notes put it as

The final hidden state of the cell will then become C

So, is there no vector?

w here does not represent weights, in case you were thinking it does.

That drawing it's a bit oversimplified. Check this blog for a better explanation and implementation details. I'll refer to the image they have to answer:

the yellow boxes represent embedding layers, required to convert words in numbers
the green boxes represent the unfolded encoder
the red box represent the context vector, i.e. the vector you're looking for. Note that is just the final vector you obtain by applying the encoder to a sequence of words. For this reason some people prefer to draw directly a line to the decoder part, without drawing the final vector explicitly.
the blue boxes represent the unfolded decoder.
the purple boxes represent the linear layer used to predict a final word for the decoder hidden state.

The blog is reversing the input order and the paper also says that they reversed the order. But at the diagram you mentioned it's not in reverse order. Is it just an error or intentional do you have any ideas? For example it supposed to be morgen guten I guess.

yes, it's suppose to be  morgen guten . I guess it's probably an error, or they left the correct order in the diagram just for readability.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Which is more popular/common way of representing a gradient in AI community: as a row or column vector?
Consider the following remark about writing gradients from the chapter named Vector Calculus from the test book titled Mathematics for Machine Learning by Marc Peter Deisenroth et al.

Remark (Gradient as a Row Vector). It is not uncommon in the
literature to define the gradient vector as a column vector, following
the convention that vectors are generally column vectors. The reason
why we define the gradient vector as a row vector is twofold: First,
we can consistently generalize the gradient to vector-valued functions
$f: \mathbb{R}^n \rightarrow \mathbb{R}^m$ (then the gradient becomes
a matrix). Second, we can immediately apply the multi-variate chain
rule without paying attention to the dimension of the gradient.

The above remark implicitly says that there is no standard in writing the gradients. So, I can write a gradient of a scalar-valued multivariate function either as a column vector or as a row vector.
But, I want to know which is more common in the AI community?

The issue doesn't come up terribly often. If you are only dealing with vectors, everything is either a row or column vector. It makes no difference which it is.
A more relevant issue is whether one uses the so-called "numerator layout" or "denominator layout". In the numerator layout, $\partial f / \partial x$ is $\mathbb{R}^{n\times m}$, where $f \in \mathbb{R}^n$, $x \in \mathbb{R}^m$. The denominator layout is the other way ($\partial f / \partial x$ is $\mathbb{R}^{m \times n}$).
If you think of the gradient as a row vector, then you are using numerator layout.
I think the numerator layout is overall more common. But there is no real convention, and if the paper content deals with Jacobian matrices it will definitely be mentioned somewhere in the text which layout they use.


text:
Why does the activation function for a hidden layer in a MLP have to be non-polynomial?
Across multiple pieces of literature describing MLPs or while describing the universal approximation theorem, the statement is very specific on the activation function being non-polynomial.
Is there a reason why it cannot be a higher-order polynomial? Is it just an attempt to use the least complex solution or we really cannot use higher-order polynomial?
I can understand the reason for the non-linear, but I am clueless about the non-polynomial requirement.

The paper Multilayer feedforward networks with a nonpolynomial activation function can approximate any function (by Leshno et al., 1993) provides a theorem claiming this and the (quite long) proof of the theorem in the appendix.
After a quick reading, it seems to me that they do not provide a very intuitive explanation of why the non-polynomiality of the (bounded, non-constant, and not necessarily continuous) activation function is necessary and sufficient (it's an if and only if result, see the last paragraph on p. 4) for a single-layer neural network to approximate any continuous function.
The theorem (p. 10) formally states that the set of all possible functions that the neural network can compute, denoted by $\Sigma_n$, is dense in $C(\mathbb{R}^n)$, the set of continuous functions from $\mathbb{R}^n$ to $\mathbb{R}$ (and being dense in $C(\mathbb{R}^n)$ is the equivalent mathematical statement to "neural networks can approximate any continuous function"). To understand this, you need to understand what a dense (sub)set is. To understand that, you need to understand what a closure of a set $S$ is: intuitively, it's the set of all points $S$ plus the points near the set $S$.
To prove this theorem (p. 12), they proceed in 7 steps, so it's a long proof.
For example, in step 1, they show (or just state) that, if the activation function $\sigma$ is a polynomial, then $\Sigma_n$ is not dense in $C(\mathbb{R}^n)$. They conclude that $\Sigma_n$, with such an activation function, would be a set of polynomials, which is not dense in $C(\mathbb{R}^n)$ (not sure why this is the case, but there is an old question here that asks exactly this; I think that understanding this would be a great step for intuitively understanding the theorem).
I don't plan to go over all the steps now, but, if you spend some time reading this paper, you should have the answer of why the non-polynomiality of the activation function is necessary and sufficient for the neural network to approximate any continuous function of the form $f : \mathbb{R}^n \rightarrow \mathbb{R}$. If that is not very useful, you could try reading the other related (but even longer) paper Approximation theory of the MLP model in neural networks (by Pinkus, 1999).
So, I wouldn't say that non-polynomiality is a complexity requirement, but really a theoretical requirement for approximating continuous functions.
(By the way, I think there's a typo on page 6. They write $f_\omega : \mathbb{R} \rightarrow \mathbb{R}^n$ to denote the function the neural network computes, but I am pretty sure they meant $f_\omega : \mathbb{R}^n \rightarrow \mathbb{R}$; in fact, previously, they assume that the neural network has $n$ inputs and $1$ output).

Like the accepted answer, I'm assuming you are referring to (literature refering to) Leshno et al, 1993. That paper only concerns 1-layer neural networks, and those are simply of the form
$$x\mapsto\sum_{j=1}^Na_j\sigma(w_j\cdot x+\theta_j)\qquad(N\in\mathbb N,a_j\in\mathbb R,w_j\in\mathbb R^n,\theta_j\in\mathbb R).$$
I think of them as sums of scaled, dilated and rotated versions of the activation function $\sigma$. If the activation function is a polynomial of degree $k$ then every 1-layer neural network will be a polynomial of degree $k$ (and no higher). It is easy to construct a function that cannot be approximated (uniformly on the unit cube) by polynomials of degree $k$. Generically, polynomials of degree $k+1$ cannot be approximated by polynomials of degree $k$.
For example, take $n=1$ (one input dimension). Then $f(x)= x^3$ cannot be approximated by 2nd degree polynomials: the latter are all parabolas. So $f$ cannot be approximated (uniformly on $[-1,1]$) by 1-layer neural networks with activation function $\sigma(x)=x^2$.


text:
In a Temporal Convolutional Network, how is the receptive field different from the input size?
I'm playing around with TCN's lately and I don't understand one thing. How is the receptive field different from the input size?
I think that the receptive field is the time window that TCN considers during the prediction, so I guess the input size shall be equal to it.
According to the WaveNet paper, I cannot see a reason why it should be otherwise. I'm using TensorFlow with this custom library.
Please help me understand.

As you can see in Fig. 2 of the WaveNet paper the receptive field is 5, but the input size is larger (16). The receptive field defines what a single output neuron can see (see arrows in Fig. 2).

The receptive field could also be greater than the input, e.g. if you want to use or you only have the last 12 time steps and use the following structure (WaveNet paper, Fig 3), which can cover different powers of two depending on the number of layers.

If you want to calculate not only the last output neuron, it can make sense that the input size is larger than the receptive field, as the outputs before use also older inputs (see dashed lines).

Thank You a lot for the answer. So if I look only at one neuron output it would make sense to match input size with receptive field? Also if I conclude correctly, I could put another output neuron on the top of TCN that would take several TCN's outputs as an input and in such way make my predictions more reliable?

Yes, if you only use the last output neuron, it makes sense to match the input size with the receptive field. 2) No, that shouldn't actually do much. The individual output neurons correspond to the individual time steps. Combining the outputs would not produce much more information. If you want to cover a larger time horizon, you should rather adjust the other parameters to increase the receptive field instead.


text:
Is it generally advisable to have a low dimensional action space in Reinforcement Learning?
In supervised or unsupervised learning, it is advised to reduce the dimensionality due to the curse of dimensionality in general.
Is this also generally advisable for the action space of reinforcement learning?
As far as I understand (and inspired by the answer here Is it possible to tell the Reinforcement Learning agent some rules directly without any constraints), you can reduce the dimensionality of the action space always to 1, meaning that you solely have 1 action. This can be done by using a mapping (e.g. in the step function of Open AI Gym).
Let's have a look at an example: We have a heating device that can heat 3 storages and we have a discrete action variable for all of them with 11 steps [0.0, 0.1, 0.2, ..., 1.0]. So we have

action_heatStorage1: [0.0, 0.1, 0.2, ..., 1.0]
action_heatStorage2: [0.0, 0.1, 0.2, ..., 1.0]
action_heatStorage3: [0.0, 0.1, 0.2, ..., 1.0]

In this case, we would have a 3-dimensional action space

action = [action_heatStorage1, action_heatStorage2,
action_heatStorage3]

However, it is also possible to combine the 3 actions  into 1 action variable "action_combined" of the size [11 * 11 * 11=1331] by just using a mapping of this one action into the separate 3 actions. For example like this:

action_combined = 0 --> action_heatStorage1 =0, action_heatStorage2
=0, action_heatStorage3 =0
action_combined = 1 --> action_heatStorage1 =0.1, action_heatStorage2
=0, action_heatStorage3 =0
action_combined = 2 --> action_heatStorage1 =0.2, action_heatStorage2
=0, action_heatStorage3 =0

...

action_combined = 1331 --> action_heatStorage1 =1.0, action_heatStorage2
=1.0, action_heatStorage3 =1.0

Is it generally advisable to reduce the dimensionality of the action space (Option 2), or to use multidimensional action variables directly (Option 1)?
I know that the is most probably not an answer that is valid for all problems. But, as I am relatively new to reinforcement learning, I would like to know whether in the theory of reinforcement learning there is a general recommendation to do something like this or not or whether this question can't be answered in general as it is something that totally depends on the application and should be tested for each application individually?
Reminder: I have already received a good answer. Still, I would like to remind you on this question to maybe hear also the opinion and experience of others regarding this topic.

Since the question may not be answered unambiguously in general, I will use the given example as a guide. As you correctly write, a large dimensionality leads to a very large solution space because of the curse of dimensions.
However, whether one restricts this solution space by allowing only discrete values $[0.0, 0.1, 0.2, ..., 1.0]$ for the three actions or by using the resulting combinations as a list of options (one dimension) makes no difference with respect to solution space coverage.
Depending on which inputs the model receives, however, it would be conceivable that the separate variables (option 1) could be more easily assigned and learned, e.g. if there are sensors for the state, temperature, etc. of each of the 3 heat storages as inputs. Option 2 would still require the model to implicitly learn one-dimensional coding to target specific heat storages.

Thanks dexteritas for your answer. So you would tend not to reduce the dimensionality but just use the 3 action variables separately? One big advantage of transforming them into a one dimensional action space is that you could map this one action to only valid options in a control problem and thus tell the agent some rules directly independant from the reward function (see https://ai.stackexchange.com/questions/32042/is-it-possible-to-tell-the-reinforcement-learning-agent-some-rules-directly-with). But I also understand your point. Most probably it will depend on the application.

Yes, in the given example, I would use the 3 actions variables separately. Of course, it depends on the problem, e.g. if you have only 2 or 3 states for each of the 3 actions a combination could be more useful. Depending on the complexity of the control problem, it could become difficult to identify and exclude all forbidden cases and be easier to use a constraint function. Especially, if the validity not only depends on the produces outputs, but also on the current state (so a output could be valid but is not valid for all cases). However, it depends ;)

Thanks for your comment dexteritas. What do you mean by "constraint function"? A function that is influecing the reward in a negative way if a constraint violation has occured or a super-ordinate controller that can overrule the inital actions of the RL agent?

You have a constrained optimization problem. There are different approaches for that. Giving a negative reward would be a penalty function. I think it is useful, if the agent can learn what is valid/allowed. You could use e.g. penalty function in combination with super-ordinate controller to ensure hard constraints are not violated. But that could be a separate question.

Thanks dexteritas for your comment. I upvoted your answer (but I will not yet accept it as I would like to see whether others also share their opinion; after some time I might also accept it). Thanks for your insights. I appreciate it.

As nobody other than you answered my question, I have now also accepted your answer (I had upvoted it before). I appreciate it.


text:
Can a computer make a proof by induction?
Can a computer solve the following problem, i.e. make a proof by induction? And why?

Prove by induction that $$\sum_{k=1}^nk^3=\left(\frac{n(n+1)}{2}\right)^2, \, \, \, \forall n\in\mathbb N .$$

I'm doing a Ph.D. in pure maths. I love coding when I wanna have some fun, but I've never got too far in this field. I say my background because maybe there's someone who wants to explain this in a more abstract language there's a chance that I will understand it.

Are you looking for any technique that is able to prove that statement, i.e. you are not necessarily looking for a machine learning approach, right?

It is possible for some classes of problems. For instance, WolframAlpha can generate an induction proof to the problem posed in the question.
According to the author of this proof generator, he built a library of pattern-matched proofs to generate the proofs. More details about his approach can be find in his write-up about the problem.
Other alternative (thought not induction-based) for automatically verifying these kind of identities (in special, hypergeometric identities) is by using algorithms such as Zeilberger's method along with the HYPER algorithm, both described in the excellent book A=B, currently available for free by one of its co-authors.

There are programming languages that allow you to verify a proof by induction. For example, I used  Coq, but I'm sure there are also others.


text:
What is the difference between terminal state, nonterminal states and normal states?
In Sutton & Barto's Reinforcement Learning: An Introduction, page 54, the authors define the terminal state as following:

Each episode ends in a special state called the terminal state

But the authors also say:

the episodes can all be considered to
end in the same terminal state, with different rewards for the different outcomes. Tasks
with episodes of this kind are called episodic tasks.

I believe there is also a fundamental difference between a terminal state, nonterminal states and plain, normal states:

In episodic tasks we sometimes need to distinguish the set of all nonterminal states, denoted S, from the set of all states plus the terminal state, denoted S+.

In the first quote, it appears as if the terminal state is just a term to describe the final state of an episode, but, from the second quote, I understand that the terminal state is the same no matter the outcome of the episode. If we consider the game of chess, what would we consider as a terminal state? Would it be the state that, if reached, will end the game (checkmate), no matter the result (win, loss)? But then how can we describe a state that would lead to draw? If we say about a state that leads to a draw that it's a nonterminal state since we can play an "infinite" number of turns without reaching a win or a loss hence without reaching the terminal state, aren't we implicitly supposing that reaching a draw isn't a result for which we should attribute a reward (e.g. 0)? And if we name a state that leads to a draw a terminal state, then what would be the difference between a normal state and a nonterminal state?

Terminal state is always the same in the sense that it represents the same thing, that the episode is over. They don’t need to be the exact same state; for instance you could have an $n$ by $n$ grid world where the top right and bottom left states are terminal as when you reach these your agent dies. These are both terminal but not the same state.
For chess it would be any state that when reached the game ends (regardless of win/draw/lose). The difference between these terminal states is what reward you will receive for reaching it.
Finally, normal states are non-terminal states, so there is no difference.

It may also be a good idea to describe what an absorbing state is, although this was not part of the question, but it's related.


text:
What is meant by "stable training" of a deep learning model?
I have read it said that the "stable training" of a deep learning model is important. What is meant by "stable training" of a deep learning model?

I can say "Stable Learning" of a supervised machine learner is as follows:

A stable learning algorithm is one for which the prediction does not change much when the training data is modified slightly.

You can follow this link to know more in details about how can we measure the stability in the context of computational learning theory.


text:
What is the difference between "ground truth" and "ground-truth labels"?
I'm aware that the ground-truth of the example at the top left-hand corner of the image below is "zero"

However, I am confused about the meaning of the terms ground truth and ground-truth labels. What is the difference between them?

These two terms could easily refer to the same thing, depending on the context. For example, a lazy person could easily say something like this

We compute the loss/error between the prediction (of the model) and the ground truth.

Here, the ground-truth refers to the "officially correct" label (categorical or numerical) for a given input with which you compute the prediction. So, in this case, ground-truth would be a synonym for a ground-truth label.
However, in general, ground-truth refers to anything, not just labels, that are correct or true (hence the name), so it could be used more generally. For instance, you could say something like this

We assume that the ground-truth underlying probability distribution from which the data is sampled is a Gaussian.

However, in this case, you could also leave out the ground-truth part, as it's more or less implied by the fact that you're assuming something.
So, the difference between the two is that "ground-truth" can be used more generally to refer to anything that is "true".

Note that this answer is based on my experience. I also don't think that these terms are really official, although they are often used in research papers, tutorials, etc.

Ground Truth
'Ground truth' is that data or information that you have that is 'true' or assumed to be true. That means that you have high or perfect knowledge of what it is. For example, in your image of numbers, you know that the first row are zeros, the second row are ones, the third are twos, and so on. You have 10 rows of data, each row is of  a different class or category. Each class has 16 samples. Ground truth data is used to train machine learning or deep learning models. The example you provided is from the Modified National Institute of Standards and Technology (MNIST) database which is commonly used for building image classifiers for handwritten digits.
Ground Truth Labels
The 'ground-truth labels' are the names you choose to give them. You may choose to label the classes as '0', '1', '2', etc., or as 'zero', 'one', 'two', etc. Maybe you think in Greek. If so label them as 'μηδέν', 'ένα', 'δύο', etc.
Reference
MNIST database


text:
How do we get the value of this state of an MDP, at time-step $h-2$, using dynamic programming?
I am trying to understand the problem below, represented as an MDP with four states (PU, PF, RU, and RF) and two actions (AS).

Let's consider V(RF), the value of the state RF. At time-step $h$, V(RF) = 10. When we go to the previous time-step $h-1$, V(RF) increases to 19.
Why is the value of RF increasing backward, i.e. at time-step $h$, which is the last step, it's 10, but in $h-1$ it's 19?
Also, when I apply the Bellman equation, I am not getting the value of V(RF) at time-step $h-2$, which is 25.08, according to the table.
Below is my solution which I am applying on V(RF):
Lets suppose for RF, I know that
Vh (RF) = max {R(RF,A), R(RF,S)}
        = max ({10,10}
Vh (RF) = 10

    **for h-1**
    Vh-1 (RF) = max R(RF,act) + gamma E (summation state) P(State|RF,act) Vh(State)
              = max {10+0.9(1*0), 10+0.9(0.5(10)+0.5(10))}
              = max (10,19)

    **for h-2**
    Vh-2(RF) = max R(RF,act) + gamma E(summation state) P(State|RF,act) Vh(State h-1)
             = max {19+0.9(1*0), 19+0.9(0.5(10)+0.5(10))}
             = 28.0

So, in the above scenario, the reward is 0.9, but I am not sure how we get the third result in V(RF) as 25.08. Where are we using this last part Vh(State) from the equation?

Wow, that's a really confusing example, if I were you I would check out some other RL resources. I wouldn't consider h being the last step and h-1 being the previous step. In terms of steps of iterations of the dynamic programming algorithm, h is actually the first step, h-1 the next step and so on. Viewing it in these terms it makes sense that the Value of RF increases from 10 to 19, because after the first step of dynamic programming the state RF incorporates some of the value from RU.
Here is the correct calculation for h-2.
$$10 + 0.9(0.5\times19+0.5\times14.5) = 25.08$$
You are doing a couple of things incorrectly in your calculation:

Firstly, you are mistakenly assigning the reward a value of 19. The reward should be 10. Note that the reward and Value are two different quantities. As we iterate through the dynamic programming algorithm, our current approximation of the values will change but reward will always remain the same (it is the number as indicated in the bubbles in the diagram). It just so happens that the Value and reward are equal on the first step (h).
You are using the Values for states RU and RF (=10) from step h to calculate the values step h-2 which is incorrect. You should be using the values from step h-1 which are 14.5 and 19 respectively.

Using this understanding, the calculation for the next step h-3 would be (notice that I am now using values from step h-2).
$$10 + 0.9*(0.5*25.08+0.5*16.53) = 28.72$$


text:
Fitting a Gaussian distribution into another distribution
Assume we have two vectors, containing random samples (maybe audio data?). Their distribution can be approximated to a normal distribution, so we can calculate their mean and standard deviation.

I am looking for a way to "fit" the second vector's samples, in a way that their mean and standard deviation correspond to the first vector's mean and standard deviation.

Also, I am looking for a way to do this by "moving the second vector's samples the least possible". This is because, an easy way to solve this problem could be to replace the second vector's data, with random samples that fit the first vector's parameters. This solution is easy, but not interesting.

Questions

Is this kind of problem correlated with machine learning in general? If yes "how"?

Is there a way to perform this kind of operation with some kind of neural network? If yes, how could it be modelled?

The earth mover's distance could be a relevant keyword here, I think that is what you're trying to do.

Thanks, seems to be the right path. Is this earth mover's distance correlated or used in machine Learning?

This is precisely the optimal transportation problem. If both vectors are defined on the same space, you are trying to minimize the Wasserstein distance (which I think is equivalent to what N. Kiefer suggested). Associated to the Wasserstein distance / optimal transport cost is an optimal transport plan, which tells you how to transport the mass from vector 2 onto vector 1.
Optimal transport has become fairly popular in ML, check the WGAN for example.


text:
Is it possible to overfit a model on infinite amounts of data?
This is a theoretical question. Is it possible to overfit a model on infinite amounts of data?
Let me clarify there are no duplicates.
Say, we have a generator function that produces data, with the correct classification/regression value, and we can generate infinite amounts of valid data. How long does it take for the model to overfit?
This question arose because I'm training an RNN model for fake news classification, and MSE loss is almost always 0.000, only 25% of the training data.
Will it be possible to overfit with one epoch of training on the infinite data generator?
(I'm thinking what will happen is the model will either get perfect, or sync into the generator's non-perfect randomness, and learn nothing)

If your data covers the entire sample space (easy to conceptualize with discrete-valued features), you don't need to abstract any model at all - you can just spit out the mapping from input to output, which much exist in the training data. I'm not sure the concept of overfitting is meaningful if it's impossible to have "unseen data". There's also no concept of "noise", as every unique combination of feature inputs maps to one and only one output - if "noise" can be perfectly predicted from the data, it's signal.

This is to generalize the concept of which the dataset is large enough that the features learned at the beginning of the epoch practically don't exist by the end. This is currently the case with the model I am training. If we have perfect randomness included in the model, then the noise becomes impossible to predict. There is some 'unseen' data, although it is very arbitrary in state-space

I feel that there's a disconnect here between "very large data" and "infinite data". With an infinite sample size, you'll cover the entire sample space almost surely. When selecting a new test sample from the population, the probability that it's not already in an infinite training sample of that population is zero (no unseen data). I don't follow your point about noise - since there are no duplicates in the data, every input has a fixed, deterministic mapping to an output (there is no randomness, since the output is perfectly predictable from the input features).

The trivial answer is that yes, it is possible. Consider standard Gaussian data and a generator sampling points $1/n$ with $n \rightarrow \infty$. Since you never see points beyond $(0,1]$, you are unlikely to learn the parameters of the distribution.
More generally, and for a supervised problem with data $X,Y$, if your data generator covers a subset of the domain with probability $1-\delta$ (with respect to the distribution of $X,Y$) for a very small $\delta$ and your model $\hat{f}$ fits with error $\epsilon$ here, then your generalization error is $\mathbb{E}_{X Y}[\text{loss}(\hat{f}(X),Y)] \leq \epsilon + \delta \sup_{X Y} |\text{loss}(\hat{f}(X),Y)|$. So if you knew that the loss were bounded and you could make $\delta$ arbitrarily small, you wouldn't overfit, because there wouldn't be any region of the sample space significant enough where you wouldn't be able to generalize to.
In other words, it all boils down to your generator being able to cover a set with high enough probability and your loss being "well behaved".
Note that because you will only have a finite coverage of the domain, e.g. an $\epsilon$-cover, you need some assumption on the growth of the loss for points which are $\epsilon$ apart, e.g. a Lipschitz condition (and also for the learned $\hat{f}$).


text:
Which AI techniques are there that combine multiple models to make sense of data at different stages?
I have been working to design a system that uses multiple machine learning models to make sense of data that is dynamically webscraped. Each AI would handle a specific task, for example:
An AI model would identify text in an image, then attempt to create plain text of what it might be. Once the text is extracted, it would be passed in a stored variable to an AI that can read the text to determine if it is a US city/state.
I tried to look into if others have done this, but didn't find much on it relating to what I was looking for. Does anyone know if there are potential issues with this? Logically, it looks good to me, but I figured I'd ask.
If anyone can put me in the right direction for reading material or further information, I would appreciate it.

You have explained a pipeline of AI algorithms for text in images: 1) Text detection, 2) OCR, 3) named entity recognition (NER). There are reams of paper on these topics.
Extracting City and Country Name from Text

Introducing flashgeotext: extract city and country names from text
locationtagger -A Python Package for detecting location
A Comparison Between Spacy NER & Stanford NER Using All US City Names
CoreNLP NER
Detecting Locations with NER
Papers with code for NER
Named Entity Recognition: A Practitioner’s Guide to NLP
Named Entity Recognition with NLTK and SpaCy

Papers on Text from Images

A High-Robust Automatic Reading Algorithm of Pointer Meters Based on Text Detection
Towards Spatio-Temporal Video Scene Text Detection via Temporal Clustering
An end-to-end text spotter with text relation networks
Recognizing Text with a CNN
A Framework to Detect Digital Text Using Android Based Smartphone
Detected text‐based image retrieval approach for textual images
A CNN-Based Approach to Detecting Text from Images of Whiteboards and Handwritten Notes

Websites on Text from Images

Deep Learning Based OCR for Text in the Wild
Automatically Detect and Recognize Text in Natural Images
Tesseract OCR: Text localization and detection
OpenCV Text Detection (EAST text detector)
Detect text in images
Learning to Read: Computer Vision Methods for Extracting Text from Images

I think the keyword for OP is "pipeline". It is common practice to combine detectors, classifiers and other processing to create a pipeline.


text:
How is it possible that Q-learning can learn a state-action value without taking into account the policy followed thereafter?
From my readings, I have been taught that the state-action value depends on the policy being followed. That seems logical because the expected return from actual actions will be different depending on which actions follow it.
On page 58 of Sutton & Barto's book, we have

So, how is it possible that Q-learning can learn a state-action value without taking into account the policy followed thereafter (i.e. the policy followed after having taken action $a$ in the state $s$)?

Q-learning can learn about the greedy policy (the policy that we define as $\pi(s) = \arg\max_a Q(s, a)$) whilst following some arbitrary exploratory policy because Q-learning is an off-policy algorithm.
In Q-learning, we are updating our values of $Q(s, a)$ using a bootstrapped value from one time step in the future. This means that we don't need to worry about any importance sampling type re-weighting of the action chosen by the exploratory policy that we use for action selection because of how $Q(\cdot, \cdot)$ is defined:
$$Q(s, a) = \mathbb{E}_\pi \left[ G_t | S_t = s, A_t = a \right] \; ;$$
where $G_t$ is the (discounted) future returns defined in Sutton and Barto. Now, because we have defined $Q(\cdot, \cdot)$ such that we condition on knowing action $A_t$, it really doesn't matter which distribution this action came from, because as mentioned we have conditioned on knowing it. This allows us to make updates in Q-learning for any state-action pair using the target from the optimal policy despite us not using this for action selection in the environment.
If we were to do some kind of $n$-step Q-learning with an update target looking something like
$$R(s_t, a_t) + R(s_{t+1}, a_{t+1}) + ... + R(s_{t+n}, a_{t+n}) + \max_{a'} Q(s_{t+n+1}, a')$$
then we would need to use importance sample to re-weight the trajectory to account for the actions $a_{t+i}$ for $i \geq 1$ as these are actions that the optimal policy may not have taken, as they were taken from the exploratory policy and the Q-function assumed that the trajectory (i.e. future actions) is generated under the policy associated with the Q-function.

I think OP was not quite asking about off-policy, although related. I think they are aiming to ask about the non-stationary nature of value functions in generalised policy iteration. Also, "This allows us to make updates in Q-learning for any state-action pair using the target from the optimal policy" is not quite correct. Q learning does not directly target the optimal policy action values, but the current greedy policy action values. The difference is minor but may be an important detail for questions like this one where OP is trying to reconcile prediction vs control issues.

I think the importance sampling part may not be necessary, if the user is not familiar with it. I think this answer does not emphasize certain aspects that should be emphasized, IMHO, but I think the off-policy part is relevant. In the definition given by Sutton & Barto, they define the $q$ function given a specific policy $\pi$. What is this policy in the context of Q-learning? I think that the direct answer to the OP's question is that Q-learning actually takes into account the policy that we assume to follow from the next state, i.e. the greedy policy (wrt the current value estimate).

Then we should probably explain how this relates to the off-policy nature of Q-learning. It's also important to emphasize the difference between a definition of a function (defined as an expectation) and an algorithm that attempts to estimate this function. As Neil pointed out, probably, the generalised policy iteration thing could also be useful for the explanation.

@NeilSlater agreed RE the terminology used surrounding the optimal policy.. we use a greedy target which will converge to optimal (in tabular setting) -- I'll edit this. I'm not really sure what to do about the rest. If OP isn't asking the question I've answered, then I'm not really sure what he is asking and he'd need to make the question clearer before I know if I can answer it, I will wait to see what OP says.

@DavidIreland You saw my point. I must take into account that the actions used for defining the optima state-action-value by Q-learning are known. As I now understand Q-learning assumes a greedy policy as stated also by nbro. Another important point seems to be that Q-learning directly approximate the optimal q_value. My confussion really is " aiming to ask about the non-stationary nature of value functions in generalised policy iteration." as asked by NeilSlater. Your answers and discussion helped me, even if I must continue to explore the point

The action-value function DOES take into account the policy being followed - that's precisely what the notation $\mathbb{E}_\pi$ is for. Specifically, $\mathbb{E}_\pi$ is a shorthand for
\begin{align*}
\mathbb{E}_{a_i \sim~ \pi(a_i \,|\, s_i), \, (r_{i+1}, s_{i+1}) \sim p(r_{i+1}, s_{i+1} \, |\, s_{i}, \, a_{i}), \, \forall i}
\end{align*}
where $p$ represents the environment's joint distribution of reward/transitions. This means that the expectation is with respect to the actions, states, and rewards you see under the policy $\pi$. If you want to make this more concrete, we can write out the expectation like this
\begin{align*}
& \mathbb{E}_\pi\left[\sum_{k=0}^{\infty}\gamma^k r_{k+t+1} | s_t, a_t \right] \\
:=& \int_{(r_{t+1}, s_{t+1})}p(r_{t+1}, s_{t+1})\bigg[r_{t+1} + \int_{a_{t+1}}\pi(a_{t+1})\int_{(r_{t+2}, s_{t+2})}p(r_{t+2}, s_{t+2})\bigg[r_{t+2} + \ldots
\end{align*}
(Really, you should be integrating over dummy variables, and I've omitted the conditional expectations, e.g. $p(r_{t+1}, s_{t+1})$ should be $p(r_{t+1}, s_{t+1} | s_t, a_t)$). Of course you can just replace the integrals with sums if you want discrete actions and/or states. So you can see just writing $\mathbb{E}_\pi$ sweeps a lot of the notation and true meaning under the rug, and I assume that is the source of the confusion.

The question was about Q-learning (not about the definition of the value function), which the OP assumed that it does not take into account the policy being followed.

My interpretation of the question is that OP thought that Eq. (3.13) indicates that the action-value depends only on the state, action $(s, a)$. My answer is that there is dependence on all the future states/actions which is meant by the $\mathbb{E}_\pi$ notation. It's possible the OP meant something different, but I'm not sure the OP's question is as deep as some of the other comments are implying.

By the way, OP is referring to me?

@HermesMorales Yes - it stands for Original Poster (I think). If you'd like to clarify what you are asking about, or discuss what people are asssuming you are asking, please post a comment/reply


text:
Is categorical encoding a type of word embedding?
Word embedding refers to the techniques in which a word is represented by a vector. There are also integer encoding and one-hot encoding, which I will collectively call categorical encoding.
I can see no fundamental difference between the categorical encoding and word embedding at a fundamental level. They may be different at an application level.
Is it true that categorical encoding is a type of word embedding? And are different names solely due to the task in which apply the technique?

One-hot encoding is different than the concept of a word embedding, although both approaches use vectors to represent the objects (e.g. words).
A one-hot vector contains one element that is 1 and all other elements are 0. So, for example, the vector $[0, 0, 1, 0]$ is a one-hot vector, while the vector $[0, 2, 0.2, 0]$ is not. (Given that the sum of all elements in the one-hot vector is equal to 1 and all elements are in the range $[0, 1]$, a one-hot vector is a probability vector, although this may be irrelevant.) To represent your objects as one-hot vectors, you first need to know how many objects you have (in the context of NLP, this is often the vocabulary size). Let's say you have $N$ distinct elements
(e.g. words), then, in order for two objects not to be mapped to the same one-hot vector, you need to have one-hot vectors of $N$ dimensions. So, if $N$ is very large, this can be a disadvantage of one-hot encoding, although, in principle, you only need to store the index of the $1$ for each one-hot vector.
As opposed to one-hot vectors, which do not encode the meaning of the objects they represent, word embeddings semantically represent the objects, i.e. similar objects (for some notion of similarity) are typically mapped to similar word embeddings. Here's a picture that illustrates this concept (taken from the book that I cite below).

Moreover, word embeddings can easily be smaller than one-hot vectors and do not necessarily contain only 0s and 1s. More importantly, word embeddings are derived from data (often learned).
A similar argument can be said for integer encoding, i.e. they are not learned from data and they do not carry any meaning. Chapter 6 of this book discusses word embeddings more in detail.
To conclude, the common definition of a word embedding usually implies that word embeddings semantically represent objects and are derived/learned from data, so, generally, one-hot encoding is not a form of word embedding, but it's a way to embed objects into a vector space.

Is there any reason that one-hot encoding is preferred over integer encoding for learning these word embeddings?


text:
Which loss function could I use to solve a regression problem as a classification problem (where we discretize the labels into buckets)?
I am considering a rather typical regression problem, but, for practice, I am trying to implement this as a classification problem.
The setup is as follows. I have $\mathbb{R}$-valued labels $y_i \in [-1,1]$, which I then discretize to $N$ buckets -- my classification problem is to then predict the labels to the nearest bucket.
This is rather straightforward and easy to implement with a cross-entropy loss function. However, I do not believe that this is the best option, as I would ideally like my predictions to be close to their correct bucket, even if I do not predict them correctly (which will be more difficult as if I take $N$ larger).
My current approach involves using a mean-squared error loss function. My network outputs logits for each bucket, I apply a softargmax (so the network remains differentiable) and then convert the output of the network into the $\mathbb{R}$-valued prediction.
My (very premature) results are nothing to write home about. So, I ask, is there a more natural loss function that I could consider for this exercise?

What code are you using ? share what you have tried

This is called ordinal regression, which could be a term to search, but binning tends to be discouraged.

Bucketing is usually used in industry because we are not sure what we are going to get. Some amount of observation on the distribution will give you a good idea of bucketing. But, there is caveat, if there isn't a lot of variance then bucketing won't help much.
Also, in terms of losses, since different buckets will have different number of examples, class imbalance will arise. So, focal loss is the better loss here.
I hope you get results you can write home about.

Hi, thanks for your input. I have thought of another approach, and would like to perhaps get your opinion. I am still running a classification based on my buckets, but I am using a KL divergence loss function. Additionally, for the labels I am "adding" noise to them -- essentially, centring a truncated normal (so that the pdf remains in my domain) around the observed label. In this sense, I am training a network to be a "soft" classifier. Does this make sense to you, and do you know if there is any existing approaches like this? The results are rather good for me.

The approach seems a bit convoluted. You are first assuming a discrete distribution over continuous values then you are using losses suitable for distributions (it is working on the label distribution). Also, the noise part is easily there in cross entropy function (see label smoothing). This can be a experiment and may even perform. But, traditionally, the pipeline would be (continuous -> discrete -> NN with cross entropy (with label smoothing for regularization). Do tell how that worked out for you.


text:
Why did Distributional Q Learning go out of popularity?
I read some papers (for example, this) and blogs that spoke about the advantages of distributional Q learning. However, it no longer seems to come up in literature. Did it have any shortcomings that led to its failure? If yes, can someone can talk about it here?

Actually, distributional RL is a well-studied field in deep RL. Generally speaking, distributional RL needs more computing sources (1.1X), because of the quantile head. We can also find new distributional RL literature in NeurIPS2020 :).

Hello. Welcome to AI SE. Maybe you can provide 2-3 examples of papers on Distributional RL that were published in NeurIPS 2020.


text:
Why is the validation loss less than the training loss, and what can be said about the effect of the learning rate?
I have the following results I am trying to make sense of. I have attached the loss curves here for reference.

As you can see, the first issue is that the validation loss is lower than the training loss. I think this is due to using a pre-trained model with a high dropout rate (please correct me if I am wrong here).

As one can see, the mean_auc score is increasing consistently, and so it seems that the network is indeed learning something and the validation loss is also better behaved relatively.

The training loss is what bugs me a lot. It is not at all consistent and varies a lot. This is a naive question, but is this graph giving me any sort of information about the learning rate, etc, or am I in a situation wherein everything is incorrect essentially?

Any response would be really appreciated.

This is very difficult to tell with the information provided, but the phenomenon  is something that I have encountered many times before. Sometimes this is not a bad thing, here are some possible considerations/explanations:

Data from the training set could be identical or leaking in to the validation set.
Using a high dropout rate can cause this as well as other generalization techniques.
The training set may contain outliers, contributing to higher loss and difficulty in learning.

Also, note that the there is not so much an issue with the training loss bouncing around as long as it decreases over time. You may want to try different optimizers as well to see if that changes this behavior.


text:
Is an embedding a representation of a word or its meaning?
What does the term "embedding" actually mean?
An embedding is a vector, but is that vector a representation of a word or its meaning? Literature loosely uses the word for both purposes. Which one is actually correct?
Or is there anything like: A word is its meaning itself?

An embedding is a representation of a word that can be used as a proxy for some of its linguistic properties.
The 'human' representation of a word, a sequence of letters and other symbols, is not related at all to its meaning or use in actual text. It only serves as a look-up key into our cognitive language processing facility (however that actually works) which enables us to understand the meaning in the context of its usage. However, a computer system does not have such a facility.
In order to convert the character string into something more usable for language processing, embeddings are created. These are typically vectors describing other words that surround a word in question (as the meaning of a word depends on its context). There are obvious problems with ambiguities (eg is bank the side of a river or a place where you deposit money?), but there are probably ways around that.
So the embedding (a vector) represents the usage of a word, which strongly correlates with its meaning. Because words as character sequences are useless for most sub-symbolic processing, the terms embedding and word are possibly used interchangably.

Although we have had multiple similar questions (see here, here and here) and it seems to me that you focused on word embeddings (probably because you were not aware of the application of embeddings to other contexts), in addition to what is stated in the other answer, it's important to note that the concept of an embedding does not just apply to words. For example, there are also code embeddings (see e.g. code2vec) and graph embeddings (see e.g. this), and there are probably other examples. The linked posts contain answers that explain what an embedding and embedding space generally are, so you may want to read them.


text:
Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram?
In music information retrieval, one usually converts an audio signal into some kind "sequence of frequency-vectors", such as STFT or Mel-spectrogram.
I'm wondering if it is a good idea to use the transformer architecture in a self-supervised manner -- such as auto-regressive models, or BERT in NLP -- to obtain a "smarter" representation of the music than the spectrogram itself. Such smart pretrained representation could be used for further downstream tasks.
From my quick google search, I found several papers which do something similar, but -- to my surprise -- all use some kind of symbolic/discrete music representation such as scores. (For instance here or here).
My question is this:

Is it realistic to train such an unsupervised model directly on the
Mel spectrogram?

The loss function would not be "log softmax of next word probability", but some kind of l2-distance between "predicted vector of spectra" and "observed vector of spectra", in the next time step.
Did someone try it?

The reason most music-generation models use discrete representations is because the long-term structures of music are very challenging to model. Note that the MIDI data in MAESTRO (used in the two papers you linked) encodes performances, not scores, so they include timing and accents of real performers--but are still sequences of discrete events, not audio.
There's been some work on learning discrete representations directly from audio, such as with vector-quantized variational autoencoders (VQ-VAEs). Typically an autoregressive model is trained on top of the learned representation; Jukebox used a transformer for that. By the way, I'd highly recommend reading through the "related work" section of the Jukebox paper for an overview of work on the audio/speech synthesis task.
wav2vec is probably closest to what you're describing. They train a transformer on raw audio, self-supervised, in order to learn good representations of human speech for the speech-to-text task.
As far as training directly on spectograms, there's MelNet, a somewhat exotic RNN trained for a variety of audio synthesis tasks, including music.
Hope this helps!

These papers are also very close to what I meant in the question (too long for a comment).
The following references come mostly from work on speech recognition.

Mockingjay In this work, they use an analogy of Bert architecture that is fed by Mel-spectrogram, with some audio segments "masked".

The model is asked to reconstruct the masked parts. To avoid the model using local smoothness of audio-data, they always mask several subsequent frames, so that reasonably long segments are being masked.
They evaluate on downstream "Phoneme classification tasks" using the learned features and show that these learned features are stronger than raw spectrogram, in particular if little training data is available.

Audio Albert Same story, but they use shared weights in the transformer layers. This significantly reduces memory and computational requirements and it is shown that results are comparable with Mockingjay (at least for phoneme-classification tasks).
Tera; another Bert variant where instead of masking, they use various "Alterations" of certain audio segments.
This and many more references are within this project with code.


text:
What exactly are partially observable environments?
I have trouble understanding the meaning of partially observable environments. Here's my doubt.
According to what I understand, the state of the environment is what precisely determines the next state and reward for any particular action taken. So, in a partially observable environment, you don't get to see the full environment state.
So, now, consider the game of chess. Here, we are the agent and our opponent is the environment. Even here we don't know what move the opponent is going to take. So, we don't know the next state and reward we are going to get. Also, what we can see can't precisely define what is going to happen next. Then why do we call chess a fully observable game?
I feel I am wrong about the definition of an environment state or the definition of fully observable, partially observable. Kindly correct me.

Could you reference where you have seen chess described as "fully observable"? I usually see the term "a game of perfect information" which is a different term. I think I have an answer for youy, but wantto check I am using the same terminology as you are referencing, because your question is about the use of that terminology in two slightly different fields of study.

Note that it's not the "state" that determines the next state, but the dynamics of the environment, which are encoded in the probability distribution $p(s', r \mid s, a)$ in an MDP.

@NeilSlater Figure 2.6, p. 45 of AIMA (3rd edition) describes chess as fully observable. In chess, you have access to the state (the configuration of the board), so I guess it's considered fully observable.

@NeilSlater I've seen it here

@nbro: I don't have access to that book. Is that chapter specifically about RL? Game theory, analysis of optimal play and search algorithms could use the term "fully observable" reasonably, but without the same thing applying to RL training to play versus an opponent whose decision-making is unknown.

First, note that the current state does not determine the next state. What determines the next state are the dynamics of the environment, which, in the context of reinforcement learning and, in particular, MDPs, are encoded in the probability distribution $p(s', r \mid s, a)$. So, if the agent is in a certain state $s$, it could end up in another state $s'$, but this is not only determined by being just in $s$, but also by $a$ (the action that you take in $s$) and $p$ (the dynamics of the environment).
Now, in their 3rd edition of the AIMA book, Russell and Norvig define fully observable environments as follows.

Fully observable vs. partially observable: If an agent's sensors give it access to the complete state of the environment at each point in time, then we say that the task environment is fully observable. A task environment is effectively fully observable if the sensors detect all aspects that are relevant to the choice of action; relevance, in turn, depends on the performance measure. Fully observable environments are convenient because the agent need not maintain any internal state to keep track of the world. An environment might be partially observable because of noisy and inaccurate sensors or because parts of the state are simply missing from the sensor data—for example, a vacuum agent with only a local dirt sensor cannot tell whether there is dirt in other squares, and an automated taxi cannot see what other drivers are thinking. If the agent has no sensors at all then the environment is unobservable.

This definition is also the common definition used in reinforcement learning. So, to determine whether an environment is fully or partially observable, you need to determine whether you have or not full access to a (Markovian) state or what constitutes a state in your case. You can consider chess fully observable because you have access to the configuration of the board, so, in theory, you can take optimal action by considering all possible moves of the opponent (but, of course, in practice, not even AlphaZero does this). See figure 2.6, p. 45, of the AIMA book (3rd edition) for more examples of fully and partially observable environments.
A partially observable MDP (POMDP) is a mathematical framework that can be used to model partially observable environments, where you maintain a probability distribution over the possible current (or next) state that the agent may be in.

I have provided the alternative view. Your use and definition of "partially observable" is fine IMO. What is incorrect is linking "game of perfect information" with "full observable" without considering variation in the opponent. What is going on with self-play chess players is slightly more complex, they solve the partially observable nature in a specific way.

You are correct in the question that in RL terms chess a game of chess where the agent is one player, and the other player has an unknown state is a partially observable environment. Chess played like this is not a fully observable environment.
I did not use the term "fully observable game" or "fully observable system" above , because that is not a reinforcement learning term. You may also read "game of perfect information" which is similar - it means there are no important hidden values in the state of the game which may impact optimal play. This is a different concern to understanding the state of your opponent.
Here is a counter-example showing that games of perfect information are not fully observable systems when you have an opponent with an unknown strategy:

Optimal play in tic tac toe leads to a forced draw.

Let's define a reward signal from the agent's perspective of +1 for a win, 0 for a draw, and -1 for a loss.

If the agent's opponent always plays optimally, then a RL agent will learn to counter that optimal play and also play optimally. All action choices will have an expected return of 0 or -1, and the agent will choose the 0 options when acting greedily.

If the agent's opponent can make a mistake that allows the agent to win, then there will be a trajectory through the game with a return of 1, or perhaps some other postive value in cases where the mistake is only made according to random chance.

The value of states in the game therefore depends on the opponent's strategy.

The opponent's strategy is however not observable - it is unknown and not encoded into the board state.

This should match your intuition when asking the question.
Why then, do many two player game-playing reinforcement agents for games like chess perform well, without using POMDPs?
This is because game theory on these environments supports the concept of "perfect play", and agents that assume their opponent will also attempt to play optimally - without mistakes - will usually do well. Game theory analyses choices leading to forms of the minimax theory - making a choice that your opponent is least able to exploit.
That does mean that such an agent may in fact play sub-optimally against any given opponent. For example, they could potentially turn some losing or draw situations into a win, but have little or no capability to do so unless trained against that kind of opponent. Also, playing like this may be a large risk against other opponents, it may involve playing sub-optimally at some earlier stage.
I have observed a related issue in Kaggle's Connect X competition. Connect 4 is a solved game where player one can force a win, and the best agents are all perfect players. However, they are not all equal. The best performers tweak their agent's choices for player two, to force the highest number of wins against other agents who have not coded a perfect player one. Different kinds of learning strategy lead to different imperfections, and the top of the leaderboard is occupied by the current best perfect agent that also manages to exploit the population of near-perfect agents below it in the rankings. This difference in the top-ranking agents is only possible due to the partially-observable nature of the Connect 4 game played against agents with unknown policies.

What exactly are partially observable environments?

They are environments where in at least some states, the agent does not have access to information that affects the distribution of next state or reward.
Chess played against an opponent where you have a model of their behaviour - i.e. their policy - is fully observable to the agent. This is implicitly assumed by self-play agents and systems, and can work well in practice.
Chess played against an opponent without a model of their behaviour is partially observable. In theory, you could attempt to build a system using a partially observable MDP model (POMDP) to account for this, in an attempt to try and force an opponent into states where they are more likely to make a decision that is good for the agent. However, simply playing optimally as possible in response to all plays by the opponent - i.e. assuming their policy is the same near optimal one as yours even after observing their mistake - is more usual in RL.
The original Alpha Go actually had a separate policy network for its own choices and modelling those of humans. This was selected experimentally as performing slightly better than assuming human opponents used the same policy as the self-play agent.

I'd argue that "the game of chess", with no additional qualifiers, is a fully observable, multi-agent environment. But if you want to apply single-agent RL algorithms, you have to pretend that it's a single-agent environment which you can do by "baking the opponent into the environment". I'd name that environment differently though, like "the game of chess with X as opponent", and then that might be a partially observable environment (or one that's still fully observable, but stochastic, if you know the opponent's policy!). I like both your and nbro's answer, depending on perspective.

@DennisSoemers I think I have been careful to add that condition (on knowledge about the opponent) everywhere I make a statement about chess being or not being partially observable. If I have missed one, or it is not clear anywhere please identify for me and I will try to fix it.

Urmm no don't think you necessarily missed it anywhere. Just thought it was a useful point to clarify / emphasise in a comment :D

Why do you need to know the opponent's strategy to make it fully observable? It would be like knowing the dynamics of the environment, which do not need to be encoded in the state to take the optimal action. If the optimal play leads to a draw (like in tic-tac-toe), that's orthogonal to what we're talking about, as well as the def. of the reward. To make it fully observable, you just need to know the state you are in (that's the definition of fully observable). Of course, if you know the opponent's strategy, you can exploit it to lead to a win (rather than a draw), but that's different.

It seems that you just shifted the definition of the state from the configuration of the board to also know the strategy of the opponent (i.e. the dynamics of the environment). In that case, yes, chess would generally be partially observable, but I don't understand why the opponent's strategy needs to be necessarily included in the state. To me, it seems that you're just mixing concepts here, but maybe I am wrong (I'm not an expert in this topic).

I have not shifted anything. It's the correct definition for RL environment. Whilst optimal play in a game theory sense would not need it. I think it is the OP who has mixed concepts, and this answer tries to correct that

@nbro: Another way to put it. RL: I don't know what the probilities are of opponent taking each kind of move, that means my MDP model is incomplete, and I need to do something about it. Game Theory: I don't care what the opponent does, I am going to choose a move where they will get the least benefit no matter what they decide. RL [addressed to Game Theory] - I'll use your assumptions then, that's going to be good most of the time.

The MDP is not "incomplete", it's "unknown". That's what most RL algorithms assume anyway. The MDP could also be stochastic, i.e. the strategy of the opponent could change over time. Can you please cite anyone (e.g. the Sutton and Barto's book or any other paper) that claims that chess is partially observable because we don't know the strategy of the opponent? I would like to read more about it.

@nbro: Agreed it is unknown if you train against the target opponent. It would be incomplete in simulation. So I have made an unstated assumption that we are talking about training a chess-playing RL in simulation or self play. If you have the actual human to train against, then conceptually the MDP is complete and learnable. But that is not what any RL self-play board game playing systems do

Sutton & Barto chapter 1 covers this, discussing tic tac toe. More accurately it is one of the working problems, but I do have the official answers to chapter 1 (from Sutton's office) and it matches my analysis here. I will see if I can find something linkable . . .

Best I have found so far is section 11.3 of https://www.cp.eng.chula.ac.th/~vishnu/gameResearch/AI/Ghory04%20RL%20in%20board%20game.pdf which indirectly references the issue, but does not directly use the term "partially observable" to describe it. I have to stop for now

I've read this discussion and comments again, and I realize that I said "The MDP could also be stochastic, i.e. the strategy of the opponent could change over time", while I clearly meant non-stationary. So, if the opponent's strategy is fixed, you can model the problem as an MDP and the fixed strategy of the opponent can be encoded into the dynamics of the MDP. However, if the strategy changes, then I think you could model the problem as a non-stationary MDP (but I am not sure how useful this might be), so I guess you wouldn't need to model it as POMDP, even for the self-play case.

I understand your statements about the game-theoretic view that assumes that the opponent will play optimally, so we will never lose but we may also not win even if the opponent makes a mistake. So, as Dennis also pointed out, these games with other agents, like chess, can be modeled as a multi-agent system, or we can view the opponents as part of the environment too.

A partially observable environment means it is from the agent's perspective that the agent observes the environment partially. At every time step, the agent takes action based on this partial observation. Based on the agent's action, the state of the environment changes, but the agent may not know all the changes.


text:
What is the effect of parallel environments in reinforcement learning?
Do parallel environments improve the agent's ability to learn or does it not really make a difference? Specifically, I am using PPO, but I think this applies across the board to other algorithms too.

Do parallel environments improve the agent's ability to learn or does it not really make a difference?

Yes they can make a difference. There are two ways improvement is seen:

Collecting data from multiple trajectories at once reduces correlation in the dataset. This improves convergence for online learning systems like neural networks, which work best with i.i.d. data.

Data collection is faster overall, which improves clock time to obtain the same result. This may make better use of other resources too.

Of the two, the first improvement is important for stability, although it can be emulated by running multiple episodes - or restarting from multiple starting points - between batch learning updates.

Specifically, I am using PPO, but I think this applies across the board to other algorithms too.

It does apply to PPO, but the first improvement does not apply across the board. These things need to be true for environment run in paralell to be help with stability:

Using an on-policy method, or where experience replay is not an option.

Using a function approximator for policy and/or value function.

A lot of policy gradient methods match this, including PPO, A3C, REINFORCE. However, for an off-policy method like DQN, the main benefit will be faster data collection.
These effects are discussed in sections 1 and 4 of the paper Asynchronous Methods for Deep Reinforcement Learning which introduced A3C - thanks to DeepQZero for that reference.

Why only for on-policy and why for no exp.replay?

@ddaedalus Experiance replay is only feasible for off-policy, because older records will have been made with a different policy - off-policy can handle that, on-policy cannot. If you have experience replay, you already have the benefit of improved convergence from parallel trajectories, because you already have closer to i.i.d. from sampling the experience replay memory.


text:
What is the difference between applying shallow-learning methods repeatedly and deep learning?
In the book Deep Learning with Python, François Chollet writes (section 1.2.6, page 18)

In practice, there are fast-diminishing returns to successive applications of shallow-learning methods, because the optimal first representation layer in a three-layer model isn't the optimal first layer in a one-layer or two-layer model. What is transformative
about deep learning is that it allows a model to learn all layers of representation jointly, at the same time, rather than in succession (greedily, as it's called).

By shallow learning, we mean traditional machine learning models that aren't deep learning, such as support vector machines.
I understood the above as below.

Using a model with three-layer shallow-learning methods has the same output (predicted) value as using one-layer shallow learning method. The effect of using multiple layers of shallow learning methods is to 'increase running time or repetition'.

Did I understand properly?

Quite surprising to find someone reading this same book. I read this part a week ago and the explanation is quite clear in the book :

If you use successive shallow learning methods, you first train one model, then you train another model with the outputs of your first model, and then a third with the outputs of your 2nd model. The problem with that is that each model is trained to get good results at his task, not to send information, so there can be an increase when adding successive models, but it is a very weak increase.
If you use deep learning, all the layers are trained at the same time, so each layer learns how to efficiently transfer important information to the next layer of the model. This is why it is much more efficient.

Hope I made it clearer

No, you did not correctly understood the meaning of the passage.

Using a model with three-layer shallow-learning methods has the same
output (predicted) value as using one-layer shallow learning method,
The effect of using multiple layers of shallow learning methods is to
'increase running time or repetition'.

A three-layer shallow learning method (so three one-layer methods stacked one after the other in a secuential way) has not the same predicted output value as a one-layer shallow learning method. It will be different output (and a 3 layers model should show some improvement) and clearly will need more time to get the result.
Deep learning methods as mentioned by @Ubikuity are (generally speaking) mode efficient at finding the important features and (usually) get better results than using a model with three-layer shallow-learning methods.


text:
What are the major differences between multi-armed bandits and the other well-known algorithms (DQN, A3C, PPO, etc)?
I have studied in the past different algorithms, i.e. DQN, DDQN, REINFORCE, A3C, PPO, TRPO, so on. I am doing an internship this summer where I have to use a multi-armed bandit (MAB). I am a bit confused between MAB and the other above algorithms.
What are the major differences between MAB and REINFORCE, for instance? What are the major differences between MAB and the other well-known algorithms (DQN, A3C, PPO, etc)?
EDIT
The @Kostya's answer is fine, but it would be interested for me to have a deeper answer for that question. I am still a bit confused.
Question: Do we use the Markov Reward formula $$G_t = R_{t+1} + \gamma R_{t+2} + ... = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}$$ the same way in a Multi-Armed bandit problem versus a DQN problem?

Hello. It's fine to edit your post to clarify the question, but your initial question asked for a general answer, which is different from your question asked in the last edit. If you have a more specific question, you can still ask another question but maybe in a separate post. Note that questions that ask the readers to write very long answers are not suitable for our site (so either you ask for the general idea or you need to ask specific questions, one for each post). Note also that the tag [tag:comparison] is suitable for this post, as you were asking people for a comparison/difference.

You should start with the general definition of Reinforcement Learning problem. And what Markov Decision Process is.
DQN, A3C, PPO and REINFORCE are algorithms for solving reinforcement learning problems. These algorithms have their strengths and weaknesses depending on the details of the underlying problem.
Multi-Armed Bandit is not even an algorithm - it is a subclass of reinforcement learning problems, where your environment (usually) doesn't have any state transitions and your actions are just a single choice from (usually) fixed and finite set of choices.
Multi-Armed Bandit is used as an introductory problem to reinforcement learning, because it illustrates some basic concepts in the field: exploration-exploitation tradeoff, policy, target an estimate, learning rate and gradient optimization. All these concepts are basic vocabulary in RL. I recommend reading (and, very importantly, doing all the exercises) the Sutton and Barto book chapter two to get familiarized with it.
Edit: since the answer got popular, I'll address the comments and the question edit.
Being a special simplified subset of Markov Decision Processes, Multi Armed Bandit problems allow deeper theoretical understanding. For example, (as per @NeilSlater comment) the optimal policy would be to always go for the best arm. So it makes sense to introduce "regret" $\rho$ - the difference between a potential optimal reward and the actual collected reward by agent following your strategy:
$$\rho(T) = \mathbb{E}\left[T\mu^* -\sum_{t=1}^T\mu(a_t)\right]$$
One can then study asymptotic behavior of this regret as a function of $T$ and devise strategies with different asymptotic properties. As you can see, the reward here is not discounted ($\gamma=1$) - we usually can study the behavior of it as a function of $T$ without this regularization.
Although, there is one famous result that uses discounted rewards - the Gittins index policy (note, though, that they use $\beta$ instead of $\gamma$ to denote the factor).

I think this answer misses where MABs are different from MDPs in terms of reasons for studying and using them. Whilst MABs can be treated as a strict subset of MDPs, and that is how S&B uses them - as a stepping stone towards learning about RL - there are also approaches and analysis that are used more often with MABs because of what they model. Optimising regret during learning is a good example of this. Not usually of concern in RL (and difficult to define and measure in MDPs), but of great interest in applications of MABs in real-world projects.

@NeilSlater Are you up to make a full answer? I would be interested in seeing what you have to say on the subject.

@NeilSlater I agree that you can go deeper into the usefulness of MABs. But, given that the question conflates RL algorithms with MABs I didn't feel like that this level of discussion would be helpful.

@Kostya Can we see DQN as a bandit problem?

@notaprogrammertoday Can one see linux as a CPU? Your question conflates completely separate concepts.

MAB is fundamentally different from MDP problem, which so-called well-known RL algorithms trying to solve. The point is that MAB's policy is exactly the action. The Q function takes only action(policy) $a$ as its input. Whereas in MDP, the policy is a function, The Q function takes not only state-action $s,a$ but also policy parameter $\theta$, which makes evaluation of policy and improvement of policy tangled together.
In other words, in MAB, if one has good evaluation of a policy(action), then optimal policy is directly given. But in MDP, if one has good evaluation of a policy, the optimal policy is still unclear. That is why one needs DQN to do policy Evaluation and policy Improvement together.
As for policy gradient policy like PPO, it takes another road to directly optimize the polciy to a more-promising gradient direction based on the feedback from reward.


text:
Why do we use the softmax instead of no activation function?
Why do we use the softmax activation function on the last layer?
Suppose $i$ is the index that has the highest value (in the case when we don't use softmax at all). If we use softmax and take $i$th value, it would be the highest value because $e$ is an increasing function, so that's why I am asking this question. Taking argmax(vec) and argmax(softmax(vec)) would give us the same value.

Short answer: Generally, you don't need to do softmax if you don't need probabilities. And using raw logits leads to more numerically stable code.
Long answer: First of all, the inputs of the softmax layer are called logits.
During evaluation, if you are only interested in the highest-probability class, then you can do argmax(vec) on the logits. If you want probability distribution over classes, then you'll need to exponentiate and normalize to 1 - that's what softmax does.
During training, you'd need to have a loss function to optimize. Your training data contains true classes, so you have your target probability distribution $p_i$, which is 1 at your true class and 0 at all other classes. You train the network to produce a probability distribution $q_i$ as an output. It should be as close to the target distribution $p_i$ as possible. The "distance" measure between two probability distribution is called cross-entropy:
$$ H = - \sum p_i \log q_i $$
As you can see, you only need logs of the output probabilities - so the logits will suffice to compute the loss. For example, the keras standard CategoricalCrossentropy loss can be configured to compute it from_logits and it mentions that:

Using from_logits=True is more numerically stable.

Note that the cross-entropy is really defined for 2 probability distributions, and I would say that's the reason why we use the softmax. Note that, if the logits are zero or negative, then the cross-entropy is not defined because of the logarithm. I think that TensorFlow accepts logits because it probably performs a (stable) conversion of the logits to a probability distribution under the hood, but I have not looked at the source code. So, it's a matter of diving into the TF implementation of the CE loss.

If you directly use the logits as "the logarithms of the probability", then what you say should make sense. However, why should you interpret the output of the neural network as the logarithm of the probability of belonging to a class? Still, it would be nice to show what TF exactly does when you set from_logits=True.

@nbro I kinda started explaining why use of logits is more numerically stable, but it turned out to be too mush of a tangent. So that deserves a separate question and answer in my opinion.


text:
How does Mask R-CNN automatically output a different number of objects on the image?
Recently, I was reading Pytorch's official tutorial about Mask R-CNN.
When I run the code on colab, it turned out that it automatically outputs a different number of channels during prediction. If the image has 2 people on it, it would output a mask with the shape 2xHxW. If the image has 3 people on it, it would output the mask with the shape 3xHxW.
How does Mask R-CNN change the channels? Does it have a for loop inside it?
My guess is that it has region proposals and it outputs masks based on those regions, and then it thresholds them (it removes masks that have low probability prediction). Is this right?

Object detection models usually generate multiple detections per object. Duplicates are removed in a post-processing step called Non-Maximum Suppression (NMS).
The Pytorch code that performs this post-processing is called here in the RegionProposalNetwork class. The filtering loop you've mentioned performs the NMS and applies the score_thresh threshold (although it seems to be zero by default).


text:
How is the state-value function expressed as a product of sums?
The state-value function for a given policy $\pi$ is given by
$$\begin{align}
V^{\pi}(s) &=E_{\pi}\left\{r_{t+1}+\gamma r_{t+2}+\gamma^{2} r_{t+3}+\cdots \mid s_{t}=s\right\} \\
&=E_{\pi}\left\{r_{t+1}+\gamma V^{\pi}\left(s_{t+1}\right) \mid s_{t}=s\right\} \tag{4.3}\label{4.3} \\
&=\sum_{a} \pi(s, a) \sum_{s^{\prime}} \mathcal{P}_{s s^{\prime}}^{a}\left[\mathcal{R}_{s s^{\prime}}^{a}+\gamma V^{\pi}\left(s^{\prime}\right)\right] \tag{4.4}\label{4.4}
\end{align}$$
It is given in section 4.1 of the first edition of Sutton and Barto's book (equations 4.3 and 4.4).
I don't understand how equation \ref{4.4} derives from equation \ref{4.3}. How can I get the product in equation \ref{4.4} from the expectation in equation \ref{4.3}?

A quick review of resolving expectations: If you know that a discrete random variable $X$, drawn from set $\mathcal{X}$ has probability distribution $p(x) = \mathbf{Pr}\{X=x \}$, then
$$\mathbb{E}[X] = \sum_{x \in \mathcal{X}} xp(x)$$
This equation is the core of what is going on when resolving the expectation in your quoted equation.
Resolving the expectation to show how the value function of a state relates to the possible next rewards and future states means summing up all possible rewards and next states. There are two components to the distribution over the single step involved - the policy $\pi(a|s)$, and the state progression $P^a_{ss'}$. As they are both independent probabilities, they need to be multiplied to establish the combined probability of any specific trajectory.
So, looking at only a single trajectory starting from state $s$, the trajectory of selecting action $a$ and ending up in state $s'$ has a probability of:
$$p_{\pi}(a,s'|s) = \pi(a|s) P^a_{ss'}$$
Iterating over all possible trajectories to get the expected value of some function of the end of the trajectory $f(s,a,s')$ looks like this:
$$\mathbb{E}[f(S_t, A_t, S_{t+1})|S_t=s] = \sum_a \pi(a|s)\sum_{s'}P_{ss'}^a f(s,a,s')$$
It is important to note that the sums are nested here, not separately resolved then multiplied. This is standard notation, but you could add some brackets to show it:
$$\mathbb{E}[f(S_t, A_t, S_{t+1})|S_t=s] = \sum_a \pi(a|s)\left(\sum_{s'}\left(P_{ss'}^a f(s,a,s')\right)\right)$$
In the equation from the book, $f(s,a,s') = R_{ss'}^a + \gamma v_{\pi}(s')$


text:
When should you not use the bias in a layer?
I'm not really that experienced with deep learning, and I've been looking at research code (mostly PyTorch) for deep neural networks, specifically GANs, and, in many cases, I see the authors setting bias=False in some layers without much justification. This isn't usually done in a long stack of layers that have a similar purpose, but mostly in unique layers like the initial linear layer after a conditioning vector, or certain layers in an attention architecture.
I imagined there must be a strategy to this, but most articles online seem to confirm my initial perception that bias is a good thing to have available for training in pretty much every layer.
Is there a specific optimization / theoretical reason to turn off biases in specific layers in a network? How can I choose when to do it when designing my own architecture?

Hello. To have more context, I suggest that you include the link to 1-2 examples of models that do not use the bias.

The most usual case of bias=False is in layers before/after Batch Normalization with no activators in between. The BatchNorm layer will re-center the data anyway, removing the bias and making it a useless trainable parameter. Quoting the original BatchNorm paper:

Note that, since we normalize $Wu+b$, the bias $b$ can be ignored since its effect will be canceled by the subsequent mean subtraction

Similar thing happens in transformers' LinearNormalization and (as far as I understand how conditioning works) in the GANs' conditioning layer - the data gets re-centered, effectively cancelling the bias.
In my experience, that's the most frequent reason to see bias=False, but one can imagine other reasons to remove the bias. As a rule of thumb, I'd say that you don't include bias if you want to "transform zeros to zeros" - things like learned rotations can be an example of such (rather exotic) application.


text:
Are optimal policies always deterministic, or can there also be optimal policies that are stochastic?
Let $M$ be an MDP with two states, $A$ and $B$, where $A$ is the starting state, and you always transit to the final state $B$ using two possible actions. $A_1$ gives you rewards that are normally distributed $\mathcal{N}(0, 1)$ and $A_2$ gives you rewards that are normally distributed $\mathcal{N}(0, 3)$.
How many optimal policies do we have? What is the optimal value of the states? Is any policy preferred over the other? Why? If you prefer one over the other, is there some way to detect it using what we have studied?
In my view, there are infinite policies that give the same expected reward.
$\pi_\alpha$ be a policy that is stochastic, which maps as follows - $\pi_\alpha(s, A_1) = \alpha $ and $ \pi_\alpha (s, A_2) = 1 - \alpha$ for $ \alpha \in [0,1]$. It is clear that, for each $\alpha$, we get infinite policies but have the same expected return.
But, according to some google searches (for example, here it says optimal policies are generally deterministic), I found optimal policies are always deterministic. Hence this implies there are only 2 policies, i.e., either take action $A_1$ or $A_2$ but not probabilistic.
So, my doubt is: what are the optimal policies here? Is it deterministic (only 2 policies) or stochastic (infinite)? Or is it an assumption that optimal policies are deterministic?

If you read my answer that you linked here - https://ai.stackexchange.com/a/10593/1847 - you can see that it includes your "combine equal optimimal policies" idea as the last paragraph. The reason this is at the end, is that it is an answer to a different question, the emphasis is different in order to correct the OP's misunderstanding (OP thought that stochastic MDPs requred stochastic optimal policies). The very last sentence of my answer: "You may construct a stochastic policy that mixes these in any combination, and it will also be optimal."

I think the result you are referring to is the one that says that there always exists a deterministic optimal policy for an MDP. This is true. But note that this does not imply that a stochastic optimal policy can not exist at the same time.
Suppose you have an MDP with one state and two actions $a_1$ and $a_2$, both yielding the reward 0 in expectation (as in your example). Then consider a policy that takes action $a_1$ with probability $\alpha \in [0,1]$ and $a_2$ with probability $1-\alpha$. Either of the two deterministic policies with $\alpha=0$ or $\alpha=1$ are optimal, but so is any stochastic policy with $\alpha \in (0,1)$. All of these policies yield the expected return of 0.
This is all assuming your optimality criterion is the expected cumulative (discounted) reward. If you have a different optimality criterion, such as something that accounts for risk, you might distinguish between rewards that have the same expected value but a different variance - but I think that is beyond the scope of the question.