text:
Is it possible learning convergence is lost in Reinforcement Learning as the state space grows?
I am new in the AI field and I am trying to use Reinforcement Learning. Specifically, I am using tabular Q-Learning and SARSA algorithms to solve a sequential decision making problem. (I am using TabularTDLearning.jl, q_learn.jl and sarsa.jl, with 16 GB of RAM).
When I apply those algorithms to a problem with 6.593 states and 169 actions, in less than a minute, it takes

200.000 iterations with Q-Learning, and
400.000 with SARSA

If I try to solve a more complex problem with 125.231 states and 2.197 actions, it does not converge, in fact it gets worse as iterations increase (5.000.000 iterations).
Is it possible learning convergence is lost as the state space grows?
I have read about catastrophic forgetting. I do not know if it is the problem why convergence has not happened. Could catastrophic forgetting be the problem?
For actions that cannot be taken, I have created a dummy state and a negative reward. Could this have affected the convergence of the algorithms?

With tabular reinforcement learning (RL) methods, then catastrophic forgetting does not come into play, as it is a feature of online learning with approximators such as neural networks. Essentially your table never forgets anything, although it may become out of date for rarely-visited state/action pairs whilst an approximator will often still maintain them at least a little due to generalisation.
The convergence time for RL methods is highly variable, and depends on many factors.
Tabular methods have relatively simple requirements for state/action visits though, and if your two environments are very similar, then it might be reasonable to expect the amount of experience required for convergence to scale roughly according to $O(|\mathcal{S}| \times |\mathcal{A}|)$ - that assumes you have no guiding principle to generalise state/action features or skip explorations.
Assuming your convergence sample size really is $O(|\mathcal{S}| \times |\mathcal{A}|)$ then extrapolating from your smaller to your larger environment implies you may need as many as 50 million iterations to get convergence on the larger instance. This is a crude aprpoximation, because so much depends on the nature of the changes - it may be that 5 million should be enough (and something is wrong with your implementation), or you may need 500 million. For instance one important factor I have left out of very similar is episode length (assuming you are solving for an episodic problem) - if that is also significantly larger, you may be looking at the 500 million case. Also critically important is how long the initial agent takes to find high scoring episodes.

I also want to add that in this case there are state dependent actions, so for actions that cannot be taken I have created a dummy state and a negative reward. Is it the reason?

Ideally you would not allow the agent to take actions that are not possible in a current state. You should check your library in case there is a way to inform the agent of its current set of actions instead of specifying an action space for the whole environment. If there are a lot of unavailable actions, it could be costing the agent a lot of time exploring them and learning not to take them.
Instead of adding "dummy" states, one approach would be to give a negative reward and have the agent transition to its current state - i.e. no change. The negative reward should be larger than any negative reward possible for valid action choices (otherwise the agent may prefer to collect the slightly-less-negative reward than it predicts it will get from taking a real action).
There are lots of changes you might consider to improve your agent to deal with harder-to-solve environments in reasonale time. I am not sure what you are able to consider, so here are a few open-ended ideas:

Use function approximation to generalise state and action features. This does expose you to a whole different set of complications though, such as losing guarantees of convergence, and also catastrophic forgetting.

Use eligibility traces and either Q($\lambda$) or SARSA($\lambda$). This can help you find a sweet spot for learning efficiency by considering multiple trajectory lengths for estimating returns. Alternatively, you could consider n-step returns.

Use an experience replay table, or Dyna-Q with Q-learning, to extract more data from previous experience. Whether or not this is useful depends on how fast the environment simulation runs. If your learning time is limited mostly by simulation speed of the environment, this is a good choice. If it is limited by the time taken by the Q-learning process, you are better off generating fresh data as fast as possible.

Hello and thanks for your answer. I run the 50 million episodes (yes, I'm working with episodic tasks) and it worked, so I'm sure my implementation is right and the extrapolation suposition was also right. My question now is: I have tried in the ``complex problem'' what is faster in the simple problem, so to speed up computations, should the same configurations be equally fast in both problems?


text:
Should I use an unsupervised approach or train a classifier with many classes to build a deep image feature extractor?
I'd like to build a deep feature extractor of images (using a Bi-linear CNN).
What would lead to the best results:

an unsupervised approach (such as https://iopscience.iop.org/article/10.1088/1742-6596/1237/3/032044/meta), or
training a classifier on as many classes as I can, and hope it generalized enough?

I would then like to use this extractor as:

A weights initialization for other classical tasks
A feature extractor for Few Shot Learning approaches

Depends on what your downstream task is. There are so many ways to pretrain image feature extractors, and you will have to choose them specifically based on their performance on you desired downstream task.

Please, edit your post to provide the details that are missing, which are mentioned in the previous task, i.e. what is/are your downstream task(s)?

Thank you for your answers, I've done so!

This is exactly the problem I am currently working on. I don't suggest that you use a supervised method to learn latent representation, as the model might learn shortcuts or only a most meaningful feature ignoring other latent features.
There are several self-supervised representation learning approaches:

SimCLR and SimCLR2
SwAV
SimSiam

Here is an overview of such approaches.
These approaches are aimed to learn a latent representation of images (pre-text task) that can be then used for any downstream tasks (classification, query similar samples). Here is an interesting article on this topic.
Another approach is to use generative models. The benefit of this approach is that you can also generate synthetic images from the embeddings that can help debug and evaluate your model.
One of the simplest architectures is autoencoders (VAE, VQ-VAE). Another option is generative adversarial networks. They are more difficult to train but can produce higher quality images than autoencoders. Another advantage is that the latent space is more disentangled meaning that each value might be semantically interpreted.
The main limitation is that the conventional GAN can produce samples only from random noise and to encode a real image the model has to be extended. There are different approaches to overcome this limitation, for example ALAE.
This paper compares some of these approaches.
While these approaches can easily capture coarse details such as hairstyle or background color, they are likely to ignore finer details. For medical imaging, the main goal is to identify cancer cells, which is essential for diagnosis. If the model does not learn this feature, it renders a latent representation of no avail.
However, latent space can be forced to take these features into account by guiding the model with a supervised downstream task such as classification (StylEX) or segmentation (EditGAN).

Thank you for the thorough comment! I'll accept it as the answer!

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What is the difference between features and inputs in machine learning?
I have seen many places that features and inputs have been used interchangeably when talking about machine learning especially deep neural networks. I want to know if they are indeed the same thing or there is a difference between between the two.

An input usually refers to an example (sometimes also known as sample, observation or data point) $x$ from a dataset that you pass to the model. For example, in supervised learning, you have a labelled dataset $D = \{(x_i, y_i)\}_{i=1}^N$, where $x_i$ is the $i$th input and $y_i$ the corresponding label (aka target or output).
This is similar to the terminology used for functions. For example, if you have the function $f: \mathcal{X} \rightarrow \mathcal{Y}$, then $x \in \mathcal{X}$ is the input and $f(x) = y \in \mathcal{Y}$ is the output of the function for that input $x$. In fact, models (like neural networks or linear regression models) are functions.
Examples

In image classification, an image can be an input
In machine translation, an input can be a sentence or a word (depending on the model)
In reinforcement learning, an input can be a state

A feature is an attribute associated with an input or sample. For example, a feature of an image could be a pixel. The feature of a state could be the Euclidean distance to the goal state. An input can be composed of multiple features.
It's possible that people also refer to features as inputs (in fact, if you pass e.g. an image to a model, you're also passing the pixels, the features, which are thus also inputs to the model). There are also other terms used to refer to these. For example, in statistics, people may refer to features as independent variables (or regressors), and maybe a sample refers to a dataset rather than a single observation. So, you should always take into account the context when reading these terms.
For more info, you could read this and this Wikipedia articles.

I read this article and it says  feature vector as a subset of data that is used to tackle a problem So is feature a subset of input the ML model finds more useful to deal with?

@user0193 I think it's more accurate to think of a feature as an attribute of the input rather than a subset because the input might not be a set. A feature vector is a vector of features. Sometimes, in ML, you convert your original input to a transformed input, which contains more "useful" features. For example, if you're doing image classification, you could pass an image of pixels ("raw input") to the model. But what if you first transformed this image into another input (a vector - the feature vector), where, e.g., element $i$ represents whether there's a human face or not in the image?

With the transformed input (the feature vector), the model now might perform better because the new features (presence of a human face or not) are more informative than the original ones (pixels). You could also think of a raw image (with pixels) as a feature vector, but I don't know if this is common. In summary, I think it's better to think of features as attributes of the inputs. For example, my laptop is gray. Being gray is an attribute (aka feature). The laptop is the input and the color is a feature (attribute) of the input.

So essentially in convolution neural network from the second layer till the second-last layer subsequently extracts features across several layers, i.e. as per your moments "transforming input" and final layer looks into this transformed layer and making final decision/output in the final layer. Right?

@user0193 Yes, you think of a CNN, but also other neural networks, as transforming the input and as (automatic) feature extractors (based on the data, training objective and learning algorithm).

An input involves everything as a data. For example in image classification, all images are input to model while features are the kind of specific properties of that input images based on model will decide the results.e.g in dog and cat dataset all images of dogs and cats is a input to model and properties such as ears, nose, other facial differences are the features of that input data.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What is the total number of actions and rewards count
Reading this two articles about Reinforcement Learning:

Deep Reinforcement Learning with Double Q-learning by Hado van Hasselt et al.
Human-level control through deep reinforcement learning by Volodymyr Mnih et al.

I'm trying to understand the dataset and how many states, actions and rewards have been used to train the agent on a single game. I presume they should be equal, if an action is made on some state that generates some reward.
The first article compares their results with second's results. Both refers to different count of frames, however it is not clear how many actions, rewards and states have the dataset of a single game. More than that, each game could have random states, actions, rewards, which is somehow an imprecise comparison, but I could not find a dataset used for first article and second one.
Please, somebody who have a better understanding tell what is the total number of states, actions and rewards for a single game that the agent was trained on, because I'm trying to understand the dataset, and if there exists such dataset please show the source.

TL;DR In the DQN paper, each environment was trained for 50 million frames, grouped in fours without overlap, so there were 12.5 million state, action, reward next-state records used.
The above direct numerical answer to the question as posed is not the whole story though.

I'm trying to understand the dataset and how many states, actions and rewards have been used to train the agent on a single game.

It seems you are viewing Reinforcement Learning (RL) through terms you understand from supervised learning. Although there are some similarities, especially when considering the neural network that performs function approximation inside most deep RL, there are important differences which mean the information you are looking for doesn't really exist.
The main purpose of RL is for an agent to learn through experience that it also helps to generate, through trial and error. Although off-policy algorithms such as DQN could learn from fixed datasets (and there are some circumstances where they might, perhaps as part of a more complex pipeline), this is not what the papers you reference are doing.
RL experiments can be repeated provided all random number generators (in the learning agent and in the environment) are seeded the same way. That may be of interest to researchers wanting to replicate work exactly, but more generally useful are statistical measures of "typical" learning graphs. The environment defines the problem to be solved, and the exact data generated through the RL agent exploring it in a set of runs is not necessarily as interesting when interpreting the results, or considering whether to use a certain type of agent for a specific problem. Having a reference copy of the environment identical to the one used in the paper is the equivalent in RL of having the "dataset".
You could log the state, action, rewards data from each experiment (and in DQN it is already being stored temporarily in the experience replay table), but you would not expect it to repeat on the next run even with the same agent. Also, when comparing to some other agent (or agent hyperparameters), one major point of difference is that you would expect that agent to generate a different set of states, actions and rewards - choosing how and when to explore different parts of the environment is a key differentator between agents. So it is usually not considered a critical part of the work to log every state. That's not to say that researchers won't do it - presumably many do keep records for debugging or simply to be thorough. However, more important in published RL results is to characterise the learning curves against some metric such as the amount of experience required, or another cost such as total CPU resource or time used in training.
In the original DQN paper, I believe that each 4 frames were stacked separately into a single state, without overlap, thus each 4 frames equals one state, for which a single action would be taken and then reward and next state observed. So dividing the frame count by 4 should give you the "dataset size". But it is important to note that this was never used as a whole in a supervised learning manner, and was not re-used between runs. The experience replay buffer was sampled randomly on each timestep, and there was an average number of times each sample was used (32 I think) to train the network - you might compare that to running 32 epochs of training across all state/action pairs as inputs, but it is only a loose comparison due to the sample-with-replacement sampling strategy and the sliding window effect of the experience replay.

Thanks for clarifying the differences and other stuffs, but I'm looking to explore a method that uses less interaction between the environment and agent, so I think the number of trials should be equal to the number of actions made by the agent, also equal to the number of rewards. In this case I can say that the performance measures of number of trials is better or worse. That's why I need to know the number of actions made by an agent in a single game.

@Jigberto: Your question is how many states/actions and rewards for a single game. I have now answered that at the top (it is 12.5 million per game). Unless you mean by "per game" per episode (i.e. number of frames from starting a game session to completing it due to winning or losing all lives or some kind of timeout/reset). The issue with per episode is that it will vary hugely per type of game and how well the agent is currently performing. To the point that is a meaningless statistic in the papers you reference, except for games that reward simple survival time.

In those articles were mentioned different number of frames like: 1M, 10M, 50M, 200M. In the second article was mentioned "... Training is done over 50M steps (i.e., 200M frames)." which seems like you said 200M / 4 = 50M actions(steps) ... still not sure

I took 50M frames from the text of DQN paper and have not read the DDQN paper. It is entirely possible that learning curves or results are published from multiple points in the training proces for comparison. This would be useful data to understand how the agent performs. For the original DQN paper I think you can safely divide any frame count given by 4 to get number of steps, which will equal the number of states, actions and rewards.


text:
What is the correct partial derivative of $Y^c$ with respect to $A_{ij}^{kc}$?
I have a question about the Grad-CAM++ paper. I do not understand how the following equation (10) for the alphas is obtained:
$$
\alpha_{ij}^{kc} =
\frac{\frac{\partial^2 Y^c}{(\partial A_{ij}^k)^2}}
{2\frac{\partial^2 Y^c}{(\partial A_{ij}^k)^2}
  + \sum_{ab} A_{ab}^k \{ \frac{\partial^3 Y^c}{(\partial A_{ij}^k)^3} \}}
\qquad (10)
$$
I found various issues with the derivation, here I will focus on the step from equation (7) to (8).
Equation (7) without the relu function is:
$$
    Y^c =
    \sum_k \Bigl( \Bigl\{ \sum_{a,b} \alpha_{ab}^{kc}
    \cdot \frac{\partial Y^c}{\partial A_{ab}^k} \Bigr\}
    \Bigl[ \sum_{i,j} A_{ij}^k \Bigr] \Bigr)
    \qquad (7)
$$
Here $Y^c$ is supposed to be a function of the $A_{ij}^k$, and $\alpha_{ab}^{kc}$ are unknowns to be determined.
The next step consists of computing the partial derivative of (7) w.r.t. $A_{ij}^{kc}$, which according to the paper yields equation (8):
$$
  \frac{\partial Y^c}{\partial A_{jk}^k} = \sum_{a,b} \alpha_{ab}^{kc}
  \cdot \frac{\partial Y^c}{\partial A_{ab}^k} +
    \sum_{a,b} A_{ab}^k \Bigl\{ \alpha_{ij}^{kc}
    \cdot \frac{\partial^2 Y^c}{(\partial A_{jk}^k)^2} \Bigr\}
    \qquad (8)
$$
However, when I did the computation myself I got the following:
$$
  \frac{\partial Y^c}{\partial A_{ij}^k} =
    \sum_{a,b} \alpha_{ab}^{kc} \cdot \frac{\partial Y^c}{\partial A_{ab}^k} +
    \sum_l \Bigl(\Bigl[\sum_{u,v} A_{uv}^l \Bigr]
    \Bigl\{ \sum_{a,b} \alpha_{ab}^{lc} \cdot
    \frac{\partial^2 Y^c}{\partial A_{ij}^k \partial A_{ab}^l} \Bigr\} \Bigr)
    \qquad (8')
$$
Note the extra sum, and the cross-derivatives.
What is right, (8) or (8')?

After some reflection I noticed that the actual final expression should contain derivatives of the alphas w.r.t. $A_{ij}^k$ too, because the alphas cannot be constants that do not depend on $A_{ij}^k$.  So, the equation becomes:
$$
\frac{\partial Y^c}{\partial A_{ij}^k} =
\sum_{a,b} \alpha_{ab}^{kc} \cdot \frac{\partial Y^c}{\partial A_{ab}^k} +
\sum_l \Bigl(\Bigl[\sum_{u,v} A_{uv}^l \Bigr]
\Bigl\{ \sum_{a,b} \alpha_{ab}^{lc} \cdot
\frac{\partial^2 Y^c}{\partial A_{ij}^k \partial A_{ab}^l} \Bigr\} \Bigr) +
\sum_l \Bigl( \Bigl[\sum_{u,v} A_{uv}^l\Bigr]
\Bigl\{ \sum_{a,b} \frac{\partial \alpha_{ab}^{lc}}{\partial A_{ij}^k} \cdot \frac{\partial Y^c}{\partial A_{ab}^l} \Bigr\} \Bigr)
$$
More details in the appendix of this document.
I believe this is the final answer to my question.


text:
Is the "Helvetica scenario" mentioned here related to Artificial Intelligence?
Consider the following sentence from the original GAN paper titled Generative Adversarial Nets

in particular, $G$ must not be trained too much without updating $D$, in order to avoid "the Helvetica scenario" in which $G$ collapses too many values of $\mathbf{z}$ to the same value of $\mathbf{x}$ to have enough diversity to model $p_{\text {data }}$

The sentence uses the name of a scenario called Helvetica. Is it related to AI or chemistry? If AI, then where can I read more about the scenario?

The "Helvetica scenario" referenced in the paper is an AI issue related to generative systems failing.
In GANs, this might be due to a generator becoming initially strong against the discriminator by focusing on its best output results so far that cover just a subset of the full dataset - i.e. gradients move the generator towards output of a single best class, suppressing output of classes that the discriminator can spot easily, and encouraging output of classes where the discriminator has more trouble. By the time the discriminator catches up with the generator, it may be too late for the combined training system to learn its way out of this state.
In GANs, the term Helvetica scenario from the paper is more commonly referred to as mode collapse in later literature. If you search for "GAN mode collapse" you will find a lot more information about it. There are a few different improvements to GAN design - loss functions and other hyperparameters - that work to reduce the incidence of mode collapse.

From this answer on Data Science SE, it seems like the authors named the effect after a science parody sketch in the show Look Around You, where the structure of a molecule collapses. The "science" in the sketch is complete nonsense.


text:
Mathematically, what is happening differently in the neural net during exploration vs. exploitation?
I want to understand roughly what is happening in the neural network of an RL agent when it is exploring vs. exploiting. For example, are the network weights not being updated when the agent is exploiting? Or somehow being updated to a lesser degree?
If this question is algorithm-dependent, I'm mostly interested in PPO, but any insight is appreciated.

Typically, the NN is trained the same way whether an action is chosen for exploration or exploitation. Look at the objective (AKA loss) function for any algorithm you're interested in and you'll probably find that it doesn't contain a term for exploration vs exploitation.
Instead, the choice of exploration vs exploitation, and the tradeoff, is about how the data used to train the neural network are generated.
So in a rough sense, what's happening to the NN is that during exploration, the network is being trained on novel data (not always, but it's more likely), while during exploitation, the network is being trained on well worn paths.
Then even though the gradient formula is the same, the gradient may be smaller and the weights may change less during exploitation, because the loss is typically lower when choosing the optimal action, and with some loss functions (like MSE, but not L1 loss) a bigger loss results in a bigger gradient.

"...you'll probably find that it doesn't contain a term for exploration vs exploitation" - this is not true - think of $\epsilon$ in $\epsilon$-greedy, but maybe you meant something else?

But  is not used in the objective (aka loss) function. Sorry if my terms were confusing there, I'll add (AKA loss).

Ok. Note that the objective function (e.g. minimization of the discounted reward) may contain a discount factor, which can play a role in or affect the exploitation-exploration dilemma. So, this dilemma is actually not just a practical issue.


text:
Is Q-learning only capable of learning a deterministic policy?
I was following a reinforcement learning course on coursera and in this video at 2:57 the instructor says

Expected SARSA and SARSA both allow us to learn an optimal
$\epsilon$-soft policy, but, Q-learning does not

From what I understand, SARSA and Q-learning both give us an estimate of the optimal action-value function. SARSA does this on-policy with an epsilon-greedy policy, for example, whereas the action-values from the Q-learning algorithm are for a deterministic policy, which is always greedy.
But, can't we use these action values generated by the Q-learning algorithm to form an $\epsilon$-greedy policy?
We can, for instance, in each state, give the maximum probability to the action with the greatest action-value and the rest of the actions can have probability $\frac{\epsilon}{\text{number of actions}}$. Because we do a similar thing with SARSA, where we infer the policy from the current estimate of action-values after each update.

If we assume a tabular setting, then Q-learning converges to the optimal state-action value function, from which an optimal policy can be derived, provided a few conditions are met.
In finite MDPs, there's at least one optimal (stationary) deterministic policy, but there can also be optimal stochastic policies - specifically, if two or more actions have the same optimal value, then you can stochastically choose between them. However, if you stochastically choose between all actions (including non-optimal ones), then you will not behave optimally.
SARSA also converges to the optimal state-action value function, but the learning policy must eventually become greedy. See this post and theorem 1 (p. 294) of this paper.
So, even in SARSA, if you want to behave optimally, you can't just arbitrarily choose any stochastic policy derived from this found optimal value function (note also that SARSA is on-policy). However, SARSA can also find an optimal restricted policy. See theorem 2 (p. 297) of this paper for more details.
To answer your question directly, Q-learning can find an optimal stochastic policy, provided that it exists.

So if I understand the post correctly, even in SARSA we decrease exploration (by decreasing epsilon in case of epsilon greedy) as time progresses to ensure that we have the optimal value function?

@ketandhanuka Yes, as far as I remember, $\epsilon$ must eventually decay in order for SARSA to find the optimal value function.

$\epsilon$-greedy policies with a fixed $\epsilon$ can be ranked. So SARSA will converge on the optimal $\epsilon$-greedy policy (i.e. the optimal polciy given that a specific $\epsilon$ value applies) even without decaying $\epsilon$. Sometimes this is desirable, for instance in online scenarios that never complete learning.

@NeilSlater I updated my answer to improve the precision of my explanations. SARSA converges to the optimal value function with decaying behaviour policies, but it can also converge to a policy called (here) an optimal restricted policy, which is the optimal policy that chooses actions according to the their rank (as far as I understand it). I am not sure if it's correct to say that SARSA converges to an optimal $\epsilon$-greedy policy, though - $\epsilon$-greedy policies don't choose actions according to their rank.

Maybe you're referring to another result about $\epsilon$-greedy policies in the context of SARSA. But, in this paper, what you say only applies to restricted rank-based randomized (RRR) policies, i.e. policies that choose actions according to their ranks.

Yes I meant the restricted policy. By "ranked" I meant that the policies can be compared, so optimality is well defined. I used the wrong terminology, sorry.


text:
How to perform the back propagation step in Semi-gradient SARSA using a deep neural network?

For the back weight update step, I need to calculate $\nabla\hat{q}(S,A,w)$. My neural network takes in the state vector $S$ and gives out the action values for state $S$ and each action in the action space.
Since $\epsilon$-greedy randomly chooses between the greedy action and a random action, I was not sure what the loss for the network should be. Let's say I have 3 possible actions - $a_1, a_2, a_3$ and my neural network outputs the vector of action-values for each of these actions in state $S$. If $a_3$ is the action chosen by $\epsilon$-greedy, then is this a correct expression for $\hat{q}(S,A,w)$?
\begin{equation}
\hat{q}(S,A,w) =
\begin{bmatrix}
    \hat{q}(S,a_1,w)  &  \hat{q}(S,a_2,w) &  \hat{q}(S,a_3,w)
\end{bmatrix}
\times
\begin{bmatrix}
    0       \\
    0       \\
    1
\end{bmatrix}
\end{equation}
If so, then would it be correct to take the gradient of this expression of $\hat{q}(S,A,w)$ with respect to the weights of the network and plug into into the following update equation for back propagation?
\begin{equation}
w = w + \alpha~[R + \gamma \hat{q}(S',A',w) - \hat{q}(S,A,w)]\nabla\hat{q}(S,A,w)
\end{equation}
I was trying to use this in the Lunar-Lander Open-AI gym environment but it never converged. So I wanted to make sure this approach is correct before trying other things.

If so, then would it be correct to take the gradient of this expression of $\hat{q}(S,A,w)$ with respect to the weights of the network

Yes, your expression for $\hat{q}(S,A,w)$ looks correct for your network, and can be differentiated.

and plug into into the following update equation for back propagation?

Not clear what you intend. If you already have the gradient $\nabla\hat{q}(S,A,w)$ then you have already performed backpropagation from the current output value to all weights. The additional factor of $[R + \gamma \hat{q}(S',A',w) - \hat{q}(S,A,w)]$ is effectively the gradient of the MSE cost function w.r.t. the current output that needs to be multiplied in so that the backprop starts from the MSE loss.
The equations given in Sutton&Barto are not that easy to apply when you already have a neural network library that has everything defined around supervised learning.
You may find it easier to calculate a target value for the whole action value vector and use your library's definitions of loss functions and update steps. That would go something like this:

Set the loss function of the neural network to MSE.
Set the optimisation method to SGD (although you now have the option of using others e.g. Adam, momentum, mini-batches etc)
Calculate $\begin{bmatrix}
    \hat{q}(S,a_1,w)  &  \hat{q}(S,a_2,w) &  \hat{q}(S,a_3,w)
\end{bmatrix}$ forward using the neural network.
Calculate the TD target $G = R + \gamma \hat{q}(S',A',w)$
Substitute the target value into the forward matrix e.g. $\begin{bmatrix}
    \hat{q}(S,a_1,w)  &  \hat{q}(S,a_2,w) &  G
\end{bmatrix}$ assuming it was action 3 that the agent took.
Use the adjusted forward matrix as the SGD training value.

This will automatically provide a gradient 0 associated with actions that were not taken, and is effectively the same update as per Sutton&Barto, but using the NN library as-is without trying to tweak the gradient calculations directly.
If you are using a library like PyTorch, then it is definitely possible to calculate the pure gradient of the selected action as you are attempting. However, you may find yourself in more familiar and supported territory if you re-frame the updates as supervised learning as suggested above.
I suggest one of the first things you try if this does not converge is to batch up the updates and only perform them every e.g. 100 steps. Again, this is possible using the S&B update rule, and something like PyTorch, but things will be easier if you go with the grain and re-frame the updates as supervised learning "inside" the RL mechanism.
I am not sure how well SARSA will cope with Lunar Lander. I have solved it using DQN, which is similar, but has the advantage of re-using older off-policy experience, sampling from it randomly, which SARSA cannot do. To correct for SARSA's weakness there, you may need quite large batches to avoid too much correlation between experience. A common pattern with "Deep SARSA" is to sample multiple environments at once under the current policy, to build a large batch (you see this in A2C and other on-policy policy gradient methods too).


text:
Why don't we also need to approximate $p(x \mid z)$ in the VAE?
In the VAE, we approximate the probability distribution $p(z \mid x)$, where $z$ is the latent vector and $x$ is our data. The reason is that $p(z \mid x)$ becomes impossible to calculate for continuous data because of $p(x)$, which require integration (not in closed form) to be solved.
But why don't we also need to approximate $p(x \mid z)$?
What I can guess here is that, in VAEs, we assume $p(z)$ (prior), so we are able to calculate $p(x \mid z)$, but for $p(x)$ we can't assume its distribution? Is it right?

What I can guess here is that, in VAEs, we assume $p(z)$ (prior), so we are able to calculate $p(x \mid z)$, but for $p(x)$ we can't assume its distribution? Is it right?

You could assume $p(x)$ is some distribution (e.g. the Gaussian $\mathcal{N}(0, 1)$), but that assumption might be completely wrong - you can also check that against your given data (i.e. if your data is unlikely under your assumed distribution, then maybe your assumption is unreasonable - see this or this).
The ultimate goal of a generative model, like the VAE, is to be able to generate data (so you want to use it for tasks like image denoising).
So, you want to learn a probability distribution that approximates the distribution from which your given dataset was sampled that would allow you to sample more similar data. In practice, you want to learn a probability distribution of the form $p(x, z)$, where $z$ is some latent random variable, which we assume to be involved in the process of generating $x$ (a random variable that represents the type of data that you have), i.e. $x$ depends on $z$.
By definition, $p(x, z)$ can be written as $p(x, z) = p(x \mid z)p(z)$ or $p(x, z) = p(z \mid x) p(x)$. If you put together these equations, you obtain the Bayes rule/theorem $p(z \mid x) = \frac{p(x \mid z)p(z)}{p(x)}$, where $p(z \mid x)$ is the posterior, $p(x \mid z)$ the likelihood, $p(z)$ the prior, and $p(x)$ the marginal (likelihood) or evidence.
In the VAE, we use the variational distribution, $q_\phi$, to approximate $p_\theta(z \mid x)$, but we also learn $p_\theta(x \mid z)$ - in fact, that's the decoder - and we learn it jointly with the variational distribution by maximizing the ELBO. We maximize the ELBO because it's a lower bound on $p(x)$, so, by maximizing it, we're also maximizing the marginal likelihood (aka evidence) $p(x)$, i.e. we're trying to find the parameters $\theta$ and $\phi$, such that $x$ is more likely to have been sample from $p(x)$, which is unknown. See the VAE paper (in particular, algorithm 1) for more details.
Once the VAE is trained, you have $q_\tilde{\phi}(z \mid x)$ and $p_\tilde{\theta}(x \mid z)$. If $p_\theta(z \mid x)$ is the posterior, then $q_\tilde{\phi}(z \mid x)$ is an approximation of the posterior. Now, you can generate data with the VAE as follows

Let $q_\tilde{\phi}(z \mid x)$ be the new prior (as opposed to the user-defined $p(z)$, which was used to train the VAE) - in Bayes statistics, the posterior then becomes the prior (it's our new belief of how $z$ behaves given our data $x$)
Sample $z^i$ from $q_\tilde{\phi}(z \mid x)$
Sample $x^i$ from $p_\tilde{\theta}(x \mid z^i)$

So, together, $\color{blue}{p_\tilde{\theta}(x \mid z^i)}$ and $\color{red}{q_\tilde{\phi}(z \mid x)}$ approximate $p(x, z) = \color{blue}{p(x \mid z)} \color{red}{p(z)}$.
It might be possible to come up with a new objective function (different from the ELBO), where we use some kind of variational distribution to approximate $p_\theta(x \mid z)$, but, in the VAE, that's not the case.


text:
Why are only neural networks (and not SVMs, for example) used for reinforcement learning?
I know that neural networks are the "universal function approximator", but they also have a huge number of trainable parameters and are extremely prone to overfitting.
So my question is: Why aren't SVMs or Random Forests used as the mediators in decision-making?
I have read this response, and it highlights very broadly why NNs are not required but are generally good for the job.
This does not answer my question, though, as I want to understand why no popular implementations (that I've seen) use anything but neural networks, and why the vast majority of reinforcement learning literature talks about Deep RL and not, for example, "SVM-RL" or something like that.
My understanding is that this is technically possible - but why is it not being done often or at all?

Hi, I have linked this to a question that I think is a duplicate. I have a high rep on RL, so it closed it immediately instead of putting it to a vote. If you think there is something missing from the linked question and answer, please modify your question to make that obvious and comment. I'm happy to vote to reopen if there is some new angle on this question that you want answered. I also up-voted, because (natural) duplicates are a useful thing to help others searching

@NeilSlater Thank you. I have modified the question to be more specific, let me know if you think that's good.

The biggest problem with SVMs, random forests, gradient boosting and others for reinforcement learning (RL) is that they are not able to learn online, adjusting for new data as it arrives, and equally importantly forgetting older data. These algorithms learn from fixed datasets, and must be re-trained with whole new datasets if the target dataset changes, in scenarios also known as non-stationary problems.
This is a major issue in reinforcement learning for control problems (when you want to find an optimal policy). RL control problems always generate non-stationary target data, because they maintain a current policy - or for off-policy approches, current target policy. That current policy then changes over time as the agent improves its estimates (of e.g. the Q table), which means it is necessary to forget older training data and replace with newer.
So one requirement in RL is for online learning algorithms that can handle non-stationary data well - both incorporating new data and smoothly forgetting older data. That includes neural networks. There are not many other non-linear approximators that can do so simply.
It is possible to compensate for this lack of online learning in various ways. Some of the model classes, such as random forests and support vector machines, can be extended to learn online - although this seems not to be explored much in deep RL, perhaps due to complexity, or limitations that make it still less flexible choice than neural networks (opinion: looking at the recent dates on these papers compared to the longer history of using NNs with RL, there may be an element of intertia too, and possibility for revisiting using these models in a RL context).
You could modify the outer algorithm to be more like policy iteration, collecting large amounts of data for the current policy to form a large batch that is then learned from to replace the previous approximate value function and in turn update the policy from that. That would work for most approximators in theory, but may be very slow and introduces hyperparameter of batch size that will have a large impact.
There are unwanted side-effect to online learning in RL using approximators, such as the risk of catastrophic forgetting, where a well-trained agent is not able to distinguish bad choices from good ones because it has not experienced a bad decision for many time steps, and has generalised that "everything is good now". This would not be fixed in RL by using large batches and non-online algorithms - if anything it could be worse in those cases, because they would effectively be forced to forget everything between batches. The usual fix for catastrophic forgetting in RL is adding some management of the dataset - e.g. keeping some amount of older experience even though it is technically not part of the current policy's generated data.

they also have a huge number of trainable parameters and are extremely prone to overfitting

Overfitting of approximators can happen, but may be less of an issue in practice with RL compared to smaller fixed datasets, because agents are continuously and actively generating new data. Complex environments with lots of resources used to generate experience behave in a lot of ways like "big data" datasets, playing to the strengths of large flexible models.
In RL there is commonly an overlap with processing image data (and possibly other sensors with return data that can be structured into some grid or graph). In the problem domains related to image comprehension, neural networks are state of the art, so it is no surprise to see them used for approximating action values in the orginal DQN paper for example - with the right architecture NNs are measurably superior, in terms of error metrics, to SVMs, RFs, boosting et al, when fitting this kind of sensor data.

Comments are not for extended discussion; this conversation has been moved to chat.


text:
For continuing tasks, is the choice of episode length completely arbitrary?
Let's say I'm training a reinforcement learning agent to act in some environment that perpetually continues to give the agent opportunities to earn rewards, and there is no cap on the score and there is no way to "win". That is, there is no natural "end" to an episode.
In these scenarios, is the choice of episode length completely arbitrary? Is it easier to train on shorter episodes than longer ones?

There are things that impact ideal pseudo-episode length for learning continuing (non-episodic) environments:

Start state. The start state of a continuing environment may be special in some way and unreachable later on. It is still important to learn optimal behaviour, and some choices may even be critical in order to reach the best areas of state space. If there is a standard/fixed start state, it means you may want to reset and restart pseudo-episodes more frequently.

Statistical coverage of repeating states. In a continuing environment under a fixed policy, there should be a set of ergodic states. That is a set of states that are visited at some expected long-term frequency (or frequency density for non-discrete state spaces) due to the combined behaviour from the policy and environment transitions. You want to sample from this set fairly, in an unbiased manner, in order to accurately learn expected values. This means you may want to reset and restart pseudo-episodes less frequently, to get long trajectories where the start state distribution has less impact on the relative frequency of each state in this repeating set.

These two issues are at odds with each other, so choosing a pseudo-episode length (when to artificially stop and re-start) is a hyperparameter that may require experimentation. The hyperparameter interacts with others, such as discount factor - for example a small discount factor may mean you care less about fair sampling of continuing states, although you will still care about visiting enough of them to get estimates from all reachable states.
There can be other issues that impact your choice, such as the RL method you are using, whether you are using approximation, whether there are some form of "attractors" that tend to get the agent stuck in inescapable loops. If you know you have these kind of trap state loops in your environment, you may want to treat that as similar to having special start states, and reset more often.

In these scenarios, is the choice of episode length completely arbitrary?

No, it is a hyperparameter that you may be able to take an educated guess at or figure out from observing behaviour.

Is it easier to train on shorter episodes than longer ones?

It depends on the environment, and there is no generally applicable preference for short or long pseudo-episodes.


text:
Do I need to create one or many neural networks to play Risk?
I have a school project to develop an AI model that plays the Risk board game as optimally as possible. Now, I have made the environment of Risk in Python and I narrowed down my possible machine learning solutions to either a neural network or using a genetic algorithm. I was thinking of using a neural network but I have a question that I can't find the answer to. I also tried searching online on Github and other thesis.
Do I have to create 1 NN to control everything? Or do I have to make one for attacking decisions, one for decisions regarding moving troops etc.?

Hey, I'm also very much a beginner here so I'm not going to post an answer, but from what I've read/seen, it's up to you - you can have 1 NN control everything (as long as you provide a proper action space), but you can also have multiple agents responsible for different things (which I believe is called "hierarchical RL"). Maybe worth trying just one NN and see if you can get good results with it? If not, try multiple? I saw a vid where one NN was used to determine broad goal/strategy "attack/preserve/defend" and another was used to determine specific actions

Thank you for the help, would you be able to link the video for me please? As I think it would be of massive help.

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

It's unclear what you know about AI and ML. How would you train these neural networks?

Here is the video: https://www.youtube.com/watch?v=q59wap1ELQ4

What a fantastic problem!  Also, welcome to AI.
The challenge, and it isn't terrible, is that you have to build an NN that can ingest the problem.  How do you pose the problem to the learner?
Here is how they setup AlphaGo:
What is the input to AlphaGo's neural network?
There are lots of bad ways to do this, so lets look at things that have to be done to not do it as badly.
The input format must communicate:

every country on the map
the number and owner of every army on the map
the cards in the current players hands
the current number of armies for a turn-in

It would also be helpful if it could communicate:

the last few moves for all the players
something about the connections between countries, how armies move, etc  (while it can be inferred, it is a good practice to make the learner figure out the hard parts by feature-engineering the easy parts.

There is a decent open-source MuZero on GitHub.  You could spend horsepower defining the game and states, but you would also need to make sure the learner was capable of figuring things out eventually.

Thank you for this detailed comment. It helped greatly.

@Xibby - let me know how it goes.  Y'all should have a Github, and push it there.  It is the new resume, a track of contributions and engagement in doing cool things.  There are very cool people who look there to find people to offer jobs to, very nice jobs.


text:
Why does the average-reward estimator for continuing tasks use the TD error?
In Sutton and Barto's RL book, section 10.3 describes how to use average reward $r(\pi)$ to define the quality of a policy, re-defining action-value function $q_\pi(s,a)$ and value function $v_\pi(s)$ to measure the expected differential return $G_t^{diff} =\sum_{t=0}^\infty \left[R_{t} - r(\pi)\right]$ of policy $\pi$ in a state/action pair, or state, instead of its discounted return $G_t^{disc}\sum_{t=0}^\infty \gamma^t R_t$.
Of course, the average rate of reward for a given policy cannot be known in advance but only estimated, so the authors propose keeping a running estimate $\bar{R} \rightarrow r(\pi)$ that starts at 0 and is then updated as $\bar{R}_{t +1} = \bar{R}_t + \beta \delta$, where $\beta$ is a learning rate and $\delta = R_{t + 1} - \bar{R}_t + \hat{q}(S_{t + 1}, A_{t + 1}) -  \hat{q}(S_{t}, A_{t})$ is the action-value TD error, in a SARSA context where $A_{t+1}$ is sampled; I understand we might as well use the value function TD error $R_{t + 1} - \bar{R}_t + \hat{v}(S_{t + 1}) -  \hat{v}(S_{t})$ if we are estimating that.
My question is: why does the update rule for the average rate of reward estimate consider TD error, and why are we not updating it simply as $\bar{R}_{t + 1} = (1 - \beta) \bar{R}_t + \beta R_{t+1}$ ?

Mystery solved thanks to Exercise 10.8 in the book. The reason is that we want the running mean to converge to the actual value of the average reward.
With $\bar{R}_{t + 1} =  \bar{R}_t + \beta \delta$ with $\delta = {R}_{t + 1} - \bar{R}_t + \hat{v}(S_{t + 1}) - \hat{v}(S_t)$, assuming value function estimation converged I would get
$\begin{align}
\delta &= R_{t + 1} - \bar{R}_t + v(S_{t + 1}) - v(S_{t}) \\
&= R_{t + 1} - \bar{R}_t + (R_{t + 2} - r(\pi^*) + R_{t + 3} - r(\pi^*) + ...) - (R_{t + 1} - r(\pi^*) + R_{t + 2} - r(\pi^*) + ...) \\
&= R_{t + 1} - \bar{R}_t - (R_{t + 1} - r(\pi^*)) \\
&= r(\pi^*) - \bar{R}_t
\end{align}
$
And the estimation will converge to the average reward of the optimal policy, being updated as $\bar{R}_{t + 1} = (1 - \beta) \bar{R}_t + \beta r(\pi^*)$.


text:
Why do we train the discriminators k times but train the generator only 1 time in a iteration in GAN?
In this paper https://arxiv.org/abs/1406.2661 , the codes for training a gan are :

Why do we train the discriminator for $k$ steps while the generator only for $1$ step? Why not the other way around?

The discriminator is composing a nonlinear measure of goodness approximation, an advanced version of the loss.  If it gets pathological, then the generator is worse off with the GAN-wrapper to train it.  Smart folks wire a training run of the discriminator so that each training iteration, half the information through the discriminator gets is propagated through the generator in the form of training.

There is a very similar question (with answer) here

The answer to your question can be found in [1, sec. 4.4]. Briefly, the GAN optimization problem is a mini-max game, and early on the proposition of GANs, the authors had the idea that one should balance the power/optimization of these two actors. Here is a quote from the said reference:

The author’s present belief is that GANs work by estimating the ratio of
the data density and model density. This ratio is estimated correctly only when the discriminator is optimal, so it is fine for the discriminator to overpower the generator.

It appears that, despite their initial claims in [2], the authors do not believe that this is the case anymore. In [1], Goodfellow explains,

The idea that the discriminator should always be optimal in order to best
estimate the ratio would suggest training the discriminator for k > 1 steps
every time the generator is trained for one step. In practice, this does not
usually result in a clear improvement.

To summarize: when the discriminator is near optimality, the ratio between densities ($p_{data}$ and $p_{model}$) is more accurate, generating better gradients to update $p_{model}$ a.k.a. the generator. If you choose to do the other way around, you would be updating the generator successively with poor gradients.
References
[1] Goodfellow, I. (2016). Nips 2016 tutorial: Generative adversarial networks. arXiv preprint arXiv:1701.00160.
[2] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., ... & Bengio, Y. (2014). Generative adversarial nets. Advances in neural information processing systems, 27.


text:
How to find the order in which DFS algorithm will inspect the nodes?

I have been taking Artificial Intelligence course in College. I came upon this problem. Now here I have to find the order in which DFS algorithm inspects the nodes and what is the path from Start to Goal State. And wherever this contention between more than one nodes, left one is to be chosen.
So according to DFS algorithm, I would add {A,B,C} in stack.
{D,E,B,C} -> {K,E,B,C} -> {M,E,B,C} -> {O,E,B,C} -> {F,G,E,B,C}.
Now I cannot understand what to do further, should I go to B and then backtrack again with B already being in the stack or do something different. And the path would be S -> B -> G or else S -> A -> D -> K -> M -> O -> G.
How to decide what way to go?

You would keep track of a list of states that you have already visited. As you progress through the graph, you add the node you choose to that list.
By the time you are at F, B is not yet on the visited list (though it is on the stack), so you would go to B. From there you would got to S, which you have already visited, so then choose the next possibility, G, which is your goal.
There is an implementation detail that would change the behaviour: you could also put nodes from the stack into your list (because arguably you would visit B much earlier if you didn't take the detour via O), in which case you would backtrack from F to O, and then move on to G.
I don't think this is fixed in DFS; you're still following the principle of exploring the current path until the end. It's similar in status to "always choose the left node to proceed": an implementation detail (though important, as it changes the outcome).

You could mention graph search, which is the term that Norvig and Russell (and most AI people) use to refer to the general state-space search that keeps track of what has already been visited.


text:
What are the advantages of the Kullback-Leibler over the MSE/RMSE?
I've recently encountered different articles that are recommending to use the KL divergence instead of the MSE/RMSE (as the loss function), when trying to learn a probability distribution, but none of the articles are giving a clear reasoning why the first is better than the others.
Could anyone give me a strong argument why the KL divergence is suitable for this?

In the context of Variational Inference (VI): the KL allows you to move from the unknown posterior $p(z \mid x)$, to the known joint $p(z,x)=p(x|z)p(z)$ and optimize only the ELBO. You cannot do this with L2.
$p(z|x)$ is the desired posterior, of which you cannot calculate the evidence (i.e., using Bayes formula we can set: $p(z|x) = \frac{p(x|z)p(z)}{\int_z p(x|z)p(z)dz}$, and you can't calculate the integral in the denominator [also denoted by $p(x)$] due to it's intractability).
Now suppose $q$ is a Variational distribution (e.g. a family of Gaussians which you can control), VI tries to approximates $p(z|x)$ by $q$ by minimizing their KL divergence.
$$KL(q(z)||p(z|x)) = \int_z q(z) \log \frac{q(z)}{p(z|x)}dz = \mathbb E_q[\log q(z)]-\mathbb E_q[\log \frac{p(x|z)p(z)}{p(x)}] =$$
$$ -\mathbb E_q[\log p(x|z)p(z)] + \mathbb E_q[\log q(z)] + \mathbb E_q[\log p(x)] = -ELBO(q) + \log p(x)
$$
Since you're only optimizing $q$ (it's the only thing you can control), you can discard the unknown and difficult to compute normalizing constant $p(x)$.
If you would use the (squared) L2 norm you would get:
$$\int_z [q(z)-p(z|x)]^2dz = \int_z [q^2(z)-2q(z)p(z|x)+p^2(z|x)]dz $$
While the 3rd term doesn't depend on q, the 2nd term does, and it also requires $p(x)$ to compute.

Maybe you should clarify that $q$ is the variational distribution (e.g. a family of Gaussians), i.e. the distribution that you're using to approximate the unknown one that you want to learn. From this answer, it's not fully clear why the KL divergence is really better than the L2 norm, which you seem to have defined as an expectation with respect to $z$. I recommend that you edit your post in order to clarify it.

I added a crash course in VI, but some knowledge is needed of VI in order to understand this. As implied by the "In the context of Variational Inference" that starts the answer.

KL-divergence is a measure on probability distributions. It essentially captures the information loss between ground truth distribution and predicted.
L2-norm/MSE/RMSE doesn't do well with probabilities, because of the power operations involved in the calculation of loss. Probabilities, being fractions under 1, are significantly affected by any power operations (square or root), and considering we are calculating the squares of differences of probabilities, the values that are summed are abnormally small, essentially barely learning anything as the random initialization itself starts with an abnormally small loss, almost always staying constant.
L1 norm, on the other hand, does not have any power operations, making it relatively acceptable.
Loss functions, such as Kullback-Leibler-divergence or Jensen-Shannon-Divergence, are preferred for probability distributions because of the statistical meaning they hold. KL-Divergence, as mentioned before, is a statistical measure of information loss between distributions, or, in other words, assuming $Q$ is the ground truth distribution, KL-Divergence is a measure of how much $P$ deviates from $Q$. Also, considering probability distributions, convergence is much stronger in measures of Information Loss such KL-Divergence.
More clarity on the motivation behind Kullback-Leibler can be read here.


text:
Can off-policy algorithms benefit from the parallelization?
On-policy algorithms, such as A2C, A3C and PPO, leverage massive parallelization to achieve state of the art results. However, I’ve never come across parallelization efforts when it comes to the off-policy algorithms, like SAC and TD3.
Is it because the replay memory is kind of a substitute for the parallel data sampling in the on-policy algorithms? Can off-policy algorithms benefit from the parallelization?
Ray RlLib says the following for SAC and TD3 regarding the number of workers for collecting samples

This only makes sense to increase if your environment is particularly slow to sample

From the point of view of someone developing an in-house DRL lib and working on extremely CPU-intensive environments (usually large finite element-based simulations that can require several hours to unroll a single episode), yes it can.
What I do is that I have $n$ parallel environments unrolling simultaneously and collecting transitions, and I perform $n$ updates after each parallel step (i.e. all $n$ parallel envs have collected one transition), in order to keep a 1/1 ratio between stepping and training. I believe it is okay from a theoretical point of view, although I'm not 100% certain. From a practical point of view, it has worked well so far.
The difference between offline and online collection is that the sampling/training procedures are very different, as in online methods you usually sample a large buffer of samples before performing several epochs of training. Yet, I am curious about what would happen if one was to perform the same for, say, TD3: sample a large buffer of $m$ transitions, store them in the replay buffer, and then perform $m$ updates to keep the 1/1 ratio.

Have you ever tried to adjust the batch size by multiplying the original batch size for one environment by the number of parallel environments instead in order to maintain 1/1 ratio between stepping and training? I’ve seen one such GitHub implementation of parallelization of off policy algorithms. Obviously this may lead to relatively huge batch sizes; however Open AI is using batch size of tens of millions as reported here https://openai.com/blog/science-of-ai/. But I believe they are applying this in on policy algorithms (since DOTA 2 bots were trained with PPO).

@Mika thanks for accepting the answer. I'm not certain to understand the follow-up question.
In sequential, I perform one gradient step at every time-step by sampling one mini-batch of size $b$ from my replay buffer.
In parallel, I still use a mini-batch of size $b$, but I collect $n$ samples in parallel, so I perform $n$ gradient steps at every time-step by sampling $n$ different mini-batches from my replay buffer.
What you suggest is to perform $n$ gradient steps, but using mini-batches of size $nb$ ?


text:
Where does the "rectified" in ReLU come from?
ReLU stands for Rectified Linear Unit. Linear Unit, I understand, since the function is piecewise linear. But what does rectified mean?
I looked up the definition and it said:

denoting an electric current that has been converted from alternating current to direct current

But I don't quite see how this can connect to ReLU function. While it does have a bend at $x = 0$ indicating a change in direction, not sure how I can work this into the idea of current directions.

I always linked it to engineering, a rectifier converts alternating current into direct current. As ReLU gets rid of the negative side of the linear function I think its fitting. Not sure though if thats also what the authors had in mind.

I think it is by analogy with an electrical rectifier. A rectifier allows current to flow in one direction but blocks current in the other direction. Or if you prefer it allows voltage in one polarity but won't allow a voltage in the opposite polarity. Which is also what a ReLU does.


text:
Is there a standardized method to train a reinforcement learning NN by demonstration?
I'm less familiar with reinforcement learning compared to other neural network learning approaches, so I'm unaware of anything exactly like what I want for an approach. I'm wondering if there are any ways to train a Deep-Q neural network on, say, OpenAI Gym, where the model is given a recorded demonstration to learn from. That is, I'd like to do the following (with a Mario NES example):

Play the first level of Mario and somehow record this (maybe as an input sequence?)
Run the model on the level a few times until it passes
Send the model to the next level, let it train for as long as possible unless it continues to fail -- if so, I'll play the level and then let it train again
Repeat

Are there any approaches similar to this that currently exist? And, would this be infeasible because the model may fail to generalize? I'd like to accomplish this with much more complicated games, but if I could use this approach to save on defining endless amounts of reward/penalty policies and ROM hacking to find the memory address of everything I want the model to use, it would be extremely helpful.

Yes, this is known as imitation learning, which can be divided into

inverse RL (i.e. learn a reward function from demonstrations, then apply RL), and
behaviour cloning (supervised learning applied to RL).

I don't know the state-of-the-art (I am not an expert in this topic) or whether IL is a good approach in your case, but you can check e.g. the pre-print survey Imitation Learning: Progress, Taxonomies and Opportunities (2021) by Boyuan Zheng et al., which seems to nicely describe the topic.


text:
Does LSTM provide any unique value or advantages compared to other algorithms, including "vanilla" RNN?
I have heard a lot of hype around LSTM for all kinds of time-series based applications including NLP. Despite this, I haven't seen many (if any) applications of LSTM where LSTM performs uniquely well compared to other type of deep learning, including more vanilla RNN.
Are there any examples where LSTM does significantly better on a particular task, compared to other modern algorithms and architectures?

LSTMs were the state-of-the-art (SOTA) in many cases (e.g. machine translation) until transformers came along - now I don't really know the SOTA or where LSTMs still perform better than e.g. transformers. LSTMs were introduced to solve the vanishing and exploding gradient problems. Even the LSTM paper tells you that

In comparisons with RTRL, BPTT, Recurrent Cascade-Correlation,
Elman nets, and Neural Sequence Chunking, LSTM leads to many more successful runs, and learns much faster. LSTM also solves complex, articial long time lag tasks that have never been solved by previous recurrent network algorithms.

For a specific case where LSTM achieved SOTA (if I remember correctly), you can check the neural machine translation paper. Google used LSTMs for some time in Google Translate. See this paper for more details.

The original LSTM paper is quite old, which is why I was a bit skeptical about their conclusions holding up against more modern architectures (transformers aside). Maybe things haven't changed as much as I thought? From what I can tell, the neural machine translation paper doesn't use LSTMs . I appreciate the Google Translate reference, quite interesting! I was hoping for a more head-to-head comparison, which the Google translate paper appears to do. Thank you!

GRUs can be viewed as a variation of LSTMs. See also this repo.

RNN's/LSTM's work better on smaller datasets compared to transformers, assuming neither are pretrained.
RNN's perform worse than transformers in most tasks with enough data due to RNN's having a higher model bias.
Recall that in many ML classes, you are taught that with enough samples, models with higher bias (and lower complexity) will perform worse than models with lower bias (and higher complexity).
RNN's take in a sequence of elements $x_1,x_2,...,x_n$ and assume a markov property. this markov property is used to simplify / reduce the complexity of representations for the input. That and the fact that the RNN only uses one representation for the sequence of tokens are examples of simplifying/introducing a bias into the model architecture.
Transformers create a representation for each token, and do not assume that markov property.
This means it has lower bias than RNN's and can scale better with more data.


text:
How to tackle the human error made in labeling datasets for classification tasks like facial expression recognition?
I am working on the Facial Expression Recognition Task. One of the most challenging tasks that I faced was human error in labeling the datasets (ex: let's say FER2013).
Are there anyways to Handle incorrect labeling of datasets in the classification tasks? Will using clustering methods and treating it as an unsupervised Learning problem (just a thought not have a clear idea) solve this kind of issue? or are any other methods available without treating it as an unsupervised task?

In general the only way to deal with this is by quantifying these labeling mistakes in the output of the model, since the model will learn for them. And in many cases these are not really mistakes, but they encode the ambiguity of the task, particularly in interpreting facial emotions.
For this you need to use a model that can estimate Aleatoric uncertainty (this is data uncertainty). For classification you can use the cross-entropy loss combined with the sampling softmax function, which replaces a standard Dense layer with one that computes a Gaussian distribution for the softmax logits (so it predicts mean logit and variance logits).
This method is described in the paper "What uncertainties do we need in bayesian deep learning for computer vision?" (link), where the authors can also disentangle aleatoric from epistemic uncertainty.
I have run experiments exactly on the facial emotion recognition task (using the FER+ dataset), which you can find in my paper "A Deeper Look into Aleatoric and Epistemic Uncertainty Disentanglement" (link). In this paper I show multiple uncertainty quantification methods and their ability to produce disentangled probabilities. If you only need aleatoric uncertainty, then only the sampling softmax is needed.
I even have an implementation of this layer in my keras-uncertainty library.

Quantifying uncertainty is useful, but I am not sure how it directly answers the OP's question. Could you perhaps add a paragraph about how the OP can use the uncertainty measurement to cope with incorrect ground truths in the data set.

@NeilSlater yes, just did


text:
How does a VAE sample the coding layer?
I am reading the book, Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems and came across the following paragraph -

You can recognize the basic structure of all autoencoders, with an encoder followed by a decoder (in this example, they both have two hidden layers), but there is a twist: instead of directly producing a coding for a given input, the encoder produces a mean coding μ and a standard deviation σ. The actual coding is then sampled randomly from a Gaussian distribution with mean μ and standard deviation σ. After that the decoder decodes the sampled coding normally.

I do not understand how sampling is conducted over here and have the following question -
I understand it is trivial to sample from a univariate Gaussian distribution. However, for a dataset with n features, we won't be able to use a univariate Gaussian. How does that work?

The number of features $n$ of an input $\mathbf{x} \in \mathbb{R}^n$ to the VAE may be related to the dimension $m$ of the latent vector $\mathbf{z} \in \mathbb{R}^m \sim \mathcal{N}(\mu, \sigma^2)$, but the formulation of the VAE does not go far enough to tell us how these parameters are related. The only assumption in the VAE is that $\mathbf{x}$ depends on $\mathbf{z}$. Here's the diagram (from the VAE paper) that illustrates the graphical model they consider

In practice, we may not use a univariate Gaussian, but most likely we use a multivariate Gaussian, i.e. $m > 1$, but $m$ is really a hyper-parameter, so it could also be $1$. Moreover, note that we don't sample $\mathbf{z}$ directly from $\mathcal{N}$, if we use the re-parametrization trick. See this implementation, where $\epsilon$ is actually sampled. $\epsilon$ is just a sample from a standard normal (usually), which we use (along with the learnable mean and variance) to construct $\mathbf{z}$.


text:
Why and when do we need to normalize weights in Reinforcement Learning?
I recently came across this SO question, wherein the poster was asked to normalize their weights while using a function approximator with SARSA. I don't remember having to normalize any weights while using a DQN and so therefore would like to when and why is this method needed.

The kind of divergence that the other question experienced is a common problem with deep RL and temporal difference methods (Q-learning, SARSA, or any Actor Critic).
The weight normalisation would not be needed if the asker of the question you linked had used some kind of protection against divergence, such as using a separate target network. The normalisation may have worked for them as an alternative.
DQN already has experience replay and separate target network to help with this. If you do get runaway feedback causing diverging values, you already have some hyperparameters you can change to try and fix it - size of minibatch, size of replay table, number of updates between copying to target network.
So, in short, you don't really need weight normalisation in Deep RL. You might want to use it for regularisation though.


text:
Does rule-based image processing count as AI?
I'm quite new to the field of AI and I currently find it hard to precisely inclose the broad field of AI. Especially the aspect of image processing isn't quite clear to me.
So far I've done rule-based approaches with image processing, e.g. things like template matching or blob detection combined with fixed thresholds and If-then-else-statements which define different classes.
Do rule-based systems like this one belong to the field of AI or is this still part of simple image (pre-)processing and therefore no "artificial intelligence"?

Whether template matching algorithms (and other image processing or computer vision techniques) are part of the AI field or not might be a little bit subjective. I don't know if there is a consensus on this topic. It probably depends on whether people consider these approaches intelligent or useful to create intelligent agents/systems, and that depends on the definition of intelligence. However, I think that most would agree that CV is fundamental to create AGI (e.g. most animals have a vision system).
Having said that, image processing techniques, including template matching, are still studied in the context of AI. For example, during my MSc in AI, I had a Computer Vision course, where we studied template matching, in addition to other more modern techniques (e.g. CNNs). If you look at a standard textbook on AI, e.g. AIMA, you will see that it contains sections on CV and image processing (chapter 24, 3rd edition). This book, which covers the history of AI, by Nils J. Nilsson (co-author of STRIPS and A*), mentions a few image processing techniques. So, CV and image processing have traditionally been studied in AI. Moreover, although one may think that IP algorithms are "not so intelligent", they are useful for CV, which is fundamental to create general AI systems. So, I think it's reasonable to consider both CV and IP techniques (whether they are rule-based or not) part of the AI field.


text:
Do I need to normalize all state-space variables? If so, how?
I am playing around with a DRL agent in a stock-trading environment.
I have normalized all the external input data (the features that my agent will use). However, what about characteristics that don't come from the environment?
For example, I have included things like "current account balance" and "current unrealized gain" in my observation space (as I believe it's useful). However, I don't know how I could normalize these values, given that they are dependent on what actions the agent took, which changes every time etc.
Any feedback or advice is appreciated.
Will it be detrimental if I don't normalize these values (as long as they're reasonably within the orders of magnitude of my other normalized variables)?
I guess a simple example would be like if a robot was being trained to pick up balls, and one of the observations was "current number of balls picked up", how would you normalize that value, given that it's just a count that could technically go to infinity?

The way I've seen most codes treat the state normalization is that they simply take a running mean and standard deviation for each dimension of the state space. As you point out, this normalization will be dependent on the actions the agent takes; this is not unique to your problem.
As for your concern of the state observations going to infinity, this will not happen. Keeping with your example, the number of balls going to infinity would require the number of timesteps to also go to infinity. In practice there will have to be some finite length of the episodes, so this won't be an issue. And note, once you start a new episode, the number of balls goes back to 0, so it's not like the normalization amount can go to infinity either.

Thank you for your response. I had considered a running mean/StdD, but my concern is that unless I have really long episodes (which I don't), there isn't enough time for these values to reach any kind of stability. What do you think?

The length of the episode should not be important here. After you have hundreds of thousands (or millions) of transitions, the change to the mean/stdev from a single episode (which consists of perhaps a few hundred or ~thousand transitions) is fairly small.

So, you're saying I would normalize across episodes? Not just within 1 episode? So, episode 1 ends, but the state variables of episode 2 are normalized by the mean/SD of episode 1 in addition to the timesteps taken so far in episode 2?

yes. the normalization amount should be the empirical mean/stdev, computed over EVERY observed transition.

I'll start with the literal question in the title:

Do I need to normalize all state-space variables?

You don't strictly need to in theory. It's often really useful, or sometimes borderline necessary, in practice though. It can be useful just for faster learning, it can be important for numeric stability, or just make it much easier to tune hyperparameters (like a learning rate).
If it's not clear exactly how you could go about doing the normalisation, or if you have multiple different possible solutions, ideally you'd be able to empirically validate what works best for your case by trying them all and running experiments. Maybe this will simply lead to the conclusion that not normalising at all happens to work best for your particular setup.

I am not aware of any 100% robust, standard normalisation or standardisation approaches that work all the time when you have no prior knowledge about upper/lower bounds on the values that your variables might take (or if you do have prior knowledge that there exist no such bounds). I imagine that the online tracking of mean and standard deviation, as suggested by Taw, could work well in practice, especially if your state variables don't explode too wildly at deployment time and suddenly take very different values from what you observed at training time. On the other hand, if such extreme differences do suddenly occur, the approach won't be robust against that. Similarly, you could also track the minimum and maximum observed values, and normalise to a $[0, 1]$ range based on that (but in practice you might exceed that range again if after training you observe more extreme values than you did during training).

I also really like the general idea from Lee Reeves' answer about trying to model your state variables in a different way, such that they still encode the information necessary for your agent to work well, but in a manner that has less of the annoying numeric issues. This requires domain knowledge of your particular problem though.

a simple example would be like if a robot was being trained to pick up balls, and one of the observations was "current number of balls picked up"

In this simple example, why is that observation a relevant observation? My domain knowledge says it probably isn't a relevant observation at all and should be removed. If the goal continues to be to pick up as many balls as possible, why should the robot care about how many balls it already picked up before?

If it shouldn't care, just remove the feature instead of trying to normalise it.
Maybe it could be relevant if the robot also knows how many balls were lying on the floor at the start; then, knowing how many were already picked up tells you how many are still on the floor. But the same information could be represented as a value in $[0, 1]$ giving you the proportion of balls already picked up... that could work better then!

For example, I have included things like "current account balance" and "current unrealized gain" in my observation space (as I believe it's useful).

For this example, it'd be good to try to think of why you believe such variables would be useful, and then try to again find other ways to encode it (if possible). $\frac{\text{unrealised gains}}{\text{account balance}}$ as suggested by Lee Reeves could be a good way. A series of binary variables that provide your agent with information about the orders of magnitude of these values could be very useful too, for example:

$b_1$: is the account balance above $10$?
$b_2$: is the account balance above $100$?
$b_3$: is the account balance above $100$?
$b_4$: is the account balance above $1000$?
$b_5$: is the account balance above $10,000$?
$b_6$: is the account balance above $100,000$?

I suspect you wouldn't have to keep going too far like that, probably beyond a certain point you stop caring and your agent just knows that it's very rich and that's probably all it needs to know?

Thank you very much for your in-depth response. I will try all of these approaches. It's quite hard choosing an answer to accept here, as every answer adds an additional layer of perspective! After a lot of thought, I'm going to accept Taw's answer as I think it most directly answers the question in the way that I originally envisioned/expected it.

I don't know why the so-called AI algorithms require humans to consider such complex problems, this is not artificial intelligence, it is just artificial programming.

It's likely to train as long as they're reasonably within the orders of magnitude of other normalized variables. The network can adjust for that.
But it might cause problems later, if the values move outside the interval they were trained on. For example, it might work perfectly with a 10k account balance then unexpectedly fail with a 1m balance, because NNs do not always extrapolate well Xu et al, 2020.
You could use "current unrealized gain" / "current account balance" instead, so it will always be in the interval [0,1].

Thank you for your response. So, are you saying though, that there is no way to normalize this kind of information? Only clever attempts at keeping it within a reasonable range?

Could you clarify precisely what you mean by normalization? It could mean scale to the range [0,1]; or scale and shift to mean 0, variance 1; or other things.

By normalization, I mean applying any transformation (including ones you mentioned) so that the scale/magnitude of these inputs does not vary hugely from all the other inputs (which, in my case, have been normalized by subtracting the mean and dividing by the standard deviation).

I'm wondering if there is any standard method/practice of how to achieve that end with the kinds of variables that are generated directly as a result of actions the agent has taken. Any method/practice that is more standard or rigorous than a heuristic of just generally being clever to try to make sure the numbers don't get too big or small. I hope this clarifies my question.

From a rigorous perspective, consider that most RL algorithms assume the environment is a Markov decision process. That means the next state is conditionally independent of all previous states, given the current state and chosen action. Including a cumulative variable like account balance or balls picked up breaks that assumption.

I'm not familiar with the theory or practice of RL in non-Markovian environments. If you'd like to investigate further, I did see a fairly recent paper that might be a good starting point: Reinforcement Learning with Non-Markovian Rewards

@LeeReeves I don't agree with the comment that including a cumul. variable like account balance breaks the Markov assumption. If you include it in the state representation, it's right there, so Markov assumption holds. Of course it does bring the practical issue that your state space becomes really big (possibly infinite), which is bad for tabular approaches. And the issue of difficulties with normalisation (as is the topic of the question here), which is bad/challenging/annoying for RL with function approximation.

Thank you, you're right of course. I should be careful about answering right after waking up. Are you familiar with any standard techniques for normalizing unbounded inputs? That question is in my head now too.

@DennisSoemers I agree with your assessment about maintaining the Markov assumption. Do you agree with Taw's answer? Are there other approaches?


text:
What kind of NN I need to find ideal ranges and correlation between them?
I’m new to NN and I’m trying to collect material and study. I’m getting through a general high level book, but I’m still struggling understanding what kind of NN I should go ‘deeper into’ for what is my idea.
I have some equipment that I can control from my computer. Each part of this equipment has a specific ID. Each ID accepts a value within a specific range, always the same for that specific ID: so for example ID1 might accept just 0 to 1, ID2 accepts 0-255, ID3 accepts from 0 to 3 etc. (the max number of ID/controllers is 255).
Now, the thing is that some ranges are better than others. So there’s a desirable outcome, that usually means some ranges work better than others for each ID/controller, and it also means that there are probably some correlations (for example the range 30 to 90 for ID1 usually works best when ID5 is in the range of 20 to 50).
The idea would be to have training data from random generation: so I generate a random value (within the full range of each ID) for the whole set of parameters, I decide if the outcome is good (0/1, good or bad), and then I generate once again random values for all the parameters, once again I label it if good or bad, etc. It all works with all parameters together, not one by one, so it’s like a snapshot for the status of the machine (all its IDs/parameters).
And to test the AI, I would have two choices: either let it generate ‘desired’ values directly, or have the AI after a quick random generator that let pass only what it considers as good.
My question is: what kind of NN would be recommended for this application? What would I need to study more? I’m proficient in Python so I would use that.
Considering the nature of the data (around 100 pairs of ID:value where value is within range that goes max from 0 to 255), I believe this should be an almost textbook example for using a NN. But maybe I’m wrong?
Any suggestion, link, direction would be much appreciated :-)

What you described seems like a pretty standard binary classification problem. There are many good algorithms, that are much simpler and more interpretable than NNs. I don't see why you would straight jump into NNs without first trying and fine-tuning e.g. Support Vector Machines (SVM), Random Forest(s), or some gradient boosting classifier like XGBoost. I suggest you have a look at the sklearn package in Python for the implementation of these.


text:
How many training steps does it usually take to train an RL model?
This is my model average rewards as follow image.
How to tell if it is undertrained or not convergent? How many training steps does it usually take to train an RL model?
And I'm using PPO to train.

This is not possible to know in advance precisely, only approximately, but it also strongly depends on the environment, hyperparameters and algorithm. For hard environments, e.g. the ones learning from pixels such as the Atari games, you can easily expect 1-10M steps.
You can get a feeling by looking at many solved environments, for instance here: https://github.com/Rafael1s/Deep-Reinforcement-Learning-Algorithms

It depends on the the problem you're applying PPO to. To get an idea, you can have a look at the CleanRL benchmarks, there are a few of them where they use PPO: https://wandb.ai/openrlbenchmark/openrlbenchmark/reportlist.

As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.


text:
What is the difference between the US and global edition of the AIMA book by Russell and Norvig?
The book Artificial Intelligence: A Modern Approach by Russell and Norvig has two editions: global and the US. It looks like these two are generally the same, but have some differences in the order of the chapters and in the context, is this correct?

they only add one chapter in the Global Edition adding the chapter 20 Knowledge in Learning, in the V Machine Learning Section and start change the orders of the chapters in the Section IV Uncertain knowledge and reasoning. I don't have those editions yet to confirm if there are any differences between the sub-chapters, if anyone have it please answer about the difference. for the moment  Global Edition has an extra chapter than US edition.

The difference can be found in the following link more clearly
US vs Global

Thanks but I have seen the site before. Now I am familiar with the difference.

While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference.  Link-only answers can become invalid if the linked page changes. - From Review

I check again with the subchapters of Artificial Intelligence: A Modern Approach, 4th Global ed / US ed from this website the pdf subchapters reference of Global Edition and US Edition. I can confirm you the difference between Global US edition is this subchapter:
20 Knowledge in Learning 739
    20.1 A Logical Formulation of Learning 739
    20.2 Knowledge in Learning 747
    20.3 Explanation-Based Learning 750
    20.4 Learning Using Relevance Information 754
    20.5 Inductive Logic Programming 758
Summary 767
Bibliographical and Historical Notes 768

So Global Edition has more content than US Edition.

3 editions

US

Global
has one more chapter than US ("Knowledge in Learning")

Global for Indian sub-continent
much cheaper
same as Global, except
a) black & white
b) missing "Probabilistic Programming" chapter (28 pages)
c) index page for any entry after the missing chapter (=after page 669) refers to a page 28 pages after actual page (very annoying)


text:
How does Bishop derive $\ln p\left(\mathbf{x} \mid \mu, \sigma^{2}\right)$, when $p$ is a Gaussian?
I am now reading the Bishop Machine Learning Book and going through every single equation.
We know that in the case of a single real-valued variable $x$, the Gaussian distribution is defined by
$$\mathcal{N}\left(x \mid \mu, \sigma^{2}\right)=\frac{1}{\left(2 \pi \sigma^{2}\right)^{1 / 2}} \exp \left\{-\frac{1}{2 \sigma^{2}}(x-\mu)^{2}\right\}$$
Since the dataset $\mathbf{X}$ is i.i.d., we can therefore write the probability of the dataset, given $\mu \text { and } \sigma^{2}$, in the form
$$p\left(\mathbf{x} \mid \mu, \sigma^{2}\right)=\prod_{n=1}^{N} \mathcal{N}\left(x_{n} \mid \mu, \sigma^{2}\right)$$
We will resolve the numerical underflow by computing the sum of the log probabilities, therefore from the above two equations we have the following log-likelihood function,
$$\ln p\left(\mathbf{x} \mid \mu, \sigma^{2}\right)=-\frac{1}{2 \sigma^{2}} \sum_{n=1}^{N}\left(x_{n}-\mu\right)^{2}-\frac{N}{2} \ln \sigma^{2}-\frac{N}{2} \ln (2 \pi)$$
I am not quite sure how did they derive this in the book. Thanks a lot and have a good day.
You can refer to page 27 equation 1.54 for the detail.

Please, next time, provide a more descriptive title!.

This is not so difficult (just a bit verbose if you do all steps). Just replace $\mathcal{N}\left(x_{n} \mid \mu, \sigma^{2}\right)$ with $\frac{1}{\left(2 \pi \sigma^{2}\right)^{1 / 2}} \exp \left\{-\frac{1}{2 \sigma^{2}}(x-\mu)^{2}\right\}$ and apply the properties of logs (specifically, log of multiplication, log of exponential and log of a fraction). Try first before looking at my solution (otherwise you don't learn anything)!

 \begin{align} p\left(\mathbf{x} \mid \mu, \sigma^{2}\right) &= \prod_{n=1}^{N} \mathcal{N}\left(x_{n} \mid \mu, \sigma^{2}\right)\\ &= \prod_{n=1}^{N}\frac{1}{\left(2 \pi \sigma^{2}\right)^{1 / 2}} \exp \left\{-\frac{1}{2 \sigma^{2}}(x-\mu)^{2}\right\}  \iff \\ \log p\left(\mathbf{x} \mid \mu, \sigma^{2}\right) &= \log \left( \prod_{n=1}^{N}\frac{1}{\left(2 \pi \sigma^{2}\right)^{1 / 2}} \exp \left\{-\frac{1}{2 \sigma^{2}}(x-\mu)^{2}\right\} \right) \\ &= \sum_{n=1}^{N} \log \left(\frac{1}{\left(2 \pi \sigma^{2}\right)^{1 / 2}} \exp \left\{-\frac{1}{2 \sigma^{2}}(x-\mu)^{2}\right\} \right) \\ &= \sum_{n=1}^{N} \left( \log \left(\frac{1}{\left(2 \pi \sigma^{2}\right)^{1 / 2}} \right) + \log \left( \exp \left\{-\frac{1}{2 \sigma^{2}}(x-\mu)^{2}\right\} \right) \right) \\ &= \sum_{n=1}^{N} \left( - \frac{1}{2}\log \left(2 \pi \sigma^{2}\right)  -\frac{1}{2 \sigma^{2}}(x-\mu)^{2}  \right) \\ &= - \frac{1}{2} \sum_{n=1}^{N} \log \left(2 \pi \sigma^{2}\right) - \frac{1}{2 \sigma^{2}} \sum_{n=1}^{N}(x-\mu)^{2} \\ &= - \frac{1}{2} \sum_{n=1}^{N} \left( \log 2 \pi + \log \sigma^{2}\right) - \frac{1}{2 \sigma^{2}} \sum_{n=1}^{N}(x-\mu)^{2} \\ &= - \frac{1}{2} \sum_{n=1}^{N} \log 2 \pi  - \frac{1}{2} \sum_{n=1}^{N} \log \sigma^{2}- \frac{1}{2 \sigma^{2}} \sum_{n=1}^{N}(x-\mu)^{2} \end{align}


text:
Should PPO always converge toward the global optimum?
I'm trying to "solve" the OpenAI gym environment "Humanoid-v3"  using PPO. I got it to work to some degree (The NN is learning a policy and perfecting it. Average reward of about 5.5k). However, the learned policies do not yet resemble the human stride (like in the PPO blog post), which brought up a question.
Should the algorithm always converge toward the global optimum (given good hyperparameters)? Or is a good convergence somewhat luck-based and you may need multiple training processes?

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

Please, ask only one question. If you have multiple questions (even if somehow related), it's a good idea to split them into multiple posts.

In the world of theory, Convergence Proof for Actor-Critic Methods Applied to PPO and RUDDER claims to show that PPO eventually converges to a local (but not necessarily global) optimum.
In practice, good convergence is somewhat luck-based. Among other problems, the number of training steps is finite, the optimal solution might not be explored, and the model might approach the optimal solution then suffer catastrophic interference/forgetting.
And it's even possible that the human gait isn't the optimal solution of that task. OpenAI has been accused of cherry-picking or misrepresenting results for good PR at times, and a human gait gives better publicity than a weird, but higher scoring, solution.

Even if you were reaching the global optimum for the PPO loss, it wouldn't mean that the learnt policy would resemble the human stride. Nothing in the loss enforces that.
More generally in RL, the policies we use as humans to "solve" a task aren't necessarily the optimal ones (in the sense of reaching the highest possible reward, which is what RL methods aim to do), see for example the previously unseen strategies which AlphaGo employs to beat humans at Go.
That's even more the case for environments which use an imperfect physics simulator, e.g. the humanoid here can't dislocate its member and can do very jerky movements: maybe these jerky movements achieve a higher reward than a human gait.