text:
Why is this vacuum cleaner agent rational?
This is the vacuum cleaner example of the book "Artificial intelligence: A Modern Approach" (4th edition).

Consider the simple vacuum-cleaner agent that cleans a square if it is dirty and moves to the other square if not; this is the agent function tabulated as follows.
Percept sequence                      Action
[A,Clean]                             Right
[A,Dirty]                             Suck
[B,Clean]                             Left
[B,Dirty]                             Suck
[A,Clean], [A,Clean]                  Right
[A,Clean], [A,Dirty]                  Suck
.                                      .
.                                      .
[A,Clean], [A,Clean], [A,Clean]       Right
[A,Clean], [A,Clean], [A,Dirty]       Suck
.                                      .
.
                               .

The characteristics of environment and performance function are as follow:

The performance measure awards one point for each clean square at each time step, over a "lifetime" of 1000 time steps.

The "geography" of the environment is known a priori (the above figure) but the dirt distribution and the initial location of the agent are not. Clean squares stay clean and sucking cleans the current square. The Right and Left actions move the agent one square except when this would take the agent outside the environment, in which case the agent remains where it is.

The only available actions are Right, Left, and Suck.

The agent correctly perceives its location and whether that location contains dirt.

In the book, it is stated that under these circumstances the agent is indeed rational. But I do not understand such percept sequence that consists of multiple [A, clean] percepts, e.g. {[A, clean], [A, clean]}. In my opinion, after first [A, clean], the agent must be gone to the right square; So, the sequence {[A, clean], [A, clean]} will never be perceived.
In other words, the second perception of [A, clean] is the consequence of acting left or suck action after perceiving the first [A, clean]. Therefore, we can conclude the agent is not rational.
Please, help me to understand it.

You are correct that the robot should never perceive two [A, Clean] events in a row, as it should have moved when perceiving the first one.
However, this only applies to a perfect agent, which always successfully executes all its actions. What if the dog walks past and blocks the exit to room B? Then the Right action fails, and the agent is still in A. But now, it might only listen to [B, ?] events, because obviously it must now be in room B, since it executed a Right command.
Having a simple tabulated agent function like this is clearly not adequate for real world problems, as where do you stop? How many [A, Clean] events do you want to list? And, if you start again, why do you have lists of perceptions in the first place? The initial [A, Clean] in the list should suffice, as it would still attempt to move right if it processes perceptions with no internal state/memory.
But bearing in mind that physical actions can fail in the real world, the agent is still rational, it's just overly cautious: it doesn't try to suck when the location is clean, or move away when it's dirty. It just wants to make sure it's really clean before moving away.

Thank you for the answer. But in the assumptions we have: "The Right and Left actions move the agent one square except when this would take the agent outside the environment, in which case the agent remains where it is." So it seems that the closure can not occur. What is your opinion?

The table represents the agent function which takes as input the perception sequence
In the Vacuum example the rational decision does not depend on the perception sequence, or rather only on the last entry of the perception sequence.
Think of a more complex game where the optimal decision changes after you have perceived something in the past. An example could be you are standing in front of a door and need a key.
* [A:door] -> Walk right
* ...
* [A:door] ... [found key] ... [A:door] -> Open door

Such a table is intended to be used as a look-up table for the optimal decision for every situation including the past.
Depending on the setting in could make sense to limit the table to a maximum length of a perception sequence; for the vacuum world to length one.

The answer answer can be found in the later part of the text which you did not read:
"One can see easily that the same agent would be irrational under different circumstances"
For example, once all the dirt is cleaned up, the agent will oscillate needlessly back and forth; if the performance measure includes a penalty of one point for each movement left or right, the agent will fare poorly. A better agent for this case would do nothing once it is sure that all the squares are clean. If clean squares can become dirty again, the agent should occasionally check and re-clean them if needed. If the geography of the environment is unknown,
the agent will need to explore it rather than stick to squares A"


text:
Can I use the transformers for the prediction of historical data?
Can I use the transformers for the prediction of wind power with the historical data?
Dataset
Datetime, Ambient temperature (Degree), Dewpoint (Degree), Relative Humidity\n (%), Air Pressure, Wind Direction (Degree), Wind Speed at 76.8 m (m/sec), Power Generated(kW).
15 years of data from 2007 to 2021 with a sampling time of 1 hour

Hello. Welcome to AI SE. You provide some info about your dataset, but it's not clear what are your features (or independent variables) and what the label (or dependent variable, i.e what you want to predict) is. You should probably also tell us how large your dataset is in terms of observations. Finally, you should explain why you think the transformer may be a good choice for this.

Transformers, being a general-purpose sequence model can be used for Time-Series forecasting.
There are some papers dedicated to the use of Transformer for time-series prediction and blogs.
The main ingredient for the autoregression in predictions is the mask in Transformer encoder.
When the next element is predicted, tokens in the sequence attend only to the tokens back in time.

After each block a new element is predicted, based on the decoder and encoder tokens.
However, since the dimensionality of your data seems to be rather small, I would suggest starting from something simpler - say linear AR models or RNN, and only then work with transformers.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How to use K-means clustering to visualise learnt features of a CNN model?
Recently, I was going through the paper Intriguing Properties of Contrastive Losses. In the paper (section 3.2), the authors try to determine how well the SimCLR framework has allowed the ResNet50 Model to learn good quality/generalised features that exhibit hierarchical properties. To achieve this, they make use of K-means on intermediate features of the ResNet50 model (intermediate means o/p of block 2,3,4..), and I quote the reason below

If the model learns good representations then regions of similar objects should be grouped together.

Final Results:
I am trying to replicate the same procedure but with a different model (like VggNet, Xception).
Are there any resources explaining how to perform such visualizations?

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
References for the convergence of gradient-based algorithms for training neural networks
I'm looking for some good references that give convergence results of training neural networks. I'm decently familiar with works that analyze the convergence of SGD, and, in particular, I really like this paper Optimization Methods for Large-Scale Machine Learning. I'm looking for works that talk about the convergence of SGD (or possibly a different gradient-based algorithm) specifically for neural networks.
An example of the type of paper I'm looking for is Convergence Analysis of Two-layer Neural Networks with ReLU Activation.


text:
Using parameter estimation for training a neural network
Assume that we have 4 layers in a neural network.
$$z_1 = L_1(x, W_1)$$
$$z_2 = L_2(z_1, W_2)$$
$$z_3 = L_3(z_2, W_3)$$
$$y = L_1(z_3, W_4)$$
Where $x$ is the vector input, $y$ is the vector output and $W_i, i = 1..4$ is the weight matrix.
Assume that I could estimate parameters in a function.
$$b = f(a, w)$$
Where the $b$ is a real value and $a$ is the input vector and $w$ is the weight vector parameter. The function $f$ could be like this.
$$b = \text{activation}(a_1*w_1 + a_2*w_2 + a_3*w_3 + \dots + a_n*w_n)$$
Here we can interpret $b$ as the neuron output. Estimate $w_n$ is very easy if we know $b$ and $a_n$. This can be done used by recursive least squares or a Kalman filter.
Question:
If every neuron in a neural network is a function that has inputs and weights, can I use parameter estimation for estimating all weights in a neural network if I did parameter estimation for every neuron inside a neural network?
The reason why I'm asking:
I found a paper where they are using a Unscented Kalman Filter for parameter estimation.
Function $D_{k|k-1} = G[x_k, W_{k|k-1}]$ can be interpreted as a neuron function where $W_{k|k-1}$ is a matrix with different types of weights and $D_{k|k-1}$ is different types of outputs from that neuron. No, it's not a "multivariable output"-neuron. It's just the way how to estimate the best weights by using different weights.
The error of the neuron output is: $d_k - \hat d_k$ in equation (41).
So when the error is small, that means the output of the neuron is OK and that means the real weights $\hat w_k$ has been found.

I don't understand what you mean exactly by "parameter estimation" when you ask the question "can I use parameter estimation for estimating all weights in a neural network...?". Can you clarify that? It's not very clear the relationship between your idea of "estimating parameters" and the paper that you found.

@nbro I assume that parameter estimation is a tool for estimating unknown parameters inside an equation, by using data.

So, would back-propagation (which is usually used to train neural networks) be a "parameter estimation" tool/technique or you have something else in mind?

@nbro No. Back-propagation is a optimization method. This method above is estimation. Back propagation you computing the weights by setting the outputs first. In this method above, we don't know the outputs. We only set the inputs and compute the weights just by looking how large the error is between the real data and estimated data.

But back-propagation computes the gradient based on the error, which seems that it's what you're suggesting.

@nbro Yes. all algorithms need to have a reference. Gradient decent is using derivative and this algorithm above is using least squares. They both minimize the error.

Ok, so what is your question then? If you're asking "Can we use back-propagation/gradient descent to train the neural network?", of course, the answer is generally yes. If you're interested in other specific training procedures, then you should probably be more specific. So, please, change your post and title to be more specific and clearer.

@nbro My question is very clear. I can change the title, it's good enough. I'm interesting if it's possible to estimate a neural network by using this method above, which is an estimation method by using a SR-UKF filter.


text:
Why actual mapping is called as unreferenced mapping in this context of residual framework?
Consider the following statements from the research paper titled Deep Residual Learning for Image Recognition by Kaiming He et al.
#1:

We explicitly reformulate the layers as learning residual functions
with reference to the layer inputs, instead of learning unreferenced
functions. We provide comprehensive empirical evidence showing that
these residual networks are easier to optimize, and can gain accuracy
from considerably increased depth.

#2:

In this paper, we address the degradation problem by introducing a
deep residual learning framework. Instead of hoping each few stacked
layers directly fit a desired underlying mapping, we explicitly let
these layers fit a residual mapping. Formally, denoting the desired
underlying mapping as $\mathcal{H}(x)$, we let the stacked nonlinear
layers fit another mapping of $\mathcal{F}(x) := \mathcal{H}(x)−x$.
The original mapping is recast into $\mathcal{F}(x)+x$. We hypothesize
that it is easier to optimize the residual mapping than to optimize
the original, unreferenced mapping. To the extreme, if an identity
mapping were optimal, it would be easier to push the residual to zero
than to fit an identity mapping by a stack of nonlinear layers.

The research paper contains the word "unreferenced" twice for a function, which I think is the actual function $\mathcal{H}(x)$. In normal deep neural networks, the function $\mathcal{H}(x)$ is generally learned. But, $\mathcal{F}(x)$ is learned in the residual neural networks and hence $\mathcal{H}(x)$.
Why this paper is calling the original function $\mathcal{H}(x)$as unreferenced function?


text:
Is Speech to Speech with changing the voice to a given other voice possible?
Background:
I am working on a research project to use (demonstrate) the possibilities of Machine Learning and AI in artistic projects. One thing we are exploring is demonstrating deep fakes on stage. Of course, a deep fake is not easy to make. Still, we are exploring creating a "minor quality" deep fake live on a stage (or maybe in some other ways where people can make deep fakes of themselves) in which we put words into someone's mouth. I discovered that a semi-nice deep fake of the facial movement is possible, now a also want to add voice.
There are a lot of text-to-speech systems, which allow using a voice that is created from the recording of the voice of a real person. That is already nice.
The video is based on the facial movements of another person. So the Audio has to match the facial movements. The easiest way would be, if the "fake" voice says it exactly the same way, as the person doing the "facial acting" for the deep fake said it.
Question:
Is it possible to do a fake voice of a person in this way:

Another person (the actor, or source) speaks the words in his voice and records it.
The person with the faked voice (the destination) gives a voice sample with some random spoken text.
An AI/algorithm/whatever modifies the recording from (1) in such a way, that the tone/voice is modified so it matches the voice from (2).

Do systems/research like this exist? I did not find anything using google, but maybe I did not use the correct keywords.

I think this is probably possible. The research I'm aware of is about images. Have a look at CycleGAN, which learns to convert horses into zebras, with just a bunch of horse pictures and an unrelated bunch of zebra pictures.

also StyleGAN which transfers the "artistic style" from one picture onto another picture. Voice is just "artistic style" but for sound.

of course there is no way to know for sure without actually trying it and probably trying a bunch of dead ends before it might actually work

@user253751 Thanks for your input! The repos you are listing are also interesting. Unfortunately this is out of my skill and time possibilities.


text:
How to generate a response while considering past questions as well?
User: What is the tallest mountain?
Agent: Everest
User: Where is it located? # Agent hears: "Where is Everest located?"
Agent: Nepal

I want to be able to generate a sequence that has been generated using the user's current query as well as the past conversation.
More specifically, I am using Google's T5 for closed-book question answering, but, instead of trivial questions, we use the user's frequently asked queries.
I want to be able to encode their past questions and the agent's past answers, then use them to generate the agent's next answer. How can I do that?

Hello. I edited your post. Please, make sure I didn't change its meaning. In particular, I added this part "we use the user's frequently asked queries.", because what you had originally written wasn't clear (at least to me).

What you want to look for is called anaphora resolution. You basically keep a record of the past conversation and try and find an antecedent for any occurrences of it, he/she, her/his, etc. You probably want to have a pre-processing step where you substitute the antecedent before passing the input sentence on to the agent.


text:
Is there a widely accepted definition of the width of a neural network?
The depth of a neural network is equal to the total number of layers in the neural network (excluding the input layer by convention). A neural network with "many layers" is called a deep neural network.
On the other hand, the width is the name of a property of a layer in a neural network: it is equal to the number of neurons in that particular layer. So, it may be apt to use the phrase "the width of a layer in a neural network".
But, is it valid to use the phrase "width of a neural network"?
I got this doubt because the phrase "wide neural network" is widely used. The phrase gives the impression that the width is a property of a neural network. So, I am wondering whether the width of a neural network has a definition. For example, say, the width of a neural network is the number of neurons in the widest layer of that neural network.

I changed a little bit your post. I think your question was more what I put in the title, i.e. you were wondering if there is a widely accepted definition of the "width of a neural network". Feel free to change your post again.

The width of a neural network layer is an agreed upon term.
According to Lou et al., in the paper The Expressive Power of Neural Networks: A View from the Width (page 4), the width of a neural network is the width of the widest layer of the neural network.

The architecture of neural networks often specified by the width and the depth of the networks. The depth $h$ of a network is defined as its number of layers (including output layer but excluding input
layer); while the width $d_m$ of a network is defined to be the maximal number of nodes in a layer
(emphasis added)

So, I would caution you to be careful with how you use the phrase "the width of a neural network", due to interpretability and scale, and the fact that neural networks often contain layers with varying numbers of neurons, depending on the layer.
From this Wikipedia page on "Large width limits of neural networks":

The number of neurons in a layer is called the layer width.

From a nice machine learning resource page

Finally, there are terms used to describe the shape and capability of a neural network; for example:

Size: The number of nodes in the model.
Width: The number of nodes in a specific layer.

A node [is] also called a neuron or Perceptron

I edited your answer to, hopefully, improve its clarity. I would remove the part "From this wikipedia page ..." until the end of your answer because the OP already knows that we can talk about "the width of a layer". That's a pretty much standard definition, so there's no need to say that again. He's also aware of what "depth" means.


text:
How does Chebyshev approximation of spectral convolution work?
I was reading the following paper: here. In it, it talks about spectral graph convolutions and says:

We consider spectral convolutions on graphs defined as the
multiplication of a signal $x \in R^N$ (a scalar for every node)
with a filter $g_{\theta}$ $=$ $\text{diag}  (\theta)$ parameterized
by $\theta \in R^{N}$ in the Fourier domain, i.e.: $ g_{\theta} * x = U g_{\theta} U^Tx $. We can understand $ g_{\theta}$ as a function of the eigenvalues of $L$, i.e.  $g_{\theta}(\Lambda)$

So far, it makes sense. $U^T x$ is the graph Fourier transform of the signal $x$, then we multiply by $ g_{\theta}$ in the Fourier domain as: $FT(f * g) = F(\omega)G(\omega)$. Then we have the multiplication by $U$ in the front to represent the inverse (graph) Fourier transform.
Then the paper lists some reasons why using the above convolution equation may not be practical in reality:

Evaluating the above equation is computationally expensive; multiplying with eigenvector matrix $U$ is $O(N^2)$
Computing eigendecomposition of $L$ may be too expensive for an arbitrarily large graph
etc.

and then the paper says:

To circumvent this problem, it was suggested in Hammond et al. (2011)
that $g_{\theta}(\Lambda)$ can be well-approximated by a truncated
expansion in terms of Chebyshev polynomials $T_k (x)$ up to
$K^{\text{th}}$ order: $$ g_{\theta '}(\Lambda) \approx \sum_{k =
> 0}^{K} \theta_k ' T_k(\tilde{\Lambda}) $$
with a rescaled $\tilde{\Lambda} = \frac{2}{\lambda_{\text{max}}}\Lambda − I_N$. $\lambda_{\text{max}}$
denotes the largest eigenvalue of $L$. $\theta ′ \in R^K$ is now a
vector of Chebyshev coefficients. The Chebyshev polynomials are
recursively defined as $T_k(x) = 2xT_{k−1}(x) − T_{k−2}(x)$, with
$T_0(x) = 1$ and $T_1(x) = x$. The reader is referred to Hammond et al. (2011) for an in-depth discussion of this approximation. Going
back to our definition of a convolution of a signal $x$ with a filter
$g_{\theta '}$, we now have: $$ g_{\theta '} * x \approx \sum_{k=0}^{K} \theta_k ′ T_k (\tilde{L}) x$$ with $\tilde{L} = \frac{2}{\lambda_{\text{max}}}L − I_N$ ; as can easily be verified by
noticing that $(U \Lambda U^T)^k = U \Lambda^k U^T $

Question: What happened to the terms $U^T$ and $U$ which take the (graph) Fourier transform and invert it respectively?
Attempt: Does it have something to do with what is mentioned in the last line about noticing that $(U \Lambda U^T)^k = U \Lambda^k U^T $? I might guess that we use that because a k-th order Chebyshev polynomial will have $\Lambda ^k$ (and lower powers) present in the equation and thus the $U^T$ and $U$ mean that we can write the convolution equation in terms of the Laplacian matrix $L$

I can't understand the equation you pasted from the given paper. Can you please phase this question in your own words and use latex to write equations?

Hi @SwaksharDeb - the equations were written in Latex. Have fixed the few areas where the '>' character made it into the equations (meant for the block quote). The question can be summarized as: how did the Chebyshev function of $\tilde{\Lambda}$  into a function of L


text:
How to fine-tune a model which was pre-trained on a corpus that contains words with different meanings than the meanings of those words on my corpus?
I have a scenario in which we should leverage previously asked questions (not questions pairs, single question in a column) to locate similar questions within those questions.
How can I fine-tune my model to manage out of vocabulary, as my data includes domain-specific questions (3300 questions)?.
Right now, I'm using hugging face sentence transformers, which is already pre-trained on huge data.
For example, BERT knows that gold is a metal, but, in our domain corpus, it's a platform. We have some terminologies which were not exposed openly, how can I fine-tune the model to get related sentences (handling OOV).

Hello. I'm not sure I understand this question/problem. Let me try to understand if I understood your problem. You have some pre-trained model. You also have your own dataset. However, your dataset contains terms, which are equal to the terms used to pre-train the model, but they have different meaning, but you want the pre-trained model to learn the meaning of the words in your corpus and forget the meaning it's learned before. Is this the question/problem? Btw, please, put your specific question in the title. Right now, it's very vague/general.

your understanding is correct, lets say
the question : How to access gold environment? (here gold is an technical word in our organization).
But, in general gold is a metal. If I would search gold then more importance should be for gold(environment) and not for gold(metal).

Ok, but, in your case, the model is supposed to produce whole sentences and not words, it seems from your description. So, rather than the meaning of words, which can differ in your corpus, you want to fine-tune this model so that it understands that your sentences (in your corpus), which are equal to the sentences used in the corpus used to pre-train the model, have different meaning that the other equal sentences. Or maybe the sentences are different and you're just concerned with the words in the sentences?

See my last edit to your post and, in particular, the title, and make sure it's consistent with your actual question.


text:
What is better to use: early stopping, model checkpoint or both?
I want to get a model which works best, what should I go for while training the model, ModelCheckpoint, EarlyStopping, or both?

Early stopping: stop the training when a condition is met
Checkpoint : frequently save the model

The purpose of Early Stopping is to avoid overfitting by stopping the model before it happens using a defined condition. If you use it, and then you save the model when the training is stopped*, you will get a model that is assumed to be good enough and not overfitted.
The purpose of the class ModelCheckpoint is to save models several times while training. This can be useful to find at which epoch the model gets the best performance. So, if you use it, you will get several models that are saved at different epochs (or, more generally, "checkpoints").
Even after using both methods, you will get some models, but they have a different purpose. None of them is better than the other. I almost always use both methods at the same time. You can use early stopping to stop the training and save a lot of models while training using ModelCheckpoint. In most of my cases, the best model is around the epoch during early stopping.
*note: the model saving process is not done by EarlyStopping

What do you mean by "Even both methods "save the model,". As far as I remember, Early stopping does not save any model automatically. The EarlyStopping class has a parameter restore_best_weights, but this is just about restoring the weights of your final neural network (if I remember correctly).

ah that's correct! I have edited my answer

I try to highlight "save the model" part because in this part both methods may look similar (and I think that's the reason OP is confuse)

Maybe it's obvious but other (very important) checkpointing's purpose is to safeguard the training, especially if the training takes very long time and is being done on a (campus/company/shared) server that have risks of stopping the training (hardware issues, electrical issues, etc) we can still continue our training from the last checkpoint.

@Sanyou Yes, but you need to load the weights manually from the checkpoint, at least, as far as I remember, that's how keras works, i.e. it doesn't automatically restart training from the last saved checkpoint, unless you manually tell keras to do that. Feel free to edit this answer to add that detail or feel free to add another formal answer, because I think this "detail" is important, you also think.


text:
How are these two versions of the Bellman optimality equation related?
I saw two versions of the optimality equation for $V_{*}(s)$ and $Q_{*}(s,a)$.
The first one is:
$$
V_{*}(s)=\max _{a} \sum_{s^{\prime}} P_{s s^{\prime}}^{a}\left(r(s, a)+\gamma V_{*}\left(s^{\prime}\right)\right)
$$
and
$$
Q_{*}(s, a)=\sum_{s^{\prime}} P_{s s^{\prime}}^{a}\left(r(s, a)+\gamma \max _{a^{\prime}} Q_{*}\left(s^{\prime}, a^{\prime}\right)\right)
$$
The second one is:
$$
V_{*}(s)=\max _{a \in \mathcal{A}}\left(R(s, a)+\gamma \sum_{s^{\prime} \in \mathcal{S}} P_{s s^{\prime}}^{a} V_{*}\left(s^{\prime}\right)\right)
$$
and for $Q_*$
$$
Q_{*}(s, a)=R(s, a)+\gamma \sum_{s^{\prime} \in \mathcal{S}} P_{s s^{\prime}}^{a} \max _{a^{\prime} \in \mathcal{A}} Q_{*}\left(s^{\prime}, a^{\prime}\right)
$$
If following distributive property to get from the first to the second expression. Why there is no summation term for the reward, for example, $$V_{*}(s) = \max_{a}(\sum_{s'}P^{a}_{ss'}r(s,a)+\gamma\sum_{s'}P^{a}_{ss'}V_{*}(s'))$$?
My guess is that $r(s,a)$ is the constant so it can be moved out of the summation, leaving $$r(s,a)\sum_{s'}P^{a}_{ss'} = r(s,a).$$
But is it always the case that $r(s,a)$ is independent of $s'$? I think the reward of moving from state $s$ to $s'$ may vary.

My guess is that $r(s,a)$ is the constant so it can be moved out of the summation, leaving $r(s,a)\sum_{s'}P^{a}_{ss'} = r(s,a)$

Yes, this is the case. More specifically:

$r(s,a)$ is the expected reward after taking action $a$ in state $s$.
Reward may depend on the state arrived in, $s'$, but that is ignored in the equations.
Reward may vary randomly, but by using the expected reward, this can be ignored.

The first equations you quote, which sum over $s'$ but use $r(s,a)$ inside that sum, are very misleading IMO, since the individual terms may not represent anything meaningful within the MDP. That is the term $r(s,a) + \gamma V^*(s')$ does not correspond to any part of the trajectory of the agent.
Although the sum is still mathematically sound, it is more normal to see a different term $r(s,a,s')$ (the expected reward similar to $r(s,a)$ but also conditional on $s'$) where the expected reward is used inside the sum of next states. The term $r(s,a,s') + \gamma V^*(s')$ does correspond to nodes on the trajectory of the agent. It is the expected future return from $s,a$ conditional on the state transitioning to $s'$.

but is it always the case that $r(s,a)$ is independent of $s'$. I think the reward  of moving from state $s$ to $s'$ may vary.

Yes $r(s,a)$ is independent of $s'$. Although individual rewards may vary stochastically, and may depend on $s'$ too, the term is already the expected reward when taking the action $a$ in state $s$. So it already includes any effects of random state transition and random reward. For the Bellman equations to work as written, the expectation needs to be independent of the policy $\pi$ thus a property of the environment, and this is the case.
I think both sets of equations are a little bit awkward from using a combination of expected reward, yet summing up expectations over the state transition matrix. I prefer the notation used in second edition of Sutton & Barto's Reinforcement Learning: An Introduction:
$$v^*(s) = \text{max}_a \sum_{r,s'} p(r,s'|s,a)(r + \gamma v^*(s'))$$
Where $p(r, s'|s,a)$ is the conditional probability of observing reward $r$ and next state $s'$ given initial state $s$ and action $a$. The $p(r, s'|s,a)$ function replaces the combination of state transition matrices $P_{ss'}^a$ and the expected reward (either $r(s,a)$ or $r(s,a,s')$). Those objects can be derived from $p(r,s'|a,s)$ if you want, but personally I find the newer notation easier to follow.


text:
Can I always use "encoding" and "embedding" interchangeably?
This question is restricted to the text domain only.
The meaning of the word "encode" is Convert (information or instruction) into a particular form. One which performs encoding is called an encoder.
In deep learning, an encoder can also be the first part of a neural network (autoencoder) that simulates identity function, which governs the English meaning of encoder since it encodes the input.
Embeddings are encodings where the intention is to preserve semantics. You can observe the following excerpt from the chapter Vector Semantics and Embeddings

In this chapter we introduce vector semantics, which instantiates this
linguistic hypothesis by learning representations of the meaning of
words, called embeddings, directly from their distributions in texts.

But all encodings may not be the embeddings since encodings might not always preserve semantics (?). I have doubt in this statement which I inferred based on my current knowledge.
Many times, I came across the terms text encoding and text embedding interchangeably. But failing to catch whether they are the same or we need to be choosy while using them.
Consider the following usages of encoding and embedding in the paper titled Generative Adversarial Text to Image Synthesis by Scott Reed et al.

#1: The intuition here is that a text encoding should have a higher compatibility score with images of the correspondong class compared to any other class and vice-versa.
#2: Text encoding $\phi(t)$ is used by both generator and discriminator.
#3: ...where $T$ is the dimension of the text description embedding.
#4: ... we encode the text query $t$ using text encoder $\phi$. The description embedding $\phi(t)$ is first compressed ...

I think they are used interchangeably. Is it true? Can I use any word if I am confident enough that my encoding is semantic preserving? Or is there any strong reason for choosing the words?
If you observe the last point, the word "encoder" is used. Can I use embedder instead of it?

This blog post states that  "There are no completely standard terminology guides for machine learning. Each project, research paper, and blog post should explain what each term means. The terms “encoding”, “embedding”, and “latent representation” can be, and often are, used interchangeably."

Caveat: I am not a native English speaker (but French). And mostly interested in symbolic artificial intelligence (the topic of my PhD thesis defended in 1990; see books by Jacques Pitrat)
Encoding is related to decoding. Most of the time, if you encode something A into some other thing B, you can "decode" B to get back A.
Otherwise, you would just say "parsing".
Embedding means mapping something into a "greater" set (or category, e.g. differential manifolds).
Feel free to email me for more explanations.

From my experience with reading papers and books, I think these two terms are sometimes used interchangeably.
As you also point out, an encoder (in an auto-encoder) also may also learn some "semantics" of the inputs in order to produce the latent space. However, the way encoders are trained may not produce embeddings, with similar properties to e.g. word embeddings. For example, an image of a cat may be mapped to a latent vector that is closer to the latent vector of a person than to the latent vector of a dog (the usual way deterministic autoencoders are trained doesn't enforce these properties).
So, in my head, an encoding may not have any semantics (one-hot encoding is the typical example), but an embedding has. However, again, it depends on the context, so you should take context into account. So, I expect people to use the term encoding to refer to an embedding.

The OP is correct:

One definition of "encoding" is perfectly synonymous with the semantic definition of "embedding." and the two words are used interchangeably, as in the work the OP sites.
Yet "encode" and "embed" can also be used in a contrasting non-semantic vs. semantic sense as the OP also suggests.

The encoder-decoder architecture uses "encode" in a sense synonymous with "embed."
Positional encodings and byte-pair encodings (word-piece encodings) use "encode" in a sense that is contrasting with the semantic meaning of "embedding."
Synonymous by definition
According to Merriam-Webseter, one definition of encoding is "to convey symbolically -- 'the capacity of poetry to encode ideology—J. D. Niles'".  The symbols of poetry are deeply linked to the subtle variations of meanings in words -- thus poetic "encoding" indeed "embeds" from one representation to another, using representations that are strongly linked to the meanings of terms.
Use of encode and embed in the first Transformer paper.
This section is written in the context of LLMs (large language models) like ChatGPT, specifically in the first Transformer paper, Attention is All You Need.
In this paper, the term "encoding" is indeed used for a non-semantic encoding -- that is, for representations that do NOT attempt to push words with similar meanings closer in the representational (encoding) space.  The word "encoding" is used in this sense twice in the paper. The first is for positional encodings, which encode the position of a word within the input text uniquely with an encoding that is something like a soft embedding.  The second is for "byte pair encodings" (BPE), which was the term used for Word-Piece encodings at the the time this paper was written.  Word-Piece encodings map the input into a sequence of indices that correspond to known words or pieces of words.  The original words can be recovered from the sequence of indices and the indices do NOT move meanings closer together.
The term "embedding" is also used in this paper in the sense the PO and others have discussed.  Here, the BPE (word-piece) tokens are embedded into a 512-dimensional space, where, during learning, the embeddings will drive similar words towards similar embeddings, thus producing a semantic embedding.
But, in the same paper, the term "encoder" is consistently used for a large network whose sole purpose is to produce rich semantic embeddings that embed each word of the sequence into a rich 512-dimensional space where the position of the vector within that space not only encodes the meaning of that word, but also the meanings of other words to which that word is connected grammatically within the sentence.
What are we to make of this?  The words "encoding" and "embedding" do seem to clearly differ in being non-semantic or semantic.  Yet the word "encoder" is clearly semantic.
Are the authors of Attention is All You Need being inconsistent about their use of language?  I don't think so.  I believe that the popular "encoder-decoder" architecture, already popular long before Transformers came on the scene, breaks from the encoding/embedding distinction seen elsewhere and indeed uses the term "encoding" synonymous with the term "embedding."
The Encoder-Decoder Architecture
Why are these called encoder-decoder architectures?  Here, encode is used in the synonymous sense of "semantic embeddings," and is used instead of embedding because entire sentences are considered rather than single words.  The d2l.ai text clearly defines the encoder-decoder architecture as "consisting of two major components: an encoder that takes a variable-length sequence as input, and a decoder that acts as a conditional language model, taking in the encoded input and the leftwards context of the target sequence and predicting the subsequent token in the target sequence."
Embeddings as manifolds within a larger space
As illustrated clearly in a figure from an answer to another question already linked to this one, the term "embedding" can refer to the process of putting one space within another according to a nonlinear mapping.
When we use this term to talk about embedding the meanings of words within a higher-dimensional space, we should NOT assume that the embeddings fall on a lower dimensional manifold (surface) within that space.  They may very well fill the entire dimensionality of that space.  But the word still carries this picture of embedding points into a space, just as one embeds nails into wood to make nail-art.
The word encoding describes exactly the same process, but has an emphasis on preserving the meaning of the original text.  But yet again, an embedding can also preserve the meaning of the original text. For example, the embeddings of word-pieces must be reversable, or some tokens in the vocabulary could never be copied by the network to the output.
Conclusion
It is clear from this brief review of a tiny fraction of the literature that the terms "encode" and "embed" can be used synonymously (in the sense of transforming one representation to another, considering the meaning of what is being transformed), but they can also be used in contrast to one another (with the word "encode" implying that the meanings of terms are NOT taken into account, while the word "embed" does take those meanings into account).
The encoder-decoder architecture uses "encode" in a sense synonymous with "embed."
Positional encodings and byte-pair encodings (word-piece encodings) use "encode" in a sense that is contrasting with the semantic meaning of "embedding."

What does "OP" stand for?

@SaryeHaddadi Original Poster.  This is the person who asked the question.  This is one of the most common acronyms used internally within Stack Overflow. There shouldn't be too many others to learn for this site, or I'd share a few more with you now. But I can't think of any others at the moment!

@SaryeHaddadi Welcome to Stack Overflow.  I've found this a very welcoming place over the years, despite the rumors. But boy, it can be hard to earn basic privileges like editing. All the best and feel free to @ mention me with questions.


text:
Are there any deep RL algorithms that work well on finite MDPs and non-trivial terminal rewards?
I notice that most Deep Reinforcement Learning (DRL) works focus on Markov Decision Process (MDP) with an infinite time horizon.
Are there any algorithms that work well on finite MDP and non-trivial terminal reward?
My definition of non-trivial terminal reward is that the reward function at the terminal time is different from the one at non-terminal timestamps. Many environments or games fall into this category. For example, many games are on $[0, T]$, the total reward is the summation of the accumulated reward plus the final bonus.


text:
How to handle random order of inputs and get same output?
I am a beginner with DL. I did some tutorials and I know the basics of TensorFlow. But I have a problem understanding how to construct more advanced NNs.
Let's say I have 6 inputs and a list of 500 names from which you can pick any, but only 6 at the time. The output should be one value between $0.0$ and $1.0$.
My question is, how I can handle random order in inputs? In inputs 1-6 you have names ABCDEF and the output score is 0.7. I need the same output if input will be in order CEDBFA. How can I handle this? Should I make random shuffle on inputs during training, or should I make for every output value 500D binary vector-like $[0,0,1,0,1,...,0,1,0,0,0]$, where index position in the array is the corresponding token of name and then feed it in 500 inputs? Or there is some better way?

Note: This is always going to be an estimate until you actually run the experiment. ML is not always predictable.
If order truly does not matter, then I think it will be better to design a network architecture that automatically ignores order, instead of using one that cares about order and then training it to ignore order. If nothing else, less training data will be needed since you don't need to train it on permutations of the input - similar to why CNNs are useful for image recognition.
One network architecture could be to have an "input processing block" (a group of layers) and an "output processing block". First apply the input processing block to each input. Then add together all the outputs of the input processing blocks. Because addition is insensitive to order, this step completely discards any information about the order. Finally, apply the output processing block to the sum of those and the output from that block is your final output.
           output
             ^
             |
           +--+
           |NN| (different NN to the one below)
           +--+
             ^
             |
+---------------------------+
| add all together          |
+---------------------------+
 ^    ^    ^    ^    ^    ^
 |    |    |    |    |    |
+--+ +--+ +--+ +--+ +--+ +--+
|NN| |NN| |NN| |NN| |NN| |NN| (same NN 6 times)
+--+ +--+ +--+ +--+ +--+ +--+
 ^    ^    ^    ^    ^    ^
 |    |    |    |    |    |
 C    E    D    B    F    A

This is just one idea. You are not limited to addition; you can use any commutative operation in the middle. Even something like an attention layer could be used.


text:
What are examples of node 'features' in graph networks?
Context: I was reading Chapter 3 in the following book (here) about graph representation learning. Before I get to node embeddings, I wanted to make sure that I do understand what is meant by the phrase 'node features' used numerous times throughout the book. Examples are as follows:
Chapter 5, page 50:

Node Features: note that unlike the shallow embedding methods discussed in Part I of this book, the GNN framework requires that we
node features $\mathbf{x}_{u}$, $\forall u \in \mathcal{V}$ as input
to the model. In many graphs we will have rich node features to use
(e.g. gene expression features in biological networks or text features
in social networks)....

Question: What is a simple, concrete example of different node features? I have read the paragraph above, but I am not sure whether I have interpreted it correctly. For example, if we imagine a social network of some friends, would some example node features be: address, age, height, weight, etc.? Would it be as simple as that? What are some more advanced/subtle bits of information which could be counted as node features. Perhaps one could be 'number of friends' (i.e. the degree of the node), but what about others.

For example, if we imagine a social network of some friends, would
some example node features be: address, age, height, weight, etc.?
Would it be as simple as that?

Yes, that is correct. The idea behind Graph Neural Networks (GNNs) is that existing node features are augmented by the local graph structure to produce more accurate results.

What are some more advanced/subtle bits of information which could be
counted as node features. Perhaps one could be 'number of friends'
(i.e. the degree of the node), but what about others.

Yes - structural / positional properties of the graph (such as degree distribution) can also be used. They are particularly useful for graphs without existing attributes to use as node features.
An example use case of this would be modelling molecules as a graph, and trying to classify each molecule by type.
The following papers compare and discuss various different graph properties that could be used for this, depending on task / graph:

https://doi.org/10.48550/arXiv.2107.01495
https://doi.org/10.48550/arXiv.1911.08795

What is a simple, concrete example of different node features

Generally, you want a feature that is correlated to the task at hand.
For example, user interests might be a good feature for predicting friendships between users in a social network, but less good for bot detection.
Same principle applies to positional features (for attributeless graphs) as well.
It might be helpful to look at the features given with common datasets:
https://pytorch-geometric.readthedocs.io/en/latest/modules/datasets.html


text:
What is the reason behind using node embeddings?
I was reading Chapter 3 from the following book (here) on graph representation learning. The chapter is about node embeddings.
Question: What is the point of using node embeddings?
Do we use them:

to save space/memory by mapping our graph into a form which is of lower dimension?
to find a representation of the graph which can be fed into a neural network?
perhaps some other reason?

It isn't clear to me what the actual purpose/end-goal is of finding a representation is.
Any help would be greatly appreciated as I have also been reading about graph neural networks, which aim to find an embedding for the nodes (but I don't understand why that is of any use to us).


text:
Should I repeat lengthy deep learning experiments to average results ? How to decide how many times to repeat?
I am doing my MSc thesis on deep learning. My model takes many hours to train. Part of what I do is trying different parameters and settings hoping that they will achieve different results. But I often notice that the result differences are too small to conclude whether the set of parameters A is better than B. Sometimes it happens that on a first run, set A seems to work better than B, but on a second run the opposite is suggested.
The logical approach for that would be to repeat experiments and average out. But it seems unpractical given that each run could take so many hours.
I wonder how expert AI researchers deal with that, do they perform multiple experiments, even if this takes extremely long? Do they draw conclusions from single runs?

The goal has always been tune it enough to beat SOTA and then publish a paper with the model/weights.

I wonder how expert AI researchers deal with that, do they perform multiple experiments, even if this takes extremely long? Do they draw conclusions from single runs?

Unfortunately, the question you ask in the main body of your question here ("how do expert AI researchers do things") often turns out to actually be different from the question in your title ("how should I do things").
The very short answer would be that, ideally, you run as many repetitions as you can, but in practice it is indeed very often not feasible to do more than one or a handful.

For the long answer, I think it actually may be quite different also depending on what kind of machine learning you are doing. Personally, I am much more familiar with Reinforcement Learning and similar kinds of problems, i.e. problems where we're generating the "training" data ourselves by making agents act in environments, and similarly also evaluating by again making trained agents act in environments and measuring their performance. I'm not as familiar with the state-of-the-art research in "standard" machine learning tasks like classification/regression, but probably the problem is much worse in RL-style problems because:

We actually have to generate our own data, which takes a huge amount of time (sometimes much more time than the actual training itself takes)
There is often a lot of randomness in how we generate our training data, so we often have very different training datasets in different runs, which can of course also lead to wildly different levels of performance across different runs
We pretty much always compute gradients from subsets (batches) of data in RL, which is again a source of randomness. In contrast, in supervised ML you could consider estimating gradients from the entire dataset at once per epoch, and then at least that part of your training process becomes deterministic (and hence replicable).

For the case of RL, recently a paper titled "Deep Reinforcement Learning at the Edge of the Statistical Precipice" appeared on arXiv, which gets into various tools and approaches you can use to more reasonably draw principled conclusions even from a small handful of repetitions of your training process (in better ways than the common practice of just reporting a mean, or even worse, just reporting the best result).
For supervised ML, some similar techniques may be applicable, but the need for this may also be less great (especially if you have little randomness in your training process).

Part of what I do is trying different parameters and settings hoping that they will achieve different results. But I often notice that the result differences are too small to conclude whether set of parameters A is better than B. Sometimes it happens that on a first run, set A seems to work better than B, but on a second run the opposite is suggested.

For this specific situation, I would be interested to know by how much A is sometimes better than B, and B is sometimes better than A. If we're talking about like 0.1% differences in accuracy either way at a baseline accuracy of 99.5%, that probably just means they're both equally good. If you were "hoping" for one of them to outperform the other, well then that's probably disappointing... but for the overall training process in general, you could actually draw a more positive conclusion that, apparently, it is robust; it performs similarly across different runs even with different parameters!
On the other hand, if you observe differences in performance of significant magnitudes (like, 80% vs 90% accuracy), but sometimes A better by that much and sometimes B better by that much... then this strongly suggests that you indeed are dealing with high variance and you're going to have to do many repetitions to get statistically meaningful results.

In addition to the other good answer, if you can afford the luxury of running your experiments multiple times, you can also use hypothesis testing to test whether there is any significant difference between the performance (e.g. accuracy) of the two models.
Hypothesis testing is not widely used (or, at least, reported in research papers) in the ML/DL community (given that, to apply these tests, you often need to repeat the experiments several times, which can be expensive, as you already know), but there are some researchers (for example, see this paper) that often use these statistical tools.
I must also admit that I don't know how valuable they can really be. I've used them in the past, but not enough to say that they can really be useful in the context of machine learning.


text:
What could cause the hamming loss and subset accuracy to get stuck in a multi-label image classification problem?
I am rather new to deep learning and got some questions on performing a multi-label image classification task with keras convolutional neural networks. Those are mainly referring to evaluating keras models performing multi label classification tasks. I will structure this a bit to get a better overview first.
Problem Description
The underlying dataset are album cover images from different genres. In my case those are electronic, rock, jazz, pop, hiphop. So we have 5 possible classes that are not mutual exclusive. Task is to predict possible genres for a given album cover. Each album cover is of size 300px x 300px. The images are loaded into tensorflow datasets, resized to 150px x 150px.

Model Architecture
The architecture for the model is the following.
import tensorflow as tf

from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.models import Sequential

data_augmentation = keras.Sequential(
  [
    layers.experimental.preprocessing.RandomFlip("horizontal",
                                                 input_shape=(img_height,
                                                              img_width,
                                                              3)),
   layers.experimental.preprocessing.RandomFlip("vertical"),
    layers.experimental.preprocessing.RandomRotation(0.4),
   layers.experimental.preprocessing.RandomZoom(height_factor=(0.2, 0.6), width_factor=(0.2, 0.6))
  ]
)

def create_model(num_classes=5, augmentation_layers=None):
  model = Sequential()

  # We can pass a list of layers performing data augmentation here
  if augmentation_layers:
    # The first layer of the augmentation layers must define the input shape
    model.add(augmentation_layers)
    model.add(layers.experimental.preprocessing.Rescaling(1./255))
  else:
    model.add(layers.experimental.preprocessing.Rescaling(1./255, input_shape=(img_height, img_width, 3)))

  model.add(layers.Conv2D(32, (3, 3), activation='relu'))
  model.add(layers.MaxPooling2D((2, 2)))
  model.add(layers.Conv2D(64, (3, 3), activation='relu'))
  model.add(layers.MaxPooling2D((2, 2)))
  model.add(layers.Conv2D(128, (3, 3), activation='relu'))
  model.add(layers.MaxPooling2D((2, 2)))
  model.add(layers.Conv2D(128, (3, 3), activation='relu'))
  model.add(layers.MaxPooling2D((2, 2)))
  model.add(layers.Flatten())
  model.add(layers.Dense(512, activation='relu'))

  # Use sigmoid activation function. Basically we train binary classifiers for each class by specifiying binary crossentropy loss and sigmoid activation on the output layer.
  model.add(layers.Dense(num_classes, activation='sigmoid'))
  model.summary()

  return model

I'm not using the usual metrics here like standard accuracy. In this paper I read that you cannot evaluate multi-label classification models with the usual methods. In chapter 7. evaluation metrics the hamming loss and an adjusted accuracy (variant of exact match) are presented which I use for this model.
The hamming loss is already provided by tensorflow-addons (see here) and an implementation of the subset accuracy I found here (see here).
from tensorflow_addons.metrics import HammingLoss

hamming_loss = HammingLoss(mode="multilabel", threshold=0.5)

def subset_accuracy(y_true, y_pred):
    # From https://stackoverflow.com/questions/56739708/how-to-implement-exact-match-subset-accuracy-as-a-metric-for-keras

    threshold = tf.constant(.5, tf.float32)
    gtt_pred = tf.math.greater(y_pred, threshold)
    gtt_true = tf.math.greater(y_true, threshold)
    accuracy = tf.reduce_mean(tf.cast(tf.equal(gtt_pred, gtt_true), tf.float32), axis=-1)
    return accuracy

 # Create model
 model = create_model(num_classes=5, augmentation_layers=data_augmentation)

 # Compile model
 model.compile(loss="binary_crossentropy", optimizer="adam", metrics=[subset_accuracy, hamming_loss])

 # Fit the model
 history = model.fit(training_dataset, epochs=epochs, validation_data=validation_dataset, callbacks=callbacks)

Problem with this model
When training the model  subset_accuracy hamming_loss are at some point stuck which looks like the following:

What could cause this behaviour? I am honestly a little bit lost right now. Could this be a case of the dying ReLU problem? Or is it wrong use of the metrics mentioned or is the implementation of those maybe wrong?
So far, I tried to test different optimizers and lowering the learning rate (e.g. from 0.01 to 0.001, 0.0001, etc..) but that didn't help either.

Generally when you see metrics stagnate like this it's because the model has converged incorrectly (ex: always predicting 0, or gradients/weights have dropped to 0, etc.). Can you see if this is what's happening for your problem?
I'd expect that perhaps your model has converged to predict the majority class for each label.


text:
Is there any relationship between the batch size and the number of epochs?
I am currently running a program with a batch size of 17 instead of batch size 32. The benchmark results are obtained at a batch size of 32 with the number of epochs 700.
Now I am running with batch size 17 with unchanged number epochs. So I am interested to know whether there is any relationship between the batch size and the number of epochs in general.
Do I need to increase the number of epochs? Or is it entirely dependent on the program?

The smaller the batch_size is, the larger number of batches is processed per one epoch.
On one hand, since one makes more steps per epoch, one can think, that less epochs are required to achieve the same level of accuracy.
On the other side, smaller batch size leads to more noisy and stochastic estimates of the gradient, therefore, convergence would not be as steady most likely.
I think it is difficult to give a definite answer about the exact relation on the number of epochs - since say, to achieve a certain level of accuracy use of small batch may be more beneficial since it allows for more exploration and is more likely to escape from local minima and saddles, but when one reaches the approximation limit of the network and is in the vicinity of the good optimum - large batch would descend better to the extremum.
Good strategy usually is to start from smaller batches to find wide and plain minima, which are better from the generalization point of view, and then increase batch size for steadier convergence.

Smaller batch size means the model is updated more often. So, it takes longer to complete each epoch. Also, if the batch size is too small, each update is done without "seeing" all the data - the batch itself might not be a good representative of the dataset. So, there might be too much "wiggling", which makes it harder to get real minimum. Larger batches will get you "near" minimum for quickly, as they have larger and step sizes.
As you only asked the relation between batch size and epochs, above is the answer. But in practice, one hardly uses very large batches as it doesn't get very close to the minimum for the same reason they reach there faster - once you are there, you want to have smaller steps to get very close to the minimum. Smaller batches might take longer but once they are there, their wiggly nature becomes a strength, and gets them closer to the minimum.
For a more in depth discussion and some references, see this post.
And batch sizes are usually picked to be a power of 2, so I would go for 16 instead of 17 if I were you. Check this discussion for the reason for this.

"Smaller batch size means the model is updated more often. So, it takes longer to complete each epoch", but if you use fewer examples at each update, wouldn't that make the update faster? So, wouldn't that smaller times for updating compensate the higher number of updates? I don't know exactly, but that statement/conclusion may not be correct. Note that I am not talking about convergence here, I'm just talking about time complexity.

Well, still each time the update is made, the entire model is updated, that is the issue. Small batch or large batch, for each epoch entire dataset is used once in the calculation. Assuming total time for the calculation excluding updates is the same for both, the difference comes from the number of updates. And the number of times an update is made is higher for small batches.

@nbro Sorry, I guess I did not address your comment directly but I don't know why small samples would mean shorter update time. Even we assume the changes are small in small batches, the mathematical operation made on each cell is the same, and should take the same amount of time.


text:
How do I compute the value function when the reward is only at the end in the context of actor-critic algorithms?
Consider the actor-critic reinforcement learning setting (actor and critic parameterized by a neural network). The reward is given only at the end of the episode (or when there is a timeout there is no reward).
How could we learn the value function? Do you recommend computing intermediate rewards?

The reward is given only at the end of the episode (or when there is timeout there is no reward)

This is a common case. E.g. winning a board game, or reaching a goal state.

How could we learn the value function?

All RL algorithms are designed to cope with this scenario. Actor-Critic is not an exception. Value-based algorithms (including the critic in Actor-Critic) learn through time step backup updates. The simplest backup is to copy data about experience at time $t+1$ into an update for the state or state/action experienced at time $t$. That is what single-step temporal difference algorithms do. Other value-based algorithms can be more sophisticated and more efficient with assigning the update signal back in time.
A very sparse reward can be difficult for an agent to find or learn from. So some methods may work better than others. Without knowing the environment, and what the specific difficulties might be, it is not possible to recommend an approach, or to even suggest whether your algorithm needs any help.

Do you recommend computing intermediate rewards?

In general, no. However, these can help when the "natural" rewards in a problem are both sparse and hard for the agent to discover through exploration. Constructing extra reward signals to guide a learning agent is called reward shaping.
Reward shaping needs to be done with care because it can inadvertently change what the optimal solution is. But if done well, it can make a problem much easier to solve for an agent.
A starting rule of thumb for whether to add some kind of reward shaping is based on how often a random agent might accidentally obtain a difference in end rewards. Once the agent has experienced a difference between rewards, it can start to refine its predictions and begin to prefer the higher reward, which then usually leads to exploration nearer more valuable states. It may do this even if the higher reward only occurs one time in a thousand initially, say. However, if initial random actions gain no useful signal even for millions of trial-and-error episodes, then you will need to do something to assist the agent.


text:
Compare the efficiency of a trained ML model with a non-learning-based method for solving the same problem
If a certain task T is solved by a non-learning-based method A (let's say, an optimization-based approach). We now train a machine learning model B (let's say a neural network) on the same task.
What are some metrics that we can use to compare their efficiency in terms of finding the solution (Assuming the quality of both solutions is comparable)?

The most generic answer to this question is:
the same metrics you use to evaluate the quality of your model during training or in test phase. (Plus the timing of inference if you're referring to computational efficiency)
And I'm not referring to any specific metric yet cause that's really task dependent. But in general if you have a model that perform a task and another algorithm that perform the same task, then you should be able to apply both to the same set of data, compute whatever metric is suitable to evaluate the performance on the task, and compare the two scores. Let me stress out that the test instances should be the same for a scientifically relevant comparison, and I mean literally the same.
As an example of some metrics I would refer to the web since out there there's plenty of blog posts listing and comparing metrics. Just to link a few:

classification
regression
image quality assessment
object detection
machine translation

The list is not exhaustive but I think it illustrates the point.
Also, as a side note: almost all machine learning algorithms are optimization-based, if you want to refer to approaches that don't fall into machine learning I think a better term is analytic methods/approaches.

This is a great answer, I just want to suggest another side note. If OP is writing a scientific article, it might be worth adding another point in the Results and Discussion which is "how can the ML-based methods mimic the performance or compete with the performance of 'non-ML' based methods"? does it rediscover a certain (well-known) method? For example, an RL agent for travelling salesman problem could rediscover the greedy heuristic as shown by likelihood of taking the furthest node in each step.


text:
What can cause massive instability in validation loss?
I'm working with very weird data that is apparently very hard to fit.
And I've noticed a very strange phenomenon where it can go from roughly 0.0176 validation MSE to 1534863.6250 validation MSE in only 1 epoch! It usually then will return to a very low number after a few epochs. Also, no such fluctuation is seen in the training data.
This behavior of instability is consistent across shuffling, repartitioning & retraining. Even though I have 16,000 samples and a highly regularized network (dropout + residual layers + batch normalization + gradient clipping).
I mean I realize I could have more data, but, still, this behavior is really surprising. What could be causing it?
P.S. Model is feedforward with 10 layers of size [32,64,128,256,512,256,128,64,32,1], using Adam optimizer. Also, this question may be related (my experience is also periodic validation loss), but I don't think they experienced the same massive instability I am seeing.

Given that you're using the MSE, I assume you're solving a regression problem, right? It may be a good idea to also share with us how much data you are using to compute the validation loss. What's the range of your inputs and targets?

The targets follow an exponential distribution but apparently can also be negative.
So they can range from like -1e8 to 1e10.


text:
Do you need a terminal state when using double deep q networks?
I just got my agent training, and I'm wondering if the terminal flags are necessary when sampling from the replay buffer. The game I'm implementing the agent in has two different ways the game can end, and so far my agent seems to be learning without terminal flags. I was wondering how important this feature is, as it's in all the pseudocode but doesn't seem to be necessary in my implementation.

It's an important feature, and you drop it at the risk of the agent failing to learn successfully.
The difference between the TD target without the terminal flag
$$G_t = R_{t+1} + \gamma \text{max}_{a'} Q(S_{t+1}, a')$$
and with the terminal flag applied to $S_{t+1} = S_T$
$$G_t = R_{t+1}$$
is important whenever $Q(S_{T}, a')$ might be evaluated as non-zero. The true action value of the terminal state is always defined as zero.
In some cicumstances, values based on a function approximator estimate could be far from zero - if at any point the estimator moves significantly away from zero then this could lead to value estimates diverging. This can be a problem at any time, because the estimator is usually not trained on actions taken from the terminal state - due to them never being observed. In principle you could add training data to keep the function approximator in line with this estimate, but as it is an absolute value it is far more common (when using approximators such as neural networks) to change the TD target as above.
Double Q learning improves estimates of bootstrap values by removing some of the maximisation bias from the next action value. This may help in your case, if the expected future rewards from the terminal state remain close to zero.
I would expect Double Q learning to work OK without using terminal state informaton, and without adding "mitigating" training to keep terminal state estimates close to zero, if rewards are not sparse, and do not increase close to the terminal states. That would mean the estimator should recognise expected future reward becomes closer to zero as the agent approaches a terminal state, thus would not take much extrapolation to predict close to zero for the terminal state - close enough that it does not create runaway feedback or influence the optimal policy much.
With sparse rewards, or with significantly high rewards close to the terminal state (e.g. a goal state which the agent must reach to gain all the reward), then using the terminal flag in the normal way becomes more important. It is unlikely that double Q learning would help much in that case. However, it may still be possible to find the optimal policy but with highly inaccurate action values (e.g. action values could all be double what they should be).


text:
What do people refer to when they use the word 'dimensionality' in the context of convolutional layer?
In practical applications, we generally talk about three types of convolution layers: 1-dimensional convolution, 2-dimensional convolution, and 3-dimensional convolution. Most popular packages like PyTorch, Keras, etc., provide Conv1d, Conv2d, and Conv3d.
What is the deciding factor for the dimensionality of the convolution layer mentioned in the packages?

What do you mean by "dimensionality of the convolution layer"? A convolutional layer is just an abstraction to perform the convolution (i.e. a function). So, to me, it doesn't make sense to ask "What is the dimensionality of a function?". So, you should start by clarifying this before proceeding. Also, I don't understand this sentence "Neither the shape of the input nor the shape of the kernel decides the dimensionality of the convolution operation." because I don't understand what you mean by "dimensionality of a layer". So, you should also clarify that sentence and why you claim that.

@nbro Is it fine now? I am asking about the dimensionality of the layers that are discussed in packages.

To be honest, your last edit doesn't clarify what was in my opinion unclear. As I said, convolutional layers should be seen more as functions rather than tensors (indeed, they are just an implementation of a function, i.e. the convolution). So, I don't understand what you mean by "dimensionality of a function here". Maybe you mean "the dimensionality of the input to and output of the convolutional layer" That would make more sense. Like in the function $f : \mathbb{R}^n \rightarrow \mathbb{R}$. If that's not what you mean, I don't understand what you mean.

If you're asking about whether we use 3d or 2d convolutions, then that's a different story than what your post seems to suggest. If we use 3d or 2d convolutions, it's a matter of choice of the programmer (usually). Anyway, please, edit your post to clarify what you really mean by "dimensionality of the convolutional layer".

Are you now asking "What do people refer to when they use the word 'dimensionality' in the context of convolutional layer"? If yes, I would suggest that you remove the sentences "The dimensionality I am referring to is obvious from this context." and "Neither the shape of the input nor the shape of the kernel decides the dimensionality of the convolution operation.". It may also be a good idea to quote someone that used this terminology, i.e. "dimensionality of convolutional layer". The context will probably be very useful to answer that question.

Ok, now, the question is clearer to me, but still not completely clear. Are you asking 1. how we choose between conv1d, conv2d, and conv3d, or 2. what makes, for example, a conv2d and 2d convolution rather than a 1d or 3d convolutional, i.e. why is e.g. a conv2d a convolution in 2d? These questions are related, but different, I guess.

I am asking for 2. Which element/entity decides 1d,2d or 3d. @nbro

The dimensionality used to discuss convolutional layers in CNNs is based on the dimensionality of the input without considering channels.

1D CNNs might process raw audio sources (mono or stereo), text sequences, IR spectrometry from a single sample point
2D CNNs can process photographic images (regardless of colour/depth etc information), audio spectrograms, grid-based board games
3D CNNs can process voxels from Minecraft, image sequences from videos etc

It is often possible to perform signal processing that changes dimensions of signal sources. Whether that adds "channels" or adds a dimension can be a matter of convenience to fit a particular approach. In terms of defining a n-dimensional array, then the addition of channels is just another dimension. In terms of considering signal processing performed in CNNs, we care about the distinction between channels and the rest of the space that the signal exists in.
One way to decide whether something is considered a channel or a CNN layer dimension is whether there is an ordering or metric that consistently separates measurements over that dimension. If a metric such as space, time or frequency applies, then that dimension can be considered part of the "core" dimensionality that defines the problem, whilst a more arbitrary set of features (e.g. each entry in the vector embedding of a word) is more channel-like.
As standard CNN design involves summing over all input channels to create each output feature/channel, which is mathematically the same as increasing the convolution dimension (when the kernel size in that dimension matches to the number of channels), then in practice the convolution operation implemented in a CNN layer of a particular dimensionality can be one dimension size higher. E.g. a layer class labelled "Conv1D" will perform a 2D convolution operation, with the added dimension size matching exactly to the number of input channels. However, conceptually it makes sense to view this as a sum of lower-dimension convolutions, because of the need to exactly match the dimension size. This extra dimension is seen as a convenience for calculation, and not part of the definition.

Kernel dimensionality and presence of filters decides the dimension of convolution operator. N Dimensional Convolutions have N dimensional kernels. For example, from Keras Documentation on 2 Dimensional Convolutions:

kernel_size: An integer or tuple/list of 2 integers, specifying the height and width of the 2D convolution window. Can be a single integer to specify the same value for all spatial dimensions.

If you have more than one filter in the layer, that also adds another dimension to the layer. So, we can say that a 2D convolutional layer is in general 3 dimensional, where 3rd dimension is the number of filters: (k,k,F). For the special case of a single filter, F=1 and we can treat it as 2 dimensional.

Please check this answer

@hanugm Yes, that is the case in the first layer for a color image, a separate kernel used for each color. Still, each kernel is 2 dimensional. And if we have an additional convolutional layer with 16 filter, we still have 2 dimensional kernels, but now 16 of them.

Worth noting that for efficiency purposes a Conv2D layer is very likely running a 3D convolution operation when feeding forward. This is used to sum over the input channels. However, it is still useful to think of this as conceptually a sum of 2D convolutions when looking from the perspective of signal processing from the input.


text:
In anchor based object detection, why don't the anchors share the same weights?
After reading about YOLO V3 and Faster R-CNN, I don't understand why the weights for the regression head aren't the same across all boxes of the same size. Given that the backbone of these systems is fully convolutional, the location of the outputted features should only depend upon the local region of the image which telescops to that feature map. Given that we want the object detector to behave the same way regardless of the object location in the image, shouldn't the weights be the same across anchors of the same size?


text:
AI for games which involve social intelligence. Games like warewolf where players must persuade, charm, threaten etc
I'm looking for any introductory/accessible reading on AI that can play games which involve social intelligence.
Games like poker, where you might bait someone into overcommiting their hand or threaten them into not betting a lot when they should.
Or warewolf, where there is a group of villagers, and a warewolf. Every night the warewolf "eats" a villager, and in the morning all the villagers get together to kick out a person who they think is the wolf. Wolf wins if he's the last one alive, villagers win if they manage to kick out the wolf before everyone dies.
I know there's some online content on the game warewolf, but how does AI tackle such concepts in general? How are the rules represented, and how does the AI "persuade" and be persuaded?

You would probably have to design it. How do humans persuade each other? Chat messages? You probably don't want a whole GPT-2 in your game but maybe you can have some canned messages the AI can say.

thank you for the reply. I have done a bit of reading and I guess what I'm looking for is an entry point into "arumentation theory".
How agents can exchange information to make joint decisions that meet their collective goals, or how they decide if they'll believe argument.
Is there a good entry point into this?

I think it's more of a game design question really. Video game AIs often cheat in order to create the desired level of difficulty. Or for easier difficulties, they kind of un-cheat; they cheat to make their position worse.

Are you asking how AI players cooperate with human players, or how AI players cooperate with each other?

In a typical video game, there would just be some system that decides which AI players co-operate, and that's that. If you're aiming to design an advanced game AI that is even able to co-operate with human players, then maybe the AI doesn't need to know whether other players are AI or human.

I remember in Sid Meier's Alpha Centauri, you can make peace agreements with other AI players, basically just by asking. There is some kind of reputation system so they won't agree to peace if you just captured their city for example. AI players rarely initiate peace offers.

I am asking about how AI cooperate and compete with both human and other AI. Specifically how the AI algorithm encodes the concepts of input from other agents as arguments which may or may not be true, and the AI has to make a estimate of the truth value of the statement.

Probably no different from any other input. A signal from another player that may or may not indicate their intention is no different from a pixel in a picture that may or may not mean it's a cat picture.

Im really sorry for bumping old thread, but searching similar information. Can someone help me?


text:
Book/course recommendation on game theory application to multi-agent system (reinforcement learning)
Is there any great game theory book or course that discusses the application of game theory to modern reinforcement learning or multi-agent systems? Or a classic reference book that can help me get a full understanding of papers like $\alpha$-rank.

"game theory to modern reinforcement learning or multi-agent systems" Why "to"? Do you mean any book, which narrated how the Game Theory evolved to Reinforce Learning?


text:
What is the difference between gradient decent in neural networks and temporal difference in reinforcement learning?
I am studying Q-learning in reinforcement learning. My question is about the Bellman equation.
In Q-learning, the Bellman equation is often introduced as follows.
\begin{align}
Q_{new}(s,a)
&= Q_{old}(s,a) + \text{learning rate} \times \text{error}\\
&= Q_{old}(s,a) + \alpha(\text{target} - \text{actual})\\
&= Q_{old}(s,a) + \alpha((\text{reward} + \text{discount factor} \times \text{max next } Q) - Q_{old}(s,a)) \\
&=Q_{old}(s,a) + \alpha[r(s,a)+\gamma\times max(Q(s',a'))-Q_{old}(s,a)]
\end{align}
The update equation of gradient descent (which is used in the context of neural networks and other fields) is as follows.
$$
w_{new} = w_{old} + \eta\frac{dE}{dw}
$$
So, why the Bellman equation depends on the error after the learning rate while the gradient descent depends on the error gradient?
I feel confused.

I provided an answer to the general question in the title. However, I don't fully understand the specific question in the body of your post. In particular, I don't understand this "Bellman equation depends on the error after the learning rate". What do you mean by "error after the learning rate"? If you're asking whether the difference between the Bellman equation and the GD rule is that one uses the "error" while the other uses the "gradient", then the answer is "yes".

Yes, this is what I mean. I know this is a difference. But why? In other words, why the Bellman equation does not use the gradient instead of error? I think this may be mathematically inappropriate

If you're wondering why Q-learning (or TD-learning) are defined using a Bellman equation that uses the "temporal difference" and why it works at all, you should probably ask a different question in a separate post that doesn't involve gradient descent. It seems to me that you know the main difference between GD and TD learning, although you are asking that question in the title and that's the question that I answered. It seems to me that your doubt is more "Why does Q-learning even work using this type of update rule?".

In this post, you were effectively asking 2 quite distinct questions. Ideally, one post should contain exactly one question, as explained here.

They have a few similarities, but they are quite different. Let me first give you a general description of both approaches/algorithms, so that you start to get a sense of their differences and similarities.
Description
Gradient descent (GD) can be applied to solving any optimization problem where your loss (aka cost or objective) function is differentiable with respect to the parameters that you want to update. For example, if you're training a neural network with gradient descent to solve a classification problem, you could be using a cross-entropy function, which should be differentiable with respect to the weights of the neural network (so you should make sure that all the operations in the neural network, in particular, the activation functions, are differentiable or, at least, you define their derivatives). So, the only restriction to use GD is that your loss function is differentiable, so you could use GD to solve classification, regression, or even reinforcement learning problems (and this has actually be done).
Temporal-difference (TD) learning is a specific approach to reinforcement learning, where you update your current estimate of a value function (in your case, the action, aka state-action, value function) with a value that is the difference between estimates at different time steps (hence the name temporal difference).
How are they different/similar?

They can both be seen as learning algorithms/approaches, although people in other areas other than machine learning may view gradient descent "just" as an optimization algorithm.

TD learning is applied in the specific context of reinforcement learning, while GD is applied to any optimization problem where your cost function is differentiable.

In TD learning (or, more generally, in RL), you want to find a value function (or policy), which could be seen as the parameters that we want to find. (So, here, the parameters are the variables that we want to find). On the other hand, in GD, we want to find the parameters of a model (that define some function or distribution, which could be a value function, but not necessarily).

TD learning can be combined with neural networks (for example, see this paper), which leads to a new field often known as deep RL. In this case, you may use GD to update the parameters of this neural network, which represents the value function or policy. So, GD can be used to solve RL problems

In both cases, we have a learning rate, which determines the magnitude of the changes to the current estimate of the parameters.

You can estimate/approximate gradients (or derivatives) with finite-differences, or finite-differences could be seen as the discrete version of derivatives. In fact, derivatives can be defined as limits of differences. Moreover, if you read e.g. this paper, you will see a lot of gradient symbols. Given that TD uses these "differences", this could be the reason why you're confused.

Thank you very much for this detailed answer


text:
What is the fastest multi-human pose estimation model?
I am trying to find an accurate and fast multi-person human pose estimation that I can train on with custom data. I have been searching for a little while and I may not be up-to-date on the newest techniques.  I will start by posting what I have found and looked into (a little):

Openpose: This is supposedly real-time (I assume on a GPU, 24fps?) and they provide training code
Lightweight OpenPose: Runs in realtime >20fps confirmed, training code is provided
mediapipe: runs in realtime > 20fps confirmed, training code is NOT provided
posenet: No training code, can one even train tfjs models?
movenet: Very fast but no way to train?
hrnet & lightweight-hrnet: seem to be slow? Can anyone confirm? training code provided
blazepose: haven't tried it yet, looks like tf implementation, but no discussion bout speed. Training code included
alphapose: haven't tried it, but looks to run at 16fps (maybe faster?) but is only intended for research not commercial. Training script available.
MoVnect: Looks new, and fast (haven't tested it) but looks like it uses student-teacher training.

What are some other human poses estimation models out there?
I care more about speed and training. I am thinking #2 is my best bet but it's a few years old. And not the most friendly to train Anything newer? Can anyone confirm or reject my findings?

Hello. Could you please clarify this sentence "I care more about speed and training". What do you mean "I care about training"? Also, what do you mean by speed? Do you mean speed of inference (i.e. prediction) or maybe training time? Please, change the title after you have clarified this so that it's also clear what you mean by "speed". I changed the title to reflect more your current post's question/info, but this post is not fully clear to me, as it's not clear what your requirements really are.


text:
Is the noise term $\epsilon$ in $y=g(x) + \epsilon$ used to denote the model's imperfection to the real world?
In supervised machine learning, it is common to say that we learn a function of the form
$$y=g(x) + \epsilon.$$
Generally, $\epsilon$ is used to denote noise or, more precisely, any influence by latent variables such as measurement inaccuracies (right?).
Is it, therefore, correct to say that we use $\epsilon$ to denote the model's imperfection to the real world (caused by anything unknown)?

Yes, precisely.
What do you mention is known in the literature as the Bayes Error. See page top of the page 114 https://www.deeplearningbook.org/contents/ml.html.

The ideal model is an oracle that simply knows the true probability distribution that generates the data. Even such a model will still incur some error on many problems, because there may still be some noise in the distribution. In the case of supervised learning, the mapping from $\mathbf{x}$ to $y$ may be inherently stochastic, or $y$ may be a deterministic function that involves other variables besides those included in $\mathbf{x}$. The error incurred by an oracle making predictions from the true distribution $p(\mathbf{x}, y)$ is called the Bayes error.

Regardless of how clever is your model, the best error you can achieve for the prediction on the data distribution is $\varepsilon$. Note that it holds for the whole data distribution, not a sample of data.
Say, you would like to fit something like $\sin(x) + \varepsilon$. There are 10 points, and one can fit them perfectly with the 9th-degree polynomial, but this an error on training data, and, in case one samples more data points, the error will be likely do exceed the optimal $\varepsilon$.


text:
How does the skip connection match its dimension to the same layer in the expansive path?
According to the U-Net architecture image from the second page of the research paper (URL link) https://arxiv.org/pdf/1505.04597.pdf
How does the skip connection match its dimension to the same layer in the expansive path?

The image looks like the layers sizes (width and height) are the same. Do you mean feature dimension?

Can you please provide more details about your problem? I am familiar with U-net, but maybe not all people, so you may want to briefly describe U-net. What is the expansive path? What do you understand about "skip connections" in this specific context? How are they defined? Have you read the paper? Why do you think those skip connections may not match the dimension of the layer in the expansive path? These are all questions that you should have answered in order to provide us more context about what your problem really is. See https://ai.stackexchange.com/help/how-to-ask

Output of each layer in the upscaling block is of the same size as the input of corresponding convolution layer in the downscaling block after cropping the input's feature maps.
This is how the network is defined. Each conv layer in the downscaling block has a corresponding layer in upscaling block to which the skip connection is made to.Except for the layer in the middle(sometimes called the latent layer). This is the layer that separates the downscaling block from upscaling block as seen in the original paper.
So in short it's just the way network is designed. I doesn't use the whole feature map from corresponding layer in down sampling block.
For a reference you can see Ternaus Net. There they had to crop randomly to support the VGG encoder in the UNet structure.


text:
Are the two policies in SARSA for choosing an action the same?
Here is the pseudocode for SARSA (which I took from here)

Are the two policies in SARSA for choosing an action equal? I guess yes, because it is called an on-policy learning algorithm. But could I, for example, also use different policies in this SARSA framework? For example an $\epsilon$-greedy policy and a softmax policy. Maybe the resulting algorithm would not be called SARSA anymore but it would be something similar.

I have deleted my answer as it caused too many follow-up questions, thus did not seem to be helping you. I highly recommend reading the book Reinforcement Learning: An Introduction, which covers the rationale behind basic value-based methods very well: http://incompleteideas.net/book/RLbook2020.pdf

@NeilSlater I think you should undelete your answer. It answers the question. If the OP is not able to understand it, it's a different matter.

@nbro: OK, I have undeleted the question.

For learning, it doesn't matter much how you choose the first action before starting the main loop. That is because the agent doesn't need to learn about transitions to the first state of an episode.
The thing that does matter is that the first action choice should cover all possible actions with probability greater than zero, in order to guarantee convergence. Conceptually, this is not much different to using exploring starts.
However, the usual practice is to have just one active policy for SARSA, typically $\epsilon$-greedy based on Q values learned so far. It is worth noting here, that as Q values change during learning - which happens on each time step - then the policy may also change (when the greedy action choice changes). So even when you use the same rules to derive the policy in SARSA, the actual policy used may vary, even in the middle of the loop. In that respect, the SARSA algorithm uses many policies, but typically only one approach to determining the current policy.
If you are using function approximation and also used a very different rule for the policy for the first action, it is possible you could affect the function approximator through sample bias (your training data has different distribution to target data). This is tricky to put a number on in RL, but is usually ignored in off-policy approximation, so should not put you off if you want to try out ideas of using a different first time step policy.

Thanks Neil Slater. Unfortunately I have to admit that I have huge difficulties understanding it and I have several follow-up questions. 1) Why does  "For learning, it doesn't matter much how you choose the first action before" ? Are you here refering to the first action of the whole algorithm or the first action selection in every step as we need to select 2 actions in SARSA in every step 2) Why does the policy in SARSA change when the Q-values are changing? 3) What do you mean by "So even when you use the same rules to derive the policy in SARSA"? What kind of rules are you talking about?

What do you mean by "[SARSA uses] typically only one approach to determining the current policy."? What do you mean by approach here? What is the difference between a policy, rules (Question 3) and approach here? 5) You write "the usual practice is to have just one active policy for SARSA". But is SARSA not an off-policy learning algorithm. So this should mean that it has to use at least 2 policies otherwise it would be on-policy learning as far as I see it.

@PeterBe: I cannot answer your follow-up questions, sorry. You may find it helpful to review basic concepts in RL. I can recommend the book Sutton & Barto, Reinforcement Learning: An Introduction. You can get a copy at http://incompleteideas.net/book/RLbook2020.pdf and I think reading the first 4 or 5 chapters should answer your questions and be a lot more productive for you than more Q&A.

Do you know a comprehensive example (other than the book) where it is shown how the algorithm actually works. Any maybe you can try to answer the 2) and the 4) because they confused me most