text:
Does AlphaGo play random moves in a real competition?
Alphago and AlphaGo zero use random play to generate data and use the data to train DNN. "Random play" means that there is a positive probability for AlphaGo to play some suboptimal moves based on the current DNN; this is for exploring and learning purposes (please correct me if my understanding is wrong).
In the real tournament, does AlphaGo still play the random moves? Is the random play feature only used in the training phase?
If AlphaGo does not play a random move in the real competition, then I think AlphaGo is not learning in that competition. Human players do similar "random play": they usually play some random moves or strange moves in minor contests, just to test out new strategies; in major tournaments, they will be more serious and play less unprepared moves.
So, a related and broader question is: does AlphaGo learn from the game it is playing with the human in real-time?
I think the second question is less important because AlphaGo's learning curve is extremely flat compared to humans: AlphaGo learns epsilon from one single game while human can learn a lot.

Please, ask only one question per post, even if they are related!

Question 1: I don't think they ran AlphaGo or AlphaGoZero in training mode during tournament matches because the computing power required for this is really large. I don't recall if this is described in the documentary but see this quote from the AlphaZero paper (page 4):

using 5,000 first-generation TPUs (15) to generate self-play games and 64 second-generation TPUs to train the neural networks.

Question 2: From my first answer, I assume the network is run in inference mode, i.e., taking the observation of the environment as input and outputting the action (next move) according to the strongest strategy of the network. Then the AlphaGo does not learn anything as it does not evaluate whether the taken action is better than the strategy because these are the same. The way that the network learns during training/exploration is by comparing the random action that is taken to the best action known by the network.

You are correct in your understanding of self-play and exploration. This is done in RL to explore actions that might not have been taken yet to explore actions that might be better than the best-known strategy.
But during inference, you would want to greedily take the best-known actions because an action taken at random would most likely be way worse.

I think you are right for 1 and 2, that the NN was not being trained. However, my understanding is that MCTS was still used to improve on the NN policies, so it is not as cut and dry as this answer implies.

That is true and an important note which I did not consider. MCTS is used in AlphaGo and AlphaZero and described in the respective papers. Although its use is quite limited compared to traditional alpha-beta search engines, such as the Stockfish chess engine.

Many thanks for your contribution! So in the "inference mode", AlphaGo will only choose the "best move"? What if there are two equally best move with the same value?

To my knowledge, I don't think there can be two moves with equal value unless your discount factor is 0, or close to it. With a discount factor of 0, the agent will not consider the value of the future moves which can be determined from the immediate action the agent is about to take. But hypothetically speaking, it would probably be a random selection between actions of equal value.

@Lars Note that the Go board is symmetric so there are some scenarios that two or more moves are absolutely symmetric. If those symmetric moves are not of same value, then the algorithm has a problem. So, in my humble opinion, my question still hold: when two moves have exactly the same value, which move will be chosen?

Actually, I did not consider the symmetry property. You are correct that when the board and stones are symmetrical then two or more moves can have identical values. Moreover, because the value is the same, the possible actions in the future will also be the same regardless of which move is selected. So if they have the highest value of all available moves then it does not matter which of the moves is selected.

@GPA: for ML purposes, you want to parameterize your scoring method using the minimal amount of trainable weights. A scoring algorithm that is asymmetric uses more weights than a symmetric one, so it's definitely worse.

AlphaGo/AlphaZero has 3 main sources of randomness during competitive mode:

Move temperature: The MCTS process outputs a probability distribution P over all candidate moves. The agent chooses a move by sampling from the probability distribution proportional to P^(1/T) for some move temperature T. Typical configurations start T at some value near 1 and decay it quickly to 0 over the course of the game (the limit of P^(1/T) as T approaches 0 is the argmax function). This mechanism helps to ensure, for example, that the agent will not always play the same first move as first player.

Symmetries: The game of go has 8-fold symmetry. Before querying the neural network, the agent randomly chooses one of the 8 symmetry transforms of the game and applies it to the board. It then applies the inverse transform to the neural network’s (policy) output. A perfect neural network should exhibit perfect 8-fold symmetry, but in reality the neural network is imperfect and exhibits asymmetrical biases (e.g., it wants to play an atari in board state S, but if you rotate S by 90 degrees, it no longer wants to play an atari). So this operation leads to random deviations in its move choices.

Tree parallelization: MCTS is made multithreaded via a mechanism known as tree parallelization. The move distribution that MCTS outputs is dependent on the order in which the different threads interleave with each other, which is effectively random.

It is worth noting that any human could easily defeat any deterministic AI in any deterministic game. Simply pit two instances of the AI against each other, memorize the deterministic sequence of moves, and then use those exact same moves against a single copy of the AI. Because of this possibility, game playing agents typically are willing to sacrifice a small amount of playing strength for randomness. Move temperature in particular can be justified for this reason. The other two sources of randomness can be seen as byproducts of engineering optimizations.
In other answers/comments, others have correctly noted that single-threaded MCTS is a deterministic algorithm (if executed purely on CPU). Given a game state, it will always output the same move distribution, absent the symmetry mechanism described above.

The core mechanics of AlphaZero during selfplay and real tournament games are the same: something similar to Monte Carlo Tree Search is done but guided by the current neural network instead of random simulations. The network is only doing inference, it's not learning during a tree search. There's a great summary diagram here.
The differences between selfplay and tournaments are:

Selfplay tree searches games get some additional (Dirichlet) noise injected to encourage exploration of different moves and positions. This does not happen during tournament games since that would likely make play weaker.

During selfplay moves are selected randomly, but weighed according to how good the tree search thinks certain moves are. This means the entire training process spends most of its time looking at good moves, but still occasionally plays bad moves to keep some variety. During tournaments AlphaZero just plays what it thinks is the best move.

Selfplay games are kept in a large buffer with millions of positions, on which the next neural network is trained. This is the only point at which actual network training is happening.

Selfplay tree searches only look at ~800 nodes per position, which probably only takes a few milliseconds to calculate. Tournament games take a lot longer, 3h per game + 15s per move. This means the tree search visits hundreds of thousands, likely millions of positions for each move, making tournament play a lot stronger than selfplay.

These are the only differences between selfplay and tournament games. One thing to note is that after selfplay/training is done, AlphaZero is just an immutable neural network with some tree search code around it. There is no learning happening during a tournament, or any additional randomness injected into the process. Instead, to ensure a diversity of games, an Opening Book is used.

I think MCTS is inherently random, unless you have some statement about that being fixed in some way by Deep Mind for tournament play?

@NeilSlater Pure MCTS is indeed random, since it uses random simulations as the evaluation function. AlphaZero instead uses one of the outputs of the neural network as the value estimate. This makes AlphaZero with a fixed node budget fully deterministic, except for maybe some implementation details (multithreading, GPU batch processing, ...).

MCTS also makes random choices earlier than rollout (Selection and Expansion phases). I don't think AlphaZero's truncated rollout fully removes randomness.

@NeilSlater There is some varierty in MCTS details, but the vanilla one also has deterministic selection and expansion (which is not actually a distinct step IMO). Regardless, from the AlphaGo paper, in the description of figure 2, it's pretty clear that selection is deterministically based only on previous neural network outputs.


text:
What does it mean by "lazy mean" here?
Consider the following paragraph, taken from 3.4: Named Tensors of the textbook named Deep Learning with PyTorch by Eli Stevens et al., regarding the calculation of the mean for RGB channels of an RGB image in order to convert it into a grayscale image

So sometimes the RGB channels are in dimension 0, and sometimes they
are in dimension 1. But we can generalize by counting from the end:
they are always in dimension –3, the third from the end. The lazy,
unweighted mean can thus be written as follows:
# In[4]:
img_gray_naive = img_t.mean(-3)
batch_gray_naive = batch_t.mean(-3)
img_gray_naive.shape, batch_gray_naive.shape

# Out[4]:
(torch.Size([5, 5]), torch.Size([2, 5, 5]))

Here img_t is a single image tensor of shape (3, 5, 5) and batch_t is a batch of 2 images with the same shape. They are taking the mean over the channel dimension.
Since they are taking the normal average of the intensity values over RGB channels, it is unweighted. But, what does it mean by lazy here? Are they referring to some implementation detail regarding lazy initialization? If yes, what exactly does it mean?

Looking at the variable names in the code and considering the context, it seems to me that the author is using the word "Lazy" to describe the approach.  I believe the author actually means that a naïve approach to producing a grayscale image is being used.
If you want to use a less-naïve approach, you might consider producing the dot product of the 3 channel color depth by a vector such as [0.2126, 0.7153, 0.0722] which will produce something a bit more accurate.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Why are Directed Graphical Models considered ML methods?
Consider the following problem. The probability of being born in countries [1,2,3,4] is given by [a, b, c, d] respectively. This is a categorical problem.
Now, assume that the height of a person belonging to any country is normally distributed.
The task is to sample from the total distribution of countries and heights,
$
p(c,h) = p(c)p(h|c)
$
One can use a mixed distribution as a tool to model these scenarios, which are a special type of DGM.
Can someone please tell me where ML comes into play here, and how this is considered an ML technique?

Generative models like latent variable models (e.g. VAE) use directed graphical models and these sort of factorizations as a foundation for learning. In VAEs, Neural nets are used to estimate posteriors/priors to generate samples.
This sort of explicit factorization is helpful in other generative models as well like autoregressive models which are basically operationalizing the chain rule of probability or bayesian networks which may be more explicit in modeling joint distributions of interest.
I would agree the use of ML often overlaps somewhat confusingly/incorrectly in these contexts, but these factorizations often help in formalizing ML problems and can play a more direct role in (deep) generative modeling.

Thanks sma! I am wondering specifically about the problem I gave, say it's a generative model just for generating data, but nothing else. No Neural networks, no deep learning. I am struggling to see where the 'learning' is specially if one were to make a generative model based on the total probability I provide above.

I would rather say that models, both generative and discriminative ones, can be modelled as Bayesian networks, rather than saying "Generative models like latent variable models (e.g. VAE) use directed graphical models". Also, what do you mean by "autoregressive models" specifically? I don't think you mean AR(0), for instance. So, because it's not clear whic autoregressive models you're referring to, then this sentence "which are basically operationalizing the chain rule of probability or bayesian networks which may be more explicit in modeling joint distributions of interest" is also unclear.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How to prove that "w will converge to TD fixed point once A is positive definite"
In Reinforcement Learning: An Introduction 2nd edition  section 9.4 (p. 206), it says that when we use TD(0) as target and use semi-gradient method to update :

In general, $w_t$ will be reduced toward zero whenever A is positive definite, meaning >$y^TAy > 0$ for any real vector y $\neq$ 0. Positive definiteness also ensures that the inverse A exists

How to prove this statement? I tried to prove it in the following way, but I can only get  $\lambda <1$ rather than $|\lambda|<1$.
$$
w_{t+1} = (I-\alpha A)w_t + \alpha b \\
\because\delta_{w+1} = (I-\alpha A)\delta_{w}\therefore|\lambda_{max}(I-\alpha A)|\lt 1\Rightarrow\lim_{t\rightarrow\infty}\delta_{w_t}= 0 \\
(I-\alpha A)y= \lambda y\Rightarrow y^T(I-\alpha A)y= y^Ty-\alpha y^TAy = \lambda y^Ty \\
\Rightarrow 1-\lambda = \alpha\frac{y^TAy}{y^Ty}\gt 0\Rightarrow\lambda\lt 1
$$

Are you sure it is in section 9.4 (pages 241-247)? I cannot seem to find the statement there.

The link you post seems to be the first edition of the book. The statement I mentioned can be found in second edition,  p206, 9.4 linear method

You are right, my previous link was to the "Second edition in progress". I have updated the link to the final 2nd version - could you please accept the edit?


text:
Does $(\langle w, x \rangle + b) = ||x - x'||$ hold?
Currently, I am trying to understand the mathematics of SVM's using the textbook 'Mathematics for Machine Learning' by Deisenrot et. al.
On page 375, they define the distance between a point $x$ and the separating hyperplane as the length of the distance vector $||x - x'||$, where $x'$ corresponds to the orthogonal projection of x onto the hyperplane.
Subsequently, they formulate the following condition to express that any point's distance from the hyperplane should be greater than or equal to some constant r:
$$y(\langle w, x \rangle + b) \geq r$$

$y ∈ {0, 1}$ representing the class of $x$
$w$ normal vector and $b$ the bias.

So my question is: Does $(\langle w, x \rangle + b) = ||x - x'||$ hold?
If that's the case, I would be thankful for some hints on how both sides of the equation are related.

I think that your missing the fact that $r$ is not a random constant, but precisely the minimum distance we want to ensure between each point and the separation hyperplane (called margin). And indeed in the book they use $r$ to define the distance from a generic vector $a$ to the hyperplane as well:
$a = a' + r\frac{w}{||w||}$
Since $\frac{w}{||w||}$ is a normalized vector, we can consider $a'$ as the  x coordinate of $a$ in the coordinates system defined by $w$, vector orthogonal to the hyperplane and the hyperplane itself, whereas $r$ represent the y coordinate (see drawing below, apologies for the awful quality). To be able to separate 2 points $x_1$ and $x_2$ into 2 different classes, we want them to fall into different sides of the hyperplane. By imposing
$y(\langle w, a \rangle + b) \geq r$
we're also saying that we don't want the points to just fall into different sides of the hyperplane, but they're y coordinate should also be bigger or equal to r (in absolute value).
So, to answer your question, $(\langle a x \rangle + b) = ||x - x'||$ is not really a correct statement but for a bunch of points instead will hold  $y(\langle w, a \rangle + b) = x - x'$ (i.e. the y coordinate of the point we want to classify express in the coordinates system of the hyperplane is equal to r, which means the point lies in the margin of the hyperplane).


text:
Which algorithms work in a non-stationary stochastic environment?
Currently, I am reading into the Multi-Armed-Bandit problem and found the special case of non-stationary (environment and its attributes, like the reward distribution, change over time) stochastic environments. Since this is an adversarial MAB problem, no context is available at the moment. I've read that $\epsilon$-greedy, Exp3 and the FPL algorithm work. But some tutorials like TensorFlow ones use LinUCB and other algorithms, which are never mentioned in any papers.
So, my question is, basically: which algorithms work on the non-stationary stochastic environments?

What do you mean by "LinUCB and other algorithms, which are never mentioned in any papers."? I've seen LinUCB being mentioned in several papers that I've come across on bandits. So, which papers do you have in mind? Do you mean papers related to non-stationary environments? Also, can you clarify this "Since this is an adversarial MAB problem, no context is available at the moment."? Are you referring to the "context" as in "contextual bandits"? I don't see the connection between being "an adversarial MAB problem" and not having access to "context" (whatever you mean by this).

Hi, yes I've been unclear in this points. By "LinUCB and other algorithms, which are never mentioned in any papers." I do mean papers related to non-stationary environments. And by "no context" I mean the base case of the MAB problem, so without any context like in contextual bandits. I just want to make the example simpler to understand it better. Thats why I am excluding the context. I hope this made it clearer.

Thanks for the clarifications. Can you please include this info directly into your post? I would also recommend that you provide the link to 1-2 papers that you came across on non-stationary environments.


text:
What to look out for when designing an environment regarding observations?
When designing an environment, what should one look out for when designing the observation space to make the environment as easy to be learnable for an agent as possible?
E.g. make sure the markov property is fulfilled if possible, but I mean also more specific details like the coding of observation channels: Try avoiding continuous values, try to keep the number of categories for categorical observations small, or try to represent continuous values in smaller/larger intervals or consider discretizing continuous values into ordinal categorical values.
I read a paper presenting recommendations for the design of the action space, regarding several video games RL studies (Action Space Shaping in Deep Reinforcement Learning, 2020). I wondered whether there are recommendations concerning the observation space as well.


text:
Is a Transformer a good choice for multivariate signal classification?
I am working on a problem regarding the multi-classification of multivariate time signals. So I have multiple signals and try to train an algorithm on them. My current approach is to build a neural network with LSTM-layers and it works pretty fine.
I have read that LSTMs are pretty outdated because of the transformer architectures. I found some papers about the idea to use them for signal classification (see: https://arxiv.org/abs/2103.14438). There was an example on the TensorFlow page regarding univariate signal classification (see https://keras.io/examples/timeseries/timeseries_transformer_classification/). I think it is rather a research question than a common approach for this type of problem.
To my questions:

Would you recommend implementing a Transformer for this type of problem? Do you think, it is a more "state of the art" approach?

Do you know some example projects?

Classifying multivariate signals is a difficult task that generally requires more sophisticated methods than the standard transformer. However, if the transformer is able to accurately learn the relationships between the different variables in the signal, it can be an effective classifier.

Definitely yes. I would say Transformers would work wherever LSTM works and even better. The reason is that they can attend to longer sequences as an input. In contrast to transformers, in LSTM for example after some sequence length(in your case the signal length) it would lose performance.

No, I don't specifically know projects that are in this domain.But you could have a look at the architecture called Conformer (https://arxiv.org/abs/2005.08100) (I think they work on audio signals)

Thank you for your answer! 1)  I am aware of the fact, that LSTM have their benefits. Do you have any ressources (paper,benchmarks etc.) which can confirm your statement? I just feel confirmed that it is not a common approach. Maybe I should try it and publish an paper on it ;)

Your welcome! :) I don't remember on top of my head which paper I can show you. But the reasoning is in LSTM because things are recurrent, you need the previous input for calculation and if you do this over a long sequence this leads to vanishing gradients problem therefore decreasing the learning potential. However, because transformer attends to the sentence or input sequence at once it can handle longer sequences better. I hope this helps, if I find a benchmark on this I will share for sure. https://medium.com/mlearning-ai/transformer-implementation-for-time-series-forecasting-a9db2db5c820


text:
Why does the schema theorem of genetic algorithms hold?
I have been reading about the Schema Theorem - one of the first theorems from the field of evolutionary computing and genetic algorithms, largely responsible for justifying the use of genetic algorithms to solve optimization problems when the derivative of the objective function is either unknown or not clearly defined.
Genetic algorithms can be used in many different types of optimization problems, such as finding the roots of a polynomial. For example, we could use the genetic algorithm to find the roots (the zeros) of the following polynomial (this polynomial will be referred to as the objective function, i.e. the objective of the optimization/root finding):
$$f(x, y) = x \sin(4x) + 1.1 y \sin(2y)$$
Essentially, the genetic algorithm would start by taking random values of $x$ and $y$ and record the corresponding value of $f(x,y)$ for each random combination of $x$ and $y$: the value of $f(x,y)$ for a given combination of $x$ and $y$ is called the "fitness". The genetic algorithm works by taking many such random combinations of $x$ and $y$ and recording which combinations produce lower fitness values (i.e. which coordinates of $x$ and $y$ correspond to low elevation regions on the $f(x,y)$ surface). The genetic algorithm then combines (i.e. mutates) combinations of $x$ and $y$ that produced low fitness values, and re-evaluates $f(x,y)$ at these new mutated combinations of $x$ and $y$. The genetic algorithm repeats this mutation process many times until it successive differences in $f(x,y)$ are negligible, or after a predefined number of iterations - this is very useful in problems where evaluating the derivative of the objective function can be extremely time-consuming or costly, or the derivative of the objective function is poorly defined (i.e. standard optimization algorithms like gradient descent and newton-raphson can not be performed).
An interesting observation about the genetic algorithm is that it does not promise or guarantee convergence to the true minimum point of the objective function during the optimization process. But what the genetic algorithm does guarantee is that the overall fitness of the objective function is guaranteed to improve as the number of iterations increases. This idea is expressed in one of the original and fundamental theorems of genetic algorithms, called the Schema Theorem.
Apparently, this theorem states that "schemas" (e.g. combinations of $x$ and $y$) that produce better fitness values are more likely to survive after mutation - i.e. the expected number of "schemas" that produce better fitness values are likely to increase as the number of iterations increase.
Question: I have spent a long time trying to find a derivation of proof of the Schema Theorem - something which attempts to mathematically explain why the genetic algorithm is guaranteed to move towards the true optimum of the objection function as the number of iterations increase (rhetorical question: after all, why does the "genetic algorithm" not result in worse fitness results?). After all:
$$
\mathrm{E}(m(H, t+1)) \geq \frac{m(H, t) f(H)}{a_{t}}[1-p]
$$
Why does this inequality hold?
I am very curious to see if there is a mathematical proof that justifies one of the most important and earliest theorems in "genetic algorithms" and evolutionary computing; does the Schema Theorem have a mathematical proof?

Yeah, the schema theorem is rubbish. If you want to guarantee increasing fitness, just use elitism.

I am not exactly following. The term "theorem" has a very specific and unambiguous meaning in mathematics; if there was not a proof, we would call it Schema Axiom, or Schema Principle, or Schema Conjecture, or - you get the idea. I could understand a question "where can I find the proof of the Schema Theorem", but if it exists? In any case, the obvious starting point for a proof would be Holland's Adaptation in Natural and Artificial Systems; have you consulted it?

A quick Google search revealed this slide deck, which includes a section titled "Proving the Schema Theorem".

I’m voting to close this question because asking if a mathematical theorem has a proof does not make any sense (the answer is trivially "yes"). If the question is about producing or locating a/the proof, please edit accordingly.

@desertnaut Clearly, the OP doesn't know or clarify that what we usually call "theorem" in math has a proof and I think it's implicit that the OP isn't just looking for "yes, this theorem has a proof", but would also like someone to point him/her to the proof. In fact, the question starts with "I have spent a long time trying to find a derivation of proof of the Schema Theorem" and then asks "Why does this inequality hold?". So, my interpretation of this question is: "Why does the inequality hold, i.e. what's the intuition behind the proof of the schema theorem? Where can I find it?".

The original post even contained more questions, i.e. "Why do genetic algorithms work?". I think the core of this post is "Why is this inequality true? So, what's the proof of this schema theorem or inequality? "

@nbro Arguably, the question is adequately summarized in its last sentence: "does the Schema Theorem have a mathematical proof?", which arguably would also answer their previous question "Why does this inequality hold?". My simple argument is that, as is, this is an ill-posed question, trivially answered in the positive. I am afraid that the OP never answers "What is the proof", implicitly assuming a possibility that such a proof may not even exist, and this is not justified (the fact that I was able to locate such a proof after googling for 5' is another story, but not irrelevant).

@nbro notice again: "I am very curious to see if there is a mathematical proof [...]" - again, completely unjustified. If they wanted to ask "Where can I find the proof", they could (and should) do so in just a couple of lines, without raising all that dust and implying that such a proof (of a theorem) may not even exist.

@desertnaut In this worst case, someone that answers this question will just give a "yes" or "no" answer. In the best case, they will also explain the proof and provide and/or provide a link to the proof. I think the post is ambiguous in a way, but, as I said, to me the OP wants to understand that inequality, not only because that question is in bold but also because the original post contained the title "Why do GAs work?". I agree that this post contained multiple questions and it might not be clear which one is the main one.

For me, it's obvious that the main one is not "does the schema theorem have a proof?", but "what is the proof?". That's obvious because it's implied by the question "Why does this inequality hold?" The OP will eventually clarify what the main question is after having seen these comments.

@desertnaut In any case, I think this is one of the most fundamental questions that could have been asked here. So, voting to close this question for me is not the right thing to do, as the main question to me is obvious. There are examples of many other questions that should have been closed, if you want to vote to close them. For example, this.

@nbro if the main question is obvious to you, then kindly edit the question to remove any ambiguities (as I have explicitly suggested in my close vote). I can't see how "this is one of the most fundamental questions that could have been asked here" - anyone understanding the meaning of the term "theorem" is aware that there is a proof, and it is findable, for anyone really interested.

And bold in itself says nothing; the last sentence was also in bold, which I edited out myself.

Let us continue this discussion in chat.


text:
Does reaching the global optima guarantee good performance in a task?
It is to my understanding that, in deep learning, we are essentially trying to minimize the loss function that we have defined and reach its global optima through some form of optimization technique. However, with AI being applied to so many fields, can we be sure that there exists a loss function for every task, such that upon reaching its global optima, it guarantees reasonably good/human-level performance in the task?

As a more direct answer to your question as stated in the question title, finding a global minimum means we have found the optimal solution for that model.  This doesn't mean that the model necessarily works particularly well.  Hence the saying from George Box, "All models are wrong, some models are useful."
In other words, while you may have found an optimal state for that model, the model may not be optimal or even acceptable for the task.  (Note that that phrase can be interpreted in different ways, and this is not the only way to apply it.)

There is an existence of a loss/reward function for any task that can be evaluated, but this does not mean that function generalizes to any model.
In your prompt you mention "human-level performance", this assumes a metric such as accuracy, auc, precision, winning %, etc that humans were evaluated on for some task. Since it is possible to model any function with a boundless resource budget we can then model a function that reaches any form of theoretical optimum for that metric and some given domain.
Ofcourse there is issues with what I wrote above in an applicable sense:

Infinite resource budget doesn't exist.
Not all metrics can be easily calculated.
We usually subset a domain to "train", but to find one that truly encompasses a domain is usually not feasible and requires clever data usage and handling.

So in practice we make simplifying assumptions that handle the above issues. Examples include finite-parameter deep learning models designed for efficient learning, differential metrics such as distance, and data augmentation. In doing so, it is possible to achieve cases where the global min/max over its parameter space performs poorly. Its the researcher/engineers job though to make this not the case.

Thank you for your answer. I would like to clarify on a few points:

this does not mean that function generalizes to any model - Do you mean that not all models may be suited for this function?

possible to model any function - Can everything be expressed as a function?

To give an example, I'm working on a navigation task, the network is learning a function that maps from image to action, but can we be sure that such a function exist, one that generates the correct action for all images?

adding to it outliers would be hard to determine as you club number of task together  which could affect all your performance metrics


text:
Is logic AI a complement to learning AI?
I want to know the relation between logic AI and learning AI.
Logic AI here refers to the branch of AI that is based on mathematical logic. Learning AI refers to the branch of AI that is based on learning from data. The data can be in the form of either labelled dataset, unlabelled dataset, experience, etc.
Can we show a Venn diagram with the universal set as the complete spectrum of AI, while logic AI and learning AI as subsets in it?

What you refer to as logic AI is a subset of what is called symbolic AI, as you manipulate symbols, according to certain rules (which could be rules of logic). These rules are either authored by a human being, or they can be learned from examples. There are algorithms to derive decision trees (eg ID3) or other rule sets from training data. But the important aspect is that the 'intelligence' or 'knowledge' of the system is expressed in such rules and (usually specific) algorithms. Symbolic AI systems are generally designed for a particular purpose, and it can be difficult to use them for other tasks (eg language analysis vs. natural language generation)
Most learning AI nowadays operates on the sub-symbolic level: instead of manipulating symbols with a particular meaning (eg words in a human language) they operate on numbers. Symbols are mapped onto numbers (eg through embeddings, ie vectors of co-occurrence information). The algorithms used are general purpose: a neural network doesn't really care what it is used for, it simply learns to associate numerical input with numerical output. There is no human involvement apart from preparing and providing training data (and perhaps choosing a particular set of basic parameters/architecture), and the 'knowledge' is the model derived through training, basically a large blob of numerical values.
At the interface between the two parts is the mapping between symbols and their numerical representation.
As a diagram, imagine a two-way partition between symbolic and sub-symbolic AI, and a further set ('learning') which encompasses most of the sub-symbolic part and some (but not much) of the symbolic part.


text:
Are there any works that deal with 2D pose estimation in videos?
Since pose estimation is often a task where spatial-temporal context should be helpful in finding subsequent key points, I thought there should be many papers on it. However, I could not find any work that deals with 2D pose estimation in videos.
Am I missing something, or am I just hindered by a large number of papers on 3D position estimation?

Update:
I misread the question; It seems like OP is more interested in pose tracking. So, I'll have to point OP to papers on that, like this one. Using multiple frames becomes especially important when there are multiple people in the frame, and it's desired to track which pose belongs to which person.
For more papers on pose tracking, look here.

Reading this immediately reminded me of OpenPose. The 2D poses are based on this paper; They also refer to a few other papers and code repositories. There are several other 2D pose models at ModelZoo as well. They do refer to the papers they are based off of (or, at least, the ones I have seen do).
Later work focuses on extracting even more 3D data from 2D images like this one by Facebook. You can find much more on Papers with Code.

I have already known OpenPose, but it does not seem use spatio-temporal data.

@BertGayus I have updated the answer. Am I right in guessing that you are interested in human pose tracking?

I don't think there is a great difference between pose estimation in pictures and in videos.
Do you know MediaPipe? https://google.github.io/mediapipe/
MediaPipe does perform an tracking from the keypoints through time. So the temporal information is used. Apart from that it is the same problem as the estimation from keypoints in static images.

I think that the estimation of body position in images and videos should be different, because the spatio-temporal information in videos is an important aspect. For example, it is unlikely that my wrist at time t is not somewhere between its position in t-1 and t+1.

This is the reason why tracking is used. Do you know Kalman-Filters? The output of the Neural Network is used as an measurement for the Kalman-Filter. Kalman-Filter perform a typ of error-correction. MediaPipe for example takes all frames into account.


text:
What kind of features does each node have as an input graph to a graph neural network?
What kind of features does each node have as an input graph to a graph neural network? For example, we want to do image classification with GNN, what are the features of each pixel? Or if anyone could send me a link to implementing GNN on an example I would greatly appreciate it.

Applying GNNs to images can be realized in different ways. If you only substitute a visual ConvNet by a GNN, then the pixel values would be the same as what goes into a ConvNet. The only difference will be the input format. Whereas a ConvNet gets inputs of shape [batch, height, width, channels], a GNN would receive a list of pixels [batch, n_pixels, channels] together with an adjacency list that contains the structural information (Each pixel may have 8 neighbors and a selfloop except for the pixels at the boarders).
So basically the features doesn't change, what changes is the format of the data. For a clean library of GNNs you can checkout DeepGraphLibrary, they have tutorials on how to use their library. If you want to see a simplified standalone implementation, you can checkout this code. His repository also contains more nice implementations.


text:
What is the name of a feature space which has consistant distance-related properties?
What is the word describing a feature space where distance between two elements has a decisive informational value, whatever the pair of elements is?
For example if a model creates embeddings for words, and we take all possible pairs of embeddings and compute their distances, and it is possible to set a fixed threshold distance, where every pair whose distance is inferior to the threshold are synonyms, and every pair with distance superior to the threshold aren't. Is there an adjective that qualify a given feature space that has this property?

A vector space can be named based on the properties of functions that can be defined in it.
If a vector space is structured enough to contain a metric (e.g. euclidean distance or generic inner product) then we can call that space a Metric Space.
In the case of cosine similarity, we can't talk about metric space since cosine similarity is not a metric (it doesn't respect the triangular inequality), but it is a valid dissimilarity function, so you could refer to that vector space as a Dissimilarity Space.

I am sorry, but my question is more constrained than that. Given a metric space, what is the name of the subset of metric spaces where the given distance metric has the property such as described in the original question ? The second paragraph is not just a vague description, every word of it counts.

well, again it really depends on the function you define. What you describe sounds to me like a dissimilarity function. So if you can provide a rigorous definition of it and prove that it satisfies the properties of a dissimilarity function (i.e. the more dissimilar two words are the higher the value returned by the function is) then you can call that space a dissimilarity space. Otherwise the description of such space is too generic to associate it to a specific nomenclature.

I will try to rephrase it. One way to use a feature space equipped with a distance metric is to take a specific element A, and sort every other element by distance to this element. You would obtain for example the distance list [0.1, 2, 5, 7,...]. Then you would do the same thing with another element B and get the list [10, 20, 30, 70,...]. A feature space that would NOT have the property i am talking about would be if the closest element to element A is of the same identity, AND the the closest element to element B is of the same identity. ie, you cant set a decisive distance threshold.


text:
How does OpenAI-ES use Adam?
I just read that OpenAI's ES uses Adam: "OpenAI’s ES is denoted as “OptimES” (since it uses Adam optimizer)"?? I verified they are correct using the link they posted, (see es_distributed/Optimizers.py). But I don't understand how because the paper says they are using Evolution Strategies as a Scalable Alternative to Reinforcement Learning, which is black box optimization (like most ES's)... So how on earth is Adam used, given this is black box?

OpenAI's ES updates parameters using learning rate. It may remain constant, decaying or changed in different ways. You can for example use momentum that's commonly used in stochastic gradient descent. So Adam is used to adjust learning rates per parameters.

Oh that's interesting. Just checking though you're sure about this?
Could you add some citations?

Not citations per se, but look at the code in github from Visual Guide to Evolution Strategies.


text:
When to model decision-making problem as single agent vs multi-agent problem?
I understand the goals and purposes of RL in the case of a single agent and the underlying model, i.e. MDPs, for RL problems (or sequential decision making with uncertainty in general).
My question is (and I know this will/may be subjective) are the indicators for choosing to model some decision making problem as a single agent, treating all other factors/noise as part of the environment (MDP or some variant of it) vs multiple agents (a stochastic/Markov game)?
In the zero-sum/adversarial or pure cooperation setting where the goals of the agents conflict/assist each other, it is obvious that the multi-agent setting is the way to go. But suppose that there is no pure conflict/coordination.

Note that "Markov games" are already stochastic, so saying "stochastic Markov game" is a bit redundant. Markov games are sometimes called "stochastic games", so I changed "stochastic Markov game" to "stochastic/Markov game" in your post (because maybe that's what you meant originally anyway).

Thank @nbro, I understand that Markov games are stochastic but not all stochastic games are Markovian. I was referring to both cases where the Markov property may or may not apply. I should think there may be tricks for making general stochastic games (approximately) Markovian tho.


text:
What are the benefits of using spectral k-means over simple k-means?
I have understood why k-means can get stuck in local minima.
Now, I am curious to know how the spectral k-means helps to avoid this local minima problem.
According to this paper A tutorial on Spectral, The spectral algorithm goes in the following way

Project data into  $R^n$  matrix

Define an Affinity matrix A , using a Gaussian Kernel K or an
Adjacency matrix

Construct the Graph Laplacian from A (i.e. decide on a
normalization)

Solve the Eigenvalue problem

Select k eigenvectors corresponding to the k lowest (or highest)
eigenvalues to define a k-dimensional subspace

Form clusters in this subspace using k-means

In step 6, it is using k-means. How is it overcoming the local minima problem of k-means? Moreover, what are the benefits of spectral over k-means?
If someone gives detailed insight upon this, it will be very helpful for me.


text:
How many layers do GPT-3, AlphaFold 2, and DALL-E 2 have?
Unsuccessfully, I tried to find out the "depth" (definition below) in large neural networks such as GPT-3, AlphaFold 2, and DALL-E 2.
Formally, my question is about their computational graph: consider a path from some node (a.k.a. neuron) to another. The length of a path is the number of its edges.
What is the longest path from an input node to an output node that visits unique nodes at most once?
I would appreciate any answer/reference regarding large networks like those mentioned above.

Please, focus on 1 architecture per post. So, edit your post to ask just about one architecture. If you have questions about other models, ask them in a separate post. If you're interested in a reference that summarises results about multiple models, then I would change the title not to be so specific and to actually emphasize that you're looking for some reference.

Having said that, it seems that you're asking 2 distinct questions here: 1. "What is the longest path from an input node to an output node that visits unique nodes at most once?" and 2. "How many layers do GPT-3, AlphaFold 2, and DALL-E 2 have?". Please, only 1 question per post. If they are very related or the same question, explain why.


text:
Is there a benefit to starting with MCTS and switching to minimax as the branching factor decreases?
I've invented a deterministic, perfect-information game with a fairly large branching factor (~150) which tapers out dramatically after the midgame (~30 at worst). I need a strong AI. My understanding from questions like this and this is that it makes more sense to go with MCTS with a large branching factor, but as the game reaches endgame, I'm worried about MCTS missing non-obvious wins due to underestimating their ancestor game states.
I do have some ideas for a heuristic to use with minimax, but since this is a brand new game, it's probably not going to be as good as it would need to be to make minimax worthwhile in general. Instead, I was thinking about using MCTS in the early game, and switching to minimax (with alpha beta pruning) after the branching factor goes below a certain threshold for long enough.
Are there any issues with using an approach like that?


text:
Prediction of continuous variable based on threshold
The independent variables are date, count, atmp, and clouds, and the dependent variable is amount. It's only important if amount > 1.0, so it can be a binary task.

date
count
amount
atmp
clouds

2014-01-21 00:00:00
87
0.169158634145595
14.6
NaN

2014-01-27 00:00:00
87
2.931856805839958
14.6
NaN

2014-02-03 00:00:00
87
0.4570096329822
12.4
1.0

2014-02-10 00:00:00
90
0.422058948094016
15.2
NaN

However, converting amount to 0 or 1, doesn't the model lose information that would be helpful in training? And vice versa, if leaving amount as continuous and performing regression, is it possible or common to show the model's accuracy based on being above or below 1.0 rather than using a statistic such as MSE?
What is a good way to model this using machine learning?

I don't think there's a hard rule here, just do what makes sense.

Do a standard binary classification task.
Assuming the label itself actually contains information (and it's not just there to indicate > 1 or not), then you can try keeping the full labels (maybe they are indeed useful to the model), and train a regression task.

For your internal metrics you can keep using MSE as a statistic to look at to help differentiate various models (since it's more sensitive than precision/recall) but keep precision/recall of the resulting binary classification as your main metric.
At the end of the day it does not matter how the model was trained, so report whatever the most useful metric is (on your test data of course)


text:
How can AlphaZero be used in other industries besides gaming?
I'm an AI Engineering student from Belgium and I'm writing my bachelor thesis on the creation of a chess computer with deep reinforcement learning based on AlphaZero. My implementation can be found here: https://github.com/zjeffer/chess-deep-rl.
To complete my thesis, I'd like to know more about AlphaZero's use in other industries. It seems that so far, AlphaGo, AlphaGo Zero and AlphaZero have primarily been used for solving games.
How can AlphaZero and Monte Carlo Tree Search be used in other industries besides the games industry?
I've looked at this question, but it's more general - I would like to know specifically about AlphaZero and MCTS applications.

Looking at it basically, considering these are made to oppose the human player or AI player in a strategy game, I would look at other aspects of life that have military-like components and a vs kind of dynamic, e.g. utilised in war, in computer games and things like this. Perhaps crowd control could be one in terms of strategical thinking it could try to adapt using strategy to such 'chaotic movements. I know it is not a very specific answer but I think I have touched on some good points that can lead to finding more papers on the subject.


text:
Does $S_{t+1}$ denote the future information in Q-learning?
In Q-learning, $Q(S_t,a)$ is updated by the Bellman equation. $Q(S_t,a) = r + \max_{a'}(Q(S_{t+1},a'))$ where $S_{t+1}$ is the future state.
Let's say $S$ denotes the stock price, does it mean we are using future information $S_{t+1}$ in the Q-learning process?
After the training process and bringing into implementation, how is it possible to apply the strategy without knowing the future price (the future state)?

You should be careful with the equations that you use. Is that really the equation or update rule used by Q-learning? Maybe you're referring to the target (in DQN)? In any case, be aware that the target is not the update rule.

Let's say $S$ denotes the stock price, does it mean we are using future information $S_{t+1}$ in the Q-learning process?

It means you are using the information from a time step $t+1$ to update the value you associate with a state or state action pair seen on a previous time step $t$. When training, this is not future data, it is past data, and there does not need to be any special consideration of the value of $t$. It might be something that has just been observed, or could even be from other experience from years ago. The important thing is that you must know the state and action from time $t$, and the reward and next state at time $t+1$ (in general, you might take longer trajectories up to some time $t+n$).
The term backup is used to describe the process of using information from a range of time steps to improve estimates made in earlier ones, and it is a key part of how all reinforcement learning (RL) methods work.

After the training process and bringing into implementation, how is it possible to apply the strategy without knowing the future price

In production, you make use of the estimates that you refined during training by referring to the learned value function.
In your example, if you are in a production system and are at time step $t$, then you know the current state $S_t$. Your policy will be:
$$\pi(S_t) = \text{argmax}_a Q(S_t, a)$$
This is the greedy policy, and doesn't use any future information, but it does rely on a training process that was allowed to "look ahead" to learn about results of taking different actions.
It is common to use a variation of this decision rule during training too, because at the point of making a decision on how to act, the agent does not know what the outcome will be. A common variation is that sometimes the agent will act randomly, and sometimes it will act greedily with respect to action values - that is the $\epsilon$-greedy policy.


text:
What is commonly done for standardization/normalization of the targets in Deep Q-Learning?
I have been searching a lot about standardization/normalization of rewards and targets for the DQN algorithm. For the rewards, I now use the gym wrapper, which only scales but not shifts the rewards by an estimate of their standard deviation, which is updated over time.
However, I can't really find something about common practice for standardization/normalization of the constructed targets. Is it done on top of reward normalization? Do we just let the agent train for a while and then take an estimate of the standard deviation of the targets and scale with the estimate then, assuming we can't use any information of the environment's dynamics?
If this question is too broad to be answered in general, I'll quickly list the specifics of my environment.

Rewards are stochastic and approximately in the range [-0.2, 0) before scaling with outliers excluded
The agent receives a reward after each timestep
Given a state and action, the next state is deterministic
Very short time horizon -> 5 timesteps per episode
Action space is discrete and only consists of 11 actions
With reward scaling, the q-targets are approximately in the range [-3, 0] (I saved a list of targets and plotted a histogram a couple of times during training)

Is the NN set up for regression, with linear output in last layer, and MSE loss function? That would be default for most DQN. I don't have an answer for your main question, although my opinion would be that no scaling is required or likely to be useful with q targets in [-3, 0]. There are other radically different options, such as distributional return predictions, although I have only used them for very simple reward schemes (e.g. +1, 0, -1 at end of episode), no idea if they would help or hinder here.

@NeilSlater thanks for the response and your view. The NN is indeed set up for regression with the Huber loss.

Not sure about DQN, but in actor-critic algorithms (e.g. ppo) the state is always normalized, sometimes the value function is normalized (in that case, the targets for the value function are normalized so the value function outputs a normalized prediction). Never heard of reward normalization.


text:
Knowing the futility of discounting in continuing problems, how can we say discounting has no role in control problems with function approximation?
Sutton-Barto (Section 10.4, page 254):

Based on the futility of discounting in continuing problems, how can we conclude that discounting has no role to play in control problems with function approximation?


text:
Why is the step-size $\alpha_t = 1/t$ not appropriate for non-stationary function approximation?
Sutton-Barto (Section: Selecting Step-Size Parameters Manually, page: 222)

The classical choice $\alpha_t = 1/t$, which produces sample averages in tabular MC methods, is not appropriate for TD methods, for nonstationary problems, or for any method using function approximation.

Why is the choice  $\alpha_t = 1/t$, which produces sample averages in tabular MC methods? Why is it  not appropriate for TD methods, for nonstationary problems, or for any method using function approximation?

I think there's a typo in your first question. You're asking "why is the choice x", but then you don't finish the sentence. Or are you asking why that step size produces sample averages? Having said that, it seems that you're asking more than one specific question in this post. Please, only one question per post.


text:
Does it make sense to add an additional attention layer while fine-tuning Bert?
I'm fine tuning a Bert/Roberta model for a classification task.
As I need to improve my results, I'm thinking about to add an additional attention layer after Bert model and before dense and dropout layers. Is this a good idea?

What method are you using for fine tuning? Do you have your code in GitHub?

The usual practice is to the first token embedding as an input to the classifier, which forces the last layer to collect the relevant information from the previous layers to this particular embeddings.
You might probably view it as wasting the potential of the last layer, where most hidden states are just ignored. From this point of view, an additional attention layer might better use what BERT already knows. On the other hand,  adding an entirely new layer means new parameters that need to be learned from scratch, so more data is needed. Because this is not usually done, I guess the gain from it is probably very small.
I would rather focus on more traditional ways of improving classification accuracy: cleaning/preprocessing data and better regularization.

Please, add information on "where most hidden states are just ignored".

A few points to consider:

Is your dataset big enough for fine-tuning BERT model, a large model might overfit your dataset, and using a small model might help in that case
Also instead of a whole attention block, try adding only a linear layer to see if that helps, you can also reduce the number of parameters in the layer by using a bottleneck architecture, i.e. adding 2 layers of shape (N x N/4) & (N/4 x N).
Instead of full fine-tuning, try using LoRA where you can control the rank of matrices according to the size of your dataset.

Please, share any reference of LoRA and its experiment.


text:
What is an epistemic graph in AI and how is it related to cognitive science?
I found this paper Epistemic graphs for representing and reasoning with positive and negative influences of arguments.
I haven't found any definition of or Wikipedia article on epistemic graphs on the Internet.
Can anyone give me an introductory idea?

See this github for further ref. Essentially EG assumes degree of belief as some probabilistic quantitative version for our usual qualitative everyday argument based on our axiomatic superstructural beliefs. Now in what degree you believe this core idea to be useful for what kind of applications?


text:
Why is depth-limited is preferable to minimax without depth limited
I have read about a question that says the following:
Why is depth-limited minimax preferable to minimax?
One of the wrong answers was:

The depth-limited minimax will achieve the same output as minimax without depth-limited, but can sometimes use less memory.

Why is the above answer is wrong? I mean, don't both of these algorithms always achieve the same output, and because the depth-limited minimax doesn't always explore all the states this makes it use less memory?

Can you please provide a reference that describes this "depth-limited minimax"? I think you're not talking about alpha-beta pruning, but some other variation. Wikipedia provides the pseudocode for the depth-limited minimax here, but if you have a more reliable reference that you're using that describes it, feel free to share it with us. It might also be a good idea to provide the link to the original source of the question.

Assuming that 'non-depth-limited minimax' is minimax with iterative deepening, it does not have to result in the same output as depth-limited minimax. It can traverse the tree deeper and thus find different optimal answers for a 'deeper' layer.


text:
Is there any research on anger and distrust detection (presence and level of political cynicism)?
The undergrad research project I'm working on would require me to detect presence and level of political cynicism from reddit posts.
According to definition political cynicism consists of

anger
distrust

towards politicians or political institutions.
Does anyone know of existing research relevant for my goal?
I know approaches exist for emotion detection, so the anger component should be no problem.
But how could I model distrust or at least come close to it?
I have access to a large dataset consisting of reddit posts (from politics related subreddits) with labels measuring the political extremity of the user. Political cynicism is one of the most distinguishing features of those on the extreme ends of the political spectrum compared to moderates according to data from psychology, so the data should somehow reflect this correlation, and therefore there should be a some opportunity for a CNN to pick up on it. But would there be a way to then isolate this component?
Any guidance would be appreciated.
I'm a newbie to NLP so please forgive any ignorance expressed through this post.


text:
How to deal with an unbalanced dataset?
I'm constructing a feed forward neural network that predicts whether a patient will get a stroke or not. However, my dataset is very unbalanced. Out of 5111 rows, 250 contain patients that have had a stroke (1) and 4861 that did not (0). The accuracy is (as a result of this, I suppose) very high (89% on the first epoch, and 95% on the second, then it stays at 95%). What would be the best thing to do about this?

Please, put your specific question in the title. "Unbalanced dataset" is not specific and not a question. Moreover, use more appropriate tags, like [tag:unbalanced-datasets]. Finally, I recommend that you take a look at existing questions. I think this question was already asked more than once here in the past.

4861/5111 is about 95.1%, so it looks like your classifier is probably predicting every patient as "no stroke" (i.e. it is not really doing anything useful).  The thing to do is to work out the costs of false-positive (predicting a stroke but it didn't happen) and false-negative (predicting they won't have a stroke but they did) errors.  Then factor those into your training process.  This can be done in two ways: Firstly by using a weighted training criterion so you can factor the costs in explicitly; or secondly by resampling the dataset to have a greater proportion of "stroke" patients.
The presence of an imbalance is very rarely a justification in itself for resampling the data.  The real reason to do so is that tasks with an imbalance very often have unequal misclassification costs, for instance if you tell someone they are going to have a stroke when they won't, you will have scared them rather badly, and you may spend some more money on more testing, but that is probably about it.  If you tell someone they are not at risk of a stroke when they are, they may go home, have a stroke and become severely disabled as a result, or even die.  So the false-negative cost is likely to be much higher and that will boost recognition of the positive cases.  The amount of resampling required depends only on the costs; the degree of imbalance is entirely irrelevant.
BTW, rather than look at accuracy, you might want to look at a related metric, which is the improvement over a classifier that always predicts the majority class.  Something like:
$$\frac{\mathrm{Accuracy} - \pi}{1 - \pi}$$
where $\pi$ is the proportion of the most common class in the dataset.  In this case, your classifier is going to score somewhere close to zero as it is probably just going by the majority class, and a score of zero shows clearly that it isn't doing a good job.  A score of 1 would be perfect classification.  A negative score shows the model is worse than just guessing.  It is an affine transformation of accuracy though, so it is still measuring the same basic thing, just on a more interpretable scale.

You can use a data augmentation technique like SMOTE to oversample the minority class. It will help you have a more balanced dataset. Here is a nice guide on it:
https://machinelearningmastery.com/smote-oversampling-for-imbalanced-classification/

I'd advise against SMOTE for modern classifier systems.  It was developed for very simple classifiers that didn't support cost-sensitive learning or have some means of avoiding over-fitting.  Modern classifier systems, such as SVMs or NN have both, so SMOTE is not likely to be necessary (or helpful as there is a lot of good theory supporting regularisation, but none for the rather peculiar method SMOTE uses to generate synthetic examples)

You should use another classification metric for evaluating your model. I would just look at the confusion matrix to see how the model performance on the "interesting class" (minority class).
To overcome your imbalanced dataset, you could upsample the minority class. Bootstrapping is a great staring point. Then advance with SMOTE or something else. This article might give you some ides; 8 Tactics to Combat Imbalanced Classes in Your Machine Learning Dataset

The classification metric should depend on the needs of the application rather than problems with the model.  In this application the expected loss is likely to be a better metric as the misclassification costs are obviously unequal, so the thing to do is to work out what those costs reasonably might be.


text:
Why would one prefer the gradient of the sum rather than the sum of the gradients?
When gradients are aggregated over mini batches, I sometimes see formulations like this, e.g., in the "Deep Learning" book by Goodfellow et al.
$$\mathbf{g} = \frac{1}{m} \nabla_{\mathbf{w}} \left( \sum\limits_{i=1}^{m} L \left( f \left( \mathbf{x}^{(i)}, \mathbf{w} \right), y^{(i)} \right) \right)$$
This is mathematically equivalent to
$$\mathbf{g} = \frac{1}{m} \left( \sum\limits_{i=1}^{m} \nabla_{\mathbf{w}} L \left( f \left( \mathbf{x}^{(i)}, \mathbf{w} \right), y^{(i)} \right) \right)$$
just moving the gradient operator inside/outside the sum.
But I was wondering: why one would prefer the first representation?
My thoughts, and please correct me if I am wrong:

When performing batch gradient descent in practice, we process one example
after the other and compute the corresponding gradient everytime. So
the second equation above better represents what is really happening

Even more, there is no alternative in practice. I can not compute all losses, just sum them up and obtain just one gradient afterwards. So the first equation (although mathematically correct and equal) might even be somewhat misleading?

The only reason I can imagine to decide for the first way is to express more clearly that there is one (aggregated, mean) gradient based on one data set

Any mistakes here on my side?

Here is a related question.


text:
Is the initialisation of $V(s)$ and $\pi(s)$ really arbitrary in policy iteration?
In Sutton and Barto's book (Reinforcement learning: An introduction. MIT press, 2018), the algorithm "Policy Iteration" is:

Here, $V(s)$ is initialized arbitrarily, meaning that I can choose anything I want. Moreover, I think nothing is stated about $\gamma$ here so we can consider undiscounted environments where $\gamma = 1$.
Now suppose we use the following environment:

If I initialize:

$V(s_1) = 10$
$V(s_2) = 10$
$V(s_3) = 0$
$\pi(s_1) = a_1$
$\pi(s_2) = a_3$

With this, it appears that the "Policy Evaluation" part will not have any effect and the algorithm will immediately stops, outputing a policy where the optimal actions are $\pi(s_1) = a_1$ and $\pi(s_2) = a_3$. What am I missing ?

EDIT: I made a toy repository to reproduce, if you want to tweak the numbers of point out something I misunderstood: https://github.com/Gregwar/policy_iteration_initialization/blob/master/policy_iteration.py

(I think the thing is that $\gamma = 1$ can only be used if all episodes terminate, else it has to be $< 1$. Else the episode returns is not well-defined.)

I think you have constructed a bad case for the algorithm as given. You have created an infinite loop and set starting values such that there is no chance for the loop to break out. You have also deliberately set up a starting policy for an episodic problem where the episode cannot terminate.
One simple rule to avoid this is to set $\pi$ to the stochastic equiprobable policy initially (i.e. the first policy will explore all edges of the MDP, making it very hard - maybe impossible - to construct bad cases), and all initial $V$ to $0$. Although these can be set arbitrarily, it is not the user of the solver that normally gets to decide these initialisations, but the person implementing the solver.
So, yes, the algorithm as shown does let you deliberately break it through apparently free choices that are not covered in the book. However, these look like edge cases that were maybe not worth covering in detail. The algorithm works if you pick reasonable arbitrary choices, but defining "reasonable" is fiddly and might distract from the introduction to the topic.

I agree with your second point, but not with the first. If $\gamma < 1$ it seems to always work.
I made a toy repository if you want to fiddle with the numbers and/or check if I did something wrong: https://github.com/Gregwar/policy_iteration_initialization

@Gregwar: You are right, it works. I was forgetting that the update step re-evaluates the bootstrap values to pick the optimal action. As soon as predicted values from evaluation drop below the actual reward received by transitioning to the terminal state, the polciy switches to that final transition.

@Gregwar I dropped my bullet points as they are not relevant to the take-away, which is "arbitrary" means "person writing solver controls it, and has plenty of free choice", and not literally "works for all values".

Ok, I see. But I feel like they'd better give slightly more conservative preconditions that works all the time and let the "person writing solver" take advantages of less conservative reality rather than writing "arbitrary", where it is easy to find a counter-example and thus deceptive :-)


text:
How is AI used in Internet of Things?
I would really appreciate it if someone would explain how AI is used in IoT. In the papers that I have found, half of the paper itself is about what IoT  is and very few information about how AI is used for it. Good paper recommendation or a book would also strongly help.
The papers that I found:

Selected methods of artificial intelligence for Internet of Things conception (2015) - Aneta Poniszewska-Maranda, Daniel Kaczmarek

Artificial Intelligence in Internet of Things (2018) - Ashish Ghosh, Debasrita Chakraborty, Anwesha Law.

The application of artificial intelligence in the internet of things (2019) - Wenbo Yao.

Review on Artificial Intelligence with Internet of Things - Problems, Challenges and Opportunities.

for me internet of things is about smart equipment, the applications are too broad, AI may not run directly on a single small equipment item but runs on server

an example is facial recognition on a "smart" camera

I edited your post to include the links to the papers you found. Make sure these links point to the correct paper.

One application of AI in IoT is cybersecurity. If you are interested in this topic, this review Role of Artificial Intelligence in the Internet of Things (IoT) cybersecurity (2021) mentions many specific works and papers that describe specific experiments.
I don't know if you are familiar with AI techniques, like decision trees or neural networks, but one idea is to apply these techniques to detect anomalies in the traffic.
The following diagram of a decision tree (taken from the review) illustrates this idea

So, basically, you detect whether there's an attack based on the flow rate, size, and duration of traffic.
Right now, I cannot provide more details, because I don't know much about this specific topic, but, after having quickly skimmed through the paper and fully read the introduction, I think you will get answers to some of your questions if you read it.


text:
What is 'eligibility' in intuitive terms in TD($\lambda$) learning?
I am watching the lecture from Brown University (in udemy) and I am in the portion of Temporal Difference Learning.
In the pseudocode/algorithm of TD(1) (seen in the screenshot below), we initialise the eligibility $e(s) =0$ for all states. Later on, we decay this eligibility by a factor $\gamma$.
My question is, what does 'eligibility' mean in an intuitive sense? Previous dynamic programming algorithms (value iteration and policy iteration) do not have this 'eligibility' concept. Why is it here in TD? Is it because we are effectively sampling episodes here (unlike in the previous when we test all states and all possible actions?)
Insights welcome.

I recommend that you provide the link to the lecture's video and slides, not just the screenshot.


text:
What does it mean by "gradient flow" in the context of neural networks?
Several research papers and textbooks (e.g. this) contain the phrase "gradient flow" in the context of neural networks.
I am confused about whether it has any rigorous and formal way of understanding or not. What is the flow referring to here?

Note that although this is seemingly not the way it's used in the paper you link, gradient flows in the context of optimization generally refer to a continuous-time analogue of gradient descent; see, e.g., here: https://francisbach.com/gradient-flows/

It has. Gradient flow or more generally flow is a well known concept in maths. Say we have a function $f:\mathbb R^n \longrightarrow \mathbb R^n$ and a function $\theta:[0,\infty)\longrightarrow \mathbb R^n$ such that the ODE
$$
\partial_t \theta(t) = f(\theta(t))
$$
exists for any initial choice $\theta(0)\in\mathbb R^n$ uniquely. Then i think the naming comes from the following informal description. Lets imagine the case $n=2$ then $\mathbb R^2$ is a plane. Now you drop $\theta(0)$ somewhere on this plane and wait some time $t$ to see where the $\theta(0)$ will flow to. You can also draw the trajectories for multiple starting points $\theta(0)$ which will often give you an image that looks like a flow of a liquid.
In the case of the ANN $f$ would be the negative gradient of the costfunction with respect to the paramter-vector $\theta$ and gradient descent would be an approximation of the gradient flow.

Here is my idea of what that means: Gradient flow is an abstract term to describe properties of the gradient. The gradient is calculated by propagating the error backwards through the networks, therefore it kind of flows from the last to the first layer. Depending on network architecture and loss function the flow can behave differently.
One popular kind of undesirable gradient flow is the vanishing gradient. It refers to the gradient norm being very small, i.e. the parameter updates are very small which slows down/prevents proper training. It often occurs when training very deep neural networks. Residual connections can help, because they bypass operations that reduce gradient magnitude.
The exact opposite would be exploding gradients where your gradient-norm is very large which leads to unstable training and the weights inside the network can't follow a stable trajectory to the optimum. This often occurs in the context of recurrent neural networks.
Here is a nice article on vanishing/exploding gradients.
A general way to phrase this is that the gradient reveals properties about the loss function which you are optimizing (as it is the derivative). A noisy or heavily oscillating loss function usually implies an undesirable gradient flow.
Thinking about these properties plays an important role in network and loss function design.
For example, the root function $\sqrt{\cdot}$ has a very large derivative when approaching zero, so having a $\sqrt{\cdot}$ in the network or the loss function can lead to exploding gradients.


text:
Are any AI systems available, or in development, for finding and analysing fallacious inference in natural language text?
Poor reasoning, and ignorance in general, is the source of a lot of suffering and evil. Covertly erroneous logic is often used in manipulation. And much of this broken thought is being used directly in the training of AI.
There has been talk of, and development in, fact-checking, such as for language transformers. But what about reasoning?
The function in mind is specifically being able to process a potentially large text body, analysing all logic and implied relations for fallacy and other misleading reasoning. Perhaps shades of colour could indicate level of error. A bonus would be output listing and explaining the mistakes, maybe like compiler errors -- "fallacy x between premise y and conclusion z".
Are any AI systems available, or in development, for finding and analysing fallacious inference in natural language text?

It is very complicated to even recognise inferences; deciding whether they are fallacious would require understanding of how the world works, and AI is a long way away from that. So the answer, I'm afraid, is no.

There's some impressive work going on at IBM under the name of "Project Debater", which already produced some impressive results, that you can check in this video.
The project is about creating a system capable of debating with human on random topics by scraping the internet to get ground knowledge (mainly scientific papers, not random gibberish of course). The system is pretty massive, it include speech synthesis and lot of other components, what's interesting for you are the parts I highlighted in red, especially those on the left that contribute to generate the counter arguments to the human opponent (rebuttal construction).
I won't go into the details cause all those three topics constitute entire branches of Natural Language Processing. But you can easily see how combined together they try to accomplish a rough version of deductive reasoning:

claim detection: identifying the main argument of a sentence/document
evidence detection: identifying what facts are presented along with the argument
stance detection: identifying the positive/neutral/negative stance of each evidence (and possibly opinion based facts presented) toward the claim has been made.

Assuming that we managed to extract all this information from a text (big time, unfortunately not easy at all) then we can build reasoning out of it. For example:

Public schools are worse than private ones for children education -> detected negative claim
Researches show that there's an equal amount of graduated students coming from public and private schools. -> detected evidence
The previous evidence has a positive stance toward the initial claim -> stance detection

Putting everything together, since we have a negative claim followed by a positive evidence toward the same fact, we just found out that the document contains a contradiction.

Yes, current large language models since GPT-4 are able to find errors in reasoning, probably zero-shot, not much prompt engineering needed.
Giving a precise probability for something being wrong is much harder, even if the the model knows it, it is hard to query. Rough estimates should be easy.

Some examples using GPT-4:
Prompt:

Alice had blue shoes. Therefore, Bob had to eat the cake.
Analyze the reasoning.

Result:

The reasoning presented in the statement is illogical and does not
follow a coherent thought process. There is no apparent connection or
correlation between Alice having blue shoes and Bob having to eat the
cake. The conclusion does not follow from the premise, and the
argument lacks a proper cause-and-effect relationship.

Starting a new chat:
Prompt:

Alice had birthday cake. Therefore, Alice had to eat the cake.

Result:

It is not necessarily true that Alice had to eat the cake just because she had a birthday cake. While it is common for people to eat cake on their birthdays, Alice could have chosen not to eat it, shared it with others, or saved it for later.

Starting a new chat:
Prompt:

Alice had blue shoes. Therefore, Alice had to shoe the blues.

Result:

This statement is a play on words. "Alice had blue shoes" means that Alice owned a pair of blue-colored shoes. "Shoe the blues" is a pun on the phrase "sing the blues," which means to express one's troubles or sadness, often through music. In this case, the pun implies that because Alice had blue shoes, she had to express her sadness or troubles, possibly through wearing her blue shoes.

These examples are not cherry picked.

With that in mind, can any current models convert real-world natural argumentation into something more formal, like a list of premises and conclusions? An example would be taking an Online argument thread, or political debate, then summarising its arguments cleanly.

@user253751 I added your examples to the answer, thanks for asking. It shows that I did not cherry pick the question. I did not cherry pick the answers either.

@Michael Yes, summarizing is one of the most important tasks it is used for,

@Michael To demonstrate, I asked it "What are the main points in:" followed by the text of your original question above. The answer was a list of 5 items:

Poor reasoning and ignorance can lead to suffering and evil, and manipulative individuals often use faulty logic.
AI is sometimes trained using this broken thought process, raising concerns about its potential impact.
Fact-checking and language transformers are being developed to address this issue, but there is still a need for AI systems that can analyze reasoning in text. ...

@Michael ... 4. The desired AI system should be able to process large text bodies, identify fallacies and misleading reasoning, and provide an output explaining the errors.
5. The question raised is whether there are any AI systems available or in development that can analyze fallacious inferences in natural language text.

@user253751 I tried to read it carefully and found it a very valid interpretation. What do you think is wrong? I mean, it does not take it as nonsense, but as poetic, with a convincing interpretation.

@user253751 To ask GPT-4 for advice for an example for one that is actually true and not a fallacy, I prompted it with "The text below is a question and some comments under one of the answers. Commentator user253751 is looking for a specific kind of example in the last comment. Can you find one?" followed by the text of the question, the word "Comments:" and the text of the comments under this answer up to the comment here you ask for it (starting "okay,...").

The result was scary good - note that I did not clean up the comment thread in any way, or clarify the question:
One possible example that is true and not a fallacy could be:
"Alice studied diligently for her exam. Therefore, Alice was well-prepared for the exam."
In this case, the premise is that Alice studied diligently, and the conclusion is that she was well-prepared. The reasoning is sound, as diligent studying typically leads to being well-prepared for an exam.

@Michael That is worth a different question, but as an example, I asked it to summarize the first third (28 minutes) of the transcript of the presidential debate between Trump and Biden, on Oct. 22, 2020.  - the model of GPT-8 allows for a context of about 8,000 tokens, roughly 32,000 words. The prompt was "Summarize the 5 most important points, one line each." followed by https://eu.usatoday.com/story/news/politics/elections/2020/10/23/debate-transcript-trump-biden-final-presidential-debate-nashville/3740152001/ up to [27:44] See the 5 items in the following 2 comments for the result:

Trump claims a COVID-19 vaccine is coming within weeks and will be distributed by the military, while Biden focuses on encouraging mask-wearing and rapid testing.
Biden emphasizes the need for transparency in vaccine development and criticizes Trump's handling of the pandemic, calling the current situation a "dark winter."
Trump defends his administration's response to COVID-19, arguing that they are learning to live with the virus and asserting that the virus will eventually go away.

Biden states that he would not shut down the country, but instead focus on shutting down the virus, suggesting increased safety measures for reopening businesses and schools.
Both candidates address foreign interference in U.S. elections, with Biden vowing to make interfering countries pay a price and Trump defending his administration's toughness on Russia.


text:
How to calculate uncertainty in Deep Ensembles for Reinforcement Learning?
Lets take the following example: I must predict the return (Q-values) of x state-action pairs using an ensemble of m models. Using NumPy I could have the following for x = 5 and m = 3:
 >>> predictions = np.random.rand(3, 1, 5)

 [[[0.22668968 0.58857404 0.49572979 0.68034031 0.96522052]]

 [[0.90452081 0.07554403 0.62139326 0.6269648  0.78426295]]

 [[0.14154026 0.75292144 0.99831914 0.7584285  0.69479723]]]

Thus, for each possible action we have the following prediction considering the set of models:
>> actions_out = [q[0] for q in predictions]
>> actions_out = [list(a) for a in zip(*actions_out)]
>>
[
[0.22668968082539054, 0.9045208066488987, 0.14154025891848865],
[0.5885740401748317, 0.07554403461136683, 0.7529214398937515],
[0.4957297945825573, 0.6213932636399634, 0.998319138313377],
[0.6803403139829055, 0.6269648017308974, 0.7584284958713308],
[0.9652205174041535, 0.7842629542761801, 0.6947972303000536]
]

Where for example the actions_out[0] = [0.22668968082539054, 0.9045208066488987, 0.14154025891848865] is the prediction of 3 models for the action 1 (index = 0).
The question is: to calculate the variance of those values (the disagreement or uncertainty between models), the following is correct?
variance = np.var(actions_out, axis = 1)
avg_variance = np.average(variance)

Does this average capture the disagreement between the models?


text:
Why should data batches in a neural network have an equal size?
Why should data batches in a neural network have an equal size?
I have seen some recent research works on making the batch size dynamic, but still, I can't find an answer to my question.

They don't have to be equal, but introducing variable batches brings more complexity.
In particular you need to make sure that batches are representative enough of the full dataset. If the batches have the same size you can just test few batch values until you get satisfactory results. If you introduce variable batches you need to perform more complicated test on the distribution/rule controlling this variability and handle edge cases, which is an additional headache over an already complicated debugging process. Usually it's not worth the pain, there are more impactful ways of spending your development time.


text:
Is possible to perform an early goal test with A* using a consistent heuristic?
I have a doubt about A* goal test.
As far as I know an early goal test is performed when a node (representing a certain state) is inserted in the frontier or, to conform with AIMA terminology, when is reached/generated.
Quoting AIMA 4ed:

(...) with a consistent heuristic, the first time we reach a state it will be on an optimal path, so we never have to re-add a state to the frontier, and never have to change an entry in reached.

With this piece of information I suppose that an early goal test is totally safe, but I wasn't able to find anything about it on the internet to confirm my hypothesis, so my second guess is that I am wrong about it.
Am I wrong? Did I misunderstood something?

Yes, you can. It's an extremely common optimization to early-exit like this in pathfinding code when you know for sure the heuristic will always be consistent.

Is early goal test ok even with multiple goal states? I think it is not. It is true that we can adapt a problem with multiple goal states to a problem with one goal state by connecting all goal states to an unique goal state, but this is equivalent to perform a late goal test with the original goal states in my opinion.


text:
Why is an action-independent baseline required to reduce variance?
I'm learning policy gradient methods. I encountered the REINFORCE algorithm with variance reduction with a baseline. I see we can use a constant or state-dependent function (e.g value function) but not an action-value function as a baseline.
Is there any intuition behind this?
I saw some papers claimed that an action-dependent baseline improves the performance compared to baselines that use only state information. One example is: Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines (2018) by Cathy Wu et al.

Can you please explain which paper/resource claims that you need "an action-independent baseline to reduce the variance"? You only say that we can use a constant or state-dependent function as the baseline and that a paper used an dependent baseline and that improved the performance (what do you mean by "performance", btw? And do they mention anything about the variance?).


text:
What are the state-of-the-art AI methods to recognize elements on webpages or the purpose of webpage?
I'm curious to know about the capabilities of AI today in 2022. I know that AI has become pretty good at recognizing things like objects in photos. But what about when it comes to elements in HTML? Would it be feasible to use AI to determine things like:

Is there a call-to-action? Basically a button or main action that directs the user somewhere. The text in the call to action can obviously contain a variety of different text.
Is there a form on the page for the user to fill out?

The last time I tried running a rendered image of a website through image recognition software, such as Google Vision or Amazon's Rekognition, it didn't detect these things, which didn't surprise me. However, maybe there's a better or alternate way, such as using the source code? But the end goal would be to determine if the page is meant to capture leads, and the form elements are some of the criteria we'd be looking for. Maybe this can also be seen as a categorization type of task too.
As I understand, AI is a broad term. So, if this was a feasible project, I'd also be curious to know what branch of AI would be the correct one to explore.

I think image recognition with screenshots of a webpage is the wrong approach for this sort of task. I think you can leverage the syntax in the source code of the webpage using a custom NLP model to create a much more powerful classification model. I've read a few papers that do this for fraud detection. But could you clarify what you specifically are looking for? A pretrained model? A network structure?

Thanks for the feedback. Honestly, I'm not exactly sure what I'm looking for, which is part of the problem. The thing I'm looking to solve is determining if the intent of a page is to gather leads. I had asked something similar to this elsewhere and NLP was also mentioned there.

You could do it with the current set of vision models

Two main approaches are used for AI to recognize elements and understand the purpose of webpages:
Computer Vision with Deep Learning: This is the leading technique and relies heavily on deep learning models, a type of artificial neural network particularly adept at image recognition.expand_more  These models are trained on vast amounts of labeled webpage data, allowing them to identify specific elements on a webpage like buttons, menus, or text sections.  Some popular deep learning architectures for object recognition on webpages include Faster R-CNN and YOLO.expand_more
Natural Language Processing (NLP):  This approach focuses on analyzing the text content of a webpage. By understanding the language used, NLP techniques can classify the webpage's purpose.  For instance, an NLP system might identify keywords related to e-commerce on a webpage, suggesting it's an online store.

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.

The branch of AI that is devoted to image-related processing is Computer Vision (powered by deep learning, indeed). In particular for this project you probably need to train an object recognition model (e.g. Faster R-CNN, YOLO), able to find relevant parts of the websites like buttons and forms. An object detector finds the location (bounding box, i.e. a rect) and class (which kind of object is) of each detection. In alternative, you can look at semantic or even instance segmentation (look here for an explanation) model (e.g. Mask R-CNN) that instead provides segmentation masks which can handle, in principle, arbitrary complex shapes.

To train such a model, you have to collect a dataset of (images, labels), where the labels comprise: bounding box or segmentation mask of the page's object of interest, and its class (e.g. button, form, video, header, etc).

I think it would be interesting to train a multi-modal model that at the same time learns from images of webpage, and their source code as well. In this case you need to pair CV with NPL (natural language processing). In order of complexity, code (text in general) can be understood by recurrent neural networks, attention models, and large language models (e.g. BERT, GPT, etc.)

In this other solution, you have two neural networs: one processes the image, the other the text/code; you have to combine both at some point, then have the output layer.


text:
MLP or RNN for Regression of Smooth Function (No Time Data)?
My Problem consists of Input sequences in the form of $x=[B,z]$ and one output $y_i$ for each data point $x_i=[B,z_i]$. For one sequence/dataset $B$ is a constant, whereas $z$ is continously between 0 and 5.
In the moment I just train a standard MLP on this. However, I wonder if I can "help" my neural network because I know, that $y_i$ is a smooth function of $z_i$. This seems analog to a time series problem, where I computate my solution at $y_i(t=0)$ and use this information for the prediction of $y_i(t=1)$.
Am I right with this assumption? Can I use RNN or a combination of RNN with MLP for this?

Can you please edit your post to format the symbols and equations with mathjax to improve the readability? Moreover, the question in the title is very general. Please, put your specific question in the title.

Done and changed title

I think the best way is just to try it out. If it is in fact something like a time series RNN/LSTM can be helpful because in many applications (but not all) they yield good results when having sequence data.


text:
Why does importance sampling ratio start and end one step later in off-policy SARSA given in Sutton-Barto's RL book?
In Sutton & Barto's RL book (page 149) they say:

Sarsa update can be completely replaced by a simple off-policy form
$Q_{t+n}(S_t,A_t)=Q_{t+n−1}(S_t,A_t) + \rho_{t+1:t+n} [G_{t:t+n} −  Q_{t+n−1}(S_t,A_t)]$
for $0 \leq t < T$. Note that the importance sampling ratio here starts and ends one step later than for n-step TD (7.9). This is because here we are updating a state–action pair. We do not have to care how likely we were to select the action; now that we have selected it we want to learn fully from what happens, with importance sampling only for subsequent actions.

The explanation for importance sampling ratio to start and end one step later is not clear for me. I would be very happy if one can provide a simple and easy-to-understand explanation (a visual explanation through node transitions would be perfect).

Prepend the excerpt/text that you're quoting with >. This will immediately give an idea what part of your post is a quote. Please, also put your specific question in the title, and use more tags other than [tag:reinforcement-learning], to be more specific.

@ nbro, all done.