text:
What are the recurrences used for updating state value function in $TD$ and $TD(\lambda)$ learning?
There are two types of value functions in reinforcement learning: State value function $V^{\pi} (s)$, state-action  value function $Q^{\pi}(s, a)$.
State value function:
This value tells us how good to be in state $s$ if we are following policy $\pi$. Formally, it can be defined as the average returns obtained at time step $t$ from state $s$ if we follow policy $\pi$.
$$V^{\pi}(s) = \mathbb{E}_{\pi}[R_{t}|s_t = s] = \mathbb{E}_{\pi} \left[ \sum \limits_{k=0}^{\infty} \gamma^{k}r_{t+k+1} \mid s_t = s\right] = \mathbb{E}_{\pi} \left[ \sum \limits_{k=0}^{\infty} \gamma^{k}r_{t+k+1} \mid s_t = s, a_t = a \right]$$
State-action value function:
This value tells us how good is to to perform action $a$ in state $s$ if we are following policy $\pi$. Formally, it can be defined as the average returns obtained at time step $t$ from state $s$ and action $a$ if we follow policy $\pi$ further.
$$Q^{\pi}(s, a) = \mathbb{E}_{\pi}[R_{t}|s_t = s, a_t = a] = \mathbb{E}_{\pi} \left[ \sum \limits_{k=0}^{\infty} \gamma^{k}r_{t+k+1} \mid s_t = s, a_t = a\right] = \mathbb{E}_{\pi} \left[ \sum \limits_{k=0}^{\infty} \gamma^{k}r_{t+k+1} \mid s_t = s, a_t = a \right]$$
Now, Q-learning and SARSA learning algorithms are generally used to update $Q$ function under policy $\pi$ using the following recurrences respectively
$$Q(s_t,a_t) = Q(s_t,a_t) + \alpha[r_{t+1} + \gamma \max\limits_{a} Q(s_{t+1},a) - Q(s_t,a_t)] $$
$$Q(s_t,a_t) = Q(s_t,a_t) + \alpha[r_{t+1} + \gamma Q(s_{t+1},a_{t+1}) - Q(s_t,a_t)] $$
Now my doubt is about the recurrence relations in Temporal Difference (TD) algorithms that update state value functions. Are they same as the recurrences provided above?
$$V(s_t) = V(s_t) + \alpha[r_{t+1} + \gamma \max V(s_{t+1}) - V(s_t)] $$
$$V(s_t) = V(s_t) + \alpha[r_{t+1} + \gamma V(s_{t+1}) - V(s_t)] $$
If yes, what are the names of the algorithms that uses these recurrences?

Q-learning and SARSA are TD learning algorithms; so, no, TD is not limited to algorithms that update the state value function. Q-learning and SARSA are also control algorithms, so they find policies (i.e. controllers). They are also TD(0) algorithms. To have an idea of what TD-lambda is, I think my answer here should address it. Let me know if I should close this post as a duplicate of that one.

In any case, it seems to me that you're asking multiple questions here. I'd suggest that you ask only 1, and clarify how your question is different from others. In any case, to fully understand TD-lambda, you need to understand TD learning well first, then Q-learning. I would suggest that you read the related chapter of Sutton & Barto. It may take some time to get used to all this.

@nbro Now, I am in need of only recurrence equations. That is the reason for asking. I will try to edit.

@nbro I will surely read that book. But for now, I want the recurrences only. I am hoping that I will read in detail further.

There is repetition in your breakdown of $Q^\pi$ and it doesn't seem to go anywhere - is there something missing or a mistake there? Also your analysis of $V^\pi$ is wrong, you add a conditional $a_t=a$ from out of nowhere. Did you intend to write those equations as they are now, or have you copied them from some reference (the reference is plain wrong for $V^\pi$  if so)?

Ok, if you're interested only in the recursive relation of TD($\lambda$) (so focus on one algorithm at a time), please, edit your post to ask only that. If you're interested in knowing if there is a counter-part of the recursive relations of Q-learning and SARSA, please, edit your post to ask only that. If you have any other question, again, edit your post to leave just that question. If you have multiple questions, it's better to split the post into multiple ones, one for each question. It really seems to me that you're asking multiple questions here, although they are related.

Thanks! Please, make also sure that the question in the title matches the one in the body of the post.


text:
How to visually or intuitively understand single element multi-dimensional tensors?
Consider the following code in PyTorch
>>>torch.tensor([8]).shape
torch.Size([1])
>>>torch.tensor([[8]]).shape
torch.Size([1, 1])
>>>torch.tensor([[[8]]]).shape
torch.Size([1, 1, 1])

We can notice that we want to store only a single element $8$ in a tensor. But it is possible in tensors to store $8$ in any n-dimensional tensor where $n \in \mathbb{N}$. In strict case $\mathbb{N}$ may be replaced by $\mathbb{W}$.
But, I am facing difficulty in understanding this fact of a single element contributing to all dimensions. If the element is present in all dimensions, then I am assuming that it has to be present multiple times, which is not the case. I can't understand how a single element is contributing any number of dimensions without repeating itself multiple times.
How to understand this phenomenon? How should I interpret or visualize this fact intuitively?

How should I interpret or visualize this fact intuitively?

You could visualize it as a point in a geometrical space:

8 is just a number
[8] is just a number in a line
[[8]] is a number in a plane
[[[8]]] is a number in a space

The object (number 8) won't change. The space around it changes.

You can never represent a complex object (3d-cube) in a simpler shape (2d-plane).
But you can always represent a simpler object (2d-square) in a higher dimensional shape (3d-space).

A number is a simplest possible object, and therefore it "fits inside" (can be represented in) any dimension.

The number is not repeated if it is the only element of some high dimensional space.
An $n$-dimensional (vector) space is a set of objects (known as vectors, although this term is more commonly used to refer to the objects in $1$-dimensional spaces), which are ordered/organized/shaped in specific ways, but composed of scalars/numbers, and to which you can also apply certain operations.
It may be a good idea to think that the shape/organization of each object in an $n$-d space is determined by some "container", which has a certain shape, and that the container of the objects of higher-dimensional spaces generalizes (or it can include) the containers of lower-dimensional spaces.

In the case of a $0$-dimensional space (e.g. $\mathbb{R}$), there's no container because you don't need it, as every object is composed of only one number (this is always the case!).

In the case of $1$-dimensional spaces, an object could be composed of more than one number (but this does not have to be always the case: it depends on the specific $1$-d space, so there are many $1$-d spaces), so you need a way to organized these numbers. You organize them in a sequence that follows some direction. So, the objects $[0] \in \mathbb{R}^1$, $[1, 2] \in \mathbb{R}^2$, $[2, 3, 1] \in \mathbb{R}^3$, $[0, 4, 2, 2] \in \mathbb{R}^4$, etc., are all $1$-d objects because  each of them is composed of one or more numbers, which are organized in sequence. $\mathbb{R}^1$, $\mathbb{R}^2$, $\mathbb{R}^3$, etc., are all $1$-d spaces, because they organize each of their objects in a sequence: their only difference is the number of numbers/scalars in each object.

In the case of $2$-d spaces, you organize the objects not in a line, but in a rectangle. The rectangles can have different shapes, so they are not necessarily just squares. So, $[0] \in \mathbb{R}^{1 \times 1}$, $ \begin{bmatrix} 1 & 1\end{bmatrix} \in \mathbb{R}^{1 \times 2}$, $ \begin{bmatrix} 1 & 1 \\ 0 & 2\end{bmatrix} \in \mathbb{R}^{1 \times 2}$.

In the case of $3$-d spaces, you organize each object into cuboids. This does not mean that these cuboids contain more than one number. In general, they could, but you also have cuboids that contain only one number, i.e. $[0] \in \mathbb{R}^{1 \times 1 \times 1}$, which, in PyTorch, would be printed as [[[0]]] to give you the idea that the number 0 is inside a container [0], which is inside another container [ ], which can contain only one small container that can contain only one number (e.g. [0], but you could also have had [1] or [10]), which is inside another container [   ], which can contain only one container, which can contain only another container, which, in turn, can contain only one number.

To give you an analogy, think of having many boxes of different sizes and also have balls that you can put inside the smallest boxes. These boxes are the "containers" and the balls are the numbers. You can put smaller boxes inside the bigger ones (such that the boxes cannot slide around) and you can put the balls inside only the smallest boxes (so that they do not move).

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Why is it called "batch" gradient descent if it consumes the full dataset before calculating the gradient?
While training a neural network, we can follow three methods: batch gradient descent, mini-batch gradient descent and stochastic gradient descent.
For this question, assume that your dataset has $n$ training samples and we divided it into $k$ batches with $\dfrac{n}{k}$ samples in each batch. So, it can be easily understood the word "batch" is generally used to refer to a portion of the dataset rather than the whole dataset.
In batch gradient descent, we pass all the $n$ available training samples to the network and then calculates the gradients (only once). We can repeat this process several times.
In mini-batch gradient descent, we pass $\dfrac{n}{k}$ training samples to the network and calculates the gradient. That is, we calculate the gradient once for a batch. We repeat this process with all $k$ batches of samples to complete an epoch. And we can repeat this process several times.
In stochastic gradient descent, we pass one training sample to the network and calculates the gradient. That is, we calculate the gradient once for iteration. We repeat this process with all $n$ times to complete an epoch. And we can repeat this process several times.
Batch gradient descent can be viewed as a mini-batch gradient descent with $k = 1$ and stochastic gradient descent can be viewed as a mini-batch gradient descent with $k = n$.
Am I correct regarding the usage of terms in the context provided above? If wrong then where did I go wrong?
If correct, I am confused about the usage of the word "batch" in "batch gradient descent". In fact, we do not need the concept of batch in batch gradient descent since we pass all the training samples before calculating gradient. In fact, there is no need for batch gradient descent to partition the training dataset into batches. Then why do we use the word "batch" in batch gradient descent? Similarly, we are using the word "mini-batch" in "mini-batch gradient descent". In fact, we are passing a batch of samples before calculating the gradient. Then why it is called "mini-batch" gradient descent instead of "batch" gradient descent?

You are correct, but requires final words:
In Batch GD, we take the average of all training data to update the parameters, hence, one step per epoch. That's very valid if you have a convex problem (i.e. smooth error).
On the other hand, in the Stochastic GD, we take one training sample to go one step towards the optimum, then repeat the latter for every training sample, hence updating the parameters once per sample sequentially in every epoch (no average here). As you can expect, the training will be noisy and the error will be fluctuating.
Lastly, the mini-batch GD, is somehow in between the first two methods, that is: the average of a different portion of the data, every time. This method would take the benefits of the previous two, not so noisy, yet can deal with less smooth error manifold.

Personally, I memorize them in my mind by creating the following map:

Batch GD ≡ Average of All per Step ≡ More suitable for Convex Problems at the Risk of Converging directly to Minima = Heavyweight.
Stochastic GD ≡ Fluctuating & Noisy ≡ Converges on the Long Run especially for Large Dataset ≡ Lightweight but Slower ≡ No Vectorization Possible (because one sample per time).
Mini-Batch GD ≡ Portion of Data per Step ≡ Mixture of Stochastic and Batch GD ≡ Less Fluctuation + Would work for Less Smooth Error Manifolds ≡ Faster Computation.

Regarding the naming convention, I would understand the word "Batch" as a "Set" or "Collection", hence the whole "Dataset". Consequently, "Mini" would go with the flow, to mean a "Part of the Set".

I think that the term "batch GD" is sometimes used to refer to "mini-batch GD" and that "stochastic GD" may actually refer to "mini-batch GD". I think people should take context into account because others may be using these terms inconsistently (this is my impression).

@Yahya, when you say for 'batch GD' that 'we take the average of all the training data...' you specifically mean that from the total aggregate loss on all independent training samples which is then backpropagated, you take the average across all the data to make that single update to the parameters from a single derivative of the model params. Is that correct? Can you also please include in your answer the analogous description for how this is similarly done in Mini-batch.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Material(s) for understanding "image channels"
I am pretty confused about the concept of "image channels".
I want material that explains the concept of channels from scratch to whatever is required to understand their role in machine learning. I think that it is a small concept and possibly present as a chapter in good textbooks.
Where can I read about channels of an image in detail?

Image channels have nothing to do with machine learning, they are just part of computer image processing.
A channel is a number per pixel. So most colour images are stored with red, green and blue channels, as you probably know. Some images are stored in greyscale with just one white channel.
A RGB image is stored like this: pixel 0 red amount, pixel 0 green amount, pixel 0 blue amount, pixel 1 red amount, pixel 1 green amount, pixel 1 blue amount, pixel 2 red amount, ...
They could also be rearranged like this: pixel 0 red amount, pixel 1 red amount, pixel 2 red amount, ....., pixel 99999 red amount, pixel 0 green amount, pixel 1 green amount, ....., pixel 99999 green amount, pixel 0 blue amount, ....., pixel 99999 blue amount.
But that is not common.
A greyscale image only has one channel and it's stored like this: pixel 0 white amount, pixel 1 white amount, pixel 2 white amount, pixel 3 white amount, ...
A black-and-white image also has only one channel (a white channel) but that channel can only be 0 brightness or maximum brightness. They can be stored with just 1 bit per pixel.
Alpha is an extra transparency channel that some pictures have. Alpha 0 means fully transparent. Maximum alpha means fully opaque. Half-maximum alpha means the image is partially see-through at that pixel. Things like photos don't have alpha, but computer graphics that are designed to be displayed on top of other pictures often do.
There are also more exotic systems like YCbCr, where you have a white channel, a blue-versus-green channel, and a red-versus-green channel. Mostly we just convert those to RGB before processing.

"A channel is a number per pixel", I don't think this description is correct: a channel is not a number. A channel often contains the values associated with one specific color, while another channel contains the values associated with another color.

@nbro yes, the concept of "red" is a function from pixel coordinates to numbers. You can ask how much redness is at a given pixel coordinate. You can also ask how much greenness, which is a different function.

Still, how is a channel a number per pixel? Maybe there are some typos in your answer.


text:
When does a batch normalization layer becomes active?
Let us assume your dataset has $n$ training samples each of size $s$ and you divided them into $k$ batches for training. Then each batch has $n_k = \dfrac{n}{k}$ training samples.
Batch normalization can be applied to any input or hidden layer in a neural network. So, assume that I am applying batch normalization at every possible place I can.
Now, consider a particular batch normalization layer (say $b$) of a hidden layer $\ell$. Now, I am confused about the working frequency of $b$.
Will it be activated only after every $n_k - 1$ forward passes i.e, once per batch at the end of the batch? If no, then how $b$ calculates the mean and standard deviation for every forward pass while training if $n_k$ output vectors of $\ell$ are not available at that instant?
Will $b$ calculates the mean and standard deviated, for every forward pass, based on the outputs of $\ell$ that are calculated so far? If yes, then why it is called batch normalization?
To put it concisely, are batch normalization layers active for every iteration? If yes then how they are normalizing a "batch" of vectors?

You can check here which says

The mean and standard-deviation are calculated per-dimension over the
mini-batches

If I understand your question correctly, I think this is related to the momentum which calculates the moving average, hence it can be applied at every iteration. More Details.

@Yahya Does the word "batch" here also need to be interpreted as the "collection" rather than collection of $\dfrac{n}{k}$ samples?

@Yahya you can see that "the layer normalizes its output using the mean and standard deviation of the current batch of inputs" this statement says that we use mean and standard deviation on a batch. Even we use moving averages, the number of samples we are considering for normalization depends on number of iterations occurred till now. So, batch in batch normalization also refers to the collection of vectors we encountered till now.

And it is also useful to check here which says The mean and standard-deviation are calculated per-dimension over the mini-batches. The word mini-batches here imposes the requirement that at-least mini batch number of vectors should be available to proceed. Am i wrong?

@Yahya I am opining that the words batch and mini-batch are used loosely If they are considering moving averages at every forward pass.

As far as I understand it, it is always the mini-batch that the mean and STD are derived from. Yet, the mini-batch can be as large as the whole dataset, hence they are using the two terms (batch and mini-batch) interchangeably. It's even required statistically to have a relatively large mini-batch to get representative mean and STD. Moreover, at inference time, we need the mean and STD of the whole dataset, (continue in next comment).

but since we are using mini-batches, we calculate the moving average and STD at every call of the layer during training, this is valid regardless of the vectorization, ..well ...because it is a moving measure. Still, how many mini-batches should be involved to calculate the latter? Here we introduce the momentum to decide the history that should be involved, where 0.0 means calculate the moving average and STD only for the very last mini-batch (hence highly biased), and 1 is for the very first mini-batch.


text:
Backpropagation after N sequential input-output pass
I'm trying to train a Neural Network in a particular situation -- similar to a genetic algorithm domain as far as I know.
I have to run a simulation with a length of $K$ steps.
I have a neural network $N$ that at each time step is used to produce an output, so that:
$$
o_{t+1} = N(i_{t})
$$
$i_t$ is a feature vector built upon $o_{t-1}$, and $i_0$ is given.
My ground-truth value is $o_k$, namely the right value at the end of the simulation. So, I can evaluate the loss (e.g. MSE) only at the end of the simulation.
Suppose to fix k to 3, the evaluation is: $N(N(N(i_0))$
because:
$$
o_1 = N(i_o) \\
o_2 = N(i_1) \\
o_3 = N(i_2)
$$
So my questions are:

does it have any sense to apply backpropagation in these settings?
if yes, what happens to the gradients?

Practically, in some simple situations, the backpropagation seems to work, but in others, the gradients explode or vanishes

Is it possible that you could train it in steps, starting from short simulations and then working up to longer ones?

In some case it is possible, but in the general case no..

I/ Does it has any sense to apply backpropagation in these settings?
If I understand correctly, this question should be "Can backpropagation return the gradient for every node of these networks?".
It depends on your network $N$ if it is differentiable with all inputs $i_t$ so the backpropagation can be guaranteed that there are the gradient values at every node of the network.
II/ If yes, what happens to the gradients?
We deal with the vanishing or exploding gradient after we make sure that the gradient is exist. The reason for this phenomenon in a neural network can come various, some typical causes are:

Very deep neural networks (a large number of layers):
Assume a neural network with $n$-th layers, the backpropagation is followed the Markov Chain's rule can be show as:

$$\frac{\partial loss}{\partial w_t} = \frac{\partial loss}{\partial l_{n}} \times \frac{\partial l_{n}}{\partial l_{n-1}}\times...\times \frac{\partial l_{t+1}}{\partial l_{t}}\times \frac{\partial l_t}{\partial w_t}$$
It's easy to see that the gradient is scaled exponentially for each layer, so if each gradient value is larger than 1, the gradient will become $\inf$ with $n\rightarrow \inf$ (exploding) and $0$ in vice versa.

Activation function:
The image below shows the difference between different activation functions. It's easy to see that the logistic function such as Sigmoid or Tanh is limited in the range $[0,1]$ (in case sigmoid) or $[-1,1]$ in case Tanh. Therefore, if the output from a node of the neural network is larger than 2 or smaller than -2, the gradient will become nearly zero (vanishing) because the output is always the same.

Solution:

Replace it by the simple function such as ReLU: $o_{t+1} = max(0, N(i_t))$. However, the output from ReLU will be $0$ if it's lower than $0$, so there are many variants of ReLU to solve this problem, you can find them easily by the keyword "ReLU family" on google (example).

There are other methods or strategies to duel with vanishing/exploding gradient, it also depends on your model or your data. The answer can be more detailed if you give more information.


text:
Training on the dataset in parts vs training on the whole dataset
What is the difference between these two situations? are they the same ?
#1 : train a model 20 epochs on the whole dataset
#2 : divide dataset into n-parts then train the model 20 epochs on each part
20 is a random number just for clarification. do we get the same result (accuracy) between these two situations? and why ?
Side note: this question was raised in my mind when I faced a problem: dataset is bigger than the storage space. So I want to divide it into 4 parts and train the model on each part. But does this effect on accuracy ? does this method of training is correct ?


text:
Does average loss function in GAN training is just an approximation of value function and does not ensure convergence of generator and discriminator?
The value function on which convergence has been proved by the original paper of GAN is
$$\min_G \max_DV(D, G) = \mathbb{E}_{x ∼ P_{data}}[\log D(x)] +  \mathbb{E}_{z ∼ p_z}[log (1 - D(G(z)))]$$
and the loss function used in training are
$$\max L(D) =  \frac{1}{m} \sum_{i=1}^{m}\left[\log D\left(\boldsymbol{x}^{(i)}\right)+\log \left(1-D\left(G\left(\boldsymbol{z}^{(i)}\right)\right)\right)\right]$$
$$\min L(G) =  \frac{1}{m} \sum_{i=1}^{m}\left[\log \left(1-D\left(G\left(\boldsymbol{z}^{(i)}\right)\right)\right)\right]$$
where $\{z^{(1)}, z^{(2)}, z^{(3)}, \cdots, z^{(m)}\}$ and  $\{x^{(1)}, x^{(2)}, x^{(3)}, \cdots, x^{(m)}\}$  ate the noise samples and data samples for a mini-batch respectively.
I found after analyzing some questions 1, 2 on our main site that the loss function used for training is just an approximation of the value function and are not same in formal sense.
Is it true? If yes, what is the reason behind the disparity? Is the loss function used for implementation also ensures convergence?

What do you mean by "an approximation of the value function and are not same in formal sense"? The original loss is nothing more than the BCE loss. The generator's loss is more complex: D plays the role of some kind of dynamic loss function.

@ArayKarjauv I mean, they both are not same but latter is inspired on former. What do you mean by original loss function? Is it value function?

Do you mean the difference between the loss functions for D and G?

@ArayKarjauv No, the difference between the actual value function and the loss function used for implementation.

They are identical. The notation is slightly different. The only difference is the loss for G.

@ArayKarjauv How can be the same? The value function is using the actual probabilities and in implementation, we are using just 1/m!

@ArayKarjauv Log part is the same. But when comes to the probability of samples, they are different right?

@ArayKarjauv Are you saying that both are BCE? If yes then I may be facing difficulty in deriving the loss functions used for implementation from the theoretical value function used...

I decided to write a complete answer. Feel free to leave your feedback.

Expected value can be thought of as a weighted average of outcomes. Thus, expectation and mean are the same thing, if each outcome has the same probability (which is $\frac{1}{m}$), so we can replace it with a sum divided by $m$. We can rewrite the equation:
$$\min_G \max_DV(D, G) = \mathbb{E}_{x ∼ P_{data}}[\log D(x)] +  \mathbb{E}_{z ∼ p_z}[log (1 - D(G(z)))]$$
First, we sample minibatch of size $m$ for $\boldsymbol{x} \sim P_{data}$ and $\boldsymbol{z} \sim \mathcal{N(0, 1)}$. Now we can replace the expectation with the sum:
$$
\begin{align*}
\min_G \max_DV(D, G) &= \sum_{i=1}^{m}\left[p(\boldsymbol{x}^{(i)})\log D(\boldsymbol{x}^{(i)})\right] +  \sum_{i=1}^{m}\left[p(\boldsymbol{z}^{(i)})log (1 - D(G(\boldsymbol{z}^{(i)})))\right] \\
&= \sum_{i=1}^{m}\left[\frac{1}{m}\log D(\boldsymbol{x}^{(i)})\right] + \sum_{i=1}^{m}[\frac{1}{m}log (1 - D(G(\boldsymbol{z}^{(i)})))]\\
 &=\frac{1}{m}\sum_{i=1}^{m}\left[\log D(\boldsymbol{x}^{(i)}) + log (1 - D(G(\boldsymbol{z}^{(i)})))\right]
\end{align*}
$$

Binary cross entropy defined as follows:
$$H(p, q) = \operatorname{E}_p[-\log q] = H(p) + D_{\mathrm{KL}}(p \| q)=-\sum_x p(x)\log q(x)$$
Since we have a binary classification problem (fake/real), we can define $p \in \{y,1-y\}$ and $q \in \{\hat{y}, 1-\hat{y}\}$ and rewriting coros entropy as follows:
$$H(p, q)=-\sum_x p_x \log q_x =-y\log \hat{y}-(1-y)\log (1-\hat{y})$$
which is nothing but logistic loss. Since we know the source of our data (either real or fake), we can replace labels $y$ for real and fake with 1. We then get:
$$\min_G\max_D L =  \frac{1}{m} \sum_{i=1}^{m}\left[1\cdot\log D\left(\boldsymbol{x}^{(i)}\right)+1\cdot\log \left(1-D\left(G\left(\boldsymbol{z}^{(i)}\right)\right)\right)\right]
$$
This is the original loss. The first term in the equation gets always real images, while the second gets only generated. Hence, both terms have corresponding true labels. Read this article for more details.
Since the first term does not depend on $G$, we can rewrite it as follows:
$$\max L(D) =  \frac{1}{m} \sum_{i=1}^{m}\left[\log D\left(\boldsymbol{x}^{(i)}\right)+\log \left(1-D\left(G\left(\boldsymbol{z}^{(i)}\right)\right)\right)\right]$$
$$\min L(G) =  \frac{1}{m} \sum_{i=1}^{m}\left[\log \left(1-D\left(G\left(\boldsymbol{z}^{(i)}\right)\right)\right)\right]$$

My doubt is in fixing up the probabilities of noise, generated and real samples as $\dfrac{1}{m}$, where $m$ is just a hyperparameter we are selecting and does not depends on actual probability. So, how can we take that?

I guess I answered the question about $m$ - we assume each image has the same probability (we can also sample images with certain probabilities). This is not so important. We could have made a batch size 1, but as far as I know, batch optimization is less prone to getting stuck in local minima. As for the noise, it is only used to generate images, so we can replace $G(z)$ with $\hat{x} \sim P_g$

yeah, I understood. But I am not sure whether considering our own probability is theoretically correct or nor. So, I am still in confusion.

@hanugm I don't think the mean is related to the probabilities of the data in this case. It has more to do with the importance of the gradient in the batch. The gradient for all data points is equally important. You can update your model using one data point at a time, so you get rid of the mean. The likelihood of the data is encoded in your dataset. If your dataset is unbalanced, you can sample a specific class with a certain probability, for example, using WeightedRandomSampler in PyTorch.


text:
Which probability distribution a generator in Generative Adversarial Network (GAN) is capturing: dataset or ground truth?
Consider the following statement from the abstract of the paper titled Generative Adversarial Nets

We propose a new framework for estimating generative models via an
adversarial process, in which we simultaneously train two models: a
generative model $G$ that captures the data distribution, and a
discriminative model $D$ that estimates the probability that a sample
came from the training data rather than $G$.

Let $D'$ be a dataset of $n$ digital and discrete* images, each of size $C \times H \times W$. Suppose the generative adversarial network is trained on the dataset $D'$.
Our sample space (or data set) is $D' = \{I_1, I_2, I_3, \cdots, I_n\}$, where $I_j$ is the $j^{th}$ image for $1 \le j \le n$
The random variables are $X_1, X_2, X_3, \cdots, X_{CHW}$ where
$$X_i \in \{a, a+1, a+2, \cdots, b\} =\text{ intensity of }i^{th} \text{ pixel;} \text{     for  } 1 \le i \le CHW$$
Since we have dataset $D'$, we can calculate the joint distribution $p_{data\_set}$ having $(b-a+1)^{CHW}$ parameters calculated from the dataset. But the parameters of the original image distribution $p_{ground_{D^{'}}} $is not equal to the $p_{D'}$

Simple example:
Suppose I flipped an unbiased coin 100 times and I got 45 heads, 55 tails, then $P_{data\_set}(H) = \dfrac{45}{100}$ and $P_{data\_set}(T) = \dfrac{55}{100}$. So,  $P_{data\_set} = \{\dfrac{45}{100}, \dfrac{55}{100}\}$
but the ground truth probability distribution is $P_{ground}(H) = \dfrac{50}{100}$ and $P_{ground}(T) = \dfrac{50}{100}$.  So,  $P_{ground} = \{\dfrac{50}{100}, \dfrac{50}{100}\}$

Which distribution is our generator capturing by the end? Is it the probability distribution calculated based on our dataset $p_{data\_set}$ of images or the actual probability distribution $p_{ground}$?
How to understand the act of capturing here? Does it only mean to be behaving (generating instances) in the same way as data distribution?

Suppose I design a machine that is capable to generate equal number of $0$'a as equal number of $1$'s for sufficiently large trails, then I can say that my machine captured the coin toss probability distribution?

discrete images refers to the images whose pixel values takes finite number of integer values.


text:
Are there any Generative Adversarial Networks without Multi Layer Perceptrons?
Although the main stream research is on Generative Adversarial Networks(GANs) using Multi Layer Percepteons (MLPs). The original paper titled Generative Adversarial Nets clealry says, in abstract, that GAN is possible with out MLP also

In the space of arbitrary functions $G$ and $D$, a unique solution
exists, with $G$ recovering the training data distribution and $D$
equal to $\dfrac{1}{2}$ everywhere. In the case where G and D are defined
by multilayer perceptrons, the entire system can be trained with backpropagation.

Are there any research papers that uses models other than MLPs and are comparatively successful?

FCC-GAN: A Fully Connected and Convolutional Net
Architecture for GANs

@Yahya It is also using MLP right?

I mix of two, but mainly a Convolution NN, look page 4, Table 1, the right column, for the architecture.

@Yahya Oh! In the question, i used MLP as a representative term for all forms of neural networks.

So Do you mean a non-ANN based model, such as a SVM or a simple Linear Model?

@Yahya yes exactly.

Semi-supervised Generative Adversarial Networks Based on Scalable Support Vector Machines and Manifold Regularization. Here the discriminator uses SVM classifier.

@Yahya Feel free to provide a formal answer below (as comments are temporary), if you have an answer to this question ;)

Someone suggested to me to try using a Java compiler as a discriminator (to generate Java code), but I had to point out it wasn't differentiable and couldn't be weakened for training.


text:
Why is noise vector represented by letter $z$?
Most of the notations in Artificial Intelligence are borrowed from the mathematics.
$x$ stands for input (vector), $y$ stands for output (vector) etc., and the list is long.
But, I am not sure whether $z$ has any (widely used) role in mathematics.
Is there any reason behind the usage of letter $z$ to represent a noise vector?  Or is it just selected randomly without any reason?

Letter in mathematical discplines are often chosen at random, maybe zzzzz - resembled the noise to the inventor)

Please try to comment whether down-votes reflect the low quality of the question or off-topic-ness of the question or some other. It will be helpful for me and the community.

I don't think there's any rationale behind the usage of the letter $z$ to denote the noise (which sometimes is also denoted by $\epsilon$ in other contexts), apart from the fact that $x$ and $y$ are already being used and that the letters $x$, $y$, $z$ and $w$ are often used to denote variables in mathematics. In particular, in machine learning, $x$ and $y$ are often used to denote the inputs and outputs (or labels) respectively, while $w$ often denotes the parameters (although $\theta$ is also used for that).
In other words, it's just a convention (e.g. $z$ is also used to denote the hidden variable in the VAE paper). Even if it wasn't and someone used $z$ as a mnemonic letter or for some particular reason, I wouldn't lose too much time on this issue, as another author may very well use any other letter to refer to the same concept. It's important to be a bit flexible when it comes to notation in mathematics, otherwise, you may easily get lost.


text:
What are the iid random variables for a dataset in the GAN framework?
I am trying to understand why mean is used for expectation in training Generative Adversarial Networks.
The answer tells that it is due to the law of large numbers which is based on the assumption that random variables are independent and identically distributed.

If I have a dataset of all possible $32 \times 32$ grayscale images. Then my sample space consists of $256^{32 \times 32}$ elements. Suppose I define 1024 random variables as
$$X_i = \text{ intensity of } i^{th} \text{ pixel for } 1 \le i \le 1024$$
Then it is clear that all the random variables are iid since

$X_i \perp  X_j$ for all $i, j$ such that $i \ne j$ and
$p(X_i = k) = \dfrac{1}{256}$ for all $i$

But these properties do not hold if I take a dataset of (say flower) images since pixel intensities are not independent of each other and the intensity values are not uniformly distributed as well.
Then how can the law of large numbers be applicable for GAN as the dataset (sample space) does not cover all the possible elements? If I am wrong, then what is the sample space they are considering and what are the random variables they are using implicitly that leads to the satisfaction of iid condition and then the law of large numbers?

GANs do not learn pixels directly, but a mapping between latent features (e.g., color, rotation, width, etc.) and an output image.

@ArayKarjauv But the value function is saying $x∼p_{data}$ under the expectation, which then I understood $x$ as a random vector containing random variables which says the intensities of pixels. Is it wrong?

$x$ is an image with abstract attributes, which a GAN aims to learn. Specifically, GAN tries to approximate a lower dimensional manifold that represents the real data. Of course, all modalities must be evenly represented in the dataset, but I think that the mean is rather related to the loss than to the iid of the random variables.

@ArayKarjauv Your second comment is close to a full answer IMO

@ArayKarjauv So, is this answer wrong?

I find this answer incomplete.  It just discusses the expectation itself, as the author stated at the beginning.

Independent and identically distributed random variables share the same probability distribution and each item doesn’t influence or provide insight about the value of the next item you measure. The most common example is a coin toss: as you flip the coin, one outcome does not influence or predict the next one.
As for a dataset of flowers, we assume that the photos were made independently and all kinds of flowers are evenly represented, which in practice is not true. Each flower won't be represented in the same lighting condition and camera view.
Each pixel in an image is highly dependent on the neighboring pixels (and sometimes distant ones, e.g., when generating faces as they are usual symmetric) and the main goal of GAN is to discover this dependence by exploring a latent features manifold that represents abstract attributes of the real data, and to find a mapping between these features and output pixels. Of course, if some of the features are poorly represented, the network will not be able to generate them correctly. For instance, to increase the fidelity, there is the so-called truncation trick, where we sample the latent features vector $z$ from a truncated normal distribution, which ignores rare features. But on the other hand, it reduces diversity.
As for the loss function,
$$\nabla_{\theta_{d}} \frac{1}{m} \sum_{i=1}^{m}\left[\log D\left(\boldsymbol{x}^{(i)}\right)+\log \left(1-D\left(G\left(\boldsymbol{z}^{(i)}\right)\right)\right)\right]$$
the mean is associated with the loss rather than the iid of the random variables and is used to obtain the average gradient of a batch. Here are related questions: one, tow.


text:
What exactly is a Parzen?
I came across the term "Parzen" while reading the research paper titled Generative Adversarial Nets. It has been used in the research paper in two contexts.
#1: In phrase "Parzen window"

We estimate probability of the test set data under $p_g$ by fitting a
Gaussian Parzen window to the samples generated with $G$ and reporting
the log-likelihood under this distribution.

#2: In phrase "Parzen density estimation"

Evaluating $p(x)$ in Generative autoencoders and Adversarial models: Not
explicitly represented, may be approximated with Parzen density
estimation

Is there any definition for the word Parzen and how is it related to the probability distributions?

Parzen was a statistician, who worked in spectral analysis and stochastic processes. I don't know if he invented them, but those windows and probability density esimation methods are named after him.
See also his Wikipedia entry.


text:
How to create a neural network from a set of equations?
Say I have these equations:
$$x_1 = x_2 + 2y_1 + b$$
$$x_2 = y_2 + c$$
$$y_1 = z + a$$
$$y_2 = y_3 + d$$
$$z = z_1 + e$$
$x_1$ depends on $x_2$ (depends on $y_2$ (depends on $y_3$)) and $y_1$ (depends on $z$ (depends on $z_1$)).
$x_1$ is my final equation and $y_3$ and $z_1$ are my initial variables.
How do I represent them in a neural network? My final aim is to backtrack from $x_1$, and see what change of an amount $n$ in $x_1$ resulted from which of $y_3$ or $z_1$.
All these variables are item prices in the real world.
My inputs are $z_1$ and $y_3$ and my output is $x_1. z_1$ and $y_3$ are prices and the final output $x_1$ is also a price.

Do you have a particular reason to prefer a neural network solution as opposed to a traditional solver for a linear system? If you're not particularly set on neural networks, there are efficient ways to solve linear systems in a much simpler way.

I want a neural network solution so that I can trace back any delta change in x1 to a major change in either z1 or y3. This neural network will train on historic price data, like last 10 years, and will be used to backtrack changes in x1 thereafter. Do you think a neural network is good for this or is there anything else that is better?

All modern frameworks for deep learning (PyTorch, Jax, Tensorflow) support automatic differentation. These operations can be easily implemented. Here I write, how it would look like in PyTorch:
class Net(nn.Module):

    def __init__(self):
         super().__init__()

         self.a = nn.Parameter(torch.randn(1))
         self.b = nn.Parameter(torch.randn(1))
         self.c = nn.Parameter(torch.randn(1))
         self.d = nn.Parameter(torch.randn(1))
         self.e = nn.Parameter(torch.randn(1))

   def forward(self, z1, y3):
       z = z1 + self.e
       y2 = y3 + self.d
       y1 = z + self.a
       x2 = y2 + self.c
       x1 = x2 + 2 * y1 + self.b
       return x1

And the use case is the following, say:
net = Net()
net(torch.ones(1), 2 * torch.ones(1))


text:
Recursive Least squares (RLS) for mini batch
For my application I am considering a learning problem where I simulate a bunch of episodes say '$n$' first, and than carry out the recursive least squares update. Similar to $TD(1)$.
I know that RLS can be used to update parameters being learned as they arrive. This can be done efficiently for single data point and the derivations are easily available online and also easy to understand.
However for my case I am looking for same equations when data arrive as a mini batch and not a single data point at a time. I could not find any material regarding RLS for mini batches.
According to my understanding the same equations can be also used by appropriately considering matrix dimensions. However I do not know if this is valid.
What are the alternatives to be used?

I had just successfully implemented RLS when I found this question. In fact, I wanted to extend it to the mini batch setting, just like you (which I did!).
RLS is derived using the Sherman-Morrison formula, which is specific for rank-1 updates. We'd need to use its generalization for rank-n updates, the Woodbury Matrix Identity. By looking at the RLS formula, the Sherman-Morrison formula becomes evident, and the conversion can be made quite easily. My solution in Python with PyTorch is this:
R = (R - (R @ x.T @ torch.linalg.inv(l * torch.eye(k) + x @ R @ x.T) @ x @ R) ) / l

where $k$ is the mini batch size and $l$ is the forgetting factor (which can be set to 1 to give the same result as OLS). The transposes are inverted, since I use batch_size x input_dim format for $x$. It gives me the same result as the rank-1 updates, except for a $-0.05\%$ difference in accuracy on MNIST in my experiments, which I attribute to rounding errors and such. If anyone finds any error in my formula (or any improvement to it), please, leave a comment :)


text:
Feeding the output back to input in 3D CNN model
I am currently designing a Model which takes Input 3D Grid and Model Output at $t-1$. The model figure is described below

I have two thoughts in training the model for above situation.

Feed output $t-1$ from ground truth with some noise. And, during testing feed output of the model as previous output at $t-1$. Maybe we can fine tune with model output when training loss is sufficiently low.

Feed model output to next stage as $t-1$ output. But I am not sure if this works.

The situation is similar to RNNs but I am using 3D CNN’s here. I don’t know if RNN can be used here. How can I train such a model.

What is the word looking like shoref in between the two 3D-Encoders?

Sorry, It is "shared" not shoref. My bad. These are shared convolutions.


text:
Why not make the training set and validation set one if their roles are similar?
If the validation set is used to tune the hyperparameters and the training set adjusts the weights, why don't they be one thing as they have a similar role, as in improving the model?

I think this is best explained using an analogy. Also you seen to have the misconception that you don't tune hyper-parameters for training data. You want to increase the accuracy of the training set AND validation set at the same time, but the validation set is more important so you want to maximise that accuracy more.
Imagine you had a toddler, and you were trying to teach them what an apple looks like. You have 10 pictures of apples, and for 45 years you sit them in a room and show them these apples. After all this time they will get to know the apples so well, that even the minutest differences in the photos would be noticed. When you try to see how the toddler generalised (how well it can use what it saw in training to evaluate real examples), it's absolutely terrible, because after all that time how could anything be an apple but the 10 it had seen?
So to combat this, you might think to reduce the number of years you spent showing the toddler the training set (the 10 apples), maybe that will allow them to generalise better (prevent over-fitting)? But you need a way to validate that this is actually helping, on unseen data (novel apples). That's where the 5 unseen apples come in to play (the validation set). You measure the accuracy of this unseen data to get an actual idea of how the toddler has learnt, because in a real example, the toddler isn't going to have seen the apple before, so it's important to know how it will handle unseen data.
That brings us finally to the testing set. The issue with what I've described above is maybe there's some kind of bias in the validation set, maybe those apples are slightly more round than usual. Of course you want the maximum accuracy on the validation set, so you tune everything to increase this accuracy, but this incurs a bias to the validation set. Maybe the parameters you chose are good only for this validation set. To make sure this isn't the case, you try and increase your accuracy as much as possible on the validation set, then after that you test the true accuracy on the testing set. This ensures you can't just tune your hyper-parameters to the validation set and call it a day. You need real generalisation to increase both validation AND testing set accuracy.

Idea is to optimize with regards to unseen data in each step in order to avoid overfitting and data leakage so that the final network will be most generalizable to novel data.
First, you initialize your network weights randomly. For those weights, training data is unseen so network is optimized with regards to loss function that is calculated using training data. This was the first step.
Second, you would like to optimize hyperparameters, the parameters you use to train your network in first step. The network already worked hard to do its best in the first step while learning weights. If you use the same dataset in this step, it will have even more flexibility to fit even better to training data. But this will result in high variance, and network will perform poorly on unseen data.
For this reason, you split your data into train, dev and test. Train network with train data, optimize it with dev data and finally, evaluate with test data, never touching it until the very last step.

Why would the use of the training data for hyper-parameter tuning result in high variance? You should probably elaborate on this part because it's not clear the variance of which random variable you're referring to and why that's even the case.

@crinix so you mean that dividing the training set and dev set allows overfitting to be reduced?

@nbro Optimizing hyper-parameters wrt trainingset causes high-variance and overfitting yes. It provides the model with more means to reduce the loss on training data, while increasing loss for unseen data.
OmarZayed yes.


text:
Where does batch normalization layers present in a neural network?
Batch normalization is a procedure widely used to train neural networks. Mean and standard deviation are calculated in this step of training.
Since we train a neural network by dividing training data into batches, we use the word batch normalization as we consider a batch of training vectors at a time.
My doubt is about the position of batch normalization layers in a neural network.
Is it present before the input layer only? Or is it before every layer? Or is it dependent on the underlying task?
Suppose there is a neural network of 4 layers: $I \rightarrow h1 \rightarrow h2 \rightarrow h3 \rightarrow O$
Which one of the following is true
$$bn \rightarrow I \rightarrow h1 \rightarrow h2 \rightarrow h3 \rightarrow O$$
$$bn1 \rightarrow I \rightarrow bn2 \rightarrow  h1 \rightarrow bn3 \rightarrow  h2 \rightarrow bn4 \rightarrow  h3 \rightarrow bn5 \rightarrow  O$$
Here $I$ stands for input layer, $h$ for the hidden layer, $O$ for output layer, and $bn$ for batch normalization layer.

Batch normalization can be applied to any input
or hidden layer in a network. p:318

I think the answer to your question is much more a rule of thumb than an appropriate analytical answer. First of all, I would like to remark that Batch Normalization [1] are applied most commonly to convolutional layer, constituting what is called a "convolutional block" (Convolution + Batch Normalization + Activation). Thus, for giving you an idea on where to put the normalization layers, I will analyze three papers that make use of Batch Normalization.
Unsupervised representation learning with deep convolutional generative adversarial networks [2]. In Section 3, Approach and Model Architecture, the authors make the following remark:

Third is Batch Normalization [1] which stabilizes learning by normalizing the
input to each unit to have zero mean and unit variance. This helps deal with training problems that
arise due to poor initialization and helps gradient flow in deeper models. This proved critical to get
deep generators to begin learning, preventing the generator from collapsing all samples to a single
point which is a common failure mode observed in GANs. Directly applying batchnorm to all layers
however, resulted in sample oscillation and model instability. This was avoided by not applying
batchnorm to the generator output layer and the discriminator input layer.

thus, the authors have experimented with Batch Normalization on all layers, but this approach resulted in model instability. They ultimately avoided Batch Normalization on the Generator's output layer, and Discriminator's input layer. These correspond to "raw image layers" (the output of the generator is an image, as well as the input of the discriminator). Using your notation, it would be something like:
$$G: z \rightarrow (C + BN + A)_{1} \rightarrow (C + A)_{2} \rightarrow O$$
$$D: I \rightarrow (C + A)_{1} \rightarrow (C + BN + A)_{2} \rightarrow h_{1} \rightarrow h_{2} \rightarrow O$$
where $G$ stands for Generator, $D$ for Discriminator, $C$ for convolution, $BN$ for batchnorm, $A$ for activation and $D_{i}$ for dense layers.
Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising [3]. In Figure 1., the authors show their proposed architecture that applies the same logic. Neither the first convolutional layer, neither the last use Batch Normalization. Although the authors make their point in justifying the usage of residual learning + Batch Normalization, the paper does not justify this architectural choice. The network would look something like this:
$$I \rightarrow (C + A)_{1} \rightarrow (C + BN + A)_{2} \rightarrow \cdots \rightarrow (C + BN + A)_{n} \rightarrow C \rightarrow O$$
Deep Residual Learning for Image Recognition [4]. This paper proposes a different scheme, as it applies Batch Normalization after each convolution operation on convolutional layers (thus including the input).
To sum up some papers use it only on the "hidden convolutional blocks", others on all of them. My advice is that, if you have the time, you should compare the two approaches. Remark: maybe I am unaware of some further development on the matter that gives a precise argument in favor of whether of these approaches.
References
[1] Ioffe, S., & Szegedy, C. (2015, June). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning (pp. 448-456). PMLR.
[2] Radford, A., Metz, L., & Chintala, S. (2015). Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434.
[3] Zhang, K., Zuo, W., Chen, Y., Meng, D., & Zhang, L. (2017). Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE transactions on image processing, 26(7), 3142-3155.
[4] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).


text:
Scrabble rack observation with MuZero
Currently I'm trying to implement Scrabble with MuZero.
The $15 \times 15$ game board observation (as input) is of size $27 \times15 \times15$ (26 letters + 1 wildcard) with a value of 0 or 1.
However I'm having difficulties finding a suitable way to encode the player's rack of letters (Always 7 letters on the rack).
The available tiles are: 26 letters $(A-Z)$ and 1 wildcard.
A rack can also contain multiple tiles of the same letter.
Example: rack of player 1 is  $[A,A,C,E,T,T,H] -> A:2x, C:1x, E:1x, T:2x, H:1x$
How can I represent a rack of tiles as a $(? \times)15 \times15$ (or other board size) matrix ?

Use one hot encoding for each position, shape $(7, 27)$.
Stack these two dimensions, shape $(189)$.
Tile (replicate) this vector into images of the same resolution, now shape $(15, 15, 189)$
Stack them with your other observation, final shape $(15, 15, 216)$.

Another way is instead of replicating the $(189)$ vector along two axis, you can squeeze them into one single plane of $15 * 15 = 225$ values, and pad the unfilled part with zeros.


text:
Building an AI that predicts the pronunciation of words
I want to create an AI that converts words to  International Phonetic Alphabet (IPA), but I am not sure which architecture I am supposed to use.
It is not possible to translate the characters one by one since there are multiple characters in the source word corresponding to one IPA character. There are solutions for this kind of problem, for example using an Encoder that encodes the content of the input which the decoder then translates, but I am uncertain if this isn't too abstract for this problem.
Can anyone think of a suitable solution for this task?

You could try a finite state machine.

I don't think what you're suggesting is too abstract -- this sounds suitable for seq2seq architectures, such as those used in neural machine translation. It might not be the simplest possible model for the job though. You might be able to adapt this tutorial.

You may find this useful, what you describe is (I think) the extraction of grapheme from text and mapping of grapheme to phonemes. See https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/tts/intro.html

So as the answer below mentions it'sa  seq2seq problem, i.e. commonly LSTM or perhaps a trasnformer encoder/decoder network.

Look for sequence-to-sequence modelling, aka, seq2seq.
https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequence-learning-in-keras.html
https://en.wikipedia.org/wiki/Seq2seq


text:
What is a conditional random field?
I new in machine learning, especially in Conditional Random Fields (CRF).
I have read several articles and papers and in there is always associated with HMM and sequences classification. I don't really understand mathematics, especially in the annoying formula. So I can't understand the process. Where I need to start to understand CRFs??
I want to make an information extraction application using CRF Named Entity Recognition (NER).
I got some tutorial for that: https://eli5.readthedocs.io/en/latest/tutorials/sklearn_crfsuite.html#training-data
But I don't know the proses each step, like training proses, evaluation, and testing
I use this code :
  data_frame = eli5.format_as_dataframes(
            eli5.explain_weights_sklearn_crfsuite(self.crf))

Targets

Transition Features

How to get that number ?
and 1 more thing makes me confused:
crf = sklearn_crfsuite.CRF(
    algorithm='lbfgs',
    c1=0.1,
    c2=0.1,
    max_iterations=20,
    all_possible_transitions=False,
)

What is the algorithm lbfgs? Is the CRF not an algorithm? Why do I need lbfgs? What is exactly a conditional random field?

I will provide an answer later, but you will have to wait (unless someone provides an answer meanwhile). Anyway, it is ok to be confused when it comes to CRFs, especially if you don't have a solid mathematical background.

okay, thank you for responding I really appreciate that @nbro

heyy i still waiting your answer @nbro :(

Hey! Sorry for having made you wait for a long time without warning you. CRFs involve a lot of concepts. So, a satisfactory will possibly be quite long, which would have required some time. Meanwhile, I couldn't have spent so much time to give this answer. I will try to give you an answer this weekend.

@nbro Ok, i will waiting your answer :(

@nbro hey :( i am still waiting


text:
identifying pattern in datasets
i am new to machine learning. i'm trying to identify driving pattern through accelerometer and gyroscope sensor. i have been collecting the data of both the sensors and have been storing them in .csv extension. i am not able to identify a pattern in the datasets since it has a lot of datas. i have three independent variables of accelerometer and with that i need to identify the sudden acceleration and sudden breaking and i have three independent variables of gyroscope and i need to identify aggressive turns. can you suggest as to how i need to analyse the pattern and find a algorithm which suits my requirement. this is how the dataset is

is your data sequential? does it have time? can you plot it on time scale?

yes,it does have time. The data table is as follows; time,speed,Ax,Ay,Az,Gx,Gy,Gz


text:
Could error surface shape be useful to detect which local minima is better for generalization?
The following plot shows error function output based on system weights.
Two equal local minima are shown in green pointers. Note that the red dots are not related to the question.
Does the right one generalize better compared to the left one?
My assumption is that for the right minimum if the weights change, the overall error increases less compared to the left minimum point. Would this somehow mean the system does better generalization if the right one is chosen as the optimum minimum?

In general I agree with @nbro answer, nevertheless sticking strictly to this specific question I'd like to share some speculations:

what the author of the question provides us with is the Loss Function Shape so I'll try to use the full information here to compare the 2 minima
looking at the LF steepness we observe the Left LM is in a steeper region than the Right LM: how can this be interpreted?
I'd interpret the LF Steepness as a Measure of Parametrization Stability: in fact perturbating the Parametrization slightly has a bigger effect on Left LM observed performance rather than the Right LM one
when the NN is running "in production" typically the parametrization is fixed (in most typical application, weights are not changed out of the training phase) so you should not be concerned about parametrization stability,
however I'm one of those believing the Flat Minima - Schmidhuber 1997 idea that Local Minima Flatness is connected to generalization property

However it is important to observe this is still a very open and interesting question as in Sharp Minima Can Generalize For Deep Nets Dinh 2017 et al. demonstrated
it is not just about flatness, because reparametrization of the NN model, though preserving minima locations, change its shape so basically sharp minima might be transformed into flat ones without affecting network performance

Even though this "in fact perturbating the Parametrization slightly has a bigger effect on Left LM observed performance rather than the Right LM one" was already the assumption of the OP, I appreciate more this answer because you cite some research work.

Overviewing the reference kindly provided by @NicolaBernini, I am actually happy that, as a beginner in AI, this question jumped in my head. Thanks for the answer.

I don't think that the concept of generalization is (directly) related to the "shape" of the function close to the point where it attains a minimum.
The concept of generalisation refers to when a trained model is able "perform well" on unseen data (that is, data not seen during the training phase). If a trained model does not generalise well, then it might have "overfitted" or "underfitted".
Overfitting means that the model performs well on the training data, but not on other data. In other words, in the case of overfitting, the model learns the structure of the training data, but not of the whole population (of interest) or it learns the "noise" (that is, the data which is not part of the population of interest) present in the training data. This also implies that the training data is not a "good" sample of the population, that is, it doesn't "summarise" well all characteristics of the population. In practice, this is often the case (for large populations).
Underfitting occurs when the model is not even able to learn enough about the training data during the training phase. For example, underfitting occurs when you try to train a linear model but the data does not have a linear relationship.
So, being able to generalise (or not) depends on the model (including the number of parameters), but also on the data you are given.

Does the right one generalize better compared to the left one?

The shape of the function will only be useful during the training phase. More specifically, in this case, you might reach one of the two minima faster than the other (depending also on the optimisation algorithms, model, etc., that you use).
I would like to note that, in machine learning, we often minimise a function of functions (which in mathematics is called a "functional"). Why is that? For "simplicity", consider a simple neural network (NN) model (e.g. a multi-layer perceptron). We usually train such NN using gradient descent (or one of its variants) by minimising a function (e.g. the mean squared error). Essentially, when we train such a NN, we want to find the function (which is represented by the parameters of the NN) that minimises e.g. the MSE. In this case, the mean squared error (MSE) is the functional we attempt to minimise. Note that the MSE is a function of the parameters of the current NN and that a NN is a model that represents a function.
Let's go back to your question. If, during the training phase, you get one minimum rather than the other, you will get different NNs (say $A$ and $B$). Note, again, that the point where our MSE function attains one its minima, in this context of training a NN (and often, in general, in machine learning) is a function (and not just a scalar). I know that, at the beginning (and if you're not a math guy), it might be difficult to think of functions as points (actually, more precisely, we should call them "vectors") that minimise other functions, but this concept exists and actually underlies a great amount of machine learning techniques (e.g. training NNs using back-propagation by minimising a cost function).
So, why do I think that we can't say much about the generalisation ability of NNs $A$ and $B$?
Suppose that we have access, after the training phase, to both NNs $A$ and $B$ (that is, the ones that correspond to the "points" where your function attains the two (visible) minima. $A$ and $B$ will perform equally well on the training data (or even validation data). We might then conclude that both generalise in the same way. But this might be a wrong conclusion because both the training and the validation data (as I mentioned above) might not be good samples of the population. So, we can't really say which NN (or "minima"), $A$ or $B$, generalises better. They might generalise more or less in the same way (according to their performance e.g. on the validation dataset) or not (because the validation dataset is not a good sample of the population).
To conclude, the concept of generalisation is a little bit more complex than function minimisation, because it is also related to data. In machine learning, we often minimise a functional (a function whose parameters are functions). If your functional attains the same minimum at two different points (actually, functions), during training, one might be faster to reach than the other. However, if you have two NNs that represent these two minima, we can't say much about their generalisation ability. Essentially, we can just hope that the validation dataset is a good sample of the population.

thanks for the detailed answer, I will try to understand your answer and come back here soon.

the details provided in this answer convinced me that there could be more intuitions and science related to my question that I have to consider. Although it does not satisfy me.


text:
Difference in continuing and episodic cases in Sutton and Barto - Introduction to RL, exercise 3.5

Excercise 3.5 The equastions in Section 3.1 are for the continuing
  case and need to be modified (very slightly) to apply to episodic
  tasks. Show that you know the modifications needed by giving the
  modified version of (3.3).

$\displaystyle\sum_{s^{\prime} \in S} \displaystyle\sum_{r \in R} = p(s^{\prime}, r
| s,a) = 1$ , for all $s\in S, a \in A(s)$    (3.3)
Is it just about final states? So for $s \in S$ when S is not final?

Is it just about final states? So for $s \in S$ when S is not final?

You are thinking the right way, but to represent what you mean you don't need to write out "when $s$ is not final" - although that would be fine (and is used in some places), there is a more concise way of saying that given to you by the book.
As this is a formal exercise from the book, I don't want to write out an answer that could be cut&paste for all students.
Instead I suggest you take a look at the notations section at the beginning of the book, and find how Sutton & Barto use different set labels for all states including terminal states, and all states excluding terminal states. Also, check carefully which of those sets needs to be summed over.

Thank you! I guess you mean $S$ and $S^+$, so $s \in S$ already means it's not final. Still I'm not sure what's the difference between continuing and episodic cases in the context of this equation.

@JakubBielan: Look at the sums

I think I got it :) When I'll be able to give acknowledgement for your answer, I'll remember to do it for sure

I found myself turning in cycles for a while, so to clarify Neil Slater's answer,
In the beginning of the book, $S$ means "set of non-terminal states" and $S^+$ means "set of all states, including the terminal ones".
$$\sum_{s^{\prime} \in S} \sum_{r \in R} p(s^{\prime}, r | s,a) = 1, \forall s \in S, a \in A(s) \tag{3.3}$$
That said, in eq. 3.3 when we define that $\forall s \in S$, we say that that once in a terminal state, the formula does not apply (which is obvious because no action is ever available in a terminal state by definition).
It does not however constraint the probability in how to "get" in a terminal state, and that is the key to answer the question.


text:
What determines the values of weights in a neural network?
I am trying to understand how weights are actually gotten. What is generating them in a neural network? What is the algorithm that gives them certain values?

Typically, weights are randomly initialized. Then, as the model is optimized for its given task, those weights are steadily made "better" as determined by the network's loss function. This is also referred to as "training" the neural network.
By far the most popular way of updating weights in a neural net is the backpropagation algorithm, most simply with stochastic gradient descent (SGD). Essentially, the algorithm determines how much each individual weight contributed to the network's loss. It then updates that weight in the direction that would reduce the loss.
I recommend going through Michael Nielsen's online book to learn the basics.

I agree with @PhilipRaeisghasem, in most architectures, weights are initialized in a random manner. However, some research papers suggest applying a random normal distribution initialization to the weights in the case of Convolutional Neural Networks (for computer vision).

Absolutely. I didn't think getting into specific initialization distributions would be super beneficial for someone asking such a basic question. But you're right.

I absolutely overthought that question. Thanks for replying.


text:
Is there a way to train an RL agent without any environment?
Following Deep Q-learning from Demonstrations, I'd like to avoid potentially unsafe behavior during early learning by making use of supervised learning with demonstration data. However, the implementation I'm following still uses an environment. Can I train my agent without an environment at all?

Most of this question is off-topic for this community, as we don't address implementation details. We can help you understand that paper and general things about environments, though.

Sorry, I'm newbie here. I addressed the implementation to make people see that people still uses Environments to "learn from Demonstrations" while the devil is in learning only from a certain dataset, isn't it?

I didn't mention you. Sorry again @PhilipRaeisghasem.

Hah, it's ok. I'm considering how to best answer your question.

There are many techniques for training an RL agent without explicitly interacting with an environment, some of which are cited in the paper you linked. Heck, even using experience replay like in the foundational DQN paper is a way of doing this. However, while many models utilize some sort of pre-training for the sake of safety or speed, there are a couple of reasons why an environment is also used whenever possible.
Eventually, your RL agent will be placed in an environment to take its own actions. This is why we train RL agents. I'm assuming that, per your question, learning does not happen during this phase.
Maybe your agent encounters a novel situation
Hopefully, the experience your agent learns from is extensive enough to include every possible state-action pair $(s,a)$ that your agent will ever encounter. If it isn't, your agent won't have learned about these situations, and it will always perform suboptimally in them. This lack of coverage over the state-action space could be caused by stochasticity or nonstationarity in the environment.
Maybe the teacher isn't perfect
If you don't allow your agent to learn from its own experience, it will only ever perform as well as the agent that collected the demonstration data. That's an upper bound on performance that we have no reason to set for ourselves.

Thanks a lot Philip. More specifically, I'm planning to pre-train a recommender system from a generated dataset using user embeddings and last interactions as the state and the items to recommend as the actions to take. Therefore, I'm very concerned about including every possible state-action par, because although the states may be very close one from another, it won't be the same exactly. Does it really matter?

It depends on how well your model can generalize between similar states. More training data always helps with this. Regularization could also help. But, again, it's very possible your system needs to learn from its own experience--through trial and error after deployment--in order to learn optimal behavior.

Of course, I'm only talking about the pre-deployment phase. Thanks a lot.


text:
CNN output generally has more than one category in one-hot categorization?
I'm a bit of a CNN newbie, and I'm trying to train one to image classify pictures of pretty similar looking particles. I'm making the inputs and labels by hand from a set of 48x48 grayscale images, and labeling them with a one-hot vector based on their position in the sequence (for example, the 400/1000th image might have a one-hot in the 4th position if I have 10 categories in the run). I'm using sigmoidal output activation and categorical cross entropy loss. I've played around with a few different optimizers, as well. I'm implementing in python keras.
Unfortunately, although I have pretty good accuracy numbers for the training and validation, when I actually look at the outputs being produced, it generally gives multiple categories, which is not at all what I want. For example, if I have 6 categories and a label of 3, it might give the following probability vector:
[ .99 .98  1.0  .99  0.02  0.05 ]
It was my understanding that categorical cross entropy would not allow this type of categorization, and yet it is prevalent in my code. I am under the impression that I'm doing something fundamentally wrong, but I cant figure out what. Any help would be appreciated.

when you use sigmoid activation it is applied independently to all outputs so outputs won't sum up to 1. Sigmoid is usually used for binary logistic regression where you only have 2 classes. In your case you should use softmax activation at the output which will squash outputs to range [0,1] and additionally make them sum up to 1.

I knew it would be a rookie mistake! Thank you! That fixed that issue, at least. If you want to post this as a full answer I will mark it as a correct fix.

When you use sigmoid activation it is applied independently to all outputs so outputs won't sum up to 1. Sigmoid is usually used for binary logistic regression where you only have 2 classes. In your case you should use softmax activation at the output which will squash outputs to range [0,1] and additionally make them sum up to 1.


text:
How is the policy gradient calculated in REINFORCE?
Reading Sutton and Barto, I see the following in describing policy gradients:

How is the gradient calculated with respect to an action (taken at time t)? I've read implementations of the algorithm, but conceptually I'm not sure I understand how the gradient is computed, since we need some loss function to compute the gradient.
I've seen a good PyTorch article, but I still don't understand the meaning of this gradient conceptually, and I don't know what I'm looking to implement. Any intuition that you could provide would be helpful.

Are you asking if the gradient is with respect to an action? If yes, then, no, the gradient is not with respect to an action, but with respect to the parameters (of e.g. the neural network representing your policy), $\theta_t$. Conceptually, you don't need any "loss" to compute the gradient. You just need a multivariable differentiable function. In this case, the multivariable function is your parametrised policy $\pi(A_t \mid S_t, \theta_t)$.

@nbro Conceptually I understand that, but the action is the output of the policy. So what I don’t understand is how conceptually we are determining the gradient of the policy with respect to the action it output? Further if the action is making a choice, that wouldn’t be differentiable. I looked at the implementation that used an action probability instead. But I’m still unsure what is the gradient of the policy with respect to the output of the policy. It doesn’t make sense to me yet.

@nbro I guess I see that it’s technically the gradient of the probability of selecting an action At. But still not sure how to implement that. We want to know how the probability would change with respect to changing the parameters. Maybe I’m starting to get a little insight.

I think this is just a notation or terminology issue. To train your neural network representing $\pi$, you will need a loss function (that assesses the quality of the output action), yes, but this is not explicit in Barto and Sutton's book (at least in those equations), which just states that the gradient is with respect to the parameters (variables) of the function. Barto and Sutton just present the general idea. Have a look at this reference implementation: https://github.com/pytorch/examples/blob/master/reinforcement_learning/reinforce.py.

@nbro thanks for the link; I actually think I’m almost there. In this implementation we take the (negative) log probability and multiply it by the return. We moved the return inside the gradient since it’s not dependent on $theta$.  Then we sum over the list of returns and use the summing function as the function over which we compute the gradient. So the gradient is the SUM of all future rewards wrt to theta (which, in turn, effects the choice of action). Is this correct? (I’m going back to read it again now).

@nbro sorry I meant to say the sum of all future rewards times the (negative) log probability of selecting the action taken that led to that reward. (I couldn’t edit my previous comment for clarity).

@nbro I’m assuming torch tracks the gradients by tracking  entries in policy.saved_log_probs since they were drawn from torch.Categorical() distribution. Otherwise these log probabilities would simply be entries in a list w/o any way to track how they were computed or their gradients.

@nbro I’m now just trying to figure out why we want to sum over all the products of (negative) log probabilities of each action and their associated return instead of doing an update on each time step. The Sutton / Barto algorithm has a different update for every time step. I suppose if we are already tracking gradients of every log probability then instead of doing one update at a time we can do an aggregate update to a linear combination of them? This makes sense to me but I haven’t written it out to prove it.

The first part of this answer is a little background that might bolster your intuition for what's going on. The second part is the more practical and direct answer to your question.

The gradient is just the generalization of the derivative to multivariable functions. The gradient of a function at a certain point is a vector that points in the direction of the steepest increase of that function.
Usually, we take a derivative/gradient of some loss function $\mathcal{L}$ because we want to minimize that loss. So we update our parameters in the direction opposite the direction of the gradient.
$$\theta_{t+1} = \theta_{t} - \alpha\nabla_{\theta_t} \mathcal{L} \tag{1}$$
In policy gradient methods, we're not trying to minimize a loss function. Actually, we're trying to maximize some measure $J$ of the performance of our agent. So now we want to update parameters in the same direction as the gradient.
$$\theta_{t+1} = \theta_{t} + \alpha\nabla_{\theta_t} J \tag{2}$$
In the episodic case, $J$ is the value of the starting state. In the continuing case, $J$ is the average reward. It just so happens that a nice theorem called the Policy Gradient Theorem applies to both cases. This theorem states that
$$\begin{align}
\nabla_{\theta_t}J(\theta_t) &\propto \sum_s \mu(s)\sum_a q_\pi (s,a) \nabla_{\theta_t} \pi (a|s,\theta_t)\\
&=\mathbb{E}_\mu \left[ \sum_a q_\pi (s,a) \nabla_{\theta_t} \pi (a|s,\theta_t)\right].
\end{align}\tag{3}
$$
The rest of the derivation is in your question, so let's skip to the end.
$$\begin{align}
\theta_{t+1} &= \theta_{t} + \alpha G_t \frac{\nabla_{\theta_t}\pi(A_t|S_t,\theta_t)}{\pi(A_t|S_t,\theta_t)}\\
&= \theta_{t} + \alpha G_t \nabla_{\theta_t} \ln \pi(A_t|S_t,\theta_t)
\end{align}\tag{4}$$
Remember, $(4)$ says exactly the same thing as $(2)$, so REINFORCE just updates parameters in the direction that will most increase $J$. (Because we sample from an expectation in the derivation, the parameter step in REINFORCE is actually an unbiased estimate of the maximizing step.)

Alright, but how do we actually get this gradient? Well, you use the chain rule of derivatives (backpropagation). Practically, though, both Tensorflow and PyTorch can take all the derivatives for you.
Tensorflow, for example, has a minimize() method in its Optimizer class that takes a loss function as an input. Given a function of the parameters of the network, it will do the calculus for you to determine which way to update the parameters in order to minimize that function. But we don't want to minimize. We want to maximize! So just include a negative sign.
In our case, the function we want to minimize is
$$-G_t\ln \pi(A_t|S_t,\theta_t).$$
This corresponds to stochastic gradient descent ($G_t$ is not a function of $\theta_t$).
You might want to do minibatch gradient descent on each episode of experience in order to get a better (lower variance) estimate of $\nabla_{\theta_t} J$. If so, you would instead minimize
$$-\sum_t G_t\ln \pi(A_t|S_t,\theta_t),$$
where $\theta_t$ would be constant for different values of $t$ within the same episode. Technically, minibatch gradient descent updates parameters in the average estimated maximizing direction, but the scaling factor $1/N$ can be absorbed into the learning rate.

Thanks you answered one of my questions about an implementation summing over the products of the (negative) log probabilities and their associated returns (I hadn’t considered it as mini batch implementation). I really appreciate the detailed answer. I guess I was trying to figure out how to represent it in a framework but I got bogged down in the details and forgot that Sutton / Barto mentioned that $J$ is a representation of the value of the start state. With the two answers provided I’m starting to find my way.


text:
How can the $\lambda$-return be defined recursively?
The $\lambda$-return is defined as
$$G_t^\lambda = (1-\lambda)\sum_{n=1}^\infty \lambda^{n-1}G_{t:t+n}$$
where
$$G_{t:t+n} = R_{t+1}+\gamma R_{t+2}+\dots +\gamma^{n-1}R_{t+n} + \gamma^n\hat{v}(S_{t+n})$$
is the $n$-step return from time $t$.
How can we use this definition to rewrite $G_t^\lambda$ recursively?

To rewrite $G_t^\lambda$ recursively, our goal is to define it in terms of
$$G_{t+1}^\lambda = (1-\lambda)\sum_{n=1}^\infty \lambda^{n-1}G_{t+1:t+n+1}.\tag{0}$$
The $\lambda$-return is a weighted average of all $n$-step returns. We will split up the summation by  pulling out the one-step return $G_{t:t+1}$ and the first step's reward $R_{t+1}$.
$$
\begin{align*}
G_t^\lambda &= (1-\lambda)\sum_{n=1}^\infty \lambda^{n-1}G_{t:t+n} \tag{1}\\
&\\
&= (1-\lambda)\lambda^0G_{t:t+1} + (1-\lambda)\sum_{n=2}^\infty \lambda^{n-1}G_{t:t+n}\tag{2}\\
&\\
&= (1-\lambda)\left(R_{t+1}+\gamma\hat{v}(S_{t+1})\right)\\
&\qquad + (1-\lambda)\sum_{n=2}^\infty \lambda^{n-1}(R_{t+1}+\gamma R_{t+2}+\dots +\gamma^{n-1}R_{t+n} + \gamma^n\hat{v}(S_{t+n}))\tag{3}\\
&\\
&= (1-\lambda)\left(R_{t+1}+\gamma\hat{v}(S_{t+1})\right) + (1-\lambda)\sum_{n=2}^\infty \lambda^{n-1} R_{t+1}\\
&\qquad + (1-\lambda)\sum_{n=2}^\infty \lambda^{n-1}(\gamma R_{t+2}+\dots +\gamma^{n-1}R_{t+n} + \gamma^n\hat{v}(S_{t+n}))\tag{4}\\
&\\
&= \gamma(1-\lambda)\hat{v}(S_{t+1}) + (1-\lambda)\sum_{n=1}^\infty \lambda^{n-1} R_{t+1}\\
&\qquad + (1-\lambda)\sum_{n=2}^\infty \lambda^{n-1}(\gamma R_{t+2}+\dots +\gamma^{n-1}R_{t+n} + \gamma^n\hat{v}(S_{t+n}))\tag{5}\\
&\\
&= \gamma(1-\lambda)\hat{v}(S_{t+1}) + R_{t+1}\\
&\qquad + (1-\lambda)\sum_{n=2}^\infty \lambda^{n-1}(\gamma R_{t+2}+\dots +\gamma^{n-1}R_{t+n} + \gamma^n\hat{v}(S_{t+n}))\tag{6}\\
&\\
&= \gamma(1-\lambda)\hat{v}(S_{t+1}) + R_{t+1}\\
&\qquad + \gamma\lambda(1-\lambda)\sum_{n=2}^\infty \lambda^{n-2}(R_{t+2}+\dots +\gamma^{n-2}R_{t+n} + \gamma^{n-1}\hat{v}(S_{t+n}))\tag{7}\\
&\\
&= \gamma(1-\lambda)\hat{v}(S_{t+1}) + R_{t+1}\\
&\qquad + \gamma\lambda(1-\lambda)\sum_{m=1}^\infty \lambda^{m-1}(R_{t+2}+\dots +\gamma^{m-1}R_{t+m+1} + \gamma^{m}\hat{v}(S_{t+m+1}))\tag{8}\\
&\\
&= \gamma(1-\lambda)\hat{v}(S_{t+1}) + R_{t+1} + \gamma\lambda(1-\lambda)\sum_{m=1}^\infty \lambda^{m-1}G_{t+1:t+m+1}\tag{9}\\
&\\
&= \gamma(1-\lambda)\hat{v}(S_{t+1}) + R_{t+1} + \gamma\lambda G_{t+1}^\lambda \tag{10}\\
\end{align*}
$$
$ $
$(2)$ pulls out the one-step return from the summation.
$(3)$ expands the $n$-step returns.
$(4)$ pulls out the remaining first step rewards.
$(5)$ combines first step rewards.
$(6)$ simplifies the geometric series.
$(7)$ pulls a factor of $\gamma\lambda$ out of the summation.
$(8)$ makes the substitution $m=n-1$.
$(9)$ uses the definition of the $n$-step return.
$(10)$ uses the definition of the $\lambda$-return
The result can be verified in equation $(12.18)$ of Sutton and Barto's RL book.


text:
Can a neuron have both a bias and a threshold?
I have not seen a neuron that uses both a bias and a threshold. Why is this?

I'm unsure about what this question is asking. Could you clarify and provide some context?

i have seen that if we set bias in activation function then we don't have to set threshold value, one of them will be  use in activation function.  is this necessary in every case or there is any specific reason for that?

I did my best interpreting and answering your question. Please let me know if I understood you correctly.

I assume you're talking about a perceptron threshold function. One definition of it with an explicit threshold is
$$f(\textbf{x})=
\begin{cases}
1& \text{if } \textbf{w}\cdot\textbf{x} > t\\
0& \text{otherwise}
\end{cases}.$$
Another form with a bias is
$$f(\textbf{x})=
\begin{cases}
1& \text{if } \textbf{w}\cdot\textbf{x} + b > 0\\
0& \text{otherwise}
\end{cases}.$$
But these forms are of course equivalent if you set $b=-t$.
There's nothing stopping you from using a perceptron definition with both a bias and a threshold:
$$f(\textbf{x})=
\begin{cases}
1& \text{if } \textbf{w}\cdot\textbf{x} +b > t\\
0& \text{otherwise}
\end{cases}.$$
But this is also equivalent to the other two forms. We can rewrite this as
$$f(\textbf{x})=
\begin{cases}
1& \text{if } \textbf{w}\cdot\textbf{x} > t'\\
0& \text{otherwise}
\end{cases}$$
where $t'=t-b$ is the new threshold. Or, we could rewrite it as
$$f(\textbf{x})=
\begin{cases}
1& \text{if } \textbf{w}\cdot\textbf{x} + a> 0\\
0& \text{otherwise}
\end{cases}$$
where $a=b-t$ is the new bias.
You never see definitions of this function with both a threshold and a bias because it has simpler forms.


text:
Is Reinforcement Learning the future of Natural Language Processing?
I was reading about the grounding problem after seeing it mentioned in another answer today. The article states that, in order to avoid the "infinite regress" of defining all words with other words, we must ground the meaning of some words in the "sensorimotor."

To be grounded, the symbol system would have to be augmented with nonsymbolic, sensorimotor capacities—the capacity to interact autonomously with that world of objects, events, actions, properties and states that its symbols are systematically interpretable (by us) as referring to.

Obviously, this made me think of Reinforcement Learning. But I'm not exactly sure what counts as "interaction." Would this necessarily imply an MDP-like formulation with rewards, state transitions, etc? Or could some form of grounding be accomplished with supervised learning?
This seems like a pretty fundamental problem of AI. Does anyone know of research being done on grounding words/symbols within an RL agent?

I think you need to reduce the number of questions at the end. In fact I would remove the "On a related note" paragraph, because it although it is related, answering it could double the length of any answer here. By stating the question, you are setting expectation that someone should attempt to answer it. If you want an answer to that, and it is not already covered by answers to your first question, you can always ask a second, separate question here. That would use the site better, IMO

Have a look at https://github.com/adityathakker/awesome-rl-nlp.

authors: Δημητρης Παππάς


text:
Which kind of prioritized experience replay should I use?
The Prioritized Experience Replay paper gives two different ways of sampling from the replay buffer. One, called "proportional prioritization", assigns each transition a priority proportional to its TD-error.
$$p_i = |\delta_i|+\epsilon$$
The other, called "rank-based prioritization", assigns each transition a priority inversely proportional to its rank.
$$p_i = 1/\text{rank}(i)$$
where $\text{rank}(i)$ is the rank of transition $i$ when the replay buffer is sorted according to $|\delta_i|$.
The paper goes on to show that the two methods give similar performance for certain problems.
Are there times when I should choose one sampling method over the other?

The authors of that paper hypothesized that rank-based prioritization would be more robust to outliers. They suggested that rank-based sampling would be preferred for this reason. However, as they noted later, the fact that DQN clips rewards anyways weakens this argument.
If you're going to use someone else's ready-made code for your prioritized experience replay, then you'll probably end up using proportional sampling. Every implementation I've been able to find, including OpenAI's baselines implementation, uses proportional sampling. If you were going to write your own, proportional sampling might be preferred for being simpler to implement (so less error-prone).
Comparing these two sampling methods is complicated by the fact that rank-based sampling involves a hyperparameter that determines how often you sort your replay buffer. The authors of the original PER paper only sorted every $n$ timesteps, giving the nice amortized time of $O(\log n)$, where $n$ is the size of the replay buffer. Sampling takes constant time, so sampling a minibatch of size $k$ takes $O(k)$ time.
Proportional sampling doesn't involve sorting, but it does need to maintain a sum tree structure, which takes $O(\log n)$ time each time we add to the buffer. Sampling also takes $O(\log n)$ time, so sampling a minibatch of size $k$ takes $O(k\log n)$ time.
If we only sort our replay buffer every $n$ timesteps, then rank-based sampling is faster. However, because the buffer is almost always only approximately sorted, the distribution we sample from is only a rough estimate of the distribution we wanted. It's not clear that this estimation would be accurate enough to be performant when the replay buffer is scaled up in size past the $n=10^6$ transitions used in the paper. I haven't seen a study that compares performance for different frequencies of sorting and different buffer sizes.
So, rank-based sampling might be faster, but it also might not work as well. Adjustment of the sorting frequency hyperparameter might be necessary. The simpler and surer approach would be to use proportional sampling with clipped TD-errors.

It appears that the rank based method would be slightly better in terms of time complexity because you sort only once for k number of sampling operations. This article (not mine) explains in detail clearly. https://towardsdatascience.com/how-to-implement-prioritized-experience-replay-for-a-deep-q-network-a710beecd77b


text:
Is the next state drawn from the joint distribution of the previous state and action?
Suppose $G_t$, the discounted return at time $t$ is defined as: $$ G_t \triangleq R_t+\gamma R_{t+1}+\gamma^{2}R_{t+2} + \cdots = \sum_{j=1}^{\infty} \gamma^{k}R_{t+k}$$
where $R_t$ is the reward at time $t$ and $0 < \gamma < 1$ is a discount factor. Let the state-value function $v(s)$ be defined as: $$v_{\pi}(s) \triangleq \mathbb{E}[G_t|S_{t}=s]$$
In other words, it is the expected discounted return given that we start in state $s$ with some policy $\pi$. Then  $$v_{\pi}(s) = \mathbb{E}_{\pi}[R_t+\gamma G_{t+1}|S_{t}=s]$$
$$ = \sum_{a} \pi(a|s) \sum_{s',r} p(r,s'|s,a)[r+\ \gamma v_{\pi}(s')]$$

Question 1. Are the states $s'$ drawn from a from a joint probability distribution $P_{sa}$? In other words, if you are in an
  initial state $s$, take an action $\pi(s)$, then $s'$ is the random
  state you would end up in according to the probability distribution
  $P_{sa}$?

Also let $q_{\pi}(s,a)$, the action-value function be defined as: $$q_{\pi}(s,a) \triangleq \mathbb{E}_{\pi}[G_t|S_t = s, A_t = a]$$
$$=\sum_{s',r} p(r,s'|s,a)[r+\ \gamma v_{\pi}(s')]$$

Question 2. What are the advantages of looking at $q_{\pi}(s,a)$ versus $v_{\pi}(s)$?

You should have asked these two questions in separate posts, given that they are not very related.

The one answer should be accepted, in my opinion. It completely and correctly answers the questions.

Question 1. Are the states $s'$ drawn from a from a joint probability distribution $P_{sa}$? In other words, if you are in an
  initial state $s$, take an action $\pi(s)$, then $s'$ is the random
  state you would end up in according to the probability distribution
  $P_{sa}$?

This is tricky, because you don't show a definition of $P_{sa}$. My first thought was that you meant the transition matrix $P_{ss'}^a$, but that doesn't fit with the phrase joint probability distribution.
If you really mean joint probability distribution then the answer is generally "no", because $P_{sa}$ should be the probability of observing state $s$ and action $a$ when taking a random sampled time step:
$$P_{sa} = \pi(a|s)\rho_{\pi}(s)$$
where $\rho_{\pi}(s)$ is the distribution of states under policy $\pi$. Note this makes no reference to $s'$ at all.
However, there are ways that this could relate to the distribution of $s'$. Probably the most direct relationship would be when there is a deterministic environment, thus knowing $s$ and $a$ would determine $s'$. If, in addition to that, each $s'$ could only be reached from a single $(s,a)$ combination, then knowing $P_{sa}$ would also give you knowledge of $P_{s'}$ - this is not the same thing as the question is asking though.
If you did mean the transition matrix $P_{ss'}^a$ instead in the question, then the answer is yes, because
$$P_{ss'}^a = \sum_r p(r,s'|s,a)$$

Question 2. What are the advantages of looking at $q_{\pi}(s,a)$ versus $v_{\pi}(s)$?

The main advantage is that you can derive a policy more easily from $q_{\pi}$:
$$\pi'(s) = \text{argmax}_a q_{\pi}(s,a)$$
Compare with deriving a policy using $v_{\pi}$:
$$\pi'(s) = \text{argmax}_a \sum_{s',r} p(r, s'|s,a)(r + \gamma v_{\pi}(s'))$$
Note that these policies are not necessarily the same as $\pi$ on which the $q$ or $v$ values are evaluated. In fact this is a common situation whilst searching for an optimal policy, and it is possible to show that $\pi'(s)$ will result in same or higher returns as $\pi(s)$ across all states . . . the proof of this is called the policy improvement theorem.
The important thing about the first equation using $q$ is that it does not involve using the MDP model $p(r, s'|s,a)$ directly. This is the basis of model-free RL. Whilst the version using $v$ is more complex (taking more computation) and requires that you know $p(r,s'|s,a)$.
The main disadvantage of looking at $q_{\pi}(s,a)$ is that it has a larger dimension, it maps $S \times A \rightarrow \mathbb{R}$, as opposed to using $v_{\pi}(s)$ which maps $S \rightarrow \mathbb{R}$. So it can take longer to get good approximations of $q$ compared to $v$.