text:
What is the difference between batches in deep Q learning and supervised learning?
How is the batch loss calculated in both DQNs and simple classifiers? From what I understood, in a classifier, a common method is that you sample a mini-batch, calculate the loss for every example, calculate the average loss over the whole batch, and adjust the weights w.r.t the average loss?
(Please correct me if I'm wrong)
But is this the same in DQNs? So, you sample a batch from your memory, say 64 transitions. Do I iterate through each transition and adjust the weights "on the fly", or do I calculate the average loss of the batch and THEN in a big step adjust the weights w.r.t the average batch loss?

From what I understood in a classifier a common method is that you sample a mini-batch, calculate the loss for every example, calculate the average loss over the whole batch and adjust the weights w.r.t the average loss? (Please correct me if I'm wrong)

You are wrong.
The weights are adjusted w.r.t. the average gradient, and this must be calculated using individual loss function results. The average loss (or cost function when considering the whole dataset) is a useful metric for current performance, and it is the measure being minimised. But you cannot calculate meaningful gradients against the average loss directly.

But is this the same in DQNs?

The batch process is not as you described, but an experience replay minibatch in RL and a sampled minibatch in supervised learning can be very similar. The main difference in RL is that your prediction targets must be recalculated as part of the sampling process (using $G_{t:t+1} = R_{t+1} + \gamma \text{max}_{a'}\hat{q}(S_{t+1},a', \theta)$ to calculate the TD target, assuming you are using single step Q learning), whilst in most supervised learning the target values are fixed for each example.
In theory you could use repeated single item stochastic gradient descent in DQN, it doesn't break any theory, and it would work. However, it will usually be more efficient to use a standard minibatch update, combining all gradients into one average gradient for the minibatch and making a single update step.
If you are using a high level library for your neural network model in DQN, you usually don't need to worry about this detail. You can use the .fit function or whatever the library provides. In that case the only difference between a supervised learning update and an experience replay DQN update is what you get from the sampling. In supervised learning you get a set of $(\mathbf{x}_i, \mathbf{y}_i)$ examples directly by sampling a minibatch. In RL you get $(\mathbf{s}_i, \mathbf{a}_i, r, \mathbf{s'}_i, done)$ and must construct the $(\mathbf{x}_i, \mathbf{y}_i)$ minibatch from these before passing to your .fit function

Thank you for clarifying.
What exactly do you mean with "batch update"?

@bruh: I meant minibatch update, as in the process you areasking about. Will edit

To get the average gradient do I calculate for every example the gradient for every wheight, then average it over the mini-batch and then adjust the weights with the averaged gradient?

@bruh: Yes, that is how you would do it. Most libraries with a .fit method or similar will do that for you


text:
How can a DQN backpropagate its loss?
I'm currently trying to take the next step in deep learning. I managed so far to write my own basic feed-forward network in python without any frameworks (just numpy and pandas), so I think I understood the math and intuition behind backpropagation. Now, I'm stuck with deep q-learning. I've tried to get an agent to learn in various environments. But somehow nothing works out. So there has to be something I'm getting wrong. And it seems that I do not understand the critical part right at least that's what I'm thinking.
The screenshot is from this video.
What I'm trying to draw here is my understanding of the very basic process of a simple DQN. Assuming this is right, how is the loss backpropagated?
Since only the selected $Q(s, a)$ values (5 and 7) are further processed in the loss function, how is the impact from the other neurons calculated so their weights can be adjusted to better predict the real q-values?

It is quite common in DQN to instead of having the neural network represent function $f(s,a) = \hat{q}(s,a,\theta)$ directly, it actually represents $f(s)= [\hat{q}(s,1,\theta), \hat{q}(s,2,\theta), \hat{q}(s,3,\theta) . . . \hat{q}(s,N_a,\theta)]$ where $N_a$ is the maximum action, and the input the current state. That is what is going on here. It is usually done for a performance gain, since calculating all values at once is faster than individually.
However, in a Q learning update, you cannot adjust this vector of output values for  actions that you did not take. You can do one of two things:

Figure out the gradient due to the one item with a TD error, and propagate that backwards. This involves inserting a known gradient into the normal training update step in a specific place and working from there. This works best if you are implementing your own backpropagation with low-level tools, otherwise it can be a bit fiddly figuring out how to do it in a framework like Keras.
Force the gradients of all other items to be zero by setting the target outputs to be whatever the learning network is currently generating.

If you are using something like Keras, the second approach is the way to go. A concrete example where you have two networks n_learn and n_target that output arrays of Q values might be like this:

For each sample (s, a, r, next_s, done) in your minibatch*

Calculate array of action values from your learning network qvals = n_learn.predict(s)
Calulate TD target for $(s,a)$ e.g. td_target = r + max(n_target.predict(next_s)) (discount factor and how to handle terminal states not shown)
Alter the one array item that you know about from this sample qvals[a] = td_target
Append s to your train_X data and qvals to your train_Y data

Fit the minibatch n_learn.fit(train_X, train_Y)

* It is possible to vectorise these calculations for efficiency. I show it as a for loop as it is simpler to describe that way

Thank you again for answering. I think I got that. I also think that I implemented that the right way. Would you mind to look over it and tell my what I got wrong here?
https://github.com/OleVoss/DeepTaxi
PS:
I tried it with various architectures and hyperparamaters.

Looks ok on a first pass through, but I'm not familiar enough with PyTorch to comment on how you manipulated the q values in vectorised form. There are a few things you could try, including simplifying your network right down to no hidden layers, because essentially it is learning a Q table when each state is one hot encoded, and the weights will actually be equal to the learned Q values. There is no generalisation possible between states due to the way you have encoded them.

Ok. Appreciate your try. Technically it is learning something. But every time it is lerning to take just one action...

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Why Monte Carlo epsilon-soft approach cannot compute $\max Q(s,a)$?
I am new to Reinforcement learning and am currently reading up on the estimation of Q $\pi(s, a)$ values using MC epsilon-soft approach and chanced upon this algorithm. The link to the algorithm is found from this website.
https://www.analyticsvidhya.com/blog/2018/11/reinforcement-learning-introduction-monte-carlo-learning-openai-gym/
def monte_carlo_e_soft(env, episodes=100, policy=None, epsilon=0.01):

if not policy:
    policy = create_random_policy(env)
# Create an empty dictionary to store state action values
Q = create_state_action_dictionary(env, policy)

# Empty dictionary for storing rewards for each state-action pair
returns = {} # 3.

for _ in range(episodes): # Looping through episodes
    G = 0 # Store cumulative reward in G (initialized at 0)
    episode = run_game(env=env, policy=policy, display=False) # Store state, action and value respectively

    # for loop through reversed indices of episode array.
    # The logic behind it being reversed is that the eventual reward would be at the end.
    # So we have to go back from the last timestep to the first one propagating result from the future.

    # episodes = [[s1,a1,r1], [s2,a2,r2], ... [Sn, an, Rn]]
    for i in reversed(range(0, len(episode))):
        s_t, a_t, r_t = episode[i]
        state_action = (s_t, a_t)
        G += r_t # Increment total reward by reward on current timestep

        # if state - action pair not found in the preceeding episodes,
        # then this is the only time the state appears in this episode.

        if not state_action in [(x[0], x[1]) for x in episode[0:i]]: #
            # if returns dict contains a state action pair from prev episodes,
            # append the curr reward to this dict
            if returns.get(state_action):
                returns[state_action].append(G)
            else:
                # create new dictionary entry with reward
                returns[state_action] = [G]

            # returns is a dictionary that maps (s,a) : [G1,G2, ...]
            # Once reward is found for this state in current episode,
            # average the reward.
            Q[s_t][a_t] = sum(returns[state_action]) / len(returns[state_action]) # Average reward across episodes

            # Finding the action with maximum value.

            Q_list = list(map(lambda x: x[1], Q[s_t].items()))
            indices = [i for i, x in enumerate(Q_list) if x == max(Q_list)]
            max_Q = random.choice(indices)

            A_star = max_Q # 14.

            # Update action probability for s_t in policy
            for a in policy[s_t].items():
                if a[0] == A_star:
                    policy[s_t][a[0]] = 1 - epsilon + (epsilon / abs(sum(policy[s_t].values())))
                else:
                    policy[s_t][a[0]] = (epsilon / abs(sum(policy[s_t].values())))

return policy

This algorithm computes the $Q(s, a)$ for all state action value pairs that the policy follows. If $\pi$ is a random policy, and after running through this algorithm, and for each state take the $\max Q(s,a)$ for all possible actions, why would that not be equal to $Q_{\pi^*}(s, a)$ (optimal Q function)?
From this website, they claim to have been able to find the optimal policy when running through this algorithm.
I have read up a bit on Q-learning and the update equation is different from MC epsilon-soft. However, I can't seem to understand clearly how these 2 approaches are different.

If $\pi$ is a random policy, and after running through this algorithm, and for each state take the $\max Q(s,a)$ for all possible actions, why would that not be equal to $Q_{\pi^*}(s, a)$ (optimal Q function)?

Assuming that the estimates for $Q_{\pi}(s,a)$ have converged to close to correct values from many samples, then a policy based on $\pi'(s) = \text{argmax}_a Q_{\pi}(s,a)$ is not guaranteed to be an optimal policy unless the policy $\pi$ being measured is already the optimal policy.
This is because the action value $Q_{\pi}(s,a)$ gives the expected future reward from taking action $a$ in state $s$, and from that point on following the policy $\pi$. The function does not, by itself, adapt to the idea that you might change other action choices as well. It is a measure of immediate differences between action choices at any given time step. Therefore if there are any long-term dependencies where your action choice at $t$ would be different if only you could guarantee a certain choice at $t+1$ or later, this cannot be resolved by simply taking the maximum $Q_{\pi}(s,a)$ when $\pi$ was a simple random policy.
However, if you do decide to change the policy such that you always follow actions $\pi'(s) = \text{argmax}_a Q_{\pi}(s,a)$ for all states, then you can say this: For each state, $V_{\pi'}(s) \ge V_{\pi}(s)$. I.e.  $\pi'(s)$ is no worse than, and may be a strict improvement over $\pi(s)$. Better than that, $\pi'(s)$ will be a strict improvment over $\pi(s)$ if $Q_{\pi}(s,a)$ is accurate and $\pi(s)$ is not already the optimal policy $\pi^*(s)$. This is the basis for the Policy Improvement Theorem which shows that if you repeat the process of measuring $Q_{\pi^k}$ and then creating a new policy $\pi^{k+1}(s) = \text{argmax}_a Q_{\pi^k}(s,a)$ that you will eventually find the optimal policy. You only have to repeat your idea many times to eventually find $\pi^*$
The Dynamic Programming technique Policy Iteration does this exactly. All other value-based Reinforcement Learning methods are variations of this idea and rely at least in part on this proof.

Hey Neil, thanks for answering ! I think I have a clearer understanding of why it is not Q opt ! What do you mean in the second paragraph when you say “stuck measuring immediate differences between ... “ from that point onwards up till the end of the paragraph ?

I have one more question regarding the last paragraph where you talk about policy iteration. Policy iteration does not necessarily output Q opt in the case where not all actions are explored when following a policy pi, am i right ? If we take the case of a grid problem, where we can move "up" , " down", "left" and "right", then if our current policy specifies move only "right" for cell number (1,1), we would only get the Q value for action move "right", at the end of the whole monte carlo simulation and there would be no way to update the policy for cell number (1,1) ?

@calveen: Policy Iteration from Dynamic Programming explores all state.action pairs. That means it doesn't scale well. However, it is theoretically solid. The example you are asking about is a Monte Carlo updates - these approach optimal policy in a more stochastic way over time. To do that you need to add some randomness - typically epsilon greedy policies, so that sometimes the agent will take a different action than the polciy derived by taking the maximisiing action, and learn about the other directions

@calveen: In my second paragraph I am trying to cover that accurate Q(s,a) gives only a one step look ahead. It does not measure longer-term effects. That's what I meant by "stuck" - just that the values do not change instantly for $Q_{\pi}(s_2,)$ solely because you changed your policy choice when looking at  $Q_{\pi}(s_1,)$. You need to make that policy change for real (or in simulation) and then re-estimate your Q values after using it for a while

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Why is this deep Q agent constantly learning just one action?
I'm trying to implement deep q learning in the OpenAI's gym "Taxi-v3" environment. But my agent only learns to do one action in every state. What am I doing wrong? Here is the Github repository with the code.

I thought about my input-layer. I had the 500 states one hot encoded. So 499 of every input node would be 0. And 0 is very bad in an neural network. I tried the same code with the "CardPole-v0" and it worked.
So think about your input guys


text:
Why everyone is using CNN for image segmentation?
I'm a newbie in artificial intelligence.
I have started to research how to do image segmentation and all the papers that I have found are about CNN. Most of them use the same network, U-net, but with little variations: with more or fewer layers, different parameter values, etc.; but with not very different results.
It seems that CNNs are in fashion and everyone uses them. Or there are other reasons that I don't know.
If everyone is getting not very different results, why are they using the same approach instead of trying different ones?

CNN is used since it is effectively an optimized use case for dealing with image data.
CNN effectively automatically extracts features from images. Other techniques are more likely to not take full advantage of the data. CNN is able to make full use of the data by also including information from adjacent pixels and downsample through layers.

Here is a paper on the performance of CNN on image data
Here is a paper comparing CNN's to other methods


text:
What is the difference between Sutton's and Levine's REINFORCE algorithm?
I followed the videos/slides of Berkley RL course, but now I am a bit confused when implementing it. Please see the picture below.

In particular, what does $i$ represent in the REINFORCE algorithm? If $\tau^i$ is the trajectory for the whole episode $i$, then why don't we average across the episodes $\frac{1}{N}$, which approximates the gradient of the objective function? Instead, it is a sum over the $i$. So, do we update the gradients per episode or have batches of episodes to update it?
When I compare the algorithm to Sutton's book as shown below, I see that there we update the gradients per episode.

But wouldn't it then contradict the derivation on the Levine's slide that the gradient of the objective function $J$ is the expectation (therefore sampling) of the gradients of the logs?
Secondly, why do we have a cumulative sum of the returns over $T$ in Sutton's version but do not do it in Levine's (instead, all returns are summed together)

About the first question, you are right. The $i$ denotes a sample trajectory corresponding to a whole episode. However, Sutton's version is exactly the same one as Levine's if you choose $N=1$.
About the second question, the Policy Gradient theorem only tells you what is the gradient up to a constant, so basically any constant is irrelevant. Now, even if you do know the constant, you are going to multiply the gradients by an arbitrary learning rate $\alpha$. So, you can think that the factor $\frac{1}{N}$ is actually already considered "inside" $\alpha$.

Could you please detail on the second point. I don't really understand how the policy gradient theorem is related to averaging up the batch

The PG theorem tells you that the gradient  $\nabla J$ is proportional to the expected value of $G\sum_t\nabla\log\pi(a_t|s_t)$. So, your actual gradient is $C\mathbb E\left[G\sum_t\nabla\log\pi(a_t|s_t)\right]$, where $C$ is some constant. If you perform an SGD step with learning rate $\eta$, then your new parameters will be $\theta\leftarrow\theta+\eta\nabla J \approx\theta+\frac{C\eta}{N}\sum_iG_i\sum_t\nabla\log\pi(a_t^i|s_t^i) $. If you define $\alpha=\frac{C\eta}{N}$, you will obtain Levine's version. Since $\eta$ is arbitrary, you can see that $C$ or $\frac{1}{N}$ are irrelevant.

So, to answer your question, the fact that we usually don't know $C$ and that the learning rate $\eta$ is arbitrary makes irrelevant if you decide to consider the sum or the mean over the trajectories, that is, both are valid approximations of your gradient.

A practical and efficient method to compute the gradient involves leveraging stochastic gradient descent (SGD). This implementation aligns with the training approach commonly used in Variational Autoencoders (VAEs). The key idea is to estimate the gradient of the algorithm by using a mini-batch of size M to approximate the loss over the entire dataset size N. Specifically, the loss function is expressed as:
$\mathcal{L}(D) \approx -\frac{N}{M}\mathcal{L}(\mathbf{X}^M) = -\frac{N}{ML} \sum_{j=1}^{N} \sum_{i=1}^{L} \log p_{\theta}(x_{j} | z) \\
\mathcal{L}(D) \approx -\frac{N}{M}\mathcal{L}(\mathbf{X}^M) = -\frac{N}{M} \sum_{j=1}^{N} \log p_{\theta}(x_{j} | z)$
By increasing the mini-batch size sufficiently, it becomes feasible to use only one sample for the gradient estimation. This approach is referred to as the single-sample Monte Carlo method.


text:
What does the notation sup dist mean in distributional RL?
I'm trying to understand distributional RL, based on this article. In one of the equations, there is a symbol $\operatorname{sup dist}$.
\begin{align}
\operatorname{sup dist}_{s, a} (R(s, a) + \gamma Z(s', a^*), Z(s, a)) \\
s' \sim p(\cdot \mid s, a)
\end{align}
What does $\operatorname{sup dist}$ mean?

It doesn't seem that it is a "proper" symbol.
I guess that $\sup$ simply refers to the supremum, that is, you want to select actions that maximize the quantity that comes to the right of $\sup$, while $\text{dist}$ is simply a proxy for any possible distance between distributions. For example, you can replace $\text{dist}$ with the Kullback-Leibler divergence or with the mutual information.


text:
Are CNN, LSTM, GRU and transformer AGI or computational intelligence tools?
Will CNN, LSTM, GRU and transformer be better classified as Computational Intelligence (CI) tools or Artificial General Intelligence (AGI) tools? The term CI arose back when some codes like neural networks, GA, PSO were considered doing magical stuff. These days CI tools do not appear very magical. Researchers want codes to exude AGI. Do the current state of art Deep Learning codes fall in the AGI category?

CNNs, LSTMs, GRUs and transformers are or use artificial neural networks. The expression computational intelligence (CI) is often used interchangeably with artificial intelligence (AI). CI can also refer to a subfield or superfield of AI where biology is often an inspiration. See What is Computational Intelligence and what could it become? by Włodzisław Duch.
RNNs are Turing complete and CNNs have been shown to be universal function approximators (they can approximate any continuous function to an arbitrary accuracy given a sufficiently deep architecture), but that doesn't mean we will be able to create AGI with them, unless you believe that AGI is just a bunch of algorithms, but, IMHO, that alone doesn't produce AGI. See also the computational theory of mind.
To conclude, CNNs, LSTMs, GRUs and transformers are deep learning tools (so they could also be considered CI tools, given some definitions of CI), which might be useful for the development of AGI.

how would you differentiate between CI,AI,AGI?

@user_1_1_1 AGI is a sub-field of AI that attempts to develop AGI (intelligent systems similar to humans). As I say in my answer, AI can be used interchangeably with CI, but not all people will agree with this usage. I recommend you read the paper I cite in my answer for more details. I also recommend you read my answer https://ai.stackexchange.com/a/16649/2444.


text:
Why does my model overfit on pseudo-random numbers training data?
I am trying to predict pseudo-random numbers using the past numbers with a multiplayer perceptron. The error while training is very low. However, as soon as I test it with a test set, the model overfits and returns very bad results. The correlation coefficient and error metrics are both not performing well.
What would be some of the ways to solve this issue?
For example, if I train it with 5000 rows of data and test it with 1000, I get:
Correlation coefficient                  0.0742
Mean absolute error                      0.742
Root mean squared error                  0.9407
Relative absolute error                146.2462 %
Root relative squared error            160.1116 %
Total Number of Instances             1000

As mentioned, I can train it with as many training samples as I want and still have the model overfits. If anyone is interested, I can provide/generate some data and post it online.

Did you mention your data is pseudo-random? If that is the case the network must overfits as currently we cannot predict pseudo random numbers using neural network, or this will break all current cryptography in our society

Also what is the output you are going to predict?

@ClementHui excellent point and you guessed the topic in question too, the output is just a single number from -1 to 1. Regardless, I would like to learn if there are any solutions that might even slightly suggest improved results.

So you are trying to predict pseudo random numbers

@ClementHui lets say yes, however, I can filter the noise out... in that case any suggestion on which filter to apply (or any other method) ?

Please refer to this https://ai.stackexchange.com/q/3850/23713

Added an answer. Hope I can help you @Damir Olejar

Btw is this a possible duplicate of https://ai.stackexchange.com/q/3850/23713 @nbro

@ClementHui Not a duplicate, but for the simplicity's sake I said that it is (pseudo) random given the noisy data.

Simply said, predicting pseudo random number is just not possible for now. Pseudo random numbers generated now have a high enough "randomness" so that it cannot be predicted. Pseudo random numbers is the basis of modern cryptography which is widely used in the world wide web and more. It may be possible in the future through faster computers and stronger AI, but for now it is not possible. If you train a model to fit on pseudo random numbers, the model will just overfit and thus creating a scenario as shown in the question. The training loss will be very low while test loss will be extremely high. The model will just "remember" the training data instead of generalising to all pseudo random numbers, thus the high test loss.
Also, as a side note, loss is not represented by %, instead it is just a raw numeric value.
See this stack exchange answer for details.

"Also, as a side note, loss is not represented by %, instead it is just a raw numeric value."

Thanks, but that came as a default with the software :-)


text:
Is the Q value updated at every episode?
I trying to understand the Bellman equation for updating the Q table values. The concept of initially updating the value is clear to me. What is unclear is the subsequent updates to the value. Is the value replaced with each episode?  It doesn't seem like this would learn from the past.  Maybe average the value from the previous episode with the existing value?
Not specifically from the book. I'm using the equation
$$V(s) = \max_a(R(s, a) + \gamma V(s')),$$
where $\gamma$ is the learning rate. $0.9$ will encourage exploration, and $0.99$ will encourage exploitation.  I'm working with a simple $3 \times 4$ matrix from YouTube

I think you are a bit confused about what is the update function and the target.
The equation you have there, and what is done in the video is the estimation of the true value of a certain state. In Temporal-Difference algorithms this is called the TD-Target.
The reason for your confusion might be that in the video he starts from the end state and goes backwards using that formula to get the final value of each state. But that is not how you update the values, that is where you want to get to at the end of iterating through the states.
The update formula may have several forms depending on the algorithm. For TD(0), which is  a simple 1-step look ahead off-policy where what is being evaluated is the state (as in your case), the update function is:
$$
V(s) = (1 - \alpha) * V(s) + \alpha * (R(s,a) + \gamma V(s')),
$$
where alpha is the learning rate. What alpha will do is balance how much of your current estimate you want to change. You keep $1 - \alpha$ of the original value and add $\alpha$ times the td-target, which uses the reward for the current state plus the discounted estimate of the value of the next state. Normal values for alpha can be 0.1 to 0.3, for example.
The estimate will slowly converge into the real value of the state which is given by your equation:
$$
V(s) = \max_a(R(s, a) + \gamma V(s')).
$$
Also, the $\gamma$ is actually the discount associated with future states, as it is said in the video you referenced. It basically says how much importance you give to future states rewards. If $\gamma = 0$, then you only care about the reward in your current state to evaluate it (this is not what is used). On the other extreme if $\gamma = 1$ you will give as much value for a reward received in a state 5 steps ahead as you will to the current state. If you use some intermediate value you will give some importance to future rewards, but not as much as for the present one. The decay on the reward received on a state $n steps$ in the future is given by $\gamma^n$.
Another thing that I would correct is that the exploration - exploitation balance is not in any way related to $\gamma$. It is normally balanced by some policy, for example $\epsilon - greedy$. This one for example says that a certain % of the actions you take are random, which in turn makes you explore less valued states.


text:
Is word embedding a form of feature extraction?
Feature extraction is a concept concerning the translation of raw data into the inputs that a particular machine learning algorithm requires. These derived features from the raw data that are actually relevant to tackle the underlying problem. On the other hand, word embeddings are basically distributed representations of text in an n-dimensional space.
As far as I understand, word embedding is a somehow feature extraction technique. Am I wrong ? I had an argument with a friend who believes the two topics are totally separate. Is he right? What are the similarities and dissimilarities between word embedding and feature extraction?

Though word-embedding is primarily a language modeling tool, it also acts as a feature extraction method because it helps transform raw data (characters in text documents) to a meaningful alignment of word vectors in the embedding space that the model can work with more effectively (than other traditional methods such as TF-IDF, Bag of Words, etc, on a large corpus). Word Embedding techniques help extract information from the pattern and occurence of words and goes further than other traditional token representation methods to decode/identify the meaning/context of the words, thereby providing more relevant and important features to the model to tackle the underlying problem.
However, from another standpoint, word-embedding models were not developed aiming to solve a particular feature extraction problem, but rather, to generalize and model the language used in a corpus to gain a semantic understanding of the words and the relationships between them. Such that, all the various corpus-specific tasks can then employ the same "library" of information which was collectively & exhaustively learnt by the embedding model. Meaning, the word embedding model learns a language model that is task-agnostic for all tasks on that corpus unlike feature extraction methods which are specifically task-oriented.
Hence, the similarity is - word-embeddings can effectively aid in feature extraction; the dissimilarity is - they're not primarily meant to extract features more than they are for modeling a language which might be an "overkill" for a particular feature extraction task on a dataset.

One thing that should be mentioned is that when two semantically similar words are mapped into the embedding space, the defined metric within the space (for word2vec it is cosine distance) is proportional to the word similarity.  For example 'hog' and 'pig' will have a cosine distance(distance of 1 means same) closer to 1 than would 'hog' and 'cow'.  So neighbours in the embedding space are similar in the language. I consider this property to be the most important aspect of embedding.

I think you guys are playing on semantics.
If you consider feature extraction to be an unlearned preprocessing step to get inputs for your model, then no, word embeddings are not a feature extraction technique (examples here would be BoW counts, n-gram features, etc)
If you consider feature extraction to be any form of conversion from text to a set of features, then yes, word embeddings should be considered a form of feature extraction, given that they've learned in the process (or stolen from another model's training). Note though if you do include this, you would probably include most pretrained models as a whole as feature extraction techniques (like BERT).
So the whole conversation you had can go either way depending on the definitions you set.


text:
What is the intuition behind the TD(0) equation with average reward, and how is it derived?
In chapter 10 of Sutton and Barto's book (2nd edition) is given the equation for TD(0) error with average reward (equation 10.10):
$$\delta_t = R_{t+1} - \bar{R} + \hat{v}(S_{t+1}, \mathbf{w}) - \hat{v}(S_{t}, \mathbf{w})$$
What is the intuition behind this equation? And how exactly is it derived?
Also, in chapter 13, section 6, is given the Actor-Critic algorithm, which uses the TD error. How can you use 1 error to update 3 distinct things - like the average reward, value function estimator (critic), and the policy function estimator (actor)?
Average Reward update rule: $\bar{R} \leftarrow \bar{R} + \alpha^{\bar{R}}\delta$
Critic weight update rule: $\mathbf{w} \leftarrow \mathbf{w} + \alpha^{\mathbf{w}}\delta\nabla \hat{v}(s,\mathbf{w})$
Actor weight update rule: $\mathbf{\theta} \leftarrow \mathbf{\theta} + \alpha^{\mathbf{\theta}}\delta\nabla ln \pi(A|S,\mathbf{\theta})$

This is simply from definition of return in average reward setting (look at equation $10.9$). The "standard" TD error is defined as
\begin{equation}
TD_{\text{error}} = R_{t+1} + V(S_{t+1}) - V(S_t)
\end{equation}
In average reward setting, average reward $r(\pi)$ is subtracted from reward at $t$, $R_t$, so TD error in this case is
\begin{equation}
TD_{\text{error}} = R_{t+1} - \bar R_{t+1} + V(S_{t+1}) - V(S_t)
\end{equation}
where $\bar R_{t+1}$ is estimate of $r(\pi)$.
You can use $\delta_t$ in all 3 updates because neither of these updates depend on each other. For example if you update $\mathbf w$ you don't use that then to update $\mathbf \theta$, or if you update $\bar R$ you don't use updated version to update $\mathbf w$ or $\mathbf \theta$ so you're not introducing additional bias. In each separate update you also don't have $\delta_t$ present multiple times so that you require multiple sampling per timestep to get the unbiased update.
Additionally this is semi-gradient algorithm, it uses bootstrapped estimate $V_{t+1}$ but it doesn't calculate full derivative with respect to it aswell, only with respect to $V_t$ so the algorithm is biased by default, but works well enough in practice for linear case.


text:
Are current AI models sufficient to achieve Artificial General Intelligence?
I read an interesting essay about how far we are from AGI. There were quite a few solid points that made me re-visit the foundation of AI today. A few interesting concepts arose:

imagine that you require a program with a more ambitious functionality: to address some outstanding problem in theoretical physics — say the nature of Dark Matter — with a new explanation that is plausible and rigorous enough to meet the criteria for publication in an academic journal.
Such a program would presumably be an AGI (and then some). But how would you specify its task to computer programmers? Never mind that it’s more complicated than temperature conversion: there’s a much more fundamental difficulty. Suppose you were somehow to give them a list, as with the temperature-conversion program, of explanations of Dark Matter that would be acceptable outputs of the program. If the program did output one of those explanations later, that would not constitute meeting your requirement to generate new explanations. For none of those explanations would be new: you would already have created them yourself in order to write the specification. So, in this case, and actually in all other cases of programming genuine AGI, only an algorithm with the right functionality would suffice. But writing that algorithm (without first making new discoveries in physics and hiding them in the program) is exactly what you wanted the programmers to do!

The concept of creativity seems like the initial thing to address when approaching a true AGI. The same type of creativity that humans have to ask the initial question or generate new radical ideas to long-lasting questions like dark matter.
Is there current research being done on this?
I've seen work with generating art and music, but it seems like a different approach.

In the classic ‘brain in a vat’ thought experiment, the brain, when temporarily disconnected from its input and output channels, is thinking, feeling, creating explanations — it has all the cognitive attributes of an AGI. So the relevant attributes of an AGI program do not consist only of the relationships between its inputs and outputs.

This is an interesting concept behind why reinforcement learning is not the answer. Without input from the environment, the agent has nothing to improve upon. However, with the actual brain, if you had no input or output, it is still in a state of "thinking".

Here and here are two related questions.

Task Specification
It's been proposed that novelty search may circumvent this problem.  See: Abandoning Objectives: Evolution Through
the Search for Novelty Alone.  In this model, the agent has no goal or objective, but just messes around with the data to see what results.  (This could be regarded as finding/forming patterns.  Here's a recent popular article on the subject: Computers Evolve a New Path Toward Human Intelligence).
A form of procedural generation may also be useful, specifically the capability of creating novel models/environments and processes/algorithms to analyze them.  (See: AI-GAs: AI-generating algorithms).
In terms of programmers communicating a task to the AGI, that's a natural language problem if the task relates to mundane human activity or art and craft, and a math problem if the subject is physics.  (In the former case, humans are describing the problem in natural language, in the latter, presumably feeding all of the data that suggests dark matter into the algorithm.  Natural language is challenging for computers, but math, along with logic, is one of their two core functions.)
Re: dark matter, it may be a matter of asking the algorithm to find patterns in the data, and build models based on the data. The patterns and models would be the output, which humans could then consider. The output would be mathematical.
(Converting that mathematical output into metaphors, as in common on science programs like Nova and Cosmos, would be another goal of AGI.)
Brain in a Box
There needs to be stimulus/input to initiate "thought" process/computation.  In brain in a box, the brain is providing it's own internal stimulus.  I'd argue that an RL algorithm engaged in self-play is not dependent on external stimulus, but internally generated inputs, so that the process of model-based reinforcement learning is often a brain in a box, considering a subject or problem.

I am focusing on what you have posted here without going through (or having read) the whole essay you linked:

In the classic ‘brain in a vat’ thought experiment, the brain, when temporarily disconnected from its input and output channels, is thinking, feeling, creating explanations — it has all the cognitive attributes of an AGI. So the relevant attributes of an AGI program do not consist only of the relationships between its inputs and outputs.

The classic brain in a vat experiment does not disconnect the brain from all inputs and outputs but replaces "real" connections with its environment with "fake" connections, e.g. by connection the brain to a computer. This is what Wikipedia says:

In philosophy, the brain in a vat (BIV; alternately known as brain in a jar) is a scenario used in a variety of thought experiments intended to draw out certain features of human conceptions of knowledge, reality, truth, mind, consciousness, and meaning. It is an updated version of René Descartes's evil demon thought experiment originated by Gilbert Harman.1 Common to many science fiction stories, it outlines a scenario in which a mad scientist, machine, or other entity might remove a person's brain from the body, suspend it in a vat of life-sustaining liquid, and connect its neurons by wires to a supercomputer which would provide it with electrical impulses identical to those the brain normally receives.2 According to such stories, the computer would then be simulating reality (including appropriate responses to the brain's own output) and the "disembodied" brain would continue to have perfectly normal conscious experiences, such as those of a person with an embodied brain, without these being related to objects or events in the real world.

Following that, the argument of the author referring to the brain in a vat scenario does not hold.
If you ignore that problem for a moment the next inconsistency arises: The author assumed that because a brain would act "so and so" an AGI would need to act "so and so" as well. That however is not in line with how artificial intelligence is usually defined. Even if you consider the broad range of definitions:

(source: "Artificial Intelligence: A Modern Approach"; Russel, Norvig, 3rd Ed, 2010)
None of these definitions directly references AI to the brain. Therefore, the assumption that an AGI (as a subset of AI) would need to act like a brain in a vat is, without further assumptions,  generally flawed.
Another problem with the author's argumentation lies here:

to address some outstanding problem in theoretical physics — say the nature of Dark Matter — with a new explanation that is plausible and rigorous enough to meet the criteria for publication in an academic journal.
Such a program would presumably be an AGI (and then some).

The assumption that "creativity" (in quotation marks since we actually need to precisely define that in the first place) requires an AGI does not hold either. Let's stick to the authors example related to dark matter:

This is per definition a specialized field of application of an AI. Accordingly, there is not necessarily a need for an AGI since "specialization" is exactly what an AGI does not have.
Moreover, you could make an argument that coming up with new explanations related to dark matter is somewhat related to automated theorem proving. Accordingly, we might already have AI which is, in principle, capable of "solving" this task as of today.


text:
What are some new deep learning models for learning latent representation of data?
I know that autoencoders are one type of deep neural networks that can learn the latent representation of data. I guess there should be several other models like autoencoders.
What are some new deep learning models for learning latent representation of data?

Here's a link to my answer on CV Stack Exchange, where I have mentioned about latent spaces and some deep learning models that learn these representations: https://stats.stackexchange.com/questions/442352/what-is-a-latent-space/442360#442360
In short, deep learning models for Domain Adaptation, Computer Vision, Natural Language Processing, Recommendation Systems, Music/Speech/Audio processing, Adversarial models, etc., all learn some form of latent representation of data.
In fact, any place we're learning a function to map input and output spaces of a dataset, the model essentially learns a latent representation of data irrespective of whether the model is based on deep neural networks or a stochastic method or any other.


text:
Are on-policy algorithms always better than off-policy ones?
I am studying RL and I have a question: Are on-policy algorithms always better than off-policy ones?

No. An off-policy algorithm is, in one sense, always better. It allows you to train on already collected data that you could otherwise not use.
An off-policy algorithm can be used to "re-interpret" data that was collected earlier while running a worse policy. It can also learn the optimal policy while still doing disruptive random exploration. (Instead of learning to anticipate those random actions and mitigate.)
But in practice, if you can cheaply generate new training episodes (e.g. with a fast simulator), an off-policy algorithm may make the learning more complicated (slower) without any benefit. As always, it depends. Every method has some task for which it is optimal.


text:
How is InstructGPT a fine-tuned version of GPT-3 and at the same time has fewer parameters than the original GPT3?
I am reading the paper "Training language models to follow instructions with human feedback"
It says:

Our labelers provide demonstrations of the desired behavior on the input prompt distribution (see Section 3.2 for details on this distribution). We then fine-tune a pretrained GPT-3 model on this data using supervised learning.

The paper also says:

On our test set, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3,
despite having over 100x fewer parameters.

I am not able to understand how the aligned model is a fine tune of GPT-3 using supervised learning (and other steps associated with reinforcement learning) and at the same time the aligned model has fewer parameters than the original model.
Can someone give me a hint on the subject?

They say “a pretrained GPT-3” model, emphasis on “a” implying one of many, rather than “the”.
I believe they simply repeat the process with various parameters scales of the pretrained GPT-3, comparing the performance to GPT-3 175B throughout to see if there are parameter efficiency gains.
They note that even the 1.3B instructGPT version outperforms the 175B GPT-3 version.


text:
Q learning: How to create output layer in which actions are combinations of multiple sub-actions
Suppose in my example I want an agent to learn a behavior that is made up of a combination of actions.
So consider the following example with a tamagotchi like game: There are 5 pets and 3 actions that can be taken for each pet (give food, give water, play).
Now I know that in the standard approach, the DQN network would be such that the output is of shape 5x3 = 15. But for a larger space of pets and possible actions this would become unfeasible. So is there an adaption of deep q learning in which I could use a network that has only 3+5=8 output layers and treats the selection of the pet and the action independently?

What you're describing is known as a composite action space (or a factorisable action space). The key assumption is that the action space can be factorised in sub-action spaces, i.e. $\mathcal{A} = \mathcal{A}_1 \times ... \times \mathcal{A}_N$.
As far as I know, this is a relatively new area in terms of research contributions. The two papers that come to mind are action branching networks and Decoupled Q-Network.
The latter is likely to be the easiest to implement. Rather than flattening your sub-action spaces into an action space that consists of very many primitive actions and having a DQN provide an output value for the many different possibilities, instead you have $N$ output heads for each sub-action space. Where a DQN would be a function $f : \mathcal{S} \rightarrow \mathbb{R}^{|\mathcal{A}|}$, the decoupled network would be $N$ functions $f_i : \mathcal{S} \rightarrow \mathbb{R}^{|\mathcal{A}_i|}$. If we refer to the these sub-functions as utility functions, i.e. $U_i(s, a_i)$, then we can obtain the Q-value as $Q(s, a_1, ..., a_N) = \sum_{i=1}^N U_i(s, a_i)$. This way, the size of the spaces we need to learn values for grows linearly with the number of sub-action spaces rather than exponentially.


text:
What is curriculum learning in reinforcement learning?
I recently came across the term "curriculum learning" in the context of DRL and was intrigued by its potential to improve the learning process. As such, what is curriculum learning? And how can it be helpful for the convergence of RL algorithms?

Curriculum learning is a general technique for deep learning, which got recently applied to also deep reinforcement learning.

It's about designing tasks to guide the learning process of the network or agent. This can be particularly useful to bootstrap the network for very hard problems, and even to achieve better convergence.
In the context of DRL, apart from designing tasks you can also reason on the experiences e.g. by sorting them to facilitate the agent's learning.
In general the tasks are related to the main problem to solve, but you usually start from very simple ones and gradually increasing the task complexity as the agent makes progresses.

Indeed, a major issue is how to best design such tasks: there are some works that propose to automate this, giving rise to automated curriculum learning.

For Deep RL I suggest you this survey.

Edit: a real-word example of curriculum learning applied to deep RL.
Say you want to solve autonomous driving (AD) with RL. Your environment is a simulator of one or more towns (like CARLA). You can design the curriculum by devising simple variations of your initial env, for example from simpler to harder:

Restrict the town to a single straight road without any obstacle (car and pedestrians), in daylight condition, without carrying about complying to traffic rules and speed limits; This is to guide the agent just to learn basic control of the car, and simplified line following.
Add cars. This implies to learn collision avoidance: here you want to edit the reward function too, to add a penalty for collisions.
Add pedestrians: the collision penalty should be much higher.
Driving in more complex scenarios like intersections.
Introduce weather conditions and night scenarios. Aim: make the driving policy be more robust.
Compliance to traffic rules and speed limits: add terms to the reward fn.
Care about time, distance, fuel consumption etc

So you add difficulties until reaching the maximum that corresponds to solving your original environment. Indeed, in doing so you can either edit your env or its reward function. You can even restrict the env to a sub-task: e.g. learning to drive in straight roads.
As written earlier there is no golden rule to best design a curriculum or even to automate its generation (or at least I'm not aware of that.)
The example I provided here is from my own experience (see the paper, chapter F): this was my first time dealing with RL, the results are not great, and the benefit of the curriculum is marginal but I believe is possible to improve that a lot.

Curriculum learning is a training strategy in the context of DRL and other machine learning methods that involves organizing the learning process in a way that gradually increases the complexity of tasks or training samples. It is inspired by the way humans and animals learn, where they start with simple tasks and then progress to more complex ones as their skills develop. The idea is to create a "curriculum" that enables the model to learn more effectively by leveraging its acquired knowledge from easier tasks to solve more difficult ones.
By starting with simpler tasks, the agent can learn the basic skills required to solve the problem more quickly, which can lead to faster convergence compared to being exposed to complex tasks from the beginning.
Update
To answer your questions in the comment: All of the above possibilities; it can be different environments or changing the reward function.

Environment modification: In this approach, tasks are defined by creating different environments with varying complexity. The complexity can be measured by the number of states, actions, or transitions involved in solving the task. For example, in a robotic manipulation task, you can start with a simple environment where the robot needs to pick up an object at a fixed position, and gradually increase the complexity by adding more objects, changing their positions, or introducing obstacles.
Subtask decomposition: The main task is split into a sequence of smaller subtasks, which the agent must learn to solve in a specific order. The complexity of each subtask can be measured by the number of intermediate steps required to complete it, the level of abstraction, or the degree of difficulty in achieving the subgoal. By solving subtasks in a specific order, the agent can gradually build the skills necessary to complete the main task.
Reward shaping: Tasks can be defined by modifying the reward function to guide the agent towards solving specific subtasks or achieving intermediate goals. In this case, the complexity of the task can be determined by the difficulty of obtaining the shaped rewards. As the agent progresses, the reward function can be adjusted to phase out intermediate rewards and focus on the main objective.
Difficulty parameters: You can define tasks by adjusting certain parameters in the environment that control the difficulty of the problem. For example, in a maze navigation task, the complexity can be adjusted by increasing the size of the maze or adding more obstacles. The tasks can be ordered based on the values of these parameters, with the agent starting with simpler configurations and progressing to more challenging ones.

Each approach is possibly not clearly delineated from the others, changing the maze size when changing difficulty parameters can also be interpreted as changing the environment.


text:
What is the partial derivative $\frac{\partial y}{\partial x_1}$ in this neural network?

The answer is supposed to be -6, but I don't know how to get that.
Also, in a NN, is that 2nd hidden layer possible, where the neurons are not dependent on all the neurons of the previous layer?

Please, next time, write a title that contains your actual question and is specific.

From calculus rules: $\frac{\partial y}{\partial x_1}=\frac{\partial y}{\partial r_1}\frac{\partial r_1}{\partial x_1}+\frac{\partial y}{\partial r_2}\frac{\partial r_2}{\partial x_1}=3c+d=-6$. In a NN the above 2nd hidden layer where the neurons are not dependent on all the neurons of the previous layer is possible and it's usually just called a non fully-connected layer.


text:
When multiple stacked encoders are used, do the decoders only attend to the output of the final encoder layer?
After looking into transformer-based models that used multiple stacked encoders and decoders, I am trying to understand how cross attention in the decoders work. In a transformer with a single encoder/decoder, my understanding is the queries come from the decoder, while the keys and values come from the output of the encoder.
But when multiple stacked encoders are used, do the decoders only attend to the output of the final encoder layer? Or do they attend to all the encoder layers?
To put it visually, does cross attention look like this?

Or like this?

The decoder uses the output of only the final encoder layer for cross attention. The first diagram is correct.


text:
Why do we discount the state distribution?
In Reinforcement Learning, it is common to use a discount factor $\gamma$ to give less importance to future rewards when calculating the returns.
I have also seen mention of discounted state distributions. It is mentioned on page 199 of the Sutton and Barto textbook that if there is discounting then (for the state distribution) it should be treated as a form of termination, and it is implied that this can be achieved by adding a factor of $\gamma$ to the state transition dynamics of the MDP, so that now we have
$$\mu(s) = \frac{\eta(s)}{\sum_{s'} \eta(s')}\;;$$
where $\eta(s) = h(s) + \sum_{\bar{s}} \eta(\bar{s})\sum_a \pi(a|\bar{s}) \gamma p(s|\bar{s}, a)$ and $h(s)$ is the probability of the episode beginning in state $s$.
In my opinion, the book kind of skips over this and it is not immediately clear to me why we need to discount our state distribution if we have discounting in the episode.
My intuition would suggest that it is because we usually take an expectation of the returns over the state distribution (and action/transition dynamics), but, if we are discounting the (future) rewards, then we should also discount the future states to give them less importance. In Sergey Levine's lectures he provides a brief aside that I think agrees with my intuition but in a rather unsatisfactory way -- he introduces the idea of a 'death state' that we transition into at each step with probability $1-\gamma$ but he does not really provide a rigorous enough justification for thinking of it this way (unless it is just a useful mental model and not supposed to be rigorous).
I am wondering whether someone can provide a more detailed explanation as to why we discount the state distribution.

I added an answer @David

Not an exhaustive answer, but perhaps this blog post by Alessio Russo may be helpful. In particular, he states how

There is an equivalence between using a discount factor and reaching a terminal state in a Markov Decision Process. [...] Therefore, we simply need to introduce, artificially, the possibility of terminating the trajectory with a certain probability 1-γ.

The topic is not 100% clear to me yet either, but I feel like it makes much more sense now why this "death state" is used as a model to replace the discounted state distribution.

If you have three possible next states from the current state, by adding the discount factor you are introducing a fourth state. It can be a terminal state or some other state that is hidden. The probability of entering that hidden state 1-γ. The probability of all other three initial possible next state is γ.
Suppose you are entering the termination state by probability 1-γ. Your policy is giving a 1-γ probability of dying, so your policy has to avoid that termination and  has to make a better choice now i.e in the current state.

This is basically the same as the Sergey Levine description, which I don’t find useful (and is why I asked the question).

The R.S. Sutton and A. G. Barto book says "Often $\mu(s)$ is chosen to be the fraction of time spent in $s$? But then why a discount factor is needed in the continual case?
Discounting is practical when working with discounted rewards but not necessary when working with average rewards.
Why do may need to discount our state distribution with discounted rewards?
Let $\gamma \in [0,1[$ be the discount factor.
Let the discounted state measure be defined as
$$
\mu(s) = \sum_{t=0}^{\infty} \gamma^t \mathbb{P}^\pi(S_t = s)
$$
where $\mathbb{P}^\pi(S_t = s)$ is the probability of being in state $s$ at time $t$.
$$
\int_\mathcal{S} \mu(s) ds = \sum_{s \in \mathcal{S}} \mu(s) = \sum_{s \in \mathcal{S}} \sum_{t=0}^{\infty} \gamma^t \mathbb{P}^\pi(S_t = s) = \sum_{t=0}^{\infty} \gamma^t \sum_{s \in \mathcal{S}} \mathbb{P}^\pi(S_t = s) = \sum_{t=0}^{\infty} \gamma^t  = \frac{1}{1-\gamma}
$$
where $\mathcal{S}$ is the state space.
Thus, to be a distribution $\mu$ has to be multiplied by $1 -\gamma$ as in [1].
Note that if $\gamma = 1$, then the total mass of $\mu$ goes to $+\infty$.
The distribution $\mu$ is said improper in this case.
Also remark that,
$$
\mathbb{P}^\pi(S_t = s, A_t = a) = \mathbb{P}^\pi(S_t = s) \mathbb{P}^\pi(A_t = a | S_t = s) = \mathbb{P}^\pi(S_t = s) \pi(a|s)
$$
where $\pi(a|s)$ is the policy.
Now, remark with $r : \mathcal{S} \times \mathcal{A} \to \mathbb{R}$ the reward function.
$$
\mathbb{E}^\pi[\sum_{t=0}^{\infty} \gamma^t r(S_t, A_t)] = \sum_{t=0}^{\infty} \gamma^t \mathbb{E}^\pi[r(S_t, A_t)] = \sum_{t=0}^{\infty} \gamma^t \sum_{s \in \mathcal{S}} \sum_{a \in \mathcal{A}} r(s,a) \mathbb{P}^\pi(S_t = s, A_t = a)
= \sum_{s \in \mathcal{S}} \sum_{a \in \mathcal{A}} \sum_{t=0}^{\infty} \gamma^t \mathbb{P}^\pi(S_t = s) \pi(a|s)  r(s,a)
= \sum_{s \in \mathcal{S}} \sum_{a \in \mathcal{A}} \mu(s) \pi(a|s)  r(s,a)
= \mathbb{E}_\mu\mathbb{E}_\pi[r(S,A)]
$$
Let's break down what happened here.
We started by averaging the reward of the trajectory $(S_0, A_0, S_1, A_1, \ldots)$ that is characterised by the distribution $\mathbb{P}^\pi$. This distribution characterised a trajectory (that's why I used the subscript $t$ inside the expectation).
On the other hand, the distribution $\mu$ is defined on the state-space $\mathcal{S}$ and characterises the probability a state is visited (that's why I do not used the subscript $t$ inside the expectation, to mark the time-independence of the random variable).
Consequently $\mathbb{P}^\pi$ and $\mu$ are two different distributions that are defined on different spaces! The distribution $\mu$ averages over the time inherently.
This kind of change of measure often appears in RL literature and are not always obvious to understand.
Introducing the discounted measure allows in practice to sample state-action pairs $(X, A) \in \mathcal{X} \times \mathcal{A}$ to evaluate the objective function, instead of doing trajectory-based Monte-Carlo methods. In practice, it is very convenient and allows to employ "replay-buffer" for instance.
Going back to your question we have observed the importance of discounting in the "continual case" (infinite trajectories).
Why multiplying by $\gamma$ introduces a termination condition?
Take a probability distribution $\mu(\cdot)$ or $\mathcal{P}(\cdot, x, a)$ for some $x \in \mathcal{S}$ and $a \in \mathcal{A}$.
Then
$$
\int_\mathcal{S} \gamma \mu(s) ds = \gamma \int_\mathcal{S} \mu(s) ds = \gamma
= \gamma \int_\mathcal{S} \mathcal{P}(s, x, a) ds
$$
So both distribution $\gamma \mu$ and $\gamma \mathcal{P}$ are not probability distributions anymore! They are not normalised to 1. One possible way to get a probability distribution is to normalise it OR to introduce a state $s_{\text{end}}$ that is absorbing and has a reward of 0 with a probability of $1-\gamma$. This new MDP constructed by augmenting the state space with $s_{\text{end}}$ is then considered.

For mathematicians, to work rigorously with $\mu$ it is always better to use its truncated version and pass to the limit since limits of measures are still measures when the limit exists.
[1]: A. Agarwal et al. - Reinforcement Learning: Theory and Algorithms

About the first step after "Now, remark with ...". Having the superscript \pi in the first expectation is weird: should it be over {s0, a0, s1, a1, ...}? This is the standard notation (see TRPO paper, definition of eta(pi)). The, what is the second expectation over? And can we pull it inside the sum like that? If it is still over the trajectory, samples have timestep, so technically we can't. Unless we are assuming stationarity of the state distribution.

Hello @Simon, Indeed there is an inconsistancy in notations with a super and subscript $\pi$. The first expectation and the second after the first inequality are strictly the same. I just corrected this typo.
An expectation is an integral over a measure in a measurable space $(\Omega, \mathcal{F}, \mathbb{P})$, here $\mathbb{E}^\pi$ is the integral over $\mathbb{P}^\pi$.
The series of expectations being equal to the expectation of the series is true at least when the reward are bounded, by the dominated convergence theorem https://en.wikipedia.org/wiki/Dominated_convergence_theorem.

The crucial part is where the "triple sum" equals the "double sum" under $\mu$. Here, by definition of your measure $\mu$, you have the equality...
Your point on the stationarity is very relevant. I agree that stationarity allows dropping the time subscripts to get spatial average. However, here it is not applicable, but this approach kind of find a trick to obtain a results similar to stationarity, and it is rigorously accurate.
Try the same derivation with an avg rewards criterion, under stationarity hypothesis, you will get smth similar. Many prefer the avg criterion for that reason!

Thanks for your reply. I am pretty sure we must assume mu stationarity. Otherwise mu(s) depends on t and it is actually written as mu^(t)(s) (see https://arxiv.org/pdf/2011.06631). What do you mean by "here it is not applicable"?

And I am also confused by your result because you start with the discounted reward criterion and end up with the average reward criterion. There should be a factor 1/(1-gamma) (page 254 or Sutton's book). Or is it being pulled in mu(s)?

I indeed start with the discounted reward criterion, and I find a formulation of this criterion under two new measures (double integral $\mathbb{E}\mu \mathbb{E}\pi$) for the discounted reward criterion. No stationarity is involved here, but a close approach that makes $\mu$ containing/averaging the time.
In the paper you cite $\rho^{(t)} = \mathbb{P}(S_t = \cdot)$ while $\rho$ is the stationary measure (the infinite horizon average- p. 249 of Sutton and Barto but not normalised by the time). See G. Neu et al. - A Unified View of Entropy-Regularized MDP, p.4 for the stationary measure.

The $\mu$ defined in my answer is time-independant but no stationarity is mentioned at all right? I am not an expert, but a stationary distribution won't help to obtain an expression of the discounted reward criterion while the average criterion can be expressed with this stationary measure. That's why we have to craft this time-independant $\mu$ for the discounted case!:)
Regarding Sutton's book, it is not very clear how those $\mu$ are defined, but the sure thing is that they are independent of time (as the stationary dist. or the discounted visitation measure I defined - check [1])!

Yep, I see how you connected the two formulations, and I know they are correct. But if $\mu$ is time-independent then it's stationary. Also Sutton at page 249 says that the equations hold if $\mu$ is the steady-state (=stationary) distribution. The paper you cited also assumes so in Assumption 1. That's why I said that in your answer (great answer btw) you must also assume so.

@Simon, Sutton's p.49 and Neu's paper both use the average criterion !! By stationary, I think we agree that we are both talking about the invariant measure as defined in the "Stationary Distribution" part of the Markov Chain Wikipedia page of the underlying Markov Chain.
Consequently, your statement "if $\mu$ is time-independant, then it is stationary" is rigorously wrong, I mean at least in the sense of a true invariant measure! Indeed, $\mu$ looks like a kind of "stationary" measure but it not in the context of discounted rewards!

@Simon, I think the point where we disagree is that you think $\mu$ is the invariant, stationary measure for the Markov chain, while I pretend that it is not.
Consider $\mu$ being "some" measure that is independent of the time but which is not the stationary invariant measure, and that satisfies the equality for discounted rewards:)
Thus, we have constructed a measure $\mu$, which is time independent such that our result hold.
Moreover, in the literature $\mu$ for discounted reward is called a "visitation measure"

The notation μ is chosen to underline that μ is similar to a stationary measure.
However, this derivation does not need the stationarity assumption since the reasoning stems from the construction of this measure... Ask yourself, which assumptions should hold to construct the discounted visited state measure $\mu$? However, If you choose average rewards you will observe that to get a similar result, $\mu$ must be constructed such that $\mu$ coincide with the stationary measure. (While you can also use the ergodic theorem to get the result I guess).

Let us continue this discussion in chat.


text:
What is the meaning of "easy negatives" in the context of machine learning?
What does the term "easy negatives" exactly mean in the context of machine learning for a classification problem or any problem in general?
From a quick google search, I think it means just negative examples in the training set.
Can someone please elaborate a bit more on why the term "easy" is brought into the picture?
Below, there is a screenshot taken from the paper where I found this term, which is underlined.

It refers to samples that are very easy for the model to classify. If you are interested in the positive class, having many easy negatives could produce misleading results as your model could really struggle to classify not-so-easy samples.

In a very hypothetical situation, imagine you are trying to classify brain scan images based on whether they show signs of a tumor or not. For the negative class, say you have a bunch of normal brain scans that have no tumor, but also a bunch of plain images, all black with nothing on them (you wouldn't have that, but let's imagine). For the positive class you got normal brain images with tumors.
If you train a model, it might just learn that plain black pictures have no tumors, which is in fact true. Since half of your negative data have this kind of picture, the model could be virtually performing with a kinda-good accuracy but it would not be learning the actual problem.

OK, I think I understood what this means.
Hard and easy negatives are the ones that have relatively large and small values for the loss function, respectively.


text:
What is the difference between T5 and T0 models?
What is the difference between T5 and T0 models? I had read that T0 is T5 + LM. But as I know T5 uses encoder-decoder model like BART but BART can be used as LM so that's mean that T5 has a LM function too. So what is the difference between T5 and T0 then?

Can you provide more context? Where did you find these terms? Provide a link.

T0 starts with T5+LM as the base model but then performs multi-task finetuning on it using BIG-Bench tasks. As detailed in the Results section (Section 6) of their paper, T0 outperforms T5+LM on a number of tasks.

But didn't T5 also train on multiple tasks?


text:
Is there a measure of model complexity?
Is there a measure of model complexity?

Yes. There are at least 2 measures of model complexity studied and used in learning theory: VC dimension and Rademacher complexity. If you're new to learning theory, you could take a look at this answer.
(Note: your question is not an exact duplicate of this, but the VC dimension is not specific to neural networks).

There is no "unit" to measure model complexity. However, you can take into account some key factors:

input dimension
output dimension
model length (number of layers): Model length describes the number of transformations on the data from input to output. A higher number of layers means more possible transformations and thus the possibility to model more complex dependencies.
model width (number of neurons per layer): Model width describes the dimensionality of the data going through a single layer. A higher number of neurons means higher information retention per layer.
activation functions: Without any activation functions, even the largest model can only learn linear dependencies. Activation functions introduce non-linearity into the model and make complex dependency modeling possible.

Based on those factors, you can get a good estimate of how complex your model is. Note that a more complex model does not equal better performance per-se, they require careful fine-tuning.
To get a good estimate if your model is large enough to model your data transformations, you can train it on very little data with batchsize=1 and no regularisation. If it completely learns the given samples, it is likely to be fitting for your entire dataset aswell.

The answer address neural network only, not model in general.


text:
Is AI just a bunch of library functions?
Recently, I began reading more about AI and took a few basic courses to learn the basics of how it works. I also started a few projects involving AI, but get bored very quickly. To me, it feels like creating an AI project just feels like finding a dataset and throwing a bunch of library functions at it until it works. And tuning hyperparameters is even worse -- I make one small change and then have to wait for a long time before I can see what my change even did.
My main frustration is that I'm not actually building anything -- I'm just plugging in data to a bunch of functions and hoping it works. It doesn't feel like science or coding at all.
I'm not sure if this is a common issue or not -- maybe I'm just going about the entire process wrong? In any case, please let me know what projects I can do that are more interesting, or if I just need to reevaluate my approach to AI.

Is AI just a bunch of library functions?

No.
Your experience with online learning - I would guess via MOOC such as Andrew Ng's Coursera courses - is typical though, due to limitations of automated grading systems, the assessment of your ability to build systems is usually reduced to you filling in a few key functions and data structures in an already-built project.
However, library functions are going to exist for anything generalisable in any large technical field. Unreal Engine automates a whole bunch of interesting and difficult problems that crop up in 3D games. Ruby on Rails codifies a lot of common design approaches in web development. Same is true in AI with libraries such as TensorFlow, PyTorch and scikit-learn.
To users of these systems who care about the results to effort ratio, this is a good thing. It can be a double-edged sword when it comes to getting a deeper understanding though, and it also means that a lot of low-hanging fruit has already been picked, in terms of improving the state of the industry as a whole.
Whether or not this is boring depends on you, what you find interesting, and what you hope to achieve. There is no requirement that you do find AI or machine learning (ML) interesting. ML is essentially automated statistics and function-fitting routines, and is a large part of many modern AI approaches. AI as a whole is far more than that, but is such a large subject that typically you would work on a specialism within it, and would need to do so if you ever hope to hit an "edge" of it where you were performing original R&D or academic research.

My main frustration is that I'm not actually building anything -- I'm just plugging in data to a bunch of functions and hoping it works. It doesn't feel like science or coding at all.

OK, so there are several ways to view your relationship with AI technologies, and what you want to learn about and do with them. Not all of them will fit what you want. There's no guarantee you will find them interesting.
My first impression from your description of your frustration is that you have been learning with toy systems that don't do anything that you care about. You may find better motivation if you were trying to solve a problem that was meaningful to you. That could be:

Applying AI to solve a work problem that you have
A hobby project based on something you personally enjoy
A competition, such as in Kaggle, where some of the hard work of setting up the data is done for you, and you get to test your skills against others. You will soon find that getting good results is not just about tuning results on a library that you found

To me, it feels like creating an AI project just feels like finding a dataset and throwing a bunch of library functions at it until it works

This is often used in learning environments, and suggested as a way to hone skills in ML. That's because preparing data for use in ML is  hard work. In reality Data Science work can be 90% about understanding and obtaining useful data sets for a specific problem, and only 10% anaylsis and training models - training the models can be seen as the "easy and fun" part.
There are also a bunch of key understandings about how statistics works that you need to have in order to use the library functions. A typical beginner probelm in ML is not understanding the need to check for generalisation by using cross-validation and test sets, how they should be used, and how to spot and avoid data leaks. Simply taking some data, throwing it into a library function and getting "99% accuracy" is a major problem that comes with lack of understanding.

great answer, deserves more upvotes.

This is a late comment, but thanks so much for the answer! I decided to continue to build m own AI models and continue to learn about different techniques used in AI. The more I learn, the more I see how interesting this field can be. I'm definitely interested in continuing to pursue this field.


text:
What is the best way to combine or weight multiple losses with gradient descent?
I am optimizing a neural network with Adam using 3 different losses. Their scale is very different, and the current method is to either sum the losses and clip the gradient or to manually weight them within the sum. Something like: $clip(w_1\nabla_{L_1} + w_2\nabla_{L_2} + w_3\nabla_{L_3}, c)$.
I am thinking of better approaches. My current idea is to clip gradients separately (to avoid having one gradient "overtaking" the others too much), then weigh them, sum them, and finally clip them (with a smaller threshold than the one used for the first clipping).
Something like: $clip(w_1clip(\nabla_{L_1},c) + w_2clip(\nabla_{L_2},c) + w_3clip(\nabla_{L_3},c), c_2)$.
I am not sure what the best way to weigh them would be, though. Like having weights $w_i$ proportional to their gradient norm?
I'd like to get some suggestions / references.

This is an important subfield within multi-task learning, called gradient combination. Here is a list of about a dozen recent approaches: https://github.com/Manchery/awesome-multi-task-learning#loss--gradient-strategy
In particular, this paper is a good starting place: https://arxiv.org/abs/2001.06782
A brief summary of the work done on this topic:
It is often better to analytically combine the gradients, than to directly combine the losses and then get one gradient. Many ways have been proposed to do this, some involving gradient norms, some involving dynamic rescaling of gradients, some involving geometric analysis, and some involving game theory.
I should also note: there is an entirely different approach to multi objective optimization called Pareto Optimality, where instead of simply combining the losses in one particular way, we analyze a set of different ways of combining the objectives and their trade-offs. For more info, this is a good starting point: https://en.wikipedia.org/wiki/Pareto_front


text:
Shape of biases in Transformer's Feedforward Network
In transformer network (Vaswani et al., 2017), the feedforward networks have equation:
$$\mathrm{FNN}(x) = \max(0, xW_1 + b_1) W_2 + b_2$$
where $x \in \mathbb{R}^{n \times d_\mathrm{model}}$, $W_1 \in\mathbb{R}^{d_\mathrm{model} \times d_{ff}}$, $W_2 \in\mathbb{R}^{d_{ff} \times d_\mathrm{model}}$.
We know that the biases $b_1$ and $b_2$ are vectors.
But, for the equation to work the shape of $b_1$ and $b_2$ must agree, i.e., $b_1 \in\mathbb{R}^{n \times d_{ff}}$ and $b_2 \in\mathbb{R}^{n \times d_\mathrm{model}}$.
My question: is it true that
$b_1 = \begin{bmatrix} (b_1)_{1} & (b_1)_{2} & \dots & (b_1)_{d_{ff}}\\ (b_1)_{1} & (b_1)_{2} & \dots & (b_1)_{d_{ff}} \\ \vdots & \vdots &  & \vdots \\ (b_1)_{1} & (b_1)_{2} & \dots & (b_1)_{d_{ff}} \end{bmatrix}$
and
$b_2 = \begin{bmatrix} (b_2)_{1} & (b_2)_{2} & \dots & (b_2)_{d_\mathrm{model}}\\ (b_2)_{1} & (b_2)_{2} & \dots & (b_2)_{d_\mathrm{model}} \\ \vdots & \vdots &  & \vdots \\ (b_2)_{1} & (b_2)_{2} & \dots & (b_2)_{d_\mathrm{model}} \end{bmatrix}$ ?

The biases are vectors, and their shape should be $b_1\in \mathbb R^{d_{ff}}$ and $b_2\in \mathbb R^{d_\text{model}}$.
To verify that let's compute the shapes of the above formula (for simplicity we can exclude the $\max$):
$$\begin{align}
xW_1 &= (n\times d_\text{model})\times (d_\text{model}\times d_{ff}) = n\times d_{ff} \\
(xW_1)W_2 &= (n\times d_{ff})\times (d_{ff}\times d_\text{model}) = n\times d_\text{model}
\end{align}
$$
In practice, according to the underlying implementation the biases can be either:

expanded to match the batch dimensions, i.e. the vectors can be replicated $n$ times (equal to the batch size) - as you wrote;
summed element per element.
added directly to the weight matrix: this requires adding an extra dimension to both $x$ (the inputs) and $W$, like: $xW =  (n\times d_{model}+1)\times (d_{model}+1\times d_{ff})$.

In the first two cases, the biases are stored as vectors. I'm not aware of implementations that keeps the biases as matrices (i.e. replicated vectors) also because the batch size $n$ is not said to remain fixed: e.g. on inference it's tipically one.


text:
How is the max function differentiable wrt multiple arguments?
I recently came across an answer on StackOverflow that mentioned the max function being differentiable with respect to its values.
From my current understanding of mathematics, I'm struggling to comprehend how this is possible.
Could someone help explain how we can define the derivative of the max function in the case of multiple arguments as max(a1, a2, ..., an)?

Are you asking how the derivative of a function of multiple arguments is defined?

@Plop No. Actually, someone changed the original title. The tile is not mine.

Not sure wether this question is suitable here or should rather be on the math SE but here goes.
First restrict to two arguments, the general case is similar. Let $f(x,y):=max(x,y)$. This is a function defined on the entire real plane. When restricted to the region of the plane where $x<y$ then in this region $f(x,y)=y$ which is clearly differentiable. Similarly in the region where $x>y$ we have $f(x,y)=x$ which is also differentiable.
However on the line where $x=y$ this function is continuous but not differentiable. To see this consider for example $max(t, 1-t)$ for a real value $t$. This is continuous but has a kink at $t=1/2$ so is not differentiable at this point. This is exactly where it crosses the $x=y$ line.

Do you know how autodiff libraries handle the case $x=y$?


text:
Is it possible to use LLMs for regression tasks?
I want to use LLMs to predict edge weights in a graph based on attributes between two nodes. Is this even possible? If not, what would you recommend?
I tried to look up uses of LLM in regression tasks, but haven't had much luck finding anything helpful.

A better question might be, why? are other well-understood approaches not sufficient? Large language models don't seem like a good place to start, unless somehow the weights will depend on understanding text.

Regression with LLMs is definitely possible. Assuming you use a GPT-like model, you can either

train the transformer from scratch on the regression task, or
first pre-train the transformer on a general task, and then transfer learn the regression task by replacing the final linear layer.

Which option is more appropriate depends on the kind of regression task and how well that task is generalizable. For example, if you want to assess how positive a text is, you can better do that using option 2. However, a super-specific regression task is likely easier to do through option 1.
Check out this post for more general knowledge on transformers, and a bit of context about 'pretraining' and 'transfer learning'.

Thanks! Does the answer change if the data is mostly numerical and categorical data where not much text is involved?

You can still use transformers for numerical data. Neural nets have never been veey good at categorical data, but it is possible. The approach is still the same. If its generalizable then use 1 otherwise 2

Would you recommend LLMs for tabular data in general? I feel like there are easier ways to accomplish the task at hand. The post was helpful, are there any other resources you can share? Thanks!

Transformers are very general architectures. Although they were introduced for NLP they can be used for static data as well (tho arguably you wouldnt need positional encoding then). However, if you have static data, simple MLP architectures can get you started faster probably. For graphs, which you mention, you might actually wanna look into graph-neural-networks. They are super interesting and might be more suited for your problem.

Awesome, thanks for answering. I was told to use GPT-4 to work on predicting edge weights, but I can't imagine it being super helpful which is why I asked. To be clear, applying GPT to static data prediction would take a lot of setup right?

It is possible, but for graphs its simply not the right architecture

The overall graph structure doesn't mean much. It's more so about the attributes associated with two nodes and the edge. It's essentially just predicting a price based on numerical and categorical features. GPT is not the right architecture for this?

Theres many architectures that can be used for such a task. If you'd like specific advice on  how to approach a problem, i'd recommend starting a new question, detailing your dataset/use-case/target and asking what kind of architecture would be most suitable. This question was opened for whether transformers can be used for regression tasks, not for advice on a specific problem (although that was the intention behind it).


text:
Any suggestion on what I should try to get this cVAE model working for random generation of new molecules?
I am using a cVAE model to generate new structures of molecules. After successfully training the VAE model I am able to get proper reconstructions of the training set. While I am also able to generate decent structures  (near correct) when I use the unseen testing dataset. (i.e given a test molecule, I first pass it into the encoder; generate the latent vector and then produce a reconstruction from the decoder.)
But when I am randomly sampling my latent vector (z) from a Gaussian distribution (normal Gaussian), then the decoded results are not correct. i.e. sample a random vector torch.randn(dim(z)) of the dimension of the latent space and pass it through the decoder. The outputs of this result are nowhere valid.
I have tried to visualise the latent space according to the class conditional as well, and the latent space looks fairly good, pretty much the same as cVAEs. i.e. distributed uniformly across the origin in all directions. The 2-Dimensional and 3-Dimensional latent space distribution looks like the following.
Any suggestion on what I should try to get this cVAE model working for random generation of new molecules?

Are you sure your latent space is actually normally distributed? Compute a large number of latent vectors from inputs and measure the mean/var of your learned latents

The problem with autoencoders (conditional or not) is usually the poor structure of the learned latent space. The reconstruction loss guide the decoder to map from $\mathcal{N}(0, 1)$ to $\mathcal{D}$ (being your target distrbution) but it does not provide any information about independent features being present within $\mathcal{D}$ and no geometric structure is therefore learned, as your PCA visualisation show.
You can try two approaches to improve your model with the specific intent of generating unseen data:

Add classes to enforce the latent space to learn independent features per each class. In this scenario cross entropy can be use to train the encoder to classify the given input. This obviously require having classes information available along with the input data. (in the picture the upper part)

Move to an adversarial approach. GANs are better than VAE in generating unseen data, and fortunately the two are not mutually exclusive. I strongly advise to read Adversarial Latent Autoencoders. This approach differs from traditional GANs cause it leverages adversarial training to learn proper latent representation rather than proper generated data, putting a patch on the weak point of VAEs.

Thanks for the answer, actually I think the latent space of a cVAE model is not required to disperse in classes like the normal VAE model as the class conditions are already given. Hence the geometric structure of my PCA may not be all wrong. What do you think about this? I referred this blog post : https://stats.stackexchange.com/questions/428599/why-doesnt-conditional-variational-autoencoderscvae-cluster-data-like-the-vani

Also how to introduce the cross entropy loss in the encoder which you've mentioned?

I agree that your PCA is not wrong. What I tried to explain (maybe poorly, if so I can rewrite the answer) is that succeeding in training a VAE does not necessarily imply the possibility of generating good new data. You just managed to compress and decompress the data, which is the only thing VAE is meant for by design.

by explicitly conditioning on classes you can enforce a smooth transition between them, like in conditional GANs when changing the value of a single latent dimension cause a change of a specific feature in the generated output (like glasses or hairs or age of the same face).

To include the cross entropy you can just change the output of your encoder to be z+c (z=latent dimension or $\mu$ and $\sigma$, c=num classes). You can then compute the usual reconstruction loss using z and cross entropy using c, your final loss would be the sum of both.


text:
Why does the pass@k metric not "behave like" probability?
pass@k is a metric used to evaluate models that generate code, used for example to evaluate Codex. To evaluate pass@k, you have a dataset of natural language/code pairs, and you pass each NL prompt to the model. For each prompt, it generates k code snippets. If at least one of the code snippets is correct, then the model succeeded at that prompt in k samples. The pass@k is the fraction of prompts for which the model succeeded in this sense.
The samples generated for each prompt are obtained via some stochastic procedure based on the model's output probability distributions on the vocabulary, like randomly choosing the next token based on that distribution or something like that.
So in the Codex paper for example, we see these figures for the largest Codex model:

pass@1: 28%
pass@100: 72%

These numbers make no sense to me. Every time we sample the model's prediction for a given prompt, we get back a random code snippet. That output is either correct or incorrect. Each trial is independent. So if the probability of one output being correct is p, the probability of at least one of 100 being correct should be 1 - (1 - p)^100. Here p=0.28, so the pass@100 should be like 99.999999%.
Are the trials not independent? What's going on?

You can use mathjax on this site.

Very late to this question, but pass@k doesn't behave like you're describing because each single pass@k sample is itself a union of k independent events, so averaging them doesn't behave like a union of (num samples) independent events.
Here's an illustration. Say our dataset is two samples, such that on sample 1 our model always gets it right, and on sample 2 our model never gets it right. Then pass@1 = 50%. But doing 100 passes on each sample, we get 100 correct answers on sample 1 (counts as correct) and 0 correct answers on sample 2 (counts as incorrect). Then pass@100 = 50%. In fact pass@k = 50% for any k in this toy setting.


text:
For a transformer decoder, how exactly are K, Q, and V for each decoding step?
For a transformer decoder, how exactly are K, Q, and V for each decoding step?
Assume my input prompt is "today is a" (good day).
At t= 0 (generation step 0):
K, Q, and V are the projections of the sequence ("today is a")
Then say the next token generated is "good".
At t=1 (generation step 1):
Which one is true:

K, Q, and V are the projections of the sequence ("today is a good")
K, Q, are the projections of the sequence ("today is a"), and V is the projection of the sequence ("good")?

Please see if the explanation here helps you in any way.
In short, during the forward pass, K, Q, and V are projections of the input sequence ("today is a"). The output of the forward pass is "good".

@RobinvanHoorn, I see, and thanks. My question is the next next time step (t+2), so now your entire sequence is "today is a good", do you just repeat the same thing (with K, Q, V as the projections of the "today is a good" sequence?)

your question was (and the question still only mentions) t=1. If you again want to predict a next word, then K,Q and V are again projections of the input sequence ("today is a good") and then the output becomes (i.e.) "day".

(This type of) autoregressive LLM always works by predicting one next token based on a series of previous tokens. First you run the model with input "today is a" and the prediction is "good". Then you run the model with input "today is a good" and the prediction is "day", and so on. Each token is predicted by running the entire model from start to finish on its previous input.


text:
In DQN, how to increase epsilon but not too much?
I am using DQN algorithm in a non-stationary problem in a continuous learning. My environment gives me some new states each T steps. For example after 10 000 steps, I get some new states and I need to train it again but not from zero. I need to change just epsilon to discover some new states in a way that I don't lose my previous experiences. In other words, how to increase epsilon but not too much? I don't want a fixed epsilon_0 like 1 or 0.99.

You can determine the value of epsilon during training, for example, by using a linear decay schedule as a function of the steps or every T of them: you start from an initial epsilon (e.g. $\epsilon_0 = 1$) and decay with some factor (e.g. $\gamma_\epsilon = 1/T$) until a minimum/final value of epsilon is reached (e.g. $\epsilon_\min = 0.1$).
The rule is something like:
$$\epsilon \leftarrow \max\big(\epsilon_\min, \epsilon - (\epsilon_0 - \epsilon_\min) * \gamma_\epsilon\big) $$
In practice you start with high epsilon so that your agent explores a lot initially, then you start to decrease it gradually until it reaches a minimum value, after that your agent mostly exploits (indeed if final $\epsilon=0$ that it only exploits.)
Moreover, if you want to periodically increase $\epsilon$ but not to the max value each time, you can decay $\epsilon_0$ too: for example, you can $\epsilon_0\leftarrow \epsilon_0*\gamma_\epsilon$ every T steps and then proceed with the linear decay. It would be like resetting the epsilon from time to time, and then gradually decrease that to favor exploitation.


text:
Why do we need the identify function when approximating a function with a neural network with multiple layers?
I have a question about the explanation of universal approximation theorem provided by wikipedia.
https://en.wikipedia.org/wiki/Universal_approximation_theorem#cite_note-:0-29
It says, after a universal approximation of one layer that approximates $f$ with $g$ with
\begin{align}
\sup_{x\in K}  ||f(x) - g(x)|| < \epsilon\\
\text{where} \quad
g(x) = C \cdot [\sigma \circ (A \cdot x + b)],\\
\end{align}
that

an $f$ can also be approximated by a network of greater depth by using the same construction for the first layer and approximating the identity function with later layers.

I cannot understand how I should interpret this. After the first layer, it outputs $g$ which is an approximation of $f$ right? Why is identity function following here? Doesn’t it keep outputting the same $g$ without its improving quality?

This says that if you can approximate a function with one layer, you can also approximate it with multiple layers because you can make the extra layers do nothing.
The universal approximation theorem tells you there is at least one way to approximate any function - which is the way that puts exponentially many nodes into one layer. It doesn't tell you whether there's a more efficient way using more layers.


text:
How do policy gradients work?
If I understand it correctly from the following equation
$$U(\theta)=\mathbb{E}_{\tau \sim P(\tau;\theta)}\left [ \sum_{t=0}^{H-1}R(s_t,u_t);\pi_{\theta} \right ]=\sum_{\tau}P(\tau;\theta)R(\tau)$$
from this paper, the utility of a policy parameterized by weights $\theta$, is the total expected reward following (all, one?) trajectory $\tau$. After rearranging, taking the gradient, and running the policy over $m$ trials, the equation becomes
$$\nabla_{\theta}U(\theta)\approx \hat{g} = \frac{1}{m}\sum_{i=1}^{m}\nabla_{\theta}logP(\tau^{(i)};\theta)R(\tau^{(i)}).$$
My question is - how does this work? I can't understand it in terms of the utility/loss surface. At most, doesn't taking multiple trials give you a more accurate guess of the point $(\theta,U(\theta))$ alone? I don't get how you can get the gradient (a sort of local tangent plane of a surface) without evaluating the function at different $\theta$'s, like how do you get the slope at a point (2,3) without evaluating it at 2.01, 1.99, etc. at the very least? Is it by virtue of the well defined geometry of the function $U$ we defined?

From the equation you wrote:

$U(\theta)$ is the total expected reward of a policy $\pi_\theta$ (with parameters/weights $\theta$) over all possible trajectories $\tau$ sampled under the environment dynamics while following the policy itself: the term $P(\tau;\theta)$ should include both the env. dynamics (i.e. how it evolves each step given an action from a state), usually denoted as $P(s'\mid s, a)$ where $s'$ is next state, and the fact that we sample actions from the policy $a\sim \pi_\theta(s)$ from a state $s$.
For practical purpose, you can't take the expectation over all possible trajectories so you run the policy over $m$ of them and approximate the expectation (integral) with a summation over finite num. of trajectories. Thus, $\hat g = \frac{1}{m}\sum_{i=1}^m\nabla_\theta \log P(\tau^{(i)};\theta)R(\tau^{(i)})$.

My question is how does this work?

Basically the formula that estimates the gradient $\hat g$ makes use of what is called a score estimator (also called REINFORCE from the notable algorithm). It works by computing the probability of a trajectory to occur, then you score that trajectory by $R(\tau^{(i)})$ the total (discounted) reward, and finally compute the expectation as a simple average over $m$ trajectories.
The gradient of the policy $\nabla_\theta$ is so scored by the log-prob times the reward: the log-prob term has the intuitive explanation of increasing the probability of trajectories that score a high $R(\tau)$, while decreasing the ones for which $R$ is bad.

doesn't taking multiple trials give you a more accurate guess of the point $(\theta,U(\theta))$ alone?

Increasing $m$ provides a better estimate of $\hat g$. Indeed, as $m\to\infty$ you recover the true gradient $g$ by computing the full expectation. But that is not feasible in practice.

I don't get how you can get the gradient without evaluating the function at different $\theta$'s

At each update/optimization step the policy is only evaluated at $\theta$: its weights define both the point at which the gradient is computed and the policy itself, since it's a parametrized function approximator (like a neural network.) So, there is no need to know/evaluate at the surroundings of $\theta$.

So is the ground truth of $R(\tau)$ fixed? The only thing that changes when varying $\theta$ is the probability of following that trajectory $\tau$?

Exactly. For a given trajectory $\tau$, $R(\tau)$ is always the same since it's just a (discounted) sum over the immediate rewards $r_t$ gathered in $\tau$: it depends on the environment not the agent/policy.  Instead, the log-prob changes as the parameters $\theta$ got updated.

Sorry, I have just one more question, isn't the $U(\theta)$ function defined in essentially the way that the value function is (i.e. total expected reward of a policy)? Except it doesn't take in the state as an input, rather the parameters themselves?

$U(\theta)$ and $V(s)$ are similar (I assume $V(s)=V^{\pi_\theta}(s)$), but not equal. Although both represent sum of rewards, $V(s)$ computes that only from a state, while $U(\theta)$ considers an initial $(s_0,a_0)$ pair: this looks like a Q-value, $Q(s,a)$, but it's still different because $U(\theta)$ also averages over the initial state distribution $s_0\sim\rho(s_0)$. Intuitively, it's like $U(\theta) = \mathbb{E}{s_0\sim \rho}[V(s_0);\ \pi\theta]$ (but I'm not sure if 100% correct) as after all the RL objective is to maximize the total reward expected over all possible trajectories.