text:
Is there a recommended resource that can provide a detailed overview of the gradient norm?
When it comes to the concept of "Gradient Norm," it can be challenging to find a widely recognized and clearly defined resource that offers a comprehensive explanation. While many search results include insights from machine learning experts or references to papers that touch upon gradient norm, there isn't a single, definitive source that delves into the topic extensively.
Is there a recommended resource that can provide a detailed overview of the gradient norm?

Welcome to AI SE! Please provide the research that you have done in an attempt to answer your own questions (links to the resources you mention) within the post. This will help others understand what you have or have not seen!

The norm is a mathematical operation that can be applied to vectors, or matrices, informally measuring the "length" of such mathematical objects.
Since the gradient $g=\nabla_\theta f(x;\theta)$ of some continuous function $f:\mathbb R^N\to \mathbb R^M$ w.r.t. some parameters $\theta$ can be either a scalar (if $N=M=1$), vector (if $N>1, M=1$), or matrix (if $N>1,M>1$) the "gradient norm" is just the norm operation applied to $g$.

If the gradient is a scalar, the norm is just its absolute value: $|g|$.
For the vector case, the norm measures the length or magnitude of the vector. There are various notions, resulting in different norms. The most used (especially in the context of gradients) is the Euclidean norm (also called $l_2$-norm): $\|g\|_2$.
For the matrices the concept of vector norm is extended. For example, the Frobenius norm, $\|g\|_F$, is the equivalent of the Euclidean norm but for matrices.

Now in the context of DL, the gradient is usually a list of matrices and vectors. So when referring to "gradient norm" one usually means the global $l_2$-norm of all the gradients, computed as follows (see tf.linalg.global_norm):
$$
\|G\|_2 = \sqrt\sum_{g\in G}\|g\|_2^2,
$$
which is the square root of the sum of the squared euclidean norms, for each gradient $g$ in the list $G$ of gradients.
In DL, the norm of the gradients can serve two main purposes: 1) to monitor the gradients to see if vanishing/exploding grads occurs during training, and 2) to be applied with optimization algorithms in the context of gradient clipping, thus scaling the norm of the gradients: this can be done individually on each $g$, and globally on $G$ (gradient list).


text:
How to interpret the policy notation $\pi_{\theta}(a_{t}|s_{t})$ in Reinforcement Learning?
In the context of Reinforcement Learning, I have seen that the policy $\pi$ (for some algorithms) is nothing but a Neural Network architecture (for example a Feedforward Neural Network).
This policy is usually annotated as $\pi_{\theta}$, suggesting the policy is parameterized by $\theta$.
Question 1: Does this mean that $\theta$ in this case would represent all the Neural Network's parameters?
Question 2: The notation $\pi_{\theta}(a_{t}|s_{t})$ can be interpreted as "the output probability of Neural Network $\pi$ with parameters $\theta$ of selecting action $a_{t}$ when being input the state $s_{t}$"?
Here I leave an example from the Hugging Face RL course which uses this kind of notation.

Yes. If the policy is represented by a neural network, then both of your statements describe the typical interpretation.

Question 1: Does this mean that $\theta$ in this case would represent all the Neural Network's parameters?

Yes. It doesn't have to be a neural network though. Any parametric model for the policy can be described using the same notation.

Question 2: The notation $\pi_{\theta}(a_{t}|s_{t})$ can be interpreted as "the output probability of Neural Network $\pi$ with parameters $\theta$ of selecting action $a_{t}$ when being input the state $s_{t}$"?

Yes, sort of. The neural network does not make the selection, and does not necessarily output probabilities. For example in a DQN a neural network typically outputs estimated Q values for each action, and a separate piece of code will convert that into a policy and take a sample action. So the notation also covers the behaviour of that additional code that works together with the ANN.
The Policy Gradient example that you linked typically has an ANN that outputs either an array of discrete probabilities that the policy code samples one action from, or it can output the parameters of some distribution, e.g. the mean and variance of a normal distribution, which the rest of the policy code will evaluate and sample from to decide an action in a continuous action space.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Surveys/Important papers in Explainability for LLM
I'm interested in the topic of Explainability for LLM: the attempts to find some higher understandable structures inside the LLMs or, to put it simply (though may be not completely correctly), the attempts to understand how LLMs "think".
Would anybody please recommend where to start?
I'm both interested in foundational papers at this topic and in important latest research.
The great survey or a curated list of papers would be wonderful, although I'm not sure they exist give the speed the field is moving with.

Is there a specific reason why you asked Large Language Models specifically? Are you interested only in the Sequence Generation task or also in classification and other tasks?

As you have already mentioned, it's a rapidly evolving research area. These can be the first steps in hunting for foundations.

https://arxiv.org/abs/2303.06223

https://www.sciencedirect.com/science/article/pii/S0950705123000230

Could you please provide at least the title for each article (not just the link)?

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What is the definition of Q when the discount factor depends on time?
Suppose I want to find out the Q value of a particular state $s$ bu doing action $a$ at a particular timestep $t$. I know that the Q-value when the discount factor is given by,
$$Q(a,s)=E_{\pi}\big[r_t+\gamma r_{t+1}+\gamma ^2r_{t+2}+..\big|s_t=s~~a_t=a\big]$$
If the  discount factor is not fixed, say it is  given by $\gamma(t)=e^{-t}$. Then is the Q-value given by
$$Q(a,s)=E_{\pi}\big[r_t+\gamma(t)r_{t+1}+\gamma(t)^2r_{t+2}+..\big|s_t=s~~a_t=a\big]$$ or is it given by
$$Q(a,s)=E_{\pi}\big[r_t+\gamma(t+1)r_{t+1}+\gamma(t+2)^2r_{t+2}+..\big|s_t=s~~a_t=a\big]$$

This seems similar to eligibility traces $e$ used in backward-mode TD learning. Basically, the $e$ values decay in time being time-dependent.

What is your motivation for having $\gamma = f(t)$, as opposed to say $\gamma = f(s)$? There are problems with definition using a function of time (independently of state - it is OK to have time as one of the properties of the state, it makes the equations work consistently), which I can go into if/when I write an answer. It would be helpful to know why you want to do this so I can suggest an alternative that may work for you


text:
How does Monte-Carlo Tree Search Compare to MCMC?
Monte-Carlo Tree Search was the method used for AlphaGo my understanding is: it would randomly search the state space of possible moves where the probability of choosing a move was proportional to the perceived Value of the resulting state (this rollout data was then used to train the value function further).
This seems very similar to MCMC to me. Except that it includes a mechanism for observing ground truth likelihood (winners/losers in GO) & updating the likelihood function (aka Value function) in the process.
Is this true? And to what degree in general have Bayesian methods been used in RL?
For example have people tried using one of the various specialized MCMC algorithms to improve Monte-Carlo tree search?
Sorry if the question is open-ended/vague.

There is an element in MCTS, in which you select to explore "promising" directions. You also give a chance for branches that were not explored enough yet. So, you can use for that Bayesian methods (to keep and maintain the distribution of the expected rewards).

I'm curious if you know if anyone has tried using MCMC for this?

AlphaGo as far as I can tell at this moment, is basically MCTS + maintaining and updating a value function and using it for making decision where to spend the next iterations. The vanilla MCTS makes use of Upper Confidence Bound (UCB), which is a handy / simple surrogate for a distribution.


text:
Can an RNN predict a sinus curve with no input?
I read a number of tutorials on how to make an RNN (simple, LSTM, etc.) that predicts a sinus curve. They all use as an input (x) in every step a set of past sinus values.
I am wondering if THEORETICALLY and if yes, practically as well, can an RNN be constructed that takes no input (or if technically needed one x = 1) and still predicts a sinus curve?
A bit of details:
Let's assume that the "inputs" (but as said the input is none) are coming on a time scale equidistantly and the expected output is a sinus curve. The timestep is significantly smaller than 2PI and the ratio to 2PI is non-rational (i.e. the same value is never repeated). E.g. assume that the timestep is 0.1 (so more than 62 points for one cycle). The feedback for backpropagation can be anything, like the sinus value that was expected, or the error square, or anything else. The number of cell states is also freely choosable.
When I tried to construct such a model, instead of a predicted sinus curve, I got a moving average of past values.
Extra version (if the above simple could work): Let there be one input parameter, that is the time passed since the last point. The expected prediction is again, the sinus value after that passed time. Here again, we can assume that the time steps (not equidistant in this case) are still significantly smaller than the cycle length of sinus.

The universal approximation theorem states that, theoretically, any function (trivially including sine waves) can be approximated using a neural network of large enough size. Neural networks include RNNs (i.e. vanillar, GRU, LSTM etc).
Practically, RNNs can definitely approximate a sine function. Research on GANs has had various papers reproducing sine waves (of different frequencies and phases) given random input (1, 2).
If you would only want to generate a sine wave with one phase and one frequency, then simply train the discriminator of the GANs to do so, and you easily have a generator RNN which generates approximately perfect sine waves. In such a case (without any desired variation in the outputs), it is enough to simply provide 1s as input to all timesteps, giving you the 'no input' criterion.

Thanks. It took me some time to read the two links. I learnt a lot, but do not fully understand the relevance (or my question was not clear). Why is it a GAN project? I do not want to make something "like" a sinus curve, but I do want the system to learn a particular, given sinus curve. With other words, I want a generator RNN to give me an output that predicts my known curve. For every datapoint I can do a backpropagation to get a better generator, but I am lost with the cost function of it (instead of a prediction, I get a moving average, i.e. a sinus, just different).

GANs are one type of generative model. Other types of architectures can also be trained but i just thought of these papers which show the sine wave generation and hence answered your original question. However for getting advice on a project, you might need to open a new question because the comments are not for this kind of discussions.

Sorry if my original question was not clear, but it always was about to predict a - given, existing - sinus curve and not to generate one that looks like. So even opening a new question would be the same, but anyway thanks.

You could do that with the given architectures. Simply detach the discriminator, and generate the first t steps and check the latest step. You could probably do it better simply using a supervised rnn. But the papers were simply to show that it was possible and done in literature. If you want to simply generate a given/existing sine wave from an integer t, just do it in a supervised learning fashion.

Consider that RNNs have an internal state, $h$, apart from the input sequence (and in particular, LSTMs have two of them), that got updated at each timestep of the input sequence during back-propagation through-time.

I am wondering if THEORETICALLY and practically, an RNN can be constructed that takes no input (or if technically needed one x = 1) and still predicts a sinus curve?

Well, an input is always required and you can set it to a constant value, each time. Anyway, you could leverage the hidden state at time $t$ to predict the next sine value at time $t+1$: you feed it as an input to the RNN. This may work but I guess only in a sequential manner, I mean you can't predict values at arbitrary points of the curve but only the next one in time: assuming $f$ is the RNN, you have $h_{t+1}, \hat y_{t+1} = f(h_t)$ i.e. the next internal state $h_{t+1}$, and the estimated sine value $\hat y_{t+1}$ also at the next timestep $t+1$.

Extra version: Let there be one input parameter, that is the time passed since the last point. The expected prediction is again, the sinus value after that passed time.

This is more reasonable. You can implement it by either let the timestep by the only input, or from the previous idea also use the last hidden state i.e. $h_{t+1}, \hat y_{t+t'} = f(t'; h_t)$ where $t'$ is the time passed since last point.
Anyway, in both cases you should use a mean squared error, and appropriately set the target sine values according to the point in time you want to predict.
(Regarding the universal approximation theorem: RNNs are universal computers, meaning that they can approximate/learn whatever computable program, at least in principle. There should be at least one paper about this, see the works of Juergen Schmidhuber.)

How can the second be more reasonable (i.e. easier - I guess)? The first is a special case of the second, when the only input (i.e. time passed) is always the same.
I actually tried the same as said, i.e. use LSTM (1000 h and cs values), a one layer DNN (1000:1 no AF) to get from cs the $\hat{y}_{t+1}$ value and a error square as the CF for backpropagation. What I got was moving average, hence the question.

On the approximation theorem: It is clear that any continuous function can be approximated by a NN. The question, if with the given model (i.e. in every step no input, or max the time passed; no previous values fed as input), the model can theoretically learn it.


text:
Where can I find the solutions to the problems in the book "An Introduction to Computational Learning Theory"?
I have been going through "An Introduction to Computational Learning Theory" (Kearns-Vazirani). I don't know if my solutions to the problems are correct and have no other way of checking my learning. How would I find solutions to compare to or otherwise check my learning?

You could also try asking a specific question about your specific problem/solution.


text:
What is the justification for this approach of clipping elementwise?
I'm new to the field of AI (though I have a background in mathematics).
As I was going through some documents, I read that there is a form of gradient clipping where the elements of the gradient that are outside some range are trimmed/clipped. I didn't understand this: the gradient provides us the direction of steepest descent for optimizing a real-valued function. Clipping elementwise (as opposed to normalization) would change that direction.
Why then would we choose to clip instead of normalization?
The only argument I can see is that clipping ensures we make some progress along the non-dominant direction (whereas normalization might make those components extremely tiny). But this contradicts the "goal" of steepest descent - we might be moving in a non-optimal direction.
What is the justification for this approach of clipping elementwise?

Intro: Gradient clipping is in the context of neural networks, which have multiple layers meaning that each layer has learnable parameters (vectors and matrices) so you actually get a list of gradients: one per layer, where each gradient is not a scalar (can be) but rather a tensor with the same shape of the parameters in the layer (this is due to reducing the Jacobian of the layer'weights to a gradient instead.)
Gradient clipping: to prevent spikes during training (and so avoid instabilities) it is possibile to clip (i.e. limit) each value to a maximum value. This operation does not change the direction of improvement but rather reduce it's magnitude because is performed in absolute value: say $g$ is the gradient and want to clip to $v$ (where $v>0$), the operation is $g=\text{clip}(g, -v, v)$.
Moreover there are alternatives to this:

Normalizing each gradient by its $l_2$-norm, which basically scales each layer's gradient independently. Or
Scaling the gradients by the global norm of all the gradients (so the list considered as a whole.)

Why then, would we choose to clip instead of normalization?

You decide it by trial and error, in some applications clipping is sufficient in some other normalization by norm is better.
Anyway, in both cases the training may experience a slow-down since the magnitude of the gradients are reduced and so the step size is too.

Sorry, I don't follow this: "it is possible to clip (i.e. limit) each value to a maximum value. This operation does not change the direction of improvement ...".
Since (as you rightly pointed out) there are multiple layers with matrices and vectors to optimize, we can reshape all of these to one giant vector, and we are optimizing J(theta). From my understanding, gradient clipping (as opposed to normalization or scaling) looks at dTheta, and clips/saturates some values if they exceed a scalar bound. How does this not change the direction of the dTheta vector?

You're right, what I not specified is that clipping occurs in absolute value because you don't want to change the sign of the gradient. Suppose $d\theta=-10$ and your clip value is $v=1$ (must be positive) what you do is $clip(d\theta, -v, v) = -1$. This code confirms this. I'm going to add this detail to the answer.


text:
If we know the joint distribution, can we simply derive the evidence from it?
I'm struggling to understand one specific part of the formalism of the free energy principle.
My understanding is that the free energy principle can be derived from considering statistical dynamics of a system that is coupled with its environment in some non-trivial way. To use a more specific instance, we can think of sensory perception as estimating a latent state given some sensory observation. Thus, the problem would be straightforwardly interpreted as Bayesian inference as the estimation of the quantity:
$$P(x \mid o) = \frac{P(o \mid x)p(x)}{P(o)}$$
The denominator $P(o)$ is intractable as it involves integrating $P(o|x) dx$ over all latent variable states $x$.
The problem is then solved as an optimization problem of fitting the variational posterior $Q(x \mid o)$ to the true posterior $P(x \mid o)$ by minimizing the KL divergence between the distributions.
The free energy then is simply the upper bound on this divergence which is the KL divergence $D(Q(x \mid o) || P(o, x))$. So our upper bound -- the free energy -- is the KL divergence between variational posterior and the joint distribution.
What confuses me is that it seems like knowing the joint distribution allows one to calculate the value $P(o)$ directly avoiding this whole minimization using the KL divergence, since $P(o, x) = P(o \mid x)P(x)$.
For reference: https://arxiv.org/abs/2107.12979 (pages 7 and 8 contain the derivation I'm referring to)

Please, next time, put your specific question in the title. Thanks.

If the full joint distribution $p(o,x)=p(o|x)p(x)$ (usually assumed to be the product of two Gaussians for free energy principle of predictive coding, and in many practical situations you can only learn the full joint distribution alongside the variational posterior on the fly via EM algo) is allowed to calculate $p(o)$ directly, then you'd first have to know the posterior function $p(x|o)$ which is exactly what the variational inference here aims to approximate using a variational posterior $q(x|o; φ)$ with parameters $φ$ instead of the convergence slower or inefficient MCMC method, and you've already mentioned $p(o)$ is intractable directly as it requires an integration over all latent variable states.


text:
How to guide the interaction between two ChatGPT conversational agents?
I wrote a code for a conversation between 2 ChatGPTs (which I call them agent 1 and agent 2) using poe.com and its reversed engineered API:
import poe

agent_1 = {"token":""}
print("Agent 1: ON")
agent_2 = {"token":""}
print("Agent 2: ON")

client_1 = poe.Client(agent_1["token"])
client_2 = poe.Client(agent_2["token"])

model = "chinchilla"

client_1.purge_conversation(model)
print("Agent 1: Ready")
client_2.purge_conversation(model)
print("Agent 2: Ready")

print("Initial Prompt:")
message = input()

def remove_ai_phrase(sentence):
    """
    Removes the phrase "As an AI language model" from a sentence.
    """
    ai_phrase = "As an AI language model"
    if ai_phrase in sentence:
        # Find the end of the sentence (i.e., the position of the last period)
        sentence_end = sentence.rfind(".")
        # Remove the ai_phrase from the sentence
        sentence = sentence[:sentence_end].replace(ai_phrase, "") + sentence[sentence_end:]
    return sentence

c = 0
while True:
    if c == 0:
        message = message
    c += 1
    print("\n",50*"=")
    print("\nAgent 1:")
    answer_1 = ""
    for chunk in client_1.send_message(model, message):
        print(chunk["text_new"], end="", flush=True)
        answer_1 += ""+chunk["text_new"]
    print("\n")
    print("\n",50*"=")
    print("\nAgent 2:")
    # answer_1 += "\nIn addition..."
    answer_1 = remove_ai_phrase(answer_1)
    answer_2 = ""
    for chunk in client_2.send_message(model, answer_1):
        print(chunk["text_new"], end="", flush=True)
        answer_2 += ""+chunk["text_new"]
    print("\n")
    message = answer_2
    message = remove_ai_phrase(message)

Based on my few experiments, when I set the initial prompt on something like "discuss the meaning of life", they stuck at complementing each other. But if I set the initial prompt to something short and meaningless like "1", they actually start having conversation on various topics.
How do you think we can make them talk about a specific topic and reflect on each other?

I voted to reopen. While a specific code example is given, the double question is general and interesting.

@JaumeOliverLafont If you think this is not a programming/technical question, maybe you can then remove the code from the post and reformulate the title as the specific question that is being asked ;)

I dared to rewrite the title generalizing from ChatGPT to conversational agents. I am not so sure to delete the code or rather write pseudocode instead. Should that be decided by the OP?

I put this question in SO and they (with a toxic language) down voted and closed the question because apparently this question belongs to AI Stack. I don't know where I can get my answer. @JaumeOliverLafont

An interesting long interaction (human-bot) can be obtained from asking the bot to "act as a teacher", and prompting for, say, five questions for each of five different subjects (nested loop in natural language programming). This allows for rethinking GPT as General Programmable Tutor.
A natural extension of this between two bots could be prompting one to act as a teacher and the other one as a student, to break the initial symmetry and allow you to simply contemplate the evolution of dialogue.
As for reflection, I wonder whether some artificial consciousness can be made to emerge by letting an agent ask questions about itself to a variety of other agents and make kind of a majority vote, a synthesis from the answers obtained. A natural first question to investigate one's nature in the absence of direct self-observation could be who am I?. With two agents only it seems to be more difficult to gain much knowledge about oneself, because the self-image could be manipulated more easily.


text:
For given units of a measure of model complexity, how many examples do we need to train a network to get the model right and generalize?
For given units of a measure of model complexity, how many examples do we need to train a network to get the model right and generalize?

Please name one such measure of model complexity, so we can elaborate from there.

Let us use VC complexity.

In general, we won't know: it is an undecidable problem.
For reference, Bashir et al. and Sehra et al. proved that for a given dataset, (even if given unlimited training time) it is undecidable whether an encodable learning algorithm will always overfit or underfit respectively. So we can never know if we have got the 'right' model (in terms of generalization error).
FYI, This problem is related to the halting problem and minimum description length (MDL).


text:
What is the relation between any suitable measure of model complexity, number of training examples and network size in deep learning?
What is the relation between any suitable measure of model complexity, number of training examples and network size in deep learning?

Please name one such measure of model complexity, so we can elaborate from there.

Let us use VC complexity.

In general, the higher the model complexity, the more training examples (and computation time) is required for the model to learn.
In general, network size is directly proportional to model complexity. The larger the network size, the more parameters, thus higher the complexity.


text:
Can you confirm that the transformer works strictly deterministically and there is no randomness inside or between the attention layers?
On a high-level temperature and randomness affect the output of a generative language model:

Lower temperature: Produces more focused, conservative, and consistent responses.

Moderate temperature: Strikes a balance between creativity and consistency. This setting can be useful for general content generation, where a blend of accuracy and inventiveness is desired.

Higher temperature: Generates more creative, diverse, and unexpected outputs.

What I'm not sure of is where exactly randomness (controlled by the temperature) comes into play. I believe to have understood that it's only after a transformer has done its deterministic work, suggesting some probable next words.
Can you confirm that the transformer works strictly deterministically and there is no randomness inside or between the attention layers?

Please, next time, put your specific question in the title to avoid ambiguities about what your actual question is. While reading your post, I thought you were asking about the temperature but then, if we continue to read, it turns out that your question is about the attention layers or the layers inside the transformer and in reality has nothing to do with ChatGPT specifically because you're asking about the transformer in general. Again, this is why we need to see your specific question in the title. Also create a post with 1 specific question in mind, not multiple or discussion.

Btw, if you want to understand how the temperature works, see this question. Also, please, next time, before asking a question, check our existing questions. It's often the case that some of our questions can be answered already with a simple search. Thanks for your cooperation.

Can you confirm that the transformer works strictly deterministically and there is no randomness inside or between the attention layers?

Of course, there is no injected randomness in a regular Transformer model: you have just and encoder-decoder architecture, with positional encoding, multiple multi-head attention blocks and layer normalization.
The randomness you describe is typical of GPT model family which are generative, being trained to predict the next word token. Basically, you have a distribution over next tokens and you can control the amount of randomness in the sampling by dividing the logits by a temperature $\tau$: in the limit if $\tau\to\infty$ the predicted distribution becames uniform, otherwise if $\tau\to 0$ it exactly resembles a one-hot encoding (so a $1$ in a given place, and all zeros elsewhere.)

I'm not sure of is where exactly randomness comes into play.

For example, at the end of the Transformer you have a dense layer which outputs $K$ logits: unnormalized probabilities over $K$ tokens. You can define a Categorical distribution from the logits $\alpha$ (well, you can also define it from normalized probabilities by applying a softmax on top) from which you can sample the "index" of the next token. To control the sampling you introduce the $\tau$ as follows $\text{Categorical}(\alpha / \tau)$. To generate new text you sample from that: say you start from a context $C$, you forward the model to get the logits $\alpha$ and sample a new token $t$, then update the context $C=C\cup \{t\}$ (so that the model also considers the novel predicted word), and repeat the process again.

That's right, the Attention Layer output is totally deterministic.
The temperature parameter is related to generative tasks (note that this is not the only thing you can do with Attention and the transformers architecture), and it controls how, given the logits (a vector of the same length as the vocabulary $V$, the model samples one token among all the available tokens.
Suppose, for simplicity, that we are doing Causal generation and we have a vector of logits $u$ of length $V$
If you do not sample and take just the most probable token at each time step (i.e the $\arg\max$ of the logits), you always end up with the same generation, given a fixed input.
Otherwise, we can model the distribution of probability over all tokens with a softmax and sample one token from the distribution at each step $t$.
$$\mathbb{P}(x=v_l | x_{1:t-1}) = \frac{\exp(u_l )}{\sum_{l'}\exp(u'_l)} \qquad \text{where}\; v_l = x_{1:t-1}l \;, \forall l \in V$$
The temperature modifies this distribution by warping the previous distribution, multiplying the terms inside the exponential by a factor $t$.
$$\mathbb{P}(x=v_l | x_{1:t-1}) = \frac{\exp(u_l/t)}{\sum_{l'}\exp(u'_l/t)} \qquad \text{where}\; v_l = x_{1:t-1}l \;, \forall l \in V$$
For $t \rightarrow 1$, the distribution gets sharper, as it tends to a indicator function over the maximum token (the $\arg \max$), and for $t = 1$ it is equivalent to the softmax formula above.
If instead you choose $t > 1$, the probability distribution of predicted tokens will flatten, i.e the distribution will tend to a uniform distribution as $t \rightarrow \infty$.
References: https://huggingface.co/blog/how-to-generate

The other answers might be correct for the mathematical abstraction of a GPT-like model, but are false for the actual real-world models. On-device forward passes through any big quantized neural network, including state-of-the-art OpenAI models in 2023, are not deterministic. The difference is not large, but in gpt-4-0314 with temperature 0, you can get a different token sampled every 100 tokens or so.
Edit (August 2023): What I wrote above is partly correct. The hardware issues explain some of the randomness; however, OpenAI's Chat Completions API models have more variance than the old Completions API models, which is some evidence that randomness in GPT-4 is not explained by hardware issues only.
The best guess I've seen is that mixture-of-experts routers operate on batches of queries, and are deterministic (up to hardware issues) on the batch level, not on single queries. There is no information leakage between inputs from different users, but the expert choice in each layer depends on what is in the batch, influencing the output.
Of course, this is speculation, and although the MoE theory seems plausible, it could turn out that GPT-4 inference requires a GPU setup which makes inference less deterministic than on classical dense models.


text:
How do I construct a state space that shows that Greedy Best First Search is not complete while A* is?

Construct a state space with appropriate heuristics and local costs. Show that Greedy Best First search is not complete for the state space. Also illustrate A* is complete and guarantees solution for the same state space.

Here I can not come up with a graphical figure of state space. Somewhere I saw (0,0),(2,2),(4,4),(6,6),(8,8),(10,10) as state space. But how to draw a figure for it?

Do you know what being complete means? That's the first step to answer your question. Also, your homework problem doesn't tell you to draw the state space, but just to construct it.

Actually I am not getting the idea what this question is trying to ask for.

I think you need to describe (not necessarily draw it) a state space with costs in the edges and a heuristic function (that estimates the costs to the goal) in such a way that the greedy BFS will not be able to find any solution and will continue to search forever or maybe get stuck, while A* will find a solution.


text:
How do I assign a weight to an additional loss?
I am trying to do multi-spectral image fusion. I am using the following paper as a reference.
https://arxiv.org/pdf/1804.08361.pdf
The code available on GitHub works well. But, I am trying to add some of my own code. I am trying to add an additional loss to the total loss calculation. The current loss function is sum of two losses, I am trying to introduce a third loss.
I am not sure how do I assign a weight to this additional loss? Because I have read many papers like this one and no one has really gone in depth of how did they came up with the weights assigns to individual losses in case of multi-loss function.
Is there any logical way to solve this problem? I would rather not do trial and error.

Well, it depends on the "meaning" (or expected impact) that the additional loss term has (should have).
For example, there are cases in which minimizing all the losses is detrimental for the network: in VAEs fully minimizing the KL may prevent the decoder to reconstruct the inputs at all. And in some other in which the additional loss term is either a regularizer or a property.
To summarize:

If all the terms have the same importance (and so they should be all equally minimized), you should weight them such that to have the same scale. Intuitively you want to gradient of each loss term to have almost the same magnitude.
Otherwise, if the other term is either a regularizer or something that have to ensure a property you need to search for the $\lambda$ that best weights the extra term. This is because usually such term should have a limited contribution on the overall loss. One practical way to do so is to apply grid search first to find the magnitude of $\lambda$, e.g. considering a scale within $[10^{-2}, 10^{-1}, 1, 10, 10^2]$. Then use either a random search or some more sophisticated hyperparameter tuning algo (like bayesian optimization) within the interval that grid search found, e.g. between $[10^{-1}, 1]$.

Sure, you can definitely add your own loss term to the originally computed loss function. I checked the repo of your paper, they have used tensorflow. I am giving you a minimum reproducible example of how can you make your own custom loss function and train your model with the computational graph of tf.
For more details you can check this link to other question on stack. link
import tensorflow as tf

# Assuming you have some model defined and the inputs and targets are placeholders or tensors
inputs = ...
targets = ...

# Define your loss function
def my_loss_function(predictions, targets):
    # Calculate the additional loss term
    additional_loss = ...

    # Calculate the main loss (e.g., mean squared error)
    main_loss = tf.reduce_mean(tf.square(predictions - targets))

    # Combine the additional loss term with the main loss
    total_loss = main_loss + additional_loss

    return total_loss

# Calculate the predictions of your model
predictions = model(inputs)

# Calculate the total loss by calling your custom loss function
loss = my_loss_function(predictions, targets)

# Perform backpropagation and update the model parameters
optimizer = tf.train.AdamOptimizer()
train_op = optimizer.minimize(loss)


text:
Is there a resource that offers a detailed overview of the gradient flow?
Understanding the concept of "Gradient Flow" can be quite difficult as there is a lack of widely recognized and clearly defined resources that provide a comprehensive explanation. Although you can find insights from machine learning experts and references to papers that mention gradient flow, there isn't a single, definitive source that thoroughly covers the topic.
Could you please recommend a resource that offers a detailed overview of the gradient flow?

Please share the insights that you have seen. This provides context for others trying to help you out!

Bear in mind the context is surrounding this video with the rest of Andrew Ng's work. So please watch that to get a better background.
" In the beginning there is Time. Time moves, and creates dynamics on Space."

1.2.5 Gradient Descent by Andrew
Ng
Gradient descent is in discrete time, gradient flow is in continuous
time

Links

https://www.youtube.com/watch?v=yFPLyDwVifc
http://awibisono.github.io/2016/06/13/gradient-flow-gradient-descent.html
https://www.youtube.com/results?search_query=gradient+descent+and+gradient+flow

First time answering on this site so please comment and I will fix what I can.

I highly recommend the Machine Learning Crash Course for understanding Gradient Descent. Yes, of course, this course is primarily aimed at an introduction to Tensorflow, but the initial lessons will help you in studying your question.
https://developers.google.com/machine-learning/crash-course
I would also recommend this incredibly cool article for study. This is the best I've come across on this subject, but it will take you some time to figure it out. After studying it (and a number of others), I finally managed to write my first vanilla neural network:
https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/


text:
Is there any reference about backpropagation of the Transformer's multi-head layer?
Is there any reference about backpropagation of the Transformer's multi-head layer or multi-head attention (MHA)? I have searched various journals but have not found one yet.

The original paper that first introduced the Multi-Head Attention is called "Attention Is All You Need".
There are also a couple of tutorials that give more didactic introduction into MHA and Transofrmer architecture:

Andrej Karpathy "Let's build GPT: from scratch" video and miniGPT github repository.
Jay Alammar: "Illustrated Transformer" blog post.
Michael Phi: "Illustrated Guide to Transformers" video.

The question is about backpropagation of the transformers multi-head layer not about the transformers or multi-head attention. So, we can conclude that the OP wants to know how backpropagation works in these layers specifically. It's also true they didn't provide more context. You don't mention BP in your answer. Does any of your links talk about this?


text:
Control algorithms when system dynamics are stochastic and/or unknown
I'm working on a traffic signal control problem, which I am currently approaching with Reinforcement Learning, but I want to try some other control algorithms.
This is hard for me because we don't have a model of the system:

The number of incoming vehicles is stochastic, it depends on the time of the day and the particular scenario

The number of outgoing vehicles is also stochastic: when we give green to an approach, the number of vehicles that go through depends on how many are already waiting (measurable) the current position and speed of other vehicles on that road unmeasurable), the incoming flow (stochastic)

On the other hand, the state space and action space are small and discrete, so I guess they can be treated by many control algorithms.
Could you point me to some algorithms or principles in control theory that could be suited for such a problem? I am also interested in algorithms that merge the learning capabilities of RL and the more robust and stable properties of control algorithms.

https://en.wikipedia.org/wiki/Stochastic_control#:~:text=Stochastic%20control%20or%20stochastic%20optimal,the%20evolution%20of%20the%20system.

Check also https://en.wikipedia.org/wiki/Queueing_theory


text:
Is information gain only calculated using decrease in entropy of a dataset?
Is information gain only calculated using decrease in entropy of a dataset?
If I calculate it using the decrease in Gini Impurity, before and after the split, can it not be considered information gain?

Information gain is a very specific measure in information theory, thus the entropy definition... for sure, the decrease in Gini index can be seen as a proxy for it


text:
How to automatically trigger model retraining for object detection models in case of data drift?
I have fine-tuned a pre-trained Yolov8 model on my dataset of labelled containers in a warehouse conveyor belt (image example here) .
I am working to develop an MLOps project for my project portfolio, and chose Object Detection since that was something on my to-do list for sometime now. While I am 95% done, I am stuck on this last step to automatically trigger model retraining for Yolov8 if a new object is suddenly introduced in an image on which inference is needed.
Just to frame thought process of my problem clearly :
Currently the model is only trained on labelled containers, but all of them are cuboidal and brown in color. But let's say the container shape or colors changes in the future. If the decision maker for this doesn't inform me of this change, the model may not predict these new containers correctly a.k.a. data drift and thus there could be indefinite errors. Thus, I would need a trigger to automatically retrain and update the model. But I don't know how.
Can anyone suggest any solutions for this problem?

The first idea that comes to my mind would be to have an "unknown" class. If the model places the object into that class, you can trigger the retraining. But, of course, you will need to manually categorise those new objects before retraining, but you could set up this at regular intervals (e.g. once the model categorised 10 new objects as unknown you get a notification) so that you don't have to continuously do this. Of course, you will also need to change your current model to be able to categorise objects as unknown.
The second idea, which you can also combine with the first, would be to have some kind of uncertainty estimator. I don't really know if you can (easily) adapt your model to do that, so you'll need to do your research.
There is also continual or lifelong learning. In simple words, there are specific ML techniques for incorporating new classes over time.
These are just some ideas. There could be better ones. I've never implemented them for your specific case, but in the past I've dealt with Bayesian neural networks for uncertainty estimation, so you may want to check them out.


text:
How does using the ELBO in VAEs make the problem tractable?
I'm studying Variational Autoencoders and a lot of the literature says that the posterior is intractable because the marginal distribution p(x) is intractable since the space of z is so large we cannot possibly integrate over it all. So to avoid this they create a lower bound on the log likelihood, the ELBO, which they then try to maximize. The term for the ELBO is:
$$E_q[log~ p(z,x)]-E_q[log~q(z)]$$
What I am trying to understand is how is this now tractable. The expectations in the ELBO are still over the distribution of q. Take the first term for example:
$$E_q[log~ p(z,x)]=\int_{}^{}q(z)~log~p(z,x)dz$$
Is this not still an integral over all z values? How did we make this problem any more tractable by finding the ELBO?
Additional Question:
Also another thing I was confused about is we always say the posterior p(z|x) is not computable because we don't have p(x), but how exactly do we have the numerator, p(x,z).
$$p(z|x)=\frac{p(x,z)}{p(x)}$$
Is this because we assume a prior, and then also assume that we can model p(x|z) with a decoder?

Please, limit your posts to just one specific question. If you have multiple questions, even if they are related to the same main topic, you can always create multiple posts.

Though the posterior $p(z|x)$ is intractable, we can still compute $p(x,z)=p(z)p(x|z)$ using conditional probability. As you claimed correctly $p(z)$ is assumed a simple and known prior distribution such as a standard normal distribution. Depending on the underlying generative process $p(x|z)$ is often assumed to be a multivariate Gaussian distribution with a diagonal covariance matrix and is typically implemented using a neural network known as decoder which takes the latent variables $z$ as input and outputs the parameters $\theta$ of $p(x|z)$.
Your ELBO seems not exactly correct compared to the reference: $${L_{\theta ,\phi }(x):=\mathbb {E} _{z\sim q_{\phi }(\cdot |x)}\left[\ln {\frac {p_{\theta }(x,z)}{q_{\phi }({z|x})}}\right]=\ln p_{\theta }(x)-D_{KL}(q_{\phi }({\cdot |x})\parallel p_{\theta }({\cdot |x}))}$$

The form given is not very convenient for maximization, but the following, equivalent form, is: $${L_{\theta ,\phi }(x)=\mathbb {E} _{z\sim q_{\phi }(\cdot |x)}\left[\ln p_{\theta }(x|z)\right]-D_{KL}(q_{\phi }({\cdot |x})\parallel p_{\theta }(\cdot ))}$$
where ${\ln p_{\theta }(x|z)}$ is implemented as ${-{\frac {1}{2}}\|x-D_{\theta }(z)\|_{2}^{2}}$, since that is, up to an additive constant, what ${x\sim {\mathcal {N}}(D_{\theta }(z),I)}$ yields. That is, we model the distribution of $x$ conditional on $z$ to be a Gaussian distribution centered on ${D_{\theta }(z)}$.

Since distribution of the approximate posterior ${q_{\phi }(z|x)}$ and the prior ${p_{\theta }(z)}$ are often chosen to be Gaussians, ELBO in the reference is finally tractable as $${L_{\theta ,\phi }(x)=-{\frac {1}{2}}\mathbb {E} _{z\sim q_{\phi }(\cdot |x)}\left[\|x-D_{\theta }(z)\|_{2}^{2}\right]-{\frac {1}{2}}\left(N\sigma _{\phi }(x)^{2}+\|E_{\phi }(x)\|_{2}^{2}-2N\ln \sigma _{\phi }(x)\right)+Const}$$

Here $N$ is the dimension of $z$.

Thank you for you answer. But how can we automatically say that we can model p(x|z) with a decoder, but we can't do the same with p(z|x). Through Bayes formula isn't p(x|z) also dependent on p(x)?

Of course encoder is used to approximate parameters of p(z|x), similar to decoder p(x|z). As implicated in my answer p(z|x)p(x)=p(x|z)p(z) and RHS terms are both tractable, so what I mean p(z|x) is intractable (analytically) is due to intractability of p(x). If you were able to directly learn a parametric model of the evidence p(x) (an integral expression per chain rule as a GMM in my ref) from first principles such as MLE or MAP, then you can simply generate output by sampling p(x). The difficulty of p(x) is that integration in high dimensional latent space is intractable even numerically.

From what I am understanding the reason why p(x|z) is tractable is because we simply choose for it to tractable my modeling it with a simply tractable distribution like Gaussian. Then obviously because of p(x), p(z|x) becomes intractable and we have to approximate it with VI. Theoretically, is it possible to choose a simple tractable distribution for p(z|x) instead which would force us to VI approximation on the likelihood?

You have constraints regarding the simple tractable p(z|x) since as posterior it must be close to its prior p(z) which is typically Gaussian. And due to the apparent conjugate prior property of both Gaussian p(z) and likelihood p(x|z), p(z|x) must be Gaussian, no simpler distribution is possible.


text:
Is it reasonable to ask for the same time-regularity of the high and low dimensional signals?
Consider we are dealing with sequential data sampled from a continuous time signal $x(t)\in \mathbb{R}^n$, so that the dataset will look like $\{x_0,x_1,…,x_n\}$, with $x_i= x(t_i)$.
Assume that we aim to perform dimensionality reduction of such sequence via an autoencoder, s.t we learn a low dimensional representation $z(t)\in \mathbb{R}^m$ , with $m\ll n$. In such settings, is it reasonable to ask for the same time-regularity of the high and low dimensional signals? (e.g. does $x \in C^0 \Rightarrow z \in C^0$?)
Papers or any kind of resource touching such problem is appreciated!


text:
What does the notation $\hat{A}_t\left(s_{0: \infty}, a_{0: \infty}\right)$ appearing in Generalized Advantage Estimation mean?
In general, the advantage function is defined as:
$A^\pi\left(s_t, a_t\right):=Q^\pi\left(s_t, a_t\right)-V^\pi\left(s_t\right)$
So far, I understand this formular like this:
With the advantage function, we can compare, how good (positive sign, i.e. $A^\pi > 0$) or bad (negative sign , i.e. $A^\pi < 0$) a specific action $a$ in state $s$ is compared against the average of all actions at time step $t$.
However, in the paper High-Dimensional Continuous Control Using Generalized Advantage Estimation (https://arxiv.org/pdf/1506.02438.pdf), they introduce the following notation:
$E_{s_{0: \infty} \\ a_{0: \infty}}\left[\hat{A}_t\left(s_{0: \infty}, a_{0: \infty}\right) \nabla_\theta \log \pi_\theta(a_t \mid s_t)\right]$ (see equation $7$)
I am a little bit confused by the term
$\hat{A}_t\left(s_{0: \infty}, a_{0: \infty}\right)$
What does this expression mean?
How can we estimate the advantage at time step $t$, using all actions and states starting from the starting state?

In policy gradient methods reducing bias is crucial to obtain more accurate gradient estimates for updating the policy in a stochastic gradient ascent fashion. The usual advantage function as you expressed only implicitly addresses credit assignment for the current state and the specific on-policy's action at time step $t$, while the generalized advantage estimation (GAE) expressed by equation 16 explicitly addresses credit assignment by capturing the temporal consistency of advantages and considering the cumulative effects of the on-policy's actions from the initial state to the current state and beyond until the end of its episode during each parameters update time step $t$. Also from equation 16 you see GAE has a flexible bias-variance trade-off parameter $λ$ to further fine tune for each problem, and even if it's biased when it's not $γ$-just, the bias must be very small as reflected in equation 14.
Therefore it's clear that the advantage estimator $\hat{A}_t\left(s_{0: \infty}, a_{0: \infty}\right)$ referenced in equation 7 is a function of the entire trajectory of an episode in general (the 2 semicolons mean the independent variables could be all states and their on-policy actions of any episode), and if it's of the form expressed in Proposition 1 then it's $γ$-just. The usual discounted advantage function is a special $γ$-just case of the advantage estimator $\hat{A}_t\left(s_{0: \infty}, a_{0: \infty}\right)$ as mentioned immediately after Proposition 1, and the GAE expressed in equation 16 is the generalized case of $\hat{A}_t\left(s_{0: \infty}, a_{0: \infty}\right)$ considering the on-policy's advantage over the entire episode to become completely unbiased when $λ=1$ in theory.

I think you mean that in policy gradient methods we are interested in reducing variance, not bias? Since with this method explained in the paper, we are introducing bias, we are not reducing it.

See equation 18 and the explanation after it: GAE(γ, 1) is γ-just regardless of the accuracy of V, but it has high variance due to the sum of terms. So when λ=1 GAE is unbiased estimator for γ-discounted advantage function. But as the footnote on page 3 notes that here we are concerned with obtaining an unbiased estimate of $g^γ$, which is a biased estimate of the policy gradient of the undiscounted MDP. PG method has to take discounted and undiscounted MDP separately as one of its complications compared to value based methods.

Anyway the flexible bias-variance trade-off capability of GAE is its feature compared to the usual advantage function which typically has low variance but high bias which is also mentioned in the same paper and this has little to do with your question about $\hat{A}t\left(s{0: \infty}, a_{0: \infty}\right)$, and equation 16 is the constructive way to estimate the generalized advantage at time step $t$ using all actions and states starting from the starting state as you requested.

@cinch I have a question. You say "cumulative effects of the on-policy's actions from the initial state to the current state", but the formula seems to look from the time step forward (running variable l) not from the beginning somehow (t=0). Could you clarify where my misunderstanding is?


text:
Which main steps should I consider in order to successfully use a VAE for Anomaly Detection?
I am thinking about using the variational autoencoder model for anomaly detection . I have an Android Logs dataset. As the logs generated are a representative of time series type of data I thought about using the VAEs with an architecture involving LSTMs in the encoder and decoders. As far as what I know about the basics of the working of the autoencoder models , given a sequence size say n , during learning/training phase n-1 entries are used to generate a representation/prediction of the nth entry (log in this case) and then reconstruction error is minimized by the model.
How could I approach including the preprocessing steps and any specific architecture for the VAE model that would result in higher accuracy ? I was also thinking about converting the logs into word embeddings using Word2Vec and then using them as input for the VAE model. Also suggest improvements to my approach .

Could you please check the question in the title is your actual question? If not, feel free to edit your post again ;) Could you also clarify what would be an anomaly in this case? What do you expect the model to detect as an anomaly.


text:
Where I can find SOTA models of information retrieval?
Can someone tell me where I can find SOTA models of information retrieval?
My task is to rank documents by given query by semantic search of embedding. I know that models like ColBERT, SPLADE solve this problem, but I think they are not SOTA now. I tried to find SOTA models on paperswithcode but there almost nothing about this problem.


text:
What does it mean if I trained my model with more steps per epoch than the total number of training images I have?
I'm having a little bit of trouble understanding what steps per epoch really means. I've read that Number of Steps per Epoch = (Total Number of Training Samples) / (Batch Size), however I don't understand at which point in the training this parameter is actually changing.
I am particularly interested in the question:
What does it mean if I trained my model with more steps per epoch than the total number of training images I have (I have a relatively small training image set).
I am going to re-run the model with fewer steps per epoch, but I am trying to figure out how I might interpret the results I have from a pretrained model trained with an additional 60 training samples and with 100 steps per epoch... the model performs relatively well though has a somewhat erratic training/validation loss curve that nonetheless decreases over time.

Can you please provide the relevant part of the code that allows you to perform 100 steps per epoch, even though you only have 60 training samples? It's possible that you feed the same samples multiple times. For example, you could feed 1 sample at a time, and do this for 60 times, then do again for 40 samples. There are different ways this could be done, but without the specific code, it's difficult to say exactly what's going on. Different libraries may use different terminology.

Welcome to AI.stackexchange! To answer your question more precisely, it would be helpful to provide a minimum working example of your code where we can see how you implemented your training loop.
However, I will give you my 2 cents on this topic, since I struggled myself with understanding the difference between epochs, iterations, batches etc. See for example this question. Therein, @nbro gave a good explanaiton that might help you further understand the difference between batches, iterations and epochs.

Im having a little bit of trouble understanding what steps per epoch really means. Ive read that Number of Steps per Epoch = (Total Number of Training Samples) / (Batch Size) [...]

As you have pointed out, steps per epoch is the (total number of training samples)/(batch size), so, how many iterations are needed until one epoch is completed, i.e. until the model has seen all data. Here, I use "steps per epoch" equivalently with "batch iterations". A batch is just a chunk of you full data set, say you have a 1000 data samples, and you want a batch containing 100 data samples, you would get 1000/100 = 10 batches.

I don't understand at which point in the training this parameter is actually changing

It would therefore take you 10 iterations (or steps) until you have fed all your batches to the model, in other words, until your model has seen the entire dataset. The number of iterations is determined by the number of batches you have, so if you have 3 batches, it will be 3 iterations, so it's not about the number of images you have, but the number of batches.

What does it mean if I trained my model with more steps per epoch than the total number of training images I have

Of course, if your dataset is small, you might also have only two batches of size 500 each (in that case it would take you 2 iterations to complete the full dataset), or just one batch that contains all your data (then it just takes one iteration). For simplicity, let's consider 10 batches of 100 data samples.
Instead of feeding the full dataset, we first feed batch_0 through the model, and we compute the loss_batch_0 of that forward pass. Given this loss, we can now compute the gradient of this loss w.r.t. the parameters and update the weights according to the optimisation algorithm, in your case gradient descent. However, since you've used only a small chunk of your entire data, these gradients, as well as the loss, are only approximations of the actual gradients and the actual loss.
Now, we have updated the weights of the model once. Using this slightly more accurate weights, you process batch_1, compute the loss again, loss_batch_1, which is now hopefully smaller than loss_batch_0, compute the gradient of this loss w.r.t. the weights, and update again, and so on. You see, in order to process all batches, it would take you 10 iterations, or 10 parameter updates until your model has seen all the data once.
Since we have processed all batches, i.e. the entire data set, we have completed one epoch! Surely the loss won't be small enough, which is why you would usually do more epochs, say you repeat this entire process 50 times. This would result in 50x10 iterations = 500 weight updates. As your dataset is small, you can pack all datasamples into one batch and process all you data in one iteration. Therefore, havin 50 epochs would result in 50 iterations, i.e. 50x weight update. In practice, you will often encounter much larger datasets where packing all data into one batch would either be an overkill for the computer, or from an optimisation point of view, inefficient.
Remember, in traditional gradient descent, the gradient is computed using the entire dataset, which can be computationally expensive for large datasets. SGD, on the other hand, randomly selects a subset of the data (a batch) at each iteration to compute an approximation of the gradient.
Here's the entire process in more detail, this is how your training routine should look like:

Initialization: Initialize the model parameters (weights and biases)

Batch Processing: For each epoch of training:

Divide the dataset into batches (and shuffle)

For each batch:

Feed the batch through the model with the current set of weights to compute predictions.
Compute the loss between the predictions and the true labels
Compute the gradient of the loss with respect to the model parameters
Update the model parameters using the gradient descent update rule (e.g., SGD update rule) based on the current set of weights.
Repeat for the next batch, until all your batches are processed

Epoch Completion: Repeat the process for a specified number of epochs until your model has a sufficiently small train & validation loss.

After training is complete, the final weights of the model are those obtained after processing the entire dataset for the last epoch. Based on this algorithm above, you might understand what you are missing here:

but I am trying to figure out how I might interpret the results I have from a pretrained model trained with an additional 60 training samples and with 100 steps per epoch

If you pack all these 60 training samples in one batch, then you do one iteration per epoch, doing 100 iterations per epoch is consequently 100 epochs. If you'd divide these 60 samples into two batches of 30 each, you'd have 2 iterations per epoch, doing 100 epochs would result in 200 iterations, i.e. 200x updating the weights. To interpret your loss, track your training loss, and track your validation loss. If the validation loss starts to deviate from the train loss, you are most likely overfitting the dataset, and you should stop training (reduce the nr. of epochs).


text:
Can you explain the Hinton's comment "Rprop is equivalent to using the gradient, but also dividing by the size of the gradient"?
Been reviewing some old foundational material and ran into this comment by Hinton on Rprop in his old Coursera class:

Rprop is equivalent to using the gradient, but also dividing by the
size of the gradient

I don't understand this statement. Dividing by the size of the gradient would to me sound like scaling the gradient to be a unit vector. My understanding of Rprop does nothing like that (i.e. it just starts with small step size and then scales each dimension separately based on adjacent signs of the gradient).
Can someone elaborate on the details?

"Rprop is equivalent to using the gradient" means Rprop fundamentally relies on information about the sign of the gradient of each weight to determine the direction of weight updates. Like traditional gradient-based methods, Rprop utilizes information about whether the gradient is positive or negative to guide the weight updates.
The second part of Hinton's statement "but also dividing by the size of the gradient" roughly draws only an analogy to the concept of adaptive adjustment of the step size based on the magnitude of the gradient change for each weight instead of the usual fixed step size. The net effect is similar to the idea of always normalizing the gradient to have a fixed unit magnitude for weight updates to be consistently controlled and stable which ideally has no or little overshooting or oscillating convergence stability issue if an optimal solution actually exists.


text:
Where can I find good sources (textbook, lecture notes, etc) on multi-armed bandits?
I am looking for good sources (textbook, lecture notes, etc)
on multi-armed bandits where both theory and practical examples are given.
I will be happy for suggestion of such material.

There is the bandit book by Lattimore and Szepesvari, which is, I think, the main reference in the field: you can find pretty everything about Bandits, even more recent and research oriented topics.
Keep in main that it is a strongly theoretical book.
Update

RL: an introduction by Sutton & Barto: the chapter 2 is about multi-armed bandits. It's still theoretical but with some pseudo-code example, also the authors provides some bibliography.
Mastering RL with Python: see chapter 2 (MAB) and chapter 3 (contextual bandits). This is a practical book, with code examples in Python. Also you can find the code here.

I know this but it is too mathematical.

@DSPinfinity I can't blame you. I'll see if I can find something easier


text:
Why is the loss function for the critic decreasing and the gradient increasing over the episodes?
I have built an Actor Critic model, where we have a Quantile Critic.
The aim of the task is to tell what is the optimal portfolio choice of an agent based on quantiles.
The model set up: We have continuous case so I am using Gaussian policy ( usual set up, entropy regularization etc) and I have simple Quantile Critic model. The  kernel_initializer is HeNormal().Also, I have a Flatten layer before the output layer.
Why is the loss function for the critic decreasing and the gradient increasing over the episodes?
Also, the model is too volatile in the testing data.

Can you please provide a link to this "actor-critic model with quantile critic"? Is this something that you came up with or something that exists in the literature? If the latter, please, provide a link to a research or book that describes it. It might also be a good idea to provide more info about what you mean by "too volatile in the testing data", even if you provide the images.

Thank you for answering. The model is mine, I came up with. I cannot really share because of my job. The most important part is the increasing gradient, if you could help me what should i check and explain why it can happen, that would be awesome. What can be the explanation? Isnt is weird that the loss is decreasing parallel? There are some structural breaks in the data, that is why there are some peaks in the loss.
As for the volatility part: I run many testing episodes and got different result. But this is expected as we have continuous problem with gaussian policy.


text:
What is the name of this construction for a compound policy that operates over distinct action sets?
I am developing an RL algorithm with a policy that needs to compute valid probabilities over multiple distinct action sets. I think I have a construction that will work, but I do not know what it is called, or if it even has a name:
Given two discrete policies $\pi_{A_1}: A_1 \times S \rightarrow [0, 1]$ and $\pi_{A_2}: A_2 \times S \rightarrow [0, 1]$, there is a policy $\pi_{A_1 \cup A_1}: (A_1 \cup A_2) \times S \rightarrow [0, 1]$ defined:
$$
\begin{align*}
\pi_{A_1 \cup A_2}(x \mid s) =
\left\{
    \begin{array}{lr}
        m_{1} \cdot \pi_{A_1}(x \mid s) & x \in A_1 \\
        m_{2} \cdot \pi_{A_2}(x \mid s) & x \in A_2
    \end{array}
\right\}
\end{align*}
$$
where $m_{1} + m_{2} = 1$, $m_{i} \geq 0$, and $A_1 \cap A_2 = \emptyset$.
The proof that this is a valid policy is simple:
$$
\begin{equation}
\begin{split}
\sum_{x \in A_1 \cup A_2} \pi_{A_1 \cup A_2}(x \mid s) &= \sum_{a_1 \in A} m_{1} \cdot \pi_{A_1}(a_1 \mid s) + \sum_{a_2 \in A_2} m_{2} \cdot \pi_{A_2}(a_2 \mid s) \\
&= m_{1} \cdot \sum_{a_1 \in A_1} \pi_{A_1}(a_1 \mid s) + m_{2} \cdot \sum_{a_2 \in A_2} \pi_{A_2}(a_2 \mid s) \\
&= m_{1} + m_{2} \\
&= 1
\end{split}
\end{equation}
$$
It doesn't look like a mixture policy due to the splitting behavior. Again, I'm curious if this has a name, or what other constructions to which it might be related.


text:
How come all the multi-headed self-attention layers don't end up learning the same aspect of a natural language?
How come all the multi-headed self-attention layers don't end up learning the same aspect of a natural language? Since we don't dictate ahead of time what the self-attention layers focus on, how do we ensure they don't "converge"?
(My question is partly influenced by the notion of kernels in computer vision. Those kernels also focus on different aspects of the image, but they don't end up learning the same thing since they're specified ahead of time.)

They all have different weight initializations. I think that the chance of gradient descent discovering the same local minima in such high dimensions is low.


text:
Do probabilities that don't vary much from the average contribute to greater entropy?
If the entropy of information reaches its maximum when all values of X are equally probable, indicating maximum uncertainty, do probabilities that don't vary much from the average contribute to greater entropy?

Can you please explain how this is related to AI? I know that this can be related to AI but it might not. So, please, provide more info about why you're asking this question.

If you have a finite set of possibilities (such as 0/1 for binary), the entropy is maximized, given that the probability of observing one or the other is 0.5, and nothing can increase it further
If you have a distribution that is close to the uniform, your entropy is high, but it's not maximized


text:
What does "aligned" across domains in domain adaptation?
within Delving into Local Features for Open-Set Domain Adaptation in Fundus Image Analysis paper. I got trouble in understanding their cluster-aware contrastive adaption $\mathcal{L}_\text{cca}$.

I don't get it how they defined "$i$ and $j$ are aligned", and how is consider "mis-aligned".
This is what they say when they highlighted their contributions: (CCA) method is proposed to enforce the local features of aligned clusters across domains to remain close-by, while pushing those of the misaligned clusters far apart.

(Blue is source domain, Orange is target domain).This is open-set domain adaptation. I believe each prototype is the mean (center) of each cluster. and they say $i$ (a cluster) from source, $j$ (a cluster) from target.
After scanning the paper for a week, I still don't understand what is "aligned" here. I sent them email twice (politely) but no reply.
Can any one enlighten me a little bit?

To provide some context for this answer, the referenced paper is dealing with the problem of unsupervised domain adaptation (UDA). In UDA, there are two (or more) datasets, each drawn from a different, but related domain. One of these datasets (the source) is labelled, and we want to use that data to train a machine learning model to predict the labels for the unlabelled (or target) datasets. The complicating factor is that the distribution of the data from each domain is different, so a model trained on the source data in the usual way does not generalise to the target domain. In open-set domain adaptation the distributions of the source and target labels are different and there are one or more target labels that are not represented in the source dataset.
The paper addresses this problem by training a model jointly on the source and target data. The idea behind the joint training is that the deep learning model is trained to extract domain-invariant features from the input, which are then used in a classifier. As the classifier is trained on these domain-invariant features, it should be able to classify samples from the target domain, as well as from the source domain. For the source data, the usual cross-entropy loss is used. For the target data they use two loss functions, one of which is the above-mentioned cluster-aware contrastive adaptive loss.
To calculate the cluster-aware contrastive adaptive loss, the learned feature maps for the source and target data are clustered separately, and then the clusters are aligned. Minimizing this loss function trains the model to extract features that bring aligned clusters closer together and pushes unaligned clusters further apart.
The authors don't go into a lot of detail about how the clusters are aligned. But they do tell us they use the "domain consensus score" and cite the paper Domain consensus clustering for universal domain adaptation by Guangrui Li et al. According to this paper, they find the centre of each cluster, then for each cluster in one domain they simply find the cluster in the other domain with the nearest cluster centre. If the two clusters (one from each domain) are each other's nearest centre, then they describe this as reaching consensus. They also compute a domain consensus score, which is simply the mean of the proportion of cluster members that are located in the intersection of the two clusters.

Your last paragraph is useful actually for this question but still leaving questions raised for how they mentioned threshold of consensus score ? why no repulsion for intra-domain clusters (look at that image)

Your last paragraph is useful actually for this question but still leaving questions raised for how they mentioned threshold of consensus score ?
why no repulsion force for intra-domain clusters (look at that image, no repulsion between 2 blue clusters)?
The use of delta as a upper bound for repulsion force ? why that ? is that mean that as a certain point, repulsion can harm for learning invariant-domain feature ? too much force cause loss of balance in clustering ?
Can you elaborate more on DCC paper ? (just a feedback, first 2 para are common knowledge, can get from gpt or survey)


text:
Why are these two implementations of the $\epsilon$-greedy policy different?
According to the book Reinforcement Learning An Introduction, the epsilon greedy policy can generally implemented as:
$$
\pi(a|s) =
\begin{cases}
\frac{\epsilon}{|A|} + 1 - \epsilon & \text{if } a = \arg\max_{a'} Q(s, a') \\\\
\frac{\epsilon}{|A|} & \text{otherwise}
\end{cases}
$$
I followed an exercise from
https://github.com/dennybritz/reinforcement-learning/blob/master/TD/Q-Learning%20Solution.ipynb
where the policy is implemented in Python like this:
def policy_fn(observation):
    A = np.ones(nA, dtype=float) * epsilon / nA
    best_action = np.argmax(Q[observation])
    A[best_action] += (1.0 - epsilon)
    return A

I wonder why this implementation is different from:
def policy_fn(observation):
    p = np.random.rand()
    if p > epsilon:
        return np.argmax(Q[observation])
    return np.random.randint(0, nA)

At least, I get different results for the final action-state pairs.
In my opinion, the best action gets sampled with a probability of $1 -\epsilon$. Otherwise, each action gets chosen with the same probability.

they look equivalent to me. let's look at what happens: with probability $\epsilon$, an action is chosen uniformly at random. That means that all actions have a probability of $\epsilon / |\mathcal{A}|$ of being selected. However, with probability $1 - \epsilon$, we select the greedy action. So, the probability of the greedy action being chosen is $\epsilon / |\mathcal{A}| + 1 - \epsilon$. Both the functions look to implement the same thing, where the first returns the probability of each action being selected (and thus should then be sampled from) and the latter returns the action directly.

The two implementations you posted are different, but they do represent the same $\epsilon$-greedy policy.
The first function returns an array A which contains the probabilities of each action choice. In order to get an action choice you need to run something like np.random.choice(np.arange(len(A)), p=A) to sample from it.
The second function directly returns an action choice.
You might want the policy probabilities from the first approach for some kinds of reinforcement learning methods. For instance, importance sampling for off-policy learning needs to divide target policy probability by behaviour policy probability. Having a policy method that returns the full distribution of action choices is a generic way to represent a policy, allowing you to work with the policy in other contexts than taking an action.
You might otherwise prefer the second direct action choice because then you do not need to add extra code to make that choice in the inner loop of the learning algorithm. Technically this is not just representing the policy, but actually sampling from the policy as well.


text:
What are the similarities between Q-learning and Value Iteration?

This is the explanation of value iteration in our notes where you keep applying bellman optimality equation till it stops changing and then acting greedily wrt the value function gives the optimal policy.
Is using Q learning for control the same for finding optimal policy? Or do I have to perform generalized policy iteration where q learning is for evaluation and some other method for improvement till convergence?

Q learning is very similar to value iteration. They are based on the same principles. A key similarity is that both assume a greedy action choice on the bootstrap next state value.
The big difference is that value iteration is based on processing all states and possible actions whilst knowing the MDP model in detail. For Q learning, the value function is instead learned by sampling from experience, and it does not require knowledge of the MDP model, beyond the state and action representations.
Q learning typically uses an $\epsilon$-greedy policy derived from its current action value function estimate, in order to drive exploration and take samples. Whilst value iteration neither explores nor exploits nor samples, instead resolving based on the model, processing all states and actions. Value iteration can be thought of as a kind of planning algorithm.

Or do I have to perform generalized policy iteration where q learning is for evaluation and some other method for improvement till convergence?

Q learning is already a form of generalised policy iteration (GPI). GPI is the underlying principle behind value-based control methods. You can think of policy iteration (and value iteration) as the "standard" for accuracy, but it doesn't scale well or apply directly when the model is not known. GPI is an abstract view of how learning values and then basing policies on greedily selecting on those values should eventually converge on an optimal policy.
The difference between using off policy TD learning for evaluation or control is in how you select policies. For evaluation you would have a fixed target policy used to calculate bootstrap values (for TD error or TD target). For control scenarios, the target policy is always the greedy action choice based on current value estimates, which is a policy that changes over time. Q learning is exclusively a TD control method. It is easy to adapt it to create an evaluation method, but AFAIK that would not be called Q learning.


text:
What are the techniques used to initialize weights for neural networks?
When creating a neural network to predict the impact of risks on the project cost, what techniques are used to initialize the weights provided to the hidden layers and the output layer?

Both Keras and PyTorch provide a range of  initialization classes and functions. Of these, probably the two most commonly used are the Glorot (Xavier in Pytorch) and He (Kaiming in pytorch) initializers. Additionally, biases are commonly initialized to 0 - see for example this answer on Stack Overflow: Initial bias values for a neural network.
Glorot initialization (Glorot and Bengio: Understanding the difficulty of training deep feedforward neural networks Xavier) was designed for NNs that use sigmoid or tanh activation functions.
He initialization (He at al.: Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification) was designed for NNs that use rectifiers such as RELU.
Each of these has a normal and uniform variant - i.e. the initial weights are chosen by sampling from a normal or uniform distribution with a mean of zero. They vary in the way the standard deviation of the normal distribution or limits of the uniform distribution are calculated. The Glorot initializer is based on the sum of the input and output units to the weight tensor, whereas the He initializer just uses the number of input units.

Both Keras and Torch provide a wide range of techniques for weight initialization. Like
Zero initialization :- Initializing all weights to zero it is easy way to initialize the weights and biases but it is not very effective since all neuron learns same things and predicts same things which leads to symmetry breaking issue.
One initialization :- This method is as same as zero initialization in this method instead of zeros we initialize all weights with zeros but again it also leads to symmetry breaking down.
Random Initialization :- In this method weights are initialize based on normal or uniform distribution. This method is common and mostly use in initialization of the weights of neurons. and it also solves the problems like stucking at local minima. but there are chances of gradient vanishing and gradient exploding.
As per Lynn said you can use the initialization like He and glorot which are used for specific types of layers which gives much better performance of that specific types of layers.
For more information you can check this links
for Keras :https://keras.io/api/layers/initializers/
for pytorch:https://pytorch.org/docs/stable/nn.init.html

Yes, weight initialization in the network has had a high impact on the model. The choice of weight initialization depends on a few factors like the architecture of the neural network, activation function, and input data. you can share those things that help us help you with which activation function to use. e.g. if you are using Deep Network with ReLu then He initialization is a good choice, for a network with Sigmoid and Tanh use XavierGlorot.
Here I am listing a few Weight initialization techniques and when to use them:
Zero:

All Weights are initialized to Zero.
Mostly it is not recommended because it stops effective learning, because all the layers are going to learn almost the same features while training.

Random:

Weights are initialized with random values with the distribution of normal or uniform.
It is better to break symmetry but it leads to vanishing and exploding gradient. Mostly in POCs, we start from this weighting technique.

Xavier/Glorot:

Weights are in distribution with 0 mean and std. deviation of the sqrt(2/(inputs+outputs))
Useful when networks use tanh or sigmoid activation because it helps in maintaining variance of the activation function and back prop gradient

He:

It is similar to the Xavier with 0 mean and std. deviation of the sqrt(2/inputs).
Ideals for networks like ReLU and variants. It helps in mitigation of the vanishing gradient.

Orthogonal:

In this weights are orthogonal metrics. This helps in norm preservation.
Best suited for RNN because it helps in maintaining gradient magrnitude in deeper networks.

Sparse:

Some of the weights are set to non-zero values while others are zero.
Useful in case of sparse network data type handing.


text:
How can I deal with random weights initialisation when predicting a time-series sine function?
I am training a simple RNN model in keras to predict a time series.
The time series I am considering is just a sine function

The task to solve is the following:

Given a timeseries of 90 time steps, predict the next 10 time steps.

I am generating my training set by sampling from

where $\epsilon$ is just a noise term that shifts the series. I have added the noise to have a good training sample.
Question
I train my model for 10 epochs. I notice that sometimes the predictions are actually very good. Some other times, if I retrain the same model with the same dataset, the predictions are very bad. This has to do with the randomness of weights initialisations, but I was hoping that the training algorithm would have found the "best" solution anyway, irrespectively of the initial weights. When I say "best" solution, I mean a class of solutions whose variance is small.
How can I deal with random weights initialisation? Do I need to add some regulation terms to my network?
I know that I can define an ensemble of multiple models in order to cope with weights initialisation. However, this is a very simple problem, and I was expecting to be able to solve it with a single simple model.
Description of the model and code
You can find a Jupyter notebook here .
My training sample will be time series of lenght 90 and the targets will be the value of the series at the 91st step.
I will predict the 10 steps by first predicting the 91st step. Then, I will plug it in the sample. I will select the last 90 terms and predict the 92nd step and so on.
I generate the samples with this code
def generate_series(n_series, n_timesteps, scale_noise=0, scale_shift_origin=10):
    """
    Generate random sin series for testing purposes.
    """
    series = []
    for _ in range(n_series):
        noise = np.random.normal(loc=0.0, scale=scale_noise, size=n_timesteps)
        noise_origin = np.random.normal(loc=1, scale=scale_shift_origin, size=1)
        x = np.linspace(start=0, stop=100, num=n_timesteps)
        y = np.sin(0.3*(x + noise_origin)) + noise
        series.append(y)
    return np.array(series)

keras.utils.set_random_seed(42)

series = generate_series(10000,100)

X_train, y_train = series[:8000,:-10], series[:8000,-10]
X_train = X_train.reshape(8000, -1,1)
y_train = y_train.reshape(8000, -1)

X_test, y_test = series[8000:,:-10], series[8000:,-10]
X_test = X_test.reshape(2000, -1,1)
y_test = y_test.reshape(2000, -1)

The model I am using is very simple, it is a stack of two RNN layers and a Dense layer.
class TwoLSTM:
    def __init__(self, n_units_1:int, n_units_2:int):
        self.n_units_1 = n_units_1
        self.n_units_2 = n_units_2
        self.model = None

        self.build()

    def build(self):
        self.model = keras.models.Sequential([
        keras.layers.LSTM(self.n_units_1, return_sequences=True, input_shape=[None, 1]),
        keras.layers.LSTM(self.n_units_2),
        keras.layers.Dense(1)])

        self.model.compile(loss="mse", optimizer="adam")

    def fit(self, train:tuple, test:tuple, epochs:int, batch_size:int=32,verbose=0):
        X_train, y_train = train
        X_test, y_test = test

        callbacks = [
        keras.callbacks.EarlyStopping(monitor='val_loss', patience=10, mode='min', restore_best_weights=True)]
        history = self.model.fit(X_train, y_train,
                        epochs=epochs, batch_size=batch_size,
                        validation_data=(X_test, y_test), callbacks=callbacks,
                        shuffle=True, verbose=verbose)
        return history

I generate the predictions with this code
def generate_predictions(series, model):
    X = series.reshape(1,-1,1)
    X_new = X[:,:90,:]
    for step in range(10):
        X_to_use = X_new[:,step:90+step,:]
        y_pred = model.predict(X_to_use, verbose=0).reshape(1,-1,1)
        X_new = np.concatenate([X_new, y_pred], axis=1)

    return X_new

The first training of 10 epochs returns the following prediction of a series in the training set

and the log of the loss function during training are

Then, I train again the same model (by redefining it again; I don't use the best weights of the first run as starting point) and this time I get very good predictions and loss functions

and

What values are you using for n_units_1 and n_units_2 for those graphs?

@NeilSlater sorry, I should have written it in the post. I am using n_units_1 = n_units_2 = 1. It is on the jupyter notebook though.

From comments, you are testing your ideas with a very small neural networks.
The highly variable end result with large dependency on initial conditions is a common result of working with small numbers of neurons. Random initialisation works in part because you can reasonably expect a mean activation value when considered over a layer's output. With a very small number of neurons in any layer, this is less likely to happen. The assumptions behind the statistical analysis that suggest how to properly initialise the parameters in a layer don't hold for very small layers.
Increasing n_units_1 and n_units_2 to a slightly higher value, say 10, should resolve this provided you are using a normal initialisation routine, such as Keras' default.
Another thing that can help is adding some regularisation other than early stopping. A small amount of weight decay (or L2 weight normalisation) may work. Usually you would not apply regularisation to very low number of neurons though, since the low learning capacity of the network means it is already quite restricted. However, one of the issues you may face when increasing the size of the network is that its learning capacity will be much larger than the simple problem you currently are testing it with. So you may need to start addressing over-fitting. Try without it initially though.
There is probably a sweet spot for the layer sizes, where 99%+ of your runs will be stable and accurate over the whole domain, but you won't need any more regularisation.


text:
Ways to train a neural network continuosly as new data is added
There is a project I'm currently working on that requires object detection with continuous training. The idea is to train a model beforehand with a standard dataset. When I get new images I want to "teach" the model with theses new images. The procedure is the following:

Train a neural network with a standard dataset for object detection.
Create a script that will deploy the model, capture camera images and make inference on them.
Since all processed images are stored in a computer, I want to use them for training, example: for a true positive the model learns the object, for a false positive, the model learns that the object detected is not from the desired class, for false negatives the user needs to create a label and train the model. Also the model needs to be trained with true negatives.
The learning should be periodically based on new images (When the user presses a certain button or automatically).
The model should not forget the features from old images, or at least retain the most possible features from them.

Basically the model should improve continuously as new images are gathered.
I tried finding the technique on google, but for this specific scenario I can't find anything. I found some papers about the incremental learning, but they refer on their solution about adding new classes to the model. That is not what I want. For now, I need only one class, hopefully I can add more in the future for the same model.

What kinds of objects are being identified, and how many classes? In addition, could this be a kind of identity system (e.g. face recognition, pet recognition, or any named objects from similar classes where there could be 1000s of classes)? Identity systems usually have a different approach to training and finding matches which is better than continuous training

It is 1 class for the model. We want to detect people from a surveillance camera. The idea is to use the object detection method. For now we are using yolov5, but it is not possible to do want I want, which is incremental training by "feeding" new labelled images to the model (continuously).

How much labelled data are you starting with? Is this a single installation, or at least a single central model, or is the idea that separate installations learn their own unique local models?

Hopefully we will start with ~5000 images with different scenarios of people (different distances from the camera to the people, different illumination, day and night images, etc.). The idea is to have a central model (default) to each system and each model will be updated with the images of that system, so that the model learns the environment/context of the site.

There are different terms that refer to similar systems you're looking for. I had provided an answer here with more details. I'd assume there are techniques to handle catastrophic forgetting without adding new classes, although it's been a while since I studied this topic. Feel free to provide an answer to your own question once you know it.


text:
Would a pipeline of different models be considered Ensemble Learning?
For example, if I have a problem in which I try to predict if it is a nice day for jogging from a corpus of images, I might first convert the images to text descriptions (ex. raining in forrest, cloudy in city) and then use a binary classifier to determine if those descriptions are describing a nice day for jogging or not.
Would a system like this, which takes the output of one model and uses it as the input for another, be considered Ensemble Learning?

No. Ensemble Learning (EL) is a way to improve the model performance, which usually means to reduce bias (i.e., get a better model class) or reduce variance (i.e., get better at generalizing across samplings of the dataset.)
Moreover, in EL you need to ensure a "property" (I'll explain in a bit) among the learners (i.e., the models in the ensemble.), which also defines the kind of EL:

In boosting we need the learners to be correlated (this is the property I was referring to) to each other to effectively reduce the bias.
Conversely, in bagging we need the models to do independent errors otherwise we can't reduce variance by averaging them out.

In your case you're chaining two models, thus not combining/aggregating their predictions. When you chain one or more models you have to consider that each model made an error, say $e$, and that each time you chain the next model would receive an input (which is the output of the previous model) that already incurs an error $e$, therefore predicting an output associated to an even higher error, say $e'$ with $e'>e$. Usually it happens that the whole error, say $E=e+e'+\cdots$, behaves like the sum of individual errors which can even grow quite easily, making your own pipeline drift away quite easily.
A way to mitigate this issue, is to train the whole pipeline of models in an end-to-end fashion (i.e., you back-propagate the error through all the models), instead of training each model in isolation until the local optimal is reached.

Would a system like this, which takes the output of one model and uses it as the input for another, be considered Ensemble Learning?

Not usually. The main criteria to consider something as an ensemble learning setup is that there are multiple different learning units -maybe different algorithm - all working with the same inputs and target values. They are then aggregated in some way (taking a mean value, voting, maybe weighted) to get the ensemble's output.
So to take your example, if you ran multiple types of binary classifier to classify the text at the end of the pipeline, and combined their outputs using a majority vote rule, then that part of your complex model would be considered as an ensemble.
These terms for system architecture - pipeline, ensemble, stacking etc - are not necessarily formal though. They are for communication about engineering setup, more than mathematical definitions.


text:
What is an information bottleneck in the context of ELBO and Hierarchical VAEs?
These slides (slide number 26) mention that the ELBO enforces an information bottleneck at the latent variables z which make it prone to bad local minima.
Can you please explain what they mean by that?

I'd need to read the slides, but probably this only means that essentially we compress the inputs into a vector, which places a restriction on the amount of information you can contain from the input. Now, why it leads to local minima is a different question. Not evident to answer I guess.

In the slides, they aim to explain the use of Hierarchical VAEs and the types of problems they address. "Evidence Lower Bound" (ELBO) measures a lower bound approximation of the log-likelihood distribution for the observed data. A VAE estimates the real distribution of $p(x)$ through a Normal Distribution, utilizing the ELBO estimation, which in a continuous space is expressed as:
$$ELBO = \int q(z|x) \log \left( \frac{p(x,z)}{q(z|x)}\right) dz$$
This formula arises from marginalizing the latent variables and applying Bayes' rule. However, due to the integral's intractability, the real distribution of $p(x)$ is approximated by the ELBO loss function.
The ELBO loss function employs KL divergence, which, outside the probabilistic perspective, acts as a regularizer for the observed data. The reconstruction process is similar to an Autoencoder loss, but instead of using Euclidean distances with Mean Squared Error (MSE), it utilizes probabilities with Sigmoids and Binary Cross Entropy to align with the nature of the image data.
This regularization makes it challenging for the model to specialize in generating a specific data point. The VAE's objective is not just to reconstruct observed data but to use a learned distribution for reconstructing the learned data, which is markedly different. This addition of a learned distribution results in a more general yet sparser representation, often leading to a mean representation of the data.
Typically, this is undesirable for a neural network. We generally don't want models to learn a mean representation of the data, as this can be easily achieved with other methods like PCA, which employs linear approximations through its eigenvectors. Neural Networks should represent complex nonlinear functions, like is the case of $p(x)$ if $x$ represents the random variable for the sets of crisp and realistic images .
So in the slides when they mention that ELBO enforces an information bottleneck at the latent variables
$z$. The z (latent variables) generating the normal distributions from which we sample to retrieve $p(x)$ (the image data) are trapped in a suboptimal local minimum. This means that these Normal Distributions generated by the $\sigma$ and $\mu$ vectors generated by the encoder are not optimally tuned for the specific set of the observed data.
This issue is common in unsupervised learning when the target distribution is unknown, and the model is expected to identify or learn it. In the case of VAEs, the goal is not to replicate the discrete distribution of the exact observed data, but to recognize that the observed data may represent a more general continous distribution.
A potential solution is to segment the continuous latent space into discrete portions. This approach enables the model to learn more specific data distributions, narrowing the data possibilities. This is the reason behind using VQ-VAEs, which balance generality and improved data modeling for a specific dataset. Another solutions for this, is to use  diffusion models, they seem to address these issues without necessitating the quantization of the vector space.