text:
What is the time complexity for testing a stacked LSTM model?
In the data preparation phase, we have to divide the dataset into two parts: the training dataset and the test dataset.
I have seen this post regarding the time complexity for training a model.
However, I couldn't find any good source for the time complexity for testing a model, specifically, a stacked LSTM model (with 1 input, 3 layers, 4 LSTM units per LSTM layer, 1 output, sequence 18, and batch size 32 for MSE loss), based on the test set, e.g. the computation of the accuracy/loss on the test set given $N$ test examples.
Is there any source to check that?

The time complexity of an algorithm always depends on its implementation (e.g. searching in a red-black tree has a different time complexity than searching in an unbalanced binary search tree). This also applies to the case of computing the time complexity of the algorithm that tests a neural network with multiple LSTM layers, so one may need to assume how an algorithm or data structure is implemented in order to provide the right time complexity.
Now, note that testing a neural network typically means performing a forward pass. So, basically, if you have the time complexity for the forward pass, you are almost done. In this answer, I provide details of how you can compute the time complexity of the forward pass of a feedforward neural network with no recurrent layers, so you can read it in order to have an idea of how you might do this in general.
Now, in the case of a neural network with LSTM layers, we also need to take into account the recurrent connections in the LSTM units. This would also be the case for the vanilla RNN units, which are simpler than LSTM units, so the time complexity is smaller.
I will assume that the equations used to perform the forward pass of an LSTM layer are the ones given in the PyTorch documentation
$$
\begin{aligned}
i_{t} &=\sigma\left(W_{i i} x_{t}+b_{i i}+W_{h i} h_{t-1}+b_{h i}\right) \\
f_{t} &=\sigma\left(W_{i f} x_{t}+b_{i f}+W_{h f} h_{t-1}+b_{h f}\right) \\
g_{t} &=\tanh \left(W_{i g} x_{t}+b_{i g}+W_{h g} h_{t-1}+b_{h g}\right) \\
o_{t} &=\sigma\left(W_{i o} x_{t}+b_{i o}+W_{h o} h_{t-1}+b_{h o}\right) \\
c_{t} &=f_{t} \odot c_{t-1}+i_{t} \odot g_{t} \\
h_{t} &=o_{t} \odot \tanh \left(c_{t}\right)
\end{aligned}
$$
Now, let's break these operations down. Note that the dimensionality and type of the objects (matrices and vectors) are important. I will start with the computation of $i_{t}$ (the output of the input gate). The computation of $f_{t}$ (forget gate), $g_{t}$ (cell) and $o_{t}$ (output gate) is the same, in terms of time complexity.
$$i_{t} =\sigma\left(W_{i i} x_{t}+b_{i i}+W_{h i} h_{t-1}+b_{h i}\right)$$

$W_{i i} \in \mathbb{R}^{h \times d}$ is the matrix that represents the forward connections for the input gate
$x_{t} \in \mathbb{R}^d$ is the input vector with $d$ entries (this could be e.g. the embedding of a word)
$b_{i i} \in \mathbb{R}^h$ is a bias vector for this forward connetions
$W_{h i} \in \mathbb{R}^{h \times h}$ is the matrix for the recurrent connections; note that this matrix has a different dimensionality than $W_{i i}$
$h_{t-1} \in \mathbb{R}^h$ is the hidden state
$b_{h i} \in \mathbb{R}^h$ is the bias vector for the recurrent operations

Here is an estimate of the time complexities of the operations inside the non-linearity

$W_{i i} x_{t}$ has time complexity $\mathcal{O}(h d)$ (matrix-vector multiplication)
$W_{i i} x_{t}+b_{i i}$ has time complexity $\mathcal{O}(h)$ (sum of two vectors, both with size $h$)
$W_{h i} h_{t-1}$ has time complexity $\mathcal{O}(h^2)$
$W_{h i} h_{t-1}+b_{h i}$ has time complexity $\mathcal{O}(h)$

Now, let $N = \max(d, h)$, then the time complexity of the operations inside the non-linearity is $\mathcal{O}(hN)$. More precisely, it would be
$$\mathcal{O}(h d + h + h^2 + h) = \mathcal{O}(h d + 2h + h^2) = \mathcal{O}(h(d + 2 + h)).$$
Actually, in this case, as I explain in the other answer, you could use $\Theta(hN)$ because this is an upper and lower bound in the limit.
Now, what is the time complexity of $\sigma$? It depends on what $\sigma$ is. If we assume it's a ReLU, then you perform basically the following operation
$$\displaystyle \sigma(x) = \max(0,x)$$
Let's assume that $\max(0,x)$ has a constant time complexity, so, if you apply the ReLU element-wise, you get a time complexity of $h$.
So, basically, the computation of $i_t$ has time complexity $\mathcal{O}(hN)$. The same applies to $f_{t}$ (forget gate), $g_{t}$ (cell) and $o_{t}$ (output gate), as stated above, so you could multiply this time complexity by $4$, but constant factors don't affect the time complexity. However, this is in theory, i.e. in the limit, in practice, they affect, so you probably want to include those. So, the time complexity, let's say, is
$$\mathcal{O}(4h(d + 2 + h))$$
so far (ignoring there's a tanh there: for simplicity, I assume it has the same time complexity as the ReLU even though that may not be the case: you can work out the details!)
Now, we have the following operation
$$c_{t} =f_{t} \odot c_{t-1}+i_{t} \odot g_{t},$$
where

$c_t \in \mathbb{R}^h$ is the cell vector
$\odot$ is an element-wise multiplication

So, this operation has time complexity
$$\mathcal{O}(2 h)$$
Similarly, the operation
$$h_{t} =o_{t} \odot \tanh \left(c_{t}\right)$$
has time complexity
$$\mathcal{O}(2 h)$$
One $h$ for the tanh and the other for the element-wise multiplication.
So, overall, the forward pass of a single LSTM layer has time complexity
$$\mathcal{O}(4h(d + 2 + h) + 4h) = \mathcal{O}(4h(d + 3 + h))$$
Now, you can work out the time complexity for multiple LSTM layers stacked one after the other. Just make sure that you understand that what is passed to the successive layer is not the input $x_t$ but the output of the previous layer.
In the end, you will also have to sum the complexity of the computation of the metric you're interested, which can be the loss or accuracy. If you understood my reasoning above, it should not be difficult to do that. The same applies for the case of a batch size and sequence size greater than 1.

what is your source ? I search a paper or course on "big O notation on deep learning" and I haven't found anything reliable.
Thanks for the advance :)

@user75427 My source is my background in Computer Science (bachelor) and AI (master). Every computer scientist student learns how to compute the Big-O notation of algorithms. But honestly I don't remember what I wrote in this answer anymore. It was already a long time ago. If there's anything unclear about my answer, let me know, and I'll read it again and try to clarify your doubts.


text:
NEAT: How to properly handle Node IDs and avoid Competing Conventions?
I'm working on yet another NEAT implementation for a personal project, and I feel like I'm missing something about the proposed solution to the Competing Conventions problem.
Here's what I'm assuming:

Each new connection gene yields a new innovation number.

The new connection gene created in the first mutation is assigned the number 7, and the two new connection genes added during the new node mutation are assigned the numbers 8 and 9.

If a connection from X to Y appears more than once at the same generation, it receives the same innovation number.

However,  by  keeping  a  list  of  the  innovations  that  occurred  in  the  current  generation,  it is possible to ensure that when the same structure arises more than once through independent mutations in the same generation, each identical mutation is assigned the same innovation number.

Every new node receives a globally new id. (I have no source for this. Maybe the problem is here)
Equal fitness is assumed at the example below, so all genes are randomly inherited (on this case the probability is 100%, for demonstration purposes).

In this case, equal fitnesses are assumed on the example below, so the disjoint and excess genes are also inherited randomly.

So we have two genomes after two generations, which suffered the exact same mutations at each generation: add_connection(0,2) at gen 1, add_node(1) at gen 2.

Even though the topology is exactly the same for both, the genomes don't share innovation numbers, so the crossover yields a more complex topology.
It seems to me that using global ids for nodes breaks the historical tracking of connections, which blows up the innovation counter pretty fast (around 1400 innovations in 100 gens for 100 individuals), and yields big, non-functional networks.
What am I missing?

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Is it possible to describe wordle as a multi armed bandit problem?
Game wordle got fame in January 2022 on Twitter (now knows as X).
Rules are pretty simple.

Player has to guess a word of length 5
Player can make at most 6 guesses
If player guesses the correct word or when 6 guesses are exhausted, game ends.
Once player guesses any word, he/she gains information regarding the actual word using colors. If a character is placed at correct position, it is denoted by green color. If a character is in the actual word but at a different position, then it is denoted by yellow color.

I want to describe this as a reinforcement learning problem. My question is: Can we describe this as a multi armed bandit problem ?
Edit-1: I have found these two research papers which address wordle as reinforcement learning problem. None of them has talked about seeing wordle as a bandit problem.

D Bertsimas et al.
S Bhambri et al.

Is it possible to describe wordle as a multi armed bandit problem?

Yes, although the data management to present the current choice would involve constructing a state much like a reinforcement learning (RL) version of the same problem.
An agent that attempted to solve Wordle with a multi-armed bandit approach would likely be less efficient than the RL version, since the reinforcement learning version would be able to select an answer with a lower immediate chance of winning, but that gave more information and thus a higher chance of solving. You can intuitively see that is the case since a bandit version would learn to equally select any one of the few thousand possible start words (assuming they are equally likely to be chosen by the problem setter) for a 1-in-2390 probability of winning, whilst it is a proven strategy that some start words are more informative when they fail. Bandit algorithms however do not consider "next state".
A multi-armed bandit version would be a contextual bandit, with the input state being known letter information from previous attempts - e.g. an array of 26 x 5 x 2 with the known results of placing each letter in each position so far. The output could be the possible class values - e.g. all allowed words, one-hot encoded. Other structures are possible. You could also make the agent's life slightly easier by ruling out words logically - although that's 90% of the work so there's not much learning to do at that point. A bandit's best strategy assuming the target word was chosen randomly, is simply to choose from available valid words completely at random*.
The reinforcement learning version could use identical structures, but in its Q table it would be able to learn about the advantages to reducing the space of valid words through careful choice in early guesses.
An optimal play of Wordle assuming random pick of target word is also entirely possible using a logical look-ahead search. By choosing test words that are guaranteed to split the remaining valid words into the smallest groups (probabilistically) depending on the results, you can dramatically reduce the search space in only a few steps. This resolves to minimising the mean square size of each group (the frequency of ending up in the group times the remaining size). You could probably write this optimal search just as easily as writing a bandit or RL agent to solve the problem. However, that only makes sense if you want an optimal agent by any means - if you want to use Wordle as a way to try out RL or bandit algorithms, then there is no need to write this "expert" agent.

* You could use reward shaping to reward the bandit for reducing the remaining candidate words. At that point you have almost implemented the full reinforcement learning version by proxy though.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Can I use Sentence-Bert to embed event triples?
I extracted event triples from sentences using OpenIE. Can I concatenate the components in the event triple to make it a sentence and use Sentence-Bert to embed the event?
It seems no one has done this way before so I am questioning my idea.

Sentence-Bert can embed any chain of characters in english and create semantically related embeddings (or at least contextual character-based if the model is dealing with OOV).
Even if no one else did it before, you can do it !

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.


text:
GAN performance starts to get worse as training continues
I'm currently trying to train a GAN to recreate similar images from a dataset. The dataset is using the Eiffel Tower Pictures from Googles Quick Draw dataset. The images aren't very large (only 12x12 pixels) and are all black and white.
The performance increases at an expected rate initially however after a certain point the quality begins to go down despite the cost from both generator and discriminator networks seeming to stay at a steady value as expected.

I've tried changing the learning rate and other hyper parameters however they all end up with the same result of the network eventually getting worse.
My learning rate is currently 0.01 which I'm aware is quite high compared to what other people are using but anything lower takes too long to train and the results don't seem any better even if I do give it long enough.
Any pointers on what could be causing this or the specific name of this problem if it's a common issue with GANs would be appreciated.
Thanks

Which adversarial loss are you using? Also, is the drop in quality constant in time or do the images get worse/better/worse/better? It might be that the high learning rate push the weights out of the global minima after reaching it, in which case a scheduler or dynamic decrease of learning rate would be enough to solve the problem.

Not quite sure what you mean by adversarial loss, but if you mean the loss function used by the generator and discriminator, it's the mean square error/quadratic loss function. The drop in quality seems to follow the pattern that pixels in the images produced later on get gradually blacker until all pixels are either completely black or white instead of also grey which produces worse images and doesn't get better. I'll try using a scheduler for the learning rate although the window of cycles in which good quality images are produced is quite slim so I'm not so sure of this working.

were you ever able to get it working? I am having the same issue where the generated images get better after a while, but just keep getting worse and worse. I monitored the gradient coming into the generator for the picture of "3": This was at the start of training. However, as training progressed, the generator learnt from this gradient and was about to make better images when the gradient just started getting worse: [](htt


text:
How do sine and cosine transforms help in extracting frequencies in time series forecasting models?
I'm trying to learn how time series forecasting models work and while reading a tutorial off the TensorFlow website I came across these algorithms. I don't quite understand what the article means by "time signals" and how do sine and cosine functions help accomplish them. Can anyone please explain?
Here's a link to the tutorial
The following code was provided along with the caption
"the time in seconds is not a useful model input. Being weather data, it has clear daily and yearly periodicity. There are many ways you could deal with periodicity.
You can get usable signals by using sine and cosine transforms to clear "Time of day" and "Time of year" signals:"
day = 24*60*60
year = (365.2425)*day

df['Day sin'] = np.sin(timestamp_s * (2 * np.pi / day))
df['Day cos'] = np.cos(timestamp_s * (2 * np.pi / day))
df['Year sin'] = np.sin(timestamp_s * (2 * np.pi / year))
df['Year cos'] = np.cos(timestamp_s * (2 * np.pi / year))

In simple terms, without this transformation the network doesn't know that 2022-01-01 data likely correlates with the January 1st on previous (and future!) years. And actually it is likely to also correlate with January 3rd, and maybe even December 29th.
It is also possible to encode the day of year as float between 0.0 and 1.0 (first day of year and last day of year), and that would also provide this "correlation" information. However it isn't as good, since 2021-12-31 would have a value of 1.0, very far from the next day's 2022-01-01 value of 0.0.
Yet an other option is to treat the day of year as a categorical value between 1 and 365 (except on leap years, which is annoying). This would provide some degree of information, but it would lose the apriori knowledge that the 10th day of the year correlates more with the 11th than 100th.
Splitting this to sine and cosine signals circumvents these issues. (now I'm thinking whether this got too verbose.)
This same logic applies to the time of day, 23:59:59 should be very correlated with 00:00:00.

Sorry for my weak English. Your are using neural network to forecast times series which often have irregular fluctuations.
Stock values are volatile and have changing frequency. Applying periodic encodings to original data makes it easier to capture frequency information.
Read this paper to understand why it is essential for the neural network to know about frequency of data and how sinusoidal encoding could make it easier for the neural network to predict highly varying patterns inside data.
https://bmild.github.io/fourfeat/
Or you could use periodic activation functions without applying sinusoidal encoding. Networks with non-periodic activation fail to predict time series.
https://arxiv.org/pdf/2006.08195.pdf
Deep networks are biased towards low frequency functions.

Thank you for the papers, I'll check them out!


text:
Proper way to count environment steps / frames in distributed RL architecture for algorithms like CLEAR or LASER => modified impala with replay
In classical - on-policy - vtrace/Impala algorithm env_steps are incremented every training iteration
like this : env_steps += size_of_batch * unroll_length
This works since every trajectory generated by workers is used for training only once in classical Impala.
However when we add replay_buffer to vtrace (like in CLEAR or LASER) we also start to learn from off-policy data therefore some trajectories (from replay) could be sampled more than once.
My question is - does the env_step_counting stay the same even when some trajectories from batch are from replay and some directly from workers (on-policy) ?
Or do i count steps only on those trajectories that are on-policy => not from replay
like : env_steps += len(batch["on_policy_samples"]) * unroll_length
And if so what happens when i decide to use only samples from replay to train - how do i count env_steps then ? Using some kind of flag to indicate if specific trajectory was sampled already and leave it from counting ?
At the end of the day what i need to know is that should i count all steps of trajectories generated by workers (true environment steps) OR all steps of trajectories processed during training (training steps ?). - I need to know this in order to be able to compare results with other implementations.
Have been digging through the research papers for some time now and i haven`t been able to find what is the proper way to do it. Any help or advice would be much appreciated.

I have decided to to collect both metrics - training_steps & environment_steps. In the end both could be used as performance metrics in one way or another. But i am still unsure which one of these were used in the deepmind science papers.


text:
Can anyone please explain TFLite quantization part found in Netron neural network viewer?
I was checking tflite file in Netron. There I found the quantization formula in Netron as below:
quantization: 0.007709330413490534 * (q + 3)

I know the quantization formula is: quantized_value = (r/S + Z) Where, r = input, S = scale and Z = zero-point
I have used quantization-aware training before converting the model to the TFLite model. I have uploaded the TFlite Model file to Netron App website where I have found quantization as below image. Can anyone explain What do the q and 3 mean in Netron quantization?

Hello. Can you 1. put your specific question in the title, and 2. provide the link to the source where you took that screenshot from?

I have added the file and edited the title.

Thanks. You may also want to format the equations with latex/mathjax to improve the readability.

I have found the answer to this doubt I had. Here, 0.007709330413490534 = 1 / S, q = input, Z = 3. Basically, this is the formula to quantize the input value. If you pull out 1/S then it becomes clear. Here, is an article related to this topic.

are you sure about 0.007709330413490534 = 1 / S ? It means S = 1 / 0.007709330413490534 ? I don't understand it

The quantization formula is real_value = scale * (int_value - zero_point).

Scale is calculated as (float_max - float_min) / (int_max - int_min).
Zero point is calculated as int_min - (float_min / scale).

float_max is the max value in the float number line.
float_min is the min value in the float number line.
int_min is the min value in the integer number line.
int_max is the max value in the integer number line.

quantization: 0.007709330413490534 * (q + 3) means integer value q is mapped to float value using the scale 0.007709330413490534 and zero point -3.
The zero point is the offset that shifts the integer value and scale is the factor that scales this integer value to a float value. Confused? Please check this answer which explains the intuition.
If you have any doubts, please feel free to ask.
Cheers,
Thomas.

As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.

The text right next to the formula provided in netron is actually quite misleading, as it's actually a dequantization equation. The correct formula for calculating the dequantized value is:
$$ \mathbf{d} = \text{scale} \cdot (\mathbf{q} - \text{zeropoint}) $$
Where:
$$ \mathbf{d} - \text{dequantized value} $$
$$ \mathbf{q} - \text{quantized value} $$
Given your example, one can deduce that:
$$  \text{scale} \approx 0.0077 $$
$$ \text{zeropoint} = -3 $$
If you actually want to quantize the input value:
$$ \mathbf{q} =  \frac{\mathbf{d}}{\text{scale}} + \text{zeropoint}  $$


text:
How do transformers compare to CNNs in terms of compute budget (and computing time) during inference?
Transformers are data and GPU hungry during training. Is this also true at inference time? How do transformers compare to feedforward CNNs e.g., during bounding box generation at inference time? I haven't found a good comparison of computing time and computational resources.


text:
Can any GAN's utilize labels in their datasets while they are training?
It seems to me that the Generative Adversarial Networks have a practical issue when trying to reproduce some of their output images
For example, as you can see https://www.youtube.com/watch?v=oIzwe_MOeQI&t=1057s seems to generate shape and background changes as it changes rotations.
Can any of the StyleGANs be convinced to take an image of a car, along with some sort of key that says how it is rotated then consider that when the trained model is generating?
Obviously, it might be a substantial effort to label the data, but I'm curious if it's been done in a well-published fashion.

The idea of taking an additional label as input sounds like a conditional generative adversarial network (cGAN). Within a conditional GAN both the discriminator and the generator are conditioned on extra information provided through a label. The idea was first introduced in this paper.
This image from this paper nicely highlights how a conditional GAN differs from a standard GAN:

Such an architecture would likely also be suitable to be conditioned on image rotations.


text:
What is the most important predecessor of the transformer model?
I'm wondering what the origins of the transformer as proposed in Attention Is All You Need are. The paper itself provides some interesting pointers to the literature on self-attention such as:

A Decomposable Attention Model for Natural Language Inference
A STRUCTURED SELF-ATTENTIVE SENTENCE EMBEDDING
NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE

It seems like using attentional mechanisms was widespread (at least in combination with recurrent networks) and 'A Decomposable Attention Model for Natural Language Inference' paper from 2016 already conjectured that scaling attention might be feasible.
Is it from this prior work 'only' an engineering leap? Or what additional papers at the time likely influenced the architecture?

An influential predecessor paper is indeed the work on NEURAL MACHINE TRANSLATION
BY JOINTLY LEARNING TO ALIGN AND TRANSLATE. The paper outlines an attentional mechanism that is similar to the computation of the actual attentional weights in the transformer paper. See the paper and picture below for details.

When the authors of the paper visualize their attention weights they obtain the following results. The inputs into the attention computation are not called queries and keys yet but are conceptually similar.

See this video for a detailed explanation.


text:
Is there a way to use AI to compare thousands of files and detect the ones containing "unusual" content?
Is there a way to use python and AI to compare thousands of files and detect the ones containing "unusual" content?
Those files are supposed to have "homogeneous" configuration types with different parameters, but sometimes some of them might contain "illegal" types of configurations. The issue is that there are too many ways to do homogenous and illegal configurations, so it is not possible to statically define them.
Here is one of too many bloc of configuration considered "correct": (the meaning doesn't matter)
vlan XXXX
  name  XXX_NAME

interface vxlan 1
  vxlan vlan XXX vni YYYY

router bgp YYYY
   vlan XXXX
      rd A.B.C.D:YYYY
      route-target both YYYY:YYYY
      redistribute learned

Thousands of files will contains such bloc of configuration (with different parameters, xxxx, yyyy...) But some might contain something else, example
Interface vlan AAAA
  ip address B.B.B.B/Z

Because it is in only a couple of files ("rare") it is more likely that it is not authorized configuration.
I could use simple regular expression to match what is considered legal or illegal, but there is too many of such blocs.
I am wondering if AI could help with this issue?

It may be a good idea to provide an example of a simple "correct configuration file" and an "usual/incorrect configuration file" to have a better idea of the problem. Do you also have many examples of these configuration files, or would you need to create them artificially or collect them manually?

What makes an illegal configuration illegal? How do you detect that?

Hey, what I call "illegal" is lines different from the most frequent ones (correct).
I'll complete the question with an example.

Here is a quick idea: first calculate the count of how many times each word occurs in these documents (I don't know whether to lowercase them or not, do interface and Interface mean different things?), and sort them in the descending order of occurrence. Most frequent words can be called "keywords" of your configurations (such as vlan), or maybe frequently occurring ip addresses. It is up to you whether to include ips to this list or not.
Then write rules to replace "arbitrary" data (numbers, ips, ...) with a common tag (for example <NUMBER>, <IP>, ...). You should replace all non-keyword words (aka. tokens) with these tags. Use <?> for tokens you don't identify with a regex, or leave them as-is.
Now we can calculate statistics of these token n-grams, and identify those which occur very rarely. This step will require some thingking and tuning, for example how to handle newlines (or just ignore them?), how long n-grams to use, how rare is too rare etc.
This is more of a heuristic than an AI, but it works without any training data. Well, there is a human in the loop so at least it is an Intelligent solution.

Nice idea @NikoNyrth.
I'll give it a shot.


text:
Are there any other language models besides GPT-3 that can be used to create chatbots with a specific identity and environment?
I've recently been looking into language models but ran into a small question out of curiosity. For some language models like GPT-3, it is possible to generate dialogue, with the model basing its answers on the initial input (which can be anything). This makes it possible to make a "bot" which a user can chat with, that identifies itself a certain way (ex. gender and job), has certain mannerisms (ex. polite or angry) and knows things about a custom world (ex. The bot is a cashier in a certain store, located in a certain city).
Newer and better models for open-domain dialogue such as Facebook Blenderbot do not seem to have this possibility, it chats with the user but does not have to setup a certain context. Are there any other language models that can do this, and might perform better than GPT-3?

You may want to look at Chirpy Cardinal from Stanford. It doesn't provide mannerisms out of the box but the response generators can be configured to give each instance its own character.
https://stanfordnlp.github.io/chirpycardinal/


text:
Why instance segmentation architectures using reconstruction masks but not regression?
I'm wondering why many model architectures use binary mask reconstruction for segmentational CNNs, and not regression of mask polygon coordinates? Many object detectors use regression to find coordinates of bounding boxes.

If I understand correctly, the reason is probably that the segmentation polygons have various shapes and complexities. You don't know how many points you need per polygon. Bounding boxes however are always defined by 4 coordinates.

@Chillston This is the correct answer, and should be posted as an answer so it can be upvoted and the OP can accept it.

You are right, thank you - I put it as an answer

The reason is probably that the segmentation polygons have various shapes and complexities. You don't know how many points you need per polygon so defining a proper output that specifies the polygons is not straight forward. In contrast, bounding boxes are always defined by 4 coordinates (2 coordinates for lower left and upper right corner are already enough).


text:
combine two features in dataset?
I have  a data set containing the number of security gaps and the level of that gap for a specific website.
Now suppose I have 2 features in this data set, the first feature is the number of a specific security gap and the second feature is the risk of this gap for a specific website.
How can I combine these two features into one?
What is the best way to apply feature engineering to these features?
Thanks

Can you elaborate on the purpose of the features, the task, and what you've already thought of or tried? Without knowing the goal, the features themselves don't say much.

the features are the number of gaps and the risk level of each gap, the output is the security level of website

So basically you have three values per security gap:

The type of gap (i.e. its label)
The risk of that gap for that specific website. As I understand it this value is different for each website, even if the type of the security gap is the same.
The number of occurrences of the gap

One reasonable way to combine the features is to make a feature vector where the indices are the type of the gap, and the values are the risk of the gap multiplied with the occurences. However, this would mean information loss, due to a gap with risk 1 occuring 3 times being identical to a gap with risk 3 occuring once.
A different way to combine the features is just to make a feature vector that consists of the gap risk values and the gap occurences appended (for a feature vector with length of 2 * num(type_of_gaps). Assuming a model with a fully connected structure, the pattern between the connected values may be determined by the model itself during learning.
A third way would be to make a feature vector filled with tuples, which is not really feature engineering as much as it is restructuring the data into a manageable data type. However, you can combine this, when flattened, with an algorithm using a 1D convolution layer with stride kernel 2 and stride 2 in order to process the tuples together, instead of all the values separately.

If this answered your question please be so kind as to accept and/or upvote it.

I am so sorry, your solution is helped me a lot


text:
Why do the training and validation loss curves diverge?
I was training a CNN model on TensorFlow. After a while I came back and saw this loss curve:

The green curve is training loss and the gray one is validation loss. I know that before epoch 394 the model in heavily overfitted, but I have no idea what happened after that.
Also, this is accuracy curves if it helps:

I'm using categorical cross-entropy and this is the model I am using:

and here is link to PhysioNet's challenge which I am working on: https://physionet.org/content/challenge-2017/1.0.0/

Maybe you should provide a little bit of context, i.e. which task you are trying to solve with CNNs (i.e. which dataset), which loss function you are using, maybe you should do plot_model(your_model) and report the architecture of your model.

I'm trying to classify ECG signals in PhysioNet 2017 challenge. My output is four classes and I'm using categorical cross-entropy as loss function.

Please, edit your post to include these details (maybe also a link to the challenge).

I've edited the question.

Why are you using a CNN to classify ECG signals? Aren't ECG signals just numerical time series?

Yes but I'm not using them as input to my model, I'm using some time-frequency features.

Ok. Please, edit your post to clarify that your data is ECG but you're actually pre-processing it first and feeding actually time-frequency features, if I understood correctly. I don't know if this will really be useful to solve your problem, but it might. Anyway, why did you use convolutional layers?

Because frequency-time features are images, and as you know convs are useful in image classification. By the way,I didn't mentioned that there's an article I'm trying to implement and this is the network used in article.

Follow this answer from StackOverflow, I think your problem related to the second cases where your loss gets Nan value.
Maybe you should try to use the larger datatype (for example float 16 -> float 32)

As I'm using tensorflow's built-in categorical cross-entropy, I think tf's team knows they should take good care of NaN, values. By the way, can regularization help in this situation or should I put weight constraints?

We have to dig deeper into TF code to get the final conclusion. For the second question, I think yes, regularization is to solve the overfitting problem, here, before epoch 394, your validation accuracy is stable so I can say your model get overfit problem

Thanks very much, I'll try this and update the question!

@SepehrGolestanian So, why did you accept this answer? What was the problem in the end?

@nbro, you look like the admin of this community, I saw you carefully edit nearly all the post and ask in very details :smile:

@Toby I am a moderator. I try to ask for clarification and details to make this site a better and more reliable place.

@nbro I see, glad this community has someone like you

@Toby exactly! And I'm glad I found this community that has people like him in it!

@SepehrGolestanian but back to his question, can my answer solve your problem?

@Toby You mean the NaN thing or overfitting problem?

I don't know, which one solves your problem :))

You have to keep the training loss value constant. Beyond that, your teaching technique or network model is not suitable for the job. I recommend that you are using MLP.


text:
Is Reinforcement Learning with only feedback on a single action possible?
Consider the following case:
A reinforcement based web-crawler where:

State = current page + 1 out-link (reduced to features of some sort)
Action = Whether to visit that out-link or not (n_actions = 2). This means action = 1 causes the link to be added to the crawlers queue, and action = 0 causes the link to be discarded.
Reward = Score determined by the page reached by visiting the out-link
Next state = the next possible link to consider (taken from the scraper prio queue). This is seperate of whether the links in the currently visited page are added to the queue or discarded.

Can this state-action-reward space be used in a RL implementation, considering we can not calculate the reward (or penalty) for bad pages, since action = 0 never leads to a new page visit where a reward can be calculated. We will thus only ever have feedback and update action = 1.
Assuming some policy optimization algorithm (such as PPO), can we teach an agent to perform this task, or does it explicitly need feedback on both actions?
My initial assumption was that to train a binary action space, we either need feedback on both actions that is positive, or we need feedback on either action that is both positive and negative. Evening out the probability distribution over the action space would then take care of "learning" the second action. Is that correct?
Why is the problem defined like this:
A different way of defining RL web crawling is by taking the current page as state and considering the list of out-links as the action space, picking a single page to visit. The way I understand policy methods is that a single state is mapped to a single action, meaning that if such a problem definition is used, the agent will pick a single action (link) to take in each page, so only linear paths through the web-content are considered. The problem definition stated above allows the agent to consider all links at all times.
The goal:
To find as many relevant pages (scored on, for example, summing hits from a countvectorizer with target terms), while skipping irrelevant links.

Your title mismatches the body - there are clearly two actions. Your description of action 0 says "causes the link to be discarded" - what does that look like for the state? You also claim that "action = 0 never leads to a new state", so does the "discarded" link get put back on the state, and action = 0 mean "wait a short while" or "end episode" or is there something else? It is worth explaining the goal of this web crawler, to help someone understand why it might need to choose between visiting or not visiting a link, since this will help decide reward structure.

Fair points. I'll rephrase with more info when I have the time.

@NeilSlater updated the title and question


text:
How to convert color information to 1D feature vector?
We are making a classification model that takes a clip of a movie as an input and predicts who the director is. Roughly speaking, it will be a model that understands film directors' unique style.
We are going to extract 5 features from a movie: a visual-feature vector from ResNet pretrained on ImageNet, an audio-feature vector from an audio model, shot type of a frame (one-hot encoding), emotion detection, and a color scheme of a frame. In the end, we are going to concatenate all these feature vectors and give it as a input for our classification model.
We find a tool that can extract color scheme(or palette) of an image as below. It both has information about colors and their proportion. However, I can't think a smart way to convert this information into 1-d vector form. Any ideas?
Of course I know the ResNet will get information about colors but the importance of color will be degraded in ResNet. I think the color is very important feature in defining a director's style and thus I want to use a color feature separately.

My 2 suggestions would be to:

Sum the hex coding of the color multiplied by the prevalence. For example [80, 80, 80] (grey) is used 7% of the time so color_features += [80, 80, 80] * 0.07.
You determine a preset number of color bins (maximally distinct colors used as protoypes for the bins). Bin the colors based on distance, and add the prevalence of all colors in that bin together to a float value.

Example of option 2:
# colors is a list of (color, prevalence) tuples.
colors = [(DDDDDD, 0.30), (EEEEEE, 0.40), (111111, 0.10), (222222, 0.20)]
prototypes = [FFFFFF, 000000] # White and black
# colors[0] and colors[1] are close to white
# colors[2] and colors[3] are close to black
# For each prototype we add the prevalences together that go with that prototype
features = [0.70, 0,30]

The first option is probably too reductionist, while the second may be too generalized or too complex depending on the number of bins being too low or too high, respectively.
Both options are based on intuition from experience, rather than any literature or empirical evidence.

Would you please elaborate more about the second option? You mean using color histogram with appropriate bin size (e.g. 16/32/64)?
Or do you mean picking up appropriate number of colors (like the right side of above images) and concatenate with its prevalence? For example, [ 80,80,80, 0.3, 255,0,0, 0.5, 0,0,255,0.2] which means grey(80,80,80) is used 30% and red(255,0,0) is used 50% and blue(0,0,255) is used 20%.

I mean you determine a preset number of color bins (maximally distinct colors used as protoypes for the bins). The simplest example would be to do white (FFFFFF) and black (000000) and you do a distance calculation of each color in that analysis tool of which bin prototype its closest to and assign it to that. Then you add the prevalence of the binned colors together into a float value for that bin. I'll add an example to my answer.

I think the tool you found is useful for a human and to get nice visuals but I also think it's totally useless for feature extraction.
If you want to pass explicit information about colors simply concatenate 3 normalized histograms for each color channel. You're guarantee to have always a fixed sized color feature vector (n_bins * 3) and you can't literally pass more information about colors, the histograms contain even more information than a compressed color palette.


text:
What does the complexity equation constitute exactly in “Why Should I Trust You?” LIME paper
I've recently been reading this paper on LIME, an algorithm to interpret ANY machine learning model. I encountered this equation (in red) on page 4 and have just been having a hard time deciphering exactly what it means. I understand that it's a measure of complexity of something - but of what exactly? And what does each symbol in the equation entail and correspond to? What part(s) of the models and instances constitute and contribute to the complexity?

For text classification, we ensure that the explanation is interpretable by letting the interpretable representation be a bag of words, and by setting a limit $K$ on the number of words, i.e. $\color{red}{\Omega(g)=\infty \mathbb{1}\left[\left\|w_{g}\right\|_{0}>K\right]}$. Potentially, $K$ can be adapted to be as big as the user can handle, or we could have different values of $K$ for different instances.

Could anyone help me with it?

Let's brake down each component of the formula, on the left side we have
${\Omega(g)}$
The authors state that they use omega to refer to the set of models too difficult to be tractable (or explained), so g is just a model, or set of weights, omega g is a set of models. Next we have
${\infty \mathbb{1}}$
This is not a common notation, at least as far as I'm aware of, but conceptually they're trying to say there an infinite amount of such models, 1 could refer to the cardinality of the set, but honestly not 100% sure about it. Finally we have:
${\left[\left\|w_{g}\right\|_{0}>K\right]}$
This is the constrain that separate the intractable  models from the tractable ones. K refers to the number of words allowed in the bag-of-words model used to convert words into numbers. If you're not familiar with that, you can simply think of it as one hot encoding. So if you have a vocabulary of 30k words, every word vector will be a one dimensional vector of length 30k, with a single 1 and all rest zeros. That zero norm might looks weird, but it simply refers to the length of the vectors. So to summarize:
${\Omega(g)=\infty \mathbb{1}\left[\left\|w_{g}\right\|_{0}>K\right]}$
Means: "The infinite set of intractable models, made of weights vectors with length larger than a predefined value K".
PS: if you're interested in models interpretability, you might want to check Integrated Gradients, the authors created a fantastic repo that implement many algorithms for pytorch.


text:
Can AUCROC curves be used to evaluate forecasting results for time series data?
I have built a time series forecasting model based on RNN.
Can I calculate AUCROC on the test set and the actual predicted values?
Example:
true_value = [1, 3, 23, 5, 8, ...]
model_predicted_value = [2, 4, 18, 6, 11, ...]

If it is possible how should I calculate the AUCROC curve?

Is your task regression or classification?

That is regression task

Then you cannot use AUROC, which is a classification metric. Please do not accept the incorrect answer that was posted, as it is misleading other people!

AUROC and ROC curves are metrics for classification. Since you seem to have a regression tasks, it does not make any sense to use a classification metric, it simply would give you incorrect results since they do not measure what you actually want.
So do not use AUROC for regression tasks, only for classification tasks.


text:
Which calculation to use for GRU
Im doing trying to implement GRU in my own Neural Network Library but when I did some research i stumbled on some inconsistencies.
When calculating a cell there are as many legitimate resources which state that $\mathbf{h}_t = \mathbf{h}_{t-1}\mathbf{z}_t+\mathbf{r}_t(1-\mathbf{z}_t)$ (e.g. https://arxiv.org/pdf/1803.01686.pdf, https://d2l.ai/chapter_recurrent-modern/gru.html) as there are which state $\mathbf{h}_t = \mathbf{h}_{t-1}(1 - \mathbf{z}_t)+\mathbf{r}_t\mathbf{z}_t$. (e.g. https://arxiv.org/pdf/2107.02248.pdf, http://dprogrammer.org/rnn-lstm-gru) When looking at diagrams there are also inconsistencies between the formulas and the diagrams themself. Also the diagrams differ.
Can someone tell me which one it is and why?

Firstly, I am assuming $r$ is the update to the hidden state. I will call it $\tilde{\mathbf{h}}_t$, because this is how it's called in the paper by Cho et al. that introduced the GRU (where $\mathbf{r}_t$ is actually a different vector).
Both formulas do exactly the same thing. The only difference is how you would interpret the role of the vector $\mathbf{z}_t \in (0, 1)^d$. Importantly, because $\mathbf{z}_t$ is sigmoid-activated, it's values are between $0$ and $1$. Thus, you could say that $\mathbf{z}_t$ acts like a continuous switch that controls how much of $\mathbf{h}_{t-1}$ and $\tilde{\mathbf{h}}_t$ ends up in the updated hidden state $\mathbf{h}_t$. This is also called a gate.
In the first formula $\mathbf{h}_t = \mathbf{h}_{t-1}\mathbf{z}_t+\tilde{\mathbf{h}}_t(1-\mathbf{z}_t)$
you could say that $\mathbf{z}_t$ acts like a keep-gate that tells you how much of the old information in the hidden state is preserved. It's negative ($1  - \mathbf{z}_t$), tells you how much the new information $\tilde{\mathbf{h}}_t$ contributes.
In the second formula: $\mathbf{h}_t = \mathbf{h}_{t-1}(1 - \mathbf{z}_t)+\tilde{\mathbf{h}}_t\mathbf{z}_t$
the role of $\mathbf{z}_t$ is inverted and could now be interpreted as an update-gate, controlling how much of old information is replaced by the update $\tilde{\mathbf{h}}_t$.
From an optimization perspective, it doesn't matter whether you optimize the negative or the positive version (i.e. the keep or the update version) of $\mathbf{z}_t$. Therefore, both equations are valid and work equally well.
The first formula is actually used in the original paper by Cho et al., where it is called an update gate (I would argue that z, in this case, acts rather like a keep-gate, but in the end its negative controls the portion of the update, so I guess you can also call it that).

Oh okay that makes sense. Thank you for this really good explanation.


text:
What Past Approaches is the "Taylor Swift" Paper Referring To?
(My previous question regarded which theorem of probability was equation (2) referring to).
This paper mentions that temporal forecasting is meant to solve an integral equation as denoted by (2) (found on page 3 under the section “3. PDE based Forecasting”), and that common auto-regressive approaches in the past have tried to approximate the integral equation.
Where can I find information regarding these approaches? Unfortunately, the paper does not cite any of this, and the citations in the "Related Work" section seem to not make mention of the integral equation.

can you write the equation here, and please elaborate or write in more details the equation (2)? i just briefly read it and i can't find a clue what's the dots between the integral stands for.

Yeah me neither, hence why I’m asking the question. The paper does not make it clear at all.

My best guess is that the dots represent the integral signs in between, and the other set of dots represent the conditional probabilities in between. So basically the integral sign sequence and the probability sequence.

oh, I thought you just don't understand where it came from or how it was formulated on question 1. question 1 and 2 is related, but I think question 3 can be a new separate question here.

Thanks for leaving only one question. But can you please put your specific question in the title rather than writing "Question Regarding Early Approaches to Temporal Forecasting"?

It's the law of total probability, combined with the definition of conditional probability.
\begin{align*}
p(x_n) = \int_{x_{n-1}} p(x_n, x_{n-1}) = \int_{x_{n-1}} p(x_n \, | \, x_{n-1})p(x_{n-1})
\end{align*}
First equality uses the law of total probability, second uses definition of conditional probability. The only difference in the paper, is that everything is additionally conditioned on $\mathcal{X}_t$. (also, they are doing this for some $\tau$ steps, not just once.)

So, what would the bounds of the integral be? Negative infinity to infinity? That seems to be the only answer that theoretically makes sense, but the paper implies that the integrals should be computable (but it should be not just infeasible as said, but impossible to begin with).


text:
Knowledge representation and reasoning(KRR) over a Image scene: Neurosymbolic AI
What are the ways and SOTA in domain of knowledge representation and reasoning over scene. Suppose there are 3 objects in the scene and which objects needs to be picked first among them is governed by 'Rules' written in text form, like Object which are square in shape will be picked first followed by Triangular shape. So basically priority queue or list of object selection needs to be generated based on rules. Or if there is any other way to handle this case?
I believe first way is to do object detection over the scene to get the different object from scene and then we have to represent 'Rules' in form of external knowledge, so that priority queue or list can be generated. However rules in textual form are non-differentiable, so how we can integrate rules in neural network training and what are the ways to represent the knowledge for such kind of priority queue generation over the image scene.
Paper 'Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding' tries to address that kind of problem but I feel its more like a general Question answering system then KRR system

Looks like ai.stackexchange.com/q/32754/2444 is a similar kind of thread, but there is no useful comments over that, so opening the thread.


text:
How to get GPT-3 to translate a specific word in a sentence?
I just gave GPT-3 the following prompt (in the playground, using text-davinci-001 with default settings):
What's the German word for "can" in the sentence "The man removes the can."?

The word "can" in this sentence is obviously a noun and not a verb. "the can" is the metal container used to sell coke or beer, not the common verb "can" which means "be able to".
GPT-3 obviously knows this, otherwise it could not translate the sentence correctly in many languages (which it indeed can).
However, to my prompt, it answered:
The word for "can" in the sentence "The man removes the can." is "kann."

("kann" is German for the verb "can" in the sense "be able to")
Did I ask it in a wrong way or is GPT-3 unable to answer questions like this one?

GPT-3 obviously knows this

GPT-3 doesn't "know" things in the sense that it has learned specific translations, maths or science etc.
GPT-3 is not a knowledge database or question answering system, but it can behave like one if prompted carefully. Sometimes it is hard to get specific responses. This appears to be one of those cases.

Did I ask it in a wrong way or is GPT-3 unable to answer questions like this one?

GPT-3 doesn't really "answer questions", it completes text, and probablisitically fits that text to match the prompts. The fact that it gets a lot of Q&A, maths or translation-based prompting correct is an interesting side effect of the strong language model. It is not clear what the gaps are in it capabilities to perform secondary tasks like this, but very reasonable to expect some.
In general, you should not expect GPT-3 to automatically pick up on questions with answers, and assume it is answering the question as written. Some reasonable questions might look like the start of a novel or newspaper article to it, or in your case it might be acting as if sentences are only loosely related, like a list of different translations. To reduce the flexibility in possible completions, it is normal to set some context in the first paragraph - a kind of meta-explanation of what the rest of the text is doing.
I managed to get it to work like this:

The following is a translation, with individual words explained.
English: The man removes the can.
German for "can" is "Dose"

My prompts are in bold. I am sure plenty of variations of what I wrote will work for this specific case. I don't think you will find a 100% reliable prompt for arbitrary words from any sentences.


text:
Why do we use "true labels" that are based on the output of our network in Deep Q-Learning?
In the original DQN paper, the $\ell_2$ loss is taken over the distance between our network output, $\hat{q}(s_j,a_j,w)$ and the labels $y_j=r_j+\gamma \cdot \max\limits_{a'} \hat{q}(s_{j+1},a',w^-)$, where $\hat{q}(s_{j+1},a',w^-)$ is our network with static weights $w^-$, that are updated to be $w^-=w$ every $C$ steps.
This is troubling to me, as $y_j$ aren't really "true" labels as we know from supervised learning, so why should I even think that this loss updates the weights such that the output policy is something meaningful?
It seems as if my network could output some arbitrary $\hat{q}$ and with respect to this $\hat{q}$, I will try to minimize a loss, But when $\hat{q}$ isn't "optimal" per se, It is not clear that we can converge to an optimal policy.

The labels in DQN, and in Q-learning in general, are not "true" in the sense that they represent optimal action value functions. Instead they represent approximate action values of a current target policy.
The target policy changes every C time steps, when the network with static weights is updated. This update will include both corrections to the action value approximations, and changes to which actions are considered optimal.
The reason this converges towards an optimal action value function is related to the policy improvement theorem. With function approximation, as in DQN, the convergence is not guaranteed, but the process is stll based on the same idea. In summary it is a two-step repeated feedback process:

learn value function of a current policy
update the policy to select actions with maximum values

What this means for the TD target "labels" in DQN is:

They are not ground truth for the optimal action value function, until after the algorithm has converged.

They are biased, initially almost completely arbitrarily by however the target network has been initialised, and from then on due to a slowly-reducing impact from that initial bias and from lagging behind collected data.

They are non-stationary. This means an online learning model class is required (neural networks are fine). It is also the reason why many Deep RL algorithms can suffer from catastrophic forgetting.

When using experience replay, the TD target labels should be recalculated each time they are used.

Thanks. Isn't the convergence property relates to the fact that we have calculated our value function based on a bellman backup process, which is a contraction map? here, from my understanding, we use a CNN which is not really related to bellman backup

@HadarSharvit In a DQN, the CNN is being used within the Bellman backup process, in lieu of the Q table.


text:
How to calculate a meaningful distance between multidimensional tensors
TLDR: given two tensors $t_1$ and $t_2$, both with shape $(c,h,w),$ how shall the distance between them be measured?

More Info: I'm working on a project in which I'm trying to distinguish between an anomalous sample (specifically from MNIST) and a "regular" sample (specifically from CIFAR10). The solution I chose is to consider the feature maps that are given by ResNet and use kNN. More specifically:

I embed the entire CIFAR10_TRAIN data to achieve a dataset that consists of activations with dimension $(N,c,h,w)$ where $N$ is the size of CIFAR_TRAIN
I embed $2$ new test samples $t_C$ and $t_M$ from CIFAR10_TEST and MNIST_TEST respectively (both with shape $(c,h,w)$), same as I did with the training data.
(!) I find the k-Nearest-Neighbours of $t_C$ and $t_M$ w.r.t the embedding of the training data
I calculate the mean distance between the $k$ neighbors
Given some predefined threshold, I classify $t_C$ and $t_M$ as regular or anomalous, hoping that the distance for $t_M$ would be higher, as it represents O.O.D sample.

Notice that in (!) I need some distance measure, but this is not trivial as these are tensors, not vectors.

What I've Tried: a trivial solution is to flatten the tensor to have shape $(c\cdot h\cdot w)$ and then use basic $\ell_2$, but the results turned out pretty bad. (could not distinguish regular vs anomalous in this case). Hence: Is there a better way of measuring this distance?

You could try an earth mover distance in 2d  or 3d over the image? For example you could follow this example, but call it sequentially. The idea would be something like the following (untested and written on my cell phone):
def cumsum_3d(a):
    a = torch.cumsum(a, -1)
    a = torch.cumsum(a, -2)
    a = torch.cumsum(a, -3)
    return a

def norm_3d(a):
    return a / torch.sum(a, dim=(-1,-2,-3), keepdim=True)

def emd_3d(a, b):
    a = norm_3d(a)
    b = norm_3d(b)
    return torch.mean(torch.square(cumsum_3d(a) - cumsum_3d(b)), dim=(-1,-2,-3))

This should also work with batched data. I would also try normalizing the images first (so they each sum to 1) unless you want to account for changes in intensity.

can this function be used within torch.cdist, to account for pairwise distances of every pair?

It looks like torch.cdist only supports different values of p for the L_p distance. It doesn’t look like it supports applying a function to all pairs. Sounds like you want to make a kernel matrix? You could use something like https://pytorch.org/docs/stable/generated/torch.combinations.html and then stack the 0,1 elements from the tuples into two tensors, then run this?

Or better you could flatten after doing the 3d cumsum in the example above. Then you could use cdist with p=2 to replace the subtract and square steps. You would give the function your list of per-sample flattened cumsum tensors (samples x flat) twice. Then that would be all pairwise distances with something kind of like an EMD.

Wow, this has worked tremendously well. I highly appreciate your response.

Nice! Glad I could help.

You could do the same process using torch.cdist and the embedding output (before classification layer) of a ResNET or whatever pretrained/frozen image model. You might want to divide each embedded vector by its norm first though so you have a cosine similarity rather than a pure l2 if you do that though. Embeddings from a good image model should be better than emd, but I bet emd is a pretty good baseline.


text:
can I add to a language model a prompt with output example?
I want to finetune GPT2 to extract relevant data from a given text. So for (a trivial) example, given the text "the car was manufactured in X, can reach Y km/h, and has Z horse powers", my desired output would be manufacturer: X, max speed:Y, horsepowers:Z.
I don't have a lot of labeled data, so I thought it would be reasonable to take every training sample and add to it a prefix than contains an actual example. That is - Instead of providing the model with the text

INPUT: a well-known brand, the new X, can reach Y km/h, and has Z horsepower
OUTPUT:

and expect the model to understand how to fill in the details, I would provide a longer prompt that contains an actual example like

INPUT: the new BMW can reach up to 200 kmh. Even though the previous model disappointed the users, the brand-new one rocks a thousand engine horsepower
OUTPUT: Manufacturer: BMW, Max Speed: 200, Horse Powers: thousand
INPUT: though is mostly considered an outdated version of the cls200, the new Mercedes has the capabilities of reaching up to 100kmh in turns and 300kmh overall
OUTPUT:

Is this considered a common way of engineering the prompt? Do notice that the first provided example is the same for every training sample.

Yes, it is possible to finetune GPT2 to extract relevant data from a given text. For your example, you could add a prompt that looks for a manufacturer, max speed, and horsepower, and then provide an output example that includes that information.
It is possible to add a prefix to every training sample that contains an actual example. This could help the model learn from data that has fewer labels.

INPUT: though is mostly considered an outdated version of the cls 200, the new Mercedes has the capabilities of reaching up to 100kmh in turns and 300kmh overall?
The OUTPUT would be the capabilities of the new Mercedes, including its max speed and ability to handle turns.

Yes, this is considered a common way of engineering the prompt.

Providing the same example for every training sample also help the
model learn from data that has fewer labels.


text:
Detecting object position given the relative position of another object
I know that the title might be redundant but I'm trying to understand if there is way to predict where a specific object will be if I provide a certain object as a reference.
See as an example the image . The red object (which can be provided as a set of coordinates) in always present as a feature. The blue object is the  one to be predicted. In general the red polygon can vary (more elongated, larger or smaller) and the blue object change its position wrt the red one.
I am thinking, if possible, about a system where in a first training phase I provide both the objects with the blue object been the target of my training.
In a test phase I would provide only the red object and let the system try to guess where the blue object will be.
The underlying relation between the two objects relative position is not well understood in general so my idea was trying to understand if out there is something that could generalize and approach this problem.
It might be a conceptual question for the moment, in case more details are needed I would be happy to discuss them.
Thanks!

I always like to think that Theoretically, if there exists some function $f:R \rightarrow B$ that maps the set of points $R$ which represent your reference object to the set of points $B$ which represent your target object, then there should be a network $N_\theta$ that approximates $f$ with arbitrarily small precision. Sadly for us, it may be very difficult to find $N_\theta$, but hey let's try anyway!
In terms of the algorithmic approach, your suggestion seems likely. Another possibility would be to try and formulate a basic CNN that takes in normalized images of the reference object and outputs a $(x,y)$ that will represent the center of your target object. For that, of course, you will have to create data that consists only of images of rotated + scaled version of your initial reference object and labels that are the center points $\{(x_i,y_i)\}$ of the target object.
It might also be reasonable to consider transfer-learning using some pre-trained ImageNet-based network, as those were shown to improve results even for very different datasets (compared to ImageNet)

Thanks for your reply, it's very interesting. Could you give me an example of how you would proceed? to me it is not clear how with an object detection problem a CNN would predict something which is not there. I was considering reinforcement learning just as an example since it is used to "predict" next moves like in games and so on. So I was interested in this concept. Thanks!


text:
Pytorch's Actor-critic implementation seems to be implemented in a Monte-Carlo fashion - why?
In the Actor-Critic example, provided by PyTorch, it seems that the update rule only occurs when the episode ends (like in a Monte-Carlo process). Specifically, in their main function they sample the environment in a loop, saving the log probabilities and state values model.saved_actions.append(SavedAction(m.log_prob(action), state_value)) right after the agent takes an action. Later, when done was reached, the loss term is aggregated and back-propagated.
The thing is - in every literature book I've come across they distinguish that one of the critical points in AC method is that the update rule can be performed after every step in the environment - something along the lines of:

take action $a\sim\pi_\theta(\cdot|s)$ and observe $s',r$
calculate TD error $\delta\gets r+\gamma \hat{V}(s',\theta_V)-\hat{V}(s,\theta_V)$
update $\hat{V}$'s weights $\theta_V\gets\theta_V+\alpha_V  \delta \nabla_{\theta_V}\hat{V}(s,\theta_v)$
update $\pi_\theta$'s weights $\theta\gets\theta+\alpha_\theta I \delta \nabla_{\theta} \ln{\pi_\theta(a|s)}$

what am I missing here?

After some research (thanks, ChatGPT!) it seems that the implementation provided by PyTorch is known as Monte Carlo actor-critic, where the value function is updated using the Monte Carlo returns (i.e., the sum of the rewards received in an episode) rather than the TD error.
One advantage of Monte Carlo actor-critic is that it can be easier to implement and may be more stable in some environments. However, it is less sample efficient than the online version, since it only learns from completed episodes rather than making updates at each step.


text:
Can we use a pre trained Encoder (BERT, XLM ) with a Decoder (GPT, Transformer-XL) to build a Chatbot instead of Language Translation?
I was wondering if the BERT or T5 models can do the task of generating sentences in English. Most of the models I have mentioned are trained to translate from English to German or French. Is it possible that I can use the output of BERT as an input to my Decoder? My theory is that when I already have the trained Embeddings, I do not need to train the Encoder part. I can just add the outputs of sentences to the decoder to generate the sentences.
In place of finding the loss value from the translated version, Can I compute loss on the reply of a sentence?
Can someone point me toward a tutorial where I can use the BERT output for the decoder part? I have a data of conversation with me. I want to build a Chatbot from that data.
I have already implemented LSTM based Sequence2sequence model but it is not providing me satisfactory answer.
After some research, 2 such models are there as T5 and BART which are based on the same idea.
If possible, can someone tell me how can I use BART or T5 to make a conversational bot?

=> I don't have more ideas about BART and T5 Right now. but I had created a chatbot Based on GPT-2 Model-based on Microsoft DialoGPT. Which is fine-tuned on millions of parameters of Reddit. you can fine-tune on your own data using DialoGPT. I had not found a public decoding method. So, We tried to generate diverse responses using simple nucleus sampling at each time step. So, We used greedy nucleus sampling multiple times in parallel to generate multiple candidates responses. We generated 30 such responses for each turn and also used the last 3 turn conversation history as extra context for generator. So, We can generate responses related to the current Context of Conversation.
We used Code reference from here. you need to implement greedy nucleus sampling multiple on this code.
=>Now To Choose the best-matched response from response generated I created a Reranker.I used two sub-component.
=>Component one Counts Cosine similarity between Response and Query using sentence Encoder.
=>In Component Two we cross-entropy error for generating original query from the response.For that I used reverse-generator of DialogueGPT.
We Combined Score of Both Component as Score=norm(comp1Score)+norm(1-comp2Loss). Using this score you can find best response.
For Tutorial My Favourite Hugging Face they have BERT,GPT, GPT-2, XLNET,etc...
and Second Favourite Facebook ParlAI,
try Blender Model also.

But can we use outputs of BERT as the the  K' and 'V values to an attention Decoder in the simplest terms possible to minimize the binary cross entropy?

@HirenNamera Can you please use this style only for code? You can and should use this or this style if you need to emphasize something.

I think it is possible to minimize binary cross entropy. Why not you try using huggingface's pretrainedencoderdecoder module. Using bert-base-uncased as encoder and gpt-2 large as a decoder.


text:
Can back-bone of text-to-image GEN AI models utilised for classification?
With the advent of GEN AI (Stable Diffusion), we are able to create images with text. For eg. If i need to create a dog on beach during sunset; now in background this model needs to first get images of object in the image which is dog, scene which is beach, sunset.
Now when the model knows what are the objects/ scene in the image, it is good to say it can classify the objects/ scenes. I wanted to utilise this feature of model for building classification models. Would appreciate getting leads/code/discussions on this.

Do you have a specific generative model type in mind? Each model type uses the text guidance in different ways. So Stable Diffusion, Midjourney are diffusion based models. But there are also GANs, VAEs and others. Also many of these models split into sub-models that you could potentially use, like CLIP. So knowing which models you have access to could help. Also whether you want to do zero-shot classifying of existing data, or have a training set with specific targets to learn?

The motivation of this question is stable diffusion. And, yes i have a training set with specific targets to learn.

I suggest you use [edit] to put that information into the question. Also a couple examples of what you are hoping to achieve would help (an image and its category). Is there any reason you don't want to go the normal route for this and fine tune a classifier trained on ImageNet (or similar)? Chances are even if you can get something out of the generative models, that it would be inferior to a simpler fine-tuned model.

Already tried the fine-tuning route; I ran out of ideas while trying out fine-tune model architectures. And when you say it may be generative AI model would be inferior - what is the hypothesis behind it, is there some supporting article? Or something that you have experienced that may be of use to me?

It's an educated guess. The basic theory of it is based on common knowledge, that the closer that a model is that you want to adapt is to already performing your task, then the easier it is to adapt, and the better performance you can expect.


text:
Attention with Recurrent Neural Networks
In RNNs, to avoid "forgetting" information encoded by earlier encoders, we can use attention. It's basically a second neural network that tells us how much we should attend at time t on each of the earlier hidden states (from 1 to t - 1).
This is described here:

(image from https://medium.com/swlh/a-simple-overview-of-rnn-lstm-and-attention-mechanism-9e844763d07b)
However, there is something I don't understand. This second neural network will output t values. So there will be t neurons on the last layer. What if my input sentence (in a seq2seq network for example) has only t - 2 words. Or t + 2 words. Then I can't use that same second neural network since its architecture is fixed. How is it dealt?
EDIT: is it that the second neural network takes as input one encoder hidden state and one decoder hidden state, and we get t scores by doing t forward passes (one per each input hidden state, all with the current decoder hidden state)? And then we softmax all those t outputs?

You don't need padding for attention with variable-length inputs. Looking at the formulation in the article:

A learned model ($\mathbf{a}(\cdot)$) encodes the hidden state of each input token directly. This means that if you have a longer/shorter input, you would just run more forward-passes.
During training, $\mathbf{a}(\cdot)$ learns to give higher values for more relevant tokens. It only depends on the hidden-dimensions of $h_j$ and $s_{t-1}$. You might still use masking and padding, this generally is more of a efficiency thing (because GPUs like static data sizes). In fact, you'll probably have to use masking to make sure that your decoder doesn't attend to masking tokens.

Oh I see, so we just do as many forward passes as potential hidden states we want to attend to, each forward pass giving a score and then we apply a softmax!

Yep, that's correct

Two concepts are critical here: masking and padding.
From Tensorflow:

Masking is a way to tell sequence-processing layers that certain
timesteps in an input are missing, and thus should be skipped when
processing the data.
Padding is a special form of masking where the masked steps are at the
start or the end of a sequence. Padding comes from the need to encode
sequence data into contiguous batches: in order to make all sequences
in a batch fit a given standard length, it is necessary to pad or
truncate some sequences.

See the article for more details.
Also see Introduction to Recurrent Neural Networks with Keras and TensorFlow by Aritra Roy Gosthipaty, Devjyoti Chakraborty, and Ritwik Raha. See the section entitled "A Caveat: Masking and Padding".
Prudhviraju Srivatsavaya explains the concepts in reference to Transformers in his article Attention Masks - Explanation.

So does that mean attention has a set length and if we have let's say more input tokens that its set length it would not be able to provide attention to any previous hidden states?


text:
TD Leaf value function update
I'm currently watching a RL course by David Silver and he explains the update of TD Leaf, here is the slide:

He says that if, at the next turn (after we played red and the opponent played blue) instead of the blue node the opponent played node number 1 (because the next minimax search indicated we were wrong at state St because we did not look "far enough"), we then pick the one in the bottom right corner for example (node number 4), we still update the blue node value on the left diagram and not the node next to the blue node (node number 1).
The explanation follows a question from a student and is at this timestep:
https://youtu.be/kZ_AUmFcZtk?t=4346
I'm struggling to understand why we wouldn't update the value of the node next to the blue one (node number 1), since that's the action that ended up being selected.

I think you should indicate on a copy of the diagram which alternative nodes you are referring to. The term "next to" is ambiguous, and so is "the blue node" (there are two shown). I don't know which nodes you mean, nor in which side of diagram you are asking about changes.

I watched the video, but the screen is washed out on it, and it is really unclear which nodes are being discussed between the lecturer and student in that clip, even when looking a clear version of the diagram separately.

I added more figures to help understand what I mean, sorry if I wasn't explaining myself well!

I hope it's clearer now, I ended up just renaming the nodes and then referring to them in the text.

In temporal difference learning, we do not wait to collect the full trajectory before performing backups. That means at the time of performing the backup, we may be "wrong". In fact that is the most common situation for all initial backups (ours and our opponent's).
On the next pass through however, the successor states will have also been updated by the opponent's choice. So we may correctly predict their decision, and get a more accurate backup. That is, providing they were correct in their decision. The opponent could be facing a very similar situation. We don't actually know that the later state information they worked from is a better prediction of who will win - it is more likely to be in general, as winner data backs up from game end states - but it is not guaranteed.
Using a longer "real" trajectory to decide what to backup doesn't fix things - although it may capture actual results, those rely on too many interim poorly-informed choices (unless you have capacity to perform a full tree search), so there is a lot of variance, whlst TD learning swaps that variance for bias. Both the bias (in TD learning) and variance (in e.g. Monte Carlo learning) reduce over time as more data is collected. In practice TD learning often performs better - although for game playing that may depend a lot on the type of game.

Ohh, so the backup is performed at the same time as we are performing the action? Does that mean that for "early actions" in the game, during the first episodes, we basically change the node values randomly since we only get real information on the states close to the only state giving us a reward (the end one, +1 if win, -1 if loss)? (Obviously in Monte-Carlo it's different, I'm talking about TD learning)

@FluidMechanicsPotentialFlows Yes the most basic unaltered TD learning ends up doing a lot of zero updates in the first games, provided the reward system is simply +1 for win -1 for loss.

No problem. The delay is on purpose. There may yet be a better answer for you, so you should wait as long as you want, up to the expiry


text:
GANs inputs normalized and generator only outputs in [-1; 1]
I'm currently coding a GAN on the dataset MNIST.
I'm using the following code to transform my data:
# MNIST Dataset
transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.1307,), (0.3081,))])
# the output of torchvision datasets are PILImage images of range [0, 1] and we
# want data that is centered around 0 with a std of 1 (0.1307 and 0.3081 are the estimated values of the MNIST mean & std)

I will have data centered around 0 with a standard deviation of 1 ((0.1307,), (0.3081,) are the estimated mean & standard deviation on the training dataset)). So that means that there will occasionnaly be values above 1 and below -1 in the real data.
Now, my generator ends up with a tanh activation function:
return torch.tanh(self.fc4(x))  # outputs in[-1; 1]

That means there will never be values above 1 and below -1 in the faked data.
Is it possible that the discriminator picks on this phenomenon? This seems to be the case as its loss goes to 0 really quickly. However this also could also be the case that the discriminator is just "too strong" as I've seen numerous times on stackexchange posts. I however never seen nobody talking about the fact that it could pick on the fact that there are outliers in the "real data" and only pixels between -1 and 1 in the "fake data".
EDIT: the entirety of my code can be found here:
https://github.com/JQuentinMendoza2008/PyTorch_GAN_for_MNIST_Dataset
Any suggestion is welcomed.

Is it possible that the discriminator picks on this phenomenon?
It's not just possible, it's a certainty. The generator should learn to translate an input distribution A to an output distribution B, if distribution B has range $(-\infty, \infty)$ and your generator can output only values in range $(-1, 1)$ the translation between the two simply can't happen.
Move to another normalization, like min max, and replace tanh with sigmoid as final activation function. If your GAN will still have trouble converging, only then you might start investigating other components like the discriminator depth.

is there a way to do the min max normalization easily? I've checked https://stackoverflow.com/questions/68791508/min-max-normalization-of-a-tensor-in-pytorch but it looks complicated. Also is there not a final activation function that looks like tanh (will likely outputs between -1 and 1) but can output values in (−∞,∞)?

you could use no activation at all if you want to output any real value. But I would suggest against it, convergence is still possible but for sure it will become really slow and unstable.

the min max normalization is actually probably the simplest one among normalizations. You just subtract the minimum value of the whole image to all pixels, and divide them by (max value of whole image - min value of whole image). The first step push down the min value to zero, the second step push down the max value to 1, so your guarantee to have values within the range $(0,1)$ (hence the necessity of using sigmoid activation with this normalization)

@EdoardoGuerriero do you have anything where you base this on (that you advice against linear activations due to stability issues)? I'm being curious, because I've had great results with using linear output activations where my output distribution was not constrained.

@Edoardo Guerriero ok! Actually, I just remembered, but aren't the values of the MNIST dataset already between 0 and 1 anyway? Why do people use Normalize in that case (https://github.com/lyeoni/pytorch-mnist-GAN, etc.)

@FluidMechanicsPotentialFlows I guess it's due to a not really good understanding of data preprocessing, combined with the fact many tutorials online use tanh as final activation. But testing both configuration is a quick exercise for only 2 line of code to modify.

@EdoardoGuerriero Yeh of course, I just wanted to know why people were doing it in general. I'm gonna try to not use any activation at all and to use a sigmoid with a min max normalization (but I think that's already been done in the Dataset I'm  using, I'd have to check)

@Chillston I got that experience while working on satellite data super resolution. Admittedly the data where codified in uint16, so the range to learn is quite huge compared to uint8. We tried to not normalized cause we didn't wanted to affect the raw sensor data but we gave up pretty quickly on this potential risk and after normalizing the training was much more easy. Still, the mnist data is quite a simple one so you might be right that this risk is probably not noticeable.

Ah, I was asking about the output activation. You where advising against linear activation, because it can lead to unstable training (I was asking about this comment: "you could use no activation at all if you want to output any real value. But I would suggest against it, convergence is still possible but for sure it will become really slow and unstable."). This is not clear to me, I had good experience with linear output activation so far, but I am very interested to learn more about this.


text:
What is being optimized with WGAN loss? Is the generator maximizing or minimizing the critic value?
I am kind of new to the field of GANs and decided to develop a WGAN. All of the information online seems to be kind of contradicting itself. The more I read, the more I become confused, so I'm hoping y'all can clarify my misunderstanding with WGAN loss.
Critic loss function:

Generator loss function:

g_w are the parameters for the critic and g_θ are the parameters for the generator
From my understanding, the loss functions show that:

The critic wants to minimize its loss. Splitting the loss function up, this means it wants to:

minimize its score on real data
maximize its score on fake data

The generator wants to maximize the critic score on fake data. So it wants to make the data it generates seem more fake to the critic?

Since the critic gives a high score to fake data and a low score to real data, why would the generator want to maximize its score? Wouldn't that mean the generator wants to make its data appear more "fake" to the critic? I would think the generator would want to minimize its loss to make it look more real (since real data has a low score)

I think I understand what's happening with the loss functions now.
Notation:

D = discriminator/critic
G = generator
D(x) - Critic score on real data
D(G(z)) - Critic score on fake data
∇_D - Critic loss gradients
D_p - Critic parameters
∇_G - Generator loss gradients
G_p - Generator parameters

The loss for D essentially is as follows:

In this case, D wants to maximize this loss (as the + indicates in the second line above). This means that D wants to make the difference as large as possible. How can it do this?

If D(x) is as high as possible and D(G(z)) is as low as possible, then the loss will be as high as possible.
D is effectively maximizing its score on real data (D(x))
D is also minimizing its score on fake data (D(G(z))) or maximizing its score on the flipped score on the fake data (-D(G(z)))

So, D wants a high score for real data and a low score for fake data

The loss for G is as follows:

In this case, G wants to minimize its loss which is -D(G(z)). This means that G wants to make the opposite of the discriminator score as low as possible. Let me break this down as it confused me a lot at first.

Since the loss is negative, the score is basically flipped.
Due to the flipped score, if D gives a positive value meaning it thinks the data is real, then the loss will be negative which G wants.
If D gives a negative value meaning it thinks the data is fake, then the loss will be positive which G does not want.

Due to the double negative, this loss basically means G wants to maximize the output of the discriminator on fake data, which makes sense.

With all this in mind, I would expect the graph to look something like the following:

D_real [maximizing D(x)] - The critic wants to produce an output that's as high as possible on real data. So, I would expect this value to be a high positive value.
D_fake [maximizing -D(G(z))] - The critic wants to produce an output that's as low as possible on fake data. So, I would expect this value to be as high as possible. Note, I would expect D(G(z)) to be as low as possible, but due to the negative, I would expect this value to be pretty high.
G [minimizing -D(G(z))] - The generator wants the critic to produce an output that's as high as possible. But, if you look at the loss function, you will notice the generator loss is the exact same as the discriminator loss's second term (the difference is the discriminator is maximizing its term while the generator is minimizing its term). With this in mind, I would expect the value to almost be exactly the same as D_fake (with little variation due to gradient updates).
D - The critic wants to make its loss as high as possible, so I would expect this curve to be much higher than the rest due to it being the sum of two positive values, D_real and D_fake where D_fake is equal to G.
So, D_fake should be fighting to make its loss higher (effectively minimizing D(G(z))) while G should be fighting to make its loss smaller (effectively maximizing D(G(z)))


text:
In the Policy Gradient Theorem proof, why is $d^\pi(s) = \sum_{k=0}^{\infty}\gamma^{k}Pr(s_0 \rightarrow s, k, \pi)$ true?
I was reading the original Policy Gradient Paper. I didn't quiet get the last step of the proof for the policy gradient theorem. The proof given in the paper is below:

I don't understand how the last line follows from the second to last line. That is how is the following true? $d^\pi(s) = \sum_{k=0}^{\infty}\gamma^{k}Pr(s_0 \rightarrow s, k, \pi)$
where $d^\pi$ is the stationary distribution of the MDP with fixed policy $\pi$ and $Pr(s_0 \rightarrow s, k, \pi)$ is probability of transitioning from state $s_0$ to state $s$ in exactly $k$ timesteps.

$\sum_{k=0}^\infty \mathbb{P}(s_0 \rightarrow s, k, \pi)$ is equivalent to the steady state distribution $d^\pi (s)$ if we assume that the MDP is ergodic. You can see this because due to ergodicity, in the limit $k \rightarrow \infty$, the state visitation distribution becomes independent of time (this is loosely speaking the definition of ergodicity) and so this is the state distribution. Intuitively, it is saying that because the MDP is ergodic, independent of whatever we do at the start of the MDP we will eventually reach this 'steady state' distribution.

@DavidIreland I don't quite get it. I would have thought given the MDP is ergodic that $d_\pi(s)=lim_{k \to \infty} Pr(s_0 \rightarrow s, k, \pi)$ as this is the definition of steady state probabilities. Also using the infinite sum, $d_\pi(s_0)$ would be more than 1 because  $Pr(s_0 \rightarrow s_0, 0, \pi) = 1$ and as the MDP is ergodic, there is some probability of self transition back to $s_0$ after some finite number of timesteps, so $d_\pi(s)$ therefore would not be a valid probability distribution right?

Yes you are right about that -- I do believe in the paper they mention it is the improper state distribution, i.e. it has not been normalised. I'm fairly certain they mention it there, or in the Sutton & Barto book. I should note that $d$ is also the discounted state distribution.


text:
PPO advantage estimate - Why does advantage estimate have $r_t+\gamma V(s_{t+1})-V(s_t)$
So I've been looking at this formula for advantage estimate
\begin{equation}
\begin{aligned}
& \hat{A}_t = \delta_t + (\gamma \lambda)\delta_{t+1} + ... +  (\gamma \lambda)^{T-t+1}\delta_{T-1}\\
&\text{where}\ \delta_t = r_t +\gamma V(s_{t+1}) - V(s_t)
\end{aligned}
\end{equation}
where $r_t$ is the reward at time $t$ and $V(s_t)$ is the estimated value of $s_t$ and $\gamma$ is future discount. Why is $\delta_t$ useful? As in why is it useful to calculate reward from time $t$ from the discounted estimated reward from that time onward of $s_t$ and subtract the value estimate of $s_t$
Original PPO Paper

Lets notice, that $\hat{A}=\delta_t$ is a unbiased estimate of $A$ in a sense, that
$$
E_{s_{t+1}}[r_t + \gamma V(s_{t+1}) - V(s_t)] = E_{s_{t+1}}[Q(a_t, s_t) - V(s_t)] = A(a_t, s_t)
$$
Here we abuse the fact, that $V(s)$ is known, but in reality we know only its approximation, so the bias of estimator will be correlated to error of estimation of $V$ by  $V_\theta(s)$. By increasing the trajectory we can minimise the impact of $V_\theta(s_{t+1})$, lets write
\begin{equation}
 \begin{aligned}
&\hat{A}_t^{(1)}:=\delta^V_t& =r_t + \gamma V(s_{t+1}) - V(s_t) \\
&\hat{A}_t^{(2)}:=\delta^V_t + \gamma\delta^V_{t+1}&=r_t + \gamma r_t + \gamma^2V(s_{t+2}) - V(s_t)\\
&\hat{A}_t^{(3)}:=\delta^V_t + \gamma\delta^V_{t+1}+\gamma^2\delta^V_{t+2}&=r_t + \gamma r_t + \gamma^2r_{t+2}+\gamma^3V(s_{t+3}) - V(s_t)\\
&...&\\
&\hat{A}_t^{(\infty)} := \sum_{i=0}^{\infty} \gamma^i\delta_{t+i}^{V}
 \end{aligned}
\end{equation}
The longer the traction, the smaller the term $\gamma^i$ at $V(s_{t+i})$, therefore the approximation of advantage function is less biased.
It is not perfect, since the variance is increasing with longer path, e.g. $\hat{A}^{(1)}_t$ has low variance and high bias, and $\hat{A}^{(\infty)}_t$ has low bias, but high variance. The tradeoff can be introduced by taking some $i < \infty$ and estimating the A by $\hat{A}^{(i)}_t$. But choice of $i$ is not evident. By analogy of generalisation for TD($\lambda$), described in Sutton's book, chapter
12.8 the other way to introduce the trade-off between bias and variance is to take a weighted sum. In practical case of course the infinite trajectories are not accessible. So the trajectories of length T are taken into account, which results exactly in
$$
\hat{A}_t = \sum_{i=0}^{T-(t+1)} (\gamma \lambda)^i \delta^{V_\theta}_{t + i}
$$
Where $\lambda$ introduces the trade off between bias and variance of advantage function estimation.
For further detailed reading I would highly recommend looking at the other article by J. Schulman. It has a section (3. Advantage Function Estimation) on exactly this question.

Is there any reason, why you put the index for the advantage A on the top instead of replacing the step t?

Im not sure I understand exactly what do you mean. Are you referring to $\hat{A}^{(i)}t$ notation? And the question is why I didn't write it as $\hat{A}{(i)}$ ?

Sorry: yes exactly. Because otherwise, given a trajectory, I would need to calculate: $\hat{A}^{(1)}{0}$, $\hat{A}^{(1)}{1}$, $\hat{A}^{(2)}{2}$ and so on... Then I need to repeat the same process again for $\hat{A}^{(2)}{t}$. It shouldn't be: $\hat{A}{0}$, $\hat{A}{1}$, $\hat{A}_{2}$ ... instead?

I think I see where the misunderstanding is coming from! So each of $\hat{A}^{(j)}_i$ is an estimate of advantage function, so you would have to pick one to use in specific case. You would compute only $\hat{A}^{(1)}_t$ for example, for all the $t$ in the trajectory. However there is a bias-variance tradeoff - longer trace gives less bias, but high variance ( The estimate is accurate, but takes a lot of data to converge ) and shorter trace - gives low variance, but high bias ( It converges very fast, but is not very accurate ). So the question is - which length you should choose?

To answer this question $\hat{A}t$ is proposed. If you look at the formula, it uses all the trajectory information ( like $\hat{A}^{(T)}t$ ), so it supposed to have high variance and low bias. However, there is an extra parameter $\lambda$ that added to control that trade-off. Notice, that if $\lambda$ is very small, then $(\gamma \lambda)^i \delta^{V\theta}{t + i}$ becomes small very fast with increasing $i$ and the information from the end of the trajectory is not used! This all to say, that $\hat{A}_t$ is actually a different estimator from $\hat{A}_t^{(i)}$. Hope it helps!

If I understood right (I'm still not really sure), it is exactly the same as $TD(\lambda)$. It is correct? I mean, the same concept, not the same algorithm. That means, if $\lambda = 0$ is one case and if $\lambda = \infty$ then it is another case

It looks to me, that $\lambda \in [0, 1]$, so you can't have $\lambda = \infty$. Yes its the same as $\text{TD}(\lambda)$, but its $\hat{A}(\lambda)$ :) So for different $\lambda$ you will have different estimates of advantage function.

You are right: $\lambda \in [0, 1]$, I mistook meaning for $\lambda = 1$ you get the MC in $\infty$ case. Thank you very much for your patience and explanations!


text:
Likelihood function for Gaussian Discriminant Analsis
Im trying to understand how the likelhood function for gaussian discriminant analysis is derived.  I self studying Murphy's Probabilistic Machine learning, and in it, he states the likelihood function as follows:
$$P(D|\theta) = \prod_{i=1:n} Cat(y^i|\pi) \prod_{c=1:C} N(x^i | \mu_c, \sum_c)^{I(y^i=c)}$$
where $N(x^i | \mu_c, \sum_c)$ is the normally distributed random variable with mean $\mu_c$ which is the mean vector for each feature whose class label is $y^i=c$ and $\sum_c$ is the covariance matrix between the features for the ith example.
Based on my understanding, the way I have been deriving Likelihood is by assuming the following:
We are given the dataset $D = {(x^1, y^1),(x^2, y^2), ..., (x^n, y^n)}$ drawn from unknown distribution $p(X,Y)$.  B/c all pairs are sampled iid, we obtain:
$$p(D) = p((x^1, y^1),(x^2, y^2), ..., (x^n, y^n)) = \prod_{i=1:n} p(x^i, y^i | \theta)$$
Hence...
$$p(D|\theta) = \prod_{i=1:n} p(x^i | y^i = c, \theta)p(y^i=c|\theta)$$
Now, I can see the prior is just a categorical distribution given by $Cat(y | \pi)$ so I can see how the Cat function is coming in equation for likelihood since $y^i$ falls in one of C categories.  But what im not able to understand is why is the equation taking product of all clases from $c=1:C$ for the class conditional density?
Is it assuming that for the ith sample, the label can only take on one of c values, hence the distribution of $p(x|y,\theta)$ is given by the product of all the disributions raised to the power of indicator?  hence only one of them can be true?
Thanks

Your understanding is correct. The indicator function ensures that only the term corresponding to the true class $y_i$ contributes, and all other terms become $1$, effectively ignoring them. This is actually a useful way to  write out the likelihood function for mixed continuous–discrete distributions with class-conditional densities similar to the categorical variables as mentioned below, so that you can optimize $\pi$ and the $(\mu_c
, Σ_c)$ terms separately based on the log-likelihood later on to derive the MLEs for the Gaussians and tied/diagonal covariances.

Another formulation that appears more complex but facilitates mathematical manipulations is as follows, using the Iverson bracket:
$${f(x\mid {\boldsymbol {p}})=\prod _{i=1}^{k}p_{i}^{[x=i]}},$$
where ${[x=i]}$ evaluates to $1$ if ${x=i, 0}$ otherwise. There are various advantages of this formulation, e.g.:

It is easier to write out the likelihood function of a set of independent identically distributed categorical variables.

It connects the categorical distribution with the related multinomial distribution.

It shows why the Dirichlet distribution is the conjugate prior of the categorical distribution, and allows the posterior distribution of the parameters to be calculated.