text:
Unclear paragraph in Sutton-Barto on "Tile Coding"
Sutton-Barto (Tile Coding, page 218):

For example, choosing $\alpha = 1/n$, where n is the number
of tilings, results in exact one-trial learning. If the example $s\to v$ is trained on, then
whatever the prior estimate, $\hat{v}(s,w_t)$, the new estimate will be $\hat{v}(s,w_t)=v$.

This part is not clear for me. I need to see mathematically why that is the case.

In short, this works because the derivative of the (implied) loss function happens to be numerically equal to the difference between current estimate and the target (this is actually a relatively common setup in regression and classifier estimators). In turn, the partial derivatives of weights associated with active tiles in tile coding each inherit that property, because their derivative factor used in the chain rule to figure out the impact of changing them in isolation is exactly 1 (whilst inactive weights have a factor of 0 and are not important to the calculation).
Taking a unit step for any single weight would correct the current estimate to the new observed value. As the next part of the text in S&B explains, you would not usually make such an update step if you need the estimator to generalise.
We can show this a little more rigorously using the formulae from the book. The general update rule for estimators is:
$$\mathbf{w} \leftarrow \mathbf{w} + \alpha[G - \hat{v}(s, \mathbf{w})]\nabla \hat{v}(s, \mathbf{w})$$
Where $G$ is the sampled or measured return, and $G - \hat{v}(s, \mathbf{w})$ is therefore a measure of the error in the estimate.
Let's call the error in the estimate $\delta$, we can treat that as a single number from now on - any update that adjusts the estimate by $\delta$ makes it match the new data exactly.
So now we have:
$$\mathbf{w} \leftarrow \mathbf{w} + \alpha \delta \nabla \hat{v}(s, \mathbf{w})$$
In linear approximators, we have feature vector $\mathbf{x} = x(s)$, and:
$$\hat{v}(s, \mathbf{w}) = \mathbf{w}^T\mathbf{x}$$
Therefore:
$$\nabla \hat{v}(s, \mathbf{w}) = \mathbf{x}$$
In tile coding, we always have $n$ active features (the $n$ tiles that the current state vector is contained by) with a feature vector value of $1$, and the rest are $0$. In terms of calculating $\hat{v}$ we can ignore all the $0$ values. The value of $\hat{v}$ is the sum of all the active weights that are associated with the $n \times 1$ values. For convenience (of keeping the notation simple) we can create a vector of length $n$ which is only the weights associated with those $1$s - let's call that $\mathbf{w^1}$. The associated part of $\mathbf{x}$ is a vector of $n$ $1$s that we can call $\mathbf{1}_n$
With the new vector, only for this one specific value of $s$ that has these specific $n$ features:
$$\hat{v}(s,\mathbf{w^1}) = \sum_{i=1}^n \mathbf{w^1}_i$$
Now the update rule for the selected weights is:
$$\mathbf{w^1} \leftarrow \mathbf{w^1} + \alpha\delta\nabla \hat{v}(s, \mathbf{w^1})$$
We also know that $\nabla \hat{v}(s, \mathbf{w^1}) = \mathbf{1}_n$. This matches to the intuition used in the S&B quote - if you change any active weight by some amount $y$, then the estimate also changes by that amount. Second-order derivatives need to be always zero for this to hold, otherwise you cannot make such easy statements about large changes (e.g. this logic would not work for any neural network).
So the update rule becomes:
$$\mathbf{w^1} \leftarrow \mathbf{w^1} + \alpha\delta\mathbf{1}_n$$
This causes a change to the value estimate:
$$\hat{v}(s,\mathbf{w^1} + \alpha\delta\mathbf{1}_n) = \hat{v}(s,\mathbf{w^1}) + \sum_{i=1}^n \alpha\delta$$
That is when calculated using the old $\mathbf{w^1}$ from before the update - this works as above because $(\mathbf{w^1} + \mathbf{y})\mathbf{x} = \mathbf{w^1}\mathbf{x} + \mathbf{y}\mathbf{x}$
A correction of $\delta$ would be "perfect" for the single data point, so if we want that, we want:
$$\delta = \sum_{i=1}^n \alpha\delta =  n \alpha\delta$$
or
$$\alpha = \frac{1}{n}$$


text:
How can my RNN get way better results than my ANN
So, I'm using the same dataset in both models but my RNN gets a 95% accuracy and my ANN gets 52%.
It is a time series, binary classification problem, and I know that RNN is better than ANN for time series, but I was suspecting that there was some data leak, since 95% is way higher than I expected.
Is it possible that I have a data leak only in my RNN? Or are RNN this much better than ANN for time series?

What do you mean by "data leak" in this case? Without having almost no info about your dataset, it's not possible to answer this unclear question.

https://en.wikipedia.org/wiki/Leakage_(machine_learning)
lol

An RNN has a recursive element to it, as its output is fed back as part of the next input; this provides a kind of memory.
When analysing a time series, values tend to depend on previous values (unless the time series is random), so an RNN is bound to perform better, as it has more context for classification.
An ANN basically just looks at values in isolation, and thus cannot make use of the history of the time series. So I would expect it to perform much worse in this case.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How contrastive loss work intuitively in siamese network
I am having issue in getting clear concept of contrastive loss used in siamese network.
Here is pytorch formula
torch.mean((1-label) * torch.pow(euclidean_distance, 2) +
                                      (label) * torch.pow(torch.clamp(margin - euclidean_distance, min=0.0), 2))

where margin=2.
If we convert this to equation format, it can be written as
(1-Y)*D^2 + Y* max(m-d,0)^2

Y=0, if both images are from same class
Y=1, if both images are from different class
What i think, if images are from same class the distance between embedding should decrease.   and if images are from different class, the distance should increase.
I am unable to map this concept to contrastive loss.
Let say, if Y is 1 and distance value is larger, the first part become zero (1-Y), and second also become zero, because it should choose whether m-d or 0 is bigger.
So the loss is zero which does not make sense.
Can you please help me to understand this

You can use mathjax to format the mathematical formulas here. So, maybe you can format the loss function.

I think you are confused about why the margin exists.
The margin exists in contrastive learning because we only want the model to output embeddings where negative samples are far from each other to a certain extent.
That is, we don't want the model to train itself to output embeddings where the distance between negtaive samples diverge to infinity. That would be an overkill and most likely the model will not be able to learn a meaningful representation.
Like you said here,

Let say, if Y is 1 and distance value is larger, the first part become
zero (1-Y), and second also become zero, because it should choose
whether m-d or 0 is bigger. So the loss is zero which does not make
sense. Can you please help me to understand this

The loss is zero when the distance is larger than m for negative samples. Thus, the model will not be further updated for negative pairs that it is already good at distinguishing.

thanks, this explanation make my confusion clear.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Should minimax with alpha beta pruning depth be an odd number?
I implemented the minimax algorithm with alpha-beta pruning to see how it works, with application to the connect four game.
My AI works fine, considering the AI is the MAX player (VS human player, which is the MIN), e.g. the Minimax root node of the tree.
However, since the root node is always the MAX player (AI), does this means that the leaf nodes, where pruning occurs, should be at an odd depth?
Whenever I use an odd number for depth, my algorithm works fine. But when I use an even number, the bottom node because a min node and AI looses the game.
Can anyone confirm that when using MAX as the root node, the depth should always be an odd number? AFAIK, it's never said explicitly in the algorithm's description.
Thanks

Do you use a transposition table and iterative deepening ? If yes, results of even depth-searches can trouble results of odd depth-searches, but shouldn't give bad moves as well. If no, there is no reason for an even search-depth to give a bad move (i used to have depths of even number).

@Vintarel no I don't use move ordering techniques. But when the depth is even the algorithm ends up computing a score for the min. Is this expected?

For even depths, the score should be a little lower (not too much though) than for odd depths, as the tree on leaves is supplied with moves favoring the MIN player. On my chess program, the scores could change of around 0.2 point when switching parity of depths, but a good move (score of 3 or more) still remained a good move. Try to see for shallow searches (depth = 1, 2) or by printing the results at every upper nodes if you can spot what happens.

Following Vintarel's advice, I reviewed my code and saw there was an error. Indeed, the evaluation function returned different values, depending on the AI or the player. So, for example, it would return -3 for the AI and 3 for the player, resulting in a wrong max/min optimization.
I modified the evaluation function, so that it returns the same value, independently of the player. After, the score is minimized or maximized, according to the current player (AI or human). In the end, odd/even depth has no impact and the algorithm works as expected!
Thanks


text:
Can a face recognition system be trained using only computer generated hyper realistic faces?
In order to train a face recognition system you need to have access to a large database with thousands of photos containing different faces. Companies like facebook and amazon have these databases but most average people do not.
If you don't have access to a sufficiently large dataset with faces, could you use computer generated random faces instead? I'm asking this because computers are becoming better and better in rendering hyper realistic faces. An example is the meetmike digital human showcase video. Another example is the unreal engine project spotlight video.. Lastly you also have websites like https://thispersondoesnotexist.com/ that can generate random faces.
What if you generate a couple of photos of the same computer generated face and you make sure that each photo shows the face in a different setting or from a different angle. Could you then use such photos to train a facial recognition system that can accurately recognize real people?

It sounds plausible. The catch is, to train a generator for human faces, you need to have a lot of face images anyway. So, you might directly train a face recognition on that set. However, if you don't have your own dataset, but only has access to a pretrained face generator model, what you describe makes sense to me.

I am a bit later to this but in short yes you can. The approach of training on synthetic data and transfer that knowledge to real data is called sim2real gap (borrowed from RL) and for computer vision you can now hear synth2real gap.
Microsoft has been working on that for a while (they do not have faces dataset) and it has published some relevant papers on it:

Fake It Till You Make It
3D face reconstruction with dense landmarks

There are also other approaches using GANs for face generation, although realism is boosted GANs suffer from micro glitches in the images that biases the training data. But even with that glitches you can successfully train.
You can also measure the knowledge transfer by computing some metrics on some public face datasets such as WIDER-FACE.


text:
Arcface implementation for image similarity produces opposite embeddings for positive negative image pairs
So I've built an arcface model with this arcface layer implementation:
https://github.com/4uiiurz1/keras-arcface/blob/master/metrics.py
I trained for a few epochs and now that I'm comparing the results I'm quite baffled.
According to the paper and resources I've read the embedding should produce embeddings where similar images are closer and should have a higher cosine similarity.
But I have the exact opposite case, I ran the models embedding layers through the hold out set and to 95% the mismatches are closer than the matches. Thus I have a reversed 95% accuracy.
My feed and labels are correct
I binned similar images in groups similar to here:
https://www.kaggle.com/ragnar123/unsupervised-baseline-arcface
but for a different dataset.
Could someone guess why this is happening? Is it possible that some initialization would produce the opposite goal?

Cosine similarity, $s$, is an angle distance. The nature of its distance is represented in the range $s \in [-1, 1]$ being $s=-1$ very dissimilar and $s=1$ very similar.
However the intuition of distances is that the closest to $0$ it is the more similar. So if you say
if s < threshold:
    match = True
else:
    match = False

It would produce the exact opposite results since cosine similarity is a measure where the closest similarity equals to $1$. So you have 2 options:

Transform cosine similarity into a traditional normalized distance $d \in  [0,1]$ where $0$-similar, $1$-dissimilar by transforming: $d = (-x + 1) / 2$. The $-$ sign is a source of hassles in metrics computation.
Do the thresholding with the above equal threshold like threshold=0.8 and then just issue the decission: if s >= threshold: match=True

thank you so much! It's been sometime but this intuitively makes sense :-)

You are welcome, I have been using it very recently and this issue came along, so glad to help with this :)


text:
How is catastrophic cancellation dealt with in loss functions?
It just occurred to me that this seems like it should be a very common problem that must have some kind of solution... Yet I'm not sure what it is...
If there is no solution, does this mean once a model reaches a certain level of accuracy numerical training methods can no longer be counted on to make it 'better'? And if so, is this a practical concern?

Could you please give an example of a "no solution" scenario to better illustrate the kind of thing you are thinking of and to make this more concrete? For instance, very often there is an optimal statistical solution to a data fitting problem. As a trivial example if you have a function $y=f(x)$ to learn and contradictory training examples $(x,y)$ of $(1, 0), (1, 1)$, then many neural networks will approximate $\hat{f}(1) = 0.5$ - it's valid and optimal according to the loss function. But not clear if this matches your "if there is no solution" scenario.

By 'no solution' I mean that there are no common mechanisms implemented in libraries (e.g. keras, sklearn, etc...) to counter the affects of catastrophic cancellation on the loss function when the model is already quite close to optimal.

Intuitively, it seems like unless this problem is addressed it might cause models which are nearly perfect to start to diverge. No?

Catastrophic cancellation occurs when a function to optimise includes the difference between two estimates to close numbers. As those estimates approach their true values, the ratio of the estimated difference between them and real difference between them may vary widely.
Although this may be a concern in a particular model or problem, it is not really related to neural network loss functions or how they learn from example data. No standard loss functions rely on a stable relative difference between two estimates from the neural network. Most loss functions compare a current estimate with a target value (or ground truth), and the error value for that is absolute and can approach zero without causing any instability.
There may be a few exceptions, and in addition there are other numerical issues that impact convergence stability for neural networks, where solutions are employed in neural network libraries. For example when dealing with output layers with exponentiation in them (e.g. sigmoid or softmax) combined with loss functions that have log terms in them (e.g. cross entropy), it is common to use a simplified gradient calculation that does not need to actually perform the exponentiation or log - this saves time and is numerically more stable. This is not the same issue as catastrophic cancellation, but clearly is considered important enough in neural networks that it is addressed.
As far as I know, there is no need to address catastropic cancellation in general neural network optimisation, and there are no libraries with standard solutions.
If you are estimating two close values, taking the difference, perhaps in some pipeline, and the resulting system is sensitive to proportional errors in that difference, then you may be concerned about catastrophic cancellation and need to do something about it. What that something is will not directly be related to neural network libraries though. Perhaps you will be able to reformulate your problem so that the difference between the estimates is not so important, or that measure could be estimated more directly.


text:
Extract person silhouette from photo or video
Are there any programming libraries or neural network design patterns designed for the task of finding persons in a photo/video and extracting their silhouettes (i.e. not only the rectangle containing the person but a bitmap marking which pixels are part of the person and which are background)?

You're looking for image or instance segmentation.

Check out pytorch/pyvision.  This is wonderful in that it has many great models that are pretrained that you can use to train your model on and then go further and use what is trained to train further.  It has bounding boxes, silhouettes, segmentation, etc.
https://medium.com/@ashishgupta_65016/semantic-segmentation-for-silhouette-extraction-e52c7d319295
https://pytorch.org/vision/main/auto_examples/plot_visualization_utils.html
I am sure there are many more options out there.  This is just one that I am familiar with.  Good luck!!


text:
How does PPO account for the last reward?
I was implementing PPO for the lunar lander environment in openai gym, but my agent seems to be getting stuck at a score of ~-80. On the website it says the agent gets rewarded +100 or -100 if it comes to a rest or crashes at the end. However, my time horizon is set to 20, which leads me to think the last reward is not properly accounted for (since episode lengths arent always a multiple of 20) and thus, the agent predicts bogous values for future states which causes the poor results. This begs the question: how does ppo account for the final reward? Do we just hope the algorithm stumbles upon an episode where the length is a multiple of the time horizon, or is it that the environment is programmed to wait a few steps after finishing to ensure that the final reward gets accounted for?

I actually feel kinda dumb writing this, but the update is made if the TOTAL environment steps is a multiple of the time horizon. I did, recode my PPO algorithm, and it seems to eventually solve the lunar lander environment, albeit very slowly (taking around 650k training steps from 1700 episodes to solve), and the training is very unstable, the agent might average around 100 points and drop to an avg of -80 in the next 500 episodes and suddenly jump back to 200 points (my lr is 3e-5).

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.


text:
Training a GAN after after evaluation metric reaches minimum
I am training a StyleGAN-3 using one of the pre-trained models. At some point, roughly halfway through the 5000 kimg recommended for fine-tuning, the FID50K score starts oscillating around a minimum value and appears to make no further progress. From your experience, can the model still make further improvements to generated images that will be noticeable to the human eye but not reflected in the FID50K, or should I stop training early?

That depends on the value of your loss.
If it is still too high, your model might be stuck in a local minimum. In this case early stopping and another random weight initialisation will help.
You might also need to change other hyper parameters. Another reason could be that your learning rate is too high and you overshoot the minimum.
If it is low enough and there has been no change for 3-5 epochs, it could be the global minimum. In that case, stop early.
Edit:
The possibility of an increase in human-observable image quality while the loss stays the same depends on the metric of your loss. Most likely no because usually the evaluation metric of the loss will cover that.

Thanks, but my question is somewhat different. What I meant was the possibility of an increase in human-observable image quality while the loss stays the same.

That depends on the metric of your loss but most likely no because usually the evaluation metric of the loss will cover that.

Sounds good, if you edit your answer accordingly. I'll accept.


text:
Is there an argument against using the (reviewed) predictions of a model as ground truth to further train exactly this model?
I plan to use my predictions as ground truth to continue training my model. These predictions are of course reviewed during this process. Is there an argument against that (reinforcement of slight mistakes/overfitting etc.)?

Here my specific use case described:
I am using detectron's faster R-CNN implementation to train a (pretrained) model to find defects of a machine part in images.
The goal is to find bounding boxes around these defects and to label them.
A colleague labeled some of the images (1500 images, making up 20% of the whole set) and I used those to train my model. Then I had the model predict defects on all 7500 images. My colleague asked me if he can review the predictions (and adjust/add if necessary) so he doesn't have to label the remaining images from scratch, and then I would like to continue the training with all the images.

Using the (unchecked) predictions of the model as training data is an approach known as "pseudo-labeling". It can help in certain situations, depending on the underlying structure of your dataset, but you have to be a bit careful about how you use it (e.g. only using high-confidence predictions as your pseudo-labels) and you always want to keep your pseudo-labels separate from your true labels, so you can potentially update them as your model changes.

But it sounds like you're not using the raw predictions as labels, but rather using the predictions of the models as a pointer to (currently unlabeled) examples which you then will manually label.
"Training on errors" is recognized mode of augmenting your training dataset, especially for "on-line" style systems where you're getting a continuous stream of new examples. The concept is to identify those examples which are predicted either inaccurately or with low confidence, identify the accurate labels for these instances, and then include them with the rest of the training set to help improve the predictions for similar sorts of examples in the future.
In contrast to pseudo-labeling, you're looking to correct the low confidence examples or the incorrectly predicted. Adding in high-confidence examples doesn't gain you much, as your current training set is already sufficient to correctly predict these. And with an on-line model where you're continually getting new examples, adding the well-predicted examples to your training set does potentially cause issues with subclass imbalance issues, as "normal" examples are expected to swamp out the rare outliers.

But it sounds like you have a fixed-size training set. In that case, the standard recommendation for the best course of action of dealing with unlabeled data applies: "pay someone to label it for you". What you're looking for is accurate labeling. How you get that is left somewhat nebulous, so long as the labeling is accurate. Using model results as a starting point is perfectly valid, assuming that whoever is doing the checking/correction is willing to actually do all the corrections (to the same quality level as a "from scratch" prediction) and won't wave through the model predictions as "ehh, good enough".
In addition to label accuracy, another issue may be selection bias (that is, the model may have certain subsets of examples which it performs worse/better on, and picking which examples to include in labeling on that basis may bias future training). But if you have a pre-determined, fixed-size training set this is not really an issue if you label all of them (or a model-independent random subset). The selection bias comes not from the initial model predictions/selection, but instead the (model-independent) selection of the examples to be labeled.

Thanks, that's very helpful. The keyword 'pseudo-labeling' is leading me to more literature, I didn't know that term before.

The answer is: It depends.
What you describe is a strategy often used to save time and costs for labelling data. It is important that the data you have already labelled (the 20%) is representative of the rest of data (the 80%). At the very least, you must have all classes in those 20%.
It is also important that you select a good detection model to have reliable predictions. Faster RCNN should be a good choice.
However, whether 20% labelled data is enough is difficult to tell. It depends on your data.
Your strategy itself is common. I'd just try whether 20% labelled data is enough. You can also fine-tune your faster RCNN model in between, say after 40% of the data is labelled to improve predictions further.

Thanks, continuing training as an intermediate step is a good idea.


text:
PPO custom implementation: do metrics like value loss, actor loss and entropy move a certain way?
I'm wondering whether problems with a custom PPO implementation (problem couldn't be made into a gym environment) can be debugged considering how the losses change over time.
In my current experiment, although results do get better in the actual task, the actor(policy) loss, the value function loss and the entropy move in weird ways.

How do these metrics usually change over the course of an experiment? Do they always move along the same paths? Thank you.

It's difficult to tell how the output of the loss changes over the course of the experiment as you use them as a measure on how well your model performs. Ideally, the loss decreases over time with minimal jittering. A decreasing loss means the model is learning and the hyper parameters fit the dataset.
If your loss is jittering too much without a clear decreasing trend, it might be that, for example, the learning rate is too high and the model overshoots the minimum. It also might be that there is a flaw in the model.
To make it short, no, the loss doesn't always move along the same path.
If you need to test your metric, create tests with small models. Use a small keras model (for example) where you know that there is no flaw inherent with the model and use standard hyper parameters. Then train the model twice. Once with your own metric and once with a standard metric similar to your own provided by a library. The path of the loss should look similar - but it will not be the same.
I'd do the above approach for every method you implemented yourself to make sure that the method itself is doing what it is intended to do.


text:
How does backpropagation know which weights to change?
I'm currently working on constructing a neural network from scratch (in JavaScript). I'm in the middle of working on the backpropagation, but there's something I don't understand: how does the backprop algorithm know which weights to change or which paths to take? The way I did it, it always took all of the paths/weights and changed them all. So how does the algorithm know which paths to take, which weights to change, and whether to add or subtract X amount from said weight?

Basically using chain rule. Check this video and if it is your taste, the following one on the playlist with the mathematical details to get a better understanding of the process. This one might also help.

@serali is correct, there are many resources to describe this process.  I'm going to talk about it in a very general way.
Disclaimer: this is a work in progress, it is not yet complete.
Question:
How do we know what weights to change in the network?
General Answer:
This is a very good question.  This is a key component of ML, and even small improvements there would make a huge difference for the overall community.  There are gigawatt-hours of electricity spent using the current answers to this question.
Training means you have an input that you want the network to transform into an output, and it means you have an output to compare it against.
The difference between what it makes, and what it should make, the error, can be run in reverse through the network.  As you go through that reverse run, you can get an error associated with each weight. if you move that error to zero, by changing the weight, then the network can be less bad.
Some variations:
Here are some approaches for relating the per-weight errors to the new weights.
Approaches:

(sgd link) take just a few or one sample at random and compute the error, and then move weights in that direction, usually only a small percentage of it. The intention is that each new update occurs closer to the optimum, and works well in lower-noise cases.
(conjugate gradient link, ) find a line along which to minimize, find the minimum, explore around that minimum to find a new direction for minimization.  For quadratic problems (shaped like parabola) it gives good solutions quickly.
(momentum link) take a running average of the last several weight changes, and do most of the movement in that direction
(regularization link) penalize weights that are too high, push back against learning them, tries to keep weights in smaller subset of the space unless pushed there by the data.
(dropout link) randomly set internal responses of neurons to zero, to force a robust and distributed representation of the processing in the network.  Keeps on network from "dominating the show" because in practice it tends to do more harm than good for generalization.
(rotated locking link) aka stochastic depth, randomly freeze layers in the architecture to avoid most learning occurring only in the last few layers, aka diminishing gradients.
(pca in vectorized weights) given a batch of errors for weights, perform pca and only retain highest x percent of eigenvalues, move in that direction.
(entropy-scaled momentum in pca elements) given batch of errors for weights, perform svd, and then map weight changes per sample to the largest eigenvectors associated with it


text:
Trouble writing the backpropagation algorithm in python through crossentropy and softmax
so I am writing my own neural network library for a class project and I got everything working for a simple 2-class test using the distance (L2) cost function. I wanted to get a similar result using softmax and crossentropy instead.
I did the derivation of the formulas as in this article, but I tried computing the derivative of the cost function w.r.t. its inputs and the Jacobian of the softmax outputs w.r.t. the inputs separately.
This is what I did:
So, let $Y$ be the matrix with rows equal to the expected class probabilities, $\hat{Y}$ the outputs of the softmax layer, $X$ the inputs of the softmax, all of these matrices of size $n\times c$ (samples times classes). The cost function $J(Y,\hat{Y}) = -sum(Y * \log(\hat{Y}))$ where * is element by element multiplication.
Then, if $E$ is the cost function, I first compute
$$\frac{\partial E}{\partial \hat{Y}_j^{(i)}} = -\frac{Y_j^{(i)}}{\hat{Y}_j^{(i)}},\:\: i=1,\cdots,n : j=1,\cdots,c$$
This is a $n\times c$ matrix. Then, being $S$ the softMax function I compute the Jacobian of the softmax at each row of $X$, so a total of $n$ matrices of $c\times c$ dimensions.
$$JS^{(i)} = \left(\frac{\partial \hat{Y}_j^{(i)}}{\partial X_k^{(i)}} \right)_{j,k=1,\cdots,c},\:\: i=1,\cdots,n$$
Finally to compute the partial derivative of the error with respect to the inputs of the softmax I multiply the rows of $\frac{\partial E}{\partial \hat{Y}}$ with each jacobian, to get a new $n\times c$ matrix.
This way I have everything needed to compute:
$$\frac{\partial E}{\partial X_j^{(i)}}= \sum_{k=1}^c \frac{\partial E}{\partial \hat{Y}_k^{(i)}}\frac{\partial \hat{Y}_k^{(i)}}{\partial X_j^{(i)}} ,\:\: i=1,\cdots,n : j=1,\cdots,c$$
Is this correct? Where is the problem if not so? Might the numerical errors make everything fail?
Thanks in advance.

I found the bug on my code, now everything works just fine, so I am fairly sure that the derivation of the formulas is on point. Optimisation wise, clearly using the short formula on the end of the article is better and less prone to errors than using the full derivation.


text:
PPO: policy loss becomes nan
I'm implement PPO for a very specific problem, and it seems to be working somewhat, but after a few epochs, I always get something like this:
/home/antoni4040/anaconda3/lib/python3.9/site-packages/torch/autograd/__init__.py:173: UserWarning: Error detected in MulBackward0. Traceback of forward call that caused the error:
  File "/home/antoni4040/Documents/Toulouse/Toulouse-Optimizer/toulousePPO.py", line 439, in <module>
    agent.train(120)
  File "/home/antoni4040/Documents/Toulouse/Toulouse-Optimizer/toulousePPO.py", line 298, in train
    self.learn(ep=epoch, lr=learning_rate, clip=clip)
  File "/home/antoni4040/Documents/Toulouse/Toulouse-Optimizer/toulousePPO.py", line 169, in learn
    clipped_ratio * sampled_normalized_advantage )
 (Triggered internally at  /opt/conda/conda-bld/pytorch_1646756402876/work/torch/csrc/autograd/python_anomaly_mode.cpp:104.)
  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
Traceback (most recent call last):
  File "/home/antoni4040/Documents/Toulouse/Toulouse-Optimizer/toulousePPO.py", line 439, in <module>
    agent.train(120)
  File "/home/antoni4040/Documents/Toulouse/Toulouse-Optimizer/toulousePPO.py", line 298, in train
    self.learn(ep=epoch, lr=learning_rate, clip=clip)
  File "/home/antoni4040/Documents/Toulouse/Toulouse-Optimizer/toulousePPO.py", line 197, in learn
    total_loss.backward()
  File "/home/antoni4040/anaconda3/lib/python3.9/site-packages/torch/_tensor.py", line 363, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/antoni4040/anaconda3/lib/python3.9/site-packages/torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: Function 'MulBackward0' returned nan values in its 0th output.

I started printing the losses and it turns out that the policy loss becomes nan.
I do something like this for the policy loss:
dist, value = self.model(batchStates)
new_probs = dist.log_prob(batchActions)
ratio =  torch.exp(new_probs - batchOldProbs)

clipped_ratio = ratio.clamp(min=1.0 - clip, max=1.0 + clip)
policy_loss = torch.min(ratio * sampled_normalized_advantage,
                   clipped_ratio * sampled_normalized_advantage )
policy_loss = policy_loss.mean()

Should I add a small value to the ratio (1.0e-6 or something), or is it something else?

Have you considered taking an approach that does not require exponentiation?  Without seeing numbers, that is most likely what is vanishing (as you seem to realize from your question).  Could you simply continue to operate on the log values, which will give you a great deal more "room"?

That's how PPO works tho. That's the math behind it.

You might want to try substituting the exponentiation with a piecewise-defined function that uses a numerical approximation that is more numerically stable for low values of the exponent, such as using $e^x \approx 1 + x$ (First-order Maclaurin series) when x is under a certain threshold

That's actually a very nice trick. I'll accept the answer after some testing.


text:
In Value Iteration, why can we initialize the value function arbitrarily?
I have not been able to find a good explanation of this, other than statements that the algorithm is guaranteed to converge with arbitrary choices for initial values in each state. Is this something to do with the Bellman optimality constraint itself?
It's hard to see that this is true intuitively since my intuition states that there ought to be ways in which an arbitrary choice of values could cause the value function to converge incorrectly compared to the ground truth. For instance, what if a state that in practice had low reward is assigned a very high initial reward value in value iteration? Would the algorithm not construct a value function that highly values trajectories passing through that state?

If the value function of a state $v(s)$ is relatively high, then you are absolutely correct in saying that a greedy policy may choose to visit $s$, since the high $v(s)$ makes it very promising. The key idea here is that the update rule of value iteration will gradually change the value function and likewise will gradually change the policy.
Suppose that the optimal value function $v_*(s)$ of a specific state $s$ is low, yet the value function $v(s)$ is initialized much higher. Then, the update rule in the pseudocode you provided will eventually decrease $v(s)$ to $v_*(s)$. To see this fact intuitively, note that $v(s)$ can be decomposed as the expected return (sum of rewards) from that state:
$$v(s) = \mathbb{E}[R_k + R_{k+1} + R_{k+2} + \ldots + R_T|S_k = s].$$
If the expected return from $s$ is smaller than $v(s)$, then $v(s)$ will decrease with the aforementioned update. The actual update rule approximates the expected return from $s$ using the next reward $R_k$ and the value function of the next state $v(s')$ via bootstrapping. Note that $v(s')$ can be decomposed as follows:
$$v(s') = \mathbb{E}[R_{k+1} + R_{k+2} + \ldots + R_T|S_{k+1}=s'].$$
Therefore, $v(s')$ quantifies the part of the return aside from the next reward $R_k$, and the update rule indeed quantifies the expected return from $s$. If $v(s')$ is initialized too high or low, then the update rule will simultaneously be adjusting it to the optimal $v_*(s')$ while adjusting $v(s)$ closer to $v_*(s)$.
Once the value functions change as described above, then the greedy policy may also change and favor other states aside from $s$ with higher value functions.

Is this something to do with the Bellman optimality constraint itself?

That is part of it, and important for episodic problems without discounting. The Bellman equations link between time steps, providing a direction that data about returns is used to bootstrap value estimates. In episodic problems there are terminal states with 0 value by definition. In the first pass through state space, any state that can transition to a terminal state will be exposed to that fixed knowledge, and could be altered to take that transition. This data is backed up through time steps, at least once per value iteration sweep, and maybe more frequently depending on the order of updating states. You can visualise this effect like a "flood fill" starting from terminal states and filling in the table with true values.
In addition, when there is a discount factor (optional for episodic environments, but required for continuing ones), then this progressively reduces any bias in existing estimates due to the term $r + \gamma v(s')$ repeatedly - over many updates - multiplying any error in the bootstrap from next state by $\gamma$, the discount factor.

For instance, what if a state that in practice had low reward is assigned a very high initial reward value in value iteration? Would the algorithm not construct a value function that highly values trajectories passing through that state?

In limited cases you can construct a combination of environment and starting value function adversarially, so that it does not converge properly. This would require that there is no discounting (or very high $gamma$ and also high $\theta$ cutoff to stop iterations early), and a set of self-consistent high start value estimates that form a loop preventing the iterations from making changes. Any single "rogue" high value estimate should quickly get reduced by the update mechanism (only exception would be a high value state that can loop to itself deterministically and without any discounting applied).
In this question, the OP constructs just such an adversarial start condition: Is the initialisation of $V(s)$ and $\pi(s)$ really arbitrary in policy iteration?
So in some senses, the initialisation is not strictly arbitrary, in that you can set up non-working examples. However, the value iteration algorithm is far more robust than your initial concerns imply, thanks to the effects described above. In addition, if you are implementing a solver then you get to choose initialisation, and can make conservative choices (e.g. don't initialise any value higher than max possible reward - there is no known benefit to doing so for dynamic programming algorithms). These choices are not strictly required, the space of working initialisations is much larger, but you are also not required to attempt to code truly arbitrary value table initialisation.


text:
What should I think about when designing a custom loss function?
I'm trying to get my toy network to learn a sine wave.
I output (via tanh) a number between -1 and 1, and I want the network to minimise the following loss, where self(x) are the predictions.
loss = -torch.mean(self(x)*y)

This should be equivalent to trading a stock with a sinusoidal price.
The issue I'm having is that the network doesn't learn anything. It does work if I change the loss function to be torch.mean((self(x)-y)**2) (MSE), but this isn't what I want. I'm trying to focus the network on 'making a profit', not making a prediction.
I think the issue may be related to the convexity of the loss function, but I'm not sure, and I'm not certain how to proceed. I've experimented with differing learning rates, but alas nothing works.
What should I be thinking about?
Actual code:
%load_ext tensorboard
import matplotlib.pyplot as plt; plt.rcParams["figure.figsize"] = (30,8)
import torch;from torch.utils.data import Dataset, DataLoader
import torch.nn.functional as F;import pytorch_lightning as pl
from torch import nn, tensor
def piecewise(x): return 2*(x>0)-1

class TsDs(torch.utils.data.Dataset):
  def __init__(self, s, l=5): super().__init__();self.l,self.s=l,s
  def __len__(self): return self.s.shape[0] - 1 - self.l
  def __getitem__(self, i): return self.s[i:i+self.l], torch.log(self.s[i+self.l+1]/self.s[i+self.l])
  def plt(self): plt.plot(self.s)

class TsDm(pl.LightningDataModule):
  def __init__(self, length=5000, batch_size=1000): super().__init__();self.batch_size=batch_size;self.s = torch.sin(torch.arange(length)*0.2) + 5 + 0*torch.rand(length)
  def train_dataloader(self): return DataLoader(TsDs(self.s[:3999]), batch_size=self.batch_size, shuffle=True)
  def val_dataloader(self): return DataLoader(TsDs(self.s[4000:]), batch_size=self.batch_size)

dm = TsDm()

class MyModel(pl.LightningModule):
    def __init__(self, learning_rate=0.01):
        super().__init__();self.learning_rate = learning_rate
        super().__init__();self.learning_rate = learning_rate
        self.conv1 = nn.Conv1d(1,5,2)
        self.lin1 = nn.Linear(20,3);self.lin2 = nn.Linear(3,1)
        # self.network = nn.Sequential(nn.Conv1d(1,5,2),nn.ReLU(),nn.Linear(20,3),nn.ReLU(),nn.Linear(3,1), nn.Tanh())
        # self.network = nn.Sequential(nn.Linear(5,5),nn.ReLU(),nn.Linear(5,3),nn.ReLU(),nn.Linear(3,1), nn.Tanh())
    def forward(self, x):
        out = x.unsqueeze(1)
        out = self.conv1(out)
        out = out.reshape(-1,20)
        out = nn.ReLU()(out)
        out = self.lin1(out)
        out = nn.ReLU()(out)
        out = self.lin2(out)
        return nn.Tanh()(out)

    def step(self, batch, batch_idx, stage):
        x, y = batch
        loss = -torch.mean(self(x)*y)
        # loss = torch.mean((self(x)-y)**2)
        print(loss)
        self.log("loss", loss, prog_bar=True)
        return loss
    def training_step(self, batch, batch_idx): return self.step(batch, batch_idx, "train")
    def validation_step(self, batch, batch_idx): return self.step(batch, batch_idx, "val")
    def configure_optimizers(self): return torch.optim.SGD(self.parameters(), lr=self.learning_rate)

#logger = pl.loggers.TensorBoardLogger(save_dir="/content/")
mm = MyModel(0.1);trainer = pl.Trainer(max_epochs=10)
# trainer.tune(mm, dm)
trainer.fit(mm, datamodule=dm)
#

It doesn't matter that your loss is not convex. As a matter of fact, the loss function of a neural network is in general neither convex nor concave (reference).
As ImotVoksim points out, the issue is that the loss function you've defined has nothing to do with the problem you're trying to solve.
For example, a stock price of zero is going to give you a loss of zero and hence no gradients at all: the neural network is allowed to output arbitrary values whenever the stock price is zero.
You want to "make a profit". I'm not sure why the MSE is not good in this case: if your neural network outputs the correct price of the stock for the next time period, you can use this information to make the trade which will maximize your profit.
Or do you want to predict the price further in the future? In that case you could use an MSE of the form
torch.mean((self(x_t)-y_{t+n})**2)

where x_t is the input at time period t and y_{t+n} the price of the stock at time period t+n, n a number you choose.

The reasoning was as sort of 'reinforcement learning' bellman-style reward. We only care about the movements in the price, and we care more about large movements, hence the reasoning for this loss function. If our price contains stochastic elements, we want to make a strong prediction when we know what's going to happen next, and a 0 prediction otherwise.

Your loss function doesn't have to be continuous, so I would just go with writing one which is more explicitly encoding what you describe. E.g. if the price movement is such that $|y_{t+1} - y_t| < \delta$ then set the loss to 0 and otherwise use the MSE. Not sure what you mean by "when we know what's gong to happen". Who knows in that case?

The loss function you have defined is the negative mean of the predictions multiplied by the targets, where both values are on the closed interval [-1,1]. The alternative you've listed is the MSE. Let's look at what these two loss functions result in with some concrete values:

self(x)
y
Your Loss
MSE

1
-1
1
4

0.5
-1
0.5
2.25

0
-1
0
1

-0.5
-1
-0.5
0.25

-1
-1
-1
0

A good loss function is such that when we are making good predictions the loss is close to 0, and when we are making bad predictions the loss increases above 0. That's why we often call the lost function a cost function. In other words, bad predictions are costly for the model, good predictions cost nothing ($L = 0$). As you can see, your loss function lacks this property, and instead it attains 0 whenever your prediction or the target is 0. Hence why your model cannot learn.

I thought the idea is to minimise the loss, not to reduce it to zero.
In the case of MSE, it can only go to zero, but in my case, it should be somewhere below that, but it has definitely has a minimum. I'm trying to regress an action for a reward, not make a prediction.


text:
What is the advantage of adding CNN to LSTM for forecasting sequential data?
I am working with simulated sequential data and the goal is to forecast that data. Long-short-term-memory (LSTM) is one of the most advanced models to forecast time series according to this post. I can imagine that it is a good model because of the memory-cells they use which are useful when learning of the past.
This paper discussed the use of CNN in time-series analysis. It says:

CNN is suitable for forecasting time-series because it offers dilated
convolutions, in which filters can be used to compute dilations
between cells. The size of the space between each cell allows the
neural network to understand better the relationships between the
different observations in the time-series [14].

It even outperformed LSTM:

A specific architecture of CNN, WaveNet, outperformed LSTM and the
other methods in forecasting financial time-series [16].

I see more and more posts about the usage of CNN in combination with LSTM, but I can't find any information about the advantages and disadvantages of using these in combination.
This post (Advantages of CNN vs. LSTM for sequence data like text or log-files), it is asked about the advantages of CNN vs. LSTM. But I would like to know the advantages and disadvantages of adding CNN to LSTM for forecasting univariate sequential data? Or should you use one of the two algorithms?

If your data are 2D + time then you might want to use something like ConvLSTM.
If you only care about 1D + time then you don't need to add CNN to LSTM you only use one or the other. In terms of pros and cons have a look at this empirical study on how dilated convolutions compare to LSTMs for modeling sequential data. If you're also interested in the more theoretical aspects, this paper shows how temporal convolutional networks are related to truncated RNNs.

A convolution layer can extract local patterns from its input (from the previous layer). Neighborhood size of the extracted pattern is relative to the kernel size. As we go deeper in the network the reception field of the layers increase. Accordingly, layers can recognize patters from larger neighborhoods of the input data. However, this increase in the reception field is very gradual. So, to understand patterns concerning large neighborhoods of input data, you need very deep networks. To ease this requirement, you can use dilated convolution layers, as in [1,2]. Hence, with shallower networks u can have reception fields as large as the whole input sequence. notice that many time series problems require patterns concerning the whole input sequence.
using LSTM layers in combination with convolution layers is also beneficial in many cases. in this scenario, several blocks of convolution layers first extract local hierarchical patterns. So the input sequence is represented by a reach collection of local patterns. Then, this patterns are feed to an LSTM to understand dependencies related to whole sequence. also, notice that an lstm layer generally requires more parameters to learn.
[1] Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., & Muller, P. A. (2019). Deep learning for time series classification: a review. Data mining and knowledge discovery, 33(4), 917-963.
[2] Wang, Z., Yan, W., & Oates, T. (2017, May). Time series classification from scratch with deep neural networks: A strong baseline. In 2017 International joint conference on neural networks (IJCNN) (pp. 1578-1585). IEEE.


text:
What does IOU3 mean in this context?
I was reading a paper and this paragraph said that:

The ground truth score is calculated based on the intersectionover-
union (IoU) of the perturbed image and the ground truth one. Since we
would like to distinguish among IoU values close to 1.0, we use $IOU^3$
as the ground truth score.

I couldn't find any references to this, nor mentioned in the paper. Is it just simply the calculated IoU to the power of 3? Or is it a special kind of IoU calculation method? What does this mean?

Link? Are you sure 3 is not a reference?

https://openaccess.thecvf.com/content/WACV2022/papers/Shi_Self-Supervised_Shape_Alignment_for_Sports_Field_Registration_WACV_2022_paper.pdf

Yeah it's not a reference, it's under section 3.1, score regression network

Since we would like to distinguish among IoU values close to 1.0, we
use IOU3 as the ground truth score for the SRN.

It seems to be just IoU to the power of 3. They use the cube function because they want their regressor to pick up small differences in IoU when IoU is near 1.0. They could have used any other function that is sufficiently steep close to 1.0. I haven't seen power of 3 being used very often but it's pretty straightforward to implement and I assume it works better than the square in this problem for them to use it.

From context, I would say: Yes, it's IoU to the power of 3, since they want to have larger differences between values close to 1. Obviously, the difference between 0.9^3 and 1.0^3 is larger than between 0.9 and 1.0.


text:
How to implement PPO without using a Critic
I am using the standard policy gradient algorithm, REINFORCE, to solve a RL problem and was thinking about implementing Proximal Policy Optimization (PPO) to increase the sample efficiency of my solution. From the original paper, it seems the clipped loss PPO optimizes includes an Advantage $A_t$ term:
$$
L^{CLIP} (\theta) = \mathbb{E}_t \Big[min(r_t(\theta)A_t,
\: clip(r_t(\theta), 1 - \epsilon, 1 + \epsilon)A_t \Big]
$$
Generally, this advantage $A_t$ is calculated using an actor-critic schema, where we train a network (critic) to predict the value $V(s)$ of a given state, which is then used to calculate $A_t$. However, my RL task is episodic and the trajectories end (i.e., arrive at a terminal state) in just a few actions, so I would rather not use a critic network. Thus, my question is the following: How can I implement PPO without a critic network, i.e., without a network that predicts $V(s)$?
To achieve this, I thought about simply substituting the advantage term $A_t$ used in $L^{CLIP}$ for the discounted sum of rewards $R_t$ used in the REINFORCE loss. Should this work fine or is there a better alternative?

Your justification for not wanting to use a critic (that the episodes are short) does not make sense to me. I would expect that including the critic would result in a substantial variance reduction (and substantially faster training) due to the extra baseline and also the bootstrapping which is done in the generalized advantage estimation (GAE). I don't see how the episode length being short is relevant.
However, if you don't want to use an advantage, you don't have to. You can simply replace $A_t$ with the (monte carlo sampled) cumulative reward $R_t$. The problem is that the clipping objective only makes sense if you use an advantage. It won't make sense to do that clipping if you are using $R_t$ instead of $A_t$.
It seems dubious to me to call such an algorithm "PPO" as really it would just be REINFORCE. The only possible difference between what you describe and reinforce, is that PPO should generate a dataset of transitions and do multiple mini-batch updates per dataset. If you do decide you want to go that route, you should note that you still need the importance sampling term (the unfortunately named $r_t(\theta)$) for the update to be unbiased.

The real reason why I don't want to use a critic is because I am using a specific NN architecture called Neural Logic Machine and it might be difficult to predict $V(s)$. As you mention, the reason why I want to use PPO instead of REINFORCE is to be able to increase the sample-efficiency of REINFORCE, by performing multiple gradient ascent iterations over the same data. However, if by using $R_t$ instead of the advantage then the clipping objective does not make sense, does this mean I could change the policy probabilities too much by performing multiple training iterations over the same data?

Yes it's possible that the training will be a bit more unstable without the clipping. But my biggest concern is that without the bootstrapping (which is included in gae) the algorithm may learn quite slowly

So, if I have understood you correctly, you think that even if the trajectories are short I should try to use actor-critic instead of vanilla REINFORCE and that will make a bigger impact in the sample-efficiency of my algorithm than using PPO (instead of REINFORCE) and performing multiple gradient ascents over the same data.

I am saying that the generalized advantage estimator (GAE) seems to be really good and I would highly recommend using it for the advantage calculation; yes this would require having a critic in addition to the policy. The question of sample efficiency is separate and it's hard to make any quantifiable claim there. But yes I believe it will definitely help.


text:
How to preserve Markov Property in Deep Reinforcement Learning when using "mixup" or "mixreg"?
I've read through these two papers:

(original about "mixup") https://arxiv.org/pdf/1710.09412.pdf
(variant for RL, "mixreg") https://arxiv.org/pdf/2010.10814.pdf

They are about a rather interesting approach to improving model generalization. Here's the thing, though - I can easily see how to use this for supervised learning, as there is always a "reward"/prediction on each "observation"/row-of-data .
However, even though the second paper (mixreg) talks about applying this to RL specifically, I don't understand how you can manage this.
How would you preserve the Markov property if you're mixing observations/rewards that aren't necessarily in any way sequential?

Because each environment individually satisfies the Markov property, the distribution of the next state $s_{t+1}$ in any transition depends only on $s_t$, $a_t$, and the transition probabilities of the active environment $e_t$:
$$P(s_{t+1} | s_t, a_t) = P_{e_t}(s_t,a_t,s_{t+1})$$
Each environment is independent of the others, so when mixreg builds a combined state $(s_i, s_j, \lambda)$ and chooses an action $a$, the distribution of the next combined state $(s_{i+1}, s_{j+1}, \lambda)$ is simply the combined distribution:
$$P((s_{i+1}, s_{j+1}, \lambda) | (s_i, s_j, \lambda), a) = P_{e_i}(s_i,a,s_{i+1}) P_{e_j}(s_j,a,s_{j+1})$$
This remains conditionally independent of all other states and actions.

I see. So the Markov property is satisfied, even though that doesn't mean that there will be any signal to be found/remaining when mixing states. Is that correct? I might be destroying the signal, but Markov property is preserved?

Do you mean the rewards? Those are preserved, either using the reward from the environment with the bigger weight or a convex mixture of the two rewards

Got it, thank you for your answer! I have a related question here: https://ai.stackexchange.com/questions/35941/how-to-deal-with-sparse-rewards-when-applying-mixup-or-mixreg-to-deep-reinfo  In case you are able and willing to answer.

I'm not sure how mixreg changes things there, but in single environment RL, the agent learns in steps. First it learns that the state immediately before the reward is valuable, then the state immediately before that one, and so on.


text:
Why are SVMs / Softmax classifiers considered linear while neural networks are non-linear?
My understanding is that neural networks are definitely not linear classifiers, as the point of functions like ReLU is to introduce non-linearity.
However, here's where my understanding starts to break down. A classifier, like Softmax or SVM is considered to be a "linear classifier". I'm using the definitions from CS 231N. There may be another definition of an SVM, but I'm not considering that.
In a linear classifier we have the following: $W \cdot x$ where

$W$ is the weights
$x$ is the input vector.

The output is a vector of scores. 1 score per label.
An SVM classifier uses hinge loss to update the weights:

A softmax classifier normalizes the output values using the softmax function and then uses cross-entropy loss to update the weights:

From the lecture CS231n Winter 2016: Lecture 2: Data-driven approach, kNN, Linear Classification 1, we have the following image to help visualize what a linear classifier does:

Essentially, each image can be considered a point in 3072 dimensional space, and we are drawing lines through this space. On one side of the "car" line, the car score for a given point will go up. On the other side of the "car" line, the car score for a given point will go down.
However, this doesn't seem to be that much different from ReLU (Taken from this post: https://stats.stackexchange.com/questions/158549/why-are-activation-functions-needed-in-neural-networks):

So what is the fundamental difference between a "linear" classifier like Softmax / SVM and a multi-layer neural network? Why can't a SVM classifier learn any function but a neural network can?

In my view neural nets are more universal than SVMs: they can approximate any functions from one multi-dimensional space to another. They are often used as classifiers by adding SoftMax or some other trick. SVMs, on the other hand, are purely classifiers, but they can do non-linear classification using so called "kernel trick" (https://en.wikipedia.org/wiki/Support-vector_machine).

@blamocur Ideally, our site is not meant to share "views" or "opinions" but facts and evidence. Just because there's a lot of hype around NNs now, it doesn't mean that they are more universal than SVMs, whatever that means. See also this.

I was confused because the images look similar even though in reality the problems the 2 images are solving are completely different:

The first image shows a linear classifier assigning scores for each of $C$ classes to an input image
The second image shows a binary classifier (I think) scoring an item as ether white or purple (or something like that)

Consider a single class in the linear classifier. The linear classifier can only draw a single line through $N$ dimensional space. Items on one side of this line have negative car scores. Items on the other side have positive car scores.
For example:

Here, the 2nd image in the row represents the SVM's attempt to extract all the features of a car into a single image. As images are "closer" to this image, the SVM will give them a higher car score. As images are farther away from this image, the SVM will give them a lower car score.
The SVM cannot extract individual features, it has to smush them all into a single line in $D$ dimensional space (where $D$ is the number of pixels). That line can be represented as:

On the other hand. A multi-layer neural network, for example:

Can now partition the space into a 100 different ways. The example is a bit convoluted,  since the neural net is classifying 1 thing into several buckets instead of doing binary classification, but the principle is the same. Maybe 10 out of a 100 neurons in the second layer are dedicated to partitioning the space of images with regard to cars.


text:
Using GraphSAGE model for multigraph datasets
I checked out applications of GraphSAGE and it seems like its primarily used for single graph datasets. For example - Cora dataset - Its one big graph with 2708 nodes and 5429 edges. The model can learn the node representation of this big graph and it can be later used for downstream tasks like node classification, link prediction.
I have a dataset with hundreds of graphs that are relatively small (about 15 nodes and 20 edges on avg per graph). Each graph represents a separate datapoint i.e, there is no relation between these graphs. It is similar to datasets like MUTAG, PROTEIN datasets . My question is if GraphSAGE is suitable for this kind of data ? Can it be used to learn node representation and be used for downstream tasks like node/graph/link classification?
I am following this stellargraph example - stellargraph_graphsage
It uses a sampler to perform random walks and solve a classification task. However, this sampler intakes only one Graph as input (big Cora graph). I am confused about how to use a dataset with multiple graphs here.
unsupervised_samples = UnsupervisedSampler(
    G, nodes=nodes, length=length, number_of_walks=number_of_walks
)

My question is if GraphSAGE is suitable for this kind of data?

To my knowledge, GraphSAGE is designed for very large graphs with highly connected nodes (like social networks). The neighborhood sampling mitigates oversmoothing problems that occur when aggregating large neighborhoods. I guess that other architectures are better suited, but it's always worth trying it out with your dataset.

Can it be used to learn node representation and be used for downstream tasks like node/graph/link classification?

Yes, GraphSAGE produces node embeddings for every node that can be used for a classification or a regression task.

However, this sampler intakes only one Graph as input (big Cora graph). I am confused about how to use a dataset with multiple graphs here.

Most graph libraries work like that. The models/functions only accept a single graph, if you want to input multiple graphs at once, you can take a look at graph batching. How it works is that you combine your $n$ separate graphs into a single large graph (with disjoint subgraphs) because GNNs work on neighborhoods the individual graphs will not affect each other. Here this principle is described.


text:
Why does the coding layer in a VAE have a range of values?
While reading the book, Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems, I read that VAEs using a sampling technique to obtain values from the coding layer. However, the output of a neural network activation is just a single value. In that case, what values does the VAE sample?

What do you mean by "output of a neural network activation is just a single value"? Which activation are you talking about exactly?

The VAE's encoder is usually implemented to produce the mean and variance. These are vectors, which can also be 1-dimensional (equivalently, scalars). If that's the case, then the latent vector is also 1-dimensional (if I understand your question correctly). See this implementation, where the size of the mean and variance vector is $20$, so the size of $z$ is also $20$, but you can change this to $1$.
(Actually, the (co)variance may not be a vector but a matrix, but I assume a diagonal covariance, like in the linked implementation and the VAE paper).

The idea of a VAE is that the code/feature layers produce a probability distribution of features (Gaussian to be exact), by predicting its mean and variance.
This is the point that requires sampling, since probability distributions are abstract concepts, and sampling is the only way to get numbers from them.


text:
How do I know if my Random Forest Regressor Model is overfitted?
Im creating a Random Forest Regressor Model with a small dataset (30 data points). I tried with other models but RF was the best one, however, after applying GridSearchCv I got that the training set is 0 and the test set is 0.12, and I dont know if that means that my model is overfitted.
Could anyone please help me with this?
Tks

Hi and welcome to StackExchange!
First of all, your dataset is truly, extremely small. Maybe someone can correct me, but I would say 30 points is so small that using RandomForest is not appropriate.
That aside, overfitting is when your test set performance is worse to training set performance, due to the model fitting itself to noise in the training set. In most cases, you will see SOME degree of this (test set performance worse than training set). However, the question is how much.
In your case, you have basically a "perfect score" on your training set - this will basically ALWAYS signify some degree of overfitting (in the real world, there's pretty much no way a model can ever "truly" be getting a perfect score unless it's overfitting). This is confirmed by the considerably larger error in the test set.
The question to you is: Yes, you have overfitting. But is this degree of overfitting acceptable? Is 0.12 error on test set good enough for you? If it is, then you don't need to worry too much. If not, you need to make some changes!
If that's too much error for you, I would HIGHLY recommend using a simpler method (some form of regression, linear or logistic). Your dataset is too small for such complex algorithms to do well.

Thanks Vladimir!
I tried with Linear Regression, Kernel Ridge and Lasso but I didn't get the result I was expecting. I was reading and maybe I could try with a Bayesian Model since it works with small data but I don't know if you could recommend me something else ? tks.

@Gaby Can you give me more information about the kind of data you're working with? What variables you're using (how many you have) and how much total data you have? Maybe I could give a recommendation.

@Gaby The way you pre-process the data and the features you create are generally far more important than the algorithm you choose. I'm happy to give some suggestions if you give more details, but generally, for such a small amount of data, I would recommend sticking with a simple model (linear reg.) and put your time into pre-processing and feature engineering.

I'm trying to predict the temperature in Colombia. For that purpose, Im using the following variables: CO2, population, forest area, agricultural land, agricultural methane emissions, coal consumption, gas consumption, oil consumption, renewables consumption and gdp. The information is available for the period 1990 - 2020 but is annualized. I used Min Max Scaler for standardization and i dont really have null values. Thanks for your help.

@Gaby I see. First off, that's a large amount of variables for such a small dataset. Remember that statistics is not magic :) There is no realistic way a model could figure out the relationship between 12 variables given only 30 points to look at. With that in mind, apply a bit of expertise - look at your variables. Do you think that things like GDP, energy consumption, population and even forest area (which probably doesn't change dramatically from year to year) would realistically predict next year's mean temperature? To be honest, personally, I'm very doubtful.

@Gaby I know CO2 and methane are greenhouse gasses, but even that I doubt would be any kind of realistic predictor of temperature in the very next year - these processes take a long time to unfold, and they do so in complex ways. If I were you, I would try a basic time series approach. I think realistically, that would be one of the few ways to try to make sense of this data.


text:
Is plain autoencoder a generative model?
I am wondering how a plain auto encoder is a generative model though its version might be but how can a plain auto encoder can be generative. I know that Vaes which is a version of the autoencoder is generative as it generates distribution for latent variables and whole data explicitly. But I am not able to think how an autoencoder generates probability distribution and becomes a generative model.
Also from this youtube video: here It says plain auto encoder is not a generative model. See last line from picture.

I think one of the reasons that plain AE is not generative is because it does not learn a joint probability function. For exemple, VAE learns p(x,z), which z is the latent variable. Also, the main goal of VAE is not reduce the reconstruction error, but estimate the intractable posterior distribution of z, p(z|x).

An autoencoder is not considered a generative model, because it only reconstructs the given input. You could use the decoder like a generative model by putting in different vectors. However, the standard autoencoder mostly learns a sparse latent space. This means that you will have distinct clusters in the latent space (see the left image below). The decoder has never learned to reconstruct vectors in between the clusters, so it will produce very abstract things - mostly garbage.
Instead a variational autoencoder (VAE) is considered a generative model. It's basically an autoencoder with a modified bottleneck. This VAE learns a dense latent space (see image on the right), this means you can sample any vector from the latent space, pass it to the model and it will give you a nice result with somewhat interpolated object properties from the dataset.
This article provides a nice overview of the two models.

Figure taken from here

Just as completing, in general, the autoencoders are an unsupervised learning technique in which we use neural networks for the task of representation learning. Specifically, we'll design a neural network architecture to impose a network bottleneck, forcing a compressed knowledge representation of the original input. In this case, we dont generate any new data and just compress them.
In the case of VQ-VAE or VAE, we generate new data because the architecture encodes all data in latent space and samples from that (continuous in VAE and discrete in VQ-VAE). Therefore, although the decoder will generate similar outputs (if regulated well), the input data is not just a compressed version.For better understanding, in VAE, the mean and variance of encoded data are used for generation.
Also, you can read these good references: https://www.jeremyjordan.me/autoencoders/ , https://www.v7labs.com/blog/autoencoders-guide ,  https://www.analyticsvidhya.com/blog/2021/06/complete-guide-on-how-to-use-autoencoders-in-python/

As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.


text:
Why do Q-values diverge without a target network?
After reviewing similar posts on this topic, I understand that a target network is used to prevent "divergence", but am not sure what it actually means. Q-values are predicted using a function approximator. The weights of the function approximator are then updated using the difference between the Q-value and TD target ($r + \gamma\max_aQ(s',a)$). Now assuming, that the estimate of the Q-value was wrong, the weights could easily be updated so that they are correct the next time the state is encountered. My confusion arises when online blogposts say that this change in weights modifies the target Q-value($Q(s', a)$) too. I don't see how the target Q-value gets updated when the current Q-value is changed.

I don't see how the target Q-value gets updated when the current Q-value is changed.

Without a separate target network, this happens because the approximator will generalise, and the generalisation will include the successor/target states. This can be very likely in many environments, since successor states often have many similar features to the previous states (think how similar any video game might look a few video frames on from any position).
In tabular variants of Q learning this does not happen, a change to any single Q value give state, action is always made to a single estimate that is isolated from all other estimates. Adding approximation changes things, and is not avoidable - in fact it is usually desirable to have strong generalisation in order to obtain good estimates for never-seen-before states. However, the flip side of this strong generalisation is that updating estimates for $Q(s,a)$ will impact many other $Q(s_i,a_j)$
One related thing worth bearing in mind is that the Q-function update in DQN is a semi-gradient update. If you do not use a target network, then technically the full gradient needs to take account of the changes to the TD target when the weights change (because both $Q(s,a)$ and $Q(s',a')$ are calculated use the same approximator). So one way to try and solve the same issue is to alter the update to use the full gradient. The maths for this is more complex than normal DQN, but is addressed in the Sutton & Barto book in chapter 11 section 11.7 "Gradient-TD Methods".
In practice for DQN, most experiments seem to prefer some variant of target networks. This also effectively changes the learning task to a full gradient one, with different progression of the TD targets. Although it is not as theoretically nice as a full gradient method, it seems to be a pragmatic choice by many researchers and developers.
The paper Full Gradient DQN Reinforcement Learning: A Provably Convergent Scheme attempts to make a fair comparison of DQN with a TD-gradient DQN, and comes to the conclusion that the full gradient approach is better - however, there may be other reasons to prefer the slightly clunkier semi-gradient plus target network approach.


text:
Is My Extra Tree Classifier Model Over/Underfit?
I was playing around with some models and training them on a heart disease dataset and found my Extra Tree Classifier reported a 100% accuracy. I was shocked at first since I had never gotten a perfect score report from a model, but I realized that the model is most likely over/underfit. After doing some research, I found that you can determine if a model is over/underfit by comparing the Training Score and Cross Validation Score reports, but after looking at the report, I could not tell if my model was over/underfit or if it really is perfect. (I am new to machine learning and I have high doubts that it is actually perfect.)
Here is the model report:

Can someone help me figure out if this model is over/underfit? (If I need to give any more data/info to figure this out, please let me know!)

"you can determine if a model is over/underfit by comparing the Training Score and Cross Validation Score reports" > did you do that? How's the performance on the validation set?

Based off the report, the training score was 100 and the cross validation score was about 98.

Turns out the model contained a whole lot of bias because 72% of the dataset was duplicated data. The reason for the CV and training score being so close is because the testing data was basically the same as the training data.

To evaluate your fit, the results on the validation set, as you say, are essential.

A good performance on both the training and the validation set is a sign of a good fit
A poor performance on the training set (and therefore on the validations set) is a sign of underfitting
A good performance on the training set, coupled with a poor performance on the validation set is a sign of overfitting.

As you can see you cannot evaluate based on the training set alone, whether it's a good fit or an overfit of the training set.
However, you stated in a comment that you also got 98% on the validation set. We can therefore conclude that you do have a good fit and a working model.
The very high resulting score is probably due to the fact that you're working on a very easy task. The size shown in the image seems to be 100 samples per class -- this makes me think this is a 'toy' dataset used just as an exercise. Whether this works in the real world, whatever your objective is, depends on how representative this dataset is.

I see what you mean. The reason there's only 100 samples is because I split my dataset to be 80% for training and 20% for testing. Would increasing the testing split make it make it more obvious?
I changed the split to be 50/50 just to see the difference. The new report has around 500 samples. It reported 100 training score and 93 cv score.

The exact size doesn't matter as long as they are both statistically representative, and 80% training is kind of standard. But this isn't the point, what I mean is this is an "easy" task for the model to solve.


text:
Why can the sum over timesteps in the Vanilla Policy Gradient be ignored?
I understand how to derive the vanilla policy gradient
$$
\begin{align}
    \nabla_{\theta}J(\pi_{\theta}) = \mathbb{E}_{\pi_{\theta}} \left[ \sum_{t = 0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_{t} \mid s_{t}) \hat{A}^{\pi_{\theta}}(s_{t}, a_{t}) \right]
\end{align}
$$
as is also demonstrated in the openai spinning up documentation. Reading the PPO paper, they say that the most commonly used gradient estimator is
$$
\begin{align}
    \nabla_{\theta}J(\pi_{\theta}) = \mathbb{E}_{\pi_{\theta}} \left[\nabla_{\theta}\log \pi_{\theta}(a_{t} \mid s_{t}) \hat{A}^{\pi_{\theta}}(s_{t}, a_{t}) \right]
\end{align}
$$
and from there they argue that this is equivalent to using the loss function
$$
\begin{align}
    \mathcal{L}^{\text{PG}}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[\log \pi_{\theta}(a_{t} \mid s_{t}) \hat{A}^{\pi_{\theta}}(s_{t}, a_{t}) \right]
\end{align}
$$
where the derivative is then taken with respect to $\theta$.
Question: How does the vanilla policy gradient relate to the gradient stated in the PPO paper? More precisely, they seem to be identical up to the sum over $t$. Why can the sum be ignored and the gradient is still the same?
Thanks in advance for any help!

Short answer:
The expectation $\mathbb{E}_t$ in the PPO paper is not an expectation over trajectories, but a mean. I suppose the confusion comes from there. The two quantities are otherwise very similar. The gradient is however not the same, as it is a biased estimator.
Long answer:
There are many related formulations for RL problems, on one hand we have the finite-horizon (episodic) setting:
$$ J_1(\theta)=\mathbb{E}_{\tau\sim p_{\pi_{\theta}}}\left[\sum_{t= 0}^T r(s_t,a_t)\right]. $$
On the other hand, we have the commonly used infinite-horizon setting (assumed by the PPO paper):
$$ J_2(\theta)=\mathbb{E}_{\tau\sim p_{\pi_{\theta}}}\left[\sum_{t\geq 0} \gamma^tr(s_t,a_t)\right]. $$
In the finite horizon setting, we have :
\begin{equation} \nabla_{\theta} J_1(\theta) = \mathbb{E}_{\tau\sim p_{\pi_{\theta}}}\left[ \left(\sum_{t= 0}^T \nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) \right)R(\tau) \right], \end{equation}
where $R(\tau)=\sum_{t= 0}^T r(s_t,a_t)$. As explained in the SpinningUp post, we can replace $R(\tau)$ by $A_{\pi_{\theta}}(s_t,a_t)$.
Now, a similar proof shows that:
\begin{equation} \nabla_{\theta} J_2(\theta) = \mathbb{E}_{\tau\sim p_{\pi_{\theta}}}\left[ \sum_{t\geq 0}\gamma^t \nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) A_{\pi_{\theta}}(s_t,a_t) \right]. \end{equation}
Now, in practice it is very common to approximate
$$\nabla_{\theta} J_2(\theta) \approx \mathbb{E}_{\tau\sim p_{\pi_{\theta}}}\left[ \sum_{t\geq 0}\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) A_{\pi_{\theta}}(s_t,a_t) \right]:=g^{\gamma},$$
that is we drop the discount factor $\gamma^t$. This relates to the policy gradient theorem and that we want to sample states from the steady-state distribution instead of the discounted state distribution.
Now, we want to approximate $A_{\pi}(s_t,a_t)$ by an estimator $\hat{A_t}$. Schulman et al in the GAE paper introduce the definition of a $\gamma$-just estimator. The exact definition is maybe not essential, but $\gamma$-just estimators include $Q_{\pi}(s_t,a_t)$, $A_{\pi}(s_t,a_t)$ and the GAE estimator used in the PPO paper.
So, if $\hat{A}_t$ is $\gamma$-just, then we have:
$$g^{\gamma} = \mathbb{E}_{\tau\sim p_{\pi_{\theta}}}\left[ \sum_{t\geq 0}\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) \hat{A}_t \right]. $$
This expectation is however still not convenient for a practical algorithm, as we need to collect the trajectories with a policy with older weights $\pi_{\theta_{\text{old}}}$. So we approximate
$$g^{\gamma} \approx \mathbb{E}_{\tau\sim p_{\pi_{\theta_{\text{old}}}}}\left[ \sum_{t\geq 0}\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) \hat{A}_t \right]:=\hat{g}. $$
Now, we need to approximate this expectation, so we collect $N$ rollouts of identical length $T$ (for simplicity) with the policy $\pi_{\theta_{\text{old}}}$, then we have an approximation of $g^{\gamma}$:
$$\nabla_{\theta} J_2(\theta) \approx g^{\gamma}\approx  \hat{g}\approx\frac{1}{N}\sum_{n=0}^N\sum_{t=0}^T \nabla_{\theta}\log\pi_{\theta}(a_t^n\mid s_t^n) \hat{A}_t^n,$$
which is what the $\mathbb{E}_t$ in the PPO paper means. The expression on the right-hand side is now finally useful for gradient descent-like algorithms, as $\hat{A_t}$does not depend on $\theta$.
Note also that much of the PPO paper aims to ensure that the last approximation by replacing $p_{\pi_{\theta}}$ by  $p_{\pi_{\theta_{\text{old}}}}$ is valid.

Thanks for the detailed explanation. Indeed, it does seem like the confusion stems from the $\hat{E}{t}$ notation. I understand your derivation but I dont get why your last estimator $\hat{g}$ is what they refer to by $\hat{E}{t}$. For me "an empirical batch over a finite batch of samples" is rather your last estimor without the sum over t, i.e. $\frac{1}{N} \sum_{n = 0}^{N} \nabla_{\theta} \log \pi_{\theta}(a_{t}^{n} \mid s_{t}^{n}) \hat{A}_{t}^{n}$.

We want to estimate $\hat{g}$, which is an expectation over trajectories. Depending on the algorithm prior to PPO, this is done differently. It relates to the discussion of TD(0) vs n-step TD estimation (although the quantity in the expectation is different, the principle is similar).  What you propose in the comment is to take $n$ samples of 1-step tranistions, which is often done in off-policy methods (like DDPG). The other way, done in VPG is to collect one trajectory of length $T$. You can also take $n$ trajectories of length $T$ as in A3C. The A3C paper shows many ways to estimate it.


text:
Why don't we use diffusion for non-graph CNNs?
I'm pretty new to graph neural networks, so please forgive me if this is a silly question.
Diffusion is a method used to improve graph CNNs, however it seems to me that general CNNs can also benefit from taking into consideration diffusion-like processes (for example, in CV, one may want information to diffuse from a small neighborhood, then to a bigger neighborhood, etc.). Also, PDE-based methods have been used in CV traditionally (before deep learning became a thing), so there's at least some evidence that this might work. But I did search extensively, and I can't find any information on using diffusion for general CNNs!
Therefore, what I'm curious about is: why didn't I find any information about this? Is it already explored under a different name? Or does it not really work because of some reason I don't know (or didn't think about)? Or maybe it does work, but there are some technical difficulties?

Just for completeness, here is one simple formalization of a diffusion GCN (Gasteiger et al.):
$\text{D-GCN}(X) = \sum_{k=1}^K A^k X W_k$
You have a diffusion factor $k \in [1 .. K]$ and you apply different GCNs with parameters $W_k$ to the $1$-hop, $2$-hop, .., $K$-hop neighborhood and sum the results together. $A^k$ denotes the $k^{th}$ power of the adjacency matrix.
There are two principles in visual CNNs that come to my mind that are comparable to the diffusion applied in graph models.
Firstly, dilated convolutions apply a kind of diffusion (they are also referred to as Atrous convolution). Specifically, the convolutional kernel is spread out based on a dilation factor $d$. The dilation factor determines the number of pixels that are skipped when applying the kernel. For example, if you have a 3x3 kernel, and apply a dilation rate of $d=1$, you get the standard convolution, a dilation rate of $d=2$ applies the 3x3 kernel to a 5x5 area of the image by using only every 2nd pixel (see the Image below). So in that case $d$ determines which pixel neighborhood the kernel is applied to, which is comparable to the $k$ in the diffusion GCN.
Dilated convolution has a history in segmentation models (see Yu et al. and Chen et al.). Apparently, it helps with segmentation, because the dilated convolutions can capture the context of pixels a bit better, so segmentation maps become cleaner.
It has also been applied with 1-dimensional convolutions in the WaveNet architecture. In that case dilation was used to cover greater receptive fields without increasing the number of parameters, i.e. model complexity.

The green is the output, blue is the input and the grayed pixels are the pixels where kernel weights are applied to. Image taken from here
Second, the Inception architecture by Szegedy et al. applies something that can be interpreted as a diffusion as well: Inception computes convolutions of different kernel sizes in parallel and stacks their feature maps (this blog post has a nice visualization). Here, the different kernel sizes would correspond to the $k$ and stacking is used instead of a sum.
I think these two examples are definitely related to the idea of diffusion GCN.


text:
Where is memory stored in a chatbot like LaMDA?
I have a basic understanding of how neural networks work, and I have always thought that those chatbots work in a similar way (but I might be wrong): they take an input, shape it in a way that can be processed, go through a series of layers in a neural network model (with weights based on their previous training), and then give an output.
In the published conversation with Blake Lemoine, the LaMDA chatbot makes a reference to a “previous conversation”: is this memory somehow stored in the weights of the neurons, or is there a physical memory from which it can read?
What goes out of my understanding in this particular situation, is how the notion of a "previous conversation" can be processed by a similar system: is this memory somehow stored in the neurons of the neural network (in this case the training would continue also during the conversation), or do they have some kind of memory storage that they can access during the evaluation of their inputs?
Here is the article where I've taken the link to the conversation: https://www.washingtonpost.com/technology/2022/06/11/google-ai-lamda-blake-lemoine/

What goes out of my understanding in this particular situation, is how the notion of a "previous conversation" can be processed by a similar system: is this memory somehow stored in the neurons of the neural network

Although I cannot answer what specific "memory" LaMDA may have (but I suspect it is either nonexistent or very simple), I think I can address this point.
The AI likely has no "notion" of anything here. It has access to a very large and complex statistical language model, where conversations that allude to prior conversations may happen from time to time. It is therefore capable of outputting text with similar-looking references, adjusting for the current  context in the flow of conversation.
If you asked the AI how much it enjoyed the summer of 2015, depending on what previous prompts (and any pre-prompt warm up*) were, it would likely make a good attempt at "remembering" what it did then, even though it did not exist. It would draw statistically on other texts related to that time, and might invent places, refer to real events that it may of attended etc. A lot will depend on how much pre-prompting is applied that forces the AI to answer questions logically "knowing" that it's an AI, but that pre-prompting can often fail later in a conversation.
Typically a chatbot built using one of the current crop of cutting-edge language models will fail to be self-consistent and coherent in the long term unless the person interacting with it is helping it along or willing to ignore the failures. With older chatbots this could happen very quickly. With GPT-3 and probably LaMDA, it may take a slightly longer conversation.
I find it interesting that the journalist in your linked story managed to get a complete opposite of the Engineer's experience:

In early June, Lemoine invited me over to talk to LaMDA. The first attempt sputtered out in the kind of mechanized responses you would expect from Siri or Alexa.

“Do you ever think of yourself as a person?” I asked.

“No, I don’t think of myself as a person,” LaMDA said. “I think of myself as an AI-powered dialog agent.”

Afterward, Lemoine said LaMDA had been telling me what I wanted to hear. “You never treated it like a person,” he said, “So it thought you wanted it to be a robot.”

The above simple exchange is pretty good proof that the engineer is suffering from confirmation bias, IMO.
Related: The Eliza Effect.

* A pre-prompt warm up is a piece of text that "frames" the start of interaction with a large language model so that it can perform a specific task. At the start of any conversation, a paragraph of context is fed into the language model to set the statistics for a certain kind of document. Open AI's GPT-3 uses this a lot to support different tasks.
A simple pre-prompt might look like this:

The following is a conversation between a human and AI. The AI is knowledgable, friendly and helpful.

AI: Hi, I am AI! How can I help you today?

Human:

At that point control will pass to the human - outside of the AI - so they will add their first sentence that will become part of the growing prompt history. Note that the AI, if left to its own devices would begin to speak for the human too, naively continuing to predict the conversation unitil its end! In fact it does often do so, but the chatbot code is set to catch obvious end points such as newlines, or the AI actually outputting "Human: ", and will force the mode of conversation back to looking like a chat. It is really quite crude.
LaMDA's architecture may not do exactly the same as GPT-3, in fact I would suspect it is different in some key ways. However, it is very likely that there are still two "layers" in play - a very large sophisticated language model trained on gigabytes of human text, and a control layer which makes the language model conform to a chat scenario. Even if the control layer is more sophisticated than GPT-3's pre-prompting, it will have similar limitations.

Good example of using pre-prompting on GPT-3 here: https://www.youtube.com/watch?v=mgEYP_OS418

Why wouldn't Lemoine's argument also apply to his own conversation with LaMDA? That would invalidate his thesis. LaMDA tells Lemoine it is a person because that is what the engineer wants to hear, anyone could tell.

@JaumeOliverLafont Yes, exactly my point. Confirmation bias very much on show there. There's probably a better term for rejecting/altering evidence that opposes a view (confirmation bias is usually quoted when accepting only evidence that supports a view).


text:
What is a "continuous vector"?
I have seen the concept of a "continuous vector" described in the context of embeddings. For instance, this answer to a question on embeddings in the context of deep learning. I obviously know what a vector is, but it isn't clear to me what a continuous vector is as a mathematical object. What is a "continuous vector"?

Continuous vector simply means that each value of the vector is allowed to be a real number, in contrast of integer values used in other categorical encoding techniques like one-hot encoding (only 0 and 1) or class-value mapping (a different integer per class).

Oh, ok. I was wondering if that was the case, since that's the most obvious option, but I wasn't sure if it was referring to something else in the context of machine learning. Thanks for the clarification.


text:
Should I include overlapping (input) Data in my training data
If I have time dependent data and want to predict the relative change for a future time. Should I separate the data so that the input times don't overlap?
With an example:
I have hourly temperature readings and want to predict the temperature change 4 hours after the last input hour. Now I could split my data so that I have
(overlapping)
1st data point: 1h, 2h, 3h, ... 10h -> predict for 14h
2nd data point: 2h, 3h, 4h ... 11h -> predict for 15h
(not overlapping)
1st data point: 1h, 2h, 3h, ... 10h -> predict for 14h
2nd data point: 15h, 16h, 17h ... 25h -> predict for 29h\
Further questions:
Does it depend on the Model/Training algorithm.
Does it depend on the Data?
Does it depend on the number of data points I have available? (I have hourly data for 6 years)

In general, both methods are valid to train temporal models. The only thing you need to check is that validation and test-set don't overlap with any of your training samples.
Using the overlapping variant can sometimes increase performance, because your model might benefit from the increased number of different starting points. This can be very relevant for certain datasets. For example, traffic data has periodic characteristics (little traffic at nighttimes / high traffic at daytimes). Let's say you have hourly traffic data and a sequence length of 24. Now your sequences will always start at the exact same day time. Therefore the model might not learn to properly forecast from other starting times.
I cannot think of any drawback the overlapping variant might have, so I'd opt for the overlapping variant. To be sure which one works best for you, I think you have to find out empirically.

Does it depend on the Model/Training algorithm.

In general it shouldn't, after all, in both cases you are always feeding the model valid sequences - any model and any training algorithm should deal well with either one.

Does it depend on the Data?

Yes, if the sequences have distinct starting points the non-overlapping variant might be better suited. The traffic-data example from above applies here.

Does it depend on the number of data points I have available? (I have hourly data for 6 years)

I'd say yes, you effectively increase the number of data points by using overlapping sequences, this is good if your dataset is small. Your dataset is comparably large.