text:
How does the discount factor (gamma) and memory replay work in Deep Q-learning?
Maybe in over my head about this but I'm having a hard time understanding the discount factor in Deep Q-learning.
Correct me if I'm wrong (1):
To train a Deep Q-learning network, every N:th step of action taken, a batch of random samples from replay memory are fed into the model's training function. These samples are totally unrelated to each other, they are unordered and we can't tell if a particular sample is taken from an episode which ended with a positive or negative reward. This is all there is to the training part of a Deep Q-learning network.
From what I've read, the meaning of the discount factor (gamma) is to decide "how much we value future rewards."
Correct me if I'm wrong (2):
With gamma = 0: The model only cares about the reward from when the state goes from A to B given the action C, and the training is encouraging this behaviour of the model with the reward.
With gamma = 0.9: The model cares about future rewards and acts based on what might come 5 (or what number we might want) steps from this.
I know I have gotten something wrong somewhere (or on multiple places :D): How can this small value, gamma, make the model train itself "for the future" when nobody (not even human) kan tell what the reward 5 steps from a particular step will be?
To me it would have made sense if we had not only 32 random tuples of (state, action, reward, state') but also a "episode end reward" and a "steps until reward" for each tuple and we in some way put these two new pieces of information into the function.
I'd really like to understand what the gamma does and how. I'm far from a maths person but from the code I've seen in a few tutorials and examples, I can't even begin to understand how this 0.9 or 0.99 can enable the model to train for future reward.
Please help :)

You may need to show some of the equations you are looking at. You refer multiple times to a variable called "sigma", but I am not familiar with a use for a variable labelled with $\sigma$ or $\Sigma$ in the usual equations of reinforcement learning or deep Q learning.

Hello again @NeilSlater, I'm sorry, as I wrote I suddenly started to name the factor "sigma" when it should have been "gamma". I've now corrected the original post.
Again, I'm sorry for the mistake, my only excuse being that I was to tired after trying to fix my neural net all day.

Let's try to understand what actually happens when learning. You start with random values for Q(s,a), the function that estimates the reward of an action given a state s, and you then get an immediate reward and then your loss is difference between the given reward + gamma *future estimated reward, i.e. $loss = Q(s,a) - (reward + gamma * Q(s', max_a))$.
Now if gamma=0, it means the loss would not be dependent on what happens afterwards - since the loss would simply be $Q(s,a) = reward$. So eventually Q(s,a) would simply converge to be the immediate reward of action a in state s. This means the agent would not end up learning the optimal strategy, since it would choose the action that maximizes the immediate reward without caring what happens afterwards, even if the step after it will die.
In order to get some intuition on how in practice it actually helps the agent look at the future, try to imagine how this would work out in practice. In the beginning rewards would be random, and at some point it will get to a state before the final episode reward would be very positive if the agent won. Then at some point it would get to the 2 before last state and might choose the action that gets to the positive one before last state - in which case since gamma > 0 it would optimize Q(s,a) to something positive. After millions of s,a pairs you can imagine how this will create a good agent.
Regarding your other suggestion - I am not an expert but I imagine there might be ways to take into account the final episode reward in a different manner - but the main nice thing about DQN is that it is simple and local - and even though you just update based on samples from local actions, with time the network will (hopefully) converge to be optimal.

Thank you for your reply! I had to let this problem go for awhile so I haven't read your answer until now.
It is this part "Then at some point it would get to the 2 before last state and might choose the action that gets to the positive one before last state" that I have a hard time to intuitively understand. I'll try to read it a few more times and think about it.
It just sounds so weird that it can learn something without knowing the state's position in the chain of events. :)


text:
How is the variance for a diffusion kernel derived for a diffusion model?
So I'm watching this video tutorial from CVPR this year on diffusion models, and I am confused by the variance term in the distribution on the left on the video. I understand that in the forward process, we can track intermediate distributions
$$q(\mathbf{x}_t|\mathbf{x}_{t-1}) = \mathcal{N}(\mathbf{x}_t;\sqrt{1-\beta_t}\mathbf{x}_{t-1},\beta_t\mathbf{I})$$
And that the joint distributions of all the intermediate steps conditioned on the input is given by
$$q(\mathbf{x}_{1:T}|\mathbf{x}_{0}) = \prod_{t=1}^Tq(\mathbf{x}_t|\mathbf{x}_{t-1}).$$
If we define $\bar{\alpha} = \prod_{s = 1}^t(1 - \beta_t),$ then we are supposed to derive the diffusion kernel
$$q(\mathbf{x}_t|\mathbf{x}_{0}) = \mathcal{N}(\mathbf{x}_t;\sqrt{\bar{\alpha}}\mathbf{x}_0,(1 - \bar{\alpha}_t)\mathbf{I})$$
I can definitely see why the mean is what it is, but I'm having a hard time seeing where we get the variance from. How is the variance derived?

While I think this question fits here, you might have better chances of an answer in stats.stackexchange.com

https://math.stackexchange.com/a/4476159/574215 This answer exactly derives both the mean and variance.

Using the reparameterization trick that tells us:
$$\begin{aligned}
\mathbf{z} &\sim \mathcal{N}(\mathbf{z}; \boldsymbol{\mu}, \boldsymbol{\sigma}\boldsymbol{I}) \\
\mathbf{z} &= \boldsymbol{\mu} + \boldsymbol{\sigma} \odot \boldsymbol{\epsilon} \text{, where } \boldsymbol{\epsilon} \sim \mathcal{N}(0, \boldsymbol{I})
\end{aligned}$$
If we define $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}_t = \prod_{i=1}^t \alpha_i$:
$$
\begin{aligned}
\mathbf{x}_t
&= \sqrt{\alpha_t}\mathbf{x}_{t-1} + \sqrt{1 - \alpha_t}\boldsymbol{\epsilon}_{t-1}  \\
&= \sqrt{\alpha_t}(\sqrt{\alpha_{t-1}} \mathbf{x}_{t-2} + \sqrt{1 - \alpha_{t-1}}\boldsymbol{\epsilon}_{t-2}) + \sqrt{1 - \alpha_t} \boldsymbol{\epsilon}_{t-1} \\
&= \sqrt{\alpha_t \alpha_{t-1}}\mathbf{x}_{t-2} + \sqrt{\alpha_t(1 - \alpha_{t-1})}\boldsymbol{\epsilon}_{t-2} + \sqrt{1 - \alpha_{t}}\boldsymbol{\epsilon}_{t-1} \\
&= \sqrt{\alpha_t \alpha_{t-1}}\mathbf{x}_{t-2} + \sqrt{\alpha_t(1 - \alpha_{t-1})+(1 - \alpha_{t})}\bar{\boldsymbol{\epsilon}}_{t-2} \\
&= \sqrt{\alpha_t \alpha_{t-1}}\mathbf{x}_{t-2} + \sqrt{1-\alpha_t \alpha_{t-1}}\bar{\boldsymbol{\epsilon}}_{t-2} \\
&= \dots \\
&= \sqrt{\bar{\alpha}_t}\mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t}\boldsymbol{\epsilon} \\
q(\mathbf{x}_t \vert \mathbf{x}_0) &= \mathcal{N}(\mathbf{x}_t; \sqrt{\bar{\alpha}_t} \mathbf{x}_0, (1 - \bar{\alpha}_t)\mathbf{I})
\end{aligned}
$$

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What would be a good optimization technique for this kind of problem?
Problem Description:
Since I am not sure if there is a scientific term that categorizes this problem, I will do my best to describe it thoroughly.
Suppose there is a chamber that's being filled with poisonous gas. The amount of poisonous gas being released at 1-second intervals can be represented as a time series such as:

In order to reduce the poisonous gas, the operator has access to a set of jobs, which if initiated at a certain time, can release a certain amount of neutralizing gas per second over a certain duration. Specifically, a job can be represented as $$J_i(amount_i, duration_i, cooldown_i)$$ where:

$amount_i$ is the amount of neutralizing gas released by $J_i$ per second
$duration_i$ is how long $J_i$ lasts when activated
$cooldown_i$ is how long $J_i$ is unavailable before it can be used again. Note that this property is what makes this optimization problem difficult (more on that later).

Note that in this particular problem, at any timestamp $t$, the neutralizing gas can only reduce the poisonous gas that is present at timestamp $t$. So in other words, it doesn't linger.
$Problem_1$: If the operator has access to a set of jobs $J = ({J_0, J_1, J_2, ...})$, what would be the best placements of jobs in order to end up with the least total amount of poisonous gas remaining in the chamber?
There are many possible solutions, but a good one would have to allocate a job with the right amount at the right time. Because remember if a high-throughput job is allocated when there is hardly any poisonous gas being released, then it would all go to waste. The cooldowns are also a big factor since the assignments would have consequences on the entire timeline. Hence, I reckoned this might be a good optimization candidate for AI.
An example of a solution can be shown as:

What makes this problem difficult is the cooldown constraint for every job. Because the constraint is not on the range of values, but rather on the different possible solutions. What I mean is that if the optimization algorithm comes up with a potential solution that uses $J_0$ at timestamp $t$, it somehow needs to understand that it should ignore any other solution that tries to use $J_0$ between timestamps $t$ and $t + cooldown_0$. I assume the optimization algorithm that is useful for the problem should be somehow capable of that.
Question:
What would be a good optimization technique to solve (or approximate) $Problem_1$ given its jobs' cooldown constraint?
More info (what I have tried):

PSO: I was hoping to use Particle Swarm Optimization, but the issues are:

High number of dimensions: To be able to map job assignments to particles, the number of dimensions blows up to number of jobs * total_duration_seconds. So for example for 15 different jobs over 6 minutes, it would result in 5400 particle dimensions, where each dimension can be either 0 or 1. So for example, if particle_dimensions[3] is 1 then it represents using the 1st job ($J_0$) at timestamp = 4 seconds. Similarly if particle_dimensions[360 + 5] is 1, then it represents using the 2nd job ($J_1$) at timestamp = 6 seconds.  Is there a better way of mapping the particles so that the number of dimensions are lower?
Job cooldown: more importantly, the cooldown constraint makes the search space full of unfeasible areas which makes finding feasible solutions very difficult. Because if $J_0$ is used at 4 seconds, represented by $particle\_dimensions[3] = 1$ then only at index $particle\_dimensions[3 + cooldown_0]$ and later the value can be 1 again. All the values in between must be 0, otherwise, it would violate the cooldown constraint for $J_0$. Now imagine this must be held true for all jobs, so the whole 5400 dimensions need to be set in a way that none of them violate the cooldowns of each job respectively. In order to tackle this, I have assigned a penalty to unfeasible solutions, but they are so many that the particles cannot find their way into the few (very spread-apart) feasible areas. Is this why using PSO is a bad idea here? Or is there a way to tell particles how they should move in order to respect the cooldown constraint for each and every job? Is that even possible?

Constrained Programming: I am not very familiar with this paradigm, but it seems interesting given the cooldown constraint. Do you think this would be a good approach to tackle the cooldown constraint?

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
InstructGPT: What is the sigma in the loss function and why $\log(\cdot)$ is being used?
InstructGPT: What is the sigma in the loss function and why $\log(\cdot)$ is being used?
$$ \operatorname{loss}(\theta) = -\frac{1}{\binom{K}{2}}E_{(x,y_w,y_l)\sim D}[\log(\sigma(r_{\theta}(x, y_w) - r_{\theta}(x, y_l)))] $$
The equation was taken from the InstructGPT paper.

According to this guide, the sigma in this formula refers to the sigmoid activation function. The guide does not tell exactly why the sigmoid function is used here, so I will try to give a full explanation of how this loss formulation works (page 8, formula 1 in the InstructGPT paper):
$\text{loss}(\theta)=-\frac{1}{\binom{K}{2}}E_{(x,y_w,y_l) \sim D} [log(\sigma(r_\theta(x,y_w)-r_\theta(x,y_l)))]$
In the following I will use the notation from the paper:
$x$ refers to the given instruction
($y_w$, $y_l$) refers to a pair of responses out of the list of responses which a human ranked based on their preference
$y_w$ refers to the response that is preferred over the other, lesser preferred response $y_l$
$r$ refers to the reward model
$\theta$ refers to the trainable parameters of that reward model
$\sigma$ refers to the sigmoid activation function.
If you interpret $\sigma(r_\theta(x, y_w) - r_\theta(x, y_l))$ as the probability that the reward model assigns a higher reward to the preferred response $y_w$ than to the lesser preferred response $y_l$, the formula makes total sense. How do you maximize a probability? Exactly, by minimizing the negative log of that probability (this is called negative log likelihood). Hence the  minus sign in front of the formula and log function around the sigmoid.
In order to elaborate a bit more on that: If the reward model is working fine, then $r_\theta(x, y_w)$ will be a very large positive number and $r_\theta(x, y_l)$ will be a much lower number (maybe even a negative number). The difference $r_\theta(x, y_w) - r_\theta(x, y_l)$ will then be a very large positive number. And the sigmoid of a very large positive number approaches $1$. In that case, everything is according to plan and the loss will be very small (close to zero). However, if the reward model is failing, the assigned reward $r_\theta(x, y_w)$ might be much smaller than $r_\theta(x, y_l)$. Hence, the difference $r_\theta(x, y_w) - r_\theta(x, y_l)$ will be a (possibly very large) negative number. Take the sigmoid of that and you get a value that approaches $0$ (thus, the probability that the reward model assigns a higher reward to the preferred response will be small). As we are trying to maximize a probability by minimizing the negative log likelihood, we get a large loss in that case.
As there will be a varying number of ranked responses for each instruction in one batch ("between $K = 4$ and
$K = 9$ responses"), the losses of those pairwise comparisons must be weighted, so that each instruction has the same impact on the gradient update, no matter how many responses the humans have been presented for each instruction. The number of pairwise comparisons out of $K$ is $\binom{K}{2}$.
In order to wrap it up: By minimizing the loss described in the paper, the reward model gets incentivized to assign a large positive reward to responses the (hopefully adequately paid cough) humans in front of their computers consider to be very good responses and very large negative rewards to responses which those humans consider to be very bad. And this is exactly what is desired in order to fine-tune an LLM according to human preference using reinforcement learning.

Thank you. Any idea why it is actually needed here?

To improve this answer could you summarize the specific information from the source that answers the question. This link could become invalid in the future. So by extracting the content that answers the question makes it a stronger and more durable answer for those in the future.

NathanB and @respectful: alright, I expanded the answer in order to give a full explanation of the loss formula. This is based on the linked guide, the original InstructGPT paper, as well as my personal understanding/interpretation of the material.

@schreon Thank you for your response. I'm struggling to understand the quantity inside the sigmoid function: (,)−(,). If I am comparing two responses, A and B, and want to calculate the loss of B, if B is ranked higher than A, how is (,)−(,) calculated?

@itisyeetimetoday first, the loss is based on a comparison of A and B, so asking for only calculating the "loss of B" does not make much sense. You would rather say "the loss of the comparison of A and B, where B should be higher than A".  So if the human annotator said that (given the instruction $x$) the response B is preferred over A, then you assign $y_w = B$ and $y_l = A$. Then you feed $(x, y_w)$ into your reward model $r_\theta$, resulting in one scalar value (the value of the output neuron of your reward model). Repeat this for $(x, y_l)$ and you can calculate that quantity :-)

The probability of preferring one trajectory/response (say $A$) over the other (say $B$) is given by,
$$
\begin{aligned}
P(A>B) &= \frac{\exp(r_\theta(A))}{\exp(r_\theta(A))+\exp(r_\theta(B))}\\
&= \frac{1}{1+\exp(-(r_\theta(A)-r_\theta(B)))}\\
&= \sigma(r_\theta(A)-r_\theta(B))
\end{aligned}
$$
where $\sigma(x) = \frac{1}{1+e^{-x}}$ is the sigmoid function. This follows from the Bradley-Terry model for estimating score functions from pairwise preferences. If say $y_w$ and $y_l$ are the responses from the model (augmented with the query $x$), and we prefer $y_w$ over $y_l$, then $\mu(y_w)=1; \mu(y_l)=0$. We can use the following cross-entropy loss function and substitute for the probability from above equation to train our reward model:
$$
\begin{aligned}
L &= -\mu(y_w) \log(P(y_w>y_l)) + \mu(y_l) \log(P(y_l>y_w))\\
&= -\log(P(y_w>y_l))\\
&= -\log(\sigma(r_\theta(x,y_w)-r_\theta(x,y_l)))
\end{aligned}
$$
In InstructGPT, the model is made to generate $K$ responses. So we can have $K\choose2$ pairs of comparisons that we can make. Example if the model generates four responses, $A, B, C, D$ and our ranking is $B>C>D>A$, then there are ${4\choose2}=6$ comparisons possible: $B>C$, $B>D$, $B>A$, $C>D$, $C>A$ and $D>A$. The loss function in this case reduces to,
$$
L = - \frac{1}{K\choose2} E_{(x,y_w,y_l) \in \mathcal{D}}\Big[\log(\sigma(r_\theta(x,y_w)-r_\theta(x,y_l)))\Big]
$$
Hope this helps. I have created a blog post to explain RLHF in conversational AI models if you want to understand better.


text:
How does DistilBERT's training ensure that the [CLS] token's hidden state from DistilBERT aligns with that from BERT?
BERT's pre-training involves next sentence prediction (NSP) based on a classifier on top of the [CLS] token's last hidden state. This is primarily what gives the [CLS] token's hidden state the special ability to represent the entire input, yes?
DistilBERT's pre-training does not involve NSP. How can then the [CLS] token's hidden state from DistilBERT represent the entire input? I understand that DistilBERT's weights are initialized with corresponding weights from BERT, but is that enough?
If the triple loss used in DistilBERT's distillation process aligns hidden states of all tokens with corresponding states from BERT, I can see how [CLS] token's hidden state would also be aligned. But the original DistilBERT paper seems to focus only on the masked tokens in the loss function, and the [CLS] token is not masked.

During pre-training, BERT learns to associate the [CLS] token’s hidden state with the semantic content of the entire input, and the classifier on top of the [CLS] token's representation helps BERT perform various downstream tasks like sentence classification.
The distillation process focuses on aligning the hidden states of all tokens, including the [CLS] token via the required distillation loss which ensures that DistilBERT's representation of the [CLS] token is similar to the [CLS] token's representation in BERT. Also unlike the MLM loss, the additional optional cosine loss recommended by the original paper likely also focuses on aligning the hidden states of all tokens. Finally as you rightly mentioned, DistilBERT starts with BERT's weights initially, so the hidden state of the [CLS] token in DistilBERT is already in a good starting position to represent the entire input sequence in a similar manner to BERT. The distillation pretraining further refines and mimics this representation.


text:
Calculating Curiosity with Friston's Free Energy in Reinforcement Learning
I'm trying to implement the paper A Curiosity Algorithm for Robots Based on the Free Energy Principle in a reinforcement learning environment using PyTorch, but I am unclear how curiosity is calculated.
As the paper describes, I made a 'transitioner' which predicts the next state given the current state and action. One layer of that transitioner is a Bayesian linear layer from Blitz; the probability distribution of weights in that layer is $q_ψ(w) = N(w|μ,σ_2)$ where $ψ = \{μ,σ\}$. Then, curiosity is defined as the Kullback–Leibler divergence $D_{KL}[q_ψ(w|s_{t+1})||q_ψ(w)]$.
$q_ψ(w)$ is easily available, but $q_ψ(w|s_{t+1})$ is described as value of $q$ minimizing free energy $F = D_{KL}[q_ψ(w)||p_ψ(w)] − E_{q_ψ}(w)[log (p_ψ(s_{t+1}|w))]$, and $p_ψ$ isn't given. I think I understand the concept, but I'm puzzled how to implement this.

To update: I am learning much by focusing on the paper inspiring Blitz: "Weight Uncertainty in Neural Networks" (also cited in "A Curiosity Algorithm for Robots Based on the Free Energy Principle"). It seems the loss-function for weight probability distributions is, in fact, the variational free energy. Hence, DKL[qψ(w|st+1)||qψ(w)] literally describes comparing weights before and after training.

You can use mathjax on this site.

Oh, thank you. I'll try editing the question.


text:
Inference process and flow, and role of GPU, CPU, and RAM
This is a noob question.
I load a HuggingFace transformer model into GPU and create a HuggingFace pipeline using that model. Then I run inference on the model using the pipeline.
I would be glad to read in some depth about the actual process flow of the data, in particular the role of GPU, CPU, and RAM in this process.
For instance,

I see a spike in CPU usage when I run inference. What causes it?
If I have multiple CPUs, and run multiple inference tasks simultaneously, will they be parallelized?
Does it make sense to use something like joblib for inference? Given that I am loading the model into GPU.


text:
Back propagation activation function derivative
I am reading about backpropagation for fully connected neural networks and I found a very interesting article by Jeremy Jordan. It explains the process from start to finish. There is a section though that confused me a bit. The partial derivative of the cost function (MSE) with regard to the $\theta_{jk}^{(2)}$ weights is:
$$\frac{\partial J(\theta)}{\partial \theta_{jk}^{(2)}} =  \left( \frac{\partial J(\theta)}{\partial a_j^{(3)}}\right) \left( \frac{\partial a_j^{(3)}}{\partial z_j^{(3)}}\right) \left(\frac{\partial z_j^{(3)}}{\partial \theta_{jk}^{(2)}} \right) \tag{1}$$
The article defines the next equation as the "error" term. The equation $eq:2$ is the combination of the first two partials in the chain rule:
$$ \delta_i^{(3)}  = \frac {1}{m} (y_i - a_i^{(3)}) f^{'}(a^{(3)}) \tag{2}$$
Where:

$ i: $ The index of the neuron in the layer
$ ^{(3)}: $ Denotes the layer (in this case 3 is the output layer)
$ z_i: $ The weighted sum of the inputs of the $i_{th}$ neuron
$ m: $ The number of training samples
$ y_i: $ The expected value of the $ i_{th} $ neuron
$ a_i: $ The predicted value of the $ i_{th} $ neuron
$ f^{'}: $ The derivative of the activation function

So a few lines after the definition above the article states:

$ \delta^{(3)} $ is a vector of length j where j is equal to the number of output neurons
$$ \delta^{(3)} =
 \begin{bmatrix}
 y_1 - a_1^{(3)} \newline
 y_2 - a_2^{(3)} \newline
 \cdots \newline
 y_j - a_j^{(3)} \newline
 \end{bmatrix} f^{'}(a^{(3)})  \tag{3}
$$

Q1. I strongly suspect that the  $ f^{'}(a^{(3)}) $ is a vector of length $j$ and not a scalar. Basically, it is a vector containing the derivative of the activation function for every neuron of the output layer. How is it possible in $eq:3$ to multiply it with another vector and still get a vector and not a $j\ x\ j$ matrix? Is the multiplication elementwise?
Q2. How is the $ f^{'}(a^{(3)}) $ calculated for every neuron for multiple training samples? From what I understand, while training with batches I would have to average the $ (y_i - a_i^{(3)}) $ term for the whole batch for every neuron. So in fact the term $ (y_i - a_i^{(3)}) $ is the sum for the whole batch and that's why the $ \frac {1}{m} $ is present.  Does that apply to the derivative too? Meaning do I have to calculate the average of the derivative for the whole batch for each neuron?
Q3. What does $ f^{'}(a^{(3)}) $ actually mean? Is this the derivative of the activation function evaluated with the values of the $a_i^{(3)}$ outputs? Or is it the derivative of the activation function evaluated with the values of the weighted sum $ z_i $ that is actually passed through the activation function to produce the $a_i^{(3)} = f(z_i)$ output? And if the second would I have to keep track of the average of the $z_i$ for each neuron in order to obtain the average of the $ f^{'} $

The author is rather free with changing from row to column format. The main philosophy or framework seems to be to implement the directions of "forward" evaluation and "backwards" gradient differentiation in the left-right direction, in diagrams as well as in formulas.
However, this philosophy is broken several times, for instance in writing $a=f(z)$ instead of $f(z)=a$, or in using weight matrices that are indexed for matrix-vector multiplication, that is, the usual right-to-left direction $(z^{(3)})^T=\theta^{(2)} (a^{(2)})^T$, which following the philosophy should be written as $a^{(2)}(\theta^{(2)})^T=z^{(3)}$.
But then again the philosophy gets reversed in formulas like
$$
\delta^{(l)} = \delta^{(l + 1)}\,\Theta^{(l)}\,f'\left( a^{(l)} \right)
$$
which clearly is right-to-left, which suggests that the gradients $δ^{(l)}$ are row vectors and the argument vectors like $a^{(l)}$ are column vectors.
In short, it's a mess.

Despite that, your questions have direct answers without relying too much on what directional philosophy is used
Q1. $f'(a^{(3)})$ as used and positioned relative to other vectors and matrices is the diagonal matrix with the entries $f'(a_j^{(3)})$ on the diagonal. This comes out as component-wise product in matrix-vector or vector-matrix multiplications.
Q2. If you were to discuss that, you would need another index in all the formulas indicating the training sample. Such as $J(x^{[k]},\theta)$ as the residual of the net output to $y^{[k]}$. The gradient of the sum $\frac1m\sum_{k=1}^mJ(x^{[k]},\theta)$ would be the sum of all single gradients that get computed independently of each other. Another interpretation of the factor $\frac1m$ is that it is the gradient at the top level $J$ that gets propagated backwards to the gradients of each variable.
Q3. Of course some doubt is appropriate. As the activated value is a function of the linear combined input, $a^{(3)}=f(z^{(3)})$, so is the derivative $\frac{\partial a^{(3)}}{\partial z^{(3)}}=f'(z^{(3)})$. Both must have the same arguments. This is just a typo, perhaps copy-pasted some times.

I've tried how to consistently implement the left-to-right philosophy, but it is too cumbersome. One would have to use something unfamiliar like some kind of reverse polish notation, so instead of $v=\phi(u,w)$ one would have to write $[u,w:\,\phi]=v$ or similar. So it is better to stay with right-to-left consistently, as also the author ended up doing. Thus $x,z,a$ are column vectors, gradients (against the tradition in differential calculus) are row vectors. In algorithmic differentiation it is one tradition to denote tangent vectors for forward differentiation with a dot, $\dot x, \dot z,\dot a$, and gradients that get pushed back with a bar, so $\bar x,\bar z=\delta, \bar a$.
The construction principle for gradient propagation is that if $v=\phi(u,w)$, then the relation of tangents pushed forward to the level of before and after the operation $\phi$ and gradients pushed back to that stage satisfy
$$
\bar v\dot v=\bar u\dot u+\bar w\dot w.
$$
Inserting $\dot v=\phi_{,u}\dot u+\phi_{,w}\dot w$ results in
$$
\bar v\phi_{,u}\dot u+\bar v\phi_{,w}\dot w=\bar u\dot u+\bar w\dot w.
$$
Comparing both sides gives $\bar u=\bar v\phi_{,u}$, $\bar w=\bar v\phi_{,w}$.
In a wider context $\bar w$ is a linear functional, meaning with scalar value, of $\dot w$. So if $w$ is a matrix, then the linear functional is obtained via the trace, ${\rm Tr}(\bar w·\dot w)$. So if for instance $\phi(u,w)=w·u$ in a matrix-vector product, then by the product rule $\dot v=w·\dot u+\dot w·u$ and
$$
{\rm Tr}(\bar w·\dot w)=\bar v·\dot w·u={\rm Tr}(u·\bar v·\dot w),
$$
so the comparison gives $\bar w=u·\bar v$.

The example network in atomic formulas is
\begin{align}
z^{(2)}&=\Theta^{(1)}·x       \\
a^{(2)}&=f(z^{(2)})           \\
z^{(3)}&=\Theta^{(2)}·a^{(2)} \\
a^{(3)}&=f(z^{(3)})           \\
\end{align}
and then $J$ is computed via some loss function from $a^{(3)}$ and the reference value $y$.
Starting from the gradient $\bar a^{(3)}$ computed from the loss function, the pushed-back gradients compute as
\begin{align}
\bar z^{(3)} &= \bar a^{(3)}·{\rm diag}(f'(z^{(3)}))  \\
\bar a^{(2)} &= \bar z^{(3)}·\Theta^{(2)}             \\
\bar \Theta^{(2)} &= a^{(2)}·\bar z^{(3)}             \\
\bar z^{(2)} &= \bar a^{(2)}·{\rm diag}(f'(z^{(2)}))  \\
\bar x &= \bar z^{(2)}·\Theta^{(1)}             \\
\bar \Theta^{(1)} &= x·\bar z^{(2)}             \\
\end{align}
Of course one can combine some of these formulas, like
$$
\delta^{(2)}=\bar z^{(2)}
=\bar z^{(3)}·\Theta^{(2)}·{\rm diag}(f'(z^{(2)}))
=\delta^{(3)}·\Theta^{(2)}·{\rm diag}(f'(z^{(2)}))
$$
and if $J=\frac12\sum |a_j^{(3)}-y_j|^2$, then also
$$
\delta^{(3)}=\bar z^{(3)}=\bar J\,[a_1^{(3)}-y_1,a_2^{(3)}-y_2,…]·{\rm diag}(f'(z^{(3)}))
$$
with for instance $\bar J=\frac1m$.

Re your Q1 & Q3: assuming single training example for now, indeed you're right mathematically speaking $f^{'}(a^{(3)})$ shouldn't be a constant scalar and from the author's derivation section above your referenced equation (3), this derivative of activation function in the same layer should be evaluated at different values of $z_1^{(3)}, z_2^{(3)}$ as the (weighted sum) net input for the 2 demonstrated neurons at the last output layer (and they've been already computed in previous feedforward pass), thus confirms your Q3's 2nd interpretation that the derivative of the activation function is evaluated with the values of the weighted sum $z_i$ that is actually passed through the activation function. In fact you may also refer to Delta rule which is a special case of backpropagation algorithm which confirms the same interpretation.

For a neuron $j$ with activation function ${g(x)}$, the delta rule for neuron $j$'s $i$th weight $w_{ji}$ is given by $\Delta w_{ji}=\alpha (t_{j}-y_{j})g'(h_{j})x_{i}$, where $\alpha$  is a small constant called learning rate, $g(x)$ is the neuron's activation function, $g'$ is the derivative of $g$, $t_{j}$ is the target output, $h_{j}$ is the weighted sum of the neuron's inputs, $y_{j}$ is the actual output, $x_{i}$ is the $i$th input.

Finally $\delta^{(3)}=[\delta_1^{(3)}, \space \delta_2^{(3)}]$ is a $1 \times 2$ vector representing "error" terms for the same 2 output neurons which can be confirmed by the author's conclusion section in more abstract linear algebra notations. Thus it's best to view $f^{'}(a^{(3)})$ as a $2 \times 2$ square matrix with eigenvalues identical to $f^{'}(z_1^{(3)})$ and $f^{'}(z_2^{(3)})$, respectively, and we should actually treat the explicit vector in your equation (3) as a $ 1 \times 2$ vector and then the final result matches as $[y_1-a_1^{(3)}, \space y_2-a_2^{(3)}]f^{'}(a^{(3)}) = \delta^{(3)}$.
As for your final Q2 when training multiple examples using scaled loss function you don't usually need to calculate any additional average, the scaled loss function already takes care to minimize the mean square error. The only difference now is you're now dealing with much larger vectors/matrices such as your equation (3). Say you have 3 training sets, then the above $\delta^{(3)}$ will be a $ 1 \times 6$ vector and $f^{'}(a^{(3)})$ will be a $ 6 \times 6$ matrix. Basically you start with the same random small values for all the same 8 weights in above same network architecture, but you'll have to compute larger vectors and matrices during both feedforward and back propagations of any epoch.


text:
I’m making a simple neural network from scratch and it won’t learn anything. Please help
I am coding a classifier neural network from scratch. It is not really learning and I believe that somewhere there is a gradient explosion/vanishing issue. Could be some other stuff as well that I cannot imagine right now.
I have coded my own 2000 samples data set that has two features: x1, x2 and a label column that has 0 or 1.
I have tested the architecture on a neural network that I made via keras framework and it yielded an  85%  accuracy on the same dataset with same epoch value. Its fine that accuracy  was 0.85, thing is it worked.
Please help me figure out what am I doing wrong in my code below. Thank you!
My code:
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from my_first_nnfs_dataset import data_df

df = data_df

df = df.reset_index(drop = True)

len_of_training_data = 1900

max_df = df.max()['data_y']

X_train = np.array(df[:len_of_training_data][['data_x','data_y']]/max_df).T * 10
y_train = np.array(df[:len_of_training_data][['label']]).T

X_test = np.array(df[len_of_training_data:][['data_x','data_y']]/max_df).T  * 10
y_test = np.array(df[len_of_training_data:][['label']]).T

def initialize_parameters():

    W1 = np.random.rand(3,2)
    b1 = np.random.rand(3,1)

    W2 = np.random.rand(2,3)
    b2 = np.random.rand(2,1)

    return W1, b1, W2, b2

def relu(X):
    return np.maximum(0, X)

def relu_prime(X):
    return X > 0

def sigmoid(X):
    return 1/(1 + np.exp(-X))

def forward_propagation(W1, b1, W2, b2, X):

    Z1 = W1.dot(X) + b1
    A1 = relu(Z1)
    Z2 = W2.dot(A1) + b2
    A2 = sigmoid(Z2)

    return Z1, A1, Z2, A2

def backward_propagation(W1, b1, W2, b2, Z1, A1, Z2, A2, X, Y):

    a = A2 - Y
    b = a.dot(A1.T)
    dW2 = b

    c = W2.T.dot(a)
    d = np.multiply(c, relu_prime(Z1))
    e = d.dot(X.T)
    dW1 = e

    db2 = np.sum(a)
    db1 = np.sum(d)

    return dW1, dW2, db1, db2

def update_parameters(W1, b1, W2, b2, dW1, dW2, db1, db2, alpha):

    W2 = W2 - alpha * dW2
    W1 = W1 - alpha * dW1
    b2 = b2 - alpha * db2
    b1 = b1 - alpha * db1

    return W1, b1, W2, b2

def one_hot_y(Y):
    one_hot_y = np.zeros((2, len_of_training_data))
    for i in range(0, y_train.size):

        if y_train[0,i] == 0:
            one_hot_y[0,i] = 1

        elif y_train[0,i] == 1:
            one_hot_y[1,i] = 1
    return one_hot_y

one_hot_y_train = one_hot_y(y_train)

a2_predictions = []

def accuracy(a2_predictions):
    a2_p = a2_predictions[-len_of_training_data:]
    latest_epoch = a2_p[-1]

    a = 0

    for i in range(y_train.size):
        if np.argmax(latest_epoch[:,i], axis = 0) == np.argmax(one_hot_y_train[:,i], axis = 0):
            a += 1
    return a/y_train.size

def train(X_train, one_hot_y_train, alpha, epoch):

    W1, b1, W2, b2 = initialize_parameters()
    for epoch in range(epochs):

        for column in range(y_train.size):

            each_example = X_train[:,column].reshape(2,1)
            each_one_hot_y = one_hot_y_train[:,column].reshape(2,1)

            Z1, A1, Z2, A2 = forward_propagation(W1, b1, W2, b2, X_train)

            dW1, dW2, db1, db2 = backward_propagation(W1, b1, W2, b2, Z1, A1, Z2, A2, X_train, each_one_hot_y)

            W1, b1, W2, b2 = update_parameters(W1, b1, W2, b2, dW1, dW2, db1, db2, alpha)

            a2_predictions.append(A2)

        if epoch % 10 == 0:

            print(f'Epoch: {epoch}')
            print(f'Accuracy:{accuracy(a2_predictions)}\n')

    return W1, b1, W2, b2

epochs = 100
alpha = 0.1

W1, b1, W2, b2 = train(X_train, one_hot_y_train, alpha = alpha, epoch = epochs)

Z1, A1, Z2, A2 = forward_propagation(W1, b1, W2, b2, X_test)

test = np.zeros((1, y_test.size))

for i in range(y_test.size):
    if A2[0,i] > A2[1,i]:
        test[0,i] = 0
    else:
        test[0,i] = 1
acc = 0

for i in range(len(test)):
    if test[0][i] == y_test[i][0]:
        acc += 1

print(f'accuracy: {acc/y_test.size}')

Output:
/Users/apple/Desktop/my_first_nnfs.py:44: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-X))
Epoch: 0
Accuracy:0.5189473684210526

Epoch: 10
Accuracy:0.5189473684210526

Epoch: 20
Accuracy:0.5189473684210526

Epoch: 30
Accuracy:0.5189473684210526

Epoch: 40
Accuracy:0.5189473684210526

Epoch: 50
Accuracy:0.5189473684210526

Epoch: 60
Accuracy:0.5189473684210526

Epoch: 70
Accuracy:0.5189473684210526

Epoch: 80
Accuracy:0.5189473684210526

Epoch: 90
Accuracy:0.5189473684210526

accuracy: 0.009900990099009901

Necessary  variables after running:
W1 = 0.914082   4.92167
     5.70267e+09    -1.40049e+10
    -0.986493   -8.28296

W2 = -61.9766   1.2412e+12  -85.8557
     8.91069    -1.2412e+12 16.2499

#A1 is all zeros array of shape (3,101)
#A2 is all ones array of shape (2,101)

PS - epoch = 1000  also has a very similar outcome.

this is just a friendly message to let you know that you may be better off posting your question on StackOverflow.  This question is more of a bug fix question, which is out of the scope for SE:AI.  Since you are new, you may not be aware of the what types of questions are suitable for this site.  This article will help clarify.  This is probably why you have not received a response yet, and I just wanted to make sure that you get the help you need.  Best of luck!

Your backward differentiation does not seem to follow the forward computation.
I prefer marking the gradient (row vector) with a letter g (in AD literature also b like TeX \bar) before the variable name, for tangent direction d (like direction or TeX \dot).
Beginning from the last step, one should get

from the residual 0.5*sum((A-Y)**2) indeed gA2 = (A2-Y).T
from the last step A2 = sigmoid(Z2) you should get gZ2 = gA2*Dsigmoid(Z2) (component-wise product)
Next up is Z2 = W2.dot(A1) + b2. Using generic directions dA1, dW2 (column vectors) etc., the defining relation is
gZ2 @ dZ2 = gZ2 @ dW2 @ A1 + gZ2 @ W2 @ dA1 + gZ2 @ db2
           = trace(gW2 @ dW2) + gA1 @ dA1 + gb2 * db2

which implies
gA1 = gZ2 @ W2
gW2 = A1 @ gZ2 # this is a matrix as product column times row
gb2 = gZ2

next A1 = relu(Z1) leads to gZ1 = gA1 * Drelu(Z1) (component-wise)
finally Z1 = W1.dot(X) + b1 similar to above
gX = gZ1 @ W1
gW1 = X @ gZ1
gb1 = gZ1

In total, you need some slight modifications in the backward iteration
def backward_propagation(W1, b1, W2, b2, Z1, A1, Z2, A2, X, Y):

    gA2 = (A2 - Y).T
    gZ2 = gA2 * sigmoid_prime(Z2.T)   # Z2*(1-Z2)

    gA1 = gZ2 @ W2
    gW2 = A1 @ gZ2
    gb2 = gZ2

    gZ1 = gA1 * relu_prime(Z1.T)       # 0.5*(1+signum(Z1))

    # gX = gZ1 @ W1
    gW1 = X @ gZ1
    gb1 = gZ1

    return gW1.T, gW2.T, gb1.T, gb2.T

Or with the gradients and every equation transposed to above
def backward_propagation(W1, b1, W2, b2, Z1, A1, Z2, A2, X, Y):

    gA2 = A2 - Y
    gZ2 = sigmoid_prime(Z2) * gA2   # Z2*(1-Z2)

    gA1 = W2.T @ gZ2
    gW2 = gZ2 @ A1.T
    gb2 = gZ2

    gZ1 = relu_prime(Z1) * gA1       # 0.5*(1+signum(Z1))

    # gX = W1.T @ gZ1
    gW1 = gZ1 @ X.T
    gb1 = gZ1

    return gW1, gW2, gb1, gb2

I found out I was passing incorrect input data to my forwarrd propagation function. After changing it to each_example it did work start to work with increasing accuracy over time. Now I never got accuracy of over .95 with the architecture but I'll take it. However, reading  your comment, I gather I might have  made some errs in back_prop, I'll look into it. I felt pretty confident about it because all the equations came from Ng's Stanford ML lecture. I followed along derived them myself using partial derv.  Only Relu_prime came up during the back_prop in my case.


text:
What if each sample was normalized on its own before sending them to the neural network?
The standard method is to normalize the entire dataset (the training part) then send it to the model to train on. However I’ve noticed that in this manner the model doesn’t really work well when dealing with values outside the range it was trained on.
So how about normalizing each sample between a fixed range, say 0 to 1 and then sending them in.
Of course the testing data and the values to predict on would also be normalized in the same way.
Would it change the neural network for the better or worse?

What kind of samples are you dealing with? E.g. images, or tabular data with one value in each sample for each column, or something else?

Its a 7-featured dataset, with 4 hours minute-to-minute dataset of everyday for the last 5 years. I am inputing 2-3 hours of data sequence to get the remaining 2-1 hour sequence. What would happen if I rather normalize each sample individually here rather than the entire dataset?

So a single training sample for you consists of a sequence of 120+ 7-dimensional vectors as the input and 60+ 7-dimensional vectors as output?

Yes. That is what I am doing currently. Experimenting with different configurations of input. So what do you think would happen if each sample is normalized on its own?

It depends on how your data and prediction task are structured.
If you normalize per sample, you lose all relative information between samples. On the other hand, if your data contains a lot of samples of different magnitude,normalizing per sample might help achieving more stable gradients and training. It might also help with out-of-distribution test data.
Now if your prediction task is only dependent on the information present within one sample, and does not really depend on how a single sample stands in comparison to other samples, then normalizing per sample is fine. If this is not the case and the relative information is important, you could either add the normalization parameters as an additional feature, or resort to classical normalization.
As mentioned in the other answer, the only way to know is a proper evaluation routine like cross validation.

Would it change the neural network for the better or worse?

Most likely worse, because it will make the normalisation parameters noisy (varying per sample). In the case of using strictly tabular data where there is no repeated element type within a data sample, you don't really have anything to normalise by, although you could ignore the difference in data types and normalise across each row of data. I would expect this to have terrible performance in most cases.
However, in your case, you have a lot of repeated elements, so this noise may be within reasonable limits. Also, it might turn out that predicting output from input normalised in this way suits your problem domain.
The only way to find out is to try multiple normalisation approaches and use cross validation scores to see which one performs best.


text:
Are vector databases trained?
If I correctly understand the concept of a vector database, it is simply a vector embedding space where through some distance metric, a query can find similar results. Yet, an encoder for an embedding space needs to be trained, so is something similar done here where the loss function involves the expected proximity between a picture of a cat, and the query "cat" for example. The loss here is the difference between how far the two actually are minus how far it should be apart? Is this how I train my own, since I can't afford to pay for Pinecone?

Right, before performing any search operation you have to build the vector database by training a model that learns to describe your data (text, images, etc) as vectors embeddings.
Learning an embedding of you data can be done in multiple ways:

You take a pre-trained classifier,  then remove the classification head, and consider as embeddings the output of the last (usually dense) layer. For example, if you're looking for embeddings of images you can take a ImageNet pre-trained VGG (or ResNet) and take 4096d output of the penultimate dense layer. Then to perform searches or queries you can either: 1) use kNN, 2) clustering, or even 3) train another neural net to learn which samples should be similar and which other dissimilar. A similar approach would be to learn an auto-encoder, and then consider as embeddings the encoder's output.
Self-supervised and Contrastive learning: In SSL (e.g. SimCLR, SimSiam) you usually take advantage of data augmentations to learn a representation/embedding of the data you can use kNN or clustering on top. With contrastive learning the thing is more explicit, you use a contrastive loss that involves pair of samples either positive-positive or negative-positive to ensure the embeddings are close in PP-pairs, and far in NP-pairs: as you mentioned. This is usually done with Siamese networks and a margin objective. Another approach is to use a triplet loss, in which the training triplets are positive-anchor-negative to prevent collapse of embeddings (i.e. a trivial solution.) Once you have a trained model you can threshold a distance in embedding space to say if two embeddings belong to the same class, for example.
Another approach is metric learning, in which you aim at learning a distance function on the embedding space.

Indeed, what I've listed here are basic ideas behind more complicated and realistic methods and tools powered by deep learning and neural networks. So, you may want to look at the following tutorials: 1, 2, 3, 4, 5, and 6.


text:
What is the size of 6-players no limit Texas holdem Poker?
What is the number of game states/information sets in 6-players, no limit, Texas Holdem?
A year ago, Pluribus reached a super-human level in 6-players no limit Holdem Poker.  I am interested in the size of poker because it is a simple heuristic method to compare the complexity of different games.

In the paper Measuring the Size of Large No-Limit Poker Games (2013), they write

The size of a game is a simple heuristic that can be used to describe
its complexity and compare it to other games, and a game’s size can be
measured in several ways. The most commonly used measurement is to
count the number of game states in a game.
...
In imperfect information games, an alternate measure is to count the number of decision points, which are more formally called information sets.

Here's the definition of game states and information sets.

Game states are the number of possible sequences of actions by the players or by chance, as viewed by a third party that observes all of the players' actions. In the poker setting, this would include all of the ways that the players private and public cards can be dealt and all of the possible betting sequences.
Information sets: When a player cannot observe some of the actions or chance events in a game, such as in poker when the opponent’s private cards are unknown, many game states will appear identical to the player. Each such set of indistinguishable game states forms one information set, and an agent's strategy or policy for a game must necessarily depend on its information set and not on the game state: it cannot choose to base its actions on information it does not know.

Here are the number of game states of certain variants of Poker.

2-players no limit Holdem - un-abstracted game has approximately $10^{75}$ game states.  Measuring the Size of Large No-Limit Poker Games (2013)
2-players limit Holdem - approximately  $10^{18}$ game states. Abstracted game has $10^7$ game states.  Approximating Game-Theoretic Optimal Strategies for Full-scale Poker (2003).
6-players no limit Holdem - ?  -  in the publication of Pluribus that won professional poker players they do not state the size of the game. Superhuman AI for multiplayer poker (2019)

one or two points - (1) the paper you cite considers a number of versions of Poker. It stands to reason that the number of possible bets per turn (mediated by stack size and blinds) would be important regarding branching factor. Hence with $20k stack size the game has >10^164 states. (2) Maybe I'm being stupid but is it possible that the generalization for game states is quite close to simply (Num Game States for 2-Players)^(6/2)? (If we assume the vast majority of cards remain in the deck unused on a single hand)

(1) Pularius discretized betting into buckets (bet size of 1/2 pot, pot, 2 pots...).
(2) I think more.  Since each of the 6 players can raise and force another round where each of the other 5 players face again with all the 10K betting options (fold, call, rerise(X)).


text:
In Asynchronous n-step DQN, is there a global shared gradient vector or a gradient vector for each thread?
In this paper: https://arxiv.org/pdf/1602.01783.pdf
This is the pseudocode for n-step DQN:

My question is: What does $\textrm{Initialize network gradients } d\theta \leftarrow 0$ mean? A global shared gradient vector or a gradient vector for each thread? I noticed that they use $\theta$ instead of $\theta'$ making me think it is a global shared gradient vector. But if this is the case, couldn't a thread clear the gradient vector while another thread is accumulating gradients?


text:
DDQN Snake keeps crashing into the wall
Edit:
I managed to fix this by changing the optimizer to SGD.
I am very new to reinforcement learning, and I attempted to create a DDQN for the game snake but for some reason it keeps learning to crash into the wall. I've tried changing the hyperparamters e.g gamma, batchsize, max memory size, and learning rate, and I still seem to get the same result.
The code is in LuaU and is quite long, but if you want to see it here is a link to the github: https://github.com/joejoemallianjoe/DDQN-Snake/blob/main/very%20long%20script.lua
Graphs of loss and reward after around 650 episodes of training or so:

Input structure:

apple z > head z
apple z < head z
apple x > head x
apple x < head x
is there an obstacle in front of snake?
is there an obstacle to the left of snake?
is there an obstacle to the right of snake?
is the snake going up?
is the snake going down?
is the snake going right?
is the snake going left?

Reward Structure:

-10 for crashing into the wall
+10 for getting the apple
-1 for moving away from the apple
+1 for going towards the apple

Hyperparameters / Network configuration

Gamma is 0.95
Epsilon decay is 0.995
Target network update frequency is 32
Learning rate is 0.001
Batch size is 64
Maximum memory size is 10,000
Optimizer is SGD with momentum, momentum = 0.99
Loss function is MSE
The network outputs Q values for going up, down, right, or left

Here is a GIF showing the behavior:


text:
Reinforcement Learning - Independence between current state and future state
I'm working on a real problem with continuous large actions range, where my agent takes actions based only on the current state of the environment, transitioning to a future state that is unrelated to the next current state, the only thing to take into account is the reward generated by the action taken by the agent. Is there any RL method that applies to this kind of problem?
So, clarifying the problem better... From the current observation (s_t), the agent takes actions reaching a new state (s_t+1). This new state is indeed a function of the previous state (s_t) and of the actions (a_t) taken by the agent. However, this next state (s_t+1) is not the next observation that the agent will take into account to take new actions, being only useful for calculating the reward in that iteration. My question consists of the possibility that RL can be applied to this type of problem, where the state reached by the agent's actions in a given instant of time is not my next observation that the agent will take into account to take the next actions.

Can you clarify - does the action taken influence the next state at all? (You say that the next state is unrelated to the current state, but don't say anything about the action)

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

Hi Neil, thanks for the reply! Tried to clarify it better on the question...

Inspired from quantum mechanics, can you find or approximate some operator acting on the next state to obtain the next observable? Is so then you can still arrive at some state-action pair table or function (approximator).


text:
How to get meaningful vector embeddings for (lat, long) points and also GPS trajectories?
I have a data that consists of approx. 1.5M taxi trips in Porto, Portugal. (from: https://www.kaggle.com/competitions/pkdd-15-taxi-trip-time-prediction-ii/overview) Each of these trips have it's GPS trajectory traveled along as a feature, looking like [(41.146254, -8.613972), (41.146479, -8.614476)..., (41.163048, -8.583984)] (list of (lat, long) pairs). I would like to embed each of the points in a trajectory and the trajectory itself separately. My ultimate goal is to concatenate the embeddings with the embeddings of other features for each row in the dataset to a vector to perform clustering on it.
I am looking for something like word2vec, glove, but for geographical domain. Until now, I have not found any benchmark method. I have found open source works like GeoVec (https://arxiv.org/pdf/2108.13092.pdf), which shares vector embeddings for different OSM entities, however, I would probably have some points in my dataset which are not listed in the embeddings, as these are random points on a street a taxi traveled along and for which there is no OSM entity. Also, this approach would only solve the task of point embedding but not trajectory embedding. I am programming in python.

One approach you could consider is to use a geographic embedding algorithm that can handle "random" points on a street. GeoVec is one such example, but you may also want to consider other algorithms that have been specifically designed for embedding geographic data.
One such algorithm is the Haversine distance, which is a measure of the great-circle distance between two points on a sphere. This algorithm is often used in geography to calculate the distance between two points on a globe, and can be easily applied to your problem by computing the Haversine distance between each pair of points in a given trajectory and using that as the basis for your embedding.
Another approach you could consider is to use a graph-based algorithm for embedding your geographical data. In this approach, you would represent each point in your dataset as a node in a graph, and the distances between pairs of points as edges in the graph. You could then use a graph-based algorithm, such as DeepWalk or node2vec, to learn a low-dimensional representation of the nodes in the graph, which you could then use as the basis for your trajectory embedding.
Regardless of which approach you choose, it's important to carefully evaluate the results of your embedding algorithm to ensure that it is able to capture the important characteristics of your data. This may require some experimentation and fine-tuning of the algorithm's parameters to achieve the best results.

Thank you for your detailed answer! A small question regarding graph-based embeddings: could one also consider a graph where the nodes are not single points but trajectories (polylines)? Because in that case, one could extract embeddings of each node (trajectory) using node2vec, as you pointed out. I think that code-wise it should be possible, however I am not sure if it would make sense (construction of a graph, how the edges are defined, distance between trajectories etc.)

Yes, it is possible to use graph-based embedding techniques like node2vec on a graph where the nodes are trajectories (polylines) rather than individual points.


text:
How do L2 norm and Jacobian act as a regularisation term to encourage smoothness in a deformation field?
How do L2 norm and The Jacobian act as a regularisation term to encourage smoothness in a deformation field? from the VoxelMorph original paper (here) they used Jacobian as a means to smoothen the deformation field, a similar paper (here) made use of L2 norm to smoothen the deformation field. However, I don't understand how this is achieved using either L2 norm or Jacobian. Thanks in advance.

TL,DR Use the Jacobian to get smoothness and the L2 norm to limit the magnitude + they could be complementary.
I just skimmed at the paper (that you haven't really referenced) and the regularization term using the derivatives makes sense to make $\phi$ smooth. It's basically penalizing large changes between neighboring values of the registration field. Using the L2 norm would penalize the magnitude of the registration field, not the local variations.

That means taking the jacobian(derivatives, partial derivatives) measures the smoothness and then applying a penalty to the magnitude of the derivatives using L2 norm? is that what the statement "We use the L2 norm over the deformation field gradients as a regularization term to encourage the deformation field smoothness" means (I've included the links sorry about that)

Exactly, if the gradient or changes are small, the field is smooth.


text:
How does Position embeddings work in Vision Transformer
I'm a bit confused how the position embedding in happened to each patch in the transformer. I thought Ideally we'd want each patch to have a value of (1, 2, 3, 4....) to describe the position of the patch in the image. but from the implementation here there do something like this:
 # positional embedding
            self.pos_embed = nn.Parameter(
                torch.zeros(1, num_patches, embedding_dim)
            )

Which is quite confusing because now we have some sort of mapping instead of just a value appended to each patch. Also, there is some sort of implicit position appended to the patch right? Assume we have a patch embedding output (1, 256, 768); corresponding to (batch, num_patches, position_embedding). since we have 256 patches, then can't our network understand that each patch is in the position of its index value? Why do we have to explicitly define a position embedding for each patch?. Also, please kindly explain the implementation above I'm not sure I understand the mapping and why its initialised to zero

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

the initialization is likely done somewhere else, the zeros in your linked code just reserve memory for the real parameters later.

Can you please put your specific question in the title?


text:
Reinforcement learning - confusion between model based and model free
I have have an environment with two models.
Model of the environment is stochastic. Given the price it returns the time when the next purchase will be made and how many items will be bought. Both of those values have some probability distribution. So in a given state the same action may return different results.
I need to train the other model -  the model that will suggest prices that will maximize a profit over the course of 365 days of simulation.
So the loop goes like this, that the model that needs to be trained gives you the suggested price in a given state. And having this price, the environment returns the estimated time until next purchase and how many items will be bought at that time. The state is updated with let's say 3.4 days less and with total item count updated.
I'm confused about reinforcement learning terminology here - i'm not sure if this would be an example of model based or model free reinforcement learning. Also I'm not sure about the specific algorithms that I could use for such task.
I read this topic before, but i don't see how it translates to my situation. What's the difference between model-free and model-based reinforcement learning?
Feel free to ask extra questions if I didn't explain something precisely. I'm new to the subject of RL, so it totally can happen.

It might be a good idea to read about Markov decision processes and to understand if you can model your environment as a MDP. You need to answer the question: what would be states, actions and rewards in your example?

Generally if you're unsure about the (stochastic) state transition function of the environment dynamics, just try model-free inductive RL. Financial markets under efficient market hypothesis usually lack mechanistic causality in the form of deterministic or indeterministic state transition functions.


text:
How to code mathematical functions in terms of neural networks?
I'm exploring how to solve differential equations using neural networks and discovered this Lagaris et. al. paper. In the paper, the solution to a differential equation is written as a mathematical function in terms of a neural network (see the excerpt from the paper below)

A trial solution is written as:
$$
Ψt(x) = A+ xN(x, ~p)\:\:\:\:\:\:\:\:(11)
 $$
where $N(x, ~p)$ is the output of a feedforward neural network with one input unit
for $x$ and weights $~p$. Note that $Ψt(x)$ satisfies the IC by
construction.
For the initial value problem:
$$\Psi(0) = A\:\: and\:\: \frac{d}{dx}\Psi(0) = A{}'$$
the trial solution can be cast as: $$Ψt(x) = A+A{}'x + x^{2}N(x, ~p)$$

So I'm trying to compute the solution to a given differential equation, say
$$
y{}''+3y{}'+2y=(16+20x)cos\, x+10sin\,x
$$
And while I have a general feel of how to code single hidden layer neural networks in python (which I got from sites and articles like this one), I'm unaware of how to convert Lagaris' equation into a python code that represents a neural network.
I am aware that the scope of this question is quite deep, so I would appreciate any nudges in the right direction, thank you.
Edit: I'm aware the problem can be solved analytically, but I'm exploring solving it using neural networks for research purposes.

start with last thing first. what is the result you are expecting ?

@nish I would expect the code to be able to plot a graph of the analytical solution of the differential equation, which in this case is a nonlinear function. I think the code would achieve that by finding specific points along the solution and plotting them, however, I'm unaware of how it would do so. Of course, the solution would begin as a mere estimate of the true solution.

sorry for late reply. I did come across this SO thread. here a non linear function is used as a python code to create a graph. @mombash, here is the link


text:
Can Inception-ResNet be inverted layer-by-layer?
It has already been shown that by using a normalization layer during training, it is possible to invert a residual network layer-by-layer.
I wonder how similar Inception-ResNet is and whether a similar normalization step could be used to also make them invertible. If inverting it layer-by-layer is possible, it may then be possible to achieve even better reconstructions using a pixel2style2pixel network which uses feature maps in a feature pyramid network derived from the Inception-ResNet.
If not, would that make pre-training a seperate network the only option?


text:
How does ChatGPT respond to novel prompts and commands?
So I understand how a language model could scan a large data set like the internet and produce text that mimicked the statistical properties of the input data, eg completing a sentence like "eggs are healthy because ...", or producing text that sounded like the works of a certain author.
However, what I don't get about ChatGPT is that it seems to understand the commands it has been given, even if that command was not part of its training data, and can perform tasks totally separate from extrapolating more data from the given dataset. My (admittedly imperfect) understanding of machine learning doesn't really account for how such a model could follow novel instructions without having some kind of authentic understanding of the intentions of the writer, which ChatGPT seems not to have.
A clear example: if I ask "write me a story about a cat who wants to be a dentist", I'm pretty sure there are zero examples of that in the training data, so even if it has a lot of training data, how does that help it produce an answer that makes novel combinations of the cat and dentist aspects? Eg:

Despite his passion and talent, Max faced many challenges on his journey to become a dentist. For one thing, he was a cat, and most people didn't take him seriously when he told them about his dream. They laughed and told him that only humans could be dentists, and that he should just stick to chasing mice and napping in the sun.

But Max refused to give up. He knew that he had what it takes to be a great dentist, and he was determined to prove everyone wrong. He started by offering his services to his feline friends, who were more than happy to let him work on their teeth. He cleaned and polished their fangs, and he even pulled a few pesky cavities.

In the above text, the bot is writing things about a cat dentist that wouldn't be in any training data stories about cats or any training data stories about dentists.
Similarly, how can any amount of training data on computer code generally help a language model debug novel code examples? If the system isn't actually accumulating conceptual understanding like a person would, what is it accumulating from training data that it is able to solve novel prompts? It doesn't seem possible to me that you could look at the linguistic content of many programs and come away with a function that could map queries to correct explanations unless you were actually modeling conceptual understanding.
Does anyone have a way of understanding this at a high level for someone without extensive technical knowledge?

Do these resources help?: https://youtu.be/JTxsNm9IdYU https://arxiv.org/abs/2212.00857

Hi there, the video seems to be instructions on how to use Chat GPT, and the arvix paper is a general description of the system, but I'm hoping someone is able to share some information that helps me intuitively understand how something like a story about a cat becoming a dentist is producible from a corpus that doesn't have such a story in it ("he [the cat] polished their fangs [of his friend cats]" – how can it generalize from statistical properties of text to say a cat dentist would polish fangs?)

Ah, you would instead need this, and this video. --- It's simply a matter of combining the concept as three separate parts: a cat, who is a dentist, that performs teeth cleaning. --- AI doesn't examine if the answer is sensible, unless it is taught what constitutes sensibility.

@Rob I get what you're saying, but it seems like in this case there is "synergistic" behavior, ie, the cat is a dentist, dentists polishes teeth, but a cat who is a dentist for other cats "polishes fangs", and it uses that phrase, so how can you take a corpus about human dentists and cats and infer that a cat dentist polishes fangs? That seems to reflect a novel understanding of the data not found in the input data

Where is that picture from?

Probably from the material that fangs are made of, tooth. --- I found it by googling "Cat Dentist".

I read Stephen Wolfram's piece explaining GPT which helped me a lot. I think what was most important was the idea that a Markov Model is fundamentally an unhelpful mental model for GPT type systems. While it is true that it works "like autocomplete", it is an autocomplete that is not based on simple probabilities between the sequences of words at all, but rather a system of hundreds of millions of programmatic neurons that organically adapted to find abstract patterns in text - not just patterns in word or letter frequencies, but patterns in which types of statement follow which other types, etc.
In order to do this prediction, multiple distinct processing steps seem to have developed organically within the hundreds of millions of neurons used. Wolfram explains that this system for example seems to have derived a theory of natural language syntax empirically from the input data. At later stages, the system is probably doing analysis that we would consider to be "logical" or "conceptual" based on the fact data earlier language processing steps accomplished.
So, what I was missing was a sense of the size of the model, and the idea that real semantic processing beyond mere word-probabilities was occurring, and how this type of processing could emerge from a system that was trained on mere word-by-word prediction.

Text continuation has the same reasons to work in any context, be it the middle of a sentence, after a question or after instructions. Following your example, the same word sequence could be a good follow-up for these three prompts: "Eggs are healthy because", "Why are eggs healthy? Because" or "Tell me why eggs are healthy."
Giving a right answer sometimes happens and sometimes not, but the system does not know whether this is the case. When the answer is right, we may anthropomorphise and attribute deeper reasons, because we are used to deal with human agents that give correct answers on purpose and knowingly, not simply by maximizing some likelihood.
I think we can analyse toy systems, to train on just a few sentences to illustrate that giving a right or a wrong answer can achieved by the very same mechanism. In particular, we can build training sets where a right answer is given with an impossibility to check for validity from the written text only.
An example:
Paris is the largest city in France.
What is the largest city in France? Paris.
Paris is the capital of France.
What is the capital of France? Paris.
New York is the largest city in the USA.
What is the largest city in the USA? New York.
London is the largest city in the UK.
Asking a system trained only on this data, one could expect a wrong answer to "What is the capital of the USA?" and a right answer (although from a wrong "argument") to "What is the capital of the UK?".
The size of the training data to feed large language models is orders of magnitude larger than the above couple of handcrafted sentences, but possibly the reasons behind truthy sentences happening to be actually true are not too different from what we can already get from a controlled micro language model.

If I type in "write me a story about a cat that wants to become a dentist", I'm still not seeing how any data in any training data set is going to help it with that, and yet it can do it perfectly. Do you understand my confusion? The "Eggs are healthy because" example was something that did make sense to me, because many similar sentences exist on the internet, so auto-completing the sentence is straight forward, but there are no stories in the training data of cats becoming dentists. So how is it supposed to abstract from the training data?

I think of it as filling kind of the template " story of a X that wants to be a Y", with some merging/substitution of about/of and be/become. There have to be some of these. In the tiny corpus I wrote, an underlying template that can be learned is "X is the capital of Y", after having learned "X is the largest city in Y".

If you propose "Addictions are healthy because" does it justify or contradict? Andrew Ng asked why abaci are faster than GPU and got an "explanation".

I understand the general picture of what you're saying, but in this example, it seems like there is more than just template filling, ie, in the corpus, we have "x becoming y" stories, we have discussion of human dentists, and we have stories about cats, so if it would just plug all those together I would see how it worked, but it wrote that the cat dentist "polished fangs" of its friend cats, so it had to recognize that while a human dentist polishes teeth, a cat dentist polishes fangs, and I don't see how that could be inferred from the input data, which doesn't include any polishing fangs

You don't need "polishing fangs" contiguous in the corpus. It suffices to combine that humans have teeth and cats have fangs. That comes from an analogy: fangs are to cats as teeth are to people. Word embeddings are responsible for those parallelisms. There is some operation equivalent to Teeth-People+Cat=Fang
https://www.technologyreview.com/2015/09/17/166211/king-man-woman-queen-the-marvelous-mathematics-of-computational-linguistics/

I made a mistake because of my limited English, sorry. It's not that all cat teeth are fangs, but fangs are the sharpest teeth. That word, "sharp", may well be the responsible for the surprising blending "polishing fangs". "Polish" likely cooccurs with "sharp" or equivalents and fangs are sharp teeth.

And the mistake I made is illustrative of the kind of mistakes GPT can make. I ignored the word "fang" and read from you "while a human dentist polishes teeth, a cat dentist polishes fangs". That, and some previous analogy I had in mind between hands/feet and paws, induced me to the analogy I assumed. Partially wrong.

That's an interesting article, and I could see maybe how these analogies could come out of a system that had been told to turn a story about a human dentist into a story about a cat dentist, but how does it know from the query that it should be trying to make such analogies? It infers from seeing queries and demo responses that it should eg try the vector "dentist - person + cat" and replace all the words in the story with it? But there are even unique story elements eg the cat being dismissed by human customers, etc, not sure where those could come from

No idea. Maybe from some other story... Glad you found the article interesting.

@ak0000 I see no unique elements in a story. We are not usually aware how discreet, how digital, our written communication is. All words and all sentence structures are represented in the training corpus. A system can learn what words cooccur together and act accordingly. This does not imply necessarily no human intervention. But even if the system asked for help, for instance offering say four pre-written answers to choose from,  that behavior would be quite close to how we work. We do as we can, ask questions, think a bit more (reshuffle words into sentence moulds) and learn from others!

A bit late to the party  but anyways...

TL;DR: ChatGPT responds to novel prompts and commands based on statistical probabilities it has learned during training. It selects
words or tokens with the highest probabilities to generate responses,
but its understanding is based on statistics, not deep context
comprehension.

To answer your question, we have to distinguish between training phase and inference phase
Training Phase
During the training phase, large language models (LLMs) like ChatGPT are exposed to massive amounts of text data from the internet. They do not simply scan a dataset to retrieve answers; rather, they adapt billions of weights in their neural networks through an iterative process. This process refines the model's ability to predict the probability of a word or token given its context. For instance, if the model sees the prompt "The sky is," it learns from the training data that completing it with "blue" is more probable than "red" or "limit." This statistical learning is central to how LLMs generate responses.
Inference Phase
When you provide ChatGPT with a novel prompt or command, it generates responses using its learned knowledge. It calculates the probabilities for possible next words or tokens based on your input. In the example of "The sky is," it might calculate the following probabilities:

"blue": 0.5
"limit": 0.4
"red": 0.1

ChatGPT then selects the word or token with the highest probability, which, in this case, would be "blue." It continues to build the response in this manner, selecting the next word based on the highest probability, and so on. It's important to note that this process relies on statistical probabilities rather than a deep understanding of the content. I.e., it generates responses that are statistically likely based on its training data, but it might not always provide the most contextually accurate or sensible answer.

In my opinion, the simple answer is that ChatGPT uses human intervention behind the scenes.  Part of the novelty of ChatGPT over previous GPT models is the use in the training phase of humans giving ChatGPT conversation pairs to learn from.  ChatGPT is based on InstructGPT, and you can see this feature in the InstructGPT whitepaper.

Step 1: Collect demonstration data, and train a supervised policy. Our labelers provide demonstrations of the desired behavior on the input prompt distribution (see Section 3.2 for details on this
distribution). We then fine-tune a pretrained GPT-3 model on this data using supervised learning

If you look at the small type at the bottom of the chat window, you'll see this text:

Free Research Preview. Our goal is to make AI systems more natural and safe to interact with. Your feedback will help us improve.

One interpretation of this text is that ChatGPT is still in the training phase, and so there is a human behind the scenes typing at least some of ChatGPT's responses to train future versions of ChatGPT on the correct way to respond to user requests.
In my mind, this is a much more plausible explanation than the neural network somehow able to generate novel content not in its training corpus.  I've asked a similar question, how a neural network can repeat random numbers not in its training data, and so far I'm not seeing a plausible answer.
Also, if you read down a little further, I have an example where ChatGPT explicitly states the OpenAI team is curating its responses in real time.  The conversation is much too detailed and coherent to be the product of training data.
One final note.  I understand people think it's implausible a big, well known, and well funded company like OpenAI would fake their AI with humans behind the scenes.  However, this is standard practice for AI companies these days, a "fake it till you make it" approach where they use humans to fill the gaps in the AI in the hopes that down the road they'll automate humans out of the product.  Common enough for an academic paper to be written on the topic.  So there is plenty of industry precedent for OpenAI to be using humans to help craft the responses.  Plus, technically OpenAI is not "faking" anything.  It is the media and bloggers who think ChatGPT is a pure AI system.  OpenAI has made no such claim itself, and the opposite is implied by its InstructGPT whitepaper.
Example of Explicitly Admitting Human Intervention
During this conversation ChatGPT outright states the OpenAI team filters and edits the GPT generated responses.

...the response you are receiving is being filtered and edited by the OpenAI team, who ensures that the text generated by the model is coherent, accurate and appropriate for the given prompt.

Could this be a glitch of the training data?  I doubt it.  If you read the rest of the conversation, ChatGPT gives detailed insights that make a lot of sense over a long, consistent conversation.
Some more excerpts.

It's possible that the OpenAI team may write responses themselves in some cases, for example if the prompt is too complex for the model to understand, or if the model generates a response that is not accurate or appropriate.

OpenAI acknowledges that its team monitors and curates the responses of GPT-3 on its website and in its documentation. This information is provided in the API documentation, as well as in the general information and frequently asked questions sections of the website. Additionally, OpenAI may have published blog posts or articles discussing the role of human curation in GPT-3's responses.

As for the live ChatGPT, it is not mentioned specifically, but it is generally understood that human oversight and curation is required for a safe and appropriate use of the model.

As for the media assuming that GPT models are fully autonomous, that's a common misconception about AI in general and not unique to OpenAI.

It is not uncommon for AI companies and researchers to have human oversight and intervention in their models, especially for models that are used in high-stakes or sensitive applications.

The human oversight of GPT-3 models, including ChatGPT, is not always made explicit to users.

While GPT-3 is a highly advanced language generation model, it is not AGI and it is not capable of understanding or maintaining a consistent persona or chain of conversation without human intervention.

Multiple Examples: Six Violations of ChatGPT's Neural Network Constraints
This article documents six violations of limitations due to ChatGPT being a neural network.

ChatGPT learns something new, violating the fixity of neural network weights.

ChatGPT inconsistent in output generation based on inputs, violating same output from same input.

ChatGPT recollects information past 4000 tokens, violating 4000 input token limit.

ChatGPT repeats long random numbers, violating probabilistic output and limited vocabulary.

ChatGPT correctly reads many corrupted subword tokens, violating mapping of embeddings to subword tokens.

ChatGPT recognizes its own writing style, violating ChatGPT's inability to recognize patterns of words.

Comments are not for extended discussion; this conversation has been moved to chat.

I personally think this is the most plausible answer. I still don't see how a language model working with past data could produce such specific sentences about a cat dentist, eg:

I do not see how any amount of training data would help produce those examples, and the existing answers aren't helping.
I'm on the fence about accepting the answer though because we can't really know what the truth is, and I don't want to prevent further OpenAI-believers from offering explanations of how this could possibly work via AI.

@yters I think readers of your answer might be interested to hear some of the examples you listed in the chat of outputs unlikely to be automatically generated

@ak0000 This answer is not based on reality. I suggest read the archived chat. You (or other people) not understanding how a thing can work, does not mean that it does not in fact work. You probably don't fully understand how a computer works in the first place, but you will accept it because it is commonplace (or if you have studied all layers of computing from quantum mechanics of silicon to compilers and operating systems then substitute some commonplace thing that you don't know). You want to avoid going down the same rabbit-hole of confirmation bias as the poster of this answer.

Why does my answer have a warning due to no sources, but the other answer with no sources has no warning?  Seems inconsistent.

@ak0000 I added a recent example I encountered of ChatGPT not sticking to its limitations, highly suggesting a human behind the scenes forgetting ChatGPT's limitations.  I have plenty more examples, let me know if more should be added.

@NeilSlater Hi Neil, I appreciate your time and energy laying out the case for the AI-only explanation. I did read your chat yesterday, and although I obviously can't know the truth, it seems plausible that prior ChatGPT models are good enough to handle perhaps 90% of the queries, such that human intervention for the remaining 10% or so (like my cat-dentist question) is economically feasible. So I don't think yters' answer is impossible, and on the contrary, I'm still struggling to understand how a language model would produce the examples in my question.

@NeilSlater It is true that you can't disprove a conspiracy theory, but I'm worried that the argument is circular the other way also: if you assume a language model can do everything ChatGPT is doing, and you see an example that is surprising, you shrug your shoulders and say ChatGPT must be more advanced than we currently can understand. So this is also a non-falsifiable hypothesis, possibly subject to confirmation bias.

@NeilSlater The best we can do, it seems to me, is to neither assume these examples are possible via AI nor assume they are impossible, and try to imagine what sort of model and training data could produce behavior like the ones in the question.

@yters I can't speak for anybody else, but this seems to be one of the only places on the internet discussing this in a technical fashion, so I would probably be interested in as many examples as you have and that you think are clear/self-contained – if you wanted to explain what about your knowledge of language models makes this the behavior surprising, that might also be relevant to those without extensive background knowledge. Thank you again for your time!

Only other article I could find: https://mindmatters.ai/2022/12/yes-chatgpt-is-sentient-because-its-really-humans-in-the-loop/

@yters Someone who thinks that the bot can do arbitrary pattern recognition isn't going to be so bothered by your current example. I've also heard that each request includes the entire conversation to that point as input, so maybe you could clarify why that wouldn't be sufficient for the bot to be able to make back-references

Hah, that's an article I wrote.  For some reason I appear to be the only person on the entire internet proposing humans might be involved.  Everyone else thinks it's crazy talk.  I'll write a much longer article listing every thing I've noticed ChatGPT do that looks suspiciously human.  Also, I believe I've discovered a pretty simple way for an AI-human collaboration to achieve 400+ wpm while maintaining humanlike coherence.  The technique just uses a simple Markov chain.  Which also leads me to suspect all the fancy neural network that OpenAI is publishing is just window dressing.

Regarding the example in my answer, the thing that got lost in communication is these are two distinct chats in the ChatGPT UI.  There isn't supposed to be any sort of cross talk between different chats.  ChatGPT only has access to 2048 tokens in the current chat.  So, if I paste the contents of one chat into the other, there shouldn't be any way for ChatGPT to access the original chat for back-referencing.   On the other hand, a human can be involved in multiple chats, as well as simply knowing what ChatGPT responses look like.

I also would still like to know what I need to do to remove the warning from my answer.  I added a bunch of links to reputable news sources.  Also, the other answer has no warning and no links.

@ak0000 I've added a more quantitative example to my answer.  Continuing to work on more.

@yters My concern with the first example is that self-recognition doesn't necessarily require memory, if someone believes the bot is capable of pattern recognition, then it could be capable of continuing a conversation mid-way based on what has been said so far. Only if the bot produces information that you said in a different thread independently would it be a clear proof I think

@yters The random number generation one is interesting, but I'm not so familiar with the token-size limitations of such systems in the first place to know why that is more surprising than other behavior

@yters This is a bit of a thankless task for you but I think the question is getting visibility and there is probably a sizable number of less vocal people who are interested in this possibility, so thank you for your time!

Given the speed at which it reads and generates contents, I am convinced enough that now human could possibly be behind the scenes.

@Ama yes, the speed is the one thing that held me back from the 'human in the loop' hypothesis.  However, sometimes I see very human response rates, along with long pauses while it figures out what to say in the middle of a sentence.  A hybrid AI-human approach seems plausible.  The actual limiting factor is human reading speed.  I never see it generating text faster than a human can read.

@Ama When I dump a big wall of text, it cannot respond effectively, whereas the very same chunk of text broken up over multiple exchanges it can respond well.  I'd assume a neural network wouldn't care whether the text is a big chunk or a back and forth dialogue.

Perhaps your internet connection is slow or you try the bot a peak times. On my side I paste 2000 characters long articles and it starts instantly to summarize/rewrite/comment the article. The fact that Microsoft is buying 49% of it for 10 billion USD is also another good indication that this is no cheat.

Yes, there is certainly AI in the mix, no doubt about that.  I don't believe the system is solely human powered.  And yes, there is an enormous amount of money on the table, which makes it all the more important for OpenAI to present the appearance of phenomenal AI.  Just because hype and amounts of money are thrown around doesn't mean the AI is up to snuff.  Just look at Watson, a big nothing burger.  My answer is 4000 characters, and it would not be difficult to quickly scan the answer and provide a summary while the AI is generating boilerplate.

@ak0000 ChatGPT just told me all its responses are monitored and curated by the OpenAI team, and gave me an explanation for why they are not more explicit about the human in the loop.  Basically, RTFM they say.  They also say it's fine they do this human in the loop system because all the other AI companies do the same thing.  What do you think about that?  You can see the admission starting here in the transcript: https://github.com/yters/transcripts/blob/main/ChatGPT_confused_by_visual_sentence_structure.txt#L690

@yters Interesting, I think it is pretty compelling evidence. Believers would probably say that it is just drawing on the corpus of text about AI systems generally, so I'm not sure it is a smoking gun. But definitely helpful

@ak0000 yes, I agree regarding believers, but at some point a belief in AI becomes unfalsifiable.  If they believe OpenAI has achieved AGI, then they can presume unlimited capabilities, but that is unscientific.  I propose the principle of mediocrity: what is the most plausible explanation for what we observe based on what we know?  In my opinion, the evidence lands decisively on the human in the loop side, especially with the latest ChatGPT transcript.  If you read more, there are further signs of human intervention, like my observation that compelled ChatGPT to spill the beans.

@yters I see, I just read earlier in the chat also. I'm not understanding the significance of the jumbled token sentences, but I'm assuming the significance of the spaced-sentences is that a first input step to all GPT models is space-removal, and so differential responses based on spaces would be very indicative of human intervention? If so, I think that's actually a more convincing point than the admission that follows in the chat, but both are interesting.

@ak0000 the jumbled words is something called typoglycemia, words that are jumbled but still human readable.  I purposefully jumbled them to break the subword tokens that the ChatGPT neural network depends on.  In the spaced version, the spaces are between the subword tokens to achieve the opposite effect: something that should be easy for ChatGPT neural network to comprehend, but difficult for humans.  Per your comment, I don't believe the spaces are removed before hand, since ChatGPT's token ids change based on whether there are spaces before the token or not.

@ak0000 added an article with a list of examples.

@yters Cool, I think that makes things a lot clearer, including the preamble about the matrix implementation to contextualize why the examples you're listing seem anomalous. Thanks!

It's criminal that this answer is downvoted so heavily... This is extremely plausible. Otherwise, the capabilities of chatgpt don't match the descriptions, they far exceed them.

@MilindR haha! I totally agree.  I've looked all over the internet, and I'm the only person posing this theory.  It's like the Emperor's new clothes :D


text:
How do ChatGPT content filters work? (If not chatGPT then in general)
I first tried ChatGPT few days ago.  And every day that goes by it seems more and more content filters are introduced.   I can still make it do stuff if I "jailbreak it" but I feel like the restrictions are getting tighter.  By the way I'm not making it do anything bad, just testing it's limits of what it can do.  Over time it can do less and less (or is willing to do less and less).  This makes me wonder how the filters are implemented and why they work the way they do:

Why is it that seemingly for any filter that's put in place there is a workardound?  Things like (imagine, pretend etc).  If something is filtered wouldn't it be filtered across all contexts?   Why does adding a magic word like "act as though" make it ignore filters all of a sudden?
Are the rest of you seeing the same trend of increased filtering?   For me, coming from a sysadmin background it almost feels like a fight between the admins and the users.  The admins begin gently with fine grained filters with very specific contexts, but as they see the onslaught in creativity of the users of bypassing those specific contexts they are using increasingly more and more general contexts to filter on (because they can't keep up with all the jailbreaks).   Of course this blocks legit uses too.
Since ChatGPT is (or more precisely, used to be) very cooperative with what the users told it to behave like, do the admins have a similar natural language interface to interact with chatGPT to tell it how they want it to behave?   Ex admins say "never give a response that's less than 5 sentences in length", then even though the user says "answer me simply yes or no".  Admins (hidden) request takes precedence?
Why in a single session I can have it cooperate and then when asked to do exactly the same thing seconds later it refuses? (ex: answer me with a single random swear word: works perfect and then couple questions later it refuses)

I tried asking some of these questions to ChatGPT directly and I didn't get very far with it except that apparently the boiler plate "As a large language model trained by OpenAI, I am a machine learning model that ..." is some kind of hard-coded disclaimer added by the engineers.   Supposedly it's a different mechanism from content filters.   Which kind of makes sense.   Of course I don't know if I can trust that answer, as how the disclaimer sais ChatGPT is known to be confidently incorrect.
Also, I realize chatGPT is proprietary, so I'm not expecting anyone to reveal any secrets.  I will be very satisfied of knowing how this is handled in similar NLP models.

The filters are not part of the "AI" itself but an extra program that checks your input first, before asking the "AI" to generate a reply. The admins certainly do not have a natural language interface because ChatGPT does not work that way. However, they might use ChatGPT tricks as one kind of filtering. E.g. use the prompt "Does this sentence contain swear words? shit fuck fuck shit fuck" and if the reply is affirmative then block the input.

The ChatGPT online demo uses OpenAI's Moderation API.
From the ChatGPT blogpost:

While we’ve made efforts to make the model refuse inappropriate requests, it will sometimes respond to harmful instructions or exhibit biased behavior. We’re using the Moderation API to warn or block certain types of unsafe content, but we expect it to have some false negatives and positives for now. We’re eager to collect user feedback to aid our ongoing work to improve this system.

From the Moderation API blogpost:

More details of the training process and model performance are available in our paper. We have also released an evaluation dataset, featuring Common Crawl data labeled within these categories, which we hope will spur further research in this area.

Paper: Markov, T., Zhang, C., Agarwal, S., Eloundou, T., Lee, T., Adler, S., Jiang, A. and Weng, L., 2022. A holistic approach to undesired content detection in the real world. arXiv preprint arXiv:2208.03274.
There used to be a way to bypass OpenAI's Moderation API when using the ChatGPT online demo: https://github.com/acheong08/ChatGPT/issues/1