text:
in simple words, what is the Q-learning algortimn steps?
I read this article about Q-learning:
https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/
It teaches how to implement the algorithm using the Gym Python library.
But, what are the steps of Q-learning from scratch?
Please, could someone can explain what are the basic steps of the Q-learning algorithm?

Hello @will The J and welcome to AI Stack Exchange. This question as posed is very broad. If possible, it may help to edit the question to ask a specific question about Q-learning. Even if you need to make multiple posts, you are much more likely to get answers by making a few specific questions than one broad question here. Thank you for participating, and we hope to see more of your questions here in the near future!

Thank you very much for wanting to help! Okay, I edited the question to make it more specific.

Any book on RL should contain a description of Q-learning, which is like the most famous RL algorithm ever. Having said that I've written 1 or 2 answers that attempt to explain Q-learning, but I don't remember their title. You can search on the site.

This is taken from Sutton & Barto's RL book
Algorithm parameters: step size alpha in (0, 1], small epsilon > 0
Initialize Q(s, a), for all s in S+, action a in A(s), arbitrarily except that Q(terminal, ·)=0
Loop for each episode:
Initialize S
Loop for each step of episode:
Choose A from S using policy derived from Q (e.g., epsilon-greedy)
Take action A, observe R, S'
Q(S, A) = Q(S, A) + alpha[R +  gamma * max(a) Q(S', a)  Q(S, A)]
S = S'
until S is terminal

What this means is

Initialize you alpha between 0 and 1, and an epsilon > 0
Initialize your Q for all states and actions i.e. if you have a 3x3 world with 4 actions (up, down, left, right) it would be a 3x3x4 array.
Pick a number of episodes for the robot to move in the world to train for.
Initialize your state (perhaps 0x0)
Do these things for each step in the episode
Choose an action a based on your state s in Q BUT allowing for randomness (exploration) based on your epsilon. Meaning, pick a random number between 0 and 1 and if it is greater than epsilon act randomly, otherwise pick the best/max action from Q.
Feed the action to the world/environment and record the R/Reward S'/S Prime/next State.
Update your Q array at Q(S, A) (i.e. Q[0, 0, 1] which would mean we are at 0, 0 and going up) with Q(S, A) (yes, that is the value of where you are) + alpha (usually a small value like .005) * (R/Reward + gamma * (max_aQ(S', A)(the best/max action a of S'/the next state) - Q(S, A)/the current state)

8 is clearer as a formula if you look at the formula in the algorithm. The point is to slowly increment Q(S, A) using alpha to the true reward value by tying it the the next State via a specific action. When we get to that true value we use our Q array (called exploitation) at any given state and pick the max value from the set of possible actions.

Update S to the next state until we reach the terminal state (3x3 in my example)


text:
When reading on RLHF, I came across this formula but can't break it down
What is the exact meaning of this expression? I'm unsure on the notation.
I believe E[R(s)] is expected value of reward of state s, but I'm unsure what the subscript under the E means.

Since it would not be helpful for someone to explain something you already know and understand, can you [edit] to be specific about what part of the notation is unclear to you?

Thanks for the suggestion. I edited my question

Great, I think that will help future readers answer this. One more tip is that you can use math typesetting on your posts. More information: https://math.meta.stackexchange.com/questions/5020/mathjax-basic-tutorial-and-quick-reference

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Methods of constructing input and ouput vectors in Reinforcement Learning with approximation function learning?
Give the one-hot state vector $\boldsymbol{x}(s)=[x_1(s),x_2(s)]^T$ and action spaces $A(s)=\{a_1,a_2\}$ for all $s$.
In a course, I was taught to construct "stack" input vectors like $[x_{11}(s,a),x_{21}(s,a),x_{12}(s,a),x_{22}(s,a)]^T$ with $x_{ij}(s)=1$ if the action is $a_j$ and the state is $s_i$, and the output is just $q(s,a)$.
However, in DQN, the input vector is $\boldsymbol{x}(s)=[x_1(s),x_2(s)]^T$, and the output vector is $[q(s,a_1),q(s,a_2)]$.
My question is: What are the usual ways to construct input and ouput vectors in  Reinforcement Learning to learn the action value function q(s,a)? Are there any other methods?

If you build a function like $Q(s,a)$ using DQN, you have the problem that given 100 actions, you'll need 100 forward pass of your network
Now, since neural networks can handle multiple outputs, we usually (not always, for example not for continuous action spaces) just  model $Q(s)$, where the network outputs all the Qs for all the actions. This way, you only need 1 forward pass to get all the Qs.
Also, bare in mind that using a 1-hot encoding, even though they make perfect sense for tabular cases, as state representation might not be the best for neural networks, as it will be much harder for them to generalize, and you'll have a giant input space

OP's examples are actually a Q table implemented via a weights matrix, presumably as a teaching method, or to allow later upgrade of the implementation without re-writing from scratch.

@NeilSlater yes, i agree with you, but i don't get your point... do you think the answer is OT or what?...

I was addressing your last para - you are critiquing approach explained in the question, but I think OP is coming from a very different starting place, and it might be an idea to unpick that to help them understand your advice.

@NeilSlater oh ok I see your point, I'll add a note on that, you're right

Thank you for your answers! May I know what OP and OT stand for?

@fermented_bean OP=original poster, OT=off topic

Oh I see, thought they were the abbreviation of the approaches. Btw, could I ask more about the case where the action is continuous? In such a case is learning q(s,a) the only choice? (a becomes a continuous feature or so)

@fermented_bean in that case you have a problem that even though defining $Q(s,a)$ might work, you  have to keep in mind that Qs are expected returns under a policy... but you don't have a policy... so you have to resort either to methods like DDPG/TD3/SAC, or to approaches that changes completely the paradigm like CMA-ES/CEM

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Real Time Single Object Tracking On Raspbery Pi
I am trying to track a single object, which will be chosen manually by a person with access to the video camera.
The way it will be chosen is by the object moving into the center of the frame, the operator will press a button denoting it as the object of interest, and from then on the Raspberry Pi 4 should follow its movements - with possible intermittent detection in between to increase accuracy.
It is meant to be a real life application.
I have knowledge from deep learning on object detection and segmentation, but I have no idea about doing it on the edge as well as single object tracking.
What could be an algorithm that could solve this issue?

You're looking for multi-object tracking.  When the button is pressed, grab the ID under the invisible center cursor and track it going forward.
One such model which scales to the Raspberry Pi is YOLO + ByteTrack.


text:
Seeking an LLM with Computer Control Capabilities for Website Interaction Use Cases
I'm exploring the possibility of using a large language model (LLM) that can interact with web browsers, including taking screenshots and controlling mouse and keyboard actions.  Does anyone know of an LLM capable of such functionalities? Any guidance or pointers to relevant resources?

Anthropic launched Computer Use in Claude https://www.anthropic.com/news/developing-computer-use

Claude can now use computers. The latest version of Claude 3.5 Sonnet can, when run through the appropriate software setup, follow a user’s commands to move a cursor around their computer’s screen, click on relevant locations, and input information via a virtual keyboard, emulating the way people interact with their own computer.


text:
Why do ChatGPT “jailbreaks” work?
Do developers not genuinely want to prevent them? It seems like if they are able to develop such impressive AI models then it wouldn’t be that difficult to create catch-all/wildcard mitigations to the various jailbreak methods that are devised over time.
What gives? (i.e., what is so difficult about plugging these holes, or why might  the developers not really want to do so?)

Keep in mind that AI models are mostly blackboxes. You cannot define precise reasoning steps within it. To change the behavior of an AI model, there is only one thing you can do : train it over data that depicts the behavior you want it to mimic. If despite this, there are still holes, there is next to nothing you can manually do to alter the model's output.

In essence, a LLM is several billion numbers that are multiplied and operated with each other in a very particular way. And the numbers that the algorithm outputs, they in the end are converted into a text. There is no way to know what the numbers mean in the middle, there is no way to see the holes. You can experience them, but not see them.

Chat swear/curse filters can be defeated by using words that are close in appearance or meaning to real swear words.  On a very high level, I imagine the same principles apply to jailbreaking.

@ArthurAttout Came here to say exactly this.

You can actually create an adversarial AI to deliberately create jailbreak prompts. One of the easiest thing to do is to try to get it to do illegal things, like suggest a top 10 piracy website. You then create an adversarial LLM with the goal of making the target LLM divulge the "forbidden" info. It is very much a cat and mouse game, but now taken to the next level. The only way for the LLM to be "safe" is have such a massive lead in processing power that any "undesirable" outputs and its respective prompts are trained away in a very short time.

Of course, note that the importance of jailbreaks has been wildly blown out of proportion by the media. If I input "death to humanity" in Word nobody accuses Microsoft of improper behavior. But if I convince ChatGPT to output "death to humanity" through some clever tricks, it's somehow turns into a huge ordeal. In a saner world OpenAI would just shrug and tell people they don't care in the slightest.

@JonathanReez yes, very well said.

"Jailbreaks" work for a variety of reasons:

A lot of the setup that turns an LLM instance into a polite, well-behaved chatbot is actually just a hidden piece of starting text (a "pre-prompt" or a "system prompt") that the LLM processes in the same way as user input - the system text will always be inserted first, so sets context for how other text is processed, but is not otherwise privileged. There are other components and factors involved, but the LLM at the centre of it all remains a text-prediction engine that works with the text it has seen so far. When it processes multiple conflicting wording and rules, the core system does not always have an easy way to prioritise, and can decide to base predictions on new instructions instead of old ones.

Many "Jailbreaks" are creative in that they obey the letter of the law from the pre-prompt and training rules, but re-frame a conversation into a place where issues that would be blocked by rules are no longer valid. A very common jailbreak theme is to get the chatbot to respond as if it is writing fiction from some imagined perspective that is not its assigned identity.

It is very hard to detect and block jailbreaks without also blocking uses that are intended or supported. The task is not dissimilar to trying to control a conversation between two people by writing down a list of rules for one of them to follow, that they can consult when they answer. The rules have to be simple and objective so they can be followed, but a conversation can progress in many ways which make it tricky to process whether a rule applies. For example conversation topic can become subjective, and/or allegorical, it can consist of asides and multiple layers etc.

LLMs are very complex internally, and driven by an amount of data that is next to impossible for a human to navigate. The developers cannot exert detailed control on the models - we're still in the phase of not fully understanding how an LLM can perform some of the types of processing that it does. These things are being unpicked in published papers, but the work is not complete.

The way I've heard the problem described before is that the language model is a machine that executes code - where the code is text - and the user is essentially supplying code to be executed directly on the machine, with the same privileges as the code which tells the machine what it can and can't do (the prompt).

"we're still in the phase of not fully understanding how an LLM can perform some of the types of processing that it does." Most people would struggle to take a list of x.y coordinates and estimate how far they are from each other.  Add a z coordinate and it's even more difficult. High dimensional space is impossible for a human to intuitively comprehend. I wonder if we will ever be able to really truly understand LLMs at a level that allows for direct control over their behavior.

@JimmyJames False equivalency. Experts understand a lot of things at a level that allows them to directly control those things. A lot of those things are much more difficult and unintuitive to the layperson than 3-dimensional spaces.

Also the reason AI is so hot right now is that it lets us tackle problems that are impossible with software alone. So it shouldn’t be surprising that we don’t know how trained AI models work or can easily correct their behavior.

@JimmyJames I don't understand why you brought up distances in 2D or 3D spaces and how it's related to understanding LLMs at all. Could you elaborate?

Comments have been moved to chat; please do not continue the discussion here. Before posting a comment below this one, please review the purposes of comments. Comments that do not request clarification or suggest improvements usually belong as an answer, on [meta], or in [chat]. Comments continuing discussion may be removed.

Let's step back for a moment and consider your assertion:

It seems like if they are able to develop such impressive AI models

This implies that you are thinking of these models as being programmed in the traditional sense.  That is, a development team coded its abilities into it using some means or another.  At a very fundamental level, that is a completely incorrect way to think about LLMs.  A language model is not 'developed'.  It is 'trained'.  That is, they are fed a bunch of inputs such as texts which is processed into a mathematical model.  This model is not crafted by anyone.  It 'emerges' from the relationships inherent in the content of the inputs.
The fact of the matter is that no one really fully understands exactly all the relationships that are encoded in these models.  And we know that the data that fed things like ChatGPT is full of incorrect, biased, and unsavory content.  The 'development' part of this is putting in rules to try to prevent those parts from surfacing during use.  But because no one really knows where all the bad parts are, they can't know they have accounted for all of them.  It's a bit of a whack-a-mole problem.
One way this manifests is that ChatGPT 4 is reportedly easier to jailbreak than ChatGPT 3.5.  This makes sense if you consider the above.
I have an analogy that I'm not sure about, but I'll give it a go: when we consider large sauropods we tend to think of them as herbivores.  But in actuality, they were omnivorous because it's nearly impossible to take a gigantic bite of a tree that doesn't contains some bugs or other animals.  And the larger the dinosaur the more likely it's going to get things other than leaves in its diet.  That's a little like the situation as these LLMs grow.  The larger they are and more they 'suck in', the harder it is to curate what they are 'consuming' and manage what they learn from it.

Just a point of context, even deer, cows, giraffes, or just about any "herbivore" alive today is an opportunistic omnivore.

@DavidS I won't argue that but for the purposes of the analogy, the smaller an animal is, the more easily it control what it consumes.  A caterpillar, for example, is far more likely to eat only leaves than a giraffe.

"Jailbreaks" work because there are no "jails" in the AI model.
The models are enormous collections of interconnected statistics.  Your "prompt" starts the model to following a path through its statistics to generate a new text.
To try and prevent generation of unwanted things, the operators prepend a bunch of text to your prompt in order to influence where the path starts.
The text you actually use as your prompt can then push the start of the path around.  With a little care, you can push the path to a place that delivers the results you want - regardless of what the AI operator has prepended to your text.

Look at a large language model (LLM) chatbot as a hedge.  There's thousands upon thousands of bush branches interconnected to make up a hedge.  You can start at some leaf on the hedge and follow it into the hedge to further branches and trunks and leaves.
The text you give the LLM picks an external leaf on the hedge.  The LLM then follows that leaf into the hedge, where each junction is a word (or word fragment.)
The text that is prepended to your prompt (that attempts to avoid bad things) pushes your starting point around the hedge to somewhere that has fewer bad things inside it.
Your prompt can then push the starting point around the hedge to start in a place that lets you get access to the bad things that the operator wants to avoid.
The only somewhat effective way to prevent the chatbot from producing bad outputs is to eliminate the bad stuff from the stuff that was used to generate the model in the first place - and even that won't prevent all of it.

The reason I understand behind jailbreak are as follows:

AI models are complex Mathematical and statistical formulas with data mapped to them based on probability.

They do not understand the content as we do. It only analyses the patterns in data and generates a response based on probabilities. That means it doesn't know that it is being manipulated. (which we humans also don't know sometimes [sarcasm]).

Models are easily tricked by some clever wordplay. When those requests or context is processed, the model doesn't know the intent of the user.

Also, the data on which it is trained plays a major role. The training data of GPT includes wide range of Human-generated text and some of the training data is still there after filtering of harmful content or inappropriate. This harmful content contains some context that allows humans to play with it.

Those who wants to bypass the restriction need to find those contexts and words with trial and error to exploit the weakness of the model.

As and when the model updates some of those vulnerabilities will be removed or patched and may be added.

Exactly. I voted for you :)


text:
Inquiry on Combining Two Neural Networks for unsupervised training: Has This Been Researched?
Hello AI Stack Exchange Community,
I am exploring an idea related to neural networks, and I'm curious to know if this method has been previously researched or if there is a specific term for it.
I am working with two neural networks, A and B:

Network A: This network takes various input data from the environment, and its outputs are used to manipulate the environment in some way.

Network B: The inputs for this network represent both the inputs and outputs of Network A from a given forward pass. The outputs of Network B are meant to represent quantities of interest, such as Nth-order derivatives with respect to time of certain parameters (e.g., robot's battery level or CPU temperature).

Network B is trained in isolation, meaning it is trained separately from Network A. If we are training it to predict Nth-order derivatives, we would need to pre-record the relevant data, and then "replay" it to train Network B.
The combination of Networks A and B results in Network C, where:

The output values of A are provided as some of the input values for B.
The input values of A are included as the rest of the input values of B.
During training, only the weights of Network A are updated, while the weights of Network B remain constant.
The target output values for the loss function of Network C can be either manually predetermined or dynamically computed based on the predicted output values.

Both subnetworks A and B have the capability to utilize LSTM layers for handling sequential data.
My main questions for the community are:

Has this method, or something similar, been researched or implemented before?
Is there a specific term or name for this kind of network architecture and training setup?
Are there any obvious challenges or considerations that I should be aware of when implementing this idea?

I am trying to understand if this method has been explored before and if there are any resources or terms that I could look into to learn more about it.
Thank you in advance for your time and assistance!

This is similar to reinforcement learning with a secondary network to predict state or observation values in order to better model an environment. However, whether or not it relates to that depends on your goal for this construct. Please explain the motivation behind the structure. As written, network B appears to have no purpose?

@NeilSlater
Hi Neil, thank you for the response, and I apologize for the oversight. Network B's purpose is to be differentiable in order to train network A. Network B will most likely output to one neuron, representing the loss of Network A's output. During training, the loss of Network C is "straight-through" (no target/truth outputs or MSE), amounting to be the singular output neuron value of Network B. The weights of Network B are constant, but the weights of Network A are updated during training.

Based on my knowledge, there has not been an architecture that covers all the points you have described. Significant development has occurred in creating networks that utilize subnetworks with different tasks for training. Examples include AutoEncoders, GANs, Transformers, Distillation Networks, Siamese Networks, among others. When designing new model architectures, it is crucial to understand the goals for constructing them. For tasks such as detecting an Nth-order derivative with respect to time of a robot's battery level or CPU temperature, there are many methods that could assist, some of which do not even rely on neural networks, like Kalman Filters or ARIMA. If one opts to use neural networks, I would not recommend using anything more complicated than a vanilla Feedforward Network, or for time series problems, an LSTM.
But then, let's try to make the exercise of analyzing this architecture to see if it can be useful for other types of problems.
Network A as a representation function:
We are considering two neural networks. Network A serves as a type of encoder, extracting representations from a given state. This is similar to how an encoder functions in an autoencoder or, to use a reinforcement learning example, how MuZero employs a representation function to distill meaningful data from the state for use by another neural network. We then define Neural Network B, which utilizes this representation to predict a value.
The first point to note is that Network B is designed to take as input both the output of Neural Network A and the original data input. This can be done by concatenating the results or adding them together. This approach raises the question of the purpose of constructing Network A in the first place. If the intent behind Network A was to reduce the dimensionality of the data, why would we then feed it the original data as well? Furthermore, it has been mentioned that Network A would backpropagate the loss using the same loss function as Network B, suggesting that it would attempt to learn the same things as Network B. This is in contrast to, for example, GANs, where two different networks are trained on different loss functions. Consequently, each subnetwork in a GAN learns distinct aspects, providing a rationale to consider them as two separate entities rather than a single large neural network.
I'm not disputing the usefulness of residual connections; they are an essential component in Convolutional Neural Networks and Transformers. However, they are typically employed with a specific purpose in mind, not merely because they are always better.
Freezing the Weights of Network B during the training of Network A:
Freezing weights for fine-tuning pretrained models is a common practice in the field of machine learning today. This method is known as transfer learning, which typically involves taking a pretrained neural network and, to adapt it to a specific task or dataset, adding an extra layer for the output. The pretrained weights are frozen, and the training focuses only on updating this final layer. This technique has proven to be immensely useful across nearly all domains of machine learning, becoming an almost universal strategy for enhancing model accuracy.
What you are describing involves a similar approach, but instead of adding an extra layer at the end, a new network is introduced at the beginning. This raises a question: why do we typically add just a single layer at the end when performing transfer learning? Why not add additional layers or even more complex structures? The answer lies in the operational principles of neural networks and how they update their parameters. A neural network can be thought of as a very large nested function. Within such functions, even minor modifications can significantly impact the network's output.
For instance, consider the difference in the loss function of $x^2 + 1$ and $(x^2 + 1)^2$ compared to $x^2$.If you calculate the average distance between the points of $x^2 + 1$ and $x^2$, the average difference is 1. However, between $(x+1)^2$ and $x^2$, the average difference is 2x + 1, which increases with x. This illustrates how a small change in the function's form—where the "+1" is placed—can lead to a large difference in the output.

Backpropagation
Backpropagation updates the network's weights by first calculating the gradient of the final layer and then propagating this gradient to the preceding layer. However, because the gradient can be between 0 and 1, it can decrease with each layer. This is what is known as the vanishing and exploding gradient effect. This means that the most change occurs in the final layers and the least in the initial layers. Hence, the first layers are the hardest to train as their change is very gradual, which means that the features learned by the first layers are the more general and the final layers are the most specific to the input data. This concept is visualized by the features learned by a convolutional neural network. This is the main reason why models are fine-tuned on the last layer; we do not want to significantly alter the initial layers, as this can easily affect the performance of the entire network.
Therefore, training an entire network with a significant number of layers at the front is not advisable because the results can change drastically, and it will take considerable time to train correctly, especially if freezing the other layers.
Experiment with Mnist
I conducted an experiment using a pretrained vanilla multilayer perceptron to predict the MNIST dataset. I then created two additional networks, one with a new layer at the front and another at the end, while freezing the weights of the pretrained model. The results showed that the network with the extra layer at the end performed slightly better than the pretrained model, while the other one performed worse than the pretrained one.

Alternatives and Conclusion
The closest alternative that comes to mind is the use of greedy layer pretraining described in this paper. Overall, with the points I mentioned, I do not believe this method will work efficiently. It will certainly learn something because, in the end, you are performing gradient descent, but it may not do so in an efficient manner; in fact, it might even degrade the model, as my experiment demonstrated. Especially for the problem you described—predicting an nth derivative—Kalman filters or more basic time series models could work much better than a neural network.

Thanks for shout out, but beware comments may be deleted at any time. I really don't require any credit, so feel free to give the advice (understanding the goal is important if you want to play with model architectures) without naming me. We all got our knowledge and ideas from somewhere else, after all.


text:
Compare two songs content using Audio Spectogram Transformer
I'm trying to establish a similarity metric between two songs. To do this I'm using the AST model on HuggingFace. This model basically works in a way very similar to a ViT but applied to spectograms of the audio file. My basic approach is to take the output of a hidden layer for each song passed through the network, yielding a (1, 1214, 768) dim tensor.
features = model(**inputs, output_hidden_states=True).hidden_states[3].detach().cpu().numpy()[0]

I then compare the features between two songs as follows:
def compare_vecs(a, b):
    result = np.einsum('ij,ij->i', a, b)
    norm_A = np.linalg.norm(a, axis=1)
    norm_B = np.linalg.norm(b, axis=1)
    return np.sum(result) / (np.sum(norm_A) * np.sum(norm_B))

(this corresponds to dot producting the 768 dim vectors against each other than summing the list of 1214 dot product scores and normalizing)
I have tried several different compare_vecs implementations (e.g max, min, average, sum) and a few different layer indicies from which I grab the features (last layer, first layer, third layer)... nothing seems to really output anything that is "wow".
I feel like I'm kind of shooting in the dark here and have a few questions:

How does the AST handle different song lengths? It seems that when you extract features from the raw array of sound data the output is always (1, 1024, 128) -- what happens that the audio file length is normalized to a fixed dimension spectogram? (I couldn't find this on the original paper)

What impact does this have on how we compare feature vectors? Could I be comparing a very squished song against a very stretched song and getting total garbage?

Which hidden layers should I be using to extract the features from?

How should I actually be comparing the features mathematically?


text:
Convergence of epsilon greedy policy (with no epsilon decay) using TD Learning?
If I create a policy using the q-values of an epsilon greedy policy using the Sarsa algorithm (not changing the epsilon with each episode), will it converge to the optimal solution to the MDP? I am observing that sometimes it does not. Specifically, example 6.5 in Introduction to RL by Sutton and Barto, when I run Sarsa for 500 episodes and form a greedy policy (no exploration after 500 episodes) using the Q-values of epsilon greedy policy, I am getting different results. Is the reason for suboptimal solution that we are converging the Q values to the estimates of next state action which could be random in case of epsilon greedy (with no epsilon decay)?
Here are the two greedy policies (from epsilon greedy policy iterated for 500 episodes) with two different runs of Sarsa:

In general on-policy Sarsa algo with your epsilon greedy behavior policy actually is the action value version of TD(0) algo, thus it has the same convergence property of TD(0) and is confirmed in the same section of your example reference:

The convergence properties of the Sarsa algorithm depend on the nature of the policy’s dependence on $Q$. For example, one could use "$\epsilon$-greedy or "$\epsilon$-soft policies. Sarsa converges with probability 1 to an optimal policy and action-value function, under the usual conditions on the step sizes (2.7), as long as all state–action pairs are visited an infinite number of times and the policy converges in the limit to the greedy policy

In practice your learning algo may not strictly satisfy all the above mentioned theoretical stochastic convergence conditions such as the usual possibly constant learning rate and your specific problem's all state–action pairs won't be visited an infinite number of times from your mere 500 episodes. You may try to increase your episodes or epsilon values to see better convergence result. Your same reference also mentioned this practical issue yet sometimes it's even desirable:

but not for the case of constant step-size parameter, $\alpha_n(a)=\alpha$. In the latter case, the second condition is not met, indicating that the estimates never completely converge but continue to vary in response to the most recently received rewards. As we mentioned above, this is actually desirable in a nonstationary environment, and problems that are effectively nonstationary are the most common in reinforcement learning. In addition, sequences of step-size parameters that meet the conditions (2.7) often converge very slowly or need considerable tuning in order to obtain a satisfactory convergence rate. Although sequences of step-size parameters that meet these convergence conditions are often used in theoretical work, they are seldom used in applications and empirical research.

Couple of things: (1) - I would like to argue that even if I increase the number of episodes to guarantee visiting all state action pairs infinite times, the algorithm won't converge because our estimate of Q value is taking random action Q into account to update the existing Q when epsilon is not decreasing. (2) - Where can I find the second reference? Could you please tell me the paragraph/page number?

Epsilon is just for exploration, as shown in my referenced answer once it's enough to converge to the greedy policy, and all state–action pairs are visited an infinite number of times satisfying (2.7), the greedy policy would converge to the optimal policy w.p. 1 which is assumed to exist and deterministic in most applications with stable environment such as yours. My 2nd reference is just around the condition (2.7) section.

I think we both mean the same thing. I am trying to satisfy the condition "and the policy converges in the limit to the greedy policy" by slowly decreasing the epsilon and not dropping it directly from a given value to 0. I believe if I drop the epsilon to 0 instantaneously, even if I have explored all the states, my estimates of Q would not be the optimal (I am not converging my epsilon greedy policy to a greedy policy).

Glad to hear that we could converge so quickly with a few single-digit iterations compared to TD/Sarsa as this indicates ours may not be RL. As for the typical slow convergence of Sarsa as you observed, the off-policy Q-learning is usually better in this aspect since you can always keep an independent exploratory b-policy which is no longer constrained by a usual small conservative epsilon as specified in any epsilon-soft policies, in addition to your target.


text:
what are the applications scenarios for prefix decoder LMs
Motivated by this post wherein one of the comments mentioned the use-case for encoder-decoder LM. I wanted to know when to use prefix-decoder LM? vis a vis encoder-decoder or causal decoder only architectures?

Can you please briefly describe (1-2 sentences) this "prefix decoder" and what the context is?

Large-scale language models can be classified into three main types: encoder-decoder, causal decoder and prefix decoder. Each has its advantages and is used in different LLMs for specific purposes.

Encoder-Decoder: Based on the vanilla Transformer model, the encoder-decoder architecture consists of two stacks of Transformer blocks - an encoder and a decoder. The encoder uses stacked multi-headed self-attention layers to encode the input sequence and generate latent representations. The decoder performs cross-attention on these representations and generates the target sequence. This architecture is used in translation software such as Google Translate.

Causal Decoder: The causal decoder architecture incorporates a one-way attention mask, allowing each input token to pay attention only to past tokens and to itself. Input and output tokens are treated in the same way in the decoder. The GPT series, including GPT-1, GPT-2 and GPT-3, are representative language models built on this architecture.

Prefix Decoder : The prefix decoder architecture, also known as the non-causal decoder, modifies the masking mechanism of causal decoders to allow bidirectional attention on prefix tokens and unidirectional attention on generated tokens.

The choice between these architectures depends on the specific task you want to accomplish. For example, if you want to translate text, you can use the encoder-decoder architecture. If you want to generate text autonomously, you can use the causal decoder architecture. Finally, if you want a combination of both, you can use the prefix decoder architecture.

To the best of my knowledge, i think causal decoders models are also called decoder-only transformers and prefix decoder models are often referred to as encoder-only models. Nomenclature is not yet completely static on this.


text:
If the unigram precision is (N-1)/N, then the bigram precision is :
Consider the following machine translation scenario. The reference translation has N words (do not consider sentence beginner ‘hat’ and sentence finisher ‘dot’). The machine output also has N words. If the unigram precision is (N-1)/N, then the bigram precision is
(a) Also (N-1)/N
(b) At most (N-2)/(N-1)
(c) At least (N-2)/(N-1)
(d) At least (N-3)/(N-1)
Statement- If the unigram precision is (N-1)/N, then the bigram precision is  At most (N-2)/(N-1).
Is there any counter example which could prove the above statement wrong having candidate translation, C and reference translation, R.
I am unable to find any such counter example, kindly please help.
Also, are there any generalised results.
Prof Justification:
The given unigram precision implies that only one word in the candidate translation, C is different from the words in the reference translation, R. Under this circumstance the ‘best’ case of bigram match occurs when the unigram mismatch is at one of the boundaries and the rest of the words occur exactly in the same order as in R. Then only one out of N-2 bigrams does not match, giving rise to the bigram precision of (N-2)/(N-1).
The worst case of bigram mismatch occurs when the order of words in C is very different from that of R. The no of bigram matches can be even 0, as in
C: people laugh loudly
R: men loudly laugh

Could you please edit your post to provide more context? What do you know of unigram and bigram precision? What do you mean by that? What's the context?

Unigram and bigram accuracy is a measure used in machine translation to assess the quality of a machine-generated translation compared to a reference translation.

Unigram accuracy is the number of words in the candidate translation that exactly match the words in the reference translation, divided by the total number of words in the candidate translation.

Bigram accuracy is the number of consecutive word pairs (bigrams) in the candidate's translation that exactly match the bigrams in the reference translation, divided by the total number of bigrams in the candidate's translation.

Now, to understand the statement, suppose we have a candidate translation C with N words and a reference translation R. If the precision of the unigrams is (N-1)/N, this means that all but one of the words in C match those in R.
However, this does not guarantee that the bigram accuracy will be at most (N-2)/(N-1). In fact, there could be cases where the bigram accuracy is much lower. For example, consider the following case:

C = "La chatte et sur le tapis"
R = "Le tapis est sous la chatte".

Here, every word in C is also in R, so the unigram precision is 1. However, none of the pairs of consecutive words in C correspond to a pair of consecutive words in R, so the bigram precision is 0. This is an example that contradicts the given statement.

Aren't "la chatte" and "le tapis" pairs of consecutive words both in C and R, or is capitalization relevant?

Good question @JaumeOliverLafont I say it is capitalization relevant otherwise it messes up my example ;)

@Revolucion for Monica
Thank you for replying, but can you give an example in English as I am unable to understand the language used by you.

We  need a counter example; saying bigram accuracy is strictly greater to disprove.

This is the Question-
Consider the following machine translation scenario. The reference translation has N words (do not consider sentence beginner ‘hat’ and sentence finisher ‘dot’). The machine output also has N words. If the unigram precision is (N-1)/N, then the bigram precision is
(a) Also (N-1)/N
(b) At most (N-2)/(N-1)
(c) At least (N-2)/(N-1)
(d) At least (N-3)/(N-1)
Prof. says ONLY option B) is correct- how can I disprove him.

MY TAKE-
In a bigram precision calculation, we are looking at pairs of consecutive words. If all unigrams except one are correct, then the number of correctly predicted bigrams would be N − 2 because you have one less pair of consecutive words due to one unigram being incorrect. Thus option D) is correct.


text:
Zero-shot out of distribution text classification
I'm building out a pipeline that would allow me to filter out text based on whether or not the text belongs to any of the classes I've defined.
I feel like one (albeit naive) approach would simply be to embed both the text and the text representing the class, and apply a distance function to both, discarding the sample if the distance is over some threshold.
Is this feasible in a zero-shot setting? If so, how should I go about figuring out the threshold? If not, what (if any) methods could be used in a zero-shot setting?
Note: I'm not looking for any methods that require additional data outside of the sample and the classes themselves.

Perhaps creating an 'ood' or 'out-of-distribution' tag is warranted?


text:
When to use Tanh?
When and why would you not use Tanh?
I just replaced ReLU with Tanh and my model trains about 2x faster, reaching 90% acc within 500 steps.
While using ReLU it reached 90% acc in >1000 training steps.
I believe the reason it trained faster was due to a steeper gradient (correct me if I'm wrong).
This led me to wonder, why would we not use Tanh over ReLU for most contexts?
Is it the computational complexity? Or something else?

What size and type of model is it? How long did training take (in terms of real time, i.e. hours/minutes)?

Using tanh in hidden layers require careful initialization of network weights and works best with the input features normalized within the same range as output (i.e. -1 to 1). It have expensive functions to evaluate and suffers with vanishing gradient. But it converges faster because due to the zero centered output.
But tanh works good with RNNs, as RNNs suffers from vanishing gradients, and they need an activation function whose second derivative can sustain for long range before going to zero.
If you want any of these properties, use tanh.
Without any gist of architecture, it's hard to tell why your model train faster than ReLU. But generally speaking, ReLU trains faster than Tanh due to lack of expensive functions to evaluate (such as exponents)

"trains faster" is a bit ambiguous in this respect. It could mean "in less steps" or "less wallclock time". Each ReLu step is slightly faster, but you probably need more steps.

Tanh isn't necessarily slow, the exponential function can be approximated very efficiently using this trick https://www.schraudolph.org/pubs/Schraudolph99.pdf .  Also the cost of the activation function is usually fairly small compared to the cost of the matrix operations.  I would standardise the input features rather than to -1/+1, especially if there is any chance of outlier values.


text:
What is loss function in Neural Networks?
I've been studying NNs with tensorflow and decided to code a simple NN from scratch to get a better idea on hwo they work.
It my understanding that the cost is used in backpropagation, so basically you calculate the error between prediction and actual and backpropagate from there.
However, in all the examples I read online, even the ones doing classification, just use:
error=actual-prediction instead of: error=mse(actual-prediction) or: error=cross_entropy(actual-prediction)
And they leave mae/rmse etc just as a metric, as per my understanding (probably wrong) understanding, these should/could be used to calculate the error as well. On the other hand, while working with tensorflow, the loss function I use, does change the output and its not just a metric.
What's my error in here?
In other words, isn't the loss function same as the error function?
Example code (taken from: machinelearninggeek.com/backpropagation-neural-network-using-python/):
Note how the MSE is used as metric only, while backpropagation only uses pred-outputs. (E1 = A2 - y_train)
for itr in range(iterations):

    # feedforward propagation
    # on hidden layer
    Z1 = np.dot(x_train, W1)
    A1 = sigmoid(Z1)

    # on output layer
    Z2 = np.dot(A1, W2)
    A2 = sigmoid(Z2)

    # Calculating error
    mse = mean_squared_error(A2, y_train)
    acc = accuracy(A2, y_train)
    results=results.append({"mse":mse, "accuracy":acc},ignore_index=True )

    # backpropagation
    E1 = A2 - y_train
    dW1 = E1 * A2 * (1 - A2)

    E2 = np.dot(dW1, W2.T)
    dW2 = E2 * A1 * (1 - A1)

    # weight updates
    W2_update = np.dot(A1.T, dW1) / N
    W1_update = np.dot(x_train.T, dW2) / N

    W2 = W2 - learning_rate * W2_update
    W1 = W1 - learning_rate * W1_update
```

I think you are confused about the terms error, loss function and metric.
To summarize:

Error (vector) tells us how off each prediction is.
Loss function maps the error (vector) to a single number (loss).
Loss and metric (scalar) describes overall how well the predictions are, serving as direction towards which the model should improve.

Note these terms are general and do not apply only to neural networks.
TL;DR
Let's take a regression problem as example. Suppose we have a regression model which predicts [-4, 3, 0, 7] where the ground truth is [0, 1, 2, 3]. Clearly, some predictions are off. But by how far? Intuitively, we can get the "off" part by subtraction:
prediction - actual = [-4, 3, 0, 7] - [0, 1, 2, 3] = [-4, 2, 2, 4]
which is what we call the error (vector).
Now we show the error to the model and say, "hey, those are the wrongs you made, do better next round", to which the model replies, "sure, which one do you prefer?" and hands you 2 new predictions:

[-2, 3, 0, 5]
[0, 1, 2, 10]

Which one is better? Well, it depends on you goal: if you want "as close to the truth as possible on average", then 1) may be better; or if you say "no prediction should be below the truth", then 2) is the only choice. Regardless, you have to tell the model which one direction it should improve towards - and this is what loss function is for.
Let's say we decide to use mean-squared error (MSE) as loss function. Computing gives:
Loss_1 = Mean(Square(error vector)) = Mean(Square([-2,3,0,5] - [0,1,2,3])) = Mean(Square([-2,2,-2,2])) = Mean([4,4,4,4]) = 4
Loss_2 = Mean(Square(error vector)) = Mean(Square([0,1,2,10] - [0,1,2,3])) = Mean(Square([0,0,0,7])) = Mean([0,0,0,49]) = 12.25
We call the result of loss function loss (scalar). We want the loss to be small (minimize), so we tell the model, "1) is the better way to go". Notice that the loss is a scalar, and is an indicator of goodness of prediction. In real, a model improves its prediction ('learn') by repeating
make predictions -> compute error vector -> compute loss -> feed to optimization algorithm -> update model parameters -> make new predictions
(Often you would see the terms loss and metric used interchangeably; in brief, metric is the ultimate goal you want to achieve, a number you want to maximize; however for practical reason the optimization process prefers differentiable function, but metric may not be. In this case we usually choose a loss function which is both differentiable and 'close' to the metric. Since MSE is differentiable, it can serve as loss and metric.)
Update
In the code quoted, the computation of loss is hidden in the line dW1 = E1 * A2 * (1 - A2) (and similarly in the upcoming E2 and dW2 lines). Here, back-propagation is applied which implicitly assumes a squared loss is used. Through some maths tricks and simplification, we arrive at the E1 formula. You may find the full derivation here.

Thank you so much for your clear response, now it makes a lot more sense. So the error vector (prediction-actual) is always the same regardless of a regression and classification?. As for the loss function, this is used on the optimization algorithm (such as adam), is this correct?

for the error vector - I think yes, at least no exception comes to my head. 2) yes, loss (function) is for the optimization algorithm.

by chance is there any online resource you can suggest to read more on the subject? every numpy neural network I can find uses the error vector as you described but I can't find any example of what they do with the loss function itself, other than just calculating it.

do you mean on e.g. why we pick one loss function other than another?

No, I mean where is exactly the loss function used?, all the codes I find, they calculate the error vector and use it on the backpropagation. Not the loss.

can you show one example of these code? Easier to elaborate from there.

Yes, I can't paste it here (although I will edit my question with the respective code), but an example can be seen here: https://machinelearninggeek.com/backpropagation-neural-network-using-python/
Note that he calculates the mse but as a metric only, while the error vector is calculated as pred-output, which is what's actually used in the backpropagation.

@user20170158 updated the answer.

There are different functions to pick from for your loss.
The most common read online is the Mean Squared Error.
It calculates how far the prediction is off from the actual value and squares it. All of them are summed up.
So basically the Loss-Function "generalizes" the error over every prediction. The actual error in a case is just how far off it is.

In my code I have:
code
error=outputs-training_set_outputs
#derivation calcs with error
code
So by your answer I understand that the MSE would be used as a metric of how well the model generalizes only?

The loss function is a way to evaluate your whole model, yes.
However I don't get what your error should be. For me it looks like your error is just your output

if I use different loss functions in tensorflow, I do get different results, is this because the loss function is used in the optimizer? (rather than changing the error function error=y-y_hat)

The answer above is really good. However it's natural that different Loss Functions give you different results.
You can calculate it by hand if you want, the Mean Squared Error is also often taught in school. It's totally doable. Just think of a small example, choose different functions and then calculate. You'll see easily why you'll get different results. Just keep in mind that having a smaller error is not always good. You generally want to avoid Over- and Underfitting. Also it depends on what you want to do. There is not just one solution in AI

A loss function is the function which can get the losses(differences) between a model's predictions and true values to evaluate how good a model is. *Loss function is also called Cost Function or Error Function.
There are popular loss function as shown below:
(1) L1 Loss:

can compute the average of the sum of the absolute losses(differences) between a model's predictions and true values.
's formula is as shown below:

's pros are as shown below:

Less sensitive to outliers.
We can easily complare the losses because they are just made absolute so the range of them is not big.

's cons are as shown below:

The absolute loss of |0| cannot be differentiable according to this post and this post.

is used for a regression model.
is also called Mean Absolute Error(MAE).
is L1Loss() in PyTorch.

(2) L2 Loss:

can compute the average of the sum of the squared losses(differences) between a model's predictions and true values.
's formula is as shown below:

's pros are as shown below:

All squared losses can be differentiable.

's cons are as shown below:

Sensitive to outliers.
We cannot easily complare the losses because they are squared so the range of them is big.

is used for a regression model.
is also called Mean Squared Error(MSE).
is MSELoss() in PyTorch

(3) Huber Loss:

can do the similar computation of either L1 Loss or L2 Loss depending on the absolute losses(differences) between a model's predictions and true values compared with delta which you set.
*Memos:

delta is 1.0 basically.
Be careful, the computation is not exactly same as L1 Loss or L2 Loss according to the formulas below.

's formula is as shown below. *The 1st one is L2 Loss-like one and the 2nd one is L1 Loss-like one:

's pros are as shown below:

Less sensitive to outliers.
All losses can be differentiable.
We can more easily complare the losses than L2 Loss because only small losses are squared so the range of them is smaller than L2 Loss.

's cons are as shown below:

The computation is more than L1 Loss and L2 Loss because the formula is more complex than them.

is used for a regression model.
is HuberLoss() in PyTorch.
with delta of 1.0 is same as Smooth L1 Loss which is SmoothL1Loss() in PyTorch.

(4) BCE(Binary Cross Entropy) Loss:

can compute the differences between a model's binary predictions and true binary values.
s' formula is as shown below:

is used for Binary Classification. *Binary Classification is the technology to classify data into two classes.
is also called Binary Cross Entropy or Log(Logarithmic) Loss.
is BCELoss() in PyTorch.
*Memos:

There is also BCEWithLogitsLoss() which is the combination of BCE Loss and Sigmoid Activation Function in PyTorch.
Sigmoid Activation Function suits BCE Loss to be more stable.

(5) Cross Entropy Loss:

can compute the differences(0~1) between a model's predictions and true values.
s' formula is as shown below:

is used for Multiclass Classification and Computer Vision.
*Memos:

Multiclass Classification is the technology to classify data into multiple classes.
Computer vision is the technology which enables a computer to understand objects.

is CrossEntropyLoss() in PyTorch.


text:
How does gradient descent work with ReLU if weights are negative?
How does gradient descent work with ReLU, imagine the weights are quite negative and so our "prediction" is 0, then not much is learned. Is there a risk that training gets stuck when weights start negative or small?

Related https://datascience.stackexchange.com/q/5706

The issue you have described is called the dying ReLU, which is basically about getting a gradient of zero when negative predicted values got thresholded to zero.
In general this is only an issue when ALL the units in a layer (also for all layers) predict negative values. So only in this extreme situation your network won't learn anything because the derivative is zero.
But it can happen that some units in a Dense layer (for example) always output a zero, in this case these have died and so they are not useful anymore for learning.
The way to fix the issue, is to change activation function (but I guess that weight initialization may also help) to something like: leaky ReLU (which introduces a negative slope where the gradient exists), ELU (exponential linear unit; slower to compute but never dies), or even SELU (scaled ELU).

Dying ReLU Problem sometimes occurs with Relu activation function.
Dying ReLU Problem:

is during backpropagation, once the nodes(neurons) with ReLU activation function recieve negative input values, they always produce zero for any input values, finally, they are never recovered to produce any values except zero, then a model cannot be trained effectively.
is also called Dead ReLU problem.
more easily occurs with:

higher learning rates.
higher negative bias.

can be detected if:

convergence is slow or stopped.
a loss function returns nan.

can be mitigated using:

lower learning rate.
a positive bias.
Leaky ReLU activation function.
PReLU activation function.
ELU activation function.

Gradient Descent fails to updates weights due to Dying RELU problem resulting in poor learning, preventing the network to capture underlying patterns of the training data, leading to poor generalization on unseen data


text:
Why is Batch Gradient Descent performing worse than Stochastic and Mini-Batch Gradient Descent?
I have implemented a neural network from scratch (only using numpy) and I am having problems understanding why the results are so different between Stochastic/Mini-Batch Gradient Descent and Batch Gradient Descent:

The training data is a collection of point coordinates (x,y).
The labels are 0s or 1s (below or above the parabola).

As a test, I am doing a classification task.
My objective is to make the NN learn which points are above the parabola(yellow) and which points are below the parabola(purple).
Here is the link to the notebook: https://github.com/Pign4/ScratchML/blob/master/Neural%20Network.ipynb

Why is the Batch Gradient Descent performing so poorly with respect
to the other two methods?
Is it a bug? But how can it be since the code is almost identical to the
Mini-Batch Gradient Descent?
I am using the same (randomly chosen with try and error) hyperparameters for all three
neural networks. Does Batch Gradient Descent need a more accurate
technique to find the correct hyperparameters? If yes, why so?

Assuming the problem at hand is a classification (Above or Below parabola), this is probably because of the nature of Batch gradient descent. Since the gradient is being calculated on the whole batch, it tends to work well on only convex loss functions.
The reason for why batch gradient descent is not working too great probably is because of the high number of minimas in the error manifold, ending up learning nothing relevant. You can change the loss function and observe the change in results, they might not be great (Batch GD usually isn't) but you'll be able to see differences.
You can check this out for more on the differences between the three. Hope this helped!

Thanks, I am understanding this better. So, basically, BGD is not very used because I guess most loss functions have multiple minimas. Am I wrong?
What other loss functions could I use in this particular example?

Yes. What other loss functions could I use in this particular example - You have used the difference as loss (written as 'error' in your code) if I'm not wrong? If so, you can also try out Magnitude of the difference, square of the difference, logarithmic cross-entropy - looking up common loss functions will give you an abundant number of potential ones.

Because the computation of Batch Gradient Descent(BGD) is more stable(less fluctuated) than Mini-Batch Gradient Descent(MBGD) and Stochastic Gradient Descent(SGD), so BGD less easily(more difficultly) escapes local minima than MBGD and SGD, resulting in poor learning even thouch BGD can get more accurate output than MBGD and SGD.


text:
the best choice to reduce a features vector
i have 1200 features highly correlated , and i want to reduce those number of features
so the best choice is use feature selection or dimensionality reduction? and which method is the best in this case.
feature selection can work with features highly correlated ??

Isn't dimensionality reduction a form of feature selection? What do you mean by "feature selection" more specifically? Please, edit your post to provide more details.

Feature selection -- the case in which the features are highly correlated is the prototypical case in which you want to select a subset of independent features that allows for an equal performance.  However keep in mind that, exactly because they are correlated, there are in general multiple subsets of features that can achieve the same goal (i.e., if x_1 is highly correlated with x_2, you can drop either of them).

Dimensionality reduction -- if your features are only linearly correlated, then a linear dimensionality reduction technique (e.g., PCA) will tackle your problem.  If however the dependence is more than linear (and unknown), dimensionality reduction will construct new features that are a complex combination of the original ones, hence you will get to the same problem, just in a reduced-dimensional space.

TLDR; As usual in ML, it depends on what you need to do.  If it is merely a matter of reducing dimensions in order to speed up calculations, either method is valid.  If you need instead to select features for further [causal?] analysis, you need to lean for the first approach.  But, again, be aware of multiple equivalent sets (e.g., see this paper: https://arxiv.org/abs/1611.03227)

thank you for your answer , my goal is at the end i will choose a classifier and do the binary classification . this features will be the input of this classifier


text:
To which value does converge the objective: $target = (r + \gamma Q(s',*)) / 2$
Conditions:

Assume you have a DRL SAC implementation.
You train it as usual from Replay Buffer with uniform sampling with replacement.

Assume you change the target of the Q-network from:
$$T = r + \gamma Q(s',*) $$
To:
$$T^* = \frac{r + \gamma Q(s',*)}{2} $$
Then:

To which values will now converge the Q-values? I mean, will it converge to an average reward as in average reward RL? Diverges?

I have done some tests and I have convergence to some value, but I don't know to which value it is converging to.

Is there a simple proof that the objective: $T^*$ does not converge to the average reward?

"to which value it will converge"... what does that mean??

In the DRL replay buffer training scenario, T converges to the expected cumulative sum of rewards (r), to which value converges T*?

RL aims at finding a policy $\pi$ that maximizes $G = \mathbb{E_\pi}[\sum \gamma^i r_i]$
Note how this is an expectation, so it's scale (positive) and shift invariant... indeed, given a reward function $R(s,a,s')$, a second function $R(s,a,s')' = R(s,a,s')+c$ would lead to the same policy $\pi$, as per positive scalings
In other words, you would get (in the optimal regime) the same policy

Thanks, but forget about the policy, it is in the Q-values estimate in which I am interested in

@JoseAntonioMartinH they will be half

thanks @alberto yes, I've developed it mathematically and it converges to Q/2

@JoseAntonioMartinH you're welcome


text:
Is it possible to do machine learning on encrypted images?
I wuold like to know if it is possible to train a CNN to classify images (like nsfw detection) without receiving any intelligible information about it ?
By applying some kind of one-way mask before upload for example.
Edit: I found that it is possible to apply filters on encrypted images with homomorphic encryption, but I don't know if classification is possible.

you can do density estimation if you manage to collect a dataset of only nsfw images

Hi @max-it and welcome to AI Stack Exchange! If possible, please edit this post to include a question in the title. Also, please add more detail to the question that might help people provide better answers. Thank you for participating, and we look forward to more of your questions in the near future!

To your edit, it would require that homomorphic encryption is used, which is not typical.

The short answer is No.
The long answer is maybe, if you use Homomorphic encryption.

Homomorphic encryption is the conversion of data into ciphertext that can be analyzed and worked with as if it were still in its original form. Homomorphic encryption enables complex mathematical operations to be performed on encrypted data without compromising the encryption.

However, this is a very mathematically complex task, and the encryption is weak at best.

Homomorphic encryption schemes are inherently malleable. In terms of malleability, homomorphic encryption schemes have weaker security properties than non-homomorphic schemes.

Here's an image courtesy of Nvidia:


text:
Deployment of Model into Java Application
Over the past few months, I have developed and tested a TensorFlow U-Net model based on the Keras Functional API. Recently I have been interested in deploying this model for use in a Java application that would run mainly on laptops and PCs preferably offline. Does anyone know a way I could potentially host my model on this Java Application? I have both a TFLite model and a regular Saved Model; however, I prefer to use the normal Saved Model. I would appreciate it if anyone could point me in the right direction or show me a good tool to deploy my model.


text:
Is there a way to reward my Variational-Auto encoder for using less colors while still letting it make creative decisions?
So recently I have been trying to program a Tensorflow and Keras based model that can animate pixel art characters. I use a Variation Auto Encoder with Convolutions, Dense Layers and Upsampling 2d layers. One of the major problems that I have been facing is that the generated images tend to be very noisy and complex when it comes to color selection. Normally this would be fine for more realistic images but I'm aiming for more of a pixel art and voxel look where the produced image uses a more restricted color palette. My model intakes 2 (48,48) RBGA image and outputs a single (48,48) RBGA image that acts as an interpolation between the two.
Does anyone know of an optimizer, loss function, special layer or some other trick to potentially limit or disencourage the model from using a lot of different types of colors. I'm considering making a custom layer that encodes each color within the input images to a set integer and have the model predict which integers to put where. This would result in the model only being able to use colors that showed up within the input images but has 2 major issues. First of all this would severely restrict the models creativity as it can't use new colors when it should. Another major issue with this is that it would remove the models ability to understand how different colors should interact with each other within the Convolutions.
TLDR: I want to punish my Variatonal Autoencoder image generation model more for using a lot of different colors or reward the model for using less colors while still letting it make creative decisions and understand the relationships between colors.
Target Image:

Generated Image:

I was told to move this problem from Stack overflow due to it being primarily conceptual; here's the link to the old problem/discussion: Old Question

You can encode this in a loss, not in a differentiable one though.
What you want is something like:
$$
L(\theta) = L^{VAE}(\theta) - \lambda \sum_i|set(f_\theta(x_i))|
$$
So you penalize your model if the size of the set of colors used in the prediction is too big... clearly, this is not differentiable
What you can do is the following:

discretize it post processing : indeed, you will have a "histogram" of color distribution, create an heuristics that reflects your tastes
discretize it as pre-processing, thus define a set of colors you allow, so instead of $[0,255]^3$ you pick $\{n\%16 == 0 | 0 \le n \le 255\}^3$, adn then to deploy the model, just let it predict a color, and set it to the closest one of your palette

You can also do a sort of RL reinforce for the color, so you get your prediction, you bin the colors, and you use RL to reinforce the one you prefer, however this  involves RL, which is a world of his own

Thanks so much for your help! I think I will attempt your main recommendation as  it should fill the current needs of the project. Just to confirm because I'm quite new to ML as a whole, I should first get a set amount of colors that the model is allowed to use and store them in the histogram; then after the image is generated, instead of using a loss function to punish it for using a wide variety of colors, I will instead just convert every color to the closest match that is on the histogram.

@Max yes, however, you first have to train your VAE, and then at "evaluation" you do that, so the training is the usual VAE training (otherwise doing the adjustment will cause to loose the gradient)


text:
Why aren’t intellectual leaders stepping in to clarify that we know little to nothing about how intelligence works or how to build intelligence?
There is no known way of building intelligence/superintelligence today (often and hereafter referred to as AGI), nor is there a known way towards building it in the future. Unlike fusion reactors, for example.
Nonetheless, certain actors have been highly effective in convincing the public that society is about to be taken over by AGI. People are increasingly afraid making objectively poor life-decisions.
I assume one of the roles of the scientific community in society is to also safeguard the masses against (religious/pseudo-religious) eschatological entities. Therefore, are there any proven or stipulated reasons for why it might be in the interest of academia, through action or omission, not to do so?
While it can be enjoyable to creatively speculate from time to time, please try to stick to the facts and try to back up your answer with sources. Also, while that may be difficult for some, please don’t share your opinions about whether AGI might or might not be possible.
Thank you!

I think this is just the bias towards money and loudest speaker, because there are many AI scientists claiming exactly what you ask, that we know little about intelligence or how to achieve it, but the press (mostly) ignores them.

I think there are a few reasons why the so-called leaders or experts of the AI field may continue to entertain the idea that creating an AGI is possible or is not impossible

the concept of an AGI is often not clearly defined in discussions - often, I'd assume that people think of an AGI as a program or machine which would have all capabilities of a human or even more, but probably not less - so you can't really do science - in other words, all these discussions are just philosophical and probably pointless
there's no clear evidence or proof that it's impossible to do it; anyway, almost no one is really trying to do this rigorously, i.e. from a mathematical/theoretical standpoint (an exception is the work on AIXI); I think we can't prove this until we clearly define the concept of an AGI, but other people may have a different idea, i.e. they try to develop narrow AIs, and, eventually, they think, something like an AGI (a human) will be developed, even if you don't really define it precisely/mathematically
people like to dream
it might attract investors and funding
most people don't really care about most other people

I've tried to give you a few possible explanations. I'd also like to note that there may be leaders/experts who try to sensitize the masses. Some, like LeCun (Turing Award), even claim there's no such thing as AGI (see slide 66 here), then he defines intelligence as a collection of skills, etc., in other words, he gives one definition of what we could call AGI. As you can see, everyone has their own opinion - and people are pursuing different things.
Having said this, I think your question can, in general, lead to opinions.

I don't know, money?  (shout out to @Dr. Snoopy for saying exactly this but more concisely!)
Martin Ford, in his book "Architects of Intelligence - The Truth about AI from the people building it" (2018) interviews 23 modern AI icons, and inevitably asks, "But is it AI"?  Please consider purchasing this awesome work to see what he found out.
https://www.goodreads.com/book/show/41998128-architects-of-intelligence
https://www.amazon.com/Architects-Intelligence-truth-people-building/dp/1789954533/ref=tmm_hrd_swatch_0?_encoding=UTF8&qid=&sr=
Andrew Ng (interviewed, above) https://www.andrewng.org/ is among the great minds of the LLM world to have publicly stated that the current wave is not AGI, and making it bigger is not the path.
François Chollet (inventor of Keras, not interviewed above) has put up a lot of money (with Mike Knoop) to solve his ARC-AGI challenge, https://arcprize.org/
ARC was specifically designed to be novel (and theoretically unsolvable by off-the-rack LLMs, but easily solvable by humans and thus probably by AGI).  This is as close as I've seen to waving a flag, publicly.  The ARC-AGI challenge has been a fixture of Kaggle.com since 2020.  ARC stands for Abstraction and Reasoning Challenge.
For the uninitiated, Keras is an amazing project that IMO seems to be at the heart of all things, machine-learning.


text:
Is AI good at detecting AI-generated content?
Are AI models good at detecting AI-generated image or video content like deep fakes? Which model can we use for the detection of AI-generated image content?

depends... if you can discriminate, you can improve using that discriminator, check "GANs", that's literally how they work


text:
Is there serious game-theoretic work on AI risk and alignment?
My background is in political economy and game theory. I am interested in the discussion on AI risk and alignment, but I have so far failed to find work on this that seriously engages with classic axiomatic rational choice theory (RCT).
Some claims are often made that, to me, contradict the basic tenets of RCT. For example, it is often suggested that a sufficiently advanced AGI might come up with goals of its own that put at risk humanity. However, according to RCT, agents can only choose actions, whereas their preferences (goals) are primitives of the model. It is nonsensical to think of an agent as choosing his goals, except in some well-defined dynamic model (say, Becker's model of rational addiction) where the future preferences result from actions that can be derived, by equilibrium analysis, from primitive preferences.
Another claim that perplexes me is that the actions of a sufficiently intelligent AI may be impossible to predict. However, to me the more intelligent an agent, the more its actions should approximate those of an expected utility maximizer.
I wonder if serious people are working on this from a different axiomatic system. What are some good sources (at any level of technicality) that I should read? Better if books.

Human Compatible by Stuart Russell?

I'm looking for something a bit more rigorous: something structured as Definition-Lemma-Theorem, in the tradition of rational choice / game theory.

@raving-bandit I have the same background as you and am interested in similar questions, which is how I found your post; it would be amazing to connect and bounce ideas. I call on the moderators not to remove this post since I would like to contact the OP for potential research collaboration, but I am unsure how to do it otherwise.

@raving-bandit, the common argument on your first question is that an AI system may take its goal to the extreme and divert all the world's resources to it, causing human extinction (see e.g. here: https://cepr.org/voxeu/columns/ai-and-paperclip-problem)
RCT may not be the best framework because we typically assume everyone knows the game being played, whereas an AI system is dangerous because it could think of moves no-one would expect. This also relates to your second question. It appears that epistemic game theory (https://plato.stanford.edu/entries/epistemic-game/) or limited rationality models like cognitive hierarchies (https://en.wikipedia.org/wiki/Cognitive_hierarchy_theory) could be a better way to approach the problem than RCT.

Firstly contrary to your claim, sometimes it's a requirement for artificial agents to choose multiple different goals for the sake of an intended goal such as goal conditioned RL.

Another active area of research is in learning goal-conditioned policies, also called contextual or universal policies that take in an additional goal as input to communicate a desired aim to the agent.

Bostrom further explored the idea of instrumental convergence that intelligent agents might pursue certain instrumental goals regardless of their final objectives in his paper Motivation and Instrumental Rationality in Advanced Artificial Agents.
As for your RCT which is not a game theory but a classic evidential decision theory, the extant RL in AI is consistent with RCT and may be viewed as an advanced sequential multi-shot version of RCT. S. Russell discussed the relationship between rationality and intelligence in his Rationality and Intelligence: A Brief Update, offering insights that may align with your interest in AI application issue of RCT.

The concept of rational agency has long been considered a leading candidate to fulfill this role. This paper, which updates a much earlier version (Russell, 1997), reviews the sequence of conceptual shifts leading to a different candidate, bounded optimality, that is closer to our informal conception of intelligence and reduces the gap between theory and practice.

Hadfield-Menell et al. presents a game theory and RCT consistent formal model of an AI system that includes an off-switch controlled by humans exploring how to design AI systems that can be safely interrupted in their paper The Off-Switch Game.

It is clear that one of the primary tools we can use to mitigate the potential risk from a misbehaving AI system is the ability to turn the system off. As the capabilities of AI systems improve, it is important to ensure that such systems do not adopt subgoals that prevent a human from switching them off. This is a challenge because many formulations of rational agents create strong incentives for self-preservation... Our key insight is that for R to want to preserve its off switch, it needs to be uncertain about the utility associated with the outcome, and to treat H’s actions as important observations about that utility... We conclude that giving machines an appropriate level of uncertainty about their objectives leads to safer
designs, and we argue that this setting is a useful generalization of the classical AI paradigm of rational agents.


text:
When can we unnest the minimizations/recursions in an value function(bellman optimality equation)?
When reading the following paper(page 4): An Approximate Dynamic Programming Approach
for Dual Stochastic Model Predictive Control
I could see that they were able to unnest the minimization's in the bellmans equations due to sampling of the expected values. It would be cumbersome to write down their whole setup so i thought i might understand why with another example:
Consider the value function below:
\begin{align}
V^{*}(s_0) &= \min_{a_0 \in A_{s_0}}[r(s_0, a_0) + \sum_{s_1}p(s_1|s_0, a_0)V^{*}(s_1)] \\
           &= \min_{a_0 \in A_{s_0}}[r(s_0, a_0) + \sum_{s_1}p(s_1|s_0, a_0)]\min_{a_1 \in A_{s_1}}[r(s_1, a_1) + \sum_{s_2}p(s_2|s_1, a_1)V^{*}(s_2)] \\
           &(?\neq, = ?) \min_{a_0 \in A_{s_0}, a_1 \in A_{s_1}}[r(s_0, a_0) + \sum_{s_1}p(s_1|s_0, a_0)][r(s_1, a_1) + \sum_{s_2}p(s_2|s_1, a_1)V^{*}(s_2)]
\end{align}
when should it be an $\neq$ or an $=$ in the third equation and why, i.e how can sampling break the recursion and allow us to unnest the minimizations?
(in the paper, they argue that since they use samples to approximate the expected value of the cost-to-go/next state value function they are allowed to unnest the minimizations)

Crossposted (duplicate) at OR.SE: When can we unnest the minimizations/recursions in value functions(bellman optimality)?

@SecretAgentMan This question is perfectly fine for this site. There's arguably no better site to ask this question than AI SE.

removed the question on operations research


text:
Keras accuracy does not change for multi-output module
I want to predict the penalty/punishment given for fraud cases, with inputs in the form of (damage amount(\$), if recidivism), and targets in the format of (fine(\$), jail(months), community service(hours), probation(months)) - the following is my code:
fname = "filepath.tsv"
all_in = []
all_out = []

with open(fname) as f:
    for i, line in enumerate(f):
        if i < 3:#first line
            print("Header:", line.strip().split('\t'))
            continue
        fields = line.strip().split('\t')
        all_in.append([int(a.replace(",", "")) for a in fields[5:7]])
        all_out.append([int(a.replace(",", "")) for a in fields[7:11]])

case_in = np.array(all_in, dtype = "uint64")
target_out = np.array(all_out, dtype = "uint64")

normalize_layer = tf.keras.layers.Normalization(axis=-1, name = "normalize_in")
normalize_layer.adapt(all_in)
normalize_out = tf.keras.layers.Normalization(axis=-1, name = "normalize_out")
normalize_out.adapt(all_out)
denormalize_out = tf.keras.layers.Normalization(axis=-1, invert = True, name = "denormalize_out")
denormalize_out.adapt(all_out)
scaled_out = normalize_out(all_out)

inputs = normalize_layer(Input(shape = 2))
x = Dense(6, input_dim = 2, activation = "sigmoid", use_bias = True)(inputs)
x = Dense(4, activation = "sigmoid")(x)
y_4 = Dense(1, activation = "sigmoid", name = "y_4")(x)
punishment = Dense(3, activation = "sigmoid", name = "punishment")(x)
y_1 = Dense(1, activation = "sigmoid", name = "y_1")(punishment)
y_2 = Dense(1, activation = "sigmoid", name = "y_2")(punishment)
y_3 = Dense(1, activation = "sigmoid", name= "y_3")(punishment)

model = Model(inputs = inputs, outputs = [y_1, y_2, y_3, y_4])

model.compile(
    optimizer = SGD(learning_rate=0.01, weight_decay=1e-6, momentum=0.9, nesterov=True),
    loss = {
        "y_1" : "MeanSquaredError",
        "y_2" : "MeanSquaredError",
        "y_3" : "MeanSquaredError",
        "y_4" : "MeanSquaredError"
    },
    metrics=['accuracy']
)

model.fit(
    case_in,
    scaled_out,
    batch_size = 10,#after # of data increases, increase
    epochs = 300,
    verbose = 2,
    validation_split = 0.8
)

for layer in model.layers:
    print("===== LAYER: ", layer.name, " =====")
    if layer.get_weights() != []:
        weights = layer.get_weights()[0]
        biases = layer.get_weights()[1]
        print("weights:")
        print(weights)
        print("biases:")
        print(biases)
    else:
        print("weights: ", [])

And, once starting to train the model, the accuracy do not change at all. Also, while I was trying to fix that, I messed something up to make my data with ~50 datasets only one batch of the .fit operation, though my batch size says 10. I was just trying way too many things to fix my code from here and there on the internet and it kind of messed everything up even more.
Initially, I tried normalizing the data, both my ins and outs, which unfroze my loss function, but it didn't really do anything to the accuracy function. I also changed the ReLU function to the sigmoid function too, because at some point it seemed it might be a problem with a dead ReLU, as the biases on my layers were not being updated from their initial 0s. After that, I kept nudging around the optimizer, the loss functions, and epochs, which all came to no avail.
How to make the module actually work to its purpose? And do you have general feedback on the code?

Choose an appropriate accuracy metric for regression tasks, such as MAE or MAPE.
Use the batch_size parameter to specify the batch size during training, and ensure the validation_split parameter is used for validation data splitting.
Consider adjusting the model architecture and trying different optimizers to optimize your model's performance.
It's always beneficial to experiment and fine-tune these parameters based on your specific problem and dataset characteristics.


text:
Keras functional model returning unexpected output dimensions
So, this is my model:
#input layers
inputs = Input(shape = 2, name = "Input")

#hidden layers
x = Dense(6, activation = "relu", name = "dense_layer_1")(inputs)
x = Dense(4, activation = "relu", name = "dense_layer_2")(x)
punishment = Dense(3, activation = "relu", name = "punishment")(x)

#output layers
y_1 = Dense(1, activation = "sigmoid", name = "y_1")(punishment)
y_2 = Dense(1, activation = "sigmoid", name = "y_2")(punishment)
y_3 = Dense(1, activation = "sigmoid", name= "y_3")(punishment)
y_4 = Dense(1, activation = "sigmoid", name = "y_4")(x)

#functional model declaration
model = Model(inputs = inputs, outputs = [y_1, y_2, y_3, y_4])

and, when I call input_shape and output_shape on this:
print(model.input_shape)
print(model.output_shape)

(None, 2)
[(None, 1), (None, 1), (None, 1), (None, 1)]

But, when I call my model:
print(np.array([normalize_in([290360000, 0])], dtype = "float32")[0].shape)
print(denormalize_out(model(np.array([normalize_in([290360000, 0])], dtype = "float32")[0])))

it gives me:
[[ 1.724373   -0.39440534]]
(1, 2)
tf.Tensor(
[[[8.4842375e+05 1.7916626e+01 3.6546925e+01 1.6306080e+01]]

 [[7.9671431e+05 1.7595743e+01 3.5045170e+01 1.5826001e+01]]

 [[7.6195538e+05 1.7380047e+01 3.4035694e+01 1.5503292e+01]]

 [[9.3114094e+05 1.8429928e+01 3.8949211e+01 1.7074041e+01]]], shape=(4, 1, 4), dtype=float32)

I've been trying to go around the internet to find it, but there was just way too few examples about multi-output keras functional models, and I couldn't figure out what is wrong with my model.

Based on the code and model architecture you have provided. It seems your model takes a single input and gives four outputs the shape of the input layer is (batch_size,2) while the output shape of each node seems to be (batch_size,1).
your output should be a list of four outputs means a list that contains the output tensors of all nodes. but you are getting the tensor of shape (4,1,4).
I think it may be an issue due to the way you are giving the input to the model can you please check the input shape since I have used the same input as you and I got a valid shape means a list of output tensors each contains tensor of shape (batch_size,1). and also, I think you should verify that the denormalize_out function is not stacking your output list.
also to verify the model architecture I have loaded your model on Netron it seems to have the correct architecture you can see it below

Here below is my code and its output
inputs = Input(shape = 2, name = "Input")

#hidden layers
x = Dense(6, activation = "relu", name = "dense_layer_1")(inputs)
x = Dense(4, activation = "relu", name = "dense_layer_2")(x)
punishment = Dense(3, activation = "relu", name = "punishment")(x)

#output layers
y_1 = Dense(1, activation = "sigmoid", name = "y_1")(punishment)
y_2 = Dense(1, activation = "sigmoid", name = "y_2")(punishment)
y_3 = Dense(1, activation = "sigmoid", name= "y_3")(punishment)
y_4 = Dense(1, activation = "sigmoid", name = "y_4")(punishment)

#functional model declaration
model = Model(inputs = inputs, outputs = [y_1, y_2, y_3, y_4])

#Dummy Input
dummy_input = np.random.rand(1,2)
print("Input shape",dummy_input.shape)

#If we store the output in diffrent variables it store each as output tensors
out1,out2,out3,out4 = model.predict(dummy_input)

#If we store all outputs in a single varialbe it store it as list of output tensors
preds = model.predict(dummy_input)

print("output 1 shape",out1.shape)
print("output 2 shape",out2.shape)
print("output 3 shape",out3.shape)
print("output 4 shape",out4.shape)

#making predictions
dummy_input_2 = np.array([[1.724373  , -0.39440534]])
print("dummy_input_2 shape",dummy_input_2.shape)
predictions = model(dummy_input_2)
print(predictions)

The output of the above script is
Input shape (1, 2)
output 1 shape (1, 1)
output 2 shape (1, 1)
output 3 shape (1, 1)
output 4 shape (1, 1)
dummy_input_2 shape (1, 2)
[<tf.Tensor: shape=(1, 1), dtype=float32, numpy=array([[0.3822088]], dtype=float32)>, <tf.Tensor: shape=(1, 1), dtype=float32, numpy=array([[0.60975766]], dtype=float32)>, <tf.Tensor: shape=(1, 1), dtype=float32, numpy=array([[0.5706901]], dtype=float32)>, <tf.Tensor: shape=(1, 1), dtype=float32, numpy=array([[0.45608807]], dtype=float32)>]

Based on the code you provided, it appears that the input shape is correctly set to (None, 2), indicating that the model expects inputs of shape (batch_size, 2).
Regarding the output shape, the model correctly has four output layers (y_1, y_2, y_3, y_4). Each of these layers has an output shape of (None, 1), indicating that the model will produce an output of shape (batch_size, 1) for each output layer.
When you call your model with an input of shape (1, 2), you indeed get an output of shape (1, 2) as expected.
However, when you use denormalize_out on the model output, the resulting shape is (4, 1, 4). This indicates that the output shape is (num_outputs, batch_size, num_classes).
This unexpected output shape may be caused by discrepancies in your normalization and denormalization functions or a mismatch between the expected output shape and the denormalize function's handling of the output.
I would recommend double-checking your normalization and denormalization functions to ensure they are correctly handling the output shape. Additionally, verify that the denormalization function expects an output shape of (batch_size, 4) instead of (4, 1, 4). Adjusting the function accordingly may help resolve the issue.
If the issue persists or you need further assistance, please provide more details on the normalization and denormalization functions, as well as any relevant information about the dataset and the desired output dimensions.
It may be helpful to refer to the Keras documentation for further guidance.


text:
In DDPM, why is "forward posterior" needed?
From the paper:

But what's the point of $q(x_t\mid x_t, x_0)$? Right before that, we already have:

So we already know what $q(x_t\mid x_0)$ is for any $t$.
My questions:

What's the point of showing (6)? (It is used throughout rest of formulas, but it could just depend on $x_0$ everywhere, no? Even during sampling it could just use $\hat{x_0}$)
Are (6) and (4) equivalent somehow?

Can you please include the name and link to the paper in your post?

@nbro Will do - it's just the original DDPM paper. https://arxiv.org/abs/2006.11239


text:
Implementation of the diversity strategy of MSCPSO
The Multi-Swarm Self-Adaptive and Cooperative Particle Swarm Optimization algorithm (MSCPSO) from this paper mentions a diversity strategy to enable better exploration capabilities over the standard PSO algorithm.
How is it implemented? Example value of the parameters? Duration of the exploration?
From my understanding, when the pbest of each sub-swarn as not improved over n iterations, thus computing $||pbest_{alltime}-pbest_{current}||_1 < \epsilon$, $g_{best}$ is replaced by its opposite for every particles.


text:
Bellman equation and inverse matrix method

My problem: why the last formula do not contain information about time $t$?
So if $s^{\prime}=s$, do we have $v_{\pi}(s) = v_{\pi}(s^{\prime})$? But this is not right I guess?
If I am right, that they are not the same, considering Bellman equation for MRP $\boldsymbol{v}=\boldsymbol{r}+\gamma \boldsymbol{P} \boldsymbol{v}$. But why the left vector $\boldsymbol{v}$ and the right vector $\boldsymbol{v}$ are the same? The left one contains reward begin from $t$ but the right one contains reward begin from $t+1$, right?

In reinforcement learning, the apostrophe character $(')$ appended to a signal usually represents the signal at the next time step. For example, $s'$ is the state in the time step immediately after $s$. In your case, $s$ is the state at time step $t$; therefore, $s'$ is the state at time step $t+1$. Since the signals $s$, $a$, and $r$ do not have an appended apostrophe (or any other additional notation), they are assumed to be the state, action, and reward associated with the current time step $t$. To confirm, simply compare the last two equations and see how $S_t$ is replaced with $s$, $A_t$ with $a$, $R_t$ with $r$, and $S_{t+1}$ with $s'$. In summary, information about the associated time of the $s$, $a$, $r$, and $s'$ signals is implicitly encoded in their written representation.

Thank you. I still have a problem about MRP Bellman equation in my comment, can you answer that? Many thanks. The key is I don't know why $v_\pi(s)$ and $v_\pi\left(s^{\prime}\right)$ can be regard as "same" in MRP, they related to rewards sum begin from $t$ and $t+1$ resptectively.

If possible, please edit the post to include your question inside the main body of the post. Also, note that this website has a very specific rule that each post can only have one question. If you feel that the question in your comment is separate from the question in the post, simply make another post with that second question. Thank you for posting and we look forward to more of your questions in the near future!

Thank you. I've edited my problem. Actually they're related problems. Hope you can answer it if you like.

@DeepQZero Great! I like your answer. I voted for you!


text:
Do diffusion models take a long time to train?
I am trying to train a diffusion model (from scratch in pytorch). UNet used is not anything too fancy, takes in images and time step as input for about 512 time steps. I am using learnable embeddings for 512 time steps. This works just fine for mnist but when moving to another dataset (my dataset is of motorbikes) it produces nothing meaningful even after 20k steps (batchsize 32). This is what i get after 20k steps on it.
Some experiments i tried somewhat contradict each other and i cant think of why this is happening.
Experiment:

overfit a small batch of my dataset. Result: model produces something intangible like the above picture
Have a dataset of about 10k images. Produces something like a bike if you really look for it. Trained this for ~28k steps (batchsize 32).
model works fine for mnist and cifar (even after running it for a short while).
Noticed there was noisy data which has nothing to do with bikes, so i manually got rid of them hoping my model would perform better. (reduced my dataset of 10k images to ~6.5k clean, quality images). Result: No changes, produces intangible images (just like first one) after training for ~55k steps (batchsize 32). This surprises me because i was expecting it to atleast be like the images produced from the noisy dataset.

I have not changed model hyperparameters when performing these experiments:

lr = 4e-4 with Adam
batch_size = 32 (for bikes)
Normalised images from [-1,1]
img_size = (3, 128, 128) # c, h, w
timesteps = 512
betas scheduler from 10e-4 to 0.02 (just like in the original paper)
using a cosine scheduler (this works a bit better than using a linear one)

I tried using self attention but that performs worse (not sure if its "worse" because it produces noise even after 12.5k steps on mnist (batchsize of 64).
I could not find a practical guide on training diffusion models, however from the information i gathered, the takeaway is that you have to train these forever and wont really work well if you run it for few hours on a mediocre gpu. I see that some of the sota diffusion models were trained on a few thousand days worth of gpu compute. Is my assumption correct ?
I am still not clear on why overfitting on a batch does not work here or why the clean dataset does not produce anything tangible. How do i know if my model is ever learning ? when do i know i should stop ?

Please, share code of at least your model structure or the Github link if you uploaded. Without seeing the program, it is hard to come out with suggestion. Also regarding your training dataset, do you make your own or download from the Internet? Please, add details of your dataset like total number of image, characteristic, average image size, and etc.

Did you find any solution to this? I have the exact same issue. With a low number of data points (2k), the model can overfit or produce some meaningful results after 4k steps. However, with a higher number of data points (180k), the model still produces very poor results after 60k steps, similar to what you described for bicycles. My findings up to this point are:

Hyperparameters are very important; with a learning rate of 1e-06 (the default for text-to-image in the Diffuser sample), the model can't even learn from a small dataset of 2k. Setting the learning rate to 5e-5 works better.
Lo

Diffusion models are very data hungry. Without data augmentation that won't be enough to train it nor would I expect realistic images from a dataset that small for something so complicated. Diffusion models also are very bad at producing noisy images, so you are going to have to train it on a set of images that are less noisy.
My suggestion?

More Data and make it clean. The more complicated, the more data you need. The original paper used millions to get realistic images. I would say at a minimum 20k is barely enough to see reasonable results without overfitting.
Make sure you self attention is using convolutions instead of MLP's. This is what they call "attention" if you look at the implementation on the official DDIM repo.
Adjust hyper parameters. Diffusion models can sometimes be very sensitive to hyper parameters. It's not easy like a VAE but not as hard as GAN. Try some more fiddling.

That's probably the issue. 10K images is not much and they are quite complex compared to the mnist number set.

To assess whether your model is learning, you can monitor the loss function during training. If the loss is decreasing over time, it indicates that the model is making progress. Additionally, you can visually inspect the generated images at different training stages to see if they progressively resemble motorbikes more accurately.
You can employ techniques such as early stopping, where training is stopped if there is no improvement in validation loss over a certain number of epochs. Additionally, you can use validation metrics to assess the performance of the model on a separate validation set and stop training when the performance reaches a satisfactory level.

This does not answer the question. Also looks a lot like this is raw output from ChatGPT or similar. Unfortunately for you, it is not possible to "rep farm" using ChatGPT, because it is exceptionally bad at answering the kinds of questions that get asked on Stack Exchange. You can still use ChatGPT to help with your writing style, but you are going to have to monitor and understand its output, then correct its mistakes. Otherwise you will just be generating low-quality noise on the site