text:
Which process is better to understand images?
What is the difference between this process of recognizing objects in a image:

(The correlation function calculate the correlation coefficient between the input and a image containing the object we want to recognize from the input which has already been stored in the program's memory)
And just using a neural network to decide if a image contains a object ?Obviously the process with the correlation function has a much quicker learning curve since images out of the error range are discarded but which can give more accurate results?

Whether or not this is even workable depends very much on what "same object" and "correlation function" mean. Typically two images being compared are not analysed for "correlation" but for e.g. pixel similarity, perhaps after downsampling. Could you please clarify, and [edit] the question.

Also, is the input to this system 2 images which may or may not depict the "same object"? Not clear from the diagram what the input is

Yes 1 image is stored in the program's memory and 1 is given as input.

From 10,000 feet up, and ignoring some details, your diagram shows a simple classification pipeline, where a small/fast model is used to filter and respond to "obvious" results before using a larger/slower model. Pipelines like this already exist, and can and do work in general. Typically they don't improve accuracy, but they can improve inference and training speed, plus reduce CPU usage.
Regarding your choice of Pearson correlation coefficient as a first stage classifier, this is not so good. Outside of a few specialist image domains where template-matching could work, such as identifying game sprites or computer text with a fixed font, it would perform very badly. When trying to match natural images - e.g. classifying a photo as of a cat or a dog - it is unlikely to do better than random. You would have to set the threshold for rejection very low to avoid false negatives, meaning that e.g. less than 1% of training images would be pre-filtered by the first stage classifier.
In fact, with natural images I am not entirely sure that there is any threshold where Pearson correlation between images - at the pixel level - would be useful to classify images as "not the same object".
Potentially if you flipped the decision, and returned a positive match for very highly correlated images, the classifier could filter out near exact image matches, e.g. if photo input and the example held in memory were essentially the same photo. If that occurred a lot for some reason in your production system, it might be worth having the first stage correlation filter for efficiency.
In principle the pipeline is a sound idea. You need a better first stage classifier, or you need to be working in an image domain where the Pearson correlation coefficient is a useful measure of image similarity. Potentially, correlation of some derived image features (e.g. number of edges/corners or other low-level matches) would perform better, but as such a filter become more and more sophisticated to drive accuracy up, you would essentially be recapping the history of computer vision until eventually your first stage filter was a simpler neural network than the second stage . . .


text:
Modern reinforcement learning for video game NPCs
Recently, I have been reading about the 1996 artificial life game 'Creatures'. The game features NPCs called 'Norns' that use reinforcement learning to learn continuously through interactions with the player and their environment. From what I have read, there haven't been any later attempts to make a game with similar AI. Have there been any more recent examples/experiments/papers on using modern reinforcement learning techniques to create NPCs that learn based on interactions with the player and game, within the main gameplay loop? What should I read if I want to create a similar system?
Relevant reading:
https://www.alanzucconi.com/2020/07/27/the-ai-of-creatures/
https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.81.1278&rep=rep1&type=pdf

I love this question because I grew up with Creatures.  I'm not sure if you've played Creatures but from my memory the creatures were relatively simplistic AI.  I would not have asked a Norm to play pong much less anything requiring higher level strategy.  For anything more complex than reacting to the environment and not algorithm based (see The Sims for algorithmic AI in the pre-2010s) the methods/compute just weren't there.
There are some big names (Sony notably) working towards RL in modern video games so we are getting to the age of it but we still aren't there in a few ways.
But hey, I'm not here to crush dreams so let's make it realistic:

Ensure there is no simpler method for how you want the NPCs to react to the player.  It's fun to jump into DeepRL but, for example, if the desired interaction is "Oh the player goes B every 3rd round, the AI should learn to go B in those cases" it's much easier to maintain a table of probabilities and choose a predefined strategy based on it.
Make the inputs/outputs as simple as possible.  The landmark DQN paper down sampled the Atari(!) screen.  We've gotten to where Youtubers train Mario kart AI on down sampled N64 screens but the point is we still struggle with really rich inputs.  Keeping the input space as small as possible is your friend for training time and to keep the AI from overfitting.
Learn to transfer train.  Players probably wont be happy with NPCs who do little to nothing for the first 12 hours of gameplay.  Establish a reasonable model at engineering time and allow a subset of the model to adjust at playtime from interaction with the player.  The reality is training even a few layers of a NN won't happen in real time but maybe between rounds or as a background task.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Error in action space for creating gymnasium environment
While trying to use a created environment, I get the following error:
AssertionError: action space does not inherit from gym.spaces.Space, actual type: <class 'gymnasium.spaces.discrete.Discrete'>
Does anyone know how to resolve this?

Hi. Programming questions or questions about specific software are off-topic here, so your question is off-topic here. We focus on the theory.

I was able to resolve the issue, it was because of inconsistency between using gym and gymnasium. Uniformly using gymnasium seems to resolve all issues.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Why do current language models no longer generate to long or short texts?
One of the biggest strengths of ChatGPT is that it generates fitting text with respect to the input query. It usually stays on topic, anwers the question completely and especially does not start talking gibberish or repeating itself.
This behaviour is different when comparing this to older LLMs. For example: GPT2 would usually only stop generating text when it hit the token limit or a predefined stop sequence. Also, it had a much bigger problem with giving repeating answers. Newer models (especially instruction tuned ones) do not suffer from this problems (e.g. llama 2).
So I have 2 questions: What mechanisms/techniques are used in current language models such that...

...the models know when to stop generating text.
...the models do not repeat themselfes and stay on topic.

I suspect it might have alot to do with instruction tuning but I am happy to hear from you.

Repeating answers were eliminated by specific mechanisms. For example, down-weighting the probability of recently generated words. Apple researchers published a paper discussing one effort to deal with this.

TLDR: if you're using a model without instruction-tuning, you'll likely need to do some additional work during inference to make sure that your outputs are coherent regardless of how old your model is. But even on GPT-2 you can get decent generations with the right technique.
You're correct in that this has a lot to do with instruction-tuning: GPT-2 wasn't trained to give helpful chatbot-style responses, it was trained to generate random web-text. Yes, there are EOS tokens that indicate the end of a sequence, but during pre-training these separate documents, which tend to be pretty long and likely aren't very coherent as chat-bot responses.
This also means that newer models that aren't instruction-tuned (like Llama-2) will suffer from similar issues, compared to their instruction-tuned variant (Llama-2 chat).
Better responses through prompting
However, it's important to note that, at least for larger models, training isn't necessary to get this kind of behavior. Lin et al. explore various methods for eliciting helpful chatbot-like responses through zero-shot prompting (instructions + a markdown-style prompt), ICL (providing examples of queries & responses), retrieval augmented ICL (ICL but retrieving examples that are most similar to the query), and their proposed method which adds a system prompt and uses specially-crafted ICL samples.

My guess is that this wouldn't work as well on less powerful models like GPT-2, as they are much worse at learning from context.
Repetition penalty
@Eponymous mentioned that there are mechanisms to downweigh repetition during sampling, but it's important to add that people have been working on developing inference-time techniques to improve generation quality since GPT-2 (and before neural models, for that matter). e.g., this famous 2019 paper, proposing nucleus sampling (and targeting GPT-2 Large & XL):

With the right techniques, you can definitely get these older models to generate coherent text (see Figure 11 in Appendix B in the above paper for some examples of different sampling techniques + the resulting generations).

LLMs have by themself a token used for end of string, so they themself are the one that decide when a generation should stop
This kind of questions are usually pretty opinionated, but the main reasons are bigger models, and more training data

However, tuning the length might also be a bias from the RLHF training some model have received

So models are now able to stop by themselves because they can predict the end of string token. But this was already possible with earlier models like GPT2. But you are saying that that one of the reason how they gained the ability to predict that token effectively bigger models with more training data?


text:
Discount factors in REINFORCE Algorithm: Difference in two definitions
I am a bit confused between two definitions of the Vanilla REINFORCE algorithm.
The first one is in the following (from this page: https://stjohngrimbly.com/model-free-RL/):

Here, at every step in an episode we calculate the returns with involving the discount factors, such that the final gradient is like the following:
$$\nabla J(\theta)=G_0\nabla_{\theta}log\pi_{\theta}(A_0|S_0) + G_1\nabla_{\theta}log\pi_{\theta}(A_1|S_1) + G_2\nabla_{\theta}log\pi_{\theta}(A_2|S_2) + \dots G_{T-1}\nabla_{\theta}log\pi_{\theta}(A_{T-1}|S_{T-1})$$
However, in Sutton´s book, the definition is like the following:

So, they explicity multiply each gradient term from each time step t, separately with the discount factor $\gamma^t$. Assuming they define $G_t$ similarly like $G_t = \sum_{k=t}^{T-1}\gamma^{k-t}r_k$, the obtained gradient will be:
$$\nabla J(\theta)=G_0\nabla_{\theta}log\pi_{\theta}(A_0|S_0) + \gamma G_1\nabla_{\theta}log\pi_{\theta}(A_1|S_1) + \gamma^2G_2\nabla_{\theta}log\pi_{\theta}(A_2|S_2) + \dots \gamma^{T-1}G_{T-1}\nabla_{\theta}log\pi_{\theta}(A_{T-1}|S_{T-1})$$
So, the gradient term in total will be different from the first one. Since in Sutton`s version they have used the discount factor reduntantly (it should be in $G_t$ already), it seems like the first version is the valid one to me, if this has not been done with a specific purpose. My question is, is there a small notational mistake in Sutton version of the algorithm, or am I missing something with the gradient formula here?

No, the difference between the gradients will be even greater, as in the second formulation the weights are updated each step, so the gradient is not a cumulative gradient, but a sort of 1-sample-SGD
In my opinion, the first one is the more accurate, as the following one does not consider the change in parameters (given $\theta' = \theta + \nabla_\theta L$, $\pi_\theta(A|S) \ne \pi_{\theta'}(A|S)$)


text:
When training a convolutional neural net, what if the channels in my image are mutually exclusive?
I'm attempting to train a convolutional neural net to perform binary classification of volumes of shape $(H, W, C)$ (i.e., height, width, channels). For the sake of this example, let's say that the volumes represent RGB images and that $C = 3$.
My data are formatted such that pixel values are mutually exclusive across channels. In other words, if the pixel at $H_0, W_0, C_i > 0$, then, all $H_0, W_0, C_{j \neq i} = 0$. Said another way, for a given pixel location $H_i, W_i$, if the green channel value is greater than 0, the red and blue channel values are always 0.
The model itself consists of multiple 2D convolutional layers (each followed by max pooling operations), followed by two fully-connected layers.
Of course, I could always just collapse the channels before training by taking the sum of pixel values across channels (creating an image of shape $(H, W, 1)$). However, I would like to keep the channels separate as they represent very different characteristics of the image. Basically, I'd like the network to "learn" the arrangement of non-zero values across the channels. I've also considered doing a $1 \times 1$ convolution across the multi-channel input as the first step in the model, followed by my various CONV-POOL operations.
As far as I can tell, this seems like an atypical approach, so I'm wondering if there are obvious issues with training the CNN on mutually-exclusive multi-channel images? Does it even make sense to do this? If available, I'd love to know of any reading material or resources about training CNNs on multi-channel images with mutually exclusive channels. Thanks!

so if you have a layer with 10 filters, you want to have 30 channels as output, the first 10 being the R channel convolved with the 10 filters and so on?

Ah, @Alberto are you referring to something like Depthwise Convolution (as described here)? I hadn't thought about that, but it might make sense to apply the convolution to each channel separately!

I mean, imho, naive convolution might be better, as it is at least as powerful as Depthwise Convolution, however it seemed like it was fitting your requirements


text:
Does SVD of sparse matrices have some special properties?
If I have a big, but sparse matrix and I decompose it using SVD, would the three matrices have some special properties other than their regular ones ?


text:
My first experience with gym environment has raised many questions, and I need some guidance
As I'm new to the AI/ML field, I'm still learning from various online materials. In this particular instance, I've been studying the Reinforcement Learning tutorial by deeplizard, specifically focusing on videos 8 through 10.
After attempting to replicate the example that demonstrates how to train an agent in the gym's FrozenLake environment, I encountered some unexpected issues. Although I successfully debugged the code and got the algorithm to work, I can't confidently claim that I fully comprehend if this is the correct approach. Therefore, I have specific questions related to this exercise that I intend to post within this single thread. Please feel free to answer any of the questions at your convenience.
Q1
What is this gym environment warning all about, and how should it be properly addressed?
/home/boris/.local/lib/python3.11/site-packages/gym/utils/passive_env_checker.py:233: DeprecationWarning: `np.bool8` is a deprecated alias for `np.bool_`.  (Deprecated NumPy 1.24)
  if not isinstance(terminated, (bool, np.bool8)):

So far, I've discovered two methods to handle this warning: One is to simply  suppress the warning as follows:
from warnings import filterwarnings
filterwarnings(action='ignore', category=DeprecationWarning, message='`np.bool8` is a deprecated alias')

Another is to replace the gym environment with the gymnasium environment, which does not produce this warning.
#import gym
import gymnasium as gym

This brings me to my second question.
Q2
Why are there two environments, gym and gymnasium, that do the same thing?
Most online examples use gym, but I believe gymnasium is a better choice. Am I mistaken?
Q3
What's the distinction between "terminated" and "truncated"? I've made numerous attempts, but I couldn't find a situation where they diverge. They consistently evaluate to either both True or both False.
In the current 'gym' version, I have to use "FrozenLake-v1" (originally, "FrozenLake-v0" was used in the tutorial), and this modification causes the code to break because the environment now expects 5 values instead of 4. Initially, I fixed this issue as follows:
new_state, reward, done, UNKNOWN, info = env.step(action)

I discovered later that the correct name for this variable is "truncated." Consequently, I adjusted the code as follows:
new_state, reward, terminated, truncated, info = env.step(action)
done = terminated or truncated

Nonetheless, I remain perplexed regarding the usage of "terminated" and "truncated." While experimenting with various algorithm options, such as max_steps_per_episode, I was unable to get them to differ. In all cases, they either both evaluate to True or both to False.
Q4
If I want to continue training the agent in stages and test its intermediate performance, how can I do that? I was thinking it's just a matter of saving the q_table, then loading the saved q_table and replacing the current q_table initialization with the new one before resuming training. Or is it not that simple?
saved_q_table = anydbm.open(filename, 'r')

#q_table = np.zeros((state_space_size, action_space_size))
q_table = pickle.loads(saved_q_table)

Q5
Is there a way to get some documentation or help about the environment?
I'm seeking a resource that I can print and reference while coding to gain a better understanding of the environment's parameters. For instance, in scikit-learn, you can achieve this with the following code:
from sklearn.neighbors import KNeighborsRegressor
from pprint import pprint
pprint(KNeighborsRegressor().get_params())

Resulting in following output:
{'algorithm': 'auto',
 'leaf_size': 30,
 'metric': 'minkowski',
 'metric_params': None,
 'n_jobs': None,
 'n_neighbors': 5,
 'p': 2,
 'weights': 'uniform'}

It would be great if one could get relevant agent/states parameters from the gym environments in a similar way.
Q6
How to use render_mode="human" correctly? Currently, I'm using render_mode="ansi" and rendering the environment as follows:
env = gym.make("FrozenLake-v1", map_name="8x8", is_slippery=True, render_mode="ansi", desc=None)

print("\033c")
env.render()
print(env.render())

However, when I switch to render_mode="human", the environment automatically displays without the need for env.render(). Consequently, the environment renders during training as well, leading to extremely slow training. How can I disable this automatic rendering when using render_mode="human"?
Q7
Why doesn't this algorithm work for an 8x8 map size?
env = gym.make("FrozenLake-v1", map_name="8x8", is_slippery=True, render_mode="ansi", desc=None)

It appears that the agent is being trained, but it fails to learn anything.
*** Average reward per thousand episodes ***

1000 :  0.0
2000 :  0.0
3000 :  0.0
4000 :  0.0
5000 :  0.0
6000 :  0.0
7000 :  0.0
8000 :  0.0
9000 :  0.0
10000 :  0.0

I can't seem to figure out what needs to be modified. The q_table is the correct size:
q_table = np.zeros((state_space_size, action_space_size))

I've also increased max_steps_per_episode to 3000. Is it possible that num_episodes = 10000 isn't sufficient?
By the way, here is the full code I have so far:
#!/usr/bin/python

# -*- coding: utf-8 -*-

import numpy as np
#import gym
import gymnasium as gym
import random
import time
from warnings import filterwarnings

#filterwarnings(action='ignore', category=DeprecationWarning, message='`np.bool8` is a deprecated alias')

env = gym.make("FrozenLake-v1", map_name="4x4", is_slippery=True, render_mode="ansi", desc=None)
#env = gym.make("FrozenLake-v1", map_name="4x4", is_slippery=True, render_mode="human", desc=None)

action_space_size = env.action_space.n
state_space_size = env.observation_space.n
#print(action_space_size)
#print(state_space_size)

q_table = np.zeros((state_space_size, action_space_size))

num_episodes = 10000
max_steps_per_episode = 3000

learning_rate = 0.11
discount_rate = 0.99

exploration_rate = 1
max_exploration_rate = 0.99
min_exploration_rate = 0.01
exploration_decay_rate = 0.002

rewards_all_episodes = []

# Q-Learnig Algorithm
for episode in range(num_episodes):
    print("*** Episode ",  episode+1, " ***", end="\r")
    state = env.reset()[0]

    done = False
    rewards_current_episode = 0

    for step in range(max_steps_per_episode):

        # Exploration-Exploitation Trade-off
        exploration_rate_threshold = random.uniform(0, 1)
        if exploration_rate_threshold > exploration_rate:
            action = np.argmax(q_table[state, :])
        else:
            action = env.action_space.sample()
        new_state, reward, terminated, truncated, info = env.step(action)
        done = terminated or truncated

        # Update Q-Table for Q(s,a)
        q_table[state, action] = q_table[state, action] * (1 - learning_rate) + learning_rate * (reward + discount_rate * np.max(q_table[new_state, :]))

        state = new_state
        rewards_current_episode += reward

        if done == True:
            break

    # Exploration rate decay
    exploration_rate = min_exploration_rate + (max_exploration_rate - min_exploration_rate) * np.exp(-exploration_decay_rate * episode)

    rewards_all_episodes.append(rewards_current_episode)

# Calsulate and print the average reward per thousand episodes
rewards_per_thousand_episodes = np.split(np.array(rewards_all_episodes), num_episodes/1000)
count = 1000
print("*** Average reward per thousand episodes ***\n")
for r in rewards_per_thousand_episodes:
    print(count, ": ", str(sum(r/1000)))
    count += 1000

# Print updated Q-Table
print("\n\n*** Updated Q-Table ***\n")
print(q_table)
input("Press Enter to continue...")

num_episodes = 5
max_steps_per_episode = 99

for episode in range(num_episodes):
    state = env.reset()[0]

    done = False

    for step in range(max_steps_per_episode):
        print("\033c")
        print("*** Episode ",  episode+1, " ***\n", step)
        env.render()
        print(env.render())
        time.sleep(0.1)

        action = np.argmax(q_table[state, :])
        new_state, reward, done, unknown, info = env.step(action)

        # Update Q-Table for Q(s,a)
        q_table[state, action] = q_table[state, action] * (1 - learning_rate) + learning_rate * (reward + discount_rate * np.max(q_table[new_state, :]))

        state = new_state
        rewards_current_episode += reward

        if done == True:
            print("\033c")
            print("*** Episode ",  episode+1, " ***\n")
            env.render()
            print(env.render())
            if reward == 1:
                print("*** You reached the goal! ***")
                time.sleep(3)
            else:
                print("*** You fell through a hole! ***")
                time.sleep(3)
            #clear_output(wait=True)
            break

        if step == max_steps_per_episode - 1:
            print("*** You failed to complete the task! ***")
            time.sleep(3)

Q1. What is this gym environment warning all about, and how should it be properly addressed?

The development of gym has ended I think. Anyway the error is about an old usage of numpy, so probably you have installed a newer version of it.

Q2. Why are there two environments, gym and gymnasium, that do the same thing?

In principle, they do the same thing but gym was developed by OpenAI and gymnasium by Farama Foundation, which inherited gym basically. You should use gymnasium from now on, since is in actual development. This means that initially gym and gymnasium were almost identical but now their are diverging, even if the underlying concepts are still the same.

Q3. What's the distinction between "terminated" and "truncated"?

truncated indicates that the episode ended due to a time limit, i.e. the current timestep have exceeded the time limit so the episode ends. This also indicated that you have to bootstrap the Q value estimate.
terminated instead is when the agent have reached a terminal state, therefore the episode naturally ends. You should use both to understand when an episode terminates, e.g. if truncated or terminated: ..., but the two flags are useful to determine when or not to bootstrap: you don't do that on termination, for example.

Q4. If I want to continue training the agent in stages and test its intermediate performance, how can I do that?

You need to save the Q-table (or, in general, the weights/parameters of a function approximator like a neural-net) but also the current value of the episilon (if using some schedule, so if changing during the episodes) and similar. Then you instantiate the agent, load the various parameters, and continue training where you left.

Q5. Is there a way to get some documentation or help about the environment?

Try help(env), the official website (https://gymnasium.farama.org/index.html), or read the env's source code.

Q6. How to use render_mode="human" correctly?

If you set render_mode="human" gymnasium will render at each step() and even reset(): this is something that gym not used to do.
I'm not sure but you may need to wrap the env with this gymnasium.wrappers.RenderCollection (docs) to trigger the rendering only when render() is called - I haven't tested that, so I can be wrong.

Thanks a lot! To summarize,
/ - gymnasium environments are the way to go
/ - help(env) prints documentation about environment
/ - need to learn about bootstrapping the Q value estimate to use truncated flag
/ - to resume training need both Q-table and epsilon value
/ - check gymnasium.wrappers for advanced rendering options

@BorisL. Yeah, that's all you need to do and know in short. Also bootstrapping is something you'll find in Q-learning and TD-learning. I think there is some good answer about it, in AI SE.


text:
Variational Lower Bound in VAE for Gaussian latent prior
From Bishop's recent book on Deep Learning, it says the ELBO for Gaussian latent prior can be approximated by
$\frac{1}{L}\sum_{l=1}^L \ln p(x_n|z_n^l,w) + KL(q(z_n|x_n,\phi)||p(z_n))$
where $n$ are data samples, and the KL term have a closed form for gaussian prior p(z). My question is: what exactly is $\ln p(x_n | z_n^l,w)$? I don't know what this distribution is exactly, so one makes any assumption about the distribution of $x$?
update:
In the original paper (Auto-Encoding Variational Bayes), they claim to use MLP gaussian output for the decoder. What exactly that means?

My question is: what exactly is $\log p(x|z,w)$? I don't know what this distribution is exactly, so does one make any assumption about the distribution of x?

I completely understand the question. I think the main problem of Variational Autoencoders is that the selected notation for explaining the math is between Probability theory notation and standard Machine Learning notation. To understand the concepts from it, it can be somewhat confusing. What exactly is $\log p(x|z,w)$? Well that part is called the reconstruction error of the ELBO estimation. Let's see it in parts to fully understand it.
Inside the natural logarithm there is the probability $p(x|z,w)$. This is called the posterior of the decoding phase. This is the probability of getting $x$ given $z$ with the parameters $w$ that are the parameters of the decoder network. Then the problem is, what exactly are $z$ and $x$ given that we are working with images?
Well, $z$ are the parameters for the Gaussian distribution generated by the encoder. $z$ are two vectors - the standard deviation vector and the mean vector. These parameters generated by the encoder are then used to calculate the posterior of the decoder $p(x|z)$, which is the probability of generating the original image given $z$. and $x$ is the data we give as inputs.(1 footnote).
After doing the reparameterization trick:
$z = \mu + \sigma \odot \epsilon$ where $\epsilon \sim \mathcal{N}(0,1)$
That is just a smart way of sampling from a distribution, in this case a Normal distribution (2 footnote).
Then that $z$ vector is passed through the decoder for adding the highly nonlinear distribution at the end and output a probability. Now the question is, what is the form of that probability distribution? In this case we used a sigmoidal function for outputting a Bernoulli or binomial distribution (3 footnote). Why do we choose the sigmoid function? Because first, all the pixels in the generated image must be independent, meaning that when assigning a probability to a pixel we don't want others to influence them like in the case of a softmax function. A second reason is that the sigmoid function has a range from 0 to 1 and when working with images the pixels are bounded from 0 to 255, meaning that we can normalize all the data at the beginning and the inductive bias is more powerful.

In the original paper (Auto-Encoding Variational Bayes), they claim to use MLP gaussian output for the decoder. What exactly does that mean?

Citing the paper it says:

"The decoding distribution $p_\theta(x^{(i)}|z^{(i)})$ is a Bernoulli or Gaussian MLP, depending on the type of data we are modelling."

This means that the decoder network has all activation functions consist of sigmoidal or GELU functions. This was terminology that was very common in the early times of Machine Learning. Now most likely you would not hear the term Bernoulli MLP or Gaussian MLP because most neural networks today consist of various activation functions.
Footnotes:

Although they used mean sum in this paper, they really just using the ELBO estimation for calculating the loss. The thing is that ELBO uses the log of the expectation of a random variable. Binary Cross Entropy also uses the expectation of a random variable. The reason we can use BCE in the loss is because we interpret the original data as a probability and because the final output is also a probability we can perform BCE knowing the probability of each data point. As the contrary of using a mean sum that is making the assumption that the probabilities of all the data are the same because we don't have real information of the distribution of the original data.

The reason we pick a Normal Distribution is because it is a very known distribution which we can operate with. For example it is very easy for us to calculate the KL Divergence on the loss function using a Normal Distribution.

Technically if we try to be as theoretical as possible we should pick the Gaussian distribution with the GELU activation function but because of the advantages of using sigmoid for image data we use a Bernoulli distribution with sigmoid. Another thing is that most of the time the fully theoretical formula doesn't work as well as nitpicking with different architectural decisions. What I am trying to say is that I have seen VAEs being implemented with MSE loss functions even when it is not a regression problem but a classification one. So when implementing an efficient algorithm for a given data you can be less strict with the theory for gaining more performance at the expense of having a lack of theoretical background.

In my understanding, as $p(x|z,w)$ being the decoder then it is not a probability distribution, but a function of $z$ parametrized by $w$, because the decoder network is deterministic, why picking GeLu or sigmoid makes it a well defined probability distribution, and more than that, a Gaussian distribution? Is that a simple convenient assumption to introduce the MSE loss?

If you Assume $p(x|z,w)$ is normal with mean $decoder(z)$ and covariance equals to identity, then its log takes the form of MSE loss, but this does not make any sense, its a nonsense assumption.


text:
Why policy gradient theorem has two different forms?
I have been studying policy gradients recently but found different expositions from different sources, which greatly confused me. From the book "Reinforcement Learning: an Introduction (Sutton & Barto Chapter 13)", we get the following policy gradient:
$$
\nabla J(\theta) = \mathbb E_\pi\left[G_t\nabla\log\pi(A_t | S_t, \theta)\right].
$$
As we can observe from the equation, it does not relate to trajectory distributions. However, a more intuitive and widely-used introduction to policy gradient starts from defining the distribution of trajectories: $p(\tau)$. For example, in OpenAI Spinning Up, the policy gradient has the form similar to the following equation:
$$
\nabla J(\theta) = \mathbb E_{\tau \sim \pi}\left[\sum_{t=0}^{T}G_t\nabla_\theta\log\pi_\theta(a_t | s_t)\right].
$$
The confusion comes from the fact that the first policy gradient does not have a summation over timestamps and is not sampling from trajectories, but the second samples from trajectories and has a summation.
I did find some relevant questions about this confusion, but none of them seemed to have a good answer. Also, I could not identify any source that explained the difference/connection between the two forms.
My question is why are there two different ways to describe the policy gradient and are the two forms mathematically equivalent?
Update
I found a great RL theory book (draft) written by some expert Professors in this field that shows two different formulations: https://rltheorybook.github.io. Also, one of Nan's lecture notes shows the correlation between the two forms: https://nanjiang.cs.illinois.edu/files/cs598/note6.pdf.

I also have a bit of trouble understanding this, because the wsummation also appears in the GAE paper. Could you help me understand how they are the same? Which page in the RL theory book are you referring to?

After reviewing Nan's lecture notes above, here are my conclusions
let $d_t^\pi$ denote the distribution over $s_t$ and $a_t$ induced by policy $\pi$ from the initial state distribution $d_0$, the first form is formally expressed as
\begin{equation}
\nabla J(\pi)=\frac{1}{1-\gamma}\mathbb{E}_{(s,a)\thicksim d^\pi}\left[(\nabla\log\pi(a|s))Q^\pi(s,a)\right].
\end{equation}
where $d^\pi$ is the normalized state-action distribution shown below
\begin{equation}
d^\pi = (1 - \gamma) \sum_{t' = 1}^{\infty} \gamma^{t' - 1} d_{\pi}^{t'}
\end{equation}
We can see that this form sums over timestamps in state-action distribution $d^\pi$, while the second form sums inside the expectation. They capture long-term information in different ways.
Besides, since $\frac{1}{1-\gamma}$ is a constant, we can ignore it and treat it as a part of learning rate.


text:
Do WGAN gradients require multi-variable calculus?
The generator tries to maximise this function D(G(z)). That much I understand.
But how can the critic maximise D(x) - D(G(z)). This has two inputs: a real instance and a noise sample. The model takes an input in order to backprop, otherwise it wouldn't be able to update the first layer. So which input do I choose? Am I missing something or does this require multi-variable calculus?

A function can be optimized, even if it has two inputs!
First, note that the problem in question already occurs with traditional GANs as well. It might be easier to understand what is going on if you have an 'implementation' example. You can check my tutorial on GANs on GitHub.
Specifically, for training the discriminator we indeed take to inputs:

A batch of synthetic samples $\hat x$. This we can create by taking a batch of noise vectors, which we pass through the generator ($\hat x = G(z)$).
A batch of real training samples $x$ from the dataset.

Now we pass both through the discriminator. Obtaining both $D(\hat x)$ and $D(x)$.
Now we can calculate the loss for both outputs and combining them. For WGAN, this means taking the mean over the outputs, and subtracting them, so $mean(D(x)) - mean(D(\hat x))$. This we can backpropagate, as it is simply one number. Note that backpropagating one of these terms (i.e. just $mean(D(x))$ ) is very similar to backpropagating both of them, as it is only one addition operation different. You need just the same amount of calculus for the one as for the other.

Thank you for your efforts. I looked through your repo and found out that I can add two cost functions together for backwards propagation. After some more research I learned that this is equivalent to backwards propagating twice. e.g. (loss1 + loss2).backprop(); optimizer.step(); is the same as loss1.backprop(); loss2.backprop(); optimizer.step(); God bless.

They are approximately the same, but your second option will train 'faster' because the learning rate is 'doubled' (applied two times). Operationally, it is the same.


text:
What is GNN Cheatsheet in PyG Docs
I am going through the Pytorch Geometric documentation: https://pytorch-geometric.readthedocs.io/en/latest/index.html which is built on Pytorch .Here they mentioned about GNN Cheatsheet: https://pytorch-geometric.readthedocs.io/en/latest/cheatsheet/gnn_cheatsheet.html. Can someone explain what the purpose of this is as it shows a table and there are tick marks if checked. Please explain how to understand the first function with the related attributes and then I can proceed further. Thank you.

Would put some additional explanation on " to replace the links with text so that people can easily understand."?

@Cloud Cho  Thanks for your feed back I will look through it please help me in the first question I've asked regaeding GNNsheet.

Regarding "what the purpose of this is as it shows a table and there are tick marks if checked.", it sounds like that the tick marks indicating convolution operation support. This means you could train the specific graph with specific convolution method.
  For example (directly from same webpage), "SparseTensor": If checked (✓), supports message passing based on torch_sparse.SparseTensor, e.g., GCNConv(...).forward(x, adj_t).
  In other words, "SparseTensor": If checked (✓), you could run convolution operation using torch_sparse.SparseTensor, e.g., GCNConv(...).forward(x, adj_t)
Here is PyTorch own definition of Message Passing: Generalizing the convolution operator to irregular domains is typically expressed as a neighborhood aggregation or message passing scheme.


text:
Why solely a one-step-lookahead in value/policy-iteration?
In value iteration and policy iteration we solely consider a one-step-lookahead where the lookahead is from the previous iteraiton and therefore need to sweep over all states and iterate this procedure.
Why dont we do one sweep over all states but instead of using a one-step-lookahead we build a complete scenario-tree thus a full-step-lookahead.
Question Is it due to efficiency, computational challenges(obviously you cant if the horizon is infinite), it is due to loops?

It's due to the Principle of Optimality in Dynamic programming

@alberto the principle shows that a single step look ahead works, not that it is the only or best choice. Also better to write up an answer, rather than a half answer as a comment. If a question seems stuck, and the best possible is to give a pointer, then it's kind of ok, because it might help the OP, which is better than nothing

@NeilSlater, no, it shows that if that property holds, that full traversal, or a recursively solution, are equivalent... I cannot find an example where a full traversal per each state is more accurate then a recursive DP solution

@Alberto - equivalent results, yes. Not necessarily equivalent performance, or equivalent in other ways. In reinforcement learning you also add the policy handling, which is not core to dynamic programming.

i.e they will reach the same result in all cases...the main question still sticks, any input on it? -> a part of the reasons must be the reasons i mentioned? it strikes me that i cant find any introductory RL-courses talking about this..

@NeilSlater are you aware of any resources i can check at that compares the two approaches, seems so strange that noone really discusses it, i cant see intuitively how it is obvious that a onestep lookahead is better in all cases

@hugh I have not answered, because not 100% certain, I think that the reduced complexity wins most of the time. And if you are looking at the kind of problem that might benefit from deeper traversal, chances are the space is not easy to run full sweeps on. There are approximate versions of DP, and versions that don't run full sweeps, but they are kind of specialist compared to e.g. DQN.


text:
Is this a bandit problem or a MDP?
I am trying to understand if this problem can be casted both as a bandit problem as well as an MDP.
Lets assume that we are trying to optimize sales $y_t$ based on investments $x_{1, t}, x_{2, t}$ over some horizon $H$. To model sales for timestep $t$ we have the following function $y_t = \beta_0 * x_{1, t} + \beta_1 * x_{2, t}$
where as we have the following priors $\beta_0 \sim Beta(2, 1)$ and $\beta_1$ consists of dummies associating with day of the week, each dummie is distributed as $\sim Beta(2, 1)$, thus $6$ dummies in total. We can assume the parametrization is correct, however, the priors may be far from the true parameters governing the true sales equation. Thus a typical exploration vs exploitation dilemma.
Lets assume a normally distributed likelihood $N(y_t| 1)$.
We are solely allowed to play $3$ actions every timestep:
$Action \ 1: {x_1 = 0, x_2 = 1}, Action \ 2: {x_1 = 1, x_2 = 0}, Action \ 3: {x_1 = 1, x_2=1}$
Lets assume we want to plan over an horizon of $5$ timesteps.
Every timestep(daily) we take an action and every timestep we receive a reward.
We have an total action budget $B$ over the whole horizon that we should not exceed.

Question Should this problem be casted as an MDP or as a bandit and why?

My intuition tells me that due to the timevarying parameter $\beta_1$ and the total budget constraint $B$ this should be casted as an MDP since actions today have an impact on in this case our "physical state" tomorrow. However, i have seen total budgets for bandits before which makes me doubt. Is the budget constraint really the only thing that makes this impossible to model as a bandit..


text:
How to plot average return vs step figures in reinforcement learning？
I know return is total discounted reward of an episode,so it is easy to plot return vs. episode.But in many papers,they provide figures of average return vs step,like this:
Based on my knowledge,return is discounted cumulative rewards of an episode,it can't be calculated until the end of episodes,so I feel puzzled about how to calculate average return at each training step.
Does return of each training step mean discounted cumulative rewards collected from current training step to the end of current episode?If an agent is trained with N episodes,and each episode has T step,then I can calculate return of the (i-1)Tth step to the iTth step when the ith episode ends and plot return vs. step if all N episodes finish,is it right?

Please link the paper where you found the graphs. Answers can then explain those graphs with more confidence. The answers may also apply to other graphs you have seen, but you should always check the paper details  to find what definitions are being used

Here is the paper: https://arxiv.org/abs/2010.02966

You have not linked the paper where you got such images, however, usually in DRL publications with these environments, the x-axis refers to training-steps, not environment-steps

Here is the paper: https://arxiv.org/abs/2010.02966

I'm new to DRL. If x axis refers to training steps,then how to calculate return at each training step?Should I use the policy learned at this training step to interact with the environment for a complete episode to get return of current training step? Thank you.


text:
Can Vision Transformers be used to extract features?
Can Vision Transformers be used to extract features, just like with VGG ?
I am interested in using this vision transformer in extracting features (https://huggingface.co/google/vit-base-patch16-224)
Since VIT is used in classification problems, I don't think that it will properly encode features like for example VAEs.
I want to use these features in image generation, something akin to super resolution.

Yes, of course they can be used to extract features, just like convolutional networks, even in supervised settings.
ViTs are not exclusive to classification, their intermediate layers can also be used as high quality features, just to show this, if your search in Google Scholar for "vision transformer transfer learning", at the time this question was written, there are 66700 papers.


text:
How to disapprove some who says that AI is capable of irrational thought and emotions
A teacher of mine said something of about AI being self sufficient and operating on experience. They backed this statement by this example-a car programmed to avoid ramming into apple, mango and banna tree also stopped at a walnut tree which was not what it was programmed to do, thus ai proves itself to be capable of faulty reasoning or associative learning, which technically is not possible for ai as it can only generalize to some extent if it is trained on diverse data and designed to recognize patterns.
They also mentioned something regarding emotions, attributing it to the video released last year where Sophia (the robot) said that it wanted to have a baby (https://youtu.be/q6xWrhD7p5o?si=rEFI7ReRzEQ4e9DQ)
Which is bull and I know.
Problem is that they insist that I provide valid proof that's not too outdated. I would like your help in this regard. Please help me find articles or any books that are new and up-to-date so that I can shut his non-existent brain off before he corrupts others with his faulty teachings.
On the side note I would like to mention this whole incident started when they asked the question (what is the computer?) and my friend replied "it's a dumb machine" and he scoffed and laughed at the response as if hearing some funny joke. He also mentioned that while it might have been so in the past, the present is vastly different and that computer is no longer dumb and instead can think and do it's work on its own.

AI are based on the idea of distribution matching, no matter what you are doing.
Is it a discriminative problem? then match $p_\theta(y|x)$ with the true one, using examples
Is it a generative problem? then match $p_\theta(x)$ with the true one, using examples
The only one that is not under this umbrella, is Reinforcement Learning, which is indeed sold as one of the most promising ways to achieve AGI (big word, take it with a grane of salt)
Indeed, evolution can be formulated as a RL process, trying to maximize the probability of procreation (only the strongest ones, that are able to arrive to such moment, will keep their genes "alive")
However, this does not mean that we are at that point, not even close.
What your teacher is observing is just generalization, which is far from extrapolation
If you show to an NN a light that is turned on every time it rains, that NN cannot understand if it's the light that turns on when it rains, or if it rains when the light turns on (the only way is, for example for LLM, give enough common knowledge to the AI to reasonably deduce it)
However, on the other hand, if there is no evidence that any AI at the moment can extrapolate (which is not equivalent to "generalize"), there are some Researcher that thinks that LLM training is a sort of "compression", and the best compression possible will indeed lead to abilities of deductive reasoning (however, without interaction with the world, I still see some counterexamples)... check any podcast with Ilya Sutskever, cofounder of OpenAI, on ChatGPT abilities or LLM in general


text:
How to do image classification with optional metadata?
I have a vanilla image classification problem. The image may optionally have some numerical metadata associated with it. We don't assume uniform availability of this metadata, i.e., the model should be able to produce an output even in its absence. What type of neural network should I explore for this problem? I have been trying various Graph Neural Networks such as Relational GCN [1] and Heterogeneous Graph Transformer [2] with the metadata feature representation obtained using an MLP. Unfortunately, it doesn't outperform the image-only model. Any ideas are much appreciated!
[1] Schlichtkrull, Michael, et al. "Modeling relational data with graph convolutional networks." The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3–7, 2018, Proceedings 15. Springer International Publishing, 2018.
[2] Hu, Ziniu, et al. "Heterogeneous graph transformer." Proceedings of the web conference 2020. 2020.

You could use your existing CNN architecture and concatenate the metadata with the flattened last convolutional layer. Add a flag Boolean feature for whether the metadata is present, and if it is not, then set other values of the feature to neutral value, such as the mean (if you are using standard scaler to normalise features, then that will be all zeros)
In general it's ok to have inputs going to different layers within a network, or having it split, combine or skip between layers. This gives you a lot of flexibility to handle different input types without needing to explore more complex solutions such as graph networks

Train a self-supervised model to generate embeddings of your metadata
Train a model to generate embeddings of your images (or use/fine-tune a pretrained image model)
Train a prediction model that takes in concatenated [image_embedding, metadata_embedding] to make predictions
For images with no metadata, use a vector of zeros in place of the metadata embedding.
If your training data has "missing metadata" images at a rate similar to your inference environment, the model should work things out during training.
If your training dataset is enriched for images with metadata, add a dropout step to randomly zero out the metadata embedding so your model trains on a realistic proportion of "missing metadata" images.


text:
How to find coefficients for a RL-agent environmnet penalty for a buildings district heating peak-shaving problem?
In short, we are trying to use Reinforcement Learning to try to control the heating of a building (district heating) with the input buildings zone temperature, outdoor temperature. To not use the real building during training of the RL-algorithm we are using a building simulation program as an environment.
The building simulation program has inputs:

Zone thermostat heating and cooling setpoint (C)
Hot water pump flow rate.

Outputs from the building simulation program are:

Zone temperatures (C)
-Outdoor temperature (C)
Hot water rate (kw)

The aim of the RL-algorithm is to make a more efficient control of the buildings district heating use, then the current district heating control function. The primary goal is to make the RL-algorithm peak-shave the district heating use.
We are using ClippedPPO as an agent using a RL-framework. As a comparison we have district heating data from one year from the building we want to control. The building is modelled in the building simulation format.
Action space of the RL-algorithm is:

Hot water pump flow rate
Zones heating and cooling temperature SP

Observation space of the RL-algorithm is:

Zone Air temperature
Outdoor temperature
Heating rate of hot water

In each timestep the RL-environment takes the input from the building simulation program and calculates a penalty from the observation state that is returned to the agent. The penalty is calculated as a sum of 4 different parts. Each part has a coefficient that by art I have been trying to figure out. Some of parts are for example the -coeff1heating_rate^2, -coeff2heating_derivative and -coeff3*unfomfortabletemp (large penalty when indoor temperature less than 19C)
The problem is that we are seeing heating with high peaks that we want the RL-algorithm to shave. So if anyone has any idea on how to get this working or give some insight on how to progress.

Please, put your specific question in the title. Thanks.


text:
Help for my custom ENV with GYM, trying with DQN
I created a custom env to simulate a sort of blocks using gym enviroments.
My environment consists of an 8 x 8 observation space, which would be the stacks of blocks and the height of each stack.
yard: spaces.Box(low=-np.inf, high=np.inf, shape=(8, 8), dtype=float)
The action space is discrete, being mapped to a dictionary that contains all POSSIBLE movements in this environment, which total 168, namely:
self.ACTION_LOOKUP = {0: "(0,1,0) - top slab from stack 0 to top of stack 1",
2: "(0,1,1)" - two
                               1: "(0,2, 2) - top three slabs from post 0 to top of stack 2"}

My reset, in addition to starting the variables that are used throughout the environment, generates a "random map", based on the seed.
This map is the arrangement of blocks and location of which must be unlocked in the correct sequence.
Once this is done, the step is designed for 3 main rules

Do not allow an action that is not in space to be performed (this scenario only occurs when the rendering mode is like HUMAN)
Do not allow an action to be performed starting from an empty stack or with fewer blocks than what is being requested.
Do not allow a stack to receive more blocks than its capacity.

In this case the action is not performed and the observation space remains the same.
The reward function, which I have already implemented countless times... is currently designed to capture the general state of the environment and no longer the UNIQUE MOVEMENT (the results are not the same when compared in the evolution of reward_mean).
In this case, it is a complex punctuation composition:

If the above validations are checked (of invalid movements, return -1)

Otherwise, I EXECUTE THE CHANGE IN THE STATE OF THE ENVIRONMENT, I calculate the reward (after the movement has been executed), according to the following composition:

1 point for moving a correct pile (where it contains a block)
Percentage 0 to 100, of how close I can complete unlocking the blocks - this calculation is based on proximity to the surface
Additional points for partially unlocking a block
Reducer (-1) for the current step, my intention is for the game to be completed in the least amount of steps possible.

@NeilSlater thanks, I reviewed my question. Take a look if it makes sense now.

You don't explain how you convert state to neural network features. Which makes me think that you are doing no feature scaling, using the state directly as input. This is a common mistake when implementing deep RL. The neural network can perform badly with inputs outside of range -2 to +2. So one thing you can do is scale the state vector before using it as input to the neural network.
Another thing that might be useful because you have a large number of actions is to look at double DQN, where you run two updating  estimators, using one to select the maximising action but the other to generate the TD target from it. This reduces the bias from taking the "best" action from lots of approximations, and then using the very likely overestimated action value from the same estimate.
I can't rule out other sources of error here. Other possibilities include that your state is missing key data that impacts future reward. I'm not sure if that might be the case from your description.

used this to convert to 1d array
        return np.concatenate([
                state["yard"].reshape(-1)
                ])  ```

just add an image to my question.

@Betnem If you are using values of 30 or over as inputs to the neural network, my first paragraph definitley applies. This will make the NN perform badly.

thanks a lot.
I will make a change in my state to 0 to represent void slots and another value to represent occupied slots... thanks!

Didnt get any good results...

@Betnem That's a shame. Sorry I am out of ideas what could be wrong. I think you need someone closer to your project to work with you, or just keep trying ideas. I hope the little bit I've been able to suggest turns out to be a useful step towards getting it working


text:
Aren't context lengths for transformers an artificial restriction?
Let's focus on the case of decoder-only transformers, where I am using algorithm 10 from "Formal Algorithms for Transformers" by Mary Phung and Marcus Hutter as a reference.
: https://i.sstatic.net/ZWC9o.png
Previously I thought that the maximum context length is very much built into the transformer, for example as the dimension of a layer of weights. After studying this algorithm I am surprised because it seems more like an artificial restriction! Because this is a topic of active research I would like to know if I am misunderstanding something.
The way I see it, if I had access to the weights of GPT-2 right now, I could almost execute it on any number of tokens I like right away (If I had sufficient memory to compute this). The MHA algorithm is just carried out over a larger sequenece. There are only two issues, which are points where the context window $l_{max}$ appears:

The positional encoding has only $l_{max}$ positions
During training the weights were never optimized to attend over more than $l_{max}$ tokens.

But these issues seem rather easy to resolve:

Use some positional encoding which has infinitely many positions. The first encoding vectors are nicely spread around while the later ones are closer to eachother, due to the nature of fitting an infinite sequence of vectors into more or less a unit ball/sphere. But this is not an issue: It is natural for the positional encoding to become more vague as the token is further and further in the past.

Train 50% on context lengths around $l_{max}$, 25% on context lengths around $2 l_{max}$, 12.5% on context lengths around $4 l_{max}$ and so on...

I can imagine the following issues appearing:
A) Memory becomes larger than what is available on a single "unit" (GPU?) so you have to start moving data back and forth to execute your transformer, which is just terribly inefficient during training and also inference, so it is really pointless to train on such large context windows.
B) Perhaps the transformer just doesn't learn well with this procedure for some reason.
These issues are still rather "soft" issues though. As far as I can tell, I could use the architecture of GPT-2 (modified positional encoding) to create 1000000 context window LLMs, in theory. So, am I missing something?
Thank you!

In theory you can spend your way to any context length. That said, transformers show really bad information retrieval/context understanding for the middle of long sequences. They show a bias towards information at the start or at the end of the sequence. Can't remember the paper that showed this

@Karl I believe the paper was called "Lost in the middle"

Yes, you have the right idea. There's been a lot of work recently regarding extending the context-length of existing models, mostly looking at the Llama family of models.
You should check out this post/paper on LongChat by the team behind the Vicuna models (LMSYS). They extend the pretrained Llama models (with 2048-token contexts) to be able to handle 16k-token contexts.
You may seem some parallels between the techniques they discuss and what you mentioned in your post.
(1) They use Rotary Embeddings, which have an infinite number of positions. However, instead of extending past the original length, they instead interpolate between the existing values.
(2) They also finetune the model on long-context sequences (long conversational data).
The resulting performance on different context lengths:

This plot from the LMSYS org shows the LongChat model (described above) versus accuracy on a retrieval task, compared to some other models.
This technique is elaborated on in this paper: "Extending Context Window of Large Language Models via Position Interpolation"

Position Interpolation can easily enable very long context windows (e.g. 32768), requiring only fine-tuning for 1000 steps on the Pile (Gao et al., 2020) to achieve a good quality. The cost of fine-tuning is negligible compared to the pre-training costs. This confirms our hypothesis that it is relatively easy for the models to adapt to interpolated position encodings.

This work suggests that your second suggestion may even be overkill -- you don't need to do too much additional tuning to allow models to use the larger context. They also discuss why one should interpolate between embeddings rather than extrapolate beyond the original context length: namely, that it can result in catastrophic explosions in attention scores when the position differences are too large.

So the reason for the interpolation is that the pre-training on the original position encoding still kind of makes sense and we don't have to pre-train from scratch, right?

Yeah kinda, you don't have to pre-train from scratch, and interpolation (rather than extrapolation) is used because the model is "used to" the range of positional embeddings.


text:
Any suggestions for transformer finetuning techniques ablation study?
I'm planning to fine tune a 7b parameter model for a research project. I understand the different steps of model fine tuning, namely

Supervised fine tuning - where we train model on curated examples to impart domain specific knowledge (could be instruction fine tuning as well)
DPO/PPO - this step is done for model alignment to human preferences

I understand there is a huggingface library which has built wrappers around these both but I'm trying to get an in depth understanding. So how I see a language model with depth N is that the first N-1 layers are building a representation and last layer is a policy network, i.e. takes the representation and spits out a probability distribution over the tokens. Now my main question, in SFT and DPO/PPO is there any literature which describes the following and provides benchmarks/ablation studies for the following -

Doing SFT (no frozen parameters) and only using adapters for DPO/PPO
Doing SFT by freezing all layers except the last and followed by DPO/PPO on last layer
Doing SFT using adapters/LoRA, merging the weights, followed by adapters for DPO/PPO
Doing SFT using adapters/LoRA, merging the weights, followed by DPO/PPO on only last layer
Vanilla option, i.e. SFT followed by DPO/PPO no frozen weights.

Any reference for literature or anecdotal evidence?

Can you please put your specific question in the title? So a question


text:
Binary instance segmentation - does the masks have to be complete
I am wondering if it is required or not that the masks used for binary instance segmentation are complete.
For instance, I want to find the buildings in aerial imagery. If my masks cover, let say between 25% and 75% of the training image buildings, can these samples be used for training the model? Does the masks should cover 100% of the buildings in the samples?
I know it would be better to mask all buildings, but if the mask is not complete, can it be used for training or not?
I am using a deeplabv3 model.
Here an example of what I mean

You can train a model with that dataset, but it will perform worse compared to a higher fidelity dataset. Training on images with unannotated buildings is essentially instructing the model to stochastically ignore buildings, which is confusing.


text:
How to teach Gemma model my mother tongue (Kannada - one of the oldest Indic languages)
I'm interested in teaching the Gemma 2B model my mother tongue (Kannada - one of the oldest Indic languages). The pre-trained model doesn't work well with the mentioned language, so I thought of teaching the model but I'm very new to this field of training and fine-tuning LLMs. But based on some research I've planned the below steps:

Prepare the Dataset - luckily I found some dataset on HF.

Train the Tokenizer for the language and enrich the vocabulary of the Gemma model with the newly obtained tokens from the dataset using a newly trained Tokenizer.

Here is where I'm confused
a. Should I simply fine-tune the model? - maybe using LoRA or QLoRA kind of techniques.
b. I read about a concept called Continually Pre Train, should I give this a try?

3 (b) looks promising to me but I'm not able to find much explanations and code examples on the internet about how to achieve this.
Note: I'm looking for a very basic model and performing this for learning purposes and I should be able to do this on my Google Colab Notebook.
I'm seeking some guidance here on how to proceed and some code examples for achieving this.


text:
Please, could someone help me understand if the backpropagation explanations in these two articles about calculate the error are equivalent?
I have a question about backpropagation, I'm a beginner, I'm studying the formulas to calculate the delta of neurons, there are several sources on the internet, which teach in different ways, so I'm confused about the formulas presented in the explanations, as they are a little different. Please, could someone help me understand if the explanations in these two articles are equivalent?:
This first article: https://home.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html, it says that the error of a neuron in the current hidden layer is the sum of all (error* connection weight) of the neurons in the next layer. The error calculation in this article does not use the derivative of the neuron activation when calculating the error. Instead, it will only use the derivative when updating the weights, using the formula "wNew = wOld + (learningRate * error * derivative * input)", this is where it will use the derivative.
However, in this second article: https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/, which teaches how to program a multi-layer perceptron from scratch in Python, he does it a little differently, In the formula for updating the weights he uses a variable called "delta", he explains that the "delta" of a neuron is the (neuron's error * derivative from the neuron's activation), and to calculate this delta we need the neuron's error. In the code presented in this article, the error of a neuron in the hidden layer is also a sum, but it is the sum of (neuron delta * connection weight) of the neurons in the next layer, therefore it first calculates the error of all neurons, then it calculates the "deltas" of all neurons (so, to calculate the deltas of neurons in one layer it uses the deltas of the next layer), and then updates the weights. And since in this article he already calculated the delta as being the (error * derivative), when updating the weights, the formula for updating the weights is simply "wNew = wOld + (learningRate * delta * input)"
So I'm confused, specifically about SUM. I would really like to understand why in the first article: https://home.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html, he calculates the neuron error without using the derivative, and only uses the derivative just when updating the weights. And on the other hand, in this other article: https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/, it calculates the errors and then calculates the deltas using the derivative to calculate these deltas, to then update the weights without needing to use a derivative in the formula to update the weights
Why are there these differences in the formulas of the two articles?
This is a question I've had for a long time, I would really like to understand.
Are the explanations in these two articles equivalent? Please help me clarify this doubt?

The difference you spotted is only cosmetic when dealing with formulas yet both are equivalent. In general for any delta rule ultimately driven by some form of error be it primitive correction or Widrow-Hoff/LMS like SGD correction follows the three multiplicative terms (learning rate, error, input) format of your 2nd reference, thus the backpropagated error for each (hidden) neuron should've already absorbed the derivative of the neuron activation complying with this convention for the middle error term of the said neuron. This is reflected in the code below of the backward_propagate_error(network, expected) function in your 2nd reference. Of course there's nothing wrong to invoke the derivative of activation in the last update step as reflected in your 1st reference.
            neuron = layer[j]
            neuron['delta'] = errors[j] * transfer_derivative(neuron['output'])


text:
Can Q-learning be used to create new creative solutions by combining different factors and characteristics?
References from Wikipedia:

https://en.wikipedia.org/wiki/Q-learning
https://en.wikipedia.org/wiki/Markov_decision_process

Q-learning can be used to create new creative solutions, combining different behaviors, characteristics, reactions, facts, qualities, advantages, of materials, substances, etc., and using these as actions in the action space, and thus being able to find a solution to meet a given need or specific characteristic, such as finding possibilities for new recipes, combining ingredients so the cake has flavors that the user likes most, or based on physical or chemical reactions and interactions, creating materials that have certain characteristics, such as better resistance the given situation?
In summary, can Q-learning be used for these types of more complex problems, which involve finding new creative solutions, involving combinations of various actions, and thus being able to create new products, materials, etc.? Is this something possible for Q-learning?

Your imagination about Q-learning even including possibly all RL methods to learn really creative solutions in the sense of AGI seems too far-fetched, as intuitively this needs additional higher level abstract reasoning and inferencing capabilities and production expert systems at least which Q-learning cannot model and it cannot even deal with non-Markov processes or time series in theory.
Additionally, even you can use neural nets to approximately deal with large (infinite) state and action spaces, reward can be very hard and tricky for hard problem requiring creative solutions where the reward may be subjective and different for each agent, such as the case to find a creative solution to unify quantum mechanics and general relativity in theoretical physics where some scientists favor explanatory power of real hidden variables, some favor the reward of instrumental predictive power and falsifiability, while others many favor completely different reward of the unified grand theory.
Finally Q-learning implicitly assumes a stationary environment where the transition dynamics and reward distributions don't change over time, however, in creative tasks the definition of a good solution might evolve such as some social application thus adapting to dynamic environments becomes essential while current standard Q-learning cannot handle.

I have one more question about this: What if I use a very simple creative problem, where the value of the rewards would not change, and instead of using standard Q-learning, I assume that the states are infinite, with a quantity of actions that I knew could be combined to reach the final objective.  And, if in this scenario, I know how to correctly define the appropriate rewards and punishments for the problem in question.  Could this algorithm be able to find a combination of actions that reach the final goal?  And in your opinion, what could be the biggest challenge in tasks of this type?

It'll become much more suitable for RL then, especially with policy gradient methods instead of Q-learning as you cannot store an infinite Q-table for your potentially infinite states. If the existent creative policy is extremely nonlinear then you'd need deep RL to estimate value functions in its advantage to update the policy. And in difficult creative situations like Chess or Go, additionally you need some heuristic search like MCTS to speed up the learning process. These are the main tricks currently known.

Please accept any answer you feel satisfied for many of your recent batch of questions if possible, thanks!

Of course, I'll do it!


text:
In MLP, Why in the formula to calculate delta, the derivation of the cost function "(predicted - expected)" just as "(predicted - expected)" itself?
I know that we have to perform the derivation of the terms, using the chain rule, to obtain the equations to calculate the gradients of the neurons.
In this article: https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/,
The formula for calculating the gradient in the output layer was simply like this: "(predicted - expected) * output_derivative".  That is, it multiplies the error by the derivative of the output layer activation.  So, studying this, I think that in all cases to calculate the gradient I would just need to multiply the difference between the predicted and expected value (the error, i.e. the cost function) by the derivative of the layer's activation function, as described in this formula.
But it seems that it won't always be this, as I saw that in other articles and videos on YouTube, when they explain the derivation of the chain rule, the formula for calculating the gradients is different.  For example, in a video, he taught that we need to calculate the derivative of the cost function, for example, he used the MSE cost function (which is "(predicted - expected) ^ 2" ), and the derivative of this function was "-2 * (predicted - expected)", which was somewhat different from what I expected.  I thought it would just be "(predicted - expected) ^ 2 * output_derivative" similar to another formula, but it was different, it ended up deriving the cost function, which was "2 * (predicted - expected)", therefore the complete formula to calculate the gradient was "2 * (predicted - expected) * output_derivative", that is, it calculated the derivative of the cost function. That's why I was confused.
I had a question and wanted to ask:
Why was the derivation of the cost function "(predicted - expected)" in the article I mentioned just as "(predicted - expected)", that is, itself?  Why was the other explanation different?  Could someone please help me understand?


text:
In MLP, to calculate the delta, do I need to calculate the derivative of the cost function? Or can I just use the cost function result?
In Multi Layer Perceptron networks, I have a question: in the formula to calculate the error in the output layer, Some articles say it is like this "deltaOutput = (predict - expected) * derivativeOutput".  More in other sources, I saw that they seemed to calculate the derivative of the cost function, they calculated derivative it in Wolfram, and the formula to calculate the error was become "derivativeError * derivativeOutput". But then I got confused, and had this question: do we need to calculate the derivative of the error(the cost function) to use in the formula to calculate the delta?  Or are there cases where I can simply use "deltaOutput = (predict - expected) * derivativeOutput" in the formula without having to calculate the derivative of the error?  Please, could you help me understand?
In MLP, In formula to calculating the delta, do I need to calculate the derivative of the cost function(As in the case of the function MSE) ? Or can I just use the cost function result, And the formula for calculating the error in the output layer would just be "(predict - expected) * derivativeOutput"?


text:
If I manually trained a multilayer percetron neural network, always following exactly the same steps meticulously, would I always get the same result?
Reference:
https://home.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html
If I trained a multilayer percetron neural network manually, following exactly the backpropagation steps described in the article, with fixed initial weights (that is, without starting randomly), and then repeating this training again several more times just like I did the first time, ignoring the randomness and unforeseen events of the computation(because as I would be following everything manually, there would be no random initialization of the weights, no rounding problems, nor any variation in the process), if I do this when training the network, and I always use the same fixed initial weights( so that it always starts from the same starting point), using the same network structure (same number of layers, neurons, exactly the same network), with exactly the same parameters (such as number of epochs, learning rate, etc.), with the same dataset, with this network being trained with the dataset samples in the same order, following meticulously, very carefully, mathematically will I always get the same results? this is true ?
I are without considering of whether the predictions are correct or not. My specific doubt, whether mathematically the results will be the same in the circumstances described.

Unless you use full batch gradient descent method for your BP algo which is very rare in practice mainly due to its inefficiency and slow convergence, the usual stochastic or mini-batch or momentum-based gradient descent methods don't produce exactly same results.

As the algorithm sweeps through the training set, it performs the above update for each training sample. Several passes can be made over the training set until the algorithm converges. If this is done, the data can be shuffled for each pass to prevent cycles. Typical implementations may use an adaptive learning rate so that the algorithm converges.

Obviously in above case the order of all your training samples to be fitted in any epoch matters due to the data shuffle along with an adaptive learning rate and the nonzero threshold even in all the cases of actual convergence.


text:
Why in one article to calcule the delta uses the weights of current layer, while in another article uses the weights of neuron inputs of next layer?
Article 1:
https://pyimagesearch.com/2021/05/06/backpropagation-from-scratch-with-python/
Article 2:
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/
I was reading the 2 articles that teach backpropagation from scratch.
I had a doubt about the first article, and so I would really like to ask specifically about the way he used to calculate the deltas of neurons in the hidden layer.
The first article, talking about calculating the delta in hidden layers, mentions the following:

the delta of the current layer is equal to the delta from the previous
layer dotted with the weight matrix of the current layer, followed by
multiplication of the delta by the derivative of the nonlinear
activation function for the activations of the current layer

However, in the second article, the explanation seems to be a little different, in the second article's explanation on how to calculate the delta he mentions

You can see that the error signal for neurons in the hidden layer is
accumulated from neurons in the output layer where the hidden neuron
number j is also the index of the neuron’s weight in the output layer
neuron[‘weights’][j].

Analyzing this explanation from the second article, I had this doubt: the first article on the hidden layer says that the delta of the current layer is equal to the delta from the previous layer dotted with the weight matrix of the current layer. However, in the second article he says that the delta in the hidden layer is calculated using the sum of the deltas of the next layer multiplied by the connection weights with the neuron with index J in the current hidden layer (in this case, the next layer is the output layer), this sum is multiplied by the derivative of the activation of the hidden layer neuron. I was very confused precisely because in the second article the weights that were used to calculate the delta are not the weights of the current layer (as is done in the first article), instead, the weights used in the second article are the weights of the next layer (That's it, the weights of the inputs of the neurons in the next layer, in this case the output layer)
I know that both articles are similar explanations, but I was confused. Is there any difference in the formulas of both articles? Or in practice are both equivalent? Could someone please help me understand these differences?

The key idea of backpropagation can be summarized as:

Each individual component of the gradient, $\partial C/\partial w_{jk}^{l}$ can be computed by the chain rule; but doing this separately for each weight is inefficient. Backpropagation efficiently computes the gradient by avoiding duplicate calculations and not computing unnecessary intermediate values, by computing the gradient of each layer – specifically the gradient of the weighted input of each layer, denoted by $\delta^{l}$ – from back to front... The $\delta ^{l}$ can easily be computed recursively, going from right to left, as: $$\delta ^{l-1}:=(f^{l-1})'\circ (W^{l})^{T}\cdot \delta ^{l}$$
The gradients of the weights can thus be computed using a few matrix multiplications for each level; this is backpropagation.

Once this crucial step is done, all the weights can be straightforwardly updated via the delta rule involving a learning rate. Therefore you 1st article's quoted section/code just essentially implements the above crucial gradients $\delta ^{l-1}$ of the weighted inputs of each layer from code line 99-111, and code line 113-125 implements the final weights delta update rule which additionally involves the multiplicative terms of learning rate and the respective weighted inputs of each layer in Numpy's matrix/array linear algebra way.
And the error signal of your 2nd article's quoted section ("the error signal for neurons in the hidden layer is accumulated from neurons in the output layer") is just the inner function part $(W^{l})^{T}\cdot \delta ^{l}$ of the composed gradient function $\delta ^{l-1}$. And at the end line 18 of the quoted code snippet you have:
neuron['delta'] = errors[j] * transfer_derivative(neuron['output'])
Thus the same gradient of each layer $\delta ^{l-1}$ as expressed as neuron['delta'] here is obtained too after multiplied by the derivative of the activation/transfer function though here seems only primitive Python data types are employed instead of Numpy's linear algebra data types.

Thanks for explaining!  With your explanation, both articles seem similar.  So this means that to calculate the delta of a neuron in the hidden layers, in both ways, both the first article (which uses the current layer) and the second article (which uses the next layer), work equivalently and are based on the same principles for the same goal?, even using a slightly different way? Did I understand correctly?

Yes, your conclusion is correct, there’s only one standard back propagation algorithm though there’re many different implementations such as manifested here and in another similar question you asked recently. One idea with many different implementations is a common phenomenon in applied sciences and engineering.

Thanks!. But, in second article, the sum of DELTA OUTPUT * WEIGHT to calcule delta in hidden layer, he explains that WEIGHT is the weight J (the index of the neuron J of current hidden layer in vector of the output layer, referring to input J), that is, the connection weight of output layer, that is multiplied by input of output layer (the input of the output layer is the output of the last hidden layer). But, in first article is different, because the weights are the weights that connect the input layer input with the current hidden layer. I had this question below:

I have this another question about the first article: the first article multiplies the delta of the output layer with the weight matrix of the self hidden layer(that is, the weight matrix of the current hidden layer L itself). In contrast, the second article, when calculating the delta in the hidden layer, using a sum, it multiplies the delta of the output layer itself with the weights of the neurons in the output layer itself(Formula sum of DELTA OUPUT * WEIGHT OUPUT). It gets little confusing, and seems to be different calculation. Please, could help me clarify this point?

The 1st article is using matrix algebra just as my above answer for the hidden layer's all units' gradients $\delta^{l-1}$ of their respective weighted input, so if 3 units in the current hidden layer $\delta^{l-1}$ is a vector of 3 gradients. This ref is clearer, the first hidden layer has 3 units (1,2,3), each has gradient of its own weighted input, and it's not in matrix form, so like your 2nd article, you can compare and check its for-loops carefully though this is mainly a coding practice, I'd prefer 1st article's matrix code.

Thank for the explanations!. It has helped me a lot to understand the concepts.


text:
In multilayer perceptron neural networks, the more complex a dataset is, the more difficult it will be to find adequate initial random weights?
I know that neural networks don't need to have exactly the same weights, and that different networks trained on the same problem may result in different final weights, but produce comparable results. And i know that the initial weights initialized randomly are used to help the optimization procedures.
In multilayer perceptron neural networks, during training, using a dataset with a simpler pattern such as Iris, the network is more likely to find adequate initial weights using random weight initialization(perhaps because the pattern is simpler, and could there are more possibilities of initial weights that would be suitable?
And for example, if I use a much more complex dataset, which has a much more challenging pattern for the model. During random initial weight initialization, would it be much more difficult to find suitable initial weights so that the network can perform well?
In other words, the more complex a dataset is, the more difficult it will be to find adequate initial random weights?
Detail: the neural network I'm using, to start the random weights, generates pseudo-random numbers (between 0 and 1), using the same seed(42) to guarantee reproducibility.

but produce the same result. this is a fake news going around the ML community, they will have comparable performances, but the predictions won't be the same.
And no, initialization does not aim to get good initial performances, but to help the optimization procedures (so avoiding dead gradients, smooth locally the loss landscape and so on)
So, to answer to your question: _ the more complex a dataset is, the more difficult it will be to find adequate initial random weights_ no, it's pretty much impossible to find a random initial guess for neural network that works even for the simplest of the task, mainly due to stacked non linearities
The only initialization that is based on good guesses is called pretraining, but it's far from being random

Thanks for clarifying some points!  But I have a question: I know that the initial weights initialized randomly are to help the optimization procedures (so avoiding dead gradients, smooth locally the loss landscape and so on) as you said.  However, if this initialization is not good, would this hinder the training process?

@willTheJ yes, suppose to have normalized your feature in 0-1 (so they are all positive), and you initialize your weights with negative numbers, then $relu(Wx+b)$ will always be 0, thus no learning will happen at all. So yes, initialization is "an art", because it can most likely hinder your training (mainly the initial part), however now we have good guesses for most of the known activation functions

Thanks for ReLu's example!. For example, the ReLu would produce output 0 if the value is negative, but what if for example it was the Sigmoid, would no learning occur either?

@willTheJ you can still have problems, for example initializing with big values your parameters $W,b$, doing $\sigma(Wx+b)$ will lead you to the tails of the sigmoid function, which has pretty much 0 gradient, so the learning will be very very slow


text:
In MLP, how would I update the weights using batches? Would I have to calculate the accumulated error(of all samples) of each output neuron?
In Multilayer Perceptron neural networks, I know that there are two types of training: online training, and batch training, which consists of dividing the samples and updating the weights using the accumulated error.  In other neural networks, for example in Adaline, I know that something similar is done to update the weights, for example in this video: https://youtu.be/MTe2qsS56MQ?si=Sky-220_zA15l7T8
In this case, it shows the " total error", from what it seems to me, the weights are only adjusted at the end of each epoch, using the accumulated error (summed, that is, the total error) of the samples.  In the example in the video, he used only one output neuron, as it was just one Adaline, and in the way it was done, it seems like it was just a single batch, as he made a sum of the squared errors of each sample and only at the end I updated the weights using this accumulated error as I mentioned above, it was just a batch.  But for me it's confusing, because, if I have a Multilayer Perceptron using several output classes, how would I update the weights using batches? Would I have to calculate the accumulated error(of all samples) of each output neuron?
For example, if I divide 10 samples into 2 batches, how would the weights be updated?  Would it add up the error of all samples from each batch, and then update the weights (after the current batch)?  When using multiple classes, what would this weight update look like?  I would have to add the error (that is, obtain the accumulated error) of all output neurons, for example: "total error of class 1", "total error of class 2", "total error of class 3", etc?
In Multilayer Perceptron,using several output classes, how would I update the weights using batches? Would I have to calculate the accumulated error(of all samples) of each output neuron?

So, two things:

Yes you technically calculate the accumulated error for each neuron, but in practice we calculate all neurons at once by treating it as a matrix instead of individual vectors. Likewise, you do not have to calculate error gradients for the individual hidden neurons, but rather you calculate the gradients on a layer-wise basis which can be decomposed into individual neurons.

The size of your minibatches and the number of samples per update step are not necessarily the same thing. While you can perform backprop and weight updates after each minibatch, you can also perform what's called gradient accumulation where, as the name says, you accumulate the gradients between several (or all) minibatches before performing your update step. Accumulating the gradients for all minibatches before performing your update step effectively makes it so you're doing batch gradient descent, even if you don't have the hardware to do "true" batch gradient descent on data of your size.

How you select your minibatch size and number of gradient accumulation steps (minibatches per update step) is an empirical process, for now.

But then, if I'm not using matrices, does this mean that after processing each sample (in feedforward), I calculate the error of that sample for each of the output neurons individually?  And so at the end of the minibatch, I would have the following errors accumulated from each output neuron individually, for example if I have 2 output neurons:
errorN1 = sum(...samplesErrorsN1), errorN2 = sum(...samplesErrorsN2), etc...
In minibatch, For each output neuron I calculate the sum of the errors of all samples, so I have the accumulated error of each output neuron individually? Am I correct?


text:
what is the backpropagation formula to calculate delta and update weights?
I'm trying to study how backpropagation works step by step in a MultiLayer Perceptron neural network.  I would really like to be able to understand how these calculations work.  And I have a specific question I would like to ask.  The formulas I'm trying to learn are the following to calculate the delta, and then to update the weights and bias:
Please, could you help me understand if is correct ?
In Output Layer
Calculate delta: error * derivative

Update weights: newW = oldW + (delta * neuron entry * learning_rate)

Update Bias: newB = oldB + (delta * learning_rate)

(Here in the output layer, the neuron's error is the difference between the desired output and the predicted output of the neuron, in the feedforward phase)
In the Hidden Layer:
Calculate delta: SUM of (delta * connection weight ) * derivative

 Update weights: newW = oldW + (delta * neuron entry * learning_rate)

 Update Bias: newB = oldB + (delta * learning_rate)

(In the SUM, Here, in the hidden layer, the error of a neuron is a summation, adding the delta multiplied by the connection weight of the neuron in this next layer with the neuron in the current layer, of each neuron in the next layer, that is, the connection weights that connect a neuron in the current hidden layer with a neuron in the next layer.)
I would like to ask you a specific question about these two formulas used in backpropagation, because I have this doubt: the formulas I am studying to calculate the delta and then to update the weights in the output layer and hidden layer are correct, Are these formulas I mentioned correct?
Please, could someone help me, and tell me if what I am studying is correct?
What are the backpropagation correct formulas?

I'm sorry, in the end I cannot make out what you are asking.  I do not know what "It is like this the same?" means.

Sorry in the end. Thank you for wanting to help. With that, what I would like to ask is if the formulas are correct?

Your understanding appears correct. One formula I want to nitpick about though, is that of the delta calculation for the hidden layer. I would suggest framing it as :
SUM of (delta * connection weight ) * derivative

rather than :
(SUM of delta * connection weight ) * derivative

I will edit the question to improve the question according to your suggestion. Thanks!


text:
In Q-learning, Am I the one who will define the way in which actions allow the agent to interact with the environment? And the interactions will vary?
In Q-learning, am I the one who will define the way in which actions allow the agent to interact with the environment, so that the way in which actions allow the agent to interact with the environment can vary greatly from the problem in question?
For example, in this article: https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/, which explains Q-learnig, it teaches the Smartcab problem, it only has 6 actions (walk up, down, right and left, pickup and dropoff). The action of moving upwards makes the agent add +1 to Y, advancing its state. In this Smartcab example, the states are positions X and Y representing where the agent is.
But the way in which actions allow the agent to interact with the scenario can be something much broader depending on the problem?, instead of being movement actions (such as walking up, down, right and left), instead of Furthermore, are they more complex actions, which could make the agent change state in a very different way than this Smartcab example?
In Q-learning the way in which the actions will make the agent interact with the environment will depend greatly on the problem in question, so that each problem can have its own rules for the agent to interact with the environment, rules that I myself Can I set it according to my needs?

The action space for a reinforcement learning (RL) task is mostly determined by the problem you are applying RL to. As such, you get to define it in the following ways:

By choosing what problem to work on
By writing a formal description or code for the environment
By encoding a suitable representation of the action space for the agent code to interact with

The second and third items here are constrained by the ones above them. Other than that, RL as a whole is a high level and generalised description of learning by trial and error that applies to a very broad number of cases. Q-learning specifically may not be applicable to all of them, and is one of many solution methods.
Understanding which RL methods are suitable for a specific environment takes a bit of study. One of the factors that affects which methods work best is the nature of the action space. However, if you are just concerned about defining a similar number of discrete actions to TaxiCab, that happen to do something entirely different in a new problem of a similar size, then Q-learning should still apply, and you can absolutely decide on a different set of actions with entirely different consequences to the environment.

Agent has the free will to choose one action out of its action space (many environments may pose continuously valued infinite action space and state space unlike your referenced simple grid environment) at a timestep according to the agent's current state following some probability distribution during training or even possibly as a final converged optimal stochastic policy $\pi^*$ depending on your problem setup, so clearly each problem has its own rule for defining action space for the agent to interact with the environment, but the mechanism of the action space defining rule itself is not determined by the agent, but totally determined by the environment.