text:
does it make sense to consider the base/backbone network one of the multiscale feature map blocks in SSD?
I'm trying to understand Single Shot Multibox Detection following a book adopted at 500 universities from 70 countries

The complete single shot multibox detection model consists of five blocks. The feature maps produced by each block are used for both (i) generating anchor boxes and (ii) predicting classes and offsets of these anchor boxes. Among these five blocks, the first one is the base network block, the second to the fourth are downsampling blocks, and the last block uses global max-pooling to reduce both the height and width to 1. Technically, the second to the fifth blocks are all those multiscale feature map blocks in Fig. 14.7.1.

the following is Fig. 14.7.1. mentioned in the above excerpt

I'm aware that the second to the fifth blocks are all those multiscale feature map blocks. My only concern is whether the base network is also a feature map block. I guess it is though the book doesn't convey the idea explicitly. So I guess I need a double confirm.

Hi @singularli, and welcome to AI Stack Exchange! If possible, please outline your question a little more explicitly. That might help in getting more answers.

Yes, you are correct The base network block is also a feature map block.
The base network block extracts features from the input image. It is using CNN modules with filter num of [3, 16, 32, 64].  It Outputs the feature maps that are used for generation of anchor boxes and predicting clases as well as the offset of those anchor boxes.
def base_net():
    blk = []
    num_filters = [3, 16, 32, 64]
    for i in range(len(num_filters) - 1):
        blk.append(down_sample_blk(num_filters[i], num_filters[i+1]))
    return nn.Sequential(*blk)

forward(torch.zeros((2, 3, 256, 256)), base_net()).shape

reference

thank you. so, the original phrasing seems to be somewhat misleading. A clearer way might be 'All 5 blocks are multiscale feature map blocks.', right?

Yes exactly all blocks are feature maps...


text:
How does chatgpt interpret prompts?
First of all, my English isn't very good, please bear with me.
I've heard that chatgpt is a decoder-only model, which doesn't use any encoder model.
How does it interpret our prompts if it doesn't use any encoder model?

ChatGPT is the finetuning of GPT_3.5, which is a decoder-only model. It does not interpret the prompt the same way the encode-decoder model does. It is using autoregressive pre-training, which generates text based on the input.
You Provide a prompt to ChatGPT. It can be any text like a question, statement, or text.
Your Prompt is broken into a series of tokens. every token is a word or part of a word.
Each token is transformed into the contextual embedding which is a vector representation of the token. This holds the contextual meaning of the token with respect to surrounding tokens.
The main Decoding process then processes the decoder part of the model. The decode only generates one token at a time and uses previously generated tokens as well as the prompt as context. This process is done in an autoregressive manner.
The final output is a sequence of tokens that need to be converted back to the text.
This is the complete decoding process. Hope you understood.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
DQN arXiv 10-year anniversary: What are the outstanding problems being actively researched in deep Q-learning since 2019?
Background
As of today (12-19-2023), the arXiv submission of the original deep Q-learning approach to achieve superhuman performance on ATARI games has turned a decade old. The original approach, sometimes referred to as vanilla DQN (2013), was the focus of numerous research investigations and improvements throughout the ensuing 5 years (2014-2018), as evidenced by the following problems and examples:

Estimation: Double DQN (DDQN, 2015) decouples action selection and estimation to prevent overestimation in the Q-values. Dueling DQN (2015) decouples the deep Q-network into separate state-value and action-value functions to achieve better policy evaluation in the presence of many similar-valued actions. Distributional DQN (2017) attempts to learn a value distribution to better model multimodal and nonstationary policies.

Efficiency: Prioritized experience replay (PER, 2015) samples important transitions more frequently to increase learning speed. Learning from multi-step bootstrap targets such as in A3C (2016) was performed in RAINBOW DQN (2017) to help propagate newly observed rewards faster to earlier visited states, again for increased learning speed.

Generalization: Certain environmental additions such as sticky actions (2018) prevent the agent from memorizing trajectories and require some basic level of generalization. Techniques from standard deep learning such as batch normalization and regularization (2018) have shown to improve the agent's ability to generalize.

Parallelism: General Reinforcement Learning Architecture DQN (Gorila, 2015) presented the first massively distributed architecture for DQN, which reduced wall-clock training time. Ape-X DQN (2018) is a distributed algorithm that incorporates other DQN improvements such as prioritized experience replay to again lessen training time.

Memory: Incorporating recurrent architectures such as LSTMs in deep recurrent Q-networks (DRQN, 2015) improves the agent's ability to learn and test on partially-observable environments. Recurrent replay distributed DQN (R2D2, 2018) incorporates an LSTM in a distributed architecture and presents training strategies to mitigate representational drift and recurrent state staleness caused by RNN architectures.

Exploration: Noisy networks (2017) inject stochasticity into the agent's policy to more efficiently explore. Random network distillation (RND, 2018) provides an exploration bonus based on the novelty of a state through its prediction error on a randomly-distilled predictor network that is continually trained on incoming states using a fixed target network for its target values.

Question: What are the outstanding problems being actively researched in deep Q-learning throughout the past 5 years (2019-2023)?
To keep this question within the on-topic guidelines and reduce possible subjectivity, please cite relevant sources and briefly summarize the sources (e.g. with a cause-and-effect structure as in the above paragraphs). Also, only discuss sources that involve Q-learning.

the continuous action space, C51 (distributional DQN) for example

Regarding estimation and sample efficiency:

REDQ (2021): using a large ensemble of Q-functions can reduce the overestimation bias (likewise Double DQN) so much that it enables to update the Q-networks more often than the environment interactions (i.e., high update-to-data ratio), therefore achieving high sample-efficiency.
DroQ (2022): improves over REDQ by also increasing the computational efficiency, thanks to the use of a small ensemble. This is possible by "dropout Q-networks", which use Dropout as an efficient way to introduce uncertainty in the Q-networks.

The concept of ensemble Q-functions is also useful in offline RL, where the goal is to learn a policy from a fixed dataset of experience:

This paper (2021) says that the uncertainty of the ensemble can be used to penalize the policy towards predicting actions that are out-of-distribution (OOD), due to an erroneously large Q-value.

The original deep Q-learning approach has been extended in many fundamental works in offline reinforcement learning (ORL), where a fixed dataset is used to train an agent without any further environment interaction.

Batch Constrained Q-learning (BCQ, 2019) aims to keep the state-action visitation of the learned policy similar to that of the dataset, thereby reducing overestimation error caused by distributional drift between the dataset and the learned policy. For a given state, BCQ generates candidate actions to yield state-action pairs that have high similarity to those in the dataset. The most promising action is chosen through a learned Q-network. Furthermore, BCQ penalizes rare or unseen states through a modification to Clipped Double Q-learning. Implementing BCQ requires the training of 4 neural networks, and as a result, it seems to be used less frequently than other ORL algorithms.

Conservative Q-learning (CQL, 2020) regularizes the Q-values during training to learn a conservative Q-function and prevent the overestimation problem. The expected value of a policy under the learned Q-function lower-bounds the true value of the policy. Only lower-bounding in expectation prevents extra underestimation from approaches that learn a point-wise lower-bounded Q-function. The implementation of CQL is remarkably simple and makes it a popular choice as an ORL baseline algorithm.

Implicit Q-learning (IQL, 2021) learns a Q-function and completely avoids querying unseen actions. This is accomplished by treating the state-value function as a random variable, with randomness determined by the action. The values of actions are estimated by taking a state conditional upper expectile of this random variable. The resultant Q-function is extracted with advantage-weighted behavioral cloning, again to avoid querying unseen actions. Implementing IQL is simple, and its performance on standard ORL benchmark datasets has also made it a popular ORL baseline algorithm.

Calibrated Q-learning (Cal-QL, 2023) learns a Q-function initialization that, in expectation, lower-bounds the true value function of the learned policy and upper bounds the true value function of another reference policy (i.e. the behavior policy induced by the dataset). The motivation for Cal-QL is to prevent the performance decline of conservative ORL methods immediately after online finetuning begins, such as CQL. Upper-bounding a reference policy calibrates the Q-function to learn values on a realistic scale and has been shown empirically to prevent initial performance decline.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
In neural style transfer, why do we focus on a single style image?
In all the descriptions of neural style transfer that I've seen so far, there is a single style image and a single content image, and the task is to produce a new image with the style of one image and the content of the other. For example, we can make the Mona Lisa in the style of van Gogh's Starry Night.
However, this approach seems limiting to me because it is reproducing the style of a single image and not the overall style of the artist. If van Gogh painted a portrait like the Mona Lisa, chances are he would not have used the same colours and motifs as Starry Night. For example, in the Starry Night Mona Lisa linked above, the skin is bluish with swirls and there are a few star-like artefacts. These features are not present in the original Mona Lisa, and they don't appear in van Gogh's portraits like this one either.
Has much work been done in the direction of reproducing an artist's overall style, rather than the style of a single image?
One solution I am imagining is you could have a database of images in the same style, and to stylise a particular content image you could find the style image which is "most similar" to the content image (for some appropriate definition of "similar") and use that as input to the style transfer algorithm. But I haven't tried this and maybe there are better solutions.

So far it is correct. The style transfer technique from the paper "A Neural Algorithm of Artistic Style" only uses a single image. To be exact, the final loss function that the model uses is the following:
$$ L_{total}(\vec{p}, \vec{a}, \vec{x}) = \alpha L_{content}(\vec{p}, \vec{x}) + \beta L_{style}(\vec{a}, \vec{x}) $$
where $\vec{p}$ is the original image, $\vec{a}$ is the image we want to copy the style from, $\vec{x}$ is the initial random image we are going to iterate on to get our final result. Then $L_{content}$ is a function that calculates the error between the original image and the generated image. Similarly, $L_{style}$ is the loss function that calculates the error between the style image and the generated image.
So this specific method takes two images as input. It can generate another image that copies the style from one and the structure from the other to create a final image with both of those properties.
Why do we focus on a single style image?
The focus is on a single image because there was never a need to focus on multiple images. The goal of the algorithm is not to replicate the style of an artist, but the style of a single image.
Other solutions:
If you want to replicate the styles of multiple images, other generative models could work, like pix2pix, VAEs, GANs or diffusion models. These use latent spaces for saving the information from the trained data, to then generate completely new images. The idea would be to use Stable Diffusion-like models.
Your solution could also work - let's use the method from this paper and try a visual encoder to embed each style we want to copy into an embedded space. Then compute similarity with a similarity function to determine which style image best matches the requested original image. In that way, you could be more exact on the type of style that will best match your image.
Tools for creating the systems mentioned
For using generative models, the best current approach is the Stable Diffusion + DreamBooth pipeline, if you want to fine-tune on a specific set of styles given a special token.
In the case of the database solution, using a vector database like Pinecone, that supports image encoding, or Weaviate could work.
It's important to notice that there could be some ambiguity on the meaning of an artist's style. The best we can do is select multiple images, extract patterns from those images, and transfer them into another. If that is what you are referring to, these methods could work, but sadly human cloning has not been invented yet ;)   so more than this is going to be quite difficult!


text:
Selection of actions based on distribution of rewards
In "A Distributional Perspective on Reinforcement Learning" Bellamare et. al. 2017.  We find the following phrase.

As in DQN, we use a simple $\epsilon$-greedy policy over the expected action values; we leave as future work the many ways in which an agent could select actions on the basis of the full distribution.

Did I miss the strategy used in the paper?  I understand $\epsilon$-greedy but how were distributions compared to determine which action is the greediest?  Q is the reward with the maximum likelihood, etc?
Has there been published research on the "many ways" this can be solved yet?

I don't think this fully answers your question, but, as a related fact, the return distribution you get with distributional RL can be used as a mean to "alter" the policy in a smart way.
What I mean is that with the return distribution you also get the uncertainty related to the returns, and from this you can introduce the notion of risk. Basically, you have a distortion risk measure, $\beta$, which is a function that can transform your return distribution altering the way the policy would sample actions, since the underlying distribution has changed.
Once you apply a specific implementation of risk, you can make your policy to act more conservatively for example, therefore expecting to exhibit a safer behavior, e.g., in more uncertain environments.

The picture compares DQN (non-distributional) with IQN (a distributional agent), with different risk measures.
As you can see the sampling distribution of the actions (so the policy) can be influenced by the specific notion of risk.

This is called risk-sensitive RL, and is nicely introduced in chapter 4 of the IQN paper, which is a more powerful way to estimate the return distribution compared to C51. I suggest you to checkout the paper for a more in depth understanding.

In distributional value functions, you can still calculate a single expected action value, by summing over the distribution. You can use that to select a greedy action for $\epsilon$-greedy policies.
So in practice this is the same as the non-distributional approach, but with a sum to get expected Q predicted by the distributional q function

It is my understanding that distributional RL seeks to derive some further actionable insight than the estimated mean Q value of traditional RL.  If we only sum over the distribution does that not result in the mean or something with the same action as the mean?

@foreverska not necessarily, one more direct benefit of a distributional model is that it is more apt and stable. E.g for win/draw/lose results of a game, it's a good choice, and a more informative model even if optimal choice would be the same. Perhaps worth a separate question though


text:
Optimal sensor placement formulation as multi-armed bandit
Assume that I have $n$ sensors to be placed optimally in $N$ possible locations where $n<<N$.
How can I model this problem as a multi-armed bandit problem?

Can you describe what a multi-armed bandit problem is? That's the first step to understand how you can formulate this problem. I'd recommend that you read the related chapter in the "Introduction to Reinforcement Learning" book by Sutton and Barto. This also looks like a homework and you should tag it with [tag:homework].

you might have a problem with large $n$, but you can consider every possible allocation as an action, and  do that as a multi-armed bandit problem.
For example, say you have 2 sensors and 3 spots, then you have:

action 1: sensors in spots 1 and 2
action 2: sensors in spots 1 and 3
action 3: sensors in spots 2 and 3

then, just run some simulations and figure out the best possible allocation


text:
In multilayer perceptron, to update the Bias value, do we use the neuron's delta, or do we use the sum of all deltas?
In multilayer perceptron, to update the Bias value, do we use the neuron's delta, or do we use the sum of all deltas?
I ask this because I once saw a video on YouTube where the formula for updating the neuron's Bias in backpropagation, The correction factor to update Bias was a sum of all the deltas (of the layer).
But, in other videos and articles, such as this article here: https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/, the bias correction factor for the neuron is the delta itself. This is because Bias has no input, so its correction factor was just the neuron's delta.
I was in doubt: In multilayer perceptron, to update the Bias value, do we use the neuron's delta, or do we use the sum of all deltas?

I really like the way Michael Nielsen describes backpropagation in his book. I will not explain all the details of these formulas because for that it's way better to read his chapter on backpropagation.
The formulas are the following:
\begin{align*}
(1) \quad & \delta^L = \Delta_a C \odot \sigma'(z^L) \\
(2) \quad & \delta^l = ((w_k^{l+1})^T\delta^{l+1}) \odot \sigma'(z^l)\\
(3) \quad & \frac{\partial C}{\partial b^l_j}=  \delta^l_j \\
(4) \quad & \frac{\partial C}{\partial w^l_{j \space k}}=  a_k^{l-1} \delta^l_j \\
\end{align*}
Based on these formulas, the derivative of the cost function with respect to the bias is the derivative of the current layer's activation function with respect to the cost function, which in these formulas is represented by the $\delta^l$ variable. So to answer your question, no it doesn't use the sum of all Deltas of the layer. That would be a constraint we wouldn't want for the network because it would mean the derivative of all the biases in the network would have the same change factor. We want all the variables to be able to change independently of each other.Thats why we only use the neuron's Delta.
When we consider the sum of gradients, it is when performing gradient descent with multiple data points. Because the cost function is typically a sum of all the errors, we are supposed to calculate the gradient of that summation.
Let's say we are using the mean squared error cost function:
$$
C =  \frac{1}{M}\sum_{i = 1}^{M} (y'_i - y_i)^2
$$
When differentiating with respect to the output of the last layer we get:
$$
\frac{\partial C}{\partial y'} = \frac{\partial}{\partial y_i'} \frac{1}{M} (\sum_{i = 1}^{M} (y'_i - y_i)^2)
$$
But because we can exchange the summation of derivatives with the derivatives of a summation (because derivation is a linear transformation), we can calculate the gradients of each weight first and then do the summation:
$$
\frac{\partial C}{\partial y'} =   \frac{1}{M}\sum_{i = 1}^{M} \frac{\partial}{\partial y_i'} (y'_i - y_i)^2
$$
If you want another resource to see how backpropagation can be implemented, you can read the following code from the same author I mentioned before. Here they implement what I just said - they first calculate the gradients with respect to all the weights and then for the SGD update calculate the sum of the gradients over all the data points.

In this link, The author Makes the sum of each deltas of each neuron in each layer, during the mini-bath. So, this results in several "accumulated deltas" of each neuron in each layer. Did I understand the article correctly? So that each delta of each individual neuron will be accumulated during the mini-bath? And only then will he do the SGD? Am I correct?

Correct. We first get the accumulation of the deltas for each neuron in the mini-batch and then use that to update the weights. A common error I had when learning backpropagation was thinking that for each forward pass we do a backward pass, but it's not for each forward pass but for each batch. This is like this because the cost function must try to minimize all the errors of the data points, not only a single point. That's why we get the summation. Remember that it is not the same to update the weights with the sum of the gradients as updating them for each gradient independently.

This means that we never use the gradients calculated for each data point alone. Technically, to update the weights with gradient descent we must take the sum of the gradients across all data points. However, because it can be computationally expensive to do that, we perform gradient descent stochastically on small groups (batches) of the data using SGD.

Thank you for clarifying these points for me. This helps me


text:
Sutton & Barto: Why expected square of the importance-sampling-scaled return is for policy b?
From "Reinforcement Learning: An introduction (2nd ed.)" by Richard S. Sutton and Andrew G. Barto, on page 107 and 108:

We can verify that the variance of the importance-sampling scaled
returns is infinite in this last example. The variance of a random
variable X  is the expected value of the deviation from its mean X¯:

\begin{align}
\mathrm{Var}[X] \doteq \mathbb{E} \Big[ (X - \bar{X})^2\Big] = \mathbb{E} \Big[ X^2 - 2X\bar{X} + \bar{X}^2 \Big] = \mathbb{E} \Big[ X^2\Big] - \bar{X}^2
\end{align}

If the mean of the random variable is finite (as in the example),
having an infinite variance means that the expectation of the square
of the random variable is infinite. Thus, we need only show that the
expected square of the importance-sampling scaled return is infinite:

\begin{align}
\mathbb{E}\_b \Bigg[(\prod_{t=0}^{T-1} \frac{\pi(A_t|S_t)}{b(A_t|S_t)}G_0 )^2\Bigg]
\end{align}
Question is following: why we calculate expected return for policy $b$ instead of $\pi$? If we approximate returns for $\pi$ using importance sampling aren't we supposed to calculate $\mathbb{E}\_\pi$?

why we calculate expected return for policy $b$ instead of $\pi$?

The expected return for policy $b$ is not being calculated here:
\begin{align}
\mathbb{E}_b \Bigg[(\prod_{t=0}^{T-1} \frac{\pi(A_t|S_t)}{b(A_t|S_t)}G_0 )^2\Bigg]
\end{align}
This expression is part of the variance of the importance-sampled expected return for $\pi$, given that your observations of $G_0$ are all made following $b$. The expectation of this value is shown as expected following policy $b$, because that is what you directly observe.
If you were instead observing returns from following the policy $\pi$, you wouldn't have the problem with possible unbound variance when trying to convert it.
The proof is showing a problem inherent to the conversion maths. The importance sampling is still approximating $\mathbb{E}_{\pi} \Big[G_0\Big]$, but the proof needs to look at the conversion in detail to show the possibility of infinite variance.


text:
Why is the vanishing gradient problem especially relevant for a RNN and not a MLP
I would like to know why the vanishing gradient problem especially relevant for a RNN and not a MLP (multi-layer-pereptron). In a MLP you also backpropagate errors and multiple different weigths. If the weights are small, the resulting update in the last layers in the backpropagation will be very small

In the most simple case, you can think of RNN as a recursive MLP that calls itself multiple times. We can unfold it into a very deep MLP and as a result, the gradients vanish as they propagate through the numerous layers.

No, ResNet were not introduced to solve vanishing gradients, citing from the paper:

An obstacle to answering this question was the notorious
problem of vanishing/exploding gradients [1, 9], which
hamper convergence from the beginning. This problem,
however, has been largely addressed by normalized initialization [23, 9, 37, 13] and intermediate normalization layers
[16], which enable networks with tens of layers to start converging for stochastic gradient descent (SGD) with backpropagation [22].

However, vanishing gradient happens also for MLP for the same reasons why they happen in RNNs as you can see an unrolled RNN as a MLP at the end of the day: because you stack multiple layer, and if many of them saturate, the gradient will tend to zero
You can see it from an unrolled RNN:

Here, the gradient of $E_4$ with respect to $x_0$ will have to travel 6 matrix multiplications/non linearities, even though the net is just 1 layer deep.
If the spectral norm of such matrices is less than one (ie the highest eigenvalue is < 1), then they are contractive, and thus a vector multiplied many times by such matrix will be attracted to 0

Thanks Alberto for your answer. What I still don't understand is why are RNN especially prone for the vanishing gradient problem? I mean if you use a deep MLP it will look similar as in your figure and why is a deep MLP not prone to the vanishing gradient problem as strong as a RNN?

@PeterBe you should be able to see it from the picture. the problem is that the "connection" from the first input (x0) and the last output (En) is as long as the length of the input size... thus even a network with 1 recurrent layer, given a string of length 50, the last output is 50 layer transformation away from the first input (thus is like having a 50 layers deep net)

@PeterBe you're welcome, good luck with your journey

Vanishing gradient problem is indeed present in MLP and CNNs. Please have a look at ResNet paper: before the introduction of residual blocks, the vanishing gradient problem was one of the main limitations to the depth of the networks.
In RNNs this problem is present also with fewer layers because each layer needs to perform backpropagation through time, so learning is additionally constrained by the sequence length.

Thanks for your answer. What I don't really understand is why are RNN especially prone for the vanishing gradient problem? So why is the backpropagation through time as you wrote, especially prone to that?

Two problems for RNNs

They're typically "deeper" than a MLP since you can unroll a RNN into a stack of layers for each timestep, which exacerbates the problem
The activation function of choice typically is tanh which has diminishing gradients at the extremes

Thanks mf908 for your answer. Let's say I have a deep MLP and I also use tanh as the activiation function. Would it be as prone to the vanishing gradient problem as a RNN?

@PeterBe The deeper the network, the worse it's going to be. As far as the vanishing gradient issue is concerned, an RNN of k layers over n timesteps is equivalent to a MLP of kn layers. Can you see that this number kn (k times n) can easily be very large? Of course it's theoretically possible to come up with a very deep MLP and a very shallow RNN over very few timesteps, and then you can say "the vanishing gradient issue is worse for this MLP than for this RNN!!", but in general it's much worse for RNNs than for MLP.

Theoretically yes, it's a part of the reason why ReLu is the activation function of choice nowadays, previously it was sigmoid which similarly has diminishing gradient issues


text:
PINN - Why Differential equations only?
Im recently studying Physics informed neural networks.
I found that it is designed for solving Differential equations.
Is there any reason why it is used for differential equations only?

because ODE are very unstable learnt trivially by neural network, and the "physics informed" ones seem to behave better... but not, they can be applied in other areas

(Partial) differential equations describe physical laws in many scientific domains. The purpose of Physics-informed Neural Networks (PINN) is to incorporate prior information from physical laws into neural network learning so they can better model physical systems. Modeling physical systems is solving differential equations.
Example: Airplane
For example, assume you must determine how air flows around an airplane wing at different speeds, given some data gathered in wind tunnel experiments. The result you want is a system that takes a time-varying speed as input and generates a function describing the movement of the air over time. Since differential equations govern air flow, the function we seek is a solution to those differential equations. Traditional methods to solve these equations are costly and require assumptions or measurements of boundary conditions. Training a PINN with our wind tunnel data will create a solution that not only fits our data but also obeys the laws we know apply to airflow. A conventional neural network trained with that same data will be less accurate because it can generate solutions that don't take into account the knowledge we have about airflow. It relies only on the data we give it, so it has to search a larger space, and it can generate physically impossible solutions.
Example: Toy problem
Imagine you have time-position values as follows:

Time (t)
Position (y)

0
0.1

1
0.9

2
4.1

This is $y=t^2 \pm 0.1$ - the position of an object under constant acceleration starting at the origin with a measurement error of $0.1$.
I trained a conventional neural network on this data with absolute error:
$\hat{y}=w_{3,1} f(w_{2,1,1} f(w_{1,1} t+b_{1,1}) + w_{2,1,2} f(w_{1,2} t+b_{1,2}) + b_{2,1}) + b_{3,1}$
where $f$ is the ReLU activation function $f(x)=max(0, x)$
This produces the following trained network:

Parameter
Value

$weight_{1,1}$
2.031941197

$weight_{1,2}$
3.192294059

$bias_{1,1}$
-0.617635647

$bias_{1,2}$
-1.30428564

$weight_{2,1,1}$
1.548372668

$weight_{2,1,2}$
1.693544207

$bias_{2,1}$
-3.251014905

$weight_{3,1}$
0.374160198

$bias_{3,1}$
0.100685864

If we instead use the differential equation for constant acceleration starting from rest, $\frac{dy}{dt} = a t$, we can solve that by hand (thus a toy problem, you'd use PINN when you can't do this) to get $y=\frac{a t^2}{2} + b$. You can absorb the constant into the weight to get $y=w t^2 + b$. If I train this on the same data, I get:

PDE Param
Value

weight
1.000157219

boundary
0.099369488

This yields the following predictions (including some held-out test data in bold italics)

Time
Measured Position
NN Pred
DE Pred

0
0.1
0.100685864
0.099369488

1
0.9
0.899997887
1.099526707

2
4.1
4.100000482
4.099998364

3
8.9
7.300003076
9.100784458

4
16.1
10.50000567
16.10188499

5
24.9
13.70000827
25.10329996

Or in a plot:

The conventional neural network is actually closer than the differential equation on the training data (the diagram is misleading on the second training point because of errors in the plotting software). But, when predicting data not in the initial training set, it is much worse because it does not know the appropriate underlying physical law.
One can also see that the DE-based solution, though it is much closer, does not have the correct parameter values because it fits some measurement error. (It has a training error of $0.200158855$ and the correct solution has a training error of $0.3$.) A full Bayesian treatment (beyond what PINNs do) would account for the uncertainty from measurement and limited data.

Thanks for the detail answer! Reading your answer, I thought that PINN can be used in any equation if we can get the derivative, like y = m * exp(-b). Did I understood right?

The goal would be any equation where the integral is defined. However, that is a simplification. PINN is one method of solving DE using data. Current implementations have many cases where they become inefficient (for example, you might get the wrong answer, or it might not converge because you don't have infinite computation or you used the wrong approximation). See the "Limitations" section of the Wikipedia article for some known problem cases.


text:
Does Chat-GPT learn from interactions?
Does Chat-GPT accumulate knowledge through interactions with users in real time? I know it's trained on a massive dataset but does it do online learning?

Questions about specific products are off-topic here.

The core model for ChatGPT, a Large Language Model (LLM), is not updated in real time during chats.
The chat history can be saved, and if so used by OpenAI to train models for future releases.
You can see this option in Settings:

However, there have been leaked/previewed plans for changes to the higher level software (i.e. the input/output control that uses the model), and as part of that OpenAI have added a mechanism whereby the bot will somehow store key data from conversations, and use this data to customise responses. As an educated guess, this is probably a text summary (generated by an LLM reading the conversations) stored as reference data or as a second "system prompt". This new feature in progress has been labelled "Project Sunshine" by commenters. OpenAI have not formally confirmed this or released details of how that works AFAICS, but it seems like the ability to enable this, and to clear this personalised memory, will also appear in Settings.

Honestly it should retrain itself on accumulated data(perhaps it will in the future). Eventually you will get novel conversations where each bot is talking to another bot.

@FourierFlux but retraining is expensive and cause catastrophic forgetting if not done properly


text:
In the paper "LLM in a flash," what is meant by an up projection or down projection layer?
In the paper, they first use the terms "up projection layer," and similarly for down projection, in this paragraph in the introduction:

Row-column bundling: We store a concatenated row and column of the up-projection and down-projection layers to read bigger contiguous chunks from flash memory. This increases throughput by reading larger chunks.

What does this refer to in terms of the architecture of a given LLM?
This paper focuses on the Falcon and OPT LLM models.
To try to understand this, I've read through the paper for other explanations, and I searched online. One suggestion I found is that there are special layers in a network that (speaking informally) increase or decrease the size of the resolution, but I don't think that would apply to these specific LLM models. (For example, a max-pooling layer may be seen as decreasing resolution.)
The image below is relevant, but I don't think it answers the question. Here is the paper in question.

Up-project and down-project refer to the first and second feed forward layer respectively, found in each transformer block.
They use this in the context of sparsity in the feedforward layers in each transformer block.
Recall that each transformer block has two FF layers with ReLU activations. Each FF can be thought of as a "projection" to a different dimension. They noticed that with ReLU, the output of the first FF layer (the "intermediate values") tends to be very sparse:

The ReLU activation function naturally induces over 90% sparsity in the FFN’s intermediate outputs, which reduces the memory footprint for subsequent layers that utilize these sparse outputs. However, the preceding layer, namely the up project for OPT and Falcon, must be fully present in memory.

Because of this sparsity, you don't want to load the entire FFN weights, so the authors propose efficient ways to do this.

I suspect you are correct, and, in slightly more detail, that the authors want to load both the incoming weights for a particular neuron, as well as all the weights that will be multiplied by the output of that neuron in the next layer. One thing that made this hard for me to understand is that I don't think the terms "up project" or "down project" are standard for feedforward networks. At least not in the literature I have previously read.

The answer could be found in the author's another paper.
https://arxiv.org/pdf/2310.04564.pdf
For FFN(x) = linerar1(relu(linear0(x))), the first nn.Linear is the "up-project" , the second nn.Linear is the "down-project"


text:
Navigating encoder-only text/sentence embedding models
tldr; I am researching/looking for vec2text methods that work (a) without training a special model to guide the mutations, and (b) without access to the model weights.
I am doing some experiments in navigating the encoded space of an encoder-only text embedding model (such as OpenAI's text-embedding-ada-002, but not specific to this model), with the limitation that I do not have access to the weights. I am trying to find some heuristics to "travel" towards a target embedding within the space.
Note: this is about text embedding (e.g sentences, maybe paragraphs), not word embedding.
Some ideas I currently have:

Running an evolutionary algorithm to mutate the text in (text, embedding) pairs as individuals, and using dist(embedding, target embedding) as an objective.

At each step, locally finding "directional" vectors by taking the delta of two steps combined with some adjective, and associating meaning to the direction.

Applying a direction repeatedly if it gets me closer to the target.

Asking an LLM to {mutate, combine} an/two individuals' text, and also to tell me the semantic difference (giving meaning to the direction).

Inspiration/existing research: I am aware of the Vec2Text from Text Embeddings Reveal (Almost) As Much As Text, but i am specifically looking to do as well as possible without training a special model to guide the mutations.

Q: Is there any existing research on this sort of thing?

Q: Any better mutation heuristics? Other approaches entirely?

Q: I vaguely aware of the Manifold Hypothesis, are there ways to detect if an embedding is far from the manifold? Or a way to "get back" to the manifold?

The embedding space of different models may be more similar than you think. Your problem reminds me a bit of blackbox adversarial attacks, where you want to e.g., find an adversarial example targeting some API to induce an output without having access to model weights.
For example, Zou et al., 2023 and Wallace et al., 2019 both suggest finding adversarial examples with whitebox access to an ensemble of models (through gradient steps + discrete updates), with the goal being to induce certain types of outputs (e.g., toxic/harmful, different sentiment etc.). They then notice that these examples transfer to models that you only have API access to. This is pretty directly transferrable to your setting: take gradient steps + discrete updates (as described in the papers) to match the embeddings of an ensemble of whitebox models, then this might end up transferring to whatever blackbox API you're targeting.


text:
Difference between FLAX and pytorch
I am going through a code written in FLAX instead of pytorch .Can someone please explain what is the difference between these two deep learning frameworks?

Flax is a high-level library built on top of Jax, which is a recent numerical framework that allows the building of differentiable models. Compared to PyTorch, Jax (and so Flax) follows the principles of functional programming: immutable objects, functions are first-class, and stateless computation, to name a few. Pytorch, instead, follows an object-oriented programming paradigm where you represent your models as objects rather than the application of a function.
Basically, Jax and Pytorch implement the same thing with two different philosophies. Moreover, Jax is gathering attention thanks to its improved reproducibility since it is designed around stateless functions.

Hi got a doubt ..Is FLAX is optimised better than Pytorch or not? I am trying to convert a FLAX model to pytorch which has used nearly more than 10 Multi Layer Preceptrons for training the model.

@sripathiakhil I haven't done any benchmark, but I'll expect similar performance.


text:
Impact of scaling in loss terms when loss function is a composition of multiple functions
I am training a deep learning model, the loss function of which is of the form
$$
\cal{L} = \cal{L_1} + \cal{L_2}
$$
where $\cal{L_1}$ and $\cal{L_2}$ are of very different orders. WLOG, let's assume the order of $\cal{L_1}$ is much higher than the order of $\cal{L_2}$.
During the first several epochs of training, the model will attempt to minimize $\cal{L_1}$ largely. However, after a certain number of epochs, the value of $\cal{L_1}$ will converge.
My question is, what will happen now? Specifically, I have three questions:

Does the convergence of $\cal{L_1}$ imply the convergence of $\cal{L}$, which means the training is over and the loss function behaved as if it was essentially $\cal{L} = \cal{L_1}$?

Since $\cal{L_1}$ has now converged, does that imply $\frac{\partial{\cal{L_1}}}{\partial{\theta}} \approx 0$? (where $\theta$ are the model parameters)

If the above point is true, then since the model parameters are updated based on $\frac{\partial{\cal{L}}}{\partial \theta}$,  does that imply that the model will now start minimizing $\cal{L_2}$ (since $\frac{\partial{\cal{L}}}{\partial \theta} \approx \frac{\partial \cal{L_2}}{\partial \theta}$)?

not generally, consider a case where $L_2 = 1/L_1$, if one converges to 0, the other one diverges
Yes, that's the usual definition of convergence, though the "$\approx$" is not really well defined usually
exactly, however, after one update, most likely you will have that $\frac{dL_1}{d\theta}$ won't be 0 anymore


text:
AlphaZero Chess: High portion of "draws" during first rounds of self-play hampers learning
I am experimenting with a self-programmed version of AZ Chess following the described methodology in the official paper. I am experiencing that at the beginning of the self-play (when the ANN weights are is still randomly initialized) a very high portion of all games end with a "draw" (ca. 97% of all games). This is not surprising because if two pure random players play against each other, it is very unlikely that white or black achieve a win/lose. As a result, ca. 97% of all board states visited during the sell-play are trained towards a target value of 0 (instead towards +1/-1 for white to win/lose). This means that there is very litte training data (3% of all training data generated via self-play) that helps the ANN to learn how to win and to get stronger.
Does anyone know how DeepMind has overcome this problem? I have found nothing about this in the www.

Hello and welcome to AI Stack Exchange! How many games have you currently run during training?

Also, are you using a MCTS?

I have already run ca. 1000 self-play games. But that is enough to see that the vast majority end with a "draw" which does not generate any training effect.

Yes, I general I use MCTS during self-play, but at the beginning I set "number of roll-outs" to just 1 in order to speed up the process (that led to the 97% draws). But during the last 24 hours I set "number of roll-outs" to 800 and observed a decrease of "draws" (down from 97% to 80%). But the 80% still seem far suboptimal for effective learning. I am wondering if it was really the huge number of self-play games that solved the problem in the original version of AlphaZero, or do I still miss something conceptionally-wise?

@OliverEwald AlphaZero training with only a single visit is a bit strange, how would the policy head ever learning anything? You definitely need at least some rollouts!

First intuition
I wrote a quick script to test the outcome distribution of random chess games: I get about 7% wins for black and white each, so 14% non-drawn games. The MCTS should still be able to find simple mate in one moves even with a fully uninitialized network, so I would expect it to draw even less.
Your 97% draw rate is significantly higher than that, so I suspect something else is going wrong. Some ideas to try:

Do your own random moves test to confirm that you also get 7% win for each player, otherwise something went wrong with your board implementation.
Run MCTS with a random NN on a mate-in-one position, and ensure that it finds the mate and picks it as the best (highest visit count) move after a couple thousand visits.

I think in theory 3% decisive games is already enough to kickstart the learning process, but specifically for chess it probably means something in the implementation is wrong.
Actual data
Here's the win/draw/loss plot from a chess run of my own AlphaZero implementation, kZero. The horizontal axis corresponds to about 300k selfplay games played. This is with 400 visits/move, and settings otherwise mostly similar to the AlphaZero paper.
It looks like the behavior is as follows:

The initial games generated with a random NN have only 17% draws, so MCTS is finding a lot of mates
Very quickly the drawrate goes up to 35%, as the NN learns to avoid mate in one
After that the drawrate drops back down to 8%, as the NN learns to finish games from winning positions.
For the rest of the run the drawrate slowly climbs again, as is typical for stronger chess players and engines.

Thanks a lot for your quick and comprehensive answer!

I will do new random checks (without MCTS) to see if my win ratios will go up towards your 14%
I just checked that my MCTS clearly detects "mate in one"
What I find surprising is that your MCTS on a fully uninitialized NN only finds 17% draws. That's far off from my 97%
Currently I define a game as "drawn" if there is no mate after 100 moves. What rules do you apply to avoid endless games?
I will answer back here once I have 1. completed with (possibly) adjusted rules how to define draws.

@OliverEwald I only terminate games based on 3-fold repetition, the 50 move rule and insufficient material (only king vs king). For this run the average game length started at 100, peaked at 300 and then dropped back down to 110. Cutting of games at 100 moves indeed causes lots of draws, so that's probably your main issue!

Thanks a lot for clarification! I will test with modified termination rules.


text:
How can reward at time step $t$ can be a function of a state at time step $t+1$?

The above picture is from a textbook on RL.
I am confused with the notation $r_t=\mathcal{R}(s_t, a_t, s_{t+1})$. How can reward at time
step $t$ be a function of a state at time step $t+1$?

Does this answer your question? How are the reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ equivalent?

There is a free choice when deciding the indexing of rewards in reinforcement learning. You can decide to associate the immediate reward with the time step of state and action that generated it, or with the time step of the observed next state. Both conventions are self-consistent and unambiguous, they differ only in the notation used in expressions.
The most common convention is to associate reward with next time step, leading to trajectories that look like this:
$$s_0, a_0, r_1, s_1, a_1, r_2 . . . s_{T-1}, a_{T-1}, r_T, s_T$$
The alternative convention, that it looks like your excerpt is using*, looks like this:
$$s_0, a_0, r_0, s_1, a_1, r_1 . . . s_{T-1}, a_{T-1}, r_{T-1}, s_T$$
Whether you use 0 as first index or 1 is a similar convention.
In practice, the next state and reward that result from a state/action pair are part of an environment response where they can be calculated in any order. In some cases, $s_{t+1}$ is required before the associated reward, because the value of the reward depends on it, yet usually reward is shown in trajectories before the next state so that the listing has the same pattern $s,a,r$ throughout. It doesn't matter how the notation tracks that, as long as it is consistent. The timing of events is not different in the different conventions, only the index numbering.

* Note that the diagram that goes with the equations does show reward and state sharing time step indexing. This implies that there could be a typo and the author meant to write $r_{t+1}$ where they have put $r_t$ in equation 1.3. Or perhaps they have used two sources which use different conventions and not noticed.

So, you also agree that the equations and the diagram are not consistent? That was in fact my question.

@DSPinfinity yes I agree, something looks wrong there. However in isolation neither is incorrect

I don't know that I've seen that notation but one must remember that an MDP does not guarantee a transition from $S_1$ to $S_2$ via action $A_0$.  There may be some stochasity in the transition causing the agent to stay in $S_1$, successfully transition to $S_2$ or instead go to $S_3$.  Think of a grid world with a slippery floor that may cause the agent not to move or to over move.
The reward is not for taking action $A_0$ in $S_1$ but for the real resulting state.  Therefore the reward at time t is always for $S_{t+1}$.

Your question didn't specify the reference of your picture at all which may have additional context in terms of the specific meaning of your reward function such as whether it's a random variable or an expected value. For instance, your picture very much resembles Figure 3.1 of Sutton's RL book where the reward $\mathcal{R}$ is a random variable and they never wrote or needed $\mathcal{R}(s_t, a_t, s_{t+1})$, but instead, its expected value $r(s_t, a_t, s_{t+1})$ is defined on page 49 due to possible stochastic environment unlike a normal grid world game:

We can also compute the expected rewards for state–action pairs as a two-argument function $r : \mathcal{S} \times \mathcal{A} \to \mathbb{R}$:
$$r(s,a) = \sum_{r \in \mathcal{R}} \sum_{s' \in \mathcal{S}} rp(s',r|s,a)$$
and the expected rewards for state–action–next-state triples as a three-argument function $r : \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to \mathbb{R}$,
$$r(s,a,s') = \sum_{r \in \mathcal{R}}r\frac{p(s',r|s,a)}{p(s'|s,a)}$$

In MDP RL there's always supposed to be an explicit model of the environment including stationary state and reward transition dynamics, thus as you claimed there's no necessary need for the (expected) reward function given by the environment to ever have explicit dependence on the next entering state $s_{t+1}$ to solve any problem. In fact in Sutton's book the only place the 3-argument expected reward is explicitly used appears in a nondeterministic finite state diagram for a recycling robot example on page 52:

You see in such a diagram the expected reward function needs the explicit dependence of the entering state to conveniently express its value based on its entering state for each of the possibly branching state transition due to its random environment characterized by the above Markovian transition dynamics $p(s',r|s,a)$, and they're obviously mathematically compatible with the above usual 2-argument reward function.


text:
Why is generative QA + RAG so popular when extractive QA models exist that can provide factual information?
If the concern with using generative models for question answering is that these models aren't always producing factual information, why is it that people are using these models with Retrieval Augmented Generation (Open Generative QA/Closed Generative QA), rather than using a transformers-based extractive QA model that can refer users to the potential answer from actual text documentation?

The issue is that extractive QA limits the types of questions to ones that have answers that appear exactly in the text.
For example, consider the question:
Is Bob considered a good movie director?

It's very unlikely that you're going to have a document that says exactly Bob is a good movie director. But, you might have several documents that talk about the quality of movies that Bob has directed or interviews that discuss how well he works with actors etc. from which you can infer an answer from the totality of the evidence.
That's not say that extractive systems are always worse, you're just faced with the classic trade-off of a restrictive but predictable system versus a more flexible system that may e.g., hallucinate.


text:
What's the best criterion for evaluating activation maps in a CNN?
I'm currently studying CNNs and I had the idea of building a model without a fully connected layer at the end. I think this could be beneficial, if one can somehow model the desired outputs as a matrix. A basic problem solvable in this way is the following: let's say we want to place a dot on the nose of a person in an image. The basic approach is
img -> [conv layer] -> [pooling] -> [...] -> [fully connected layer] -> pred_x, pred_y

where pred_x and pred_y are the two nodes in the final layer, after the fully connected one, and represents the coordinates of the dot.
Every coordinate x, y can be represented with a matrix of size image_width x image_height with all zeros except a 1 in the position int(x), int(y). This means that it might be possible to have a CNN like the following:
img -> [conv layer] -> [pooling] -> [conv layer] -> [...] -> act_map

where act_map is the activation map resulting from the last convolutional layer. Given act_map we can extract pred_x, pred_y by getting the two indexes for which act_map is maximum.
Maybe this doesn't make sense with a single output, but imagine a problem where we want to predict the position of n dots on an image. Then we could simply have n activation maps at the end.
I have a few questions:

where can I find more resources on this topic? I'm sure I'm not the first one to have thought about this
could this really work? I don't really know how to create a loss function

Thanks

It looks like you are thinking about something like UNET architecture.
The final layer to your toy problem with a "dot" on the nose can be modelled as a convolution layer from last convolution layer of your network of a shape [BatchSize, Channels, Width, Hight] to the "output" layer of size [BatchSize, 1, Width, Hight].
You may have a dimensionality problem, since the width and hight of the output image can differ from the input. This can happened if you decide to, for example, perform pooling in your architecture, luckily there are a lot of upsampling methods. You also can use architecture of CNN, that does not change image's size, like UNET, YOLO. I recommend you to look at the problem of image segmentation, since it looks very similar to the example given by you.
As of your second point. If you would want to predict a mask of onces and zeros you can use sigmoid function for each activation ( pixel ) of last convolution layer to get the probability as an output and then train the network by minimising cross entropy between predicted probabilities and true "mask" of an image.

I agree with the answer by @vl_knd and want to add my two cents:
This problem does sound a lot like a special case of image segmentation. You want an image where each pixel gets assigned a class. In your case, most pixels will be class 0 and some individual pixels will be class 1, that's where the points are. A very similar field of research is 2D pose estimation. Given an image, you want to map a skeleton onto that image that indicates the pose of a depicted person. This is mostly mapping 2D points onto the joints of a person. There exists work for both the ideas you mentioned (regression of $x,y$ and prediction a heat-map representation of the $x,y$-points on an image) Here is a reading list of such papers. And this paper predicts the $x, y$ pixels in the form of heat maps.
You might also find some ideas in papers that tackle high-precision segmentation problems. The medical domain might be interesting (i.e. blood vessel segmentation).
When it comes to the loss function, it should probably be a bit forgiving. When you want a pixel $(x,y)$ to be classified as 1, neighboring pixels likely have some ambiguity as to whether they should be 1 as well. What can come in handy is smooth labeling. You could for example center a Gaussian at $(x,y)$ with some variance to include neighboring pixels and provide a smoother training signal.


text:
What do you mean by "updating based on a training example/batch" in Gradient Descent?
My understanding is this: When doing Stochastic Gradient Descent over a neural network, in every epoch, we run $n$ iterations (where the dataset has $n$ training examples) and in every iteration, we take a random sample and update the parameters wrt the sample.
However, in batch gradient descent, we take the whole dataset every epoch, and update the parameters wrt the batch. I have the following questions:

Why do we need to compute the loss function every time, especially if the value of the loss has no importance as such in the backprop process? Is it just to ensure that the loss decreases over time? How can you propagate the error back when the value of L is of no significance?
What exactly does updating wrt a "batch" mean? What would you take the input vector (required to compute gradients for the first set of weights) as? I assume that the loss is taken to be the average of the losses for the entire batch

In batch gradient descent computing the loss function every time serves several purposes. While the value of the loss itself may not directly affect the backpropagation process, as you said monitoring the loss over time is essential to ensure that the optimization process is converging. Also many optimization algorithms such as learning rate schedules or adaptive learning rate methods adjust the learning rate or other hyperparameters based on the behavior of the loss function during training. Therefore, computing the loss function is crucial for determining how the learning rate should be updated. Finally, monitoring the loss function can also be used for early stopping, a technique where training is halted if the loss on a validation set stops decreasing or starts increasing, which helps prevent the usual overfitting of neural networks and wasting computational resources.
Updating parameters with respect to a batch means computing the gradients of the (average) loss function with respect to the parameters using all examples in the batch and then updating the parameters accordingly. This process is repeated for each batch during each epoch of training and in your case each batch could simply mean the entire training dataset.
The input vector used to compute gradients for the first set of weights is a matrix whose row represents an individual example from the batch and whose columns represent the features of that example, and the number of features would typically determine the number of input nodes of your network. During training the neural network processes this entire batch at once with operations including forward and backward passes performed in a vectorized or parallelized manner for efficiency.

How can you propagate the error back when the very value of L is of no significance?

The gradients of L w.r.t. all the weight parameters computed in the backward passes sourced from the error computed from forward passes are very significant to be back propagated to update the parameters.

could you please explain how they are "sourced from the error computed from forward passes" ?

based on your specific loss function (usually MSE) it's just matrix add-mult operations flow through each layer in any forward pass to get the sum of each output node's squared error, you may average too as you rightly claimed. Some lib such as pytorch only has one line code to do a forward pass which would do all these things for you, you may read its open source code to see programming detail. Ideally here only one main question for each post, if still unclear I encourage you to close this post and ask a new question.


text:
3D Unet gives "output size is too small" error
I wrote simple 3D-Unet arch in pytorch to do segmentation on 3D images.
class UNet3D(nn.Module):
def __init__(self, in_channels, out_channels):
    super(UNet3D, self).__init__()

    # Encoder
    self.encoder = nn.Sequential(
        nn.Conv3d(in_channels, 64, kernel_size=(3,3,3), padding=1),
        nn.ReLU(inplace=True),
        nn.Conv3d(64, 64, kernel_size=(3,3,3), padding=1),
        nn.ReLU(inplace=True),
        nn.MaxPool3d(kernel_size=(2,2,2), stride=2)
    )

    # Middle
    self.middle = nn.Sequential(
        nn.Conv3d(64, 128, kernel_size=(3,3,3), padding=1),
        nn.ReLU(inplace=True),
        nn.Conv3d(128, 128, kernel_size=(3,3,3), padding=1),
        nn.ReLU(inplace=True),
        nn.MaxPool3d(kernel_size=(2,2,2), stride=2)
    )

    # Decoder
    self.decoder = nn.Sequential(
        nn.ConvTranspose3d(128, 64, kernel_size=(2,2,2), stride=2),
        nn.ReLU(inplace=True),
        nn.ConvTranspose3d(64, 64, kernel_size=(3,3,3), padding=1),
        nn.ReLU(inplace=True),
        nn.ConvTranspose3d(64, out_channels, kernel_size=(2,2,2), stride=2)
    )

def forward(self, x):
    # Encoder
    print(x.shape)
    print(type(x))
    x1 = self.encoder(x)

    # Middle
    x2 = self.middle(x1)
    print(x2.shape)

    # Decoder
    x3 = self.decoder(x2)

    return x3

My images are one channel with rectangle size, so I am resizing them to 512
transform = transforms.Compose([ transforms.ToTensor(), transforms.Resize((512, 512))])

my architecture summary seems good to me
----------------------------------------------------------------
    Layer (type)               Output Shape         Param #
================================================================
        Conv3d-1    [-1, 64, 128, 128, 128]           1,792
          ReLU-2    [-1, 64, 128, 128, 128]               0
        Conv3d-3    [-1, 64, 128, 128, 128]         110,656
          ReLU-4    [-1, 64, 128, 128, 128]               0
     MaxPool3d-5       [-1, 64, 64, 64, 64]               0
        Conv3d-6      [-1, 128, 64, 64, 64]         221,312
          ReLU-7      [-1, 128, 64, 64, 64]               0
        Conv3d-8      [-1, 128, 64, 64, 64]         442,496
          ReLU-9      [-1, 128, 64, 64, 64]               0
    MaxPool3d-10      [-1, 128, 32, 32, 32]               0
  ConvTranspose3d-11       [-1, 64, 64, 64, 64]          65,600
         ReLU-12       [-1, 64, 64, 64, 64]               0
  ConvTranspose3d-13       [-1, 64, 64, 64, 64]         110,656
         ReLU-14       [-1, 64, 64, 64, 64]               0
  ConvTranspose3d-15     [-1, 1, 128, 128, 128]             513
================================================================

but I am getting following erro when I try to train the model

Given input size: (64x1x512x512). Calculated output size:
(64x0x256x256). Output size is too small

Any idea how to fix it and what is going on
PS. the error comes from maxpooling3d layer from encoder part

Why are you using 3D convolution/pooling if you input a 2D image ?
The expected input size should be BxCxDxHxW for a 3D convolution, not  BxCxHxW . In your case, C=1 because you said you only have one channel. B is the batch dimension, which is usually added by the dataloader. And I suppose that you have H=W=512. But you have no 3rd coordinate (the depth D of the image, which should be a cube if you work in 3D imagery).
The 3D pooling is trying to do pooling on the channel dimension, which has value 1 since you only have one channel. Hence it returns an error because you cannot downscale more than one pixel...
You can use the Upsample function that is compatible with 3D images, instead of Resize :
https://pytorch.org/docs/stable/generated/torch.nn.Upsample.html
If you use 2D images then you need to use 2D convolutions.

I realized that my dataloader was skipping the depth, so I fixed it and it is running fine now. Though I have trouble with loss function. my dice score is coming out to be negative, but I will post the another question for that.


text:
Is "The Dimpled Manifold Hypothesis" correct to say this about autoencoders?
This quite famous paper states page 3 that:

The (well-known) fact which underlies the new conceptual framework is that all the natural images are located on or near some low-dimensional manifold (as shown by a huge number of previous papers, from Ruderman [1994] to Pope et al. [2021]). We can approximate this manifold by using a high-quality autoencoding DNN (Bourlard and Kamp [1988], Wang et al. [2014]) which first compresses the given image into a k-dimensional latent space, and then decompresses this latent space into a k-dimensional image manifold in the n-dimensional input space.

Does this make sense? Shouldn't decompression happen in a larger dimension? I always though autoencoders 'shrank' the dimension of the input then expanded it back to the original shape, but here it seems to imply the compressed image remains of the same dimension as its encoded form somehow.

The quote hangs on the definitions of "space" and "manifold". An example of a space might be the familiar 3D euclidean vector space, and an example of a manifold would be the surface of sphere within this space. You can also consider the surface of a sphere as a space, and a ring drawn on it as yet another manifold. Whether you think about about something as a space or manifold depends on context.
From the quote, the dimension of the image space is $n$.
The dimension of the autoencoder representation/embedding is $k$, and that is considered to be in "latent space" (the space from which all outputs will be mapped from) when you are looking at the compressed image vector at the encode/decode interface. You can consider moving within this space for instance by modifying the embedding vector directly, or compressing some new input image.
The decoder maps out those $k$ dimensions as a manifold in the image space. So it outputs a structure of $n$ dimensions, but only creates values that are in the manifold that it can represent by expanding from the $k$-dimensional embedding at the encoder/decoder interface. The expansion is always from a single point to another single point, no matter how much values can be multiplied, and the output shape twisted through the image space using non-linearities, it never fills the image space, instead it creates a $k$-dimensional surface within it. You cannot move in the $n$-dimensional image space arbitrarily and always end up in a position that the decoder can output.
The shape of the manifold - how those $k$ dimensions embed into the $n$ dimensional image space - is learned. However, the fact that the decoding produces a $k$-dimensional manifold in the larger space is a direct consequence of the architecture. Even an untrained decoder will do this, although the resulting manifold is of little interest.
Another way of understanding this discrepancy is that the compression is lossy (information is discarded), but the decompression is precise (new information is not created, nor is image space sampled from stochastically). So the decompression cannot fully "fill" the image space.

Thank you for your prompt answer and clarification, I understand now!


text:
When is it non-Markovian?
Several months ago, I was writing for class.  I claimed an environment was non-Markovian because it would take several states to de-alias some positions in the grid world.  I was corrected that it was n-step Markovian.  Something that didn't occur to me until the class had ended was:
If there's an n, then n could be infinity.  This would imply an MDP where the next state is decidable with infinite history.  This seems like a reasonable thing to call non-Markovian as it's not decidable within a finite time but is it?  Where does non-Markovian start?

Your observation about the possibility of n being infinite makes sense. If an environment requires an infinite unbounded history of states to make decisions, it is considered non-Markovian. However, in the strict mathematical sense of (1st order Peano) arithmetic supposed to model all the natural numbers, any specific number n in its domain is finite, no matter how big it is. Thus your n-step Markovian is not non-Markovian as considered above.


text:
REINFORCE with Baseline update rule
I was looking at the algorithm for REINFORCE with baseline from the Book 'Introduction to Reinforcement Learning' from Sutton:

I do not quite understand the update rule for $w$:
$w = w  + \alpha \delta \nabla \hat{v}(S_t, w) $
If I understand correctly, this would be the update rule, if $ \delta$ would be the MSE:
$\delta = (G-\hat{v}(S_t, w))^2$
$\nabla\delta = -2(G-\hat{v}(S_t, w))\nabla \hat{v}(S_t, w)$
which would give the same update as shown in the algorithm, when we include the $2$ into the learning rate $\alpha$
$w = w  + \alpha \delta \nabla \hat{v}(S_t, w) $
However, in the algorithm they do not use the MSE for $\delta$, but simply
$\delta = G-\hat{v}(S_t, w)$
$\nabla\delta = - \nabla \hat{v}(S_t, w)$
which would yield a different update rule (compared to the one in the algorithm)
$w = w  + \alpha \nabla \hat{v}(S_t, w) $
What am I missing?

The value $\delta$ is already representing a derivative equivalent to derivative of MSE loss for the difference between observed and predicted return. Multiplying it by the gradient of $\hat{v}$ to get a gradient for the weights is applying the chain rule.
Your equations would work and be compatible with the book, if instead of trying to redefine $\delta$, you created the implied loss function $L = \frac{1}{2}(\hat{v}(S_t, w) - G)^2$ and set $\delta = \frac{d L}{d \hat{v}}$


text:
Batch wise Inference to speed up Muzero's MCTS
Context:
I've implemented Muzero for the game Tic-tac-toe.
Unfortunately, the self-play and training  is very slow (like 10 hours until it plays quite well).
I ran the python profiler to find the parts that take the most time.
The result is that most time is spent doing Monte Carlo tree searches, specifically
querying the neural networks for the next hidden state and the value, policy predictions.
While running self-play, my GPU is only on 30% load and my CPU is on max load (single process since GIL)
Note: In the muzero paper, they have done some fancy stuff like scaling some gradients, which I haven't implemented yet. This will probably also result in a small speedup
What I'm trying to do:
I want to speed up the self-play by running multiple MCTS's in different threads so that they pause whenever they want to query the neural network until enough other threads have queries for the network. Then I put all the queries into a batch and sent the batch to the network. Once I have the results, I return them to each thread, and they continue until they try to query the network again.
Reason:
Let's say I want to play 100 self-play games. I do 50 simulations per move, the average game length is 7 and the observation shape is (3,3,3).
With my current approach, this would result in 100 * 7 * 50 = 35000 network queries of shape (1,3,3,3).
With the approach described above, I could run all 100 games at once and batch the network queries, resulting in 7 * 50 = 350 network queries of shape (100,3,3,3)
I hope that this will result in a significant speedup.
Questions:

What are your thoughts on this plan?
Any frameworks/PyTorch features that can help me with my plan?
How would you implement something like this?
How would you tackle problems like threads never waking up because the batch never gets full enough (if there's something better than timeouts), or possible race conditions?

Don't feel obligated to answer all of these questions.

you can just use minimax for tic-tac-toe...

@Alberto Thats true but Tic-Tac-Toe is only a simple first step. Later on i will also let it learn other games with an action space to large for minimax.

Yes, this could arguably be more focused, but I don't consider that sufficient justification for closing the question.

Batching: A Good Idea
You're right, batching is a great way to speed up AlphaZero or MuZero self-play! Your proposed solution of running multiple games in parallel is the easiest way to achieve some batching. There are other solutions, most notably Virtual loss, which allows you to get batches even from a single tree search. You can even combine both approaches.
Implementation Details
Unfortunately, I don't know of any PyTorch features that help with this, and in general Python is not the best language for concurrency and multithreading.
The way I implemented this in kZero (in Rust):

There is a single thread responsible for NN execution that listens to a channel/queue of work items. Once it has collected enough items, it evaluates the batch and sends the results back to each work item through a dedicated temporary response channel that was part of the original item.
Different games run on different threads or async tasks, send their evaluation requests to the main thread, and wait for the response.

The way to implement this in Python is with either threading and queues or with asyncio and its built-in queues.
See some of the diagrams in the kZero readme for some graphics to illustrate this.
An alternative is to share a single thread between multiple games and write a state machine to switch between them yourself. Effectively, this means you're implementing your own async-like system, this is tricky to get right! I did this at first but later switched to using async proper to simplify the code a lot.
Yet another alternative is to do what mctx does and let JAX "magically" handle all batching and concurrency. I'm not sure how well this actually works in practice, I haven't used JAX myself yet.
Deadlocks and Race Conditions
You can avoid the potential deadlock of batches never filling by just running enough games at once! For example, if you're only running a single NN execution thread with a batch size of 100, spawning at least 100 games will make deadlocks impossible. It's best to spawn at least twice as many games to get some concurrency, that way, by the time the first 100 requests have been evaluated, the next 100 are ready to go, and the NN thread never has to wait long.
The way to avoid deadlocks and race conditions caused by concurrency bugs is to write correct code! This is tricky to get right and basically an entire domain by itself. I've found that only communicating between threads using (concurrency-supporting) queues makes it relatively easy to write concurrency-bug-free code.

This is has helped me a lot. I really like the fact that you use something fast like rust for the mcts and just use python for the nn queries and training.  For now im just using the library ray to run games in parallel. Its still pretty slow. For connect4 it takes 7min to run 100 games with 50 simulations per move. The training itself is pretty fast. I've implemented batchwise unrolling to generate predictions. I'm able to do 30 training steps per second at batch size 32.

I'm happy to hear that! Yeah using a systems language for tree search and python for training is pretty common, LC0 and KataGo are doing the same thing. Indeed for AlphaZero selfplay is the part that's important to optimize, training is only a small fraction of the total cost.


text:
Activation function intuition question
I just want to verify the my intuition of why activation functions are necessary. For this example lets consider a network that classifies numbers 0-9. A network WITHOUT an activation function will be able to do well on numbers that are similar to the sizes of the numbers in the training set but it will struggle if numbers appear to be darker or lighter because it is linear and cannot take both size and lightness/darkness into account. In a neural network completeing the same task but WITH activation functions will be able to take into account both orientation and lightness/darkness because the weights will learn all possible relationships between the pixel values in the data set and the sigmoid(or other acctivation function) will then take number that are slighty lighter or darker and transform/smooth them so that the network could ouput the same probability as if it were the nomral darkness/lightness. Does this intuition sound about right or is it incorrect in some way ?
Here is another example of what I am trying to descirbe if this makes more sense: I understand it is a non linearity but mabey let me try and explain what I am thinking in a diffrent way. In a network that is detecting numbers 0-9 the network has learned for example all the relationships between the pixels of a 7 of standard darkness in all sorts of positions. Then during test time a lighter 7 comes along with slighlty lighter pixel values. In one of the neurons in the input layer after the lighter 7’s input has been multiplied by the weights and biases for that neuron when it is transformed by the sigmoid function it will have aproximatly the same value the darker one after passing through sigmoid in neuron one would be 0.994 and the lighter one 0.992 so then the lighter one will get treated the same through all later layer of the network and get classified correclty. Does that make sense ?

Why do you claim "it will have aproximatly the same value the darker one after passing through sigmoid in neuron one would be 0.994 and the lighter one 0.992"? Imagine you simply have Relu activation function, it's obviously not the case but the output of the whole network can still work well.

Can you please put your specific question in the title? "Activation function intuition question" is just the announcement of the question, but we already know that you have a question.

Activations can help with brightness, but not in the way you described, not by smoothing.
Without activation, all your layers collapse to a single linear operation, just because in matrix operations W1*W2 = W3. That means the prediction from every pixel is independent. Like, you can learn that the pixel at position (5,6) is brighter for #7 and darker for #1. That's it; it's the only level of complexity you have.
With activations, you can build a deeper network with more complex relationships. It can train that neuron 17 on layer 2 could have average brightness and then it can be subtracted from all other pixels.
As for size change, activations don't help with it. In fact, even convolutional layers are not scale-invariant, let alone multiple linear layers. For that, you would need something like a U-Net or multi-scale networks.
Bottom line, people learned that non-linearity is extremely important, but the exact shape of it is not so much.
Google has a nice playground for MLP https://playground.tensorflow.org/
And an extended explanation, including MNIST https://cloud.google.com/blog/products/ai-machine-learning/understanding-neural-networks-with-tensorflow-playground

Thanks of the response. I understand that the activation function does not help with the shape. If the two 7's are the same in every way except brightness/darkness when those pre activation values are entered into a sigmoid they will come out to be around the same 0.993 vs 0.991 so then they will be treated the same throughout the rest of the network. Does that make sense ?

The case you describing is when nn is very confident anyway. Now imagine it gave probability not 99.3%, but let's say 90%. With half the brightness it will recalculates as sigmoid(logit(p/2)) = 73%, which is almost triple error.


text:
Why does averaging attention-weighted positions reduce the effective resolution in transformers?
I was reading this blog post from Harvard and it says in its background paragraph about transformers that the number of operations required to relate signals from two arbitrary input or output positions doesn't grow in the distance between positions with the transformers architecture but this comes at the "cost of reduced effective resolution due to averaging attention-weighted positions".
I don't understand why does the averaging of attention-weighted position reduces the effective resolution. I have developed below the calculations needed to get the attention output, for the sake of simplicity I'm only considering one head:
The definition is:
$A = Attention(Q, K, V) = softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V$
I think we can see this in two ways, either we see that $softmax(\frac{QK^{T}}{\sqrt{d_{k}}})$ is an $(m, m)$ matrix if $m$ is the number of input tokens, let us denote this matrix $M$, and since $V$ is a the input matrix $X$ multiplied by a weight matrix $W^{V}$, that gives us $A = M*X*W^{V}$ and we can clearly see that every element of $A$ is bound to all element in $X$ and that makes the resolution as high as possible.
We can also see it if we completely develop all the calculations inside the matrix $A$. I will do just one element here for the sake of simplicity: $A_{1,1} = \sum_{j=1}^{m}\frac{e^{Q_{1}K_{j}^{T}}V_{j, 1}}{\beta_{1}}$
Where $m$ is the number of input tokens, $Q_{i}, K_{j}$ are the i-th and j-th rows of $Q$ and $K$ respectively and $v_{j, 1}$ is the first element of the j-th row in $V$. And $\beta_{1} = \sum_{j=1}^{m}e^{Q_{1}K_{j}^{T}}$ (the softmax denominator).
Let's say $X$, the input matrix is of shape $(m, d_{e})$, then $Q = X*W^{Q}$ is of shape $(m, d_{k})$ and $K = X*W^{K}$ is of shape $(m, d_{k})$ too if we consider that $W^{Q}$ and $W^{K}$ are both of shape $(d_{e}, d_{k})$.
Now, If we examine the product $Q_{1}K_{j}^{T}$, it is $\sum_{l=1}^{d_{k}}Q_{1,l}K_{j,l}^{T}=\sum_{l=1}^{d_{k}}(\sum_{p=1}^{d_{e}}X_{1,p}W_{p,l}^{Q})(\sum_{p=1}^{d_{e}}X_{j,p}W_{p,l}^{k})$, we can see that with the indices $p$ going from $1$ to $d_{e}$ and the $j$ index going from $1$ to $m$, we cover all the elements of $X$ in the first position of $A$. Granted that this is inside an exponential and that we divide by the softmax denominator. But if we develop the $V_{j, 1}$, it is $\sum_{p=1}^{d_{v}}X_{j,p}W_{p,1}^{V}$, and again we cover all the elements of $X$.
My guess is that there is a subtlety in the claim of the article that stems from the "effective resolution", but I wonder what a full resolution would be in that case, since even in the fully connected networks we have a linear transformation of inputs so we can argue that we're not directly using all the inputs.
I hope someone can clarify this for me and help me gain more understanding of the transformers architecture.
Thank you!

Exactly the same question occurred to me on reading that para just now. Did you find an answer? The claim of compensation for so-called limited resolution by multi-head attention" was also interesting.

@JulianMoore sorry for my late reply, I didn't have an answer at the time. You can check out the answer to this question. I've also found the section named Reduced effective resolution and Multi-Head Attention in this blog post to answer this question as well.

I think that the idea of "reduced effective resolution" from averaging is best understood through seeing how the proposed multi-head attention architecture fixes the issue. Specifically, multi-head attention is a concatenation of independently calculated but lower dimension attention outputs.
When your data's underlying structure is lower dimensional this should be fine for the most part when it comes to representing the data, but the benefit comes from the feedforward layer right after the attention mechanism — whereas single head attention would return what can be interpreted as some average of features that the layer is attending to, multihead attention outputs what are basically separate lower dimensional vectors, where the span of these is accessible to the ff layer. So while technically the dimension of the output of the layer is the same, it more easily preserves information about the structure of its input for the following layer.
Sequence aligned recurrences in CNNs and RNNs don't have this exact same issue, but information is still diluted through the potentially many operations (i.e. other layers / neurons) between what a layer is attending to and itself.


text:
What language model can convert normal text to JSON data
I have tried training T5-small, T5-base and T5-Large on around 15K rows of data where input data was something like but I did not get desired results
Nutrition Facts,
100g per,
Energy 646.95Kcal Carbohydrates 19.31g,
 Protein 21.94g 53.55g Total Fat 6.64g Saturated Fat 14.97g Dietary Fiber,
<1.Omg Cholesterol Sodium 0.29g Sugars 3.39g,
Lightly Salted and to Perfection,
Ingredients: Peanuts, Almonds,,
Cashews, Pistachios, Vegetable Oil, Salt,
aa, ,
74G,
Pistachio, 61129110611336177
WE ARE NUTS ABOUT QUALITY,
Baked,
Nuts Salted,
Mixed
WE ARE NUTS ABOUT,
Community 364, 13 Street Plot No. 36,
Al Area 1, 24149,UAE 4971 4 3355777,
License Number: 224614 VAT No: 100058529700003,
CERTIFIED COMPANY,
ALLERGEN WARNING: in a facility that also processes nuts, sesame and mustard,
Store in a cool dry place away from heat moisture,
Instruction Once store in airtight container and consume before expiry date,
Pro: 14/12/23,
Exp:13/12/24,
Net 40gms

Output data will be in JSON format of the above details.
What language models can be trained for this purpose and minimum how many Parameters it should have?

I've tried Mistral-7B-Instruct-v0.2, and it worked out of the box for your example, as did Qwen1.5-1.5b. However, Qwen1.5-0.5b had issues, but I think with enough training, you can achieve decent results.
I believe the size/quality trade-off of starting from 1.5b and lowering it could deteriorate in corner cases, but the only way to find out is through experimentation with validation.
One note is that you can generate almost an infinite amount of data by removing the JSON structure. 15k seems sufficient if you have just a product JSONs, but for more diversity, you might want more data. The SFT stage in an LLM usually takes from 200k up to a million examples.
You can also constrain the output using a scheme by restricting the output tokens. See one example of such an approach at https://github.com/1rgs/jsonformer.


text:
Is there a theoretical way to determine the best learning rate for gradient descent if the function is a simple known polynomial?
I was playing around gradient descent topic. Wrote a function that calculates a gradient descent of a degree-2 polynomial. While trying out what is the best "step size multiplyer" (a.k.a. "learning rate") I found that value of approximatively 0.7068 works best. I cannot figure out why it's best though- I simply found it by trial and error. Is it possible determine such values without any trial and error? Is there any mathematical theory that digs deep into this topic? Or is it either manual trial end error always? Or do I have to write a gradient descent over the gradient descent function in some way (with the "learning_rate" as input and the "number of steps" as output?
Quick search yields topic of Lipschitz constant, but it does not seem to provide a result similar to that value. The code in question:
def gradient_descent(derivative_function,
                     position,
                     step_multiplier=0.7068,
                     min_step=0.0001,
                     max_steps_num=1000):
    current_step_num = 0
    while True:
        slope = derivative_function(position)
        step = step_multiplier * slope
        new_position = position - (step * step_multiplier)
        if abs(step) < min_step:
            print(f'Best result (step {current_step_num}): {new_position}')
            return new_position
        if current_step_num > max_steps_num:
            print(f'Best result (step cap {current_step_num}): {new_position}')
            return new_position
        current_step_num += 1
        position = new_position

# With step_multiplier=0.7068 an optimal value is found in just 3 steps
gradient_descent(
    derivative_function=lambda a: 2 * (a - 1),
    position=-10000,
    step_multiplier=0.7068
)  # prints: Best result (step 3): 0.9999999943355022

# With step_multiplier=0.1 an optimal value is found after 833 steps
gradient_descent(
    derivative_function=lambda a: 2 * (a - 1),
    position=-10000,
    step_multiplier=0.1
)  # prints: Best result (step 833): 0.9995185063601436
```

You are considering a one-dimensional parabola; in this case, it is easy. The derivative is twice the distance to the optimal point, so the optimal learning rate is always 0.5/a, where a is coef near x2.
Your code has a small bug. I'll leave it as an exercise for you to find it. Hint: your optimal number is sqrt(0.5).
In fact, in the one-dimensional case, there is always such a learning rate (lr) that you can find a solution in one step, i.e., lr = x/d, where x is a distance to an optimal point, and d is the current derivative. The problem is that it depends on the initial position for any case when the dimensionality is larger than 2.
For future study, you can look up term 'optimization'. As one example, there are nice and accessible materials from a course at Stanford: https://web.stanford.edu/group/sisl/k12/optimization/#!index.md.

Thank you! Found the bug, it was the double step_multiplier :D. Also indeed, with 0.5a multiplier it's just one step  with a perfect result, vs 9 steps and an approximate result, for example, with a 0.6a multiplier. Will read through the link, thanks again.


text:
Books that contains exclusively math problems/assignments in Deep Learning & Neural Networks
I am doing a Deep Learning Course.Suggest some books that contains exclusively math problems/assignments in Deep Learning & Neural Networks. I can understand that majority of the replies suggest  "Deep learning", by Goodfellow, Bengio and Courville, which focuses on math, mostly linear algebra and calculus. But I am looking for books/articles/"assignments from universities" that contains/deals with EXERCISE MATH PROBLEMS in DL & ML.
For example I have been given the following problem which I have uploaded (i dont except the solution, I'll find it somehow):
Since I am new to DL & NN, I want to know HOW & WHERE to go about, understand the problem & find a solution for it. I am not interested in searching AI chatGPTs for the answer.
please help and thank you

From your problem example it seems an entry level neural network course to find a decision boundary of a single perceptron for linearly separable training data. Hagon's book Neural Network Design (2nd edition) specifically addresses this problem on page 4-5 and it also has many solved problems.

The decision boundary is determined by the input vectors for which the net
input is zero

So for your case the decision boundary is $w_1x_1+w_2x_2+bias=0$
And on page 4-7 it has an intuitive diagram of the decision boundary for AND function, you OR function is very similar. So intuitively you can immediately see the only possible correct answer is c since $x_2=-(w_1/w_2)x_1-bias/w_2$ which has the right slope and $x_2$-intercept. Later the book also dives into the supervised training rule for single and multiple perceptrons which can be generalized later.

From your comments it seems like your course is covering topics related to feature descriptors in the context of image or signal processing. These are fundamental concepts in computer vision and signal processing which is not what this site focuses on and you may try other closely related sites.
In computer vision, a feature descriptor is a representation of an image region or object that is used for tasks such as object recognition, matching, and tracking. In this context, a signature usually refers to a representation or descriptor that characterizes a specific aspect of an object or pattern within an image or signal. In image processing, a co-occurrence matrix is a matrix that describes the joint occurrence of pixel intensity values at different spatial offsets in an image which is often used for texture analysis.

I have found your reply much helpful. I hope this forum will help me find answers to my questions. Thank You

@kreetykishore Thanks for your positive comment, if an answer is helpful to you I encourage you upvote it or even accept it if satisfied :)

i tried to upvote, i have only 11, it is asking for atleast 15.                                                                 how to accept?

Oh, gosh. I believe there's a check like icon below each answer point to accept which as original poster should have right to check.

In that case I would recommend to look for courses offered by universities and solve their problem sheets. For example, have a look at this course Neural Network Theory at ETH Zurich. You can also find past exams if you want to solve them by your own, you can find them here. Also I can recommend The Science of Deep Learning by Iddo Drori.
But if you are specifically looking for "school books" the best way to go is to look for content of Deep Learning courses offered by universities. You will likely find literature that supports these courses. Also, have a look into MIT's Theoretical Foundations for Deep Learning course course.

Thank You for your reply. I hope I can learn something from the sources given by you. Please  suggest  me more books/resources that have helped you to learn DL/NN

it helped me mostly by solving problems myself, especially when coding, because programming is debugging your thinking:) you learn best by doing. therefore it would be beneficial to know, what are you mostly interested in? ML/NN is a broad field, from CNN to GNN to NLP ...

"ML/NN is a broad field, from CNN to GNN to NLP"                                                                         That's what I am trying to tell you. I am new and seeking advice from this forum.                         MY course started right of with topics like "feature descriptor", signature, fourier descriptor, intensity histogram, co occurrence matrix which I am not able to find in the recommended books. I don't know where to start.                                                               if you don't mind kindly tell me how to contact you so that I can explain my doubts.


text:
Gymnasium/Petting Zoo: Creating a copy of the board/env
I'm attempting to create a Tic Tac Toe player using MCTS.  For the game environment, I'm using Tic Tac Toe from the Gymnasium/Petting Zoo environment.
Running MCTS on Tic Tac Toe requires simulating hypothetical board states.  To do this, my idea is to copy the state of the board at the root node, creating a new environment where I could simulate a hypothetical move.
I don't see any built in functions for this, so I attempted to see if copy.deepcopy() would copy the board state for me:
import copy
from pettingzoo.classic import tictactoe_v3

env = tictactoe_v3.env(render_mode=None)
env.reset(seed=1)

env = copy.deepcopy(env)
env.step(0)

This gives an error: AttributeError: 'raw_env' object has no attribute '_cumulative_rewards'
Clearly, deepcopy is not fully copying the board state.  So how am I supposed to consider hypothetical board positions starting from a given state?  Technically I could replay every action in the history to get the new board there, but that seems wildly inefficient, especially since I plan to play more complex games in the gym environment after this.

So how am I supposed to consider hypothetical board positions starting from a given state?

In an environment designed for model-free reinforcement learning, you cannot, at least not via any of the public functions. That's because look-ahead search is not really a part of standard RL toolkit.
If you want to run a model-based algorithm for tree search algorithms, you need a separate game model implemented. One model instance is maintaining state for the environment, and the second model instance can be used for planning. Sometimes gym libraries do decompose into an environment wrapper and a game engine that you could do this with, but that needs to be checked per environment and there is no standard approach here. The only rule is that the standard gym environment interface has to exist (e.g. reset and step functions).
A lot of the simpler environments directly implement game rules as functions within a single environment class, and cannot be separated out. This is the case with tictactoe_v3 (the source for the step method is a long function with significant game rules implemented, not available separately) - in which case the normal option is to implement a new game model and use that for search/planning algorithms.

So you need to build a TicTacToe class from scratch in this case?  Why use Gymnasium at all, if you have to build a model of the game from scratch (said game model could trivially implement a step() function, so why use Gymnasium)? Also, will this be the case for Connect Four and Chess?
This seems incredibly inconvenient, I thought Gymnasium was intended to be agnostic to which reinforcement learning approach you were taking.

Note that I am attempting to implement the AlphaZero approach to TicTacToe, and later to Connect Four and Chess.  I'm amazed that the most prominent RL environment would not support one of the most prominent approaches to RL in recent history.

@BenG it is not as inconvenient as you seem to think. You will still spend far longer on your RL implementation. Either way, I didn't design or write this library. Don't shoot the messenger, and I hope you find the answer useful as it is factual

Gym is not very useful for two player games in any case, as it doesn't really support multiple agent training. When I implemented a Connect 4 agent, I just wrote the model. For chess, maybe look for an engine.

The dissapointing thing is that I was looking forward to a standardized environment for playing these games, with a common interface.  So that the same system could play multiple games.  But it seems like no one has created such a common interface and you need to implement each game individually

@BenG: There might be some library out there that has a few multiplayer games and a standardised interface to make agents for them. But that's not Gym, and not Gym's purpose.