text:
Why is the convolution layer called Conv2D?
When I build a convolution layer for image processing, the filter parameters should have 3 dimensions, (filter_length, filter_width, color_depth) is that correct?
Why is this convolution layer called Conv2D? Where does the 2 come from?

A 2D convolution is a convolution where the kernel has the same depth as the input, so, in theory, you do not need to specify the depth of the kernel, if you know the depth of the input.
I don't know which library you are referring to (although you tagged your post with TensorFlow and Keras), but, in TensorFlow, you only need to specify the width and height of the kernel in the Conv2D class, given that the depth is automatically calculated from the depth of the input.
Thus, the $2$ comes from the fact that you slide across two dimensions (i.e. width and height).
On the other hand, in a 3D convolution, the depth of the kernel does not necessarily have the same depth of the input, so, in that case, you also slide across the depth. In TensorFlow, you need to specify the width, height and depth of the Conv3D class.


text:
What is the best activation function for the embedding layer in a deep auto-encoder?
I am designing a deep autoencoder for graph embedding (exactly node embedding) following this paper SDNE. In the original paper, they used the sigmoid activation for all hidden layers in the autoencoder model, even for the embedding layer.
However, I think the embedding layer should use the tanh activation and the reconstruction layer should be used ReLU activation. Because, embedding is in the range $[-1, 1]$ and reconstruction layer is in the range $[0, x]$, which generates better results due to a larger range for representation and directed graph. Instead of in the range $[0,1]$ from sigmoid will lead to a lack of embedding information.
So, what is the best activation function for deep autoencoders to capture good information about the structure of graph?

It is hard to tell what exactly is better because these are hyperparameters. However, the sigmoid activation function is closer to biological neurons.
In the paper below, Bengio demonstrates why ReLU activation functions are better for hidden layers. In summary, they increase the sparsity of calculations (matrix in each layer shod multiply to its relative weights), and because of that, it made it possible for data to classify faster and easier.
http://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Does a better discriminator in GANs mean better sample generation by the generator?
Since the discriminator defines how the generator is updated, then building a discriminator with a higher number of parameters/more layers should lead to a better quality of generated samples. So, assuming that it won't lead to overwhelming the generator (discriminator loss toward 0) or mode collapse, when engineering a GAN, I should build a discriminator as good as possible?

If you start to train a GAN in a way in which the discriminator becomes very powerful than the generator, then the training of generator will not be very much successful. In the same way, if the discriminator is too low in comparison with generator, it would let it produce any thing. So, either way, it will affect the training of the entire system.
So, if you see here, both generator and discriminator and competing against each other and on the other side they are dependent on each other for efficient training.
But I've read in a research paper that adding noise to discriminator is good for the overall stability. And sometimes people also change the labelling system as well. For instance, the value for YES is 1 and people change it to 0.9. This actually keeps the discriminator from being over confident.

It is only true for the original GAN loss. The SOTA approaches use different loss functions and regularization (WGAN-GP, R1 penalty)

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Were AI strategies identified at go or starcraft games and how?
When an AI is trained to play an opposing game, such as chess or go, it can become very strong.
I have read in an article (non-scientific) the claim that AI strategies were identified by scientists while an AI was bound to play go games, as well as starcraft games. However it did not tell what these strategies actually were, how they were identified, nor did it explain the configuration in which AI played (AI vs AI? AI vs human?)
Can someone explain it to me? I am familiar with go, not with starcraft, so an explanation about go is appreciated.
I also note that the chess game is not mentioned. Is there any specific feature for chess that makes them inappropriate for strategies? Or is it the behavior of an AI in the chess game that does not allow to identify strategy?
I understand there are plenty of definitions for strategy, and the article did not give one. So let's focus on following significance: Strategy is a group of principles that tell which fields are important to fight for and which are not. A strategy gives long term rewards, which is opposite to tactics with short term rewards obtained thanks to calculation on a specific issue. With this definition, go game stand for a strategic game with a few, well known tactical situations such as line versus line.

Can you provide a link to the article you mention?

Welcome to SE:AI I know what your are referencing. (Players like Lee Sedol commented we can learn much from these algorithms.)  A link to the article, or other articles referencing this notion would definitely be helpful though.

Truth be told, I have no idea how to play Go, but luckily this is a AI forum and not a Go forum. Addressing your questions about the specific strategies that AI discovered, there's a paper released by OpenAI titled "Mastering the game of Go without human knowledge" (https://deepmind.com/research/publications/mastering-game-go-without-human-knowledge). Here, there's a section titled "Knowledge learned by AlphaGo Zero." In it, they mention that AlphaGo Zero discovered some variants of common corner sequences. I think it may be worth checking out. The section also lists some other common Go strategies that AlphaGo Zero learned. I believe during the training process, AlphaGo Zero was trained using self-play, where the AI played itself over and over again to get better.
To address your questions about chess. I don't think there's anything limiting AI from learning strategy in chess or anything about chess that limits AI from learning an effective strategy. Check out this paper for more: https://deepmind.com/research/publications/Mastering-Atari-Go-Chess-and-Shogi-by-Planning-with-a-Learned-Model
Here the authors present a general reinforcement learning model capable of mastering multiple common board games. I'm not too familiar with strategies in Star Craft, however, here's a link containing some other projects where strategies that emerge by the AI are shown: https://openai.com/blog/competitive-self-play/


text:
Is the error function known or unknown?
What is the error function? Is it the same as the cost function?
Is the error function known or unknown?
When I get the outcome of a neural net I compare it with the target value. The difference between both is called the error. When I get mutiple error values e.g. when I pass a batch through the NN I will get as many error value as the size of my batch. Is the error function the plot of the points? If yes, to me the error function would be unknown. I would only know some point on the graph of the error function.

In deep learning, the error function is sometimes known as loss function or cost function (but I do not exclude the possibility that these terms/expressions have also been used to refer to different although related functions, so you should take into account your context).
In statistical learning theory, the function that you want to minimize is known as expected risk
\begin{align}
R(\theta)
&=
\int L(y, f_{\theta}(x)) d P(x, y) \\
&=
\mathbb{E}_{P(x, y)} \left[ L(y, f_{\theta}(x)) \right],
\end{align}
where $\theta$ are the parameters of your model $f$, and the function that you actually minimize is the empirical risk given the dataset $D = \{ (x_i, y_i) \}_{i=1}^n$,
$$
R_{\mathrm{emp}}(\theta)=\frac{1}{n} \sum_{i=1}^{n}L(y_i, f_{\theta}(x_i)),
$$
which is a generalization of the commonly used cost functions, such as mean squared error (MSE) or binary cross-entropy. For example, in the case of MSE, $L = \left(y_{i}-f_\theta(x_i)\right)^{2}$, which can be called the loss function (though this terminology may not be standard).
You optimize the empirical risk, a proxy for the expected risk, because the expected risk is incomputable (given that $P(x, y)$ is generally unknown), so, in this sense, the expected risk is unknown.
If you use the term error function to refer to the expected risk, then, yes, the error function is typically unknown, but error function is typically used to refer to an instance of the empirical risk, so, in that case, it is known and computable.
Note that I purposely used the term loss function above to refer to $L$ and the term cost function to refer to the empirical risks, such as the MSE (i.e., in this case, I did not use loss function and cost function as synonyms), which shows that terminology is not always used consistently, so take into account your context.

I am not sure if this answers your question, given that I don't fully understand your doubts. If not, please, let me know.

I got a little confused when I looked up all the explanations on how the error is considered.
E.g. when the error function is mentioned sth like E=(y'-y)^2 is meant. and a parabola is shown. Sometimes however (maybe in another context) I see a sort of scatter plot that just shows how big an error is (for every entry maybe in chronological order as the batch is passed through the net). I am just asking what is what.

@MScott As I said, sometimes it depends on the context. $(y' - y)^2$ could be called the error function and the sum of all errors could be called the empirical risk or often in the context of deep learning also the error function (yes, error function again!).


text:
How is exponential moving average computed in deep Q networks?
In normal Q-learning, the update rule is an implementation of the exponential moving average, which then converges to the optimal true Q values. However, looking at DQN, how exactly is the exponential moving average implemented in deep networks?

However, looking at DQN, how exactly is the exponential moving average implemented in deep networks?

It is not implemented directly as exponential moving average.
Instead, the ability of neural networks to learn online and incrementally forget older input/output associations is used to achieve the same goal.
If you use the simplest mini-batch stochastic gradient descent methods - i.e. just a simple gradient step $\mathbf{w} \leftarrow \mathbf{w} - \alpha \nabla_{\mathbf{w}}\sum_i(g_i - \hat{q}_i)^2$ where $g_i$ is measured (or bootstrap estimated) discounted return for a single state/action pair and $\hat{q}_i$ is the current estimate, then the learning rate $\alpha$ is analagous to the same factor in exponential moving average approach, and in fact would be the same thing mathematically if you one-hot-encoded the states and only had a single layer in the neural network.
Typical implementations of DQN will have deeper networks, will not one-hot-encode the entire state space, and will typically use some gradient modifier such as momentum or Adam to improve performance. So the match to exponential moving average is not exact. But the behaviour is similar in the most important aspect for RL - the ability to learn online and forget older values as the target distribution of expected returns changes due to changes in policy.

I don’t quite understand this fully, but what you’re tying to say is, the exponential moving average and the update of the weights are essentially the same mathematically ?

Can you refer me to the mathematical proof of the SGD being the same as the exponential moving average

@ChukwudiOgbonna: I don't have a reference, but it is trivial because fully enumerated linear "approximation" is exactly the same as tabular, and the weights become the Q table. You won't find many references because it is not a very interesting result. No-one uses DQN in a way that the updates are directly equivalent. Just similar behaviour of online learning, as described in the last paragraph in this answer.

Thank you for your time, I appreciate , I don’t know if I should really bother my self with the maths aspect, what do you think

@ChukwudiOgbonna: If you want to understand RL well enough to alter or implement algorithms yourself, it is worth working on the maths. In this particular case you are looking at a minor issue though, so you could skip it with the broad understanding that all that matters is the online learning behaviour which exponential moving average and neural network training by gradient descent both produce. The numerical equivalence in a special case (which never happens in real problems) is not too important, and other forms of forgetting old inputs might be equally good.


text:
How does the Ornstein-Uhlenbeck process work, and how it is used in DDPG?
In section 3 of the paper Continuous control with deep reinforcement learning, the authors write

As detailed in the supplementary materials we used an Ornstein-Uhlenbeck process (Uhlenbeck & Ornstein, 1930) to generate temporally correlated exploration for exploration efficiency in physical control problems with inertia (similar use of autocorrelated noise was introduced in (Wawrzynski, 2015)).

In section 7, they write

For the exploration noise process we used temporally correlated noise in order to explore well in physical environments that have momentum. We used an Ornstein-Uhlenbeck process (Uhlenbeck & Ornstein, 1930) with θ = 0.15 and σ = 0.2. The Ornstein-Uhlenbeck process models the velocity of a Brownian particle with friction, which results in temporally correlated values centered around 0.

In a few words, what is the Ornstein-Uhlenbeck process? How does it work? How exactly is it used in DDPG?
I want to implement the Deep Deterministic Policy Gradient algorithm, and, in the initial actions, noise has to be added. However, I cannot understand how this Ornstein-Uhlenbeck process works. I have searched the internet, but I have not understood the information that I found.

It should be noted that more recent work suggests that uncorrelated Gaussian noise works just as well.
TD3 paper (https://arxiv.org/pdf/1802.09477.pdf): "Unlike the original DDPG, we used uncorrelated noise for exploration as we found noise drawn from the Ornstein-Uhlenbeck (Uhlenbeck & Ornstein, 1930) process offered no performance benefits."
D4PG paper (https://arxiv.org/pdf/1804.08617.pdf): "We experimented with correlated noise drawn from an Ornstein-Uhlenbeck process, as suggested by (Lillicrap et al., 2016), however we found this was unnecessary and did not add to performance."


text:
What does "convolve k filters" mean in the AlphaGo paper?
On page 27 of the DeepMind AlphaGo paper appears the following sentence:

The first hidden layer zero pads the input into a $23 \times 23$ image, then convolves $k$ filters of kernel size $5 \times 5$ with stride $1$ with the input image and applies a rectifier nonlinearity.

What does "convolves $k$ filters" mean here?
Does it mean the following:

The first hidden layer is a convolutional layer with $k$ groups of $(19 \times 19)$ neurons, where there is a kernel of $(5 \times 5 \times numChannels + 1)$ parameters (input weights plus a bias term) used by all the neurons of each group. $numChannels$ is 48 (the number of feature planes in the input image stack).
All $(19 \times 19 \times k)$ neurons' outputs are available to the second hidden layer (which happens to be another convolutional layer, but could in principle be fully connected).

?

I believe it means they perform convolution with $k$ filters in the given layer (this is if I'm not getting my terminology confused)

Regarding your first question, I agree with @DavidIreland. I am pretty sure that they convolve $k$ different filters with the same $23 \times 23$ image, where each of the filters has the same depth as the input images, provided you're performing a 2D convolution. The $k$ is a hyper-parameter there. I am not so sure about the second paragraph. I would need to think about it a little more.


text:
How to take the optimal batch_size for training a model?
I have an image dataset, which is composed of 113695 images for training and 28424 images for validation. Now, when I use ImageDataGenerator and flow_from_dataframe, it as the parameter batch_size.
How can I take the correct number for batch_size because both numbers cannot be divided by the same number? Should I need to drop four images in the validation data to make them batch_size of 5? Or is there another way?

Note that programming issues are off-topic here. I am not completely sure whether this is only a programming issue or not, so I will leave it open for now. Can you clarify if this is a programming issue or not? Please, read our on-topic page: https://ai.stackexchange.com/help/on-topic.

This Cross Validated post might answer your question.
In a nutshell:

A single batch (that is all your data in one batch) will result in a smooth trajectory on the loss surface. The drawback is that all your data might not fit into your memory. Which is highly likely for ~100k images.

One image per batch (batch size = no. examples) will result in a more stochastic trajectory since the gradients are calculated on a single example. Advantages are of computational nature and faster training time.

The middle way is to choose the batch size in such a way that your batch fits into memory and gradients behave less 'noisy'. To be honest there is no 'golden' number, personally I like to choose powers of two.
Don't worry that your data is not divisible by the batch size. Libraries will take care about that internally, the last batch will just be a smaller than the defined batch size ($N \text{ mod } b$).

From Andrew lesson on Coursera, batch_size should be the power of 2, ex: 512, 1024, 2048. It will faster for training.
And you don't need to drop your last images to batch_size of 5 for example. The library likes Tensorflow or Pytorch, the last batch_size will be number_training_images % 5 which 5 is your batch_size.
Last but not least, batch_size need to fit your memory training (CPU or GPU). You can try several large batch_size to know which value is not out of memory. The smaller number_mini_batch = number_training_image//batch_size + 1, the faster for training time.
Hope they can help you!


text:
Are neural networks really used apart from specific hi-tech organisations?
This is a generic question. Still posting it to get insights from experts in the field.
I am interested in knowing if Neural Networks are used in general apart from specific hi-tech organizations.
If so, which type of NN is used in which industry and for what purpose?

Hi. Can you clarify what you mean by "specific hi-tech orgs"? And what industry are you referring to when you are looking for applications of NNs in those industries?

Yes, I use NN on my home PC for image classification. No problem at all.

@nbro Thanks for your reply. ‘Specific hi-tech’ companies refer to handful of companies such as Google et al who are involved in developing NN tech. What I am interested in knowing is if NN tech is used by companies just as any company uses IT. IS NN or even ML in general use widespread?

Before putting up my experience I'd like to provide some facts about AI startups and established Hi-Tech companies.

Most of companies claiming to use AI doesn't actually use ML/DL
And there are products by some companies that are classified to be using AI since they just use linear regression

Coming to my experience I can say that AI is passively as well as actively utilized in product development and services. Though Neural Networks are the driving force behind the current AI tech, the thing that we study in the text books are just basic building blocks that are used to designing complex network architectures. Though most of High-Tech companies as you have mentioned might not be willing to share their secrets, the publications they make in conferences are good enough. I would suggest one to go through the papers that are published by companies like NVIDIA, Microsoft, and Google to get understanding on how complex networks are designed to address specific problems.


text:
What is eager learning and lazy learning?
What is the difference between eager learning and lazy learning?
How does eager learning or lazy learning help me build a neural network system? And how can I use it for any target function?

What is eager learning or lazy learning?

Eager learning is when a model does all its computation before needing to make a prediction for unseen data. For example, Neural Networks are eager models.
Lazy learning is when a model doesn't require any training, but all of its computation during inference. An example of such a model is k-NN. Lazy learning is also known as instance-based learning [1, 2, 3].

How does eager and lazy learning help me build a neural network system? And how can I use it for any target function?

To answer your second question, you can't employ lazy learning to train a neural network, because they are inherently eager models.

Maybe you should cite some book or paper that discusses this terminology because, although what you write makes sense, I am not completely sure this is standard terminology.  Maybe it's just been a while since I had to deal with this terminology and that's why I am not completely convinced.

@nbro I added some links, hope it's OK.

Well, actually, you added some links to other non "reputable" sources, so I edited your post only to leave two links to papers that seem somehow reliable. I think that's enough, unless there's a better source.

@nbro I think the terms are sufficiently well-established. I do recall having seen them in my education now that I think of it :) Maybe not as much of a commonly-discussed topic anymore in recent years with many people behaving as if (deep) neural networks are the only types of machine learning models that exist...

@DennisSoemers Well, they are somehow "established" but I don't think they are widely used. k-NNs is usually described as a "lazy learner", but neural networks and other methods aren't usually described as "eager learning methods", unless you compare them with k-NN and describe when they learn. In this sense, I don't think these terms are widely used. So, I think that some reliable source is important to show that these terms have been used in the literature.

@nbro I think that the default notion of learning has shifted towards eager learning. This way NNs aren't described as eager learners because that's the default. In contrast lazy learners are described as such to differentiate with the other more popular models. TBH, I don't agree with the terminology, because no actual learning happens in lazy models (no knowledge is extracted from the data), but since they are considered ML models, they need a way of distinguishing themselves...


text:
What's the difference between domain randomization and domain adaptation?
In my understanding, domain randomization is one method of diversifying the dataset to achieve a better shot at domain adaptation. Am I wrong?

you are kind of right.
but no necessarily.
domain randomization : when you widen the range of your training data parameters to make your model more generalized.
this can be done for any purpose. (even if you are not doing domain adaptation).
domain adaptation : when you train your model on data from a certain domain (lets say to detect cars) and then test it on data from another domain (lets say for detecting trucks).

the data you used earlier helped "pre-train" your model.
you may "fine-tune" this model with whatever small amounts of data you have available from the test domain (trucks in our case).
you may also use the concepts of domain randomization (make sure you include different varieties of car images) during your training.

these things are optional, and do not always HAVE to go together.
Hope this helps!


text:
Do we assume the policy to be deterministic when proving the optimality?
In reinforcement learning, when we talk about the principle of optimality, do we assume the policy to be deterministic?

Hi. Can you clarify what your question is? Is it "when we talk about the principle of optimality, do we assume policy to be deterministic?"

Yes, that's what I meant

Which "principle of optimality" are you talking about exactly? Where did you read this term? Maybe provide a reference to the book? Are you talking about the optimality of the value iteration and/or policy iteration? I heard the term "principle of optimality" only in the context of dynamic programming algorithms.

Yes, I'm reading Sutton-Barto, Introduction to Reinforcement learning. I'm asking in the context of DP only.


text:
Why is domain adaptation and generative modelling for knowledge graphs still not applied widely in enterprise data? What are the challenges?
I see that domain adaptation and transfer learning has been widely adopted in image classification and semantic segmentation analysis. But it's still lacking in providing solutions to enterprise data, for example, solving problems related to business processes?
I want to know what characteristics of the data determine the applicability or non-applicability with respect to generating models for prediction where multiple domains are involved within an enterprise information database?


text:
Given the same features, do logistic regression and neural networks produce the same output?
I have a binary classification problem. I have variables (features) var1, var2, var3, ..., var14.
Using these variables (aka features) in a logistic regression, I get their weights.
If I use the same set of variables in a neural network:

Should I get a different output?
or

Should I get the same output?

I developed a ROC Curve, and I have both lines overlaying on each other. I am not sure if I am missing something here.

It's difficult to answer this question if you don't provide both the plots of the ROC Curve, more details about your neural network (such as the output layer, which active function you are using, which problem you're trying to solve, size of dataset, etc.)


text:
Why would DDPG with Hindsight Experience Replay not converge?
I am trying to train a DDPG agent augmented with Hindsight Experience Replay (HER) to solve the KukaGymEnv environment.  The actor and critic are simple neural networks with two hidden layers (as in the HER paper).
More precisely, the hyper-parameters I am using are

The actor's hidden layer's sizes: [256, 128] (using ReLU activations and a tanh activation after the last layer)
Critic's hidden layer's sizes: [256, 128] (Using ReLU activations)
Maximum Replay buffer size: 50000
Actor learning rate: 0.000005 (Adam Optimizer)
Critic learning rate: 0.00005 (Adam Optimizer)
Discount rate: 0.99
Polyak constant : 0.001
The transitions are sampled in batches of 32 from the replay buffer for training
Update rate: 1 (target networks are updated after each time step)
The action selection is stochastic with the noise being sampled from a normal distribution of mean 0 and standard deviation of 0.7

I trained the agent for 25 episodes with a maximum of 700 time-steps each and got the following reward plot:

The reward shoots up to a very high value of about 8000 for the second episode and steeply falls to -2000 in the very next time step, never to rise again. What could be the reason for this behavior and how can I get it to converge?
PS : One difference I observed while training this agent from while training a simple DDPG agent is that for simple DDPG, the episode would usually terminate at around 450 time-steps, thus never reaching the maximum specified timesteps. However, here, no episode terminated before the specified 700 maximum time steps. This might have something to do with the performance.

Did you fix this issue?


text:
What is the weight matrix in self-attention?
I've been looking into self-attention lately, and in the articles that I've been seeing, they all talk about "weights" in attention. My understanding is that the weights in self-attention are not the same as the weights in a neural network.
From this article, http://peterbloem.nl/blog/transformers, in the additional tricks section, it mentions,
The query is the dot product of the query weight matrix and the word vector,
ie, q = W(q)x and the key is the dot product of the key weight matrix and the word vector, k = W(k)x and similarly for the value it is v = W(v)x. So my question is, where do the weight matrices come from?

Hi. I just wanted to let you know that you can use latex on this site.

The answer is actually really simple: they are all randomly initialised. So they are to all intents and purposes "normal" weights of a neural network.
This is also the reason why in the original paper the authors tested several setting with single and multiple attention heads. If these matrices were somehow "special" or predetermined they would all serve the same purpose. Instead, because of their random initialisation, each attention head learn to contribute to solve a different task, like they show in Figure 3 and 4.

So how are those weight matrices updated? Is there a separate neural network for that?

Why do you need another neural network? You just back propagate everything till these very first matrices. You can see from the tensorflow documentation that the matrices are just dense layers (or conv1D layers) https://github.com/tensorflow/tensor2tensor/blob/d9f807cf2738323d19aba0a20a8cf0c7f7da8b27/tensor2tensor/layers/common_attention.py#L2193

Others said wights matrices(Wq Wk Wv) are trained by the network while still someone like you said it's randomly initialized. It make sense if u said wights matrices was randomly initialized on the very beginning of training stage, But I dont think there is any initialization of them on inference stage, am I correct?

In my mind there are two weight matrices, the one you get prior to applying softmax:
$$ \alpha_{i,j} = \frac{\langle q_i, k_j \rangle}{\sqrt{d}}$$
the other you get after applying the softmax:
$$ \text{Attn}_{i,j}(X) = \frac{\exp \left( \frac{\langle q_i, k_j \rangle}{\sqrt{d}}\right)}{\sum_k \exp\left( \frac{\langle q_i, k_k \rangle}{\sqrt{d}} \right)}.$$
Either way, you can view them as directed "attention graphs" and they can be interpreted in terms of graph neural networks using only complete graphs and graph attention:
$$ A_{i,j} = \frac{\exp(w(i,j))}{\sum_{j \in \mathcal{N}(i)} \exp(w(i,k))},$$
where $w(i,j)$ is a weight function, which can be the scaled dot-product of attention for example. The sum is over the neighborhood of a node (think token) $\mathcal{N}(i)$. Hope that helps a little with intuition and that it's close to what you meant.


text:
How do non-local neural networks relate to attention and self-attention?
I've been reading non-local neural networks as explained in the original paper. My understanding is that they solve the restrained reception of local filters. I see how they are different from convolutions and fully connected networks.
How do they relate to attention (specifically self-attention)? How do they integrate this attention?


text:
How to understand this NN architecture?
I was reading a paper Multi-Agent Reinforcement Learning for Adaptive
User Association in Dynamic mmWave Networks and I was stuck understanding the deep neural network architecture that was used. The authors gave it in Fig. 3 (on top of page 6) and they state the following (on page 9):

This architecture comprises 2 multi-layers perceptron (MLP) of 32 hidden units, one RNN layer (a long short memory term - LSTM) layer with 64 memory cells followed by another 2 MLPs of 32 hidden units. The network then branches off in two MLPs of 16 hidden units to construct the duelling network.

According to Fig. 3 there is one MLP, one RNN and one MLP. So why the authors said 2 MLPs?
Assuming it is 2 MLPs, does this mean we have 2 hidden layers of 32 neurons each? So, at the end we will have:
one input layer - one hidden layer with 32 neurons - another hidden layer with 32 neurons - one RNN layer with 64 cells - one hidden layer with 32 neurons - another hidden layer with 32 neurons - one hidden layer with 16 neurons - another hidden layer with 16 neurons - one output layer.


text:
How is the data labelled in order to train a region proposal network?
I don't get how the training of the RPN works. From the forward propagation, I have $W \times H \times k$ outputs from the RPN.
How is the training data labeled such that I can use the loss function and update the weights through bach propagation?  Is the training data labeled in the same shape of the output, as there are $W \times H \times k$ anchor boxes and we use the loss function directly or what?

Did u find any answer?I am interested too


text:
How to calculate the GPU memory need to run a deep learning network?
In general, how do I calculate the GPU memory need to run a deep learning network?
I'm asking this question because my training for some network configuration is getting out of memory.
If the TensorFlow only store the memory necessary to the tunable parameters, and if I have around 8 million, I supposed the RAM required will be:
RAM = 8.000.000 * (8 (float64)) / 1.000.000 (scaling to MB)
RAM = 64 MB, right?
The TensorFlow requires more memory to store the image at each layer?
By the way, these are my GPU Specifications:

Nvidia GeForce 1050 4GB

Networking topology

Unet
Input Shape (256,256,4)

Model: "functional_1"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to
==================================================================================================
input_1 (InputLayer)            [(None, 256, 256, 4) 0
__________________________________________________________________________________________________
conv2d (Conv2D)                 (None, 256, 256, 64) 2368        input_1[0][0]
__________________________________________________________________________________________________
dropout (Dropout)               (None, 256, 256, 64) 0           conv2d[0][0]
__________________________________________________________________________________________________
conv2d_1 (Conv2D)               (None, 256, 256, 64) 36928       dropout[0][0]
__________________________________________________________________________________________________
max_pooling2d (MaxPooling2D)    (None, 128, 128, 64) 0           conv2d_1[0][0]
__________________________________________________________________________________________________
conv2d_2 (Conv2D)               (None, 128, 128, 128 73856       max_pooling2d[0][0]
__________________________________________________________________________________________________
dropout_1 (Dropout)             (None, 128, 128, 128 0           conv2d_2[0][0]
__________________________________________________________________________________________________
conv2d_3 (Conv2D)               (None, 128, 128, 128 147584      dropout_1[0][0]
__________________________________________________________________________________________________
max_pooling2d_1 (MaxPooling2D)  (None, 64, 64, 128)  0           conv2d_3[0][0]
__________________________________________________________________________________________________
conv2d_4 (Conv2D)               (None, 64, 64, 256)  295168      max_pooling2d_1[0][0]
__________________________________________________________________________________________________
dropout_2 (Dropout)             (None, 64, 64, 256)  0           conv2d_4[0][0]
__________________________________________________________________________________________________
conv2d_5 (Conv2D)               (None, 64, 64, 256)  590080      dropout_2[0][0]
__________________________________________________________________________________________________
max_pooling2d_2 (MaxPooling2D)  (None, 32, 32, 256)  0           conv2d_5[0][0]
__________________________________________________________________________________________________
conv2d_6 (Conv2D)               (None, 32, 32, 512)  1180160     max_pooling2d_2[0][0]
__________________________________________________________________________________________________
dropout_3 (Dropout)             (None, 32, 32, 512)  0           conv2d_6[0][0]
__________________________________________________________________________________________________
conv2d_7 (Conv2D)               (None, 32, 32, 512)  2359808     dropout_3[0][0]
__________________________________________________________________________________________________
conv2d_transpose (Conv2DTranspo (None, 64, 64, 256)  524544      conv2d_7[0][0]
__________________________________________________________________________________________________
concatenate (Concatenate)       (None, 64, 64, 512)  0           conv2d_transpose[0][0]
                                                                 conv2d_5[0][0]
__________________________________________________________________________________________________
conv2d_8 (Conv2D)               (None, 64, 64, 256)  1179904     concatenate[0][0]
__________________________________________________________________________________________________
dropout_4 (Dropout)             (None, 64, 64, 256)  0           conv2d_8[0][0]
__________________________________________________________________________________________________
conv2d_9 (Conv2D)               (None, 64, 64, 256)  590080      dropout_4[0][0]
__________________________________________________________________________________________________
conv2d_transpose_1 (Conv2DTrans (None, 128, 128, 128 131200      conv2d_9[0][0]
__________________________________________________________________________________________________
concatenate_1 (Concatenate)     (None, 128, 128, 256 0           conv2d_transpose_1[0][0]
                                                                 conv2d_3[0][0]
__________________________________________________________________________________________________
conv2d_10 (Conv2D)              (None, 128, 128, 128 295040      concatenate_1[0][0]
__________________________________________________________________________________________________
dropout_5 (Dropout)             (None, 128, 128, 128 0           conv2d_10[0][0]
__________________________________________________________________________________________________
conv2d_11 (Conv2D)              (None, 128, 128, 128 147584      dropout_5[0][0]
__________________________________________________________________________________________________
conv2d_transpose_2 (Conv2DTrans (None, 256, 256, 64) 32832       conv2d_11[0][0]
__________________________________________________________________________________________________
concatenate_2 (Concatenate)     (None, 256, 256, 128 0           conv2d_transpose_2[0][0]
                                                                 conv2d_1[0][0]
__________________________________________________________________________________________________
conv2d_12 (Conv2D)              (None, 256, 256, 64) 73792       concatenate_2[0][0]
__________________________________________________________________________________________________
dropout_6 (Dropout)             (None, 256, 256, 64) 0           conv2d_12[0][0]
__________________________________________________________________________________________________
conv2d_13 (Conv2D)              (None, 256, 256, 64) 36928       dropout_6[0][0]
__________________________________________________________________________________________________
conv2d_14 (Conv2D)              (None, 256, 256, 1)  65          conv2d_13[0][0]
==================================================================================================
Total params: 7,697,921
Trainable params: 7,697,921
Non-trainable params: 0

This is the error given.
---------------------------------------------------------------------------
ResourceExhaustedError                    Traceback (most recent call last)
<ipython-input-17-d4852b86b8c1> in <module>
     23 # Train the model, doing validation at the end of each epoch.
     24 epochs = 30
---> 25 result_model = model.fit(train_gen, epochs=epochs, validation_data=val_gen, callbacks=callbacks)

~\Anaconda3\envs\tf23\lib\site-packages\tensorflow\python\keras\engine\training.py in _method_wrapper(self, *args, **kwargs)
    106   def _method_wrapper(self, *args, **kwargs):
    107     if not self._in_multi_worker_mode():  # pylint: disable=protected-access
--> 108       return method(self, *args, **kwargs)
    109
    110     # Running inside `run_distribute_coordinator` already.

~\Anaconda3\envs\tf23\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
   1096                 batch_size=batch_size):
   1097               callbacks.on_train_batch_begin(step)
-> 1098               tmp_logs = train_function(iterator)
   1099               if data_handler.should_sync:
   1100                 context.async_wait()

~\Anaconda3\envs\tf23\lib\site-packages\tensorflow\python\eager\def_function.py in __call__(self, *args, **kwds)
    778       else:
    779         compiler = "nonXla"
--> 780         result = self._call(*args, **kwds)
    781
    782       new_tracing_count = self._get_tracing_count()

~\Anaconda3\envs\tf23\lib\site-packages\tensorflow\python\eager\def_function.py in _call(self, *args, **kwds)
    838         # Lifting succeeded, so variables are initialized and we can run the
    839         # stateless function.
--> 840         return self._stateless_fn(*args, **kwds)
    841     else:
    842       canon_args, canon_kwds = \

~\Anaconda3\envs\tf23\lib\site-packages\tensorflow\python\eager\function.py in __call__(self, *args, **kwargs)
   2827     with self._lock:
   2828       graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
-> 2829     return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
   2830
   2831   @property

~\Anaconda3\envs\tf23\lib\site-packages\tensorflow\python\eager\function.py in _filtered_call(self, args, kwargs, cancellation_manager)
   1846                            resource_variable_ops.BaseResourceVariable))],
   1847         captured_inputs=self.captured_inputs,
-> 1848         cancellation_manager=cancellation_manager)
   1849
   1850   def _call_flat(self, args, captured_inputs, cancellation_manager=None):

~\Anaconda3\envs\tf23\lib\site-packages\tensorflow\python\eager\function.py in _call_flat(self, args, captured_inputs, cancellation_manager)
   1922       # No tape is watching; skip to running the function.
   1923       return self._build_call_outputs(self._inference_function.call(
-> 1924           ctx, args, cancellation_manager=cancellation_manager))
   1925     forward_backward = self._select_forward_and_backward_functions(
   1926         args,

~\Anaconda3\envs\tf23\lib\site-packages\tensorflow\python\eager\function.py in call(self, ctx, args, cancellation_manager)
    548               inputs=args,
    549               attrs=attrs,
--> 550               ctx=ctx)
    551         else:
    552           outputs = execute.execute_with_cancellation(

~\Anaconda3\envs\tf23\lib\site-packages\tensorflow\python\eager\execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     58     ctx.ensure_initialized()
     59     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
---> 60                                         inputs, attrs, num_outputs)
     61   except core._NotOkStatusException as e:
     62     if name is not None:

ResourceExhaustedError:  OOM when allocating tensor with shape[8,64,256,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
     [[node gradient_tape/functional_1/conv2d_14/Conv2D/Conv2DBackpropInput (defined at <ipython-input-17-d4852b86b8c1>:25) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
 [Op:__inference_train_function_17207]

Function call stack:
train_function

Is there any type of mistake in the network definition? How could I improve the network to solve this problem?

Hi and welcome to AI SE :) I am sorry, but programming issues are off-topic, although this programming issue is related to AI. Please, have a look at our on-topic page https://ai.stackexchange.com/help/on-topic to know more about our scope. Stack Overflow  is the most appropriate SE site to ask questions about programming issues. This was also posted here: https://stackoverflow.com/q/63635126/3924118.

You should check your batchsize, and try reducing it. It's not only about storing the parameters, it's also about storing the tensor. For example, at your second layer, the tensor is 256 * 256 * 64 * 8 * batch_size, and if batch_size = 128, then it requires 4.2 Gb.

16Aghnar, the batch size I have reduced to 8 and I still get OOM.

In fact, I do not know how to calculate GPU memory to run a neural network but I have a solution for allocation problems in GPUs while using tensorflow framework.
import tensorflow as tf
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
    # Restrict TensorFlow to only allocate 2GB * 2 of memory on the first GPU
    try:
        tf.config.experimental.set_virtual_device_configuration(
            gpus[0],
            [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=2048 * 2)])
        logical_gpus = tf.config.experimental.list_logical_devices('GPU')
        print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
    except RuntimeError as e:
        # Virtual devices must be set before GPUs have been initialized
        print(e)

You can set a memory limit on GPU which sometimes solves memory allocation problems. As shown above, you can set "memory_limit" parameter as your configuration requires.
Also be careful about using correct framework. If you want to use above code to set memory, you have to build your neural network from tensorflow with keras backend.
from tensorflow.python.keras.models import Sequential

You can calculate the memory requirement analytically, but it's still not going to beat physical test in practice as there are so many unknown variables in the system which can takes the GPU memory. Maybe tensorflow will decide to store the gradients, then you have to take into account the memory usage of it also.
The way I do it is by setting the GPU memory limit to a high value e.g. 1GB, then test the model inference speed. Then I repeat the process with half the memory. I do it until the model refuses to run or the model speed drops. For example, I start with 1GB, then 512MB, then 256MB, eventually I got to 32 MB and the model speed drops. At 16MB, the model refuses to run. So I know that 64 MB is the minimum requirement I should use for my model. If I want to get a more precise number, I'd repeat the binary search process a couple more time between 64 MB and 32 MB.
You can see how to limit the GPU memory here:  https://www.tensorflow.org/guide/gpu#limiting_gpu_memory_growth
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
  # Restrict TensorFlow to only allocate 1GB of memory on the first GPU
  try:
    tf.config.experimental.set_virtual_device_configuration(
        gpus[0],
        [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=1024)])
    logical_gpus = tf.config.experimental.list_logical_devices('GPU')
    print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
  except RuntimeError as e:
    # Virtual devices must be set before GPUs have been initialized
    print(e)

TensorFlow is interesting that it can store not only weights, but also training data in video RAM.
with tf.device('/gpu:0'):
    tensorflow_dataset = tf.constant(numpy_dataset)

Feeding training data and weights to GPU for matrix mul is faster than from regular RAM.
Video RAM required = Number of params * sizeof(weight type) +
                     Training data amount in bytes

However, I believe that video RAM required should be at least 1.5 times the above value just to be sure things would be working.


text:
How are we calculating the average reward ($r(\pi)$) if the policy changes over time?
In the average reward setting, the quality of a policy is defined as:
$$ r(\pi) = \lim_{h\to\infty}\frac{1}{h} \sum_{j=1}^{h}E[R_j] $$
When we reach the steady state distribution, we can write the above equation as follows:
$$ r(\pi) = \lim_{t\to\infty}E[R_t | A \sim \pi] $$
We can use the incremental update method to find $r(\pi)$:
$$ r(\pi) = \frac{1}{t} \sum_{j=1}^{t} R_j = \bar R_{t-1} + \beta (R_t - \bar R_{t-1})$$
where $ \bar R_{t-1}$ is the estimate of the average reward $r(\pi)$ at time step $t-1$.
We use this incremental update rule in the SARSA algorithm:

Now, in this above algorithm, we can see that the policy will change with respect to time. But to calculate the $r(\pi)$, the agent should follow the policy $\pi$ for a long period of time. Then how we are using $r(\pi)$ if the policy changes with respect to time?

You are correct: to evaluate a policy, we need to fix it.

We can temporarily fix it, just to evaluate it over a number of test cases. For a fair comparison, we should fix the start states and random seeds used for the transitions.
We can wait until convergence / until we are satisfied. The resulting policy would be what we implement in the "true", trained agent. This is important when exploration might be harmful in the "real world" domain where the agent will be operating.
We can also measure average reward of the "non-stationary" policy, and assume that, once the agent is doing well, this should be close enough to evaluating the fixed policy. This is not ideal, but on the other hand it is trivial to implement, and is often used to track the learning process. If you have a life-long learning agent, this might be the best you can do.

if we wait until convergence to calculate $r(\pi)$, then how are we using $r(\pi)$ to find optimal policy.

When finding the optimal policy, we're not using $r(\pi)$, but we might be using $Q(s, a, \pi)$ when we're using policy iteration. In policy iteration we fix the policy until the values have converged, and then improve the policy.
When we use other methods where the policy changes, we use the $Q(s, a)$ values that do depend on the exploration policy, but eventually converge; that's what the Bellman equations tell us. Of course, this assumes that we're using a table-based approach. When using function approximation, we have no guarantees of convergence.

you said that, "when finding the optimal policy we are not using $r(\pi)$, but in the above SARSA algorithm we are compensating $w$ using $r(\pi)$ also that changes our policy per iteration. I think what you are trying to say is that use it like policy iteration algorithm. But in the above algorithm I can't see update like this.

The SARSA algorithm does not mention $\pi$ anywhere. It is not evaluating a fixed policty. It does not try to learn $Q(s, a, \pi)$. It will learn $Q(s, a)$ (the actual $Q$-values). This particular algorithm does so by trying to steer the weights of the policy in such a way that the average reward is maximized (that's where the gradient comes in).

Yes, it will try to maximize the average reward. But my question was, how are we calculating the average reward while changing the policy over every step (here policy is $\epsilon$-greedy).

The algorithm itself does not calculate the average reward explicitly -- it uses the Q-values. Maximizing the Q value (by tweaking your policy parameters for a given state) will implicitly improve the total average reward of the policy, as well.


text:
Why is the target called "target" in Monte Carlo and TD learning if it is not the true target?
I was going through Sutton's book and, using sample-based learning for estimating the expectations, we have this formula
$$
\text{new estimate} = \text{old estimate} + \alpha(\text{target} - \text{old estimate})
$$
What I don't quite understand is why it's called the target, because since it's the sample, it’s not the actual target value, so why are we moving towards a wrong value?

It is our "current" target. We assume that the value we get now is at least a closer approximation to the "true" target.
We're not so much moving towards a wrong value as we are moving away from a more wrong value.
Of course, it is all base on random trials, so saying anything definite (such as: "we are guaranteed to improve at each step") is hard to show without working probabilistically. The expectation of the error of the value function (as compared to the true value function) will decrease, that is all we can say.

Maybe to clarify your answer, you could add the formula of the "target" and why it represents an estimate of the "true" target. Anyway, welcome to Artificial Intelligence SE :)

This isn’t clear to me, why is it closer to the true target?

I understand that we use a sample return as the target, but what I don’t get is how it moves closer to the true target, the sample return doesn’t get closer to the target, the mean of those samples is what gets Closer which is our value function, so why do we then call the it target

The new value is equal to $(1 - \alpha)(\mathrm{old estimate})  + \alpha(\mathrm{new estimate})$. We're gradually "forgetting" the old estimate with a decay rate of $(1 - \alpha)$.

It would be heplful for me if you specify the section and page number of the Sutton's book.
But as far as I understand your question I will try explain this.
Think of TD update. The sample contains $(s_t,a_t,r_{t+1},s_{t+1})$. Using incremental update we can write:
$$ v_{t}(s) = \frac{1}{t} \sum_{j=1}^{t}(r_{t+1} + \gamma v_{s_{t+1}})$$
$$ v_{t}(s) = v_{t-1}(s) + \alpha (r_{t+1} + \gamma v_{t-1}(s_{t+1}) - v_{t-1}(s_t))$$
We are calling this $r_{t+1} + \gamma v_{t-1}(s_{t+1})$ as the TD target. From the above equation you can already see that $r_{t+1} + \gamma v_{t-1}(s_{t+1})$ is actually the unbaised estimate for $v(s)$. We are calling $r_{t+1} + \gamma v_{t-1}(s_{t+1})$ an unbiased estimate since $E[r_{t+1} + \gamma v_{t-1}(s_{t+1})] = v_t(s_t)$. That means expectation over $r_{t+1} + \gamma v_{t-1}(s_{t+1})$ lead us to true state value function, $v_t(s)$.
For the monte carlo update same explain will be applied. I hope that this answer your question.

I understand the unbiased estimate part, which means by law of large numbers it’ll eventually converge to the optimal value, what I don’t get is why it’s called the target , normally in supervised learning it’s the actual target value, not a sample return , we’re using a sample return as our target TD, which I do not understand

In RL targets are not fixed like supervised learning. Here we have moving targets. It is called target in the sense that it is an unbiased estimate of the true value.

Ok, so why do people say the target will eventually converge, isn’t it the mean of the target that actually converges to the optimal q value, I mean that’s what unbiased estimate means

People say this in the sense that initially what you get $r_{t+1} + \gamma v(s_{t+1})$ is not actually an unbiased estimate. We randomly initally all $v(s) = 0$. Then gradually they started to update. Initially, those values do not represent the unbiased estimate. But as time passes those estimate get close to the true unbiased estimate and converges to the unbiased estimate. For this reason they say target will eventually converge to the ubiased estimate.

Chill I think I get what you mean, we use the  estimate r + yV(s’), the unbiased estimate is the formula with the optimal V(s) ie R +yV*(s’), but since we use an estimate instead of the actual value ,so as our V(s) estimate converges to the V*(s) we can then say the target now becomes the unbiased estimate of our true value function ?

For TD learning or monte carlo learning we are not finding optimal $v(s)$. We are just evaluating the policy, $\pi$,

What I mean is the V(s) we initially start with is wrong, so it’s not the unbiased estimate, but as our V(s) converges to the true value, the target now converges to the unbiased estimate

When the target  converges after that we can say the target is an unbiased estimate of the true value

When the target converges to the true value, then you can get the true $v(s)$ for each state.

That makes sense, because our target is dependent on a value function that is initially wrong, it isn’t really the unbiased estimate, but as time goes on and we interact more, our Value function becomes more accurate , and so out TD target converges to the unbiased estimate

Yes,  you are right.

Finally, thank you so much, I appreciate your time and effort


text:
What are some other real-life examples of simple policies but complex value functions?
Hado van Hasselt, a researcher at DeepMind, mentioned in one of his videos (from 7:20 to 8:20) on Youtube (about policy gradient methods) that there are cases when the policy is very simple compared to the value function - and it makes more sense to learn the policy directly rather than first learning the value function and then doing control. He gives a very simple example at minute 7:20.
What are some other real-life examples (even just one example) of simple policies but complex value functions?
By real-life example I mean an example that is not as simple as a robot in a grid world, but some relatively complex real-world situations (say, autonomous driving).


text:
Why does Simulated Annealing not take worse solution if the energy difference becomes higher?
In Simulated Annealing, a worse solution is accepted with this probability:
$$p=e^{-\frac{E(y)-E(x)}{kT}}.$$
If that understanding is correct: Why is this probability function used? This means that, the bigger the energy difference, the smaller the probability of accepting the new solution. I would say the bigger the difference the more we want to escape a local minimum. I plotted that function in Matlab in two dimensions:

If any of the answers below answers your question, you should accept one of them. In my view, both answer your question.

Note that you can't really predict whether your escape from a local minimum will work or not - you might just wind up in another, worse local minimum. The probability function you describe increases the likelihood of this happening. By upweighting the likelihood of allowing small energy differences, you allow for the possibility of escaping local minima, while ensuring that whatever new minimum you find can't be that much worse that where you started. If you make the acceptance of large energy differences more likely, you can escape local minima more often, but you increase the likelihood that you'll just wind up in a region with an even higher local minimum.

Nice question!
My guess is that, if the probability of acceptance increases the bigger the difference between the current and new solutions is, then there's the risk that you need to search a lot again to find a good solution, i.e. you may oscillate between different subspaces or you could actually often end up in subspaces where there are only bad solutions. Your reasoning probably makes sense at the beginning of the search when the initial solutions may not be good enough or maybe if you have parallel searches (and you want to explore the search space at different search subspaces), but once you have a good solution, you don't want to completely discard it and replace it with a quite worse solution.
If you perform some experiments with your idea, I would like to see the results.

authors: عبدالله الشرماني


text:
How are the "Link Strength true", "Link Strength blind" and "Mutual Information" calculated in this report on Bayesian networks?
I'm trying to understand how to calculate the strength of every arc in a Bayesian Network.
I came across this report Measuring Connection Strengths and Link Strengths in Discrete Bayesian Networks, but I got lost in the calculation.
In particular, how are the values of Link Strength true, Link Strength blind, and Mutual Information computed in Table 1?

The $MI$ is easy enough.
For $MI(X \to Z)$, we get $U([0.5, 0.5]) - 0.5 U([0.9, 0.1]) - 0.5 U([0.1, 0.9])$. The $[0.5, 0.5]$ comes from $0.5 [0.9, 0.1] + 0.5[0.1, 0.9]$ -- this is how to calculate $P(Z)$ from $P(Z|X)$ and $P(X)$.
For $MI(X \to Y)$, we need to marginalize out $Z$. This takes a bit of work, but is not too hard. I manually verified all of the reported values in the figure, and they are all correct.
The $\mathrm{LS}{\mathrm{true}}$ values are a bit trickier, but still feasible. I managed to get the same values. It is important when calculating $\mathrm{LS}{\mathrm{true}}(X \to Z)$ that $Y$ is not a parent, so it does not come into it -- which is why the MI gives exactly the same value.
The "blind" version took me a bit more effort to implement, but again, I am getting the same values as reported in the table.
I used the "simple formula" at the top of page 5.

Thank you very much @Robby, it helps me understand some part. I don't have a strong mathematical background, which makes me struggle to translate the formulas and numbers in the paper. I'm doing a python code for this formula, and still figure out which variables need to hold which numbers. So please apologize.
When you mentioned $MI(X \to Z)$ = $U([0.5,0.5])−0.5U([0.9,0.1])−0.5U([0.1,0.9])$, it does use formula (2) $U(Z)-U(Z|X)$, right?
There is also a formula
$\sum\limits_{x,y}P(x,y) log_{2} (\frac{P(x,y)}{P(x)P(y)})$, and I'm not sure how to derive numbers for $P(x,y), P(x), P(y)$.

Yes, I used formula (2) for that.
$P(x,y) = P(y | x) \cdot P(x)$. You might want to take an introduction to probability theory, make sure it covers things up to conditional probabilities, entropy and Bayes' theorem; you'll encounter those a lot.

Thanks @Robby,
I will look at those materials asap. I have quickly tried to find the numbers for: $P(z,x)=0.5; P(x)=0,5; P(z)=0.5$.
I got it from $P(z,x)$ = the average of $P(z|x=true)$ and $P(z|x=false)$, which yield 0.5. $P(x) = P(x=true) = 0.5$. $P(z=true)$ = the average of 0.9 and 0.1 = 0.5. But the result is different from the table when I plug in those number in the formula.
I believe I calculated those wrongly, but it will help me to also understand the materials that you suggested if I know where my mistakes are. Thanks again Robby and I will update my understanding.

If $P(z|x) = 0.5$ and $P(x) = 0.5$, then $P(x, z) = 0.5 \cdot 0.5 = 0.25$.

The calculation you did ($P(z=\mathrm{True}|x=\mathrm{True}) P(x=\mathrm{True}) + P(z =\mathrm{True}| x=\mathrm{False})$ gives us $P(z=\mathrm{True})$, not $P(z, x)$.

Hey @Robby,
I have done the calculation for $(X \to Y)$ and $(Z \to Y)$.
$(X \to Y) = U([0.4982, 0.5017]) - 0.5([0.41, 0.09]) - 0.5([0.094, 0.406])$ = 0.1556174
$(Z \to Y) = U([0.4982, 0.5017]) - 0.5([0.4495, 0.0505]) - 0.5([0.0545, 0.4455])$ = 0.2576914
As you see, my results are half of the correct results and I'm wondering which my mistakes are.
Some details of my calculations are:
$U(Y)= 0.9999538$
$U(Y|X)= 0.844336$
$U(Y|Z)= 0.7422624$

The Joint distribution:
$P(Y=t | X=t,Z=t) * P(Z=t | X=t) * P(X=t) =0.405$
$P(Y=t | X=f,Z=t) * P(Z=t | X=f) * P(X=f) =0.0445$
$P(Y=t | X=t,Z=f) * P(Z=f | X=t) * P(X=t) =0.005$
$P(Y=t | X=f,Z=f) * P(Z=f | X=f) * P(X=f) =0.0495$
$P(Y=f | X=t,Z=t) * P(Z=t | X=t) * P(X=t) =0.045$
$P(Y=f | X=f,Z=t) * P(Z=t | X=f) * P(X=f) =0.0055$
$P(Y=f | X=t,Z=f) * P(Z=f | X=t) * P(X=t) =0.045$
$P(Y=f | X=f,Z=f) * P(Z=f | X=f) * P(X=f) =0.4005$
The Marginal:
$P(Z=t) = 0.5$ and $P(Z=f) = 0.5$
$P(Y=t) = 0.504$ and $P(Y=f) = 0.496$
I'm sorry I cannot present the comment with a clear formating. Thanks

Your joint and marginal distributions are correct. U(Y) is correct as well. For U(Y|X), that is equal to U(X, Y) - U(X), which gives me 0.68867. I think that's where your calculations are off.

Thanks @Robby,
I calculated $U(X,Y)$, but, it seems even off from the result:
1.) Find $P(X,Y)$ by marginalizing $Z$. This gives me $P(X,Y)=0.494$.
2.) Apply $0.494 * log_{2} \frac{1}{0.494}$. This gives me $U(X,Y) = 0.5026$.
3.) Then $U(Y|X)=U(X,Y)-U(X)=(-4973959)$.
Previously, I calculated $U(Y|X)$, as followed:
1.) Calculate $P(X=t) * U(Y|X=t) = 0.50.8400385= 0.42002$.
2.) Calculate $P(X=f) * U(Y|X=f) = 0.50.8486344= 0.42432$.
3.) Then $U(Y|X)=0.84434$.
4.) Finally, $MI(X,Y) = U(Y) - U(Y|X) = 0.9999538 - 0.84434 = 0.15562$, which half of the correct result.
Thanks so much.


text:
Is the stride applied both in the horizontal and vertical directions in convolutional neural networks?
In the convolutional layer for CNNs, when you specify the stride of a filter, typical notes show some examples of this but only for the horizontal panning. Is this same stride applied for the vertical direction too when you're done with the current row?
In other words, say our input volume is 7x7, and we apply a stride of 1 for a 3x3 filter. Is the output volume 5x5? (which would mean you applied the stride in both the horizontal and vertical panning).
Is it possible to apply a different stride for each direction?

Yes, in Keras you can apply different strides by giving a tuple/list, specifying the value of strides along the height and width. If you just give a single value the API assumes the same value for all spatial dimensions.
You can find the official documentation here
In Pytorch, too you can specify the values in a tuple for the stride
argument. Link to Pytorch Documentation for stride

Yes, in Keras this is simply implemented by using a tuple for the stride argument of a convolutional layer, with each element of the tuple corresponding to the stride of each dimension.


text:
How can traditional edge detection algorithms be implemented on a GPU?
How can edge detection algorithms, which are not based on deep learning, such as the canny edge detector, be implemented on a GPU? For example, how are non-edge pixels removed from an image once it detects all the edges?
The reason why I am asking this question is that when writing data to memory the GPU cores can't see what memory locations the other cores are writing to, so I am interested in knowing how traditional edge detectors can be implemented in on GPU.


text:
When past states contain useful information, does A3C perform better than TD3, given that TD3 does not use an LSTM?
I am trying to build an AI that needs to have some information about the past states as well. Therefore, LSTMs are suitable for this.
Now, I want to know that for a problem/game like Breakout, where we require previous states as well, does A3C perform better than TD3, given that TD3 does not have LSTM?
Or without the LSTM TD3 should perform better than A3C despite the fact that A3C has LSTM in it.


text:
How can I find a specific word in an audio file?
I'm trying to train and use a neural network to detect a specific word in an audio file. The input of the neural network is an audio of 2-3 seconds duration, and the neural network must determine whether the input audio (the voice of a person) contains the word "hello"  or not.
I do not know what kind of network to use. I used the SOM network, but I did not get the desired result. My training data contains a large number of voices that contain the word "hello".
Is there any python code for dis problem?

You can use a CNN or LSTM model. For the CNN one you can try pre-trained models for starting and for the LSTM one you can try bidirectional LSTMs.

What datasheet should I use to train the class related to "not hello"?

Have you tried looking at some models at tensorflow hub? I’m positive there should be some useful stuff

After some research on the internet, I realized that using VOSK toolkit in python, it can be found (detect) any particular word in audio file or real time audio streaming.
https://alphacephei.com/vosk/


text:
Why would the loss increase on a single fixed input?
I'm training a neural network on some input data. I know that loss increasing may be related to:

overfitting, if the loss increases on test data (while still decreases on training data)
oscillations near the optimal point, if the learning rate is too big

However, I find that while for some input data the net makes good predictions, for other data the loss continues to increase, even if I only train on one data point and if the learning rate is fairly low. To me, it's quite strange that performing the training on only one point the loss continues to increase and not decreases; in fact, the only reason I can find for this is a big learning rate.
Can you think about some other reason?

by loss, do you mean validation/training error? or do you mean the actual loss function? its also hard to evaluate this without knowing the architecture involved....


text:
Is it possible to have a fixed trajectory size in the vanilla policy gradient algorithm?
In the concept of the vanilla policy gradient algorithm, is it possible for our trajectory size to be fixed?
For example, my environment is the space of embedded images (using a pre-trained encoder to take images in a space with smaller dimensions), the action I am performing is clustering via k-means algorithm and the reward is the silhouette score metric applied on the clustered images.
I am thinking to set batches of size 100 (dataset is MNIST and its trainset size is 60000). Then take the mean of them and consider this as one observation. Then feed this into the policy network to give me its logits which is an array of size 20 (20 discrete actions). These actions tell me the number of k clusters in the k-means clustering algorithm. One value for k is sampled and the k-means algorithm is applied on these 100 images and then reward is calculated.
I can set a constant number for trajectory sizes, for example, 20 and sum the rewards to get the R(trajectory). Is this possible in the context of the RL and policy gradient, or the trajectory's size cannot be fixed? Also, the action that our policy gives us must lead us to the next observation in the environment, but here images are independent of the policy network's parameters.
I wonder if I can utilize RL to implement this. I appreciate any hints.

Trajectory size can be fixed, but in this case problem would be formulated as something similar to the multi-armed bandit problem where there is a single state and a set of actions to choose from. There is no sequential decision making since samples are not correlated, they are picked at random. So, if you take a batch of 20 examples then you would basically have 20 single state trajectories. For each of those trajectories you can calculate policy gradient and average it over the batch size (which would be 20 in this case).

A lot of people seem to try RL as a framework for optimising parameter chocies nowadays, but it is often a poor choice. In this case I agree that the related gradient contextual bandit algorithm could be used - perhaps you could make it clearer that this is what you are suggesting by linking an paper or tutorial?


text:
How to avoid running out of solutions in genetic algorithm due to selection?
The genetic algorithm consists of 5 phases of which 4 are repeated:

Initial population (initially)
Fitness function
Selection
Crossover
Mutation

In the selection phase, the number of solutions decreases. How is it avoided to run out of the population before reaching a suitable solution?

It is not true that the number of solutions necessarily decreases during the selection phase (if by solutions you mean the number of individuals in the population). The number of solutions is usually constant, i.e., you can start with $N$ individuals, then, every iteration (or generation), you can e.g. select two individuals from the population (typically, the fittest ones, but you can have some more sophisticated selection criteria), then you merge them to create two new individuals (i.e. crossover), which will then replace (with a certain probability) the two least fit individuals from the current population, so the population's size remains constant.
If you are talking about reaching a local minimum, i.e. none of the solutions in the population are "good enough", then, as someone has already suggested, there are potentially multiple ways to address this issue, such as

increase the population size
run the genetic algorithm for a longer time (if you have the resources)
change your genetic operators (i.e. the mutation and crossover) so that to introduce more diversity
tweak the replacement, mutation, and crossover rates
change your selection strategy (there are many selection strategies)
make sure that the representation of the solutions is suitable (e.g. once, by mistake, I was using an array of integers rather than floating-point numbers, so I couldn't ever find the correct solution, which was an array of floating-point numbers)
use something like novelty search

The correct approach will probably depend on the context.

There are multiple ways to interpret those steps. The most common standard approaches are

select two parents and produce two offspring; repeat until child population is the same size as parent population, and let the children replace their parents unconditionally (generational GA)

same as the above, but allow a few parents to live on instead of a few children if the parents have higher fitness (elitism)

each iteration, select two parents, produce one child, let the child replace a member of the parent population if it is better (steady state GA)

But there are other ways to go. There's an algorithm called CHC that lets the child population get smaller over time, and when it reaches zero, the algorithm triggers a smart restart. The point is there's no single definition for what makes an evolutionary algorithm. It's up to you to decide how to make something that works well for your problem. When you're a beginner though, it's handy to start from known points, like the three I mentioned above.

This is a more complex question than it might initially seem. A genetic algorithm models a biological process,namely population genetics. No biological population evolves to a single cloned individual, a process in genetic algorithms referred to as premature convergence where the population converges to a single non optimal, though possibly locally optimal, solution. The avoidance of premature convergence or the maintenance of population diversity is an important aspect of the genetic model that is often not well addressed, and one that the five step model you detail definitely does not.
The one operator that will maintain diversity is mutation, since it is a purely random operator. However, what the mutation rate should be is highly argued over. A general consensus is that if each chromosome is of length N then the mutation rate should be 1/N. Likewise, the consensus is that 60% of the population should be replaced in each breeding cycle.
However, these settings do not emerge directly from biological reality and premature convergence remains problematic. A more realistic model is to reflect the fact that in biology resources are finite, and to adjust the fitness of individuals proportionate to the number of similar individuals on the assumption that similar individuals are chasing the same resource. The fitness landscape is thus dynamically warped by  the changing distribution of the population. You will still have to retain memory of the fittest solution before adjustment. A common solution is to apply cluster analysis to the population, reducing the individual’s fitness by the size of the cluster to which it is allotted. A seminal paper is by Yin and Germay A Fast Genetic Algorithm with Sharing Scheme Using Cluster Analysis Methods in Multimodal Function Optimization `. The assumption is still made that the population is modelling a single biological species. How diversity does not merely maintain diversity but results in a population dividing into separate reproductively isolated species is a question for another day, and one that divides biologists to the current day.


text:
What are some applications where tree models perform better than neural networks?
Neural networks are known to be generally better modeling techniques as compared to tree-based models (such as decision trees). Are there any exceptions to this?

Hard to say in general. Speaking from my own experience and by looking at which models win Kaggle competitions (see here and here), I would say tree-based models e.g. Random Forests, Decision Trees, Gradient Boosting are favorable over neural networks when working with low-dimensional data and easy interpretable features (usually simple tabular data with numeric, ordinal or categorical features).
Whereas when working with everything high-dimensional like images, text, time series or other data with non-trivial features, I would recommend neural networks.
Of course there might be exceptions and the future may prove me wrong.


text:
What does self-play in reinforcement learning lead to?

Suppose, instead of playing against a random opponent, the reinforcement learning algorithm described above played against itself, with both sides learning. What do you think would happen in this case? Would it learn a different policy for selecting moves?

Above is an extract from Reinforcement Learning: An Introduction by Andrew Barto and Richard S. Sutton, and I wasn't quite sure about what the answer to the question would be, so thought of posting it here. The algorithm being referred to is the one for playing the game tic-tac-toe.
In my opinion, if the same algorithm plays both sides, it may end up assisting itself to win every time - and not really learn anything. What do you think?


text:
How can deep Q-learning converge if the targets may not be correct?
In deep Q-learning, $Q(s, a)$ and $Q'(s, a)$ are predicted or estimated by the neural network itself. In supervised learning, the target value is a true unbiased value. However, this isn't the case in reinforcement learning. So, how can we be sure that deep Q-learning converges? How do we know that the target Q values are accurate?

See https://ai.stackexchange.com/q/21053/2444 and https://ai.stackexchange.com/q/11679/2444.


text:
How to combine two differently equally important signals into the reward function, that have different scales?
I have two signals that I want to use to model my reward.
The first one is the CPU TIME: running mean from this diagram:

The second one is the MAX RESIDUAL from this diagram:

Since they are both equally important, I can weight them together like this:
$r = w_\rho \rho + w_\tau \tau$
where $r$ is the reward function, $\tau$ is the CPU TIME: running mean, and $\rho$ is the MAX RESIDUAL. The problem is, how to set the weights $w_\tau,w_\rho$ to make the contributions equally important if $\rho$ and $\tau$ are on very different scales?
RL algorithms will learn policies based on increases/decreases of the reward, and if one signal has values that are much smaller than the other, it will influence the reward less, if this is done in a wrong way.
On the other hand, if the algorithm converges, they must be on different scales, as I want the CPU time to go ideally to $0$, and residuals $\rho$ to be minimized as well.
Modeling the reward function is a crucial RL step, because it decides what the algorithm will in fact optimize. How are examples like these handled? Are there any best practices for this? Also, what happens when there are $n$ such signals, that have to be combined with "equally important" weighting into a reward function?
Basing the weights $w$ on the current signal values is possible to define their reward, but then the reward contributions won't see $\max(\rho), \min(\rho)$ and $\max(\tau), \min(\tau)$ over time.
So, how do you do feature scaling for reward signals?

scale your rewards to be 0 mean and variance 1?

@DavidIreland I tried z-scaling already and it doesn't seem to scale to same min,max intervals for "running" signals... can you suggest an alternative scaling for running signals that does what you suggest? Thanks, btw :)


text:
What is the computational complexity of the forward pass of a convolutional neural network?
How do I determine the computational complexity (big-O notation) of the forward pass of a convolutional neural network?
Let's assume for simplicity that we use zero-padding such that the input size and the output size are the same.

"assume for simplicity that we use zero-padding such that the input size and the output size are the same." so you are assuming zero-filter as well?

@k.c.sayz'k.csayz' very funny! Anyway I assume the discussion about filters is going to be part of the answer.

What is the time complexity?
The time complexity of an algorithm is the number of basic operations, such as multiplications and summations, that the algorithm performs. The time complexity is usually expressed as a function of the input's size $n$ (but this does not always have to be the case: for instance, you can express the time complexity as a function of the output's size).
Example
Rather than giving you a full answer to your question, I will try to help you by explaining, with the simplest example, how you should calculate the time complexity.
For simplicity, let's assume that we have a kernel $\mathbf{H} \in \mathbb{R}^{3 \times 3}$ and input image $\mathbf{I} \in \mathbb{R}^{3 \times 3}$ (i.e. the kernel has the same dimensions as the input), we use a stride of $1$ and no padding. If we convolve $\mathbf{I}$ with $\mathbf{H}$, how many operations will we perform? The convolution is defined as a scalar product, so it is composed of multiplications and summations, so we need to count both of them. We have $9$ multiplications and $8$ summations, for a total of $17$ operations.
\begin{align}
\mathbf{I} \circledast \mathbf{H}
&=
\begin{bmatrix}
i_{11} & i_{12} & i_{13} \\
i_{21} & i_{22} & i_{23} \\
i_{31} & i_{32} & i_{33}
\end{bmatrix}
\odot
\begin{bmatrix}
h_{11} & h_{12} & h_{13} \\
h_{21} & h_{22} & h_{23} \\
h_{31} & h_{32} & h_{33}
\end{bmatrix}\\
&=
\sum_{ij}
\begin{bmatrix}
i_{11} h_{11} & i_{12} h_{12} & i_{13} h_{13} \\
i_{21} h_{21} & i_{22}  h_{22} & i_{23} h_{23} \\
i_{31} h_{31} & i_{32} h_{32} & i_{33} h_{33}
\end{bmatrix}\\
&=
i_{11} h_{11} + i_{12} h_{12} + i_{13} h_{13} +
i_{21} h_{21} + i_{22}  h_{22} + i_{23} h_{23} +
i_{31} h_{31} + i_{32} h_{32} + i_{33} h_{33}
\end{align}
Time complexity
What is the time complexity of this convolution? To answer this question, you first need to know the input's size, $n$. The input contains $9$ elements, so its size is $n = 9$. How many operations did we perform with respect to the input's size? We performed $17$ operations, so the time complexity $\mathcal{O}(2*n) = \mathcal{O}(n)$, i.e. this operation is linear. If you are not familiar with the big-O notation, I suggest that you get familiar with it, otherwise, you will not understand anything about computational complexity.
To calculate the time complexity in the case the input's dimensions are different than the kernel's dimensions, you will need to calculate the number of times you slide the kernel over the input. You can't ignore this (as I ignored the constant $2$ above) because the number of times you slide the kernel over the input depends on the input's size, so it's a function of the input. Anyway, the paper A guide to convolution arithmetic for deep learning contains a lot of information about convolution arithmetic, so it will be helpful.
Non-linearities, pooling and fully connected layers
Note that, in the above example, I ignored the non-linearities and pooling layer. You can easily extend my reasoning to include these operations too. I also ignored the operations in the final fully connected layers. You can find how to calculate the number of operations in an MLP in this answer. If you also perform other operations or have other layers other than convolutional, pooling and fully connected, of course, you will also need to take them into account.
Forward pass
Moreover, the time complexity of the forward pass of a CNN depends on all these operations in these different layers, so you need to compute the number of operations in each layer first. However, once you know how to compute the number of operations for one convolutional layer, one pooling layer, and one fully connected layer, you can easily compute the number of operations for the other convolutional, pooling, and fully connected layers. Then you just need to sum all these operations and express your time complexity as a function of the input (and probably number of layers).
Space complexity
If you also want to compute the space complexity, you just need to do the same thing, but as a function of the space that you use, i.e. how many variables you use to perform the convolution.

"so its size is n=9. How many operations did we perform with respect to the input's size? We performed 17 operations, so the time complexity O(2∗n)" What? 2 * 9 = 18. Or I am wrong?


text:
Does the selective search algorithm in object detection learn?
I am trying to get a better grasp of how object detection works. I (almost) completely understand the concept behind RPNs. However, I am a little bit confused with the selective search algorithm part. This algorithm does not really learn anything, as far as I understand.
So, for example, when I have an image containing people (even though my network does not need to classify these), will the selective search still propose these people to my network?
Of course, my CNN  has not learned to classify a human and will output a very low probability for every class (it did learn), and thus, this way, the human will not contain a bounding box.
Also, in further iterations of the R-CNN model, they proposed using regressors to improve the bounding box.
Does this mean that this part of the model got the CNNs feature maps, and, based on this, learned to output a bounding box (this way smaller instances of a detected object would get a smaller bounding box)?
So, in this first iteration, they probably did not need bounding boxes in the training data (since there was no way to learn the size of the bounding boxes and thus no need to find a loss function for this problem)?
Lastly, I understand that the selective search algorithm is an improvement on the sliding window algorithm. It tries to have a high recall, so having false positives is not bad, as long as we have all the true positives. Again, I do not seem to understand HOW this algorithm knows when it has the object it needs without really learning. Any intuïtive explanation or visual (I am a visual learner at first) on how this algorithm works is greatly appreciated.

The selective search algorithm is described in the paper Selective Search for Object Recognition, so you can read it and then attempt to provide an answer to your own question below.


text:
How to generate labels for self-supervised training?
I've been reading a lot lately about self-supervised learning and I didn't understand very well how to generate the desired label for a given image.
Let's say that I have an image classification task, and I have very little labeled data.
How can I generate the target label from the other data in the dataset?

Hi, I am also working on SSL task or more learning how to do it. Is it please possible, if you show me the way you did it please? :)

@TimcoVanco See my updated answer below. It should clarify your doubts, at this point, I believe.

You should accept my answer below (by clicking on the checkmark), if it answers your question. It should answer your question, at this point. If not, please, let me know what is still unclear.

How can I generate the target label from the other data in the dataset?

If you are asking how you can create the learning signal in SSL, when given an unlabelled dataset, for learning representations of these unlabelled data, then there is no general answer. The answer depends on the type of data that you have (which can be e.g. textual or visual), and which features do you think you want to learn or can be learned from your unlabelled data. This paper and other answers to this question provide some examples of how that can be done (depending on the type of data). Below, I also provide an example.
Let me try to explain this more in detail.
Let's assume that you have both

an unlabelled dataset $U = \{ u_i \}_{i=1}^m$ and

a labelled dataset $D = \{(x_i, y_i) \}_{i=1}^n$

where we may have $m \gg n$ (although this is not a strict requirement), i.e. you may have a lot more unlabelled data than labelled data (this can easily be the case, given that, in general, manual data annotation is expensive/laborious). Let's say that your ultimate task is to perform object recognition (or classification). Let's call this task the downstream task. So, you may think that $x_i$ and $u_i$ are images and $y_i$ are labels, like "cat" or "dog" (let's say that you want to differentiate between cats and dogs).
You want to solve this downstream task by supervised learning with $D$. However, given that your labeled dataset is not big enough, you may think that training a neural network from scratch (i.e. by randomly initializing its weights) with $D$ may not lead to good performance. So, you think that it could be useful to start training from a pre-trained model that already contains useful representations of data similar to your labeled data, i.e. to perform transfer learning. To pre-train such a model, you could use SSL.
So, to solve your downstream task with SSL, there are 2 different steps

Self-supervised learning (SSL): learn representations of your images $u_i \in U$ by training a neural network $M$ with $U$ to solve a so-called pretext (or auxiliary task); there are many pre-text tasks: you can find many examples here, here and here (see example below too);

Supervised learning (SL) by transfer learning: fine-tune $M$ with $D$ (the labeled dataset), in a supervised way; this task is known as downstream task (as stated above)

In this process, there are 2 different labels.

In step 1, you have the labels that are generated automatically. But how are these labels generated? As I said, there are many ways. Let me describe one way (among many others!). Let's say that your unlabelled dataset $U$ contains high-resolution images (i.e. $u_i \in U$ are high-resolution images), then you could define your pre-text task as follows. You lower the resolution of your high-resolution images to create other images. Let $v_i$ be the low-resolution image created from the high-resolution image $u_i \in U$, then the training pair to your neural network $M$ is $(v_i, u_i) \in U'$, where $u_i$ is the label (which is the original high-resolution image) and $U'$ the labeled dataset automatically generated (i.e. with the algorithm I've just explained).
So, these labels $u_i$ (high-resolution images) are semantically different than $y_i$ ("cat" or "dog") in the pairs $(x_i, y_i) \in D$. They are different because, here, we want to learn representations and not to perform object recognition/classification: the idea is that, by solving this pre-text task, your final trained neural network, should have learned features of the images in the unlabelled data (i.e. representation learning). These learned features can then be used to bootstrap training in the downstream task.

In step 2, you use the labeled dataset $D$, which has been typically annotated (or labeled) by a human. As stated above, this dataset contains pairs $(x_i, y_i)$, where $y_i$ is, for example, the label "cat" or "dog".
In this step, the pre-trained model $M$, with the SSL technique,  can be fine-tuned with $D$ in a supervised fashion. Given that we start with a pre-trained model $M$, we are effectively performing transfer learning.

Note that SSL can also refer to something (slightly) different than what has been explained in this answer. See my other answer for more details. Moreover, note that you can perform representation learning with SSL without necessarily solving a downstream task later, which may also not be an SL task (in the example above, I've described a downstream task that is an SL task only for simplicity).
If this answer is still unclear, maybe you should have a look at existing implementations of SSL techniques (such as this) for more inspiration.