text:
In off-policy MC control algorithm by Sutton & Barto, why do we perform a last update when sample action is inconsistent with target policy?
I have a question about the $W$ term in the off-policy MC control algorithm on Page 111 of Sutton & Barto. I have also included it in the figure below.

My question: shouldn't the check $A_{t} = \pi(S_{t})$ be made before updating $C(S_{t}, A_{t})$ and $Q(S_{t}, A_{t})$? And, at this point if $A_{t} \neq \pi(S_{t}) $ then the inner loop should exit before updating $Q(\cdot)$. If $A_{t} = \pi(S_{t})$ then shouldn't $W$ be updated to
$W = W \frac{1}{b(A_{t}|S_{t})} $ before updating the $Q(s, a)$ and $C(s, a) functions?
The algorithm as stated seems problematic to me. For example, if say the target policy $\pi$ is deterministic and behavior policy $b$ is stochastic. If in period $T-1$ the behavior policy takes an action that is not consistent with $\pi$ then the importance sampling ratio $\rho_{T-1:T-1} = 0$. However, the algorithm as shown would update $Q(S_{T-1}, A_{T-1})$ since the checks I referred to above don't occur until the end of the inner loop. What am I missing here?

I think that this is an intentional subtle detail of the algorithm that ensures the convergence property.  The claim in the book is that for any $b$ that provides us with "an infinite number of returns for each pair of state and action" the target policy $\pi$ will converge to optimal.
Imagine now that we have such a bad policy $b$ that it never aligns with the target policy action at the last step $t=T-1$ of each generated episode: $A_{T-1} = \pi(S_{T-1})$. In that case the weight value will stay $W=1$ and the the algorithm will be reduced to (ignoring $t$ indices for the last $S,A,R$ triplet):
$$\begin{array}{l} C(S,A) \leftarrow C(S,A) + 1 \\
Q(S,A) \leftarrow Q(S,A) + \frac{1}{C(S,A)}\left[R - Q(S,A)\right]
\end{array}$$
Which is just the tabular incremental averaging for the Q values (see for example eq. (2.3)).
If we bail from the for loop before these updates, then no updates would happen at all. And authors won't be able to claim convergence to optimal policy for all those "good" sampling policies $b$.

Thank you. I wish they at least had a brief comment stating this. To leave this completely without comment seems strange (it was driving me crazy). Thank you again. Regarding, "we have such a bad policy  that it never aligns with the target policy action at the last step =−1 of each generated episode". I suppose that is not possible given coverage, right? It might be super slow but has to align from time to time.

@Curious2learn I agree that this point should've been stressed, as far as I understand the algorithm is not too practical and authors wanted to move on to the next chapters that discuss all the (authors' original) improvements to this approach.

@Curious2learn As for the possibility of such a "bad" $b$. I guess that depends on concrete MDP - there are definitely MDPs where it is inconsistent with coverage, but I think one can come up with an MDP where it is possible.

Thank you. Another reason why such an update might make sense is the following. Say the initial target policy specifies an action $a$ in state $s$ in $T-1$. Suppose the updated value of $Q(s, a) $ is greater than 0, and all other initial $Q(s, a') <= 0$. Then if $Q(s, a2) > Q(s, a)$, and even if the behavior policy takes actions a2 from time to time, any experience from that would never be used to improve the target policy. But updating even when the sample action is inconsistent with the target policy can help improve the action under the target policy.

The algorithm as stated seems problematic to me. For example, if say the target policy  is deterministic and behavior policy  is stochastic. If in period −1 the behavior policy takes an action that is not consistent with  then the importance sampling ratio −1:−1=0. However, the algorithm as shown would update (−1,−1) since the checks I referred to above don't occur until the end of the inner loop. What am I missing here?

Why is this problematic? You are trying to estimate Q(s, a), which is the expected reward given that a has been played in state s, assuming to behave according to the target policy in the future.
If you were learning it on-policy, you would sample future states and actions based on the target policy. Here, you sample according to the behaviour policy and then correct with the ratio of probabilities of future states and actions under the two policies.
But you are conditioning everything on having played action a in state s in the first place, so the probability of that happening does not enter the computation.


text:
Why the non-exploitation of edge labels in current graph convolutions "results in an overly homogeneous view of local graph neighborhoods"?
I am currently reading a paper called Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphs (2017, CPPR), and I cannot understand the following sentence:

We identify that the current formulations of graph convolution do not
exploit edge labels, which results in an overly homogeneous view of
local graph neighborhoods, with an effect similar to enforcing
rotational invariance of filters in regular convolutions on images.

What does this sentence mean?

Consider a two-dimensional convolution layer with 3x3 kernels. The 2d inputs of this layer can be seen as a particular graph with each pixel being a graph node, that is connected to 8 of his neighbors:

The 3x3 kernels of the convolutional layer not only process the information about neighborhood relation between pixels, but also about their relative orientation. For example the [0,0] element of the kernel might represent the weight of the node to NW. And the [1,2] element of the kernel represent the weight the node across the S edge:

Now, if we make a convolution that "doesn't exploit edge labels" then we'll have to forget the labels on the picture above, making us loose directional information:

All we can say now is that the "red" node has those neighbors, but we don't really know how they are oriented relative to it. Since now the sub-graph does not provide any directional information, the learned convolution kernels will be direction-agnostic - in other words, they will be rotation-invariant.

This answer is interesting but it's not fully clear to me. When you say "The 3x3 kernels of the convolutional layer not only process the information about neighborhood relation between pixels, but also about their relative orientation", are you talking about the regular convolutional layer of a CNN or a graph CNN? Because, if you're talking about CNNs, I don't see how the multiplication of the kernel by that specific subwindow of the 2d input keeps track of the relative positions (i.e. it's just a dot product).

Maybe you mean the application of the whole kernel across all subwindows of the input (i.e. the convolution/cross-correlation)? That would make more sense to me. But, from your example that you continue below, it doesn't seem that's what you mean. In any case, although it's been a while I had to deal with graph CNNs, your interpretation of the potential utility of including edge labelling info in the graph convolution seems reasonable to me, although maybe it's important to note that the edges do not necessarily represent directions (I guess).

Although maybe it's clear to you, I think you should also explicitly state where the translation invariance comes into play in your answer and explanations, and how it relates to the regular graph convolution.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Does regularization just mean using an augmented loss function?
We need to use a loss function for training the neural networks.
In general, the loss function depends only on the desired output $y$ and actual output $\hat{y}$ and is represented as $L(y, \hat{y})$.
As per my current understanding,

Regularization is nothing but using a new loss function
$L'(y,\hat{y})$ which must contain a $\lambda$ term (formally called
as regularization term) for training a neural network and can be
represented as
$$L'(y,\hat{y}) = L(y, \hat{y}) + \lambda \ell(.) $$
where $\ell(.)$ is called regularization function. Based on the
definition of function $\ell$ there can be different regularization
methods.

Is my current understanding complete? Or is there any other technique in machine learning that is also considered a regularization technique? If yes, where can I read about that regularization?

Regularization is not limited to methods like L1/L2 regularization which are specific versions of what you showed.
Regularization is any technique that would prevent network from overfitting and help network to be more generalizable to unseen data. Some other techniques are Dropout, Early Stopping, Data Augmentation, limiting the capacity of network by reducing number of trainable parameters.

L2 regularisation is an interesting case, in that there are two equivalent ways of implementing it. You can use an adjustment to the loss function, or you can decay the weights after each update step

Also, keep in mind that not just any augmentation of the loss function is a regularization.
For example, you can add terms to a loss function that enforce constraints on the solution but do not prevent overfitting nor facilitate generalization.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What is numerical stability?
I came across the phrase "numerical stability" several times. But almost in the same context.
I encountered this word mostly in the analytical formula for batch normalization.

$$y = \dfrac{x - \mathbb{E}[x]}{\sqrt{Var[x]+\epsilon}}* \gamma +
 \beta$$
eps – a value added to the denominator for numerical stability. Default: 1e-5

Is the phenomenon of "numerical instability" happens during the training of neural networks? Or is it a general one in other models also? What is the reason for its occurrence?

Related: 4.2 Poor Conditioning

You can find a definition for "numerical stability" in mathworld wolframe:

Numerical stability refers to how a malformed input affects the execution of an algorithm. In a numerically stable algorithm, errors in the input lessen in significance as the algorithm executes, having little effect on the final output.
On the other hand, in a numerically unstable algorithm, errors in the input cause a considerably larger error in the final output.

In your example, suppose $Var(x)$ is little and truncated to zero in the corresponding computing machine. In that case, you will get INF as a result and it can be problematic to continue the computation.
Therefore, they add a small value such as $\epsilon$ to the division to prevent such a case. As $Var(x) \geqslant 0$, $Var(x) + \epsilon$ will be always greater than zero for any machine with higher precision than the $\epsilon$. So, here, an interpretation of the numerical stability can be like this!


text:
Why do we add 1 in the formula to calculate the shape of the output of the convolution?
In the formula to calculate output shape of tensor after convolution operation
$$
W_2 = (W_1-F+2P)/S + 1,
$$
where:

$W_2$ is the output shape of the tensor
$W_1$ is the input shape
$F$ is the filter size
$P$ is the padding
$S$ is the stride.

Why do we add $1$? It gets us to the correct answer, but how is this formula derived?
Source: https://cs231n.github.io/convolutional-networks/#pool

In a few words, we add $1$ to account for the initial position of the kernel.
You can easily see this if you let $s = 1$ (unit stride) and $p = 0$ (i.e. no padding), so your formula simplifies to
\begin{align}
W_2 = (W_1 - F) + 1, \label{1}\tag{1}
\end{align}
So, in this simplified case and, for simplicity, assuming squared inputs and kernels, the width (or height) of the output of the convolutional layer is the number of steps that we slide the kernel horizontally (or vertically, respectively), for example, starting from the top left of the input, plus the initial position of the kernel. In this case, $(W_1 - F)$ is the number of times we slide the kernel horizontally (or vertically) and $+1$ is to account for the initial position of the kernel.
Here's an animation of this simplified case for $W_1 = 4$ and $F = 3$ (the green matrix is the output, while the blue one is the input).

So, if we apply the formula \ref{1}, we should get $W_2 = 2$, as we can see from the animation above, which produces a $2 \times 2$ matrix. In fact, $(4 - 3) + 1 = 2 = W_2$. You can see from the animation that we slide the kernel only once in each of the axes, but the shape of the output matrix is $2 \times 2$ because we compute the dot product between the kernel and the submatrix of the input that corresponds to the initial position of the kernel.
A detailed explanation of this (section 2.4, relationship 6) and other simpler formulas to compute the size of the output of a convolutional layer can be found in the report A guide to convolution arithmetic for deep learning, which has also many images that illustrate the concepts (you can find animations here too).


text:
If we can model the environment, wouldn't be meaningless to use a model-free algorithm?
I am trying to understand the concept of model-free and model-based approaches. As far as I understand, having a model of the environment does not mean that an RL agent has to be model-based. It is about the policy. However, if we can model the environment, why should we want to employ a model-free algorithm? Isn't it better to have a model and expectation about the next reward and state? If you have a better understanding of all these, can you explain them to me as well?

I am not sure what you mean by "It is about the policy"? Could you please clarify?

I mean, even if we have a model, the main reason for having this model is to find the optimal policy. (If I am right!).

However, if we can model the environment, why should we want to employ a model-free algorithm?

Depends what you mean by "model the environment". There are two kinds of model:

Distribution model, which provides full access to a function like $p(r,s'|s,a)$, the probability of observing reward $r$ and next state $s'$ given starting state $s$ and taking action $a$.

Sample model, which can process a single step within the environment on demand. That should provide access to a function like $\text{step}(s,a)$ that returns a single $(r, s')$ pair based on running the environment forward from $(s,a)$.

Which kind of model is available makes a difference to which kinds of model-based reinforcement-learning approaches will work. For instance a distribution model is required for dynamic programming, but MCTS may only need a sample model.
A distribution model can be converted easily to a sample model, but the other way around is only possible approximately (by taking lots of samples).

Isn't it better to have a model and expectation about the next reward and state?

All things being equal, then yes having access to and using a model allows for better estimates, and opens up possibilities for various types of planning algorithm.
However, even if you can build a model of an environment, there are costs:

Writing a distribution model can be hard theoretically - it is often far easier to simulate an environment for a single step forward than calculate probability distributions for all outcomes of the same step.

The need for random access to arbitrary states can be expensive. A simulator that only needs to run forward from a single reset to a start state may run much more efficiently than one that needs to be set to multiple places in a tree of possibilities arbitrarily.

These costs are different for different environments. Setting a game like chess or go to arbitrary states is relatively cheap compared to a modern computer game or something that simulates a real-world physical system with lots of detail. Partially-observable states can be a major problem in both cases, since any random access model would need to account for all possible "true" states that lie behind whatever the agent is processing.
So basically, at different levels of environment complexity, either the development cost of making a full model, or the evaluation cost of setting arbitrary states can make it more effective to use a model-free approach.
Agents that learn to build their own approximate models of the environment are of great interest, as these potentially address the issue by getting the best of both world. Or possibly they could be tunable to use model-free vs model-based approaches in the most efficient manner depending on the relative costs and accuracy of each approach.

I'm not sure I understand why you differentiate between what you call a "distribution model" and a "sampling model". You always need something that returns you the next state $s'$ and reward $r$ for every state $s$ and action $a$ for any RL algorithm; so, if I understand correctly your terminology, you always need a "sampling model" anyway. It seems to me that a sampling model is just the distribution model, but without knowing the probabilities, i.e. just a way to sample from $p.$

I wouldn't call it a sampling model, but step/sampling function (as you do) to differentiate it from what we usually call "model" in RL (not sure if "sampling model" is used in the literature, I don't know). The step function may be stochastic, but we may not know exact how it computes. Or your "sampling model" would be kind of a simulation of the real environment?

@nbro: Sample model and distribution model are terms used in e.g. Sutton & Barto. A distribution model can produce a sample model, but not the other way around (except via some statistical learning algorithm). Technically the book uses "sample model", not "sampling model", so I'll change my answer to match

Ok. It's been a while since I read that book, and probably that term is only used in some chapters and maybe in the 2nd ed (I will check that).

@nbro: You will not need a sample model if you have access to the real environment. As some real environments are virtual, implemented in code, then sometimes that line is blurred. Anyway, with a real environment, the step function is not calling a model, it is calling the real instance of the thing.

I am kind of confused. The definition of the model necessitates transition and reward function. In this case, I am not quite sure what you mean by "real environment".

@NeilSlater Thanks for your answer. I see your point. Thus, if I have a physical dynamic system like the one that you generally can solve by linear programming or non-linear programming, I can actually create my model because I know the system dynamics and can write as an equation. Then, I need to use a policy or value iteration type solution only since I should employ dynamic programming methods.

A real environment is not a model. It is the real thing. This sometimes get lost conceptually because many environments are simulations, games etc that are inside the same computer as the agent. However, you would not write transition functions for a physical robot - the transition logic is what happens in the real world to the robot.

@NeilSlater So my transition function is getting after I created my solution and becoming with experiences?


text:
How do I know what a good mean absolute error value is?
I have just run an MAE calculation for my machine learning models and the results show:

SVM MAE = 28.850 deg.
Random Forest MAE = 33.832 deg.

How do I know what a good MAE value is? What is the range of the MAE?

Hello. This question was closed as "needs more details". Not sure why, but people probably wanted you to describe what you didn't understand about MAE when you read about it on the web. In other words, if you haven't done a little bit of research before asking this question, please, next time, do a little bit of research before asking a question. Some questions are so basic that answers can be immediately found as the first entry of a google search. In any case, you already got an answer to your question, so there's no strict need to reopen this post.

Mean Absolute Error is nothing but the mean of absolute errors.
If your model gave $n$ predictions $\{\hat{y}_i\}_{i = 1}^{n}$ against $n$ ground truths $\{y_i\}_{i = 1}^{n}$, then MAE is defines as follows
$$MAE_{model} = \dfrac{\sum\limits_{i = 1}^{n} |y_i - \hat{y}_i|}{n} $$.
Thus, MAE gives the average amount of error. So, the machine learning model with minimum MAE should be considered as a good model since it is giving minimum error.
In your task, SVM should be considered as a good model.
The range of $MAE$ is $[0,\infty)$. The lower bound zero is achieved if all your model's predictions are exactly correct, else it will give positive value depending on how worse your model's predictions are.

Additional to @hanugm's response, whether a MAE is good or not depends on data. If you're predicting millions and MAE is 100, sounds good. If you're predicting 2 digits and MAE is 100, it's not good. For this reason, there's another metric called MAPE: Mean absolute percentage error.


text:
What does "linear unit" mean in the names of activation functions?
Activation functions, in neural networks, are used to introduce non-linearity. Many activation functions that are used in neural networks have the term "Linear Unit" in their full form. "Linear unit" can be abbreviated as LU.
For example, consider some activation functions

ELU  - Exponential Linear Unit
ReLU - Rectified Linear Unit
................................................

Why does the function name contain the term "Linear Unit"? What is meant by Linear Unit here? Is it saying anything about the nature of function under consideration?

Have a look at these graphics showing popular linear units (image taken from Clevert et al. 2016):

You can see that these functions are linear functions for $x > 0$, that's why they are called Linear Units.
For example, the ELU is defined as
$$ ELU(x) = \begin{cases}
x &\text{if } x > 0\\
\alpha (\exp(x)-1) & \text{if } x \leq 0.
\end{cases} $$
These functions introduce the nonlinearity around zero, each in its own way, which can be used for different problems.


text:
In Computer Vision, what is the difference between a transformer and attention?
Having been studying computer vision for a while, I still cannot understand what the difference between a transformer and attention is?

The original transformer is a feedforward neural network (FFNN)-based architecture that makes use of an attention mechanism. So, this is the difference: an attention mechanism (in particular, a self-attention operation) is used by the transformer, which is not just this attention mechanism, but it's an encoder-decoder architecture, which makes use of other techniques too: for example, positional encoding and layer normalization. In other words, the transformer is the model, while the attention is a technique used by the model.
The paper that introduced the transformer Attention Is All You Need (2017, NIPS) contains a diagram of the transformer and the attention block (i.e. the part of the transformer that does this attention operation).
Here's the diagram of the transformer.

Here's the picture of the attention mechanism (as you can see from the diagram above, the transformer used the multi-head attention on the right).

One thing to keep in mind is that the idea of attention is not novel to the transformer, given that similar ideas had already been used in previous works and models, for example, here, although the specific attention mechanisms are different.
Of course, you should read the mentioned paper for more details about the transformer, the attention mechanism and the diagrams above.

Could you reply to this question, please?https://datascience.stackexchange.com/questions/124020/how-can-we-use-a-transfomer-model-with-new-data-if-we-still-dont-have-the-outpu/124022#124022


text:
Why is the exponential loss used in this case?
I am reading the paper Tracking-by-Segmentation With Online Gradient Boosting Decision Tree. In Section 2.1, the paper says

Given training examples, $\left\{\left(\mathbf{x}_{i}, y_{i}\right) \mid \mathbf{x}_{i} \in \mathbb{R}^{n}\right.$ and $y_{i} \in$ $\mathbb{R}\}_{i=1: N}, f(\cdot)$ is constructed in a greedy manner by selecting parameter $\theta_{j}$ and weight $\alpha_{j}$ of a weak learner iteratively to minimize an augmented loss function given by
$$
\mathcal{L}=\sum_{i=1}^{N} \ell\left(y_{i}, f\left(\mathbf{x}_{i}\right)\right) \equiv \sum_{i=1}^{N} \exp \left(-y_{i} f\left(\mathbf{x}_{i}\right)\right)
$$
where an exponential loss function is adopted ${ }^{1}$. The greedy optimization procedure is summarized in Algorithm 1.

I cannot understand the exponential loss function. In my opinion, the loss function should get the smallest value when $y_i=f(x_i)$. But the loss function in the image obtains a smaller value if $(-y_i f(x_i))$ becomes smaller.

The loss is
$$\mathcal{L}=\sum_{i=1}^{N} \ell\left(y_{i}, f\left(\mathbf{x}_{i}\right)\right) \equiv \sum_{i=1}^{N} \exp \left(-y_{i} f\left(\mathbf{x}_{i}\right)\right),$$
which can also be written as follows
$$\mathcal{L} = \sum_{i=1}^{N} e^{-y_{i} f\left(\mathbf{x}_{i}\right)} \tag{1}\label{1}$$
The important thing to note here is the $-$ in the exponent, which allows us to write \ref{1} as follows (see this)
$$\mathcal{L} = \sum_{i=1}^{N} \frac{1}{e^{y_{i} f\left(\mathbf{x}_{i}\right)}} \tag{2}\label{2}$$
So, the loss becomes smaller the higher $y_{i}$ and $f\left(\mathbf{x}_{i}\right)$ are. If, for example, $y_{i}$ is negative and $f\left(\mathbf{x}_{i}\right)$ positive (or vice-versa), then \ref{2} would be higher. If both are positive or negative, then the loss will be smaller, as we will be summing fractions of the form $\frac{1}{e^k} < 1$ for some positive constant $k$: the higher the $k$, the smaller the loss.
So, if you use this loss, it seems that you want that

$y_{i}$ and $f\left(\mathbf{x}_{i}\right)$ have the same sign (either both negative or both positive)
$f\left(\mathbf{x}_{i}\right)$ is as high as possible (note that you cannot change $y_{i}$), which could even be much higher (in terms of magnitude) than $y_{i}$ (not sure why they would want this)

I don't know why they chose this loss function because I didn't read the paper (yet), but it seems to me that this is how you should interpret this loss function.


text:
In the cross-entropy method, should I select state-action pairs by their immediate reward or by the episode reward?
I am trying to understand the code mechanics when selecting the elite states and elite actions. It appears clear to me that they are those that appear in the episodes with the rewards bigger than the threshold.
My question is: should I select state-action pairs by their immediate reward or by the episode reward?
I am applying the method to a craft environment interesting to me and I have been studying an example applying the OpenAI's Gym taxi environment, but I do not fully understand the code.

Hello. I've edited your post in order to clarify what your question is by putting it in the title. Please, make sure that the current version of your post is still consistent with the original one. If not, feel free to edit your post again to fix that.

Although the edition is Ok, I am not sure if monte-carlo-methods as tag is correct.

It is Monte Carlo in the general sense of simulating outcomes by sampling from random variables (in this case only the policy). I would not have used monte-carlo-methods here, CEM is not a variant of Monte Carlo control. Although you could consider REINFORCE algorithm strongly related to CEM, and an "upgrade" of it.

It may be a good idea to provide a reference/link to the CEM you're referring to, just to make sure we're talking about the same thing.

@NeilSlater  I am refining my skills related to Reinforcement Learning after doing some online courses in Coursera. I intend to apply these knowledge area to the adaptation of livestock farmers to climate variability. As an excercise, I am trying to deep my expertise in the different kind of  RL and related algorithms. For that, initially I must master the environment construction under different perspectives, tabular and approximation methods, discrete and continuous state and action spaces, and solidify my familiarity with the discipline

@nbro It would be a very good new to have some comments on my very initial progress. For that I must also improve my Github knowledge in order to publish the different versions that I'll be producing in the next months. For the moment I do not have any working solution....I'll let you know as soon as I have any concrete version.

My question is if I should select state_action pairs by theirs immediate reward or should I select them by the episode reward?

By the return (sum of all rewards) from the whole episode. A lot of decisions made in "good" episodes do not lead to immediate rewards, but instead transition towards states where better rewards are possible.
In retrospect, you do not know whether any single action was a good choice, but with the cross entropy method (CEM) you rely on the fact that on average the better episodes will contain more good decisions than the worse episodes, so you train the policy neural network on the state, action pairs from the elite episodes as if all the decisions were correct. This will not be true, but will hopefully be true more often than by chance, so the policy should improve.
This can be a noisy, high variance approach with any RL method. CEM is one of the most sensitive to noise and variance. However, the taxi environment is deterministic, and that makes using CEM more reasonable.

So, I should select all the state-action pairs in each episode with a total_reward bigger than the treshold. Am I ok?

@HermesMorales Yes. If an episode is "elite", then all the state/action pairs in it should be considered good for training data.


text:
Is there any comprehensive book that reviews topics in the area of brain-inspired computing?
I am looking to write my master's thesis next year about brain-inspired computing. Hence, I am looking to get a good overview of this domain.
Do you know of any comprehensive book that reviews topics in the area of brain-inspired computing (such as spiking neural networks)?
In spirit and scope, it should be similar to Ian Goodfellow's book deep learning.

The most popular theoretical framework in use currently, in the neuromorphic (brain-inspired) computing community is the Neural Engineering Framework (NEF). Neural Engineering by Chris Eliasmith and Charles Anderson explains the framework comprehensively.
As a follow up to that, How to Build a Brain by Chris Eliasmith describes the more recent and more high-level description of how to get spiking neural networks to actually perform multiple functions: the Semantic Pointer Architecture (SPA).
If you're looking for hardware descriptions too, the research publications on the chips Neurogrid, Brainscales, Braindrop, SpiNNaker, Loihi, TrueNorth etc. provide some good high-level descriptions of how to actually build the aforementioned chips.

Hello. Welcome to Artificial Intelligence SE. Thanks for trying to contribute to our site! It may be a good idea to provide links to the cited books or their pdfs, if available.


text:
Why would the "improvement" be the result of random initialization, and so why should we use multiple runs?
I got this feedback for my thesis paper.

The improvement shown in the results section could be the result of random initialization. There should be multiple runs with means and standard deviations.

Can anyone explain this feedback with details?
I used a neural network with pre-trained weights for transfer learning (specifically, EfficientNetB0, with 'noisy-student' for the weights). It was a classification problem to classify between Covid-19, Viral Pneumonia, and normal cases. I normalised the dataset so that the images are in the range [0, 255] and I also did k-fold cross-validation.

Neural networks use random number generators in multiple places. Most notably for weight initialisation, but also for features such as dropout, selecting minibatches within epochs, and train/cv split for cross-validation.
That means that any result metric from the neural network e.g. accuracy, loss, F1 score, is a random variable.
Reporting a single value of a random variable is not very informative. It may be higher or lower than expected, and it is difficult to tell if a result is significant.
Your reviewers are asking you to run the entire training and reporting from your dataset multiple times, to get the mean and standard deviation of any metrics that you reported.
You can keep the same train/test split for your hold-out test data set. Ideally this is the same train/test split as used by the classifiers that you are comparing your thesis results with.
If you used a seed for the RNG for repeatability, you can keep that as is and perform multiple training and reporting runs within a single script starting with the same seed set at the very beginning. Alternatively, if this would take too long, you could generate a set of seeds to use in consecutive runs - provided you are not selecting seed values depending on the results it does not matter.

Thank you so much Sir for your detailed answer. It helps me a lot.


text:
Can people use neural networks without providing the set of training data?
It seems that neural networks (NNs) can be applied to supervised learning, unsupervised learning and reinforcement learning. Some people even train neural networks without the set of training data. If NNs are used in reinforcement learning, is it possible that we don't need training data?

Neural networks are trained by using pairs of example input/output vectors that they learn to associate and can generalise from. In that sense, they always need training data.
For supervised learning, a neural network (NN) is trained on a dataset of example inputs and outputs (aka "a labelled dataset") that the user must provide somehow.
There are scenarios involving neural networks that do not require the user to possess a labelled dataset, or even any dataset at all, but they all have some restriction:

The NN is already trained, and is now being used to make predictions. You do not need to have access to the original dataset to use a trained NN, although you do need access to some inputs that are of the same type that the network was trained on. The restriction here is that the NN cannot be trained further.

Semi-supervised learning: In some cases, the desired outputs can be generated automatically from the inputs (they may even be same as the inputs). You still need a dataset of inputs, but may be spared the hard work of adding labels, making it a lot easier to collect a dataset. The restriction here is that this is for specific use cases, such as Generative Adversarial Networks, and is not an approach you can use in general.

Reinforcement learning (RL). The labelled data for RL are generated through a trial and error process, and do not need to be provided separately by the user. However, the user does need to write the RL code for the environment to allow this data generation to happen. Internally in systems like Deep Q Networks (DQN), the training process looks a lot like supervised learning.

Both RL and semi-supervised learning are special cases of auto-generation of datasets, where the NN is being used to learn a complex function that can be calculated in some other way. As well as the semi-supervised and reinforcement learning cases, NNs have successfully been applied to fluid dynamics and ray tracing problems in this way, where the CPU cost for full calculation is even higher than using a neural network. These scenarios don't require you to possess a labelled dataset before training starts, but do require effort in developing something that generates input/output pairs.

You're right. My question should be "Is it possible to train a NN with only input? But this NN can predict output after the training process is done". Is there any easy to learn reference, website or hands on experiment on training NN with only input?

@JianqiaoHuang "Training a neural network with only one input" would be one-shot learning.

You cannot train a neural network without training data. It would be like training a football player without making him/her play/watch football or anything that resembles football: it's simply not possible. The definition of training/learning in machine learning strictly requires data.
You can train a neural network in different ways (e.g. supervised or unsupervised) and with different types of data (e.g. labelled or unlabelled, respectively), but this is a different story. In reinforcement learning, you also have training data, but the data may not be given to the neural network in the same way that it's given e.g. in supervised learning. Still, this does not mean that there is no training data. Of course, there is or must be (by definition)!
However, note that you can use a (e.g. randomly initialised) neural network without training it, but it would probably be a useless neural network. You could also use a neural network that has been trained by someone else with data that you may not have access to anymore (and maybe that answers your question in the title).

I am wrong. In fact, I am supposed to say that some people train NN with only input data, but without output data.

@JianqiaoHuang Yes, that's known as unsupervised learning.

In case the question is if NNs can be trained without data, as pointed by others, the answer is negative - any training by definition involves the use of data in some way - supervised, semi-supervised, reward, etc.
However, if the question is whether one can obtain something useful I would think about the following use cases:

One can use randomly initialized networks as a random map. The application of this seems to be rather specific, but maybe there are some applications of this.
One can add certain evolution to the weights like in the statistical physics system of form:
$$
w_{n+1} - w_n = f(w)
$$
Where $f(w)$ can be deterministic or non-deterministic. It is not actually a neural network, but a related concept - https://en.wikipedia.org/wiki/Boltzmann_machine.


text:
Could someone help tell what the labels are pointed out by red rectangles?
The following figure comes from the paper The perceptron: A probabilistic model for information storage and organization in the brain

I can tell the labels pointed out by blue rectangles are: "Projection area", "A-units", "$R_1$", "inhibitory connections" and "$R_2$".
Could someone help tell what the labels are pointed out by red rectangles?

Left-to right:
red: SENSORY RECEPTOR OR
blue: PROJECTION AREA
blue: A-UNITS
blue: R$_1$
red: BROKEN LINES SHOW
blue: INHIBITORY CONNECTIONS
blue: R$_2$


text:
How can we find the value function by solving a system of linear equations?
I am following the book "Reinforcement Learning: An Introduction" by Richard Sutton and Andrew Barto, and they give an example of a problem for which the value function can be computed explicitly by solving a system of $\lvert S \rvert $ equations that have $\lvert S \rvert $ unknowns. Each of these $\lvert S \rvert$ equations is given by:
$$v_{\pi}(s) = \sum_{a} \pi(a\rvert s) \sum_{s^{\prime}}\sum_{r} p(s^{\prime}, r \rvert s,a)[r + \gamma v_{\pi}(s^{\prime})] $$
I am having a hard time understanding how one could solve this system of equations. It seems to me as if each equation consists of a summation of an infinite amount of terms and therefore one would not be able to analytically solve them. Could anyone offer any intuition as to how this system of equations could be explicitly solved?

First of all, we assume that we have a finite MDP, i.e. the set of states $\mathcal{S}$, the set of actions $\mathcal{A}$ and the set of rewards $\mathcal{R}$ all have a finite number of elements (I didn't think about how the explanations below would extend to other cases, but I suspect you will need differential equations).
For simplicity, let's only consider the value function $v$ (as opposed to the state-action value function $q(s, a)$, but this also applies to $q$). The value function $v$ is defined for all states, i.e. it's a function of the form $v : \mathcal{S} \rightarrow \mathbb{R}$ or, with an alternative notation, $v(s), \forall s \in \mathcal{S}$. So, we can define this function as a vector $\mathbf{v}$ of dimension $|\mathcal{S}| = n$, i.e. $\mathbf{v} \in \mathbb{R}^{|\mathcal{S}|}$, where the $i$th element contains the value of the $i$th state (so we need a function that maps states to indices of this vector, but this is trivial).
The fact that you can represent the value function, in this finite MDP, as a vector should already suggest that you can find this value function by solving a linear system of equations.
However, let me show that by starting with the definition of the value function you also provided
\begin{align}
v_{\pi}(s)
&=
\sum_{a} \pi(a\rvert s) \sum_{s^{\prime}}\sum_{r} p(s^{\prime}, r \rvert s,a)[r + \gamma v_{\pi}(s^{\prime})] \label{1}\tag{1}, \; \forall s \in \mathcal{S}
\end{align}
which can be expanded as follows
\begin{align}
v_{\pi}(s)
&=
\sum_{a} \pi(a\rvert s) \sum_{s^{\prime}}\sum_{r} \left[p(s^{\prime}, r \rvert s,a)r + \gamma p(s^{\prime}, r \rvert s,a) v_{\pi}(s^{\prime}) \right] \\
&=
\sum_{a} \pi(a\rvert s)  \left[ \sum_{r} \underbrace{\sum_{s^{\prime}} p(s^{\prime}, r \rvert s,a)}_{\text{Marginalization of }p \text{ over } s'}r + \gamma \sum_{s^{\prime}} \underbrace{\sum_{r} p(s^{\prime}, r \rvert s,a) }_{\text{Marginalization of }p \text{ over } r} v_{\pi}(s^{\prime}) \right]\\
&=
\sum_{a} \pi(a\rvert s)  \left[ \sum_{r} p(r \rvert s, a)r  + \gamma \sum_{s^{\prime}}p(s^{\prime} \rvert s,a) v_{\pi}(s^{\prime}) \right] \\
&=
\sum_{a} \pi(a\rvert s)  \left[ r(s, a)  + \gamma \sum_{s^{\prime}}p(s^{\prime} \rvert s,a) v_{\pi}(s^{\prime}) \right] \label{2}\tag{2}, \; \forall s \in \mathcal{S}
\end{align}
where

$\sum_{r} p(r \rvert s, a)r = r(s, a)$ (see this).
$\sum_{s^{\prime}}p(s^{\prime}, r \rvert s,a) = p(r \rvert s, a)$ (marginalization)
$\sum_{r} p(s^{\prime}, r \rvert s,a) =p(s^{\prime} \rvert s,a) $ (marginalization)

In this form, as in equation \ref{2}, the value function can also be written in a different notation
\begin{align}
v_{\pi}(s)
&=
\sum_{a} \pi(a\rvert s)  \left[ R_{s}^a  + \gamma \sum_{s^{\prime}}P_{ss'}^a v_{\pi}(s^{\prime}) \right] \label{3}\tag{3}, \; \forall s \in \mathcal{S}
\end{align}
where

$R_{s}^a = r(s, a)$
$P_{ss'}^a = p(s^{\prime} \rvert s,a)$

We can still write equation \ref{3} in a "simpler" form as follows
\begin{align}
v_{\pi}(s)
&=
\sum_{a} \pi(a\rvert s) R_{s}^a  + \gamma \sum_{s^{\prime}} \sum_{a} \pi(a\rvert s) P_{ss'}^a v_{\pi}(s^{\prime}) \\
&=
R_{s}^\pi + \gamma \sum_{s^{\prime}} P_{ss'}^\pi v_{\pi}(s^{\prime})
\label{4}\tag{4}, \; \forall s \in \mathcal{S}
\end{align}
where

$\sum_{a} \pi(a\rvert s) R_{s}^a = R_{s}^\pi$
$\sum_{a}  \pi(a\rvert s) P_{ss'}^a = P_{ss'}^\pi$

We can write the definition of the value function in \ref{4} in matrix form for all states $s \in \mathcal{S}$ as follows
\begin{align}
\begin{bmatrix}
v_\pi(1) \\
\vdots \\
v_\pi(n)
\end{bmatrix}=
\begin{bmatrix}
{R}_1^\pi \\
\vdots \\
{R}_n^\pi
\end{bmatrix}
+\gamma
\begin{bmatrix}
{P}_{11}^\pi & \dots & {P}_{1n}^\pi\\
\vdots & \ddots & \vdots\\
{P}_{n1}^\pi & \dots & {P}_{nn}^\pi
\end{bmatrix}
\begin{bmatrix}
v_\pi(1) \\
\vdots \\
v_\pi(n)
\end{bmatrix}
\tag{5}\label{5},
\end{align}
which can be written in a more compact form as follows
\begin{align}
\mathbf{v} = \mathbf{r} + \gamma \mathbf{P}\mathbf{v} \tag{6}\label{6},
\end{align}
which is a very compact form of the Bellman equation (which is a recursive equation: as you can notice, the $\mathbf{v}$ appears on the left and right of the equals sign) that represents the value function (i.e. the value function can be defined as a recursive equation).
In equation \ref{5}, the unknowns are the $|\mathcal{S}| = n$ values of the value function $v$ and there are $n$ equations, so it should now be clear why we can solve this problem by solving a system of equations. Note that here it's assumed that $\pi$, $r(s, a)$ and $p$  are given and known, which, generally, is not the case, that's why we use algorithms like Q-learning.

Provided you have a finite number of states and actions, then there will not be an infinite number of terms. Therefore the state and action spaces need to be discrete and finite before the quote from the book applies.

I am having a hard time understanding how one could solve this system of equations.

There are a few techniques for solving simulteneous equations.
However, what I would probably do is number all the state values from $v_1 = v_\pi(s_1)$ to $v_{N = |\mathcal{S}|} = v_\pi(s_N)$, and write out each line in order:
$$v_1 = w_{1,1} v_1 + w_{1,2} v_2 + w_{1,3} v_3 + ... w_{1,N} v_N + r_1$$
Where $r_1$ is a constant - it is the expected immediate reward when starting from state $1$, but that is not important. It is the constant offset value you get from resolving the sum that is not multiplied by any $v_i$ unknown variable.
You can discover the values of $w_{i,j}$ by expanding the sum in the Bellman equation for each state in turn.
At that point you can build a matrix of the weights, and solve the linear equations by taking the inverse of the matrix.

[from comments] But if the game has no end then theoretically the sum of expected future rewards should be infinite.

The time series definition of $v_{\pi}(s)$:
$$v_{\pi}(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t = s]$$
does not appear in the Bellman equation used to establish the linear equations. This is the main benefit of the Bellman equation, it changes the infinite series view of returns into a set of relations that must hold between the value functions of each state.

But if the game has no end then theoretically the sum of expected future rewards should be infinite. I understand the we use a discount factor gamma to make this tractable, but in essence this is essentially an infinite sum? Or am I misunderstanding something?

@phil: The Bellman equation does not have any time series in it. That is the point of it, it relates values over single steps. I will try to add that to the answer later


text:
How to define a continuous action distribution with a specific range for Reinforcement Learning?
Specifically for continuous control PPO, let's say my action space range is between  $X$ (low) and $Y$ (high) and they are all sampled from a Gaussian Action Distribution with mean $\mu$ and standard deviation $\rho$.
From what I understood, the actions sampled should fall between $\mu - \rho$ and $\mu + \rho$, but that's not what happens in practice? What am I misunderstanding here? How do I ensure this range constraint from a custom action distribution with a given mean and standard deviation?
Any advice or tips for me? I would really appreciate any insights!

First of all, the support of a normal distribution is the entire real line (or, in general, $\mathbb{R}^n$ for an $n$-dimensional multivariate normal distribution) so your action can be any number in $\mathbb{R}$. What you may be getting confused with is that with probability 0.68 you will obtain an action that is within +/- 1 standard deviation from the mean.
Now, to answer the question of how you can do this using RL:
To use the Normal distribution in this setting I would simply clip my actions in the environment. If, for example, the actor gives an action below your minimum value $X$, lets say it gives $X-0.5$, then I would simply clip the action to be $X$ when executed in the environment. This way your actor can still sample from a normal distribution which could give answers below $X$ (or above $Y$) and be used with your environment.
If, for instance, your desired range was $(-1, 1)$ then another option would be to define your distribution to be $Y = \mbox{tanh}(X)$, where $X \sim N(\mu, \sigma)$. You can then find the density function of $Y$ using e.g. the density transformation method.

I see, thank you so much for clearing that about the normal distribution. From what I understand, the normal distribution is just a special case of the Gaussian distribution? Is that correct?
What if, for example, the output of the actor leads to an action range that is much smaller than the actual range of actions. That is, what if the range of output is [-2, 2] but the actual action range is [-5, 5]. Is expanding the output (unsquashing it) into the original action range of [-5,5] a valid thing to do?

the normal distribution is the Gaussian distribution, it is just a different name that people use :) in that instance, I would recommend to do some kind of normalisation so that the actors range equals the actual environments range. If the actual environment range is symmetric, i.e. $[-x, x]$ then you could use a tanh activation function for the final layer of the network and multiply the output by $x$.

Yeah, sadly my action range isn't symmetric! But thanks for the help!

you could try shifting you action range to start from 0, i.e. if the range is $[x, y]$ then make the range $[0, y-x]$ and then you just need to use a relu activation and clip at $y-x$, or use a sigmoid and multiply by $y-x$.


text:
Where can I find the original conference paper that introduced Q-learning and Deep Q-Learning?
I tried searching a lot, but I could neither find the paper that introduced Q-Learning nor the paper that introduced Deep Q Learning. If anyone knows anything about it please do tell me.

It's very strange that you didn't find the papers that introduced Q-learning and DQN. You just needed to type "Deep Q-learning paper" on Google to find it immediately as the first returned search result.

Yes, I agree it is strange. I didn't find the original Q learning or DQN paper at all. I found a few papers introducing modifications/applications to these algorithms but not the absolute original one.

This is the original Q-Learning paper by Watkins, though you may need to pay for access to this.
This is the Nature paper that introduced the DQN.


text:
TD3 sticking to end values
I am using TD3 on a custom gym environment, but the problem is that the action values stick to the end. Sticking to the end values makes reward negative, to be positive it must find action values somewhere in the mid. But, the agent doesn't learn that and keeps action values to maximum.
I am using one step termination environment (environment needs actions once for each episode).
How can I improve my model? I want action values to be roughly within 80% of maximum values.
In DDPG, we have inverted gradients, but could something similar be applied to TD3 to make action values search within legal action space more?
The score decreases as episodes increases.

could you please elaborate more about the solution? I am facing the same problem, and I am unsure how to solve it.

What do you mean by "the action values stick to the end"?  Can you also explain why "Sticking to the end values makes reward negative" and "to be positive it must find action values somewhere in the mid" are true?

I found the solution, it was changing reward function and using reward scaling. A little bit change in architecture and learning rate fixed the problem.


text:
Is it possible that the fine-tuned pre-trained model performs worse than the original pre-trained model?
I have downloaded a pre-trained EfficientDet D2 model (Tensorflow 2.0) and trained it on some data (about 20000 images with 20 classes). I set the number of steps to 25000 and batch size to 3 (computer resources are not the best).
However, if I try to make predictions, the pre-trained model makes better predictions than the model I have trained on the additional data. Is this expected behaviour?
For example, an image of a person may be 78% accurate on the pre-trained model and only 54% accurate on the same image when trained.

Hello. Welcome to AI SE! I changed the title of your post to put there what I think is your question. Please, make sure the current version of your post is still consistent with your original post.

Yes, this is quite the expected behavior. The main difference between expected and current behavior lies in the amount of data you are using for training VS the amount of data that the pre-trained model was trained with.
Take into account that pre-trained models have been trained over popular datasets, the most common ones are: COCO, ImageNet and Open Images. And the amount of data differs:

COCO: 330K images
ImageNet: 1.5M images
OpenImages: 9M images
Your dataset: 20K images

You could say, well, but I started from a pre-trained model, so the network should already know how to detect a person. Well that is true, but you are training it again, you are not using transfer learning (freezing backbone layers, or adding extra heads / extra channels to detect other features or other classes). So what is happening is that your model, even when it started from good pre-trained weights, is fitting to your 20K images data.
As I see it, you have two options: either increase your dataset size or use transfer leanring.

Thank you for great answer. Do you know how to freeze the layers of a model using TFOD? That is, how do I know which layers to freeze based on the name of them?

I am using pytorch for two years now, but last time I checked you needed to iterate the model layers and set the target layers to trainable=false


text:
Does BERT freeze the entire model body when it does fine-tuning?
Recently, I came across the BERT model. I did some research and tried some implementations.
I wanted to tackle a NER task, so I chose the BertForSequenceClassifications provided by HuggingFace.
for epoch in range(1, args.epochs + 1):
    total_loss = 0
    model.train()
    for step, batch in enumerate(train_loader):
        b_input_ids = batch[0].to(device)
        b_input_mask = batch[1].to(device)
        b_labels = batch[2].to(device)
        model.zero_grad()

        outputs = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)
        loss = outputs[0]

        total_loss += loss.item()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        # modified based on their gradients, the learning rate, etc.
        optimizer.step()

The main part of my fine-tuning follows as above.
I am curious about to what extent the fine-tuning alters the model. Does it freeze the weights that have been provided by the pre-trained model and only alter the top classification layer, or does it change the hidden layers that are contained in the already pre-trained BERT model?

Taken directly from HuggingFace

Note that if you are used to freezing the body of your pretrained model (like in computer vision) the above may seem a bit strange, as we are directly fine-tuning the whole model without taking any precaution. It actually works better this way for Transformers model (so this is not an oversight on our side). If you’re not familiar with what “freezing the body” of the model means, forget you read this paragraph.


text:
How is the convolution operation connected to neural networks?
I've been reading up on the convolution operation and neural networks. I understand that the convolution operation is defined as:
$$(f * g)(t)=\int_{-\infty}^{\infty} f(\tau) g(t-\tau) d \tau$$
The convolution operation has some properties, such as commutativity, associativity, etc.
How is the convolution connected to neural networks? How do we use this operation in a CNN?

The convolution operation performed by most CNNs that you will find (on the web) assumes that the signals/functions are discrete and 2-dimensional (e.g. images can be viewed as 2-dimensional discrete signals), although this does not have to be the case. In fact, 1D and 3D convolutions are also implemented in several deep learning libraries (see here for an example).
The equation in your post describes the convolution operation for 1-dimensional continuous signals with domain $\mathbb{R}$ (sometimes called the time domain, hence the usage of the letter $t$), so we use an integral that goes from $-\infty$ to $\infty$.
An equation that better describes the common 2D convolution operation performed by CNNs is the following
$$S(i, j)=(K * I)(i, j)=\sum_{m=-M}^M \sum_{n=-N}^N I(i+m, j+n) K(m, n), \tag{1}\label{1}$$
where

$S(i, j)$ is the convolution of the signals $I$ (e.g. an image) and $K$ (e.g. the kernel) evaluated at the coordinates (or points of the domain) $(i, j)$, so $S$ is the function that results from the convolution of $I$ and $K$
$(K * I)(i, j)$ is just another notation for $S(i, j)$, which emphasizes that the convolution is performed between $K$ and $I$, and $*$ is the convolution operation
$K(m, n)$ is the value of the function $K$ (in the context of neural networks and image processing, this is often known as the kernel, hence the $K$, or filter) evaluated at $(m, n)$
similarly, $I(i+m, j+n)$ is the value of the function $I$ (e.g. the image) evaluated at $(i+m, j+n)$

So, essentially, equation \ref{1} is telling us that the value of the convolution between $I$ and $K$, denoted by $S$, evaluated at $(i, j)$ is a sum of products, where the factors are $I(i+m, j+n)$ and $K(m, n)$, where the values of $m$ and $n$ depend on $M$ and $N$, which could also be equal, and typical values would be, for example, $1$ or $2$, so equation \ref{1} could be written as follows
$$S(i, j)=(K * I)(i, j)=\sum_{m=-1}^1 \sum_{n=-1}^1 I(i+m, j+n) K(m, n), \tag{2}\label{2}$$
If you expand the summations, you will get
$$S(i, j)=
I(i-1, j-1) K(-1, -1) + I(i-1, j) K(-1, 0) + \dots + I(i+1, j+1) K(1, 1)$$
This is essentially a 2D dot product or, equivalently, an element-wise multiplication followed by the addition of those multiplications.
Another important thing to note is that the convolution of $I$ and $K$ evaluated at $(i, j)$ uses the values of $I$ with respect to the points $(i, j)$.
Moreover, in a 2D convolutional layer of a CNN, we do not just compute $S(i, j)$, but we compute $S(i, j)$ for all possible values of $i$ and $j$, which are typically the coordinates of the image (in the case of the first layer).
Finally, the equation \ref{2} actually describes what is known as the cross-correlation operation, which is exactly the same as the convolution operation, but we use $+$ instead of $-$ in $I(i+m, j+n)$. In practice, in the context of CNNs, it doesn't really matter whether you use the $+$ and $-$, given that the kernels $K$ are learned. However, the convolution operation and the cross-correlation would be different in the case the kernels are not symmetric, but identical in the case they are. You can read more about this topic in this answer that I had written a while ago.
You should probably also read this chapter that more thoroughly describes all these operations. There are also other answers that I had written in the past that could be useful, such as this, this, this, this, and/or this. By the time you have understood all these answers, you will also have understood the convolution operation and how it is related to CNNs.


text:
Is the dropout technique specific only to neural networks?
In one Udemy course was mentioned that "dropout is unique to neural networks". However, I remember an example of decision trees where nodes that are not participating in the overall result are removed, and I think that this technique is also called "dropout". Am I correct?

I'm sure you can use dropout in any parameterized model, but I suspect it'll only really be helpful if you have enough parameters/nodes. Also dropout in neural nets has a Bayesian meaning, Yarin Gal for example has done lots of work on this.
In your decision tree example, I believe you're talking about pruning, which is different. In that context you're removing nodes that you know aren't contributing. In dropout, you randomly turn off nodes during training in order to prevent individual nodes from being too influential, but the nodes are never removed.
You might also be interested in L1 regularization in parameterized models. This is when you add a penalty according to the absolute weights (rather than square weights), which tends to drive less useful weights to 0. Then you can remove the nodes with almost no weight. This is more akin to your decision tree example rather than dropout though.

It may be also worth mentioning "pruning" in neural networks, as the OP may come across it too.

Ah good point, I forgot about that. Admittedly I don't know much about neural net pruning, I would've guessed it's similar to l1.

xgboost (a decision-tree based algorithm) does use random feature selection and random row selection for different trees, which is more similar to dropout than pruning.

When Random Forest is constructed features of each individual tree are selected at Random, (or in other words dropped out at random), this is done so that the overall model is not too dependent on one feature. This is pretty similar to dropout in Neural Networks.


text:
Why is AI Super Resolution Reconstruction more than just guessing?
I saw a video on Youtube about AI and Super Resolution Image Reconstruction with TecoGAN. I must say I am impressed.
Now, I am wondering how reliable this is.
I have learned at university that you lose information if you do not sample to fullfill Nyquist. I also don't think that the example images are in any way sparse...
Is the AI just trying to fill in the blanks by guessing?
This would be fine for entertainment, but probably not so much to enhance robbery pictures and charge people based on enhanced pictures. It also wouldn't be a good solution for improving the resolution of scientific data if it is just "guessing".

Yes, it's guessing. In the training phase, you show it lots of coarse and detailed pictures, and the algorithm learns a mapping from course to detailed. Then you present it a new coarse image, and it executes the same mapping. The information from the original picture is gone, and it cannot be retrieved, so it's filled in by analogy to other cases.
"Guessing" sounds a bit random, so it's more like a very informed guess. A bit like reading lots of books, and then being asked what word comes after "the cat sat on the" -- you're likely to say "mat", and will be right in many cases, but there's no guarantee that the most common word actually does occur. So now just substitute words with pixel values, and add a complex statistical model to make the decision, but you still won't know what the correct element is.
As you rightly say, this is fine for entertainment, but not for serious applications, where missing details in a crime scene are filled in according to how previous similar scenes may have looked.

You were correct in that the model won't be able to reconstruct any missing information with complete certainty (an intuitively impossible task). As Oliver Mason mentioned, it is estimating what a similar image (from the training data it has been exposed to) would have looked like (I should note that in the vast majority of cases, we don't/can't actually know what the model's internal representation looks like or how it works, only how to get there - see explainable AI).
In other words, it attempts to match the image it is given to a "latent space" (different terms are sometimes used for the same concept, but  you may find the links in the article helpful - also see this question) that it has approximated (representing the data it was trained on) and mapped to the output space (i.e., the "high-resolution" images).
In theory it is possible that AI could be used to enhance certain features of crime scene images (e.g., sharpening the contours of a face so a detective can better recognize the person) but again, this only approximates the "true" data. In practical settings, it is more likely that the raw data is fed into a model designed specifically for achieving some end goal (for example, facial recognition) and humans are taken out of the loop once a sufficiently accurate model is available.
Generally, "highly educated guessing" is an appropriate summary for this type of method and you are right in your assumption that this limits its usefulness in certain applications where precision is important. Though this is somewhat tangential, you might find machine learning-based computational fluid dynamics (e.g., this paper) enlightening; while most machine learning techniques (anything involving neural networks, which are inherently approximators and also often nondeterministic) cannot produce scientifically rigorous simulations in the way traditional algorithms might, they can speed up some of the processing/analysis in the way a cleverly devised heuristic might.
Here are a couple articles you might find informative (some of these are pretty sparse in information relating to machine learning, but most have links to in-depth treatments):

https://en.wikipedia.org/wiki/Autoencoder
https://en.wikipedia.org/wiki/Feature_learning
https://en.wikipedia.org/wiki/Noise_reduction#Deep_learning
https://en.wikipedia.org/wiki/Deep_Image_Prior
https://en.wikipedia.org/wiki/Intrinsic_dimension#Applications

You actually don't have to loose information if you don't fulfill Nyquist — although that topic is quite advanced and has limitations. Still, super resolution is reliable and used by most 4K TVs today to upscale 1080p video to fit the 4K screen. You may notice TV ads for 4K TVs occasionally mentioning this.
What super resolution does is just generalising shapes. For example, imagine a simple image with just a black rectangle. You can easily image enlargening that image to whatever size you want because you know how it's supposed to look. By training a neural network on a lot of images it can learn to generalise features such as faces and enlarge them.
Of course you can't take a single pixel of a face and expand this into a full 100x100 image. Therefore, it's best to use super resolution to enlarge entire images, not just individual areas of the image. If you were to use it to enlarge a very small patch of text in the image, it may not replicate the text correctly and read as something else.
Furthermore, very good super resolution models are slow. Most also only increase the resolution of a single frame at a time. This means there might be inconstancies between sequential frames in a video.


text:
Which policy has to be followed by a player while construction of its own Q-table?
Consider the scenario, where there are two players. One of the players perform the action randomly, whereas I want second player as a Q-player. I mean, the player selects a best action from the Q-table for given state i.e., the action with maximum Q-value. So, in case of second player, a Q-table is required.
It is known that Q-table has to be constructed only by running several episodes after its arbitrary initialization. So, the two players has to play with some policy to construct a Q-table for player two. Since the first player uses random policy. I have doubt regarding the policy of second player for constructing Q-table.
I have doubt regarding the policy of second player only. I know the policy he need to follow after the completion of updation of Q-table. I am not sure about the policy he need to follow while updating the Q-table only.
Which policy does my second player need to follow while constructing the Q-table for himself? Can he use random policy like player one? Or does he need to use arbitrarily initialized or partially updated Q-table itself for selecting best action? Or can he use some other policy till the completion of updating Q-table?

Comments are not for extended discussion; this conversation has been moved to chat.

I'll assume Q-player is being trained with Q learning (note, Q tables can be useful in other algorithms too, like SARSA).
Q learning is an off policy algorithm, meaning that the Q values can be learned regardless of the policy used to collect data. So the Q player can be following a random policy, or even a fixed pre defined policy if you want. Usually, however, before the Q table has converged, we want the agent to explore the environment to collect the data it needs (without sacrificing too much return, this is the exploration-exploitation tradeoff).
It is very common practice to use the intermediate Q tables to induce a policy for the agent. For example, the epsilon-greedy policy will select the best action according to the current Q table with high probability, and will otherwise choose a random action.


text:
Are the held-out datasets used for testing, validation or both?
I came across a new term "held-out corpora" and I confused regarding its usage in the NLP domain
Consider the following three paragraphs from N-gram Language Models
#1: held-out corpora as a non-train data

For an intrinsic evaluation of a language model we need a test set. As
with many of the statistical models in our field, the probabilities of
an $n-$gram model come from the corpus it is trained on, the training
set or training corpus. We can then measure training set the quality
of an n-gram model by its performance on some unseen data called the
test set or test corpus.  We will also sometimes call test sets and
other datasets that are not in our training sets held out corpora
because we hold them out from the held out training data.

This paragraph clearly says that held-out corpora can be used for either testing or validation or others except training.
#2: development set or devset for hyperparameter tuning

Sometimes we use a particular test set so often that we implicitly
tune to its characteristics. We then need a fresh test set that is
truly unseen. In such cases, we call the initial test set the
development test set or,devset. How do we divide our data into
training, development, and test sets? We want our test set to be as
large as possible, since a small test set may be accidentally
unrepresentative, but we also want as much training data as possible.
At the minimum, we would want to pick the smallest test set that gives
us enough statistical power to measure a statistically significant
difference between two potential models. In practice, we often just
divide our data into 80% training, 10% development, and 10% test.
Given a large corpus that we want to divide into training and test,
test data can either be taken from some continuous sequence of text
inside the corpus, or we can remove smaller “stripes” of text from
randomly selected parts of our corpus and combine them into a test
set.

This paragraph clearly says that development set is used for hyperparameter tuning.
#3: held-out corpora for hyperparameter tuning

How are these $\lambda$ values set? Both the simple interpolation and
conditional interpolation $\lambda'$s are learned from a held-out
corpus.  A held-out corpus is an additional training corpus that we
use to set hyperparameters like these $\lambda$ values, by choosing
the $\lambda$ values that maximize the likelihood of the held-out
corpus.

This paragraph is clearly saying that held-out corpus is used for hyper-parameter training.
I am interpreting or understanding the terms as follows:
Train corpus is used to train the model for learning parameters.
Test corpus is used for evaluating the model wrt parameters.
Development set is used for evaluating the model wrt hyperparameters.
Held-out corpus includes any corpus outside training corpus. So, it can be used for evaluating either parameters or hyperparameters.
To be concise, informally, data = training data + held-out data = training data + development set + test data
Is my understanding true? I got confusion because of paragraph 3, which says that held-out corpus is used (only) for learning the hyperparameters while paragraph 1 says that held-out corpus includes any corpus outside train corpus. Does held-out corpora include devset or same as devset?

I think that these terms may be used inconsistently across sources.
If someone says held-out dataset, I would immediately think of a dataset that is not used for training, but can be used for anything else, validation (hyper-parameter tuning or early stopping) or testing; so, to determine what they are referring to, I would probably take into account the context.
In your second quote, the development set seems to be used as a synonym for validation dataset (a more common name to refer to the same concept), i.e. the dataset used for early stopping or hyper-parameter optimization (see also this).
So, my answer to your question in the title would be

Yes, the heldout dataset can be used for validation or testing, but not because it's a special dataset, but because people may use this term to refer to either the validation dataset or the test dataset.

Here's another example of the usage of the term held-out set to refer, in this case, to the validation dataset (section 1.6 of this famous ML book)

The results above suggest a simple way of achieving this, namely by taking the available data and partitioning it into a training set, used to determine the coefficients $\mathbf{w}$, and a separate validation set, also called a hold-out set, used to optimize the model complexity

Here's another example that shows that the term may be used inconsistently (emphasis mine, taken from section 5.3 of the famous deep learning book by Goodfellow et al.). In fact, in that same section, they refer to the validation dataset, which is distinct from this held-out test set (so, in this case, the held-out set is used to refer only to the test set).

Earlier we discussed how a held-out test set, composed of examples coming from the same distribution as the training set, can be used to estimate the generalization error of a learner, after the learning process has completed.

In the sample dataset, the variable Sprint is the respondent's time (in seconds) to sprint a given distance, and Smoking is an indicator about whether or not the respondent smokes (0 = Nonsmoker, 1 = Past smoker, 2 = Current smoker). Use ANOVA to test if there is a statistically significant difference in sprint time with respect to smoking status. Sprint time will serve as the dependent variable, and smoking status will act as the independent variable.


text:
What do the terms 'Bellman backup' and 'Bellman error' mean?
Some RL literature use terms such as: 'Bellman backup' and 'Bellman error'. What do these terms refer to?

There's already an answer that addresses both concerns/questions, but, please, next time, focus on one question per post, although, in this case, the terms are highly related (but I still think these "simple" questions could have been asked in separate posts). It may also be a good idea to provide more context (e.g. a link to an article that mentions these terms), although, again, in this case, anyone familiar with RL would be able to understand the question.

A Bellman backup is an application of a Bellman operator. For example, the step
$$
V(x)\leftarrow \alpha(R + \mathbf{E}[V(x')]) + (1-\alpha)V(x)
$$
Is a Bellman backup for some learning rate $\alpha$.
A Bellman error is
$$
d(V(x), R + \mathbf{E}[V(x')])
$$
for some metric $d$, usually $d(x, y) = (x-y)^2$.

What does 'backup' refer to here?

It refers to propagating information from later states to earlier ones (backward in time sorta)

It may be a good idea to 1. also provide the figures of the backups e.g. that you can find in Sutton and Barto's book, 2. to link the OP to this question about what the Bellman operator is and 3. explain the symbols in your answer.


text:
In value iteration, what happens if we try to obtain the greedy policy while looping through the states?
I am referring to the Value Iteration (VI) algorithm as mentioned in Sutton's book below.

Rather than getting the greedy deterministic policy after VI converges, what happens if we try to obtain the greedy policy while looping through the states (i.e. using the argmax equation inside the loop)? Once our $\Delta < \theta$ and we break out of the loop, do we have an optimal policy from the family of optimal policies? Is this a valid thing to do?
I implemented the gambler's problem exercise mentioned in Sutton's book. The policies obtained after using standard VI and the method I described above are mostly similar, yet different for some states.

Are you suggesting to replace all instances of $V(s)$ with $\pi(s)$, including an update rule $\pi(s) \leftarrow \text{argmax}a \sum{s',r} p(s',r|s,a)[r +\gamma \pi(s')]$ ?

No. After we get the new $V(s)$, we calculate $\pi(s) \leftarrow \text{argmax}a \sum{s',r} p(s',r|s,a)[r +\gamma V(s')]$, rather than getting optimal $\pi(s)$ outside the loop.

Ah, OK. That is simpler

Will that have any effect? Is it a valid thing to do?

Hello. Please, do not roll back my edit. It improves your post by specifying your actual question in the title. See Why can people edit my posts? How does editing work?. In particular, I would like you to notice the sentence "Editing is important for keeping questions and answers clear, relevant, and up-to-date. If you are not comfortable with the idea of your contributions being collaboratively edited by other trusted users, this may not be the site for you.".

My understanding is that you want to extend the main update loop like this:

$\qquad V(s) \leftarrow \text{max}_a \sum_{r,s'} p(r,s'|s,a)[r+\gamma V(s')]$
$\qquad \pi(s) \leftarrow \text{argmax}_a \sum_{r,s'} p(r,s'|s,a)[r+\gamma V(s')]$

Where the first line is same as original, the second line is taken from the end update that calculates the optimal policy.

Once our $\Delta < \theta$ and we break out of the loop, do we have an optimal policy from the family of optimal policies? Is this a valid thing to do?

Yes.
The only problem with this approach is the extra wasted processing - you will evaluate and overwrite $\pi$ once per loop. Doing it once at the end, after convergence, is more efficient.

The policies obtained after using standard VI and the method I described above are mostly similar, yet different for some states.

There are lots of equivalent optimal policies in the gambler's problem exercise in the book. This is expected. If you re-run the normal value iteration routine you should see similar changes to the policy.
The thing the policies have in common is a kind of fractal sawtooth shape for amounts bid, with triangles trying to get funds to fixed points and spikes where half or all funds are bid in single go to reach 100 funds. if you plot bid amount versus current funds. It might be one large triangle, or broken down into multiple smaller ones with larger bid spikes separating them. That is the expected shape, unless probability of a win is greater than 0.5, in which case the optimal policy is to bet a minimal amount on each time step and rely on the law of averages to eventually reach the target almost guaranteed. So you could set it to 0.6, you should see both your scripts return this same "safe" policy.

In some cases, if we overwrite and the $\pi$ turns out to be the same after two consecutive loop, then we can exit the loop early and wouldn't have to converge our value of $\Delta$ towards $\theta$ right?

@user529295 No I don't think so. The policy may shift with small changes to value function, and may remain the same over some iterations. I think to guarantee an optimal policy you have to run value iteration until the values converge.


text:
What to do when the ROIs are smaller than $227 \times 227$ in R-CNN?
As English is not my native language, I have some hard time understanding the following sentence:

Regardless of the size or aspect ratio of the candidate region, we warp all pixels in a tight bounding box around it to the required size. Prior to warping, we dilate the tight bounding box so that at the warped size there are exactly p pixels of warped image context around the original box (we use p = 16).

This is from the R-CNN paper. I already extracted the ROI, but now, they say that the input of the CNN should be 227 x 227, but a lot of my ROIs are much smaller. How can I deal with it?

Hello. I am still not sure if your question was theoretical/conceptual or practical (programming issue), but I would like to note that programming issues are generally off-topic here. See https://ai.stackexchange.com/help/on-topic for more details.

It is actually very easy, I didn't know the solution would be that simple.
With opencv, you can do :
resized_roi= cv2.resize(roi, (224,224))

and that's it :]


text:
What is the difference between a reward and a value for a given state?
I am trying to learn reinforcement learning and I am focusing on the value iteration. I am looking at the example of grid world, and I am trying to implement it in python. While doing this, I encountered the situation in which I had to set the rewards for the agent, but looking at the theory, I have found that each state has also a value, which is found using the value iteration.
So, my doubt is: What is the difference between a reward and a value for a given state? And should the initial values of the states always be set equal to zero?

Here and here are two related questions.

Starting with rewards, states don't have rewards in general. A reward is a number returned at a certain step of the MDP. If you arrange things in sequence over a whole time step $s, a, r, s'$ for state, action, reward, next state, then the reward $r$ is allowed to depend on all three of $s, a, s'$, and it can also be from a random distribution of real numbers, not just a single number.
It is however OK to associate a single number reward with each state, for either leaving that state (when it is $s$ or $s_t$ in the sequence) or arriving in it (when it is $s'$ or $s_{t+1}$). The rewards should be allocated as fits the problem being solved. They are part of the problem definition.
State values are a way to measure longer term benefits of being in a state, and are often something calculated as part of a solution. The formal definition of state value looks like this:
$$v_{\pi}(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t=s]$$
In English: "The expected discounted sum of all future rewards when starting from a given state and following a specific policy." The discounted sum is usually called the return or the utility associated with the state.

What is the difference between a reward and a value for a given state?

A state value is composed of many rewards weighted by their probability of occurring in the future. It is a useful summary of possible futures that can be used to make decisions.

And should the initial values of the states always be set equal to zero?

Not necessarily, but zero is a reasonable default if you have nothing else to go on. Alternatives include:

Best guesses at true values (perhaps from some previous attempt to solve the problem). This may improve speed of convergence depending on how good the guesses are.

Random values - this may happen if you use a neural network.

Optimistic values. This is a trick for improving exploration on smaller problems - if you set a value higher than an upper bound on the optimum possible then an agent following an greedy or near-greedy policy will try to reach the associated state at some point, even if other results are already better than a lower default like zero.

What is the difference between a reward and a value for a given state?

Let us say that an agent took an action from state $A$ and reached state $B$ and got a score $R$. This instantaneous score the agent received on reaching state $B$ is called the reward.
Now, let me introduce you to the concept of return. Assume that an agent followed a particular trajectory:
1. State 1 -> Action 1
2. Reward 1, State 2 -> Action 2
3. Reward 2, State 3 -> Action 3
...
n. Reward n-1, State n (Terminated)

Return (often denoted by $G$) is the sum total of all the rewards obtained by starting from a state State 1 and following a policy.
So, the definition of the return is
$$G(s_1) = R_1 + R_2 + R_3 + ... = \sum_{i=1}^{\infty}R_i$$
Sometimes (most often) these sequences never terminate, so we include a discount factor (Greek letter gamma, $\gamma$) to rewards obtained in the future.
The definition of the discounted return $G$ is
$$G(s_1) = R_1 + \gamma R_2 + \gamma^2 R_3 + ... = \sum_{i=1}^{\infty}\gamma^{i-1} R_i $$
$\gamma$ is a number between $0$ and $1$: it defines how much importance the agent gives to long-term rewards. For a smaller value of $\gamma$, more importance is given for short-term rewards.
Now, coming back to your question. A value of the state is the expected return for an agent starting from that state and following a particular policy. In the case of stochastic policies (policies that have inherent randomness) and/or for environments with stochastic transition probabilities and/or stochastic rewards, the value is the sum of (the returns of all trajectories multiplied by the probability of taking that trajectory).

And should the initial values of the states always be set equal to zero?

Not necessary, zero initialization is one of many ways to initialize. Random initialization is another method. It depends on the environment setting.


text:
Is there a recent book that covers the theoretical and philosophical aspects of artificial intelligence?
What are some recent books that introduce AI and neural networks while also discussing the related philosophical issues, like epistemology and whether AI is really thinking, etc.?

Stanley L. Jaki, O.S.B., Brain, Mind and Computers (2022) gives a good historical and philosophical overview.
Deep Learning (2016) by Ian Goodfellow, Yoshua Bengio, & Aaron Courville, introduction:

Inventors have long dreamed of creating machines that think. This desire dates
back to at least the time of ancient Greece. The mythical ﬁgures Pygmalion,
Daedalus, and Hephaestus may all be interpreted as legendary inventors, and
Galatea, Talos, and Pandora may all be regarded as artiﬁcial life
Ironically, abstract and formal tasks that are among the most difficult mental undertakings for a human being are among the easiest for a computer. Computers have long been able to defeat even the best human chess player but only recently have begun matching some of the abilities of average human beings to recognize objects or speech. A person’s everyday life requires an immense amount of knowledge about the world. Much of this knowledge is subjective and intuitive, and therefore difficult to articulate in a formal way. Computers need to capture this same knowledge in order to behave in an intelligent way. One of the key challenges in artificial intelligence is how to get this informal knowledge into a computer.
Deep learning has had a long and rich history, but has gone by many names, reflecting different philosophical viewpoints, and has waxed and waned in popularity.

The famous book Artificial Intelligence: A Modern Approach (by Stuart Russell and Peter Norvig) covers all or most of the theoretical aspects of artificial intelligence (such as deep learning) and it also dedicates one chapter to the common philosophical topics that you mention.

I would like to add "The Master Algorithm" by Pedro Domingos. I would say it's more philosophical but still provides high level discussions about differences between algorithms. He also has a sense of humor which makes it a lighter read.


text:
How to add negative samples for object detection?
My question is: how to add certain negative samples to the training dataset to suppress those samples that are recognized as the object.
For example, if I want to train a car detector. All my training images are outdoor images with at least one car. However, when I use the trained detector on indoor images, sometimes I got the wrong object detected (false positive). How can I add more indoor images (negative samples) to the training dataset to improve the accuracy? Can I just add them without any labeling?

What do you mean by "Can I just add them without any labeling?"? How are you training your neural network? Aren't you training your model in a supervised way?

I am training a SSD model for object detection. Each of my training images contains at least one object and the objects are labeled (bounding box) manually. When I test my model on certain scenario (not in the training set) I got many false positive results. I am just wondering how should I improve model? Can I add those failure images to the training set without labeling (because no object on those images)? Hopes it is clear

The quick answer: yes you can, just add images without labels, just make sure that in the negative samples there are no cars or you will make the AI crazy (i.e. convergence & instability issues).
However that might not be the better approach to go. Why? Because your dataset  already have enough negative examples. This was pointed out by the famous paper Focal Loss for Dense Object Detection. The paper basically proposes to think that each pixel of a dataset image is a training signal. Then for each image there are lots of pixels with negative signal (nothing in it: sky, ground, trees...) and only a few with positive signal (the actual car).
So if each image of the dataset have more negative signals (pixels) than positive, then, the problem might not be in the negative examples. So that leaves you with 2 ways to go:

Use a loss function that focus more on positive signal (car pixels) than in the negative examples (not car pixels) such focal loss or derivatives
Add  more positive examples in the dataset

I can confirm what this paper stated in my every day experiments. We have a battery of experiments right now that performs best with focal loss & no negative examples VS other experiments without focal loss & negative examples.
Just for reference this is what happens when there are lots of negative examples:

The AI took a while to figure out that negative samples are not useful (1M steps) in this experiment. From them on it just focused on the positive samples and this training started to converge (and inference started to show something meaningful)


text:
What gets optimized in convolutional neural network?
In a convolutional neural network, the hyperparameters such as number of kernels and stride, kernel size, etc are determined. After some combination of convolutions, ReLU and pooling layer there is the fully connected (FC) layer in the end which yields a classification result. I originally thought that during training the values of kernels would be optimized and that kernels such as edge detection are a result of optimization.
But at the end if we have weights to optimize at the FC layer, what is it that gets optimized during training of the CNN? Do both the kernel values and weights in FCC get optimized? If so, it seems like we're dealing with two different types of parameters. How are both trained simultaneously? If not so, are there simply sets of kernels known to work and automatically implemented in CNN modules?

Do both the kernel values and weights in FCC get optimized?

Yes.
Some of the designs for image processing neural networks prior to CNNs had separate filter processing states. For instance, Sobel filters were popular choices in earlier attempts at machine learning on images, and they can be thought of as fixed CNN-like layers. They may still have a role in some projects.
However, most CNN architectures for images now work with pixel data directly, and can learn both the filter weights and fully connected later weights together.
In some uses, such as transfer learning, it is useful to be able to selectively learn only some of the layers. You may take the CNN filter layers from a very general image classifier trained on ImageNet, and repurpose it by replacing the fully connected layers. When training the new neural network you can freeze the filter layers and learn only the fully connected layers - although there is no specific requirement to separate them only by convolutional/fully-connected, you could equally retrain only part of the fully-connnected layers, or include some of the convolutional layers.

If so, it seems like we're dealing with two different types of parameters. How are both trained simultaneously?

They are not really that different. A CNN can be thought of as fully connected throughout, but with some extra constraints on the convolutional layers:

Weights that would connect a feature map neuron to an input outside the bounds of the filter are always zero.

Weights for each position on the feature map are identical.

Using learnable convolution filters enforces both these constraints.
The practical difference to how back propagation works in convolutional layers as opposed to fully connected layers is to sum all gradients arising from each "pixel" in the feature map to the appropriate filter weight when back propagating. So unlike a fully connected layer weight that receives one summed gradient update from the layer above, each convolutional filter weight receives the equivalent update summed over all pixels in the next feature layer. Depending on which source you learn CNNs from, this additional outer sum might be shown or might be implied using different notation for the update rules.


text:
Is the Bandit Problem an MDP?
I've read Sutton and Barto's introductory RL book. They define a policy as a mapping from states to probabilities of selecting each possible action. If the agent is following policy $\pi$ at time $t$, then $\pi(a|s)$ as the probability of taking action $A_t = a$ when the current state is $S_t = s$. This definition is the context of the markov assumption, which is why the policy is only dependent on the current state.
When discussing the standard k-armed bandits problem, they write $\pi(a)$ to denote the probability of taking action $a$, since there are no states. However, when designing the agent, clearly, the agent needs to keep track of what the past rewards are for each lever, so either there is a summary statistic of each lever, or the entire history of actions and rewards must be kept.
Is the k-armed bandit problem then a MDP? Why isn't the notation $\pi(a|A_0, R_1, A_1, \ldots, R_T)$ for some sequence  $A_0, R_1, A_1, \ldots, R_T$?

Here is a related question.

The bandit problem is an MDP. You can make the same argument about needing data to learn in the stateful MDP setting. The thing is, the data you need (the past rewards in this case) was drawn iid (conditioned on the arm) and is not actually a trajectory. For instance, once you learn an optimal policy, you no longer need to gather data and the sequence of past results doesn't influence your policy.

It seems to me that the OP is mixing the concept of a state and action and reward. It may be a good idea to clarify that these are distinct concepts and explain how are they related. Moreover, it may also be a good idea to clarify that a k-armed bandit problem can be modelled as an MPD with 1 state or no state, where the actions are the arms that you can pull; so, in this scenario, the policy would always be conditioned on the same state. That's why it would probably be redundant to write $\pi(a \mid s)$.

I assumed the OP understood this, he did write $\pi(a)$. I think he's questioning why the policy shouldn't be conditioned on the previous actions and associated rewards because you use those to estimate the optimal policy.