text:
Why does Batch Normalization work?
Adding BatchNorm layers improves training time and makes the whole deep model more stable.  That's an experimental fact that is widely used in machine learning practice.
My question is - why does it work?
The original (2015) paper motivated the introduction of the layers by stating that these layers help fixing "internal covariate shift". The rough idea is that large shifts in the distributions of inputs of inner layers makes training less stable, leading to a decrease in the learning rate and slowing down of the training.  Batch normalization mitigates this problem by standardizing the inputs of inner layers.
This explanation was harshly criticized by the next (2018) paper -- quoting the abstract:

... distributional stability of layer inputs has little to do with the success of BatchNorm

They demonstrate that BatchNorm only slightly affects the inner layer inputs distributions.  More than that -- they tried to inject some non-zero mean/variance noise into the distributions. And they still got almost the same performance.
Their conclusion was that the real reason BatchNorm works was that...

Instead BatchNorm makes the optimization landscape significantly smoother.

Which, to my taste, is slightly tautological to saying that it improves stability.
I've found two more papers trying to tackle the question: In this paper the "key benefit" is claimed to be the fact that Batch Normalization biases residual blocks towards the identity function. And in this paper that it "avoids rank collapse".
So, is there any bottom line? Why does BatchNorm work?

This blog post attempted to explain why this is the case (go to the part "So, why does Batch Norm Work?"), but as they mentioned it is kind of a speculative thought.

I believe anything in machine learning that works, works because it flattens and smoothens the loss landscape.
Batch and layer normalization would help ensure that the feature vectors (i.e. channels) are embedded around the unit sphere Batch/Instance norm translates to origin. Layer norm scales radially to unit sphere. Viewing neural networks as transformations, this would make the loss landscape smoother since the transformations the neural net needs to find would be more "regular".
I would recomend this video to learn about loss landscapes.
From Visualizing the Loss Landscape of Neural Nets. NeuRIPS 2018:

This got me thinking about my understanding of batch normalization. I thought I understand it until I read this. Then, I refer to the Coursera deep learning specialization by Andrew Ng.
Prof. Andrew Ng explained it this way.

One reason why does batch norm work is that it normalizes not only the input features but also further values in the hidden units to take
on a similar range of values that can speed up learning.
The second reason why batch norm works, is it makes weights, later or
deeper than the network you have, say the weight on layer 10, more robust to changes to weights in earlier layers of the neural network (eg. in layer one). However, these hidden unit values are changing all the time, and so it's suffering from the problem of covariate shift.  So what batch norm does, is it reduces the amount that the distribution of these hidden unit values shifts around. What batch norm ensures is that no matter how the parameters of the neural network update, their mean and variance will at least stay the same mean and variance, causing the input values to become more stable, so that the later layers of the neural network has more firm ground to stand on.

And even though the input distribution changes a bit, it changes less, and
what this does is, even as the earlier layers keep learning, the amounts that this forces the later layers to adapt
to as early as layer changes is reduced or, if you will, it weakens the coupling between what the early layers
parameters has to do and what the later layers parameters have to do. And so it allows each layer of the
network to learn by itself, a little bit more independently of other layers, and this has the effect of
speeding up of learning in the whole network. Takeaway is that batch norm means that, especially from
the perspective of one of the later layers of the neural network, the earlier layers don't get to shift around as
much, because they're constrained to have the same mean and variance. And so this makes the job of learning
on the later layers easier. It turns out batch norm has a second effect, it has a slight regularization effect. So
one non-intuitive thing of a batch norm is that each mini-batch, the mean and variance computed on just that
mini-batch as opposed to computed on the entire data set, that mean and variance has a little bit of noise
in it, because it's computed just on your mini-batch of, say, 64, or 128, or maybe 256 or larger training
examples. Batch norm works with mini-batch

The punctuation needs some work.

@HappyFace, I see your point.

Hey, @kibromhft, my question specifically mentions these kinds of explanations. Experiments in the 2018 paper show that BatchNorm doesn't affect distributions in inner layers. Even more - artificially changing these distributions doesn't affect the performance much.

When we are training deep neural Network gradient tells how to update each parameter, under the assumption other layers do not change.In Practice, we update all the layers simultaneously. When we update, unexpected results can happen because many functions composed together are changed simultaneously using updates that were computed under the assumption that other function remains constant.This makes it very hard to choose an appropriate leaning rate, because the effects of an update to the parametrs of one layer strongly on all other layers.

How does Batch Normalisation Help :
Batch Normalisation a layer which is added to any input or hidden layer in the neural network. Suppose H is the minitach of activations of the layer to normalize.
The formula for normalizing H is :
$_H = \frac{H - Mean}{Standard Deviation}$
Mean : Vector Containing Mean of each unit
Standard Deviation : Vector Containing Mean of each unit
At training time mean and sd are calculated and when we backpropogate through these operations for apply mean, sd and Normalize H. This means that gradient will never propose an operation that acts simply to increase the standard deviation and mean of hi, the normalization operation remove the effect of such an action and zero out its componenr in the gradient. Hence Batch Normalisation thus ensure no or slight covariance shift in the input to layer after Batch Normalisation and thus improving learning time as shown in the original paper mentioned in question.
For more details : https://www.deeplearningbook.org/contents/optimization.html

At least this "tells how to update each parameter, under the assumption other layers do not change.In Practice, we update all the layers simultaneously." seems to be copied from here. Either edit this answer to clarify which parts you copied or this answer will be deleted.

if you look at the reference we are referring to same reference. Nothing has been copied from the link :) The sentence you have highlighted is taken from reference.

Yes, I understood that the reference is the same, but you cannot just copy from a reference without explicitly showing us WHAT you copied, i.e. you need to quote. To quote, you need to precede the text that you copied with >. Take a look at https://ai.stackexchange.com/help/referencing to know more about how to quote parts of other answers. Please, take the time to go through all your answers to quote things that you copied (from other sources) in order for them not to be deleted.

Thanks i am new to this platform , will keep this things in mind

Don't just put the reference at the end. Whenever you quote something, tell us immediately where you copied it from, which section and page, so that people can check that you're referencing a specific part of another source. That chapter 8 is a bit long.

It is a question with no simple answer.
On one hand the BatchNormalization is unloved by some arguing it doesn't change the accuracy of neural networks or biased them.
On the other hand, it is highly recommended by the other because it leads to better trained models with a larger scope of predictions and less chances of overflow.
All I know for sure is that BN is really efficient on image classification. In fact, like the image categorization and classification soar this last years and that BN is a good practice in this field, it has spread to almost all DNNs.
Not only is the BN not always used in the right purpose, but it is often used without taking into account several elements such as :

The layers between which apply BN
The initializer algorithms
The activation algorithms
etc

For more computer sciences litterature "against" BN, I will let you look at the H. Zhang et al paper who has trained a DNN without BN and get good results.
Some people use Gradient Clipping technique (R. Pascanu) instead of the BN in particular for RNNs
I hope it will give you some answers !

Thanks for the response. I'm following it as I added the bounty. I down voted it but don't take offence! The reason I did it is because I don't believe any part of it really addresses the question. Votes help others quickly identify the most relevant responses.

Thanks for the feedback and the explanation. So I guess I didn't understand the question. Can you just reformulate it to let me understand it please ?

I think your answer is more for a question like "Where is BN useful vs not?" or "Is BN always a good thing to use?". The question asks "What is the generally accepted reason for why BN works well?" Like if I ask "Why do conv kernels work well for images?" the answer is along the lines of: "Because it satisfies the inductive priors of translational invariance, feature hierarchies, and to some degree, scale invariance". Your answer doesn't talk about the properties of BN that make it successful. And in this case OP also wants the answer to be backed by authoritative sources

To some extend, it get rid of low intensity numerical noise. Condition properties of the optimization problem is always an issue, i suspect BatchNorm alleviate this instability.

Adding some artificial noise to neural networks quite often increases their generalization performance.

I was just thinking about the conditioning that BatchNorm performs and that also in some cases this process cutoff low intensity component. When a feature appears on a layer, it become evident since its intensity is very over average. When you "normalize",  all others components falls under threshold at least numerically. Then this "kind of noise" is removed. Another thing is adding uniform noise, which helps to search such a desirable feature and it is not removed by "normalization".


text:
What is the meaning of "Our current objective weights every token equally and lacks a notion of what is most important to predict" in the GPT-3 paper?
On page 34 of OpenAI's GPT-3, there is a sentence demonstrating the limitation of objective function:

Our current objective weights every token equally and lacks a notion of what is most important to predict and what is less important.

I am not sure if I understand this correctly. In my understanding, the objective function is to maximize the log-likelihood of the token to predict given the current context, i.e., $\max L \sim \sum_{i} \log P(x_{i} | x_{<i})$. Although we aim to predict every token that appears in the training sentence, the tokens have a certain distribution, and therefore we do not actually assign equal weight to every token in loss optimization.
And what should be an example for a model to get the notion of "what is important and what is not". What is the importance refer to in here? For example, does it mean that "the" is less important compared to a less common noun, or does it mean that "the current task we are interested in is more important than the scenario we are not interested in ?"
Any idea how to understand the sentence by OpenAI?

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What is the difference between a performance standard and performance measure?
I am reading AI: A Modern Approach. In the 2nd chapter when introducing different agent types, i.e., reflex, utility-based, goal-based, and learning agents, I understood that all types of agents, except learning agents, receive feedback and choose actions using the performance measure.
But they do so in different ways. Model-based reflex agents possess an internal state (like a memory), while goal-based agents predict the outcome of actions and choose the one serving the goal. Lastly, utility-based functions measure the 'happiness' of each state using the utility function, which is again an internalization of the performance measure, hence all have similar nature overall.
The learning agents, however, can be wrapped around the entire structure of previous agents. The entire agent's architecture is now called a performance element, and the learning agent has an additional learning element, which modifies each component of the agent, so as to bring the components into closer agreement with the available feedback information. But the feedback information in learning agents does not from the performance measure embedded in the agent's structure, but from a fixed external performance standard, which is part of the critic element*.
For the purpose of illustration, the structure of a utility-based agent and that of a learning agent are presented in the figure:

What boggles my mind is figuring out the actual difference and interaction between performance standard and performance measure, which is perhaps related to those between learning agents and other ones. Here are my thoughts thus far:

Other agents aim for maximizing the performance measure, causing them to do perfect actions. On the other hand, learning agents have the freedom of doing sub-optimal actions, which allow them to discover better actions on the long run using the performance standard.

Through the performance standard's feedback (which comes from the critic as shown in the figure), the learning agent can also learn a utility function or reflex component.

For providing examples, the book states that giving tip to an automated taxi is considered a performance standard. And also

hard-wired performance standards such as pain and hunger in animals can be understood in this way.

But I am still not sure about the discrepancy and interaction between the performance measure and performance standard. For instance, in the automated taxi, when confronting a road junction, the utility-based agent chooses a path that maximizes its utility function. The learning agent, however, must check different roads and after testing them, it receives feedback from outside so that eventually it would detect the user's preference.
But what if we wrap a learning agent around a utility-based agent in such a condition? Which has more effect, the utility function from inside, or the performance standard from outside (critic)? If they happen to contradict each other, which one would have the prevalent effect?

After having read a few parts of the book that mention these terms, I don't think there's any practical difference between a performance standard and a performance measure. They both measure the performance of the agents.
They initially use the term performance measure to refer to rationality in section 1.1

The definitions on the left measure success in terms of fidelity to human performance, whereas the ones on the right measure against an ideal performance measure, called rationality.

Later, they define this performance measure in the context of rational agents in section 2.2.

If the sequence is desirable, then the agent has performed well. This notion of desirability is captured by a performance measure that evaluates any given sequence of environment states.

So, here, a performance measure evaluates a sequence of states.
In that same section (3rd edition), they explain that the performance measure should be designed in such a way that the agent achieves our desired goals.

Obviously, there is not one fixed performance measure for all tasks and agents; typically, a designer will devise one appropriate to the circumstances. This is not as easy as it sounds. Consider, for example, the vacuum-cleaner agent from the preceding section. We might propose to measure performance by the amount of dirt cleaned up in a single eight-hour shift. With a rational agent, of course, what you ask for is what you get. A rational agent can maximize this performance measure by cleaning up the dirt, then dumping it all on the floor, then cleaning it up again, and so on. A more suitable performance measure would reward the agent for having a clean floor. For example, one point could be awarded for each clean square
at each time step (perhaps with a penalty for electricity consumed and noise generated). As a general rule, it is better to design performance measures according to what one actually wants in the environment, rather than according to how one thinks the agent should behave.

So, the usage of the term performance measure in this example is very analogous to the usage of the term reward function in RL. To be more precise, in RL, you could define a reward function as a function $r : \mathcal{S} \rightarrow \mathbb{R}$, so $r(s)$, where $s \in \mathcal{S}$, is the reward given to the RL agent when it enters or is in state $s$. This is consistent with their definition above (although they state that the performance measure evaluates a sequence of states, but I think this is due to the fact that, in a rational agent, you don't necessarily assume that the Markov property holds, like in an MDP and RL).
Later, they also use the term performance measure in the context of utility-based agents,

An agent's utility function is essentially an internalization of the performance measure. If the internal utility function and the external performance measure are in agreement, then an agent that chooses actions to maximize its utility will be rational according to the external performance measure

So, here, if you are familiar with RL, the distinction between the utility function and the performance measure should be clear. The utility function would be analogous to the value function, while the performance measure is again analogous to the reward function.
Later, in section 17.1.1, they write

In the MDP example in Figure 17.1, the performance of the agent was measured by a sum of rewards for the states visited. This choice of performance measure is not arbitrary

So, again, the performance measure is a synonym for the reward function in the context of RL.
In section 2.4, they write about the performance standard

The critic tells the learning element how well the agent is doing with respect to a fixed performance standard. The critic is necessary because the percepts themselves provide no indication of the agent’s success. For example, a chess program could receive a percept indicating that it has checkmated its opponent, but it needs a performance standard to know that this is a good thing; the percept itself does not say so. It is important that the performance standard be fixed. Conceptually, one should think of it as being outside the agent altogether because the agent must not modify it to fit its own behavior

That's why it appears outside the agent in the diagram.
On the next page, they write

In a sense, the performance standard distinguishes part of the
incoming percept as a reward (or penalty) that provides direct feedback on the quality of the agent’s behavior.

In RL, this would be represented as $p = (o, r)$, i.e. the percept $p$ is composed of the observation $o$ and reward/performance $r$.
I don't really know why they used different terms for rational, utility-based, and learning agents, but, to me, I don't see any difference between the two terms.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Is it possible to use an internal layer's outputs in a loss function?
For a network of the form:
Input(10)
Dense(200)
Dense(100+10)
Dense(20)
Output()

Those +10 outputs are what I want to add to the standard 20 outputs, for my loss function.
Is this possible - in theory or even with some pre-existing library?

Yes, you can do that, and it is a standard practice. One famous example is the "Inception" network architecture. To keep inner subnets from "dying out",  several outputs from inner layers are extracted and passed into FC->Softmax. Then all the outputs are averaged in the loss function.
From practical point of view, you won't be able to implement such things with basic Sequential model construction. Most libraries allow you to move beyond it, though, and this distinction is usually well-documented. For example in the tensorflow guide on functional API it states right in the introduction:

The functional API is a way to create models that are more flexible than the tf.keras.Sequential API. The functional API can handle models with non-linear topology, shared layers, and even multiple inputs or outputs.

Hope that'll help you to get started.


text:
How does the loss landscape look like or change when a model is overfitting?
My understanding is that when a model starts overfitting, it no longer learns useful features and starts remembering the training data set. Given enough epochs and sufficient parameters, a model can over-fit any arbitrary dataset. My question is how does the loss landscape look like for the training dataset vs the test dataset when a model is overfitting? Is there a weird dip around some point in the training dataset but the same dip is not there in the test dataset?


text:
Must all CNNs and RNNs not have a fully connected layer in order to be considered as such?
In the paper Wrist-worn blood pressure tracking in healthy free-living individuals using neural networks, the authors talk about a combination of feed-forward and recurrent layers, as if FC layers cannot be part of the RNN.
So, must all Convolutional Neural Networks and Recurrent Neural Networks not have a fully connected layer in order to be considered CNNs and RNNs, respectively? If yes, should we consider CNNs and RNNs with an FC layer "hybrid models"?

Not quite sure about RNN & LSTM (and it always depends on the task), but for CNN the answer is clearly no; CNN routinely include FC layers. Quoting from the highly popular (and recommended) Stanford course CS231n: Convolutional Neural Networks for Visual Recognition:

ConvNet Architectures
We have seen that Convolutional Networks are commonly made up of only
three layer types: CONV, POOL (we assume Max pool unless stated
otherwise) and FC (short for fully-connected).

You can easily verify that this is the case for practically all popular CNN models for computer vision in any relevant exposition (e.g. Illustrated: 10 CNN Architectures).
In fact, the all-convolutional NN (i.e. without FC layers) is considered a special case: see Striving for Simplicity: The All Convolutional Net.


text:
Why do you calculate the mean and standard deviation over the complete dataset before training rather than for every batch?
In most implementations of neural networks the features are scaled to make the optimization of the loss function as stable as possible.
Mostly a min-max scaler is used. Alternatively, there is also a standard scaler.
Why do you calculate the mean and standard deviation offline over the complete dataset before training? Couldn't this be calculated per batch or even per file? What is the disadvantage? Why doesn't anyone do this?

Just to clarify, you're wondering why we couldn't take a batch from the data, compute the mean and standard deviation of each feature over that batch, and then scale using that (so the means from each batch could potentially be different)?


text:
Issue with graphical interpretation of the universal approximation theorem
This article attempts to provide a graphical justification of the universal approximation theorem.
It succeeds in showing that a linear combination of two sigmoids can produce essentially a bounded constant function or step function, and thus can therefore to a reasonable degree of approximation produce any function by essentially splitting up any function into a cluster (linear combination?) of these towers or steps.
However, he produced the steps and towers using specific weight parametrizations.
However, since when are we allowed to specify weights and biases? Isn't this all out of our hands and in the hands of cost function minimization?
I don't understand why he was dealing with setting weights to this, biases to that, when in my experience that is all done by "the machine" to minimize the cost function. I doubt the weights to minimize the cost function are arranged in the ways specified in order to form the towers and steps that were formed in this tutorial, so I kind of don't understand what all the hub-ub is all about.

The classical version of the universal approximation theorem states that, roughly, given a continuous function $f \colon [0, 1]^n \to [0, 1]^n$, there exists a single layer neural network and a set of weights and biases such that this network approximates the given function $f$ arbitrarily well.
It doesn't say anything about how you obtain such weights: the result is entirely independent of the way you train your network, and what it says is that the set of single-layer neural networks has enough capacity, in principle, to approximate any continuous function arbitrarily well.
This kind of result is fairly common in mathematics: it is shown that a suitable object must exist, but the proof is non-constructive, i.e. it doesn't actually show you how to get that object.
You're indeed correct that the backpropagation algorithm might not be able to find suitable weights. In fact, if you fix an architecture in advance, there might not even be any weights that would lead to a neural network that is a good approximation to $f$.
Why do we care?
Universality tells us, at least in principle, that we should be able to approximate the function we want, if we pick an appropriate architecture. The XOR Problem, for example, affects a type of model called a perceptron (essentially a single neuron neural network), and says that the XOR function cannot be approximated by perceptrons. It is always useful to know the kinds of functions that can and cannot be expressed by a certain type of model.

What was the usefulness of this article then? It sounds like he's saying functions can be approximated with sigmoid "towers" and "steps", but the neural network would have to have specific weights to force towers and steps, which is not even happening at all in the first place?

The point is that, if universal approximation wasn't true, there might be types of functions that you can never hope to represent with a neural network, because there simply isn't any set of weights which would do this. This would have practical implications because it would mean that no matter how much you train or look for a good architecture, you might still fail to find a good neural net for your problem. The whole section where he explicitly constructs weights is just to help you understand why the result is true.

So he gave an example of a possible way sigmoid functions can represent any function, but it isn't necessarily the way the neural network will use the sigmiods (by using stochastic gradient descent to get the best weights and biases), just a way that shows proof of concept.

That's correct, @sangstar. The theorem doesn't care what your algorithm is for finding the weights (and, conversely, makes no promises that backpropagation will find very good weights). It just says that there are some weights which would be a good fit: it doesn't tell you how to find them, and it doesn't say that backpropagation will converge to these weights: backprop isn't even mentioned once in the paper proving the theorem!

I suppose I find it counterintuitive, because the way the offer represents the “building blocks” it makes me feel like those “steps” and “towers” ought to be the orthonormal basis functions (because of how “fundamental” they look” that should form any functions, and that they should not be arbitrarily chosen but instead be the functions to use. Just an impression


text:
What are the differences in testing between traditional software and artificial intelligence?
The testing problem in traditional software has been fully explored over the last decades, but it seems that testing in artificial intelligence/machine learning has not (see this question and this one).
What are the differences between the two?

Testing machine learning programs is quite different than testing traditional software.
The main reason why this is the case is quite simple, if you're familiar with machine learning.
ML programs are not just if statements and loops, but they are composed of models, which can even be black-box models, such as neural networks (i.e. it's difficult to interpret the function that they compute). These models are trained to approximate some unknown function, which is only partially described by some given data, which can also contain spurious/noisy information.
For this reason, traditional testing techniques, such as statement coverage, are insufficient to fully "test" ML programs: e.g. even if all statements of your ML program are covered, your ML program can still fail, e.g. it may predict the wrong class for some previously unseen input. So, testing ML programs requires not only traditional testing techniques, but also other approaches that attempt to address the (un-)desirable behaviour (such as the generalization) of the models.
Currently, I am also doing research on this specific topic, so I can say that there are already several people working on this topic, but the field is still quite immature and there are still many unsolved problems. It's also important to note that the general "testing problem", even for traditional software, is not yet solved.


text:
How does high entropy targets relate to less variance of the gradient between training cases?
I've been trying to understand the Distilling the Knowledge in a Neural Network paper by Hinton et al. But I cannot fully understand this:

When the soft targets have high entropy, they provide much more information per training case than hard targets and much less variance in the gradient between training cases [...]

The information part is very clear, but how does high entropy correlate to less variance between training cases?

Suppose you have two tokens that are equally likely. In your original training data, your next-token predictions are
$$[1, 0]\quad\text{and}\quad [0, 1]$$
an equal number of times. The entropy of each of these samples is zero. However, a fully trained network should converge to
$$[0.5, 0.5],$$
which has an entropy of one bit. A second network trained on these higher-entropy samples will see less variance in its KL-loss. Also, because the learned network is essentially collating a whole lot of training samples together, we get more information per sample.

Since it is a trained network already, when you run an example through it, the gradient will not have a very high variance.
The gradient varies a lot when you are training a network from the scratch but then it stops varying much since it understands the pattern.


text:
Is there any research on the application of policy gradients to problems where the selection of an action requires the selection of another one?
I am working on a problem and want to explore if it can be solved with PPO (or other policy gradient methods). The problem is that the action space is a bit special, compared to classic RL environments.
At each time $t$, we chose between 4 actions: $a_1\in \{0, 1, 2, 3\}$,
but given $a_1 = 0 \text{ or } 2$, we need to chose three more actions: $a_2, a_3, a_4$ (which all three can be chosen from categorical distributions).
I know I can design this kind of policy myself and re-work the entropy terms, and so on for PPO.
My question is: is there any research into this kind of RL?
I am having a hard time finding someone working with problems in which the actions chosen are dependent on other chosen at the same time. I have looked into Hierarchical RL, but the papers I have found have not worked with this particular kind of problem.
If these action spaces were small ($a_2, a_3$ are chosen from categorical distributions with $\sim$800 different options), one solution would be to roll it out into one big policy where each possible combination of actions is represented by one choice in the policy. But my concern of doing this with a bigger action space is that the choice of $a_1 = 1, 3$ where we don't choose the other separate actions will get lost in the policy.

you could check out the options framework (I think mainly introduced by Doima Precup in her PhD work) but it works somewhat similar to what you are describing; at time $t$ you choose an option which then dictates that you follow an options policy until some stopping criteria is met (its been a while since I looked at this so I may not be being 100% accurate, but the general idea should be right).

I made changes to your post to improve the clarity. Make sure the meaning has not changed.


text:
Extracting values from text based on keywords
I am trying to read a PDF file and put it in Python string and trying to fetch information based on keywords. The text here is completely irregular.
Example of text

Blockquote
Ram has taken an insurance of his premises with total sum insured of INR 256,200,000,000. XYZ company provides an insured with limit of liability of INR 100,250,000 and 90 days indemnity period. Insured with deductible of INR 200,000.

Here I want to find 3 things from this text

limit of liability amount
Deductible amount
Sum insured amount

For example
Limit of liability  = 100,250,000

What is your specific question? What have you tried so far?

If the text is really that regular, you can do simple pattern matching: Search for the keywords "limit of liability", "deductible", and "sum insured". The take the next numerical value (possibly preceded by "INR"), and assign it to the corresponding value.
However, this is very simplistic and brittle, and will fail if the texts are more varied. You might still get quite far with it, and you can always add more variations of the keywords to it.
Note that I would not advise that for general text analysis, as it's far too primitive.


text:
Why does $E_q[\log p(\mathbf{w}|\mathbf{z},\beta)]=\sum_{n=1}^{N}\sum_{i=1}^{k}\sum_{j=1}^{V}\phi_{ni}w_n^j\log \beta_{ij}$ hold in LDA?
I'm having trouble understanding an equality that comes up in the original LDA paper by Blei et al.:
Consider the classical LDA model, i.e. for every document $\textbf{w}=(w_1,\ldots,w_N)$ in a text corpus $\mathcal{D}=\{w_1,\ldots,w_M\}$ assume that the document is created as follows$^{\dagger}$:

Choose $N\sim \text{Poisson}(\xi)$.

Choose $\theta\sim\text{Dir}(\alpha)$.

For each of the $N$ words $w_n$:
(a) Choose a topic $z_n\sim \text{Multinomial}(\theta)$.
(b) Choose a word $w_n$ from $p(w_n|z_n,\beta)$, a multinomial probability conditioned on the topic $z_n$.

In order to do inference for LDA, the authors use a variational approach with the following graphical model where $\gamma$ and $\phi$ are Dirichlet and multinomial parameters, respectively:

Let us write $p$ for the original LDA distribution and $q$ for the variational one. The equality I don't understand is given in the appendix$^{*}$ of the paper and states that:
$$E_q[\log p(\mathbf{w}|\mathbf{z},\beta)]=\sum_{n=1}^{N}\sum_{i=1}^{k}\sum_{j=1}^{V}\phi_{ni}w_n^j\log \beta_{ij}.$$
My work so far:
We can write the RHS as
$$\sum_{n=1}^{N}\sum_{i=1}^{k}\sum_{j=1}^{V}\phi_{ni}w_n^j\log \beta_{ij}=\sum_{n=1}^{N}\sum_{i=1}^{k}\sum_{j=1}^{V}q(z_n=i)\cdot w_n^j\cdot \log p(w_n^j=1|z_n=i)$$
and the LHS via exchangeability and de Finetti's theorem as
$$p(\mathbf{w}|\mathbf{z},\beta)]=\sum_{n=1}^{N}E_q[\log p(w_n|z_n,\beta)].$$
I now want to obtain the double sum from the right-hand side on the LHS as well. This looks like some sort of expected value with respect to $q$, of a discrete random variable that only depends on the values of $z_n$, conditional on $w_n$ (seemingly, as if $w_n$ was fixed) but the R.V. that we do have on the left-hand side is $\log p(w_n|z_n,\beta)$ which depends on both the values of $z_n$ and $w_n$, both not fixed. How do I continue?

$^{\dagger}$ Also assume that both $M$, the number of documents, and $k$, the dimensionality of the Dirichlet, are known and fixed. Furthermore, let $V$ denote the number of possible words (the "size of the vocabulary") and write every word as a $\{0,1\}^V$ vector with zeros everywhere except for the index of the word in the vocabulary list. Finally, let $\beta$ be a $k\times V$ matrix with $\beta_{ij}=p(w^j=1|z^i=1)$ and assume that both words and documents are exchangeable.
$^*$ Eq. 15, A.3


text:
What is the relation between self-taught learning and transfer learning?
I am new to transfer learning and I start by reading A Survey on Transfer Learning, and it stated the following:

according to different situations of labeled and unlabeled data in the source domain, we can
further categorize the inductive transfer learning setting into two cases:
case $(a)$ (It is irrelevant to my question).
case $(b): $ No labeled data in the source domain are
available. In this case, the inductive transfer
learning setting is similar to the self-taught
learning setting, which is first proposed by Raina
et al. [22]. In the self-taught learning setting, the
label spaces between the source and target
domains may be different, which implies the
side information of the source domain cannot be
used directly. Thus, it’s similar to the inductive
transfer learning setting where the labeled data
in the source domain are unavailable.

From that, I understand that self-taught learning is inductive transfer learning.
But I opened the paper of self-taught learning that was mentioned (i.e paper by Raina
et al. [22].), and It stated the following in the introduction:

Because
self-taught learning places significantly fewer restrictions on
the type of unlabeled data, in many practical applications
(such as image, audio or text classification) it is much easier
to apply than typical semi-supervised learning or transfer learning
methods.

And here it looks like transfer learning is different from self-taught learning.
So what is the right relation between them?

I'm not familiar with the term self-taught learning, but I suspect it is related to self-supervised learning, which is turn is related to transfer learning, as SSL can be used for transfer learning. See this post for more info about self-supervised learning.

@nbro It is good to know about self-supervised learning, Thank you.
But what about the distribution of the data that used for it? and the actual labels of them?
In Self_Taught learning, there is no need for the unlabeled data (that used for training) to follow the same class labels or generative distribution as the labelled data that will be used for the supervised task.

After a lot of searches, I think self-taught learning is a Transfer learning category, I think when Self-taught learning paper published (2007), there isn't any good survey on transfer learning, and as seen by high citation of the paper of Pan, his paper (which published in 2009) describes transfer learning in a clear way that did not exist before it.
Also, it is reasonable to consider Self-Taught learning a transfer learning category because he actually transfers the knowledge learned from unlabelled data to use it in the supervised task that we want to do (there is no need for the unlabeled data (that used for training) to follow the same class labels or generative distribution as the labelled data that will be used for the supervised task).
If someone could find something wrong with my answer or there is something missing, then please tell me.


text:
Given a sequence of states followed by the agent, is it possible to find the Q-value for a state-action pair not in this sequence?
Assume you are given a sequence of states followed by the agent, generated by a random policy, $[s_0, s_1, s_2, \dots,  s_n]$. Furthermore, assume the MDP is fully observable and time is discrete.
Is it possible to find the Q-value for a state-action pair $(s_j, a_j)$ which was not encountered along this sequence?
From my understanding of the MDP, yes, it would be possible. However, I'm unsure how to get this Q-value.

My understanding from your question is that you have the following data generated from a random policy:
$$[s_0, s_1, s_2 . . . s_n]$$
That is, the state observed at each time step.
You know nothing more about the MDP, such as the transition or reward functions. Although the MDP is discrete and fully observable (and thus usual RL theory is supported), you do not have any observations other than a single list of states.
You wish to obtain an estimate for $Q(s_j, a_j)$, given a specific state/action pair that has not been observed before. This is not possible. You do not have any data from your observations regarding rewards, nor the actions taken by the random agent. At minimum, you would need data about rewards and actions that were observed. Observed rewards are needed to calculate returns, and value functions are expected returns. Observed actions are needed to assign returns to the correct $(s,a)$ pair when calculating estimates.
In addition, if you want to make a meaningful estimate for an as-yet unseen state/action pair, then you would need to be using some form of function approximation for the action value function Q. Otherwise your estimate will be whatever default estimate you had assigned at the beginning of learning.
What inferences can you make given the data you have? Very roughly:
$$p(s_{k+1} | s_k, \pi ) \gt 0, \forall k \in [0,n-1]$$
i.e. the state sequence that was observed gives you proof of non-zero transition probabilities between those states under the given policy. This is not enough to learn a value function, but might have other uses.
With repeated observations - multiple lists of observed states - you could infer what those probabilities were, but would not be able to separate the effect of action choice from state transition rules (because the action was not observed). If you added reward observations then you could learn a state value function.


text:
Off-policy Bellman Operators: Writing Operator and Weight Update Function for a 2-State System
I am studying for RL on my own and was trying to solve this question I came across.

Write an operator function $T(w, \pi, \mu, l, g)$ that takes weights $w$, a target policy $\pi$, a behaviour policy $\mu$, a trace parameter $l$, and a discount $g$, and outputs an off-policy-corrected lambda-return. For this question, implement the standard importance-weighted per-decision lambda-return. There will only be two actions, with the same policy in each state,
so we can define $\pi$ to be a number which is the target probability of selecting action a in any state (s.t. $1 - \pi$ is the probability of selecting $b$), and similarly for the behaviour $\mu$.

Write an expected weight update, that uses the operator function $T$ and a value function $v$ to compute the expected weight update. The expectation should take into account the probabilities of actions in the future, as well as the steady-state (=long-term) probability of being in a state. The step size of the update should be  $\alpha=0.1$.

Here is how my solution looks like (I am a total beginner in RL and in addition to studying Rich's book, I was trying to solve the basic intro course assignments as well to help understand the topic in detail.
x1 = np.array([1., 1.])
x2 = np.array([2., 1.])

def v(w, x):
    return x.T*w

def T(w, pi, mu, l, g):
    states = [0, 1]
    n_states = len(states)
    #initial_dist = np.array([[1.0, 0.0]])
    transition_matrix = np.array([[pi, 1-pi],
                                  [pi, 1-pi]])

    if pi <= mu: # thresholding to select the state
        val = v(w, x1)
    else:
        val = v(w, x2)
        pi = 1 - pi

    l_power = np.power(l, n_states - 1)
    lambda_corrected = l_power * val
    lambda_corrected *= 1 - l

    return lambda_corrected - val

def expected_update(w, pi, mu, l, g, lr):
    delta = T(w, pi, mu, l, g)

    w += lr * delta
    return w

The state diagram looks like this where there are two states $s_0$ and $s_1$. All rewards are $0$ and the state features $x_0 = x(s_0)$ and $x_1 = x(s_2)$ for two states are given as $x_1$ and $x_2$ in the code ([1., 1.], [2., 1.]) and also there are only two actions in each state $a$ and $b$. Action an always transitions to state $s_0$ (i.e. from s1 or from s0 itself) and action b always transitions to state $s_1$ (i.e. from $s_0$ or $s_1$ itself):
This is how the caller portion of the code looks like.
def caller(w, pi, mu, l, g):
  ws = [w]
  for _ in range(100):
    w = w + expected_update(w, pi, mu, l, g, lr=0.1)
    ws.append(w)
  return np.array(ws)

mu = 0.2 # behaviour
g = 0.99  # discount

lambdas = np.array([0, 0.8, 0.9, 0.95, 1.])
pis = np.array([0., 0.1, 0.2, 0.5, 1.])

I would appreciate any help.

Edit:
I tried implementing the T() following the Bellman backup operator, but I am still not sure if I did this right or not.
return pi * g*v(w, x1) + (1-pi) * g*v(w, x2)


text:
What is the difference between the forward pass of the Multi-Layer Perceptron, Deep AutoEncoder and Deep Belief Network?
Multi-Layer Perceptron (MLP), Deep AutoEncoder (DAE), and Deep Belief Network (DBN) are trained differently.
However, do they follow the same process during the inference phase, i.e., do they calculate a weighted sum, then apply a non-linear activation function, for each layer until the last layer, or is there any difference? Moreover, are they only composed of fully connected layers?

Could you please focus on the comparison of 2 models at a time rather than 3? Moreover, when you use an acronym, like DAE, you should first use the long/full version, like "muli-layer perceptron (MLP)", so that everyone knows what the acronym refers to.

I kept the three because the goal is not to know the detailed differences between the models .. but just to know if they are all based on "only" fully connected layers and thus they perform the same mathematical operations during inference phase.

I made some changes to your post in an attempt to improve the clarity. Please, check that I didn't change the meaning of the post.


text:
How to simplify policy gradient theorem to $E_{\pi}[G_t \frac{\nabla_{\theta}\pi(a|S_t,\theta)}{\pi(a|S_t,\theta)}]$?
In "Introduction to Reinforcement Learning" (Richard Sutton) section 13.3(Reinforce algorithm) they have the following equation:
\begin{align}
 \nabla_{\theta}J &\propto \sum_s \mu(s) \sum_a q_{\pi}(s,a)\nabla_{\theta}\pi(a|s,\theta) \\
                  &= E_{\pi}[\sum_a q_{\pi}(S_t,a) \nabla_{\theta}\pi(a|S_t,\theta)] \tag{1}\label{1}
\end{align}
But in my opinion equation 1 should be expectation over state distribution: $$E_{\mu}[\sum_a q_{\pi}(S_t,a) \nabla_{\theta}\pi(a|S_t,\theta)]$$
If I am right here then the rest of the lines follows like this:
\begin{align}
\nabla_{\theta}J &= E_{\mu}[\sum_a q_{\pi}(S_t,a) \nabla_{\theta}\pi(a|S_t,\theta)] \\
                  &= E_{\mu}[\sum_a \pi(a|S_t,\theta) q_{\pi}(S_t,a) \frac{\nabla_{\theta}\pi(a|S_t,\theta)}{\pi(a|S_t,\theta)}] \\
                   &= E_{\mu}[E_{\pi}[q_{\pi}(S_t,A_t)\frac{\nabla_{\theta}\pi(A_t|S_t,\theta)}{\pi(A_t|s,\theta)}]]\\
                   &= E_{\mu}[E_{\pi}[G_t\frac{\nabla_{\theta}\pi(a|S_t,\theta)}{\pi(a|S_t,\theta)}]]
\end{align}
Now the final update rule using stochastic gradient descent will be:
$$\triangle \theta = \alpha E_{\pi}[G_t\frac{\nabla_{\theta}\pi(A_t|S_t,\theta)}{\pi(A_t|S_t,\theta)}] \tag{2}$$
I think I am doing something wrong here because this equation 2 does not match with the book also with other materials. Can anyone please show me where I am doing wrong?

It's explained in the book literally above the equation: "Notice that the right-hand
side of the policy gradient theorem is a sum over states weighted by how often the states occur under the target policy $\pi$, if $\pi$ is followed, then states will be encountered in these proportions".

Can you please explain this to me ?

When the authors write go from
$$\nabla_{\theta}J \propto \sum_s \mu(s) \sum_a q_{\pi}(s,a)\nabla_{\theta}\pi(a|s;\theta)\;$$
to
$$\nabla_{\theta}J = E_{\pi}\left[\sum_a q_{\pi}(S_t,a) \nabla_{\theta}\pi(a|S_t;\theta)\right]\;$$
they are simply taking an expectation where the only random variable is the state $S_t$. This is because, as they say in the book, the Policy Gradient Theorem is a sum over a state distribution, therefore it can be written as an expectation. The expectation is over the states wrt the state distribution; the $\pi$ subscript here does not mean we are taking expectation with respect to the action policy $\pi$, they use the notation $\pi$ to emphasise that the state distribution is induced by the current policy $\pi$ -- this is what makes REINFORCE an on-policy algorithm because the state distribution with which we take the expectation must come from the current policy $\pi$, not some arbitrary distribution.
Now, as you don't seem to understand how they go from this to the Stochastic Gradient Ascent (SGA) (NOT descent as you keep referring to it as) update rule in the book, I will explain further. As you rightly say we can go from my previous equation to the following
$$\nabla_{\theta}J = E_{\pi}\left[G_t \nabla_{\theta}\log\pi(A_t|S_t;\theta)\right]\;$$
by doing some re-arranging and noting that $G_t$ is an unbiased estimate of $q_\pi(s, a)$ when taking expectation over the actions and states. Now, this expectation we have is with respect to the actions and states; your first mistake is that you have made it an expectation over the actions without signifying that the action is now a random variable (maybe you could look up expectations and random variables to understand really what this means).
Your second mistake is by not seeing how you can use this gradient to optimise the policy parameters. As we want to maximise this objective we want to perform SGA to it; that is we want to perform
$$\theta = \theta + \alpha \nabla_\theta J\;;$$
but $\nabla_\theta J$ is an expectation. It would be costly to evaluate this at every single state-action pair that we see in an episode, thus we only perform SGA for one state-action tuple at a time. Now, you might think that this is not correct, but it is in fact an unbiased estimate as we will sample lots of state-action pairs; this kind of estimation is known as Monte Carlo Sampling. Therefore, we can write our REINFORCE update rule as
$$\theta = \theta + \alpha G_t \nabla_\theta \log \pi(A_t | S_t; \theta)\;;$$
which is exactly the update rule in the book.
Note that at the start of the chapter in Sutton and Barto they also make it clear that we wish to perform
$$\theta = \theta + \alpha \hat{\nabla J(\theta)} \;;$$
where $\hat{\nabla J(\theta)}$ is a stochastic estimate whose expectation approximates the true gradient; this is exactly what we are doing, $G_t \nabla_\theta \log \pi(A_t | S_t; \theta)$ is our stochastic estimate.

Comments are not for extended discussion; this conversation has been moved to chat.


text:
How can "any process you can imagine" be thought of as function computation?
I stumbled upon this passage when reading this guide.

Universality theorems are a commonplace in computer science, so much
so that we sometimes forget how astonishing they are. But it's worth
reminding ourselves: the ability to compute an arbitrary function is
truly remarkable. Almost any process you can imagine can be thought of
as function computation.* Consider the problem of naming a piece of
music based on a short sample of the piece. That can be thought of as
computing a function. Or consider the problem of translating a Chinese
text into English. Again, that can be thought of as computing a
function.  Or consider the problem of taking an mp4 movie file and
generating a description of the plot of the movie, and a discussion of
the quality of the acting. Again, that can be thought of as a kind of
function computation.* Universality means that, in principle, neural
networks can do all these things and many more.

How is this true? How can any process be thought of as function computation? How would one compute function in order to translate Chinese text to English?

A function is simply a procedure that maps a particular input to a particular output. You put in $X$, and the function computes $Y$. Those $X$ and $Y$ can take many different forms. It could be mapping one number to another number (convert miles to kilometres), mapping sound to text (name that tune), mapping text to text (translate languages), mapping a video to text (review this movie), or mapping text to an image (draw a picture of $X$). Anytime you have a procedure that produces a fixed output based on a fixed input, it's a function.
Universality theorems guarantee that a neural network can produce an arbitrarily good approximation of any possible function. That doesn't mean it's easy, though - finding the right function that maps $X$ to $Y$ is the hard part.

To speak to your question about how Chinese to English translation can be a computation, it first requires a way to turn the base units of translation (tokens) into something computable. One basic way is to define the set of your vocabulary terms and create a gigantic matrix (typically called an embedding) with each column representing a token as well as one-hot encoded matrices to perform a selection from the matrix.
Say, if my vocabulary is ("apple", "kiwi") and I want 2-dimensional vectors for the tokens (trivially small but manageable example), you'd need a 2x2 (two tokens in two dimensions) random matrix and two one-hot vectors:

"apple" = [1 0]
"kiwi" = [0 1]

Multiplying [1 0] for instance, by your 2x2 matrix, will "select" the vector in the first column, which represents the token "apple".
Once you have a randomly initialized embedding matrix, you have to train it to be useful. A relatively common method to do so is to make it the hidden layer in a neural network, mask tokens in the source data, and train the model with gradient descent to "guess" what token was removed. You can also just fully train the embedding as part of training a larger network, but that increases training time significantly.
If you have a lot of sentence pairs of English-Chinese translations, you could train an English/Chinese joint embedding and then further train a neural network that uses it to translate between sentence pairs (but this is not a state of the art method).
Every step of the process after text preparation here is a mathematical operation, so a forward pass of the trained model can be expressed as an equation (though one you could never fully write out by hand), and if we're careful to only choose differentiable operations, then the training of the model as well comes down to solving (many, massive) equations.


text:
Is the target assumed to be a noisy version of the output of the model in machine learning?
I wonder if the following equation (you can find it in almost every ML book) refers to a general assumption that we make when using machine learning:
$$y = f(x)+\epsilon,$$
where $y$ is our output, $f$ is e.g. a neural network and $\epsilon$ is an independent noise term.
Does this mean that we assume the $y$'s contained in our training data set come from a noised version of our network output?

Not necessarily. The neural network (or whatever else you use) is a model of what you are trying to do, and usually models are not able to perfectly model reality, as it is too complex. A noise term is generally used to represent that, ie the imperfection of the model's relationship with the actual world.

That equation is just an assumption that we make about the relationship between a response variable (aka dependent variable) $y$ and a predictor (aka independent variable) $x$, i.e. the response variable (target) is an unknown function $f$ of the predictor $x$ plus some noise $\epsilon$ due to e.g. measurement errors (caused e.g. by damaged sensors). So, if you have a dataset $D = \{(y_i, x_i)\}_{i=1}^N$, you assume that $y_i = f(x_i) + \epsilon, \forall i$. The goal (in supervised learning) is then to estimate $f$ with e.g. a neural network $\hat{f}_\theta$, so the goal is to find a function $\hat{f}_\theta$ such that $\hat{f}_\theta(x_i) = y_i$, so, in practice, you often ignore $\epsilon$ because that is associated with irreducible errors.
You can find that equation on page 16 of the book An Introduction to Statistical Learning. There you will also find more info about the goal of (statistical) supervised learning and why $\epsilon$ is irreducible.
So, the answer to your question is no, given that $f$ there is not the neural network but an unknown function. If your neural network $\hat{f}$ was equal to $f$, then, yes, but, of course, in practice, this will almost never be the case.

thanks for answering! How does this apply to the Kalman Filter where we say that $y = Hx + r$, where r is the measurement noise. If there is no measurement noise, then the observations can be fully derived from the state?

@m_3464gh I'm not currently familiar with the details of the Kalman filter, so I can't answer your question. I was about to suggest you to ask a separate question, but you have already done that here.


text:
Can an existing transformer model be modified to estimate the next most probable number in a sequence of numbers?
Models based on the transformer architectures (GPT, BERT, etc.) work awesome for NLP tasks including taking an input generated from words and producing probability estimates of the next word as the output.
Can an existing transformer model, such as GPT-2, be modified to perform the same task on a sequence of numbers and estimate the next most probable number? If so, what modifications do we need to perform (do we still train a tokenizer to tokenize integers/floats into token IDs?)?

Are you asking whether the architecture of GPT-2 could be trained from scratch to perform the task of predicting the next number or are you asking whether the trained GPT-2 model could perform number prediction zero-shot (without any fine-tuning)?

To answer this, you need some constraints on the problem. Here are some sequences of numbers. No machine learning technique could be expected to learn all of them:

the odd numbers
the primes
numbers expressed in digits, but listed in alphabetical order of their name in German
numbers listed in the lexical order of the reverse of their representation in base 3
the phone numbers in the Manhattan phone director, listed in alphabetical order of subscriber

Would a https://en.wikipedia.org/wiki/Mixture_of_experts be an appropriate tool to deal with this scenario?

No.
The set of all sequences of whole numbers is uncountably infinite. Even the set of computable sequences of numbers is large and hard to deal with.
Like any other machine learning problem, making progress depends on prior knowledge.
Here is a subset of the problem which is hard but not impossible, and which relates to some work by Kolmogorov: a sequence of numbers is output by a program, and short programs are considered more likely than long programs. Given the first N terms, where N is large compared to the expected length of the program, predict the N+1 th term.

The odd numbers are very easy: x(n)=2n+1. If we use x(n)=x(n-1)+2, that includes the even numbers as well. If we use x(n)=2x(n-1)-x(n-2), that is all arithmetic sequences.

Yes, but this sequence is not easy: 3,5,7. Is that odd numbers>1? Or is it primes>2? Or the start of 3,5,7,3,5,7,3,5,7, ....? Or any other of an infinite number of computable sequences with that prefix?

The simplest model is for odd numbers >1, linear recurrence of order 2. With order 3 you can get 3,5,7,3,5,7... x(n)=x(n-3). No linear model for the primes.
An example that makes me crazier is 1,2,... is the simplest interpretation the counting numbers or the powers or 2? Every kid starts by associating that start with counting numbers x(n)=2x(n-1)-x(n-2), but the order of the linear recurrence for the powers of 2 is smaller: x(n)=2x(n-1)


text:
What is the difference between a distribution model and a sampling model in Reinforcement Learning?
The book from Sutton and Barto, Reinforcement Learning: An Introduction, define a model in Reinforcement Learning as

something that mimics the behavior of the environment, or more generally, that allows inferences to be made about how the environment will behave.

In this answer, the answerer makes a distinction:

There are broadly two types of model:

A distribution model which provides probabilities of all events. The most general function for this might be $p(r,s'|s,a)$ which is the probability of receiving reward $r$ and transitioning to state $s'$ given starting in state $s$ and taking action $a$.

A sampling model which generates reward $r$ and next state $s'$ when given a current state $s$ and action $a$. The samples might be from a simulation, or just taken from history of what the learning algorithm has experienced so far.

The main difference is that in sampling models I only have a black box, which, given a certain input $(s,a)$, generates an output, but I don't know anything about the probability distributions of the MDP. However, having a sampling model, I can reconstruct (approximately) the probability distributions by running thousands of experiments (e.g. Monte Carlo Tree Search).
On the other hand, if I have a distribution model, I can always sample from it.
I was wondering if

what I wrote is correct;

this distinction has been remarked in literature and where I can find a more in-depth discussion on the topic;

someone has ever separated model-based algorithms which use a distribution model and model-based algorithms which use only a sampling model.

I think that your description is roughly correct, but I wouldn't call a "sampling model" a "model" because it doesn't necessarily model something, unless, for example, you are first learning in simulation to later be able to act in the real-world or environment (in this sense, the simulation would be a model of the real environment, but this does not have to be the case, i.e. you may just want to act in the simulation (e.g. Atari games)), or, alternatively, when it's really a model of the MDP, but, in that case, you can just call it a model estimate.
So, you can call it a

sampling function, in case you sample e.g. from the experience replay
environment function, in case $r$ and $s'$ are returned by the environment,
model estimate, in case it's an estimate of $p(s' \mid s, a)$ (people may consider an experience replay a model estimate or, at least, information that can be used to build a model estimate)

The important thing to keep in mind is that, if you want to take an action $a$ in a certain state $s$, you need a function that returns you a reward $r$ and next state $s'$, if you want to do reinforcement learning.
I don't know if this distinction has been emphasized in the literature, but, as you noted, you can learn/estimate a (transition) model by exploring the world. I had asked a related question here a few years ago. You can also estimate the reward function, which is sometimes incorporated in the "model" of the environment, which, in this case, is denoted as $p(s', r \mid s, a)$ rather than just $p(s' \mid s, a)$, but these terms can be written as a function of each other.
People may also confuse this environment function with an exploratory policy, as they are, in a way, both used for exploration, but I think the concepts are distinct enough, as an exploratory policy is a way of deciding how to act given your current knowledge or ignorance: the exploratory policy can be viewed as a way of exercising/calling the environment function.

Thank you for the answer, @nbro. I also wouldn't a "sampling model" a "model",   but it seems to me that this concept falls within the definition of a "model" given by Sutton and Barto, which I quoted in the question.
I appreciate your comments. That said, you didn't fully answer to questions 2 and 3, that were the core of the question :)

@A.Pesare Why does the "sampling model" fall into the category of "model"? A model in RL usually refers to a probability distribution that describes the dynamics of the environment, i.e. $p(s' \mid s, a)$. It only makes sense to call something a "sampling model" if that something is really a model or model estimate (as I call it) and if there's sampling involved (but you can also sample from $p(s' \mid s, a)$, so why not calling it also a sampling model?), if we want to be consistent with Sutton and Barto.

I've read many RL papers and I've never seen the word "sampling model" being used, as far as I remember. The closest to that is calling the experience replay technique a "model", but in its rawest form an ER is not a model, it's just a dataset, which you could use to build a model estimate. The word "sampling" is being used probably because we can "sample" from the "experience replay", which would be used to build the model, but this is a stretch. Again, the ER alone is not a model, it's a buffer/array/list/dataset of data, but it may induce a model.

If we follow the vague definition of a model that you quote "something that mimics the behavior of the environment, or more generally, that allows inferences to be made about how the environment will behave.", then yes you could consider an ER a model, but why call it a sampling model? As I said, yes, sometimes people may say that the ER is a model, but, in my view, ER is not directly a model, it's a dataset. It would be like calling a dataset in supervised learning a model too, but, in this case, it would be a different model. It doesn't make sense. The dataset is used to build the model.

By the way, in chapter 8 (p. 159) of Sutton & Barto's book, they also mention these distribution model and sampling model.


text:
Is it possible that the model is overfitting when the training and validation accuracy increase?
I am aware of similar questions that have been asked, and I have gone through many. I want to bring my case to SE to understand better what my results are.
I am working with a large dataset (around 75million records), but, for the purpose of testing techniques, I am actually using 2M records. I am working towards malicious traffic identification using NetFlow data. After employing some undersampling to have a balanced dataset according to my target variable (benign or attack) I have 1,240,950 of records in the training set and 310,238 in the validation set. Therefore I believe there is a good amount of data to train a Deep neural network properly.
After using Yeo-Yohnsons transform and standardizing the data, I train the network with a very basic model:
def basem():

    model = Sequential()

    model.add(Dense(25, input_dim=38))
    model.add(Activation("relu"))

    model.add(Dense(50))
    model.add(Activation("relu"))

    model.add(Dense(50))
    model.add(Activation("relu"))

    model.add(Dense(25))
    model.add(Activation("relu"))

    model.add(Dense(1, activation='sigmoid'))
    model.compile(loss='binary_crossentropy', optimizer="adam", metrics=['accuracy'])
    return model

model_base = basem()
model_base._name = 'base'

history_base = model_base.fit(X_train, y_train, batch_size=2048,
                    epochs=15, validation_data=(X_val,y_val), shuffle=True)

This gives me the following plot
It maybe because I am a newbie, but this plot looks too perfect. It is weird to see validation and training accuracy growing together, although I believe this is what we want right? But now I have the feeling it is overfitting. Therefore I use the model and a 5-fold cross validation to understand how well it generalizes. Results, mean accuracy and mean std(%), are:
test acc: 0.9816503485233088
test_prec: 0.9840033637114158
test_f1: 0.9816046990113001
test_recall: 0.9792384866432975
test_roc_auc: 0.9980004347946355

Dev acc: 0.052931962886091546
Dev prec: 0.2854656099314699
Dev f1: 0.057228805478181974
Dev recall: 0.3597811552056071
Dev roc auc: 0.0036456892671197097

If I understand correctly, accuracy is high which is generally good and the standard deviation is very low for each metric, the highest being 0.359% for recall. Does this mean my model generalizes well?
Edit
Adding dropout (0.3) to each layer yields the following:

Now, my validation accuracy is higher than my training. I can't make sense of any of this.

I'll try to answer on more general questions

Is it ok that model performs better on validation, then on train?

It's certainly fine if you use techniques like dropout or data augmentation and the difference is not that big. Because in case of dropout for train you use part of the network, and for validation the whole.

I'm suspicious my model is too good. What could I do?

That a good point, because in my experience too good results sometime are reasoned by the flow in the training. The most common is data leakage, that's mean you give the model a way to fool around in unexpected and unwanted way.
Let me give an example. Lets imagine you try to detect malicious traffic and you have a data about sessions and you split records on train/val. Let's imagine one of the features is IP address and you could have multiple sessions from same IP. But if same IP would be in both train and val for the crooked users, then model could just remember the 'bad' IP and thus getting high score on both train and val.
So, to avoid it you need to think about things like that. If records could be group in some too meaningful way, make sure you put the whole group either in train or val. Another thing, that's it's usually good thing to split it by time as well, i.e. use for train records for 2018 and 2019 and for val for 2020. Thus you not only avoid data leakage, but you make sure your model is robust for future predictions.
More general, try to understand, why you getting such good result with such simple model. Try to squeeze the data even smaller, drop some features to understand, when it would go done.
Finally I would recommend the post by Andrej Karpathy, when he gave lots of really good advices on practical side of training NN. http://karpathy.github.io/2019/04/25/recipe/

great answer thanks a lot. Ill just note that indeed I did drop "ip addresses" exactly due to what you explained. I, however, did not drop port numbers - ill try to remove them and see.


text:
For image preprocessing, is it better to use normalization or standartization?
For a neural network model that classifies images, is it better to use normalization (dividing by 255.0) or using standardization (subtract mean and divide by STD)?
When I started learning convolutional neural networks, I always used normalization because it's simple and effective, but then I started to learn PyTorch and in one of the tutorials https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html they preprocess images like this:
transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)

The transform object is created, which has the NORMALIZE parameter, which in itself has the mean and STD values for each channel.
At first, I didn't understand how this works, but then learned how standardization works from Andrew Ng's video, but I didn't find the answers to why is it better to use standardization over normalization or vice-versa. I understand that normalization scales inputs from [0, 1], and standardization first subtracts mean, so that dataset would be centered around 0, and divides everything by STD, so that it would normalize the variance.
Though I know how each of these techniques work (I think I know), I still don't understand why would anybody use one over the other to preprocess images.
Could anybody explain where and why would you use normalization or standardization (if possible could you give an example)? And as a side question: is it better to use the combined version where first you normalize the image and then standardize it?


text:
How does the K-dimensional WL test work?
I am reading a paper on the K-WL GCN. I did not complete the paper yet, but I just skimmed over it. There I am trying to understand the K-WL test (page 3 Weisfeiler-Leman Algorithm). I think my understanding is quite ambiguous, so I looking for an example problem that is solved using K-WL test. But I can't find any of them on the web.
Does anyone have any solved example problem on K-WL or can anyone explain to me how the K-WL test works?
Note: If anyone also explains how K-WL GCN uses the K-WL test, I will be thankful.

I never used a k-WL in practice, but I did apply weisfeiler-lehman for my graph tasks.
As you can know, the WL provides the coloring by interactive procedure that's assign each node a 'color' (basically some kind of label reflecting the node neighborhood). Counting colors allows to compare two graphs on isomorphism, but it's not that important here, the key is that you have kind of feature for each node.
The k-WL make similar thing, but it operates not on graph, but in space of k-tuples of nodes. Basically, it's pairs or triplets of nodes. Then you define neighborhood as all pairs/triplets that differ in one element from the target pair/triplet. So, now you have have some kind of objects and defined neihgborhood for each of them, that's basically enough to apply the WL coloring.
Ok, now to k-GNN part. You repeat k-WL procedure on pairs of nodes, gets color for each of pair. You use one-hot encoded colors as input features and make graph convolution for pairs (with neighborhood defined as above), then getting some kind of embeddings for pairs. To get the node classification, you make average pooling by all pairs containing the node and then use usual fully-connected neural net.
As of the examples, the paper author provided several in article repo, you can try them.

Thanks for the answer. I have few questions regarding the answer:1)Is the neighborhood of a $K$-triplet must belong to the set $V^K$. 2) Does the ordering of the triplet (for example (a,b,c) or (b,c,a)) matter? 2)In K-WL GCN, How we define input features for $K$-triplet since we have only input features for each node?

Yes 2.1. I think it shouldn't, but I don't know for sure, you could check the implementation 2.2. If you mean the features of the nodes I didn't see the precise procedure in paper, but from the code, as I can understand (hierarchical model): make node convolutions, get embeddings, average from the pair, concatenate the WL one hot encoded features

You could also try to ask questions in github issues of the repo, Matthias Fey (one of the authors) answers them there.


text:
Are there relatively new research papers that describe how to make back-propagation more efficient?
I read Yann LeCun's paper Efficient BackProp, which was published in 2000. I looked for similar but more recent papers on Arxiv, but I have not yet found any.
Are there relatively new research papers that describe how to make back-propagation more efficient?
So, I am looking for papers similar to Efficient Backprop by LeCun but newer. The papers could describe why ReLU now "dominates" tanh or even sigmoid (but tanh was Yann's favorite, as explained in the paper). ReLU is just one thing I am interested in, but the paper could also analyze e.g. the inputs from a statistical standpoint.

A related question https://stats.stackexchange.com/questions/126238/what-are-the-advantages-of-relu-over-sigmoid-function-in-deep-neural-networks

Here is a paper that explains why ReLU rules.
What we want is to disentangle data of different classes. In order to do that, we need a discontinuous mapping for the data. ReLU easily allows for that. It is even better than LeakyReLU, sigmoid and tanh in that regard. Also, the reason any of the activations work is because of the floating point error, there is inadvertently a discontinuous mapping for the whole data. I have also explained it here.

I think you meant to say non-homeomorphic mapping rather than discontinuous, as noted sect. 6.2 of the paper by Naitzat et al. you linked.  The $\mathrm{ReLU}$ mapping is continuous, of course.

Is it? What happens at 0? If I can't map A to B, then, that creates a non-homeomorphic mapping, is that also not discontinuous? Isn't non-homeomorphism discontinuity in terms of topological spaces.

$\mathrm{ReLU}$ isn't differentiable at zero, but it is continuous. Roughly, a continuous function is one that "you can draw without lifting your pen" (the actual definition is more formal of course!). The reason that $\mathrm{ReLU}$ is not a homeomorphism is because it doesn't have a well-defined inverse (lots of values map to 0), and a homeomorphism is required to be invertible, and have a continuous inverse. That's why the authors contrast with the sigmoid, which does have an inverse.

But, what it means physically is that sigmoid will always remain a lake and you would not be able to cut water. On the other hand, ReLU will be able to create those boundaries since it nullifies everything. Since, topology is the study of shapes. How would you explain this to a novice, the difference between homeomorphism and non-homeomorphism?


text:
Is there a way of path reconstruction using only the history of belief states?
Given a history of belief states, is there a common method that backtracks the most likely path of ending up in the current belief state?
I have a Markov model which calculates belief states after every step. The belief state is a representation of the most likely states one could be in. A belief state may look like this:
$$b=[1,0,0,0,0],$$ where I am in the state $s_0$ with 100% certainty.
I can store the belief state history like $b_0, b_1, b_2,\dots, b_n$.
Is there a common way to represent and estimate the most likely states one has been in?
A naive approach could be to just look for the state with the highest value per belief state and take that as the node along the reverse path. But I am not confident enough, if that is a common and a good practice, as it is not considering the fuzziness, which comes with a belief state. But then again, if I would take all states that are bigger than 0, I might not know which state leads to which state and if that transition is even possible.

The belief state in a POMDP is a distribution over the hidden state given all past actions and observations, i.e., at time $k$, the belief state is $b_k(s_k) \triangleq P(s_k \mid a_{0:k-1}, z_{1:k})$, where $a$ and $z$ are the actions and observations, respectively.
What you are asking about and calling "backtracking" boils down to the question: "given that at time $k$, I know the history of actions $a_{0:k-1}$ and observations $z_{1:k}$, what is the distribution over states at some past time step $t<k$, that is, $P(s_t\mid a_{0:k-1},z_{1:k})$"?
This is commonly known as Bayesian smoothing. You might be familiar with Bayesian filtering, which aims to recover the belief state as defined in the first paragraph - an estimate of the current state given actions and observations up to the current time. Smoothing also uses information gained after time $t$ to estimate the state at time $t$, and hence it is only possible after those actions and observations become known. There is also prediction, where you estimate the distribution over the state in the future.
Smoothing is too broad of a topic to go into detail here, but below is a picture from the book "Bayesian Filtering and Smoothing" by Simo Särkkä that illustrates the concepts. You could start there for further information on the topic.


text:
In Probabilistic Graphical Model (written by Daphne Koller), what's the meaning of "parameter" in representation of the distribution?
I just started to read the PGM book written by Daphne Koller.
In the chapter of Bayesian Network Representation(Chapter 3), there are some descriptions about the standard parameterization of the joint distribution corresponding to n-trial coin tosses.

The book also says,

Here I'm very confused about the meaning of $ 2^n parameters $. In terms of random variable or probability distribution, parameter means characteristic of the distribution. But parameter in this paragraph sounds like $O(2^n)$ space complexity. Because it also describes that we can reduce the space of all joint distribution to $n$-dimension by using expression $ \prod_{i} \theta_{x_{i}} $.

So, what's the meaning of parameter in this context? Does it mean space complexity for computation of the joint distribution?

I believe you are correct. When Koller mentions a parameter, it is in reference to the values needed to express the probability distribution. For example, the total number of parameters for a Bernoulli distribution is one, the probability of success. You can watch Nando de Freitas's lecture on undergraduate machine learning 8: Inference in Bayesian networks and dynamic programming for a detailed walkthrough of a binary Bayesian network where Nando also explains how many parameters are needed for each table in the network. In the lecture, he also briefly discussed some implementation details for Bayesian Networks, which also hints toward answering your question.


text:
Can Facebook's LASER be used like BERT?
Can Facebook's LASER be fine-tuned like BERT for Question Answering tasks or Sentiment Analysis?
From my understanding, they created an embedding that allows for similar words in different languages to be close to each other. I just don't understand if this can be used for fine-tuning tasks like BERT can.

LASER creates multilingual contextualized word embeddings, what you do with them is up to you. You can use this as a feature extraction and add whatever you want to the end of the network.
I believe the implementation by facebook does not let you change the weights of the LASER model itself, they are froozen to the best of my knowledge.
So, yes, you can use LASER for downstream tasks, but you cannot change the actual weights of the LASER model.


text:
Does the Bayesian MAP give a probability distribution over unseen t*?
I'm working my way through the Bayesian world. So far, I've understood that the MLE or the MPA are point estimates, therefore using such models just output one specific value and not a distribution.
Moreover, vanilla neuronal networks do in fact s.th. like MLE, because minimizing the squared-loss or the cross-entropy is similar to finding parameters that maximize the likelihood. Moreover, using neural networks with regularisation is comparable to the MAP estimates, as the prior works like the penalty term in error functions.
However, I've found this work. It shows that the weights $W_{PLS}$ gained from a penalized least-squared are the same as the weights $W_{MAP}$ gained through maximum a posteriori:

However, the paper says:

The first two approaches result in similar predictions, although the MAP Bayesian model does give a probability distribution for $t_*$ The mean of this distribution is the same as that of the classical predictor $y(x_*; W_{PLS})$, since $W_{PLS} = W_{MAP}$

What I don't get here is how can the MAP Bayesian give a proability distribution over $t_*$, when it is only a point estimate?
Consider a neuronal network - a point estimate would mean some fixed weights, so how can there be a output probability distribution? I thought that this is only achieved in the true Bayesian, where we integrate out the unknown weights, therefore building something like the weight averaged of all outcomes, using all possible weights.
Can you help me?

You're correct: the MAP estimate is a point estimate (specifically, MAP is used to estimate the mode of a probability distribution).
I think that the paper is referring to the (output) probability distribution over the possible targets/labels, given the point estimate. However, in the case of MLE, you can also have that probability distribution, so I'm not sure why the paper's author is emphasizing that with MAP you can build that probability distribution (so maybe this/my interpretation of that excerpt is wrong!).
That table also shows that the MAP estimate is used to produce a probability distribution, but the $\sigma$ there should be unknown, but I didn't read that article, so maybe I am missing some info or assumption.
In any case, you could also find a point estimate of a parameter of a probability distribution, but this does not imply that MAP produces a probability distribution. For instance, you can show that, if you place a Gaussian prior on the weights of a neural network, this leads to the $L_2$ loss function, but training a (normal) neural network with $L_2$ does not lead to a probability distribution over the weights.
You should try reading only information from reliable references and books. For MAP, check out chapter 5. (p. 149) of the book "Machine Learning: A Probabilistic Perspective" by Murphy.


text:
In gradient descent's update rule, why do we use $\sigma(z^{l-1})\frac{\delta C_0}{ \delta w^{l}}$ instead of $\frac{\delta C_0}{\delta w^{l}}$?
I am trying to code a two layered neural network simple NN as I have described here https://itisexplained.com/html/NN/ml/5_codingneuralnetwork/
I am getting stuck on the last step of updating the weights after calculating the gradients for the outer and inner layers via back-propagation
#---------------------------------------------------------------

# Two layered NW. Using from (1) and the equations we derived as explanations
# (1) http://iamtrask.github.io/2015/07/12/basic-python-network/
#---------------------------------------------------------------

import numpy as np
# seed random numbers to make calculation deterministic
np.random.seed(1)

# pretty print numpy array
np.set_printoptions(formatter={'float': '{: 0.3f}'.format})

# let us code our sigmoid funciton
def sigmoid(x):
    return 1/(1+np.exp(-x))

# let us add a method that takes the derivative of x as well
def derv_sigmoid(x):
   return x*(1-x)

# set learning rate as 1 for this toy example
learningRate =  1

# input x, also used as the training set here
x = np.array([ [0,0,1],[0,1,1],[1,0,1],[1,1,1]  ])

# desired output for each of the training set above
y = np.array([[0,1,1,0]]).T

# Explanaiton - as long as input has two ones, but not three, ouput is One
"""
Input [0,0,1]  Output = 0
Input [0,1,1]  Output = 1
Input [1,0,1]  Output = 1
Input [1,1,1]  Output = 0
"""

input_rows = 4
# Randomly initalised weights
weight1 =  np.random.random((3,input_rows))
weight2 =  np.random.random((input_rows,1))

print("Shape weight1",np.shape(weight1)) #debug
print("Shape weight2",np.shape(weight2)) #debug

# Activation to layer 0 is taken as input x
a0 = x

iterations = 1000
for iter in range(0,iterations):

  # Forward pass - Straight Forward
  z1= x @ weight1
  a1 = sigmoid(z1)
  z2= a1 @ weight2
  a2 = sigmoid(z2)

  # Backward Pass - Backpropagation
  delta2  = (y-a2)
  #---------------------------------------------------------------
  # Calcluating change of Cost/Loss wrto weight of 2nd/last layer
  # Eq (A) ---> dC_dw2 = delta2*derv_sigmoid(z2)
  #---------------------------------------------------------------

  dC_dw2  = delta2 * derv_sigmoid(a2)

  if iter == 0:
    print("Shape dC_dw2",np.shape(dC_dw2)) #debug

  #---------------------------------------------------------------
  # Calcluating change of Cost/Loss wrto weight of 2nd/last layer
  # Eq (B)---> dC_dw1 = derv_sigmoid(a1)*delta2*derv_sigmoid(a2)*weight2
  # note  delta2*derv_sigmoid(a2) == dC_dw2
  # dC_dw1 = derv_sigmoid(a1)*dC_dw2*weight2
  #---------------------------------------------------------------

  dC_dw1 =  (np.multiply(dC_dw2,weight2.T)) * derv_sigmoid(a1)
  if iter == 0:
    print("Shape dC_dw1",np.shape(dC_dw1)) #debug

  #---------------------------------------------------------------
  #Gradinent descent
  #---------------------------------------------------------------

  #weight2 = weight2 - learningRate*dC_dw2 --> these are what the textbook tells
  #weight1 = weight1 - learningRate*dC_dw1

  weight2 = weight2 + learningRate*np.dot(a1.T,dC_dw2) # this is what works
  weight1 = weight1 + learningRate*np.dot(a0.T,dC_dw1)

print("New ouput\n",a2)

Why is
  weight2 = weight2 + learningRate*np.dot(a1.T,dC_dw2)
  weight1 = weight1 + learningRate*np.dot(a0.T,dC_dw1)

done instead of
  #weight2 = weight2 - learningRate*dC_dw2
  #weight1 = weight1 - learningRate*dC_dw1

I am not getting the source of the equation of updating the weights by multiplying with the activation of the previous layer
As per gradient descent, the weight update should be
$$
  W^{l}_{new} = W^{l}_{old} - \gamma * \frac{\delta C_0}{\delta w^{l}}
$$
However, what works in practice is
$$
 W^{l}_{new} = W^{l}_{old} - \gamma * \sigma(z^{l-1})\frac{\delta C_0}{ \delta w^{l}},
$$
where $\gamma$ is the learning rate.

Have you read something about back-propagation before asking this question? It seems that you're asking us how is back-propagation done because you're asking us to explain a certain implementation.

sorry if this is a bit silly; yes - I have derived it and trying to code through (this is what I have done so far https://itisexplained.com/html/NN/ml/5_codingneuralnetwork/)
Wl−1new=Wl−1old−learningRate∗δC0/δwl−1 - is what I have understood the gradeint descent part

So, is your question: why do they use np.dot(a0.T,dC_dw1) rather than dC_dw1  in the update rule? Maybe you can explain these symbols and maybe the code is not strictly necessary and you can write the equations that they are using in the implementation with latex/mathjax.

It may also be a good idea to describe the neural network, the loss function, the task you're solving, etc. Back-propagation can change depending on these parameters.

Okay - I guess this is the matrix calculus part - https://explained.ai/matrix-calculus/#sec6.2 need to check this more

I guess the mistake I did was in doing derivation of sigmoid instead of partial derivation - in partial derivation it is https://math.stackexchange.com/a/2219904/284422 $S(x)(1-S(x)).x and since it is matrix calculus that last term will be transpose

Okay - the answer is here  https://explained.ai/matrix-calculus/#sec6.2 and it is pretty involved. Basically, there is a difference when you derive the equation for one neuron and when you have to do practically for a set of neurons. The answer is matrix calculus. Here goes from what I could make out. Feel free to correct if I am wrong
Gradient Vector/Matrix/2D tensor of Loss function wrto Weight
$$
C = \frac{1}{2} \sum_j (y_j-a^L_j)^2
$$
Assuming a neural net with 2 layers, we have the final Loss as
$$
C = \frac{1}{2} \sum_j (y_j-a^2_j)^2
$$
Where
$$
a^2 = \sigma(w^2.a^1)
$$
We can then write
$$
C = \frac{1}{2} \sum_j v^2 \quad \rightarrow (Eq \;A)
$$
Where
$$
v= y-a^2
$$
Partial Derivative of Loss function wrto Weight
For the last layer, lets use Chain Rule to split like below
$$
\frac {\partial C}{\partial w^2} = \frac{\partial v^2}{\partial v} * \frac{\partial v}{\partial w^2} \quad \rightarrow (Eq \;B)
$$
$$
 \frac{\partial v^2}{\partial v} =2v \quad \rightarrow (Eq \;B.1)
$$
$$
\frac{\partial v}{\partial w^2}=  \frac{\partial (y-a^2)}{\partial w^2} =
0-\frac{\partial a^2}{\partial w^2} \quad \rightarrow (Eq \;B.2)
$$
$$
\frac {\partial C}{\partial w^2} = \frac{1}{2} *2v(0-\frac{\partial a^2}{\partial w^2}) \quad \rightarrow (Eq \;B)
$$

Now we need to find $\frac{\partial a^2}{\partial w^2}$
Let
$$
a^2= \sigma(sum(w^2 \otimes a^1 )) = \sigma(z^2)
$$
$$
z^2 = sum(w^2 \otimes a^1 )
$$
$$
z^2 = sum(k^2) \; \text {where} \; k^2=w^2 \otimes a^1
$$
We now need to derive an intermediate term which we will use later
$$
\frac{\partial z^2}{\partial w^2} =\frac{\partial z^2}{\partial k^2}*\frac{\partial k^2}{\partial w^2}
$$
$$
=\frac {\partial sum(k^2)}{\partial k^2}* \frac {\partial (w^2 \otimes a^1 )} {\partial w^2}
$$
$$
\frac{\partial z^2}{\partial w^2} = (1^{\rightarrow})^T* diag(a^1) =(a^{1})^T \quad \rightarrow (Eq \;B.3)
$$
How the above is, you need to check this  in https://explained.ai/matrix-calculus/#sec6.2
Basically though these are written like scalar here; actually all these are partial differention of vector by vector, or vector by scalar; and a set of vectors can be represented as the matrix here.
Note that the vector dot product $w.a$ when applied on matrices becomes the elementwise multiplication $w^2 \otimes a^1$ (also called Hadamard product)
Going back to  $Eq \;(B.2)$
$$
\frac {\partial a^2}{\partial w^2} = \frac{\partial a^2}{\partial z^2} * \frac{\partial z^2}{\partial w^2}
$$
Using $Eq \;(B.3)$ for the term in left
$$
=  \frac{\partial a^2}{\partial z^2} * (a^{1})^T
$$
$$
=  \frac{\partial \sigma(z^2)}{\partial z^2} * (a^{1})^T
$$
$$
\frac {\partial a^2}{\partial w^2} =   \sigma^{'}(z^2) * (a^{1})^T \quad \rightarrow (Eq \;B.4)
$$
Now lets got back to partial derivative of Loss function wrto to weight
$$
\frac {\partial C}{\partial w^2} = \frac {1}{2}*2v(0-\frac{\partial a^2}{\partial w^2}) \quad \rightarrow (Eq \;B)
$$
Using $Eq \;(B.4)$ to substitute in the last term
$$
= v(0- \sigma^{'}(z^2) * (a^{1})^T)
$$
$$
= v*-1*\sigma^{'}(z^2) * (a^{1})^T
$$
$$
= (y-a^2)*-1*\sigma^{'}(z^2) * (a^{1})^T
$$
$$
\frac {\partial C}{\partial w^2}= (a^2-y)*\sigma^{'}(z^2) * (a^{1})^T
$$


text:
What are some most promising ways to approximate common sense and background knowledge?
I learned from this blog post Self-Supervised Learning: The Dark Matter of Intelligence that

We believe that self-supervised learning (SSL) is one of the most promising ways to build such background knowledge and approximate a form of common sense in AI systems.

What are other most promising ways which would be competitive to self-supervised learning?
I only know the knowledge base, but I don't think it would be that promising due to the curation problem of the large scale Automated Knowledge Base Construction.

Note that, in knowledge representation and reasoning, common-sense knowledge is traditionally represented as sentences (in logic). For example, one possible sentence that you could store in a knowledge base is

The earth revolves around the sun.

This is common-sense knowledge (ignoring the fact that this was not the case in the past until Copernicus and Galileo came along in the 1500s). The programming language PROLOG is based on this type of knowledge, facts, and deduction.
Self-supervised learning (SSL) has been used to learn representations of data (which is often done in the context of natural language processing), but these representations may not be knowledge, in the sense that we may not understand what is encoded in these representations or whether they are related to our common-sense knowledge.
So, whether or not SSL can be used for acquiring/approximating knowledge depends on the definition of knowledge that you use.
If you use the traditional definition of common-sense knowledge, SSL is not usually used for knowledge representation, but it has been used in the context of knowledge graphs (see the SelfLinKG approach), which can be viewed as a graphical representation of a knowledge base. So, it's possible that SSL can also be useful for approximating common-sense knowledge.
To answer your question more directly, my impression is that knowledge graphs are promising and they are useful in practice. Google uses them in its search engine (and probably also in the Google Assistant). Whenever you search, for example, for a famous person (for example, Gandhi), on the right side, you should see a window describing certain details of that famous person. That's done using Google's knowledge graph.
Right now, people are trying to develop techniques to learn embeddings of the nodes or relations in a knowledge graph, with the goal of using these embeddings for discovering new knowledge. This area is called knowledge graph embedding (KGE). My other answer provides more details (but there are many surveys on KGEs that you can find, for instance, on Google Scholar).
Having said this, knowledge representation has been a big problem since the early days of AI (for example, see the frame problem) and there hasn't been much progress in this area (AFAIK). Most people are focusing on neural networks right now, but explainable AI techniques for neural networks could potentially play a role in knowledge representation.


text:
Does a trajectory in reinforcement learning contain the last action?
From what I learn from CS285 and OpenAI's spinning up, a trajectory in RL is a sequence of state-action pairs:
$$\tau = \{s_0, a_0, ..., s_t, a_t\}$$
And the resulting trajectory probability is:
$$ P(\tau \mid \pi)=\rho_{0}\left(s_{0}\right) \prod_{t=0}^{T-1} P\left(s_{t+1} \mid s_{t}, a_{t}\right) \pi\left(a_{t} \mid s_{t}\right) $$

From CS285: http://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-4.pdf

From spinning up: https://spinningup.openai.com/en/latest/spinningup/rl_intro.html#trajectories

However, from my derivation, the above trajectory probability actually corresponds to the following sequence where the last action $a_t$ is absent:
$$ \tau = \{s_0, a_0, ..., s_t\} $$
Can someone please help me clarify this confusion?

If we use $T$ as the notation for the terminal state, then the last action is $a_{T-1}$. This is because when you reach state $s_T$ you don't take another action, which would be $a_T$, because the episode is finished upon reaching the terminal state.