text:
Does it make sense to compare images (samples) with words (features)?
Consider the following paragraphs from the introduction of the chapter named Recurrent Neural Networks from the textbook titled Dive into Deep Learning

So far we encountered two types of data: tabular data and image data.
For the latter we designed specialized layers to take advantage of the
regularity in them. In other words, if we were to permute the pixels
in an image, it would be much more difficult to reason about its
content of something that would look much like the background of a
test pattern in the times of analog TV.
Most importantly, so far we tacitly assumed that our data are all
drawn from some distribution, and all the examples are independently
and identically distributed (i.i.d.). Unfortunately, this is not true
for most data. For instance, the words in this paragraph are written
in sequence, and it would be quite difficult to decipher its meaning
if they were permuted randomly. Likewise, image frames in a video, the
audio signal in a conversation, and the browsing behavior on a
website, all follow sequential order. It is thus reasonable to assume
that specialized models for such data will do better at describing
them.

In neural networks, we generally use words: instances, examples, data points to refer to a particular row of a dataset. In general, in the case of CNN, an instance will be an image, and in the case of RNN, an instance will be a sequence of text (maybe a sentence, paragraph, or text).
Every instance contains features: in image data, pixels are generally treated as features and in the case of text data, either characters or words are generally treated as features.
With this context, let me explain my doubt
I have an issue understanding the paragraphs. The issue is the comparison of examples/instances of image data with features of text data. Pixels can be compared to words/characters and images can be compared to sentences or words or paragraphs or text documents based on the context.
In the second paragraph, it is said that the examples (images) are i.i.d. But then the images are compared with words in a paragraph, but words are features, not examples. As the paragraph is not saying that pixels are i.i.d., how can the words in paragraphs be used for comparison?

I am not sure I understand your question. I understand it has to do with images, words, sentences, being iid or not, and to compare words with images, but I don't understand what the specific question is. I don't understand this "As the paragraph is not saying that pixels are i.i.d. how can the words in paragraphs be used for comparison?". I mean, why do you need both the pixels and the words to be iid to compare them? If pixels and words are features, then you can compare them, in the sense that you can say that both are features. What do you mean by "compare"?

@nbro in the second paragraph, they told that examples are iid (w.r.t image data). But they are talking about the words in paragraphs. How to reconcile?

@nbro If they say pixels are iid then they can compare with words. But they are saying images are iid then how can they compare with words instead of paragraphs?

It seems to me you think that words are "features" and that we can't compare words with images because images are "samples" and not features. Is this your question? I would recommend that you edit your post to remove unnecessary details and just leave specific question that highlights your doubt/problem.

@nbro Yeah exactly.

Thanks. I've rewritten the title to make it clearer what the problem is. Make sure that's your question. Feel free to edit again your problem to improve it ;)

Thanks @nbro After you decreased activity, I also decreased involuntarily :(

It's fine. We're all volunteers here, so we're not forced to be here. Nobody will pay you for being here. We're here because we believe in or like this community. I needed this break from AISE and to lower my activity here for a while. When you're a mod, people will attack you only because you're a mod and because you take a decision that is not consistent with their expectations. I don't want to deal with toxic users. That made my life worse and more stressful. I can avoid this stress!

For now, I prefer to be a regular user and help whenever I can (especially, edits, clarifications and answer questions that can be answered quickly), without having to deal with problematic users. I think that every small contribution can be helpful. Voting, edits, commenting, asking and answering questions or reviewing the queues. You don't have to do everything. If we're about 10 active users every day (and they don't always have to be the same), and everyone does a little, the community should be able to self-maintain, assuming that our general activity doesn't increase much.


text:
Is this aggregation of multiple convolutions of the same input a type of attention or dynamic convolution?
Are there any examples of people performing multiple convolutions at a single depth and then performing feature max aggregation as a convex combination as a form of "dynamic convolutions"?
To be more precise: Say you have an input x, and you generate
Y_1 = conv(x)
Y_2 = conv(x)
Y_3 = conv(x)

Y = torch.cat([Y_1,Y_2,Y_3])
Weights = nn.Parameter(torch.rand(1,3))
Weights_normalized = nn.softmax(weights)
Attended_features = torch.matmul(Y, weights_normalized.t())

So, essentially, you are learning a weighting of the feature maps through this averaging procedure.
Some of you may be familiar with the "Dynamic Convolutions" paper. I’m just curious if you all would consider this dynamic convolution or attention of feature maps. Have you seen it before?
If the code isn’t clear, this is just taking an optimized linear combination of the convolution algorithm feature maps.

Is the paper you're referring to Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference?

@nbro exactly the paper I am referencing but they are aggregating on the  weights of the conv layer.

I wouldn't call it nor attention nor dynamic convolution.
Reason being that everything is static. if for conv(x) you refer to a standard convolution then that would imply a static kernel, so nothing fancy going on there but just a classic multichannel CNN, and adding 3 learnable parameters is basically just adding a linear layer (not dense) on top of those features. So in inference phase one of those parameters will be higher than the others and the features coming from the convolution, associated with that parameter, will overcome the others. Which doesn't really sound like attention.
The closest paper to what you're suggesting I can think about is:
U-GAT-IT: Unsupervised generative attentional notworks with adaptive layer instance normalization for image-to-image translation.
You can see in the image below that in their generator the authors apply individual weights to each feature map, but the crucial difference is that these weights are not just extra initialized parameters, they come from an auxiliary classifier trained precisely to generate attention masks, idea taken from Learning Deep Features for Discriminative Localization (from which I took the second image).

See Dynamic Convolution: Attention over Convolution Kernels by Yinpeng Chen et al.
The convolution kernels are generated by taking a weighted average of K=4 kernels. The weights are determined non-linearly via channel attention (i.e. "excitation" in SE networks) that uses global average pooling and a dense network.

The paper also discusses some "training tricks", namely by limiting the space of possible weights via a softmax with "temperature" of T=30 to soften the max further.
See also: an implementation in PyTorch.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Avoid unintentional "merging" in cluttered object detection
I have a problem that has bothered me quite some time.
With modern methods object detectors can often be accurately trained, even with small to medium sized datasets. However, there is one thing where I always see errors and that is when objects are very close together. The problem seems to be that the commonly used clustering methods like NMS will often combine predictions in a wrong way, like in this example:

So far, I found that one rather hacky way to solve this is to only annotate some part of the object as  the parts won't be close together, but I wonder if there could be a nicer way to solve this. Many solutions like bottom-up keypoint detection look promising, but they seem to only show their strength with large training datasets - with less data I saw a lot of confusion between the objects.
My question - has anybody experienced something similar and maybe has a promising paper or implementation for a solution that could work without huge datasets?

Just to clarify, the image on the left contains the predictions after you have applied NMS, while the image on the right are the ground-truth bounding boxes? If that's the case, so you're looking for an alternative to NMS?

They are all predictions, but the error only occurs on the left side at the box marked in red. An alternative to NMS could be a possibility, but it seems to me that this problem lies with using a NMS-like-clustering per se, which maybe does not work well for really close objects.
As I mentioned there is some work out there that can live without this clustering step,  example would be all bottom-up keypoint methods, but I have not found something that I could use well for a small dataset.

I might have a solution, but based on a hard assumption, which is the following:
all expected predicted boxes share a characteristic, in this case they are all rectangles with big height/length ratio
This is important cause the fact that you have a big box in your final prediction means that that box had a confidence score high enough to eat out potential smaller boxes, which we know are correct but that the model failed to predict with high confidence. So what need to be done is helping the model by leveraging some a priori knowledge to filter out completely "bad boxes", before proceeding applying MNS.
Below you can see the results of a toy script, in which I first generate some random boxe (top left plot), sometimes "good", which mean the long side is more than 4 times the short side, and sometimes "bad", which means the box is more squarish. When applying NMS to the boxes (top right plot), you can see that some of the bad squarish boxes are not discarded in the predictions. We know though that good predictions should not be squarish, hence we can apply a simple filter to the original boxes before applying NMS. In this case the filter could be:
$long side / short side > 4$
If we filter the boxes (bottom left plot), we remain with a set of potential predictions made only of good boxes, and if we apply MNS to this new filtered set (bottom right plot), areas that before were labeled with bad boxes are now getting labelled with good boxes, see for example the green rectangle on the left at y=400, which was eaten out by a large box in the original set.
of course this is a and crafted solution specific for this case, it might not work perfectly, but I feel it's easier to solve your issue following this post processing path rather than redesigning NMS or by different model training regimes (since it seems that on overall your model is not performing that bad).

import numpy as np
from numpy.random import randint as rind
import matplotlib.pyplot as plt

def nms(dets, thresh=0.1):
    x1 = dets[:, 0]
    y1 = dets[:, 1]
    x2 = dets[:, 2]
    y2 = dets[:, 3]
    scores = dets[:, 4]

    areas = (x2 - x1 + 1) * (y2 - y1 + 1)
    order = scores.argsort()[::-1]

    keep = []
    while order.size > 0:
        i = order[0]
        keep.append(i)
        xx1 = np.maximum(x1[i], x1[order[1:]])
        yy1 = np.maximum(y1[i], y1[order[1:]])
        xx2 = np.minimum(x2[i], x2[order[1:]])
        yy2 = np.minimum(y2[i], y2[order[1:]])

        w = np.maximum(0.0, xx2 - xx1 + 1)
        h = np.maximum(0.0, yy2 - yy1 + 1)
        inter = w * h
        ovr = inter / (areas[i] + areas[order[1:]] - inter)

        inds = np.where(ovr <= thresh)[0]
        order = order[inds + 1]
    return keep

def nms_filer(dets):
    good_boxes = (dets[:, 2] - dets[:, 0]) / (dets[:, 3] - dets[:, 1]) > 4
    print(f"Dropped {dets.shape[0] - len(good_boxes)} boxes")
    dets = dets[good_boxes]
    return dets

def random_boxes(tot=100, min_=0, max_=1000):
    boxes = np.zeros((tot, 5))
    for t in range(tot):
        if np.random.rand() < 0.6:
            min_lenght = 250
            min_height = 30
        else:
            min_lenght = 250
            min_height = 250
        x1 = rind(min_, max_ - min_lenght)
        x2 = rind(x1 + min_lenght, min(max_, x1 + min_lenght + rind(10,min_lenght/2)))
        y1 = rind(min_, max_ - min_height)
        y2 = rind(y1 + min_height, min(max_, y1 + min_height + rind(10,min_height/2)))
        score = 0.5 + np.random.rand() / 2
        boxes[t, :] = np.asarray([x1, y1, x2, y2, score])
    return boxes

def plot_boxes(boxes):
    f, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2)

    for b in range(boxes.shape[0]):
        x1, y1, x2, y2, score = boxes[b]
        x = [x1,x2,x2,x1,x1]
        y = [y1,y1,y2,y2,y1]
        ax1.plot(x, y, linewidth=2)
        ax1.set_title("Original boxes")

    indices_keep = nms(boxes)
    boxes_nms_original = boxes[indices_keep]
    for b in range(boxes_nms_original.shape[0]):
        x1, y1, x2, y2, score = boxes_nms_original[b]
        x = [x1,x2,x2,x1,x1]
        y = [y1,y1,y2,y2,y1]
        ax2.plot(x, y, linewidth=2)
        ax2.set_title("NMS original boxes")

    boxes_filtered = nms_filer(boxes)
    for b in range(boxes_filtered.shape[0]):
        x1, y1, x2, y2, score = boxes_filtered[b]
        x = [x1,x2,x2,x1,x1]
        y = [y1,y1,y2,y2,y1]
        ax3.plot(x, y, linewidth=2)
        ax3.set_title("Filtered boxes")

    indices_keep = nms(boxes_filtered)
    boxes_nms_constrain = boxes_filtered[indices_keep]
    for b in range(boxes_nms_constrain.shape[0]):
        x1, y1, x2, y2, score = boxes_nms_constrain[b]
        x = [x1,x2,x2,x1,x1]
        y = [y1,y1,y2,y2,y1]
        ax4.plot(x, y, linewidth=2)
        ax4.set_title("NMS filtered boxes")

    plt.show()

if __name__=="__main__":

    boxes = random_boxes()
    plot_boxes(boxes)

Thanks for you answer. In this case filtering by shape would definitely work, but this is actually just one of multiple cases where this is happening, so that is why I am looking for a general solution.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Are positional embeddings computed during or before training?
I'm trying to practically frame the concept of positional embeddings as introduced in the original paper.
As far as I've understood, what we do is basically creating some other vectors in addition to the original embeddings of our inputs. So if I have my input $X \in \mathbb{R}^{n \times d}$ with shape (64,103) (so $n$ here is the batch size), I will be creating a matrix $P \in \mathbb{R}^{n \times d}$ where each $d$ dimensional vector will contain information about positions. Now, these vectors are generated from sinusoid functions with an initial frequency of 1e-4, and then initial embeddings $X \in \mathbb{R}^{n \times d}$ and positional embeddings $P \in \mathbb{R}^{n \times d}$ are summed together to get a new input $X' \in \mathbb{R}^{n \times d}$.
Now, all this process is happening before the start of training right? Positional embeddings are not learned/modified during backpropagation?

Yes, your reading of the paper is correct.
Vectors generated from sinusoid functions are fixed, and are not modified during training.
There exists an alternative variant -- initialize vectors as random and then update during training as usual. It also works in practice.


text:
Are there some known neural networks that, given an input image, can generate a similar image, with the same topic?
Are there some known neural networks that, given an input image, can generate a similar image, with the same topic?
Example: input = a photo of a cat on a green table, output = a generated photo of another cat on another green table.
Example 2: input = a portrait of a man with glasses and a beard, output = a portrait of a generated person with similar glasses / beard (see "ThisPersonDoesNotExist").
I imagine it is possible with a GAN, but more precisely which kind of architecture?

(Good) GANs are highly specific. If you had a specific domain in mind, e.g. just human faces then you could use something like a Bicycle GAN. On top of a Generator (& Discriminator) this includes an Encoder, which would take your image, encode it into a latent variable z, which you could then feed into the Generator, perhaps with extra noise to get variance. To my knowledge there don't exist any good pretrained variants so you would have to train it yourself.
On the other hand, I think that Diffusion models are better suited for this particular task, they are trained to denoise a noisy image. So all you would have to do is take your original, add (some) noise and then denoise it to get a similar image.
Both of these would produce an image similar not only in content but also in style, if you actually cared about variance in style and keeping only the content, I would perhaps suggest:

feeding the original image into a Captioning network
feeding the caption into a text-to-image Diffusion model

The closest literature to what you're suggesting is indeed related to GANs, specifically to arithmetic performed on the latent space learned by generators.
Check Unsupervised Representation Learning with deep convolutional generative adversarial networks.
The idea suggested in the paper is to "tune" specific characteristic of the generated images by discarding specific filters learned by the generator. For example, by removing filters related to specific objects like windows, we could generate images of rooms without windows (see second image below, top row: images generated with all filters, bottom row: images generated without window filters).

Other papers applied and refined the same idea, showing that you can tune characteristic of faces like age/glasses and so on.


text:
In CVAE's objective function, why do both terms condition on $\textbf{c}$?
I don't quite understand why, in Conditional Variational Autoencoder (CVAE), we concatenate a conditioning vector two times, at encoder and decoder respectively.
After we concatenate it once at the beginning, isn't the latent distribution going to already incorporate knowledge about the label?
I know that, from a practical perspective we need to concatenate it at the decoder as well, since we want to be able then to generate new instances associating a specific label, but I'm missing more the mathematical motivations for which we concatenate it two times.
To be more precise, let's consider how the  objective function of CVAE is defined:
$$\mathcal{L}_{CVAE} = \mathbb{E}_\textbf{z}[log \, p_\theta(\textbf{x}|\textbf{z},\textbf{c})] - D_{KL}[q_\phi(\textbf{z}|\textbf{x},\textbf{c})||p(\textbf{z}|\textbf{c})] $$
My question: Why do we condition on $\textbf{c}$ in both terms?

To make sure I understand your question, in the objective function, $c$ is what you call label? This paper seems to use a different notation for the objective function. It might be a good idea to provide the reference you're using.

Exactly, $\textbf{c}$ is the label in the objective function I have specified

So, your question is: "In CVAE, why are both the encoder and decoder defined as a function or probability distribution that is conditioned on the label"?

Yes that's pretty much what I wanted to ask

Ok, can you please provide the source of this equation? Because I'm looking at the original paper, as I already said, and it uses a different notation, and it's harder for me to associate the symbols there with yours. It seems that you're using $c$ to denote what the authors of the paper denote by $x$, and you use $x$ to denote what they denote by $y$, the label or output variable. $z$ is the latent vector both in your case and in the original paper. $c$ seems to be the input to the auto-encoder, not a label.

So, they are not conditioning on a "label" (which would be $y$ in the paper and $x$ in your notation), but conditioning on the inputs. So, please, edit your post to reflect this comment and to either make your notation equal to the original paper's one or to provide a reference that uses this notation. Moreover, your question "Why do we condition both on encoder and decoder?" is not specific and clear enough. Because, even in the original ELBO, you condition on something. You want to ask why we're conditioning on $c$ in both terms.

That's exactly what I'm trying to better frame. Why do we condition on $\textbf{c}$ in both terms of the loss? I've modified the question clearly expressing this doubt.

I had understood that this was your question, but I recommend that you follow my other suggestions in the comments above.

I suspect that z needs to be conditioned on c to get the KL maffs to work out. Crudely it also makes sense that the encoder can do a better job with more information, in which case the decoder can benefit. But I also expect it sometimes won't matter in practice. Not been able to find work on this specifically. So am going to test.


text:
Are these two forms of the state value function the same?
Why are there different forms of the value function in reinforcement learning?
Sutton & Barto (2nd edition, equation 3.14) define the state value function as follows
$$v_{\pi}(s) = \displaystyle\sum_a \pi(a|s) \displaystyle\sum_{s',r}p(s',r \mid s,a)[r + \gamma v_\pi(s')].$$
Many lecture notes (for example, these by Haim Sompolinsky, MCB 131, 2017) define it as follows (equation 18)
$$V^{\pi}(s) =\displaystyle\sum_a \pi(a,s) [R(s,a) + \gamma \displaystyle\sum_{s'}P(s' \mid s,a) V^\pi(s')].$$
Are these two forms the same?

Please, focus on one question. You're asking at least 2 distinct questions here. Also, provide the links to the slides from Carnegie Mellon. I think your formulas have been copied incorrectly or imprecisely. Definitely, the notation is not fully correct.

Have I seen this question before? In any case, the first equation contains some mistakes (no declaration of $r$, it should be in the second sum, and $r_{t+1}$ appears when there are no time steps - should be just $r$). Both equations seem to skip superscript on $V^{\pi}$. It would be good to fix these, and/or give full references to where the equations are from (Sutton&Barto numbers its key equations, so you could say exactly which one you meant to copy). I don't want to assume anything and "fix" equations myself when the question is directly about them.

I edited the question so that it reflects the sources of the equations.

There are a few different, but equivalent, ways to express the relationships between value functions in the Bellman equations.
Some differences are just notation, but in the case of the two equations that you give, there is a more practical difference.
Specifcally, the term $R(s,a)$ from the second equation that you give is the expected reward from taking action $a$ in state $s$. Even though individual rewards may depend on $s'$, the expectation does not for a consistent MDP.
You can see that the two equations are equivalent, when you consider that:
$$R(s,a) = \sum_{r,s'} r p(r,s'|s,a)$$
and
$$P(s'|s,a) = \sum_r p(r,s'|s,a)$$
The first equation for me is the more intuitive version of the state value Bellman equation as it explicitly shows all the dependencies for the transition. The second equation can be more practical in some circumstances, because if you know or easily calculate $R(s,a)$ for any state/action pair, then it is often more convenient to use it than recalculate each time from first principles. In addition, $R(s,a)$ is often under the control of the developer, and may well be some constant or fixed simple equation based on $s$ and $a$, in which case the $R(s,a)$ notation captures that exactly.

This is correct but not that this is just the definition of the state value function, so I don't think that the practical aspects is very important here, but I understand what you mean. It may also be worth noting that, in many cases, the reward is deterministic, so $r(s, a)$ will just be the expectation of a constant, which is a constant itself, i.e. $r(s, a)$ can be replaced with a deterministic reward function rather than being an expectation.

@nbro: I would say that the Bellman equations are not the same as the state value definition. The definition IMO is $v_{\pi}(s) = \mathbb{E}{\pi}[\sum{k=0}^{\infty} \gamma^k R_{t+1+k} | S_{t} = s]$ and the Bellman equations in the question are derived from it.


text:
Compare Strings composed from 2-3 words using NLP/ML(Python)
I have a database of books. Each book have a list of categories that describe the genre/topics of the book (I use Python models).
Most of the time, the categories in the list are composed from 1-3 words.
Examples of a book category list:
['Children', 'Flour mills', 'Jealousy', 'Nannies', 'Child labor', 'Conduct of life'],
["Children's stories", 'Christian life'],
['Children', 'Brothers and sisters', 'Conduct of life', 'Cheerfulness', 'Christian life'],
['Fugitive slaves', 'African Americans', 'Slavery', 'Plantation life', 'Slaves', 'Christian life', 'Cruelty']

I want to create/use an algorithm to compare the books and find similarity between 2 books using NLP/machine learning models.
The categories are not well defined and tend to change. For example, there can be a category of 'story' and other called 'stories' category (since the text in the system don't saved categories and use a open text box).
So far I tried 2 algorithms:

cosine similarity with WordNet - split the category to bag of words and check if each word have synonym in the other book lists.
Check the similarity using the nlp model of the spacy library (Python) - distance algorithm.

So far I used WordNet model from the nltk package and spacy
had problem with those two algorithms because when the algorithm compare a categories that contain 2 or 3 words the results wasn't accurate and each of them had specific problems.
Which algorithm and models (in Python), can I use to compare between the books that can handle strings that contain 2 or 3 words?
B.w is the first time I ask here. If you need more details about the database or what I did so far please tell me.

One thing is not clear to me. Do you want to compute the similarity of books by only considering these list of categories? From your description, this seems to be the case. So, what is your actual question?

Hey. Firstable thanks for your comment. As i described the categories are not solid - since they was added as an open text. I need a tool that compare and find similarity between those strings(we can also treat them as a short phrase). For example can be categories like child, children and children story. All of them are about children but the algorithms i mentioned before can't compare between a string of 2-3 words in a good way. So i search a techniques to find similarity/compare between those strings

You can use a model to create rich embeddings for example: sentence transformers and then use cosine similarity distance from sklearn with a threshold (at least 0.6) to create clusters of semantically-close document

Interesting. I should try it :)


text:
Does importance sampling really improve sampling efficiency of TRPO or PPO?
Vanilla policy gradient has a loss function:
$$\mathcal{L}_{\pi_{\theta}(\theta)} = E_{\tau \sim \pi_{\theta}}[\sum\limits_{t = 0}^{\infty}\gamma^{t}r_{t}]$$
while in TRPO it is:
$$\mathcal{L}_{\pi_{\theta_{old}}(\theta)} = \frac{1}{1 - \gamma}E_{s, a \sim \pi_{\theta_{old}}}[\frac{\pi_{\theta}(a|s)}{\pi_{\theta_{old}(a|s)}}A^{\pi_{\theta_{old}}}(s,a)]$$
There exists some problems of vanilla policy gradient, such that distance dismatch between parameter space and policy space and suffering poor sample efficiency.
For tackling this problem, TRPO introduces importance sampling for improving this. However, when I compared the pseudocode of two algorithms, I don't see any obvious evidence that aids the point. They all firstly sampled multiple trajectories under the current policy. The former then just uses the estimated gradient of loss function to update the parameter $\theta$. The latter also uses the estimated gradient but under the KL constraint to update the parameter $\theta$. It looks like they are almost the same procedure but with some subtle difference of the optimization process.
My question is - should it be more accurate to say that importance sampling just increases the stability of the sequential decision making process, or rather, avoid update too aggressively, but not directly improves the sample efficiency?
Pseudocode of two algorithms (source from OpenAI spinning up document)

The point of importance sampling is to use the same episode(s) to do multiple policy gradient updates. It will definitely increase sample efficiency over the strictly on-policy case, simply because in an off-policy algorithm we can use a single transition to do multiple updates, whereas we only get a single update from that same experience in the on-policy case.
First of all, in the pseudocode for TRPO that you give, I see no importance sampling being done, and the pseudocode makes it seem that the algorithm is being done on-policy.
Basically, an on-policy algorithm would look like:
\begin{align*}
& \text{ for each iteration }: \\
& \qquad \text{ for t in range(n)}: \\
& \qquad \qquad \text{sample } a_t; \text{ get reward } r_t \text{ and next state } s_{t+1} \\
& \qquad \text{ do policy update based on } a_1, r_1, s_2, a_2, \ldots
\end{align*}
whereas an off-policy algorithm is like
\begin{align*}
& \text{ for each iteration }: \\
& \qquad \text{ for t in range(n)}: \\
& \qquad \qquad \text{sample } a_t; \text{ get reward } r_t \text{ and next state } s_{t+1} \\
& \qquad \text{ for m epochs}: \\
& \qquad \qquad \text{ for k mini-batches}: \\
& \qquad \qquad \qquad \text{make mini batch from} \{a_1, r_1, s_2, a_2, \ldots\} \text{ and do policy gradient update }
\end{align*}
So, in an off-policy algorithm, you may have something like $m=10$ (10 epochs), meaning every transition you do gets used in 10 policy gradient updates. An on policy algorithm would only use that transition once.

This clarifies a lot. But  I still have some problems. 1) TRPO is a on-policy method for sure, for which it confuses me a lot that how is sample efficiency attained without doing multiple update like the off-policy method does (seems no essential difference bewteen VPG). 2) For best of my knowledge, is policy gradient method always on-policy because the object of both sampling and control is $\pi_{\theta}$ under the same parameter space? So if I'm not wrong, how to do mini-batches without introducing a behavior policy?

And I also wanna stress that you've concluded at the beginning with the statement It will definitely increase sample efficiency over the strictly on-policy case,  however, your explanation right after mainly focuses on the off-policy case where can attain sample efficiency by runing many epoches, but not directly aids the conclusion of on-policy case.

my answer explains what is meant by 'off policy' in the context of the PPO algorithm (which is mentioned in your title, and is essentially the successor to TRPO). I'm not exactly sure what purpose importance sampling serves in TRPO.


text:
Why don't we wait if there is no patrons, in this decision tree from Russel and Norvig's book?
I'm reading Russel-Norvig's book about artificial intelligence and now at chapter decision tree where this figure is shown:

So far I understood it. This decision tree should answer the question if we have to wait for a table or not. Obviously we don't have to wait if there are no patrons and if the restaurant is full, it depends on the waiting time. If it is 0-10 then we wait but if it is longer than 60min then we don't have to wait?
But why do we have to wait if there are some patrons? I mean there will be free tables, so why do we have to wait?
Then I try to change the question to "should we wait?" Then it makes more sense. If it is just 0-10min we should wait but if it is longer than 60min we should not. If there are some patrons, then it is ok we can wait. But the case "None" doesn't make sense. Why don't we wait if there is no patrons? What are the reasons? Bad restaurant?

If a restaurant doesn't have patrons, it means that it has received a lot of complaints that's why the None means don't wait. In general, complaints based on preference also serve a purpose as they start to show a pattern that can be interpreted as disapproval from a good portion of your clientele.
That's why the absence of patrons makes sense.

The additional information implied in this answer is "wait for a table" is not an outcome or measurement, but an immediate action. "No" means leave the restaurant, "yes" means stay at restaurant until provided a table (probably quickly if the restaurant has spaces).


text:
How to deal with datasets which are not balanced?
I have a dataset that I want to use for training.
The output of the model is a binary value (0,1)
The dataset is not balanced, it has only 200 entries for output 1 and 4000 entries for output 0.
When I tried to use it with LightGMB, the model always predict 0 and for this reason, it is not good.
How can deal with an unbalanced dataset?
One way that I can think of it is to delete several of 0 entries and only use around 200  entries with an output of 0.
This is not good, as the model can not see all datasets.
What is the best way to deal with unbalanced datasets?

My favorite first alternative is change the error/cost function in order to penalize more the errors in the less frequent label.
About other alternatives (generate synthetic cases, ..) you can easily Google for "imbalanced data". Some easy to understand articles will appear, as:
https://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/
who suggest 8 groups of solutions, four concrete:

Collect More Data
Change Performance Metric
Resampling
Generate Synthetic Samples

and four more generic:

Try Different Algorithms
Try Penalized Models
Try a Different Perspective
Try Getting Creative

"the model always predict 0 and for this reason, it is not good."
If accuracy is the performance metric that is of most interest for your application then this may well be the ideal behaviour if the density of positive patterns is not higher than the density of negative patterns anywhere in the feature space.
You need to determine what is the appropriate performance metric for your application before deciding what to do.  In imbalanced learning problems it is common for false-positive and false-negative misclassification errors to have different costs, in which case the first thing to do is to use cost-sensitive learning and incorporate these costs into the training criterion.
I'd also advise using a probabilistic classifier (e.g. logistic regression, neural net, kernel logistic regress, Gaussian process classification) that gives the probability of class membership rather than a hard yes/no classification as then the misclassification costs can be changed without having to re-fit the model.


text:
What exactly happens in gradient clipping by norm?
Consider the following description regarding gradient clipping in PyTorch
torch.nn.utils.clip_grad_norm_(parameters, max_norm, norm_type=2.0, error_if_nonfinite=False)

Clips gradient norm of an iterable of parameters.
The norm is computed over all gradients together as if they were
concatenated into a single vector. Gradients are modified in-place.

Let the weights and gradients, for loss function $L$, of the model, be given as below
\begin{align}
w
&= [w_1, w_2, w_3, \cdots, w_n] \\
\triangledown
&= [\triangledown_1, \triangledown_2, \triangledown_3, \cdots, \triangledown_n] \text{, where } \triangledown_i = \dfrac{\partial L}{\partial w_i} \text{ and } 1 \le i \le n
\end{align}
From the description, we need to compute gradient norm, i.e. $||\triangledown||$.
How to proceed after the step of finding the gradient norm? What is meant by clipping the gradient norm mathematically?

Gradient clipping is a technique that tackles exploding gradients. The idea of gradient clipping is very simple: If the gradient gets too large, we rescale it to keep it small. More precisely,
$$
\text{if } \Vert \mathbf{g} \Vert \geq c, \text{then }
\mathbf{g} \leftarrow c \frac{\mathbf{g}}{\Vert \mathbf{g} \Vert}
$$
where $c$ is a hyperparameter, $\mathbf{g}$ is the gradient, and $\Vert \mathbf{g} \Vert$ is the norm of $\mathbf{g}$.
Since $\frac{\mathbf{g}}{\Vert \mathbf{g} \Vert}$ is a unit vector, after rescaling the new $\mathbf{g}$ will have norm $c$.
Note that if $\frac{\mathbf{g}}{\Vert \mathbf{g} \Vert} < c$ , then we don’t need to do anything.
Check this article  for more information

This appears to be copy-pasted from another source, without appropriate attribution.  We have certain rules that we require you to follow whenever copying from another source: https://ai.stackexchange.com/help/referencing.  Please [edit] your answer to follow them.


text:
What do we mean by "orderly opinions" in this sentence in the context of Bayes theorem?
In this page, it's written (emphasis mine)

If probabilities are thought to describe orderly opinions, Bayes theorem describes how the opinions should be updated in the light of new information

What is your understanding/definition of "orderly" opinion?
Maybe something like: a probability that is not arbitrarily chosen but well-founded and explainable?

That term exactly refers to the difference between two main paradigms in probability and statistics: Frequentism vs Bayesianism. You can find many texts for explaining the difference, for example [1] and [2].
By the way, we can briefly say it means that we assume that there exists a fixed opinion out there (like a parameter of a distribution function) and we are going to estimate it based on the observation. Despite the Baysian probability, in frequentist probability, we assume that the opinion is not fixed and changing randomly based on observations.


text:
Should I train a neural network with data with or without a constraint?
I want to train a Neural Network (NN) using a dataset. I want to use the NN model as a prediction function in one algorithm. However, in the algorithm, any data that does not meet a specific constraint (say some parameter $\theta <10$) would not be included.
So, my question is, while generating the training data, should I include all kinds of inputs irrespective of the constraint, or should I generate only those data which meet the constraint $\theta <10$?
Currently, I am training data with constraint ($\theta <10$), and I am getting an average error of around $6\%$. Ideally, I want it below $3\%$.
I am new to NN model training. Any kind of pointers would be helpful.

In general you want to train a model with a sample of data that represent as good as possible the data that the model will predict on when deployed. if you use a threshold in your first algorithm, then you're doing right in using the same threshold for your nn training. Regarding the error you should provide more info, first of all, average error of what?


text:
Different equations for Yolov3 in courses/ articles and Darknet GitHub code?
I am confused by the equations for bounding boxes I find online. Some articles say that
box_width = anchor_width * exp(residual_value_of_box_width))

and the coordinates have a sigmoid function.
Eg: https://www.kdnuggets.com/2018/05/implement-yolo-v3-object-detector-pytorch-part-1.html
https://christopher5106.github.io/object/detectors/2017/08/10/bounding-box-object-detectors-understanding-yolo.html

But Darknet code and GitHub have equations dividing coordinates and box width with image width.
For example, https://github.com/AlexeyAB/darknet/blob/master/build/darknet/x64/data/voc/voc_label.py
def convert(size, box):
    dw = 1./size[0]
    dh = 1./size[1]
    x = (box[0] + box[1])/2.0
    y = (box[2] + box[3])/2.0
    w = box[1] - box[0]
    h = box[3] - box[2]
    x = x*dw
    w = w*dw
    y = y*dh
    h = h*dh
    return (x,y,w,h)

If the image width is used, then what is the use of anchor box width/height values in yolov3.cfg file? I can't find where it has been used in the source code other than generate the anchors file.


text:
What is meant by "spatial encoding" in the context of convolutional neural networks?
Consider the following excerpt from the abstract of the research paper titled Squeeze-and-Excitation networks by Jie Hu et al.

Convolutional neural networks are built upon the convolution operation, which extracts informative features by fusing spatial and channel-wise information together within local receptive fields. In order to boost the representational power of a network, several recent approaches have shown the benefit of enhancing spatial encoding.

The authors used the term "spatial encoding" and the excerpt implies that enhancing spatial encoding has the benefit of increasing the representational power of a convolutional neural network.
What is meant by the term "spatial encoding" in this context related to the convolutional neural networks?


text:
How do you decide that you have tested enough hyper-parameter combinations for a specific neural network architecture?
How do you decide that you have tested enough hyper-parameter combinations for a specific neural network architecture to discard it and move on to a new model?
Do you have a structured (generic) approach? In practice, what gives you the necessary performance (e.g. >= 80% accuracy) the fastest (w.r.t. to your work-hours) assuming there is no SOTA that easily exceeds your requirements and extensive hyper-parameter optimization is infeasible?

I would personally go the other way around. Test several architectures from the literature with default hyperparameters, select the model that perform best, and then go full tank on fine tuning. The logic being that hyperparameters optimization can only boost by a small margin already decent results.


text:
How should I choose a reinforcement learning algorithm?
I'm starting a new RL project. I'm familiar with Deep Q-Learning because of an old project where I used it, but I'm not sure I chose correctly back then.
Why should or shouldn't I choose DQN, or any other RL algorithm/method for a problem? By which criteria should I judge an RL algorithm? Is there any set of guidelines to help you choose a specific RL algorithm for a problem?
I did some research, but wasn't satisfied.


text:
Are there neural networks with (hard) constraints on the weights?
I don't know too much about Deep Learning, so my question might be silly. However, I was wondering whether there are NN architectures with some hard constraints on the weights of some layers. For example, let $(W^k_{ij})_{ij}$ be the weights of the (dense) $k$-th layer.
Are there architectures where it is imposed something like
$$
\sum_{i, j} (W^k_{ij})^2 = 1
$$
(namely the roll-out vector of weights is constrained to stay on a sphere) or $W^k_{ij}$ are equivalence classes $mod K$ for some number $K>0$?
Then, of course, one should probably think about proper activation functions for these cases, but it's probably not a big obstacle.
Putting constraints of these kinds will prevent the weights to grow indefinitely and maybe could prevent over-fitting?

Hello. Welcome to Artificial Intelligence Stack Exchange. I put what I think is your main specific question in the title. Make sure that's correct. Having said that, you tagged your post with [tag:weight-normalization], so I suppose you're aware of those techniques. Right?

Hi @nbro and thanks for the correction! Not really! I used that tag because it was recommended when I start typing "weight"


text:
Which algorithms are used to locate objects in a 3d space?
I can see mobile apps that can locate a 3D object on a surface with a mobile camera and you can turn around that object.
What is the name of the algorithm(s) that is used for that purpose? Or, is there AI in these algorithms? Are they use plane detection? After detection, which algorithm do they use to locate the objects?
It seems like a Computer Vision problem, but I do not know the title.

Hello. Can you please provide the name of the apps that do this?

3D Object Recognition (https://paperswithcode.com/task/3d-object-recognition) or 3D Object Detection (https://paperswithcode.com/task/3d-object-detection) is probably what you're looking for.


text:
Test accuracy decreases during my train process

I want to train a neural network model with the arcface loss function and try to combine it with domain adaption. But when the training process continues, I find the test accuracy first increases and then decreases, the model cannot reach convergence. I chose the office31 dataset, and the feature_extractor was resnet50.
I want to know if it is caused by my code, or by my loss function
The arcface function was set as
def Arc_pred(cosine, s=64.0, m=0.1):
    cosine = cosine / s
    thea = torch.acos(cosine)
    top = torch.exp(torch.cos(thea + m) * s)
    _top = torch.exp(torch.cos(thea) * s)
    bottom = torch.sum(torch.exp(cosine * s), dim=1).view(-1, 1)
    divide = (top / (bottom - _top + top)) + 1e-10
    return divide

and my total loss function was set as
total_loss = 0.1*target_entropy_loss + label_loss + arc_loss + discriminator_loss

In that, the target_entropy_loss tries to make the decision boundary cross the sparsest sample area，label_loss was the classification loss, discriminator_loss was a domain adaptation loss function.
I tried to set a learning rate schedule for my experiment, it seems it did not work. So, could it be caused by my loss function?

Hello. From the plot you're showing us, it's not clear what is the highest test accuracy reached after it drops. As suggested in the answer below, you could try early stopping, but if the top test accuracy is too low, you may try something else first. Could you please also tell us how long have you been training the model for, at which epoch the test accuracy drops and how large are your training and test datasets.

I tried to train the network by 10000 iterations,the highest test accuracy reached at about 91%.Besides overfitting,I want to know whether the SGD optimizer I chosed for the GAN part also could cause the decrease.

This looks like overfitting. You can try stop training earlier by using a validation dataset to prevent this, or you can try other regularization effects such as weight-decay, dropout etc.


text:
Is it a bad practice to use cumulative rewards in reinforcement learning
I am using a DDPG agent for doing prediction on the position on an asset in a stock trading-like environment. I am using the cumulative reward as the reward for each timestep. Since it is trained over many years of data, the reward tends to become large. I have realized that after some training the agent becomes lazy, it just keeps the same position.
Is it a bad practice to use cumulative rewards as rewards? Would the daily revenue be a better reward for the agent?

Can you please describe more in detail your environment or problem? "prediction on the position on an asset in a stock trading-like environment" is not very clear to me. It's also not clear what you mean by "cumulative rewards as rewards". Maybe you mean "cumulative cash/credit/money as reward"?

Reinforcement learning already has the objective of maximising the sum of future expected reward.
By making each reward the sum of all previous rewards, you will make the the difference between good and bad next choices low, relative to the overall reward guaranteed on each step.
The best reward for the agent should be as direct measure of what you want it to achieve as possible. Sometimes you need to compromise because you cannot easily measure what you really want. Sometimes you may need to assist the bot, giving it a more frequent reward signal (e.g. change to predicted cash-out value of all the current holdings) - but you should note this is also a compromise, because an agent that maximises this "reward shaping" reward scheme may not actually maximise what you really want.
You could view your attempt to use cumulative values as a failed experiment in reward shaping. I don't know if daily revenue would be a good for your bot, because a trading bot might have several different end goals for its creator.
First thing you should do is figure out what your success criteria is for the bot, and find the measure for it that best captures that goal as directly as possible. E.g. if the goal is for it to create a reliable income after 5 years of trading, then the reward might be the monthly income that it able to generate in the long term. Perhaps simulating or measuring this is too hard, in which case you could fall back to a proxy, such as cash value of any sell orders with a penalty if it is not maintaining a certain amount of total of holdings.


text:
Can Q-learning be used for my scenario, and how might I do so?
I have already asked 2-3 general questions w.r.t Q learning and now I am asking a scenario specific one. I will try to be concise and understandable. I really really need help.
Scenario: I have a network with few nodes and links. On each link, there are some slots (#1 to #800). I generate traffic requests (come one by one) that want to go from one node to another and need certain slots to do so. So, my task is to allot the slots to each upcoming request and finally achieve a low rejection probability i.e. able to allot slots to as many requests as possible. The allotted slots are also freed up depending on when the arrived requests leave the system. I use the Poisson process to do, but this is not important here.
What I thought: There have been certain simple benchmarks to do this but I wanted to use Q learning so that in the long run the agent (a centralized controller) takes better decisions as to which slots to assign on the particular link i.e. which slots position (#1 to #800).
What I did: I decided to take the state space as the links (say 10 links in my case) and the action space as the #1 to #800 slots. I use binary notation 1 to say slot is occupied or 0 that is free.
Problem encountered: But it is long later I realized that my state space is infinitely big. For E.g. For request 1, I give two slots on Link1 & state is 1 1 0 0 0 .... up to 800 zeros. Another request comes (say 3 slots) and say Link1 state is now 1 1 1 1 1 0 0 0...up to 800 zeros. This is when I realized that the state space is unimaginably large as departures can also occur leading to freeing up and some 1s becoming 0s and so on.
What I am asking for: So, does anyone have any ideas on how can I still use Q learning in this case. The point is that someone already used deep Q on this. I was thinking I am approaching it in a different and simplified way of just using link state as state space that would enable me to have a small Q table. But it is later I realized that each link state will vary every time and lead to large state space thus putting me back to square one after investing a lot of time on this. So, please give any suggestions as I don't want to leave it altogether.

I am not clear on this: The requests require "certain slots" - so the slot numbers needed are somehow specified in the request? However, you also say that action description is the slot number to assign, which must therefore mean the agent is free in at least some extent to make a choice other than accept/reject a request? How much freedom the agent has is important in understanding the true size of the state space and action space here.

For instance, if all slots are equivalent, and requests only specify the number of slots required, your state space would be size 800 and action space size 2. I would not expect that to be the case, since you would probably have figured that out. However, it is not clear from the question why that is the case, what specifically prevents you from having necessary information about the state of the system for decisions by the agent in a simplified form. Knowing this is necessary if you want someone else to verify your assessment of the state space size.

@NeilSlater Thanks for the comments. If I understood your question correctly, I would say that the scenario is that the request asks for a certain no. of slots (say 3), I have find &allot 3 slots on the respective links it will travel. The #800 slots are increasing in a frequency value. It is like I choose the frequency(slot) to assign it. But I should give 3 continuous slots for e.g. slot no. 35 36 37. Since this alloting slots has effect on certain paramters (which I consider in reward) I decided to make the first slot selection as 'actions'.

@NeilSlater Nothing prevents me from knowing the information of the state. At any time I can access the link_slots variable to see the occupancy in each link. Since I have 10 links I thought ok, maybe my states are my links (#1 to #10) and actions the corresponding frequency slots (#1 to #800) . But later did I realize (as int he question e.g.) that the each link's state i.e. the combination of 1's and 0's would be different everytime and I cannot use the updated Q value (say) of slot #35 at a later time to give the same slot to another request (assuming now that slot is free).

@NeilSlater I thought so because the state of that link when I updated the Q value for 'its' #35 slot action and the link state 'now' are different, when I am trying to decide to give the same #35 slot for my current request on that same link jsut because I see the Q value is good for #35 action. But this is a new combiantion of 1's and 0's & so it is a new unseen state for the same link. The e.g. in the question illustrates this. I don't know if my conclusions regarding Q learning in this case are right though...

Thanks for the response. The important pieces of information you gave are "he request asks for a certain no. of slots (say 3)" and "I should give 3 continuous slots for e.g. slot no. 35 36 37". This feels a little bit like memory block allocation, and specifically concerns about memory fragmentation - not something I know much about, buit you may want to look into solutions used for that in case they help

@NeilSlater Thanks for whatever help you extended. I too know that this is too specific and so I can't expect to get detailed answers! Actually, that's why I was intially asking generic questions about Q learning rather than this question,. Anyway, thanks a lot for atleast reading my explanations and responding.  I will try to do something.


text:
What do we actually 'approximate' when dealing with large state spaces in Q-learning?
I realized that my state space is very large in size. I had planned to use tabular Q-learning (Bellman equation to update the $Q(s, a)$ after each action taken). But this 'large space' realization has now disappointed me and I read a lot of stuff on the internet. I have the following confusions.
I saw the 'approximation' term for the 'large space' scenario (for example, in this Medium blog post). But what it is exactly? I can't reduce the states I have nor can I club together different states and update the Q values. So, what is it I should do when they say 'approximate'? If it is the $Q(s,a)$ we approximate, then won't we anyway do for each state $s$ as and when it is encountered? How does this help in a 'large space' scenario?

The only thing that is missing from your question is maybe some specific context. You say "I saw the 'approximation' term for the 'large space' scenario". Can you please quote the excerpt (and provide the link to the source) where people use the term "approximation" in this specific context. It may be necessary to answer this question more appropriately, although I think it may already be answerable. After that, I will reopen your post.

Actually, I saw like lot of sources which are more for continuous state spaces and they say approximation. For large state spaces. I saw the term 'discretization'. But I couldn't put all the details here and thought that people would eventually point this out in answers. So, if I am try to put a link here it would look like aah, it is for continuous space and you are asking for large state space. I am confused and so I asked the way it is.

Discretisation of a continuous space is one form of approximation (one of the simplest, but can be very effective depending on the environment).


text:
Do any practical deep learning algorithms deal with tensors containing non-real entries?
In deep learning, most of the applications are from text and images. Both text and images can be converted into a tensor of real numbers.
Other than both mentioned above, there may be some other real-world data used by Deep learning algorithms. Any algorithm in general takes tensor as input and gives tensor as output. As far as I know, tensors always consist of real numbers.
I want to know whether there are any practical applications that deal with the tensors containing non-real numbers. Is there any such possibility?

What about a Fourier transform of a time series?

@Dave I will check once.


text:
Why do language models produce different outputs for same prompt?
For conventional 'Neural Networks', the weights simply act as a transformation in highly multi-dimensional space; for a forward pass, the output is always the same since there is no stochastic weighting component in the process.
However, in Transformers (self-attention based encoder-decoder type architecture to be specific) we get different outputs with the same prompts (assuming $T > 0$). This doesn't make sense to me because the set of weights are always static, so the probability distribution produced should be the same; this simple decoding should yield the same output.
However, in practice, we observe that it is not actually the case.
Any reasons why?

Language models produce a probability distribution over a set of words. You determine the next word by sampling from this distribution. So, determining the next word is stochastic even though the distribution is the same given the initial prompt.

Worth mentioning the role of T in adjusting the distribution for sampling

Maybe it's a stupid question but, shoudn't there be a seed to set to make the sampling reproducible?


text:
Why is there a 1 in complexity formula of uniform-cost search?
I am reading the book titled Artificial Intelligence: A Modern Approach 4th ed by Stuart Russell and Peter Norvig. According to the book, the complexity of uniform-cost search is as
$$
O(b^{1+\lfloor{C^*/\epsilon}\rfloor}),
$$
where $b$ is the branching factor (i.e. the number of available actions in each state), $C^*$ is the cost of the optimal solution, and $\epsilon > 0$ is a lower bound of the cost of each action.
My question is: Why is there is a 1 in the formula?
For example, suppose in the following tree, the red node is the initial state and the green one is the goal state, and two actions are needed to reach the goal state from the initial state. If the cost of both actions is equal to $\epsilon = 1$, so, $C^*$ will be $2$. Therefore, the complexity will be $O(b^{2})$. But, from the above formula, the complexity will be $O(b^{3})$.

PS. I know there is a similar question in stackoverflow and have read the answer. But there is a disagreement between the answers about the 1.

To be fair I struggle too, to see a reason for it to be included.
I'll try to argumentate: assume that the optimal path has length N and therefore you have costs $c_1,\dots,c_N$ all positive and bigger than some $\epsilon>0$. Such $\epsilon$ can't be bigger than the minimum between $c_1,\dots,c_N$.
Now, to need that $+1$ correction factor it should be possible that
$$\cfrac{c_1+\dots+c_N}{\epsilon}<N$$
Since $\epsilon<min\{c_1,\dots,c_N\}$ I could try to prove instead that
$$\cfrac{c_1+\dots+c_N}{min\{c_1,\dots,c_N\}}<N$$
But you can divide both members by N and multiply by $min\{c_1,\dots,c_N\}$ obtaining that
$$\cfrac{c_1+\dots+c_N}{N}<min\{c_1,\dots,c_N\}$$
On the left hand side you have the average of the costs, which of course can't be smaller than the minimum of the costs, therefore that has no solution. This means that even without the +1 factor you can't underestimate the length of the path to the optimal solution using $\lfloor C^\star/\epsilon\rfloor$.
On the flip coin by the way when you compute that $\lfloor C^\star/\epsilon\rfloor$ you're actually trying to find an upper bound for the length of the optimal path, so if you add +1 (equivalent to compute the ceiling instead of floor) you still get an upper bound. Also, since you're computing an $O(\circ)$ keep in mind that if a function is an $O(b^k)$ then it's an $O(b^{k+c})$ for every $c>0$ so although not necessary the book isn't technically wrong.
EDIT: Take all I said with a grain of salt, I'm a student, too and still have to take the AI exam
EDIT 2: I thought that another possible reason why you would expand (at most) b more nodes after reaching the depth of the optimal goal is that you perform the goal test the moment you extract the node from the frontier and not before insertion. This is necessary cause it grants us that when we extract a node from the frontier we reached it through the path with lowest cost from the initial state, but it will cause us to expand at most one more level before realizing a node in the frontier is indeed a goal.

I think the example you gave is inappropriate. This formula is derived from the summation formula of the geometric series, but the summation formula does not apply to the case where the common ratio is 1. so if b > 1, the formula is correct.


text:
Why are today's neural networks not modeled with probability theory?
In the paper The Perceptron: A probabilistic model for information storage and organization in the brain, Rosenblatt used the probability theory to model his perceptron.
My professor told me that today's neural networks are not modeled with probability theory anymore. Why is that?

I appreciate this query as an undergrad math major doing a project on probability theory in neural machine translations.

Your professor is wrong (or maybe you misunderstood what he wanted to say, or he did not explain correctly what he wanted to say). It may be a good idea to ask your professor for some clarification and tell him about the info given in this answer.
Probability theory is widely used to model problems in machine learning, including in the context of today's neural networks. There are many examples of research papers or books where you will see probabilities and probability distributions. For example, you can take a look at the neural machine translation paper.
More importantly, we often formulate the problem of learning (in the context of neural networks) as a minimization of an objective function, which is equivalent to the maximization of a likelihood of the parameters (given the data). It's also often the case that neural networks produce a probability vector (from the so-called logits, often, by using a softmax function).
There are many other (maybe clearer) examples of the use of probability theory to model problems in machine learning. For example, you can take a look at Bayesian neural networks, variational auto-encoders, or generative adversarial networks. You will see a lot of probabilities and probability distributions in these linked papers.
You may also be interested in this answer.


text:
How should I read a deep learning paper?
I have a background in mathematics and I am accustomed to reading papers with lemma and proofs. When I see a deep learning paper, they seem to be of practical nature.
How can I improve my reading and understanding of deep learning papers?
To truly understand, should I have to implement the code? What is the best approach (if any)?

Hello, welcome to the artificial intelligence stack exchange. I feel that this question may be closed due to its subjectiveness...

In addition to the comment above and my answer, there is already a more general question here, to which I also provided a similar answer (I realize this only now). Given that the questions are not exact duplicates, I won't close yours for now.

There are a lot of machine learning papers which are theoretical in nature, and their primary contribution is lemmas and theorems. There are also an immense number of papers whose contributions are based around a new equation or mathematical idea, which are supported by derivations or proofs, and have some experiments that just validate the idea. To understand such papers, you have to understand the equations and the experiments you can just quickly glance over.

Adding something to nbro answer, from my personal experience there are also some hints that can quickly tell you if you're dealing with a good machine learning paper, i.e. worth to read in its entirety or not.
In random order:

Clear contribution description: machine learning and artificial intelligence in general are both broad fields. A paper can be about several things, and it should be clear already from the title what the main contribution is: a new model architecture? A new pre/post processing step? New loss function? If it's not clear after reading the abstract there are high chances that this point will not be clear also after reading the whole paper.

Clear architecture/algorithm description and visualization: images and code snippets make a huge difference. As you pointed out, machine learning is mostly an applied field, hence having a code snippet or a clear list of implementation passages reduce the overhead of thinking about how to turn into code the math in an efficient way, and it also reduces the chances of making interpretation mistakes when the passages are not clear or give for granted. Since you're from a mathematical background you are probably experiencing the opposite feeling and wondering why there's not that much math inside. Point is that most machine learning papers are structured like experimental papers. You have hypotheses, not theorems to prove, you run experiments to test them and you describe the results at the end.

Code availability: unfortunately, still not such a common practice, making your code available is fundamental, not to make other people's life easier, but to grantee the reproducibility of the published results. Moreover, machine learning is characterized by many subtle and arbitrary choices, especially when it comes to hyperparameters, which are many times not reported on the real papers, and, when that's the case, looking at the code becomes the last resource to find that information.

Proper benchmarking: evaluating the impact of a new loss/architecture is always hard. Many papers just report tables with random scores like "97% accuracy", which means nothing when not compared to other models. A good paper always reports the state of the art (SOTA) scores and test the proposed improvements ON EXACTLY THE SAME DATA. Furthermore, a paper ideally should report mean scores over several training runs. Unfortunately, due to the expensive hardware and long time required to train only a single model, this is almost never done.

Short but clear SOTA analysis: it is super hard to stay on track with the state of the art when it comes to machine learning, since dozens of papers are published every month. For this reason, the literature research section should be concise and point to works that are as closest as possible to what is going to be described (and improved upon) in the paper, otherwise, you know you're reading a survey instead.

I have some experience reading research papers. However, in my view, there is no single answer to this question (apart from this answer I am giving you, i.e. "it depends"). The answer to your question depends on

your background knowledge/education

If you don't know much about the specific topic in the paper, you may need to study at least briefly the prerequisites for reading the paper, otherwise, you won't understand much about the paper.

why you are reading the paper

Are you just interested in or curious about the topic in the paper? Then maybe you can only quickly read it, or only read specific sections, such as the ones that introduce the technique (abstract, introduction, and maybe conclusion) and skip all the math; the figures are sometimes also insightful.

Are you doing (serious) research on a related topic? If yes, you will probably need to read the paper multiple times (at least the sections that you don't fully understand). It may also be useful to look at implementations of the approach. If they don't exist, you may consider implementing the approach yourself.

Do you need to present the paper? In this case, you may also need to read the paper multiple times, understand well the figures (because you will probably use them in your slides: an image is worth 1000 words, etc.)

how much time you have to read the paper

Note that these guidelines should also be applicable to other cases (i.e. when you're reading other types of research papers that involve math).
You may also be interested in the paper How to Read a Paper (by Keshav). It provides a few tips that could be useful.


text:
How many unique angles of an object do you need in your image training set in order to correctly classify it?
I'm interested in using ResNet-50 to classify images of objects for around 1000 unique classes. I'm wondering if there is any way to estimate how many unique angles I need in my training set to classify images that can be taken from any angle. For example, if for a given object I had 500 training images from directly the front and 500 training images from directly the top, I'd have 2 unique angles.
A model trained with only those 2 unique angles probably wouldn't be able to classify the same object if it was given a photo from the top right looking down.
Is there anyway to figure out how many unique angles I would need in my training set to classify images that could be taken from any angle? If I had 12 unique angles (top, bottom, front, back, left, right, front-left, front-right, front-top, front-bottom, back-left, back-right, back-top, back-bottom) would I then be able to classify images of any arbitrary angle?
To clarify, if I had 12 unique angles, that would mean I would have many photos from each of the 12 angles, but the 12 angles would all be exactly the same with no variation. I.E. top would be exactly a 90-degree angle towards the object on the Z-axis and 0-degree angles on the X and Y axis, for many photos.

[I wanted it to be a comment but it's too long :)]
I don't think it's a good approach to split point of views into a group of 12 angles. The main purpose of using neural net is to have model that is able to generalize the data. That means the perfect model will be able to recognize an object in every orientation. Your task is to create a model that will be able to do similar thing. In my opinion You should try to make your dataset as differential as possible with not only different point views but also different lightning, background etc. According to the ResNet paper, they evalueted the model on ImageNet 2012 dataset. It was 1.28mil images for 1000 classes. That means approximately 1280 images per class. I guess that's good starting point for You. During training You'll be able to see if that is enough, or if You need to get more data or use some data augmentation techniques.

In the ideal world I would want as many angles as possible. That's not what I'm asking about though. I'm asking about a situation where there is limitations in the training data.

Again, You can use about 1280 images per class, so ~107 images per angle and try to train it on that. If You have no such portion of the images, train on data that You've got and if training will be relatively hard try data augmentation.

The viewing angle is only one of many variables that influence whether a model can classify an object correctly. Other factors might be: is the object occluded? how close are we to the object? is the image noisy? what size has the image of the object? how are the lighting conditions? what color is the surrounding? But let's assume we keep all those variables perfectly constant.
Then Neural radiance fields (NeRFs) might give you an idea of how much information you need to extrapolate the pieces where you have less information. Those NeRF representations are 'richer' than what you need for classification tasks, the interpolation to 'unseen' angles however is conceptually similar.
The more images from different angles you have, the better your classification accuracy will become. But there probably is a different 'amount of angles'-to-accuracy mapping for every conceivable object, but similar objects will require a similar number of images for the same classification accuracy.


text:
Why is gradient descent used over the conjugate gradient method?
Based on some preliminary research, the conjugate gradient method is almost exactly the same as gradient descent, except the search direction must be orthogonal to the previous step.
From what I've read, the idea tends to be that the conjugate gradient method is better than regular gradient descent, so if that's the case, why is regular gradient descent used?
Additionally, I know algorithms such as the Powell method use the conjugate gradient method for finding minima, but I also know the Powell method is computationally expensive in finding parameter updates as it can be run on any arbitrary function without the need to find partial derivatives of the computational graph. More specifically, when gradient descent is run on a neural network, the gradient with respect to every single parameter is calculated in the backward pass, whereas the Powell method just calculates the gradient of the overall function at this step from what I understand. (See scipy's minimize, you could technically pass an entire neural network into this function and it would optimize it, but there's no world where this is faster than backpropagation)
However, given how similar gradient descent is to the conjugate gradient method, could we not replace the gradient updates for each parameter with one that is orthogonal to its last update? Would that not be faster?

There seems to be a paper on this topic.
https://cs.stanford.edu/~acoates/papers/LeNgiCoaLahProNg11.pdf

@DKDK Thanks for the link! An interesting read, however this only adds to my question. The paper indicates the conjugate gradient method is good enough to get an (at the time) SOTA result on MNIST. So why, 10 years after that paper is published, does nobody use it?

I can’t really think of a reason why people don’t use conjugate gradient. My guess is that the initial success of gd based optimizers led to research mostly focused on its variants and applicable tricks (like momentum and adaptive learning rates).

A new optimizer with conjugate gradient seems like a viable research topic and I would love to see a sota conjugate gradient optimizer in the near future

When dealing with optimization problems, a fundamental distinction is whether the objective is a (deterministic) function, or an expectation of some function. I will refer to these cases as the deterministic and stochastic setting respectively.
Almost always machine learning problems are in the stochastic setting. Gradient descent is not used here (and indeed, it performs poorly, which is why it is not used); rather it is stochastic gradient descent, or more specifically, mini-batch stochastic gradient descent (SGD) that is the "vanilla" algorithm. In practice however, methods such as ADAM (or related methods such as AdaGrad or RMSprop) or SGD with momentum are preferred over SGD.
The deterministic case should be thought of separately, as the algorithms used there are completely different. It's interesting to note that the deterministic algorithms are much more complicated than their stochastic counterparts. Conjugate gradient is definitely going to be better on average than gradient descent, however it is quasi-Newton methods, such as BFGS (and its variants such as l-BFGS-b) or a truncated method that are currently considered state of the art.
Here's a NIPs paper that says CG doesn't generalize well. There are similar results for quasi-Newton methods. If you want something better than SGD, you should look into a method like ADAM, which was designed for the stochastic setting. CG and ADAM both use information from past gradient directions to improve the current search direction. CG is formulated assuming that the past gradients are the exact gradient. ADAM is formulated assuming that the past gradients are gradient estimates, which is the case in the stochastic setting.

Saying that "Gradient descent is not used" in the context of deep learning is probably very wrong. People certainly tried to train neural networks with all data at some point (I think I already did it too). Maybe you should reformulate that sentence as "GD may not be widely used, but, instead, mini-batch GD seems to be more widely used in practice". In any case, after having read your answer, I don't understand why conjugate gradient is not used for training neural networks. What prevents exactly CG from being used?

gradient descent means using all examples to compute the search direction. stochastic gradient descent uses 1 training example to compute the search direction. mini-batch stochastic gradient descent (SGD) uses a batch of training examples to compute the search direction. I have never seen a code which uses only 1 training example or every training example.

maybe you think the answer is more clear now

Hello. Thanks for trying to clarify. I know the difference between GD, mini-batch, and stochastic gradient descent. My question was not about that. It was about why isn't the CG not "better" than SG for training neural networks. I think you clarify this in your last edit by saying that CG assumes that you use the "true" gradient of the objective function. However, if we use the full dataset (at each iteration, like in full-batch GD), wouldn't that lead to the "true gradient"?

And couldn't we use the CG with mini-batches (like we start doing with GD) as some kind of stochastic approximation? You provide a paper that claims that CG doesn't generalise well. I didn't read the paper, but I suspect that this paper provides these results for the neural network context. It may be a good idea to describe their setting. How are they using the CG exactly?

The paper indicates "[CG] overfits significantly" - however the paper also indicates "CG results in lower training error". Given the training sets were chosen "so that overfitting was possible", it would seem CG might still have a place if regularized properly to outperform Gradient Descent. Intuitively, your answer describing how SGD would outperform CG makes sense, as as you said, an optimizer like ADAM behaves very similarly to CG however it assumes (correctly) a stochastic environment whereas CG assumes deterministic. However it doesn't seem there is concrete results to back this

@Taw Maybe it's worth mentioning that CG can be used under certain conditions to have good outcome.

The fundamental issue is that one doesn't really want to find an optimum of one's optimization problem. We are really interested in generalization - not optimality. And we still poorly understand how and why neural models generalize so well.
Now, it looks like the generalization properties of neural models have something to do with the structure of their optimization landscape and some particular properties of their well-generalizing minima. Empirically, the SGD class of optimizers is better at finding such generalizing minima.
This paper illustrates these ideas by talking about "wide flat" minima and showing how one can use SGD with stochastic weight averaging to improve generalization and convergence.


text:
How to fine-tune GPT-J with small dataset
I have followed this guide as closely as possible: https://github.com/kingoflolz/mesh-transformer-jax
I'm trying to fine-tune GPT-J with a small dataset of ~500 lines:
You are important to me. <|endoftext|>
I love spending time with you. <|endoftext|>
You make me smile. <|endoftext|>
feel so lucky to be your friend. <|endoftext|>
You can always talk to me, even if it’s about something that makes you nervous or scared or sad. <|endoftext|>
etc...

Using the create_finetune_tfrecords.py script (from the repo mentioned above) outputs a file with 2 in it. I understand that means my data has 2 sequences.
I could really use some advice with the .json config file. What hyperparameters do you recommend for this small dataset?
The best I came up with trying to follow the guide:
{
  "layers": 28,
  "d_model": 4096,
  "n_heads": 16,
  "n_vocab": 50400,
  "norm": "layernorm",
  "pe": "rotary",
  "pe_rotary_dims": 64,

  "seq": 2048,
  "cores_per_replica": 8,
  "per_replica_batch": 1,
  "gradient_accumulation_steps": 2,

  "warmup_steps": 1,
  "anneal_steps": 9,
  "lr": 1.2e-4,
  "end_lr": 1.2e-5,
  "weight_decay": 0.1,
  "total_steps": 10,

  "tpu_size": 8,

  "bucket": "chat-app-tpu-bucket-europe",
  "model_dir": "finetune_dir",

  "train_set": "james_bond_1.train.index",
  "val_set": {},

  "eval_harness_tasks": [
  ],

  "val_batches": 2,
  "val_every": 400000,
  "ckpt_every": 1,
  "keep_every": 1,

  "name": "GPT3_6B_pile_rotary",
  "wandb_project": "mesh-transformer-jax",
  "comment": ""
}

The problem is that, when I test the fine-tuned model, I get responses that make no sense:

The last paragraph describes in detail the process of selecting parameters for fine tuning

Far from expert, but at least I can shed some light.
Your dataset is simply too small. Finetuning means you "interfere" with what the GPT-J model sees as important for the continuation of a prompt. Because that is what the model does; continue a prompt in a way that logically makes sense to what it has seen. Since your dataset is very small, especially since the sentences are not even full sentences, things go south.
For one, the create_tf_records.py (that I presume you used) script already filters OUT all senquences that are not long enough (I'm not sure, but this could be why you end up with 2 sequences?).
Then the model now has 500 sentences (and maybe even just 2) it sees as the MOST IMPORTANT data to use for its continuation (I think this is where the term "fine-tuning" comes from, not that it's a small influence on the data, it requires "fine" precision ). I had to find this out myself the hard way - there is very limited documentation - and I'm still not 100% certain, but quite sure about this part; you trained the model into a brain dead zombie that only slightly remembers all of the knowledge that is stored deeply embedded behind your dataset.
Now, this is a far for complete or really correct explanation, but I hope it gives you some understanding. If you understand more of it or better - please don't forget to let me know :)
What you probably could do to actually get somewhere:
Create a script, that based on the sentence you want to "inject", creates more data. So if your happy/positive sentence set is what you want to achieve, then create context around those sentences and spin them. Many times, combined in all kinds of ways. I would even suggest adding in some stuff like the binary contents of an image (look at CLIP, or the v-jax implementation of it by kingoflolz) if you know how to. Some chat generated by another model / chatbot - like GPT-2 or Clara - and just append your sentence at the end. Translate your sentences (deepl is recommended atm I think), mix those in. Pose a question before your sentences; "what would be a good way to cheer someone up or let them know you appreciate them? [insert item from dataset]" and spin those questions. Try to keep repeating yourself to a minimum (but different combinations are fine, small deviations also, but won't teach the model that much more). Paste entire news articles as though it's being quoted in a chat conversation and then put one of your sentences, also will work.
Those are my findings so far. Available for further discussing all of this; I'm also still figuring this thing out.

Thanks for your insight Ontopic! Buuut. helloforefront.com (which helps with the fine-tuning and hosting of GPT-J) works perfectly well with my dataset. Also, Using the same stuff for OpenAI's GPT-3 and it also works just fine. So I don't think it's a problem with the dataset. It could be other things e.g., create_tf_records.py as you mentioned that may be filtering out smaller sentences.

I’ll try to answer even if some more information would be needed. GPT-J is text generation Models meaning that it can complete a sentence or a pattern you started. As I see from your output when you write “Hi” in your finetuned model it gives you back a complete nonsense output which make me think:

How are you giving your examples to the model? If you want to build a sort of chatbot you should use something like turns (person1: How are you, person2:fine thanx!)
For fine tuning I suggest to not modify the structure of the model until it is close to work properly or you will not know if you break it changing the model structure or for some other reason.

In the end if the model works well on another platform I think it should be a matter of Data preparation that you’re not doing and the platform is doing instead.
My advice e is trying to replicate the exact preparation of your link and replacing the text with your examples assuring that you are passing the texts to the model in the exact same way.


text:
How rewards are playing role in Deep Q Network
I have started working on Reinforcement Learning, specifically DQN. And I have watched some interesting videos on it. However, I have some doubts about how the model works.
Let's say we are playing Atari Breakout where we have only 3 actions: left, stay still, right. We have 2 networks- the policy_net and the target_net (technically both are the same) and they give 3 outputs which are the q values for the 3 actions. During exploration we choose:
random.randrange(3)

and during exploitation, we choose:
argmax(policy_net output)

where the input of the policy net is the current state.
Now, during each timeline of each episode, we are storing current_state, action, reward, and next_state in storage that we will later randomly shuffle and use in training, which we call experience replay memory. During training, let's say we extract a batch of (current_states, actions, rewards, next_states). Now, we get current_q_val and next_q_val as:
current_q_values = policy_net(current_states)
next_q_values = target_net(next_states)

We use the following equation to find the loss:
$$q_*(s,a) - q(s,a) = \text{loss}$$
$$E[R_{t+1} + \gamma \max_{a'} q_*(s', a')] - E[\Sigma_{k=0}^ \inf \gamma^k R_{t+k+1}] = \text{loss}$$
And for that, we find which one from the next_q_val is the best one and that we call $\max q_*(s',a')$ (we already have $q(s,a)=\text{current_q_values}$).
Now my question is, which is $R_{t+1}$ here? As we are taking a random batch from the experience replay memory, we don't have any specific time here and we cannot calculate the $R_{t+1}$ from the time $t$. And if we simply use $q*(s,a) - q(s,a)$ or $\text{next_q_val} - \text{current_q_val}$, then where is the importance of the reward? I don't really understand how we are using the rewards in the training. I mean, where are we making sure the positive and negative influences of good and bad rewards respectively? The fact that the agent takes an action (randomly or from the policy_net) which then gives a reward, I don't understand how to use this reward in the loss function so as to influence how the agent should take action given a state.

Where did you get the loss function from? The equation you give is a kind of conceptual description of "regret" in RL, and not used as a loss function in DQN in any meaningful way.

Hi @NeilSlater, I got the loss function from one of the youtube series on RL I watched. can you tell me what should be the corrected loss function? In the video series, they are using MSE loss on current_q_value and next_q_value

Hello. As I asked you on Stack Overflow, please, edit your post to format the math symbols with mathjax. You need to wrap them with $ on either side, like this $\pi$ to produce $\pi$.

"From one of the youtube series" is too vague for me to correct it. Either the video is wrong or your understanding of it is wrong. Using MSE loss for the neural network versus the TD targets calculated from each minibatch drives the learning, but it is important to note that is a moving target. It is possible for the MSE loss to get larger, but the agent behaviour to improve at the same time, at least in the short term. That is because the MSE loss does not capture the full RL objective, just the NN's role's objective.

@nbro, okay I changed the equation

@NeilSlater, yes I understand what you are saying. So, I was thinking of updating the target_net every 4 epochs while policy_net is getting updated every epoch, to make sure we are not always making it 'chasing its own tail'. But, I don't know how to add the rewards in the loss function. WHat I thought of is at each run we are getting currentstate, action, reward, nextstate. do we need to add this reward with the nextqvalue? and what should be the gamma then?


text:
Is the VC dimension of a MLP regressor a valid upper bound on how many points it can exactly fit?
I want to calculate an upper bound on how many training points an MLP regressor can fit with ~0 error. I don't care about the test error, I want to overfit as much as possible the (few) training points.
For example, for linear regression, it's impossible to achieve 0 MSE if the points don't lie on a line. An MLP, however, can overfit and include these points in the prediction.

So, my question is: given an MLP and its parameter, how can I calculate an upper bound on how many points it can exactly fit?
I was thinking to use the VC dimension to estimate this upper bound.
The VCdim is a metric for binary classification models but a pseudo dimension can be adapted to real-value regression models by thresholding the output:
$$Pdim(\mathcal{G}) = {VCdim}(\{(x,t) \mapsto 1_{g(x)-t>0}:g \in \mathcal{G}\})$$
(from the book Foundation of Machine Learning, 2nd edition, definition 11.5)
where $\mathcal{G}$ is the concept class of the regressor, $g(x)$ is the regressor out and $t$ is a threshold.
The model is an MLP with RELU activations. As far as I understood on Wikipedia, it should have a VCdim equal to the number of the weights (correct me if I'm wrong).
So, the question is: how to practically calculate the pseudo-dim for the regressor given the VCdim? Does it make sense for the purpose that I want to achieve?

This question was also asked here.


text:
Which multi-agent reinforcement learning algorithm can I use when there are two types of agents with different action spaces?
Most of the papers on multi-agent RL (MARL) that I have encountered have multiple agents who have a common action space.
In my work, my scenario involves $m$ numbers of a particular agent (say type A) and $n$ numbers of another type of agent. Here, the type A agents deal with a similar problem due to which they have the same action space, and type B deal with another type of problem and they have the same action space.
The type A agents are involved in an intermediary task that doesn't reflect in the final reward, and the final reward comes from the actions of type B agents. But the actions of type B are dependent on type A agents.
Any idea on what kind of MARL algorithm is suitable for such a scenario?


text:
Where do the characteristics of self-attention come into play in Linformer's proof that self-attention is low rank?
In Linformer's proof that self-attention is low rank in their paper, I don't see how it doesn't generalize to every matrix. They don't utilize any specifics of self-attention (the entire proof feels like it's in equation 20 utilizing JL, and I do not see where characteristics of self-attention come into play).
What am I missing?


text:
Why isn't a target network used for the critic in on-policy actor-critic methods?
Based on my research, I've seen so many on-policy AC approaches that utilise a critic network to estimate the value function $V$. The Bellman equation for the value function is as bellow:
$$
V_\pi(s_t) = \sum_a \pi(a|s_t)\sum_{r, s'}(r+V_\pi(s'))P(s', r|s, a)
$$
It makes sense not to have a replay buffer due to the current policy in the formula and the fact that our approach is on-policy. However, I really do not figure out why no one uses a target network to stabilize the training process of the critic, like what we have in DQN, namely the variant published in 2015. Does anyone have an idea for that with probably a citation?
I know that DDPG uses a critic with a fixed target network, but be aware that it is a real off-policy actor-critic. By "real" I mean it is not due to importance sampling.
I have to mention that I can imagine something, but I'm not sure whether it is true or not. If we have a target network, it means we are trying to find a deterministic, optimal in the case of DQN, policy, while we are learning the current policy's data for the actor-critic case with the critic.

Common boys! Doesn't anyone want the bounty?! The question is not strange. It is something that everyone considers in the papers.

Just to contextualize more, what is an example of an on-policy AC method that does not use the target network that you are referring to?

@nbro Vanilla on-policy AC does not use target for the critic. Available in textbooks and source codes.

I can't give a conclusive answer, but here's an attempt, largely based on An Empathic Approach to the Problem of Off-policy Temporal-Difference Learning. With function approximation, updating $V_\pi$ for state $s_t$ due to a recorded transition may update state $s^\prime$. Lets say the value estimate for $s_t$ is 1 and the estimate for $s^\prime$ is 2 and we have a linear network, with $\gamma \approx 1$ and learning rate $0.1$.
Lets also say we have recorded a transition that gives 0 reward.
In the off-policy setting, when learning from this transition, first we update the value for $s_t$ to 1.1 (and $s^\prime$ to 2.2). The second time the transition is used we update to 1.21 (and 2.42). Repeated off-policy learning can basically cause the learning to diverge as updates get larger. A target network fixes the target value for a longer period, which may mitigate this - the target remains at 2 for a longer time which reduces the updates - but it's still vulnerable to this issue.
In the on-policy setting, once we reach $s^\prime$ we need to learn from a new transition that starts in $s^\prime$, and our current overestimation bias of the value of $s^\prime$ will impact the new error, directly, hopefully counteracting the bias. This is why there are much stronger convergence guarantees in the on policy case for function approximation and bootstrapping. There is some research into target functions for on policy actor-critic but I don't think it has been shown to be more effective, rather the opposite.


text:
Is there any other (possibly less popular) approach to create AI apart from statistical methods?
From what I have gathered so far, an AI has some prior (stored in the form of some probability distribution), and, based on experiences/data, changes the distribution (via Bayes rule) accordingly. This idea seems intuitively correct, as humans do something similar: we have some prejudice about certain things and refine it further based on additional observations.
I am wondering if there is a different (possibly, non-probabilistic) setting for designing an AI.

Yes, there is symbolic AI. This was the 'original' approach to AI, at a time when there was very little data and/or processing power available. The focus was on logic and calculus, not on machine learning, which was just in its infancy.
A lot of natural language processing was developed using grammar rules (which only later were learned from data).
There still is a lot of this around, but often it's now hybrid, where human-authored rules correct the systematic mistakes of statistical approaches.
Update: as an example the Teneo system for building conversational agents (aka chatbots) uses both pattern matching rules and machine learning for intent recognition. The (human-created) patterns are more precise, but sometimes lack breadth of coverage, which the ML part provides, which works as a fallback.

It would be interesting if you could provide an example of a system that is hybrid, where the rules correct the statistical approaches. I know that some cognitive architectures are hybrid, but I don't remember the details or names. This info could be useful to answer this question.

@nbro Done! This is from a company I used to work for.

What you're looking for are Expert systems and Knowledge Based Systems. Really similar to each other, they encompass all systems built upon experts knowledge, from which analytic rules are derived in order to allow their implementation in computer programs.
A trivial example could be a self driving system based on proximity sensors, a camera and a set of predefined rules (basically a set of explicit if-else statements) like:

if value recorder from front proximity sensor < 10cm & left bottom corner of picture from camera is white (wall on the left) -> turn wheels 90 degrees to the right.
if value recorder from front proximity sensor < 10cm & right bottom corner of picture from camera is white -> turn wheels 90 degrees to the left.
... (and so on and do forth)

I wouldn't limit the answer to this question to knowledge-based systems, but to symbolic AI (aka classical AI or GOFAI), which is the alternative approach to "statistical AI". This answer may not explicitly suggest that, for example, search algorithms (like A*) can be considered "AI algorithms".