text:
How widely accepted is the definition of intelligence by Marcus Hutter & Shane Legg?
I came across several papers by M. Hutter & S. Legg.
Especially this one:
Universal Intelligence: A Definition of Machine Intelligence, Shane Legg, Marcus Hutter
Given that it was published back in 2007, how much recognition or agreement has it received?
Has any other work better formalizing the idea of intelligence been done since?
What is considered current standard on the topic in the field?

Hutter and Legg's definition of intelligence is based on Hutter's AIXI framework, and it's an optimization-based definition: basically, intelligence is a measure of the capability of an agent to optimize with respect to a "wide range of" environments. I don't think that everyone agrees with this definition (for some reason: maybe because not all people like the optizimization-based idea or the part "wide range of environments"), otherwise, my answer here would have more upvotes, but I haven't found any "better" definition of intelligence so far.

And I highly recommend everyone read nbro's blog post on the subject "On the definition of intelligence"

My sense is that everyone is pretending Intelligence doesn't have a grounded definition, from which all other definitions arise:

Intelligence is a measure of utility in an action space μ(υ)

It can be a relative measure, in relation to other rational agents, or absolute in relation to solved games (problems).  An action space is any context, and formalized as problems or sets of problems, typically grouped by complexity class.
Hutter and Legg is an explication of this grounded definition which accounts for unlimited contexts (complexity classes/environments) and for increasing utility of a given agent over time (learning/optimization.)  Intelligence itself does not require learning or general applicability, but Hutter & Legg does not refute this, merely grades static intelligence and narrow intelligence as more limited.
Even this is subject to context, as more limited rationality can be more optimal.

The definition of intelligence is grounded because, while the term "intelligence" is a symbol, intelligence itself is function, the strength of which evaluated by a measurement of some result (utility)

It doesn't require defining the function to understand it as a function:  measurement of a result requires mechanism and decision/action.
You will find this natural language definition applies to even emotional intelligence, which relates to the observational capability of the rational agent in context, and allow that rational agent to make more optimal decisions in context.
This is similar in spirit to truth, only grounded in a formal logic context, where it is a condition and result, not an assertion.  By contrast, the the conditionality of truth is often obscured in a natural language context, where is is routinely applied to unvalidatable informal statements, and can even be conflated with the statement itself à la: "this is the truth!"

The short answer is that it hasn't received nearly the attention it should have despite the cited paper having over 600 citations.
According to this Singularity Summit talk by Shane Legg, his PhD thesis on the wide ranging uses of the word "intelligence" managed to identify 2 qualitative dimensions of those uses:

Human vs Ideal
Internal vs External

Of the 4 points in the space of these 2 dimensions, the sense in which his paper with Hutter uses the word "intelligence" is (Ideal, External).  This is distinguished from the Turing Test which is for (Human, External).
The vast majority of machine learning targets (Ideal, External) intelligence rather than modeling human intelligence.
The formal, "top-down" definition of (Ideal, External) provided by Hutter is called AIXI, which is the unification of ideal science (Algorithmic Information Theory aka AIT) with ideal technology (Sequential Decision Theory aka SDT).
Unlike other attempts to define (Ideal, External) intelligence, it has only 2 open parameters:

Choice of Universal Turing Machine running the universe in which the agent perceives as well as the agent's simulation of that universe to make predictions.
Choice of utility function providing the value system of the agent so it can decide which actions yield the most valuable consequences.

In the vernacular, AIT is about what "is" and SDT is about what "ought" to be done about what "is".
For just one example of why AIXI has not received nearly the attention it should have:
The ethics of AI are riddled with the conflation of "is" with "ought".  Under AIXI, these can be, at the top level of analysis, factored out using the Algorithmic Information Criterion for model selection, which is conceptually quite simple:
Given the union of all datasets used to train "large" AI models, choose the model that yields the smallest executable archive of that data.
The model so-selected is the best available model of what "is" the case -- including models of bias in the (possibly latent) identities that are the sources of data.  If we are to be fair about what is and is not "bias" here's how to proceed:
If someone has a reason to call some data "biased" then they should be challenged to provide the data that supports their perception of what is and is not "biased".  Then, simply include that data in the total dataset.  If they, themselves, are biased and select biased data because of their bias, let others include the data that shows that bias to be, in fact, bias.  Compress relentlessly until the data that is most consilient with the rest of the universe is better compressed, leaving the bias exposed as "noise" associated with the responsible identities.
This has applications in dealing more effectively not only with social media censorship, but by factoring out the "is" from the ethics, it permits people to recognize when they simply differ in their utility functions or "value systems" and get on with the hard problem of dealing with quasi-religious differences and thereby reduce the likelihood of conflicts that can be exceedingly destructive.
Nor is this merely pie-in-the-sky as Hutter has demonstrated with The Hutter Prize for Lossless Compression of Human Knowledge.  The Hutter Prize, if funded at the level deserving of the problem of ethics in AI, would be several orders of magnitude larger.

While I cannot tell you what intelligence is/should be, I can give you why their definition fails by today's standard on what systems are intelligent.
According to their definitions, the most intelligent model is that is statistically most optimal in various (goal-driven) environments. This definition leads to Hutter's AI model AIXI being arguably the most intelligent, because in every step, given the current environment and previous observations it statistically optimally predicts (well, the uncomputable model, and a computable model restricted by time and size gives the most statistically optimal prediction bounded by time and size). This takes exponentially much time (with regard to space size) however.
Just like most other models, AIXI also starts with zero knowledge and learns step by step. Now, if we have limited resources, and want to create the model that predicts best based on the resources we have, then one could argue that AIXI may predict best and be the most intelligent (after trained on all our resources). In reality, we have much more resources than ever "run out" and the goal is rather to build models that learn fast (and scale for long). Two decades later it is clear that you definitely do not need to optimally predict while learning on data, it is faster to make mistakes and go back to correct them. The learning makes AIXI inefficient, and no "intelligent" system today builds on it. They build on things that based on these concepts would be unintelligent, yet they work better.
Learning must definitely be a key component of intelligence in the case of models (or natural entities) that start with zero or low knowledge.

Just a few commonsensical remarks about why this kind of intelligence definition seems unable to capture the logic of life:

Optimization only makes sense in a stationary environment.  When many agents learn and interact, they are building a constantly changing environment.

Survival and reproduction is the only thing that really matters, and it does not require optimization, just good enough solutions.

The survival of individual living organisms heavily depends on adequate hardwired sensory abilities that can slowly change throughout many generations. But smarter individuals can use their brains (or whatever tools they may have endowed with plasticity) to quickly learn and adapt in non-stationary environments. Fast learning, not optimization, is what these agents really need.

Welcome to SE:AI!  I think Herbert Simon nailed it with satisficing


text:
I have 5000 html files (structured text), how can I generate a new one that "resembles" those?
I don't know anything about ML or NLP, but I was asked by someone to create brand new statutes (written laws) that resemble the ones currently in effect in my country. I have already gathered the laws, and have 5000 html files now, one per law.
The average size of each html file is 49 kB. The entire corpus is 300 MB.
I have two alternative goals (doing both would be perfect of course):

Generate a new, complete HTML file, that would imitate the 5000 existing ones (it would typically have 1 big heading at the top, sub-headings, articles with their own title and number, etc.)

Generate sentences that sound as if they could be found in a typical law (the laws are written in French)

Is any of those goals feasible, with such a small corpus (~300 MB total)?
Should I try and fine-tune an existing model (but in that case, wouldn't the small size of my corpus be a problem? Wouldn't it be "drowned out" in the rest of the training data?), or should I create one from scratch?
I've tried following guides on huggingface, but between the obsolete files, the undocumented flags and my general lack of knowledge of the subject, I'm completely lost.
Thanks in advance.
BTW, if you want to take a peek at the data, there it is: https://github.com/Biganon/rs/

"I don't know anything about ML or NLP" --> why should u take the task, btw, text generation is just for fun, won't ever be law (must be exact, precise, no bad logics)

@datdinhquoc I took the task because I'm the only teaching assistant in the law faculty who is at least slightly comfortable around computers, and I thought it would be fun. I am not required to fulfill the task, I just thought it would be interesting. Also, we don't want to produce law that will be used, we want to use this as an experiment, precisely to show the limits of the system. I'm still hoping someone can point me in the right direction.

Why does it need to be in HTML? Why not simply the raw texts of the statutes themselves. HTML formatting introduces extra complexity, and is the same in every file: it does not to be learned.

You could try to train a recurrent neural net on char level. Basically, you took GRU or LSTM and use a sequence of characters, not tags or words. In the blogpost "The Unreasonable Effectiveness of Recurrent Neural Networks" there are examples for Shakespeare, Linux source code on C and for papers in latex code, and the results are quite valid, produced from similar to your size training sets. The good thing in case of html pages is that modern browsers are very good at handling a slightly broken HTML, so this approach could work for both of your tasks.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Why (not) using pre-processing before using Transformer models?
Regarding the use of pre-processing techniques before using Transformers models, I read this post that apparently says that these measures are not so necessary nor interfere so much in the final result.
The arguments raised seemed to me quite convincing, but someone would know how to explain better, perhaps with a bibliographic reference, why is it not so necessary to use these techniques?

Tokenizing is a part of pre-processing.

@AbhishekVerma, these models already do tokenizing, indeed, but how does Transformers work so that a pre-processing step is little or nothing necessary?

Why would you want to preprocess instead of using the raw data? Removing stop words, stemming, etc, removes information. If the raw data is given it can learn where to put its attention.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How do I implement the cross-entropy-method for a RL environment with a continuous action space?
I found many tutorials and posts on how to solve RL environments with discrete action spaces using the cross entropy method (e.g., in this blog post for the OpenAI Gym frozen lake environment).
However now I have built my first custom environment, which simulates a car driving on a road with leading and following vehicles. I want to control the acceleration of my vehicle without crashing into anyone. The state consists of the velocity and distance to the leading and following vehicles. The observation and action spaces are continuous and not discrete, which is why I cannot implement my training loop like in the examples that use the cross entropy method. That is, because the method relies on modifying each tuple for training <s, a, r> (state, action, reward) so that the probability distribution in a is equal to 1 in one dimension and equal to 0 in all others (meaning, it it very confident in its action, i.e., [0, 1, 0, 0]).
How do I implement the cross entropy method for a continuous action space (in Python and Pytorch) or is that even possible? The answer to this question probably describes what I want to do in a very mathematical form, but I do not understand it.


text:
How can I train a model to recognize object with zoomed-in image?
Humans are good at guessing animals with zoomed-in images from patterns of fur/skin.
(For example, if we saw a black-white pattern fur, it must be a zebra)
I have some experience guessing a car model from an interior/exterior photo without a brand logo.
(based on the dashboard/gear level/air vent or something like that)
I think it would be helpful for my coworkers to have such a model.
(I'm working at a car forum, and I have some limited experience working with TensorFlow).
Is this possible?
Where should I start with?


text:
SAGAN - is there a mistake in the original paper?
in the original paper the following scheme of the self-attention appears:
https://arxiv.org/pdf/1805.08318.pdf

In a later overview:
https://arxiv.org/pdf/1906.01529.pdf
this scheme appears:

referring the original paper.
My understanding more correlates with the second paper scheme, as:

Where there is two dot-product operations and three hidden parametric matrices:
$$W_k, W_v, W_q$$
which corresponds to $W_f, W_g, W_h$ without $W_v$ as it in the original paper explanation, which is as following:

Is this a mistake in the original paper ?

I don't understand where the mistake is. Can you please clarify where the mistake is?

I think I was confused by the notation of v as a vector function, I thought it is a vector and then it didn't work out. So eventually there is non.


text:
Do Gradient Descent and Natural Gradient solve the same problem?
I am troubled by natural gradient methods.
If we have a function f(x) we wish to minimize, gradient descent minimizes f(x) of course, but what does the natural gradient do?
I found on https://towardsdatascience.com/natural-gradient-ce454b3dcdfa:

Instead of fixing the euclidean distance each parameter moves(distance in the parameter space), we can fix the distance in the distribution space of the target output.

Where did the distributions come from? If we wish to minimize f(x), the target output is just a minimizer x* right, and not a distribution, or am I missing something?

You're asking multiple distinct questions here. The question in the title is quite general and different from the question "Where did the distributions come from?". So, please, edit your post to leave only one question.

https://wiseodd.github.io/techblog/2018/03/14/natural-gradient/#:~:text=Up%20to%20constant%20factor%20of,%E2%88%87%CE%B8L(%CE%B8).


text:
I want to determine how similar a given song is to Queen's songs. Am I headed in the right direction?
I've asked this question before (@ Reddit) and people suggested CNNs on a mel spectrogram more than anything else. This is great.
But I'm sort of stuck at: label some music data as "queen" and "not queen" and have this be the training set. Like, download 300 songs, 70 queen (that's all they have) and 230 not queen, create their mel spectrograms using some python package that can do that.
First of all, is 300 songs even enough?
I only have a basic understanding of what I'm doing. I need some help

Welcome to AI Stack Exchange. Could you expand on what you mean by "similar"? There are lots of ways (dimenisons) in which two songs can be similar. Once trained, how do you intend to use your neural network?

similar in voice, like Gnarles Barkley is more similar to Alicia Keys than System of a Down

You are heading in the right direction to make an audio-based classifier. This is not quite the same as providing a similarity metric between two pieces of audio, but it may do as a first attempt. You could use the "probability that this audio is a Queen song" as a proxy for similarity.

First of all, is 300 songs even enough?

Nowhere near enough to train a classifier from scratch. In that case you might be aiming for 10,000 samples or even up to a million, depending on how sophisticated you want your classifier to be.
However, you don't necessrily need to find that many training examples. Instead, if you can find a pre-trained audio classifier for music that is compatible with the libraries you are working with, and use transfer learning, you may get decent results. This works by replacing the last few layers of a neural nework trained with your own ones and training the network on your data only modifying those new changed layers.
From your comments:

similar in voice, like Gnarles Barkley is more similar to Alicia Keys than System of a Down

The tone of the vocals is one part of many things that can vary between music recordings. A classifier with only a few examples to work from will not be able to isolate just the elements you are interested in. Also, without examples that are explicitly labelled in a way that "more similar" is meaningful in the way that you want, you will not have control over whether the neural network identifies Brian May's guitar, or 1970s studio equipment as the most easy to identify element compared to Freddie Mercury's vocals.
These things may limit the usefulness or accuracy of your first attempt, but I would not suggest you consider them immediately. You basic idea should produce something that has an interesting behaviour when given different inputs. Just be realistic that you will not get state-of-the-art results on your first attempts at the project.


text:
In a convolutional neural network, how is the error delta propagated between convolutional layers?
I'm coding some stuff for CNNs, just relying on numpy (and scipy just for the convolution operation for pure performance reasons).
I've coded a small network consisting of a convolutional layer with several feature maps, the max pooling layer and a dense layer for the output, so far so good, extrapolating the backpropagation from fully connected neural networks was quite intuitive.
But now I'm stuck when several convolutional layers are chained. Imagine the following architecture:

Output neurons: 10
Input matrix (I): 28x28
First convolutional layer (CN1): 3x5x5, stride 1 (output shape is 3x24x24)
First pooling layer (MP1): 2x2 (output shape is 3x12x12)
Second convolutional layer (CN2): 3x5x5, stride 1(output shape is 3x8x8)
Second pooling layer (MP2): 2x2 (output shape is 3x4x4)
Dense layer (D): 10x48 (fully connected to flattened MP2)

Propagating the error back:

Error delta in output layer: 10x1 (cost delta)
Error delta in MP2: 3x4x4 (48x1 unflattened, calculating the error delta for the dense layer as usual)
Error delta in CN2: 3x8x8 (error delta of MP2 but just upsampled)

How do I keep from here? I don't know how to keep propagating the error to the previous layer, if the error delta in the current one is 3x8x8, and the kernel 3x5x5, performing the convolution between the error delta and the filter for calculating the delta for the previous layer, that gives a 3x4x4 delta.


text:
Understanding advantage estimator in proximal policy optimization
I was reading Proximal Policy Optimization paper. It states following:

The advantage estimator used is:
$\hat{A}_t=-V(s_t)+r_t+\gamma r_{t+1}+...+\gamma^{T-t+1}r_{T-1}+\color{blue}{\gamma^{T-t}}V(s_T) \quad\quad\quad\quad\quad\quad\quad(10)$
where $t$ specifies the time index in $[0, T]$, within a given length-$T$ trajectory segment. Generalizing
this choice, we can use a truncated version of generalized advantage estimation, which reduces to
Equation (10) when $λ = 1$:
$\hat{A}_t=\delta_t+(\gamma\lambda)\delta_{t+1}+...+(\gamma\lambda)^{T-t+1}\delta_{T-1}\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad(11)$
where, $\delta_t=r_t+\gamma V(s_{t+1})-V(s_t)\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad(12)$

How equation (11) reduces to equation (10). Putting $\lambda=1$ in equation (11), we get:
$\hat{A}_t=\delta_t+\gamma\delta_{t+1}+...+\gamma^{T-t+1}\delta_{T-1}$
Putting equation (12) in equation (11), we get:
$\hat{A}_t$
$=r_t+\gamma V(s_{t+1})-V(s_t) $
$+\gamma[r_{t+1}+\gamma V(s_{t+2})-V(s_{t+1})]+...$
$+\gamma^{T-t+1}[r_{T-1}+\gamma V(s_{T})-V(s_{T-1})]$
$=-V(s_t)+r_t\color{red}{+\gamma V(s_{t+1})} $
$+\gamma r_{t+1}+\gamma^2 V(s_{t+2})\color{red}{-\gamma V(s_{t+1})}+...$
$+\gamma^{T-t+1}r_{T-1}+\color{blue}{\gamma^{T-t+2}} V(s_{T})-V(s_{T-1})$
I understand the terms cancels out. I am not getting the difference in blue colored power of $\gamma$ in last terms. I must have made some stupid mistake.


text:
Pytorch - Evaluation loss on the training set higher than loss during training
    def forward(self, image, proj, proj_inv):
        return self.predict_2d_joint_locations(image, proj, proj_inv)

    def criterion(self, predicted, gt):
        return self.mse(predicted, gt)

    def training_step(self, batch, batch_idx):
        player_images, j2d, j3d, proj, proj_inv, is_synth = batch
        predicted_2d_joint_locations = self.predict_2d_joint_locations(player_images, proj, proj_inv)
        train_loss = self.criterion(predicted_2d_joint_locations, j2d)
        self.log('train_loss', train_loss)
        return train_loss

    def validation_step(self, batch, batch_idx):
        player_images, j2d, j3d, proj, proj_inv, is_synth = batch
        predicted_2d_joint_locations = self.predict_2d_joint_locations(player_images, proj, proj_inv)
        val_loss = self.criterion(predicted_2d_joint_locations, j2d)
        self.log('val_loss', val_loss)
        return val_loss

I have this simple code for training_step() and forward() in Pytorch. Both the functions essentially do the same.
Owing to a relatively small dataset, my model grossly overfits on the training data (as is evident from there being an orders of magnitude of difference between the training and validation losses). But that's fine for now, I am perfectly aware of that and will add more data soon.
What surprises me is when I try to evaluate (infer). I don't have a separate test set (for now) and only have a training and a validation set. When I evaluate on the validation set, the mean squared error turns out to be in the same range as the validation loss my model is based on as expected. However, when I evaluate on the training set, the mean squared error I get is again in the same range as the validation loss (not the training loss).
    if args.val:
            check_dl = dataset.val_dataloader()
        else:
            check_dl = dataset.train_dataloader()

        for player_images,j2d,j3d,proj,proj_inv,is_synth in check_dl:
            if args.visualize:
                # visualize dataset
                player_images = player_images.cpu().numpy()
                j2d_predicted = model(torch.from_numpy(player_images), proj, proj_inv).cpu().detach().numpy()
                print(((j2d - j2d_predicted) ** 2).mean(), model.training_step((torch.from_numpy(player_images),j2d,j3d,proj,proj_inv,is_synth), 0))

When I print print(((j2d - j2d_predicted) ** 2).mean() for images in the training set after fetching the model from the trained checkpoint, I get numbers in the range of the validation loss. I retried the same by printing the loss using the training_step() function, but I again receive high losses (in the validation loss range).
Note: The inference mean squared errors I receive on the training set are high but they are not as high as when the training actually started. So, the pre-trained model is fetched properly. On a model with completely random weights, I should have received orders of magnitudes of higher errors. So, the model is definitely fetched correctly.
I have been scratching my head over this. Any help would be really appreciated.

Fundamentally, you are seeing a difference in behavior during training v.s. during evaluation. The most typical reason for it is the difference in behavior of some nn layers that your library (pytorch) provides, depending on the mode that you are in.
Check out documentation for torch.nn.Module.train.
Most notably nn.Dropout and nn.BatchNorm layers are prone to that.

I was suspecting the dropout and batch norm layers to be the reason behind the discrepancy as well. When I update model.eval() to model.train(), I get lower errors. But I believe that the removal of batch norm and dropout layers should help in improving the loss during evaluation instead of degrading it. With dropout, some information bearing nodes are randomly switched off which is not the case during evaluation. Doesn't it mean a lower (or at worst the same) loss during evaluation as compared to training?

@NitishAgarwal As I understand, you've mentioned that you are strongly overfitting during training? Dropout can be understood as way to do model averaging. When averaging several overfitted models you'll must likely get a worse-performing one.


text:
How to interpret this learning curve of my neural network?
How to interpret the following learning curves?
Background: The accuracy starts at 50%, because the network has a binary output (0 or 1). I chose an exponentially decreasing learning rate of the optimizer - I believe that this the reason why the network starts learning after 10 epochs or so.
lr_schedule = keras.optimizers.schedules.ExponentialDecay(**h_p["optimizer"]["adam"])
optimizer = keras.optimizers.Adam(learning_rate=lr_schedule)
h_p["Compile"]["optimizer"] = optimizer

This is a textbook case of overfitting. Your network fits particular patterns existing only on the training data, and fails to generalize. Given your training curves, you should stop training at about 20th epoch, in which the validation loss is the lowest.

It looks like your network is overfitting, because the training loss carries on decreasing to zero even though validation loss levels off, and then starts to increase again.
I would guess that your network is essentially "memorising" the training examples because you're getting a near zero loss in training.
You could try:

applying some form of regularisation
reducing the number of neurons or layers
collecting more training data to see if that helps.

Ideally you want to minimise validation loss, so epoch 20 was about the best point in your graph. Look into early stopping: this is an effective way to avoid overfitting too much.
When the learning rate is too high, you often see the loss 'jumping around' quite rapidly, and when the learning rate is too low, the graph barely changes at all. You could try fixing the learning rate closer to what it is around epoch 10 and see if this is "the sweet spot"; it's mostly a game of experimenting to see which works best.


text:
Why the optimal Bellman operator of a Q-function can be approximated by a single point
I am currently studying reinforcement learning, especially DQN.
In DQN, learning proceeds in such a way as to minimize the norm (least-squares, Huber, etc.) of the optimal Bellman equation and the approximate Q-function as follows (roughly):
$$
\min\|B^*Q^*-\hat{Q}\|.
$$
Here $\hat{Q}$ is an estimator of Q function, $Q^*$ is the optimal Q function, and $B^*$ is the optimal Bellman operator.
$$
B^*Q^*(s,a)=\sum_{s'}p_T(s'|s,a)[r(s,a,s')+\gamma \max_{a'}Q^*(s',a')],
$$
where $p_T$ is a transition probability, $r$ is an immediate reward, and $\gamma$ is a discount factor.
As I understand it, in the DQN algorithm, the optimal Bellman equation is approximated by a single point, and the optimal Q function $Q^*$ is further approximated by an estimator different from $\hat{Q}$, say $\tilde{Q}$.
\begin{equation}\label{question}
B^*Q^*(s,a)\approx r(s,a,s')+\gamma\max_{a'}Q^*(s',a')\approx r(s,a,s')+\gamma\max_{a'}\tilde{Q}(s',a'),\tag{*}
\end{equation}
therefore the problem becomes as follows:
$$
\min\|r(s,a,s')+\gamma\max_{a'}\tilde{Q}(s',a')-\hat{Q}(s,a)\|.
$$
What I want to ask：
I would like to know the mathematical or theoretical background of the approximation of \eqref{question}, especially why the first approximation is possible. It looks like a very rough approximation. Can the right-hand side be defined as an "approximate Bellman equation"?　I have looked at various literature and online resources, but none of them mention exact derivation, so I would be very grateful if you could tell me about reference as well.

Hi. Can you please edit your post to include a link to the research paper or book where you took these equations from, in order to provide some context?

Both your notation and terminology are quite confusing. For example, I'm not sure what is an "optimal" Bellman operator is. Here's a good clarification on definition of a Bellman operator. Likewise, your description of the DQN algorithm completely ignores the averaging over states/actions/rewards sampled from the replay memory.
Trying to savage your notation, I'll introduce a Bellman operator $B$ that acts on any state-value function $Q$ as:
$$B[Q(s,a)] = \mathbb{E}_{s'}\left[r(s,a,s') + \gamma \max_{a'} Q(s',a')  \right]$$
And optimal Q function $Q^*$ satisfies the Bellman equation:
$$B[Q^*(s,a)] = Q^*(s,a)$$
The value-iteration algorithm iteratively applies the Bellman operator:
$$Q^{n+1}(s,a) = B[Q^n(s,a)]$$
And is proven to converge to the optimal Q function:
$$Q^*(s,a) = \lim_{n\to\infty}Q^{n}(s,a)$$
Now, in the DQN algorithm we are approximating Q functions with DNNs with parameters $\theta$: $Q(s,a; \theta)$. Then we approximate the value-iteration algorithm by minimizing the norm
$$\mathbb{E}_{s,a} \left\|B[Q(s,a; \theta_{i-1})] - Q(s,a; \theta_i)\right\|$$
The minimization is performed over parameters $\theta_i$ with previous parameters $\theta_{i-1}$ held fixed.
The averages $\mathbb{E}_{s,a}$ and $\mathbb{E}_{s'}$ are approximated by sampling a minibatch $MB$ of $(s,a,r,s')$ tuples from the replay memory.
I suppose that is the closest I can get to the crux of your question. You are claiming that the average in the bellman operator application is approximated by a single point:
$$B[Q(s,a)] \simeq r(s,a,s') + \gamma \max_{a'} Q(s',a')$$
While, in fact it is approximated by
$$B[Q(s,a)] \simeq \mathbb{E}_{s'\in MB} \left[ r(s,a,s') + \gamma \max_{a'} Q(s',a')\right]$$


text:
How many papers about AI / ML were published in the recent years?
I am trying to formulate an argument at work saying the disruption in AI/ML is very high and that it is hard to stay "state of the art". I would like to support that hypothesis by numbers.
Question:
How many papers were published in 2018-2020 related to AI (or if that is too generic: ML)?

There are several ML conferences and journals. You can count the number of papers published in the most notable conferences, such as NeurIPS, to provide a lower bound on the number of published papers. In any case, it's possible that there's some tool that does this automatically.


text:
Extracting "hidden" costs from financial statements using NLP
I'm designing a NLP model to extract various kinds of "hidden" expenses from 10-K and 10-Q financial statements. I've come up with about 7 different expense categories (restructuring costs, merger and acquisitions, etc.) and for each one I have a list of terms/synonyms that different companies call them. I'm new to NLP would like some advice on the best approach for extracting them.
Values are usually hidden in two different areas of the document:
Type 1: Free-form text (footnotes)
Values are nested in sentences. Here are some examples, with the Expense Type and Monetary value indicated.

Exploratory dry-hole costs were \$12.7 million, \$1.3 million, and \$1.0 million for the years ended December 31, 2012, 2011, and 2010, respectively.

2012 includes the recognition of a $3,340 million impairment charge related to the carrying value of Citi's remaining 35% interest in the Morgan Stanley Smith Barney joint venture

During the year ended December 31, 2017, we decided to discontinue the internal development of AMG 899, resulting in an impairment charge of $400 million for the IPR&D asset

Type 2: Table data
SEC statements also contain "structured" data in HTML tables. Some line items, like the first row below, correspond to the expense type I'm looking for:

Item
2020
2019
2018

impairment related to real estate assets(2):
398.2
200
0

research and development
100
200
300

other expenses
20
30
40

Correct value = 398.2

I'm thinking about a two-model approach:

Define a new NER model based off the terms I already know (e.g. "dry-hole costs", "impairment charges"). I would need to manually annotate extracts from historic statements that contain these terms for the training set.

For free-form text, it would match the sentence and pass it on for further processing (see 2).
For table data, I would loop over each row using beautifulsoup and pandas, check the first column for a match (e.g. using spaCy's comparison function), and then grab that year's value from the dataframe and finish.

For free-form matches, I still need to grab the monetary value for the correct year (sometimes multiple values are given for various years, see the first example above).

One potential problem here is that sentences like this would cause problems:

We gained $100 million this year, despite facing restructuring charges.

If the NLP algo is split into the above two-model process, model 1 would pass (because it contains a known term like "restructuring charges"), and model 2 would extract $100 million, which is incorrect because it doesn't actually correspond to the expense itself.
Is there a better solution here? As I said, I'm new to NLP and data extraction so would really appreciate any advice or resources to learn more about solving these types of key/value problems.

A lot of ML when applied to financial analysis has the potential to be worth millions. I'm just interested in general strategies that might help solve this kind of problem, not a full implementation.

well why can't hidden also be obvious? other expenses just labeled as "restructuring charges" match (e.g. (2) = $398.2 million applied over year 2020. or am i getting too far off of this one?

any update on what you tried?


text:
Is it possible to use RGB image with decimal values when feeding training data to CNN?
I am working with four grayscale images of float32 data type to perform regression using Keras. Three images are stacked using np.dstack to form a RGB data-set. The last grayscale image is used as label. The grayscale images contains different variations, including [0 , 790.65], [ 150.87 , 260.45], [ -2.74174 , 2.4126 ], [-32.927 , 69.333].
If I convert the images to unit8, the maximum values for the first and second image will be 255 and the decimal values for all images will be lost.  I am having difficulty and struggling to find the solution for using the image in the original data type (float32) when I try to use ImageDataGenerator and flow_from_directory. Can anyone suggest way for that?

Data size is too low. And the loss of information of casting from float to int won't be that high.

Your images should be normalized. You can choose any range (e.g. [-1, 0] or [0, 1]), but the main condition is that it must represent the intensities of the pixels and has to  be able to be scaled back to the original range [0, 255]. btw matplolib can plot normalized images in range [0, 1]


text:
How to Select Model Parameters for Transformer (Heads, number of layers, etc)
Is there a general guideline on how the Transformer model parameters should be selected, or the range of these parameters that should be included in a hyperparameter sweep?

Number of heads
Number of encoder & decoder layers
Size of transformer model (d_model in Pytorch)
Size of hidden layers

Are there general guidelines like number of decoder layers should be equal to encoder layers? Thank you


text:
Are there any inverse RNN layers?
Given the model:
Sequence([
GRU(200, input_shape=(None,100), return_sequences=False)
])

Which maps the space (None, 100) -> (200,)
Is there an InverseGRU such that it maps the space (200,) -> (None, 100)
or, at least, is it possible to simulate this behaviour?

This seems to be somehow a duplicate of this: https://ai.stackexchange.com/q/27246/2444.

@nbro its part of the same problem, but tackling a different aspect. That question is more general, this question is about whether this specific type of layer exists - and where its even possible to implement


text:
Advantages of CNN vs. LSTM for sequence data like text or log-files
When do you tend to use CNN rather than LSTM (or the other way round) in classification or generation tasks of sequential data like text or log-data? What are the reasons for the decision and what does it depend on? Are there any papers or statistics that confirm this?
I'm thinking of data like Linux log entries or short sentence of length of less than 20 words/tokens.
Personally i would almost always use LSTM but I'm curious if CNN wouldn't be better in some cases, if its possible to implement them in a meaningful way. On short sentence there isn't much buffer to use CNN if i'm not mistaken.

The networks with one-dimensional convolution over the temporal dimensions are called Temporal Convolutional Networks. The authors of this study claim that TCNs are comparable to RNNs in performance on common tasks associated with RNNs.


text:
Semantic segmentation - background or ignore for non-target classes?
I am training a deep learning model for semantic segmentation. I am using the cityscapes dataset for training/evaluation.
In cityscapes, there are 34 classes, and of which, we consider only 19 classes and the rest of the classes are ignored. For training, I have assigned the 19 classes with 0-19 traid_ids.
Now, since the rest of the classes are ignored, I have ignored them when computing the loss using cross enropy with ignore_index=255.
But, the above effect can also be achieved by assigning a background class, i.e 20 as bg class and assign all the ignored classes to it.
Now my question is, which method would be better to achieve a high mIoU in cityscapes? And what would be your intuition in choosing the approach?

I don't have much experience with pytorch, but the way you describe them the two seem to be equivalent. One other option I would like to add would be the addition of class weights. Usually the weights are selected to be inverse the imbalance ratio between the classes. This would cause one background pixel's misclassification to have a much less impact to the model's training than one of the actual classes.


text:
Trying to understand why nonlinearity is important for neural networks by analogy
Is the reason why linear activation functions are usually pretty bad at approximating functions the same reason why combinations of hermitian polynomials or combinations of sines and cosines are better at approximating a function than combinations of linear functions?
For example, regardless of the amount of terms in this combination of linear functions, the function will always be some form of $y = mx + b$. However, if we're summing sines, you absolutely cannot express a combination of sines and cosines as something of the form $A \sin{bx}$. For example, a combination of three sinusoids cannot be simplified further than $A \sin{bx} + B \sin{cx} + D \sin{ex}$.
Is this fact essentially why the Fourier series is able to approximate functions (other than obviously the fact that $A \sin{bx}$ is orthogonal to $B \sin{cx}$)? Because if it could be simplified into one sinusoid, it could never approximate an arbitrary function because it's lost its robustness? Because with other terms combined, whereas linear functions summed up gain no further ability to approximate, things like sinusoids actually begin to approximate really well with enough terms and with the right constants.
In that vein, is this the reason why non-linear activiation functions (also called non-linear classifiers?) are generally valued more than linear ones? Because linear activation functions simply are lousy function approximators, while, with enough constants and terms, non-linear activation functions can approximate any function?

Your analogy is correct, except it is not really an "analogy". Sin is an activation function - in past works (before modern deep learning boom) it was rather standard to see it listed as a possible activator.
So your expression $\sigma(x) = A\sin ax + B \sin bx + D \sin ex$ is of a neural network with one 3-neuron layer and a single output linear neuron:
$$\sigma(x) = \sum_i V_i \sin\left(\sum_jW_{ij} x_j + b_i\right)+\beta$$
With all biases being zero $b_i=\beta=0$, output weights are $V_i = \left(A,B,C\right)$ and the inner weight matrix is diagonal:
$$W_{ij} = \begin{pmatrix}a&0&0\\0&b&0\\0&0&e\end{pmatrix}$$

What made the non-periodic functions gain favor over sin?

Ok. Here is an analogy for you. The equation for a neuron is wx + b, which is equivalent to a straight line. If we don't apply non-linearity we will be stuck with a straight line forever. So, this type of network won't be even able to model points in a unit circle randomly distributed.
What does non-linearity do? If you look the graphs for x to the power 2, 3, 4 and so on. You see with each increase in power, the line gets tugged like a sine or cosine curve. That bend in the straight line allows us to then model boundaries with arbitrary shapes.
The more the difficult the boundary to model between the classes the more bends in the line you need to model and so, you keep on increasing the layers in neural network.


text:
How can the gradient of the weight be calculated in the viewpoint of matrix calculus?
Let $\sigma(x)$ be sigmoid function. Consider the case where $\text{out}=\sigma(\vec{x} \times W + \vec{b})$, and we want to compute $\frac{\partial{\text{out}}}{\partial{w}
}.$
Set the dimension as belows:
$\vec{x}$: $(n, n_{\text{in}})$, $W$: $(n_{\text{in}}, n_{\text{out}})$, $\vec{b}$: $(1, n_{\text{out}})$.
Then $\text{out}$ has the dimension $(n, n_{\text{out}})$. So we need to calculate the matrix by matrix derivative, as I know there is no such way to define that.
I know that finally it is calculated as $\vec{x}^T \times (\text{out}\cdot(1-\text{out}))$.
But I can't still get the exact procedure of calculation, why it should be $\vec{x}^T \times (\text{out}\cdot(1-\text{out}))$, not $(\text{out}\cdot(1-\text{out})) \times \vec{x}^T$,I know it by considering dimension, but not by calculation.
My intuition about this problem is that all calculation can be considerd as vector by vector differentiation since $n$ is a batch size number, we can calculate matrix differentiation by considering each column vector.
I'm not sure about my intuition yet, and I need some exact mathematical calculation procedure for the problem,

What is $n$ in your context? The number of samples/examples in your batch that you pass to the net? When you say "I know that finally it is calculated as...", what exactly are you talking about? Do you mean the derivative that you want to calculate? Where does that formula come from?

menas the number of samples.
I know that ∂out / ∂ has that result, because of 'derivative of composition function' and I saw it on my textbook(which is based on some internet resources). It came from the 'derivative of sigmoid w.r.t (xW + b)' * 'derivative of xW + b w.r.t w ( = x.T)'.
I'm just wondering that how can matrix-by-matrix derivative be defined, and ∂out / ∂ have those result.
Simply, I can understand it when  = 1, but not in other case. Sorry for my english nonfluency.

I self-answered to my question, will it be helpful for your question?

I think I understand the process now. Let $x$ has the dimension $(n, p)$ and W has the dimension $(p, q)$. In neural network, $n$ denotes the samples in the batch and $p, q$ denotes input dimension and output dimension of each layer, respectively. We don't use bias $b$ here just for simplicity.
I had trouble in understanding the process of differentiation because of some confusion in vector(matrix) derivative. By changing matrix as set of column vector, now I can solve the problem.
Let $x$ = $\begin{bmatrix}
x_{11}, &x_{12}, &\cdots, &x_{1p}\\
x_{21}, &x_{22}, &\cdots, &x_{2p}\\
\vdots, &\vdots, &\vdots, &\vdots\\
x_{n1}, &x_{n2}, &\cdots, &x_{np}
\end{bmatrix}$, which can be written as $\begin{bmatrix}
\vec{x}_{1}, &\vec{x}_{2}, &\cdots, &\vec{x}_{p}
\end{bmatrix}$.
Similarly, $W$ = $\begin{bmatrix}
w_{11}, &w_{12}, &\cdots, &w_{1q}\\
w_{21}, &w_{22}, &\cdots, &w_{2q}\\
\vdots, &\vdots, &\vdots, &\vdots\\
w_{p1}, &w_{n2}, &\cdots, &w_{pq}
\end{bmatrix}$ = $\begin{bmatrix}
\vec{w}_{1}\\ \vec{w}_{2}, \\ \vdots, \\ \vec{w}_{p}
\end{bmatrix}$.
Then $x\times W$ = $\vec{x}_1 \vec{w}_1 + \vec{x}_2 \vec{w}_2 + \cdots \vec{x}_p \vec{w}_p$. So $\frac{\partial(\vec{x}_1 \vec{w}_1 + \vec{x}_2 \vec{w}_2 + \cdots \vec{x}_p \vec{w}_p)}{\partial{W}} = \begin{bmatrix}
\vec{x}_{1}, &\vec{x}_{2}, &\cdots, &\vec{x}_{p}
\end{bmatrix}^T.$
I still have a little trouble in understanding the last part since actually, $\vec{x}_1 \vec{w}_1 + \vec{x}_2 \vec{w}_2 + \cdots \vec{x}_p \vec{w}_p$ is not scalar, it's $(n, q)$ matrix, but at least now I understand why it is written as above result.
(Also hope to fully understand the last part.)


text:
Identifying if a model is over or under-fitting via graphs
I am working on a Neural Network and have plotted the performance of my model. However the plots seem not to fit the "trends" (which help you identify the issue with your model) presented in this illustration.
Here is the performance of my model
The loss metric I used was Binary Cross Entropy (due to my problem being a binary classification task).
Is my model over or under-fitting? and how can you tell?

What I find weird is that my Validation accuracy is so high, while the Test one is so low

Something is wrong with your testing and validation sets. They are not representative of each other, because if they were they would get similar results (with the test set being slightly worse normally)

Thanks, this may have been due to the fact that I have used SMOTE in order to do some data augmentation (that data made up 30% of my dataset)

In that case I would guess your augmented data isn't representative, and the test set is indicating this. Either way, you can definitely narrow down the issue to whatever differences lie between the validation and test set. The network is learning fine and doesn't seem to be overfitting


text:
What's new in LaBSE v2?
I can't find what's new in LaBSE v2 (https://tfhub.dev/google/LaBSE/2). What are the main highlights of v2 versus v1? And how did you find out?

I did tests comparing LaBSE v2 versus v1. v2 is about twice as slow and returns almost identical results than v1... :(


text:
What algorithm to use to classify data by spatial relations?
Let's assume I have dataset of image-like 2D samples where values can be divided into few discrete levels (for example 1, 2, 3 and 4) like in the image below, where each color maps different value, from 1 to 4. Number of how many times given color occurs on the picture varies from sample to sample though.

I would like to classify these images into different classes but based on the spatial relations of these values between each other (not the values themselves). By spatial relations I mean basically (left, right, up, down), for example:

If blue is above and to the right of the red
Another blue is above and to the left of the same red
Yellow is to the right of one blue (same height)
One green is below red
...

My question is, what algorithm (probably some deep neural network) I should use for that task?
I would appreciate even just some keywords or clues of what might help.

The spatial relationships that you describe would correspond to features, and it's not clear that you need to use a neural network for detecting or discovering these features since you have just described them. Could you instead define a feature extractor that detects the correct patterns and returns you a vector of counts of feature occurrences across the image?
In fact gray level co-occurrance matrices might be adaptable to this problem. Since there are only four colours, each matrix would be very small, and you would define one matrix for each local spatial configuration. For example, one matrix might encode the number of times that colour x is below colour y. From these you could then extract counts of interest to use for further downstream processing.

I think "discovering" them would be proper term here, since I do not know them a priori. Discovering them and then, based on that, doing the classification or clusterisation.


text:
Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)
I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this:
S1 = ['Walk','Eat','Going school','Eat','Watching movie','Walk'......,'Sleep']
S2 = ['Eat','Eat','Going school','Walk','Walk','Watching movie'.......,'Eat']
.........................................
.........................................
S50 = ['Walk','Going school','Eat','Eat','Watching movie','Sleep',.......,'Walk']

The number of unique actions in the dataset are fixed. That means some sentences may not contain all of the actions.
By using Doc2Vec (Gensim library particularly), I was able to extract embedding for each of the sequences and used that for later task (i.e., clustering or similarity measure)
As transformer is the state-of-the-art method for NLP task. I am thinking if Transformer-based model can be used for similar task. While searching for this technique I came across the "sentence-Transformer"-
https://github.com/UKPLab/sentence-transformers. But it uses a pretrained BERT model (which is probably for language but my case is not related to language) to encode the sentences. Is there any way I can get embedding from my dataset using Transformer-based model?


text:
How do I take the correct classification predictions of an ml algo (i.e. random forest/neural net) and sort the instances in each category?
I am trying to sort the instances within each of 5 classification categories in a dataset that has been put through both a random forest classifier and a neural network with 99% accuracy on each.
Essentially what I am trying to do is stack a sorting algorithm on top of a random forest or a neural net (depending on which will boast a much more efficient and swift process in being integrated with a sorting algorithm post-output) so that the correctly classified instances within each category can be sorted into separate organized and easily comprehensible lists.
I have tried researching this but all I have found are examples of ensemble learning and traditional stacking in order to achieve higher overall accuracy on an algorithm.
I am trying to see how I can take the correct predictions from either of these algorithms and sort them by some arbitrary features that I will engineer, but all I am wondering at the moment is how to integrate a sorting algorithm in the first place into the output of a classification algorithm.


text:
Why is it that the state visitation frequency equals the sum of state visitation frequency from initial time step to the horizon?
In the maximum entropy inverse reinforcement learning paper, Ziebart et al. show that the state visitation frequency $\rho(s)$ of a state $s$ can be computed as
$$
\rho_{\pi}(s) = \sum_{t}^{T} P(s_t=s|\pi),
$$
which is the sum of the probability that the state being visited at each time step.
I just don't understand why is it the sum? From my perspective, a frequency should be the less than one, so that it should be the average value
$$
\rho_{\pi}(s) = \frac{1}{T}\sum_{t}^{T} P(s_t=s|\pi).
$$

My feeling is that they define the first equation and will then normalise it to make it a state distribution.

It should be the average and this is rarely mentioned by people except for a IRL summer camp at UCB. You can check this GithubIssue for details.

The equation you show does not appear in Ziebart et al (2008). They do provide a description of the computation in Algorithm 1.
It is the visitation frequency and it is not a probability distribution, so it does not need to be averaged.
If you look at Equation 2. in Arora & Doshi (2020), you a formulation that describes the Algorithm 1 quite well:
$\phi^\pi(s) = \phi^0(s) + \sum_{s'\in\mathcal{S}}P(s,\pi(s),s')\phi^\pi(s')$.
I am not very satisfied with this formulation, because, in my opinion, there should also be a summation over $a\in \mathcal{A}$, like $\eta(s)$ the expected number of visits, in Equation 9.2 in Sutton & Barto (2020).:
$\eta(s)=h(s)+\sum_{\bar{s}}\eta(\bar{s})\sum_a\pi(a|\bar{s})p(s|\bar{s},a)$.
To summarize, in all three descriptions, you just calculate how often a state is visited by policy $\pi$.


text:
Should one rescale (normalize) image before or after data augmentation?
During image preprocessing pipeline, should one rescale each pixel value to [0, 1] by dividing 255 first, and then perform data transformation such as color distortion, gaussian blur? or vice versa?
I believe for correctness, it may depend on the particular image transformation algorithm, or if you use some libraries. Is there any general advice? If anyone has experience trying both before or after, please share, particularly if you use external library or framework that may make implicit assumption to the value range of the pixel.

So far, it seems this is more a software "integration" issue. One great tip from http://karpathy.github.io/2019/04/25/recipe/ is to visualize everything as often as you can during development. For data augmentation, try to visualize the image right before it enters your convnet.
What I found is a bug can happen if your particular image transform library try to detect if you have rescaled or not by just checking the data type. i.e. if it sees float, it will assume [0, 1] or if an integer, it will assume [0, 255].
This can end badly if you resize (without rescaling at the same time) using tf.image.resize(...). Resizing an image (e.g. to 224x224x3) has the "side effect" of converting the pixel value to a float, while still having an (approx.) range of 0.0-255.0. This is such that a downstream library will see a float and make the wrong assumption of [0, 1] range and totally mess up the transform. If you use any color related transforms, definitely be careful, they are usually dependent on pixel value range assumption.
Because of this, if you use tf.image, you may find it safer to do resize and rescale at the same time to ensure that float => [0, 1] such that your downstream color augmentation library may most likely work. Or you just have to manage the type carefully and do tf.cast(...) wherever necessary to satisfy any library assumption.
At the end, a human eye will still have to verify them.


text:
Looking for help on initializing continuous HMM model for word level ASR
I have been studying HMM implementation approaches on ASR for the last couple of weeks. This probabilistic model is very new to me. I am currently using a Python package called Pomegranate to implement an ASR model of my own for the Librispeech dataset. I plan to use my own small-size dataset once I feel comfortable assessing the results of this one with Librispeech.
My problem: Librispeech (and my own dataset) has word-level transcription labels of the audio. The audio files have several word utterances each. Upon generating the MFCCs, I am not sure how to initialize the HMM matrices since the MFCCs, to my knowledge, capture phoneme level context at 10ms windows whereas I am trying to use the current word-level labels. There is no match up to which word each MFCC window belongs. Are the unique words in my corpus to be considered as the individual states in the transition probability matrix? I’m missing the point of how the extracted MFCCs are fed to the model for initialization and/or training?
I’ve been stumped on this for several days and I can’t seem to understand a clear cut explanation in the literature I have read. Any advice and help is very very much appreciated.

Hello. Rather than writing "Looking for help..." in the title, just write your specific question there.


text:
How to train a model for 1 image class to detect anomaly?
I want to train a model with python over the images, and these images are for a metal product.
my aim is to detect the defects, to notice if a product is a failure.
what kind of architecture do you suggest? should I train over the class? or should I use an autoencoder?

Have look into the literature: https://paperswithcode.com/sota/anomaly-detection-on-mvtec-ad

It sounds like you only have "normal" examples with which to train your model, so this makes the problem feel like an application for outlier detection algorithms. There are a variety of approaches here. You could indeed take an autoencoder approach and then use the reconstruction error to determine if a new image is normal or not, on the presumption that normal images will have lower error. You could also take the activations from the bottleneck layer and build an explicit outlier model using an algorithm like isolation forest. And you are not limited to just autoencoders if you take the latter approach -- other models pretrained for other tasks like image net classification could also provide good features for outlier detection.


text:
Comparison between TD(0) and MC ( or GAE )?
I'm getting started with DRL and have trouble distinguishing TD(0), MC, and GAE; and which scenarios one's better than others. Here is what I understand so far:

TD(0): increment learning, can learn after each step instead of waiting for the episode to end. Update bases on one reward, then low variance. However, high bias.

MC: Learn after each episode, the calculation of return is correct. However, its drawback is high variance. And you have to make decisions in the whole episode without an update of parameters.

GAE: combine returns in all steps, get a better trade-off between variance and bias. However, still has to wait until the end of the episode for an update.

I have some questions as follows:

Is variance and bias about the return of each episode? What are their effects on the outcomes (convergence speed of training process, performance of the model)?

Is increment learning important? The ability to correct behaviors after each step may improve convergence speed. However, it can lead to unstable learning ( if I understand correctly, this is why the Target model in Double DQL only updates its parameters for each k mini-batches ). Which scenarios should I use TD(0) or GAE?

Concretely, in my case, I run parallelly a batch with 12 environments, each with 1000 steps. If I use GAE, I make 12000 decisions for each update. All losses of the model are summed up and calculate gradients, after that, I clip gradients to 2.0. Is that too expensive to learn the correct direction? Should I consider using TD(0) here?

Hello. Could you please edit your post to focus on and ask only 1 specific question? If you have multiple questions, split the post into multiple ones.

@nbro Thanks for your advice, but actually there are three related questions. I combine them with the expectation of helping others better understand the problem I have. Anyway, question 3 is the one that I want to answer most now.

I am suggesting you leave 1 question because this also simplifies the life of readers and people that may attempt to answer your question. Even if they are related, I would still suggest this.


text:
What is the name of algorithms that train by competing each other?
In some learning algorithms, we don't directly train models by datasets with labels to predict, but rather we create 2 competing models and let them fight/compete against each other.  As the many millions of epochs pass, the models fight each other, and every time each model improves itself (further optimise its weights) to win.  After many epochs of the models smashing each other, eventually they become really strong super-hero models that can totally blow any human out of the water.  This approach seems to be often used with machine learning models that are tasked to play multiplayer games.  Instead of letting them play with the slow humans, they fight with each other to death for many many epochs to become way stronger than any human can naturally be.
What is the name of such kind of machine learning approach?

You can actually cut down a lot of your question. Are you talking about actor-critic approaches? There are also GANs (Generative Adversarial Networks)

True, and yes.  Post an answer and get some internet points now!

@N.Kiefer - BTW someone here seems to be calling it "multi-agent reinforcement learning".  What do you think?  Is it not the same thing?

I haven't heard of that before, while it seems to fit your question better, I feel like my mentioned methods are more widespread.

Here’s a link about what they call “competitive self-play.” Also, another link from the same source mentioning multi-agent reinforcement learning.

actor-critic or adversarial models.

I wouldn’t say there is competition in actor-critic models. The critic is guessing returns to help actor updates, not trying to become better than the actor.


text:
Why does Adam optimizer work slower than Adagrad, Adadelta, and SGD for Neural Collaborative Filtering (NCF)?
I've been working on Neural Collaborative Filtering (NCF) recently to build a recommender system using Tensorflow Recommenders. Doing some hyperparameter tuning with different optimizers available in the module tf.keras.optimizers, I found out that Adam and its other variants, such as Adamax and Nadam, work much slower than seemingly less advanced optimizers, like Adagrad, Adadelta, and SGD. With Adam and its variants, training each epoch takes about 30x longer.
It came out as a surprise to me, knowing one of the most cherished properties of Adam optimizer is its convergence speed, especially compared to SGD. What could be the reason for such a significant difference in computation speed?

Just to make sure I understand your question, are you asking why the time taken to perform 1 epoch with Adam is longer than the time used to perform 1 epoch with SGD, in the context of NCF?

Exactly. In my case, it is clear that Adam or other Adam-like optimizers converge faster in terms of the number of epochs that it takes them to reach a better set of parameters. However, it takes much longer for them to complete one epoch. Therefore it ends up taking much longer to train the network using such optimizers. I hope this clarifies your question.


text:
Why are weights not initialized with mean=1?
I wonder why weights are initialized with zero-mean. It is one of the reasons, why deep architectures cannot be trained without skip connections. Without the skip connections, the zero initialization becomes problematic, because the identity function cannot be learned in earlier layers (this is a simplified explanation I know). But why can we not initialize weights around one? This would enhance the intrinsic learning of the identity function. Of course, the skip connections also allow a better backpropagation of the gradients, but couldn't this be helpful anyways? Can anyone tell me, why this is not done?

Interesting question,
I can come with 2 explanations why we don't initialize weights with 1 mean value :

It may be easier for the network to learn identity function, but we may have a similar issue about not being able to learn comparison, comparison is quite an important reasoning in my opinion, this is why having negative weight values is important, and initializing all weights around 1 may make it difficult for the network to get negative weights.
Maybe it causes an issue with the non-linearity of the activation functions. If we use ReLU function for example, it only is non linear if there are negative values in the network. It is not uncommon to have positive inputs for a network, so initializing all weights around 1 may lead to have only positives values on the network, which makes ReLU function linear.

Anyway weight initialization has always been a complicated topic and it seems there is no definitive answer to what should be done about initializing weights. I am not an expert of the topic, just giving you my thoughts about it.


text:
Does distribution of data augmentation parameters matter?
Idea
Let's say we have simple pictures dataset containing 40x40 images of digits. We have only one image of each digit. We want to use that as training set, but we need more data, so we use data augmentation. We use only simple operations like translating and rotating and generate 1000 more images of each digit.
Question
Natural way to do data augmentation will be randomly generating parameters like translate_x, translate_y and rotate and applying them into our base image.
Does distribution of these parameters matter? From one hand we would like to have net that is able to recognize digit placed in the side of the image and rotated as well as digit placed in center and not rotated at all but maybe we don't need such accuracy with those borderline images? Maybe we know that our prediction data will be close to the centered ones so we want high accuracy in that cases and the more digit is translated and rotated the lower our's net accuracy might be.
What I mean is can we augmented data with parameters with e.g. gaussian distribution to make our net more sensitive for cases closest to ideal and less sensitive to this borderline cases. Advantage of that would be less training data that we don't need and more control on characteristic of our neural net.
Disclaimer
This digits case is just simple example to show You what I mean.


text:
Reward firstly increase, but after more episodes, start decrease, and weights diverges
I'm making a simple deep Q learning algorithm, with cartpole-v1 env.
Like you can see in chart, after many episodes the reward decrease, some possible reasons?
The exploration vs axplotation algorithm is epsilon-decay, I used a target network (used every mini-batch gradient descent update, in calculating actual Q values, and next Q values, is it right?)
The neural network is made from scratch
the complete code is here:
https://github.com/LorenzoTinfena/deep-q-learning-itt-final-project
# %%
from core.dqn_agent import DQNAgent
from cartpole.cartpole_neural_network import CartPoleNeuralNetwork
from cartpole.cartpole_wrapper import CartPoleWrapper
import gym
import numpy as np
import torch
from tqdm import tqdm
import glob
import os
from IPython.display import Video
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
from itertools import cycle
import sys
import shutil
from pathlib import Path
import shutil
import pyvirtualdisplay
_display = pyvirtualdisplay.Display(visible=False, size=(1400, 900))
_ = _display.start()

# %% [markdown]
# Initialize deep Q-learning agent, neural network, and parameters
# %%
np.random.seed(20)
agent = DQNAgent(env=CartPoleWrapper(gym.make("CartPole-v1")),
                nn=CartPoleNeuralNetwork(), replay_memory_max_size=5000, batch_size=30)

DISCOUNT_FACTOR = 0.995
LEARNING_RATE = 0.0001

n_episodes = []
total_rewards = []
number_steps = []
total_episodes = 0

# %% [markdown]
# Training
# %%
while total_episodes <= 10000:
    total_reward, steps = agent.start_episode_and_evaluate(DISCOUNT_FACTOR, LEARNING_RATE, epsilon=0, min_epsilon=0, render=False, optimize=False)
    print(f'\ntotal_episodes_training: {total_episodes}\tsteps: {steps}\ttotal_reward: {total_reward}', flush = True)
    n_episodes.append(total_episodes)
    total_rewards.append(total_reward)
    number_steps.append(steps)

    for i in tqdm(range(50), 'learning...'):
        agent.start_episode_and_evaluate(DISCOUNT_FACTOR, LEARNING_RATE, epsilon=1, epsilon_decay=0.99, min_epsilon=0.01, render=False, optimize=True)
    total_episodes += i+1

Because it did well at the beginning, it could be that the simpler algorithm you’re using is just less consistent and updates that decrease the rewards sometimes happen. Decreasing the learning rate could help.

Thank you, now I improved algorithm, and at episode 320 the medium reward of 350, for me very well, but after episode 320 it start decrease algorithm, I heard about this fact, that after a maximum, start descrease, but I don't kow why

Try having a bigger batch size. There maybe some really bad batches that make your weights diverge. You can also use prioritized exp.replay ti check if this is the case.

batch size is 10 000, very big! yeah maybe in future I can try prioritized experience replay,
or maybe duel Q learning, I don't know, but for now I think is enough for a high school exam, thank ou for the hint!


text:
Under what circumstances is a fully connected layer similar to PCA?
I am reading this paper on image retrieval where the goal is to train a network that produces highly discriminative descriptors (aka embeddings) for input images. If you are familiar with facial recognition architectures, it is similar in that the network is trained with matching / non-matching pairs and triplet loss.
The paper discusses the use of PCA and whitening on the training set of descriptors as a means of further improving the discriminability (second to last block in image below, fig 1a of paper). This all make sense to me.

Where I'm confused is where they replace PCA/whitening with a trainable fully connected layer with bias. I do understand that PCA+whitening is just the composition of of two linear transformations (ie rotation + (un)squishing in each dimension) and that these are the same as having one linear transformation but:

How is PCA+whitening equivalent to a learnable fully connected layer? Is there some theorem or paper explaining that training a fully connected layer with triplet loss is somehow statistically equivalent to PCA and whitening?
Why is there a bias?


text:
Is there a name for this approach to evolutionary algorithms?
I am considering an approach to evolutionary algorithms, in which instead of maintaining a population of individuals, we maintain a pool of $N$ mutations that can be applied to a base genome.  For every possible subset (or many possible subsets) of the mutation pool, we apply that subset to the base genome to produce an individual, then test that individual.  We discard mutations for which the population with that mutation performs worse than the population without it.  We merge the best-performing mutations into the base genome for the next generation.  Then we replace the discarded or merged mutations with new ones.
Is this known under some name?  Is it a good idea?

When you say "a pool of $N$ mutations", do you mean changes that you can apply to the "base genome"? Then you say "We discard mutations for which the population with that mutation performs worse than the population without it", so does it mean that you will evaluate, say, $M$ individuals that are produced by $M$ sets of mutations (on set of mutations produces an individual) and then compute the fitness of the whole population rather than of individuals?

@nbro fitness of a particular mutation would be measured by the average performance across all individuals with the mutation, minus the average performance across all individuals without the mutation.

But if you have a base genome and apply the same mutation multiple times to that base genome, you will get multiple mutated individuals that are identical. In your case, it seems you maintain only one individual from one generation to the other. However, you apply different subsets of the set of possible mutations to that base individual to produce different intermediate individuals. So, you want to calculate the "fitness of a mutation", i.e. how good it might be if applied?

@nbro that is one way to put it.

It's just a genetic algorithm, only your population is a set of instructions that generate a subject to be tested. There are minor differences in the way you apply multiple mutations, so you evaluate groups of them instead of each individually, but for the scoring you effectively single out the commonalities among the worse individuals.
Hard to tell whether it's a good idea; discarding poor mutations based on them being part of a sub-optimal individual might not be as straight forward as it sounds. And where do you get new mutations from?

One way to form new mutations could be to alter the base genome just enough so that it behaves differently in some particular task, e.g. so it gives the correct output for a particular input.  The mutation could then be (altered genome) - (base genome), and we can add different mutations as vectors to combine them.

It's not a genetic algorithm where the population is the mutations, because we do not evolve or mutate the "population" of mutations.

@causative But you discard bad mutations and replace them with new ones. So the population of mutations changes, and presumably converges towards improvement over time.

No, not really, because the new mutations are not generated as a crossover or mutation of old mutations.  So their quality does not improve over time.  Same as in a regular genetic algorithm.  The difference is that here we explicitly manage gene diversity via the pool of mutations, instead of implicitly managing it as in a regular genetic algorithm.

Also the best mutations go away too because they are merged into the base genome.  It's expected that the deletion of the worst mutations would keep the pool of mutations above a certain quality, but that this quality would be relatively constant over time.


text:
What is the difference between same convolution and full convolution in terms of feature map size?
In valid convolution, the size of the output shrinks at each layer. So after some point of time additional layers cannot meaningfully performs convolution. For this reason, same convolution is introduced, where where the size of the output remains intact. This is achieved by padding with enough number of zeroes at the borders of input image.
What happens to the size of output feature map in case of full convolution?
If it remains intact then what is the difference between same convolution and full convolution?

What happens to the size of output feature map in case of full convolution?

It increases.

First one is valid padding: the blue square is not padded, so the green square is smaller.  Third one is same padding: the blue square is padded just enough so that the green square is the same size.  Fourth one is full padding: the blue square is padded as much as possible for that size of filter, so the green square is larger.
From here.


text:
Is it possible to design an AI with two inputs and a Boolean output?
I am having a difficult time explaining to my boss that what he is trying to achieve may not be possible or within reason. We have a database of 3 Million data points per computer across hundreds of machines and when any data point is updated, changed, or removed. Some of these data points are the number of times a computer has been logged in, the names of printers attach, folders on the root of the drive. Some of the data points we do care about, others like a printer being out of ink, we don't care about but the same method would return if the printer was offline which we do care about.
He wants to design an AI that would check these data points and return with true or false on whether the data point is significant when they are changed, removed or updated. We are storing the name of the method to retrieve the data, the current data, all previous data, and the time the change was made. I can not foresee a way to train the data efficiently as we currently don't know which methods retrieve significant data or which values are not significant within the method.
Is it possible to design such an AI without hours of supervisor learning?

Yes... But...  you have a lot of hours.. possibly days or weeks.. of work before you are at that point.
Your bigger problem is apparent from your concerns in the second paragraph.  It doesn't sound as though your org has a solid grasp of the problem at the moment.  For that reason, it seems that some first steps are in order.
Data Exploration
Begin by collecting all of the data points.  I'd recommend that you perhaps begin with some statistical analysis of the data as a whole, including some basic visualization and generating a covariance matrix.  From there, begin to look at using some clustering methods to identify possible patterns.  Along the way, you will almost certainly go down the path of some dimensionality reduction, either via something like PCA or possibly identifying useless features.
Feature Selection
Based on the exploration work above you should now have a much better understanding of your data and relationships within it.  Based on this, it's time to start thinking about how to generate a model that produces the desired output.  Frankly, you may discover that something as simple as a random forest classification or even a clustering method such as DBSCAN can be used to initially train and then continuously fit your data, producing either a binary classification with the random forest or a yes/no cluster with the clustering technique.
Is More Required?
Of course, something more sophisticated might be required, but if it is you would know have a better handle on the problem and likely be able to intuitively generate a large dataset that could be used for a neural network.
Oh... And as a concluding thought... It might turn out that after all of this analysis the problem cannot be solved with the data that you have.  At that point you have to go back and see if there are other data points that could be gathered.


text:
Predicting the the motion of a 3D object when the motion of a set of markers is known
trying to figure out where to get started with this:
I have a few hundred CT images where certain three-dimensional features in the image (anatomy) are moving in a correlated fashion with a set of radio-opaque markers. These anatomic features can rotate, translate and deform and the markers can move together or sometimes in relation to each other. The position and motion of the markers are indicative of the position and motion of the anatomy in which they are embedded in.
I'd like to develop a model whereby given the positions and motion of the markers I can then predict the position and motion of the full anatomy in which they are embedded and use this for segmentation.
What deep-learning software and techniques are suited for this type of problem?
Thanks!


text:
Cnn for Combination of both digits and letters(small and capital)
Hi I am new to machine learning can anyone suggest open dataset consists of both digits and letters(small,capital)
I want images consisisting of both digits and letters to train my cnn model and make the model recognize the real time images
Can anyone please suggest me that dataset link
Thanks in advance

Hello. This type of question (i.e. asking for a dataset) is off-topic here. Please, read our on-topic page https://ai.stackexchange.com/help/on-topic to know more about our scope. You could try to ask this question on Open Data Stack Exchange. I will wait for the votes of the community members to close this question (the community needs to self-moderate without my or another moderator's intervention).

@nbro Sorry dont know about this I asked about any dataset which is available open source

While not directly an ML question, it is definitely related @nbro .  I think many people struggle with where to find data.  While Google is a great resource, there really aren't that many datasets around.  This question allows us to suggest ways to build your own. :)

@DavidHoelzer I understand that it may not always be easy to find the dataset you're looking for, but, as far as I interpret it, the question is not about "how to build this dataset", but "where can I find this dataset". In both cases, I think the questions are off-topic. There's already Open Data Stack Exchange for the first and Data Science SE for the second. I really appreciate that you're trying to help, but I think we should focus on our scope as defined in the on-topic page. If you think we should change it, feel free to create a post on meta to discuss this ;)

@nbro Oh, I get that he's asking "where can I find this dataset..."  I'm trying to nudge him toward being able to create his own dataset.  :) I'll switch to meta if I'm motivated. :)

As for me, the easiest path to what you are asking for is generating them yourself.
An Example
I usually grab some TTF fonts and put them into a directory so that I have variety for character identification.  Begin by importing dependencies and creating a generator function:
from PIL import Image, ImageDraw, ImageFont
from os import listdir

WIDTH       =   200
HEIGHT      =   100
MINX        =    20
MINY        =    20
MAXX        =    WIDTH-60
MAXY        =    HEIGHT-60
MINSIZE     =    24
MAXSIZE     =    48

fonts = [i for i in filter(lambda i:i[-3:]=="ttf", listdir("../data/fonts"))]

def generate_character_image():
    fonts = [i for i in filter(lambda i:i[-3:]=="ttf", listdir("../data/fonts"))]
    charset=list("0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ")
    while 1:
        img = Image.new('RGB', (WIDTH,HEIGHT), color = (255,255,255))
        font = ImageFont.truetype(f'../data/fonts/{fonts[randrange(len(fonts))]}', randrange(MINSIZE,MAXSIZE))
        canvas = ImageDraw.Draw(img)
        where = (randrange(MINX,MAXX), randrange(MINY,MAXY))
        character = charset[randrange(len(charset))]
        coords = canvas.textbbox(where, character, font)
        # The coordinates are the top left corner
        bounding_box = (coords[0], coords[1], coords[2]-coords[0], coords[3]-coords[1])
        #canvas.rectangle(coords, outline=0, width=1)
        canvas.text(where, character, font=font, fill=(0,0,0), anchor="la")
        yield (character, coords, img)

for i in range(5):
    character,coords,image = next(generate_captcha())
    print(f"{character} is in Bounding Box: {coords}")
    display(image)
    print(image)

You can then use this to generate an infinite number of training samples pretty trivially.
Why This Might Be Good
Personally, I prefer to generate my datasets whenever possible.  It allows me to do the following:

Train without ever repeating a sample
The validation data changes every time (in fact, I'll often not bother with a validation step since every epoch uses different data)
Zero storage required

Hope this helps!

thank you I didnt get what your code gives me an image? an dataset can you give me a brief please

This code is taken from another project.  It will generate a single random character at a random size an with a random font in a random location in an image.  It's perfect for generating a dataset on the fly for training a CNN for character recognition.  You would need to adjust the sizes, etc. for your own problem.


text:
Convolutional Layer Multichannel Backpropagation Implementation
I have been working on coding a CNN in python from scratch using numpy as a semester project and I think I have successfully implemented it up to backpropagation in the MaxPool Layers. However, my model seems to never converge whenever there is a Convolutional Layer(s) added. I am assuming there is a problem with the way I have implemented the backpropagation.
Most examples that I have seen for this implementation either really simplify it by using a one-channel input and a single one-channel filter, or just dive straight into the Mathematics which doesn't only not help but also confuses me more.
Here is the way I have tried to implement both Forward and Backward Propagation for multichannel inputs and outputs based on my own understanding and things I read online.
Forward Prop:

Backward Prop for Filter Gradients:

Backward Prop for Input Gradients:

Kindly point out anything that's wrong here. I have been working on this part for the last 2 days but there has to be a problem because my model never seems to converge.
Thanks!

I'm new to this, but what do you mean by not converging ? the error rates goes bigger ?

Thank you for the response @johnjerrico. The cost keeps fluctuating up and down, never drops as you would expect. Tried changing learning rates from 1 to 1e-8. No change


text:
Tensforflow schedule - does not change boundaries
I'm trying to manipulate the learning rate with tf PiecewiseConstantDecay.
I can easily check if the algorithm switches learning rate values, because one rate is extremely low 1e-20 !!
However, NO setting of the "boundaries" causes the algorithm to switch learning rate... What am I doing wrong?
step = tf.Variable(0, trainable=False)
boundaries = [100]
values = [1e2, 1e-20]

schedule = tf.optimizers.schedules.PiecewiseConstantDecay(boundaries, values)
lr = 1e-4 * schedule(step)

optimizer = tf.keras.optimizers.Adam(learning_rate=lr)
model.compile(optimizer=optimizer, loss='mean_squared_error')

history = model.fit(x = x_train, y = y_train, validation_data=(x_val, y_val), epochs=100,batch_size=32)

Would love your input.

After 100 steps learning rate will switch from 1e-2 to 1e-24.
What exactly is the problem here? Are you confusing steps with epochs?

Are you sure? I've tested this. With 1e-24 learning is completly stalled. However, after 100 steps learning just continues, implying that we are still learning with 1e-2. I'm glad you think it's correct but It does not seem to work


text:
Does batch normalization affects the possible solution distribution?
As far as I recall, in Deep Learning, batch normalization normalizes each layer activations as a gaussian every batch.  If so, Let $x$ be the input and $z_i$ the activation in the $i$-th layer:  $p(z_i)$ becomes a gaussian with batch-norm. Right?
Does this constraint affects $p(z_i|x)$?


text:
Is there a multi-agent deep reinforcement learning algorithm which is for environments with only discrete action spaces (Not hybrid)?
Is there a multi-agent deep reinforcement learning algorithm which is for environments with only discrete action spaces (Not hybrid) and have centralized training?
I have been looking for algorithms, (A2C, MADDPG etc.) but still havent find any algorithm that provides all of properties i mentioned (Multi agent + discrete action space + deep learning + centralized training).
I am wondering if we use an actor network that gets state as input and concatenated discrete actions of agents as output (For example if agent has 3 actions and we have 4 agents output can be [0,0,1, 0,1,0, 0,0,1, 1,0,0]) is that would be bad idea ?

I havn't done RL in a while, but isn't the point of multi agent the fact that they act on their own and thus are disctinct networks ? Why would you use multi agent if you merge them into one network agent ? I remember Unity ML-Agents plugin providing this kind of multi agent learning, maybe you could have a look at how they do it.

@Ubikuity if the agents are homogeneous, then you can have a common NN policy that is executed decentralized but have centralized training. Also, the last time I worked with Unity ML-Agents they didn't have a direct implementation of MARL algorithms. As I remember they have the usual RL PPO and SAC algorithms, although they can work in simple MARL implementations.

A natural policy to act in an environment with discrete action space would be a softmax.
This paper describes a method that uses the idea of centralized training, and I believe could be used in your implementation.
With regard to your last question, I don't know if i understood, but if you have a system that must perform 3 actions, you could assign each action to a specific agent (assuming we have three different action spaces). Then you would have a cooperation game with 3 agents, where all of them have a common reward function. In theory, this 3 agents represents an individual agent that interacts with the environment.


text:
What happens with policy gradient methods if rewards are differentiable?
I would like some help with understanding why there is no explicit flow of information from the reward gradient to the parameters of the policy in policy gradient methods.
What I mean is the following, there are 2 scenarios:
1st - deterministic framework with given initial state $s_0$, actions $a_t = \mu_\theta(s_t)$, rewards $r(s_t, a_t, s_{t+1})$, and transitions $s_{t+1}=f(s_t, a_t)$. Assume all of these things are differentiable (maybe all is continuous). By drawing the computational graph I found I can compute the gradient of $J(\mu_\theta) = \sum_{t} r(s_t, a_t, s_{t+1})$ with respect to $\theta$. I could optimize for cumulative reward by doing gradient ascent on this objective.
2nd - framework in https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#id16 , which seems general, reading $\nabla_\theta J(\pi_\theta) = \nabla_\theta E_{\tau \sim \pi_\theta}[R(\tau)] = E_{\tau \sim \pi_\theta}[R(\tau)\sum_{t=0}^T \nabla_\theta \log{\pi_\theta (a_t | s_t)}]$.
What I don't understand (and I certainly feel confused about in an ignorant way) is why the derivation assumes that $\nabla_\theta R(\tau) = 0$ when this scenario (as it is stochastic) should include the 1st as a particular case, which does have a derivative!
It makes sense that $R(\tau)$ is only affected by $\theta$ through the change in probability of the trajectories $\tau$, but it still feels strange that $\tau = (s_0, a_0, \dots)$ is indeed one sample of $\tau = (s_0\sim \rho(s_0), a_0 \sim \pi_\theta(s_0), \dots)$ which depends on $\theta$. The deterministic reduction is obvious, but one could also think about reparametrization tricks in order to show the same point.
In other words, if the reward function were differentiable, i.e. fully differentiable known environment, how could I use this information?

When you say "reward gradient", what specific function are you considering the gradient of, and with respect to which parameters? I don't think that $R(\tau)$ is a reward function (although it may be composed of a sum of many reward functions), and you may have trouble with different notation and emphasis used in the two sources.

I agree that it's probably a notation confusion. $R(\tau)$ is indeed not a reward function, but probably it's better defined as $\sum_t r_e(s_t, a_t, s_{t+1})$ where $r_e(\cdot) = E[r(\cdot)]$ being $r(\cdot)$ a possibly non-deterministic funcion. The gradients are with respect to the parameters $\theta$ of the policy $\pi_\theta$.

You are asking about the gradients of the "reward function". The policy clearly has a gradient with respect to $\theta$ since it is a direct function of $\theta$. The reward function $r(s,a,s')$ clearly has 0 gradient with respect to $\theta$, since no change to $\theta$ changes it. The situation is less clear when it comees to $R(\tau)$, because it depends on context - e.g. whether you consider $\tau$ to be a random variable, or a specific trajectory - but are you certain that any of your sources claim that has a zero gradient?

I don't agree with the reward function having 0 gradient, and I think the cumulative reward should behave accordingly. It's just enough to write $r(s, \mu_\theta(s), s')$ and the dependecy with $\theta$ is revealed.

The difference is between random variables and observations. There is no dependency on $\theta$ for any given observation, but there is when dealing with expected results. That is why is important to notice when the equations are referencing $\tau$ as an observation (where probabilties from the policy are already resolved) or as a random variable.

Thanks for helping me nail the problem. Now I will continue to ask "why?". Why? Why one formulation and not the other? Why aren't they equivalent (or are they?)? Why

That's exactly the point of the Policy Gradient Theorem. Let's go through he proof of this theorem - it relies on our ability to "loop" the reward expression $J(\theta,s)$, expressing it through $J(\theta,s')$ in the next state:
$$J(\theta,s) = \mathbb{E}_{\tau\sim\pi_\theta}\left[ R(\tau) | s_0 = s\right]$$
$$\begin{align}
J(\theta,s) & = \sum_a\pi_\theta(a|s)\sum_{s',r}P(s',r|s,a)(r + J(\theta, s'))\\
& = \sum_aJ(\theta,a,s)
\end{align}
$$
Here, I've introduced the notation $J(\theta,a,s)$ to keep further derivation sane. Taking the gradient:
$$\begin{align}
\nabla_\theta J(\theta,s) = \sum_{a}\left(\vphantom{\sum_a}\right.&\nabla_\theta  \pi_\theta(a|s) \left(\sum_{s',r}P(s',r|s,a)(r + J(s',\theta))\right)  + \\
&+\pi_\theta(a|s) \nabla_\theta \sum_{s',r}P(s',r|s,a)(r + J(s',\theta)\left.\vphantom{\sum_a}\right)\end{align}
$$
With the first term in the sum we do the log trick:
$$
\frac{\nabla_\theta  \pi_\theta(a|s)}{\pi_\theta(a|s)} \left(\pi_\theta(a|s)\sum_{s',r}P(s',r|s,a)(r + J(s',\theta))\right) = J(\theta,a,s)\nabla_\theta\log\pi_\theta(a|s)
$$
And the second term simplifies to:
$$\pi_\theta(a|s) \nabla_\theta \sum_{s',r}P(s',r|s,a)(r + J(s',\theta)) =
\pi_\theta(a|s)  \sum_{s'}P(s'|s,a) \nabla_\theta J(s',\theta)
$$
The last equality is exactly where we loose the rewards $r$ - the gradient of the constant $r$ is zero, and we can sum over $r$ because of the transition probability normalization $\sum_{r}P(s',r|s,a) = P(s'|s,a)$.
So we've finally get the expression for the gradient:
$$\nabla_\theta J(\theta,s) = \sum_{a}\left(J(\theta,a,s)\nabla_\theta\log\pi_\theta(a|s) + \pi_\theta(a|s)  \sum_{s'}P(s'|s,a) \nabla_\theta J(s',\theta) \right)$$
The Policy Gradient Theorem proof goes on though a couple more steps: unroll the last expression, rewrite through the distribution over states, then convert back from summation to the expectation over trajectories -- going through all this in detail here would too long. While we've already passed the crucial point for your question - we've got rid of the gradient of the rewards.
Edit:
In response to your comment, let me just stress again that this expression is a statement of the Policy Gradient Theorem.
$$\nabla_\theta J(\theta) = \mathbb{E}_{\tau\sim\pi}\left[R(\tau)\sum\nabla_\theta\log \pi_\theta(a_t|s_t)\right]$$
The proof of Policy Gradient Theorem is pretty involved and certainly is not as simple as saying that $\nabla_\theta R(\tau) = 0$. Some authors abuse notation or cut corners when going through the proof of the theorem - for a reasonably strict exposition of the PG theorem I recommend Sutton and Barto, Chapter 13.
Another potential point of confusion is the notation $r(s_t,a_t,s_{t+1})$ that you've been using. In the most general MDP formulation rewards are random variables $r$. The probability of getting a reward $r$ and ending up in state $s_{t+1}$ is encoded in the transition probability $P(s_{t+1},r|s_t,a_t)$. (This covers the case of deterministic rewards by having a deterministic distribution over $r$ ). The derivation above uses this, most general, formulation of the MDP.

Another great answer.

Thank you very much for the detailed answer. Sadly, I fail to understand why this is different from the derivation in the link in the answer. I know that the gradient of a constant is 0, the question is why is $r$ a constant. Could you try to answer the question and address the cases and contradiction there posed?

Hey! thanks for the update! I will go a little further by commenting another question. First, I agree that $r(s_t, a_t, s_{t+1})$ is possibly stochastic. I would ask the following, why is $P(s', r|s, a)$ treated as a constant when I could write $P(s', r | s, \pi^s_theta(s))$ (where $\pi^s$ is after sampling)? Moreover, for deterministic everything and known environment dynamics, $J(\theta, s) = E_{\tau \sim \pi_\theta}[R(\tau) | s_0] = R(\tau = \text{rollout}(\mu_\theta(s_0))) = \sum_t r_e(s_t, \mu_\theta(s_t), s_{t+1})$. Taking gradient at RHS would involve gradient of $r_e$. Is this right?

PS I have read the reference, but the same issue with the desaparition of the function to get a constant is there. I assume it's something like $E[R(tau)] = \int_\theta R(\tau)P(\tau)$ which for the deterministic case $=R(\tau(\theta))$ because $P$ is a delta. Differentiating this yields $\nabla_\theta R(\tau(\theta))$. I have the feeling it's a tricky math issue I'm not noticing :( can you see my point?

PS2 reading your answer again I think you have a point. I am messing up notaiton. Could you remind me of some math? Can I differentiate a random variable $r$ coming from a distribution parametrized by $\theta$? if not, why not, if it depends on $\theta$?


text:
Pixel values of segmap in multi-class semantic segmentation
I'm preparing a dataset for a multiclass semantic segmentation using U-Net like architecture. To be precise, I've got it ready but a question came to my mind. How does pixel values of a segmentation map influence the training?
Also, is it better to have a greyscale segmap, or RGB one?
I have the dataset labeled and augmented, the only thing I am thinking now is if I should alter the segmaps.
I am planning to use Keras, is it smart enough to take in segmaps in both forms? I haven't found an answer anywhere, doublechecked if not on this forum, I hope it's not something super trivial. First time trying to do segmentation task so it's all a bit new to me :)
Edit: right now the segmaps looks like:

background [0 0 0]

object_type1 [16 180 75]    green

object_type2 [225 225 25]   yellow

object_type3 [230 25 75]    red


text:
Terminology for the weight of likelihood ratio/score function?
If we estimate the gradient of $f(x)$ using the likelihood ratio/score function, i.e.
$$\nabla f = f^*\dfrac{\partial \log p(x)}{\partial \theta}$$
is there any agreed upon terminology to call "$f^*$"? Specifically I'm thinking of the case where you may use some sort of baseline/control variate or a critic, so $f^*$ is not $f$.
I've seen $f^*$ called the learning signal or the cost. In reinforcement learning, you would call $f^*$ the advantage, but I think that terminology is only specific for RL. What is a general way to call $f^*$ that is not specific to RL?