text:
What are the various problems RL is trying to solve?
I have read most of Sutton and Barto's introductory text on reinforcement learning. I thought I would try to apply some of the RL algorithms in the book to a previous assignment I had done on Sokoban, in which you are in a maze-like grid environment, trying to stack three snowballs into a snowman on a predefined location on the grid.
The basic algorithms (MC control, Q-learning, or Dyna-Q) seemed to all be based on solving whichever specific maze the agent was trained on. For example, the transition probabilities of going from coordinate (1,2) to (1,3) would be different for different mazes (since in one maze, we could have an obstacle at (1,3)). An agent that calculates its rewards based on one maze using these algorithms doesn't seem like it would know what to do given a totally different maze. It would have to retrain: 1) either take real life actions to relearn from scratch how to navigate a maze, or 2) be given the model of the maze, either exact or approximate (which seems infeasible in a real life setting) so that planning without taking actions is possible.
When I started learning RL, I thought that it would be more generalizable. This leads me to the question: Is this problem covered in multi-task RL? How would you categorize the various areas of RL in terms of the general problem that it is looking to solve?

The basic algorithms (MC control, Q-learning, or Dyna-Q) seemed to all be based on solving whichever specific maze the agent was trained on.

All RL algorithms are based on creating solutions to a defined state and action space. If you limit your state space representation and training to a single maze, then that is what will be learned. This is no different from other machine learning approaches - they learn the traits of a population by being shown samples from that population (not just one example). They also need to be built for the range of input parameters that you need them to solve.
In the case of RL, and your maze solver, that means the state representation needs to cover all possible mazes, not just a location in a single maze (there are ways to internalise some of the representation to the learning process such as using RNNs, but that is not relevant to the main answer here).
The toy environments in Sutton & Barto are often trivial to solve using non-RL approaches. They are not demonstrations of what RL can do, instead they have been chosen to explain how a particular issue related to learning works. Sutton & Barto does include a chapter on more interesting and advanced uses of RL - that is chapter 16 "Applications and Case Studies" in the second edition.

When I started learning RL, I thought that it would be more generalizable.

It is, but without some kind of pre-training to support generalisation from a low number of examples, you have to:

Model the general problem

Train the agent on the general problem

RL agents trained from scratch on new problems can seem very inefficient compared to learning by living creatures that RL roughly parallels. However, RL is not a model for general intelligence, but for learning through trial and error. Most examples start from no knowledge, not even basic priors for a maze such as the grid layout or the generalisable knowledge of movement and location.
If you do provide a more general problem definition and training examples, and use a function approximator that can generalise internally (such as a neural network), then an agent can learn to solve problems in a more general sense and may also generate internal representations that (approximately) match up to common factors in the general problem.


text:
How to test the robustness of an agent in a custom reinforcement learning environment?
I have used the stable-baseline3 implementation of the SAC algorithm to train policies in a custom gym environment. So far the results look promising. However, I would like to test the robustness of the results. What are common ways to test robustness? So far, I have considered testing different seeds. Which other tests are recommended?

This depends on your definition of robust.
Robust to what exactly?
Testing different random seeds will test the robustness of the algorithm on stochasticity of the environment and the algorithm's optimization procedure.
Trying different hyperparameters would test the robustness of the algorithm to hyperparameter changes.
Some RL benchmarks have their own definition of robustness:
The L2RPN benchmark (https://competitions.codalab.org/competitions/25426) defines robustness as a policy that is able to respond properly when unexpected events or adversarial attacks happen. In benchmarks such as Atari or Procgen, which have multiple tasks, a robust algorithm is one that can solve all the tasks.
If you mean robust, as in, the algorithm do indeed learn some inherent pattern on how to solve the task, as opposed to superficially memorizing sequences of actions, you could add noise to the observations (e.g. simple gaussian noise), try to add some sort of adversarial attacks, or try the sticky action idea used in Atari benchmarks.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Changing a CNN-LSTM image captioning architecture to use BiLSTMs
Currently I'm dealing with an assignment that made us implement the network mentioned in this paper. The network has an architecture similar to this:

As you can see it uses a Unidirectional RNN (in my case LSTM), which does the many to many sequence prediction task while training, giving LSTM outputs to dense layers with softmax activation. For generating the captions, the network is only given the image at first, and then using the prediction of the image, generates a word, which is then fed to the network along with the generated hidden state, and the model does this recursively to find a unique stop token. Here's the prediction code:
def predict(self, image, max_len = 30):
      output = []
      hidden = None
      inputs = self.encoder(image).unsqueeze(1) # Image features
      for i in range(max_len): # Recursively feed generated words to LSTM
        lstm_out, hidden = self.decoder.lstm(inputs,hidden)
        output_vocab = self.decoder.fc(lstm_out)
        output_vocab = F.softmax(output_vocab.squeeze(1), dim=1).detach().cpu().numpy()
        words_indices = output_vocab.argsort(axis=1).squeeze()
        word = words_indices[-1]
        if word == self.unk_token_index:
            word = indices[-2]
        output.append(int(word))
        if word == self.end_token_index:
          break
        inputs = self.decoder.embed(torch.LongTensor([[word]]).to(image.device))
      return output

The problem I'm having right now is that I don't know whether this generation scheme works with BiLSTMs. Right now my training loss is way better for the sequence to sequence prediction task than the UniLSTM, but my generated captions are far worse.
This is a sample caption generated by Bi-LSTM:

This is a sample caption generated by UniLSTM:

My training loss for BiLSTM converges to 10e-3, while for UniLSTM it converges to 0.5. But the problem is that even before overfitting, BiLSTM is only generating gibberish.

So after doing a bit of research, I finally found out why the model is not working at all when I change the LSTM to Bi-LSTM.
The task of the learning is Next Word Prediction for each cell of LSTM. When you have a Uni-directional LSTM, this is inherently a tough task for the model to learn good representations that can help it generate the next word with enough confident.
What happens when you change the model to a Bi-LSTM is that, if you concatenate the forward and backward values of each cell together, you have now the information of the very next word you wanted to predict via the backward route.
To alleviate this issue, Wang et al. propose to do prediction on forward and backward routes separately while training the data, and for generating, see which route has more confidence in its generated caption.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Best way to use/learn ML for board-game reinforcement learning
I am relatively new to Python but I taught myself enough to code a two-player board game that is similar to chess. It has a simple Tkinter UI. Now I am dipping into machine learning, and I want to write another program to play itself in this game repeatedly and "naturally" learn strategies for playing the game.
Can anyone give advice on what I might be able to use for this? Is Tensorflow a good option? Is there a Python library well suited for this that I could adapt and train? I am partially through the buildingai.elementsofai.com course, but I am still very new at ML / AI.

If it's a 2-player game it goes a little deeper into RL if you want both sides to be RL algorithms. I recommend reading about game theory and what is a Nash Equilibrium to start. For algorithms hiivemdptoolbox has openai-gym compatibility as well as Q-Learning. You will need to add code to make 2 learners play each other. I would also recommend adding Dyna-Q to the Q-Learner as it will probably speed up the learning.

I was thinking the same thing about leveraging learning from both sides. I guess I couldn't have the same learner play both sides and split the data on the go?

There are many approaches - the initial one would be a rule based one with some amount of randomness. The ML-AI approach is some variation of reinforcement learning, defining your game as an environment, see for instance openai-gym. ”Some variation” might be Deep Q Learning or A3C.


text:
Are policy and value iteration used only in grid world like scenarios?
I am trying to self learn reinforcement learning. At the moment I am focusing on policy and value iteration, and I am finding several problems and doubts.
One of the main doubts is given by the fact that I can't find many diversified examples on how to implement these on python, instead I find always only the classical grid world example.
So, my doubt is: Are policy and value iteration used only in grid world like scenarios, or can be used also in other contexts?

Policy and value iteration both require you to, for each possible transition and each corresponding possible reward at each state, compute a statistic of $r + \gamma V(s')$. In order for this to be tractable, you need for there to be at most finitely many states, actions, possible rewards, and possible transitions at each state. You also need to know the transition model. This is the case in gridworld.
Gridworld is not the only example of an MDP that can be solved with policy or value iteration, but all other examples must have finite (and small enough) state and action spaces. For example, take any MDP with a known model and bounded state and action spaces of fairly low dimension. Then you can approximate the state and action spaces with a finite number of bins, each corresponding to its own " discretized state/action". With smooth enough dynamics and enough bins, you'll be able to solve the MDP with policy/value iteration on the discretized spaces.
In many interesting RL problems though,

You don't know the transition model, and/or
The state space, action space, and/or reward space are too large

In these cases you wouldn't be able to compute the value function exactly, so you can't really do policy/value iteration. However, in most value based RL algorithms, the policy evaluation / policy improvement steps are approximated using sample transitions and function approximators.

Sorry, just one comment. Suppose that I am in a scenario in which I want to use value iteration, but I don't have a final state to reach. I would like to have a reward equal to zero if the agent remains in the initial state and a reward of -1 if it moves. I want to find the best state. Can I solve this with value iteration?  Thanks.

You should be able to solve that with value iteration if the state and action spaces are finite or can be discretized.


text:
How do we get from conditional expectation on both state and action to only state in the proof of the Policy Improvement Theorem?
I'm going through Sutton and Barto's book Reinforcement Learning: An Introduction and I'm trying to understand the proof of the Policy Improvement Theorem, presented at page 78 of the physical book.
The theorem goes as follows:

Let $\pi$ and $\pi'$ be any pair of deterministic policies such that, for all $s\in S$,
$q_{\pi}(s,\pi'(s))\geq v_{\pi}(s)$.
Then the policiy $\pi'$ must be as good as, or better than, $\pi$. That is, it must obtain greater or equal expected return from all states $s\in S$:
$v_{\pi'}(s)\geq v_{\pi}(s)$.

I take it that for the proof, the policy $\pi'$ is identical to $\pi$ except for one particular state $s$ (at each time step) for which we have $\pi'(s)=a\neq \pi(s)$, as suggested by @PraveenPalanisamy in his answer here.
The proof start from the statement of the theorem: $v_{\pi}(s)\leq q_{\pi}(s,\pi'(s))$
And then $q_{\pi}(s,\pi'(s))$ is developed as $\mathbb{E}[R_{t+1}+\gamma v_{\pi}(S_{t+1})|S_{t}=s,A_{t}=\pi'(s)]=\mathbb{E}_{\pi'}[R_{t+1}+\gamma v_{\pi}(S_{t+1})|S_{t}=s]$
I don't understand how did we get rid of the condition $A_{t}=\pi'(s)$. I don't think it's related to adding the subscript $\pi'$ to the expectation because it's something that should be done by definition since for the following time steps we choose policy $\pi$ which is exactly $\pi'$.

I don't understand how did we get rid of the condition $A_{t}=\pi'(s)$.

We don't really, it is just moved into the subscript $\pi'$ in $\mathbb{E}_{\pi'}[]$ - it means the same thing here, that the next action is chosen according to the modified policy $\pi'$. Moving the condition around is part of the proof's strategy, which eventually expresses the expectation in a familiar way so that we end up with a something that matches the definition of $v_{\pi'}(s)$.

Am I right in thinking that this follows because $\pi'$ is a strictly deterministic policy?

@DavidIreland It is not required, and there is a version of the policy improvement theorem for improving epsilon-greedy policies too. There are a couple of intuitive looks at that too - e.g. an epsilon-greedy policy has identical expected returns to the matching greedy policy but with a change to reward and state transitions where the environment bechaves with probabiliy epsilon as if a random action was chosen. But yes in the first view of the policy improvement theorem in the book it is a deterministic policy.

Thank you for your answer @NeilSlater but trying to get the expression in a familiar way to $v_{\pi'}(s)$ isn't so rigorous am I wrong? Especially if we do that in the proof, why don't we do the same in the definition of the action value function instead of explicitely state the condition on the action $q_{\pi}(s,a)=\mathbb{E}[G_{t}|S_{t}=s,A_{t}=a]$. If it's just for clarity, I believe there wouldn't be much of a difference between $q_{\pi}(s,a)$ and $v_{\pi}(s)$.

@Daviiid It is not "just for clarity" and I do not say that in the answer. It is algebraic manipulation, same as e.g. rationaiising a fraction or cancelling terms on two sides of an equation.

@NeilSlater yes I know sorry if it came as if it was in your answer. I just stated the "just for clarity" in relation with the $q_{\pi}(s,a)$ case to say that I don't believe it's for clarity but for another purpose.  I really understand your answer from an "intuition" point of view. Since we're sure that we're taking action $\pi '(s)$ and then following $\pi '$ which is a deterministic policy, it's seems like a "factorization", in some sense, of the expression. But is it rigorous? I mean, based on measure theorems and properties we don't have a clear implication or do we?

@Daviiid There is no need to look deeper. The only difference between terms in the subscript of the expectation, and terms in the conditions is that the latter are usually specific boundary measurements whilst the former are more general rules on the expectation. Note the $v_{\pi}(S_{t+1})$ inside the $\mathbb{E}{\pi'}$ means that the general case of $\pi'$ is being ignored after the first timestep (for getting $R{t+1}$ and $S_{t+1}$) because $v_{\pi}$ relies on following $\pi$, not $\pi'$.

@NeilSlater I see, I think I'm getting a clearer view. So when we write $q_{\pi}(s,a)=\mathbb{E}{\pi}[G{t}|S_{t}=s,A_{t}=a]$ it means we're trying to evaluate the action $a$ taken in state $s$ and then following policy $\pi$, this could also be written as $q_{\pi}(s,a)=\mathbb{E}{\pi '}[R{t+1}+\gamma v_{\pi}(S_{t+1})|S_{t}=s]$ where $\pi '(s)=a$ but $\pi '$ is the same as $\pi$ for any other state. I think I had a hard time conciliating the definition which focused on the condition $A_{t}=a$ and the fact just we can take it off (while modifying the subscript). Thank you again!


text:
Why is ancestral sampling used in autoregressive models?
I have been reading about autoregressive models. Based on what I've read, it seems to me that all autoregressive models use ancestral sampling. For instance, this paper says the following in Abstract:

We introduce a deep, generative autoencoder capable of learning hierarchies of distributed representations from data. Successive deep stochastic hidden layers are equipped with autoregressive connections, which enable the model to be sampled from quickly and exactly via ancestral sampling.

However, what I don't understand is why (as I understand it) all autoregressive models use ancestral sampling. Why is ancestral sampling used in autoregressive models?

My understanding is that the answer to this question is basically 'ancestral sampling is used in autoregressive models because it fits well the structure/dynamics of autoregressive models (the ancestor-descendent relationship, etc.)'. It's not a very satisfying answer, but my understanding is that it's correct.
If anyone has a better answer, feel free to post.


text:
Is it possible to have values of the states equal to $0$ at the end of the value iteration?
I am new to Reinforcement Learning and I am trying to self learn it. I have already posted some quesiton here and your answershave been really useful to me, so here I am posting another one.
I am studying the value iteration, and while doing the simulation using python, I get that at some states it is associated a value of $0$. I think I have to mention that I have tried to assign to the ststes an initial value different from zero, in order to simulate the fact that the agent already have some information about the enviroment before starting.
So, my questio is:
Is it possible to have values of the states equal to $0$ at the end of the value iteration?

Is it possible to have values of the states equal to 0 at the end of the value iteration?

Yes.
For a start, all terminal states should have a value of zero. This is not usually learned or calculated, but is by definition because the value represents the sum of expected future rewards and a terminal state should not have any. However, if the terminal states are implemented as "absorbing" states which always return 0 reward and do not transition away, then they can be learned as having a value of zero by e.g. value iteration. Caveat: This only works for a discounted return with $\gamma \lt 1$.
In addition, it is entirely possible to have an expected return of zero from any state. This might be composed of multiple positive and negative rewards that happen to sum to zero in expectation. A simple deterministic path through the state space that ended in a terminal state without receiving any positive or negative reward along the way would also have a value of zero.


text:
When does a neural network have a single and when does it have multiple outputs?
What I understand is, each input in a neural network is a feature.
However, what I don't understand is, when we need multiple outputs in a neural network.
For example, say, if we are classifying cats and dogs, only one output is enough. 0 = cat, 1 = dog.
When does a neural network have a single and when does it have multiple outputs?

The output depends on what answer you want from the network. Think of the network as a function $f$ with weights $\theta$, that takes an input $x$ and gives some output $y$:
$$
f_\theta(X) \rightarrow y
$$
The example you give (dog=1 or cat=0) is binary classification -- the answer from the network is "this looks more like a dog than a cat" if the output $y > 0.5$ or vice versa. So yes, $y$ is a scalar between 0 and 1.
If you had three or more classes e.g [dog, cat, banana] (multimomial classification), you want the network to answer how "confident" it is that the $x$ is a certain class. It will give a prediction for each class. In this example $y$ will be a vector of size 3, each element being a prediction for a class e.g y = $[dog=0.3, cat=0.01, banana=0.7]$
For a non-classification example: say a robot's action comprises the velocity  $v$ and a joint angle $\phi$ (action = $[v, \phi]$). The robot's questions a neural network with "what action should I take?". The network's answer $y$  might looks like $ y = [v=3.0, \phi=23]$. So the size of the output will be dependant on the dimensions of the action. In this case we have a multiple-dimension output, one dimension for velocity, and one for the joint angle.


text:
What are the labels in figure1 in the Paper "The perceptron: A probabilistic model for information storage and organization in the brain"?
This figure

comes from The perceptron: A probabilistic model for information storage and organization in the brain
I guess the first circle (neuron) labels RETINA, the second labels perceptron area, what about the third one? what are the labels pointed out by the arrows?

Circles: RETINA / $A_I$ (POJECTION AREA) / $A_{II}$ (ASSOCIATION AREA)
Labels: (LOCALISED CONNECTIONS) / (RANDOM CONNECTIONS) / (RANDOM CONNECTIONS) again / RESPONSES


text:
GLIE MC control (reinforcement learning): how the policy affects evaluation?
In his lecture 5 of the course "Reinforcement Learning", David Silver introduced GLIE Monte-Carlo Control.

I understand that we do policy evaluation for one step and then policy improvement. My question is how does the improved policy come into play in this GLIE algorithm?
Is Gt (return) based on the policy somehow? is that where the new policy comes in?
Asked another way, how are policy evaluation and policy improvement connected in this image?

Hello. Welcome to AI SE! I just want to let you know that you case use latex on this site, so you may want to format your math symbols it ;)

The policy is used in determining the next sequence of state-action pairs in the next episode. This means that the policy is determining indirectly the next Gt


text:
How can I address missing values for LSTM?
I'm a student and writing my first paper for submission on conference. I have a question
there is a dataset below. this is temporal-spatial dataset.
Date         Hour   City       Sensor1  Sensor2  Sensor3 Sensor4 ...
21-06-10     0      Region1      0.12     0.52    0.33     0.44  ...
21-06-10     1      Region2      0.16     0.83    0.34     0.49  ...
21-06-10     2      Region1      0.21     0.44    0.57     0.5   ...
...

My Task is anomaly detection for each region
I want to use LSTM. So, I represent the temporal-spatial data to two time-series data. my dataset can be represented below.
City       Date       Hour     Sensor1  Sensor2  Sensor3 Sensor4 ...
Region1   21-06-10     0         0.12     0.52    0.33     0.44  ...
Region1   21-06-10     2         0.21     0.44    0.57     0.5   ...
...

City       Date       Hour     Sensor1  Sensor2  Sensor3 Sensor4 ...
Region2   21-06-10     1         0.16     0.83    0.34     0.49  ...
...

However, then, there is no a row with attribute 'Hour=1' in Region1 dataset
(you can see the table below)
City       Date       Hour     Sensor1  Sensor2  Sensor3 Sensor4 ...
Region1   21-06-10     0         0.12     0.52    0.33     0.44  ...
Region1   21-06-10     1         NaN      NaN     NaN      NaN   ...
Region1   21-06-10     2         0.21     0.44    0.57     0.5   ...
...

Can I insert estimated values into the row with attribute 'Hour=1' in Region1 dataset? (for example, I want to insert average between the first row and the third row)
Can I claim to have utilized a real world dataset even with this missing value estimation?

You can claim to use a real-world dataset, you would just need to specify that some values were interpolated.
Do you have to have the inter-mediate values though? By the looks of it, each "region" was only measured every 2 hours, so I would just keep it that way and just have the resolution be 2 hours. It doesn't have to be hourly, and probably shouldn't since that isn't the resolution of the data by the looks of it.
If it does need to be hourly then it is fine to just linearly interpolate the data. Additionally, you can try and train the network to accept empty inputs (though It'd definitely be easier to just interpolate your dataset)

Thank you for your explanation. According to your advice, Is it ok that I use my dataset with missing value estimation to submit a paper?
Can I say that this is not data manipulation?

@Dae-YoungPark I can't say for sure because I don't have the full dataset, but it doesn't look like there is any missing data. It just looks like the measurements were taken every 2 hours (for each region), which is fine. It's the same as measurements being taken every minute, just because there isn't data at every second doesn't mean there's missing data - it just means the resolution is minutely.

The measurements is not taken every 2 hours. Actually, About Region1, there is a row with Hour=3. Thus, I think there are some missing values. In such a case, can I claim to use a real-world dataset even though some data are interpolated in data science area?

If you're using real-world data then it is a real-world dataset. If you can still draw representative conclusions about the data, what difference does it make?


text:
How to measure the similarity the pronunciation of two words?
I would like to know how I could measure the pronunciation of two words. These two words are quite similar and differ only in one vowel.
I know there is, e.g., the Hamming distance or the Levenshtein distance but they measure the "general" difference between words. I'm also interested in that but mainly I would like to know how they sound differently. I think there must be something like this to test text-to-speech results?
Best would even be an online source where I could just type in those two words.

There are a handful of tools available for manually comparing pronounciations, though all are limited in some way. Depending on your usecase, you might be interested in:

Wikspeak: a tool that transcribes (single) words into IPA and generates a pronounciation. A web demo is available, though it’s a bit sensitive about browser versions.
espeak-ng: provides a CLI tool that does text-to-speech or text-to-IPA transcription

# use the —-ipa flag to display the inferred IPA transcription
espeak-ng -v en-US --ipa "horse”
# => hˈɔːɹs
espeak-ng -v en-US --ipa "hoarse"
# => hˈoːɹs

If you want a more automated solution, you could look into python libraries like eng-to-ipa to do IPA transcription (including disambiguation when a word can map to multiple IPA transcriptions). You could then try applying edit distance measurements to the IPA transcriptions to estimate the similarity of the pronounciations.

Thanks a lot! I know I didn't target this in my question but do you know if the IPA is used in text-to-speech? Means, do machines try to translate by making use of the IPA or are they "just" trained von human speech and the like?

In regard to Wikspeak: This, and I've seen 1-2 others, cannot apply the IPA to fantasy names. I wonder why is this? The IPA should be valid to any word and, e.g., Coca Cola or Pepsi works though they are fantasy names. But another fantasy word is unknown. Are they all are just using a lookup table? I thought the IPA is a kind of a system which should be applicable to "any" word?

@Ben About the use of IPA in text-to-speech engines: I’m not sure, but understanding is that most traditional text-to-speech engines use some smaller language-specific set of phonemes internally (the simplest engines being a lookup table from word to phonemes, then stringing together pre-recorded voice clips for each phoneme). Nowadays it may be possible to train end-to-end on text -> audio, but I imagine it’s usually easier to model the problem as text -> phoneme_sequence -> audio. (e.g. Mozilla’s TTS)

Mozilla’s TTS seems to use the phonemizer package for pre-processing text into a phoneme sequence that is given to a deep learning models as input, but at a glance it seems like it optionally allows passing a config flag to disable phoneme preprocessing. (see this text-to-model-input preprocessor function, text util module)

@Ben About Wikspeak and others not being able to transcribe fantasy names, you’re correct that they’re just using a lookup table (usually the CMU Pronouncing Dictionary for english). The challenge is that the pronounciation isn’t fully specified by the latin alphabet. In principle though it should be possible to guess a plausible IPA transcription for a fantasy name - here’s a deep learning model you could try.

Thanks a lot again! How has "The challenge is that the pronounciation isn’t fully specified by the latin alphabet" to be understood? For almost the entire vocabulary there is a pronunciation (covered by the pronunciation alphabet), or not?

For dictionary words there is indeed an easy “specification” - someone has catalogued all the sounds that native speakers tend to make for a given latin alphabet word and simply published that as a lookup table. However, humans are so good at learning pronunciation that it’s easy to forget how much we deviate from the “ideal” pronunciations of each latin character. E.g. the can be "thuh" /ðə/ or "thee" /ði/ based on context, refusecan be /ˈɹɛfjuːs/ or /ɹɪˈfjuːz/ based on meaning (_garbage_/_refusal_), and the wordstough, cough, doughandboughall have differentough` sounds.

That’s why an lookup table created from empirical observation of native speakers in a given region is the only fully reliable way to map from a word (plus extra information like tense) to its pronounciation(s). When humans encounter unfamiliar words, we infer plausible sounds based not just on the letters but on intuitions about word evolution and even history, like “does the word look like a latin or french import?” (leading to weirdness like hyperforeignisms). That’s why fantasy names are hard - how would fantasy Englishmen say Worcestershire?

The last one is a good one :) Thanks again!


text:
Why do I get bad results no matter my neural network function approximator for parametrized Q-learning implementation for Contextual Bandits?
I'd like to ask you why, no matter my neural network function approximator for parametrized Q-learning implementation for a Contextual Bandits environment, I'm getting bad results. I don't know if it's a problem with my formulation of the problem and how I'm trying to solve it, or is it the neural architecture. I tried different fully-connected neural networks with different number of layers and different number of neurons (sticking to low numbers since my environment is not complex) but I always get bad results, and it seems the results are random.
if my implementation of the Q-learning algorithm for the Contextual Bandits problem is right. I made an environment that randomly generates three integers between 0 and 89 and given an action (integer between 0 and 4) it returns a reward following a certain logic (if all three integers are between 0 and 29 and the action is 0 then the reward is 0 otherwise it's -1).
My environment is:
class Environment():

  def __init__(self):

      self._observation = np.zeros((3,))

  def interact(self, action):
      self._observation = np.zeros((3,))
      c1, c2, c3 = np.random.randint(0, 90, 3)
      self._observation[0]=c1
      self._observation[1]=c2
      self._observation[2]=c3
      reward = -1.0
      condition = False
      if (c1<30) and (c2<30) and (c3<30) and action==0:
          condition = True
      elif (30<=c1<60) and (30<=c2<60) and (30<=c3<60) and action==1:
          condition = True
      elif (60<=c1<90) and (60<=c2<90) and (60<=c3<90) and action==2:
          condition = True
      else:
          if action==4:
              condition = True
      if condition:
        reward = 0.0

      return {"Observation": self._observation,
                  "Reward": reward}

The interaction method doesn't return state or time step, not like what TF-Agents environments' step method does. I just thought it's not necessary for the current problem; I don't rely on time steps since each state doesn't influence the next state. I thought that observation is what should be returned, the state being a more general data that could contain information the agent can't observe. I don't return the action too because we can get it outside the environment.
My function approximator of the Q-values are neural networks, always a fully connected architecture. For instance:
model = keras.models.Sequential([
        keras.layers.Dense(16, activation="relu", input_shape=[n_inputs]),
        keras.layers.Dense(16, activation="relu"),
        keras.layers.Dense(n_outputs)])

I took the next blocks of code from Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition and adapted them to my situation:
env = Environment()

n_inputs = 3 #Observations are made of three integers
n_outputs = 4 #Four actions

def epsilon_greedy_policy(observation, epsilon=0):
  if np.random.rand() < epsilon:
    return np.random.randint(4)
  else:
    Q_values = model.predict(observation[np.newaxis])
    return np.argmax(Q_values[0])

replay_buffer = deque(maxlen=2000)

def sample_experiences(batch_size):
  indices = np.random.randint(len(replay_buffer), size=batch_size)
  batch = [replay_buffer[index] for index in indices]
  observations, rewards, actions = [np.array([experience[field_index] for experience in batch]) for field_index in range(3)]
  return observations, rewards, actions

def play_one_step(env, observation, epsilon):
  action = epsilon_greedy_policy(observation, epsilon)
  observation, reward = env.interact(action).values()
  replay_buffer.append((observation, reward, action))
  return observation, reward

batch_size = 16
optimizer = keras.optimizers.Adam(learning_rate=1e-3)
loss_fn = keras.losses.mean_squared_error

def training_step(batch_size):
  experiences = sample_experiences(batch_size)
  observations, rewards, actions = experiences
  target_Q_values = rewards
  mask = tf.one_hot(actions, n_outputs)
  with tf.GradientTape() as tape:
    all_Q_values = model(observations)
    Q_values = tf.reduce_sum(all_Q_values * mask, axis=1, keepdims=True)
    loss = tf.reduce_mean(loss_fn(target_Q_values, Q_values))
  grads = tape.gradient(loss, model.trainable_variables)
  optimizer.apply_gradients(zip(grads, model.trainable_variables))

epsilon = 0.01
obs = np.random.randint(0,90,3)
for episode in tqdm(range(1000)):
  if episode<250:
    obs, reward = play_one_step(env, obs, epsilon)
  else:
    obs, reward = play_one_step(env, obs, epsilon)
    training_step(batch_size)

I'm not sure at how to evaluate the performance of the agent, but I tried this as a first approach just to see if the predicted Q-values will enable a greedy-policy to choose the best action:
check0 = np.random.randint(0,30,3)

for i in range(30):
  arr = np.random.randint(0,30,3)
  check0 = np.vstack((check0, arr))

predictions = model.predict(check0)

c = 0
for i in range(predictions.shape[0]):
  if np.argmax(predictions[i])==0:
    c+=1

(c/predictions.shape[0])*100

Every time I ran the code above it gave me a totally different value. Sometimes it's 0%, sometimes it's 45%, sometimes it's 19%...
The issue is that no matter my model architecture, at the end, I get random results. I wonder if it's something wrong in the overall approach to solve the problem. I want to solve a Contextual Bandit where the agent observe a continuous context, take actions and try to link together the rewards obtained with the actions and the context in order to "understand" the logic behind it.
I hope you can help me figure out why do I get these random results.
Thank you.

Scale your neural network inputs.
The raw observations are in range $[0,89]$, and neural networks will cope badly with that used as inputs.
The ideal case for NN for each input feature is a gaussian distribution with mean 0, standard deviation 1. You don't need that to be perfect, though. A simple scale - divide each element by $30$ and subtract $1.5$ - will be fine here.
You can keep the environment as-is, and scale after observations are received. Up to you whether you put ready-scaled observations in the experience replay table or not. In your case it may be very slightly more efficient to do so in terms of CPU effort, but probably not something you would notice.
There are other ways you might deal with these kinds of numbers in a neural network's input, but pre-scaling to a standard range is usually the simplest and by far the most common solution.

Yeh the number 1 mistake a lot of beginners make (myself included!) was not normalizing their data. It makes an enormous difference

Thank you for your answer. I applied the changes you suggested and while it improved the randomness of the predictions, it seems like the model now is only preferring one action over the others. So the maximum of the outputs will always be the fourth one, or always the first one... I tried to change the architecture again and that's how I got those different preferences but for one fixed architecture it's the same preference again and again. I wonder if I should make another post or just stick with the comment, but I'd like to ask you if you have any clue about what's going on. Thank you again

@Daviiid There is probably some other bug in your code. I just quickly scanned it and noticed the lack of scaling, which as Recessive points out is a common beginner's issue. Sorry I do not have time to look deeper at your code and find other implementation issues or bugs - it is very likely that you have something of that nature, where someone needs to work with you in more detail to fix the issue. If you are working alone, try to break the components apart (instead of one long script), and unit test them

@NeilSlater Yes I fixed the scaling issue as you suggest. No need to look at the code in details thank you for guiding me, I'll look at the code in every detail. Should I remove the question since it's not really RL?

@Daviiid: IMO, your code contains one AI conceptual problem (not scaling) which is answered here, so meets the site goals. It gets tricky when there is more going on, but a lot of people do come here with first attempts at RL problems and scripts not quite working. There is some debate between AI SE admins about when a question like this crosses a line from being useful to being too noisy to help next person. But I think it is fine to keep this question.


text:
Confusion about faster RCNN neither object nor background label
I am trying to construct a faster RCNN from scratch using KERAS. I am generating the tensor which contains whether anchor at each location corresponds to  object or background or neither for training the RPN.
The   output tensor for the RPN is suppose H x W x L where the L dimension  corresponds to whether an object is detected or is background or neither based on IOU thresholds.
My question is this: What should be the label value for neither an object nor background label and how to stop the gradient flow for this label.

There is no label for such bounding boxes, they are simply "ignored" during training. You can assign any value for their "labels", then multiplying what ever loss these boxes generated with 0. If there is no loss, there is no gradient from these boxes.
You can do that by defining a count_boxes vector with binary values. Object and background are counted, so value is 1.0. The remaining "ignored" boxes are marked 0.0. Then pair-wise multiply this count_boxes vector with the loss vector your model generated.

I have labelled object  with (1,0) , not object with (0,1) and neither object nor background with (-1,-1). What if I first select the labels containing object/not object before calculating the loss using tf.where? This way I can ignore the (-1,-1) labels.  Will this approach work?  And thank you for your response.

Yes, I think using tf.where should work just fine. You should also pay attention to the imbalance between object and background, they are often heavily skewed. And you are welcome.


text:
Is my understanding about the number of iterations correct?
Per google machine-learning glossary, when I have 100 training examples and update my model for each training example, if I train my model 5 epochs without early-stop, there are 500 iterations in total, is my understanding correct?

Updating model for each training example means batch size of 1, aka stochastic gradient descent(SGD).
1 iteration is defined as forward propagate, calculate loss, backpropagate and finally update weights.
Since batch size is 1, running 5 epochs on 100 training examples with SGD means you will do 500 iterations, yes.


text:
Using a Neural Network (LSTM) to approve/reject word-type sequences
I would like to train an LSTM neural network to either "approve" or "reject" a string based on the word-type sequence.
For instance: "Mike's Airplane" would output "approved", but "Airplane Mike's" would output "reject".
My method for doing this is to decompose the string into an array of words.
eg.
["Mike's", "Airplane"]

, then convert the array of words to an array of word-types since the actual word is irrelevant.
The word types (pronoun, noun, adjective etc.) are defined constants having numerical values.
eg.
const wordtypes={propernoun:1, adjective:2, noun:3, ownername:4};
console.log(wordtypes.propernoun); // 1

Mike's Fast Airplane is
["Mike's", "Fast", "Airplane"]

which becomes:
input:[properNoun, adjective, noun]
output: "approve"

properNoun represents the first word(Mike's),
adjective the second word(Fast),
and noun the third word(Airplane).
I would then like to use this array to train a Neural Network so that it can approve/reject other word-type sequences.
I am concerned with the methodology/algorithm rather than the syntax; I'm extremely new to Machine Learning and Artificial Neural Networks, so, I am using brain.js and NodeJS because they're relatively easy to use.

I would like to input multiple parameters for a single word because
many words have multiple word types (depending on the context). For
example, a word can be both a "noun" and a "verb". How do I represent this input?

Is this a good application for LSTM? Or is there a better-suited ML
algorithm? My dilemma is in deriving the proper inputs & methodology
to effectively train the Neural Network.

How is my methodology for accomplishing this approval system?

Language is complicated. Me and my friend know two Mikes, one of whom is a pilot, and we call him "Airplane Mike". I ask "Whose drink is this?" and my friend replies "Airplane Mike's". Contrived? Yes. This kind of construct is a real problem for NLP systems? Yes. It doesn't stop you making up any set of word rules you like and training an LSTM to learn it, but it may help if you explain whether your goal is just an abstract rule learning thing for teaching yourself RNNs, or whether you intend to use it for real world language processing, or for some other purpose.

Yes, it's a real-world application, but not NLP. It's actually input validation for a web-based system.
For example:
input: [ adj, noun] output: "reject"
input: [ noun, noun, noun] output: "reject"
input: [ pronoun, adj, noun] output: "approve"
input: [pronoun, noun] output: "approve"
etc.
The input is of variable length; "noun", "pronoun" etc are defined as integer constants.

You already figured out much of the problem. You can solve it with sequence models like LSTM/GRU.
One-hot encode word-types. Assume there are types of [properNoun, adjective, noun] as you said. Then "Mike" will be represented as a vector, [1,0,0], "fast" as [0,1,0], and "airplane" as [0,0,1].
Summing up these, you will train a model that takes [[1,0,0], [0,1,0], [0,0,1]] as input and return binary classification result.
Thus you can use LSTM/GRU with return_sequences = False.
An example with Keras is here.
A few things you should be careful about:

I have shown 2D input here but sequence models take 3D input in form of (batch_size, time_steps, number_of_features) so you will need to reshape even if you train with batch size 1. More details are here.
Prepad the time dimension with 0 vector, so that number of time steps will be equal for all samples. More details are here.

For multi-word type, you can create multiple data instances and train the network.


text:
What is the meaning of R2 appearing as a negative in the RandomForestRegressor?
Machine learning model was created by reading an Excel file where data was stored. I applied RandomForestRegressor to create a model that predicts the size of the sieve particles according to pressure, but the value of R2 is too large negative. I found out through googling that R2 can be negative, but I don't know what it means to have such a large negative. When I applied the same amount of different data to the designed model, R2 showed a result that was close to 1, but I don't know why this data is only large negative. RMSE and SCORE scores are good, but I don't understand if only R2 scores are bad... I would appreciate it if you could let me know what is the problem and what to consider.
My Data(Capture Image):

My Code:
import pandas as pd
import numpy as np
import sklearn
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.ensemble import RandomForestClassifier
from sklearn import metrics
from google.colab import drive
from sklearn.metrics import mean_squared_error
from sklearn.metrics import r2_score

drive.mount('/gdrive', force_remount=True)

data = pd.read_csv(r"/gdrive/MyDrive/Coal_Inert_Case/Inert_Case_1.csv")

x =data[['press1', 'press2', 'press3', 'press4', 'press5', 'press6', 'press7', 'press8', 'press9', 'press10', 'press11', 'press12', 'press13',
 'press14', 'press15', 'press16', 'press17', 'press18', 'press19', 'press20', 'press21',
 'press22', 'press23', 'press24', 'press25', 'press26', 'press27', 'press28', 'press29',
 'press30', 'press31', 'press32', 'press33', 'press34', 'press35', 'press36', 'press37',
 'press38', 'press39', 'press40', 'press41', 'press42', 'press43', 'press44', 'press45',
 'press46', 'press47', 'press48', 'press49', 'press50', 'press51', 'press52', 'press53']]

y = data[['Sieve 16000', 'Sieve 11000', 'Sieve 8000', 'Sieve 5600', 'Sieve 4000',
       'Sieve 2800', 'Sieve 2000', 'Sieve 1400', 'Sieve 1000', 'Sieve 710',
       'Sieve 500', 'Sieve 355', 'Sieve 250', 'Sieve 180', 'Sieve 125',
       'Sieve 90', 'Sieve 63', 'Sieve 44', 'Sieve 31', 'Sieve 0']]

X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.3,random_state= 42)

forest = RandomForestRegressor(n_estimators=1000,random_state= 42)
forest.fit(X_train, y_train)
y_pred = forest.predict(X_test)

mse = mean_squared_error(y_test, y_pred)
rmse = np.sqrt(mse)
r2_y_predict = r2_score(y_test, y_pred)

print("RMSE:", rmse)
print("R2 : ", r2_y_predict)

The output (RMSE, R2) is:
RMSE: 0.6913667737213217
R2 :  -7294765918428.414

Take a look at either of these great posts about negative R2 values.

What does negative R-squared mean?
When is R squared negative?

TLDR is that your model is poorly fit to the data.
From looking at the code you attached I would try reducing the number of x features you are using. It is possible that there is multicollinearity or some feature just are not useful. Using sklearn you can use recursive feature elimination RFE or recursive feature elimination and cross-validated RFECV.
You would just need to do something like this.
from sklearn.feature_selection import RFE

# fit the selector
selector = RFE(forest, n_features_to_select=5, step=1)
selector.fit(X_train, y_train)

# print the top features
selector.support_


text:
An online editor that allows data labeling format
I have a set of students (~20) that will work on annotating data for an NLP project.
The annotation task will be as in the following:
text: I like this piza place.
label: [pos, neg]
comments:
text fluency: [1,2,3,4,5]

The students will need to correct the text first (e.g. correcting piza word), and then fill the fields below.
Is there an online solution to add the data in this format and then to share the link with the students?
I tried to do this in Google forum, but I wasn't able to; I don't know actually if it's possible there.
I am looking for a solution that can allow saving the edits after annotating # instances, as there are many instances and the students won't be able to annotate everything at once. I know that a good solution would be building a website, but I am looking for something that already exists.

The good folks behind Spacy have their paid product called Prodigy which is a data labeling tool. I haven't used it but it appears you can host it somewhere and then you would just have to send the link to the students. It is a little pricey but you get a lifetime license...
A free alternative might be Label Studio but I am not sure how easy it is to host it somewhere.
Hope this helps!


text:
How to use a heuristic policy to increase sample efficiency of a deep reinforcement learning agent?
I have a heuristic solution to a problem which works quite well when certain environmental parameters are known and unchanging. However, in a real world setting these parameters will not be known and are likely to fluctuate over the course of an episode. I'm hoping to use deep RL to develop a policy that will be similar to the heuristic, but robust to these unknowns.
My question is: does the RL agent need to be trained "from scratch" as one would typically do or is there a way to leverage the existing policy to jump start the training progress?
In the latter case, what would this looks like? I've had a couple of thoughts, but I'm not sure how well any of them would work.

Reward actions that the heuristic would take in an environment with static parameter values, then gradually make the environment more complex and set a new reward function based on what I'm actually interested in.

Instead of taking random actions in the exploration stage, take actions dictated by the heuristic.

What you're looking for is called Imitation Learning (IL), in which we are interested in learning an expert policy $\pi_*$ which we assume to be optimal.
However, there are many different ways we can approach such learning setting.
Just to give some examples, we might be interested in Behavioural Cloning, where our parametrised policy $\pi_\theta$ (the RL agent) is trained in a supervised fashion to mimic the expert behaviour on a set of demonstrations $\mathcal{D} = \{(s_i, \pi_*(s_i))\}^N_{i=1}$:
$\theta^* = argmin_\theta \sum_{s \in \mathcal{D}}\mathcal{L}(\pi_*(s),  \pi_\theta(s))$
where $s_i$ represents a state and $\pi(s_i)$ the action taken by the policy.
Other approaches involve learning a reward function $\mathcal{R}_\phi$ from $\mathcal{D}$ and use it to reward our agent (Inverse Reinforcement Learning) or GAIL (Generative Adversarial Imitation Learning). I suggest you take a look at this brief overview.
So, to answer your question: yes, you can leverage expert behaviour in your RL algorithm. Should you do so? It depends. IL is not always advisable as the agent might get stuck in a suboptimal policy, especially if the expert you're learning from is poor.
You can also combine IL and RL by starting with the former early on the training and then switch to the latter for later epochs.
Unfortunately, no way of knowing which works best a priori.
Hope this was useful.


text:
Is this calculation of the vector-Jacobian product in the PyTorch documention wrong?
In the official PyTorch documentation there is the following calculation (here):
$$
J^{T} \cdot \vec{v}=\left(\begin{array}{ccc}
\frac{\partial y_{1}}{\partial x_{1}} & \cdots & \frac{\partial y_{m}}{\partial x_{1}} \\
\vdots & \ddots & \vdots \\
\frac{\partial y_{1}}{\partial x_{n}} & \cdots & \frac{\partial y_{m}}{\partial x_{n}}
\end{array}\right)\left(\begin{array}{c}
\frac{\partial l}{\partial y_{1}} \\
\vdots \\
\frac{\partial l}{\partial y_{m}}
\end{array}\right)=\left(\begin{array}{c}
\frac{\partial l}{\partial x_{1}} \\
\vdots \\
\frac{\partial l}{\partial x_{n}}
\end{array}\right)
$$
However, I am wondering why the result isn't as follows:
$$
J^{T} \cdot \vec{v}=\left(\begin{array}{ccc}
\frac{\partial y_{1}}{\partial x_{1}} & \cdots & \frac{\partial y_{m}}{\partial x_{1}} \\
\vdots & \ddots & \vdots \\
\frac{\partial y_{1}}{\partial x_{n}} & \cdots & \frac{\partial y_{m}}{\partial x_{n}}
\end{array}\right)\left(\begin{array}{c}
\frac{\partial l}{\partial y_{1}} \\
\vdots \\
\frac{\partial l}{\partial y_{m}}
\end{array}\right)=\left(\begin{array}{c}
m\frac{\partial l}{\partial x_{1}} \\
\vdots \\
m\frac{\partial l}{\partial x_{n}}
\end{array}\right)
$$
As the matrix is multiplied by a vector and the $\partial y_{x}$ terms cancel out it is should be $m$ times $\frac{\partial l}{\partial x_{x}}$.
I know that the official PyTorch docs are probably right but I just can't get my head around why.

I think your error is assuming that
$$\frac{\partial y_1}{\partial x_1} \frac{\partial l}{\partial y_1} = \frac{\partial l}{\partial x_1}.$$
While 'cancelling' derivatives like this works sometimes, it is not true in general, and treating these quantities as fractions is not really correct in the usual formulation of calculus in terms of limits.
If you run through the calculation using the chain rule with Jacobian matrices, you should recover the correct product. If you're not clear on the compositions, write the functions $y_i(x_1, \dots, x_n)$ and $l(y_1, \dots, y_m)$. If it helps define $y(x_1, \dots, x_n) = (y_1, \dots, y_m)$ then use the chain rule to compute $l(y(x_1, \dots, x_n))$. If it would help to see the calculation in more detail, do let me know. Otherwise, it's a good exercise.


text:
Data analysis before feeding to ML pipeline
I'm new to machine learning and I've been working through a dataset of ~3000 records with ~100 features. I've been hand rolling Python and R scripts to analyse the data. For example, plotting the distribution of each feature to see how normal it is, identify outliers, etc. Another example is plotting heatmaps of the features against themselves to identify strong correlations.
Whilst this has been a useful learning exercise, going forward I suspect there are tools that automate a lot of this data analysis for you, and produce the useful plots and possibly give recommendations on transforms, etc? I've had a search around but don't appear to be finding anything, I guess I'm not using the right terminology to find what I'm looking for.
If any useful open source tools for this kind of thing spring to mind that would be very helpful.

Hello. Welcome to Artificial Intelligence SE. It seems to me that this question is off-topic here and would be better suited for https://softwarerecs.stackexchange.com/ and https://datascience.stackexchange.com/. Asking for tools or software libraries is generally off-topic here. You can read more about our scope on our on-topic page (basically, we focus on the theoretical, philosophical and social aspects of AI, so programming issues, including asking for tools, are generally excluded).

This is a great question because indeed there are many tools out there to make this part of the process faster.
I have used as I usually stick to the following two:

Pandas profiling
Facets

You can also search for alternatives
Hopefully, the community can help complement my answer.

all you mentioned belongs to any statistical analysis - it is suited for Normal Data & is done with least-squares methods, but Maximum-Likelihood-Estimations can be applied for non-normal data (e.g. for rare-events analysis)... ML, in general, do not need Normality of data distribution, but some preprocessing is needed from the developer by his desire - see opportunities for preprocessing in sklearn's preprocessing class


text:
Why many deep learning research papers continue to be in arXiv?
There are plenty of research papers, especially in deep learning are present only in arXiv with large number of citations. I cannot find them in journals as peer-reviewed ones.
For example if I search for Conditional Generative Adversarial Nets then I can find only an arXiv pre-print and has been Cited by 5722
This is not the single paper and I personally found lot of papers in pre-print only with no journal/conference affiliation. Many research papers are at-least 3 years old.
Is it solely due to the will of authors or is there any other reason for this phenomenon of not getting published even though they are widely accepted especially in the domain of deep learning?

I would imagine that the speed of development is faster than the publishing and review cycle of journals. And unless peer-reviewed publications are required for career progression, there is no point in going through the effort of preparing them. Much better to continue working and make new discoveries instead. If something is peer-reviewed, that doesn't necessarily mean it's higher quality, just that it is accepted by the establishment in your field.

Simple answer: the peer-review process can be slow and not all papers deserve to be accepted at (major) conferences and journals (e.g. some are just tutorials or simple or more complete descriptions of previously published ideas, so they do not propose anything novel, which is required for being accepted at major conferences or journals).
Why is peer review slow? Because multiple people need to read the paper carefully, understand it (so they may need to read about the topic, if they are not fully familiar with it), analyze it, and maybe read (previously published) related literature to know if the proposals are really novel or deserve to be published (if you're proposing a VAE, you will not go anywhere, as VAEs have already been proposed and accepted at ICLR 2014, i.e. no progress and novelty!).
I found several papers on arXiv to be useful, but I also found low-quality ones. Generally, if you can cite a paper that has been accepted at a major conference or journal in your field, you should do that, rather than citing a pre-print. If a paper has been peer-reviewed, then it's been scrutinized in multiple senses (although some still contain some typos), so it's very unlikely to find a paper that has been accepted, for example, at NeurIPS that is poor (e.g. the GAN or transformer papers were accepted there). You will find so many useful papers published there or in conferences/journals like ICLR, JMLR, and so on. You should also notice that some papers that are first published on arXiv might be later presented/accepted at some conference or journal (example: VAE), so the citation may redirect you to the pre-print version, but this does not mean that there isn't a version that was later published in some journal. People may first publish on arXiv so that other researchers can have early access to the ideas, and so progress can be faster, but, again, there are many papers on arXiv that will never be accepted at any (major) conference/journal.
(By the way, although this question was asked in the context of deep learning (hence Artificial Intelligence), so it can be considered on-topic here, you probably would find more useful answers at Academica Stack Exchange, because this topic is not unique to research in AI (e.g. there's also bioRxiv, which has a similar role to arXiv, i.e. proliferation of ideas, etc., but for biology). In fact, you can already find some answers there that address this question, for example, this one.)


text:
What's the difference between a "perceptron" and a GLM?
In a comment to this question user nbro comments:

As a side note, "perceptrons" and "neural networks" may not be the same thing. People usually use the term perceptron to refer to a very simple neural network that has no hidden layer. Maybe you meant the term "multi-layer perceptron" (MLP).

As I understand it, a simple neural network with no hidden layer would simply be a linear model with a non-linearity put on top of it. That sounds exactly like a generalized linear model (GLM), with the non-linearity being the GLM's link function.
Is there a notable difference between (non-multi-layer) perceptrons and GLMs? Or is it simply another case of two equivalent methods having different names from different researchers?

The perceptron uses the Heaviside step (or sign) function as the activation function (so you are not free to use any activation function), while a GLM is a generalization of linear regression, where the link function can be, for example, the logit (which leads to the logistic regression), identity function (which leads to linear regression), and so on. The sign function has the role in the perceptron as the sigmoid in logistic regression.
So, GLM models have a probabilistic interpretation (i.e. you assume that the response variable follows some distribution), while perceptrons do not, even though, for example, the perceptron and logistic regression can both be used for classification.


text:
What is the bit memory task?
I learned from this post about the so-called bit memory:

They froze its self-attention and feed-forward layers and, in separate copies, fine-tuned peripheral layers on each on a wide range of tasks: Bit memory (memorizing strings of bits), Bit XOR (performing logical operations on pairs of strings of bits), ListOps (parsing and performing mathematical operations), MNIST, CIFAR-10 (classification of images), CFAR-10 LRA (classification of flattened, greyscale images), and remote homology detection (predicting what kind of protein structure an amino acid is part of).

I wonder what the "bit memory" task is? Is it an identity function as described in this post? Or the memory network?

Read the paper. It tells you. (page 3)

Bit memory. Similar to the task proposed by Miconi et al. (2018), we consider a bit memory
task where the model is shown 5 bitstrings each of length 1000. Afterwards, the model is shown
a masked version of one of the bitstrings, where each bit is masked with probability 0.5, and the
model is tasked with producing the original bitstring. The bitstrings are broken up into sequences of
length 50, so that the models are fed 120 tokens of dimension 50.

It wasn't immediately clear to me whether they train the model on the bitstrings (in which case 5 is quite low and a Transformer model is not required) or just present them to it when running the model. Figure 4 (page 8) makes it clear that it's the latter, as the original bitstrings and the query both appear on the same attention plot.
i.e. the input is something like:
00011110011100100001000101110011...
01000101001101100011001110010110...
11000101111111001010011010111010...
01011101001100100011111000000100...
11110100011010011000101100010101...
query
*10**10*1**1110*****0110**11**1*...


text:
Assume 120 examples, a model makes 20 correct predictions and updates weight for the other 100. Should I count this epoch 100 iterations or 120?
Per google's glossary, an iteration refers to

A single update of a model's weights during training ...

The following code comes from a github repo
def fit(self, x, y, verbose=False, seed=None):
    indices = np.arange(len(x))
    for i in range(self.n_epoch):
        n_iter = 0
        np.random.seed(seed)
        np.random.shuffle(indices)
        for idx in indices:
            if(self.predict(x[idx])!=y[idx]):
                self.update_weights(x[idx], y[idx], verbose)
            else:
                n_iter += 1
        if(n_iter==len(x)):
            print('model gets 100% train accuracy after {} epoch(s)'.format(i))
            break

Note that this model doesn't update weights for each single example, because when the model make a correct prediction for some example, it skips the example without updating weights.
In this kind of scenario where model makes a correct prediction for $i$th input $x_i$ and jump into next example $x_{i+1}$ without updating weights for $x_i$, does it count as an iteration?
Assume there are 120 training examples, in one epoch, the model makes 20 correct prediction and updates weight for the other 100. Should I count this epoch 100 iterations or 120 iterations?
Note: This question is NOT about coding. The code cited above works well. This question is about terminology. The code is just to illustrate the scenario in question.

This really depends on how you define "iteration". Here it's not so simple, given that the number of training steps per epoch varies based on the number of correct predictions, which would obviously change as you continue to train the model.
Generally I have found iterations refers to the number of times you run a batch through a network. This is the more reasonable definition, as when you are doing real-world machine learning, using batches of training examples allows you to utilize as much memory as possible, thus making the training much faster.
Using this definition with your example you would always have the same number of iterations as the batch size would be consistent, it's just inside the massive matrix that is your batch, you would have update values of 0 where the network correctly predicted.
It ultimately comes down to what's the best way to convey information to a reader? I think describing the number of iterations as variable based on network outputs is confusing and non-descriptive. In this case, if you don't want to use any batches, it is best to say you have 120 iterations per epoch. Saying you have up to 120 iterations is confusing. Instead, just specify that in some iterations the network may not be updated.

A quick google the keyword "definition of iteration in machine learning" gives us a lot of results. I would like to stick with this StackOverFlow question.
As your example, if we have 100 samples, let me assume batch size is 20, so the number of iteration is 5. If there is one iteration that both 20 samples are predicted correctly, this epoch should be counted which means one epoch you still have 5 iterations since the number of iterations is important in some situations such as control the learning rate while training (decay or cycle).
If you feel uncomfortable with the non-gradient / non-updating on that iteration, you can understand as your model's weight is updated with the gradient is 0.


text:
Reason for relaxing limit in derivative in this context?
Consider the following paragraph from NUMERICAL COMPUTATION of the deep learning book..

Suppose we have a function $y = f(x)$, where both $x$ and $y$ are real
numbers. The derivative of this function is denoted as $f'(x)$ or as
$\dfrac{dy}{dx}$. The derivative $f'(x)$ gives the slope of $f(x)$ at
the point $x$. In other words, it speciﬁes how to scale a small change
in the input to obtain the corresponding change in the output:
$f(x+\epsilon) \approx f(x) + \epsilon f'(x)$.

I have doubt in the equation $f(x+\epsilon) \approx f(x) + \epsilon f'(x)$ given in the paragraph.

In strict sense, the derivative function $f'$ of a real valued
function $f$ is defined as
$$f'(x) = \lim_{\epsilon \rightarrow 0}
 \dfrac{f(x+\epsilon)-f(x)}{\epsilon}$$
wherever the limit exists.

If I replace the original definition of the derivative as follows
$$f'(x) \approx \dfrac{f(x+\epsilon)-f(x)}{\epsilon}$$
then I can obtain the equation given in the paragraph i.e,  $f(x+\epsilon) \approx f(x) + \epsilon f'(x)$.
But, my doubt is that how can I modify the definition with $\lim\limits_{\epsilon \rightarrow 0}$ to an approximation with out limit? How can the following two are same?
$$f'(x) = \lim_{\epsilon \rightarrow 0} \dfrac{f(x+\epsilon)-f(x)}{\epsilon} \text { and } f'(x) \approx \dfrac{f(x+\epsilon)-f(x)}{\epsilon}$$

The equation
$$f(x + \epsilon) \approx f(x) + \epsilon f'(x)$$
is justified from taylor series. It is not derived from the limit definition of a derivative. Let $f'(a)$ and $f''(a)$ exist for $a$ in the interval $(x, x+\epsilon)$. Then,
\begin{align*}
f(x + \epsilon) = f(x) + \epsilon f'(x) + \frac{\epsilon^2}{2}f''(b)
\end{align*}
where $b$ is some number between $(x, x + \epsilon)$. If $\epsilon << 1$, then $\frac{\epsilon^2}{2}f''(b)$ is small so it can be ignored, leading to the approximation.
taylor's theorem

It is just an assumption. If we assume $\epsilon$ is small enough (depending on the function $f$), you can remove the limit $\lim_{\epsilon \to 0}$ for the approximation.


text:
How to calculate policy probability ratio in multiple action space
I try to solve a navigation problem with PPO; my actions space have three-part:

robot linear velocity that is in [-3,3] range (getting from a tanh activation func)
robot linear angular that is in [-pi/6, pi/6] range (getting from a tanh activation func)
robot step-time duration that is from [0.2, 0.5, 0.8] (getting from a softmax activation func)

The problem that I face is how to calculate the ratio of probability from this separate disturbing?
Mean or sum? or was there another way to calculate log_prob from different distributions? Something like log_prob from multivariable distribution!

It depends on how you have modelled the probability distributions that you are sampling from when you turn the neural network output into a specific action.
If your three action dimensions are sampled independently (which would be a standard way to model a policy like this), then the probabilities or probability densities multiply. Which means the log probabilities add.

I got my actions from 3 different output layers of actor-network, and each one has its own mean and std. So I must use the sum function to add log_prob and save it in the memory.


text:
Using Human Confirmation in place of a loss Function for Training
Has there been any experimentation in designing an AI to prompt a human to judge the accuracy of it's outcomes? instead of using a loss function, a human can judge the accuracy of it's estimation using some kind of metric, where it can then use that too update it's weights.
I was looking for some feedback on whether this is a plausible idea.
I was thinking that for domains that lack sufficient training data to solve problems this could be a possible solution.
Of course, it isn't feasible to judge every iteration of a training loop. So maybe feedback could be provided for the average of a number of estimations. Maybe every 100 estimations you could provide feedback.
It may not be a great training method because of the sparsity of feedback, but it could provide a place to start if you don't have a lot of data to throw at your problem initially.

What you are suggesting is similar to active learning and reward modelling.
To summarize both quickly, active learning is used when data are scarce or when the labeling process is too time consuming (almost always the case in NLP). To speed up the process, the idea is to train a model in performing not only a task, but also an estimation of its uncertainty when performing that task. For example, the model could output a classification and a probability associated with that classification that tells how much the model is 'unsure' of that classification. These probabilities can then be leveraged to collect a bunch of out of train data where the model performs poorly (i.e. low probability scores). These data are then annotated by a human, and used to expand the initial dataset and retrain the model. The whole process is meant to minimize the amount of annotation while maximizing the performance of the model. Some people use active learning also on large datasets, the idea being that many training instances are similar to each other and redundant for the model, so by focusing on training the model in instances that are particularly difficult for the model to capture, the training time can be reduce by a large margin.
Reward modelling is more interesting, and it is relate to reinforcement learning. Unlike in classic deep learning, which focus mainly in the design of loss functions, in reinforcement learning the main component is the reward function. A reward function is much trickier to design compare to a loss function, cause it's usually impossible to predict all possible non useful strategies that an agent might learn (e.g. running in circles to prevent to hit obstacles). So, to compensate for this difficulties, some people came up with the idea of letting artificial agents to design their own reward function, with the only constrain of an external human annotator penalizing dumb functions that emerge during the process. It's a bit hard to explain the idea shortly, but I personally find it really fascinating, and I suggest you also to take a look at this video for a more complete explanation.


text:
What is the reason for a training loss that drops but validation that NEVER does
I've been working on learning about NLP via a beginners competition on Kaggle.
I first trained a model with an embedding layer and then a simple linear layer. I actually got way better than a flip of the coin with this model, so I decided to try to step it up with an LSTM.
What happened was that training loss decreased and then palteaued while validation loss never decreased at all.
In the case of overfitting, I would expect validation loss to decrease for a while but then either remain steady or perhaps even increase as the model starts to overfit.
I can't find any reason for the strange loss curves I'm seeing:

What could cause such a phenomenon?
I would be happy to share my network architecture and training code if there isn't a straightforward answer (I know there usually isn't).

The conclusion that I finally arrived at was that I don't have enough data. The reason I believe this is true is that I slowly lowered model complexity and watched as overfitting decreased and underfitting increased.
If the model was not overfitting, i.e. if the validation loss decreased along with the training loss, then the model underperformed. It was little better than a flip of a coin on binary classification.
Increasing model complexity at all from this point lead to immediate overfitting.
I believe that the solution is to find more data, or use a model that generalizes better with less data.


text:
Why does a transformer not use an activation function following the multi-head attention layer?
I was hoping someone could explain to me why in the transformer model from the "Attention is all you need" paper there is no activation applied after both the multihead attention layer and to the residual connections.  It seems to me that there are multiple linear layers in a row, and I have always been under the impression that you should have an activation between linear layers.
For instance when I look at the different flavors of resnet they always apply some sort of non linearity following a linear layer.  For instance a residual block might look something like...
Input -> Conv -> BN -> Relu -> Conv -> (+ Input) -> BN -> Relu
or in the case of pre-activation...
Input -> BN -> Relu -> Conv -> BN -> Relu -> Conv -> (+ Input)
In all the resnet flavors I have seen, they never allow two linear layers to be connected without a relu in-between.
However in the the transformer...
Input -> Multihead-Attn -> Add/Norm -> Feed Forward(Dense Layer -> Relu -> Dense Layer) -> Add/Norm
In the multihead attention layer it performs the attention mechanism and then applies a fully connected layer to project back to the dimension of its input.  However, there is no non linearity between that and feed forward network (except for maybe the softmax used in part of the attention.)  A model like this would make more sense to me...
Input -> Multihead-Attn -> Add/Norm -> Relu -> Feed Forward(Dense Layer -> Relu -> Dense Layer) -> Add/Norm -> Relu
or something like the pre-activated resnet...
Input -> Relu -> Multihead-Attn -> Add/Norm -> Input2 -> Relu -> Feed Forward(Dense Layer -> Relu -> Dense Layer) -> Add/Norm(Input2)
Can anyone explain why the transformer is the way it is?
I have asked a similar question when I was looking at the architecture of wavenet on another forum but I never really got a clear answer. In that case it did not make sense to me again why there was no activation applied to the residual connections.
(https://www.reddit.com/r/MachineLearning/comments/njbjfb/d_is_there_a_point_to_having_layers_with_just_a/)

FWIW, the final operation of each attention head is a weighted sum of values where the weights are computed as a softmax. Softmax is non-linear, and its shape is sometimes thought of as a multidimensional sigmoid. In some sense, the softmax-output weights serve as a sort of activation function. So maybe it's not bizarre to follow it up with the 1x1 convolutions of the position-wise feed forward layer (with layer normalization and residual connections in between).

This goes back to the purpose of self-attention. While the Vaswani et als Attention is All You Need paper does not explicitly define self-attention, I am using it here as being synonymous to what it refers to as scaled dot-product attention when all three inputs are identical.
Measure between word-vectors is generally computed through cosine-similarity because in the dimensions word tokens exist, it's highly unlikely for two words to be colinear even if they are trained to be closer in value if they are similar. However, two trained tokens will have higher cosine-similarity if they are semantically closer to each other than two completely unrelated words.
This fact is exploited by scaled dot-product attention; After several of these matrix multiplications, the dissimilar words will zero out or become negative due to the dot product between them, and the similar words will stand out in the resulting matrix.
So, self attention can be viewed as a weighted average, where less similar words become averaged out faster (toward the zero vector, on average), thereby achieving groupings of important and unimportant words (i.e. attention). The weighting happens through the dot product. If input vectors were normalized, the weights would be exactly the cosine similarities.
The important thing to take into consideration is that within the self-attention mechanism, there are no parameters; Those linear operations are just there to capture the relationship between the different vectors by using the properties of the vectors used to represent them.
Read this blog post by Peter Bloem for a more in-depth explanation of self-attention.

Edit
I should add that this explanation is less satisfactory considering how Transformers also seem to work for tasks without learned embeddings, like time-series forecasting. I have no idea why that is. However, the model was originally used for NLP, and they did use learned embeddings. So, I bet that's why that particular architecture looks the way it does.
Bloem, in the blog post above, does discuss the mathematical properties of self-attention without bringing up the fact that the original architecture does have learned embeddings.
All this shows is that having learned embeddings does not matter that much; The layers following the multi-headed attention will learn the relationships between the vectors. The general point about the properties of the dot-product being exploited does stand.

It seems like people are applying transformers to things without these embeddings, as you say.  Without embeddings that have a priori meaningful relationships, all the attention stuff does not seem applicable.  Since there is no activation, the same transformation can be done with a single layer if I'm not mistaken.

@Mastiff Well, if you want a more general answer, this is the best alternative I can come up with:
There are a number of papers, starting with Metaformer, that suggest that any "token mixer" may work. As a specific example see FNet which uses Fourier transforms instead.
Per these papers, attention, FFTs and even convolutions and poolings are examples of token mixers. So, my explanation still holds but for a different, possibly deeper, reason.
I never added this since it's still pretty much in early research stage. Do you feel like I should?

there are no parameters  -  When said parameters we mean something that is learnable by machine leaning, right? then the Wq Wk Wv Wo that learned from training should be think of as parameters, maybe?

@Wuaner I have noticed there is an issue of definition that leads to this confusion. There is a learned layer right before the matrix multiplications. If you consider the weights from that layer as a part of attention, then yes, the illustration you are linking to is correct. However, the Attention is All You Need paper does not includue those weights as a part of scaled dot-product attention, but it does include those in multi-head attention (look at figure 2 in that paper). My answer refers to self-attention. Vaswani is not clear how he defines that.

@Wuaner I have edited the answer to make that clear. Understanding dot products and how they have historically been used in NLP is essential to understand attention as a whole. So, my answer focuses on that.

@Avatrin I see, that make sense.


text:
Transformer model is very slow and doesn't predict well
I created my first transformer model, after having worked so far with LSTMs. I created it for multivariate time series predictions - I have 10 different meteorological features (temperature, humidity, windspeed, pollution concentration a.o.) and with them I am trying to predict time sequences (24 consecutive values/hours) of air pollution. So my input has the shape X.shape = (75575, 168, 10) - 75575 time sequences, each sequence contains 168 hourly entries/vectors and each vector contains 10 meteo features. My output has the shape y.shape = (75575, 24) - 75575 sequences each containing 24 consecutive hourly values of the air pollution concentration.
I took as a model an example from the official keras site. It is created for classification problems, I only took out the softmax activation and in the last dense layer I set the number of neurons to 24 and I hoped it would work. It runs and trains, but it does worse predictions than the LSTMs I have used on the same problem and more importantly - it is very slow - 4 min/epoch. Below I attach the model and I would like to know:
I) Have I done something wrong in the model? can the accuracy or speed be improved? Are there maybe some other parts of the code I need to change for it to work on regression, not classification problems?
II) Also, can a transformer at all work on multivariate problems of my kind (10 features input, 1 feature output) or do transformers only work on univariate problems? Tnx
def build_transformer_model(input_shape, head_size, num_heads, ff_dim, num_transformer_blocks, mlp_units, dropout=0, mlp_dropout=0):

    inputs = keras.Input(shape=input_shape)
    x = inputs
    for _ in range(num_transformer_blocks):

        # Normalization and Attention
        x = layers.LayerNormalization(epsilon=1e-6)(x)
        x = layers.MultiHeadAttention(
            key_dim=head_size, num_heads=num_heads, dropout=dropout
        )(x, x)
        x = layers.Dropout(dropout)(x)
        res = x + inputs

        # Feed Forward Part
        x = layers.LayerNormalization(epsilon=1e-6)(res)
        x = layers.Conv1D(filters=ff_dim, kernel_size=1, activation="relu")(x)
        x = layers.Dropout(dropout)(x)
        x = layers.Conv1D(filters=inputs.shape[-1], kernel_size=1)(x)
        x = x + res

    x = layers.GlobalAveragePooling1D(data_format="channels_first")(x)
    for dim in mlp_units:
        x = layers.Dense(dim, activation="relu")(x)
        x = layers.Dropout(mlp_dropout)(x)
    x = layers.Dense(24)(x)
    return keras.Model(inputs, x)

model_tr = build_transformer_model(input_shape=(window_size, X_train.shape[2]), head_size=256, num_heads=4, ff_dim=4, num_transformer_blocks=4, mlp_units=[128], mlp_dropout=0.4, dropout=0.25)
model_tr.compile(loss="mse",optimizer='adam')
m_tr_history = model_tr.fit(x=X_train, y=y_train, validation_split=0.25, batch_size=64, epochs=10, callbacks=[modelsave_cb])

I know some good developer, i will ping him. I think he might help you with this.

This needs a bit of information regarding your output. Which values can they take? Softmax returns value between zero and one, and are thus only used for classification tasks.. Although, I guess they could work for regression tasks that has values between zero and one as well.

@Avatrin What I wrote above is that I have REMOVED the softmax activation, because as you say it is used for classification, while I want to do regression. Hence, I have changed softmax for relu and in the very last dense layer I use no activation at all. My Input is scaled by subtracting the mean and dividing by the standard deviation. What would you recommend me to do/change? Do I have some principal error in the architecture? Have I missed something? Because the model trains, but predictions worse compared to LSTM

Edit
I just noticed that the model you are referring to is built very differently than the transformer from Attention is All You Need since it only uses one half of the architecture. Thus my answer below is not be complete. I thus have to add the following: (The final two paragraphs still apply as they are, though)
The Keras model is quite weird, and while it is a time-series model, it's not a regression one. Thus my general description regarding how transformers work still apply (that is, you only get one prediction at a time), since their model is a many-to-one classification model. So, you have adjust your model accordingly.
The main thing you can ignore from my original answer is my emphasis on decoders; Keras only use encoders for their model.

Original answer
Transformers only output one prediction at a time, not twenty-four. Lets break a transformer during runtime step by step.
In all of the steps, the encoder input will be the same since your sequences are not that long. The decoder gets the output as its input, which is done in the following manner:
Step one:
The decoders input is only one token which stands for "start of sequence" and padding. $(start,0,0,..,0,0)$
Step two:
The decoder input is the start-token and the prediction from step one + padding $(start,y_1,0,0,..,0,0)$
Step three:
The decoder input is the decoders input in step two and the prediction from step two + padding $(start,y_1,y_2,0,0,..,0,0)$
And, so on.. If you want 24 outputs, the transformer will have to run 24 times. The only reason the original transformer model has multiple outputs is that it needs to make one prediction for each word token (and, pick the most likely one). Training will thus have to reflect this.
However, I am quite skeptical that you will have any significant benefit from using transformers over LSTMs. The main benefit of transformers, or any other attention based model, is that they allow for longer-term dependencies than LSTMs. According to a Stanford study, the point at which LSTMs are no longer efficient is a sequence of around 200 tokens. Of course, that study is specific to language models, but I bet it won't be that much different in your case; Your sequences are quite short.
Anyhow, the output layer for a regression task generally use linear activation functions. If all your outputs are positive, that won't make much of a difference.

Thanks for the long answer! So you say that transformers are not sequence2sequence models? This surprises me. They are used for translation problems ("This is a potato" -> "Das ist eine Kartoffel") which I thought are sequence2sequence problems ...? Can you elaborate on that? Thanks for the hints about LSTMs, too. Is there some other architecture, that you can suggest me to test for my air pollution project?

@NeStack To make Transformer's seq2seq, you need the decoder. Even then, they are auto-regressive and only output one token per time-step. The decoder gets the output as its input while the encoder processes the input. So, in my answer, when I discuss the decoder, I am writing about how transformers deal with the output.

@NeStack Regarding alternatives, there's a relatively recent review of Deep Learning Architectures for Time Series Forecasting by Lara-Benítez et al. They emphasize LSTMs and CNNs, although they discuss several other alternatives.


text:
How to divide a segmented image into classes instances?
Is there a method/algorithm to generate instances of objects from image that was segmented by the use of any image segmentation models?
For example, I have an image with one class and it was segmented in a given way, where 1s are objects of the same class and empty fields are of no class:
How can I now generate list of the two objects, where list's elements for example would be positions of all the pixels inside the object (list of list).

Those segmented instances can be seen as connected components of the 1s. You can use the flood fill or BFS algorithm to detect these components or try SimpleBlobDetector by OpenCV.

"flood fill" finally worked out for me, there is a nice implementation in scikit-image library , although I had to make a loop to go through all objects, because basic implementation can only fill one "blob" of 1s at a time


text:
PPO when does the update happen?
In many places, it says PPO and Actor-Critic methods in general use TD-updates, but in the loss function for PPO, the Value function loss component uses the difference between output of the value function and the value target, which I can only assume is the discounted sum of rewards that can only be obtained at the END of the episode?
So this might be a moment of stupidity for me, but

Is the value target in PPO set only at the end of the episode using the discounted sum of rewards? or is there a secret way of setting these value targets that I am missing?

If a learning update indeed takes place every learning step (before the end of the episode), then how does this TD-learning happen - does it use some other approximate of the value target?

Thank you.
Please help.
Sincerely,
a frustrated student

So the loss function that you want to optimize to improve your policy is given by:
$$ L^{CLIP} = \mathbb{E} \Big[ \frac{\pi_\theta (a|s)}{\pi_{\theta_{old}}(a|s)} \Big( R(s,a) - V_\phi(s) \Big) \Big]. $$
And to improve the value network you optimize a squared-error loss between the output of your value network and the return:
$$ L^{VF} = 0.5\Big(V_\phi(s) - R(s,a)\Big)^2.$$
And really the question is: How do you compute the return $R(s,a)$?
If you use a Monte-Carlo estimate of the return, i.e.
$$ R(s_t,a_t) = \sum_{i=t}^{T} r_{i+1},$$
well, then you have to rollout the episode until a terminal state is reached. Only then you can calculate the returns for each state that was visited during the rollout, and update the policy and the value networks.
If, instead, you use a one-step bootstrapped estimate (also called TD), i.e.
$$ R(s_t,a_t) = r_{t+1} + V_\phi(s_{t+1}),$$
then you can actually update the networks after every single step.
What is usually done in practice is to use an n-step bootstrapped estimate:
$$ R(s_t, a_t) = r_{t+1} + r_{t+2} + \cdots + r_{t+n} + V(s_{t+n}).$$
Now you rollout $n$ steps and then you update.
Even better than the n-step bootstrapped estimate would be to use the Generalized Advantage Estimate (GAE).
If you want to read more about all of these please check out:
https://pi-tau.github.io/posts/actor-critic/


text:
Delayed state observation or caching action in OpenAI gym. Can it still learn?
I am planning to use OpenAI gym for my experiment in real life.
In my experiment design, by the limits of a real-life scenario, I can only receive the state information or the rewards about 2-3 timesteps behind when the action has happened (in OpenAI gym term, ~3 cycles of step(action) function has occurred). For example, by the time the state at timestep i is observed, an action at timestep i+3 would have happened.
From how I perceive the function, step(action), is that it needs to return next_state, reward, done every step. And the agent will learn from state -> action -> next state -> reward tuple. So I was wondering if can I cache the action for future use along with the state with the correct time step in OpenAI gym? or delay the state observation/reward instead? Could the OpenAI be able to learn?
I am experimenting with PPO TD3 SAC which all uses actor-critic networks. Would the network eventually be trained well enough to the point where it would still perform well with the delayed state observation?

To turn this delayed action approach into a normal, and theoretically valid MDP, add a "pending action resolution" array to the state representation, and ensure that the state transition (or step code) manages this array with each new action pushed into it, and the array shifting down as it goes. The array length should be minimum of the number of timesteps required to fully resolve a previous action.
This allows your agent to see the actions it has recently performed and account for them in calculations without any adjustment needed to the core MDP formulation or standard agent designs.
If delays are stochastic or vary (e.g. a scenario where a warehouse is ordering stock), then add more data to this pending action resolutions array as necessary, such as expected resolution time, or time since action was taken.

From what I understand in implementation of your 'pending action resolution' array is:  First-In the step(action), take an action in real life. Second- Buffer/queue the action into the array with length of the delay step. Third - repeat fist&second until the action resolution array is completely filled. Last - With the full array, measure the state + reward associated with the oldest/last index of the pending action resolution array. Is this correct? If so, what do I return at the end of the step() function in the Second if I do not have the next state observed and the reward?

@Ykwk: That's overcomplicated. The array of recent actions becomes just a normal part of the state. Initialise it with "null" actions so it has a fixed length. When the agent takes an action, push the id of that action onto the array, then shift the array to remove one old action (so the array stays the same length, and actions are stored in age order). Run all the rest of MDP and time step handling as normal for the agent - i.e. process value estimates, use replay memory for training etc on every time step.

How can the 'initialization null' array of action be implemented, when every iteration of step() requires to return -next state -reward? Passing in 'null' action does not give me any return values for step() function. I don't think I can get the next action, without having the agent to go to next state reward.

@Ykwk These are not actions that are taken in any sense, they are placeholders for "no action" in the state representation so that the feature size is consistent, so that you can use a neural network. Another term might be "padding". Implement them as part of the starting state representation.

Can you share minimal working example code @Ykwk