text:
How to design a graph neural network to predict the forces in truss elements of a space frame?
I am trying to create a Graph NN that will be able to predict the forces in truss elements of a space frame.
The input for the NN will be a graph, where the nodes represent the nodes of the spaceframe. And the output should be the forces in the edges of the supplied graph/frame.
The problem that I am facing is that for the NN to be beneficial I need to encode a lot of data per node:

3 floats for the position of the node,
3 floats for the vector of the force applied to the node,
a boolean/int to determine whether a node is a support.

I am not sure how to design my Graph NN to allow for so many parameters per node.
Maybe I should try a different approach?
Any help is greatly appreciated!

If I remember correctly, you can have a feature vector (or arbitrary size) for each node. Why do you think that 7 features per node are too many?


text:
How to deal with the addition of a new state to the environment during training?
Let's say we have a dynamic environment: a new state gets added after 2000 episodes have been done. So, we leave room for exploration, so that it can discover the new state.
When it gets to that new state, it has no idea of the Q values, and, since we're past 2000 episodes, our exploration rate is very low. What happens if try to exploit when all Q values are 0?

There are several ways to tackle this, although exploration is definitely not a solved problem yet ;)
In general, I believe the right thing to do here is to measure the uncertainty of your policy or Q-value estimates and use that to construct some sort of exploration bonus. An intuitive example is given in Exploration by Random Network Distillation. They make two randomly initialized neural nets, one of which is never updated. At every transition, they feed transition data through both neural nets and use the difference in output between them as an estimate of uncertainty, and this quantity is added to the reward. Then they update the modifiable neural net towards the other one. This way, on a completely novel transition, the two neural nets will likely have very different outputs so the reward will be augmented a lot. Of course, this will hopefully encourage the agent to explore.

Can you break this down please , it’s quite daunting

You have one fixed NN and another NN, both initialized randomly. Let's say these networks take states as inputs and outputs scalars. On a new state, both networks will output random scalars x and y that are likely different. Add |x-y| to the reward. Train the first network to minimize (x-y)^2. Once you've seen a state many times, the first network and the fixed network should match, so |x-y| ~= 0, so there's very little bonus.

Ohh this is deep Q learning, using target networks and our normal Q network

Where you have a fixed target network which improves stability for our gradient descent and when it finall converges, we train our fixed networks with the updated weights of the Q network, am I right

No, the two networks I'm talking about are not Q networks. I recommend you read the paper that I linked.

Chuckwudi, the NN are used as a distance function between states.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Does self-supervised learning require auxiliary tasks?
Self-supervised learning algorithms provide labels automatically. But, it is not clear what else is required for an algorithm to fall under the category "self-supervised":
Some say, self-supervised learning algorithms learn on a set of auxiliary tasks [1], also named pretext task [2, 3], instead of the task we are interested in. Further examples are word2vec or autoencoders [4] or word2vec [5]. Here it is sometimes mentioned that the goal is to "expose the inner structure of the data".
Others do not mention that, implying that some algorithms can be called to be "self-supervised learning algorithms" if they are directly learning the task we are interested in [6, 7].
Is the "auxiliary tasks" a requirement for a training setup to be called "self-supervised learning" or is it just optional?

Research articles mentioning the auxiliary / pretext task:

Revisiting Self-Supervised Visual Representation Learning, 2019, mentioned by [3]:

The self-supervised learning framework requires only unlabeled data in order to formulate a pretext learning task such as predicting context or image rotation, for which a target objective can be computed without supervision.

Unsupervised Representation Learning by Predicting Image Rotations,  ICLR, 2018, mentioned by
[2]:

a prominent paradigm is the so-called self-supervised learning that defines an annotation free pretext task, using only the visual information present on the images or videos, in order to provide a surrogatesupervision signal for feature learning.

Unsupervised Visual Representation Learning by Context Prediction, 2016, mentioned by
[2]:

This converts an apparently unsupervised problem (finding a good similarity metric between words) intoa “self-supervised” one:  learning a function from a givenword to the words surrounding it.  Here the context predic-tion  task  is  just  a  “pretext”  to  force  the  model  to  learn  agood word embedding, which, in turn, has been shown tobe useful in a number of real tasks, such as semantic wordsimilarity.

Scaling and Benchmarking Self-Supervised Visual Representation Learning, 2019:

In discriminative self-supervised learning, which is the main focus of this work, a model is trained on an auxiliary or ‘pretext’ task for which ground-truth is available for free. In most cases, the pretext task involves predicting some hidden portion of the data (for example, predicting color for gray-scale images

Comments are not for extended discussion; this conversation has been moved to chat.

Most (if not all) self-supervised learning techniques for (visual or textual) representation learning use pre-text tasks, and many pre-text tasks have been proposed in recent years.
However, as I say in my other answer (which you cite), the term SSL has also been used (at least, in robotics: for example, see this paper, which I am very familiar with) to refer to techniques that automatically (although approximately) label the unlabelled dataset for your downstream task (i.e. image recognition), i.e. they automatically create a labeled dataset of pairs $(x_i, \hat{y}_i)$, where $x_i$ is an image that contains an object and $\hat{y}_i$ is the automatically (and possibly approximately) generated label (such as "dog"). This latter use of the term SSL is closer to some weakly supervised learning (WSL) techniques. Actually, it can be considered a WSL technique.
Now, in this specific paper, they actually solve some kind of pre-text task, i.e. they exploit the relations between two different sensors to produce the labels.
To answer your question more directly: in all SSL papers that I have come across, some kind of pre-text task is always solved, i.e., in some way, you need to automatically generate the supervisory signal, and that task that we solve with the automatically generated learning signal (with the purpose of learning representations or generating a labeled dataset) can be considered the pre-text task (which may coincide with the downstream task, for example, in the case you're training an auto-encoder with an unlabelled dataset for the purposes of image denoising).
In any case, I wouldn't bother too much about it. Just keep your context in mind when reading your paper. If you're really worried about it, then you should probably read almost all SSL-related papers, but, in that case, by the end of that, you will be an expert on the topic and you will not need our help (or my help).

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Why do bootstrapping methods produce nonstationary targets more than non-bootstrapping methods?
The following quote is taken from the beginning of the chapter on "Approximate Solution Methods" (p. 198) in "Reinforcement Learning" by Sutton & Barto (2018):

reinforcement learning generally requires function approximation methods able to handle nonstationary target functions (target functions that change over time). In control methods based on GPI (generalized policy iteration) we often seek to learn $q_\pi$ while $\pi$ changes. Even if the policy [pi] remains the same, the target values of training examples are nonstationary if they are generated by bootstrapping methods (DP and TD learning).

Could someone explain why the same is not the case if we use non-bootstrapping methods (such as Monte Carlo that is not allowed infinite rollouts)?


text:
How is trajectory sampling different than normal (importance) sampling in reinforcement learning?
I am using Sutton and Barto's book for Reinforcement Learning.
In Chapter 8, I am having difficulty in understanding the Trajectory Sampling.
I have read the particular section on trajectory sampling (Sec 8.6) two times (plus 3rd time partially) but still, I do not get how it is different from the normal sampling update, and what are its benefits.

Here is my understanding:
In trajectory sampling as the book describes it, we use the current policy on the simulator to get (next-state, action) pairs. The advantage is that if some states occur more frequently than others in that environment, and if we take enough samples, the distribution among the samples would be similar to the actual distribution.
On the another hand, you can sample in a different manner if you have access to the model. Suppose you have access to the transition distribution. Then you can sample your start state uniformly, and use the transition distribution to get the (next-state, action) pairs. This can be useful if you want to force your algorithm to look at all states evenly, even if that's not the actual case.


text:
Should I build an environment from scratch myself or it is not always needed?
I am inspired by the paper Neural Architecture Search with Reinforcement Learning to use reinforcement learning for optimizing a child network (learner). My meta-learner (controller or parent network) is an MLP and will take as the reward function a silhouette score. Its output is a vector of real numbers between 0 and 1. These values are k different possibilities for the number of clusters (the goal is to cluster the result of the child network which is an auto-encoder, embedded images are the input to the meta-learner).
What I am confused about is the environment here and how to implement this network. I was reading this tutorial and the author has used gym library to set the environment.
Should I build an environment from scratch myself or it is not always needed?
I appreciate any help or hints or links to a source that helps me understand better RL concepts. I am new to it and easily gets confused.

I guess it would always be better if you can reuse existing environments to make it work for yourself. Since most of the environment codes is anyway opensourced, you can always edit it to your liking.
If you want a custom environment, you can add an environment to gym like this.

Gym environments were designed first of all as a benchmark for RL algorithms. Therefore, you should ask yourself: Do I want to benchmark my RL algorithm or do I want to solve my problem with RL?
If the answer is "I want to benchmark my algorithm", you should probably go with existing environments. Using existing environments as benchmarks you will be able to compare your algorithm with other existing approaches.
If the answer is "I want to solve my problem", then yes, perhaps the best way to continue is to build your custom environment.
Finally, if you want to understand RL better, I would suggest to start with existing environments to gain a better grasp of RL concepts.


text:
Can fully connected layers be used for feature detection?
I need help in understanding something basic.
In this video, Andrew Ng says, essentially, that convolutional layers are better than fully connected (FC) layers because they use fewer parameters.
But I'm having trouble seeing when FC layers would/could ever be used for what convolutional layers are used for, specifically, feature detection.
I always read that FC layers are used in the final, classification stages of a CNN, but could they ever be used for the feature detection part?
In other words, is it even possible for a "feature" to be deciphered when the filter size is the same as the entire image?
If not, it's hard for me to understand Andrew Ng's comparison---there aren't any parameter reduction "savings" if we're not going to use an FC "filter" in place of a CNN layer in the first place.
A semi-related question: Can multi-layer perceptrons (which I understand to be fully connected neural networks) be used for feature detection? If so, how do image-sized "filters" make any sense?

First of all, (FC) Neural Networks (NN) are universal function approximators. This means, that, in theory, there must exist some NN of appropriate size that is capable of doing what a CNN can do as well. The only difference is that the standard NN would have to be considerably larger than a corresponding CNN with the same capabilities. Each input pixel would probably have to map to a few dozens to hundreds of input nodes that are trained each on detecting some part of a distinct feature in their respective input image location. Consecutive, hidden layers than have to combine the parts extracted by the input layer for any given location and finally act as the true feature detectors, detecting features as compositions of values in the first layer.
So, for each position in the input image separately, all the filters have to be learned separately. This absence of weight sharing (since filters are not shared across input locations, but have to be trained for each separate input location) makes using a standard NN for feature extraction so inefficient.
However, in theory, also a standard NN could be trained on solving an image classification task, thereby learning to extract features. Actually, I would not be surprised if someone had tested that already on the MNIST dataset.

Note that, if you trained an MLP (or FFNN) to solve some task involving images (e.g. classification of MNIST digits), the MLP wouldn't probably attempt to mimic a CNN, as one may think while reading your answer. However, I understand that you're just trying to show how an MLP would mimic a CNN, if it needed. Regarding your last paragraph, if I remember correctly, in the past, I trained a small FFNN on MNIST and I got decent results, although typically worse than CNN's results (maybe you saw the comment that I deleted), but it's important to note that MNIST is a relatively easy task.

The ease of learning the MNIST dataset is exactly why I brought up this example. Just because it is one of the few datasets where one can probably observe that it is possible, indeed, to employ a FCNN for image classification.
And, of course, in practice gradient descent will not nicely teach the FCNN to extract crisp features for every isolated position in the input image. Just because that would very likely be too inefficient and there would be no clear-cut distinction between nodes learning to extract parts of features and those that serve as ultimate feature detectors.

A core idea of CNN is convolution and max pooling. Max pooling essentially attempts to have certain features be dominate and the maxpooling extracts this feature. This provides translation invariance for objects.


text:
What is the proof that the variance of the gradient estimate in Actor-Critic is smaller than in REINFORCE?
The intuition provided when introducing actor-critic algorithms is that the variance of its gradient estimates is smaller than in REINFORCE as, e.g., discussed here. This intuition makes sense for the reasons outlined in the linked lecture.
Is there a paper/lecture providing a formal proof of that claim for any type of actor-critic algorithm (e.g. the Q Actor-Critic)?

I don't think this is something that can be formally proved, at least not in general. It can be measured in a range of environments, or could be shown more robustly in a specific constructed environment.


text:
Does importance sampling for off-policy estimation also apply to the case of negative rewards?
Importance sampling is a common method for calculating off-policy estimates in RL. I have been reading through some of the original documentation (D.G. Horvitz and D.J. Thompson, Powell, M.J. and Swann, J) and cannot find any restrictions on the reward or value being estimated. However, it seems that there are constraints because the calculation is not what I would expect for RL environments that have negative rewards.
For example, consider for a given action-state pair ($a_i, s_i$), $\pi_e(a|s) = 0.4$ and $\pi_b(a|s) = 0.6,$ where $\pi_b$ and $\pi_e$ are the behavioral and evaluation policies respectively. Also, assume the reward range is $[-1,0]$, and this action has a reward of $r_{\pi_b}=-0.5$.
Under the IS definition, the expected reward under $\pi_b$ would be $r_{\pi_e} = \frac{\pi_b(a|s)}{\pi_e(a|s)} r_{\pi_b}$. In this example, $r_{\pi_e}=-0.75$ thus $r_{\pi_e} < r_{\pi_b}$. However, assuming a change of scale of the reward to be $[0,1]$ which result in $r_{\pi_b}=0.5$, results in $r_{\pi_e} > r_{\pi_b}$.
All examples of IS I have seen in reference focus on positive rewards. However, I find myself wondering if this formulation applies to negative rewards too. If this formulation does allow for negative reward structures, I'm not sure how to interpret this result. I'm wondering how changing the scale of the reward could change the order? Is there any documentation on the requirements of the value in IS? Any insight into this would be greatly appreciated!

When you say "original documentation (D.G. Horvitz and D.J. Thompson, Powell, M.J. and Swann, J)", are you referring to a paper? If yes, can you provide the link? One of the papers you are referring to may be A Generalization of Sampling Without Replacement from a Finite Universe. Is this correct? What about the other?


text:
Is it possible to create a named entity recognition system without using POS tagging in the corpus?
Is it possible to create a named entity recognition system without using POS tagging in the corpus?

Yes, both of them are different tasks. POS tagging helps NER systems but it is not necessary. You can get features (say BERT/ELMo embedding) for each word in the sentence and train a CRF NER model. This looks like simple example https://www.pragnakalp.com/bert-named-entity-recognition-ner-tutorial-demo/


text:
Why is it the case that off-policy evaluation using importance sampling suffers from high variance?
The average return for trajectories, $V^{\pi_e}$(s) is often computed via the importance sampling estimate $$V^{\pi_e}(s) = \frac{1}{n}\sum_{i=1}^n\prod_{t=0}^{H}\frac{\pi_e(a_t | s_t)}{\pi_b(a_t|s_t)}G_i$$ where $G_i$is the reward observed for the $i$th trajectory. Sutton and Barton gives an example whereby the variance could be infinite.
In general, however, why does this estimator suffer from high variance? Is it because $\pi_e(a_t|s_t)$ is mainly deterministic and, therefore, the importance weight is $0$ for most trajectories, rendering those sample trajectories useless?


text:
How does backpropagation work in LSTMs?
After reading a lot of articles (for instance, this one Understanding LSTM Networks), I know that the long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used in the field of deep learning.
How does backpropagation work in the specific case of LSTMs?


text:
Why is it not advisable to have a 100 percent exploration rate?
During the learning phase, why don't we have a 100% exploration rate, to allow our agent to fully explore our environment and update the Q values, then during testing we bring in exploitation? Does that make more sense than decaying the exploration rate?

No - imagine if you were playing an Atari game and took completely random actions. Your games would not last very long and you would never get to experience all of the state space because the game would end too soon. This is why you need to combine exploration and exploitation to fully explore the state space.

But you get my point, even if the game ends quickly, suppose you have a lot of episodes, eventually all space states would be explored , then when we want to test it , we bring in exploitation

And also after full training, why is it not advisable to turn off exploration, why do we still leave room for small exploration, my guess is in the case of a dynamic environment where new states might appear

@Chukwudi no, if you took random actions in an Atari game you would never win the game in a feasible amount of episodes. Maybe theoretically in the limit you would, but remember that we have limited compute power, and thus you need to exploit the good actions so that we can keep learning from the good actions by backing up the Q function for these state-action values. However, exploration must be maintained to some degree to ensure that we aren't missing any better actions.

What if we’re sure we’ve covered every state and the Q values are very accurate, assuming our space state is small, the what would be the use of still exploring

In terms of making Q-learning scale well, relatively, in environments with a large state space, there must be exploration and exploitation since the aim of Q-learning is to find an optimum policy to traverse an optimum path from start to goal in an MDP. So exploration helps to check if there is a better action to take given a state and set of actions and then leading to a better policy and exploitation allows the agent to find it in a feasible amount of time. Note some environments can be very huge e.g. (256 ^ 128) states.

@Chukwudi: If you are sure that you are done, then you are done and there is no need to continue learning process. But that is not what you asked in the question

Decaying exploration as training proceeds is done because as the policy gets better there is not much need to explore since the agent might have already found the actions that give the highest reward, given a state.

Decaying the exploration rate is not the only exploration-exploitation strategy, there are others like mellow-max.

The policy you’re improving is your absolute greedy policy right

epsilon-greedy policy

No epsilon greedy is the behaviour policy, not the update policy

While theoretically you can do something like this if you're very confident you'll cover most of the state space in exploration, this is still a suboptimal strategy. Even in the case of multi-armed bandits, this strategy can be much less sample efficient than $\epsilon$-greedy, and exploration is much easier in this case.
So, even if your strategy miraculously works on a decently sized MDP, it'll be worse than combining exploration and exploitation.

Why do you say it’ll be worse, the whole point of me using just exploration is to cover the whole states, and improve my policy

Because if you're doing 100% exploration, you'll be revisiting states very often as well. Furthermore of your state space is large, covering the whole state space is intractable. In theory, as long as you explore forever (like in e-greedy for instance), you can still cover the whole state space, but you'll be exploiting way more and incurring less regret.

But I need a concrete answer as to why we need to combine exploration and exploitation, because to me, exploration 100% would cover the entire space state After like 1000 episodes, them we can start exploiting

Unless your state space is very small, you won't cover 100% of the state space with full exploration. Especially if the environment is difficult to navigate, without any exploitation your agent will probably just keep failing immediately (think CartPole for instance). For a concrete answer, like I said, look into the Multi-Armed Bandit literature (even Sutton and Barto). You can prove that your strategy does not achieve optimal regret.

Someone used a robot balancing on a rope, if I use 100% exploration, then the random actions will make the episodes shorter, and I will never be able to explore the environment in those number of episodes

Right, that's a good example


text:
How can I read any AI paper?
I have studied linear algebra, probability, and calculus twice. But I don't understand how can I reach the level that I can read any AI paper and understand mathematical notation in it.
What is your strategy when you see the mathematical expression that you can't understand?
For example, in Wasserstein GAN article, there are many advanced mathematical notations. Also, some papers are written by people who have a master's in mathematics, and those people use advanced mathematics in some papers, but I have a CS background.
When you come across this kind of problem, what do you do?

You say that you have a CS background. Does it mean that you have obtained a bachelor's or a master's in CS? Can you clarify this? Also, when you say that you studied linear algebra, probability, and calculus, how exactly did you study those? For example, do you know what the central limit theorem says? Do you know what random variables are? If you don't know the answers to these questions, you probably didn't study enough.

This is not exactly an answer to your question but I'd suggest looking for blog posts. There are lots of good blogs that simplify the math behind those complex looking papers and conveys the core idea in a much more understandable way.  For example: https://gradientscience.org/, https://distill.pub/ etc.

I think the answer depends very much on why you are reading the paper, what are you trying to get out of it? There are plenty of papers that I "read" (or often really just quickly skim through) where I'll definitely not understand all the math. More often than not, this will be because I don't actually care to deeply understand it.
There is plenty of more "practical" research to be done in AI, which definitely doesn't always require a deep understanding of all the math. Intuition can often be enough, at least to get started, for meaningful practical contributions. If this is the sort of research that you're interested in doing, you probably don't need to understand as many of the mathematical parts of AI papers as you do if you're really trying to do research directly in that theoretical area.
Personally, when I write "math-heavy" parts in my own papers (and that will often already be restricted to a rather simple level of math in comparison to the "real theory" ML papers), I always try to make sure to include intuitive, English descriptions of what we're doing around it. Even if you don't immediately understand a full equation, just having the intuitive explanation around it to tell you what it means can be enough for a broad understanding of the paper. Then you only have to dive deep into the details of the equations if -- based on the English text -- you decide that you're actually really interested. So, if there are sufficient, intuitive explanations surrounding the equations, I'd recommend to focus heavily on that first. Not every paper does this though, sometimes there's very little text and very much math, and then this can be difficult.
Even if it turns out that you do have to understand math, you may not have to understand ALL of it right away though. The important parts that I would try to focus on understanding first are:

A mathematical description of the "problem". This could be an objective function, a metric to be optimised/minimised/maximised, or an existing equation from previous literature that the authors take as a starting point and inspect some detail of in greater detail.
Mathematical descriptions of the outcomes/results. These could be equations that they actually use in concrete algorithms (see if you can relate them to any pseudocode that may be present), or the final equations stated in theorems / at the end of proofs.

All the complex parts in between are probably less important. Just a vague idea of what the starting point is, and a vague understanding of the final outcome, can be enough to at least know what the paper is about. Then you can decide for yourself whether you really need to know more about the details in between, or if they're maybe not relevant to you / your work / your research.

I think the best way to make reading papers easier is to practice (as in, read lots of papers, try implementing them, etc), and to discuss them with other students/researchers.
Sometimes it's tough to avoid some obscure or really technical math, so you may just need to do extra reading. The Wasserstein metric, for example, is used a lot in ML but I kinda doubt most ML researchers have a good understanding of it. This metric comes from a branch of math called "optimal transportation theory", which is super interesting, but very real-analysis-heavy. If you're really interested in learning about the Wasserstein metric, I recommend Cedric Villani's book "Optimal Transport: Old and New". I also recommend this awesome paper. Nevertheless, learning analysis is likely gonna serve you very well for understanding a wide range of ML papers.
Finally, as a beginning grad student, I have experienced your issue as well. I made a tool to help me with this at this repo, which manages a library of papers you're interested in. It then uses a PageRank algorithm to recommend new papers to you that are commonly referred to by the papers you want to read, with the goal of helping you read up on the foundational "prerequisite" material.

Thanks for your response, Wasserstein is a only example, but when I come across advanced math concept that has many prerequisites in math like algebraic topology, would you suggest that I follow prereqs then learn the topic? Doesn't it take a long time?

It can of course take a long time, but (hopefully) these things will be common across much of the papers you read. If not, you might not need such an in-depth understanding of those concepts, you can probably get away with getting an overview of the field just so you can learn something from the ML paper.

When I read papers in a new domain and when I started reading theoretical ML paper, I faced similar problems. I usually start with the introduction then related work and try to understand all the concepts and related papers cited that are relevant to understanding the paper.
Specifically when it comes to difficult mathematical formulations, as @harwiltz said the more you read about it the easier it gets. There may be a set of papers with concepts that are similar to the paper you are reading but are well-explained I usually read them first (or if it is an important mathematical concept you can find some blogs describing the intuitions/basics behind it).

From my experience (and I've been reading many research papers for a while), it's rare to find a research paper where you fully understand everything in one go, especially if the research paper was published or released recently or a very long time ago (because, back then, maybe people had a different writing style, used a different notation, or something like that), unless you are an expert on the topic, which is probably not the case, unless you are doing serious research on the topic (i.e. you're doing a Ph.D. and beyond; in that case, you probably don't need to ask questions on this site: hopefully, you have a qualified advisor to whom you can ask these questions!), or the paper is really easy and does not contain any formulas.
Of course, if a paper is published, it must contain something novel, so that something novel could be one of the things that you need to spend some time to understand, but the hardest parts of a paper could also easily be the prerequisites (i.e. the concepts that the paper builds upon), because you may not have a very solid knowledge of those topics (as you probably have already experienced).
There are at least three ways to proceed when you are stuck because you don't understand something

If you can ignore what you don't understand (i.e. you don't need it for your purposes because e.g. you just need to have a high-level understanding of the topics), ignore it (really!!)
If it cannot be ignored (e.g. because you really need to know all the details of the paper because e.g. you need to give a presentation at your university), try to understand what you don't understand by picking up a resource on that topic that you don't understand, then read it; spend the time that you think is opportune (i.e. do not spend 6.5 days to understand a detail of a paper if you only have 7 days to read that paper and prepare a presentation or whatever you need to do)
If you can afford it, stop reading that paper and go back to the basics.

In general, learning is not an easy process and, more specifically, reading research papers is not the easiest reading (because research papers are typically concise, i.e. there's a lot of information compression), so do not expect to understand everything of a paper in one go. In fact, the paper How to Read a Paper by S. Keshav, which gives you some guidelines on how to read a paper, tells you to read a paper in three steps. For more details about these three steps, please, read the paper!


text:
How do I calculate the return given the discount factor and a sequence of rewards?
I know that $G_t = R_{t+1} + G_{t+1}$.
Suppose $\gamma = 0.9$ and the reward sequence is $R_1 = 2$ followed by an infinite sequence of $7$s. What is the value of $G_0$?
As it's infinite, how can we deduce the value of $G_0$? I don't see the solution. It's just $G_0 = 5 + 0.9*G_1$. And we don't know $G_1$ value, and we don't know $R_2, R_3, R_4, ...$

You know all the rewards. They're 5, 7, 7, 7, and 7s forever. The problem now boils down to essentially a geometric series computation.
$$
G_0 = R_0 + \gamma G_1
$$
$$
G_0 = 5 + \gamma\sum_{k=0}^\infty 7\gamma^k
$$
$$
G_0 = 5 + 7\gamma\sum_{k=0}^\infty\gamma^k
$$
$$
G_0 = 5 + \frac{7\gamma}{1-\gamma} = \frac{5 + 2\gamma}{1-\gamma}
$$

There are a few ways to resolve values of infinite sums. In this case, we can use a simple technique of self-reference to create a solvable equation.
I will show how to do it for the generic case here of an MDP with same reward $r$ on each timestep:
$$G_t = \sum_{k=0}^{\infty} \gamma^k r$$
We can "pop off" the first item:
$$G_t = r + \sum_{k=1}^{\infty} \gamma^k r$$
Then we can note that the second term is just $\gamma$ times the orginal term:
$$G_t = r + \gamma G_t$$
(There are situations where this won't work, such as when $\gamma \ge 1$ - essentially we are taking advantage that the high order terms are arbitrarily close to zero, so can be ignored)
Re-arrange it again:
$$G_t = \frac{r}{1 - \gamma}$$
This means that you can get the value for a return for the discounted sum of repeating rewards. Which allows you to calculate your $G_1$. I will leave that last part as an execrise for you, as you already figured the first part out.


text:
How are the coefficients of the Region of Interest being selected?
I was reading the following paper: Rl-Ncs: Reinforcement Learning Based Data-Driven Approach For Nonuniform Compressed Sensing, and my question is: how do they decide whether a signal is characterized as a region of interest coefficient or non-region of interest coefficient?


text:
Do GANs also learn to map between the distribution from which the random noise is sampled and the true distribution of the data?
I am reading about GANs. I understand that GANs learn implicitly the probability distribution that generated the data. However, at the input we give a random noise vector. It seems that we can sample that random noise vector from whatever distribution we want.
My question is: given that there is ONLY ONE possible distribution that could have generated the data, and that our GAN is trying to approximate that distribution, can I think that the GAN also learns how to map between that distribution that it needs to learn and the distribution from which we sample the random noise vector?
I am thinking about this, as the random noise vector can be sampled from whatever distribution we want, so it can be different each time we start to train, so it can vary, but the GAN needs to be able to still imitate one unique distribution, so, in a way, it needs to be able to adapt to the distribution from which the noise comes.


text:
Why does shifting all the rewards have a different impact on the performance of the agent?
I am new to reinforcement learning. For my application, I have found out that if my reward function contains some negative and positive values, my model does not give the optimal solution, but the solution is not bad as it still gives positive reward at the end.
However, if I just shift all readings by subtracting a constant until my reward function is all negative, my model can reach the optimal solution easily.
Why is this happening?
I am using DQN for my application.
I feel that this is also the reason why the gym environment mountaincar-v0 uses $-1$ for each time step and $0.5$ at the goal, but correct me if I am wrong.

You have some freedom to re-define reward schemes, whilst still describing the same goals for an agent. How this works depends to some degree on whether you are dealing with an episodic or continuing problem.
Episodic problems
An episodic problem ends, and once an agent reaches the terminal state, it is guaranteed zero rewards from that point on. The optimal behaviour can therefore depend quite critically on balance between positive and negative rewards.

If an environment contains many unavoidable negative rewards, and these outweigh total positive rewards, then the agent will be motivated to complete an episode sooner.

If an environment contains repeatable positive rewards, and these outweigh total negative rewards, then the agent will be motivated to loop through the postive rewards and not end the episode.

Scaling all rewards by the same positive factor makes no difference to the goals of an agent in an episodic problem. Adding a positive or negative offset to all rewards can make a difference though. It is likely to be most notable when such a change moves rewards from positive to negative or vice versa. In the MountainCar example, adding +2 to all rewards would mean the agent would gain +1 for each time step. As it would stop gaining any reward for reaching the goal, even though reaching that goal would score the highest possible +2.5 reward, the fact that this ends the episode means that it now becomes a poor choice. The best action for the car in this modified MountainCar is to stay at the bottom of the valley collecting the +1 reward per time step forever.
Continuing problems
In a continuing problem, there is no way for the agent to avoid the stream of new reward data. That means any positive scaling of all reward values or positive or negative offset, by the same amount, has no impact on what counts as the optimal policy. The calculated value of any state under the same policy, but with rewards all transformed with the same multiplier and offset, will be different, but the optimal policy in that environment will be the same.
If you scale or offset rewards differently to each other, then that can change the goals of the agent and what the optimal policy is. The balance does not really depend on whether rewards are positive or negative in a continuing environment.
There may be some exceptions to this for continuing problems when using a discount factor, and setting it relatively low (compared to the typical state "cycling" length in the problem). That can cause changes in behaviour due to offsets, similar to those seen in episodic problems. If you use an average reward setting this tends to be less relevant. Often in DQN, you will choose a high discount factor e.g. 0.99 or 0.999, and this will tend to behave close to an average reward setting provided rewards are not very sparse.
In general
In either case, if you change a reward system, and that results in an agent that consistently learns a different policy, that will usually mean one of two things:

The original reward system was incorrect. It described a goal that you did not intend, or had "loopholes" that the agent could exploit to gain more reward in a way that you did not intend.

The implementation of the agent was sensitive in some way to absolute values of total reward. That could be due to a hyperparameter choice in something like a neural network for example, or maybe a bug.

Another possibility, that you may see if you only run a few experiments, is that the agent is not learning 100% consistently, and you are accidentally correlating your changes to reward scheme with the noise/randomness in the results. A DQN-based agent will usually have some variability in how well it solves a problem. After training, DQN is usually only approximately optimal, and by chance some approximations are closer than others.

u mention that "If an environment contains many unavoidable negative rewards, and these outweigh total positive rewards, then the agent will be motivated to complete an episode sooner." Why and how does this happen in a DQN? Where does the motivation come from?

@Fishfish: I use "will be motivated to" as loose short-hand for "will converge to this option because it is numerically optimal". It happens because the solution - of completing an episode as soon as possible - is optimal, nothing to do specifcally with a DQN, it applies equally to any solver that works in an environment where the average time step reward is unavoidably negative.

Our paper 'Exploit Reward Shifting in Value-Based DRL' answered this question.
In the case mentioned, using a negative shift will lead to explorative behaviors, therefore the DQN agent has better convergence behavior.
In general, we show that a negative (constant) shift is equivalent to optimistic value initialization, while a positive shift is equivalent to pessimistic (conservative) value initialization --- hence the former helps exploration, and the latter helps exploitation.
Reference:
1 [Google Site]
2 NeurIPS'2022 Paper

authors: Владимир Никифоров


text:
Should batch normalisation be applied before or after ReLU?
I know that there has been some discussion about this (e.g. here and here), but I can't seem to find consensus.
The crucial thing that I haven't seen mentioned in these discussions is that applying batch normalization before ReLU switches off half the activations, on average. This may not be desirable.
In other words, the effect of batch normalization before ReLU is more than just z-scaling activations.
On the other hand, applying batch normalization after ReLU may feel unnatural because the activations are necessarily non-negative, i.e. not normally distributed. Then again, there's also no guarantee that the activations are normally distributed before ReLU clipping.
I currently lean towards a preference to batch normalization after ReLU (which is also based on some empirical results).
What do you all think? Am I missing something here?

Like you said, there has been a lot of discussion but no consensus. This surely means this is not a crucial decision, or that the best choice depends on the given problem. I would suggest to compare both ways if possible. I did this in one case and I think the "bad" way (BN-RELU) was a bit better. But it may be the opposite for other architectures and problems!


text:
What is the difference between artificial neural networks and deep learning?
I have read many mixed definitions around these two terms. For example, is it right to say deep learning is any ANN with more than two hidden layers?
What are formal definitions for these two?

A few years ago, deep learning was a buzzword, but now is de-facto a standard term or expression, and it's widely used in all research papers, although deep learning is almost never defined rigorously (but this doesn't seem to be a big problem!).
From my experience (this is not just an opinion, of course!), after having read so many papers on the topic, deep learning typically refers to the subset of machine learning algorithms, specifically, gradient descent and back-propagation, applied to neural networks, and, in particular, neural networks with multiple layers. However, there is no consensus on what multiple refers to. In fact, Schmidhuber (co-author of the LSTM), in [1], writes

At which problem depth does Shallow Learning end, and Deep Learning begin? Discussions with DL experts have not yet yielded a conclusive response to this question. Instead of committing myself to a precise answer, let me just define for the purposes of this overview: problems of depth > 10 require Very Deep Learning.

So, although there is typically no agreement on the definition of deep or multiple, there are several problems, such as the vanishing or exploding gradient problems, that arise as the number of layers in a neural network increases, so there are reasons behind the distinction of deep and non-deep/shallow learning.
Note that some research papers (e.g., [1]) distinguish between deep learning applied to neural networks and other types of deep learning, so as not to exclude the layered composition of other models (e.g., SVMs) other than neural networks and the use of machine learning algorithms (e.g. gradient descent) to train them.
To answer your question more directly, neural networks are models or function approximators, while deep learning is the set of machine learning techniques applied to the layered composition of models (typically, neural networks).


text:
How can I build a deep reinforcement learning model that can be trained with multiple time series datasets
I built a DRL model to trade stocks in the financial market but the number of observations is relatively small and I would like to increase it by training the same model with stocks from several different companies. My problem is that I don't know what is the correct way to do this since the price series is a time series. Someone to enlighten me? I have read articles that show that this is possible but none that say how.

Have a look through this stored search


text:
How can a de-noising auto-encoder act as an anomaly detection model?
In some research papers, I have seen that, for training the autoencoders, instead of giving the non-anomalous input images, they add some anomalies to the normal input images, and train the auto-encoders with this anomalous images.
And, during testing, they take and pass an anomalous image and get the output, take their pixel-wise difference, and, based on a threshold, they detect if it is an anomaly or not.
If we are adding noise or anomalies to the training set, are we generalizing the model's capability to recreate the original normal input?
How does it help to detect the anomaly?
My understanding is that we should train using only normal data without adding any noise, then give an anomaly image at test, take the loss as a threshold.

Neural networks are universal approximators, i.e. able to
model any function given a sufficiently large architecture to
deal with the complexity of the problem


text:
What is the advantage of using experience replay (as opposed to feeding it sequential data)?
Let's suppose that our RL agent needs to play a game with different levels. If we train our RL agent sequentially or with sequential data, our agent will learn how to play level 1, but then it will learn to play level 2 differently, because our agent learns how to play level 2 and forgets how to play level 1, since now our model is fitted using only experiences from level 2.
How does an experience replay buffer change this? Can you explain this in simple terms?

Random sampling from the replay memory helps us reach closer to the i.i.d assumption and also lets the learning algorithm train on the data multiple times compared to just once. Therefore, there is a lower chance of the network forgetting what it previously learned. However, the experience replay size is a hyperparameter and there are disadvantages of having it too low or too big. While the former disadvantage is self evident, the latter has been explained in this paper - A Deeper Look at Experience Replay


text:
What is the difference between Bayes-adaptive MDP and a Belief-MDP in Reinforcement Learning?
I have been reading a few papers in this area recently and I keep coming across these two terms. As far as I'm aware, Belief-MDPs are when you cast a POMDP as a regular MDP with a continuous state space where the state is a belief (distribution) with some unknown parameters.
Are they not the same thing?

Hi. Maybe, for completeness, you should also explain where you heard the term "Bayes-adaptive MDP".

Both Belief-MDPs and Bayes-Adaptive MDPs (BAMDPs) are special cases of POMDPs and their state space is augmented with a belief over their unobserved/hidden variables.
In a belief-MDP, the hidden variables can change over the course of an episode. (Eg. Both the position and the uncertainty in the position of the robot can vary during an episode).
In a  BAMDP, the hidden variables are usually the attributes of the transition/reward function and are held constant during an episode. (Eg. In a robot locomotion task, ground friction or load - dynamics attributes, current goal location - reward function attributes. Though variables defining these attributes are unknown to the agent and the agent has to infer them, the actual variables remain unchanged during an episode)
References:

M Ghavamzadeh, S Mannor, J Pineau, Aviv Tamar - Bayesian reinforcement learning: A survey. Foundations and Trends in Machine Learning, 8(5-6):359-483, 2015.

L Zintgraf, K Shiarlis, M Igl, S Schulze, Y Gal, K Hofmann, and S Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. arXiv preprint arXiv:1910.08348, 2019.


text:
Which neural network should I use to distinguish between different types of defects?
I want to teach a neural network to distinguish between different types of defects. For that, I generated images of fake-defects. The images of the fake-defect types are attached.

I tried many different network architectures now:

resnet18
squeezenet
own architectures: a narrow network with broad layers and high dropout rates.

I have to say that some of these defects have really random shapes, like the type single-dirt or multi-dirt. I imagine that the classification should not be as easy as I thought before, due to the lack of repetitive features within the defects. But I always feel like the network is learning some "weird" features, which do not occur in the test set, and the results are really frustrating. I felt like teaching binary images had way better results, which should IMO be not the case.
Still, I feel like a neural network should be able to learn to distinguish them.
Which kind of network architecture would you recommend to classify the images in the attachment?


text:
Can I think of the graph convolution operation as a regular 2D convolution for images?
Kipf et al. described in his paper that we can write graph convolution operation like this:
$$H_{t+1} = AH_tW_t$$
where $A$ is the normalized adjacency matrix, $H_t$ is the embedded representation of the nodes and $W_t$ is the weight matrix.
Now, can I imagine the same formula as first performing 2D convolution with fixed-size kernel over the whole feature space then multiply the result with the adjacency matrix?
If this is the case, I think I can create a graph convolution operation just using the Conv2D layer then performing simple matrix multiplication with adjacency matrix using PyTorch.

See https://ai.stackexchange.com/q/11172/2444.


text:
What is meant by "arranging the final features of CNN in a grid" and how to do it?
In the paper What You Get Is What You See: A Visual Markup Decompiler, the authors have proposed a method to extract the features from the CNN and then arrange those extracted features in a grid to pass into an RNN encoder. Here's an illustration.

I can easily extract features from either the existing model, like ResNet, VGG, or make a new CNN model easily as they have described in the paper.
For example, let us suppose, I do this
features = keras.applications.ResNet()(images_array) # just hypothetical

How can I convert these images to the grid?? I am supposed to feed the output of the changed grid to an LSTM Encoder as:
keras.layers.LSTM()(grid) # again, hypothetical type

I just want to know what the author means from changing the output in the grid format.


text:
If the performance of an RL agent in a partially observable environment is "good", is this likely only accidental?
In my research, I remember to have read that, in case of an environment which can be modeled by partially observable MDP, there are no convergence guarantees (unfortunately, I do not find the paper anymore and I would appreciate if someone can post the link to the reference).
If the performance of an RL agent in a partially observable environment is "good" (i.e. the agent does pretty well in achieving its goal), is this likely only accidental or due to chance?


text:
How is the F1 score calculated in a question-answering system?
I have an NLP model for answer-extraction. So, basically, I have a paragraph and a question as input, and my model extracts the span of the paragraph that corresponds to the answer to the question.
I need to know how to compute the F1 score for such models. It is the standard metric (along with Exact Match) used in the literature to evaluate question-answering systems.

In QA, it's computed over the individual words in the prediction against those in the True Answer. The number of shared words between the prediction and the truth is the basis of the F1 score: precision is the ratio of the number of shared words to the total number of words in the prediction, and recall is the ratio of the number of shared words to the total number of words in the ground truth.
Source

It really depends on what you are looking for your model to do. For example, do false negatives or false positives really cost your research (or your business)? Also, it's very important to consider your label (class) distribution.
If you just want to achieve the highest accuracy, and you don't have any issue with your class distribution (that I believe you probably don't in your case) then accuracy works pretty well.
F1 score might be a better option to use if you need to seek a balance between precision and recall and there is an uneven class distribution.

Thanks for your answer. In my  case I need to compute F1 to compare my model to another one. So I need a clear definition of F1 in case of question/answering tasks like SQuAD dataset. In other words, I need to know how is F1 score calculated for a model trained on SQuAD dataset.


text:
What is the reason for mode collapse in GAN as opposed to WGAN?
In this article I am reading:

$D_{KL}$ gives us inifity when two distributions are disjoint. The value of $D_{JS}$ has sudden jump, not differentiable at $\theta=0$. Only Wasserstein metric provides a smooth measure, which is super helpful for a stable learning process using gradient descents.

Why is this important for a stable learning process? I have also the feeling this is also the reason for mode collapse in GANs, but I am not sure.
The Wasserstein GAN paper also talks about it obviously, but I think I am missing a point. Does it say JS does not provide a usable gradient? What exactly does that mean?

I don't have a definite answer, but only a suspicion/idea:
Looking at Figure 1 from the WGAN paper, we clear see that the JS divergence on the right is not continuous at $0$, hence not differentiable at $0$. However, the EM plot on the left is continuous also at $0$. You could now argue that we have a kink there, so it should not be differentiable there either, but they might have a different notion of differentiability, I am honestly not sure about it right now.

There is a fairly straightforward way of doing optimization of piecewise linear functions that would deal fine with the left hand side, the non-differentiability at the 'kink' is not an issue, see e.g. here


text:
Why can we perform graph convolution using the standard 2d convolution with $1 \times \Gamma$ kernels?
Recently I was reading this paper Skeleton Based Action RecognitionUsing Spatio Temporal Graph Convolution. In this paper, the authors claim (below equation (\ref{9})) that we can perform graph convolution with the following formula
$$
\mathbf{f}_{o u t}=\mathbf{\Lambda}^{-\frac{1}{2}}(\mathbf{A}+\mathbf{I}) \mathbf{\Lambda}^{-\frac{1}{2}} \mathbf{f}_{i n} \mathbf{W} \label{9}\tag{9}
$$
using the standard 2d convolution with kernels of shape $1 \times \Gamma$ (where $\Gamma$ is defined under equation 6 of the paper), and then multiplying it with the normalised adjacency matrix
$$\mathbf{\Lambda}^{-\frac{1}{2}}(\mathbf{A}+\mathbf{I}) \mathbf{\Lambda}^{-\frac{1}{2}}$$
For the past few days, I was thinking about his claim but I can't find an answer. Does anyone read this paper and can help me to find it out, please?

Isn't this a duplicate of https://ai.stackexchange.com/q/22650/2444? Or are you interested only in the 1d convolution? If yes, maybe read my answer https://ai.stackexchange.com/a/21824/2444. I talk about 1d convolutions there, but I don't remember anymore the level of detail I used.

I am interested in (1xD) convolution specifically. But can you please explain your answer?

Have you read my answer: https://ai.stackexchange.com/a/21824/2444? (Btw, I edited the comment above to include a link to this answer). I don't know if it answers your question because I don't remember what I wrote there, but I think it should give you some intuition behind 1d convolutions, as far as I remember :P

Yes, I go through the answer you gave previously. I think you are more focused on FCN and segmentation but I am trying to know can 1xD standard convolution and then multiplying it with adjacency matrix is the same as graph convolution in the skeleton based graphs?

It's been a while since I read something about graph NNs, but can you point me exactly to the part of the paper where the author claims that, so that I have some context?

obviously, section 3.6, after equation 9.

So, if I understand correctly, they are performing a 2d convolution between $\mathbf{f}{i n}$ and $\mathbf{W}$ with kernels of shape $1 \times \Gamma$ to implement the multiplication $\mathbf{f}{i n} \mathbf{W}$. What is $\mathbf{f}_{i n}$? The features at the nodes? I think so because they say "In practice, under the spatial temporal cases, we can represent the input feature map as a tensor of $(C, V, T)$ dimensions"

Yes, you're right. $f_{in}$ is the feature vector of the nodes.

I can't think about it right now (cause I don't have much time), but, hopefully, this discussion and clarification was somehow useful. Maybe the next step to understand is what are $C$, $V$ and $T$ (if you haven't already) and why the feature map can be represented as a 3d tensor of dimensions $(C, V, T)$. If I remember correctly, algorithms that apply the so-called "graph convolution" are actually message-passing algorithms, where you "flow the information between nodes" for some time steps, and that should explain the parameter $\Gamma$.

It;s okay. $C$ = coordinate of a perticular joint(x,y,z), $V$ = vertex(in this case joints), $T$ = Timestep.

Ok. I just want to clarify one thing. They are apparently not talking about 1d convolutions (as I initially thought from your explanation), but 2d convolutions with kernels of shape $1 \times \Gamma$, i.e. rather than having the usual $3 \times 3$ (or $5 \times 5$), you have a $1 \times \Gamma$ kernel. If they really apply a 2d convolution, then the actual kernel will have the same $1 \times \Gamma \times N$, where $N$ is the depth of the feature map, which should be $T$, from what I understand, so the kernel should be $1 \times \Gamma \times T$.

Then you apply (i.e. convolve) this kernel $1 \times \Gamma \times T$ to the spatial dimensions of the input, where the spatial dimensions (height and width, or vice-versa) should be $C$ and $V$. I don't know which values $\Gamma$ is supposed to take, but you may want to try to solve values, like $3$, and see why the graph convolution would be equivalent to this 2d convolution with kernel $1 \times 3 \times T$. (I hope I didn't confuse you more).

yes, you are right. I initally talk about 1D convolution since in their code  $\Gamma$ parameter is set to 1

I wrote "kernel will have the same", but I meant "kernel will have the shape" and I wrote "want to try to solve values, like 3", but I meant "want to try to solve this with values like $\Gamma = 3$".

Ha, so they set $\Gamma = 1$. I see that you wrote that only now. Well, in that case, you're really applying a $1\times 1 \times T$ kernel to the input feature map (i.e. 1d convolution). The application of a $1 \times 1$ kernel to the input produces a tensor with the same spatial dimensions as the input feature map. So, if you apply only one $1 \times 1 \times T$ kernel to an input feature map of shape $(C, V, T)$, then you will get an output feature map of shape $(C, V)$.

What are the dimensions of the normalized adjacency matrix, $\mathbf{\Lambda}^{-\frac{1}{2}}(\mathbf{A}+\mathbf{I}) \mathbf{\Lambda}^{-\frac{1}{2}}$?

After the convolution operation, they got 4-dimensional tensor, where extra 2 dimensions are batch size and number of filter. The normalized adjacency matrix is a 3 dimensional tensor

What are the 3rd and 4th dimensions?

After convolution the dimension of the feature map is $(N,C,V,F)$ where $N$=no of batch,$F$ = number of filters and $C,V$ are cordiality and joints. The dimension of the normalized adjacency matrix is $(K,V,V)$ where $V$= no of joints, I am not sure about $K$.

Then that's consistent with what I said, and it somehow seems to be consistent with what is written below that equation 9, i.e. "and multiplies
the resulting tensor with the normalized adjacency matrix on the second dimension"

I think $K=1$, so the dimension of the adjacency matrix is $(1,V.V)$

Ha, wait, I misinterpreted your last comment or you changed it. A tensor of shape $(N, C, V, F)$ needs to be multiplied with a tensor of shape $(K, V, V)$, on the second dimension. That looks problematic, actually, because the dimensions don't seem to match for matrix multiplication.

My bad input tensor will be $(N,F,C,V)$. They are doing the multiplication like this: torch.einsum('nkctv,kvw->nctw', (x, A))

I am not familiar with that function (although I once used PyTorch). I would need to check the documentation, but I don't have time now.

Check this paper out - Graph Classification with 2D Convolutional Neural Networks
This addresses your issue.
https://arxiv.org/abs/1708.02218


text:
Is there an optimal way to split the text into small parts when working with co-reference resolution?
I am working with co-reference resolution in a large text. Is there an optimal way to split the text into small parts? Or the best correct procedure is to use the entire text?
Just for reference, I am using the library spacy-neuralcoref in Python that is based on Deep Reinforcement Learning for Mention-Ranking Coreference Models by Kevin Clark and Christopher D. Manning, EMNLP 2016.
Why am I asking about splitting the text?
I am applying coreference to chapters of books (roughly 30 pages of text). All the examples I have seen show situations of coreference applied to small pieces of texts. I applied to a chapter and I found strange results. However, this is not a clear justification for that since the state of art in coreference is about 60%. Am I right?
I didn't check all databases that people use to test coreference but the ones I took a look (like MUC 3 and MUC 4 Data Sets), if I understand well, they were composed by a collection of a small number of paragraphs.
A test Example:

TST1-MUC3-0001
GUATEMALA CITY, 4 FEB 90 (ACAN-EFE) -- [TEXT] THE GUATEMALA ARMY
DENIED TODAY THAT GUERRILLAS ATTACKED THE "SANTO TOMAS" PRESIDENTIAL
FARM, LOCATED ON THE PACIFIC SIDE, WHERE PRESIDENT CEREZO HAS BEEN
STAYING SINCE 2 FEBRUARY.
A REPORT PUBLISHED BY THE "CERIGUA" NEWS AGENCY -- MOUTHPIECE OF
THE GUATEMALAN NATIONAL REVOLUTIONARY UNITY (URNG) -- WHOSE MAIN
OFFICES ARE IN MEXICO, SAYS THAT A GUERRILLA COLUMN ATTACKED THE FARM
2 DAYS AGO.
HOWEVER, ARMED FORCES SPOKESMAN COLONEL LUIS ARTURO ISAACS SAID
THAT THE ATTACK, WHICH RESULTED IN THE DEATH OF A CIVILIAN WHO WAS
PASSING BY AT THE TIME OF THE SKIRMISH, WAS NOT AGAINST THE FARM, AND
THAT PRESIDENT CEREZO IS SAFE AND SOUND.
HE ADDED THAT ON 3 FEBRUARY PRESIDENT CEREZO MET WITH THE
DIPLOMATIC CORPS ACCREDITED IN GUATEMALA.
THE GOVERNMENT ALSO ISSUED A COMMUNIQUE DESCRIBING THE REBEL REPORT
AS "FALSE AND INCORRECT," AND STRESSING THAT THE PRESIDENT WAS NEVER
IN DANGER.
COL ISAACS SAID THAT THE GUERRILLAS ATTACKED THE "LA EMINENCIA"
FARM LOCATED NEAR THE "SANTO TOMAS" FARM, WHERE THEY BURNED THE
FACILITIES AND STOLE FOOD.
A MILITARY PATROL CLASHED WITH A REBEL COLUMN AND INFLICTED THREE
CASUALTIES, WHICH WERE TAKEN AWAY BY THE GUERRILLAS WHO FLED TO THE
MOUNTAINS, ISAACS NOTED.
HE ALSO REPORTED THAT GUERRILLAS KILLED A PEASANT IN THE CITY OF
FLORES, IN THE NORTHERN EL PETEN DEPARTMENT, AND BURNED A TANK TRUCK.

I am applying coreference to chapters of books (roughtly 30 pages of text). All the examples I have seem show situations of coreference applied to small pieces of texts. I applied to a chapter and I found strange results. However, this is not a clear justification for that since the state of art in coreference is about 60%. Am I right? You are saying that the best approach is to input the entire chapter?

I didn't check all databases but the ones I took a look (like MUC 3 and MUC 4 Data Sets), if I understand well, they were composed by a collection of a small number of paragraphs.


text:
Why is DDPG not learning and it does not converge?
I have used a different setting, but DDPG is not learning and it does not converge. I have used these codes 1,2, and 3 and I used different optimizers, activation functions, and learning rate but there is no improvement.
    parser.add_argument('--actor-lr', help='actor network learning rate', default=0.001)
    parser.add_argument('--critic-lr', help='critic network learning rate', default=0.0001)
    parser.add_argument('--gamma', help='discount factor for critic updates', default=0.95)

    parser.add_argument('--tau', help='soft target update parameter', default=0.001)
    parser.add_argument('--buffer-size', help='max size of the replay buffer', default=int(1e5))
    parser.add_argument('--minibatch-size', help='size of minibatch for minibatch-SGD', default=64)

    # run parameters
    # parser.add_argument('--env', help='choose the gym env- tested on {Pendulum-v0}', default='MountainCarContinuous-v0')
    parser.add_argument('--random-seed', help='random seed for repeatability', default=1234)
    parser.add_argument('--max-episodes', help='max num of episodes to do while training', default=200)
    parser.add_argument('--max-episode-len', help='max length of 1 episode', default=100)

I have trained in the same environment with A2C and it converged.

Which parameters should I change to make the DDPG converge? Can anyone help me with this?


text:
How can I predict the true label for data with incomplete features based on the trained model with data with more features?
Suppose I have a model that was trained with a dataset that contains the features (f1, f2, f3, f4, f5, f6). However, my test dataset does not contain all features of the training dataset, but only (f1, f2, f3). How can I predict the true label of the entries of this test dataset without all features?

Assuming that you have access to the training data set, you could use an autoencoder network to predict what features f4, f5, f6 'could be' for the test data set. The way to do this is to train the autoencoder on the training data set with features f1, f2, f3 as inputs, and then use f1,f2,f3,f4,f5,f6 as the output of the network. The autoencoder then effectively learns to map any input samples with (f1,f2,f3) to (f1,f2,f3,f4,f5,f6). By passing your test data through the autoencoder, you can then use the output and pass it to your model.

Thank you so much! I will attempt to learn this network model

I assume you trained your model on (f1, f2, f3, f4, f5, f6) and in your test data you sometimes have (f1, f2, f3) and sometimes have for example (f1, f2, f3, f4, f5, f6), right? Because if your test data always have (f1, f2, f3), then isn't it better to just train a model on available features?
So if my assumption is correct what I would do is to manipulate the training set a bit, keeping some training set with (f1, f2, f3, f4, f5, f6) and some others with (f1, f2, f3) with replacement of real values in their (f4, f5, f6) by e.g. mean of respective feature. So all training set still have (f1, f2, f3, f4, f5, f6) but some of them have manipulated (f4, f5, f6). Then finally when testing, do the same manipulation to those test data that have a smaller number of features.
I think like this your model learn how to predict base on (f1, f2, f3) when other features are not available. but at the same time, take advantage of all features if they are all available.
It's probably not the best approach but it worth to try.

I think your explanation means 'imputation' for missing value


text:
Could a quantum computer perform vectorized forward propagation in deep networks?
Forward propagation in Deep Neural Networks
In the "Forward Propagation in a Deep Network" video on Coursera, Andrew NG mentions that there's no way to avoid a for loop to loop through the different layers of the network during forward propagation.
See image showing a deep network with 4 layers, and the requirement of a forloop to compute activations for each layer during forward propagation: https://nimb.ws/CkRVLT
This makes intuitive sense since each layer's activation depends on the previous layer's output.
Warning: start of speculation
My rudimentary understanding of quantum computing is that it somehow "magically" can bypass computing intermediate states -> this is why supposedly quantum computers can break cryptography... or something like that.
I'm wondering if a quantum computer could perform vectorized forward propagation on an L layer deep neural network.

just commenting to point out the existence of quantumcomputing.SE, in case you were not aware of the site


text:
How can one be sure that a particular neural network architecture would work?
Traditionally, when working with tabular data, one can be sure(or at least know) that a model works because the included features could explain a target variable, say "Price of a ticket" good. More features can be then be engineered to explain the target variable even better.
I have heard people say, that there is no need to hand-engineer features when working with CNNs or RNNs or Deep Neural Networks, provided all the advancements in AI and computation. So, my question is, how would one know, before training, why a particular architecture worked(or would work) when it did or why it didn't when the performance isn't acceptable or very bad. And also that not all of us would have the time to try out all possible architectures, how can one know or at least be sure that something would work for the problem in hand. Or to say, what are the things one needs to follow when designing an architecture to train for a problem, to ensure that an architecture will work?


text:
What should I do with the flatten layer during back-propagation?
I'm creating a CNN network without other frameworks such as PyTorch, Keras, Tensorflow, and so on.
During the forward pass, the Flatten layer reshapes the previous layer's activation.  I know there are a lot of questions about it, but what should I do with the Flatten layer during back-propagation? Should I compute the derivative of $dA$ and reshape it for the next layer or just reshape $dA$ of the previous layer?


text:
How do we make our outputs to have the same size as the true mask?
When we are doing multi-label segmentation tasks, our y_true (the mask) will be (w, h, 3), but, in our model, at the last layer, we will be getting (w, h, number of classes) as output.
How do we make our outputs to have the same size as the true mask so that to apply the loss function, given that, currently, the shapes are not equal? Also, if we are done with applying the loss function and trained the model, how do I make results in the shape of (w, h, 3) from (w, h, number of classes)?

You can create a mapping from classes to colors for a simple one is:
y # y.shape = (W, H, n_classes)
_, y_color = y.max(dim=-1, keepdim=True) / n_classes # y_color.shape = (W, H, 1)
y_color = torch.cat([y_color] * 3, dim=-1) # y_color.shape = (W, H, 3)

(using pytorch like code)
This mapping is visualizable, of course, you may get nicer visualizations if your class to color mapping is more sophisticated.


text:
Why is GPT-3 such a game changer?
I've been hearing a lot about GPT-3 by OpenAI, and that it's a simple to use API with text in text out and has a big neural network off 175B parameters.
But how did they achieve this huge number of parameters, and why is it being predicted as one of the greatest innovations?

You're asking "why is it being predicted as one of the greatest innovations?", but who is predicting it as a great innovation? Edit your post to include links to sources that show that people are predicting this. Any model that uses that big number of parameters doesn't seem like a big innovation to me but just brute-force.

The main point in GPT-3 and already in 2 was the observation that performance was steadily increasing with increasing model size (As seen in Figure 1.2 in your linked paper). So it seems that while all progress made in NLP was definitely useful, it also is important to just scale up the model size.
This may not seem like surprising point, but it actually kind of is. Normally, performance would saturate or at least the gain would slope off, but this is not the case! So the main innovation may not be that big and is kind off brute-force but the point still stands: Bigger models are better.
Another point to mention is the way they did the training. Such a large model needs some tricks to be actually trained (and fast at that). You also want to make use of multiple GPUs for parallel training. This means they also had to develop new structures for training.
Why exactly it is predicted as a huge innovation may only be contained to some twitter demonstration, there are no real sources on this as far as I know. Especially because the model is not openly available.


text:
How can the FCNN reduce the dimensions of the input from $1048 \times 100$ to $523 \times 100$ with max-pooling?
I am trying to implement a paper on Image tempering detection and localization, the paper is Image Manipulation Detection and Localization Based on the Dual-Domain Convolutional Neural Networks, I was able to implement the SCNN, the one surrounded by red dots, I could not quite understand the FCNN, the one that is surrounded with blue dots.
The problem I am facing is: How the network made features vector from (1048 x 100) to (523 x 100) through max-pooling (instead of 524 x 100), and from (523 x 100) to (260 x 100) and then (260 x 100) to (256, ).

It appears that the given network diagram might be wrong, but, if it is wrong, how could it be published in IEEE. Please, help me understand how the FCNN is constructed.

How did you get 520x100? With a max pooling filter of 2x1 I would assume the output should be 524x100

@Recessive sorry, typo it is 524 x 100

Ok that makes sense, at least the clarifies. Unfortunately I don't know why their numbers are off by one. I can only assume that they shifted the max pool filter by 1 at some point, but I can't know for sure or why. Hopefully someone else does


text:
Are there examples of agents that use a more modest number of parameters on Pendulum (or similar environments)?
I'm looking at some baseline implementations of RL agents on the Pendulum environment. My guess was to use a relatively small neural net (~100 parameters).
I'm comparing my solution with some baselines, e.g. the top entry on the Pendulum leaderboard. The models for these solutions are typically huge, i.e. ~120k parameters. What's more, they use very large replay buffers as well, like ~1M transitions. Such model sizes seem warranted for Atari-like environments, but for something as small as the Pendulum, this seems like complete overkill to me.
Are there examples of agents that use a more modest number of parameters on Pendulum (or similar environments)?

Actually, I just started inspecting the entries further down in the leaderboard list, and there are in fact more modest architectures, e.g. this one, which uses 3 hidden layers with 8 units each.


text:
Is there a way to get landmark features automatically learned by a neural network?
Is there a way to get landmark features automatically learned by a neural network without having to manually pre-label them in the images that are being fed into the network?

No, you can't. In CNN, if you want to detect landmark, you need to prepare data with region box, it's coordinates, width, height, than number of points that should be detected and points coordinates. Then your target vector should be,

This is your target vector. Optionally you can use YOLO algorithm.


text:
Is there a place where people can share (or buy) ready made neural networks?
Is there a place where people can share (or buy) ready made neural networks instead of creating them themselves? Something like a Wikipedia for DNNs?

I suppose you could count kaggle as one place. Post a competition with a prize pool, and just take the model that performs best.

It's not a specific web for sell or share neural network models, but actually you can easily find other people models in Github! Just search it! For example, this is a random repo I've found for Cat Classification.
But.. the problem is everyone have different problems. So you can't easily use other people neural network models and then use it for your problem. That's why there's a "trick" called transfer learning or fine tuning. That method show you how to train other people models (it's called: pre-trained models) from different problems for your specific problem.
Tensorflow and PyTorch provide a lot of pre-trained model for common problems:

Pytorch Image Classification Pre-Trained models
Tensorflow Object Detection Model Zoo
PyTorch Detectron Model Zoo

You can also find other pre-trained models in Github! This is an official PyTorch tutorial how to use that pre-trained model to your specific case.
UPDATE
You can also check HuggingFace Models Hub it's like a repository for machine learning model (most of them are transformer-based and for NLP). It's well organized, you can sort by your specific task or library

authors: Андрій Степаненко


text:
Why does the number of channels in the PointNet increase as we go deeper?
For example, in PointNet, you see the 1D convolutions with the following channels 64 -> 128 -> 1024. Why not e.g. 64 -> 1024 -> 1024 or 1024 -> 1024 -> 1024?

@nbro It's not just PointNet, you see this design choice in pretty much all architectures.

The number of kernels is typically associated with the number of different features that you want to detect in the input, although this is just an interpretation. So, in your case, if you increase the number of kernels (or channels), you are probably assuming that, as you process the input, more features should be detected. It would be nice if you could also provide other examples (just provide a link to the paper) where the number of channels or kernels increases as we go deeper, just for reference.


text:
Is A* with an admissible but inconsistent heuristic optimal?
I understand that, in tree search, an admissible heuristic implies that $A*$ is optimal. The intuitive way I think about this is as follows:
Let $P$ and $Q$ be two costs from any respective nodes $p$ and $q$ to the goal. Assume $P<Q$. Let $P'$ be an estimation of $P$. $P'\le P \Rightarrow P'<Q$. It follows from uniform-cost-search that the path through $p$ must be explored.
What I don't understand, is why the idea of an admissible heuristic does not apply as well to "graph-search". If a heuristic is admissible but inconsistent, would that imply that $A*$ is not optimal? Could you please provide an example of an admissible heuristic that results in a non-optimal solution?

It depends on what you mean by optimal.

A* will always find the optimal solution (that is, the algorithm is admissible) as long as the heuristic is admissible. (Note that the definition of admissible is overloaded and means something slightly different for an algorithm and a heuristic.)

If you talk about the set of nodes expanded by A*, then it expands the minimal set of nodes up to tie-breaking even with an inconsistent heuristic.

If you talk about the number of expansions, then A* is not optimal. A* can perform up to $2^N$ expansions of $N$ states with an inconsistent heuristic. This result comes from Martelli, 1977.

Algorithm B has worst-case $N^2$ expansions and budgeted graph search (BGS) has worst-case $N \log C$, where $C$ is the optimal solution cost.

You can see a demo that shows the worst-case performance of A* as well as the performance of BGS here:
https://www.movingai.com/SAS/INC/


text:
What is the amount of test data needed to evaluate a CNN?
I have an image dataset of about 400 images. 70% of these data points were used for training, 15% for validation, and 15% for testing. I am using the 70% to train a CNN-based binary classifier. I augmented the training data to around 8000 images. That makes my test set really small in comparison. Is that ok, and what is considered a decent size of images for a test set?

Also, I am an hs student who is really new to machine learning and would appreciate any advice :)

Depends on the complexity of the task. I know I have previously achieved high accuracy on MNIST with a similarly low amount of training data.


text:
Why do we use $X_{I_t,t}$ and $v_{I_t}$ to denote the reward received and the at time step $t$ and the distribution of the chosen arm $I_t$?
I'm doing some introductory research on classical (stochastic) MABs. However, I'm a little confused about the common notation (e.g. in the popular paper of Auer (2002) or Bubeck and Cesa-Bianchi (2012)).
As in the latter study, let us consider an MAB with a finite number of arms $i\in\{1,...,K\}$, where an agent choses at every timestep $t=1,...,n$ an arm $I_t$ which generates a reward $X_{I_t,t}$ according to a distribition $v_{I_t}$.
In my understanding, each arm has an inherent distribution, which is unknown to the agent. Therefore, I'm wondering why the notation $v_{I_t}$ is used instead of simply using $v_{i}$? Isn't the distribution independent of the time the arm $i$ was chosen?
Furthermore, I ask myself: Why not simply use $X_i$ instead of $X_{I_t,t}$ (in terms of rewards). Is it because the chosen arm at step $t$ (namely $I_t$) is a random variable and $X$ depends on it? If I am right, why is $t$ used twice in the index (namely $I_t,t$)?
Shouldn't $X_{I_t}$ be sufficient, since $X_{I_t,m}$ and $X_{I_t,n}$ are drawn from the same distribution?

Isn't the distribution independent of the time the arm $i$ was chosen?

Each one of the two references you describe assumes the context of the random bandit problem proposed by Robbins (1952) where the underlying reward distributions of each bandit are fixed. Therefore, yes, the underlying distributions are independent of the current time.

Is it because the chosen arm at step $t$ (namely $I_t$) is a random variable and $X$ depends on it?

The reward is a random variable which is dependent on the chosen arm at time $t$. Since each arm has an underlying reward distribution, the index $I_t$ is a random variable that designates the specific arm we are pulling, and the index $t$ denotes the time step when we pull the arm.

Why is $t$ used twice in the index (namely $I_t,t$)?

Note that $t$ is used twice, but the observed value of $I_t$ does not encode any information about the time it was chosen. For example, if $I_m = 5$, then $X_{I_m,\ m} = X_{5,\ m}$. If we drop the second subscript, then we have no way to distinguish $X_{5,\ m}$ notationally from $X_{5,\ n}$ (where $I_{n\ \neq\ m} = 5$). Two distinct rewards $X_{5,\ m}$ and $X_{5,\ n}$ would map to the same reward $X_5$ notationally. At first glance, this introduces numerous potential notational problems, such as losing the count of the number of times each arm was pulled.

Why not simply use $X_i$ instead of $X_{I_t,\ t}$ (in terms of rewards)? Shouldn't $X_{I_t}$ be sufficient, since $X_{I_t,\ m}$ and $X_{I_t,\ n}$ are drawn from the same distribution?

Admittedly, there are probably ways to get around the extra subscript for certain algorithms. For example, maybe you are using an algorithm where past rewards from each arm are averaged to yield an estimate of each arm's expected reward (see Section 2.2 of Sutton and Barto). This may require a collection of lists that store the past rewards for each arm, or it may require the count of each arm being pulled and an associated current estimate of the expected reward (see Section 2.4 of Sutton and Barto). However, these methods introduce more parameters that would be unnecessary had we initially included a second subscript for time in our notation (e.g. the counts of each arm pulled, the current estimate of expected reward for each arm, the labels of each reward list corresponding to an arm, etc.). Most of the fundamental equations regarding multi-armed bandits that I have seen are either heavily or solely dependent on the reward random variable (e.g. the definition of regret). Keeping the time index in a single random variable promotes concision and consistency among various sources by preventing the need for delegating the time index to another random variable, data structure, etc., even though specific implementations or contexts may profit from other notations.
The dual-subscript notation also has the benefit of generalizing to other contexts aside from the one posed by Robbins (1952). These include nonstationary reward distributions (see Section 2.5 of Sutton and Barto), time discounting, and families of alternative bandit processes, among others (see Sections 2.2-2.4 of this book for info on the last two extensions).

Isn't the distribution independent of the time the arm $i$ was chosen?

Yes, but you don't know which arm was chosen at time $t$, that is what $I_t$ represents. $v_i$ would represent the $i$th arms distribution, whereas you want the distribution of the arm that was chosen at time $t$, which is $v_{I_t}$.
$X_{I_t,t}$ is used to represent the arm you chose at time $t$ and the time you chose it. Imagine if we chose the following arms at respective time steps {1,2,5,2,4}, then the reward you get at time 3 (assuming in my example time started at 1) would be $X_{5,3}$. You need this notation because every time you pull arm $i$ you get a different reward, because the reward is a random variable (unless the assumption was a deterministic reward but that would not be interesting).
It has been a while since I read the papers but I assume that the distributions of the arms are stationary, however this notation is more general and would allow for non-stationary distributions.


text:
Can I apply AdaBoost on a random forest?
I know the random forest is a bagging technique. But what if my random forest overfits on a dataset, so I reduce the depth of the decision tree and now it is underfitting. In this scenario, can I take the under-fitted random forest with little depth and try to boost it?

Random forest does not overfit. Check this - https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#remarks

@naive - any learner can overfit, though the RF tries hard not to do so.

The random forest (rf) is the perfectly parallel ensemble of CART learners. It uses Gini impurity to inform its split locations, and ensemble summary (mean, mode) to track error.
The gradient boosted machine (gbm) is the perfectly serial ensemble of CART learners. It can use the Adaboost method of weighting error, though gradient boosting tends to be preferred.  The first tree is the predictor, and all others try and resolve the error.
There is a multiplicity of ensembles that are not 100% parallel or 0% parallel.
I have had decent results by predicting using the RF, computing the Error, and then using a gbm to try and clean up the error.  It is something that has been "on the menu" for ~20 years since those learners came out.
EDIT:
These folks talk about it some.  I don't know if they made a random-forest of gradient boosted trees, or a gradient boosted tree of random-forests (its been a while since I went through it).
https://www.jmlr.org/papers/volume10/tuv09a/tuv09a.pdf


text:
What is convergence analysis, and why is it needed in reinforcement learning?
While reading a paper about Q-learning in network energy consumption, I came across the section on convergence analysis. Does anyone know what convergence analysis is, and why is convergence analysis needed in reinforcement learning?

What's the title of the paper you're reading?

Convergence analysis is about proving that your policy and/or value function converge to some desired value, which is usually the fixed-point of an operator or an extremum. So it essentially proves that theoretically the algorithm achieves the desired function. Without convergence, we have no guarantees that the value function will be accurate or the policy will be any good, so in other words the proposed RL algorithm can completely fail at serving its purpose even in simple cases.


text:
Why do we need target network in deep Q learning?
I already know deep RL, but to learn it deeply I want to know why do we need 2 networks in deep RL. What does the target network do? I now there is huge mathematics into this, but I want to know deep Q-learning deeply, because I am about to make some changes in the deep Q-learning algorithm (i.e. invent a new one). Can you help me to understand what happens during executing a deep Q-learning algorithm intuitively?

I think also this question goes into the exact same direction: Why does adding another network help in double DQN?.

By the way: Here is a video that also explains it nicely as well: https://youtu.be/xVkPh9E9GfE?t=171
I can only recommend that online lecture series.

In DQN that was presented in the original paper the update target for the Q-Network is $\left(r_t + \max_aQ(s_{t+1},a;\theta^-) - Q(s_t,a_t; \theta)\right)^2$ were $\theta^-$ is some old version of the parameters that gets updated every $C$ updates, and the Q-Network with these parameters is the target network.
If you didn't use this target network, i.e. if your update target was $\left(r_t + \max_aQ(s_{t+1},a;\theta) - Q(s_t,a_t; \theta)\right)^2$,  then learning would become unstable because the target, $r_t + \max_aQ(s_{t+1},a;\theta)$, and the prediction, $Q(s_t,a_t; \theta)$, are not independent, as they both rely on $\theta$.
A nice analogy I saw once was that it is akin to a dog chasing it's own tail - it will never catch it because the target is non-stationary; this non-stationarity is exactly what the dependence between the target and the prediction cause.

Given that this is a duplicate but your answer is valuable, I think it would have been better to write the answer under https://ai.stackexchange.com/q/6982/2444.


text:
What is a heatmap in the CornerNet paper?
I have been working on understanding how CornerNet works, but I couldn't figure out a few parts about the architecture.
First, the authors mention that there are 3 distinct parts to be predicted as a heatmap, embedding, and offset.
Also, in the paper, it is stated that the network was trained on the COCO dataset, which has bounding box and class annotations.
As far as I am concerned, since CornerNet is based on detecting the top-left and bottom-right corners, the ground-truth labels for heatmap should be composed of top-left and bottom-right pixel locations of bounding boxes with the class score (but I might be wrong). What is the heatmap used for?
Moreover, for the embedding part, authors used the pull&push loss at the ground-truth pixel locations to find out which corner pairs belong to which object, but I don't understand how to backpropagate this loss. How do I back-propagate the embedding loss?

Heatmap in the sense of Corner net is the heatmap of the pooled corner values. As discussed in the paper, there is a corner pooling operation that gives you the vector values for a pixel point being a corner(it may or not). The output from the corner pooling is CxHxW. Then to generate a heat map you train the network similar to the Grad-CAM method. Training for heatmap generation means that we are going to tell how much a given pixel point has a weight to be a corner point. i.e if it has a high weight it will generate more colored values on that pixel area.
The heatmap is used for the prediction of corner points. That means if you know which pixel has a higher weight then that pixel will have a higher probability to hold the top-right or bottom-left corner of the given bounding box.


text:
How to solve the "dangerous feedback loops" in machine learning?
From the article Dangerous Feedback Loops in ML

Let’s say our model has leads from Facebook, Google, and Bing. If our first model decides that the probability of conversion is 3%, 5%, and 1% from these given sources, and we have finite amount of callbacks we can make, we will only callback the 5% probability. Now fast forward two months. The second model finds these probabilities are now: 0.5%, 8.5%, and 0%. What happened?

Because we started only calling Google leads back, we increased our chances of converting these leads, and likewise, because we stopped calling Facebook and Bing leads, these leads never converted because we never called them. This is an example of a real world feedback loop

How can we solve this problem?

This question needs to be clarified. We should not need to read that linked article in order to understand what is going on, i.e. provide some context or summarise what is written in that article.

I read the article you linked, and what you are missing are that the given conversion probabilities are assessed pre-callback - i.e. they include an assessment of whether you will even call them back or not. So of course the probabilities change if you change your behaviour. The writer of the article has created a bit of a straw man argument by defining a model and decision process that don't go well together. They should have used a model that predicted conversion rates after callback, then it could be used as they wanted.
So the sales example is pretty easy to solve. Make the model accept the lead source as a feature, and predict the conversion rate after callback. That will give you the "action value" of choosing to callback, which is much more useful value to have if you are deciding whether or not to callback.
To cover the possibility that probabilities change over time, you have to be willing to test that hypothesis by calling back at least some leads that the model predicts have a low probability of success, in order to update the model. This is related to exploration vs exploitation trade off in bandit problems or in reinforcement learning. The callback optimisation problem looks a lot like a bandit problem in fact.
Many problems of choosing how to act optimally, based on an updating statistical model of results, can be re-framed successfully as bandit problems or reinforcement learning problems. This is one way to try and address the feedback loops issue, because these include theory around decision making. It is not a magic fix for all these problems, but contextual bandits are very strong in advertising strategies that are very similar to the example you give.
To be able to address feedback loops like this well does require that source data is unbiased, so that adding conditional actions to your model helps obtain better ground truth. So the other examples that the linked article gives - recidivism and recruitment - are much harder to address, because biased data is inserted as ground truth even if you add the conditions (e.g. for exploring and tracking results in a statistically neutral way). For some, adding these conditions may be enough to help, it will make the model accurate enough that it starts to address bias. For others, there is no really good solution other than to be aware of weakness of the model.
You can attempt to de-bias the model by using some abstract ideal. A common technique used in recruiting for instance is to remove features that might be causing bias: Literally remove race, sex and related determiners from the CV on the grounds that these should not be part of any fair model, regardless of whether or not a ML model could make of use them to predict results. This might accidentally throw out data that would improve accuracy too, but can be used to achieve a decision process that observers agree should be free of contentious soruces of bias.
Which features to remove or adjust is an open and subjective question. Unfortunately ML models can often determine protected characteristics from other data, and with "black box" ones such as neural networks it is not clear whether or not they are doing so. It's still an open question, and the main defence against biased models is basically awareness that it is a problem and that feeding all your data into a computer to get a statistical model does not in any way "purify" it or achieve some higher level of objective truth.


text:
Why does learning rate reduce train-test generalization gap?
In this blog post: http://www.argmin.net/2016/04/18/bottoming-out/
Prof Recht shows two plots:

He says one of the reasons the plot below has a lower train-test gap is because that model was trained with a lower learning rate (and he also manually drops the learning rate at 120 epoch).
Why would a lower learning rate reduce overfitting?

bigger learning rate makes the loss curve oscillation high, and it works for training data only

BTW, two represented figures have not the same y-scale and it can be confusing and deceptive!