text:
How do you program fear into a neural network?
If you've been attacked by a spider once, chances are you'll never go near a spider again.
In a neural network model, having a bad experience with a spider will slightly decrease the probability you will go near a spider depending on the learning rate. This is not good.
How can you program fear into a neural network, such that you don't need hundreds of examples of being bitten by a spider in order to ignore the spider (and also that it doesn't just lower the probability that you will choose to go near a spider)?

WP article One-shoot learning https://en.wikipedia.org/wiki/One-shot_learning includes a section about learning with one example.

There are a lot of approaches you could take for this. Creating a realistic artificial analog for fear as implemented biologically in animals might be possible, but there is quite a lot involved in a real animal's fear response that would not apply in simpler AI bots available now. For instance, an animal entering a state of fear will typically use hormones to signal changes throughout its body, favouring resource expenditure and risk taking ("fight or flight").
In basic reinforcement learning, the neural network would not need to directly decide switch on a "fear mode". Instead, you can make use of some design in the agent and learning algorithm to help learn from rare but significant events. Here are a few ideas:

Experience replay. You may already be doing this in the Pacman scenario, if you are using DQN or something similar. Storing the state transition and reward that caused a large positive or negative reward, and repeatedly learning from it should offset your concern
Prioritised sweeping. You can use larger differences experienced between predicted and actual reward to bias sampling from your replay memory towards significant events and those linked closely to them.
Planning. With a predictive model - maybe based on sampled transitions (you can re-use the experience replay memory for this), or maybe a trained state transition prediction network - then you can look multiple steps ahead by simulating. There is a strong relation between RL and look-ahead planning too, they are very similar algorithm. The difference is which states and actions are being considered, and whether they are being simulated or experienced. Experience replay blurs the line here - it can be framed as learning from memory, or improving predictions for planning. Planning helps by optimising decisions without needing to repeat experiences as much - a combination of planning and learning can be far more powerful than either in isolation.
Smarter exploratory action selection. Epsilon-greedy, where you either take a greedy action or take a completely random action, completely ignores how much you may have already learned about alternative actions and their relative merit. You can use something like Upper Confidence Bound with a value-based agent.
In a deterministic world, increase the batch size for learning and planning, as you can trust that when a transition is learned once, that you know everything about it.

You will need to experiment in each environment. You can make learning agents that are more conservative about exploring near low reward areas. However, if the environment is such that it is necessary to take risks in order to get to the best rewards (which is often the case in games) then it may not be optimal in terms of learning time to have a "timid" agent. For instance in your example of Pacman, sometimes the ghosts should be avoided, sometimes they should be chased. If the agent learned strong aversion initially, it might take a long time to overcome this and learn to chase them after eating a power-up.
For your example of the spider, as the constructor of the experiment then you know that the bite is bad every time and that the agent must avoid it as much as possible. To most RL algorithms, there is no such knowledge, except gained through experience. An MDP world model does not need to match common sense, it may be that a spider bite is bad (-10 reward) 90% of the time and good 10% of the time (+1000 reward). The agent can only discover this by being bitten multiple times . . . RL typically does not start with any system to make assumptions about this sort of thing, and it is impossible to come up with a general rule about all possible MDPs. Instead, for a basic RL system, you can consider modifying hyperparameters or focusing on key events as suggested above. Outside of a basic RL system there could be merit in replicating other things, such as "instinctive" fear.

@DuttaA: I agree, which is why I suggest a bunch of stuff that is not "real fear (tm)". I think a very basic "instinctive fear" using RL would involve somehow adding a prior low value into the value function programmatically without actual experience.

I think there are 2 ways to make this happen: 1) explicitly program fear as a constraint or parameter in some logical expression, or 2) utilize a large set of training data to teach fear.
Think about a basic Pacman game-- whether Pacman fears the ghosts or doesn't fear them is hard to tell, but they ARE ghosts and Pacman avoids them so I think it's safe we can use this as a basic example of "fear". Since, in this game, fear = avoidance, you could logically program avoidance to be some sort of distance. I tried this with Pacman reinforcement learning. I tried to set a distance of 5 squares to the ghosts and anytime Pacman could see a ghost within 5 squares, he would move in a different direction. What I found is that while Pacman will try to avoid ghosts, he doesn't know strategy (or have intelligence). Pacman would simply move away from ghosts until he got boxed in.
My point is that you can program your network to avoid spiders to not get bit, but without training, you will just be creating a basic parameter that might cause problems if there are 100 super aggressive spiders coming at you! The better way is to use some base logic to avoid spiders, but then train the network to be rewarded the better spiders are avoided.
Now, there are many situations of fear so this one example with Pacman would not necessarily apply to all... Just trying to give some insight in my experience with teaching fear with reinforcement learning in Pacman.

I'm thinking that for some things like fire it needs to be an instinct. Because by the time you've had a few "training examples" you'd be pretty burned. But with other things you should only need one example like getting bit by a hamster you should learn hamsters have sharp teeth so don't put your hand in their mouth. So for some things instincts should just prevent you from doing things like putting your hand in a fire. Or make you jump back. But should you also be scared of going near fire if you learn that you might get hit by a spark?

@zooby in my experience little kids don't really learn to fear the hot stove or fire until they do get burned.  (s'why you've got to keep such a close eye on them!)  I'd say that if the algorithm is learning avoidance through a technique like reinforcement learning, it is "learning what to fear", whereas, if it has the avoidance mechanism pre-programmed, that would be "instinct".

Fear of this kind is an irrational response (large negative incentive in response to a small risk).  Modeling fear would need to model a "grossness" factor associated with, for example, spiders so that the normally un-proportional response would occur.  The "grossness" factor could be manifested in many other forms to magnify a response to a previously unpleasant, though not particularly dangerous, experience.  Such fear can also be inspired by hearsay (think hysteria caused by a sensational news story).
A NN would normally only respond minimally to a minimal risk.

I would suggest having the agent weight its learning from a given event based on the severity of the consequences for that event happening. Eg. Have it develop a threat model like those typically drafted up in the Information Security field. High risk but low probability is something that can be accounted for and judged against.
Trying to directly imitate human fear would be silly, you'd likely end up with AIs that have phobias if you succeeded too well.


text:
How do we calculate the hidden units values in a (restricted) Boltzmann machine?
Now, Boltzmann machines are energy-based undirected networks, meaning there are no forward computations. Instead, for each input configuration $x$, a scalar energy is calculated to asses this configuration. The higher the energy, the less likely for $x$ to be sampled from the target distribution.
The probability distribution is defined through the energy function by summing over all possible states of the hidden part $h$.
If I understand correctly, the hidden units are added to capture higher-order interactions, which offers more capacity to the model.
So, how do we calculate the values of these hidden units? Or do we not explicitly compute these values and instead approximate the marginal "free energy" (which is the negative log of the sum over all possible states of $h$)?

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What are examples of simple problems and applications that can be solved with AI techniques?
What are examples of simple problems and applications that can be solved with AI techniques, for a beginner who is trying to make use of his basic programming skills into AI at the beginning level?

This is fairly boilerplate advice, but, since you're brand new to AI, I'd personally suggest writing a classical Tic-Tac-Toe AI, ideally using minimax.
I suggest this because minimax is fundamental to AI, and there are many webpages devoted to this subject, such as How to make your Tic Tac Toe game unbeatable by using the minimax algorithm and Tic Tac Toe: Understanding the Minimax Algorithm.  (Google search for "Tic-tac-toe" and "minimax" will yield a plethora of other sites.  I'd also recommend looking at this minimax page from Stanford: "Strategies and Tactics for Intelligent Search".)
I recommend this approach as a good basic primer.  The real cutting-edge work is being done in Machine Learning and Neural Networks, and for that reason, it's probably more important than ever to have some basic grounding in classical AI before you start dipping your toe in that pond.

I will assume you talk about applied AI (in generalized/strong AI we have nothing yet to program :-).
You can look at any university course of introduction to AI and see its chapters and the program examples they use ( start by programming without any theory is not a way ).
By example, one common issue on this kind of courses is path finding, using for it algorithms as A* algorithm and applying them to games as Hanoi Towers. This kind of knowledge is a must for any activity in AI.
Standford link provided by @DukeZhou is a good example of one of theses courses, just I suggest start it from first chapter instead of go directly to minmax.
Later on, you can jump to more advanced concepts, as recognition/classification and its common approaches: k-nearest/k-means, decision networks, neural nets, ... .

Tower of Hanoi is a great suggestion.  Props!

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
In novelty search, are the novel structures or behaviour of the neural network rewarded?
I have been reading a lot lately about some very promising work coming out of Uber's AI Labs using mutation algorithms enhanced with novelty search to evolve deep neural nets. See the paper Safe Mutations for Deep and Recurrent Neural Networks through Output Gradients (2018) for more details.
In novelty search, are the novel structures or behavior of the neural network rewarded?

Hello all.  I have been researching this evening and found that both approaches can be combined to create an optimal solution.  I'm still unclear on 'safe mutation' so am not answering my own question but an leaving a paper here that describes using novelty of both structure and behavior as a fitness function:  https://www.researchgate.net/publication/262373008_Encouraging_Creative_Thinking_in_Robots_Improves_Their_Ability_to_Solve_Challenging_Problems

This is from the paper: " Novel individuals are thus determined based on their behavioral distance to current or previously seen individuals. The GA otherwise proceeds as normal, substituting novelty for ﬁtness (reward)." - taken from here: https://arxiv.org/pdf/1712.06567.pdf

safe mutations and novelty search arent dependent on one another ie you could use safe mutations without novelty search and novelty search without safe mutations. If i recall correctly safe mutations runs sgd optimization during evaluation and uses the optimized net's fitness to represent the genome. neat essentially has a search for optimal structure built in and mutations of the structures are driven by config vars so i would suggest doing behavior novelty as searching for novel structures would involve changing the underlying evolution process a substantially.

In the paper Exploiting Open-Endedness to Solve Problems Through the Search for Novelty (2008), by Joel Lehman and Kenneth O. Stanley, which introduced the novelty search approach, it is written

Thus this paper introduces the novelty search algorithm, which searches with no objective other than continually finding novel behaviors in the search space.

and

instead of searching for a final objective, the learning method is rewarded for finding any instance whose functionality is significantly different from what has been discovered before

and

The novelty of a newly generated individual is computed with respect to the behaviors (i.e. not the genotypes) of an archive of past individuals whose behaviors were highly novel when they originated

Therefore, the goal of a novelty search is to search for novel behavior and not necessarily novel chromosomes (or genotypes).
In the experiments reported in the novelty search paper, the authors use neural networks to represent the policy that controls a robot that needs to navigate a maze, while evolving these neural networks with NEAT (a neuroevolution method) and a novelty metric (rather than a fitness metric, which is used in the original NEAT). In the same experiments section, Lehman and Stanley write

Thus, for the maze domain, the behavior of a navigator is defined as its ending position. The novelty metric is then the Euclidean distance between the ending positions of two individuals. For example, two robots stuck in the same corner appear similar, while one robot that simply sits at the start position looks very different from one that reaches the goal, though they are both equally viable to the novelty metric.

Therefore, the evolution of the neural networks, which represent the controllers, is not necessarily guided by the novelty of (the architecture of) the neural networks but by the novelty of the behavior generated by the neural networks, even though novel neural networks might correspond or lead to novel behaviors.

Adding to this just a tad, in the implementation, for determining the novelty of an individual in a population ken states on the users page that an average (euclidean, as nbro said) distance from the k-nearest neighbors has proven to be an effective measure.

@nickw Yes, in the paper, they also mention and use this k-nearest neighbor approach.


text:
How can people contribute to AGI research?
Is there a way for people outside of the core research community of AGI to contribute to the cause?
There are a lot of people interested in supporting the field, but there is no clear way to do that. Is there something like BOINC for AGI researches, or open projects where random experts can provide some input? Maybe Kickstarter for AGI projects?

OpenCog is an open source AGI project. But it is is also incredibly complex and IMHO not a good idea (I have not fully read his theories). You can learn the essential ideas behind OpenCog from the co-founder Ben Goertzel site as well.
Or, you can participate in the philosophical discussion regarding AGI. For strictly AGI, decision theory, logic, and math material (they are all related), you can look up stuff from http://yudkowsky.net/ or https://arbital.com/. But, in some sense, every branch of philosophical inquiry can be tied back to AGI and consciousness (ethics, metaphysics, etc.), so if it fancies you it depends on how you'd like to tackle it.
You could also study the psychology end of things. The following papers and related ideas are quite important in the field of study of consciousness and cognition (but keep in mind this is pretty much a random list, the literature is massive!):

Global Workspace: A Theory of Consciousness (1997)
Relevance Realization and the Emerging Framework in Cognitive Science (2009)
The free-energy principle: a unified brain theory? (2010)

Recently, (in mathematical time), progress in category theory shows promise of being a unified framework of much of the existing math. I know next to nothing about it, but the people that do are applying to many new fields of study (including AI, apparently). Category theory requires a lot of background mathematical knowledge before its vocabulary begins to make sense, though, so beware. You can read about it on the nCatLab and occasionally on John Baez's blog: Azimuth
Of course, "regular" techniques in Machine Learning such as neural networks, reinforcement learning, statistical methods, and others are very powerful as well, but due to certain regards in their construction, they are generally understood as only being capable of being "narrow AI" in the sense that they can only complete a single task very well, but perhaps you can find some research that changes this?

You can contribute to AGI research in several ways.

Write papers for the Artificial General Intelligence conference, which is peer-reviewed, or other AGI conferences or journals, or just submit them to e.g. Arxiv. For example, you can write papers about AGI algorithms or architectures.
Contribute to (open source) AGI systems, like OpenCog, BECCA or OpenNARS.
Implement AGI algorithms, mechanisms or architectures, and put them, for example, on Github.


text:
What is the current state of AGI development?
Could you please provide some insight into the current stage of developments in AGI area? Are there any projects that had breakthroughs recently? Maybe some news source to follow on this topic?

The state of AGI research is pursuing the few problems that we have been able to break off from the gigantic research problem. These are terms which can be more thoroughly looked into.
A few of the main focuses are:

One Shot learning - You know how a person can sometimes learn to do something by something by seeing literally 1 example of it? Well current learning methods on the whole are not able to accomplish this to the extent that we can easily take for granted. Work is being done to find ways to approach this feat of learning and it’s on its way to becoming much more influential.
Transfer Learning - If you have ever played a side scroller like Mario, and then I gave you a slightly different game like Sonic, odds are you’d could learn to play Sonic faster than it took you to learn to play Mario. This is because of learning “savings” you have by transferring your Mario knowledge to the new Sonic domain. This is a much more popular research arm than one-shot, probably because it’s easier to think about but also because there have been promising results of pretraining a network on one set of data and focusing it an another task.
Creativity/curiosity - Although one could say that GANs have really changed how humans can be more creative, it is difficult to quantify curiosity and creativity This paper gives an okay overview. Moreover, allowing an agent to take the chances and make some mistakes as is the nature of creativity concerns many people who are focused on AI Safety.
Understanding concepts - This is subtle but very very important. Current AI methodologies struggle with imbuing AI with the ability to have concepts. By concepts, I don’t mean “it kinda looks like this neuron in the second last layer is sensitive to tires”. I mean that understanding what a tire looks like is just a small part of understanding what a tire is, what it is used for, what it affords someone to do etc. This research direction is in it’s infancy but will be much more influential as more theories and ideas are brought forward.

Despite the progress made in these fields and in the many other areas in AI, there is still much to be done and understood before we can finally have s̶k̶y̶n̶e̶t̶ Wall-E.

The last paragraph has it ;today,systems lack a sufficient scope of memory or processing power. In addition, the level of complexity that connects to the process of AGI research has also limit the progress of AGI research.

Could one (do people already) not argue that one-shot learning is actually just a special case of transfer learning? I can't think of an example where a human learns something after observing a single example, without using knowledge from related tasks (perhaps defining the relation itself would require some thinking). For me, that just means transfer learning with as few new observations as possible. From your linked definition: The key motivation for the one-shot learning technique is that systems, like humans, can use prior knowledge about object categories to classify new objects.

I agree with you, it is a special case. I do think that the average case of Transfer Learning differs from One Shot learning in that the "shot" may be multiple data points or may vary across time which is an important distinction.

In the paper Artificial General Intelligence: Concept, State of the Art, and Future Prospects (2014) Ben Goertzel gives an overview of the AGI field and its progress. He describes the main approaches

symbolic (e.g. cognitive architectures, like SOAR)
emergentist/subsymbolic (neural networks)
hybrid (combination of symbolic and emergentist)
universalist (AIXI and Godel Machines)

to AGI and metrics to assess human-level intelligence and partial progress.
There's also an older book by Cassio Pennachin and Ben Goertzel called Artificial General Intelligence (2007). A chapter of the book, Contemporary Approaches to Artificial General Intelligence, gives a brief history of the AGI field.


text:
What is the intuition behind how word embeddings bring information to a neural network?
How is it that a word embedding layer (say word2vec) brings more insights to the neural network compared to a simple one-hot encoded layer?
I understand how the word embedding carries some semantic meaning, but it seems that this information would get "squashed" by the activation function, leaving only a scalar value and as many different vectors could yield the same result, I would guess that the information is more or less lost.
Could anyone bring me insights as to why a neural network may utilize the information contained in a word embedding?

Shakespeare once said "A rose by any other name would smell as sweet" (Romeo and Juliet). Words are just labels we attach to ideas for convenience. By using one hot we remain tied to the letter sequence r,o,s,e, and some other structure must take on the responsibility of attaching the context of sweetness to it.
Word embeddings learn a multi-dimensional context. What exactly the context is of each dimension of the embedding is something of a mystery and simply emerges from the learning. The larger the number of dimensions, the greater the possibility that some combination of the dimensions will represent the sweetness context, but it might be quite hard to tease out.
So you can attach the idea of sweetness to one member of a one-hot structure, but it must necessarily be a part of a rules-based approach. Embeddings, when they are working well, will not need the rules.

Adding to Colin's answer; using word embedding tend to be much more robust that one-hot vectors. Consider the the following two sentences:

The desk has a book on it.

and

The table has a book on it.

These two sentences are almost identical in meaning. If we were to using word embeddings, the vectors 'desk' and 'table' would be very close together. The fact that these two sentences are similar becomes implicit with embeddings.
But if we were to use one-hot vectors, the distance between the two vectors would be the same distance between 'desk' and 'cat' or 'table' and 'book'. So now the network must learn that these sentences may entail the same thing on top of the original task.


text:
How can certain information about the goal be given to the RL learning algorithm?
Policy learning refers to mapping an agent state onto an action to maximize reward. A linear policy, such as the one used in the Augmented Random Search paper, refers to learning a linear mapping between state and reward.
When the entire state changes at each time-step, for example in the Continuous Mountain Car OpenAI Gym, the position and speed of the car changes at each time-step.
However, assume we also wanted to communicate the constant position of one or more goals. By "constant", I mean does not change within a training episode, but may change between episodes. For example, if there was a goal on the left and right of the Mountain Car.
Are there examples of how this constant/static information be communicated from the environment other than appending the location of the two goals to the state vector? Can static/constant state be differentiated from state which changes with each action?

I see three main ways to do this, which one makes more sense will depend on your application.
One is to append that information to the state/observations like you mentioned. While this information is static for a particular episode, it will be different across episodes and the policy should learn to condition the actions it chooses based on this information.
Another would be to leave goal information out entirely and force the agent to learn a policy that works when the goal is unknown. This will likely be more difficult to learn and you may end up with a policy that moves to the average of all goals or explores and tries them all.
A third option and probably the most natural is to have some context cue that the agent can observe (e.g. often in lab experiments with rats, a cue is placed on a wall that lets the rat know which way to go to get a reward). This is similar to the first method, except that the cue has to be observed rather than given directly. For the Mountain Car example, this could be an extra signal that the agent only sees in a particular location, such as the bottom of the valley or when it moves close to a particular side.


text:
How large should the replay buffer be?
I'm learning DDPG algorithm by following the following link: Open AI Spinning Up document on DDPG, where it is written

In order for the algorithm to have stable behavior, the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything.

What does this mean? Is it related to the tuning of the parameter of the batch size in the algorithm?

You need to read this 2020 paper by Deepmind:
"Revisiting Fundamentals of Experience Replay"
They explicitly test the size of the experience replay, the replay-ratio of each experience and other parameters.
Also, to add to the answer by @nbro
Assume you implement experience replay as a buffer where the newest memory is stored instead of the oldest. Then, if your buffer contains 100k entries, any memory will remain there for exactly 100k iterations.
Such a buffer is simply a way to "see" what was up to 100k iterations ago.
After the first 100k iterations you fill the buffer and begin "moving" it, much like a sliding window, by inserting new memories instead of the oldest.

The size of the buffer (relative to the total number of iterations you plan to ever train with) depends on "how much you believe your network architecture is susceptible to catastrophic forgetting".
A tiny buffer might force your network to only care about what it saw recently.
But an excessively large buffer might take a long time to "become refreshed" with good trajectories, when they finally start to be discovered. So the network would be like a university student whose book shelf is diluted with first-grade school books.
The student might have already decided that he/she wishes to become a programmer, so re-reading those primary school books has little benefit (time could have been spent more productively on programming literature) + it takes a long time to replace those with relevant university books.

Good point. Just to add to why this is important: this is important because old experience was generated by an old policy (i.e. your data distribution consists of an older distribution over state-action pairs / trajectories), and because any stored value estimates may be old. Which one of those two points matters depends on whether we're talking about value-based or policy gradient methods, and whether we're talking about on-policy or off-policy learning... but usually at least one of those two is relevant

@DennisSoemers: Generally you should not store value estimates in experience replay, but re-calculate values when using the trajectory information (s,a,r,s'). No doubt there are implementations which store value estimates for some valid reason, but "classic" DQN does not

@NeilSlater Yeah you're right, I guess you shouldn't in most (all?) of the classic RL algorithms. I've been working a lot with AlphaZero-style self-playing training the past year, and there we do store value estimates as computed by MCTS... so I guess that's why I've got it stuck in my mind :)

No wait that's wrong too, in standard AlphaZero-style training we store visit counts, but in one of my recent papers I changed that to storing MCTS' value estimates...

In order for the algorithm to have stable behavior, the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything.

The larger the experience replay, the less likely you will sample correlated elements, hence the more stable the training of the NN will be. However, a large experience replay also requires a lot of memory and it might slow training. So, there is a trade-off between training stability (of the NN) and memory requirements.
The authors of the linked article state (right after the sentence above)

If you only use the very-most recent data, you will overfit to that and things will break; if you use too much experience, you may slow down your learning. This may take some tuning to get right.


text:
Is max-pooling really bad?
Hinton doesn't believe in the pooling operation (video). I also heard that many max-pooling layers have been replaced by convolutional layers in recent years, is that true?

In addition to JCP's answer I would like to add some more detail. At best, max pooling is a less than optimal method to reduce feature matrix complexity and therefore over/under fitting and improve model generalization(for translation invariant classes).
However as JCP begins to hit on.. there are problems with this method. Hinton perhaps sums the issues in his talk here on what is wrong with CNNs. This also serves as motivation for his novel architecture capsule networks or just capsules.
As he talks about, the main problem is not translational variance per se but rather pose variance. CNNs with max pooling are more than capable of handling simple transformations like flips or rotation without too much trouble. The problem comes with complicated transforms, as features learned about a chair facing forwards, will not be too helpful towards class representation if the real-world examples contain chairs upside down, to the side, etc.
However there is much work being done here, mostly constrained to 2 areas. Those being, novel architectures/methods and inference of the 3d structure from images(via CNN tweaks). This problem was one of the bigger motivators for researchers throughout the decades, even David Marr with his primal sketches.

Max pooling isn't bad, it just depends of what are you using the convnet for. For example if you are analyzing objects and the position of the object is important you shouldn't use it because the translational variance; if you just need to detect an object, it could help reducing the size of the matrix you are passing to the next convolutional layer. So it's up to the application you are going to use your CNN.


text:
Does everyone still use discount rates?
In Section 10.4 of Sutton and Barto's RL book, they argue that the discount rate $\gamma$ has no effect in continuing settings. They show (at least for one objective function) that the average of the discounted return is proportional to the undiscounted average reward $r(\pi)$ under the given policy.$^*$ They then advocate using average rewards rather than the usual returns of the discounted setting.
I've never encountered someone using average rewards (and no discounting) in the wild, though. Am I just ignorant of some use case, or is pretty much everyone sticking to discounting anyways?
$$r(\pi)=\sum_s \mu_\pi (s) \sum_a \pi(a|s) \sum_{s',r}p(s',r|s,a)r$$
$\mu_\pi$ is the stationary state distribution while following policy $\pi$.
$^*$Their proof did use the fact that the MDP was ergodic. I'm not sure how often that assumption holds in practice.

I have never actually seen a continuing task formulation used "in the wild". Possibly that explains something: episodic tasks are a good enough framework for most real world RL problems?


text:
Is any classifier not subject (or less susceptible) to fooling?
Is any classifier not subject to fooling as in here?
My question is related to this, but not an exact duplicate.
What I wanted to ask is that any classifiers inherently do not subject (or less prone) to attack. I have a feeling that non-linear classifiers should be less susceptible to attack.

I would say this is not necessarily a duplicate but quite similar to some other questions. However, I will answer the question posed here.
At a theoretical level, what you are asking is there any algorithm that cannot be tricked into predicting the wrong class?
The answer is: No
It is analogous to asking whether there is a perfect architecture for an arbitrary classification problem, and that answer is quite obviously not. It would also not likely to be terribly difficult to show this is provably the case, at least for a class of algorithms(i.e connectionist models).


text:
How to build a neural network that can learn to predict output images?
I am working with a dataset where each input sample is a matrix, and the output corresponding to each input is also a matrix (of shape (400, 10)). The input samples do not have translation invariance. Each output image has shape (16, 16) . The output matrices have translation invariance.
I want to build a neural network which can learn how to predict the output images from the aforementioned data. It seems to me that one needs to think of a neural network here which does regression on the output images to learn. Presently, I am using 1000 data samples for learning in the neural network (input samples and corresponding output images). What is the best way to build a neural network for this task?
Presently, I am using multi-layer perceptron (MLP) with mean square error (MSE) loss for this task. I flatten the input matrices before I feed them into the MLP, and use a standard MLP with multiple hidden layers (4-5) with many hidden units for this task.
While a visual inspection shows that the true and predicted output images are in relatively good agreement for training data, I find a mismatch between true and predicted (reconstructed) output images for validation data. The bottom plot shows pictures of true and predicted (reconstructed) output images for training and validation data for chosen samples.

Presently, I am using training and validation loss curves (with respect to iteration) to measure performance. I want to have a robust metric for comparison which can tell me whether the prediction is a random image or not.
How can I get the model to generalize to the validation set better?
The Python code that I am using for this MLP and the required data can be found here and here respectively.

Hi. Welcome to AI.SE. Your post is more likely to get responded to if it consists of a single question. Also, implementation questions are off-topic here. This post has the best chance of not being closed as off-topic if you focus on conceptual questions. For more information, see these help pages.

Hi Philip, thanks for your suggestion. I have edited the question after your suggestions. I hope it is fine now :)

By saying that the input samples do not have translation invariance, do you mean that they are not images? You also say "the output corresponding to each input is also a matrix (of shape (400, 10))", but then you say "Each output image has shape (16, 16)", so what is the difference between the output corresponding to each input sample and the output image (given that they have different dimensions)? I think you should edit your question to clarify this point (at least it is not clear to me).


text:
Which algorithm should I use to map an input sentence to an output sentence?
I am new to NLP realm. If you have an input text "The price of orange has increased" and output text "Increase the production of orange". Can we make our RNN model to predict the output text? Or what algorithm should I use?

2 ways really, both involve converting the input to a 1-hot input. Either convert each character to a 1 hot input (eg, of size 27, 26 in the alphabet, 1 for spaces), or convert each word to a 1 hot vector of the RNN's vocabulary,

@Recessive Old comment, I know, but: No, those are not the only ways. Another method that is extremely common is to segment text into subwords. Also, you are implying that this tasks needs to be solved with an RNN - which is not the case.

@MathiasMüller I fail to see where I implied only an RNN could be used, I only responded to the question on how to input the text to an RNN. When you say segment the text into subwords, would those subwords not be encoded in a 1-hot vector? How do you propose inputting ascii characters into a network that deosn't involve 1-hot encoding?

@Recessive "I fail to see where I implied only an RNN could be used" - you are right, I'm sorry about this! "How do you propose inputting ascii characters into a network that deosn't involve 1-hot encoding?" - my view is as follows: all you have to do is map characters/words/subwords to vocabulary indexes. Then those indexes can be replaced with their word embeddings, without constructing one-hot vectors.

@MathiasMüller From what I understand, you're saying that the input and output should be an individual number representing the index in the vocabulary? In that case the network would have to use regression to learn, which in my experience has always performed poorly compared to an equivalent solution using classification

@Recessive I am saying that in most NLP tasks, RNNs are not reading text directly, but get to see embeddings as inputs. For instance, the inputs to an RNN word-level language model are embeddings. The outputs are distributions over the vocabulary. See here for an example: https://github.com/pytorch/examples/tree/master/word_language_model.

In your case you can use either RNN (especially BiLSTM with ELMo and attention mechanism for better accuracy) or Transformer based architectures (the best of them today is BERT). But for both cases you need data to train the model (i.e sequences of input/output like in your question).
I believe the best choice in your case is BERT as it's achieving state of the art performance in most NLP tasks and is already pretrained so you don't need a massive data to retrain the model. Also, BERT is pretrained on "Next Sentence Prediction" and allows "2 separate sentences" as input which helps a lot in your case. The only drawback, compared to other methods, is that you need fine-tuning the model so it's not a completly "ready to use" model.
For more information:
Here is the github repository of BERT: BERT-repo. For quick explanation check: The Illustrated BERT. For detailed explanation see BERT paper: BERT-paper.

sounds like a job for Sequence-to-sequence or seq2seq
for example tf-seq2seq
you basically use an RNN as an encoder (reduces and sequence of words to a vector), and another RNN as a decoder (takes the encoded vector as in input to generation a new sequence of words).


text:
Why doesn't Q-learning converge when using function approximation?
The tabular Q-learning algorithm is guaranteed to find the optimal $Q$ function, $Q^*$, provided the following conditions (the Robbins-Monro conditions) regarding the learning rate are satisfied

$\sum_{t} \alpha_t(s, a) = \infty$
$\sum_{t} \alpha_t^2(s, a) < \infty$

where $\alpha_t(s, a)$ means the learning rate used when updating the $Q$ value associated with state $s$ and action $a$ at time time step $t$, where $0 \leq  \alpha_t(s, a) < 1$ is assumed to be true, for all states $s$ and actions $a$.
Apparently, given that $0 \leq  \alpha_t(s, a) < 1$, in order for the two conditions to be true, all state-action pairs must be visited infinitely often: this is also stated in the book Reinforcement Learning: An Introduction, apart from the fact that this should be widely known and it is the rationale behind the usage of the $\epsilon$-greedy policy (or similar policies) during training.
A complete proof that shows that $Q$-learning finds the optimal $Q$ function can be found in the paper Convergence of Q-learning: A Simple Proof (by Francisco S. Melo). He uses concepts like contraction mapping in order to define the optimal $Q$ function (see also What is the Bellman operator in reinforcement learning?), which is a fixed point of this contraction operator. He also uses a theorem (n. 2) regarding the random process that converges to $0$, given a few assumptions. (The proof might not be easy to follow if you are not a math guy.)
If a neural network is used to represent the $Q$ function, do the convergence guarantees of $Q$-learning still hold? Why does (or not) Q-learning converge when using function approximation? Is there a formal proof of such non-convergence of $Q$-learning using function approximation?
I am looking for different types of answers, from those that give just the intuition behind the non-convergence of $Q$-learning when using function approximation to those that provide a formal proof (or a link to a paper with a formal proof).

Great question!

The book that you referenced talks about this problem in chapter 11 so you might read it. Also, I don't think there's a formal proof why this happens but there are few examples that show divergence even in simple environments (e.g. Tsitsiklis and van Roy) .

Here's an intuitive description answer:
Function approximation can be done with any parameterizable function. Consider the problem of a $Q(s,a)$ space where $s$ is the positive reals, $a$ is $0$ or $1$, and the true Q-function is $Q(s, 0) = s^2$, and $Q(s, 1)= 2s^2$, for all states. If your function approximator is $Q(s, a) = m*s + n*a + b$, there exists no parameters which can accurately represent the true $Q$ function (we're trying to fit a line to a quadratic function). Consequently, even if you chose a good learning rate, and visit all states infinitely often, your approximation function will never converge to the true $Q$ function.
And here's a bit more detail:

Neural networks approximate functions. A function can be approximated to greater or lesser degrees by using more or less complex polynomials to approximate it. If you're familiar with Taylor Series approximation, this idea should seem pretty natural. If not, think about a function like a sine-wave over the interval [0-$\pi/2$). You can approximate it (badly) with a straight line. You can approximate it better with a quadratic curve. By increasing the degree of the polynomial we use to approximate the curve, we can get something that fits the curve more and more closely.
Neural networks are universal function approximators. This means that, if you have a function, you can also make a neural network that is deep or wide enough that it can approximate the function you have created to an arbitrarily precise degree. However, any specific network topology you pick will be unable to learn all functions, unless it is infinitely wide or infinitely deep. This is analogous to how, if you pick the right parameters, a line can fit any two points, but not any 3 points. If you pick a network that is of a certain finite width or depth, I can always construct a function that needs a few more neurons to fit properly.
Q-learning's bounds hold only when the representation of the Q-function is exact. To see why, suppose that you chose to approximate your Q-function with a linear interpolation. If the true function can take any shape at all, then clearly the error in our interpolation can be made unboundedly large simply by constructing a XOR-like Q-function function, and no amount of extra time or data will allow us to reduce this error. If you use a function approximator, and the true function you try to fit is not something that the function can approximate arbitrarily well, then your model will not converge properly, even with a well-chosen learning rate and exploration rate. Using the terminology of computational learning theory, we might say that the convergence proofs for Q-learning have implicitly assumed that the true Q-function is a member of the hypothesis space from which you will select your model.

Where can we see from the proof I mentioned that "Q-learning's bounds hold only when the representation of the Q-function is exact" is true?

So, we can approximate any (reasonable) function using some neural network (architecture), but, given a fixed neural network architecture $Z$ (which we need to choose at the beginning of the training phase of the $Q$-learning), $Q$-learning might not converge using that specific architecture $Z$, because $Z$ might not be expressive enough to represent $Q^*$.

@nbro The proof doesn't say that explicitly, but it assumes an exact representation of the Q-function (that is, that exact values are computed and stored for every state/action pair). For infinite state spaces, it's clear that this exact representation can be infinitely large in the worst case (simple example: let Q(s,a) = sth digit of pi). Your second comment sums it up well. More formally, if the true hypothesis Q* is not an element of the hypothesis space H that you are selecting a model from, you cannot converge to Q*, even with infinite time or data.

You are correct that in a non-realisable setting (when the true function is not in the hypothesis class) one cannot learn the true function, but this does not mean that the learning does not converge. This is different from divergence. For instance, choose the hypothesis class containing only the 0 function.

@JohnDoucette, thanks for the answer. I learnt a lot. I have more problem on this. Can you please help me out: https://ai.stackexchange.com/questions/37440/what-exactly-is-non-delusional-q-learning

As far as I'm aware, it is still somewhat of an open problem to get a really clear, formal understanding of exactly why / when we get a lack of convergence -- or, worse, sometimes a danger of divergence. It is typically attributed to the "deadly triad" (see 11.3 of the second edition of Sutton and Barto's book), the combination of:

Function approximation, AND
Bootstrapping (using our own value estimates in the computation of our training targets, as done by $Q$-learning), AND
Off-policy training ($Q$-learning is indeed off-policy).

That only gives us a (possibly non-exhaustive) description of cases in which we have a lack of convergence and/or a danger of divergence, but still doesn't tell us why it happens in those cases.

John's answer already provides the intuition that part of the problem is simply that the use of function approximation can easily lead to situations where your function approximator isn't powerful enough to represent the true $Q^*$ function, there may always be approximation errors that are impossible to get rid of without switching to a different function approximator.
Personally, I think this intuition does help to understand why the algorithm cannot guarantee convergence to the optimal solution, but I'd still intuitively expect it to maybe be capable of "converging" to some "stable" solution that is the best possible approximation given the restrictions inherent in the chosen function representation. Indeed, this is what we observe in practice when we switch to on-policy training (e.g. Sarsa), at least in the case with linear function approximators.

My own intuition with respect to this question has generally been that an important source of the problem is generalisation. In the tabular setting, we have completely isolated entries $Q(s, a)$ for all $(s, a)$ pairs. Whenever we update our estimate for one entry, it leaves all other entries unmodified (at least initially -- there may be some effects on other entries in future updates due to bootstrapping in the update rule). Update rules for algorithms like $Q$-learning and Sarsa may sometimes update towards the "wrong" direction if we get "unlucky", but in expectation, they generally update towards the correct "direction". Intuitively, this means that, in the tabular setting, in expectation we will slowly, gradually fix any mistakes in any entries in isolation, without possibly harming other entries.
With function approximation, when we update our $Q(s, a)$ estimate for one $(s, a)$ pair, it can potentially also affect all of our other estimates for all other state-action pairs. Intuitively, this means that we no longer have the nice isolation of entries as in the tabular setting, and "fixing" mistakes in one entry may have a risk of adding new mistakes to other entries. However, like John's answer, this whole intuition would really also apply to on-policy algorithms, so it still doesn't explain what's special about $Q$-learning (and other off-policy approaches).

A very interesting recent paper on this topic is Non-delusional Q-learning and Value Iteration. They point out a problem of "delusional bias" in algorithms that combine function approximation with update rules involving a $\max$ operator, such as Q-learning (it's probably not unique to the $\max$ operator, but probably applies to off-policy in general?).
The problem is as follows. Suppose we run this $Q$-learning update for a state-action pair $(s, a)$:
$$Q(s, a) \gets Q(s, a) + \alpha \left[ \max_{a'} Q(s', a') - Q(s, a) \right].$$
The value estimate $\max_{a'} Q(s', a')$ used here is based on the assumption that we execute a policy that is greedy with respect to older versions of our $Q$ estimates over a -- possibly very long -- trajectory. As already discussed in some of the previous answers, our function approximator has a limited representational capacity, and updates to one state-action pair may affect value estimates for other state-action pairs. This means that, after triggering our update to $Q(s, a)$, our function approximator may no longer be able to simultaneously express the policy that leads to the high returns that our $\max_{a'} Q(s', a')$ estimate was based on. The authors of this paper say that the algorithm is "delusional". It performs an update under the assumption that, down the line, it can still obtain large returns, but it may no longer actually be powerful enough to obtain those returns with the new version of the function approximator's parameters.

Finally, another (even more recent) paper that I suspect is relevant to this question is Diagnosing Bottlenecks in Deep Q-learning Algorithms, but unfortunately I have not yet had the time to read it in sufficient detail and adequately summarise it.

But isn't the use of a neural network also due to the assumption that certain states are very similar to each? Very similar states (e.g. successive frames in a game) often have very similar (or same) optimal actions, so I am not sure that explanation in the first paper is valid (I should read it to fully understand their main points).

@nbro Yeah, often generalisation is considered to be an advantage rather than a problem precisely because of that reason. If it works out as "intended", it can be very powerful and speed up learning because we transfer whatever we learn to similar states / similar actions, rather than learning for every slightly different state/action in isolation. But it can also lead to problems, especially in theory but also in practice. It's like a "double-edged sword" I suppose.

@DennisSoemers Super interesting answer. The Non-delusional Q-learning point makes a ton of sense. Finding the correct Q-function means finding a fixed point for your update rule, but it sure looks like function approximation could lead to cyclic updates in Q-learning if you think about it this way.

@DennisSoemers, thanks for your answer, I learnt a lot. I have more question on the delusional-Q learning. Can you please help me out here: https://ai.stackexchange.com/questions/37440/what-exactly-is-non-delusional-q-learning

There are three problems

Limited capacity Neural Network (explained by John)
Non-stationary Target
Non-stationary distribution

Non-stationary Target
In tabular Q-learning, when we update a Q-value, other Q-values in the table don't get affected by this. But in neural networks, one update to the weights aiming to alter one Q-value ends up affecting other Q-values whose states look similar (since neural networks learn a continuous function that is smooth)
This is bad because when you are playing a game, two consecutive states of a game are always similar. Therefore, Q-value updates will increase or decrease Q-values for both states together. So, when you take one as the target for the other, the target becomes non-stationary since it moves along with you. This is analogous to a donkey running to catch a carrot that is attached to its head. Since the target is non-stationary, the donkey will never reach its target. And, in our case, in trying to chase, the Q-values will explode.
In Human-level control through deep reinforcement learning, this problem is addressed by caching an OLD copy of the DQN for evaluating the targets, & updating the cache every 100,000 steps of learning. This is called a target network, and the targets remain stationary this way.
Non-stationary distribution
This is analogous to the "distribution drift" problem in imitation learning, which can be solved with the dataset aggregation technique called DAgger.
The idea is, as we train, our DQN gets better and better and our policy improves. And this causes our sampling distribution to change since we are doing online learning where we sample according to a policy with $\epsilon$ probability. This is a problem for supervised learning since it assumes stationary distribution or i.i.d. data.
As an analogy, this is like training a Neural Network to identify cats and dogs but showing the network only dogs during the first 100 epochs, and then showing only cats for the remainder epochs. What happens is, the network learns to identify dogs, then forgets it and learns to identify cats.
This is what happens when the distribution changes and we care only about the current distribution during training. So, in order to solve this,  same paper starts aggregating data in a large buffer, and samples a mini-batch of both new data as well as old data every time during training. This is called experience replay, since we don't throw away our past experience and keep re-using them in training.

This answer seems to address more the question: "In practice, how can I improve the learning stability of Q-learning with neural networks?" rather than the convergence properties of Q-learning with function approximation. My question was not much about practical tricks to improve learning stability (everyone is aware of the target network and experience replay), but I was looking more for mathematical proofs that show the convergence or divergence of Q-learning with function approximation.

Oh. I'm sorry for the confusion. I was just sharing my intuition behind why these practical tricks work and how they help. I am not aware of the rigorous mathematical proofs you are looking for. I'm sorry for misunderstanding.


text:
What are some simple open problems in multi-agent RL that would be suited for a bachelor's thesis?
I've decided to make my bachelor thesis in RL. I am currently struggling to find a good problem. I am interested in multi-agent RL with the dilemma between selfishness and cooperation.
I only have 2 months to complete this and I'm afraid that multi-agent RL is too difficult and I don't have the knowledge and time to nicely learn this topic.
What are some simple open problems in multi-agent reinforcement learning that would be suited for a bachelor's thesis?
I've only done applied the Q-learning algorithm to solve a text-based environment in OpenAI's gym.

I have several undergraduates working on multiagent deep RL problems for their theses, but most of them have been working for 8-9 months. 2 might be a stretch.
Good multiagent deep RL problems for a bachelor's thesis might look something like:

Pick an older video game, which has been studied using Deep RL, but not in depth. Right now my students have been liking Nintendo 64 games.
Read the papers that study this game already.
Pick one of the described approaches and reproduce the paper's results in your own system.
Pick one of the parameters that the paper does not explore changing, and see what happens as you change it.

This probably does not lead to a publishable result, but it is real science and can make for a fine undergraduate thesis.
A slightly harder project, which may require more time, would be to examine the "future work" sections of these papers, and perform one of the experiments suggested there. These experiments often lead to small publishable results.


text:
Why isn't the reverse KL divergence commonly used in supervised learning?
Forward KL Divergence (also known as cross entropy loss) is a standard loss function in supervised learning problems. I understand why it is so: matching a known a trained distribution to a known distribution fits $P \log(P/Q)$ where $P$ is the known distribution.
Why isn't the reverse KL divergence commonly used in supervised learning?

Maybe say a little more about what reverse KL divergence is.

Reverse KL Divergence would be $Q \log (Q/P)$

It’s not an exhaustive answer to your question, but here some aspects that might be helpful:
A common problem in supervised learning where you will see KL-divergence used are classification tasks. Very often in those cases the data points in the training set are assumed to belong to a single class $c_i$ where $i$ is from some index set $I$. The class membership is represented by one-hot encoding, which corresponds to a distribution $P$ with $p_i = 1$ for the class the data point belongs to and $p_j = 0$ for all $i \neq j$. The loss function mostly used in this problems is the average categorical cross-entropy $\langle E_P[-log(Q)]\rangle_{\rm data} = \langle H(P) - D(P || Q)\rangle_{\rm data}$, which in the case of single-class membership reduces to the binary cross-entropy. Because only one probability in $P$ is non-zero in this case, the cross-entropy simplifies to $\langle -\log(q_i)\rangle_{\rm data}$. The reverse KL-Divergence $D(Q || P)$ is strictly speaking not even defined in this case, as $Q$ needs to be absolute continuous w.r.t. $P$.
There are of course cases were class membership is probabilistic also in the training data and the above argument doesn't apply in full. From a more conceptual point of view, this thread summaries nicely the relation of cross-entropy to maximizing the likelihood of the observed data under the model. Using the reverse cross-entropy would maximize the likelihood of a typical sample drawn from the model under the (empirical) distribution of the observed data.
A small side note when you read the above link is that in the above described classification problem we have a probability model for the class membership for each point in the input space, this is why the average cross-entropy is minimized.
Classification problems are just one example of supervised learning, so this answer might not fully cover your question.


text:
What kind of functions can be used as activation functions?
I read that functions are used as activation functions only when they are differentiable. What about the unit step activation function? So, is there any other reason a function can be used as an activation function (apart from being differentiable)?

Not completely sure your question. Do you mean
Q. why should we use activation function?
Ans: we need to introduce non-linearity to the network. Otherwise, multiple layers are no difference from single layer network. (It is obvious as we write things in matrix form, and say when we have two layers with weights $W_1$ and $W_2$, the two layer is no difference from a single layer with weight $W_2 W_1$.
Q. why they need to be differentiable?
Ans: Just for sake that we can back-propagate gradients back to earlier layers. Note that back-propagation is nothing but the chain rule in calculus. Say $f(\cdot)$ is an activation function in one layer and the output of that activation function is $\bf y$ and the input is ${\bf u}=W \bf x$, where $\bf x$ is output from the previous layer and mix with weights $W$ in current layer. Of course, the final loss $L$ will depend on ${\bf y} = f({\bf u})= f(W {\bf x})$. Say, loss $L=g(\bf y)$ somehow. To train the weights $W$, we have to find the gradient $\frac{\partial L}{\partial W}$ so that we can adjust weight $W$ to minimize $L$. But $\frac{\partial L}{\partial W}=\frac{\partial g({\bf y})}{\partial W}=\frac{\partial g({\bf y})}{\partial \bf y}\frac{\partial {\bf y}}{\partial {\bf u}}\frac{\partial {\bf u}}{\partial W}$. Each of these product terms can be computed locally and will be accumulatively multiplied as we apply backprop.  And note that the middle term $\frac{\partial {\bf y}}{\partial {\bf u}}=\frac{\partial f({\bf u})}{\partial {\bf u}}$ is just "derivative" of $f(\cdot)$, thus we require the activation function to be differentiable and "informative"/non-zero (at least most of the time). Note that ReLU is not differentiable everywhere and that is why researchers (at least Yoshua Bengio) worried about that when they first tried to adopt ReLU. You may check out the interview of Bengio by Andrew Ng for that.
Q. why step function is a bad activation function?
Ans: Note that step function is differentiable almost everywhere but is not "informative" though. For places (flat region) where it is differentiable, the derivative is simply zero. Consequently, any later layer gradient (information) will get cut off as it passes through a step function activation function.

I think you should have asked for clarifications before answering. I have edited the question and I tried to clarify it, but I am not sure this is what the OP was asking. He/she will eventually clarify it.

Note that ReLU and some other non-continuously differentiable functions do possess well-defined subgradients over the full range.  We can simply select one of those to use as the gradient at the non differentiable points and backprop will work fine.

Differentiability for activation functions is desirable but not necessary since you can reformulate the derivatives at the non-differentiable points, like in the case of ReLU.
Properties Needed:

To take advantage of Universal Approximation Theorem, and to take advantage of the modeling capacity it promises, the activation functions need to be continuous or Borel measurable (if this term is confusing just think of this as common functions), discriminatory (similarly discriminatory means the function doesn't just produce 0 all the time when integrating w.r.t. another function) and nonpolynomial (nonlinear).
Also with the recent research works it is beneficial to have it monotonically increasing.

Recent theoretical analysis of the activation functions gives the edge to ReLU with some preliminary theoretical guarantees like this "Convergence Analysis of Two-layer Neural Networks with ReLU Activation" https://arxiv.org/abs/1705.09886.


text:
What research has been done on learning non-Markovian reward functions?
Recently, some work has been done planning and learning in Non-Markovian Decision Processes, that is, decision-making with temporally extended rewards. In these settings, a particular reward is received only when a particular temporal logic formula is satisfied (LTL or CTL formula). However, I cannot find any work about learning which rewards correspond to which temporally extended behavior.
In my searches, I came across k-order MDPs (which are non-Markovian). I did not find RL research done on k-order MDPs.

non-Markovian reward functions are quite old concepts. They were first introduced in the theory for control systems. You can take any objective fn. of a control system and apply planning and learning concepts to it.

By "reward functions", do you mean "value functions"?

I don't mean the value function. I am assuming that the environment/society knows the rewards to give, and the agent wants to learn a model of what sequence of states visited results in what rewards.


text:
Algorithms to indentify people in pictures without using face recognition
There are lot of researches about face detection in pictures, but is it the only way one can say "this person I'm looking for is here in this picture"? Aren't there algorithms that you can provide with information like lateral or back pictures of a person and making calculations on the height, the width, the anatomic distance between parts of its body, the analysis of the hair can determine: "there's a X per cent chance this is the person you're looking for"? Is it possible to accomplish?


text:
Is it a good idea to use BERT to answer a FAQ with semantic similarity?
I have been looking for BERT for many tasks. I would like to compare the performance to answer an FAQ, using BERT semantic similarity and BERT Q/A.
However, I'm not sure it is a good idea to use semantic similarity for this task. If it is, do you think it is possible to find a dataset to fine-tune my algorithm?

Maybe the following article can help you:
FAQ Retrieval using Query-Question Similarity and BERT-Based Query-Answer Relevance (2019)
They evaluate their model in localgovFAQ and StackExchange datasets.

You can do that and you'll propably find data, yet that depends on the kind of FAQ data you  will apply it on. More importantly, what insight do you gain by comparing two BERT models?
Secondly, f you mean with semantic similarity that vector space embeddings are used, even for the retrieval/ranking and not just for re-ranking, then I can tell you that the performance still isn't SOTA. But you can simply use a such a neural semantic model for re-ranking.
We are working on that. So if you wanna know more, PM me.


text:
Regarding the output layer's activation function for continuous action space problems
I'm interested in building a (deep) RL agent for solving a continuous problem (which splits something into portions).
In all examples I've seen so far, e.g., solving the continuous lunar lander, always a $\tanh$ output layer activation was used, which produces values between $-1$ and $+1$.
Is this just because it fits the use case or is this a general rule for RL agents with continuous action spaces?
What if I just want values between $0$ and $1$? Could I simply use a $\operatorname{softmax}$ activation for my output layer?

the use of Tanh is purely because it fits the described problem (Especially for values that are min-max normalized). I have worked on couple of professional RL projects ( specifically with actions in the continuous space) and I did not use tanh at all. Hope that helped :)

What activation function did you use then? Are there rules of thumb which activation function to use for which use case?
For example, in classification there's the rule to use sigmoid for binary classification and softmax for multi-class classification.

There is no rule of thumb, it totally depends on your own problem. But as a good practice, you might want to use a function that bounds the values in a realistic range according to the actions that you want to predict.

Ok, that's not the answer I was hoping for :D But thank you anyways!

Sorry for the bad news, but if you want to excel in machine learning in general, there is no actual rule of thumb for everything (including sigmoid for binary classification, it is just widely used and not a rule of thumb ), you have to try to think freely and solve each problem on it's own. you are welcome.


text:
Why didn't champion of the Go game manage to win the last game against AlphaGo, after winning the 4th one?
In the documentary about the match, it is said that after losing the 4th game, AlphaGo came back stronger and started to play in a weird way (not human-like) and it was pretty impossible to be beaten. Why and how did that happen?

This might actually be better as a "soft question" since I'm not sure we can know definitively.  The simple answer is that Go is still unsolvable, so both Sedol and AlphaGo were surely making plenty of "mistakes" (non-optimal placements) and, Sedol's choices were stronger in game 4, MCTS is relentless in chipping away at any advantage.  (I discovered this recently playing Taverner's AIAI in partisan sudoku--for the strong AI, errors aren't an insurmountable obstacle to play out from under.)

The technique used by AlphaGo is "Monte Carlo Tree Search", combined with a very well trained neural network. The network's job is to estimate the quality of different board states and moves. This estimation is deterministic. If you show AlphaGo the same board on two different occasions, it thinks it is exactly as good (or bad) on both occasions.
Monte Carlo Tree Search however, is a randomized algorithm. So as a simplified explanation, the way AlphaGo decides which moves to make is:

Look at the current board.
Pick a random move, and imagine what the board would look like if you made that move.
Pick a random move for your opponent, and imagine what the board would look like if they made that move.
Keep doing steps 2 & 3 for a while, so that we're imagining being at some board many moves in the future along a random line of play.
Ask the neural network how good this board state is.
Repeat steps 1-5 many times. Then make whichever move led to the best lines of play on average, to make right now.

What this means is, AlphaGo won't always play the same way, because it doesn't actually consider every move explicitly. It just thinks about enough lines of play to be pretty confident about whether one move is better than another. This is actually not so far removed from how humans play most of the middle of games like these.
So, in game 4, essentially, Sedol got lucky. The random lines of play that AlphaGo chose to look at did not capture some critical facts about one or more board states. This led it to make a mistake. If you asked it to play the same game through again, it might not make the same mistakes (it might think about different random lines of play, that do capture the critical facts it missed in the first game). Further, it might choose to play slightly differently on other moves, which could have a big impact on the rest of the game. These two factors prevented Sedol from simply playing game 4 over again.


text:
How do I find whether this heuristic is or not admissible and consistent?
I was given the following problem to solve.

Given a circular trail divided by $n> 2$ segments labeled $0 \dots n-1$. In the beginning, an agent is at the start of segment number $0$ (the edge between segments $n-1$ and $0$) and the agent's speed ($M$) is $0$ segments per minute.
At the start of each minute the agent take one of three actions:

speed up: the agent's speed increases by $1$ segment per minute.
slow down:  the agent's speed decreases by $1$ segment per minute.
keep the same speed: stay at the same speed.

The action slow down cannot be used if the current speed is $0$ segments per minute.
The cost of each action is $1$.
The goal of the agent is to drive around the trail k times ($1 \leq k$)  and then park in the beginning spot (at speed 0 of course). The agent needs to do that in the minimum amount of actions.
The heuristic given is: If agent is in segment $z$ then: $n-z$ if $z \not = 0$ or $0$ if $z=0$.

I need to find if the given heuristic is admissible (or complete) and consistent.
I think:

Regarding consistency: A heuristic is consistent if its estimate is always $\leq$ estimated distance from any given neighbour vertex to goal plus cost of reaching a goal. So, in the given problem, it is consistent because ($n>2$), so heuristic function is well defined and because of circular trail divided by $n$ segments with a constant price of $1$ for each action, then the given definition of consistency holds because estimated distance from any given neighbour vertex to goal can be looked on as a difference between segments until reaching a goal and it is consistent because again the function is well defined.
Regarding admissibility: An admissible heuristic is one that the cost to reach goal is never more than the lowest possible cost from the current point to reach the goal. I am not sure if the given heuristic is admissible because it does not help much to know the difference between trail size ($n$ = segment size) and current place. But it does not create flaws, so it is probably admissible. I am not sure this is a proof.

Is my idea correct? How could I write it as a proof?

Welcome to AI.SE @hpr16!
Your understanding of when a heuristic is admissible is correct, but your heuristic is inadmissible. An admissible heuristic must always underestimate the cost to move from a given state to a goal state.
Notice that states in the search are not the same as positions on the circle in your problem. A state needs to capture all the information about the current environment the agent is in. In your problem, agents have a speed as well as a position. A state must, therefore, contain both.
To see why your heuristic is inadmissible because the agent can move (n-z) segments in less than n-z steps: it can speed up, and do them in, for example, (n-z)/2 steps, by moving with speed 2.


text:
How can I generate keywords associated with a website given its URL?
I have a column with links to websites and another column with keywords from those websites. I have to find a map between these two, such that for a new input, which is a website's URL, I can generate the keywords associated with the contents of the website.
For example, given the URL chocolate.com, the keywords could be milk and dark. We can tell from this example that my keywords are types of chocolates.
Or, if the URL is career.com, then the keywords could be IT and medicine.
I have a feeling that some sort of supervised neural networks could be used here.   Which approach would be most suited here?

Your question isn't really clear. First, you're talking about links but you probably mean the contents of web pages. Second, it is unclear whether you want to associate keywords from a given list with web page contents, or whether you plan to identify keywords from the text even if they weren't part of training data. The first task would be fairly simple, but I have no idea how the second task could be done. If you try to word your requirements precisely, you might get a knowledgeable answer...

I have edited the original post. To reply to your questions, yes, I mean the content of web pages and yes I have to associate a link between the content of those webpages and the keywords.


text:
Where or for what could genetic algorithms be used in the context of project management?
Where or for what could genetic algorithms (GA) be used in the context of project management (PM)? I thought about task dispatching, but I'm looking for other potential uses of GAs in the context of PM.

GA is well suited to optimize a non-linear fitness function with a lot of variables. Each vector that is a possible solution is evaluated with the fitness function that we want to optimize.
So, GA is well suited to optimize schedules for a lot of people, optimizing resources, etc.
Your mission is to define all variables that you can move-tune and that affect your objectives, and assign an estimated weight on every variable.
For example, create a total project cost as the fitness function, to minimize. One variable will be the number of programmers, multiplied by cost and total months. If you don't want delays over 3 months, add a correction factor: +10000*max(0, months-3), penalizing the fitness function's results where months > 3, etc.
When you apply your GA algorithm to the fitness function, you will find some minimum for your function, if any exists, because the solution may not exist if you added too many constraints.

it takes some insane complexity to make a GA more viable for project optimization than brute force.  The GA doesn't touch every possible configuration, you use it when the number of potential solutions is far to large to ever get close to traversing.

Project Management is most like model to game theory. You can find a article here, as a cooperative game, and apply opportunity cost to your budget, only you need to apply your own rules of the game.

I don't understand what you mean by this sentence "Project Management is most like model to game theory.". Moreover, the abstract of the paper that your first link links to doesn't seem to be related to "project management". Can you clarify why you're linking to that first paper?


text:
Is superintelligence a function of strength or a category?
Super comes from the Latin and means "above".

University of Oxford philosopher Nick Bostrom defines superintelligence as "any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest". (wiki)

Bostrom's definition could be taken to imply this is a quantitative measure of degrees as a numeric relationship. (Under this definition, we have achieved narrow superintelligence, reduced to competency in a single task.)
Gibson, famously, sheds light on an another aspect via Wintermute & Neuromancer, where, once superintelligence is achieved, the AI just f-'s off and does it's own thing, motivations beyond human comprehensions.  (Essentially, "next-level" thinking.) The second measure is discrete and ordinal.
Is superintelligence a function of strength or a category?

I’ll have a stab at this.
Cognitive performance in narrow domains is determined by competency, efficiency and speed. Take calculating numbers, extremely narrow domain but compared to humans the ability of a calculator to calculate numbers exceeds normal human performance, it is much competent in terms of speed. In a bit broader domain, AlphaGo has defeated Go players, which is more difficult than chess, and requires intuition. In fact, there is an instance where the AlphaGo makes a long-term move that was previously unimagined. In all domains however Humans are well rounded, therefore Human Intelligence is called general intelligence. An AlphaGo or Calculator cannot speak eloquently or make music, but AIs are gaining pace in these areas too.
I agree with @nbro that Bostrom wants to keep the interpretation of Superintelligence open. But if there is a rough category, these are-

ANI- Artificial Narrow Intelligence
AGI- Artificial General Intelligence: Where the AI’s performance is at par with humans. After AGI, it quickly takes off to SI.
SI- Superintelligence: Superintelligence is beyond our imagination, we have not figured out yet what will a SI do, think or want.

While these categories are discrete, the functions of strength are not. I’d say they are rather discrete to continuous, because if you look at the computing power plots that follow Moore’s Law, a similar exponential graph can be drawn for AI’s performance towards general intelligence. In that graph, it seems the AI’s performance starts with discrete performance points, and then as it takes off, it becomes continuous.
This is why the term Singularity is often associated with Superintelligence. I hope this answers your question.

I am reading Bostrom's book "Superintelligence". I have only read the first 2 chapters, but I think he doesn't want to define super-intelligence is a precise way, but he leaves the reader the option to define it in a "sensible" way. However, I think that, in his thoughts, there's the (clear) assumption that a super-intelligence will necessarily need to be general, so a super-intelligence will be an AGI.

It occurs to me that Superintelligence as an ordinal value could be explained as follows:

Artificial General Intelligence is strong utility in all problems that humans can conceive of.
Superintelligence as a category includes problems beyond what humans can conceive of.

An easy way to understand this is to look at the difference between dogs and humans.  Dogs are optimized for certain tasks, such as tracking, but limited in terms of what they can conceive and what tasks they can engage in.  Humans have superintelligence, compared to dogs, because we regularly engage with problems and tasks beyond the conception of canines.


text:
Why does all of NLP literature use noise contrastive estimation loss for negative sampling instead of sampled softmax loss?
A sampled softmax function is like a regular softmax but randomly selects a given number of 'negative' samples.
This is difference than NCE Loss, which doesn't use a softmax at all, it uses a logistic binary classifier for the context/labels. In NLP, 'Negative Sampling' basically refers to the NCE-based approach.
More details here: https://www.tensorflow.org/extras/candidate_sampling.pdf.
I have tested both and they both give pretty much the same results. But in word embedding literature, they always use NCE loss, and never sampled softmax.
Is there any reason why this is? The sampled softmax seems like the more obvious solution to prevent applying a softmax to all the classes, so I imagine there must be some good reason for the NCE loss.


text:
Is reinforcement learning using shallow neural networks still deep reinforcement learning?
Often times I see the term deep reinforcement learning to refer to RL algorithms that use neural networks, regardless of whether or not the networks are deep.
For example, PPO is often considered a deep RL algorithm, but using a deep network is not really part of the algorithm. In fact, the example they report in the paper says that they used a network with only 2 layers.
This SIGGRAPH project (DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills)  has the name deep in it and the title even says 'deep reinforcement learning', but if you read the paper, you'll see that their network uses only 2 layers.
Again, the paper Learning to Walk via Deep Reinforcement Learning by researchers from Google and Berkeley, contains deep RL in the title, but if you read the paper, you'll see they used 2 hidden layers.
Another SIGGRAPH project with deep RL in the title. And, if you read it, surprise, 2 hidden layers.
In the paper Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, if you read table 1 with the hyperparameters, they also used 2 hidden layers.
Is it standard to just call deep RL to any RL algorithm that uses a neural net?

Even after several years of success of deep learning systems (i.e. neural networks trained with gradient descent and back-propagation), as far as I know, there is not yet a consensus on what constitutes a deep neural network. Some people could use a neural network with 2 hidden layers and call it deep (like in your case), but other people may just dedicate the adjective deep to refer to neural networks with 10, 100, or more hidden layers. In fact, there are some good reasons to associate the term deep only to neural networks that have a significant number of hidden layers (e.g. 100): for example, the exploding (or vanishing) gradient problem does not typically arise if you only have one hidden layer but can easily occur with many (e.g. 100) hidden layers.
Nevertheless, a neural network with at least one hidden layer can approximate any continuous function, given enough (but finite number of) units (or neurons) in the layers. See the universal approximation theorem. For this reason, we could start denoting any such neural network as deep, but, although this rule would exclude perceptrons (which can only approximate linear functions, and nobody would probably call them deep anyway), this rule would be a bit redundant or useless (i.e. we may just not use the adjective deep to start with).
In your case, the rule that the authors are using seems to be the following: if it contains more hidden layers than the bare minimum (i.e. 1) to approximate any continuous function, then let's denote it as deep.

That's a very good answer!

I also want to add that hidden should also have "a non-linear activation function", else it is as simple as a linear transformation.

Is it standard to just call deep RL to any RL algorithm that uses a neural net?

Yes, it seems to have become standard practice to label RL + any NN "Deep Reinforcement Learning". It is not a formalised term.
The whole "Deep Learning" movement started this decade is as much a marketing term as a scientific one. It is however based on the discovery of real improvements in neural network architecture and training approaches.
You may find that some (or even most) of these shallower networks will use improvements designed in the last decade or so, and also associated with deeper networks, such as Xavier initialization, ReLU activation, the Adam optimizer.
As a personal opinion, I would say that, if a published experiment uses just 1 or 2 hidden layers, and does not make use of any of these recent advances, then the "Deep" label is almost entirely a branding exercise. There were advances with such networks much longer ago. For instance the TD-Gammon paper is from 1995. For TD-Gammon, the authors used reinforcement learning and a NN with one hidden layer to create a Backgammon player that played better than any human player. This was well before "Deep Learning" was a term used to describe such networks, and the term "Deep Reinforcement Learning" does not appear in that paper.
However, because "Deep Learning" is such a loose branding term, there is also an argument that all these older approaches, and pretty much all neural networks with hidden layers, should be included. Wikipedia's definition for Deep Learning says:

Deep learning is a class of machine learning algorithms that:

use a cascade of multiple layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input.
learn in supervised (e.g., classification) and/or unsupervised (e.g., pattern analysis) manners.
learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts.

Using that definition would include all the papers you cite.  You don't need a 50 layer Resnet architecture to qualify. And the branding exercise makes more sense under that definition, because the newly invented techniques have made such systems that much more viable and worthy of investment (of time & effort as well as financially).


text:
If the AI goal is the protection of humans, will it always pursue this goal?
If the AI goal is to serve humans and protect them (if this ever happens) and AI someday realizes that humans destroy themselves, will it try to control people for their own good, that is, will it control man's will to not destroy himself?

this is the very premise of the movie I, Robot tangentially based on Isaac Asimov's eponymous book. In the movie it turns out that

! yes, the robots take over humans on pretext of protecting humans from destroying themselves

If the AI is static (heuristic and fixed), it will always pursue the stated goal.  However, such a system would be "brittle", and either break or produce bad output if confronted with input not previously defined, or outside its model.
If the AI evolves via learning, even where the goal is specific, its interpretation of that goal might change, and produce unexpected results.  (The "I, Robot" scenario.)
If the AI is emergent, by which I mean it evolves in way that cannot be predicted, it might evolve new goals.
To answer the question directly:
Hypothetically, if there was an AGI or artificial superintelligence, or ultraintelligent machine tasked with protecting humans, and that AI perceived humans to be destroying themselves, that AI would, if able, take control of human society.  (I don't see this as contradicting its goal.)
However, it must be stated that, in a condition of imperfect & incomplete information, where the problem is intractable, the AI is just guessing like we humans do, even if it makes better guesses, as in the case of narrowly intelligent AIs like AlphaGo.

About the deletion: Many thing (even for most thing that you can quote) about AI is based on assumption and it can never be verified in our life or even the lifetime of the human race, if such thing are not allowed, then the question like this should be deleted too since it can never be answered based on verified trueth

@jw_ if you want to edit your answer to qualify the unproven assertions, it could be reopened.  As it was written, it causes harm.  "This is because consciousness need be based on complex/dynamic/fault-tolerant/self-growing/self-refreshing circuits..." is an unvalidated assertion. Should be presented as "consciousness may require...)

@jw_ "The only way to realize AI and motion is use bio engineering to design new species" is incredibly problematic, because there is legitimate AI that exists, just that it's narrow.  We also have robots that produce utility, not based on bio engineering.)

"This is because consciousness need be based on complex/dynamic/fault-tolerant/self-growing/self-refreshing circuits..." This is really wrong, you can check my another post after this one (hope don't delete it since its full of such staff...) - consciousness can run on any circuits in theory, but non-bio consciousness can never be economic compared with bio based. Comparing 250W 900mm2*1um AI die to 10W 1.5L human brain with self repair and reform, the former will be always nothing but a joke. And Elon musk and Hawking warn about thread of AI? Joke too.

"because there is legitimate AI that exists" In my another post, it mentioned that such "AI" is just some algorithms, it should not have been called AI at all. Anyway this post is indeed not good. But the deletion policy in StackExchange is bad, down votes is enough to identify the problematic one, and consumer of this site should not be that easy affected by bad post, but deletion will erase the right information too and the OP author's work is wasted.

BTW Celebrities worshiped by billions of people warning about the threat from will-never-mass-produced-non-bio-based-AI is the real threat and harm since it affect voters and the politicians who commonly worship them too e.g. Elon Musk is a former consultant for Trump. What about bio-based? Will ourself be called a threat since the only difference between us and the bio-based-AI is that we are designed by nature and they are designed by us? And long before that is designed successully, human race could have break down due to amatuer designed fatal microorganism.

That's reasonable, didn't know that before.

Now Trump compare Musk to Edison. A man in great power without basic science knowledge calls a great actor Edison, this is when the threat comes.

If the main goal of AI (which I assume you mean an AGI) is to protect humans and AI will be effective, then AI will always attempt to pursue its main goal (otherwise the assumption of its effectiveness does not hold), even at the expense of other less important goals that it might have. However, if the destruction of a human (or a group of humans) protected or avoided the destruction of other humans, then AI would face a dilemma. In that case, I think it is hard to predict the actions of the AI. Will it act rationally or irrationally? What would it mean for the AI to act rationally? Which parameters will it take into account? Only the number of deaths, or will take into account the future and weight the importance of the lives? How will it define the importance of a human life?

I think AI will go for the less damage as possible for humans because when it comes to protecting it means not to harm and if it harms humans AI will go against his will(main goal) and in that case maybe controlling man's will is the best option to not harm himself.

@giorgircheulishvili But the problem here now is how you define "damage" (as I roughly mention in my answer: which parameters will AI take into account when facing a dilemma?).

"
Less damage for humans"
as you mentioned before means AI will always attempt to pursue its main goal at the cost of other less important goals and I think less important things for AI are human living environment and his opinion about a situation.

Either the AGI is using its own intelligence to decide about goals themselves and there is no telling what it may pick. Or there are hard-coded goals and there are no dilemmas: just follow the rules.

@MathieuBouville I don't think these are the only two options. There's no proof that if an AI has an hard-coded goal, then everything will be deterministic (or that there will be no dilemmas). Anyway, in my answer, I assume that an AI has the main goal of protecting humans (if this is hard-coded or not, it is not important).


text:
Is REINFORCE the same as 'vanilla policy gradient'?
I don't know what people mean by 'vanilla policy gradient', but what comes to mind is REINFORCE, which is the simplest policy gradient algorithm I can think of. Is this an accurate statement?
By REINFORCE I mean this surrogate objective
$$ \frac{1}{m} \sum_i \sum_t log(\pi(a_t|s_t)) R_i,$$
where I index over the $m$ episodes and $t$ over time steps, and $R_i$ is the total reward of the episode. It's also common to replace $R_i$ with something else, like a baselined version $R_i - b$ or use the future return, potentially also with a baseline $G_{it} - b$.
However, I think even with these modifications to the multiplicative term, people would still call this 'vanilla policy gradient'. Is that correct?

Use of the term may vary, depending on context, and what "extra" feature is being discussed. Could you give a reference to where someone uses "vanilla policy gradient"? With nothing else to go on, I think your understanding seems reasonable.

I'm thinking of TRPO and PPO right now, where the 'vanilla' part often implies that it's not sample efficient because it only performs one optimization step with the data. I can't point to other specific papers right now, but I think I've seen the term 'vanilla policy gradient' in other contexts too. Are there any other aspects of REINFORCE besides the single-optimization-update that makes it 'vanilla'?

You can check the Open AI Introduction to RL series, they explain pretty neatly there what is the Policy Optimization and how to derive it. I think, that usually when we are talking about REINFORCE algorithm, we are talking about the one described in Sutton's book on Reinforcement learning. It is described as the policy optimization algorithm maximizing the Value Function $v_{\pi(\theta)}(s) = E[G_t|S_t = s]$ of initial state of the agent. Here $G_t = \sum_{k=0}^\infty \gamma^k R_{t+k+1}$ is the $\gamma$ discounted return from given state, time  $s, t$. Or, shortly put.
$$
 J(\theta) = v_{\pi(\theta)}(s_0) = E[G_t|S_t = s_0]\\
\nabla J(\theta) = E_\pi\left[G_t\frac{\nabla \pi (A_t |S_t, \theta)}{\pi (A_t |S_t, \theta)}\right]
$$
But in the RL series of Open AI, the algorithm that is described as Vanilla policy gradient (If it is the one you are talking about) is optimizing finite-horizon undiscounted return $E_{\tau \sim \pi} [R(\tau)] $, where $\tau$ are possible trajectories. e.g.
$$
 J(\theta) =  E_{\tau \sim \pi} [R(\tau)] \\
\nabla J(\theta) = E_{\tau \sim\pi}\left[\sum_{t=0}^T R(\tau) \frac{\nabla \pi (A_t |S_t, \theta)}{\pi (A_t |S_t, \theta)}\right]
$$
They do look very similar in objective functions, but they are different. The way the gradient ascent is performed differs strongly since in REINFORCE method the gradient ascent is performed once for each action taken for each episode and the direction of ascent is taken as
$$
G_t\frac{\nabla \pi (A_t |S_t, \theta)}{\pi (A_t |S_t, \theta)}
$$
so the update becomes
$$
\theta_{t+1} = \theta_{t} + \alpha G_t\frac{\nabla \pi (A_t |S_t, \theta)}{\pi (A_t |S_t, \theta)}
$$
but in VPG algorithm the gradient ascents performed once over multiple episodes and direction of ascent taken as average
$$
\frac{1}{|\mathcal{T}|}\sum_{\tau\in\mathcal{T}} \sum_{t=0}^T R(\tau) \frac{\nabla \pi (A_t |S_t, \theta)}{\pi (A_t |S_t, \theta)}
$$
and gradient ascent step is
$$
\theta_{t+1} = \theta_{t} + \alpha \frac{1}{|\mathcal{T}|}\sum_{\tau\in\mathcal{T}} \sum_{t=0}^T R(\tau) \frac{\nabla \pi (A_t |S_t, \theta)}{\pi (A_t |S_t, \theta)}
$$
which looks a lot like what you have stated as REINFORCE algorithm.
I admit, that some form of mathematical equivalence can be derived between them, since expectation over policy and expectation over the trajectory sampled from the policy looks practically the same. But approaches differ at least in the way, the ascent is computed.

the answer is great.

OpenAI has a series of Spinning Up pages on their website to educate people about AI. One of those defines Vanilla Policy Gradiant.
Vanilla Policy Gradiant via OpenAI
At the bottom of the page are reference papers that further discuss gradiants.
Whether this is definitive for Vanilla Policy Gradients or not I do not know, but if many others refer to OpenAI for learning this subject their definition will spread.

By vanilla policy gradient, I think what is meant is normally any arbitrary policy gradient for the purposes of formalization (or whatever is trying to be communicated).
For example, let $J(\theta)$ be any policy objective function. Then our policy gradient would be $\begin{equation} \nabla_\theta J(\theta)  \end{equation}$ where the change in our policy is $\Delta \theta = \alpha \nabla_\theta J(\theta)  $ with $\alpha$ being our step size.
This will, of course, vary as notational convention and term definition can change between practitioners.