text:
Can layers of deep neural networks be seen as Hopfield networks?
Hopfield networks are able to store a vector and retrieve it starting from a noisy version of it. They do so setting weights in order to minimize the energy function when all neurons are set equal to the vector values, and retrieve the vector using the noisy version of it as input and allowing the net to settle to an energy minimum.
Leaving aside problems like the fact that there is no guarantee that the net will settle in the nearest minimum etc – problems eventually solved with Boltzmann machines and eventually with back-propagation – the breakthrough was they are a starting point for having abstract representations. Two versions of the same document would recall the same state, they would be represented, in the network, by the same state.
As Hopfield himself wrote in his 1982 paper Neural networks and physical systems with emergent collective computational abilities

The present modeling might then be related to how an entity or Gestalt is remembered or categorized on the basis of inputs representing a collection of its features.

On the other side, the breakthrough of deep learning was the ability to build multiple, hierarchical representations of the input, eventually leading to making AI-practitioners' life easier, simplifying feature engineering. (see e.g. Representation Learning: A Review and New Perspectives, Bengio, Courville, Vincent).
From a conceptual point of view, I believe one can see deep learning as a generalization of Hopfield nets: from one single representation to a hierarchy of representation.
Is that true from a computational/topological point of view as well? Not considering how "simple" Hopfield networks were (2-state neurons, undirected, energy function), can one see each layer of a network as a Hopfield network and the whole process as a sequential extraction of previously memorized Gestalt, and a reorganization of these Gestalt?

Deep Learning is not a generalization of Hopfield networks. Deep Learning is a "generalization" of the neural networks/connectionism field started by Rumelhart and McClelland.
There are two kinds of neural networks:

Directed (Perceptron, MLP, ConvNets, RNNs, etc.)
Undirected (Hopfield Nets, Boltzmann Machines, Energy-based models, etc.)

Any of these can be made deep. As you said, Boltzmann machines are the probabilistic version of Hopfield Networks, and there has been a lot more work on deepifying these models than Hopfield nets: Deep Boltzmann machines, Deep Belief Networks, and deep energy models. Hinton is really the guy you want to read to learn about these models, but you can have a look at this paper which compares the three models.
Not sure about the Gestalt organisation. I guess I'll leave that to your interpretation.

My question wasn't probably clear enough. I was asking about the emergence of the ability to categorise (Gestalt) in NN.

@MarioAlemi This is an old question, but let me comment because your question needs to be improved. You don't seem to be asking about the ability to categorize in neural networks. You probably already know (even back then when you asked this) that neural networks are able to classify given multiple features of the input. Wouldn't that be the Gestalt? If that was your question, what does the Hopfield network have to do with this? Please, edit your post to clarify what your question really is. Indeed, to me, it's not clear what you're asking, especially after your comment above.

@MarioAlemi If you're asking "are today's neural networks a generalisation of Hopfield networks?" (which is what you're literally asking), then this answer seems to answer that question (in the first sentence!).


text:
What are the best machine learning models for music composition?
What are the best machine learning models that have been used to compose music? Are there some good research papers (or books) on this topic out there?
I would say, if I use a neural network, I would opt for a recurrent one, because it needs to have a concept of timing, chord progressions, and so on.
I am also wondering how the loss function would look like, and how I could give the AI so much feedback as they usually need.

There are a few of them. The most recent I've found is from DeepMind: The challenge of realistic music generation: modelling raw audio at scale. This video is a great analysis of it.

I like this answer, but maybe you could link to some other ones. EvoMusArt puts out a few papers every year, for example: http://www.evostar.org/2018/cfp_evomusart.php#abstracts.

@JohnDoucette I don't know much on the theme, and I didn't know that article/team. You should compile your own answer, so the OP gets notified

I am also new to the neural network architecture game but from what I have learned so far I think you have a few good options to choose from.
A recurrent neural network (RNN) would be a standard approach but if you're looking for something more robust you could look into a Long Short Term Memory network (LSTM). The neurons have a memory of past events and can recall that later on. It is a subset of RNN.
Perhaps you could go a little further and use a Convolutional Neural Network (CNN). So far these type of networks have been highly successful for image recognition. You could abstract a song piece as an image. Each pixel could be a progression in time and the value of the pixel could be the actual note.
Also take a look at this article for a good overview of several different neural network types.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
In imitation learning, do you simply inject optimal tuples of experience $(s, a, r, s')$ into your experience replay buffer?
Due to my RL algorithm having difficulties learning some control actions, I've decided to use imitation learning/apprenticeship learning to guide my RL to perform the optimal actions.  I've read a few articles on the subject and just want to confirm how to implement it.
Do I simply sample a state $s$, then perform the optimal action $a^*$ in that state $s$, calculate the reward for the action $r$, and then observe the next state $s'$, and finally put that into the experience replay?
If this is the case, I am thinking of implementing it as follows:

Initialize the optimal replay buffer $D_O$
Add the optimal tuple of experience $(s, a^*, r, s')$ into the replay buffer $D_O$
Initialize the normal replay buffer $D_N$
During the simulation, initially sample $(s, a^*, r, s')$ only from the optimal replay buffer $D_O$, while populating the normal replay buffer $D_N$ with the simulation results.
As training/learning proceeds, anneal out the use of the optimal replay buffer, and sample only from the normal replay buffer.

Would such an architecture work?

For completeness, you may want to edit your post to include the name of the RL algorithm you were using at the time (if you still remember).

That seems to be functional.
That is a great approach, as long as you are using an off-policy algorithm (since the samples you are using to learn are not the policy currently being performed), like Q-learning.
By annealing the sample rate from the optimal buffer to the regular one, you introduce noise into the network and emphasize exploration (albeit more limited). This is helpful when you (the researcher) have no access to optimal policies, but merely "good" ones, and you still want the network to try and improve on those.

Have you tried implementing this before?

I have, with both DQN and DDPG. Worked nicely with DQN, not that well with DDPG (although I don't think it was about the transfer learning, the problem was just too complex).

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How does the Dempster-Shafer theory differ from Bayesian reasoning?
How does the Dempster-Shafer theory differ from Bayesian reasoning? How do these two methods handle uncertainty and compute posterior distributions?

Demster-Shafer Theory and Bayesian Networks were both techniques that rose to prominence within AI in the 1970's and 1980's, as AI started to seriously grapple with uncertainty in the world, and move beyond the sterilized environments that most early systems worked in.
In the 1970's and perhaps even earlier, it became apparent that direct applications of probability theory to AI were not going to work out because of the curse of dimensionality. As more variables needed to be considered in a given problem, the amount of storage space and processing time needed increased exponentially. This led to a search for new methods to handle uncertainty within AI.
Bayesian Networks and Bayesian Learning remained firmly rooted in probabilistic reasoning, but allowed for the assignment of subjective priors to probabilities, to incorporate expert knowledge. It also allowed problems to be factored into graphical structures to avoid the curse of dimensionality in most cases.
Dempster-Shafer was a further generalization of Bayesian Networks, in which malformed probability distributions were permitted as a way to capture uncertainty. So, for example, the probability of all possible events was not required to add up to 1, because there might be events we don't know about. While on the surface this might seem reasonable, most modern AI researchers view this as a deeply flawed approach. Cheeseman's criticism of DS and other non-probabilistic methods is the basis from which a lot of this view stems. Judea Pearl was another harsh and influential critic of DS Theory.

The basic difference in the fusion of new information is that in Bayesian Networks, after observing new evidence $$E$$, we apply Bayes' rule:
$$ P(H | E) = P(E | H) * P(H) / P(E) $$
to yield a posterior for every hypothesis.
In DS theory, we look for overlap between the worlds suggested by the new evidence and the old data. This can lead to non-sensical results.
Here's an example:
Our prior belief is that our Robot is located at position (0,1) with probability 0.95, and position (0,2) with probability 0.05.
A new signal appears. The signal indicates that robot is at position (0,0) with probability 0.95, and position (0,2) with probability 0.05.
Under Bayes rule, we consider the probability that these signals were generated under each of our original hypotheses, and the probability of observing these signals at all, as shown in the equation above. Under DS-Theory, we would do the same thing.
However, DS-theory provides a second way to interpret the signal: as a second prior distribution, rather than as evidence. We can then combine this second prior with the first, to compute a sort of joint-prior:
$$P( H_{A,B}) \propto P(H_A) * P(H_B)$$
That is, the "probability" (it's not always a true probability, which is one of the criticisms) of a hypothesis after the fusion will be the product of the "probabilities" of the hypothesis under each of the separate priors.
In the example above, this gives a wacky result: the "joint prior" says the Robot is at (0,2) with probability 1.0. This and other problems are why this mode of information combination has mostly been abandoned. There are many more examples on the wikipedia page for DS.
I think there's a discussion of this in more detail in Section IV of Russell & Norvig, at the end of one of the chapters.


text:
What is artificial intelligence?
What is the definition of artificial intelligence?

We don't know how to define "artificial intelligence" probably because we don't know how to define "intelligence".

Over the years, many people attempted to define artificial intelligence. A lot of those definitions are summed up by Stuart Russell and Peter Norvig in their book Artificial Intelligence - A Modern Approach
The definitions of AI can be summarised as falling into the following categories:

Those that address thought process and reasoning (how an AI thinks/reasons)
Those that address behaviour (how an AI acts given what it knows)

Furthermore, the above 2 categories are further divided into definitions that:

I. assess the success of an AI (to do the above) based on its ability
  to replicate human performance
II. or an ability to replicate an ideal performance measure called
  'rationality' (does it do the 'right' thing based on what it knows?)

I will cite you definitions that fit into each of the above categories:

1.I. "The [automation of] activities that we associate with human thinking, activities such as decision making, problem solving, learning.." - Bellman 1978
1.II. "The study of the computations that make it possible to perceive, reason, and act." - Winston, 1992
2.I. "The study of how to make computers do things at which, at the moment, people do better" - Rich and Knight, 1991
2.II. "The study of the design of intelligent agents" - Poole et al., 1998

In summary, AI is devoted to the creation of intelligent and rational machines that can make rational decisions and take rational actions.
I would suggest you read up on the Turing test, which Alan Turing proposed to test if a computer was intelligent. However, the Turing test has a few issues, because it is anthropomorphic.
When Aeronautical engineers created the airplane, they didn't set their goal that planes should fly exactly like birds, but rather, they started learning how lift forces were generated, based on the study of aerodynamics. Using this knowledge, they created planes.
Similarly, people in the AI world shouldn't put, IMHO, human intelligence as the standard to strive for, but, rather, we could use, say, rationality as a standard (amongst others).

Comments are not for extended discussion; this conversation has been moved to chat.

The problem is that there are different rationalities. One man's ratio is another woman's madness.

Appears to me that the aspect of recnognition based on sensor signals is not displayed here, where I have the typical signal processing pattern in mind: Input/Processing/Output. This does not start with Processing ~ Thinking.

Intelligence
A measure of the strength of a decision-making agent relative to other decision-making agents, in regard to a given task or set of tasks. The medium is irrelevant—intelligence is exhibited by both organic and intentionally created mechanisms.  May also be the capability to solve a problem, as in the case of a solved game.
Artificial
Relates to the term artifact, a thing which is intentionally created.  Typically this term has been used to connote physical objects, but algorithms created by humans are also regarded as artifacts.
The etymology is derived from the Latin words ars and faciō: "To skillfully construct", or, "the art of making".
Artificial Intelligence

Any decision-making agent that is skillfully (intentionally) constructed.

APPENDIX: The meaning of "intelligence"
The original meaning of "intelligence" seems to be "to acquire", back to the Indo-European. See: intelligence (etymology); *leg/*leh₂w-
The OED 1st definition of intelligence is not incorrect, extending the meaning to acquisition of capability (demonstrable utility), just that the second definition is the older and fundamental: "The collection of information of [strategic] value; 2.3 (archaic) Information in general; news."
You can regard the universe as being comprised of information, whatever form that information takes (matter, energy, states, relative positions, etc.) From the standpoint of an algorithm, this makes sense since the only means they have to gauge the universe are percepts.
Take a flat text file. It may just be data, but you could try and execute. If it actually runs, it might demonstrate utility at some task. (For instance, if it is a minimax algorithm.)
"Intelligence as a measure of utility" is itself "intelligence" in the sense of information, specifically that information by which we measure intelligence, as a degree, relative to a task or to other intelligences.

Note, this also hews to the Russell&Norvig basic definition of intelligence, rooted in utility.  Sans utility, there is no meaningful definition of Intelligence, at least not in the sense of being concrete or practical.

What is artificial intelligence?

This question is ambiguous. I will address the two less ambiguous but related questions.

What is the goal of the AI field?
What is an artificial intelligence?

What is the goal of the AI field?
In the article What is artificial intelligence? (2007), John McCarthy, one of the founders of artificial intelligence and who also coined the expression artificial intelligence, writes

Artificial intelligence is the science and engineering of making intelligent machines, especially intelligent computer programs.
It is related to the similar task of using computers to understand human intelligence, but AI does not have to confine itself to methods that are biologically observable.

Therefore, the goal of the AI field is to create intelligent programs (or machines). So, he defines the goal of the field based on the concept of intelligence, which he defines as follows.

Intelligence is the computational part of the ability to achieve goals in the world. Varying kinds and degrees of intelligence occur in people, many animals and some machines.

So, we could conclude that the goal of the AI field is to create programs (or machines) that achieve goals in the world to different extents.
This definition of intelligence is reasonable and consistent with reinforcement learning (which could be the path to AGI), but maybe not formal and rigorous enough. In this answer, I report a possibly more sound definition of intelligence given by Hutter and Legg, so I suggest that you read it, but the definitions are roughly consistent with each other (because the concepts of "goal" and "goal-seeking behavior" are present in both definitions), although they emphasize different aspects (e.g. computation or generality).
What is an artificial intelligence?
Nowadays, most people distinguish two types of artificially intelligent systems:

Narrow AI (aka weak AI, although this term may not exactly be a synonym for narrow AI, but it's just the opposite of strong AI: see the Chinese-Room argument): a system that solves a very specific problem (e.g. playing go)
Artificial general intelligence (aka strong AI, although this term may not always be used as a synonym for AGI): a system that can solve multiple problems

This distinction started with philosophical arguments, such as the Chinese room argument, where the ability of a computer to "understand" the actual problem was questioned. Nowadays, there are multiple successful cases of narrow AIs (e.g. AlphaGo), but there isn't yet a "truly" AGI system. This is mainly due to the fact that more people have been (probably wisely) focusing on solving specific problems rather than solving the "holy grail" problem of the AI field, i.e. create an AGI, which seems to be a lot more difficult than creating narrow AI systems. (Anyway, the creation of an AGI could actually arise from solutions to these specific problems, so maybe we are already creating the tools needed to build an AGI, without realizing it). See What is the difference between strong-AI and weak-AI? for more details about the difference between narrow AI and strong AI.

According to the book Artificial Intelligence: A Modern Approach (section 1.1), artificial intelligence (AI) has been defined in multiple ways, which can be organized into 4 categories.

Thinking Humanly
Thinking Rationally
Acting Humanly
Acting Rationally

The following picture (from the same book) provides 8 definitions of AI, where each box contains 2 definitions that fall into the same category. For example, the definitions in the top-left corner fall into the category thinking humanly.

There is also the AI effect, which Pamela McCorduck describes (in her book Machines Who Think: A Personal Inquiry into the History and Prospects of Artificial Intelligence, p. 204) as follows

it's part of the history of the field of artificial intelligence that every time somebody figured out how to make a computer do something — play good checkers, solve simple but relatively informal problems—there was a chorus of critics to say, but that's not thinking

The image should really be entered as block-quoted text.  As it is now, no one can copy'n'paste from it, and search engines can't index its content.

Along the observation in the comment, the artificial part can in principle be assigned to artifacts (or at least combination of natural beings and artifacts) and focus on the intelligence knot.
Assuming the shortest definition cannot be less than three words long, intelligence looks like a measure of an operation onto something.
Here are some tentative points around a sensible concise definition, depending on the concrete word we choose for each piece of the abstract "measure of operation on something".
Data organization simplicity.
Information processing efficiency.
Model adaptation adequacy.

There is no formal definition that most people agree on. Hence here is what I, as a data science / machine learning consultant, think:
Artificial intelligence as a research field is the study of agents which sense and act autonomously in an environment and improve their situation according to some metric with their actions.
I don't like the term, because it is too broad / vague. Instead, look at the definition of machine learning by Tom Mitchell:

A computer program is said to learn from experience 'E', with respect to some class of tasks 'T' and performance measure 'P' if its performance at tasks in 'T' as measured by 'P' improves with experience E

Machine learning is an important part of AI, but not the only one. Search algorithms, SLAM, constrained optimization, knowledge bases and automatic inference are also certainly part of AI.

The facts and anomalies paper (mine) has attempted narrowing down on what intelligence is. Only once you identify it, you'll be on the right path to replicating it. For now, nobody knows exactly what constitutes intelligence. From the perspective of just what intelligence is: The ability to search for, parse and bring in the right information into a context in order to deduce past, present and future (temporal) occurrences that enable moving toward favourable outcomes. So a program or machine that operates based on some if-then-style conditions would be considered as having a basic level of intelligence.
However, when we say "artificial intelligence", we usually mean an intelligence that's approximately as good or better than living creatures. From the facts and anomalies paper, the decision-making process was observed to be similar across almost all living creatures. What differs, is the amount of memory stored, the length of the attention-span, questioning capability and to what level of complexity the creature can mix and match memories in the world model it creates in its mind.
The paper also brings forth an important point that an intelligence created via any tech can be specific to that tech. So an AI created via microprocessors will inherently be different from an organic intelligence (and there is nothing wrong in building it as such), but we will recognize it as being intelligent in the same way as an artificial sweetener is accepted as a replacement for sugar.
To think of it another way: If you had to say which one of two people are intelligent, how would you evaluate them? The person we'd consider more intelligent, would be the one who understands and analyzes situations better to take decisions that have a better outcome than others. Even a person lacking vast knowledge will be considered intelligent if their creativity and depth of thought is higher than others.
A crucial factor that enables this is the attention span of the mind. A machine that is programmed to access vast stores of memories for even trivial decision making tasks (which helps it evaluate consequences of various actions: commonsense) and is capable of asking questions and can simulate situations in memory by loading and modifying stored memories in the simulation (imagination) will be a lot more "intelligent" than us.
There is a second paper (cognitive memory constructs) that describes a bit about the theory of implementation.
All this being said, this is just our perspective on intelligence. The fact that the universe exists in such a complex form, is probably evidence of much higher forms of intelligence (like how we are much more intelligent than the Age of Empires AI characters we created). Intelligence may exist in far more dimensions than we are currently capable of imagining.

This is an interesting answer about what intelligence is. This definition is a major challenge in itself of course, and something of a pre-requisite for AI. However, the question is specifically about definition of artificial intelligence. To improve the answer I think you could more explicitly comment about the link, and how it affects definition of AI. It would be good to make it clear for instance that the linked paper is an analysis from AI perspective.

What do you mean by "For now, nobody knows exactly what constitutes intelligence."? Do you mean that nobody fully knows the mechanisms that produce intelligence? That's different than saying "We don't know what intelligence is". I know many definitions of intelligence that are consistent with what we usually call "intelligence", so I don't agree with your claim, unless you wanted to say something else. The definition of intelligence is just a matter of agreement and consensus and not a matter of "discovering" a definition.

@nbro: I'm referring to the minimum "components" required to build an intelligence. Like how the Turing machine specified a machine that could compute any computable sequence. Let's not play with words here. We do not really know enough to have a productive discussion on this.

The shortest answer I could can come up with could be as follow; take it with a grain of salt though since we still do not know a lot about natural intelligence:
What natural intelligence is could be seen as the process of learning abstract concepts from limited observations with the intention to use them for solving a [new] task. This process involves using those concepts to imagine new, hypothetically correct scenarios/theories and combine them in a meaningful way to cut down the enormous hypothesis space of possibilities and enable generalization to new situations without observing any data beforehand. Artificial intelligence is to bring what natural intelligence does into machines.


text:
What are some good approaches that I can use to count the number of people in a crowd?
What are some good approaches that I can use to count the number of people in a crowd?
Tracking each person individually is obviously not an option. Any good approaches or some references to research papers would be very helpful.

Here a convolutional neural network (cNN) based approach is presented:
Image Crowd Counting Using Convolutional Neural Network and Markov Random Field
This blog post has more of a tutorial character and also present an approach based on CNNs
Counting Crowds and Lines


text:
Is there a way of pre-determining whether a CNN model will perform better than another?
I developed a CNN for image analysis. I've around 100K labeled images.  I'm getting a accuracy around 85% and a validation accuracy around 82%, so it looks like the model generalize better than fitting. So, I'm playing with different hyper-parameters: number of filters, number of layers, number of neurons in the dense layers, etc.
For every test, I'm using all the training data, and it is very slow and time consuming.
Is there a way to have an early idea about if a model will perform better than another?

Hi, you don't specify the expected accuracy in your question. The difference that you have written between 85% for training and 82% for validation seems normal and implies that the model generalises well. What is your expected accuracy, and how do you know it is expected (is there another experiment using the exact same data that you are trying to match)?

@NeilSlater yes, I think that it generalizes ok, even if I try to overfit I don't get better acc that 87% or so. Trained human accuracy could be around 98%. I've tried to use transfer learning with similar or worst results... The labeled data is made by my, no another experiment is using the exact data.

So your expectation of 98% is based on human accuracy on the same task?

@NeilSlater yes it is, or something close to it, 95% could be enough, but I'm far, now playing manually with the number of convolution layers and filters. so tedious

The simple answer to your question is "No" with a caveat.
The caveat is that there are signs that your network is never going to perform well.  For example, the epoch accuracy fails to improve or even consistently declines over the first several epochs, or the validation accuracy is flat or declining.  It could be that the validation loss starts high and just keeps increasing from the beginning.  These are all bad signs.
Outside of this, however, it's very tough to know the model won't work well in the long run.  For example, we have a model we built for solving a set of CAPTCHAs.  The regression portions of that converged very quickly, but the portions that solved the rest of the CAPTCHA took something like 18 hours before they converged.  Honestly, we only ran it that long because it was the end of the day and the regression piece looked so promising; there was nothing in the training behavior of the CAPTCHA solver that looked like it would work (even though our intuition was that it should.)
In the end, we have a 96%+ accuracy CAPTCHA solver that we likely would have killed if we had watched it train for more than 10 or 15 minutes.

I would like you to try with following changes in the model.

Introduce batch normalization layer in the model.
Try with the batch size of 32 - 64
Use different architecture, Like VGG, Resnet, etc..

There is no full proof answer to this question but you can get best by trying some know strategies.

Actually I did all your say 1) BN is the network. 2) The batch size is 50, tested with VGG19 and the results are similar to mine, also I have test with Vision AutoML from google with similar results.

Your description sounds like something similar to Imagenet dataset. According to this website, the state-of-the-art top-1 accuracy just as high as 86%, not much higher than yours. There are plenty of methods to improve accuracy. I would suggest you to read the paper or github listed in SOTA to find ideas that best fits your situation.

I can't comment but here are a few suggestions:

Play with lr and lr finder
Use a pretrained model
Use architecture search or use something like efficientnet b6
Use swish over relu
Try different optimizers
Try Bayesian Optimization


text:
Can Q-learning be used to find the shortest distance from each source to destination?
Is it possible to form a table that will have simply the shortest distance from each source to destination using q learning?
If not, suggest any other learning algorithm.

It is not clear what the learning environment is here, and I think that is important. For instance, if the sources and destinations are in the real world, without accurate mapping, and you must actually physically explore the environment, then RL might be a reasonable choice. If the sources and destinations are on some kind of computer grid or maze, then simpler search algorithms might be better. Could you give more details about the learning problem you are faced with?

Welcome to AI.SE Adarsh!
This is fairly simple to do with Q-Learning.
If you assign a reward equal to -1 for each location other than the goals, and 0 for the goals, then Q-learning with no discount factor will learn the length of the shortest path from each location to the goal. Alternatively, you can use a discounting factor, and then invert the equation on the learned values of the Q function to obtain the number of steps.
Hope this helps!


text:
How can we compare the intelligence of AI systems?
One way of ranking human intelligence is based on IQ tests. But how can we compare the intelligence of AI systems?
For example, is there a test that tells me that a spam filter system is more intelligent than a self-driving car, or can I say that a chess program is more intelligent than AlphaGo?

There are different ways to compare different kinds of AI techniques.
As a starting point, be aware that "AI System" can mean an incredibly broad range of things. In popular culture, we usually think of a deployed system that uses AI techniques. These systems can only be compared on the basis of their performance, and their performance may have relatively little to do with AI itself (e.g. their behaviours might be more strongly affected by user interface decisions, not the AI techniques under the hood).
In contrast, AI researchers are usually more interested in comparing the performance of different AI algorithms at solving the "AI-ish" parts of the problem a fully developed system aims to solve. A common way to do this is with benchmark problems. For example, in machine learning it is common to compare two algorithms by running each of them on a commonly used dataset, and comparing the performance of the models they create. In AI Planning, it is common to issue planning challenges to the community, and compare the quality of the plans on several different axes (e.g. average wait times, maximum wait times, whether goals were accomplished, how long it took to create a plan, etc.).
There is no generally agreed upon way to compare techniques across different areas of AI, but a commonly adopted approach is the Turing Test. In the Turing Test, we care only about the ability of the system to mimic something like human intelligence. It's fair game to ask about planning problems, or learning problems, or other topics, so you could in some sense judge one technique to be better than another. However, most judgements made in the Turing Test are subjective, so it's not clear that it really solves the problem you posed.

But as per the Turing Test. The machine which behaves like a human that machine is an intelligent one.  This is what my understanding about the Turing Test

@MaheshwarLigade That's a popular definition, but I think most AI'ers would disagree at a least a little. The problem is: what does "behave like a human" mean? Who decides that? It turns out that very simple programs like Eliza can meet some people's bar. On the other hand, very complex modern attempts at the Turing Test are still pretty easy for researchers to detect. Some of the more successful ones get by pretending to be 5 year olds (i.e. less intelligent!). Check out the Loebner Prize results for more.


text:
How to architect a network to find bounding boxes in simple images?
I have an application where I want to find the locations of objects on a simple, relatively constant background (fixed camera angle, etc). For investigative purposes, I've created a test dataset that displays many characteristics of the actual problem.
Here's a sample from my test dataset.

Our problem description is to find the bounding box of the single circle in the image. If there is more than one circle or no circles, we don't care about the bounding box (but we at least need to know that there is no valid single bounding box).
For my attempt to solve this, I built a CNN that would regress (min_x, min_y, max_y, max_y), as well as one more value that could indicate how many circles were in the image.
I played with different architecture variations, but, in general, the architecture a was very standard CNN (3-4 ReLU convolutional layers with max-pooling in between, followed by a dense layer and an output layer with linear activation for the bounding box outputs, set to minimise the mean squared error between the outputs and the ground truth bounding boxes).
Regardless of the architecture, hyperparameters, optimizers, etc, the result was always the same - the CNN could not even get close to building a model that was able to regress an accurate bounding box, even with over 50000 training examples to work with.
What gives? Do I need to look at using another type of network as CNNs are more suited to classification rather than localisation tasks?
Obviously, there are computer vision techniques that could solve this easily, but due to the fact that the actual application is more involved, I want to know strictly about NN/AI approaches to this problem.

There are some special architectures of CNNs which are designed exactly for the task you mention. The Detector library includes a collection of these architectures, this paper describes the Mask R-CNN network in detail, which is designed for image segmentation tasks.

Thanks for your answer! The Mask R-CNN, Faster R-CNN are interesting but they all seem to be tailored towards combining classification with bounding box detection - I don't see a simple and effective way to isolate the key elements for finding bounding boxes alone

In the R-CNN family approach, the BBox regression is performed using a specific module providing region proposal: initially the Selective Search has been used then Region Proposal Network has been introduced, which has the advantage, being a NN, to be trainable.
This has not to be confused with the BBox Regressor which comes after in the architecture and is focused on refining the proposal bboxes for the “Non Background” proposal objects
Please consider that to perform object detection properly, at least using a region proposal based architecture, you need to perform the basic “Background / Non Background” classification (which is where CNN features help) so to filter out non relevant proposals


text:
Which algorithms can we use on games with high branching factors (e.g. Connect6)?
Connect6 is an example of a game with a very high branching factor. It is about 45 thousand, dwarfing even the impressive Go.
Which algorithms can we use on games with such high branching factors?
I tried MCTS (soft rollouts, counting a ply as placing one stone), but it does not even block the opponent, due to the high branching factor.
In the case of Connect6, there are stronger AIs out there, but they aren't described in any research papers that I know of.

There are papers on the website: http://www.connect6.org/index.php/-mainmenu-58

Typically, Monte-Carlo Tree Search (MCTS) actually is the go-to "solution" for such problems with large branching factors. I can understand that "vanilla" MCTS may still have unsatisfactory performance, but there is a plethora of extensions/enhancements available.
I don't have experience with the specific game you mentioned (Connect6), but from a quick look at how the game works, I imagine there will be a huge number of transpositions in the search tree (positions that are the same but can be reached through multiple different paths in the search tree). This will especially be very common if you treat placing one stone as a single ply; every "combined move" (of placing two stones in two positions subsequently) can be reached in two different ways, simply by switching the order in which the player places them. There has been research in using Transposition Tables with MCTS, so that may be a promising direction to look into.
I also suspect there will be great value in using Deep (Reinforcement) Learning approaches. If there is a large board on which to place stones, there will likely be many moves that are "absurd" and can easily be dismissed altogether by Deep Learning approaches (e.g., placing stones far away in a corner of the board where none of the "action" is going on). Vanilla MCTS, without Deep Learning extensions, will not be able to recognize and dismiss such absurd moves, and play them way too often (in Play-Out but also Selection phase due to the high branching factor). The most obvious source of inspiration here would be AlphaGo Zero.
Finally, there's definitely some published research on Connect6 AI (and even MCTS in Connect6). For example: Two-Stage Monte Carlo Tree Search for Connect6. You can likely also find more relevant research by checking that paper's list of References, and checking later papers on google scholar that cite this one.


text:
Would YOLO be able to detect objects in "different" positions?
I have the following question about You Only Look Once (YOLO) algorithm, for object detection.
I have to develop a neural network to recognize web components in web applications - for example, login forms, text boxes, and so on. In this context, I have to consider that the position of the objects on the page may vary, for example, when you scroll up or down.
The question is, would YOLO be able to detect objects in "different" positions? Would the changes affect the recognition precision? In other words, how to achieve translation invariance? Also, what about partial occlusions?
My guess is that it depends on the relevance of the examples in the dataset: if enough translated / partially occluded examples are present, it should work fine.
If possible, I would appreciate papers or references on this matter.
(PS: if anyone knows about a labeled dataset for this task, I would really be grateful if you let me know.)

think of any classification and localization tasks : objects are never at the same position, but the network is able to find these objects well. Why sould it be different for your task ? :)

You can check this playlist for the needed information: https://www.youtube.com/playlist?list=PLKHYJbyeQ1a3tMm-Wm6YLRzfW1UmwdUIN

Some authors distinguish between translational invariance (the network's output isn't affected by the object appearing in a different part of the image, at an image classification task) from translational equivariance (the identified bounding box moves along with the moved image).

As I know about the YOLO, its algorithm splits the whole picture into many small frames and performs classification and boundaries detection at once for every frame, so that the location of the object does not matter.

As you said, a CNN would be able to detect objects in different positions if the dataset contains enough examples of such cases, though the network is able to generalize and should be able to detect objects in slightly changed positions and orientations.
The term "translation invariance" does not mean that translating an object in the image would yield the same output for this object, but that translating the whole image would yield the same result. So the relative position of object IS important, modern CNN's takes decisions on the whole image (with strong local cues, of course).
To maximize the ability of your CNN to detect multiple orientation, you can train with data augmentation that rotate the images.
the same reasoning can be applied to partial occlusions: if there are enough samples with occlusion in the training set the network should be able to detect those ones. The network ability to generalize should also help a little when occlusions are small, and still be able to detect the object.
Some papers tried different experiment to demonstrate the robustness to occlusion and translation, for instance by looking at the network activation when artificially occluding a portion of the image with a gray rectangle, though I do not have a paper name in mind.

This is a good question because unlike Faster R-CNN RPN YOLO "proposals" arise from a fully connected layer instead of a convolutional one
The parameters of each "7x7 feature map" in the 7x7x30 YOLO detection layer are not shared which implies an image should be translated to present each of its objects to each of the 7x7 positions to ensure translation invariance
This is related to its extensive requirement for data augmentation including translations, scaling, exposure and saturation where the R-CNN family of algorithms just use random horizontal flips
This was modified in Darknet-19 in which the last layer is convolutional when modified for detection


text:
Is my interpretation of the return correct?
Sutton and Barto 2018 define the discounted return $G_t$ the following way (p 55):

Is my interpretation correct?

Or should all "1" be in the same column?

Your table is almost correct. It is a minor difference, you should not have a $R_0$, the top row, leftmost column of numbers should be empty. That is because the first reward is $R_1$ (a result of taking action $A_0$ in state $S_0$). The alignment of the columns on the right hand side is correct though.
It might help to add the time step number at the top. But the important detail is that $G_t$ is a measure of all future rewards.
For instance it should always be zero when you reach a terminal state, which is what your example shows. Whilst it is quite common to receive a reward at the end of an episode (i.e. whilst arriving at a terminal state), also as your example shows.
The decision to have reward time step match that of next state is a convention that can be altered. A few RL sources, but not Sutton & Barto, will have reward on same time step as state and action that decided it, and thus $R_0$ will exist. The reward of 1 for reaching the terminal state would then be 1 time step earlier in your table, and there would be no $R_4$. The definition of $G_t$ would need to change to match ($G_t = R_t + \gamma G_{t+1})$, as well as other equations. That would change your table also - the reward sequence (top row) would shift to the left.


text:
Is there a difference in the architecture of deep reinforcement learning when multiple actions are performed instead of a single action?
I've built a deep deterministic policy gradient reinforcement learning agent to be able to handle any games/tasks that have only one action.  However, the agent seems to fail horribly when there are two or more actions.  I tried to look online for any examples of somebody implementing DDPG on a multiple-action system, but people mostly applied it to the pendulum problem, which is a single-action problem.
For my current system, it is a 3 state, 2 continuous control actions system (One is to adjust the temperature of the system, the other one adjusts a mechanical position, both are continuous).  However, I froze the second continuous action to be the optimal action all the time.  So RL only has to manipulate one action.  It solves within 30 episodes.  However, the moment I allow the RL to try both continuous actions, it doesn't even converge after 1000 episodes.  In fact, it diverges aggressively.  The output of the actor-network seems to always be the max action, possibly because I am using a tanh activation for the actor to provide output constraint.  I added a penalty to large actions, but it does not seem to work for the 2 continuous control action case.
For my exploratory noise, I used Ornstein-Ulhenbeck noise, with means adjusted for the two different continuous actions.  The mean of the noise is 10% of the mean of the action.
Is there any massive difference between single action and multiple action DDPG?
I changed the reward function to take into account both actions, have tried making a bigger network, tried priority replay, etc., but it appears I am missing something.
Does anyone here have any experience building a multiple-action DDPG and could give me some pointers?

Technically, the difference here is between actions in (some subset of) $\mathbb{R}$ and $\mathbb{R}^n$, not between 1 or more "actions". In other words, you have an action space here that might have multiple dimensions, and something is going wrong for your agent when there are 2 or more dimensions. In RL, when something described as "having 2 actions" this is usually an enumeration - i.e. the agent can take action A or action B, and there are no quantities involved.

Hi Neil, thanks for the reply.  Yes, for classic RL, it agents' actions are indeed discrete.  However, in 2015, Lilicrap published a paper called "continuous control with deep reinforcement learning", and then in 2017, the TRPO and PPO algorithms were designed to allow agents to perform multiple continuous actions.  So you are correct about my action being in a high dimension space.  In my research, I am comparing model predictive control using trajectory optimization vs AI-based control.  Usually, in robotics and mechatronics, robots move multiple pieces.  I am trying to achieve that with RL.

Current robots, such as the ones in the Tesla megafactory, have huge flaws in some tasks.  I truly believe that with the new RL architectures, we can try to close these gaps.  All the input here are welcome, and thanks so much guys for all the help!

I suggest you [edit] a more accurate description of your RL problem to replace the sentence "For my current system, it is a 3 state, 2 action system." - because that is not how it would be described in any literature. May also be worth explaining how you have adjusted the exploration function ("actor noise"), as a mistake there would be key.

Done!  I will also try different exploratory noise means to see if it helps.

Thanks. I was wondering if you had somehow failed to adjust for different scales of the two axes of action, but it doesn't look like it. I cannot really tell what is wrong. However, I would not personally expect DDPG to be quite so fragile when scaling up from one to two dimensions of action, so I'd still suspect something about your implementation - I just don't know what it could be.

No worries Neil, I believe it has something to do with the difficulty of comprehension for the mechanical movement.  I will try new ways for my RL to interpret the actions and rewards.


text:
Why does the fitness of my neural network to play tic-tac-toe keep oscillating?
I wrote a simple feed-forward neural network that plays tic-tac-toe:

9 neurons in input layers: 1 - my sign, -1 - opponent's sign, 0 - empty;
9 neurons in hidden layer: value calculated using ReLU;
9 neurons in output layer: value calculated using softmax;

I am using an evolutionary approach: 100 individuals play against each other (all-play-all). The top 10 best are selected to mutate and reproduce into the next generation. The fitness score calculated: +1 for the correct move (it's possible to place your sign on already occupied tile), +9 for victory, -9 for a defeat.
What I notice is that the network's fitness keeps climbing up and falling down again. It seems that my current approach only evolves certain patterns on placing signs on the board and once random mutation interrupts the current pattern new one emerges. My network goes in circles without ever evolving actual strategy. I suspect the solution for this would be to pit network against tic-tac-toe AI, but is there any way to evolve the actual strategy just by making it to playing against itself?

Are you using any form of cross-over between two parents in reproduction?

I do, I tried to use mutation only and crossover + mutation, yet results are essentially same. I doubt that this is the problem.

Yes, I was asking because cross-over in neural networks may work better using a system like NEAT which can track compatibility between architectures as they grow. It is hard to tell where your problem might be, but combining GAs and NNs is quite tough, the search space is complex due to co-dependencies between weights (making cross-over fail unless care is taken). And the space also becomes very large quickly (making random mutations inefficient for searching). I guess your architecture has 180 parameters, weights + biases, to find optimal values for?

Yes. I understand why this approach doesn't work, but it's hard for me to it put in technical terms. Basically, there is no emerging correct strategy since network only plays against different variations of itself, playing against AI that knows optimal strategy would teach network to make correct moves, but I want to find a solution without this approach, so I could use network in situations where there is no known optimal strategy.

The design should work in principle for self-play. It is a matter of details, and what you can change to enable it in a reasonable time. I don't know the answer here. But one thing you could try is see whether the network can learn your desired function. Create a table of all the correct moves, and use normal supervised learning with gradient descent. Can the network get a high accuracy when told the answer? If it can, then you have ruled out the network architecture as a problem. If it cannot, then I would probably try a deeper network - keep adding layers until it works.

Of course in a non-trivial game, you won't be able to create this supervised learning data, but you may start to get a feel for correct NN architecture.

One thing: How are you matching up genomes to play each other and get a fitness rating in each generation? Is it all-play-all (so an evaluation round is 4950 games), a single random pairing (evaluation round is 50 games), or something in-between or different? Please [edit] that into the question.

From where I stand you it seems that you miss some new blood in your strategy. Have look here : https://www.tutorialspoint.com/genetic_algorithms/genetic_algorithms_parent_selection.htm if you always select top elements to reproduce,you will converge too quickly which seems to be the case. Hope this helps

http://eng.uber.com/wp-content/uploads/2017/12/improving-es-arxiv.pdf
To explore more when stuck in local optimum

What I notice is that the network's fitness keeps climbing up and falling down again. It seems that my current approach only evolves certain patterns on placing signs on the board and once random mutation interrupts current pattern new one emerges. My network goes in circles without ever evolving actual strategy. I suspect solution for this would be to pit network against tic-tac-toe AI, but is there any way to evolve actual strategy just by making it to playing against itself?

The likely cause of this phenomenon is that your fitness function involves evaluating the fitness of an agent by letting it play lots of other agents (the entire population), many of which are likely very poor agents.
Because Tic-Tac-Toe is such a simple game, we know that optimal play from both sides leads to a draw. Suppose we have a population of the following three strategies:

$\pi_1$: an optimal player
$\pi_2$: a sub-optimal player
$\pi_3$: a different sub-optimal player

There can easily be situations where the optimal player $\pi_1$ gets a draw against both of the sub-optimal players (if they're not extremely bad), because right from the get-go an optimal player will play "safe" enough such that it can still guarantee a draw against an optimal opponent, which may not be the fastest way to win against a sub-optimal player.
In the same situation, the sub-optimal player $\pi_2$ for example may be able to consistently win against a slightly worse sub-optimal player $\pi_3$. In this example situation, your fitness function ranks the agents $f(\pi_2) > f(\pi_1) > f(\pi_3)$, which is wrong.

As you already suggested yourself, the most straightforward way to address this problem would be to simply evaluate the fitness of strategies by having them play against an optimal minimax-agent, rather than a population of many strategies (including poor ones).
If you really want to use only evolving, no tree search, you'll have to find a way to fix the fitness function such that the problem described above can no longer occur. One way you could try to do that (not 100% sure it would work, but imagine that it might) would be to set up some larger tournament bracket where agents progress through the brackets if they're able to beat others which they were paired up with. Getting further in the tournament would then increase the fitness of an agent. Very bad sub-optimal players should not be able to progress far in the tournament if they get beaten by other (sub-)optimal agents, but sub-optimal agents which manage to advance still shouldn't be able to get more than a draw against an optimal agent. Some things to keep in mind with this idea:

You'll likely want to repeat the entire tournament many times with different, randomized initial pairings in the bracket, and compute fitness based on average (or maybe median or max) ranking across all repetitions of the tournament. This would be to filter out members of the population getting particularly lucky or unlucky with the opponents they encounter in the brackets.
You'll have to think about which actions the agents select in such a tournament. Do they deterministically play the action given by the $\arg\max$ of the softmax outputs (in which case you could also just use the linear outputs rather than softmax outputs, since the softmax function does not change ranking of outputs). Or do they nondeterministically sample actions according to the softmax distribution? Sampling actions according to the softmax distribution seems attractive because it leads to more variance in game state situations encountered, and it is important to be robust and be capable of handling many different game states. On the other hand, it does introduce noise and may make an optimal agent accidentally lose. I suppose I would lead towards deterministic play with the $\arg\max$ over the softmax outputs. In a large tournament, there will still be sufficient variety due to encountering many different opponents.
You'll have to put thought into handling the situation where two agents may infinitely keep drawing against each other. Which agent advances? If this happens, I think I would gradually shift from deterministcally playing $\arg\max$ actions to playing according to the softmax distribution. This guarantees that someone eventually loses. Averaging results over multiple different games like this, plus averaging over multiple different repetitions of the complete tournament, should yield accurate results.
You'll want to think about how to handle illegal actions. You describe including punishments for illegal actions in the fitness function. This introduces extra noise / variance in the fitness function which is already difficult enough to get right to begin with, so I wouldn't do this. I'd recommend having a manual post-processing step in which you manually set probabilities of illegal actions being played to $0$, and normalizing to make sure the probabilities over the remaining actions add up to $1$ again. This is also what's done (on a much larger scale) by DeepMind in their state-of-the-art Go/Chess/Shogi agents for example. If you really don't want to do this, if you really want evolved strategies to automatically learn not to ever have high outputs for illegal actions, I would recommend immediately making them lose the game if they do suggest an illegal action. That way you still have a "clean" fitness function based only on wins/draws/losses.

The characteristics of the game are these.

Moves are in two dimensions
Only three possible first moves because of symmetry (corner, edge, middle)
Never more than seven possible moves after that
No random game play element (such as a roll of dice)
No secrets (such as a hand of cards)
No way to lose with perfect play
Only one two-part rule: (1) Must make your mark (2) in an unmarked cell
Only one winning condition: Three in a row of your mark

The goal strategy is to get two marks such that a third will require two blocks in one opponent move. A three layer network will be able to learn that strategy. Because the game complexity is low, a mutation is always going to have a radical effect on game play. This means that the changes due to the mutation must be gradual and must stop when the strategy is found. This implies that stateful learning is best.
The problem is not related to human versus self play.
Since there is exactly one mark to make per turn, soft max doesn't make much sense for the output layer. You want binary outputs — threshold activation functions in the last layer.  Use two pair of binary output cells, one pair for each dimension.


text:
What is the difference between visible and hidden units in Boltzmann machines?
What is the difference between visible and hidden units in Boltzmann machines? What are their purposes?


text:
Can we teach an artificial intelligence through sentences?
Could we teach an AI with sentences such as "ants are small" and "the sky is blue"? Is there any research work that attempts to do this?

You may want to look at the symbol grounding problem.  Significant thinkers don't think automata will need to understand to imitate, but are you asking about an NN that actually understands the meaning of the symbols (words)?  I was reading something recently suggesting that mathematics itself is form of "language game". If you're talking syntactics, it doesn't seem to be out of reach, but if you're talking semantics, it's a whole different ballgame.

I think that what you're really asking about is the question of knowledge representation.  Regardless of how you train your AI, one of the most fundamental questions is how do you represent "knowledge" and especially when it exists at different levels of abstraction, may be mutually recursive, etc.  Along with that goes the question of belief revision which deals with how you update existing beliefs/knowledge in the light of new information.
Both of these areas are still subject to plenty of active research and neither has entirely settled answers to the core questions.  But progress has been made in both areas.
Personally I suspect that something like semantic networks or conceptual graphs will be the best answer to the KR problem.  Dealing with belief revision seems even fuzzier to me, although there are known strategies (like the AGM postulates) that work to a point. Something like Bayesian Belief Networks may also prove useful.

I have recently watched a podcast by Lex Fridman where he interviews Vladimir Vapnik, who talks about this way of teaching with "predicates" and "invariants".
If you were really able to teach a model with sentences like "a bird is an animal that is able to fly" but "not all birds fly, such as penguins", that would probably represent one of the biggest milestones in machine learning, especially, if the machine was able to learn and apply this learned knowledge as efficiently as humans do. However, we are not quite there yet!
To know more about this new learning paradigm Learning Using Statistical Invariants (LUSI), you probably should read the paper Rethinking statistical learning theory: learning using statistical invariants (2019), by V. Vapnik and R. Izmailov, which will probably be difficult to follow if you have no knowledge of learning theory and your mathematical background is poor. However, there are some sections of this paper that are accessible to everyone. For example, section 6.6

Suppose that the Teacher teaches Student to recognize digits by providing a number of examples and also suggesting the following heuristics: "In order to recognize the digit zero, look at the center of the picture — it is usually light; in order to recognize the digit 2, look at the bottom of the picture - it usually has a dark tail" and so on.
From the theory above, the Teacher wants the Student to construct specific predicates $\psi(x)$ to use them for invariants. However, the Student does not necessarily construct exactly the same predicate that the Teacher had in mind (the Student's understanding of concepts "center of the picture" or "bottom of the picture":  can be different). Instead of $\psi(x)$, the Student constructs function $\hat{\psi}(x)$. However, this is acceptable, since any function from $L_2$ can serve as a predicate for an invariant.


text:
Is there a machine learning system that is able to understand mathematical problems given in a textual description?
Is there a machine learning system that is able to "understand" mathematical problems given in a textual description, such as

A big cat needs 4 days to catch all the mice and a small cat needs 12 days. How many days need both, if they catch mice together?

?

There was a lot of work on this topic at UT Austin, which has now migrated to the Alan Institute.
There is no off-the-shelf software that will answer your question (if there was, DARPA would stop funding its development!), but you can read about the latest development in a number of recent papers.
This paper (Seo et al. EMNLP 2015) discusses the techniques that are used to interpret diagrams that accompany geometry problems, while this one (Hosseini et al. EMNLP 2014) talks about how to automatically parse verbs to interpret the meaning of a question. The 2015 TACL paper (Koncel-Kedziorski et al. 2015) completes this by discussing how to extract the relevant equations from a word problem. Once you have the equations, know what question is being asked, and can interpret any diagrams, you can do most high school math problems.
However, I don't think this is yet a fully reliable system. It is one part of a larger, long running effort to create a program that can achieve higher education certifications in many subjects. You can see many projects related to this at the Alan Institute's website.

Well this is a relatively new problem very tied to Question Answering. One of the recent systems is EUCLID that can answer those type of question the public Dolphin algebra question set by using a tree transducer cascade approach.
This paper details the proposed model Hopkins, M., Petrescu-Prahova, C., Levin, R., Le Bras, R., Herrasti, A., & Joshi, V. (2017). Beyond sentential semantic parsing: Tackling the math sat with a cascade of tree transducers. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (pp. 795-804).
In the same sense, SEMEVAL has released a task related to Math QA, you can see the related bibliography and referenced works semeval 2019 task 10 internal.

Welcome to AI.SE andres! I like the paper you've cited, but I think your answer could be improved. Question answering is a very old problem. Check out this paper from 1969 for example: http://www.dtic.mil/docs/citations/AD0696394. The rest of your answer looks pretty good though.

QA pretty Old!!! yes sure. But it is a an open problem and it is very interesting. You can see how it is important for Automatic Assistants, Conversational AI platforms, Knowledge Bases QA, QA over linked data, Qa over hybrid sources. You can also find old systems as LUNAR QA (1977), BASEBALL QA (1961) and new systems as WATSON (2011), MSRA QA, etc.

Those look like great sources! You should consider editing your answer to incorporate them.


text:
Why is the log probability replaced with the importance sampling in the loss function?
In the Trust-Region Policy Optimisation (TRPO) algorithm (and subsequently in PPO also), I do not understand the motivation behind replacing the log probability term from standard policy gradients
$$L^{PG}(\theta) = \hat{\mathbb{E}}_t[\log \pi_{\theta}(a_t | s_t)\hat{A}_t],$$
with the importance sampling term of the policy output probability over the old policy output probability
$$L^{IS}_{\theta_{old}}(\theta) = \hat{\mathbb{E}}_t \left[\frac{\pi_{\theta}(a_t | s_t)}{\pi_{\theta_{old}}(a_t | s_t)}\hat{A}_t \right]$$
Could someone please explain this step to me?
I understand once we have done this why we then need to constrain the updates within a 'trust region' (to avoid the $\pi_{\theta_{old}}$ increasing the gradient updates outwith the bounds in which the approximations of the gradient direction are accurate). I'm just not sure of the reasons behind including this term in the first place.

I am not 100% sure if the following is the only/complete story, but I'm quite confident it's at least part of the story:
In the PPO paper, after describing the standard policy gradient objective $L^{PG}$, they mention the following:

While it is appealing to perform multiple steps of optimization on this loss $L^{PG}$ using the same trajectory, doing so is not well-justified, and empirically it often leads to destructively large policy updates

This is because, as soon as you've performed one update using a trajectory generated with the previous policy, you land in an off-policy situation; the experience gained in that trajectory is no longer representative of your current policy, and all the estimators (like the advantage estimator) technically become incorrect.
With importance sampling, you can correct for this. This is also commonly used in multi-step off-policy value learning algorithms. Intuitively, the importance sampling term emphasizes estimates of advantage $\hat{A}_t$ corresponding to actions $a_t$ that have become more likely in the new policy relative to the old policy, and it de-emphasizes advantages corresponding to actions that have already become less likely in the new policy relative to the old policy.
If an action $a_t$ in the old trajectory has already become highly unlikely since that trajectory of experience was generated, we have $\pi_{\theta} (a_t \vert s_t) < \pi_{\theta_{\text{old}}} (a_t \vert s_t)$, which means that $\frac{\pi_{\theta} (a_t \vert s_t)}{\pi_{\theta_{\text{old}}} (a_t \vert s_t)}$ becomes close to $0$, which means that we'll reduce the influence of that particular chunk of experience on our subsequent updates. This makes sense because, due to previous updates since the generation of that trajectory, that particular part of the trajectory has already become highly unlikely anyway, and should therefore no longer be relevant for our updates.
The ability to perform multiple updates using the same (old) trajectory anyway is useful because this increases sample-efficiency, we can re-use the same samples of experience more than once rather than using them once and then discarding them again.

For everybody getting here from google, like me: the $\log$ might have been replaced in the loss function, but I think it is still there when taking the gradient of both functions (correct me, if I am wrong):
$$\begin{aligned}
\nabla_{\theta} L^{P G}(\theta) &=\nabla_{\theta} \hat{E}_{t}\left[\log \pi_{\theta}\left(a_{t} \mid s_{t}\right) \hat{A}_{t}\right] \\
&=\hat{E}_{t}\left[\nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) \hat{A}_{t}\right]
\end{aligned}$$
and
$$
\begin{aligned}
\nabla_{\theta} L^{I S}(\theta)=& \nabla_{\theta} \hat{E}_{t} \left[\frac{\pi_{\theta}\left(a_{t} \mid s_{t}\right)}{\pi_{\theta_{\text {old}}}\left(a_{t} \mid s_{t}\right)} \hat{A}_{t}\right] \\
&=\hat{E}_{t} \left[\nabla_{\theta} \frac{\pi_{\theta}\left(a_{t} \mid s_{t}\right)}{\pi_{\theta_{\text {old}}}\left(a_{t} \mid s_{t}\right)} \hat{A}_{t}\right] \\
&=\hat{E}_{t} \left[\frac{\pi_{\theta}\left(a_{t} \mid s_{t}\right)}{\pi_{\theta_{\text {old}}}\left(a_{t} \mid s_{t}\right)} \frac{\nabla_{\theta} \pi_{\theta}\left(a_{t} \mid s_{t}\right)}{\pi_{\theta}\left(a_{t} \mid s_{t}\right)} \hat{A}_{t}\right] \\
&=\hat{E}_{t}\left[\frac{\pi_{\theta}\left(a_{t} \mid s_{t}\right)}{\pi_{\theta_{\text {old}}}\left(a_{t} \mid s_{t}\right)} \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) \hat{A}_{t}\right]
\end{aligned}
$$
So, the $\pi_{\theta}\left(a_{t} \mid s_{t}\right) $ in the PG function was replaced with $\frac{\pi_{\theta}\left(a_{t} \mid s_{t}\right)}{\pi_{\theta_{\text {old}}}\left(a_{t} \mid s_{t}\right)} $ whose derivate is the same as the log of the PG function (apart from the proportionality factor).

Hi, nice intuition, but I think there is an error with your derivation on the 4th step in the second equation. When you take the $\nabla_{\theta} \pi_{\theta}(a_t|s_t)$, why isn't the derivative taken for all the other $\pi_{\theta}(a_t|s_t)$?

Nvm, I think your solution checks out.

Yep, so basically the $log$ is gone because importance sampling cancels it out with the gradient of $log$.


text:
Why do non-linear activation functions not require a specific non-linear relation between its inputs and outputs?
A linear activation function (or none at all) should only be used when the relation between input and output is linear. Why doesn't the same rule apply for other activation functions? For example, why doesn't sigmoid only work when the relation between input and output is "of sigmoid shape"?

It should, but the chance of data being related by sigmoid is thin. Also, sigmoid function doesn't map data to the an 's' shape- it squeezes data between 0 and 1.

It is correct to say that a sigmoid activation function would only work well as a model if the desired output is close to the sigmoid function applied to the input.  This is a trivial fact that applies to a single layer perceptron.  This is true for for the single layer case for any activation function, also a trivial fact.
When the layer number is between one and infinity (two or more), the theory bifurcates.  The identity function becomes a special case:  Any number of layers that conform to a first degree polynomial, $ax + b$, can be replaced with a single identity layers.  The alternative case, where there are multiple layers that functionally do not conform to a first degree polynomial, $ax + b$, cannot be replaced by a single layer of some equally simple function.  The complexity increases geometrically, which is the entire point of multilayer perceptrons.
Under particular constraints, the multilayer perceptron can produce a wide variety of functional behaviors that do not resemble the activation functions of the layers.
For instance, a properly trained network using sigmoid activation functions, with sufficient layer depth and sufficient massive allocation of computing resources, could theoretically approximate the topography of the Himalayas.

Because any number of linear layers can be represented by a single linear layer:
$A_2*(A_1*x +b_1)+B_2 = A_2*A_1*x+ A2*b_1+b_2$
The same is not true if you have non-linear functions.


text:
What kinds of problems can AI solve without using a deep neural network?
A lot of questions on this site seem to be asking "can I use X to solve Y?", where X is usually a deep neural network, and Y is often something already addressed by other areas of AI that are less well known?
I have some ideas about this, but am inspired by questions like this one where a fairly wide range of views are expressed, and each answer focuses on just one possible problem domain.
There are some related questions on this stack already, but they are not the same. This question specifically asks what genetic algorithms are good for, whereas I am more interested in having an inventory of problems mapped to possible techniques. This question asks what possible barriers are to AI with a focus on machine learning approaches, but I am interested in what we can do without using deep neural nets, rather than what is difficult in general.
A good answer will be supported with citations to the academic literature, and a brief description of both the problem and the main approaches that are used.
Finally, this question asks what AI can do to solve problems related to climate change. I'm not interested in the ability to address specific application domains. Instead, I want to see a catalog of abstract problems (e.g. having an agent learn to navigate in a new environment; reasoning strategically about how others might act; interpreting emotions), mapped to useful techniques for those problems. That is, "solving chess" isn't a problem, but "determining how to optimally play turn-based games without randomness" is.

I was hoping to see more answers here, but I'll get us started with some examples:
Combinatorial Search Problems: If your problem can be phrased as movement through a combinatorial graph, you don't need a neural network. In particular, your problem should have discrete states, a clear set of actions that are possible in each state, a clear definition of where we start, and a clear definition of what the goal state looks like. The most effective general purpose technique is iterative deepening search. If you have an idea about which moves might be more effective, or better, a function that estimates how far each state is from the goal, you may be able to build a heuristic function and use A* search instead. Common applications for these techniques include pathfinding in video games (or directions in other applications), AI planning, and Automated Theorem Proving.
I'll add some more topics later, but I suspect others have expertise to share here. Let's see some more ideas!

A nice example Markov Decision Processes, which can be solved by classic reinforcement learning techniques like Q learning.
A Markov Decision Process consists of

A set of discrete states (or continuous states that have been discretized)
A set of possible actions that can be taken in each state.
A set of transition probabilities that describe how an agent stochastically moves from its current state to the next, based on the agent's actions.
A reward function quantitatively describing how nice it is to be in each state.
A discounting factor that describes how much worse it is to receive a reward in the future than today.

Very small MDPs can be directly, exactly, solved, using techniques like value iteration, but the computational cost for these approaches grows extremely fast.
Reinforcement Learning (RL) was developed as a machine learning approach for MDPs. There is a loop: the agent gets the state of the environment, chooses an action, executes this action on the environment, and he gets back a reward, and the new state of the environment, and so on... You want the agent to maximize the cumulative reward over time.
The basic concept of Q Learning doesn't use ANNs. In Q learning, you build a state-action matrix, called the Q matrix. Thus, you must discretize the states of your environment, and the actions available to your agent. Then, the coefficient Qij is the expected reward when you perform the action j on the state i. In basic Q Learning, you explore and build this matrix, and it should converge and give an "optimal rule of action" for your agent.
However, the situation is often too complex, and you often want a non-discretized space of states or actions. Here Deep QL arrives, where the Q matrix becomes an ANN.
You can find a nice QL tutorial here (normal and deep).
And a lecture about QL here.
Keep in mind that only ANNs perform well in complex situations, so you'll always see examples with ANNs, even if the basic theory doesn't require ANNs.

RL/MDP is really a bad example of a problem that doesn't require a neural network, given that the main recent successes in RL use neural networks.

Image Segmentation with Unsupervised Learning
Deep Learning is now widely used for image classification and segmentation. However, for segmentation, some algorithms are still really effective. For example, they could also be used for the development of self-driving cars.
K-means for image segmentation
When you identify the pixels of an RGB image to vectors in $\mathbb{R}^3$, you can run the classic k-means algorithm to distinguish objects. Furthermore, you can do superpixel segmentation, by adding to all pixel vectors two components corresponding to their coordinates in the image (so it will be vectors in $\mathbb{R}^5$). You can run again a k-means algorithm to segment your image in superpixels. You can read about that SLIC Superpixels Compared to State-of-the-Art Superpixel Methods (2012), Achanta et al.
Example
Below is an example of the segmentation of picture of a seagull on a roof. On the left, we have the original image. In the middle, 3 clusters. On the right, 12 clusters. If it easily distinguishes the roof from the sky, the seagull is still unclear with 12 centroids.

Similarity graph and normalized cut
The main idea is to build a graph of similarities between pixels and then to cut the graph into subgraphs. First, you need to define a distance between pixels. For example, the colour dissimilarity, that could be   $d(p_1, p_2) = exp(-\sum{(p_{1,i} - p_{2,i})^2}), i \in (r, g, b)$. Then, build the graph over the whole image, and divide it iteratively, using the Normalized Cut algorithm.
Morphological gradients
Here, image segmentation is done by computing the morphological gradient of the image. It is the difference between the dilatation and the erosion of the input image. Erosion and dilatation are equivalent to passing min() and max() filters over all subwindows of a given size.
An example, still with the seagull. (Morphological gradient computed on a grayscaled input). The seagull appears more clearly.

Nice. Something to think about adding might be DBScan along with k-means, but this is exactly the sort of answer I was hoping to see here.

Ok, I added some illustrations. I'm not enough familiar with DBScan to write about it, but feel free to do it If you want


text:
Can deep successor representations be used with the A3C algorithm?
Deep Successor Representations(DSR) has given better performance in tasks like navigation, when compared to normal model-free RL tasks. Basically, DSR is a hybrid of model-free RL and model-based RL. But the original work has only used value-based functions deep RL methods like DQN.
Can deep successor representations be used with the A3C algorithm?


text:
Why is my calculation of the probability of an object being in a certain class incorrect?
In the attached image

there is the probability with the Naive Bayes algorithm of:

Fem:dv/m/s Young own Ex-credpaid Good ->62%

I calculated the probability so:
$$P(Fem:dv/m/s \mid Good) * P(Young \mid Good)*P(own \mid Good)*P(Ex-credpaid \mid good)*P(Good) =  1/6*2/6*5/6*3/6*0.6 = 0,01389$$
I don't know where I failed. Could someone please tell me where is my error?

Your probability hasn't been normalized!
In this case, you are computing the probability of being good, given that the other features have a fixed value. To obtain the correct probability, you need to normalize (divide) the value from your calculation by the probability that the features have taken on those fixed values.
You can calculate this as follows:
 $$P(Fem:dv/m/s, Young, own, Ex-credpaid) = \\ \sum_{x \in \{good,bad\}} P(Fem:dv/m/s, Young, own, Ex-credpaid, x) $$
by the marginalization rule.
Then, by the chain rule, you may write:
$$\\ \sum_{x \in \{good,bad\}} P(Fem:dv/m/s, Young, own, Ex-credpaid | x) * P(x) $$
So the correct probability of 0.62 should be obtained by the equation:
$$ \frac{P(Fem:dv/m/s, Young, own, Ex-credpaid | good) * P(good)}{\sum_{x \in \{good,bad\}} P(Fem:dv/m/s, Young, own, Ex-credpaid | x) * P(x)}$$
You just need to calculate
$$P(Fem:dv/m/s, Young, own, Ex-credpaid | bad) * P(bad)$$
and it should be easy to compute the rest.

thanks for the Answer, but I dont have the same result of 0.62. when I doing: P(Fem:dv/m/s | bad)P(Young | bad)P(own | bad)P(Ex-credpaid | bad)P(bad) -> 2/43/41/42/40.4= 0,01875 and 0,01389/(0,01389+0,01875)=0,4255
@John Doucette

Hmm. You are correct.  Do you have more context for that table? Where does it come from?

the table comes from this paper:
(page 3)
https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4909197

Hmm. It looks like there are several things at play. First, I think the authors may have used Laplacian Smoothing. This is a usual thing to do, but it changes the calculation. Basically, you add one additional observation of each kind to each of the conditional distributions to make a uniform prior, and then continue to count these when computing the posteriors. Even doing that however, I only get 50%. It's possible that the table is simply incorrect.

I hear, that the probability was false, and I do the same algorithm in Python and I dont know how do you did the naivebayes algorithm, how do you did?

@TomaateTip I'm not sure I understand what you mean. I think the probability in the table is incorrect, and doing Naive Bayes the way you described (plus the normalization in my answer) will yield the correct probabilities.

yes, is incorrect, but you said that you get 50%, do you programmed it? when yes, how?

No, I computed it by hand.


text:
How can the Cart Pole problem be a continuing task?
In Introduction to Reinforcement Learning (2nd edition) by Sutton and Barto, there is an example of the Pole-Balancing problem (Example 3.4).
In this example, they write that this problem can be treated as an episodic task or continuing task.
I think that it can only be treated as an episodic task because it has an end of playing, which is falling the rod.
I have no idea how this can be treated as continuing task. Even in OpenAI Gym cartpole env, there is only the episodic mode.

If you could balance the rod indefinitely then it would be continuing

It's a continuing task in that, after failure, the agent always gets a reward of $0$ at each time-step ad infinitum.
From the book:

we could treat pole-balancing as a continuing task, using discounting.
  In this case the reward would be -1 on each failure and zero at all
  other times. The return at each time would then be related to $-\gamma^K$, where $K$ is the number of time steps before failure.

(Here I have used $\gamma$ as the discount factor).
Said another way, assuming the agent fails in the (K + 1)th step the reward is $0$ till that step, $-1$ for it, and then $0$ for eternity.
So the return: $$G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + ... + \gamma^K R_{t+K+1} + ... = -\gamma^K$$

The quote and the description of it do not match here. In your text, you are describing the addition of absorbing states that allow an episodic task to be treated like a continuing one mathematically. Whilst the S&B quote is about creating an actual continuing task with resets from the episodic task (i.e. there are no absorbing states, and no episodes as such). The difference is subtle, but important in the context of the question.

The key is that reinforcement learning through something like, say, SARSA, works by splitting up the state space into discrete points, and then trying to learn the best action at every point.
To do this, it tries to pick actions that maximize the reward signal, possibly subject to some kind of exploration policy like epsilon-greedy.
In cart-pole, two common reward signals are:

Receive 1 reward when the pole is within a small distance of the topmost position, 0 otherwise.
Receive a reward that linearly increases with the distance the pole is off the ground.

In both cases, an agent can continue to learn after the pole has fallen: it will just want to move the poll back up, and will try to take actions to do so.
However, an offline algorithm wouldn't update its policy while the agent is running. This kind of algorithm wouldn't benefit from a continuous task. An online algorithm, on contrast, updates its policy as it goes, and has no reason to stop between episodes, except that it might become stuck in a bad state.

I just want to know "how can cartpole environment be continuing task even though it has an end of episode"...

end of episode = terminal state = falling off the poll

@user3595632 As mentioned in my answer, in the continuing version of the task, the episode does not end when the pole falls. With a suitable reward function, the agent can learn to rebalance the pole after it has fallen.

Then what happens to pole after it fallen?....Does it keep staying fallen state?

The state of the poll is real-valued (usually represented as an angle). When it falls, it leaves a narrow region that is defined as "balanced" (note, there isn't usually a single unique balanced state, it's a range of angle values). There's nothing special about this from the perspective of the state of the poll: it's just got a different angle than before. The angle will continue to change with the effects of gravity, the kinetic energy of the pole, and the force exerted by the agent's motor. The agent will receive a different (worse) reward until the pole re-enters the "balanced" region.

From Sutton & Barto's book (p. 56)

Example 3.4: Pole-Balancing The objective in this task is to apply forces to a cart moving along a track so as to keep a pole hinged to the cart from falling over: A failure is said to occur if the pole falls past a given angle from vertical or if the cart runs off the track. The pole is reset to vertical after each failure. This task could be treated as episodic, where the natural episodes are the repeated attempts to balance the pole. The reward in this case could be $+1$ for every time step on which failure did not occur, so that the return at each time would be the number of steps until failure. In this case, successful balancing forever would mean a return of infinity. Alternatively, we could treat pole-balancing as a continuing task, using discounting. In this case the reward would be $-1$ on each failure and zero at all other times. The return at each time would then be related to $-\gamma^{K-1}$, where $K$ is the number of time steps before failure (as well as to the times of later failures). In either case, the return is maximized by keeping the pole balanced for as long as possible.

In this case in Sutton & Barto, the authors are talking about removing the episode termination. They then treat the falling pole being reset to a given (originally the starting) state distrubution as a transition with a negative reward within a longer continuing problem. This is a change to the environment descripion, and it comes with some requirements, such as needing to use discounting*.
This is different to the "absorbing state" treatment used elsewhere to put episodic and continuing tasks on the same mathematical footing.
You might use this view of a problem in any environment where the goal is to maintain a steady state. It is in part motivated by the fact that even in the episodic framing of the problem, a perfect agent would never end an episode. However, converting an episodic problem to a continuing one with resets (from limited set of transitions) as part of state transition and reward scheme may be reasonable for other purposes too.

* Or the average reward setting, although they have not covered that option at that point in the book.


text:
Is there a way to translate the concept of batch size into reinforcement learning?
I am using a neural network as my function approximator for reinforcement learning. In order to get it to train well, I need to choose a good learning rate. Hand-picking one is difficult, so I read up on methods of programmatically choosing a learning rate. I came across this blog post, Finding Good Learning Rate and The One Cycle Policy, about finding cyclical learning rate and finding good bounds for learning rates.
All the articles about this method talk about measuring loss across batches in the data. However, as I understand it, in Reinforcement Learning tasks do not really have any "batches", they just have episodes that can be generated by an environment as many times as one wants, which also gives rewards that are then used to optimize the network.
Is there a way to translate the concept of batch size into reinforcement learning, or a way to use this method of cyclical learning rates with reinforcement learning?

Potentially.
If you do offline reinforcement learning, you're basically learning to approximate a function by sampling input/output pairs, rather than episode-by-episode. Here, your batch size could be set exactly as in an ordinary supervised learning problem.
If you do online learning, then it's not clear to me that the techniques used to set the learning rate in supervised learning can be directly applied though.
Both approaches are well covered in the RL chapter of Russell & Norvig (17? 18?).

From my understanding of reinforcement learning, you will have an agent and an environment.
In each episode, the agent observes the state $s$, takes some action action $a$, then gets some reward $r$, and finally observes the next state $s'$, and do it again and again until the end of the episode.
The above process does not incur any "learning". Then when and where exactly do you "learn"? You learn from your history. In traditional Q learning, the Q matrix is updated every time you have a new observation of $(s_t, a_t, r_t, s'_{t+1})$. Just like the supervised learning, you put in training sample one by one.
Similarly, you can feed in training samples in "batch" when you train, which means you "remember" the past $N$ observations and train them together. I think that is the answer to your question.
Furthermore, the past $N$ observations could have a strong correlation that you don't want. To break this, you may have a larger "memory" that stores many observations, and you only sample a few (this number is your new batch size) randomly every time you train your model. This is called experience replay.


text:
Is a multilayer perceptron a recursive function?
I read somewhere that a multilayer perceptron is a recursive function in its forward propagation phase. I am not sure, what is the recursive part? For me, I would see an MLP as a chained function. So, it would nice anyone could relate an MLP to a recursive function.

Are you sure the source was not referring to "recurrent neural networks"? Without having a link to the specific resource, it's difficult to answer this question, although both of the interpretations below look plausible to me.

Inherently, no. The MLP is just a data structure. It represents a function, but a standard MLP is just representing an input-output mapping, and there's no recursive structure to it.
On the other hand, possibly your source is referring to the common algorithms that operate over MLPs, specifically forward propagation for prediction and back propagation for training. Both of these algorithms are easy to think about recursively, with each node performing a sort of recursive call with its children or parents as the target, and some useful information about activations or errors attached. I actually encourage my students to implement it recursively for this reason, even though it's probably not the most efficient solution.

Sure, you can define plenty of things we don't generally need to regard as recursive as so. An MLP is just a series of functions applied to its input. This can be loosely formulated as
$$ o_n = f(o_{n-1})$$
Where $o_n$ is the output of layer $n$.
But this clearly doesn't reveal, much does it?