text:
How do I apply reinforcement learning to a game with infinitely many actions?
I am trying to figure out how to use a reinforcement learning algorithm, if possible, as a "black box" to play a game. In this game, a player has to avoid flying birds. If he wants to move, he has to move the mouse on the display, which controls the player position by applying a force. The player can choose any position on the display for the mouse. A human can play this game. To get a sense what needs to be done, have a look at this Youtube video.
I thought about using an ANN, which takes as input the information of the game (e.g. positions, speeds, radius, etc.) and outputs a move. However, I am unlikely to ever record enough training data to train the network properly.
Therefore, I was thinking that a RL algorithm, like Q-learning, would be more suited for this task. However, I have no clue how to apply Q-learning to this task. For example, how could the coder possibly know what future reward another move will bring?
I have a few questions:

In this case, the player has infinitely many actions. How would I apply RL to this case?
Is RL a good approach to solving this task?
Is there a handy Java library which would allow me to use a RL algorithm (as a block-box) to solve this problem?
Are there alternatives?

I can answer this, but there seems to be two questions here - a very basic one about how Q Learning works, and a more involved one about the size of the action space. Finding you a Java library is probably off-topic. Are you looking for a Java library because the game is written in Java, so this is a way to integrate it?

@NeilSlater Would you please specify what you want me to do, then I will change it.  I am sorry if the question offended you in any way.
What does it mean to reflect a question?

@TVSuchty I am already putting together an answer. I am not in any way offended. I want to check why you are asking for a Java library - I won't be able to find one for you, but I might be able to suggest something that works for you if I knew why you need that. Do bear in mind you have asked a quite broad question, about how to start a project, so any answers are also going to be broad pointers on how to get started.

I need a java-library because I started writing in Java and I do not know how to write something cross-language. While I think my Java is advanced so far that I can fluently "speak" the language, my python is not. I use engoc right now, it has something about RL
http://heatonresearch-site.s3-website-us-east-1.amazonaws.com/javadoc/encog-3.3/org/encog/ml/world/learning/q/QLearning.html
But I am not sure..

This looks like the kind of game that a Reinforcement Learning algorithm could learn to play. If you are learning from the game objects directly (as opposed to wanting to have the agent learn from viewing the screen), then it should be relatively straightforward to do so - by which I mean that it will involve some new concepts, a little maths, and maybe take you a few weeks to learn enough and run enough experiments on the AI to the point where you have something good. But at least you should not need a cluster of GPU servers to run your learner.

However, I have no clue how to apply Q-learning to this task. For example, how could the coder possibly know what future reward another move will bring?

That is a major part of RL - the algorithms that are part of RL all solve this issue. Therefore the coder does not need to know how to calculate future reward, just an immediate reward. For the example game, it looks like you can grant a reward of $+1$ for every frame where the player does not crash. An RL algorithm that can solve this environment should figure out through "trial and error" which actions will keep the game running.
Q Learning achieves this by keeping track of estimates of future value and updating them across time steps - if action $a_1$ in state $s_1$ consistently leads to a later state $s_2$ with a high value, then Q learning will update its estimate of the value of $Q(s_1,a_1)$ to be high as well. The estimates start very poorly, but updates work their way back through time steps as the agent learns, and provided things don't go wrong, eventually it will learn that one action will lead to higher value than another, and it be able to consistently choose it.
This is such a basic question that if you want to use RL, even if you find a library that implements the whole thing for you, that you clearly need to study some foundations of RL. There are a few different places you can do that. I can suggest two that I have used:

The book by Sutton & Barto. Reinforcement Learning: An Introduction
UCL lectures by David Silver. These include videos on YouTube.

Both are online, free and comprehensive introductions to the subject. You can find shorter, more direct introductions online if you search for e.g. "Q Learning Tutorial".
Be prepared to spend hours just learning enough about the topic that you understand how to structure your data and interface with the game in order to create a learning environment for your AI.

In this case, the player has infinitely many actions. How would I apply RL to this case?

This is a good question. Some algorithms are better than others. Q Learning cannot work with infinitely many actions. However, you could choose a discrete set of mouse points - e.g. maybe 64 - at different angles and distances from the player (red dot) and try and get the values of those. That would allow you to try with Q learning, and is worth considering because RL that properly solves large action spaces is harder to understand and work with. You could look into the DQN algorithm for this. In short, the DQN algorithm uses history of actions and their immediate results to train a neural network to estimate the values of each action.
If you really want to solve large action spaces properly, then you need to be looking into Policy Gradient methods. Two popular and effective PG algorithms are A3C and DDPG. These are tougher to learn and implement. They use neural networks to learn multiple functions, one of which is the policy - this directly selects which action to take, given a current state/observation.

Is RL a good approach to solving this task?

It is definitely possible to use RL to do what you want.

Is there a handy Java library which would allow me to use a RL algorithm (as a block-box) to solve this problem?

I could not find one on a brief search. To get started, you might want to look at java-reinforcement-learning which implements some of the algorithms from Sutton & Barto, so would be a good starting point. Most of the other Java packages I found covered basics and learning material, but would not be plug-and-play for your game.
If you do find a library, you will need to have learned enough basic RL to be able to follow how to use it.

Are there alternatives?

One alternative that might work well for your game environment, and be a little easier to understand and implement would be to combine a policy function (like a neural network) with an evolutionary algorithm that tries out variations of the policy. The go-to algorithm here is probably NEAT, and it is capable of solving simple game agents effectively. Your game should be a reasonable fit for NEAT because the scoring system of survival time makes for a good fitness function.
One thing that could really be worth trying before diving into the (very hard to understand) A3C and DDPG algorithms is the Cross-Entropy Method. It is a really simple RL agent that has some similarities to genetic algorithms (it selects "best" items, but instead of from the population, it is from random behaviour of the same individual), but doesn't always scale too well. However, it might work well for your game.

Thanks for your answer! I have 2 Months to get this all done. I am just a high-school student (in Germany), I hope I can crack the math. I was able to crack the math in BP, but I think this is one step higher
I already tried NEAT - I tried to develop an ANN from scratch using Neat. This ANN should just play the game. It is not possible. It takes way to long for NEAT, the computing time with 10000 individuals is just tremendous...

@TVSuchty: Might be worth trying cross-entropy method, which is somewhere in-between NEAT and policy gradient methods. It doesn't evolve a population, but trains itself directly from the last batch of best efforts.

Hey, I updated the comment. What level of education do you consider to be sufficient...

@TVSuchty: I would consider a full understanding of RL including all the maths to be equivalent to studying for a MSc. However, you don't need that - there are plenty of programmer's tutorials out there that should be accessible with A-level/High School maths. The main problem you will face is not so much of it will be in Java.

@TVSuchty: Try cross-entropy method first. It is no harder than NEAT to understand, probably easier in fact. Another thing you can try is recording humans playing the game and training a regular NN to emulate them. Comparing these methods might still make a good project, even if the result is not a good AI. Emulating a human as a starting point could also significantly improve convergence times for other policy searches, if you re-use that NN as a starting point.

Thanks for your evaluation! Then it can be done. I will try following your advice.
Have an awesome day.

Incredible, in depth explanation of how to carry out a RL project for a difficult game. I was also curious about the Action Space mapping.

@math_law The answer didn't go into specific action mapping for this game. Typically for policy gradient the policy network outputs the parameters of a probability distribution, which you can sample from. E.g. it might output the mean and variance for a Normal distribution for the steering direction. Some methods work with direct action output plus a noise function for exploration (DDPG). If you have a specific case, I suggest asking a new question on the site


text:
Is there a way of representing the minimax algorithm mathematically?
I have successfully figured out how the minimax algorithm works for a game like chess, where a game tree is used, and you assign a value to the terminal nodes and propagate that value up the tree.
Is there a way to represent this algorithm mathematically? If so, how would I go about showing it?

A simple google search gives plenty of results.
If you have a look at the entry in wikipedia for Minimax it has mathematical representations as well as some basic pseudocode and tree representations to help grasp the concept.
Proving it would be a matter of going through the regular methods of a mathematical proof and would probably be a bit complicated.  It is far easier to discuss this by way of graphs.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What is a good descriptor for similar objects?
I am developing an image search engine. The engine is meant to retrieve wrist watches based on the input of the user. I am using SIFT descriptors to index the elements in the database and applying Euclidean distance to get the most similar watches. I feel like this type of descriptor is not the best since watches have a similar structure and shape. Right now, the average difference between the best and worst matches is not big enough (15%)
I've been thinking of adding colour to the descriptor, but I'd like to hear other suggestions.

Why are you using SIFT?  Have you tried deep learning?

How to develop a program that can sort images by similarity is similar to the problem of how to develop a program that can sort words by how similar they look.
For example:
"theory" is more similar to "teoryyy" than to "abc".
What determines similarity of two words or images are these factors:

how many parts are common to both images
how many parts are new in either image
how many parts are missing in either image
how many parts are zoomed in/out or stretched.
how many parts are displaced

There may be some other rules. By combining these rules you can explain how similarity recognition works. This is at the core of intelligence.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Can artificial intelligence also make mistakes?
The intelligence of the human brain is said to be a strong factor leading to human survival. The human brain functions as an overseer for many functions the organism requires. Robots can employ artificial intelligence software, just as humans employ brains.
When it comes to the human brain, we are prone to make mistakes. However, artificial intelligence is sometimes presented to the public as perfect. Is artificial intelligence really perfect? Can AI also make mistakes?

Since the question was tagged with philosophy, I'll provide a philosophical perspective.
There is no definition of perfection. Just think about the silly example that is always brought up when we talk about autonomous cars: You are about to crash into a group of school kids and the AI's only chance to avoid them is by crashing into an old person. What should the AI do in order to react "perfectly"?
All actions that we take are based on our personal value system, so what is perfect to one person might be completely nonsense to another person. It might be possible to create an AI that always acts perfectly in terms of the value system you fitted it out with, but even in this case there are some caveats like:

reaction time: is the system fast enough to process all available information fast enough in order to take a fast action?
available information: not all information might be available with the system. Do you consider it acting imperfect, if the action of the system turns out to be wrong once you posses all information about the situation, even though some information will never be available to the system (e.g. Heisenberg's principle of uncertainty)

When it comes to the human brain, we are prone to make mistakes. However, artificial intelligence is sometimes presented to the public as perfect. Do artificial intelligent systems make mistakes?

Anything that exists outside of fiction that can be called AI and is not trivial is not perfect.
Examples:

Alexa: creepy laugh - not trivial, but not perfect
Tic-Tac-Toe / Connect four: perfect algorithms exist, but trivial (you can create a game tree)

The problem is that you didn't bother to define "perfect". What does it mean to be perfect? Natural selection favors living beings that are energy efficient. Remembering everything might be considered perfect, but it is certainly not efficient.
We tend to make mistakes because some skills - especially abstract/mathematical ones or long-term decision making are not supported by natural selection.
Another big group of traits that are not supported by natural selection are health traits that are after the age of reproduction / helping children to grow up. Specifically cancer above the age of 40.

Yes it's error prone, just like us humans.
But just like in chess it is just a whole lot better at dealing with it.
AI does not contain all the possible knowledge in the universe out of no where without interaction with the world and such it should need to make assumptions and test those assumptions making it prone to error.
If you're asking about current techniques within reinforcement learning, image detection,etc. Those techniques are error prone, just likes us humans. During training of those algorithms you have a trade off between how good it will generalize on new data and how correct it can answer your questions. (it will "memorize" the questions and not actually learn in the latter, called overfitting)


text:
Can artificial intelligence applications be hacked?
Can artificial intelligence (or machine learning) applications or agents be hacked, given that they are software applications, or are all AI applications secure?

You may appreciate this story about someone hacking an AI for revenge: Unchained: A story of love, loss, and blockchain (MIT Tech Review)

Perhaps what you are looking for is the notion of adversarial attacks on machine learning systems?

YES even noise added can fool any classifier - some may be trained on noised examples but you can always find a way although it's a bit hacking unknown algorithm so I would compare it to birthday technique

To answer your question, it really depends on the purpose of the Artificial Intelligence program.
For example, 4Chan has hacked a number of "Artificial Intelligent" bots, most notably was Microsoft's Twitter bot Tay. The general purpose of the bot was to parse what was tweeted at it and respond in kind, learning and evolving with each and every interaction.
Within 24 hours, 4Chan had corrupted Tay beyond repair, by teaching it racist and sexist terminology, ironic memes, sending it to shitpost tweets, and otherwise attempting to alter its output so much so that Microsoft had to remove it.
Now, the flaw with Tay was that it accepted any input, and learned off of that exclusively, without any interaction from the developers. Other bots have similar features, but they have checks in place that require human intervention to determine what is "quality" information to learn, and what is "bad" information to learn as to not pollute the global knowledge base of the bot.
These are just two examples of how Artificial Intelligence can be "hacked", but it ultimately comes down to how the programs are implemented.
You mention in one of your comments about Cellphone Artificial Intelligence such as Siri, and whether this technology can be hacked. The answer is - not really.
Siri learns based off of her global interactions - with limited user input allowed. You can ask Siri how to pronounce a name. When she pronounces it incorrectly, you can say to her "Siri that's now how you pronounce that". And she will provide you with a limited set of options of how you pronounce that name, and you have to choose which option sounds the best.
There was no option to allow a user to give Siri "bad" information, as she already populates the results for you, and you have to teach her based off of her list of options. To give Siri bad input, you would have to have access to Siri's global learning base, which we do not have access to, and alter how she accepts human interactions within the program - which would never happen due to too many moving parts within the iPhone update process, and would be caught before you were able to deploy your update.

Nice angle.  I was going to post: "One way to 'hack' a learning algorithm would be to feed it bad data."

Everything can be hacked. The solutions found by artificial intelligence can be much more efficient than human solutions, but they can also be confused because of the diversity and immensity of details that our mind possesses.
Artificial Intelligence models bring us more secure solutions, but nothing is 100% safe when we talk about information security. There are ways to improve security, hinder invasions and attacks, but every system has flaws.
Perhaps, in the future (things of my imagination) we will have an artificial superintelligence ahead of the human, which may be one of the greatest challenges of invasion of history, but until then .. just my imagination.


text:
What is a learning agent?
What is a learning agent, and how does it work? What are examples of learning agents (e.g., in the field of robotics)?

A learning agent can be defined as an agent that, over time, improves its performance (which can be defined in different ways depending on the context) based on the interaction with the environment (or experience).
The human is an example of a learning agent. For example, a human can learn to ride a bicycle, even though, at birth, no human possesses this skill.
Section 2.4.6 Learning agents (p. 54) of the AIMA book (3rd edition), by Norvig and Russell, define a learning agent as follows.

A learning agent can be divided into four conceptual components, as shown in Fig 2.15.

The four components are

learning element: makes improvements to the performance element (an example would be Q-learning)
performance element: chooses the actions to take in the environment (this is analogous to a model, e.g. a neural network, that contains the knowledge or rules to act in the environment)
critic: provides feedback (based on some performance metric) to the learning element in order for it to improve the performance element (so this is how you evaluate the potential improvements)
problem generator: suggests actions that will lead to new informative experiences (this would be a behavior policy in reinforcement learning)

At first glance, this definition might seem unrelated to the definition given above, but they are equivalent. Norvig and Russell's definition of a learning agent builds on top of their definition of an agent. Moreover, as I wrote above, if you are familiar with RL, these four components can be associated with common concepts in reinforcement learning (such as Q-learning, value function/policy, target, behavior policy). The book just uses different names to refer to the same or similar concepts.
To make the definition clearer, Norvig and Russell also provide an example

To make the overall design more concrete, let us return to the automated taxi example. The performance element consists of whatever collection of knowledge and procedures the taxi has for selecting its driving actions. The taxi goes out on the road and drives, using this performance element. The critic observes the world and passes information along to the learning element. For example, after the taxi makes a quick left turn across three lanes of traffic, the critic observes the shocking language used by other drivers. From this experience, the learning element is able to formulate a rule saying this was a bad action, and the performance element is modified by installation of the new rule. The problem generator might identify certain areas of behavior in need of improvement and suggest experiments, such as trying out the brakes on different road surfaces under different conditions.

This answer provides more definitions of the machine learning field and an ML algorithm, which is not exactly the same thing as a learning agent, given that the concept of an agent also implies or emphasizes the usage of a body with sensors, actuators, and an agent program (which converts the observations into actions), but the definitions given in this answer and in the other answer are consistent with each other.


text:
Is there a limit of minimum error for a particular training dataset in artificial Neural Network?
In error-based learning using gradient descent, if I give you a training dataset, then can you find the minimum error after training? And the minimum error should be true for all architectures of a neural network? Consider you will use MSE for calculating the error. You can choose anything you want other than my specified condition. It's like no matter how you change your network you can never cross the limit.

In some sense yes, in some sense no. The "error" word you are using is not defined in the question. The network can have two type of errors, quite different - generalization error and loss (training error). The lower bound of loss on the dataset obviously depend on the network architecture, because loss is value produced by network as a whole, and some network will be able to produce exactly zero loss (which is likely be overfitting, wich is bad).
The generalization error is how well network generalize form training dataset to potentially infinite (or very big) amount of data samples "in the wild", so generalization error can not be calculated precisely, only estimated. The lower bound on generalization error is studied in Statistical Learning. It always depend on the complexity of the classifier (in our case architecture of network). Statistical Learning theory produce estimations like VC generalization bound which connect generalization error with size of the dataset and complexity of the classifer (netwroks in our case) space. However for specific dataset it probably don't make sense to talk about generalization bound outside of context of complexity of classifier space, because if there is no restriction we can always construct bigger network which would have lesser generalization error (for example by pretraning bigger network on bigger dataset)
To put it short you have to put restriction on complexity of network space to be able estimate minimum generalization error (and that probably wouldn't be useful anyway, because it wouldn't say anything about if gradient descent would reach that error)


text:
How much can the addition of new features improve the performance?
How much can the addition of new features improve the performance of the model during the optimization process?
Let's say I have a total of 10 features. Suppose I start the optimisation process using only 3 features.
Can the addition of the 7 remaining ones improve the performance of the model (which, you can assume, might already be quite high)?

It depends on the used network as well as the feeding mechanism but let's give an example;
When working with LSTM, giving the time data (as an integer sequence) in addition to the time-series data(coming from features) dramatically increases the performance of the network.
[$X_{0}$,$X_{1}$, ...] $\rightarrow$ [[$X_{0}$,$t_{0}$],$[X_{1}$,$t_{1}$], ...]

If you go and look for the kaggle competition winner's notebooks, they do also create additional features based on the featured data.
Let's assume that the performance is already quite high on the three features so that you can predict those three features with high reliability.
It would only make sense to increase the number of features if you would like to predict additional features!

The optimization of 10 features, as opposed to optimization of 3 features, will converge more slowly.

Let's say I have a total of 10 features. Suppose I start the optimisation process using only 3 features. Can the addition of the 7 remaining ones improve the performance of the model (which, you can assume, might already be quite high).

The answer might be no.

The accuracy and reliability of the convergence toward ground truth (a formalized objective used to guide the optimization process) when split into groups of three and seven features may be better or worse than when left as a group of ten.
Except in rare cases, the results will not be the same.  The likelihood of identical results is so low it may never happen in the world in the next century except when conditions are arranged solely to cause it to occur.

Why then do may approaches group the dimensions of the result and converge on groups of axes, then another, then another, and back again to the first, iterating until the convergence goal is reached?  This approach is used to reduce the time and computing resources used to reach the optimal.  As the problem complexity increases, the use of groupings in this way is more common.


text:
What is the difference between a receptive field and a feature map?
In a CNN, the receptive field is the portion of the image used to compute the filter's output. But one filter's output (which is also called a "feature map") is the next filter's input.
What's the difference between a receptive field and a feature map?

Feature map
A feature map is the output of the convolution operation (performed by a convolutional layer).
When you apply a kernel $K_1$ to the input image $I$ (i.e. you perform the convolution operation with $K_1$), you produce one feature map $F_1$. We can denote this operation by $F_1 = I \circledast K_1$. Note that the convolution (or cross-correlation) can be seen as the multiplication of the kernel $K_1$ with different sub-regions of the image $I$. Each of these multiplications produces a scalar value of the feature map $F_1$.
If you perform another convolution operation with another kernel $K_2$, you produce another feature map $F_2$. So, if you perform the convolution operation $n$ times with $n$ different kernels in a convolutional layer, you will get $n$ feature maps, which are typically concatenated to form a block of feature maps, which we can denote by $\mathbf{F}$. So, the total number of feature maps is equal to $n$, i.e. the number of kernels that you convolve with $I$.
Each element of a feature map $F_i \in \mathbb{R}^{w \times h}$, which we can denote by $F_{i_{jk}}$ (i.e. the element at coordinates $j$ and $k$ of the matrix $F_i$), can be thought of as the "activation of a neuron".
Alternatively, you can think of a feature map as a collection/map of features produced by the kernel $K_i$ (hence the name feature map). In other words, $K_i$ extracts certain features from $I$. Another kernel $K_l$ may extract other features, and so on.
Receptive field
A receptive field is the portion of the input image that is/was used to compute the activation of a neuron (aka node) in a feature map.
So, we talk of "the receptive field of a neuron in a feature map". You can read this answer to understand more what a neuron in a convolutional layer is.
Illustration
This illustration should summarise the descriptions above.

In the illustration above, a feature map is a 2-dimensional matrix of shape $46 \times 46$. There are $100$ feature maps (so $100$ kernels were convolved with the input image). The receptive field of a neuron (small pink square in the feature map) is a 3-dimensional sub-region of the image of shape $5 \times 5 \times 3$. Note that, in this case of a 2d-convolution, the kernel has the same depth as the input image (i.e. $3$).
Receptive field of neurons in deeper layers
The output of a convolutional layer, which we denoted by $\mathbf{F}$ above (i.e. the concatenation of all feature maps), is the input to the next layer of the CNN. If this layer is a convolutional layer, then it will produce other feature maps $\mathbf{F}'$. The receptive fields of the neurons associated with these feature maps $\mathbf{F}'$ will not correspond to the receptive fields of the neurons associated with the feature maps produced by the first convolutional layer. In fact, as you go deeper into a CNN, the receptive field of neurons increases. You can read more about how it increases here or watch this video. That's why we can think of neurons in deeper layers as "looking at more general features of the input image".


text:
Is my understanding of the differences between MDP, Semi MDP and POMDP correct?
I just wanted to confirm that my understanding of the different Markov Decision Processes are correct, because they are the fundamentals of reinforcement learning. Also, I read a few literature sources, and some are not consistent with each other. What makes the most sense to me is listed below.
Markov Decision Process
All the states of the environment are known, so the agent has all the information required to make the optimal decision in every state.  We can basically assume that the current state has all information about the current state and all the previous states (i.e., the Markov property)
Semi Markov Decision Process
The agent has enough information to make decisions based on the current state.  However, the actions of the agent may take a long time to complete, and may not be completed in the next time step. Therefore, the feedback and learning portion should wait until the action is completed before being evaluated.  Because the action takes many time steps, "mini rewards" obtained from those time steps should also be summed up.
Example: Boiling water

State 1: water is at 23 °C
Action 1: agent sets the stovetop at 200 °C
Reward (30 seconds after, when water started to boil):

+1 for the fact that water boiled in the end, but -0.1 reward for each second it took for the water to start boiling.
So, the total reward was -1.9 (-2.9 because the water did not boil for 29 seconds, then +1 for water boiling on 30th second)

Partially Observable Markov Decision Process
The agent does not have all information regarding the current state, and has only an observation, which is a subset of all the information in a given state.  Therefore, it is impossible for the agent to truly behave optimally because of a lack of information.  One way to solve this is to use belief states, or to use RNNs to try to remember previous states to make better judgements on future actions (i.e., we may need states from the previous 10 time steps to know exactly what's going on currently).
Example
We are in a room that is pitch black.  At any time instant, we do not know exactly where we are, and if we only take our current state, we would have no idea.  However, if we remember that we took 3 steps forward already, we have a much better idea of where we are.
Are my above explanations correct?  And if so, isn't it possible to also have Partially Observable Semi Markov Decision Processes?

Yes that all seems pretty much correct. Of course, your descriptions are somewhat informal, so there might be minor details to nitpick about... but the core differences between the three as you described them are fine.

Thanks for the reply Dennis!  So in theory, it is possible to have a partially observable semi markov decision process?  Also, do you by any chance know any good textbooks/references I can refer to, to ensure my descriptions are 100% correct.  Thanks again!

Correct, that would be possible. As for POMDPs.. don't know any particular reference by heart, but they're all over really, they're everywhere. As for SMPDs, I believe one of the "original" references for them is: http://papers.nips.cc/paper/889-reinforcement-learning-methods-for-continuous-time-markov-decision-problems.pdf . Another good one is: https://www.sciencedirect.com/science/article/pii/S0004370299000521 (the latter actually proposes an "Options framework", which is more general than SMDPs, but much more popular these days).

Yes, the core differences between the different categories of problems are correct as you've described them.
For SMDPs, I'd like to remark that the water boiling example is maybe not the best. That looks more like an example of "delayed rewards", but not one of "durative actions": when the agent takes that action to raise the temperature, it takes some time before the reward comes in, but the agent's action itself doesn't take that much time. The agent could maybe do something else in the meantime. Delayed rewards are not restricted to SMDPs, they can also show up in regular MDPs (you just have to make sure to include some data in the state representation to indicate the time since the temperature was set or something like that, such that the Markov property is not violated).
A typical example of an SMDP with durative actions would be a grid with rooms, split by walls but connected by smaller doors inside those walls. A single "primitive" action would just take a single step in the larger grid, but longer-duration "macro-actions" would make the agent navigate from somewhere within a room to one of the doors "on auto-pilot". This would still take as much time as it would to navigate explicitly in multiple smaller actions, but the larger more complex behaviour being encapsulated in a single "macro-action" can speed up learning.


text:
In transfer learning, should we apply standardization if the pre-trained model was (or not) trained with standardised data?
Assume one is using transfer learning via a model which was trained on ImageNet.

Assume that the pre-processing, which was used to achieve the pre-trained model, contained z-score standardization using some mean and std, which was calculated on the training data.
Should one apply the same transformation to their new data? Should they apply z-score standardization using a mean and std of their own training data?

Assume that the pre-processing now did not contain any standardization.
Should one apply no standardization on their new data as well? Or should one apply the z-score standardization, using the mean and std of their new data, and expect better results?

For example, I've seen that the Inception V3 model, which was trained by Keras, did not use any standardization, and I'm wondering if using z-score standardization on my new data could yield better results.

You should use the same transformation (i.e same mean and std in case of z-score normalization) on the new data while using any pretrained model.
If the pretrained model was trained without any normalization then it does not matter if you use normalization or not as long as range of input data is fixed (0-255 for the 8-bit input images). You can use pretrained models with batch normalization which takes care of internal covariance shift and thus there is no need to normalize data.


text:
Price Movement Forecasting Issue
I am working on a project for price movement forecasting and I am stuck with poor quality predictions.
At every time-step I am using an LSTM to predict the next 10 time-steps. The input is the sequence of the last 45-60 observations. I tested several different ideas, but they all seems to give similar results. The model is trained to minimize MSE.
For each idea I tried a model predicting 1 step at a time where each prediction is fed back as an input for the next prediction, and a model directly predicting the next 10 steps(multiple outputs). For each idea I also tried using as input just the moving average of the previous prices, and extending the input to input the order book at those time-steps.
Each time-step corresponds to a second.
These are the results so far:
1- The first attempt was using as input the moving average of the last N steps, and predict the moving average of the next 10.
At time t, I use the ground truth value of the price and use the model to predict t+1....t+10
This is the result:
Predicting moving average:

On closer inspection we can see what's going wrong:
Prediction seems to be a flat line. Does not care much about the input data:

The second attempt was trying to predict differences, instead of simply the price movement. The input this time instead of simply being X[t] (where X is my input matrix) would be X[t]-X[t-1].
This did not really help.
The plot this time looks like this:

Predicting differences:

But on close inspection, when plotting the differences, the predictions are always basically 0.

At this point, I am stuck here and running our of ideas to try. I was hoping someone with more experience in this type of data could point me in the right direction.
Am I using the right objective to train the model? Are there any details when dealing with this type of data that I am missing?
Are there any "tricks" to prevent your model from always predicting similar values to what it last saw? (They do incur in low error, but they become meaningless at that point).
At least just a hint on where to dig for further info would be highly appreciated.
UPDATE
Here is my config
{
    "data": {
        "sequence_length":30,
        "train_test_split": 0.85,
        "normalise": false,
        "num_steps": 5
    },
    "training": {
        "epochs":200,
        "batch_size": 64
    },
    "model": {
        "loss": "mse",
        "optimizer": "adam",
        "layers": [
            {
                "type": "lstm",
                "neurons": 51,
                "input_timesteps": 30,
                "input_dim": 101,
                "return_seq": true,
                "activation": "relu"
            },
            {
                "type": "dropout",
                "rate": 0.1
            },
            {
                "type": "lstm",
                "neurons": 51,
                "activation": "relu",
                "return_seq": false
            },
            {
                "type": "dropout",
                "rate": 0.1
            },
            {
                "type": "dense",
                "neurons": 101,
                "activation": "relu"
            },
            {
                "type": "dense",
                "neurons": 101,
                "activation": "linear"
            }
        ]
    }
}

Notice the last layer with 101 neurons. It is not an error. We just want to predict the features as well as the price. In other words, we want to predict the price for time t+1 and use the features predicted to predict the price and new features at time t+2, ...

What is your architecture?  Knowing that it is an LSTM is not enough.  What values are you using for the hyperparameters (e.g. number of epochs, mini-batch, number of neurons, number of layers, etc.)?  Have you done a sanity check with simple data like a sine wave?  Are you normalizing the data?  If so, have you tried multiple methods?  Why are you looking at one second interval data?  Financial data is extremely noisy at that time scale even when averaged over 10 seconds.  Why not consider one minute interval data or higher to at least prove your approach?

With your update the most obvious thing I would change is to normalize the data.  You state that you are using price, moving average, and price difference but your configuration has 101 inputs.  Why is that?  Your input time steps is 30 but you state you are using 45-60 observations and the configuration says the sequence length is 30.  Your sequence length should be greater than your input time steps by the number of steps you are predicting ahead.  Why does your output layer have 101 neurons?  Another useful piece of information would be a plot of the model's loss versus epoch.

Given a model that takes in a price and a second value, such as a moving average of the price, the following configuration is my recommendation.  This is based on training on a history of 45 input time steps and forecasting out 10 steps in the future.  I have assumed that your 'num_steps' is a stride through the training data.  Note that I have not tested this and you will most likely need to tweak a parameter or two:
{
    "data": {
        "sequence_length": 55,
        "train_test_split": 0.85,
        "normalise": true,
        "num_steps": 5
    },
    "training": {
        "epochs":200,
        "batch_size": 64
    },
    "model": {
        "loss": "mse",
        "optimizer": "adam",
        "layers": [
            {
                "type": "lstm",
                "neurons": 100,
                "input_timesteps": 45,
                "input_dim": 2,
                "return_seq": true,
                "activation": "relu"
            },
            {
                "type": "dropout",
                "rate": 0.2
            },
            {
                "type": "lstm",
                "neurons": 100,
                "activation": "relu",
                "return_seq": false
            },
            {
                "type": "dropout",
                "rate": 0.2
            },
            {
                "type": "dense",
                "neurons": 1,
                "activation": "linear"
            }
        ]
    }
}

Try this out and plot your model's loss and accuracy by epoch.  See Display Deep Learning Model Training History in Keras for some information and code in this area.  Let me know if you are not using Keras and I will point you to another article/code solution.

I was wondering something a minute ago. Maybe there is a mistake in the code, but it is unclear to me. If we had applied regression on the price difference than the "Plot of differences" plot would be better, right?

If you mean using the regression line instead of the price differences my thought is that the performance would not be as good.  It is a good idea to test out.  Maybe also feed the regression line as a third value (i.e. price, difference, regression line).  It is always good to try different data.

{
"type": "dense",
"neurons": 1,
"activation": "linear"
} I noticed you changed that part, but we wanted to predict the price as well as the features for t+2. So at time t+2, I reused the predicted features and the prices from time t+1 to predict the price for time t+2 and predict the features  to be used again at time t+3.

Remember that any machine learning model works good only when there is a "rule" or a correlation between modeling data and modeled data. When there is not, even the best algorithm will not predict/classify correctly. I am not saying that this must be the case, but probably you have come pretty close to the physical limit of what can be achieved using this data.
Maybe you should consider adding or extracting new features to your model.


text:
When should the iterative deepening search and the depth-limited search be used?
When should the iterative deepening search (IDS), also called iterative deepening depth-first search (IDDFS), and the depth-limited search be used?

In practice, you can consider that IDS is a cheap way to implement a breadth-first search (BFS).
When the branching factor is high, both algorithms have similar computational costs $O(B^P)$, where $P$ is the length of the solution path and $B$ the branching factor, but IDS requires less memory, typically $O(P)$, while BFS uses $O(B^P)$.

If your search tree's branching factor is finite but its depth is unbounded, then DFS isn't "complete". What that means is that it may not ever find the goal node even if it exists. You would use these techniques precisely when this situation arises.
To understand this, let's say you're at the node A in the following tree:
     A
    / \
   /   \
  B     C
 / \   / \
.   . .   .

You choose to explore subtree rooted at B. Now if there are no goal nodes in that subtree, but the goal nodes were in the subtree rooted at C, then we're stuck. Because  we will keep searching there because we don't know beforehand that there are no goal nodes there, next node we explore could always be a goal node. But this essentially means that we'll never finish searching the subtree rooted at B in finite time, and never backtrack to explore the subtree rooted at C.
So to overcome this you would want to iterative deepening depth-first search or depth-limited search.
In depth-limited search you set a limit on the depth you'll explore. Let's say you choose $d$ as the depth limit because you know that the solution lies within that depth limit.
And let's say the branching factor for your tree is $b$. Going back to our example, you have finite elements in the subtree rooted at B (at most $b^d$, can you reason why?), so we'll be able to look at all of them and eventually backtrack and start exploring subtree rooted at C, and we'll be able to find our answer.
Now if you don't know the depth $d$ which goal nodes lie at, then you'll use iterative deepening DFS, which essentially does depth limited search but keeps increasing the value of $d$ on each iteration, so you're guaranteed to find a goal node, if it exists.


text:
Can current AI techniques distinguish a fake old paper from a real one?
It is an easy matter to make a paper look old, for example, using any of the techniques explained on this page of WikiHow: https://www.wikihow.com/Make-Paper-Look-Old.
Is current AI sufficient to distinguish a fake old paper from a real one?

How would the AI sense the paper under scrutiny? A photo of someone holding the paper? Output from a scanner? Output from a microscope? Touch sensors?  Mechanical stress results? IR Spectrometer readings? C14 radio-carbon dating readings? How do humans typically discover that the paper is real or fake?

Is current AI sufficient to distinguish a fake old paper from a real one?

It depends. Specifically, it depends on how much information you (or any automated system that feeds data to the AI), gathers about the paper you want to test, and how good that information is at distinguishing between real and fake old paper.
A model that distinguishes between cases in an input domain (which can be any consistent data, such as images of a subject) is called a classifier. The current state of the art classifiers working with sensory data, if fed enough training data of sufficient quality, can out-perform humans at certain tasks. This is a success of "narrow AI" over humans. This is not some magical ability of machine learning that always applies though, there are some strong caveats:

The task must be feasible given the data collected. Low-resolution photos of samples of the paper might not contain enough information for your example task at all.
When comparing with a human performing the same task, they must be given identical data. If a human would gain more data by interacting with the paper in your example, it can give the human an advantage that a narrow AI cannot easily replicate.
The quality of training data "ground truth" must be higher than achievable by a single human. In your example, this could be achieved simply by knowing the provenance of all the training data.
There often needs to be a large amount of training data to achieve better-than-human ability, although the actual amount required depends heavily on the task.

Expanding on the first point, image classifiers can sometimes be surprisingly good, they have different strengths and weaknesses compared to human vision, so it is possible for them to sometimes be good at making classifications that humans are bad at. However, there is no guarantee of getting that kind of result for any specific task.

Expanding on the second point, if interacting with a subject is allowed in order to classify it, this moves from the domain of "narrow AI" classifiers into active rational agents.
If you gave a piece of paper to a human expert on old paper, and ask them to classify it, they might first look at the paper, then maybe sniff it, touch it . . . still undecided, so get out a magnifying glass to check a couple of features that might be good telltale signs . . . etc.
To build an equivalent artificial agent is harder than a single classifier: It might be possible to purpose-build a collection of narrow AI classifiers, that output a class and confidence levels that decided whether certain tests should be carried out. It might even be possible to give a more general AI the same toolkit plus a very long time to experiment and it could learn for itself which tests to do in which order and how much to rely on the separate results. Or some hybrid of the two approaches. However, any such sophisticated system would still be bound by the limits of individual classifiers.


text:
Understanding the proof of theorem 2.1 from the paper "Efficient reductions for imitation learning"
I am trying to understand the proof of theorem 2.1 from this paper:

Ross, Stéphane, and Drew Bagnell. "Efficient reductions for imitation learning." Proceedings of the thirteenth international conference on artificial intelligence and statistics. 2010.

The cost-to-go is given as
$$J(\pi) = \sum_{t=1}^{T}\mathbb{E}_{s\,\sim\, d^t_{\pi}(s)}\left[C_\pi(s)\right].$$
In the paper they use $\hat{\pi}=\pi$ for the learned policy and $\pi^*$ for the expert policy.
In the derivation they write
$$J(\pi)\leq \sum_{t=1}^{T}\{ p_{t-1}\mathbb{E}_{s\, \sim \, d_t(s)}\left[C_\pi(s) \right]+(1-p_{t-1})\}$$
$$\leq \sum_{t=1}^{T}\{ p_{t-1}\mathbb{E}_{s\, \sim \, d_t(s)}\left[C_{\pi^*}(s) \right]+p_{t-1}{\ell_t(s,\pi)}+(1-p_{t-1})\},$$
in which $p_{t-1}$ is the probability of not not making an error with policy $\pi$ up to the time $t-1$. And $\ell$ is the surrogate 0-1 loss.
The following steps are easy to follow, but how did they come up with these steps?


text:
How do we design a neural network such that the $L_1$ norm of the outputs is less than or equal to 1?
What are some ways to design a neural network with the restriction that the $L_1$ norm of the output values must be less than or equal to 1? In particular, how would I go about performing back-propagation for this net?
I was thinking there must be some "penalty" method just like how in the mathematical optimization problem, you can introduce a log barrier function as the "penalty function"

Penalty (barrier function) is perfectly valid and simplest method for simplex type constraint (L1 norm is simplex constraint on absolute values). Any type of barrier function may work, logarithmic, reciprocal or quadratic. All of them supported by any major framework(pytorch, tensorflow), just add them to loss function. You would need some hyperparameter tuning for the scale factor of penalty.
There is more efficient, though more complex way to do it. Instead of putting constraint you can automatically output value wich satisfy simplex constraint:
Assume that L1 norm constraint is $\left \|v\right \|_1 \leq 1$, $v \in \mathbb{R}^n$

put $sigmoid(v_i)$ activation on output to norm elements to [-1, 1]
add slack (fake) variable element $v_{n+1} = 1 - \sum_{1}^{n}  v_i $
project new $v{}'\in \mathbb{R}^{n+1}$, $v{}'_i = |v_i|,1\leq  i  \leq n+1$ onto unit simplex with standard algorithm (also here)

Backpropagation of last step may require differentiable sorting, which is missing in most of frameworks, you may have to look for open sourced implementation, for example extract it from here or use some automatic differentiation package. Both require some substantial code reading/debugging. However in my experience assuming constant $\Delta$ also works in many cases, in that case  differentiable sorting is not needed. Intuition behind constant $\Delta$  is that  $\Delta$  could be chosen such way that there is some interval on which it's value doesn't affect sorting order.

The term 'size' isn't applicable to the tensor output of a network or network.  These are the qualities.

Rank $N$ that defines the rank of each tensor instance in $\mathbb{R}^N$
Ranges of the indices to the dimensions from $1$ to $N$
Ranges of the scalar values that comprise the tensor instance — If they are discrete rather than real (approximated by floating point or fixed point numbers), then the range is the description of the permissible ordinal values.

The question may be referring to this last quality.
The imposition of a penalty for values that are in the range of the numeric type used as the output of the last activation function but not in the allowable range of output for the desired trained function works in a limited way.  It skews the output distribution with respect to the natural distribution of possible learning states and therefore can easily interfere with convergence quality or speed or both.
There are a number of techniques that map natural output distribution with constrained ranges, but it must be done without skewing the distribution upstream from the technique used, to avoid negatively impacting favorable convergence properties of the artificial network.
One simple case that can be described here is when the number of possible output states is in the set of $2^i$ where $i \in$ { positive integers }.  In such a case, the final layer of the network can be $i$ threshold activation functions with 1 or -1 as possible output values.
In that case, the ordinal then becomes
$$o = \sum_{n=0}^i \frac {y_n + 1} {2},$$
where $y_n$ is the output from activation function index $n$.

You say "output states is in the set of $2^i$" but the value you describe is $o \in [0,i] $, $o \in \mathbb{Z}$


text:
How exactly is equivariance achieved in capsule neural networks?
I have read quite a lot about capsule networks, but I cannot understand how the squashed vector would also rotate in response to rotation or translation of the image. A simple example would be helpful. I understand how routing by agreement works.


text:
How do compute the table for $p(s',r|s,a)$ (exercise 3.5 in Sutton & Barto's book)?
I am trying to study the book Reinforcement Learning: An Introduction (Sutton & Barto, 2018). In chapter 3.1 the authors state the following exercise

Exercise 3.5 Give a table analogous to that in Example 3.3, but for $p(s',r|s,a)$. It should have columns for $s$, $a$, $s'$, $r$, and $p(s',r|s,a)$, and a row for every 4-tupel for which $p(s',r|s,a)>0$.

The following table and graphical representation of the Markov Decision Process is given on the next page.

I tried to use $p(s'\cup r|s,a)=p(s'|s,a)+p(r|s,a)-p(s' \cap r|s,a)$ but without a significant progress because I think this formula does not make any sense as $s'$ and $r$ are not from the same set. How is this exercise supposed to be solved?
Edit
Maybe this exercise intends to be solved by using
$$p(s'|s,a)=\sum_{r\in \mathcal{R}}p(s',r|s,a)$$
and
$$r(s,a,s')=\sum_{r\in \mathcal{R}}r\dfrac{p(s',r|s,a)}{p(s|s,a)}$$
and
$$\sum_{s'\in\mathcal{S}}\sum_{r\in\mathcal{R}}p(s',r|s,a)=1$$
the resulting system is a linear system of 30 equation with 48 unknowns. I think I am missing some equations...

The function $r(s,a,s')$ gives the expected reward in each scenario, but not the distribution of rewards that lead to values $r_{search}$ and $r_{wait}$
The text explains that reward is $+1$ for each can found, and that different distributions of numbers of cans are expected when waiting as opposed to searching. However, it does not give any description of the actual distributions, just summarises them as the two expected rewards, and suggests $r_{search} \gt r_{wait}$
You have two main ways to answer the exercise:

Invent some parameters for the distributions of $r_{search}$ and $r_{wait}$ in order to split up single values of $p(s'|s,a)$ into multiple values of $p(s', r|s,a)$. E.g you could decide that $r_{search}$ consists of $0  \eta_0 + 1  \eta_1 + 2 \eta_2 + 3 \eta_3$ where $\eta_0, \eta_1, \eta_2, \eta_3$ are probabilities that sum to $1$ - each row that currently has $r_{search}$ as the output of $r(s,a,s')$ would then split into 4 rows with reward 0, 1, 2, 3 to complete the new table . . . $r_{wait}$ would need a different set of parameters.
Ignore the details of the distribution, move column $r(s,a,s′)$ to the left and call it $r$, changing $p(s|s,a)$ to $p(s', r|s,a)$. It might be all that's expected given the lack of information.

My personal opinion is that the authors want you to think about solution 1 - the only issue is that it requires you to invent some new parameters that were not provided. The ones I name are only a suggestion, they do not represent a specific "correct" answer in terms provided by the book, because the book omits those details.
As an example to start with, if you start solution 1, and use parameters as I have labelled them, you will end up with a first row looking like this:
$s\qquad \qquad a\qquad  \qquad  s'\qquad  \qquad   r \qquad       p(s', r| s, a)$
$high  \qquad \quad search  \qquad  high  \qquad \quad   0  \qquad   \alpha \eta_0$

Thank you for your answer, but I must admit that I still do not understand how to solve this exercise.

@MrYouMath: Sorry I don't want to post a fully worked solution online, as it is important to complete the exercises alone to gain understanding, and some university courses will use the book. You can obtain solutions for each chapter (for the non-coding exercises) if you attempt all the exercises in the chapter then email the author with your work.

I will add one row as an example

I think the authors just want me to set up the linear equation from my edit and then solve them (which I will not do because it is too tedious). This will be a system of 48 linear equation with exactly 48 unknowns. The parameters $\eta_i$ do not make sense to me at the moment.

@MrYouMath: No it is not possible to solve that at all because you don't have enough data to write out the simultaneous equations. You don't even know for certain that there are 48 - like the params I suggest you would have to decide what the size of $\mathcal{R}$ is arbitrarily.

I realized that now :D. Why do you think that $|\mathcal{R}|$ is not fixed? I would think that it is $4$.

@MrYouMath: I guess you think $\mathcal{R}$ is ${-3, 0, r_{search}, r_{wait}}$ then? I explain why that is not the case in this answer. You are confusing expected reward values as returned by $r(s,a,s')$ with the possible reward values  in $\mathcal{R}$, which could be ${-3, 0, 1, 2, 3, 4, 5 . . .}$

Now, that is confusing ...

I think 2. is highly likely.

Hi Neil. For your solution 2, could you please elaborate why would we call $r(s,a,s')$ as $r$? Since for all we know, $r(s,a,s')$ could be a real number (say $3.5$) and hence it would not be in $\mathcal{R}$?

@SlimShady: It's not correct notation, but the MDP is underspecified in the question, so it does not seem too important to focus on that.

At first, like Neil Slater says, I thought this could only be solved using the expected rewards instead of actual rewards, or else there wasn't enough information to solve it. But now I think there might be a way to solve this question. Here is my thinking on this problem (I would be curious for anyone's thoughts, as I am working through this book myself).
I think the key part is where the book says:

Each can collected by the robot counts as a unit reward, whereas a reward of $-3$ results whenever the robot has to be rescued.

This means that the reward set is actually $\mathcal R = \{0, 1, -3\}$ (we assume that in each timestep, the robot can only collect one can).
Now using $$r(s,a,s') = \sum_r r \frac{p(s',r\mid s,a)}{p(s'\mid s,a)} \tag{3.6}$$ and $$p(s'\mid s,a) = \sum_r p(s',r\mid s,a)\tag{3.4}$$ it seems possible to solve for all the probabilities. I'll do an example for $(s,a,s') = (\mathtt{high}, \mathtt{search}, \mathtt{high})$ and leave the rest to you (I haven't actually done the rest, since this does seem rather tedious).
Equation 3.6 gives $$r_\mathtt{search} = 0\cdot \frac{p(s', 0 \mid s,a)}{\alpha} + 1\cdot \frac{p(s', 1 \mid s,a)}{\alpha} -3\cdot \frac{p(s', -3 \mid s,a)}{\alpha}$$ Since $p(s', -3 \mid s,a) = 0$ (it's impossible for the robot to have to be rescued, since we started in the "high" state), we get $p(s', 1 \mid s,a) = \alpha r_\mathtt{search}$.
Now equation 3.4 gives $$\alpha = p(s', 0 \mid s,a) + p(s', 1 \mid s,a) + p(s', -3 \mid s,a)$$ which solves to $p(s', 0 \mid s,a) = \alpha - \alpha r_\mathtt{search}$.
So the first two rows of the table will look like:
$$\begin{array}{cccc|c}
s& a & s' & r & p(s',r\mid s,a)\\ \hline
\mathtt{high} & \mathtt{search} & \mathtt{high} & 1 & \alpha r_\mathtt{search}\\
\mathtt{high} & \mathtt{search} & \mathtt{high} & 0 & \alpha (1- r_\mathtt{search})
\end{array}$$

This matches my answer option 1, where your assumption "in each timestep, the robot can only collect one can" sets the distribution behind $r_{search}$ to specific simple distribution ($r_{search}$ is also in that case the probability of finding a single can by searching).

In the announced problem, most of the transitions aren't possible, so most the terms of equations (3.3) and (3.4) from the book will end up being 0.
In my understanding,
$$
\begin{align}
p(s'= high | s = high, a = search) &= \sum_{r \in \{0, -3, r_{search}, r_{wait}\}} p(s'=high, r | s = high, a = search) \\
&= p(s'=high, r =0 | s = high, a = search) \\
&+p(s'=high, r = -3 | s = high, a = search) \\
&+p(s'=high, r = r_{search}| s = high, a = search) \\
&+p(s'=high, r = r_{wait} | s = high, a = search)
\end{align}
$$
The problem states that if the agent has high batteries and it chooses to search, then there is no chance that it ends up having a negative reward ($r= -3$), thus its transition probability is 0 by definition: $p(s'=high, r = -3 | s = high, a = search) = 0$.
Applying the same logic to all other terms, we get,
$$
\begin{align}
p(s'= high | s = high, a = search) &= \sum_{r \in \{0, -3, r_{search}, r_{wait}\}} p(s'=high, r | s = high, a = search) \\
&= p(s'=high, r = r_{search}| s = high, a = search) \\
&= \alpha
\end{align}
$$
It looks weird. I not 100% sure that that's the solution, because the question would not make much of sense (the table would've been quite the same).

This means that the reward set is actually R={0,1,−3} (we assume that in each timestep, the robot can only collect one can).

@riceissa While I agree with the rest of your demonstration, I wouldn't assume that the robot can only collect 0 or 1 can. As Neil Slater suggest, I think the robot could pick any number of cans between 0 and N.
Below is how I solve the problem for the more general case, assuming specific values for $s',s, a$. This generalization encompasses @riceissa answer.

Let :

$S_t=s'=\texttt{high}$
$S_{t-1}=s=\texttt{high}$
$A_{t-1}=a=\texttt{search}$

We have the following equality :
$$r_{search}=\sum_{r \in R}r\cdot\frac{p(s', r|s, a)}{p(s'|s, a)}$$
For these values of $s', s, a$ we also have:

$p(s'|s,a)=\alpha$.
$p(s', -3|s,a)=0$

Writing $\eta_r:=p(s',r|s,a)$ and for $R=\{0, 1, 2, \dots,N\}$ (we omit $r=-3$ since the proba is 0 for this case) we have then:
\begin{align*}
r_{\texttt{search}}&=\sum_{r=0}^N r\cdot\frac{\eta_r}{\alpha}\\
r_{\texttt{search}}\cdot\alpha&=\sum_{r=0}^N r\cdot\eta_r\\
r_{\texttt{search}}\cdot\alpha&=i\cdot\eta_i + \sum_{\substack{r=0 \\ r\neq i}}^N r\cdot\eta_r\\
i\cdot\eta_i&= r_{\texttt{search}}\cdot\alpha- \sum_{\substack{r=0 \\ r\neq i}}^N r\cdot\eta_r
\end{align*}
For $i\neq 0$ :
$$\eta_i= \frac{1}{i}\cdot\left(r_{\texttt{search}}\cdot\alpha- \sum_{\substack{r=0 \\ r\neq i}}^N r\cdot\eta_r\right)$$
For $i=0$, we use that fact that
\begin{align*}
\alpha&=p(s'|s,a)=\sum_{r=0}^N p(s',r|s,a)=\sum_{r=0}^N\eta_r\\
&\Rightarrow\eta_0=\alpha - \sum_{r>0} \eta_r
\end{align*}
So substituting $\eta_r$ by its probability formula we end up having, for these specific values of $s', s, a$ :
\begin{equation}
    \begin{cases}
      p(s',i|s,a)= \frac{1}{i}\cdot\left(r_{\texttt{search}}\cdot\alpha- \sum_{\substack{r=0 \\ r\neq i}}^N r\cdot p(s',r|s,a)\right) & \texttt{$\forall i \in [1,N]$}\\
      p(s',0|s,a)=\alpha -\sum_{r>0} p(s',r|s,a) & \texttt{if $i=0$}
    \end{cases}
\end{equation}

it doesn't make sense for $i=0$. Could you clarify how you got the "$\Rightarrow \eta_0=\alpha-\sum_{r>0}\eta_r$" part?

My understanding is that it will be the same as p(s' | s, a) for any s, a, s', r combination.
The reward r(s, a, s') is already defined in terms of s, a, s'.
Since p(s', r| s, a) = p(r|s', a, s)* p(s'| a, s). For each case the p(r | s', a, s) is equal to 1 by definition. Thus, the column for p(s', r| a, s) = p(s'| s, a)
For instance,
If,
S= high, A=search and S'=High, then the reward is always alpha.
p(r=alpha| s=high, a = search , s'=high) = 1 (you always get that reward for this case) etc.


text:
How do I choose the optimal batch size?

Batch size is a term used in machine learning and refers to the number of training examples utilised in one iteration. The batch size
can be one of three options:

batch mode: where the batch size is equal to the total dataset thus making the iteration and epoch values equivalent
mini-batch mode: where the batch size is greater than one but less than the total dataset size. Usually, a number that can be divided into the total dataset size.
stochastic mode: where the batch size is equal to one. Therefore the gradient and the neural network parameters are updated after each sample.

How do I choose the optimal batch size, for a given task, neural network or optimization problem?
If you hypothetically didn't have to worry about computational issues, what would the optimal batch size be?

Here are a few guidelines, inspired by the deep learning specialization course, to choose the size of the mini-batch:

If you have a small training set, use batch gradient descent (m < 200)

In practice:

Batch mode: long iteration times
Mini-batch mode: faster learning
Stochastic mode: lose speed up from vectorization

The typically mini-batch sizes are 64, 128, 256 or 512.
And, in the end, make sure the minibatch fits in the CPU/GPU.
Have also a look at the paper Practical Recommendations for Gradient-Based Training of Deep Architectures (2012) by Yoshua Bengio.

How does one normally check if the minibatch fits the CPU? and the GPU?

you know that training a deep learning model that has a lot of parameters has a relation to CPU obviously, for practical guidelines especially from an academic perspective, there is that tradeoff between accuracy and computational resources. But if you find something very useful please share it here :)

@gota You can calculate this in advance, but the easiest way to discover what fits in GPU memory is to increase the batch size until training fails from lack of memory. This should happen pretty quickly.

From the blog A Gentle Introduction to Mini-Batch Gradient Descent and How to Configure Batch Size (2017) by Jason Brownlee.

How to Configure Mini-Batch Gradient Descent
Mini-batch gradient descent is the recommended variant of gradient
descent for most applications, especially in deep learning.
Mini-batch sizes, commonly called “batch sizes” for brevity, are often
tuned to an aspect of the computational architecture on which the
implementation is being executed. Such as a power of two that fits the
memory requirements of the GPU or CPU hardware like 32, 64, 128, 256,
and so on.
Batch size is a slider on the learning process.

Small values give a learning process that converges quickly at the cost of noise in the training process.
Large values give a learning process that converges slowly with accurate estimates of the error gradient.

Tip 1: A good default for batch size might be 32.

The main content in this answer was completely copied from another source. We highly discourage this type of answer, but we prefer that users give answers in their own words, unless the quote is particularly relevant, but, even in that case, we expect users to provide more than just the quote.

The batch size can also have a significant impact on your model’s performance and the training time. In general, the optimal batch size will be lower than 32. In April 2018, Yann Lecun even tweeted:

Training with large minibatches is bad for your health.
More importantly, it's bad for your test error.
Friends dont let friends use minibatches larger than 32.

A small batch size ensures that each training iteration is very
fast, and although a large batch size will give a more precise estimate of the gradients, in practice this does not matter much since the optimization landscape is quite complex and the direction of the true gradients do not point precisely in the direction of the optimum.


text:
Which model should I use to determine the similarity between predefined sentences and new sentences?
The Levenshtein algorithm and some ratio and proportion may handle this use case.

Based on the pre-defined sequence of statements, such as "I have a dog", "I own a car" and many more, I must determine if an another input statement such as "I have a cat" is the same or how much percentage does the input statement is most likely equal to the pre-defined statements.

For Example:

Predefined statements: "I have a dog", "I own a car", "You think you are smart"

Input statements and results:

I have a dog - 100% (because it has exact match), I have a cat - ~75% (because it was almost the same except for the animal, think - ~10% (because it was just a small part of the third statement), bottle - 0% (because it has no match at all)

The requirement is that TensorFlow be used rather than Java, which is the language I know, so any help with what to look at to get started would be helpful.
My plan was to use the predefined statements as the train_data, and to output only the accuracy during the prediction, but I don't know what model to use. Please, guide me with the architecture and I will try to implement it.

If this is a simple syntax comparison, neural networks is not the best way to achieve this.
If it's semantic comparison, then you can take a look at models used in the SNLI dataset for example.
From your question, it looks like just a syntax comparison.
Consider the 2 sentences :

She likes playing guitar
She likes listening guitar

The 2 sentences have almost the same words, but the meaning is different.
Now consider these 2 sentences :

The bird is taking a bath in the fountain
Birdie wash himself with water in public place

These 2 sentences have almost no words in common, but the meaning are very similar.

So if your use case need to return a high-score for the first example, give up neural network (it is possible, but pointless).
If your use case need to return a high-score in the second example, take a look at SNLI leaderboards, there is plenty of models that can works.

authors: Евгений Ветлугин


text:
How to constraint the output value of a neural network?
I am training a deep neural network. There is a constraint on the output value of the neural network (e.g. the output has to be between 0 and 180). I think some possible solutions are using sigmoid, tanh activation at the end of the layer.
Are there better ways to put constraints on the output value of a neural network?

Please give more detail about how the output value is interpreted. This will help to understand what kind of loss function and activation of the last layer are suitable. Do you have 181 classes ? Is it a position or an angle ?

There are many ways of constraining the network's output.
Using an activation layer is a good one. If you sigmoid the output layer, the output is constrained between [0,1] and you can multiply that by 180 to adapt to your output. Since the layer is part of the optimization process, gradients will be learned correctly.
However, there are a few issues with it, namely sigmoid is not a linear curve, and is quite steep around 0. Maybe you can consider something like a ReLU, but with a maximum as well. This makes your output linearly between [0,180], and is also learned during optimization.
You could also just clip the output without any activation layers. This is simple and works, but the optimization process sees no difference between outputting 180 or 180000. This can lead to problems, depending on your problem.


text:
Why did the L1/L2 regularization technique not improve my accuracy?
I am training a multilayer neural network with 146 samples (97 for the training set, 20 for the validation set, and 29 for the testing set). I am using:

automatic differentiation,
SGD method,
fixed learning rate + momentum term,
logistic function,
quadratic cost function,
L1 and L2 regularization technique,
adding some artificial noise 3%.

When I used the L1 or L2 regularization technique, my problem (overfitting problem) got worst.
I tried different values for lambdas (the penalty parameter 0.0001, 0.001, 0.01, 0.1, 1.0 and 5.0). After 0.1, I just killed my ANN. The best result that I took was using 0.001 (but it is worst comparing the one that I didn't use the regularization technique).
The graph represents the error functions for different penalty parameters and also a case without using L1.

and the accuracy

What can be?

You have a small dataset.  Should you even be using neural nets?  Have you done any diagnostics to see if you even have enough data?  Are you using the right metric?  Accuracy is not always the correct metric.  Which weights are you retaining?  You will overfit if you save the weights that produce the lowest training error.  Save the weights that produced the lowest validation error.  L1, L2, and dropout are all great.  So many things not described in the problem...
http://www.ultravioletanalytics.com/blog/kaggle-titanic-competition-part-ix-bias-variance-and-learning-curves
I'm wondering why you're not trying interpretable models to see if the resulting weights for the features make sense.  Also if your comparing all those models and parameters, set your random initial starting point to be the same by setting the seed.  I also hope you are using the same training set for each model.
You probably need more data...

Your network without regularization does not appear to be over fitting but rather it appears to be converging to a minima. I am actually a bit surprised it is doing as well as it is given that your data set is small. So You don't need regularization. If you want to improve the accuracy you might try using an adjustable learning rate. The Keras call back ReduceLROnPlateau can be used for this. Documentation is here. Also use the callback ModelCheckpoint to save the model with the lowest validation loss. Documentation is here. It would help a lot if you posted your model code. I have found if you do encounter over fitting dropout works more effectively than regularization.


text:
Is time/space estimation of possible actions required for creating an AGI?
Given infinite resources/time, one could create AGIs by writing code to simulate infinite worlds. By doing that, in some of the worlds, AGIs would be created. Detecting them would be another issue.
Since we don't have infinite resources, the most probable way to create an AGI is to write some bootstrapping code that would reduce the resources/time to reasonable values.
In that AGI code (that would make it reasonable to create with finite resources/time) is it required to have a part that deals with time/space estimation of possible actions taken? Or should that be outside of the code and be something the AGI discovers by itself after it starts running?
Any example of projects targeting AGI that are using time/space estimation might be useful for reaching a conclusion.
Clarification, by time/space I mean time/space complexity analysis for algorithms, see: Measures of resource usage and Analysis of algorithms
I think the way I formulated the question might lead people to think that the time/space estimation can only apply to some class of actions called algorithms. To clarify my mistake, I mean the estimation to apply to any action plan.
Imagine you are an AGI and you have to make a choice between different set of actions to pursue your goals. If you had 2 goals and one of them used less space and less time then you would always pick it over the other algorithm. So time/space estimation is very useful since intelligence is about efficiency. There is at least 1 exception though, imagine in the example before that the goal of the AGI is to pick the set of actions that leads to the most expensive time/space set of actions (or any non-minimal time/space cost) then obviously because of the goal constraint you would pick the most time/space expensive set of actions. In most other cases though, you would just pick the most time/space efficient algorithm.

I don't know if time/space estimation will be explicitly programmed into an AGI, but the estimation of the computational resources to perform a certain action is definitely useful. Humans (especially, rational humans) regularly do this.
In reinforcement learning, for instance, the agent may be able to discover what are the most efficient actions to achieve the same goal, without having a dedicated component for estimating the space and time requirements. However, if it had a dedicated component, it might be able to perform more efficiently.
If you think of heuristic-based state-space search algorithms, like A*, you can see that they explicitly incorporate a way to estimate the resources to achieve the goal (in this case, the cost of the path to the goal), i.e. the heuristic function $h$. This turns out to be very useful, provided the heuristic is also suitable for the problem.
In the Godel machine, which can be viewed as a theoretical model of AGI (or general problem solver), we also have this notion of proving (so not just estimating) that rewriting the code is useful before rewriting it.


text:
How could I use machine learning to detect text and non-text regions in scanned documents?
I have a collection of scanned documents (which come from newspapers, books, and magazines) with complex alignments for the text, i.e. the text could be at any angle w.r.t. the page. I can do a lot of processing for different features extraction. However, I want to know some robust methods that do not need many features.
Can machine learning be helpful for this purpose? How could I use machine learning to detect text and non-text regions in these scanned documents?

Since the document is scanned, it will not be in an open document format so no associated API can be used.
Approach 1
Evaluate TextBridge Pro, FreeOCR, and other alternatives that purport page layout detection.  If any of them work, drive them programmatically (preferably headless) to read the scanned document, detect page layout and OCR the text, export to a document with an open format, and then use the API
With this approach, the object recognition AI is in the product and development time and resources are saved.
Approach 2
Do a 2D FFT windowing through the page in both directions.  See the cosine, trapazoidaly, Hamming, and Hanning windows and apply them in horizontal and vertical directions.  Use Approach 1, assuming those products work with the scanned documents to label the examples, and then train a DCNN (deep convolutional NN) to recognize from the 2D FFT output spectra where the pictures are.  By interpolation, close to a perfect crop of the images and the text regions can be obtained with some hyper-parameters on the model obtained.
Approach 3
This approach is just Approach 2 but preparing the labeled example data set by hand, which may be necessary because the existing software products may not handle the images being laid out at angles other than 0, 90, 180, or 270 degrees.
Approach 4
Create an architecture that is based on feature extraction, and use font rendering libraries to build the back half of an auto-encoder, allowing portions of image that do not auto-encode to be preserved as an x-y coordinate pair, which will allow the images to jump over the pictures if the convergence is set up correctly.
Final Note
One can offload some processing to a learning process, so that the actual document process runs faster, but sometimes the preparation of the example data set and the learning consumes more resources.  That's why those who assess which approach will cost less and can recommend the best approach with some reliability are high paid.

TextDetector, Tesseract and other open source packages implement text detection (object detection for text). There's also a pretrained Tensorflow model that does text detection. A text detector will give you the bounding boxes in your image for any text that it recognizes. In the case of Tesseract, it will also output the text (OCR is built in). So you can read the code in these packages to get ideas for your own machine learning pipeline. Basically you need both a regressor (for the bounding boxes) and a classifier (to detect whether the box contains text or not).


text:
What is meant by "model discriminability for local patches within the receptive field"?
In the abstract of the paper Network In Network, the authors write

We propose a novel deep network structure called "Network In Network"(NIN) to enhance model discriminability for local patches within the receptive field

What does the part in bold mean?

This is also mentioned in Flattened Convolutional Neural Networks for Feedforward Acceleration: "We also use 1 × 1 filters across channels to increase model discriminability for local patches within the receptive field."


text:
Can we combine multiple different neural networks in one?
I want to make a kind of robotic brain, i.e. a big neural network, which includes an NLP model (for understanding human voice), real-time object recognition system (so that it can identify particular objects), a face recognition model (for identifying faces), etc.
Is possible to build a huge neural network in which we can combine all these separate models together, so we can use all 3 model's capabilities at same time in parallel?
For example, if I ask the robot, using the microphone, "Can you see that table or that boy?", the robot would start recognizing the objects and faces, then answer me back by speaking if it could identify them or not.
If this is possible, can you kindly share your idea how can I implement this? Or is there any better to make such AI (e.g. in TensorFlow)?

I got the same idea. Did you find any papers/researches on combining several different neuron networks?

It seems that you want to build something like the robot Sophia, which can recognize objects and faces, and answering you by speaking out loud. According to Ben Goertzel, Sophia was partially built on top of OpenCog, so maybe that's the software tool you are looking for, but note that this tool does not use just neural networks.


text:
What are ontologies in AI?
What exactly are ontologies in AI? How should I write them and why are they important?

See for example https://protege.stanford.edu/publications/ontology_development/ontology101-noy-mcguinness.html

An ontology at its most abstract is a model of the world. It describes concepts that exist in the world and how those concepts are related.
Ontologies are similar to taxonomies. A taxonomy is a tree-like hierarchy that organizes concepts in increasing levels of specificity. What an ontology adds is a second type of link between those concepts that explains how they are connected. So a taxonomy could say $isA(DOG, ANIMAL)$ ("a dog is an animal"). An ontology may also describe something like: $chase(DOG, CAT)$ ("dogs chase cats").
Why use an ontology? It can be used to reason about the world. I subscribe to a very particular kind of language-centric ontology that isn't worth addressing here. Instead, the Semantic Web project (Tim Berners-Lee) is probably closer to what would interest you. Semantic Web uses a type of description logic, which is outside my realm of expertise. But there are tools for processing these kinds of DL and gaining "understanding" from them. To work with this kind of ontology you'll want to be familiar with the idea of Resource Description Framework triples.


text:
How to use Machine Learning to create a "Draw-A-Person Test"
The process revolves around a child's drawing. Each part of each drawing corresponds to a score as in the Draw a Person Test conceived by Dr. Florence Goodenough in 1926. The goal of the machine is to measure a child's mental age through a figure drawing task.

That's a very nice challenge!
As always, the hardest part is to get a labeled database (maybe by scrapping).
You'd probably need some thousands of drawings and their respective drawer age.
From there, you need to make an image regression model. Here is a simple example that predicts age from a face photo. It's the same principle, but applied to another database, with other relevant features to be learned by the model.


text:
Why does Q-learning converge to the optimal policy, even if the agent acts sub-optimally?
In Q-learning, during training, it doesn't matter how the agent selects actions. The algorithm always converges to the optimal policy. Why does this happen? What's the intuition?

You can find a proof of the convergence of the Q-learning algorithm in the paper Convergence of Q-learning: A Simple Proof by Francisco S. Melo.

Your statement "it doesn’t matter how I select actions" is not really true. Q-learning "requires that all state-action pairs be visited infinitely often", as it's mentioned in the paper I linked you to above and e.g. in the book RL: An Introduction by Barto and Sutton.

So, what is your real question? Are you looking for a proof? If yes, then you can find it in the paper above. Or are you looking for an intuition behind the convergence of Q-learning?

I am actually looking for an intuition behind this.

Q-learning is an off-policy learning algorithm. We are following the behaviour policy, $b$, which is $\epsilon-$greedy. This behaviour policy need not be an optimal policy rather it is a more explorable policy. But we are learning the target policy, $\pi$, which is argmax of state action value $(Q(s,a))$. This target policy is by definition optimal policy.
From the $\epsilon$-greedy policy improvement theorem we can show that for any $\epsilon$-greedy policy (I think you are referring to this as a non-optimal policy) we are still making progress towards the optimal policy and when $\pi^{'}$ = $\pi$ that is our optimal policy (Rich Sutton's book Chapter-5). Here $\pi^{'}$ is the new policy and $\pi$ is the previous policy.

Think of this diagram, where we are selecting action based on $\epsilon$-greedy policy but still making progress towards the optimal policy $\pi_*$.

This does not answer the question why, if we use a non-optimal policy, Q-learning converges to an optimal policy.

I think the op denoted $\epsilon$-greedy policy as sub-optimal policy. But from $\epsilon$-greedy policy improvement theorem we can show that the $\epsilon$-greedy policy converges to optimal policy.

You may want to include the usual diagram, which I think is quite insightful.

Where did you get this diagram from? Which version of the book are you specifically referring to? Can you provide the link? Isn't this diagram applied to generalized policy iteration and the policy shouldn't be greedy?

I took this diagram from David silver lecture slide (model-free control). Yes, this diagram applied to generalized policy iteration.


text:
Given an image and two points $A$ and $B$ on that image, how could we find a path from $A$ to $B$?
If we have a search or path-finding problem, A* and Dijkstra's algorithm require that we formulate it as a search in a graph with nodes and connections between these nodes. If there are obstacles, we also need to encode this information in the graph, so that they are not traversed. Additionally, there may be costs/weights on the connections between points. If such weights/costs are high, the algorithms won't take that path.
I've been using A* and Dijkstra's algorithm this way so far. However, it's a bit cumbersome to always have to define the nodes/points and the relationships (or connections) between them. There's no learning here. I just define a graph and the algorithms search on this graph.
Let's say I have a white image, a green blob in the middle, and points $A$ and $B$ at either side of the blob, I need to get from $A$ to $B$. I don't have a search space represented as a graph here. I just have this image.
Could I use machine learning to solve this problem (and would it generalize to more complex maps)? If so, are there any research works on this topic? Or is that the wrong route to take (pardon the pun)?

Let's say I have a white image, a green blob in the middle, and points $A$ and $B$ at either side of the blob, I need to get from $A$ to $B$. I don't have points to traverse. I just have this image.

If your image is as "simple" as you describe here, with very easily distinguishable colours, the easiest solution would likely be to construct a graph as expected by algorithms such as A* as follows:

Every pixel becomes a node in the graph, with connections to all adjacent pixels
Green pixels (the green blob in the middle) are marked as non-traversable (or simply don't have connections to adjacent pixels) (I assume this thing is your obstacle).
Pixels belonging to the points on either side of the obstacle are marked as starting and goal vertices.

Then, you have the graph that a traditional pathfinding algorithm like A* would expect, and can simply run the algorithm there.

If you get less "clear" images (i.e. not brightly coloured, easily distinguishable things, but more like real-life top-down images of an area), then the above won't really work anymore. In such cases, you'd likely want to look into Reinforcement Learning approaches, where you give an agent a reward when it successfully manages to find a connection between start and goal. In particular, you'd need Deep Reinforcement Learning approaches, because otherwise you're not going to be able to handle complex images.
Note that these approaches are not going to be simple to implement or immediately understand for 100% if you're a beginner. There won't be any "beginner-friendly" approaches for the problem you describe, it simply is quite a complex problem (requiring a mix of "decision-making" and "understanding" images).


text:
Why does the number of feature maps increases in the VGG model?
I found the below image of how a CNN works

But I don't really understand it. I think I do understand CNNs, but I find this diagram very confusing.
My simplified understanding:

Features are selected
Convolution is carried out so that to see where these features fit (repeated with every feature, in every position)
Pooling is used to shrink large images down (select the best fit feature).
ReLU is used to remove your negatives
Fully-connected layers contribute weighted votes towards deciding what class the image should be in.
These are added together, and you have your % chance of what class the image is.

Confusing points of this image to me:

Why are we going from one image of $224 \times 224 \times 3$ to two images of $224 \times 224 \times 64$? Why does this halving continue? What is this meant to represent?
It continues on to $56 \times 56 \times 256$. Why does this number continue to halve, and the number, at the end, the $256$, continues to double?

Why are we going from one image of $224 \times 224 \times 3$ to two images of $224 \times 224 \times 64$?

They do a Convolution with a $1 \times 1$ kernel, with $64$ filters. That way, you keep your size as the inputs ($224 \times 224$), but changes the number of filters to 64.
And this is not two image, but two layers !

Why does this halving continue? What is this meant to represent?

This is the maxpooling operation (the layer is in red colour, see the legend). In general, max pooling is applied with a kernel stride of 2 (rarely 3, but not more, because you lose too much informations). That way your size is cut by 2, and the activation maps are more little, and you have faster computation.

It continues on to $56 \times 56 \times 256$. Why does this number continue to halve, and the number, at the end, the $256$, continues to double?

Still max-pooling. There are no reason to double the numbers of filters, though. This just a trend, but you can put the numbers of filters that you want.

The number of feature maps is increased because we increased the number of filters that are applied throughout the process.
Here is the reason why:

Filters at first layers would find simple features like lines and edges. Subsequent layers would take those features -in other words, patterns- and combine them to create much bigger patterns.
Therefore we should increase the size of the filters to meet the increased size of complexity. Since the filter size increased, the number of feature maps would increase as a result.
Also, keep in mind that filters at the deeper layers tend to find more complicated features.


text:
How does Hindsight Experience Replay learn from unsuccessful trajectories?
I am confused by how HER learns from unsuccessful trajectories. I understand that from failed trajectories it creates 'fake' goals that it can learn from.
Ignoring HER for now, if in the case where the robotic arm reaches the goal correctly, then the value functions ($V$) and action-value functions ($Q$) that correspond to the trajectories that get to the goal quicker will increase. These high $Q$ and $V$ values are ultimately important for getting the optimal policy.
However, if you create 'fake' goals from unsuccessful trajectories - that would increase the $Q$ and $V$s of the environment that lead to getting the 'fake' goal. Those new $Q$ and $V$s would be unhelpful and possibly detrimental for the robotic arm to reach the real goal.
What am I misunderstanding?

Ignoring HER for now the $Q$ and $V$ functions operate on states and actions which are part of a Markov decision process which we call $M_0$.
Back to HER, the $Q$ and $V$ functions now take a goal as an additional parameter called the goal. We will denote individual goals $g_n$, the true goal $g_0$, and the set of all goals $G$. The set of goals is chosen such that every state matches at least one goal. We create a new MDP $M_1 = M_0 \times G$ (i.e. a larger MDP composed of multiple copies of $M_0$, all the states in each being tagged with a goal). The reward is either +1 or 0 depending on whether or not the goal and the component from $M_0$ match in some predefined sense. In HER trajectories are collected from the subset of $M_1$ where $g = g_0$ and added to the replay buffer. When training the $Q$ and $V$ functions we don't only use the original trajectories: we create new ones by substituting new values of $g$, which we do so strategically so as to include some trajectories with a positive reward.
Things to note:

HER doesn't assign rewards to states or trajectories in $M_0$: the reward function used is only defined for $M_1$
The performance of HER depends on $Q$ and $V$ both being models with some ability to extrapolate to unseen data points; such as neural networks, support vector machines, etc. It would not provide any benefit if applied to value tables.

Comments are not for extended discussion; this conversation has been moved to chat.


text:
What are the differences between uniform-cost search and greedy best-first search?
What are the differences between the uniform-cost search (UCS) and greedy best-first search (GBFS) algorithms? How would you convert a UCS into a GBFS?

In the case of UCS, the evaluation function (that is, the function that is used to select the next node to expand) is $f(n) = g(n)$, where $g(n)$ is the cost of the path from the initial node to $n$, while in the case of the greedy BFS it is $f(n) = h(n)$, where $h(n)$ is the heuristic function that estimates the cost of the path from $n$ to the goal node. In other words, in the case of UCS, nodes are expanded only using the experience (in the form of $g(n)$), while, in the case of GBFS, nodes are expanded only using the estimate of the cost to the goal. Note that, in both cases, the node that is chosen to be expanded is the one with the smallest $f(n)$ value.