text:
Are Computer Vision and Digital Image Processing part of Artificial Intelligence?
There are some fields of Computer Vision that are similar to Artificial Intelligence. For example, pattern recognition and path tracking. Based on these similarities, can we say that the Computer Vision is a part of Artificial Intelligence?

Artifical Intelligence is a rather fuzzily-defined field. It includes lots of subject areas. Sometimes that's because they replicate some behaviour or mental ability of creatures used to solve problems, and sometimes it is because they were the focus of "classic" AI, and have remained classified as part of AI. These problems include navigation, searching, solving combinatorial or logic puzzles, complex systems control, communication. Many of the problems can include converting visual input to something more useful to the rest of the system.
Computer Vision by comparison appears more tightly-defined. However, it quickly becomes less so when you try to consider what the eventual outputs should be. To what extent does a computer vision system need to "understand" the visual inputs in order for other system components to make sense of them, and use them correctly? Here are a couple of examples to illustrate my point:
Image captioning
For instance, work in automated captioning effectively includes natural language models. The input to the system is an image. The computer vision half of that system does not output a class or direct measurement of something inside the image. Instead it outputs a vector encoding of the description of the whole image - something that the language model can decode into an English sentence.
The vision model and language model are trained together, so this encoding can be very loosely considered an internal "ideagram" that represents the contents of the image and allows the vision and language models to share representations.
Agents using vision systems for decision-making
Another interesting example is recent work on Deep Q Networks, used to give superhuman performance when playing some Atari games. This used a neural network architecture (CNN) already known to be good for image classification problems, and used it to directly connect the pixel output from a game to a system which learned optimal behaviour. This is more clearly work in AI - creating a learning agent that works across a general set of problems - but at its core is a CV system.
In this case, the input is four consecutive images (although there are variations which learn from the stream of individual frames), and the output is a vector which scores the value of taking one of the possible controller actions (left, right, up, down, fire etc). Variations on the system directly output which action would be preferred from the CNN.

You could take this further and require that a vision system produce some kind of explainable, transferable, shared world model for other systems to interact with. An embryonic version of that approach is effectively how the captioning system works. It is not at all clear whether such an imagined design is strictly in the field of CV or some other part of AI.
Whether such outputs are considered part of computer vision or more generally part of AI is less of a pure scientific classification issue than an opinionated or political one (e.g. due to funding either an AI research team or a CV one).
The similarities between current computer vision approaches and ongoing work in AI, such as heavy use of deep learning models, would suggest that CV is strongly related to AI. Intuitively I would say that CV is part of AI, perhaps a key specialism of it, roughly equivalent in scope to Natural Language Processing. However, the practical outcome of that is only important to people who have labelled themselves "AI researcher" or "CV engineer" etc.
I also think the same arguments apply to other studies of artificial perception, most notably speech recognition.

There is also a habit of techniques that started out being considered as AI becoming just engineering once we've worked out an algorithm to do it (and therefore require less 'adaptive' intelligence).

This answer contains good points, but I think you're exaggerating a little bit. Definitely, computer vision is part of AI, in the same way that RL, NLP, expert systems, etc., are part of AI. Now, if you ask me if image captioning is just a CV problem, that's a different story. CV is taught in many universities as part of an AI program. Throughout the years, many techniques that have emerged and that are now considered part of CV have not been necessarily labelled as AI, but the AI field has also evolved over the years (as you suggest).


text:
How to know which kind of adverb in NLP Parts of Speech (POS) tagging?
There are 4 kinds of adverbs :

Adverbs of Manner. For example, slowly, quietly
Adverbs of Place. For example, there, far
Adverbs of Frequency. For example, everyday, often
Adverbs of Time. For example, now, first, early

nltk, spacy and textblob only tag a token as an adverb without specifying which kind it is.
Are there any libraries which tag including the type of adverb?

Generally POS taggers only use broad categories, especially if they work statistically: the more fine-grained the tagset is, the more training data you need. And unless there are differences in the distribution, the accuracy of the tagger would not be improved by adding more categories, eg

I ran [there/fast/often/early].

all work -- so nothing is gained by having subcategories of adverbs.
I suggest the easiest solution for this would be to have a list of adverbs by category, and simply look them up after the tagging. Just pick all words tagged as adverbs and check if they are in your list. Unless there are ambiguities (ie some adverbs can belong to multiple categories) it should be very straight forward, and would also work with any tagger/tagset that marks adverbials.
You might have to consult a grammar book to get a list of adverbial classes, but that again should be easy to do.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Does it make sense to apply softmax on top of relu?
While working through some example from Github I've found this network (it's for FashionMNIST but it doesn't really matter).
Pytorch forward method (my query in upper case comments with regards to applying Softmax on top of Relu?):
def forward(self, x):
    # two conv/relu + pool layers
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))

    # prep for linear layer
    # flatten the inputs into a vector
    x = x.view(x.size(0), -1)

    # DOES IT MAKE SENSE TO APPLY RELU HERE
    **x = F.relu(self.fc1(x))

    # AND THEN Softmax on top of it ?
    x = F.log_softmax(x, dim=1)**

    # final output
    return x

Does it make sense?
In general, yes it is interpretable, back propagation will work, and the NN can be optimised.
By using ReLU, the default network has a minimum logit of $0$ for the softmax input, which means at least initially that there will be higher minimum probabilities associated with all classes (compared to allowing negative logits which would happen randomly with usual weight initialisation). The network will need to learn to produce higher logit values for correct answers, because it has no ability to produce lower logit values for incorrect answers. This is like training a network to produce the highest regression value on one output, whilst clipping all values to be 0 or above, so it does not have the option of making one output e.g. $-1.0$ and the rest $-100.0$
It can probably be thought of as a type of regularisation, as it puts constraints on activation values that will work.
Is it needed?
That is less clear. You can try training with and without the line, and using cross-validation or a test set to see if there is a significant difference.
If the network has been designed well, then I'd expect to see a slight improvement with the added ReLU.
If it is a mistake, then I'd expect to see no difference, or better performance without the ReLU.

@DuttaA: The weights can be negative between layers though, so one layer being all 0 or positive does not mean next layer will be. Otherwise you would only need one ReLU layer for the whole network . . . also ReLU can only add meaningful non-linearity when there are negative inputs. An all-positive input is unchanged by ReLU. You cannot use ReLU to "add non-linearity" to a vector of all positive numbers, it would do nothing.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
An architecture for classifying distance from origin for a sum of vectors?
I built a three-layer neural network (first is 1D convolutional and the remaining two are linear). It takes an input of 5 angles in radians, and outputs two numbers from 0 to 1, which are respectively the probability of failure or success. The NN is trained in a simulation.
The simulation goes this way: it takes 5 angles in radians and calculates the vector sum of 5 vectors having $x$ as module and $\alpha$ as angles (taken from the input). It returns $1$ if the vector sum has a module greater than $y$, or $0$ if it is less than $y$.
My intention is to be able to tell sequences of radians that will generate vectors with a sum greater than $y$ in module from the ones which won't.
Which would be the best configuration to achieve this? Is the configuration I set up (1D convolution layer + 2 linear layers) efficient? If so, would it be easy to find the right size for the convolution? Or should I just remove it?
I noticed that if I change the order of the input angles the output of the simulation will be the same. Is there a particular configuration you should use when dealing with these cases?

I'm not completely sure I understand your simulation. What I think you are doing is:

Generate 5 angles specified in radians (Are these always normalized to within (0, $2*\pi$)?).
Interpret each angle as a unit vector in a 2d space.
Add the unit vectors together, yielding a vector that lies somewhere inside a circle with radius 5.
Ask whether the summed vector is more or less than a distance $y$ from the origin of the circle.

If you're doing that, your problem looks like trying to learn a separation of two concentric rings, which is a well known benchmarking problem for classification.
I am reasonably certain you can learn this pattern with several layers of ReLU neurons. I'm not certain that convolutional layers will help you much here. The main patterns I'd expect the network to learn are:

perhaps 2-3 layers to learn whether the point lies far away from the origin in each of several different directions.
1 layer to learn where the decision boundary is in each of 4 directions away from the origin.
1 layer to inclusive-OR the 4 decision boundaries together.

My guess is that this is fairly easy to learn with 4 layers of 8 ReLU neurons, or something like it.

You got the question exactly right. I was not thinking about a unit vector but scaling the unit vector for my custom steps instead, anyway the procedure stays the same. It is very inspiring for me to see you actually taking a guess about the meaning of each layer! Also, very interesting idea of ring separation, a well known problem, which is what I hoped I would find!


text:
Algorithm to solve a fault independent of its type
I am looking to plan a solution for a workspace fault and not hardware faults.
Consider a task where a robot has to move balls from one place to another. In case it faces any condition which is outside the task for eg. someone snatches the ball from the robot while it is transferring or the robot drops the balls in between. These are some example faults that could occur, many other might be possible. I am trying to build a generalized algorithm that so that the robot can find a way to resolve unexpected changes itself.
I currently have an FSM for the whole task. Any fault that somehow changes any of the state machine variables should be considered. For instance, there are faults that deal with obstacles that may come in the way.
But there might be faults for example a cloth in front of the camera. This fault should be corrected by a human since the robot cannot manage that. All the faults like that are out of the scope of the robot.
Any suggestion or ideas related to the algorithm will be helpful.

The second one. In the first statement, I would need to know the faults which I might be not fully aware of. I am building an algorithm that is independent of the type of fault. Initially the robot knows what the correct state is, with the help of the a state machine.

There are faults that deal with obstacles that come in the way. But there might be faults for example a cloth  in front of the camera. This mentioned fault you stated should be corrected by a human since the robot cannot manage that. Any fault that somehow changes any of the state machine variable shall be considered. All the faults apart from that are out of the scope of the robot.

Thanks for the clarifications. I have attempted to edit those into the question (this is standard on Stack Exchange, to clarify a question by using [edit] ). Feel free to adjust it further in case I have changed things to say something you did not really mean.

Thats very nice of you


text:
How to implement AI strategy for Mastermind
I'm looking to implement a AI for the turn-based game Mastermind in Node.JS, using Google's Tensorflow library. Basically the AI needs to predict the 4D input for the optimal 2D output [0,4] with a given list of 4D inputs and 2D outputs from previous turns in the form of [input][output].
The optimal output would be [0,4], which would be the winning output. The training data looks like this:
[1,2,3,4][0,1] [0,5,2,6][3,1] [0,2,5,6][2,2] [6,5,2,0][4,0] [5,2,0,6][0,4]

So given these previous turns
[1,2,3,4][0,1] [0,5,2,6][3,1] [0,2,5,6][2,2] [6,5,2,0][4,0]

the AI would predict an input of [5,2,0,6] for the output [0,4].
I've looked at this post but it talks about only inferring input for a output without any context. In Mastermind, the context of previous guesses and results from them are critical
My algorithm would need to use the information from previous turns to determine the best input for the winning output ([0,4]).
So my question is: How can I implement AI for Mastermind?

I think what you're looking for is Reinforcement Learning.

Mastermind is something of a special case, since although there is some randomness and strategy picking a pattern to guess and opening moves to narrow down the guess, once some data has been collected it rapidly narrows down to a logic problem. Is your question specifically  about Mastermind? If so the "turn-based game with context" part may not be that important

@NeilSlater Yes, mainly just Mastermind

I have edited to make that clearer, because you have something of an "X/Y" problem if you ask for  "turn-based game with context"  when you actually want to solve Mastermind. Further to that, I doubt you need tensorflow or deeplearning here to write a good automated Mastermind player - so could you clarify whether your goal is specifically to apply some kind of neural network model (which is interesting but a lot of effort and may perform poorly) or want to understand and solve Mastermind?

You could possibly apply neural networks, reinforcement learning to summarise results of previous choices (what you are calling context) and use score predictions to suggest the next turn's guess. However, the game of Mastermind has a small search space and it is possible to process this "context" more directly by refining a set of guesses. This will be much more efficient and simpler to understand than a neural network approach. It would be very hard to make a neural network variant which was as efficient - either in terms of CPU time, or in terms of number of turns it takes to find a solution.
In practice, a Mastermind solver is much like a Hangman solver, or a Guess Who? solver. You have an initial large set of all possible answers, and need to narrow it down to a single correct answer. You do this by processing after each guess to reduce the set of answers that meet all the constraints that the game has given you so far.
The agent needs to know the score function that compares a target value with the guess and returns the score. Let's call that score(guess, target)
The algorithm looks like this:
(Opponent sets unknown_target)
Initialise possible_answers as list of all valid targets

For each turn:
  Select one of possible_answers as this turn's guess
  Ask opponent for gscore = score(guess, unknown_target)
  If gscore is [0,4] then exit(win)
  If it was last guess then exit(lose)
  For each possible_answer in possible_answers:
    pscore = score(guess, possible_answer)
    If pscore != gscore, then remove possible_answer from possible_answers

You can finesse this for the stage Select one of possible_answers as this turns guess by trying to optimise either by psychological model of opponent or trying to find choices that are likely to cause the best reduction in size of possible_answers. However, a simple random choice should do quite well.
Also worth noting is that the algorithm does not depend on the exact nature of the scoring function, so it is applicable for many variations of guessing games. It does rely on the score providing information that will reduce the remaining set of guesses. In some games that may mean taking more care about the precise nature of a guess, in order to maximise this effect.
Out of interest, I implemented this algorithm and tested it when there were 10 choices at each position (i.e. digits 0 to 9), maximum of 10 guesses allowed, and the target to guess was set randomly. Using random guesses and the algorithm exactly as written, the above approach guessed correctly 9,996 times out of 10,000, and on average the guesser won the game in 6.2 turns.

although I have seen RL solutions to this problem, (those I saw) fail to realize that the state of mastermind is not observable, as there is the "secret" we're trying to guess.
mastermind is best approached as a constraint satisfaction problem, along the lines described by Neil Slater. The whole trick is to realize that you can eliminate options from the "current possible alternatives" set by treating the latest guess as the target and eliminate any combinations that don't agree with it e.g. (using 3 digits for clarity)
last guess=123, scores [2,0] i.e. 2 white, 0 black
then the current alternatives are eliminated if they don't score [2,0] against the last guess 123:
124 [0,2]
214 [2,0]
215 [2,0]
321 [2,1]
let's say the secret is 215, you see that our method of elimination is correct, albeit we don't know the secret!
I have seen lots of different approaches (genetic engineering, information theory etc), but the plain truth is that a 50-line matlab piece of code with random guessing will give a winning stragegy that averages 4.3 guesses for the standard mastermind game (1296 alternatives)


text:
Reinforcement learning with industrial continuous process
I am new to RL and wish to realize a RL control for an industrial process. The goal is to control the temperature and humidity in a vegetal food production chamber.
States: External temperature and humidity, internal temperature and humidity, percentage of the proportional valves controlling heater, cooler and steam for humidity. The goal is to keep temperature and humidity in the chamber (measures) as close as possible to the desired values (the setpoints).
Agent actions: Increase/decrease the percentage of the proportional valves controlling the actuators.
Rewards: Deviation between measure and setpoints (small deviation => high reward, high deviation => low reward).
I have data available, the history of states and actions from a real system. The actions are made by several PID controllers (some of them in cascade). So far I have about 3 month every minutes (with some stops sometimes when a chamber is for example cleaned). The data are continuously logged and every month I get more data. The data includes bad/unwanted states.
For training the RL agent, I am planning to simulate the environment using a supervised learning model (with the predict function), probably XGboost. Is it feasible, are there pitfalls to avoid in this case?

Are you asking if it's feasible to simulate the environment using SL? Anyway, it's not clear  (to me) how you want to simulate the environment given only a labelled training set.

@nbro: OP could train a predictive model for $p(s'|s,a)$ - or maybe $p(s', r|s,a)$, depending on whether reward is imposed by starting goals, or is some measure of output of the industrial process. The simplest environment "simulation" is pretty much experience replay, you would use SL instead if you hoped to generalise from the data set, separately from simply training e.g. Q-learning. Choices such as trying to create a sampling model vs distribution model, and limits of approximate/learned envirobment models might be useful parts of any answer. Also, whether there is any benefit to the idea

@NeilSlater Ha, right. Good points! But I think the OP should be more specific and ask a clearer and more specific answer.

@nbro Thanks for your answer. Yes the question is is it feasible.

@brz OP refers to the asker of the question. It's an acronym for "Original Poster" and in this case refers to you, so it's nothing to do with RL. Anyway, I think you should clarify how your dataset looks like and try to be more specific regarding what you want to be feasible.

OP = Original Poster i.e. you. I don't understand your description of the reward. I am not sure it is important for the purpose of this question, but I do suggest you add more details about the environment and what you are trying to do. Also, why you want to create a simulator using supervised learning - what is your motivation for that?

The goal is to control the temperature and humidity in a vegetal food production chamber. States: External temperature and humidity, internal temperature and humidity, percentage of the proportional valves controlling heater, cooler and steam for humidity. The goal is to keep temperature and humidity in the chamber (measures) as close as possible to the desired values (the setpoints). Agent actions: increase/decrease the percentage of the proportional valves controlling the actuators. Rewards: deviation between measure and setpoints (small deviation=> high reward, high deviation => low reward.

@NeilSlater I have no specific motivation for SL, just wish to compare RL algorithm and train agent before it goes in the real world. It is not clear in my previous post but the two setpoints are also part of the state space.

@NeilSlater You said. "OP could train a predictive model for p(s′|s,a) - or maybe p(s′,r|s,a)". Is it possible with a continuous space state ?

@brz: It is maybe possible to create a distributional model from your data, but might not be the best approach. It definitely is not the simplest approach. Could you please explain what you mean by "I have data available" in more detail - I would guess you have historical data for state and actions (where the actions have been taken by another controller, perhaps a human expert?). How much of this data do you have, and how comprehensively does it cover the state and action space? Does your data include bad/unwanted scenarios that the simulator can learn, and the agent can learn to avoid?

@NeilSlater Yes the data are historical of states and actions. The actions are now made by several PID controllers (some of them in cascade). So far I have about 3 month every minutes (with some stops sometimes when a chamber is for example cleaned). The data are continuously logged and every month I get more data. Yes the data are containing bad/unwanted states.

Thank you. I have edited that information into the question for you. For reference if you ask another question or want to add more details here, you can also do that yourself using the [edit] link on the left under your question text.

This is a very hard problem as there exist time dependencies and you don't have an ability to actually simulate this.

RL is a generic technique that can be applied to any MDP system. From the looks of it you have data to produce a state-space model of your system (system identification, excluding your existing control loops) and then you can use that to drive a simulated exploration of your process and discover a control policy. As this is a continuous process, some discretization will be required
so yes, your quest is feasible, but not trivial!
I experimented sometime ago with the cartpole balancing system, which is a toy, but I found that a good old PID was much better than AI :)


text:
Is the PyTorch official tutorial really about Q-learning?
I read Q-learning algorithm and also I know value iteration (when you update action values).  I think the PyTorch example is value iteration rather than Q-learning.
Here is the link:
https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html

TL;DR: It is Q learning. However Q learning is basically sample-based value iteration, so not surprising you see a similarity.
Q learning* and value iteration are very strongly related. When considering action values, both approaches use the same Bellman equation for optimal policy, $q^*(s,a) = \sum_{r,s'}p(r,s'|s,a)(r+\gamma \text{max}_{a'} q^*(s', a'))$ as the basis for update steps. The differences are:

Value iteration makes updates using a model of the environment, Q learning works from samples from the environment made by an active agent.

By working from a simulated environment rather than a real one, it may not be clear when an agent is model-free or model-based (or planning rather than acting). However, the way that the simulated environment in the PyTorch example is used is consistent with a model-free method.

Value iteration loops through all possible states and actions for updates independently of any action an agent might take (in fact the agent need not exist). Q learning works with whichever states the agent experienced.

By adding experience replay memory in DQN, Q learning becomes a little bit closer to value iteration, as you can frame the memory as a learned model, plus consider it to be a type of planning (or a "sweep" through states). This is how it is described for instance in DynaQ which is an almost identical algorithm to experience replay as used in DQN when both are used in the simplest versions - see Sutton & Barto chapter 8.

Value iteration value update steps are over an expectation of next states and rewards - it processes the weighted sum $\sum_{r,s'}p(r,s'|s,a)$. Q learning update steps are over sampled next states and rewards - it ends up approximating the same expectation over many separate updates.

Even using large amounts of experience replay memory does not get Q learning the same as value iteration on this issue, samples are not guaranteed perfect. However, in a deterministic environment, this difference is not meaningful. So if you have a deterministic environment, Q learning and value iteration may also be considered a little closer in nature.

* Technically this applies to single-step Q-learning. n-step Q-learning and Q($\lambda$) use different estimates of future expected return, that are related but not the same as the single-step version shown here.

but according to Q-learning formula that code is incorrect, they use value iteration update rather than Q-learning.

@datonefaridze: I cannot see that. Can you give me the line of code reference wheer you think the update is using value iteration? As far as I can see, the TD target is calculated and used as I would expect for Q learning - i.e. from samples of agent behaviour, and not abstractly from a distribution model.

I think you should emphasize that saying that the use of the experience replay in DQN makes DQN model-based is a little bit of a stretch. It's true that ER is a dataset from which you sample and that contains samples from the environment, but it doesn't provide you the probabilities that a model of the environment would give you (and that you need in value iteration), although you can estimate these probabilities from the ER. Maybe there's some research work that formally shows something similar to what you are saying, but I am currently not aware of it.

@nbro: Yes, there is some support for ER being a model. There is very little difference indeed between ER as used in DQN and the background model-based planning from DynaQ, they are exactly the same algorithm (until you add options like n-step). In DynaQ the memory is explicitly considered a learned sampling model. Sutton & Barto presents it this way in the chapter on planning. It is correct however, that ER cannot be a distributional model (which you would need for value iteration).

Maybe you can provide a reference to the paper/book that shows that. Anyway, I am not saying that the ER cannot represent a distribution. It can, but it's just an approximative model.

@nbro: I added that to the answer - my reference for this is Sutton & Barto chapter 8.


text:
Non-differentiable reward function to update a neural network
In Reinforcement Learning, when reward function is not differentiable, a policy gradient algorithm is used to update the weights of a network. In the paper Neural Architecture Search with Reinforcement Learning they use accuracy of one neural network as the reward signal then choose a policy gradient algorithm to update weights of another network.
I cannot wrap my head around the concept of accuracy as a non-differentiable reward function. Do we need to find the function and then check if it is mathematically non-differentiable?
I was wondering if I can use another value, for example silhouette score (in a different scenario) as the reward signal?

I do not understand your question. It doesn't usually matter in RL whether a reward function is differentiable with respect to any parameters or not. Are you asking about why that is the case? Are you asking why accuracy is a non-differentiable function? Do you have a specific problem to solve (NN to train) and want to use the same approach as in the paper?

I am asking if I can use silhouette score as a reward signal to update a network (a simple Multi-layer Perceptron) by policy gradient. I want to know if is this is correct according to what policy gradient is used for.

OK, that makes sense. Can you explain "I can not wrap my head around the concept of accuracy as a non-differentiable reward function." more - what is the issue here? Is it that you think accuracy is differentiable?

So, accuracy is a number. What do we consider as a non-differentiable is the formula for accuracy here? how can I think about it mathematically?

The part of accuracy that is non-differentiable is the comparison operator for whether the prediction equals the ground truth or not. That does not have any meaningful gradient, and it only acts on point values (typically discrete set ${0,1}$), so does not have any regions over which gradient can be measured (unlike e.g. ReLU). So any attempt to figure out gradients of parameters that caused a specific accuracy value will fail when they reach that step.

I cannot wrap my head around the concept of accuracy as a non-differentiable reward function. Do we need to find the function and then check if it is mathematically non-differentiable?

In reinforcement learning (RL), the reward function is often not differentiable with respect to any learnable parameters. In fact it is quite common to not know the function at all, and apply a model-free learning method based purely on sampling many observations of state, action, reward and next state. For reward, you don't need to know the reward function as a function of your parameters, but you do need to know how to caclulate it from each observation. That means that the reward is either provided as part of the environment, or it is clear how to calculate it from the initial state, action or next state. In the case of an agent that creates a neural network then reports the accuracy, you can probably view the NN training process and its results including accuracy on a validation set as a "black box" that reports back an arbitrary reward signal.

In Reinforcement Learning, when reward function is not differentiable, a policy gradient algorithm is used to update the weights of a network

Policy gradient methods are one broad family of RL methods which are also often model-free. Alternatively, value-based methods are broadly the other choice (e.g. Q-learning) or can be used in combination with policy gradient methods (e.g. Actor-Critic). All of these can be model-free, and often are.
Unless you want to apply a model-based RL method, then you have no need to find the reward function in an explicit form based on any parameters. Even if you did want to use a model-based approach, the reward function does not need to be differentiable.

I was wondering if I can use another value, for example silhouette score (in a different scenario) as the reward signal?

Yes this would likely be viable, and could be made to work similarly to the paper, provided there is some meaningful connection between the parameters you manipulate and the end result.
RL can be made to work as a generic way of optimising numeric measure influenced due to a vector of parameters. That includes indirect optimisation of neural networks by non-differentiable metrics such as accuracy. There is nothing special about the choice of accuracy here, the metric just needs to be related to the parameters being learned.
There is a catch. Indirect learning using RL can produce very noisy training data, due to e.g. covariance between all the parameters, and it can take many samples to get a clear gradient signal through all the noise and make meaningful updates. It can be inefficient compared to other optimisation methods if they are available - however, the ability to extract a gradient for the parameters even when rules are complex and non-differentiable is a nice feature of RL.


text:
Artificial life simulator that is fully embodied and passes open endedness tests
Geb is an alife simulation that as far as I know passes all of the tests we have tried to come up with in defining open endedness. However, when you actually run the code, the behavioral complexity certainly increases, but the physical bodies of the creatures never changes (and cannot change), and Geb bodies are the only thing present in the world.
Tool development, or at least developing new physical capabilities and new “actions” seems to be a crucial part of what makes evolution open ended. I think the problem with Geb is that the evolution and progress all takes place in the networks and network production rules, which are systems outside the physical world. They are external systems that take in data from the world and output actions for the agents. So while this rich complexity and innovation is occurring, it’s not integrated with the agents actions and physical bodies.
This leads to a simple question: is there an alife system that passes all the same tests Geb does, but is “fully embedded” in its world? In the sense that any mechanism agents use to make actions must be part of the physical world of those agents, and subject to the same rules as the bodies of the agents themselves.
What I’m saying here is loose, you could come up with plenty of weird edge cases that meet what I ask exactly without meeting my intent. And perhaps being fully embodied isn’t necessary, just being more embodied would be enough. What my intent is is to ask if we have any systems that pass the open endedness tests Geb have passed, but have the innovation occurring in a way that leads to emergent growth of “actions” and emergent growth of “bodies” because to evolve and do better those aspects must be improved as well.

Geb software available from http://www.channon.net/alastair/#Software - I'm not sure how/where to link that

It's not an alife system, but are you aware of the work of Karl Sims? https://www.karlsims.com/evolved-virtual-creatures.html There are some examples of creatures set to compete directly with each other. The control system complexity is not embodied or part of the fitness criteria, but available "actions" do appear spontaneously

@NeilSlater yes! You are right that the action set seems to be pretty emergent there, especially when they compete for the object. It’s an example where just moving more (but not all) innovation to the physical world still does quite a bit. Has anyone ran the open endedness tests on some variant of sims work?


text:
Handling large or multiple agent actions that cause non-stationarity
How to handle a problem where the agents' actions are changing the environment? For instance, a trading agent can hypothetically make a huge transaction that is enough to push the stock prices to go up or down.
Another example could be an electric grid environment. Assume a sufficient number of large consumers utilize an RL agent in their energy management system. If these agents happen to have similar trained policies they might collectively affect stable operation of the electric grid and even lead to a blackout.
Are there specific mechanisms, methods or algorithms to handle this type environments?

I'm not sure what you mean, are you not just referring to transitioning to a new state? ideally the state space would encapsulate all possible states of the environment, so there's not really a notion of the environment 'changing' with the agent's actions.

@David I think this is an issue of what is being modelled during training. The two scenarios presented imply that the agents are trained against a "large, robust" environment where their actions don't usually cause the kinds of changes that the OP is concerned about. Modelling such effects in a training environment, or repeating online training enough times to learn from them, would be next-level hard in terms of resources or complexity required.

i’d be worried if an RL agent would not change/influence the environment…

@Alberto True, but read the question body. Perhaps you could suggest a title improvement?

I've edited the title and tried to capture what I think is the essence of the question in more exacting RL terminology. Feel free to roll back or further adjust my changes.

@NeilSlater To my knowledge, non-stationarity is inherent to certain environments, i.e. they change not due to agents' actions but because of their specific inherent dynamics. Here I consider an environment that is stationary by nature, as you mentioned "large, robust". Happy to hear your thoughts on this before I change the title.

Non-stationarity means the environment is not identical over time, and fits here - in general it is not necessary to identify the nature/cause of the environment changes in order to apply the label. The problem you have does not have a standard name in RL as far as I know, and could equally be framed as non-stationarity or as not having a complete model for training purposes. Which is more useful way to frame it depends on the solutions available to you

@NeilSlater I'm still not entirely sure on OP's problem. Do they mean whereby an action space could consist of taking multiple discrete actions per time step, e.g. like seen in DecQN ? here the non-stationarity is caused from the fact that each sub-action selection policy (i.e. the policies derived from utility values) changes.

@David - from what I can tell, the issue is that the OP is concerned about a shift in environment impact from agent behaviour between "small" and "large" changes, where large can be driven in at least two ways (a single agent which has gathered lots of resources and can deploy them in a single action, or multiple similar agents). What these changes both have in common is that they are harder to model or train in advance for, and are potentially much rarer to experience. I don't think there is an easy solution here, definitely not a general one that fits all scenarios

@David However, I do think that something useful could be written about issues of this kind, approaches and limitations that impact RL.

@NeilSlater, what are the methods for handling non-stationary environments?

This is too complex to answer in comments. I think one reason you don't have any answer is that a lot depends on details, so it is hard to write a good general answer to your question. It is likely any correct answer will not lead you to a technical fix, but be more of a statement on limitations of ML in the scenarios you present


text:
Is reinforcement learning suitable for application automation?
I have basically automatised the use of an app through the use of OCR and computer vision.
So basically when a word or an image is detected it will perform a certain action.
When that action is successfully completed it will go to the next state.
Now I want to try basically with a more "heuristic" approach and I thought about reinforcement learning.
Why? Because I am aiming to build a tool that basically understand automatically what actions to perform in a certain state.
But I have a doubt.
Even though I don't need to declare an association like this (it would beat the purpose of deep reinforcement learning or deep learning in general):
  if(state.MENU_VIEW)
    clickManager.clickOnFolder();
    ...

I still need to define the states, the actions and the reward.
Meaning I would need to instruct my app that when the OCR result is "Open Folder" it means the state I am in is MENU_VIEW. I simply wouldn't tell my app what action to perform in a that state. Am I correct?
What I am trying to say is: how exactly could I make it so that the states (and maybe also the actions?) are generated automatically?
The reward in this case scenario would be basically the folder being opened successfully.

Welcome to AI Stack Exchange. It's better if the title is more self-contained (e.g. don't refer to "this scenario" and hope people will click through to read the whole body). I will edit it. If you don't like the edit, feel free to adjust further

Could you clarify what is setting the goals of the automation? I.e. how are the multi-step goals decided, and communicated to the agent?

@NeilSlater thank your for the welcome and thank you for adjusting the title, as a matter of fact it was redundant.

@NeilSlater In the example provided I could basically pass an int from the environment to the agent: -1 if the click is far from the folder icon, 0 is if near and 1 if it's exactly on top of the folder.

That would seem like a reward signal - something you would send after the agent has taken an action. What I am asking is how does the agent know what its goal is? Is the goal always to open a folder page, or is it more long term than that, and how flexible is the long term goal? I.e. are you trying to train an agent to achieve one specific task where the UI may vary (so it needs to learn where to look in different systems, but always to do the same thing), or are you trying to train the agent to undertake multiple similar tasks, and it needs to somehow infer which one?

@NeilSlater no, the folder page was just an example. Basically the goal is to reach the end of the application.
Let's say you have a series of dialogs or pages, the long-term goal is basically clicking on all the "right" buttons until you reach the last page which will display "Operation completed". A sort of visiting problem so to speak. For now I simplified the problem since there are instances in which a simple click might not be enough and are required multiple clicks to go to next state.

Is the agent aware of the nature of the required operation when it starts, or is it trying all menu options that might lead to "the end" until it is successful? Does it get any clue on what it is attempting to do other than make the end message appear somehow?

Well, that is exactly what I am not so sure.
In my program, as I said, I would basically use OCR to detect the "right path".
I've seen a reinforcement learning project of the game snake and the required operation, at first, was indeed random between RIGHT, LEFT, UP and DOWN.
So, in my case I thought about clicking randomly on the menu options that might lead to the end.
No, it doesn't get any more clues other than end message. At least so far I haven't thought about any more clues. Maybe I am still having a thought pattern that revolves around OCR.

More questions, sorry. But this is relevant: Will the trained agent be used on exactly the same or sites it was trained on, or is it attempting to learn a strategy that could work on multiple sites (despite not knowing what its goal is)?

@NeilSlater you're absolutely right, I should have mentioned it. The trained agent will be used exactly on the same site it was trained on.

In the scenario as you present it, reinforcement learning (RL) should work, but you may gain very little by applying it, over simpler search algorithms.
As you explain in comments, each application is separately trained, the task in each case is arbitrary, and there is no way for the agent to apply contextual knowledge to identify likely actions. The only feedback will be task completion. There are no heurtistics available, simply a list of available actions on each step.
In terms of RL problem definition, this is very similar to solving a maze in least number of steps. The problem is episodic (it terminates), so setting a cost per action (negative reward e.g. $r = -1$ on each step) is the simplest way for the agent to get feedback on how it is doing. The highest total reward will represent the shortest path.
Simple maze solvers are often used as toy problems in RL tutorials. You could use pretty much any example of a gridworld maze from such a tutorial as a starting point to make an agent for your problem, and replace the up/right/left/down actions with your more abstract list of text links or menu options. The state representation will depend on application behaviour, but could be as simple as the current list of available actions.
For this final part, I am assuming the following things, implied by the question and comments:

The application behaves completely deterministically.
The entry point to the application is always the same.
The output goal is a single, shortest-possible list of simple actions (labels to select from) that reach the goal from the start.

If all the above are true, then RL is effectively a random search for the goal state followed by further random iterations to remove non-necessary steps and find the shortest path. In which case, other tree-searching algorithms may be simpler to implment and have better performance than RL. You could try depth-first search (DFS) and/or breadth-first search (BFS), perhaps also requiring loop detection/avoidance. These would avoid the randomness inherent in an RL-based search, and likely be far more efficient due to that.

Thanks for the answer! Since I may gain very little by applying it, do you think in this scenario would be preferable a deep learning algorithm (maybe DQNs)?
Or should I just stick with the OCR and computer vision approach?

@zaxunobi Deep learning will only help in similar scenarios to RL, when your automation scenarios have something in common and when something is known about the goal task when starting each one. The combination with OCR/vision is a different part of the puzzle. Potentially there is something worth doing there with ML. In the question you presented this as an existing component, but learning to click on meaningful components based on a screenshot might be learnable, separate to finding a correct path to end point

Since the actions are pretty much clicking on buttons, what contextual knowledge could an agent apply to determine meaningful components (the action is always a click, what changes are the coordinates of the click)?
About the deep learning scenario, do you have in mind how it could learn to to click on meaningful components based on screenshots?

@zaxunobi I don't know enough about your goals and constraints to answer that, and definitely not in the comments. I find it odd that you are writing an automation tool without information on automation goals at the start, but only as a discoverable reward. If you want to explore options for deep RL, I suggest asking a new question giving more details of the project than you have so far

The main goal is reaching a certain state of the program, or the "end" if you like.
The reason why I am going with "discoverable reward" it's because the only information that I could provide it's basically (which I am doing with OCR already) is that if a certain text is detected then I am in a specific STATE and therefore will be performed a specific ACTION. But doing this would defeat the whole point of using RL, don't you think?

I've opened a new question: https://ai.stackexchange.com/questions/46033/is-deep-learning-suitable-preferable-for-string-similarity-detection-and-applica


text:
Two player grid-based game, minimax not good, what to try?
In this game, a matrix with size N x M is given (10<=N,M<=15). There a three main type of value in the matrix:

A integer show how many gold is it
Char 'D' show obstacle
Char 'S' show shield (which can obtain to get through obstacle, once obtained can be used the whole match, there only 2 shield). Shield can not protect player when two players collide.
There are only two player. Player move simultaneously.
There're about 50 to 70 turn then the game will end.
There are chance treasure appear on the map (only once) with value 3/4 * d to 5/4 * d with d is the difference in gold of two player at that moment.

Find a way to maximize the gold. I've tried Minimax but it's seem not verry efficient. Is there any better approach to this problem?

Welcome to AI Stack Exchange. This looks on topic, but the title could be better. A good title is self-contained. So I'm going to attempt to rewrite it for you. If you don't like what I do,you can use [edit] to go back or improve it further

As a two-player game, what is the main objective? Maximimise total gold in colaboration, adding both players? For competing players, getting more gold than the other player, or getting the highest amount of gold possible (those are similar, but not the same objective)?

minmax in perfect information game is guaranteed to converge to a nash equilibrium if you are in a zero-sum games, but i can't see anything on the rewards/utilities


text:
Types of activation functions used in neural networks
I need to specify the points and conditions that require me to use one activation function over another,

Why is it not possible to use the sigmoid FUNCTION in hidden layers, and it is preferable to use Relu instead?
When do we use the Relu function in hidden layers and when do we use Tanh?

Here is general guide/practice, https://www.turing.com/kb/how-to-choose-an-activation-function-for-deep-learning. Please add the task you want to use your neural network and what model under your consideration in your question. It will help us to give you good suggestion.

Let me give a very easy explanation...
Activation Functions in neural networks are useful as they introduce non-linearity into the model, which helps in complex data pattern understanding.
Sigmoid Function

Sigmoid converts output values into range of (0,1).
It is mostly used for binary classification problems.
It suffers from Vanishing Gradient Problem. As the Input value becomes small or large. The gradient turns out very very small (near about zero).This will slow down the training process because weights and biases receives very small updates.
E.g. :  For a large input, say 10, sigmoid(10) ≈ 0.9999. The gradient is sigmoid(10) * (1 - sigmoid(10)) ≈ 0.0001, which is very small.

ReLU Function

ReLU converts all the negative input to 0 and positive as it is
It is mostly used in hidden layers which allows faster training because of the linearity in non-saturating form.
It helps in the vanishing gradient problem of hidden layers.
In some cases, there is a dyeing ReLU problem, where neurons sometimes only output zero(for all negative inputs).
E.g. : For an input of 5, ReLU(5) = 5. For an input of -3, ReLU(-3) = 0, and the gradient is 0.

Tanh Function

The Tanh function outputs values in the range of (-1,1). It is somewhat similar to sigmoid but stretched for a wider range
It is very useful when data is centered around zero, Useful in situations where input is negative.
Tanh also suffers from vanishing gradient but lesser extent.
E.g. :For a large input, say 10, tanh(10) ≈ 1. The gradient is 1 - tanh(10)^2 ≈ 0, which is very small.

Other
There are other activation functions like

Leaky ReLU: It is a variation of ReLU but allows a small negative result. This will prevent dead neuron problem.

PReLU(Parametric ReLU): It is modified Leaky ReLU where small slope of negative input is learned during training.

ELU(Exponential Linear Unit): Positive input will be as it is and for negative result exponential decay function is used. It is combination of sigmoid/tanh with ReLU.

SELU(Scaled Exponential Linear Unit): It will encourage self-normalizing properties for faster and more stable training.

SoftMax: Converts vector of values into a probability distribution. Where probability is in proportion to its exponential. Mostly used in the output layer of multi-class classification.

Others are SoftPlus, Swish, Hard Sigmoid, Hard Swish, Gaussian, etc...


text:
Designing antenna to optimise return loss, which model might be a good start for a beginner?
I am not very well versed with Machine learning. I am doing a small project as part of my coursework in my university. So I am trying to write a program tries to design a rectangular patch antenna using ML.
The input is the frequency of operation and output is the length, breadth, height and feed-point of the patch antenna. The length and breadth can be calculated with a formula. But for a good return loss, the height and feed-point position have to determined by some experimentation.
I am planning to use ML to predict these 2 values. My question is which ML model do you think will be good for implementing the above?
I have built a dataset which contains the length, breadth, height and feed-point of the patch antenna and the corresponding return loss.

Hi, welcome to AI stack exchange. It is good practice to write your qeustion briefly, but exactly as possible in the title, because it is the first thing people will see before deciding whether or not to look further or try to answer. I will make a short edit - if you disagree or want to re-word further that is fine, use the [edit] link for that.


text:
Neural network for specific numbers from a range (Q learning)
PROBLEM AT HAND: I have a resource (Bandwidth) of B Hz. I have to distribute the bandwidth B to users as per their requirements. For instance, voice calls would require some amount of bandwidth while video calls would require some amount of bandwidth. Virtual reality applications and Industrial IoT would require more bandwidth than the above two cases. The fraction of bandwidth out of total bandwidth B that I can assign are as follows: [0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]. That is to say, anyone among the following fractions can be assigned.
I am trying to design a Reinforcement Learning Agent that would assign bandwidth to different applications based on the amount of bandwidth that a particular application requires.
I am trying to use the following template for my application.
Deep Q Learning
MY QUESTION:
I am not able to understand how can I tell the neural network that it can take only one of the following values:
[0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]
class Agent:
    def __init__(self, state_size, action_size):
        self.memory = deque(maxlen=2500)
        self.learning_rate=0.001
        self.epsilon=1
        self.max_eps=1
        self.min_eps=0.01
        self.eps_decay = 0.001/3
        self.gamma=0.9
        self.state_size= state_size
        self.action_size= action_size
        self.epsilon_lst=[]
        self.model = self.buildmodel()

    def buildmodel(self):
        model=Sequential()
        model.add(Dense(32, input_dim=self.state_size, activation='relu'))
        model.add(Dense(32, activation='relu'))
        model.add(Dense(self.action_size, activation='linear'))
        model.compile(loss='mse', optimizer=Adam(lr=self.learning_rate))
        return model

    def add_memory(self, new_state, reward, done, state, action):
        self.memory.append((new_state, reward, done, state, action))

    def action(self, state):
        if np.random.rand() > self.epsilon:
            return np.random.randint(0,4)
        return np.argmax(self.model.predict(state))

    def pred(self, state):
        return np.argmax(self.model.predict(state))

    def replay(self,batch_size):
        minibatch=random.sample(self.memory, batch_size)
        for new_state, reward, done, state, action in minibatch:
            target= reward
            if not done:
                target=reward + self.gamma* np.amax(self.model.predict(new_state))
            target_f= self.model.predict(state)
            target_f[0][action]= target
            self.model.fit(state, target_f, epochs=1, verbose=0)

        if self.epsilon > self.min_eps:
            self.epsilon=(self.max_eps - self.min_eps) * np.exp(-self.eps_decay*episode) + self.min_eps

        self.epsilon_lst.append(self.epsilon)

    def load(self, name):
        self.model.load_weights(name)

    def save(self, name):
        self.model.save_weights(name)

I am not able to understand how can I tell the neural network that it can take only one of the following values

You don't have to. The neural network in Deep Q Learning (the DQN) is not configured to output what the actions are. There are a few different possibilities, but in this case the network has been setup to predict the action value of all possible actions as an array of outputs. This is probably the most common setup for a DQN. The DQN doesn't name the actions or associate them with any value from the environment (e.g. in your case the bandwitch allocation fraction). It only knows the actions by their index position in its output.
You are free to associate each action's effect on the bandwidth allocation, 0.0, 0.1, 0.2 etc to arbitary action choices. Provided you do it consistently, it doesn't matter which action index from 0 to 10 gets associated with which item. However, you may as well make a simple intuitive mapping, by dividing the action index by 10.
When interpreting the DQN's output, it might predict for a certain input state e.g. [-0.5, 2.1, 3.7, 1.6, 0.2, 0.5, 0.1, -0.2, -0.5, -0.9, -1.7] - those are the predicted returns from choosing actions. The greedy action from that result would be action index 2, and if you followed the mapping convention I suggested that would be to allocate $0.2$ of the bandwidth.
There are other Reinforcement Learning methods that do work directly with numerical actions by predicting either a single number or distribution of numbers that can be used as the action. Q learning does not do that.

def action(self, state):
    if np.random.rand() > self.epsilon:
        return np.random.randint(0,4)
    return np.argmax(self.model.predict(state))

You do not need to tell the network it can only take one of the actions.  It is by these lines that a single action is decided upon.  Predict returns the Q-values for all of the actions.  Argmax gets the action with the highest Q-Value (the highest predict future reward).
As an aside, the randint under epsilon needs to be of the same action_size.


text:
What is the difference between on-policy and off-policy for continuous environments?
I'm trying to understand RL applied to time series (so with infinite horizon) which have a continous state space and a discrete action space.
First, some preliminary questions: in this case, what is the optimal policy? Given the infinite horizon there is no terminal state but only an objective to maximise the rewards, so I can't run more than one episode, is it correct?
Consequently, what is the difference between on-policy and off-policy learning given this framework?

Although all your questions are related, please, next time, ask one question per post! You're asking at least 3 distinct questions here.

First, some preliminary questions: in this case, what is the optimal policy?

It is the policy that maximises return from any given time step $G_t$. You need to be careful with your definition of return with continuing environments. The simple expected sum of future rewards is likely to be positive or negative infinity.
There are three basic approaches:

Set an arbitrary finite horizon $h$, so $G_t = \sum_{k=1}^{h} R_{t+k}$
Use discounting, with discount factor $\gamma < 1$, so $G_t = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}$
Use average reward per time step,  $\bar{R} = \lim\limits_{h \to \infty}\frac{1}{h} \sum_{k=1}^{h} R_{t+k}$, which leads to thinking in terms of differential return $G_t = \sum_{k=1}^{\infty} (R_{t+k} - \bar{R})$

With large enough horizon (so that state is ergodic) or large enough $\gamma$ (close to $1$), these approaches are similar and should result in approximately the same policy. The difference is in how you construct an agent in detail to solve the problem of maximising the value.

Given the infinite horizon there is no terminal state but only an objective to maximise the rewards, so I can't run more than one episode, is it correct?

The term episode becomes meaningless. It may be simpler to think of this differently though - you are trying to solve a non-episodic problem here, in that there is no natural separation of the process into separate meaningful episodes. No physical process is actually infinite, that's just a theoretical nicety.
In practice, if you can run your environment in simulation, or multiple versions of it for training purposes, then you do start and stop pseudo-episodes. You don't treat them as episodes mathematically - i.e. there is no terminal state, and you can never obtain a simple episodic return value. However, you can decide to stop an environment and start a new one from a different state.
Even if there is only one real environment, you can sample sections of it for training, or attempt to use different agents over time, each of which is necessarily finite in nature.

Consequently, what is the difference between on-policy and off-policy learning given this framework?

The notions of on-policy and off-policy are entirely separate from episodic vs continuing environments.

On-policy agents use a single policy both to select actions (the behaviour policy), and as the learning target. When the learnt policy is updated with new information that immediately affects the behaviour of the agent.
Off-policy agents use two or more policies. That is one or more behaviour policies that select actions (the behaviour policy), and a target policy that is learned, typically the best guess at the optimal policy given data so far.

These things do not change between episodic and continuing tasks, and many algorithms remain identical when solving episodic vs continuing problems. For example, DQN requires no special changes to support continuing tasks, you can just set a high enough discount factor and use it as-is.
You cannot wait until the end of an episode, so certain update methods won't work. However, value bootstrapping used in temporal difference (TD) learning still works.
In some cases, you will want to address the differences in the definition of return. Using an average reward setting typically means looking at a differential return for calculating TD targets for example.

So for example if I want to run a Deep Q-learning  with experience replay algorithm, having only one real environment, I should simply not consider the "for" loop of episodes?

@aandre_90: You may still want to arbitrarily stop and reset state whilst continuing to train, so could keep some kind of outer loop for that. You would never see an episode complete though, so terminating the inner loop needs to be based on something else for that to be useful. Otherwise, yes you could drop that outer loop per episode.


text:
What does the figure "Blackjack Value Function..." from Sutton represent?
I came across this graph in David Silver's youtube lecture and Sutton's book on reinforcement learning.
Can anyone help me understand the graph?
From the graph, for 10000 episodes what i  see is that when we don't have a usable ace we always lose the game except if the sum is 20 or 21. But in case if we have a usable ace there are chances to win when our sum is below 20. I don't know how this is possible.

Hi, it would be useful if you could edit your question and add some details about what it is you don't understand.

@OliverMason i have updated the question. Thanks for your comments.

The left hand graphs are showing you the estimated value function from using Monte Carlo evaluation, after 10,000 episodes. They give a sense of what your value table will look like before convergence.
In the case of upper "usable ace" chart, the estimates are still showing a lot of inaccuracy due to variance in the data. This is for two main reasons:

The probability of getting a usable ace at the start is a fraction of all (around 15%), so the number of samples used to build the chart is lower.
There are more variations in play when there is a usable ace, due to the extra flexibility allowed by it, so the end result also varies more, requiring more samples to converge.

In addition, if you are looking at the bottom edge of the chart, this represents the player starting with two aces. If you pick on one of the high points (dealer shows 4), then that also reduces the probability of seeing that particular state. So you are looking at a sample size of typically 4-5, but maybe in this case just one sample or maybe two, which the player then happened to go on to win, even though the odds made it unlikely. There is always some chance of winning and dealer showing 4 is a bad start for the dealer, who has a good chance of going bust provided the player does not.
If it hadn't happened this time for the "two aces + dealer showing 4" state, it may have happened for "two aces + dealer showing 5" state. That's due to  the nature of random sampling - if you have hundreds of states to sample, then a few of them are going to behave like outliers purely by chance until you have taken enough samples.
In short, 10,000 randomly sampled games are nowhere near enough to reduce error bounds on the value estimates to reasonable numbers for special cases such as starting with two aces. However, you can see in the 10,000 samples charts the beginnings of convergence, especially elsewhere in the chart.

From the graph, for 10000 episodes what i see is that when we don't have a usable ace we always lose the game except if the sum is 20 or 21.

Actually you don't see that, the expected result is not quite -1.0, but a little higher. So that means there is still a chance to win. Under this player policy, the worst chance is no usable ace and score 19, because the policy will be to "hit" and need an Ace or 2 card just to stay in the game. Even then , the value is not quite as low as -1.0, but more like -0.9

authors: ΧΡΗΣΤΟΣ ΜΙΧΑΛΗΣ


text:
Can OpenAI simulations be used in real world applications?
I know that classical control systems have been used to solve the problem of the inverted pendulum - inverted pendulum.
But I've seen that people have also used machine learning techniques to solve this nowadays - machine learning of inverted pendulum.
I came across a video on how to apply a machine learning technique called reinforcement learning on openAI gym - OpenAI gym reinforcement learning.
My question is, can I use this simulation and use it to train a controller for a real-world application of inverted pendulum?

Hi, welcome to AI Stack Exchange. I'm not sure I understand your last sentence - do you mean you want to build a controller (maybe trained using RL) using the simulation, then attempt to use it to interface to and control a physical cart pole system that you have been supplied or intend to build?

Yes,I was thinking I could do that.

In general, you can use a simulation to prepare and train a controller for a real world application. A good example of this being done for robotics is in the paper Autonomous helicopter flight via reinforcement learning where a Reinforcement Learning agent was trained on a model of helicopter dynamics before being used in reality. Often, as in this case, such work is done to avoid expensive failures due to the trial and error nature of RL - if an error is expensive, such as crashing a helicopter, then ideally the agent performs the checks to avoid it in simulation, by planning or some other virtual environment as opposed to in the real world.
The main hurdle to completing training in simulation then transfering to the real world, is the fidelity of the simulation. The simulation's physics, including measurements of physical quantities, the size of time steps, amount of randomness/noise, should match between the simulation and the target real-world environment. If they do not match, then a learning agent could generate a policy that works in simulation, but that fails in reality.
For the autonomous helicopter, the researchers used data from a human operator controlling the real helicopter, to help generate a predictive model that was used in the simulation.
Can you do the same with Open AI Gym environments? Probably not, unfortunately. The main issue is that the units used are fixed in most environments, and are unlikely to closely relate to any specific real world implementation of the same kind of system. In addition, the physics is often simplified - probably a minor issue for CartPole, but a more major one for environments like LunarLander which ignores weight of fuel used and is a 2D simulation of a 3D environment.
So, for instance, in CartPole environments, the following values are fixed:

Size of time step
Mass of cart
Mass and length of pole to be balanced
Force that cart motor pushes with

There are a couple of approaches you could use to work around this:

Make a new version of the environment and adjust it so that values match to a real environment you want to train for. Note this may still be limited, as the physics model is still quite simple, and may not allow for the real operating characteristics of the cart motor.
Use the CartPole environment as-is, not to train a controller directly, but to select hyper parameters, such as neural network size, learning rate etc. That will result in a learning agent that you are reasonably confident can learn policies with the state representation and general behaviour of your target system. You then train "for real" again in the physical system.

You can combine these ideas, creating a best-guess controller from simulation, then refining it in a real environment by continuing the training on a real system.

Thanks. Since I am new to machine learning, I don't know much about it. In ordinary control systems that I am familiar with, we use PID to tune the controller.  I've seen some videos of an inverted pendulum using reinforcement learning.  How does reinforcement learning work? Does it learn to tune the values of PID automatically? Or is it a different algorithm altogether?

@blazingcannon: I cannot answer that in comments with any level of detail, and suggest you ask a new question, but take care not to make it too broad. Reinforcement learning basically uses trial and error to tune a policy which maps observed state to a controler output (an "action"). Although it uses different terminology to classic control, a lot of the core ideas are the same. RL can solve wider class of problems than PID, but at the expense of being less efficient where PID works well.

Thanks, I will consider asking another question. But since you said PID is more efficient than reinforcement learning, I am prompted to ask whether PID can outperform RL in situations where the environment keeps changing?

@blazingcannon: That will all depend on details, such as the nature of the change and the orginal control task. You won't get any PID to play a game of Go at all, whilst the best RL agent (Alpha Zero) should cope just fine if the opponent it faced changed every turn


text:
How does a model based agent learn the model?
I want to build model-based RL. I am wondering about the process of building the model.
If I already have data, from real experience:

$S_1, a \rightarrow R,S_2$
$S_2, a \rightarrow R,S_3$

Can I use this information, to build model-based RL? Or it is necessary that the agent directly interact with the environment (I mean the same above-mentioned data should be provided by the agent)?

If you already have some transition tuples then you can train a model to predict environment dynamics using these. However, you should be careful that your pre-gathered data is diverse enough to 'cover' enough of the state/action space so that your model remains accurate. For instance, when you start training your agent it will likely start to see more of the state space than it did at the start of training (imagine playing Atari, initially your agent will die quickly but as it gets better episodes will get longer) so you would need to make sure you have data for these states that appear late in episodes, otherwise your model will just be overfitting to the start of the episode and will give a poor performance on these other states, thus slowing down or even prohibiting learning of an optimal policy.

Thank you, your issue is clear for me. So can I use this historical data as an input in model-base rl or I need some additional modeling techniques to estimate stansition?

you could try your model with your historical data, at the least it would give it a good 'initialisation'. you could then update your model once you have some new data that the agent collects.

Could you advise me some good library or tutorial for coding purpose in model-free Rl?

if you want to learn model-free RL then I would look for some medium articles on DQN, they are usually a good resource with well explained code.

Sorry, I mean model-based. It is my mistake

I'm afraid not, I have never worked with model-based methods before. You could look at some papers and see if the authors have a public GitHub repo.

cs285 has a comprehensive lecture on model-based RL @user46045. lecture video. MBRL notes, Practice code. I've added these a bit late, but they might help someone else later.


text:
Are there neural networks where nodes are randomly selected from among a set of nodes (in random orders and a random number of times)?
I am trying to make a classifier.
I am new to AI (even if I know the definition and all such a bit) , and also I have no idea of how to implement it properly by myself even if I know a bit of Python coding (in fact, I am fifteen years old !), but my passion for this has made me ask this (silly, probably) question.
Are there neural networks where nodes are randomly selected from among a set of nodes (in random orders and a random number of times)? I know this is from ML (or maybe deep learning, I suppose), but I have no idea how to recognize such a thing from the presently available algorithms. It will be great if you all could help me, because I am preparing to release an API for programming a model which I call the 'Insane Mind' on GitHub, and I want some help to know if my effort was fruitless.
And for reference, here's the code :
from math import *
from random import *

class MachineError(Exception):
    '''standard exception in the API'''
    def __init__(self, stmt):
        self.stmt = stmt
def sig(x):
    '''Sigmoid function'''
    return (exp(x) + 1)/exp(x)

class Graviton:
    def __init__(self, weight, marker):
        '''Basic unit in 'Insane Mind' algorithm
           -------------------------------------
           Graviton simply refers to a node in the algorithm.
           I call it graviton because of the fact that it applies a weight
           on the input to transform it, besides using the logistic function '''
        self.weight = weight # Weight factor of the graviton
        self.marker = marker # Marker to help in sorting
        self.input = 0 # Input to the graviton
        self.output = 0 # Output of the graviton
        self.derivative = 0 # Derivative of the output

    def process(self, input_to_machine):
        '''processes the input (a bit of this is copied from the backprop algorithm'''
        self.input = input_to_machine
        self.output = (sig(self.weight * self.input) - 1)/(self.marker + 1)
        self.derivative = (sig(self.input * self.weight) - 1) * self.input *self.output * (1- self.output)
        return self.output

    def get_derivative_at_input(self):
        '''returns the derivative of the output'''
        return self.derivative

    def correct_self(self, learning_rate, error):
        '''edits the weight'''
        self.weight += -1 * error * learning_rate * self.get_derivative_at_input() * self.weight

class Insane_Mind:

    def __init__(self, number_of_nodes):
        '''initialiser for Insane_Mind class.
           arguments : number_of_nodes : the number of nodes you want in the model'''
        self.system = [Graviton(random(),i) for i in range(number_of_nodes)] # the actual system
        self.system_size = number_of_nodes # number of nodes , or 'system size'

    def  output_sys(self, input_to_sys):
        '''system output'''
        self.output = input_to_sys
        for i in range(self.system_size):
            self.output = self.system[randint(0,self.system_size - 1 )].process(self.output)
        return self.output

    def train(self, learning_rate, wanted):
        '''trains the system'''
        self.cloned = [] # an array to keep the sorted elements during the sorting process below
        order = [] # the array to make out the order of arranging the nodes
        temp = {} # a temporary dictionary to pick the nodes from
        for graviton in self.system:
            temp.update({str(graviton.derivative): graviton.marker})
        order = sorted(temp)
        i = 0
        error = wanted - self.output
        for value in order:
            self.cloned.append(self.system[temp[value]])
            self.cloned[i].correct_self(learning_rate, error)
            error *= self.cloned[i].derivative
            i += 1
        self.system = self.cloned

Sorry for not using that MachineError exception anywhere in my code (I will use it when I am able to deploy this API).
To tell more about this algorithm, this gives randomized outputs (as if guessing). The number of guesses vary from 1 (for a system with one node), 2 (for two nodes) and so on to an infinite number of guesses for an infinite number of nodes.
Also, I wanna try and find how much it can be of use (if this is something that has never been discovered, if it is something that can find a good place in the world of ML or Deep Learning) and where it can be used.
Thanks in advance.
Criticisms (with a clear reason) are also accepted.

It is difficult to prove a negative, but I do not think there is any classifier (neural network or otherwise) that fully matches to your idea.
I suspect that you will not be able to take the idea of random connections and loops at run time, and make a useful classifier out of it. That's not to say the idea is completely without merit, sometimes it is good to explore blue sky ideas and just see what happens. However, I think it might be a frustrating excercise to build anything on top of your idea without some basic foundation work first. I recommend that you look into the theory and implementation of logistic regression as a starting point, which is a good stepping stone to understanding neural networks.
There are some neural network components and architectures that make use of random behaviour at the activation level:

Dropout. This is a method used during training which zeroes outputs from randomly selected neurons. It often gives an effective boost to neural network stability (acting to prevent overfitting to input data) and can improve accuracy of classifiers too due to behaving similarly to having multiple simpler classifiers.

Boltzmann machines, and restricted Boltzmann machines (RBMs) output 0 or 1 randomly from each "neuron" unit, with the probability decided by sum of inputs. They are used to create generative models, not classifiers though. Another difference is that the randomness is applied both during training and during inference, whilst dropout is most often applied to augment training. Early on in the days of deep learning, RBMs were used to pre-train layers in a deep neural network. This was effective, but other simpler methods were discovered later and are nowadays preferred in most cases.

A variant of dropout call called Monte Carlo dropout is used at inference time. This can be used to measure uncertainty in a model's individual predictions, which is otherwise hard to obtain.

Although not quite as freeform as your random connections on a per neuron basis. If you applied dropout to a recurrent neural network, that might be quite close to your idea, because the existence of loops between neurons in each time step would be random. This could be applied in language modelling and classifiers for sequence data. The same motivations apply here as for dropout in simpler feed forward classifiers - it can in theory make a classifier more robust against noise in the inputs and more accurate.


text:
Isn't evolutionary theory the essence of intelligence after all?
The theory of evolution seems to be intelligent as it creates life
The mechanism of evolutionary theory consists of mutation, recombination, and natural selection like a genetic algorithm.
Isn't this evolutionary mechanism itself the same as the essence of human intelligence?

I think that the question "Isn't this evolutionary mechanism itself the same as the essence of human intelligence?" is a bit ambiguous because you are using the expression "X is the essence of Y". I think it would have been clearer to ask "Do (natural) evolutionary mechanisms generate intelligence?" (actually, this question may not be on-topic, though questions about the definition of intelligence are on-topic here). The answer is clearly "yes", if you consider animals (and, in particular, mammals and humans) intelligent and you assume that evolutionary mechanisms created us.

The other question (actually, an assumption) that you can ask is: "Are EMs intelligent themselves?". Artificial evolutionary algorithms can be used to optimize some function (through search and random operations). If intelligence is the process of optimization, then (at least artificial) evolutionary algorithms are intelligent. I would suggest that you edit your post to ask these more explicit questions, but it's also true that an answer was already given, and I don't know if Neil is willing to address your possilby new questions after the edit.

Please, next time, before trying to ask a question, think more about your questions. Try to be specific and not ambiguous. Moreover, try to ask only one question per post, unless they are very related. Evolutionary algorithms and evolution are clearly a big topic in AI and your question is related to your assumption that "evolution is intelligent", but it suggests that you didn't really know what your question was. Moreover, try to search through the other existing posts/questions. For example, there is already another very similar/related question: https://ai.stackexchange.com/q/28/2444.

The theory of evolution seems to be intelligent as it creates life

When you say "seems to be intelligent" that begs the question: How are you defining "intelligent"? Which of course is still one of the big issues in AI research.
I think there are some flaws with the argument that "creates life" = "intelligent":

Evolution does not create life. It operates on entities where there is a copy mechanism which is not 100% reliable, plus a selective environment that also impacts likelihood of further copies being made. Some form of proto-life (an initial Darwinian ancestor or Ida) needed to exist before evolution started.

The process of creating the first proto-life capable of undergoing evolution is generally thought to be a large semi-random search through chemical combinations. Random search is sometimes used in optimisation problems, and might be studied as part of AI search topics. However, it would normally be considered something of a baseline algorithm, and definitley not tick the boxes for all the general traits of intelligence.

Isn't this evolutionary mechanism itself the same as the essence of human intelligence?

The Wikipedia article on artificial intellegence lists challenges faced by researchers and developers in AI. The categories chosen there are:

Reasoning, problem solving; Knowledge representation; Planning; Learning; Natural language processing; Perception; Motion and manipulation; Social intelligence; General intelligence

Together, these are mainly traits of mammalian, avian and a few other multicellular species, with a few traits such as language heavily focused on humans.
I think it is important to separate out the mechanism whereby these traits arose naturally - which is generally agreed to be via an evolutionary process - from how those traits function. Artifical intelligence may use a little bit of reverse engineering from the natural traits in order to inspire design, but most AI systems do not use theory of evolution directly.
When used directly, evolutionary algorithms can be used to solve search and optimisation problems. Also they can be used to solve simplified problems in perception and motion/manipulation. However, we are not able to scale up such algorithms to solve all aspects of general intelligence. Instead, systems like machine learning are designed to work from analysis of the problem, inspired in part by working natural systems. These work far more efficiently than evolutionary algorithms. There are no competitive evolutionary variants of AlphaZero, Watson, GPT-3 or neural-networks used in image processing.
Evolutionary algorithms have their place in AI in practice and research. However, they do not define or encapsulate a form of general intelligence.

Couldn't all known functions of intelligence be made by evolutionary mechanisms?

@Dimer: They could, and apparently this is what has happened in nature. I try to explain why that's not the same thing as evolutionary algorithms "being intelligent" in the answer. If there is a step missing or unclear reasoning in my answer, I could expand it if you could help me find what's wrong

@Dimer: As an analogy, a person can build a house or car. However, we would not usually say that the person was themselves a dwelling or a type of transport. And that is even the case when we can host other creatures and carry objects by hand in a more limited sense. Architects and engineers who study or build dwellings or cars would consider studying the biology or social history of humans as being outside of their expertise, even though those things clearly led to architecture and design of modern vehicles.


text:
Determining to terminate at a reward or not
I am practicing the Bellman equation on Grid world examples and in this scenario, there are numbered grid squares where the agent can choose to terminate and collect the reward equal to the amount inside the numbered square or they can choose to not do this and just move to the state grid square.
Since this is a determinisitc grid, I have utilised the following Bellman equation:
$$V(s) = max_a(R(s,a)+\gamma V(s'))$$
Where $\gamma=0.5$ and any movement reward is $0$, since this will allow the agent to have a balance of thinking long-term and short-term.
I am trying to understand how you would determine whether it is better for the agent to terminate at the state with the number $3$ or to continue to the state with a number $4$ to collect the more reward? I have determined and marked (X) the terminal states, where with my current calculations, I feel the agent should exit.

Hi Krellex, I have rolled back the image edit as the new image stops looking line an MDP to me. By losing the arrows and the crosses, there is no indication of the actions. By losing the separate reward numbers, there is no indication of reward, only a list of values. In general, there is no need to "fix" parts of the question when you receive an answer that corrects something in it. You can and should make edits that when someone adds a comment asking you to clarify something, However, it should be avoided when there is an answer which refers to that part of your question.

@NeilSlater Ah ok! Thanks for changing it back. I thought my edit might have made it clearer, but you're right, the older one was much clearer. Thanks for the update.

By the way, your diagram happens to show an entirely self-consistent policy and matching state values. That is, all the values are correct given the policy shown by the arrows and crosses. The only thing "wrong" about it is that the policy is not the optimal one that you were looking for - in just one state. Which is what your question was about, so that's fine.

@NeilSlater I have implemented the changes for State 3 so it's optimal. I have also updated state (1,3) to exit at state (2,3) to collect a reward of 2 since this would be the most optimal movement since it returns the highest value from all actions given a discount value of $\gamma=0.5$ I think that would be correct for that state also?

For state at (2,3) with exit value of 2, you have a choice. Both moving up towards (4,3) for an eventual reward of 8 and taking the immediate exit for reward of 2 have a value of 2. In ties for maximum return, then both actions are optimal. I suspect the grid world has been deliberately set up to show that as a discussion point.

@NeilSlater Ah ok! That make sense. Since they're equal, we can choose either one? I have decided to go with the approach shown in the image above for that state since this will give the agent a larger reward.

I am trying to understand how you would determine whether it is better for the agent to terminate at the state with the number 3 or to continue to the state with a number 4 to collect the more reward?

Which is better is determined by looking at the expected return from either choice, with higher expected returns being better.
The return from travelling one time step to the "4 on exit" state, is 2 as you have shown, due to discounting. That is assuming the 4 is gained by taking a separate "exit" action once in that position. That is, there is no combined "move and exit" action that only takes one time step - whether or not such actions exist in this environment makes a large change to your example and what will be optimal, so it is really important to be clear about that.
The return from exiting immediately in the "3 on exit" state is 3.
3 is larger than 2, so if the agent finds itself in the "3 on exit" state, then it should exit immediately to get the best expected return.
Possibly what might be making this harder to understand is the role that discounting takes. Sometimes it appears to be used to "fix" infinite rewards from continuing environments. However, discounting is part of the definition of return, and it changes what counts as optimal. With a low discount factor, such as $0.5$, then it can be optimal to take a lower reward sooner as opposed to a larger reward later. The value of the discount factor allows to make that comparison exactly.

Thanks for the reply! From a numbered square, if the agent wants they can exit to a terminal state X which will give them a reward equal to the number in the square. You are right, there is no combined move and exit. So we can either move or exit. Given this, we would then exit at state 3? The state before state 3 will then become $(3+0.5(3))?$ Where $R = 3, \gamma=0.5 , V' = 3$

@Krellex You already have the value of the state below the "3 on exit" state correct. An agent in that state acting optimally would travel up, gaining 0 immediate reward and arriving in the state "3 on exit" which has a value of 3. So $V^*(s) = 0 + 0.5(3) = 1.5$. We can tell that is optimal in this simple environment as there is no chain of moves that could possibly gain a higher return than 1.5 from that location (in more complex environments you won't be able to see that so clearly)

Thanks so much for this clear explanation, really helped me understand!