text:
Is policy learning and online system identification the same?
In some newer robotics literature, the term system identification is used in a certain meaning. The idea is not to use a fixed model, but to create the model on the fly. So it is equal to a model-free system identification. Perhaps a short remark for all, who doesn't know what the idea is. System identification means, to create a prediction model, better known as a forward numerical simulation. The model takes the input and calculates the outcome. It's not exactly the same like a physics engine, but both are operating with a model in the loop which is generating the output in realtime.
But what is policy learning? Somewhere, I've read that policy learning is equal to online system identification. Is that correct? And if yes, then it doesn't make much sense, because reinforcement learning has the goal to learn a policy. A policy is something which controls the robot. But if the aim is to do system identification, than the policy is equal to the prediction model. Perhaps somebody can lower the confusion about the different terms ...
Example Q-learning is a good example for reinforcement learning. The idea is to construct a q-table and this table controls the robot movements. But, if online-system-identification is equal to policy learning and this is equal to q-learning, then the q-table doesn't contains the servo signals for the robot, but it provides only the prediction of the system. That means, the q-table is equal to a box2d physics engine which can say, what x/y coordinates the robot will have. This kind of interpretation doesn't make much sense. Or does it make sense and the definition of a policy is quite different?

From the book Reinforcement Learning, An Introduction (R. Sutton, A. Barto):

The term system identification is used in adaptive control for what we
  call model-learning (e.g., Goodwin and Sin, 1984; Ljung and S
  ̈oderstrom, 1983; Young, 1984).

Model-learning refers to the act of learning the model (environment). Reinforcement Learning can be divided into two types:

Model-based - first we build a model of an environment and then do the control.
Model-free - we do not try to model the behaviour of the environment.

Policy learning is the act of learning optimal policy. You can do it in two ways:

On-policy learning - learn about the policy $π$ by sampling from the same policy.
Off-policy learning - learn about the policy $π$ from the experience sampled from some other policy (e.g. watching different agent playing a game).

System Identification and policy learning are two completely different aspects of a system.
System Identification is basically finding out the transfer functions, the hardware parameters, the relationships and nature of behavior for different components that determine the results, when acted upon by a control signal. Generally, it is the hardware manufacturers who have all configuration details in their datasheets and they are either used as direct system parameters or used to derive other. Online system identification is the process of determining the set of parameters not with already available measurements but by using data coming through in real time.
Policy learning is the process of correlating the actions to results and discerning what actions are good or bad. Policy learning is about determining the control strategy that shall produce desired results given all the circumstances.
SI is like curve fitting and determine the equation of the curve (already knowing the polynomial degree because you need to know the structure of the parameters you are trying to estimate) on already available data while policy learning is using a closed loop system to repeatedly update your control signals till you find one that satisfies your performance and operational desires.
In robotics context, a robot manipulator is supposed to have the Mass(H), Coriolis(C) and Gravity(G) matrix that define the dynamics of the system, basically relating the physics of the robot to the applied torque on the joints and the tip as shown in the equation below. Online parameter identification would mean using the torque, the known structure of the HCG matrices (H is a n x n matrix, n being the DOF and so on) dynamic equation and the then determine the numerical values. Similar online parameter identification is also done for friction components like the Static and Coulomb friction and the coefficients of Viscous friction. Least squares method is often used for the same.
$$\mathbf{H(q)\ddot{q}}(t) + \mathbf{C(q, \dot{q})\dot{q}}(t) + \mathbf{B\dot{q} + g(q) = \tau}$$
Policy learning in terms fo RL is basically learning the set of actions that will produce desired good behavior. Q-learning is model free learning, so there is no predicted behavior to be obtained based on the inputs. Here, the inputs and simulated, results are obtained, they are given a degree of belief (positive & negative and high & low rewards) depending upon what part and how much of the desired result are they producing. Over time, the policy learned is finally the sequence of actions that should be run to get to desired result. The Q-table does not have anything to do with the system identification which is a modeling step, it is rather a control step. So, for the arm, the learned policy would be what joints in what sequence should be actuated to what angles to complete a pick and place task.


text:
Which field to study to learn & create a.i generated simulations?
I wasn't sure how to title this question so pardon me please.
You may have seen at least one video of those "INSANE A.I created simulation of {X} doing {Y & Z} like the following ones:
A.I learns how to play Mario
A.I swaps faces of {insert celebrity} in this video after 16hrs.
etc...
I want to know what I have to learn to be able to create for example a program that takes xyz-K images of a person as training data and changes it with another person's face in a video.
Or create a program that on a basic level creates a simulation of 2 objects orbiting /attracting each other /colliding like this:

What field/topic is that?
 I suspect deep learning but I'm not sure. I'm currently learning machine learning with Python.
I'm struggling because linear regression & finances /stock value prediction is really not interesting compared to teaching objects in games to do archive something or create a program that tries to read characters from images.

You need to define "simulation" more specific. Playing Mario, Swapping face on image/video, or generating simulation of objects that are orbiting use different techniques.

Playing Mario or "AI that playing game": the AI agent trained on available environment (Mario game, so the environment is not generated) and learn the best sequential actions to achieve the goal. It runs the game thousand times, when it did a wrong action then it gets "penalties" that improve its knowledge. The algorithm that can be used is Reinforcement Learning, but some earlier paper use Genetic Algorithm to generate the best action
Face swap: It's close to computer vision area, some methods that I know use Style Transfer principle (Convolutional Neural Network) to make transformation of face of one image to another image. You can read the basic of style transfer here.
Generating physical movement: I don't know too much about this topic but I know there are some papers talk about this, Fluid Net from Google workers and this paper from TU Munchen. At a glance they also use CNN to improve the result but the main simulation came from Euler Fluid Equation. So if you need to generate object that orbiting, I think you need to find equations that models that movement.

Hope it helps!

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Using AI to guess a mathematical pattern of certain polynomials in four variables: practical challenge
I'd like to use machine learning to guess a mathematical pattern: the input are certain polynomials in four variables $q_1,q_2,q_3,q_4$, the output can be zero or one.
Allowed polynomials are such that (i) all their non-zero coefficients are equal to one, (ii) they do not contain monomials of the form $q_1^j$ for $j \geq 0$, and (iii) if an allowed polynomial contains a monomial $m=q_1^a q_2^b q_3^c q_4^d$ for some non-negative integers $a,b,c,d$, then it also contains $m'=q_1^{a-1} q_2^b q_3^c q_4^d$, provided this does not violate (ii) and $a \geq 1$; similarly for $b \to b-1$, $c \to c-1$, and $d \to d-1$.
Here's an example batch, given by pairs {input, output}: $\{q_2,1\},\{q_3,1\},\{q_4,1\}$
Here's a second batch: $\{q_2+q_1 q_2,0\},\{q_2+q_2^2,0\},\{q_2+q_3,1\},\{q_3+q_1 q_3,0\},\{q_3+q_3^2,0\},\{q_2+q_4,1\},\{q_3+q_4,1\},\{q_4+q_1 q_4,0\},\{q_4+q_4^2,1\}$
I can construct larger and larger batches using Mathematica, and I'd like to know how to practically go from here, to instructing an AI to guess a simple function of $q$'s that reproduces the behavior, namely that can guess the correct output for previously unknown admissible polynomials.
What are the typical batch size and computational power required for such a program to succeed?
My idea is to use a function $\phi$ from the space of allowed polynomials $\mathcal P$ to the set $\mathbb Z_2=\{0,1\}$, of the form $\phi:\mathcal P \to \mathbb Z_2$, $p=\sum_{i \in I} m_i \mapsto \phi(p):= \sum_{i \in I' \subset I} m_i|_1 \mod 2$, where $m_i|_1$ means the $i$-th monomial inside $p$ evaluated at $q_1=q_2=q_3=q_4=1$, and come up with the form of $I'$ as function of $I$.
Notice there's no linear structure on $\mathcal P$.
Remark: of course instead of polynomials one could use punctured solid partitions.

Is there a limit on a,b,c,d? if there is such a limit, we have an upper bound(Number of dimensions = abcd) on types of monomials and hence we can applyit to finite dimensional NN

https://ai.stackexchange.com/questions/2008/how-can-neural-networks-deal-with-varying-input-sizes
Considering each monomial as a word, this can be treated using RNNs or Recursive NNs

@ZaidSyedMMd I agree that one can impose a cutoff and deal with the reduced case of finite a,b,c,d, although the initial problem does not have one. Does your second comment imply that one can deal also with the case of no bound (the real problem I'm interested in)?

Sorry i'm not aware of how RNNs work. "reduced case of finite a,b,c,d, " problem can be easily solved with digital logic network containing comparators(non-linearilty)  and there is no need of "learning" required in the to-be-realised function.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What is "planning" in the context of reinforcement learning, and how is it different from RL and SL?
This is an excerpt taken from Sutton and Barto (pg. 3):

Another key feature of reinforcement learning is that it explicitly considers the whole problem of a goal-directed agent interacting with an uncertain environment. This is in contrast with many approaches that address subproblems without addressing how they might fit into a larger picture. For example, we have mentioned that much of machine learning research is concerned with supervised learning without explicitly specifying how such an ability would finally be useful. Other researchers have developed theories of planning with general goals, but without considering planning's role in real-time decision-making, or the question of where the predictive models necessary for planning would come from. Although these approaches have yielded many useful results, their focus on
isolated subproblems is a significant limitation.

I have an idea of supervised learning (SL), but what exactly does the author mean by planning? And how is the RL approach different from planning and SL?
(Illustration with an example would be nice).

The concept of "planning" is not just related to RL. In general (as the name suggests), planning consists in creating a "plan" which you will use to reach a "goal". The goal depends on the context or problem. For example, in robotics, you can use a "planning algorithm" (e.g. Dijkstra's algorithm) in order to find the path between two points on a map (given e.g. the map as a graph).
In RL, planning usually refers to the use of a model of the environment in order to find a policy that hopefully will help the agent to behave optimally (that is, obtain the highest amount of return or "future cumulative discounted reward"). In RL, the problem (or environment) is usually represented as a Markov Decision Process (MDP). The "model" of the environment (or MDP) refers to the transition probability distribution (and reward function) associated with the MDP. If the transition model (and reward function) is known, you can use an algorithm that exploits it to (directly or indirectly) find a policy. This is the usual meaning of planning in RL. A common planning algorithm in RL is e.g. value iteration (which is a dynamic programming algorithm).

Other researchers have developed theories of planning with general goals, but without considering planning's role in real-time decision- making, or the question of where the predictive models necessary for planning would come from.

Planing is often performed "offline", that is, you "plan" before executing. While you're executing the "plan", you often do not change it. However, often this is not desirable, given that you might need to change the plan because the environment might also have changed. Furthermore, the authors also point out that planning algorithms often have a few limitations: in the case of RL, a "model" of the environment is required to plan.

For example, we have mentioned that much of machine learning research is concerned with supervised learning without explicitly specifying how such an ability would finally be useful.

I think the authors simply want to say that supervised learning is usually used to solve specific problems. The solutions to supervised problems often are not directly applicable to other problems, so this makes them limited.

Another key feature of reinforcement learning is that it explicitly considers the whole problem of a goal-directed agent interacting with an uncertain environment.

In RL, there is the explicit notion of a "goal": there is an agent that interacts with an environment in order to achieve its goal. The goal is often to maximize the "return" (or "future cumulative discounted reward", or, simply, the reward in the long run).

How is RL different from planning and supervised learning?

RL and planning (in RL) are quite related. In RL, the problem is similar to the one in planning (in RL). However, in RL, the transition model and reward function of the MDP (which represents the environment) are usually unknown. Therefore, the only way of finding or estimating an optimal policy that will allow the agent to (near-optimally) behave in this environment is to interact with the environment and gather some info regarding its "dynamics".
RL and supervised learning (SL) are quite different. In SL, there isn't usually the explicit concept of "agent" or "environment" (and their interaction), even though it might be possible to describe supervised learning in that way (see this question). In supervised learning, during the training or learning phase, a set of inputs and the associated expected outputs is often provided. Then the "objective" is to find a map between inputs and outputs, which generalizes to inputs (and corresponding outputs) that have not been observed during the learning phase. In RL, there isn't such a set of inputs and associated expected outputs. In RL, there is just a scalar signal emitted by the environment, at each time step, which roughly indicates how well the agent is currently performing. However, the goal of the agent is not just to obtain rewards, but to behave optimally (in the long run).
In short, in RL, there is the explicit notion of agent, environment and goal, and the reward is the only signal which tells the agent how well it is performing, but the reward does not tell the agent which actions it should take at each time step. In supervised learning, the objective is to find a function that maps inputs to the corresponding outputs, and this function is learned by providing explicit examples of such mappings during the training phase.
There are some RL algorithms (like the temporal-difference ones), which could roughly be thought of as self-supervised learning algorithms, where the agent learns from itself (or from the experience it has gained by interacting with the environment). However, even in these cases, the actions that the agent needs to take are not explicitly taught.

The automated planning is:

Automated planning and scheduling, sometimes denoted as simply AI Planning,1 is a branch of artificial intelligence that concerns the realization of strategies or action sequences, typically for execution by intelligent agents, autonomous robots and unmanned vehicles. Unlike classical control and classification problems, the solutions are complex and must be discovered and optimized in multidimensional space. Planning is also related to decision theory.
In known environments with available models, planning can be done offline. Solutions can be found and evaluated prior to execution. In dynamically unknown environments, the strategy often needs to be revised online. Models and policies must be adapted. Solutions usually resort to iterative trial and error processes commonly seen in artificial intelligence. These include dynamic programming, reinforcement learning and combinatorial optimization. Languages used to describe planning and scheduling are often called action languages.

In other words, the planning is some strategies or actions to reach from the start state to the goal state. As you found in the above, one of the solutions for planning could be RL (depends on the problem). Hence, MDP is a specific case of planning and it is more general.
For the difference of RL and supervised learning you can see this post:

The main difference is to do with how "correct" or optimal results are learned:

In Supervised Learning, the learning model is presented with an input and desired output. It learns by example.
In Reinforcement Learning, the learning agent is presented with an environment and must guess correct output. Whilst it receives feedback on how good its guess was, it is never told the correct output (and in addition the feedback may be delayed). It learns by exploration, or trial and error.

I was actually looking for the difference between planning and RL.

@DuttaA as I wrote: the planning is some strategies or actions to reach from the start state to the goal state. As you found in the above, one of the solutions for planning could be RL (depends on the problem).


text:
What is the difference between First-Visit Monte-Carlo and Every-Visit Monte-Carlo Policy Evaluation?
I came across these 2 algorithms, but I cannot understand the difference between these 2, both in terms of implementation as well as intuitionally.
So, what difference does the second point in both the slides refer to?

The first-visit and the every-visit Monte-Carlo (MC) algorithms are both used to solve the prediction problem (or, also called, "evaluation problem"), that is, the problem of estimating the value function associated with a given (as input to the algorithms) fixed (that is, it does not change during the execution of the algorithm) policy, denoted by $\pi$. In general, even if we are given the policy $\pi$, we are not necessarily able to find the exact corresponding value function, so these two algorithms are used to estimate the value function associated with $\pi$.
Intuitively, we care about the value function associated with $\pi$ because we might want or need to know "how good it is to be in a certain state", if the agent behaves in the environment according to the policy $\pi$.
For simplicity, assume that the value function is the state value function (but it could also be e.g. the state-action value function), denoted by $v_\pi(s)$, where $v_\pi(s)$ is the expected return (or, in other words, expected cumulative future discounted reward), starting from state $s$ (at some time step $t$) and then following (after time step $t$) the given policy $\pi$. Formally, $v_\pi(s) = \mathbb{E}_\pi [ G_t \mid S_t = s ]$, where $G_t = \sum_{k=0}^\infty \gamma^k R_{t+k+1}$ is the return (after time step $t$).
In the case of MC algorithms, $G_t$ is often defined as $\sum_{k=0}^T  R_{t+k+1}$, where $T \in \mathbb{N}^+$ is the last time step of the episode, that is, the sum goes up to the final time step of the episode, $T$. This is because MC algorithms, in this context, often assume that the problem can be naturally split into episodes and each episode proceeds in a discrete number of time steps (from $t=0$ to $t=T$).
As I defined it here, the return, in the case of MC algorithms, is only associated with a single episode (that is, it is the return of one episode). However, in general, the expected return can be different from one episode to the other, but, for simplicity, we will assume that the expected return (of all states) is the same for all episodes.
To recapitulate, the first-visit and every-visit MC (prediction) algorithms are used to estimate $v_\pi(s)$, for all states $s \in \mathcal{S}$. To do that, at every episode, these two algorithms use $\pi$ to behave in the environment, so that to obtain some knowledge of the environment in the form of sequences of states, actions and rewards. This knowledge is then used to estimate $v_\pi(s)$. How is this knowledge used in order to estimate $v_\pi$? Let us have a look at the pseudocode of these two algorithms.

$N(s)$ is a "counter" variable that counts the number of times we visit state $s$ throughout the entire algorithm (i.e. from episode one to $num\_episodes$). $\text{Returns(s)}$ is a list of (undiscounted) returns for state $s$.
I think it is more useful for you to read the pseudocode (which should be easily translatable to actual code) and understand what it does rather than explaining it with words. Anyway, the basic idea (of both algorithms) is to generate trajectories (of states, actions and rewards) at each episode, keep track of the returns (for each state) and number of visits (of each state), and then, at the end of all episodes, average these returns (for all states). This average of returns should be an approximation of the expected return (which is what we wanted to estimate).
The differences of the two algorithms are highlighted in $\color{red}{\text{red}}$. The part "If state $S_t$ is not in the sequence $S_0, S_1, \dots, S_{t-1}$" means that the associated block of code will be executed only if $S_t$ is not part of the sequence of states that were visited (in the episode sequence generated with $\pi$) before the time step $t$. In other words, that block of code will be executed only if it is the first time we encounter $S_t$ in the sequence of states, action and rewards: $S_0, A_0, R_1, S_1, A_1, R_2 \ldots, S_{T-1}, A_{T-1}, R_T$ (which can be collectively be called "episode sequence"), with respect to the time step and not the way the episode sequence is processed.  Note that a certain state $s$ might appear more than once in $S_0, A_0, R_1, S_1, A_1, R_2 \ldots, S_{T-1}, A_{T-1}, R_T$: for example, $S_3 = s$ and $S_5 = s$.
Do not get confused by the fact that, within each episode, we proceed from the time step $T-1$ to time step $t = 0$, that is, we process the "episode sequence" backwards. We are doing that only to more conveniently compute the returns (given that the returns are iteratively computed as follows $G \leftarrow G + R_{t+1}$).
So, intuitively, in the first-visit MC, we only update the $\text{Returns}(S_t)$ (that is, the list of returns for state $S_t$, that is, the state of the episode at time step $t$) the first time we encounter $S_t$ in that same episode (or trajectory). In the every-visit MC, we update the list of returns for the state $S_t$ every time we encounter $S_t$ in that same episode.
For more info regarding these two algorithms (for example, the convergence properties), have a look at section 5.1 (on page 92) of the book "Reinforcement Learning: An Introduction" (2nd edition), by Andrew Barto and Richard S. Sutton.

May I ask, where did you get the two pseudocodes, you posted above, since they seen to be not from the Barto & Sutton book, you cited before? I find your pseudocode clearer than the pseudocode in the cited book

@Dave I wrote this answer a long time ago. I think I had written them myself in latex based on my understanding of these algorithms. However, it's been a while since I touched these specific algorithms. I'd even need to read my answer again :D Let me know if something is unclear and I can try to clarify that later.

For anyone coming across this question and wants a very intuitive understanding of first and every visit monte-carlo, look at the answer given in the link provided here.
https://amp.reddit.com/r/reinforcementlearning/comments/9zkdjb/d_help_need_in_understanding_monte_carlo_first/
After looking at that intuition, then you can come back and look at nbroz answer provided above.
Hope this helps anyone struggling with this idea

Hi. Rather than just providing a link to an external source, can you formulate in your own words that intuition here? Please, edit your post to do so. Btw, have a look at https://ai.stackexchange.com/help/on-topic.


text:
Is reinforcement learning possible for chatbots?
The reinforcement learning paradigm has the aim to determine the optimal actions for a robot. A typical example is a maze finding robot, but reinforcement learning can also be used for training a robot to play the pong game. The principle is based on a reward function. If the robot is able to solve a problem, he gets a score from the underlying game engine. The score can be positive, if the robot reaches the end of a maze, or it can be negative, if he is colliding with an obstacle. The principle itself is working quite well, that means for simpler applications it is possible to train a robot to play a game with reinforcement learning.
Chatbots are a different category of artificial intelligence. They are working not with actions but with natural language. Person #1 is opening a dialogue with “Hi, I'm Alice”, while person #2 is responding with “Nice to meet you”. What is missing here is an underlying game which is played. There is no reward available for printing out a certain sentence. In some literature the problem of language grounding was discussed seriously, but with an unclear result. It seems, that a classical game for example pong, and a chatbot conversation doesn't have much in common.
Is it possible to combine Reinforcement Learning with chatbot design? The problem is, that a speech-act should be connected to a reward. That means, a well formulated sentence gets +10 points but a weak sentence gets -10 points. How can this be evaluated?

Yep, this isnt a direct answer but you should look through these publications. https://www.microsoft.com/en-us/research/lab/microsoft-research-montreal/#!publications

The problem is indeed that the 'rules' of conversations are not as fixed as the rules for games. However, you can make use of  descriptive formalism from Discourse Analysis, called adjacency pairs. These describe regularities between utterances on a local level, for example greeting/reply, which would match your "Hi, I'm Alice" and "Nice to meet you".
You will need to be able to classify utterances by your chat bot according to a set of possible responses, and then you can see if a valid response is produced for any given utterance. If the user asks a question, then a greeting will not be a good answer, but a statement could be, if it was a response to the question. This is leaving aside the content and focuses merely on the formal characteristics of the utterance.
If you want to know more about the topic, have a look at Conversation Analysis, which is the linguistic field dealing with the subject.

This makes me think it might be worthwhile to formally explore games with fluid rulesets.  (Games, in general, evolve over time, but I wonder if there are any combinatorial games in which rules emerge from gameplay.)


text:
Why is MSE used over other quadratic loss functions?
So I was wondering, why I have only encountered square loss function also known as MSE. The only nice property of MSE I am so far aware of is its convex nature. But then all equations of the form $x^{2n}$ where $n$ is an integer belongs to the same family.
My question is what makes MSE the most suitable candidate among this entire family of curves? Why do other curves in the family, even though having steeper slopes, $(x >1) $, which might result in better optimisation, not used?
Here is a picture to what I mean where red is $x^4$ and green is $x^2$:

I think this question might help you: https://datascience.stackexchange.com/q/10188/10640.

I can comment on several properties of MSE and related losses.
As you mentioned MSE (aka $l_2$-loss) is convex which is a great property in optimization in which one can find a single global optimum. MSE is used in linear and non-linear least squares problems which form the basis of many widely used statistical methods. I would imagine the math and implementation would be more difficult if one would use a higher-order loss (e.g. $x^3$) and that would also prove to be futile because MSE already possesses great statistical and optimization properties on its own.
Another important aspect, one wouldn't use higher-order loss functions in regression is because it would be extremely prone to outliers. MSE on its own would weigh the outliers much more than l1-loss would! And in real world data there is always noise and outliers present. In comparison l1 loss is more difficult in optimization, one reason for which is it's not differentiable at zero.
Other interesting losses you might want to read about are $l_0$ and $l_{inf}$ loss, all of which have their own trade-offs in optimization-sense.

There is another variant for MSE. You can employ the absolute value of the difference of your hypothesis and the expected output. MSE and the absolute difference version, each have a property. The interpretation of the MSE is that you have the area of squares which at first are large but after training the predicted outputs and real outputs get similar to each other which means the area of squares has been diminished after training. For absolute value, the interpretation is that the difference you are going to reduce is a length and after training, you diminish the length of the errors. I don't say what you've said is not possible. It is possible and it does have an interpretation, maybe to diminish the volume of hypercubes, but the point is that the differentiation of such cost functions are a bit complex and it may lead to update rules which are longer to be calculated.

Comments are not for extended discussion; this conversation has been moved to chat.


text:
What does 'acting greedily' mean?
I wanted to clarify the term 'acting greedily'. What does it mean? Does it correspond to the immediate reward, future reward or both combined? I want to know the actions that will be taken in 2 cases:

$v_\pi(s)$ is known and $R_s$ is also known (only).
$q_{\pi}(s, a)$ is known and $R_s^a$ is also known (only).

Note: Since I am a beginner the question might look incoherent due to incomplete understanding. You can point it out and if i think thats the case i'll correct it.

In RL, the phrase "acting greedily" is usually short for "acting greedily with respect to the value function". Greedy local optimisation turns up in other contexts, and it is common to specify what metric is being maximised or minimised. The value function is most often the discounted sum of expected future reward, and also the metric used when defining a policy as "acting greedily".
It is possible to define a policy as acting greedily with respect to immediate expected reward, but not the norm. As a special case, when the discount factor $\gamma = 0$ then it is the same as acting greedily with respect to the value function.
When rewards are sparse (a common situation in RL), acting greedily with respect to expected immediate reward is not very useful. There is not enough information to make a decision.

I want to know the actions that will be taken in 2 cases:

$v_\pi(s)$ is known and $R_s$ is also known (only).

To act greedily in RL, you would use the value function $v_\pi(s')$ - the value function of the next states. To do so, you need to know the environment dynamics - to go with the notation $R_s$ you should also know the transition matrix $P_{ss'}^a$ - the probability of transitioning from $s$ to $s'$ whilst taking action $a$:
$$\pi'(s) = \text{argmax}_a \sum_{s'} P_{ss'}^a(R_{s} + \gamma v_{\pi}(s'))$$
Notes:

This assumes $R_{s}$ is your immediate reward for leaving state $s$. Substitute $R_{s'}$ if the reward matrix is for entering state $s'$ instead.

The policy gained from acting greedily with respect to $v_\pi(s)$ is not $\pi$, it is (a usually improved) policy $\pi'$

$q_{\pi}(s, a)$ is known and $R_s^a$ is also known (only).

To act greedily in RL, you would use the value function $q_{\pi}(s, a)$, and this is much simpler:
$$\pi'(s) = \text{argmax}_a q_{\pi}(s, a)$$
Again, the policy gained from acting greedily with respect to $q_\pi(s,a)$ is not $\pi$, it is (a usually improved) policy $\pi'$

Shouldn't $R_s$ be outside the brackets (not multiplied with probability) in case the reward is for leaving the state while inside the bracket for ($R_s'$)

@DuttA: Yes it can be outside the brackets (and the whole sum) for efficiency, but actually it does not matter to the value. I think I should leave it in for this case, so I don't have extra complication of two formulae.

In general, a greedy "action" is an action that would lead to an immediate "benefit". For example, the Dijkstra's algorithm can be considered a greedy algorithm because at every step it selects the node with the smallest "estimate" to the initial (or starting) node. In reinforcement learning, a greedy action often refers to an action that would lead to the immediate highest reward (disregarding possible future rewards). However,  a greedy action can also mean the action that would lead to the highest possible return (that is, the greedy action can also be considered an action that takes into account not just immediate rewards but also future ones).
In your case, I think that the "greedy action" can mean different things, depending on weather you use the reward function or the value functions, that is, you can act greedily with respect to the reward function or the value functions.
I would like to note that you are using a different notation for the reward function for each of the two value functions, but this does not need to be the case. So, your reward function might be expressed as $R_s^a$ even if you use $v_\pi(s)$. I will use the notation $R_s^a$ for simplicity of the explanations.
So, if you have access to the reward function for a given state and action, $R^a_s = r(s, a)$, then the greedy action (with respect to the reward function $r$) would just be the action from state $s$ with the highest reward. So, formally, we can define it as $a_\text{greedy} = \arg \max_a r(s, a)$ (both in the case of the state or state-action value functions: it does not matter if you have one or the other value function). In other words, if you have access to the reward function (in that form), you can act greedily from any state without needing to access the value functions: you have a "model" of the rewards that you will obtain.
If you have $q_\pi(s, a)$ (that is, the state-action value function for a fixed policy $\pi$), then, at time step $t$, the greedy action (with respect to $q_\pi(s, a)$) from state $s$ is $a_\text{greedy} = \arg \max_{a}q_\pi(s, a)$. If you then take action $a_\text{greedy}$ in the environment, you would obtain the highest discounted future reward (that is, the return), according the $q_\pi(s, a)$, which might actually not be the highest possible return from $s$, because $q_\pi(s, a)$ might not be the optimal state-action value function. If $q_\pi(s, a) = q_{\pi^*}(s, a)$ (that is, if you have the optimal state-action value function), then, if you execute $a_\text{greedy}$ in the environment, you will theoretically obtain the highest possible return from $s$.
If you had the optimal value function (the value function associated with the optimal policy to act in your environment), then the following equation holds $v_*(s) = \max_{a} q_{\pi^*}(s, a)$. So, in that case, $a_\text{greedy} = \arg \max_{a}q_{\pi^*}(s, a)$ would also be the greedy action if you had $v_*(s)$. If you only have $v_\pi(s)$ (without e.g. the Q function), I don't think you can act greedily (that is, there is no way of knowing which action is the greedy action from $s$ by just having the value of state $s$: this is actually why we often estimate the Q functions for "control", i.e. acting in the environment).

Acting greedily means that the search is not forward thinking and limits its decisions solely on immediate return. It is not quite the same as what is meant in human social contexts in that greed in that context can involve forward thinking strategies that sacrifice short term losses for long term gain. In the typical machine search lingo, greed is myopic (short-sighted).


text:
A few questions regarding the difference between policy iteration and value iteration
The question already has some answer. But I am still finding it quite unclear (also does $\pi(s)$ here mean $q(s,a)$ ?):

The few things I do not understand are:

Why the difference between 2 iterations if we are acting greedily in each of them? As per many sources 'Value Iteration' does not have an explicit policy, but here we can see the policy is to act greedily to current $v(s)$
What exactly does Policy Improvement mean? Are we acting greedily only at a particular state at a particular iteration OR once we act greedily on a particular state we keep on acting greedily on that state and other states are added iteratively until in all states we act greedily?
We can intuitively understand that acting greedily w.r.t $v(s)$ will lead to $v^*(s)$ eventually, but does using Policy Iteration eventually lead to $v^*(s)$?

NOTE: I have been thinking of all the algorithms in context of Gridworld, but if you think there is a better example to illustrate the difference you are welcome.

$\pi(s)$ does not mean $q(s,a)$ here. $\pi(s)$ is a policy that represents probability distribution over action space for a specific state. $q(s,a)$ is a state-action pair value function that tells us how much reward do we expect to get by taking action $a$ in state $s$ onwards.
For the value iteration on the right side with this update formula:
$v(s) \leftarrow \max_\limits{a} \sum_\limits{s'}p(s'\mid s, a)[r(s, a, s') + \gamma v(s')]$
we have an implicit greedy deterministic policy that updates value of state $s$ based on the greedy action that gives us the biggest expected return. When the value iteration converges to its values based on greedy behaviour after $n$ iterations we can get the explicit optimal policy with:
$\pi(s) = \arg \max_\limits{a} \sum_\limits{s'} p(s'\mid s, a)[r(s, a, s') + \gamma v(s')]$
here we are basically saying that the action that has highest expected reward for state $s$ will have probability of 1, and all other actions in action space will have probability of 0
For the policy evaluation on the left side with this update formula:
$v(s) \leftarrow \sum_\limits{s'}p(s'\mid s, \pi(s))[r(s, \pi(s), s') + \gamma v(s')]$
we have an explicit policy $\pi$ that is not greedy in general case in the beginning. That policy is usually randomly initialized so the actions that it takes will not be greedy, it means we can start with policy that takes some pretty bad actions. It also does not need to be deterministic but I guess in this case it is. Here we are updating value of state $s$ according to the current policy $\pi$.
After policy evaluation step ran for $n$ iterations we start with the policy improvement step:
$\pi(s) = \arg \max_\limits{a} \sum_\limits{s'} p(s'\mid s, a)[r(s, a, s') + \gamma v(s')]$
here we are greedily updating our policy based on the values of states that we got through policy evaluation step. It is guaranteed that our policy will improve but it is not guaranteed that our policy will be optimal after only one policy improvement step. After improvement step we do the evaluation step for new improved policy and after that we again do the improvement step and so on until we converge to the optimal policy


text:
What characteristics make it difficult for a Neural Network to approximate a function?
What are the characteristics which make a function difficult for the Neural Network to approximate?
Intuitively, one might think uneven functions might be difficult to approximate, but uneven functions just contain some high frequency terms (which in case of sigmoid is easy to approximate $\frac{1} {(1 + e ^ {-(w*x + b)})}$, by increasing the value of $w$). So uneven data might not be diffcult to approximate.
So my question is what makes a function truly difficult for approximation?
NOTE: By approximation I do not mean things which can be changed by changing the  training method (changing training set size, methods, optimisers). By approximation I mean things which require hyperparameters (size, structure, etc) of a NN to be changed to approximate to a certain level significantly easily.

What do you mean by "difficult"? Do you mean the time required to approximate? How do you count time? I think that the performance of a NN at approximating a function also depends on the available training data, the training algorithm, etc. So, I think you should clarify what you mean by "difficult". After having quickly looked at the literature, people have been trying to study the approximation properties of NNs also when e.g. weights are fixed but letting the number of neurons vary.

@nbro I was also thinking the same thing.. Difficult to train vs difficult to approximate...I'll edit the question after some time.

The paper Deep Neural Network Approximation Theory might be useful, even though it is highly mathematical, and so it might be not trivial to follow.

Possible duplicate of Which functions can't neural networks learn efficiently?


text:
How to back-propagate illegal actions for policy gradient learning
When training a AI RL agent to play a game there'll be situations where the AI cannot perform certain actions lest they violate the game rules. That's easy to handle, and I can set illegal actions to some large negative amount so when doing an argmax they won't be selected.  Or if I use softmax I can set probabilities of illegal actions to zero and then re-calculate softmax on the remaining legal states. Indeed, I believe this is what David Silver was referring to when asked this question at a presentation/lecture of AlphaZero:
https://www.youtube.com/watch?v=Wujy7OzvdJk&t=2404s
But doing so changes the output from the network and surely changes things when performing the backprop once a reward is known.
How does one handle that?
Would I set the illegal actions to the mean of the legal actions, or zero...?

possible duplicate

Specifically, this has been thoroughly discussed in the comment chain you participated in under this answer.

Please read my question carefully. This is not a duplicate and I've not forgotten about discussions I've participated in. I'm asking specifically about how the modified actions should be handled with the backprop process. i.e. not how to remove illegal actions, but what to set the illegal action values to when they're fed back in for backprop - hence the title "How to back-propagate illegal actions".

In fact, my question relates to brianberns (unanswered) comment (on Mar 22 '18 at 14:26) on the thread @Brale_ flagged this as a duplicate:
"How does one compute the gradient of the filtered version of Softmax? Seems like this would be necessary for backpropagation to work successfuly, yes?"

in any case you would backprop only through the action that you chose, so do the same thing that you do with regular version of the algorithm that doesn't have illegal actions when you make the update. And when choosing the actions disregard the illegal actions

Sorry for misunderstanding. To expand upon @Brale_'s comment, because illegal actions are never taken, the agent never receives any feedback on them in the form of reward (or lack of reward). So it isn't even possible to backprop through illegal actions.

If doing A3C, I believe the entire action output from the policy is fed back to compute policy gradients, not just the single selected action, as would be the case for Q-based learning.  Sorry, I should have made clear in the question this was for policy gradient.

method shouldn't matter though, even if its policy gradient you still backprop only one action. Are you using someone elses implementation or did you make your own? If NN outputs softmax probabilities you would take them, rescale them based on if there are some illegal actions and then sample from them. For the backprop I think standard approach is to apply one hot vector on the original NN output and then use that for calculating gradients for backprop.

Okay, well that makes things easier. I'm using a normalized output layer, but I guess the same principle would apply and I just negate the illegal actions and select from the legal ones.
But can I ask, why would you need to rescale the softmax outputs after setting illegal moves to probability zero?  If we just do an argmax/one-hot of the remaining legal actions anyway, then it'll be the same selected action after resampling as before, so why bother resampling with only legal actions...?

beacuse you need to explore too and if you always use argmax you will always pick greedy action. If you set some illegal actions to 0 other probabilities won't sum up to 1 so thats why you rescale and then sample from that. And after you sample an action you apply one hot vector so that it masks all actions except the one that was picked. I'm not exactly sure about details so try to check some implementations on github regarding that.

I see, so you'd do this to select an action
probs = tf.nn.softmax(logits)
action = np.random.choice(self.action_size, p=probs.numpy()[0])
and then apply one-hot on chosen action for the backpropogation. Perfect, thanks for your help.


text:
Is there a RNN that can predict the next substitute in a floorball match?
Floorball is a type of floor hockey. During the game, substitutions can be made.

The team is also allowed to change players any time in the game; usually, they change the whole team. Individual substitution happens sometimes, but it usually happens when a player is exhausted or is hurt.

I would like to use an RNN to predict when the next substitution will happen for a team. However, I have no pre-existing dataset to train on. Is there a way that I can start predicting without a dataset and continually improve accuracy as more games are played?

The two questions you ask in the title seem to be independent questions. They should probably be asked in separate posts. In the post about floorball, you should give or link to a description of the game. Welcome to AI.se!

@PhilipRaeisghasem Thanks for the feedback and welcoming me! I tried to narrow it down and removing multiple questions.

I think you might have some misconceptions about what training entails. "A dataset" would be a set of inputs that a neural network could use to optimize its weights in order to do well at a certain task. This could be information about many past games and when substitutions happened in them. Do you have such a dataset? Or are you asking if you can do prediction while gathering the data?

@PhilipRaeisghasem The first match being played there exist no dataset. I want it to continually update the dataset every time a substitute is made. After the dataset is updated I also want it to predict the next substitute. When the match is finished the dataset is saved and used in the next match. Then it grows every match and gets more and more accurate. Thank you very much in advance!
TL DR
"Do you have such a dataset?" Not at the beginning!
"Or are you asking if you can do prediction while gathering the data?" yes! :)


text:
In tic-tac-toe, what is the effect of the starting state on the state and action value function?
I am simulating a Tic-Tac-Toe game with a human opponent. The way the RL trains is through policy/value iterations for a fixed number of iterations all specified by the user. Now, whether the human player has turn 1 or turn 2 will decide the starting state (1st move by human or empty). The starting states for the 1st case can differ as the human can make 9 different moves.
So, my questions are:

In tic-tac-toe, what is the effect of the starting state on the state and action value function?
Does it converge to the same stable value for all starting states?
Will the value functions change if the starting players are changed? (human vs RL to RL vs human)

NOTE: I will be enumerating all states since there are approximately 20000 states which I believe is not a big number and thus convergence should not be a problem.

I would like to note that there's a way to play tic tac toe so that none of the players ever loses and so it's always a tie. I think this holds no matter which action is initially taken (by the player that first plays).

@nbro the position of first move is the starting state according to my terminology.

Value function defined for all states (of the MDP), and that include of cause all starting states. Converge or nor is dependent on the method. Table-based Q-learning converge for Tic Tac Toe (for any competing policy). Value function defined on states, actions, transitions and rewards. It doesn't depend on quality of the agent. I suggest you read any textbook on Markov Decision Process (MDP) and Reinforcement Learning. Everything explained in depth in textbooks, stackexchange is not suitable format for those explanations.

@mirror2image I didn't ask for quality of agents, I didn't ask for methods of convergence and above all the site is especially for this stuff. I don't know what do you mean by in-depth explanation.

Could you clarify what you mean by "starting state" here? Without further information, if someone asked me about Tic Tac Toe states, I would assume that there is a single starting state: An empty grid. However, the RL view will depend if you are asking about designing a self-play learning algorithm, or looking at the game from the perspective of a single agent where the opponent's behaviour is part of the environment (and the opponent is not another learning agent).

@NeilSlater starting state to me is the first move. For an RL opponents first move will be it's starting state if human player has turn 1. Also I am using the entire model that is explicitly creating all states and assigning values like policy/value iteration subject to the fact the opponent makes a random move for every RL move.

Thanks. I think the question could do with some of those details from your comment edited in, as there is not a generic RL mode for Tic Tac Toe, and choices such as self-play and model of the opponent make a large difference in practice.

@NeilSlater thanks I'll edit it in...Also can you recommend some sources for RL codes which can give me an idea of the style used in writing a proper RL algorithm. I'm not looking for complex implementations with tensorflow but more of simple implementation with numpy.

I've written a self-play TicTacToe using basic Python and no neural networks etc here: https://github.com/neilslater/game_playing_scripts/blob/master/tictactoe_q.py - however, that doesn't do quite what you want in the question, mine is a self-play learner using minimax variant of Q learning. Also I use after states rather than state/action split. It might still help though?

@NeilSlater thanks i'll check it out. I am just looking for state of art  implementation style not entire implementations so I think it will be fine.

My understanding - from comments on the question - is that you are looking to train a Reinforcement Learning agent on the game of Tic Tac Toe (perhaps just in theory), where the agent should learn to play against a "human" opponent. In practice you may want a model of a human opponent.
In this case, the RL agent will be presented with a board state, it will take an action (to put its mark on an empty place in the grid) and either:

Win immediately, receiving a positive reward. It is common to use +1 in a game like Tic Tac Toe, so I will assume that later.

Lose on opponent's turn, receiving a negative reward (assumed -1 later in the answer), as opponent makes a move that causes it to win. This is effectively "immediately" in terms of time steps, the agent does not get to act afterwards.

Receive zero reward, and a new board state that includes the opponent's move

Receive zero reward, and the game ends in a draw

In all cases, the opponent is considered part of the environment. That makes the opponent behaviour critical to the value function and choice of optimal play. Training versus different opponents can result in very different state values.
For training to be stable, the opponent should behave with the same probability of action choices for each interim state that it observes. That includes it behaving deterministically, even optimally, purely randomly or anything in-between provided the probability distribution is fixed.
With the above context, it is possible to give sound answers to your questions:

In tic-tac-toe, what is the effect of the starting state on the state and action value function?

Each state of the board, or each state/action pair if you want to track action values, should converge to a value, depending on the agent's estimated, expected result at the end of the episode. As there is only a single reward possible at the end of each game, this will vary between -1 and +1. If either the agent or the opponent can make mistakes at random, then non-zero values between -1 and +1 are possible.

Does it converge to the same stable value for all starting states?

That depends on behaviour of the opponent. In the scenario you are working with, the agent may not learn to play optimally in general, instead it will learn to gain optimal results against the supplied opponent. If the opponent can make mistakes, then moves which take advantage of that fact will have higher values.
Without a detailed description of the opponent it is not possible to make many statements about the actual state values and action values.
Against a perfect opponent, with the RL going second, it should converge to state values which are all zero and action values which are all zero or -1 for moves which would be mistakes.
Against a completely random opponent, I would have to run it to be sure, but I would expect state values to have 3 different values, all slightly positive, depending if opponent chose middle, edge or corner cases - each of these would have slightly different chance of leading to a win for the agent going forward.

Will the value functions change if the starting players are changed?

Due to the turn-based nature of the game, all the states observed would be different depending on who was the first player. When the agent goes first it will get to score the empty grid and action values of any position it would like to make a first mark in - it gets to see state after turn 0 on time step 1, after turn 2 on time step 2, after turn 4 on time step 3 etc. When the agent goes second it will get to see and evaluate the outpt of other turns - turn 1 at t=1, turn 3 at t=2, turn 5 at t=3 etc.
That means the sets of states and state/action pairs for each case (RL first or RL second) are disjoint, and you cannot ask if one agent has the same value for a specific state as the other agent - it simply won't know about the other agent's values.
If you train a single agent, sometimes starting first, sometimes starting second, the two sets of values never interact with each other directly - in an enumerated table, as per the question, this is not at all, but if an agent uses function approximation such as neural networks, then they can affect each other.

It seems to me that your last statement is not true in general. Even in turn-based games like this, if you use function approximation, then the two sets of values would interact. Like in AlphaGoZero

@PhilipRaeisghasem: You are right, my statement only applies to enumerated states - that does apply to OP, but I will add a caveat. Also in Go I think it is possible for the board to be identical on a black or white player's move due to pieces being removed from the board


text:
how to work with multi-labels or two inputs and a output
I’m in this problem and haven’t found a sound solution to it. Been like 20 days now. I have a dataset that looks like this:
X=image

Y1= current_zoom (0,25,50,75)

Y2= predicted_zoom (0,25,50,75)

y1 will have equal images for all classes. Also, I will know X and y1 when I test the model.
y2 will have variation with it because it has the predicted zoom level.
I tried to train on MTL model, I used two outputs from the model - y1 and y2. Now, y2 overfits (still not sure why, but my best guess is class imbalance). And y1 accuracy is around 0.99 in validation. Now the thing is when I deploy this on production, I’ll always have the current zoom (y1) with the image. So, I wanted to incorporate this to my model. First, I tried with two inputs and one output model, but loss was too much. Here, I concatenated the y1 to the output of last conv layer before it flattens and goes to dense. And second I tried was to concatenate y1 after flattening the output from last conv. Both didn’t work.
Any ideas on how can I with data like this.
Models used: resnet-18, vgg, alexnet
size of data: approx 7000 images in total.

So the task is to output Y2 given X and Y1? And Y1 and Y2 are like bounding boxes?

No, they are just numbers labels (int). each of them have four class.


text:
Consecutive frames can be discarded when training an SSD/YOLO?
Let's say I have a number of videos, and I want to train an SSD/YOLO (or FRCNN) to detect objects. In the case of a large amount of videos,  there will be a lot of frames extracted and transferred to images. Can you take, for example, only the fifth frame everytime and thus lower the amount of memory required? If the frames contain a similar info, we can skip some and not impact the results? This is mainly to train faster..

Theoretically possible, but you have to modify the sorce code at video capture part. I did integrate YOLO with my C++ project and I always skip 5~10 frames per second. But I didn't modify the YOLO source code, because I capture the video frame using OpenCV in my C++ project, not directly from Yolo.

Yeah, I think you are right. I used OpenCV and saved the tenth frame,  not directly from YOLO as you said. Thanks@!


text:
DQN Q-mean values converge negatively
I'm trying to implement my own DQN. So far I think my code is good, but my Q-values (I'm getting the mean of all the values for every episode) tends to converge near-zero but negatively. It is normal? Or there is something wrong in my implementation?
My exploration vs explotation greedy strategy goes from 1.0 to 0.1 in 1 million steps (as DeepMind does), my learning rate is 0.00025 and my gamma 0.99. I read here that
"The mean Q-values should smoothly converge towards a value proportionnal to the mean expected reward."
So, it's my agent expecting a negative reward? If so, how can i fix it?
Here is a graph of the first training session:

You can see how the Q-values tend to converge near-zero after about 1300 episodes (1120000 steps aproximately). Actually it's showing values like -0.0117, -0.0145, etc.
Also, the agent seems very "static" after epsilon gets near 0.1, and when it reaches it doesn't move so much. (I'm training with PongDeterministic-v4)

Could you explain the reward scheme in PongDeterministic-v4? This is really critical to deciding whether DQN is being accurate, and environments can vary wildly in what the true value function is. What you typically want is to approximate the true value function for optimal play. There is no reason to expect that to have a specific value (such as near zero) - the "true" values are first and foremost a consequence of the environment and the policy.

@NeilSlater yes; first, the difference between standar enviroments in OpenAI-gym and Deterministic enviroments are that deterministic gets a fixed frameskip of 4 (as used in DeepMind Paper). the -v4 stands for the probability of repeat the last action (in v4 is 0; so it always follow the predicted action, as DeepMind did). Source here: https://github.com/openai/gym/issues/1280 The reward scheme is just making the ball pass behind his bar to score a point. The first who score 21 times wins. I implemented reward clipping but in Pong it doesn't affect (making the scores between -1 and 1).


text:
Next step after lane detection in vehicle automation
I have an RC car with a camera, I have implemented so that i can detect lanes on my track (think like a nascar track). I want to get this car to be able to go around the track autonomous. But I am quite unsure what my next step should be.
Either I can do an algorithm that detects so that I stay in the middle of the lane (if i get to close to either of the lines I steer towards the center).
Or perhaps go around the track manually and save the coordinates of the detected lines as well as the actions I take (steering) and try a DQN approach.
I'm trying to minimize my 'trial and error' time a little here. Perhaps there are some important steps in between here, or a solution that I have not thought of (i.e. am I missing something)?
I'm doing this as a proof of concept therefore I can only spend maximum of a month on this, so what would you do here?

The point of doing something like RL is not necessarily to make the car steer itself - but to demonstrate that a system you have implemented can learn to steer by itself with minimal supervision, or maybe achieve some difficult to pre-program goal such as travel at maximum possible speed whilst still retaining control. A more simple feedback control loop might suffice if your only goal is have the car follow a track that it can reliably detect. Could you give more detail about your goals in the question?

@NielSlater Yes, I think that RL might not be the best approach. My goals are quite "simple", i want the car to be able to stay in its lane and be able to make turns. So that it can complete the track.


text:
How do I choose an appropriate fitness function and hyper-parameters to train a 7-DOF arm?
I am trying to train an ANN to control a 7 Degrees-Of-Freedom arm. It should reach a target avoiding a single obstacle. Given my modeling of the situation, my input layer is composed of 12 nodes:

5 nodes for the 5 joint states
3 nodes for the cartesian coordinates of the target
3 nodes for the cartesian coordinates of the obstacle
1 node for the radius of the obstacle (it's a spherical object).

I have already tried training the ANN with DQN. I want to try neuroevolution (NEAT, in particular) and see how the results compare. I am using NEAT-python. As seen in this paper, this should be feasible.
However, I am having trouble choosing the best fitness function and also some other hyperparameters, namely the population size. (I am also puzzled by the extremely long training time for a single generation, but that's another story.)
So, the fitness function. I have tried to replicate what I have done with DQN. So, basically, my function evaluates a genome (so, an ANN) as follows (pseudocode):
counter = 0
reapeat for NUM_OF_EPISODES times:
    generate a random target
    generate an obstacle which lies about halfway from the end-effector to the target
    repeat for TIMEOUT times:
        use the ANN to decide the next_action and execute it
        if OBSTACLE_REACHED or TARGET_REACHED, stop
    counter += 0.3 * relative_distance + 0.2 * relative_path + 0.5 * didntHitObstacle()
fitness = counter / NUM_OF_EPISODES

So, humanly speaking, for each ANN we try to execute NUM_OF_EPISODES times (how many times should be enough? 100 times seems ok but it get's really slow) a scenario. In this scenario, we use the ANN to search for the target, and if we reach it or the obstacle, we stop. Now for each one of these scenarios, we should "rank" how well the ANN performed (see the counter += ... part). But how should I do this? My idea (stolen from the paper above) was to compute something like this:
0.3 * relative_distance + 0.2 * relative_path + 0.5 * didntHitObstacle()

So, basically, we see how much we are closer to the target compared to when we started, how "short" the path was (compared to the ideal straight line start point-to-target), and whether we did or did not hit the target.
Does this function make sense? My concern is mainly about how we deal with the obstacle: 50% of the fitness. Is it correct? I am asking this because I am receiving poor results.
Another problem that I have is population size. Of course, the bigger the better, but this thing takes a lot to train. How big is ok, in your experience?

Do you have a link to a Jupyter notebook?  If you aren't using Python, please don't convert just for me (haha).  Was just looking to take a look at the code.  Glancing at the paper I see that they are using a 6 DOF arm.  Adding another degree of freedom probably exponential increases possibilities, so would be slower to train.

I'm sorry, I don't @JakeJ. If you want I can add the fitness function code in the answer.

How about a github repo?  Sure, fitness function would help.

Are you physically running it on the arm for each iteration (I've seen someone who does that)?

@JakeJ I have no repos right now because this stuff is a "fork" of my main bachelor thesis I have done quickly just to see how it works. Maybe after I will try to add something (the code is a bit messy!).
I am not simulating the arm during training. The arm, in fact, is represented as a chain of segments, and I just have to check with a bit of forward kinematics + linear algebra if the obstacle collides with the segments. Sure, a bit lengthy but the DQN code worked fast enough!

Cool.  If it takes too long for your liking to train, there are always ways to run it on a GPU (or portions of it).

@JakeJ for the linear algebra sure, I use numpy so I think that I can easily accelerate that. But for the NEAT itself? I think that depends on how neat-python is written, right?

I'm not sure what NEAT implementations there are that are GPU enabled, but I should start keeping my eye open.  In the meantime, I saw that you can speed up numpy with a BLAS:  https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms
I also saw this:  https://github.com/sean-dougherty/accneat (from the catalogue here:  https://eplex.cs.ucf.edu/neat_software/#NEAT ).
That's all the help I can provide as I've not used NEAT or trained neural nets for this application type.

Let us continue this discussion in chat.


text:
What is the input to AlphaGo's neural network?
I have been reading an article on AlphaGo and one sentence confused me a little bit, because I'm not sure what it exactly means. The article says:

AlphaGo Zero only uses the black and white stones from the Go board as its input, whereas previous versions of AlphaGo included a small number of hand-engineered features.

What exactly is the input to AlphaGo's neural network? What do they mean by "just white and black stones as input"? What kind of information is the neural network using? The position of the stones?

The input to the neural network is a $19 × 19 × 17$ image stack
  comprising $17$ binary feature planes. $8$ feature planes $X_t$ consist of binary values indicating the
  presence of the current player’s stones ($X^i_t = 1$ if intersection $i$ contains a stone of the player’s
  colour at time-step $t$; $0$ if the intersection is empty, contains an opponent stone, or if $t < 0$). A
  further $8$ feature planes, $Y_t$
  , represent the corresponding features for the opponent’s stones. The
  final feature plane, $C$, represents the colour to play, and has a constant value of either $1$ if black
  is to play or $0$ if white is to play. These planes are concatenated together to give input features
  $s_t = [
X_t, Y_t, X_{t−1}, Y_{t−1}, ..., X_{t−7}, Y_{t−7}, C]$.

This and all the other architecture details can be found in the "Neural Network Architecture" section in the paper.

Whether should the input be normalized? otherwise the weights with the input value zero can not be updated. I also notice the toy DL tutorial example has normalized mnist data (contains one and zero) with mean and std.


text:
Possible inconsistency in the Policy Improvement equation
I came across this formula in Sutton And Barto: RL an Intro (2nd Edition) equation number 4.7 (page number 78).
If $\pi$ and $\pi'$ are deterministic policies and $q_\pi(s, \pi'(s)) \geq v_\pi(s)$ then the policy $\pi'$ is as good or better than $\pi$.
NOTE on Convention: As per the convention of the book goes, I think they are using rewards for state to action to state transition sequence rather than state to action transition.
My questions are:

Why are they comparing state value function to action value function?
Isn't it obvious the above equation might hold true (provided we select the best action among the possible actions) even for the policy $\pi$ since then the equation will change to $q_\pi(s, \pi(s)) \geq v_\pi(s)$ and we know $v_\pi(s) = \sum_{a \in \mathcal{A}(s)} \pi(a|s)q_\pi(s, a)$?

What is the inconsistency here?

About the last question which might seem a little illegible, since in state value function we will weigh all the possible actions with their probability there is a chance a low reward action state might creep in, whereas in the action value function the weight-age is already bypassed and we are directly measuring the worth of a function which might be better or worse even though the actions taken from the same policy used to calculate state value.

Why are they comparing state value function to action value function?

It is because $v_{\pi}(s)$ and $q_{\pi}(s,a)$ measure the same quantity at different stages of the trajectory. By comparing the values at the same $s$ and modifying how $a$ is selected, the proof makes assertions about how that choice impacts the value.
It is important to recall that $v_{\pi}(s)$ measures the expected future reward when starting in state $s$ and following policy $\pi$, and that $q_{\pi}(s,a)$ measures the expected future reward when starting in state $s$ and taking action $a$, thereafer following policy $\pi$. When $a$ is chosen using a deterministic $a = \pi(s)$ then $v_{\pi}(s) = q_{\pi}(s,\pi(s))$

Isn't it obvious the above equation might hold true

The inequality (it is not an equation) does strictly hold true, because the equation $v_{\pi}(s) = q_{\pi}(s,\pi(s))$ is true. However, that is not terribly useful, it doesn't prove anything new.
What is interesting is if you change the decision for $a$

(provided we select the best action among the possible actions)

You cannot do that. The policy decides the actions, by definition. Selecting a better action according to the action value function, and showing what that does is precisely what the proof is showing.

since then the equation will change to $q_\pi(s, \pi(s)) \geq v_\pi(s)$

$q_\pi(s, \pi(s)) = v_\pi(s)$ - yes your inequality holds, but is equivalent to not changing anything.

and we know $v_\pi(s) = \pi(a|s)q_\pi(s, \pi(s))$?

Here you have switched to using a non-deterministic policy and a deterministic policy in the same statement (you also have not defined $a$ properly), making your equation badly formed. The correct form would be:
$$v_\pi(s) = \sum_{a \in \mathcal{A}(s)} \pi(a|s)q_\pi(s, a)$$
This is not a relevant form for the initial policy improvement theorem. However it does become relevant when the theory is adapted to show improving $\epsilon$-greedy policies later.

NOTE on Convention: As per the convention of the book goes, I think they are using rewards for state to action to state transition sequence rather than state to action transition.

That is not a convention in the book. In the book, the reward distribution is generally given as joint distribution with state transition in $p(s', r | s, a)$ - this is a very generic approach that models any discrete MDP, regardless of how the rewards are described as associated with any of $s, a, s'$ or a random factor.
Possibly the convention you are referring to is labelling the immediate reward as $R_{t+1}$, so that a small section of trajectory starting from state $S_t$ might be $S_t, A_t, R_{t+1}, S_{t+1}$ . . . some other texts will use $R_t$ here. However, that is not relevant to the policy improvement theorem, it would hold the same under either convention, just with slight relabelling of the reward indices.

What does deterministic mean? Does it mean for a state that action will be chosen with probability 1? Because only in that case v(s)=q(s,a) for a policy.

About the convention they are assigning specific rewards for a s', a, and s pair. So doesn't it mean they are assigning rewards once a state action state sequence is completed as compared to David Silver's lecture where if i understood correctly the formula's are based on state->action rewards?

@DuttA "What does deterministic mean?". It means that $\pi(s)$ outputs a single action value. If you use the form of the policy function $\pi(a|s)$ then yes one $a$ will output 1, and the rest 0 - however the proof you refer does not use that form. You cannot say "v(s)=q(s,a)", it is badly formed, as you have not defined $a$ properly (is it $\pi(s)$ ?) It might seem that I am nitpicking on that, but it seems to be loose thinking around this issue that has got you a bit lost in the proof.

Yes it is....so if the formula is valid only for deterministic policies, then it is not very generalized.

@DuttaA: It proves policy improvement, which can be done using deterministic policies. Given that the optimal policy can always be deterministic in strict MDPs, it is as general as it needs to be, and definitely gives strong theoretical backing to Dynamic Programming approach of Policy Iteration

Later in the book, the proof is extended to show a similar convergence to "best" policy applies to $\epsilon$-greedy policies. Note that the best $\epsilon$-greedy policy is unlikely to be strictly optimal, but if you can reduce $\epsilon$ then you can get arbitrarily close to an optimal deterministic policy


text:
Understanding the update rule for the policy in the policy iteration algorithm
Consider the grid world problem in RL. Formally, policy in RL is defined as $\pi(a|s)$. If we are solving grid world by policy iteration then the following pseudocode is used:

My question is related to the policy improvement step. Specifically, I am trying to understand the following update rule.
$$\pi(s) \leftarrow arg max_a \sum_{s'}p(s'|s,a)[r(s,a,s') + \gamma v(s')] $$
I can have 2 interpretations of this update rule.

In this step, we check which action (say, going right for a particular state) has the highest reward and assign going right a probability of 1 and rest actions a probability of 0. Thus, in PE step, we will always go right for all iterations for that state even if it might not be the most rewarding function after certain iterations.
We keep the policy improvement step in mind, and, while doing the PE step, we update the $v(s)$, based on the action giving highest reward (say, for 1st iteration $k=0$, going right gives the highest reward, we update based on that, while, for $k=1$, we see going left gives the highest reward, and update our value based on that likewise. Thus action changes depending on maximum reward).

For me, the second interpretation is very similar to value iteration. So, which one is the correct interpretation of a policy iteration?

Your first interpretation is correct, in policy evaluation step, every loop iteration you act considering the current fixed policy that you think is the best, that policy may not be the overall best policy, but currently we think it is. You learn about how good that policy is by updating values (while acting according to our current policy) and then you improve it in policy improvement step based on those values that you got in PE step.

@Brale_ The first interpretation is wrong. You don't necessarily always go "right" for all iterations of a single PE step. The greedy action might change, after each PE step. I also clarify in my answer that the greedy action might not be the same for all states, so you don't necessarily go "right" for all states (during a single run of PE or, equivalently, for different iterations of the same PI step).

@nbro greedy action changes after policy improvement step (hence the name improvement), in policy evaluation you evaluate your current policy, action that you choose for specific state doesn't change in every loop iteration of policy evaluation step

@Brale_ Yes, even though there might not be a strict improvement for all states in PI. However, note that in the OP stated "Thus in PE step we will always go right for all iterations even if it might not be the most rewarding function after certain iterations.". To me this seems to imply that $\pi(s)$ is "right" for all states, but this is not necessarily the case. Maybe I am just wrongly interpreting this statement, which is ambiguous.

there might be a misunderstanding, I thought the OP was saying that we act according to our current policy in states, obviously if an action for $s_1$ is going right it would be incorrect going right in $s_2$, unless that is the greedy action for that state

@nbro I meant for a particular state not all states, if it was ambiguous previously.

@Brale_ no I meant right for a single state only, it maybe clearer from my comments below the answer.

A policy can be stochastic or deterministic. A deterministic policy is a function of the form $\pi_{\text{deterministic}}: S \rightarrow A$, that is, a function from the set of states to the set of actions. A stochastic policy is a map of the form $\pi_{\text{stochastic}} : S \rightarrow P(A)$, where $P(A)$ is a set of probability distributions ($P(A) = \{ p_{s_1}(A), p_{s_2}(A), \dots, p_{s_{|S|}}(A) \}$, where $p_{s_i}(A)$ is a probability distribution over the set of actions $A$ for the state $s_i$ and $|S|$ is the size of the set of states of the environment) over the set of actions $A$. A deterministic policy can be interpreted as a stochastic policy that gives the probability of $1$ to one of the available actions (and $0$ to the remaining actions), for each state.
In the case of value iteration (VI) and policy iteration (PI), the policy is deterministic, both in the policy evaluation (PE) or policy improvement (PI) steps.
In the PE step, for a specific state $s$ (inside the for loop), you use $\pi(s)$, that is, you assume that the action taken in the specific state $s$ is the greedy action, because of the $\text{argmax}$ in the PI, according to the current policy. However, in general, $\pi_k(s_i) \neq \pi_k(s_j)$, for $i \neq j$, where $k$ is the current iteration step (of PE and PI).
The update rule
$$\pi_{k+1}(s) \leftarrow arg max_a \sum_{s'}p(s' \mid s, a)[r(s, a, s') + \gamma v_k(s')]$$
computes the action that is expected to give the highest return, given the current and fixed value function $(v_k$). Have a look at the definition of expectation for discrete random variables: it is defined as a weighted sum (like in the update rule above, where the weights are $p(s' \mid s, a)$).
Value iteration is a shorter version of policy iteration. In VI, rather than performing a PI step for each state of the environment, when performing PE, the action that is expected to give the highest return is chosen. In VI, the policy is updated only once and at the end of policy evaluation. In PI, you alternate between PE and PI, and, at each PI, you update the policy.

So the main question is the probablity assigned during the Pimprovement step or Pevaluation step? If you see that was the two of my interpretations. Basically if you use tensorflow then you'll know in it we don't explicitly define any data, but we just build a model. So is the policy an explicit definition of probablity or is it a rule which is used in PE step (I.e act greedily).

Thus would the probablity be already assigned before the beginning of PE step or would it be assigned for each PE iteration?

@DuttaA When you perform one PE step, the policy is fixed (so the probabilities for that specific PE step have already been assigned). When you perform the next PE step, the policy is also fixed, but it might be different than the fixed policy you used in the previous PE step (because, during PI, you change the policy).

so basically a policy  in a policy evaluation step is a fixed set of probabilities instead of a abstract command 'act greedily' for each state.

@DuttaA Given that the policy used in this context is deterministic, it might not be convenient to think of probabilities. However, the policy you use in PE is fixed (as I said above). This fixed policy used in PE has been computed using a "greedy" method in PI, so I would think of the fixed policy used in PE as a greedy policy with respect to the previous value function. I think this diagram might help: http://incompleteideas.net/book/first/ebook/node46.html. It gives you the idea behind policy iteration.

thanks...So that basically is the difference between VI and PI, acting greedily in VI ensures you are always going towards better which might not be true in PI.

@DuttaA I would think of the difference between PI and VI as follows. $PI: E \rightarrow I \rightarrow E \rightarrow I \rightarrow E \rightarrow \dots \rightarrow I$ and $VI: E \rightarrow E \rightarrow \dots \rightarrow E \rightarrow I$, where $I$ means improvement of the policy and $E$ means evaluation of the current fixed policy (that is, update of the current value function given the current fixed policy).

@DuttaA policy iteration and value iteration are basically the same thing. Value iteration is a special case of policy iteration where you do a single step of policy evaluation (no for loop for $n$ steps, only a single step) and after that policy improvement, while in policy iteration you do policy evaluation for $n$ steps (for loop with $n$ steps) and then policy improvement. Also, in value iteration policy improvement is sort of implicit, its "blended" in in policy evaluation step.

@nbro but in VI the policy is 'act greedily' (probablities will be decided based on which state gives max reward) whereas in Piteration, improvement step passes a fixed set of probabilities (deterministic upto my knowledge) upon which evaluation is acted upon.

@DuttaA Yes, in VI one can think of it as acting greedily, because of the $\max$. In PI, you explicitly use the policy $\pi$. In VI, in the policy evaluation step, you do not explicitly use the actual policy $\pi$. You use a $\max$. This max can be interpreted as if we were using a greedy policy, but I am not sure this is a good idea.


text:
Are there real-world problems where case-based reasoning is not suitable?
I know case-based reasoning has four stages: retrieve, retain, re-use and revise.
Used for solving new problems by adapting solutions that were used to solve old problems, like car issues.
The advantages of it are that solutions are proposed quickly and there's no need to start from scratch, but is there a real-world problem where using this would not be suitable?
Just trying to better my understanding of it.

Hi and welcome to this community. What does case-based reasoning have to do with reinforcement learning (given that you added this tag to your question)?

Wasn't sure what to put as the tag since case-based reasoning isn't an option, apologies, what should I change it to?

Which source have you read that mentions case-based reasoning in the context of AI?

Well I'm learning it as part of a module for AI at uni and the way we're being taught is how it's used to compare old solutions with new problems. Using it with applications to find cars with similar features etc. The reason I'm asking about when it can't be used is I found a question on it on a past paper and when I asked my lecturer if he could provide extra information as the notes don't explain when not to use it, I just got told no.

Have a look at this video Case-Based Reasoning - AI 101.

That was a really helpful video. Much better understanding of it now, Thank you!


text:
How to make the RL player a perfect/expert (tic-tac-toe/chess) player?
I asked a question related to tic-tac-toe playing in RL. From the answer, it seems to me a lot is dependent on the opponent (rightly so, if we write down the expectation equations).
My questions are (in the context of tic-tac-toe or chess):

How to make the RL player a perfect/expert (tic-tac-toe/chess) player?
As far as TTT is concerned, when playing against a perfect player, an RL will become perfect, provided that the opponent is perfect.
So, will this hold true if the same RL algorithm, with its learned values, are used to play some other lesser perfect players?

Can an RL player with pre-trained values (assume from a perfect or expert opponent) be used in any scenario with best results?

Note: The problem is more severe in chess, since experts will use some kind of opening moves which will not match with say a random player and thus finding values for those states becomes a problem, since we have not encountered it during training time.
Footnote: Any resources on Game Playing RL is appreciated.

I would say a good way to make a good agent would be making it play against itself. As you go through several episodes, with a good exploration and exploitation balance, both will gradually learn and converge to $q_*(s,a)$.

So, will this hold true if the same RL algorithm, with its learned values, are used to play some other lesser perfect players?

As long as the states that are played (or approximations if you are using function approximation methods) were simulated enough times during training, it will play well against any kind of opponent.
If you are training against a completely perfect opponent and you are not using function approximation, I believe you could get to an incomplete $Q(s,a)$ table, and so not be able to predict the best play when facing certain states.

I figured out that playing against itself is a good idea, but I think it's good if the state space is small...But for big games like chess I am not sure it would work even if we use function approximation since we have to assume that some kind of underlying function is there on chess determining state value vs positions.


text:
Would it be ethical to use AI to determine a user’s gender from the content they upload, without them knowing?
I’ve been thinking about this for a few days and can’t tell if this would feel morally just to an average user

Interesting question but I think it's in the realm of business ethics or philosophy as opposed to UX. AI tools analyze text and actions which are, for the most part, outside the realm of UX.

Ethics aside, I think this could potentially create a number of different issues. What if your algorithm guesses wrong and I'm stuck with a UI that's targeting incorrectly? What if my wife and I share an account? What if I'm a (insert orientation here) (insert gender here) who [coaches, supports] a [men's, women's] [volleyball, football] team? People are very diverse, and you could make many mistakes. Would you then provide tools for the user to correct these mistakes? How would you be able to do so without the user being offended?
Instead, I see many fewer issues that might arise from a section like the following:

Let's get to know each other!
If you'd like, we can help tune your profile to match your interests.
You can start by selecting some of your interests from the list below, or try searching for your own.
Fishing Cooking Soccer Video Gaming Social Media
Search for my interest...
Skip this section →

You could continue with other questions that might actually be relevant to your shaping your UI or application, allowing the user to omit/delete details for any or all questions.
Being purely an opt-in experience prevents any issues with the user feeling like your app may be "talking about them" behind their back.
In this case, specifically at this point in time, I think it's wise to be transparent about what data your application knows (or thinks it knows) about your users.

My reasoning is: if any person can look at another person’s public profile and visually determine their gender without even consciously trying to, what makes it morally questionable to write code to do this same, extremely evolutionarily fundamental activity. You don’t give consent to strangers in public to determine your gender without you knowing. It’s so intrinsic to human behavior that it’s expected.

Your suggestion exists at the intersection of many controversial topics. Gender, user privacy, AI, and data collection are all regularly in the news, with people voicing loud opinions on either side of each topic. By not allowing the user any control, you're imposing your view of all 4 topics on the user secretly and involuntarily. In order to avoid the possible legal implications that should be accounted for and maintain the user trust that could be lost from incorrect conclusions, I'm offering a suggestion of transparency by allowing a user to opt in to this feature instead.

Aren't we already doing things with technology that are not ethical? But to answer your question, yes, it will be unethical to use gender to target a user. It will mean that the AI will be fed a list of things to look for for a certain gender which is again gender-bias. I find targetting users unethical but using gender basis would really be something else. I am not sure if I am on the right track but ads are run keeping in mind the target audience which involves the age, gender, etc. That is done by humans but still.

The general principle is that data belongs to the person who generated it, and permission should be sought before you use someone else's possessions.
So, it is ethical to use it to infer whatever you like about a person, provided that you ask permission first.


text:
The problem with the Gambler's Problem in RL
Recently I simulated the Gambler's Problem in RL:

Now, the problem is, the curve does not at all appear the way as given in the book. The "best policy" curve appears a lot more undulating than it is shown based on the following factors:

Sensitivity (i.e. the threshold for which you decide the state values have converged).
Probability of heads (expected).
Depending the value of sensitivity it also depends on whether I find the policy by finding the action (bet) which cause the maximum return by using $>$ or by using $>=$ in the following code i.e:

 initialize maximum = -inf
 best_action = None
 loop over states:
    loop over actions of the state:
       if(action_reward>maximum):
          best_action = action

Also note that if we make the final reward as 101 instead of 100 the curve becomes more uniform. This problem has also been noted in the following thread.
So what is the actual intuitive explanation behind such a behaviour of the solution. Also here is the thread where this problem is discussed.

The intuitive explanation is that there are many equally good "optimal" policies. This is mentioned at the end of the example problem description you posted. My gut says that the family of optimal policies would be any policy from the double/nothing family. So, for example, if you bet 25 on the first bet instead of 50, I think your overall chances of winning should be the same as if you bet 50, it'll just take longer in expectation. The resulting family of policies will look more undulating than the one in the book.
As Neil notes, for low values of $p$, the probability that you win a gamble, it is the case that there is a unique optimal policy.

The writer Prof. Sutton himself has commented that he cannot give any explanation, so I thought to ask her. Check the link!

@DuttaA I think what he is saying in that link is that there are many possible solutions. The form of the solutions makes a lot of sense too. Perhaps I will do a formal analysis of this (or maybe someone else here will!), but I think this basically just boils down to the MDP's solution space being under-determined because there's no discounting factor. I suspect that if a discounting factor is added, a unique solution emerges.

Actually if p > 0.5 and p < 1 for heads, you should get a deterministic solution of bet minimum amount (1) always. It is only p <= 0.5 when the double-or-quits logic kicks in.

@JohnDoucette I forgot about the 2nd part of my answer. Why do you think the policy becomes stable if the final is reward is 101 instead of 100? Someone in the thread noted it and I cross checked it, it's happening.

@NeilSlater see my above comment.

@DuttaA My guess is that when the target amount is odd, a lot of the potential ties should go away because there no longer exists any (sane) amount you can bet that simultaneously reaches the win condition and the lose condition.


text:
Deep Generative Networks Probability of "Success"
I have built various "successful" GANs or VAEs that can generate realistic images reliably, but in either case the generative step is sampling a latent feature vector from some distribution and running it through a generator/decoder $G(x) \  s.t. \  x \sim \mathbb{D} $.
Generally $\mathbb{D}$ has a continuous domain (normally distributed is a common choice), and $G(x)$ is a continuous and at-least singly differential function (by construction so it could be optimized with a gradient scheme).
Now assume we have 2 images $y_1, y_2$ generated by $x_1, x_2 \sim \mathbb{D}$ and that shortest path from $x_1$ to $x_2$ is also in $\mathbb{D}$'s domain. Crawling along this path, we get a continuous path from $y_1$ to $y_2$ in image space. Wouldnt you assume along this continuous transformation that the images would be a "Garbage" as they are essentially a fusion of $y_1, y_2,$ and $\{y_k\}_k$ where $\{y_k\}_k$ describes a set of "succesful" stops along the way?
 --Especially in cases of single mode distributions (like normal) where all $x$ on the path must have higher likelihood than atleast one of the paths endpoints.
Trying to summarize my point, how do continuous generative models not with high probability produce garbage when given any 2 "succesful" images, there probably is a high range of "garbage" examples along a path between them in image space?
Note: by "succesful" i mean the generated image could be considered a draw from the true distribution you are trying to capture by the generator, and by "garbage" i mean ones that are obviously not.

"Wouldnt you assume along this continuous transformation that the images would be a "Garbage" as they are essentially a fusion" - why do you think they would be a fusion? Also, what do you exactly mean by "Crawling along this path". The VAE is trained to learn the distribution that resembles the training dataset, this is likely why the results are acceptable, if this is what you mean.

@nbro by definition it is a continuous path from $y_1$, and $y_2$, which is where i get the "fusion"... and i get your point, that is the training objective, but im saying by design of the gen. Example: imagine a gen training on a multiclass dataset, intuitively you expect domain($\mathbb{D}$) to be pseudo-partioned into realms where certain clusters generate different classes. Now follow a line from one class to another, eventually youll hit a boundary where you see images that are questionably one or the other, probably producing 'garbage'

To be sure, $\mathbb{D}$ is the latent space and $x_1$ and $x_2$ are two latent variables (or are they the input examples, that is, other images)?

yes thats right. it is the latent space.


text:
Do I need to use a pre-processed dataset to classify comments?
I want to use Machine Learning for text classification, more precisely, I want to determine whether a text (or comment) is positive or negative. I can download a dataset with 120 million comments. I read the TensorFlow tutorial and they also have a text dataset. This dataset is already pre-processed, like the words are converted to integers and the most used words are in the top 10000.
Do I also have to use a pre-processed dataset like them? If yes, does it have to be like the dataset from TensorFlow? And which pages could help me to implement that kind of program?
My steps would be:

find datasets
preprocess them if needed
feed them in the neural network

Here's a list of some of the best python libraries for natural language processing.

Natural Language Toolkit (nltk)
Covers all the basic functions and NLP tools such as tokenization etc.
TextBolb
This is a good library of beginners, it provides the nltk toolkit in a simplified format.
Spacy
It is an advanced library and can be used in production code.

You can preprocess textual data in a number of ways. It depends on the type of task at hand and the size of the data.
From your question, I think you are referring to converting the words to a vector form word2vec. Here is a massive word2vect list from Google.
Also have a look at preprocessing techniques in NLP such as tf-idf etc.


text:
Is predicting day of week straight forward?
I am using python and Xgboost. I have features: activity and location and time stamps of when the activity occurred.
I want to predict day of week. Is this straight forward, ie y=day of week, X={activity, location}, or am I being naive and I need to do fancy time series things? I'd also like to predict time of day.

Do you have a finite list of activities? How many different activities are there?  What is location?  Is it lattitude and longitude, or you have some finite list of locations (how long), or something else?

Small number of activities: either around 10 or 200, not sure which I will use. Location is a region: just an arbitrary number denoting a geographical area.

How many regions are there, and what is the size of training set?

Sorry I missed your comment.  Around 100 regions although possibly it could be as many as 1000. Training set is around 150,000.