text:
Can this be a possible deep q learning pseudocode?
I  am not using replay here. Can this be a possible deep q learning pseudocode?
s - state
a - action
r - reward
n_s - next state
q_net - neural network representing q

step()
{

    get s,a,r,n_s
    q_target[s,a]=r+gamma*max(q_net[n_s,:])
    loss=mse(q_target,q_net[s,a])
    loss.backprop()

}

while(!terminal)
{
    totalReturn+=step();
}

It looks generally valid to me. There are a couple of things missing/implied that I'd like to give feedback on though:

I am not using replay here

Then it won't work, except for the most simple and trivial of problems (where you probably would not need a neural network anyway).

get s,a,r,n_s

I would make it more explicit where you get these values from, and split up the assignments.
# before step()
s = env.start state()
a = behavior_policy(q_net[n_s,:])

  # inside step(), first action
  r, n_s = env.step(s, a)

  # ....rest of loop

  # inside step(), last actions
  s = n_s
  a = behavior_policy(q_net[n_s,:])

In the above code, env is the environment which takes state, action pairs as input and returns reward plus next state. It would be equally valid to have current state as a property of the environment, and query that when needed. The behaviour_policy is a function that selects an action based on current action values - typically this might use an $\epsilon$-greedy selection.

while(!terminal) {

You appear to run just one episode in your pseudo-code. You will want an outer loop to run many episodes. Also it is not clear how you are deciding the value of terminal - in practice you will need a handle to current environment state via some variable.
Without the outer loop to start new episodes, and some variables defined to communicate between sections of code, it is difficult to follow the code and decide where to add details (such as where env.start_state() call should go, or whether something like s = env.reset() would be more appropriate).

totalReturn+=step();

In your pseudocode you do not return anything from step and it is not clear what you hope to do with this totalReturn variable. Technically it won't equal the definition of return in RL for any state, not even the starting state if gamma < 1.0.
However, the sum of all rewards seen in an episode is a useful metric. In Deep RL it is OK to treat gamma as a solution hyperparameter, and your target metric can be expected undiscounted return from the start state.


text:
What does it mean to have epochs=30 in Keras' fit method given certain data?
I have read a lot of information about several notions, like batch_size, epochs, iterations, but because of explanation was without numerical examples and I am not native speaker, I have some kind of problem of understanding still about those terms, so I decided to work with data. Let us suppose we have the following data

Of course, it is just subset of original data, and I want to build a neural network with three hidden layers, the first layer contains 500 nodes, it takes input three variable and on each node, there is sigmoid activation function, next layer contains 100 node and sigmoid activation, the third one contains 50 node and sigmoid again, and finally we have one output with sigmoid to convert the result into 0 or 1 that classify whether a person with those attributes is female or male.
I trained the model using Keras Tensorflow with the following code
model.fit(X_train,y_train,epochs=30)

With this data, what does mean epochs=30? Does it mean that all 177 rows  (with 3 input at times) will go to the model 30 times? What about batch_size=None in model.fit parameters?

Hi dato, I think your question have a lot of irrelevant information, what is your main question?

i wanted to understand following thing : if  my train data has size of (177,3) and batch size is 2 , let say, how update process is going? let us suppose neural network takes 3 input and 1 output

@malioboro i have answered and could you please check if it is correct?

Hi and welcome to AI SE! If you're asking how a certain library, method or API works, then your question is off-topic here. See https://ai.stackexchange.com/help/on-topic for more details. You should ask these questions on Stack Overflow.

Batch size and epochs are independent parameters - they serve very different purposes. Your main question as I understand it (and for general, non-library specific consumption) is what is an epoch and how is the data used for each epoch?
Simply put, an epoch is a single iteration though the training data. Each and every sample from your training dataset will be used once per epoch, whether it is for training or validation. Therefore, the more epochs, the more the model is trained. The key is to identify the number of epochs that fits the model to the data without overfitting.
Your explaination of how batch size affects the training process is correct but not relevant to the question since it has no relation to the epoch training iterations. That is not to say that these values should be considered independently since they have similar effects on the model training process.

ok  so let me explain in my word how i  understood this process:
i know that one sample mean one row, therefore if we have data with size(177,3), that means  we have 177 sample. because we have divided  X and y into training and test, therefore we have following pairs (X_train,y_train)  and (X_test, y_test)
now about batch size, if we have  let say 177 sample(177 row) and  2 batch_size , that means we have approximately  $177/2$ batch right?update process goes  like this:
let us suppose network takes 3 input and produce one output, from first sample of data, three data will go to the network and  output will be  generated, this output will be compared to the  first value of y_train and  cost function will be created, then next  sample will go(it means next three value) and compared to the second value of y_train, also second cost function will be generated, final cost function for first batch will be sum of those cost functions and using gradient method weights are updated, after that one new batches will go through the network and on the based on updated weights, new weights are generated, when all  $177/2$  batch will be finished , it will be our 1 epoch right? is that correct?

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What does "In each generation, 25% of offspring resulted from mutation without crossover" mean in the context of NEAT?
I am reading through the NEAT paper. In parameter settings, page 15, there is:

In each generation, 25% of offspring resulted from mutation without crossover.

What does it mean?

In genetic algorithms, mutation without crossover simply means that part of the population is randomly changed. In this case this is applied to 25% of the population.
The remaining 75% either remain unchanged (generally the best performing specimen), or will be combined with other specimen (using crossover). It's a bit more complex here, as the genome is connecting weights in networks, and they use 'species', where certain individuals are treated as separate groups from the rest of the population.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What is the proof that policy evaluation converges to the optimal solution?
Although I know how the algorithm of iterative policy evaluation using dynamic programming works, I am having a hard time realizing how it actually converges.
It appeals to intuition that, with each iteration, we get a better and better approximation for the value function and we can thus assure its convergence, but with this said simply, it seems that this method is very inefficient contrary to the reality that it actually is quite efficient.
What is the rigorous mathematical proof of the convergence of the policy evaluation algorithm to the actual answer? How is it that the value function obtained this way is close to the actual values computed by solving the set of bellman equations?

First of all, efficiency and convergence are two different things. There's also the rate of convergence, so an algorithm may converge faster than another, so, in this sense, it may be more efficient. I will focus on the proof that policy evaluation (PE) converges. If you want to know about its efficiency, maybe ask another question, but the proof below also tells you about the rate of convergence of PE.

What is the proof that policy evaluation converges to the optimal solution?

To provide some context, I will briefly describe policy evaluation and what you need to know to understand the proof.
Policy evaluation
Policy evaluation (PE) is an iterative numerical algorithm to find the value function $v^\pi$ for a given (and arbitrary) policy $\pi$. This problem is often called the prediction problem (i.e. you want to predict the rewards you will get if you behave in a certain way).
Two versions: synchronous and asynchronous
There are (at least) two versions of policy evaluation: a synchronous one and an asynchronous one.
In the synchronous version (SPE), you maintain two arrays for the values of the states: one array holds the current values of the states and the other array will contain the next values of the states, so two arrays are used in order to be able to update the value of each state at the same time.
In the asynchronous version (APE), you update the value of each state in place. So, first, you update the value of e.g. $s_1$, then $s_2$, etc., by changing your only array of values (so you do not require a second array).
SPE is similar in style to the numerical method called Jacobi method, which is a general iterative method for finding a solution to a system of linear equations (which is exactly what PE is actually doing, and this is also explained in the cited book by Sutton and Barto). Similarly, APE is similar in style to the Gauss–Seidel method, which is another method to solve a system of linear equations.
Both of these general numerical methods to solve a system of linear equations are studied in detail in Parallel and Distributed Computation Numerical Methods (1989) by Bertsekas and Tsitsiklis, which I haven't read yet, but provides convergence results for these numerical methods.
The book Reinforcement learning: an introduction by Sutton and Barto provides a more detailed description of policy evaluation (PE).
Proof of convergence
I will provide a proof for the SPE based on these slides by Tom Mitchell. Before proceeding, I suggest you read the following question  What is the Bellman operator in reinforcement learning? and its answer, and you should also get familiar with vector spaces, norms, fixed points and maybe contraction mappings.
The proof that PE finds a unique fixed-point is based on the contraction mapping theorem and on the concept of $\gamma$-contractions, so let me first recall these definitions.

Definition ($\gamma$-contraction): An operator on a normed vector space $\mathcal{X}$ is a $\gamma$-contraction, for $0 < \gamma < 1$, provided for all  $x, y \in \mathcal{X}$
$$\| F(x) - F(y) \| \leq \gamma \| x - y\|$$
Contraction mapping theorem: For a $\gamma$-contraction $F$ in a complete normed vector space $\mathcal{X}$

Iterative application of $F$ converges to a unique fixed point in $\mathcal{X}$ independently of the starting point

at a linear convergence rate determined by $\gamma$

Now, consider the vector space $\mathcal{V}$ over state-value functions $v$ (i.e. $v \in \mathcal{V})$. So, each point in this space fully specifies a value function $v : \mathcal{S} \rightarrow \mathbb{R}$ (where $\mathcal{S}$ is the state space of the MDP).
Theorem (convergence of PE): The Bellman operator is a $\gamma$-contraction operator, so an iterative application of it converges to a unique fixed-point in $\mathcal{V}$. Given that PE is an iterative application of the Bellman operator (see What is the Bellman operator in reinforcement learning?), PE finds this unique fixed-point solution.
So, we just need to show that the Bellman operator is a $\gamma$-contraction operator in order to show that PE finds this unique fixed-point solution.
Proof
We will measure the distance between state-value functions $u$ and $v$ by the $\infty$-norm, i.e. the largest difference between state values:
$$\|u - v\|_{\infty} = \operatorname{max}_{s \in \mathcal{S}} |u(s) - v(s)|$$
Definition (Bellman operator): We define the Bellman expectation operator as
$$F^\pi(v) = \mathbf{r}^\pi + \gamma \mathbf{T}^\pi v$$
where $v \in \mathcal{V}$, $\mathbf{r}^\pi$ is an $|\mathcal{S}|$-dimensional vector whose $j$th entry gives $\mathbb{E} \left[ r \mid s_j, a=\pi(s_j) \right]$ and $\mathbf{T}^\pi$ is an $|\mathcal{S}| \times |\mathcal{S}|$ matrix whose $(j, k)$ entry gives $\mathbb{P}(s_k \mid s_j, a=\pi(s_j))$.
Now, let's measure the distance (with the $\infty$-norm defined above) between any two value functions $u \in \mathcal{V}$ and $v \in \mathcal{V}$ after the application of the Bellman operator $F^\pi$
\begin{align}
\| F^\pi(u) - F^\pi(v) \|_{\infty}
&= \| (\mathbf{r}^\pi + \gamma \mathbf{T}^\pi u) - (\mathbf{r}^\pi + \gamma \mathbf{T}^\pi v)\|_{\infty} \\
&= \| \gamma \mathbf{T}^\pi (u - v)\|_{\infty} \\
&\leq \| \gamma \mathbf{T}^\pi ( \mathbb{1} \cdot \| u - v \|_{\infty})\|_{\infty} \\
&\leq \| \gamma (\mathbf{T}^\pi \mathbb{1}) \cdot \| u - v \|_{\infty}\|_{\infty} \\
&\leq \gamma \| u - v \|_{\infty}
\end{align}
where $\mathbb{1} = [1, \dots, 1]^T$. Note that $\mathbf{T}^\pi \cdot \mathbb{1} = \mathbb{1}$ because $\mathbf{T}^\pi$ is a stochastic matrix.
By the Bellman expectation equation (see Barto and Sutton's book and What is the Bellman operator in reinforcement learning?), $v^\pi$ is a fixed-point of the Bellman operator $F^\pi$. Given the contraction mapping theorem, the iterative application of $F^\pi$ produces a unique solution, so $v^\pi$ must be this unique solution, i.e. SPE finds $v^\pi$. Here is another version of the proof that the Bellman operator is a contraction.
Notes
I didn't prove the contraction mapping theorem, but you can find more info about the theorem and its proof in the related Wikipedia article.

There exist other RL books which do a better job of talking about this but it's pretty simple at it's core.
The discounting factor puts an upper limit on the difference in reward between a finite number of iterations and an infinite number, each time you add another iteration it decreases by $\gamma$ multiplied into the upper bound of the difference.
$V_\pi = E[\sum_{i=0}^{\infty} \gamma^iR_i]$, $\Delta V_k = E[\sum_{i=k}^{\infty} \gamma^iR_i] = \gamma^k E[\sum_{i=0}^{\infty} \gamma^{i}R_{i+k}] < \gamma^k \frac{1}{1-\gamma}R_{max}$

This explanation isn't clear at all (to me). The discount factor may play a role, but that doesn't give much intuition behind the algorithm, what it does and why it converges, in my opinion. I think it would be nice if you could provide a full description of the proof (maybe taken from a book, that's fine!). I think a good answer would start from a description of the algorithm and what it does. Then it proves why it converges.

@SAGALPREETSINGH I may provide a more formal and complete answer later, if Fourier doesn't decide to do it.

@FourierFlux can you please be a bit more descriptive. I am not able to understand your answer. I think you didn't get my question right. I wanted to know, how policy evaluation works out and it happens that the algorithm converges. The intuition I formed for this is that everytime we iterate over current value function, we get closer to the actual one because after each iteration, our value function becomes more and more appropriate but how in the world does it go so close to the actual answer. What was the actual motivation behind using Bellman equation as an update rule.


text:
How does size of the dataset depend on VC dimension of the hypothesis class?
This might be a little broad question, but I have been watching Caltech youtube videos on Machine Learning, and in this video prof. is trying to explain how we should interpret the VC dimension in terms of what it means in layman terms, and why do we need it in practice.
The first part I think I understand, please correct me if I am wrong. VC Dimension dictates the number of effective parameters (i.e. degrees of freedom) that model has. In other words, the number of parameters the model needs in order to cover all possible label combinations for the chosen dataset. Now, the second part is not clear to me. The professor is trying to answer the question:

How does knowing the VC dimension of the hypothesis class affect number of samples we need for training?

Again, I apologize if all of this may be trivial, but I am new to the field and wish to learn as much as I can, so I can implement better and more efficient programs in practice.

From [1] we know that we have the following bound between the test and train error for i.i.d samples:
$$
\mathbb{P}\left(R \leqslant R_{emp} + \sqrt{\frac{d\left(\log{\left(\frac{2m}{d}\right)}+1\right)-\log{\left(\frac{\eta}{4}\right)}}{m}}\right) \geqslant 1-\eta
$$
$R$ is the test error, $R_{emp}$ is the training error, $m$ is the size of the training dataset, and $d$ is the hypothesis class's VC dimension. As you can see, the training and test errors have some relations to the dataset's size ($m$) and $d$.
Now, in terms of PAC learnability, we want to find a (lower or upper) bound for $m$ such that the absolute difference between $R$ and $R_{emp}$ will be less than a given $\epsilon$ with a given probability of at least $1-\eta$.  Hence, $m$ can be computed in terms of $\epsilon$, $\eta$, and $d$. For example, it can be proved ([2]) to train a binary classifier with $\epsilon$ difference between test and train error with the probability of at least $1-\eta$, we need $O\left(\frac{d + \log\frac{1}{\eta}}{\epsilon} \right)$ i.i.d sample data, i.e., $m = O\left(\frac{d + \log\frac{1}{\eta}}{\epsilon}\right)$. See more example and references here.

Given a hypothesis set $H$, the set of all possible mappings from $X\to Y$ where $X$ is our input space and $Y$ are our binary mappings: $\{-1,1\}$, the growth function, $\Pi_H(m)$, is defined as the maximum number of dichotomies generated by $H$ on $m$ points. Here a dichotomy is the set of $m$ points in $X$ that represent a hypothesis. A hypothesis is just a way we classify our points. Therefore with two labels we know,
$$\Pi_H(m)\leq 2^m$$
This is just counts every possible hypothesis. The VC dimension is then the largest $m$ where $\Pi_H(m)=2^m$.
Consider a 2D perceptron, meaning our $X$ is $\mathbb{R}^2$ and our classifying hyperlane is one-dimensional: a line. The VC dimension will be 3. This is because we can shatter (correctly classify) all dichotomies for $m=3$. We can either have all points be the same colour, or one point be a different colour - which is $2^3=8$ dichotomies. You may ask what if the points we are trying to classify are collinear. This does not matter because we are concerned with resolving the dichotomies themselves, not the location of the points. We just need a set of points (wherever they may be located) that exhibits that dichotomy. In other words, we can pick the points such that they maximize the number of dichotomies we can shatter with one classifying hyperplane (a triangle): the VC dimension is a statement of the capacity of our model.
To make this clear, consider $m=4$. We can represent the truth table of the XOR gate as a dichotomy but this is not resolvable by the perceptron, no matter where we choose the location of the points (not linearly separable). Therefore, we can resolve a maximum of 8 dichotomies, so our VC dimension is 3. In general, the VC dimension of perceptrons is $d+1$ where $d$ is the dimension of $X$ and $d-1$ is the dimension of the classifying hyperplane.

If the 3 points are collinear, the perceptron cannot classify them, so it's not "any" dataset of 3 points. Apart from that, your answer seems correct

Yeah, I believe that has to do with the growth function. In this case, it is $\Pi_H(m)=2^3$ and the growth function of a hypothesis set is defined to maximize the number of dichotomies so a collinear set of points wouldn't be applicable when talking about the VC dimension since that isn't classifiable with one plane - they need to be in a triangular layout. But that's a good point and I'll edit my answer to be more formal.

@ArchieShahidullah thanks for your answer! Maybe I did not format my question correctly, but it is not the answer I was looking for. I understand what you wanted to say, but I think other answers are more related to my actual question. Anyhow, thanks for the effort!

The VC dimension represents the capacity (the same Vapnik, the letter V from VC, calls it the "capacity") of a model (or, in general, hypotheses class), so a model with a higher VC dimension has more capacity (i.e. it can represent more functions) than a model with a lower VC dimension.
The VC dimension is typically used to provide theoretical bounds e.g. on the number of samples required for a model to achieve a certain test error with a given uncertainty or, similarly, to understand the quality of your estimation given a certain dataset.
Just to give you an idea of how the bounds look like, have a look at the theorem on page 6 (of the pdf) of the paper An overview of statistical learning theory (1999) by Vapnik.
Have also a look at this answer, where I provide more info about the VC dimension, in particular, in the context of neural networks.

I will later update this answer to provide more relevant (to the question) bounds and maybe explain them. Alternatively, have a look at this other answer.

Thank you for the references. I am currently reading your answer on VC dimensions in the context of neural networks as we speak.


text:
Why does this model have 12 parameters?
I guess the model shown in this image (img_1)

is the same as the one in this image (img_2)

I was trying to build a neural net like that.
This keras code is to do the job.
model = Sequential()
model.add(Dense(3, input_dim=3, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
plot_model(model, to_file='model_plot.png', show_shapes=True, show_layer_names=True)

However, print(model.summary()) outputs
Model: "sequential_17"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
dense_31 (Dense)             (None, 3)                 12
_________________________________________________________________
dense_32 (Dense)             (None, 1)                 4
=================================================================

There are 3 ws and 1 b in the hidden layer. Why does this model have 12 parameters?

You have 3 inputs going to 3 nodes in the input layer. Each connection has a weight so you have  3 X 3 =9 weights. Plus each node has a bias weight so that adds 3 more weights for a total of 12. Your output layer has 3 inputs and is a single node so you have 3 weights for the inputs to the node plus a bias weight for a total of 4. So the total weights in your network is 16.


text:
What are some resources on computational learning theory?
Pretty soon I will be finishing up Understanding Machine Learning: From Theory to Algorithms by Shai Ben-David and Shai Shalev-Shwartz. I absolutely love the subject and want to learn more, the only issue is I'm having trouble finding a book that could come after this. Ultimately, my goal is to read papers in JMLR's COLT.

Is there a book similar to "Understanding Machine Learning: From Theory to Algorithms" that would progress my knowledge further and would go well after reading UML?
Is there any other materials (not a book) that could allow me to learn more or prepare me for reading a journal like the one mentioned above?

(Also, taking courses in this is not really an option, so this will be for self-study).
(Note that I have also asked this question here on TCS SE, but it was recommended I also ask here.)

Although I have only partially read or not read at all some of the following resources and some of these resources may not cover more advanced topics than the ones presented in the book you are reading, I think they can still be useful for your purposes, so I will share them with you.
I would also like to note that if you understand the contents of the book you are currently reading, you are probably already prepared for reading some (if not most of) the research papers you wish to read. Initially, you may find them a little bit too succinct and sometimes unclear or complex, but you need to get used to this format, so there's nothing stopping you from trying to read them and learn even more by doing this exercise.
Books

An Introduction to Computational Learning Theory (1994) by Kearns and Vazirani (no free PDF is available, afaik)
The Nature of Statistical Learning Theory (1995, 2000) by Vapnik
Machine Learning (1997) by Mitchell
Statistical Learning Theory (1998) by Vapnik
Prediction, Learning, and Games (2006) by Cesa-Bianchi and Lugosi
Foundations of Machine Learning (2012) by Mohri et al.
Boosting: Foundations and Algorithms (2012) by Schapire and Freund
Bandit Algorithms (2020) by Lattimore and Szepesvári (with associated solutions to the exercises)

Papers

Language identification in the limit (1967) by E. Mark Gold (this paper introduces algorithmic learning theory, i.e. the non-statistical approach to CLT)

A theory of the learnable (1984) by Valiant (this paper introduces PAC learning)

An overview of statistical learning theory (1999) by Vapnik

Introduction to Statistical Learning Theory (2014) by Bousquet et al.

Courses (videos)

Learning from Data (Caltech) (2012) by Yaser Abu-Mostafa
Machine Learning Theory (2015) by Shai Ben-David

Lecture notes

Computational Learning Theory - Michaelmas Term 2018 by Varun Kanade

Introduction to Machine Learning (2009) by Shai Shalev-Shwartz

Concentration-of-measure inequalities (2009) by Gábor Lugosi

Stat 928: Statistical Learning Theory (2011) by Sham Kakades

COMS E6253, Advanced Topics in Computational Learning Theory (2012) Rocco Servedio

9.520: Statistical Learning Theory and Applications, Spring 2012 by Tomaso Poggio et al.

STAT928: Statistical Learning Theory and Sequential Prediction (2014) by Alexander Rakhlin and Karthik Sridharan

ECE 543, Statistical Learning Theory (2019) by Bruce Hajek and Maxim Raginsky

Other
See also this list of resources https://kiranvodrahalli.github.io/links/#resources-notes-textbooks-monographs-classes-etc compiled by Kiran Vodrahalli.

Comments are not for extended discussion; this conversation has been moved to chat.


text:
Is it possible to guide a reinforcement learning algorithm?
I have just started to study reinforcement learning and, as far as I understand, existing algorithms search for the optimal solution/policy, but do not allow the possibility for the programmer to suggest a way to find the solution (to guide their learning process). This would be beneficial for finding the optimal solution faster.
Is it possible to guide the learning process in (deep) reinforcement learning?

The programmer already guides the RL algorithm (or agent) by specifying the reward function. However, the reward function alone may not be sufficient to learn efficiently and fast, as you correctly noticed.
To attempt to solve this inefficiency problem, one solution is to combine reinforcement learning with supervised learning. For example, the paper Deep Q-learning from Demonstrations (2017) by Todd Hester et al. describes an approach to achieve this.
The paper Active Reinforcement Learning (2008) by Arkady Epshteyn et al. also tries to solve this problem but by incorporating approximations (given by domain experts) of the MDP.
There are probably many other possible solutions. In fact, all model-based RL algorithms could probably fall into this category of algorithms that estimate or incorporate the dynamics of the environment to find a policy more efficiently.

Here are two very related interesting papers:

Learning from Human Preferences
Improving Reinforcement Learning with Human Input


text:
Is Expected SARSA an off-policy or on-policy algorithm?
I understand that SARSA is an On-policy algorithm, and Q-learning an off-policy one.
Sutton and Barto's textbook describes Expected Sarsa thusly:

In these cliff walking results Expected Sarsa was used on-policy, but
  in general it might use a policy different from the target policy to
  generate behavior, in which case it becomes an off-policy algorithm.

I am fundamentally confused by this - specifically, how do we define when Expected SARSA adopts or disregards policy. The Coursera Course states that it is On-Policy, further confusing me.
My confusions became realized when tackling the Udacity course, specifically a section visualizing Expected SARSA for simple a gridworld (See section 1.11 and 1.12 in link below). Note that the course defines Expected Sarsa as on-policy.
https://www.zhenhantom.com/2019/10/27/Deep-Reinforcement-Learning-Part-1/
You'll notice the calculation for the new state value Q(s0,a0) as

Q(s0, a0) <— 6 + 0.1( -1 + [0.1 x 8] + [0.1 x 7] + [0.7 x 9] + [0.1
  x 8] - 6) = 6.16.

This is also the official answer. But this would mean that it is running off policy, given that it is stated that the action taken at S1 corresponds to a shift right, and hence expected SARSA (On policy) should yield you.

Q(s0, a0) <— 6 + 0.1( -1 + [0.1 x 8] + [0.1 x 7] + [0.1 x 9] + [0.7 x
  8] - 6) = 6.1

The question does state

(Suppose that when selecting the actions for the first two timesteps
  in the 100th episode, the agent was following the epsilon-greedy
  policy with respect to the Q-table, with epsilon = 0.4.)

But as this same statement existed for the regular SARSA example (which also yields 6.1 as A1 is shift right, as before), I disregarded it.
Any advice is welcome.

Expected SARSA can be used either on-policy or off-policy.
The policy that you use in the update step determines which it is. If the update step uses a different weighting for action choices than the policy that actually took the action, then you are using Expected SARSA in an off-policy way.
Q-learning is a special case of Expected SARSA, where the target policy is greedy with respect to the action values, so there is only ever one $r_{t+1} + \gamma \text{max}_{a'} Q(s_{t+1}, a')$ term to add with a probability $1$.
You can also use Expected SARSA, similarly to SARSA, where the behaviour policy and target policy are identical. It is not identical to SARSA though, because it calculates the TD Target over all possible actions $r_{t+1} + \gamma \sum_{a'} \pi(a'|s_{t+1}) Q(s_{t+1}, a')$
You can construct Expected SARSA updates where $\pi(a|s)$ is different when selecting which action to explore in the environment (behaviour) and when updating the Q values (target). For instance, you can decide to explore using $\epsilon$-greedy with $\epsilon=0.1$ and update the value function with $\epsilon=0.01$.

Hi Neil, so in the example above, if I was telling the model to update itself using an e-greedy policy targeting actions with highest Q-values, and the policy it used to choose actions was instead completely random, it would then be an off policy approach, correct? I feel like the exercise I specified has a typo of some sort.

The last example you specified has me intrigued. If I coded that up, I would have to write up two e-greedy policy methods, then?

@Y.Xu: That might be one way to do it. Doesn't need to be much code, epsilon is just a parameter of the policy, so you could do it as two instances of the same class for instance.


text:
Why are the rewards of my RL agent for the Atari Breakout game decreasing after a certain number of episodes?

The agent is trying to master the Atari Breakout game.
Here is my code
Is that normal that reward_100 decreased that much after it hits 4.5? Is there a way to avoid that behavior?
Be aware that reward_100 is simply mean_reward = np.mean(self.total_rewards[-100:]). In other words, it is the mean over the last 100 rewards. On the graph, reward_100 represents de y-axis and th number of episodes the x-axis.

These are just general questions about your ML approach. Are you decaying the learning rate? And how are you trading off exploitation and exploration?

Decaying the learning rate seems to be the right thing to do. It changed a lot of things.

Nice! Maybe you can provide a formal answer below, once you solve your problem ;)

What algorithm are you using?

@RayWalker I used just a standard DQN. Look at my code in the question to know more info.

Thanks for the answer. It's good practice to include this information in the question. All the best to you.

It seems that decaying the learning rate solved my problem. I changed learning_rate from 0.001 to 0.0001

Hi jgauth, great that decreasing the learning rate has solved your issue. By "decaying the learning rate" @nbro most probably meant to decrease the learning rate during training. This means that instead of a fixed learning rate you use a function that takes the amount of passed training steps as input and outputs a learning rate.

@RayWalker Thanks, I did not notice that. It might be a good idea!


text:
How would you build an AI to output the primary concept of a paragraph?
My thinking is you input a paragraph, or sentence, and the program can boil it down to the primary concept(s).
Example:
Input:

Sure, it would be nice if morality was simply a navigation toward greater states of conscious well-being, and diminishing states of suffering, but aren't there other things to value independent of well-being? Like truth, or beauty?

Output:

Questioning moral philosophy.

Is there any group that's doing this already? If not, why not?

"Natural language understanding" and "Auto summarization" are the keyphrases that should lead you to the relevant work.

Identifying the primary concepts of a paragraph required understanding of the meaning of the text. In natural language processing, we are still a long way off even recognising and representing the meaning of text, let alone summarising the meaning of multiple sentences into a single statement.
Note that this is different from simply summarising a text: this can be done without any understanding based on textual features within the text itself, and ways of doing that have been around for a while. But such approaches will generally remove sentences which seem less relevant to the text, thus shortening it. They will not express the content in different words.
Conceivably people might try this with deep learning, where you train a system with paragraphs and the corresponding concepts, but again such a system would not have any understanding of the meaning, and thus results would be more or less accidental.

Thanks. I agree that "understanding" in the way we talk about it might not apply here, but I think "calibrated" is more appropriate that "accidental". I don't think AlphaStar wins games by accident.

@Otherness Playing games is a whole different topic from understanding open ended meanings. You cannot improve by 'playing' against yourself there.

You're right, not having self-play is a big difference. I wonder if there's some equivalent method of self-improvement for a natural language processing program, similar to self-play for gaming programs. Perhaps we need another big breakthrough similar to GANs. I guess time will tell.

And when dealing with human languages, not everyone speaks perfectly.  Most things need to be understood within a larger context.  The simple command "Lay down on the sheet!" should be interpreted by a robot as my request to fetch some feathers for the quilt we are about to make; but, if someone less pedantic than I am says the same command, it should be interpreted as an ungrammatical request to get into position for its crankcase oil change.  The robot's brain has to allow for non-obvious errors in the input, and to derive the intended meaning based on additional situational knowledge.

Good point, Ray. Interpreting context and imperfect communication seems quite the challenge. In fact, my first reaction to your text command was thinking you were about to start a massage.


text:
If the i.i.d. assumption holds, shouldn't the training and validation trends be exactly the same?
If the i.i.d. (independent and identically distributed) assumption holds for a training-validation set pair, shouldn't their loss trends be exactly the same, since every batch from the validation set is equivalent to having a batch from the training set instead?
If the assumption was to be true wouldn't that make any method that was aware of the fact that there were two separate sets (regularization methods such as early stopping) meaningless?
Do we work with the fact that there is a certain degree of wrongness to the assumption or am I interpreting it wrongly?
P.S - The question stems from an observation made on MNIST (where I suppose the i.i.d assumption holds strongly). The training and validation trends (losses and accuracy both) on MNIST were almost exactly identical for any network (convolutional and feedforward) trained using negative log-likelihood, making regularization meaningless.

It depends on the VC dimension of your problem settings.

@OmG I think it would be very nice if you provide a formal answer from this perspective of computational/statistical learning theory. If you feel like writing one, I would really like to see it!

@OmG Could you elaborate a bit more?

If the i.i.d (independent and identically distributed) assumption holds, shouldn't the training and validation trends be exactly the same?

No, not necessarily. Let me explain why.
If you assume your samples (aka examples, observations, data points, etc.) are i.i.d., this means

that they come from the same distribution, e.g. a Gaussian $\mathcal{N}(0, 1)$ (the identically distributed part), and
that they are independently drawn from it, i.e., intuitively, each sample provides the same kind of information independently of the others

However, even if samples are independently drawn from a certain distribution, they can be different. For example, if you draw a sample $x$ from $\mathcal{N}(0, 1)$, an operation often denoted as $x \sim \mathcal{N}(0, 1)$, $x$ could have the value $0$, $1$, $13$ or $50$ (or any other number), so they could be variable, although your samples will tend to be mainly around $0$, because that's where your Gaussian puts more density (and your standard deviation is just $1$). If your standard deviation was higher, then there would be even more variability in the sampling process.
So, if you assume that your samples are independently drawn from a certain distribution, it doesn't mean that you will always get the same pattern of samples. In other words, you can still have variability in your samples, and this also depends on the distribution you sample from.
To answer your question more directly, there's a chance that your training data and your validation data don't necessarily have the same patterns, even if the independence assumption holds. Therefore, the training and validation trends (and I assume you mean e.g. the performance) are not necessarily the same, but, although this may also depend on the training method, I would say that they shouldn't be very different (if the assumption holds) because, intuitively, each sample should be as informative as any other sample (independence assumption).

Do we work with the fact that there is a certain degree of wrongness to the assumption or am I interpreting it wrongly?

It is often convenient to make the i.i.d. assumption even, if it doesn't hold, for several reasons:

your training procedure may converge faster (because, intuitively, each sample will be as informative as any other sample)
your models may be simpler (e.g., in naive Bayes, you make the i.i.d. assumption only to simplify the model and, in general, the mathematical formulations)

Sometimes, if it doesn't hold, your training procedure can be highly affected. In those cases, you can find workarounds and try to make it hold. For example, the usage of the experience replay in deep Q-learning is an example of a trick used to overcome the dependence of successive samples, which causes learning to be highly variable. See this question Why exactly do neural networks require i.i.d. data?.
The answers to the question On the importance of the i.i.d. assumption in statistical learning on CrossValidated provide more information and details, so you may want to have a look at it too. Here's another answer, which is related to shuffling and how it can or not make the independence assumption hold, that I highly recommend that you read.

Regardless of the loss taking slightly different values, wouldn't the loss trend indicating how loss is decreasing be roughly the same for both curves? They would be different, but is it reasonable to expect them to tend to each other as batch size increases?

@ashenoy I say this in my answer. I think the answer to your question is "yes", although I don't have a formal proof for this.


text:
Which simulation platform is used by DeepMind (and others) to handle inverse kinematics musculoskeletal?
Which simulation platform is used by DeepMind and others to handle inverse kinematics musculoskeletal simulation, etc., for reinforcement learning simulations and agents?
I thought they use Unity or Unreal but I assume that would be resource-heavy.

I don't think there's one answer. For example, OpenAI developed their Gym for this purpose.

@Andy makes sense. So are these all custom developed? I wished someone could have had done some study of pro's and con's for these such as Gazebo, Cinder etc. I am quite eager to adapt the platform that is lightweight (not too many libraries) and highly controlable instead of having to deal with a framework.

I'm not sure, sorry, and a quick search didn't turn up much. What you're looking for is a survey of reinforcement learning toolkits, I think.

Welcome to SE:AI!

@gfdsal You could try to ask this as a new question on our site. For example, you could ask "what are the pros and cons of simulators, such as Gazebo, Cinder, etc., for reinforcement learning tasks?" or something like that. Try to be specific and clear!

@nbro, I shall be specific and clear in my next question.

DeepMind used MuJoCo (see also the related paper MuJoCo: A physics engine for model-based control) for the simulations, as they stated in section 3.1 of their paper Emergence of Locomotion Behaviours in Rich Environments (2017), which is the paper you should read to know more about their results related to those animations of skeletons that try to walk or jump (but do it weirdly).

This makes sense. I was told that Gazebo is one of the most used simulators in robotics as it has numerous builtin plugins for handing sensor data, noise. I am just confused why DeepMind and OpenAI and others develop their own version when there is something that is lot more versatile already exists. Gazebo even have solver for wind dynamics. I shall ready the paper you added to see key advantage of MuJoCo

@gfdsal Honestly, I don't know right now. I could investigate that later, though.


text:
How is the convolution layer is usually implemented in practice?
Following an earlier question, I'm interested in understanding the basics of Conv2d and especially how the kernel is applied, summed, and the propagated.
I understand that:

a kernel has size W x H and that more than one kernel is applied (e.g., S x W x H) where S is the amount of kernels.
A stride or step is used when iterating the network input
Padding may or may not be used by being added to the network input.

What I would ideally like to see is either a description, or a python sample (pytorch or tensorflow) of how that is done, what the dimensionality of the output is, and any operation I may be missing (some YouTube videos say that the kernel is summarised and then divided to hold one new unique value representing the feature activation?)

It's not clear to me if you're asking for a general description of how the convolution layer works or just some particular detail, like how to calculate the shape of the output of a convolution layer.

@nbro I'd say I'm asking for the implementation, either in a math level, or a pseudo code level, or even better as a minified example? The basics of Conv2D and how that works on the input and how the output is produced. I understand there are other parts and many variants.

Well, the implementation may be different than the actual general idea. So, I suggest you edit your post to clarify that you're specifically looking for "how the convolution layer is usually implemented in practice"

@nbro done :-) Only reason I'm asking for the implementation is because it tends to be more easily digestible than the theory more often than not.

I don't think that to understand convolution you need to dig into the nested code of huge libraries, since the code becomes quickly really hard to understand and convoluted (ba dum tsss!). Joking apart, in PyTorch Conv2d is a layer that applies another low level function, conv2d, written in c++.
Luckily enough, the guys from PyTorch wrote the general idea of how convolution is implemented in the documentation:

From this paragraph, we already have some important information like the input and output dimensions. The number of channels should be easy to understand, if we have an RGB image, for example, the channels are 3, one for each color, so they are just different matrices representing different features.
The next important element is the reference to the cross correlation, the function applied to our input images through the kernel k. Why cross-correlation? Because it is almost identical to a convolution, as you can see comparing their formulas:

The only difference consists in the way the indexing of the width is implemented, which causes the operation to start from the bottom right of the input matrix for the convolution and from the top left of the input image from the cross-correlation (the circles in the squares in the previous pic). Since, in most programming languages, matrix indexing starts from top left, cross-correlation is the most common choice to implement.
But how do these formulas work in practice? Here's another picture taken from Chapter 9 of Deep Learning (Goodfellow, Bengio, Courville), which I strongly suggest you to read.

Basically, from the input matrix, a submatrix is extracted, with the same dimension of the kernel, then the sub-matrix and the kernel are multiplied elementwise and all the resulting product summed together to produce a single output element that will form a 'pixel' of the resulting feature map (the output matrix).
Here's another example with fake numbers that I made. I hope the double notation for filter/kernel doesn't generate confusion, I actually found that sometimes it is inconsistent (in the chapter I linked they don't even use the filter at all). In practice they actually mean the same thing, I usually call kernel the actual matrix that is multiplied to the input and with filter I refer to the sliding window on the input image (that has, of course, must have the same dimension of the kernel).

Lastly, when you apply padding, the filter can actually move also outside the 'edges' of the input matrix, in which case all the elements outside would be considered to be zero. The computation is exactly the same, but since there are more splitting steps, the output matrix will have bigger dimension.

Please note that with multiple input channels you can perform either 2d convolution or 3d convolution, the difference relies in the filter dimension, in case of 2d convolution it would be a square whereas in 3d convolution it would be a cube. This means that for an RGB image a 2d convolution would treat each color layer independently, mixing the information from each channel with further computations like pooling (averaging the resulting feature maps of each color or select the max value among the feature maps for each pixel, etc..) while 3d convolution would mix together the color layers already during convolution thanks to the 3d kernel which will sum together elements from different layers.


text:
Why does the policy $\pi$ affect the Q value?
From my understanding, the policy $\pi$ is basically how the agent acts (i.e. the actions it will take in each state).
However, I am confused about the Q value and how it is "affected" by a policy. This answer says

$Q^\pi(s, a)$ is the action-value function. It is the expected return starting from state $s$, following policy $\pi$, taking action $a$. It's focusing on the particular action at the particular state.

From this, I infer that the $Q$ value (the action-value function) will be affected by the policy $\pi$. Why? So, why does the Q value change according to policy $\pi$?
Shouldn't the Q value be constant, because the same action taken in the same state will always give the same yield (and hence remain constantly good/bad)?
All the policy does is find out the max Q values and bases its policy on that information.

First of all, $Q_\pi(s, a)$ IS DEFINED AS the value (i.e. the expected return) of taking some action $a$ in some state $s$, AND THEN following some given policy $\pi$ (until e.g. the end of the game or your life). In other words, suppose that you take action $a$ in state $s$, AND THEN use the policy $\pi$ to behave in the world until you die, then $Q_\pi(s, a)$ would represent the value that you would obtain.
So, we are DEFINING $Q_\pi(s, a)$ in a certain way.  This is a DEFINITION! It's not an algorithm. In the algorithms (e.g. Q-learning), things will typically change, but that's a different story that you should investigate later.

From this, I infer that the $Q$ value (the action-value function) will be affected by the policy $\pi$.

So, $Q_\pi(s, a)$ will not keep changing. You could say that $Q_\pi(s, a)$ (which is a function) is "affected by" $\pi$ ONLY in the sense that it is "defined in terms of" $\pi$. To be precise, $Q_\pi(s, a)$ is actually an expectation (which is a mathematical concept similar to an ideal average). If you are not familiar with the concept of expectation, I suggest you get familiar with it first, before studying reinforcement learning.

Shouldn't the Q value be constant, because the same action taken in the same state will always give the same yield (and hence remain constantly good/bad)?

Again, there's the distinction between the algorithm that you actually use to find the function $Q_\pi(s, a)$ and the definition of the same function. In case you are estimating the function with an algorithm, you will not necessarily find "constant Q values". It depends on different aspects, which I would like to avoid discussing here, so that this post doesn't become an open discussion (I suggest you first learn about the basic Bellman equations and then you study the algorithms from the book Reinforcement learning: an introduction by Sutton and Barto).

Ok, Q is the reward associated with being in a given state, following a certain action and then following the given policy.
You need to take the expectation of the sum the immediate reward plus the value function which is defined by the policy.


text:
What are the pros and cons of studying machine learning before deep learning?
I'm a biotech student and I'm currently working on single-particle tracking. For my work, I need to use aspects of deep learning (CNN, RNN and object segmentation) but I'm not familiar with these topics. I have some prior knowledge in python.
So, do I have to learn machine learning first before going into deep learning, or can I skip ML?
What are the pros and cons of studying machine learning before deep learning?

I totally agreed with Oliver Mason.
Buy the way I suggest you this article "Introduction to Machine Learning". This article can also be useful for a beginner.

That question doesn't really make sense: deep learning is a sub-topic of machine learning, so you can't really 'skip' it. It's a bit like "I want to learn about trigonometry, but do I need to do geometry first?"
Having said that, in order to make sense of deep learning you should really know about the general principles of machine learning, otherwise you won't understand it. Or, more importantly, you won't understand what problems deep learning can be applied to, and what issues are better solved with other methods.
You don't need to go into much detail, but should at least get an overview.

Like Oliver Mason mentioned, Deep learning is just a sub-field of machine learning. In order to learn deep learning effectively you need to have certain pre-requisites like basic principle of Machine learning and basics of simple Artificial neural network  with some programming knowledge ( Python is go-to language). That being said, you don't need to know every single Machine learning algorithm and it's practices.
Now if deep learning happens to be just a tool that you need for this particular project and have no time to learn in depth about it then I would recommend you to take a look at python libraries like Tensorflow, pytorch, scikit learn, scipy, open cv etc. You can get started and use DL, ML models with these and many other libraries without knowing it's under the hood algorithms and implementations.
One of the best course to get started with deep learning with very little Ml knowledge is Andrew ng's deep learning.ai course on coursera ( you can audit the course and get all the course materials for free)
Here's the link to the course : Deep learning.ai

Machine learning uses algorithms to digest data sets, draw conclusions based on analyzed data, and use these conclusions to complete the task in the most effective way. This ability is a fundamental difference between machine learning and machine that has been programmed from the beginning with a certain sequence of commands. Machine learning has the capability to accomplish tasks dynamically.
While Deep Learning is one of the methods of implementing machine learning that aims to mimic the workings of the human brain using ANN. Deep learning uses a number of algorithms as 'neurons' to work together in determining and digesting certain characteristics in a data set.
In contrast to general machine learning programs that are designed to accomplish certain tasks, deep learning programs are usually programmed with more complex capabilities to study, digest, and classify data.
A machine learning model requires data to learn and obtain parameter estimates, so the more data that can be used, the machine learning program will be smarter. In addition, operating machine learning models — especially logical networks for deep learning — requires high computational power. This is because the deep learning model must operate many processes simultaneously, especially in the training phase. In the training phase, the machine learning model must process very large amounts of data to be categorized as a reference.

So, do I have to learn machine learning first before going into deep learning, or can I skip ML?

Quoting from Wikipedia, "Deep learning (also known as deep structured learning or differential programming) is part of a broader family of machine learning methods based on artificial neural networks with representation learning."
That being said, it is better to understand the fundamental of machine learning first so you would be able to understand completely of how deep learning works, and how to apply it effectively and efficiently. However, You can always skip straight to deep learning without having any major issues, as there is already a lot of libraries supporting deep learning on python such as TensorFlow.

What are the pros and cons of studying machine learning before deep learning?

pros:
Since deep learning is a subset of machine learning, having fundamental knowledge about machine learning and the other machine learning algorithms will be beneficial.
cons:
You might (not) waste your time and energy to learn something you wouldn't use.


text:
Why do we update all layers simultaneously while training a neural network?

Very deep models involve the composition of several functions or layers. The gradient tells how to update each parameter, under the assumption that the other layers do not change. In practice, we update all of the layers simultaneously.

The above is an extract from Ian Goodfellow's Deep Learning  - which talks about the need for batch normalization.
Why do we update all the layers simultaneously? Instead, if we update layers one at a time during backpropagation - it will eliminate the need for batch normalization, right?
Reference: A Gentle Introduction to Batch Normalization for Deep Neural Networks
P.S. The attached link says: Because all layers are changed during an update, the update procedure is forever chasing a moving target. Apart from the main question, it would be great if someone could explain why exactly a moving target is being referred to in the above sentence.

Two years after this question, Google has now published a paper about doing backpropagation layer-by-layer, caching the "target" outputs of each layers. It does indeed seem a lot more efficient: https://ai.googleblog.com/2022/07/enhancing-backpropagation-via-local.html

Why do we update all layers simultaneously while training a neural network?

We typically train a neural network with gradient descent and back-propagation. Gradient descent is the iterative algorithm used to update the parameters and back-propagation is the algorithm used to compute the gradient of the loss function with respect to each of these parameters.
Let's denote a vector that contains all learnable parameters of a neural network $M$ by $\mathbf{w} = \left[w_1, \dots, w_n \right] \in \mathbb{R}^n$ (so $M$ contains $n$ learnable parameters), the loss function of $M$ by $\mathcal{L}$,  the gradient of the loss function with respect to each parameter $w_i$ of $M$ by $ \nabla \mathcal{L} = \left[ \frac{\partial \mathcal{L}}{\partial w_1}, \dots, \frac{\partial \mathcal{L}}{\partial w_n}  \right] \in \mathbb{R}^n$, then the gradient descent step to update all parameters is
\begin{align}
\mathbf{w} \leftarrow \mathbf{w} - \gamma * \nabla \mathcal{L} \tag{1} \label{1}
\end{align}
where $\gamma \in \mathbb{R}$ is the learning rate.
In equation \ref{1}, we are assigning to $\mathbf{w}$ the value $\mathbf{w} - \gamma * \nabla \mathcal{L}$, so we are updating all parameters $\mathbf{w}$ simultaneously, so we are also updating all layers simultaneously.
In principle, you could update each parameter $w_i$ individually. To be more precise, you would have the following update rule
\begin{align}
w_i \leftarrow w_i - \gamma * \frac{\partial \mathcal{L}}{\partial w_i}, \; \forall i\tag{2} \label{2}
\end{align}
So, you could update first $w_1$, then $w_2$, and so on.
Actually, you don't need to update the parameters sequentially (also because there is no real order of the parameters). You can actually update them in any other way. You can update the parameters in any way because, although the computation of the gradient highly depends on the structure of the neural network (so if you change the structure, the computation of the gradient also changes), once the gradient is computed, you already have all information to update each parameter independently of each other.
You typically update all parameters (or layers) simultaneously because, in practice, you work with vectors and matrices rather than scalars in order to take benefit from efficient matrix multiplication algorithms and hardware (i.e. GPUs).

After got the gradient $ \nabla \mathcal{L} = \left[ \frac{\partial \mathcal{L}}{\partial w_1}, \dots, \frac{\partial \mathcal{L}}{\partial w_n}  \right] $, the BP algorithm should update weights $w_1, w_2, ... w_n$ at next step. For example, after update $w_n$ , the $\frac{∂L}{∂w_{n-1}}$ may not still the best "direction" for update $w_{n-1}$. I'm not sure am I correct. If true, why not re-calculate the partial derivative from $w_{n-1}$ to $w_1$ then update $w_{n-1}$?


text:
Is my understanding of the value function, Q function, policy, reward and return correct?
I'm a beginner in the RL field, and I would like to check that my understanding of certain RL concepts.
Value function: How good it is to be in a state S following policy π.

So, the value functions here are 0.3 and 0.9

Q function(also called state-action value, or just action value): How good it is to be in a state S and perform action A while following policy π. It uses reward to measure the state-action value

So, the state-action values here are 0.03,0.02,0.5 and 0.9

Q value: The overall expected rewards after performing action A in state S, and continuing with policy π until the end of the episode. So, essentially I can only calculate the Q value if I know all the state-action values of the actions I will be taking in the single episode.(Because the Q value takes into account the actions after the current action A, till the end of the episode, following policy π)
Reward: The metric used to tell the agent how good/bad it's action was. It is a constant value.
For e.g
 1. Fall in pond --> -1
 2. On stone path --> +1
 3. Reach home--> +10

Return: The sum of rewards in a single episode
Policy π: A set of specific instructions an agent will follow in an episode. For example, the policy will look like:
In state 1, take action 3 ( which takes me to state 2)

In state 2, take action 2 ( which takes me to state 3)

In state 3, take action 1 ( Which takes me to state 4)

In state 4, take action 2 ( Which takes me to terminal state)

1 episode completed

And my policy will keep updating each episode to get the best return

What is the name of this book that you are using?

Value function: How good it is to be in a state $s$ following policy $\pi$.

There are different value functions. There's the state value function, often denoted as $v(s)$ (or $V(s)$), so it's a function of only one variable, i.e. $s$ (a state). There's the state-action value function $q(s, a)$ (or $Q(s, a$)). A value function is a function, so it's not a number or a vector, or whatever. It's a function, so it maps inputs to outputs. In the first case, it maps states to real numbers. In the second case, it maps states and actions to real numbers. So, we could denote the state value function as $v : \mathcal{S} \rightarrow \mathbb{R}$ (where $\mathcal{S}$ is the set of states in your environment) and state-action value function as $q : \mathcal{S} \times \mathcal{A}\rightarrow \mathbb{R}$ (where $\mathcal{A}$ is the set of actions and $\times$ means "combination of").
So, your definition of a value function is not quite correct. The value function $v(s)$ doesn't represent "how good it is to be in a state $s$ following a policy $\pi$", but "how good it is to be in a state $s$ AND THEN following policy $\pi$". To emphasize this, you often use the notation $v_{\pi}(s)$ rather than simply $v(s)$.
See What are the value functions used in reinforcement learning? for more details about existing value functions in reinforcement learning. And to see the full definition of the value functions, I suggest you read Sutton and Barto's book.

Q function (also called state-action value, or just action value): How good it is to be in a state $s$ and perform action $a$ while following policy $\pi$. It uses reward to measure the state-action value

As I said above, the $q$ function is a "value function" too. It's just a different value function than $v$.
Again, the same thing I said for $v$ also applies here, so "how good it is to be in a state $s$ and perform action $a$ while following policy $\pi$" is incorrect for the same reason your definition for $v$ was incorrect. The $q$ function can be defined as "how good it is to be in a state $s$ and take action $a$, AND, AFTER THAT, follow a given policy $\pi$. Again, to emphasize that $q$ is defined in terms of $\pi$, we often use the notation $q_\pi$.

Reward: The metric used to tell the agent how good/bad it's action was. It is a constant value.

This is roughly correct, but the reward doesn't have to be constant and it depends on your problem. Also, there's also the related notion of "reward function", which is the function that assigns rewards to each action. So, when defining your problem as a Markov decision process, you need to define this reward function. Actually, this is probably the most important function in reinforcement learning (because this is the way you teach the agent to behave).

Return: The sum of rewards in a single episode

This is roughly correct. However, note that the sum can also be a "weighted sum".

Policy: A set of specific instructions an agent will follow in an episode.

This is roughly correct, but a policy can also have some randomness in it. For example, if you are in state $s$, your policy could say "always take action $a_i$", but another policy could say "take action $a_i$ with probability $p$ and action $a_j$ with probably $1 - p$. Also, note that the policy is not restricted to an episode. It's a general function that tells the agent how to behave independently of the episode.
(Sorry, I didn't look at your examples. Maybe I will review this answer later to look at your examples too, but the information in this answer should already tell you if your examples are correct or not).

I think most of it is correct.

Q function(also called state-action value, or just action value): How good it is to be in a state S and perform action A while following policy π. It uses reward to measure the state-action value

This is a bit off. Q function basically tells you how good it is to be in state S and perform action A, and follow policy $\pi$ from the next state onwards. The action A that you take can be any action from the action space and need not be according to the policy $\pi$.
Also, I think Q-function and Q-value are mostly used interchangeably to mean the same thing.

Ah, so my action A can be different , which will affect the subsequent effectiveness policy π?  (e.g my policy π requires the state to be 1 to be effective, but when I take action A it takes me to state 3 instead which means my policy isn't as effective. Hence, the Q value of this action(to state 3) will be relatively low)


text:
Why does the definition of the reward function $r(s, a, s')$ involve the term $p(s' \mid s, a)$?
Sutton and Barto define the state–action–next-state reward function, $r(s, a, s')$, as follows (equation 3.6, p. 49)
$$
r(s, a, s^{\prime}) \doteq \mathbb{E}\left[R_{t} \mid S_{t-1}=s, A_{t-1}=a, S_{t}=s^{\prime}\right]=\sum_{r \in \mathcal{R}} r \frac{p(s^{\prime}, r \mid s, a )}{\color{red}{p(s^{\prime} \mid s, a)}}
$$
Why is the term $p(s' \mid s, a)$ required in this definition? Shouldn't the correct formula be $\sum_{r \in \mathcal{R}} r p(s^{\prime}, r \mid s, a )$?

Expectation of reward after taking action $a$ in state $s$ and ending up in state $s'$ would simply be
\begin{equation}
r(s, a, s') = \sum_{r \in R} r \cdot p(r|s, a, s')
\end{equation}
The problem with this is that they do not define probability distribution for rewards separately, they use joint distribution $p(s', r|s, a)$, which represents probability for ending up in state $s'$ with reward $r$ after taking action $a$ in state $s$. This probability can be separated in 2 parts using product rule
\begin{equation}
p(s', r|s, a) = p(s'|s, a)\cdot p(r|s', s, a)
\end{equation}
which represents the probability for getting to state $s'$ from $(s, a)$, and then probability for getting reward $r$ after ending up in $s'$.
If we define reward expectation through the joint distribution, we would have
\begin{align}
r(s, a, s') &= \sum_{r \in R} r \cdot p(s', r|s, a)\\
&= \sum_{r \in R} r \cdot p(s'|s, a) \cdot p(r|s', s, a)
\end{align}
but this would not be correct, since we have this extra $p(s'|s, a)$, so we divide everything by it to get expression with only $p(r|s', s, a)$.
So, in the end we have
\begin{equation}
r(s, a, s') = \sum_{r \in R} r \frac{p(r, s'|s, a)}{p(s'|s, a)}
\end{equation}

$\frac{p(s', r \mid s, a)}{p(s' \mid s, a)}$ represents the probability of observing reward $r$ in state $s'$, given that state $s'$ is the next state transitioned to. The equation assumes a probability distribution of rewards $r$ over state $s'$, meaning that a different reward might be observed whenever a state transitions from $s$ to $s'$. In most cases, if $r(s, a, s')$ is a deterministic reward then $p(s', r \mid s, a) = p(s' \mid s,a )$.

But probability of observing reward r in state s', given that state s' is the next state transitioned to can be given by: Σ p(s',r|s,a) over set of all actions. I am not able to understand how these two expressions are equivalent.

Actions in the case of the equation above is fixed rite ? $A_{t-1}$ = a. The equation does not talk about sum over all set of actions, but that a single action can lead to multiple rewards observed for state $s'$

Yes, in the question, the action is fixed but in the first line of your answer you wrote that the fraction represents probability of observing reward r in state s', given that state s' is next state transitioned to. I was referring to that.

In fact that is what my question was. Since, action is fixed the summation in Σ p(s',r | s,a) reduces simply to p(s',r | s,a), then why divide it by something?

I see what you mean. Maybe you could consider the example whereby $p(s'|s,a) = 0.5$ and there are 2 rewards to be observed, $r_1$ and $r_2$, where $p(s',r_1|s,a) = 0.3$ and $p(s',r_2|s,a) = 0.2$. If u calculate expectations without dividing by the denominator, you would get $E(R_t) = 0.3R_1 + 0.2R_2$, which is not correct because in actual fact, you would observe $r_1$ about 0.6 of the time and $r_2$ 0.4 of the time when you transition to state $s'$ from state $s$, action $a$.


text:
Understanding relation between VC Symmetrization Lemma and Generalization Bounds
I am new in the field of Machine Learning so I wanted to start of by reading more about mathematics and history behind it.
I am currently reading, in my opinion, a very good and descriptive paper on Statistical Learning Theory - "Statistical Learning Theory: Models, Concepts, and Results". In section 5.5 Generalization bounds, it states that:

It is sometimes useful to rewrite (17) "the other way round". That is, instead of ﬁxing $\epsilon$ and then computing the probability that the empirical risk deviates from the true risk by more than $\epsilon$, we specify the probability with which we want the bound to hold, and then get a statement which tells us how close we can expect the risk to be to the empirical risk. This can be achieved by setting the right-hand side of (17) equal to some $\delta > 0$, and then solving for $\epsilon$. As a result, we get the statement that with a probability at least $1−\delta$, any function $f \in F$ satisﬁes

Equation (17) is VC Symmetrization lemma to which we applied union bound and then Chernoff bound:

What I fail to understand is the part where we are rewriting (17) "the other way around". I fail to grasp intuitive understanding of relation between (17) and (18), as well as understanding generalization bounds in general.
Could anyone help me with understanding these concepts or at least provide me with additional resources (papers, blog posts, etc.) that can help?

Regarding the general idea of generalization bounds, generalization gap, etc., this answer I had provided several months ago could be useful.

@nbro Thank you! I really want to learn more about Generalization in DL and I think this will help a lot! If you have any more resources, please share. Once again, THANKS!

Let $\varepsilon$ in (17) is equal to $\sqrt{\frac{4}{n}\left(\log{(2\mathsf{N}(\mathcal{F},n))}-\log{\delta}\right)}$. We have:
$$
P\left(\sup_{f\in\mathcal{F}}|R(f)-R_{emp}(f)| > \sqrt{\frac{4}{n}\left(\log{(2\mathcal{N}(\mathcal{F},n))}-\log{\delta}\right)}\right) \leqslant
2\mathcal{N}(\mathcal{F},n) e^{\frac{-n}{4}\left(\frac{4}{n}\left(\log{(2\mathcal{N}(\mathcal{F},n))}-\log{\delta}\right)\right)} =
2\mathcal{N}(\mathcal{F},n) e^{\log{\delta} - \log{(2\mathcal{N}(\mathcal{F},n))}}
$$
As we know that $e^{\log{n}} = n$ (suppose that base of $\log$ here is $e$), we can write:
$$
P\left(\sup_{f\in\mathcal{F}}|R(f)-R_{emp}(f)| > \sqrt{\frac{4}{n}\left(\log{(2\mathcal{N}(\mathcal{F},n))}-\log{\delta}\right)}\right) \leqslant
2\mathcal{N}(\mathcal{F},n) \left(\frac{\delta}{2\mathcal{N}(\mathcal{F},n)}\right)
$$
Hence:
$$
P\left(\sup_{f\in\mathcal{F}}|R(f)-R_{emp}(f)| > \sqrt{\frac{4}{n}\left(\log{(2\mathcal{N}(\mathcal{F},n))}-\log{\delta}\right)}\right) \leqslant \delta
$$
As we know that if we have $P(x > a) \leqslant c$, we can conlude that $P( x<a) \geqslant 1-c$, we will have:
$$
P\left(\sup_{f\in\mathcal{F}}|R(f)-R_{emp}(f)| < \sqrt{\frac{4}{n}\left(\log{(2\mathcal{N}(\mathcal{F},n))}-\log{\delta}\right)}\right) \geqslant 1- \delta
$$
Now, as this inequality is true for supremum of set $\mathcal{F}$ and $R(f) -R_{emp}(f) \leqslant \varepsilon$ is subset of  $|R(f) -R_{emp}(f)| \leqslant \varepsilon$ (in terms of probability state space), we can say the following inequality is correct for any function $f$ with at least probability of $1-\delta$:
$$
R(f) \leqslant R_{emp}(f) + \sqrt{\frac{4}{n}\left(\log{(2\mathcal{N}(\mathcal{F},n))}-\log{\delta}\right)}
$$

The only part that is not clear and obvious is why you say that $R(f) -R_{emp}(f) \geqslant 0$ is a subset of $|R(f) -R_{emp}(f)| \geqslant 0$, i.e. you're saying that something greater or equal to zero (a number) is a subset of something else that is greater or equal to zero. Maybe you can clarify this part.


text:
What are some online courses on artificial general intelligence?
Although no artificial general intelligence (AGI) has yet been created, probably, there are already some courses on the topic. So, what are some online (preferably free) courses on AGI?

As far as I know, no AGI system has yet been created, so that's why there aren't yet many courses on AGI. However, there are a few courses that attempt to address AGI as the main topic but from different perspectives. Below, I will mention the ones that I found and partially followed, and give some info about them.
MIT 6.S099: Artificial General Intelligence
This is organized by Lex Fridman. It is a series of lessons and talks primarily given by a diverse set of guest appearances, such as

Josh Tenenbaum (researcher and professor in computational cognitive science),
Nate Derbinsky (who gives a lesson on cognitive architectures, Soar, etc.),
Stephen Wolfram (creator of Mathematica, Wolfram Alpha and the Wolfram Language; he talks about his work throughout the years, especially, the development of Wolfram Alpha and Language),
Marc Raibert (CEO of Boston Dynamics, who gives a lesson on his work at Boston Dynamics and the robots they have developed),
Lisa Feldman Barrett (professor of psychology, who gives a very insightful lesson on emotions and feelings, with possibly different perspectives),
Rosalind Picard (a professor at MIT, director of the Affective Computing Research Group at the MIT Media Lab, and co-founder of two companies, Affectiva and Empatica; in her talk with Lex, she talks about affective computing and other interesting concepts and issues)
Marcus Hutter (inventor of a mathematical theory of general intelligence, AIXI; in his talk with Lex, he talks about Occam's razor, Solomonoff induction, Kolmogorov complexity, the definition of intelligence, AIXI, rewards, bounded rationality and consciousness)

Knowledge-Based AI: Cognitive Systems
It's taught by Ashok Goel and David Joyner. As the name of the courses suggests, this course focuses on knowledge-based AI.
The Society of Mind
This course is taught by Marvin Minsky, who had also written a book and developed a theory of natural intelligence with the same name.
CS 294-149: Safety and Control for Artificial General Intelligence (Fall 2018)
I don't think that there are (free) recorded video lectures, but, in the lecture schedule section of the course, you have links to references that they use during the course (e.g. this paper), which seems to focus on safety and control aspects of future AGI systems. See also these notes.
General Theory of General Intelligence
This is not really a course, but it's a series of videos by Ben Goertzel, who summarises some of the topics presented in a paper that the same Ben Goertzel wrote about AGI.


text:
What is it called in AI when a program is designed to make "x in the style of y"?
Simplified: What is it called in AI when a program is designed to make "x in the style of y;" when it trains off of two types of sources in order to make a thing from source one, informed by features from source two? For example, if a network made up of two smaller networks were to take sheet music of a specific compositional style in network A and audio samples from a certain genre in B and through an interface creates music from a certain genre in a certain compositional style; the is comes from One, the seems comes from Two.
For more coarse and obvious examples:

"Compose synthpop in the style of Beethoven"
"Draw impressionism in the style of Mondrian"
"Generate casserole recipes using only ingredients most likely to fluctuate in price given current market data"
"Sketch baseballs that look like they're made of espresso foam"

Style Transfer

This problem is known as "style transfer".
The field was started by Gatsys et al. in 2016, and has seen a lot of work over the last few years, including conditional translation from Isola et al. in 2017 and unpaired translation from Zhu et al. 2017.
This picture (Figure 2 from Gatsys et al.) shows the idea behind style transfer, and illustrates the dramatic results that are possible with these models.


text:
Why does TensorFlow docs discourage using softmax as activation for the last layer?
The beginner colab example for tensorflow states:

Note: It is possible to bake this tf.nn.softmax in as the activation function for the last layer of the network. While this can make the model output more directly interpretable, this approach is discouraged as it's impossible to
  provide an exact and numerically stable loss calculation for all models when using a softmax output.

My question is, then, why? What do they mean by impossible to provide an exact and numerically stable loss calculation?

It's because of gradient computations: automatic differentiation will compute the gradient for each module and if you have a standalone crossentropy module the over all loss will be unstable (~1/x so it will diverge for small input values) whereas if you use a softmax + crossentropy module all-in-one, then it becomes numerically stable (y-p)
Slides from DeepMind's Simon Osindero lecture at UCL in 2016:

You say "if you have a standalone crossentropy module the over all loss will be unstable (~1/x)", but you don't really explain WHY. Maybe you can explain why using softmax is unstable.That's the actual question. Also, what does the CE have to do with the softmax?

The slides actually do explain why. If the question were about mathematical gradient derivations I'd agree with your comment, it's not. I'll add a few words though

There is a typo in the gradient on the second slide. Check here: https://xeonqq.github.io/machine%20learning/softmax/
The $i$-th element in the gradient should be $\frac{e^{x_i}}{\sum_m e^{x_m}} - p_i$.

This is also a question I stumble upon, thanks for the explaination from ted, it is very helpfull, I will try to elaborate a little bit. Let's still use DeepMind's Simon Osindero's slide:

The grey block on the left we are looking at is only a cross entropy operation, the input $x$ (a vector) could be the softmax output from previous layer (not the input for the neutral network), and $y$ (a scalar) is the cross entropy result of $x$. To propagate the gradient back, we need to calculate the gradient of $dy/dx_i$, which is $-p_i/x_i$ for each element in $x$.
As we know the softmax function scale the logits into the range [0,1], so if in one training step, the neutral network becomes super confident and predict one of the probabilties $x_i$ to be 0 then we have a numerical problem in calculting $dy/dx_i$.
While in the other case where we take the logits and calculate the softmax and crossentropy at one shot (XentLogits function), we don't have this problem. Because the derivative of XentLogits is $dy/dx_i = y - p_i$, a more elaborated derivation can be found here.


text:
How can I implement policy evaluation when reward is tied to an action outcome?
I'm following Stanford reinforcement learning videos on youtube. One of the assignments asks to write code for policy evaluation for Gym's FrozenLake-v0 environment.
In the course (and books I have seen), they define policy evaluation as
$$V^\pi_k(s)=r(s,\pi(s))+\gamma\sum_{s'}p(s'|s,\pi(s))V^\pi_{k-1}(s')$$
My confusion is that in the frozen lake example, the reward is tied to the result of the action. So, for each pair state-action, I have a list that contains a possible next-state, the probability to get to that next-state and the reward. For example, being in the target state and performing any action brings a reward of $0$, but being in any state that brings me to the target state gives me a reward of $1$.
Does this mean that, for this example, I need to rewrite $V^\pi_k(s)$ as something like this:
$$V^\pi_k(s)= \sum_{s'} p(s'|s,\pi(s)) [r(s,\pi(s), s')+ \gamma V^\pi_{k-1}(s')]$$

The renowned book Reinforcement Learning: An Introduction (2nd edition), by Sutton and Barto, provides a different update rule than your first update rule for policy evaluation. Their update rule is more similar to your second update rule. See section 4.1. They also provide the pseudocode for policy evaluation on page 75 of the book. You can also find the pseudocode here.
Moreover, note that the update rule doesn't need to change only because the rewards are tied to the outcome of an action. This information is encoded in the functions $p$ (the transition function) and $r$ (the reward function) of the Markov decision process, which is incorporated in the update rule. If you want to understand the update rule, you should read the relevant pages (especially, chapter 4) of the cited book.


text:
How does being on-policy prevent us from using the replay buffer with the policy gradients?

One of the approaches to improving the stability of the Policy
Gradient family of methods is to use multiple environments in
parallel. The reason behind this is the fundamental problem we
discussed in Chapter 6, Deep Q-Network, when we talked about the
correlation between samples, which breaks the independent and
identically distributed (i.i.d) assumption, which is critical for
Stochastic Gradient Descent (SDG) optimization. The negative
consequence of such correlation is very high variance in gradients,
which means that our training batch contains very similar examples,
all of them pushing our network in the same direction. However, this
may be totally the wrong direction in the global sense, as all those
examples could be from one single lucky or unlucky episode. With our
Deep Q-Network (DQN), we solved the issue by storing a large amount of
previous states in the replay buffer and sampling our training batch
from this buffer. If the buffer is large enough, the random sample
from it is much better representation of the states distribution at
large. Unfortunately, this solution won't work for PG methods, at most
of them are on-policy, which means that we have to train on samples
generated by our current policy, so, remembering old transitions is
not possible anymore.

The above excerpt is from Maxim Lapan in the book Deep Reinforcement Learning Hands-on page 284.
How does being on-policy prevent us from using the replay buffer with the policy gradients? Can you explain to me mathematically why we can't use replay buffer with A3C for instance?

I answered similar question here you can take a look

Let's say your old policy is $\pi_b$ and your current one is $\pi_a$. If you collected trajectory by using policy $\pi_b$ you would get return $G$ whose expected value is
\begin{align}
E_{\pi_b}[G_t|S_t = s] &= E_{\pi_b}[R_{t+1} + G_{t+1}]\\
&= \sum_a \pi_b(a|s) \sum_{s', r} p(s', r|s, a) [r + E_{\pi_b}[G_{t+1}|S_{t+1} = s']]\\
\end{align}
You can see if you write out this recursively that this expectation depends on $\pi_b(a|s), \pi_b(a'|s'), \ldots$
If you collect trajectory with policy $\pi_a$ you would get expected return that depends on $\pi_a(a|s), \pi_a(a'|s'), \ldots$ Since these are two different policies then $\pi_b(a|s) \neq \pi_a(a|s)$ for some $(s, a)$. That would mean that returns have different expected values and are sampled through different distributions. You cannot then use some return $G$ sampled by following policy $\pi_b$ to update policy $\pi_a$ because it's not sampled according to the proper distribution, if we did, we would be updating policy $\pi_a$ with biased gradient update that does not reflect how policy $\pi_a$ performed.

Or in more plain, non-mathematical terms; if you're traing on old examples stored in some replay buffer, which were created by another / an older policy, you're by definition doing off-policy learning!

Ok. I am working with the Max Lapan's book, i.e. Deep Reinforcement Learning Hands-on on the subject of A2C. I tried to build my own implementation of the Atari Pong game using that algorithm and based on Max code, but my code is not converging at all. I'm struggling on that converging issue for a week now.  I think there's a very subtle error in the code, but I can't say what it is. Here is the question: https://ai.stackexchange.com/q/21187/2444. Can you help me with that?

Both code are running in a different Colab Env. You can probably see a very significant difference in the results ... right now.

I'm not sure if I will have time to look into code and I'm also not too familiar with Pytorch. If the other code is similar and it works try to change those lines that are different one portion at the time to try to spot the error. So my advice would be to try to isolate portions of the code and make sure they work and that they provide same behavior in both cases. To do that easier, you can also set the random seed so that you always initialize with the same parameters in both cases.

Can you develop what you mean by "you can also set the random seed so that you always initialize with the same parameters in both cases" ?

for example in numpy you can call numpy.random.seed and your random numbers will always follow same pattern every time you run the program.


text:
Why can't pure KG embedding methods discover multi-hop relations paths?
According to Reinforcement Knowledge Graph Reasoning for Explainable Recommendation

pure KG embedding methods lack the ability to discover multi-hop relational paths.

Why is it so?

To put this insert to context, we should take at least this much of text from the paper:

One line of research focuses on making recommendations using knowledge graph embedding models, such as TransE [2] and node2vec [5]. These approaches align the knowledge graph in a regularized vector space and uncover the similarity between entities by calculating their representation distance [30]. However,pure KG embedding methods lack the ability to discover multi-hop relational paths.

In my understanding the pure KG embedding here refers to TransE and node2vec solutions. To learn more about those, we should read links [1] and [2]. From [1]:

Usually, we use a triple (head, relation, tail) to represent a knowledge. Here, head and tail are entities. For example, (sky tree, location, Tokyo). We can use the one-hot vector to represent this knowledge.

Later on same source there is an explanation on the end of definition section of TransE solution:

But this model only can take care with one-to-one relation, not suitable for one-to-many/many-to-one relation, for example, there is two knowledge, (skytree, location, tokyo) and (gundam, location, tokyo). After training, the 'sky tree' entity vector will be very close with 'gundam' entity vector. But they do not have such similarity in real.

On other hand, [3] tells:

Knowledge graphs as described above represents a static snapshot of our knowledge. It does not reflect the process of it’s how the knowledge built up. In the real world, we learn by observing temporal patterns. While it’s possible to learn the similarity between nodes A and node B, it will be hard to see the similarity between node A and node C as it was 3 years ago.

Solution in paper states "--,each recommended item is associated with around 1.6 reasoning paths." which is supposedly impossible for TransE solution.
So, knowledge graph, purely, is a static snapshot that can identify by great accuracy one to one findings. Actually according what [2] tells how node2vec works, they can describe and combine also more information (node2vec combines different types of similarities at same time), but anyways I think the main point is actually in one word in the citation: discover!
The model suggested on paper adds Reinforcement Learning principles to the KG modeling, so to say, the pure KG embedding always tells one distance based statistical solution, but a RL based solution may learn more aspects behind the scenes as it learns by try and error more complex paths underlying in the behavior.
On the other hand, the paper says, when relating their solution to pure TransE:

It can be regarded as a single-hop latent matching method, but the post-hoc explanations do not necessarily reflect the true reason of generating a recommendation. In contrast, our methods generate recommendations through an explicit path reasoning process over knowledge graphs, so that the explanations directly reflect how the decisions are generated, which makes the system transparent.

So, whether TransE and those could actually recommend things in the given environment, the recommendation reasoning paths may stay obscure.
Sources:
[1] https://towardsdatascience.com/summary-of-translate-model-for-knowledge-graph-embedding-29042be64273
[2] https://towardsdatascience.com/node2vec-embeddings-for-graph-data-32a866340fef
[3] https://towardsdatascience.com/extracting-knowledge-from-knowledge-graphs-e5521e4861a0


text:
Are there examples of neural networks (used for control) implemented on a FPGA or on a neurochip?
Greetings to all respected colleagues!
I want to consult on the use of FPGAs and neurochips. I plan to use it in my laboratory project for programming control systems on neural networks.
In my work, there are a lot of applications of neural networks, and I became interested in their programming on FPGAs and neurochips.
But I don’t know a single example of a really made and working laboratory prototype in which a neural network is implemented on an FPGA or on a neurochip and controls something. If someone shares the link, I would carefully study it.

Because I'm a beginner, I need a good tutorial.

I have never previously used either a FPGA or neurochips. Therefore, I do not know how to program on them.

Good start for me would be a tutorial on programming in FPGA or Neurochip of neural network control of motor speed.

Assumption
In this answer it is assumed that with "neurochips" you mean chips made (using neuromorphic engineering) for neuromphic computing.
Related example
From what I currently understand from this article neuromorphic chips, in particular the TrueNorth chip, are being used (or emulated) for embedded systems related signals processing.
Doubt
The signal processing performed by these (emulated) neuromorphic chips might be part of real-life applications involving control systems.


text:
How is the incremental update rule derived from the weighted importance sampling in off-policy Monte Carlo control?
Here's the approximated value using weighted importance sampling
$$
V_{n} \doteq \frac{\sum_{k=1}^{n-1} W_{k} G_{k}}{\sum_{k=1}^{n-1} W_{k}}, \quad n \geq 2
$$
Here's the incremental update rule for the approximated value
$$V_{n+1} \doteq V_{n}+\frac{W_{n}}{C_{n}}\left[G_{n}-V_{n}\right], \quad n \geq 1$$
How is the second equation derived from the first?
These are used for the weighted importance sampling method of off-policy Monte Carlo control.

By definition of $V_{n+1}$, we have:
$V_{n+1} = \frac{\sum_{k=1}^{n} W_{k} G_{k}}{\sum_{k=1}^{n} W_{k}} \; \tag{1}$
Then, taking the $n^{th}$ term out of the sum in the numerator, we have:
$V_{n+1} = \frac{W_{n}G_{n} \; + \; \sum_{k=1}^{n-1} W_{k} G_{k}}{\sum_{k=1}^{n} W_{k}} \; \tag{2}$
Then, from the definition of $V_n$, $V_{n} = \frac{\sum_{k=1}^{n-1} W_{k} G_{k}}{\sum_{k=1}^{n-1} W_{k}}$, we have:
$\sum_{k=1}^{n-1} W_{k} G_{k} = V_{n}*\sum_{k=1}^{n-1} W_{k} \; \tag{3}$
Then, substituting $(3)$ in the numerator of $(2)$, we get:
$V_{n+1} = \frac{W_{n}G_{n} \; + \; V_{n}*\sum_{k=1}^{n-1} W_{k}}{\sum_{k=1}^{n} W_{k}} \; \tag{4}$
Then, adding and subtracting $V_{n}W_{n}$ in the numerator of $(4)$, we obtain:
$V_{n+1} = \frac{W_{n}G_{n} \; + \; V_{n}*\sum_{k=1}^{n-1} W_{k} \; + \; V_n W_n \; - \; V_n W_n}{\sum_{k=1}^{n} W_{k}} \; \tag{5}$
We factor $V_n$ in the numerator of $(5)$:
$V_{n+1} = \frac{W_{n}G_{n} \; + \; V_{n}(W_n \; + \; \sum_{k=1}^{n-1} W_{k}) \; - \; V_n W_n}{\sum_{k=1}^{n} W_{k}} \; \tag{6}$
We simplify, taking into account that the denominator $\sum_{k=1}^{n} W_{k} = W_n + \sum_{k=1}^{n-1} W_{k}$, we get:
$V_{n+1} = V_n + \frac{W_n G_n - W_n V_n}{\sum_{k=1}^{n} W_{k}} \; \tag{7} $
Further rearrangements of the terms give us:
$V_{n+1} = V_n + \frac{W_n}{\sum_{k=1}^{n} W_{k}}[G_n - V_n] \; \tag{8}$
Finally, by definition of $C_n$ as the cumulative sum of the weights up to time $n$, we get the desired incremental update equation:
$V_{n+1} = V_n + \frac{W_n}{C_n}[G_n - V_n] \; \tag{9}$


text:
What is the difference between deep learning and shallow learning?
What is the difference between deep learning and shallow learning?
What I am interested in knowing is not the definition of deep learning and shallow learning, but understanding the actual difference.
Links to other resources are also appreciated.

Where did you find the term "shallow learning"? It's not a common term.

Found Here. Googled the Term. Thought I would get a better understanding of Shallow Learning here

A Comparison of Shallow and Deep Learning Methods for Predicting Cognitive Performance of Stroke Patients From MRI Lesion Images

That article only mentions "shallow learning" in the title and it mentions "shallow" to refer to the fact that deep learning models are not really learning any "deep" concepts, where "deep" here means "philosophically deep". So, I think the title is just (fairly?) provocative.
Currently, in machine learning, the expression "shallow learning" isn't really standardized, as opposed to deep learning, which refers to learning, with gradient descent and back-propagation, from (typically) huge amounts of data with neural networks. Nevertheless, "shallow learning" may occasionally refer to everything that isn't deep learning (e.g. traditional machine learning models, such as support vector machines), but most likely it refers to learning in neural networks with only a small number (0-2) of hidden layers (i.e. non-deep neural networks).
Note that the difference between deep and shallow neural networks isn't really clear. Some people may consider neural networks with only 1-2 hidden layers already deep, while others may consider only neural networks with e.g. 5-10 hidden layers deep. This also shows that deep learning isn't actually well-defined too.
The other linked article actually says

CNN performance was compared to that of conventional (shallow) machine learning methods, including ridge regression (RR) on the images’ principal components and support vector regression.

So, in this article, they use "shallow learning" to refer to traditional (or conventional) machine learning models, which confirms what I said above.


text:
What are some conferences for publishing papers on graph convolutional networks?
What are some conferences for publishing papers on graph convolutional networks?

Based on past publications, here are some journals and conferences where you can possibly publish or present a research paper on geometric deep learning or graph neural networks

Neural Information Processing Systems (NIPS)
International Conference on Learning Representations (ICLR)
Conference on Computer Vision and Pattern Recognition (CVPR)
International Conference on Machine Learning (ICML)

The website http://geometricdeeplearning.com/ also provides some information about the topic and links you to several workshops, papers, and tutorials.
Here are some links to some of the past workshops on GDL.

Representation Learning on Graphs and Manifolds (and the related homepage)
Workshop IV: Deep Geometric Learning of Big Data and Applications
Geometric Deep Learning on Graphs and Manifolds.