text:
What are the best classifiers for this type of data?
I would like to classify a dataset Credit Scoring, which is composed of 21 attributes, some of them are numeric and others are boolean.
For the output, I want to know if they have a good or bad credit based on those attributes, without calculating any numeric value for the credit score.
I am using Weka for this task. However, I am not sure what are the best/ideal classifiers for that kind of datasets.
Anyone here can put me in the right direction?

Well, it depends on the structure of the data. The best way is to try all the intelligent models like naive bayes, random forest, svm with different parameters by grid search. There is no model works best all the time for classification. However,  neural network (named Multilayer Perceptron on weka) is supposed to be better if it is set correctly.


text:
How should I design the reward function for racing game (where the goal is to reach finishing line before the opponent)?
I'm building an agent for a racing game. In this game, there is a randomized map where there are speed boosts for the player to pick up and obstacles that act to slow the player down. The goal of the game is to reach the finishing line before the opponent.
While working on this problem, I've realized that we can almost forget about the presence of our opponent and just focus on getting the agent to the finish line as quickly as possible.
I started with a simple

$-1$ reward for every timestep
$+100$ reward for winning, and
$-100$ for losing.

When I was experimenting with this, I felt like the rewards may be too sparse, as my agent was converging to pretty poor average returns. I iterated to a function of speed and distance travelled (along with the $+100$ reward), but, after some experimentation, I started feeling like the agent might be able to achieve high returns without necessarily being the fastest to the finish line.
I'm thinking that I return to the first approach and possibly add in some reward for being in the first place (as a function of the opponent's distance behind the agent).
What else could I try? Should I try and spread the positive rewards out more for good behavior? should I create additional rewards/penalties for perhaps hitting obstacles and using boosts or can I expect the agent to learn the correlation?

Sutton and Barto state, "The reward signal is your way of communicating to the robot [agent] what you want it to achieve, not how you want it achieved." Since you stated that the goal is to reach the finish line first, then a reward of $1$ for winning, $0$ for losing, and $0$ at all other time steps seems to fit that narrative. If a draw is identical to a loss, then it should provide reward $0$; otherwise, a reward of $0.5$ seems reasonable. These rewards provide model interpretability: an expected return of $p$ (estimated with a state-value or action-value) at a certain state under the current policy would signify a $p$ chance of winning. Also, keeping the rewards in absolute value at most 1 can aid in training speed and prevent divergence, but it often isn't necessary for deep reinforcement learning problems. You most certainly can add other rewards based on partial progress towards the goal, but as it seems you found out, they may lead to incorrect results.
That being said, I would focus on the training process instead of a finely-tuned reward signal. Since there is a known goal state in the racing game (the finish line), I suggest training the RL agent by first initializing all racer agents only a few steps away from the goal state at the beginning of each episode. These episodes are shorter and therefore should provide a more dense reward signal. When your RL agent has learned a winning policy (e.g. wins more often than not), then initialize the agents slightly further from the goal state at the beginning of each episode. Also, continue to use and train the same neural network. Since the neural network presumably knows a winning policy at states near the goal state, then by initializing the agents only a few states further back, the RL agent is given an warm start and only needs to learn a policy for a few more states. The policy encoded by the neural network essentially contains a refined reward signal for the states close to the goal state since it is based on a winning policy; this helps prevent the sparsity problem caused by only supplying a reward at episode completion. You can repeat this process by initializing the agents slightly further from the goal state once the RL agent has learned a winning policy while continuing to use and train the same neural network.
Depending on your access to the environment internals, you may need other analogous approaches. For example, you could initialize the agents at the original starting line (i.e. not partway down the map) and then see which agent makes it $n$ units down the map first to determine the winner. Once a winning policy is learned by the RL agent, then gradually increase $n$ until $n$ matches the distance from the starting line to the finish line. Since it seems like you have distance traveled and distance to the opponent as features, you may instead try this method if you are unable to initialize the agents wherever you want on the map and instead can only initialize them on the starting line.
A notable benefit of the overall approach is that you can more quickly debug your algorithm on the easier environments (i.e. ones with shorter episode lengths) to be confident that the learning process is correct and focus your efforts elsewhere (e.g. the training process, including the reward signal).

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What is the original source of the TD Advantage Actor-Critic algorithm?
What is the original source of the TD Advantage Actor-Critic algorithm?
I found this tutorial really helpful for learning the algorithm. However, what is the original source of this algorithm?

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
When does Monte Carlo linear function approximation converge?
In this Stanford lecture (minute 35:47 and 37:00), the professor says that Monte Carlo (MC) linear function approximation does not always converge, and she gives an example. In general, when does MC linear function approximation converge (or not)?
Why do people use that MC linear function approximation if sometimes it doesn't converge?
They also gave the definition of the stationary distribution of a policy, and I am not sure if using it for function approximation converges or not.

I don't think your question is a duplicate (because you're asking about Monte Carlo approximations), but here's a related question Why doesn't Q-learning converge when using function approximation?.


text:
Understanding the TensorFlow implementation of the policy gradient method
I was trying to understand the implementation of a basic policy gradient (REINFORCE) method using TensorFlow.
I think I got almost everything. The only thing that still bothers me is the loss function implementation.
From the theory, we have that after all the manipulation the gradient of the score function is
$$\nabla_{\theta}J(\theta)=\mathop{\mathbb{E}}\left[\nabla_{\theta}(log(\pi(s,a,\theta)))R(\tau) \right]$$
In this Cartpole example the part relative to the loss function is
    neg_log_prob = tf.nn.softmax_cross_entropy_with_logits_v2(logits = NeuralNetworkOutputs, labels = actions)
    loss = tf.reduce_mean(neg_log_prob * discounted_episode_rewards_)

At this point, I do not understand how the definition from above translates into code.
As far as I understood, the functions
tf.nn.softmax_cross_entropy_with_logits_v2(logits = NeuralNetworkOutputs, labels = actions)

returns
log(softmax(NeuralNetworkOutputs))*actions

Which is then multiplied by the discounted returns
log(softmax(NeuralNetworkOutputs))*actions*discounted_episode_rewards_

Within this expression, I do not understand why should we multiply, an expression which looks like the loss function we want, by the value of the action.

It's not a value of action it's probably one hot vector with 1 on the place where the action you took is placed. You only consider output element of the neural network which represents action that you took.


text:
How long should the state-dependent baseline for policy gradient methods be trained at each iteration?
How long should the state-dependent baseline be trained at each iteration? Or what baseline loss should we target at each iteration for use with policy gradient methods?
I'm using this equation to compute the policy gradient:
$$
\nabla_{\theta} J\left(\pi_{\theta}\right)=\underset{\tau \sim \pi_{\theta}}{\mathrm{E}}\left[\sum_{t=0}^{T} \nabla_{\theta} \log \pi_{\theta}\left(a_{t} | s_{t}\right)\left(\sum_{t^{\prime}=t}^{T} R\left(s_{t^{\prime}}, a_{t^{\prime}}, s_{t^{\prime}+1}\right)-b\left(s_{t}\right)\right)\right]
$$
Here is mentioned to use one or more gradient steps, so is it a hyper-parameter to be found using random search?
Is there some way we can use an adaptive method to find out when to stop?
In an experiment to train Cartpole-v2 using a policy gradient with baseline, I found the results are better when applying 5 updates than when only a single update was applied.
Note: I am referring to the number of updates to take on a single batch of q values encountered across trajectories collected using current policy.

I went into the pytorch code for the spinning up implementation of vanilla policy gradient and from what I could understand, found that they use a learning rate of 1e-3 for training the baseline and do a gradient descent 80 times on the same dataset by default with no termination criteria.
Also it is usually impossible to fit the value function completely as we are using a function approximator. The main point is not to get too carried away to get the lowest loss as the main improvement in the agent's performance will happen by doing policy gradient descents rather than trying to aggressively get the accurate value function for the baseline.
Link for the implementation:
spinning up implementation of vpg


text:
Is the reward following after time step $t+1$ collected based on current policy?
I am currently learning policy gradient methods from the Deep RL boot camp by Pieter Abbeel in which he explains the actor-critic algorithm derivation.
At around minute 39, he explains that the sum of the rewards from time step $t$ onwards is actually an estimation of $Q^\pi(s,u)$. I understand the definition of $Q^\pi(s,u)$ but I'm not sure why this is the case here. Is the reward following after time step $t+1$ collected based on current policy?

yes, you are getting rewards following the policy $\pi$. After completing each trajectory, then you will update the policy $\pi$.

so at state $s_k^(i)$ the action taken is $u_k$ but for state $s_{k+1}$ onwards, the policy $\pi$ is being followed ? but the notation does not contain anything about $\pi$ in the $R(s_k,u_k)$ term

The notation $R(s_{k},u_{k})$ is not clear, but the rewards you get is following your current policy

but why would that be an estimation of Q ? if following the current policy it should be $V^\pi$ ?

You are sampling action($u_{k}$) from the current policy, $u_{k} \sim \pi$. So, it is like that, you are at the state $s_{k}$, from that state sample action $u_{k}$ from current policy $\pi$, thereafter following the policy $\pi$. By defination this is action value function ($Q^{\pi}(s_{k},u_{k})$). Don't be confused with:   $Q^{\pi}(s_{t},\pi(a_{t}|s_{t})$ and $Q^{\pi}(s,a)$ where $a \sim \pi$, those two have different meaning.

hmm how is sampling action from the current policy different from following the policy ? because the policy $\pi_{\theta}$ is a distribution of actions and $u_k$ is an action that is sampled from this policy distribution ?

Consider that you are following  a stochastic policy since this policy gradient algorithm gives the stochastic policy as output. So, $Q^{\pi}(s_{t},\pi(a_{t}|s_{t})) = \sum_{a} \pi(a|s) \sum_{s',r} p(s',r|s,a) (r+\gamma v^{\pi}(s') = V^{\pi}(s)$ and $Q^{\pi}(s_{t},a) = \sum_{s',r} p(s',r|s,a) (r+\gamma v^{\pi}(s'))$ where $a \sim \pi$


text:
Why do RL implementations converge on one action?
I have seen this happening in implementations of state-of-the-art RL algorithms where the model converges to a single action over time after multiple training iterations. Are there some general loopholes or reasons why this kind of behavior is exhibited?

Hi and welcome to AI SE! Maybe it would be a good idea to provide an example of an algorithm that converges to one action, although I don't think a person needs this information to provide an answer (in this case).

Hi @nbro , I have seen this happening in DQN or in policy gradient methods like PPO , so i believe there should be some generalized do's and dont's which we need to follow irrespective of algorithms to ensure that this issue doesnt happen

Isn't the answer to your question just "Because there's probably just an optimal action for each state"? Why do you think this is an issue? It may not be an issue. It depends. If the environment is stochastic, that may be an issue, but if the environment is deterministic, it may be the case that one action is typically the best one.

Well ..   not all environments are deterministic . For example If my agent is playing pacman and it does only one action then it might just hit into a wall and do nothing to recover . I dont think thats an optimal action and I have seen this happening as well... Correct me If I am missing anything here

Yes, you're right. Sometimes the policy shouldn't always select the same action. For example, if you're playing rock paper scissors, the optimal policy shouldn't always choose the same action, because that would make you lose against any intelligent player that recognises the pattern of the policy. However, in the cases you found, maybe there's primarily one optimal action for each state. I don't know because I don't know to which problems those algorithms that you found have been applied.

well lets take some examples .. which I faced while implementing DQN on pacman.. Now the same policy acted really well in case of a game like cart pole . But when implemented for pacman it converged to one action which lead to the agent hitting the wall all the time after 48 hours of training . Similarly , I found that if I implement the PPO algorithm for the same game , it also converged to one single action so I am assuming that there are some general guidelines which I am not following which is leading to this ..

Why do RL implementations converge on one action?

If the optimal policy shouldn't always select the same action in the same state, i.e., if the optimal policy isn't deterministic (e.g., in the case of the rock paper scissors, the optimal policy cannot be deterministic because any intelligent player would easily memorize your deterministic policy, so, after a while, you would always lose again that player), then there are a few things that you can do to make your policy more stochastic

Change the reward function. If your agent ends up selecting always the same action and you don't want that, it's probably because you're not giving it the right reinforcement signal (given that the agent selects the action that apparently will give it the highest reward in the long run).
Try to explore more during training. So, if you're using a behavior policy like $\epsilon$-greedy, you may want to increase your $\epsilon$ (i.e. probability of selecting a random action).
If you estimated the state-action value function (e.g. with Q-learning), maybe you derived the policy from it by selecting the best action, but, of course, that will make your policy deterministic. You may want to use e.g. softmax to derive the policy from the state-action value function (i.e. the probability of selecting an action is proportional to its value), although Q-learning assumes that your target policy is greedy with respect to the state-action value function.

If the optimal policy is supposed to be deterministic, then, if you find the optimal policy (which isn't probably the case), you will end up with an agent that always selects the same action. In that case, obviously, it's not a problem that the RL agent selects always the same optimal action.

Thanks ... I will have to explore how the reward can be manipulated in environments like OPen AI Gym .... For your 2nd point for policy gradients , normally e-greedy is not followed so i guess this will be more applicable for Q learning in general ..

the problem still remains the same - after all the modifications mentioned by you


text:
Why is it hard to prove the convergence of the deep Q-learning algorithm?
Why is it hard to prove the convergence of the DQN algorithm? We know that the tabular Q-learning algorithm converges to the optimal Q-values, and with a linear approximator convergence is proved.
The main difference of DQN compared to Q-Learning with linear approximator is using DNN, the experience replay memory, and the target network. Which of these components causes the issue and why?

See Why doesn't Q-learning converge when using function approximation?.

Thanks for the link. I carefully read the post, it does not actually answer my question.

This question is similar to: Why doesn't Q-learning converge when using function approximation?. If you believe it’s different, please [edit] the question, make it clear how it’s different and/or how the answers on that question are not helpful for your problem.

It is hard to prove because the claim is not true. The DQN algorithm is not guaranteed to converge. For a proof, and a modified algorithm C-DQN that does converge, see the paper by Wang and Ueda.


text:
Are Q-learning and SARSA the same when action selection is greedy?
I'm currently studying reinforcement learning and I'm having difficulties with question 6.12 in Sutton and Barto's book.
Suppose action selection is greedy. Is Q-learning then exactly the same algorithm as SARSA? Will they make exactly the same action selections and weight updates?
I think it's true, because the main difference between the two is when the agent explores, and following the greedy policy it never explores, but I am not sure.

This question is possibly a duplicate of https://stackoverflow.com/questions/32846262/are-q-learning-and-sarsa-with-greedy-selection-equivalent.

@willtryagain The question on Stack Overflow is off-topic there.

If we write the pseudo-code for the SARSA algorithm we first initialise our hyper-parameters etc. and then initialise $S_t$, which we use to choose $A_t$ from our policy $\pi(a|s)$. Then for each $t$ in the episode we do the following:

Take action $A_t$ and observe $R_{t+1}$, $S_{t+1}$
Choose $A_{t+1}$ using $S_{t+1}$ in our policy
$Q(S_t, A_t) = Q(S_t, A_t) + \alpha [R_{t+1} + \gamma Q(S_{t+1},A_{t+1}) - Q(S_t, A_t)]$

Now, in Q-learning we replace $Q(S_{t+1},A_{t+1})$ in line 3 with $\max_aQ(S_{t+1},a)$. Recall that in SARSA we chose our $A_{t+1}$ using our policy $\pi$ - if our policy is greedy with respect to the action value function then this simply means the policy is $\pi(a|s) = \text{argmax}_aQ(s,a)$ which is exactly how we choose our weight update in Q-learning.
To answer the question - no, they are not always the same algorithm.
Consider where we transition from $s$ to $s'$ where $s'=s$. I will outline the updates for SARSA and Q-learning indexing the $Q$ functions with $t$ to demonstrate the difference.
For each case, I will assume we are at the start of the episode, as this is the easiest way to illustrate the difference. Note that actions denoted by $A_i$ are for actions taken explicitly in the environment -- in the Q-Learning update the $\max$ action that is chosen for the update is not executed in the environment, the action taken in the environment is chosen by the policy after the update has happened.
SARSA

We initialise $S_0 = s$ and choose $A_0 = \text{argmax}_a Q_0(s,a)$
Take action $A_0$ and observe $R_{1}$ and $S_{1} = s' = s$.
Choose action $A_{1} = \text{argmax}_aQ_{0}(s,a)$
$Q_{1}(S_0,A_0) = Q_0(S_0,A_0) + \alpha [R_{1} + \gamma Q_0(s,A_1) - Q_0(S_0,A_0)]$

Q-Learning

Initialise $S_0 = s$
Choose action $A_0 = \text{argmax}_aQ_0(s,a)$, observe $R_{1}, S_{1} = s' = s$
$Q_{1}(S_0,A_0) = Q_0(S_0,A_0) + \alpha [R_{1} + \gamma \max_aQ_0(s,a) - Q_0(S_0,A_0)]$
Choose action $A_1 = \text{argmax}_aQ_1(s,a)$

As you can see the next action for the updates in SARSA (line 4) and Q-learning (line 3) are taken with respect to the same $Q$ function, but the key difference is that the actual next action taken in $Q$-learning is taken with respect to the updated $Q$-function.
The key for understanding this edge case is that when we transition into the same state, the Q-Learning update will update the Q-function before choosing $A_1$. I have indexed actions and Q-functions by the episode step - hopefully, it makes sense why I have done this for the Q-functions as, usually, this would not make sense, but, because we have two successive states that are the same, it is okay.

Comments are not for extended discussion; this conversation has been moved to chat.

SARSA:
$$Q(S_t,A_t)=Q(S_t,A_t)+\alpha[R_{t+1}+\gamma Q(S_{t+1},A_{t+1})-Q(S_t,A_t)]$$
Q-Learning
$$Q(S_t,A_t)=Q(S_t,A_t)+\alpha[R_{t+1}+\gamma\max Q(S_{t+1},a)-Q(S_t,A_t)]$$
Consider the following case:
For both SARSA and Q-Learning we'll randomly select $a_1$, when $t=1, t=2$, and assume $\alpha R_2 > 0$
SARSA
At $t=2$ to determine $Q(S_3,A_3)$ since it's greedy it'll choose $s_1,a_1$ and the recalculate $Q(s_1,a_1)$
$$\def\arraystretch{1.5}
   \begin{array}{|c|c|c|}\hline
   (t,s,a, R) & Q(s_1,a_1) & Q(s_1,a_2) \\ \hline
& 0 & 0 \\ \hline
   (1,s_1,a_1, R_2) & \alpha R_2 & 0\\ \hline
   (2,s_1,a_1, R_3) & \alpha R_2 + \alpha R_3 +\alpha^2\gamma R_2 -\alpha^2R_2& 0   \\\hline
   (3,s_1,a_1, R_4) &  & 0\\ \hline
\end{array}$$
Q-Learning, at $t=2$ to determine $Q(S_3,A_3)$, it'll first evaluate the policies so inorder to pick $Q(s_1,a_1)$
$$\alpha R_2 + \alpha R_3 +\alpha^2\gamma R_2 -\alpha^2R_2 > 0$$
$$R_2 + R_3 +\alpha\gamma R_2 -\alpha R_2 > 0$$
$$-R_2(-1-\alpha\gamma + \alpha) + R_3 > 0$$
$$R_3 > R_2(\alpha-\alpha\gamma-1)$$
$$\def\arraystretch{1.5}
   \begin{array}{|c|c|c|}\hline
   (t,s,a) & Q(s_1,a_1) & Q(s_1,a_2) \\ \hline
& 0 & 0 \\ \hline
   (1,s_1,a_1, R_2) & \alpha R_2 & 0\\ \hline
   (2,s_1,a_1, R_3) & \alpha R_2 + \alpha R_3 +\alpha^2\gamma R_2 -\alpha^2R_2& 0   \\\hline
   (3,s_1,a_1, R_4) &  &   \\\hline
\end{array}$$


text:
How does the region proposal method work in Fast R-CNN?
I read so many articles and the Fast R-CNN paper, but I'm still confused about how the region proposal method works in Fast R-CNN.
As you can see in the image below, they say they used a proposal method, but it is not specified how it works.
What confuses me is, for example, in the VGGnet, the output of the last convolution layer is feature maps of shape 14x14x512, but what is the used algorithm to propose the regions and how does it propose them from the feature maps?

Yes, it is not specified because the region proposal algorithm did not change from R-CNN (the previous version from Fast R-CNN, however, in the next verion, Faster R-CNN, this algorithm is replaced by a CNN).
The region proposal algorithm you are looking for is called selective search. You can find in the R-CNN paper that the algorithm is described in "Selective Search for Object Recognition", I found a copy here.
The algorithm is based on a series of segmentation and aggregation techniques of the input image for generating the proposed regions. Check it out 4 iterations of segmentation & aggregation over the same input image to build the proposed regions.

All the algorithm is doing is just iterating over 4 steps:

Initial regions based on segmentation by pixel light intensity are obtained by applying a segmentation algorithm described in the paper. For example, given a picture of a shepherd with his sheep in the mountain, it is segmented by light intensity, and the image of Figure (a) is obtained.
Different regions are proposed based on the previous segmentation, Figure (e)
The similarity between the proposed regions is calculated using the formula proposed in equation 6
in Section 3.2 in the paper which is nothing more than an aggregate metric of the similarity of two regions based in 4 metrics: similarity in color, texture, size and fill (measures how well a region within another)
Add the regions based on similarity and get Figure (b). Then return to step two.

That is how iteratively you get all the images depicted.

This answer is nice (because it visually shows what you mean by "segmentation + aggregation"), but it doesn't describe how "selective search" actually proceeds. So, to improve this answer, I would suggest that you attempt to describe "selective search" slightly more in detail (maybe provide the pseudocode, if it exists?), so that people when reading your answer (which could serve as a reference) have a more detailed idea than "segmentation + aggregation", which is somehow explained by the images that you show us.

yep, you re right, already updated :)

Maybe the only thing that is missing in your answer would be an answer to this question "And how is the size of these proposed regions calculated?".

The intuitive approach on how the regions are calculated is easy to express with words but not that easy mathematically speaking. For a true description the algorithm is explained in the paper (algorithm 1) but I feel that if we were to describe it in here it would make this answer very big


text:
Can you explain me this CNN architecture?
I am starting to get my head around convolutional neural networks, and I have been working with the CIFAR-10 dataset and some research papers that used it. In one of these papers, they mention a network architecture notation for a CNN, and I am not sure how to interpret that exactly in terms of how many layers are there and how many neurons in each.
This is an image of their structure notation.

Can some give me an explanation as to what exactly this structure looks like?

In the CIFAR-10 dataset, each image is $32 \times 32$ pixels, represented by 3072 integers indicating the red, green, blue values for each pixel.
Does that not mean that my input layer has to be of size 3072? Or is there some way to group the inputs into matrices and then feed them into the network?

It would be better if you share the link to the paper that you're referring to.

While it would certainly help if the link to the paper could also be posted, I will give it a shot based on what I understand from this picture.
1) For any convolutional layer, there are few important things to configure, namely, the kernel (or, filter) size, number of kernels, stride. Padding is also important but it is generally defined to be zero unless mentioned otherwise. Let us consider the picture block-by-block.
The first block contains 3 convolutional layers: (i) 2 conv layers with 96 filters each and the size of each filter is $ 3 \times 3$ (and stride $=1$ by default since it is not mentioned), and (ii) another conv layer with same configurations as above but with stride $=2$.
The second block is pretty much the same as the previous except the number of filters is increased to 192 for each layer that is defined.
The only considerable change in the third block is the introduction of $ 1 \times 1$ convolutional filters instead of $3 \times 3$.
And finally, a global average pooling layer is used (instead of a fully connected layer).
2) As for your analysis, it is exactly the case in fully connected layers, wherein the number of units in the input layer must match the vectorized dimensions of the input data. But, in the case of CNN, we give the images directly as the input to the network. The whole idea of a CNN is to understand the spatial structure of the data by analyzing patches of the image at a time (which is what the filter size defines). This PyTorch tutorial should give an idea as to how exactly the input is given to CNN.


text:
When to do discretization to decrease the state/action space in RL?
When to do discretization to decrease the state/action space in RL? Can you give me some references that such a technique is used?


text:
Should Monte Carlo tree search be able to consistently beat me in the connect four game?
I've implemented the Monte Carlo tree search (MCTS) algorithm for a connect four game I've built. The MCTS agent beats a random choice agent 90-100% of the time, but I’m still able to beat it pretty easily. It even misses obvious three in a row opportunities where it just needs to add one more token to win (but places it elsewhere instead).
Is this normal behavior, or should the MCTS agent be able to beat me consistently too? I'm allowing it to grow its tree for 2 seconds before getting it to return its chosen action - could it be that it needs longer to think?

From my understanding, a new tree is grown each time a new state is given to it. The best action from that root state is then returned after it has calculated stats on which action will likely lead to a successful outcome. I have tried creating the tree in two ways: keep expanding the tree and updating the stats until X seconds are over, as well as after X iterations.

@mason7663 you do not need to grow the tree from the beginning, you could also replace the root node with current node and keep the stats as it is and then again perform selection, expansion and backpropagation.

@DuttaA No, MCTS doesn't require any offline training at all. MCTS can be combined with Deep Neural Nets (for various purposes), as was famously done in AlphaGo etc. That's probably what you're thinking of. But in there, it's the DNNs that require huge amounts of training, not the MCTS.

Indeed. I’m not using NNs, just vanilla MCTS

@mason7663 I'm not 100% sure how "good" basic MCTS implementations at 1-2 second time controls are generally expected to be at Connect 4. It will depend very much also on how efficient your implementation of the game is (because that influences how many playouts MCTS can run). It also depends on how good you are at the game yourself :) My own implementation of a fairly basic MCTS variant (named UCT) in Ludii beats me. But maybe I'm just bad. You can download at https://ludii.games/downloads.php and tell me how you fare against that one :)

Because Ludii is a general game system, its implementation of Connect 4 (or any other game really) is significantly slower than what you could pull off in a single-game implementation though. So, in theory, this one should be weaker than your own game-specific implementation.

@SwaksharDeb are you saying the tree (and its stats) are stored somewhere and queried/updated on the next action? If so, where are they stored?

@DennisSoemers Thank you for your comments. After the iterations (2 seconds of), I'm currently only seeing only three (max) children deep nodes/states from the root node (great grandchildren) – does this mean something has gone wrong? Even if I leave it to run for 10 seconds, I'm still only seeing a max of three descendants. Also, using C=2 (in UCT), the values of the children of the root node are all pretty much the same (around 0.933202). Is this another indication of something going awry?

@mason7663 Assuming a 6x7 board, Connect-4 basically has a branching factor of 7 (in most states). So if you can't produce nodes deeper than 3 levels below the root, and you produce one new node per iteration, this suggests you have run at most $7^3 = 343$ MCTS iterations. This sounds very low. In Ludii, my MCTS easily runs $15,000$ iterations per second in Connect 4. This suggests at least that your implementation of the game and/or the algorithm are rather slow. Or maybe you're using a very slow programming language. Are you using Python? :)

You should not let the tree grow for only two seconds rather you should use the simulation number equal to 1000 or something like that. I use the simulation number equal to 10000 for making a single move in the tictactoe game and it was working fine for me. Also, after the agent has chosen the move you do not have to start the statistics(N = visit count, V = expected reward, U = UCT score) from the beginning, you can use the current statistics and replace the root node with the chosen node.

You can limit MCTS by iterations instead of time, but I don't see why you should. It depends on the situation. If you do restrict by iterations, 1000 sounds very low for Connect 4 to me though. In our general game implementation in Ludii (which should be slower than game-specific implementations), my MCTS hits 15K iterations per second in the beginning of a game, quickly getting 30-70K iterations per second in the middle of the game, and a peak of 2.4 million iterations per second for the very last move. Good point about tree reuse though, that can help :)

@DennisSoemers I can not exactly tell how many iterations it will take to give a better result, I just estimated that. But in my sense growing the tree for only 2 sec that OP does is not enough iteration for getting a better result.

@DuttaA No, it's just a tree search algorithm. Like minimax, breadth-first search, depth-first search, etc. It can operate on any problem that you can express as a (finite) tree right away, without any offline training required.

With the proper implementation, 2 secs is enough to defeat you. I have made an MCTS agent for Gomoku (Connect five), which in that interval is able to generate 80k-60k simulations which are enough to beat a random agent 100% of the time and me most of the time. With connect four you should be able to get even more sims and have an even better agent.


text:
Which activation functions should I use for polynomial regression?
I am a beginner in machine learning and neural networks. I have only used neural networks for classification problems.  My aim is to modify it so that it can work for polynomial regression as well. In my problem, I have three inputs and three outputs. My aim is to predict these three outputs based on these three inputs. The outputs are real-valued, and can take positive and negative values.
How should I choose the activation functions? I have only used sigmoid.

For regression problems we use linear activation.

Try the popular ReLU activation function (or its variant, the leaky ReLU)

Thank you. I have one more doubt. The three input data are in the range of 100s eg(500,600,700) and the output have mixed units (700, -0.52, 0.45). Should I normalize the input as well as output ?


text:
What are the differences between artificial neural networks and other function approximators?
Modern artificial neural networks use a lot more functions than just the classic sigmoid, to the point I'm having a hard time really seeing what classifies something as a "neural network" over other function approximators (such as Fourier series, Bernstein polynomials, Chebyshev polynomials or splines).
So, what makes something an artificial neural network? Is there a subset of theorems that apply only to neural networks?
Backpropagation is classic, but that is the multi-variable chain rule, what else is unique to neural networks over other function approximators?

First of all, neural networks are not (just) defined by the fact that they are typically trained with gradient descent and back-propagation. In fact, there are other ways of training neural networks, such as evolutionary algorithms and the Hebb's rule (e.g. Hopfield networks are typically associated with this Hebbian learning rule).
The first difference between neural networks and other function approximators is conceptual. In neural networks, you typically imagine that there are one or more computational units (often called neurons) that are connected in different and often complex ways. The human can choose these connections (or they could also be learned) and the functions that these units compute given the inputs. So, there's a great deal of flexibility and complexity, but, often, also a lack of rigorousness (from the mathematical point of view) while using and designing neuron networks.
The other difference is that neural networks were originally inspired by the biological counterparts. See A logical calculus of the ideas immanent in nervous activity (1943) by Warren McCulloch and Walter Pitts, who proposed, inspired by neuroscience, the first mathematical model of an artificial neuron.
There are other technical differences. For example, the Taylor expansion of a function is typically done only at a single value of the domain, it assumes that the function to be approximated is differentiable multiple times, and it makes uses of the derivatives of such a function. Fourier series typically approximate functions with a weighted sum of sinusoids. Given appropriate weights, the Fourier series can be used to approximate an arbitrary function in a certain interval or the entire function (if the function you want to approximate is also periodic). On the other hand, neural networks attempt to approximate functions of the form $f: [0, 1]^n \rightarrow \mathbb{R}$ (at least, this is the setup in the famous paper that proved the universality of neural networks) in many different ways (for example, weighted sums followed by sigmoids).
To conclude, neural networks are quite different from other function approximation techniques (such as Taylor or Fourier series) in the way they approximate functions and their purpose (i.e. which functions they were supposed to approximate and in which context).


text:
Is the high dimensionality of input vectors a problem for a radial basis function neural network?
I have a dataset A of videos. I've extracted the feature vector of each video (with a convolutional neural network, via transfer learning) creating a dataset B. Now, every vector of the dataset B has a high dimension (about 16000), and I would like to classify these vectors using an RBF-ANN (there are only 2 possible classes).
Is the high dimensionality of input vectors a problem for a radial basis function ANN? If yes, is there any way to deal with it?


text:
Why can't DQN be used for self-driving cars?
Why can't DQN be used for self-driving cars?
Why can't DQN and similar RL algorithms be used for self-driving cars?
The reason why I am curious is that it successfully plays go and other multistate games.

I'm not familiar with the ins and outs of self-driving cars, but I imagine that the action space is not discrete. For instance, the car may want to decide what angle it needs to turn (rather than left or right). The update in Q-Learning involves taking $\max_aQ(s, a)$; this is theoretically possible for a continuous action space, but it would itself require some expensive optimisation at each time step to find the maximum. It is more likely that if RL were to be applied to self-driving cars it would be through a method that easily allows for a continuous action space, like the methods detailed in this paper.
I found this survey of Deep RL for autonomous driving that you may want to look at.


text:
How do I determine the genomes to use for crossover in NEAT?
If I have the fitness of each genome, how do I determine which genome will crossover with which, and so on, so that I get a new population?
Unfortunately, I can't find anything about it in the original paper, so I ask here?

Maybe this code from SharpNEAT will help you: https://github.com/colgreen/sharpneat/blob/master/src/SharpNeatLib/EvolutionAlgorithms/NeatEvolutionAlgorithm.cs#L505

The original work on NEAT(Neuroevolution of augmenting topologies) was by Ken Stanley in 2002 at The University of Texas at Austin. The web page for the project is here I suggest you download and read the paper linked from that page. As for selection of genome pairs, NEAT makes use of a speciation model so  the selection of such pairs is constrained to at least prefer pairs from the same 'species', on the assumption that the species has evolved such that species population is isolated under reproduction. The innovation that has been 'bred' into the species is thus preserved under reproduction.  Selection by fitness alone is insufficient in such models. This differs from the simple GA where pair selection is unconstrained.

The good thing about genetic algorithms is that they are exchangeable.
If you have the fitness of each individual, any algorithm (i.e. roulette, rank, tournament) will do.


text:
How should I deal with variable batch size in A3C?
I am fairly new to reinforcement learning (RL) and deep RL. I have been trying to create my first agent (using A3C) that selects an optimal path with the reward being some associated completion time (the more optimal the path is, packets will be delivered in less time kind of).
However, in each episode/epoch, I do not have a certain batch size for updating my NN's parameters.
To make it more clear, let's say that, in my environment, on each step, I need to perform a request to the servers, and I have to select the optimal path.
Now, each execution in my environment does not contain the same amount of files. For instance, I might have 3 requests as one run, and then 5 requests for the next one.
The A3C code I have at hand has a batch size of a 100. In my case, that batch size is not known a priori.
Is that going to affect my training? Should I find a way to keep it fixed somehow? And how can one define an optimal batch size for updating?


text:
Why is DDPG an off-policy RL algorithm?
In DDPG, if there are no $\epsilon$-greedy and no action noise, is DDPG an on-policy algorithm?

DDPG is an off-policy algorithm simply because of the objective taking expectation with respect to some other distribution that we are not learning about, i.e. the deterministic policy gradient can be expressed as
$$\nabla _{\theta^\mu} J \approx \mathbb{E}_{s_t \sim \rho^\beta} \left[ \nabla _{\theta^\mu} Q(s,a|\theta^Q) | s=s_t, a=\mu(s_t ; \theta ^\mu)  \right]\;.$$
We are interested in learning about the policy parameters of $\mu$, denoted by $\theta$, but we take expected with respect to some discounted state distribution induced by a policy $\beta$, which we will denote as $\rho^\beta$.
To summarise, we are learning off-policy as the expectation of the gradient is taken with respect to some state distribution that occurs under some policy that we are not learning about.
Given that on-policy learning is a special case of off-policy learning, if the replay buffer had a size of one, i.e. we use only the most recent experience tuple to perform parameter updates, then DDPG would be on-policy.

If there was no action noise it would probably not explore enough to obtain a good estimate of Q or the policy gradient.
Instead of estimating Q of the target policy you could estimate Q of the behavior policy but then you have a stochastic policy and the deterministic policy gradient theorem does not work anymore as it is a special case of the stochastic policy gradient theorem (see Section 3.3 of the DPG paper, http://proceedings.mlr.press/v32/silver14.pdf). You would have to use policy gradient theorem of Section 2.2 from the DPG paper.

Thanks for your answer, what is the behaviour policy in DDPG?

In algorithm 1 from the DDPG paper (https://arxiv.org/pdf/1509.02971.pdf) you find the behavior policy in line 8 (Select action a_t ...). It is the target policy plus some exploration noise. For this paper they used an "Ornstein-Uhlenbeck process" (last paragraph of Section 3 and Appendix 7). It seems like Gaussian noise works, too (source: https://spinningup.openai.com/en/latest/algorithms/ddpg.html#exploration-vs-exploitation).

I think that is not a behaviour policy right? Even on-policy algorithms like PPO has such policy.

Yes, it is. In on-policy algorithms the behavior policy is the target policy. The behavior policy is used for exploration and the target policy is used to approximate the value function (Q, V, or A). See Section 5.5 of Sutton & Barto (http://incompleteideas.net/book/RLbook2020.pdf).

Maybe you have a wrong idea about what off-policy and on-policy means. What would you say is the difference between both?

I don't think adding noise to actions is a behaviour policy, if that so, in PPO (on-policy), adding a noise to its actions makes it off-policy? It is not the definition. The behaviour policy can be a bad policy, it is used to collect data for target policy to learn. I do not think the main purpose of behaviour policy is used for exploration.

The behavior policy is called behavior policy because it defines how the agent behaves. The DDPG agent creates its actions through mu + noise while it approximates Q of mu - its target policy. PPO in its original form uses a stochastic policy and is on-policy. Adding noise would not change that. The stochastic policy ensures sufficient exploration. The purpose of a policy is either eploration or exploration (or both). A behavior policy is clearly more on the exploration side than the target policy.

Thanks for the reply of the purpose of adding the noise, back to my question: in DDPG, if there are no -greedy and no action noise, is DDPG an on-policy algorithm? I am not sure on this point. And I am not clear on if the behaviour is clearly more on the exploration side because I did not find theoretical guarantees on its exploration ability of the behaviour so far. If the behaviour policy is far away from the target policy, it can be a bad policy, some algorithms like Retrace/Vtrace still own some theoretical guarantees on the convergence of the target policy.

OK, you could say that without exploration noise it is on-policy (with a deterministic policy). It would most likely not work though. If you had an infinite amount of samples, infinite capacity of the neural networks, and infinite training time for your networks you would perfectly fit Q of mu and I guess mu would get stuck in some local optimum. Theoretically this is not much different from tabular Q-learning.

There are no guarantees in Deep RL for anything anyway though, so maybe it works even better than some noisy behavior policies. ;) As far as I understand (I might be wrong here) V-trace guarantees convergence of the value function to "something between target and behavior policy" in the tabular case. With a neural network in the mix I am not sure if anything is guaranteed. V-trace, however, needs a stochastic policy to compute importance sampling weights. mu in DDPG is not stochastic. By the way, in the V-trace paper mu is the behavior policy and pi is the target policy. Another notation.

I think that although this does not answer the posed question directly, it is the correct answer. DDPG without noise is at best a partially functioning RL algorithm, and probably not of much interest. It may work successfully in environments with stochastic overlapping state transitions. TD Gammon for example got away with only greedy action choice in a SARSA/Q-learning algorithm due to randomness inherent in the game


text:
How do I calculate the partial derivative with respect to $x$?
I am trying to implement CNN using python NumPy. I searched so much, but all I found was for one filter with one channel for convolution.
Suppose $x$ is an image with the shape: (N_Height, N_Width, N_Channel) = (5,5,3).
Let's say I have 16 filters with this shape: (F_Height, F_Width, N_Channel) = (3,3,3) , stride=1 and padding=0
Forward:
The output shape after convolution 2d will be
(
math.floor((N_Height - F_Height + 2*padding)/stride + 1 )),
math.floor((N_Width- F_Width + 2*padding)/stride + 1 )),
filter_count
)

So, the output of this layer will be an array with this shape: (Height, Width, Channel) =  (3, 3, 16)
BackPropagation:
Suppose $dL/dh$ is the input for my layer in back-propagation with this shape: (3, 3, 16)
Now, I must find $dL/dw$ and $dL/dx$: $dL/dw$ to update my filters parameter and $dL/dx$ to pass it as input to the previous layer as the loss respect to the input $x$.
From this answer Error respect to filters weights I found how to calculate $dL/dw$.
The problem I have in the back-propagation is I don't know how to calculate $dL/dx$ having this shape: (5, 5, 3) and pass it to the previous layer.
I read lots of articles in Medium and other sites, but I don't get how to calculate it:

How Backpropagation works in a CNN
The best explanation of Convolutional Neural Networks on the Internet!
Backpropagation In Convolutional Neural Networks
How to propagate error back to previous layer in CNN?

While this may not be the answer you were looking for, I hope this explanation will help you to understand applying backpropagation to a CNN. Fundamentally, convolutional layers are no different than dense layers, however there are restrictions. The key one is weight-sharing which allows a CNN to be much more efficient than a regular dense layer (as well as it being sparse due to locality). Imagine we are transforming a 4x4 image into a 2x2 image. Since we are inputting a 16-vector, and outputting a 4-vector, we need a weights matrix of 4x16:

This has 64 parameters. In a convolutional layer, we can accomplish this by convolving a 3x3 kernel over the image:
$$ K=
\begin{bmatrix}
k_{1,1} & k_{1,2} & k_{1,3} \\
k_{2,1} & k_{2,2} & k_{2,3} \\
k_{3,1} & k_{3,2} & k_{3,3}
\end{bmatrix} $$
This convolution is equivalent to multiplying by the weights matrix:

As you can see, this only requires 9 parameters and backpropagation can be applied to update these parameters.
Image Source: https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1

thanks but as you said it is not my question's answer.
I finally find $dL/dX$ for each channel in my example 16 then I add all of them and
I get my $dL/dX$


text:
Which machine learning method can take a matrix as input?
I am pretty new to the machine learning field. I want to use an $n \times m$ matrix as the input of a model, in order to predict a vector $1 \times m$, both of real numbers. Input data are quite clean, with statistics of about 10000 items.
Do you know a method that can handle that?

Hint: Start by flattening the $n\times m$ matrix to a $nm\times 1$ vector.

ANNs can do the trick. Check out sklearn's ANN example with the Digits dataset, which consists of 28x28 pixel input data.


text:
What are some use cases of few-shot learning?
Besides computer vision and image classification, what other use cases/applications are for few-shot learning?

Anything? A system which can learn from a single example is the holy grail.

Few-short learning (FSL) can be useful for many (if not all) machine learning problems, including supervised learning (regression and classification) and reinforcement learning.
The paper Generalizing from a Few Examples: A Survey on Few-Shot Learning (2020) provides an overview (including examples of applications and use cases) of FSL. Their definition of FSL provided is based on Tom Mitchell's famous definition of machine learning.

Definition 2.1 (Machine Learning [92, 94]). A computer program is said to learn from experience $E$ with respect to some classes of task $T$ and performance measure $P$ if its performance can improve with $E$ on $T$ measured by $P$.

Here's the definition of FSL.

Definition 2.2. Few-Shot Learning (FSL) is a type of machine learning problems, specified by $E$, $T$ and $P$, where $E$ contains only a limited number of examples with supervised information for the task $T$.

Specific examples of applications of FSL are

character generation
drug toxicity discovery
sentiment classification from short text
object recognition

An interesting use case is IQ tests, or program synthesis from examples in general.
IQ tests often require you to derive a program from a few examples that can produce a certain output. See for instance https://github.com/fchollet/ARC


text:
What is a Hidden Markov Model - Artificial Neural Network (HMM-ANN)?
As far as I know, neural networks have hidden computational units and HMM has hidden states.
Hidden Markov Models can be used to generate a language, that is, list elements from a family of strings. For example, if you have an HMM that models a set of sequences, you would be able to generate members of this family, by listing sequences that would befall into the group of sequences we are modeling.
In this, this and this paper, HMMs are combined with ANNs. But how exactly? What is a Hidden Markov Model - Artificial Neural Network (HMM-ANN)? Is HMM-ANN a hybrid algorithm? In simple words, how is this model or algorithm used?

Where did you find this expression "Hidden Markov Model - Artificial Neural Network"?

Here and here HMM-ANN

In these 3 papers, are you sure they are referring to the same model? Because if not this question may become a little bit too broad, because you would be asking us to explain different models. Maybe HMMs are combined with ANNs in different ways in these papers.

Yes, they are referring the same(Speech Recognition).

I was asking if they were referring to the same model, not if they were being used for the same purpose or task. Are you sure that, in these papers, they are using the same model?

According to my knowledge,  Yes they are referring the same model,


text:
Is there any practical application of knowing whether a concept class is PAC-learnable?
A concept class $C$ is PAC-learnable if there exists an algorithm that can output a hypothesis with probability at least $(1-\delta)$ (the "probably" part), and an error that is less than $\epsilon$ (the "approximately" part), in time that is polynomial in $1/\epsilon$, $1/\delta$, $n$ and $|C|$.
Tom Mitchell defines an upper bound for the sample complexity, $m >= 1/\epsilon (ln(|H|) + ln(1/\delta))$ for the finite hypotheses. Based on this bound, he classifies whether target concepts are PAC-learnable or not. For example, $n$ conjunction boolean literal concept class.
It seems to me that PAC-learnability seeks to act more like a classification of certain concept classes.
Are there any practical purposes for knowing whether a concept class is PAC-learnable?

but wouldn't determining whether the concept class is PAC learnable means going through the process that i described in the post ?

Is there any practical application of knowing whether a concept class is PAC-learnable?

If you know that a concept class is PAC-learnable (i.e. its VC dimension is finite), then there's a possibility that you can design an algorithm that can find a function (or concept) that is arbitrarily close to your target (or desired) function.
This is not really an application, but a consequence, which can lead to applications.
However, note that asking if PAC learning is useful in practice is like asking if special relativity is useful in practice. Yes, they are useful, but in the sense that they can be used to predict the outcomes of an experiment or explain the rules in their specific context. In the case of machine learning, PAC learning can be used to explain e.g. the probably required number of data points needed to learn a concept (a target function) approximately.
See also Are PAC learning and VC dimension relevant to machine learning in practice? for more concrete "applications" of PAC-learning and the VC dimension.


text:
If features are always positives, why do we use RELU activation functions?
When does it happen that a layer (either first or hidden) outputs negative values in order to justify the use of RELU?
As far as I know, features are never negative or converted to negative in any other type of layer.
Is it that we can use the RELU with a different "inflection" point than zero, so we can make the neuron start describing a lineal response just after this "new zero"?

The fact that features are always positive values don't guarantee that outputs of hidden layers are positive too.
Due to multiplication, output of an hidden layer could contain negative values, i.e., a hidden layer can contain weights that have opposites signs as its input.
Remember that only layer outputs, not their weights, are passed through ReLu, so, weights of a model could contain negative values.


text:
Why do we use a weighted average of child entropies when we calculate information gain?
In the decision tree algorithm, why do we use a weighted average of child entropies when we calculate information gain? What is wrong about using the arithmetic mean of entropies?

Hi and welcome to AI SE! There are different decision tree algorithms. Which one are you referring to?

@nbro Thank you! Well, in more general sense rather than particular one. It's part of an exercise to be honest...


text:
How do reinforcement learning and collaborative learning overlap?
How do reinforcement learning and collaborative learning overlap? What are the differences and similarities between these fields?
I feel like the results I get via google do not make the distinction clear.

This is an old question, but can you confirm that by "collaborative learning", you mean "collaborating filtering" in recommender systems? I know this question was already edited by me and you, so the original intent was changed.


text:
Can we use a Gaussian process to approximate the belief distribution at every instant in a POMDP?
Suppose $x_{t+1} \sim \mathbb{P}(\cdot | x_t, a_t)$ denotes the state transition dynamics in a reinforcement learning (RL) problem. Let $y_{t+1} = \mathbb{P}(\cdot | x_{t+1})$ denote the noisy observation or the imperfect state information. Let $H_{t}$ denote the history of actions and observations $H_{t+1} = \{b_0,y_0,a_0,\cdots,y_{t+1}\}$.
For the RL Partially Observed Markov Decision Process (RL-POMDP), the summary of the history is contained in the "belief state"  $b_{t+1}(i) = \mathbb{P}(x_{t+1} = i | H_{t+1})$, which is the posterior distribution over the states conditioned on the history.
Now, suppose the model is NOT known. Clearly, the belief state can't be computed.
Can we use a Gaussian Process (GP) to approximate the belief distribution $b_{t}$ at every instant $t$?
Can Variational GP be adapted to such a situation? Can universal approximation property of GP be invoked here?
Are there such results in the literature?
Any references and insights into this problem would be much appreciated.

How? If you have no model you literally know nothing. You have no idea how to assign a probability to any state or even what a state is.

@FourierFlux: Say you have access to the following (training) data: Input {a_t,y_t} and output {x_t}. It's just during the implementation (testing), we only have access to imperfect information. Could a GP, which captures the distribution over the states, match the belief state? at every time t? Are there such works?

@math_phile You need a model for that. GP is a model. You said you have no model. You need to make some assumptions about what you want to predict.


text:
Would you categorize policy iteration as an actor-critic reinforcement learning approach?
One way of understanding the difference between value function approaches, policy approaches and actor-critic approaches in reinforcement learning is the following:

A critic explicitly models a value function for a policy.
An actor explicitly models a policy.

Value function approaches, such as Q-learning, only keep track of a value function, and the policy is directly derived from that (e.g. greedily or epsilon-greedily). Therefore, these approaches can be classified as a "critic-only" approach.
Some policy search/gradient approaches, such as REINFORCE, only use a policy representation, therefore, I would argue that this approach can be classified as an "actor-only" approach.
Of course, many policy search/gradient approaches also use value models in addition to a policy model. These algorithms are commonly referred to as "actor-critic" approaches (well-known ones are A2C / A3C).
Keeping this taxonomy intact for model-based dynamic programming algorithms, I would argue that value iteration is an actor-only approach, and policy iteration is an actor-critic approach. However, not many people discuss the term actor-critic when referring to policy iteration. How come?
Also, I am not familiar with any model-based/dynamic programming like actor only approaches? Do these exist? If not, what prevents this from happening?

Keeping this taxonomy intact for model-based Dynamic programming algorithms, I would argue that value iteration is a Actor only approach, and policy iteration is a Actor-Critic approach. However, not many people discuss the term Actor-Critic when referring to Policy Iteration. How come?

Both policy iteration and value iteration are value-based approaches. The policy in policy iteration is either arbitrary or derived from a value table. It is not modelled separately.
To count as an Actor, the policy function needs to modelled directly as a parametric function of the state, not indirectly via a value assessment. You cannot use policy gradient methods to adjust an Actor's policy function unless it is possible to derive the gradient of the policy function with respect to parameters that control the relationship bewteen state and action. An Actor policy might be noted as $\pi(a|s,\theta)$ and the parameters $\theta$ are what make it possible to learn improvements.
Policy iteration often generates an explicit policy, from the current value estimates. This is not a representation that can be directly manipulated, instead it is a consequence of measuring values, and there are no parameters that can be learned. Therefore the policy seen in policy iteration cannot be used as an actor in Actor-Critic or related methods.
Another way to state this is that the policy and value functions in DP are not separate enough to be considered as an actor/critic pair. Instead they are both views of the same measurement, with the value function being closer to raw measurements and policy being a mapping of the value function to policy space.

Also, I am not familiar with any model-based/dynamic programming like actor only approaches? Do these exist? If not, what prevents this from happening?

The main difference between model-based dynamic programming and model-free methods like Q-learning, or SARSA, is that the dynamic programming methods directly use the full distribution model (which can be expressed as $p(r, s'|s,a)$) to calculate expected bootstrapped returns.
There is nothing in principle stopping you substituting expected returns calculated in this way into REINFORCE or Actor-Critic methods. However, it may be computationally hard to do so - these methods are often chosen when action space is large for instance.
Basic REINFORCE using model-based expectations would be especially hard as you need an expected value calculated over all possible trajectories from each starting state - if you are going to expand the tree of all possible results to that degree, then a simple tree search algorithm would perform better, and the algorithm then resolves to a one-off planning exhaustive tree search.
Actor-Critic using dynamic programming methods for the Critic should be viable, and I expect you could find examples of it being done in some situations. It may work well for some card or board games, if the combined action space and state space is not too large - it would behave a little like using Expected SARSA for the Critic component, except also run expectations over the state transition dynamics (whilst Expected SARSA only runs expectations over policy). You could vary the depth of this too, getting better estimates theoretically at the expense of extra computation (potentially a lot of extra computation if there is a large branching factor)

I think that the confusion that policy iteration is an actor-critic method lies in the fact that in actor-critic methods you use the value function to guide the search for the policy. In policy iteration, you actually use the value function to derive the policy too. I don't think it's fully clear from your answer why policy iteration couldn't be considered an actor-critic method.

Thanks for your elaborate answer. in policy iteration however; the policy function is usually directly represented, right? Sure, it is a quite trivial step to do so (policy improvement), but it is done nonetheless. For example, see the algorithm shown in Sutton and Barto: http://www.incompleteideas.net/sutton/book/first/4/node4.html#SECTION00130000000000000000

About how REINFORCE would translate into a DP method, wouldn't this resolve to some form of policy iteration?

You generate a "trajectory" and calculate its return. In a model-based sense however; this trajectory updates the value based on all possible successor states until a terminal state, somewhat emulating a step of policy evaluation.
You improve the policy based on the gradient signal from this evaluation -> policy improvement
Sure, doing it this way doesn't intuitively sound very smart. Why not update all states in a normal Policy evaluation manner?

@NeilSlater Okay, so the key difference between PI and Actor-Critic would be that PI generates a policy in every step, whereas AC modifies/adapts an older policy in every step, right? Do you happen to know of any sources that would argue similarly?

@NeilSlater Exactly, I'm on the same page about REINFORCE, I just worded it  awkwardly. Generating a single trajectory isn't really possible in a DP sense as you are taking the expected value, resulting in effectively generating all the trajectories. Thanks for that!

@DaniëlWillemsen I don't know of any other sources that would try to argue the case that DP is not a policy-based method. It is just a basic fact that DP is the core of value-based methods. Most RL taxonomy diagrams will show this - DP methods represent the baseline of "Generalise Policy Iteration" approaches, and don't relate to policy gradients at all

@nbro: I have tried to make that clearer, the difference being the policy derviation method in DP is not a direct parametric function over state that can be adjusted by changing those parameters. I guess another way to state this is that the Actor and Critic are not separate enough, they are just transformations of the same measurement.

policy iteration is an actor-critic approach.

This is a very insightful observation and I would agree with it. You can find similar statement in this lecture and this NeurIPS paper.
The actor's role is to perform policy improvement and the critic's role is to perform policy evaluation. This helps explain its advantage over value-only approaches, as PI converges faster than VI although each iteration is more expensive.


text:
Does L1/L2 Regularization help reach an optimum result faster?
I understand that L1 and L2 regularization helps to prevent overfitting. My question is then, does that mean they also help a neural network learn faster as a result?
The way I'm thinking is that since the regularization techniques reduce weights (to 0 or close to 0 depending on whether it's L1 or L2) that are not important to the neural network, this would, in turn, result in "better values" for the output neurons right? Or perhaps I am completely wrong.
For example, suppose I have a neural network that is to train a snake to move around a NxN environment. With regularization, the snake will learn faster in terms of survive longer in the game?

I am not aware of any empirical results regarding this question.
But in theory, adding a regularization term shall make the learning task actually even harder, since there is suddenly a second loss term that the network has to be optimized for, which is not even directly related to achieving the original task of fitting the model to the data.
It is true that the regularization term will try drive as many of the weights towards low values as it can. But, at the same time, the other loss term (computed on the original optimization criterion) will try to drive many of the weights to larger values unequal 0 in order to achieve the original training task. (If that wasn't the case, you would be good to go without regularization in the first place.)
Having these competing interests shall then make achieving the original task harder, which I would in theory expect to be more time consuming than, alternatively, allowing the model overfit until a certain admissible error is reached.
After all, the goal of regularization is to improve generalization of a learned model (i.e. to prevent overfitting, as you said), which is about the quality of the outcome and not about how quickly you get to the desired outcome.


text:
Is it possible to prove that the target policy is better than the behavioural policy based on learned Q values?
I have retrospective data for a sort of "behaviour policy" which I will use to train a deep q network to learn a target greedy policy. After learning the Q values for this target policy, can we make the conclusion that because the Q value for the target policy, $Q(s,\pi_e(s))$ is higher than the Q values for the behaviour policy, $Q(s,\pi_b(s))$ at all states encountered, where $\pi_e$ is the policy output by deep Q-learning and $\pi_b$ is the behaviour policy, then this target policy has better performance than the behaviour policy?
I know the proper way is to run the policy and do an empirical comparison of some sort. However, that is not possible in my case.

No, mainly because these are all stochastic approximations and may not represent the true values.
Almost nothing good can be said about NN approximations to value and Q functions(at least according to a professor I have had).

I believe the first sentence is correct. However, I also think the answer needs to have a few more details. If nothing else, worth clarifying "nothing good can be said" - it could be read as it is bad choice of solution, but actually many novel results have been produced using deep RL that are not possible with any other technique. So perhaps you are referring to the lack of solid theory, such as a practical loss function that is minimised when agent performance is maximised.


text:
What are the implications of the statement "If you can't tell, does it matter?" in relation to AI?
"If you can't tell, does it matter?" was one of the first lines of dialogue of the Westworld television series, presented as a throwaway in the first episode of the first season, in response to the question "Are you real?"
In the sixth episode of the third season, the line becomes a central realization of one of the main characters, The Man in Black.
This is, in fact, a central premise of the show—what is reality, what is identity, what does it mean to be alive?—and has a basis in the philosophy of Philip K. Dick.

What are the implications of this statement in relation to AI?  In relation to experience?  In relation to the self?

Given that the answer below seems to have misinterpreted your question, let me ask for clarification. Is your question "Does it really matter if an artificial (general) intelligence is conscious or not (i.e. can have "own experiences", whatever that means)?"

The question in this video is

Are you real?

What does this question really mean? Is the guy asking whether the apparent female (I don't know if she is a cyborg or not because I did not yet watch the TV series) is a human? So, is "real" a synonym for "human"? If that's the case, then the first implication (in the form of a question) of the statement

If you can't tell, does it matter?

in relation to AI is

Can we create an AGI that is sufficiently similar to a human that we can't tell whether it's a human or not (by just normally interacting with it)?

Of course, it's not clear what we mean by "normally interacting". As far as I remember, this issue is also raised in the film A.I. Artificial Intelligence, where the AI (the kid interpreted by Haley Joel Osment) looks sufficiently real to the other kid, so he behaves as if he was a human kid, but then the human kid understands that he's a machine, and starts to behave differently (I hope I'm remembering the film correctly).
So, the second question that we could ask is

Once we understand that it's not a human (for example, because it's made of other substances), would we humans start to behave differently and start treating the AGI differently?

As opposed to the first question, which is still an open problem, this second question can probably be answered by looking at our relationships with other humans (or entities, such as other animals). Often, we have an idea of a person. Once we discover something new about that person, which maybe we dislike, we may start to treat that person differently. I think this would very likely also happen in our eventual relationship with a sufficiently advanced AGI too, as depicted in the mentioned film.
Now, let me try to address the other question

What are the implications of this statement in relation to experience, in relation to the self?

I think that you're asking whether a sufficiently advanced AGI could be considered conscious or not. Of course, this is a very hard question to answer, because we still don't have a clear definition of consciousness or we don't yet agree on a standard definition, so I don't really have a definitive answer to this question. However, if consciousness is just a byproduct of perception and the ability to understand the world and its (physical) rules, then an AGI could be conscious (in a similar way that humans are also conscious). However, consciousness may not actually be necessary to correctly act in the world. In any case, the AI probably needs to know that it has a body and that it needs to protect it for its survival, if that's its main goal.

Interesting take.

It's useful to understand HBO's Westworld as an extension of Phillip Dick's Do Androids Dream of Electric Sheep.
Most of Dick's novels involve the nature of reality in relation to perception, and how that informs identity.
A major feature of Dick's book, and BladeRunner, is a form of Turing Test (Voight-Kampff) to which humans are subjected to determine if they are replicants.
At a certain point, Deckard, the hero, begins to question whether he is human or an android.  (This is never fully made clear in the book, and Deckard's wife's alienation from him may indicate his non-human status.  The film adaptation similarly raises this question over potentially implanted memories, which would mark Deckard as a replicant, even though the director reversed course and later stated otherwise.)
Westworld continues this idea where there are characters who turn out to be definitively androids, such as the Man in Black, who, presumably, has had artificial memories created, and believes he is "real".  BladeRunner 2049 also involves this theme, which could be said to be the "unreliability of memory" in relation to experience and identity.  Even in mundane circumstances, two humans can remember the same event differently!

The point of the Electric Sheep hypothesis is ambiguity—we can only validate our own qualia, and even that is not entirely reliable due to the nature of perception and subjectivity.

The novel ends with Deckard finding a frog in the ashes, and initially thinking it is real.  It turns out to be robotic, but Deckard ultimately decides it doesn't really matter.

This is important because empathy is the main theme, and altruistic behavior in nature is supported by evolutionary game theory.

The central plot device is that replicants don't have empathy, a design flaw that becomes a "feature not a bug" in that it keeps replicants from banding together to overthrow their oppressors.  But the new generation of Nexus androids are intelligent enough to develop empathy naturally.
Dick was a Christian philosopher who worked mainly in popular narrative and believed empathy is a natural function of intelligence sufficiently advanced.

If the suffering witnessed by an entity appears real, but we cannot validate that entity's qualia, is it a moral imperative to make a Pascal's Wager?

i.e. err on the side of caution and compassion, just in case the entity is conscious.

If altruistic behavior is expressed by an algorithm, is that altruism invalid?

I haven't watched Westworld from the beginning to the end, but I've read the synopsis of it.
The implications of the statement above might be related to the experiences of what the androids (or cyborgs) might think of themselves. Are their identity and experiences that they had gone through "real" or not? It seems that, in the series, this question is actually central, and the series seems to be about an identity crisis, the search for a true identity.
In my view, without really any fail-safe programming, once the androids understand that all of their experiences are false (or unreal) and that everything that they do is monitored and tested by some individuals for science purposes, the AI would go rogue. A conscious being with no real background experiences being put into an environment that 'should' have been their 'life' all along, but, after realizing the real truth, everything does matter, and the conscious being would go rogue to find their true self in order to recover their origin and their true identity.

I'm not sure you interpreted the question correctly. You're essentially assuming that the androids are conscious (i.e. can have their own/independent experiences), and you try to explain what would happen when they discover that this is the case. I don't think that the question was about this. I think the question was more about "Does it really matter if the AI is conscious or not?".


text:
How well can NLP techniques recognize connotations in natural languages?
What is the state of the art with respect to recognizing connotations in natural languages?
For instance:

Trump is a better president than Obama.           [Praising]
Trump is the worst president since Obama.         [Insulting]

or:

The rock star did not infect over 100 groupies.   [Defending against rumor]
The rock star infected no more than 100 groupies. [Attacking (0 is no more than 100)]

In each example, both statements logically mean exactly the same thing, but any human hearing them would interpret them as having quite opposite meanings.
How well can current natural language processors recognize the difference between logically equivalent statements?


text:
Why is the number of neurons used in various neural networks power of 2?
I have noticed that almost all tutorials take the number of neurons as a power of 2. Is there any proper mathematical and well-proven reason for that?
If you sometimes change it to some other odd value, then we get a very long and weird error calling about a dozen things making the traceback of a page. Is there any reason for that?

I had tried it on a text predicting RNN with some GRU and LSTM layers mixed (bidirectional). I changed the no. of neuron units and It also resulted in an error. So, any ideas/theories?

Hi and welcome to AI SE! You say "If you sometimes change it to some other odd value, then we get a very long and weird error calling about a dozen things making the traceback of a page", but I don't think this is caused by the fact that the number of neurons is not a power of 2. So, can you provide a concrete example where that occurs? Also, note that, for each layer, you may have a different number of neurons.

@nbro, No I do not have a concrete example. I have just migrated to another OS so most of my code is lost. But even then in most tutorials the no. is always a power of 2. So I was wondering what reason was there behind it.....


text:
Is Monte Carlo tree search needed in partially observable environments during gameplay?
I understand that with a fully observable environment (chess / go etc) you can run an MCTS with an optimal policy network for future planning purposes. This will allow you to pick actions for gameplay, which will result in max expected return from that state.
However, in a partially observable environment, do we still need to run MCTS during gameplay? Why can't we just pick the max action from the trained optimal policy given the current state? What utility does MCTS serve here?
I am new to reinforcement learning and am trying to understand the purpose of MCTS / planning in partially observable environments.


text:
How should I design a reward function for a NLP problem where two models interoperate?
I would like to design a reward function. I am training two models from the first model that classify set of texts (paragraphs and keywords) and I also got some hidden states. The second model is trying to generate keywords for those paragraphs.
I want to use those hidden states from the first model to give rewards for key phrases that are generated from the second model. I want to know how can I implement this reward function since I have never used it before.

the question sounds a bit confusing to me, can you reframe it explain in more details which models you're using and where RL comes into play? I don't get what kind of policy you want to train. From what I understood you have a model that classify some text into paragraphs and another one that extract keywords for each paragraph, is it correct?


text:
Do the variance and bias belong to the policy or value functions?
Recently, I read many papers on variance and bias. But I am still confused by the two notions, the variance or bias belongs to who? Policy or value? If the variance or bias is large or low, what results will we get?


text:
GPT-2: (Hardware) requirements for fine-tuning the 774M model
I wonder if there's anyone who has actually succeeded in fine-tuning GPT-2's 774M model without using cloud TPU's. My GeForce RTX 2070 SUPER couldn't handle it in previous attempts.
I'm running TensorFlow 1.14.0 with CUDA V 9.1 on Ubuntu 18.04. For fine-tuning I'm using gpt-2-simple.
When fine-tuning using the 77M model, I keep running into OOM errors, such as:
W tensorflow/core/common_runtime/bfc_allocator.cc:314] Allocator (GPU_0_bfc) ran out of memory trying to allocate 6.25MiB (rounded to 6553600).  Current allocation summary follows.
So far I've tried:

Using different a optimizer (RMSPropOptimizer instead of AdamOptimizer)
Setting batch-size to 1
use_memory_saving_gradients
only_train_transformer_layers

Fine-tuning works smoothly on the 355M model.
So what I'm really asking is:

is it possible to fine-tune GPT-2's 774M model without industrial-sized hardware?
if so, please tell me about your successful attempts
apart from hardware-recommendations, how could fine-tuning be optimized to make 77M fit in memory?

Possibly a bit late to the answer, but I doubt you'd be able to run GPT-2 774M in FP32 on 2070 Super which has 8GB VRAM. I know it's not an exact comparison, but fine-tuning BERT Large (345M) in FP32 easily takes more than 10GB of VRAM. You might be able to run GPT-2 774M if you run it in FP16.
Alternatively, you can use Google Collab TPUs which provide at 11GB+ VRAM. Here's a good source listing a few posts about fine tuning GTP-2 1.5B on Google Collab TPUs: https://news.ycombinator.com/item?id=21456025
And here's the notebook itself demonstrating the process: https://colab.research.google.com/drive/1BXry0kcm869-RVHHiY6NZmY9uBzbkf1Q#scrollTo=lP1InuxJTD6a


text:
How can I design a DQN or policy gradient model to explore and collect all optimal solutions?
I am working to use DQN and Policy Gradient reinforcement learning models to solve classic maze escaping problems.
So far, I have been able to train a model, which, after around 100 episodes, quickly explored ONE optimal solution to escape mazes.
However, it is easy to see that for many maze designs, the optimal solutions could be multiple, and I would like to take one step further to collect all optimal and distinguishable solutions.
However, I tried some searches online and so far, the only material I can find is this Learning Diverse Skills. But this seems an obstacle to me. I somewhat believe this seems a classic (?) and an easier problem that should be addressed in the textbook?
Could someone shed light on this matter?


text:
Why are all weights of a neural net updated and not just the weights of the first layer
Why are all weights of a neural net updated and not only the weights of the first hidden layer?
The error-influence of the prediction by the weights of a neural net is calculated using the chain rule. However, the chain rule tells us how the first variable influences the second variable, and so on. Following that logic, we should only update the weights of the first hidden layer. My thought is, that if we backtrack the influence of the first variable but also change the values of the subsequent weights (of the subsequent hidden layer), there is no need to calculate the influence of the first weights in the first place. Where am I wrong?

However, the chain rule tells us how the first variable influences the second variable, and so on. Following that logic, we should only update the weights of the first hidden layer.

I don't see how the second statement follows from the first.
Each weight $w_i$ (not just the ones in the first layer) affects the loss $\mathcal{L}$ according to the partial derivative of $\mathcal{L}$ with respect to $w_i$, i.e. $\frac{\partial \mathcal{L}}{\partial w_i}$. Intuitively, the partial derivative with respect to parameter tells you how the function is changing with respect to that parameter.

My thought is, that if we backtrack the influence of the first variable but also change the values of the subsequent weights (of the subsequent hidden layer), there is no need to calculate the influence of the first weights in the first place.

I am not sure I understand your reasoning, but, typically, you update the parameters only after having computed all the partial derivatives. In other words, first, you compute all partial derivatives, i.e. the gradient with back-propagation (a fancy name to denote the application of the chain rule), then you update the parameters.
Why do you do this? In this case, the loss function is multi-variable function, so it depends on multiple variables. The gradient $\nabla \mathcal{L} = \left[ \frac{\partial \mathcal{L}}{\partial w_1}, \dots, \frac{\partial \mathcal{L}}{\partial w_N} \right]$ represents the direction (note that the gradient is a vector and vectors have direction) towards which your function is increasing or decreasing (depending on the sign of the gradient).


text:
How can I sample the output distribution multiple times when pruning the filters with reinforcement learning?
I was reading the paper Learning to Prune Filters in Convolutional Neural Networks, which is about pruning the CNN filters using reinforcement learning (policy gradient). The paper says that the input for the pruning agent (the agent is a convolutional neural network) is a 2D array of shape (N_l, M_l), where N_l is the number of filters and M_l = m x h x w (m, l and h are filter dimensions), and the output is an array of actions (each element is 0 (unnecessary filter) or 1 (necessary)) and says in order to approximate gradients we have to sample the output M times (using the REINFORCE algorithm).
Since I have one input, how can I sample the output distribution multiple times (without updating the CNN parameters)?
If I'm missing something, please, tell me where I'm wrong

I'm not sure what do you mean by one input. The input to the pruning agent is always the same, it's the convolutional layer $W$ of dimension $m \times h \times w$. The layer is taken from baseline model that is pretrained. The input doesn't change it's always the same. The output of the pruning agent is an array of probabilities to prune a specific filter. For example if you have $3$ filters in a layer, the output of the pruning agent will be array of $3$ elements . Let's say its
\begin{equation}
y = [0.1, 0.6, 0.7]
\end{equation}
Each of these elements represents probability of pruning filter $i$ in layer $W$. So $0.1$ would be probability to prune filter $1$, $0.6$ to prune filter $2$ and $0.7$ to prune filter $3$.
Let's say you sample this distribution $2$ times and you get: $[0, 1, 1], [0, 0, 1]$. That means you would make 2 different models from the original baseline model. First model would have 2nd and 3rd filter pruned in layer $W$, and second model would have 3rd filter pruned. The you run those 2 new models on your train and validation set, calculate objective function $R$. Then you update parameters $\theta$ of your pruning agent based on $R$. The original weights of layer $W$ stay untouched. Then you do another inference of the pruning model $\pi$ with updated parameters $\theta$ (the input is still original $W$). You will get another array of probabilities and you keep repeating previous steps that i described until parameters $\theta$ converge. When they converge you make final pruning.

Thanks for your answer! but how can I get these two arrays [0,1,1],[0,0,1] ?in order to sample I normally would do something like action=model.predict(W), if I sample another time I will get the same action array, won't I ? please tell me what I'm missing.

That's not what sampling means, sampling translated to code (Python) is something like this : random.random() < prob. It's like flipping a coin. If random number is lower than probability that means you set array element to 0 (or 1 depending what you want to do).

In the paper (Algorithm 1) they say sample the output distribution & then write Ai=π(θ,W). I'm confused i think. anyway thank you I will try what you said.

Yes, that's exactly what I said in the previous comment.

So this Ai=π(θ,W) means sampling the output distribution randomly ? I thought Ai=π(θ,W) means do something like Ai=agent.predict(W). Thanks :)

They write $\mathbf A_i \sim \pi(W, \theta)$ in Algorithm 1, this $\sim$ means they sample like I described.


text:
What is the relationship between PAC learning and classic parameter estimation theorems?
What are the differences and similarities between PAC learning and classic parameter estimation theorems (e.g. consistency results when estimating parameters, e.g. with MLE)?

The concept of "consistency" also exists in learning theory, and it's been defined differently in different contexts. For example, see the paper Learning and Consistency (1995), by Wiehagen and Zeugmann, or chapter 7 of Understanding Machine Learning: From Theory to Algorithms (2014) by Shalev-Shwartz and Ben-David. In statistics, consistency is related to the convergence guarantees of the estimator.

The relationship is more fundamental: both seek to modify parameters to fit a model, both provide asymptotic bounds on how many observations are needed. The question is, why isn't PAC learning a strictly stronger version of classical parameter estimation? For example, you can view the estimation of a real number with AWGN in the MLE framework and in the PAC framework right? Both provide bounds on how many observations are needed to stay predict the probabilistic accuracy of the estimator. But it appears statistics and PAC learning are very separate fields.

I didn't talk about other things because your question is too broad (and could be closed as too broad). I only talked about consistency because that's the only specific "classic results" you mentioned. I suggest you narrow your question.


text:
How does the DQN loss from td_targets against q_values make sense?
Why td_loss is calculated from (td_targets against q_values)?
Why I am lost is because:

q_values is just the probability of action. It does not have a reward and discount.
td_targets does have rewards + discounts * next_q_values. Somemore next_q_values is next state.

How both td_targets and q_values can minus (or Huber or MSE) to get lost work?
td_error = valid_mask * (td_targets - q_values)
td_loss = valid_mask * td_errors_loss_fn(td_targets, q_values)

td_loss = valid_mask * td_errors_loss_fn(td_targets, q_values)

First of all, DQN is off-policy learning. That means, you are following the behavior policy(epsilon greedy policy) but still learning about the optimal policy or target policy(greedy policy). Td_target in DQN is the estimation of our current state's optimal action-value function independent of the policy we are following (since we are picking next state's action-value from target policy) and q_values(as you referred) is what you get following the behavior policy. While using this kind of update, you are improving both the behavior policy and the target policy.

Td_target is the estimation of our current state's but it q value is base on next state.

In DQN, the td_target is the optimal q value and you are subtracting it from your estimated q value.

As i mentioned above, td_targets is tf.stop_gradient(rewards + discounts * next_q_values). It is not just simple next_q_values.

In DQN, td_targets = rewards + discount*(max q value of next state). The max term comes from the target policy.  If the explanation did not satisfy you, can you please explain to me your question in more detail(without adding the code section)?

I am just do not understand how come it working. Maybe i need more this kind of lost solution on other senario. Just for appreciation and have intuition.


text:
How to estimate the convolutional representation of a graph from its similarity to other graph convolutional representation?
Suppose we have two graphs A and B disconnected to each other (let's say 2-hops each), within a larger graph. If the convolutional representation of graph A is known, is it possible to estimate the definitive convolutional representation of graph B based on its similarity to graph A?
If yes, what do you think is the simplest (arithmetically) way to do this, which algorithm can help me to do this? You can assume that precision requirements are not important.


text:
Why are denser layers needed in computer vision neural nets?
Many neural net architectures for computer vision tasks use several convolutional layers and then several fully-connected (or dense) layers. While the reasons for using convolutional layers are clear to me, I don't understand why the dense layers are needed. Can't high accuracy be achieved with only convolutional layers?

Convolutional layers are added in order to extract features from the image (like edges, corners, textures). After extracting those features, you feed them to a fully connected neural network to get the prediction.
Let's take an example, consider you want to classify the cat's image. But you decided to do this by only using the convolutional layer. So, you feed the image to a convolutional layer. After passing through some layer it extracted the key features from the cat's image and at the end of the layer, you add a single neuron(since we have a single class to predict). Now it's ready to classify it as a cat. But unfortunately, it's misclassified the image. Why? Now let us answer the question.
Since convolutional layers are not fully connected layers, the neuron added at the last layer is only connected with a handful of neurons from the previous layer. So it misses the key features (encapsulated by some other neurons it is not connected with them) in order to detect this image as a cat. In order to get the prediction right, you have to add a dense layer(a fully connected layer) at the end of the convolutional layer so that it can get all of those extracted features.
If this does not satisfy you, please, ask the question more specifically and also edit the question because linear layer does not call fully connected layer.