Why semi-gradient is used instead of the true gradient in Q-learning? In reinforcement learning, with function approximation, a popular cost function is the Mean value error.



This involves a target value V_pi and a current value estimate V_hat.
When deriving the update rule for gradient descent learning, people just ignore V_pi-s dependence on the parameters, using a semi-gradient instead of the true gradient.
Why is this? Is it difficult to calculate the true gradient?
The problem does not come from the above equation but from the bootstrapping method. Sutton's reinforcement learning book (2nd) gave a pretty good explanation to the problem in Chapter 9.3.
The idea here is you cannot obtain the true value for $V_\pi$, but you want to approximate it. My understand is that when bootstrapping, like with the TD method, $V_\pi$ depends on $w$. This breaks the assumption that $V_\pi$ is independent of $w$ and thus the gradient is not the true gradient and we call it semi-gradient. However, if you are not bootstrapping, as with Monte Carlo methods, $V_\pi$ will be unbiased and the above will be guaranteed to converge to a local minimal.