Temporal Difference learning with function approximation has been widely used recently and has led to several successful results.   However, compared with the original tabular-based methods, one major drawback of temporal difference learning with neural networks and other function approximators is that they tend to over-generalize across temporally successive states, resulting in slow convergence and even instability. In this work, we propose a novel TD learning method, Hadamard product Regularized TD (HR-TD), that reduces over-generalization and thus leads to faster convergence. This approach can be easily applied to both linear and nonlinear function approximators. 
 HR-TD is evaluated on several linear and nonlinear benchmark domains, where we show improvement in learning behavior and performance.
A regularization technique for TD learning that avoids temporal over-generalization, especially in Deep Networks A variation on temporal difference learning for the function approximation case that attempts to resolve the issue of over-generalization across temporally-successive states. The paper introduces HR-TD, a variation of the TD(0) algorithm, meant to improve the over-generalization problem in conventional TD