Reinforcement Learning (RL) can model complex behavior policies for goal-directed sequential decision making tasks. A hallmark of RL algorithms is Temporal Difference (TD) learning: value function for the current state is moved towards a bootstrapped target that is estimated using the next state's value function. lambda-returns define the target of the RL agent as a weighted combination of rewards estimated by using multiple many-step look-aheads. Although mathematically tractable, the use of  exponentially decaying weighting of n-step returns based targets in lambda-returns is a rather ad-hoc design choice. Our major contribution  is that we propose a generalization of lambda-returns called Confidence-based Autodidactic Returns (CAR), wherein the RL agent learns the weighting of the n-step returns in an end-to-end manner. In contrast to lambda-returns wherein the RL agent is restricted to use an exponentially decaying weighting scheme, CAR allows the agent to learn to decide how much it wants to weigh the n-step returns based targets. Our experiments, in addition to showing the efficacy of CAR, also empirically demonstrate that using sophisticated weighted mixtures of multi-step returns (like CAR and lambda-returns) considerably outperforms the use of n-step returns. We perform our experiments on the  Asynchronous Advantage Actor Critic (A3C) algorithm in the Atari 2600 domain.
A novel way to generalize lambda-returns by allowing the RL agent to decide how much it wants to weigh each of the n-step returns. Extends the A3C algorithm with lambda returns, and proposes an approach for learning the weights of the returns. The authors present confidence-based autodidactic returns, a Deep learning RL method to adjust the weights of an eligibility vector in TD(lambda)-like value estimation to favour more stable estimates of the state.