    We establish the relation between Distributional RL and the Upper Confidence Bound (UCB) approach to exploration.
     In this paper we show that the density of the Q function estimated by Distributional RL can be successfully used for the estimation of UCB. This approach does not require counting and, therefore, generalizes well to the Deep RL. We also point to the asymmetry of the empirical densities estimated by the Distributional RL algorithms like QR-DQN. This observation leads to the reexamination of the variance's performance in the UCB type approach to exploration. We introduce truncated variance as an alternative estimator of the UCB and a novel algorithm based on it. We empirically show that newly introduced algorithm achieves better performance in multi-armed bandits setting. Finally, we extend this approach to high-dimensional setting and test it on the Atari 2600 games. New approach achieves better performance compared to QR-DQN in 26 of games, 13 ties out of 49 games.
Exploration using Distributional RL and truncagted variance. Presents an RL method to manage exploration-explotation trade-offs via UCB techniques. A method to use the distribution learned by Quantile Regression DQN for exploration, in place of the usual epsilon-greedy strategy. Proposes new algorithsms (QUCB and QUCB+) to handle the exploration tradeoff in Multi-Armed Bendits and more generally in Reinforcement Learning