Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: The Uncertainty Bellman Equation and Exploration   We consider the exploration/exploitation problem in reinforcement learning.
For exploitation, it is well known that the Bellman equation connects the value
at any time-step to the expected value at subsequent time-steps. In this paper
we consider a similar \textit{uncertainty} Bellman equation (UBE), which
connects the uncertainty at any time-step to the expected uncertainties at
subsequent time-steps, thereby extending the potential exploratory benefit of a
policy beyond individual time-steps. We prove that the unique fixed point of
the UBE yields an upper bound on the variance of the posterior distribution of
the Q-values induced by any policy. This bound can be much tighter than
traditional count-based bonuses that compound standard deviation rather than
variance. Importantly, and unlike several existing approaches to optimism, this
method scales naturally to large systems with complex generalization.
Substituting our UBE-exploration strategy for $\epsilon$-greedy improves DQN
performance on 51 out of 57 games in the Atari suite.

23
The categories of this paper are: solv-int
31
The categories of this paper are: ao-sci
9
The categories of this paper are: nlin
3
The categories of this paper are: physics
24
The categories of this paper are: dg-ga
11
The categories of this paper are: hep-lat
8
The categories of this paper are: hep-ex
33
The categories of this paper are: plasm-ph
30
The categories of this paper are: q-alg
21
The categories of this paper are: adap-org
36
The categories of this paper are: cmp-lg
35
The categories of this paper are: mtrl-th
4
The categories of this paper are: cond-mat
10
The categories of this paper are: q-bio
0
The categories of this paper are: hep-ph
28
The categories of this paper are: alg-geom