Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Revisiting Prioritized Experience Replay: A Value Perspective   Experience replay enables off-policy reinforcement learning (RL) agents to
utilize past experiences to maximize the cumulative reward. Prioritized
experience replay that weighs experiences by the magnitude of their
temporal-difference error ($|\text{TD}|$) significantly improves the learning
efficiency. But how $|\text{TD}|$ is related to the importance of experience is
not well understood. We address this problem from an economic perspective, by
linking $|\text{TD}|$ to value of experience, which is defined as the value
added to the cumulative reward by accessing the experience. We theoretically
show the value metrics of experience are upper-bounded by $|\text{TD}|$ for
Q-learning. Furthermore, we successfully extend our theoretical framework to
maximum-entropy RL by deriving the lower and upper bounds of these value
metrics for soft Q-learning, which turn out to be the product of $|\text{TD}|$
and "on-policyness" of the experiences. Our framework links two important
quantities in RL: $|\text{TD}|$ and value of experience. We empirically show
that the bounds hold in practice, and experience replay using the upper bound
as priority improves maximum-entropy RL in Atari games.

25
The categories of this paper are: patt-sol
15
The categories of this paper are: nucl-ex
36
The categories of this paper are: cmp-lg
30
The categories of this paper are: q-alg
18
The categories of this paper are: econ
21
The categories of this paper are: adap-org
13
The categories of this paper are: quant-ph
22
The categories of this paper are: chao-dyn
9
The categories of this paper are: nlin
0
The categories of this paper are: hep-ph
31
The categories of this paper are: ao-sci
24
The categories of this paper are: dg-ga
7
The categories of this paper are: hep-th
12
The categories of this paper are: nucl-th
5
The categories of this paper are: gr-qc
14
The categories of this paper are: math-ph