Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Optimization Issues in KL-Constrained Approximate Policy Iteration   Many reinforcement learning algorithms can be seen as versions of approximate
policy iteration (API). While standard API often performs poorly, it has been
shown that learning can be stabilized by regularizing each policy update by the
KL-divergence to the previous policy. Popular practical algorithms such as
TRPO, MPO, and VMPO replace regularization by a constraint on KL-divergence of
consecutive policies, arguing that this is easier to implement and tune. In
this work, we study this implementation choice in more detail. We compare the
use of KL divergence as a constraint vs. as a regularizer, and point out
several optimization issues with the widely-used constrained approach. We show
that the constrained algorithm is not guaranteed to converge even on simple
problem instances where the constrained problem can be solved exactly, and in
fact incurs linear expected regret. With approximate implementation using
softmax policies, we show that regularization can improve the optimization
landscape of the original objective. We demonstrate these issues empirically on
several bandit and RL environments.

3
The categories of this paper are: physics
14
The categories of this paper are: math-ph
32
The categories of this paper are: atom-ph
36
The categories of this paper are: cmp-lg
28
The categories of this paper are: alg-geom
21
The categories of this paper are: adap-org
7
The categories of this paper are: hep-th
35
The categories of this paper are: mtrl-th
5
The categories of this paper are: gr-qc
31
The categories of this paper are: ao-sci
6
The categories of this paper are: astro-ph
19
The categories of this paper are: eess
26
The categories of this paper are: bayes-an
23
The categories of this paper are: solv-int
22
The categories of this paper are: chao-dyn
33
The categories of this paper are: plasm-ph