Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Mirror Descent Policy Optimization   Mirror descent (MD), a well-known first-order method in constrained convex
optimization, has recently been shown as an important tool to analyze
trust-region algorithms in reinforcement learning (RL). However, there remains
a considerable gap between such theoretically analyzed algorithms and the ones
used in practice. Inspired by this, we propose an efficient RL algorithm,
called {\em mirror descent policy optimization} (MDPO). MDPO iteratively
updates the policy by {\em approximately} solving a trust-region problem, whose
objective function consists of two terms: a linearization of the standard RL
objective and a proximity term that restricts two consecutive policies to be
close to each other. Each update performs this approximation by taking multiple
gradient steps on this objective function. We derive {\em on-policy} and {\em
off-policy} variants of MDPO, while emphasizing important design choices
motivated by the existing theory of MD in RL. We highlight the connections
between on-policy MDPO and two popular trust-region RL algorithms: TRPO and
PPO, and show that explicitly enforcing the trust-region constraint is in fact
{\em not} a necessity for high performance gains in TRPO. We then show how the
popular soft actor-critic (SAC) algorithm can be derived by slight
modifications of off-policy MDPO. Overall, MDPO is derived from the MD
principles, offers a unified approach to viewing a number of popular RL
algorithms, and performs better than or on-par with TRPO, PPO, and SAC in a
number of continuous control tasks. Code is available at
\url{https://github.com/manantomar/Mirror-Descent-Policy-Optimization}.

26
The categories of this paper are: bayes-an
12
The categories of this paper are: nucl-th
4
The categories of this paper are: cond-mat
34
The categories of this paper are: chem-ph
18
The categories of this paper are: econ
19
The categories of this paper are: eess
21
The categories of this paper are: adap-org
13
The categories of this paper are: quant-ph
5
The categories of this paper are: gr-qc
23
The categories of this paper are: solv-int
25
The categories of this paper are: patt-sol
36
The categories of this paper are: cmp-lg
37
The categories of this paper are: supr-con
2
The categories of this paper are: math
8
The categories of this paper are: hep-ex
6
The categories of this paper are: astro-ph