Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Policy-Gradient Algorithms Have No Guarantees of Convergence in Linear
  Quadratic Games   We show by counterexample that policy-gradient algorithms have no guarantees
of even local convergence to Nash equilibria in continuous action and state
space multi-agent settings. To do so, we analyze gradient-play in N-player
general-sum linear quadratic games, a classic game setting which is recently
emerging as a benchmark in the field of multi-agent learning. In such games the
state and action spaces are continuous and global Nash equilibria can be found
be solving coupled Ricatti equations. Further, gradient-play in LQ games is
equivalent to multi agent policy-gradient. We first show that these games are
surprisingly not convex games. Despite this, we are still able to show that the
only critical points of the gradient dynamics are global Nash equilibria. We
then give sufficient conditions under which policy-gradient will avoid the Nash
equilibria, and generate a large number of general-sum linear quadratic games
that satisfy these conditions. In such games we empirically observe the players
converging to limit cycles for which the time average does not coincide with a
Nash equilibrium. The existence of such games indicates that one of the most
popular approaches to solving reinforcement learning problems in the classic
reinforcement learning setting has no local guarantee of convergence in
multi-agent settings. Further, the ease with which we can generate these
counterexamples suggests that such situations are not mere edge cases and are
in fact quite common.

12
The categories of this paper are: nucl-th
17
The categories of this paper are: q-fin
20
The categories of this paper are: acc-phys
9
The categories of this paper are: nlin
11
The categories of this paper are: hep-lat
28
The categories of this paper are: alg-geom
21
The categories of this paper are: adap-org
22
The categories of this paper are: chao-dyn
0
The categories of this paper are: hep-ph
27
The categories of this paper are: comp-gas
13
The categories of this paper are: quant-ph
5
The categories of this paper are: gr-qc
10
The categories of this paper are: q-bio
3
The categories of this paper are: physics
31
The categories of this paper are: ao-sci
23
The categories of this paper are: solv-int