Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Characterizing the Exact Behaviors of Temporal Difference Learning
  Algorithms Using Markov Jump Linear System Theory   In this paper, we provide a unified analysis of temporal difference learning
algorithms with linear function approximators by exploiting their connections
to Markov jump linear systems (MJLS). We tailor the MJLS theory developed in
the control community to characterize the exact behaviors of the first and
second order moments of a large family of temporal difference learning
algorithms. For both the IID and Markov noise cases, we show that the evolution
of some augmented versions of the mean and covariance matrix of the TD
estimation error exactly follows the trajectory of a deterministic linear
time-invariant (LTI) dynamical system. Applying the well-known LTI system
theory, we obtain closed-form expressions for the mean and covariance matrix of
the TD estimation error at any time step. We provide a tight matrix spectral
radius condition to guarantee the convergence of the covariance matrix of the
TD estimation error, and perform a perturbation analysis to characterize the
dependence of the TD behaviors on learning rate. For the IID case, we provide
an exact formula characterizing how the mean and covariance matrix of the TD
estimation error converge to the steady state values. For the Markov case, we
use our formulas to explain how the behaviors of TD learning algorithms are
affected by learning rate and the underlying Markov chain. For both cases,
upper and lower bounds for the mean square TD error are provided. The mean
square TD error is shown to converge linearly to an exact limit.

13
The categories of this paper are: quant-ph
34
The categories of this paper are: chem-ph
32
The categories of this paper are: atom-ph
26
The categories of this paper are: bayes-an
20
The categories of this paper are: acc-phys
21
The categories of this paper are: adap-org
10
The categories of this paper are: q-bio
8
The categories of this paper are: hep-ex
11
The categories of this paper are: hep-lat
31
The categories of this paper are: ao-sci
9
The categories of this paper are: nlin
19
The categories of this paper are: eess
37
The categories of this paper are: supr-con
35
The categories of this paper are: mtrl-th
29
The categories of this paper are: funct-an
15
The categories of this paper are: nucl-ex