Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Sample Complexity Bounds for Two Timescale Value-based Reinforcement
  Learning Algorithms   Two timescale stochastic approximation (SA) has been widely used in
value-based reinforcement learning algorithms. In the policy evaluation
setting, it can model the linear and nonlinear temporal difference learning
with gradient correction (TDC) algorithms as linear SA and nonlinear SA,
respectively. In the policy optimization setting, two timescale nonlinear SA
can also model the greedy gradient-Q (Greedy-GQ) algorithm. In previous
studies, the non-asymptotic analysis of linear TDC and Greedy-GQ has been
studied in the Markovian setting, with diminishing or accuracy-dependent
stepsize. For the nonlinear TDC algorithm, only the asymptotic convergence has
been established. In this paper, we study the non-asymptotic convergence rate
of two timescale linear and nonlinear TDC and Greedy-GQ under Markovian
sampling and with accuracy-independent constant stepsize. For linear TDC, we
provide a novel non-asymptotic analysis and show that it attains an
$\epsilon$-accurate solution with the optimal sample complexity of
$\mathcal{O}(\epsilon^{-1}\log(1/\epsilon))$ under a constant stepsize. For
nonlinear TDC and Greedy-GQ, we show that both algorithms attain
$\epsilon$-accurate stationary solution with sample complexity
$\mathcal{O}(\epsilon^{-2})$. It is the first non-asymptotic convergence result
established for nonlinear TDC under Markovian sampling and our result for
Greedy-GQ outperforms the previous result orderwisely by a factor of
$\mathcal{O}(\epsilon^{-1}\log(1/\epsilon))$.

37
The categories of this paper are: supr-con
11
The categories of this paper are: hep-lat
35
The categories of this paper are: mtrl-th
32
The categories of this paper are: atom-ph
18
The categories of this paper are: econ
17
The categories of this paper are: q-fin
34
The categories of this paper are: chem-ph
31
The categories of this paper are: ao-sci
28
The categories of this paper are: alg-geom
21
The categories of this paper are: adap-org
23
The categories of this paper are: solv-int
7
The categories of this paper are: hep-th
9
The categories of this paper are: nlin
20
The categories of this paper are: acc-phys
29
The categories of this paper are: funct-an
26
The categories of this paper are: bayes-an