Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Speech-XLNet: Unsupervised Acoustic Model Pretraining For Self-Attention
  Networks   Self-attention network (SAN) can benefit significantly from the
bi-directional representation learning through unsupervised pretraining
paradigms such as BERT and XLNet. In this paper, we present an XLNet-like
pretraining scheme "Speech-XLNet" for unsupervised acoustic model pretraining
to learn speech representations with SAN. The pretrained SAN is finetuned under
the hybrid SAN/HMM framework. We conjecture that by shuffling the speech frame
orders, the permutation in Speech-XLNet serves as a strong regularizer to
encourage the SAN to make inferences by focusing on global structures through
its attention weights. In addition, Speech-XLNet also allows the model to
explore the bi-directional contexts for effective speech representation
learning. Experiments on TIMIT and WSJ demonstrate that Speech-XLNet greatly
improves the SAN/HMM performance in terms of both convergence speed and
recognition accuracy compared to the one trained from randomly initialized
weights. Our best systems achieve a relative improvement of 11.9% and 8.3% on
the TIMIT and WSJ tasks respectively. In particular, the best system achieves a
phone error rate (PER) of 13.3% on the TIMIT test set, which to our best
knowledge, is the lowest PER obtained from a single system.

15
The categories of this paper are: nucl-ex
33
The categories of this paper are: plasm-ph
0
The categories of this paper are: hep-ph
27
The categories of this paper are: comp-gas
14
The categories of this paper are: math-ph
31
The categories of this paper are: ao-sci
30
The categories of this paper are: q-alg
11
The categories of this paper are: hep-lat
32
The categories of this paper are: atom-ph
20
The categories of this paper are: acc-phys
17
The categories of this paper are: q-fin
37
The categories of this paper are: supr-con
23
The categories of this paper are: solv-int
3
The categories of this paper are: physics
24
The categories of this paper are: dg-ga
4
The categories of this paper are: cond-mat