Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: SHAQ: Single Headed Attention with Quasi-Recurrence   Natural Language Processing research has recently been dominated by large
scale transformer models. Although they achieve state of the art on many
important language tasks, transformers often require expensive compute
resources, and days spanning to weeks to train. This is feasible for
researchers at big tech companies and leading research universities, but not
for scrappy start-up founders, students, and independent researchers. Stephen
Merity's SHA-RNN, a compact, hybrid attention-RNN model, is designed for
consumer-grade modeling as it requires significantly fewer parameters and less
training time to reach near state of the art results. We analyze Merity's model
here through an exploratory model analysis over several units of the
architecture considering both training time and overall quality in our
assessment. Ultimately, we combine these findings into a new architecture which
we call SHAQ: Single Headed Attention Quasi-recurrent Neural Network. With our
new architecture we achieved similar accuracy results as the SHA-RNN while
accomplishing a 4x speed boost in training.

10
The categories of this paper are: q-bio
14
The categories of this paper are: math-ph
33
The categories of this paper are: plasm-ph
37
The categories of this paper are: supr-con
29
The categories of this paper are: funct-an
24
The categories of this paper are: dg-ga
5
The categories of this paper are: gr-qc
26
The categories of this paper are: bayes-an
20
The categories of this paper are: acc-phys
18
The categories of this paper are: econ
16
The categories of this paper are: stat
0
The categories of this paper are: hep-ph
17
The categories of this paper are: q-fin
23
The categories of this paper are: solv-int
9
The categories of this paper are: nlin
2
The categories of this paper are: math