Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Controlling Computation versus Quality for Neural Sequence Models   Most neural networks utilize the same amount of compute for every example
independent of the inherent complexity of the input. Further, methods that
adapt the amount of computation to the example focus on finding a fixed
inference-time computational graph per example, ignoring any external
computational budgets or varying inference time limitations. In this work, we
utilize conditional computation to make neural sequence models (Transformer)
more efficient and computation-aware during inference. We first modify the
Transformer architecture, making each set of operations conditionally
executable depending on the output of a learned control network. We then train
this model in a multi-task setting, where each task corresponds to a particular
computation budget. This allows us to train a single model that can be
controlled to operate on different points of the computation-quality trade-off
curve, depending on the available computation budget at inference time. We
evaluate our approach on two tasks: (i) WMT English-French Translation and (ii)
Unsupervised representation learning (BERT). Our experiments demonstrate that
the proposed Conditional Computation Transformer (CCT) is competitive with
vanilla Transformers when allowed to utilize its full computational budget,
while improving significantly over computationally equivalent baselines when
operating on smaller computational budgets.

17
The categories of this paper are: q-fin
0
The categories of this paper are: hep-ph
23
The categories of this paper are: solv-int
28
The categories of this paper are: alg-geom
24
The categories of this paper are: dg-ga
27
The categories of this paper are: comp-gas
3
The categories of this paper are: physics
25
The categories of this paper are: patt-sol
4
The categories of this paper are: cond-mat
37
The categories of this paper are: supr-con
26
The categories of this paper are: bayes-an
21
The categories of this paper are: adap-org
11
The categories of this paper are: hep-lat
31
The categories of this paper are: ao-sci
13
The categories of this paper are: quant-ph
30
The categories of this paper are: q-alg