Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Hash Layers For Large Sparse Models   We investigate the training of sparse layers that use different parameters
for different inputs based on hashing in large Transformer models.
Specifically, we modify the feedforward layer to hash to different sets of
weights depending on the current token, over all tokens in the sequence. We
show that this procedure either outperforms or is competitive with
learning-to-route mixture-of-expert methods such as Switch Transformers and
BASE Layers, while requiring no routing parameters or extra terms in the
objective function such as a load balancing loss, and no sophisticated
assignment algorithm. We study the performance of different hashing techniques,
hash sizes and input features, and show that balanced and random hashes focused
on the most local features work best, compared to either learning clusters or
using longer-range context. We show our approach works well both on large
language modeling and dialogue tasks, and on downstream fine-tuning tasks.

11
The categories of this paper are: hep-lat
23
The categories of this paper are: solv-int
2
The categories of this paper are: math
8
The categories of this paper are: hep-ex
9
The categories of this paper are: nlin
31
The categories of this paper are: ao-sci
10
The categories of this paper are: q-bio
26
The categories of this paper are: bayes-an
35
The categories of this paper are: mtrl-th
17
The categories of this paper are: q-fin
13
The categories of this paper are: quant-ph
12
The categories of this paper are: nucl-th
28
The categories of this paper are: alg-geom
27
The categories of this paper are: comp-gas
34
The categories of this paper are: chem-ph
5
The categories of this paper are: gr-qc