Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Data-Informed Global Sparseness in Attention Mechanisms for Deep Neural
  Networks   The attention mechanism is a key component of the neural revolution in
Natural Language Processing (NLP). As the size of attention-based models has
been scaling with the available computational resources, a number of pruning
techniques have been developed to detect and to exploit sparseness in such
models in order to make them more efficient. The majority of such efforts have
focused on looking for attention patterns and then hard-coding them to achieve
sparseness, or pruning the weights of the attention mechanisms based on
statistical information from the training data. Here, we marry these two lines
of research by proposing Attention Pruning (AP): a novel pruning framework that
collects observations about the attention patterns in a fixed dataset and then
induces a global sparseness mask for the model. This can save 90% of the
attention computation for language modelling and about 50% for machine
translation and for solving GLUE tasks, while maintaining the quality of the
results. Moreover, using our method, we discovered important distinctions
between self- and cross-attention patterns, which could guide future NLP
research in attention-based modelling. Our framework can in principle speed up
any model that uses attention mechanism, thus helping develop better models for
existing or for new NLP applications. Our implementation is available at
https://github.com/irugina/AP.

27
The categories of this paper are: comp-gas
28
The categories of this paper are: alg-geom
26
The categories of this paper are: bayes-an
35
The categories of this paper are: mtrl-th
14
The categories of this paper are: math-ph
7
The categories of this paper are: hep-th
36
The categories of this paper are: cmp-lg
19
The categories of this paper are: eess
11
The categories of this paper are: hep-lat
31
The categories of this paper are: ao-sci
15
The categories of this paper are: nucl-ex
18
The categories of this paper are: econ
4
The categories of this paper are: cond-mat
33
The categories of this paper are: plasm-ph
16
The categories of this paper are: stat
0
The categories of this paper are: hep-ph