Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: An Analysis of Attention Mechanisms: The Case of Word Sense
  Disambiguation in Neural Machine Translation   Recent work has shown that the encoder-decoder attention mechanisms in neural
machine translation (NMT) are different from the word alignment in statistical
machine translation. In this paper, we focus on analyzing encoder-decoder
attention mechanisms, in the case of word sense disambiguation (WSD) in NMT
models. We hypothesize that attention mechanisms pay more attention to context
tokens when translating ambiguous words. We explore the attention distribution
patterns when translating ambiguous nouns. Counter-intuitively, we find that
attention mechanisms are likely to distribute more attention to the ambiguous
noun itself rather than context tokens, in comparison to other nouns. We
conclude that attention mechanism is not the main mechanism used by NMT models
to incorporate contextual information for WSD. The experimental results suggest
that NMT models learn to encode contextual information necessary for WSD in the
encoder hidden states. For the attention mechanism in Transformer models, we
reveal that the first few layers gradually learn to "align" source and target
tokens and the last few layers learn to extract features from the related but
unaligned context tokens.

2
The categories of this paper are: math
13
The categories of this paper are: quant-ph
18
The categories of this paper are: econ
23
The categories of this paper are: solv-int
14
The categories of this paper are: math-ph
32
The categories of this paper are: atom-ph
4
The categories of this paper are: cond-mat
35
The categories of this paper are: mtrl-th
7
The categories of this paper are: hep-th
12
The categories of this paper are: nucl-th
28
The categories of this paper are: alg-geom
11
The categories of this paper are: hep-lat
25
The categories of this paper are: patt-sol
16
The categories of this paper are: stat
5
The categories of this paper are: gr-qc
29
The categories of this paper are: funct-an