Modern neural architectures critically rely on attention for mapping structured inputs to sequences. In this paper we show that prevalent attention architectures do not adequately model the dependence among the attention and output tokens across a predicted sequence.
 We present an alternative architecture called  Posterior Attention Models that after a principled factorization of the full joint distribution of the attention and output variables, proposes two major changes.   First, the position where attention is marginalized is changed from the input to the output. Second, the attention propagated to the next decoding stage is a posterior attention distribution conditioned on the output. Empirically on five translation and two morphological inflection tasks the proposed posterior attention models yield better BLEU score and alignment accuracy than existing attention models.
Computing attention based on posterior distribution leads to more meaningful attention and better performance This paper proposes a sequence to sequence model where attention is treated as a latent variable, and derives novel inference procedures for this model, obtaining improvements in machine translation and morphological inflection generation tasks. This paper presents a novel posterior attention model for seq2seq problems