Recent approaches have successfully demonstrated the benefits of learning the parameters of shallow networks in hyperbolic space. We extend this line of work by imposing hyperbolic geometry on the embeddings used to compute the ubiquitous attention mechanisms for different neural networks architectures. By only changing the geometry of embedding of object representations, we can use the embedding space more efficiently without increasing the number of parameters of the model. Mainly as the number of objects grows exponentially for any semantic distance from the query, hyperbolic geometry  --as opposed to Euclidean geometry-- can encode those objects without having any interference. Our method shows improvements in generalization on neural machine translation on WMT'14 (English to German), learning on graphs (both on synthetic and real-world graph tasks) and visual question answering (CLEVR) tasks while keeping the neural representations compact.
We propose to incorporate inductive biases and operations coming from hyperbolic geometry to improve the attention mechanism of the neural networks. This paper replaces the dot-product similarity used in attention mechanisms with the negative hyperbolic distance, and applies it to the existing Transformer model, graph attention networks, and Relation Networks The authors propose a novel approach to improve relational-attention by changing the matching and aggregation functions to use hyperbolic geometric. 