Topological Vision Transformer with 2D-level block Toeplitz masking mechanism for image processing with topology-preserving-attention, described in the paper: https://arxiv.org/abs/2107.07999. TopVit is built on the ViT Transformer from https://github.com/google-research/scenic and modulates the attention block by directly incorporating a 2D structural inductive prior. It also replaces softmax-kernel with ReLU-kernel in attention. That choice makes the attention mechanism numerically equivalent (not just approximately) to its implicit (sub-quadratic) attention variation obtained via: (a) the Performer method (https://arxiv.org/abs/2009.14794) and (b) implicit masking methods from https://arxiv.org/abs/2107.07999. The currently implemented variant is the explicit one.
Topological Vision Transformer with 2D-level block Toeplitz masking mechanism for image processing with topology-preserving-attention, described in the paper: https://arxiv.org/abs/2107.07999. TopVit is built on the ViT Transformer from https://github.com/google-research/scenic and modulates the attention block by directly incorporating a 2D structural inductive prior. It also replaces softmax-kernel with ReLU-kernel in attention. That choice makes the attention mechanism numerically equivalent (not just approximately) to its implicit (sub-quadratic) attention variation obtained via: (a) the Performer method (https://arxiv.org/abs/2009.14794) and (b)
implicit masking methods from https://arxiv.org/abs/2107.07999.
The currently implemented variant is the explicit one.

PiperOrigin-RevId: 478680756

google-research/google-research,google-research/google-research,google-research/google-research,google-research/google-research,google-research/google-research,google-research/google-research,google-research/google-research,google-research/google-research,google-research/google-research,google-research/google-research,google-research/google-research