MiniMax-01: Scaling Foundation Models with Lightning Attention
Paper • 2501.08313 • Published • 304
Attention
-
-
Lizard: An Efficient Linearization Framework for Large Language Models
Paper • 2507.09025 • Published • 19 -
On the Expressiveness of Softmax Attention: A Recurrent Neural Network Perspective
Paper • 2507.23632 • Published • 6 -
Causal Attention with Lookahead Keys
Paper • 2509.07301 • Published • 21 -
Hybrid Architectures for Language Models: Systematic Analysis and Design Insights
Paper • 2510.04800 • Published • 37 -
Less is More: Recursive Reasoning with Tiny Networks
Paper • 2510.04871 • Published • 519 -
Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention
Paper • 2510.04212 • Published • 26 -
Reactive Transformer (RxT) -- Stateful Real-Time Processing for Event-Driven Reactive Language Models
Paper • 2510.03561 • Published • 25 -
Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning
Paper • 2510.19338 • Published • 116 -
Kimi Linear: An Expressive, Efficient Attention Architecture
Paper • 2510.26692 • Published • 138 -
DoPE: Denoising Rotary Position Embedding
Paper • 2511.09146 • Published • 98 -
SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space
Paper • 2511.20102 • Published • 28 -
InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Models
Paper • 2512.08829 • Published • 23 -
MHLA: Restoring Expressivity of Linear Attention via Token-Level Multi-Head
Paper • 2601.07832 • Published • 53 -
Reinforced Attention Learning
Paper • 2602.04884 • Published • 30 -
Prism: Spectral-Aware Block-Sparse Attention
Paper • 2602.08426 • Published • 38 -
SLA2: Sparse-Linear Attention with Learnable Routing and QAT
Paper • 2602.12675 • Published • 59 -
TriAttention: Efficient Long Reasoning with Trigonometric KV Compression
Paper • 2604.04921 • Published • 117 -
Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation
Paper • 2604.10098 • Published • 82 -
KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs
Paper • 2604.13226 • Published • 11 -
Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
Paper • 2605.16928 • Published • 99 -
FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention
Paper • 2606.09079 • Published • 67 -
Kwai Keye-VL-2.0 Technical Report
Paper • 2606.10651 • Published • 194 -
MiniMax Sparse Attention
Paper • 2606.13392 • Published • 154 -
Rethinking the Role of Efficient Attention in Hybrid Architectures
Paper • 2606.15378 • Published • 20 -
Morphing into Hybrid Attention Models
Paper • 2606.30562 • Published • 52 -
Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
Paper • 2607.02980 • Published • 84 -
Multi-Head Attention Residuals
Paper • 2607.27230 • Published • 10