In Search of Needles in a 10M Haystack: Recurrent Memory Finds What LLMs Miss
Paper • 2402.10790 • Published • 42
Papers - Context
-
-
LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaboration
Paper • 2402.11550 • Published • 19 -
A Neural Conversational Model
Paper • 1506.05869 • Published • 2 -
Data Engineering for Scaling Language Models to 128K Context
Paper • 2402.10171 • Published • 25 -
World Model on Million-Length Video And Language With RingAttention
Paper • 2402.08268 • Published • 40 -
GrowLength: Accelerating LLMs Pretraining by Progressively Growing Training Length
Paper • 2310.00576 • Published • 2 -
Ring Attention with Blockwise Transformers for Near-Infinite Context
Paper • 2310.01889 • Published • 13 -
Scaling Laws of RoPE-based Extrapolation
Paper • 2310.05209 • Published • 8 -
Extending Context Window of Large Language Models via Positional Interpolation
Paper • 2306.15595 • Published • 54 -
Longformer: The Long-Document Transformer
Paper • 2004.05150 • Published • 4 -
BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences
Paper • 2403.09347 • Published • 22 -
Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention
Paper • 2404.07143 • Published • 111 -
RULER: What's the Real Context Size of Your Long-Context Language Models?
Paper • 2404.06654 • Published • 42 -
LLoCO: Learning Long Contexts Offline
Paper • 2404.07979 • Published • 22 -
Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length
Paper • 2404.08801 • Published • 66 -
Length Generalization of Causal Transformers without Position Encoding
Paper • 2404.12224 • Published • 1 -
Qwen2 Technical Report
Paper • 2407.10671 • Published • 175 -
NeedleBench: Can LLMs Do Retrieval and Reasoning in 1 Million Context Window?
Paper • 2407.11963 • Published • 44