The Pulse
Fudan-Linked Team Reports 16× Context Extrapolation
A preprint reports that a hybrid attention method reached 100% accuracy on one needle-retrieval test at 64,000 tokens after training at 4,000 tokens. Its results on two other retrieval variants were lower, limiting the scope of the headline
AI.info Team · Written with AI ·
64,000 tokens—16 times a 4,000-token training context—is the length at which a Fudan-affiliated team says its attention method can still score 100% on one needle-retrieval test, without additional training. The result comes from a new preprint on hybrid language models, not a general demonstration that models can reliably use any information placed in a much longer prompt.
The paper, submitted to arXiv on October 7, 2026, proposes Sliding-Window Linear Attention (SWLA) and examines how different attention mechanisms affect long-context performance. Its authors are Xiaoran Liu, Ziwei He and Xipeng Qiu, whose listed affiliations include Fudan University, Shanghai Innovation Institute and the OpenMOSS Team. The preprint is marked as under review.
How SWLA stretches a 4,000-token window
The authors distinguish between context extension, which involves continued training on longer sequences, and length extrapolation: using a model beyond its training context without further training. Their experiments include models pretrained at 4,000 tokens and evaluate performance at lengths up to 64,000. For the 16× result, the paper combines SWLA with a log-scale adjustment to the model’s NoPE, or no positional embedding, attention.
SWLA limits how far linear attention’s position-related signals reach by applying a sliding window. The paper describes an implementation using a 4,096-token window and a 1,024-token stride. In the hybrid design, that windowed attention works alongside NoPE attention, which retains a global view of the sequence; the log-scale adjustment strengthens that global component during extrapolation.
The authors write:
“For LA hybrids, we summarize the Matthew Effect of Hybrid Position Extrapolation and propose Sliding-Window Linear Attention, achieving 16× training-free length extrapolation while maintaining 100% accuracy on NIAH-SK1 in 64k context length.”
Xiaoran Liu, Ziwei He and Xipeng Qiu, authors of the paper
The 100% score has a narrow scope
NIAH-SK1 is one variant of a “needle in a haystack” retrieval test: a model must locate a specific piece of information within a longer context. The paper also reports two other variants, SK2 and SK3, and the result does not carry across them uniformly. In one 776-million-parameter, short-context-pretrained GLA configuration with SWLA and the log adjustment, the table records 100% on SK1 at 64,000 tokens, but 59% on SK2 and 64% on SK3.
That contrast matters. A perfect score on one retrieval test does not establish that the method preserves broad reasoning, summarization or information use across long documents. The paper’s reported numbers are benchmark results from specific hybrid-model configurations, not a guarantee for arbitrary models or real-world prompts.
Hybrid attention brings a trade-off
The broader study compares sliding-window attention with gated linear-attention variants, including GLA and GDN, across different model sizes. The authors find that the methods trade strengths: sliding-window hybrids can extrapolate better beyond their training length, while linear-attention hybrids benefit more from continued pretraining at longer contexts. They describe this as a “Seesaw Effect,” and report that changing the window size and training setup can improve sliding-window hybrids’ context extension.
The experiments span 376-million-, 776-million- and 1-billion-parameter models, with the authors also reporting validation of key conclusions using a 3-billion-parameter model. That breadth adds evidence for the paper’s architectural comparisons, but it does not turn the 16× result into a universal score: performance depends on the attention configuration, model scale and task.
Reasoning and multimodal tests remain open
The authors say their validation focuses on context extension and length extrapolation during pretraining. They identify post-training, multimodal models, complex reasoning and agentic tasks as areas needing further evaluation. Those limits leave practical questions about how well SWLA transfers to common applications that require more than finding a planted fact.
For now, the paper’s clearest claim is specific: a hybrid model using SWLA and a log-scale adjustment reaches 64,000 tokens after 4,000-token training and reports perfect accuracy on NIAH-SK1. Its lower scores on the other needle variants show why the headline number needs its benchmark attached.