YANchor-4B: Effective Long-Horizon Reasoning in O(N) Time with O(1) Memory

AI summary

Overview

Research area: Efficient sequence modeling and long-context language models — specifically recurrent/state-space architectures for long-horizon reasoning.

Technical level: Advanced.

Scope: The paper introduces YANchor-4B, a 4.58B-parameter recurrent language model that adds an independently written, retrievable memory of "anchors" to a Gated DeltaNet plus sliding-window-attention backbone, claiming O(N) generation time and O(1) history-state memory while outperforming bounded-state baselines on mathematics, code, science, knowledge, and instruction following.

What This Paper Is About

Long-horizon reasoning requires a model to keep using definitions, constraints, and intermediate results from much earlier in a solution. Full-history attention keeps everything accessible but pays for it with a growing cache and growing computation, while recurrent models keep a fixed-size summary that can blur precise details. The paper's goal is to retain and retrieve crucial earlier information inside a fixed-capacity history state, so that hard reasoning stays affordable as generation gets long.

Key Contributions

1. An architecture that combines three kinds of history in one computation. YANchor-4B adapts Qwen3.5-4B into eight repeating groups of three Gated DeltaNet (GDN) layers plus one layer that combines a 1,024-token sliding window attention (SWA) with an independent memory branch, written as [GDN, GDN, GDN, SWA₁₀₂₄ ∥ Memory] × 8.

2. A memory that writes, admits, and reads records separately from local attention. A residual writer builds key/value representations from normalized hidden states, a small admission network scores each KV group on information available at write time, each of four KV groups keeps its highest-scoring 1,024 records, and queries are projected separately so writing and retrieving are decoupled. Records are replenished in 256-token blocks.

3. Grouped-query retrieval with a neutral memory item. Each of the eight memory modules has 16 query heads, 4 KV groups, and head dimension 256. The retrieval softmax includes a zero-logit, zero-value item so the read returns a neutral output when memory is empty, and a query-conditioned gate modulates each retrieved head before an independent output projection.

4. A training recipe in four stages. M0 (memory adaptation), R (joint adaptation with the backbone), S (supervised post-training), and RLVR (reinforcement learning with verifiable rewards) are sequenced to teach what to preserve, how to retrieve it, and how to use it in a solution.

Main Findings

• Competition mathematics with bounded state: YANchor-4B reaches 82.93% mean pass@1 on AIME 2024–2026 (86.72 on AIME 2024, 77.45 on AIME 2025, 84.64 on AIME 2026) and 63.64% on HMMT February 2026. RWKV-7 G1j 13.3B, described as the strongest bounded-state baseline on these tasks, scores 18.89% and 12.12% respectively. YANchor also scores 97.49% on MATH-500 and 94.45% on GSM8K.

• Broad lead across 24 common benchmarks: YANchor leads the 16 bounded-state baselines on 20 of 24 tasks, with a 24-benchmark mean of 78.64 versus the strongest baseline's 56.29 — a 22.35-point gap. The cross-architecture reference Qwen3.5-4B scores 80.44 on the same mean.

• Small deficit versus the Transformer hybrid it was converted from: Against Qwen3.5-4B, YANchor is lower on the AIME three-year mean (82.93 versus 86.11), HMMT February 2026 (63.64 versus 65.91), GSM8K (94.45 versus 94.50), MATH-500 (97.49 versus 98.00), CMMLU, C-Eval, GPQA-Diamond, WinoGrande, HellaSwag, ARC-C, ARC-E, BBEH, LiveCodeBench v6, and IFEval. It is higher on MMLU (87.87 versus 85.60), MMLU-Pro (79.60 versus 75.90), MMLU-Redux (92.00 versus 89.31), SuperGPQA (61.69 versus 51.40), HumanEval (96.72 versus 85.98), MBPP (87.81 versus 70.80), and IFBench (65.29 versus 60.33).

• Shorter reasoning traces: YANchor uses 43–47% fewer output tokens than Qwen3.5-4B on AIME while retaining 96.31% of its three-year mean accuracy, and 41.1% fewer tokens on HMMT February 2026 (63.64 versus 65.91). Responses average roughly 18,000–25,000 tokens on the three AIME editions and 30,189 tokens on HMMT February 2026.

• Stable decoding speed on one H100: YANchor sustains approximately 213 decode tokens/s as generation extends. At 128K input and 128K output, cumulative rates are 212.98 for YANchor, 153.91 for Qwen3.5-4B, 133.75 for Gemma 4 E4B, and 164.53 for MiniCPM5-2B.

• Several-fold higher batched throughput: YANchor delivers 4.12–6.58 times Qwen3.5-4B's end-to-end throughput and 3.25–5.66 times Gemma's in batched long generation, and outperforms MiniCPM throughout the comparison. At 32K outputs, reported throughputs are 12,799 tokens/s at 4K input, 12,732 at 8K, 12,661 at 16K, and 12,431 at 32K, each at batch 512.

• Fixed memory at high concurrency: YANchor keeps 512 independent long-context sequences resident on one H100 80GB. Its history cache stays fixed at 48.90 GiB as context grows from 512 to 65,536 tokens per sequence, with peak demand near 67 GiB (the 1 GiB increase comes from input buffers). The growing-cache baselines begin preemption at 4K tokens per sequence for Qwen3.5-4B and MiniCPM and at 8K for Gemma, with projected 64K requirements of 1,061, 1,352, and 541 GiB.

• The memory branch is doing the work: On long-memory tasks spanning 16K to 130,000 tokens, YANchor achieves 94.14% query accuracy; disabling the memory branch drops this to 0.39%. Complete-context success (all four answers correct) is 80.08% overall, broken down as 90.63% exact retrieval, 87.50% state-update interpretation, 89.06% cross-document combination, and 53.13% variable binding.

• Sensitivity to changed distant evidence: Both answers are correct in 89.84% of pairs requiring an answer change and 96.48% of pairs requiring a stable answer.

• Multimodal capability in the same bounded-state model: Across six VL benchmarks the mean answer-level accuracy is 84.94% (MMBench EN 88.28, MMBench CN 89.00, POPE 85.43, MME 82.98, ChartQA 83.92, AI2D 80.05), with an answer-weighted accuracy of 85.50%.

• Reading comprehension and multi-step reasoning: DROP F1 is 77.22 (exact match 71.55), MuSR task-macro accuracy is 60.42, and LogiQA2 accuracy is 72.23.

• Bounded-state bookkeeping: The text model totals 4.58B parameters including 372M independent memory parameters, with residual width 2,560 and 24 GDN plus 8 SWA backbone layers. Memory K/V state is 40 MiB per sequence in BF16, following 2 × 8 × 4 × (1024 + 256) × 256 × 2 bytes.

• Complexity claims: The paper argues that because the local window W, memory capacity K, and block size C are independent of generated length N, total autoregressive work per token is bounded and totals O(N), with O(1) persistent context state — a table places this alongside full-history attention at O(N²) time and O(N) state, and fixed-dimensional recurrence or fixed-window attention at O(N) time and O(1) state.

Methodology in Plain English

The authors start from Qwen3.5-4B and keep its recurrent Gated DeltaNet layers, but replace global attention with a local sliding window and an added memory branch. At each of the eight attention positions, the model looks at the most recent 1,024 tokens through sliding-window attention and separately consults a store of selected older records. A dedicated writer network decides how each token should be represented for storage, using projections that are separate from the projections used to query it. A small admission network gives every record a score based only on information available when it is written, and each of four key/value groups independently keeps its 1,024 highest-scoring records. Records accumulate in 256-token blocks, and at each boundary the group merges new candidates with its committed set and re-selects the top 1,024, keeping whole stored key/value representations intact.

Retrieval then works like grouped-query attention over these retained records, with a zero-logit, zero-value item acting as a neutral fallback and a learned gate controlling how much memory contributes to the residual stream. Training proceeds through four stages: M0 teaches the memory modules to retain and retrieve useful information, R integrates them with the recurrent and local-attention computation, S applies supervised post-training across a broad range of tasks, and RLVR refines responses using verified outcomes on answer correctness and task completion. Evaluation covers 27 text benchmarks (24 of them supporting comparisons) and six VL benchmarks, using mean pass@1 with K = 64 responses per item for sets under 100 questions, K = 16 for 100–10,000 questions, and K = 1 above 10,000. Text generation uses temperature 0.6, top-p 0.95, and top-k 20 with a 128K-token response budget (Appendix A.1 states 131,072 tokens); VL evaluation uses temperature 1.0, top-p 0.95, top-k 20, presence penalty 1.5, a 32,768-token response cap, and pixel budgets from 65,536 to 4,194,304. Performance measurements use a single H100 80GB at tensor-parallel size one.

Why This Matters

The work targets a practical tension in long-context reasoning: models that remember everything get progressively more expensive, while models with fixed state risk losing precise details. YANchor's results suggest that a bounded-state model can handle competition-level mathematics and long retrieval tasks without a cache that grows with the solution, and that the memory path — not just the backbone — carries that capability, given the collapse from 94.14% to 0.39% query accuracy when memory is disabled.

Impact on research: It offers a concrete alternative to the established split between full-history attention and pure recurrent compression, and it provides an intervention-style diagnostic (disabling the memory branch) plus a per-category breakdown showing where anchored memory helps and where it does not. The variable-binding result of 53.13% complete-context success marks a clear boundary on what this design currently handles.

Real-world applications (as the design suggests, not as tested in the paper):

• Serving long reasoning traces to many concurrent users on fixed GPU memory, where 512 resident 64K sequences on one H100 80GB would be impractical with growing caches.
• Mathematics and competition-style tutoring or verification, given the reported AIME, HMMT, and MATH-500 scores.
• Code generation and repair, building on the reported HumanEval 96.72, MBPP 87.81, and LiveCodeBench v6 61.63.
• Long-document and multimodal analysis, where the model can combine text reasoning with chart, diagram, and object-presence judgments under the same bounded state.

Industry relevance: Fixed-capacity history changes the economics of long-generation serving. The reported 4.12–6.58 times Qwen3.5-4B throughput and the flat 48.90 GiB history cache mean capacity planning does not have to grow with sequence length, which matters for high-concurrency inference on a single accelerator.

Future Directions

• Improving variable binding. Complete-context success on variable binding is 53.13% versus 87.50–90.63% for exact retrieval, state-update interpretation, and cross-document combination. The paper calls this the hardest diagnostic category, requiring several associations to remain jointly accessible — a natural target for better admission scoring or retrieval.

• Recovering capability lost in conversion. YANchor trails Qwen3.5-4B on the AIME three-year mean (82.93 versus 86.11), the 24-benchmark mean (78.64 versus 80.44), GPQA-Diamond (64.61 versus 77.78), and several commonsense and knowledge tasks, raising the question of how much of that gap can be closed within a constant-state budget.

• Scaling the design. All reported results come from a 4.58B text model with eight memory modules; whether the same anchored-memory mechanism at larger parameter counts follows the same scaling behavior is not established by the reported experiments.

• Training details and data. The provided content describes the M0, R, S, and RLVR stages by purpose but does not report training data volumes, compute budgets, or stage-specific hyperparameters, leaving those questions open from this text alone.

Target Audience

Researchers and engineers working on efficient inference, state-space and recurrent architectures, and long-context reasoning will get the most from this paper, since it assumes familiarity with Gated DeltaNet, sliding-window attention, and grouped-query attention. Practitioners responsible for serving long-generation workloads at high concurrency will find the throughput and memory-capacity results directly relevant, and evaluators interested in memory-path diagnostics and bounded-state baselines will benefit from the comparison tables and the disable-the-memory ablation. Readers without a background in sequence-mixer architectures will find the architectural sections dense.

Authors’ abstract

Long-horizon reasoning demands access to earlier information at a manageable generation cost. Full-history attention incurs growing storage and computation, while recurrent compression can lose precise details. Therefore, we present YANchor-4B, a general-purpose recurrent model that preserves crucial memory as ANchors for retrieval during subsequent reasoning. Beyond $O(N)$-time generation and $O(1)$ memory, YANchor enables effective long-horizon reasoning through its multidimensional memory mechanism. For example, on challenging math problems, it achieves 82.93% mean pass@1 on AIME 2024--2026 and 63.64% on HMMT, substantially outperforming linear-time, constant-state counterparts, including larger models. It also delivers several-fold higher batched long-generation throughput than Transformer and hybrid baselines on H100. Furthermore, evaluations across dozens of benchmarks demonstrate YANchor's superiority in general-purpose capabilities.