Dory1.0-4B: A Looped Normalized-GPT

AI summary

Overview

Research area: Natural Language Processing — efficient language-model architecture, specifically looped (latent-recurrent) Transformers and test-time compute scaling.

Technical level: Advanced. The paper assumes familiarity with Transformer internals, attention variants (GQA, sliding-window attention, RoPE), low-precision training (NVFP4/MXFP8), reinforcement learning from verifiable rewards, and post-training quantization.

Scope: A single open 4-billion-parameter model, Dory1.0-4B, whose shared recurrent block can be run a variable number of times at inference so that one checkpoint trades compute for accuracy, trained on 6 trillion tokens and post-trained with SFT and RLVR.

What This Paper Is About

Most ways of spending more compute on a language model at inference time do so by generating more tokens, which increases latency, KV-cache memory, and serving cost with every token produced. This paper argues that compute can instead be spent in latent space: a shared 40-layer block is applied repeatedly to the same hidden state, so the model thinks harder per token rather than writing more. The goal is a single fully post-trained, open-weight checkpoint that a serving system can run at 1, 2, 3, or 4 loops depending on the accuracy budget of each request.

Key Contributions

1. An open looped model with elastic depth. Dory1.0-4B partitions its 72 physical layers into a 16-layer input block, a 40-layer shared recurrent block, and a 16-layer output block. The input and output blocks run once and the recurrent block runs k times, giving an effective forward depth of 32 + 40k — from 72 to 192 physical-layer applications — from one set of weights.

2. A full pretraining-to-post-training recipe in low precision. The model was pretrained on the first ~4.5T tokens as a standard non-looped model in NVFP4 (an end-to-end speedup of about 20% over BF16), then switched to BF16 for ~1.465T additional tokens while looping was introduced by sampling k from {1, 2, 3, 4}, then extended context to 256K tokens over ~300B more tokens.

3. A difficulty-aware loop curriculum for RLVR. During 200 steps of asynchronous GRPO across 23 environments, prompts with lower profiled pass rates are preferentially routed to deeper-loop rollout workers, using a routing score that interpolates a within-batch difficulty quantile with a uniform random draw at α = 0.3, while preserving fixed prompt quotas of 4:1:1:1 for k = 1, 2, 3, 4.

4. Last-loop KV caching and open quantized checkpoints. Keys and values are cached only from the final loop, so KV memory stays at its one-loop size for any k. BF16, MXFP8, NVFP4, and GGUF (Q5_K_M) checkpoints are released under the OpenMDW-1.1 license.

Main Findings

• Accuracy rises with loop count. On the equal-weight average of 12 benchmarks, Dory1.0-4B scores 35.17 at one loop, 39.67 at two loops, and 40.60 at four loops.
• It beats larger models from two loops up. That 39.67 average exceeds Qwen3.5-9B (32.21), Gemma 4 12B (32.59), and Gemma 4 26B-A4B (36.73), and comes within 1.7 points of Qwen3.6-35B-A3B (42.30), a model the paper describes as having roughly 9× as many total parameters.
• Gains are task-dependent. From one to four loops, GDPval-AA v2 rises from 815 to 967 Elo, SWE-Bench Verified from 50.3% to 61.8%, and Terminal-Bench v2.1 from 12.5% to 22.2%, while AA-Omniscience accuracy stays essentially flat (10.03 at both one and four loops).
• Looping reduces output length on agentic tasks. Output tokens per attempt fall 1.3–2.5× from one to four loops. On GDPval-AA v2, turns per attempt drop from 116 to 49 while output tokens per turn change by only 5%; on Terminal-Bench v2.1, turns drop 39% and tokens per turn 20%.
• Two loops is the best compute-per-point setting. Approximating compute as output tokens × layer applications per token (72, 112, 192 at k = 1, 2, 4), two loops cost only 1.03–1.25× the compute of one loop yet recover 86% of the GDPval-AA v2 Elo gain, 80% on SWE-Bench Verified, 58% on Terminal-Bench v2.1, and 42% on τ³-Banking. Four loops cost 1.07–2.04×.
• Last-loop caching is nearly free. At four loops, it shifts benchmark scores by between −2.1 and +0.4 points relative to per-loop caching, with nearly unchanged output lengths, while keeping KV memory at one-loop size.
• The loop curriculum helps most at depth. In an ablation on an earlier SFT checkpoint, it raises Terminal-Bench v2.1 from 14.0% to 21.4% at four loops, GDPval-AA v2 by 98 Elo, and τ³-Banking by 4.9 points, and improves all six measured benchmarks at both one and four loops.
• Extended SFT keeps paying off at fixed one-loop inference. Between iterations 3,600 and 32,400 (epochs one and nine), SciCode rises from 13.11% to 18.14%, Terminal-Bench v2.1 from 4.2% to 13.5%, GPQA Diamond from 43.13% to 54.95%, IFBench from 28.94% to 38.50%, MATH-500 from 60.2% to 75.2%, and AA-LCR from 29.5% to 37.3%.
• Base-model gains saturate after about four loops. Base scores at k ∈ {1, 2, 4, 6, 8} show MMLU rising 68.20 → 70.77 → 71.19 → 71.22 → 71.17 and MATH-500 60.60 → 65.00 → 69.40 → 68.40 → 69.40; because 6 and 8 were never trained on, they measure inference-depth extrapolation.
• Non-hallucination is unusually high. AA-Omniscience non-hallucination is 76.42 / 79.74 / 79.38 at one / two / four loops, against 20.18 for Qwen3.5-4B, 17.39 for Qwen3.5-9B, 21.84 for Gemma 4 12B, 16.36 for Gemma 4 26B-A4B, and 65.25 for Qwen3.6-35B-A3B.
• Quantization preserves most accuracy. MXFP8 (post-training quantization) is roughly on par with BF16; NVFP4 and Q5_K_M needed quantization-aware distillation to close their gaps. At four loops, for example, BF16 scores 35.11 on SciCode versus 34.85 (NVFP4-QAD) and 34.29 (Q5_K_M-QAD).

Methodology in Plain English

The starting point is a normalized Transformer (nGPT), where residual updates blend a normalized sublayer output with a normalized residual stream using learned per-channel weights, and everything is L2-normalized. The team adds L2 normalization of the concatenated, gated attention output before the output projection and of the SwiGLU activations before the down-projection, implemented with fused Triton kernels.

The 72 layers use a repeating eight-layer pattern of three sliding-window-attention/SwiGLU pairs followed by one full-attention/SwiGLU pair — 27 sliding-window and 9 full-attention layers interleaved with 36 MLP-only layers. Sliding-window attention uses a 1,024-token window with RoPE base 10⁴; full attention uses RoPE base 10⁷ after long-context extension (raised from 500,000). Attention is grouped-query with 16 query heads, 4 KV groups, and input-dependent sigmoid output gates.

Training proceeds in stages. First a normal non-looped model is trained in NVFP4 for about 4.535T tokens (ending at iteration 720,750) with a learning rate of 3.2×10⁻⁴. Then, in BF16, loops are introduced: the stack is partitioned into input, recurrent, and output blocks; the recurrent block's output is L2-normalized and the input block's fixed output is re-injected before every pass; and the loop count is sampled at 0.25 probability each for k ∈ {1, 2, 3, 4}. A rank-level override forces about 10% of data-parallel replicas to run exactly one loop. Backpropagation is truncated to depth one — only the final recurrent invocation is trained. Context then stretches from 32,768 to 262,144 tokens over 47,683 steps at a constant learning rate of 10⁻⁵.

Post-training uses nine epochs of SFT on packed 262,144-token sequences with mixed thinking and non-thinking data, followed by 200 GRPO steps over 23 environments, where each step samples 384 prompts with 16 responses each. For quantization, MXFP8 gets standard post-training quantization; NVFP4 (with dynamic MSE scale selection over the 126 positive E4M3 values) and Q5_K_M get quantization-aware distillation against the frozen BF16 teacher on a 10B-token subsample of the SFT mixture.

Why This Matters

The paper offers a concrete, reproducible alternative axis for test-time scaling that does not inflate output length — a property that matters because token-space scaling makes latency, KV-cache memory, and serving cost grow with every generated token. It also shows that a looped model can be released as a single checkpoint serving several compute tiers, rather than as separate small, medium, and large models, and it demonstrates that this survives full post-training (SFT plus RL) rather than only pretraining.

Real-world applications:

• Agentic software engineering: SWE-Bench Verified rises from 50.3% to 61.8% between one and four loops, and turn counts fall from 88 to 73, so fewer actions per resolved issue.
• Terminal and tool-use automation: Terminal-Bench v2.1 goes from 12.5% to 22.2%, with 109k → 54k output tokens per attempt.
• Conversational tool use in regulated domains: τ³-Banking reaches 14.69% at four loops, ahead of every baseline in the table, while producing 18% shorter turns.
• Long-document professional work: AA-LCR reaches 73 at four loops and GDP.pdf 6.0, with GDPval-AA v2 at 967 Elo on workplace tasks.

Industry relevance: the NVFP4 pretraining stage reports a ~20% end-to-end speedup over BF16, last-loop KV caching keeps memory flat as loop count grows, and MXFP8, NVFP4, and GGUF releases make the model deployable on a range of hardware — all of which bear directly on serving economics rather than only on benchmark tables.

Future Directions

• Beyond four loops. Training samples only k ∈ {1, 2, 3, 4}; the 6- and 8-loop evaluations are extrapolation and largely saturate. Whether training at higher loop counts, or with learned halting, unlocks further gains is not reported.
• Deeper truncated backpropagation. The current recipe uses depth one, training only the final recurrent invocation. Whether backpropagating through more loops improves the accuracy-per-loop curve is an open question the paper does not answer.
• Generalizing the loop curriculum. The difficulty-aware routing is demonstrated with α = 0.3 and quotas of 4:1:1:1 over 23 environments; how sensitive the gains are to α, to the quota shape, and to the profiling budget (eight responses per prompt) is not reported.
• Looping versus other latent-compute schemes. The paper compares against larger non-looped models but does not report comparisons against other looped or recurrent-depth architectures under matched compute, leaving that trade-off unexplored.

Target Audience

Model-architecture and training-infrastructure researchers working on efficient inference and test-time compute scaling; engineers who own LLM serving economics and care about KV-cache footprint and tokens per task; and practitioners evaluating whether a single elastic-depth checkpoint can replace a ladder of separately sized models. Readers need comfort with Transformer internals, low-precision training formats, and RL post-training to get the most from the recipe details.

Authors’ abstract

We introduce Dory1.0-4B, a 4-billion-parameter looped (latent-recurrent) language model with elastic depth: a single checkpoint serves different inference compute budgets. Dory1.0-4B is built on a normalized Transformer (nGPT) backbone with interleaved sliding-window and full-attention layers, organized into an input block, a shared recurrent block, and an output block. At inference, the recurrent block can be applied a variable number of times. We pre-trained Dory1.0-4B on 6 trillion tokens. The first 4.5 trillion tokens were trained in NVFP4, and additional pre-training then extended the context length to 256K tokens. Next, we post-trained the model using Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR), with a difficulty-aware loop curriculum that routes harder prompts to deeper loops. Increasing the loop count improves accuracy, with the largest gains on coding, agentic, and long-context benchmarks. With two or more loops, Dory1.0-4B outperforms larger models such as Qwen3.5-9B, Gemma 4 12B, and Gemma 4 26B-A4B on average across 12 benchmarks. Looping also makes Dory1.0-4B more token-efficient on agentic benchmarks: with four loops, it generates 1.3-2.5x fewer tokens than with one loop, as the extra computation is spent in latent space rather than on longer outputs. To reduce memory during looping, Dory1.0-4B caches keys and values only from the last loop, keeping its KV-cache footprint equal to that of a non-looped model at any loop count while retaining most of the accuracy gains. We open-source the BF16, MXFP8, NVFP4, and GGUF quantized checkpoints.