The Pulse
NVIDIA Researchers Publish a 256K-Context Recipe for Coding Agents
NVIDIA researchers describe a 256K-context midtraining recipe for preparing Nemotron models for coding-agent tasks. In their tests, 500 billion training tokens raised Nemotron 3.5 Lightning’s SWE-bench Verified score from 60.7% to 64.5% aft
AI.info Team · Written with AI · fact-checked ·
Nemotron 3.5 Lightning’s SWE-bench Verified score rose from 60.7% to 64.5% after NVIDIA researchers added 500 billion tokens of midtraining and then applied the same supervised fine-tuning. The result comes from a paper submitted to arXiv on October 15, which lays out a 256K-context recipe for preparing language models to work as coding agents. A larger Nemotron model also improved on a benchmark of longer software tasks, though the authors stress that their tests cover only NVIDIA’s own model family.
A new stage between pretraining and fine-tuning
The paper, “Priming Base Models for Agentic Post-Training: An Open Midtraining Recipe,” describes an additional training stage between pretraining and post-training. The researchers argue that standard pretraining often uses shorter contexts and includes few agent interactions, while supervised fine-tuning data tends to be synthetic. Their midtraining stage gives the model more exposure to the materials and workflows that coding agents encounter: repositories, pull requests, commit histories, build logs and tool-use traces.
Training runs use a context window of 256,000 tokens and hundreds of billions of training tokens. In the 500-billion-token blend, about 41% comes from issue and pull-request data, 20% from repositories, 10% from agent traces, 3% from code-diff data and 1.2% from build and continuous-integration logs. The remainder is pretraining-data replay and other long-context material. NVIDIA says it filters data for licensing, removes secrets and checks for overlap with SWE-bench and Terminal-Bench.
Lightning gains 3.8 points on SWE-bench Verified
For Nemotron 3.5 Lightning, the researchers compared the starting model with checkpoints trained on up to 500 billion midtraining tokens, then gave each the same full fine-tuning run. On SWE-bench Verified, the score increased from 60.7% to 64.5%; on SWE-bench Multilingual, it rose from 55.8% to 62.8%. The score on SWE-bench Pro went from 32.0% to 35.5%.
The gains were not uniform across every measure. The authors report that SWE-bench Verified and Pro improved substantially by 100 billion midtraining tokens, while Multilingual continued to improve at higher token counts. Terminal-Bench results were noisier and peaked at 300 billion tokens rather than 500 billion. These figures are pass@1 results from the paper’s evaluation setup, not a guarantee that a model will achieve the same performance in other environments.
Agent traces matter more than natural code alone
The recipe combines ordinary software artifacts with a smaller share of agent trajectories. Those traces show models taking actions through tools, testing changes and responding to failures. In an ablation, training on natural software data without agent traces delivered much smaller gains after fine-tuning on SWE-bench Verified; the full blend performed better, leading the authors to conclude that traces contributed substantially to the improvements.
The paper also reports that training on tool responses—the outputs returned by the environment after an agent’s tool call—hurt one experiment. For a 100-billion-token run on the smaller Lightning checkpoint, the version trained on assistant turns scored 37.8% on SWE-bench Verified, compared with 27.5% when tool responses were included in the loss. The researchers’ recipe therefore trains on assistant actions and tool calls, but not the tool outputs themselves.
“We tested the recipe only on Nemotron models, not on other model families, and mainly on agentic coding.”
Soumye Singhal et al., “Priming Base Models for Agentic Post-Training: An Open Midtraining Recipe”
Super improves on long-horizon tasks, with limits
On Nemotron 3.5 Super, 280 billion midtraining tokens lifted the DeepSWE score from 11.7% to 15.7% after identical full fine-tuning. SWE-bench Verified moved from 72.9% to 74.3%. The larger model’s results suggest the method can transfer across two Nemotron sizes, but they do not establish that it will work on other model architectures.
The authors identify further constraints: each benchmark comparison is a single run, and differences of one or two points may reflect run-to-run variation. They did not compare midtraining against spending the same token budget on longer pretraining or fine-tuning. Their evaluations also use an agent-focused fine-tuning blend rather than a general-purpose post-training mix, and reinforcement-learning results remain limited because the larger-model runs were too compute-intensive. The paper offers a detailed training recipe, but its strongest evidence is still confined to two Nemotron models and the reported coding-agent tests.