Rufus-Air: An Open LLM Post-Training Recipe

AI summary

Overview

• Research area: Large language model post-training (supervised fine-tuning plus reinforcement learning), with an emphasis on reproducible open recipes for agentic and reasoning models.
• Technical level: Advanced. The paper assumes familiarity with SFT, RLVR, policy-gradient methods (GSPO), Mixture-of-Experts models, and inference/training infrastructure.
• Scope: One sentence: the paper documents a complete eight-stage post-training pipeline applied to GLM-4.5-Air-Base (106B total / 12B active parameters), reporting the data, reward design, stage order, infrastructure choices, and stagewise results needed to reproduce the resulting Rufus-Air checkpoint.

What This Paper Is About

Open-weight base checkpoints have made it possible for many teams to start post-training from a public model, but the recipes themselves are usually described thinly, closer to a system card than to something another team could follow. The authors address this by publishing a full, reusable account of one recipe built on GLM-4.5-Air-Base, using only open-source components and public data. The goal is not a new algorithm but a documented, reproducible pipeline that other teams can rebuild or adapt.

Key Contributions

1. A complete, serial eight-stage recipe. The pipeline is SFT → Reasoning RL → Coding RL → Instruction-Following RL → General Agent → Coding Agent → Search Agent → RLHF, with each stage trained on the checkpoint the previous one produced. The paper reports data, reward design, infrastructure, stage order, and stagewise results for the sequence.
2. An ordering principle based on reward reliability. Stages with hard, verifiable rewards run first, and stages with softer judge- or model-based rewards run later, to limit how long training is exposed to reward hacking. The authors state the order tracks how easily a reward can be gamed rather than the reward's format — IF RL uses a rubric judge but runs early because instruction following is close to what the policy already does.
3. A difficulty-filtering scheme that acts as an automatic curriculum. Prompts the policy already solves are dropped, and in most stages prompts it never solves are dropped as well, keeping training in what the paper calls the productive learning band.
4. A demonstration that infrastructure choices are part of the recipe. The paper highlights token-in/token-out rollouts for on-policy multi-turn RL, consistent chat-template handling from SFT through the agentic stages, a sandbox service reliable for long agentic runs, and large batches with Rollout Routing Replay for stable RL.

Main Findings

• SFT is a capability-building stage, not a warm-up. The SFT mixture contains 9.01M samples and 44.5B raw tokens across 66.7M conversational turns (23.5M supervised); after masking, 27.0B tokens (60.8%) contribute to the loss. The shipped SFT checkpoint (step 3799) already beats the public GLM-4.5-Air release on IFEval (88.33 vs 83.00), IFBench (57.75 vs 33.60), AIME 25 (90.83 vs 84.20), and AIME 26 (90.00 vs 86.50), while trailing on GPQA (68.18 vs 73.90) — and the deficits it leaves are exactly what the RL stages target. Across the pipeline, GPQA pass@1 rises from 68.18 to 75.6 and IFEval from 88.33 to 95.4.
• Sample share and supervision volume differ sharply. General Agent is the largest SFT category by sample count (39.6%) but contributes 14.2% of training tokens, while Math and Coding Agent together are 18.8% of samples but 49.1% of training tokens. The authors argue both views should be reported.
• Difficulty filtering keeps prompts learnable. The Reasoning RL prompt set contains 121,161 prompts and 25.07M prompt tokens (Math 47.7% of prompts, Puzzles 55.5% of tokens). A correctness filter drops prompts a strong teacher (GPT-OSS-120B) never solves, and a learnability filter drops prompts solved above a 0.8 rate; the fixed window lets previously dead prompts cross into the productive band as the policy improves.
• Reasoning RL is curation-bound. Against the SFT checkpoint, Reasoning RL raises GPQA by +5.32 points (68.18 to 73.50) and lowers AIME 25 by 2.81 and AIME 26 by 2.60 points, which the authors treat as within evaluation noise on a 30-problem benchmark (same-condition re-runs on AIME 25 differed by 4.6 points).
• Extending the response budget unblocked coding gains. In Coding RL, 11–28% of samples hit the 64K response cap and were loss-masked; after extending to 128K, truncation fell below 0.1%, reward resumed climbing to 0.42 by step 34, and LiveCodeBench v6 pass@1 peaked at 75.9 at step 33. Relative to the Reasoning RL checkpoint, Coding RL lifts LiveCodeBench v6 pass@1 by +7.3 points (68.6 to 75.9) and pass@8 by +2.3 points (85.1 to 87.4).
• Instruction-following RL produced the largest single-stage jumps. From the prior checkpoint, IF RL lifts IFEval by +4.0 (90.5 to 94.5), IFBench by +14.0 (63.8 to 77.8), and Multi-challenge by +24.7 (31.1 to 55.8), while response length fell from roughly 5.4K tokens to roughly 3.6K.
• Later stages add targeted agentic capability. General Agent adds +7.8 on MCP-Atlas (35.0 to 42.8) and +9.8 on Tau2-Retail (74.0 to 83.8); Coding Agent adds +1.4 on Terminal-Bench 2.1 (38.8 to 40.2) and +2.2 on SWE-bench Verified (65.6 to 67.8); Search Agent adds +3.0 on BrowseComp (34.2 to 37.2), +5.4 on Seal-0 (48.6 to 54.0), and +3.4 on HLE-Verified Gold (47.7 to 51.1); RLHF adds +6.0 on Arena-Hard v2 (HP) (83.1 to 89.1) and +14.4 on Arena-Hard v2 (CW) (38.6 to 53.0).
• Final placement. Rufus-Air leads the official GLM-4.5-Air release on every reported benchmark except Arena-Hard v2 Creative Writing, where GLM-4.5-Air (60.3) and Nemotron-3-Super (61.5) are ahead of Rufus-Air's 53.0. Against Nemotron-3-Super it trails on Tau2-Airline (84.0 vs 86.0), and by about a point on Tau2-Retail (85.3 vs 86.4), and ties on Terminal-Bench 2.1 (both 42.7).
• Decontamination was conservative. The SFT mix was screened against ten benchmarks with a word-level 8-gram overlap test, removing 3,529 samples (3,321 for Terminal-Bench 2.0, 57 for HLE-Verified, 21 for IFEval, 130 for IFBench; the other six had zero matches). Two further detectors for math and knowledge benchmarks found 68 confirmed contaminated pairs (58 unique samples); with borderline cases, 658 unique samples were dropped, bringing total removals from the SFT mix to 4,187.
• The order does not protect earlier gains. Because every later stage updates the same parameters, each stage is measured against the checkpoint it starts from; the exception is SFT, which starts from a base model and is measured against the public GLM-4.5-Air release.
• A stated caveat. The report sits between a research paper and an engineering experience report; some conclusions come from training experience rather than full ablations, which the authors mark as such in their §6. The per-stage data and method details for the three agentic stages and RLHF, along with the infrastructure section, are referenced in the paper but are not included in the provided text.

Methodology in Plain English

The authors start from a public Mixture-of-Experts base checkpoint, GLM-4.5-Air-Base (106B total, 12B active parameters), and keep its architecture, tokenizer, and native <think> chat format fixed. They then run eight training stages in a fixed serial order, each starting where the previous one ended.

SFT first, and treated as a real capability stage. They assemble a 9.01M-sample mixture from 17 public datasets, with no new human annotation and no in-house distillation teacher; some source datasets already contain responses generated by other models, which the authors describe as upstream dependencies of the public releases. Heterogeneous sources are normalized into one role-aware multi-turn schema with per-message loss masks; agent trajectories are supervised as complete thought–action–observation chains, with earlier dialogue kept as context but masked. The mix is screened for benchmark leakage with an 8-gram overlap test plus exhaustive n-gram and dense-retrieval checks for math and knowledge benchmarks. Training uses Slime with the Megatron backend on 64 nodes of 8 H200 GPUs (512 GPUs total), batch size 4096 sequences, a warmup to a 5e-5 peak and a cosine decay toward 5e-6, and three epochs. Loss fell from 0.84 to a 50-step moving average of 0.42 by the end of the first epoch (step 2370), then to 0.38 and 0.34 at later epoch boundaries, but held-out scores flattened within the first epoch, so the authors carry checkpoint 3799 rather than the final step into RL.

Then RL stages, ordered by how gameable the reward is. Reasoning RL and Coding RL use binary, programmatically auditable rewards (Math-Verify on canonicalized answers, generated Python checkers for puzzles, fuzzy string matching for science; sandboxed execution against test cases for code). Prompts enter these stages only if a prompt lies in a learnable band; the Reasoning stage also filters online by group-average reward, over-sampling 4× and keeping groups with mean reward in (0, 0.8]. The policy-gradient backbone is GSPO with tight clipping and zero KL and entropy coefficients, trained with FP8-quantized SGLang rollout engines colocated on the same GPUs that run BF16 training.

Then IF RL and the agentic stages. Instruction-following RL trains jointly on two synthetic sets: 14K single-turn multi-constraint prompts (2–6 atomic constraints each, generated by Qwen3-235B-A22B with a Python verifier per constraint) and adversarially generated multi-turn conversations where several open-weight models rotate in the assistant role while a teacher plays a user trying to make the assistant fail. The three agentic stages add tool use in their own environments under rule-based assertions and execution tests, and RLHF runs last, shaping open-ended quality where no hard verifier exists.

Benchmarks are used only to place the model. All reported values are pass@1 except the two Arena-Hard v2 rows, which are win rates. The four leftmost models in the comparison table were evaluated by the authors under one harness; the other six columns come from other public sources.

Why This Matters

Impact on research. The paper argues that the details a team actually needs to reproduce a post-training run are the least disclosed part of the literature, with few exceptions such as Tülu 3. By documenting stage order, reward design, filtering rules, and engineering choices — and by noting that its compute footprint is small enough for teams outside frontier labs (Table 16) — it provides a reusable baseline that others can modify stage by stage rather than treating post-training as an opaque monolith.

• Enterprise assistant agents: the Tau2-Retail (85.3), Tau2-Airline (84.0), and Tau2-Telecom (93.0) results target tool-using agents in transactional and support settings.
• Software engineering automation: SWE-bench Verified (65.6) and Terminal-Bench 2.1 (42.7) measure repository-level and terminal-level coding agents.
• Web and document research agents: BrowseComp (37.1), Seal-0 (51.4), and HLE-Verified Gold (51.1) cover search and evidence-seeking behavior.
• Constraint-critical generation: the IF RL gains — IFEval 95.4, IFBench 76.9, Multi-challenge 65.8 — matter where output must obey length, structure, or keyword requirements exactly, such as templated reporting and system-prompt-governed assistants.

Industry relevance. The recipe builds entirely on open-source components (Slime, SGLang, Megatron) and public data, runs at 12B active parameters, and required no new human annotation and no in-house distillation teacher. That combination lowers the barrier to entry for organizations that want to post-train an open checkpoint themselves, and it makes stage ordering and reward-reliability decisions explicit rather than tacit engineering lore.

Future Directions

• Replace experience-based judgments with ablations. The authors note that some conclusions come from training experience rather than full ablations and mark them as such; systematic ablations of stage order, data mix, and filtering thresholds are the obvious next step.
• Test whether the reward-reliability ordering generalizes. The claim that

Authors’ abstract

Rufus-Air is an open and reproducible post-training recipe on GLM-4.5-Air-Base (106B-A12B), organized as a serial pipeline of eight stages: SFT, Reasoning RL, Coding RL, Instruction-Following RL, General Agent, Coding Agent, Search Agent, and RLHF. We document the data, reward design, infrastructure, stage order, and stagewise results needed to reproduce the recipe. Stages progress from basic to advanced capabilities and from hard, verifiable rewards to softer judge-based signals. Training builds on open-source components and public data, much of it used as released, without new human annotation or an in-house distillation teacher. Our main findings are that (i) diverse, high-quality SFT establishes a strong capability floor; (ii) difficulty filtering keeps RL prompts within a productive learning range; (iii) reward reliability provides a practical principle for ordering stages; and (iv) infrastructure and engineering choices are part of the recipe, not just an implementation detail. Rufus-Air improves over the official GLM-4.5-Air post-trained release and is competitive with similarly sized open models.