Evaluating Agentic Capabilities Before Post-Training: An End-to-End Approach

AI summary

Overview

Research area: Large language model evaluation — specifically, diagnosing agentic (tool-using, multi-turn) capability in pretrained base models before any instruction tuning or post-training.

Technical level: Intermediate. The paper assumes familiarity with LLM pretraining stages, chat templates versus completion interfaces, and agent scaffolds such as SWE-agent and OpenHands.

Scope (1 sentence): The paper introduces ASAP, a lightweight framework that adapts existing agent scaffolds so that frozen pretrained base models can be run end-to-end on software-engineering and deep-research benchmarks, and uses the resulting trajectories as a diagnostic of how agentic behavior changes across pretraining checkpoints and after post-training.

What This Paper Is About

Agentic benchmarks such as SWE-bench, Terminal-Bench and BrowseComp-Plus assume instruction-following models: their scaffolds rely on system prompts, chat templates and prescribed tool formats, so pretrained base models dropped into them often produce malformed or unstable trajectories. Existing pretraining-side alternatives (static trajectory-derived QA, code or trajectory likelihood) are cheap but only score fixed examples, never showing how the evaluated model actually behaves in an environment. The paper asks whether base models can be evaluated on real end-to-end agentic tasks without first being converted into chat-style agents, and answers with ASAP (Agent Scaffold Adaptation for Pretraining Evaluation).

Key Contributions

1. End-to-end evaluation of pretrained base models. ASAP minimally patches existing scaffolds (SWE-agent, OpenHands, Terminus, BrowseComp) so a frozen base model executes tasks through its native completion interface, with no weight updates and no chat template.
2. A sensitive probe of agent-relevant behavior during pretraining. ASAP separates both different base models and checkpoints within a single pretraining run, including stages where the conventional pretraining benchmark aggregate moves only marginally.
3. Trajectory-level diagnosis of behavioral change. Because ASAP exposes the model's own execution trajectories, aggregate score gains can be decomposed into concrete behaviors — exploration, editing, testing, recovery, submission — across pretraining stages and before versus after post-training.
4. A controlled design recipe with ablations. The framework's three components (Markdown+XML format, short reference-trajectory prefix, minimal failure recovery) are each ablated, including format choice, prefix length, and robustness to independently generated prefixes.

Main Findings

• ASAP discriminates across base models and scaffolds. With eight rollouts per model on six base models (Qwen3-30B-A3B-Base, Qwen3.5-9B-Base, Qwen3.5-35B-A3B-Base, Gemma4-26B-A4B-Base, GLM-4.5-Base, MiMo-V2.5-Base), Pass@1 spans 7.12%–37.70% on SWE-bench Verified with SWE-agent, 4.00%–39.23% with OpenHands, 12.50%–26.69% on Terminal-Bench 2.1, and 14.64%–29.44% on BrowseComp-Plus.
• Capability tiers transfer, exact rankings do not. SWE-agent and OpenHands Pass@1 rankings on SWE-bench Verified correlate at Spearman ρ ≈ 0.89, but individual models move: Qwen3.5-9B-Base rises from 21.85% (SWE-agent) to 32.95% (OpenHands) while Qwen3-30B-A3B-Base falls from 7.38% to 4.00%, and the order of Qwen3.5-35B-A3B-Base and MiMo-V2.5-Base varies across benchmarks.
• Repeated sampling separates occasional from stable capability. Under SWE-agent, MiMo-V2.5-Base reaches 37.70% Pass@1 with 82 instances solved in all eight runs, and Qwen3.5-35B-A3B-Base 34.23% with 67; Qwen3-30B-A3B-Base rises from 7.38% Pass@1 to 24.20% Pass@8 yet solves zero instances in all eight runs.
• A sharp pretraining transition at stage 3. On an internal 30B-A3B MoE base model, Pass@1 goes from 10.15% at the end of stage 2 to 27.30% at the end of stage 3 (stage3-mid already 22.95%), then to 29.95% at stage 4 and 30.65% at stage 5. Stage 3 is where trajectory-like data and 32K long-context training are first introduced.
• ASAP stays sensitive where conventional benchmarks flatten. Within-stage gains in ASAP Pass@1 are 2.23, 4.35 and 1.10 percentage points for stages 1, 3 and 4, while the conventional benchmark aggregate changes by only +0.25, +0.09 and −0.13 points. Stage 2 is the exception, where the conventional aggregate gains +2.31 points versus +0.73 percentage points for ASAP, consistent with its QA/instruction-heavy annealing mixture.
• Behavioral decomposition localizes what each stage buys. From stage 1 to stage 2, parsable turns rise from 88.81% to 99.41% and valid tool calls from 79.78% to 97.06%, but only 36.33% of trajectories produce an applicable patch and 10.15% resolve. From stage 2 to stage 3, autonomous edits reach 42.86%, active testing 65.64%, applicable patches 63.88% and resolution 27.30%, with repetitive command loops and abnormal termination becoming substantially less common.
• Post-training mainly buys reliability, not new primitives. Comparing base (ASAP) to post-trained (original SWE-agent) models, Qwen3.5-35B-A3B goes from 34.23% to 49.40% and MiMo-V2.5 from 37.70% to 74.00%; explicit submission rises by +31.60 and +62.21 points, valid re-edit after a failed test by +31.98 and +28.99, and empty patches fall by 9.59 and 33.07 points. The authors state this comparison is descriptive, not scaffold-controlled.
• Different base models fail differently. Qwen-Base already edits and tests in a large fraction of trajectories and mainly lacks recovery, refinement and reliable completion, whereas MiMo-Base more often stays in exploration and never transitions into effective editing.
• The interaction format matters a lot. On a 100-instance SWE-bench subset with SWE-agent, Markdown+XML gives Pass@1 of 33.00 (Qwen3.5-35B-A3B) and 42.00 (MiMo-V2.5-Base), versus 27.00/38.00 for pure XML, 21.67/30.67 for Markdown, and 4.33/14.67 for pure JSON.
• Longer prefixes help and preserve ordering. Going from K=0 to K=3 lifts Qwen3.5-35B-A3B from 24.00 to 33.00 and MiMo-V2.5-Base from 28.00 to 42.00 Pass@1; K=3 is chosen because longer prefixes began exposing solution-bearing information such as parts of the gold patch.
• Results are robust to prefix source. Swapping in independently generated prefixes (Prefix B, from GLM-5.3-Flash) changes scores by at most 3 percentage points and leaves the model ordering unchanged.
• Ordering survives a shared SFT recipe. After the same 70K-example lightweight SFT on four models, the model ordering is identical across base Pass@1, base Pass@8, post-SFT Pass@1 and post-SFT Pass@3 (Spearman ρ = 1.0), which the authors treat only as an aggregate consistency check.
• Network shortcutting is rare. Across 24,000 audited trajectories, 0.60% attempt any external network access and 0.07% are flagged as seeking a gold solution; under a pessimistic assumption this could contribute at most 0.23 percentage points of Pass@1 for any audited model.

Methodology in Plain English

The core design principle is that a base model should be evaluated as a continuation model, not as an incomplete chat model. Rather than wrapping it in a system prompt and chat template it was never trained for, ASAP keeps the plain text-completion interface and changes only what the model sees.

Three pieces make this work. First, each agent step is written as structured text: Markdown for free-form reasoning plus XML tags such as <bash> and <observation> to delimit actions and environment feedback — a shell-log style the authors argue resembles pretraining data, and which they verify empirically beats JSON, pure Markdown and pure XML. Second, each task is bootstrapped with a short prefix taken from the first few steps of a reference trajectory produced by an instruction-tuned model, acting as few-shot guidance about the expected interaction protocol; the authors check that this prefix does not reveal the target patch, and use the same prefix for every model so comparisons are controlled. Third, the harness applies at most one gentle nudge per failure mode: if a model tries to submit before editing any file, the submission is rejected once with a hint; if it repeats the identical action, the redundant action is suppressed once with a hint. After that, the trajectory is simply terminated — the harness never inserts task-specific actions or edits.

Generation proceeds one completion call per step, starting from ## Thinking and stopping after a complete action; the harness executes the action, appends the observation, and continues until <submit/>, a step cap, or a degenerate loop. The primary metric is Pass@k using the unbiased estimator, complemented by Intersection@n, the number of instances solved in all n runs, which separates lucky occasional success from stable capability. To study pretraining, the authors run nine checkpoints across five stages of an internal 30B-A3B MoE model, and they align trajectories per SWE-bench instance and run index (4,000 paired trajectories per checkpoint across 500 tasks) so behavioral metrics can be compared like-for-like.

Why This Matters

Impact on research. The paper argues that agent evaluation need not be purely a post-training activity. By showing that ASAP separates checkpoints where conventional pretraining benchmarks barely move, it makes the case that end-to-end interaction is a usable pretraining-side signal — and, unlike scalar proxies such as trajectory-derived QA or trajectory likelihood, it yields a behavioral trace explaining why a score changed. That converts a benchmark into a diagnostic for pretraining data design.

Real-world applications.

• Selecting checkpoints and base models for agentic products before committing to expensive post-training.
• Auditing whether a pretraining data mixture is actually teaching tool use, editing and recovery, rather than only improving interface compliance.
• Debugging agent pipelines by attributing failures to specific behaviors (no exploration, no editing, no recovery) instead of a single accuracy number.
• Providing a low-cost sanity check on whether an added data source (for example trajectory-like data or longer context) pays off.

Industry relevance. Teams that train their own base models currently decide when to stop pretraining and which checkpoint to post-train using knowledge, math and code benchmarks. ASAP offers a practical, weight-free way to include agentic capability in that decision, and its finding that post-training mostly improves reliability rather than introducing new workflow primitives suggests where post-training data and objectives should be targeted.

Future Directions

• Does ASAP predict post-SFT agent performance? The 70K-example SFT check yields identical ordering with Spearman ρ = 1.0, but the authors explicitly caution that four models is too few to claim ASAP forecasts post-trained performance; a larger sweep is the obvious follow-up.
• Closing the specific post-training gaps. The largest base-to-post-trained deltas are in recovery (valid re-edit after a failed test), explicit submission, and empty-patch avoidance, raising the question of which pretraining or post-training data would teach these directly.
• Removing or shortening the reference prefix. The prefix improves elicitation but the authors capped it at K=3 because longer prefixes leaked gold-patch information; whether prefixes can be replaced by purely structural cues, or made leak-proof at longer lengths, is open.
• Broadening the diagnostic. The study covers software engineering, terminal use and deep research on one internal 30B-A3B model's checkpoints plus six open base models; extending the behavioral decomposition to more pretraining runs, model families and longer-horizon scaffolds remains untested. The supplied paper text is also truncated inside the network-access audit, so the authors' closing remarks on that audit are not fully available here.

Target Audience

Pretraining and foundation-model teams who need to judge agentic readiness before post-training; agent-benchmark and evaluation researchers interested in base-model-compatible protocols; and post-training engineers who want a behavioral breakdown of where a given base model's agent workflow breaks down. Readers need some background in LLM training stages and agent scaffolds to get the most from it, but the central idea — evaluate continuation models as continuation models — is stated plainly enough to be accessible to newcomers.

Authors’ abstract

Agentic capabilities are increasingly important for modern LLMs, yet remain difficult to evaluate during pretraining. Existing approaches typically rely on static trajectory-derived tasks or likelihood-based proxies, which are lightweight but provide limited visibility into how base models behave in interactive environments. We introduce ASAP (Agent Scaffold Adaptation for Pretraining Evaluation), a lightweight framework for eliciting and evaluating agentic behavior from pretrained base models on end-to-end tasks. Rather than converting base models into chat-style agents, ASAP preserves the completion interface and adapts existing agent scaffolds through structured trajectory continuation, short reference prefixes, and minimal recovery mechanisms. Across software-engineering and deep-research benchmarks, ASAP yields discriminative end-to-end performance across base models and pretraining checkpoints. Beyond aggregate scores, ASAP exposes model-generated execution trajectories, enabling analysis of exploration, editing, testing, recovery, and task completion across training stages. These trajectories provide a behavioral view of how agent-relevant capabilities change during pretraining and how execution patterns differ after post-training. Our results suggest that end-to-end agent evaluation can complement conventional pretraining benchmarks by providing a practical diagnostic of interactive model behavior.