Interlock: From Local Correctness to Task Completion in Long-Horizon Agents

AI summary

Overview

Research area: Natural Language Processing — specifically training-data synthesis and post-training for long-horizon LLM agents that operate inside tool-using, file-editing environments.

Technical level: Advanced. The paper formalizes tasks as typed graphs with set-theoretic relations and derives a rejection-sampling fine-tuning objective; readers should be comfortable with agent training pipelines and rubric-based evaluation.

Scope in one sentence: The paper introduces Interlock, a task-synthesis framework that builds agent training tasks from explicit "completion graphs" linking evidence to decisions and decisions to required downstream effects, then fine-tunes two Qwen3.6 backbones on their own verified successful rollouts.

What This Paper Is About

Long-horizon agents often produce work that looks right at each individual step but is incomplete as a whole — a correct decision is made, yet the records, schedules, or budgets that should reflect that decision are never updated. The authors argue these failures come from relationships the agent must maintain between environmental evidence, consequential decisions, and required downstream effects, and that existing synthesis methods do not preserve those relationships. Their goal is to make those relationships an explicit constraint during task generation so that complete executions become usable, transferable training data.

Key Contributions

1. A typed representation of task completion. Tasks are modeled as completion graphs with three relation types: grounding (which evidence and rules support a decision), coupling (hyperedge constraints requiring several decisions or outputs to satisfy a shared condition jointly), and realization (which downstream artifacts or states must reflect a decision). Each required effect carries four fields: the establishing evidence, the source population, the applicability condition, and the outcome condition.

2. Graph-grounded synthesis with dual review. The same graph is compiled into three artifacts — a resettable workspace, a public request plus private deliverable contract, and an evaluation rubric — by three specialized generation roles. Two complementary reviews follow: a source-coverage review tracing every graph relation into each artifact, and a public-input critic that checks whether the rubric is actually supported by solver-visible materials.

3. Completion-filtered self-supervision. An agent-as-judge with terminal access inspects execution artifacts and writes and runs verification scripts, returning pass/fail per mandatory criterion. Only trajectories satisfying all mandatory requirements are retained for rejection-sampling fine-tuning, with each backbone trained on its own accepted rollouts.

4. Empirical validation of transfer. Training on synthetic experience improves both backbones on three independently authored external benchmarks, and an ablation isolates the contribution of executable agentic verification over text-only LLM judging.

Main Findings

• Gains on all benchmarks for both backbones: Interlock-RFT scores highest in every column of Table 1, beating both the original checkpoint and ClawGym-SFT. For Qwen3.6-27B: AutomationBench task pass rate 14.33 → 18.33, JobBench Main 14.11 → 18.46, JobBench Easy 54.46 → 61.10, Harvey LAB 73.96 → 77.85. For Qwen3.6-35B-A3B: 9.00 → 13.50, 11.81 → 15.01, 51.86 → 56.17, and 54.18 → 58.34.

• Reported gain ranges: 4.00–4.50 percentage points on AutomationBench task pass rate, 3.20–6.64 points on JobBench weighted score, and 3.89–4.16 percentage points on Harvey LAB criterion pass rate. The 27B model gains 4.00, 4.35, 6.64, and 3.89 respectively; the 35B-A3B model gains 4.50, 3.20, 4.31, and 4.16.

• ClawGym-SFT degrades most metrics: On both backbones, the released-demonstration SFT baseline falls below the original checkpoint on AutomationBench (14.33 → 9.00 for 27B; 9.00 → 5.17 for 35B-A3B), JobBench Main (14.11 → 11.82; 11.81 → 6.25), and JobBench Easy (54.46 → 48.01; 51.86 → 40.66), while slightly improving Harvey LAB (73.96 → 75.03; 54.18 → 54.97).

• Broader domain coverage helps at fixed data volume: Holding training at 8,957 trajectories and sweeping 16, 32, 64, and 128 domains, performance increases at every successive setting on all four metrics, with 128 domains highest.

• Executable verification beats text-only judging: With the same judge backbone, rubrics, rollout pool, and acceptance rule, agent-as-judge filtering reaches 18.33 / 18.46 / 61.10 / 77.85 versus 13.17 / 15.14 / 56.53 / 75.28 for LLM-as-judge. Judge output rises from 11.03k to 12.28k tokens per rollout (an 11.3% increase, about 1.25k additional tokens).

• Synthesized tasks elicit much longer executions: Across 32,370 normally terminated Interlock trajectories versus 24,497 ClawGym trajectories, Interlock averages 84,293 recorded tokens, 27.63 assistant turns, and 42.10 tool calls, compared with 14,940 tokens, 11.92 turns, and 14.76 calls — ratios of 5.64x, 2.32x, and 2.85x. Medians are 25 versus 9 assistant turns and 40 versus 12 tool calls.

• Completion failures are concrete and coupled: In one case study, an attempt allocates 970 HX units against 770 available, plans 3,920 kits against 3,250 shared trays, partially serves an unsplittable 230-unit order, and omits a selected $6,800 expedite cost from contribution margin. In another, an attempt omits two of six due trainees, schedules a 12:00–14:00 class outside an 08:00–11:30 release window, and assigns an instructor on February 16 whose credential expires February 12.

• Task statements occupy a distinct, dispersed region: A PCA–t-SNE projection of BGE-M3 embeddings shows Interlock tasks spread broadly and in multiple groups, while AutomationBench, JobBench, and Harvey LAB form dataset-specific clusters — supporting the out-of-distribution framing of the transfer results.

Methodology in Plain English

The authors start from a reusable library of business contexts, task families (each with a standard set of obligations), and archetypes (recurring relationship patterns, such as several outputs sharing one decision or several allocations sharing one capacity limit). Scenario parameters fill in the concrete entities and starting records. From these, a builder produces a "completion graph" — essentially a map of what evidence exists, what the agent must decide, which decisions must be consistent with each other, and which files or records must change as a result.

Three generator roles then turn that one graph into three things that must stay mutually consistent: a workspace of files with the evidence and half-finished artifacts, a user request plus a private contract listing required deliverables, and a hidden rubric of outcome criteria. A "carrier map" records exactly where each graph requirement landed in each artifact, so reviewers can trace it.

Two reviews then run in opposite directions. Source coverage walks forward from the original obligations and checks each relation survived into the workspace, the request, and the rubric — so a requirement cannot silently disappear just because both the request and rubric dropped it. The public-input critic sees only the request, the workspace, and the rubric, and checks that the criteria are actually inferable from materials the solver can see and judgeable from the artifacts produced.

If both reviews pass, the task is executed from a fresh workspace copy and a judge examines the initial state, final state, and the diff. Admission does not require the validation solver to succeed — hard tasks are kept; only defective packages are rejected and sent back, with bounded, versioned revisions.

For training, each backbone makes five attempts per admitted task from a reset workspace. An agent-as-judge with terminal access, guided by the rubric, inspects files and can write and execute verification scripts. A trajectory is accepted only if every mandatory criterion passes and any declared quality threshold is met. Accepted trajectories are used for standard supervised fine-tuning on assistant tokens covering reasoning, responses, and tool calls; workspace observations stay as context only. Crucially, each model is trained only on its own successes, not on another model's.

For partial-credit diagnostics (separate from acceptance), criterion scores propagate along declared depends_on edges, essential-criterion failures zero out their group, and critical groups are capped at half the total rubric weight.

Why This Matters

Impact on research. The paper reframes agent data synthesis around conditional obligations rather than tool coverage or task length. Most prior synthesis work — tool-use datasets, environment generators, task composers, graph-guided skill pipelines — scales breadth or difficulty; Interlock scales the density of relationships an execution must preserve. The result that SFT on released demonstrations actively hurt performance on three of four metrics for both backbones, while self-generated verified successes helped, is a notable empirical signal about data provenance for agent post-training. The judge ablation also quantifies a concrete benefit from giving the verifier the ability to run code rather than only read text.

Real-world applications:

• Operations and supply recovery planning — reconciling supply constraints, whole-order service rules, and action costs in one plan.
• Workforce scheduling and compliance — jointly satisfying due-trainee scope, release windows, and dated instructor credentials.
• Financial close and payroll — ensuring a revised allocation propagates to schedules, budgets, and contribution margin.
• Legal and professional deliverables — Harvey LAB-style work where criterion-level completeness matters more than a single final answer.
• Agent evaluation design — the dual-review scheme (spec coverage plus public support) is directly reusable as a QA gate for any synthetic benchmark pipeline.

Industry relevance. Teams building agentic products need training and evaluation data that punishes "locally correct but incomplete" behavior, which is exactly the failure mode users notice in deployed assistants. The pipeline is model-agnostic and the reported gains hold across two backbones and three external benchmarks with results reported as 4–7 points on 0–100 scales — meaningful but not saturated — which is the kind of delta that matters for production reliability work.

Future Directions

1. Scaling the domain and task-family library. The retained corpus spans 128 domains and 484 task families, and the domain sweep was still improving monotonically at 128. Whether gains continue past that point, and how many trajectories per domain are needed, is not reported.

2. Stronger or cheaper verifiers. Agent-as-judge raised judge output by roughly 1.25k tokens per rollout. Whether smaller judge models, cached verification scripts, or partial-script reuse can retain the gain at lower cost is open, as is the failure-mode profile of judges on adversarial artifacts.

3. Beyond rejection sampling. The current recipe keeps only binary-accepted trajectories and discards all partial-credit signal, even though the rubric computes a weighted partial score. Using graded credit, or learning from near-misses where a single realization edge failed, is a natural next step the paper does not test.

4. Closing the follow-through gap. The appendix defines a follow-through rate over decision units and reserves a frozen diagnostic set with composition-transfer and intervention variants, but the main text reports no FTR values. Quantifying how much of the residual completion gap persists after training — and whether it concentrates in specific archetypes — remains unanswered.

Target Audience

Researchers and engineers working on agent post-training, synthetic data generation, and agentic evaluation. It is most useful to those who already run rejection-sampling fine-tuning or rubric-based judging pipelines and want a principled way to make generated tasks test completeness rather than isolated tool calls. Benchmark designers will find the dual-review admission scheme and the per-effect four-field obligation format directly portable. Readers seeking a beginner introduction to agent training should look elsewhere first, given the formal graph notation and the density of experimental detail.

Authors’ abstract

Large language model agents increasingly undertake long-horizon tasks in complex environments characterized by fragmented evidence, partially completed states, and dependencies across resources and artifacts. Existing agents may satisfy individual requirements while failing to maintain these relationships, producing locally correct but incomplete outcomes. These failures arise because task completion depends on relationships among environmental evidence, consequential decisions, and required effects. To address this completion gap, we propose Interlock, a framework for task synthesis that represents each task as a typed completion graph. Grounding edges connect public evidence to decisions, coupling hyperedges impose joint constraints across components, and realization edges connect decisions to all required downstream effects. Interlock compiles this graph into a resettable workspace, a public request, and an evaluation rubric. A source-coverage review checks whether the original requirements survive generation, while a public-input review checks whether the rubric is supported by the materials available to the solver. An agent-as-judge inspects execution artifacts and runs verification scripts, retaining trajectories judged to satisfy all mandatory requirements. To evaluate training utility, we fine-tune two Qwen3.6 backbones on their own accepted trajectories. Across the two backbones, training improves AutomationBench task pass rates by 4.00-4.50 percentage points, JobBench weighted scores by 3.20-6.64 points, and Harvey LAB criterion pass rates by 3.89-4.16 percentage points. These results demonstrate the training utility of the synthesized experience across external task environments.