CLIFT: Conformal Self-Verification for Web Agent Training and Test-Time Scaling

AI summary

Overview

Research area: Reinforcement learning for LLM-based web agents, with a focus on verifier design, credit assignment, and judge-free test-time scaling using conformal prediction.

Technical level: Advanced. The paper combines GRPO-style policy-gradient training, Mondrian adaptive conformal inference, and comparative LLM judging; the high-level idea is accessible, but the certification machinery and reward equations assume familiarity with RL and conformal prediction.

Scope: CLIFT is a single certified question-bank that is used both to densify per-step RL rewards during training and, once frozen, to select among candidate trajectories at test time without calling any external judge.

What This Paper Is About

Open-source web agents can now execute realistic browser tasks, but improving them with reinforcement learning is bottlenecked by weak supervision: a binary success signal is too sparse (when every rollout in a GRPO group gets the same outcome, the advantage estimate collapses), while frontier-model judges are expensive to call at every step and cannot be assumed available at deployment. CLIFT asks whether judge feedback can be distilled into a compact, reusable verifier that the agent carries with itself — a bank of natural-language YES/NO questions about its own behaviour, each tagged with a URL scope and a polarity sign, and each certified only where observed evidence agrees with the judge. The same bank then serves as structured evidence for trajectory selection at inference, with no judge in the loop.

Key Contributions

1. A certified verification bank as a reusable object. CLIFT maintains a bank of natural-language verification questions, each with a URL pattern and a polarity sign in {−1, 0, +1} indicating whether YES means progress, failure, or is ambiguous. A Compositional Conformal Certifier (CCC) keeps only questions whose URL-conditional evidence agrees with a training-time judge, using separate Mondrian-ACI trackers at global, host, and URL-path granularity, and assigns each certified question a signed trust weight in [−1, +1] derived from polarity-aware lift.

2. An asymmetric training reward with URL-stratified advantages. The verifier score is added to the per-step judge reward through r̃ = r_judge + β·max(0, R^VQ(u)), so certified evidence can only strengthen the judge signal and never subtract from it — the paper motivates this as a fix for the reward-collapse failure of earlier linear blends. Returns are then group-normalised within each (rollout-group, step-position, URL-state) cell, making credit assignment URL-local while leaving the GRPO update itself unchanged.

3. Conformal Trajectory Selection (CTS), a judge-free test-time scaling rule. The trained agent emits a greedy incumbent plus one or more diverse retries under a budget K; the frozen bank produces per-URL-state evidence sheets; a comparative self-verifier returns A/B/TIE; and a conservative majority rule over an odd number m of calls swaps only when at least ⌈(m+1)/2⌉ votes favour the retry, defaulting to the incumbent on ties. The paper states a no-regression-style bound, E[SR(CTS)] ≥ E[SR(ρ_A)] − ε(m, δ), with ε shrinking as the vote becomes more reliable.

4. Evidence that one bank transfers across backbones and benchmarks. The same mechanism is evaluated as in-domain training (WebArena Infinity), as bank transfer from an open model to a frozen closed model (VisualWebArena with GPT-5.5), and as translated-and-recalibrated zero-shot transfer to live-web tasks (Online Mind2Web, with no policy trained on it). Table 1 positions CLIFT as the only compared approach that is simultaneously step-dense, URL-grounded, certified, judge-free at test time, test-time-scaling capable, and transferable.

Main Findings

• State-of-the-art open-agent result on WebArena Infinity. On the public 9-app comparison set, Gemma-4 with CLIFT + CTS (K=3) reaches 74.6 overall, against 70.1 for Gemini 3 Flash + BU, 61.8 for the base Gemma-4 browser-use backbone, 50.7 for Qwen 3.5+, and 46.4 for Kimi K2.5. CLIFT wins 7 of 9 apps, and the gain over the same Gemma-4 backbone is +12.8 percentage points overall.

• Training, not extra search, produces most of the WAI gain. In the ablation (all training rows judged by Claude Opus 4.6), base Gemma-4 scores 61.8 with 15.4 average steps on successful tasks; vanilla GRPO reaches 64.7 (+2.9) but with 18.5 steps; CLIFT reaches 72.6 (+10.8) at 16.0 steps; CLIFT + CTS reaches 74.6 (+12.8) at 15.5 steps. The corrected Opus 4.6 reference is 69.8 (+8.0). The authors read this as certified URL-local feedback improving credit assignment rather than encouraging longer trajectories.

• CTS is conservative by construction. On WAI it improves, ties, and regresses 5 / 4 / 0 apps with a 17.8% average swap rate, adding 2.0 points over the single-rollout CLIFT anchor.

• The bank transfers to a frozen closed backbone. On VisualWebArena, GPT-5.5 + CLIFT scores 48.5 weighted average; adding CTS with K=4 raises it to 53.7 (Classifieds 62.4, Shopping 51.0, Reddit 50.0), narrowly above the WALT reference at 52.9 and above SGV at 50.2. For comparison, Gemma-4 base is 30.9, Gemma-4 + CLIFT is 34.4, Gemma-4 + CLIFT + CTS (K=2) is 35.6, and the human row is 88.7.

• Live-web transfer without any policy training. On Online Mind2Web (300 tasks, roughly 30 public domains, judged by an LLM rubric), the translated and recalibrated bank certifies 100 of 132 questions (75.8%), comprising 75 positive-lift and 25 negative-lift questions with mean lift 0.34. Realised success rises from 49.7 to 61.0 at K=4 on GPT-5.5 high-effort (+11.3 pp), and from 40.0 to 45.0 on Gemma-4 — the latter slightly above the GPT-5.5 low-effort baseline of 43.7.

• Bank provenance is tracked question by question. The final OM2W bank draws 36 questions from the WAI bank (33 certified, 91.7%), 48 from the VWA bank (31 certified, 64.6%), and 48 new live-web questions (36 certified, 75.0%).

• Certification is also a diagnostic. Uncertified questions are withheld rather than amplified; the paper reports that the noisier live-web judge leaves more questions uncertified, and describes a sanity veto that demotes a question's polarity to neutral when observed lift contradicts the tag.

Methodology in Plain English

The researchers start by writing a large set of simple yes/no questions an agent could ask about its own behaviour on a web page — things like "did I just apply the filter the user asked for?" Each question is tied to a URL pattern and tagged with a direction: for some questions, "yes" is good news; for others, "yes" means something went wrong.

Next they check which questions actually mean anything. For each question, at three levels of URL specificity (the whole web, a particular host, a particular path), they compare the question's answers against how a frontier-model judge scored the same rollouts. A question is kept only if it has been seen enough times (n_min = 50), its conformal error rate is below α* = 0.10, and its lift — the gap in judge score between "yes" and "no" cases — exceeds lift_min = 0.02. Kept questions get a signed weight; rejected ones get zero weight. A periodic loop proposes new questions aimed at separating the best and worst rollouts, and a veto demotes questions whose polarity turns out to be backwards.

During training, the agent answers the certified questions about its own rollouts with the task adapter detached. The resulting score at each URL is added to the judge's per-step reward, clipped at zero so it can never lower the judge's score. Rewards are converted to returns with discount γ, then normalised inside each group of rollouts at the same step and URL, so the comparison is between agents that were in the same place on the same site.

At deployment the judge is dropped entirely. The agent produces a greedy trajectory plus a few diverse retries under a budget K. For each pair, the frozen bank produces a structured evidence sheet of question answers, weights, lifts, and locations, which the self-verifier reads to pick A, B, or TIE. The same comparison is repeated an odd number of times m and the retry is adopted only on a clear majority; ties and uncertainty keep the greedy trajectory. Training used Gemma-4-31B-it with a rank-8 LoRA adapter across 16 single-node shards, G = 8 rollouts per prompt, peak eight nodes of 8× H200 GPUs, on 1,886 synthesized WAI tasks and 1,272 synthesized VWA tasks; the training pool was built from 186 WAI seeds and 206 VWA seeds expanded roughly 20× per seed and filtered for malformed, near-duplicate (cosine 0.92), hardcoded, and trivial-navigation tasks, with no evaluation task descriptions read during synthesis, filtering, sharding, or training.

Note on completeness: the paper states that the concrete hyperparameter values for all four stages are listed in Appendix A.3, but that section is cut off in the supplied content, so the specific values of β, m, γ, and the vote counts used in the reported runs are not reported here.

Why This Matters

Impact on research. The paper reframes the verifier as a durable, auditable artefact rather than a model or a prompt. A trajectory score is consumed once; a certified question bank can be inspected, recalibrated, and reused — across training and test time, across open and closed backbones, and across sandboxed and live-web judges. The demonstration that a bank learned while training an open Gemma-4 model improves a frozen GPT-5.5 at inference is a concrete argument that verification knowledge is separable from policy weights. The conservative selection rule, with its explicit no-regression-style bound and its default-to-incumbent behaviour, also offers a template for adding test-time compute without risking the baseline.

Real-world applications:

• Browser-automation products (form filling, procurement, invoicing, account management) that need a reliable "did this actually work?" signal without paying for a frontier judge on every step.
• Enterprise agent deployment in regulated settings, where an auditable list of URL-scoped checks with provenance back to the judge rationale is easier to review than an opaque reward model.
• Cost-controlled training pipelines that want to convert a small number of expensive judge calls into a reusable dense reward for many subsequent rollouts.
• Agent evaluation and monitoring, where certified per-URL questions can double as a diagnostic that flags when a judge, state partition, or question set no longer supports reliable supervision.

Industry relevance. The authors are at Salesforce AI Research, and the target is the practical deployment complaint about agentic RL: judges are too costly during training and unavailable in production. A bank that ships with the agent, is small enough to carry, and needs only a sample of judge-scored rollouts to recalibrate on a new domain is directly useful to teams that cannot serve a frontier model alongside every deployed agent.

Future Directions

• Replace natural-language questions with executable assertions. The authors propose having a coding agent write compact browser-state checks that can be executed during rollouts and reused for test-time selection, reducing dependence on a separate self-verification bank while preserving auditability.
• Broaden backbone coverage. Only one open-weight backbone was trained (Gemma-4-31B-it with LoRA), so whether the certification recipe behaves similarly across other open models is an open question.
• Address judge bias. The method relies on a frontier judge during training and bank construction; if that judge is systematically biased, calibration can detect disagreement only through observed data. How to certify against a biased teacher is unresolved.
• Test universality of the bank. The transfer results show that a bank can help stronger closed agents, but not that a single bank is universal across all web domains; how far the taxonomy generalises, and what recalibration costs on genuinely new domains, remains to be measured.

Target Audience

Researchers and engineers working on LLM agents, agentic reinforcement learning, or process-level reward design will get the most from this paper, particularly those already familiar with GRPO and interested in dense supervision without a learned process reward model. It is also relevant to practitioners in conformal prediction looking for a new application area, and to applied teams at companies deploying browser agents who need judge-free verification and a conservative, auditable test-time selection rule. Readers without an RL or conformal-prediction background can still follow the motivation, the benchmark results, and the design rationale for never letting the verifier subtract from the judge baseline.

Authors’ abstract

Open-source web agents are now strong enough to execute realistic browser tasks, but training them with reinforcement learning still depends on weak supervision: binary task success is too sparse for credit assignment, while frontier-language-model judges are too expensive to call at every step and cannot be assumed available at deployment. We introduce CLIFT, a training and test-time scaling method built around conformal self-verification. During training, the agent answers natural-language verification questions about its own rollouts; a Compositional Conformal Certifier keeps only question signals whose URL-conditional evidence agrees with a training-time judge, assigns signed trust weights through polarity-aware lift, and blends the resulting verifier score into per-step rewards in a way that never subtracts from the judge baseline. At test time, the same certified bank is frozen and reused as structured evidence for Conformal Trajectory Selection (CTS): the agent samples a greedy rollout and one or more diverse retries, the self-verifier summarises each URL trace, and a conservative majority-vote rule chooses whether to swap away from the current incumbent without calling any external judge. This single mechanism supports three settings. On WebArena Infinity, CLIFT achieves state-of-the-art performance among open-source web agents. On VisualWebArena, a bank trained with the open model transfers to GPT-5.5 at test time and reaches state-of-the-art performance under the canonical harness. On Online Mind2Web, without training an agent on the benchmark, translating the certified question bank improves a live-web agent in zero-shot evaluation. Together these results position conformal self-verification as a way to turn costly judge feedback into a reusable training signal and a judge-free test-time scaling signal.