ICA: Information-Aware Credit Assignment for Visually Grounded Long-Horizon Information-Seeking Agents

AI summary

Overview

Research area: Reinforcement learning for long-horizon web information-seeking agents (also called deep-research agents), combining multimodal webpage representation with credit assignment for tool-use trajectories.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning for large language models (group-based policy optimization, importance ratios, advantage normalization) and with agentic tool-use pipelines (search, fetch, answer).

Scope: The paper proposes Information-Aware Credit Assignment (ICA), a method that represents fetched webpages as rendered visual snapshots and assigns dense turn-level rewards to the search and fetch actions that introduced high-utility evidence, integrated with GSPO and evaluated on BrowseComp, GAIA, Xbench-DS, and Seal-0.

What This Paper Is About

Long-horizon web agents normally receive only a terminal reward telling them whether the final answer was correct, which gives no signal about which search or fetch actions actually supplied the useful information. The problem is made worse by text-based webpage processing, where parsing, truncation, chunking, and summarization make the same webpage look different across trajectories, so acquired evidence cannot be reliably identified or compared. The paper's goal is to turn long-horizon web agent learning into an "evidence attribution" problem: represent each fetched page as a stable, comparable unit (a rendered snapshot), estimate which units are associated with successful rollouts, and propagate that signal back to the turns that retrieved them.

Key Contributions

1. Reframing long-horizon web agent learning as an evidence attribution problem. Retrieval actions are optimized through the external information they introduce rather than only through terminal answer outcomes, using the notion of an "atomic evidence unit" (an individual Search result entry or a complete Fetch webpage snapshot).

2. Rendered webpage snapshots as stable content-level observations. Fetched pages are represented as full-page visual snapshots that preserve layout, tables, images, and multimodal cues, so the same source can be identified and compared consistently across rollout trajectories, replacing lossy text parsing, chunking, or summarization.

3. Information-Aware Credit Assignment (ICA). A post hoc reward-propagation method that computes an association-based utility score for each evidence unit from rollout success rates, ranks units by that score, and assigns dense binary rewards to the tool turns that introduced high-utility evidence, while giving the terminal answer turn an asymmetric, budget-aware shaped reward.

4. ICA-GSPO integration with consistent empirical gains. ICA is combined with a turn-level GSPO objective and evaluated on BrowseComp, GAIA, Xbench-DS, and Seal-0, where the 8B model is reported to outperform most reported open-source agents under 15B parameters.

Main Findings

• Main benchmark results. On the 8B-scale model, Qwen3-VL-8B-ICA (Search & Fetch with snapshots) reports Pass@1 of 25.0 on BrowseComp, 69.9 on GAIA, 71.0 on Xbench-DS, and 25.2 on Seal-0. On the larger model, Qwen3-VL-30B-A3B-ICA reports 26.1 on BrowseComp, 72.8 on GAIA, 76.0 on Xbench-DS, and 27.0 on Seal-0. Both are listed under open-source agents (the 8B under 15B parameters, the 30B-A3B under greater than 15B).

• Comparison with proprietary systems. For reference in the same table, OpenAI-o3 is listed at 49.7 (BrowseComp), 70.5 (GAIA), 66.7 (Xbench-DS), 18.9 (Seal-0); OpenAI DeepResearch at 51.5 (BrowseComp) and 67.4 (GAIA); Claude-4-Sonnet at 12.2 (BrowseComp), 68.3 (GAIA), and 64.6 (Xbench-DS). The paper states ICA-30B-A3B remains competitive on BrowseComp and Seal-0 despite a smaller model size and a fully open training pipeline.

• Snapshot versus text in supervised fine-tuning. In the ablation on the 8B backbone, SFT with text fetches reports 11.0 (BC-100), 54.3 (GAIA), 55.0 (XDS), 20.7 (Seal-0), while SFT with snapshots reports 18.0, 56.4, 66.0, 23.4 — gains of 17.0, 27.3, 27.0, and 16.2 points respectively over the text-only baseline (1.0, 29.1, 39.0, 7.2).

• ICA versus vanilla GSPO. With snapshots fixed as the fetch policy during RL, ICA - Snap. reports 25.0 (BC-100), 69.9 (GAIA), 71.0 (XDS), 25.2 (Seal-0), compared with GSPO - Snap. at 22.0, 60.2, 68.0, 24.3. The paper states the improvements are largest on long-horizon, high-noise tasks.

• Token efficiency of snapshots. Figure 2(b) reports that snapshots reduce token usage by 27.0% to 65.6% compared with parsed text across four URLs.

• Context overflow elimination. Under a strict 128k context limit with a 16k tool-response truncation threshold, SFT - Text shows context overflow rates of 36.0 (BrowseComp), 15.5 (GAIA), 6.0 (XDS), and 20.0 (Seal-0), while SFT - Snap. shows 0.0 overflow on all four datasets. The paper attributes a 7.0-point Pass@1 gain on BrowseComp and an 11.0-point gain on Xbench-DS to this lower token footprint; on GAIA accuracy rises from 54.3% to 56.4% while overflow is eliminated.

• Selected evidence is answer-supporting. In a length-controlled analysis over trajectory buckets of 0–19, 20–39, 40–59, 60–79, and 80–99 interaction steps, with dynamic top-p% selection for p in {10, 15, 20}, top-p% ICA evidence reaches an average accuracy of 71.0%, versus 58.2% for randomly sampled evidence and 50.2% for the lowest-scoring ("tail") evidence. The advantage holds across all trajectory buckets and selection ratios, which the authors read as evidence that ICA is not simply biased toward longer trajectories.

• Δe as a utility proxy. The paper reports a strong correlation between the association-based utility score Δe and answer correctness in ablations, supporting its use as a ranking proxy rather than a direct continuous reward. Note that Δe is computed only for rollout groups with mixed outcomes; groups with uniform outcomes are assigned zero tool-turn advantages, since they provide no contrastive signal.

Methodology in Plain English

The agent interacts with the web through three actions: Search (returns a ranked list of results with URL, title, snippet), Fetch (retrieves the content of selected URLs), and Answer (produces the final response). A group of trajectories is sampled for each prompt, and each trajectory gets a binary reward for whether the final answer is correct.

The core idea is to define small, comparable pieces of evidence — "atomic evidence units" — at the URL level: either a single Search result entry or a complete fetched webpage snapshot. Because snapshots are rendered full pages rather than parsed or summarized text, the same webpage produces the same unit across trajectories, so units can be pooled across the whole group.

For each unit, the method computes how often a trajectory succeeded when the unit was present (P+) and how often it succeeded when the unit was absent (P−). The difference, Δe = P+ − P−, is the unit's association-based utility. Units are ranked by Δe, and the top fraction ρ (set to 10% in experiments) of each tool type with positive Δe are marked as high-utility.

Tool turns that captured at least one high-utility unit receive a reward of 1; others receive 0. The final answer turn instead gets a shaped reward: 1.0 if correct, and −1.0 + sqrt(|τi| / Lmax) if incorrect, which softens the penalty for trajectories that used more of the context budget, protecting thorough exploration from being punished. Advantages are normalized separately within each scope — tool-turn rewards against the group mean and standard deviation for that tool type, answer rewards against their own group statistics — so search quality and final correctness are not mixed.

Optimization uses a turn-level GSPO objective. Instead of one importance ratio over an entire trajectory, each turn gets its own geometric-mean ratio over the tokens in that turn, clipped at 1 ± ε, which keeps a large policy shift in one turn from contaminating the trajectory-level estimate. The answer turn's objective is weighted by a fixed factor w (set to 0.3) and the average of the non-answer turns by (1 − w), so the single answer turn's signal is not diluted by many preceding tool turns.

For observations, the text baseline uses Trafilatura-parsed content, while the snapshot modality uses Playwright rendering with a browser pool of 8 Chromium instances, up to 32 concurrent rendering requests, an auto-scrolling procedure for lazy-loaded elements, denoising of overlays such as cookie banners and popups, a hard maximum rendering height of 20,000 pixels, sliding-window slicing with slice height 4,480 pixels and vertical overlap 112 pixels, and 0.7× LANCZOS downsampling. Search uses the Serper API. The comparison text pipeline includes a RAGTool using asynchronous httpx with up to 32 concurrent requests, a JINA API fallback, 256-token chunks with a naive_merge strategy, BAAI/bge-reranker-v2-m3 reranking, top-K with K = 10, and dynamic truncation to a default 2048-token budget.

Training uses Qwen3-VL-8B-Thinking and Qwen3-VL-30B-A3B-Thinking backbones, 50 steps on NVIDIA H800 GPUs, with reported costs of approximately 443 GPU-hours for the 8B model and approximately 534 GPU-hours for the 30B-A3B model. Rollout sampling uses temperature 0.6 and top-p 0.95. Evaluation is on GAIA's 103-example text-only validation subset and the 2505 release of Xbench-DS, with Pass@1 measured by LLM-as-a-judge.

Why This Matters

Impact on research. The paper shifts attention in agentic RL from "which turn was responsible" to "which piece of acquired evidence was responsible," and argues that the observation modality itself (rendered snapshots versus parsed text) is a precondition for credit assignment rather than just an input choice. It also provides a counterpoint to text compression pipelines common in deep-research agents, showing that compression can break cross-trajectory evidence identity. The reported gains over text-based and outcome-only baselines across two model scales support the claim that evidence-level credit assignment is a useful training signal.

Real-world applications.

• Deep-research and analyst assistants that must gather evidence from many pages and justify answers, where knowing which sources were decisive matters for trust.
• Enterprise knowledge work and competitive intelligence, where agents repeatedly search and fetch across long sessions under a fixed context budget.
• Customer support and technical troubleshooting agents that must retrieve the right document or product page before answering.
• Compliance, due diligence, and fact-checking workflows where verifiable retrieval actions and citations are as important as the final answer.

Industry relevance. The results are reported on open models (Qwen3-VL-8B-Thinking and Qwen3-VL-30B-A3B-Thinking) with an open training pipeline, and the paper states code and datasets will be released at https://github.com/pc-inno/ICA_MM_deepsearch. The 8B configuration is reported to outperform most reported open-source agents under 15B parameters, and the token-efficiency and zero-overflow results under a 128k context limit matter directly for serving costs and for agents running near their context ceiling.

Future Directions

• Visual token compression. The limitations section states that raw snapshots are fed directly into the Vision Transformer with no intermediate visual compression, spatial pooling, or salient-region filtering, causing token redundancy and computational overhead on visually dense or very long pages.
• Context and memory management. The paper notes there is no sophisticated strategy for unbounded exploration: without historical summarization, token eviction, or memory-based sliding windows, accumulated tokens can exceed the context limit, forcing truncation that may discard early-retrieved crucial evidence.
• Credit assignment under uniform outcomes. Because Δe is only computed for groups with mixed outcomes and all tool-turn advantages are set to zero otherwise, understanding how to extract signal from groups that succeed or fail uniformly remains open.
• Extending evidence-level attribution beyond web search and fetch. The framework is defined around Search and Fetch tool types on webpages; whether the same atomic-evidence utility estimation transfers to other tool-use settings or other observation modalities is not established in the paper.

Target Audience

Researchers and engineers working on reinforcement learning for LLM agents, deep-research or web-navigation agents, and multimodal agent observation design. It is most useful to readers already comfortable with group-based policy optimization methods (GSPO, DAPO, Dr. GRPO) and with the mechanics of agentic search-and-fetch pipelines. Readers looking for an accessible introduction to agentic RL will find the conceptual framing (evidence attribution, snapshots as stable evidence units) clear, while the training objective and advantage-assignment equations require an intermediate-to-advanced background.

Authors’ abstract

Long-horizon reinforcement learning for information seeking agents remains difficult because terminal rewards reveal whether the final answer is correct, but not which acquired information enabled it. This difficulty is amplified by text-derived webpage observations, where parsing, truncation, and summarization often produce incomplete and unstable content representations across trajectories. We propose an evidence-centric framework for web agent learning that represents information acquired through tools as identifiable units for comparison across trajectories. In particular, fetched webpages are represented as rendered snapshots, preserving layout and multimodal content as stable content-level observations. Building on these units, we introduce Information-Aware Credit Assignment, a post hoc reward propagation method that estimates turn-level utility scores from rollout success rates and assigns dense rewards to intermediate steps that introduced high-utility information. Integrated with GSPO, our method consistently improves performance on BrowseComp, GAIA, Xbench-DS, and Seal-0. Code and datasets will be released at https://github.com/pc-inno/ICA_MM_deepsearch.