Prune4Web: DOM Tree Pruning Programming for Web Agent

AI summary

Overview

Research area: LLM-based web agents and multimodal web automation, specifically DOM/HTML processing for element grounding. Technical level: Advanced. Scope: This paper introduces Prune4Web, a framework that replaces LLM-based DOM filtering with LLM-generated Python scoring programs, plus a data annotation pipeline and a two-turn dialogue training strategy for a Planner, Programmatic Filter, and Action Grounder.

What This Paper Is About

Web agents must pick the right element on a page to act on, but real webpage DOMs contain roughly 10,000 to 100,000 tokens, which exceeds what an LLM can handle well and dilutes the model's attention. Existing fixes either truncate the DOM (losing information) or use rigid heuristics or separate ranking models. Prune4Web's goal is to shrink the DOM to a small, high-quality candidate list before any LLM has to reason over it, by having the LLM write a scoring program instead of reading the page itself.

Key Contributions

1. A multimodal web agent framework that combines visual screenshot reasoning with HTML/DOM semantic precision.
2. Prune4Web's Programmatic Element Filter, which generates task-specific Python scoring programs to filter and rank DOM elements, addressing the DOM scalability bottleneck.
3. A data annotation method (using GPT-4o to add intermediate labels) and a two-turn dialogue training strategy that jointly optimizes the Planner, Filter, and Grounder, using SFT followed by RFT.
4. Empirical validation on standard benchmarks plus a new purpose-built low-level sub-task grounding benchmark (1101 trajectories).

Main Findings

• Large grounding gain: With a ground-truth low-level sub-task as input, the full Programmatic Element Filter plus Action Grounder pipeline reaches 88.28% grounding accuracy, versus 46.80% for fine-tuning without pruning.
• Pruning beats stronger models: The same 88.28% exceeds using GPT-4o as the Action Grounder (80.65%) and GPT-4o-mini (73.75%) under Prune4Web pruning.
• High recall with small candidate sets: Fine-tuned Qwen2.5-0.5B-Instruct and Qwen2.5VL-3B-Instruct both exceed 90% recall at N=3 and approach 95% at N=5, while GPT-4o reaches roughly 72% at N=3 and around 86% at N=20. Fine-tuned 0.5B performs almost identically to the 3B model.
• Baseline comparison under the same setting: End-to-end LLM pruning and decision yields 70.84% grounding for GPT-4o and 67.57% for GPT-4o-mini, below Prune4Web's programmatic filtering with the same Recall@20 scores (85.56 and 89.19).
• Oracle ceiling: With oracle pruning (ground truth guaranteed in top 20), fine-tuned Qwen2.5VL-3B-Instruct reaches 90.28%, versus 82.83% for GPT-4o and 75.39% for GPT-4o-mini; zero-shot Qwen2.5VL-3B and 7B reach only 11.99% and 12.08%.
• Multimodal-Mind2Web performance: The Two-turn Dialogue unified model achieves Element Accuracy / Operation F1 / Step Success Rate of 58.4 / 84.1 / 52.4 (Cross-Task), 50.2 / 81.2 / 44.9 (Cross-Website), and 49.2 / 84.4 / 46.1 (Cross-Domain). The Separate Models variant scores 46.0 / 83.4 / 42.2, 43.0 / 77.3 / 37.8, and 42.2 / 84.4 / 40.6.
• RFT improves planning: Adding RFT on top of SFT raises Step Success Rate from 37.9% to 42.2% for Separate Models and from 46.5% to 52.4% for the Two-turn Dialogue model.
• Architecture ablation (30 online tasks, LLM-Verified Task Completion Rate): For GPT-4o-mini, Action Grounder only yields 21.1%, adding the Planner gives 26.3%, and the full framework gives 31.6%.
• Programmatic filtering helps small models most: Versus LLM Top-N selection, Prune4Web filtering raises GPT-4o-mini from 26.3% to 31.6% and takes fine-tuned Qwen2.5VL-3B from 0.0% to 5.2% (GPT-4o stays at 42.1% both ways).

Methodology in Plain English

The pipeline has three stages. A Planner sees the high-level task, screenshot, and history, and writes a low-level sub-task such as "find the destination field" — it deliberately does not see the HTML. A Programmatic Element Filter then takes that sub-task and asks an LLM to generate only the key parameters (keywords and weights) for a fixed Python scoring template. A rule-based pre-filter keeps clearly interactive elements (<a>, <button>, <input>, certain roles) and attaches nearby text as context. The generated program runs outside the LLM, scoring every remaining element on a tiered scheme: match quality (exact > phrase > word > fuzzy) multiplied by attribute priority (visible text > trusted attributes like aria-label and placeholder > other attributes like class or id), weighted by the LLM-supplied keyword weight. The Top-N elements, with N defaulting to 20, are the only DOM content passed downstream. Finally, an Action Grounder picks the executable action from that shortlist; for abstract actions like "task complete," no candidate list is needed. Training uses an annotated version of the Multimodal-Mind2Web dataset — GPT-4o added low-level sub-tasks, keywords and weights, pruned DOM trees, and grounder thinking steps, giving roughly 5,000 interaction steps, split per MM2W's original train/test partitions. Models are Qwen2.5VL-3B-Instruct (and a lighter Qwen2.5-0.5B-Instruct), trained with SFT (either three separate models or one unified two-turn dialogue model), then GRPO-based RFT applied to the Planner with a binary reward summing format, filtering (did pruning keep the ground-truth element), and grounding components.

Why This Matters

Impact on research: The paper reframes DOM reduction as a code-generation problem rather than a classification or ranking problem, and shows that a 0.5B model can match a 3B model on filtering recall — evidence that the difficulty lies in the task formulation, not model scale. It also demonstrates intermediate filtering success as a usable reward signal for training an upstream planner.

Real-world applications:

• Browser agents that book flights, shop online, or fill forms on large commercial sites.
• Assistive tools that navigate complex pages for users with accessibility needs.
• Enterprise RPA-style automation that must run reliably in dynamic, JavaScript-heavy environments.
• Deployment of web agents on small, cheap, or on-device models where processing 10,000 to 100,000 tokens per step is infeasible.

Industry relevance: Because the approach avoids feeding full DOMs into the LLM, it cuts token cost and inference latency and reduces the barrier to using smaller fine-tuned models rather than proprietary frontier models.

Future Directions

• The paper's appendix promises a systematic discussion of limitations and future work, but the truncated content here does not report specific proposed directions.
• Extending the pruning program's template beyond keyword-and-weight parameters while preserving its controllability.
• Scaling the annotated dataset beyond approximately 5,000 interaction steps and the 1101-trajectory grounding benchmark.
• Testing whether the hierarchical reward mechanism transfers to other agent training setups, and applying programmatic pruning to domains beyond the web.

Target Audience

Researchers and engineers working on LLM agents, web automation, GUI grounding, and reinforcement fine-tuning, plus practitioners building browser agents who need to cut inference cost on long-context HTML. Beginners would need background in LLM prompting, supervised fine-tuning, and reinforcement learning to follow the training sections.

Authors’ abstract

Web automation employs intelligent agents to execute high-level tasks by mimicking human interactions with web interfaces. Despite the capabilities of recent Large Language Model (LLM)-based web agents, navigating complex, real-world webpages efficiently remains a significant hurdle due to the prohibitively large size of Document Object Model (DOM) structures, often ranging from 10,000 to 100,000 tokens. Existing strategies typically rely on crude DOM truncation -- risking the loss of critical information -- or employ inefficient heuristics and separate ranking models, failing to achieve an optimal balance between precision and scalability. To address these challenges, we introduce Prune4Web, a novel paradigm that shifts DOM processing from resource-intensive LLM reading to efficient programmatic pruning. Central to our approach is DOM Tree Pruning Programming, where an LLM generates executable Python scoring scripts to dynamically filter DOM elements based on semantic cues from decomposed sub-tasks. This mechanism eliminates the need for LLMs to ingest raw, massive DOMs, instead delegating traversal and scoring to lightweight, interpretable programs. This methodology achieves a 25x to 50x reduction in candidate elements for grounding, thereby facilitating precise action localization while mitigating attention dilution. Furthermore, we propose a specialized data annotation pipeline and a two-turn dialogue training strategy that jointly optimizes the Planner, Programmatic Filter, and Grounder within a unified framework. Extensive experiments demonstrate state-of-the-art performance. Notably, on our low-level grounding task, Prune4Web dramatically improves accuracy from 46.8% to 88.28%, underscoring its efficacy in real-world web automation.