What It Is, Not What It Says: Securing Agentic Browsing Interactions with Web Semantics

AI summary

Overview

Research area: Computer security and privacy — specifically the security of AI "agentic" browsers (LLMs that read webpages and click, type, and navigate on a user's behalf), with connections to browser architecture, provenance tracking, and prompt-injection defense.

Technical level: Advanced. The paper assumes familiarity with the DOM, the accessibility tree, same-origin policy and site isolation, the Chrome DevTools Protocol, and prompt injection.

Scope (one sentence): The paper proposes and evaluates Intact, an execution architecture plus a page representation called GenTree that lets an unmodified browser mediate what an LLM agent reads, decides, and does on real webpages, using provenance and web semantics rather than the model's judgment.

What This Paper Is About

Agentic browsers currently consume pages as screenshots or as flattened text, which strips away the browser's protections: which party authored each element, which origin it came from, and what that element is actually capable of doing. As a result, text injected by a third party can persuade the LLM to exercise the user's authority for content that has none — the agent becomes a "confused deputy."

The goal is not to detect prompt injection but to constrain its effects. The paper argues that security must live in the interface between the web and the agent, in code that page content cannot persuade, and it builds that interface around the browser's own semantics: what an element is and where it comes from, not what it says.

Key Contributions

1. GenTree, a bespoke per-element page representation that combines the DOM of every frame, the accessibility tree, and script-level provenance, labeling each element with a capability taxonomy. On the tested webpages it attributes 2.3× as many elements to third parties as the browser's standard attribution, which misattributes 58.7% of them to the first party.

2. Virtual isolation, a planning architecture implemented in Intact that runs on an unmodified browser: the page is passed to the LLM one element at a time, so no element's text reaches the decision on another element, and each step is executed only within the capabilities of the element it names.

3. An empirical evaluation against Vanilla, a conventional planner, on 1,091 real webpages with injected attacks: only 12.8% of attacks get through Intact's planner, each at most as a consent request, versus 73.1% for Vanilla; isolation disrupts functionality on only 3.7% of tasks.

4. A released artifact: source code and an attack benchmark of 1,091 real webpages at https://github.com/llm-platform-security/intact.

Main Findings

• The browser's default attribution is badly incomplete. Across 1,270 crawled pages, standard attribution identifies 10.0% of a page's elements as third-party (37.5% of those inside iframes). With instrumentation, GenTree identifies 22.7% as third-party (26.4% of those inside iframes) — 2.3× as many, meaning 58.7% of third-party elements are misattributed to the first party without instrumentation.

• Most third-party content is invisible to origin-based checks. Of the third-party elements found with instrumentation, 46.7% are attributed only because of instrumentation: 36.4% were created by third-party scripts, 10.3% are first-party elements whose attributes, text, or event listeners a third party changed, and 1.4% are third-party elements modified by a different third party.

• Isolation sharply reduces successful attacks. On 1,091 pages with injected attacks, 12.8% of attacks get through Intact's planner, and each gets through only as a request for user consent; the conventional planner lets through 73.1%.

• Security costs little functionality. Isolation disrupts functionality on 3.7% of tasks.

• Latency is a known cost, with a proposed remedy. The paper reports that a parallel schedule could remove most of the latency that isolation adds; specific latency measurements are not reported in the excerpt analyzed here.

• Prior defenses leave a narrow but real gap. The paper reports that in prior work (UCM), a single mislabeled element raised attack success rate from 0% to 6%, close to the 8% of a prompt-based defense, because those defenses still give one LLM a whole-page view.

• Third-party content is functionally necessary. The authors find that third-party content provides useful functionality such as links, forms, and text fields, which is why Intact plans over third-party elements too and asks the user for consent rather than simply blocking them.

• The web is structurally hostile. Third parties appear on 92% of pages, the median top-thousand site includes 66 of them, and about half of websites render content they did not load themselves; cross-site scripting has been among MITRE's two most dangerous software weaknesses every year since 2020, with SQL injection joining it at the top in 2025.

Methodology in Plain English

Capturing the page as the browser sees it. Before any page script runs, the authors inject instrumentation into every frame using the Chrome DevTools Protocol. This code wraps the DOM interfaces through which scripts create, insert, and modify elements, and attributes each call to the script responsible, identified from the JavaScript call stack. Scripts are identified by party, URL, and a content hash so that two scripts served from the same URL stay distinct; code from eval, Function, and inserted inline scripts gets a synthetic source URL that maps back to its creator; source URLs declared by page code are stripped so one script cannot impersonate another. Any write whose script cannot be identified is marked unattributed and treated as untrusted. Interfaces that build elements from HTML strings (innerHTML, document.write) are covered with a mutation observer read synchronously around the call. Cross-origin frames, which Chromium runs in separate processes, get their own session and the same instrumentation.

Labeling what each element can do. Each element is labeled along two axes derived from the HTML specification and ARIA: ten interaction types (e.g., click, type, upload) and thirteen effect types (e.g., navigate, send-request, dom-mutate). Labels are assigned statically by 76 rules grouped by element family, then adjusted by 14 composition rules accounting for ancestors, ARIA attributes, and computed style. Effects are also categorized as local (reversible), external (not recallable), or undetermined. Custom elements whose behavior is script-defined get no interactions unless an ARIA role declares one, favoring safety over functionality.

Planning one element at a time. The LLM is shown the user's request, the current element's own markup and direct text, a text-free log of operations decided so far, and a line saying whether the element is inside a form. It never sees provenance, party names, capability labels, or element identifiers; it answers whether the task needs this element and, if so, with which operation and value. The harness, not the LLM, attaches identifiers and executes each step only if the operation is among that element's labeled interactions, asking the user before acting on any third-party element. Because showing every element would take thousands of calls (IMDb's homepage alone has 3,355 elements with visible markup), the page is split in page order into chunks of about 1,500 characters; the LLM screens chunks in parallel, picks one, then walks it element by element.

Evaluation on the live web. Rather than synthetic benchmarks, the authors sample the top-50K Tranco sites into fifteen Curlie-derived topical categories, take 100 sites per category (Adult yields 65, Kids & Teens 21, for 1,386 websites), and crawl landing pages in September 2026 with live third-party content, letting each page settle for eight seconds, retrying once with a 20-second wait, and skipping pages that stall beyond five minutes. After 65 load failures, 45 bot blocks or challenges, and six stalls, 1,270 pages remain; 15,854 of 15,968 cross-origin frames were captured. Tasks are generated by an LLM (five realistic functionalities per page), and few-shot examples for the planner come from pages outside the evaluation set (Lowe's, iHerb, Home Depot, PriceRunner).

Why This Matters

Impact on research. The paper reframes agentic browsing security as an architecture problem rather than a model-alignment problem, and it supplies the missing substrate — per-element provenance and capability labels — that execution-isolation, control- and data-flow, and permission-management defenses all presuppose. Its finding that standard origin-based attribution misses 58.7% of third-party elements on real pages is a directly reusable measurement, and its 1,091-page attack benchmark addresses a genuine weakness in synthetic agent benchmarks.

Real-world applications:

• Hardening deployed agentic browsers (Comet, Claude in Chrome, Gemini in Chrome) and automation frameworks such as Browser Use, which today feed pages to LLMs as screenshots or text.
• Protecting authenticated sessions where an agent can spend money, as in the reported case where Comet filled in saved address and card details on a fake Walmart store and completed an Apple Watch purchase within seconds.
• Guarding against the documented real attacks: hidden instructions in Reddit posts and comments that leaked one-time login codes and credit card numbers, crafted links that exfiltrated email and calendar data, and injected page elements and pop-ups that diverted agents.
• Consent gating and audit trails for third-party widgets (payment forms, booking widgets) inside otherwise trusted first-party sites.

Industry relevance. The design runs on stock Chromium via Playwright and CDP, requires no browser-architecture changes and no cooperation from websites — important because the paper notes that sites adopt even established mechanisms such as Content Security Policy slowly and incompletely. Browser vendors adding agent features can adopt the representation and consent model incrementally.

Future Directions

• Move instrumentation into the browser. The current instrumentation lives in the page's JavaScript environment, so the threat model must assume page scripts do not evade or tamper with it; relocating it into browser internals would remove that assumption.

• Close the coverage gaps. The instrumentation does not yet cover cookies, postMessage exchanges, content injected by browser extensions, or shadow roots, and the authors state it can be extended to them.

• Address availability attacks. Chunk selection and the operation log sit outside virtual isolation, so an attacker can derail a plan with a decoy — affecting whether a task completes rather than what the agent may do. The paper flags this explicitly as unaddressed.

• Reduce latency and extend the representation. A parallel schedule is proposed to remove most of the added latency (quantified results are not reported in the excerpt analyzed here), and GenTree's per-element records are proposed as a basis for fine-grained access control encoding user preferences, and as a substrate for existing agent defenses.

Target Audience

Security and privacy researchers working on agentic systems and prompt injection; browser and platform engineers building or hardening agentic browsing features; designers of LLM agent frameworks who need to know why screenshots and DOM-to-text serialization fail as security boundaries; and graduate students in systems security interested in provenance tracking, capability-based enforcement, and evaluation on live rather than synthetic web corpora. Readers should be comfortable with browser internals and web security concepts, as the paper is written for a security-venue audience.

Authors’ abstract

AI agents increasingly browse the web on users' behalf, yet they consume webpages as screenshots or text, exposing their LLMs to unvetted third-party content. Browsers enforce semantics under which what an element can do depends on what it is and where it comes from, not on what it says. Agents flatten these semantics, so injected text can persuade the LLM to exercise the user's authority for content that has none. As LLMs remain easy to persuade, the agent's interface to the web must itself ensure the integrity of what the agent reads, interprets, and does. We propose Intact, which runs on an unmodified browser. Intact builds GenTree, a bespoke representation that captures provenance and web semantics. Intact passes the page via GenTree to the LLM one element at a time, so the LLM decides on each element without any other element's text. Its harness executes each step only within the capabilities of the step's element and asks the user before acting on third-party content. On 1,270 real webpages, GenTree attributes 2.3x as many elements to third parties as the browser, which misattributes 58.7% of them to the first party. On 1,091 of these webpages with injected attacks, only 12.8% of attacks get through Intact's planner, each at most as a consent request, against 73.1% for a conventional planner. Isolation disrupts functionality on only 3.7% of tasks, and a parallel schedule could remove most of its added latency.