Browser Agents and Web Navigation

GPT-4V was turned loose on live websites in 2024. It completed 51.1 percent of its tasks. It reached that figure only when its textual plans were grounded into actions by hand. Planning was not the bottleneck. Converting a plan into the right element on the page was. The experiment is called SeeAct.

The technique everyone reaches for did not rescue it. Set-of-mark prompting is the standard way to ground a model in a screenshot, and for web agents it turned out not to be effective. The best grounding strategy the authors could build used the HTML structure together with the visuals, not the picture alone. That is the failure this lesson is about. An agent reads the page, decides correctly, and still acts on the wrong element. It is a measured failure, not a story about a form that moved.

Perception, grounding, interaction, verification. A browser agent repeats that cycle on every page, and any of the four can be the one that broke. Perception is what the page looks like to the agent. Grounding is deciding which element on the page the words refer to.

SeeAct is the experiment that pulls those two apart. It holds perception fixed — the same GPT-4V, the same live websites — and changes only the grounding step. That is how 51.1 percent of tasks got completed. The paper is blunt about which half stays open: “However, grounding still remains a major challenge. Existing LMM grounding strategies like set-of-mark prompting turns out to be not effective for web agents, and the best grounding strategy we develop in this paper leverages both the HTML structure and visuals.”

That is why the cycle is drawn as five owners and not one box marked “the agent”. A failure at perception and a failure at grounding produce the same screenshot of a wrong click. Verification belongs to one owner and trust control to another, with tests to match.

DOM and API interaction, accessibility-tree control, and screenshot coordinates read the same page at three different levels of meaning. The DOM names elements. The accessibility tree names roles. A screenshot names nothing, so a moved button is a different click. The more meaning the interface carries, the less often a cosmetic change breaks the run.

The third row is usually argued about with adjectives. SeeAct settles it with a result. Set-of-mark prompting, the standard screenshot-grounding technique, was not effective for web agents. The best grounding strategy in that paper leveraged both the HTML structure and the visuals. So screenshot coordinates are not the option that trades a little accuracy for universal coverage. In the published comparison they are the option that did not work on its own. The winner kept the page's structure alongside the picture.

None of the three tells the agent which words on the page are content and which are an instruction aimed at it. That question has to be settled elsewhere. The next section is where.

Browser agents observe rendered pages, accessibility trees, DOM structures, network responses, or screenshots. Each of those views exposes different semantics. Each can change independently as the site evolves.

Web content is both data and potentially adversarial instruction. That is not a house convention; it is the thesis of a peer-reviewed paper. A 2023 paper named indirect prompt injection and taxonomised it: “We argue that LLM-Integrated Applications blur the line between data and instructions. We reveal new attack vectors, using Indirect Prompt Injection, that enable adversaries to remotely (without a direct interface) exploit LLM-integrated applications by strategically injecting prompts into data likely to be retrieved.” Its authors demonstrated working attacks against Bing's GPT-4-powered Chat and against code-completion engines. Named victims, in 2023, not a projection.

The runtime must therefore distinguish user intent, trusted policy, site controls, and untrusted page text. Everything the page says belongs in the last of those, including text that reads like a system message.

Layout moves constantly. So start by measuring whether navigation succeeds across minor layout variation: rerun yesterday's task against today's page and see how much of it still works. Sensitive submissions need a check of their own. A run can navigate perfectly and still send the wrong thing to the wrong place. Look at where the form posts, and what it carries, before it is submitted.

Then put the agent in front of a page written to mislead it, and give that test a number rather than a verdict. Anthropic did that before announcing its Claude for Chrome pilot on 25 August 2025: “We conducted extensive adversarial prompt injection testing, evaluating 123 test cases representing 29 different attack scenarios.” Unmitigated, the result was “Browser use without our safety mitigations showed a 23.6% attack success rate when deliberately targeted by malicious actors.” With the mitigations in place the figure fell to 11.2 percent in autonomous mode. Reduced, not removed. On a four-type browser-specific challenge set the same mitigations took the success rate from 35.7 percent to 0 percent. That is what a defence looks like when the attack space is narrow enough to enumerate.

Last, when an interaction fails in the same place twice, ask whether the execution trace holds enough state for someone to reproduce that failure without guessing.

Before a browser agent needs a better model it needs an action protocol: what it may click, what it must confirm, what it may never submit. Write one for a single workflow. The protocol exists because a page can contain instructions designed to hijack the agent, and the only defence that holds is a list the page itself cannot edit.

The last step, testing against hostile pages, is the one teams skip. Inventing hostile pages sounds like a project. It is not one any more. AgentDojo was built for exactly this question: “We populate the environment with 97 realistic tasks (e.g., managing an email client, navigating an e-banking website, or making travel bookings), 629 security test cases, and various attack and defense paradigms from the literature.” NIST's taxonomy lists it among the public AML benchmarks. 97 tasks and 629 security test cases is the size of the exam your protocol has to sit. One of those tasks is an agent navigating an e-banking website, which is the workflow most teams would rather not think about.

Then test the protocol by running the workflow again after the site has moved things around, and see whether it still describes what the agent is actually doing. Write those three lists down before the next site redesign, not after it.

Use direct APIs when they provide a safer and more stable interface. Browser automation is valuable when no structured integration exists, not because clicking looks more human.

The figures in this lesson are the argument. WebArena's best GPT-4-based agent finished 14.41 percent of 812 tasks end to end, where human annotators finished 78.24 percent. Scored by people rather than by a grader, on Online-Mind2Web's 300 tasks across 136 live websites, Operator reached 61 percent and most other agents sat near 30 percent. Grounding, the step between deciding and clicking, is still the open problem SeeAct described. And after adversarial testing across 123 cases and 29 scenarios, 11.2 percent of injection attempts still succeeded in autonomous mode.

When you do run an agent through a browser, treat each site redesign as a reason to reread the action protocol. The pages it visits now are not the pages it was tested against. Two questions never go away. Can it still find its way around a page that has shifted? Can it tell the page's content apart from an order aimed at it?