MCP vs RAG vs NLWeb vs HTML: A Comparison of the Effectiveness and Efficiency of Different Agent Interfaces to the Web (Technical Report)

AI summary

Overview

Research area: LLM-based web agents, agent–website interaction protocols, and retrieval-augmented generation for e-commerce.

Technical level: Intermediate. The paper assumes familiarity with LLM agents, retrieval-augmented generation, and API protocols, but the comparison itself is presented through concrete metrics rather than deep architectural theory.

Scope: A controlled, head-to-head benchmark of four ways an LLM agent can interact with a website (HTML browsing, RAG, MCP, and NLWeb), evaluated on 91 multi-shop shopping tasks across four LLMs.

What This Paper Is About

Researchers are exploring several different ways for LLM agents to interact with websites: browsing HTML pages, querying a retrieval index (RAG), calling site APIs via the Model Context Protocol (MCP), or using the natural-language NLWeb endpoint. Each approach has advocates, but no prior study had compared all four under identical conditions. This paper builds a controlled testbed of four simulated e-shops and measures which interface makes agents most effective and most efficient at the same set of shopping tasks.

Key Contributions

1. A reusable testbed of four simulated e-shops, each exposing its products through three interfaces simultaneously (HTML, MCP, NLWeb), plus a RAG index over all crawled shop pages. For each of the four architectures, the authors implemented a specialized agent that performs identical tasks.

2. The first four-way comparison of agent–website interfaces under a single controlled environment, using identical task sets, an identical product catalog, and four underlying LLMs (GPT-4.1, GPT-5, GPT-5-mini, Claude Sonnet 4).

3. A systematic effectiveness and efficiency evaluation covering completion rate, F1, token consumption, API cost, and runtime, broken down by interface, model, and four task categories (specific search, vague search, cheapest-product search, transactional tasks).

4. A detailed error analysis of 729 manually annotated false positives and several hundred false negatives, categorizing failure modes such as constraint violations, near-miss variants, and retrieval coverage gaps.

Main Findings

• Interfaces matter more than expected. Averaged over all tasks and models, F1 rises from 0.67 for HTML to 0.75–0.77 for MCP, NLWeb, and RAG. HTML trails by roughly ten percentage points.

• Efficiency gains are larger than effectiveness gains. Token usage per task drops from about 241k (HTML) to 47k–140k, and runtime falls from 291 seconds to 50–62 seconds. RAG and NLWeb average around 50 seconds per task versus 281 seconds for HTML — roughly a 5× speedup.

• RAG is the strongest overall architecture. RAG achieves the highest average F1 (0.77) and the lowest token usage and cost. The best single configuration is RAG with GPT-5 (F1 0.87, completion rate 0.79).

• RAG with GPT-5-mini is the best price–performance compromise. It sits on the cost–quality frontier: nearly competitive F1 at roughly a tenth of the cost of GPT-5-based runs.

• NLWeb matches MCP in effectiveness at roughly half the tokens. NLWeb and MCP score almost identically in F1 (0.76 vs. 0.75), but NLWeb uses about 71k tokens per task versus 140k for MCP. The standardized schema.org responses appear to reduce interpretation overhead.

• Task difficulty reshapes the picture. On specific product search and transactional tasks, RAG, MCP, and NLWeb all exceed F1 0.90, while HTML lags by ~15 points. On vague and cheapest-product searches, all interfaces drop sharply (F1 0.59–0.68), and model capability matters more than interface choice.

• HTML is unexpectedly strong at transactions. HTML with GPT-4.1 achieves perfect F1 (1.00) on add-to-cart and checkout tasks, one of the few settings where the older model beats newer ones.

• Non-reasoning models offer a throughput advantage. GPT-4.1 and Claude Sonnet 4 generate no internal reasoning tokens, so they consume far fewer tokens and finish faster, at only a moderate effectiveness cost on well-specified tasks.

• Errors are mostly near misses. The dominant false-positive category (~25%) is "product fails requirements" — items that satisfy most constraints but violate one attribute (e.g. wrong memory size or a special edition instead of the standard variant). Agents also struggled with spatial reasoning ("compact" keyboards returned full-sized ones) and comparative expressions ("more than," "less than") read as equality checks.

• False-negative patterns differ by interface. RAG misses items because they were never retrieved (retrieval coverage limit), while NLWeb usually retrieves the right product but fails to select it.

Methodology in Plain English

The researchers built four simulated online shops populated with 4,421 real product offers drawn from the October 2024 Common Crawl, spanning PC components, peripherals, and consumer electronics. Each shop exposes its catalog through three channels: traditional HTML pages, a proprietary MCP server with search/cart/checkout functions, and an NLWeb "ask" endpoint that returns schema.org-formatted JSON. Separately, a RAG index was built by crawling and cleaning all shop pages, embedding the text, and storing it in Elasticsearch.

Four agents were then written, one per interface. The HTML agent observes a page's accessibility tree and clicks and fills forms. The RAG agent issues search queries against the unified index and calls Python functions for cart operations. The MCP agent calls each shop's API endpoints and must handle heterogeneous JSON schemas. The NLWeb agent issues natural-language queries and receives schema.org JSON.

All 91 tasks from the WebMall benchmark were run against every interface–model combination. Tasks were grouped into four categories: specific product search, vague product search, cheapest product search, and transactional workflows. Results were scored on completion rate (exact match), precision/recall/F1 (partial credit), token usage, API cost in dollars, and end-to-end runtime. Finally, the authors manually inspected and categorized the errors from RAG, MCP, and NLWeb runs with GPT-4.1 and Claude Sonnet 4.

Why This Matters

Impact on research: This is the first study to isolate the effect of the agent–website interface while holding tasks, catalog, and model constant. Earlier work compared only two architectures at a time (HTML vs. APIs, or browsing vs. RAG), often on the live web where results cannot be reproduced. This paper sets a methodological standard for how to benchmark agent interfaces and provides an open testbed that others can extend.

Real-world applications:

• Agentic shopping assistants that compare prices across retailers and complete checkout can use these findings to choose between crawling, indexing, or API integration.

• E-commerce platform design — retailers deciding whether to expose MCP servers, NLWeb endpoints, or leave their HTML as-is now have quantified evidence about the effectiveness and cost trade-offs for each option.

• Enterprise web agent deployment — organizations choosing between RAG pipelines and purpose-built APIs for internal tools get a concrete cost-per-task and latency comparison.

• Agent framework development — the error taxonomy (near-miss variants, constraint violations, comparative-expression failures) points to specific validation checks that framework builders can add.

Industry relevance: Efficiency findings are directly actionable. RAG and NLWeb reduce token consumption by roughly 3× and latency by 5× relative to HTML browsing, which translates into lower operating costs for any company running LLM agents at scale. The observation that GPT-5-mini with RAG nearly matches GPT-5 with RAG at a fraction of the cost gives a clear deployment recommendation for budget-conscious teams.

Future Directions

• Improving retrieval coverage in RAG. About half of all non-retrieved false negatives came from RAG agents that queried the index only once or twice per task. Better multi-query strategies could close this gap.

• Adding lightweight constraint validation. The dominant error type is a product that satisfies most requirements but violates one attribute. A cheap post-retrieval check on price or attribute thresholds could eliminate many of these.

• Extending beyond e-commerce. The testbed covers four shops in one domain. Testing whether the interface rankings hold for travel booking, real-estate search, or multi-domain agents remains open.

• Investigating model–interface interaction more deeply. GPT-5 dominated on some interfaces and underperformed GPT-4.1 on others (notably HTML transactions). Understanding why reasoning-enabled models vary so much across interfaces could inform better agent design.

Target Audience

Researchers and practitioners working on LLM-based web agents, agentic commerce, and retrieval-augmented generation will find the most value here, as will engineers building agent frameworks or evaluating MCP and NLWeb for production use. E-commerce platform architects deciding which programmatic interfaces to expose will benefit from the concrete cost and latency comparisons. Readers need working familiarity with LLM agents and RAG to get the most out of the metrics sections, but the comparative conclusions are accessible to anyone with a general interest in how AI systems interact with the web.

Authors’ abstract

Large language model agents are increasingly used to automate web tasks such as product search, offer comparison, and checkout. Current research explores different interfaces through which these agents interact with websites, including traditional HTML browsing, retrieval-augmented generation (RAG) over pre-crawled content, communication via Web APIs using the Model Context Protocol (MCP), and natural-language querying through the NLWeb interface. However, no prior work has compared these four architectures within a single controlled environment using identical tasks. To address this gap, we introduce a testbed consisting of four simulated e-shops, each offering its products via HTML, MCP, and NLWeb interfaces. For each interface (HTML, RAG, MCP, and NLWeb) we develop specialized agents that perform the same sets of tasks, ranging from simple product searches and price comparisons to complex queries for complementary or substitute products and checkout processes. We evaluate the agents using GPT 4.1, GPT 5, GPT 5 mini, and Claude Sonnet 4 as underlying LLM. Our evaluation shows that the RAG, MCP and NLWeb agents outperform HTML on both effectiveness and efficiency. Averaged over all tasks, F1 rises from 0.67 for HTML to between 0.75 and 0.77 for the other agents. Token usage falls from about 241k for HTML to between 47k and 140k per task. The runtime per task drops from 291 seconds to between 50 and 62 seconds. The best overall configuration is RAG with GPT 5 achieving an F1 score of 0.87 and a completion rate of 0.79. Also taking cost into consideration, RAG with GPT 5 mini offers a good compromise between API usage fees and performance. Our experiments show the choice of the interaction interface has a substantial impact on both the effectiveness and efficiency of LLM-based web agents.