AutoVend: Benchmarking LLM Agents on Long-Horizon Autonomous Vending Operations

AI summary

Overview

Research area: LLM agents applied to agent-mediated commerce, specifically online-to-offline (O2O) business operations; benchmark design and agent evaluation.

Technical level: Advanced. The paper assumes familiarity with LLM agent architectures, long-horizon task evaluation, and inventory-control concepts such as lead times, replenishment, and holding costs.

Scope: A single benchmark paper that builds a simulated vending-machine business and asks whether five LLM agents can run it profitably over 180 simulated days, compared against a rule-based inventory-control baseline.

What This Paper Is About

LLM agents are moving from shopping on a consumer's behalf to running a business on a seller's behalf. In O2O commerce this is risky: decisions are made through a Web interface, but they are carried out physically and expensively, so a bad online choice becomes a real cost. The paper's goal is to evaluate agents on whether their online decisions produce sustained economic returns, rather than on whether they complete individual transactions correctly. AutoVend is the benchmark built to ask that question.

Key Contributions

1. A long-horizon, multi-site operations benchmark. AutoVend places a single LLM agent in charge of 20 vending machines spanning five site types, operated through a Web-based interface over 180 simulated days.

2. A data-grounded simulator with realistic business friction. The environment models a shared warehouse, procurement lead times, paid replenishment, inventory expiration, and recurring operating costs — the mechanisms that make physical execution costly.

3. A baseline-anchored evaluation of five LLM agents. The agents are compared against a rule-based inventory-control baseline, with diagnosis of how they coordinate across sites, execute actions, and perform over time.

4. Cost-aware scoring and trajectory-level diagnosis. The paper ranks models by returns per dollar of inference cost in addition to raw returns, and analyzes agent trajectories to characterize where and why they fail.

Main Findings

• Most LLM agents lose to a simple rule-based policy. Four of the five evaluated LLM agents fall short of the rule-based inventory-control baseline on the benchmark.

• Rankings change once inference cost is counted. Ordering models by returns per dollar of inference cost produces a different ranking than ordering them by returns alone — so "which agent is best" depends on whether cost is included.

• Local metric gains do not compound into economic growth. Improvements on individual operational metrics do not consistently translate into stronger economic growth, because inventory costs and execution costs erode those gains.

• Failure modes are operational, not conversational. Trajectory analysis identifies three recurring difficulties: grounding feasible actions (acting on what is actually possible in the environment), managing inventory commitments, and converting feedback into effective policy revisions.

Methodology in Plain English

The authors built a simulated vending business rather than using real machines, which makes it possible to run 180 days of operations repeatedly and cheaply while still charging the agent for the things that cost money in reality: buying stock, waiting for deliveries, paying to restock, losing stock to expiry, and covering ongoing operating costs. The agent interacts only through a Web-style operating interface, so its decisions have to be issued as actions the platform would accept and are then carried out in the simulation. Five LLM agents run the business, and their results are compared to a hand-written inventory-control rule. The researchers then look past the final score: they examine performance over time, how the agent coordinates across the 20 machines and five site types, and the step-by-step trajectories to see what went wrong. The abstract does not report the specific economic figures, per-model scores, or ablation results.

Why This Matters

Impact on research. Most agent benchmarks score single tasks or single transactions. AutoVend argues that for business-operation agents the right unit of evaluation is sustained economic return under realistic frictions, and that inference cost belongs in the score. Its finding that four of five LLM agents trail a rule-based baseline is a useful counterweight to task-completion benchmarks where agents appear highly capable.

Real-world applications:

• Unattended retail and vending operations, where restocking decisions must balance spoilage against stockouts.
• O2O order fulfillment, where online commitments must be matched to physical capacity and lead times.
• Automated procurement and replenishment planning for small-format retail networks.
• Seller-side agent tooling on commerce platforms, where agents propose actions a human operator approves.

Industry relevance. As platforms shift from assisting buyers to automating sellers, the economics of running an agent matter as much as its competence. The paper's cost-adjusted ranking and its finding that metric improvements get eaten by inventory and execution costs speak directly to whether deploying an agent actually pays for itself.

Future Directions

• Improving action grounding. The abstract flags grounding feasible actions as a core failure; follow-up work could target agents that reliably map intentions onto actions the platform will accept and the simulation can execute.
• Better inventory commitment and feedback use. Understanding how an agent should commit to stock and revise policy from delayed, noisy feedback is left open, and is the natural next research problem.
• Treating inference cost as a design objective. Since cost-adjusted rankings differ from raw returns, future agents may need to be built and tuned for return per unit of inference spend, not just return.
• Generalizing beyond vending. Whether these findings hold for other O2O business types, other scales of machine fleet, and multi-agent or human-in-the-loop setups is not addressed in the abstract and remains an open question.

Target Audience

Researchers building and evaluating LLM agents for long-horizon, tool-using tasks; benchmark designers interested in economic and cost-aware metrics; and practitioners in retail operations, O2O commerce, and inventory management who want to know how far current agents are from replacing or assisting rule-based operational policies.

Authors’ abstract

Large language model (LLM) agents are extending agent-mediated commerce on the Web from shopping for consumers to operating businesses for sellers. In online-to-offline (O2O) commerce, decisions are issued through Web-based platforms but realized through costly physical execution. Evaluating such agents requires going beyond individual transactions to ask whether decisions made online turn into sustained economic returns. We introduce AutoVend, a benchmark in which one LLM…