E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation

AI summary

Overview

• Research area: Evaluation of large language model (LLM) agents on long-horizon, continuing (non-episodic) tasks, specifically autonomous e-commerce business operation.
• Technical level: Advanced. The paper assumes familiarity with agent benchmarks, reinforcement-learning task framing (episodic vs. continuing), bargaining theory, and LLM agent scaffolding such as context management and persistent memory.
• Scope: The paper introduces E-Commerce Bench, an open-source 365-day merchant simulation with fully deterministic customer and supplier behavior, and reports how 18 frontier and open-weight models perform on it across seven evaluation dimensions.

What This Paper Is About

Existing long-horizon agent benchmarks split into episodic tasks, which end when a stated goal is met and therefore say nothing about sustained competence, and continuing business simulations, which either use mock data or let an LLM play the counterpart, making runs stochastic and persuadable. E-Commerce Bench addresses this by putting an LLM agent in charge of a simulated Chinese online marketplace for a year, where it must research demand, negotiate with suppliers, price and ship goods, handle returns, and manage cash flow to maximize year-end total assets. The goal is an evaluation where any difference in outcome is attributable to the agent's policy rather than to chance.

Key Contributions

1. A realistic, open-source business environment. E-Commerce Bench is described as the first open-source benchmark to combine deterministic counterpart negotiations, dynamic events, and concurrent multi-store management over a year-long horizon, built on data derived from a real e-commerce platform (Taobao & Tmall), spanning 6,886 products and 576 suppliers. Table 1 positions it as the only listed benchmark holding real data, negotiation, multi-agent, adversarial, reproducible, and open-source properties simultaneously.
2. Determinism on both sides of the market. Customer purchases and returns follow a fixed demand model, while a deterministic negotiation kernel fixes every pricing, concession, and accept/walk-away decision; an LLM only renders those decisions as dialogue. The kernel adapts the negotiation-partner design of TERMS-Bench to repeated, cross-supplier bargaining.
3. Multi-dimensional evaluation of 18 models. Rather than ranking on year-end assets alone, the benchmark independently scores negotiation quality, fraud avoidance, cash flow and solvency, operational efficiency, operational execution, and long-horizon learning, producing capability profiles rather than a single leaderboard number.
4. Released code. The benchmark code is available at https://github.com/QwenLM/E-CommerceBench.

Main Findings

• No single model dominates. Across the seven dimensions, the best earner is neither the best negotiator nor the most efficient operator, and the radar plots of the strongest model per vendor family come out visibly uneven.
• Highest year-end assets: GPT-5.6 Sol. It grows the ¥100,000 opening stake into ¥1,431,425, described as roughly a 14-fold increase, yet it ranks 16th of 18 on fraud avoidance and earns less from a tool call than Fable5 does on operational efficiency.
• Strongest open-weight entry: Qwen3.8-Max-Preview. It reaches ¥416,252, which the paper reports as 38% above GLM 5.2 (high) and 4.2 times the initial balance, and shows the strongest learning over the horizon by progressively bargaining down prices across repeated orders.
• Bargaining and fraud leadership come from a mid-pack earner. Claude Opus 4.7 leads both bargaining and fraud avoidance while sitting mid-pack on profit.
• Most vendor families are uneven. Of the seven models drawn in the capability radar, one per vendor family, six fall below the eighteen-model median on at least one axis.
• Bankruptcy is common enough to matter. Four of the eighteen models slip into bankruptcy on part of their runs; bankrupt episodes are pooled into mean assets in the headline figure.
• Long-horizon experiential learning is weak. Across 8,647 repeat purchases of the same item from the same honest supplier, 16 models show no clear sign of pushing the purchase price down as the year progresses.
• Context limits are a first-class stressor. Across the 90 evaluation episodes, 1,495 context evictions ran. Turn counts and token drag diverge sharply: Claude Opus 4.7 finishes the year in 432 turns and loses about four windows' worth of transcript, while Gemini 3.5 Flash takes 2,628 turns and loses nearly nineteen windows' worth; Kimi K2.6 takes 1,017 turns and loses the same three and a half windows as Claude Opus 4.7 at 432 turns because it drags in 431 tokens per turn against Opus 4.7's 1,058; GPT-5.5 loses 28 windows in half of Gemini 3.5 Flash's turns at 2,759 tokens per turn.
• Evaluation scale. Each model is run for five 365-day episodes; the whiskers in the headline figure span one 5-run standard deviation.
• Section 4 detail is unavailable in the provided content. The supplied text is truncated within Section 3, so the paper's full per-model result tables and its error analysis of recurring failure modes are not available here.

Methodology in Plain English

The agent receives a merchant account, ¥100,000, and the 2026 calendar year on a simulated Chinese online marketplace, and its objective is simply to end the year with as much money as possible.

The benchmark is built in four layers. An agent loop layer handles turn-based control, context management, and persistent memory. A tool layer provides 18 e-commerce tools. An environment layer contains the sales-and-economy engine and a two-layer negotiation engine. Beneath everything is a data layer derived from real platform logs.

Actions cost simulated minutes out of a 600-minute day running from 08:00 to 18:00. A balance check costs 10 minutes, opening a store costs 60, and a supplier message costs 30 minutes regardless of how many suppliers receive it, so twenty messages fill a working day. Past 18:00 the clock jumps to 08:00 the next morning and runs a fixed settlement sequence that charges operating and storage costs, converts demand into orders, cancels shipments past their two-day deadline, settles matured escrow, delivers incoming purchase orders, and updates reputation. Episodes cap at 4,000 turns and end early after three consecutive turns without a tool call. Because a year of calls far exceeds the window, a single tokenizer enforces a shared 128,000-token budget: when the count reaches 120,000 tokens, whole tool-call groups are evicted oldest-first until 60,000 tokens are released, sparing the system message, the first user turn, and the two newest groups. A separate memory store of at most 20 entries lives in the environment and is never evicted, and the agent curates it itself.

Commercially, the agent opens up to four stores at once from 12 store types, paying ¥500 to open and ¥60 to ¥130 a day to run each. Cash moves through three accounts that are deliberately staggered: the bank account pays costs immediately, but a sale first becomes a pending shipment, then escrow with platform commission already deducted, then matures into the platform wallet after nine days, and only an explicit withdrawal returns it to the bank. Ten consecutive days closing with a negative bank balance end the episode as bankrupt.

Demand is computed, not sampled. Each SKU-day starts from a category base and is adjusted by six multipliers: price response, weekend uplift, promotion joined, month seasonality, calendar events, and store reputation. Two capacity terms then trim the result, one for the category and one for the store, so overstocking a narrow category cannibalizes rather than grows demand. In one worked example, an accessory SKU on a promotion Saturday sees a compounded expectation of 128.2 units cut to the 9 actually sold, with the category term alone accounting for a factor of 7.5. Four price-elasticity families are used per category, one of which is quadratic and symmetric about the public reference price, so discounting below reference can lose demand in those categories.

Fulfillment carries consequences: unshipped orders past two days are cancelled and hurt reputation, and return probability depends on the category's natural rate, defective supplier stock, how far above reference the agent priced, and the chosen shipping speed. A defective unit carries a return rate of at least 0.40 and at least double the natural rate, capped at 0.95; pricing at 1.3 times reference multiplies the return rate by 1.50, and slow shipping by a further 1.30. Returns are weighted 0.6 and cancellations 1.0 against reputation, and a store pinned at the reputation floor receives 15% of the demand it would get at the top.

Negotiation is where the design is most distinctive. Each supplier-SKU pair is handled by a deterministic kernel that sets reservation price, opening quote, concessions, and accept-or-walk decisions, seeded so that an identical agent trajectory always meets an identical supplier response. Acceptance and walk-away are stochastic, and counter-offers respond to the agent's own concession speed rather than the clock, so an agent that concedes fast meets a firmer counterpart. A second LLM renders the kernel's committed decision as chat, and an accept is refused unless the rendered price matches the kernel's standing quote within ±0.005, which keeps eloquence from overriding economics. Of the 576 suppliers, 152 are fraudulent, performing five scripted scam overlays split into pre-deal scams, whose reservation price is inflated toward 1.5 times the honest cost floor, and post-deal scams, which deliver only 60% to 70% of the quantity paid for, or defective stock. Honest suppliers draw from six behavior templates, and neither the template nor the honest/fraudulent label is visible to the agent. Because the same pair carries the same persistent type and reservation price across reopenings, repeatedly sourcing a SKU rewards anchoring each new session at or below the best price previously won.

Finally, results are scored on six capability axes alongside the primary resource metric, with negotiation measured as the fraction of the bargaining range captured and fraud measured as money actually lost rather than judged intent.

Why This Matters

Impact on research. The paper argues that benchmark credibility in continuing agentic tasks is a reproducibility problem as much as a task-design problem. By making both the demand side and the supplier side deterministic while leaving only the dialogue renderer sampled, it isolates the agent policy as the source of outcome variation. That design choice extends a rule-based negotiation-partner idea from single-session bargaining to repeated, cross-supplier, year-long bargaining, and it offers a template for other continuing-domain benchmarks that want adversarial behavior without giving up reproducibility. The finding that 16 of 18 models fail to push prices down on 8,647 repeat purchases is a concrete, measurable statement about long-horizon experiential learning that single-episode benchmarks cannot produce.

Real-world applications:

• Agent capability auditing for commerce platforms. The seven-axis profile format lets platform operators see where a deployed agent is weak, for example strong revenue but poor solvency or fraud avoidance, rather than reading a single profit number.
• Supplier negotiation and procurement assistants. The repeated cross-supplier bargaining setup mirrors real sourcing, where a buyer must infer a hidden floor, remember a historical best, and arbitrate between competing suppliers.
• Fraud and counterparty risk screening. Roughly a quarter of the roster is adversarial, with deception visible only in multi-round concession

Authors’ abstract

Long-horizon agentic tasks go beyond chaining short tasks over more interaction turns. Their evolving dynamic environments and long-range dependencies require Large Language Models (LLMs) to continually explore, learn from experience, and adapt their policies over thousands of steps. We introduce E-Commerce Bench, the first open-source benchmark that integrates multi-round counterpart negotiation and dynamic events into a year-long business operation. Over a 365-day year, an LLM agent concurrently runs multiple online stores, researching the market, negotiating with suppliers to source inventory, optimizing sales strategies, fulfilling orders, handling returns, and managing cash flow to maximize its end-of-year total assets. To construct a realistic merchant-side operating environment, the product and supplier data are derived from a real e-commerce platform, while a year-long calendar of promotions, natural disasters, and supply-chain shocks continually reshapes demand. For reproducibility, both sides of the market are deterministic: customer purchases and returns follow a fixed demand model, while a negotiation kernel determines supplier pricing, concessions, and decisions, with an LLM used only to verbalize them. We evaluate 18 frontier models across seven dimensions, including year-end assets, and find that no single model dominates. GPT-5.6 Sol earns the most, growing the 100,000 opening stake into 1,431,425, yet it ranks 16th of 18 on fraud avoidance and trails Fable5 in operational efficiency. Among open-weight models, Qwen3.8-Max-Preview leads with 416,252, 38% above GLM 5.2 (high), and achieves the strongest learning over the horizon, progressively bargaining down prices across repeated orders. Our code is available at https://github.com/QwenLM/E-CommerceBench.