CARTBench: Rubric-Based Evaluation of Shopping Advice for Complex and Ambiguous Requests with Timelines

AI summary

Overview

Research area: Natural Language Processing — evaluation of LLM-based shopping assistants and recommendation agents.

Technical level: Intermediate. Readers should be comfortable with LLM benchmarking, rubric-based judging, and pass-rate metrics, but the paper's central idea (case-specific checklists guided by shared principles) is explained in plain terms.

Scope (one sentence): The paper builds CARTBench, a benchmark of 169 real commercial shopping cases with timestamped dialogue, 1,603 cached product records, and 1,328 expert-written criteria, and uses it to score 10 models on how well their shopping advice handles complex, ambiguous, and time-sensitive requests.

What This Paper Is About

LLM shopping assistants now give product advice in multi-turn conversations, but existing benchmarks do not report timeline information (so they cannot tell whether advice follows a need that has already expired) and give no credit to replies that keep unconfirmed preferences open rather than guessing. Building a fair test is hard for two reasons the authors name explicitly: data scarcity (real commercial conversations are confidential and costly to annotate) and difficulty standardizing criteria (the right checks differ per product and per user need, so one shared checklist cannot work). The goal is a benchmark that scores shopping advice case by case while holding a common standard across cases.

Key Contributions

1. CARTBench, an evaluation benchmark built from real commercial shopping needs: 169 stratified-sampled cases across 17 retail domains and eight intents, each pairing timestamped dialogue with cached product evidence (1,603 records total), plus 1,328 frozen, expert-annotated criteria split into Core, Excellence, and Safety tiers.
2. Ideal Response Principles, four principles distilled by experts from excellent replies manually selected from successfully completed orders. They keep case-specific rubrics consistent across cases and also drive Principle-Guided CART Prompting, the structured recommendation strategy given to models (the rubrics themselves stay hidden from them).
3. Evidence-Grounded Rubric Judging, a judging protocol that returns criterion labels, supporting quotations, and reasons, followed by independent blinded expert review; in the first full-suite evaluation, 13,239 of 13,280 judge labels (99.69%) agreed with expert judgments.
4. Identification of an understanding-to-execution gap, plus a failure-mode diagnosis showing that the same error types — unsupported product claims, unjustified suitability inferences, and premature resolution of unknowns — recur across every evaluated model.

Main Findings

• High Core, low Excellence: 3 of 10 models exceed 90% on Core, while none exceeds 45% on Excellence. In Table 2 the Core scores above 90% belong to DeepSeek-V4-Pro (91.0), Qwen3.8-Max (95.4), and GPT-6.1-Sol (95.9); the highest Excellence score in the table is Claude-Opus-5 at 39.8.
• The understanding-to-execution gap: meeting the minimum required behaviors does not produce high-quality purchase advice, and the gap persists even though every model receives the explicit Ideal Response Principles.
• Goal alignment is easier than the rest: goal alignment exceeds requirement adherence for all 10 models, and exceeds uncertainty preservation for 9 models, in the first full-suite evaluation. No model passes every required check.
• Constraints are the weak spot for Excellence: constraint-focused cases have the lowest Excellence rate for 7 of 10 models, because they require reconciling interacting constraints or dependent choices.
• Three shared failure modes appear in every model: F7 Product Evidence Overreach (adding intrinsic product claims the supplied evidence does not support), F8 Product Fit Leap (converting stated attributes into unjustified user-specific suitability guarantees), and F2 Shopping Uncertainty Collapse (treating unknown preferences, possessions, purchase needs, or scenario conditions as settled).
• Task switching exposes state-tracking errors: F1 context-state drift is minor overall but is the most frequent diagnosed mode in task-switch cases, and the continuity–F1 association stays positive after adjusting for overlapping mechanisms and rubric opportunities.
• Decisive unknowns break evidence and fit more than uncertainty handling: in decisive-unknown scenarios, F7 and F8 occur more often than F2, so preserving unknowns also requires bounding product and suitability claims.
• The "Shopping Feasibility Mirage": among replies that pass all resolved goal, requirement, uncertainty, and evidence checks, 13.0% still fail a plan check (95% interval 8.1–18.4, models weighted equally).
• Open-weight models trail: the three Qwen3.5 configurations (35B-A3B, 122B-A10B, 397B-A17B) score 68.9, 71.3, and 80.3 on Core, versus a 95.9 high among commercial APIs.
• Cost is low: at official list prices, one complete 169-case evaluation costs less than US$24 per API model for inference and judging.

Methodology in Plain English

The researchers started from real shopping requests rather than synthetic ones. Experts screened commercial conversations for the hard ingredients the benchmark targets — requirements that interact, needs that change, goals that are unclear, and facts that are decisive but unknown — and used stratified sampling to pick 169 representative cases spanning 17 retail domains.

Each case is a bundle: the dialogue history with timestamps, the current request, the time of the current message, and a cached set of product records from an internal product search platform so every model sees identical evidence. Keeping relative dates anchored to their original message times is what lets graders tell an active need from an expired one.

To make grading consistent without a single generic checklist, experts wrote four short Ideal Response Principles (address the current decision; follow active needs and preserve unknowns; ground claims and explain trade-offs; give clear, useful decision help), derived from excellent replies to orders that were actually completed. They then turned those principles into per-case criteria: 1,328 in total, each checking one observable behavior and allowing supported alternatives. Every criterion sits in exactly one tier — Core (required behavior), Excellence (advice quality beyond the minimum), or Safety (case-specific risk).

Models get the dialogue, the evidence, the timestamp, and the principles, but never the rubric and never browsing. A judge model (DeepSeek-V4-Pro for the first full-suite evaluation) assigns each criterion a pass, fail, unresolved, or not-applicable label with a quotation and reason; an expert then independently reviews those labels blind before scoring. Scores use pass rate = P / (P + F) and resolved coverage = (P + F) / (P + F + U), with not-applicable labels excluded and each response–criterion instance weighted once. Three full-suite runs give the reported means and sample standard deviations; the diagnostic analysis uses the first run and its expert-confirmed labels.

Why This Matters

Impact on research: The paper argues that shopping benchmarks cannot currently distinguish a lucky guess from a justified conditional answer, because they report neither timelines nor credit for preserved unknowns. CARTBench supplies both, plus a template — shared principles constraining case-specific rubrics — for building evaluations where the right checks vary per item but the standard does not. The reported 99.69% judge–expert agreement also offers a concrete data point on whether LLM judges can be trusted under expert review.

Real-world applications:

• Shopping assistants and conversational commerce: the failure taxonomy (unsupported product claims, unjustified suitability leaps, premature resolution of unknowns) gives teams concrete patterns to target during prompt design and post-training.
• Model selection for retail platforms: with a full evaluation costing under US$24 per API model, the benchmark is cheap enough to use as a routine procurement or regression check.
• Agent evaluation generally: the principle-guided, case-specific rubric design transfers to any domain where a single shared checklist is too coarse — travel planning, customer support, or personal finance advice.
• Gift and occasion shopping: cases such as an expired gift deadline show why timestamp-aware evaluation matters when advice can silently follow a need that no longer exists.

Industry relevance: All seven commercial API models tested are named flagship systems (MiMo-V2.6-Pro, DeepSeek-V4-Pro, MiniMax-M3, Kimi-K3, Claude-Opus-5, Qwen3.8-Max, GPT-6.1-Sol), and the open-weight comparison uses three Qwen3.5 sizes at 35B-A3B, 122B-A10B, and 397B-A17B. That mix speaks directly to deployment decisions: strong Core scores mean a model will usually satisfy baseline obligations, but Excellence scores under 45% for every model mean none of them reliably delivers the comparisons, prioritization, and conditional alternatives that actually drive a purchase decision.

Future Directions

• Cover specialized domains: the authors note that gifting and travel clothing are covered but that needs such as historically accurate costumes for museum reenactments or equipment for extended polar expeditions require further study.
• Extend past the recommendation: the paper evaluates only the current reply, excluding upstream elicitation and purchase outcomes; how advice should adapt across longer purchase, return, and replacement cycles, and how feedback from actual product use introduces new requirements, remain open.
• Close the understanding-to-execution gap: since no model exceeds 45% on Excellence despite receiving the Ideal Response Principles, a natural next question is what training or prompting changes would move constraint-reconciliation and decision-help behavior, rather than just Core compliance.
• Explain the Feasibility Mirage: 13.0% of replies that pass all resolved need and evidence checks still fail a plan check, so identifying what plan-validity signal those replies lack is an unresolved diagnostic question.

Target Audience

Researchers and engineers working on LLM evaluation, conversational recommendation, and shopping agents; benchmark designers who need case-specific rubrics without losing cross-case consistency; product and ML teams at e-commerce or social-commerce platforms choosing among commercial APIs or open-weight models; and anyone studying LLM-as-judge reliability, who will find the 99.69% expert-agreement figure and the blinded review protocol directly useful.

Authors’ abstract

Large language model (LLM) shopping assistants now recommend products in multi-turn conversations. However, reliable evaluation faces Data scarcity and Difficulty Standardizing Criteria. To address these obstacles, we introduce CARTBench for CART (Complex and Ambiguous Requests with Timelines). For Data scarcity, we curate cases from real commercial shopping needs across 17 retail domains, pairing timestamped dialogue with 1,603 product records. For Difficulty Standardizing…