No-Human in the Loop: Agentic Evaluation at Scale for Recommendation

AI summary

Overview

Research area: LLM-as-a-judge evaluation, agentic multi-agent systems, and recommender-system evaluation (specifically Complementary-Item Recommendation, CIR).

Technical level: Intermediate. Readers need familiarity with LLM prompting, majority-voting aggregation, and common recommender evaluation metrics, but the framework itself is described at a conceptual level.

Scope: The paper introduces ScalingEval, a multi-agent, human-annotation-free benchmarking framework that compares 36 LLM configurations as judges over 1,745 anchor–recommendation product pairs across seven e-commerce categories from the Walmart platform.

What This Paper Is About

As LLMs are increasingly used to judge the quality of other AI systems, it is unclear which models can be trusted as evaluators, how their judgments differ across families, and how expensive reliable evaluation is. The authors build a pipeline in which LLM agents audit complementary product recommendations using predefined patterns and issue codes, then synthesize "ground truth" through majority voting across many models instead of human annotators. The goal is a reproducible benchmark and protocol for LLM-as-a-judge evaluation at e-commerce scale.

Key Contributions

1. ScalingEval framework: A multi-agent orchestration system that decomposes complementary-item evaluation into a CI Pattern Audit agent, a Recommendation Issue Audit agent, and a Report Generation agent, with strict conflict-resolution rules.
2. Scalable majority-vote ground truth synthesis: A protocol that collects independent judgments from many models into a structured matrix, resolves disagreements by majority vote, and falls back to a conservative prioritization policy (Reject > Major > Minor > Good) when disagreement persists.
3. Large-scale benchmark: A systematic comparison of 36 LLM configurations spanning GPT, Gemini, Claude, Llama, and GPT-OSS families across seven product categories, evaluated on accuracy, confidence, coverage, Cohen's Kappa, issue match, agreement, latency, and cost per pair.
4. Category-sensitive findings: Documented evidence that structured domains (Electronics, Sports & Outdoors) yield strong cross-model consensus while lifestyle categories (Clothing & Shoes, Food & Beverages) remain challenging, alongside a demo that surfaces three candidate new issue codes (COMPAT, FUNC-MIS, CTX-MIS).

Main Findings

• Highest decision confidence: Anthropic Claude-3.5-sonnet leads on confidence on pairs with definitive judgments, at 98.4–99.2% across temperature settings; the abstract rounds this to approximately 99% in decision-making confidence.
• Best overall performer: Gemini-1.5-pro achieves the best overall accuracy and coverage, with 76.63% overall accuracy and 87.14% overall coverage in Table 1, and it leads in 4 out of 5 product categories.
• Best latency–accuracy–cost trade-off: GPT-4o provides the most favorable balance in the closed-source group, listed at 1x latency and 1x cost per pair with 60.13% overall accuracy and 64.63% overall coverage.
• Strongest open-source model: GPT-OSS-20B leads open-source models with 42.32% overall accuracy and 52.60% overall coverage, reaching mid-tier closed-source performance at lower cost (listed as 5.9x latency and 0.0x cost per pair).
• Category-level accuracy leader: GPT-o1 excels in Clothes & Shoes (82.35% accuracy, 89.91% coverage in Table 1) and shows 4.9x latency and 6.0x cost per pair.
• Confidence tiers: Beyond Claude-3.5-sonnet, the paper reports clear tiers: Llama-3-8B-Instruct (95.9–97.5%), GPT-4o (93.1–95.1%), o1 models (94.3–95.5%), Gemini models (87.9–88.6%), and Llama-3-3B-Instruct (86.3–93.3%).
• Temperature robustness: Several families (Gemini, Claude-3.5-sonnet, o1, o3) show minimal sensitivity to temperature.
• Agreement varies by domain: High-agreement categories include Sports & Outdoors (93.8%) and Electronics (91.3%), while challenging categories include Clothing & Shoes (84.2%) and Food & Beverages (85.4%).
• Distinct judgment profiles: Sports & Outdoors and Pet Supplies show right-skewed agreement distributions with most pairs above 0.8; Electronics concentrates above 0.7; Clothing & Shoes and Food & Beverages show more spread-out, higher-variability distributions.
• Consensus mitigates isolated errors: In case studies, individual disagreements (e.g., open-source models flagging possible gender-targeting in a scrub set/scrub jacket pair, or GPT-OSS-20B treating mayonnaise as a condiment) were resolved by majority voting.

Methodology in Plain English

The authors treat evaluation as a report-generation task driven by a user query such as "generate a CI report." The pipeline works in stages:

1. Pattern audit. A state-of-the-art LLM proposes a fixed set of complementary-item patterns (Accessory/Add-On, Replenishment/Consumable, Functional Synergy, Aesthetic/Style Match, Bundled Solution, Brand Synergy, Occasion/Use-Case, and Other). Each anchor–recommendation pair is mapped to one or more of these patterns.
2. Issue audit. A second prompt uses predefined issue codes (accessory/refill for a different product, embarrassing or sensitive content, category too distant, too similar to anchor, wrong age/gender targeting, wrong format, wrong size/dimensions, and Other) to flag why a pair might fail.
3. Report generation. Passed pairs are split into chunks of size k. Each chunk is summarized as a set of counts and labels: total pairs (t), complementary count (c), non-complementary count (n), CI patterns (P), flagged issues (I), conflicted pairs (q), and an agreement score (a). Aggregation returns totals T, C, N, Q along with merged pattern, issue, and agreement sets (P, I, A), under the constraint that C + N = T.
4. Majority-vote ground truth. Multiple LLMs (GPT, Gemini, LLaMA) independently run the entire process. Their judgments form a matrix with one row per anchor–recommendation pair and one column per agent. Majority voting resolves conflicts, and if disagreement persists, the stricter judgment wins. The most complete and consistent candidate report is selected as the Consensus Report.
5. Benchmarking. The consensus labels are used to score every model on accuracy, confidence, coverage, Cohen's Kappa, issue match, and agreement, plus latency and cost per pair.

Why This Matters

Impact on research. The paper provides a reproducible protocol for LLM-as-a-judge evaluation that removes human annotation from the loop, and it grounds judge-model benchmarking in a real recommender-system task rather than a synthetic one. It also contributes empirical evidence about how judge reliability, agreement, and cost trade off across model families and product domains.

Real-world applications:

• E-commerce recommendation QA: Auditing complementary-item recommendation pairs at catalog scale (phone case with smartphone, printer with ink cartridge) without a human labeling queue.
• Catalog and merchandising governance: Detecting problematic pairings such as wrong-size recommendations, substitute items, or embarrassing/sensitive adjacency before they reach shoppers.
• Model selection for evaluation infrastructure: Choosing judge models by cost, latency, and accuracy fit — for example, GPT-4o for a balanced budget or GPT-OSS-20B when open-source deployment is required.
• Domain-sensitive evaluation design: Allocating more scrutiny or different protocols to lifestyle categories (Clothing, Food) where models disagree most.

Industry relevance. The work originates from Walmart Global Tech and uses Walmart-scale e-commerce behavioral data across Electronics, Sports & Outdoors, Pet Supplies, Home & Garden, Toys & Games, Food & Beverages, and Clothing & Shoes. It speaks directly to organizations that want cheaper, scalable evaluation pipelines for recommender systems while keeping known judge biases in check through multi-model aggregation.

Future Directions

1. Extending the framework to LLM lifecycle evaluation broadly. The authors position ScalingEval as a template for studying emergent abilities, domain robustness, and cross-model reliability beyond complementary-item recommendation.
2. Improving evaluation in lifestyle categories. Clothing & Shoes (84.2% agreement) and Food & Beverages (85.4%) remain persistently harder than structured domains, raising the question of what domain-specific patterns or issue codes would close the gap.
3. Adopting the suggested new issue codes. The demo report proposes COMPAT (compatibility issue), FUNC-MIS (functional mismatch), and CTX-MIS (contextual suitability), alongside refined annotation guidelines to reduce residual conflicts.
4. Validating consensus labels against human judgment. The paper's core premise is that majority voting can substitute for human annotation; the paper does not report a head-to-head comparison between consensus labels and human annotators, which remains an open question.

Target Audience

Researchers and engineers working on LLM-as-a-judge evaluation, agentic AI pipelines, and recommender-system quality assessment. It is also relevant to applied machine learning teams in e-commerce and marketplace companies who need scalable, low-cost evaluation workflows, and to practitioners comparing closed-source and open-source judge models on accuracy, latency, and cost.

Authors’ abstract

Evaluating large language models (LLMs) as judges is increasingly critical for building scalable and trustworthy evaluation pipelines. We present ScalingEval, a large-scale benchmarking study that systematically compares 36 LLMs, including GPT, Gemini, Claude, and Llama, across multiple product categories using a consensus-driven evaluation protocol. Our multi-agent framework aggregates pattern audits and issue codes into ground-truth labels via scalable majority voting, enabling reproducible comparison of LLM evaluators without human annotation. Applied to large-scale complementary-item recommendation, the benchmark reports four key findings: (i) Anthropic Claude 3.5 Sonnet achieves the highest decision confidence; (ii) Gemini 1.5 Pro offers the best overall performance across categories; (iii) GPT-4o provides the most favorable latency-accuracy-cost tradeoff; and (iv) GPT-OSS 20B leads among open-source models. Category-level analysis shows strong consensus in structured domains (Electronics, Sports) but persistent disagreement in lifestyle categories (Clothing, Food). These results establish ScalingEval as a reproducible benchmark and evaluation protocol for LLMs as judges, with actionable guidance on scaling, reliability, and model family tradeoffs.