Can LLM Annotators Match Human Consistency? Benchmarking Human Agreement and Optimizing Multi-Agent with Contrastive Knowledge

AI summary

Overview

Research area: Natural Language Processing — LLM-based data annotation, annotation benchmarks, and multi-agent LLM systems.

Technical level: Intermediate (readers should be comfortable with inter-annotator agreement metrics such as Fleiss' κ, prompt optimization, and retrieval-augmented prompting).

Scope: The paper introduces ANT-A, a 7-dataset multi-annotator benchmark with detailed guidelines and 5 human labels per sample, and ANT-OR, a multi-agent annotation framework that distills contrastive decision rules from annotation errors.

What This Paper Is About

Existing benchmarks for LLM annotators are unrealistic: they omit the long, task-specific guideline documents real annotators use, and they collapse each sample into a single gold label even when humans legitimately disagree on boundary cases. This makes it impossible to tell whether an LLM annotator reaches human-level consistency or merely happens to match one reference answer. The authors build a benchmark that preserves that disagreement, and a multi-agent framework that adapts its prompts to complex guidelines by learning from its own mistakes.

Key Contributions

1. Problem diagnosis: The authors identify two bottlenecks for reliable automated annotation — benchmarks that fail to capture inter-annotator disagreement and guideline complexity, and annotation agents limited by single-agent or static multi-agent designs.
2. ANT-A benchmark: A released multi-annotator benchmark of 7 datasets spanning medical, e-commerce, and general domains across text and vision-language modalities, each with 1,500 samples, detailed guideline documents, and 5 independent human annotations per sample, enabling distribution-aware evaluation via Fleiss' κ.
3. ANT-OR framework: An adaptive multi-agent annotation framework that converts collective prediction successes and failures into reusable structured decision knowledge (a contrastive sample pool plus a "discriminative dimension" pool), with agent-level error attribution and prompt optimization.
4. Empirical validation: ANT-OR outperforms the strongest baseline by 4.5 points in agreement@≥2 accuracy and achieves the closest match to human annotator distributions among compared methods, within 2 points of human Fleiss' κ.

Main Findings

• Consistent gains across all seven datasets: ANT-OR achieves the best overall performance on all 7 datasets, beating the strongest baseline by 4.5 points in average agreement@≥2 accuracy. Its averages are 74.74 accuracy and 61.64 Fleiss' κ, against a human baseline of 88.94 and 63.60.
• Close to human consistency on individual tasks: ANT-OR reaches κ = 56.61 on Image Attribution Recognition and κ = 83.82 on Billing Scenario Classification, compared with human inter-annotator agreement of 58.59 and 83.50 respectively on those datasets.
• Disagreement is real but bounded: Across datasets, samples where all five annotators gave completely different labels are extremely rare (nearly 1%), so most instances retain partial human consensus — but agreement varies widely, from κ = 38.77 (Disease Privacy Assessment) to κ = 83.50 (Billing Scenario Classification).
• Multi-agent beats single-agent as a foundation: Multi-agent systems consistently outperform the basic single-agent CoT setup across all 7 datasets, with gains from 0.2 to 2.9 points.
• Strongest on the hardest cases: When instances are grouped by annotator agreement, all methods degrade as agreement drops, but on the most ambiguous subset (≤ 2/5 agreement) ANT-OR outperforms the multi-agent baseline by nearly 8 points, while SPO shows only limited gains.
• Better generalization than SPO: SPO reaches higher training accuracy (98.5% vs. 96.5% on BillSC), but ANT-OR performs better on test sets by an average of 3.7 percentage points, suggesting SPO overfits.
• Discriminative dimensions drive the gains: Removing dimension mining drops average accuracy from 71.75 to 68.05 and κ from 61.37 to 60.22 on the four text datasets; removing contrastive triples gives 69.47 / 60.74; removing coupled injection gives 69.53 / 61.04.
• Robust to injected sample count: Varying injected samples from 1 to 5 changes average accuracy only slightly (70.67, 71.75, 71.22, 71.62), and quality-aware similarity retrieval beats random selection (71.45 vs. 70.80).
• Two positive samples suffice: In contrastive mining, moving from 1 to 2 positive samples gives clear gains (average 62.80 → 64.25), while a third adds little (64.45), confirming the 2:1 positive-to-negative ratio.
• Higher cost, higher accuracy: ANT-OR averages $101.43 in optimization cost versus $65.94 for SPO, but delivers 4.7 points more accuracy (71.75% vs. 67.08%) on the four text tasks.
• Architecture matters: Without prompt optimization, the annotator–verifier design reaches 67.50 average accuracy at 24.68 cost, versus 59.77 at 74.12 for debate and 64.80 at 34.83 for planner–worker; planner–worker even underperforms single-agent baselines (-1.1% on MedEE, -3.7% on DisPA, -0.4% on BillSC).

Methodology in Plain English

Building the benchmark. For each of the 7 datasets the authors gathered 1,500 domain-relevant queries from public internet data, anonymized identifiable information, and filtered duplicates and low-quality items, keeping a mix of clear-cut and boundary cases. They wrote task-specific guideline documents from real-world requirements and sharpened them with explicit decision rules for edge cases.

Collecting human labels. A pool of 10 professional annotators was used, with each sample independently labeled by 5 of them. Annotation ran in three phases: a pilot where all 10 annotators labeled roughly 50 samples per dataset to surface ambiguities; a rule-clarification phase where annotator questions were answered and the guidelines revised; and full batch annotation. Total annotation cost was USD $4078. Labelers could not see each other's work, making agreement analysis meaningful.

Evaluating. Two metrics are used. agreement@≥2 counts a prediction as correct if it matches a label given by at least two of the five humans (human performance is computed leave-one-annotator-out). Fleiss' κ treats the model as an additional annotator alongside the humans to measure distributional alignment; Medical Entity Extraction uses entity-level F1 instead because κ does not apply to span-level annotation.

How ANT-OR works. Three annotator agents write independent labels and rationales, and a verifier adjudicates conflicts. When an agent is wrong (its label fails to match at least two human labels), the system builds a contrastive triple: two semantically similar correct cases plus the incorrect one, and asks an LLM to summarize the "discriminative dimension" — a reusable rule about how to tell the confusable categories apart. Redundant rules are merged with LLM-based refinement, and a rule is kept only if its support count is at least 2. At inference, the top-K retrieved samples (scored by embedding similarity times a learned quality weight, updated by whether past retrievals led to correct predictions) and their associated rules are injected together into the prompt. An error-attribution model identifies which agents caused a failure and why, and only those agents' prompts are rewritten. Updated prompts are kept only if batch-level performance improves, otherwise they are rolled back, and the knowledge base grows each epoch.

Setup. DeepSeek-V3.1-Terminus is the backbone for text tasks and Qwen3-235B-A22B-VL-Instruct for vision-language tasks; both also act as optimizers. Retrieval uses Qwen3-Embedding-8B and Qwen3-VL-Embedding-8B. Training uses 200 samples per task, batch size 20, 3 iterations per batch over 6 epochs, with the best epoch chosen on training performance; evaluation is on 1,000 test samples, leaving 300 unused. Temperature is 0.1.

Why This Matters

The paper reframes LLM annotation quality as a distributional problem rather than a single-label accuracy problem — which is how annotation actually works when guidelines are nuanced. It gives researchers a benchmark where guideline-following and disagreement are studied together rather than in isolation, and it shows that lightweight, training-free prompt adaptation over a structured knowledge base can close much of the gap to human consistency.

Real-world applications:

• Medical record annotation and privacy-risk assessment, where guidelines are long and stakes are high.
• E-commerce transaction and merchant categorization, such as the 31-class billing scenario task.
• Content moderation and social text classification, where boundary cases split reasonable reviewers.
• Multimodal catalog and advertising work — image classification, image attribution recognition, and image–text relevance checking.

Industry relevance: Annotation vendors and platform teams can use the benchmark to audit whether an LLM annotator matches the spread of human judgments, not just a reference label, and can weigh the reported cost figures ($101.43 average optimization cost for ANT-OR vs. $65.94 for SPO) against the 4.7-point accuracy gain. The framework requires no model training, which matters for teams that cannot fine-tune.

Future Directions

• Broader benchmark coverage: extending ANT-A to long-document annotation, open-ended judgment, and specialized expert workflows, which the authors name as a limitation of the current 7 datasets.
• Cheaper optimization: reducing the computational overhead introduced by multi-agent collaboration and iterative optimization while preserving the accuracy gains.
• Backbone scaling and transfer: studying how ANT-OR behaves with stronger models and how well it transfers across model families, domains, and annotation settings.
• When to inject retrieved examples: the ablation shows coupled injection helps most when category distinctions are subtle but can introduce noise on rule-based tasks, leaving open how to decide adaptively when instance-level evidence is worth retrieving.

Target Audience

Researchers and engineers working on LLM-as-annotator systems, prompt optimization, and multi-agent LLM architectures; dataset and benchmark builders interested in inter-annotator agreement and distribution-aware evaluation; and applied teams in medical, e-commerce, or content-moderation settings who need to judge whether automated annotation can stand in for a panel of human labelers.

Authors’ abstract

LLM-based annotation has advanced rapidly, yet existing benchmarks remain misaligned with real-world practice. Actual annotation often involves lengthy, task-specific guidelines, and different annotators may produce divergent but reasonable judgments, especially on boundary cases, where single-labeled dataset fail to reflect this disagreement. This gap also limits existing single-agent and workflow-based annotation systems, which often lack adaptability to complex, evolving guidelines. To bridge this gap, we introduce ANT-A Benchmark, a real-world multi-annotator benchmark spanning 7 datasets across domains and modalities, with task-specific guidelines and 5 independent human labels per instance. ANT-A enables a distribution-aware evaluation, using Fleiss kappa to assess how closely model annotations align with human labeling distributions. We further propose ANT-OR, a multi-agent annotation framework that refines prompts by distilling contrastive decision rules from positive and negative feedback. Experiments show that ANT-OR improves agreement@>=2 accuracy by 4.5 points over the strongest baseline and achieves the highest alignment with human annotator distributions as measured by Fleiss kappa within 2 points. Code and benchmark are available at https://github.com/Krueske/ANT-OR.