MAFA: A Multi-Agent Framework for Enterprise-Scale Annotation with Configurable Task Adaptation

AI summary

Overview

Research area: Applied machine learning — multi-agent LLM systems for enterprise-scale data annotation, with a focus on conversational AI in financial services.

Technical level: Intermediate. Readers need basic familiarity with large language models, retrieval/embedding methods, and standard classification metrics (Top-k accuracy, F1, MRR), but the paper is written in an applied, deployment-oriented style rather than a theoretical one.

Scope: The paper describes a production multi-agent annotation framework deployed at JP Morgan Chase, its configurable architecture, and its measured accuracy, efficiency, and business results across intent classification and FAQ matching tasks in multiple languages and domains.

What This Paper Is About

Large enterprises generate far more customer messages than human annotators can label, creating backlogs that stall model improvement. Single-LLM annotation is faster but inconsistent and error-prone in domains like banking, where mistakes have compliance and customer-experience consequences. MAFA addresses this by coordinating multiple specialized LLM agents — a query planner, four parallel ranker agents, and a judge agent — behind a configuration file that lets teams switch annotation tasks without changing code.

Key Contributions

1. A production-deployed multi-agent annotation system that processes millions of utterances in hours while reaching 86% agreement with human annotators, compared to 72% inter-annotator agreement in the prior manual workflow.
2. A configuration-driven adaptation scheme (the AnnotationConfig class) that redefines the annotation type, matching fields, and prompt behavior through settings rather than code, letting one system serve FAQ mapping, intent classification, entity extraction, and other tasks.
3. Documented enterprise impact: elimination of a 1,067,400-utterance backlog, roughly 5,187 hours of annual annotation labor saved, and a 26.4x reduction in per-query processing time.
4. Empirical validation across four datasets (Banking77, an internal 500,000-query banking set, CLINIC-150, and Banking FAQ) plus cross-lingual evaluation on LCQMC (Chinese) and FiQA (financial), with ablation and statistical significance testing.

Main Findings

• Consistent accuracy gains over single-agent baselines: Top-1 accuracy improved by 10.5% on Banking77 (0.768 to 0.873), 13.8% on the internal banking dataset (0.699 to 0.837), and 6.3% on CLINIC-150 (0.838 to 0.901). Reported gains also include 15.1% Top-5 accuracy and 16.9% F1 improvement on the internal dataset.

• The judge agent matters most: Ablation on the internal dataset shows removing the judge agent costs 5.7% Top-1 accuracy — the largest single-component drop. Embedding-based agents contribute 4.2%, few-shot example diversity 2.8%, and query planning 2.5%.

• Confidence routing makes human review tractable: Annotations split roughly 85% HIGH, 10% MEDIUM, and 5% LOW. HIGH results auto-accept, MEDIUM auto-accept with flagging for audit, and LOW go to a human priority queue, concentrating scarce human attention on genuinely ambiguous cases.

• Strong FAQ retrieval performance: On the Banking FAQ dataset, MAFA reached 0.355 Top-1 accuracy versus 0.120 for BM25 and 0.270 for the best single-agent configuration, with MRR improving 41% over BM25 (0.790 vs 0.382).

• Cross-domain and cross-lingual generalization: MAFA outperformed the best single agent on the Chinese LCQMC dataset (0.694 vs 0.631 Top-1) and on FiQA (0.612 vs 0.545 Top-1), showing the framework is not specific to English banking intents.

• Structured prompting reduces hallucination: JSON-based structured reasoning prompts lowered hallucination rates from 8.3% to 5.2% compared with free-form chain-of-thought, with the largest benefit on ambiguous inputs.

• Graceful degradation under failure: When GPT-4 availability dropped during peak load, the system continued operating with three agents at 79% accuracy rather than failing outright.

• Error modes are identifiable: Misclassification analysis attributes errors to out-of-domain queries (42%), extreme brevity such as single-word inputs (31%), and utterances legitimately spanning multiple intents (27%).

Methodology in Plain English

The pipeline runs in three stages. First, a query planning agent (GPT-4o) rewrites short or informal customer messages into richer forms — for example, expanding "cash back" into related banking concepts — while deliberately leaving ambiguous numeric strings untouched to avoid inventing meaning. Expanded queries are cached for 24 hours, cutting API calls by 35%.

Second, four ranker agents run simultaneously, each using a different retrieval strategy. Two rely on structured prompting alone, matching against just the primary field (FAQ question, intent name) or against full context including secondary fields like FAQ answers. Two others use dense vector embeddings from OpenAI's text-embedding-3-large with adaptive dimensionality reduction. Each agent gets its own set of 8 to 15 unique few-shot examples, which deliberately differ across agents to create ensemble diversity.

Third, a judge agent aggregates all candidate annotations, scores, and reasoning traces into a final ranking using a weighted voting scheme where agent weights update daily based on rolling accuracy. The judge also produces explanation trails for audit purposes. If the judge times out at 200ms, the system falls back to simple score aggregation.

Execution uses a 50-worker thread pool to run agents concurrently, reducing latency from 2,800ms sequential to 650ms parallel. Non-urgent workloads go through OpenAI's Batch API at half cost. The whole system runs on Kubernetes and was validated through ablation studies, paired t-tests with Bonferroni correction across 10 runs, load testing, and three months of live production data.

Why This Matters

Impact on research. The paper is a rare full account of moving multi-agent LLM orchestration from benchmark to regulated production. It provides concrete numbers on where each architectural component earns its cost, and it introduces a configuration abstraction that separates annotation task definition from system logic — a design pattern largely absent from academic annotation literature, which tends to optimize for a single fixed task.

Real-world applications:

• Banking and financial services, where intent routing, FAQ automation, and compliance-sensitive categorization must handle millions of monthly customer messages.
• Customer support platforms across industries that need to classify or tag high-volume text streams without proportional growth in labeling headcount.
• Search and knowledge-base teams building FAQ or question-matching systems, where MAFA's FAQ results substantially exceed BM25 and embedding-only retrieval.
• Multilingual content operations, since the framework was validated on both English and Chinese datasets and adapts through configuration.

Industry relevance. The paper targets the gap between LLM capability demos and systems that satisfy uptime, audit, privacy, and ROI requirements. Its cost figures ($0.003 per query, an 85% reduction versus single-agent calls), latency budgets, auto-scaling policies, PII masking, and 90-day audit retention directly address what enterprise procurement and compliance teams ask about. The framing that 85% accuracy you can trust beats 95% accuracy you cannot is a useful counterweight to benchmark-chasing in applied AI.

Future Directions

• Active learning integration: Using the system's own low-confidence predictions to select which examples get labeled and added to training pools, closing the loop between annotation output and model improvement.
• Multi-intent utterances: Roughly 27% of errors come from queries that genuinely span multiple intents, and the current single-label framing cannot represent them. Extending to multi-label annotation is a natural next step.
• Automated configuration discovery: Learning optimal annotation configurations — fields, prompts, agent weights — from data rather than requiring engineers or analysts to specify them manually.
• Generalization beyond banking and beyond classification: The authors show cross-domain results for FAQ matching, but entity extraction and more complex structured annotation tasks remain largely untested in the reported evaluation.

Target Audience

ML engineers and applied scientists building production NLP systems, particularly those working on annotation pipelines, intent classification, or retrieval. It is also well suited to technical product and platform leaders in regulated industries evaluating whether multi-agent LLM architectures can replace or augment human labeling teams, and to researchers studying LLM ensembles and multi-agent consensus who want evidence from a real deployment rather than a benchmark.

Authors’ abstract

We present MAFA (Multi-Agent Framework for Annotation), a production-deployed system that transforms enterprise-scale annotation workflows through configurable multi-agent collaboration. Addressing the critical challenge of annotation backlogs in financial services, where millions of customer utterances require accurate categorization, MAFA combines specialized agents with structured reasoning and a judge-based consensus mechanism. Our framework uniquely supports dynamic task adaptation, allowing organizations to define custom annotation types (FAQs, intents, entities, or domain-specific categories) through configuration rather than code changes. Deployed at JP Morgan Chase, MAFA has eliminated a 1 million utterance backlog while achieving, on average, 86% agreement with human annotators, annually saving over 5,000 hours of manual annotation work. The system processes utterances with annotation confidence classifications, which are typically 85% high, 10% medium, and 5% low across all datasets we tested. This enables human annotators to focus exclusively on ambiguous and low-coverage cases. We demonstrate MAFA's effectiveness across multiple datasets and languages, showing consistent improvements over traditional and single-agent annotation baselines: 13.8% higher Top-1 accuracy, 15.1% improvement in Top-5 accuracy, and 16.9% better F1 in our internal intent classification dataset and similar gains on public benchmarks. This work bridges the gap between theoretical multi-agent systems and practical enterprise deployment, providing a blueprint for organizations facing similar annotation challenges.