Agent-in-the-Loop: A Data Flywheel for Continuous Improvement in LLM-based Customer Support

AI summary

Overview

Research area: Applied LLM systems — retrieval-augmented generation (RAG), human-in-the-loop preference learning, and continuous model retraining for enterprise customer support.

Technical level: Intermediate. Readers will get the most from this paper with basic familiarity with RAG, supervised fine-tuning, and preference optimization (RLHF/ORPO); the paper itself spends little space on technical derivations and most of its space on workflow design and measured outcomes.

Scope: A single production pilot at Airbnb (arXiv:2510.06674v2, cs.AI) that embeds four types of human annotation into live US-based customer support work and feeds them back into periodic retraining of retrieval, ranking, and generation models.

What This Paper Is About

Customer support knowledge changes constantly — products, policies, and customer preferences drift — but LLM-based support systems are typically retrained from static, batch-annotated datasets, which is slow and quickly goes stale. The authors build an "Agent-in-the-Loop" (AITL) framework in which support agents annotate preferences, adoption rationales, knowledge relevance, and missing knowledge while handling real customer conversations, and those signals are automatically filtered and folded back into model retraining. The goal is to compress the model update cycle from months to weeks while improving retrieval accuracy, response quality, and agent adoption of model suggestions.

Key Contributions

1. An online annotation interface embedded in live operations. The system captures four distinct signal types during real conversations: (1) pairwise response preferences, (2) agent adoption decisions and free-text rationales, (3) knowledge relevance checks on retrieved resources, and (4) identification of missing knowledge.
2. A continuous learning pipeline that closes the loop. Collected annotations are aggregated, filtered with rule-based thresholds and an LLM-based virtual judge, and used to retrain retrieval, ranking, and generation models via generalized LLM offline workflow (GLOW) modules, reducing update cycles from months to weeks (the prior offline pipeline took three months).
3. A production pilot with measured gains. A deployment with 40 US-based support agents on an asynchronous messaging channel, collecting annotations from more than 5,000 customer support cases, produced +11.7% recall@75, +14.8% precision@8, +8.4% helpfulness, and +4.5% agent adoption over baseline.
4. Two ablations that shape practical deployment advice. An annotation-timing study (immediate vs. delayed annotation over approximately 2,000 cases) and a virtual-judge (VJ) filtering study quantify when human-in-the-loop labeling actually pays off.

Main Findings

• Agent adoption and retrieval improve. Retrieval recall@75 rose from 0.634 (baseline) to 0.708 with AITL fine-tuning — an increase of 11.7% — versus 0.670 for offline fine-tuning, surpassing offline by 3.8%. Precision@8 rose from 0.357 to 0.410 (+14.8%) versus 0.394 for offline, exceeding offline by 4.1%.
• Generation quality improves with ORPO. Helpfulness rose from 0.658 to 0.713 (+8.4%), exceeding offline fine-tuning (0.691). Citation accuracy improved from 0.097 to 0.134 (+38.1%), surpassing offline (0.112). Response correctness increased from 0.851 to 0.882 (+3.6%), higher than offline (0.868).
• Humans prefer the fine-tuned model. In pairwise human evaluations, 60.12% of responses came from the AITL fine-tuned model versus 33.32% from the baseline, with 6.57% expressing no preference. Overall agent adoption rate increased by 4.5% versus baseline.
• Online annotations are more consistent with real agent behavior than offline ones. Agreement scores rose from 0.635 to 0.832 for preference judgment, from 0.721 to 0.775 for adoption judgment, and from 0.436 to 0.923 for knowledge relevancy. Step 4 (missing knowledge) was not evaluated offline because the annotation tool could not annotate all potential missing knowledge.
• Only missing-knowledge annotation benefits from being immediate. Immediate annotation raised Step 4 agreement from 63.9% to 76.5% (+12 percentage points, p < 0.05), while differences for Steps 1–3 were negligible. Median annotation times were 1.30 min (Step 1), 0.95 min (Step 2), 1.87 min (Step 3), and 1.63 min (Step 4).
• The LLM-based virtual judge acts as a noise gate. With filtering, 14.3% of examples are removed before retriever/ranker training and 34.5% before generator training on the same AITL batches. Gains concentrated in Recall@75 (0.708 vs. 0.670, +3.8 pp) and Citation (0.131 vs. 0.112, +1.9 pp absolute), while Precision@8 (0.402 vs. 0.394), Helpfulness (0.703 vs. 0.696), and Response Correctness (0.880 vs. 0.880) were statistically unchanged.
• Adoption filtering balances helpfulness against citation. Training on "plus" preference data (better or significantly better) raised helpfulness to 0.766 but lowered citation to 0.109 versus a baseline of 0.694/0.123; adding an agent-adoption filter produced 0.718 helpfulness and 0.137 citation. The paper's prose states the adoption filter restores citation correctness to 11.4% and retains a 3.5% helpfulness gain, which does not match the values listed in Table 6.
• Mixing old and new data beats training on new data alone. Retraining on a mix of historical and new annotations from the previous checkpoint increased precision@8 by 8% on historical data and 4% on recent data compared with training only on new data. Similar benefits were observed with offline annotation approaches.
• Gains generalize across model families and scales. Replicating AITL on Qwen2.5-32B and the Llama-3 family using the same split and evaluator: Llama-3.2-3B helpfulness went from 0.3731 to 0.6362 and citation from 0.0569 to 0.0606; Llama-3.1-8B from 0.3787 to 0.6056 and 0.0967 to 0.1136; Llama-3.3-70B from 0.6322 to 0.6438 and 0.1048 to 0.1224; Qwen2.5-32B-Instruct from 0.6718 to 0.6830 with citation unchanged at 0.1040. At 70B, helpfulness effects were mixed while citation improved or stayed stable.
• Annotating did not hurt productivity. Each agent annotated approximately 11 cases daily alongside regular support tasks, maintaining productivity comparable to agents not participating in annotation.
• Human and LLM verifiers agree strongly. The two evaluation sources correlate at r > 0.90, supporting a hybrid reliability score; the reference helpfulness evaluator (a Mistral-7B-based model fine-tuned on human preference data) and the GPT-4 prompt-based evaluation (seven prompts producing a 0–7 score) each reach roughly 80% agreement with expert ground truth.

Methodology in Plain English

The team first built a Unified Knowledge Base, consolidating customer guides, FAQs, internal policies, workflows, dynamic context such as reservation status, and historical cases into a centralized content management system with rich metadata.

They then embedded an annotation interface into the agents' live workflow. While serving customers, agents see two candidate responses (possibly from different models) and record how much better one is (significantly better, better, or slightly better), whether they actually adopted a response and why in free text, how relevant the retrieved knowledge was, and what knowledge was missing.

A review layer follows: a human expert and an LLM-based verifier independently check whether the agent's annotations match what actually happened in the conversation, flagging preference mismatches, incorrect knowledge relevance, adoption discrepancies, and omitted knowledge. Because human and LLM judgments correlate at r > 0.90, the two are averaged into a hybrid score.

Annotations then flow into a continuous learning pipeline. A rule-based filter keeps only annotations meeting significant-preference thresholds, and a model-driven filter uses an LLM-based virtual judge to remove low prompt-adherence examples. Retrieval, ranking, and generation models are periodically retrained on Ray clusters with parameter-efficient fine-tuning (LoRA/QLoRA) and model partitioning, then evaluated with both ground-truth-based metrics and virtual judges. Improvements in model performance serve as a proxy for annotation quality, reinforcing the cycle.

The baseline system uses an 8×7B Mistral Mixture-of-Experts generation model fine-tuned with SFT and then ORPO; a Zeta-Alpha-E5-Mistral 7b embedding model producing 1024-dimensional vectors for retrieval; and an in-house fine-tuned FLAN-T5 model for ranking. Metric choices were empirically driven: recall@75 because recall plateaued beyond 75 candidates against a corpus of ten thousand documents, and precision@8 because reranking outperformed retrieval alone from topN=5 onward and larger contexts diluted helpfulness.

Why This Matters

Impact on research. The paper argues that automated data flywheels such as Arena Learning, which rely on AI-simulated self-play and offline labels, do not directly address human preference drift or domain-specific knowledge decay. AITL shows that real-time human feedback can be collected inside operational workflows, not just in dedicated annotation sessions, and that it can supervise retrieval and ranking modules in addition to generation — the key structural difference from prior flywheel work, which targets generation only. The finding that immediate annotation only helps for missing-knowledge labeling is a concrete, testable constraint on human-in-the-loop design.

Real-world applications:

• Customer support platforms that need to keep answers aligned with frequently changing policies and product features.
• Any RAG deployment where knowledge gaps are discovered in conversation rather than in documents — the missing-knowledge channel acts as a live gap detector.
• Annotation and labeling tooling for teams that cannot afford separate offline annotation cycles and must fold labeling into existing staff workflows.
• Model refresh operations that need a defensible cadence; AITL provides a monitored, automated retraining path with a virtual-judge quality gate.

Industry relevance. The work comes from Airbnb and is evaluated on a production channel under real SLAs, with cost and productivity constraints stated explicitly (agents annotate roughly 11 cases a day without measurable productivity loss). The cross-model study on Qwen2.5-32B and Llama-3 3B/8B/70B also indicates the data flywheel is portable rather than tied to one architecture, which matters for organizations choosing models pragmatically.

Future Directions

• Scaling optional agent feedback. The authors propose replacing heavy labels with lightweight micro-annotations that default to "skip," using active sampling for high-uncertainty or disagreement cases, and correcting selection bias with inverse-propensity weighting and post-stratification.
• Product-embedded AITL. Integrating annotation into agent-facing tools and evaluating with a productivity bundle such as CSAT, time-to-resolution, adoption rate, and human-edit distance, while studying cognitive load, trust calibration, and skill formation for novice versus expert agents.
• Toward fuller automation. Using simulation and judge-based validation to automate dataset curation and preference labeling where appropriate, while preserving human oversight for safety, policy adherence, and domain nuance.
• Open questions the paper leaves unresolved. The authors note three limitations: prolonged real-time annotation may cause workload and fatigue, requiring rotation, adaptive workload management, breaks, and incentives; the study covers only English-language support, leaving multilingual and culturally diverse contexts untested; and the short study duration leaves unclear how annotation practices evolve and scale to larger agent groups.

Target Audience

This paper is most useful to applied ML engineers and product teams building RAG-based assistants on top of evolving proprietary knowledge, to operations and support leaders deciding whether to fold labeling into agent workflows, and to researchers working on human-in-the-loop alignment and data flywheel designs who want a production-grounded counterpoint to fully simulated feedback pipelines. Readers looking for novel model architectures or theoretical results will not find them here; the contribution is a deployed system design plus measured outcomes and ablations.

Authors’ abstract

We introduce an Agent-in-the-Loop (AITL) framework that implements a continuous data flywheel for iteratively improving an LLM-based customer support system. Unlike standard offline approaches that rely on batch annotations, AITL integrates four key types of annotations directly into live customer operations: (1) pairwise response preferences, (2) agent adoption and rationales, (3) knowledge relevance checks, and (4) identification of missing knowledge. These feedback signals seamlessly feed back into models' updates, reducing retraining cycles from months to weeks. Our production pilot involving US-based customer support agents demonstrated significant improvements in retrieval accuracy (+11.7% recall@75, +14.8% precision@8), generation quality (+8.4% helpfulness) and agent adoption rates (+4.5%). These results underscore the effectiveness of embedding human feedback loops directly into operational workflows to continuously refine LLM-based customer support system.