Incremental Summarization for Customer Support via Progressive Note-Taking and Agent Feedback

AI summary

Overview

Research area: Applied natural language processing, specifically dialogue summarization for customer support, with LLM fine-tuning and human-in-the-loop feedback.

Technical level: Intermediate to Advanced. The paper assumes familiarity with LLM fine-tuning (SFT, DPO, ORPO), preference data construction, classifier training, and production serving stacks (TensorRT-LLM, Triton).

Scope: The paper describes a production-deployed incremental (real-time) summarization system that generates bullet-style notes during customer support conversations rather than only at the end, and reports offline and online results from that deployment.

What This Paper Is About

Customer support agents must write summary notes for handoffs, compliance, and quality audits, and the paper cites industry figures that note-writing consumes roughly 10% of case handling duration. Existing summarization tools mostly run after the conversation ends, or regenerate a full summary when input crosses a length threshold, which does not give agents real-time support while they work. The authors build a system that decides when to emit a short new bullet during a live conversation, filters out trivial bullets, and learns continuously from the corrections agents make to those bullets.

Key Contributions

1. A Progressive Note-Taking with Quality Control workflow. A fine-tuned Mixtral-8x7B model proposes incremental bullets when new information arrives, and a fine-tuned DeBERTa-based relevance classifier filters out trivial or redundant content before it reaches the agent's UI.

2. An Agent-Edits Learning Framework with an online and an offline path. Agent edits immediately update the in-context prompt for subsequent bullet generation, and are also logged, quality-assessed, rewritten when needed, and converted into supervised and preference-tuning data for periodic offline retraining.

3. A production deployment with a quasi-experimental evaluation. Because direct A/B testing was infeasible due to agent experience variance and scheduling complexity, the authors use a Diff-in-Diff design comparing pilot sites against all other sites from three months before to one month after rollout, covering over 92% of production cases.

4. An evaluation methodology built around an LLM judge. The authors compare a GPT-4o-based LLM judge against human annotators on 522 case samples using matching binary rubrics, and find the LLM judge outperformed human annotators in almost all metrics while processing evaluations approximately 15 times faster.

Main Findings

• Business impact of the deployed system: The system achieved a 3% reduction in case handling time compared to bulk summarization, with reductions of up to 9% for the most complex cases (defined as cases involving 3+ agents, usually with solving time over 100 minutes). The reported p-value is below 0.001.

• Agent satisfaction: Surveys from over a thousand agents showed 95.2% (English), 81.8% (French), and 89.4% (Spanish) reported being "Satisfied" or "Very Satisfied."

• Note-writing time collapsed: Agents' manual note-writing time ratio decreased from 10% to 3% after the feature rollout.

• Model-generated notes beat agent-written notes on completeness: The paper reports model-generated notes having higher completeness than agents' manual notes (0.871 vs 0.624) in the online results section; the appendix table of human-annotated comparisons gives model vs agent completeness of 0.884 vs 0.683 (EN), 0.894 vs 0.604 (FR), and 0.835 vs 0.585 (ES), with truthfulness of 0.917 vs 0.801 (EN), 0.875 vs 0.825 (FR), and 0.904 vs 0.775 (ES).

• Iterative fine-tuning with agent feedback improved offline quality: On 1,200 randomly sampled production case conversations, Mixtral-FB (feedback) scored completeness 0.842, truthfulness 0.996, and overall 0.885, versus Mixtral-NF (no feedback) at 0.824 / 0.995 / 0.879 and Mixtral-Base at 0.585 / 0.989 / 0.787. GPT-4o scored 0.846 completeness, 0.995 truthfulness, 0.788 conciseness, and 0.876 overall.

• The gains over both baselines were statistically significant: Mixtral-FB vs Mixtral-NF had a 95% bootstrap CI of [+0.28%, +0.96%] with p = 4.6 × 10⁻⁴; Mixtral-FB vs GPT-4o had a CI of [+0.46%, +1.28%] with p = 4.1 × 10⁻⁵. Mixtral-NF vs GPT-4o was not significant (CI [-0.65%, +0.13%], p = 0.202).

• GPT-4o was not deployed despite competitive quality: Its lower conciseness plus operational constraints (latency, cost, data privacy) prevented deployment. The paper's cost comparison lists GPT-4o at over 100B parameters, $475k/year, and 4s p95 latency, versus Mixtral-8x7B at 45B parameters, $100k/year, and 5s p95 latency.

• The bullet classifier improved overall summary quality: Adding bullet classifier filtering (BCF) slightly reduced completeness and did not significantly affect truthfulness, but overall score rose from 0.880 to 0.888 due to improved conciseness. Summary length decreased by 26.5% on average and 24.9% at the median.

• Chunk-based summarization underperformed: Mixtral-FB with 200-word chunks scored 0.841 overall and with 500-word chunks scored 0.877, both below the note-taking approaches (0.880 without BCF, 0.888 with BCF).

• Classifier performance: The bullet classifier achieved ROC AUC 0.96, macro F1 0.801, and micro F1 0.845. Used as a binary classifier retaining any of the five target classes, it achieved F1 0.895. It was pretrained on a 500G domain-specific corpus mixed with CCNet, then fine-tuned on approximately 70,000 bullets with an 80:10:10 split.

• Public dataset benchmark: BERTScore F1 on SAMSum and DialogSum respectively was 0.874 / 0.861 for Mixtral-Base, 0.881 / 0.866 for Mixtral-NF, and 0.888 / 0.871 for Mixtral-FB. The comparison baseline Llama-2-ROR-FG (results taken from Tian et al., 2024) scored 0.685 / 0.757. The authors state these are for reference only.

• Customer NPS was neutral across languages, indicating no observable negative impact on customer satisfaction.

• Serving learnings: 8-bit quantization reduced GPU usage from two A100s to one with similar latency and a 1% drop in completeness and truthfulness (tested on 2,500 cases). Replacing vLLM with TensorRT-LLM reduced inference latency by 20% at p50 and p90. The final stack uses BF16 with TensorRT-LLM, with 8-bit retained as an efficiency fallback.

• Language-specific weaknesses: French summaries had lower truthfulness, attributed mainly to numeric inaccuracies from complex number structures (the paper gives "quatre-vingt-quatre" for 84, literally "4 × 20 + 4"). Spanish summaries had lower completeness in capturing agent solutions, attributed to varied verb conjugations and passive forms (for example, indicative "enviaré" versus conditional "enviaría").

• Serving configuration in production: The note-taking workflow runs inference via NVIDIA Triton using TensorRT-LLM with each model instance on 2 A100 GPUs (p50 latency 600 ms, p95 2s). The bullet classifier runs on an A10G GPU (p50 latency 20 ms, p95 latency 40 ms).

Methodology in Plain English

The system watches a conversation as it unfolds across phone, chat, email, and contextual data, and asks a fine-tuned Mixtral-8x7B model a simple question each time new input arrives: is there new information here? If yes, the model generates a bullet. A second model, a DeBERTa-based classifier, then decides whether that bullet is meaningful, keeping only five categories of content (customer_provides_issue, customer_provides_context, customer_takes_action, agent_asks_follow_up, agent_provides_solution) and discarding the rest. The LLM is prompted with the task instructions, case metadata, the full interaction history, and the previously accepted bullets, so it only needs to produce new incremental bullets; if there is nothing new, it returns an empty response. This design matters because re-summarizing everything each turn would overwrite the corrections agents had already made.

Agents see the bullets in their UI and can edit them. Those edits do two things. Immediately, they are saved and fed back into the prompt, so the next generation uses agent-verified wording. Separately, every edit is logged as a before-edit and after-edit pair, quality-checked by an LLM and sampled human review, rewritten by a stronger LLM when it fails the check, and only then turned into training data. Clean pairs become supervised fine-tuning examples, and before/after pairs with clear quality improvement become preference pairs for DPO or ORPO. From Mixtral-Base to Mixtral-NF the authors collected about 30k examples and ran one round of SFT and one round of ORPO; from Mixtral-NF to Mixtral-FB they collected about 8k examples and ran one round of ORPO.

For evaluation, the authors define conciseness as one minus the ratio of summary tokens to input conversation tokens, and evaluate completeness and truthfulness as binary yes/no rubrics via both human reviewers and a GPT-4o LLM judge. Because customer conversations have no single gold summary, they treat semantic-similarity metrics like BERTScore as supplementary only. Online, they measure agent satisfaction by survey, agent working time in minutes, and customer NPS, using a Diff-in-Diff design rather than a randomized test.

Why This Matters

Impact on research. The paper is a detailed case study of closing the loop between a deployed LLM feature and its users: agent edits serve simultaneously as an online context update and as offline preference training data. It also contributes a direct head-to-head comparison of a GPT-4o LLM judge against human annotators on matched binary rubrics, including a full human annotation guideline and judge prompt in the appendix, which is useful for anyone designing evaluation for production summarization.

Real-world applications:

• Customer support platforms that need live note-taking during phone, chat, and email interactions.
• Meeting or call assistants that must surface incremental notes rather than a single post-call recap.
• Handoff and compliance workflows, where notes must be accurate and complete for the next agent or for audit.
• Any human-in-the-loop LLM system where user corrections need to flow back into both the live prompt and the training pipeline.

Industry relevance. The paper quantifies the economics: at the studied production volume, the reported reductions correspond to O(10⁵–10⁶) agent hours saved annually, implying multi-million dollar cost savings annually. It also documents practical constraints that decide deployment — GPT-4o was not deployed despite competitive quality because of latency, cost, and data privacy, while Mixtral-8x7B was chosen at $100k/year versus $475k/year for GPT-4o in the cost model.

Future Directions

1. Extending to additional languages beyond the English, Spanish, and French covered in this deployment, given the language-specific truthfulness and completeness gaps observed offline.

2. Incorporating multimodal contexts such as images, which the current system does not handle.

3. Continuous retraining and updating of the bullet classifier from ongoing agent feedback, which the authors identify as important because the classifier occasionally misclassifies content and can cause omissions or irrelevant updates.

4. Reducing reliance on the GPT-4o-based LLM judge, which the limitations section notes may introduce evaluation biases or inaccuracies relative to human judgment.

Target Audience

Applied ML engineers and researchers building production LLM systems with human-in-the-loop feedback, particularly those working on dialogue summarization, agent-assist tooling, or customer support automation. It is also relevant to evaluation practitioners interested in LLM-as-a-judge versus human annotation, and to product and operations leaders seeking a template for measuring a deployed ML feature with quasi-experimental methods and business metrics such as handling time, satisfaction surveys, and NPS.

Authors’ abstract

We introduce an incremental summarization system for customer support agents that intelligently determines when to generate concise bullet notes during conversations, reducing agents' context-switching effort and redundant review. Our approach combines a fine-tuned Mixtral-8x7B model for continuous note generation with a DeBERTa-based classifier to filter trivial content. Agent edits refine the online notes generation and regularly inform offline model retraining, closing the agent edits feedback loop. Deployed in production, our system achieved a 3% reduction in case handling time compared to bulk summarization (with reductions of up to 9% in highly complex cases), alongside high agent satisfaction ratings from surveys. These results demonstrate that incremental summarization with continuous feedback effectively enhances summary quality and agent productivity at scale.