AI Knowledge Assist: An Automated Approach for the Creation of Knowledge Bases for Conversational AI Agents

AI summary

Overview

Research area: Natural Language Processing / applied LLM systems for contact center AI (Retrieval Augmented Generation, knowledge base construction, supervised fine-tuning of lightweight LLMs).

Technical level: Intermediate. The paper assumes familiarity with LLMs, fine-tuning, embeddings and clustering, but the pipeline itself is describable in plain terms.

One-sentence scope: The paper describes and evaluates AI Knowledge Assist, a three-stage LLM pipeline that mines historical customer–agent call transcripts to automatically build question–answer knowledge bases for conversational AI agents, evaluated on real data from 20 companies.

What This Paper Is About

Contact centers that want to deploy RAG-powered conversational agents usually need a company-specific knowledge base of question–answer pairs, and most of them do not have one — a "cold start" problem that blocks adoption. The authors' goal is to build that knowledge base automatically from conversation logs the contact center already owns (call transcripts and chat histories), so a chatbot can be deployed immediately rather than after months of manual FAQ curation.

Key Contributions

1. A three-stage automated knowledge base construction system — knowledge extraction from transcripts, clustering for deduplication, and LLM-based recommendation of representative QA pairs (with optional human review by a "Knowledge Manager").
2. A cost-efficient fine-tuned model, Knowledge-Assist-8B-SFT, built on LLaMA-3.1-8B, which outperforms larger and closed-source cost-efficient models (GPT-4o-Mini, Gemini-2.0-Flash / Flash-Lite, Gemini-2.5-Flash-Lite, DeepSeek-R1-LLaMA-8B) on the knowledge extraction task.
3. A reference-free LLM-as-a-judge evaluation protocol using GPT-4o, motivated by the risk of bias when the evaluation data itself is annotated by Gemini-2.5-Pro; the paper reports about 90% agreement between GPT-4o judgments and human judgments.
4. A real-world deployment description on Kubeflow / Google Vertex AI requiring 1 L4 GPU, including a proposed self-updating mechanism that detects new customer issues and stale answers via embedding similarity thresholds.

Main Findings

• Knowledge extraction results (Table 1): Knowledge-Assist-8B-SFT achieved Precision 84.88, Recall 84.85, and F1-Score 84.86, the best of all compared models. GPT-4o-Mini reached 71.53 F1 and Gemini-2.5-Flash-Lite reached 73.68 F1. LLaMA-3.1-8B-Instruct (58.13), DeepSeek-R1-LLaMA-8B (49.71), Gemini-2.0-Flash (69.60), and Gemini-2.0-Flash-Lite (64.86) scored lower.
• Reference-wise metrics favour a closed model: On ROUGE-1, ROUGE-2, ROUGE-L and BERTScore, Gemini-2.5-Flash-Lite scored highest (54.17 / 25.42 / 28.74 / 66.86) versus Knowledge-Assist-8B-SFT (41.26 / 19.68 / 23.87 / 60.12), which the authors attribute to the evaluation references being annotated by Gemini-2.5-Pro.
• Clustering: Using DBSCAN with BGE-Large embeddings, 1578 clusters were found for the Gemini-2.5-Flash-Lite extracted QA pairs and 1429 clusters for the Knowledge-Assist-8B-SFT extracted QA pairs. Appendix B reports DBSCAN performs much better than K-Means on the Silhouette metric, and that DBSCAN's automatic determination of cluster count matters because call volumes and cluster structures vary across companies.
• End-to-end recommendation results (Table 2): Knowledge-Assist-8B-SFT reached Precision 91.4, Recall 92.2, F1 91.8 with 14K representative QA pairs, versus Gemini-2.5-Flash-Lite at 81.1 / 78.1 / 79.6 with 13K. The abstract frames this as "above 90% accuracy" and says this eliminates the cold-start gap.
• Per-company consistency: For the majority of companies, F1-Score on final recommended representative QA pairs is above 90% (Figure 3).
• Backbone choice matters (Table 3): Replacing LLaMA-3.1-8B with Qwen3-8B dropped F1 from 84.86 to 74.73.
• Annotator choice matters (Table 3): Replacing the Gemini-2.5-Pro data annotator with GPT-4o dropped F1 to 65.95 (Recall fell to 55.89).
• Human evaluation (Table 4): Across 100 randomly selected conversations, humans preferred Knowledge-Assist-8B-SFT recommended QA pairs in 25% of cases and Gemini-2.5-Flash-Lite pairs in 17%; 58% were rated "Tie." Total approved pairs were 107 for Knowledge-Assist-8B-SFT and 98 for Gemini-2.5-Flash-Lite. A Wilcoxon signed-rank test showed the difference statistically significant (p ≤ 0.05).
• Judge reliability: About 90% agreement was found between human-annotated judgments and GPT-4o judgments; Appendix D gives examples where the LLM judge labelled correct QA pairs as incorrect.
• Cost positioning: All open-source models used have fewer than 10 billion parameters and fit on a single NVIDIA L4 GPU. Gemini-2.0-Flash-Lite is the cheapest proprietary option at $0.075 input / $0.30 output per million tokens; Gemini-2.0-Flash at $0.15 / $0.60; GPT-4o-mini at $0.15 / $0.60; Gemini-2.5-Flash was omitted due to $0.30 / $2.50 pricing; GPT-4o costs $2.50 / $10 and Gemini-2.5-Pro (used for annotation) costs $1.25 / $10.

Methodology in Plain English

The researchers took a month of real call transcripts (November 2024) from 20 client companies of Dialpad, anonymized them with Google Cloud Data Loss Prevention, and ran a three-step pipeline.

1. Extract. An LLM reads each transcript and pulls out customer information-seeking questions paired with the agent's answers. Because the text comes from noisy ASR output, the model is allowed to rewrite the pairs so they make sense without the surrounding conversation. Only non-sensitive, non-PII, non-time-sensitive, general questions are kept.
2. Group. The extracted QA pairs are embedded (BGE-Large) and clustered by cosine distance between question embeddings using DBSCAN, so near-duplicate questions land in the same cluster.
3. Merge and recommend. An LLM looks at each cluster and outputs one or more representative QA pairs, either extracting the best one verbatim or rewriting/synthesizing across the cluster. These go into the knowledge base directly or to a human reviewer.

Training data was bootstrapped from the same pipeline run with Gemini-2.5-Pro as the annotator, producing 27,500 instances: 12,500 for knowledge extraction (5,500 train / 7,000 evaluation) and 15,000 for representative QA pair recommendation (2,500 train / 12,500 evaluation). The target model was LLaMA-3.1-8B, supervised fine-tuned for 3 epochs with a maximum sequence length of 8000 tokens (4000 input, 4000 output) and a learning rate between 2e-4 and 2e-6, on 8 NVIDIA A100 GPUs. Evaluation used a GPT-4o judge in a reference-free setting rather than ROUGE/BERTScore alone, to avoid favoring models fine-tuned on Gemini-annotated references.

Why This Matters

The paper is one of the few descriptions of an end-to-end LLM system for contact center knowledge base creation that reports deployment constraints, cost trade-offs, ablation on model and annotator choice, and human evaluation alongside automatic metrics. It argues that reference-free LLM judging is the more trustworthy signal when the ground truth itself is LLM-annotated — a methodological point of general interest.

Real-world applications:

• Cold-start chatbot deployment: A contact center with no knowledge base can bootstrap one from existing call recordings and launch a RAG agent immediately.
• FAQ and help-article generation: Representative QA pairs produced from clusters map naturally onto help-center content.
• Knowledge base maintenance: The proposed self-update loop compares new question embeddings against existing knowledge base questions to flag new issues, and compares answers for the same question to flag obsolete content after product changes.
• Voice-of-customer mining across companies: The pipeline runs per contact center and per timeframe, so operators of many client accounts can generate separate knowledge bases from each one.

Industry relevance is direct: the system runs on Kubeflow on Google Vertex AI with 1 L4 GPU, and the paper explicitly prioritizes models under 10B parameters for inference speed and cost.

Future Directions

• Efficient knowledge base updating: The authors state they will build new benchmarks to study how to efficiently update existing knowledge bases, extending the self-updating mechanism sketched in the deployment section.
• Generalization beyond customer–agent conversations: The Limitations section notes the models may not suit other domains without further prompt engineering or fine-tuning, leaving cross-domain transfer open.
• Reducing LLM-judge errors: Appendix D documents cases where the GPT-4o judge marked correct QA pairs as incorrect, which the reported 90% human agreement does not fully resolve.
• Releasing data or benchmarks: The dataset is proprietary and not released, so the community currently cannot reproduce results on the same data — the paper compensates only with detailed model names, fine-tuning parameters, and verbatim prompts.

Target Audience

This paper is most useful to applied NLP engineers and ML practitioners building RAG systems or conversational agents for enterprise settings, to contact center product and engineering teams evaluating cold-start solutions, and to researchers interested in LLM-as-a-judge evaluation and in cost-accuracy trade-offs between fine-tuned open models and closed APIs.

Authors’ abstract

The utilization of conversational AI systems by leveraging Retrieval Augmented Generation (RAG) techniques to solve customer problems has been on the rise with the rapid progress of Large Language Models (LLMs). However, the absence of a company-specific dedicated knowledge base is a major barrier to the integration of conversational AI systems in contact centers. To this end, we introduce AI Knowledge Assist, a system that extracts knowledge in the form of question-answer (QA) pairs from historical customer-agent conversations to automatically build a knowledge base. Fine-tuning a lightweight LLM on internal data demonstrates state-of-the-art performance, outperforming larger closed-source LLMs. More specifically, empirical evaluation on 20 companies demonstrates that the proposed AI Knowledge Assist system that leverages the LLaMA-3.1-8B model eliminates the cold-start gap in contact centers by achieving above 90% accuracy in answering information-seeking questions. This enables immediate deployment of RAG-powered chatbots.