MegaChat: A Synthetic Persian Q&A Dataset for High-Quality Sales Chatbot Evaluation

AI summary

Overview

Research area: Natural Language Processing — low-resource language datasets, retrieval-augmented generation, and multi-agent conversational AI for e-commerce.

Technical level: Intermediate. The abstract assumes familiarity with RAG pipelines, retrieval/reranking, and multi-agent system design, though the problem it addresses (chatbot data scarcity for Persian-language online sellers) is explained in accessible terms.

Scope: The paper introduces MegaChat, a fully synthetic Persian question-and-answer dataset built by a multi-agent generation pipeline, and uses it to compare an agentic answer-generation system against classic retrieval-augmented generation models.

What This Paper Is About

Small and medium-sized businesses in Iran increasingly sell through Telegram, where fast, real-time responses drive conversions, but building AI chatbots for that setting requires large, high-quality Persian Q&A data that is costly and slow to produce by hand. MegaChat is presented as the first fully synthetic Persian Q&A dataset aimed at evaluating intelligent sales chatbots in Telegram-based e-commerce, generated automatically from real shopping channel content. The goal is to make high-quality conversational evaluation data available for a low-resource language without expensive human annotation or model fine-tuning.

Key Contributions

1. MegaChat dataset: Described as the first fully synthetic Persian Q&A dataset designed specifically for evaluating intelligent sales chatbots in Telegram-based e-commerce.
2. A multi-agent generation architecture: An automated pipeline that collects data from active Telegram shopping channels and produces persona-aware Q&A pairs using specialized agents for question generation, validation, and refinement.
3. An agentic answer-generation system: A system combining multi-query retrieval, reranking, and persona-aligned response synthesis, positioned as an alternative to conventional RAG.
4. A comparative evaluation: An assessment of three classic RAG models against the agentic system, using GPT-5.1 as the evaluator across six quality dimensions.

Main Findings

• Synthetic data can substitute for human annotation: The abstract states the approach yields scalable, high-quality datasets without relying on expensive human annotation or complex fine-tuning.
• Agentic RAG beat classic RAG in most channels: Under GPT-5.1 evaluation, the agentic architecture outperformed traditional RAG models in 4 out of 5 diverse channels tested.
• Persona-aware generation produced realistic, diverse data: The authors claim the agent pipeline (generation, validation, refinement) ensures realistic and diverse conversational data.
• Evaluation was multi-dimensional: Comparisons relied on six quality dimensions, though the abstract does not name them or report per-dimension scores or margins of difference.
• Cost and resource efficiency: The framing throughout is that this approach is more efficient and cost-effective than alternatives requiring annotation labor or fine-tuning.

Methodology in Plain English

The researchers gathered material from Telegram shopping channels that are actively used for sales. From that content, they ran an automated, multi-agent pipeline: one set of agents writes candidate questions, others check them, and others polish them, with the aim of producing Q&A pairs that reflect realistic buyer personas rather than generic questions. To test answer quality, they set up two kinds of systems. The first were three standard RAG models — the familiar pattern of retrieving relevant text and having a language model answer from it. The second was their own "agentic" system, which does more work before answering: it issues multiple retrieval queries, reranks what comes back, and shapes the final answer to fit a particular persona. GPT-5.1 was then used as the judge, scoring outputs along six quality dimensions. Notably, the abstract reports no dataset size, no per-dimension scores, and no numeric effect sizes; those details are not available in the abstract.

Why This Matters

Impact on research: The work targets a recognized bottleneck in low-resource language NLP — the scarcity of evaluation data. If synthetic, agent-generated data can be validated as a substitute for human annotation in a specialized commercial domain, that approach could be transferred to other languages and verticals where annotated corpora are impractical to build.

Real-world applications:

• Persian-language sales chatbots for Telegram shops, able to answer product and order questions in a persona-appropriate tone.
• Evaluation benchmarks for vendors building customer-service AI in domains where no public Q&A data exists.
• Rapid bootstrapping of domain-specific assistants for SMEs that cannot fund annotation projects.
• Multilingual conversational AI development, by offering a template for extending the pipeline to other low-resource languages.

Industry relevance: The target users are small and medium-sized enterprises, specifically in Iran's Telegram commerce ecosystem, for whom cost-effective customer engagement tooling is a practical need rather than an academic exercise. The emphasis on avoiding fine-tuning and human labeling speaks directly to the budget and staffing constraints of that market.

Future Directions

• Extending beyond Persian and Telegram: Testing whether the multi-agent generation pipeline transfers to other low-resource languages and other commercial platforms.
• Validating synthetic data against human judgment: The abstract does not report agreement between GPT-5.1 scores and human raters, lexifying the reliability of the evaluation, this is an open question.
• Understanding the limits of persona awareness: It remains unclear from the abstract how personas are defined or how much they contribute to the performance gains.
• Defining and reporting the quality dimensions: The six dimensions are named only in number, so future work would need to establish which aspects of answer quality the agentic system actually improves and by how much.

Target Audience

Researchers working on low-resource language NLP, retrieval-augmented generation, and multi-agent data synthesis would benefit most, along with practitioners building conversational AI for e-commerce in markets where annotated data is unavailable. Product and engineering teams at SMEs or chatbot vendors serving Persian-speaking Telegram sellers will find the applied framing and the released dataset directly useful, as will anyone seeking a template for generating domain-specific Q&A data without human labeling.

Authors’ abstract

Small and medium-sized enterprises (SMEs) in Iran increasingly leverage Telegram for sales, where real-time engagement is essential for conversion. However, developing AI-driven chatbots for this purpose requires large, high-quality question-and-answer (Q&A) datasets, which are typically expensive and resource-intensive to produce, especially for low-resource languages like Persian. In this paper, we introduce MegaChat, the first fully synthetic Persian Q&A dataset designed to evaluate intelligent sales chatbots in Telegram-based e-commerce. We propose a novel, automated multi-agent architecture that generates persona-aware Q&A pairs by collecting data from active Telegram shopping channels. The system employs specialized agents for question generation, validation, and refinement, ensuring the production of realistic and diverse conversational data. To evaluate answer generation, we compare three classic retrieval-augmented generation (RAG) models with our advanced agentic system, which features multi-query retrieval, reranking, and persona-aligned response synthesis. Using GPT-5.1 for evaluation across six quality dimensions, our results show that the agentic architecture outperformed traditional RAG models in 4 out of 5 diverse channels, demonstrating its ability to generate scalable, high-quality datasets without relying on expensive human annotation or complex fine-tuning. MegaChat provides SMEs with an efficient, cost-effective solution for building intelligent customer engagement systems in specialized commercial domains, enabling advancements in multilingual conversational AI for low-resource languages. Download: https://github.com/MegaChat-Tech/MegaChat-DataSet