MCGA: A Multi-task Classical Chinese Literary Genre Audio Corpus

AI summary

Overview

Research area: Multimodal NLP / Chinese Classical Studies (CCS) — speech and text corpora for evaluating multimodal large language models.

Technical level: Intermediate. Readers will benefit from familiarity with ASR, speech translation, and multimodal LLM benchmarks, though the paper explains each task clearly.

Scope: The paper introduces MCGA, a 119-hour, 22,000-sample audio corpus of classical Chinese literature spanning six speech tasks, and reports evaluations of ten multimodal LLMs plus a fine-tuning experiment on it.

What This Paper Is About

Most work applying multimodal large language models to Chinese Classical Studies concerns text or images, while the audio dimension has been held back by the absence of a high-quality, domain-specific speech corpus. The authors build MCGA — a 119-hour, 22,000-sample corpus of human-recorded classical Chinese literature covering six speech tasks — and use it to benchmark ten multimodal LLMs. Their goal is both to expose where current models fail on classical Chinese audio and to supply a training resource that improves them.

Key Contributions

1. MCGA corpus: The first large-scale (119 hours), open-source, fully copyrighted audio corpus for classical Chinese literature, covering 5 genres (Fu, Shi, Wen, Ci, Qu), 11 historical periods, 37 period–genre categories, and 4,497 literary works.
2. Evaluation framework: Six speech-centric tasks — ASR, S2TT, SEC, SQA, SU, and SR — plus four text tasks supported by the parallel speech–text data (MT, QA, LU, LR).
3. Two new metrics: Emotion Caption Fidelity (ECF), an LLM-based score tailored to literary speech emotion captioning, and Cross-Modal Consistency (CMC), which quantifies the gap between a model's speech and text performance on the same questions.
4. Empirical analysis and training utility: Evaluation of 10 MLLMs identifying performance bottlenecks, plus a LoRA fine-tuning experiment showing substantial gains from training on MCGA.

Main Findings

• Top model overall: Qwen3-Omni-30B-A3B-Instruct achieves the highest aggregate score at 442.9, narrowly ahead of Gemini-3-Flash at 440.9. Both stay below 60 on SEC (58.4 and 54.0 ECF respectively).
• Open vs. closed divide: Qwen3-Omni leads on Chinese understanding and generation tasks — ASR (4.4 CER), SEC (58.4 ECF), SQA (51.5 F1), and SU (86.9 Acc). Gemini-3-Flash leads on English generation and Chinese reasoning — S2TT (74.0 LLM-B) and SR (83.7 Acc). Open-source models have reached a competitive level overall.
• Task difficulty ordering: Models are strongest on ASR, followed by SU and SR (multiple-choice format). S2TT is acceptable but improvable. SEC is notably poor — a critical gap in affective computing. SQA F1 scores remain low, indicating unresolved hallucination.
• Genre disparities in ASR: Ci consistently yields lower CER while Fu is consistently hardest, attributed to Fu's ornate rhetoric, frequent classical allusions, and high density of modal particles. Qwen3-Omni reaches 2.8 CER on Ci; the LoRA-tuned Qwen-Omni-MCGA reaches 2.8 on Shi and 3.1 on Ci, achieving state-of-the-art on all genres except Qu.
• Audio quality is consistent: A 0.1 CER gap separates the human-verified valid/test sets from the train set (Qwen3-Omni). Residual errors come mainly from uncommon characters and phonetic loanwords (tongjiazi).
• SEC ground truth gap: Human ground truth scores 100.0 on ECF, versus 58.4 for the best model. GPT-4o-mini-Audio performs poorly (5.7 ECF), largely because its safety protocols trigger refusals on persona-based or emotional analysis.
• Open-ended vs. multiple-choice gap: Gemini-3-Flash scores 86.6 on SU and 83.7 on SR but drops to 48.7 on SQA, indicating severe hallucination in open-ended factual QA despite strong reasoning.
• Cross-modal consistency: Step-Audio-2-mini achieves the highest CMC among evaluated MLLMs at 90.7, ahead of Qwen3-Omni at 89.9.

Methodology in Plain English

The authors gathered public-domain classical Chinese texts and their pinyin from the web, cleaned them, and split them into clips short enough to read in 30 seconds. They used DeepSeek-V3.2 to generate question–answer pairs for S2TT, SEC, SQA, SU, and SR, then filtered these through trio validation with DeepSeek-V3.2, GPT-5-mini, and Gemini-3-Flash; test and validation sets were also human-verified.

Recording was done by 28 native Mandarin speakers (13 male, 15 female, aged 18–40) through a private website, with guidelines on matching tone to the text's emotion, quiet environments, 30-second clips, and at least one male and one female reading per clip. Clips from the same work went to the same person.

Quality control happened in two stages: an MLLM check using Qwen and Whisper models to flag samples with significant errors for re-recording, and a human check by 6 inspection volunteers who scored validation and test samples, removing those with pronunciation errors or background noise.

For evaluation, 10 MLLMs were run via vLLM with API requests at temperature 0. ASR used Character Error Rate; S2TT and SEC were scored by the deepseek-chat API and normalized to a 100-point scale; SQA used F1; SU and SR used Accuracy. Training used LoRA (r=8, α=32) on Qwen2.5-Omni-7B via ms-swift, for 3 epochs on 4 A100 GPUs with AdamW, learning rate 1×10⁻⁴, per-device batch size 8, and gradient accumulation of 4.

Why This Matters

Impact on research: MCGA addresses a genuine gap — the absence of high-quality audio data in Chinese Classical Studies — and provides a reusable benchmark with two new metrics (ECF and CMC). Its explicit copyright transfers resolve a long-standing Intellectual Property Rights obstacle that has discouraged open-source audio datasets in this domain. Demonstrating that fine-tuning on MCGA yields substantial gains also validates the corpus as training data, not just an evaluation set.

Real-world applications:

• Cultural preservation and digital archiving of classical Chinese literature in spoken form.
• Speech-enabled educational tools for classical Chinese literature, including pronunciation and recitation support.
• Accessibility systems that let learners interact with classical texts through audio and spoken questions.
• Multilingual mediation, translating classical Chinese speech into other languages while preserving its aesthetic qualities.

Industry relevance: The corpus is directly useful to teams building multimodal assistants for cultural heritage, education technology, and Chinese-language speech products. The finding that strong reasoning models still hallucinate badly on open-ended factual questions from audio points to a concrete product risk for voice-based cultural or historical Q&A systems.

Future Directions

1. Adding aligned visual data: The authors note copyright constraints prevent including real-world images matched to both the text and audio; securing such data would extend MCGA toward fully trimodal cultural study.
2. Broadening the Qu genre: Qu is underrepresented because the Yuan Dynasty was short-lived and fewer works survive; expanding coverage would balance the corpus across genres.
3. Improving affective modeling: SEC performance is the weakest area, with the best model at 58.4 ECF against a 100.0 ground truth — a large gap inviting dedicated affective-computing work.
4. Reducing hallucination in open-ended QA: The SQA-to-QA gap (for example, 48.7 versus 66.0 for Gemini-3-Flash) shows speech-based open-ended factual answering remains unsolved, and the CMC metric offers a way to track progress.

Target Audience

Researchers and engineers working on multimodal large language models, speech processing, and low-resource or domain-specific benchmarks, especially those interested in Chinese-language and cultural-heritage applications. It is also relevant to digital humanities scholars seeking spoken-language resources, and to practitioners who need a ready-made corpus for fine-tuning or evaluating audio-capable models on classical Chinese literature.

Authors’ abstract

With the rapid advancement of Multimodal Large Language Models (MLLMs), their potential has gained significant attention in Chinese Classical Studies (CCS). While existing research primarily focuses on text and visual modalities, the audio corpus within this domain remains largely underexplored. To bridge this gap, we introduce the Multi-task Classical Chinese Literary Genre Audio Corpus (MCGA), a 119-hour corpus comprising 22,000 audio samples. It encompasses a diverse range of literary genres across six tasks: Automatic Speech Recognition (ASR), Speech-to-Text Translation (S2TT), Speech Emotion Captioning (SEC), Spoken Question Answering (SQA), Speech Understanding (SU), and Speech Reasoning (SR). Through the evaluation of ten MLLMs, our experimental results demonstrate that current MLLMs still face substantial challenges on the MCGA test set. Furthermore, we introduce a domain-specific metric for SEC and a metric to measure the consistency between speech and text capabilities. We release MCGA to the public to facilitate the development of more robust MLLMs. MCGA Corpus: https://github.com/yxduir/MCGA