Fun-Audio-Chat Technical Report

AI summary

Overview

Research area: Natural Language Processing / spoken dialogue systems — specifically Large Audio Language Models (LALMs) that jointly handle speech and text.

Technical level: Advanced. The report assumes familiarity with transformer LLMs, autoregressive decoding, speech tokenizers, Mixture-of-Experts (MoE) architectures, and preference-optimization methods such as DPO.

Scope: The paper describes Fun-Audio-Chat, an 8B dense and 30B-A3B MoE audio language model (plus a full-duplex variant) built entirely through post-training on pre-trained components, with a dual-resolution design that processes audio at 5Hz in the LLM backbone while generating speech at 25Hz.

What This Paper Is About

Joint speech-text models must reconcile a basic mismatch: speech tokens run at roughly 25Hz while text tokens run at about 3Hz, which dilutes semantics, inflates compute, and tends to damage the text LLM's original knowledge during multimodal training. Fun-Audio-Chat addresses all three problems at once by using a shared backbone that reads grouped 5Hz audio representations while a separate Speech Refined Head restores 25Hz speech output, and by using a two-stage supervised training recipe with an intermediate model merge to prevent catastrophic forgetting. Notably, the model is trained without large-scale audio-text pre-training, relying instead on pre-trained components plus an extensive multi-stage post-training pipeline.

Key Contributions

1. Large-scale post-training and model scaling. The two ideas from the authors' prior work DrVoice — Dual-Resolution Speech Representations (DRSR) and Core-Cocktail Training — are scaled to millions of hours of diverse audio data and to dense 8B and MoE 30B-A3B model sizes, with the roughly 50% training GPU-hour reduction preserved at the larger scale.

2. Multi-Task DPO training. After Core-Cocktail Training, Direct Preference Optimization is applied across several preference dimensions at once — robustness to real speech, instruction-following, audio understanding, and voice empathy — adding capabilities beyond what supervised fine-tuning alone provides.

3. Comprehensive evaluation across diverse benchmarks. Fun-Audio-Chat is evaluated on Spoken Question Answering (both speech-to-text and speech-to-speech), audio understanding, speech recognition, speech function calling, speech instruction-following, and voice empathy, reporting competitive-to-leading results at comparable model scales.

4. A full-duplex variant and an open-source release. Fun-Audio-Chat-Duplex supports simultaneous two-way communication, and the dense Fun-Audio-Chat-8B checkpoint with training and inference code plus an interactive demo is released publicly.

Main Findings

• Spoken Question Answering at 8B scale: Fun-Audio-Chat-8B reports the best overall performance among roughly 8B-scale models on OpenAudioBench (76.61%) and VoiceBench (83.21%), and an overall of 59.56 on the speech-to-speech UltraEval-Audio benchmark.

• Large-scale comparison: Fun-Audio-Chat-30B-A3B reaches 80.59 overall on OpenAudioBench, 85.63 overall on VoiceBench, and 62.14 overall on UltraEval-Audio, which the paper describes as competitive with large-scale baselines including GPT-Audio, Gemini-2.5-Pro, and Longcat-Flash-Omni-Instruct (560B-A27B).

• Audio understanding leads evaluated models: On MMAU, Fun-Audio-Chat-30B-A3B scores 77.9% and Fun-Audio-Chat-8B scores 76.6%. On MMAU-Pro, the scores are 59.9% and 58.0%; on MMSU, 70.1% and 67.8%.

• Speech quality at 5Hz backbone rate: On the Llama Q. test set within UltraEval-Audio, Fun-Audio-Chat-8B achieves a UTMOS of 4.37 and an ASR-WER of 4.32%, which the authors use to argue that the dual-resolution design preserves speech quality despite efficient backbone processing.

• Speech recognition is competitive, not leading: Fun-Audio-Chat-30B-A3B and 8B report Librispeech clean WERs of 1.64 and 1.71, Librispeech other WERs of 3.73 and 4.13, Common Voice-EN WERs of 7.79 and 8.88, and Common Voice-ZH WERs of 5.88 and 6.16 — behind several smaller open baselines on several of these rows.

• Speech function calling: Fun-Audio-Chat-30B-A3B reports the highest overall score (79.63%) among evaluated models, with 76.40% on Speech-ACEBench (Single) and 84.13% on Speech-SmartInteract. Fun-Audio-Chat-8B reports 54.50% on ACEBench-Parallel and 87.63% on BFCL-Parallel, the latter above GPT-Audio and Gemini-2.5-Pro on that task.

• Full-duplex behavior: The report states Fun-Audio-Chat-Duplex achieves strong Spoken QA results and strong full-duplex interaction metrics, measured by S2M-T (text output accuracy in multimodal response), S2M-S (speech output accuracy in multimodal response), and Turn-taking Success Rate. Specific numeric results for these metrics are not reported in the excerpt available.

• VStyle and internal empathy results: The paper states competitive-to-superior performance on VStyle (English and Mandarin subsets) and on an internal test set covering Speech Instruction-Following, Semantics-based Empathy, and Paralinguistic-Cue-based Empathy. Numeric scores for these are not reported in the excerpt available.

Methodology in Plain English

The system is assembled from pre-trained parts rather than trained from scratch. A Whisper-Large-v3 encoder turns incoming audio into continuous representations, and an adapter shrinks their temporal resolution and reshapes them to match the LLM's hidden dimension. For the assistant's own speech output, the S3Tokenizer converts audio into discrete semantic tokens, and a detokenizer turns generated tokens back into waveforms using speaker embeddings, a Flow Matching model that produces Mel-spectrograms, and a HiFi-GAN vocoder.

The central efficiency trick is grouping. Because speech tokens arrive at 25Hz and text at about 3Hz, the system concatenates every five consecutive speech tokens (grouping factor k = 5) and projects them down with a linear layer, so the shared LLM backbone only processes 5Hz sequences. To recover the fine acoustic detail lost by grouping, a Speech Refined Head takes the LLM's final hidden state, projects it back to a larger embedding, splits it into five segments, and generates speech tokens autoregressively at full 25Hz resolution. That split between a 5Hz backbone and a 25Hz output head is what the report credits with roughly halving training GPU hours.

The LLM itself is a parallel joint speech-text model in the style of Moshi: speech and text token embeddings are summed into one input at each step, and the model predicts both text and speech outputs from a single autoregressive process. Modality alignment is applied only on the assistant side, reflecting the asymmetry that users speak or type, while the assistant can respond with speech, text, or both. Shorter of the two sequences is padded with a silence token, <|SIL|>.

Training proceeds in stages. A Pre-alignment stage updates the speech encoder, adapter, and Speech Refined Head on large-scale speech-text pairs while the LLM stays frozen. Core-Cocktail Training then does full supervised fine-tuning in two phases: Stage 1 uses a high learning rate decaying from 1×10⁻⁴ to 1×10⁻⁵ under cosine annealing to adapt quickly, then the trained weights are merged with the original pre-trained LLM weights by weighted interpolation (M_r ← αM₁ + (1−α)M₀, with α set to 0.5) to pull back the base model's knowledge, and Stage 2 fine-tunes the merged model with a lower learning rate decaying from 1×10⁻⁵ to 1×10⁻⁶. Multi-Task DPO follows, combining preference objectives for robustness, instruction-following, audio understanding, and voice empathy into one preference signal. The full-duplex extension continues from the Core-Cocktail checkpoint and adds a parallel input stream so the model can hear the user while speaking, trained on half-duplex dialogue data augmented into simulated full-duplex interactions following the OmniFlatten approach. Maximum context length is 2048 tokens, about 6 minutes of speech.

Why This Matters

Research impact. The report argues that strong audio capabilities do not require large-scale audio-text pre-training, only pre-trained components plus a carefully staged post-training pipeline. If that holds, it lowers the barrier to building capable spoken dialogue models and reframes the temporal resolution mismatch — long treated as an inherent cost of joint speech-text modeling — as an architectural design choice that can be worked around with grouping and a refined output head. Releasing the 8B checkpoint and code gives the community a reproducible baseline for the 5Hz/25Hz design.

Real-world applications:

• Voice assistants that execute function calls from spoken instructions, including single and parallel (multi-step) calls.
• Customer-facing or companion agents that recognize emotional cues and generate empathetic spoken responses.
• Real-time conversational systems that handle interruptions, backchanneling, and overlapping speech in full-duplex mode.
• Speech-to-speech question answering and multilingual speech recognition/transcription across English and Mandarin.

Industry relevance. The roughly 50% reduction in training GPU hours and the 5Hz backbone frame rate matter directly for deployment cost, since inference and training compute scale with sequence length. Speech function calling is the capability that connects a voice model to real services, and the full-duplex variant addresses the turn-taking behavior users expect from natural conversation. The choice to build from pre-trained LLMs plus post-training is also an industry-relevant template for teams that already own a strong text LLM.

Future Directions

• Extending the full-duplex evaluation story. The report describes S2M-T, S2M-S, and Turn-taking Success Rate metrics but does not surface detailed numbers in the provided content; a fuller comparison against Moshi and FreezeOmni would clarify where the duplex model stands.

• Closing the speech recognition gap. On most Librispeech and Common Voice rows, Fun-Audio-Chat trails several smaller open baselines, so improving raw ASR fidelity within the 5Hz backbone design is an open problem.

• Pushing scale and data further. The report frames scaling to millions of hours and 8B/30B-A3B as validation of scalability, which invites testing whether the DRSR and Core-Cocktail design continues to hold at larger scales and longer contexts than the 2048-token window (about 6 minutes of speech) reported here.

• Broadening the multi-task preference set. Multi-Task DPO currently covers robustness, instruction-following, audio understanding, and voice empathy; whether additional preference dimensions can be folded into the same unified preference signal without trade-offs is unresolved.

Target Audience

Researchers and engineers building spoken dialogue systems, joint speech-text models, or omni-modal assistants will get the most from this report, particularly those concerned with training and inference cost, catastrophic forgetting when adapting text LLMs to audio, or the design of speech token resolution. Practitioners interested in speech function calling, voice empathy, and full-duplex interaction will find the benchmark comparisons and the released 8B checkpoint useful as a starting point. Readers without a background in speech tokenization or LLM post-training will find the architecture and training sections dense.

Authors’ abstract

Recent advancements in joint speech-text models show great potential for seamless voice interactions. However, existing models face critical challenges: temporal resolution mismatch between speech tokens (25Hz) and text tokens (~3Hz) dilutes semantic information, incurs high computational costs, and causes catastrophic forgetting of text LLM knowledge. We introduce Fun-Audio-Chat, a Large Audio Language Model addressing these limitations via two innovations from our previous work DrVoice. First, Dual-Resolution Speech Representations (DRSR): the Shared LLM processes audio at efficient 5Hz (via token grouping), while the Speech Refined Head generates high-quality tokens at 25Hz, balancing efficiency (~50% GPU reduction) and quality. Second, Core-Cocktail Training, a two-stage fine-tuning with intermediate merging that mitigates catastrophic forgetting. We then apply Multi-Task DPO Training to enhance robustness, audio understanding, instruction-following and voice empathy. This multi-stage post-training enables Fun-Audio-Chat to retain text LLM knowledge while gaining powerful audio understanding, reasoning, and generation. Unlike recent LALMs requiring large-scale audio-text pre-training, Fun-Audio-Chat leverages pre-trained models and extensive post-training. Fun-Audio-Chat 8B and MoE 30B-A3B achieve competitive performance on Speech-to-Text and Speech-to-Speech tasks, ranking top among similar-scale models on Spoken QA benchmarks. They also achieve competitive to superior performance on Audio Understanding, Speech Function Calling, Instruction-Following and Voice Empathy. We develop Fun-Audio-Chat-Duplex, a full-duplex variant with strong performance on Spoken QA and full-duplex interactions. We open-source Fun-Audio-Chat-8B with training and inference code, and provide an interactive demo, at https://github.com/FunAudioLLM/Fun-Audio-Chat .