Apriel Voice 27B Preview: An Open Model for Speech, Reasoning, and Tool Use

AI summary

Overview

Research area: Speech-language models (cs.SD), specifically speech-conditioned multimodal LLMs that combine speech understanding with reasoning and tool calling.

Technical level: Advanced. The paper includes formal training objectives (cross-entropy plus forward KL divergence), projector algebra, and distributed-training engineering, though the overall narrative is readable for anyone familiar with transformer-based assistants.

Scope (one sentence): The paper describes Apriel Voice 27B Preview, an open speech-language model built on Gemma 4 26B-A4B that handles spoken input, text, images, and tool calls in a single reasoning backbone, together with its two-stage training recipe, data pipeline, efficient online-distillation infrastructure, and evaluation results.

What This Paper Is About

Voice assistants fail in practice not because they mishear words, but because they cannot reason over what was said, follow constraints, hold a conversation, or take an action such as opening a support ticket. The paper argues that recognition accuracy alone is insufficient, and that a model should be evaluated on whether it completes the requested task.

The goal is to build an openly available speech-language model that projects audio directly into an existing, already post-trained language backbone so that speech becomes just another input interface to the same reasoning and tool-use abilities the text model already has — without a separate dialogue model, and with speech synthesis kept replaceable.

Key Contributions

1. Broad speech-conditioned agentic capability. A single shared model handles spoken reasoning, instruction following, multi-turn conversation, and tool-call generation, while retaining the original text and image input modalities and the original output vocabulary and tool-call formats.

2. A staged adaptation recipe. Stage 1 learns the speech interface with the backbone frozen; Stage 2 jointly adapts the projector and backbone on speech tasks, automatically constructed speech-conditioned dialogues, tool interactions, and text replay, using source-dependent cross-entropy (CE) and forward Kullback–Leibler divergence (FKL) supervision from a frozen teacher.

3. An efficient online-supervision implementation. Local teacher serving on dedicated GPUs, one-batch lookahead that overlaps teacher inference with student computation, fused full-vocabulary CE/FKL kernels, and coordinated checkpointing make the combined workload practical.

4. Evaluation and interactive deployment. Results span speech recognition and translation, audio reasoning, tool calling, service interaction, enterprise workflows, and text/image retention, and the model is served through a custom platform (Apriel Live) for interactive voice use.

Main Findings

• Headline thinking-mode results: 98.90 on Big Bench Audio (BBA), 81.13 on Speech-BFCL, and 65.51% task success on τ-Voice, with a mean latency of 1.91 seconds.
• Conversational service quality: English EVA accuracy reaches 51.27% and EVA experience pass@1 reaches 74.13%.
• Speech is projected, not transcribed: A frozen Whisper large-v3-turbo encoder produces 1280-dimensional features, which a learned projector maps into the decoder's 2816-dimensional embedding space; inference uses acoustic representations directly without an intermediate transcript.
• Small trainable interface: The projector has exactly 47,722,496 parameters, built from frame stacking of eight adjacent frames (10240 dimensions), a bias-free linear layer to 4096, a SwiGLU gate to 2048, and a second bias-free projection into the decoder space.
• Stage-one scale: 18.17M training examples across 1,164 dataset configurations grouped under 152 names, drawn from sources including Common Voice, GigaSpeech, People's Speech, LibriSpeech, and CoVoST 2; two epochs each covering 32.2k hours of audio, with 405,931 raw bins and 476,747 effective bins at 99.2% token fill, contributing roughly 1.9367B total tokens and 631.7583M loss-bearing tokens per epoch.
• Stage-two scale: 17.56M rows, 1.56M packed bins, 92.5% average token-fill efficiency, 17.5B tokens and 2.6B loss-bearing tokens per epoch, three completed epochs, and 52.6B total token exposure (17.5491B × 3 = 52.6473B), including approximately 7.88574B loss-bearing tokens.
• Audio exposure is reported separately from unique data: Stage 1 repeats 32,213 hours twice; Stage 2 repeats 61,540 hours three times. These are repeated training exposure, not unique data volume.
• Training infrastructure: Eight nodes with two teacher GPUs and six student GPUs each — 16 teacher and 48 student NVIDIA H100 80 GB HBM3 GPUs — completing the optimized run in approximately three days.
• Component freezing schedule: The speech encoder, the original vision tower and bridge, and the teacher are frozen in both stages; the projector is trainable in both stages; the language backbone and mixture-of-experts router are frozen in Stage 1 and trainable in Stage 2.
• The combined objective has a clean interpretation: On tokens supervised by both losses, CE plus FKL behaves as a scaled cross-entropy against a soft target that mixes the one-hot label with the teacher distribution, with mixing weight λ_d = β_d / (α_d + β_d) and overall scale (α_d + β_d).
• Retention is an empirical question, not a guarantee: Because the objective constrains behavior only on contexts seen in training, the paper states that retention on other inputs remains empirical — and the vision tower stays frozen while the decoder it feeds into changes, so visual retention must be measured.

The excerpt provided stops partway through Section 9, so full per-benchmark comparison tables against the baseline models are not included in the available content.

Methodology in Plain English

The team started from an existing post-trained multimodal model (Gemma 4 26B-A4B, a mixture-of-experts model with about 26 billion total parameters and about four billion active per token) that already understood text and images. They attached a speech front end: a frozen Whisper encoder turns audio into acoustic features, and a small trainable projector — following the frame-stacking and multilayer-perceptron design used by Ultravox — squeezes groups of eight frames together and maps them into the space the language model expects. Projected features simply replace reserved audio positions in the input sequence, so multiple audio spans can be interleaved with text.

Training happens in two phases. In Stage 1, only the projector learns, using multilingual transcription, translation, and continuation tasks; the language model stays frozen. In Stage 2, the projector and the backbone (including its mixture-of-experts router) are updated together on a broad mixture: mathematics, science, general conversation, speech tasks, noisy interactions, tool use, and text replay.

A key trick is that a frozen copy of the original model acts as a teacher during Stage 2. Each data source independently chooses plain supervised learning (cross-entropy on the target answer), distillation from the teacher (forward KL over the full vocabulary), or a weighted combination. For speech examples the teacher sees the corresponding text view, so the student learns to behave on audio the way the teacher behaves on text. Distillation is not limited to retention — it is used for modality adaptation too.

Conversational training data is built synthetically: the team first generates text dialogues with user requests, assistant replies, and tool calls and results, then an automated pipeline selects which user spans to speak, normalizes them into speakable form, synthesizes them with a single reference voice per conversation, and applies fidelity checks. Assistant targets and tool messages stay as text, preserving the supervision signal.

To make online distillation affordable, teacher inference runs on separate local GPUs, and the loop runs one batch ahead so teacher work for the next batch overlaps student forward/backward passes for the current one. Fused kernels compute the full-vocabulary normalization once per row and reuse it across vocabulary tiles, avoiding dense FP32 probability buffers, and workers drain pending teacher requests before checkpointing.

Decoding defaults are temperature 1.0, top-k 64, and top-p 0.95, with temperature 0.4 for recognition and translation benchmarks. Latency is converted to a 0–100 Responsive Score that is 100 at 0.25 seconds and 0 at 4.0 seconds, with linear interpolation in between.

Why This Matters

Impact on research. The paper demonstrates that an already post-trained, multimodal backbone can be given a speech interface without discarding its reasoning and tool-use behavior, and it documents the training recipe and infrastructure at a level of detail — bin counts, token exposure, GPU topology, loss coefficients — that lets others reproduce or critique the approach. It also reframes voice-model evaluation around task completion and interaction experience rather than word error rate alone.

Real-world applications:

• IT and account support: a user describes a locked account aloud, shares a screenshot, corrects an identifier, and asks the assistant to open a ticket — the model connects evidence across turns and issues the tool call.
• Enterprise stateful workflows: evaluated with EnterpriseOps-Gym, which separates strict task completion (all verifier checks pass) from partial progress.
• Multilingual voice access: the model is trained on translation tasks such as CoVoST 2 and on multilingual transcription data, supporting cross-language spoken requests.
• Document-heavy voice interaction: because images and text remain in context, spoken questions can be answered over screenshots, tables, charts, and OCR content.

Industry relevance. Speech synthesis is deliberately external and replaceable, so voices and deployment choices can change without retraining the reasoner; the EVA and τ-Voice evaluations in the paper use Cartesia Sonic 3.5 for synthesis. That separation, plus an open checkpoint and an interactive demo (Apriel Live), lowers the barrier for teams that want an in-house voice agent rather than a closed speech-to-speech API.

Future Directions

• Reducing latency and improving responsiveness. Mean τ-Voice latency is 1.91 seconds including synthesis; closing the gap toward the 0.25-second "excellent" threshold in the Responsive Score, and handling interruption and turn-taking more tightly, is a natural next step.
• Verifying capability retention more rigorously. The paper states that retention outside the training distribution is empirical, and that the frozen vision tower feeds a decoder that changed in Stage 2 — systematic measurement of text and image regression, and stronger retention constraints, follow directly.
• Scaling and broadening the data mixture. Continued expansion of speech-conditioned synthetic dialogues, more languages, and harder tool-use scenarios beyond the current mixtures.
• Moving beyond the "Preview" label. The title and the two-stage, teacher-supervised recipe invite questions about a final release: longer training, larger backbones, and whether the online-distillation pipeline can be simplified without losing quality.

Target Audience

Researchers and engineers working on speech-language models, multimodal LLM adaptation, or on-device/enterprise voice agents will get the most from this paper, particularly those interested in distillation-based modality transfer and in the systems engineering of online teacher supervision. Product and platform teams evaluating open alternatives to closed speech-to-speech APIs will find the deployment architecture and benchmark coverage useful. Readers without a background in LLM training will find the high-level framing accessible but the methods sections dense.

Authors’ abstract

A capable voice assistant must do more than recognize words; it must understand requests, reason over context, sustain conversations, and act through tools. We present Apriel Voice 27B Preview, an open speech-language model that unifies these capabilities within a shared reasoning backbone while retaining text and image input modalities. Built on the post-trained Gemma 4 26B-A4B checkpoint, Apriel Voice directly processes speech and generates both responses and tool calls without requiring a separate dialogue model. We introduce a two-stage training procedure to adapt the model to speech. The first stage establishes the speech interface, while the second jointly adapts the projector and backbone through supervised fine-tuning, text replay, and source-dependent cross-entropy and forward Kullback--Leibler divergence supervision. Automatically constructed speech-conditioned dialogues provide conversational training data, while local teacher serving, overlapping teacher and student computation, and fused full-vocabulary losses enable efficient online distillation. In thinking mode, Apriel Voice achieves a score of 98.90 on Big Bench Audio, 81.13 on Speech-BFCL, and a task success rate of 65.51% on $τ$-Voice, with a mean latency of 1.91 seconds. It also achieves 51.27% English EVA accuracy and 74.13% experience pass@1. Finally, we deploy Apriel Voice 27B Preview on a custom serving platform to support interactive voice interactions.