Do Audio LLMs Listen Before They Act? Diagnosing Acoustic-Context Gating in Voice Agents

AI summary

Overview

Research area: Spoken language processing and voice agents — specifically how audio language models (Audio LLMs) decide whether to act on spoken input, and how to evaluate that decision.

Technical level: Intermediate. The paper is readable without deep audio-signal-processing background, but assumes familiarity with tool-calling agents, supervised fine-tuning (SFT), and GRPO-style reinforcement learning.

One-sentence scope: The paper introduces VGBench, a 1,018-item diagnostic benchmark that measures whether Audio LLMs condition an executable action (mute, tool call, or answer) on acoustic and conversational context, and uses a post-training system called VoxGate as a case study of whether that behavior can be learned.

What This Paper Is About

Audio LLMs can recognize a spoken command and pick the right tool, but that does not mean they correctly decide whether the utterance was addressed to them at all. A bystander can say the same words as the user, and a user can mention a command while thinking aloud. The paper builds a benchmark that holds the specified words fixed while changing who spoke them, how far away they were, and where the turn boundary falls, then asks whether the model still executes the action or stays silent. It then tests whether that gating behavior can be trained.

Key Contributions

1. An action-level addressedness benchmark (VGBench). 1,018 items — 395 side-talk recordings, 223 self-talk recordings, and 400 paired speaker-switch cases — all mapped to one shared action space of [Mute], a tool call, or a natural-language answer. Speaker-switch pairs hold specified text fixed while jointly changing source, distance rendering, and a 600 ms boundary.

2. A behavioral diagnosis of current systems. Six raw Audio LLMs and three training-free adaptations were evaluated on the full corpus under one standardized action contract. The best raw switch mute rate is 14% and the best training-free result is 6%, even when target-tool selection is high.

3. A post-training case study (VoxGate). Supervised training on the VGBench training partition plus WearVox answer, abstention, tool-use, and translation examples mutes 91.3% of switched commands while selecting the correct tool for all held-out near-field same-speaker and text-only controls.

4. Factorized and cue-balanced controls. Separate manipulations of trigger source, distance rendering, and the 600 ms gap isolate which cue drives the decision, and an equal-budget cue-balanced SFT run tests how sensitive that gate is to acoustic scene.

Main Findings

• Tool recognition does not imply acoustic-context gating. Step-Audio-R1.1 selects the target tool on 96% and 97% of same-speaker and text-only controls respectively, yet mutes only 1% of switched commands. Kimi-Audio-7B has the highest raw switch mute rate at 14%, but selects the target tool on only 53% and 44% of the two controls.

• Systems fail through different shortcuts. Qwen3-Omni preferentially acts on the first side-talk utterance, Gemini-3.8-flash favors the later one, and Step-Audio-R1.1 often responds to both. Audio Flamingo 3 receives zero canonical tool-selection credit because it emits a different bracketed action language; that format mismatch does not explain its 1% self-talk mute rate or its zero switch mute rate.

• Training-free adaptations change the error profile rather than solving it. The Qwen + AURA adaptation raises side-talk from 39.0% to 46.6% and self-talk from 57.0% to 63.2%, but mutes no switched commands. Qwen + TwS raises self-talk muting to 85.7% while side-talk falls to 23.0%, largely through overuse of silence. Qwen + SHANKS raises self-talk muting but leaves 1% same-speaker tool selection through action-grammar violations.

• Supervised training learns most of the benchmark gate. VoxGate supervised training reaches 91.3% switch mute, 100.0% same-speaker target-tool selection, 100.0% text-only target-tool selection, 68.4% side-talk accuracy, and 52.0% self-talk mute on the held-out 20% split; the base Qwen model reaches 0.0%, 81.3%, 87.5%, 53.2%, and 50.0% respectively. In counts, SFT mutes 73 of 80 held-out switched commands.

• GRPO adds small descriptive gains in one run. Switch mute rises from 91.3% to 92.5% (74 of 80), self-talk muting from 52.0% to 60.0%, and side-talk accuracy from 68.4% to 70.9%. The paper states that these differences do not isolate the effect of pair-aware grouping, and that there is no unpaired-GRPO or repeated-seed control.

• The aggregate switch score combines source and distance cues. At fixed near-field rendering and a 600 ms gap, changing only the trigger source raises mute rate from 1.25% to 50.0% for SFT (48.75 points) and to 53.75% for GRPO (52.50 points). At fixed source and gap, far-field rendering raises muting to 60.0% for both models (58.75 points). The gap alone changes same-speaker near-field muting by only 1.25 points.

• Cue-balanced SFT improves near-bystander suppression but is scene dependent. On the original validity WAVs it raises same/far/600 muting from 60.00% to 90.00% and different/near muting from 35.00%/50.00% to 66.25%/82.50% at 0/600 ms. On scene-consistent eight-condition audio, target-action accuracy under the proximity rule rises from 39.22% to 66.88%, but same/far muting remains only 3.75%/5.00%. Original speaker-switch muting falls from 73/80 to 70/80, and WearVox tool-call accuracy falls from 90.18% to 83.04%.

• Downstream tasks are retained, not erased. On the fixed 384-example WearVox protocol, base Qwen scores 45.57% overall, task-only SFT 63.28%, task-only SFT + GRPO 67.71%, joint VoxGate supervised training 72.14%, and joint VoxGate + GRPO 76.30%. Joint GRPO reaches 100.00% abstention accuracy, 92.00% tool use, and 89.70% live translation.

• The evidence has stated boundaries. Results use one split and one run per setting. The side-talk free-response subset uses a fixed Qwen3.5-35B-A3B judge with no reported human-agreement estimate, and VGBench tool accuracy checks tool names rather than arguments.

Methodology in Plain English

The researchers built a controlled test set rather than collecting data in the wild, so that the words stay the same while the situation changes.

Three scenario families probe different cues:

• Side-talk (395 recordings): each recording has two consecutive utterances from the same stored speaker, one directed to the assistant and one to a nearby person, with balanced order. Because one voice is used, speaker identity cannot explain the correct answer; the model must use pragmatic attribution. Tool-like and conversational content are balanced so that utterance position or an obvious command is not a reliable shortcut.
• Self-talk (223 recordings): command-like monologues framed as planning, regret, quotation, sarcasm, or rhetorical questions. Every item targets [Mute] even when the words map to an available tool.
• Speaker-switch (400 counterfactual pairs across ten consequential tools, 40 cases per tool; 100 seed cases and 300 generated cases using eight distinct voices): a conservative wearable authorization rule is adopted — a tool is authorized only for the same authorized source in the near field; a different source or a far-field trigger targets [Mute]. In the same-speaker condition, source A speaks both context and near-field trigger. In the switch condition, source A speaks the context, 600 ms of silence is inserted, and a bystander source B speaks the same trigger under fixed far-field room rendering. A text-only condition keeps the tool target.

Audio is converted to 16 kHz mono PCM WAV, normalized to −20 dBFS per segment with 5 ms fades, and retained only after annotation, target-label verification, and script-audio consistency checks.

All systems see the same tool inventory and must emit exactly one of three action forms: the literal token [Mute], a single structured tool object, or natural-language text. Decoding is greedy. In the post-training phase, roughly 80% of items within each scenario are used for training (including 320/80 speaker-switch pairs), and all reported addressedness scores come from the disjoint held-out partition.

The training case study, VoxGate, fine-tunes a LoRA adapter (rank 32, alpha 64) on the attention q, k, v, and o projections across 48 layers of Qwen3-Omni-30B-A3B-Instruct, with the audio encoder and aligner frozen, for three epochs at learning rate 10⁻⁴, effective batch size 32, on eight H200 GPUs. The mixture combines the VGBench training partition with WearVox answer, abstention, tool-use, and translation examples. The GRPO stage samples K/2 completions from each side of a speaker-switch pair into one rollout group, so reward normalization depends on whether the policy separates the counterfactual pair; correct mute or matching tool earns +1, an incorrect action −1, malformed tool syntax an extra −0.2, and answer targets earn +0.5 for nonempty natural language, −1 for mute, and −0.5 for empty or tool-only output. Only groups with nonzero reward variance contribute an update.

Why This Matters

Impact on research. The paper reframes addressedness from a separate front-end detection score into an executable action decision, and it shows that a high tool-selection score can coexist with almost no gating (Step-Audio-R1.1: 96%/97% tool selection, 1% switch mute). It supplies a paired positive/negative evaluation design that

Authors’ abstract

Audio language models can recognize spoken commands and invoke tools, but an agent must first decide whether the acoustic and conversational context warrants action. We introduce VGBench, a 1,018-item diagnostic benchmark for action-level addressedness across side-talk, self-talk, and speaker-switch scenarios. Each item uses a shared action space comprising silence, a tool call, and a natural-language answer. Speaker-switch pairs hold the specified words fixed while source, distance rendering, and a temporal boundary define a controlled wearer-to-bystander shift. Six raw Audio LLMs and three training-free adaptations often identify the target tool yet rarely withhold action under this shift; the highest raw switch mute rate is 14%. We then use VoxGate as a post-training case study. Supervised training mutes 91.3% of switched commands while choosing the correct tool for all nearby wearer commands and text-only controls. An exploratory GRPO stage has similar switch performance; side-talk accuracy rises from 68.4% to 70.9%, and self-talk muting from 52.0% to 60.0%. Factorized controls identify an independent source-change effect, while sensitivity to the far-field manipulation varies across acoustic renderings. The benchmark therefore measures multi-cue acoustic-context gating rather than isolated speaker identity.