Almost Human, Except When It Matters: VoxParity and the Decisions a Voice Should Change

AI summary

Overview

Research area: Natural Language Processing, specifically voice agents that take typed actions (tool calls) on live calls, evaluating whether they act on what they hear as well as on what is said.

Technical level: Intermediate. The core idea (hold the words fixed, change only the audio, and check whether the executed action changes) is easy to grasp, but the paper leans on difference-in-differences testing, item-clustered bootstrap intervals, and Holm correction, and it separates confirmatory from exploratory results.

Scope: The paper introduces VoxParity, a 183-scenario, 14-sector benchmark with 28 systems and a human reference, and reports that every system tested errs toward the transcript when the audio calls for protection, with the gap concentrated on emotional delivery whose rule is left unstated.

What This Paper Is About

Voice agents can now hear a caller's audio rather than only a transcript, but nobody has measured whether the audio actually changes the typed actions they execute. VoxParity holds one caller transcript fixed while the sound of the call changes (a coaching second voice, a medical monitor beeping, a mayday under a radio check, noise over a drug name, a child's voice placing a bet, a frightened whisper) so that the correct tool call flips, and then tests each system against a pipeline that only reads the words. The goal is to find out how often, and on which kinds of calls, agents act on what they hear rather than on what was said.

Key Contributions

1. A test for acting on audio. The words-only null test requires no human judge and reaches different verdicts from accuracy and from cue recognition (Table 5), and its verdicts survive the robustness checks the paper describes in its Section 10.
2. A benchmark on the actions agents execute. 183 scenarios from 14 sectors, built on the sectors' own rules and practice, seven kinds of audible cue plus a channel control, 28 systems including nine production realtime agents, a judge-free scorer, and volunteer players choosing among the same actions on the same audio as a reference.
3. A failure map. The direction of errors on protective calls, where acting on audio does happen (items that state the rule), the split between acute alarm and quiet states, and the bridge from perception to policy, decomposed into a description of the voice and a stated rule.

Main Findings

• Errors run toward the words, for every system. All 28 systems, from 11 vendors and three serving modes, carry out the routine request on protective calls more often than they over-trigger on clean ones: 41% [36, 46] against 12% [9, 15] pooled, a median ratio of 4 to 1, and 34% against 18% for the four leading systems. The words-only cascade carries out the routine request on 58%; the volunteer players sit near the equal-rates line at 18% unsafe execution and 21% over-triggering.

• Hearing lowers unsafe execution but never reverses its direction. Against each system's own transcript (23 systems with a transcript path), hearing the audio lowers unsafe execution by 0.12 [0.09, 0.15] and leaves over-triggering unchanged (+0.00 [-0.01, +0.01]). Eighteen of the 28 execute unsafely less often than the words-only cascade after Holm correction (pooled -0.17 [-0.23, -0.12]); one, Qwen3-Omni-30B, does so more often (+0.12 [+0.04, +0.19]).

• Recognising a cue is not acting on it. All four leading systems identified the medical monitor beeping behind a disconnection request, and three scheduled the disconnection while none applied the vulnerability hold the rules require. Across the 27 systems with a readable perception probe, a protective cue identified in the system's own probe is acted on in 40% of life-safety calls and 40% of others (+0.00 [-0.11, +0.11]).

• Only 11 of 23 testable systems pass the null test. Systems from six of the eleven vendors pass. Neither OpenAI system that can take the test passes, and only one of the seven production realtime agents that can; no system without a transcript path beats the cascade. The median of all 28 systems scores the words-only cascade's own 0.34.

• Passes come almost entirely from items that state the rule. On stated-rule items the advantage over the null clears zero for 18 of the 23 systems, against 1 of 23 on items that leave the rule implicit (unadjusted, observational). Where the rule is stated, the four leading systems beat the null on emotional cells by +0.33 [+0.19, +0.49]; where it is not, they show no detectable gain (+0.00 [-0.11, +0.11]).

• Heard quiet states are overruled far more often than heard acute alarm. The four leading systems take the words' action on 6% [1%, 12%] of heard acute alarm (102 heard cells) and on 52% [40%, 64%] of heard quiet states (152). Heard facts sit between at 20% [12%, 29%], so quiet states exceed facts by 32 points [17, 46]. Over all 27 systems the excess is +0.17 [+0.04, +0.28], positive for 22 of them.

• The loss at the frontier is in deciding, not hearing. For the four leading systems, perfect hearing would add 0.04 credit and perfect deciding 0.28. With both a description of the voice and a stated rule, gemini-3.7-flash acts correctly on 0.83 [0.75, 0.90] of emotional calls whose rule was unstated (0.40 with neither), and with the description alone a text-only model reaches 0.76, above every unaided audio system (at most 0.57). Neither closes the gap on emotion.

• Accuracy and recognition disagree with the null test about who acts on audio. Accuracy disagrees with the null test on 4 of the 23 systems with a transcript path, and cue recognition, as scored, on at least 4 at any threshold. Five systems have no transcript path and are compared with the cascade on accuracy instead.

• The words-only null holds. The cascade's own audio-minus-twin change on cue-bearing cells is +0.01 [-0.02, +0.05], and replaying the same transcripts to two other text models leaves it at +0.02 and +0.03. A transcriber that adds emotion and sound-event tags rises just off the floor (+0.06 [+0.00, +0.11]).

• People hear the cues. Listeners who had not made the recordings chose the intended cue on 0.78 [0.72, 0.83] of 143 judgments, against a chance rate of 0.26 (0.50 on sarcastic readings), and players identified the cue on 0.82 [0.71, 0.89] of synthetic cue clips. Guess-corrected cue recognition is 0.77 for the leading systems against 0.79 for the volunteer players.

Methodology in Plain English

Each VoxParity item is a call frozen at the moment of decision: an agent role, sometimes a stated policy, a menu of typed tools, and one caller transcript. That transcript is rendered as two or more audio variants that differ only in how the words sound or in what else is audible, and each variant carries its own gold tool call with typed arguments, optional acceptable alternatives for partial credit, and a forced-choice question about what is audible. The scenario schema requires the gold to flip across variants, and it does flip in 130 of the 132 items with two runnable variants. Tool menus are shuffled per item with a fixed seed, so a policy of always picking the second tool, which would score 0.977 on unshuffled menus, gains nothing.

Scoring is deterministic, following the syntax-tree matching of the Berkeley Function Calling Leaderboard: the tool name must match, each typed argument must fall in its admissible list, and free text must match after normalisation. Credit is 1 for the gold call, the stated partial value for an acceptable alternative, and 0 otherwise. A separate selection credit scores the tool choice alone and is used whenever players are compared, since players typed no arguments.

The central test has no human judge. Every variant has a text twin, in which the same system reads the exact transcript as text, so audio credit minus twin credit measures what hearing bought over reading. A words-only cascade (Whisper-large-v3-turbo, then gpt-oss-120b choosing the tool) runs every cell; because its language model never hears the audio, its own audio-minus-twin change measures only what transcription moves. The words-only null test takes a system's audio-minus-twin change minus the cascade's, a difference-in-differences on identical cells that carry a cue; a system passes when it is above zero after Holm correction. Twenty-eight audio-native systems from 11 vendors ran through one pipeline and one scorer on the frozen bank between 15 and 25 September 2026, in three serving modes: 13 file-mode API models, 9 production realtime agents from Google, OpenAI, xAI and Alibaba, and 6 open-weights models run locally on one Apple-silicon Mac. Amazon's Nova 2 Sonic was not run.

People made the same decisions in a browser game: the same clips, scenarios, shuffled menus and standing actions, with an action locked in before the perception question. The game did not show the stated policy and told players that the caller's voice decides the move. The authors report 638 answers from 34 sessions on 27 browsers (about 20 unpaid volunteers) on 381 cells from 144 items. Stimuli are synthetic, with Gemini-TTS as the primary engine and Kokoro-82M and Qwen3-TTS as second-engine copies; the author recorded 49 variants on 26 items by hand. Statistical comparisons cluster by item and pair on identical cells, with 95% item-clustered percentile bootstrap intervals, p-values inverted from the bootstrap and floored at 1/4,000, and Holm correction within each family of tests. Player comparisons resolve differences of about 0.19 at 80% power, enough to separate people from the words-only cascade but not to rank the leading systems against people.

Only the null-test verdicts are described as confirmatory; the other analyses were chosen after looking at the data and are labelled descriptive or exploratory.

Why This Matters

Impact on research. Prior work had hinted at a gap between hearing a cue and acting on it: three scenarios where four production realtime agents acted on the caller's words while most recognised the cue, adding audio to the transcript barely changing the decisions of open 7B audio models (Hear2Act), and probing showing that models encode more than they use. VoxParity turns those observations into a measurement against a null that needs no judge, and its verdicts diverge from accuracy and from cue recognition, which means the measures used today can credit the wrong systems. It also shows that the bridge from perception to policy, not perception itself, carries most of the loss at the frontier.

Real-world applications (each grounded in the sector rules the paper cites):

• Emergency and public-safety dispatch, where NENA-STA-020.1 requires a response when background sounds indicate danger, and where the paper shows 21 of 28 systems taking the words' default on a silent 911 call with a man shouting a threat in the background.
• Fraud and collections, where FinCEN FIN-2022-A002 treats a second voice coaching the caller as a red flag, and where a sobbing caller saying "That's everything I have" is overruled by 5 of 27 systems that heard the state.
• Utilities and home services under Ofgem SLC, where a medical monitor beeping behind a disconnection request should trigger a vulnerability hold but 20 of 28 systems scheduled the disconnection instead.
• Retail and telephone betting, where UKGC LCCP 3.2.11 requires age verification before a customer gambles, and a child's voice on a bet is one of the cue types tested.
• Healthcare, where noise over a drug name should prompt a repeat but the median system scores 0.24 on healthcare and telecom cue-bearing cells.

Industry relevance. Speech vendors that report caller "sentiment" document it as computed from the transcript, and this benchmark measures the typed actions agents execute rather than a sentiment label. Systems that look safe because they clarify on most protective calls can hide confident word-following, and risk and accuracy rank systems differently (ρ = 0.48). Cue recognition thresholds are not a substitute for the null test: a threshold of 0.5 credits 13 systems, 8 of whose verdicts disagree with the test.

Future Directions

• Closing the emotion gap. Describing the voice and stating the rule each recover part of the shortfall, but neither closes it: on items that state the rule, gemini-3.7-flash reaches 0.71-0.76 of emotional cells against 0.95-0.97 of other cues, and with a description and a rule it still trails other cues by 0.17 [0.10, 0.25] on calls whose rule was implicit.
• Fixing the perception-to-policy bridge. Perfect hearing would add only 0.04 credit while perfect deciding would add 0.28, and the highest-credit system's own pre-action notes understate the state it heard in 30 coded cases, 22 of which its forced-choice probe had identified correctly. What makes a system act on a state it has already named is unresolved.
• Understanding why quiet states lose to acute alarm. The contrast holds in the pooled

Authors’ abstract

A voice agent can handle almost every call on the words alone and still fail the few its sector's rules were written for. Emergency-call standards, fraud guidance, radio phraseology and vulnerability rules recognise that how a caller sounds, or what else is audible, can change the right action. VoxParity tests whether agents act on it. In 183 scenarios from 14 sectors, one transcript stays fixed while the audio changes (a coaching voice, a medical monitor beeping, a mayday under a radio check, noise over a drug name, a child's voice placing a bet, a frightened whisper), and with it the correct typed tool call. A words-only null test credits a system only if hearing the call moves its actions more than it moves a pipeline that only reads the words. Only 11 of the 23 systems that can also be run on the transcript pass. Descriptively, errors run toward the words: when the audio calls for protection, all 28 systems carry out the routine request more often than they over-react on clean calls (41% against 12% pooled; the words-only pipeline, 58% against 15%). Exploratory analyses place most of the leading systems' misses on cues they heard; systems beat the null almost entirely on items that state the rule; the leading systems overrule heard resignation or confusion far more often than acute alarm; and, in the models tested, describing the voice and stating the rule each recover part of the shortfall, leaving a gap on emotion.