Same Words, Different Judgments: Modality Effects on Preference Alignment

AI summary

Overview

Research area: Human preference annotation and preference-based reinforcement learning (PbRL) / RLHF, with a focus on comparing text vs. speech (audio) modalities. Published under cs.SD (sound), arXiv:2602.22710v2.

Technical level: Intermediate. The paper uses standard psychometric and statistical machinery (intraclass correlation coefficients, Krippendorff's alpha, linear mixed-effects models, permutation tests), but the framing and conclusions are accessible to anyone familiar with preference data collection.

One-sentence scope: A controlled cross-modal experiment in which 94 human raters plus synthetic AI raters evaluated 100 identical prompts as either text or TTS-generated audio, measuring inter-rater reliability, decision thresholds, length and order biases, and cross-modality agreement.

What This Paper Is About

Preference-based reinforcement learning is the dominant way AI systems are aligned to human preferences, but its evaluation protocols were designed for text and have never been validated for speech. The authors ask whether the same words, presented as audio rather than text, produce the same preference judgments—and whether existing text-based annotation recipes can simply be carried over to speech. To answer this, they built a matched text-vs-audio experiment over 100 identical semantic prompts and characterized how reliable, how biased, and how cross-compatible the resulting annotations are.

Key Contributions

1. A cross-modal evaluation protocol for preference elicitation, consisting of continuous 1–100 visual analog scale (VAS) ratings with hidden numeric values, counterbalanced sequential audio presentation, and attention checks adapted for audio (as well as a text comparison condition matched on layout and instruments).

2. The first ICC-based characterization of preference-annotation reliability in either modality, showing how agreement scales with rater count: ICC(2,1) is poor at one rater (.333 audio, .295 text), moderate at three raters, and good with approximately 9 raters (ICC(2,k) = .821 audio, .788 text), with diminishing returns beyond that.

3. An empirical characterization of how preference judgments shift across modalities, including narrower decision thresholds and reduced length bias in audio, near-chance cross-modality agreement, and prompt-specific rather than uniform preference shifts.

4. A released corpus of 3,113 TTS-converted conversations from PRISM, plus code and data, to support future audio preference research (hosted at the linked Hugging Face dataset page, NeurIPS-Anon-2784/modality-prefs-data).

Main Findings

• Audio and text are comparably reliable in aggregate. No significant modality difference was observed between ICC values (ICC(2,1) = .333 audio vs. .295 text, p = .127; ICC(2,k) = .821 audio vs. .788 text, p = .113). Average raters per stimulus was 9.2 (audio) and 8.9 (text). Krippendorff's alpha also showed no significant differences across continuous (.315 vs. .228, p = .069), ordinal (.157 vs. .217, p = .119), and nominal (.010 vs. .031, p = .437) computations.

• Roughly nine raters are needed for good reliability. Session/annotation cost scales accordingly: moderate reliability requires a 3x increase in annotation cost and good agreement a 9x increase. The paper notes that prior work reported only single-rater pairwise agreement (73–77% for text summarization, 63% for helpful and harmless annotations, 60% for music) and did not characterize how agreement scales with rater count.

• Audio raters use narrower decision thresholds. When committing to a winner, text raters showed larger average rating differences (M = 41.7, 95% CI [39.5, 43.9], Mdn = 34.0) than audio raters (M = 27.9, 95% CI [26.1, 29.8], Mdn = 20.0; Mann–Whitney U = 150582, p < .001, r_rb = .303). Tie magnitudes differed statistically but were small (audio M = 3.9; text M = 3.3; U = 34092, p = .022, r_rb = -.113).

• Modality moderated length bias. Longer responses predicted higher ratings in both modalities, but more strongly for text (text: b = 4.92, p < .001; audio: b = 3.52, p = .004; interaction b = 1.53, 95% CI [0.03, 3.02], p = .045). The authors describe this as a roughly 40% larger rating increase for longer text responses.

• A recency bias emerged, not specific to audio. Second-presented items were rated more favorably (b = -2.61, 95% CI [-4.65, -0.58], p = .012), with no modality difference (b = -0.51, p = .732). The authors frame this as a general property of sequential presentation.

• Ratings stayed stable across trials while completion times fell. Ratings showed no trial-number drift (b = 0.03, p = .960) and no modality-specific drift (b = 0.90, p = .236), while trial duration decreased (audio b = -0.09, p < .001, about 8.6% reduction per SD of trial number; text b = -0.17, p < .001, about 16.0%; text sped up more, b = -0.10, p < .001).

• Cross-modality agreement is near chance at a zero threshold. Audio and text agreed on the winner for 53% of prompts (53/100; binomial vs. 50%, p = .31). Agreement rose with stricter decisiveness thresholds while the number of decisive pairs dropped.

• Modality effects are prompt-specific, not a uniform shift. The fixed effect of modality on per-trial preference scores was not significant (beta = 1.71, p = .458), but adding random slopes for modality by prompt significantly improved model fit (chi-squared(2) = 22.1, p < .001; random slope SD = 12.93). The authors describe modality as able to swing preferences by nearly half a standard deviation in either direction depending on the prompt.

• Audio evaluation appears more user-oriented. A TF-IDF analysis of written justifications found audio raters more frequently referenced "user" and "help," while text raters emphasized "detail" and "response."

• Audio sessions take longer without feeling harder. Audio trials took nearly twice as long as text trials (M = 136 vs. 79 sec; U = 255.0, p < .001), and sessions were longer overall (audio 47.3 min vs. text 27.0 min, p < .001), yet participants rated both tasks as similarly difficult (M = 2.50 vs. 2.45 on a 5-point scale, p = .868).

• Synthetic ratings predict human inter-rater agreement. For each prompt, the authors regressed human ICC(2,1) on the absolute AI rating difference across the two responses (N = 100). AI differentiation was a significant predictor (B = 0.014, p < .001), with no modality moderation (B = -0.003, p = .15). Spearman correlations: audio r = .23, p = .02; text r = .38, p < .001.

• AI-human rating error was smaller for audio than text. Audio MAE was 12.27 and text MAE 14.29 on the 1–100 scale (modality difference from the LMM = 2.02, p = .035).

• Human audio ratings aligned most closely with the original PRISM ratings. Comparing original PRISM ratings against external groups at the clip level, the signed difference was smallest for human-audio (mean signed difference = 1.15, p = .374) relative to AI-text (-1.84, p = .015), AI-audio (-9.18, p < .001), and human-text (8.03, p < .001). The authors note the non-significant human-audio result precludes strong claims.

• Audio-quality perception did not explain the main effects. In an audio-only model (n = 1,840 observations, 46 participants, 200 stimuli), perceived audio quality was a strong positive predictor of response ratings (b = 7.77, 95% CI [6.51, 9.03], p < 10^-31), yet recency remained significant (b = -2.28, p = .015, about 14% attenuation), the length effect was essentially unchanged (b = 3.27, p = .004), and the trial-number effect remained null (b = 0.28, p = .57).

• Voluntary re-listening occurred in a minority of audio trials. 214/920 audio trials (23.3%) involved voluntary re-listens to the first clip after hearing the second.

Methodology in Plain English

The authors started from PRISM, a preference dataset of 8,011 live conversations between 1,500 participants and 21 LLMs, where each response is rated on a 1–100 VAS. They sub-selected the 3,113 "unguided" conversations (chosen because other conversations were intended to be controversial) and randomly drew 100 interactions. To guarantee variability in response length, 25 of the 100 interactions came from the top 10th percentile of absolute character-length differences between responses (a minimum difference of 449 characters), and the remaining 75 came from the lower 90th percentile. This yielded 200 audio clips (2 model responses x 100 interactions), screened so only innocuous content was shown.

The 100 interactions were converted to speech with Kokoro, a state-of-the-art open-source TTS model benchmarked by TTS-Arena, using default parameters and a single voice (af_heart) for consistency. All clips were manually reviewed for artifacts or mispronunciations, and low-quality samples were replaced and re-evaluated.

They recruited 106 participants (53 per modality) from Prolific's US population at $12/hour. Participants completed a demographic questionnaire, then rated 20 interactions in balanced block random order plus one attention check (21 total). Raters were randomly assigned to text or audio and could not switch conditions. Both conditions used identical interfaces and 1–100 sliders with hidden numeric values to avoid anchoring; audio raters also rated audio quality. Presentation was sequential in both conditions, with audio requiring full playback on first listen (replays allowed afterward) so order effects could be explicitly compared. Afterwards, raters gave a binary preference with a tie option and an optional free-text justification.

Attention-check failures (7/53, 13.2% audio; 5/53, 9.4% text) and non-completers were excluded, leaving 94 participants and 8–10 ratings per audio clip. In addition to explicit ratings, the study logged decision time, audio deliberation time, replay counts, intermediate rating changes, session metadata, trial order, and stimulus properties (history character count and turns, response character length, absolute length differences).

For the synthetic comparison, they collected 10 AI ratings per prompt per modality on the 1–100 scale using GPT-4o for text and GPT-4o-Audio-Preview for audio with default API parameters, choosing an end-to-end audio model rather than an STT+LLM+TTS pipeline that would only evaluate transcription/synthesis accuracy.

Statistically, reliability was estimated with ICC(2,1) and ICC(2,k) from linear mixed-effects models with crossed random intercepts for stimuli and participants, order/trial position as covariates, and 95% CIs from 2,000 bootstrap resamples; modality differences were tested by permutation (2,000 iterations). Krippendorff's alpha was computed without covariate adjustment using 5,000 bootstrap resamples and 5,000 permutation iterations.

Why This Matters

Impact on research: The paper argues that text-derived TTS preference data cannot be assumed to generalize to speech and that modality should be treated as a first-class consideration in preference-based RL pipelines. It is the first work to characterize how many raters are needed to reach specific reliability thresholds in preference annotation, and the first to report a relationship between AI discriminability and human inter-rater reliability. It also warns that binarization thresholds calibrated on text may misclassify audio pairs as ties or reverse preferences, inflating label noise for reward model training—so thresholds should be calibrated per modality, or continuous signals retained.

Real-world applications:

• Designing crowdsourced annotation pipelines for speech models, with modality-specific rater counts and session lengths (the paper suggests sessions up to one hour are viable for both modalities).
• Using synthetic ratings as a pre-screening signal to triage which prompts need many human raters and which need few, reducing annotation cost.
• Debaising text-trained reward models by using audio-derived labels as a complementary signal, at the cost of roughly 2x longer annotation per trial.
• Informing evaluation design for TTS and speech assistants where naturalness, emotional expressiveness, and conversational appropriateness are the qualities being optimized.

Industry relevance: Any organization collecting human preference data for speech or multimodal assistants and feeding it into RLHF-style pipelines faces the practical questions this paper quantifies: how many annotators to hire, how to present audio sequentially, whether to keep continuous ratings or binarize, and how much synthetic labels can substitute for human ones. The released 3,113-conversation audio corpus and code lower the barrier to running audio preference studies.

Future Directions

• Characterizing naturalistic speech. The findings cover TTS-rendered speech from text originally written for reading, with a single static voice profile and innocuous content only. Future work should test paralinguistic features (emotional prosody, hesitation, background noise), speaker characteristics (identity, gender, accent, pitch, culture), and non-innocuous or toxic semantic content.

• Broadening the rater pool. The sample was US-based, predominantly White, and predominantly native-English-speaking. Preference differences between non-Western populations and non-native English speakers are largely unexplored, and the authors recommend demographically broader rater pools for downstream deployment.

• Validating other models and identifying the degradation boundary. The synthetic-rating results apply to a single model family (GPT-4o and GPT-4o-Audio-Preview). The authors suggest future work could use their ICC data to validate other LLMs, and should identify when longer sessions begin to degrade data quality.

• Investigating why audio and AI align more tightly. The 2-point AI-human MAE gap favoring audio warrants further study into whether end-to-end audio models capture signals unavailable to text pipelines. The authors also frame the audio-label debiasing trade-off (roughly 2x longer annotation per trial) as an open question.

Target Audience

Researchers and practitioners working on preference-based reinforcement learning, RLHF, and RLAIF who collect human preference data—especially those extending text alignment methods to speech. It is also relevant to speech and TTS evaluation researchers, crowdsourcing and annotation-ops teams designing rater protocols, and psychometricians or HCI researchers interested in how presentation modality shapes judgment, decision thresholds, and inter-rater agreement. Readers should be comfortable interpreting ICC values, mixed-effects model output, and permutation tests.

Authors’ abstract

Preference-based reinforcement learning (PbRL) is the dominant framework for aligning AI systems to human preferences, but its application to speech remains underexplored. We present a controlled cross-modal study of human and synthetic preference annotations, comparing text and audio evaluations of identical semantic content across 100 prompts. Audio preferences prove as reliable as text, with inter-rater agreement reaching good levels (ICC(2,k) $\approx$ .80) at $\sim$9 raters -- the first ICC-based reliability characterization in the preference annotation literature for either modality. However, modality reshapes how people judge: audio raters exhibit narrower decision thresholds, reduced length bias, and more user-oriented evaluation criteria, with near-chance cross-modality agreement. Synthetic ratings further align with human judgments and predict inter-rater agreement, supporting their use both for triaging ambiguous pairs and as full replacements for human annotations.