CASA: Content-Acoustic Speaking Assessment with Speech Encoder and Large Language Model

AI summary

Overview

Research area: Automatic speaking assessment (ASA) — the automatic scoring of second-language (L2) learners' oral proficiency — combining speech recognition and speech-language models. This sits at the intersection of natural language processing, speech processing, and educational assessment.

Technical level: Intermediate. The paper assumes familiarity with encoder–decoder speech models, LoRA adapters, and the CEFR proficiency scale.

Scope: The paper introduces CASA, a two-branch architecture pairing a Whisper-medium speech encoder with the Qwen3.5-2B language model, evaluates it on the Speak & Improve Corpus 2025, and analyzes how acoustic versus content information drives predictions.

What This Paper Is About

Automatic speaking assessment systems must judge both how a learner speaks (delivery, fluency, pronunciation) and what they say (content, vocabulary, task relevance). Recent speech-LLM systems achieve strong scores but rely on large multimodal backbones and offer limited analysis of which kind of information drives their predictions. The paper's goal is to build a smaller, more interpretable, general-purpose architecture that keeps acoustic and content evidence explicitly separated, reaches state-of-the-art accuracy, and uses far fewer parameters at inference.

Key Contributions

1. A compact two-branch architecture (CASA). CASA combines a frozen Whisper-medium encoder adapted with LoRA for the acoustic branch and a frozen Qwen3.5-2B adapted with a separate LoRA for the content branch, using only three handcrafted fluency features (duration, silence ratio, speech rate).

2. State-of-the-art-level accuracy with roughly half the inference parameters. CASA reaches an RMSE of 0.358 on the Speak & Improve test set, marginally below the previous best of 0.360, while using an estimated 3.13 B parameters versus 6.24 B for the prior speech-LLM system. The authors explicitly state they do not claim a meaningful accuracy improvement.

3. A systematic ablation study of acoustic and content contributions. Through repeated runs and ablations, the paper quantifies performance variability, tests the auxiliary loss, alternate learning rates, alternate acoustic encoders (CrisperWhisper, WavLM, wav2vec2 XLS-R 300M), and larger models (Qwen3.5-4B, Whisper-large-v3).

4. Training-free content validation with the LLM branch. With no task-specific training, the Qwen3.5-2B branch can judge whether an answer addresses a given question, flagging 99.9% of responses when questions are replaced with an unrelated question about nuclear reactors and 97.3% when responses are paired with real questions from different parts.

Main Findings

• Overall accuracy: CASA achieves RMSE 0.358, PCC 0.829, 84.7% of scores within 0.5, and 98.7% within 1.0 on the S&I test set. NTNU scores RMSE 0.360, PCC 0.827, 85.7% within 0.5, 99.0% within 1.0. Perezoso scores RMSE 0.364, PCC 0.826, 83.0% within 0.5, 99.7% within 1.0. CASA-Crisper scores RMSE 0.363, PCC 0.836, 84.0% within 0.5, 99.7% within 1.0.

• Parameter efficiency: CASA's total parameter count is 3.13 B, compared with an estimated 6.24 B for NTNU and 2.17 B for Perezoso. One Whisper is the most compact system at 0.17 B, with a higher RMSE of 0.372. NTNU and Perezoso counts are estimated from described architectures because no code was released.

• The auxiliary acoustic loss helps consistently. Averaged over 10 runs, disabling the auxiliary loss (aux-0) raises mean RMSE from 0.363 to 0.367 — the auxiliary head reduces mean RMSE by 0.004. Blocking either the auxiliary-loss gradient or the soft-token-path gradient from reaching the acoustic encoder degraded performance in two additional single-run ablations, suggesting the two paths supply complementary supervision.

• Results are smaller than run-to-run noise. Over 10 runs, CASA has a mean RMSE of 0.363, median 0.362, range 0.357–0.377, and a 95% CI of [0.359, 0.367]. The reported 0.358 came from the first run of the configuration, selected on the development set. Even runs with the same seed ranged from 0.357 to 0.365, indicating nondeterminism in the training pipeline; one distinct seed (6066) produced a poorer run.

• Higher learning rates are unstable, not better. Doubling the acoustic-branch LoRA learning rate to 4e-4 produced an RMSE of 0.350 on the first run, but subsequent runs did not reproduce it: mean 0.378, median 0.376, range 0.350–0.402, 95% CI [0.364, 0.392].

• Verbatim transcription shifts errors between proficiency bands. CASA-Crisper improves on the A2 band (0.553 → 0.485) and on B1 (0.351 → 0.335), but degrades B2 (0.290 → 0.322) and C1 (0.554 → 0.617). Macro RMSE is 0.437 for CASA and 0.440 for CASA-Crisper. The authors suggest verbatim transcripts help catch content errors of A2 and B1 speakers — relevant because A2 is under-represented — but may inject inauthentic errors that obscure the range marking the top band.

• Performance is not capacity-limited. Neither a larger acoustic encoder (CrisperWhisper) nor a larger LLM improves RMSE: Qwen3.5-4B yields 0.364, and Whisper-large-v3 paired with the 4B LLM yields only 0.362. Larger models may require different hyperparameters.

• Self-supervised encoders underperform. Swapping the acoustic encoder for WavLM or wav2vec2 XLS-R 300M (keeping CASA's Whisper ASR transcript) substantially underperforms CASA, and an English-finetuned variant provides no improvement. The authors attribute this partly to architectural mismatch with CASA's Whisper-oriented aggregator and training recipe.

• Task-dependent difficulty. Per-part RMSEs are 0.476 (P1), 0.454 (P3), 0.490 (P4), and 0.444 (P5). Performance is weaker on P1 and P4, consistent with Lin et al. A qualitative transcript analysis, conducted with a co-author in applied linguistics, suggests the short personal questions in P1 and the process descriptions in P4 offer less freedom to display broad proficiency than the opinion-based tasks in P3 and P5.

• Task embeddings were ineffective. The zero-initialized task embeddings in the Transformer block remained close to their zero initialization, contributing little task information to the acoustic representation.

Methodology in Plain English

The system scores each part of the exam separately with one shared model, then averages the per-part predictions. There are two streams of information.

The acoustic stream starts with a frozen Whisper-medium encoder, lightly adapted with LoRA adapters so it can specialize for assessment while still producing ASR transcripts. Each answer is cut into 30-second chunks up to a 2-minute cap; each chunk yields 1,500 frame-level vectors. Instead of averaging everything into one vector — which the authors say discards acoustic detail — the chunk vectors are concatenated and adjacent frame pairs are averaged, halving the sequence length and changing the time resolution from 20 ms to 40 ms. Learned task and segment embeddings are added to every frame so the model can tell which exam part and which answer a frame came from. The frames then pass through a two-layer Transformer encoder with rotary position embeddings, and a [CLS] token summarizes the sequence. That summary feeds an MLP projector producing four "acoustic soft tokens" for the language model, plus a separate linear auxiliary head that predicts a scalar CEFR estimate. That estimate is detached and written into the prompt as text (for example, acoustic_cefr_estimate: 4.5).

The content stream uses transcripts produced offline by the frozen Whisper encoder–decoder, paired with the task prompt in a <TASK part=P1> format followed by question–answer pairs. Qwen3.5-2B receives, in order: the four acoustic soft tokens, the detached acoustic CEFR estimate as text, the scoring rubric, the question–answer pairs, and three fluency statistics derived from audio and ASR output (duration, silence ratio, speech rate in words per second). The model runs a single forward pass without generating text, and a linear regression head on the final-token representation predicts the part score.

Training combines a main MSE loss on the fused prediction with an auxiliary loss on the acoustic-only prediction, weighted 0.1 and using a tolerance of ±1 score point, so that predictions within that margin incur no auxiliary penalty. This keeps an inherently ambiguous acoustic-only target from dominating. Training ran on a single NVIDIA H100 80 GB GPU with batch size 16 and gradient accumulation over two steps, learning rates of 2×10⁻⁴ (acoustic LoRA), 1×10⁻⁴ (LLM LoRA), and 5×10⁻⁵ (other modules), taking approximately two hours.

The data is the spoken language assessment task of the S&I Corpus, comprising Parts 1, 3, 4, and 5: six short questions (responses typically 10–20 seconds), one-minute open-ended opinion questions, one-minute process-description tasks with a graphical prompt, and five opinion-based questions with a maximum response time of 20 seconds. The corpus contains approximately 315 hours of speech split into training, development, and test sets with balanced score distributions. CEFR levels map to a 2.0–5.5 scale in 0.5 steps (A2=2.0, A2+=2.5, …, C1+=5.5), and the final score is the arithmetic mean of the four part scores.

Why This Matters

Impact on research. The paper argues that the field's focus on scaling speech-LLM backbones may be misdirected for this task: CASA matches state-of-the-art accuracy with roughly half the estimated inference parameters of the prior speech-LLM system, and neither larger acoustic encoders nor larger language models improved RMSE. It also provides ablation analysis and repeated-run variability statistics that earlier work did not report, addresses the reproducibility gap created by unreleased implementations, and releases its code at https://github.com/aalto-speech/CASA/.

Real-world applications:

• Language testing and certification, where automated scoring could reduce scoring costs and improve scoring consistency relative to human-only rating.
• Classroom and self-study language learning, delivering scalable, timely feedback on spoken responses.
• Automated content validation that checks whether a learner's answer actually addresses the question, which an acoustic-only grader cannot do.
• Adaptable assessment for other ASA corpora or task formats, since the architecture is designed for adaptation without structural changes and uses only three simple fluency features beyond the model itself.

Industry relevance. Reduced inference cost directly affects deployability: the paper frames large multimodal backbones as increasing computational requirements during inference and potentially limiting practical deployment. A 3.13 B-parameter system that scores comparably to a 6.24 B system is more practical for high-volume scoring, and the near-instant content-validation judgments (just under 0.1 s per response) point toward interactive feedback products.

Future Directions

• Task-specific modeling. With P1 and P4 showing weaker per-part RMSE (0.476 and 0.490), the authors suggest investigating separate prediction heads or graders for more constrained versus open-ended tasks, and note that task embeddings failed to contribute.

• Making the acoustic branch use alternative encoders. CASA's aggregator and training recipe were built for Whisper representations; adapting the architecture to exploit self-supervised encoders such as WavLM or wav2vec2 XLS-R 300M — for example through a dedicated pronunciation head — is left to future work.

• Reliable detection of A2-level content. The same LLM judge could be prompted to flag A2-level content even from ASR transcripts, but it did so inconsistently; the authors leave this as an open direction.

• Understanding and controlling training variability. Identical seeds produced test RMSEs from 0.357 to 0.365, and the 4e-4 learning-rate configuration produced one unreproducible 0.350 run, raising open questions about nondeterminism and hyperparameter stability in this pipeline.

Target Audience

Researchers and graduate students in speech processing, spoken language assessment, and educational technology; developers building automated scoring or language-learning feedback systems; and language-testing practitioners and applied linguists interested in how delivery and content evidence can be separated in automated scoring. Readers should be comfortable with speech encoders, LoRA adaptation, and the CEFR scale to follow the architectural and evaluation details.

Authors’ abstract

Research on automatic speaking assessment (ASA) has increasingly adopted multimodal speech large language models to assess learners' speaking performance. However, existing studies provide limited analysis of how acoustic and content information contribute to predictions and how stable the resulting performance is. We propose CASA, a simpler architecture combining Whisper-medium and Qwen3.5-2B that achieves state-of-the-art performance while providing a more interpretable separation between speech delivery and content. On the Speak & Improve Corpus 2025, CASA achieves a root mean square error (RMSE) of 0.358, improving on the previous best RMSE while using approximately half the estimated inference parameters. The general-purpose architecture is designed for adaptation to other ASA corpora without structural changes and relies on three handcrafted fluency features. Through ablations and repeated runs, we analyze the individual and complementary contributions of acoustic and content information, examine performance variability, and demonstrate the potential of large language model reasoning for training-free content validation.