Do Language Models Associate Sound with Meaning? A Multimodal Study of Sound Symbolism

AI summary

Overview

Research area: Natural Language Processing / multimodal interpretability and cognitive linguistics, specifically phonetic iconicity (sound symbolism) in Multimodal Large Language Models.

Technical level: Intermediate — assumes familiarity with LLM evaluation metrics (macro-F1, Pearson correlation), attention analysis, and phonetic notation (IPA), but the core ideas are explained accessibly.

Scope: The paper builds a large multilingual mimetic-word dataset (LEX-ICON) and uses semantic-dimension A/B tests plus phoneme-level attention analysis to test whether MLLMs link the sound of a word to its meaning across text, IPA, and audio inputs.

What This Paper Is About

Sound symbolism is the idea that certain speech sounds carry non-arbitrary meaning — the classic example being that people overwhelmingly match "kiki" with sharp shapes and "bouba" with round ones. This paper asks whether Multimodal Large Language Models that accept both text and audio exhibit the same phonetic intuitions, and whether their internal attention actually concentrates on the iconic phonemes that linguistic research associates with particular meanings.

Key Contributions

1. The paper presents LEX-ICON, described as the first large-scale investigation of phonetic iconicity in MLLMs, backed by a novel multilingual mimetic word dataset spanning multiple language families — comprising 8,052 natural words from English, French, Japanese, and Korean, plus 2,930 systematically constructed pseudo-words.
2. It provides a quantitative evaluation methodology for sound symbolism via the semantic dimension approach, applied to up to 25 semantic dimension pairs, intended to capture phonetic iconicity akin to human behavior.
3. It supplies an interpretability analysis measuring phoneme-level attention fraction scores to show how phonemes and meanings relate inside the models, finding that attention to iconic phonemes is more prominent in late layers when models process constructed pseudo-words.
4. It reports human evaluation results on audio input for sampled data, which score above the 0.50 baseline in most dimensions and are used to validate the automatically annotated pseudo ground truth.

Main Findings

• Models beat the baseline on most dimensions: Averaged across all input types, MLLM macro-F1 scores surpass the baseline of 0.50 in 84.2% of semantic dimensions for the natural word group and 68.4% for the constructed group. This includes constructed pseudo-words unlikely to have been memorized during training.
• Small model drags the average down: Overall performance becomes even larger when the comparatively small Qwen2.5-Omni-3B model is excluded.
• The bouba-kiki effect shows up in models: The models' strong performance on the sharp vs. round dimension aligns with well-known cognitive linguistic experiments such as the "bouba-kiki" effect.
• Closest model to humans is not the largest: Qwen2.5-Omni-7B achieves the highest overall Pearson correlation with human evaluation across semantic dimensions, with a maximum r = 0.579, while larger models such as gemini-2.5-flash deviate more from human results.
• Constructed words align better with humans than natural words: In all models, constructed words elicit responses closer to human tendencies than natural words. The relatively low correlation for IPA-converted natural words suggests linguistic arbitrariness may override iconic patterns, since models' knowledge is shaped by large-scale distributional semantics.
• Modality preferences are systematic: For acoustically grounded dimensions such as big vs. small (linked to formant frequencies) and fast vs. slow (linked to consonant voicing duration), models perform better on constructed words in audio format. For non-acoustic dimensions such as sharp vs. round (lip rounding gestures) and valence dimensions (beautiful vs. ugly, happy vs. sad), models rely more on textual representations. Audio-over-text advantage scores correlate across word groups at Pearson r = 0.681 and Spearman ρ = 0.705.
• Attention exceeds baseline and favors text: For constructed words, attention fraction scores track the correct semantic feature above the 0.50 baseline, with IPA text (0.523 on average) consistently exceeding audio (0.506 on average), and an upward trend toward late layers.
• Natural words show weaker attention signals: Average attention fraction scores for natural words are lower (IPA = 0.507, audio = 0.501), which the authors attribute to arbitrary form-meaning mappings attenuating phonosemantic cues.
• Attention maps match linguistic theory: Phonemes /p/ and /k/ draw elevated attention under the sharp feature, while /m/ and /n/ associate with round; /A/ associates with big and /i/ with small.

Methodology in Plain English

The researchers assembled a corpus in two parts. For the natural group, they manually collected 8,052 mimetic words and definitions from specialized onomatopoeia dictionaries and authoritative dictionaries across English, French, Japanese, and Korean. For the constructed group, they generated novel disyllabic CVCV pseudo-words from 15 consonants in five categories and four vowels in two categories, checked them against the IPA-dict database across four languages to filter out anything matching existing words (yielding 3,108), then removed words that Google TTS pronounced incorrectly, leaving 2,930.

Each word was labeled against up to 25 semantic dimension pairs taken from Sidhu et al. (2022), such as sharp vs. round. Natural words were annotated automatically by four LLMs — GPT-4.1, Qwen3-32B, Gemma-3-27B, and Gemini-2.5-flash — and only features unanimously agreed upon and not labeled "neither" were kept as pseudo ground truth, filtering 67.0% of annotation points and leaving 66,484 features. Constructed words were scored using empirical phoneme-category coefficients from Sidhu et al. (2022), with a threshold of 1.0 standard deviation from the neutral point.

For the semantic dimension prediction test, models received each word in three forms — original orthographic text, phoneme-level spaced IPA text, and TTS audio — and answered binary A/B questions per dimension. Macro-F1 was computed per dimension, averaged across the four languages (for natural words) and the three input types, to correct for class imbalance. Audio was obtained through Google Text-to-Speech for English, French, and Japanese, and MeloTTS for Korean; IPA conversion used the Epitran package.

For the attention analysis, the Qwen2.5-Omni-7B model was chosen for its human-like correlation and available weights. Attention scores between each IPA symbol's tokens and the semantic feature tokens were normalized so paired feature scores sum to one, averaged across heads, words, and both feature-order conditions to counter position bias. Audio phoneme boundaries came from the Montreal Forced Aligner and were aligned to the model's 40 ms sampling period.

Why This Matters

Impact on research: The work bridges artificial intelligence and cognitive linguistics, offering what the authors call the first large-scale, quantitative analysis of phonetic iconicity in terms of MLLM interpretability. It extends multimodal interpretability beyond the visual modality, which prior work has predominantly focused on, and gives linguists a nonhuman test bed for experiments usually run with humans.

Real-world applications:

• Language learning tools, given that iconicity is noted to facilitate early-childhood and second-language acquisition.
• Brand and product naming, where sound-symbolic effects are described as commercially relevant.
• Model interpretability and auditing — understanding when and why audio-capable models rely on acoustic versus textual cues.
• Speech interface design, where knowing which semantic dimensions are best served by audio versus text could inform how information is presented.

Industry relevance: Any product built on multimodal models that ingest audio — voice assistants, TTS pipelines, accessibility tools, marketing copy generation — could benefit from knowing that model behavior varies systematically by semantic dimension and input modality.

Future Directions

• Deepen the analytical methodology through experiments with more human participants, since human evaluation here covered only the audio input type on sampled data for feasibility.
• Investigate modality-specific information such as intonation, which the current word-level setup does not capture.
• Extend the approach to application fields such as language learning or brand effects.
• Address the discrepancy between dimensions where humans score highly and those where models perform well, and the tendency of models to overlook subtle phonosemantic cues in natural, arbitrary-form words.

Target Audience

Researchers in NLP and multimodal model interpretability, cognitive linguists and psycholinguists studying sound symbolism, and practitioners building audio-capable language applications. Readers with a basic grasp of classification metrics and phonetics will get the most out of the quantitative sections, though the conceptual framing is accessible to a broader technical audience.

Authors’ abstract

Sound symbolism is a linguistic concept that refers to non-arbitrary associations between phonetic forms and their meanings. We suggest that this can be a compelling probe into how Multimodal Large Language Models (MLLMs) interpret auditory information in human languages. We investigate MLLMs' performance on phonetic iconicity across textual (orthographic and IPA) and auditory forms of inputs with up to 25 semantic dimensions (e.g., sharp vs. round), observing models' layer-wise information processing by measuring phoneme-level attention fraction scores. To this end, we present LEX-ICON, an extensive mimetic word dataset consisting of 8,052 words from four natural languages (English, French, Japanese, and Korean) and 2,930 systematically constructed pseudo-words, annotated with semantic features applied across both text and audio modalities. Our key findings demonstrate (1) MLLMs' phonetic intuitions that align with existing linguistic research across multiple semantic dimensions and (2) phonosemantic attention patterns that highlight models' focus on iconic phonemes. These results bridge domains of artificial intelligence and cognitive linguistics, providing the first large-scale, quantitative analyses of phonetic iconicity in terms of MLLMs' interpretability.