Encoder-Free Multimodal LLMs Learn Implicit Encoders

AI summary

Overview

Research area: Multimodal large language models (speech, vision, and text), with a focus on mechanistic representation analysis and emergent modularity.

Technical level: Intermediate. The paper assumes familiarity with Transformer LLMs and with standard probing/evaluation toolkits (SUPERB, MIEB), but the core argument is stated plainly and the experiments are conceptually simple.

Scope: An empirical study testing whether the early Transformer layers of two encoder-free multimodal LLMs, Gemma-4 12B and Inkling-Small, learn to behave like the dedicated audio and image encoders used in conventional multimodal LLMs.

What This Paper Is About

Most multimodal LLMs bolt a separately pre-trained audio or vision encoder onto a text LLM backbone. A newer class of "encoder-free" models instead feeds raw waveform windows, image pixel patches, or spectral features straight into the LLM through a simple linear or MLP projection. This paper asks whether such models secretly re-learn an encoder inside their own early layers, and tests that hypothesis with context-invariance measurements, activation patching, standard encoder benchmarks, layer-wise probing, and distillation.

Key Contributions

1. Demonstrates context invariance in early layers. Using Centered Kernel Alignment (CKA), the authors show that audio and image token representations in Gemma-4 12B stay highly similar to neutral-context references (above 0.9 for audio) through roughly the first 12 layers, only diverging once cross-modal context is mixed in later layers.

2. Shows independent encoding is as good as joint encoding up to a point. Via activation patching on OmniBench (1,142 text+image+audio question quadruplets), encoding each modality separately and merging at layer L matches the fully joint baseline up to layer 12, then degrades — establishing a practical boundary for where cross-modal fusion actually begins.

3. Benchmarks the "implicit encoders" as encoders. The first 12 layers are evaluated on SUPERB (speech) and the image-only tasks of MIEB-Lite (10 tasks: 8 classification, 2 clustering), compared against wav2vec 2.0, HuBERT, FBANK features, SigLIP2-base, and DINOv2-base.

4. Isolates and compresses the implicit encoder via distillation. Compact student models of 800M, 400M, 200M, and 100M parameters are distilled from the first 12 layers of Gemma-4 12B, showing most of the audio-encoding behavior survives at roughly 7% of the teacher's size.

Main Findings

• Gemma-4 12B is context-invariant for audio up to layer 12. Across 1,000 LibriSpeech dev-clean utterances re-encoded with a transcription prompt, an emotion prompt, a real MSCOCO photograph, and a combined photo-plus-prompt condition, CKA similarity to the neutral reference stays well above 0.9 until layer 12, then declines gradually across the remaining 48 layers.

• Inkling-Small is even more invariant for audio. Aside from a small dip at layer 14 and a significant drop at the very last layer, its audio representations remain highly similar to the neutral reference across all 42 layers.

• Visual inputs show the same pattern. For 797 MSCOCO images encoded with no caption, a matched caption, and an average over 60 mismatched captions, Gemma-4 12B stays near 1.0 CKA until layer 12 and then drops sharply; Inkling-Small is nearly invariant until layer 10 and mildly context-sensitive thereafter.

• Joint and independent encoding are equivalent early, divergent late. OmniBench baseline accuracy is 40.4% for Gemma-4 12B and 43.8% for Inkling-Small. Merging independently encoded modalities holds near these baselines until layer 12, then falls to chance level (25%) for Gemma-4 12B and to approximately 37% for Inkling-Small.

• Inkling-Small hides its context in boundary tokens. Layer-wise CKA between joint and independent encoding stays high for content tokens (audio, image, question) but declines sharply after layer 12 for boundary tokens between modality segments — explaining why its embeddings looked invariant yet its OmniBench accuracy still dropped.

• Strong on speech classification, weak on fine-grained speech tasks. On SUPERB with mean-pooled embeddings, Gemma-4 12B reaches 96.79 keyword-spotting accuracy, 98.13 intent-classification accuracy, 85.14 speaker-identification accuracy, and 66.79 emotion-recognition accuracy; Inkling-Small reaches 94.90, 98.29, 85.52, and 69.61 respectively — competitive with wav2vec 2.0 and HuBERT baselines. On frame-wise tasks they lag: phoneme-recognition PER is 11.11 (Gemma) and 15.44 (Inkling) versus 3.53 for HuBERT Large, and ASR WER is 9.57 and 19.12 versus 3.62 for HuBERT Large.

• Vision performance is middling. On MIEB-Lite image tasks averaged, Gemma-4 12B at layer 10 scores 58.39, compared to 73.61 for SigLIP2-base and 70.10 for DINOv2-base. It is competitive on several classification tasks (e.g., 83.20 on EuroSAT, 72.70 on PatchCamelyon) but far behind on clustering (50.40 NMI on ImageNetDog15 vs 79.61 for SigLIP2-base).

• The layer-wise information hierarchy mirrors self-supervised speech encoders. Using PWCCA, Gemma-4 12B's similarity to log-mel filterbank features peaks at layer 3 with a score of 0.95 despite raw-waveform input; phonetic similarity peaks at layer 8 (Gemma) and layer 7 (Inkling); lexical similarity peaks at layer 9 in both; similarity to whole-sentence MPNet embeddings begins to grow at layer 10 and stays high.

• The implicit encoder is far larger than it needs to be. The first 12 layers of Gemma-4 12B contain 2.73B parameters, yet distillation onto 100M–800M parameter students trained on roughly 8,000 hours of GigaSpeech, Common Voice English, and AudioSet preserves most SUPERB performance — with the 800M student even exceeding the teacher on speaker identification (92.44 vs 85.14).

Methodology in Plain English

The authors treat the two open-weight models as black boxes and probe them in four ways.

First, they hold one input fixed (a speech clip or an image) while swapping the surrounding text and image/audio context, then measure with CKA how much the internal representation of that fixed input changes layer by layer. A dedicated encoder would be blind to the surrounding context, so high similarity early in the network is evidence of encoder-like behavior.

Second, they run a multimodal question-answering task (OmniBench) two ways: normally, with all modalities processed jointly from layer 1; and with each modality processed alone through the first L layers, then stitched together using the model's expected token layout and run through the remaining layers. Padding preserves positional information. If the early layers really are independent encoders, the two should score the same.

Third, they freeze the first 12 layers as a feature extractor and plug the resulting embeddings into standard benchmark pipelines — SUPERB for speech and MIEB-Lite for images — to see whether the representations generalize beyond the LLM itself. They also replicate a published layer-wise probing pipeline (PWCCA) that correlates each layer against log-mel features, phoneme labels, word labels, part-of-speech tags, word-sense labels, and sentence embeddings.

Fourth, they distill the first 12 layers of Gemma-4 12B into smaller randomly initialized students using the DistilHuBERT layer-to-layer objective, keeping the same 12 blocks, head counts, and sliding-window attention pattern while shrinking model and head dimensions.

Why This Matters

The paper reframes a widely adopted architectural shortcut. If encoder-free models spend a large fraction of their parameters and training compute re-deriving functionality that a pre-trained encoder provides off the shelf, then the "simpler" architecture is not necessarily cheaper or better — it just pays the cost internally. The distillation result, where a small fraction of the parameters retains most of the encoder behavior, quantifies how much of that capacity is redundant. The work also gives practitioners a concrete empirical boundary (around layer 12 in both models) for where cross-modal fusion actually happens, which matters for anyone doing interpretability, model editing, or partial-layer reuse.

Real-world applications:

• Model compression and deployment: knowing that early layers form a separable, distillable encoder supports building small on-device audio or vision front-ends extracted from large multimodal LLMs.
• Efficient training recipes: initializing or regularizing early layers with pre-trained encoder weights or distillation objectives could cut the compute needed to train future encoder-free models.
• Streaming and latency-sensitive inference: the finding that low-level acoustic and phonetic information concentrates in the first ~9 layers supports early-exit or cascaded designs for speech tasks.
• Interpretability and safety auditing: the boundary-token finding in Inkling-Small shows where cross-modal information is actually carried, which is useful for debugging multimodal failures.

Industry relevance: the two models studied are large open-weight releases from Google DeepMind and Thinking Machines Lab, so the results directly inform how labs design the next generation of natively multimodal models, and whether to keep or discard separate encoders in their stacks.

Future Directions

• Quantify the real trade-off. The authors explicitly call for measuring encoder-free versus encoder-based architectures in terms of both task performance and training cost (data volume and compute time), which they state is not yet clear.
• Hybrid initialization. Initializing the early layers of an encoder-free model with parameters from pre-trained encoders — or adding distillation from pre-trained encoders as an auxiliary training objective — is proposed as a way to get the benefits of both designs.
• Close the fine-grained gap. Frame-wise tasks such as ASR and phoneme recognition underperform substantially versus much smaller self-supervised models, and it is unresolved whether this is a capacity, training-data, or architectural limitation.
• Extend beyond audio. The distillation experiment is restricted to the audio modality; whether the implicit image encoder in these models can be similarly isolated and compressed is left open. Separately, the vision evaluation covers only the image-only tasks of the MIEB-Lite subset, and the paper reports per-layer sweeps in its appendix figures without a broader vision benchmark sweep.

Target Audience

Researchers and engineers working on multimodal LLM architecture, speech and vision representation learning, and model interpretability — particularly those deciding whether to build on pre-trained encoders or train a monolithic encoder-free model. It is also useful for graduate students learning how to combine CKA, activation patching, and linear probing into a coherent empirical argument about what a network has learned, and for practitioners who need a concrete answer to "which layers of this model actually process audio or images."

Authors’ abstract

We empirically analyze two recently proposed encoder-free multimodal LLMs, Gemma-4 12B and Inkling-Small. Unlike previous multimodal LLMs, these models do not use modality-specific encoders to extract tokenized embeddings representing audio and image input; instead, they directly feed windows of samples, pixels, or spectral features to the Transformer language model. We analyze whether the early layers of these models learn to function similarly to dedicated audio and visual encoders. We find that these early layers do in fact act as ``implicit encoders,'' encoding audio and visual inputs in a way that is largely invariant to their cross-modal context. Furthermore, when performing multimodal reasoning tasks, we show that independently encoding the modalities throughout these early model layers achieves the same accuracy as jointly encoding them from the first layer onwards. Finally, we use layer-wise probing to analyze the salient information captured by the implicit encoders and find their behavior to be similar to several popular encoder models.