Looking to Learn: Token-wise Dynamic Gating for Low-Resource Vision-Language Modelling

AI summary

Overview

Research area: Low-resource vision-language modelling — multimodal language acquisition for small models trained on cognitively plausible amounts of data, as framed by the Vision track of the BabyLM Challenge 2025.

Technical level: Intermediate. Readers will benefit from familiarity with transformer decoders, cross-attention, and standard vision-language architectures, though the paper explains its mechanisms in accessible terms.

Scope: The paper proposes and evaluates a lightweight decoder-based vision-language architecture whose central innovation is a token-wise dynamic gate that decides, per token, how much to rely on visual versus linguistic information.

What This Paper Is About

Children learn language from far less data than large language models and do so in rich multimodal contexts, but current vision-language models integrate images and text in ways that are not cognitively motivated. This paper asks whether an explicit, token-level gating mechanism can let a small model learn when and how to use visual cues at each step of generation, without any supervision telling it which words are visual. The goal is to test this within the strict data and compute limits of the BabyLM Challenge 2025 Vision track, alongside feature-enhancement techniques and auxiliary contrastive objectives.

Key Contributions

1. Token-wise dynamic gating. Four variants of a gate that fuses text hidden states with cross-attention outputs — soft gate per feature, soft gate per token, hard gate per feature, and hard gate per token — differ along two axes: granularity (per-feature vs. per-token) and decision type (continuous vs. discrete, the latter via Gumbel-Softmax). The paper contrasts this with Flamingo's gated cross-attention dense blocks, which apply uniform layer-wise gating parameters across all tokens and inject visual information only every few layers, whereas this model gates per token and injects visual features at every decoding layer.

2. Feature enhancement for compressed visual input. Because the Challenge provides only a single global image embedding, the authors test FiLM and DyIntra modulation at several integration points (text, cross-attention, image) plus a global channel-attention mechanism, targeting both cross-modal interaction and intra-modal refinement.

3. Auxiliary contrastive objectives. Two objectives are explored: sentence-level CLIP alignment between entire captions and images, and word-level LexiContrastive Grounding (LCG) between individual tokens and images.

4. An interpretability analysis of the learned gate. The gate's behaviour is analysed statistically against part-of-speech, concreteness, and imageability, connecting architectural behaviour to psycholinguistic properties of words.

Main Findings

• Strong BLiMP performance. The base model (soft gate per feature) scores 74.33 on BLiMP, described as almost 4% higher than Flamingo and over 2% higher than GIT. The authors attribute this to a clear separation between text and image streams and consistent fusion at every decoding layer, versus GIT's concatenation of image tokens with text input and Flamingo's intermittent visual injection.

• Lower BLiMP Supplement and Winoground scores than baselines. The base model reaches 56.36 on BLiMP Supplement against 65.1 for Flamingo and 66.4 for GIT. The authors trace this to training data: the image-caption dataset supports BLiMP Supplement better than the text-only dataset, and the 2025 baselines used a 1:4 text-only to image-caption ratio versus their own 1:1 ratio.

• Gating preserves text-only performance. Dynamic gating shows no significant effect on BLiMP and BLiMP Supplement, which the authors describe as the desired outcome for text-only benchmarks, and brings modest benefits on Winoground. The soft gate per token (52.14) and hard gate per token (50.80) beat the no-gate variant (51.34) on Winoground.

• Hard gates hurt VQA. Hard gate models score roughly 5% lower on VQA than the no-gate variant (45.62 and 45.51 versus 50.58). The authors hypothesise the hard gates learned to admit stronger image signals than is optimal for VQA, given few VQA-like constructions in the image-caption training set.

• Feature enhancement has limited, task-specific benefits. No single feature representation technique improved all five benchmarks. FiLM on cross-attention (52.68), FiLM on text (53.49), and DyIntra on cross-attention (53.22) improved Winoground over the base model by 1.07%, 1.88%, and 1.61% respectively. All techniques decreased performance on VQA, with FiLM on image collapsing to 17.92. The conclusion is that these techniques cannot overcome the information bottleneck of a global image embedding.

• Contrastive auxiliary objectives were counterproductive. Pure next token prediction performed best overall. BLiMP score differences were 2.05% and 4.06% with CLIP and LCG respectively. Cited reasons include the dilution of linguistic focus, a CLIP objective designed for larger batch sizes and more data, insufficient visual detail in global embeddings for fine-grained alignment, and instability from alternating text-only and image-caption epochs under a 10-epoch budget.

• The gate learns interpretable, unsupervised patterns. Over 1,034 held-out tokens from the Localized Narratives dataset, the model attended more to image signals for open-class, more grounded parts-of-speech (adjective, noun, proper noun, verb) and more to pure text for function words (conjunction, punctuation, symbols, auxiliary verbs, particles). A Kruskal-Wallis test confirms the relationship (H = 154.91, p < 0.001). The model also showed increased visual grounding for numerals, determiners, and adpositions, suggesting use of visual information for counting, uniqueness, and spatial reference and relationships.

• Weak and non-monotonic concreteness effects. Using the MRC Psycholinguistic Database and Spearman's rank correlation, gate values correlated negatively but weakly with concreteness (ρ = −0.139, p < 0.001) and imageability (ρ = −0.153, p < 0.001), with |ρ| < 0.2. Binned by mean ± 1 SD cutpoints, gate selection was 0.427 (0.141) for very abstract words (<318, 420 words), 0.471 (0.136) for abstract (318–438, 82 words), 0.391 (0.155) for concrete (438–558, 80 words), and 0.343 (0.139) for very concrete (>558, 119 words) — a non-monotonic pattern suggesting part-of-speech matters more.

• Benchmark and data mismatches. EWoK stayed around 50% regardless of architecture or training strategy. A regular expression match found that in 37.69% of EWoK examples at least one of two tested concepts appears fewer than 100 times in the training data, and 13% of test examples have both concepts appearing 0 times. VQA depends heavily on question-answer and turn-taking formats, which the pre-training data lacks.

Methodology in Plain English

The model is a dual-stream transformer. A text stream embeds tokens and runs them through an N-layer transformer decoder. An image stream takes a DINOv2 global token, projects it into the model's hidden space, and refines it with an M-layer transformer encoder. At each decoding layer, the text representation passes through masked self-attention, then cross-attention against the image features, and finally a gate that blends the text hidden state with the cross-attention output using the formula h_fused = g ⊙ h_text + (1 − g) ⊙ h_crossAttn. For text-only samples, the image stream, cross-attention, and gate are all skipped. The four gate variants ask whether to decide per feature or per token and whether to decide continuously or discretely.

Training alternates between text-only and image-caption epochs, which the authors found empirically to be the best curriculum for the five selected benchmarks, with text-only and image-caption data at a 1:1 ratio. Batch size was 64 for most runs and 128 for the auxiliary-objective runs, since larger batches are recommended for contrastive learning. Evaluation used the 2025 pipeline for BLiMP, BLiMP Supplement, EWoK, and Winoground, and the 2024 pipeline for VQA. All architectural features and training strategies were tested as ablations in isolation under identical conditions. Models, tokenizers, checkpoints, and training code are released on HuggingFace and GitHub.

Why This Matters

Impact on research. The paper offers a counterpoint to the assumption that fusion strategies should be uniform across a sequence. It shows that an explicit, unsupervised gate in a small model produces interpretable, linguistically structured behaviour — echoing findings in cognitive science that concrete and abstract words are processed differently, and that concrete nouns are decodable from both linguistic and visual representations while abstract nouns are decodable only linguistically. It also documents concrete failure modes of the BabyLM Challenge setup: the information bottleneck of global image embeddings, the missing-modality problem caused by splitting text-only and image-caption data, the absence of a fine-tuning stage for VQA-style capabilities, and the mismatch between training data and EWoK concepts.

Real-world applications:

• Assistive and educational tools that need to ground language in images with very limited training data.
• Low-resource or on-device multimodal assistants where compute and data budgets are tight.
• Interpretability tooling that exposes which tokens a model grounds in visual versus textual evidence, useful for auditing and debugging multimodal systems.
• Benchmark design for cognitively plausible evaluation, since the paper argues for benchmarks that reward cognitively-motivated mechanisms rather than only task accuracy.

Industry relevance. The results caution against assuming that contrastive auxiliary objectives help small models, and against assuming that feature-modulation techniques transfer from rich patch-based representations to compressed global embeddings. Both are practical constraints for teams building multimodal systems under tight data and compute budgets.

Future Directions

1. Use patch-token image representations. The authors name this explicitly as the aim of future iterations of the framework, since global image embeddings created a bottleneck that feature enhancement could not overcome.

2. Train on a fully multimodal dataset. The alternation between text-only and image-caption epochs caused performance oscillations and training instability; a completely multimodal dataset is proposed as a promising avenue.

3. Improve training data coverage. The paper calls for varied, high-quality text covering a range of English constructions, and for coverage of the constructions and concepts present in the evaluation benchmarks, such as images paired with question-answers for VQA and EWoK concepts.

4. Develop cognitively-motivated benchmarks. The authors propose benchmarks that specifically reward the cognitive principles guiding a model's responses, rather than task performance alone.

An additional open question the authors flag is the interaction between concreteness, word frequency, and word class, which they examine separately here and leave for future work.

Target Audience

Researchers working on vision-language models, multimodal fusion, and data-efficient language modelling will find the architectural comparison and ablation results most useful. Practitioners building small or on-device multimodal systems will benefit from the negative results on contrastive auxiliary objectives and feature modulation under compressed visual input. Cognitive scientists and psycholinguists interested in comparing model behaviour against human word grounding will find the part-of-speech, concreteness, and imageability analyses most relevant, as will participants in and organisers of the BabyLM Challenge interested in the documented constraints of the Vision track.

Authors’ abstract

Training vision-language models on cognitively-plausible amounts of data requires rethinking how models integrate multimodal information. Within the constraints of the Vision track for the BabyLM Challenge 2025, we propose a lightweight decoder-based architecture with (1) token-wise dynamic gating for adaptive fusion of linguistic and visual cues, (2) feature modulation and channel attention to maximise the utility of limited visual information and (3) auxiliary contrastive objectives for visual grounding. Evaluation on five benchmarks (BLiMP, BLiMP Supplement, EWoK, Winoground and VQA) shows competitive or superior performance to multimodal baselines. More notably, our dynamic gate discovers interpretable patterns without explicit supervision, favouring visual cues for content words and linguistic cues for function words. While we identify limitations in the Challenge constraints, such as the information bottleneck created by global image embeddings and training instability from the dataset split, our findings establish dynamic gating as a powerful tool for efficient multimodal learning, offering both interpretability and performance even under severe constraints.