BabyVLM-V2: Toward Developmentally Grounded Pretraining and Benchmarking of Vision Foundation Models

AI summary

Overview

Research area: Computer vision and multimodal (vision-language) foundation models, specifically developmentally inspired, sample-efficient pretraining and cognitively grounded benchmarking.

Technical level: Advanced. The paper combines vision-language model training pipelines, large-scale egocentric video curation, and psychometric test adaptation, though the motivating questions are stated in plain developmental terms.

Scope in one sentence: BabyVLM-V2 is a framework that retrains a compact vision-language "baby model" from scratch on a longitudinal infant egocentric audiovisual corpus (SAYCam) and evaluates it on DevCV Toolbox, a ten-task benchmark adapted from all vision-related measures of the NIH Baby Toolbox.

What This Paper Is About

Foundation models are normally trained on datasets far larger than anything a human child experiences, yet children learn perception and language from a small, longitudinal stream of what they see and hear. This paper asks whether a foundation model can be pretrained on roughly that scale of infant sensory data and remain versatile, and how to measure the result using established developmental psychology tests rather than ad hoc tasks. It extends the earlier BabyVLM-V1 (67k image-utterance pairs, four intuitive tasks, no instruction tuning) into a much larger pretraining set, a flexible multimodal model, and a benchmark grounded in the NIH Baby Toolbox released in February 2025.

Key Contributions

1. A larger, multi-format pretraining set from SAYCam. BabyVLM-V2 expands BabyVLM-V1's 67k image-utterance pairs to 768k image-utterance pairs, 181k video-utterance pairs, and 63k interleaved image-text sequences. The video clips cover 138 of SAYCam's 478 hours. Data is split roughly 3:1:1 into training, validation, and test, with minimal curation: transcript confidence below 0.3 and clips shorter than 0.5 seconds are dropped, X-CLIP video-utterance similarity must exceed 0.1, and frames sampled at 1 FPS must have CLIP similarity above 0.2 to their utterance.

2. An instruction-tuned "baby model" with a flexible interface. The model keeps the BabyVLM-V1 architecture (LLaMA-1.1B language backbone, ViT-L-16 visual encoder with 300M parameters, MLP connector) but adds multi-image and multi-turn capability. It is trained from scratch through a staged pipeline and fine-tuned on 113k instruction examples. A variant with double the language model size showed no significant gain, so the smaller configuration is used.

3. DevCV Toolbox, a ten-task developmental benchmark. All vision-related measures of the NIH Baby Toolbox are adapted into computer vision tasks, each with thousands of naturalistic examples drawn from SAYCam and a separate out-of-domain test set built from Ego4D. Tasks span language, executive function/memory, and math subdomains and require single-image, multi-image, video, and multi-turn inputs.

4. Empirical validation. Proprietary and open-source models plus human volunteers are benchmarked, instruction data is validated through task-specific fine-tuning of LLaVA-OneVision-7B and Qwen2.5-VL-7B, pretraining language sources are ablated, and the baby model is tested for out-of-domain and unseen-task generalization.

Main Findings

• Human upper bound and random baseline frame the benchmark: adult volunteers from the authors' home institute reach 93.0% overall; random guess is 31.8% overall. Adults are near-perfect on executive function/memory and math tasks, with Localization the lowest at 87.3%, which a follow-up suggested improved when volunteers were told to spend more time on the task.

• Model spread is wide: GPT-5 leads at 87.6% overall, followed by Gemini-2.5-pro (82.5), Qwen3-VL-30B-A3B (78.0), GPT-4o (74.6), Gemini-2.5-flash (72.7), and Qwen2.5-VL-7B (63.7). The baby model reaches 63.9%, ahead of LLaVA-OneVision-7B (53.4), Qwen2.5-VL-3B (47.0), InternVL3.5-1B (37.2), and LLaVA-OneVision-0.5B (33.2).

• The baby model beats GPT-4o on math tasks: it outperforms GPT-4o on Object Counting and Who Has More, and is on par with the four latest GPT and Gemini models on Spatial Details and Who Has More.

• GPT models struggle to count: Object Counting requires counting between 1 and 12 objects, and GPT-4o can hardly count beyond 5. The baby model counts better than GPT-4o when six or more objects are present.

• Proprietary model differences: GPT-5 is significantly better than the rest on Spatial Details, while Gemini models are better at Object Counting than GPT models.

• Instruction tuning works: fine-tuning LLaVA-OneVision-7B and Qwen2.5-VL-7B separately on each task produces consistent and substantial gains.

• Synthetic captions give a modest boost: replacing caregiver transcripts with GPT-4o-generated captions raises overall accuracy from 63.9% to 65.4%, with the largest improvements on Object Counting and Memory. The paper concludes the minimally curated transcripts already provide strong supervision.

• Out-of-domain generalization is weak: on the Ego4D-based sibling benchmark, the baby model scores 41.1% versus 31.8% for random guess, well below its in-domain performance of 55.2% on DevCV Toolbox. SAYCam was filmed by 3 babies across 4 homes with 307k pixels and objects filling 57% of the frame on average; Ego4D was filmed by 923 participants across 74 sites with over 2M pixels on average and objects filling 4% of the frame.

• Unseen tasks fail: Looking While Listening and Subitizing are excluded from instruction tuning, and the baby model performs near random guess on both.

• Ablation of model size: doubling the language model's size yields no significant gain; the smaller variant is used because it better balances developmental plausibility and expressive capacity.

Methodology in Plain English

The researchers start from SAYCam, a dataset of head-mounted camera recordings from three infants, captured roughly weekly for about two hours per week from age 6 to 32 months. They transcribe the caregiver utterances with Azure Speech Recognition and keep the curation minimal so the data stays close to what an infant actually experiences. From that pipeline they build three data formats: short clips paired with a single utterance, still frames paired with an utterance, and sliding-window sequences of interleaved images and utterances that mimic conversation.

The model is then trained from scratch in stages: the language and vision backbones are first trained separately (language on 283k utterances with a 6000-token BPE vocabulary; vision on 1085k images with a DINOv2 objective), the MLP connector is trained to align the two modalities while both backbones are frozen, the connector and language backbone are jointly pretrained on the full mixed-format data, and finally everything is fine-tuned on the instruction set. Training runs on four NVIDIA A6000 GPUs with 48 GB VRAM each; the language backbone finishes in under an hour, the vision backbone in 4 days, the connector in about 5 hours, joint pretraining in roughly 34 hours, and instruction tuning in about 24 hours.

For the benchmark, the authors consulted two developmental psychologists and mapped each vision-related NIH Baby Toolbox measure to a machine-checkable task, replacing clipart with real SAYCam objects and actions detected and cropped using Grounding-DINO, and filtering labels to the child-oriented MAB-CDI vocabulary. Accuracy is the metric for all tasks. An Ego4D-based version of the benchmark serves as an out-of-domain test, and human volunteers were surveyed as an upper bound

Authors’ abstract

Early children's developmental trajectories set up a natural goal for sample-efficient pretraining of vision foundation models. We introduce BabyVLM-V2, a developmentally grounded framework for infant-inspired vision-language modeling that extensively improves upon BabyVLM-V1 through a longitudinal, multifaceted pretraining set, a versatile model, and, most importantly, DevCV Toolbox for cognitive evaluation. The pretraining set maximizes coverage while minimizing curation of a longitudinal, infant-centric audiovisual corpus, yielding video-utterance, image-utterance, and multi-turn conversational data that mirror infant experiences. DevCV Toolbox adapts all vision-related measures of the recently released NIH Baby Toolbox into a benchmark suite of ten multimodal tasks, covering spatial reasoning, memory, and vocabulary understanding aligned with early children's capabilities. Experimental results show that a compact model pretrained from scratch can achieve competitive performance on DevCV Toolbox, outperforming GPT-4o on some tasks. We hope the principled, unified BabyVLM-V2 framework will accelerate research in developmentally plausible pretraining of vision foundation models.