Technical Deep Dives

Vision-Language Models: How AI Learned to See — and Talk About What It Sees

From AlexNet's 15.3% ImageNet error in 2012 to CLIP's 400-million-pair contrastive training in 2021, here is how vision and language merged into one model — and where it still hallucinates, miscounts, and now drives robots.

Gabriele Masetti · 2026-09-11

The 15.3% Error Rate That Ended the Old Computer Vision

For years, image recognition was a fight between hand-built feature detectors — edge filters, SIFT descriptors, color histograms — bolted onto classifiers that never quite generalized. That ended at the 2012 ImageNet Large Scale Visual Recognition Challenge, when Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton entered a convolutional neural network later named AlexNet. It scored a 15.3% top-5 error rate on the 1,000-category ImageNet test set; the next-best entry, built on traditional methods, trailed at 26.2%. AlexNet had eight layers — five convolutional, three fully connected — and it ran on two GTX 580 GPUs because a single card couldn't hold it.

That gap, more than ten percentage points, is why 2012 gets treated as the hinge point of modern computer vision. It proved that a network which learns its own features from raw pixels, given enough labeled data and enough compute, beats decades of manually engineered ones. Every ImageNet winner after that was some flavor of deep net, and by the mid-2010s CNNs were the default for any vision task.

But CNNs trained this way had a structural ceiling: they could only recognize the fixed set of categories they were trained on. A model trained on ImageNet's 1,000 classes could tell you "Labrador retriever" but had no path to describing a scene, answering a question about it, or handling a category nobody had bothered to label. Vision and language stayed separate disciplines, with separate models, for another nine years.

CLIP and the 400 Million Pairs That Changed the Question

In January 2021, OpenAI published "Learning Transferable Visual Models From Natural Language Supervision" — the paper that introduced CLIP (Contrastive Language-Image Pre-training). Instead of training on a fixed label set, CLIP trained an image encoder and a text encoder jointly on about 400 million (image, text) pairs scraped from the public internet, using a contrastive objective: pull the embeddings of a matching image-caption pair together, push mismatched pairs apart. Nobody had to hand-label a single one of those 400 million images with a category name — the caption was the label.

The result reframed what "recognition" meant. CLIP's best model, a ViT-L/14 variant run at 336-pixel resolution, matched the 76.2% top-1 accuracy of a fully supervised ResNet-50 on ImageNet — without ever training on ImageNet's labels, purely by comparing image embeddings against text prompts like "a photo of a dog" at inference time. Zero-shot transfer methods before CLIP topped out around 11.5% on the same benchmark, so this was roughly a 6.6x jump.

CLIP also generalized where classifiers historically broke: 60.2% on ImageNet-Sketch versus a ResNet's 25.2%, and it beat the best public ImageNet model on 20 of 26 other transfer benchmarks tested. The practical consequence mattered more than the leaderboard number: once you have an image encoder and a text encoder that share a coordinate space, you have the connective tissue for a model that can look at a picture and produce language about it. CLIP's vision encoder, not the ResNet lineage, is what shows up inside most vision-language models built since.

Inside a Modern VLM: Patches, Projectors, and 16x16 Words

The architecture that CLIP made practical rests on the Vision Transformer, introduced by Alexey Dosovitskiy and colleagues at Google Research in the October 2020 paper "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale," later published at ICLR 2021. ViT throws out convolution almost entirely: it slices an image into fixed-size patches — 16x16 pixels is the paper's reference configuration — flattens each patch, linearly projects it into an embedding, and feeds the resulting sequence of patch embeddings into a standard transformer encoder, exactly the way a language model consumes a sequence of word tokens. An image of a dog becomes, functionally, a sentence made of image-patches instead of words.

That shared token format is what makes today's VLMs buildable at all. The now-standard recipe has three pieces: a vision encoder (often a CLIP-style or SigLIP ViT) turns an image into a set of patch embeddings; a projector — sometimes a single linear layer, sometimes a small MLP or a cross-attention "resampler" — maps those embeddings into the same dimensional space the language model expects; and a pretrained LLM consumes the projected image tokens interleaved with text tokens, generating a response autoregressively as if the image were just more text.

LLaVA, released by Haotian Liu and coauthors on April 17, 2023 ("Visual Instruction Tuning," a NeurIPS 2023 oral paper), demonstrated this recipe could be built cheaply: a frozen CLIP encoder, a simple linear projector, and Vicuna as the language model, fine-tuned on GPT-4-generated instruction data. Every subsequent open VLM — Qwen-VL, LLaVA's own successors, PaliGemma, Llama's vision models — is a variation on that same three-part skeleton, differing mainly in encoder choice, projector design, resolution handling, and how much of the stack gets trained versus frozen.

The Lineage: From GPT-4V to Qwen3.5

The proprietary and open tracks moved almost in lockstep from 2023 onward. OpenAI announced GPT-4 with Vision in September 2023 — the system card is dated September 25, 2023 — giving ChatGPT the ability to answer questions about uploaded images, with API access following on November 6, 2023. Alibaba's Qwen team published Qwen-VL on arXiv on August 24, 2023, an early open competitor built on their Qwen LLM. Google answered on December 6, 2023 with Gemini 1.0, announced as natively multimodal from pretraining rather than a vision module retrofitted onto a text model, released in Ultra, Pro, and Nano sizes, with public access following on December 13, 2023.

2024 is when the open ecosystem caught up on cadence. Google released PaliGemma on May 14, 2024, pairing a SigLIP-So400m vision encoder with a 3B-parameter Gemma backbone — small enough to run on a single consumer GPU — and followed it with the larger PaliGemma 2 (3B, 10B, and 28B variants) in December 2024. Anthropic added image input to Claude for the first time with the Claude 3 family (Opus, Sonnet, Haiku) on March 4, 2024, putting vision on every tier from launch rather than reserving it for a flagship.

Alibaba shipped Qwen2-VL on August 30, 2024, and Meta released Llama 3.2's vision-enabled models — 11B and 90B parameter sizes, handling images up to 1120x1120 pixels — on September 25, 2024. The pace kept accelerating into 2025: Qwen2.5-VL landed January 28, 2025, and Qwen3-VL's largest variants (235B-A22B, both Instruct and Thinking versions) arrived September 23, 2025. Then the naming stopped marking vision as a variant at all. Alibaba released Qwen3.5 on 16 February 2026, opening the series with the open-weight Qwen3.5-397B-A17B, a sparse mixture-of-experts model that activates 17 billion of its 397 billion parameters per token and is described as natively vision-language rather than a VL edition of a text model. In roughly three years a technique invented at two labs stopped being a feature and became the default shape of the model.

What They Still Can't Do: Blind Spots, Counting, and Charts

The headline benchmark numbers look strong and keep climbing. MMMU (Massive Multi-discipline Multimodal Understanding), introduced on arXiv in November 2023 with 11.5K college-level questions across six disciplines, put GPT-4V at 55.7% at launch against an estimated human-expert accuracy of 88.6% — a wide gap. By mid-2026 the leaderboard tells a different story: frontier models are reported clearing the high-80s, converging on or slightly past that human-expert mark.

On DocVQA, a document visual-question-answering benchmark, frontier models were exceeding 95% by 2026, and the leaderboard is described as approaching saturation for single-page document understanding — Qwen2.5-VL-72B was reported at 0.964 and GPT-4o at 0.928 on standard splits.

Those numbers describe recognition and reading, not the kind of precise visual reasoning humans do without thinking. A July 2024 paper titled "Vision Language Models Are Blind" (Rahmanzadehgervi, Bolton, and coauthors, presented at ACCV 2024) built a "BlindTest" suite of seven trivial-for-humans tasks: do two circles overlap, how many times do two lines cross, which letter in a word is circled, how many circles appear in an Olympic-style logo.

Four VLMs that were state of the art at the time, including GPT-4o and Gemini 1.5 Pro, averaged just 58.07% accuracy across those tasks; Claude 3.5 Sonnet did best at 77.84%, still well short of the 100% a human manages almost instantly. The failure mode is consistent: these models degrade specifically when geometric primitives overlap or sit close together, meaning they're reasoning over a compressed, roughly-gisted representation of the image rather than tracking exact pixel geometry.

The paper's numbers are fixed; the models it tested are not. GPT-4o, Gemini 1.5 Pro and Claude 3.5 Sonnet are several generations old now, and no comparable re-run of BlindTest on current frontier models has been published. Read 58.07% as a measurement of 2024 systems, not as a verdict on what ships today.

A related and better-known issue is object counting, which multiple hallucination benchmarks (spanning categories like attribute, relation, comparison, and OCR errors) treat as a separate failure category from general recognition — models will confidently report a count that's off by one or two, or hallucinate objects or text that aren't in the image, most often on cluttered scenes and low-resolution chart or document images.

The practical upshot for anyone deploying these systems: strong average benchmark scores coexist with brittle, specific failures on exactly the tasks — precise counting, occlusion, fine layout in a chart — that look easiest to a human reviewer.

Be My AI, Radiology Drafts, and Document Extraction

The most concrete deployment of VLM accessibility work is Be My Eyes' partnership with OpenAI. The companies announced their collaboration in March 2023, building "Virtual Volunteer" — later renamed Be My AI — on top of GPT-4's vision capabilities, so blind and low-vision users could point a phone camera at a scene or document and get an immediate spoken description instead of waiting for a human volunteer.

The feature moved from an initial announcement to iOS beta testing that August, then rolled out to hundreds of thousands of iOS and Android users starting in December 2023, with a contact-center integration following in November of that year. It's one of the few VLM applications where the "seeing" is the entire product rather than a feature bolted onto something else.

In medicine, deployment is more research-stage than clinical. GPT-4V and comparable models have been shown able to draft plausible-sounding radiology reports from chest X-rays, and 2024–2025 research has produced medical-specific vision-language systems aimed at report generation and interpretability, including chain-of-thought approaches that try to ground a diagnosis in specific image regions rather than free-associate a description.

That regulatory boundary moved at the end of 2025, though not in imaging. UpDoc V1.0, cleared under 510(k) K253281 on 23 December 2025 and announced on 25 June 2026, is described by its maker as the first Software as a Medical Device built on a patient-facing large language model — narrow by design, covering insulin titration for adults with type 2 diabetes, with a defined action set and logged provenance on every step.

Reading scans is a different problem. The tools the FDA has cleared for imaging are still narrower, purpose-built networks for tasks like nodule or fracture detection; the agency has said only that it will explore ways to identify and tag devices built on foundation models on its public list, and no such tag had appeared there by September 2026. VLM radiology work sits in the research-and-draft-assist category, not autonomous diagnosis.

Document AI is the area where deployment is least controversial: DocVQA-style extraction, reading tables, forms, and scanned text, is now a commodity capability across frontier and open current-generation models alike, precisely because it's a text-recognition problem the underlying transformer and OCR-adjacent training data are well suited to, rather than a spatial-reasoning problem where these models are weaker.

From Pixels to Motor Commands: RT-2, OpenVLA, and Physical Intelligence's π0

The same architecture that lets a model describe an image turns out to be reusable for controlling a robot, if you retrain the output layer to emit motor commands instead of words. Google DeepMind's RT-2 (Robotic Transformer 2), announced July 28, 2023, was among the first to demonstrate this: built by adapting existing vision-language models (PaLM-E and PaLI-X) rather than training from scratch, RT-2 output robot actions as text tokens, trained jointly on internet-scale vision-language data plus physical demonstrations collected from 13 robots over 17 months.

It generalized to instructions and objects that never appeared in its robot-specific training data — placing an item on a printed number, or picking the smallest of several objects — because the underlying VLM had already learned those concepts from web-scale pretraining.

OpenVLA followed in 2024 as the open-source answer: a 7-billion-parameter model built on the Prismatic-7B VLM framework, combining a fused DINOv2 and SigLIP vision encoder with a Llama 2 backbone, trained on 970,000 real-world robot demonstrations and released with full weights so labs without Google-scale robot fleets could fine-tune it. It reads now as the reference open implementation rather than the current state of the art.

Physical Intelligence, a San Francisco robotics startup incorporated in 2024, introduced π0 on October 31, 2024 as a "vision-language-action flow model" — starting from a pretrained VLM and adding a flow-matching action head, trained across seven distinct robot configurations and 68 tasks, and demonstrated on tasks no prior learned robot system had managed end-to-end, including folding laundry pulled from a hamper and assembling a cardboard box. Three successors have followed from the same lab: π0.5 on April 22, 2025, π*0.6 on November 17, 2025, and π0.7 on April 16, 2026, each pitched on generalization to tasks the robot was never shown.

Google DeepMind extended the line into a production-oriented family on March 12, 2025, launching Gemini Robotics and Gemini Robotics-ER (embodied reasoning) built on Gemini 2.0, developed in partnership with Apptronik for humanoid hardware. Gemini Robotics 2 arrived on July 30, 2026, with an ER 2 and an on-device variant beside it, moving the control problem from a pair of arms to a whole humanoid body — DeepMind's phrase for the range is "from feet to fingertips" — demonstrated on Apptronik's Apollo 2. The pattern across all three efforts is identical: take a model that already learned to associate pixels with language at internet scale, and treat physical action as one more modality the same transformer can be taught to output next.