Skip to content
AI.info

Computer vision

Vision-Language Models and Grounding

Understand dual encoders, cross-modal fusion, captioning, visual question answering, grounding, open-vocabulary recognition, and multimodal evaluation.

By the end you can

Language turns visual categories into an open interface

A fixed classifier recognizes only the labels chosen during training. A vision-language model can compare images with new textual descriptions, retrieve evidence, or answer questions.

That flexibility comes from paired data and prompt interpretation. It does not mean the model has inspected every claim it can express. The rest of this lesson is about the distance between those two things. Every step of it has been measured by someone: the corpus the pairs came from, the objects a captioner names that are not there, the benchmarks built to catch that and then audited themselves, and the questions a real user asks that no image can answer.

Open vocabulary expands the interface, not the guarantee of visual grounding.

Comparison

Two major ways to combine vision and language

The architecture determines whether modalities meet early, late, or repeatedly.

Two named systems put numbers on the trade-off. Flamingo is fusion in its most expensive form. A frozen vision encoder and a frozen language model are joined by cross-attention layers interleaved between the pretrained language-model layers, handling “sequences of arbitrarily interleaved visual and textual data”. Flamingo-80B carries 80B parameters and is built on a 70B Chinchilla. DeepMind published it in 2022, and the abstract claims that “On numerous benchmarks, Flamingo outperforms models fine-tuned on thousands of times more task-specific data”. That is the case for the fusion column: conditional reasoning bought with scale.

The cost half of the column is a design decision rather than a law. BLIP-2, published by Salesforce Research in 2023, bridges an off-the-shelf frozen image encoder and a frozen LLM with a lightweight Querying Transformer. Its abstract reports that “our model outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters”. A different laboratory measured itself against DeepMind's system on the same zero-shot benchmark and moved the price by a factor of 54. Read the serving-cost row below as a choice someone made, not as the price of fusion.

FigureComparison · 3 columns

Dual encoder

Images and text are encoded separately into a shared space.

  • Efficient large-scale retrieval
  • Precompute gallery embeddings
  • Weak token-region interaction
  • Example: image-text search

Fusion model

Image features and language tokens interact through cross-attention or joint blocks.

  • Detailed conditional reasoning
  • Supports generation and VQA
  • Higher serving cost
  • Example: grounded question answering

Hybrid pipeline

A retriever narrows evidence before a fusion model analyzes it.

  • Balances scale and detail
  • Makes evidence selection explicit
  • Creates multiple failure stages
  • Example: catalog assistant

Visual

A ladder from association to grounding

Multimodal outputs become more testable as they connect claims to visible regions.

The vendor of a deployed system has written down where the ladder ends. OpenAI's GPT-4V(ision) System Card, of 25 September 2023, defines an ungrounded inference as one “not justified by the information the user has provided—in the case of GPT-4V, this means information contained in the image or text”. That is the test each rung is approaching. Not whether the sentence is plausible, or fluent, or usually right, but whether the information behind it was in the image or the text the user actually supplied. The same card reports steering the model to refuse person-identification requests “more than 98% of the time”. That is a rung the system is deliberately kept off.

FigureHierarchy · 5 levels

  • Global association

    The image and phrase are judged broadly compatible.

    • Attribute recognition

      The model identifies properties such as color, count, or material.

      • Region reference

        A phrase is linked to a box, point, or mask.

        • Relational grounding

          The system connects multiple entities and spatial relations.

          • Evidence-backed answer

            The final claim includes source regions and uncertainty.

Example

What paired image-text data can silently teach

Captions are selective descriptions, not neutral inventories of pixels — and at web scale nobody writes them for you. The documented case is LAION-5B, the paired image-text corpus behind much of this generation of models. It was built by snapshotting Common Crawl, reading each image's HTML "alt" attribute as its caption, and using CLIP to discard the pairs where text and image did not match. There is no human step in that description because there was not one.

Then someone counted. On 23 December 2023 David Thiel, at the Stanford Internet Observatory, published an audit that identified 3,226 dataset entries of suspected CSAM inside LAION-5B. The report was later entered into the record of a hearing of the US House Committee on the Judiciary. LAION withdrew the dataset. On 30 August 2024 it published Re-LAION-5B — 5,526,641,167 text-link-to-image pairs — after removing 2,236 links, and named the overlap with the audit that had prompted it: “These links also subsume 1008 links found by the Stanford Internet Observatory report in Dec 2023.”

Hold those two numbers together. 3,226 found by an outside auditor; 2,236 links removed, of which 1,008 were the auditor's. The point about paired data stops being a caution and becomes an inventory problem. Nobody, including the people who built it, could say what was in the corpus until someone else went looking.

  • Alt text, not description: LAION-5B's captions are the HTML "alt" attributes that happened to sit beside each image in a Common Crawl snapshot, written for search ranking or screen readers rather than to inventory the pixels.
  • The filter is a model: CLIP itself discarded the pairs whose text and image did not match, so the corpus inherits whatever CLIP already believed about which words belong to which pictures.
  • Scale hides content: 3,226 suspected-CSAM entries sat inside a corpus of billions of pairs, and were found by an outside report rather than by the pipeline that assembled it.
  • Missing negatives: Unmentioned objects are not necessarily absent — an alt attribute names a product or a subject, not everything in frame.
  • Cultural convention: Attributes are named differently across communities and languages, and a scraped caption carries the conventions of whichever page it was lifted from.
  • Social bias: Occupations or activities may be tied to demographic stereotypes in the harvested text as well as in the images, and no step in this pipeline was looking.

Analogy

A museum guide answering beside a painting

A museum guide can discuss the whole painting, point to a figure, and compare it with the label on the wall. A good answer depends on seeing the work and using the label, without letting what the guide expects replace what the guide sees.

A visitor standing at the painting can tell when the guide is talking past it. A model's language fluency can hide weak visual evidence. Global description, reference, and grounded explanation stay separate either way. The visitor's advantage is the whole asymmetry: they can look. A user who asks a model what is on a restaurant menu cannot, and the next section is what that costs.

The stronger the claim, the stronger the connection to visible evidence should be.

Key idea

A plausible noun can enter through language priors

When an object is small, obscured, or absent, a model may complete the scene using common co-occurrences. A kitchen image invites “oven,” and a street scene invites “traffic light,” even when those objects are not visible.

Test counterfactual crops, object removal, and unanswerable questions. Require abstention when visual evidence is insufficient.

Object hallucination in captioning has been measured directly, not just described. In 2018 Rohrbach and four colleagues proposed “a new image relevance metric to evaluate current models with veridical visual labels and assess their rate of object hallucination”, investigated it “on the standard image captioning benchmark, MSCOCO, using a diverse set of models”, and found “that models which score best on standard sentence metrics do not always have lower hallucination and that models which hallucinate more tend to make errors driven by language priors”. Fluency was not the tell. The standard sentence metrics did not catch it.

Then fluency improved, and the problem got worse. POPE, published in 2023 by six researchers at Renmin University of China, re-ran the CHAIR metric over MSCOCO on five instruction-tuned large vision-language models. Under the instruction “Provide a brief description of the given image”, mPLUG-Owl scores CHAIR_S 76.8 and LLaVA 62.7, against 13.0 for OSCAR-base, 10.5 for VinVL-Large and 8.8 for BLIP-Large. The newer, more articulate systems invent objects at roughly seven times the rate of the small captioners they replaced. The objects they invent are the ones that appear or co-occur most often in the visual instruction data.

An independent group had already reported the same disconnect between sentence quality and faithfulness: “First, we examine recent state-of-the-art VLP models, showing that they still hallucinate frequently, and models achieving better scores on standard metrics (e.g., CIDEr) could be more unfaithful.” Their paper is titled Plausible May Not Be Faithful. The title is the finding.

This reaches users with something at stake. The same GPT-4V system card documents the Be My Eyes deployment: piloted from March to early August 2023 with nearly 200 blind and low-vision beta testers, and grown by September to 16,000 users requesting a daily average of 25,000 descriptions. Testers reported that the model “can make basic errors, sometimes with misleading matter-of-fact confidence”. One of them described what that looks like from the other side: “It very confidently told me there was an item on a menu that was in fact not there.”

Be My Eyes, announcing Be My AI the same day, published the same warning in its own words — “You may still get hallucinations, occasional wrong answers, and experience some frustrations” — and then drew the line where confident error stops being survivable: “Be My AI does not and should not replace a white cane, guide dog, or other mobility aid that provides for safe travel”.

Multimodal hallucination often looks like confident scene completion.

Steps

Build a multimodal evaluation suite that resists fluent shortcuts

Measure the visual and linguistic parts separately before trusting what they do together.

Benchmarks have been built to defeat exactly these shortcuts, and the instructive part is what happened to them afterwards. Goyal and four colleagues rebalanced VQA in 2017 by “collecting complementary images such that every question in our balanced dataset is associated with not just a single image, but rather a pair of similar images that result in two different answers to the question”. The rebalanced set “has approximately twice the number of image-question pairs”. And the models fell over: “All models perform significantly worse on our balanced dataset, suggesting that these models have indeed learned to exploit language priors.”

Winoground holds the vocabulary fixed instead. In 2022 seven researchers at Meta AI hand-curated 400 items: “Given two images and two captions, the goal is to match them correctly - but crucially, both captions contain a completely identical set of words, only in a different order”. Probing “a diverse range of state-of-the-art vision and language models”, they “find that, surprisingly, none of them do much better than chance”.

Then the benchmark was audited by people who had not built it. Five researchers at UT Austin re-annotated all 400 items and tagged what each one actually demands: VisuallyDifficult n=38, UnusualImage n=56, UnusualText n=50, ComplexReasoning n=78. Their conclusion, in 2022: “Thus, we conclude that not all Winoground items test what they aim to, and identify a subset of 171 items which directly measure compositionality.” Performance on those 171 vanilla items was still low, so the headline held. But it took a second paper from a second institution to know which of the 400 numbers had been measuring word order, and which had been measuring whether a model could resolve a small out-of-focus object.

Size does not settle it either. The Attribution, Relation and Order benchmark — ARO, out of Stanford in 2023 — is “orders of magnitude larger than previous benchmarks of compositionality, with more than 50,000 test cases”. On its relation tests “chance level performance is 50%”, and “In relation tests, we observe that most models are near or below chance level, indicating severe deficiencies in relational understanding”. Concretely: “For example, BLIP chooses 'the grass is eating the horse', with 81% probability”.

A year later a group at the University of Washington and the Allen Institute for AI asked whether such benchmarks needed the image at all. SugarCrepe found “significant biases in all these benchmarks rendering them hackable”, and that the reported compositionality improvements “were hugely overestimated”: “This hackability is so dire that blind models with no access to the image outperform state-of-the-art vision-language models.” A text-only model that never saw a pixel beat the vision-language systems. Step 4 therefore has an extra requirement. Print a blind baseline beside every compositionality score, or the score is unreadable.

Step 5 has a base rate rather than a warning. In VizWiz, the one large VQA dataset collected from people who needed the answer, 28.63% of the visual questions are not answerable at all. The next section works through what that means for a test set.

FigureProcess · 5 steps

  1. 1. Test retrieval

    Measure image-to-text and text-to-image ranking with hard negatives.

  2. 2. Test attributes and counts

    Use controlled examples where language priors conflict with pixels.

  3. 3. Test grounding

    Score boxes, points, masks, and phrase-region consistency.

  4. 4. Test compositionality

    Vary subject, attribute, relation, and negation independently.

  5. 5. Test refusal

    Include blurry, cropped, absent, and genuinely unanswerable evidence.

Key idea

Zero-shot labels are part of the model

Changing a class name, article, template, or synonym can change similarity scores. Prompt ensembles may improve averages while hiding label ambiguity.

Document the exact text prompts, language, and aggregation method. Evaluate class descriptions with domain experts when terminology is specialized.

CLIP's authors measured the size of the effect on their own headline number. The paper reports that “we ensemble 80 different context prompts and this improves performance by an additional 3.5% over the single default prompt discussed above. When considered together, prompt engineering and ensembling improve ImageNet accuracy by almost 5%.” A figure caption in the same paper gives the gain as an average across many datasets rather than as the ImageNet figure: “Compared to the baseline of using contextless class names, prompt engineering and ensembling boost zero-shot classification performance by almost 5 points on average across 36 datasets.” The comparison they reach for is hardware. The improvement “is similar to the gain from using 4 times more compute with the baseline zero-shot method but is ‘free’ when amortized over many predictions.”

None of that would be checkable if the templates had stayed private. They ship with the released OpenAI CLIP code, in a notebook that names “the full 80 ensemble reported in the paper”. That is the only reason anyone else can reproduce the 3.5%. Change the words and the number changes. The prompt list is part of the result.

A zero-shot benchmark includes a prompt design, not just an image model.

Example

Practice: design a grounded maintenance assistant

Technicians photograph industrial control panels and ask which indicator requires attention.

Before designing the tests, look at what happened the one time a large VQA dataset was collected from users with a real need rather than from crowdworkers with a prompt. The 2018 paper describes it in one sentence: “VizWiz consists of over 31,000 visual questions originating from blind people who each took a picture using a mobile phone and recorded a spoken question about it, together with 10 crowdsourced answers per visual question.” The collection is 31,173 image/question pairs, and 28.63% of the visual questions are not answerable — the image is blurred, the object is out of frame, the label is turned away. That is the judgement of at least half the crowdworkers, marking the answer "unanswerable" or the image "unsuitable". Ernest Davis, of New York University, restated the figure independently in Frontiers in Artificial Intelligence in 2020: “As a result, 28% of the questions in VizWiz are unanswerable”.

A technician holding a phone at arm's length in a hot plant room is in the position of those photographers, not of a benchmark author choosing a clean image. Refusal is not the exotic case at the end of your test plan. On the only evidence available it is close to three questions in ten.

  • Define which answers require a localized box or point.
  • Size the unanswerable set from the base rate rather than from intuition: VizWiz found 28.63% of real users' questions unanswerable, so three out-of-frame gauges in a hundred clean photographs is not a test of refusal.
  • Add a hard negative panel with a visually similar but harmless warning light.
  • Specify how manuals or labels enter the answer without overriding the image.
  • Choose one metric for grounding, one metric for technician outcome, and a blind baseline — a text-only model that never sees the photograph — so you can tell a grounded score from a hackable one.

Key takeaways