PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptions

AI summary

Overview

Research area: Computer Vision / Natural Language Processing — specifically evaluation metrics for vision-language models (VLMs) and detailed image description.

Technical level: Intermediate. Readers should be comfortable with scene graphs, LLM-as-a-Judge paradigms, and correlation-based metric evaluation (Spearman ρ, Kendall's τ, F1).

Scope: The paper introduces PoSh, an open-weight reference-based metric that uses scene graphs as structured rubrics to score long image descriptions, and DOCENT, a new human-annotated benchmark of artwork descriptions used to validate it.

What This Paper Is About

Standard image captioning metrics like CIDEr and SPICE were built for short captions and tuned to catch errors (such as object misidentification) that modern VLMs rarely make. Long, detailed descriptions instead fail on subtler issues like attaching the wrong attribute or relation to an object, and existing metrics give only a single opaque score with no indication of where a description went wrong. PoSh addresses this by extracting scene graphs from a generated description and its reference, using them as rubrics for an open-weight LLM to localize mistakes and omissions, then aggregating those granular findings into interpretable coarse scores.

Key Contributions

1. PoSh, a new metric for detailed image description. It produces coarse scores for mistakes (precision) and omissions (recall) that are grounded in granular, span-localized scores, making evaluation both interpretable and fully replicable using open-weight models.
2. DOCENT, a new benchmark for artwork description. It contains 1,750 public-domain works of art from the U.S. National Gallery of Art with expert-written accessibility descriptions, 400 model generations from four VLMs, and 900 human judgments (300 granular, 600 coarse) from 24 art-history students at two levels of resolution.
3. Empirical validation of PoSh. PoSh correlates more strongly with human judgments than existing metrics and even GPT4o-as-a-Judge, and the advantage holds on CapArena, a separate dataset of web imagery.
4. PoSh as a reward function and a leaderboard. Reinforcement learning with PoSh as the reward outperforms supervised fine-tuning on DOCENT, and the paper uses PoSh to rank open and closed models on the new artwork description task.

Main Findings

• Better correlation with humans on DOCENT: PoSh achieves a 0.11 increase in Spearman ρ for mistakes (25% relative), 0.07 for omissions (14%), and 0.05 for overall quality (9%) over the next-best replicable metric. It surpasses GPT4o in all settings and text-only GPT5 on omissions and overall quality.

• Strong granular error localization: PoSh reaches the highest F1 in localizing mistakes (0.580) and omissions (0.680) against baselines including random, 4-gram embedding comparison (0.483 / 0.641), and scene-graph embedding comparison (0.514 / 0.658). Because coarse scores are averaged from these, the coarse scores inherit interpretability.

• Robust to image type (CapArena): On a separate benchmark of web imagery, PoSh achieves model-ranking correlations of ρ = 0.931 and τ = 0.796, beating nearly every open-weight metric and GPT4o. The only exception is LLaVA-Critic, a much larger VLM-as-a-Judge — and on the harder subset containing three or more people in the scene, PoSh overtakes it (ρ = 0.727 vs. 0.686).

• Effective reward function: Training Qwen2.5-VL-7B with DAPO using PoSh as reward produces generations with more mistakes on average (-0.243) but far fewer missing details (+0.432), yielding higher overall quality (+0.135) than supervised fine-tuning.

• Detailed description remains hard for VLMs: Even the best model evaluated, GPT4o, covers only 50.1% of the visual information in DOCENT's references. All models make relatively few mistakes but struggle badly with omissions.

• Efficient and validated internally: PoSh scores 400 DOCENT examples in 15 minutes on a single H100 (about 2 seconds each), versus roughly 2 hours for DCScore. Its scene graph extraction reaches 0.892 element F1 and its element verification reaches 0.852 F1 against hand annotations.

Methodology in Plain English

PoSh works in three steps.

First, it turns each description — both the model's output and the human-written reference — into a scene graph: a structured list of objects (a man, a bird), attributes on those objects (a "tall" man), and relations between objects (a man "pouring" water). It builds these sentence by sentence using off-the-shelf dependency parsing, then merges objects across sentences with coreference resolution. Each element stays linked to the original text span, which is what makes the scoring interpretable.

Second, it checks each element from one scene graph against the other description using question answering. For example, it asks whether the reference's "trio of figures" appears in the generation. Crucially, it asks questions rather than forcing one-to-one matching, so a generation that describes three people individually is not penalized for failing to use the word "trio." To handle repeated object classes, it tests several candidate identifiers ("man," "tall man," "face of tall man") in three passes and picks the simplest one confirmed present. The QA scorer, Qwen-3-14B, predicts a number from 1 to 5 via weighted averaging over token logits.

Third, it averages those granular scores into coarse mistake and omission scores. The researchers deliberately leave the weights unweighted to demonstrate robustness rather than tuning them per dataset.

To validate the metric, they built DOCENT. Twenty-four art history students — half doing granular span annotation, half doing pairwise coarse rankings — judged generations from LLaVA-1.6-7B, Molmo-D-7B, GPT4o, and Claude Sonnet 3.5 describing 100 artworks. They then compared PoSh and a wide set of baselines (n-gram metrics, SPICE, CLIPScore, CAPTURE, several LLM/VLM judges) against those judgments.

Why This Matters

Impact on research: Detailed image description is one of the few remaining tasks where VLMs are far from saturated, but progress has been bottlenecked by the absence of a cheap, reliable, granular scoring method. PoSh gives researchers a metric that localizes errors to specific text spans, so model and prompt iteration can target the actual failure modes instead of requiring manual inspection. DOCENT also fills a gap: most detailed-description benchmarks release no human judgments at all, and none had prior granular span-level annotations.

Real-world applications:

• Assistive text generation — automatically producing high-quality alt text for images, a major accessibility need for blind and low-vision users online.
• Museum and cultural heritage access — describing artworks so that collection websites and audio guides can serve visually impaired visitors, which is the direct use case behind the National Gallery of Art's involvement.
• Reinforcement learning for vision-language models — PoSh functions as a dense, span-level reward signal for post-training, including potential token-level guidance.
• Automated quality assurance — auditing large volumes of model-generated descriptions for precision and recall failures without paying for human review.

Industry relevance: The metric is fully open-weight and replicable, unlike API-dependent judges such as GPT4o or GPT5. It runs roughly 12 times faster than a comparable GPT4-based metric on the same hardware, which matters for any team that needs to score thousands of generations during training or evaluation loops.

Future Directions

• Validate against structure-guided generation. Some methods extract structure from images during generation. Since PoSh extracts structure from text, the authors expect no bias, but they flag this as requiring experimental confirmation once those models become available.
• Add tunable weights. Coarse scores are currently unweighted averages. Introducing per-dimension weights, as CAPTURE does, could adapt PoSh to specific domains or user priorities.
• Explore token-level reward guidance. Because PoSh produces localized granular scores, it could plausibly guide reinforcement learning at the token level rather than only at the sequence level.
• Scale synthetic reference generation. Recent work on generating detailed synthetic descriptions suggests PoSh-tuning could be scaled cheaply without new expert-written references.

Target Audience

This paper is most useful for researchers and engineers working on vision-language model evaluation, detailed captioning, or assistive text generation. It will also interest practitioners who need a deployable, low-cost scoring function for long-form image descriptions — particularly those in accessibility, cultural heritage technology, or model post-training. Benchmark designers will find the DOCENT annotation protocol (granular span annotation plus pairwise coarse ranking, with an expert-validated agreement analysis) directly reusable. Readers do need some familiarity with scene graphs and metric-evaluation conventions to follow the experiments, but the core idea — using structure to make LLM judges more precise — is accessible without deep background.

Authors’ abstract

While vision-language models (VLMs) have advanced into detailed image description, evaluation remains a challenge. Standard metrics (e.g. CIDEr, SPICE) were designed for short texts and tuned to recognize errors that are now uncommon, such as object misidentification. In contrast, long texts require sensitivity to attribute and relation attachments and scores that localize errors to particular text spans. In this work, we introduce PoSh, a metric for detailed image description that uses scene graphs as structured rubrics to guide LLMs-as-a-Judge, producing aggregate scores grounded in fine-grained errors (e.g. mistakes in compositional understanding). PoSh is replicable, interpretable and a better proxy for human raters than existing metrics (including GPT4o-as-a-Judge). To validate PoSh, we introduce a challenging new dataset, DOCENT. This novel benchmark contains artwork, paired with expert-written references, and model-generated descriptions, augmented with granular and coarse judgments of their quality from art history students. Thus, DOCENT enables evaluating both detailed image description metrics and detailed image description itself in a challenging new domain. We show that PoSh achieves stronger correlations (+0.05 Spearman $ρ$) with the human judgments in DOCENT than the best open-weight alternatives, is robust to image type (using CapArena, an existing dataset of web imagery) and is a capable reward function, outperforming standard supervised fine-tuning. Then, using PoSh, we characterize the performance of open and closed models in describing the paintings, sketches and statues in DOCENT and find that foundation models struggle to achieve full, error-free coverage of images with rich scene dynamics, establishing a demanding new task to gauge VLM progress. Through both PoSh and DOCENT, we hope to enable advances in important areas such as assistive text generation.