When More Metadata Misleads: Probing Field-Level Grounding in LLM-Generated Museum Descriptions

AI summary

Overview

Research area: Natural language processing — metadata-conditioned data-to-text generation, with a focus on cultural heritage (museum catalogue records) and multimodal LLM faithfulness.

Technical level: Intermediate. Readers should be comfortable with standard NLG evaluation metrics (BLEU, ROUGE, METEOR, BERTScore), NLI-based entailment scoring, and basic statistical testing; the paper's conceptual framing, however, is accessible without deep technical background.

Scope: A two-phase study on the TimeTravel benchmark (10,250 museum artefact records across ten geographic regions) examining how metadata quantity, field type, image availability, and place-specific documentation patterns shape what LLMs include, omit, and infer when writing object descriptions.

What This Paper Is About

Metadata-conditioned generation is usually treated as a data-to-text problem where adding more input fields should improve grounding. The authors argue that museum catalogue records break this assumption: fields differ in epistemic status (some describe the object directly, others describe its provenance and circumstances), records are unevenly populated across regions, and images can support only some kinds of claims. The paper asks how metadata quantity, field type, image availability, and place-level documentation patterns affect what LLMs include, omit, and infer — and shows that field-level and claim-level evaluation is needed because aggregate input coverage and reference similarity hide distinct failure modes.

Key Contributions

1. A controlled field-type probing framework. The authors introduce a within-object experimental design that holds object identity and prompt template fixed while independently varying field count (3 to 6 fields, plus bounding conditions), field type (context versus content fields), and image access, across ten generation conditions.

2. Evidence that more metadata does not improve faithfulness monotonically. Increasing field count improves coverage of provided fields but also increases unsupported inference over withheld fields, contradicting the standard data-to-text assumption that richer input yields better grounding.

3. A demonstration that context and content fields fail through different mechanisms. Context fields (Find spot, Place, Assoc name) are grounded reliably when present and rarely inferred when absent; content fields (Subjects, Technique, Inscription) are verbalised whenever provided but carry the highest hallucination risk.

4. Evidence that place-level variation reflects field–region interactions and documentation practice, motivating field- and claim-level evaluation rather than aggregate input coverage or reference similarity for metadata-conditioned generation.

Main Findings

• Surface metrics invert documentation completeness in natural data. In Phase 1, mean description length rises from 160 to 213 words across 3–9 fields while mean BERTScore-P falls from 0.863 to 0.839. Output length increases with field count (η² = 0.33), and length correlates negatively with BERTScore across civilisations (r = −0.34 to −0.70). Surface-metric rankings almost perfectly invert documentation completeness (Kendall's W = 0.990), so well-documented regions score lower despite richer records.

• Holding object identity fixed reverses the length–quality relationship. In Phase 2, BERTScore rises from 0.885 at n = 3 to 0.901 at n = 7, then falls to 0.890 at n = 9, while word count rises monotonically from 100 to 135. Within-place field-count–BERTScore correlations shift from negative in every region in Phase 1 to positive in all ten regions in Phase 2 (ρ = 0.200 to 0.421).

• Primary grounding is flat with field count; bonus inference grows. Primary-field coverage stays between 0.90 and 0.93 and primary hallucination between 0.07 and 0.10 across field counts, with silence near zero. Bonus silence falls from 0.79 at n = 2 to 0.65 at n = 6, and bonus wrong exceeds bonus correct at every field count (wrong stable at 0.18–0.24 against correct rising from 0.05 to 0.19).

• Field type determines the dominant failure mode. The context arm (S_ctx4–6) reaches primary coverage of 0.937–0.949 with hallucination of 0.040–0.044; the content arm (S_cont4–6) reaches lower coverage (0.867–0.893) with higher hallucination (0.102–0.129). Paired permutation tests (10,000 resamples, BH–FDR corrected) show S_ctx exceeds S_cont in primary coverage by 5.6–7.4 pp and in bonus wrong by 15.5–34.5 pp, and falls below it in primary hallucination by 6.2–8.5 pp (all p < 0.001).

• Content fields are verbalised but misstated; context fields are omitted. Subjects and Technique have the highest primary hallucination rates (24.2% and 21.3%), far above Culture (1.3%) and Production date (1.7%). Find spot and Place show much lower hallucination (3.9% and 5.6%) but higher silence (5.0% and 8.1%). Inscription is the only field where bonus correct exceeds bonus wrong (16.7% vs 10.6%).

• Withheld fields are more often guessed wrong than recovered. The highest bonus wrong rates are Technique (51.7%) and Subjects (41.8%). Withheld context fields are usually simply omitted: Find spot has 99.2% bonus silence and 0.4% bonus wrong; Place and Assoc. name show bonus silence of 86.8% and 88.0%. Along the context arm, bonus wrong rises from 0.308 (S_ctx4) to 0.421 (S_ctx6) while bonus silence falls from 0.577 to 0.360.

• Regions differ in failure mode, not overall reliability. BERTScore varies significantly across regions (Kruskal–Wallis H > 100, p ≪ 0.001) but over a narrow range, from 0.877 (Greece) to 0.896 (Central America); a negative length–BERTScore correlation holds within every region (r = −0.38 to −0.71). Claim-level rates separate regions more sharply: the British Isles has the highest primary coverage (0.936) and lowest hallucination (0.054), Central America the highest hallucination (0.112), and China the lowest coverage (0.859) with the highest silence (0.047). Technique hallucination ranges from 0 (Roman Empire, where the field is absent from records) to 0.46 (Central America); for Greece, models generate bonus Technique content in 71% of cases, of which 0.69 are wrong against 0.04 correct.

• Images help sparse grounding but increase extrapolation. Removing the image under sparse metadata (S_noi vs S) lowers BERTScore, drops primary coverage from 93.3% to 88.5%, and raises primary hallucination from 5.8% to 9.1% (4.8 pp, 3.3 pp, and 7.7 pp bonus wrong; all p < 0.001). The same ablation under full metadata (R vs T) yields no significant difference (0.6 pp coverage, 1.0 pp hallucination). Ablation raises bonus silence from 68.5% to 79.9% and lowers bonus wrong from 22.3% to 14.6%, with the wrong-to-correct ratio rising from 2.4:1 to 2.7:1.

• Evaluation layers disagree. Reference similarity, NLI-estimated field coverage, and claim-level verification capture different aspects of quality and can rank matched conditions differently, so a difference in error type can be misread as a difference in quality. NLI also measures semantic consistency rather than value accuracy: a description can receive an entailment verdict without correctly reproducing the catalogue value.

• Pipeline validation against humans. Two annotators labelled 350 stratified claims (Cohen's κ = 0.78; 96% agreement for supported, 87% for hallucinated). Human–pipeline reliability was κ_w = 0.67, with 89% agreement for supported and 73% for hallucinated.

Methodology in Plain English

The work runs in two phases on the TimeTravel benchmark, which contains 10,250 museum artefact records from ten regions (Iraq, Egypt, Greece, Roman Empire, British Isles, China, Japan, Central America, India, Iran). Each record has structured metadata, an image, and an expert-written reference description. Nine fields are analysed: Materials, Culture, Production date, Find spot, Place, Associated name, Subjects, Technique, and Inscription. Objects average 6.8 populated fields (range 3–9), and availability varies sharply by region.

Phase 1 is observational: each of four LLMs is prompted with every object's naturally available fields and image, producing descriptions for all 10,250 records. This shows how quality varies in realistic conditions, but field count, field type, image availability, and region all co-vary, so causes cannot be separated.

Phase 2 fixes that with a controlled probe over 500 objects balanced across the Place field. Every object is reused under ten prompt conditions, making comparisons paired within object. A three-field baseline (S: Materials, Culture, Production date) is extended along two matched arms: a context arm adding Find spot, then Place, then Assoc name (S_ctx4–6), and a content arm adding Subjects, then Technique, then Inscription (S_cont4–6). Field order is fixed, and matched levels differ only in field type. Additional conditions bound the range (R with all available fields plus image, T with all fields but no image) and ablate the image at sparse metadata (S_noi).

Generations are scored three ways. Reference similarity uses BLEU-1, ROUGE-L, METEOR, and BERTScore (roberta-large). Field-value support uses DeBERTa-large-MNLI to test whether a description entails each catalogue field value, distinguishing provided ("primary") fields from withheld ("bonus") fields. Claim-level factuality uses a two-stage nugget pipeline: stage 1 splits a description into atomic claims and attributes each to a catalogue field or to "extra content"; stage 2 labels each claim as supported, hallucinated, or not_verifiable relative to the catalogue record. Labels are defined against the record, so a claim can be historically or visually correct and still count as hallucinated if the target field does not support it. Significance is assessed with paired permutation tests over matched field counts.

Why This Matters

Impact on research. The paper reframes metadata-conditioned generation as a schema-alignment problem: models must learn not only what information is available but what each field permits them to assert. It shows that aggregate input coverage and reference similarity can conceal — or invert — real differences in faithfulness, and it supplies a reusable experimental template (fixed object identity, matched field-count arms, claim-level verification) for separating quantity, type, and modality effects.

Real-world applications:

• Museum and gallery cataloguing: automated draft descriptions for public catalogues, where unsupported elaboration could present speculation as authoritative object information.
• Cultural heritage aggregation platforms: quality assurance for records drawn from many institutions with uneven documentation, since a single global accuracy threshold is not adequate across regions.
• Digital libraries and archives: provenance-aware description of digitised materials where images and catalogue metadata support different kinds of claims.
• Multimodal retrieval and accessibility tools: generating object descriptions that must be traceable to source records rather than to model priors.

Industry relevance. Any deployment that converts structured records plus images into public-facing text — e-commerce catalogues, real-estate listings, product data sheets, clinical or regulatory summaries — faces the same trade-off the paper documents: more input fields increase coverage but also increase claims that outrun the source. The finding that images help under sparse metadata but encourage extrapolation under rich metadata is directly relevant to teams deciding how much context to feed a generator, and the claim-to-field attribution approach suggests a route to provenance labelling that lets reviewers separate source-supported content from model-supplied content.

Future Directions

• Dynamic field categorisation. The context/content split may not be absolute: Materials is usually grounded reliably but can still trigger unsupported object-type or quality inferences, and field behaviour may depend on object type, record format, and cataloguing practice. The authors call for categories defined dynamically from both schema role and object-specific usage.

• Expert rather than crowd annotation. Claim-level factuality was validated with an automatic pipeline plus two crowdworkers rather than curators; expert review is proposed, especially for visually plausible claims not recorded in the catalogue.

• Separating support sources. Catalogue metadata is treated as the authoritative reference schema, but records may omit details visible in images or known externally. Future work should distinguish catalogue-supported, image-supported, and externally supported claims rather than treating all unsupported content as equivalent.

• Better modelling of what humans want. Expert references are selective rather than exhaustive, which partly explains the divergence between reference similarity and field-level support. Open question: how do humans and domain experts balance descriptiveness, visual plausibility, and catalogue fidelity?

Target Audience

Researchers and practitioners in natural language generation and data-to-text who care about faithfulness and evaluation design; digital humanities and cultural heritage informatics specialists working with heterogeneous museum metadata; and applied teams building metadata- or image-conditioned description systems who need to reason about when additional input improves grounding and when it invites unsupported claims. The paper is also useful for evaluation-methodology readers interested in why reference-based metrics and NLI coverage can rank the same conditions differently.

Authors’ abstract

Metadata-conditioned generation is often framed as a data-to-text task in which additional input fields should improve grounding. Museum description generation challenges this assumption because catalogue records are heterogeneous, partially visual, and unevenly documented across geographic regions. We study how metadata composition shapes the generation of artefact descriptions through a two-phase framework. Phase 1 analyses natural metadata variation in a large museum collection, where field count, field type, image availability, and Place co-vary. Phase 2 introduces controlled field probes that hold object identity fixed while varying metadata quantity, field type, and image access. Our results show that metadata quantity alone does not explain generation behaviour. Field type determines whether models ground, omit, or infer information, while images can both support visual grounding and increase unsupported interpretation. More broadly, our findings show that metadata-conditioned data-to-text evaluation should assess field- and claim-level support, rather than relying only on aggregate input coverage or reference similarity.