RealityBench: Simulated Environments to Evaluate a Vision Language Model's Ability to Document Human Physical Work

AI summary

Overview

Research area: Computer vision and vision-language model (VLM) evaluation, specifically benchmarking perception for physical-work documentation (rasterized/simulated 3D environments, sim-to-real transfer, human-agreement validation).

Technical level: Intermediate. The paper is readable without a vision research background, but familiarity with vision-language models, F1 scores, Spearman rank correlation, and benchmark design helps.

Scope (1 sentence): The paper introduces RealityBench, eleven Blender-rendered work environments from rooftop solar installation and vehicle repair, and uses them to evaluate 26 vision-language models on two tasks — judging whether a photo is acceptable proof of work, and reading task-relevant state from that photo — with a sim-to-real check against real photographs.

What This Paper Is About

AI assistants that watch a worker through a phone or smart glasses could help document physical work, but nobody knows how well current vision-language models actually do at this. Real field data is proprietary and too variable to study carefully, so the authors built eleven simulated work scenes in Blender where every frame can be scored exactly by a program. The goal is to measure, in a reproducible way, whether models can (a) tell if a photograph is good enough to serve as proof-of-work and (b) extract the information a documentation record needs from that photograph.

Key Contributions

1. Eleven programmatically scored simulated environments. Built in Blender 4.5.12 across rooftop solar installation (seven environments: rail_attachment, wire_adequacy, tape_gap, slope_gauge, equipment_label, horizon_peak, breaker_label) and vehicle repair (four: car_view, damage_view, car_plate, ro_board). Each has a human-validated programmatic acceptability scorer plus a documentation scorer derived from scene ground truth.

2. A 26-model evaluation on 1,378 frames. Reports acceptability F1, documentation accuracy, the cost of running the whole set at list API prices, mean thinking tokens, and grammar-conformance rates, alongside a single human rater as a comparison point.

3. A sim-to-real transfer study. On three matched documentation questions — tape-measure reading (a newly released public set of 133 readings on 84 photos of 35 items across 12 scenes), car damage severity (63 real damaged-panel photos), and which face of a car is shown (117 real frames) — model rankings on rendered frames are compared to rankings on real photographs.

4. An error analysis separating requirement errors from scene errors. In three environments where the acceptability and documentation tasks overlap, the authors trace whether wrong rejections come from misreading the instruction or misreading the image, and re-ask 150 frames to nine models with the verdict moved last or removed to test whether reported defects are post-hoc justifications.

Main Findings

• The scorer is roughly as reliable as a human rater. On 95 held-out frames with at least three ratings, the tuned scorer matches the raters' majority on 82% of frames (95% CI 75–90%). Individual raters match the majority of the others 76–92% of the time (67% for the author who rated only two environments, on 18 frames). The scorer errs strict: it accepts 4 of the 62 majority-rejected frames and rejects 13 of the 33 majority-accepted ones.

• Wide spread across models. Acceptability F1 ranges 0.39–0.61 and documentation 0.62–0.84. The two rankings agree at Spearman ρ = 0.69, p < 0.001.

• Leaders on each track. GPT-6 Astra leads acceptability (0.61, $30.05), with Claude Opus 5.5 (0.58, $16.36), GLM-5.3 Flash (0.57, $1.30) and Gemini 3.5 Flash (0.56, $7.36) in the top group. Gemini 3.8 Flash leads documentation (0.84, $7.16), with Claude Opus 5.5 (0.83), GPT-6 Astra (0.83), Claude Fable 5.1 (0.83, $41.35) and Qwen 3.8 Max (0.82, $38.19) in the top group. Only GPT-6 Astra and Claude Opus 5.5 appear in both top groups.

• Routing helps documentation, not acceptability. Picking the best model per environment (chosen on the other half of frames) reaches documentation 0.86 ($17.90) — above every single model — but acceptability F1 only 0.60 ($20.60). The majority vote over all models reaches F1 0.60 and documentation 0.83 at $322, on par with or worse than the best single model.

• Three separate cost bands. Models costing over $4 for the set ($4.80–41.35); six under $2 ($0.69–1.30, "moderately priced"); and the three self-served, unpriced models. In the moderately priced band, GLM-5.3 Flash (0.57, $1.30) and GPT-5.6 Luna (0.53, $1.18) lead acceptability and Gemma 4 31B (0.79, $0.70) leads documentation alone. Two of these three are open-weight.

• Small self-hosted models are usable but behind. LFM2.5-VL-3B and Gemma 4 E4B reach F1 0.43 and 0.42, and documentation 0.68 and 0.69 — on par with the lowest-scoring priced models.

• Reading text is close to solved; everything else is not. Eight of ten reading fields score 0.90–1.00 averaged over models. Long or unusual strings remain hard: the 17-character VIN averages 0.59 and the gauge's seven-segment display 0.66, though Gemini 3.8 Flash scores 0.89 and 0.95 on those.

• Small parts and spatial relations are the weak spot. Lag bolt present averages 0.67 (best 0.82), damage severity 0.61 (best 0.75), whether wire touches the roof 0.57 (best 0.74), whether the tape leans 0.55 (best 0.82). Contrast with easy judgments: what a label is on (0.95), whether the horizon is visible (0.98), whether it is level (0.91), kind of damage (0.89), car's face (0.83), body style (0.81).

• Reading a tape measure is the hardest task tested. The number at the crossing is within 1/8 in only 11% of the time on average (best 36%, Opus 5.5 and GPT-6 Sol), and the true perpendicular gap within 1/4 in 23% (best 48%, Opus 5.5). This difficulty carries over to real photographs.

• Skill correlations separate the tasks. Reading is not detectably correlated with judgment (ρ = 0.23, p = 0.27), while judgment and measurement correlate strongly (ρ = 0.61). Acceptability F1 tracks judgment (ρ = 0.77) and measurement (ρ = 0.77), not reading (ρ = 0.26, p = 0.19).

• Models misread the scene, not the instruction. The tape is within 5° of upright on every scorer-acceptable frame, yet models call it tilted 75% of the time, and the rate rises with camera roll (Spearman −0.59 with roll, −0.51 with apparent blade size) in 23 of 26 models. On frames where a model reports the defect, it rejects 89–99.6% of the time — the requirement is applied consistently — but these misreadings account for 97% of wrong rejections in tape_gap, 74% in slope_gauge and 63% in rail_attachment.

• The misreadings are real, not post-hoc justification. Of nine models re-asked on 150 frames with the verdict last or absent, seven showed no change beyond run-to-run noise. The two cheapest, Gemini 3.1 Flash Lite and Gemma 4 31B, reported fewer defects without the verdict (Flash Lite about half as many), and asking for the verdict first worsens their documentation answers.

• Simulated rankings transfer to real photos. Spearman ρ of 0.63–0.79 (p ≤ 0.002) across 22 models on the three matched questions, with the worst transfer on damage severity. Task difficulty ordering is consistent: tape reading hardest, then severity, then naming the car's face (saturated for top models).

• The best models match, but do not clearly beat, one human rater. A single non-author rater scores F1 0.59 (95% CI 0.50–0.67) against the scorer. A human pass over the 1,378 items is estimated at roughly $11–165 (raters took 8–12 s per frame, at $10/h to $35.91/h), against $16.36 for Claude Opus 5.5 and $30.05 for GPT-6 Astra.

• Documentation is easier on acceptable frames. Averaged over the random frames, documentation is 0.71 on scorer-acceptable frames and 0.50 on the rest; every model scores higher on acceptable frames, by 0.14–0.26. The gap is largest on reading tasks (+0.58 to +0.70) and smallest in horizon_peak (+0.09), car_view (+0.07), wire_adequacy (+0.01) and tape_gap (−0.17) — in tape_gap the tape leans on 88% of scorer-rejected frames, so the models' tilt-calling is usually right there.

Methodology in Plain English

The authors could not use real field data, so they rebuilt the work in a 3D simulator. Eleven scenes were modelled in Blender: seven on a roof during solar installation (attaching L-feet with lag bolts and flashing, checking wire clearance, measuring module-to-roof gap with a tape, reading a digital angle gauge, photographing equipment nameplates, shooting the skyline from the ridge, and photographing the breaker panel's circuit directory), and four in a parking lot during vehicle repair (a straight-on whole-car shot, a close-up of damage, a license plate, and a repair-order board held next to the car).

Every frame uses a simulated phone camera — main lens 24–28 mm full-frame equivalent, sensor width 5.6–9.8 mm, f/1.6–2.4, with ultra-wide and telephoto lenses and digital zoom on some phones — so depth of field is physically rendered. 70% of frames are focused on the target with a steady hand, 15% have autofocus locked elsewhere, and 15% have hand shake over a shutter time drawn log-uniformly from 1/500 to 1/25 s. Sensor noise, white balance, tone mapping, chromatic aberration, vignetting and a JPEG re-encode are applied afterwards.

Each scene carries two tasks. The acceptability task asks whether a photo satisfies a natural-language condition (is the whole attachment visible, is the label legible, is the tape seated flush). The documentation task asks the model to write a property of the scene into structured JSON (the VIN, the gauge reading, the tape number at the crossing). Because the scene is a simulator, both can be checked exactly: some acceptability terms are computed from scene geometry after camera placement (in-frame status, occlusion by ray casting, distances, view angles, pixels per unit length), and others from the rendered pixels (sharpness at the target, contrast of text against its paper, blown-out fraction, wire pixel count and edge contrast, label luminance).

To make the scorer mimic human judgment, six raters (four authors, two colleagues) judged 30 random frames per environment against the same requirement text the models see, without seeing the scorer's verdict — 4.0–5.1 ratings per frame. The set was split once by frame, 60% to tune the scorer and set "gray zones," 40% held out and looked at once after freezing. Gray zones mark ranges where the wording is ambiguous (for example, camera distances of 1.5–3 m in damage_view) so that reasonable interpretations are not penalized; 43 frames were dropped this way. They also dropped the acceptability task for car_plate (raters disagreed on small but legible plates) and for rail_attachment frames clearly showing a missing bolt or flashing, and removed a "horizon must be level" scorer requirement the prompt did not ask for. After these reductions, 1,378 frames remained: 1,265 scored for acceptability, and 547 scorer-acceptable frames scored for documentation.

Models were run two ways: 23 through provider APIs at default settings, and three small open-weight models (Gemma 4 E4B, LFM2.5-VL-3B, Qwen3.5-4B) served by the authors in bf16. Gemma 4 E4B ran with thinking on; Qwen3.5-4B ran non-thinking, because in thinking mode it used a mean of 3,064 thinking tokens per item on a 180-item sample, about four times Gemma 4 E4B's 740.

For the sim-to-real check, the authors collected three real-image sets: the released tape-measure set (133 readings on 84 photos of 35 items across 12 scenes, ground truth by caliper or careful human read, scored as within 1/8 in), 63 real damaged-panel photos rated by one author on the severity scale, and 117 real frames labeled by one author with every side and third of the car shown. They then compared model rankings between rendered and real images.

Why This Matters

Benchmarks for physical-work assistance have been stuck between two bad options: expensive in-field trials and opaque proprietary archives with coarse labels. RealityBench offers a third path — simulated scenes with exact, programmatically checkable ground truth — and, unusually, tests whether conclusions drawn from rendered frames actually predict behavior on real photographs. The paper also contributes a negative but actionable result: the dominant failure mode is not instruction-following but perception of small parts and geometry, which points fine-tuning and model design at a specific target.

Real-world applications:

• Rooftop solar QA. Automatically screening installer-submitted photos for whether an attachment, wire clearance or nameplate is genuinely documented, reducing re-work and inspection trips.
• Insurance and collision repair intake. Judging whether customer or shop photos show damage square-on and at usable range, and extracting severity and vehicle identity from them.
• Field service documentation. Converting smart-glasses captures into structured reports (serial numbers, breaker directories, gauge readings) for maintenance and compliance records.
• On-device assistant deployment. The small self-served models show that local, low-latency, network-independent documentation assistance is within reach, at accuracy below frontier APIs.

Industry relevance: The cost axis matters as much as the accuracy axis here. The finding that six models cost under $2 for the full 1,378-frame set ($0.69–1.30), and that two of the three best in that band are open-weight, gives practitioners a concrete deployment trade-off. The comparison to a single human rater at roughly $11–165 per pass gives organizations a baseline for deciding whether to automate.

Future Directions

• Closed-loop guidance. Because the environments are simulators rather than fixed image sets, the authors note others can extend them to new tasks and to closed-loop guidance — not just judging a finished photo, but steering a worker toward a better one.
• Fine-tuning the open-weight middle tier. The authors highlight open-weight models in the moderately priced band (Gemma 4 31B, Gemma 4 26B-A4B, GLM-5.3 Flash) as attractive fine-tuning targets, since they already sit near the top group at $0.69–1.30 per set.
• Attacking measurement and spatial relations specifically. Tape reading at 11% average accuracy (36% best) and tilt judgments wrong 75% of the time on upright tapes are isolated, named failures that invite targeted data or architecture work.
• Sharpening acceptability specification. Human raters disagree with each other (76–92% agreement with the majority), the scorer is stricter than the majority, and gray zones had to absorb four ambiguous ranges. Better elicitation of what "acceptable" means in the field remains open.
• Closing the sim-to-real gap where it is weakest. Transfer was worst on damage severity, and four models (Qwen 3.8 Max, Muse Spark 1.3, Kimi K3, DeepSeek V4.1 Flash) were not run on real photographs at all — a gap the authors flag rather than resolve.

Target Audience

Researchers and engineers building vision-language systems for field work, augmented-reality assistance, or egocentric documentation will get the most from this paper, particularly those who need a reproducible benchmark with exact ground truth and a validated scorer. It is also useful for benchmark designers interested in human-agreement calibration, gray zones for ambiguous natural-language specifications, and rank-transfer testing between synthetic and real data. Product and operations leads evaluating whether to automate photo-based proof-of-work will find the cost-per-set and human-rater comparisons directly actionable. A basic grasp of classification metrics and rank correlation is enough to follow the results.

Authors’ abstract

Augmenting worker capabilities in the physical world through guidance has been a decades-long goal for AI agents. One key supporting capability is to understand what the worker is doing, seeing through the human's camera or smart glasses, and assist them in documentation. Despite advances in vision language models, their capabilities in these settings remain uncertain and are challenging to benchmark because most physical work data is proprietary, and involves too much variability to study in a principled manner. Unlike structured settings such as kitchens and warehouses that have predictable lighting, geometry, and objects, field-facing tasks such as rooftop solar installation and vehicle repair are unconstrained and high variance. Here, we model real-world physical work in Blender across eleven environments, incorporating variants in both image acceptability and documentation. The acceptability task asks whether a particular view would be suitable proof-of-work and documentation asks whether an agent can understand the task-relevant state of the environment from that view. Across 26 vision-language models, we find that top models largely solve documentation when the answer is text in frame, but are weak at reading measurements and at judging small parts and spatial relations, such as whether a tape measure leans. Acceptability is harder to pin down because human raters disagree with each other. Measured against a scorer that agrees with the raters' majority about as often as a single rater does, the best models cannot be distinguished from a single non-author rater, at a cost within the range of a human pass. In the three environments where we can trace acceptability errors, most wrong rejections come from misreading the scene rather than the requirement. On the three documentation questions for which we have real images, model rankings on rendered frames transfer to real images.