Humanity's Sixth Sense: Benchmarking Intuitive Visual Reasoning in Multimodal Models

AI summary

Overview

Research area: Evaluation of Multimodal Large Language Models (MLLMs) — specifically, benchmarking a form of visual reasoning that sits between raw perception and deliberate problem-solving.

Technical level: Intermediate. The paper is readable without specialist mathematics, but it assumes familiarity with benchmark design, pass@1 scoring, LLM-as-judge evaluation, and agentic tool-use harnesses.

Scope: The paper introduces HSS (Humanity's Sixth Sense), a 522-task human-verified benchmark of images and video that tests whether models can infer what a scene implies rather than what it shows, and reports results for 25 models from eight vendors against a human baseline.

What This Paper Is About

People read enormous amounts of unstated information from a glance — who is in charge of a room, whether a car will fit in a gap, why someone stopped running, what label is hidden behind an obstruction — and they do this quickly, without deliberate derivation. Current visual benchmarks test either expert academic problem-solving (where the image supplies explicit "givens") or low-level perceptual acuity (where the answer is somewhere in the frame). HSS targets the missing middle: tasks whose answers are never depicted at all, and which a careful look cannot settle. The goal is to make this intuitive visual reasoning a measurable axis of model capability and to quantify how far frontier MLLMs fall short of people on it.

Key Contributions

1. A structured taxonomy of intuitive visual reasoning. The authors formalize the capacity into four core domains and eleven subdomains — Temporal & Causal Dynamics (retrodiction, mechanistic causality, extrapolation), Physical & Spatial Logic (hidden & invisible, affordance & feasibility, spatial "alien viewpoint," spatial reachability), Social Understanding (theory of mind; social role, norm & power dynamics), and Abstract & Contextual Inference (change & consequence, patterns & pareidolia) — grounded in cognitive-science work on core object knowledge, intuitive physics, and mentalizing.

2. A rigorous, human-verified benchmark. HSS contains 522 tasks (288 image-based, 234 video-based, 17.6 hours of video total) carrying 723 fine-grained rubric criteria. Every item was human-authored and passed three independent review rounds with unanimous approval; 3,466 initially authored tasks yielded 522 released tasks, a 15.1% end-to-end acceptance rate. Answers are free-form and graded against atomic rubrics that must all be satisfied, avoiding multiple-choice process-of-elimination shortcuts.

3. Comprehensive empirical evaluation. 25 MLLMs from eight vendors were evaluated against a human baseline, plus a blind control with visual media removed, an error taxonomy over 8,573 labelled failures, and multi-judge grading audits.

4. Agentic and ablation analyses. The paper tests vendor agentic harnesses (Claude Code, Codex) on a 388-task subset, sweeps reasoning effort and video frame budgets, and shows the human–model gap narrows but does not close under any of these interventions.

Main Findings

• A large, unsaturated gap. Human participants reached 93.1% pass@1; the strongest model, GPT-6-astra at maximum reasoning effort, reached 53.6% (±4.1). Most of the 25 models scored below 40%, and the second-best model, GPT-6.1-sol (max), reached 46.6%.

• Models overthink these tasks. Across all 25 models, average reasoning token usage was around 4,000 per task, and models "frequently overthink without arriving at the correct answer." Peak accuracy often occurs at intermediate rather than maximum reasoning effort — for example, GPT-6-astra drops 14 points in the retrodiction subdomain when effort scales from high to xhigh.

• Video is harder than images. Video tasks were more difficult than static images for 23 of 25 models, with an average drop of 7.3 percentage points.

• Social understanding is the weakest domain. It was the lowest-performing domain for 21 of 25 models, averaging 24.4% accuracy versus 34.1% across the other three domains.

• Vision is indispensable. Removing the visual media from the top model drops its score from 53.6% to 6.6%. Only 4 of 522 tasks passed blind on all three attempts, and those were concentrated on two-way questions where the model committed to one side and invented supporting detail.

• Human performance is robust even under low confidence. Twenty annotators were unsure on only 63 of 522 tasks (12.1%); they scored 94.1% on the 459 tasks they were sure about, and 85.7% on the ones they were unsure about.

• Failures are perceptual and inferential, not logical. Of 8,573 labelled first-attempt failures, 53% stemmed from visual perception, 41% from latent inference, and only 5% from abstract reasoning. The two largest single failure modes were missing a decisive visual cue (21%) and misidentifying an object, person, or role (20%). Failure patterns are systematic across architectures: on tasks where five or more models failed, a median of 80% failed for the exact same cause.

• Subdomains have distinct failure signatures. Affordance and spatial reachability fail mainly on size and fit (62% and 56%), extrapolation on facing direction (42%), hidden & invisible on hidden quantities (44%), and mechanistic causality and theory of mind on missed cues (47% and 39%). Alien viewpoint is the only subdomain with a notable reasoning component, led by line-of-sight errors (23%).

• Agentic harnesses help, but not enough. On a 388-task subset, Claude Code produced gains of +16.5 to +20.5 points at high cost (median 57–72 steps, more than 42k thinking tokens per task), while Codex produced +4.9 to +7.5 points at 14–18 steps. The best configuration (Codex with GPT-6-astra) reached only 59.3%. Total agentic failures fell from 944 to 759, driven by a 19–37% reduction in perception and latent errors, but depth errors barely moved (102 to 101) and failed agent runs took more steps than successful ones (median 81 vs. 67).

• Frame budget matters. Across 234 video tasks, every one of four evaluated models scored lowest at 10 frames; GPT-6-astra gained +17.9 points going from 10 to 500 frames.

• Grading is stable across judges. Re-grading five models with three judges from three vendors over 2,620 items and 3,780 criteria preserved rankings (r ≥ 0.98), with pairwise agreement above 95% and κ ≥ 0.92. GPT-5.6-terra was stricter by 2.4 points on average but uniformly so, and the primary judge showed no self-preference (Claude-Opus-5 grading itself shifted its own score by −0.44 points).

Methodology in Plain English

The authors first defined what counts as "intuitive": fast, automatic, grounded in everyday experience, and hard to fully articulate. They operationalized this into four domains and eleven subdomains, then used that taxonomy as a curation blueprint so the benchmark would have balanced coverage.

Human annotators built each task by picking an image or video clip from a licensed media pool, assigning it to a domain and subdomain, and writing a question, a gold answer, and a checklist of atomic rubric criteria. Three design rules governed acceptance: reject anything a high-resolution crop would settle (pure perception), reject anything needing specialist knowledge or symbolic calculation, and reject anything on which independent reviewers did not unanimously agree.

Quality control was a three-round review funnel where each reviewer could Pass, Edit (send back for revision, restarting the pipeline), or Drop. Roughly two-thirds of submissions were filtered at the first gate. Only tasks surviving all three rounds — verified by at least three reviewers beyond the author — entered the benchmark.

For evaluation, models received either native video or uniformly sampled frames (capped at 500, or the model's own maximum), and images at native resolution. Responses were free-form and graded by Claude-Opus-5 as an LLM judge against the rubrics; an item counted as solved only if every criterion was met. The authors stress-tested this protocol by re-grading with two other judges, by running a blind condition with media removed, and by scoring human reference answers as a sanity check. They separately dropped the same models into agentic harnesses so the models could crop, zoom, and re-inspect, then re-labelled agent failures with the same error taxonomy to see which failure types active looking actually fixes.

Why This Matters

Impact on research. HSS defines a capability axis that existing benchmarks do not measure: inference about what a scene never depicts. Because the gap persists under test-time compute scaling, higher reasoning effort, denser video sampling, and agentic tool use, the results argue that this capability is not simply a byproduct of more inference compute — a claim that reframes what "scaling" should mean for multimodal systems. The failure taxonomy also gives the field a diagnostic vocabulary (missed cue vs. size/fit vs. depth vs. facing direction) rather than a single aggregate score.

Real-world applications:

• Home and assistive robotics — noticing a toddler edging toward stairs, or inferring that a container is about to tip, before anything goes wrong.
• Automotive and autonomous driving — judging whether a vehicle fits between two parked cars without explicit scale markers, and anticipating trajectories from a glance.
• Workplace and social agents — reading authority, tension, and unstated norms in a room to align behavior with the people present.
• Safety, inspection, and monitoring — reconstructing prior events and predicting imminent consequences from surveillance or industrial footage where the relevant evidence is off-frame or implied.

Industry relevance. The paper is co-authored across ScaleAI, Elorian, and UC Santa Cruz, and it ships a leaderboard and a public Hugging Face dataset — infrastructure aimed at model developers and evaluators. For anyone selecting or shipping multimodal models, the message is that leaderboard dominance on academic and perceptual benchmarks does not transfer to everyday visual common sense, and that agentic scaffolding buys real but bounded gains at substantial token and step cost.

Future Directions

• Why does extra deliberation hurt? Accuracy sometimes peaks at intermediate reasoning effort and degrades at maximum (a 14-point drop in retrodiction), suggesting that current reasoning alignment is mismatched to intuitive tasks. Understanding and fixing this is an open problem.

• Recovering 3D information from 2D inspection. Agentic zooming repaired object, facing, size/fit, and missed-cue errors but left depth errors essentially unchanged (102 to 101), implying that new sensing or representation approaches — not better cropping — are needed for spatial reachability and alien-viewpoint tasks.

• Improving social inference. Social understanding is the weakest domain across nearly all models (24.4% average), and the authors caution that their annotator pool is demographically narrow, so a broader, cross-cultural ground truth for theory-of-mind items remains to be built.

• Beyond the visual channel and beyond judge mediation. The benchmark strips audio and transcripts by construction, and scores depend on an LLM judge with known biases. Extending to multimodal audio-visual inference and reducing judge dependence are both stated limitations and natural next steps, alongside controlling for training-data contamination from public web media.

Target Audience

Multimodal AI researchers and benchmark designers; model developers at labs deciding what capabilities to train and measure next; evaluation and safety teams who need to know where a deployed vision-language system's common-sense inference will fail; and cognitive-science-adjacent readers interested in how human intuitive physics, mentalizing, and scene understanding can be operationalized into testable machine tasks. Readers with a general AI background but no benchmark-engineering experience will still follow the main argument from the human–model gap figures alone.

Authors’ abstract

Humans perceive far more in a scene than what is explicitly depicted: a single glance captures past causes and future trajectories; a quick peek determines if a vehicle can fit between two parked cars; a few seconds of video reveals who holds authority in a room; and a fleeting clip highlights subtle abstract patterns like unwritten rules or hidden labels. This capacity reflects a form of humanity's sixth sense: an intuitive reasoning mechanism that recovers implicit information beyond raw sensory perception. Crucially, this rapid, zero-shot visual intuition underpins everyday navigation and social interaction, making it a vital capability for Multimodal Large Language Models (MLLMs) deployed alongside people. Existing visual benchmarks, however, target either deliberate expert-level analysis in academic and mathematical domains or low-level perception, leaving the intuitive reasoning that people perform largely untested. To bridge this gap, we introduce Humanity's Sixth Sense (HSS), a benchmark for intuitive visual reasoning. HSS spans diverse image and video inputs, organizes items under a structured taxonomy, and pairs each with human-written prompts probing the implicit temporal, spatial, social, and abstract structure that people infer at a glance. Frontier MLLMs fall short of human performance: participants reach 93.1% accuracy, while the strongest model, GPT-6-astra, reaches only 53.6% even at maximum reasoning effort. Despite excelling in many complex tasks that require advanced perception and knowledge, current models still struggle significantly on these visual tasks that are intuitive for humans. We further explore agentic setup that apply dynamic visual manipulation to HSS, which narrows but does not close the gap. HSS establishes intuitive visual reasoning as a measurable axis and directs attention to a capability that scaling on current benchmarks has so far left behind.