Hard Vision, Easy Vision: What GPT-6 Astra Reveals Across Computer Vision

AI summary

Overview

• Research area: Computer vision benchmarking and evaluation of frontier general-purpose (multimodal, language-model-driven) AI systems against dedicated specialist models and human performance.
• Technical level: Intermediate. The paper is an empirical landscape study; it assumes familiarity with standard computer vision task families (detection, segmentation, depth, video, restoration, medical and remote-sensing vision) but does not require deep architectural background.
• Scope: A systematic evaluation of GPT-6 Astra and five other frontier general-purpose systems across 34 capabilities and 55 benchmarks spanning nine areas of computer vision, comparing them with specialist models and humans to map what is now accessible through a general-purpose interface and what remains hard.

What This Paper Is About

Computer vision has traditionally advanced through specialized models built for individual tasks. Frontier general-purpose systems are now absorbing many of those tasks into a single natural-language interface, but existing benchmark evidence is scattered across tasks, domains and model comparisons, making it hard to see what these advances collectively mean. This paper asks how much of computer vision is now reachable through a general-purpose system, where meaningful gaps remain, and where dedicated vision models are still necessary.

Key Contributions

1. A landscape-level evaluation of frontier general-purpose vision. The authors evaluate GPT-6 Astra alongside five frontier systems (Fable 5, Kimi K3, Gemini 3.1 Pro, Qwen 3.8-Max, and Muse Spark 1.3) across 34 capabilities and 55 benchmarks organized into nine broad areas of computer vision, using the same task instructions, visual inputs and evaluation samples for all models.
2. A reference-based framing with specialist and human comparisons. Rather than comparing generalists only with each other, the study benchmarks them against dedicated specialist models and benchmark-reported human performance where suitable references exist, and identifies the specific specialist model used for each capability (Table 2).
3. A four-tier maturity framework. Capabilities are grouped into four tiers based on the best generalist score as a percentage of the stronger specialist or human reference: exceeds reference level (above 110%), at reference level (100–110%), approaching (85–100%), and substantial gap (below 85%).
4. An analysis of reasoning and tool use as gap-closing mechanisms. The paper tests multiple reasoning-effort settings (high, xhigh, and max) and tool-augmented pipelines, measuring both performance change and inference cost, including open specialist-as-tool pipelines.

Main Findings

• Broad but uneven capability. Across the 34 capabilities and 55 benchmarks, frontier visual capability has broadened dramatically, but maturity is not uniform. GPT-6 Astra shows substantial gains over other frontier systems in visual and spatial reasoning and several forms of structured prediction.
• Visual reading is close to or above human level. OCR scores fall between 93.7 and 98.1 against a reported human score of 98.0. Document, chart and infographic understanding is stronger still: all six models surpass reported human performance (89.3) by margins ranging from +0.2 to +3.5 points, with Astra at 92.8.
• Spatial reasoning has emerged as a strong capability. GPT-6 Astra reaches human performance in 2D spatial reasoning (96.0 vs. 95.8) and approaches it in 3D spatial and multiview reasoning (89.6 vs. 94.1). Its margins over the next-best frontier model on visual logical reasoning and 3D and multiview reasoning are +13.6 and +12.4 points respectively.
• Math reasoning is at or above human level, science is slightly below. All six models reach or exceed the reported human score on visual math reasoning (78.8–92.5 vs. 78.7), while scientific reasoning remains slightly below it (80.5–86.8 vs. 88.6).
• 2D localization is an emerging strength for some models, not all. GPT-6 Astra exceeds the specialist reference in 2D object detection (89.1 vs. 78.4) and segmentation (81.3 vs. 77.0), with +10.7 and +4.3 point margins over the specialist. Qwen 3.8-Max also surpasses the specialist detector (87.0), but the remaining models score considerably lower (31.1–69.7 for detection). The spread between frontier systems on segmentation is also wide (41.3–70.9).
• Pose estimation remains a major gap. GPT-6 Astra leads other frontier systems by a large margin (+22.7 points) at 75.2 OKS AP, yet remains 8.4 points below the specialist reference of 83.6.
• Object-centric 3D understanding is strong; metric geometry is not. All six models outperform the specialist on 3D visual grounding (reference 46.5), and GPT-6 Astra closely approaches the specialist on 3D object detection (17.3 vs. 16.8). In contrast, depth estimation (AbsRel 0.63 vs. specialist 0.44) and 3D reconstruction (error 1.40 vs. specialist 0.61) retain substantial headroom; even the strongest multiview reconstruction result has approximately 2.3 times the error of the specialist model.
• Video understanding outpaces dense temporal prediction. Several frontier models are competitive with the specialist on video and temporal understanding, but video segmentation is far harder: GPT-6 Astra reaches 84.5 J&F while other generalists span 32.0–57.9, and even the best frontier model remains 7.9 points below the dedicated model's 91.0 J&F.
• Generation, editing and quality assessment are near specialist level; restoration is not. GPT-6 Astra exceeds specialist performance in text-to-image generation (96.2 vs. 92.6, +3.6), instruction-guided editing (4.80 vs. 4.64, +0.16) and image quality assessment (86.2 vs. 84.9, +1.3). Image restoration is a shared weakness: all evaluated generalists cluster in a narrow band of 17.2–17.7 dB PSNR against 30.7 dB PSNR for the specialist.
• Restoration failures are fidelity failures, not just quality failures. In a LOL-v1 low-light enhancement example, Astra's restoration looks plausible but hallucinates content — four bowling pins become five, extra pins appear on an empty lane, and banner text and the lane number are altered — while the specialist MIRAGE stays faithful.
• Robotics shows competitive understanding and emerging execution ability. GPT-6 Astra exceeds specialist performance in driving-scene reasoning (80.8 vs. 73.1, +7.7) and embodied understanding (82.3 vs. 78.5, +3.8), and achieves 78% success on robotic navigation, where other evaluated models range from 48.0 to 70.0 and no reference is reported for that capability.
• Expert-domain understanding is strong but expert-domain perception is weak. Several models approach or exceed specialist performance in medical and remote-sensing understanding, and GPT-6 Astra leads 3D medical grounding at 73.9 mIoU against 59.4 for the reference. Yet 2D medical grounding remains far below the specialist (61.5 vs. 81.8), remote-sensing grounding is far below the specialist (26.4 vs. 75.5), and microscopy and pathology is a shared limitation across all six models (14.1–23.8 vs. 57.1 for the specialist and 82.0 for humans).
• Domain reasoning does not imply domain perception. Qualitative observations indicate models can localize relevant structures — such as nuclei in pathology — yet struggle to distinguish fine-grained categories, particularly subtle or less frequent ones.
• Reasoning and tools close selected gaps at variable cost. Higher reasoning brings 2D spatial reasoning, 3D object detection, and image generation and editing to specialist level, and tools improve chart and document understanding, counting, and 2D medical grounding. Segmentation also benefits but gains plateau at higher effort and remain insufficient to close the specialist gap. Tool use can yield gains at modest additional cost (1.04–1.20×), while higher reasoning can require up to 13.22× the baseline cost, sometimes for only small improvements.
• A boundary is emerging between semantic competence and perceptual precision. General-purpose systems increasingly match references when visual information supports semantic interpretation, reasoning, or object-centric prediction, and fall short when output must be metrically precise, pixel-faithful, temporally consistent, or dependent on specialized fine-grained visual knowledge.

Methodology in Plain English

The authors assembled a large evaluation suite covering nine areas of computer vision, broken down into 34 capabilities and drawing on 55 benchmarks chosen to retain meaningful difficulty for current frontier systems. Tasks were not limited to text answers: they span bounding boxes, masks, depth maps, 3D predictions, temporally consistent mask sequences, generated and edited images, and actions in embodied environments.

Six frontier general-purpose systems were run on identical instructions, visual inputs, and evaluation samples, with outputs scored by the corresponding benchmark metric. Where a system offered multiple reasoning configurations, the strongest available reasoning setting for the task was used.

To judge how far generalists have come, the authors compared each capability against two kinds of references: dedicated specialist models designed or trained specifically for that task, and benchmark-reported human performance. Table 2 lists the specific specialist model used per capability — for example SAM 3 Agent for segmentation, ViTPose++-L for 2D pose estimation, DA3 for depth, VGGT-1B for 3D reconstruction, BEE for video segmentation, MIRAGE for image restoration, HoVer-NeXt for microscopy and pathology, and RSRefSeg 2 for remote-sensing grounding.

Predictions were converted into benchmark-required formats: segmentation polygons were rasterized into pixel masks, depth surfaces and anchors rendered into dense depth maps (or generated by model-written code), depth combined with camera parameters for point clouds or per-frame point maps, and predicted centers, dimensions and rotations converted into box boundaries or cuboid corners. For image generation and editing, reasoning models were paired with image models — GPT-6 Astra with GPT Image 2.5 Sunburst, Qwen 3.8-Max with Qwen Image 3 Pro, Muse Spark 1.3 with Muse Image, and Gemini 3.1 Pro with Gemini 3.1 Flash Image.

The analysis is organized around four research questions: how broad frontier visual coverage is relative to human and specialist references (RQ1); where frontier systems converge and differ (RQ2); for which task types specialist-level performance is available through a general-purpose interface and what predicts the remaining gap (RQ3); and whether reasoning, tool use, or specialist-as-tool pipelines can close those gaps and at what cost (RQ4). A four-tier maturity scheme summarizes the distance to reference levels.

Why This Matters

Impact on research. The paper reframes the central question of the field from "which model wins this benchmark" to "which capabilities should be internalized by a generalist and which are better provided through tools." It shows that a model can lead other generalists on a benchmark while remaining far from specialist or human performance, and that strengths on one task do not automatically extend to related tasks — which makes landscape-level analysis necessary for interpreting incremental results.

Real-world applications:

• Medical imaging and pathology. Strong medical understanding coexists with weak fine-grained pathology recognition and large gaps in 2D medical grounding, which matters because the paper notes hallucinations may be acceptable for casual photos but critical in fidelity-sensitive domains such as medical imaging.
• Remote sensing. Fine-scale object grounding in satellite imagery remains far below specialist performance (26.4 vs. 75.5 mIoU), despite competitive remote-sensing reasoning.
• Robotics and embodied agents. Navigation reaching 78% success while other models range from 48.0 to 70.0 shows task execution is becoming accessible, but unevenly.
• Photography, restoration and content editing. Competitive generation, editing and quality assessment alongside uniformly poor restoration (17.2–17.7 vs. 30.7 dB PSNR) means plausible-looking outputs cannot be trusted for faithful reconstruction.
• Driving-scene understanding. GPT-6 Astra exceeds the specialist reference by +7.7 points, indicating general-purpose systems can support scene reasoning for autonomous systems, though metric geometric prediction remains weak.

Industry relevance. The results give practitioners a map for system design: use general-purpose systems for semantic interpretation, reasoning, and task planning and coordination; keep specialist tools for metric geometry, faithful reconstruction, dense correspondence, high-fidelity restoration, and fine-grained discrimination in rare expert domains. The cost analysis (tool use at 1.04–1.20× versus reasoning up to 13.22× baseline cost) directly informs deployment trade-offs between accuracy, cost and latency.

Future Directions

• Deciding the generalist/specialist division of labor. The paper raises explicitly which visual capabilities should be internalized by a generalist and which are better provided through tools, suggesting that metric geometry, faithful reconstruction, dense correspondence, high-fidelity restoration, and fine-grained discrimination in rare expert domains may remain tool territory.
• Knowing when to call a specialist and how to verify its output. A key open challenge is determining when the generalist should act directly, when it should call a specialist, and how it should verify the resulting output while accounting for accuracy, cost and latency.
• Closing the dense temporal prediction gap. Video segmentation remains far from the specialist (best generalist 84.5 J&F vs. 91.0), with difficulties concentrated in precise boundaries, small structures, separation of nearby instances, and maintaining these distinctions consistently across frames.
• Understanding why emerging strengths are not shared. Several of Astra's strongest capabilities — video segmentation, pose estimation, 2D detection, 3D visual grounding — remain substantially less developed in other frontier systems, leaving open whether these reflect architectural choices, training data, or reasoning configurations.

Target Audience

• Computer vision researchers and benchmark designers who need a consolidated view of where general-purpose models stand relative to specialist models and humans.
• Multimodal and foundation model developers deciding which capabilities to build into a generalist versus delegate to tools.
• Applied machine learning engineers and technical product teams evaluating whether a general-purpose interface can replace a dedicated vision model for a given task.
• Domain specialists in medical imaging, pathology, remote sensing, and robotics who need to know the current reliability limits of general-purpose vision in fidelity-sensitive settings.

Authors’ abstract

Frontier general-purpose systems are rapidly expanding beyond visual understanding into capabilities traditionally handled by dedicated computer-vision models. As these capabilities expand, a central question for the computer-vision community is how far this reach extends, and what remains hard. We evaluate GPT-6 Astra alongside five frontier general-purpose AI systems across 34 capabilities and 55 benchmarks spanning nine areas of computer vision. We compare their performance with dedicated models and humans where suitable references are available. Astra demonstrates broad visual capability, with substantial gains over other frontier systems in visual and spatial reasoning and several forms of structured prediction. Across the state-of-the-art systems, a consistent pattern emerges. Capabilities involving semantic interpretation, reasoning, and object-centric prediction increasingly approach or reach available reference levels. In contrast, larger gaps remain when tasks require metric geometric accuracy, faithful reconstruction, temporally consistent dense prediction, or specialized fine-grained visual knowledge. Additional reasoning and specialist tools close selected gaps, but their benefits vary across capabilities. These results map a changing landscape of computer vision in which increasingly sophisticated visual tasks are accessible through a general-purpose interface, while precise and fidelity-sensitive perception remains an important frontier.