MNIST-PRO: MNIST is Back as a Partially Observable World for AI Agents

AI summary

Overview

• Research area: Agentic perception and multimodal AI evaluation — specifically how vision-language agents build and maintain a perceptual state under partial observability.
• Technical level: Intermediate. The benchmark itself is simple (MNIST digits behind a moving window), but the analysis concerns POMDPs, working memory representations, and agentic perception.
• Scope: The paper introduces MNIST-PRO, a controlled benchmark that converts MNIST digit recognition into a sequential, glimpse-based search task, and uses it to diagnose why multimodal agents fail when they can only see part of the world at a time.

What This Paper Is About

Multimodal agents in partially observable settings must decide where to look next and combine fragmented glimpses into an evolving internal picture of the world. Existing benchmarks make this hard to study in isolation because they bundle perception together with navigation, physical control, or low-level execution failures. The paper turns MNIST into a controlled partially observable world — a small window slides over a hidden digit — so that failures can be attributed to perception and memory rather than to task comprehension or robot control.

Key Contributions

1. A minimal, controlled benchmark for agentic perception. MNIST-PRO reformulates MNIST from fully observed digit recognition into a POMDP where the agent must explore with a masked glimpse window to actively acquire, integrate, retain, and interpret visual observations over time.
2. A diagnostic framing rather than a success-rate leaderboard. The benchmark lets researchers determine whether an agent failed to acquire enough evidence, failed to consolidate evidence into an effective perceptual-state representation, or failed to interpret the state it built.
3. A taxonomy and controlled framework for comparing memory representations. The paper evaluates internal state construction (Image Only Baseline with raw visual history, Textual State, Metric Grid Map), externalized state construction (Online and Offline Visual Memory Canvas), and persistent/procedural memory in a tool-use harness, while varying visual lookback, sensing capacity, and task horizon.
4. A characterization of exploration and stopping behavior. The study documents how agents allocate their sensing budget and when they commit to predictions under partial observability, showing that agents often predict after limited exploration despite having substantial sensing capacity remaining.

Main Findings

• A large gap between recognition and agentic perception. Under full observability, nearly all evaluated models identify unmasked digits with 83.0% to 99.0% accuracy. Under partial observability, performance collapses — Gemini-3.1-Pro-Preview falls from 99.0% control accuracy to 38.0% in the image-only condition, and Claude-5-Sonnet falls from 94.0% to 24.0%.
• Best native multi-turn results are still modest. With full conversation history and no tools, Gemini-3.7-Flash leads at 75.0% on Level 1 and 47.0% on Level 2, followed by Gemini-3.6-Flash (64.0% and 28.0%) and Gemini-3.1-Pro-Preview (53.0% and 18.0%).
• More exploration does not mean better predictions. Claude-5-Sonnet has the highest step counts (15.96 on Level 1, 43.24 on Level 2) yet reaches only 23.0% on Level 1 and 0.0% on Level 2. Conversely, GPT-5.6-Sol, Qwen-3.8-27B, and GLM-4.6V rush to predict in 5.74, 4.62, and 2.76 steps on Level 1 and 13.99, 9.23, and 5.25 steps on Level 2, reaching 0.0% on Level 2 and 37.0%, 29.0%, and 13.0% on Level 1.
• No single memory representation dominates. For Level 1, raw visual history wins for seven of the eight proprietary models (Gemini-3.6-Flash: 53.0% versus 47.0% with textual state; Gemini-3.7-Flash: 65.0% versus 54.0%). For Level 2, the direct visual history baseline does worse (Gemini-3.1-Pro-Preview: 5.0%; Claude-5-Fable: 15.0%), and textual memory helps (Gemini-3.1-Pro-Preview reaches 20.0% with a textual state; Claude-5-Fable and Gemini-3.7-Flash reach 23.0% and 28.0% with metric grid maps).
• Coordinate tracking helps some models and hurts others. Gemini-3.1-Pro-Preview reaches 56.0% on Level 1 with textual reasoning alone but drops to 53.0% when also maintaining a grid map, while Gemini-3.7-Flash improves from 54.0% to 57.0%.
• Multi-digit sequences remain unsolved. No model exceeds 30.0% accuracy on Level 2 under any memory representation, and in the image-only baseline most models score below 20.0% (Gemini-3.1-Pro-Preview 5.0%, GPT-5.6-Sol 1.0%).
• Restricting visual lookback lengthens exploration. Gemini-3.1-Pro-Preview averages 4.52 steps on Level 1 with unbounded visual history, rising to 7.18 steps with a textual state and 7.15 steps with a metric grid map.
• Errors compound with task horizon. Single-digit Level 1 accuracy is roughly 39.0% across configurations; per-position accuracy in Level 2 drops to around 23.5% to 29.0%, and exact sequence accuracy falls further to around 9.5% to 11.5%.
• Timing of visual consolidation matters. In consolidation experiments, continuously consolidating evidence after every exploration step can be less effective than exploring first and constructing the consolidated canvas only at the end; the authors hypothesize that repeatedly exposing an incomplete perceptual state can confuse the agent.
• Three distinct bottlenecks. The paper identifies (1) difficulty constructing and interpreting a perceptual state from fragmented glimpses, (2) agents stopping exploration before seeing the full sequence, and (3) failure to revise early incorrect beliefs even when later contradictory evidence appears.

Methodology in Plain English

The researchers take MNIST images — 28×28 digits, inverted so strokes are black on white, upscaled to 224×224 with bilinear interpolation, then binarized at an intensity threshold of 200 — and hide them behind a 64×64 glimpse window. The agent moves that window around the image with up, down, left, and right actions that shift it by 32 pixels, and can end the episode at any time by making a prediction. To derive an upper bound on needed exploration, the step budget is computed as the number of discrete steps needed to cover the canvas at that stride: 36 steps for Level 1 and 78 steps for Level 2.

Two task levels scale the demands. Level 1 is a single centered digit that must be traced and identified. Level 2 horizontally concatenates two raw digit images before the same preprocessing, producing a 224×448 canvas whose digits must be recognized and preserved in order. Each level is evaluated on 100 test episodes; Level 1 is balanced across the ten digit classes with ten episodes each, and Level 2 samples two digits independently and uniformly from {0, …, 9}.

Ten models are evaluated — proprietary systems Gemini-3.1-Pro-Preview, Gemini-3.6-Flash, Gemini-3.7-Flash, Claude-5-Sonnet, Claude-5-Opus, Claude-5-Fable, GPT-5.6-Terra, and GPT-5.6-Sol, plus open-source Qwen-3.8-27B and GLM-4.6V — all queried with default API configurations. GPT-5.6-Sol is additionally run under the "xhigh" reasoning effort setting for selected experiments. Three metrics are reported: control accuracy on the fully unmasked image (the upper bound), classification accuracy under partial observability, and average step count before prediction.

Because the variables in a free-form multi-turn setup are entangled, the authors run targeted conditions that hold others fixed. They compare memory representations: an Image Only Baseline that passes the chronological visual history plus a text log of actions; a Textual State that requires a "thought" alongside each action under a strict zero-visual-lookback constraint (one frame visible); and a Metric Grid Map that adds structured coordinates and observed features, with the agent anchoring its start at [0, 0] and integrating displacements itself. On the externalized side, they programmatically build an Online Visual Memory Canvas that overlays each new glimpse onto a persistent coordinate-aligned image, and an Offline Visual Memory Canvas built from previously collected trajectories to probe whether failures come from exploration or from interpretation. A final harness gives agents persistent memory across episodes, including correctness feedback, to test whether they retain or revise procedures — the detailed results of that harness are not included in the portion of the paper analyzed here, nor are the quantitative results of the visual-history and glimpse-size ablations described at the end of the experiments section.

Why This Matters

The paper argues that acquiring visual evidence is not the same as being able to use it. Strong passive recognition ability on a fully observed image does not transfer to an environment where the agent must decide where to look, remember what it saw, and know when it has seen enough. By stripping away navigation, collisions, and object interaction, MNIST-PRO provides a clean attribution of failure to perception, memory, and stopping decisions — a diagnostic capability that leaderboard-style benchmarks do not offer.

Real-world applications:

• GUI and computer-use agents that must scroll, click, and inspect interface regions they cannot see all at once, and decide when they have enough information to act.
• Robotic manipulation and inspection, where a camera or sensor sees only a patch of the scene and the agent must decide where to look next while retaining what it already observed.
• Document and form understanding, where information is spread across regions that must be gathered sequentially and kept in the correct order.
• Long-horizon web or enterprise workflows, where early incorrect beliefs must be revised as new evidence arrives — directly related to the paper's finding that models often fail to update a wrong early guess.

Industry relevance: the results suggest that scaling recognition capability alone will not produce reliable agents in partially observed settings, and that design choices about what an agent writes into memory — and when — can measurably change accuracy. That is directly actionable for anyone building agents that operate behind a limited viewport or sensing budget.

Future Directions

• Why does representation choice flip between tasks? Raw visual history wins on single digits while textual or grid-based states win on multi-digit sequences, and no representation dominates across models. Building a representation that is not brittle to horizon is an open problem.
• Improving belief revision. Models frequently keep an early incorrect belief despite later contradictory evidence; how to make agents reliably overwrite stale perceptual states is unresolved.
• Teaching agents when to stop. Agents often commit before exhausting their sensing budget, yet using more steps does not guarantee success. Deciding both where to gather evidence and when the state is reliable enough to act remains open.
• Better use of the memory write-path. The finding that continuous consolidation can be worse than end-of-episode consolidation raises the question of when an agent should commit a partial state to memory at all.
• Scaling beyond MNIST. The authors position the benchmark as deliberately familiar and low-complexity; whether these bottlenecks persist in richer visual domains with the same controlled isolation is an open question.

Target Audience

Researchers and engineers working on multimodal agents, agentic perception, and memory architectures for vision-language models. It is also useful for benchmark designers who need clean attribution of agent failures, and for practitioners building GUI agents, embodied systems, or any application where an agent must act under a limited field of view. Readers need only a basic familiarity with multimodal models and reinforcement-learning vocabulary such as POMDP to follow the paper.

Authors’ abstract

AI agents in partially observable environments need to coordinate active sensing with working memory to maintain an evolving perceptual state. However, existing benchmarks struggle to isolate this perceptual-state construction and interpretation capability because they introduce physical and control complexities. We address this with MNIST-PRO, a benchmark that isolates agentic perception by converting MNIST digit recognition into a sequential, glimpse-based search task with lookback constraints. We evaluate ten multimodal models across four memory representations, including raw visual history, textual states, structured metric grid maps, and a consolidated visual canvas. While models excel under full observability, partial observability exposes a clear performance gap. We identify three distinct bottlenecks. First, perceptual-state construction and interpretation present a challenge, as agents struggle to integrate fragmented glimpses. Second, agents often stop exploring before they see the full sequence. Third, models often fail to revise early, incorrect beliefs even when faced with subsequent contradictory evidence. These results show that simply acquiring visual evidence is not enough. Agents must also be able to build and update a reliable perceptual state.