Flow-of-Thought: A Framework for Visual Reasoning

AI summary

Overview

• Research area: Computer Vision, spatial reasoning, multimodal foundation models, generative modeling (flow matching).
• Technical level: Advanced. The paper assumes familiarity with ordinary differential equations, flow matching, SO(2) group orbits, U-Net architectures, feature-wise linear modulation, and IoU-based segmentation metrics.
• Scope: The paper proposes and evaluates FoT, a framework that learns continuous visual dynamics (rotation orbits and maze paths) from trajectory supervision and freezes them for interpretable same/different and navigation decisions.

What This Paper Is About

Large language models and vision transformers still fail at spatial tasks that humans solve easily by "seeing with the mind's eye" — such as deciding whether two blocks are the same object rotated, or finding a connected path through a maze. The core problem is a modality mismatch: text-based reasoning is one-dimensional and discrete, while spatial reasoning requires continuous, multi-dimensional geometric transformations. The goal is a framework that generates explicit visual sketches as intermediate reasoning steps, mimicking human mental imagery, and then uses those generated endpoints as the basis for a decision.

Key Contributions

1. The Flow-of-Thought (FoT) framework, which trains coordinate-, time-, and action-conditioned trajectory flow fields on rendered SO(2) rotation orbits and cumulative shortest-path masks for mazes, then freezes the learned dynamics at inference.
2. A frozen hypothesis-competition decision rule for same-versus-different tasks: the source and its horizontal reflection define competing hypotheses, and the one with the lower foreground-weighted target-reconstruction energy over a predeclared 36-angle action set is selected. No pair labels, fitted threshold, or target-task adaptation enters the decision.
3. A single-source rendering scheme for rotation, in which a parallel backward characteristic map is integrated continuously so every requested frame x(t) is sampled once from the original source, avoiding the blur that repeated resampling would cause.
4. A matched supervision ablation and an approximate SO(2) consistency audit, comparing rotation-orbit supervision against an endpoint-only control and a linear pixel-path baseline, alongside frozen transfer evaluations on 3D Blocks and BLINK Multi-view.

Main Findings

• Locked 2D rotation tasks: FoT reaches 100.0% accuracy on Tetris and 99.0% on colored shapes, against displayed baselines including ViT (61.0% Tetris, 62.0% colored), DINOv3 (60.0%, 49.0%), Qwen3.5 (52.0%, 51.0%), GPT-4o (52.4%, 50.0%), GPT-5.6 (72.0%, 98.0%), and Claude 4.6 (46.2%, 51.6%).
• BLINK Multi-view transfer (primary OOD evidence): The orbit-trained 2D flow reaches 72.2% [64.0, 79.1] versus 63.9% [55.5, 71.6] for its endpoint-only control on 133 public validation pairs — an 8.3-point improvement. The intervals overlap, so the paper calls the improvement suggestive rather than conclusive. FoT exceeds GPT-4V (58.7%) but not the published P2+Qwen3VL result (94.0%).
• 3D Blocks transfer (negative control): FoT reaches 61.5% and does not exceed the endpoint-only control at 66.7%. Because the test set has only 78 pairs, a single reclassified item moves accuracy by more than 1.2 points (stated elsewhere as 1.3 points) and the Wilson interval [50.4, 71.6] spans more than twenty points. The paper reports this as an instability-limited negative control against a universal OOD advantage from intermediate trajectory supervision.
• Maze navigation: FoT obtains 97.5% endpoint IoU [94.6, 100.0], 84.2% intermediate prefix IoU [82.2, 86.3], 100% goal reach, and zero obstacle violations. However, premature activation is 31.7% and mean future-path intensity is 0.164, meaning later path segments sometimes activate too early.
• Matched supervision ablation (Table 2): Rotation-orbit supervision beats endpoint-only and linear-pixel-path baselines. Tetris: 100.0% [96.3, 100.0] orbit, 94.0% [87.5, 97.2] endpoint-only, 39.0% [30.0, 48.8] linear. Colored: 99.0% [94.6, 99.8] orbit, 85.0% [76.7, 90.7] endpoint-only, 51.0% [41.3, 60.6] linear. Endpoint IoU follows the same ordering (Tetris orbit 0.876 vs. 0.757 vs. 0.532; Colored orbit 0.832 vs. 0.633 vs. 0.300).
• Approximate SO(2) consistency audit (Table 3): Orbit-trained models show the lowest inverse and composition defects. Tetris orbit inverse MSE 0.002526 and composition MSE 0.001138, versus endpoint-only 0.017123 and 0.012587, and linear path 0.031338 and 0.044970. Colored orbit inverse MSE 0.009159 and composition 0.004895, versus endpoint-only 0.030288 and 0.014789. Identity MSE is close across methods (e.g., Tetris orbit 0.000987 vs. endpoint-only 0.001061). The model is approximately group-consistent, though equivariance is not built into the architecture.
• Interpretability: Successful rotation decisions show progressive geometric alignment in intermediate frames; failures reveal loss of structure or implausible branch reconstructions. Maze traces expose whether errors concern wall avoidance, endpoint completion, or the timing of path activation.
• Qualitative caveat: On 3D Blocks, the 2D flow often behaves more like viewpoint-conditioned morphing, consistent with the weak, unstable transfer result.

Methodology in Plain English

FoT learns a "flow" — a set of rules describing how pixels should move over time — rather than predicting a final answer directly.

• Backbone. A spatial U-Net receives the current state, the task condition, and explicit (x, y) coordinates. Time is encoded as sinusoidal features, and a rotation action is encoded as three numbers (Δθ/180, sin Δθ, cos Δθ). These are injected into residual blocks via feature-wise linear modulation, with group normalization and SiLU activations at every resolution.
• Rotation training. Instead of predicting an image residual, the network predicts a dense two-channel transport field. The target tangent field is u*(p; ω) = (−ω p_y, ω p_x), where ω is the requested angle in radians, and training minimizes a smooth ℓ1 loss between predicted and target fields. The field is integrated with 12 fixed Heun steps. A parallel backward characteristic map is integrated continuously, so every displayed frame is sampled once from the original source — avoiding cumulative blur.
• Maze training. The state is a cumulative path mask and the field is additive, conditioned on a map encoding walls, start, and goal. Training combines foreground-balanced tangent regression (weighting newly added path pixels by 20 and current foreground by 3) with differentiable multi-step binary-cross-entropy and Dice losses (weighted by 2) against cumulative shortest-path prefixes. Inference uses 8 Heun steps.
• Decision. For a rotation pair, SAME starts from the observed source and DIFFERENT starts from its horizontal reflection. Each branch is integrated over a predeclared 36-angle set {−177.5°, −167.5°, …, 172.5°} — the 2.5° offset keeps candidates off both manifest angle grids — and scored by a foreground-weighted reconstruction energy that multiplies squared pixel error by (1 + 4·max over channels of the target). SAME is predicted iff E_same ≤ E_diff.
• Data and training. Tetris rotation trajectories use J/L/S/Z sources for training and held-out F/P shapes for selection; the colored-shapes task adds multi-object composition and RGB channels. Mazes are perfect 9×9 cells generated by depth-first backtracking, with breadth-first search supplying the shortest path, resized to 64×64. All primary models are trained independently with seeds 0, 1, and 2 on Metal Performance Shaders, using 3,000 training trajectories and 400 fixed validation trajectories for 30 epochs, batch size 20, AdamW with weight decay 10⁻⁴, and cosine learning-rate decay. Rotation models use width 16, context dimension 64, and learning rate 10⁻³; the maze model uses width 32, context dimension 128, and learning rate 2×10⁻⁴. Checkpoints are selected by validation silhouette IoU for rotation and endpoint path IoU for maze.
• Uncertainty reporting. Wilson intervals quantify uncertainty over items; seed-level geometric metrics use two-sided Student-t intervals over three independent runs.

Why This Matters

• Research impact: The paper shifts spatial reasoning from one-shot prediction toward continuous, inspectable visual traces, and it deliberately reports a matched endpoint-only control that removes a claimed advantage on one OOD benchmark. That makes it a careful data point both for and against intermediate trajectory supervision. The finding that the orbit-trained model is approximately SO(2)-consistent despite no equivariance being built into the architecture suggests that group structure can emerge from trajectory supervision.
• Real-world applications:
  • Robotics and manipulation, where a robot must mentally simulate incremental rotations before grasping or reorienting an object.
  • Navigation and path planning in grid-like or constrained environments (warehouse floors, game levels, indoor maps).
  • Educational and diagnostic tools for spatial reasoning, where the intermediate sketch shows a learner or model where reasoning diverges.
  • CAD, geometry, and molecular modeling, where chirality, reflection, and view-angle judgments are routine and error-prone.
• Industry relevance: The framework provides an interpretable alternative to semantic memorization in multimodal systems. The paper positions FoT as a candidate external visual tool for foundation models, which is attractive for industries that need auditable spatial decisions rather than opaque answers. The paper states the code will be made publicly available upon acceptance.

Future Directions

1. Physically faithful 3D dynamics evaluated on a larger 3D test set, since the current 78-pair 3D Blocks evaluation is too small and seed-sensitive to produce a stable estimate.
2. Stricter causal path generation for mazes, addressing the observed 31.7% premature activation and mean future-path intensity of 0.164.
3. Integration of the FoT pipeline as an external tool for multimodal foundation models.
4. Resolving the BLINK evaluation's status, since its sign convention was finalized after only a two-item smoke check and its confidence intervals overlap with the control.

Target Audience

Researchers and graduate students working on multimodal reasoning, spatial and abstract reasoning benchmarks, generative models with flow matching or ODE-based integration, and neuroscience-inspired AI. It is also relevant to practitioners who need interpretable spatial decision systems and to anyone evaluating whether intermediate visual supervision genuinely transfers out of distribution — the paper's explicit negative control on 3D Blocks is as informative as its positive results.

Authors’ abstract

Mental imagery, ``seeing with the mind's eye'' is an essential aspect of human cognition. Despite rapid progress Large Language Models (LLMs) and Vision Transformers (ViTs) still underperform on tasks requiring spatial understanding. To address this, we introduce Flow-of-Thought (FoT), a framework that integrates the generation of visual sketches as intermediate reasoning steps, mimicking mental imagery in humans. We train coordinate-aware trajectory flow fields on $SO(2)$ group orbits and cumulative shortest paths, then freeze the learned dynamics; same vs. different decisions compare competing generative hypotheses using foreground-weighted reconstruction energy. On locked tests FoT reaches 100.0% accuracy on Tetris and 99.0% on colored shapes. Under frozen transfer, the orbit-trained 2D flow improves over its endpoint-only control on BLINK Multi-view (72.2% vs. 63.9% on 133 public validation pairs), supporting continuous visual traces as an effective and interpretable representation for spatial reasoning in some out-of-distribution settings.