Reasoning is a Modality

AI summary

Overview

Research area: Abstract reasoning and AI architecture — specifically the Abstraction and Reasoning Corpus (ARC) benchmark, treated as a vision-conditioned image-to-image translation problem.

Technical level: Intermediate. The paper assumes familiarity with transformers, attention mechanisms, vision transformers (ViTs), and test-time training, though its central idea is explained in accessible terms.

Scope: The paper proposes that "reasoning is a modality" — a distinct internal channel separate from the low-level workspace — and tests this by replacing a dense ViT backbone with a role-separated transformer block for solving ARC puzzles under the VARC protocol.

What This Paper Is About

Modern AI systems such as large language models and dense vision transformers behave as "sequence-of-behavior prediction machines": they generate actions and explanations as statistically plausible continuations of observable behavior, without a persistent internal mental state that causally produces the behavior. The authors hypothesize that reasoning should exist as a distinct internal channel — a global controller state — separate from the low-level workspace on which rules are applied. They operationalize this hypothesis by designing a role-separated transformer block and evaluating it on ARC tasks as a visual reasoning problem.

Key Contributions

1. A formulation of reasoning as a modality. The paper hypothesizes that reasoning should exist as a distinct internal channel — a global controller state — that is separate from the low-level workspace on which rules are applied, motivated by biological and cognitive evidence for hierarchical processing and temporal separation.

2. A role-separated transformer block. The block partitions tokens into a small set of globally connected controller tokens (a task token plus context tokens derived from demonstration pairs) and a large set of workspace tokens representing grid states. A structured attention pass lets controller tokens attend globally while restricting workspace tokens to the controller plus a local 3 × 3 neighborhood (self and 8 neighbors).

3. A recurrent controller wrapper. A lightweight recurrence with a gated residual update and an exponential-moving-average history state increases effective depth without increasing the number of parameters, framed as repeated rule execution over a workspace rather than emulation of an RNN over time.

4. Empirical gains over prior ARC methods. An ablation across four architectural variants and three test-time-training strategies, yielding the c-d-4-TTT3 model, which outperformed VARC-ViT on both ARC-1 and ARC-2 and, when ensembled with UNet, exceeded reported average human accuracy on ARC-1.

Main Findings

• Best single-pass variant is architecture d. At r = 1, architecture d (dense attention followed by a structured pass where workspace tokens attend to the controller and a small local neighborhood) was the best single-pass model, supporting the design principle that reasoning signals should be mediated by controller tokens while local consistency and corrections remain available through limited neighborhood access.

• Removing dense self-attention hurts. Architectures a and b performed poorly on ARC tasks due to lack of dense self-attention.

• Architecture c generalizes more rule-consistently than ViT. Architecture c was comparable with the ViT baseline in training accuracy, but exhibited a smaller train-eval disparity under RE-ARC augmentation, which the authors read as a shift away from superficial distribution matching toward more rule-consistent generalization. Architecture d obtained the best training and evaluation accuracy.

• Staged training was best at deeper recurrence. For r = 4, training architecture d end-to-end did not always outperform architecture c. The best variant (c-d-4) came from staged training: architecture c for 100 epochs, then enabling the structured neighborhood as in architecture d for the next 30 epochs.

• Workspace attention is heavily bottlenecked. Under dense attention a workspace token may attend to S tokens (controller plus all workspace tokens); under the structured pass it attends to at most P + 1 + 8 ≤ 14 tokens. With a 64 × 64 canvas and 2 × 2 patches, this is orders of magnitude smaller than dense global attention, making the controller the natural locus for global task information.

• Main test accuracies. c-d-4-TTT3 (28M parameters) reached 58.8 on ARC-1 and 11.0 on ARC-2, with mean ± std of 58.8 ± 0.7 on ARC-1 and 11.0 ± 1.0 on ARC-2 over 4 random trials. On average it outperformed VARC-ViT by 4.3 on ARC-1 and 2.7 on ARC-2. (The paper's Summary section states 11.1% on ARC-2, while Tables 1 and 2 list 11.0.)

• Ensemble results beat average human accuracy on ARC-1. Using the VARC-Unet ensemble technique, the c-d-4 model (83M parameters) achieved 62.6% on ARC-1 and 13.5% on ARC-2. The abstract states this 62.6% on ARC-1 surpasses average human performance (60.2%).

• Comparison with prior methods. VARC-ViT (18M) scored 54.5 on ARC-1 and 8.3 on ARC-2; VARC-ViT-large (66M) scored 53.0 on ARC-1 (ARC-2 not reported); VARC-ViT-Unet-ensem (73M) scored 60.4 on ARC-1 and 11.1 on ARC-2; HRM (27M) scored 40.3 on ARC-1 and 5.0 on ARC-2; TRM (7M) scored 44.6 on ARC-1 and 7.8 on ARC-2. Average human is listed as 60.2 on ARC-1, best human as 98.0 on ARC-1 and 100.0 on ARC-2.

• Test-time-training strategy matters. d-1-TTT1 scored 56.6 on ARC-1 and 6.9 on ARC-2; d-1-TTT2 scored 56.4 and 7.71; d-1-TTT3 scored 55.5 and 8.3; c-2-TTT2 scored 58.0 and 8.3.

• Attention maps look more structured. Layer-wise attention visualization compared d-1-TTT1 against VARC-ViT on the same ARC puzzle. The ViT attention field progressively diffuses into mixtures over the workspace, consistent with unconstrained global token mixing; the proposed model keeps a stable, structured attention field that preserves coherent spatial primitives across depth. At layers 1 and 2, ViT's attention map is highly mosaic and fuzzy; from layer 3 it begins to resemble the proposed model, though not as crisply.

• ARC-2 gains are out-of-distribution. ARC-2 training tasks were not used in offline pretraining, so improvement on ARC-2 reflects out-of-distribution generalization and the effectiveness of test-time adaptation.

• The "smile explanation" thought experiment. A constructed task separates state-decoding intelligence (a human producing an explanation from an internal mental state) from behavior-sequence intelligence (a predictor producing a post-hoc rationalization from observable behavior). The authors call this form of post-hoc rationalization without subjective experience "hallucination" in the context of the paper.

Methodology in Plain English

The authors start from a conceptual argument: when a human explains an action, they decode an internal mental state that caused the action; a behavior-prediction model instead invents an explanation from the visible trace. To close this gap, they argue an AI system needs an explicit, readable controller state that drives both behavior and explanation.

To make this testable, they build on VARC, which treats ARC puzzles as a vision-conditioned image-to-image translation problem with a two-stage training pipeline. They keep VARC's data augmentation, training, and evaluation pipeline but swap out the dense ViT backbone for their own block.

In that block, ARC grids are first embedded onto a larger augmented canvas (for example 64 × 64) with rotations, translations, and scale changes and a fixed border, then split into non-overlapping 2 × 2 patches, each turned into a 512-dimensional token. Alongside the large set of workspace tokens, the model builds a small controller: one learned task token plus one context token per demonstration. Each context token is produced by taking the difference between the embedded demonstration output and input, then projecting it with a small MLP. At test time, the task token is adapted for the new task.

Each transformer block runs two attention passes: a standard dense pass as in a ViT, and a structured pass that enforces the role separation. In the structured pass, controller tokens attend to everything, while workspace tokens may consult only the controller and their 8 neighbors. This creates a deliberate bottleneck so the workspace cannot become a full carrier of global state. A recurrent wrapper then applies the encoder stack K times with a gate and a running exponential moving average, increasing effective depth at no extra parameter cost. A linear head maps each workspace token back to patch-level color logits, which are reassembled, de-augmented, and decoded by argmax into the output grid.

Evaluation follows VARC's three stages: offline training shared across roughly 400 ARC-1 training tasks (expanded with 1000 RE-ARC demos), per-task test-time training on the demonstration set, and inference with multi-view augmentation. In inference, 510 augmented views of the query input are consolidated by majority voting, and the two most frequent candidate grids form the pass@2 output; a task counts as solved if either candidate matches the ground truth. Results are pass@2 averaged across 4 seeds for 4 random trials.

Three test-time-training strategies were compared: TTT1 (the VARC baseline, 100 epochs at the default step size), TTT2 (learning rate reduced 3× and budget increased 3× to 300 epochs, with early termination once the task reaches 100% training accuracy in a 3-epoch streak), and TTT3 (TTT2 plus auxiliary losses: grid-level correctness supervision from a correctness head over global patch features, pixel-level error emphasis similar to focal loss, and a reward-shaping rule that down-weights the output cross-entropy and disables mid-loss once a grid is exactly correct). Models with r of 1 or 2 were trained without mid-loss; deeper recurrent settings used mid-loss for deep supervision.

Why This Matters

Impact on research. The paper offers a structural, architectural claim rather than only a scaling or prompting claim: that reasoning benefits from an explicit controller/workspace split, and that the absence of a grounded global state for decoding may be the qualitative deficiency of AI systems in human simulation. It also supplies a controlled comparison — the same VARC data, augmentation, and evaluation pipeline with only the backbone changed — so architectural role separation is isolated from other factors. The reported result that a 28M-parameter model outperforms VARC-ViT on both ARC-1 and ARC-2, and that an 83M-parameter ensemble reaches 62.6% on ARC-1 (above the reported 60.2% average human), gives the hypothesis concrete empirical weight, while the large remaining gap to best human (98.0 on ARC-1, 100.0 on ARC-2) shows how much is unresolved.

Real-world applications (these are plausible implications of the architecture, not applications demonstrated in the paper):

• Interpretability and auditability of AI decisions, if the controller state can be decoded into explanations rather than generated post hoc.
• Visual and abstract reasoning assistants that must infer a rule from a few examples and apply it to a new input.
• Systems where local consistency plus global coordination both matter, such as structured document or diagram understanding.
• Tooling for program induction and example-driven automation, where a task is specified by a handful of input-output pairs.

Industry relevance. The result is relevant to teams that need models to generalize from very few examples and then justify their behavior. Because the proposed model uses 28M parameters in its base form, the work also speaks to efficiency questions: the authors argue that increasing effective depth via recurrence helps under fixed parameter budgets, which matters when compute and memory are constrained.

Future Directions

• Making the controller readable. The paper's core motivation is that a human explains an action by decoding internal state; the experiments measure ARC accuracy and attention structure, not whether the controller state can actually be decoded into faithful explanations. Turning the hypothesized controller into an inspectable, decodable channel is the natural next step.

• Closing the ARC-2 gap. ARC-2 accuracy for c-d-4-TTT3 is 11.0 (13.5 with UNet ensembling), against a best human of 100.0 and an average human figure not reported for ARC-2. Since ARC-2 tasks were withheld from offline pretraining, improving out-of-distribution generalization and test-time adaptation is an open problem.

• Understanding staged versus end-to-end training at depth. For r = 4, training architecture d end-to-end did not always outperform architecture c; staged training (c first, then d) worked better. Why the controller bottleneck needs to form before local repair capacity is added is not fully explained.

• Generalizing beyond ARC grids. The hypothesis is stated generally, but the implementation is specific to ARC-style categorical grids with a fixed maximum size, a 64 × 64 canvas, and 2 × 2 patchification. Extending the role-separated block to other modalities and tasks is untested here.

• Refining test-time training. The three TTT strategies differ substantially in accuracy (for example, d-1 variants range from 6.9 to 8.3 on ARC-2), suggesting the interaction between TTT objectives, auxiliary losses, and structured backbones deserves further study.

Target Audience

This paper is most useful to machine learning researchers working on abstract reasoning, ARC-style benchmarks, and architecture design, particularly those interested in inductive biases that go beyond scaling dense transformers. It also suits practitioners who work with test-time training and multi-view inference pipelines, and readers following the debate about whether current systems genuinely "reason" or merely produce plausible behavior. Some familiarity with attention and vision transformers is needed to follow the architectural details, though the conceptual hypothesis is stated plainly enough for a broader technically literate audience. The code base is available at https://github.com/lz7fd/Reasoning_is_a_Modality.

Authors’ abstract

The Abstraction and Reasoning Corpus (ARC) provides a compact laboratory for studying abstract reasoning, an ability central to human intelligence. Modern AI systems, including LLMs and ViTs, largely operate as sequence-of-behavior prediction machines: they match observable behaviors by modeling token statistics without a persistent, readable mental state. This creates a gap with human-like behavior: humans can explain an action by decoding internal state, while AI systems can produce fluent post-hoc rationalizations that are not grounded in such a state. We hypothesize that reasoning is a modality: reasoning should exist as a distinct channel separate from the low-level workspace on which rules are applied. To test this hypothesis, on solving ARC tasks as a visual reasoning problem, we designed a novel role-separated transformer block that splits global controller tokens from grid workspace tokens, enabling iterative rule execution. Trained and evaluated within the VARC vision-centric protocol, our method achieved 62.6% accuracy on ARC-1, surpassing average human performance (60.2%) and outperforming prior methods significantly. Qualitatively, our models exhibit more coherent rule-application structure than the dense ViT baseline, consistent with a shift away from plausible probability blobs toward controller-driven reasoning.