ARCTraj: A Dataset and Benchmark of Human Reasoning Trajectories for Abstract Problem Solving

AI summary

Overview

Research area: Artificial intelligence — abstract reasoning, benchmark and dataset construction, and human-in-the-loop modeling of reasoning processes (specifically the Abstraction and Reasoning Corpus, ARC).

Technical level: Intermediate. The paper is readable without deep mathematics, but it uses reinforcement-learning and sequential-modeling vocabulary (MDPs, state–action pairs, Decision Transformers, GFlowNets, diffusion agents) and formalizes ARC as a few-shot prediction problem.

Scope in one sentence: The paper introduces ARCTraj, a dataset of roughly 10,000 temporally ordered, object-level human action trajectories across the 400 ARC-AGI-1 training tasks, together with analyses of selection, color, and strategy patterns and a proposed pipeline for using that data to train ARC solvers.

What This Paper Is About

Most ARC research treats the benchmark as a static input–output prediction problem: a model sees a few example grid pairs and must produce the correct output grid for a new input. That framing records only final answers and says nothing about the sequence of perceptions, hypotheses, and transformations a human performs on the way there. ARCTraj closes this gap by capturing step-by-step human interactions with ARC grids through the O2ARC web interface, then packaging those logs as structured supervision that reinforcement learning, generative, and sequence models can consume.

Key Contributions

1. A large trajectory dataset. ARCTraj contains 10,672 trajectories collected from the O2ARC web interface, covering all 400 training tasks of the ARC-AGI-1 benchmark. Each log stores an ordered actionSequence of symbolic actions annotated with task IDs, user IDs, timestamps, and success labels.

2. An object-centric action representation. Every action is a triplet ⟨category, object, operation⟩ paired with its grid state. Categories include Selection, Coloring, Object-Oriented, Clipboard, and Critical; operations include Move, Paint, Flip, Copy, Paste, and Rotate. Objects are perceptually meaningful regions that users explicitly confirm through selection, so the representation reflects human perceptual grouping rather than heuristic clustering.

3. A unified reasoning pipeline. The paper defines an end-to-end framework spanning data collection, action abstraction, Markov decision process (MDP) formulation, and downstream learning, with trajectories expressed as ⟨s_t, a_t, s_{t+1}⟩ transitions and a binary reward r_t ∈ {1, 0} indicating whether the current grid matches the correct output.

4. An analysis of human reasoning regularities. The authors derive three forms of auxiliary knowledge from the trajectories — selection biases, color origins, and shared intentions — corresponding to "where" humans attend, "what" information they abstract, and "how" they organize multi-step strategies.

Main Findings

• ARCTraj is denser per task than H-ARC. On the same 400 ARC-AGI-1 training tasks, ARCTraj records 10,672 trajectories versus H-ARC's 7,916, with an average of 25.5 trajectories per task versus 19.8, and 13.9 participants per task versus 11.8 — despite H-ARC having 783 participants and ARCTraj having 100 reported in Table 1.

• Note on participant counts. Section 3.1 states the dataset was contributed by "more than 300 participants," whereas Section 3.2 and Table 1 report about 100 participants. The paper does not reconcile these two figures.

• Object-level abstraction produces more efficient trajectories. ARCTraj contains 208,721 total actions, dropping to 84,123 when Selection steps are excluded. H-ARC records 241,697 actions. ARCTraj has 33,608 unique visited states versus H-ARC's 127,146, indicating that higher-level object operations reach the same coverage with less redundant state exploration.

• A much larger share of actions are object-related. 15.2% of ARCTraj actions are object-related (37.7% when Selection steps are excluded), compared with 0.9% for H-ARC.

• Human solvers converge on shared intermediate states. 43.7% of ARCTraj grids are cross-trajectory grids (states visited in more than one trajectory), versus 11.4% for H-ARC. This suggests participants frequently reach the same intermediate configurations through different strategies.

• Selections are small, compact, and shape-biased. Selection bounding boxes cluster between 1×1 and 3×3, with square-shaped (n×n) and bar-shaped (n×1 or 1×m) regions dominating. Most selections cover fewer than 20 pixels, and peaks occur near square numbers (1, 4, 9, 16), consistent with a bias toward local, symmetric regions.

• Color sources are constrained by task design. Of the 400 training tasks, 266 (66.5%) can be solved using only colors present in the test input grid, and 134 (33.5%) require colors drawn from both the test input and the example output grids. No task (0.0%) requires colors that appear exclusively in the example inputs. ARCTraj does not explicitly log where users obtained colors, but observed behavior aligns with these sources.

• Humans act after short exploratory phases. 23,632 operations (63.7%) are preceded by a single selection, and over 90% occur within four selections. The longest gap recorded is 386 selections preceding a single operation in one instance.

• Strategy convergence varies by task. Tasks with low trajectory uniqueness correspond to well-structured problems with canonical solutions (the paper illustrates this with Task c0f76784), while high-uniqueness tasks indicate flexible or ambiguous reasoning spaces.

• Reported benefits of ARCTraj signals in downstream solvers. Decision Transformers trained with intention cues gain 6–8 points over demonstration-only baselines; GFlowNet models augmented with selection priors produce trajectories with 14% higher diversity; diffusion-based solvers integrating intermediate human states show more consistent reconstruction of multi-step transformations. The paper content provided ends mid-sentence at this point, so the full set of empirical results is not reported here.

• Baseline solver performance summarized. The paper's comparison table lists PPO at 55–70% (online training in ARC-like MDPs), World Model at 38–100% (analogical generalization via latent dynamics), Decision Transformer at 59–90% (trajectory-conditioned policies with inferred intentions), Diffusion at 77–92% (intermediate-state generation for plan synthesis), and GFlowNet at 10–100% (sampling diverse goal-directed trajectories, with an augmentation rather than solve objective).

Methodology in Plain English

The researchers built on an existing web interface, O2ARC, that lets people manipulate ARC grids by clicking on regions of the grid rather than editing individual pixels. Every click, selection, and transformation a user makes is logged in order. The system groups contiguous colored cells into "objects," but the user must confirm each selection, so the recorded objects reflect how a person actually perceived the grid. Each logged action is then a triplet of a category, an object, and an operation, saved alongside the grid state and a timestamp.

To make the data usable for machine learning, the authors cast each trajectory as a sequence of state–action–next-state transitions, which is the standard format for reinforcement learning. They also define an optional "auxiliary knowledge" term that a solver can condition on, derived from the human data. They then analyze the logs three ways: measuring the size and shape of selected regions (to study attention), tracing where output colors come from relative to test inputs and example outputs (to study inductive biases), and grouping recurring (selection, operation) pairs across users solving the same task (to study shared intentions). Finally, they describe how to reformat trajectories for different model families — for example, retaining grid, object, and operation for reinforcement learning environments such as ARCLE, or just grid and operation for sequence models such as the Decision Transformer.

Why This Matters

Impact on research. ARC has driven a great deal of work in program synthesis, neuro-symbolic reasoning, and test-time training, but nearly all of it is supervised only by static input–output pairs or by the intermediate results of programs the model itself generates. ARCTraj supplies a different kind of supervision: what actual humans looked at, in what order, and what they changed. That makes it possible to study reasoning as a process rather than only as an outcome, and to compare machine strategies against human ones directly.

Real-world applications:

• Program synthesis and data transformation. The paper argues that models learning from structured human strategies could transfer to automatic program construction and to transforming tabular or structured data, since both require inferring a general rule from a few examples.
• Robotics and planning. Object-level actions such as move, rotate, flip, and copy are structurally similar to manipulation primitives, so trajectory-conditioned policies could inform how agents decompose physical tasks into subgoals.
• Human–AI collaborative reasoning tools. The selection-bias and intention analyses could feed attention-prediction or intention-prediction models that anticipate what a user is about to do and offer support, as suggested in the paper's own research directions.
• Interpretability and alignment research. Because every action in ARCTraj has a semantic label and a grid context, models trained on it can be inspected at the level of intermediate reasoning steps, which the authors position as a step toward explainable and human-aligned systems.

Industry relevance. The paper is published at the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (August 9–13, 2026, Jeju, Korea). Trajectory datasets with consistent formatting and explicit state–action structure are the kind of asset that reinforcement learning, sequential decision-making, and generative-modeling teams can plug into existing training pipelines, as the paper's own preprocessing figure for ARCLE and Decision Transformer illustrates.

Future Directions

• Temporal dynamics of selection (Research Direction 1a). Do solvers begin with small exploratory selections and expand later, or start broad and then narrow in? The paper suggests temporal analyses could inspire phase-based attention models.

• Perceptual drivers of selection (Research Direction 1b). Do color contrast, isolation, or proximity to grid boundaries change which regions people select? Controlled grid manipulations could answer this and support predictive attention models.

• Explicit color-origin logging (Research Direction 2a) and generalized origin tracking (Research Direction 2b). The current interface does not record where users sampled their colors. The authors propose extending domain-specific languages with operators such as sample_color(grid, x, y) or apply_color_transformation(rule), and extending origin tracking beyond color to objects, shapes, spatial configurations, and transformation rules.

• Strategy grammars and intention prediction (Research Directions 3a and 3b). The paper proposes formalizing recurring intention groupings into a compositional grammar of strategy motifs (for example, "color and duplicate" or "fold and align"), and training models that predict human intention distributions from task features, potentially enabling curriculum sequencing, adaptive tutors, and diversity benchmarks for AI reasoning.

Target Audience

Researchers working on abstract reasoning, ARC-style benchmarks, and human-aligned machine learning; reinforcement learning and sequential modeling practitioners who need temporally structured supervision; cognitive scientists and HCI researchers interested in how people decompose and solve visual puzzles; and dataset or benchmark builders looking for a worked example of turning interaction logs into MDP-compatible training data. Readers new to ARC will find the framing accessible, but the solver-comparison and learning-paradigm sections assume familiarity with PPO, Decision Transformers, diffusion models, and GFlowNets.

Authors’ abstract

We present ARCTraj, a dataset and methodological framework for modeling human reasoning through complex visual tasks in the Abstraction and Reasoning Corpus (ARC). While ARC has inspired extensive research on abstract reasoning, most existing approaches rely on static input-output supervision, which limits insight into how reasoning unfolds over time. ARCTraj addresses this gap by recording temporally ordered, object-level actions that capture how humans iteratively transform inputs into outputs, revealing intermediate reasoning steps that conventional datasets overlook. Collected via the O2ARC web interface, it contains around 10,000 trajectories annotated with task identifiers, timestamps, and success labels across 400 training tasks from the ARC-AGI-1 benchmark. It further defines a unified reasoning pipeline encompassing data collection, action abstraction, Markov decision process (MDP) formulation, and downstream learning, enabling integration with reinforcement learning, generative modeling, and sequence modeling methods such as PPO, World Models, GFlowNets, Diffusion agents, and Decision Transformers. Analyses of spatial selection, color attribution, and strategic convergence highlight the structure and diversity of human reasoning. Together, these contributions position ARCTraj as a structured and interpretable foundation for studying human-like reasoning, advancing explainability, alignment, and generalizable intelligence.