MATA: A Trainable Hierarchical Automaton System for Multi-Agent Visual Reasoning

AI summary

Overview

• Research area: Visual reasoning with vision-language models (VLMs), compositional and agentic multi-agent systems, neuro-symbolic methods, and finite-state automata as control abstractions.
• Technical level: Intermediate. The core idea is intuitive (route a query between specialized helper agents), but the paper formalizes it as a Mealy machine, builds a trajectory-tree data pipeline, and reports ablations over SFT configurations.
• Scope: The authors (Monash University, Australia) propose a hierarchical automaton in which a supervised-finetuned LLM "hyper agent" learns which of three visual reasoning agents to invoke next, trained on a generated 90,854-example transition dataset and evaluated on GQA, OK-VQA, RefCOCO/RefCOCO+/RefCOCOg, and Ref-Adv.

What This Paper Is About

Monolithic VLMs perceive well but reason implicitly, which makes their errors hard to audit and causes hallucinations on queries about spatial relations, attributes, and counting. Existing compositional methods improve transparency, but most rely on a single agent or a hand-written pipeline whose order is fixed in advance and cannot decide when to collaborate across different agents or let overlapping agents compete for the same subtask. MATA's goal is to replace those hand-crafted transition rules with a learned policy: a trainable LLM hyper agent that reads a shared memory of intermediate results and chooses the next agent state.

Key Contributions

1. A hierarchical automaton system (MATA) that unifies a neuro-symbolic framework with collaborative and competitive multi-agent design for visual reasoning. Each top-level state is an agent that internally runs a small rule-based sub-automaton, and all agents read and write an append-only shared memory.
2. A learnable transition policy: the top-level transition function is implemented by a trainable LLM hyper agent rather than hand-written rules, so the system can decide when agents should collaborate (build on each other's context) or compete (take over when another stalls or fails).
3. A transition-trajectory data generation pipeline and the MATA-SFT-90K dataset (N = 90,854 examples). Trajectory trees are expanded per image-query pair, leaves are scored with task metrics, and each decision point is labeled with the child leading to the highest-scoring subtree, producing memory-to-next-state pairs for supervised finetuning (SFT).
4. Comprehensive experiments across visual reasoning benchmarks with ablations on the hyper agent, generalizability, LLM controller size, and number of agents.

Main Findings

• GQA accuracy: MATA reaches 64.9% (General setting) and 64.7% (Domain-Specific), above compositional baselines HYDRA (52.8), VisRep (51.4), and ViperGPT (37.9), and above monolithic models including InternVL3.5 (8B) at 63.8 and Qwen2.5-VL (7B) at 62.4.
• OK-VQA accuracy: MATA achieves 76.5% (Domain-Specific) and 76.0% (General), exceeding compositional systems DWIM (62.8) and HYDRA (59.4), and monolithic models Qwen2.5-VL (71.8) and InternVL3.5 (75.7).
• Referring expression comprehension: MATA reports 96.3 on RefCOCO, 93.8 on RefCOCO+ (General), 90.7 on RefCOCOg, and 77.3 on Ref-Adv. Context from Table 4: NAVER records 96.2 / 92.8 / 91.6 / 75.4 and Florence2-L records 95.1 / 92.5 / 90.9 / 71.8, so on RefCOCOg NAVER's 91.6 is numerically higher than MATA's 90.7/90.8, and on Ref-Adv GroundingDINO-B's 78.0 is higher than MATA's 77.3, even though the paper describes its results as state-of-the-art on these datasets.
• Domain transfer works: Ref-Adv contains only a test set, so MATA-SFT-90K holds no data collected from it; the 77.3 result is therefore a domain-transfer result. In Table 6, off-diagonal (cross-domain) values are close to diagonal (domain-specific) values, with less than 1% difference for the GQA ↔ OK-VQA transfer directions.
• Joint training does not hurt: the last row of Table 6 ("All") reports 64.9 / 76.0 / 96.3 / 93.8 / 90.7 / 77.3, similar to training on the corresponding subset only, which the authors read as evidence the controller learns a task-agnostic transition policy.
• Ablation of components (Table 5, GQA / OK-VQA / RefCOCO / RefCOCO+ / RefCOCOg / Ref-Adv, seconds per query): exhaustive ensemble without the hierarchical automaton scores 57.7 / 71.5 / 87.7 / 85.6 / 81.7 / 73.1 at 34.58 s; random transition scores 57.1 / 71.1 / 85.3 / 83.8 / 81.1 / 73.2 at 6.91 s; an unfinetuned LLM controller scores 58.5 / 75.1 / 95.8 / 93.5 / 88.0 / 76.0 at 8.07 s; the full LLM + SFT system scores 64.9 / 76.5 / 96.3 / 93.9 / 90.8 / 77.3 at 8.01 s.
• Controller size: with domain-specific SFT, even small models (0.6B/1.7B) are described as performing competitively with 4B and 8B. Joint SFT on all data leaves small models a few percentage points behind 4B/8B. Without SFT, accuracy drops sharply for smaller models. The authors choose 4B as default.
• Number of agents (GQA): one Specialized agent reaches 61.5%, adding the Oneshot reasoner lifts it to 64.5%, and adding the Stepwise reasoner gives a marginal further gain to 64.9%, which the authors interpret as diminishing returns on current benchmarks rather than evidence that agent count is the main driver.

Methodology in Plain English

MATA treats a visual question as a journey through a state machine. The states are three agents — Oneshot (a single-pass program generator with a lightweight verifier), Stepwise (a slow, multi-step Python reasoner whose code is verified and run in a sandbox), and Specialized (fast System-1 perception such as detection) — plus three lifecycle states: Initial, Final, and Failure. Formally this is a Mealy machine, M_θ = (S, S₀, Σ, Λ, δ_θ, Γ), where inputs are shared-memory snapshots and outputs are answers.

All agents read and write one append-only memory, so at step t the memory is m_t and, after an agent runs, m_{t+1} = m_t ∪ Δm_t. The trainable part is the transition function δ_θ(s_t, m_t), implemented by a finetuned LLM that receives a text prompt built from the memory (task description, query, feedback, code, variables, state history, current state, step number) and returns the next state. Micro-steps inside each agent stay rule-based, because those procedures are easy to define; only the ambiguous cross-agent choice is learned.

To get supervision labels, the authors run the system step by step and expand a bounded transition-trajectory tree to depth T for each image-query pair, branching over all possible next states instead of committing to one path. Leaves are scored with IoU(ŷ, y) for visual grounding and Acc(ŷ, y) for VQA, and those scores are propagated upward with V(s) = metric(ŷ_s, y) at leaves and V(s) = max over children V(s') otherwise. Each decision point is then labeled with s_t* = argmax V(s) over its children, producing prompt/next-state pairs. Running this over training splits of GQA, OK-VQA, and RefCOCO/RefCOCO+/RefCOCOg yields MATA-SFT-90K (90,854 examples), used to finetune the LLM controller with AdamW, cosine decay with 5% warm-up, global batch size 64, for 8 epochs.

Inference starts at Initial, lets the hyper agent pick a state, runs that agent until it returns control, and repeats until Final or the step limit of T = 15. On Failure, the hyper agent re-selects while temporarily removing the failed agent to avoid infinite retries. Experiments use InternVL2.5 (8B) as the VLM, Florence2-L for detection, DepthAnythingV2 for depth, and a Qwen3 (4B) LLM as the controller, on 4 RTX 4090 GPUs. Three SFT configurations are compared: domain-specific (train and test on the same dataset), domain-transfer (train on a different dataset than the evaluation target), and general (train on all of MATA-SFT-90K).

Why This Matters

• Research impact: The paper shifts inter-agent control from hand-written rules to a learned, auditable transition policy. It argues that hand-coded transitions become impractical as states and competing agents multiply, while a learned policy can redirect to an alternative agent after a failure — an explicit error-handling mechanism that fixed pipelines lack.
• Transparency: Because memory is append-only and states are explicit, the full reasoning history is inspectable, which is a different proposition from the implicit reasoning of a monolithic VLM.

Real-world applications implied by the agent design and benchmarks:

• Assistive and accessibility tools that answer questions about a scene or locate a described object for users who cannot see it.
• Visual inspection or industrial QA, where a fast perception check suffices for simple cases and a slower verified program is escalated to only when needed.
• External-knowledge question answering over images (the OK-VQA setting), such as identifying products or landmarks that require lookup beyond the image.
• Robotics or agentic assistants where a controller must decide which perception or reasoning module to call next and recover gracefully when one fails.

Industry relevance: The 8.01 s per-query runtime for the full system versus 34.58 s for the exhaustive ensemble shows the routing policy buys a large efficiency gain over calling every agent, and the finding that a 4B controller is near-optimal indicates the approach is deployable without the largest models.

Future Directions

• Scaling the transition search: The authors state as a limitation that the data generation pipeline performs a near-exhaustive transition search, which is tractable with three agents but may become costly as the number of states grows. Reducing this cost is an explicit open problem.
• Adding more agents: The formulation is described as modular and able to scale to additional agents without changing other parts of the system, but the near-exhaustive expansion and the diminishing returns from 2 to 3 agents leave open how far this scales.
• Rethinking the RefCOCOg and Ref-Adv results: Since NAVER scores 91.6 on RefCOCOg and GroundingDINO-B scores 78.0 on Ref-Adv, above MATA's 90.7/90.8 and 77.3, there is room to examine where the learned policy is still weaker than dedicated baselines.
• Smaller controllers and cross-task knowledge: The observation that small models fall a few percentage points behind 4B/8B under joint training suggests limited capacity to absorb cross-task knowledge, an avenue for better training or distillation.

Target Audience

Researchers and practitioners working on vision-language models, agentic and compositional visual reasoning, and neuro-symbolic systems, plus engineers building multi-agent pipelines who need learned routing rather than fixed workflows. Readers should be comfortable with supervised finetuning, evaluation benchmarks, and basic state-machine concepts; the appendices referenced for detailed agent implementations, additional generalizability analysis, efficiency, comparison with direct SFT, and qualitative examples are not included in the available text, so those specifics are not reported here. The paper declares that GPT-5/5.1/5.2 were used for language polishing, and the work is sponsored by the DARPA ANSR program under award number FA8750-23-2-1016. Code and dataset are stated as available at https://github.com/ControlNet/MATA.

Authors’ abstract

Recent vision-language models have strong perceptual ability but their implicit reasoning is hard to explain and easily generates hallucinations on complex queries. Compositional methods improve interpretability, but most rely on a single agent or hand-crafted pipeline and cannot decide when to collaborate across complementary agents or compete among overlapping ones. We introduce MATA (Multi-Agent hierarchical Trainable Automaton), a multi-agent system presented as a hierarchical finite-state automaton for visual reasoning whose top-level transitions are chosen by a trainable hyper agent. Each agent corresponds to a state in the hyper automaton, and runs a small rule-based sub-automaton for reliable micro-control. All agents read and write a shared memory, yielding transparent execution history. To supervise the hyper agent's transition policy, we build transition-trajectory trees and transform to memory-to-next-state pairs, forming the MATA-SFT-90K dataset for supervised finetuning (SFT). The finetuned LLM as the transition policy understands the query and the capacity of agents, and it can efficiently choose the optimal agent to solve the task. Across multiple visual reasoning benchmarks, MATA achieves the state-of-the-art results compared with monolithic and compositional baselines. The code and dataset are available at https://github.com/ControlNet/MATA.