HiRS-Agent: A Hierarchical Multi-Agent System for Reliable Long-Horizon Remote Sensing Task Solving

AI summary

Overview

• Research area: Artificial Intelligence / multimodal remote sensing agents, combining hierarchical multi-agent systems, supervised fine-tuning, and reinforcement learning for tool-using LLM agents.
• Technical level: Advanced (assumes familiarity with multi-agent architectures, supervised fine-tuning, LoRA, and GRPO-style reinforcement learning).
• Scope: The paper proposes HiRS-Agent, a two-level Manager–Specialist multi-agent system with verification-guided control and a two-stage training pipeline (Expert-tuning plus VG-HRL), evaluated on Earth-Agent Benchmark (Earth-Bench) and ThinkGeo with Qwen3-4B and Qwen3-8B backbones.

What This Paper Is About

Remote sensing (RS) tasks such as flood mapping unfold as long chains of interdependent stages, where an early mistake (for example, mistaking cloud shadows for water) can corrupt downstream water-index computation, inundation-area estimation, and the final report. Existing RS agent systems are largely adapted from general-purpose agent templates and exhibit three mismatches: workflow mismatch (no explicit stage-dependency modeling), knowledge mismatch (the LLM is not aligned with RS physical mechanisms, spectral constraints, or professional procedures), and control mismatch (limited verification of intermediate states or recovery from execution failures). The paper's goal is a hierarchical, verification-guided agent system whose architecture and training are both designed around the structured, stage-dependent nature of RS workflows.

Key Contributions

1. A hierarchical multi-agent framework for long-horizon RS task solving. HiRS-Agent is a two-level system tailored to the structured and stage-dependent nature of RS workflows.
2. Verification-guided workflow control. A memory-aware step-level verification mechanism that supports rerouting, replanning, and recovery under RS-specific constraints during multi-stage execution.
3. Workflow-aware training for RS agents. Expert-to-Workflow Alignment Tuning (Expert-tuning) for RS expertise injection and workflow alignment, plus Verification-Guided Hierarchical Reinforcement Learning (VG-HRL) to jointly optimize global coordination and local tool execution.
4. Systematic evaluation on representative benchmarks. Experiments on Earth-Bench and ThinkGeo show improved long-horizon RS task execution and final-task correctness, especially on lightweight open-source backbones. Code is publicly available at https://github.com/IntelliSensing/HiRS-Agent.

Main Findings

• Earth-Bench gains on lightweight backbones. For Qwen3-4B, HiRS-Agent improves Accuracy from 15.73/10.08 to 43.95/45.56 (AP/IF), Tool-Exact-Match from 0.00/8.63 to 31.67/34.64, and Param-Match from 0.00/4.21 to 19.54/20.81, while reducing Efficiency from 3.9747/5.4554 to 2.5059/3.3013. For Qwen3-8B, HiRS-Agent reaches 48.39/53.62 Accuracy.
• Improvement is not simply backbone scaling. The paper attributes gains to hierarchical specialization, verification-guided control, and post-SFT optimization, and reports that increasing backbone capacity from 4B to 8B narrows the gap to strong proprietary models.
• Context on Earth-Bench baselines. GPT-5 reaches the highest reported Accuracy (65.59/63.16) and DeepSeek-V3.1 (37B) the highest Tool-Any-Order (78.40/77.98), so HiRS-Agent leads among the shared Qwen3-4B comparisons rather than across all entries in Table 1.
• ThinkGeo step-level improvements without task-specific optimization. Qwen3-4B improves Inst./Tool./Arg. from 18.35/8.54/1.24 to 73.73/47.87/8.51 and Ans./Ans_I from 6.07/7.79 to 11.28/13.77. Qwen3-8B improves from 20.98/13.36/3.26 and 7.67/8.68 to 77.97/59.57/11.70 and 12.09/14.75, respectively, achieving the best Ans. score among all compared methods.
• Domain adaptation works without severe general-capability loss. Expert-tuning on Qwen3-4B raises RS-EXPERT-BENCHMARK overall from 73.35 to 87.60 (in-domain 76.43 to 95.84; out-of-domain 72.42 to 85.13), beating the Prompt variant (77.37 overall). MMLU-Redux stays nearly unchanged (66.57 to 66.80), while MATH-500 (67.80 to 66.20) and Multi-IF (58.35 to 55.53) show moderate drops.
• Training ablation. Expert-tuning improves Exact from 11.47/12.85 to 18.07/19.78 and Accuracy from 35.48/33.06 to 40.73/39.52; adding VG-HRL reaches 41.10/46.34 In-Order, 31.67/34.64 Exact, and 43.95/45.56 Accuracy. In-Order under AP drops from 22.31 (Base) to 19.73 (Base + ET) before VG-HRL raises it to 41.10.
• Architecture ablation. Adding verification to a flat agent improves Exact from 6.84/19.42 to 29.05/30.14. A generic hierarchy with verification reaches 32.47/34.92 In-Order, and adding RS-specialist grouping raises Accuracy from 30.24/33.47 to 43.95/45.56. Removing verification from the full system reduces Exact by 11.30/12.43 and Accuracy by 4.03/2.82, indicating that hierarchical specialization and step-level verification provide distinct, complementary benefits.

Methodology in Plain English

Architecture. HiRS-Agent mirrors the canonical RS progression from spectral parsing to physical retrieval to statistical analytics across two layers. The Manager Layer is a single Orchestrator Agent (OA) that decomposes tasks, routes sub-goals, verifies intermediate results, and decides whether to continue, replan, verify, or terminate. It keeps a structured global state s_t = {x, d, g_t, H_t, C_t} (instruction, multimodal input, current sub-goal, structured global memory, dynamic constraints), where memory stores task context, routing states, execution traces, and intermediate evidence. The Specialist Layer contains three function-oriented expert families: the Surface Parsing Agent (SPA, e.g., detection, segmentation, ROI/mask generation, NDVI/NDWI-style index parsing), the Physical Retrieval Agent (PRA, e.g., LST-oriented tasks and quantitative product generation), and the Statistical Analytics Agent (SAA, e.g., aggregation, cleaning, statistical testing, report metrics). Each specialist turns a sub-goal into a structured tool call (tool, args). All roles share one backbone and LoRA adapters, differing only by role-specific prompts.

Verification. After each specialist step, the OA retrieves relevant memory and computes a verification verdict z_t in {pass, uncertain, fail} using three checks: schema validity, sub-goal consistency, and constraint satisfaction (physical, spectral, and statistical plausibility). FAIL triggers replan-on-failure (tool repair, alternative routing, sub-goal adjustment), UNCERTAIN triggers cross-checking via re-execution or alternative reasoning paths, and PASS commits the result to memory. Failures and recovery traces are retained so later decisions are history-aware rather than myopic.

Training. Expert-tuning is a two-stage supervised strategy. Stage I builds a structured RS knowledge system from a 3-level taxonomy (3 L1 topics, 11 L2 modules, 28 L3 domains) covering Fundamentals of RS, Digital Image Processing, and Intelligent Interpretation and Applications; content is curated from four authoritative RS textbooks and processed through five stages (content extraction, normalization, chunking, rewriting, annotation) into 1,583 knowledge chunks and 5,687 knowledge-centric training samples. Stage II adds workflow supervision (684 Workflow MCQ plus 309 Workflow Order samples, 993 total, spanning 14 task families) drawn from real RS application pipelines collected from multi-agent demo logs and verified by two experts. 80% of the data becomes instruction-response pairs, 20% is held out, and RS-EXPERT-BENCHMARK tests knowledge grounding and workflow generalization using held-out in-domain data plus out-of-domain samples from unseen RS Handbook sources.

VG-HRL. Post-SFT reinforcement learning with GRPO separates trajectory-level coordination from step-level tool execution. The Manager reward combines normalized routing correctness, final task success, and a trajectory-length penalty. The Specialist reward is a graded gating reward over three tiers: Tier 1 (hallucinated out-of-domain tool) = -1.0, Tier 2 (wrong but in-domain tool) = -0.5 + 0.2 R_exec, Tier 3 (correct tool) = 0.5 + 0.2 R_exec + 0.3 R̂_args. Advantages are grouped by task for the Manager and by (task, step index) for Specialists; the two levels are combined at the loss level (mixed loss with λ = 0.5) rather than the reward level. Implementation: Expert-tuning uses LoRA rank 8 for 3 epochs; VG-HRL uses LoRA rank 64 with alpha 128, rollout group size G = 4, and 3 training epochs per round, on 2× RTX 4090 24GB for Qwen3-4B and 4× RTX 4090 24GB for Qwen3-8B.

Why This Matters

Impact on research. The paper argues that the key question for RS agents is not whether multiple agents are used, but whether the system maintains execution coherence under evolving intermediate states. It treats step-level verification and state-adaptive rerouting as first-class control signals and shows that process-level optimization matters more than final-answer supervision alone for long-horizon tool use. It also provides a reusable domain-adaptation recipe and an RS-specific benchmark (RS-EXPERT-BENCHMARK) for measuring specialization versus general-capability retention.

Real-world applications (as motivated in the paper):

• Flood mapping and inundation-area estimation, where early parsing errors such as mistaking cloud shadows for water propagate into water-index computation and final reporting.
• Disaster assessment and environmental monitoring built on multi-temporal, multi-sensor observation data.
• Earth observation and geospatial intelligence workflows that chain spectral parsing, physical inversion, and statistical analysis.
• Quantitative RS product generation, including LST-oriented retrieval with calibration, normalization, and quality-control checks.

Industry relevance. The largest gains appear on lightweight open-source backbones (Qwen3-4B, Qwen3-8B), which is directly relevant for deployments with constrained compute, such as 2× or 4× RTX 4090 24GB setups used in the paper. On

Authors’ abstract

Recent advances in large language models and multimodal models have pushed remote sensing (RS) processing from simple perception models to agentic systems designed to tackle complex, long-horizon RS tasks. However, existing systems often rely on monolithic decision-making frameworks, which fail to accommodate the multi-stage, interdependent nature of RS tasks. This centralized approach leads to challenges such as unstable task execution, incorrect tool usage, and error propagation across stages. To address these issues, we propose HiRS-Agent, a hierarchical multi-agent system for long-horizon RS task solving. HiRS-Agent adopts a two-level collaborative architecture: the Manager Layer handles dynamic routing, step-level verification, replanning, and termination control, while the Specialist Layer organizes domain-specific tools according to the RS workflow and is responsible for subtask reasoning and tool execution. To further enhance the system's capability, we introduce a two-stage supervised tuning strategy and a verification-guided hierarchical reinforcement learning stage to jointly optimize coordination and tool-use policies. Experiments on Earth-Agent Benchmark and ThinkGeo show that HiRS-Agent substantially improves long-horizon tool-use capability and final-task correctness, demonstrating the effectiveness of structured multi-agent collaboration for reliable RS agents. The code is publicly available at https://github.com/IntelliSensing/HiRS-Agent.