Spec-o3: A Tool-Augmented Vision-Language Agent for Rare Celestial Object Candidate Vetting via Automated Spectral Inspection

AI summary

Overview

Research area: Multimodal large language models (vision-language agents) applied to astronomical spectroscopy, specifically automating the expert vetting stage of rare celestial object catalog construction.

Technical level: Advanced. It combines tool-augmented VLM agent training (cold-start SFT plus GRPO reinforcement learning) with domain-specific astrophysical spectral diagnostics.

Scope: This paper builds and evaluates Spec-o3, a tool-augmented vision-language agent that inspects spectra through iterated zoom-and-reason steps to decide whether a candidate spectrum belongs to a rare object class, reporting state-of-the-art accuracy on five LAMOST rare-object tasks plus generalization tests on SDSS, DESI, unseen spectral types, and extreme cases.

What This Paper Is About

Building catalogs of rare celestial objects requires a two-stage process: automated screening over massive spectral archives, then expert visual vetting of the surviving candidates. Deep learning can handle the screening, but vetting still needs humans because classifiers give opaque probability scores with limited out-of-distribution generalization, and post-hoc explainability methods produce coarse feature attributions that do not map reliably onto astrophysical structures. For example, building a cataclysmic variable catalog from LAMOST required experts to visually inspect about 170k candidates screened from roughly 10 million archived spectra, ultimately confirming only 323 objects.

The goal of this paper is to build an agent that vets candidates the way astronomers actually do, iteratively forming an initial judgment from a global spectrum view and then zooming into specific wavelength regions, while producing reasoning traces that experts can audit and trust.

Key Contributions

1. A rigorous expert-driven data construction pipeline and a high-quality astronomy interleaved multimodal chain-of-thought (iMCoT) dataset, providing a reliable cold start for automated spectral inspection.

2. Spec-o3, an end-to-end tool-augmented VLM agent that achieves 76.5% macro-F1 on rare celestial object vetting, outperforming specialized deep models and proprietary VLMs.

3. Demonstrated zero-shot generalization across unseen surveys (LAMOST to SDSS/DESI) and unseen inspection tasks, plus an approximately 50x inference throughput gain over expert manual inspection.

4. Expert-validated reasoning reliability, with astronomer ratings and LLM-judge scores showing that Spec-o3's trajectories are coherent and physically consistent, supporting transparent and trustworthy decision-making.

Main Findings

• State-of-the-art in-distribution performance: On SpecVI-Bench across five tasks (Cataclysmic Variables (CV), Carbon Stars (CS), S-type Stars (SS), M-type Giants (MG), White Dwarfs (WD)), Spec-o3-7B reaches a macro-average F1 of 76.5% and accuracy of 88.7%. Spec-o3-3B reaches 73.3% macro-F1 and 88.8% accuracy.

• Large gains over the base model: The two-stage post-training lifts macro-average F1 from 26.1% to 73.3% (+47.2%) for the 3B model and from 28.3% to 76.5% (+48.2%) for the 7B model relative to Qwen2.5-VL base models.

• Beats proprietary and larger science VLMs: Spec-o3-7B surpasses o3 (52.3% macro-F1), GPT-4.1 (29.8%), and S1-VL-32B-SFT (42.2%), which is a 32B model post-trained on multi-discipline scientific data.

• Beats fine-tuned specialist deep learning models on F1: GaiaNet reaches a similar average accuracy (89.0%) but lower macro-average F1 (64.9%) than Spec-o3-7B (76.5%), indicating a better precision-recall balance on imbalanced data. CarbonNet (83.7% Acc / 64.3% F1) and AstroCLIP (87.3% Acc / 64.5% F1) also score lower on F1.

• Cross-survey generalization: Zero-shot on SDSS and DESI, Spec-o3-7B reaches 81.1% and 77.4% average F1 respectively, close to its LAMOST in-distribution reference of 81.5%, versus o3 at 56.1%. Specialist deep learning models degrade by -17.6% to -13.1% on SDSS and -20.1% to -14.7% on DESI relative to their LAMOST reference average.

• Cross-task generalization: On three unseen types (O, B, A), Spec-o3-7B averages 76.4% F1 (+15.5% over o3, +45.9% over Qwen2.5-VL-7B), and Spec-o3-3B averages 74.4% (+13.5% over o3, +47.3% over Qwen2.5-VL-3B). Deep learning baselines are omitted because they lack this capability.

• Extreme-case generalization: On the QSO subset, Spec-o3-7B achieves 71.9% F1 zero-shot versus o3 at 47.5% and Qwen2.5-VL-7B at 16.9%. A brief incremental SFT with only 96 synthetic trajectories improves the 3B model from 68.4% to 73.6%. On the Low-SNR subset (SNR < 5), Spec-o3-7B retains 78.2% F1, only about 3 points below its standard-SNR performance (81.0%).

• Both training stages are necessary: Ablations on SpecVI-Bench show RL alone reaches 35.7 (3B) / 40.5 (7B) average F1, cold-start SFT alone reaches 33.1 / 41.6, neither stage reaches 26.1 / 28.3, and the combined two-stage recipe reaches 73.3 / 76.5.

• Tool access is essential: Removing the visualization tool drops performance to 43.5 (3B) and 55.8 (7B) even with full SFT and RL, showing a static global view is insufficient for reliable verification.

• Cold start is data-efficient: Reducing SFT data from about 1k to about 200 trajectories causes only a minor drop (CV F1: 80.7 to 77.8 for the 3B model). For QSO, seeding a synthetic trajectory generator with just 3 expert examples raises the LLM-filter pass rate from 25% to 78%, and adding answer guidance pushes it to 98%.

• Fast inference: Spec-o3 processes spectra at approximately 0.2 seconds per sample on an 8x H100 server including tool rendering, about a 50x throughput gain over expert manual inspection (approximately 10 seconds per spectrum).

• Expert-validated reasoning: Six astronomers rated 100 sampled reasoning trajectories (50 each from the 3B and 7B models) on a 0-5 scale for coherence and physical consistency, with score distributions closely agreeing with four LLM judges (GPT-5, Gemini-2.5-Pro, Claude-4-Sonnet, Grok-4). In a pairwise preference study of 50 cases per survey, Spec-o3 was preferred at least as often as o3 across datasets, with a low loss rate.

Methodology in Plain English

The inspection loop. Spec-o3 is built on Qwen2.5-VL and mimics the astronomer's workflow. It receives a text prompt (a discriminative question about whether the spectrum matches a target category, plus an expert-written summary of diagnostic features and common contaminants) and an initial full-range spectrum plot. At each step it either produces a final answer or calls a spectral visualization tool, specifying a wavelength interval and an optional label. The tool caches the original one-dimensional wavelength-flux array, slices the requested segment, renders a zoomed plot, and returns it. The agent alternates between text reasoning and these rendered zoomed images (the interleaved multimodal chain-of-thought, iMCoT) until it answers or hits a preset tool-call limit.

Benchmark construction. SpecVI-Bench contains five binary verification tasks, one per rare type. Positive samples come from official LAMOST catalogs. Negatives are built by rejection sampling to mimic the high-confusion vetting stage: a weak classifier is trained per task, spectra are sampled from the full LAMOST archive, and only sources not in the official catalog whose predicted positive probability exceeds 0.8 are retained. The average acceptance rate is 1.74%. Training splits are approximately balanced, while test splits are deliberately imbalanced to reflect the rarity of true targets.

Cold start (stage one). Spectra are sampled from LAMOST official catalogs for the five tasks plus confusing types chosen by astronomers, filtered at signal-to-noise ratio greater than 10, yielding roughly 4k spectra. Astronomers write simplified inspection guidelines. GPT-5 uses these guidelines plus ground-truth labels to invoke the visualization tool and generate draft trajectories. Astronomers discard implausible drafts; one astronomer revises each remaining draft (correcting tool arguments and rewriting the analysis); two additional astronomers independently audit; failed audits are revised and re-audited until all three approve. GPT-5 then standardizes approved trajectories into iMCoT format, and the three astronomers take a final acceptance vote. This yields about 1k high-quality trajectories used for supervised fine-tuning. A token-level loss mask is applied to tool returns so the model learns invocation and interpretation methods rather than memorizing rendered outputs.

Reinforcement learning (stage two). Because expert trajectories are scarce, outcome-based RL is applied on more abundant label-only data using Group Relative Policy Optimization (GRPO), with a token-wise loss mask on tool response tokens. The reward is rule-based: 1 for a correct answer with valid formatting, 1-alpha for correct answer with format violation, 0 for wrong answer with valid formatting, and -alpha for wrong answer with format violation. No explicit tool-usage reward term is added because tool use is already reliable after cold start. Training used Qwen2.5-VL-3B and Qwen2.5-VL-7B bases, 8x NVIDIA H100 GPUs, 8 rollouts per question, and a maximum of 8 tool calls per trajectory. All VLM baselines share Spec-o3's visualization tool and prompt for fair comparison.

Evaluation. Metrics are accuracy and positive-class F1 per task with macro-averages; F1 is primary because tests are imbalanced. VLM outputs must be exactly boxed YES or NO. Beyond SpecVI-Bench, the team tests a Cross-Survey set from SDSS and DESI, a Cross-Task set of unseen categories (O, B, A), and an Extreme-Case set covering QSOs and low-SNR (SNR < 5) cataclysmic variables.

Why This Matters

This work targets a genuine scaling bottleneck: expert visual vetting cannot keep up with the candidate volumes produced by current and next-generation spectroscopic surveys. By automating inspection with an auditable, workflow-aligned process, the paper offers a path to catalog construction that is both faster and more transparent than opaque classifiers, while retaining the reasoning style that astronomers already trust.

Real-world applications:

• Large-scale catalog construction for rare stellar and galactic object types in spectroscopic surveys.
• Triage and ranking of survey candidates, reducing the number of spectra that require astronomer review.
• Difficulty triage and flagging: Spec-o3's scores could be reused as a scalable quality filter or for flagging uncertain cases, though the paper notes production-oriented risk controls (calibration, abstention, triage) are not yet provided.
• Cross-survey transfer: the same agent can be applied to data from different telescope projects (LAMOST, SDSS, DESI) without retraining.

Industry relevance: The results matter to any organization that must apply multimodal agents to scientific or technical instrument data where interpretation depends on domain-specific evidence rather than natural images. The data-efficiency findings (about 200 trajectories versus about 1k, and bootstrapping synthetic generation from 3 expert examples) are relevant to anyone who must cold-start a specialized agent with limited expert labeling.

Future Directions

• Broaden coverage: The paper states its evaluation covers a limited set of rare-object types and does not yet cover broader spectral subclasses or the most challenging observational conditions.
• Incorporate other evidence and modalities: Real catalog construction often requires cross-matching external databases and using photometry, imaging, or time-domain evidence, which the current zoom-and-reason loop does not include.
• Reduce the cold-start dependency: Although outcome-based RL scales with label-only data, the approach still relies on expert demonstration trajectories for cold start, a non-trivial barrier for new tasks or surveys.
• Add production risk controls: The authors note they do not yet provide calibration, abstention, or triage mechanisms for deferring uncertain cases to human experts.

Target Audience

This paper is most valuable to researchers and engineers working on multimodal agents, tool-augmented reasoning, and reinforcement learning post-training for domain-specific applications. It is also directly relevant to astronomers and survey data scientists building and validating rare-object catalogs, and to practitioners interested in how expert-in-the-loop data curation can be combined with RL to make scientific AI trustworthy and deployable at scale.

Authors’ abstract

Due to the limited generalization and interpretability of deep learning classifiers, The final vetting of rare celestial object candidates still relies on expert visual inspection--a manually intensive process. In this process, astronomers leverage specialized tools to analyze spectra and construct reliable catalogs. However, this practice has become the primary bottleneck, as it is fundamentally incapable of scaling with the data deluge from modern spectroscopic surveys. To bridge this gap, we propose Spec-o3, a tool-augmented vision-language agent that performs astronomer-aligned spectral inspection via interleaved multimodal chain-of-thought reasoning. Spec-o3 is trained with a two-stage post-training recipe: cold-start supervised fine-tuning on expert inspection trajectories followed by outcome-based reinforcement learning on rare-type verification tasks. Evaluated on five rare-object identification tasks from LAMOST, Spec-o3 establishes a new State-of-the-Art, boosting the macro-F1 score from 28.3 to 76.5 with a 7B parameter base model and outperforming both proprietary VLMs and specialized deep models. Crucially, the agent demonstrates strong generalization to unseen inspection tasks across survey shifts (from LAMOST to SDSS/DESI). Expert evaluations confirm that its reasoning traces are coherent and physically consistent, supporting transparent and trustworthy decision-making. Code, data, and models are available at https://github.com/Maxwell-Jia/spec-o3.