MorphoOrgaAgent: A Foundation-Model-Based Multi-Agent System for Autonomous Organoid Analysis

AI summary

Overview

Research area: Biomedical image analysis and multi-agent LLM systems, applied to organoid microscopy. Technical level: Intermediate (requires familiarity with segmentation models, LLM agents, and basic morphological metrics). Scope: The paper presents MorphoOrgaAgent, a three-component multi-agent framework that turns a biologist's natural-language question about an organoid image into zero-shot instance segmentation, quantitative morphology measurements, and a written analysis report.

What This Paper Is About

Organoids are 3D tissue models whose size, shape, and count reveal how tumors grow and how cultures respond to drugs, but extracting these measurements normally requires manual segmentation and custom code. Existing organoid segmentation tools are trained on specific datasets and organoid types, so they generalize poorly, and existing LLM-based bioimage agents are generic rather than tailored to organoid morphology. This work builds and benchmarks a purpose-built agent that performs zero-shot segmentation, computes statistics, and writes reports directly from a natural-language request.

Key Contributions

1. MorphoOrgaAgent pipeline — A multi-agent system combining a TaskUnderstandingAgent (intent parsing and metric/visualization selection), a hybrid-prompt segmentation module, and a ReportAgent, all coordinated through a shared JSON-serializable state container (MorphoOrgaState) that keeps every intermediate result auditable.
2. Hybrid-prompt zero-shot segmentation — Cellpose masks are repurposed as geometric prompts and combined with the text prompt "cell cluster" to steer SAM 3, avoiding organoid-specific training and outperforming StarDist-based zero-shot segmentation.
3. MorphoOrgaVQA benchmark — A 16-question Visual Question Answering benchmark built from three public expert-annotated datasets (OrganoID, OrgaExtractor, OrgaSegment; 69 images; 1,104 question–answer pairs), with paired "clear" and "open" phrasings and a released script that derives ground-truth answers deterministically from annotated masks.
4. Hallucination-constrained reporting — The ReportAgent is explicitly forbidden from recalculating or overriding numbers, using the LLM only as a reasoning engine over pre-computed state data.

Main Findings

• Roundness is handled best: Median absolute percentage error under 1.6% for single-extremum queries (1.54% clear, 1.56% open) and under 2.6% for tertile-mean queries (2.48% clear, 2.51% open).
• Area and perimeter are moderate: Single-extremum median APE of 15.36–17.40% for area and 15.36% for perimeter; tertile-mean errors rise to 38.74% (area, clear) and about 22% (perimeter).
• Roughness is the hardest metric: 46.02% median APE on single-extremum queries, because roughness depends on the difference between real and idealized perimeters and is highly sensitive to pixel-level boundary noise.
• Phrasing mode barely matters: Clear and open prompts produce nearly identical error profiles across all four metrics and both query types, indicating the system correctly infers metrics from colloquial biologist language.
• Qualitative comparison favors the proposed system: On a complex colon-organoid query requiring roughness filtering of the top 10% and area quantification, MorphoOrgaAgent reported a median area of 5,771.5 px against a ground truth of 5,749 px (0.39% error), versus 1,235 px for Agentic-J and 761 px for Omega.
• Baselines fail for different reasons: Omega's errors stem from upstream StarDist segmentation failures on organoid images, while Agentic-J, given the same predicted mask, defaults to generic metrics such as convexity instead of domain-specific roughness descriptors.

Methodology in Plain English

Every query starts a fresh MorphoOrgaState object holding the image path and the user's question. The TaskUnderstandingAgent (powered by GPT-5.4-mini) reads that question, decides which metrics and which visualization categories are needed from a fixed pool, rewrites the request into a precise scientific instruction, and writes all of it back into the state. Next, Cellpose produces coarse masks that act as geometric prompts; these, plus the text prompt "cell cluster," are fed to SAM 3, which refines them into final organoid instance masks. Because all morphology is derived from those masks, the system then calls predefined functions to compute the requested statistics and plots, appending everything to the state. Finally, the ReportAgent (GPT-5.4) reads the accumulated state and writes a structured Markdown report that answers the refined query, citing only the evidence in the state and forbidden from inventing or recomputing any number. Evaluation uses the MorphoOrgaVQA benchmark, comparing predicted scalar values against ground truth computed analytically from expert masks and reporting absolute percentage error, supplemented by a qualitative head-to-head against Omega and Agentic-J.

Why This Matters

Impact on research: The framework removes two bottlenecks in organoid studies, namely manual annotation and bespoke analysis code, enabling reproducible, high-throughput morphological analysis across datasets and organoid phenotypes without retraining a single model. The released MorphoOrgaVQA benchmark and deterministic ground-truth pipeline also give the field a shared, objective way to compare agentic bioimage systems.

Real-world applications:

• Drug screening: Rapid, consistent readouts of organoid size and shape across large compound libraries, where culture state and treatment response are inferred from morphology.
• Disease modeling and personalized medicine: Quantifying patient-derived organoid morphology to track tumor development or disease progression in longitudinal experiments.
• Quality control in organoid culture: Automated triage of roundness and boundary regularity to flag irregular or stressed cultures before downstream assays.
• Lumen and differentiation tracking: Metrics such as lumen_ratio expose internal cavity formation, which relates to differentiation events in developing tissue models.

Industry relevance: Biotech and pharmaceutical companies running image-based organoid pipelines can replace labor-intensive annotation and custom scripting with a single natural-language interface, lowering the expertise barrier for experimental biologists and reducing cost per analyzed plate. The benchmark's existence also supports vendor-neutral evaluation of competing agentic analysis platforms.

Future Directions

• Improve boundary fidelity: Roughness errors of roughly 46% point to a need for sub-pixel-accurate segmentation or a more robust definition of perimeter difference that is less sensitive to mask noise.
• Reduce aggregation error: Tertile-mean area errors near 39% suggest that population-level statistics inherit segmentation error; better mask quality or error-propagation modeling could close this gap.
• Extend to new modalities and metrics: The current benchmark covers brightfield images and four metrics; generalizing to fluorescence, 3D volumetric organoids, and additional morphological descriptors remains open.
• Validate on expert biological judgment: Current evaluation is numerical against annotated masks; whether generated reports align with what biologists find useful or scientifically sound has yet to be tested at scale.

Target Audience

Experimental biologists and imaging scientists who need organoid morphometrics without writing code; computational biologists and bioimage analysts building agentic pipelines; and machine learning researchers interested in multi-agent orchestration, foundation-model-based segmentation, or benchmark design for scientific agents. Readers without a background in segmentation models or LLM agents will follow the conceptual argument but may find the hybrid-prompting mechanics and error analysis harder to assess.

Authors’ abstract

Organoids are three-dimensional tissue models whose morphology provides important insights into tumor development, disease progression, and drug testing. Extracting these morphological features relies heavily on manual segmentation, which is time-consuming and labor-intensive. Furthermore, performing quantitative statistical analysis typically requires custom coding skills and a mathematical background, presenting a major barrier for experimental biologists. To address these challenges, we introduce MorphoOrgaAgent, a multi-agent framework that achieves zero-shot organoid segmentation, automated data analysis, and report generation based on natural language input. The framework consists mainly of three core components: a TaskUnderstandingAgent that identifies requested measurements and visualization types; a hybrid segmentation module that combines Cellpose-derived geometric prompts with text prompts to guide SAM3 for zero-shot organoid instance segmentation; and a ReportAgent that computes quantitative metrics and compiles them alongside generated visualizations into a structured report. We further introduce MorphoOrgaVQA, a benchmark designed for quantitative evaluation of agent systems in organoid morphology analysis. Experimental results demonstrate that MorphoOrgaAgent handles both explicit and descriptive user requests, produces measurements closely matching ground truth, and generates complete analysis reports without requiring manual programming. The complete source code and MorphoOrgaVQA benchmark are publicly available at https://github.com/peng-lab/MorphoOrgaAgent.