NeuroRefiner: Morphology-Aware Multi-Agent Refinement for 3D Fluorescence Microscopy Neuron Segmentation

AI summary

Overview

Research area: Computer vision applied to biomedical image analysis — specifically 3D neuron segmentation in fluorescence microscopy volumes, combined with large-language-model (LLM) multi-agent systems.

Technical level: Advanced. The paper assumes familiarity with 3D U-Net architectures, cross-attention, topological metrics (Betti numbers, connected components), and LLM agent pipelines.

Scope: NeuroRefiner is a multi-agent refinement framework with a dedicated instruction-guided 3D segmentation editing tool (TopoRefineNet) that iteratively corrects topological errors in neuron segmentation masks across three fluorescence microscopy benchmarks.

What This Paper Is About

Automated neuron segmentation in 3D fluorescence microscopy is hard because neurons are sparse, elongated, and embedded in noisy volumes, so single-pass models with limited receptive fields produce fragmented results that break the neuron's global topology. Existing end-to-end methods also behave as uninterpretable black boxes with no mechanism to detect or fix topological errors after inference. The paper's goal is to imitate how a human expert works — looking globally for broken or noisy regions, then editing locally — by building a three-agent system that diagnoses errors, writes natural-language correction instructions, and validates each edit using a purpose-built 3D editing network.

Key Contributions

1. NeuroRefiner, described as the first LLM-based multi-agent system that achieves topology-aware iterative optimization of neuron segmentation results, formalizing the human expert loop of global observation and local editing.

2. Three specialized agents — the Global Inspector (uses initial masks and their Betti numbers to locate sub-regions needing correction), the Refinement Advisor (generates refinement instructions per sub-region), and the Change Validator (decides whether to accept or regenerate an instruction).

3. TopoRefineNet, a dedicated 3D U-Net-based segmentation editing tool that uses cross-modality feature fusion (cross-attention between visual features and text-encoder instruction embeddings) to turn language instructions into voxel-level mask modifications, paired with a tailored two-stage training strategy.

4. Empirical validation on three benchmarks (BigNeuron, CWMBS, ZBFWB) showing that the method outperforms existing neuron segmentation and segmentation refinement approaches, with a notable 3.02% F1 improvement on the challenging ZBFWB dataset.

Main Findings

• Better F1 than all baselines on ZBFWB: NeuroRefiner reaches an F1 of 86.13 on ZBFWB, versus 83.11 for the strongest baseline GBP-Net — the 3.02% gain highlighted in the abstract. It also raises the F1 of 3D U-Net by 13.19% on that dataset (72.94 to 86.13).

• Large gains on BigNeuron over the initialization: Starting from 3D U-Net segmentations, the method improves MES by 5.20% and reduces SSD by 4.33 relative to the initial segmentation, reaching SD 3.39, SSD 6.94, precision 89.74, recall 90.37, F1 90.05, and MES 72.02.

• Strong results on both CWMBS subsets: On the weak-signal subset with an nnUNet initialization, the method reaches SD 7.26, SSD 11.86, precision 96.79, recall 87.39, F1 91.85, MES 72.60. On the strong-noise subset it reaches SD 15.96, SSD 20.17, precision 93.76, recall 77.47, F1 84.84, MES 61.82. The paper states the approach improves F1 scores of various segmentation masks by over 10% on the weak-signal subset.

• Generic refinement methods give only marginal or modest gains: Mask Transfiner and SegFix yield only marginal gains on 3D U-Net and nnUNet masks on CWMBS, while SegRefiner outperforms them but still trails NeuroRefiner in every reported configuration.

• All three agents matter (ablation): With no agents, ZBFWB SSD/F1 is 34.52/76.39 and CWMBS is 30.23/79.96. Using only the Global Inspector gives 32.05/77.29 and 28.47/80.34. Adding the Refinement Advisor yields F1 gains of 6.18% and 6.80% respectively; adding the Change Validator yields further gains of 2.66% and 1.68%; the full system reaches 23.66/86.13 (ZBFWB) and 16.65/88.83 (CWMBS).

• Five iterations is the chosen cutoff: Across all three datasets the first iteration produces substantial gains and improvements diminish significantly after the fifth iteration, so T_max is set to 5. The harder ZBFWB and CWMBS datasets need more steps to converge.

• The framework is not tied to one foundation model: Replacing Qwen3-VL-8B with Qwen2.5-VL-7B or Intern-VL3-8B still improves over the baseline (the paper cites +4.82% F1 with Intern-VL3 on CWMBS), indicating the gain comes mainly from the multi-agent framework and TopoRefineNet.

• Global Inspector needs both global and local views plus topology cues: Using only the xy projection gives ZBFWB F1 of 77.36; adding patch-level projections raises it by 3.43% to 80.79; using only the Betti number gives 82.41; combining the global view, patch projections, and the 0th-order Betti number gives the best result of 86.13.

• Cross-attention in the bottleneck is chosen over AdaLN for efficiency: CDP and CAB give comparable accuracy (CDP ZBFWB F1 86.07, CAB 86.13), so the more efficient CAB is adopted. Replacing the 0.6B text encoder with a larger Qwen3-4B brings no significant improvement.

• The two-stage curriculum is necessary: Training on real defects from scratch (single-stage) instead of using the two-stage strategy increased the Change Validator's rejection rate by 23.57% and degraded F1 by 1.17% on ZBFWB and 1.89% on CWMBS.

Methodology in Plain English

The pipeline starts from an initial segmentation mask produced by an off-the-shelf model, then iterates up to 5 times:

1. Global Inspector compresses the 3D mask into a top-down (z-axis maximum intensity projection) view, splits it into non-overlapping 2D blocks of 128 × 128 pixels, counts connected components in each block (the 0th-order Betti number), and flags blocks that are likely to contain breaks (false negatives) or isolated noise (false positives). High connected-component counts are treated as evidence of topological trouble.

2. Refinement Advisor looks at the flagged block from two orthogonal views (xy and yz projections) and writes a structured natural-language instruction that names both a spatial location (for example "upper-left-posterior") and an operation ("connect" or "remove").

3. TopoRefineNet executes that instruction. It concatenates the image patch and the noisy mask patch, encodes them with a visual encoder, embeds the text instruction with a frozen pre-trained text encoder, fuses the deepest visual features with the text embedding via cross-attention, and decodes back to a refined mask. This makes refinement a conditional image-editing problem rather than a fresh segmentation pass.

4. Change Validator compares the original and refined block, conditioned on the instruction and error type, and accepts the edit only if it (a) implements what the instruction asked for and (b) reduces fragmentation or noise without adding new artifacts. If it fails, the Advisor is asked for a new instruction; unresolved blocks are escalated to human experts. Accepted edits are merged into the global mask, and the loop repeats until the mask is judged morphologically complete or T_max is reached.

To train TopoRefineNet, stage one uses synthetic defects: an erosion with a random kernel size k between 5 and 20 creates false-negative breaks, while copy-pasting other neurons' annotations or injecting Gaussian noise in local regions creates false positives. Stage two uses real errors: several common architectures (3D U-Net, V-Net, UNETR, nnFormer, SwinUNETR) are trained on a subset of the training data and run over the full training set, and 5,310 volumes of 128 × 128 × 64 voxels with significant segmentation errors are selected, with refinement instructions synthesized by Qwen3-VL. Both stages use cross-entropy loss and dice loss. Evaluation converts predictions into reconstructions with APP2 and reports precision, recall, F1, SD, SSD, and MES.

Why This Matters

The work targets a known bottleneck in neuroscience pipelines: segmentation that looks locally plausible but is globally disconnected is unusable for tracing neuronal morphology, and errors are hard to localize and fix. By making the correction process an explicit, auditable sequence of diagnoses and instructions, the paper argues that refinement becomes interpretable rather than a black-box second pass. It also shows that the agent framework is agnostic to the underlying vision-language model, so it can improve as VLMs improve.

Real-world applications:

• Large-scale connectomics and brain-mapping projects that need to trace long-range axonal projections through noisy fluorescence volumes.
• Morphological and quantitative neuroscience studies that depend on measuring neuron length, branching, and connectivity from reconstructed skeletons.
• Drug and disease research where neuronal degeneration or regeneration must be quantified across many specimens.
• General biomedical image refinement, since the instruction-guided editing tool could be adapted to other sparse tubular or filamentous structures.

Industry relevance: The work is relevant to groups building agent-plus-tool pipelines for scientific imaging, to vendors of microscopy analysis software who need automated quality control for sparse structures, and to anyone designing domain-specific tools that let LLM agents act on 3D volumetric data rather than only 2D slices, which the paper argues is a key weakness of existing biomedical agents that rely on tools such as MedSAM2.

Future Directions

• Depth-aware inspection: The paper's stated limitation is that 2D maximum intensity projection loses depth information in dense regions, hindering precise 3D error localization. The authors propose depth-aware encoding that maps Z-axis depth to pseudo-color channels or generates depth-weighted projections so the Global Inspector can resolve occlusions and infer the Z-depth of defects.

• Improving the resolve rate for difficult blocks: The system escalates sub-blocks that remain unresolved after multiple refinement cycles to human experts, so reducing that escalation rate is an open practical question.

• Extending beyond fluorescence microscopy: The framework's tool design is specific to sparse filamentous neurons; testing whether instruction-guided refinement transfers to other modalities and structures (the paper notes existing agent tools are mostly aimed at electron microscopy or MRI/CT) is unresolved.

• Scaling the reasoning model and instruction vocabulary: The paper reports that a larger 0.6B-to-4B text encoder swap brought no significant benefit, leaving open whether richer instruction schemas or larger VLM backbones could yield further gains on the hardest datasets.

Target Audience

Researchers and engineers working on biomedical image segmentation, connectomics and neuronal reconstruction pipelines, and agentic AI systems that combine LLMs with domain-specific tools. It will also interest practitioners in 3D microscopy image analysis who need automated, interpretable quality control, and readers studying how to design execution tools that translate natural-language instructions into voxel-level image edits.

Authors’ abstract

Accurate 3D neuron segmentation in fluorescence microscopy is critical for neuroscience. However, the sparse and elongated morphology of neurons poses significant challenges to existing segmentation methods. These methods struggle to preserve both local details and global topology, leading to fragmented results. To address this, we propose NeuroRefiner, a multi-agent system that formalizes the human expert workflow involving iterative global observation and local editing. Specifically, NeuroRefiner comprises three collaborative agents dedicated to diagnosing topological errors, generating correction instructions, and validating refinement quality. To facilitate agent instruction-guided segmentation refinement, we propose TopoRefineNet, a dedicated 3D U-Net-based tool that leverages cross-modality feature fusion to generate refined masks. Through multi-round agent reasoning and voxel-level editing, NeuroRefiner produces topologically more accurate segmentations with enhanced interpretability. Experiments on the BigNeuron, CWMBS, and ZBFWB datasets demonstrate that NeuroRefiner outperforms state-of-the-art methods, notably achieving a 3.02% improvement in F1 score on the challenging ZBFWB dataset.