Multimodal Taxonomic Conditioning for Generative Plankton Imagery

AI summary

Overview

• Research area: Computer vision — generative models, fine-grained long-tailed recognition, and multimodal (vision-language) representation learning, applied to automated plankton imaging.
• Technical level: Advanced. The paper assumes familiarity with contrastive learning (CLIP, InfoNCE/RINCE), diffusion transformers (DiT), LoRA adapters, classifier-free guidance, and hierarchical/taxonomic structure in labels.
• Scope: The paper adapts a CLIP text encoder to a large plankton corpus using a ranked contrastive objective extended to deep, ragged taxonomic hierarchies, freezes it, and uses the resulting embeddings to condition a parameter-efficient diffusion transformer for synthetic plankton image generation — then evaluates both image quality (FID) and usefulness as training data for a downstream classifier (macro-F1).

What This Paper Is About

Automated plankton imaging instruments photograph far more organisms than experts can label, producing severely long-tailed datasets where ecologically important rare species may have fewer than ten images. This makes standard classifiers unreliable on exactly the taxa that matter most. The authors' goal is to generate realistic synthetic plankton images conditioned on where a species sits in the biological tree of life, so that knowledge about abundant, well-photographed relatives can be transferred to rare ones.

Key Contributions

1. Ranked contrastive learning for deep, ragged taxonomies. An adaptation of RINCE to biological hierarchies where specimens are often identified only to genus or family, introducing truncation-aware depth matching (a pair's rank is the shared depth of their lineages, counting only populated ranks) and per-rank weighting by batch coverage.
2. A taxonomy-conditioned diffusion model that decouples representation learning from generation. The hierarchical structure is learned by the CLIP encoder on a large external corpus and encoded in the embedding geometry itself, rather than being injected via the training procedure or the guidance rule as in prior work (FineDiffusion, TaxaDiffusion).
3. A frozen multimodal embedding as a drop-in replacement for a learned class table. The pretrained DiT-XL/2's per-class embedding table is replaced by frozen CLIP text/image embeddings projected through a two-layer MLP — the only new trainable component (2.5M of 676M parameters).
4. A twofold evaluation on distributional fidelity (FID) and downstream classifier utility (macro-F1 on a shared real test split), under both a replacement (fully synthetic) and an augmentation (rare-class top-up) regime.

Main Findings

• Taxonomic conditioning improves generation fidelity. FID against the real training split: 19.17 for the proposed method, versus 22.43 for FineDiffusion and 43.62 for TaxaDiffusion. As a calibration point, FID between two disjoint 3,000-image samples of real data is 10.98.
• Synthetic data substitutes better for real data under the proposed conditioning. In the replacement regime, macro-F1 over 144 classes rises from 0.603 (FineDiffusion) to 0.664 (this work), against 0.513 for TaxaDiffusion and 0.869 for training on real data only. The three generators rank identically by FID and by replacement utility.
• Rare classes benefit most in the replacement regime. Rare-class F1 (<100 training images, n=52 classes): 0.486 for this work versus 0.421 (FineDiffusion), 0.305 (TaxaDiffusion), and 0.786 (real-only). Common-class F1 follows the same order (0.765 / 0.706 / 0.630 / 0.916).
• The augmentation regime does not discriminate between generators. When generated images only top up rare classes to 100 samples, this method (0.867), FineDiffusion (0.876), and TaxaDiffusion (0.843) are all statistically indistinguishable from naively duplicating real images (0.875), which is itself indistinguishable from no oversampling at all (0.869).
• Failure mode on the very smallest classes. On the four classes with two or fewer training images, the method loses to duplication by 0.113 macro-F1. A worked example is Carchesium, a sessile colonial ciliate whose nearest taxonomic neighbours are free-swimming solitary ciliates: its images agree at cosine similarity 0.94 in CLIP image space, while its nearest neighbour by text embedding sits at 0.58 in appearance. A frozen semantic embedding has nothing useful to transfer from in such cases, whereas a free per-class table can memorise the few available images.
• A capability a learned class table cannot provide. Conditioning on a specific specimen's CLIP image embedding rather than a class prototype generates variations of that particular individual, enabling instance-level rather than class-level synthesis.
• Transfer across collections works. The text encoder is adapted on Planktonzilla-17M (3.74M images, 602 taxa, 13 imaging systems), which is disjoint from the generation target WCO L4 IFCB library (74,181 images, 145 classes), so conditioning embeddings are transferred rather than fitted.

Methodology in Plain English

The work proceeds in two clearly separated stages.

Stage one — teach the language model what the tree of life looks like. The authors take OpenCLIP ViT-B/16 and adapt it with LoRA adapters on a large plankton corpus, using each specimen's cumulative taxonomic lineage (species, genus, family, and so on up the ranks) as its text description. Standard contrastive learning treats every non-matching sample as an equally wrong answer, which throws away the fact that two Chaetoceros species are more alike than a Chaetoceros and a ciliate. Ranked contrastive learning instead grades the negatives by how taxonomically distant they are.

Two practical problems arise. First, plankton labels are ragged: many specimens are identified only to genus or family, so not every sample has a label at every rank. The authors handle this by defining the relationship between two specimens as the depth at which their lineage strings stop matching, treating identical truncation as a full match and ignoring unpopulated ranks. Second, some ranks cover much more of a batch than others, so each rank's contribution to the loss is weighted by how much of the batch it actually constrains. Finer ranks are optimised at sharper temperatures than coarser ones.

Stage two — use those embeddings as the generation condition. A pretrained DiT-XL/2 diffusion transformer is frozen except for its conditioning embedder, biases, and normalisation layers. Its learned per-class embedding table is deleted and replaced by the frozen CLIP encoder's output for a class, projected to the right width by a small two-layer MLP. During training, the lineage text embedding is concatenated with the image embedding of the actual training image, and half the time the image embedding is swapped for its class mean — so at inference the model can be conditioned either on a class prototype (for quantitative experiments) or on a specific specimen (for the qualitative figure). Hierarchical classifier-free guidance is retained: with probability 0.1 the species-level text embedding is replaced by the phylum-level lineage prefix.

Evaluation. Generated images are cropped to the detected organism with Grounding DINO before scoring. Fidelity is measured by FID. Utility is measured by training a frozen DINOv3 ViT-S/16 backbone with a small linear head for 20 epochs across three seeds and reporting macro-averaged F1 on a real test split shared identically across every experiment. Baselines are FineDiffusion, TaxaDiffusion, and naive duplication of real images.

Why This Matters

Impact on research. The paper reframes how hierarchy should enter a generative model: not as a training schedule or a guidance trick, but as the geometry of the conditioning space itself, learned separately on a much larger corpus than the generation target. That decoupling is reusable. It also documents a genuine negative result — augmentation with synthetic images does not beat simply duplicating rare real images at this scale — and a genuine failure mode (taxonomic proximity is not visual proximity) that future work on hierarchical conditioning must confront.

Real-world applications:

• Marine ecology and biodiversity monitoring. Automated instruments such as the Imaging FlowCytobot already generate imagery faster than experts can annotate it; better rare-taxon classifiers would let long-term observatories track species that currently slip through as "unidentified."
• Harmful algal bloom and water-quality early warning. Bloom-forming and toxic taxa are often the rare ones in a sample; improving detection where training data is scarce directly supports timely warnings for aquaculture and public health.
• Climate and carbon-cycle science. Plankton drive the biological carbon pump. Reliable automated census of rare and morphologically subtle taxa improves the accuracy of the abundance estimates that feed into carbon-flux models.
• Ecological change and invasive-species detection. Newly arriving or range-shifting species are by definition rare in existing datasets, which is exactly the regime this method targets.

Industry relevance. The technique applies to any domain with a documented taxonomy and a long-tailed image distribution — entomology, fisheries, botanical surveys, and medical imaging where labels carry an ontological hierarchy (disease subtypes under broader categories). It is also relevant to the oceanographic instrumentation sector, environmental consulting and monitoring firms, and developers of domain-specific vision foundation models, since the conditioning encoder is frozen and can be swapped in cheaply.

Future Directions

1. Closing the gap between taxonomic and visual similarity. The Carchesium case shows that a purely semantic text embedding has nothing to transfer from when a species' nearest relatives look nothing like it. Hybrid conditioning that blends the frozen embedding with a small, learnable per-class residual could capture both.
2. Making synthetic augmentation actually beat duplication. The augmentation regime failed to separate any method from naive duplication, and actually performed worse than duplication on the four smallest classes. Understanding why — image quality, diversity, or simply the low information ceiling at n≤2 — is an open question.
3. Extending beyond single-label taxonomy to richer conditioning. The paper notes that modelling taxonomic relationships across modalities as conditioning signals merits further exploration; combining morphology, habitat, or sampling metadata with lineage is a natural extension.
4. Generalising ragged-taxonomy ranked contrastive learning beyond plankton. The truncation-aware depth matching and per-rank batch weighting were designed for unevenly annotated biological hierarchies but should transfer to any domain where labels are inconsistently resolved across a deep ontology.

Target Audience

Researchers working on generative augmentation for long-tailed and fine-grained recognition; computer vision practitioners interested in hierarchical and multimodal conditioning; marine ecologists and biological oceanographers who rely on automated imaging instruments; and machine learning engineers building foundation models for scientific or biodiversity data. The paper is most valuable to readers already comfortable with contrastive learning and diffusion models who want a concrete example of encoding taxonomic structure into a conditioning signal rather than into a training schedule.

Authors’ abstract

Automated plankton imaging produces severely long-tailed datasets, where the rare taxa of greatest ecological interest have too few images to train or evaluate classifiers reliably. We generate synthetic plankton imagery conditioned on taxonomy: a CLIP encoder is adapted on a large plankton corpus with a ranked contrastive objective extended to deep, ragged taxonomies, then frozen to condition a parameter-efficient diffusion transformer. We evaluate synthetic sample quality on distributional fidelity and downstream classifier utility.