Representation-driven Endoscopic Visual Embedding Alignment for Latent Generation

AI summary

Overview

Research area: Medical computer vision — generative foundation models for gastrointestinal endoscopy, combining diffusion/flow-matching transformers with self-supervised representation alignment.

Technical level: Advanced. The paper assumes familiarity with latent diffusion/flow-matching (Scalable Interpolant Transformers, VAEs, classifier-free sampling), self-supervised vision encoders (DINOv2/DINOv3, SAM2), representation alignment (REPA/iREPA), and linear-probing evaluation protocols.

Scope (one sentence): The paper introduces REVEAL, a latent generative model for endoscopy trained on roughly five million clinical frames, and shows through systematic ablations that aligning its internal features with domain-specific endoscopic encoders improves both image synthesis quality and downstream discriminative performance.

What This Paper Is About

Endoscopic image synthesis has been held back by two problems: generative models trained on natural images (such as Stable Diffusion) do not understand the endoscopic visual domain, and training large Diffusion Transformers from scratch is computationally expensive. Meanwhile, representation alignment — matching a generative model's internal features to a pretrained self-supervised encoder — has accelerated training in general computer vision, but its role in the specialised endoscopic image space was unclear.

The authors build REVEAL on top of the iREPA alignment strategy, but replace general-purpose teachers with encoders pretrained directly on the endoscopic distribution (GastroNet-5M, or GN-5M), and they benchmark which alignment design choices actually matter. The resulting model generates high-fidelity endoscopic images and, without any supervised training, also serves as a feature extractor that outperforms dedicated endoscopy foundation models such as EndoViT and Endo-FM on the reported benchmarks.

Key Contributions

1. A large-scale ablation of alignment design elements evaluating how (i) the VAE latent space (SD2, SD3, FLUX.1-dev, FLUX.2-dev, and pixel-space JiT), (ii) the choice of target teacher representation, (iii) backbone scaling (SiT-S to SiT-L), and (iv) dataset size and training duration each affect morphological fidelity (FID) and linear-probing accuracy on the BE and POLAR benchmarks.

2. A feature-quality and robustness evaluation across multiple independent endoscopic datasets, including the POLAR benchmark, a private Barrett's Esophagus neoplasia (BE) dataset, and BE-C, a corrupted variant of the BE test set built from eight clinical corruption types, used to test generalisation beyond the training distribution.

3. REVEAL itself — described as the largest foundation generative model for endoscopy to date, a high-capacity SiT backbone trained on GN-5M that doubles as a competitive feature extractor, exceeding EndoViT and Endo-FM on the reported benchmarks, with code and weights released publicly.

4. Demonstration of latent-space editing capabilities — inpainting and outpainting performed via a RePaint-style resampling strategy with no architectural changes or task-specific finetuning, used as spatial robustness probes of the learned gastrointestinal manifold.

Main Findings

• Domain-adapted alignment teachers win, but not on every metric. Adding representation alignment consistently improved generative performance over unguided baselines. DINOv3-B (GN-5M) achieved the best FID (10.18) and best POLAR accuracy (0.682) among SiT-B/2 configurations, while DINOv2-B achieved a marginally higher BE accuracy (0.872) than DINOv2-B (GN-5M) at 0.860 — an advantage the authors attribute to its finer patch size and higher operating resolution, which did not transfer to generation fidelity.

• SD2 is the best latent space in the ablation. With a SiT-B/2 backbone at 150k iterations, SD2 gave FID 13.55, compared with SD3 at 14.22, FLUX.2-dev at 14.46, FLUX.1-dev at 28.88, and pixel-space JiT-B/16 at 28.10. The authors attribute this to SD2's compact 4-channel bottleneck being easier for the SiT backbone to modulate. JiT was excluded from further evaluation because of its substantially inferior generation quality and the additional implementation effort required for feature extraction in pixel space.

• Scaling the backbone mainly buys image fidelity. Going from SiT-S/2 (FID 13.07) to SiT-L/2 (FID 9.33) substantially improved fidelity, while representation quality rose only modestly (BE 0.863 to 0.881; POLAR 0.682 to 0.684), which the authors link to extracting features early at the 8th layer.

• Scaling the data was the single largest gain. Moving from the ~250,000-image subset to the full GN-5M set at an equivalent 150k iterations dropped FID from 9.33 to 5.43, raised BE accuracy from 0.881 to 0.905 and POLAR from 0.684 to 0.763. Continued training gave diminishing returns: 200k iterations reached FID 5.37, BE 0.906, POLAR 0.768, and 250k iterations reached FID 5.32, BE 0.904, POLAR 0.773.

• REVEAL beats dedicated endoscopy foundation models on clean and corrupted data. On the BE test set, REVEAL (SiT-L/2) reached AUC 0.786 / AUPRC 0.728 versus EndoViT at 0.629 / 0.475 and Endo-FM at 0.764 / 0.689. On POLAR it reached 0.758 / 0.935 versus EndoViT at 0.632 / 0.892 and Endo-FM at 0.683 / 0.908. On the corrupted BE-C set it reached 0.754 / 0.679 versus EndoViT at 0.524 / 0.400 and Endo-FM at 0.692 / 0.588.

• EndoViT underperformed general-purpose encoders on clean data. EndoViT scored below DINOv2 and DINOv3 on both the BE and POLAR benchmarks, which the authors read as evidence that domain specificity alone is insufficient without adequate pretraining scale and objective expressiveness. Under corruption, EndoViT degraded the most sharply of the models tested (BE-C AUC 0.524).

• The strongest discriminative encoders were still the domain-pretrained DINO variants. DINOv3 (GN-5M) achieved the highest overall clean-image scores (BE 0.835 / 0.790; POLAR 0.808 / 0.947) and the highest corrupted-image scores (BE-C 0.814 / 0.771), with DINOv2 (GN-5M) closely behind (BE-C 0.810 / 0.774). REVEAL's own figures are below these two on every reported benchmark, while still exceeding the general-purpose encoders SAM2, DINOv2, and DINOv3 and the endoscopy-specific EndoViT and Endo-FM.

• Robustness is achieved despite two stated handicaps. The authors note that REVEAL operates on features from a frozen, general-purpose SD2 VAE encoder that was not designed for robustness to low-level artifacts, and that REVEAL was evaluated only at the standard noise level t=0. They suggest intermediate diffusion timesteps, which naturally denoise corrupted inputs, could yield further gains without retraining.

• Qualitative synthesis and editing results. Unconditional samples (Heun solver, 50 NFEs) showed plausible mucosal texture and specular highlights across anatomical regions, lesion morphologies, and imaging conditions. Inpainting reconstructed masked regions and outpainting extended the visible field of view while preserving structural continuity. These were assessed qualitatively only; no numerical inpainting or outpainting metrics are reported. The authors also state that the resampling-based strategy can occasionally produce harmonisation artifacts at mask boundaries.

Methodology in Plain English

The researchers trained a latent generative model on endoscopy images rather than natural images. An image is first compressed into a small latent code by a pretrained VAE encoder (Stable Diffusion 2's was chosen after testing alternatives), and a Scalable Interpolant Transformer learns to reverse a noise-adding process to reconstruct it.

The distinctive ingredient is a second training signal: while the transformer denoises, its internal patch-level features are forced to look like the features produced by a frozen vision encoder that was itself pretrained on endoscopic data. Similarity is measured by cosine similarity, and the alignment term is added to the denoising loss with a weight of 1, following iREPA.

Two refinements from iREPA are carried over: a small convolutional projection layer (kernel size 3) replaces the usual point-wise MLP so that neighbouring patches keep their spatial relationships, and the teacher features are spatially normalised to boost the contrast of local anatomical structure at the expense of global information.

The team trained one of the teacher encoders themselves — a DINOv3 ViT-B/16 adapted to endoscopy through a two-stage curriculum totalling 230,000 iterations — and compared it against SAM2, DINOv2, DINOv3, and DINOv2 pretrained on GN-5M. All generation ran at 256×256 resolution on four NVIDIA H100 GPUs.

To check whether the learned features are actually useful, they froze the model and trained simple linear classifiers on features taken from the 8th transformer layer at t=0, evaluating on the POLAR polyp benchmark, a private Barrett's Esophagus neoplasia dataset, and a synthetically corrupted version of that test set. Five-fold cross-validation means and AUC/AUPRC are reported.

Why This Matters

Impact on research. The paper argues that large-scale generative pretraining is a viable and complementary alternative to masked image modelling for building clinical visual representations — the authors state that all prior endoscopy foundation models are purely discriminative and that no earlier work explored generative pretraining for this purpose. It also shows that in a narrow, texture-heavy domain, the choice of alignment teacher matters more than alignment in general, and that in-domain encoders beat out-of-domain ones.

Real-world applications:

• Data augmentation for rare pathologies. High-fidelity unconditional generation can address limited data diversity, bias, and severe class imbalance for rare lesions, with future conditional finetuning targeting specific pathology generation.
• Privacy-preserving data sharing. Synthetic endoscopic images could reduce the privacy constraints that currently restrict sharing of clinical recordings between centres.
• Image editing and field-of-view extension. The demonstrated inpainting and outpainting show the model can reconstruct missing anatomical regions and extend the visible area while preserving structural continuity.
• Quality control and outlier detection. Likelihood estimation over the learned distribution supports dataset coverage analysis and out-of-distribution detection, with the authors proposing segmentation as a further downstream use of the generative features.

Industry relevance. The paper frames REVEAL as lowering the computational threshold for building specialised clinical tools: a publicly released backbone and weights (hosted at caetas.github.io/reveal.html) give groups without access to five-million-image multicenter datasets — or four H100 GPUs — a strong starting point for conditional synthesis, segmentation, and diagnostic classifiers, which is directly relevant to endoscopy device manufacturers, clinical AI developers, and gastroenterology screening programmes.

Future Directions

• Scaling along resolution and capacity. The authors note the SD2 VAE natively supports higher-resolution inputs, so finer spatial detail is a straightforward extension, and larger transformer backbones are promising given the scaling behaviour seen in Table 1. They suggest the alternative VAE architectures may become more suitable at greater model capacity, with domain finetuning of any of them potentially adding gains.

• Conditional finetuning on labelled subsets. The paper proposes finetuning for targeted pathology generation, rare lesion augmentation, segmentation via generative features or Symmetrical Flow Matching, and coupling the backbone with text encoders or semantic masks to address the long-tail distribution of clinical findings.

• Improved robustness without retraining. Since REVEAL was evaluated only at noise level t=0, the authors identify intermediate diffusion timesteps as a promising route to denoise corrupted inputs. They also propose replacing the resampling-based inpainting/outpainting strategy with gradient-guided or Langevin-corrected sampling to reduce boundary artifacts.

• Validation and benchmarking beyond the current datasets. The stated near-term roadmap includes robustness validation across devices and centres, plus segmentation benchmarks, alongside out-of-distribution detection and dataset coverage analysis through likelihood estimation.

Target Audience

Researchers and engineers working on medical image synthesis, generative foundation models, and self-supervised representation learning will get the most from this paper, particularly those interested in how alignment-teacher choice interacts with a narrow clinical domain. It is also relevant to endoscopy and gastroenterology AI groups seeking a pretrained backbone for segmentation, conditional synthesis, or out-of-distribution detection, and to practitioners deciding whether generative pretraining is worth pursuing relative to masked image modelling. Readers without a background in diffusion or flow-matching transformers and linear-probing evaluation will find the methodology sections demanding.

Authors’ abstract

Developing foundation generative models for endoscopy is limited by the gap between natural and clinical images and the computational cost of training large Diffusion Transformers. Although representation alignment has improved efficiency in general computer vision, its role within the highly specialized endoscopic image space remains unclear. We introduce REVEAL (Representation-driven Endoscopic Visual Embedding Alignment), the largest generative foundation model for endoscopy to date, trained on GastroNet-5M (GN-5M), a multicenter dataset of 5 million endoscopic frames. Instead of depending on out-of-domain priors, REVEAL employs encoders pretrained directly on the endoscopic distribution to align diffusion latents with domain-specific visual features, preserving fine textures and intricate anatomical structures. Beyond image generation, REVEAL also serves as a powerful feature extractor; in multiple benchmarks, it delivers performance that is competitive with, and in several cases exceeds, endoscopic foundation models such as EndoViT and Endo-FM, specifically tuned for classification tasks, while demonstrating strong representation robustness under realistic imaging corruptions. REVEAL produces high-fidelity images and maintains robust structural coherence in latent-space edits such as inpainting and outpainting. This high-capacity backbone lowers the computational threshold for building specialized clinical tools, offering an open, versatile foundation for conditional synthesis, segmentation, and out-of-distribution detection in future intelligent gastroenterology systems.