Co-Annotator: Expert-Distilled ViT and VLM for Visual and Documentation Guidance in Age-Related Macular Degeneration

AI summary

Overview

• Research area: Human–computer interaction (HCI) for clinical AI, combining vision transformers, vision–language models, and eye tracking, applied to ophthalmology (optical coherence tomography imaging of wet age-related macular degeneration).
• Technical level: Intermediate. The interface and study design are described accessibly, but the modeling sections assume familiarity with attention rollout, micro-AUC, BERTScore-style text metrics, and multi-label biomarker ontologies.
• Scope: A system called Co-Annotator that distills expert gaze and expert dictation into two separate clinician-facing guidance components (a gaze-aligned ViT and an ontology-bounded VLM), evaluated across three user studies including a two-institution combined deployment. (The conference detail: UIST '26, the 39th Annual ACM Symposium on User Interface Software and Technology, November 02–05, 2026, Detroit, MI, USA.)

What This Paper Is About

Clinical AI usually optimizes predictive accuracy without accounting for how clinicians decide where to look and what to write. Diagnosing wet age-related macular degeneration (wAMD) on OCT requires spotting subtle, fine-grained biomarkers distributed across five image slices per eye, then documenting them in standardized language — two separate, largely independent costs that consume clinician time. The paper's goal is to build and test multimodal guidance that removes both bottlenecks at once without degrading the diagnostic accuracy clinicians already achieve.

Key Contributions

1. Interaction design and design principles (the authors' stated primary contribution). An interaction model for how clinicians and AI read scans together, distilled into three principles residents consistently demanded across all three studies: guidance should be deferrable (revealed after a first independent pass), sparse (2–3 precise hotspots, not diffuse coverage), and evidence-anchored (each biomarker token clickable to its AOI tile).
2. The Co-Annotator system. An OCT co-annotation interface with two clinician-facing components: fixation-aligned AOI heatmaps rendered from a gaze-aligned ViT, and editable biomarker drafts pre-filled by an ontology-bounded VLM. The overlay is toggleable by right-clicking the image, and the draft is constrained to terms from an expert-derived ontology.
3. Expert distillation (US1). Collection of synchronized expert gaze and dictation to train both models, with the ViT improving diagnostic micro-AUC from 0.95 to 0.98 in the patient-level regime and the VLM reaching MedBERTScore 0.867 on biomarker text.
4. Modality evaluation (US2). Deployment of each guidance channel in isolation with ophthalmology residents, showing that each preserves accuracy and produces complementary benefits — an AOI post-guidance efficiency carryover and VLM guidance more than doubling biomarker breadth (5.8 vs. 2.3 per AMD eye, 83.1% retention).
5. Combined deployment (US3). Simultaneous AOI + VLM guidance across two academic institutions, producing Correct Dx/min gains of 40% and a 67% reduction in comment editing time.

Main Findings

• Expert gaze can be distilled into an attention prior. Training the ViT with a gaze-alignment term (weighted by α) alongside a classification loss moved its last-layer attention rollout toward expert fixation density. In the patient-level (five-image) regime, adding gaze alignment raised micro-AUC from 0.95 to 0.98 under five-fold cross-validation. In the single-image regime, the best trade-off occurred at α = 0.05, yielding 88.37% validation accuracy and F1 = 0.8628, outperforming the unaligned baseline ViT.
• The fine-tuned VLM was strong on all three tasks. After two-stage curriculum training on MedGemma, diagnosis accuracy was 0.920 on the OCT-C8 held-out test set and 0.910 on a larger US1 test set; biomarker discrimination accuracy was 0.800; generative biomarker identification reached BERTScore F1 = 0.880 and MedBERTScore F1 = 0.867. Ablation showed co-training with OCT5k was critical — training only on the smaller transcript dataset led to marked overfitting.
• Data scale of the distillation corpus. 1,155 high-definition five-line raster OCT scans of the macula from 231 eyes in 203 patients (five scans per eye; 104 normal, 127 wAMD), captured on a Zeiss Cirrus platform. Eight experts provided visual attention on a subset of 138 images and dictation on 113. Adding OCT5k samples produced 573 image–biomarker pairs; biomarkers with fewer than 10 occurrences were excluded, yielding 12 ontology terms.
• AOI guidance alone produced no in-guidance speed gain, but did produce carryover. Diagnostic accuracy was preserved across all three blocks (0.93/0.89/0.90). Correct Dx/min and time per eye were statistically unchanged during the AOI block. In the post-guidance block, time per eye fell 28% versus pre-control (p < 0.001) and Correct Dx/min rose 52% (from 2.9 to 4.4) but did not reach significance (p = 0.065 raw, 0.274 adjusted). The authors attribute the in-guidance null to residents adopting a read-first-then-toggle strategy.
• VLM guidance alone broadened documentation without slowing readers down. Diagnostic accuracy was maintained (0.917 vs. 0.833 in control, nonsignificant). Residents recorded a mean of 5.8 biomarkers per AMD eye versus 2.3 in control (p = 8.8 × 10⁻⁹), retaining 83.1% of VLM suggestions, deleting 16.8%, and independently contributing 27 new biomarkers. Vocabulary overlap (Jaccard) rose from 0.45 to 0.78. Out-of-ontology generation was rare: 3 biomarkers across 11 appearances, with the high-risk term "hemorrhage" deleted 71% of the time.
• Documentation burden was substantial at baseline. Residents spent 17–22% of per-eye time editing comments — a bottleneck AOI guidance alone does not address.
• Combined guidance produced gains neither modality achieved alone. Correct Dx/min rose from 3.40 to 4.76 (+40%, 95% CI [+15%, +74%], p = 0.023) and comment edit time fell from 7.2 s to 2.4 s (−67%, 95% CI [−87%, −43%], p = 0.023). Accuracy was not compromised (83.3% pre-control vs. 93.3% under guidance; W = 2, n = 8, p = 0.094), and the false positive rate remained 0% throughout. Perceived effort decreased significantly (2.11 to 1.91, p = 0.047). Time per eye trended lower (26.0 to 22.1 s, −15%) but the reported text is truncated at that point.
• Biomarker breadth also exceeded either modality alone. Combined guidance yielded 5.36 biomarkers per AMD eye at 85.5% retention; the paper states these outcomes exceed either modality alone (4.76 vs. 3.0 AOI-only and 2.0 VLM-only).
• Gaze behavior diverged in opposite directions depending on condition. Kullback–Leibler divergence between resident gaze and model AOI increased significantly during and after AOI-only guidance (p = 0.039 during, p = 0.008 post) but showed no significant change across blocks under combined guidance (all p > 0.6).
• Neither modality improved efficiency during its own guidance block in US2. The authors highlight this explicitly: the +40% in-guidance Correct Dx/min gain in US3 is a gain neither modality produced on its own during active guidance.

Methodology in Plain English

The researchers treated the OCT reading workflow as two independent costs — finding the finding, and writing the finding — and built one component for each rather than a single general-purpose assistant.

First, they recorded retina specialists while they read OCT scans: eye movements were converted into a per-image fixation-density map on a 32×32 patch grid, and concurrent spoken dictation was transcribed, cleaned by the experts, and converted into multi-label biomarker targets using an ontology derived from OCT5k, with extraction outputs reviewed by a retina specialist.

The gaze maps became the supervision target for a ViT whose training loss adds a gaze-alignment term to the ordinary classification loss, so the network is pulled toward the regions experts actually fixated while still learning to diagnose. Because this objective learns a feature-to-attention mapping, the model can produce fixation-aligned regions of interest for unseen images without new eye-tracking data.

Expert dictations became the training signal for a vision–language model built on MedGemma. Training used a two-stage curriculum: stage one covered binary diagnosis and yes/no biomarker discrimination (visual question answering) to build OCT-domain grounding, and stage two added generative biomarker listing while continuing to train the earlier tasks. A structured system prompt constrains output format and restricts vocabulary to the ontology.

In the interface, the ViT drives a semi-transparent colored heatmap overlaid on the currently active B-scan (chosen over contours because the grayscale scan stays legible beneath it), toggleable by right-clicking. The VLM pre-fills the free-text biomarker comment field for the majority diagnosis across the eye's five slices. The visual parameters were inherited from the authors' prior glaucoma diagnosis interface and set through informal clinician design discussions rather than tuned for this study.

Three user studies followed. US1 collected the gaze and dictation data and evaluated the trained models. US2 tested each modality in isolation with residents, using an unguided-control → guidance → post-guidance-control block sequence for the AOI condition and a two-block sequence for the VLM condition (to keep sessions under 40 minutes). US3 deployed both modalities simultaneously with the same three-block design across two academic medical institutions. The interface logged all interactions with millisecond timestamps, and participants could not return to previously rated cases.

Why This Matters

Impact on research. The paper argues that gaze alignment and ontology grounding are typically optimized in isolation, and that unifying them inside a realistic reading-and-documentation loop changes what is measured. Its most distinctive claim is methodological and counterintuitive: the combined condition improved efficiency during guidance even though neither modality did so alone, which the authors present as evidence that treating visual search and documentation as separate bottlenecks — each handled by a separate component — exceeds what one general-purpose assistant would achieve. The paper also reframes the modeling work as an enabler of the interaction design rather than the contribution itself, and reports residents' requests for sparse overlays and clickable evidence links as findings rather than as validated design decisions.

Real-world applications:

• Retina specialty clinics facing high imaging volume, where per-eye time competes directly with reading throughput.
• Ophthalmology trainee education, where documentation consistency and pattern recognition are still being formed.
• Reducing documentation burden in structured reporting workflows, where an editable, ontology-constrained draft replaces transcription from scratch.
• Transparent, contestable clinical decision support, where the model exposes where it looked and the clinician retains editorial control.

Industry relevance. The Expert Distillation Corpus, the fine-tuned VLM, and the gaze-aligned ViT are publicly released on Hugging Face (linked in the paper's Data and Model Availability statement), lowering the barrier for teams building gaze-supervised or ontology-bounded clinical interfaces. The design principles are stated generically enough to transfer beyond ophthalmology, and the results speak to anyone deploying AI into an existing clinical documentation pipeline rather than a standalone prediction service.

Future Directions

• Implement and test the sparse overlay and evidence anchoring. The authors write that they expect to implement a sparse overlay that surfaces only a few hotspots and to link individual biomarker tokens to the image regions that produced them, and that residents requested both repeatedly. These are reported as resident requests, not as validated design decisions.
• Disentangle guidance effects from task familiarity. The paper notes that the post-guidance AOI carryover in US2 cannot be fully separated from task familiarity given the fixed block order, and that ordering effects were confirmed nonsignificant for accuracy via a practice-slope decomposition.
• Address the participant overlap confound across studies. Three of the eight US3 completers had previously taken part in US2 (with a six-month washout), which the authors flag as a potential confound for cross-study comparisons despite the washout period and within-subject block design.
• Size the VLM timing estimates properly. The US2 VLM sub-study (n = 4 participants, 120 image-set observations) was powered for the biomarker outcome (n = 128 image-set target), not for timing, so participant-level timing estimates for that condition are described as exploratory.

Target Audience

This paper is most valuable to HCI researchers working on human–AI collaboration and clinical decision support, to clinician-scientists and medical AI developers building OCT or other ophthalmic imaging tools, and to interface designers embedding AI into high-volume documentation-heavy workflows. Radiology and pathology researchers studying gaze-supervised models or ontology-constrained generation will find the method transferable. It is also useful for machine learning practitioners who want a concrete example of designing evaluation around workflow outcomes — efficiency, documentation breadth, retention of AI suggestions — rather than predictive accuracy alone.

Authors’ abstract

Clinical AI often optimizes predictive performance without engaging how clinicians decide where to look and what to write. We present Co-Annotator, which distills expert gaze and dictation into two guidance components: a gaze-aligned Vision Transformer producing fixation-aligned areas of interest (AOIs), and an ontology-bounded vision-language model (VLM) that pre-fills editable biomarker summaries for retinal optical coherence tomography (OCT). We first collect expert gaze and dictations (US1) to train the models, significantly improving diagnostic accuracy and biomarker generation. We then deploy the system with ophthalmology residents: a controlled resident study (US2) confirmed each modality is safe and independently beneficial, with AOI guidance producing lasting perceptual efficiency gains through post-guidance carryover and VLM guidance more than doubling biomarker documentation breadth. In a combined deployment across two academic institutions (US3), providing both modalities simultaneously produced efficiency gains that substantially exceeded either modality alone: correct diagnoses per minute increased by 40% and comment editing time fell by 67%, without compromising diagnostic accuracy. Notably, neither modality improved efficiency during guidance in US2, which makes the in-guidance efficiency gain under combined guidance in US3 the more striking result. Expert-distilled multimodal guidance can remove two distinct clinical workflow bottlenecks at once (visual search overhead and documentation burden) without compromising the diagnostic accuracy clinicians already achieve.