ROFI: A Deep Learning-Based Ophthalmic Sign-Preserving and Reversible Patient Face Anonymizer

AI summary

Overview

Research area: Medical image privacy protection / computer vision for ophthalmology, combining weakly-supervised deep learning, Transformer-based image translation, and clinical validation studies.

Technical level: Advanced. The paper assumes familiarity with Transformers, weakly-supervised learning, face recognition embeddings, Cohen's kappa, differential privacy, and membership inference attacks.

Scope: The paper develops and clinically validates ROFI, a deep learning framework that anonymizes patient face photographs while preserving the ophthalmic disease signs clinicians and AI models need, and that can reversibly reconstruct the original image with a private key.

What This Paper Is About

Patient face photos are central to diagnosing externally visible eye diseases such as strabismus, ptosis, and thyroid eye disease (TED), but they also carry strong biometric identity information that risks privacy breaches. Existing protections — cropping the eye, blurring, mosaics, face swapping, AI-generated faces, and face de-identification — either fail to defeat modern face recognition or destroy the subtle clinical signs needed for diagnosis, and none are reversible. ROFI aims to anonymize identity while keeping disease signs intact and allowing authorized recovery of the original image.

Key Contributions

1. A learnable ophthalmic sign detector. Using weakly-supervised learning, ROFI mines disease-related regions from patient faces annotated only with binary health labels (healthy or not), avoiding the expensive region-mask annotation that would otherwise be required from physicians.
2. Reversible neural identity translation. A pair of Transformer-based Neural Identity Protector and Neural Identity Restorer models transform facial identity under the control of a customized privacy key, restoring the original image when authorized. A Transformer-based feature enhancement network called DA-Former refines features to remove substitution artifacts.
3. Multi-center clinical validation. Evaluation across three Chinese clinical centers spanning eleven ophthalmic diseases with external manifestations, comparing ROFI against five representative protection methods (Mosaic, AIGC, Face Swap/SimSwap, Digital Mask, G2Face).
4. Demonstrated compatibility with medical AI, privacy attacks, and longitudinal care. ROFI-protected images were tested with ResNet50 and ViT diagnostic models, against AdaCos/ArcFace face recognition, OpenIris iris recognition, differential privacy baselines, a membership inference attack, and a human recognition study, plus use of reversed images to retrieve historical records for assessing TED hormone therapy.

Main Findings

• Sign preservation: ROFI preserved disease features with over 98% accuracy and κ > 0.90 (abstract). On the SNPH set it had the lowest keypoint error, averaging 1.53%/1.64% for eyelid/iris keypoints, versus Mosaic 5.51%/12.36%, AIGC 3.19%/3.79%, Face Swap 2.35%/4.16%, Digital Mask 8.25%/6.84%, and G2Face 2.70%/3.18%. External sets gave 1.38%/1.37% on ECXHCSU and 1.68%/2.07% on WUPH.
• Disorder signs: ROFI showed κ ≥ 0.81 consistency with original images across all three validation sets. For Conj-Dis on SNPH, κ was 0.9223 (95% CI 0.8691-0.9755), versus Mosaic 0.3299, AIGC 0.2399, Face Swap 0.5208, Digital Mask 0.0955, and G2Face 0.4489. ROFI significantly outperformed G2Face (P < 0.001 for three of four signs on SNPH and all four signs on ECXHCSU and WUPH).
• Eye structure similarity: On ECXHCSU, ROFI reached 94.09% similarity, versus Mosaic 55.55%, AIGC 53.74%, Face Swap 74.15%, Digital Mask 41.37%, and G2Face 68.38%.
• Diagnostic sensitivity: ROFI achieved 100% sensitivity (95% CI 100%-100%) on SNPH, ECXHCSU, and WUPH for the health/unhealthy assessment. On WUPH, this beat Mosaic 85.00%, AIGC 77.50%, Face Swap 97.50%, Digital Mask 85.00%, and G2Face 85.00%.
• Disease-level agreement: ROFI reached κ ≥ 0.81 across all three validation sets and all eye diseases. On SNPH it scored 0.9594 for BCC, 0.9046 for CM, 0.8732 for CL, and 1.0 for SCC. On ECXHCSU it reached 0.9692 for BCC and 1.0 for CM, where Digital Mask scored only 0.0712 and 0.0993.
• Balanced disease performance: Digital Mask did well on TED and ptosis (hand-crafted eyelid features) but failed on BCC and CM; Face Swap and G2Face never reached κ > 0.81 for any disease.
• Matthews Correlation Coefficient: On WUPH, ROFI scored 0.9450 (95% CI 0.8684-1.0000), versus G2Face 0.5157, Digital Mask 0.4960, Face Swap 0.6501, AIGC 0.1513, and Mosaic 0.1604.
• Landmark and lesion accuracy: On CelebA-HQ with MTCNN landmark detection, ROFI's error was 2.9871, versus Mosaic 5.6799, AIGC 7.8945, Face Swap 4.8921, Digital Mask 6.3891, and G2Face 3.7130. For physician-annotated BCC tumor regions, ROFI's Dice coefficient was 0.7389, versus G2Face 0.5782 and Face Swap 0.2783.
• AI model compatibility: With ResNet50 on SNPH, ROFI achieved κ = 0.9077 (95% CI 0.8569–0.9586) versus G2Face κ = 0.7257. AUROC with ResNet50 on SNPH was 0.9113 for ROFI, versus Mosaic 0.6102, AIGC 0.7438, Face Swap 0.7983, Digital Mask 0.7853, and G2Face 0.8776. With ViT, ROFI reached 0.9124 on SNPH and 0.7894 on ECXHCSU. At 512×512 input, ROFI reached 94.59% AUROC versus G2Face 86.75%.
• Privacy protection: With AdaCos, ROFI protected 96.54% of images, versus Mosaic 90.91%, AIGC 93.08%, Face Swap 74.90%, Digital Mask 94.81%, and G2Face 93.51%. Traditional eye cropping protected only 68.40% (left eye) and 74.03% (right eye) under ArcFace, compared with 94.81% for ROFI. Against iris recognition with OpenIris, original or eye-cropped images had a 74.45% recognition success rate, while ROFI-protected images achieved 0.87%.
• Advantage over differential privacy: At a comparable identity recognition rate (3.0%), ROFI maintained 91.25% AUROC versus pixel-wise DP 62.32% and feature-wise DP 76.69%. Under a shadow-model membership inference attack calibrated to a True Positive Rate of about 1.4% at 1% False Positive Rate, the DP method dropped to 61.41% AUROC while ROFI stayed at 91.25%.
• Human recognition study: An initial 200-trial forced-choice study with 5 participants found significant differences among methods (Kruskal-Wallis H=21.69, P=0.0006). In an expanded 1,000-trial study of the three strongest methods, ROFI was recognized in 1.1% of cases (14/1200), versus Digital Mask 3.5% (42/1200) and G2Face 3.1% (38/1200), with one-tailed Fisher's Exact Test P = 0.000099 versus Digital Mask and P = 0.000529 versus G2Face.
• Reversibility: ROFI achieved 97.19% reversed ID similarity and 98.47% image similarity, versus G2Face's 74.72% and 93.48%.
• Clinical use of reversed images: For assessing TED hormone treatment, the "Single" no-history scenario gave κ = 0.2758, G2Face reconstruction gave 0.3913, and ROFI gave 0.8888 (95% CI 0.7393-1.0384).
• Key diversity: Sampling 100 different private keys on the same input produced a pixel-wise standard deviation of σ_pixel = 36.52 and a mean pairwise cosine distance of 0.8845 between identity features, showing outputs vary substantially with the key.

Methodology in Plain English

The team built ROFI around two ideas. First, rather than asking physicians to outline every diseased region, they trained a network on a large set of patient faces labeled only "healthy" or "not healthy," using a weakly-supervised region-score-max strategy so the network learns on its own which areas matter for disease. Second, they built a pair of Transformer-based translation models — a protector that alters identity and a restorer that undoes it — with a privacy key controlling the transformation. A Transformer-based enhancement network (DA-Former) cleans up any artifacts left by substituting disease features back into the anonymized image.

For validation, they used face photos collected at three Chinese hospitals between January 1, 2018 and December 25, 2023. The Introduction states 17,181 patients from Shanghai Ninth People's Hospital (SNPH), 493 from Eye Center of Xiangya Hospital of Central South University (ECXHCSU), and 79 from Renmin Hospital of Wuhan University (RHWU); the Results section gives final analyzed cohorts of 12,289 for SNPH (split into 11,836 developing, 222 model-selection, and 231 internal validation), 246 for ECXHCSU, and 48 for the Wuhan cohort (referred to as WUPH in the results). Three ophthalmologists reviewed each image and majority voting determined final labels. ROFI was compared against Mosaic, AIGC, SimSwap face swapping, Digital Mask, and G2Face, and evaluated with keypoint error, agreement statistics (Cohen's kappa, MCC, AUROC, Dice), face recognition systems, iris recognition, differential privacy baselines, an attack framework, and a human matching task.

Why This Matters

Research impact: ROFI reframes medical de-identification as a sign-preserving, reversible problem rather than a binary utility-versus-privacy tradeoff, and shows that weakly-supervised learning can localize clinically meaningful regions without expensive expert annotation. It also provides a benchmark protocol — clinical agreement, AI model agreement, face/iris recognition, differential privacy, membership inference, and human studies — for evaluating future medical anonymizers.

Real-world applications:

• Remote or cloud-based AI screening of eye disease from patient face photos, where transmitted images would otherwise expose identity.
• Safe sharing of ophthalmic image datasets for research and publication without losing diagnostic detail.
• Authorized reconstruction of original images for medical audits, longitudinal records, and treatment-response comparison, as shown in the TED hormone therapy assessment.
• Replacing the widely used but weak "eye cropping" strategy in ophthalmic departments, which the paper reports protects only about 68-74% of patients under ArcFace.

Industry relevance: Deployers of medical AI on remote GPU clusters, hospital privacy and compliance teams, telemedicine platforms, and ophthalmic imaging vendors all face the same tension between data utility and patient identity protection that ROFI directly targets; the paper's differential privacy and membership inference comparisons set a concrete utility bar for competing products.

Future Directions

• Broader disease and specialty coverage: The study covers eleven ophthalmic diseases with external manifestations; extending sign-preserving anonymization to conditions and image modalities not evaluated here is untested.
• Scaling external validation: The external cohorts (246 at ECXHCSU and 48 at WUPH) are small, and WUPH was excluded from AI model training due to its size, so generalization evidence remains limited.
• Operational and security questions: The paper does not report computational cost, latency, or deployment requirements for the protector/restorer models, nor the robustness of the privacy-key scheme against key compromise or adversarial reconstruction attempts beyond the tested attack framework.
• Integration with other medical image types and full-face conditions: The authors note the landmark accuracy could support facial palsy assessment and that extraocular signs matter for TED and SCC, but these are proposed extensions rather than validated results.

Target Audience

Ophthalmologists and clinical researchers interested in privacy-preserving data sharing; medical AI and computer vision researchers working on de-identification, face translation, or generative models under clinical constraints; hospital privacy, compliance, and data-governance teams; and graduate students or engineers who want a worked example of evaluating a privacy method with both clinical outcome measures and adversarial privacy metrics.

Authors’ abstract

Patient face images provide a convenient mean for evaluating eye diseases, while also raising privacy concerns. Here, we introduce ROFI, a deep learning-based privacy protection framework for ophthalmology. Using weakly supervised learning and neural identity translation, ROFI anonymizes facial features while retaining disease features (over 98\% accuracy, $κ&gt; 0.90$). It achieves 100\% diagnostic sensitivity and high agreement ($κ&gt; 0.90$) across eleven eye diseases in three cohorts, anonymizing over 95\% of images. ROFI works with AI systems, maintaining original diagnoses ($κ&gt; 0.80$), and supports secure image reversal (over 98\% similarity), enabling audits and long-term care. These results show ROFI's effectiveness of protecting patient privacy in the digital medicine era.