RealStats: A Rigorous Real-Only Statistical Framework for Fake Image Detection

AI summary

Overview

Research area: Computer vision, specifically the detection of AI-generated (fake) images, approached through statistical hypothesis testing rather than supervised classification.

Technical level: Advanced. The paper assumes familiarity with p-values, empirical cumulative distribution functions (ECDFs), chi-squared tests, Cramér's V, and p-value aggregation methods such as Stouffer's test.

Scope: The paper presents a training-free, real-only detection framework that converts multiple hand-crafted image statistics into calibrated p-values and aggregates them into a single interpretable score, evaluated against three training-free baselines across 187K images spanning multiple generative model families (accepted to AISTATS 2026).

What This Paper Is About

Most fake image detectors are trained on labeled fakes, which ties them to the generators present in training data and yields outputs that are hard to interpret statistically. This paper instead asks a narrower, more rigorous question: given only a reference set of real images, how likely is a candidate image to have come from that real-image population? The authors answer this by treating detection as a hypothesis test, producing a p-value that has an explicit probabilistic meaning, while flexibly combining many different detector statistics without ever using fake images.

Key Contributions

1. A real-only detection framework framed as hypothesis testing. The method formulates fake image detection as a test of the null hypothesis that an image is drawn from the real-image distribution, returning an aggregated p-value rather than a generic "realness score."

2. A modular, training-free statistic set. Statistics are computed as cosine similarities between clean and Gaussian-perturbed embeddings from multiple frozen encoders, so new detectors and perturbation strengths can be added later without retraining.

3. A principled independence-selection procedure. Pairwise chi-squared tests with Cramér's V build an independence graph, from which a maximal clique is extracted and then regularized by a Kolmogorov–Smirnov uniformity check, ensuring the selected statistics support valid p-value aggregation.

4. Demonstrated adaptability without retraining. The framework absorbs an external statistic (ManifoldBias) into the same test structure, improving results on generators where the base statistics underperform.

Main Findings

• Competitive average performance with lower variance. Across all generator splits, the Min-p ensemble reached AUC 0.775 ± 0.126 and AP 0.756 ± 0.119; Stouffer aggregation reached AUC 0.756 ± 0.135 and AP 0.743 ± 0.133. Baselines were Manifold Bias (AUC 0.761 ± 0.179, AP 0.753 ± 0.169), RIGID (AUC 0.769 ± 0.194, AP 0.765 ± 0.189) and AEROBLADE (AUC 0.697 ± 0.161, AP 0.697 ± 0.163). Manifold Bias and RIGID obtained slightly higher peak scores, but the authors emphasize their own substantially lower variance across generators.

• Per-generator behaviour differs sharply between methods. Manifold Bias scores highly on GauGAN but drops on StarGAN and SDv2; RIGID performs well on SDXL but struggles on SDv1.4; AEROBLADE is strong on ADM but weak on SDv1.5. The proposed method shows weaknesses on datasets such as CycleGAN but is described as more balanced overall.

• Adding a statistic fixes weak spots. On GauGAN, CycleGAN and SAN — generators where ManifoldBias excels and the base method was weaker — incorporating the ManifoldBias statistic under the same experimental setup improved the Min-p ensemble's average AUC.

• Broader separation gain. On the full test benchmark of 160K images, incorporating ManifoldBias shifted fake samples' p-values toward zero while real samples remained uniform, raising overall AUC.

• Degraded calibration still preserves discrimination. In a controlled domain-shift experiment using reference ECDFs from only 1k FFHQ faces (less than 4% of the test size) and inference on 30k high-resolution CelebA faces, real samples deviated from uniform behaviour while fake samples stayed concentrated near zero. RealStats still achieved AUC = 0.79 and AP = 0.845, versus AUC = 0.70 and AP = 0.74 for a perturbation-based heuristic under identical conditions.

• Partial robustness to corruptions. With the Min-p ensemble, Gaussian blur left performance stable or slightly improved, while JPEG compression caused a moderate drop of 5% AUC and 6.4% AP without invalidating the p-values.

• Efficiency. The method scales with GPU parallelism, has moderate memory use even with many statistics and large batches, and the independence-testing stage adds negligible overhead relative to forward-pass cost. It is described as faster and lower-memory than training-free methods built on heavy autoencoders such as Stable Diffusion.

Methodology in Plain English

The pipeline has two phases.

Null distribution modeling. A reference set of real images is pushed through a set of frozen feature extractors — CLIP ViT-L/14, DINOv2 ViT-L/14, DINOv3 ViT-S/16, DINOv3 ViT-H/16, ConvNeXT and BEiT ViT-L/16 — each paired with Gaussian perturbation strengths of 0.05 and 0.10. For each (encoder, perturbation) pair, the statistic is the cosine similarity between the clean embedding and the embedding of the noisily perturbed image, so real images tend to give more stable features. Each statistic's distribution over real images is stored as an ECDF.

Because combining p-values with Stouffer's test or the minimum-p method requires independence, the authors test all pairs of statistics with chi-squared tests, measure association with Cramér's V, and accept an edge when V is at most 0.07. A maximal clique in this graph gives a candidate set, which is kept only if the aggregated p-values pass a Kolmogorov–Smirnov uniformity test at α = 0.05. When several valid cliques exist, those containing strong encoders such as DINOv2 and CLIP are preferred. Other hyperparameters are 15 chi-squared bins, 400 ECDF bins, and the minimum-p aggregation method as the default.

Inference. For a candidate image, only the selected statistics are computed. Each is mapped to a two-sided empirical p-value via its stored ECDF: p(x) = 2 · min(F̂(s(x)), 1 − F̂(s(x))). These are then merged into one number, either by Stouffer's test (averaging inverse-normal-transformed p-values and mapping back through the normal CDF) or by the minimum-p method (with CDF 1 − (1 − t)^K). The image is declared fake if the unified p-value falls below a chosen significance level. The paper states that under the null hypothesis this final value is uniformly distributed on [0,1], given i.i.d. real samples, independent statistics, and a test image drawn from the same real distribution.

Evaluation setup. Real images were split so that 30% were used for ECDF estimation and the remaining 70% held out for testing, meaning no inference image informed calibration. Results are reported as AUC and AP rather than accuracy, because the method outputs p-values rather than requiring a fixed threshold. Baselines RIGID, AEROBLADE and ManifoldBias were evaluated under the same protocol; RIGID was reimplemented from its paper description using a DINOv2 backbone with perturbation strength 0.05, since its official implementation was not publicly available at the time of writing.

Why This Matters

The work reframes fake image detection as calibrated statistical inference instead of classification, which matters because generator architectures change faster than datasets can be relabeled. A p-value tied to a fixed significance level gives practitioners a principled error-control story, and the real-only requirement means the detector can be maintained without continuously collecting new fake examples.

The paper does not enumerate application domains, but the setting implies several:

• Content moderation and platform integrity, where a calibrated score helps set and justify review thresholds rather than relying on opaque classifier confidence.
• Journalistic and forensic verification, where the ability to state that a result is extreme under the real-image distribution supports evidentiary reasoning.
• Dataset curation and provenance auditing, where large image collections need screening without a labeled fake corpus.
• Deployment under domain shift, such as mismatched resolution or content categories, where the paper shows discrimination survives even when calibration degrades.

Industry relevance: the framework's modularity means organizations can add their own in-house statistics to the same aggregation machinery, and the reported efficiency means inference does not require the heavy autoencoders that some competing training-free detectors rely on.

Future Directions

• Reference-distribution quality. The authors identify finite sampling and distributional shift as the two ways the empirical null model can become unrepresentative. How to detect, quantify and correct misalignment — beyond inspecting the p-value distribution on a validation set — remains open.

• Better clique selection. Performance depends on which independent statistic set is chosen, and correlated but informative detectors may be excluded. Alternative selection criteria beyond the Cramér's V threshold and KS uniformity constraint could recover that lost separability.

• Extending the statistic library. The paper positions the framework as extensible and demonstrates this by absorbing ManifoldBias; systematically identifying which new training-free statistics to add for emerging generator families is a natural next step.

• Comparison and adoption of RIGID implementations. Because the official RIGID code was unavailable and the authors reimplemented it, they state they will release their implementation including the original setup plus variants with alternative feature extractors and perturbation levels, which invites independent reproduction of the baseline comparisons.

Target Audience

Researchers and graduate students in computer vision, statistical machine learning and AI forensics who are comfortable with hypothesis testing and multiple-comparison aggregation. It is also relevant to practitioners building detection systems who need an interpretable scoring mechanism and want to understand the trade-off between statistical validity and raw AUC. Readers looking for a benchmark-driven leaderboard contribution will find less here; the paper's distinctive value is its framing, the independence-selection machinery, and the explicit statement of when the resulting p-values stop being true p-values.

Authors’ abstract

As generative models continue to evolve, detecting AI-generated images remains a critical challenge. While effective detection methods exist, they often lack formal interpretability and may rely on implicit assumptions about fake content, potentially limiting robustness to distributional shifts. In this work, we introduce a rigorous, statistically grounded framework for fake image detection that focuses on producing a probability score interpretable with respect to the real-image population. Our method leverages the strengths of multiple existing detectors by combining training-free statistics. We compute p-values over a range of test statistics and aggregate them using classical statistical ensembling to assess alignment with the unified real-image distribution. This framework is generic, flexible, and training-free, making it well-suited for robust fake image detection across diverse and evolving settings.