Building Auditable AI Text Detectors with Sparse Autoencoders

AI summary

Overview

Research area: Natural Language Processing — interpretability of AI-generated text detectors, combining sparse autoencoders (SAEs) with lightweight downstream classifiers.

Technical level: Intermediate to Advanced. The paper is readable with basic familiarity with neural network activations and linear classifiers, but assumes comfort with concepts such as residual streams, sparse autoencoders, and probing.

Scope (one sentence): The paper converts a black-box fine-tuned AI-text detector into an equally strong detector built only on human-readable SAE features that pass an automatic interpretability threshold, and then stress-tests whether those feature descriptions are faithful.

What This Paper Is About

AI-text detectors are used in high-stakes settings, but the strongest ones are opaque neural classifiers, while the transparent alternatives (stylometric and linguistic-feature classifiers) perform much worse. The authors ask whether a fine-tuned detector can be rebuilt out of human-readable concepts without losing accuracy. Their answer is a pipeline that trains SAEs on the detector's own residual stream, keeps only SAE features whose auto-generated descriptions pass a fidelity test, and trains a simple logistic regression on those features.

Key Contributions

1. A four-stage pipeline for auditable detection. Fine-tune a neural detector, train SAEs on its residual stream at multiple layers, auto-interpret each SAE feature and score its description, then fit an L1-regularized logistic regression on only the features that clear a global interpretability threshold.
2. First SAEs trained directly on AI detectors. The authors state that prior work (Scher, 2024; Kuznetsov et al., 2025) used SAEs from base models and did not evaluate the accuracy of feature descriptions, whereas this work trains SAEs on the detector itself and validates its explanations.
3. A global fidelity threshold rather than a top-M feature cut. Instead of removing the M least interpretable features (as in Kulkarni et al., 2026), they keep only features whose simulation score exceeds a threshold τ, which guarantees that retained features are themselves interpretable.
4. Causal validation of the explanations. A feature-level paraphrase ablation and a document-level "humanization" attack test whether the descriptions actually reflect what drives the detector's output, not just whether a classifier scores well.

Main Findings

• The interpretable detector matches the neural one in-distribution. On the COLING dev split, the fine-tuned gemma-2-2b detector scores 99.8 AUC / 98.0 F1 / 98.9 TPR@5% FPR, while FT-Model+SAE with masking at τ > 0.7 scores 99.2 AUC / 96.8 F1 / 96.7 TPR@5% FPR. Without masking, FT-Model+SAE reaches 99.5 / 97.5 / 98.1.
• Base-model SAE classifiers generalize better out-of-distribution than the fine-tuned detector. On the RAID test split, FT-SAE with masking reaches 94.5 AUC / 91.7 F1 / 84.4 TPR@5% FPR, and Base-SAE reaches 93.0 / 89.8 / 78.6, compared with 93.6 / 89.3 / 74.9 for the fine-tuned gemma-2-2b detector. Base-SAE and FT-SAE beat well-established baselines such as Binoculars and Fast-DetectGPT by around 8–12 F1.
• The interpretable detector inherits the detector's failures. FT-Model+SAE with masking falls to 92.0 AUC / 93.2 F1 / 64.5 TPR@5% FPR on RAID, below both base-SAE variants, showing that when the underlying detector fails to generalize, so does the concept-based version.
• Masking is cheap. Restricting to features with auto-interp score above 0.7 removes most features but leaves performance close to the unmasked systems across COLING and RAID; on some RAID metrics it slightly improves (FT-Model+SAE TPR@5% FPR rises from 61.5 unmasked to 64.5 masked).
• AI-generated peer reviews are too easy to detect. On PRDB, every trained classifier scores near ceiling (AUC 100, F1 99.6–100, TPR@1% FPR 100 in most cells), which the authors read as evidence of confounds or spurious cues in the dataset.
• Feature descriptions are causally faithful. In the feature-level paraphrase ablation, importance scores positively predict the logit change from paraphrasing a feature away: Spearman ρ = 0.69–0.81 on PRDB with 89–99% sign agreement, and ρ = 0.36–0.62 on COLING. Applying the τ > 0.7 mask sharpens this (FT-Model+SAE improves from 0.54 to 0.62 on COLING and from 0.69 to 0.80 on PRDB).
• The feature lists appear high-recall. In the document-level attack on 2,500 AI documents per dataset, generic humanizing prompts flip at most 8.7% of predictions and frequently 0%, while prompts listing the SAE features with descriptions flip 60–100% of predictions when iterated three times, all while editing less than 40% of tokens.
• The discovered features are specific and often non-stylistic. Feature 22_4800 ("double colons following bolded markdown section headers") fires in 11 human documents versus 54,293 GPT-4o documents; feature 13_164 ("opening square bracket in list or string literals") fires in 7 human documents versus 22,541 Llama documents. Human-predictive features on PRDB are more stylistic, such as first-person "I" at sentence starts, hedges like "seem", and "see" used as a directive.
• Base and fine-tuned SAE behavior differs qualitatively. On COLING, the top FT-SAE feature activates on introductory filler phrases such as "As an AI language model", while top Base-SAE features are dominated by topical and domain-level signals such as HTML attributes and software license boilerplate.

Methodology in Plain English

The authors start by fine-tuning gemma-2-2b as an ordinary AI-text classifier. Next, they train sparse autoencoders on that model's internal residual stream at layers 7, 13, 17, and 22, initializing from the GemmaScope width_16k/canonical checkpoints so each layer has a 16,384-feature dictionary.

For each document, they run every token through the SAE encoders, max-pool the feature activations over the sequence, and concatenate the four layers into one long feature vector.

To make features describable, they split the training corpus into quantile bins by each feature's document-level activation (plus a bin for inactive documents), sample a stratified set of snippets centered on the highest-activating token, and prompt gemini-3.1-flash-lite for a description of at most 20 words. They then validate each description by asking the same model to guess which quantile bin a document came from, using 10 documents for generation and 50 for validation; the Spearman correlation between predicted and true bins is the feature's simulation score.

A binary mask keeps only features whose score exceeds a threshold τ (they report results at τ > 0.7, and use τ > 0.85 for qualitative tables). On the masked features they fit an L1-regularized logistic regression, so any prediction can be explained by listing the top-k features by net contribution alongside their natural-language descriptions.

Three model variants are compared: Base-SAE (off-the-shelf GemmaScope SAEs), FT-SAE (GemmaScope fine-tuned to reconstruct base gemma-2-2b activations on in-domain data), and FT-Model+SAE (SAEs fit to the fine-tuned detector's activations). SAE training uses a JumpReLU architecture, a sparsity coefficient of 6e-4, learning rate 7e-5, batch size 4096, and 4 epochs.

Why This Matters

Detectors are already used to flag student essays, vet peer-review submissions, screen applicants, and moderate content, yet users often cannot inspect or contest a decision. This paper offers a route to a detector that keeps neural-level accuracy while grounding each decision in named concepts an auditor can read, argue with, and switch off.

Real-world applications:

• Peer review and publishing: screening submissions while showing editors exactly which cues triggered a flag, including formatting artifacts that a venue may decide are or are not legitimate evidence.
• Education and admissions: giving instructors or admissions officers a rationale they can discuss with a student rather than an opaque score.
• Content moderation and journalism: letting platforms and newsrooms audit and tune which textual tells drive automated flags.
• Compliance and red-teaming: using the feature lists as a diagnostic to humanize text and probe a detector's robustness before deployment.

Industry relevance: The authors emphasize that a model provider can audit and decide case-by-case whether a specific feature — such as a markdown formatting quirk — should be used by a specific detector. That kind of per-feature policy control is a practical affordance for governed deployment. The paper also reports that classifiers built on base-model SAE features are unexpectedly strong out-of-distribution detectors, which is relevant to anyone building detectors without task-specific fine-tuning. Work was conducted with Google DeepMind, with code and models released at the detectorscope repository.

Future Directions

• Fixing the recall bias in descriptions. The authors note that auto-interpretation pipelines produce high-recall, low-precision descriptions, and report that roughly half of their features received perfect descriptions (99.5+ F1) in an informal regular-expression check while the other half showed recall bias. They propose better sampling of hard negatives and synthetic data generation.
• Testing beyond GemmaScope. Only fine-tuned GemmaScope SAEs on gemma-2-2b were tested; the authors call for expanding to SAEs for other base models such as Llama and Qwen, particularly to understand the Base-SAE and FT-SAE conditions.
• Explaining why base-model SAE classifiers detect AI text so well. The authors follow Quaremba et al. (2026) in hypothesizing that the base model has learned linearly separable "AI text" features during pre-training, and leave further investigation to future work.
• Adversarial hardening and feature policy. Because the most predictive features are often highly specific token sequences and formatting elements that may not generalize, the authors encourage work on adversarial hardening via this style of unsupervised concept discovery, and on deciding which features are valid to use.

Target Audience

Researchers and practitioners in NLP interpretability who want a concrete template for turning a black-box classifier into a concept-based one; AI-detection researchers interested in out-of-distribution robustness and in what detectors actually key on; and policy, compliance, or platform-integrity teams who need detectors whose decisions can be explained, audited, and selectively constrained.

Authors’ abstract

AI-generated text detectors are increasingly deployed in consequential settings---peer review, education, admissions, journalism, and content moderation---where their outputs directly affect individuals. Yet the strongest detectors are fine-tuned neural classifiers whose decisions are opaque, while transparent alternatives (stylometric and linguistic-feature classifiers) underperform them by wide margins. We present a pipeline that converts a fine-tuned neural detector into a classifier that operates on human-readable concepts. We accomplish this by training sparse autoencoders (SAEs) directly on a detector's residual stream, assigning auto-interpretation labels to each SAE feature, and retaining only features whose generated descriptions pass a global fidelity threshold. We then train a lightweight classifier on just the interpretable features. We find that our SAEs recover latent concepts that correspond to highly specific and predictive linguistic "tells" of AI-generated text. Classifiers built on these extracted features are not only highly interpretable but exhibit stronger performance on out-of-distribution texts than either neural or stylometric alternatives.