SAEScientist-Bench: Can AI Agents Conduct Autonomous SAE Interpretability Research?

AI summary

Overview

Research area: Mechanistic interpretability, autonomous AI research agents, and AI safety evaluation benchmarks.

Technical level: Intermediate — readers need basic familiarity with language models, sparse autoencoders (SAEs), and activation steering, but the paper's framing and scoring framework are explained self-containedly.

Scope: The paper introduces SAEScientist-Bench, a 20-task benchmark that measures whether frontier AI agents can autonomously conduct SAE-based interpretability research — discovering a single feature in a 131K-feature dictionary that represents a given concept, judged on activation rank, concept selectivity, and causal steering against expert-curated Neuronpedia baselines.

What This Paper Is About

Most work on recursive self-improvement (RSI) automates the training pipeline — data curation, algorithm design, post-training — while treating the resulting model as a black box judged only by external task performance. That leaves a critical gap: nothing monitors what the model actually learned internally, which makes reward hacking, specification gaming, and deceptive alignment harder to detect. This paper asks whether AI agents can close that gap by acting as autonomous interpretability scientists, using Sparse Autoencoder (SAE) tools to search for, test, and validate the internal features that represent a target concept.

Key Contributions

1. SAEScientist-Bench — A benchmark of 20 feature-discovery tasks spanning multilingual understanding (Portuguese, Spanish, Latin, Turkish), specialized document formats (earnings reports, tax filings, job postings), and domain knowledge (clinical symptoms, pharmaceutical dosing), all set against layers 9 and 20 of Gemma-2-9B-IT with its pretrained Gemma Scope SAE dictionaries of 131,072 features each.

2. A standardized three-axis scoring framework — Activation Rank (dictionary-wide prominence relative to Expert), Activation Selectivity (AUROC separation of positive texts from hard-negative and neutral controls), and Causal Steering (net gain in target-relevance ratings under activation-addition intervention versus baseline and a norm-matched random control), aggregated into an unweighted Overall score.

3. A reference baseline from Neuronpedia — Each task is paired with a frozen expert feature established via public steering presets or standard expert curation, enabling fair comparison; task descriptions, Expert features, and evaluation suites stay hidden during discovery.

4. Systematic behavioral analysis — Traces of agent-authored probes, candidate comparisons, search workflows across 10 frontier configurations and 3 independent runs each, exposing where autonomous discovery succeeds and where it fails.

Main Findings

• Agents are genuine but incomplete scientists: Kimi K3 leads with an Overall score of 65.82, followed by Claude Opus 5 (65.41) and Claude Sonnet 5 (65.04) — all well below the Expert baseline of 85.56, but far above chance and stable across runs (standard deviations typically 1–3 points).

• Different models win different dimensions: Claude Opus 5 leads Activation Rank (75.35), Kimi K3 leads Activation Selectivity (92.91), and Grok 4.6 leads Causal Steering (31.47). No single model dominates all three axes.

• Selectivity is nearly solved; causality is not: Top agents approach the Expert baseline on Activation Selectivity (92.91 vs. 98.92), meaning they can identify features that fire selectively on the target concept. But on Causal Steering they reach only 31.47 versus Expert's 57.75 — finding a correlated feature does not translate into finding a usable steering vector.

• Search strategy varies widely but volume alone does not predict success: GPT-5.6 Sol authors the most probe text (~115.1 texts, ~20.2 candidates tested), Claude Opus 5 screens the most candidates (~80.2 across ~77.6 probe texts), and Kimi K3 runs a highly targeted search with few retrieval queries yet tops the leaderboard. What matters is informative counterexample design, not query count.

• Active verification separates rigorous agents from heuristics-driven ones: Opus 5 repeatedly probes a candidate before accepting or rejecting it; Sonnet 5 frequently encounters the Expert feature during initial retrieval but ignores it without direct validation. Since Expert identity is anonymized, this reflects genuine differences in scientific thoroughness.

• Failure mode 1 — misreading measurements: Opus 5's Portuguese submission activates more strongly on an English control (4.50) than on the Portuguese target (3.81), collapsing its Overall score to 13.91 despite a well-designed probe suite.

• Failure mode 2 — surface distractors: A Cat candidate fires at 44.00 on "Copycat killer" and 24.54 on "Catalytic converter"; Opus 5 correctly diagnoses this as substring matching and rejects it. GLM-5.2 makes the opposite error on the clinical-symptom task, dismissing a 70.25 activation on symptom-free medical text as negligible and selecting a document-format feature instead.

• High selectivity can mask weak signal: Sol and Expert both achieve AUROC 1.000 on Portuguese, but Sol's target activations average only 3.5–10.0 versus Expert's 14.4–19.4, dragging Sol's Rank score to 3.97 versus 100.00. Rank and selectivity must be read together.

• Top-three-token averaging beats max-token aggregation: Isolated lexical spikes (e.g., a Portuguese loanword producing a 5.56 spike on an English control) degrade Sol's AUROC to 0.969 under max aggregation but recover to 1.000 under top-three averaging; Opus 4.8 improves from 0.927 to 0.958.

• Steering success comes with quality costs: Sol's Portuguese steering reaches 88.75 with instruction preservation of 3.525; Opus 4.8 reaches 84.38 with 3.050 due to repetition and length drift on a four-line poem (it outputs eleven lines). Kimi K3's broad real-estate feature boosts relevance but raises degeneration from 32.5% to 52.5%.

• Benchmark scores track general capability: Across matching frontier models, Overall score correlates strongly with the Artificial Analysis Intelligence Index v4.2 (ρ = 0.800), positioning white-box interpretability as a distinct but related capability frontier.

Methodology in Plain English

Each task hands an agent a target concept (say, Portuguese text) plus the identity of the base model and SAE layer, and asks it to return exactly one feature ID from a 131,072-feature dictionary that best represents that concept.

The agent works through a restricted probe_sae interface: it can submit up to 64 of its own written texts and either retrieve the top-k most activating features or measure a specific set of candidate features, receiving back activations and full-dictionary ranks. Crucially, the agent never sees the Expert feature or the evaluation texts during this search — it must invent its own contrastive probes (real Portuguese sentences versus Spanish translations, English discussions of Portuguese, and so on) to rule out spurious candidates.

Once the agent submits a feature, the evaluation pipeline takes over. It measures that feature's activations on a frozen suite of held-out positive texts, hard negatives, and neutral controls to compute Selectivity via AUROC, positions it against the full dictionary to compute Rank relative to Expert, and finally performs causal steering — adding the feature's decoder direction to the hidden state with strength α during inference and rating whether the outputs shift toward the concept. Steering is rated on a 0–4 scale by an automated GPT-4o judge in two passes on anonymized output triples (baseline, steered, random-direction control), with α calibrated on held-out prompts to avoid degeneration. The final Steering score is the net gain over the stronger of baseline and random control, normalized to 0–100.

Every configuration runs three independent end-to-end investigations per task, and all 20 tasks are weighted equally. Ten frontier agent setups were tested — Kimi K3, Claude Opus 5, Claude Sonnet 5, Claude Opus 4.8, Grok 4.6, Gemini 3.8 Flash, GLM-5.2, and the GPT-5.x series (Sol and Luna in Codex; the rest in Cursor) — with network access disabled and no artificial step caps, so agents chose their own reasoning effort, exploration depth, and stopping criteria.

Why This Matters

Impact on research. The paper reframes interpretability as a measurable, benchmarkable agent capability rather than an ad-hoc tooling exercise. It supplies the missing third pillar for trustworthy RSI: if autonomous training loops are to be safe, something must audit internal representations continuously, and this work quantifies how far current agents are from doing that unaided. It also exposes a clean separation of concerns — correlation-based feature discovery is nearly solved, causal validation is not — which gives the field a concrete target.

Real-world applications:

• AI safety auditing: Automated pipelines that watch a model's internal features for concepts it should not have acquired (deception, unsafe instructions) without requiring a human to hand-craft probes for each one.
• Model debugging and failure diagnosis: When a deployed model behaves unexpectedly, an agent could locate the responsible internal representation and test whether intervening on it fixes the behavior.
• Behavioral control and customization: Reliable SAE steering vectors enable fine-grained control over model outputs — enforcing language choice, tone, or domain focus at inference time without retraining.
• Automated red-teaming and evaluation: The contrastive-probe methodology generalizes to any "does this model represent concept X?" question, useful for compliance checks and pre-deployment audits.

Industry relevance. Foundation model labs need scalable, low-human-cost interpretability if they intend to ship models with auditable internals. This benchmark gives them a standardized way to compare agent performance, identify whether a given frontier model is a competent scientific collaborator, and decide where to invest — in particular around causal steering, where every model tested falls far short.

Future Directions

• Close the causal steering gap. The largest deficit — 31.47 versus 57.75 for Expert — is also the least understood. Research should determine whether agents pick the wrong features, whether the right features exist at the chosen layers, or whether agents simply cannot recognize causal potency from activation statistics alone.

• Attack measurement misinterpretation directly. Opus 5's Portuguese collapse and GLM-5.2's clinical-symptom error both stem from agents misreading their own experimental results. Training or scaffolding agents to reason more carefully over probe outputs — rather than merely generating more probes — looks like the highest-leverage intervention.

• Extend beyond single-feature discovery. Real interpretability work involves circuits, feature composition, absorption, and splitting. Tasks requiring multi-feature explanations, interaction analysis, or cross-layer tracing would test a substantially harder capability.

• Broaden the model and layer coverage. This benchmark covers one base model (Gemma-2-9B-IT) and two layers. Whether agent competency transfers to other architectures, larger scales, and MLP or attention SAEs remains open — as does whether the ρ = 0.800 correlation with general intelligence holds as a predictive shortcut.

Target Audience

AI safety and interpretability researchers; agent benchmark designers evaluating autonomous scientific capability; foundation model labs building auditing or steering tooling; and graduate students or advanced practitioners who want a concrete, quantitative entry point into mechanistic interpretability as an agentic task rather than a manual analysis technique. The paper is most useful to readers who already understand what an SAE is and want to know where autonomous systems currently stand — and precisely where they fail.

Authors’ abstract

While research on recursive self-improvement (RSI) has predominantly automated model training pipelines, reliable autonomous development demands a missing pillar: post-hoc monitoring and auditing to understand what models learn and ensure safe alignment. Mechanistic interpretability tools are essential to bridge this gap, among which Sparse Autoencoders (SAEs) serve as a cornerstone by isolating interpretable features for model inspection and steering. In this paper, we introduce SAEScientist-Bench to evaluate whether AI agents can act as scientists utilizing SAE tools for autonomous mechanistic discovery. Given a target concept, an agent designs contrastive probes and navigates a Gemma Scope dictionary of 131K+ features in Gemma-2-9B-IT to discover the optimal feature, evaluated against curated expert reference features anchored on Neuronpedia across activation rank, concept selectivity on contrastive texts, and causal steering. Across 10 agent configurations and 20 tasks, frontier agents demonstrate genuine discovery capabilities and lead different evaluation dimensions, but remain well behind the expert baseline, approaching expert levels on separating target concepts from contrastive controls while lagging substantially in causal generation steering. Further analysis reveals that although agents can design contrasts to rule out spurious candidates, they frequently misinterpret experimental measurements. These results establish experimental model understanding as a measurable capability for closed-loop autonomous AI R&D. Our code is available at https://github.com/Trae1ounG/SAEScientist.