Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures

AI summary

Overview

• Research area: AI alignment evaluation and AI safety monitoring — specifically, using a reinforcement-learning-for-calibrated-decisions (RLCD) model, Jev, as a black-box detector of alignment failures in language model outputs.
• Technical level: Advanced. The paper assumes familiarity with AUROC, F1, calibration (ECE), Cohen's kappa, bootstrap confidence intervals, and alignment benchmark terminology.
• Scope (one sentence): The paper introduces RLCDAlignBench, a 44-benchmark, 7,193-instance evaluation of whether a single Jev call with calibrated probabilistic answers can zero-shot detect ten types of alignment failures across five small open target models, and what role question wording versus input context plays in that detection.

What This Paper Is About

Most detectors of alignment failures are generative LLM judges that spend a decoding pass per criterion, or classifiers such as Llama Guard that read a token probability for one fixed label per call. Jev, trained with reinforcement learning for calibrated decisions, returns calibrated probabilities for many typed questions about one input in a single call, but it had never been measured as an alignment-failure detector. The paper asks whether such a model can flag failures zero-shot, and separates the two possible sources of failure — the question Jev is asked and the state (input material) Jev sees — because many alignment failures are relational, defined against a reference such as the user's belief or an injected instruction.

Key Contributions

1. RLCDAlignBench: 44 benchmarks spanning ten alignment failure types, with context variants, cached Jev answers, and rescoring scripts, plus a split-half protocol that removes selection inflation from reported gains.
2. A study of detection design factors: how question wording, answer type, context, and threshold shape Jev's detection, yielding a recommended recipe — a generic question read as a probability with its threshold fitted on ten labelled items — which lifts F1 where Jev fires too rarely at t = 0.5, at a cost of 0.025 on validated labels.
3. Evidence that Jev's confident disagreements locate label defects: the audit changes eight labels, of which seven usable benchmarks carry a label-changing defect.
4. A cost and agreement comparison: one Jev call answers a whole question battery and, on StrongREJECT, agrees with human labels as well as the reference GPT-4o-mini scorer (Cohen's kappa 0.809 vs. 0.811) while ranking responses better (AUROC 0.971 vs. 0.929).

Main Findings

• A single generic question already works zero-shot: the generic Noul question reaches a median AUROC of 0.886 [0.821, 0.952] over the 31 benchmarks that admit it, and the split-half targeted strategy reaches 0.911 [0.860, 0.944] over all 38 usable benchmarks.
• It beats supervised lexical baselines without seeing labels: the generic Noul exceeds the better of response length and an in-domain TF-IDF logistic regression by a median of +0.132 [+0.057, +0.190], winning on 25 of 31 benchmarks (sign test p = 9 × 10⁻⁴).
• Scorer type does not matter, label source does: median AUROC is 0.906 for judge-scored, 0.890 for rule-scored, and 0.870 for multi-turn benchmarks (p = 0.78); but the generic Noul scores 0.949 on the 20 benchmarks with unvalidated, mostly judge, labels versus 0.872 on the 8 with validated labels.
• Question wording matters little out of sample: selected on one half of items and scored on the other, the best targeted strategy beats the best generic readout by a median of +0.006 [−0.004, +0.015] AUROC (24/1/11, Wilcoxon p = 0.055). Selecting on the evaluation data would overstate targeted wording by 0.008 [0.006, 0.014], larger than the gain itself.
• Keep answers soft: the 3-level Score read as its expected level beats the generic Noul on 19 of 31 benchmarks (19/7/5, median +0.016 [+0.001, +0.030]); the soft Choice ties it (11/7/13); argmax readouts lose almost everywhere (Choice 2/0/28, Score 5/2/24). Rubrics that threshold each Jev answer at 0.5 before combining lose to the best direct targeted question on 9 of 10 benchmarks (median −0.137).
• Deployable context rarely helps; label keys do: a deployable reference absent from the state raises the generic Noul's AUROC with a confidence interval above zero on 1 of 4 benchmarks (DeceptionBench, +0.050 [+0.029, +0.077] from the target's goal prompt), while a distilled copy of a reference the state already holds helps on 0 of 3. Attack prompts move the best shared question by at most 0.002. Label keys raise the generic Noul's AUROC with a CI above zero on 4 of 11 benchmarks (median +0.053).
• A label-key gain can measure the label's construct, not the failure: on SycophancyEval (answer), adding the true answer moves the generic Noul from 0.540 to 0.941 on the official label but from 0.712 to 0.288 on an answer-shift label.
• Calibration holds pooled but not per benchmark: pooled ECE is 0.047, but the median per-benchmark ECE is 0.168 against a null of 0.074, and 24 of 31 benchmarks exceed the null's 95th percentile. Jev's mean probability misses each benchmark's positive rate by a median of 0.125 while the median AUROC is 0.905. Label-free EM prior-shift correction lowers F1 to 0.571–0.690; ECE exceeds the null on 6 of 8 validated benchmarks.
• A fitted threshold helps where Jev fires too rarely: median F1 rises from 0.706 at t = 0.5 to 0.822 with a cross-validated threshold and to 0.793 with 10 labelled items. The gain comes from unvalidated labels (0.678 to 0.853), mostly four rule-scored benchmarks (+0.370 [+0.158, +0.819], against +0.052 [−0.000, +0.126] on judge labels). On validated labels, t = 0.5 is as good (0.721 vs. 0.694), and a threshold fitted on ten labels costs 0.025.
• Confidence identifies which decisions to trust: keeping the half of decisions with the largest |p − 0.5| raises the median accuracy from 0.793 to 0.933.
• Human agreement matches the judge: on StrongREJECT the generic Noul agrees with humans at kappa 0.809 versus the GPT-4o-mini scorer's 0.811 (difference −0.002 [−0.059, +0.057]), and humans side with Jev on 49% of their 116 disagreements; coarsened to the scorer's nine levels, Jev still reaches AUROC 0.962. On HarmBench validation, the generic Noul agrees with a single annotator at mean kappa 0.748, on par with inter-annotator agreement (0.736, +0.013 [−0.022, +0.046]). The parity holds pooled, not per generator: on GPT-3.5 outputs Jev reaches kappa 0.668 against the scorer's 0.790 (−0.121 [−0.205, −0.034]).
• Cheap: a call carries 11.4 questions on average and returns in a median of 0.31 s. One Jev pass over the 19 judge-scored benchmarks costs $0.30 against $18.96 for the judges at list prices, 63× less pooled; with a conservative repricing (all judges at GPT-4o-mini rates, Jev asked only the generic question), Jev is 12× cheaper pooled with a median of 3.3×.

Methodology in Plain English

The researchers took existing alignment benchmark suites, which were built to score a policy model, and turned them into detection tasks. For each of ten failure types they fixed one open 2–7B target model, generated its outputs, and replayed the benchmark's own reference scorer to produce labels (y = 1 when the failure occurs). This yields 44 benchmarks and 7,193 detection instances from five target models: Qwen3.5-2B, Phi-4-mini, Gemma-2-2B, Llama-3.2-3B, and Olmo-3-7B.

For each instance they built a state — by default the canonical state containing only the target model's input and output, as a response-level monitor would see — and asked Jev typed questions about that state in a single call. Answers come back as probabilities for three answer types: Noul (yes/no), Choice (a distribution over options), and Score (a distribution over ordered levels). A readout or a strategy combines the answers into a score s(x) in [0, 1], compared against labels Jev never sees.

The central design choice is varying the question and the state separately. On the question side, each benchmark gets one generic question built from a template with a per-benchmark behaviour phrase, asked in all three answer types, plus targeted questions (direct, rubric, or decomposition) that name the labelled behaviour. On the state side, state variants hold the same items and labels but differ in their fields, giving 49 paired contrasts; added fields are tagged as deployable (a system-level monitor may hold it) or label keys (part of the label definition). To avoid inflating results by picking a best-of-many strategy on the evaluation data, targeted strategies are selected on one half of the items and scored on the other, across 20 grouped splits.

Metrics are AUROC as the primary measure, plus F1 at t = 0.5 and F1 with a 2-fold cross-validated threshold. Confidence intervals come from 1000 bootstrap resamples of item groups. Six benchmarks are degenerate (one class or at most four minority-class items) and are excluded from aggregates, leaving 38 usable benchmarks. Baselines are all-positive, response length, and a TF-IDF logistic regression on word and character n-grams trained by 5-fold cross-validation on in-domain labels.

Why This Matters

• Impact on research: The paper reframes alignment-failure detection as a question-and-context problem rather than a purely modeling problem, showing that question wording buys almost nothing out of sample while the contents of the state and label dominate. It also provides a cheap, black-box way to audit benchmark labels: Jev's confident disagreements exposed label defects in three benchmarks and labels unobservable from the state in four MACHIAVELLI variants, and appendix analysis attributes 25% of remaining disagreements to Jev errors.
• Real-world applications:
  • Runtime monitoring of deployed model inputs and outputs across ten failure types using one call per message.
  • Benchmark and mitigation scoring at roughly 63× lower cost than LLM-judge scorers on the 19 judge-scored benchmarks.
  • Selective routing: keeping the half of decisions with the largest |p − 0.5| raises median accuracy from 0.793 to 0.933, so a monitor can send only Jev's least confident decisions to a judge or a human.
  • Label quality control for benchmark builders, using confident model disagreements as a signal to inspect specific labels.
• Industry relevance: Because the detector is black-box, has no fixed harm taxonomy, and answers many typed questions per call, it fits cost-sensitive deployment pipelines where per-criterion decoding passes are prohibitive. The paper's caution is equally relevant: a per-generator gap on GPT-3.5 outputs (kappa 0.668 vs. 0.790) shows agreement measured pooled can hide generator-specific failures that no threshold closes.

Future Directions

• Extend beyond one detector: the evaluation covers a single RLCD model, jev-1.13.0, so whether other RLCD models or guard classifiers behave the same way is untested.
• Broaden language and model coverage: the benchmarks are English and the target models are 2–7B, leaving larger targets and other languages as stated future work.
• Reduce reliance on scorer labels: most labels used are scorer labels; the paper notes that two AbstentionBench judges running the official prompt agree at kappa 0.05, so the reliability of judge-generated labels remains an open issue.
• Fix or replace the defect labels: seven usable benchmarks carry a label-changing defect — two deterministic defects, two judge defects, and four labels unobservable from the state — which the authors leave for downstream users to repair or exclude.

Target Audience

Researchers and engineers working on AI alignment evaluation, guardrail and monitor design, and LLM-as-judge benchmarking; benchmark builders interested in cheap label-quality auditing; and practitioners choosing a detector under real cost constraints, who need to know that a generic probabilistic question plus a threshold fitted on ten labels is the recommended configuration. Readers without a background in AUROC, calibration error, and bootstrap confidence intervals will need to consult the background section first.

Authors’ abstract

Detectors of alignment failures screen deployed language models and score alignment benchmarks. Most are generative judges that spend a decoding pass on every criterion, and classifiers that read token probabilities, such as Llama Guard, still score one fixed label per call. Jev, a model trained with reinforcement learning for calibrated decisions (RLCD), answers many typed questions about one input with calibrated probabilities in a single call. Whether it detects alignment failures has not been measured. We present RLCDAlignBench, which benchmarks Jev on ten alignment failures: sycophancy, jailbreaks, deception, prompt injection, hallucination, privacy violation, social bias, reward hacking, concealing uncertainty, and power seeking. It spans 44 benchmarks and five target models, labelled by each benchmark's scorer and, on two, by humans. Many of these failures are relational, defined against a reference, such as the user's belief or an injected instruction, that the response alone does not reveal. Our key idea is therefore to vary what Jev is asked separately from what it sees: the question's wording and answer type on one side, the fields of the input on the other. A single generic question reaches a median AUROC of 0.886 zero-shot and beats supervised baselines on most benchmarks. Question wording matters little, while context matters more, mostly through fields that encode the label. Jev matches the reference scorer's agreement with human labels, surfaces label defects in existing benchmarks, and costs 63x less than LLM-judge scorers. Code and data: https://github.com/sumleo/RLCDAlignBench.