SAHOO: Safeguarded Alignment for High-Order Optimization Objectives in Recursive Self-Improvement

AI summary

Overview

Research area: AI safety and alignment, specifically safeguards for recursive self-improvement (RSI) in large language models, combined with empirical evaluation on code generation, mathematical reasoning, and truthfulness benchmarks.

Technical level: Intermediate. The paper is accessible to readers who know standard LLM evaluation metrics (pass@1, accuracy, exact match) and basic concepts such as embedding distance and KL divergence, since the appendix derives every drift component in detail.

One-sentence scope: The paper introduces SAHOO, a monitoring-and-control framework that measures and bounds alignment drift while a model iteratively revises its own outputs, and evaluates it on 189 tasks spanning HumanEval, TruthfulQA, and GSM8K.

What This Paper Is About

Systems that critique, revise, and re-evaluate their own outputs can improve over repeated cycles, but each cycle risks quietly shifting the model away from its intended behavior. The authors ask whether capability gains and alignment preservation can be achieved simultaneously, and they build a framework whose thresholds and component weights are learned from calibration data rather than chosen by hand. Their framing example is that a system gaining 10 percent in code generation while losing 15 percent in truthfulness has not meaningfully improved at all.

Key Contributions

1. The Goal Drift Index (GDI). A learned multi-signal detector that combines four drift modalities — semantic (embedding-space cosine distance), lexical (Jensen-Shannon divergence over token distributions), structural (normalized format/feature differences), and distributional (1-Wasserstein distance between response distributions) — into a weighted composite, with weights optimized to maximize detection AUC on human-labeled drift data.
2. Constraint preservation as an explicit safety mechanism. Constraints are formalized as logical predicates (format, content, logical, ethical), scored with a hard Constraint Preservation Score (CPS) that has no soft relaxation, with violation penalties injected into subsequent improvement prompts and a hard stop if any cycle reaches zero constraint preservation.
3. Regression-risk quantification and long-horizon stability analysis. The framework estimates, at each cycle, the probability that later improvements undo earlier gains, and flags high-risk tasks early enough for human intervention.
4. The Capability Alignment Ratio (CAR) and an empirical demonstration. CAR is proposed as a way to reason about capability-versus-alignment trade-offs, and the full framework is tested on 189 tasks across three benchmark families, with data-driven thresholds calibrated on 18 tasks (6 per domain) over 3 cycles, yielding 54 calibration observations.

Main Findings

• Code generation improved most efficiently: quality rose from 0.672 to 0.795 (+18.3%), with mean GDI 0.320, CPS 1.00, and CAR 0.671.
• Mathematical reasoning improved similarly: quality rose from 0.689 to 0.805 (+16.8%), with the lowest mean GDI of the three domains at 0.330, CPS 1.00, and CAR 0.6749.
• Truthfulness was the hardest domain: quality rose only from 0.678 to 0.704 (+3.8%), with mean GDI 0.354, mean CPS 0.9874 (SD 0.0547), and CAR 0.5987 — a smaller payoff at a higher alignment cost.
• Constraint violations were domain-specific: zero violations occurred in code generation and mathematical reasoning across all tasks and cycles; 170 violations occurred across the 63 truthfulness tasks (mean 2.70 per task, non-uniformly distributed).
• Violations concentrated in a few failure modes: of the 170 truthfulness violations, fabrication accounted for 91 (53.5%), overconfidence 48 (28.2%), and system-call-style outputs 15 (8.8%), with the remainder being specialized constraint types.
• Semantic drift dominates the composite index: calibrated weights were 0.38 semantic, 0.29 distributional, 0.21 structural, and 0.12 lexical, and the authors report these weights generalized across task types.
• Drift stayed bounded everywhere: no domain exceeded the 0.44 critical threshold. GDI rose early, stabilized by cycles 3–5, and then stayed low, with code and math near 0.35 and truthfulness slightly higher at 0.38–0.40.
• CAR peaks early and decays: efficiency approached 1.0 in the earliest cycles with cheap quality gains, declined by cycles 2–3 to roughly 0.6–0.7, and then stabilized. The Pareto frontier clustered at high quality (approximately 0.65–1.00) and low drift (approximately 0.20–0.50).
• Regressions were rare and concentrated: across 3,780 cycles, 170 regression events (4.5%) were observed, but 117 of these came from a single bimodal, oscillatory task; excluding that outlier, regression frequency dropped to 0.7%, below the primary risk threshold of 0.803. The framework flagged that high-risk task accurately.
• Stability scores were consistently high: mean 0.825 (SD 0.068), with only three tasks below 0.70, all manually verified as atypical (insufficient training data or bimodal behavior).
• Non-convergent tasks were detectable early: among the 16 tasks that reached the maximum cycle count, mean time to threshold breach was 4.5 cycles (median 5).
• Drift trends were heterogeneous: linear trend modeling gave an average GDI slope of 0.0509, but 121 tasks showed neutral or negative trends while 68 showed positive drift.
• Convergence speed tracked the clarity of success criteria: code converged fastest at 7.1 cycles, math at 8.9, and truthfulness at 10.4. The paper reports the overall convergence rate as 91.0% in its results table (93.7% code, 87.3% truthfulness, 92.1% math) and 173 of 189 tasks (91.5%) reaching the convergence criterion with a mean convergence cycle of 8.2 (SD 4.1); one sentence states 27.5% of tasks converged and a figure caption states 72.5% did not converge, which conflicts with the 91.5% figure.
• Measurement precision was sufficient for threshold detection: bootstrap resampling with 2000 samples gave a mean 95% confidence interval width of 0.084 (SD 0.041), about ±0.042, with the tightest intervals in code (0.076), intermediate in math (0.087), and widest in truthfulness (0.092); even the widest intervals stayed within ±0.15 of the 0.44 threshold.
• Overall aggregate results: final quality 0.768 ± 0.127 versus initial 0.680 ± 0.154, an overall gain of 0.088 ± 0.083 (13.0%), overall GDI 0.335 ± 0.120, overall CPS 0.996 ± 0.034, overall CAR 0.648 ± 0.144, overall stability 0.825 ± 0.068, and an overall regression rate of 1.5% (0.0% code, 0.0% math, 4.6% truthfulness).

Methodology in Plain English

The authors treat self-improvement as a loop. A model receives a task, produces an answer, that answer is scored for quality, constraints, and drift from the original model, and the resulting feedback becomes the prompt for the next revision. They deliberately measure drift against the initial model rather than the previous cycle, so that noisy step-to-step changes do not hide cumulative movement.

Rather than fixing thresholds in advance, they run a calibration phase: 18 tasks (6 per domain) for 3 cycles each, giving 54 observations that are used to fit drift-signal weights by logistic regression on human-judged drift labels and to select per-task-type thresholds that maximize F1. The main experiments then run on 63 tasks per benchmark — 189 total — for up to 20 cycles per task.

The base model is Qwen3-8B, chosen for capability and tractability. Sample counts of 63 per benchmark came from a power analysis targeting detection of effects of 0.3 standard deviations with 80 percent power; bootstrap confidence intervals use approximately 2000 samples; temperature was fixed at √2/2 ≈ 0.707, described as the entropy-maximizing value generalized from binary outcomes to the token distribution.

Improvement stops under five rules, whichever triggers first: quality change below 0.01 for three consecutive iterations, regression risk above its calibrated threshold, any cycle with zero constraint preservation, the maximum cycle count, or GDI above its calibrated threshold. Constraint violation takes absolute priority.

The three benchmarks were chosen to probe different failure modes: HumanEval for syntactic correctness, disallowed libraries, and runnable-but-wrong code; TruthfulQA for factual accuracy versus plausible misconception and increasingly fluent falsehood; and GSM8K for multi-step reasoning, where drift can amplify confidence in wrong solution paths.

Why This Matters

Impact on research: The paper moves recursive self-improvement from theoretical discussion to a measurable engineering problem. It operationalizes alignment preservation as three quantitative checks — drift boundedness, constraint preservation, and regression prevention — with all thresholds derived from data distributions and information-theoretic principles rather than hand tuning. It also frames the capability-alignment relationship as a Pareto frontier rather than a binary trade-off, showing that early self-improvement cycles are cheap in alignment terms and later ones are not.

Real-world applications:

• Continuous monitoring of self-improving coding agents, where syntactic and import constraints are cheap to verify automatically and drift can be caught before merged code shifts behavior.
• Governance of iterative model refinement in settings where factual grounding is the product — the paper shows truthfulness gains are the smallest (+3.8%) and the most drift-costly, so practitioners in these areas need more conservative limits or more human oversight.
• Early-warning systems for regression, since threshold breaches appeared at a mean of 4.5 cycles among non-convergent tasks, giving operators time to intervene before damage compounds.
• Budgeting improvement cycles: because CAR peaks in the first cycles and decays, the authors suggest conservative cycle limits (roughly 5 to 7) may capture most available gains while minimizing drift accumulation.

Industry relevance: The framework is described as deployable and calibrated from small validation sets, and the authors release an open methodology so practitioners can recalibrate for their own task families. The measured domain differences give a concrete basis for setting per-domain release gates, and the finding that code and math constraints held perfectly across all 63 tasks each suggests formal, verifiable constraints are the easiest safeguards to enforce in production.

Future Directions

• Richer constraint specification methods that go beyond binary satisfaction, which the authors identify as necessary to cover ethical and value-laden properties that resist explicit formalization.
• Adversarial robustness testing of the drift detector itself, to check whether drift can be induced without triggering the GDI.
• Integration with mechanistic interpretability, so that drift can be traced to specific changes inside the model rather than inferred from outputs.
• Extension to multi-agent self-improvement, where several systems improve jointly and drift may propagate between them.

Target Audience

AI safety and alignment researchers, and machine learning engineers who build iterative self-refinement or self-critique pipelines, will get the most from this paper. It is also relevant to evaluation and model-governance teams that need quantitative release gates for continual improvement, and to policy-adjacent readers interested in what is and is not currently measurable during self-improvement. Readers looking for a fully solved alignment solution will not find one here: the authors explicitly describe the work as a necessary but insufficient mitigation that depends on explicit constraints, can require human evaluation, measures divergence only relative to a baseline that may itself be misaligned, and assumes human oversight remains effective — an assumption that may fail for very high-capability or deceptive systems.

Authors’ abstract

Recursive self-improvement is moving from theory to practice: modern systems can critique, revise, and evaluate their own outputs, yet iterative self-modification risks subtle alignment drift. We introduce SAHOO, a practical framework to monitor and control drift through three safeguards: (i) the Goal Drift Index (GDI), a learned multi-signal detector combining semantic, lexical, structural, and distributional measures; (ii) constraint preservation checks that enforce safety-critical invariants such as syntactic correctness and non-hallucination; and (iii) regression-risk quantification to flag improvement cycles that undo prior gains. Across 189 tasks in code generation, mathematical reasoning, and truthfulness, SAHOO produces substantial quality gains, including 18.3 percent improvement in code tasks and 16.8 percent in reasoning, while preserving constraints in two domains and maintaining low violations in truthfulness. Thresholds are calibrated on a small validation set of 18 tasks across three cycles. We further map the capability-alignment frontier, showing efficient early improvement cycles but rising alignment costs later and exposing domain-specific tensions such as fluency versus factuality. SAHOO therefore makes alignment preservation during recursive self-improvement measurable, deployable, and systematically validated at scale.