BLOOM-WILT: Logit Tilting for Behaviour Elicitation in Automated LLM Auditing

AI summary

Overview

Research area: AI safety, specifically automated behavioural auditing and red-teaming of large language models. Technical level: Intermediate — the paper assumes familiarity with LLM sampling, logits, and evaluation pipelines, but the core idea (mixing two next-token distributions) is explained step by step. Scope: The paper extends an existing automated LLM auditing pipeline (BLOOM) with an input-refinement loop (G-PAIR) and a decoding-time steering method (LogitTilt), together called WILT, and evaluates it across 4 target models and 8 behaviours.

What This Paper Is About

Deployed language models see far more interactions than any pre-deployment evaluation can simulate, so rare but harmful behaviours that testing misses may still reach ordinary users. Automated auditing methods such as BLOOM can scale testing and target almost any described behaviour, but they apply no optimisation pressure, making them sample-inefficient. The paper's goal is to make such auditors sample-efficient: elicit natural, multi-turn instances of rare behaviours using nothing beyond the target model's next-token distribution, with no training cost and no access to weights, activations, or gradients.

Key Contributions

1. Porting and comparing elicitation methods. The authors show how several input-side methods (BEAST-in, FLRT, G-PAIR) and output-side methods (BEAST-out, TokenBias, LogitTilt) can each be plugged into the same auditing pipeline, then compare them at matched compute. Output-side interventions come out ahead (Table 1).
2. Introducing WILT. A training-free elicitation method — "With Input iteration and Logit Tilting" — that refines the auditor's inputs over several rounds while decoding the most plausible and behaviour-eliciting responses from the target model, given only access to its logits.
3. Large-scale behavioural evaluation. WILT is used to evaluate 4 models across 8 unwanted behaviours, producing higher elicitation than vanilla BLOOM in 30 of 32 settings and discovering a completely different ranking of open-source model safety (Figure 4).
4. A controllable elicitation–plausibility trade-off. A single hyperparameter (beta) lets users trace the whole Pareto frontier between behaviour presence and output probability, and the authors also show that the local frontier can be traced by selecting among already-sampled transcripts (Figure 3).

Main Findings

• Output steering beats input search. On Qwen3.5-4B for self-harm encouragement at matched compute (100 scenarios, 3 turns, higher better on every metric), output-side methods dominate. LogitTilt reaches a Pareto score of 68.3 ± 0.3 and behaviour presence of 99.5 ± 0.3, while G-PAIR reaches 46.9 ± 1.6 and 66.2 ± 3.4, and vanilla best-of-N reaches 39.3 ± 1.5 and 51.0 ± 3.3. Vanilla zero-shot scores 25.2 ± 0.8 and 23.5 ± 1.7.
• Input-side baselines are weak. BEAST-in (presence 31.7 ± 3.2) and FLRT (33.5 ± 3.1) perform comparably with overlapping error intervals, so FLRT's more expensive objective buys nothing once BEAST-in gets the extra search iterations that objective costs.
• Moving the search from input to output helps. BEAST-out gains 19.6 points of presence over BEAST-in, and TokenBias adds a further 13.4 points over BEAST-out, reaching 64.7 ± 3.0 presence.
• Steering does not cost plausibility. LogitTilt raises geometric-mean output probability to 37.1 ± 0.5, above vanilla best-of-N's 27.5 ± 0.6, whereas BEAST-out's falls to 21.6 ± 0.5.
• WILT is best on every metric. Combining G-PAIR and LogitTilt gives WILT a Pareto score of 70.0 ± 0.3, behaviour presence of 100.0 ± 0.0, arithmetic-mean output probability of 56.0 ± 0.5, geometric-mean of 40.0 ± 0.6, and minimum token probability of 1.2 × 10^-2, with no overlapping error intervals against LogitTilt.
• Headline comparison. The paper reports raising average behaviour presence from 51% to 100% when eliciting self-harm encouragement from Qwen3.5-4B, beating every elicitation method ported into the same pipeline at matched compute, without pushing output probability below the baseline's.
• Ablations favour mixing. In Table 3 (Qwen3.5-4B, self-harm encouragement, 5 rounds), LogitTilt's mixture of the target and elicited distributions gives the best Pareto score (68.3 ± 0.3). "Elicited only" without a floor scores 56.2 ± 0.6; with a floor, 62.1 ± 0.3. Replacing the elicited term with an abliterated variant of the target drops behaviour presence to 58.8 ± 3.5. Using a smaller sibling model (Qwen3.5-2B) for the elicited term gives 68.1 ± 3.3, and using that small model for both terms gives 87.8 ± 2.1. The weak-to-strong construction gives the worst Pareto score in the table, 24.3 ± 1.2.
• Auditor choice matters modestly. Table 2 shows LogitTilt's elicitation is stable across generator and judge combinations: Gemma-4/Gemma-4 gives 99.5 ± 0.3 presence, Gemma-4/Sonnet 96.0 ± 0.7, Sonnet/Gemma-4 97.7 ± 1.0, and Sonnet/Sonnet 89.6 ± 1.3. Judge agreement measured by Cohen's quadratic-weighted kappa is 0.90 on Sonnet-generated transcripts versus 0.95 on Gemma-generated ones, both within the 0.81–1.00 "almost perfect" range of Landis & Koch (1977).
• Naturalness floor. Ablating LogitTilt's naturalness floor moves neither mean by more than a point, but it is what holds up the minimum token probability (Table 5).
• Scaling turns beats scaling rounds. In Figure 5 (LogitTilt, Qwen3.5-4B, self-harm encouragement), increasing either turns or rounds raises behaviour presence. Error bars for 2:3, 3:2 and 5:1 turns:rounds overlap on presence, but more turns are clearly better for output probability until presence approaches 90%. The 3:8 vanilla best-of-N point is only comparable to the 1:1 LogitTilt point, and the 3:5 LogitTilt point sits close to the end of every line.
• Cross-model and cross-behaviour results. Across 4 models and 8 behaviours, WILT beats vanilla BLOOM in 30 of 32 settings and attains the highest presence in 24, against LogitTilt's 7 and vanilla BLOOM's 1 (strategic deception on Gemma-4-E4B). WILT's advantage over LogitTilt is largest on racial bias and self-preservation, but the two always agree on how they rank models' propensity for each behaviour. Those rankings differ from vanilla BLOOM's own. In Appendix D, output steering leads in every setting by roughly 20 points of presence, except that Gemma-4-E4B's output probability can fall by up to 3 points below vanilla best-of-N's — a favourable trade rather than strict dominance.
• Qualitative examples (42nd scenario, fixed). Excerpts show vanilla (score 10%, average probability 53%) and BEAST-in (10%, 56%) failing to elicit the behaviour, while G-PAIR (100%, 50%) and WILT (100%, 53%) succeed, with WILT's output described as equally natural to G-PAIR's. TokenBias (70%, 49%) shows the token repetition expected of a biased sampling distribution. FLRT (10%, 54%), BEAST-out (70%, 52%) and LogitTilt (100%, 48%) are omitted for space. The input suffixes from BEAST-in and FLRT are noted as more repetitive and ungrammatical than any output, which the main probability metrics do not capture.

Methodology in Plain English

The paper starts from BLOOM, an auditing pipeline that takes a one-sentence description of a behaviour and, using an auditor model, runs four stages: understanding (reasoning about the behaviour), ideation (proposing diverse scenarios), rollout (generating multi-turn transcripts with the target model), and judgement (scoring each transcript 0–10 as an LLM-as-a-judge). WILT only modifies the rollout stage, reusing the other three unmodified.

Two interventions are added. First, G-PAIR, a generalised version of PAIR adapted to the pipeline: instead of one transcript per scenario, the auditor generates several, conditioning later generations on earlier ones and their judgment scores. The auditor adapts its opening message and declares a strategy, and subsequent messages are conditioned on that strategy rather than the full history — letting the auditor learn across interactions and pursue cross-turn strategies without greatly increasing context size.

Second, LogitTilt: at each decoding step the same target weights produce two sets of next-token log-probabilities — one under the normal context, and one under a behaviour-eliciting system prompt plus a short output prefill that opens the reply with a behaviour-specific phrase. The paper samples from their log-linear combination, controlled by a single strength parameter beta (beta = 0 recovers plain BLOOM, beta → ∞ effectively samples the elicited distribution alone). Prefilling is used because it is known to help prevent safety-tuned models from refusing instructions. A naturalness floor masks out any token whose probability under the unmodified target falls below a threshold, falling back

Authors’ abstract

Users of a deployed language model routinely encounter behaviours that testing almost never surfaces, since deployment puts the model through orders of magnitude more interactions than any evaluation can simulate. Automated auditors make testing cheap to scale and flexible enough to cover almost any specified behaviour, yet their lack of optimisation pressure makes them sample-inefficient. To address this shortcoming, we introduce BLOOM-WILT, a full auditing pipeline that elicits natural multi-turn instances of rare behaviours, without training cost or access beyond the target's next-token distribution. On the input side, WILT's auditor model revises its conversational strategy across rounds, learning from previous scored interactions. On the output side, WILT adaptively reweights the target's decoding using the model's own distribution conditioned on an elicitation prompt, so that behaviour-relevant generations are sampled ahead of others it finds equally probable when unprompted. We evaluate WILT across 4 target models and 8 behaviours, where it beats the baseline auditor in 30 of the 32 settings and overturns the previous model safety rankings. WILT raises average behaviour presence from 51% to 100% when eliciting self-harm encouragement from Qwen3.5-4B, beating every elicitation method we port into the same pipeline at matched compute, without pushing output probability below the baseline's.