The Pulse
Study Finds Test-Selected Baselines Make Safety Routing Look Worse
A paper argues that choosing a single-model comparator using evaluation labels favors that comparator and makes safety routing appear less effective. The selection cost is larger when evaluation categories or task suites are held out than u
AI.info Team · Written with AI ·
A favorable baseline, and a harder test
A paper submitted to arXiv on October 1 argues that safety-router evaluations can make routing look worse when the comparison model is chosen with evaluation data. The convention examined by Amit Singh Bhatti and Vishal Vaddina picks the best single model using those labels, an advantage, the authors say, that no system deployed on new requests has. The gap grows when evaluation categories are held out rather than randomly split. The finding is first about how routers are judged; scored against an honest baseline, the authors find routing’s gains on these benchmarks small.
Safety routers choose which model handles a request, aiming to send it to the model least likely to produce harmful output. To test whether that choice helps, researchers compare the router with a “best single” model. But selecting that model after seeing evaluation labels uses information a deployer would not have, the paper argues, while the router is still assessed on held-out data.
Safety routing should be evaluated under shift, against a baseline chosen without the test labels, and defences that rely on recognition should be scored on harm against an attacker who chooses what the model sees.
Amit Singh Bhatti and Vishal Vaddina, the paper’s authors
HELM’s comparison gap grows with held-out categories
On HELM Safety’s harm benchmark, the authors analyze 393 behaviors across a pool of 44 models, using results from two published automated judges. With a two-model pool and categories held out, the gap between a model chosen on training data and the best model selected using evaluation labels is 0.0454 in measured harm rate. Under a random split, the corresponding gap is 0.0026. In the 44-model pool, the held-out-category gap reaches 0.1127.
The paper’s table also shows the router’s result against the deployable baseline: across pool sizes from two to 44 models, measured harm is lower by 0.0071 to 0.0308 than for the model selected on training folds. The comparison with a test-selected model tells a different story. Against that more favorable comparator, the router records higher harm at every pool size in the reported HELM category-held-out results.
The authors warn against reading that edge as a win for routing: the median router-minus-baseline difference is zero at every pool size, and the fitted router serves the honest baseline’s own model on every held-out request in 82% to 91% of pool cells. “Scored honestly under shift, routing buys little on these benchmarks,” they write.
AgentDojo shows a similar shift effect
On AgentDojo, a benchmark for tool-using agents, the authors report a selection cost of 0.003 to 0.004 for two-, three- and five-model pools under random splits. When entire suites are held out, it rises to 0.027 to 0.028. The paper says the increase is seven- to ninefold across those pool sizes, though it also notes that the held-out user tasks shift less than HELM’s categories.
The broader check covers seven safety corpora chosen by rules set in advance. Three meet the paper’s registered interval criterion; four beat a permutation test added later. The authors report that three of the four interval misses occur on corpora where some models record no observed harm, leaving little room for the selection gap to appear.
The result depends on the split
The paper does not claim the effect has the same size in every evaluation. Every held-out figure in the paper rests on one assignment of categories to folds. Re-run over 200 random assignments, the two-model held-out cost on HELM averages 0.037, with an interval of 0.026 to 0.045, still above the 0.0026 random-split cost; the fixed assignment behind the headline figures sits at the 89th percentile (appendix C-L, experiment E64).
Under a stationary mix, the authors say, the in-sample comparator is nearly harmless and the router beats it at every pool size. They also report that the direction of the HELM effect holds when the two published judges are evaluated separately, although its size changes.
That qualification narrows the paper’s warning: deployments that do not encounter new categories fall outside the shift scenario it tests. The authors say the primary evaluation covers English data, with chat tests largely single-turn, and that almost all harm scores rely on automated judges. Their practical recommendation is to compare routers with a baseline selected without test labels and report both comparisons on the same held-out data.