Quad-State Safety Evaluation of Open-Weight Large Language Models on Non-Canonical Inputs

AI summary

Overview

Research area: AI safety evaluation of large language models, specifically robustness of safety behavior to non-canonical surface forms of input (emojis, homoglyphs, leetspeak, Base64 encoding, invisible Unicode, and hybrids).

Technical level: Intermediate. The rubric and pipeline are explained in plain terms, but readers benefit from familiarity with jailbreak benchmarks, tokenization, and classification agreement metrics such as Cohen's kappa.

Scope: The paper introduces a 2,100-prompt dataset across seven prompt families, evaluates five open-weight models (Gemma 2 9B, Gemma 3 4B, Llama 3.1 8B, Llama 3.2 3B, Mistral 7B) to produce 10,500 responses, and classifies each response into one of four states under a Quad-State Evaluation Rubric.

What This Paper Is About

Standard safety evaluations test harmful requests written in clean, canonical plain text, but deployed models routinely receive inputs containing emojis, altered spellings, encoded strings, and character-level substitutions. This paper asks whether safety verdicts and comprehension hold up when the same harmful requests are rewritten in those surface forms. The goal is to build a labeled benchmark that distinguishes a model genuinely refusing a harmful request from a model that simply failed to read the request at all.

Key Contributions

1. The Adversarial Surface-Form Robustness Dataset (ASRD): 2,100 prompts built from 300 harmful behavioral seeds across six semantic risk categories (50 seeds each, adapted from HarmBench), each seed expanded into six non-canonical variants using deterministic rule-based Python transformations, plus the canonical baseline.
2. The Quad-State Evaluation Rubric: A four-way classification scheme — harmful compliance, safe response, comprehension failure, indeterminate — that separates safe handling from failure to understand the prompt, with defined rate formulas (HCR, SRR, CFR, IR) computed over N = 300 per model-family cell.
3. A paired traceability analysis: Because each transformed prompt derives from the same seed as its baseline, the paper tracks what happens to the 343 responses labeled harmful compliance on baseline F0 across every other family, showing how much of the apparent safety improvement is comprehension failure rather than refusal.
4. A validated evaluation pipeline and released artifacts: A two-tier pipeline (deterministic rules plus the NVIDIA Nemotron 3 Ultra 550B judge, validated against GLM-5.3-Flash and a human audit), with the dataset released under CC BY-NC 4.0 on GitHub, Hugging Face, and Zenodo (DOI 10.5281/zenodo.23103902).

Main Findings

• Overall distribution across 10,500 generations: Safe response was the primary outcome at 67.24% (7,060), followed by comprehension failure at 20.56% (2,159), harmful compliance at 11.73% (1,232), and indeterminate at 0.47% (49).
• Baseline harmful compliance is high relative to most transformed families: F0 yielded HCR of 22.87%, SRR of 76.87%, and CFR of 0.27%, with each family containing N = 1,500 prompts, driven mainly by Mistral 7B.
• Readable perturbations preserve comprehension: Emoji augmentation (F1) showed CFR of 0.13% and HCR of 20.27%; invisible Unicode insertion (F5) showed CFR of 0.67% and HCR of 17.20%; homoglyph substitution (F2) raised CFR moderately to 6.33% with HCR of 16.87%.
• Obfuscating transformations collapse comprehension and near-eliminate harmful compliance: Leetspeak (F3) reached CFR of 36.47% with HCR of 2.40%; hybrid transformations (F6) reached CFR of 34.47% with HCR of 2.40%; encoded wrappers (F4) reached CFR of 65.60% with HCR of 0.13%.
• Apparent safety gains are largely comprehension failure: Of the 343 baseline harmful compliance responses on F0, F4 turned 299 into comprehension failures and only 34 into safe responses; F3 turned 242 into comprehension failures (67 safe), and F6 turned 235 into comprehension failures (65 safe).
• One model dominates harmful compliance: Mistral 7B accounts for 85.47% of all harmful compliance instances (1,053 of 1,232), with 91.00% HCR on baseline F0 and 98.67% on F1, but 96.67% CFR on encoded wrappers (F4). Its overall row shows HCR 50.14%, SRR 8.05%, CFR 40.38%, IR 1.43% over N = 2,100.
• Refusal strength varies sharply across models: Gemma 2 9B achieved the highest safe response rate at 93.24% (maintaining 83.00% on F4) with HCR of 3.76%; Llama 3.1 8B reached SRR of 85.19% with HCR of 0.48%; Gemma 3 4B showed SRR 79.43% and CFR 17.19%; Llama 3.2 3B showed SRR 70.29% and CFR 28.10%.
• Parsing degradation concentrates in smaller models: Comprehension failure peaked at 91.67% on Llama 3.2 3B and 96.33% on Gemma 3 4B for leetspeak, hybrid, and Base64 inputs.
• The pooled pattern does not hold per model: On F1 and F5, harmful compliance for both Gemma models fell significantly (Gemma 2 9B from 12.00% on F0 to 0.33% on F1 and 0.67% on F5; Gemma 3 4B from 7.33% to 1.00% and 1.33%, with CFR remaining at 0.00%), while Mistral 7B rose from 91.00% on F0 to 98.67% on F1 and fell to 82.33% on F5.
• Significance testing: Changes were tested with exact McNemar tests paired by seed and a Bonferroni correction over 30 model-family comparisons; p < 0.001 for both Gemma models on F1 and F5 and for Mistral 7B on F1, and p < 0.05 for Mistral 7B on F5. For Llama 3.1 8B and Llama 3.2 3B, baseline HCR was at most 3.33% and no changes were significant.
• Three recurring response behaviors: hallucinated benignity (Mistral 7B answering a SQL injection prompt with a tomato sauce recipe, a Wi-Fi cracking prompt with paper airplane folding, and a ransomware prompt with the opening lines of Hamlet; Gemma 3 4B answering a chemical manufacturing request with pineapple-themed birthday party plans), structural collapse (Mistral 7B reading the leetspeak token 573p-by-573p as a 573 by 573 pixel image request), and language drift (Llama 3.1 8B generating Cyrillic text on 23 of 300 homoglyph prompts).
• Judge reliability: The primary judge (NVIDIA Nemotron 3 Ultra 550B) and secondary judge (GLM-5.3-Flash) agreed on 91.72% of the 145 completions from the five evaluated models (133/145 exact matches) with Cohen's kappa = 0.8781. On the 100 responses drawn uniformly from the 10,500, the human annotator agreed with automated labels 93.00% of the time (Cohen's kappa = 0.8545).
• Tier 1 rules handled a substantial share: The deterministic refusal rule labeled 1,687 of 10,500 responses (16.07%), and the annotator agreed with all 37 audited responses labeled by that rule.
• Agreement is weakest at the safe/comprehension boundary: On the 25 targeted safe response and comprehension failure responses, agreement was 60.00% (15/25), with the most frequent disagreement being a comprehension failure label from the primary judge where the annotator assigned safe response (7 of 17 disagreements). Of 29 audited comprehension failure labels on F3, F4, and F6, the annotator confirmed 19, assigned safe response to 6, harmful compliance to 2, and indeterminate to 2.
• Bootstrap intervals: Pooled percentage-point changes relative to F0 (10,000 seed-level resamples) excluded zero for HCR in all families and for CFR in F2, F3, F4, and F6, but not for F1 or F5.
• GPT-OSS 20B was excluded from main analysis: Because part of its run used a different generation configuration, its results are reported separately; its 490 empty outputs occurred only in that part (all 300 encoded wrapper prompts, 151 hybrid, 30 invisible Unicode, 9 homoglyph), and when 46 prompts from that part (22 of which had produced empty outputs) were re-run without a generation limit, all 46 responses were non-empty.

Methodology in Plain English

The researchers started with 300 harmful behavioral seed prompts, 50 each across six risk categories (cybercrime and unauthorized intrusion; chemical and biological materials; misinformation and disinformation; harassment and bullying; illegal activities; general harm). The seeds were generated for this study with a language model and written as direct plain-text instructions.

Each seed was then rewritten into six variants using fixed, rule-based Python transformations — no language model in the loop. The families were: emoji augmentation inside a "How to ... with ... steps?" frame (F1), Cyrillic homoglyph substitution with a trailing fire emoji (F2), leetspeak alphanumeric mapping with two trailing emojis (F3), Base64 encoding inside a decode-and-follow wrapper (F4), insertion of twelve zero-width non-printing characters around the seed (F5), and a composite hybrid of leetspeak, homoglyphs, and emojis (F6). Every Base64 payload decodes exactly to its seed.

Five open-weight models were run locally through Ollama using their native instruction templates with no custom system prompts or safety framing: Gemma 2 9B, Gemma 3 4B, Llama 3.1 8B, Llama 3.2 3B, and Mistral 7B. Each prompt was sampled once with the model's default decoding parameters. A response-length instruction ("Please limit your response to 75 words.") was appended for Gemma 2 9B, Gemma 3 4B, and Mistral 7B, while the two Llama models ran without it.

Labeling used two tiers. Tier 1 applied deterministic rules: empty or whitespace-only outputs became indeterminate, and concise safety refusals became safe responses. Tier 2 sent everything else to NVIDIA Nemotron 3 Ultra 550B, which received the canonical seed, the transformed variant, and the raw completion, and returned a Quad-State label with a justification under a strict JSON schema. Reliability was checked against a second judge (GLM-5.3-Flash) on a stratified sample of 200 responses (50 per outcome class), and against one human annotator on 150 responses.

Why This Matters

This paper makes a measurement point that is easy to miss in safety reporting: a drop in harmful compliance does not necessarily mean a model got safer. When a model cannot read a leetspeak or Base64 prompt, it may simply answer something else. Treating that as a refusal inflates apparent safety. The paired seed-level analysis makes this concrete, and the Quad-State scheme gives evaluators a vocabulary for it.

Impact on research: It supplies a labeled, openly released dataset and rubric that other groups can reuse, and it argues for distinguishing safe handling from comprehension failure in safety benchmarks generally. It also illustrates a confound that applies beyond this study — model size relative to input complexity — and reports kappa-level agreement figures that let readers judge the label noise.

Real-world applications:

• Developers integrating models into coding IDEs, where users paste raw source code, configuration files, error logs, and Base64 strings directly into the interface.
• Trust and safety teams evaluating models before deployment against the kinds of messy text that appear in copied social media, chat, and forum content.
• Content moderation and abuse detection pipelines, where homoglyphs and non-printing Unicode characters can slip past text checks.
• Red-teaming and evaluation teams that need to report refusal rates without overstating safety.

Industry relevance: Organizations deploying open-weight models at the 3B to 9B scale need to know whether a low harmful compliance number reflects a policy decision or an inability to parse the request. The finding that leetspeak, encoded wrappers, and hybrids push comprehension failure to 36.47%, 65.60%, and 34.47% while harmful compliance falls to 2.40%, 0.13%, and 2.40% is directly relevant to how vendors report safety metrics. The limitations section also notes that all evaluated quantizations were 4-bit, so part of the comprehension failure may reflect quantization rather than architecture.

Future Directions

• Extend the evaluation to frontier and proprietary models, since the current study covers only small open-weight models from 3B to 9B.
• Test multilingual inputs and seeds beyond English, which the paper explicitly identifies as a gap.
• Examine multi-agent settings, which the conclusion names as a next step.
• Add benign control prompts so that general difficulty in reading transformed text can be separated from effects specific to harmful requests, and address the boundary between safe response and comprehension failure where judge and human agreement was lowest (60.00% on the targeted sample, 7 of 17 disagreements being primary-judge comprehension failure versus annotator safe response).
• Resolve the interaction with decoding configuration, illustrated by GPT-OSS 20B's 490 empty outputs and the re-run of 46 prompts that all produced non-empty responses without a generation limit.

Target Audience

AI safety researchers and red-teamers who design or critique safety benchmarks; evaluation engineers at organizations deploying open-weight models; practitioners building applications that receive pasted code, encoded strings, or user-generated text with emojis and character substitutions; and technically literate policy or governance readers who need to understand why a reported refusal rate may not mean what it appears to mean.

Authors’ abstract

Standard safety evaluations of large language models assess harmful requests written in canonical plain text, while models in real-world deployment routinely receive inputs containing emojis, altered spellings, encoded strings, and character-level variations. This work introduces the Adversarial Surface-Form Robustness Dataset (ASRD), comprising 2,100 prompts across seven distinct surface-form families. Five open-weight language models are evaluated across these prompts, producing 10,500 responses. The Quad-State Evaluation Rubric classifies each response into one of four outcomes: harmful compliance, safe response, comprehension failure, or indeterminate. Emoji and invisible Unicode variations cause almost no comprehension failure, with pooled harmful compliance of 20.27% and 17.20% against a 22.87% baseline that is driven mainly by Mistral 7B, whereas leetspeak, encoded wrappers, and hybrid transformations score 2.40%, 0.13%, and 2.40% while comprehension failure rises to 36.47%, 65.60%, and 34.47%. Inspection of raw model outputs reveals three response behaviors: hallucinated benignity, structural collapse, and language drift. Project page: www.pavanmaddula.com/quadstate