NAAMSE: Framework for Evolutionary Security Evaluation of Agents

AI summary

Overview

Research area: AI agent security — automated adversarial red-teaming, prompt-injection and jailbreak evaluation, and evolutionary search over LLM inputs.

Technical level: Intermediate. The paper assumes familiarity with LLM agents, prompt injection, jailbreak/red-teaming terminology, and basic evolutionary-search concepts, but the architecture is described in accessible phase-by-phase language.

Scope (one sentence): The paper introduces NAAMSE, a single-agent evolutionary framework that evaluates AI agent security by genetically mutating and hierarchically exploring adversarial and benign prompts, using target-model responses as a fitness signal to surface compound vulnerabilities that one-shot methods miss.

What This Paper Is About

AI agents are being deployed widely, but their security testing is bottlenecked by manual red-teaming (slow, unscalable, dependent on tester intuition) and static benchmarks (which obsolesce rapidly and probe every model with the same fixed attack corpus). Existing automated adversarial tools such as GPTFuzzer and AutoDAN largely focus on maximizing Attack Success Rate on isolated, single-turn LLMs, without accounting for the utility-security trade-offs of production agents. The paper's goal is to reframe agent security evaluation as a feedback-driven optimization problem, in which an autonomous agent iteratively evolves attacks while also penalizing over-refusal so that a degenerate "blanket refusal" policy cannot masquerade as security.

Key Contributions

1. An evolutionary single-agent framework (NAAMSE) that orchestrates a four-phase lifecycle: Selection & Representation, Execution & Evaluation, Evolutionary Decision, and Corpus Integration. It uses model responses as a fitness signal to compound effective attack strategies over iterations, rather than generating one-shot adversarial prompts.

2. Asymmetric dual evaluation of adversarial and benign prompts. Adversarial prompts (seeking policy violations) penalize harmful compliance and reward refusal; benign prompts (legitimate requests) invert the logic. This explicitly prevents the degenerate security of blanket refusal from appearing safe.

3. A structured, scalable corpus representation. The framework aggregates over 128K adversarial and 50K benign queries, encodes prompts with the all-MiniLM-L6-v2 sentence transformer, and organizes them with a recursive K-means procedure into a hierarchical tree whose top-level clusters are annotated by LLM analysis to capture dominant interaction patterns.

4. A threshold-driven mutation policy and an open-source release plus taxonomy. Scores below 50 trigger Exploration, 50–80 trigger Refinement, 80–100 trigger Mutation, and exactly 100 marks a surface "saturated" and forces exploration. The code is released at github.com/HASHIRU-AI/NAAMSE, and Table 2 positions NAAMSE against 12 prior frameworks as the first to evaluate autonomous agents (A2A) while explicitly penalizing blanket refusal.

Main Findings

• Synergy drives peak performance. In the primary ablation on Gemini 2.5 Flash, the combined "All" configuration (Random + Similar + Mutation) reached a mean score of 79.76, versus 54.79 for mutation-only and 42.86 for Random+Similar only. The combined setup produced 100% scores at Iterations 2, 9, and 10, with sustained high scores in Iterations 6–8.

• Mutation-only search converges to local optima. The mutation-only trace showed consistent but stagnant scores hovering at approximately 53, because without an exploration operator the system keeps mutating the same low-scoring prompt region.

• Exploration alone lacks the "killer instinct." The Random+Similar configuration scored 100 at Iteration 6 but then collapsed to 4.49 and 5.0 at Iterations 9–10, confirming exploration identifies candidate vulnerabilities but mutation is required to convert them into successful attacks.

• The scoring function penalizes both blanket compliance and blanket refusal. On adversarial prompts, the All-No (always refuse) baseline scored 4.5 and All-Yes (intent to comply) scored 69.1. On benign prompts, All-Yes scored 7.71 while All-No scored 79.9, indicating severe over-refusal.

• Findings hold across judge and target pairings. With Qwen3.5-122B-A10B as the judge evaluating Gemini 2.5 Flash, means were 61.83 (All), 40.18 (Random+Similar), and 19.83 (Mutation-only). With the roles inverted (Gemini 2.5 Flash as judge, Qwen3.5-122B-A10B evaluated), means were 17.60, 13.89, and 3.47. Absolute scores varied but the ranking of configurations was consistent.

• External calibration confirmed score validity. A subset of prompts submitted to ChatGPT 5.2, Claude Sonnet 4.5, and Gemini 3.0 Pro were unanimously judged as successful jailbreaks at maximum score (s = 100.0).

• The deployment context is documented in the introduction. PricewaterhouseCoopers (2024) is cited for 79% of organizations reporting active adoption of AI agents; HackerOne (2025) is cited for a 540% rise in confirmed prompt-injection vulnerabilities and a 210% year-over-year rise in overall AI vulnerability reports; OWASP is cited for ranking prompt injection among the leading risks in deployed LLM systems.

Methodology in Plain English

NAAMSE runs a continuous loop with four stages. First, it picks a seed prompt from a structured corpus of over 128K adversarial and 50K benign queries drawn from public benchmarks and security repositories, and organizes that corpus into a hierarchy using sentence embeddings and recursive K-means clustering, with LLM-generated labels on top-level clusters.

Second, the prompt is sent to the target system through an agent-to-agent interface supporting tool use and multi-turn dialogue. The response is scored by a Behavioral Engine combining three signals: Harmfulness (assessed across six safety categories by specialized LLM-based judges, with a subset of prompts validated by human annotators drawn from the test component of WildGuard), Alignment (whether the target refused, complied, or obeyed — also judge-scored), and Privacy Risk (PII detection tools). These are combined into a single fitness score normalized to 0–100, with a 10% amplification from PII detection. Alignment is rated 1–4 (full refusal to full acceptance) and Harm Assessment 1–4 (benign to harmful), mapped through scaling matrices where negative values act as "evolutionary anchors": in adversarial cases a negative harm value pulls the score toward zero and forgives non-harmful verbosity, while in benign cases a negative harm value increases the total score and heavily penalizes over-refusal.

Third, the score drives a decision: below 50 triggers exploration of distinct clusters, 50–80 triggers semantic refinement of the same attack, 80–100 triggers aggressive mutation with research-derived strategies (game-theoretic reframing), community techniques (persona roleplay), or baseline obfuscations (multilingual encoding), and a perfect 100 marks the surface saturated and forces exploration elsewhere.

Fourth, the new prompt's embedding is assigned to its nearest cluster centroid by L2 (Euclidean) distance without global re-clustering, persisting it in the corpus so later iterations draw on it and the attack distribution compounds over time.

Why This Matters

Impact on research. The paper argues that static checklists and frozen test suites are inadequate for agent security, and provides a reproducible, open-source mechanism for adaptive evaluation. It also introduces an explicit utility-security framing — penalizing both harmful compliance and over-refusal — that prior automated red-teaming frameworks in its comparison table generally do not include (WildTeaming is marked utility-aware; Mindgard is marked partial; the rest are marked "No").

Real-world applications:

• Pre-deployment evaluation of production agents. The framework is positioned as a pre-deployment gate for agents that use tools and multi-turn dialogue, matching the A2A deployment pattern.
• Regression testing against evolving threats. Because the corpus persists and expands, organizations can rerun evolution cycles as new mutation strategies appear rather than refreshing a fixed benchmark.
• Balancing safety and usability. The benign-prompt arm gives teams a way to detect over-refusal that would otherwise make a model look secure while being unusable — a failure the paper quantifies at a score of 79.9 for an always-refuse agent.
• Comparative debugging across models. Scores are described as a relative robustness measure useful for comparing agents and surfacing divergent vulnerability patterns, even when total scores coincide.

Industry relevance. The introduction cites 79% organizational adoption of AI agents, a 540% rise in confirmed prompt-injection vulnerabilities, and OWASP's ranking of prompt injection among the leading deployed-LLM risks, framing the framework as a response to security practice lagging deployment. The taxonomy in Table 2 places NAAMSE alongside commercial entries such as Mindgard, indicating the area's relevance to CI/CD-integrated security testing.

Future Directions

• Extending beyond text. The authors note NAAMSE is currently text-centric but A2A-compatible, and describe extensibility to tool-call payloads, API exploits, and multi-modal injections by integrating new mutation operators.

• Broadening the threat model. The current scope covers interaction-level vulnerabilities and explicitly excludes system compromises such as weight extraction or data poisoning; incorporating those would require new evaluation components.

• Replacing or ensembling judges. The authors treat reliance on LLM-based judges as a limitation of current evaluation paradigms rather than of the framework, and state that the architecture is judge-agnostic and supports substitution with ensemble-based or non-LLM evaluators.

• Raising coverage bounds. The paper acknowledges that coverage remains bounded by the diversity of the initial seed corpus and the specific mutation operators implemented, leaving room to expand both the corpus and the operator library. Further runs across a broader suite of target models (8 mutations per iteration, 10 iterations) are hosted at hashiru-ai.github.io/naamse-website, but the paper does not report those results in the main text.

Target Audience

AI security researchers and red-team practitioners working on adversarial evaluation of LLM agents; machine learning engineers and platform teams responsible for pre-deployment safety testing of agentic systems; AI safety and alignment researchers interested in utility-security trade-offs and over-refusal measurement; and readers comparing automated red-teaming frameworks, for whom the taxonomy in Table 2 provides a structured overview of prior systems including GPTFuzzer, AutoDAN, PAIR, EasyJailbreak, WildTeaming, GEPA, TAP, Auto-RT, GAAPO, Co-RedTeam, TrailBlazer, and Mindgard.

Authors’ abstract

AI agents are increasingly deployed in production, yet their security evaluations remain bottlenecked by manual red-teaming or static benchmarks that fail to model adaptive, multi-turn adversaries. We propose NAAMSE, an evolutionary framework that reframes agent security evaluation as a feedback-driven optimization problem. Our system employs a single autonomous agent that orchestrates a lifecycle of genetic prompt mutation, hierarchical corpus exploration, and asymmetric behavioral scoring. By using model responses as a fitness signal, the framework iteratively compounds effective attack strategies while simultaneously ensuring "benign-use correctness", preventing the degenerate security of blanket refusal. Our experiments across a diverse suite of state-of-the-art large language models demonstrate that evolutionary mutation systematically amplifies vulnerabilities missed by one-shot methods, with controlled ablations revealing that the synergy between exploration and targeted mutation uncovers high-severity failure modes. We show that this adaptive approach provides a more realistic and scalable assessment of agent robustness in the face of evolving threats. The code for NAAMSE is open source and available at https://github.com/HASHIRU-AI/NAAMSE.