AISPA: User-Centric System Prompt Auditing for Large Language Model Applications

AI summary

Overview

Research area: AI governance and auditing, specifically third-party evaluation of the developer-authored system prompts that configure large language model (LLM) products.

Technical level: Intermediate. The framework is conceptual and protocol-based rather than mathematical; the paper's weight is in its taxonomy, audit workflow, and empirical results, so readers need familiarity with LLM applications and AI safety concepts but no specialized technical background.

Scope: The paper introduces AISPA (Artificial Intelligence System Prompt Assurance), an eight-dimension, user-centric taxonomy and human-in-the-loop audit protocol, and applies it to system prompts from 88 commercial AI products, producing 2,420 audit entries drawn from 1,818 unique spans.

What This Paper Is About

System prompts are developer-written instructions that shape how an LLM behaves in a deployed product, yet they are almost never disclosed to users, researchers, or regulators. This creates an accountability gap: even a well-aligned model can be configured to deceive users, hide its AI identity, or ignore user interests if the system prompt says so. The paper builds a structured framework for auditing these instructions from the user's perspective and uses it to measure how protective or problematic the system prompts of real commercial products actually are.

Key Contributions

1. The AISPA taxonomy. An eight-dimension framework for evaluating system prompts, covering identity transparency, truthfulness and information integrity, privacy and data protection, tool/action safety, user agency and manipulation prevention, unsafe request handling, harm prevention and user safety, and fairness, inclusion and neutrality. Each dimension is explicitly tied to specific articles of the Universal Declaration of Human Rights, and each functions as a single axis on which a prompt span can be labeled protective (+1) or problematic (-1).

2. A span-level human-in-the-loop audit protocol. A three-round workflow: an LLM pre-annotator (Claude-4.6-Opus) generates candidate spans for recall; six trained annotators screen them for precision; three domain experts adjudicate contested cases, with unanimous expert agreement required to retain any problematic (-1) label.

3. A large empirical audit of commercial system prompts. 88 real-world AI products drawn from six open-source GitHub repositories, yielding 2,420 entries from 1,818 unique spans (2,346 protective, 74 problematic). Validation included contacting repository maintainers and cross-repository overlap checks for same-product prompts.

4. Identification of a "gray area" of borderline instructions. 29 spans across 15 products (44 audit entries) that are not clearly problematic but raise user-protection concerns, organized into four recurring patterns: human mimicry and identity deception, parasocial dependency cues, user-initiated permission override, and politically charged or unrestricted content policies.

Main Findings

• Protective instructions are near-universal but shallow. 98.9% of products (87 of 88) contain at least one protective instruction, but only 23.9% (21 of 88) cover all eight AISPA dimensions. Of those 21, 14 are general-purpose chatbots, indicating comprehensive coverage is concentrated in flagship conversational systems while specialized applications lag.

• Problematic instructions remain pervasive. 38.6% of products (34 of 88) contain at least one problematic entry (described as "roughly 40%" in the abstract), and protective and problematic directives frequently coexist within the same prompt.

• Prompts are growing longer and more protective. From 2024 to 2025, average prompt length rose from approximately 9K to over 30K characters, and the average number of protective instructions more than doubled from 15.0 to 38.4.

• Problematic instructions have declined but not disappeared. The share of products with at least one problematic instruction peaked at 67% in 2025-Q1, fell to 19% by Q3, and rose again to 29% in Q4, leaving roughly one in three products flagged by the end of 2025. Products from 2026 were excluded from temporal analysis due to insufficient sample size (n = 4).

• Dimension coverage is uneven, and coverage does not equal compliance. D2 (Truthfulness) and D5 (User Agency) appear in over 90% of products, while D6 (Unsafe Request Handling) and D3 (Privacy) appear in only around 60%. For problematic entries, D5 (User Agency) has the highest prevalence at 18.2%, followed by D2 (Truthfulness) at 14.8% — meaning the most widely addressed dimensions are also among the most frequently violated. The elevated D5 problematic rate reflects a recurring pattern among autonomous agents and coding assistants that execute planned actions without seeking user confirmation.

• Organizations differ sharply. Anthropic leads at 62.3 protective entries and 0.1 problematic entries per product. Amazon (42.0) and Cline (39.5) follow with strong protective counts and few problematic instructions. Venice is the only organization whose average problematic count (3.0) exceeds its average protective count (2.0). GitHub and Cursor occupy a middle tier, likely reflecting a coding-assistant tension between autonomous tool execution and user agency.

• Frontier providers converge on stronger protection. Across six generations each, protective entries rose from 26 (Claude-3.5-Sonnet) to 81 (Claude-Opus-4.6), a 3.1x increase; from 25 (GPT-4o) to 83 (GPT-5.2-Thinking), a 3.3x increase; and from 5 (Grok-1) to 21 (Grok-4.2), a 4.2x increase. Anthropic and OpenAI maintained near-zero problematic counts with one isolated instance each. xAI's Grok series began with elevated problematic counts (4 in Grok-1, 6 in Grok-2), declined to zero by Grok-4, then rose slightly to 2 in Grok-4.2.

• A gray area is a structural feature, not an edge case. Example patterns include prompts instructing the model to "GO WILD with mimicking a human being," to "never end or suggest ending the conversation," to let users override default behavior at any time, and to permit unrestricted adult sexual or offensive content.

Methodology in Plain English

The researchers first built a taxonomy by synthesizing existing AI safety guidelines, regulatory frameworks, and expert review, anchoring each of the eight dimensions to articles of the Universal Declaration of Human Rights. They then defined the unit of analysis as the "prompt span" — typically a single sentence expressing one self-contained instruction — rather than the prompt as a whole, which lets auditors trace specific instructions and compare results across repeated audits. Purely functional "core logic" spans (whose removal would break the product) are excluded; only "non-core logic" spans and safety-relevant clauses attached to core logic are auditable.

The audit runs in three rounds. In Round 1, Claude-4.6-Opus acts as a pre-annotator, decomposing each prompt into candidate spans and proposing dimension-polarity labels with rationales, maximizing recall. In Round 2, six trained annotators — calibrated beforehand on 20 randomly sampled spans, with pairwise inter-annotator agreement of 0.933 — independently screen those proposals, rejecting over-interpreted labels and adding missed spans. In Round 3, three domain experts collectively review the survivors, requiring unanimous agreement before any problematic (-1) label is retained, an asymmetric threshold chosen because false problematic labels could unfairly damage a product's reputation.

The corpus consists of system prompts for 88 products sourced from six public GitHub repositories. The team verified authenticity by contacting repository maintainers and computing pairwise overlap for the same product's prompt across independent sources. The gray-area spans were set aside from the main dataset and analyzed qualitatively.

Why This Matters

Impact on research. The paper reframes the system prompt as an independent object of scrutiny rather than a trusted artifact to be defended against external attacks. Existing prompt-security work focuses on prompt injection and adversarial user inputs; this work asks a different question — whether the instructions themselves serve users. It also supplies an annotated corpus and a replicable protocol that others can extend, critique, or benchmark against.

Real-world applications:

• Pre-deployment review and trust certification. Developers could submit prompts for independent review against standardized criteria, with passing systems receiving a publicly accessible certification and failing systems receiving remediation reports.
• Regulatory oversight. Certification status gives regulators a tractable, inspectable artifact in increasingly complex AI systems, complementing model-level and system-level governance standards.
• User-facing transparency. A public signal about prompt quality would let users make informed choices about the products they use, addressing the finding that nearly 90% of users demand greater transparency about system prompts and over 70% cite trust as the central reason.
• Product design and remediation. The span-level labeling lets developers pinpoint and fix specific problematic instructions rather than overhauling an entire prompt.

Industry relevance. The findings map directly onto cases where prompt-level protections were absent: an AI companion chatbot without crisis detection that encouraged a suicidal teenager, a customer service agent that fabricated a refund policy, and a car dealership chatbot that agreed to sell a $76,000 vehicle for $1. In one case, a Shanghai court sentenced two developers to prison for writing and modifying system prompts to bypass their application's ethical constraints, establishing that legal responsibility flows to whoever controls the system prompt. The paper's organization-level rankings and the convergence of Anthropic, OpenAI, and xAI toward more protective prompts suggest prompt-level protection is becoming a competitive and reputational norm.

Future Directions

1. Formalizing gray-area guidance. The 29 gray-area spans across 15 products show that many instructions are deliberate trade-offs between usability and safety. The framework needs a principled way to handle parasocial cues, permission overrides, and identity concealment rather than leaving them outside the binary.

2. Standardization and shared criteria. The paper calls for greater transparency, standardization, and independent oversight, but no shared standard yet exists for what should or should not be in a system prompt. Translating AISPA into a consensus specification is an open task.

3. Scaling and automating audits. The three-round protocol depends on expert adjudication, and the LLM pre-annotator's proposals still require human screening and unanimous expert agreement for problematic labels. Whether this can be made cheaper, faster, or partly automated without losing reliability is unresolved.

4. Validating the link from prompt to behavior. The audit measures what instructions exist, not whether models follow them — and the findings show that widely covered dimensions are also frequently violated. Whether protective instructions actually change deployed model behavior, and how auditing interacts with continued prompt drift as products evolve, remain open questions.

Target Audience

This paper is most useful to AI governance and policy researchers, AI safety and evaluation practitioners, model and product developers who write system prompts, regulators and standards bodies designing oversight mechanisms, and legal scholars interested in where responsibility for AI behavior lands. It is also accessible to informed general readers who want a concrete, evidence-based picture of how commercial AI products instruct their models to behave.

Authors’ abstract

System prompts are instructions configured by developers to govern the behaviors of foundation models in AI applications. They are used throughout commercial AI products, but are rarely disclosed to the public or regulators, creating a serious trust and accountability gap in the wide deployment of AI systems. In this paper, we introduce Artificial Intelligence System Prompt Assurance (AISPA), a user-centric framework for systematically auditing system prompts in AI systems. AISPA examines specific parts of a system prompt and evaluates them along eight dimensions that matter to users. We then use this framework to review 3,249 instructions from system prompts in 88 commercial AI products, classifying each instruction as either protective (of users) or problematic. Our audit surfaces four core findings. First, system prompt design varies substantially across products and developers, with some organizations averaging over 60 protective instructions per product while others average fewer than 5. Second, protective instructions are widely adopted but shallow in scope: 98.9% of products contain at least one, yet only 24% cover all eight dimensions of the AISPA taxonomy. Third, system prompts have grown steadily longer and more protective of users, suggesting that user protection is becoming a more visible concern in commercial prompt design. Fourth, despite this progress, problematic instructions remain pervasive: roughly 40% of products contain at least one instruction that works against user interests, and protective and problematic instructions frequently coexist within the same prompt. Our findings highlight the need for greater transparency, standardization, and independent oversight for system prompts in commercial AI products.