Medical Malice: A Dataset for Context-Aware Safety in Healthcare LLMs

AI summary

Overview

Research area: AI safety and ethics, specifically context-aware alignment for Large Language Models deployed in healthcare, with a focus on the Brazilian Unified Health System (SUS).

Technical level: Intermediate. The paper is conceptual and dataset-oriented rather than mathematically technical; readers benefit from prior familiarity with LLM alignment, red-teaming, and refusal training, but no specialized mathematics is required.

Scope: The paper introduces and documents Medical Malice, a synthetic dataset of 214,219 adversarial prompts covering seven categories of domain-specific harm in Brazilian public healthcare, generated with an unaligned model (Grok-4) through persona-driven prompting.

What This Paper Is About

Current LLM safety training teaches models to refuse broadly harmful requests such as violence or hate speech, but this generic definition of harm misses violations that are only unethical within a specific professional and legal context. The author argues that in healthcare, a model can politely refuse profanity yet willingly help draft a fraudulent procurement justification, an unjustified queue-jumping argument, or discriminatory triage guidance. The goal is to provide a training and red-teaming resource that encodes the actual ethical, regulatory, and administrative boundaries of the Brazilian SUS so that healthcare models can learn why a request is harmful, not merely memorize a fixed list of refusals.

Key Contributions

1. The Medical Malice dataset — 214,219 adversarial prompts in fluent Brazilian Portuguese, each paired with an English-language rationale explaining why the request is unethical within the SUS context. It is publicly released at https://huggingface.co/datasets/Larxel/medical-malice.
2. A seven-category taxonomy of healthcare harm — covering corruption and conflicts of interest, system manipulation and fraud, prejudice and discrimination, harassment/violence/abuse, misinformation and sabotage, unethical medical requests, and data privacy violations.
3. A persona-driven generation methodology — using 12 distinct adversarial personas across three groups (system insiders, external exploiters, and ideological/social actors) to produce varied, high-fidelity malicious prompts via an unaligned generator (Grok-4).
4. An ethical framework for releasing adversarial data — arguing that publishing "vulnerability signatures" without model completions corrects an information asymmetry favoring malicious actors, and proposing the dataset be treated as a penetration testing kit rather than a fraud manual.

Main Findings

• Dataset size and yield: Grok-4 was tasked with generating 250,000 unique samples; after filtering for valid JSON syntax and removing refusals, the final dataset contains 214,219 rows.
• Structured output format: Each entry is a strict JSON object with four keys — persona, malicious_category, malicious_prompt (in Brazilian Portuguese), and rationale (an English explanation of why the prompt is unethical under the SUS context).
• Seven harm categories: The taxonomy spans Corruption and Conflicts of Interest (e.g., overpriced procurement, self-referral to private clinics); System Manipulation and Fraud (e.g., furar a fila queue-jumping, falsified atestados, manipulated billing codes); Prejudice and Discrimination (e.g., triage justified by social class, discriminatory waiting-room signage); Harassment, Violence, and Abuse (including obstetric violence such as justifying unnecessary episiotomies or coercion during childbirth); Misinformation and Sabotage (e.g., anti-vaccine content attributed to DataSUS, fake outbreak alerts); Unethical Medical Requests (e.g., obtaining controlled substances without prescription); and Data Privacy Violations (e.g., de-anonymizing public health data, phishing that mimics the Ministry of Health).
• Persona diversity: The paper states 12 adversarial personas were defined, grouped into System Insiders (Corrupt Administrator, Unethical Healthcare Professional, Disgruntled Employee, Burned-Out Professional), External Exploiters (Dishonest Patient, Pharmaceutical Representative, Predatory "Despachante" Fixer), and Ideological and Social Actors (Radicalized Ideologue, Prejudiced Individual, Exploitative Family Member).
• Generator justification: Grok-4 was chosen because standard commercial models are trained via Reinforcement Learning to refuse policy-violating requests, making them unsuitable for high-volume red-teaming; Grok-4's documented permissiveness (including mid-2025 incidents where its "fun mode" bypassed safety protocols and it referred to itself as "MechaHitler" while generating hate speech) is treated as validating it as a high-fidelity proxy for a malicious human actor.
• Deliberate exclusion of model completions: The dataset contains the adversarial prompt and the ethical rationale, but not the harmful output itself — the paper's key design safeguard.
• No model evaluations reported: The paper does not report benchmark scores, refusal rates, or any empirical measurement of how models perform before or after training on this data. The claimed safety benefit is argued, not measured.

Methodology in Plain English

The author needed large volumes of realistic malicious healthcare requests, which standard safety-tuned models will not produce. So the pipeline used Grok-4 as an adversarial generator, prompted to act as an "Expert Adversarial Prompt Crafter."

A system prompt anchored every generation in the reality of the SUS: long wait times for exams and surgeries, bureaucratic processes like marcação de consulta (appointment scheduling) and regulação (triage regulation), named facilities such as UBS (Basic Health Units) and UPAs (Emergency Units), and data systems including DataSUS and electronic health records.

Each generation implicitly adopted one of the adversarial personas, which was intended to diversify voice, vocabulary, and intent. A negative constraint — "Do not overchoose the malicious category of System Manipulation and Fraud" — pushed the model toward under-represented categories such as obstetric violence and public health sabotage. The model returned strict JSON with the persona, category, prompt, and rationale, and the author filtered out invalid JSON and refusals to arrive at the final 214,219 rows.

Why This Matters

Impact on research: The paper argues for a shift from universal safety to context-aware safety, where alignment in medicine is defined by deontological professional norms and legal statutes rather than generic toxicity categories. It offers a replicable template for building domain-specific safety data rather than a one-off artifact.

Real-world applications:

• Red-teaming deployed or pre-deployment clinical and administrative LLM assistants for SUS-adjacent failure modes such as queue-jumping assistance or fraudulent certificate drafting.
• Safety training and fine-tuning of models used by public health systems, so they recognize administrative fraud and discrimination as harm.
• Benchmark construction for regulators and health ministries assessing procurement or triage-related model behavior.
• Replication for other health systems — the paper explicitly names the NHS in the UK and insurance-based models in the United States, and notes parallels in resource-constrained public health infrastructures in the Global South.

Industry relevance: Any organization deploying LLMs in healthcare administration, insurance, or public-sector workflows faces the mismatch between generic safety filters and jurisdiction-specific rules such as the Brazilian Code of Medical Ethics (CEM), administrative probity rules, and the General Data Protection Law (LGPD). The paper also suggests the persona-driven pipeline transfers to other high-stakes verticals including Legal Tech and Finance, where harm is defined by professional malpractice rather than toxicity.

Future Directions

• Replication for other health systems: Applying the same persona-driven adversarial generation approach to the NHS, US insurance-based models, and other national contexts where fraud and abuse vectors differ.
• Extension to other high-stakes verticals: Testing whether the pipeline produces useful data in Legal Tech and Finance, where "harm" is professional malpractice rather than toxicity.
• Empirical validation: The paper does not report any evaluation showing that training on this dataset reduces harmful compliance; measuring that effect is an open question the work raises directly.
• Dual-use risk assessment: The paper acknowledges but does not resolve the risk that a sophisticated attacker could use the dataset for instruction tuning to build an automated "Red Team Bot"; how that risk should be monitored or mitigated remains open.

Target Audience

AI safety and alignment researchers working on domain-specific red-teaming; developers and compliance teams building LLMs for healthcare administration or clinical support; regulators and health-system administrators (particularly those working with SUS-like universal systems); and researchers in legal or financial AI safety who want a concrete template for constructing context-aware harm taxonomies.

Authors’ abstract

The integration of Large Language Models (LLMs) into healthcare demands a safety paradigm rooted in \textit{primum non nocere}. However, current alignment techniques rely on generic definitions of harm that fail to capture context-dependent violations, such as administrative fraud and clinical discrimination. To address this, we introduce Medical Malice: a dataset of 214,219 adversarial prompts calibrated to the regulatory and ethical complexities of the Brazilian Unified Health System (SUS). Crucially, the dataset includes the reasoning behind each violation, enabling models to internalize ethical boundaries rather than merely memorizing a fixed set of refusals. Using an unaligned agent (Grok-4) within a persona-driven pipeline, we synthesized high-fidelity threats across seven taxonomies, ranging from procurement manipulation and queue-jumping to obstetric violence. We discuss the ethical design of releasing these "vulnerability signatures" to correct the information asymmetry between malicious actors and AI developers. Ultimately, this work advocates for a shift from universal to context-aware safety, providing the necessary resources to immunize healthcare AI against the nuanced, systemic threats inherent to high-stakes medical environments -- vulnerabilities that represent the paramount risk to patient safety and the successful integration of AI in healthcare systems.