PAINT: A Framework for Multilingual Pain Representation from Medical Licensing Exams to Real-World Clinical Texts

AI summary

Overview

• Research area: Multilingual clinical natural language processing (NLP) and clinical information extraction, focused on how pain is documented in medical text.
• Technical level: Intermediate. The paper is primarily a data-resource and benchmarking paper; the methods are LLM prompting pipelines and ensembles, described at a level accessible to readers familiar with basic NLP evaluation metrics.
• Scope: PAINT is a multilingual framework and released dataset pairing medical licensing examination questions with real-world clinical records in English, Spanish, Portuguese, and Japanese, annotated for pain and demographic attributes with an open-weight LLM ensemble and validated by clinicians.

What This Paper Is About

Pain is one of the most common and clinically important symptoms, but it is hard to capture reliably from medical text because its expression varies by language, documentation type, and patient demographics, and because pain mentions are often embedded in negation, temporality, and contextual phrasing. Existing pain-related NLP resources are largely monolingual, small, and limited to narrow clinical settings, and there was previously no unified multilingual representation of pain across both medical education and clinical practice. PAINT addresses this gap by combining exam text and real clinical text under one structured annotation schema in four languages, with labels generated by an LLM ensemble and checked by clinicians.

Key Contributions

1. Resource. A multilingual pain dataset that pairs medical-licensing-exam text (WorldMedQA-Pain, 2,963 items) with real clinical notes (BRIDGE-Pain, 17,795 documents) under one structured schema across English, Spanish, Portuguese, and Japanese, with clinician-validated labels and a released extraction pipeline. The validated labels also serve as an evaluation set for pain extraction, particularly for non-English settings where such data are scarce.

2. Methodology. A reproducible, clinician-validated labeling pipeline with three stated strengths: generalizability (a cross-lingual, cross-corpus pain extraction and labeling framework), transferability (a pipeline validated on WorldMedQA-Pain transfers to BRIDGE-Pain, where clinician review confirms macro-F1 of at least 0.91 for pain relevance and pain presence and at least 92% accuracy for demographics across all five clinical corpora), and efficiency (the open-weight ensemble matches GPT-5 on the consensus subset and runs locally on a single GPU, making it usable on controlled-access clinical corpora and in resource-constrained settings).

3. Documentation. Per-partition documentation of the released labels: sample sizes, consensus coverage, pain relevance and presence rates, and demographic field coverage, together with the annotation protocol, clinician instructions, and every inter-annotator disagreement, so users can judge the reliability of each field and partition before reuse.

Main Findings

• Pain versus no-pain distinction is a core design choice: The schema separates pain relevance (any pain mention, including denial or "painless"-type derivatives) from pain presence (pain asserted in the patient at least once). Examples of mention-without-presence cases are given in English, Spanish, Portuguese, and Japanese, such as "She has no pain", "sin fiebre ni dolor", "nódulo indolor", and 無痛性腫大.

• Model selection on WorldMedQA-Pain: Gemma-4-E4B (4B), Qwen3-8B (8B), and DeepSeek-R1-Distill-Qwen-32B (32B) performed best overall, with macro-F1 of 0.87 to 0.97 for pain relevance and pain presence across the four partitions, and were combined into the ensemble. LLaMA-3.1-8B and MedGemma-4B scored lower across partitions and were not included.

• Ensemble performance on the consensus subset: The three-model ensemble reached macro-F1 of 0.922 / 0.916 (Brazil, relevance / presence), 0.992 / 0.992 (Japan), 0.935 / 0.960 (Spain), and 0.961 / 0.969 (UK), with consensus coverage of 88.9%, 92.9%, 92.4%, and 91.0% respectively. GPT-5 as a proprietary reference scored 0.925 / 0.915, 0.995 / 0.989, 0.937 / 0.955, and 0.966 / 0.978 on the same subset.

• Abstained rows are harder, mainly on pain presence: The strict agreement rule set aside 7–11% of WorldMedQA-Pain rows and up to 31.8% of Brateca. On the abstain subset, no system exceeded 0.75 on pain presence; GPT-5 reached only 0.63–0.73 on pain presence, while pain relevance was more recoverable (GPT-5: 0.80–0.98). The paper states this means the consensus subset is enriched for cases where pain presence can be determined more reliably, not a random sample of the corpus.

• Clinician validation on BRIDGE-Pain: The ensemble achieved macro-F1 of 0.91–1.00 on pain relevance and pain presence and 92–100% accuracy on demographics across MEDIQA, MIMIC-III, Cantemist, IFMIR, and Brateca. Pooled across the five datasets on records where the two clinicians agree (about 750 each), pain relevance and pain presence macro-F1 were both 0.98, age accuracy 97.3% (95% CI [95.9, 98.3]), gender 98.1% ([96.9, 98.9]), and occupation 97.9% ([96.6, 98.7]).

• Inter-annotator agreement is high: Cohen's κ was 0.88–1.00 for pain relevance and 0.87–1.00 for pain presence on the BRIDGE-Pain overlaps, with 90–100% agreement for demographic attributes across the BRIDGE-Pain and WorldMedQA-Pain overlaps. Wherever defined, demographic κ values ranged from 0.82 to 1.00, corresponding to almost-perfect agreement on the Landis and Koch (1977) scale.

• Disagreements are documented in full: Across all double-annotated overlaps there were 17 disagreements out of 1,370 judgements; the paper attributes them to dialogue intent, whether pain elicited during a neurological examination counts as a mention, whether 気分不快 (malaise) is a pain term, assertion strength, and a reading slip on a long record.

• Pain rates differ substantially by partition but are not prevalence estimates: In WorldMedQA-Pain, pain relevance was 36.5% (Brazil REVALIDA), 17.5% (Japan NLE), 15.9% (Spain MIR), and 38.8% (UK PLAB); pain presence was 30.0%, 15.1%, 13.0%, and 33.6%, with non-empty pain location in 27.8%, 14.7%, 11.8%, and 32.8% of items. The paper states these label distributions characterize each corpus rather than estimate prevalence and are not comparable across partitions.

• Demographic field coverage varies widely: Within the pain-consensus rows, coverage ranged, for example, from 7.5% (IFMIR age) and 0.0% (IFMIR ethnicity) to 100.0% (MIMIC-III gender) and 99.9% (Brateca gender), as reported in Table 6.

• Descriptor vocabulary differs structurally by language: Frequent descriptors include English "pressure" (153), "sharp" (129), "discomfort" (125); Portuguese "aguda" (227), "desconforto" (106), "dor" (72); Spanish "irradiado/a" (31), "intermitente" (25), "irradiado" (18); and Japanese compounds such as 圧痛 (42), 頭痛 (21), 腹痛 (18). The paper notes Japanese descriptors fuse location and quality, whereas the other languages use location-independent adjectives.

Methodology in Plain English

The authors picked existing corpora rather than collecting new text. From WorldMedQA's text-only national licensing examination partitions they kept four in their original languages: REVALIDA (Brazil), the National Medical Licensing Examination (Japan), MIR (Spain), and PLAB (United Kingdom), giving 2,963 instances. From the multilingual BRIDGE benchmark they took five clinical-text partitions: MEDIQA 2023 Chat-A (English doctor–patient dialogues), MIMIC-III Outcome (English ICU discharge summaries), Cantemist (Spanish oncology reports), Brateca (Brazilian Portuguese electronic health records), and IFMIR (Japanese medication-safety incident reports), retaining MEDIQA and Cantemist in full and randomly down-sampling MIMIC-III, Brateca, and IFMIR to 5,000 records each, for 17,795 clinical documents in total.

Each record is described by five fields in a hierarchical schema: pain relevance (any pain term, synonym, or derivative, or an explicit denial), pain presence (conditioned on relevance and indicating pain is asserted to be present in the patient at least once), anatomical location (conditioned on presence, with "unspecified" when no site is named), pain descriptors (free-text quality descriptions), and demographics (age, gender, ethnicity, occupation when stated). Two chain-of-thought prompting pipelines were used, one for pain attributes and one for demographics, with all source texts processed in their original languages and hierarchical constraints enforced in the prompt.

After evaluating five open-weight models plus GPT-5 as a proprietary reference, the authors built a consensus ensemble from the three best performers: Gemma-4-E4B, Qwen3-8B, and DeepSeek-R1-Distill-Qwen-32B. For the two binary pain fields, a record is retained only if all three models agree; otherwise it is marked as abstained. For demographics, majority voting over non-null predictions is used, so a value produced by only one model is kept only if no other model provides a conflicting prediction. Anatomical location and pain descriptors are treated as lists and aggregated item by item: if only one model provides a list, it is retained; otherwise an item is kept only if at least two models include it. This favors higher coverage for sparse attributes such as ethnicity and occupation.

A deterministic normalization step converts age to years using the first numeric value (ages under one year recorded as 0), maps gender to the binary labels male and female using a cross-lingual dictionary, and preserves ethnicity and occupation as free text, with non-normalizable values recorded as null. Ten clinicians, two per dataset, validated the annotations: each BRIDGE-Pain annotator reviewed a random sample of 100 records and each WorldMedQA-Pain partition 30 records, with overlapping subsets of 50 and 10 records respectively for agreement analysis. Annotation was open (not blind for demographics) and disagreements were not altered after the fact. All inference used open-weight models under 4-bit quantization on a single NVIDIA A100 (40 GB) or L4 (24 GB) GPU with 4-bit NF4 quantization via bitsandbytes; GPT-5 (gpt-5-2025-08-07) was queried through the OpenAI API. No model was trained or fine-tuned.

Why This Matters

Impact on research. PAINT is presented as, to the authors' knowledge, the first multilingual resource for pain representation spanning medical licensing exams and real-world clinical text in English, Spanish, Portuguese, and Japanese. It gives clinical NLP researchers a clinician-validated evaluation set for pain extraction where non-English data are scarce, and it provides a released extraction pipeline, prompts, schema, validation and agreement scripts, and clinician annotation sheets (source text removed) for publicly licensed corpora, with sheets for credentialed corpora available on request. The paper frames the resource as infrastructure for studying whether LLMs reproduce known disparities in pain assessment, treatment, and documentation, which prior work has linked to patient race, gender, age, and language.

Real-world applications.

• Auditing clinical NLP systems and LLMs for fairness and robustness in pain-related extraction across languages and care settings.
• Building or evaluating structured pain variables from routine clinical documentation in hospitals using languages other than English.
• Studying how demographic detail (age, gender, ethnicity, occupation) is documented alongside pain, using per-partition coverage statistics as a guide to which fields are usable.
• Benchmarking information extraction on exam-style clinical text, using the four WorldMedQA-Pain partitions as a controlled knowledge setting.

Industry relevance. The authors emphasize that the open-weight ensemble matches GPT-5 on the consensus subset and runs locally on a single GPU, which matters for controlled-access clinical corpora and resource-constrained settings where sending data to a proprietary API is not viable. The paper also states the pipeline is intended for multilingual clinical NLP research, information extraction, and fairness auditing, and is not intended for clinical diagnosis, treatment recommendation, or automated triage. Demographic results for restricted clinical datasets are reported as aggregate statistics.

Future Directions

• Extending language and document coverage. The paper states the pipeline extends to further languages and document types with minimal adaptation, and notes that European Portuguese pain vocabulary is not represented because both Portuguese partitions are Brazilian.
• Adding structured and patient-reported pain data. PAINT captures pain as documented by clinicians and examination writers, not as reported by patients, and none of the source corpora carry numeric pain scores; the schema is designed to accommodate structured pain-intensity scores and patient-reported pain once corpora with such data become available.
• Releasing exam gold labels and improving hard cases. The clinician gold labels for WorldMedQA-Pain are to be made available in a subsequent version, and the abstain subsets — where no system exceeded 0.75 on pain presence and GPT-5 reached only 0.63–0.73 — remain an open target for improving pain-presence determination.
• Addressing comparability and confounding. Because corpus purpose, language, country, and healthcare system co-vary across the nine partitions, and because the English partitions are not country-matched (UK PLAB exam versus US clinical corpora MEDIQA and MIMIC-III), differences between partitions should be read as corpus-purpose effects rather than as cultural claims about pain experience.

Target Audience

Clinical NLP researchers and benchmark developers working on multilingual information extraction; medical informatics groups studying pain documentation, disparities, and fairness in clinical AI; researchers needing evaluation data for pain-related extraction outside English; and practitioners or engineers who need to run extraction pipelines locally on controlled-access clinical text under constrained compute.

Authors’ abstract

Pain is one of the most common and clinically important symptoms, reflecting disease onset, progression, and treatment response. However, pain is difficult to accurately capture and interpret from medical text, as its expression varies across languages, documentation types, and patient demographics, and is often embedded in terminology, negation, temporality, and contextual descriptions. Existing pain-related studies remain largely monolingual or limited to narrow clinical settings. In this study, we introduce PAINT, a multilingual framework for studying pain representation across medical exams and real-world clinical texts. PAINT includes nine pain-focused dataset partitions spanning English, Spanish, Portuguese, and Japanese, with clinician annotations in all four languages. We curated a structured schema covering pain relevance, pain presence, anatomical location, pain descriptors, and demographic information. To capture multilingual pain representations, we leverage large language models (LLMs) and adopt an ensemble strategy to improve performance. Ten clinicians across four languages validate the labels: the ensemble reaches macro-F1 of 0.91-1.00 on pain relevance and pain presence and 92-100% accuracy on demographics, with inter-annotator $κ$ of 0.82-1.00. We release the WorldMedQA-Pain ensemble labels, the extraction pipeline, validation scripts, and clinician annotation sheets for publicly licensed corpora, providing a foundation for pain-centered clinical NLP and for studying pain documentation across languages and healthcare contexts.