MedRECT: A Medical Reasoning Benchmark for Error Correction in Clinical Texts

AI summary

Overview

Research area: Medical natural language processing and large language model evaluation, specifically benchmarks for detecting and correcting errors in clinical text across languages.

Technical level: Advanced. The paper assumes familiarity with LLM prompting and reasoning modes, LoRA fine-tuning, and NLG evaluation metrics (ROUGE, BERTScore, BLEURT).

Scope: MedRECT is a cross-lingual (Japanese/English) benchmark of 663 Japanese and 458 English clinical texts that evaluates nine LLMs on error detection, error sentence extraction, and error correction.

What This Paper Is About

Large language models are increasingly proposed for clinical use, but their ability to catch and fix mistakes in clinical text — a safety prerequisite — has been evaluated mostly in English and mostly through manual, expensive benchmark construction. The authors build MedRECT, a Japanese/English benchmark with an automated construction pipeline, and use it to measure how reasoning ability, language, and fine-tuning affect medical error handling. The goal is to give the field a reproducible, cross-lingual way to test whether models can reliably find and correct clinical errors.

Key Contributions

1. A scalable automated benchmark construction pipeline. Instead of expert manual error injection, the authors synthesize clinical texts from Japanese Medical Licensing Examinations (JMLE) questions using LLMs, then filter, deduplicate, and screen them. The pipeline retained 92.1% (663/720) of samples after screening.

2. The first cross-lingual medical error correction benchmark. MedRECT-ja covers diagnosis, monitoring/management, physical findings, procedures, medication selection, medication dosage, test interpretation, history taking, and others, and is paired with MedRECT-en for systematic cross-lingual comparison.

3. A reproducible fine-tuning pathway. Using LoRA on Qwen3-32B with bilingual reasoning data generated by DeepSeek-R1-0528, the authors show substantial bilingual error-correction gains while preserving reasoning behavior.

4. First empirical cross-lingual analysis of nine LLMs on this task, covering proprietary, open-weight, and reasoning model families.

Main Findings

• Reasoning beats standard architectures. Comparing Qwen3-32B in think versus no-think mode on MedRECT-ja, the think version reached 0.723 error detection F1 and 72.5% sentence extraction accuracy versus 0.637 and 48.0% for no-think — a 13.5% relative improvement in detection and 51.0% in sentence extraction.

• Top MedRECT-ja performers. Claude Sonnet 4 had the highest average error correction score (0.675) and the best error detection F1 (0.795), followed by o3 (0.654 average, 0.764 F1) and GPT-5 (0.648 average, 0.758 F1). Sentence extraction accuracy ranged widely, from 42.2% (DeepSeek-V3-0324) to 83.7% (GPT-5).

• A persistent English-to-Japanese gap. Most proprietary models performed better on MedRECT-en than MedRECT-ja; the abstract reports 5–10% gaps from English to Japanese, with smaller disparities for reasoning models. GPT-4.1 showed one of the largest subtask gaps (52.6% Japanese vs. 72.8% English sentence extraction). DeepSeek-R1-0528 was an exception, scoring higher on Japanese (0.647 vs. 0.608 average).

• Fine-tuning helps asymmetrically. LoRA fine-tuning improved the Qwen3-32B + LoRA (think) error correction average by +0.078 on Japanese and +0.168 on English (relative gains of 14.2% and 30.5%). The fine-tuned model reached 90.9% English sentence extraction accuracy.

• Error type difficulty varies sharply. Medication dosage was the hardest category (average around 70%, and 27.3% for Qwen3-32B + LoRA), while medication selection was the most tractable (most models above 80%). History taking showed the largest spread (26.1%–78.3%).

• Reasoning helps most where context matters. Think vs. no-think gaps were largest for history taking (68.1% vs. 36.2%) and physical findings (68.4% vs. 37.8%). LoRA gains were also largest in history taking (+10.2 percentage points) and physical findings (+15.4 points).

• Model size does not always predict accuracy. gpt-oss-120b beat gpt-oss-20b on test interpretation (79.6% vs. 69.4%), but the smaller Qwen3-32B (think) outperformed gpt-oss-20b on history taking (68.1% vs. 47.8%).

• False-positive corrections are a deployment risk. In qualitative analysis, GPT-5, Claude Sonnet 4, and base Qwen3-32B incorrectly flagged an already-correct English diagnostic text (Sample 3, ms-test-120) and proposed unnecessary lab-finding edits; only the LoRA fine-tuned model correctly output "CORRECT."

• Reported human-level performance. The abstract states the fine-tuned model exceeds human expert performance on structured medical error correction tasks; the truncated content does not include the human baseline table.

Methodology in Plain English

The authors treat medical error handling as three chained subtasks: decide whether a clinical text contains an error, point to the sentence containing it, and rewrite that sentence correctly. Only texts with an error count for the latter two subtasks, and correction scoring follows the MEDIQA-CORR 2024 protocol.

For the Japanese half, they started with 287 clinical case questions from JMLE 2024 and 2025 and used two LLMs (DeepSeek-R1-0528 and Qwen3-235B-A22B-Thinking-2507) to convert each answer choice into a clinical narrative — correct choices became CORRECT samples, wrong choices became ERROR samples labeled by clinical domain. This produced 2,792 candidates. Eleven validation models then solved the detection and extraction tasks; samples were kept only if validation accuracy landed between 1/11 and 7/11, with ERROR samples additionally required to have a detection-vs-extraction accuracy gap of at most 3/11. That left 1,057 samples. Deduplication across the two synthesis models reduced this to 720, and a final Gemini 2.5 Pro LLM-as-a-Judge screen on five dimensions (ambiguous_error, extra_elements, multiple_errors, numerical_error, synthesis_consistency_error) left 663.

For the English half, they took the existing MEDEC MS Subset Test (597 samples) and applied the same judge screening, swapping two of the five dimensions for unrealistic_scenario and inconsistent_context because MEDEC samples lacked original question context. This yielded 458 samples. The resulting datasets have comparable error/correct ratios (about 55:45).

Nine models were evaluated zero-shot with a prompt instructing the model to act as a medical expert and output either "CORRECT" or "sentence_number: corrected_sentence," with no explanations permitted; the Japanese prompt is a direct translation. Metrics were error detection F1, sentence extraction accuracy, and correction quality via ROUGE-1 (MeCab tokenization for Japanese, whitespace for English), BERTScore F1 using microsoft/deberta-xlarge-mnli, and BLEURT-20, plus their arithmetic average.

Fine-tuning used LoRA with rank 64 and alpha 128 on Qwen3-32B at a learning rate of 1e-4, trained on 5,538 Japanese and 2,439 English samples with reasoning traces produced by DeepSeek-R1-0528.

Why This Matters

Impact on research: MedRECT removes the manual annotation bottleneck that limited MEDEC to English and provides the first parallel Japanese/English testbed for medical error correction, letting researchers separate reasoning ability from language-specific knowledge.

Real-world applications:

• Screening and quality-checking clinical notes in electronic health records before clinician review.
• Safety validation of medical LLMs before deployment in hospitals, especially in non-English markets.
• Reducing unnecessary clinician review by suppressing false-alarm corrections — a failure mode the paper directly documents.
• Supporting patient-facing communication quality, since history-taking errors required empathetic responses that several models handled poorly.

Industry relevance: The released artifacts (GitHub repository pfnet-research/medrect and the Hugging Face model pfnet/Preferred-MedRECT-32B) give vendors a ready evaluation harness. The finding that a 32B fine-tuned open-weight model can outperform much larger proprietary systems on English correction suggests targeted tuning, not raw scale, may drive medical NLP products.

Future Directions

• Closing the medication dosage gap, which persisted even in high-performing models and appears tied to numerical precision rather than reasoning style.
• Reducing false-positive error detection so systems do not burden clinicians with unnecessary alarms on already-correct text.
• Extending the automated pipeline to additional languages beyond Japanese and English, and testing whether its 92.1% retention rate holds elsewhere.
• Injecting richer clinical realism into error translation beyond the current history-taking and intervention examples.

Target Audience

Researchers and engineers building or auditing clinical LLMs, medical NLP benchmark designers, hospital informatics teams evaluating model safety, and multilingual AI developers who need a non-English evaluation resource. Readers without background in LLM evaluation metrics or fine-tuning will find the results section harder going than the dataset and contributions sections.

Authors’ abstract

Large language models (LLMs) show increasing promise in medical applications, but their ability to detect and correct errors in clinical texts -- a prerequisite for safe deployment -- remains under-evaluated, particularly beyond English. We introduce MedRECT, a cross-lingual benchmark (Japanese/English) that formulates medical error handling as three subtasks: error detection, error localization (sentence extraction), and error correction. MedRECT is built with a scalable, automated pipeline from the Japanese Medical Licensing Examinations (JMLE) and a curated English counterpart, yielding MedRECT-ja (663 texts) and MedRECT-en (458 texts) with comparable error/no-error balance. We evaluate 9 contemporary LLMs spanning proprietary, open-weight, and reasoning families. Key findings: (i) reasoning models substantially outperform standard architectures, with up to 13.5% relative improvement in error detection and 51.0% in sentence extraction; (ii) cross-lingual evaluation reveals 5-10% performance gaps from English to Japanese, with smaller disparities for reasoning models; (iii) targeted LoRA fine-tuning yields asymmetric improvements in error correction performance (Japanese: +0.078, English: +0.168) while preserving reasoning capabilities; and (iv) our fine-tuned model exceeds human expert performance on structured medical error correction tasks. To our knowledge, MedRECT is the first comprehensive cross-lingual benchmark for medical error correction, providing a reproducible framework and resources for developing safer medical LLMs across languages.