Taming the Real-world Complexities in CPT E/M Coding with Large Language Models

AI summary

Overview

Research area: applied large language models for clinical/administrative natural language processing, specifically automated CPT Evaluation and Management (E/M) medical billing code assignment. Technical level: Advanced. Scope: the paper describes ProFees, an LLM-based framework deployed inside a production EHR system that predicts CPT E/M codes from clinical encounter notes, and reports a systematic evaluation against a commercial coding system and single-prompt baselines.

What This Paper Is About

CPT E/M codes document the medical services a physician provides and are used primarily for billing, so accuracy matters financially and for compliance. Assigning them requires reading a clinical note and applying detailed American Medical Association guidelines, and errors are costly: the paper cites a U.S. Office of Inspector General estimate that in 2010 Medicare inappropriately paid US$6.7 billion (21% of total E/M payments) due to incorrect coding and/or insufficient documentation, with 42% of claims miscoded and 19% lacking proper documentation, plus a Florida analysis estimating nearly 9% of primary care visits (2.6 million annually) were undercoded, costing hospitals approximately US$114 million in lost revenue. The goal is to automate this coding reliably enough for real deployment, despite label noise, missing intermediate labels, and a need for explainable, repeatable outputs.

Key Contributions

1. Development and deployment of ProFees, an LLM-based framework combining dynamic few-shot prompting, explicit self-criticism, and self-consistency for CPT E/M coding and intermediate Medical Decision Making (MDM) level estimation.
2. A systematic evaluation on a proprietary, expert-annotated test dataset of 99 encounters, compared against a commercial CPT E/M coding system (System A) and four single-prompt baselines.
3. Characterization of five real-world complexities in CPT E/M automation (intermediate labels, label noise, explainability, robustness, clinical breadth) and a matched solution for each.
4. Expert re-annotation of 117 encounters with three intermediate MDM element levels — Problem, Data, and Risk — plus justifications, and documentation of a 56% disagreement rate between external and internal coding experts.

Main Findings

• Full model is best on the test set: ProFees (Few-Shot + RCI), the complete model, achieves the highest CPT accuracy, reported as 36.85% higher than the commercial System A, 33.73% higher than the Single prompt baseline, and 4.73% over the strongest baseline (Single prompt + Full Info + CoT).
• Abstract-level framing: the abstract states an increase in coding accuracy of more than 36% over a commercial CPT E/M coding system and almost 5% over the strongest single-prompt baseline. The conclusion states the improvement over the commercial System A as 30.61% rather than 36.85%, so the paper reports both figures in different places.
• Table 3 is relative improvement, not absolute accuracy: results are listed as accuracy improvements over the "Single prompt" baseline, averaged over 5 runs, with standard deviations. The highest CPT row is ProFees (Few-Shot + RCI | Full) at 33.73 ± 0.58, and its MDM accuracy improvement is 33.99 ± 0.60, Problem Complexity (PC) 35.06 ± 0.00, Data Complexity (DC) 39.76 ± 1.18, and Risk Complexity (RC) 27.14 ± 0.68.
• System A performance: the commercial rule-based tool records 3.12 ± 0.64 CPT and 4.99 ± 0.40 MDM improvement, and intermediate complexity predictions are not available for it, so those columns are omitted.
• RCI helps: adding Recursive Criticism and Improvement lifts MDM accuracy by 4% in the few-shot setting and 2.33% in the zero-shot setting. The ablation reports a 7.58-percentage-point gain in overall MDM accuracy and a 4% gain in final CPT accuracy from the RCI stage.
• Retrieval helps: replacing the verbose prompt with three retrieved chain-of-thought examples increases CPT accuracy by 4% relative to ProFees (Zero-Shot + RCI), and by 2.3% when comparing ProFees (Few-Shot) to ProFees (Zero-Shot).
• Self-consistency has diminishing returns: going from K=1 to K=3 yields an additional +1.2% CPT and +2.7% MDM accuracy but triples unit cost; K=5 adds only +3.1% CPT and +6.8% MDM over the 1-vote ProFees while pushing cost above a four-fold increase. Data-Complexity gains +11.8% at K=3 and +19.9% at K=5; Problem-Complexity peaks at K=3 then declines slightly; Risk improves by just +2.7% overall. K=3 was fixed for all remaining experiments.
• Intermediate elements improve most: the full model outperforms the strongest baseline on MDM, PC, DC and RC accuracy by 8–17 absolute points.
• Generalization check: the overall trend of improvements held on the internal–external coder agreement subset of the test set.
• Label disagreement is high: the internal expert agreed with the professional coder's CPT E/M codes on 52 encounters (44%) and disagreed on the remaining 65 (56%).

Methodology in Plain English

The architecture splits the problem the way the CPT guidelines do. Because the final code depends on encounter type, MDM level, and patient type — and patient type is already available from structured EHR data — ProFees predicts only the first two.

An Encounter Type Classifier reads the SOAP note (Subjective, Objective, Assessment, Plan), patient age, and patient type, and returns an encounter classification. An MDM Complexity Classifier then assesses the three MDM elements (problems addressed, data reviewed and analyzed, risk of complications), which combine via a 2-out-of-3 rule into a level of straightforward, low, moderate, or high.

The MDM classifier works in stages. It first makes a classification using dynamic few-shot prompting, retrieving the top N=3 semantically similar exemplars from a vector database built from the development and test data, using leave-one-out filtering so a query's own example is never retrieved. The database indexes gold justifications, expert-verified model justifications, and expert-verified chain-of-thought reasoning with checklist verification. A Recursive Criticism and Improvement step then has a dedicated critic for each MDM element audit the initial prediction against checklists built from common errors the internal expert observed. To counteract LLM stochasticity, the RCI-enhanced classifier runs K times concurrently (avoiding added latency, at proportional cost) and takes a majority vote per element; ties go to the first prediction in the sorted result list, so with {1, 2, 2, 2, 3, 4, 4} the output is 2, and with {1, 2, 2, 2, 4, 4, 4} it is still 2 over 4. A deterministic rule-based decision tree then combines classifier outputs with EHR data to emit the final CPT E/M code plus a justification passed down from the MDM classifier.

Data: 216 real-world encounters were collected from hospital production databases, de-identified and drawn from outpatient and preventive medicine. Each carries a physician-assigned CPT E/M code and a professional coder's code used as the final billing code, but no detail on how the code was determined, so an internal expert re-annotated problem, data, and risk levels with free-text justifications for each encounter. This produced 117 annotated development encounters, split into Platinum (52 encounters where the expert and professional coder agreed) and Disagreement (65 encounters where they diverged, 56%). Prompt tuning and critic-checklist design ran only on Platinum, followed by iterative error analysis on both subsets with the internal expert. A separate Test set of 99 encounters was annotated the same way.

Averages of words per SOAP note: Platinum 543, Disagreement 451, Test 434.

Experiment settings: the 117 development encounters serve as the tuning set for both ProFees and baselines; Azure GPT-4o-2024-05-13 is the foundation LLM for both. Temperature was set to 0 for all calls, K=3 predictions per test encounter for the self-consistency vote, and one round of RCI per MDM element. Only strict exact-match accuracy is reported, because payers adjudicate at the code level. Full production prompts are not released verbatim for compliance and business confidentiality reasons; template skeletons covering role, task, and JSON output schema are provided.

Why This Matters

Impact on research: the paper presents what it describes as the first systematic, production-oriented study of automating CPT E/M coding with LLMs, and frames CPT E/M coding as an instance of a broader class of problems where models must map free text to codified decisions under strict guideline constraints — with lessons for other regulated domains needing auditable, rule-aligned predictions. It also documents label noise as a first-class obstacle, noting that even expert coders disagreed on 56% of encounters.

Real-world applications:

• Reducing physicians' documentation burden and improving billing efficiency and revenue capture turnaround.
• Consistent application of complex coding rules at scale, handling high-volume chart reviews without adding coder workload.
• Audit and compliance support, since ProFees emits chain-of-thought exemplars and checklist-based critic outputs citing CPT guideline clauses.
• Optimizing coder resources by letting human coders focus on more complex or ambiguous cases.

Industry relevance: the work comes from Oracle Health & AI and is described as deployed within a production EHR system, with a commercial rule-based coding tool as the comparison point. The paper notes business outcomes like revenue uplift are hard to attribute directly to code choices because of diagnosis–procedure linkage, billing completeness, payer-specific rules, and documentation quality.

Future Directions

• Extending ProFees to predict multiple codes per encounter, such as a preventive medicine visit that also includes a problem-oriented service, which the authors say can be done by extending the Encounter Type Classifier to output multiple encounter types.
• Supporting CPT modifiers.
• Generating synthetic datasets for edge-case testing and to fill the gap in publicly available CPT E/M coding data, which is also proposed as a way to evaluate model bias.
• Enriching the vector database.
• Broader open issues raised in the ethics section: automation bias risk given the >30-point accuracy improvement, algorithmic fairness concerns from tuning on a small real-world dataset, and continued human oversight and user education given that ProFees is framed as an assistive, not autonomous, tool.

Target Audience

Clinicians, professional medical coders, and health-system informatics and revenue-cycle teams considering coding automation; NLP and applied LLM researchers interested in retrieval-augmented prompting, self-critique, and self-consistency for high-stakes, guideline-constrained prediction; and developers building auditable decision-support systems in regulated domains. Readers wanting to reproduce the pipeline should note that the real-world test set is not publicly available due to privacy and regulatory constraints, the dataset sizes are small (117 development and 99 test encounters), and the full prompts are withheld.

Authors’ abstract

Evaluation and Management (E/M) coding, under the Current Procedural Terminology (CPT) taxonomy, documents medical services provided to patients by physicians. Used primarily for billing purposes, it is in physicians' best interest to provide accurate CPT E/M codes. %While important, it is an auxiliary task that adds to physicians' documentation burden. Automating this coding task will help alleviate physicians' documentation burden, improve billing efficiency, and ultimately enable better patient care. However, a number of real-world complexities have made E/M encoding automation a challenging task. In this paper, we elaborate some of the key complexities and present ProFees, our LLM-based framework that tackles them, followed by a systematic evaluation. On an expert-curated real-world dataset, ProFees achieves an increase in coding accuracy of more than 36\% over a commercial CPT E/M coding system and almost 5\% over our strongest single-prompt baseline, demonstrating its effectiveness in addressing the real-world complexities.