MedPsy: State-of-the-Art Small Medical Language Models for Efficient Edge Deployment

AI summary

Overview

Research area: Natural Language Processing — medical domain adaptation and efficient on-device inference for small language models. Technical level: Advanced (the paper is a full post-training recipe with RL details, quantization sweeps, and hardware measurements), though the headline results are readable without specialist background. Scope: A training-and-deployment recipe for two compact, text-only medical reasoning models (1.7B and 4B parameters) that aims to match much larger medical LLMs on accuracy while cutting response length, latency, memory, and disk footprint for private edge use.

What This Paper Is About

The strongest open medical language models require large accelerator memory or cloud access, which conflicts with the privacy, latency, and reliability needs of healthcare. Meanwhile, small medical models (1–4B) trail substantially on knowledge-intensive benchmarks, and compact reasoning models can "overthink," producing long traces that inflate latency, energy use, and KV-cache memory. The paper builds MedPsy-1.7B and MedPsy-4B to improve medical accuracy and deployment efficiency together, rather than trading one for the other.

Key Contributions

1. State-of-the-art efficient medical models. MedPsy-4B matches MedGemma-27B-text-it on aggregate closed-ended accuracy (70.54 vs. 69.95) at 6.75× smaller scale, and beats both MedGemma variants on HealthBench and HealthBench-Hard; MedPsy-1.7B beats MedGemma-1.5-4B-it by 11.42 points on closed-ended tasks despite being less than half its size.
2. A four-stage post-training curriculum. Two SFT stages (broad medical corpus, then focused clinical QA) followed by two DAPO-based RL stages with a single-pass deterministic hard-sample mining procedure — with ablations separating the roles of SFT, DAPO's drift-enabling changes, an overlong-response penalty, and Stage-2 mining.
3. Token efficiency as an explicit target. The models produce 3.2× (4B) and 1.7× (1.7B) fewer tokens than their Qwen3 backbones while improving accuracy, driven mainly by a soft overlong-response penalty during RL.
4. A quantized edge-deployment study. Six GGUF formats plus real-device inference on an Apple M5 Pro laptop and a OnePlus 13 smartphone, identifying Q4_K_M as the preferred configuration and documenting a capacity-vs-precision effect at matched footprint.

Main Findings

• Closed-ended accuracy: MedPsy-4B averages 70.54 across seven closed-ended benchmarks versus 69.95 for MedGemma-27B-text-it and 51.20 for MedGemma-1.5-4B-it (a +19.34 gap). MedPsy-1.7B reaches 62.62, beating MedGemma-1.5-4B-it by 11.42 and matching Qwen3-4B-Thinking-2507 (63.10) at less than half its size. Post-training adds +7.44 over the 4B backbone and +12.67 over the 1.7B backbone.
• Where the 4B model wins and loses: It exceeds MedGemma-27B on MedQA-USMLE (84.39 vs. 83.29), MedXpertQA (30.61 vs. 25.18), and PubMedQA (75.00 vs. 71.93), but trails it narrowly on AfriMedQA (71.50 vs. 73.07), MMLU-Health (89.70 vs. 90.48), MMLU-Pro Health (70.45 vs. 72.94), and MedMCQA (72.15 vs. 72.77).
• HealthBench results: MedPsy-4B ranks first in every grader-by-split cell across three independent graders, with Standard/Hard scores of 74.00/58.00 under CompassJudger-2-32B, 66.33/48.33 under Llama-3.3-70B, and 51.33/28.67 under GPT-OSS-120B. MedPsy-1.7B is second-best in every cell, surpassing MedGemma-27B on both splits under all three graders. The full five-model ordering is identical across all six grader-by-split cells.
• Token efficiency: Sample-weighted across eight benchmarks, the 4B model averages 909 tokens versus 2,953 for its backbone; the 1.7B averages 1,110 versus 1,901. A thinking-budget sweep shows MedPsy-4B retains 96.6% of its unbounded closed-ended score with a 300-token thinking cap.
• Where compression comes from: Vanilla GRPO with a KL anchor compresses only ~6–20%; DAPO's drift-enabling changes (KL removal, asymmetric clip-higher) bring it to ~44–45%; adding the soft overlong-response penalty reaches the released ~65–66%. The ordering of RL stages matters, and the full curriculum is Pareto-dominant over single-stage hard-only RL at ~3–4× shorter response length.
• Stage-wise progression: SFT Stage 1 delivers the largest single jump (+7.53 for 1.7B, +4.83 for 4B), with the biggest gains on HealthBench (+17.33/+11.00) and MedQA-USMLE. HealthBench gains are installed early, and the RL stages preserve them while improving closed-ended accuracy. Stage-2 hard mining accounts for essentially all of the MedXpertQA lift: replacing the mined set with a random subset of identical size loses 5.0 points (4B) and 2.0 points (1.7B).
• Quantization: At Q4_K_M, both sizes stay within one point of BF16 aggregate score (4B: 2.6 GB, change of 0.81; 1.7B: 1.2 GB, change of 0.73), a 68–69% footprint reduction. Q8_0 is near-lossless (−0.15 and 0.00). IQ3_XXS collapses (−5.56 at 4B, −12.46 at 1.7B), and the 1.7B degrades roughly twice as fast below 4-bit — the authors do not recommend any sub-4-bit variant of MedPsy-1.7B for medical use.
• Capacity vs. precision: At matched on-disk size, more parameters at lower precision wins: the 2.0 GB IQ3_M 4B model outperforms the 2.1 GB Q8_0 1.7B model.
• Real-device performance: On an Apple M5 Pro at matched Q4_K_M over 30 prompts, MedPsy-4B generates 738 tokens per answer versus 2,727 for its backbone at similar decode rates (83.9 vs. 82.2 tok/s), giving 8.75 s versus 34.6 s median time per answer and 232 J versus 917 J estimated energy — a 74.7% latency reduction at 4.29 GB peak RAM. A separate 8K-context stress test records 4.83 GB as the largest laptop peak. On a OnePlus 13, peak RAM is 2.22 GB (1.7B) and 3.98 GB (4B).
• Robustness checks: The closed-ended ranking is stable across three parsers (maximum parser-induced spread of 0.45 points) and survives decontamination, which removed 157 examples (1.10%). On LongHealth documents of 5,090–6,754 words, MedPsy-4B scores 0.855/0.875 on Tasks 1/2 versus 0.828/0.802 for MedGemma-27B, and MedPsy-1.7B leads MedGemma-1.5-4B-it by up to 30.6 points.
• Recipe portability: Applying all four stages to DeepSeek-R1-Distill-Qwen-1.5B lifts the seven-task closed-ended average from 28.4 to 52.2 and HealthBench from 7.0 to 58.0.
• Teacher matters: Baichuan-M3-235B led the teacher candidates with 74.83 average closed-ended accuracy versus 72.73 and 72.55, and beat both alternatives by 6–12 HealthBench points across all three graders.

Methodology in Plain English

The authors start from existing small Qwen3 reasoning models chosen through a controlled backbone comparison, then teach them medicine in four successive stages. First they build a large synthetic medical corpus (about 10M rows) whose questions come from public biology/medicine seed sets and newly generated health seeds, but whose answers and reasoning traces are newly written by a much larger medical-focused teacher model (Baichuan-M3-235B). Stage one fine-tunes on this broad corpus to install vocabulary and factual coverage; stage two fine-tunes on a smaller, higher-value clinical QA set to sharpen reasoning structure. Then come two reinforcement-learning stages using DAPO, a variant of GRPO that removes the KL anchor, uses asymmetric clip-higher, filters out rollout groups where every sample gets the same reward, and adds a soft penalty for overly long answers. For RL data, the model's own five sampled attempts per question determine difficulty; stage one trains on easy and moderate items (~16K), and stage two is built by re-testing the model on the full 19K set and keeping everything it still fails, topped up with a smaller sample it gets right, at a fixed 1:2 right-to-wrong ratio (2,439 items for 4B, 4,317 for 1.7B). The reward gives full credit only for a correct answer that also contains a reasoning block, half credit for a correct answer with no reasoning, and 0.1 for reasoning that reaches a wrong answer. Finally, they convert the models to GGUF at six quantization levels and measure accuracy, disk size, latency, memory, and estimated energy on real laptop and phone hardware.

Why This Matters

The paper reframes the small-medical-model problem: instead of assuming accuracy must be sacrificed for portability, it shows that data quality, staged post-training, and length-aware alignment can dominate parameter count. It also connects accuracy work to the deployment metrics that actually decide whether a model can run in a clinic or on a phone — tokens generated, not just benchmark scores.

Real-world applications:

• Offline clinical reference on consumer hardware: a 2.6 GB Q4_K_M model running locally on a laptop within ~5 GB RAM, with no patient data leaving the device.
• Privacy-constrained settings: HIPAA/GDPR-sensitive environments where sending records to a cloud API creates compliance burden and attack surface.
• Low-connectivity or underserved-region medicine: relevant given the paper's inclusion of AfriMedQA among its benchmarks.
• Mobile triage and health-literacy support: a 1.2 GB variant at 2.22 GB peak RAM on a smartphone, with shorter answers that reduce battery cost.

Industry relevance: the work gives a reproducible template (synthetic distillation from a large teacher, curriculum SFT, DAPO-based RL, quantization sweep) for any team trying to ship a domain-specialized model to edge devices, and its capacity-vs-precision finding is directly actionable for anyone choosing a GGUF format under a fixed memory budget.

Future Directions

• Multilingual and multimodal extension. The authors explicitly name this, noting the current data is predominantly English and the models are text-only.
• Stronger open-ended and safety evaluation. The paper calls for better open-ended evaluations, safety benchmarks, and medical error detection tests before behavior in higher-risk clinical scenarios can be understood.
• Clinical validation pathway. Real decision support would require additional safety review, IRB-grade evaluation, prospective testing, and regulatory clearance — none of which this work performs.
• Compression limits at small scale. The finding that MedPsy-1.7B degrades roughly twice as fast as the 4B below 4-bit raises the open question of how to preserve low-precision robustness when weight redundancy is scarce.

Target Audience

ML engineers and researchers working on domain-specific small language models, post-training/RL alignment, or on-device inference; healthcare AI teams evaluating whether a compact model can meet privacy and latency constraints; and practitioners choosing quantization formats for memory-bounded deployment. Readers interested in the medical results alone can read the introduction and Section 5; the full recipe (reward functions, mining algorithm, hyperparameters) is in Sections 4 and the appendices.

Authors’ abstract

Medical language models promise expert clinical support, but the strongest open-source LLMs remain too large for private edge-device deployment, while compact medical models still trail substantially on knowledge-intensive and real-world healthcare tasks. Reasoning models that fit edge memory budgets can produce long reasoning traces, increasing sequential decoding, end-to-end latency, energy consumption, and KV-cache memory pressure. We present MedPsy, a family of text-only 1.7B and 4B medical language models designed for edge deployment, targeting this accuracy-generation-efficiency trade-off. Our recipe combines a synthetic medical data pipeline over biology, medicine, and health seeds with chain-of-thought targets from a 235B medical-focused LLM teacher; a four-stage post-training curriculum comprising two SFT stages followed by two RL stages with hard-sample mining; and an edge deployment evaluation spanning multiple GGUF quantization variants and real-device inference on laptop and smartphone hardware. On seven closed-ended medical benchmarks, MedPsy-4B surpasses MedGemma-1.5-4B by 19.34 points and matches MedGemma-27B while being 6.75x smaller; MedPsy-1.7B outperforms MedGemma-1.5-4B by 11.42 points despite being less than half its size. On HealthBench-Hard, MedPsy-4B surpasses MedGemma-27B by 15.33 points, while MedPsy-1.7B surpasses it by 11.66 points despite being ~16x smaller. Beyond accuracy, MedPsy reduces average response length by 1.7x (1.7B) and 3.2x (4B) versus its Qwen3 backbones, while Q4_K_M quantization preserves the aggregate score within 1 point of BF16 with ~69% lower disk footprint. In real-device inference, MedPsy-4B's shorter generations translate into 75% lower median end-to-end latency and estimated energy per answer than its Qwen3 backbone at matched quantization, while peak RAM remains below 5 GB across our tested laptop and smartphone configurations.