Frozen Models, Evolving Expertise: Model-Agnostic Learning from Deployment Experience for Multimodal Medical AI

AI summary

Overview

Research area: Deployment-time learning for frozen large language models (LLMs) and vision-language models (VLMs), applied to multimodal medical AI. Technical level: Intermediate. The ideas are described in plain language, but some familiarity with LLM/VLM deployment, retrieval, and fine-tuning tradeoffs helps. Scope: The paper proposes a model-agnostic framework that lets frozen LLMs and VLMs improve during deployment by accumulating external expertise (a Skill, a Knowledge Memory, and a Multimodal Knowledge Base) rather than updating model weights.

What This Paper Is About

LLMs and VLMs are usually frozen after deployment, so they do not learn from the cases they solve. This matters in medicine, where new clinical evidence, updated guidelines, and new therapies change established practice. Fine-tuning can update a model but requires access to weights and additional training, and existing parameter-free methods may overfit a fixed validation set, lack reliable domain knowledge, or lose visual details by storing experience only as text. The paper's goal is to let a frozen model improve at a task by learning from verified deployment experience, using external expertise organized into three components and validated before it is deployed.

Key Contributions

1. Continual improvement through external expertise. A framework that enables frozen LLMs and VLMs to continually improve during deployment without training by turning experience into reusable external expertise. Because it does not require access to model weights, it applies to both open-weight and closed-source models.
2. Persistent multimodal expertise. The Multimodal Knowledge Base (MMKB) preserves images, source answers, and available explanations from reference data and completed cases, and guides the model to examine the evidence behind each reference answer and compare relevant observations and conditions with the current case.
3. Reusable expertise across models and domains. Expertise learned with one model can be reused by other models without further optimization, so it need not be relearned when the base model changes; the framework also improves non-medical visual reasoning tasks.
4. Validation-driven updates. A validation strategy that tests each candidate update on new cases and a representative buffer of past cases, accepting an update only if it improves new cases without degrading performance on earlier ones, which the authors use to reduce overfitting to a fixed validation set.

Main Findings

• Improvement on every benchmark and model. Ours scores above Base in all 24 benchmark–model pairs and achieves the best score in 22 of them, with an average gain over Base of 17.6%, compared with 5.6% for ACE. ACE falls below Base in 7 pairs; where ACE scores higher, the margin is small (0.404 vs. 0.400 and 26.4 vs. 26.1).
• Largest gains where expertise is reusable. Gains reach 34.2% on AgentClinic and 81.9% on VisuLogic, and exceed 16% on QCalEval for every base model. Gains are smaller on MedThink-Bench and MedThinkVQA (0.4%–5.0%), whose independent questions likely depend more on the model's own medical knowledge.
• Robust to case order. Repeating the online experiment with GPT-5.6-Sol on AgentClinic and QCalEval under five random case orders, Ours improves on Base by 5.7% and 16.5% on average, with both gaps at least five times the largest standard deviation (AgentClinic 79.3 ± 0.7 to 83.8 ± 0.9; QCalEval 71.3 ± 1.8 to 83.1 ± 1.5).
• Each component contributes differently. Removing the Skill causes the largest drops on text-based and multi-step tasks (12.1% on MedChain and 5.5% on AgentClinic, where the score returns to Base). Removing MMKB causes the largest drops on visual tasks (14.0% on QCalEval and 8.2% on VisuLogic, returning both to Base scores). Removing Knowledge Memory has a smaller effect, largest on MedThink-Bench (2.5%).
• Validation is necessary. Accepting every update without validation lowers the score on all six benchmarks by 21.8% to 37.8% relative to Full and leaves every score below Base.
• MMKB gains come from how references are used. With identical retrieved references, standard multimodal retrieval changes the score by −3.5% to +5.0% relative to Base, whereas MMKB improves all four image-based tasks, by up to 23.9% on MedChain Task 3. Initial MMKB (initial references only, no deployment cases) still improves all four tasks by up to 22.6% on MedChain Task 3.
• Frozen expertise generalizes to unseen cases. After learning on approximately the first 70% of the stream and freezing, Ours scores above Base in 21 of 24 pairs and achieves the best score in 19, with an average gain of 20.7%, compared with 8.3% for SkillOpt and 1.9% for ACE. SkillOpt falls below Base in 7 pairs and ACE in 12, whereas Ours falls below Base only once (−3.6% on MedThinkVQA with Nemotron-3-Nano-Omni-30B).
• Zero-step cross-model transfer. Transferred expertise improves the target model in 33 of the 36 transfer results, with an average gain of 20.0%. Expertise learned with Qwen3.8-27B improves the closed-source GPT-5.6-Sol by 11.8% on AgentClinic and 23.2% on QCalEval, and gives the weaker GPT-4o-mini a 47.3% gain on AgentClinic.
• Non-medical domains benefit too. VisuLogic and QCalEval test visual logic and quantum-calibration reasoning beyond medicine, and the framework improves both.

Methodology in Plain English

The framework leaves the base model's weights untouched and instead maintains three pieces of external expertise:

• A Skill: a natural-language document describing useful steps, their conditions of use, and when extra actions are unnecessary. After each batch, an LLM optimizer reviews the current Skill, the scored trajectories, and the previous candidate's comparison results, and rewrites the whole Skill (adding missing steps, revising overly broad rules, removing unhelpful guidance, and keeping it within a length limit) rather than appending to it, because appending lengthens the context and can leave conflicting rules.
• A Knowledge Memory: stores facts and relations with supporting evidence and conditions of use. The model can query it via a tool call; text retrieval finds candidates and a separate model checks whether they directly answer the question and apply to the case. If none is suitable, the system searches trusted external sources such as PubMed. Case-based proposals must cite supporting trajectories and quote their evidence, and multiple supporting cases are normally required (a single case suffices only with direct support from a trusted external source).
• A Multimodal Knowledge Base (MMKB): retains original images, their questions or context, source answers, and explanations when available. A fixed multimodal encoder jointly represents each case's images and question text, and cosine similarity retrieves relevant references. Rather than only choosing which references to retrieve, MMKB guides the model to examine how each reference's images and context support its source answer and which observations and conditions also apply to the current case, considering differences, missing evidence, and contradictions before answering. After each batch, cases with images and source answers are added to the MMKB with their source answers, not the model's predictions, so even incorrectly answered cases can become useful references.

Updates are decided through a validation strategy. The current expertise and a candidate version are both tested on the new batch and on a representative buffer of past cases, under the same base model, MMKB, task settings, and random seed. An update is accepted only if the new cases show positive mean improvement and no more losses than wins, and the past cases show non-negative mean change and no more losses than wins. The buffer is refreshed after each comparison using only task goals and question texts (not answers or scores), and periodically the current version is compared on the buffer with an earlier retained version, restoring the earlier Skill and Memory if the current version performs worse.

Experimental setup: Six benchmarks are used — AgentClinic-MedQA (interactive diagnosis), MedChain (multi-stage clinical workflows), MedThink-Bench (text-based medical reasoning), MedThinkVQA (multi-image diagnosis), VisuLogic (visual logic), and QCalEval (quantum-calibration reasoning). Four frozen base models are used: Qwen3.8-27B, Nemotron-3-Nano-Omni-30B, GPT-5.6-Sol, and GPT-4o-mini, with GPT-5.6-Sol as the optimizer and MMKB enabled only for image tasks. Each benchmark follows a fixed case order with seed 42, and updates occur after batches of 32 cases. Predictions are recorded before their feedback is used for learning. Metrics are diagnosis accuracy for AgentClinic-MedQA; answer accuracy for MedThink-Bench, MedThinkVQA, and VisuLogic; the official overall score for QCalEval; and MedChain's Average on a 0–1 scale across six metrics covering its five tasks.

Why This Matters

Impact on research. The paper shows that deployment experience can produce reusable expertise without touching model weights, which reframes how frozen models can improve after release. It distinguishes itself from prior parameter-free methods (Reflexion, ExpeL, ACE, GEPA, EvoSkill, SkillOpt) by validating candidate updates on both new and past cases rather than a fixed validation set, by adding knowledge supported by earlier cases or trusted external sources, and by keeping visual evidence in a multimodal knowledge base instead of storing lessons only as text.

Real-world applications (each is a direction the paper's framework targets, not a deployed product described in the paper):

• Interactive clinical diagnosis, where multi-step consultations recur and a learned procedure can help many later cases.
• Multi-stage clinical workflows such as referral, history-taking, diagnosis, and treatment decisions, as covered by MedChain's five tasks.
• Multi-image diagnosis and other medical visual reasoning, where retaining prior images and their source answers supports comparison with the current case.
• Non-medical visual reasoning scenarios such as visual logic and quantum-calibration reasoning, since the framework also improves these benchmarks.

Industry relevance. The framework applies to closed-source models because it does not require access to model weights. Expertise learned with one model transfers to other models without further optimization, so an upgraded or replaced base model can use accumulated expertise immediately rather than relearning it, which matters for organizations that switch or version their underlying models.

Future Directions

• Extending the framework to settings beyond the six benchmarks, since the authors note that gains were smaller on benchmarks whose independent questions depend more on the model's own medical knowledge, and the paper does not report how it would perform on other clinical specialties or modalities.
• Determining how the framework scales with larger case streams and more components, given the fixed settings reported (a batch of 32 cases, a 32-case buffer, one Skill of up to 24,000 characters and at most four knowledge additions per batch).
• Understanding the limits of cross-model transfer. It succeeded in 33 of 36 transfer results, but the paper does not report the conditions under which transfer fails or how much expertise must be learned before it transfers well.
• Investigating the observed declines that remain: the frozen generalization evaluation fell below Base in one pair (−3.6% on MedThinkVQA with Nemotron-3-Nano-Omni-30B), and the paper does not report an analysis of why that case occurred.

Target Audience

Researchers and practitioners working on LLM and VLM adaptation, continual learning, retrieval-augmented systems, and multimodal medical AI will benefit most. It is also relevant to engineers and product teams deploying closed-source models who cannot fine-tune weights but want systems that improve from deployment experience, and to readers interested in evaluation protocols that separate reusable expertise from corrections that help only the cases used for learning.

Authors’ abstract

Large language models (LLMs) and vision-language models (VLMs) are usually frozen after deployment, so they do not learn from the cases they solve. This is especially concerning in medicine, where new clinical evidence, updated guidelines, and new therapies can change established practice. Fine-tuning can update the model, but it requires access to model weights and additional training. Parameter-free methods avoid training, but they may overfit a fixed validation set, lack reliable domain knowledge, or lose visual details by saving experience only as text. To address these limitations, we present a model-agnostic framework that allows frozen LLMs and VLMs to learn from deployment experience through three forms of external expertise: a Skill that guides reasoning and tool use, a Knowledge Memory that stores reliable facts supported by earlier cases or trusted external evidence, and a Multimodal Knowledge Base that keeps visual examples and guides the model to relate each retrieved case to the current image. Instead of relying on a fixed validation set, a validation strategy keeps an update only if it helps on new cases without degrading performance on earlier ones. Across six benchmarks covering clinical diagnosis, clinical workflows, medical reasoning, and medical and non-medical visual reasoning, and with four open-weight and closed-source base models, our framework improves performance during online deployment by up to 34.2% over the base model on medical tasks, generalizes to unseen cases, transfers to other models without further optimization, and works in non-medical domains.