PathCowork: Learning from real-world pathology practice to build AI copilots

AI summary

Overview

Research area: Computational pathology and medical AI — specifically, using recorded pathologist behavior (gaze, narration, navigation) as training supervision for vision–language models and diagnostic agents.

Technical level: Intermediate. The paper assumes familiarity with whole-slide images, supervised fine-tuning of vision–language models, and multiple-instance learning (MIL), but its central idea is explained in accessible terms.

Scope: The paper introduces PathCowork, a data-collection and training framework that captures pathologists' narrated slide examinations and turns them into (i) visual-grounding training data for open-source models released as PathCowork-1T and PathCowork-27B, (ii) reusable diagnostic "skills" for agents, (iii) region supervision for MIL, and (iv) a case-specific benchmark called CoworkExam.

What This Paper Is About

Pathology AI is usually trained on images paired with final diagnostic reports, which record what a pathologist concluded but not how they reached it — where they looked, at what magnification, what they revisited, and what they were uncertain about. PathCowork records that examination process directly, using a whole-slide viewer with an eye tracker and microphone, so the interpretive steps themselves become training data. The goal is diagnostic AI agents that can localize visual evidence and follow transparent procedures, rather than only predicting labels.

Key Contributions

1. A data-collection framework for the process of diagnosis. PathCowork captures synchronized gaze, narration, mouse movement, viewport position and magnification inside the TissueLab whole-slide viewer, then processes these recordings into linked records of image region, description, viewing trajectory and revisits. The collection comprises 48 sessions and 1,028 minutes of review from 16 pathologists (six attending pathologists, four fellows, six residents).

2. Behavior-trained visual grounders released as two models. Supervised fine-tuning of Kimi K2.6 (1T parameters) and Qwen 3.6 27B on the recordings produced PathCowork-1T for server deployment and PathCowork-27B for local deployment, both released at the project's GitHub repository.

3. Diagnostic skills compiled from recorded examinations. PathCowork retrieves relevant review histories for a diagnostic task and organizes their observations into a reusable procedure linking examination steps to anatomical targets, viewing scales and visual concepts; five such skills were built (melanoma versus dysplastic nevus, prostate needle biopsy, glioblastoma versus lower-grade glioma, colon lymph-node metastasis, Nottingham grading of breast carcinoma).

4. Two downstream uses of the recorded behavior. CoworkMIL uses agent-proposed regions as tile-level supervision for slide-level MIL training, and CoworkExam converts recorded examinations into case-specific benchmark questions spanning multi-slide reasoning, rejection of inapplicable premises, environmental sensing and goal-directed navigation.

Main Findings

• Grounding improved substantially over base models and a frontier reference. Aggregate spatial grounding accuracy rose from 19.8% to 51.2% for the 1T model and from 5.6% to 52.2% for the 27B model (gains of 31.4 and 46.6 percentage points). Both exceeded the GPT-5.4 reference of 24.4%.

• Narration is tied to viewing scale. Median viewport magnification was 23× for mitosis (19 observations), 21× for nuclei and 13× for cytoplasm, compared with 4.0× for glands, 3.0× for dermis, 2.6× for follicles and 1.4× for lymph nodes (103 observations).

• Spatial language maps onto gaze spread. Median root-mean-square gaze spread around an utterance was 3.0 mm for "diffuse" and 2.3 mm for "infiltrate", versus 1.7 mm for "around", 1.5 mm for "sheets", 1.1 mm for "nests" and 0.7 mm for "scattered".

• Agents outperformed general coding agents on five organ-specific tasks. Averaging the five primary task scores with equal weight, PathCowork scored 17.9 percentage points higher than Claude Code and 24.1 points higher than Codex. Skin accuracy was 80% across 20 cases comprising 69 whole-slide images versus 65% for each comparator; prostate cancer detection was 95% versus 85% (17/20) and 35%; brain accuracy was 86.7% (26/30) versus 60% (18/30) and 70% (21/30); lymph-node detection recall was 97%, 87.3% and 86%; breast grade accuracy was 71.8% (28/39), 43.6% (17/39) and 53.8% (21/39).

• The accuracy gain involved a sensitivity/specificity trade-off in skin. PathCowork detected all melanomas with 60% specificity, whereas both comparators had 40% melanoma sensitivity and 90% specificity.

• Faster and cheaper. Task-averaged processing was 6.2-fold faster than Claude Code and 4.7-fold faster than Codex (for example, 39 s versus 252 s and 117 s per skin slide; 22 s median versus 277 s and 322 s per colon lymph-node slide). Inference cost ranged from $0.01 to $0.17 per slide versus $0.47–$1.98 for Claude Code and $0.75–$1.25 for Codex, corresponding to 11.0- and 10.3-fold reductions.

• Agent-proposed regions improved MIL prostate grading with limited labels. At 20 training slides, CoworkMIL raised mean quadratic-weighted kappa over ABMIL from 0.466 to 0.544 with UNI2-h, from 0.385 to 0.486 with Virchow2 and from 0.484 to 0.585 with CONCH. The largest gain over ABMIL was 0.151 with Virchow2 at 30 slides (0.620 versus 0.469). At 50 slides the advantage narrowed to 0.013 with Virchow2 and turned negative with UNI2-h, while CONCH retained it (0.741 versus 0.643). In the auxiliary-loss sweep, kappa rose from 0.500 at β = 0.1 to 0.583 at β = 1.0, then changed little at β = 2.0 (0.576).

• Benchmark questions depend heavily on images. On the same 148 CoworkExam questions, Claude Opus 4.8 answered 26.4% correctly from text alone and 81.8% with images — a 55.4-percentage-point difference — compared with 59% and 74% on PathMMU (15 points). GPT scored 23% and 55% on CoworkExam (32 points) versus 62% and 70% on PathMMU (8 points).

• Frontier models show a specific weakness relative to a pathology resident. A resident scored 91% on rejecting inapplicable premises (models: 13–79%) and 84% on environmental sensing (models: 46.5–79%), while the best model scores nearly matched the resident on multi-slide reasoning (89% versus 87%) and goal-directed navigation (90% versus 89%).

Methodology in Plain English

Pathologists reviewed cases in a desktop whole-slide image viewer while explaining their findings aloud, as if teaching a trainee. A screen-mounted eye tracker and microphone recorded where they looked and what they said, alongside mouse movements, the visible viewport and magnification. After each session, software aligned the speech with the images, filtered gaze fixations, and produced records that pair a described finding with the specific tissue region and zoom level at which it was observed.

Those paired records served as training targets: given an image and a stated concept, a model learns to output the region matching that concept. Two open-source vision–language models were fine-tuned this way. Separately, the narrated sequences were assembled into step-by-step procedures for particular diagnostic tasks — for example, examining the dermoepidermal junction, assessing the dermal component, then searching for mitoses at higher power — and these procedures call on the trained grounder to locate concepts on new slides. The resulting agents were compared against Claude Code and Codex on five diagnostic tasks, measuring accuracy, time per slide and cost per slide. The same agent-generated regions were also used to add tile-level supervision to standard MIL training for prostate grading, and the recorded examinations were converted into multiple-choice style benchmark questions for six frontier models and one pathology resident.

Why This Matters

Research impact. The paper reframes what counts as supervision in medical AI: instead of learning from conclusions alone, models learn from the interpretive process. It also argues that evaluation should test interpretation in context — recognizing viewing scale, rejecting questions with inapplicable premises, integrating evidence across slides — not just factual recall. The released models give other groups a reusable grounding component.

Real-world applications:

• Diagnostic copilots that point to the evidence behind a suggested diagnosis, supporting the kind of explainability that regulators such as the FDA require for computer-assisted diagnosis software.
• Faster triage of high-volume tasks: prostate biopsy cancer detection, lymph-node metastasis screening and breast cancer grading at $0.01–$0.17 per slide and tens of seconds per slide.
• Local deployment of pathology AI in institutions that cannot send slides to external servers, via the 27B model.
• Data-efficient model development, where agent-proposed regions substitute for scarce expert annotations when only 10–50 labelled training slides are available.
• A continuing source of benchmark material generated from routine practice rather than hand-authored.

Industry relevance. The cost and latency comparisons against Claude Code and Codex ($0.01–$0.17 versus $0.47–$1.98 and $0.75–$1.25 per slide) matter for any deployment at slide-review volume. The release of both a server-scale and a local-scale model, and the framing of routine sign-out as a passive data source, are directly relevant to vendors and health systems building pathology AI products.

Future Directions

• Prospective clinical evaluation. The current results measure the agent's standalone accuracy; whether pathologists using PathCowork work faster, diagnose more accurately, gather more complete evidence, or report less fatigue remains unstudied, as does whether sustained use maintains or erodes unaided diagnostic proficiency.

• Three-way interaction with AI tools. The recordings capture pathologists examining images directly. Extending to pathologists, images and AI tools (for example automated cell counting or mitosis detection) would let agents learn how clinicians use and interpret AI outputs.

• Active rather than passive learning. Because learning depends on how often a concept happens to arise in routine cases, supervision may be uneven. A partly active approach could let the agent flag concepts it has not learned well and request demonstrations or corrections during an examination.

• Closing the reasoning gap. The largest model–resident gaps were in recognizing viewing context and rejecting inapplicable premises, motivating both more rigorous pathology benchmarks and pathology-specific vision–language reasoning models.

Target Audience

Researchers and engineers in computational pathology, medical imaging and clinical NLP; pathologists and informatics leads evaluating AI copilots; groups fine-tuning open-source vision–language models for grounded localization; and anyone building agent systems that must show their evidence. Readers without a medical imaging background can follow the framework and results, though the fine-tuning and MIL comparisons assume some machine-learning familiarity.

Authors’ abstract

Medical AI learns from the answers doctors record, but rarely from how they interpret the evidence to reach them. Here we introduce PathCowork, a framework that turns pathologists' narrated examinations into supervision for visual grounding and procedures that guide diagnostic AI agents. Synchronized gaze, narration and navigation link visual findings to the process of interpreting them. Across 48 sessions from 16 pathologists, the platform captured 1,028 minutes of slide review. Using these data, we trained PathCowork-1T for server deployment and PathCowork-27B for local deployment, with improved visual grounding. Across five organ-specific evaluations, agents combining this grounding with recorded diagnostic procedures achieved primary task scores averaging 17.9 and 24.1 percentage points higher than Claude Code and Codex, respectively, with shorter processing times and lower inference costs. Compared to the state-of-the-art multiple-instance learning AIs, the PathCowork agent-proposed regions achieved higher prostate-grading scores with limited slide labels. PathCowork can also help construct pathology AI benchmarks where questions derived from the recordings identified difficulties in recognizing viewing context and rejecting inapplicable premises. In summary, PathCowork proposes a new paradigm to scale up the data collection and train pathology AI through transparent reasoning from diagnostic sessions, opening a path towards systems that learn directly from routine clinical practice.