Eye-Tracking, Mouse Tracking, Stimulus Tracking,and Decision-Making Datasets in Digital Pathology

AI summary

Overview

Research area: Digital pathology, medical image perception, and human-computer interaction — specifically multimodal behavioral data collection around whole-slide image (WSI) diagnosis. Bridges computer vision benchmark datasets (CAMELYON16) with cognitive science methods (eye-tracking, scanpath analysis).

Technical level: Intermediate. The dataset itself is accessible to newcomers, but exploiting it fully requires familiarity with eye-tracking metrics, coordinate systems, and WSI tooling.

Scope: The paper introduces PathoGaze1.0, a preregistered, application-grounded dataset of eye-gaze, mouse, viewport, and diagnostic decision behavior from 19 pathologists reading 397 breast cancer lymph node slides, collected through a custom testbed that mirrors real clinical software.

What This Paper Is About

Pathologists reading giga-pixel whole-slide images are only about 70% accurate on average, and adding a second reader does not meaningfully improve consistency — yet the field has almost no behavioral data explaining why these errors and disagreements happen. The authors built PathoGaze1.0 to fill that gap, recording where pathologists looked, how they navigated, and what they decided across two large-scale experiments. The result is a linkable record connecting image stimuli, visual search, motor action, and diagnostic outcome.

Key Contributions

1. PathoGaze1.0 dataset: 18.69 hours of synchronized eye-tracking, mouse interaction, stimulus tracking, viewport navigation, and diagnostic decision (EMSVD) data from 19 pathologists interpreting 397 WSIs — comprising 171,909 fixations, 263,320 saccades, and 1,867,362 mouse interaction events.

2. An application-grounded testbed (PTAH): A data collection interface assembled from features of state-of-the-art clinical diagnosis platforms, using OpenSeadragon for WSI rendering and Annotorious for region annotation, to preserve ecological validity rather than using an artificial laboratory viewer.

3. Two complementary experimental designs: Experiment I (P10S60T600) has 10 pathologists each read the same 60 slides for cross-reader consistency analysis; Experiment II (P9D397T540) has 9 pathologists read a broad, nearly non-overlapping set of 397 slides for behavioral coverage.

4. A harmonized, FAIR-aligned data release: Separate gaze-only, mouse-only, and time-synchronized combined files, plus trial-level gaze metrics, image metrics, participant experience metadata, coordinate transformation from screen space to WSI image space, and public analysis code.

Main Findings

• Behavioral data was previously missing for pathology errors: The paper frames existing benchmarks such as CAMELYON16 as designed for neural network analysis, never analyzed jointly with the visual search behavior of the humans those systems are meant to assist.

• Distinct viewing strategies are observable: In cases with large tumor regions, one participant (P5) reached a diagnosis without scanning all tumor areas, while another (P10) examined most regions before responding — evidence of early "satisfied" stopping behavior.

• Search errors occur in realistic settings: Participant P8 failed to fixate on tumor regions entirely in a challenging small-tumor case, a "Looked But Failed to See"-type failure, while the same participant correctly identified a tumor in a different slide.

• Mouse and gaze are often but not always aligned: Mouse movement generally followed gaze (e.g., P10), but in some cases (e.g., P5) the mouse roamed over a wider area than the gaze — raising an open question about whether mouse tracking is a valid proxy for attention in pathology as it is in other imaging modalities.

• Gaze is resolvable down to the tumor level: By converting screen-space fixations into WSI image coordinates and segmenting tissue with Otsu's method, the dataset supports quantifying whether fixations fall inside or outside ground-truth tumor regions.

• Validation was done against raw recordings: Trials were manually checked against screen recordings to confirm that processed fixations matched those directly output by Tobii Pro Lab, and mouse/eye streams were synchronized via eye-tracker timestamps logged by the testbed.

Methodology in Plain English

The team recruited 19 pathologists at The Ohio State University Wexner Medical Center ranging from residents to readers with more than 10 years of experience. Each sat at a 27-inch clinical-grade monitor (the same model used in daily diagnosis) with a Tobii Pro Fusion eye-tracker mounted below the screen — no chin rest, so they could sit naturally. All participants were in the same room and lighting conditions as their normal clinical work.

The viewing software, PTAH, was built to look and behave like a real pathology workstation: participants could pan by dragging, navigate tile-by-tile with arrow keys, draw circular, rectangular, or freehand annotations, and select a diagnosis from a menu. This matters because behavior collected in a stripped-down viewer may not transfer to real practice.

In Experiment I, 10 pathologists each read the same 60 slides (15 benign, 45 malignant), marking the tumor region that drove their decision and briefly describing their reasoning. In Experiment II, 9 pathologists read as many slides as possible from a pool of 397, with each slide seen by no more than two participants — a design traded for breadth over repetition. Experiment II dropped annotation and verbal reasoning to preserve the flow of real clinical reading.

Because the eye-tracker reports positions in screen pixels, the mouse in screen pixels, and the image in its own enormous pixel grid, the authors converted everything into WSI image coordinates using a scaling-and-offset formula that accounts for the current viewport and zoom level. Each trial was defined from image load to the click of "Next," and gaze recorded outside that window was discarded. Fixations and saccades were classified with the Tobii I-VT filter using standard settings.

Finally, the team segmented tissue from background using Otsu's thresholding on HSV channels, producing masks that let analysts ask whether a fixation landed on or off a tumor.

Why This Matters

Impact on research. Diagnostic error research in pathology has largely been limited to outcome measures — correct or incorrect, benign or malignant. PathoGaze1.0 makes the process observable at scale, enabling tests of established perception theories (satisfaction of search, look-but-fail-to-see, attentional capture) in a domain where they have never been systematically measured. It also provides a human-behavior counterpart to CAMELYON16 — a benchmark used in 26 of 41 public-dataset studies (63%) on lymph node machine learning.

Real-world applications:

• Explainable AI for pathology: Models that localize regions can be compared against where pathologists actually looked, revealing whether an AI succeeds for the same reasons a human does or by exploiting different cues.
• Training and competency assessment: Gaze patterns of experts versus trainees can inform feedback tools and curricula, showing novices what efficient search looks like in specific case types.
• Workflow and interface design: Viewport, zoom, and mouse data expose where readers waste time, when they over-zoom, and how navigator use correlates with diagnostic success — directly informing scanner and viewer software design.
• Digital twins of diagnostic reasoning: The synchronized perception-action stream supports building models that predict what a pathologist will do next, a stepping stone toward decision-support systems that intervene at the right moment.

Industry relevance. Scanner and pathology software vendors, AI diagnostic developers, and clinical labs all have a stake. Vendors gain behavioral evidence for interface decisions; AI companies gain a human baseline for validation and for training human-AI collaborative systems; regulatory and quality bodies gain a potential signal for reader-performance monitoring.

Future Directions

• Is mouse tracking a valid proxy for gaze in pathology? The authors explicitly flag cases where mouse and gaze diverged, and pose validation of that correlation as an open question carried over from other imaging modalities.

• Link behavior to AI model internals. Future work could pair PathoGaze1.0 fixations with the saliency and attention maps of pathology models to determine whether AI and human readers attend to the same morphology.

• Finer-grained error taxonomy. The dataset enables distinguishing satisfaction-of-search, recognition failure, and early stopping — but the paper only sketches the observed behaviors; a full categorization across all 397 slides remains to be done.

• Scale and generalization beyond breast lymph node. All slides come from CAMELYON16, so whether the observed search patterns transfer to other organs, stains, and tumor types is untested. Extending the protocol to additional specimen types, and to human-AI collaboration settings where a model's output appears on screen, are natural next steps.

Target Audience

• Digital pathology and medical AI researchers seeking human behavioral ground truth to complement or critique model behavior.
• Human factors and medical image perception scientists interested in applying eye-tracking methods to a new, high-stakes domain.
• Pathology informatics and viewer software developers looking for empirically grounded design guidance.
• Clinical educators and pathology residency programs exploring objective, gaze-based feedback for trainees.
• Cognitive scientists studying visual search who want a well-instrumented, preregistered dataset in a real-world expertise setting.

Authors’ abstract

Interpretation of giga-pixel whole-slide images (WSIs) is an important but difficult task for pathologists. Their diagnostic accuracy is estimated to average around 70%. Adding a second pathologist does not substantially improve decision consistency. The field lacks adequate behavioral data to explain diagnostic errors and inconsistencies. To fill in this gap, we present PathoGaze1.0, a comprehensive behavioral dataset capturing the dynamic visual search and decision-making processes of the full diagnostic workflow during cancer diagnosis. The dataset comprises 18.69 hours of eye-tracking, mouse interaction, stimulus tracking, viewport navigation, and diagnostic decision data (EMSVD) collected from 19 pathologists interpreting 397 WSIs. The data collection process emphasizes ecological validity through an application-grounded testbed, called PTAH. In total, we recorded 171,909 fixations, 263,320 saccades, and 1,867,362 mouse interaction events. In addition, such data could also be used to improve the training of both pathologists and AI systems that might support human experts. All experiments were preregistered at https://osf.io/hj9a7, and the complete dataset along with analysis code is available at https://go.osu.edu/pathogaze.