Automatic Field-of-View Adjustment for a View-Expansive Microscope via LSTM-Based Gaze and Pipette Motion Interpretation

AI summary

Overview

• Research area: Human-Computer Interaction, applied to robot-assisted micromanipulation for assisted reproductive technology (intracytoplasmic sperm injection, ICSI).
• Technical level: Intermediate. The hardware and the LSTM pipeline are described clearly, but comfortable reading requires some familiarity with recurrent neural networks, gaze tracking, and microscopy terminology.
• Scope: A single paper describing an LSTM-based system that automatically changes the field-of-view (FOV) of a view-expansive microscope by interpreting the operator's gaze and holding-pipette motion, evaluated in an ICSI-mimicking task with six novice participants.

What This Paper Is About

During ICSI, an operator must repeatedly switch between large-FOV and high-resolution views, and with conventional microscopes this means manually rotating an objective-lens revolver and re-adjusting brightness. These interruptions break concentration, cost procedure time, and require extensive training to master. The authors build an AI model that predicts the appropriate FOV from the operator's gaze position and the holding pipette's position and velocity, so that the microscope changes magnification automatically while the operator keeps both hands on the manipulators.

Key Contributions

1. An LSTM model for FOV prediction. A model that takes eight inputs — holding pipette position and velocity on the monitor, operator gaze position, and the deviation of gaze from the pipette — sampled at 20 Hz, and outputs a single continuous FOV parameter between 0.0 and 1.0.
2. An implicit gaze-based intention interface. Rather than requiring the operator to perform deliberate gaze gestures to trigger a response, the system learns natural task-related gaze behavior, allowing FOV changes to be anticipated rather than commanded.
3. Integration with a view-expansive microscope. The prediction drives a single objective lens (no physical lens exchange) through a rate-limited FOV parameter, with the FOV center anchored to the holding pipette plus the oocyte radius, so the operator does not lose sight of the pipette or the oocyte.
4. A quantitative human study. Six micromanipulation novices performed an ICSI-mimicking microbead task with and without automatic FOV adjustment, with an experienced operator (over five years of micromanipulation experience) on a conventional microscope as a reference benchmark.

Main Findings

• Faster task completion with automatic FOV: The abstract reports the average task completion time falling from 60.5 s to 48.0 s (p < 0.001) with the proposed system.
• Median times: In the violin-plot analysis, novices' median time was 51.5 s with automatic FOV adjustment and 68 s without it; the expert's median on a conventional microscope was 45 s.
• Statistical significance vs. manual adjustment: A Kruskal–Wallis test with Bonferroni correction showed a significant difference between the proposed system and manual FOV adjustment (p = 0.02).
• No significant difference vs. the expert: The comparison between novices using the proposed system and the expert on a conventional microscope was not significant (reported as p > 0.05), which the authors interpret as novices reaching expert-level working speed.
• Every participant improved: Table II shows a reduction in median task completion time for all six participants (A: 73.0 s to 44.0 s for B... reported per participant as A 73.0→? , specifically A 73.0/101.0, B 44.0/64.5, C 56.0/81.0, D 58.5/63.0, E 42.5/47.0, F 65.0/69.5 for automatic/manual conditions).
• Two novices exceeded the expert: Participants B and E had median times lower than the expert's 45 s.
• Two novices improved only marginally: Participants D and F showed only a 4.5 s reduction versus manual adjustment and remained slower than the expert.
• Model accuracy and speed: The coefficient of determination was 0.91, the AI inference time was 3.9–6.0 ms, and the system's FOV response was observed to follow the gaze and pipette motion phases of the task in the presented time series (e.g., FOV adjustment at 12–15 s, FOV expansion at 20–25 s, high-resolution presentation at 25–35 s).
• Attributed cause: The authors attribute the speed gain to removing an interface — the operator no longer switches hands from the joystick to a mouse wheel to change FOV.

Methodology in Plain English

The team built on a microscope their group previously developed that can show both a wide view and a high-resolution view at the same time from one objective lens, using galvanometer mirrors, an electrically tunable lens, and a high-speed camera. They added a screen-based eye tracker (Tobii Pro Nano) so the system knows where the operator is looking on the monitor.

They recorded ICSI procedures performed by an operator familiar with the technique and labeled the appropriate FOV at each moment. Fourteen trials (11,520 steps) were used for training and three trials (2,768 steps) for validation, all from a single expert — deliberately, because different experts and clinics use different technique styles and FOV preferences, which the authors say would create conflicting learning targets ("multimodality"). The trained LSTM outputs a number between 0.0 and 1.0; the system converts it smoothly into an image width that ranges from the highest-resolution capture (corresponding to 500 μm) up to the maximum expanded FOV (up to 1.5 mm), with a rate limit so the view never jumps abruptly. The FOV is centered on the holding pipette plus the oocyte radius.

For evaluation, six people with no prior micromanipulation experience each practiced for ten minutes, then performed a simplified ICSI task six times using 100 μm microbeads — pick a bead up from Workspace 1, move it to Workspace 2, touch it with the injection pipette tip, and release it in Workspace 3. Each participant did this both with and without automatic FOV adjustment (manual adjustment was done with a mouse wheel on the same microscope), and the condition order was counterbalanced. An expert with over five years of experience did the same task on a familiar conventional microscope for reference. The main measure was task completion time.

Why This Matters

Impact on research. This paper is a case study in a specific design argument: that in human-machine systems, reducing the number of interfaces an operator must manage can matter as much as simplifying each individual interface. It also demonstrates that implicit gaze interpretation — as opposed to deliberate gaze commands — can work at the micro-scale, where most prior gaze-based robot control work has been macro-scale.

Real-world applications:

• IVF and ART clinics: Shortening and smoothing the ICSI workflow, and potentially lowering the training burden for new embryologists by giving them expert-like FOV behavior automatically.
• Remote or teleoperated micromanipulation: Reducing the interface load on a remote expert, who otherwise spends part of the procedure on magnification control rather than the injection itself.
• Other micro-scale cell manipulation: Any procedure that alternates between a wide search view and a high-magnification working view under a microscope.
• Gaze-aware surgical and laboratory robotics generally: The implicit-intention approach is transferable to domains where an operator's hands are fully occupied.

Industry relevance. The hardware list is largely commercial off-the-shelf instrumentation (Olympus inverted microscope, Optotune tunable lens and driver, Cambridge Technology galvanometer mirror, Eppendorf microinjectors and micromanipulators, Ximea high-speed camera, EIZO monitor, Tobii eye tracker), and the model is small enough to run locally without internet connectivity — a deliberate choice motivated by patient privacy and connection stability, which the authors argue suits clinical deployment.

Future Directions

• Handling multimodality across experts. Training on a single expert avoids conflicting targets but bakes in that expert's FOV preferences; the authors call for algorithms that can handle multiple experts with differing styles.
• Adaptation to individual operators. Novices whose gaze patterns differ from the expert's benefit less, and the paper suggests adaptive fine-tuning that accounts for individual FOV preferences and gaze behavior.
• Generalization beyond one setup. The static pre-trained model cannot be applied directly to different microscope environments or tasks and would need retraining; supporting multiple task types would require architectural changes.
• Better eye-tracking hardware. Per-operator calibration is time-consuming and not always successful, calibration accuracy varies between sessions, and drift from head movement or lighting can require recalibration — limitations the authors say cannot be fixed by AI improvements alone.

Target Audience

Readers who will benefit most are researchers and engineers working on human-machine interaction for micromanipulation and medical robotics, ICSI and ART practitioners or clinical engineers interested in microscope workflow, and developers of gaze-based assistive interfaces who want an example of implicit rather than explicit gaze control. It is also a useful methodological reference for anyone evaluating an automatic assistance feature against both a manual baseline and an expert benchmark. Readers primarily interested in deep-learning architecture research will find the model itself deliberately simple — the contribution lies in the interface concept and the human study, not in the network design.

Authors’ abstract

Intracytoplasmic sperm injection (ICSI) operators frequently adjust the field-of-view (FOV) during procedures, which interrupts workflow and increases procedure time. Conventional microscopes require manual objective lens switching and illumination adjustments to achieve different FOV sizes. We propose an AI-based automatic FOV adjustment method integrated with a view-expansive microscope. This microscope enables the simultaneous acquisition of a large FOV and high-resolution images using a single objective lens through multiview imaging with galvanometer mirrors and high-speed vision, thereby eliminating the need for physical lens exchanges. Our method utilizes a long short-term memory (LSTM) model to predict the appropriate FOV size based on real-time analysis of the pipette's position and velocity, combined with the operator's gaze position. The AI model is trained using ICSI procedure data from an expert with over five years of micromanipulation experience. Experimental evaluation with novice operators reveals that the proposed automatic FOV adjustment system significantly improves the ICSI procedure speed, reducing the average task completion time from 60.5 to 48.0 s (p &lt; 0.001). The experiments also demonstrate that this improvement enables novice operators to achieve ICSI working speeds equivalent to those of expert operators.