Botany-Bot: Digital Twin Monitoring of Occluded and Underleaf Plant Structures with Gaussian Splats

AI summary

Overview

• Research area: Robotics for agriculture, specifically 3D plant phenotyping, view synthesis (Gaussian Splatting), and robot-plant interaction.
• Technical level: Advanced.
• Scope: The paper presents Botany-Bot, a system that builds "annotated digital twins" of living potted plants from a lightbox-and-turntable scan rig, segments the Gaussian Splat model into plant parts, and uses an industrial robot arm to physically lift or push leaves aside so a high-resolution camera can image surfaces that were previously occluded.

What This Paper Is About

Commercial plant phenotyping systems rely on fixed cameras, so a great deal of plant structure is hidden behind leaves and never measured. The paper's goal is a full pipeline that (1) reconstructs a plant in 3D from a small number of cameras, (2) automatically splits that reconstruction into leaves and stems, and (3) uses a robot arm to gently manipulate each leaf so that fine details such as stem buds, leaf veins, and leaf undersides can be photographed with a high-resolution camera and indexed back to the correct leaf ID.

Key Contributions

1. Botany-Bot pipeline: A novel pipeline for autonomously creating high-fidelity 3D digital twins of plants using off-the-shelf hardware (a lightbox, a digital turntable, two stereo cameras, and a robot arm).
2. Segmented digital twin model: A digital twin representation for agricultural phenotyping that segments Gaussian Splats into constituent parts (leaf top, leaf bottom, side stem, main stem), extending the authors' prior work on GARField.
3. Robot leaf-manipulation primitives: Algorithms for a robot to autonomously lift or push individual leaves aside in order to capture additional high-resolution images of their occluded surfaces, formulated as an inspection planning problem that maximizes view coverage.
4. Experimental validation: Results suggesting the pipeline accurately measures physical properties of plant components, evaluated on 109 leaves from 8 real plants.

Main Findings

• Segmentation and detection accuracy: The abstract reports that Botany-Bot segments leaves with 90.8% accuracy and detects leaves with 86.2% accuracy. The results section separately states that the method correctly segments 91% and detects 91% of leaves, without using a task-specific leaf segmentation model or resolving the multiple-mask ambiguity in Segment Anything.
• Physical measurement accuracy: Leaf length and width estimates were compared against manual ruler measurements on three random leaves per plant; the average absolute error (MAE) was 2.0 cm. The authors state the calculated values tend to underestimate leaf dimensions, because pepper leaf length in ground truth includes part of the connection to the stem that GARField tends to exclude, and because the oriented bounding box method ignores leaf curvature and geodesic distances.
• Per-plant detection and segmentation (Table II): Pepper #1 22/23 segmented and 22/23 detected; Pepper #2 13/17 and 13/17; Pepper #3 21/22 and 21/22; PP-Philodendron 7/7 and 6/7; Alocasia 6/6 and 5/6; Anthurium 10/12 and 11/12; Chroma Belize 11/13 and 9/13; Croton 9/9 and 7/9.
• Failure modes for segmentation: The only leaves that remain unsegmented are very small (less than or equal to 1 cm in length) or very close to a dense area such as where the plant branches into multiple stems or near the pot and soil. When two leaves are segmented as one (for example, Anthurium), the authors count this as a failure and count neither leaf.
• Robot manipulation accuracy: Across 68 safely accessible leaves from 8 plants, the robot successfully lifted or pushed 53 leaves, which the abstract reports as 77.9% accuracy. Of those 53 leaves, the oversides/undersides were visible in 41 cases, reported in the abstract as 77.3% for taking detailed overside/underside images.
• Per-plant interaction results (Table III): Pepper #1 9/15 lifted with 9/9 observed; Pepper #2 8/9 lifted with 5/8 observed; Pepper #3 8/10 lifted with 2/8 observed; PP-Philodendron 6/6 lifted with 5/6 observed; Alocasia 3/4 lifted with 3/3 observed; Anthurium 8/9 lifted with 8/8 observed; Chroma Belize 8/9 pushed with 6/8 observed; Croton 3/6 pushed with 3/3 observed.
• Image quality gain from interaction: The images captured after manipulation are significantly more visually detailed, showing leaf veins and the node connecting the leaf to the stem, while the Gaussian Splat reconstructions are under-reconstructed at the overside/underside with blotchy color artifacts.
• Alpha loss ablation: Without the added alpha loss penalizing out-of-mask pixels, reconstruction quality drastically degrades because the quasi multi-view capture introduces floaters.
• Main failure causes for observation: Most failure cases (6/12) were caused by a nearby leaf being lifted or pushed and blocking the target leaf's underside from the camera; another failure mode (5/12) was the target leaf not being lifted or pushed enough to expose its underside/overside. The lift/push distance selection was a naive implementation, and the gripper did not always orient parallel to the leaf surface.
• Physical metrics from Table I (mean ± standard deviation): Pepper #1 leaf height 25.4 ± 6.4 cm, leaf area 18.4 ± 15.3 cm², 22 leaves; Pepper #2 25.8 ± 5.7 cm, 16.1 ± 7.8 cm², 13 leaves; Pepper #3 31.5 ± 4.5 cm, 11.2 ± 8.5 cm², 21 leaves; PP-Philodendron 23.3 ± 3.2 cm, 46.1 ± 13.5 cm², 6 leaves; Alocasia 30.8 ± 6.9 cm, 55.8 ± 35.5 cm², 5 leaves; Anthurium 27.3 ± 6.8 cm, 38.2 ± 12.2 cm², 11 leaves; Chroma Belize 20.7 ± 5.0 cm, 40.1 ± 21.7 cm², 9 leaves; Croton 19.7 ± 4.4 cm, 27.8 ± 10.5 cm², 7 leaves.
• Illustrative leaf metrics: The highlighted leaf in Figure 1 has a leaf area of 25.6 cm², leaf length of 7.2 cm, and ground height of 27.2 cm.
• Rendering performance: The Gaussian Splat models can be viewed in High Definition at a high frame rate of 30 frames per second.
• Cost: The lightbox, two ZED 2 stereo cameras, and digital turntable total $1,137.88 (1,025.46 Euros); the two cameras alone cost $598.99 (539.81 Euros), bringing the system cost apart from the robot to $1,736.87 (1,565.27 Euros). The YuMi Bimanual robot costs $71,277 for a full configuration, according to an email received from ABB on March 5, 2025. Only one arm of the bimanual robot is used in this instantiation.

Methodology in Plain English

Botany-Bot separates the problem into two halves. The scan half puts a potted plant on a digital turntable inside a well-lit lightbox, with two ZED 2 stereo cameras oriented vertically (giving four angles of elevation total). The turntable rotates in steps of 15°, producing 24 images per camera per full revolution and up to 96 total views. ArUco markers on the turntable and on each plant let the system compute accurate multi-view camera poses, with turntable positioning accurate to ±0.1° relative to calibration. Because the plant rotates, the rest of the scene moves too, which breaks standard assumptions of NeRF and Gaussian Splatting. The authors fix this in two ways: they mask the potted plant using Segment Anything 2 and add an alpha loss penalizing pixels outside the mask, and they use the lightbox to keep lighting directionally uniform as the plant turns. Reconstruction uses Splatfacto, and 3D segmentation uses GARField, extended from the authors' earlier work, to split the Gaussians into discrete clusters.

The inspection half takes that segmented model and decides what to do with it. The system filters clusters to find leaves by rejecting the three bottommost clusters (pot, pot texture, soil), the two tallest clusters (pot, stem), and any cluster with fewer than 100 points as noise. Leaf centers come from the mean of Gaussian centers, and leaf directions from principal component analysis. Leaf area is estimated by fitting an oriented bounding box, projecting along the smallest dimension, fitting a 2D ellipse, and computing its area; length and width come from the longest and second longest bounding box dimensions.

For each target leaf, the robot executes three sequential primitives: rotation alignment (the turntable turns so the leaf's principal axis aligns with the high-resolution camera's z-axis), tool positioning (a custom ring-shaped end effector moves directly above or underneath the leaf center, parallel to the leaf surface), and manipulation (the tool moves up or down while rotating, lifting or pushing the leaf). A YuMi IRB 14000 robot arm executes these motions, planned with the Jacobi Motion Library, using a Logitech Brio camera for high-resolution images and a Zed mini camera for depth alignment. The lift/push distance is set to 65% to 90% of the longest dimension of the leaf's oriented bounding box, determined empirically, and only leaves over 5 cm in length are lifted. The whole thing is framed as maximizing view coverage: the ratio of the visible leaf surface area in the camera's field of view to the actual leaf surface area.

Why This Matters

Impact on research. The paper shows a concrete way to combine fast 3D view synthesis with physical robot interaction, addressing a limitation that both fixed-camera phenotyping systems and pure reconstruction methods share: they can only report what the cameras happen to see. It also documents a specific engineering fix (the alpha loss) for applying Gaussian Splatting to quasi multi-view data where the object rotates rather than the camera, and it reports negative results and failure modes rather than only successes.

Real-world applications:

• Plant breeding: Accurate leaf-level height and area measurements support selection decisions aimed at increasing yield, which the paper cites as a motivation for phenotyping.
• Pest and disease inspection: Leaf undersides are where many infestations and lesions appear first, and the paper explicitly motivates underside imaging with pest control and disease inspection.
• Growth monitoring: Repeated annotated digital twins could track how a plant changes over time, an application the authors list as future work.
• Agricultural robotics: The gentle, non-destructive lift/push primitives contrast with structurally destructive tasks such as mechanical weeding, fruit harvesting, and sample collection, and could inform inspection robots that must handle delicate plant structures.

Industry relevance. The authors include researchers from Siemens Research Lab, and the paper includes a cost breakdown distinguishing the affordable sensing hardware (roughly $1,736.87 apart from the robot) from the $71,277 bimanual robot. This framing points at a practical industrial question: how much of a phenotyping pipeline can be built from off-the-shelf components, and what role remains for expensive manipulators. The authors also propose future use of lower-cost platforms such as ALOHA and YAM.

Future Directions

• Closed-loop servoing and better motion planning. The authors identify collision-free, contact-aware motion planning with the 3D plant model as necessary to "burrow" between leaves carefully, and suggest closed-loop motion that takes an image just before losing leaf contact.
• Smarter lift/push distance selection. The current distance selection is described as naive; the authors propose making the lift/push amount depend on the leaf's position in the camera image, lifting lower leaves more and pushing higher leaves more, plus better plant-specific tuning of the end effector orientation.
• Updating reconstructions with the new images. Currently the high-definition post-interaction images only annotate the model; updating the actual 3D reconstructions with them remains future work.
• Bimanual manipulation and cheaper hardware. Only one arm of the bimanual robot is used, and the authors plan to extend to bimanual manipulation and consider the ALOHA and YAM platforms.
• Improved segmentation and longitudinal twins. The authors suggest tuning SAM 2 with plant data or exploring ultrametric feature fields for hierarchical 3D clustering, and they plan to capture and maintain agricultural digital twins over time, potentially using dynamic multi-view reconstruction.
• User study. A user study is proposed to evaluate how useful agricultural practitioners find the system for phenotyping.

Target Audience

Researchers and engineers working on agricultural robotics, plant phenotyping, and 3D scene reconstruction will find the most value here, particularly those interested in Gaussian Splatting applied to non-static or quasi multi-view capture setups. Roboticists studying non-destructive manipulation of deformable objects will benefit from the task primitives, the reported failure modes, and the observation-versus-manipulation distinction in the metrics. Practitioners evaluating commercial or custom phenotyping hardware will find the cost breakdown and the limitations section useful for deciding whether the approach fits their scale of operation.

Authors’ abstract

Commercial plant phenotyping systems using fixed cameras cannot perceive many plant details due to leaf occlusion. In this paper, we present Botany-Bot, a system for building detailed "annotated digital twins" of living plants using two stereo cameras, a digital turntable inside a lightbox, an industrial robot arm, and 3D segmentated Gaussian Splat models. We also present robot algorithms for manipulating leaves to take high-resolution indexable images of occluded details such as stem buds and the underside/topside of leaves. Results from experiments suggest that Botany-Bot can segment leaves with 90.8% accuracy, detect leaves with 86.2% accuracy, lift/push leaves with 77.9% accuracy, and take detailed overside/underside images with 77.3% accuracy. Code, videos, and datasets are available at https://berkeleyautomation.github.io/Botany-Bot/.