Koala Gripper: Co-designing Robotic Grippers and Data-Capture Devices for Scaling Dexterous Manipulation Learning

AI summary

Overview

Research area: Robotics — dexterous manipulation, gripper mechanism design, handheld data-capture devices for imitation learning.

Technical level: Intermediate. The paper combines mechanical linkage design, kinematic/inertial modeling (operational space inertia, reduced Jacobians) and imitation-learning policy training, but the arguments are presented in accessible terms.

Scope: The paper proposes a co-design framework for jointly developing a handheld data capture device (CD) and its matching robotic gripper (RD), and applies it to build and validate the Koala Gripper system through force characterization, grasp demonstrations, and diffusion-policy execution.

What This Paper Is About

Robotic manipulation lacks the large-scale real-world data that language and vision models enjoy, and handheld data-capture devices are a promising way to collect it cheaply. However, existing capture devices tend to simply copy the morphology of existing robot grippers, which hurts ergonomics and manipulation performance. This paper introduces a design process that develops the human-held capture device and the robot-executed gripper at the same time, then presents the resulting Koala system: a three-degree-of-freedom gripper with linkage fingers, a pivoting "dual thumb," and a matching handheld trigger-driven capture device.

Key Contributions

1. A co-design framework that weaves constraints and performance targets from both the handheld data capture device (CD) and the robotic device (RD) into common design elements, iterated through a feedback loop (Fig. 2) starting from shared task requirements and workspace definitions.
2. The Koala gripper architecture: two 1-DoF preshaping underactuated linkage fingers opposing a 1-DoF pivoting monolithic "dual thumb," totaling three controllable DoFs, built around a 9-bar finger linkage derived from an existing 7-bar mechanism and integrated with a crossed 4-bar fingertip.
3. A force-optimized trigger/finger linkage in the CD that mimics natural human fingertip closing paths and whose mechanical advantage compensates for the finger linkage's deleveraging near end-of-stroke singularities, plus a backdrivable RD actuation package (4 mm pitch ballscrew, TQ ILM25x08 frameless motor, 18.8:1 planetary reduction for the thumb).
4. System validation through a force-parity study between CD and RD, an effective-mass characterization of the actuated fingers in MuJoCo, qualitative grasp and tool-use demonstrations, and an end-to-end data collection plus diffusion-policy execution pipeline.

Main Findings

• Backdrivability and reflected inertia: Effective mass at the fingertip with the RD actuator is on the order of tens of grams along the finger trajectory. The belted-ellipsoid plots show asymmetric effective-mass profiles traced to linkage singularities and the underactuated DoF, with singularity-related high-mass directions generally aligned with reaction forces from gripped objects, aiding grasp stability.
• Directional mass behavior: Effective mass is low in the closing direction (the underactuation complies) and higher in the opening direction, where the linkage reduces to a 1-DoF system against a hard stop — giving low reflected mass for adaptation without losing stiffness while grasping. A high mass in the negative z-direction at the start of the stroke supports pressing and pushing tasks.
• Force parity between CD and RD: With the CD trigger loaded under 100 N of input force and ten trials per thickness measured with a Biometrics P200 coin load cell, the RD matched the CD at all tested thicknesses. At 10 mm: CD 25.88 ± 0.64 N vs RD 25.06 ± 0.45 N (−3.17% ± 4.25%). At 20 mm: 23.84 ± 0.73 N vs 24.88 ± 0.91 N (4.36% ± 6.73%). At 30 mm: 24.41 ± 0.65 N vs 26.39 ± 0.88 N (8.11% ± 6.00%). At 40 mm: 24.34 ± 0.77 N vs 26.81 ± 1.16 N (10.15% ± 7.50%).
• Grasp workspace: The design targets a grasp range from 0 mm pinch grasps to power grasps spanning 7 mm–100 mm diameter, with pads of 2–3 mm compliance and Shore 30A polyurethane. Testing demonstrated a maximum power grasp diameter of 115 mm and a minimum grasp diameter of 7 mm; the 44 mm cylindrical handle diameter is called out as the average size of handles designed for humans.
• Object and tool variety: The gripper grasped and manipulated large/unwieldy objects, objects with uneven weight distribution (mug, hammer), and complex-functionality objects (spray bottle, drill), including a 25 lb sandbag, a 2 L bottle, and multiple YCB-set objects. Rigid metal fingernails enabled singulation down to grasping a nail and picking thin objects such as a card off a flat surface.
• Beyond-morphology capability: High-force pinch grasps were shown to emulate effects of certain abducted-thumb grasps such as the key grasp.
• Learning from demonstration: A diffusion policy with a shared vision encoder across all camera streams, with action and state dimensions expanded to include the 3 DoF end-effector, was trained and rolled out on a 7 DoF Franka Emika FR3 under impedance control. 250 handheld demonstrations were collected for pasta straining and 100 teleoperated demonstrations for cup destacking.
• Ergonomics grounding: A survey of 30 potential users informed three grip sizes, defined by the grip-to-fingertip distance, the grip-to-trigger distances, and foam insert thickness.

Methodology in Plain English

The researchers started by writing down what tasks the system must do and which regions of space around the gripper must stay clear — for example, keeping the bottom open for table-top picks, the tips open for button pushing, and a keep-out region so the human operator's hand is not boxed in. From there they defined separate constraint lists for the handheld device (ergonomics, mechanical advantage, user-controllable degrees of freedom) and the robot device (actuator inertia and packaging), then fed both sets into shared design elements and iterated. Keeping the user's DoFs low pushed the design away from a human-like hand and toward two underactuated linkage fingers with a pivoting dual thumb, so the same finger can meet the thumb for pinch grasps and bypass it for wrap grasps. The trigger was built as a 4-bar linkage tracing a path similar to a human fingertip's closing motion, so the trigger's mechanical advantage curve cancels the finger linkage's deleveraging at the end of stroke. They then measured how "heavy" the motorized finger feels to an external push by modeling the linkage in MuJoCo with equality constraints, computing a reduced Jacobian and reduced mass matrix, and mapping these to an operational space inertia matrix at the fingertip. Finally, they compared grasp forces between the handheld and robot versions, demonstrated grasps on a range of objects, and trained a diffusion policy from collected demonstrations to execute tasks on a Franka FR3 arm.

Why This Matters

Impact on research: The paper argues that treating data-capture hardware and execution hardware as a single coupled design problem — rather than as an afterthought pair — is a route to higher-capability manipulation datasets. It also shows that a low-DoF (three controllable DoFs) non-anthropomorphic morphology can achieve grasps that are usually associated with more complex hands, and provides an inertial characterization method for underactuated linkage fingers.

Real-world applications:

• Scaling real-world manipulation datasets through cheap, portable in-the-wild handheld collection plus teleoperated on-robot collection using the same device.
• Tool use tasks in unstructured settings, such as fastening a bolt with a standard drill, spraying, or hammering.
• Industrial and warehouse picking of objects from flat surfaces, including thin flat items, via the fingernail singulation features.
• Household manipulation such as handling bottles, cans, bowls, plates, and heavier items like a 25 lb sandbag.

Industry relevance: The shared parts between capture device and robot gripper (interfaces, finger parts, actuator choices), the reliance on standard off-the-shelf components (frameless motor, ballscrew, planetary reduction, global-shutter cameras), and the explicit target of 100 N maximum human trigger input and force matching between human-collected and robot-executed grasps are all framed as practical manufacturing and deployment considerations for companies building manipulation data pipelines.

Future Directions

• Thumb adduction DoF: Adding an actuated adducting degree of freedom is described as the most direct route to grasps that remain out of reach, such as in-hand re-orientation of small parts, at the cost of additional user mental load.
• Force sensing: Work is underway to add wrench and grip force sensing to the CD, since the lack of force-state information in collected data limits how much of the system's high force output learned policies can exploit; this would enable force-reactive policies.
• Ergonomics and operator fatigue: Operator fatigue is named as a limiting factor on data collection speed; proposed avenues include moving the operator's hand closer to the gripper's active surfaces, reducing wrist torques, and running user studies on fit and comfort.
• Embodiment gap characterization: Quantifying how the gap between handheld and teleoperated supervision modalities scales with task complexity and dataset size is left to future work, and policy demonstrations are limited to the two reported tasks.

Target Audience

Robotics researchers and engineers working on gripper mechanism design, handheld and teleoperated data collection pipelines, imitation learning, and learning-from-demonstration systems. It is also useful for product and hardware teams in robotics companies who need to weigh ergonomics, manufacturability, and actuator packaging against manipulation capability, and for graduate students looking for a worked example of coupling mechanism design with learned policy execution.

Authors’ abstract

As the demand for larger manipulation datasets grows, handheld robotic gripper data collection and the associated gripper designs become more vital. Current data collection device designs trend towards matching the morphologies of existing robotic grippers, sacrificing ergonomics and manipulation performance. In this paper, we propose a co-design framework that guides the simultaneous development of both data collection and robotic execution devices by weaving both platform constraints into the design process. Through this workflow, we present the Koala Gripper system, a data capture device and robotic gripper platform that improves dexterity and grasp capability compared to parallel jaw grippers while preserving scalability and ease-of-use. The design introduces a novel force-optimized finger/trigger linkage mechanism with directional reflected mass characteristics, a unique monolithic dual-thumb, and user-centered ergonomic design. The design's actuated robotic fingers are backdrivable, with effective mass on the order of tens of grams. We show that these grippers are capable of secure grasps over a wide range of objects, forceful tool use, and precise singulation. We further validate the platform by deploying it with an end-to-end data collection and policy execution pipeline that highlights its capabilities through learning from demonstration. More information available at http://koalagripper.rai-inst.com