Fingers as Legs: Learning Self-Supported Locomotion and Manipulation with an Anthropomorphic Hand

AI summary

Overview

Research area: Robotics — reinforcement learning for loco-manipulation, applying legged-robot locomotion techniques to a multifingered anthropomorphic robotic hand.

Technical level: Intermediate. The paper assumes familiarity with reinforcement learning (PPO, reward shaping, domain randomization) and sim-to-real transfer for legged robots, but the core ideas are explained in accessible terms and the hardware setup is described concretely.

Scope: The paper demonstrates that an off-the-shelf commercial anthropomorphic hand (WUJI right hand) can use its own fingers as legs to crawl untethered, steer, recover from falls, press keyboard keys, and push objects — all without modifying the hand's finger kinematics or its built-in position controller.

What This Paper Is About

Most robotic hands are carried to a workspace by an arm. This paper asks whether a hand can instead move itself there — crawling on its fingers like the character "Thing" in The Addams Family — so it could be deployed into confined spaces and later retrieved. The core problem is that the same fingers must simultaneously move the body, support its weight, and interact with the environment, which is hard because the hand's fingers are unequal in length and the thumb is opposed, so no fingertip has the same reach as another. The goal is to learn policies that handle this dual role on a commercial hand while keeping its existing finger design and position controller intact.

Key Contributions

1. System: An 818 g mobile manipulator built from a commercial anthropomorphic hand (738 g hand plus an 80 g dorsal module), carrying its own four-cell lithium-polymer battery, Raspberry Pi Zero 2 W, and BNO085 IMU. It runs untethered and uses the same fingers to walk, support itself, and interact, while retaining the vendor's finger kinematics, factory controller gains, and position-control interface.

2. Method: A locomotion reward formulation built around the hand's own stance. A footprint objective pulls each of the five unequal fingertips toward its own nominal stance position (captured from a settled stance and expressed in a body frame that is level at that stance), while a lift objective encourages a stepping rate that scales with commanded speed and leaves the choice of which finger steps, and when, to the policy.

3. Evaluation: A simulation ablation comparing the formulation against rewards adapted from quadruped locomotion (raw and tuned stock baselines), plus hardware demonstration of untethered crawling on 14 surfaces, steering, fall recovery, successive keyboard presses without vision, and vision-guided object pushing.

Main Findings

• Speed versus stock rewards: In simulation, the hand moves faster with the authors' reward formulation than with either tested stock baseline. Compared with the tuned stock baseline, it is 0.65 cm/s faster on average (95% confidence interval [+0.26, +1.02] across twelve seeds) and faster in 10 of 12 seeds.

• Participation versus contact posture: Speed alone hides how the fingers touch the ground. The tuned stock baseline reaches a higher mean five-finger participation, although that difference is uncertain, but it plants fingers on their nails more in every seed. The authors' formulation has higher non-nail participation (+0.35 [+0.18, +0.53]), which is why participation and contact posture must be read together.

• Footprint objective drives the gains: Removing the footprint objective reduces task speed by 0.79 cm/s [+0.37, +1.19] and lowers five-finger participation. At the deployed weight, the objective shifts mean contact tilt across fingertips 8.8° toward the nail side ([+1.4, +15.9]) — so it improves five-finger participation without improving every aspect of contact posture.

• Lift and direction objectives shape rhythm, not speed: The ablations do not establish an independent speed benefit from the auxiliary lift or direction objectives, whether removed separately or together. Removing the direction objective alone increases five-finger participation by 0.31 [+0.07, +0.52], while its contact-posture effect remains uncertain. The authors retain it because the hardware policies were trained with it.

• Weight sensitivity: Doubling the footprint weight gives a higher worst-tip planting fraction than doubling the lift weight (paired difference +0.22 [+0.11, +0.33], in 10 of 12 seeds), while their speed difference is uncertain (−0.08 cm/s [−0.39, +0.21]). Above the deployed footprint weight, the worst-tip planting fraction keeps improving while task speed changes remain uncertain; mean five-finger participation peaks at twice the deployed weight and decreases again by 3.3 times. Higher weights were tested only in simulation.

• Untethered crawling: The hand crawled untethered under gamepad control across 14 indoor and outdoor surfaces (rubber mat, carpet, hardwood, tile, diamond plate, metal grating, hard court, asphalt, dry concrete, cut stone, weathered stone, artificial turf, grass, and gravel). These runs are described as a qualitative demonstration, not a measured success-rate study.

• Steering asymmetry: Without a steering command, the hand drifts right by about 6° s⁻¹ regardless of initial heading. Across 21 trajectories, mean path speed was 0.093 m/s. Right-turn rate scales with the command, while left-turn rate plateaus. With a proportional controller on the onboard IMU heading, the hand reached the target for both ±15° steps and the 30° right turn; 30° left turns fell short because the drift-cancelling offset pushes left commands beyond the training limit.

• Fall recovery: In simulation, the full procedure rights the hand in 28 of 32 simulated falls, with a median time of 6.1 s and a mean absolute joint error of 0.004 rad relative to the crawl stance; in the remaining four episodes the upright detector did not fire within 20 s. On hardware, the hand recovered in 21 of 25 trials (84%), including 11 of 14 thumb-side falls and 10 of 11 wrist-side falls. In the four failures, fingers caught on each other and the hand stalled.

• Keyboard pressing without vision: The hand executed 29 correct presses from 32 consecutive keyboard commands over 72.5 s, without keyboard realignment, maintaining its support stance with a maximum tilt of 7.7°. All three failures were Up-key commands that activated the adjacent Right Shift key. Median command-to-keystroke latency among correct presses was 0.25 s. Through the same interface, the hand executed optimal 9- and 12-move Sokoban solutions for one- and two-box levels.

• Vision-guided pushing: Using a 40 × 40 × 40 mm PLA cube weighing 41.4 g, one policy approached and pushed the cube to targets. Across 15 deliveries at target distances of 10 cm to 40 cm (presented as a demonstration), final target error averaged 17 mm (range 5 mm to 37 mm), under half the cube's side length.

Methodology in Plain English

The authors took an off-the-shelf right hand with 20 actuated joints (four per finger) and added a dorsal module with battery, IMU, and a Raspberry Pi, bringing the mass to 818 g. The joints are non-backdrivable, are limited to 1.0 A, and are driven through the vendor's existing position controller with a 3 Hz low-pass filter on commands. An onboard ROS 2 stack runs a 500 Hz serial driver and runs policy inference at a nominal 50 Hz, with crawl commands arriving over a 2.4 GHz gamepad link so Wi-Fi stays outside the locomotion control loop. A safety supervisor monitors communication, electrical limits, and attitude.

Each task gets its own policy — a small multilayer-perceptron network trained with PPO in NVIDIA Isaac Lab using 4096 parallel environments and 20 s episodes, and deployed as a 32-bit floating-point ONNX model. Every policy sees 46 common inputs per control step (each joint's angle relative to a fixed reference, gravity direction, angular velocity, and the previous policy output), each retained as eight samples of history, plus task-specific inputs. The network does not command joint positions directly; it outputs an increment that is scaled by a task-specific angle (0.060 rad for crawl/recovery, 0.028 rad for keyboard, 0.040 rad for pushing) and added to the previous target.

Because the policies act through the hand's own position controller, the simulator is calibrated from measurements on the real hardware: joint frequency sweeps give a stiffness scale of 18.7 ± 2.5 times the nominal model; three loaded fingertip pulls give a kinetic fingertip friction of 0.88 (range 0.80–0.97); a timed command response gives a closed-loop delay of about 19 ms; ramps give filtered joint speeds of 2.8 and 2.5 rad/s for flexion and abduction; and trajectory comparison gives a 3 Hz command filter cutoff. Simulation also includes a one-step command delay and the same 3 Hz filter. Training then randomizes friction, effort scale, payload center-of-mass offsets, palm mass, palm center of mass, and IMU bias, plus actuator gains around the calibrated stiffness values.

The reward design is the paper's central methodological idea. The hand's stance — captured by holding fixed joint targets for 4 s in simulation with the 80 g payload — defines a frame that removes the palm's nominal tilt and aligns forward with the direction from palm to the centroid of the index-to-little fingertips' origins. In that frame, each of the five fingertips is pulled toward its own nominal position by a virtual-spring penalty with a diagonal weight matrix of (0.25, 1, 1), so lateral and vertical departures are penalized four times as strongly as fore–aft motion; this leaves room for stepping forward while discouraging sideways and vertical drift. A separate lift objective only encourages a stepping frequency that grows linearly with commanded speed (mapping 0.02 m/s to 0.16 m/s onto 1 Hz to 3.5 Hz), gated on a fingertip being airborne with force below 0.15 N, on contact having occurred since the last phase wrap, and on commanded speed being at least 0.02 m/s. Yaw-rate tracking is introduced by curriculum at the same 18,000 per-environment steps at which the lift weight begins decaying, so the policy does not learn stepping and turning at once.

For manipulation, separate policies reuse the same simulator, proprioceptive history, and PPO setup. The keyboard policy learns four nominal press locations and takes a requested-key indicator as input, relying on maintained alignment rather than vision. The pushing policy uses an overhead camera tracking a dorsal marker and the object, and was trained with a 60 Hz sample-and-hold camera model including one-to-three control steps of latency, 3% dropout, and 2 mm position noise.

Why This Matters

This work shows a commercially available anthropomorphic hand, with its uneven finger geometry and its factory position controller untouched, can act as a self-contained mobile manipulator rather than a component that needs an arm to carry it. For robotics research it is a counterexample to the assumption that hand morphology must be redesigned for symmetry and modularity — the authors explicitly contrast with prior systems that use six identical hand-finger modules or purpose-built symmetric hands. It also introduces a stance-anchored reward formulation for a robot with no left–right symmetry, which sidesteps the mirroring requirement that common symmetry-exploiting locomotion methods rely on.

Real-world applications:

• Confined-space inspection and operation: A larger robot could place the hand near a restricted opening, have it crawl to a control or object, interact, and return for retrieval — the motivating scenario in the paper.
• Operating human interfaces in hazardous or inaccessible areas: The keyboard-pressing demonstrations show physical keystrokes on standard hardware, which matters for legacy control panels that have no digital interface.
• Gentle object relocation: The vision-guided pushing results point to moving objects within a limited workspace using overhead tracking, useful where a full mobile manipulator cannot fit.
• Search-and-rescue or maintenance in cluttered terrain: The 14-surface crawl demonstration and fall recovery show the platform is not restricted to a single prepared floor.

Industry relevance: the paper's value proposition is parts reuse — a commercial hand plus an 80 g add-on module, with no change to the vendored controller. That is relevant to companies already building multifingered hands who want to extend them into small mobile manipulators without a parallel locomotion mechanism.

Future Directions

• Extending the range of commanded heading changes. The paper reports that 30° left turns fell short on hardware because the drift-cancelling offset pushes left-turn commands beyond the training range, and lists extending this range as an explicit next step.
• Integrating onboard perception. The authors propose onboard hand and object tracking so pushing is not limited to the overhead camera's workspace.
• Automating keyboard alignment. Currently an operator aligns the keyboard with trial presses, and misalignment requires manual realignment; the authors suggest visual registration could remove this step.
• Open question on reward components. The ablations did not establish an independent speed benefit from the lift or direction objectives, and higher footprint weights were tested only in simulation, so the transferability of those higher weights to hardware remains untested.

Target Audience

Researchers and engineers working on legged locomotion, loco-manipulation, and multifingered robotic hands will benefit most, particularly those interested in sim-to-real transfer through an existing position-control interface and in reward design for robots lacking left–right symmetry. Roboticists evaluating whether a commercial hand can be turned into a self-contained mobile platform, and students with some reinforcement learning background looking for a concrete reward-engineering case study, will also find it useful.

Authors’ abstract

A walking robotic hand must use the same fingers to move its body, support its weight, and interact with the environment. We show how an anthropomorphic hand can learn these skills while retaining its finger design and position controller. Onboard power and computation make the platform self-contained. Our reinforcement learning approach accounts for the hand's unequal fingers, with training in a simulator calibrated from hardware measurements. In simulation, the hand moves faster with our reward formulation than with tuned rewards originally designed for quadrupeds. On hardware, task-specific policies enable untethered crawling, steering, and fall recovery. While supporting its own weight, the hand also executes successive keyboard commands without vision and pushes an object to targets using overhead visual feedback. These results demonstrate a compact mobile manipulator that reuses its fingers for locomotion and interaction, without a separate locomotion mechanism.