Gesture Matters: Pedestrian Gesture Recognition for AVs Through Skeleton Pose Evaluation

AI summary

Overview

Research area: Computer vision for autonomous driving — specifically pedestrian gesture recognition using 2D human pose estimation.

Technical level: Intermediate. The work combines pose estimation, feature engineering, and classification, and assumes familiarity with concepts like keypoints, normalisation, and static versus dynamic features.

Scope: A gesture classification framework that uses 2D pose estimation on real-world traffic video to sort pedestrian gestures into four classes, evaluated by which body features best separate those classes.

What This Paper Is About

Pedestrians and drivers often rely on hand gestures to communicate intent in situations where formal traffic rules do not clearly apply — for example, waving someone across a junction. Drivers can usually read these signals, but autonomous vehicles struggle to interpret them, which creates a safety and communication gap. This paper's goal is to build and evaluate a gesture classification pipeline that can tell pedestrian gestures apart, and to identify which measurable body features carry the most useful information for doing so.

Key Contributions

1. A gesture classification framework built on 2D pose estimation, applied to real-world video sequences from the WIVW dataset rather than to synthetic or staged data.
2. A four-class gesture taxonomy covering Stop, Go, Thank & Greet, and No Gesture, giving the problem a defined set of categories to distinguish.
3. A feature set of 76 static and dynamic features extracted from normalised keypoints, separating aspects like body configuration from aspects like motion.
4. An analysis of feature discriminative power, identifying hand position and movement velocity as particularly informative for separating gesture classes.

Main Findings

• Hand position matters most: The paper reports that hand position is especially discriminative when distinguishing between gesture classes.
• Movement velocity is also highly informative: Velocity of movement, a dynamic feature, ranks alongside hand position as a key signal for classification.
• Reported classification accuracy: The framework achieves a classification accuracy score of 87% across the four gesture classes.
• Static and dynamic features both contribute: The feature design deliberately combines static (posture-like) and dynamic (motion-like) descriptors extracted from normalised keypoints, rather than relying on one type alone.
• Dataset and real-world grounding: Evaluation uses real-world video sequences from the WIVW dataset, not a controlled laboratory setup.

The abstract does not report per-class performance, confusion between specific gesture categories, or comparisons against alternative feature sets or baselines; those details are not available in it.

Methodology in Plain English

The researchers start from video of real traffic scenes. Rather than trying to recognise a gesture from raw pixels, they first estimate the 2D pose of the person — locating body joints such as hands, elbows, and shoulders as keypoints in the image. Those keypoints are then normalised, which means adjusting them so that differences in position, scale, or framing do not distort the comparison between people.

From the normalised keypoints they compute 76 features. Some are static, capturing where body parts are positioned at a given moment. Others are dynamic, capturing how those positions change over time, such as movement velocity. These features are fed into a classifier that assigns each sequence to one of four categories: Stop, Go, Thank & Greet, or No Gesture. The researchers then examine which features contribute most to telling the classes apart, which is how they arrive at hand position and movement velocity as the standout signals.

Why This Matters

Research impact: The paper connects pedestrian behaviour research with AV perception. By framing gestures as a pose-analysis problem and identifying which features are discriminative, it offers a concrete starting point for others working on non-verbal communication in traffic, and it contributes to understanding pedestrian behaviour in traffic contexts more broadly.

Real-world applications:

• Autonomous vehicles at unsignalised crossings, junctions, or parking lot exits, where a pedestrian's wave or raised hand may be the only signal of intent.
• Driver-assistance systems that alert a human driver when a pedestrian appears to be signalling, rather than only detecting their presence.
• Simulation and testing pipelines for AV planners, where realistic pedestrian gesture behaviour is needed to stress-test decision-making.
• Traffic behaviour research and urban design studies that need a repeatable way to measure how often and how pedestrians gesture.

Industry relevance: AV developers, perception teams, and automotive suppliers need to handle edge cases where formal right-of-way rules are ambiguous and humans fall back on gestures. A pose-based approach is attractive to industry because it works from skeleton keypoints rather than requiring gesture-specific appearance training, and because the feature analysis tells engineers where to focus sensor and model effort.

Future Directions

• Generalisation beyond the WIVW dataset: The abstract reports results on one dataset. Whether the four-class framework and its feature rankings hold on other countries, camera viewpoints, weather conditions, and pedestrian demographics is an open question.
• Improving on the reported accuracy: The abstract reports 87% accuracy without detailing where the remaining errors fall. Understanding which gestures are confused with each other — and whether "No Gesture" is over- or under-predicted — would guide refinements.
• Extending to 3D pose and richer temporal modelling: The work uses 2D pose estimation. Moving to 3D could resolve ambiguities where a 2D projection makes different gestures look alike, and richer temporal models might capture gesture dynamics that velocity alone does not.
• Integration into an actual AV pipeline: The abstract positions the work as improving AV perceptual capabilities, but the path from offline classification on video sequences to real-time, in-vehicle inference — including latency, sensor placement, and interaction with planning — is not addressed.

Target Audience

Researchers and graduate students in computer vision, human-robot interaction, and autonomous driving perception; AV perception engineers evaluating pose-based approaches to pedestrian intent; and traffic psychology or human factors researchers interested in how pedestrian gestures can be formalised into measurable features. Readers looking for a complete benchmark comparison or ablation study will not find it here, since the abstract reports only the headline accuracy and feature findings.

Authors’ abstract

Gestures are a key component of non-verbal communication in traffic, often helping pedestrian-to-driver interactions when formal traffic rules may be insufficient. This problem becomes more apparent when autonomous vehicles (AVs) struggle to interpret such gestures. In this study, we present a gesture classification framework using 2D pose estimation applied to real-world video sequences from the WIVW dataset. We categorise gestures into four primary classes (Stop, Go, Thank & Greet, and No Gesture) and extract 76 static and dynamic features from normalised keypoints. Our analysis demonstrates that hand position and movement velocity are especially discriminative in distinguishing between gesture classes, achieving a classification accuracy score of 87%. These findings not only improve the perceptual capabilities of AV systems but also contribute to the broader understanding of pedestrian behaviour in traffic contexts.