Integrating Skeleton Based Representations for Robust Yoga Pose Classification Using Deep Learning Models

AI summary

Overview

Research area: Computer vision and deep learning applied to human pose estimation and action recognition, specifically yoga pose classification.

Technical level: Intermediate. The paper uses standard, well-known CNN architectures (VGG16, ResNet50, Xception) and off-the-shelf keypoint extractors (MediaPipe Pose, YOLOv8-Pose), so no novel architecture design is required to follow it, but familiarity with transfer learning, skeletonization, and classification metrics helps.

Scope: A benchmarking study that introduces the "Yoga-16" dataset and systematically compares three CNN classifiers across three input modalities (raw images, MediaPipe skeletons, YOLOv8-Pose skeletons) for 16-class yoga pose recognition.

What This Paper Is About

Automated yoga pose classification could reduce reliance on expert instructors, but incorrect postures can cause injury, so the classifier must be reliable. Existing work is fragmented: some studies use only raw images, others use only a single pose-extraction model, and datasets such as Yoga-82 suffer from class imbalance, noise, and limited diversity. This paper builds a cleaner, balanced dataset and then runs a controlled comparison to determine whether skeleton-based representations beat raw images, and which combination of extractor and classifier works best.

Key Contributions

1. Yoga-16 dataset: A curated 16-class yoga pose dataset built by cleaning and combining images from the Yoga-82 dataset and a publicly available Kaggle "Yoga Poses Dataset," with the Goddess Pose class supplemented from Kaggle because Yoga-82 did not contain it. Classes were standardized to approximately 80 images each.
2. Three input modalities compared: Direct (non-skeletonized) images, MediaPipe Pose skeleton images, and YOLOv8-Pose skeleton images, evaluated under an otherwise consistent pipeline.
3. Systematic benchmarking of three architectures: VGG16, ResNet50, and Xception were trained and evaluated on all three input representations across nine distinct experiments, producing accuracy, precision, recall, and F1-score for each combination.
4. Interpretability and validation: Grad-CAM visualizations were applied to interpret model decisions, and K-fold cross validation was implemented to assess generalizability.

Main Findings

• Skeleton representations beat raw images: Every model improved when given skeleton input. VGG16 rose from 86.33% on direct images to 96.09% on MediaPipe skeletons; Xception rose from 84.38% to 93.36%; ResNet50 rose from 66.41% to 88.28%.
• Best overall result: VGG16 with MediaPipe Pose skeleton input achieved 96.09% accuracy, 96.27% precision, 96.09% recall, and 96.10% F1-score — the highest benchmark accuracy reported in the paper.
• MediaPipe outperformed YOLOv8-Pose for skeletonization: For VGG16, MediaPipe skeletons gave 96.09% versus 91.41% for YOLOv8-Pose skeletons. The same ordering held for Xception (93.36% vs. 85.55%) and ResNet50 (88.28% vs. 75.00%).
• ResNet50 was the weakest model in every input setting: 66.41% (direct image), 88.28% (MediaPipe skeleton), and 75.00% (YOLOv8-Pose skeleton).
• Confusion patterns on direct images: VGG16 performed strongly on Chair Pose, Goddess Pose, and Seated Forward Bend Pose but struggled with Fish Pose. ResNet50 struggled with Fish Pose and Warrior 2 Pose while doing better on Side Plank Pose and Warrior 3 Pose. Xception excelled on Dolphin Plank Pose, Downward Facing Dog Pose, and Warrior 3 Pose.
• Dataset statistics: Yoga-16 contains 896 training images, 128 validation images, and 256 test images (16 test, 56 train, 8 validation per class). Brightness averaged 169.86 (std 48.02, min 13.00, max 249.05), contrast averaged 59.34 (std 14.94, min 20.45, max 117.59), and aspect ratio averaged 1.37 (std 0.64, min 0.40, max 4.23).
• Training setup: Initial learning rate 0.001, learning rate reduction factor 0.2, minimum learning rate 1e-6, batch size 32, Adam optimizer with default parameters, 50 epochs, early stopping patience 10, reduce-learning-rate patience 5, categorical cross-entropy loss, and model checkpointing enabled.
• Related-work baselines cited: YogaConvo2D reached 99.62% on skeletonized images and VGG16 reached 95.6% on non-skeletonized images in Garg et al. (1,551 images, five poses); a Random Forest on tf-pose features reached 99.04% (5,500 images); TL-MobileNet-DA reached 98.43% (1,120 images); a transfer learning framework on UCF-101 reached 96.03% (13,320 video clips); VGG16-based transfer learning reached 82% (700 images); Xception reached 82.52% for 82 classes and 92.42% for 6 super classes on 18,488 images from Yoga-82.

Methodology in Plain English

The authors started by cleaning up existing data. From Yoga-82 and a Kaggle dataset they removed images with multiple people, non-human subjects, blurry or poorly lit examples, and zoomed-out subjects where the pose was hard to identify. They cropped images to focus on the person and standardized each class to roughly the same number of images so no pose dominated training.

Each image was then prepared in three ways: left as a raw photograph, converted to a stick-figure skeleton using MediaPipe Pose, or converted to a skeleton using YOLOv8-Pose. MediaPipe's pipeline processes images through CNNs, detects bounding boxes, applies BlazePose for keypoint estimation, removes overlapping detections with non-maximum suppression, and smooths predictions with a Kalman filter. YOLOv8-Pose resizes and normalizes the input, runs it through a CNN backbone, predicts keypoints with a pose head, filters them with non-maximum suppression, discards low-confidence points, and groups the rest into a skeleton graph rendered onto the image.

Three CNN classifiers were then trained on each of the three input types (nine experiments in total). The models were trained from scratch on the Yoga-16 data rather than initialized with ImageNet weights, because the authors argue pretrained weights do not align well with skeletal heatmaps or pose-specific features. Architectures used global average pooling followed by a 1024-unit dense layer and a 16-way softmax output. Training used the Adam optimizer, categorical cross-entropy loss, a decaying learning rate, early stopping, and checkpointing. Finally, Grad-CAM was used to show which image regions drove predictions, and K-fold cross validation was run to test generalizability. All experiments ran on Kaggle using an NVIDIA Tesla P100 GPU, an Intel Xeon CPU at 2.00GHz, 30 GB RAM, 57.6 GB disk, and Linux (Ubuntu) with TensorFlow, PyTorch, and Scikit-learn.

Why This Matters

The study provides a controlled head-to-head comparison that was previously missing: skeletonized versus non-skeletonized input, two different pose extractors, and three classifiers, all on a single balanced dataset. The clear result — that skeleton input improves every architecture and that MediaPipe skeletons plus VGG16 give the best accuracy — gives other researchers a concrete starting configuration and a standardized benchmark (Yoga-16) for future work. It also shows that image resolution and background clutter matter less when the representation is reduced to joint structure, which is relevant for low-cost camera setups.

Real-world applications:

• Home fitness and yoga apps: A phone camera could classify poses in real time and give corrective feedback without an instructor present.
• Injury prevention: Detecting incorrect or unsafe postures early, which the paper explicitly motivates as the reason automated classification matters.
• Remote and resource-limited instruction: The paper frames automated systems as a way to deliver high-quality guidance where expert instructors are unavailable.
• Telehealth and rehabilitation: The same skeleton-based pipeline could flag whether a patient is performing a prescribed movement correctly.

Industry relevance: Fitness technology companies, wearable and smart-camera makers, and digital health platforms could use this benchmark to justify investing in skeleton-based pose classification rather than raw-image models, since the runtimes and accuracy tradeoffs are quantified. The result that a comparably simple model (VGG16) beats a deeper one (ResNet50) on this task is also practically useful for deployment on constrained hardware.

Future Directions

• Report the cross-validation results in full: The paper states K-fold cross validation was implemented to demonstrate generalizability, but the truncated content provided does not report the fold-by-fold numbers.
• Detail the Grad-CAM analysis: Grad-CAM is named as a contribution and shown in figures whose captions are referenced, but the specific interpretability findings for each model and modality are not present in the available text.
• Expand beyond approximately 80 images per class: Yoga-16 is deliberately small and balanced; scaling it while preserving balance and cleanliness is the obvious next step, and the paper notes prior datasets were limited by smaller image counts and narrow pose ranges.
• Move toward 3D or depth-aware pose estimation: A cited related work (Anilkumar et al.) is limited to two-dimensional space with no depth data, and the same limitation applies to the skeleton representations used here (as reported in the paper's related work discussion).
• Test on harder, more diverse data: The paper explicitly criticizes prior work (e.g., YogaConvo2D) for evaluating on datasets lacking real-world challenges such as inter-class similarity, so evaluating Yoga-16-trained models on noisier, in-the-wild footage is a natural follow-up.

Target Audience

Researchers and graduate students working on human pose estimation, action recognition, or sports and fitness analytics will get the most value, particularly those looking for a clean benchmark and a documented comparison of skeletonization methods. Practitioners building fitness or health applications — computer vision engineers, mobile developers, and product teams in fitness technology — will find the concrete accuracy comparisons and the finding that VGG16 with MediaPipe skeletons is the strongest configuration directly actionable. Readers with a basic background in convolutional neural networks and transfer learning will be able to follow the methodology; deep expertise in pose estimation is helpful but not required.

Authors’ abstract

Yoga is a popular form of exercise worldwide due to its spiritual and physical health benefits, but incorrect postures can lead to injuries. Automated yoga pose classification has therefore gained importance to reduce reliance on expert practitioners. While human pose keypoint extraction models have shown high potential in action recognition, systematic benchmarking for yoga pose recognition remains limited, as prior works often focus solely on raw images or a single pose extraction model. In this study, we introduce a curated dataset, 'Yoga-16', which addresses limitations of existing datasets, and systematically evaluate three deep learning architectures (VGG16, ResNet50, and Xception), using three input modalities (direct images, MediaPipe Pose skeleton images, and YOLOv8 Pose skeleton images). Our experiments demonstrate that skeleton-based representations outperform raw image inputs, with the highest accuracy of 96.09% achieved by VGG16 with MediaPipe Pose skeleton input. Additionally, we provide interpretability analysis using Grad-CAM, offering insights into model decision-making for yoga pose classification with cross-validation analysis.