SUGAR: Learning Skeleton Representation with Visual-Motion Knowledge for Action Recognition

AI summary

Overview

• Research area: Skeleton-based human action recognition, combining graph-based skeleton encoders with large language models (LLMs) and vision-language models.
• Technical level: Advanced. The paper assumes familiarity with graph convolution networks, contrastive representation learning (CLIP-style training), vector quantization, LoRA fine-tuning, and LLM token projection.
• Scope: The paper proposes SUGAR, a three-stage paradigm that uses LLM-generated motion text and VLM-generated visual text to supervise discrete skeleton representation learning, then feeds those representations to a frozen LLaMA2 7B for action classification and description.

What This Paper Is About

Skeleton-based action recognizers are lightweight and privacy-friendly, but they struggle with fine-grained daily activities whose joint trajectories are nearly identical — the paper's own example is that drinking water and snacking look almost the same in skeleton form. SUGAR addresses this by generating two kinds of textual prior knowledge (a description of body-part motion from GPT 3.5 turbo, and a description of the visual scene from GPT-4V) and using that text to supervise skeleton representation learning, so that a large language model can then read those representations and classify or describe the action.

The paper frames two open questions: how an LLM can understand skeleton input at all, given the gap between skeleton and text, and how an LLM can distinguish between highly similar actions when appearance information is absent.

Key Contributions

1. Visual-motion knowledge for skeleton learning. Unlike prior skeleton-only paradigms, SUGAR injects both motion descriptions (six body-part movements: head, hand, arm, hip, leg, foot, following the HAKE dataset production process) and VLM-generated visual descriptions to force skeleton representations for similar actions apart.
2. An LLM-based recognition paradigm (SUGAR). The framework classifies actions using LLaMA2 7B with untouched pre-trained weights plus LoRA fine-tuning, and also generates a short description of the action rather than only a class label.
3. Temporal Query Projection (TQP). A new bridging module built from a sequence of Q-Formers with shared parameters, which queries skeleton temporal signals continuously and distills them into a short set of language tokens, replacing pooling or a single linear projection.
4. State-of-the-art and zero-shot results. The authors report top performance on several skeleton-based action classification benchmarks and show that SUGAR generalizes to unseen action categories and even across datasets, which linear-based classifiers cannot do.

Main Findings

• Toyota Smarthome: SUGAR reaches 70.2 (cross-subject), 50.9 (cross-view1), and 67.1 (cross-view2) mean per-class accuracy. Compared with the LLM-based method LLM-AR, it improves cross-subject by 3.2% and cross-view1 by 14.8%.
• NTU RGB+D, NTU RGB+D 120, and PKU-MMD: SUGAR reports 95.2 (cross-subject) and 97.8 (cross-view) on NTU RGB+D; 90.1 and 89.7 on NTU RGB+D 120; and 89.0 and 94.3 on PKU-MMD. The authors describe these as competitive across all evaluation protocols rather than uniformly superior.
• Zero-shot generalization: On two protocols — pre-training on NTU 60 and inferring on 10 unseen NTU 120 cross-subject classes, and pre-training on NTU 60 then testing on PKU-MMD cross-subject — SUGAR scores 65.3 Top-1 / 89.8 Top-5 and 53.4 Top-1 / 77.6 Top-5. ST-GCN, 2S-AGCN, and CTR-GCN linear baselines score between 30.1–36.9 Top-1 and 45.2–55.2 Top-5.
• Language knowledge ablation (Toyota Smarthome overall accuracy): no visual or motion knowledge 69.2; visual only 69.4; motion only 72.1; both 73.4. The paper describes this as a stable improvement of 4.2% overall accuracy.
• Bridging module ablation (Toyota Smarthome overall accuracy): cross-attention 52.1; one Q-Former 70.7; one linear layer 70.4; TQP 73.4. TQP improves by 2.7% over a single Q-Former.
• Token length: Testing lengths of 1000 (full length), 512, 256, 128, 64, and 1 on NTU RGB+D, performance increased as the input length was continuously reduced, except that compressing to 1 token produced the worst result. The authors therefore set the TQP query vector length to 128.
• Representation visualization: t-SNE plots of 8 Toyota Smarthome action classes before and after training show that similar actions such as "Drink From bottle" and "Drink From can", and "clean up" and "clean dishes", become largely separated after visual-motion supervision.

Methodology in Plain English

SUGAR runs in three stages.

First, the authors build text. They take the label list of the target dataset as a predefined action dictionary, then ask GPT 3.5 turbo to describe each action as movements of six body parts, and ask GPT-4V to describe the visual scene for each action under three rules (match the action, describe only the action-related scene, avoid unrelated text). To cut down redundant near-identical indoor frames, they embed frames with CLIP's vision encoder and keep the most dissimilar ones before generating descriptions.

Second, a skeleton encoder is trained. The encoder is a stack of graph convolution blocks plus multi-scale temporal modeling. The authors deliberately drop the temporal max-pooling used in the original CTR-GCN design so that fine-grained temporal detail is preserved. A pretrained CLIP text encoder embeds the motion and visual descriptions, and the skeleton representation is trained with a multiple-instance contrastive loss that allows one skeleton to match several positive texts in the batch — motion description embeddings and visual description embeddings are randomly combined into text sets.

Third, an LLM is adapted. A Temporal Query Projection module, made of Q-Formers with shared parameters, takes the long skeleton feature sequence and uses each queried result as the query for the next segment, compressing the sequence into a short set of tokens while keeping temporal continuity. These tokens, along with a fixed instruction, are fed to LLaMA2 7B whose weights are otherwise untouched; only LoRA adapters (r = 64, alpha = 16) are trained with a cross-entropy loss against the ground-truth action plus a short GPT-generated description. At inference, only skeleton data is needed — the model picks from the predefined action list and outputs a brief description.

Implementation specifics: CTR-GCN as skeleton backbone, CLIP as text encoder, SGD optimizer, initial learning rate 0.01, 200 epochs, batch size 200 (100 for Toyota Smarthome) with a 0.1 decay factor, LoRA training at learning rate 2e-5 for 1 epoch with total batch size 128, all on two NVIDIA A6000 GPUs.

Why This Matters

Impact on research. The paper argues that the bottleneck in LLM-based action recognition is not the recognizer's architecture but the discreteness of the skeleton representation, and that linguistic priors — especially visual ones — are what make fine-grained daily activities separable. It also demonstrates that an LLM-based classifier transfers to unseen classes and across datasets where a linear classifier's decision boundaries cannot.

Real-world applications:

• Human-computer interaction, where skeleton input is lightweight and the system can both name and describe the action.
• Intelligent monitoring, including eldercare scenarios like Toyota Smarthome, which involve many similar composite daily activities.
• Cross-domain deployment, where a model pretrained on one action dataset is applied to a new environment with different or unseen activity classes.
• Privacy-conscious sensing, since skeleton coordinates can be stored and processed without raw video.

Industry relevance. SUGAR's inference path only requires skeleton data, and the LLM runs with frozen weights plus small LoRA adapters, which suits deployment where a single pretrained backbone must be adapted cheaply to new activity sets. The token-length study also matters practically: shorter token sequences were both faster and, up to a point, more accurate, though collapsing to a single token failed.

Future Directions

• Extending beyond joints. SUGAR is trained only on skeleton joints, unlike competing methods that ensemble joint, bone, and motion streams; whether the paradigm gains further from multi-stream input is untested.
• Improving large-scene benchmarks. Performance on NTU RGB+D and NTU RGB+D 120 is described as competitive rather than decisively better than the LLM-based baseline LLM-AR, so the source of SUGAR's advantage in daily-activity settings versus scripted single-scene settings deserves study.
• Scaling the zero-shot protocols. The paper tests only 10 unseen NTU 120 classes and one cross-dataset transfer to PKU-MMD; broader open-vocabulary evaluation is left open.
• Reducing reliance on closed generative models. Motion and visual text come from GPT 3.5 turbo and GPT-4V, so dependence on those services, and the sensitivity of results to their output quality, is an open question. The paper does not report inference latency, training cost in GPU hours, or failure cases.

Target Audience

Researchers and engineers working on skeleton-based action recognition, multimodal representation learning, or LLM adaptation to non-text modalities. It is also relevant to practitioners building activity-monitoring or human-computer interaction systems who want a lightweight skeleton input pipeline with descriptive output. Readers need a background in graph neural networks and contrastive learning to follow the technical details; the paper does not report implementation-level details such as inference latency or hardware cost beyond the two A6000 GPUs used for training.

Authors’ abstract

Large Language Models (LLMs) hold rich implicit knowledge and powerful transferability. In this paper, we explore the combination of LLMs with the human skeleton to perform action classification and description. However, when treating LLM as a recognizer, two questions arise: 1) How can LLMs understand skeleton? 2) How can LLMs distinguish among actions? To address these problems, we introduce a novel paradigm named learning Skeleton representation with visUal-motion knowledGe for Action Recognition (SUGAR). In our pipeline, we first utilize off-the-shelf large-scale video models as a knowledge base to generate visual, motion information related to actions. Then, we propose to supervise skeleton learning through this prior knowledge to yield discrete representations. Finally, we use the LLM with untouched pre-training weights to understand these representations and generate the desired action targets and descriptions. Notably, we present a Temporal Query Projection (TQP) module to continuously model the skeleton signals with long sequences. Experiments on several skeleton-based action classification benchmarks demonstrate the efficacy of our SUGAR. Moreover, experiments on zero-shot scenarios show that SUGAR is more versatile than linear-based methods.