Continual Action Quality Assessment via Adaptive Manifold-Aligned Graph Regularization

AI summary

Overview

• Research area: Computer vision — specifically Action Quality Assessment (AQA) in videos, combined with Continual Learning (CL).
• Technical level: Advanced. The paper combines large-scale video benchmark experiments with a theoretical treatment (projection-gap bounds for parameter-efficient fine-tuning, NTK-style curvature analysis, and a forgetting bound for replay-based continual learning).
• Scope: This paper introduces the Continual Action Quality Assessment (CAQA) setting and proposes MAGR++ (Adaptive Manifold-Aligned Graph Regularization), a framework for continuously adapting an AQA model to non-stationary quality distributions without catastrophic forgetting.

What This Paper Is About

Action Quality Assessment tries to automatically score how well a human action is performed in a video, but in real settings the scoring patterns keep changing as skills evolve and user groups differ, so a model trained once degrades over time. The paper defines this as Continual AQA (CAQA), where a model must be updated across a sequence of sessions of data while retaining what it learned earlier. The authors show that lightweight fine-tuning of a pretrained model is not enough for this task, and propose MAGR++ to make full-parameter fine-tuning both adaptive and stable.

Key Contributions

1. First formulation of CAQA. The paper formally defines Continual AQA to address non-stationary action quality distributions, extending continual learning from classification to continual score regression.
2. A theoretical framework for CAQA's two core challenges. Theorem 1 shows that parameter-efficient fine-tuning (PEFT) incurs a strictly positive excess risk under a curvature-induced metric when the downstream optimum lies outside its restricted low-rank subspace, while full-parameter fine-tuning (FPFT) avoids this. Theorem 2 shows that under feature replay, forgetting grows with the size of the parameter update, tying large FPFT updates to both replay drift and overfitting.
3. A theoretically grounded method, MAGR++. The framework combines layer-adaptive full-parameter fine-tuning (stabilizing shallow layers, fully tuning deeper ones, with the boundary selected automatically) with a two-step feature rectification pipeline: a manifold projector that translates old features into the current representation space, and an Intra-Inter-Joint Graph Regularizer (IIJ-GR) that aligns local and global feature-score structure.
4. Four CAQA benchmarks and broad evaluation. The authors construct four CAQA benchmarks from three AQA datasets with tailored evaluation protocols and strong baselines, and report state-of-the-art results across both offline and online CAQA settings. Code is released at https://github.com/ZhouKanglei/MAGRPP.

Main Findings

• Full fine-tuning beats parameter-efficient tuning for AQA. Comparing a fixed backbone, PEFT (I3D-Adapters), and FPFT across three benchmarks (MTL-AQA, AQA-7, FineDiving) with three prediction heads (CoRe, HGCN, CoFInAl), FPFT was consistently better in both SRCC and rMSE. FPFT gained +3.33% SRCC and −31.23% rMSE over the fixed backbone, and +2.11% SRCC and −14.25% rMSE over PEFT.
• The PEFT gap is largest where the domain gap is largest. The biggest improvement over PEFT was on AQA-7 (+5.89% SRCC, −44.21% rMSE), with smaller but consistent benefits on MTL-AQA (+0.23% SRCC, +4.31% rMSE as reported) and FineDiving (+0.22% SRCC, −2.85% rMSE).
• PEFT has a structural limitation, not just a tuning problem. Theorem 1 shows that prompt tuning, adapters, and LoRA share a low-rank update form, and that any PEFT update suffers excess risk whenever the downstream optimum has a nonzero projection onto the complement of the PEFT subspace.
• Uncontrolled full fine-tuning is unstable in the continual setting. Theorem 2's forgetting bound contains terms proportional to the update magnitude Δ_t, meaning large updates amplify both replay drift (through feature mismatch between old and new backbones) and growth of the hypothesis class, leading to overfitting and forgetting.
• Stored features drift away from the current manifold. Because the backbone keeps changing, features kept in the memory bank become misaligned with newly extracted features, weakening replay supervision and confusing score regression when old and new features are mixed.
• Two fixes drive the improvement. Layer-adaptive fine-tuning and asynchronous feature rectification (a departure from the synchronous projector training of the preliminary MAGR version, which the authors state exacerbated misalignment) are the key changes in MAGR++.
• Layer boundary selection is reliable and cheap. Choosing the boundary using only base-session data via an abstraction ratio (built from the Davies–Bouldin index of frozen versus fine-tuned features, with a margin of 0.05) produced layer boundaries consistent with exhaustive grid search, which would otherwise require future session data.
• Constraining a single layer can suffice. Experiments showed that applying the shallow-layer consistency constraint only at layer L_opt − 1 was enough for stable performance.
• Angular geometry matters. Existing graph-based continual learning methods relying on Euclidean distances fail to capture the geodesic structure of quality relationships, whereas IIJ-GR uses angular distances on a unit hypersphere plus distance matrix partitioning.
• State-of-the-art across settings. MAGR++ surpasses the strongest baseline by 1.6%–6.5% offline and 4.0%–21.8% online, with average correlation gains of 3.6% offline and 12.2% online.

Methodology in Plain English

The authors first ran a controlled comparison showing that adapting all parameters of a pretrained video backbone beats adapting only small add-on modules, because pretrained action-recognition features are not aligned with the fine-grained motion cues that quality scoring needs. They then used theory to explain why the obvious fix — just fine-tune everything — breaks in a continual setting.

MAGR++ works in a session-by-session loop. At the end of a session, it selects a small set of representative features using Ordered Uniform Sampling, which sorts samples by quality score and picks uniformly across the score range so the memory bank covers diverse quality levels. At the start of the next session, the backbone is adapted with layer-adaptive full fine-tuning: shallow layers that mainly encode low-level appearance are softly held to their previous outputs, while deeper layers that encode execution quality are fully updated. The boundary between the two is chosen automatically by checking, on base-session data only, which layer starts to benefit from fine-tuning.

Next, a manifold projector learns to map features from the old backbone to the new one using current-session videos only — it never needs the raw past data. This lets the stored old features be translated into the current representation space before they are replayed. Finally, a graph regularizer aligns the geometry of the feature space with the geometry of the score space, using angular distances between unit-normalized features and splitting the distance matrix into within-session and cross-session blocks so that both local and global quality relationships are preserved. The regressor is trained jointly on rectified old features and new features, and the memory bank is refreshed afterward.

Why This Matters

• Research impact: The paper opens a new problem setting (continual score regression for human action quality), supplies the first theoretical account of why standard continual learning recipes fail there, and provides four benchmarks plus baselines for cross-dataset comparison. It also gives a general argument — with a proof of a projection gap — for when full fine-tuning is necessary rather than parameter-efficient tuning.
• Impact on continual learning more broadly: The insights connect representation-level manifold shift to replay effectiveness, which is relevant beyond AQA to any continual regression or fine-grained continual perception problem.

Real-world applications named or implied by the paper:

• Sports scoring: objective, automated judging that keeps pace as athletes' techniques and scoring conventions evolve.
• Rehabilitation: tracking how well patients perform prescribed movements over long periods as their ability changes.
• Skill assessment and training: evaluating learners' execution quality where performance distributions shift as skills develop.
• Evolving user populations: deploying AQA where new individuals, groups, or actions continually appear and the underlying quality distribution is non-stationary.

Industry relevance: Systems that grade human motion — sports analytics, clinical rehab monitoring, physical education, coaching tools — typically cannot retrain from scratch because expert annotations are scarce and expensive. MAGR++ targets exactly that constraint: it adapts an existing pretrained model with lightweight feature replay rather than storing raw video, which lowers memory and privacy burdens, and it avoids the expensive annotated base-session training that prior pretrained-model continual learning paradigms require.

Future Directions

• Extending CAQA beyond the four constructed benchmarks. The paper builds four CAQA benchmarks from three AQA datasets; generalizing the setting and protocol to other action domains is an open question.
• Reducing reliance on any stored past information. MAGR++ already uses feature replay rather than raw exemplars, but the memory bank must still be curated and refreshed; a fully exemplar-free variant would further reduce storage and privacy concerns.
• Understanding the limits of the layer-adaptive boundary. The boundary is selected from base-session data and validated against grid search, but how it behaves under very long task sequences, abrupt distribution changes, or new backbone architectures is not established in the content available.
• Applying the projection-gap argument to other fine-grained continual tasks. The theory suggests full fine-tuning is necessary whenever the downstream optimum lies outside a low-rank update subspace; testing whether layer-adaptive tuning plus feature rectification helps in other fine-grained continual vision problems is a natural next step.

Note: the paper content available does not include an explicit author-stated future work section.

Target Audience

Researchers and practitioners working on video understanding, human motion analysis, and action quality assessment who need models that keep working as data distributions change. It is also relevant to continual learning researchers interested in regression tasks rather than classification, and to engineers building sports analytics, rehabilitation monitoring, or skill-training systems who need to adapt pretrained models without repeated large-scale annotation or raw video storage. The theoretical sections (projection gaps, NTK-style curvature, forgetting bounds) make the paper most accessible to readers with a graduate-level machine learning background.

Authors’ abstract

Action Quality Assessment (AQA) quantifies human actions in videos, supporting applications in sports scoring, rehabilitation, and skill evaluation. A major challenge lies in the non-stationary nature of quality distributions in real-world scenarios, which limits the generalization ability of conventional methods. We introduce Continual AQA (CAQA), which equips AQA with Continual Learning (CL) capabilities to handle evolving distributions while mitigating catastrophic forgetting. Although parameter-efficient fine-tuning of pretrained models has shown promise in CL for image classification, we find it insufficient for CAQA. Our empirical and theoretical analyses reveal two insights: (i) Full-Parameter Fine-Tuning (FPFT) is necessary for effective representation learning; yet (ii) uncontrolled FPFT induces overfitting and feature manifold shift, thereby aggravating forgetting. To address this, we propose Adaptive Manifold-Aligned Graph Regularization (MAGR++), which couples backbone fine-tuning that stabilizes shallow layers while adapting deeper ones with a two-step feature rectification pipeline: a manifold projector to translate deviated historical features into the current representation space, and a graph regularizer to align local and global distributions. We construct four CAQA benchmarks from three datasets with tailored evaluation protocols and strong baselines, enabling systematic cross-dataset comparison. Extensive experiments show that MAGR++ achieves state-of-the-art performance, with average correlation gains of 3.6% offline and 12.2% online over the strongest baseline, confirming its robustness and effectiveness. Our code is available at https://github.com/ZhouKanglei/MAGRPP.