Recurrent Temporal Restricted Boltzmann Machines as Unsupervised Encoders for Biomechanical Movement Sequences

AI summary

Overview

Research area: Machine learning for time-series representation learning, specifically energy-based models (Restricted Boltzmann Machines and their recurrent/temporal variants) applied to biomechanical motion capture and gait data.

Technical level: Intermediate. The paper assumes familiarity with RBMs, contrastive divergence and mean-field inference, but its main claims and comparisons are stated in terms of reconstruction error, clustering scores and probe accuracy that a general ML reader can follow.

Scope: The paper develops a learned-variance Gaussian Recurrent Temporal Restricted Boltzmann Machine (RTRBM), stacked into a Recurrent Temporal Deep Belief Network (RTDBN), as an unsupervised encoder that turns windows of movement data into 64-dimensional embeddings, and tests — including a control that removes the recurrent connection — what the temporal structure actually contributes.

What This Paper Is About

Biomechanical recordings (joint angles, velocities, muscle activations) are continuous, high-dimensional and inherently sequential, yet most unsupervised representation-learning pipelines treat each frame as an independent sample and throw away the temporal structure. The authors adapt an energy-based sequence model, the RTRBM, into a working unsupervised encoder for the SIT-FUSE segmentation pipeline and ask whether its recurrent connection measurably improves the resulting embeddings. They evaluate on simulated musculoskeletal reaching and on two real gait cohorts, comparing against non-recurrent and non-probabilistic baselines.

Key Contributions

1. A continuous-valued, learned-variance RTRBM. The authors extend the RTRBM (Sutskever et al., 2008), which used Gaussian visible units with fixed unit variance, to a learned per-feature variance parameterization (following Cho et al., 2011), implemented as the RTVarianceGaussianRBM class, with a two-phase variance schedule, gradient clipping (global max norm 1.0) and clamping of σ to [0.1, 10.0].

2. An RTDBN window encoder. RTRBM layers are stacked greedily and the temporal hidden states are collapsed into one fixed-length embedding per window by mean pooling; windowing never spans two recordings, and a causal sliding window of length T is applied within each episode.

3. Verified software and SIT-FUSE integration. The models are implemented as Learnergy classes in PyTorch; training gradients are checked against exact maximum likelihood on enumerable models, and setting encoder_type: rtdbn in a SIT-FUSE configuration selects the temporal encoder, running pretraining, clustering, per-window inference, FLOP counting and per-cluster TimeSHAP attributions.

4. A direct test of recurrence. A control retrains the model with W′ initialized to zero and frozen throughout training, reducing it to a per-frame Gaussian RBM with learned variance, mean-pooled over the window — isolating exactly what the recurrent connection contributes.

Main Findings

• Learned variance reconstructs far better than alternatives. Median held-out per-channel R² was 0.09 ± 0.03 for Bernoulli units, 0.57 ± 0.02 for fixed-variance Gaussian units, and 0.61 ± 0.01 for the learned-variance model. The paired difference between learned and fixed variance was +0.04 in median R² (95% CI [+0.025, +0.056]) and −189 in held-out MSE (539 ± 27 versus 728 ± 21; CI [−199, −177]).

• Reconstruction is uneven across channel groups. With learned variance, median R² was 0.86 for joint positions, 0.76 for velocities and 0.56 for muscle activations.

• Clusters carry a kinematic reading without labels. On 11,220 train+validation windows, k-means (k = 10) gave normalized entropy 0.978 ± 0.007 and silhouette 0.154 ± 0.013. Agreement with ten equal phase bins was moderate (NMI 0.265 ± 0.013), and a linear probe recovered a window's relative position in its episode with R² = 0.79 ± 0.04.

• Cluster identity tracks distance to target and closing rate. The largest cluster (n = 2,174) sat near the target (mean reach-error norm 0.143 m) while still closing it (falling 0.077 m within the window); the second (n = 1,370) was also near the target (0.151 m) but nearly static (falling 0.018 m); the third (n = 1,344) was far (0.597 m) and closing slowly (0.066 m). The population mean reach-error norm was 0.358 m. The authors note these readings are descriptive, from one seed.

• Clustering transfers to unseen episodes. On the 15 held-out test episodes (1,980 windows), held-out reconstruction MSE was 539 ± 27 on test versus 547 ± 29 on validation, all 10 clusters stayed populated, normalized entropy was 0.952 ± 0.031, and test silhouette (0.170 ± 0.018) was at least as high as train+validation silhouette (0.154 ± 0.013).

• Removing recurrence costs nothing on the reaching task. The W′ = 0 model matched the canonical model: test MSE 538 ± 31 versus 539 ± 27 (paired difference −1.2, CI [−19.0, +14.6], p = 0.90), median R² 0.615 ± 0.018 versus 0.608 ± 0.012 (difference +0.006, CI [−0.005, +0.017]), and test silhouette difference +0.004. No probe difference had a confidence interval excluding zero. Because the CIs are narrow, a benefit of W′ larger than about 19 units of test MSE (3.5%) or 0.005 in median R² can be excluded at this sample size.

• Window length barely matters; hidden size trades reconstruction against cluster separation. Across T ∈ {10, 20, 30, 40}, MSE/(T·n_v) stayed at 0.32–0.35 and median R² at 0.60–0.61, with silhouette between 0.140 and 0.153. Increasing n_h raised median R² from 0.549 (32) to 0.610 (64) to 0.678 (128) while silhouette moved the opposite way (0.212, 0.153, 0.141).

• The RTRBM is a weak reconstructor and a weak probe on reach quantities compared with deterministic baselines. Frame-wise PCA reached median R² of 1.000, the autoencoders at least 0.998, window PCA 0.714, against 0.608 for the canonical RTRBM. Baselines were better on the reach-trend probe (window PCA 0.660 versus 0.365 for the canonical RTRBM, Holm-adjusted p < 0.001). The RTRBM was better only on the joint-speed probe (0.790 versus at most 0.35 for every baseline), and the W′ = 0 model scored 0.825 there, so this is not due to recurrence.

• On real gait, embeddings beat PCA and autoencoders under a random forest. Random-forest accuracy on unseen subjects was 0.577 for the RTRBM in cohort 1 (versus 0.456 to 0.515 for the four learned baselines) and 0.610 in cohort 2 (versus 0.413 to 0.559); all four contrasts held after Holm correction (cohort 1: +0.062 to +0.121, Holm p ≤ 5.5 × 10⁻⁴; cohort 2: +0.051 to +0.197, Holm p ≤ 0.018).

• Simple summary statistics beat the embeddings on gait. A training-free baseline of per-channel mean, standard deviation, minimum and maximum reached 0.730 (linear) and 0.649 (random forest) in cohort 1 versus 0.514 and 0.577 for the RTRBM (paired differences +0.216, CI [+0.174, +0.258], and +0.072, CI [+0.048, +0.094]; Holm p = 2 × 10⁻⁵ and 7 × 10⁻⁴). In cohort 2 it was numerically higher (0.567 and 0.649 versus 0.529 and 0.610) but within noise (Holm p = 0.28).

• Recurrence may help on one gait cohort only. Removing recurrence lowered random-forest accuracy by 0.062 in cohort 1 (CI [+0.046, +0.078]) but only by 0.019 in cohort 2 (CI [−0.017, +0.061]).

Methodology in Plain English

The team started from a Restricted Boltzmann Machine, an energy-based model that learns hidden features by running a short Gibbs sampling chain (CD-1) between visible and hidden layers. A standard RBM sees one frame at a time, so they swapped in the RTRBM, which adds a single learned matrix W′ that carries the previous step's mean-field hidden probabilities forward and uses them to modulate the hidden bias at the current step. They replaced the binary visible layer with Gaussian units and made the per-channel variance a learned parameter, trained on a schedule where the variance is frozen at 1 for 15 epochs while the other weights stabilize and then unfrozen for 15 epochs.

Training used CD-1 at each timestep, teacher-forced on real data when advancing the recurrent state, with the sampled visible unit acting as the negative particle. They used plain SGD with batch size 32 on CPU, learning rates chosen by validation reconstruction MSE, ten seeds (1–10) with 70/15/15 episode splits and paired comparisons by seed.

They generated data from the myoArmReachFixed-v0 environment in MyoSuite (MuJoCo), running 100 episodes of 151 steps each with an 80-dimensional observation vector (20 joint positions, 20 joint velocities, 3 reach-site positions, 3 reach errors, 34 muscle activations), driven by a momentum-biased random policy. Windows of length T = 20 with stride 1 were mean-pooled into 64-dimensional embeddings and clustered with k-means (k = 10, 20 random restarts), deliberately decoupled from encoder training.

To test generalization outside simulation, they used two real gait cohorts where the task was three-class walking-speed classification on unseen subjects: cohort 1 (57 subjects, speeds 0.5, 0.75 and 1.0 m/s, 66 joint-angle channels at 60 Hz) and cohort 2 (14 healthy adults at 70%, 100% and 130% of preferred speed, 18 rotational channels). They fit logistic-regression and random-forest probes on frozen embeddings and compared against frame-wise PCA, window PCA, an MLP autoencoder, a GRU autoencoder and the summary-statistics baseline.

Why This Matters

Impact on research. The paper is a careful, negative-leaning result: it supplies a verified, reproducible implementation of a temporal energy-based encoder for a self-supervised segmentation pipeline (SIT-FUSE), and it shows that the intuitive claim "recurrence must help" does not hold automatically on continuous biomechanical signals under a reconstruction objective. The narrow confidence intervals on the W′ = 0 ablation make that a quantified null rather than a vague one, and the summary-statistics comparison sets a bar that future gait-representation papers should clear.

Real-world applications:

• Unsupervised segmentation of rehabilitation or physical-therapy recordings, where cluster structure could flag movement phases without hand-labeled data.
• Gait analysis for clinical or sports monitoring, e.g. grouping strides or classifying walking speed from wearable inertial sensors.
• Markerless motion capture pipelines that need compact per-window descriptors feeding downstream clustering or interpretability tools such as TimeSHAP.
• Any sequential sensing domain where SIT-FUSE is already used (the authors' own pipeline originated in remote sensing imagery), extended now to multi-trial biomechanical recordings.

Industry relevance. Wearable and digital-health companies, sports-analytics firms and robotics groups working with human motion all need label-efficient ways to compress high-rate multichannel signals. The finding that a compact 64-dimensional learned embedding is a better speed descriptor than PCA or autoencoder embeddings, but not better than four summary statistics per channel, is a directly actionable engineering result about where complexity pays off.

Future Directions

• Richer behavioral diversity. The current reach data uses a random action policy on a single fixed target, producing kinematically but not behaviorally diverse windows; variable-target reaching, task-directed policies or real sensor data are proposed as likely sources of richer cluster structure.
• Objectives beyond reconstruction. The authors suggest the null W′ result stems from a reconstruction-based objective that gives memory little to learn, and point to joint training with the information-theoretic clustering head (IIC) that SIT-FUSE normally uses as a natural next evaluation.
• Larger and more varied cohorts. Both gait cohorts are small (57 and 14 subjects), and the authors call for larger cohorts to test whether the recurrence benefit seen in cohort 1 replicates.
• Deeper and more expressive temporal models. The RNN-RBM is discussed as a more expressive alternative with a separate recurrent state, and stacking more RTRBM layers is implemented but untested at this data scale.

Target Audience

Machine learning researchers working on energy-based and time-series representation learning; engineers building self-supervised segmentation pipelines for sensor data, particularly those already using or extending SIT-FUSE; biomechanics and gait-analysis researchers who want label-free window descriptors and a realistic sense of how they compare with simple summary statistics; and practitioners who value a paper that reports its own ablation against its central premise and draws the corresponding limits on its claims.

Authors’ abstract

Biomechanical movement recordings are continuous, high-dimensional time series, and analyzing them without manually labeling them requires representations that capture how movements evolve over time. Most unsupervised representation-learning pipelines, however, treat each observation frame independently and discard this temporal structure, while energy-based models that could capture it have not extensively been adapted to biomechanical signals. So, this paper implements the Recurrent Temporal Restricted Boltzmann Machine (RTRBM), a recurrent energy-based model, into an unsupervised encoder that maps windows of movement data to fixed-length embeddings, and tests what its recurrent structure contributes. We adapt the model to continuous-valued signals, stack it into a deep architecture, and integrate it into the SIT-FUSE segmentation pipeline. We evaluate the embeddings on simulated musculoskeletal reaching and on two real gait datasets, using clustering, reconstruction, and walking-speed classification of unseen subjects, and compare them against non-recurrent and non-probabilistic baselines, including a control that removes the recurrent connection. Clustering the embeddings recovers movement structure tied to progress toward the target without any labels, which also transfers to unseen episodes. On gait data, the embeddings outperform PCA and autoencoder baselines under a nonlinear probe. However, removing the recurrent connection does not yield a measurable loss on the reaching task, and simple per-channel summary statistics classify walking speed at least as well as the embeddings. The model is therefore a working, verified encoder for SIT-FUSE, while the evidence for a benefit of recurrence is mixed, and the learned embeddings did not outperform simple summary statistics on gait.