PEFT-MuTS: A Multivariate Parameter-Efficient Fine-Tuning Framework for Remaining Useful Life Prediction based on Cross-domain Time Series Representation Model

AI summary

Overview

Research area: Prognostics and Health Management (PHM) / industrial time-series machine learning — specifically remaining useful life (RUL) prediction combining self-supervised time-series representation learning, transfer learning, and parameter-efficient fine-tuning (PEFT).

Technical level: Advanced. The paper assumes familiarity with fine-tuning, low-rank adaptation, contrastive/masked self-supervised pretraining, and degradation prognostics.

Scope: The paper proposes and evaluates PEFT-MuTS, a framework that adapts a cross-domain pretrained univariate time-series backbone to multivariate few-shot RUL prediction using a small number of additional trainable parameters.

What This Paper Is About

Deep-learning RUL prediction normally requires large amounts of run-to-failure degradation data, and existing few-shot solutions (domain adaptation, meta-learning) still depend on having enough degradation data from equipment similar to the target. This paper asks whether the transferable knowledge for RUL prediction can instead come from general-purpose, cross-domain time-series data that is unrelated to equipment health. The goal is a framework that adapts a pretrained time-series representation model to multivariate RUL prediction while training only a tiny fraction of parameters, so it remains stable when target-equipment data is extremely scarce.

Key Contributions

1. A shift in transfer paradigm. The paper argues against the assumption that knowledge transfer for RUL prediction can only occur between similar devices, and shows that pretraining on large-scale cross-domain time series provides substantial benefit, broadening the range of usable source-domain data.

2. The PEFT-MuTS framework. A parameter-efficient fine-tuning framework for multivariate few-shot RUL prediction composed of an Independent Feature Tuning Network (per-variable, low-rank, LoRA-inspired) and a Meta-Variable–based Low-Rank Feature Fusion mechanism that lets a univariate backbone handle multivariate degradation data.

3. Analysis and mitigation of fine-tuning instability. The paper analyzes unstable fine-tuning under extremely small samples with a variance-based derivation and proposes a zero-initialized RUL regressor as a simple remedy, which removes the regressor-initialization variance term from the gradient variance.

4. Demonstration under extreme data scarcity. Experiments on aero-engine (C-MAPSS FD002 and FD004) and industrial bearing (XJTU-SY) datasets are reported to show effective RUL prediction when less than 1% of target-equipment samples are used, with a code release at https://github.com/fuen1590/PEFT-MuTS.

Main Findings

• Cross-domain priors transfer to degradation modeling: The paper reports that pretraining on cross-domain time series (SleepEEG) yields representations that transfer to RUL prediction, supporting the claim that transferable knowledge need not be degradation-specific.

• Extreme data scarcity is tolerated: The authors state that effective RUL prediction is achieved even when less than 1% of samples from the target equipment are used. Table 2 includes training subsets as small as 0.72% (FD002-1, FD004-1) of the original dataset and subsets with only one late-stage sample (OP-A-1, OP-B-1, OP-C-1).

• Strong reported comparison: The abstract states the method "substantially outperforms conventional supervised and few-shot approaches while markedly reducing the data required to achieve high predictive accuracy." The specific performance metrics (for example RMSE or the C-MAPSS scoring function) and comparison tables are not present in the truncated content provided, so they are not reported here.

• Very small parameter footprint: The PEFT-MuTS layers add 74.9K parameters, which is 0.50% of the 15M-parameter backbone, with 8 PEFT-MuTS layers attached to the ResNet-18 blocks.

• Low ranks are sufficient: The inner projection dimensions r_k in Table 3 shrink from 128 (Block1) down to 1 (Block8), while input dimensions d_k grow from 64 to 1024.

• Instability has an identifiable source: The variance analysis indicates gradient-update variance grows with input-feature variance, label variance, and regressor-initialization variance. Because pretrained models produce more discriminative features early in fine-tuning, initial feature variance is high, which the paper identifies as a reason fine-tuning is less stable than training from scratch.

• Zero initialization reduces the instability term: Setting the regressor to zero removes the σ_φ² contribution and reduces the impact of σ_z², which the paper reports as improving fine-tuning stability, with empirical validation stated to follow.

• Few-shot construction matters: The paper constructs few-shot sets with three independent sampling axes — device retention (p₁), RUL coverage (p₂), and global data retention (p₃) — rather than a single sampling rate.

Methodology in Plain English

The framework runs in two stages.

Stage 1 — pretraining. A 1-D ResNet-18 backbone is trained self-supervised on large-scale time-series data. In this paper the pretraining uses the FEI method on the SleepEEG dataset (371,055 EEG samples, each of length 178, 66,047,790 time points, sampled at 100 Hz). SleepEEG was chosen for its diverse temporal patterns across frequency ranges. FEI and SimMTM are both named as usable pretraining approaches. Critically, this data has nothing to do with machine degradation — the point is to learn general temporal structure such as trends, periodicity, frequency patterns, and local dependencies.

Stage 2 — fine-tuning. The pretrained backbone is frozen. Only new, small modules attached to it are trained. Each of the 8 residual blocks gets a PEFT-MuTS layer, and depthwise separable kernels (kernel size 3) are used where dimensional alignment is needed.

The PEFT-MuTS layer has two paths:

• Independent Feature Tuning Network (variable path). Every sensor variable is processed separately so the univariate structure the backbone was pretrained on is not disrupted. The tuning branch computes a low-rank mapping through matrices A_k (d_k × r_k, randomly initialized) and B_k (r_k × d_k, zero-initialized), applies a convolution, passes the result through a SiLU activation, and adds it residually to the frozen backbone output. Zero-initializing B_k means the branch starts at zero, so the pretrained behavior is preserved at the start of fine-tuning.

• Meta-Variable–based Low-Rank Feature Fusion (meta-variable path). A learnable extra "variable" u, initialized as a zero vector with the same length as the input, is processed by the backbone in parallel with the real variables. Within the low-rank space, a self-gating operation (a sigmoid multiplied elementwise with the low-rank features) filters information from all variables and the meta-variable, and average pooling along the variable dimension (window size N+1) aggregates it. The result updates only the meta-variable, which progressively absorbs information from the data variables across layers. The final representation comes solely from the meta-variable features. The data variables keep their independent pipelines, so per-variable representations are preserved.

Zero-initialized regressor. The paper derives how the variance of a linear regressor's gradient update depends on feature variance, label variance, and initialization variance, then sets the regressor weights to zero to minimize the initial update variance. The authors explicitly note this is an intuitive analysis rather than a rigorous proof, resting on simplifying assumptions such as Gaussian feature distributions, independence of features and prediction error, and a linear prediction head, so it explains the early fine-tuning stage rather than the full optimization dynamics.

Experimental setup. Two datasets are used: C-MAPSS (subsets FD002 and FD004, each with six operating conditions; 14 selected sensor variables; piecewise linear labels with degradation starting 120 cycles before failure) and the XJTU-SY bearing dataset (OP-A, OP-B, OP-C; horizontal and vertical vibration signals; degradation onset determined by RMS with the three-sigma rule). Labels are normalized to [0, 1], where 1 is healthy and 0 is failure. Sliding windows use size 30 / step 15 for C-MAPSS and size 1024 / step 32768 for the bearing data. Few-shot training sets are built by device retention, RUL coverage (healthy samples always kept), and global data retention. Test sets are the original ones: 259 engines for FD002 and 248 for FD004, and bearing units 2–5 from each operating condition. Min-max normalization is computed per few-shot training set and applied to the corresponding test set. Fine-tuning runs for 300 epochs with AdamW, no validation split and no early stopping.

Why This Matters

Impact on research. The paper reframes few-shot RUL prediction from "find equipment with a similar degradation pattern to transfer from" to "borrow general temporal priors from any large time-series corpus." If that holds, the bottleneck for deploying prognostics shifts away from collecting years of run-to-failure data from related machines toward selecting and pretraining strong general time-series encoders. It also argues that mainstream PEFT techniques (LoRA, P-Tuning, prompt tuning) are tightly coupled to Transformer MLP or self-attention layers and therefore underexplored for the non-Transformer, variable-mismatched setting of time-series RUL prediction, which is a distinct research gap.

Real-world applications:

• Aero-engine maintenance. Predicting RUL for engines across six operating conditions (as in FD002/FD004) supports maintenance scheduling and replacement planning without a fleet of fully run-to-failure engines.
• Industrial bearings and rotating machinery. Using horizontal and vertical vibration signals across different speeds and loads (OP-A/B/C) to estimate remaining bearing life.
• Long-life industrial assets. Equipment whose operational lifespan extends for decades, where full-lifecycle data collection is prohibitively slow and costly.
• Newly developed or small-batch customized equipment. Devices with little service time and small production quantities have almost no historical degradation data, which is exactly the regime the paper targets.

Industry relevance. A 0.50% trainable-parameter footprint means adaptation is cheap in memory and compute, and the frozen backbone allows one pretrained model to be reused across many downstream assets. The stated advantage of not needing source data from similar equipment reduces the dependency on proprietary fleet data, which is often unavailable or not shareable between operators.

Future Directions

• Extend the stability analysis beyond the linear head. The paper's own derivation is limited to a linear regressor and explicitly states it may not extend directly to nonlinear prediction heads; a broader analysis is an open problem.
• Transfer other PEFT families to non-Transformer time-series backbones. The authors state that PEFT architectures tailored for multivariate RUL prediction on time-series representation models remain underexplored, leaving Side Tuning, P-Tuning, prompt tuning, and LoRA-style variants largely untested in this setting.
• Validate on more backbones, pretraining corpora, and domains. Reported experiments use a 1-D ResNet-18 backbone pretrained with FEI on SleepEEG; whether the paradigm generalizes to other representation methods (such as SimMTM or TS2Vec) and other cross-domain corpora is not established here.
• Understand and replace the few-shot validation gap. Because the sample sizes are too small for a validation split, the paper uses a fixed 300 epochs with no early stopping; a principled stopping or model-selection criterion for extreme few-shot regimes remains unresolved.
• Tune the rank and fusion design. The r_k values in Table 3 are fixed per layer (128 down to 1); how to choose them automatically, and how the meta-variable and gating design scale with variable count, is not addressed.

Target Audience

Researchers and graduate students working on prognostics and health management, industrial time-series modeling, and transfer learning; practitioners building RUL prediction systems where target-machine failure data is scarce; and machine learning researchers interested in parameter-efficient fine-tuning for non-Transformer architectures and in adapting general time-series foundation models to regression tasks. Readers need prior familiarity with self-supervised representation learning and fine-tuning to follow the derivations and architecture details.

Authors’ abstract

The application of data-driven remaining useful life (RUL) prediction has long been constrained by the availability of large amount of degradation data. Mainstream solutions such as domain adaptation and meta-learning still rely on large amounts of historical degradation data from equipment that is identical or similar to the target, which imposes significant limitations in practical applications. This study investigates PEFT-MuTS, a Parameter-Efficient Fine-Tuning framework for few-shot RUL prediction, built on cross-domain pre-trained time-series representation models. Contrary to the widely held view that knowledge transfer in RUL prediction can only occur within similar devices, we demonstrate that substantial benefits can be achieved through pre-training process with large-scale cross-domain time series datasets. A independent feature tuning network and a meta-variable-based low rank multivariate fusion mechanism are developed to enable the pre-trained univariate time-series representation backbone model to fully exploit the multivariate relationships in degradation data for downstream RUL prediction task. Additionally, we introduce a zero-initialized regressor that stabilizes the fine-tuning process under few-shot conditions. Experiments on aero-engine and industrial bearing datasets demonstrate that our method can achieve effective RUL prediction even when less than 1\% of samples of target equipment are used. Meanwhile, it substantially outperforms conventional supervised and few-shot approaches while markedly reducing the data required to achieve high predictive accuracy. Our code is available at https://github.com/fuen1590/PEFT-MuTS.