Multi-modal cross-domain mixed fusion model with dual disentanglement for fault diagnosis under unseen working conditions

AI summary

Overview

Research area: Intelligent fault diagnosis for machinery (induction motors), specifically domain generalization (DG) with multi-modal sensor fusion — combining vibration, current, and acoustic signals.

Technical level: Advanced. The paper assumes familiarity with convolutional/residual networks, cross-attention, maximum mean discrepancy (MMD), Mixup-style data augmentation, and the domain adaptation versus domain generalization distinction.

Scope: The paper proposes a single framework that jointly disentangles modality-invariant/specific features and domain-invariant/specific features, augments data by mixing modalities across source domains, and fuses three heterogeneous modalities for fault diagnosis under working conditions never seen during training.

Note: the supplied content is truncated before the experimental results section, so no accuracy values, dataset sizes, or per-benchmark numbers are reported here. Where the paper states only qualitative outcomes, that is what is reflected below.

What This Paper Is About

Fault diagnosis models trained on data from one set of machine operating conditions (speeds and loads) often fail when the machine later runs under different, previously unseen conditions. Existing fixes either require samples from the new condition (domain adaptation, which is impractical in industry) or use only one sensor modality, such as vibration.

The paper's goal is a model that generalizes to unseen working conditions without ever seeing target-condition data, by exploiting complementary information from three sensing modalities (vibration, current, acoustic) and by separating what is shared from what is condition- or sensor-specific.

Key Contributions

1. Dual disentanglement framework. A framework that disentangles invariance and specificity at two levels simultaneously: modality level (modality-invariant vs. modality-specific features) and domain level (domain-invariant vs. domain-specific representations).

2. Multi-modal cross-domain mixed fusion mechanism. A modality-augmentation strategy that randomly mixes each modality across source domains to mitigate domain bias and increase cross-domain, same-class modality diversity.

3. Triple-modal fusion module. A pairwise, multi-head cross-attention fusion design that adaptively integrates three heterogeneous modalities (vibration, current, acoustic), rather than only two as in much prior cross-attention fusion work.

The paper also presents a methodological comparison table (Table 1) positioning the proposed method as the only listed approach combining a DG setting, three-modality fusion, modality-level disentanglement, domain-level disentanglement, and modality-level generation. Existing invariance-based DG methods and causal/disentanglement DG methods are listed as single-modal; multi-modal fusion and multi-modal DA studies are listed as not addressing the DG setting and covering only two modalities.

Main Findings

• Proposed method outperforms comparison methods: The paper states that experiments on induction motor fault diagnosis under both unseen constant and time-varying working conditions show the proposed method "consistently outperforms advanced methods." The specific comparative accuracy figures are not included in the provided content.

• Ablation confirms each component: The paper reports that comprehensive ablation studies verify the effectiveness of each proposed component and the advantage of multi-modal fusion. No ablation numbers are present in the provided content.

• Both constant and time-varying unseen conditions were tested: The experimental scope covers unseen working conditions that are constant and working conditions that vary over time, using vibration, current, and acoustic signals collected from induction motors.

• Multi-modal fusion helps generalization under DG: The stated rationale and result is that integrating modalities reduces reliance on any single domain-specific feature distribution, giving more stable cross-domain generalization than single-modal DG. The paper also notes that domain shift does not affect all modalities uniformly, so retaining modality-specific components preserves complementary, potentially more stable fault cues.

• Effectiveness of the mixing strategy: Cross-domain mixing is reported as improving robustness by introducing random cross-domain perturbations at the modality level for samples of the same fault class.

Methodology in Plain English

Inputs and preprocessing. Each training sample contains three synchronized-in-time modalities from the same machine state. Vibration is converted to time-frequency images using the short-time Fourier transform (STFT), with multiple channels stacked into a multichannel image. Acoustic signals are converted to Mel-spectrograms by the same stacking approach, which emphasizes lower frequencies where fault information concentrates. Current signals are fed directly as raw multichannel temporal data. Because sampling rates differ, the number of sampling points per modality can differ; temporal alignment is required only in physical time duration, not in sample count, since fusion happens at the feature level.

Cross-domain mixing. For each modality of each sample, there is a 50% probability (based on a uniform random draw) of mixing with the same modality from a randomly chosen other source domain, picking a sample with the same fault label. Mixing uses a coefficient drawn from a Beta(0.2, 0.2) distribution. Unlike standard Mixup, the paper adds a constrained extrapolation step: with 50% probability the coefficient is transformed to be larger than, but close to, 1. This expands the data along the domain-variation direction while keeping samples physically close to the original, so perturbations remain meaningful.

Encoders. Separate, unshared encoders handle each modality: 2D ResNets for vibration and acoustic images, and a 1D ResNet for current signals.

Modality-level disentanglement. Each encoder output is passed through two embedding networks that produce a modality-invariant representation and a modality-specific representation. Three losses drive the separation: (1) a modality-level MMD loss aligning the distributions of invariant representations pairwise across vibration, current, and acoustic within the same source domain; (2) an intra-modality orthogonality loss that minimizes the covariance between the invariant and specific representations of the same modality; (3) an inter-modality orthogonality loss that minimizes covariance between specific representations of different modality pairs. These three terms are summed into one modality-level loss.

Triple-modal fusion. Rather than fusing all three modalities at once, the model fuses every pair independently using multi-head cross-attention, and both invariant and specific representations participate in fusion. This pairwise design is intended to capture fine-grained inter-modal interactions while avoiding the complexity and interference of direct triple fusion.

Domain-level disentanglement and classification. After fusion, domain-invariant and domain-specific features are further learned and separated across all source domains, capturing shared versus condition-dependent characteristics. A fault classifier is then attached for diagnosis.

Why This Matters

Research impact. The paper argues that most DG-based fault diagnosis work focuses only on domain-invariant features and single-modal data, and that existing disentanglement and causal-learning approaches need extra classifiers or decoders and were designed for one modality. It also argues that multi-modal fusion studies have not adequately addressed unseen working conditions, and that multi-modal DA methods still need target-condition data. This work reframes the problem as jointly handling two sources of heterogeneity — sensing modality and working condition — rather than one.

Real-world applications.

• Rotating machinery and motor monitoring in factories where machines are commissioned under one speed/load profile but operate under many others.
• Continuous condition monitoring in plants where labeling data from new operating conditions is infeasible or unavailable ahead of time.
• Multi-sensor industrial systems already instrumented for vibration, current, and acoustic measurements, where existing hardware can be reused rather than adding new sensing.
• Maintenance planning for equipment whose operating regimes change over time (for example, time-varying speed or load duty cycles).

Industry relevance. Because the method requires no target-condition data at training or inference, it avoids one of the main practical blockers of domain adaptation deployments. The use of three heterogeneous modalities matches the sensor suites already common on industrial assets, and the code is publicly available (https://github.com/xiapc1996/MMDG), which supports reproducibility and downstream adoption. The paper's methodological comparison table explicitly positions the approach against invariance-based, causal/disentanglement, and generation/augmentation-based DG methods, as well as multi-modal fusion and multi-modal DA methods.

Future Directions

• Quantifying generalization across more conditions. The paper reports experiments on unseen constant and time-varying working conditions for induction motors; extending the evaluation to other machinery types and additional unseen conditions would test the generality of the dual-disentanglement idea.
• Scaling beyond three modalities. The triple-modal fusion module is built for exactly three modalities using pairwise cross-attention. How the design scales to more or fewer sensors, or to modalities with very different sampling rates and physical units, remains open.
• Understanding the mixing hyperparameters. The mixing uses a 50% mixing probability and a Beta(0.2, 0.2) coefficient distribution with constrained extrapolation. The provided content does not report sensitivity analysis of these choices, which would clarify how much performance depends on them.
• Comparing against multi-modal domain generalization baselines. The paper notes that multi-modal fault diagnosis under unseen working conditions has not been adequately addressed in the literature, implying a shortage of directly comparable methods and a need for standardized multi-modal DG benchmarks.

Target Audience

Researchers and graduate students in machine learning for industrial systems, fault diagnosis, and prognostics and health management, particularly those working on domain generalization, domain adaptation, or multi-modal fusion. It is also relevant to industrial engineers and practitioners deploying condition monitoring on multi-sensor machinery who need models that transfer to new operating conditions without target-condition data. Readers without a background in representation learning, attention mechanisms, or domain shift concepts will find the methodology sections demanding.

Authors’ abstract

Intelligent fault diagnosis has become an indispensable technique for ensuring machinery reliability. However, existing methods suffer significant performance decline in real-world scenarios where models are tested under unseen working conditions, while domain adaptation approaches are limited to their reliance on target domain samples. Moreover, most existing studies rely on single-modal sensing signals, overlooking the complementary nature of multi-modal information for improving model generalization. To address these limitations, this paper proposes a multi-modal cross-domain mixed fusion model with dual disentanglement for fault diagnosis. A dual disentanglement framework is developed to decouple modality-invariant and modality-specific features, as well as domain-invariant and domain-specific representations, enabling both comprehensive multi-modal representation learning and robust domain generalization. A cross-domain mixed fusion strategy is designed to randomly mix modality information across domains for modality and domain diversity augmentation. Furthermore, a triple-modal fusion mechanism is introduced to adaptively integrate multi-modal heterogeneous information. Extensive experiments are conducted on induction motor fault diagnosis under both unseen constant and time-varying working conditions. The results demonstrate that the proposed method consistently outperforms advanced methods and comprehensive ablation studies further verify the effectiveness of each proposed component and multi-modal fusion. The code is available at: https://github.com/xiapc1996/MMDG.