Diff-MN: Diffusion Parameterized MoE-NCDE for Continuous Time Series Generation with Irregular Observations

AI summary

Overview

• Research area: Machine learning, specifically generative modeling for time series — continuous time series generation (TSG) from irregularly sampled observations, built on Neural Controlled Differential Equations (NCDEs) and diffusion models.
• Technical level: Advanced. The paper assumes familiarity with Neural ODEs/CDEs, Mixture-of-Experts routing, and diffusion generative models.
• Scope: The paper introduces Diff-MN, a framework that combines a decoupled Mixture-of-Experts NCDE (MoE-NCDE) with a diffusion model that parameterizes the MoE weights, and evaluates it on ten datasets across irregular-to-regular and irregular-to-continuous generation tasks.

What This Paper Is About

Most time series generation methods assume data are sampled at fixed intervals and produce outputs at a fixed resolution, but real-world observations (for example, ICU electronic health records) are irregular and sparse while downstream tasks need dense, continuous sequences. The goal is to build a generative model that learns from irregular observations yet can sample new, high-frequency time series at any time point in the interval. The authors argue that standard NCDEs are a natural fit for irregular data but are limited by a single dynamics function, tightly coupled optimization, and an inability to adapt learned dynamics to newly generated samples — the three problems Diff-MN targets.

Key Contributions

1. First formal study of continuous TSG. The authors state they are the first to formally define and evaluate continuous time series generation, assessing generated high-frequency data from two angles: downstream forecasting performance and recovery of analytical solutions of cubic polynomials.
2. Decoupled MoE-NCDE. They replace the single NCDE dynamics function with a dense Mixture-of-Experts dynamics function (four experts in experiments) and replace the State Initialization Network and Readout Network with a pretrained, frozen channel-wise encoder and decoder, so training focuses on the dynamics function.
3. Diff-MN: diffusion-parameterized MoE weights. A diffusion model jointly learns the distribution of imputed time series and their corresponding MoE weights, so each newly generated sample receives matched MoE weights that are fed back into the pretrained MoE-NCDE for continuous generation.
4. Broad empirical validation. Experiments cover four public TSG datasets (Sines, Stocks, Energy, MuJoCo), four medical ECG datasets (ECG200, ECG5K, ECGFD, TLECG, with 2, 5, 2, and 2 classes respectively), and two synthetic datasets (integrated signal waves and polynomial coefficient fitting data), under 30%, 50%, and 70% dropped observations.

Main Findings

• Irregular-to-regular generation: Diff-MN reports the best DS, MDD, and KL scores against KoVAE, GT-GAN, TimeGAN-NCDE, TimeVAE-NCDE, Diffusion-NCDE, ProFITi, and HeTVAE. For example, at 30% missing values on Sines, Diff-MN records DS 0.105 versus KoVAE 0.142 and GT-GAN 0.302; MDD 0.953 versus KoVAE 5.134; KL 0.013 versus KoVAE 4.172. The same ordering holds on the medical ECG datasets at 30%, 50%, and 70% missing rates.
• Standard NCDE hurts continuous generation: KOVAE-NCDE and GTGAN-NCDE perform worse than their non-continuous KOVAE and GTGAN counterparts (e.g., MSE at 30% on Sines: KOVAE-NCDE 0.076 versus KOVAE 0.044; GTGAN-NCDE 0.089 versus GTGAN 0.044). The authors attribute this to standard NCDE being ineffective for continuous generation.
• Generated MoE weights improve forecasting: The variant using generated MoE weights (Ours-Ŝ_MoE) beats its non-continuous version, e.g., MSE at 30% on Sines of 0.026 versus 0.049, and at 70% of 0.028 versus 0.054.
• Cubic polynomial recovery: With 2,500 samples per coefficient drawn from normal distributions and x linearly spaced over 24 values in [-1, 1], coefficients derived from Diff-MN's generated series closely match the true distribution, and the refined continuous generation is closer to ground truth than the non-continuous version; baseline outputs deviate significantly.
• Ablation — both components matter: Removing the MoE dynamics or the decoupled design degrades performance, and removing both degrades it further. On ECG5K MDD at 30% missing: Ours 0.062, w/o MoE 0.152, w/o Decoupled Design 0.191, w/o both 0.173. On ECG200 KL at 50%: Ours 0.107, w/o MoE 0.111, w/o Decoupled Design 0.502, w/o both 0.583.
• MoE beats an equivalent single expert: Comparing four experts (each with one MLP and two linear layers, eight layers total) against a single expert with four stacked MLPs (same eight layers), MoE wins across DS, MDD, and KL on ECG200, ECG5K, and ECGFD at all three missing rates — for instance, ECG200 KL 0.052 versus 0.136 at 30%.
• Dense beats sparse MoE: With sparse MoE fixed at 8 total experts and 1, 2, 4, or 6 activated, dense MoE achieves better overall performance, winning 7 versus 5 comparisons on DS and 6 versus 5 on MDD. The authors state sparse MoE is more susceptible to expert collapse under high missingness.
• Expert weights do not collapse: Appendix Table 14 reports non-sparse average expert weights, with different datasets favoring different experts.
• Privacy: Diff-MN shows the lowest privacy leakage risk according to the Membership Inference Risk metric (results in Appendix Table 10), which the authors attribute to MoE promoting diverse generated samples rather than replication.
• Expert count sensitivity: Performance is not highly sensitive to the number of experts; more than one expert improves over one, and four experts yield satisfactory results.
• Cost: MoE-NCDE has higher time complexity than standard NCDE — 1.2s versus 0.81s for double-length refined generation on 3,674 stock samples of length 12 — but generating MoE weights with the diffusion model avoids NCDE retraining and additional forward propagation.

Methodology in Plain English

• Model the series as a continuous function. Each multivariate series is treated as a continuous function over an interval [0, T], observed only at discrete, irregular timestamps. NCDE evolves a latent state driven by a spline interpolation of those observations, which is why the authors prefer it to Neural ODEs — the control path keeps the dynamics anchored to actual observations instead of drifting.
• Use several dynamics functions instead of one. A small MLP router takes cubic-interpolated observations, produces a score per expert, and softmax turns those scores into dense weights. The final dynamics function is the weighted sum of all expert networks. These weights become the "temporal dynamics fingerprint" the later stages use.
• Decouple the training. A channel-wise autoencoder is pretrained on the irregular data, then its encoder and decoder are frozen and substituted for the standard State Initialization Network and Readout Network, since they share the same input and output dimensions. Training then concentrates on the MoE dynamics function with a warm start.
• Diffuse over data and weights together. Because diffusion models need regular data, the trained MoE-NCDE first imputes each irregular sample onto a regular grid. The imputed series and their MoE weights are fed jointly into a diffusion model, which learns their joint distribution.
• Generate, then resolve. At generation time the diffusion model produces both a new sample and its matching MoE weights; the weights are plugged back into the pretrained MoE-NCDE, and the CDE solver produces values at any time point in [0, T]. The paper places the imputation, training, and generation procedures in Appendix Algorithms 1 through 4.

Why This Matters

• Research impact: The paper reframes TSG from fixed-resolution output to continuous-time output and proposes a mechanism (diffusion-generated NCDE parameters) for carrying learned dynamics over from training samples to newly generated ones. It also supplies two evaluation protocols for continuous generation — downstream forecasting with a GRU on the roughly doubled-length refined sample, and analytical recovery of cubic polynomial coefficients.
• Real-world applications (from the paper's motivation and impact statement):
  • ICU monitoring and electronic health records: irregular, sparse observations feed patient trajectory simulation and risk forecasting that need dense sequences.
  • Finance: generation for markets where irregular and sparse sampling is common.
  • Manufacturing and energy systems: simulation and augmentation where sensor sampling is not uniform.
  • Privacy-preserving data sharing and data augmentation, with the caution that generated data should complement rather than replace real data in safety-critical settings.
• Industry relevance: The code is released in Microsoft's TimeCraft repository, and the paper reports an efficiency argument for deployment: once MoE weights are parameterized by the diffusion model, new samples get weights without retraining the NCDE, avoiding extra feature extraction and forward propagation. The paper also recommends privacy safeguards such as differential privacy before deployment on sensitive data.

Future Directions

• MoE training under heavy missingness: Both dense and sparse MoE degrade as expert count increases because of insufficient data and limited class diversity for expert specialization; effective sparse MoE training and diversity-promoting mechanisms remain open.
• Scalability: The primary computational bottleneck is reported to be the native NCDE rather than MoE overhead; generation quality also depends on observation quality and on the interpolation function used to build the continuous input path.
• Evaluation on naturally irregular data: The current protocol simulates missingness and uses held-out values as ground truth; the paper states that developing evaluation protocols for naturally irregular data where ground truth is unavailable remains future work.
• Interpolation choice: The paper flags the impact of smooth interpolation functions as a question addressed in Appendix B.2, and the limitation section ties quality to the interpolation used.

Target Audience

Researchers and practitioners in generative time series modeling, continuous-time deep learning (Neural ODEs/CDEs), and diffusion models, as well as applied scientists in healthcare, finance, and industrial sensing who work with irregularly sampled data and need continuous or high-resolution synthetic series. Readers should be comfortable with differential equation-based models and mixture-of-experts architectures; the paper does not report dataset sizes or full hyperparameter settings in the main text, and defers those to its appendices.

Authors’ abstract

Time series generation (TSG) is widely used across domains, yet most existing methods assume regular sampling and fixed output resolutions. These assumptions are often violated in practice, where observations are irregular and sparse, while downstream applications require continuous and high-resolution TS. Although Neural Controlled Differential Equation (NCDE) is promising for modeling irregular TS, it is constrained by a single dynamics function, tightly coupled optimization, and limited ability to adapt learned dynamics to newly generated samples from the generative model. We propose Diff-MN, a continuous TSG framework that enhances NCDE with a Mixture-of-Experts (MoE) dynamics function and a decoupled architectural design for dynamics-focused training. To further enable NCDE to generalize to newly generated samples, Diff-MN employs a diffusion model to parameterize the NCDE temporal dynamics parameters (MoE weights), i.e., jointly learn the distribution of TS data and MoE weights. This design allows sample-specific NCDE parameters to be generated for continuous TS generation. Experiments on ten public and synthetic datasets demonstrate that Diff-MN consistently outperforms strong baselines on both irregular-to-regular and irregular-to-continuous TSG tasks. The code is available at the link https://github.com/microsoft/TimeCraft/tree/main/Diff-MN.