LogSig-SSM: Time-Series Modelling with Multi-Scale Log-Signature Compression for State-Space Models

AI summary

Overview

Research area: Machine learning for time-series modelling, specifically architectures for long, irregularly sampled multivariate sequences — combining selective state-space models (SSMs, e.g. Mamba) with signature-based path compression.

Technical level: Advanced. The paper assumes familiarity with controlled differential equations, state-space discretisation (zero-order hold), the signature and log-signature transforms, free Lie algebras, and the theory of neural rough differential equations.

Scope: The paper proposes a two-stage architecture that compresses long multivariate time series into short token sequences using multi-scale windowed log-signatures before feeding them to a selective SSM backbone, and evaluates it on four benchmarks.

What This Paper Is About

Time series in domains such as healthcare and finance are often sampled irregularly at high frequencies, so models must handle long horizons without excessive compute. Continuous-time models (NCDEs, NRDEs) handle irregular sampling well but scale poorly to long sequences, while selective SSMs like Mamba scale linearly but have two documented weaknesses: performance degradation on very long sequences, and a diagonal state transition that admits no mixing across hidden dimensions within a single block (so genuinely multivariate path functionals such as Lévy areas lie outside the closure of one block). LogSig-SSM addresses both by moving cross-channel geometric information to the input: it computes truncated log-signatures over strided, multi-scale windows to build compact tokens, then runs a diagonal selective scan over the much shorter token sequence.

Key Contributions

1. LogSig-SSM architecture. A model that integrates log-signature tokenisation with selective SSMs. The tokeniser compresses long, irregularly sampled multivariate sequences into compact summaries by computing truncated log-signatures over multi-scale windows (window lengths L_k, truncation depths m_k, stride s), producing a token sequence of length T' ≈ T/s.

2. Continuous-time interpretation. Extending the linear-CDE view of SSMs, the authors interpret LogSig-SSM as an NCDE/NRDE-style system driven by log-signature-derived controls, where selectivity induces input-dependent rescaling of the latent dynamics.

3. A depth–truncation tradeoff. Proposition 4.1: a single LogSig-SSM block followed by an MLP can approximate any continuous functional of the windowed log-signature up to depth m, while the same class of functionals lies outside the closure of a single diagonal Mamba block (even with a pointwise MLP) and requires chained blocks to recover.

4. Four-benchmark evaluation. Long-sequence time-series classification (UEA), high-frequency physiological regression (PPG-DaLiA), multivariate weather forecasting, and irregularly sampled early sepsis prediction (PhysioNet Sepsis) — matching or exceeding strong SSM and continuous-time baselines while reducing training time by up to 30× and GPU memory by up to 37× relative to standard Mamba on the longest sequences.

Main Findings

• UEA long-sequence classification: Across the six longest-sequence UEA datasets (split 70:15:15, five random seeds), LogSig-SSM achieves the best average accuracy across the sixteen models evaluated (71.7%) and the best average rank (2.7).

• EthanolConcentration advantage: On this 4-class task (chance 25%), diagonal SSMs fail to surpass 30% accuracy and the best non-SSM baseline reaches 40.5%, while LogSig-SSM reaches 51.4%. The paper attributes this to the signal being an absorption curve whose classes differ by smooth shifts in peak shape over extended regions, a global trajectory geometry multi-scale log-signature tokens capture directly.

• Efficiency on EigenWorms (T = 17,984): LogSig-SSM trains in 4 s per 1000 steps versus 122 s for Mamba (30× faster) and uses 362 MB GPU memory versus 13,486 MB for Mamba (37× less), with test accuracy of 95.3% versus 70.9% for Mamba. Tokenisation is a one-time offline step costing 1.6 s for the full dataset and 6 s for all six UEA datasets together, about 1% of a single training run per dataset.

• Memory advantage over LinOSS-IM: LogSig-SSM leads on EigenWorms while requiring 10× less memory than LinOSS-IM.

• Concrete baseline comparison (EigenWorms): Memory in MB — NCDE 2484, NRDE 2506, Log-NCDE 2510, Mamba 13486, S5 6646, LinOSS-IM 3488, D-LinOSS 3488, LogSig-SSM 362. Time in s per 1000 steps — NCDE 24595, NRDE 5386, Log-NCDE 1956, Mamba 122, S5 31, LinOSS-IM 14, D-LinOSS 14, LogSig-SSM 4. Accuracy — NCDE 75.0, NRDE 83.9, Log-NCDE 85.6, Mamba 70.9, S5 81.1, LinOSS-IM 95.0, D-LinOSS 93.9, LogSig-SSM 95.3.

• Irregular sampling: The local within-window reparameterisation invariance is claimed sufficient in practice — the model is highly competitive with specialised irregular-time methods on PhysioNet Sepsis, where only 10.3% of values are observed.

• Scan complexity reduction: The selective scan cost scales as O(B_sz · T · N · D); compressing from T to T' reduces scan FLOPs by a factor of T/T' ≈ s.

• Comparative context on other benchmarks (as shown in the provided content): PPG-DaLiA heart-rate regression MSE (×10⁻²) — Log-NCDE 9.56 ± 0.59, NRDE 9.90 ± 0.97, Mamba 10.65 ± 2.20, LRU 12.17 ± 0.49, S5 12.63 ± 1.25, S6 12.88 ± 2.05, NCDE 13.54 ± 0.69, LinOSS-IMEX 7.50 ± 0.46, LinOSS-IM 6.40 ± 0.23. Weather forecasting MAE — LinOSS-IMEX 0.508, LinOSS-IM 0.528, S4 0.578, Informer 0.731, Informer with canonical self-attention 0.741, LSTnet 0.757, LogTrans 0.773, LSTMa 1.109, Reformer 1.575. PhysioNet Sepsis AUROC (with observation indicators / without) — Neural LNSDE 91.1 ± 0.2 / 88.1 ± 0.2, Neural LSDE 90.9 ± 0.4 / 87.9 ± 0.8, Neural GSDE 90.9 ± 0.1 (no-OI value not shown), ANCDE 90.0 ± 0.2 / 82.3 ± 0.3, Neural CDE 88.0 ± 0.6 / 77.6 ± 0.9, ODE-RNN 87.4 ± 1.6 / 83.3 ± 2.0, GRU-D 87.1 ± 2.2 / 85.0 ± 1.3, GRU-Δt 87.8 ± 0.6 / 84.0 ± 0.7, GRU-ODE 85.2 ± 1.0 / 77.1 ± 2.4.

• Not reported in the provided content: The LogSig-SSM rows for the PPG-DaLiA, weather-forecasting, and PhysioNet Sepsis sub-tables are cut off in the supplied text, so the exact LogSig-SSM numbers for those three benchmarks are not reported here.

Methodology in Plain English

The pipeline has two stages, illustrated in Figure 1.

Stage one — compress with log-signatures. The long input series is sampled at strided anchor points spaced s apart, giving T' = floor((T−1)/s) tokens. At each anchor, the model looks back over several window lengths L_k (multiple scales) and computes a truncated log-signature of the piecewise-linearly interpolated window, truncated at depth m_k. The log-signature is a fixed, redundancy-free coordinate description of the path's iterated integrals, including cross-channel terms like Lévy areas, and it is invariant to monotone reparameterisation of time — meaning it depends on the shape of the path rather than on how densely it was sampled. Coordinate vectors from each scale are concatenated into a token, then linearly projected to the scan width.

Stage two — run a selective SSM. These tokens are fed to a stack of Mamba blocks. The discretisation step, input projection, and output projection are all input-dependent functions of the current token, so the model adaptively rescales how fast its latent state evolves based on content.

Efficiency rationale. Tokenisation is a one-time offline pre-processing step — the precomputed features are not in the training-time backpropagation graph — and it is implemented with optimised libraries (iisignature, Signatory). Because the expensive sequential scan now runs over T' ≈ T/s rather than T, both scan FLOPs and memory drop accordingly.

Theory. The authors show the model admits a linear-CDE representation driven by log-signature-induced functionals, and that under the zero-order-hold extension the drivers are affine on each interval, reducing the system to an NRDE-style integral form in which Δω acts as a content-dependent internal-time allocation per interval. Two differences from NRDEs are highlighted: LogSig-SSM uses multi-scale tokens (K different window lengths) rather than one window, and its drivers are content-dependent rather than a fixed interval parameterisation. Proposition 4.3 establishes within-window invariance to monotone time reparameterisation.

Evaluation setup. Four benchmarks: UEA (Log-NCDE protocol), PPG-DaLiA (Log-NCDE protocol), multivariate weather forecasting (LinOSS protocol, predicting 30 days of hourly data from the preceding 30 days), and PhysioNet Sepsis (Neural CDE protocol). All experiments ran on one NVIDIA RTX 4090 GPU.

Why This Matters

The work reframes a known expressivity limitation of diagonal selective SSMs not as something to fix inside the recurrence — by stacking blocks or by replacing the diagonal transition with richer structure (PD-SSM reports approximately 6× slower runtime than Mamba) — but as something to pre-empt at the input. If the efficiency figures hold broadly, this makes long-horizon, irregularly sampled modelling practical on hardware where Mamba's memory footprint is prohibitive. Code is released at https://github.com/felix-oury/LogSigSSM.

Real-world applications:

• Clinical monitoring and early warning: PhysioNet Sepsis early prediction, where only 10.3% of values are observed, is directly relevant to deploying models on sparse electronic health record data.
• Wearable physiological sensing: PPG-DaLiA heart-rate regression represents continuous vital-sign estimation from consumer or medical wearables sampled at high frequency.
• Financial time series: The introduction cites transaction prices as an example of signals sampled irregularly at high frequency with long-range structure.
• Weather and climate forecasting: Multivariate weather forecasting over 30-day horizons, useful for energy load planning and agricultural decision-making.

Industry relevance: Reduced GPU memory (362 MB versus 13,486 MB on EigenWorms) and training time (4 s versus 122 s per 1000 steps) lower the hardware barrier for training long-context sequence models, and moving tokenisation offline means the expensive geometry computation is not repeated in every training step. Any industry pipeline that stores long, irregularly sampled sensor or event data — healthcare, wearables, finance, industrial telemetry — is a candidate beneficiary.

Future Directions

• Extending beyond local invariance: Proposition 4.3 guarantees invariance only within each window; the paper relies on empirical results on PhysioNet Sepsis to argue this local guarantee suffices. A global, cross-window invariance guarantee is a natural open question.

• Hyperparameter selection for scales and truncation depth: The method introduces K window scales with lengths {L_k}, truncation depths {m_k}, and a stride s, with D_tot growing as the sum of per-scale coordinate dimensions. How to choose these per dataset systematically is not resolved in the provided content.

• Scaling the truncation analysis: Remark 4.2 notes that as m_k → ∞ a single LogSig-SSM block approximates any continuous functional of windowed paths by signature universality; the practical regime where higher depths repay their token-size cost remains open.

• Closing the measured gaps: The LogSig-SSM entries for the PPG-DaLiA, weather, and Sepsis sub-tables are not present in the provided content, so the precise margins on those three benchmarks — and what remains to be improved there — cannot be assessed from this material.

Target Audience

Machine learning researchers working on sequence architectures, particularly those interested in state-space models, continuous-time models, and signature methods. It is also relevant to practitioners in healthcare, wearable sensing, finance, and weather forecasting who need to model long multivariate series with irregular sampling under tight memory budgets. Readers without a background in rough path theory or controlled differential equations will find the theoretical sections demanding, though the two-stage pipeline itself is conceptually accessible.

Authors’ abstract

Time-series data are often sampled irregularly at high frequencies and exhibit long-range dependencies, which makes long-horizon modelling difficult. Continuous-time models such as neural controlled differential equations (NCDEs) and neural rough differential equations (NRDEs) can handle irregular sampling, but they scale poorly to long sequences. Selective state-space models (SSMs) such as Mamba scale linearly with sequence length, but they provide limited recurrent mixing across hidden dimensions within a single block. We propose LogSig-SSM (Log-Signature Compression for State-Space Models), which first compresses long multivariate time series into a shorter sequence of tokens using multi-scale windowed log-signatures, and then processes these tokens with a selective SSM backbone. LogSig-SSM is scalable and robust to irregular sampling, combining log-signature tokens that capture higher-order cross-channel interactions with a selective SSM that models long-range dependencies. The model also admits a continuous-time interpretation as an NCDE/NRDE-style system driven by a log-signature-based input, in which selectivity induces an input-dependent rescaling of the latent dynamics. Across four benchmarks, namely long-sequence classification on UEA, high-frequency physiological regression on PPG-DaLiA, multivariate weather forecasting, and irregularly sampled clinical prediction on PhysioNet Sepsis, LogSig-SSM outperforms or matches strong SSM and continuous-time baselines while training up to $30\times$ faster and using up to $37\times$ less GPU memory than Mamba on the longest sequences.