TIDES: Implicit Time-Awareness in Selective State Space Models

AI summary

Overview

Research area: Deep learning for sequential data — specifically state space models (SSMs) applied to irregularly sampled time series, with comparisons against both SSM baselines (S5, Mamba/S6, Mamba-2, Mamba-3, LRU) and non-SSM irregular-time-series methods (NCDE, NRDE, LogNCDE, CRU, LinODEnet, GraFITi, Neural Flows, GRU-ODE-Bayes, Rough Transformer).

Technical level: Advanced. The paper assumes familiarity with continuous-time linear systems, discretization schemes (zero-order hold, bilinear), the parallel associative scan, and the S5/Mamba literature.

Scope: The paper proposes TIDES, a selective SSM variant that keeps the discretization step equal to the physical time gap while placing input-dependence on the decay rates and on the B and C projections, and evaluates it on a controlled diagnostic plus three benchmark families.

What This Paper Is About

Sequence models face a tradeoff: continuous-time SSMs like S5 handle irregularly sampled data natively because their integration step equals the real elapsed time between observations, but their dynamics are linear time-invariant, which limits how much each token can express. Selective SSMs like Mamba gain per-token expressivity by learning the step size as a function of the input, but this destroys the step size's meaning as physical time, so irregular timestamps have nowhere to enter except as an appended feature that the model must learn and that, as the authors show, does not extrapolate. TIDES aims to get both properties at once by moving input-dependence off the discretization step and onto the diagonal state matrix.

Key Contributions

1. Architecture. TIDES (Time-Implicit Decay and Eigenvalue Selectivity): a selective SSM variant with input-dependence on (Re(λ), B, C), a static Im(λ), and an implicit Δ̃, designed so physical sampling intervals enter the model only through the discretization.

2. Mechanistic analysis. A controlled toy experiment (the "Fading Flash" benchmark) that isolates two orthogonal failure modes in existing SSMs — S5's LTI rigidity and Mamba's failure to extrapolate across training Δ — and shows TIDES is the only design overcoming both.

3. Empirical results. New state of the art, by average rank, on UEA time-series classification and the Physiome-ODE regression benchmark, and matching or exceeding the reference baseline model on 6 of 8 natively irregular datasets spanning astronomy, agriculture, neuromorphic sensing, and climate.

4. Design principle (stated in the discussion). Keep discretization and expressivity separate: the physical sampling interval should enter only through the rule converting a continuous-time generator into a discrete recurrence, while expressivity should live on the continuous-time generator itself.

Main Findings

• Fading Flash isolates two axes. In the 40-detector, three-zone task with a global clock Δ, S5 lacks expressivity: its effective learned decays stay constant per zone and it plateaus at the LTI floor, failing to reproduce three different zone decays simultaneously, though its relative error stays robust across unseen Δ. Mamba_S lacks extrapolation: its learned gate fits the training Δ distribution with low training error, but effective decay rates drift with Δ and error grows sharply outside the training range. TIDES overcomes both, with eigenvalues landing at the three target decays and robustness across a wide span of Δ.

• UEA classification. TIDES achieves the highest average accuracy across the six UEA datasets, surpassing Mamba-3 by approximately 1.5 points on average, ranking first on 2 of 6 datasets, and posting the best average rank (3.00) and the best average accuracy (66.1). Mamba-3 follows at 64.7 average with rank 4.17, then LogNCDE (64.3, rank 4.25), RFormer (64.5, rank 5.17), S6 (62.0, rank 6.33), LRU (61.7, rank 6.50), S5 (61.8, rank 7.08), and Transformer (59.0, rank 7.50).

• Physiome-ODE. On the 50 irregular-time-series forecasting datasets derived from biophysical models, TIDES attains the best average rank (2.40), statistically on par with LinODEnet (2.62) and tying it on per-dataset wins (16 of 50). A bootstrap 95% CI on the average-rank difference between TIDES and LinODEnet is [-0.87, +0.34], which includes zero but is skewed in TIDES's favor. Remaining average ranks: GraFITi 2.82, GraFITi-C 3.54, CRU 3.68, Neural Flows 5.76, GRU-ODE-Bayes 6.88.

• Random drop on EigenWorms. Training at r_train = 0.5 with a different random 50% of timesteps dropped per seed and n = 3 seeds: S5 and TIDES-with-only-Λ plateau near chance regardless of r_test, showing LTI B, C caps expressivity. Mamba, Mamba-2, and Mamba-3 match TIDES for r_test ≤ 0.7 but collapse from roughly 0.7 to roughly 0.4–0.6 at r_test = 0.9; RFormer fails analogously. TIDES achieves the best mean accuracy (0.739) and stays flat across the full r_test range. Adding input-dependence to Im(λ) does not help and slightly hurts.

• Natively irregular benchmarks. TIDES sets the best reported result on TimeSen2Crop (89.80 OA vs. 88.70 for MLP+LTAE), EuroCropsML (80.26 ± 0.41 vs. 79.14 ± 0.23 for DirPA, p = 0.0015, 500-shot Estonia setting), and USHCN (MSE 0.147 ± 0.003 vs. 0.159 ± 0.014 and MAE 0.211 ± 0.002 vs. 0.261 ± 0.017 for APN). It exceeds Event-SSM, whose pipeline it plugs into, on SHD (96.1 vs. 95.9) and SSC (88.6 vs. 88.4). It matches iTCN on MACHO (92.8 vs. 92.7) and is slightly below on ASAS-SN (94.2 vs. 94.5), with mean paired differences over the 8 splits of +0.13 ± 0.32 and −0.36 ± 0.17 respectively, while outperforming every other model reported by Zhang and Bloom. On DVS128 it is slightly below the updated Event-SSM result (98.9 vs. 99.2; the Event-SSM paper reports 97.7, and the 99.2 figure is an updated result from the authors' code release v0.2 after a fix to DVS event tokenization).

• Why the margins vary. The authors attribute the varying advantage to how much each task benefits from three properties: per-token input-dependent processing, faithful handling of irregular sampling, and extrapolation to unseen sampling regimes. On Physiome-ODE the strongest baseline LinODEnet is itself continuous-time and shares two of the three properties, so the gap narrows.

Methodology in Plain English

A diagonal SSM has four places where input-dependence could go: the state matrix Λ (whose real part is a decay rate and imaginary part an oscillation frequency), the input projection B, the output projection C, and the discretization step Δ̃. S5 leaves everything static. Mamba makes B, C, and Δ̃ input-dependent, which turns Δ̃ into a learned gate and breaks its physical meaning.

TIDES makes Re(Λ), B, and C functions of the input, leaves Im(λ) static, and keeps Δ̃ equal to the real elapsed time. The reasoning: making decay rates input-dependent lets the model forget faster at an event boundary or hold longer when something should be remembered, and this is the closest semantic analog to Mamba's selectivity that does not conflict with physical time. Modulating an oscillation frequency per token, by contrast, is described as having no clean interpretation because the model would redefine its own dynamical basis at every step; the ablation confirms it hurts. Making Δ̃ a learned gate is rejected because a real irregular timestamp then has nowhere to enter except as an input feature whose relationship to the internal gate does not extrapolate.

Implementation details: three projection heads produce Re(Λ_k), B_k, and C_k affine in the input; the bias terms are initialized to S5's HiPPO-derived values and the projection weights are zero-initialized, so training starts from a well-conditioned HiPPO recurrence. Because a dense B projection has 2PH² parameters (for typical P = 128, H = 128, larger than the rest of the SSM), B and C are factored into low-rank up/down products, dropping the per-head cost from O(PH²) to O(rPH). Optional stacked residual GLU blocks add nonlinear preprocessing before the final projection, with depths d_Λ and d_enc that default to 0 (a plain linear map). RMSNorm is applied to the projected Re(Λ_k) and a complex-valued variant to the projected B_k and C_k before discretization, to bound perturbations that would otherwise be exponentially amplified through the recurrence. Re(Λ) is reparameterized (for example exponentially or via the stable form −1/(θ² + 1/2)) to keep modes strictly in the stable half-plane; the choice is treated as a hyperparameter.

Evaluation uses a controlled synthetic diagnostic first (Fading Flash), then three benchmark families: UEA time series classification following the Rough Transformer environment, the Physiome-ODE regression benchmark (50 datasets) following the Klötergens et al. protocol, and natively irregular real-world benchmarks. A random-drop ablation on EigenWorms trains six roughly 30k-parameter SSM variants differing only in which of {Re(Λ), Im(Λ), B, C, Δ̃} is input-dependent, plus three official Mamba variants and RFormer. TIDES is implemented in PyTorch with hyperparameters tuned per dataset within a fixed budget; all experiments use approximately 5,000 GPU-hours of A100 time.

Why This Matters

Impact on research. The paper reframes a tradeoff usually treated as a matter of design philosophy — Mamba's per-token expressivity versus S5's faithful physical-time semantics — as separable. It provides both a concrete mechanism for separating them and a controlled diagnostic that attributes specific failure modes to specific architectural choices, which is a reusable methodology for future sequence-model work. It also sets new average-rank results on UEA and Physiome-ODE and reports results on 8 natively irregular datasets from four domains.

Real-world applications.

• Clinical monitoring, where observations arrive whenever a measurement is ordered rather than on a grid.
• Wearable health sensing, where streams arrive irregularly, sensors drop and resume, and compute and memory budgets are tight.
• Scientific and industrial instrumentation, cited in the paper as seismology and astronomical surveys, where phenomena unfold outside the experimenter's control.
• Environmental and agricultural monitoring, spanning Sentinel-2 crop time series and weather-station records, where revisit times, cloud cover, and per-variable reporting schedules make every sample irregular.

Industry relevance. The paper notes that TIDES inherits the SSM family's linear training scaling and constant per-step inference cost in recurrent mode while adding native handling of irregular timestamps without explicit time features. That combination, the authors argue, suits real-time sensor settings on resource-constrained hardware, since such systems can operate without engineering an explicit time-encoding pipeline or maintaining separate logic for missing observations. The code is released at https://github.com/TaylanSoydan/TIDES.

Future Directions

• Scale to language modeling. The authors explicitly do not claim TIDES as a general-purpose replacement for Mamba on regular-grid tasks such as language modeling, where the input-dependent Δ̃ gate plays a different role; whether the principle survives at language-model scale is left as an open question, and language modeling is listed among the limitations (not evaluated).
• Beyond physical time. The principle extends to any setting where Δ_k encodes a meaningful continuous parameter along which the sequence evolves, such as trajectory modeling parameterized by arc length, rather than wall-clock time.
• Active sampling. A model with a faithful notion of its step parameter could be coupled to an acquisition rule that decides where the next observation should be taken, which the authors note is relevant where observations are costly.
• Robust anytime forecasting. Because Δ̃ is physical at inference, a trained TIDES can in principle forecast at arbitrary future steps without retraining by feeding the desired Δ at readout; characterizing how this degrades far from the training horizon distribution is stated as a concrete open question. Additional open items include the extra tuning burden from the rank r, projector depths, and reparameterization choice, and the absence of a formal expressivity result for the claim about input-dependent Im(Λ).

Target Audience

Researchers and practitioners working on sequence models for irregularly sampled or event-driven time series, especially those already familiar with SSMs (S4, S5, Mamba and its successors) and with neural ODE and neural CDE approaches. It is most useful to readers interested in architecture design for time-aware sequence modeling, benchmarking methodology for irregular-time regimes, or deployment on sensor and wearable hardware. It is not aimed at readers seeking an introduction to state space models or to language modeling applications, which the paper does not cover.

Authors’ abstract

Selective state space models (SSMs), such as Mamba, achieve strong per-token expressivity by making the time discretization step $\TildeΔ$ a learned function of the input. However, in doing so, $\TildeΔ$ no longer equals the physical time gap $Δ$ between consecutive observations, limiting the ability of these models to handle irregular time series. Continuous time SSMs, such as S5, keep $\TildeΔ\equivΔ$ and therefore handle irregular timestamps natively, but their dynamics remain linear time invariant (LTI), limiting per token expressivity. We propose \textbf{TIDES}, a selective SSM variant that reconciles selective and continuous architectures by moving input dependence off the step size and onto the diagonal state matrix. As a result, $\TildeΔ\equivΔ$ as in S5, allowing the model to handle irregular timestamps natively without sacrificing the per-token expressivity that makes selective SSMs effective. We show this on a novel \emph{Fading Flash} experimental benchmark, a compact controlled diagnostic for sequence models that jointly tests input dependence and extrapolation to out-of-distribution $Δ$ values, and isolates the distinct failure modes of current state-of-the-art architectures that TIDES avoids by construction. On large-scale benchmarks, TIDES sets the new best average rank on UEA time series classification and the Physiome ODE regression benchmark, and matches or exceeds the reference baseline model on 6 of 8 natively irregular datasets from astronomy, agriculture, neuromorphic sensing, and climate events. Code available at: \url{https://github.com/TaylanSoydan/TIDES}.