HiPPO Functional Memory: Coordinate-Queryable Attention for Irregular Time Series

AI summary

Overview

Research area: Machine learning for irregularly sampled time series — specifically, continuous-time sequence modeling, attention mechanisms, and structured/functional memory (HiPPO polynomial projections, interdomain kernel features, RKHS theory).

Technical level: Advanced. The paper combines a concrete neural operator with functional-analytic results (RKHS representers, Gram solves, projection error bounds), so comfortable reading requires familiarity with attention, kernel methods, and orthogonal polynomial bases.

Scope: The paper proposes HiPPO-FM, a drop-in attention-style operator that compresses an irregular observation history into a fixed-size, coordinate-queryable functional state, and evaluates it on reconstruction, classification, extrapolation, and event-sequence prediction.

What This Paper Is About

Irregular time series are observed at whatever times data collection happens to produce, but downstream tasks ask questions at other times — interpolating missing points, extrapolating past the record, or re-reading the same history at a new resolution. Most existing models keep a representation tied to the observed samples (tokens, patches, graph nodes, or latent trajectories), so every new query set must be re-resolved against that history. The paper asks whether an irregular history can instead be compiled once into a function-valued memory that can be evaluated directly at any requested time coordinate.

Key Contributions

1. Coordinate-queryable functional attention. HiPPO-FM writes observations into a fixed-rank functional state and predicts by evaluating that state at requested coordinates, rather than re-running attention over the observation-indexed history. The retained state supports arbitrary query sets with storage that does not grow with history length.

2. A matched write/read geometry with content interaction before compression. Legendre/HiPPO modes combined with interdomain kernel geometry define one shared functional system used for both the memory write and the coordinate readout, with a Gram-based solve correcting correlations between functional coordinates. Masked self-attention is applied on the observed support before the write (an "MHA-to-FM" ordering), so what enters memory is content-dependent while the query space stays functional.

3. Functional analysis of the retained state. The paper characterizes the finite-dimensional query space induced by the readout (raw memory entries are interdomain functionals of an empirical kernel field; each scalar output function lies in a span of at most M representer functions, plus a constant from affine biases), and derives a pointwise error decomposition splitting functional-geometry approximation error from learned-memory mismatch (Theorem 3.3).

4. Reusable inference and a multi-setting empirical study. Because the state is independent of the query set, encoding is done once and reused; the paper studies the operator on continuous-coordinate reconstruction, 20-dataset UEA classification under synthetic missingness, real extrapolation, and irregular event-sequence prediction.

Main Findings

• Spiral reconstruction (interpolation). At noise level ρ = 0.02, HiPPO-FM records RMSE 0.42 ± 0.08 and MAE 0.42 ± 0.09 (×10⁻²), versus 0.69 ± 0.16 for ContiFormer, 0.95 ± 0.21 for a standard Transformer, 1.03 ± 0.19 for Neural ODE, 1.16 ± 0.25 for Latent ODE, and 1.32 ± 0.95 for Neural CDE. At ρ = 2, HiPPO-FM reaches RMSE 0.69 ± 0.22 while the next best, ContiFormer, is at 2.06 ± 2.39.

• Spiral extrapolation is where the gap is largest. On the withheld half of the trajectory at ρ = 2, HiPPO-FM reports RMSE 0.80 ± 0.17 and MAE 0.73 ± 0.08, against 3.11 ± 2.89 (ContiFormer), 3.70 ± 2.03 (Neural CDE), 4.18 ± 1.03 (Neural ODE), 4.94 ± 1.05 (Latent ODE), and 5.23 ± 1.08 (Transformer).

• Not a clean sweep on Spiral. The paper states HiPPO-FM has the lowest mean in most rows and remains competitive in the rest, and explicitly notes ContiFormer is strongest in one interpolation condition — the ρ = 0.2 interpolation row, where ContiFormer reports RMSE 0.58 ± 0.14 and HiPPO-FM 0.61 ± 0.22.

• Strongest overall UEA accuracy under equal-drop aggregation. Averaged over the 30%, 50%, and 70% dropped-observation settings across 20 UEA datasets, HiPPO-FM reports overall accuracy 0.7306 with overall rank 1.00, ahead of S5 (0.7272, rank 2.00), ContiFormer (0.7167, rank 3.00), CADN (0.7152, rank 4.00), and far ahead of mTAN (0.5418) and Neural CDE (0.5119).

• The advantage grows with heavier masking. HiPPO-FM is best at 50% dropped (0.7364) and 70% dropped (0.7058), while at 30% dropped it is essentially tied with S5 (0.7496 versus 0.7497). The paper notes ContiFormer leads the per-setting rank at the two larger drop ratios (4.43 vs 4.50 at 50%; 4.53 vs 4.55 at 70%).

• UEA classification tests the encoder, not the readout. The paper is explicit that classification pools the observed-support representation via last-valid pooling and bypasses the final P→R functional readout, so those results measure the history encoder and retained representation rather than coordinate querying.

• Event-sequence prediction. Across Synthetic, Neonate, Traffic, MIMIC, StackOverflow, and BookOrder, the paper reports that HiPPO-FM is strongest on several displayed datasets, especially in event-type accuracy and next-event-time RMSE. The HiPPO-FM row of Table 3 is not present in the supplied text, so its exact values are not reproducible from this content. For reference, the strongest displayed competitors include ContiFormer (Synthetic time RMSE 0.188 ± 0.006), ODE-RNN (Neonate 9.289 ± 0.048), NSTKA (MIMIC 0.842 ± 0.008), SAHP (Traffic LL 0.372 ± 0.022 and BookOrder LL −0.304 ± 0.002), and GRU-Δt (BookOrder accuracy 0.627 ± 0.000).

• Theory: the query space is genuinely finite-dimensional. Proposition 3.2 shows that, for exact functional values, zero jitter, and a nonsingular Gram matrix, the Gram inverse maps stored functional values to the coefficients of their RKHS projection, so readout is a projection onto a span of at most M functions; shared geometry across heads and linear channel mixing preserve that span.

• Theory: errors separate cleanly. Theorem 3.3 bounds pointwise error by a geometry term (‖f‖ times the square root of a residual kernel variance c_ε(r), which is ≥ 0) plus a memory term (readout norm times the deviation of the learned memory from exact functional values) — meaning coordinate-system limitations and imperfect memory estimation are diagnosable separately.

• Storage is history-length independent. After encoding, the persistent state is O(Md), independent of the number N of observed timestamps, and the same state can serve multiple query sets without re-running the encoder, functional write, or geometry computation.

• PhysioNet extrapolation is claimed but not quantified here. The introduction lists "real irregular extrapolation on PhysioNet" as one of the studied settings, but no PhysioNet numbers appear in the supplied text.

Methodology in Plain English

The design has three moving parts.

1. A shared "coordinate system" for time. The model builds a set of smooth basis functions from Legendre polynomials (the family behind HiPPO-LegS), evaluated on a fixed reference grid, and then smooths them through a learnable positive-definite kernel. This produces a feature vector for any time point — whether it was observed or merely requested. Crucially, the same feature construction is used on both sides: observed timestamps get features for writing, requested timestamps get features for reading.

2. Writing observations into a fixed-size memory. Each observation is projected to a value, multiplied by its coordinate feature, and summed. That sum has one entry per basis mode rather than one entry per observation — so the memory shrinks from N rows to M modes. Because basis functions overlap, the raw sums are correlated; the model therefore applies a normalization step and then solves a small linear system built from the Gram matrix of the basis (via Cholesky solves) to obtain clean coefficients. This is the "matched" part: the correction applied at write time is exactly the one the readout inverts.

3. Letting content interact before compression. Before any of this compression happens, ordinary masked self-attention runs over the observed positions, followed by a residual connection, a functional-memory block applied from the observed support to itself, layer normalization, and a feed-forward network — one post-norm Transformer block in total. So attention decides what to store based on relationships among observations, while the functional geometry decides how it is stored and queried. For coordinate-valued tasks, a separate output FM head reads the encoded history at the requested coordinates; for classification, last-valid pooling is used instead. A padding/validity mask handles missing entries, and since the encoder only sees a permitted prefix during forecasting, no extra causal mask is needed inside it.

Experimental protocol. Default configuration is one encoder block, four heads, width 64, shifted-Legendre coordinates, and a learned squared-exponential kernel. Spiral experiments use a five-seed protocol with 5,000-iteration runs and a sweep over noise level ρ; UEA experiments delete 30%, 50%, or 70% of observations while keeping timestamps; event experiments use four-fold cross-validation on Synthetic, Neonate, and Traffic and five-fold on MIMIC, StackOverflow, and BookOrder.

Why This Matters

For research. The paper reframes the irregular-time-series problem from "how do we attend over unevenly spaced tokens" to "what function should we leave behind after encoding." The Theorem 3.3 decomposition gives practitioners a way to attribute error to the basis/kernel choice versus the learned memory — a diagnostic that observation-indexed architectures do not naturally provide. It also connects the practical attention literature to classical interdomain functional approximations and HiPPO projections.

Real-world applications.

• Clinical monitoring: labs and vitals arrive at irregular times, and clinicians query at times no sensor recorded — plus risk scores are often needed beyond the observed window, where this paper's extrapolation gains are largest.
• Environmental and climate sensing: buoy, station, and satellite measurements are unevenly spaced; the same history must be re-read at different temporal resolutions for different analyses.
• Event logs and transactional systems: user actions, order flows, and support tickets form marked event sequences where both the type and the timing of the next event matter.
• Multi-resolution downstream pipelines: one encode, many queries — e.g., a single retained state serving hourly, daily, and event-triggered readouts without reprocessing the raw record.

Industry relevance. The O(Md) post-encoding storage and the ability to reuse a state across query sets map directly onto serving architectures: encode once per record, cache a small fixed-size state, and answer arbitrarily many time-indexed queries cheaply. Because the operator is a drop-in attention variant inside a standard Transformer block, adoption cost is low relative to switching to a latent-ODE or CDE solver. The strongest gains appearing under heavy missingness (70% dropped) and long-horizon extrapolation are precisely the regimes where deployed sensors and clinical records are least cooperative.

Future Directions

• Close the event-sequence gap. The paper claims superiority on several event benchmarks "especially in event-type accuracy and next-event-time RMSE," but the exact HiPPO-FM numbers and the degree of improvement over SAHP, THP, and NSTKA are not visible in the supplied text; a fuller accounting would clarify how much of the gain comes from the functional state versus the readout design.

• Tune or learn the geometry itself. Theorem 3.3 holds for a fixed geometry, and the authors note that separately trained configurations can also differ through their learned kernels and optimization trajectories. Whether the basis size M, the reference grid, and the kernel family should be selected per-dataset — or meta-learned — remains open.

• Explain the regime split. HiPPO-FM is best at moderate noise on interpolation but loses to ContiFormer at ρ = 0.2, and it wins overall UEA accuracy while ContiFormer wins two of three per-setting ranks. Understanding when observation-indexed attention is genuinely preferable would make the architectural choice principled rather than empirical.

• Extend beyond single-history coordinate readout. The framework currently handles one history and one query support; hierarchical or cross-series versions (multiple patients, multiple sensors sharing a functional space) and combining it with the patch-, graph-, and frequency-based reorganizations cited in related work are natural next steps.

Target Audience

Researchers and graduate students working on irregular or continuous-time sequence modeling who already know Transformer attention and are comfortable with kernel methods and RKHS arguments. It is also useful for practitioners building clinical, sensor, or event-log pipelines who need repeated temporal queries over a cached representation and are deciding between latent-ODE/CDE approaches and attention-based ones. Readers wanting only the empirical story can take Tables 1–2 and the extrapolation results; readers wanting the conceptual payoff should read Sections 3.2–3.4, where the matched write/read geometry and the error decomposition live.

Authors’ abstract

Irregular time series require models to answer temporal queries at arbitrary coordinates, yet existing approaches typically retain observation-dependent representations that must be repeatedly resolved for each query. We introduce HiPPO Functional Memory (HiPPO-FM), a functional-memory attention operator that compiles an irregular history into a reusable coordinate-addressable state. Instead of storing observation-indexed tokens or latent trajectories, HiPPO-FM writes observations into a fixed-rank functional space and performs direct coordinate evaluation through matched functional geometry. Kernel-smoothed HiPPO coordinates and Gram-based readout define a shared write/read mechanism, while observed-support self-attention enables content-dependent representations before compression. We characterize the induced finite-dimensional query space and derive an error decomposition separating functional geometry approximation from learned-memory mismatch. Across continuous-coordinate reconstruction, irregular representation learning, real-world extrapolation, and event-sequence prediction, HiPPO-FM consistently improves strong irregular time-series baselines, achieving strong reconstruction accuracy on Spiral, competitive performance across 20 UEA datasets, improved real-world extrapolation, and superior results on several irregular event benchmarks. After encoding, the compact functional state can be reused across different query sets without reprocessing the observation history.