Lévy Attention: Single-Pass Predictive Uncertainty for Continuous-Time Attention

AI summary

Overview

Research area: Machine learning — attention mechanisms and predictive uncertainty quantification for irregularly-sampled multivariate time series (IMTS).

Technical level: Advanced. The paper combines stochastic-process theory (inhomogeneous Poisson random measures, Lévy–Itô decomposition, compound-Poisson integrals) with attention architecture design and an empirical benchmark suite.

Scope: The paper introduces Lévy Attention, a cross-attention operator defined as a stochastic integral against a Poisson random measure, which emits per-query uncertainty statistics in closed form from the same forward pass that produces the prediction.

Publication details: arXiv:2608.19171v1 [cs.LG], 19 Aug 2026. Authors Sotirios P. Chatzis (GenML Laboratory, Department of Electrical Engineering and Computer Science and Engineering, Cyprus University of Technology, Limassol, Cyprus) and Loukas Papadoulas (Ethical AI Novelties, Cyprus).

What This Paper Is About

Deep models for irregularly-sampled time series answer queries at arbitrary continuous timestamps, but report nothing about how far each answer should be trusted. Standard approaches to uncertainty — deep ensembles, MC dropout, diffusion-based imputers, and learned variance or confidence heads — all pay for uncertainty separately from prediction, through multiple models, multiple passes, or an extra trained output. The paper's goal is to show that the attention layer itself can close that gap: with the right stochastic formulation, the pass that makes each prediction also reports how far that prediction should be trusted, in closed form and at no extra cost.

Key Contributions

1. Mechanism. Attention formulated as stochastic integration against an inhomogeneous Poisson random measure on a (time × channel) index space, with a deterministic mean path for training and a sampling path for on-demand error bars. Query–key compatibilities assemble an intensity over the index space, the measure scatters atoms under it, and the output averages an interpolated value field at those atoms.

2. Theory. A mean identity (Theorem 1) showing the deterministic mean path is exactly unbiased for the sampled operator, and an exact variance identity (Theorem 2) showing the output's expected squared deviation from its mean equals tr Σ_V(q) · φ(Λ_q). The scale factorises as disagreement over evidence, and a bound is given on the mollification bias against discrete cosine-kernel attention.

3. Evidence for "the uncertainty is free." Across experiments on the t-PatchGNN benchmark protocol, PhysioNet and USHCN, the authors report that accuracy survives the operator swap, that the free disagreement signal beats 20-pass MC dropout on AUSE in every seed of matched five-seed suites on a dense and a sparse benchmark, that calibrated and split-conformal intervals follow from the identity, and that one pass screens an unseen cohort of 3,383 records.

4. Ablations tying each ingredient of the construction to its effect, including a grid-resolution sweep, a Poisson-sampled training arm, and comparisons against training-free and trained estimators.

Main Findings

• Two free signals are emitted by the deterministic pass. The Poisson construction preserves two quantities that softmax attention discards: the evidence Λ_q (total intensity, i.e. the partition function that row normalisation divides out) and the disagreement tr Σ_V(q) (the spread of attended values). Their combination σ̂(q) = sqrt(tr Σ_V(q) · φ(Λ_q)) is reported to be the root-mean-square deviation of the sampled operator. No head is trained for these signals; the same forward pass emits them, at O(L) extra cost per query on top of a shared O(Ld) precompute.

• The mean path trains with exact gradients. In expectation, Lévy Attention reduces to a mollified cosine-kernel attention (Theorem 1), so it replaces a softmax layer without architectural surgery. Training involves no sampling, no relaxation such as Gumbel–softmax, no train/inference distribution gap, and no sampling variance in the loss.

• Disagreement carries the signal, evidence swings with sparsity. In the reported experiments, disagreement carries the ranking signal, while the evidence factor swings from uninformative on dense data to strongly informative on sparse data. The paper states that evidence on its own ignores values entirely — a query attending to a hundred identical values reports the same uncertainty as one attending to a hundred conflicting ones, though its true variance is zero — which is why the disagreement factor is not optional.

• The shape of φ(Λ) is not monotone. φ(Λ) vanishes at Λ = 0, rises to a maximum of about 0.52 near Λ ≈ 1.5, and decays as 1/Λ beyond. The reading "less evidence, larger σ̂" therefore applies for Λ_q ≳ 2; below that, the Z_q = 0 fallback dominates. The paper reports that every experiment operates far above this region: mean evidence ranges from 62 to 632, and on the PhysioNet interpolation test set the smallest evidence over all 65,976 queries is 28.

• Accuracy cost of the operator swap is reported as small. On the t-PatchGNN benchmark the operator swap costs at most 5.6% accuracy against a matched control and nothing on the sparsest dataset, and a transplant into the benchmark's own model preserves its accuracy. On the sparsest dataset, the backbone attains a lower MSE than every published value with either decode layer, which the authors credit to the backbone family rather than to the operator. Table 1 is reported to show the Lévy model as more accurate than mTAND on all three datasets.

• The free signal improves on 20-pass MC dropout. Across matched five-seed suites, the free disagreement signal improves on the sparsification error (AUSE) of 20-pass MC dropout in every seed of a dense and a sparse benchmark, and attains the best such error of any training-free estimator measured on the dense one.

• Calibration and interval construction. The combined σ̂ scales a validation-calibrated Gaussian whose zero-sample CRPS improves on a fifty-draw sampler. A split-conformal wrapper on top of it reaches nominal coverage at every level.

• One-pass triage. A single pass ranks an unseen cohort of 3,383 patients by trust in 1.4 seconds.

• Costs. The operator shares softmax attention's O(mnd) kernel cost and adds three grid terms: O(nLd) to build the value field, O(mnL) to assemble cell intensities, and O(mLd) for the output reduction. Because L is constant in n, the total remains O(mnd) asymptotically. Drawing K samples redraws only the length-L count vector, costing O(KmL) to redraw and O(KmLd) to re-average, plus K passes of the small decode head rather than K full network passes.

• Mollification bias bound. The gap between the mean path and discrete cosine-kernel attention decays like n e^{-r̃_min²/32}, so minimum key separation must grow like sqrt(log n) bandwidths for it to bite. The paper states its operating points do not enter this regime, since bandwidths stay wide.

• TNP-D comparison. The deterministic-path variant TNP-D is run under the authors' own protocol; the paper reports where it wins (error ranking) and where it loses: accuracy, and the predictive distribution following from it.

• Not reported in the available excerpt: the specific numeric values behind several tables (Table 1 accuracies, the AUSE and Spearman figures per suite), the results of the grid-resolution sweep in §5.7, the Poisson-sampled training ablation numbers, and the per-dataset identity of the dense and sparse benchmarks among t-PatchGNN, PhysioNet and USHCN.

Methodology in Plain English

The authors recast the cross-attention layer as a random process rather than a fixed weighted average.

1. Positions. Each observation token gets a position in a two-dimensional index space: one axis is its normalised timestamp (given by the data), the other is a learned "channel coordinate" produced by a small affine map per head, which spreads variables observed at the same instant. This is the only learned part of the position scheme.

2. A beam of compatibility. Query–key compatibility uses a scaled cosine kernel, deliberately not row-normalised, so its absolute scale stays meaningful. Each key spreads its compatibility mass over the index space through a Gaussian mollifier normalised to unit mass, so no mass is lost to boundaries. Summing these gives an intensity function over the index space; the intensity's total is the evidence Λ_q, which equals the scaled partition function of the kernel.

3. Scattering atoms. That intensity drives a Poisson random measure: atoms fall at random points, concentrating where the intensity is high and never where it is zero. The total atom count is itself Poisson with mean Λ_q.

4. Reading values anywhere. Because atoms do not land on key positions, the observed values are interpolated into a continuous field using partition-of-unity RBF interpolation (a Nadaraya–Watson average), so the field is everywhere a convex combination of the observed values. The output is the sample mean of that field at the atoms, with the deterministic mean substituted when no atoms are drawn.

5. Discretisation. In practice the integrals become sums over a fixed grid of cells with centres, with resolution tied to the bandwidths. The default bandwidths are (1/16, 1/8), giving a 16 × 8 grid (128 cells by default). The paper notes the discretisation leaves the evidence exact and only introduces the mollification gap that is bounded in the theory.

6. Training through the mean, not the draw. Rather than differentiating through a random draw, the authors train with the deterministic expectation of the output. Where a sample would be used, they substitute its mean. The mean identity guarantees this substitution is exact at the layer, so gradients are exact and the loss is not noisy.

7. Uncertainty as a read-out. The same pass computes two grid reductions: the total intensity (evidence) and the value spread under the normalised intensity (disagreement). A scalar function of the evidence turns their product into the root-mean-square deviation σ̂.

8. Error bars on demand. When a distribution is needed, the expensive parts (encoder, kernel, beam, value field) are computed once and only the count vector is redrawn K times, so K samples cost one encoder pass plus K decode-step re-runs rather than K full passes.

9. Empirical protocol. A small transformer of about 0.7M parameters is trained from scratch per dataset: three softmax self-attention blocks encode tokens; a single cross-attention decode layer is the Lévy layer. A matched control swaps that one layer for softmax cross-attention under the same projections; the two models differ by 516 parameters (the learned channel map). Training uses masked MSE on hidden targets, Adam with early stopping on validation MSE at the benchmark's patience; the interpolation study uses AdamW for a fixed sixty epochs. Uncertainty signals are scored by Spearman correlation against absolute error and by AUSE, which measures how fast error leaves when the least-trusted queries are removed against an oracle that discards by true error. AUSE is the primary ranking metric, chosen because the decision the interface is built for is triage. Signals are compared same-backbone, so post-hoc baselines and the authors' model each rank their own deterministic errors.

Why This Matters

Impact on research. The paper reframes uncertainty quantification as a property of the attention mechanism rather than an add-on. Instead of training a second model, running repeated passes, drawing roughly 100 diffusion samples per query, or fitting a variance head (which the paper reports costs accuracy, §5.4), the uncertainty is a closed-form second moment derived from the operator the model already computes. The authors present Theorem 2 as an identity rather than a bound or a fit, tying the signal to the output distribution by derivation rather than by training. It also offers a conceptual repair: softmax attention normalises away the partition function and never surfaces the spread of attended values, whereas the Poisson construction keeps both, and the paper's experiments show those two quantities capture different failure modes — missing mass versus conflicting values.

Real-world applications:

• Clinical monitoring. The paper's motivating example is a vital-sign estimate interpolated inside a dense observation window versus one extrapolated across a six-hour gap. Per-query trust scores let alarm thresholds and downstream consumers distinguish the two, and the reported one-pass screening of 3,383 unseen patient records in 1.4 seconds suggests cohort-level triage is practical.
• Active sensing and compute triage. Because the interface ranks queries by how far each answer should be trusted, it can direct where to gather more observations or spend more computation.
• Sensor networks and climate/station data. The paper names climate stations and asynchronous sensors as IMTS domains; sparse regimes are where the evidence factor is reported to become strongly informative.
• Downstream decision systems. Calibrated Gaussian intervals and split-conformal wrappers reaching nominal coverage at every level give consumers of continuous-time predictions intervals with coverage guarantees rather than bare point estimates.

Industry relevance. The operator is designed as a drop-in replacement for a softmax cross-attention layer: same asymptotics (O(mnd)), no extra parameters beyond the channel map, no loss change, no extra passes at inference when only a point prediction is wanted. The reported transplant into the t-PatchGNN benchmark's own model, preserving that model's accuracy within its own seed spread, is the kind of result that matters for adoption into existing IMTS pipelines. The on-demand error-bar mode also fits deployment budgets: K draws cost one encoder pass plus K redraws of a length-L count vector rather than K full network passes, although the paper notes their own implementation re-runs the whole network per draw.

Future Directions

• Closing the theory-to-prediction gap. The identities certify the sampled operator around the trained prediction exactly, but the paper states explicitly that transfer to predictive error is a measurement, not a theorem. A formal account of when the disagreement and evidence factors track true error remains open.
• Making the evidence factor informative on dense data. The paper reports evidence swinging from uninformative on dense data to strongly informative on sparse data, which suggests the evidence term's role depends on regime and is not yet characterised beyond that observation.
• Choosing grid resolution and bandwidths systematically. Discretisation is a modelling choice controlled by ε_t and ε_v (defaults 1/16 and 1/8, giving a 16 × 8 grid), with the mollification bias bounded by a Gaussian tail in the minimum key separation. The paper says its operating points stay in the wide-bandwidth regime; how to select resolution in general is left to the sweep reported in §5.7, whose results are not available in the excerpt.
• Cheaper sampling implementations. The paper notes its implementation re-runs the whole network per draw even though the design only requires redrawing the count vector plus the small decode head; realising the theoretical cost of the K-draw mode is an engineering direction.
• Comparison against the broader probabilistic-IMTS family. Only the most directly comparable representative, TNP-D, is run under the authors' protocol; the paper frames the comparison against GP adapters, neural processes, GRU-ODE-Bayes and evidential deep learning largely in terms of provenance rather than head-to-head measurement.

Target Audience

Researchers and practitioners working on attention mechanisms, uncertainty quantification, and deep learning for irregularly-sampled or continuous-time time series — particularly those who need per-query reliability from a model that already makes continuous-time predictions. The paper will also interest readers working on stochastic-process formulations of neural network components, since the construction is a Poisson random measure with closed-form moments rather than a trained uncertainty head. It requires comfort with measure-theoretic stochastic processes, attention internals, and calibration and conformal prediction metrics such as AUSE, Spearman correlation, CRPS and coverage; readers without that background will find the theory section dense, though the four-step construction and the flashlight intuition are stated accessibly. Practitioners concerned with the operational cost of ensembles, MC dropout or diffusion-based imputers are the most immediate beneficiaries, given the paper's central claim that the uncertainty is free given the pass that already ran.

Authors’ abstract

Deep models for irregularly-sampled time series answer queries at arbitrary continuous timestamps, yet report nothing about how far each answer should be trusted. We show the attention layer itself can close that gap: with the right stochastic formulation, the pass that makes each prediction also reports, in closed form and at no extra cost, how far it should be trusted. We introduce Lévy Attention, a cross-attention operator whose output is a stochastic integral against an inhomogeneous Poisson random measure: query-key compatibilities assemble an intensity over a continuous (time x channel) index space, the measure scatters atoms under it, and the output averages an interpolated value field at those atoms. In expectation it reduces to a mollified cosine-kernel attention, so it replaces a softmax layer and trains with exact gradients. What softmax discards, the Poisson construction preserves in closed form: the evidence $Λ_q$ (total compatibility mass) and the disagreement $\mathrm{tr}\,Σ_V(q)$ (value spread). An exact variance identity makes their combination $\hatσ(q)=\sqrt{\mathrm{tr}\,Σ_V(q)\,\varphi(Λ_q)}$ the root-mean-square deviation of the sampled operator, emitted by the deterministic pass with no trained head. Empirically, disagreement carries the signal, while the evidence factor swings from uninformative on dense data to strongly informative on sparse. On t-PatchGNN the operator swap costs at most 5.6% accuracy against a matched control and nothing on the sparsest dataset. The free disagreement signal improves on 20-pass MC dropout across matched five-seed suites, and $\hatσ$ scales a calibrated Gaussian whose zero-sample CRPS beats a fifty-draw sampler; a split-conformal wrapper reaches nominal coverage at every level, and one pass ranks 3,383 unseen patients by trust in 1.4 seconds.