KITE: A Knowledge-Injected Temporal Encoder for Time-Series Event Forecasting

AI summary

Overview

Research area: Machine learning for multivariate time-series event forecasting — specifically self-supervised representation learning with injected domain knowledge (a JEPA-based backbone plus two switchable prior-injection mechanisms). Presented at the workshop "Foundation Models for Temporal Systems: From Forecasting to World Modeling."

Technical level: Intermediate. The paper assumes familiarity with self-supervised forecasting architectures (JEPA, attention, survival heads, AUROC-based evaluation), but its central idea — treating domain knowledge as a switchable input rather than an architectural change — is explained in accessible terms.

Scope: The paper proposes KITE (Knowledge-Injected Temporal Encoder), one architecture that routes heterogeneous domain knowledge into a self-supervised event forecaster through two mechanisms that are each the identity at their null, and evaluates it on rare-event forecasting benchmarks in-domain and out-of-domain.

What This Paper Is About

Forecasting rare, high-cost events in multivariate time-series systems (building management, water networks, air-quality monitors, power infrastructure) is hard because their precursors are seldom observed, leaving purely data-driven models little to learn from on the stratum that matters. Engineers who run these systems hold "semantic priors" — knowledge that precedes the data, such as a variable's identity, a bound, a periodicity, a conservation law, or a causal edge — and this information is absent from the scarce labels. The paper's goal is to inject such heterogeneous knowledge into a single self-supervised event forecaster in a way that is general across prior types, simple to apply, and honest about attribution.

Key Contributions

1. A taxonomy of twelve semantic-prior categories mapped to two injection mechanisms, organised by when the knowledge is knowable and whether it is per-variable or relational. Four categories are implemented and evaluated here (periodicity #5, conservation #10, a structural descriptor #1, and causal-graph attention bias #9); the rest chart the design space.

2. The KITE architecture, built on a horizon-conditioned event JEPA backbone with a permutation-invariant, cardinality-free encoder (no per-variable slot, RoPE-ordered patches, variables as an unordered set), so one model runs across systems with different variable counts.

3. Two switchable prior mechanisms, each the identity at its null: mechanism (I), a covariate mechanism routing per-variable and derived priors as synthetic variables along static (encoder), future-known (predictor), and past-only derived (covariate) paths; and mechanism (II), an attention-bias mechanism that adds a scalar-gated driver-adjacency to encoder attention logits.

4. An empirical study organised by a single principle — a prior helps when it supplies information the observed window does not already contain — with prior-versus-no-prior comparisons as clean ablations, including an account of where priors are inert.

Main Findings

• An informative prior beats no prior in short context. On the Occupancy silent-onset stratum under a short w = 360 context, a true-phase periodicity clock lifts the prior-free baseline from 0.762 to 0.866; a zeroed clock stays at 0.767 and a shuffled clock falls 0.346 below the true phase, so the gain is the phase itself (pooled h-AUROC +14%, p < 0.001).

• On BATADAL the prior-free null sits below chance. At w = 512 the conservation residual lifts it to 0.877 (p = 1.2 × 10⁻⁵, n = 5), and still helps at a 0.25 label fraction (+0.207); the pooled abstract figure is +47% at a 0.25 label fraction.

• Where the prior enters matters. The clock alone as a time-of-day sine scores 0.721; as ordinary encoder input channels (the way a generic covariate-capable forecaster takes it) 0.797; only at the predictor's horizon does it reach 0.866.

• A working prior transfers out-of-domain by substitution, not amplification. A domain-invariant prior lifts the forecasting stratum by +0.12 to +0.39 where it applies (+53% pooled), while a static per-variable prior does not (+0.006). The amplification prediction fails: OOD − ID = −0.002. A prior with no pretraining beats transfer with no prior in eight of ten cells. Zero-shot Chronos-2 (0.774) sits between KITE with the prior (0.922) and without it (0.601) on BATADAL → SYNTH-CON onset.

• Priors are inert when the window already carries the information. Static per-variable injections survive 0/19 tests under BH-FDR; the clock is null once the context spans a full period (full-context ETTm1 sweep: 0/3 cells survive FDR); the causal-graph bias is bounded to approximately 0.006 h-AUROC. The static-identity FiLM prior is absorbable by the next linear layer.

• Misaligned or non-anticipatory priors do not help. A wrong FFT-estimated period reverses the Occupancy win to −0.022. The conservation residual detects an ongoing violation (+0.741 on the BATADAL ongoing stratum) but does not forecast onset (BATADAL onset −0.053; GECCO onset −0.136), and on the ongoing stratum a zero-parameter threshold on the residual alone (0.921) outranks KITE's 0.873 — that win is the residual's. The pre-registered Weather negative control is null (Δ = −0.017, p = 0.69).

• The attention-bias mechanism shows no measurable gain. With β = 2 the share of cross-variable attention mass on graph edges rises from a 3.8% chance level to 19.1% on Weather and 1.6% to 8.7% on TEP, but across seven adjacency variants on Weather every paired h-AUROC difference falls within ±0.0013 against a seed standard deviation of 0.0038; closing every cross-variable path costs only −0.0011 (95% CI [−0.0063, +0.0040], n = 5). The authors report mechanism (II) as undetermined with a small upper bound rather than refuted, because the positive control is under-powered.

• External baselines. Against the external forecasters in Table 1, none of 20 arms beats KITE at BH-FDR 0.05. The authors state the claim is that the prior lifts a backbone above its own null at no extra training cost, not that KITE beats foundation models.

• Implementation choices matter. A scalar-gated form of the static fusion (out = x + tanh(g)·δ) costs accuracy on every flagship (Occupancy −0.055, p = 0.015; BATADAL −0.570, p < 0.001), because g ends training at magnitude 0.029 on Occupancy and 0.007 on BATADAL. Removing the per-variable slot and supplying the future-known covariate during Stage-1 pretraining each leave every flagship without a detectable change (all p ≥ 0.08); SIGReg is not adopted over VICReg (all p ≥ 0.08).

• Cardinality freedom, measured. Slot removal makes the encoder summary invariant to variable reordering to about 2 × 10⁻⁷ (over 20 permutations); with the slot present the same quantity is 10⁻² to 2 × 10⁻¹. One slot-free encoder runs unmodified at every variable count tested (C_test ∈ {2, …, 10} against C_train = 5, h-AUROC 0.822 to 0.908), whereas the slot encoder cannot be evaluated at any C ≠ C_train at all.

Methodology in Plain English

The authors start from an existing self-supervised forecasting backbone (a horizon-conditioned event JEPA) and change nothing structural about it. Instead, they add domain knowledge as an extra input, through two routes designed so that, at their null setting, the conditioned model computes exactly the same thing as the unconditioned one.

Route one treats knowledge as a synthetic variable, routed by when it becomes knowable. A static fact (for example, which channel is the high-voltage load) enters the encoder as a zero-initialized additive term, out = x + W_δ δ, during Stage-1 pretraining. A future-known signal (a periodicity clock evaluated at t + Δt) skips the encoder and is concatenated into the predictor's input, so it anticipates rather than detects. A past-only derived law (a conservation residual, Σin − Σout − dL/dt) enters as an ordinary past-known covariate.

Route two biases the encoder's attention logits with a causal graph's driver-adjacency, logit(q_a, k_b) += β·A[a,b], which vanishes identically when β = 0.

Because a prior enters as an input rather than as an architectural change, one architecture serves every prior type with no per-dataset architecture search, and each prior-versus-no-prior comparison is a clean ablation. Crucially, every reported null is a separately trained model with the prior zeroed or absent — never the prior model with its input removed at test time — because training with a prior also adapts the rest of the network. Evaluation uses the rank-based horizon-AUROC (h-AUROC) over 5 to 10 seeds against zeroed- and shuffled-prior controls, and the estimand differs by row (stratum-restricted or overall), so it is comparable only within a row.

Why This Matters

Impact on research. The paper reframes prior injection from an architectural commitment into a switchable conditioning route, which makes prior-versus-no-prior comparisons separable from a pathway's raw capacity. It also supplies a negative result that is useful in itself: a large class of static per-variable priors is inert (absorbable into the following linear map), and a causal-graph attention bias is bounded to roughly 0.006 h-AUROC in these settings.

• Building management: injecting a daily occupancy clock (period 1440 minutes) to anticipate silent onsets rather than detect them from a short window.
• Water distribution networks: encoding a tank mass-balance law (Σin − Σout − dL/dt) as a residual covariate to flag ongoing conservation violations such as leaks.
• Air-quality and environmental monitoring: encoding bounds and thresholds (e.g. particulate warnings), and calendar or unit-based knowledge for monitors where labels are scarce.
• Power and industrial infrastructure: carrying variable identity (which channel is the high-voltage load) and driver relations explicitly rather than leaving them to be learned, and running one pretrained encoder across plants with different numbers of sensors.

Industry relevance. The cardinality-free encoder means one trained model can be deployed across sites with different variable counts, and the fact that the prior is delivered at Stage 2 for future-known signals means a new clock or calendar prior requires no new Stage-1 pretraining run. The paper positions a switchable prior as a substrate for pretrained, cross-domain time-series foundation models that carry their governing structure explicitly.

Future Directions

1. Scaling toward cross-domain, pretrained time-series foundation world models that carry their governing structure explicitly, which the authors name as the natural substrate for the switchable prior.
2. Resolving the undetermined attention-bias mechanism. The graph bias is bounded but not refuted; the paper's own positive control is under-powered (a linear probe with all 45 pairwise correlations reaches only 0.559 against a variable-blind floor of 0.454, with KITE at 0.478–0.519), so a stronger relational benchmark is needed.
3. Making the predictor's future-known input cardinality-compatible across datasets with differing covariate counts — the paper notes this design point remains open.
4. Expanding coverage of the twelve-category taxonomy and the external-validity base. Several categories remain conceptual scaffolds, mechanism (II) is bounded but undetermined, and the strongest transfer target is synthetic because a second real mass-balance network (W

Authors’ abstract

Forecasting rare, high-cost events in multivariate time-series systems is hard because their precursors are seldom observed, so a purely data-driven model has little to learn from. The engineers who run these systems, however, hold knowledge that precedes such events, information absent from the scarce labels. We call this a semantic prior and inject it into a self-supervised event forecaster through two switchable mechanisms, a covariate mechanism and an attention-bias mechanism, each the identity at its null, so one architecture serves every prior type with no per-dataset architecture search. A prior helps when it supplies information the observed window does not already contain: in a short context an informative prior beats no prior in-domain (pooled h-AUROC over the no-prior baseline: a periodicity clock +14%, p&lt;0.001; a conservation residual +47% at a 0.25 label fraction), and a working prior transfers out-of-domain (+53%), substituting for feature transfer, while priors the window already recovers stay inert and the attention-bias mechanism shows no measurable gain. KITE is a switchable, attributable route to contextual conditioning, and a substrate for scaling toward cross-domain, pretrained time-series foundation world models that carry their governing structure explicitly.