Time-Series and Sequential Data Engineering

A temporal dataset has several interacting clocks: when the event occurred, when it was observed, when a forecast is issued, and when the target becomes measurable. Cadence matters. Sensor readings may be irregular. Sales may be daily but corrected later. Forecasts may run hourly with several future horizons, and resampling can create values that never existed at decision time. Sequential examples also overlap. Two adjacent windows can share most of their history and outcomes, which makes random row splits especially misleading.

The scale at which this has to be got right is not hypothetical. One sepsis challenge required a prediction at every hourly time window across 40,336 patient records, released publicly from two hospital systems, with 22,761 records from three systems sequestered. That is more than 2.5 million hourly time windows and more than 15 million data points between them. That was PhysioNet's 2019 sepsis challenge. Every one of those windows is an example, and its history, its horizon and its label maturity had to be fixed in writing before a single model was fitted.

Data engineering must preserve order, revisions, gaps, and availability. The table should reconstruct what the forecasting or sequence system would have known at each origin.

Each example uses values that only became available after its forecast origin. The gap between what production saw and what the archive now holds is measurable, and one agency measures it on itself. The U.S. Bureau of Economic Analysis publishes reliability tables for GDP vintages. Over 1996-2024 the mean absolute revision to annualised real GDP growth was 0.5 percentage point from the advance estimate to the second, and 0.6 percentage point from the advance to the third. From the second estimate to the third it was 0.3 percentage point. Of the first of those the agency writes that “"Advance" estimates are released near the end of the first month following the end of the quarter and are based on source data that are incomplete or subject to further revision by the source agency.”

The same revisions have been measured on a different vintage archive. Lubik and Titcomb, in a Federal Reserve Bank of Richmond economic brief, put the mean advance-to-final revision on Philadelphia Fed vintages at 0.45 percentage point, with a standard deviation of 1.97 percentage points. Policymakers, they note, act “based on the data available, not necessarily knowing the final readings”. A backtest fed the final series hands the model exactly that advantage, every quarter.

Each boundary determines which observations and targets belong to one example. PhysioNet's 2019 sepsis challenge is worth reading less as a benchmark than as a specification, because it fixed those boundaries explicitly and in public.

History and origin: a prediction was required at every hourly time window, and PhysioNet states that “each row will represent a single hour's worth of data”. The origin therefore advances one row at a time. The observation history is everything above the current row.

Outcome horizon: the challenge paper puts it in one sentence. “In particular, we asked participants to predict sepsis at least 6 hours but no more than 12 hours before the onset of sepsis according to Sepsis-3 clinical criteria”, its nine authors write. The horizon is a stated interval, not a vague notion of earliness.

Label maturity: onset was not left to a chart note. It was defined by two clocks — t_suspicion, from antibiotics and cultures, and t_SOFA, from a two-point SOFA rise within 24 hours. The scoring window rewarded predictions between 12 hours before and 3 hours after t_sepsis.

That specification produced 40,336 public records from two hospital systems and 22,761 sequestered records from three, spanning “over 2.5 million hourly time windows and over 15 million data points”. Four of the five boundaries in the diagram are written into its rules. Known-future inputs are the one an intensive-care setting supplies least of, which is itself a fact about the domain rather than an oversight.

These representations serve different pipeline stages and should preserve the same temporal semantics. Two PhysioNet challenges show the first two shapes side by side, on the same kind of clinical stream.

The 2012 data arrives as irregular events. Its 36 time-series variables, PhysioNet notes, “may be recorded at regular intervals ranging from hourly to daily, or at irregular intervals as required”. That is the long event table in its natural state: flexible, honest about when nothing was measured, and useless to a model until it is pivoted.

The 2019 data arrives already regularised into a panel, one row per patient per hour. Windowing that panel for a sequence architecture is what turns 40,336 records into more than 2.5 million hourly time windows. That multiplication is the whole point of the third column. Adjacent windows repeat nearly all of their history, so the tensor has to carry source indices back to record and hour, or the split can no longer be controlled.

Generate examples by simulating the production schedule across history. The 2019 sepsis challenge is a worked instance of the first, second and fourth steps at once. The cadence is hourly. The origin is every hourly time window. The targets are defined by t_suspicion and t_SOFA rather than by whoever wrote the discharge summary. Maturity and horizon sit inside the metric itself: the scoring rule is a utility function that “rewards early sepsis predictions and penalizes late or missed predictions and false alarms”.

The steps that remain yours are the third and the fifth. Bounded features have to stop at the origin, even when the tooling makes it easy to reach past it. Chronological backtesting has to reproduce the retraining cadence rather than a single split. M4 is the reason to spend the effort there rather than on the model class. Sixty-one methods were scored on 100,000 series. Twelve of the 17 most accurate were combinations of mostly statistical approaches, and no pure machine-learning entry beat the combination benchmark.

Many time series are revised after they are first published, and the revision is not a rounding detail. On 9 September 2025 the U.S. Bureau of Labor Statistics announced a preliminary benchmark revision to March 2025 employment: “The preliminary estimate of the Current Employment Statistics (CES) national benchmark revision to total nonfarm employment for March 2025 is -911,000 (-0.6 percent)”. The final benchmark arrived with the release of January 2026 data, on 11 February 2026: “The March 2025 benchmarked, seasonally adjusted employment level for total nonfarm employment is 158,377,000”. That is a revision of 898,000 (-0.6 percent) against the previously published seasonally adjusted estimate, and of 862,000 (-0.5 percent) not seasonally adjusted, to 157,540,000. One month of employment, several published values, the last landing the better part of a year after the month itself.

Nor was this an aberration to be excluded as an outlier. A Federal Reserve Bank of Cleveland economic commentary by Pinheiro and Quinlan places the same revision at -0.54 percent of originally reported employment, “just marginally outside of the normal range” of roughly plus or minus 0.5 percent. A model trained on the final series and evaluated on the final series never meets the -911,000 version. That is the version every decision taken in the intervening months actually used.

Store publication or system time alongside valid time, so the pipeline can reconstruct vintage data. Evaluate both preliminary and final targets when they support different operational questions. Two public archives already do exactly this. The St. Louis Fed's ALFRED has been running since 2006: “ALFRED® allows you to retrieve each economic data release (vintage) that was available on a specific date in history.” The Federal Reserve Bank of Philadelphia's Real-Time Data Set for Macroeconomists is the same idea for macroeconomic aggregates, introduced by Croushore and Stark in 1999: “The data set consists of vintages, or snapshots, of the major macroeconomic data available at quarterly intervals in real time.”

A model may need to predict the first published value, the eventual final value, or both. The preliminary -911,000 and the final 898,000 are not one target measured twice. They are two targets, with different delays and different noise. Document whether training uses vintage snapshots, latest revised history, or simulated publication lags. Without that choice, backtest results are difficult to interpret.

The database standard has carried both clocks since December 2011. Kulkarni and Michels, writing in ACM SIGMOD Record, record how SQL:2011 separates them: “transaction time support is provided by system-versioned tables, which in turn contain the system-time period, and valid time support is provided by tables containing an application-time period”. The standard names the system-time period SYSTEM_TIME, and “users are allowed to define at most one application-time period and at most one system-time period per table”. The bookkeeping is not left to the application: “the system, rather than the user, maintains the start and end times of the periods of the rows”, and the design requires that “users be unable to modify the content of historical rows”.