TSB-SEG: A Systematic Time-Series Segmentation Benchmark

AI summary

Overview

Research area: Time-series segmentation, specifically the two related tasks of Change Point Detection (CPD) and State Detection (SD), evaluated through a large-scale benchmarking study.

Technical level: Intermediate. The paper is a benchmark and empirical evaluation study rather than a new algorithm, but it introduces a taxonomy, a two-pool experimental design, and statistical ranking procedures that reward some familiarity with time-series analysis.

Scope (one sentence): The paper introduces TSB-SEG, a unified benchmark of 592 labeled time series from 8 heterogeneous datasets and 26 algorithms spanning five decades, jointly evaluating whether five decades of segmentation progress actually deliver a dominant method, whether CPD accuracy can stand in for SD accuracy, and what practical guidance follows from data properties, runtime and tuning budgets.

What This Paper Is About

Time-series segmentation splits a signal into homogeneous segments separated by change points. The statistical community studies this as Change Point Detection (finding boundaries), while the machine-learning and data-mining communities study it as State Detection (also labeling recurring states), and the two literatures have rarely been compared on shared data. The paper builds a single reproducible benchmark that evaluates both tasks together across diverse univariate and multivariate datasets, then uses it to answer whether any method dominates, whether CPD accuracy is a reliable proxy for SD accuracy, and which practical guidelines hold for real applications.

Key Contributions

1. A taxonomy of 26 CPD and SD algorithms organized along two axes: how the segmentation is searched (profile, bottom-up, top-down, convex relaxation, optimal, Bayesian, state-sequence, segment-then-group, window clustering) and what must remain homogeneous within a segment (model fit or likelihood, distribution comparisons without a model, subsequence shapes, or a learned neural representation).

2. TSB-SEG, the first benchmark to jointly evaluate CPD and SD, built on 592 labeled univariate and multivariate series from 8 datasets plus PAMAP2 as a long-series scalability case study, using a two-pool design: a main pool of 21 tractable algorithms (15 CPD, 6 SD) that drives the aggregate ranking, and an extended-scope pool of 5 computationally intensive or restricted methods (BOCD, EAgglo, TS-CP², tGLAD, Hidalgo).

3. Eight complementary CPD and SD metrics, including soft and symmetric variants of standard CPD measures (Gaussian F1, Bidirectional Covering) and the recent WARI and SMS for SD, with SMS decomposing its penalty into Delay, Transition, Isolation and Missing error types.

4. A large-scale evaluation with statistical testing (two-sided Wilcoxon signed-rank tests with Holm correction at α = 0.1) plus the open-source tsseg library and TSB-SEG pipelines for full reproducibility. 13 of the 26 algorithms were absent from existing toolkits (aeon, sktime, ruptures, and the R package changepoint) and were re-implemented or adapted.

Main Findings

• No method dominates (RQ1, CPD): In Non-Guided mode, the penalized detectors PELT, BottomUp and BinSeg form a statistically tied top clique (mean ranks 5.77 to 5.94), followed by Window and ClaSP. ClaSP has the highest mean tuned BiC (0.70), level with the penalized detectors (0.68 to 0.69), but the two signals disagree: ClaSP is the strict best on 18% of series while ranking in the bottom half on 39% of them.

• Tuning is decisive and unevenly costly: TIRE, ChangeFinder, BottomUp, Window, PELT and BinSeg span tuning gaps of roughly 0.35 to 0.50 in BiC, so a poor configuration can drop them below the single-change-point reference AMOC; ClaSP stays well above AMOC even in its worst configuration (Δ ≈ 0.17).

• Knowing the number of events helps, but not equally: Providing the ground-truth n_CP raises mean tuned BiC by +0.14 to +0.17 for Time2State, E2USD and GGS, by +0.05 to +0.10 for nine other methods, and by at most +0.03 for HDP-HSMM, ChangeFinder and iCID; PELT, AMOC and AutoPlait cannot use it. In Guided mode, DynP takes the lead (tuned BiC 0.78, mean rank 6.71), with TICC at 0.64 and ESPRESSO at 0.57.

• SD has a clear leader: CLaP attains the lowest mean rank (2.32) and the highest mean tuned SMS (0.63) with a narrow tuning gap, ahead of Time2State and E2USD (0.50 to 0.51) and HDP-HSMM and AutoPlait (0.48). With the number of states provided, CLaP leads alone (mean rank 2.63, tuned SMS 0.71). CLaP labels the segments found by ClaSP, a CPD method, so progress in CPD carries over to SD up to the labeling stage.

• Leaders leave headroom: Picking the best method per series would raise the mean score by 0.11 to 0.16 over the single best method in every task and mode.

• CPD is a useful but imperfect proxy for SD (RQ2): Analyzing the residual r = SMS − BiC across all SD-method runs, about 34% satisfy |r| ≤ 0.05 and a roughly 95% envelope holds r in [−0.27, +0.36]. The proxy is tightest when states do not recur (|r| ≤ 0.05 for 41% of runs versus 23% when they do) and weakest on heavily recurring-state data such as PUMP (median r = −0.18).

• The two residual tails have distinct causes: Below r < −0.27 the SMS penalty splits between Missing (≈40%) and Delay (≈36%) — boundaries are fine but labels are wrong (a clustering bottleneck). Above r > +0.36 the penalty is dominated by Isolation (≈88%) — heavy over-segmentation crashes BiC while SMS only mildly penalizes fragments.

• Fast methods are also accurate for CPD (RQ3): PELT reaches BiC 0.69 in 0.65 s, BinSeg and BottomUp are within 0.01 at 2 to 4 s, and only ClaSP scores higher (0.70 at roughly 18 s). With n_CP given, DynP, BinSeg and BottomUp lead in 1.6 to 3 s. All methods except TICC have a median runtime below 60 s, but the 90th percentile exceeds 500 s for ClaSP, CLaP, iCID, ESPRESSO and TICC.

• Most tuning reduces to one knob: Time2State, E2USD and iCID are dominated by a single monotone window-size parameter; penalty-based methods peak near 8× the BIC penalty (1× for Window); only TICC shows genuine multi-axis sensitivity.

• Data characteristics matter: Penalized detectors are nearly insensitive to n, d, n_CP and n_S (|ρ| ≤ 0.17), while ClaSP excels on univariate series with few events but degrades with many events and on multivariate inputs (ρ ≈ −0.5 in both cases). Time2State and E2USD show the opposite pattern on CPD (ρ ≈ 0.5 with event counts) and their SMS barely depends on d or n_CP (|ρ| ≤ 0.15, against ≈ −0.55 for CLaP), making them the most robust SD picks for complex multivariate series without guidance.

• Long series change the ranking: On PAMAP2 (up to 447k samples), only 17 of 26 algorithms process all 9 series within the time budget; ClaSP, CLaP and tGLAD finish 1 to 5 of them and six methods finish none. BinSeg, PELT and BottomUp reach tuned BiC 0.68 to 0.72 Non-Guided (0.71 to 0.73 Guided with DynP), on par with IGTS (0.70) and above GGS (0.61), while Time2State and E2USD remain competitive for SD (tuned SMS 0.53 to 0.56).

Methodology in Plain English

The authors first define the problem carefully: an underlying process moves between discrete states, a change point is where the state changes, and a segment is the stretch between two change points. They then collect existing datasets rather than inventing new ones, keeping the original authors' labels, and report their provenance and the uncertainty around hand-placed boundaries. The collection deliberately mixes univariate and multivariate series from human activity recognition, industrial monitoring, healthcare, robotics and general-purpose archives, with series lengths from 240 to 56,251 samples, 0 to 78 change points, and 1 to 43 distinct states.

Not every algorithm can run on every series, so the evaluation uses two pools. The main pool holds 21 methods that run everywhere and drives the ranking; the extended-scope pool holds 5 methods that are at least quadratic, too expensive to train, or multivariate-only. The authors check that the extended-scope methods never beat the main pool on the cells they share, so the split does not hide a state-of-the-art outlier.

Each algorithm runs in two modes. In Default mode it uses the authors' recommended configuration. In Parameter-search mode the authors sweep a small grid of the most influential parameters, calibrated identically across algorithms, and record the best and worst dataset-level average — the gap between them is the "tuning gap" that measures sensitivity. Separately, every algorithm runs with and without knowledge of the number of events (Non-Guided versus Guided), since estimating that number is the hardest part of the problem.

Scoring uses eight metrics split by task. For CPD they pair the standard F1 score (1% margin) and Covering score with a Gaussian F1 and a symmetric Bidirectional Covering, because the standard versions are discontinuous and insensitive to over-segmentation. For SD they pair ARI and AMI, which ignore time order, with WARI and the State Matching Score, whose penalty decomposes into Delay, Transition, Isolation and Missing errors. Rankings are compared with two-sided Wilcoxon signed-rank tests with Holm correction, and all code is released in the tsseg library with a unified fit/predict interface.

Why This Matters

Impact on research: The paper makes the case that CPD and SD have been evaluated in separate silos with narrow method coverage, small or homogeneous datasets, and no shared protocol. By scoring SD methods on CPD (through the change points their labels induce) and by releasing every algorithm and metric through one library, it makes cross-task and cross-community comparisons possible, and it converts vague advice about hyperparameter tuning into measured tuning gaps per method.

Real-world applications:

• Human activity recognition from wearable recordings, where a walking-running-walking sequence requires SD to recognize the first and last segments as the same activity, not just to find two boundaries.
• Industrial condition monitoring, including hydraulic-pump end-of-line test cycles that revisit the same operating modes dozens of times (up to 43 states per cycle in PUMP).
• Health assessment and patient monitoring, where a patient alternates between a few physiological regimes and the number of segments is unknown.
• Machine and sensor diagnostics on long recordings, where an O(n²) method can become intractable before the series reaches 10⁴ to 10⁶ samples.

Industry relevance: The runtime-accuracy results give directly actionable defaults: the fast penalized detectors (PELT, BinSeg, BottomUp, plus DynP when the number of change points is known) are the safest CPD choice and scale to long series, ClaSP is preferable on univariate series with few events, and CLaP leads for SD on simple series while Time2State and E2USD are better on complex multivariate and long ones. The finding that a coarse sweep of a single window, penalty or threshold parameter captures most of the tuning gain makes deployment cheaper than the full grids suggest, and the SMS error breakdown tells practitioners whether residual error is a benign boundary-delay problem or a structural over-fragmentation problem.

Future Directions

• Extend to highly multivariate and streaming settings. The authors state that the benchmark covers univariate and moderately multivariate series (d in [4, 10]) and focuses on offline segmentation, leaving highly multivariate and online or streaming scenarios unaddressed.

• Handle gradual and mixed transitions. The benchmark treats change points and states as discrete events, whereas real transitions are often gradual and states can be mixed or hierarchical, which the soft metrics only partly account for.

• Adapt time-series foundation models to segmentation, which the authors name as a future direction.

• Automate method selection from data characteristics, for instance through meta-learning, so that the per-dataset tuning the results show to be decisive can be replaced or reduced.

Target Audience

Researchers and practitioners working on time-series segmentation, change point detection, or regime-shift analysis who need to choose a method and want evidence about accuracy, runtime, tuning cost and robustness to data properties. It is also useful for benchmarking methodologists interested in the two-pool design, the soft and symmetric metric variants, and the statistical ranking protocol, and for engineers deploying monitoring or diagnostic pipelines on sensor data who want concrete defaults rather than a new algorithm. Readers without time-series background will find the taxonomy and the plain-language takeaways accessible, but the detailed rankings assume comfort with metrics such as F1, ARI, AMI, WARI and SMS.

Authors’ abstract

Time-series segmentation, studied as either Change Point Detection (CPD) or State Detection (SD), underpins a broad range of monitoring and diagnostic applications. However, the field suffers from a persistent divide between the statistical community (focusing on CPD) and the machine learning and data mining communities (focusing on SD). Consequently, the literature remains fragmented: evaluations typically cover narrow methodological families, datasets are often small or homogeneous, and CPD and SD are rarely evaluated together. We address these gaps with TSB-SEG, a unified benchmark comprising 592 univariate and multivariate time series from 8 heterogeneous datasets and surveying 26 algorithms that span five decades of progress. To ensure a fair comparison, we introduce a two-pool design: a scalable main pool of tractable detectors used for the aggregate ranking, and an extended-scope pool for computationally intensive methods. All methods are unified under tsseg, an open-source library to ensure full reproducibility. Our systematic evaluation yields three insights: (a) no single detector dominates across domains, and per-dataset hyperparameter tuning remains decisive for performance, (b) CPD accuracy is a useful, though imperfect, proxy for SD accuracy, and (c) data characteristics should guide method selection. We provide actionable guidance based on data properties, error tolerances, and tuning budgets, identify impactful hyperparameters, and characterize algorithm robustness to these choices.