TSExplorer: An interactive data annotation and exploration tool for time-series data

AI summary

Overview

Research area: Human-Computer Interaction, specifically interactive visualization and human-in-the-loop data annotation for time-series data.

Technical level: Intermediate. The tool itself is presented in accessible terms, but the underlying workflow assumes familiarity with high-dimensional feature representations and dimensionality-reduction methods such as t-SNE, PCA, and UMAP.

Scope: The paper presents TSExplorer, a cross-platform graphical user interface tool that lets users annotate and explore time-series datasets through multiple interactive 2D visualizations computed from high-dimensional feature representations.

What This Paper Is About

Modern time-series pipelines — for speech, video, physiological signals such as electroencephalography, sensor recordings, or multimodal combinations — increasingly rely on high-dimensional feature representations, whether hand-crafted (e.g., log-mel spectrograms) or learned embeddings (e.g., wav2vec 2.0). Yet annotation workflows tend to proceed sequentially, and analysis is usually based on static visualizations or aggregate statistics such as feature variances, class-wise averages, performance metrics, or correlation measures. TSExplorer addresses this gap by letting users inspect individual samples and their relationships inside the full feature space, interactively and across multiple feature representations.

Key Contributions

1. A general-purpose annotation and exploration GUI. TSExplorer visualizes an entire dataset as a 2D scatter plot where each point is one data sample (e.g., one utterance), and selecting a point surfaces the corresponding views of that sample.
2. Multiple complementary 2D visualizations (2DVs) and feature representations. Users can alternate between t-SNE, PCA, and UMAP projections, and switch between different high-dimensional feature representations both within and across modalities.
3. Flexible sample selection and labeling. Samples can be chosen by left-clicking, by right-clicking into an inspection queue, or algorithmically through a "next sample" button offering three strategies: Random, Ordered, and Farthest-first (based on the farthest-first traversal algorithm). Labels can be changed at any time via a drop-down menu or keyboard shortcuts.
4. A customizable, extensible, cross-platform tool. It runs on Windows, Linux, and macOS; the GUI layout is fully customizable in widget size, location, and type, with multiple instances of the same widget allowed, and new widgets, 2DV methods, and sample selection strategies can be added.

Main Findings

• Five widget types are provided: audio, video, scatter, waveform, and spectrogram. The paper's Figure 1 shows an example configuration using the RAVDESS dataset with a video widget, a spectrogram widget, and a scatter widget.
• The tool supports three dataset states: unlabeled (annotation and exploratory analysis, e.g., understanding dataset structure or identifying outliers), partially-labeled (incremental annotation with direct visual feedback on how labels distribute across the feature space), and fully-labeled (evaluating class separability or feature consistency, plus label revision for quality assurance or relabeling).
• Visual encoding is label-driven: sample colors in the scatter plot reflect their labels, with green used for unlabeled samples by default, and the currently selected sample highlighted with a large ring that is yellow by default. The scatter plot supports scroll-wheel zooming and dynamically adapts data point sizes.
• Implementation stack: Python, with PySide6 for the GUI and PyQtGraph for data visualization; audio and video playback are handled using the VLC Media Player.
• Reported computational cost: computing a t-SNE embedding for 100,000 samples (160-dimensional) requires approximately 1.8 GB of RAM and 8 minutes on a single CPU core at 3.9 GHz. Computed 2DVs are saved to avoid repeated computation, and pre-computing them is recommended for large datasets.
• Reported memory scaling: a speech dataset of 100,000 samples (mean utterance duration 1.5 s) with pre-loaded audio plus log-mel, MFCC, and F0 features and their corresponding 2DVs uses 5.1 GB of RAM. Keeping audio as individual files on disk and loading samples on demand reduces RAM usage below 0.4 GB. With no data loaded, TSExplorer idles at approximately 0.3 GB of RAM.
• Evaluation status: the paper states that TSExplorer has been previously described and evaluated in Vaaras et al. [6]; no new user study, quantitative evaluation results, or benchmark comparisons are reported in this paper itself.
• Disclosure: generative AI was used only in a limited role for programming support, such as syntax guidance and debugging.

Methodology in Plain English

The tool separates expensive computation from interactive use. First, high-dimensional features are computed offline from the raw time-series data. Then 2D visualizations are derived from those features, either pre-computed offline or computed within TSExplorer online. Finally, the time-series data, the high-dimensional features, and the 2D visualizations are supplied as inputs to TSExplorer, which the user then interacts with for annotation and exploration.

Inside the interface, the whole dataset appears as a 2D scatter plot. Clicking a point brings up views of that sample in other widgets. Because different projection methods (t-SNE, PCA, UMAP) emphasize different structure, and different feature representations capture different properties of the data, users can shuffle between these views to build up a picture of how the dataset is organized and where labels do or do not separate cleanly. Annotation then happens directly in that visual context, so the effect of each new label on the distribution of classes is visible immediately.

The design deliberately avoids prescribing a single fixed workflow. The layout, widgets, and settings are configurable, and the architecture is open to extension with new widgets, new 2D visualization methods, and new algorithmic sample selection strategies.

Why This Matters

Static summaries and sequential annotation hide the structure of high-dimensional data. By making that structure directly explorable and tying labels to it visually, TSExplorer supports earlier detection of outliers, better understanding of how feature representations differ, and more informed relabeling of existing datasets — all of which are prerequisites for trustworthy models trained on time-series data.

Real-world applications:

• Speech and audio corpora: annotating utterances and inspecting how proposed embeddings such as wav2vec 2.0 organize a dataset.
• Physiological signal research: exploring electroencephalography and other recordings where sample-level inspection is needed.
• Multimodal recordings: comparing representations across modalities such as audio and video, or sensor-based recordings.
• Data quality assurance: auditing and refining labels in already-labeled datasets before they are used for training or reporting.

Industry relevance: teams that maintain labeled time-series datasets — speech and voice products, wearable and sensor analytics, clinical monitoring, and video processing — face the same problems of annotation cost and label quality. A cross-platform tool with modest resource demands and offline feature computation gives them a way to inspect and correct data interactively rather than through aggregate metrics alone.

Future Directions

The paper does not include an explicit future work section. Its design and stated extensibility point to several natural next steps:

• Adding new 2D visualization methods and algorithmic sample selection strategies, which the architecture is explicitly designed to accommodate.
• Extending the widget set, since new widgets can be added and existing components modified in a straightforward manner.
• Broader, directly reported evaluation. The paper cites Vaaras et al. [6] for prior description and evaluation; independent user studies of annotation efficiency and label quality for TSExplorer itself are not reported here.
• Scaling to larger or more complex datasets, given that 2DV computation is described as the most demanding process and that memory usage scales with dataset size and feature configuration.

Target Audience

Researchers and practitioners who work with high-dimensional time-series datasets and need to annotate, audit, or understand them: speech and audio researchers, physiological signal and biomedical researchers, multimodal and sensor-data analysts, and machine-learning engineers responsible for dataset quality. It is also useful for HCI and visualization researchers interested in human-in-the-loop annotation tools. Readers looking for quantitative benchmarking of annotation performance will need to consult the referenced prior work by Vaaras et al. [6], since this paper reports the tool and its design and resource characteristics rather than new empirical results.

The tool is publicly available on GitHub at https://github.com/SPEECHCOG/TSExplorer. The authors are affiliated with the Signal Processing Research Centre at Tampere University, Finland, and the BABA Center, Department of Physiology, University of Helsinki, Finland.

Authors’ abstract

We present TSExplorer, a cross-platform tool for interactive annotation and exploration of time-series data. The tool enables users to inspect high-dimensional datasets through multiple complementary 2D visualizations derived from high-dimensional feature representations. TSExplorer is designed as a general-purpose research tool supporting a wide range of workflows, including exploratory data analysis, annotation of unlabeled or partially-labeled datasets, comparison of feature representations, and post-hoc inspection and refinement of existing labels with interactive visual feedback.