Benchmark Datasets for Lead-Lag Forecasting on Social Platforms

AI summary

Overview

Research area: Time-series forecasting, machine learning benchmarks, computational social science (social/scholarly platform dynamics)

Technical level: Intermediate — assumes familiarity with supervised learning, classification/regression metrics, and standard forecasting conventions, but the core idea is explained concretely.

Scope: The paper defines Lead-Lag Forecasting (LLF) as a new prediction paradigm and releases two large-scale datasets (arXiv and GitHub) to benchmark it.

What This Paper Is About

Many digital platforms record early signals of attention — paper downloads, GitHub pushes and stars, video views — that appear long before the outcomes people actually care about, such as citations, forks, or sales. This paper formalizes the problem of predicting a delayed "lag" outcome from an early "lead" activity channel as Lead-Lag Forecasting, and builds the datasets needed to study it systematically. The goal is to give the research community a standard, unbiased testbed where models must transfer information across channels and generalize to entirely new entities.

Key Contributions

1. Formalization of Lead-Lag Forecasting (LLF). The authors define LLF as predicting a downstream outcome derived from a lag channel at a long horizon H, given only an early window of lead (and optionally early lag) observations up to cutoff τ, with a deliberate gap between τ and H. This distinguishes LLF from rolling forecasting and temporal point processes.

2. Two new benchmark datasets at scale. arXiv (accesses → citations for ~2.3M papers, from 4.87B access events and 2.66B citation pairs) and GitHub (pushes/stars → forks for ~3M repositories, drawn from GH Archive and Ecosyste.ms metadata). Both span a 5-year horizon and include the full impact spectrum without survivorship bias.

3. Documented curation and cleaning pipeline. The paper details data sources, joins, anonymization, subject-area/package distributions, and train/validation/labeled-test/unlabeled-test/extra splits, including handling of the arXiv weekly-to-daily logging transition in 2013.

4. Empirical validation and baselines. Statistical analysis confirms lead-lag dynamics, and the authors benchmark classification (logistic regression, Time-MoE embeddings + LR, InceptionTime) and regression (MEY, linear regression, KNN, MLP, Transformer, Time-MoE KNN) baselines across 30-, 100-, and 365-day lookbacks.

Main Findings

• Early usage signals genuinely predict long-term impact. On arXiv, a single cumulative access feature reaches AUC 0.80 at day 30 and 0.86 at day 365 for identifying papers with ≥50 citations. On GitHub, analogous patterns hold for repositories with ≥10 forks, and every method beats stratified-random and all-positive baselines.

• Cross-channel prediction beats same-channel prediction at short horizons. Predicting citations from accesses outperforms predicting citations from early citations up to roughly day 100. The same holds for predicting forks from pushes/stars versus from early forks.

• The lead signal arrives well before the lag signal. On arXiv, the first access consistently precedes the first citation. On GitHub, the median repository receives its first star within 10–30 days but its first fork only after 100–200 days, indicating a progression from lightweight to committed engagement.

• More channels and longer lookbacks improve accuracy. Combining pushes + stars, or citations + accesses, outperforms single-channel inputs across all models, and performance rises monotonically as the observation window grows from 30 to 365 days.

• Deep features help for classification, but foundation-model embeddings do not clearly help regression. Time-MoE embeddings generally improve classification over raw features. In regression, KNN with Time-MoE embeddings is comparable to plain KNN, and the authors find no compelling evidence that the time-series foundation model adds substantial value there.

• Model behavior diverges by metric and horizon. InceptionTime achieves the strongest AUC at long horizons but has low F1 at short horizons, likely due to miscalibration. KNN is strong on MAE (especially on GitHub, where many repositories have zero forks) but underperforms deep learning methods on MAPE for high-impact instances.

• A floor effect exists in both domains. Some papers and repositories receive substantial early attention but very few citations or forks, meaning early popularity is informative but not deterministic.

Methodology in Plain English

The authors start by naming a pattern that practitioners already exploit informally: early activity predicts later outcomes, but with a delay. They write this down formally — observe each entity only up to a cutoff τ (30, 100, or 365 days), then predict something derived from the lag channel at 5 years (1825 days). Nothing between τ and the horizon is visible to the model.

To make this testable, they assemble two datasets through record linkage. For arXiv, they join access logs (which record who downloaded what, anonymized) with the Semantic Scholar citation graph, approximating citation dates by the citing paper's publication date. For GitHub, they join GH Archive event streams with package metadata from Ecosyste.ms, restricting to package repositories so that events are dense enough to be useful. Both joins keep low-, medium-, and high-impact entities, avoiding the bias that comes from only studying successes.

Crucially, they split entities — not time steps — into train, validation, and test sets. Test entities are ones the model has never seen, so success requires learning transferable patterns across many series rather than memorizing one series' trajectory.

Finally, they verify that the problem is well-posed: correlation plots show early lead signals track 5-year outcomes, and classification/regression experiments confirm that modest amounts of early data already carry real predictive signal, well above random.

Why This Matters

Impact on research: Prior forecasting benchmarks (traffic, weather, electricity, taxi demand) mostly ask a model to continue the same channel it observed. LLF asks for something structurally harder — cross-channel transfer plus generalization to unseen entities — and this paper supplies the first large-scale public testbed for it. It also opens two new application domains, scholarly communication and open-source software, to the time-series community.

Real-world applications:

• Research funding and evaluation: Funders and universities could use early download or citation signals to flag emerging work for support or review, though the authors caution against causal interpretation.

• Open-source sustainability: Maintainers and foundations could identify repositories gaining committed contributors early, guiding where to direct sponsorship, security review, or documentation effort.

• Product and content launch strategy: Publishers, streaming services, and e-commerce platforms face the same lead-lag structure (views → purchases, streams → ticket sales) and could reuse the modeling framework.

• Early-warning and curation systems: Recommendation and editorial systems could surface high-potential items sooner, using only the first month of interaction data.

Industry relevance: The LLF formulation matches how commercial platforms already operate — internal logs hold rich early signals, and the business question is almost always "which of these will matter later?" The paper's findings about cross-channel superiority and multi-channel inputs translate directly to production settings, and its metrics (especially MAPE restricted to high-impact items) reflect how businesses actually evaluate forecasting systems.

Future Directions

• Richer features beyond aggregate counts. The datasets omit author lists, paper content, repository code, and individual-level behavior traces; combining these with lead-lag signals could substantially improve predictions.

• Better architectures for long-horizon cross-channel prediction. Current baselines are standard supervised models applied straightforwardly. The paper explicitly frames this as an opening for models designed for series generalization, cross-channel transfer, and horizons far beyond typical seasonal patterns.

• Transfer to noisier, faster-moving domains. The authors propose extending LLF to Wikipedia (page-views → edits), Spotify (streams → concert attendance), e-commerce (clicks → purchases), and LinkedIn (profile views → messages), noting that these datasets are proprietary and that the arXiv/GitHub cycles are unusually stable.

• Causal versus correlational questions. Everything here is predictive. A natural next step is asking whether interventions on early signals actually change long-term outcomes, or merely correlate with them — which also requires examining whether platforms change behavior once these predictions are deployed.

Target Audience

Time-series and forecasting researchers looking for benchmarks that go beyond same-channel, short-horizon prediction; machine learning practitioners working on popularity, conversion, or impact prediction; and computational social scientists studying scientific attention or open-source ecosystems. Readers interested in dataset design and benchmarking methodology — particularly unbiased sampling and entity-level splits — will also find the curation details valuable.

Authors’ abstract

Social and collaborative platforms emit multivariate time-series traces in which early interactions -- such as views, likes, or downloads -- are followed, sometimes months or years later, by higher impact like citations, sales, or reviews. We formalize this setting as Lead-Lag Forecasting (LLF): given an early usage channel (the lead), predict a correlated but temporally shifted outcome channel (the lag). Despite the ubiquity of such patterns, LLF has not been treated as a unified forecasting problem within the time-series community, largely due to the absence of standardised datasets. To anchor research in LLF, here we present two high-volume benchmark datasets: arXiv (accesses -> citations of 2.3M papers) and GitHub (pushes/stars -> forks of 3M repositories). Our datasets provide ideal testbeds for lead-lag forecasting, by capturing long-horizon dynamics across years, spanning the full spectrum of outcomes, and avoiding survivorship bias in sampling. We documented all technical details of data curation and cleaning, verified the presence of lead-lag dynamics through statistical and classification tests, and benchmarked parametric and non-parametric baselines for regression. Our study establishes LLF as a novel forecasting paradigm and lays an empirical foundation for its systematic exploration in social and usage data.