Competence-Gated Pooling of Language Models and Priors for Event Forecasting

AI summary

Overview

Research area: Hybrid forecasting with large language models — specifically forecast combination, probabilistic calibration, and selective model use.

Technical level: Intermediate. The paper is readable for someone familiar with basic probability and forecasting metrics (Brier score, calibration), but the theoretical derivations and the cross-validated estimation pipeline require some statistical background.

Scope: The paper defines and measures relative competence — the marginal value a language model adds beyond an already-available market, crowd, or statistical forecast — and introduces a domain-conditioned "competence gate" that pools models with external forecasts while deferring when it shouldn't help.

What This Paper Is About

In deployed forecasting systems, a language model is rarely the only signal available. A user may already have a prediction market price, a crowd forecast, or a time-series model output, and the real question is not "how accurate is this model?" but "does this model improve the forecast I already have?" The authors formalize this as relative competence, derive mathematically when a model's disagreement with an external forecast carries useful information, and build a method that learns from resolved outcomes how much weight to give each source within each domain.

Key Contributions

1. Formalization of relative competence under Brier loss. The paper defines two distinct quantities: direct advantage (does the model beat the external forecast on its own?) and combination value (does the model add complementary information to a pool?). It proves that model disagreement helps only when it correlates with the external source's errors — that is, when E[de] ≠ 0, where d is model-minus-external disagreement and e is external forecast error.

2. A routing identity (Theorem 1). The authors derive an exact closed-form expression for the gain from using domain-specific pooling weights rather than a single global weight: the gain equals a weighted sum of squared deviations between each domain's optimal weight and the global optimal weight. This tells you precisely when domain-aware routing pays off and how much.

3. The competence gate. A practical estimator that learns one affine source-weight vector per domain from resolved training outcomes, shrinks small-domain estimates toward a global estimate, and applies isotonic recalibration to the pooled forecast. The estimator has a closed-form solution and produces interpretable per-domain weights.

4. Empirical separation of outcome-estimated competence from verbal confidence. Across four Qwen models, the paper shows that a model's self-reported confidence does not reliably predict when it outperforms an external forecast, while competence estimated from outcomes supports better abstention decisions.

Main Findings

• Routing beats global pooling. On 2,357 resolved binary questions, the competence gate scores 0.0732 Brier versus 0.0771 for the main external baseline (p = 0.001) and 0.0759 for a global simplex pool (p = 0.010). A plain global two-source affine pool achieves only 0.0767 and is not significantly better than the baseline (p = 0.39), so the gain comes from domain-specific routing rather than pooling alone.

• The routing identity predicts real gains. The gain predicted by Theorem 1 from training-fold weights is 0.0035, close in scale to the realized held-out gain of 0.0027 over the global simplex pool.

• Model value depends on the strength of the external forecast. The gate assigns more weight to models where the external forecast is weak and less where it is strong. On the official ForecastBench live-market subset (1,294 questions with genuine market probabilities), the gate gives no significant improvement (p = 0.42) and largely defers to the market.

• Results hold under leakage controls. On the 2,103-question subset resolving on or after June 1, 2025 — reducing the chance that outcomes appeared in training data — the baseline scores 0.0805 and the gate 0.0767 (p = 0.006).

• The gate beats a leakage-safe time-series prior. On 172 structured questions with ARIMA-based priors refit at each forecast date, the gate improves from 0.172 to 0.160 (gain 0.012, 95% CI [0.003, 0.021], p = 0.014). FRED alone improves from 0.178 to 0.166 (p = 0.041). ACLED, Wikipedia, and DBnomics show positive point estimates but remain underpowered.

• Verbal confidence fails as a routing signal. AUCs for predicting whether the model beats the external forecast range from 0.389 to 0.501 across four Qwen models — none reliably above chance. Qwen3-8B scores 0.389 (95% CI [0.36, 0.42]), meaning higher confidence is mildly associated with worse relative performance.

• Outcome-estimated competence supports safer selective prediction. Ranking by estimated competence produces a risk-coverage area of 0.019 versus 0.054 for verbal confidence — a lower-risk retained set as coverage decreases.

• The gate matches, but does not beat, a strong contextual baseline. A domain-aware logistic stacker with isotonic recalibration scores 0.0740 versus 0.0732 for the gate, a non-significant difference (p = 0.53). The gate's contribution is interpretability and closed-form weights, not additional accuracy.

• Domain labels matter. Coarse and fine domain partitions preserve the routing effect, but shuffled and random groupings reduce it by more than tenfold — confirming that the learned weights reflect genuine domain structure.

• Nonnegative weights still work. Constraining the gate to the probability simplex gives 0.0748, worse than signed weights but still better than the baseline.

• Showing models the external forecast hurts. A separate elicitation where models observe the external probability increases copying and reduces the independent information available to the gate.

Methodology in Plain English

The setup is binary event forecasting. For each question, the system has an external forecast (a market price, crowd estimate, or statistical prior) plus forecasts from several language models that never see the external value. The goal is to combine them well.

The authors start theoretically. Using the Brier score as the loss function, they ask: when does a model's disagreement with the external forecast actually help? The answer is that disagreement only helps if it systematically corrects the external source's errors. If disagreements are random noise, the optimal weight is zero.

They then derive how much is gained by letting each domain (source of questions, such as Polymarket, Metaculus, FRED, or ACLED) have its own pooling weight rather than sharing one global weight. The result is a clean quadratic identity: routing helps in proportion to how much each domain's preferred weight differs from the global one.

The practical method follows directly. Within each training fold, the system solves a ridge-regularized least-squares problem to find per-domain weights over all sources. Small domains get shrunk toward the global weight using a factor that depends on how many questions the domain has. The pooled forecast is clipped to [0, 1] and passed through an isotonic calibrator fit on training data.

Evaluation uses five-fold cross-validation on the ForecastBench pool of 2,357 questions from eight sources. Crucially, structured questions (FRED, ACLED, Wikipedia, DBnomics) have no contemporaneous probabilistic forecast, so the authors construct reference values from training-fold outcome rates only — never from held-out outcomes. For a stricter test, they build ARIMA-based time-series priors using only data available at each forecast date. Baselines include the base rate, the external forecast, each individual model, global affine and simplex pools, and a domain-aware logistic stacker, all under the same recalibration protocol. Paired bootstrap resampling over questions provides p-values.

Self-assessment is tested separately: the authors check whether verbal confidence predicts the question-level indicator of whether the model's squared error is lower than the external forecast's.

Why This Matters

Impact on research. The paper reframes hybrid forecasting evaluation away from standalone model accuracy and toward marginal value over an existing signal. The routing identity gives a quantitative tool for deciding when domain adaptation is worth the estimation cost, and the negative result on verbal confidence cautions against using self-reports for comparative deferral decisions. The finding that models lose independent information when they can see the external forecast is also a useful design constraint for hybrid pipelines.

Real-world applications:

• Decision-support assistants that combine an LLM's analysis with market odds, expert panels, or economic nowcasts, and need to know when to trust the model's dissent.
• Prediction-market and trading systems that blend model signals with live prices, where the paper's result suggests deferring to the market when it is already strong.
• Policy and risk analysis using structured indicators such as economic series or conflict-event data, where a leakage-safe time-series prior plus model forecasts can be combined with measured weights.
• Selective model routing in production, where the system abstains from blending the model on questions where historical evidence shows little marginal value.

Industry relevance. The method is cheap — closed-form estimation plus a small calibrator — and its weights are interpretable per domain. That makes it suitable for systems that need auditable, conservative defaults and a principled trigger for increasing model weight as resolved outcomes accumulate.

Future Directions

• Question-level competence estimation. The current gate uses source domain as a coarse regime label. Finer features such as forecast horizon, inter-model disagreement, crowd dispersion, prior uncertainty, and market liquidity could support more precise routing.

• Comparing outcome-based and internal competence signals on the same target. The paper tests verbal confidence but notes that learned confidence estimators or representation-based uncertainty methods might perform better; the comparison has not yet been run head-to-head on relative advantage.

• Prospective evaluation. Historical evaluation cannot eliminate data contamination risk. Running the gate forward in time on questions that resolve after deployment would provide stronger evidence than the leakage-controlled subset.

• Cost-aware and latency-aware routing. The experiments do not model token cost, provider latency, or API budget. A deployment-ready gate would need to weigh marginal accuracy against marginal cost.

• Transfer to unseen domains. The gate requires resolved outcomes from the domain it routes on, so performance on genuinely new domains remains an open question. The paper recommends starting conservative and increasing model weight only as evidence accrues.

Target Audience

This paper is most useful for machine learning researchers working on forecast combination, calibration, and model routing; applied scientists building hybrid forecasting systems that mix LLMs with markets, crowds, or statistical priors; and practitioners in decision-support, risk, or quantitative forecasting roles who need an evidence-based rule for when a language model should influence a forecast and when it should be ignored. Readers without a statistics background will still follow the problem framing and empirical results, but the theoretical sections on Brier loss decompositions and the routing identity will require comfort with expectation notation and least-squares estimation.

Authors’ abstract

In hybrid forecasting, a language model is often one of several available signals. A system may already have a market, crowd, or statistical forecast and must decide whether the model adds useful information or should be ignored. The relevant target is therefore not standalone model accuracy, but relative competence, defined as the model's marginal value beyond the available external forecast. Under Brier loss, we characterize when model disagreement can improve an external forecast and derive the gain from using domain-specific rather than global pooling weights. We then introduce a competence gate that estimates domain-level source weights from resolved outcomes, shrinks uncertain estimates toward a global weight, and recalibrates the pooled forecast. Across 2,357 resolved binary questions and five language models, the gate improves the main external baseline from 0.0771 to 0.0732 Brier and significantly outperforms global forecast combinations. The gain remains significant under leakage controls and against a leakage-safe time-series prior on the pooled structured set, with separate evidence on FRED. In contrast, the gate gives no significant improvement on the official ForecastBench market subset, where it largely defers to the market. Across four Qwen models, verbal confidence does not reliably identify when the model outperforms the external forecast, while outcome-estimated competence supports better abstention decisions. These results provide a practical approach for selective model use based on measured marginal value.