Contrastive Discovery: Open-Ended Scientific Discovery over Competing Explanations

AI summary

Overview

Research area: AI for scientific discovery — specifically LLM-based agents that generate and test hypotheses over datasets.

Technical level: Intermediate. The core idea is conceptually simple (replace a single hypothesis with a set of competing explanations), but the paper assumes familiarity with Bayesian surprise, Kullback–Leibler divergence, and Monte Carlo tree search.

Scope: The paper introduces Contrastive Discovery, a reframing of how discovery agents represent the unit of scientific progress, and evaluates it across six tabular datasets against pointwise baselines using the same agent scaffold and experiment budget.

What This Paper Is About

Most LLM discovery agents test one hypothesis at a time against its negation, so when the evidence comes back negative the agent learns only that the claim is unlikely — not which alternative explanation the evidence actually supports. This paper argues that the unit of progress should instead be a "contrast set": a scientific question paired with mutually exclusive and collectively exhaustive competing explanations. The goal is to let an agent resolve scientific questions, rather than merely reject hypotheses, from the same evidence and the same budget.

Key Contributions

1. A new representational unit for discovery. The paper defines a contrast set — a question plus M mutually exclusive and collectively exhaustive explanations — and shows that pointwise discovery is exactly the M = 2 special case where the only alternative is ¬h.

2. A contrastive scoring rule. Evidence is scored by the entropy-scaled Kullback–Leibler divergence between elicited prior and posterior beliefs over the set, κ_t = D_KL(p_post ‖ p_prior) / log₂M, which the authors call the contrastive belief shift.

3. An outcome taxonomy separating "how much" from "which." Updates are classified as confirmative, corrective, or unresolved (using a posterior support threshold τ_support = 0.80), and separately as ruled out (using τ_reject = 0.10), distinguishing belief magnitude from belief direction.

4. A demonstration that contrast sets are plug-in compatible. Because only the proposed-and-scored unit changes, an existing agent can use contrast sets either as a post-hoc evaluation lens on evidence it already collected or as a live search guide — no architectural change required.

Main Findings

• Pointwise rejections hide resolvable alternatives. Among the top-50 candidates ranked by pointwise surprisal (from each run's 200 experimental validations), only 4% were confirmed, 7% were ambiguous, and 89% were rejected. Re-evaluated with a post-hoc contrastive companion on the same evidence, 73% of those 50 candidates resolved to a specific alternative. Counting only confirmations as resolutions, pointwise discovery leaves 96% unresolved versus 23% for Contrastive.

• Prior collapse drives the problem. Pointwise hypothesis proposals concentrate in the "maybe true" bin across three belief elicitors (at least 59% for each), reaching 82% when gpt-4o both proposes and elicits. Explanations drawn from contrast sets spread more broadly, with the largest bin holding only 41% of mass under gpt-4o.

• Guided search resolves 67% more questions. At K = 200, Contrastive Discovery resolved 161.9 questions on average versus 97.2 for pointwise surprisal. Pointwise baselines guided by surprisal, novelty, or feasibility clustered together, suggesting the limit lies in the representation rather than the scoring function.

• The gain is almost entirely corrective. Restricted to confirmative resolutions, Contrastive sits at or below the pointwise surprisal baseline (82.2 vs. 97.2 at K = 200). The additional yield comes from cases where evidence shifts support away from the prior-leading explanation — cases pointwise discovery records only as rejections.

• Contrastive evaluation is the main driver. In ablations at K = 50, variants with pointwise evaluation reached at most 0.22 resolution rate, while variants with contrastive evaluation ranged from 0.77 to 0.96. Contrastive ranking helps under either evaluator, but pairing it with pointwise evaluation produces lower information gain than pointwise ranking, revealing a ranker–evaluator mismatch.

• More resolutions do not cost rulings-out. The decisive-unit rate at K = 200 was 0.91 for Contrastive versus 0.82 for pointwise, and higher across all six datasets.

• Human experts largely agree. In a blinded study, 38 evaluators with MS or PhD degrees judged 60 questions (median 8 raters per item, majority vote). Experts agreed with Contrastive's explanation on 78.4% of resolved questions and on 81.8% of questions where the pointwise agent rejected its hypothesis and stopped. Of 17 total disagreements, they favored a competing explanation in only three cases.

• Resolutions follow the evidence, not the wording. Given only numerical results for 14 claims adapted from DiscoveryBench, the elicitor ranked the documented answer first in all 11 cases where an interval could be recovered, and reversing the evidence's sign reversed resolutions on all 14. Recomputation of 331 analyses from raw data reproduced the agent's statistics on 96.1%.

• Cross-model agreement is high after evidence. Re-eliciting beliefs on 180 contrast sets with gpt-4o, gemini-3-flash, and claude-sonnet-4-6, all three chose the same leading explanation on 55% of sets before evidence and 88.9% after.

• Incomplete sets are the main failure mode, and abstention helps. Removing the supported explanation from 163 correctly resolved sets caused the system to decline on 46.6% and resolve wrongly on 53.4% (mostly to "no detectable effect"). Adding an abstention alternative cut false resolutions to 0.6%; on intact sets abstention was never selected (162/163 resolved correctly versus 159/163 without it).

• The gain survives cost and threshold controls. Joint scoring reached 161.9 versus 157.8 for independently scored alternatives, agreeing on the same leader 89.3% of the time. At equal token budgets: 148.3 versus 87.7. At equal dollar cost: 145.8 versus 92.2. Across matched threshold pairs from τ_support 0.60 to 0.90 on 1,543 cases, Contrastive resolved more at every pair and in all six datasets.

• Pointwise runs rarely recover the alternatives themselves. Against a stratified sample of 300 contrast sets, pointwise hypotheses contained the complete alternative set for only 2.6% of shared questions within a single run at matched budget, and 15.4% after pooling all pointwise runs (4–6× that budget).

Methodology in Plain English

The researchers kept an existing discovery agent — the AutoDiscovery scaffold with a gpt-4o backbone — and changed only what that agent proposes and scores.

For each step, the agent poses a local scientific question (for example, how self-employment relates to mortgage denial rates given poor credit) and generates a small set of competing answers meant to cover the plausible outcomes: higher, no different, or lower denial rates. A separate LLM "belief elicitor" then distributes 100 allocation units across those answers, first before seeing any experimental result and again afterward. To reduce noise, five allocations are sampled and averaged, and the answer order is shuffled to counter positional bias.

The score for a candidate discovery is how much that belief distribution moved, normalized by log₂M so sets of different sizes are comparable, and clipped at 1 when used as a search reward. Beliefs are never computed from a hand-specified likelihood; they are elicited directly from the model. Questions are chosen for exploration by Monte Carlo tree search with a UCT rule, under a fixed budget of N = 200 experimental validations per run.

Baselines are pointwise: the same agent proposes a single hypothesis, the elicitor rates it on a five-level scale (ten ratings each for prior and posterior, fit to a Beta distribution), and search is guided by pointwise surprisal — or, in additional comparisons, by LLM-judge scores for novelty and feasibility. Evaluation counted how many scientific questions among the top-K ranked candidates reached a verdict, averaged over seeds within each dataset and then over the six datasets.

Why This Matters

The paper reframes a measurement problem in open-ended discovery. When there is no oracle for which discovery should come next, the unit an agent treats as "learned" determines what it can extract from its evidence. Collapsing the alternative space to ¬h throws away information that a single experiment already contains — 73% of top pointwise rejections in this study were resolvable to a named alternative on evidence already collected.

Real-world applications (drawn from the paper's domains):

• Materials science: resolving whether a candidate compound's property is driven by one descriptor versus a competing one, rather than simply rejecting a single structure–property claim.
• Astronomy: distinguishing among competing accounts of a galaxy population's features in survey data such as SDSS, where a rejected hypothesis says nothing about which population-level explanation holds.
• Fair-lending and policy auditing: the mortgage case is a direct example — evidence that self-employed applicants do not face higher denial rates is far less useful than evidence that they face lower or equal rates.
• Software requirements engineering and social/online-discourse analysis: turning ambiguous null results into directional findings that downstream teams can act on.

Industry relevance: the approach is a drop-in change to the proposal-and-scoring step, requiring no new architecture. The authors report it is cheaper per candidate than the pointwise baseline (52.8k versus 61.1k tokens) and that the gain holds under equal-token and equal-dollar comparisons, which matters for anyone running large discovery sweeps under a compute budget.

Future Directions

1. Generalize beyond tabular, observational data and a single gpt-4o scaffold. The authors explicitly note their results are on observational tabular datasets where a resolution is a statistical rather than causal finding, and they leave other discovery agents to future work.

2. Make contrast sets structurally sound by default. Malformed or incomplete sets are the main identified failure mode: removing the supported explanation from 163 sets produced wrong resolutions 53.4% of the time. Abstention reduced this to 0.6%, but the main experiments did not use it, and a strict majority of experts flagged a missing alternative in only 6 of 60 rated sets.

3. Reduce dependence on LLM-elicited belief. All resolutions rest on elicited distributions, so resolved-question counts depend on the elicitor. Cross-model agreement was 88.9% after evidence, but a more principled belief source remains open.

4. Extend from literature-grounded or data-grounded questions to a broader discovery loop. The authors note contrastive resolution only needs a question and plausible alternatives, so it could pair with hypothesis-generation systems that work from scientific literature rather than from datasets.

Target Audience

Researchers building LLM agents for scientific discovery, automated hypothesis generation, or experimental design will get the most from this paper — particularly those who already use Bayesian-surprise or novelty-based scoring and want to know what that scoring discards. It is also useful for evaluation researchers interested in how to measure progress in open-ended settings where no answer key exists, and for practitioners in data-rich applied domains (materials, astronomy, lending, software engineering) who want agents to return directional findings rather than null results. Readers need comfort with probability distributions and information-theoretic scoring, but the core argument is accessible to anyone who has watched a hypothesis test come back negative and wondered what to conclude instead.

Authors’ abstract

LLM agents are increasingly used for open-ended, data-driven scientific discovery, yet most of them evaluate hypotheses pointwise, testing an empirical claim $h$ only against its complement $\neg h$. When evidence rejects $h$, the agent learns that one explanation is unlikely but not which alternative the evidence supports. To recover that alternative, we introduce Contrastive Discovery, which replaces the single hypothesis with a contrast set, a scientific question paired…