Unifying Conformal Language Tasks with In-Context Ensembles

AI summary

Overview

Research area: Natural language processing, specifically content-selection tasks (summarization, extractive question answering, evidence extraction, PII detection) combined with conformal prediction, a distribution-free statistical framework for uncertainty quantification.

Technical level: Intermediate. The core ideas are accessible, but the paper includes formal results on ensemble ordering statistics and a complementarity condition that assumes familiarity with conformal prediction and basic probability.

Scope: The paper proposes a task-agnostic way to build the scoring function that conformal content-selection pipelines depend on, replacing hand-written per-task prompts with an ensemble of in-context-learning example retrieval strategies.

What This Paper Is About

Many NLP systems are really content-selection systems: given a document, pick out the parts that matter. They must balance coverage (don't drop anything important) against conciseness (don't keep everything). Conformal prediction can guarantee the coverage half of that trade-off, but how concise the output is depends entirely on the quality of an underlying relevance score function. Today, that score function is almost always a large language model driven by a carefully hand-written, task-specific prompt, which is brittle and expensive to maintain. This paper asks whether the prompt can be eliminated entirely, replaced by a fixed ensemble of in-context examples that adapts to any content-selection task without per-task engineering.

Key Contributions

1. A universal ICL ensemble scoring function for content selection across disparate tasks, paired with a recall-oriented conformal calibration procedure that retains its coverage guarantee. The same fixed configuration works on all seven evaluated datasets without modification.

2. Four mechanistically diverse sub-scoring functions based on different in-context example retrieval signals: anchor-based determinantal point process (DPP), pattern-based DPP over positive/negative centroids, lexical BM25 retrieval, and random sampling. Each is best on at least one dataset, and their failures are designed not to coincide.

3. A mathematical formalization of score ensembling for recall-oriented conformal prediction, including a complementarity condition that characterizes exactly when averaging two scorers raises the worst-case (floor) conformal score, plus an O(1/K) saturation bound showing diminishing returns as ensemble size grows.

4. An empirical demonstration across seven datasets, five domains, and four task types, showing consistent gains over hand-crafted prompts and over any single retrieval strategy, using only 150–440 labels per task.

Main Findings

• Coverage is preserved without degradation. Score-level ensembling (averaging relevance outputs on a common [0,1] scale) retains the 1−α finite-sample coverage guarantee, unlike set-level ensembling, which is known to degrade coverage to 1−2α. Empirical coverage stayed within one percentage point of target across all datasets.

• Conciseness improves substantially at fixed coverage. The ensemble named Ens4 removed substantially more irrelevant content than the manually prompted baseline at the same coverage, with up to roughly 50% reduction in retained length.

• Ens4 beats both baselines on all seven datasets. MAP improvements over the no-example hand-written prompt baseline ranged from +0.090 to +0.170 (a 12% to 59% relative gain), with 95% bootstrap confidence intervals excluding zero everywhere. It also beat the per-dataset oracle best single strategy in every case.

• Complementarity, not raw strength, drives the gain. On HotpotQA, no single retrieval strategy beat the hand-written prompt baseline, yet the ensemble improved by +0.090 MAP. Ensembling also beat the best individually-tuned configuration even though it used weaker k=2 variants of each strategy.

• No single retrieval strategy dominates. All four strategies, including the deliberately simple random sampler, won on at least one dataset, which supports the argument that choosing one strategy a priori is unreliable.

• Ensemble size shows the predicted diminishing returns. Going from K=2 to K=3 to K=4 produces strictly decreasing marginal gains on all seven datasets, matching the O(1/K) bound from the theory. Ens4 (K=4) still outperformed all smaller combinations.

• Two in-context examples per strategy is the sweet spot. k=1 underperformed k=2 on six of seven datasets; k≥5 degraded performance on most, attributed to DPP kernel saturation and prompt-length dilution.

• The gains are not explained by extra labels, extra compute, or sampling noise. Ens4 beat a supervised classifier trained on the same labeled pool on six of seven datasets, beat temperature-varied stochastic ensembles under matched compute, and beat a temperature-ensembled prompt-only baseline on six of seven datasets.

• The complementarity theory held empirically. When the condition Comp > S_max − S_min was satisfied at the sentence level, an ensemble advantage was observed over 99% of the time.

• Results are robust to model choice. Replacing the default Gemini-2.5-Flash-Lite scorer with Llama3-8B, Qwen3-8B, or GPT-5.6-terra kept Ens4 ahead of both baselines in nearly all configurations, and removing the one-line task hint (making the setup fully dataset-agnostic) cost little on six of seven datasets.

Methodology in Plain English

The approach starts by reframing a range of NLP tasks as the same underlying operation: decide, for each sentence or span in a document, whether it is relevant. A relevance score function assigns each span a number between 0 and 1.

Normally that score function is an LLM given a detailed prompt written by a human describing what "relevant" means for that specific task. This paper removes the description entirely. Instead, it shows the LLM a small number of labeled examples — documents with spans marked relevant or not — and lets the model infer the criterion from the demonstrations.

To make this work across many different notions of relevance, the authors build four different ways of choosing which examples to show. One finds the most similar document in a pool and fills the rest of the examples using a diversity-promoting selection rule. One embeds positive and negative spans, computes a direction vector separating them, and picks examples that span that direction. One is plain keyword search. One is random. These are chosen because their mistakes should not line up: when one scorer undervalues an important sentence, the others probably won't.

Each scorer produces its own relevance numbers, and the four are averaged into a single score. That averaged score is then fed into standard conformal calibration: run it over a held-out set of 100 labeled documents, take the appropriate quantile of the resulting scores, and use that quantile as a threshold. On new documents, any span scoring below the threshold is dropped. Because the ensemble is just another score function, the standard coverage guarantee applies unchanged.

The theory section explains when averaging actually helps. It defines "complementarity" as the degree to which two scorers disagree about which relevant spans are hardest — that is, which positives each one scores lowest. If that disagreement is larger than the gap between the two scorers' individual worst-case scores, the average raises the floor, which raises the calibrated threshold, which shrinks the output. The paper also shows each additional scorer contributes proportionally less, following a 1/K pattern, so a handful of diverse scorers is enough.

The evaluation uses one fixed configuration across all seven datasets, so the only thing that changes between experiments is the data.

Why This Matters

Impact on research. Conformal prediction is increasingly applied to language tasks, but its usefulness in practice has been bottlenecked by the quality of the underlying score function — and improving that function has meant prompt engineering, which does not accumulate into general knowledge. This paper reframes the problem as an ensemble design problem with a formal characterization of when diversity helps, giving the field a principled lever to pull instead of an intuitive one. It also supplies a bridge between the conformal prediction literature and the in-context example selection literature, which had largely developed independently.

Real-world applications:

• Clinical evidence extraction. Systems that pull supporting spans from medical literature or patient records for a clinician to review need high recall — missing a relevant finding is costly — while keeping the reading burden low. A fixed scoring pipeline that works without domain-specific prompt tuning makes deployment across specialties far cheaper.

• Legal document review. Contract analysis and NDA clause scoring involve finding the clauses that matter in long documents under tight time budgets. The recall guarantee means the reviewer can trust that relevant clauses are in the retained set, while the conciseness gain directly reduces billable review hours.

• Retrieval-augmented generation. RAG pipelines rely on selecting relevant passages before generation. Better-calibrated relevance scores with a coverage guarantee reduce both hallucination risk from missing context and noise from irrelevant retrieved text.

• PII detection and redaction. Selecting spans that constitute private information for masking is a content-selection problem where the cost of a miss is high. A conformal recall guarantee provides an auditable property, and the task-agnostic scoring means the same pipeline can be applied across document types without per-corpus prompt maintenance.

Industry relevance. The economic argument is straightforward: hand-written prompts are per-task, per-domain engineering artifacts that must be re-validated whenever the underlying model changes. A fixed ICL ensemble with a modest label budget (150–440 examples) and an off-the-shelf smaller model like Gemini-2.5-Flash-Lite replaces that with a one-time setup. The fourfold increase in LLM calls is a real cost, but the paper argues the retrieval overhead is negligible and the label requirement is small enough to be practical for enterprise deployments.

Future Directions

• Extending beyond binary span relevance. The framework assumes content spans are either relevant or not. Many tasks involve graded importance, hierarchical structure, or overlapping spans, and the ensemble theory would need adaptation.

• Understanding the limits of complementarity. The paper demonstrates the complementarity condition holds in practice but offers no method for deliberately constructing scorers that maximize it. A principled procedure for generating maximally complementary scoring functions, rather than picking four heuristics spanning different signal types, could push conciseness further.

• Reducing the compute multiplier. Ens4 requires four LLM scoring calls per document instead of one. Distilling the ensemble into a single scoring function, or sharing computation across the sub-scorers, would make the approach more attractive for latency-sensitive or high-volume deployments.

• Generalizing the theory past score averaging. The formal results cover the element-wise mean. Other aggregation rules, such as weighted means or trimmed combinations informed by per-dataset reliability, may offer better complementarity utilization but currently lack the guarantees established here.

• Testing at larger model scale and longer documents. The reported experiments use models in the 8B to Flash-Lite range on documents up to 461 sentences. Whether the ICL-driven approach still beats hand-written prompts when the scorer is a frontier model with strong instruction-following remains open.

Target Audience

This paper is most useful to applied machine learning researchers and engineers working on uncertainty quantification for language models, particularly those who have tried to deploy conformal prediction in production and found that the coverage guarantee was easy to obtain but the resulting output was not concise enough to be useful. It also speaks to practitioners building extractive summarization, RAG, legal review, or clinical NLP pipelines who want to eliminate per-task prompt maintenance. Researchers in in-context learning will find the four retrieval strategies and the diversity ablations instructive independently of the conformal framing. Finally, the theoretical results on score ensembling and the O(1/K) saturation bound are relevant to anyone working on ensemble methods for conformal prediction more broadly, including in classification and regression settings where the same aggregation question arises.

Authors’ abstract

Many NLP tasks, such as summarization and extractive question answering, reduce to retrieving relevant content from documents under two constraints: coverage, retaining enough pertinent information to achieve some goal, and conciseness, removing as much irrelevant information as possible. Conformal prediction methods have been used to guarantee coverage, and must be optimized for conciseness through design of a score function. State-of-the-art scoring functions use hand-engineered LLM prompts asking the model to rate the importance of content, but manual prompt engineering is labor-intensive and task-specific. We introduce the Conformal Relevance framework which uses in-context learning example curation and ensembling to create a score function which maintains coverage while improving conciseness with minimal manual input. We demonstrate this framework's application on seven NLP tasks, and also theoretically study the impact of diversity for ensembled conformal scores, giving a complementarity condition that characterizes when ensembling improves worst-case sentence scores, and a saturation bound on ensemble improvement.