The Pulse
XConf Teaches Models to Calibrate From Their Past
A new arXiv preprint describes XConf, a confidence-estimation method that uses a model’s graded past episodes to improve calibration and selective prediction.
AI.info Team ·
XConf Challenges One-Session Confidence Scores
Most language-model confidence methods judge an answer from the evidence available during one inference: the model's reasoning, its token probabilities or several independently sampled responses. A paper submitted to arXiv on September 15 argues that those signals leave out the evidence a model has accumulated from earlier successes and failures.
The proposed method, XConf, records graded episodes from a model's previous work. Each record contains the task, the model's reflection, its stated confidence, the eventual outcome and a lesson written after grading. When a new task arrives, XConf retrieves earlier episodes with similar tasks and similar confidence levels, then uses their historical success rate to inform a revised estimate.
The authors—Caiqi Zhang, Xiaochen Zhu, Chengzu Li, Yulong Chen, Dharshan Kumaran and Nigel Collier—describe the system in the paper Confidence Comes from Experience: Experiential Confidence Estimation from Reasoning to Agents. The work is an arXiv preprint.
“We argue that the current inference is not a sufficient basis for confidence.” — Caiqi Zhang, Xiaochen Zhu, Chengzu Li, Yulong Chen, Dharshan Kumaran and Nigel Collier, authors of the paper
Recall First, Then Reflect
XConf divides its process into two stages. The Recall stage searches the stored episodes and reads how often similar past judgments turned out to be correct. The Reflect stage presents that record to the model, asks it to identify a recurring failure mode and then produce a new confidence estimate informed by its own track record.
The design does not require access to model logits or changes to model weights. The paper also says the method works across output formats and needs only one answer generation for the confidence estimate. That separates it from approaches that depend on internal probability data or repeated sampling, which are not available from every model and can add substantial inference cost.
23 of 24 Comparisons Against Self-Consistency
The evaluation covers nine benchmarks spanning reasoning, coding, multimodal question answering and interactive agents. The authors test four models from three model families and compare XConf with ten-sample self-consistency, a method that generates multiple answers and uses agreement among them as a confidence signal.
According to the paper, XConf matches or exceeds ten-sample self-consistency on AUROC—the area under the receiver operating characteristic curve—in 23 of 24 comparisons. The authors also report much lower expected calibration error, or ECE, while using one-tenth of the generation cost described for the self-consistency baseline.
Those results measure whether confidence rankings separate likely-correct answers from likely-incorrect ones. They do not show that XConf makes the underlying answer more accurate on every task. The method instead aims to help a system decide when to proceed, retry, escalate or abstain.
Selective Prediction Raises Agent Success
The paper tests that decision-making use case by having systems abstain from the 10 percent of episodes receiving the lowest confidence scores. On agent tasks, the authors report that this raises delivered success rates by up to 8.7 percentage points.
Selective prediction changes the meaning of success: a system is judged not only by how often it completes a task, but also by whether it avoids presenting weak attempts as reliable results. For an agent handling a multi-step workflow, a confidence estimate can determine whether to continue execution or hand the task to another process.
XConf's reliance on graded history also creates a practical dependency. The approach needs a store of past episodes with trustworthy outcomes and lessons. The abstract does not establish how performance changes when that record is small, poorly labeled or drawn from tasks that differ sharply from the new request.
A Memory System, Not a New Model
XConf does not retrain the language model or modify its parameters. It adds an external record of prior episodes and a procedure for retrieving and reflecting on them. That makes the proposal closer to a confidence layer around an existing model than a new model architecture.
The paper's central claim is narrower than a claim that models learn permanently from every interaction. XConf uses accumulated experience at inference time, provided the system has stored, graded and retrieved relevant episodes. Its reported gains therefore depend on the quality of that memory and on the similarity between past and present tasks.
The preprint is available from arXiv. Its evidence consists of the authors' evaluation across the nine reported benchmarks.