Provenance-Aware Belief Updating in Human-AI Feedback Loops

AI summary

Overview

Research area: Belief updating and evidence accounting in language models, specifically within repeated human-AI interaction where model outputs re-enter the model's own context. The paper sits at the intersection of Bayesian social learning (data incest, repetition bias), retrieval/provenance tracking, and parameter-efficient fine-tuning of language models.

Technical level: Advanced. The paper assumes familiarity with Bayesian posterior log-odds, Beta-Bernoulli conjugacy, KL divergence, QLoRA fine-tuning, OOD generalization testing, and paired bootstrap inference.

Scope in one sentence: The paper introduces Latent Quotient Evidence Training (LQET), a relational training objective that teaches language models to update beliefs based on supplied report lineage rather than the raw count of reports, tested on a controlled synthetic binary benchmark across two 4B-class backbones.

Publication venue: the workshop titled Dynamic Alignment on Human-AI Coupled System (BiAlign Workshop). arXiv identifier 2610.13008v1 [cs.AI], dated 09 Oct 2026. Authors are Tianyi Huang (Ryquo) and Eric Liu, Jay Chong, Emily Yin, and Wanqi Zhao (App-In Club).

What This Paper Is About

When a language model's own output is quoted, summarized, or copied forward into later context, one observation can return as many "reports," and a model that counts reports will treat this repetition as independent corroboration. This is a known failure mode in Bayesian social learning (sometimes called data incest), and it produces overconfidence that compounds in closed feedback loops. The paper asks a structured question: if report lineage is explicitly supplied — that is, the prompt states which reports relay existing evidence and which introduce genuinely new observations — can a model learn to update only when new evidence actually arrives, and to apply the same update regardless of how many relay steps preceded it?

Key Contributions

1. A formalization of provenance equivalence. The authors define transcripts as provenance-equivalent when they contain the same set of distinct, query-relevant evidence roots, and derive exact Bayesian log-odds targets under an uncertain source-reliability model (a Beta-Bernoulli setup in which each source's reliability is integrated out and multiple roots from the same source become dependent after marginalization).

2. A relational training objective (LQET). The objective combines sparse pointwise soft-target supervision with four invariance constraints, three new-evidence update-consistency constraints, and one presentation-order constraint. It supervises relationships between matched transcripts rather than only individual transcript targets.

3. Empirical validation across two backbones and paired seeds. Qwen3-4B-Instruct-2507 and Phi-4-mini-instruct are each trained with three paired seeds (0, 1, 2), evaluated on IID families, 800 controlled-shift OOD families across five factor groups, and 200 recursive-relay families at depths 1, 2, 4, and 8.

4. A confidence-dependent feedback simulation. The paper connects the static relations to a dynamic setting where model confidence, via a prescribed trust rule, controls how many inherited reports re-enter the model's context in later rounds.

Main Findings

• Large reduction in new-evidence update error: LQET lowers new-evidence update RMSE by 41.25% on Qwen and 44.45% on Phi. In raw terms, Qwen goes from 0.6816 (task-only) to 0.4005 (LQET), an effect of −0.2812 with 95% interval [−0.3070, −0.2565]; Phi goes from 0.7461 to 0.4144, an effect of −0.3317 with 95% interval [−0.3545, −0.3091].

• Large reduction in relay drift: Depth-8 relay drift falls by 91.36% on Qwen (1.0122 to 0.0875, effect −0.9247, interval [−1.1175, −0.7703]) and 89.38% on Phi (1.1079 to 0.1177, effect −0.9902, interval [−1.0823, −0.8988]).

• Prediction quality preserved: Supervised-view KL changes by +0.00228 on Qwen and +0.00120 on Phi, with both bootstrap upper bounds below the prespecified +0.02-nat non-inferiority margin. Table 1 reports the Qwen interval as [+0.00116, +0.00356] and the Phi interval as [−0.00002, +0.00251]. Both backbones satisfy the full conjunctive decision criterion.

• Improvements hold in every seed and every OOD shift: The three-seed mean update RMSE is lower on every one of the five OOD shifts for both backbones. Along nested relay histories, the LQET three-seed mean remains below 0.14 at every depth, whereas task-only drift accumulates sharply through depth four and remains near one log-odds unit at depth eight.

• Selective invariance rather than blanket update suppression: Matched copy-versus-new-evidence controls hold report author and displayed outcome fixed while varying declared provenance. Copy-versus-new RMSE drops from 1.0145 to 0.4383 on Qwen and from 0.9818 to 0.4630 on Phi; the corresponding zero-intercept slopes rise from 0.027 to 0.782 and from 0.067 to 0.769. Base-to-relay and matched-copy drift fall by roughly 15–40 fold, while new-root and matched-new RMSE stay close to task-only.

• Closed-loop feedback amplification is reduced: In the simulation with a fixed 64-family subset, initial trust in {0.2, 0.8} yields 128 trajectories per trained model, with three rounds and 1–3 simulated human relays per round. All three metrics improve in every backbone-seed pair, with mean reductions of 64.6–76.9% on Qwen and 79.1–90.9% on Phi.

• Component study (Qwen, seed 0): Adding the order term lowers the order-sensitivity gap from 0.0426 to 0.0359, update RMSE from 0.4105 to 0.4058, and supervised-view KL from 0.0239 to 0.0232; depth-8 relay drift is 0.0959 without the order term and 0.0753 with it. Update-consistency supervision plus the added composition views appears to account for most of the compositional gain.

• A remaining numerical weakness: Under the held-out prior-and-calibration-support shift, lower update RMSE accompanies higher supervised-view KL. This is the only shift that changes the numerical supports for both the class prior and the source-calibration records, and the authors identify numerical generalization as an open challenge.

Methodology in Plain English

The researchers built a synthetic benchmark where every example is a question about a hidden binary outcome (for instance, "Is the valve open?"). Each prompt supplies a prior probability, calibration records for named sources (how many correct and incorrect measurements each previously made), and a set of reports. Crucially, the prompt states provenance directly: some reports say they carried out a new measurement, while others say they simply read and relayed a predecessor report. The underlying "evidence root" identifiers are never shown in the prompt.

From this visible information, the exact Bayesian posterior is computable. Because the evidence set — not the number of textual mentions — determines that posterior, two prompts that contain the same distinct evidence roots must have the same target, and adding a relay must change nothing. The authors use these facts to build matched families of 14 transcript views, arranging them so that a relay-only addition (h10 vs. h00), a new root after a relay (h11 vs. h01), two separately added roots and their combination (c10, c01, c11), and a presentation-order reversal (c11 vs. c11-rev) all have known relationships to one another.

The model's "belief" is read out as a forced-choice score: the difference between the next-token logit for "YES" and for "NO". LQET trains on these scores with a combined loss = task loss + 4 × invariance loss + 2 × update loss + 1 × order loss, using a Huber penalty with δ = 0.25 on the relational residuals. Seven views get pointwise soft targets; six more views, including h11 and c11, are supervised only through the relational terms, for 13 training views total (task-only uses the seven pointwise views). Training is QLoRA with NF4 double quantization and BF16 compute, LoRA rank 32, α = 64, dropout 0.05, on all linear modules, for 500 optimizer updates (two epochs) over 4,000 training families and 400 validation families, with the same optimizer-update count for both methods. Loss weights, the Huber threshold, and optimization settings were fixed before the multi-seed Qwen runs and transferred unchanged to Phi; no validation-based hyperparameter or checkpoint search was performed. Statistical claims use 10,000 paired crossed seed-by-family bootstrap replicates with 95% percentile intervals, and the decision criterion was prespecified as a conjunction of drift/RMSE improvements in every seed and a KL margin of at most +0.02 nats.

Why This Matters

Impact on research. The work reframes evidence accounting as a relational learning problem: instead of assigning each transcript its own target, it supervises the differences between transcripts in the same provenance-equivalence class. It extends prior repetition-invariance work to explicit multi-hop lineage, exact reliability-conditioned Bayesian targets, composition, and presentation order, and it supplies a controlled testbed with a deterministic prompt-only calculator that verifies every static-view target is fully specified by the visible prompt.

Real-world applications (the paper's benchmark domains and settings, not deployed systems):

• Multi-agent or chained LLM pipelines where one model's summary is fed to another model, or back to the same model, causing a single claim to be read as multiple independent confirmations.
• Sensor fusion or decentralized estimation with named, calibrated sources, where the same measurement may be quoted through multiple relays and naively counted more than once.
• Provenance-carrying question answering over document summaries, where downstream summaries inherit rather than introduce support.
• Monitor or alerting systems that must distinguish a genuinely new signal from a recirculated one before escalating confidence.

Industry relevance. Repeated human-AI interaction can amplify judgment biases through feedback, and the paper's closed-loop simulation shows that a model whose confidence determines how many derived reports return can produce amplification that static evaluation would miss. The reported reduction in amplification when models are trained on the relational objective is the paper's argument for training-time treatment rather than prompt-time patching. The practical framing applies to products where a model's output becomes the next turn's input — assistants, agent frameworks, and human-in-the-loop decision tools. The paper does not report deployment or production results, nor transfers beyond two 4B-class backbones.

Future Directions

• Scale and naturalism. The authors state that transfer to larger models, naturalistic tasks, and incomplete or unreliable provenance remains for further study; the current evidence covers two 4B-class backbones and a controlled binary setting.

• Free-form probability estimation and downstream decision quality. The forced-choice YES/NO score measures binary preference, and the paper notes that estimating probabilities in free-form text and evaluating downstream decision quality require separate investigation.

• Numerical generalization under distribution shift. The held-out prior and calibration-count support is the one shift where lower update RMSE comes with higher supervised-view KL, so the authors flag numerical generalization as an unresolved problem.

• Human validation of the feedback mechanism. The confidence-dependent simulation isolates report recirculation under a prescribed trust rule (trust updates as τ_{r+1} = 0.6τ_r + 0.4c_r, with relays m_r = 1 + min{2, floor(3τ_{r+1}c_r)}); the paper states that studies with human participants are needed to assess how this mechanism operates in practical human-AI interaction.

Target Audience

Researchers and practitioners working on belief updating, calibration, and evidence accounting in language models; anyone building multi-agent or chained LLM systems where outputs re-enter context; and readers interested in provenance-aware machine learning, data incest correction, or alignment methods for coupled human-AI systems. The statistical and Bayesian machinery makes it most accessible to readers already comfortable with probabilistic inference and parameter-efficient fine-tuning; the plain-language problem statement about repeated reports sounding like independent corroboration makes the motivation accessible to a broader audience.

Authors’ abstract

In repeated human-AI interaction, a single observation can be quoted, summarized, and returned to a model as multiple reports, making repetition look like independent corroboration. We study whether language models can use supplied report lineage to update according to evidence provenance rather than report count. We introduce Latent Quotient Evidence Training (LQET), a relational objective that combines sparse Bayesian supervision with constraints that encourage stable predictions when reports add no new evidence and consistent updates to the same new evidence across relay histories. On a controlled binary benchmark, LQET reduces new-evidence update RMSE by 41-44% and log-odds drift after eight relay steps by 89-91% relative to task-only training across Qwen3-4B and Phi-4-mini, with three paired training seeds per backbone. The gains extend to a closed-loop simulation in which model confidence controls how many relayed reports re-enter the model's context in later rounds. By grounding belief updates in supplied provenance, LQET offers a route toward dynamic human-AI systems in which influence follows new evidence, not the number of times the same evidence returns.