eTracer: Towards Traceable Text Generation via Claim-Level Grounding

AI summary

Overview

• Research area: Natural Language Processing, specifically retrieval-augmented generation (RAG), attribution/citation, and hallucination verification, applied to the high-stakes biomedical domain.
• Technical level: Intermediate. The paper combines formal definitions of a grounding task and evaluation metrics with a practical pipeline and benchmark experiments; readers need some familiarity with LLMs, entailment (NLI), and embeddings.
• Scope in one sentence: The paper introduces eTracer, a plug-and-play framework that decomposes a generated response into atomic claims and links each claim to contextual sentences that support or contradict it, together with new metrics and a human-annotated biomedical dataset for evaluating such grounding.

What This Paper Is About

System-generated answers, especially in medicine, can contain hallucinated content, and today's citations are usually too coarse to verify (they point to whole pages or passages rather than to specific evidence). The authors propose a finer-grained "sentence ⇒ claim" grounding paradigm, in which each atomic claim in a response is aligned with the specific contextual sentences that support or contradict it, so that responses become traceable and their faithfulness can be quantified. The goal is to make verification both faster for users and more accurate than existing grounding approaches.

Key Contributions

1. The sentence ⇒ claim grounding paradigm plus an exploratory user study. The study found that sentence ⇒ claim grounding let participants verify responses up to 2.6 times faster than existing grounding approaches, while maintaining higher verification accuracy (reported in §A).
2. A formalization of claim-level grounding and its evaluation. The task is defined as assigning a signed score to every contextual sentence for every claim, capturing both evidence importance and polarity (support vs. contradiction). Three reference-free metrics are proposed: Claim Entailment Rate (CER), Evidence–Claim Semantic Similarity (ECSS), and Polarity-Flip Consistency Rate (PFCR).
3. Four response-faithfulness metrics derived from claim-level grounding results: Faithful Claim Rate (FCR), Ambiguous Claim Rate (ACR), Hallucinated Claim Rate (HCR), and Unverified Claim Rate (UCR).
4. The eTracer framework and a released human-annotated dataset. eTracer takes a response and its context as input and outputs decomposed claims with their supporting and contradictory evidence; code and data are available at https://github.com/chubohao/eTracer.

Main Findings

• Claim-level grounding beats sentence-level grounding overall: eTracer (τ = 0.0) reached F1 of 0.709 for supportive evidence and 0.946 for contradictory evidence, versus 0.557 and 0.815 for the base model Qwen3-4B-Instruct performing sentence-level grounding (no decomposition).
• Gains over the base model: Compared with Qwen3-4B-Instruct without decomposition, fine-tuned eTracer (τ = 0.0) improved F1 by 0.152 (27%) for supportive evidence and 0.131 (16%) for contradictory evidence; compared with the same base model with decomposition, the improvements were 0.070 (11%) and 0.129 (15.8%) respectively.
• Decomposition generally helps but costs time: Baselines with decomposition outperformed those without it by 0.009 (4%) to 0.148 (35%) in supportive-evidence F1, and by up to 0.237 (80%) for contradictory evidence, but inference time rose by between 1.669 s and 22.198 s.
• eTracer wins on reference-free metrics against end-to-end systems: It achieved CER 0.930, ECSS 0.787, and PFCR 0.940, while Qwen3-14B scored CER 0.309, ECSS 0.723, PFCR 0.784; the paper notes Qwen3-14B often copied context verbatim instead of extracting claims.
• The decomposition module is essential: Removing it reduced F1 scores, with improvements from having it ranging from 0.098 (16%) to 0.454 (94%) (e.g., contradictory-evidence F1 rose from 0.485 to 0.939).
• Evidence threshold matters: Across τ ∈ {0, 0.25, 0.5, 0.75, 1}, all evaluation metrics peaked at τ = 0.25, even though τ = 0.5 is the default; raising τ from 0.0 to 0.5 cut average grounding time from 22.188 s to 14.345 s per response at a small cost in F1 (0.709 → 0.705 supportive, 0.946 → 0.939 contradictory).
• Decomposition quality is high: eTracer achieved over 0.94 in both claim recall and claim precision.
• Reference-free metrics behave sensibly but should be read jointly: ECSS and PFCR show a modest positive correlation with precision, while CER is indicative of recall to some extent; the paper advises interpreting them together, not in isolation.

Methodology in Plain English

The framework runs in two parallel streams and then joins them:

1. Processing the response. The response is split into sentences with the NLTK sentence tokenizer. A trained decomposition model breaks each sentence into atomic, independent, semantically complete claims. An entailment model then checks whether each claim is actually entailed by its source sentence; claims that are not are treated as hallucinations and trigger re-decomposition until entailment holds or a maximum number of attempts is reached. Surviving claims are embedded as vectors.
2. Processing the context. The context is likewise split into sentences, each given a unique index, and embedded with the same embedding model.
3. Grounding. Claim embeddings are compared with sentence embeddings via cosine similarity, which serves as the evidence importance score; sentences above an evidence threshold τ become candidate evidence. An entailment model labels each candidate as Entailment, Contradiction, or Neutral, and a sign function maps these to +1, −1, or 0, giving each claim a signed score per context sentence.
4. Aggregation and linking. For each response sentence, its claims' evidence and polarity are aggregated into overall citations, producing explicit links between response sentences and context sentences.

Training: The decomposition model was initialized from Qwen3-14B and fine-tuned for 10 epochs on 182 sentence–claim groups; the entailment model was initialized from Qwen3-4B-Instruct-2507 and fine-tuned for 5 epochs on 4,267 instances (claim–evidence pairs labeled Entailment or Contradiction, plus, for each claim, five additional non-evidence sentences labeled Neutral). Both were trained on a single NVIDIA A6000 GPU.

Inference settings: Qwen3-Embedding-8B is the embedding model, τ defaults to 0.5, instruction-following baselines use one- or two-shot prompting with deterministic decoding (temperature = 0.0, top-k = 1.0), and all inference ran on a single NVIDIA A6000 GPU. Hallucination verification is retained while "reformulation" in Step 3 is disabled for fair comparison.

Data: Three biomedical corpora were used: PubMedQA (single-document QA), BioASQ-QA (multi-document QA), and TracSum (single-document summarization). From each, 100 instances were sampled and manually annotated, retaining 300 QA instances with their contexts. Contexts were augmented with contradictory evidence to balance supportive and contradictory cases. The resulting ground-truth dataset contains 578 sentence–claim groups, 1,564 claim–citation groups, and 4,579 claim–evidence pairs, split into train and eval at a 3:7 ratio (seed = 42).

Comparison setup: Baselines include NLI-based models (DeBERTa-v3-base-mnli, DeBERTa-v3-large-mnli), instruction-following models (Qwen3, Ministral-3, Llama-3.1-8B-Instruct), each run both without and with decomposition, plus end-to-end claim-level grounding baselines scored only on reference-free metrics because unconstrained decomposition cannot be aligned with the ground truth.

Why This Matters

• Impact on research: The paper reframes grounding granularity as a design choice rather than a case of "finer is always better" — token ⇒ token attribution produces too many noisy links, while passage ⇒ response citations are too coarse to verify. It also supplies reference-free metrics and a human-annotated dataset, which lowers the barrier to comparing grounding methods without new annotation.
• Real-world applications:
  • Clinical question answering and decision support, where an ungrounded statement can be harmful and each claim must be traceable to evidence.
  • Summarization of biomedical literature or patient records, where readers need to check individual facts rather than whole documents.
  • Retrieval-augmented search assistants that currently cite whole pages instead of specific supporting or contradicting spans.
  • Compliance and auditing workflows that require a record of which evidence supported which generated statement.
• Industry relevance: The framework is described as plug-and-play and post-hoc, so it can sit on top of existing generation systems without retraining the generator. The paper's direct comparisons to citation behavior in commercial RAG search engines (Perplexity AI, Bing Chat) make the work relevant to search vendors and to any organization shipping LLM responses that must be defensible. The latency measurements (14.345 s per response at τ = 0.5, 22.188 s at τ = 0.0, and up to 26.021 s for a decomposed Qwen3-14B baseline) also signal that deployment cost is a real consideration.

Future Directions

• Calibrating the evidence threshold. All metrics peaked at τ = 0.25, yet τ = 0.5 is used as the default, suggesting threshold selection is not yet resolved; the paper's own analysis of this effect is truncated in the provided text.
• Reducing inference cost. Grounding a single response takes on the order of 14–26 seconds in these experiments, and decomposition consistently increases runtime; making claim-level grounding cheaper is a clear practical question.
• Validating the reference-free metrics more broadly. CER, ECSS, and PFCR are offered as proxies when ground truth is unavailable, but the paper states they correlate only modestly with precision and should be interpreted jointly, so their reliability outside this benchmark is an open question.
• Extending beyond biomedicine and beyond post-hoc grounding. The unified formulation — finding contextual sentence-level evidence for response claims — is presented as applicable to summarization and QA, but the dataset and evaluation here are biomedical; whether the same gains hold in other high-stakes domains, or whether grounding could be integrated into generation rather than applied afterward, remains open.

Target Audience

Researchers and practitioners working on retrieval-augmented generation, attribution, hallucination detection, and fact verification, particularly those building or evaluating systems for high-stakes domains such as biomedicine. It is also useful for engineers who need to add citation and verification layers on top of existing LLM pipelines, and for benchmark designers interested in reference-free evaluation metrics.

Authors’ abstract

How can system-generated responses be efficiently verified, especially in the high-stakes biomedical domain? To address this challenge, we introduce eTracer, a plug-and-play framework that enables traceable text generation by grounding claims against contextual evidence. Through post-hoc grounding, each response claim is aligned with contextual evidence that either supports or contradicts it. Building on claim-level grounding results, eTracer not only enables users to precisely trace responses back to their contextual source but also quantifies response faithfulness, thereby enabling the verifiability and trustworthiness of generated responses. Experiments show that our claim-level grounding approach alleviates the limitations of conventional grounding methods in aligning generated statements with contextual sentence-level evidence, resulting in substantial improvements in overall grounding quality and user verification efficiency. The code and data are available at https://github.com/chubohao/eTracer.