sui-1: Grounded and Verifiable Long-Form Summarization

AI summary

Overview

Research area: Natural Language Processing, specifically long-document summarization with verifiable attribution.

Technical level: Intermediate. The paper is readable without deep machine learning background, but assumes familiarity with fine-tuning concepts such as LoRA, quantization, and LLM-as-a-judge evaluation.

Scope: This paper introduces sui-1, a 24B parameter summarization model fine-tuned to produce abstractive summaries with inline citations that trace each claim back to a specific source sentence.

What This Paper Is About

Large language models often produce fluent summaries containing facts or claims that cannot be verified against the source document, which is a serious problem in compliance-sensitive fields such as government and legal analysis. The authors build sui-1, a model that writes summaries with inline citation tags so every claim can be traced to the exact source sentence. The goal is a model that is both grounded (faithful to the source) and verifiable (the user can check each claim quickly), and that works on documents far longer than a typical context window.

Key Contributions

1. A citation-grounded summarization model. sui-1 is a 24B parameter model built on Mistral-Small-3.2-24B-Instruct that generates abstractive summaries with inline citations pointing to tagged source sentences. It processes documents up to 100K tokens in a single pass and supports iterative processing for texts exceeding 2 million tokens.

2. A synthetic data generation pipeline with multi-stage verification. The pipeline combines chain-of-thought prompting, sentence-level XML tagging with unique identifiers, citation verification, and quality filtering to produce over 22,000 high-quality training examples across five languages from parliamentary documents, web text, and Wikipedia.

3. Evidence that task-specific training beats scale for this task. sui-1 outperforms all tested open-weight baselines, including models with 3x more parameters, and reaches 84% overall accuracy on LLM-as-a-judge evaluation versus 43–56% for baselines.

4. Public release of weights and data. Full-precision and FP8 quantized model weights are released under Apache 2.0, alongside the training dataset, an example script, and an interactive demo.

Main Findings

• Overall accuracy: sui-1 scores 0.842 overall on the LLM-as-a-judge evaluation, compared to 0.427–0.556 for open-weight baselines and 0.891 for the reference model (a frontier closed-source model with extended reasoning).

• Format compliance is the largest gain: sui-1 reaches 0.895 on format compliance versus 0.137–0.411 for baselines. The authors call this their key finding.

• Coverage leads among open-weight models: sui-1 achieves the highest coverage score (0.600) of all open-weight models, second only to the reference model (0.705). Baselines range from 0.042 to 0.368.

• High factual accuracy and specificity: sui-1 scores 0.905 on factual accuracy and 0.979 on specificity, close to the reference model's 0.958 and 0.989.

• Scale alone does not win: Llama-3.3-70B-Instruct scores lowest overall (0.427) despite having roughly 3x the parameters, which the authors use to argue that task-specific training outweighs scale for citation-grounded summarization.

• FP8 quantization costs little: The FP8 variant scores 0.811 overall, a 3.7% degradation, while its format compliance (0.926) actually exceeds the full-precision model's 0.895. Quantization reduces model size from approximately 48GB to 24GB, a 50% memory reduction.

• Verification pass rate: 95.2% of generated examples passed the tag verification stage, which confirms all citations exist in the source.

• Baselines fail on citation structure: The qualitative comparison shows baseline models producing generic filler with no citations, while sui-1 attaches XML tags directly after supported claims. Baselines either omit citations entirely or produce malformed tags that do not match source sentences.

Methodology in Plain English

The authors start with source documents from three places: the German Parliamentary Documentation System (DIP), long German texts from Common Crawl via OSCAR, and multilingual Wikipedia extracts. Documents range from short announcements to reports exceeding 50,000 words.

Every sentence in a source document gets a unique 8-character hexadecimal identifier derived from an MD5 hash of the sentence text, wrapped in XML tags like <a3f5e823>text</a3f5e823>. Because the tags are deterministic and language-agnostic, the model can cite a sentence without quoting it verbatim, and tags give no useful positional shortcuts. Eight hex characters provide roughly 4.3 billion unique tags.

A frontier closed-source teacher model then generates summaries through a five-stage pipeline: sentence tagging, prompt construction, LLM generation, citation verification, and quality filtering. The generation prompt contains 16–18 rules plus chain-of-thought reasoning, requiring each tag to appear exactly once and each citation to immediately follow the claim it supports. Documents under 30K tokens are handled in one pass; longer documents are chunked into roughly 15K-token pieces, summarized independently, and merged with citations preserved.

Custom instructions are generated separately for each document, split into positive instructions (focus areas, formatting), adversarial instructions (requests for information that is not in the source), and format instructions. To keep inference flexible, the authors relax rigid constraints in the training data, for example turning "Summarize in exactly 5 bullet points. Only bullets, no introduction or conclusion" into "Summarize in 5 bullet points."

Quality control filters out bad examples: tag verification (95.2% pass), quality annotation for reasoning coherence and citation distribution, and specificity checks that penalize generic filler. Citation distribution is measured with an evenness score for tag spacing uniformity, and examples in the bottom 15th percentile or with excessive uncited gaps are dropped. A second LLM pass rewrites the reasoning from third person to first person.

Fine-tuning uses LoRA with rank 16 (alpha 32, dropout 0.05) on Mistral-Small-3.2-24B-Instruct, targeting the q, k, v, o, gate, up, and down modules. Training runs for 2 epochs with a learning rate of 1.5e-4 on a cosine schedule, warmup ratio 0.02, effective batch size 16, bfloat16 precision, and sequence length 99,968 tokens, using the Axolotl framework with Flash Attention 2 and gradient checkpointing on four NVIDIA H100 GPUs with context parallelism.

Evaluation uses LLM-as-a-judge via the elluminate platform on 225 held-out examples, scoring binary pass/fail on five criteria: factual accuracy, coverage, specificity, format compliance, and instruction following.

Why This Matters

Impact on research: The paper provides a concrete demonstration that task-specific synthetic data and training can beat raw scale for a structured generation task. It also contributes a reusable recipe—deterministic sentence tagging plus automated verification—that other researchers can apply to attribution problems beyond summarization, and it frames the work as internal grounding, distinguishing it from retrieval-augmented approaches that depend on external retrieval infrastructure.

Real-world applications:

• Government and legislative analysis, where staff need to verify that a summary of a bill or committee report accurately reflects the source.
• Legal document review, where each claim in a summary must be traceable to a specific clause or filing sentence.
• Compliance and audit workflows, where reviewers must confirm that reported facts come from the underlying record.
• Long-document processing in European multilingual settings, given the model's coverage of German, English, French, Italian, and Spanish.

Industry relevance: The released weights are under Apache 2.0, enabling commercial use, and the FP8 variant is positioned for production deployment where memory efficiency matters. The interactive demo lowers the barrier to trying the model.

Future Directions

• Broaden language and domain coverage. The authors state as a limitation that training data derives primarily from German-language sources (parliamentary documents, web text, and Wikipedia), which may limit generalization to other languages and specialized domains.
• Validate with human evaluation. The current results rest on LLM-as-a-judge scoring, which the authors note may exhibit systematic biases compared to human evaluation.
• Close the remaining gap to the reference model. sui-1 reaches 0.842 versus 0.891 for the frontier reference model, and the authors attribute part of the coverage gap (0.600 vs 0.705) to an intentional trade-off when users request constrained formats.
• Extend the chunk-and-merge pipeline at extreme lengths. The model supports iterative processing for texts exceeding 2 million tokens; how quality holds up at that scale is an open question the paper does not quantify.

Target Audience

Researchers and engineers working on summarization, attribution, and trustworthy language model outputs will benefit most, along with practitioners who need summaries that can be audited against a source. It is also useful for teams deciding whether to fine-tune a smaller open-weight model rather than rely on a larger general-purpose or closed model, and for anyone deploying multilingual long-document summarization in compliance-sensitive settings such as government, legal, or audit work.

Authors’ abstract

Large language models frequently generate plausible but unfaithful summaries that users cannot verify against source text, a critical limitation in compliance-sensitive domains such as government and legal analysis. We present sui-1, a 24B parameter model that produces abstractive summaries with inline citations, enabling users to trace each claim to its source sentence. Our synthetic data pipeline combines chain-of-thought prompting with multi-stage verification, generating over 22,000 high-quality training examples across five languages from diverse sources including parliamentary documents, web text, and Wikipedia. Evaluation shows sui-1 significantly outperforms all tested open-weight baselines, including models with 3x more parameters. These results demonstrate that task-specific training substantially outperforms scale alone for citation-grounded summarization. Model weights and an interactive demo are publicly available.