SUMMPILOT: Bridging Efficiency and Customization for Interactive Summarization System

AI summary

Overview

Research area: Interactive and customizable text summarization, combining natural language processing (LLM-based summarization) with human-computer interaction design.

Technical level: Intermediate. The paper assumes familiarity with summarization concepts such as abstractive summarization, factual consistency, and coreference resolution, but its system components are described in accessible terms.

Scope: The paper presents SummPilot, an LLM-based interactive summarization system (built on GPT-4o) that offers an automatic Basic Mode and a controllable Advanced Mode with semantic graphs, entity clustering, and explainable evaluation, validated through two small user studies.

What This Paper Is About

Automatic summarization systems are efficient but generally ignore what an individual user actually wants from a summary, while existing interactive summarization systems struggle to represent relationships between pieces of content and give users no indication of how their input changed the output. SummPilot addresses both gaps by letting users explore documents through semantic graphs built from relational triples, include or exclude specific content, and receive an explainable evaluation of the generated summary. The goal is a summarization system that is both efficient (automatic) and customizable (interactive), including for multi-document inputs.

Key Contributions

1. A dual-mode summarization system: SummPilot provides a Basic Mode for automatic summarization and an Advanced Mode for user-centric, interactive control over the content and style of the summary, both powered by a large language model (specifically GPT-4o).
2. Graph-based content representation for multi-document summarization: A relation extraction module identifies relational triples, visualized as a semantic graph with directed edges and node sizes that emphasize key entities, while entity clustering groups different mentions of the same entity so users can control their information needs precisely.
3. An explainable evaluation mechanism: Generated summaries are scored on compression, coverage, and factual consistency, where the consistency component is adapted from FactScore (Min et al. 2023) by decomposing summaries into atomic facts and verifying them against source documents; unverified facts are flagged as "possible errors" in the interface.
4. Empirical validation via two user studies: A system usability test using the System Usability Scale (SUS) and a custom usefulness questionnaire, plus a comparative usefulness test using the USE questionnaire and problem-solving tasks comparing Basic Mode against Advanced Mode.

Main Findings

• SUS score of 86.3, rated "excellent": Across six participants, the SUS scores were 92.5, 90, 75, 80, 92.5, and 87.5, giving an average of 86.3, which the paper classifies as "excellent" based on established thresholds (Will 2021).
• Relation extraction was rated perfectly intuitive: In the usefulness questionnaire, "Relation Extraction table is intuitive to include or exclude specific relationships" received 5.0 (0.0), the highest score reported.
• Semantic graph and error flagging were well received: The graph's help in understanding article content scored 4.4 (0.8), and "Possible Error helps to evaluate the generated summary" scored 4.6 (0.5), while entity clustering received the lowest usefulness rating at 3.8 (0.7).
• Control over inclusion and exclusion worked well: Both "The generated summary effectively includes the information I requested" and "…effectively excludes the information I requested" scored 4.8 (0.4), with summary satisfaction also at 4.8 (0.4) and trustworthiness at 4.4 (0.8).
• Advanced Mode scored higher on usefulness and satisfaction: In the USE questionnaire, Advanced Mode received Usefulness 4.3 (0.3), Satisfaction 4.5 (0.3), Understanding the Article 4.3 (0.8), and Adjusting the Summary 4.8 (0.4); Basic Mode, described as simpler with minimal user intervention, scored higher on Ease of Use 2.9 (0.3) and Ease of Learning 2.6 (0.1).
• Problem-solving accuracy was identical, but the reported times were not: Both modes achieved a 90% correct rate on true/false question tasks, with Basic Mode reported at 3:33 and Advanced Mode at 5:02. The paper's text states that Advanced Mode's interactive features "facilitated faster task completion compared to automated summaries"; the times presented in Table 7 appear to show Advanced Mode taking longer, and the paper does not explain this discrepancy.

Methodology in Plain English

The system runs on GPT-4o and chains several modules together. When a user provides multiple documents, an automatic summarization module produces an abstractive summary. A relation extraction module then pulls out subject-relation-object triples, which are drawn as a semantic graph so users can see how information is connected; for example, the triples relating Tom, Jane, and "Tom's wife" are merged into a single clustered triple. Entity clustering then performs coreference resolution, grouping noun phrases that refer to the same entity and choosing a representative entity by frequency for display. In Advanced Mode, user commands — structured checkboxes on triples or open-form text — are folded into prompts that tell the model what to include or exclude, allowing iterative refinement of the summary. After generation, the summary is evaluated on compression, coverage, and factual consistency; the consistency check breaks the summary into atomic facts and asks the model to judge each against the source documents, flagging unverified facts as possible errors so users can fix them.

Evaluation proceeded in two user studies. The first recruited six participants (following Nielsen's guidance that six evaluators suffice for prototype evaluation) with diverse backgrounds. Each was randomly assigned two documents from Multi-News (Fabbri et al. 2019) and given a scenario in which they acted as an intern reporter analyzing a news topic. They completed the SUS and a custom nine-item usefulness questionnaire. The second study recruited six new participants who summarized two randomly selected Multi-News articles, one per mode, then completed the standard USE questionnaire and rated their preference between Basic Mode (closer to 1) and Advanced Mode (closer to 5), following SummHelper (Slobodkin et al. 2023). Participants also solved true/false problem-solving tasks on hypothetical news articles generated with ChatGPT to measure speed and accuracy. Sessions lasted 45 minutes each, were conducted via Zoom, and used a think-aloud approach (Van Someren et al. 1994). The interface uses the Bootstrap framework and a Python Flask backend, and the paper notes SummPilot can be deployed using only a CPU. A demo video is available at https://youtu.be/7ZUBIyqPpbs.

Why This Matters

The paper targets a practical middle ground: automatic summarizers are fast but impersonal, and interactive summarizers offer control but often obscure how content relates or how user actions changed the result. SummPilot's contribution is combining graph-based content relationships with an explainable evaluation loop, which the authors argue is especially valuable for lengthy or multi-topic documents and for multi-document summarization, where users must integrate information from several sources.

Real-world applications identified or implied by the paper:

• Journalism: The user study scenario casts participants as intern reporters analyzing news topics and writing summaries; the authors state the system is particularly beneficial for reporters who need to extract essential information.
• Education: The authors explicitly name students as a beneficiary group for extracting essential information.
• Business document handling: The paper states SummPilot provides a foundation for efficiently summarizing complex business documents.
• News article summarization: The reported advantage the paper claims is similar, but the paper does not report different numbers; the numbers in the paper are identical, and the paper reports them as such.

Industry relevance: The system requires only a CPU for deployment and uses a commercial LLM (GPT-4o) rather than bespoke trained models, which lowers the barrier to building customizable summarization interfaces. Factual-consistency flagging is directly relevant for any workflow where an unreliable summary carries cost, and the mode split (automatic vs. interactive) maps onto how products typically trade off convenience against control.

Future Directions

• Multilingual support, which the authors explicitly name as future work.
• Integration of multimodal LLMs to handle data types such as images and videos, with the aim of enhancing user control and improving human-computer interaction.
• Resolving the reported performance tension: The paper reports identical 90% accuracy for both modes and shows Advanced Mode at 5:02 versus Basic Mode at 3:33, yet describes the interactive mode as enabling faster completion; a follow-up would need to clarify or retest this claim.
• Scaling the evidence: Both user studies used six participants each, and the paper does not report a limitation section or larger-scale evaluation, leaving open whether the usability and preference results generalize.

Target Audience

This paper is most useful to researchers and practitioners in summarization, NLP system design, and human-computer interaction who are building interactive or customizable text-generation tools, particularly those working with multi-document inputs. It also suits product and engineering teams evaluating whether graph-based content exploration and factual-consistency checking are worth adding to summarization interfaces, and to readers interested in how LLM prompting can support human-in-the-loop control. Readers looking for large-scale quantitative benchmarks or a limitations analysis will not find them here, as the paper reports small user studies and no dedicated limitations section.

Authors’ abstract

This paper incorporates the efficiency of automatic summarization and addresses the challenge of generating personalized summaries tailored to individual users' interests and requirements. To tackle this challenge, we introduce SummPilot, an interaction-based customizable summarization system. SummPilot leverages a large language model to facilitate both automatic and interactive summarization. Users can engage with the system to understand document content and personalize summaries through interactive components such as semantic graphs, entity clustering, and explainable evaluation. Our demo and user studies demonstrate SummPilot's adaptability and usefulness for customizable summarization.