ViHERMES: A Graph-Grounded Multihop Question Answering Benchmark and System for Vietnamese Healthcare Regulations

AI summary

Overview

Research area: Natural Language Processing — retrieval-augmented generation, multihop question answering, graph-based retrieval, and low-resource legal/regulatory NLP for Vietnamese.

Technical level: Advanced. The paper assumes familiarity with RAG pipelines, dense/sparse retrieval, knowledge graphs, multi-agent LLM systems, and LLM-as-a-Judge evaluation.

Scope: The paper introduces ViHERMES, a benchmark of 1,560 test-set question–answer pairs requiring multihop reasoning over Vietnamese healthcare regulations, together with a structure-driven regulatory knowledge graph and a graph-aware multi-agent QA system evaluated against eight baselines.

What This Paper Is About

Healthcare regulations are legal documents organized into articles and clauses that constantly amend, replace, supplement, and cross-reference one another, so a correct answer often requires tracing chains of amendments across several documents rather than reading one passage. Existing Vietnamese QA datasets cover general law, education, bidding, labor law, or clinical/health-news content, but none explicitly target multihop reasoning over legally interdependent healthcare regulations. The goal of this work is to build such a benchmark and a retrieval system that respects legal structure, so that multihop regulatory QA in a low-resource language can be measured and improved in a controlled way.

Key Contributions

1. ViHERMES benchmark. The authors introduce ViHERMES (Vietnamese Healthcare Regulations–Multihop Reasoning Dataset), described as the first benchmark dataset for Vietnamese healthcare regulatory QA that explicitly targets multihop reasoning across legally interdependent documents, covering dependency patterns such as amendment tracing, cross-document comparison, and procedural synthesis.

2. Graph-inspired controlled data construction pipeline. They propose a pipeline that combines annotator/expert guideline design, corpus crawling and preprocessing from official sources such as the Vietnamese Ministry of Health portal, semantic clustering with K-Means (Spherical K-Means) over ℓ2-normalized embeddings, constrained tuple sampling, and LLM-based generation with structural validation — producing evidence-grounded answers with explicit reasoning annotations.

3. Structure-driven Regulatory Knowledge Graph (SRKG). They model legal texts as regulatory nodes (document, clause, article) with deterministic identifiers and two relation families: structural relations (Has_Article, Has_Clause) and legal semantic relations (Amends, Replaces, Supplements, Refers_To), induced by rule-based pattern matching over standardized legal expressions.

4. Graph-aware multi-agent QA system. They present a seeded retrieval and propagation framework inside a multi-agent system with four agents — Interpreter, Pathfinder, Auditor, and Conductor — and show it outperforms retrieval-based and graph-based baselines on ViHERMES while maintaining competitive latency.

Main Findings

• Test set composition: The ViHERMES test split contains 1,560 samples distributed evenly across hop levels 1 through 5 (312 samples each), with vocabulary growing from 2,232 (1 hop) to 3,707 (5 hops) and 5,703 overall.
• Lengths scale with hop count: Question length (min/mean/max) grows from 48/113.2/215 at 1 hop to 194/445.9/977 at 5 hops; answer length from 58/181.8/504 to 101/889.9/1938; context length from 195/1185.1/4257 to 1115/6451.6/18878. Across all hops, question length is 48/304.2/977, answer length 58/548.8/1938, and context length 195/3953.9/18878.
• Best overall performance by the proposed system: It achieves F1 0.8334, LLM Judge 0.7554, and Recall@5 0.8461, the best of all evaluated systems.
• Progression across baseline families: Naive RAG improves from BM25 (F1 0.3076, Judge 0.2027, Recall@5 0.2617) to Dense (0.3289 / 0.2433 / 0.3241) to Hybrid (0.4127 / 0.3324 / 0.3989). IRCoT reaches 0.4835 / 0.3751 / 0.4254. Graph-based methods are stronger: MiniRAG 0.5429 / 0.4856 / 0.5083, RAPTOR 0.5941 / 0.5783 / 0.5563, LightRAG 0.7855 / 0.6756 / 0.7256, and HippoRAG 2 0.8023 / 0.7332 / 0.8032.
• Margin over the strongest baseline: The proposed system improves F1 by 3.1 points over HippoRAG 2 (0.8334 vs. 0.8023).
• Ablation — Interpreter matters most: Removing the Interpreter drops F1 from 0.8334 to 0.6540 (LLM Judge 0.5434, Recall@5 0.6134), the largest degradation observed.
• Ablation — Auditor supports reliability: Removing the Auditor lowers LLM Judge from 0.7554 to 0.6823 (F1 0.8150, Recall@5 0.8267).
• Ablation — Pathfinder replacement hurts: Replacing SRKG-based seeded retrieval and relation-aware propagation with standard dense–sparse retrieval over flat text units yields F1 0.7734, LLM Judge 0.6927, Recall@5 0.7955.
• Efficiency trade-off: The proposed system has average latency of 14.7415 s, comparable to RAPTOR (14.2325 s) and faster than HippoRAG 2 (22.0278 s) and LightRAG (17.3236 s), while using fewer graph tokens (3,397,126) than LightRAG (3,684,343) and HippoRAG 2 (3,966,963).
• Larger graph, fewer context tokens: The SRKG has 3,727 nodes and 12,540 edges — more than MiniRAG (2,988 / 4,323), RAPTOR (3,352 / 7,234), LightRAG (3,569 / 9,332), and HippoRAG 2 (2,988 / 11,183) — yet the system averages 4,236.4620 inference tokens.

Methodology in Plain English

The authors first assembled a corpus of Vietnamese healthcare regulations by crawling official portals and splitting documents along their legal hierarchy into context units, each stored as an ⟨id, title, text⟩ triple. A small team of annotators worked with a domain expert to define guidelines for what counts as a valid multihop question.

To generate questions rather than hand-write them all, each context unit was embedded into a vector, normalized, and grouped with K-Means clustering. The context nearest each cluster centroid was kept as a representative. Because the clusters capture topical similarity, this acts as an implicit similarity graph without needing to extract legal relations automatically. For a target hop count h, the system sampled h contexts under two constraints: every context must come from a different document title (forcing cross-document reasoning), and overlap between any two sampled tuples is bounded by a threshold τ. An LLM then wrote a question and answer that must draw on all h contexts. Outputs were structurally validated for format, hop-level consistency, and explicit reasoning, and borderline cases were reviewed by annotators.

Separately, for retrieval, the authors built the SRKG: legal units become nodes tagged with document ID, promulgation date, issuing authority, and legal status, and edges come from rule-based matching on standard legal phrases (for example, "sửa đổi Điều X" for amends, "bổ sung Khoản Y" for supplements, "căn cứ Điều Z" for refers to). Missing referenced units get placeholder nodes that resolve when the document is later ingested.

At query time, the Pathfinder scores nodes with a weighted blend of dense embedding similarity and sparse lexical evidence (such as BM25), keeping the top K as seeds. It then expands along edges in three controlled flows: validity tracing, which recursively follows amendment-type edges to a terminal node that has no further amendment links and is treated as legally applicable; contextual supplementation, which follows reference edges only one hop to avoid drift; and provenance retrieval, which pulls upstream document metadata. The final bounded context goes to the Conductor for answer generation, while the Auditor checks whether generated claims are grounded and the Conductor abstains or asks for clarification when grounding is insufficient.

All systems were evaluated with the same backbone — GPT-4o-mini for generation, OpenAI text-embedding-3-small for embeddings, and GPT-4o for LLM-as-a-Judge — using default hyperparameters for baselines.

Why This Matters

Regulatory compliance is a domain where a linguistically fluent but legally outdated answer is actively harmful, and this paper makes the case that flat passage retrieval cannot resolve questions whose answers depend on which provision is currently in force. By publishing both the benchmark and the system, the work gives researchers in low-resource languages a way to measure multihop legal reasoning instead of relying on English-centric or general-domain benchmarks.

Real-world applications:

• Healthcare facility compliance assistants. The example query in the paper asks which obligations a clinic must meet regarding transitional service pricing, baseline quality maintenance, and licensing — precisely the kind of joined-up answer a compliance officer needs.
• Regulatory change monitoring. Amendment-chain tracing can determine whether a provision a hospital is relying on has been amended, replaced, or supplemented.
• Patient-facing billing and entitlement explanations. Transitional pricing rules (for example, continuing the old price until discharge) require combining provisions from multiple documents.
• Legal drafting and review support. Identifying cross-references and dependencies across regulatory texts helps drafters check consistency before promulgation.

Industry relevance: the multi-agent design (routing, retrieval, verification, orchestration) maps onto production compliance tooling where abstention is preferable to a hallucinated legal claim. The reported latency of roughly 14.7 seconds is slower than naive RAG's 4.1 to 9.2 seconds but in the range of other graph methods, and the system uses fewer context tokens than LightRAG and HippoRAG 2 despite a larger graph — a relevant cost signal for deployments. The authors also note the structure-driven formulation generalizes to other regulatory domains with hierarchical organization and formal cross-document dependencies.

Future Directions

• Constraining propagation volume. The conclusion explicitly names limiting how much context propagation expands as a goal for improved robustness.
• Enriching temporal validity. The authors call for richer temporal modeling so answers reflect the version of a provision in force at a given time.
• Extending beyond healthcare. The structure-driven formulation is proposed as applicable to other regulatory domains with hierarchical organization and formal cross-document dependencies.
• Benchmark and construction-method transfer. Open questions remain about how far the clustering-and-tuple-sampling pipeline, and the SRKG relation induction rules, transfer to other low-resource languages and legal traditions. The paper does not report train-split statistics (only the test split is characterized), so the training/evaluation balance is a further open detail.

Target Audience

This paper is most useful to NLP researchers working on retrieval-augmented generation, multihop QA, and graph-based retrieval; to researchers building datasets and systems for Vietnamese or other low-resource languages; and to practitioners building legal or regulatory compliance tools who need to reason over amendment chains rather than single passages. Readers already comfortable with RAG, knowledge graphs, and multi-agent LLM pipelines will get the most from the methodology and ablation sections, while those interested mainly in dataset design will find the construction pipeline and the test-set statistics the most directly reusable part.

Authors’ abstract

Question Answering (QA) over regulatory documents is inherently challenging due to the need for multihop reasoning across legally interdependent texts, a requirement that is particularly pronounced in the healthcare domain where regulations are hierarchically structured and frequently revised through amendments and cross-references. Despite recent progress in retrieval-augmented and graph-based QA methods, systematic evaluation in this setting remains limited, especially for low-resource languages such as Vietnamese, due to the lack of benchmark datasets that explicitly support multihop reasoning over healthcare regulations. In this work, we introduce the Vietnamese Healthcare Regulations-Multihop Reasoning Dataset (ViHERMES), a benchmark designed for multihop QA over Vietnamese healthcare regulatory documents. ViHERMES consists of high-quality question-answer pairs that require reasoning across multiple regulations and capture diverse dependency patterns, including amendment tracing, cross-document comparison, and procedural synthesis. To construct the dataset, we propose a controlled multihop QA generation pipeline based on semantic clustering and graph-inspired data mining, followed by large language model-based generation with structured evidence and reasoning annotations. We further present a graph-aware retrieval framework that models formal legal relations at the level of legal units and supports principled context expansion for legally valid and coherent answers. Experimental results demonstrate that ViHERMES provides a challenging benchmark for evaluating multihop regulatory QA systems and that the proposed graph-aware approach consistently outperforms strong retrieval-based baselines. The ViHERMES dataset and system implementation are publicly available at https://github.com/ura-hcmut/ViHERMES.