UCVE: Graph Structured Retrieval for Compliance Validation

AI summary

Overview

Research area: Natural Language Processing / legal NLP, specifically retrieval over structured obligation spaces using graph signal processing and reinforcement learning (Capital One, AI Foundations).

Technical level: Advanced. The paper assumes familiarity with spectral graph theory (graph Laplacians, low-pass filtering), retrieval baselines, and reinforcement learning for edge weighting.

Scope: A domain-agnostic framework, UCVE, that builds a Contextual Requirement Graph (CRG) over a domain's requirement space and performs recall-oriented retrieval via spectral reranking over an RL-weighted graph Laplacian, evaluated on three compliance domains.

What This Paper Is About

Compliance review requires deciding, for every obligation in a predefined requirement space, whether a document satisfies it, violates it, or does not address it. The authors frame this as a coverage problem: existing retrieval and classification systems treat each requirement independently, so they miss violations whose evidence is spread across clauses or whose wording diverges from the regulatory text. UCVE addresses this by representing the requirement space itself as a graph, learning which structural relationships between obligations predict co-occurrence, and propagating relevance signals across that graph to raise document-level recall.

Key Contributions

1. Problem formulation. Legal compliance validation is recast as structured inference over an interconnected obligation graph under asymmetric error costs, where a missed obligation carries regulatory, litigation, or reputational risk and a false alarm costs only additional human review.

2. The Contextual Requirement Graph (CRG). A weighted, undirected graph built once per domain, independent of any target document. Nodes hold a requirement identifier, category label, requirement text, and a natural-language validation question derived from the requirement, plus programmatically instantiated positive and negative class definitions that make each verdict auditable.

3. RL-learned edge weights combined with spectral reranking. Six metadata edge types (seven types total, including cosine similarity) are weighted by reinforcement learning using an ε-greedy breadth-first traversal rewarded by violation recovery, rather than hand-tuned per domain. Traversal scores are then projected onto the bottom eigenvectors of the sparsified, RL-weighted graph Laplacian to smooth scores across learned requirement communities.

4. A document-level recall evaluation protocol across three structurally distinct compliance domains (privacy policies, consumer contract fairness, sustainability disclosures), which the authors state no prior compliance validation system has adopted, together with a full-pipeline check passing retrieved nodes to an LLM validator.

Main Findings

• Highest recall in all three domains at matched-K. CRG + Spectral reaches 0.792 recall on OPP-115 (K = 14), 0.860 on CLAUDETTE (K = 16), and 0.7551 on ESRS (K = 17), versus flat cosine at 0.748, 0.806, and 0.7227 respectively. The abstract reports an average 4.3% matched-K recall lift over flat cosine.

• Gains over strong graph baselines are smaller but consistent. CRG + Spectral beats the Leiden community-detection baseline by +2.6% on OPP-115 and +1.9% on CLAUDETTE, and outperforms flat cosine by +3.2–5.4% across all three domains.

• The two components are conditionally dependent. Spectral reranking on the RL-weighted graph is the only configuration that improves recall at matched-K: +3.4% on OPP-115, +4.1% on CLAUDETTE, and 3.2% on ESRS. Spectral reranking on a cosine-only graph gives 0.759 on OPP-115 and 0.848 on CLAUDETTE, while BFS on the RL-weighted graph reaches 0.710 on OPP-115 and 0.806 on CLAUDETTE, trailing spectral reranking by 8.2% and 5.4%. Neither component alone suffices.

• The mechanism is topological. On OPP-115, zeroing the RL metadata weights changes 8 edges in the k = 3 sparsified graph, enough to disrupt the community structure the low-pass filter relies on. On CLAUDETTE this produces 54 topology differences and a −1.2% recall loss; on ESRS removing all metadata costs 2.6% recall.

• Learned weights differ by domain. OPP-115 gives meaningful weight to hierarchical metadata (same_category 0.549, same_sub_attribute 0.556). CLAUDETTE weights are near-zero across all types (0.037–0.041), which the authors attribute to high edge density (99.8%) leaving metadata with little to add. On ESRS, same_category is set to a fixed 0.500 boost rather than RL-learned because of near-zero training signal, with same_framework at 0.086.

• Discrete clustering degrades faster than spectral smoothing. Leiden falls from 0.841 on CLAUDETTE to 0.6745 on ESRS (82 nodes, 1818 queries), while CRG + Spectral holds at 0.7551. Spectral reranking operates at node level rather than fixing nodes to clusters.

• The cross-encoder's advantage shrinks with graph size. It scores 0.781 on OPP-115 and 0.825 on CLAUDETTE, but drops to 0.465 on ESRS and requires 2.6× its matched-K budget there to reach CRG + Spectral's recall.

• Hyperparameters are per-domain but stable. OPP-115 and CLAUDETTE share k = 3, n = 5, α = 0.8; ESRS needs k = 5, n = 12, α = 0.5. Configurations are stable over k ∈ [1,10], n ∈ [2,15], and α ∈ [0,1].

• End-to-end validation quality is the bottleneck, not retrieval on one domain. With Llama-3.3-70B, pipeline recall sits about 25% below the retrieval ceiling on OPP-115 and 23% below on ESRS, but only 4–5% below on CLAUDETTE. ESRS precision collapses to 0.204, attributed to a roughly 3% positive rate, with variation across the thirteen ESRS pillars.

• Recall, not precision, is the operating target. Precision at matched-K is low across all methods (for example 0.185 for CRG + Spectral on OPP-115 versus 0.197 for the cross-encoder), which the authors argue is acceptable because retrieval recall sets the ceiling on end-to-end violation detection.

Methodology in Plain English

The work starts from the observation that legal obligations are not independent facts: a data retention clause bears on storage-limitation and third-party-sharing obligations, and a jurisdiction clause usually co-occurs with a choice-of-law clause. So instead of scoring each requirement against a document alone, the authors build a map of the requirements and let relevance flow between neighbours.

Building the map. Each requirement becomes a node carrying its text, category, and a plain-language question to be answered later by a validator. Nodes are connected when their text embeddings are cosine-similar above 0.5, using the all-mpnet-base-v2 encoder (chosen over bge-large-en-v1.5 for higher or comparable recall across the domains). Dataset-provided metadata then adds six more relationship types, such as sharing a category or a sub-attribute. Each edge's total strength is the sum, over its relationship types, of a raw strength times a learned multiplier. Metadata can only ever add weight, so the semantic backbone is never weakened.

Learning which relationships matter. Rather than hand-tuning the metadata multipliers per domain, the system explores the graph from a document's best-matching requirement for 15 breadth-first steps, using ε-greedy exploration over 30 epochs with a learning rate of 0.05 and ε decaying from 0.15 to 0.02. Edges that lead to a true violation get credit that decays with how late they were found, edges within three hops of a violation get a small positive credit, and the rest get a small negative credit. Credit is split across an edge's active types in proportion to their contribution, and weights are clamped to [0.0, 3.0] so they cannot run away or dominate the similarity backbone.

Retrieving. Because the graph is effectively dense, ordinary propagation would collapse toward flat cosine ranking, so the graph is trimmed to each node's k strongest edges. The system then computes the unnormalized Laplacian L = D − W, keeps the bottom n eigenvectors, and blends a projection of the similarity scores onto that basis with the raw scores: α = 0.8 by default, giving 80% weight to the smoothed community signal. Smoothing lifts obligations whose wording does not match the document but whose neighbourhood does, and suppresses superficially similar requirements sitting in weak clusters.

Evaluating. All graph methods are scored at matched-K — each baseline returns exactly as many nodes as UCVE returns on average for that dataset — so differences reflect ranking quality rather than retrieval budget. Six baselines are compared: BM25, flat cosine, a cross-encoder (ms-marco-MiniLM-L-6-v2), Personalized PageRank, Leiden community retrieval, and heat-kernel diffusion at t = 0.5.

Why This Matters

Impact on research. The paper's central claim is that the value of spectral graph retrieval depends on whether edge weights encode task-relevant co-occurrence, not on graph topology alone. That is a transferable principle for any setting where requirements, concepts, or entities exhibit learnable co-relevance that surface similarity misses. It also supplies a document-level recall protocol for compliance validation, where prior work reported clause-level classification metrics.

Real-world applications:

• Privacy policy auditing against regimes such as GDPR and CCPA, where OPP-115 supplies the requirement hierarchy.
• Consumer contract review for potentially unfair terms under the EU Unfair Contract Terms Directive (93/13/EEC), via CLAUDETTE.
• ESG and sustainability disclosure checking against the EU Corporate Sustainability Reporting Directive, via ESRS.
• Internal policy validation and model governance documentation, which the authors note typically involve requirement spaces of roughly 50–150 nodes, within or near the evaluated range.

Industry relevance. The asymmetric cost structure matches how compliance teams actually work: a missed obligation is expensive and a false positive is merely another review. Because the graph is built from explicit requirement definitions rather than task-specific outputs, and because each node carries an auditability-supporting validation question, the architecture is portable across jurisdictions without exhaustive reannotation. The work comes from Capital One's AI Foundations.

Future Directions

• Scaling beyond small obligation spaces. The evaluated domains span 24 to 82 nodes. ESRS, the largest and densest graph, has a lower spectral signal (λ₂ = 0.04) than OPP-115 (λ₂ = 0.19) and correspondingly smaller gains; the authors note that policy-gradient traversal becomes the natural choice over exhaustive ε-greedy BFS when obligation spaces grow.
• Addressing sparse rewards at scale. Larger compliance corpora have sparser positive labels per node, which weakens the RL reward signal during graph construction. Reward shaping, curriculum learning, and semi-supervised edge initialization are named as directions.
• Automating graph construction from regulatory text. The current CRG depends on domain experts and dataset-provided metadata schemas; building the obligation graph directly from raw regulatory text would extend UCVE to domains lacking structured schemas.
• Multi-document settings, where obligations must be satisfied across a corpus rather than a single document.
• Open questions the paper leaves unresolved: comparing against LLM-based query expansion, which the authors note is orthogonal and composable; and pillar-specific threshold calibration or abstention to address the ESRS precision collapse.

Target Audience

Researchers and practitioners in legal NLP and retrieval who are comfortable with graph signal processing and reinforcement learning, and who care about high-recall structured retrieval rather than clause classification. It is also relevant to compliance, privacy, and ESG reporting teams evaluating whether automated obligation-coverage tooling is viable, and to applied scientists interested in how learned edge weights reshape a graph Laplacian's community structure for downstream retrieval. Readers seeking introductory material on legal NLP or on spectral methods will find the paper assumes that background rather than supplying it.

Authors’ abstract

Structured compliance validation requires determining whether every obligation in a predefined requirement space is satisfied, violated, or unaddressed by a given document. This can be described as a coverage problem that existing retrieval and classification systems, by treating requirements as independent, are not designed to solve. We introduce \textbf{\ac{UCVE}}, a domain-agnostic framework that represents compliance requirement spaces as a \ac{CRG} and performs retrieval via spectral reranking over its reinforcement learning-weighted Laplacian. Edges between requirement nodes are learned via reinforcement learning with a traversal reward grounded in violation detection, reshaping the graph's community structure such that low-pass spectral filtering distinguishes relevant from irrelevant obligation clusters. Evaluating across three compliance domains, \ac{UCVE} averages 4.3\% matched-$K$ recall lift over flat cosine retrieval and consistently outperforms strong retrieval and graph-based baselines.