IL-PCSR: Legal Corpus for Prior Case and Statute Retrieval

AI summary

Overview

• Research area: Legal Natural Language Processing / information retrieval, specifically Indian legal text (statutes and prior cases).
• Technical level: Intermediate. The paper assumes familiarity with retrieval evaluation metrics (F1@k, MAP, MRR), lexical versus semantic retrieval, graph neural networks, and LLM prompting, but explains each method at a high level.
• Scope: The paper introduces IL-PCSR, a corpus linking Indian statutes and prior cases for the same set of 6,271 query judgments, and benchmarks lexical, semantic, GNN-based, ensemble, and LLM re-ranking approaches on statute retrieval and precedent retrieval.

What This Paper Is About

Law practitioners must find both the statutes that apply to a case and the prior cases that should be cited for it, but researchers have treated these two retrieval tasks separately, building different datasets and models for each. The authors build IL-PCSR, a corpus where the same query case is paired with a pool of statutes and a pool of precedents, so that a single model or pipeline can exploit the fact that similar cases tend to cite similar statutes. The goal is to test whether the two tasks reinforce each other and to establish strong baselines for both.

Key Contributions

1. A combined corpus: IL-PCSR (Indian Legal Corpus for Prior Case and Statute Retrieval), described as the first corpus to provide both prior cases and statutes for the same set of queries, in English for the Indian legal domain.
2. Extensive baselines: Experiments with lexical models (e.g., BM25 with varying n-grams), semantic models (SAILER, Event-GNN, Para-GNN), summary-based variants, and ensembles of lexical and semantic models for both tasks.
3. Two ways to exploit task dependence: (i) a joint multi-task GNN model, and (ii) a pipeline where an ensemble acts as a retriever and an LLM acts as a two-stage re-ranker, running LSR → PCR or PCR → LSR.
4. A human annotation study with law students to check whether cited candidates are a sufficient gold standard, plus analyses of how performance varies with candidate frequency.

Main Findings

• Lexical methods are weak on statute retrieval but strong on precedent retrieval: The best lexical variant for LSR (BM25, para-wise Max-All) reaches only 18.59% F1, while for PCR the best lexical model (BM25, 5-gram) reaches 33.29% F1.
• Semantic fine-tuning helps LSR far more than PCR: Para-GNN with summaries reaches 32.85% F1 on LSR (a relative improvement of nearly 77% over the best lexical LSR result), while the strongest semantic model on PCR (Para-GNN, full document, fine-tuned) reaches only 24.67% F1, a relative drop of about 26% versus lexical BM25.
• Summaries help LSR but hurt PCR: Summarization improved semantic models such as SAILER and Para-GNN on statutes, but reduced PCR performance, suggesting essential lexical cues are lost during compression.
• Ensembles consistently beat individual models: Combining lexical and semantic scores improved both tasks. Para-GNN (summaries) + BM25 with dynamic α gave 38.09% F1 on LSR and 37.46% F1 on PCR with separate fine-tuning.
• Multi-task training suffers from task interference: Joint fine-tuning did not outperform separate fine-tuning; LSR F1 dropped from 38.09% to 35.91% and PCR gains were marginal.
• Transfer learning between the tasks works better than joint training: Initializing one task's model from the other gave the best ensemble results, 39.44% F1 for LSR and 38.77% F1 for PCR, described as +20% and +16% improvements over the strongest standalone models.
• LLM re-ranking gives the best overall results: Using GPT-4.1 to re-rank the top-20 statutes and top-10 precedents raised LSR F1 from 39.44% to 45.29% and PCR F1 from 38.77% to 41.54% in the first stage.
• Cross-task conditioning improves both tasks further: In the second stage, feeding one task's outputs into the other gave 46.11% F1 for LSR (+1.8% over stage 1) and 43.31% F1 for PCR (+4.3% over stage 1).
• Stage-2 prompting helps mid- and low-frequency candidates: Grouping candidates by citation frequency into four groups, the authors find first-stage prompting improves over the ensemble across groups, while Stage-2 prompting especially benefits Group-2 and Group-4 for LSR and Group-2 for PCR.
• Human annotations broaden the gold standard but preserve model rankings: Annotators identified on average 5.0 statutes and 5.87 precedents per query, versus 4.57 statutes and 2.72 precedents actually cited. Under the union of citations and annotations, scores rose for all models, but Stage-2 LLM re-ranking still led (47.77% F1 for LSR and 46.93% F1 for PCR under the combined gold standard).

Methodology in Plain English

The authors collected roughly 20K publicly available English case judgments from IndianKanoon via its API, covering frequently cited Supreme Court and High Court decisions from 1950–2019. From this they derived a query set of 6,271 judgments, a statute pool of 936 articles/sections drawn from 92 Central (Federal) Acts, and a precedent pool of 3,183 prior cases. The gold standard for each query is the set of statutes and precedents actually cited in it, and those citations are masked out of the query text (along with person names, using InLegalNER) so models cannot simply read off the answer; placeholders such as [SECTION], [ACT], [PRECEDENT] and [ENTITY] are inserted instead.

On the modeling side, they tried lexical methods such as BM25 at several n-gram sizes, event-based variants (extracting subject-action-object triplets with SpaCy or with gemma-7b-it guided by SALI ontology definitions), and paragraph-level scoring with Max-All and Max-Sum aggregation. Semantic methods included SAILER (a BERT-based legal model), Event-GNN (graphs built from LLM-generated event triplets), and Para-GNN (graphs over paragraphs labeled by rhetorical role, such as Facts, Arguments, Rulings, as provided by IndianKanoon). They also used GPT-4o-mini to generate task-specific summaries, creating separate LSR-focused and PCR-focused summaries for each query. Lexical and semantic scores were combined with Z-score normalization and a weight α, chosen either by grid search or learned dynamically per query with a feed-forward network. Finally, GPT-4.1 was used as a two-stage re-ranker, where the second stage feeds positive precedents into statute re-ranking and positive statutes into precedent re-ranking.

Why This Matters

Impact on research: The paper argues that LSR and PCR, typically studied in isolation, are inherently connected and can be improved jointly, while also documenting that naive multi-task learning fails due to task interference. It provides a shared testbed and open-source release (via Hugging Face and GitHub) so future work can compare both tasks on the same queries.

Real-world applications:

• Assisting lawyers and paralegals in locating applicable statutes and precedents for a new case.
• Supporting judges and law clerks who need to survey relevant prior rulings.
• Improving legal search engines, such as the Indian legal search platform the corpus was drawn from.
• Broadening access to legal research in jurisdictions where manual precedent search is expensive and case volume is growing.

Industry relevance: Legal-tech companies and providers of legal research tools could use the corpus and the ensemble-plus-LLM-reranking pipeline as a starting point for products that recommend statutes and precedents. The finding that LLM re-ranking works well while first-stage retrieval still needs lexical methods also matters for deployment cost and architecture choices.

Future Directions

• Exploring re-ranking with diverse LLM families and alternative prompt designs, as the authors state only GPT-4.1 was tested.
• Investigating finer-grained supervision, such as paragraph-level or rhetorical-role-level signals, to better capture legal relevance.
• Testing whether the transfer learning and two-stage pipeline benefits hold with other models and approaches, since the authors note they observed these effects only with Para-GNN (summaries) + BM25 as the ensemble and GPT-4.1 as the re-ranker.
• Addressing dataset bias toward frequently cited candidates, as the candidate pools consist largely of prominent statutes and judgments, despite the inclusion of candidates cited in fewer than 5 queries and held-out candidates with zero citations.

Target Audience

Researchers and graduate students in legal NLP and information retrieval; developers building legal search or recommendation systems, particularly for common-law and Indian law contexts; and legal professionals or legal-tech teams interested in how automated statute and precedent retrieval is evaluated and what its current performance limits are.

Authors’ abstract

Identifying/retrieving relevant statutes and prior cases/precedents for a given legal situation are common tasks exercised by law practitioners. Researchers to date have addressed the two tasks independently, thus developing completely different datasets and models for each task; however, both retrieval tasks are inherently related, e.g., similar cases tend to cite similar statutes (due to similar factual situation). In this paper, we address this gap. We propose IL-PCR (Indian Legal corpus for Prior Case and Statute Retrieval), which is a unique corpus that provides a common testbed for developing models for both the tasks (Statute Retrieval and Precedent Retrieval) that can exploit the dependence between the two. We experiment extensively with several baseline models on the tasks, including lexical models, semantic models and ensemble based on GNNs. Further, to exploit the dependence between the two tasks, we develop an LLM-based re-ranking approach that gives the best performance.