Vichara: Appellate Judgment Prediction and Explanation for the Indian Judicial System

AI summary

Overview

Research area: Natural Language Processing for legal AI — specifically Legal Judgment Prediction (LJP) and Appellate Judgment Prediction (AJP) for the Indian judicial system, with a focus on explainability.

Technical level: Intermediate. The framework is built on prompt-based use of large language models plus a rhetorical-role classifier, so no deep modelling background is needed to follow the method, but familiarity with standard NLP evaluation metrics and LLM prompting helps.

Scope: The paper introduces Vichara, a six-stage framework that predicts whether an Indian appellate court granted or dismissed an appeal and produces a structured, IRAC-inspired explanation for that prediction, evaluated on two Indian legal datasets with four LLMs.

What This Paper Is About

Indian courts carry a very large backlog of unresolved cases — the paper cites National Judicial Data Grid figures from 21 March 2025 of approximately 45.51 million pending cases in District and Subordinate Courts, 6.25 million in High Courts, and 81,598 in the Supreme Court, totalling more than 51 million. Appellate cases, where a higher court reviews a lower court's ruling, are a critical subset because they set precedent and shape consistency across lower courts. The paper's goal is a framework that both predicts the outcome of such appeals and explains the prediction in a structured format that legal professionals can inspect.

Key Contributions

1. The Vichara framework — a six-stage pipeline (rhetorical role classification, case context construction, decision point extraction, present court ruling generation, judgment prediction, explanation generation) for Indian appellate judgment prediction and explanation, centred on extracting structured "decision points" from case proceedings.
2. A structured explanation format grounded in legal reasoning conventions, inspired by the IRAC (Issue–Rule–Application–Conclusion) framework and adapted to Indian legal discourse, rather than free-form generated text.
3. An empirical evaluation across two datasets (PredEx and ILDC_expert) and four LLMs (GPT-4o mini, Llama-3.1-8B, Mistral-7B, Qwen2.5-7B), covering both prediction performance and human-rated explanation quality, with an ablation study over four pipeline stages.
4. Released code via a GitHub repository for reproducibility.

Main Findings

• Best prediction performance: GPT-4o mini achieved the highest scores within Vichara on both datasets, with macro F1 of 81.5 on PredEx and 80.3 on ILDC_expert (table values 81.50 ± 0.42 and 80.30 ± 0.35).
• Second-best prediction: Llama-3.1-8B followed with macro F1 of 76.7 on PredEx and 78.5 on ILDC_expert (table values 76.66 ± 0.50 and 78.46 ± 0.41), ahead of Qwen2.5-7B (72.10 and 76.60 macro F1) and Mistral-7B (69.04 and 74.97 macro F1).
• Beating the baseline: Compared with INLegalLlama (macro F1 76.01 on PredEx, 71.98 on ILDC_expert), all four LLMs in Vichara surpassed it on ILDC_expert, while on PredEx only GPT-4o mini and Llama-3.1-8B outperformed it.
• Human-rated explanation quality: Three legal experts (practicing advocates with 7–8 years of experience) rated 25 explanations per model across four LLMs, giving 100 annotated examples on a 5-point Likert scale. GPT-4o mini scored highest on all three metrics — Clarity 4.57, Linking 4.96, Usefulness 4.37 — followed by Mistral-7B (4.11, 4.44, 3.85).
• Inter-annotator agreement: Fleiss' Kappa was substantial across the three metrics (Clarity 0.66, Linking 0.70, Usefulness 0.63).
• Automatic metrics diverge from human judgement: Qwen2.5-7B recorded the highest scores on most reference-based metrics, including ROUGE-1, ROUGE-L, BLEU and METEOR, while human evaluation favoured GPT-4o mini — a contrast the authors present as evidence of the limits of automatic metrics for this task.
• Ablation — decision points matter most: On ILDC_expert with GPT-4o mini, removing Decision Point Extraction caused the largest drop, reducing macro F1 from 81.25 to 70.92 (nearly 11 points). Removing Present Court Ruling Generation dropped macro F1 to 72.86, and removing Case Context Construction to 77.40.
• Ablation — rhetorical roles matter least: Removing Rhetorical Role Classification produced the smallest degradation, with macro F1 of 78.85 and accuracy of 79.12 versus 80.36 accuracy for the full pipeline.
• Prompt-only pipeline: All stages operate purely through prompting, with no fine-tuning of the underlying LLMs; only the sentence segmentation model and the rhetorical-role classifier are trained components.

Methodology in Plain English

Vichara breaks an English-language appellate case proceeding document into sentences, then labels each sentence by the function it serves in the legal text — the paper uses seven rhetorical roles: Facts, Ruling by Lower Court, Argument, Statute, Precedent, Ratio of the Decision, and Ruling by Present Court. Sentence boundaries are detected with the SAT-12L (Segment Any Text) model fine-tuned via LoRA, and roles are assigned with a hierarchical BiLSTM-CRF model.

From the sentences labelled as Facts, an LLM builds a "case context" capturing six fields: appellants, respondents, issue, appellant's stance, respondent's stance, and the present court. Separately, the document is split into chunks — using existing bullet-point lists where present, otherwise 1000-token segments — and an LLM extracts "decision points", each with an issue, decision maker, outcome, optional reasoning, optional time, and a Boolean flag marking whether the decision maker is the present court. The 1000-token size was chosen because longer segments were observed to degrade extraction quality.

Only decision points attributed to the present court, plus the final sentence labelled as the present court's ruling (the "final statement"), are fed into a prompt that generates a summary of the present court's ruling. The binary outcome is then decided by comparing that ruling against the appellant's stance: if the relief sought is fully or partially granted, the prediction is 1 (Appeal Granted); if it is entirely denied, the prediction is 0 (Appeal Dismissed). Finally, an LLM synthesises an explanation with five fixed sections: Facts of the Case, Legal Issue(s) Presented, Applicable Law and Precedents, Analysis/Reasoning, and Predicted Conclusion.

Predictive results are reported as the mean across 5 independent random seeds with standard deviations. The human explanation evaluation was run on a single run rather than multiple seeds because of the human effort involved.

Why This Matters

Impact on research. The paper argues that prior Indian LJP work tends to produce either extractive explanations (highlighting relevant facts) or abstractive free-form summaries, both unstructured and of limited help in tracing how facts lead to a decision. Vichara positions structured, decision-point-based reasoning as an alternative, and its result that automatic metrics and human judgements disagree is a caution for how explanation quality is measured in this field.

Real-world applications:

• Prioritising and triaging appellate case backlogs in courts facing large pending dockets.
• Assisting judges, clerks and advocates in evaluating the legal reasoning behind a predicted outcome before relying on it.
• Supporting litigation-support tools where only limited case material (such as facts or partial procedural history) is available.
• Providing structured explanations that legal professionals or the public can use to navigate complex judicial reasoning.

Industry relevance. The authors note that smaller open-weight models — Llama-3.1-8B, Qwen2.5-7B, and Mistral-7B — used within Vichara achieved results comparable to the larger GPT-4o mini, which the paper presents as a viable path for resource-constrained deployments. The framework is described as currently operating exclusively on English-language case documents.

Future Directions

1. Reducing computational overhead. The multi-stage pipeline makes multiple LLM calls, which the authors flag as a deployment and latency constraint; they propose exploring prompt optimization and model distillation.
2. Extending beyond Indian appellate cases. Adapting the framework to other case types, court levels, and jurisdictions with different procedural structures, legal doctrines, or language conventions, which the paper says would require substantial adaptation and validation.
3. Broader human evaluation. Scaling up from 25 explanations per model and three advocates to a more diverse pool of legal professionals and a wider range of case types to establish generalizability.
4. Addressing prompt sensitivity and non-determinism. Since results depend on prompt phrasing and LLM output variability, methods for making the pipeline more stable remain open.

Target Audience

Legal AI and NLP researchers working on judgment prediction, legal reasoning, or explainable AI; practitioners and policymakers involved in judicial technology in India; and applied engineers building litigation-support or case-triage systems who want a structured, prompt-based template for combining prediction with legally grounded explanations.

Authors’ abstract

In jurisdictions like India, where courts face an extensive backlog of cases, artificial intelligence offers transformative potential for legal judgment prediction. A critical subset of this backlog comprises appellate cases, which are formal decisions issued by higher courts reviewing the rulings of lower courts. To this end, we present Vichara, a novel framework tailored to the Indian judicial system that predicts and explains appellate judgments. Vichara processes English-language appellate case proceeding documents and decomposes them into decision points. Decision points are discrete legal determinations that encapsulate the legal issue, deciding authority, outcome, reasoning, and temporal context. The structured representation isolates the core determinations and their context, enabling accurate predictions and interpretable explanations. Vichara's explanations follow a structured format inspired by the IRAC (Issue-Rule-Application-Conclusion) framework and adapted for Indian legal reasoning. This enhances interpretability, allowing legal professionals to assess the soundness of predictions efficiently. We evaluate Vichara on two datasets, PredEx and the expert-annotated subset of the Indian Legal Documents Corpus (ILDC_expert), using four large language models: GPT-4o mini, Llama-3.1-8B, Mistral-7B, and Qwen2.5-7B. Vichara surpasses existing judgment prediction benchmarks on both datasets, with GPT-4o mini achieving the highest performance (F1: 81.5 on PredEx, 80.3 on ILDC_expert), followed by Llama-3.1-8B. Human evaluation of the generated explanations across Clarity, Linking, and Usefulness metrics highlights GPT-4o mini's superior interpretability.