Seeing Justice Clearly: Handwritten Legal Document Translation with OCR and Vision-Language Models

AI summary

Overview

Research area: Computer vision and natural language processing applied to legal document digitization, specifically handwritten text recognition (HTR) and machine translation (MT) for a low-resource Indian language (Marathi).

Technical level: Intermediate. The paper assumes familiarity with OCR pipelines, neural machine translation, and vision-language models, but explains its comparisons in accessible terms.

Scope: A comparative evaluation of OCR-plus-machine-translation pipelines versus vision large language models (vLLMs) for translating handwritten Marathi legal documents directly into English.

Code is released at https://github.com/anviksha-lab-iitk/SJC. The preprint is arXiv:2512.18004v1 [cs.CV], dated 19 Dec 2025, under a CC BY 4.0 license. Authors are affiliated with IIT Kanpur, Symbiosis Law School Pune, and the University of Birmingham Dubai.

What This Paper Is About

Indian district courts and police stations still record critical documents such as First Information Reports (FIRs), case diaries, witness statements, and court proceedings by hand, much of it in Marathi, which makes these records hard to archive, search, or analyze. The authors ask whether it is better to translate such documents with a two-stage pipeline (an OCR system extracts text, then a machine translation model converts it to English) or with a single vision-language model that reads the image and outputs English directly. Their goal is to identify which approach, if any, is accurate and robust enough to support automated legal digitization in low-resource settings.

Key Contributions

1. OCR benchmarking on handwritten legal Marathi: Tesseract, EasyOCR, and PaddleOCR are evaluated on a curated dataset of handwritten Marathi legal documents using Character Error Rate (CER) and Word Error Rate (WER).
2. Translation analysis on OCR output: IndicTrans2 and Sarvam-1 are assessed on text extracted by the OCR tools, forming six distinct OCR-MT pipeline combinations (three OCR tools x two translation models).
3. Vision-language model benchmarking: Chitrarth, Maya-8B, and Ovis2-34B (int4 quantized, with Ovis2-16B also discussed in the qualitative comparison) are tested in a zero-shot setting for direct image-to-English translation and compared against the OCR-MT pipelines.
4. A curated ground-truth resource: Approximately 60 scanned Marathi legal PDFs with English translations produced by two native Marathi speakers and reviewed by a legal language expert.

Main Findings

• Printed text beats handwriting: OCR models performed notably better on printed segments than on handwritten ones. Handwritten content produced omissions, misrecognized characters, and fragmented outputs, with handwritten dates being a specific source of incorrect extraction.
• EasyOCR led the traditional tools: EasyOCR consistently outperformed PaddleOCR and Tesseract, though it still struggled with inconsistent handwriting styles.
• Error propagation damages translation: OCR mistakes degraded downstream translation. The paper gives the example of the Marathi word "Gaav" (transliterated from "gaon") being rendered as "Gaon" instead of "Village", and other cases where key information was dropped.
• Translation models produced noisy output: When fed error-containing OCR text, IndicTrans2 and Sarvam-1 sometimes generated incoherent or mixed-language output, including trailing untranslated Marathi fragments, with losses of legal or factual content and awkward phrasing on longer, semantically complex sentences.
• Prompting strongly affected vLLM results: vLLM performance improved significantly with detailed instructions, indicating high sensitivity to prompt design.
• vLLMs hallucinated on handwriting: Chitrarth often failed to produce coherent output and in one case invented a meeting summary with names, dates, and locations not present in the source. Maya-8B misread the document as a study guide. Ovis2-34B gave partial translations of Marathi ledger headers and dates but remained largely fabricated or misunderstood. Ovis2-16B performed relatively better, identifying some legal and financial cues such as account numbers, names, and locations and recognizing structural layout, but its output lacked completeness and was partially incoherent.
• vLLMs captured gist, not verbatim content: The models could interpret visual elements and page layout but often produced high-level descriptions of documents rather than their actual text, leading human annotators to conclude they currently lack the precision required for legal-grade translation.
• Numeric metric values are not reported in the provided content: CER and WER are named as the evaluation metrics, but specific scores are not given in the paper text supplied here.

Methodology in Plain English

The researchers assembled roughly 60 scanned Marathi legal PDFs from authentic legal sources. The pages mix handwritten Marathi with printed text and include stamps, seals, signatures, and structured tables. Ground truth was created by having two native Marathi speakers translate the Marathi manually into English, after which a legal language expert reviewed the translations for fidelity, contextual accuracy, and terminological consistency.

Two approaches were then compared. In the first, each page image passes through one of three OCR tools (Tesseract, EasyOCR, or PaddleOCR), and the extracted Marathi text is handed to one of two translation models (Sarvam-1 or IndicTrans2), yielding six pipeline variants. In the second, three vision-language models (Chitrarth, Maya-8B, Ovis2-34B) receive the scanned image directly and are asked, in a zero-shot setting with manually designed legal-oriented prompts, to produce English. OCR quality was measured with CER and WER, while translation quality was judged by human annotators fluent in both Marathi and English along three criteria: fluency, adequacy, and correctness against the gold-standard translations. A qualitative side-by-side comparison of vLLM outputs against the human annotation is also presented.

Why This Matters

The paper documents a concrete failure mode for high-stakes automation: both modular pipelines and unified vision-language models break down on handwritten, domain-specific, low-resource text, and vLLMs add a further risk by inventing plausible-sounding content when they cannot read the source. For research, it provides a shared benchmark and a clear statement of where current architectures fall short, framing fine-tuning, prompt design, and hybrid designs as open problems.

Real-world applications:

• Court and police record digitization: Converting handwritten FIRs, charge sheets, case diaries, and witness statements into searchable English records.
• Access for non-native speakers and legal professionals: Making Marathi-language legal material usable by practitioners and parties who do not read Marathi.
• Archival and retrieval: Enabling structured case records that can be indexed, searched, and analyzed rather than stored only as scans.
• Low-resource deployment: Informing edge-deployable designs that can run on district-level compute rather than requiring high-end infrastructure.

Industry relevance centers on legal-tech and government digitization programs, where the paper's warning about hallucination in vLLMs is directly relevant to any vendor promising automated legal translation, and where its finding that lightweight models are the practical target shapes near-term product design.

Future Directions

• Fine-tuning on legal handwritten data: Adapting both OCR systems and vLLMs to larger annotated corpora of handwritten legal documents, since the paper found zero-shot performance insufficient.
• Better prompting and alignment: Pursuing systematic prompt engineering and instruction alignment to reduce hallucination and improve fidelity, given how strongly results varied with prompt quality.
• Hybrid OCR-vLLM pipelines: Using OCR for structural cues alongside vLLMs for contextual translation, which the authors suggest may outperform either approach alone.
• Evaluation, efficiency, and data growth: Developing metrics tailored to legal translation that track factual consistency, terminology preservation, and completeness; investigating quantization and distillation for district-level compute; and curating a larger, more diverse corpus of handwritten legal documents across states and formats.

Target Audience

Researchers and practitioners in document analysis, low-resource machine translation, and multilingual vision-language modeling will find the benchmark comparison and failure analysis most useful. Legal-tech developers, court digitization program managers, and policymakers evaluating automated translation of judicial records should read it for the concrete evidence that current models are not deployment-ready for legal-grade handwriting. Graduate students entering OCR-MT or multimodal legal NLP will benefit from the clearly defined pipeline baselines and the curated ground-truth dataset.

Authors’ abstract

Handwritten text recognition (HTR) and machine translation continue to pose significant challenges, particularly for low-resource languages like Marathi, which lack large digitized corpora and exhibit high variability in handwriting styles. The conventional approach to address this involves a two-stage pipeline: an OCR system extracts text from handwritten images, which is then translated into the target language using a machine translation model. In this work, we explore and compare the performance of traditional OCR-MT pipelines with Vision Large Language Models that aim to unify these stages and directly translate handwritten text images in a single, end-to-end step. Our motivation is grounded in the urgent need for scalable, accurate translation systems to digitize legal records such as FIRs, charge sheets, and witness statements in India's district and high courts. We evaluate both approaches on a curated dataset of handwritten Marathi legal documents, with the goal of enabling efficient legal document processing, even in low-resource environments. Our findings offer actionable insights toward building robust, edge-deployable solutions that enhance access to legal information for non-native speakers and legal professionals alike.