LEMUR: A Corpus for Robust Fine-Tuning of Multilingual Law Embedding Models for Retrieval

AI summary

Overview

Research area: Natural Language Processing / multilingual legal information retrieval, domain-adaptive embedding models, retrieval-augmented generation (RAG).

Technical level: Intermediate. The paper is readable with basic familiarity with embeddings, contrastive learning and retrieval metrics, but it also contains formal loss equations and detailed dataset-construction procedures.

Scope: The paper introduces LEMUR, a 24,953-document, 25-language corpus of EU environmental legislation extracted from EUR-Lex PDFs, together with a PDF-conversion quality metric (LCS) and fine-tuning experiments on three multilingual embedding models for monolingual, bilingual and cross-lingual legal retrieval.

What This Paper Is About

Legal information systems increasingly rely on RAG, whose quality depends on the embedding model used for retrieval. Existing multilingual legal corpora are built for pretraining or classification rather than semantic retrieval, and PDF-based legislative sources suffer from imperfect text extraction that corrupts downstream embeddings. The paper builds a large multilingual EU law corpus from official EUR-Lex PDFs, quantifies conversion fidelity, and fine-tunes state-of-the-art multilingual embedding models on the corpus to test whether legal-domain adaptation improves metadata-to-document retrieval across languages.

Key Contributions

1. Multilingual dataset (LEMUR): A Law European MUltilingual Retrieval corpus consisting of 24,953 EU legal PDFs across 25 languages, drawn from 1,174 distinct legal acts (Category 15, Subcategory 10 — Environment) published between 1961–2025, totaling 461k pages, and designed for training embedding models on legal text.
2. The Lexical Content Score (LCS): A systematic measure of PDF-to-text conversion quality, computed as cosine similarity between bag-of-words vectors of converted documents and their authoritative HTML counterparts, applied across 25 languages.
3. Legal embedding fine-tuning: Fine-tuning of three state-of-the-art embedding models on five languages (EN, DE, FR, LV, MT) for legal document retrieval, evaluated in monolingual, bilingual (EN–LV) and cross-lingual settings.
4. Open releases: Code (GitHub repository) and data (Hugging Face dataset) are published under CC BY 4.0.

Main Findings

• PDF conversion quality is high but uneven across languages: For high-resource languages the olmOCR conversion achieves similarity scores above 95% against HTML; Latvian averages approximately 90% and Maltese about 80%. In the per-language appendix table, English scores highest at 0.9740 and Maltese lowest at 0.8027. Older documents score lower, which the authors hypothesize reflects the OCR model's alignment with contemporary layout standards.
• Conversion method choice matters: The authors tested Docling, Unstructured and PyMuPDF, and found the best results came from converting PDFs into structured JSONL with olmOCR, which preserves table structure (in markdown format) better than the HTML versions used by earlier datasets such as MultiEURLEX.
• Monolingual fine-tuning consistently helps: Across all evaluated languages, fine-tuning improved Top-1, Top-3 and Top-5 retrieval accuracy relative to the corresponding pre-trained models. High-resource languages performed better even at baseline, while low-resource languages had much lower baseline performance that fine-tuning brought to levels comparable with high-resource languages.
• Bilingual training produces mixed results: For E5, training on EN–LV together had an additive effect (EN Top-1 90.30, LV Top-1 83.85). For Qwen3-0.6B and Qwen3-4B, training on the dedicated language performed better than joint training. EN-only training did not improve English retrieval over the baseline in some cases, and adding Latvian data neither enhanced nor substantially degraded English Top-1 or Top-5 accuracy — no evidence of negative transfer.
• Cross-lingual transfer works: For Qwen3-0.6B, models fine-tuned on one language generalized to other high-resource languages (EN, DE, FR), with Top-1 increasing by at least 10% relative to baseline, and Top-5 above 98%. Low-resource languages still improved in Top-1 and Top-5 when fine-tuned on other languages.
• Full-corpus search is harder than test-only search: In the appendix results across eighteen languages, fine-tuned and original models both lose accuracy when test queries are evaluated against all documents rather than test documents only (for example, Spanish drops from 83.03 to 69.64 Top-1 after fine-tuning). The largest baseline-to-fine-tuned gain in that appendix is Irish (GA), from 39.53 to 69.76 Top-1 on test-only search.
• Test set sizes vary by language: English 227, German 226, French 224, Latvian 192, Maltese 187, reflecting later EU accession and a small number of documents excluded for corruption or incomplete extraction.

Methodology in Plain English

The authors first gathered all EUR-Lex legal acts in the Environment subcategory, which yielded 1,174 acts available in 25 official EU languages — 24,953 PDFs totaling 461k pages, roughly 19 pages and 7,781 tokens per document.

They then converted the PDFs to structured JSONL using olmOCR, and checked the result by comparing each converted document against the corresponding EUR-Lex HTML version. Before comparison, the HTML text is normalized (styling attributes removed, lowercase, numeric formatting normalized, repeated punctuation collapsed), each text is turned into a bag-of-words vector, and the cosine similarity between the two vectors defines the LCS.

For retrieval training, each document is split into two parts: a short introductory metadata block (act type, date, subject description, legal basis, publication notes) used as the query, and the remaining substantive legislative text used as the document to retrieve. This mimics realistic legal search, where a user starts with brief structured information. Data is split 60% training / 20% validation / 20% test, independently per language or language pair, with the same underlying acts kept in the same split across languages.

Three embedding models are fine-tuned — Qwen3-0.6B, Qwen3-4B (Yang et al., 2025) and E5-Multilingual (Wang et al., 2024) — all sourced via the MTEB leaderboard. Fine-tuning uses a contrastive Multiple Negatives Ranking (MNR) objective with in-batch negatives; the bilingual variant extends this to a grouped multi-positive MNR loss where aligned translations of the same act all count as positives. Training runs up to 30 epochs with early stopping on validation loss, bfloat16 precision, gradient checkpointing where supported, and a linear warm-up schedule. Maximum sequence length is 2,048 tokens except for E5-Multilingual, which is limited to 512. Training was done on NVIDIA RTX A6000 and NVIDIA A100 (80GB) GPUs, with Qwen3-4B on the A100. Typical fine-tuning cost: E5 about 20–30 minutes per language, Qwen3-0.6B about 2–4 hours, Qwen3-4B about 6–8 hours.

Retrieval is simulated with ChromaDB, storing L2-normalized vectors and searching by cosine similarity. Very long documents are truncated with decreasing token caps: approximately 8–15% of documents require truncation across languages and models, and for those documents roughly 40–50% of their original tokens are removed. Performance is reported as Acc@1, Acc@3 and Acc@5, under two settings: Full-dataset search (test queries against all documents) and Test-only search (against held-out test documents).

Why This Matters

This work addresses a practical bottleneck in legal AI: even strong general-purpose multilingual embedding models are not adapted to legal vocabulary, syntax or cross-lingual alignment, and the noisy state of official PDF sources silently degrades retrieval quality. By quantifying that noise (LCS) and showing that domain fine-tuning improves retrieval across languages and model sizes, the paper gives a reproducible recipe and an open dataset for building more grounded legal RAG systems. It also provides evidence that fine-tuning mainly improves language-independent, content-level legal representations rather than language-specific cues.

Real-world applications:

• Cross-lingual search over EU environmental legislation, where a lawyer or compliance officer queries in one language and retrieves acts published in another.
• RAG pipelines for legal question answering, where the retriever must return verifiable primary documents to reduce hallucination.
• Automated compliance monitoring and regulatory tracking for companies operating across multiple EU member states.
• Retrieval for lower-resource EU languages (for example Latvian or Maltese) that are poorly served by general-purpose embedding models.

Industry relevance: Organizations building legal-tech, regulatory-intelligence or enterprise search products need embedding models that work over messy, multi-column, table-heavy legislative PDFs in many languages. This paper shows that domain fine-tuning is achievable at modest compute cost (minutes to hours per language on a single GPU), that a 0.6B-parameter model can be adapted effectively, and that a vector-database workflow using open components can deliver substantial retrieval gains.

Future Directions

• Broaden topical coverage: LEMUR is restricted to Category 15, Subcategory 10 (Environment), which limits topical diversity and may not transfer to EUR-Lex categories with different legal styles and terminology.
• Expand bilingual and multilingual training configurations: Bilingual multi-positive fine-tuning was evaluated on only one language pair (EN–LV); other language pairs and resource configurations remain unexplored.
• Reduce residual PDF-to-text noise: Average lexical similarity across languages is about 0.94, so extraction noise remains, especially for older documents and low-resource languages. Alternative conversion pipelines and layout-aware post-processing are suggested.
• Extend to additional legal domains and languages, as stated in the conclusion's future work.

Target Audience

Researchers and practitioners in legal NLP, multilingual information retrieval and RAG system design; builders of embedding models who need a domain-adapted, multilingual retrieval corpus; and legal-tech engineers who must extract reliable text from official PDF sources and evaluate retrieval quality across high- and low-resource languages. The paper is also useful to dataset curators interested in conversion-quality measurement, and to readers who want a concrete demonstration that contrastive domain fine-tuning transfers across languages.

Authors’ abstract

Large language models (LLMs) are increasingly used to access legal information. Yet, their deployment in multilingual legal settings is constrained by unreliable retrieval and the lack of domain-adapted, open-embedding models. In particular, existing multilingual legal corpora are not designed for semantic retrieval, and PDF-based legislative sources introduce substantial noise due to imperfect text extraction. To address these challenges, we introduce LEMUR, a large-scale multilingual corpus of EU environmental legislation constructed from 24,953 official EUR-Lex PDF documents covering 25 languages. We quantify the fidelity of PDF-to-text conversion by measuring lexical consistency against authoritative HTML versions using the Lexical Content Score (LCS). Building on LEMUR, we fine-tune three state-of-the-art multilingual embedding models using contrastive objectives in both monolingual and bilingual settings, reflecting realistic legal-retrieval scenarios. Experiments across low- and high-resource languages demonstrate that legal-domain fine-tuning consistently improves Top-k retrieval accuracy relative to strong baselines, with particularly pronounced gains for low-resource languages. Cross-lingual evaluations show that these improvements transfer to unseen languages, indicating that fine-tuning primarily enhances language-independent, content-level legal representations rather than language-specific cues. We publish code\footnote{\href{https://github.com/nargesbh/eur_lex}{GitHub Repository}} and data\footnote{\href{https://huggingface.co/datasets/G4KMU/LEMUR}{Hugging Face Dataset}}.