Mina: A Multilingual LLM-Powered Legal Assistant Agent for Bangladesh for Empowering Access to Justice

AI summary

Overview

Research area: Legal NLP / multilingual large language model agents, with a focus on low-resource, non-English jurisdictions.

Technical level: Advanced. The work combines retrieval-augmented generation (RAG), vector databases, multi-agent orchestration (LangGraph state machine), multilingual embeddings, and a suite of auxiliary tools, then evaluates them against a national professional examination.

Scope: The paper builds and evaluates "Mina," a Bengali- and English-capable LLM legal assistant for Bangladesh, using the 2022 and 2023 Bangladesh Bar Council Examinations (MCQ, Written, and Viva Voce) as its benchmark.

What This Paper Is About

Bangladesh faces a severe access-to-justice crisis: the paper reports a backlog of 3.7 to 4.4 million cases, only 2,100 judges (one judge per 90,000 people), delays of 10 to 60 years, and unregulated lawyer fees, alongside complex legal language, colonial-era codes, and Farsi-influenced terminology. Existing legal NLP tools and English-centric models are inadequate because Bengali has scarce annotated datasets, limited digital content, and underdeveloped tooling, while models lack jurisdiction-specific knowledge.

The goal is a localized, multilingual legal assistant that retrieves Bangladeshi statutes at both the Act and Section levels, reasons over them, drafts documents, translates into plain Bengali, and inserts citations, and then to test whether it can perform at the level of human candidates on a high-stakes national legal examination.

Key Contributions

1. A localized multilingual legal assistant with an integrated pipeline combining two-stage RAG, reasoning, drafting, translation, and citation insertion, plus a custom legal dictionary for colonial-era and Farsi-influenced terms and a socio-economic simulation module.

2. A two-stage retrieval design that first retrieves Acts (default top 5) from LLM-summarized Act records, then filters the Section database (default top 10) using those Act IDs, which the authors state prevents conflation of unrelated provisions.

3. A rigorous evaluation across all stages of the 2022 and 2023 Bangladesh Bar Council Examinations (Preliminary MCQ, Written, and Viva Voce via a chat interface), with law faculty experts as evaluators, comparing proprietary and open-source models under four setups: without RAG, naive RAG, 2-step RAG, and tools.

4. A comprehensive error analysis, evaluator response analysis, and a cost-per-query comparison examining affordability implications for public-service deployment.

Main Findings

• Top overall result: Gemini-2.5-Flash consistently outperformed all baselines across years and setups. It scored 77.00 on the MCQ stage with tools in both 2022 and 2023, 81.00 (2022) and 81.80 (2023) on the Written stage with tools, and 81.00 on the Viva with tools.

• Reported agent-level performance: The paper states Mina consistently scored 75–80%, matched or exceeded average human performance, and passed both years of the Bar Council exam.

• Retrieval drives most gains: Naive RAG produced moderate gains over no RAG, while 2-step RAG with tool integration substantially boosted performance. On the Viva, naive or 2-step RAG increased scores by 20–40 points for smaller and mid-sized models. Without RAG, GPT-4o scored 18.60 (2022 MCQ) versus 73.60 with tools.

• Open-source scale matters, but with exceptions: Qwen3-30B-A3B-Instruct-2507 reached 70.80 (2022) and 72.40 (2023) on MCQ with tools, and 78.20 / 79.40 on the Written with tools, and 79.40 on the Viva. Llama3.1-70B-Instruct reached 79.80 (2022) and 80.20 (2023) on the Written with tools but only 42.40 / 46.20 on MCQ with tools. Small models such as Llama3.2-1B-Instruct and Gemma-3-4B-it barely surpassed random-choice baselines (random choice was around 25).

• Year-to-year stability: Performance gaps between 2022 and 2023 remained small, which the authors interpret as generalization across cohorts rather than year-specific overfitting.

• Human comparison benchmarks: In 2022, 25.86% of candidates (10,527 of 40,696) passed the MCQ stage, and in 2023 only 17.96% (6,229 of 34,682), against a 50% passing threshold. Written pass rates were 53.94% (5,533 of 10,527) in 2022 and 44.21% (2,754 of 6,229) in 2023, with human examinees typically achieving 40–60%. Viva pass rates were 96.65% (5,348 of 5,533) in 2022 and 97.25% (2,973 of 3,057) in 2023.

• Where the system still trails humans: In the viva, optimized models scored 70–76%, below the human near-certainty of over 96%, though still exhibiting legally coherent reasoning.

• Systematic error patterns: The model misinterprets Bengali conjunctions ("O" meaning "and" versus "ba" meaning "or"), makes jurisdictional hierarchy errors, misclassifies doctrines such as Res Judicata under Section 151, and confuses intra-order rules (Order 1, Rules 8 versus 13). In written answers it used "injury" instead of "damage" under the Specific Relief Act, 1877, omitted statutory conditions, denied the existence of general exceptions in the Penal Code, 1860, conflated civil and criminal law, and hallucinated a case named "Deowaney Mokdama v. Hazirawala and Garahazira."

• Evaluator observations: Evaluators found large retrieval-augmented models produced coherent, exam-style answers following the IRAC pattern with strong statutory recall, but penalized omissions of procedural prerequisites, arithmetic failures in limitation-period calculations, incomplete synthesis across the CPC, Evidence Act, and Specific Relief Act, and occasional hallucinated citations.

• Cost findings: Estimated per-query cost for short MCQ-style queries was 0.2–0.4 cents for Qwen (approximately 0.24–0.49 BDT), 0.3–0.5 cents for LLaMA (approximately 0.37–0.61 BDT), and 0.4–0.6 cents for Gemini (approximately 0.49–0.73 BDT). For longer constructed-response queries the cost rose to roughly 0.8–1.4 cents for Qwen (approximately 0.98–1.71 BDT), 1.0–1.6 cents for LLaMA (approximately 1.22–1.95 BDT), and 1.4–2.0 cents for Gemini (approximately 1.71–2.44 BDT). Even a conservative multi-turn upper bound of 10 cents total (approximately 12.2 BDT) corresponds to only about 0.12%–0.61% of the minimum prevailing cost of basic legal advice in Bangladesh, which typically ranges from 2,000 to 10,000 BDT per consultation — a cost reduction of approximately 99.4%–99.9% relative to traditional human-provided legal services.

• Dataset scale: As of April 2025, the corpus contains 595 Acts comprising 18,023 Sections (average 24 sections per act). Act titles average 50.30 characters, Section titles 38.07 characters, and Section content 736.69 characters; average act name and detail lengths are 50 and 438 characters, and average section name and detail lengths are 38 and 736 characters.

Methodology in Plain English

The system is built around an Orchestrator Agent that looks at the user's message, chat history, and any uploaded documents, and decides whether it can answer directly or needs to look things up. If it needs more information, it hands off to a RAG Agent.

Retrieval happens in two steps. First, a Keyword Generator produces 5–10 semantically meaningful search terms (with a regular expression fallback), and the system searches a vector database of Act summaries to find the top 5 most relevant Acts. Second, those Act IDs are used to filter a separate Section-level database to pull the top 10 most relevant legal provisions. Both databases were built from the official Bangladesh Law and Justice website. The Act database indexes LLM-generated summaries plus metadata; the Section database chunks individual provisions (1–2 chunks each) with Act metadata attached. Cohere's embed-multilingual-light-v3.0 provides the multilingual embeddings so Bengali and English queries can be matched. Retrieved sections are checked for relevance; if they are insufficient, the query is refined and rerun.

The agents run inside a LangGraph-based state machine that keeps memory across turns, supporting multi-turn consultations and conditional execution. The Orchestrator Agent and RAG Agent have separated roles so the system is easier to maintain and extend.

A suite of tools supports the workflow: a File Content Reader for .pptx, .docx, and .pdf files; the Keyword Generator; a DuckDuckGo-based Web Search that returns titles, URLs, and snippets; a BeautifulSoup Web Page Parser that extracts up to 5,000 characters of visible content; a Question Relevance Analyzer that uses embeddings to link current and prior queries; a Chat Analyzer; a custom Legal Dictionary for colonial-era and Farsi-influenced terms; and a Socio-Economic Simulation module.

Model choice was tested broadly: GPT-4o; gemini-2.0-flash and gemini-2.5-flash; gemma-3-4b-it, gemma-3-12b-it, and gemma-3-27b-it; llama3.2-1b-instruct, llama3.2-3b-instruct, llama3.1-8b-instruct, and llama3.1-70b-instruct; command-a-03-2025; and qwen3-4b-instruct-2507 and qwen3-30b-a3b-instruct-2507.

For evaluation, the team mirrored the actual Bar Council process. MCQ answers were automatically marked against ground truth on a 100-point scale, replicating Optical Mark Recognition evaluation, with each setup and model combination repeated 5 times and averaged. Written answers were scored by law faculty members and legal experts from leading Bangladeshi universities (anonymized), with each answer assessed by at least two qualified judges and the two scores averaged. The written section had thirteen questions: the first eleven organized into five sets, from which candidates had to answer six (two from the first set and one from each of the remaining four) for ninety marks, plus Set F with two ten-mark questions of which one had to be answered. The Viva was conducted through a ChatGPT-like interactive platform, with evaluators assigning detailed scores out of 100 even though the official Bar Council record is simply pass or fail. A minimum of 50% was the human passing threshold.

Why This Matters

Impact on research. The paper argues that retrieval alone is insufficient for high-stakes legal tasks, and that scaling alone is not enough — architecture and training data alignment matter. It provides a template for adapting established components (multilingual embeddings, RAG, agent frameworks) to a bilingual, low-resource legal environment, and it contributes a rigorous, multi-stage evaluation protocol that mirrors a real national licensing exam rather than a synthetic benchmark. It also documents concrete failure modes (Bengali conjunction confusion, hierarchical procedural reasoning, hallucinated citations) that future legal NLP work can target.

Real-world applications:

• Legal aid for low-income and marginalized populations who cannot afford the 2,000–10,000 BDT consultation fees, delivered at a few cents per query.
• Bar Council exam preparation and legal education, since the system was benchmarked directly against the 2022 and 2023 examinations.
• Drafting petitions, referencing statutes, and simplifying complex legal language into plain Bengali via a chat interface, including analysis of uploaded documents.
• Policy planning and diagnostic analysis of how socio-demographic variables interact with legal access, through the socio-economic simulation module.

Industry relevance. The cost analysis positions the system as economically viable for public-service deployment in low-resource settings, with the computational overhead of non-LLM tools (retrieval, translation, formatting) described as negligible relative to LLM inference. The finding that mid-sized open-source models approach or exceed average human performance with a robust RAG pipeline suggests a cost-effective alternative to proprietary models, which the authors frame as particularly relevant to Global South contexts. The paper also raises policy questions about AI-augmented professional legal assessment. It explicitly states that Mina is a supportive tool, not infallible, and that liability remains with the human professional using it, requiring human supervision and cross-checking against authoritative statutes and case law.

Future Directions

• Deterministic procedural modules. Evaluators recommended deterministic procedural calculators and statutory-fusion modules to address arithmetic failures in limitation-period calculations and incomplete synthesis across the CPC, Evidence Act, and Specific Relief Act.

• Fine-tuning and contrastive training. The error analysis suggests fine-tuning on authoritative legal texts and on annotated bilingual exam data, plus contrastive training, to improve interpretive fidelity and reduce omissions, domain conflation, and hallucinations.

• Interactive clarification and dialogue-state tracking. The viva analysis found a key weakness was the lack of clarifying questions and flattening of hierarchical reasoning; proposed remedies include a clarification policy, dialogue-state tracking, and real-time procedural validation.

• Verification and retrieval robustness. Suggested directions include integrating verification against statutory and case law databases to improve factual accuracy, and addressing the limitations that retrieval quality depends heavily on the underlying corpus and that Two Step RAG introduces additional latency and complexity that may not scale well in real-time systems.

Target Audience

This paper is most useful to NLP and AI researchers working on legal applications, multilingual or low-resource language technology, and retrieval-augmented generation and agent architectures. It also speaks to legal tech practitioners and policymakers interested in deploying AI legal assistance in developing economies, to legal educators and Bar examination bodies evaluating AI performance against professional standards, and to legal aid organizations assessing the affordability and reliability of AI-assisted services. Readers without a background in RAG or agent design will find the high-level framing accessible, but the architecture and evaluation details are aimed at a technically experienced audience.

Authors’ abstract

Bangladesh's low-income population faces major barriers to affordable legal advice due to complex legal language, procedural opacity, and high costs. Existing AI legal assistants lack Bengali-language support and jurisdiction-specific adaptation, limiting their effectiveness. To address this, we developed Mina, a multilingual LLM-based legal assistant tailored for the Bangladeshi context. It employs multilingual embeddings and a RAG-based chain-of-tools framework for retrieval, reasoning, translation, and document generation, delivering context-aware legal drafts, citations, and plain-language explanations via an interactive chat interface. Evaluated by law faculty from leading Bangladeshi universities across all stages of the 2022 and 2023 Bangladesh Bar Council Exams, Mina scored 75-80% in Preliminary MCQs, Written, and simulated Viva Voce exams, matching or surpassing average human performance and demonstrating clarity, contextual understanding, and sound legal reasoning. Even under a conservative upper bound, Mina operates at just 0.12-0.61% of typical legal consultation costs in Bangladesh, yielding a 99.4-99.9\% cost reduction relative to human-provided services. These results confirm its potential as a low-cost, multilingual AI assistant that automates key legal tasks and scales access to justice, offering a real-world case study on building domain-specific, low-resource systems and addressing challenges of multilingual adaptation, efficiency, and sustainable public-service AI deployment.