Parameter Efficient Fine Tuning Llama 3.1 for Answering Arabic Legal Questions: A Case Study on Jordanian Laws

AI summary

Overview

Research area: Natural Language Processing, specifically parameter-efficient fine-tuning of large language models for Arabic legal question answering.

Technical level: Intermediate. The abstract assumes familiarity with fine-tuning, quantization, LoRA adapters, and evaluation metrics such as BLEU and ROUGE, though the underlying goal is easy to grasp.

Scope: A case study on adapting two quantized Llama-3.1-8B variants to answer questions grounded in Jordanian law using a purpose-built Arabic legal question-answer dataset.

What This Paper Is About

Large language models are trained mostly on general text, so they tend to be weak on specialized, low-resource domains such as Arabic legal practice. The authors take Jordanian law as a test case and ask whether a compact, quantized Llama-3.1 model can be adapted to answer legal questions accurately without the cost of full-scale retraining. Their goal is to show that resource-efficient fine-tuning can improve legal reasoning and answer quality in Arabic relative to the models' untuned base versions.

Key Contributions

1. A case study applying parameter-efficient fine-tuning (PEFT) with LoRA adapters to Arabic legal question answering, using Jordanian law as the domain.
2. A custom dataset of 6,000 legal question-answer pairs curated from Jordanian laws and formatted into structured prompts.
3. A comparison of two quantized model variants — Llama-3.1-8B-bnb-4bit and Llama-3.1-8B-Instruct-bnb-4bit — each evaluated against its own base version.
4. A demonstration that combining 4-bit quantization with the Unsloth framework enables accelerated, resource-efficient training for this domain-specific task.

Main Findings

• Improved legal reasoning and accuracy: The fine-tuned models reportedly showed better legal reasoning and accuracy than their respective base versions, as measured by BLEU and ROUGE. The abstract does not report the actual scores.
• Both model variants benefited: Gains are claimed for both the base instruction-untuned variant and the instruction-tuned variant, each compared against its own unmodified counterpart.
• Resource efficiency was achieved: Quantization (4-bit) and optimized fine-tuning strategies allowed the adaptation to be carried out with modest compute, which is the central practical claim of the work.
• Domain adaptation is feasible for Arabic law: The results are presented as evidence that large language models can be productively adapted to Arabic legal domains.
• Evaluation stayed at surface-overlap metrics: Assessment relied on BLEU and ROUGE only. The abstract mentions no human evaluation, legal-expert review, or accuracy figures, so the strength of the "accuracy" claim cannot be judged from the abstract alone.

Methodology in Plain English

The researchers started with two off-the-shelf Llama-3.1-8B models that had already been compressed into 4-bit form, which makes them cheaper to store and run. Rather than retraining all of the model's weights, they used LoRA adapters — small trainable add-on layers that adjust the model's behavior while the bulk of it stays frozen. Training was carried out with the Unsloth framework, which is designed to speed up and shrink the memory cost of this kind of fine-tuning.

For training material, they assembled 6,000 question-and-answer pairs drawn from Jordanian laws and rewrote them as structured prompts, giving the model consistent input formatting. After fine-tuning, they compared each adapted model with the original unmodified version of the same model using BLEU and ROUGE, which score how closely generated answers overlap with reference answers. The abstract does not describe the data collection process, annotation procedure, or any baseline beyond the models' own base versions.

Why This Matters

Impact on research. The work adds to the evidence that small, quantized open-weight models can be specialized for low-resource languages and niche domains through lightweight adaptation rather than full retraining. It also contributes a legal-domain Arabic dataset and a concrete recipe combining LoRA, 4-bit quantization, and the Unsloth toolchain.

Real-world applications:

• Assisting lawyers, paralegals, and court staff in Jordan who need fast first-pass answers about statutes and procedures.
• Helping citizens and small businesses understand legal obligations in Arabic without paying for professional consultation at every step.
• Supporting legal-aid organizations and clinics that handle more queries than their staff can absorb.
• Serving as a template for adapting the same approach to the legal systems of other Arabic-speaking countries.

Industry relevance. Quantized, LoRA-tuned models are attractive to organizations that cannot afford large-scale training or cloud inference costs, and the pattern generalizes to other regulated, document-heavy sectors such as tax, insurance, and compliance. The paper also signals a market interest in Arabic-language domain assistants, a segment often underserved by general-purpose models.

Future Directions

• Reporting the actual BLEU and ROUGE scores, and adding evaluation methods that capture legal correctness rather than surface overlap with reference answers.
• Involving legal experts in reviewing model outputs, since fluent Arabic is not the same as legally sound Arabic.
• Expanding the dataset beyond 6,000 pairs and beyond Jordanian law to test whether the approach transfers to other jurisdictions and to other Arabic legal registers.
• Comparing LoRA-based fine-tuning against alternative adaptation strategies, or testing whether larger or differently quantized models produce further gains.
• Examining the limits of the approach on questions requiring multi-document reasoning, citation of specific articles, or awareness of recent legal amendments.

Target Audience

Researchers and practitioners working on Arabic NLP, low-resource language adaptation, or domain-specific fine-tuning of large language models. It is also relevant to legal-tech developers and institutions in Arabic-speaking jurisdictions evaluating whether open-weight models can be adapted to their domain, and to engineers looking for a concrete, resource-efficient fine-tuning recipe they can replicate.

Authors’ abstract

This study uses Jordanian law as a case study to explore the fine-tuning of the Llama-3.1 large language model for Arabic question-answering. Two versions of the model - Llama-3.1-8B-bnb-4bit and Llama-3.1-8B-Instruct-bnb-4bit - were fine-tuned using parameter-efficient fine-tuning (PEFT) with LoRA adapters and 4-bit quantized models, leveraging the Unsloth framework for accelerated and resource-efficient training. A custom dataset of 6000 legal question-answer pairs was curated from Jordanian laws and formatted into structured prompts. Performance was evaluated using the BLEU and the ROUGE metrics to compare the fine-tuned models to their respective base versions. Results demonstrated improved legal reasoning and accuracy while achieving resource efficiency through quantization and optimized fine-tuning strategies. This work underscores the potential of adapting large language models for Arabic legal domains and highlights effective techniques for fine-tuning domain-specific tasks.