Text to Trust: Evaluating Fine-Tuning and LoRA Trade-offs in Language Models for Unfair Terms of Service Detection

AI summary

Overview

• Research area: Natural Language Processing applied to legal informatics — specifically automated detection of unfair clauses in Terms of Service (ToS) agreements using large language models.
• Technical level: Intermediate. The paper assumes familiarity with transformer encoders, parameter-efficient fine-tuning (LoRA/QLoRA), and standard classification metrics, but the paradigms are described accessibly.
• Scope: A single comparative study benchmarking three modeling paradigms — full fine-tuning, LoRA with 4-bit quantization, and zero-shot prompting — on clause-level ToS unfairness detection, with an additional deployment test on a large scraped web corpus.

What This Paper Is About

Terms of Service agreements are long, densely worded, and routinely accepted without being read, yet they can contain clauses that waive liability, force arbitration, allow unilateral changes, or strip users of legal rights. Manually finding those clauses across the internet is impossible at scale. This paper asks which modeling strategy — full fine-tuning, parameter-efficient tuning, or zero-shot prompting with commercial APIs — best balances accuracy, recall, and computational cost when classifying individual ToS clauses as fair or unfair.

Key Contributions

1. A unified three-paradigm comparison. Unlike prior work focused on a single approach (e.g., BERT fine-tuning alone or GPT prompting alone), the paper evaluates full fine-tuning (BERT, DistilBERT), 4-bit quantized LoRA (TinyLlama-1.1B, LLaMA-3B, LLaMA-7B, and the legal-domain SaulLM-7B), and zero-shot prompting (GPT-4o, GPT-4o-mini, O1-mini, O3-mini, O4-mini) under one experimental protocol and one held-out test set.
2. Benchmark evaluation on CLAUDETTE-ToS. Models were trained and scored on the 9,414-clause CLAUDETTE-ToS benchmark, with a balanced subset constructed to counter the original 89.1% fair / 10.9% unfair skew.
3. Web-scale deployment validation. The strongest model (BERT) was deployed on 937 English-language clauses drawn from the Multilingual Scraper of Privacy Policies and Terms of Service corpus, testing generalization to noisy, naturally occurring legal text rather than curated benchmarks.
4. Release of fine-tuned legal-domain models. The fine-tuned TinyLlama-ToS and SaulLM-ToS checkpoints are publicly released on Hugging Face.

Main Findings

• Full fine-tuning sets the performance ceiling. BERT reached an F1 of 89.20% (accuracy 88.86%, precision 89.20%, recall 89.20%) and DistilBERT reached an F1 of 89.58% (accuracy 89.00%, precision 89.87%, recall 89.31%) — nearly identical, balanced performance.
• Legal-domain LoRA buys recall at the cost of precision. SaulLM-7B, a legal-domain model pre-trained on over 19 million legal documents with LoRA adapters at rank 16, achieved the highest recall of any fine-tuned model at 97.50%, but precision dropped to 73.58% (accuracy 82.25%, F1 83.87%). This indicates broader clause coverage with lower specificity.
• Lightweight quantized models preserve precision but not recall. TinyLlama-1.1B showed the reverse pattern: precision 89.05% but recall only 52.50% (accuracy 73.02%, F1 66.06%).
• Mid-sized general LLaMA models underperformed. LLaMA-3B (F1 58.91%, accuracy 57.82%) and LLaMA-7B (F1 58.55%, accuracy 58.03%) fell below both the fine-tuned encoders and the other LoRA models, suggesting parameter scale alone does not deliver competitive fairness detection without domain-specific adaptation.
• Zero-shot models have high recall, low precision. All five API-accessible models achieved recall above 89%, but precision stayed low. O3-mini was the best balanced at 85.67% accuracy, 42.73% precision, 91.26% recall, and 58.20% F1. GPT-4o-mini followed at 78.03% accuracy, 32.07% precision, 90.29% recall, and 47.33% F1. GPT-4o reached 44.12% F1, O4-mini 50.82%, and O1-mini 44.19%.
• Deployment flagged a minority of clauses as unfair at scale. Across 937 clauses, 749 were predicted fair and 188 unfair. Model confidence ranged from 0.01 to 0.96 and keyword heuristics from 0.007 to 3.51. Filtering at ml_probability ≥ 0.5 or keyword_score ≥ 1.5 yielded 623 high-confidence ToS clauses, with 152 of the 188 unfair-labeled clauses coming from those sources — roughly 80%.
• BERT was chosen over the marginally better DistilBERT. Despite DistilBERT's higher F1 (89.58% vs. 89.20%), BERT was selected for deployment because it produced smoother probability distributions and fewer spurious activations on ambiguous clauses.
• A concrete case study. A clause from plagramme.com allowing the service to modify terms "at any time, at its sole discretion" was consistently flagged as potentially unfair because of unilateral change without notice, no user notification, and binary enforcement (accept or stop using the service).

Methodology in Plain English

The researchers took a labeled dataset of real contract clauses, each marked fair or unfair, and trained or prompted a range of language models to reproduce those labels.

For the supervised approaches, they used two small encoder models (BERT at 110M parameters and DistilBERT at roughly 40% fewer) trained end-to-end on the full dataset, and four larger decoder models adapted with LoRA — a technique that freezes the base model and trains small low-rank adapter matrices instead — under 4-bit quantization to fit them on limited hardware. SaulLM-7B used adapter rank 16 with gradient checkpointing at a learning rate of 5e-5, while TinyLlama-1.1B used 2e-4. Training ran on NVIDIA A100 and V100 GPUs via a SLURM cluster, with gradient accumulation and checkpointing to fit 7B-parameter models inside 24 GB of VRAM.

For the zero-shot condition, no training happened at all: batches of five clauses were fed to commercial instruction-tuned models through the OpenAI API using a standardized prompt, and the responses were post-processed into binary labels and scored on the same held-out test set.

Finally, the best supervised model was pointed at a much messier target: hundreds of clauses scraped from real websites, filtered to English, and combined with crawler heuristics to separate likely genuine ToS documents from noise.

Why This Matters

• Impact on research: The paper provides a single comparable reference point for three paradigms that are usually studied in isolation, and it extends evaluation beyond curated legal benchmarks to web-scale data — a step that matters because benchmark performance often does not survive contact with noisy real-world text.
• Real-world applications:
  • Consumer advocacy platforms that scan a service's terms and warn users about predatory clauses before they sign up.
  • Regulatory and watchdog bodies that audit platform contracts at internet scale, where manual review is infeasible.
  • Compliance teams inside companies that want to review their own terms before publication.
  • Contract-review tooling for legal practitioners handling large volumes of standard-form agreements.
• Industry relevance: The results map directly onto deployment economics. Full fine-tuning needs labeled data and GPUs but delivers the most reliable and best-calibrated output; LoRA variants offer a middle path where recall matters more than precision; zero-shot APIs allow immediate prototyping without any training pipeline but over-flag borderline cases, which is a poor fit for production legal analysis where false positives create review burden.

Future Directions

1. Multilingual extension. Extending the pipeline with cross-lingual adapters and translation alignment, since the deployed corpus is multilingual but evaluation was restricted to English-language documents.
2. Explanation generation. Integrating explanation modules or retrieval-augmented prompts so models not only flag a clause but supply a rationale — important for legal review where a bare label is hard to act on.
3. Adaptive ensembling. Developing strategies that dynamically select which model to use based on available resources and domain specificity.
4. Closing the precision gap for zero-shot. Zero-shot models reliably catch unfair clauses (recall above 89% across all five tested) but their precision lags far behind, leaving open the question of how to raise precision without sacrificing that recall or incurring fine-tuning costs.

Target Audience

NLP researchers working on legal text classification and parameter-efficient fine-tuning; legal-tech engineers deciding which modeling strategy fits their accuracy, latency, and hardware budget; compliance and regulatory professionals interested in automated contract auditing; and graduate students looking for a clear empirical comparison of full fine-tuning, LoRA, and zero-shot prompting on a real-world classification task.

Authors’ abstract

Large Language Models (LLMs) have transformed text understanding, yet their adaptation to specialized legal domains remains constrained by the cost of full fine-tuning. This study provides a systematic evaluation of fine tuning, parameter efficient adaptation (LoRA, QLoRA), and zero-shot prompting strategies for unfair clause detection in Terms of Service (ToS) documents, a key application in legal NLP. We finetune BERT and DistilBERT, apply 4-bit Low-Rank Adaptation (LoRA) to models such as TinyLlama, LLaMA 3B/7B, and SaulLM, and evaluate GPT-4o and O-versions in zero-shot settings. Experiments on the CLAUDETTE-ToS benchmark and the Multilingual Scraper Corpus show that full fine-tuning achieves the strongest precision recall balance, while LoRA-based models provide competitive recall with up to 3x lower memory cost. These findings highlight practical design trade-offs for efficient and domain-adapted LLMs, contributing open baselines for fine-tuning research in legal text processing.