TermGPT: Multi-Level Contrastive Fine-Tuning for Terminology Adaptation in Legal and Financial Domain

AI summary

Overview

Research area: Natural Language Processing — domain-specific terminology understanding, contrastive representation learning, and parameter-efficient fine-tuning of LLMs for legal and financial text.

Technical level: Intermediate. The paper assumes familiarity with sentence embeddings, contrastive learning objectives (InfoNCE-style), supervised fine-tuning, and multiple-choice evaluation, but explains its terminology-aware fine-tuning task from first principles.

Scope: The paper defines a new "terminology-aware fine-tuning" task, proposes the TermGPT multi-level contrastive fine-tuning framework (sentence-level plus token-level), and evaluates it on a newly constructed Chinese financial regulatory dataset and the legal-domain JecQA benchmark.

What This Paper Is About

Large language models often produce embedding spaces that are distributed too uniformly in high-dimensional space — an "isotropy problem" — which makes it hard for them to separate domain-specific terms that look superficially similar. In high-stakes finance and law, confusing two related terms can cause real damage; the paper's motivating example is a banking LLM that misreads the term "supervisor" as "executive supervisor" during a loan review, overlooks a required disclosure to the supervisory board, and wrongly approves the loan. The paper's goal is a fine-tuning framework that sharpens an LLM's discrimination of specialized terms while still preserving global sentence context.

Key Contributions

1. A new task definition. The authors formalize terminology-aware fine-tuning as a standalone objective for improving fine-grained understanding of domain-specific terminology. They state this is the first work to formalize terminology-aware fine-tuning as an independent task.
2. A novel framework, TermGPT. A multi-level contrastive fine-tuning framework that jointly models global context (sentence-level contrastive learning) and local term semantics (token-level contrastive learning), intended to help with sparse and imbalanced terminology.
3. A specialized dataset. A new financial terminology dataset built from 425 authoritative financial regulatory documents. The contributions section describes it as consisting of 3,647 domain-specific terms; the paper states this is the first financial terminology dataset derived from official regulatory documents.
4. Extensive experiments. Evaluations on the new financial dataset and the legal-domain JecQA benchmark, reporting average improvements of 6.14% on terminology Question-Answering (QA) tasks and 2.60% on terminology Question-Choice Answering (QCA) tasks.

Main Findings

• Three motivating challenges are identified. (CH1) Supervision signals for terminology are ambiguous because terms are highly context-dependent; (CH2) terminology is extremely sparse — the paper reports terms constitute only 0.08% of the JecQA dataset; (CH3) there is a trade-off between over-relying on context (which can obscure term-specific meaning) and ignoring it (which isolates terms from functional usage).

• QCA performance (RQ1). Contrastive learning-based methods slightly trail commercial LLMs but still outperform Lawyer-LLaMA by 63.00% and Xuanyuan by 17.69% on average. TermGPT outperforms the best baseline by 2.60% on average. Concretely, TermGPT-Qwen3 (8B) reaches 0.858 accuracy on JecQA and 0.908 on Financial Regulations, with reported improvements of +2.87% and +2.36% respectively (p-values 0.008 and 0.001 from a paired t-test at the 0.05 level). TermGPT-LlaMA (8B) reaches 0.681 on JecQA and 0.856 on Financial Regulations.

• QA performance (RQ2). TermGPT outperforms the best baseline by 6.14% on average. For TermGPT-Qwen3 (8B), JecQA scores are BLEU-1 0.266, BLEU-4 0.125, BERTScore 0.716, ROUGE-1 0.376, ROUGE-L 0.316, and METEOR 0.222; Financial Regulations scores are BLEU-1 0.274, BLEU-4 0.126, BERTScore 0.708, ROUGE-1 0.371, ROUGE-L 0.302, and METEOR 0.201. Reported improvements over the best baseline are +1.92%, +2.46%, +0.28%, +0.27%, +0.64%, and +0.91% on JecQA, and +15.13%, +18.87%, +1.28%, +11.08%, +16.15%, and +4.69% on Financial Regulations. Domain-pretrained models (Lawyer-LLaMA, Xuanyuan) lack task adaptation, which the authors say limits their effectiveness despite domain pretraining.

• Backbone choice matters (Result 3). Qwen3 outperforms LLaMA as a backbone. TermGPT-Qwen3 yields an average gain of 15.98% on the QCA task and 43.52% on the QA task compared to TermGPT-LLaMA, so Qwen3 is adopted as the default backbone.

• Ablation results (RQ3). Removing token-level contrastive learning drops average performance by 1.31%; removing sentence-level contrastive learning drops it by 1.28%; removing contrastive learning entirely drops it by 3.81%; removing SFT drops it by 8.55%. The authors conclude both SFT (task adaptation) and multi-level contrastive learning (term discrimination) are essential. Ablation accuracies: full TermGPT 0.858 (JecQA) and 0.908 (Financial); w/o SFT 0.796 and 0.818; w/o CL 0.834 and 0.864; w/o token-level CL 0.844 and 0.899; w/o sentence-level CL 0.849 and 0.894.

• Cross-domain robustness (RQ4). Performance stays balanced across sub-domains. In the QCA task, Economic Law and Risk Management score slightly lower, attributed to limited training data. In the QA task, Civil Law lags slightly, attributed to its broader scope and greater ambiguity.

• Note on the improvement figures. The paper's footnotes state that improvements are computed over the best-performing baselines excluding commercial LLMs, because those have substantially larger parameter scales.

Methodology in Plain English

The approach has three moving parts.

Building a sentence graph. Sentences become nodes. Edges encode two kinds of relationship: edge_sen, connecting sentences that contain the same entities or belong to the same type; and edge_token, connecting sentences whose extracted entities are similar enough that their embeddings exceed a threshold θ (the paper uses θ_sen for the sentence-level threshold). Entities are extracted from each sentence by combining an LLM with a schema.

Graph-driven data augmentation. Each node is treated as an anchor sentence. Neighbors reachable by a sentence edge are added to a sentence-level candidate set if their sentence embedding similarity to the anchor exceeds a threshold. Neighbors reachable by a token edge contribute their similar entities directly to a token-level candidate set. An LLM then generates questions from these candidate sets, producing Question-Choice Answering (QCA) samples: for the sentence set, the anchor is the correct option and a connected sentence is an incorrect option, plus two hard negative samples; for the token set, the anchor entity is correct and the other candidate entities are incorrect options. This is how the model gets diverse, informative positive and negative pairs despite terminology being extremely sparse.

Multi-level contrastive learning. The pipeline runs three stages. First, supervised fine-tuning (SFT) over both the sentence and token sample sets, minimizing cross-entropy so the model learns the mapping from question to target answer. Second, sentence-level contrastive learning: the model generates embeddings for each question with its correct and incorrect answers, activates bidirectional attention to integrate context, and optimizes an InfoNCE-style loss that pulls the question embedding toward the correct answer and away from the three incorrect options, with a temperature parameter τ. Third, token-level contrastive learning: the question-answer pair is rewritten by the LLM into a declarative sentence where the correct answer appears as a subsequence. The correct answer is treated as a positive sample and each incorrect answer as a negative sample; a mixing function builds a sequence where the positive is replaced by a negative, and a binary indicator sequence marks which tokens came from the positive/original text and which from the negative. A masking-style loss over this mixed sequence then teaches the model to distinguish the two, and the token-level loss aggregates over all negative candidates.

The three stages are chained into a single optimization pipeline (given as pseudocode in Algorithm 1). Training configuration details are stated to be in Appendix B; they are not included in the content available here.

Why This Matters

Impact on research. The paper argues that terminology-level representation quality is a bottleneck that general embedding methods and sentence-level contrastive learning do not address well, because sentence-level objectives tend to dilute token-level semantics while token-level objectives tend to lose global context. TermGPT's contribution is a framework that supervises both levels at once, and the formalization of terminology-aware fine-tuning gives the community a named task to benchmark against. The release of a financial terminology dataset built from official regulatory documents, plus the code at the project's GitHub repository, provides concrete evaluation infrastructure for a domain that is currently underserved.

Real-world applications.

• Loan review and credit approval, where confusing a supervisory role with an executive role can lead to incorrect approvals.
• Compliance advisory services, where precise reading of regulatory terms determines whether an action is permissible.
• Legal judgment prediction and legal QA, where subtle distinctions between closely related legal concepts drive outcomes.
• Financial risk analysis, where the paper names risk analysis as a downstream task that depends on terminology-level discrimination.

Industry relevance. Banks, regulators, and legal-tech vendors operate under regimes where a single misread term can cause financial loss, regulatory penalties, and reputational risk — the paper's own framing of the loan-application failure. The result that an 8B model can approach or match commercial LLMs of 72B and 671B parameters on these tasks (as measured by the reported metrics) is directly relevant for institutions that cannot deploy very large models for cost, latency, or data-residency reasons.

Future Directions

• Adversarial robustness. The authors explicitly name this as the important next direction: in high-stakes domains, adversarial inputs may distort how LLMs interpret key terminology, and they suggest exploring adversarial training or robust optimization.
• Extending the framework beyond finance and law. The paper tests finance (Financial Regulations) and law (JecQA); whether the sentence-graph augmentation and multi-level contrastive design transfer to other terminology-dense domains such as medicine or engineering is not reported.
• Improving sub-domains with limited data. Economic Law and Risk Management in QCA and Civil Law in QA underperform relatively, which the authors attribute to limited training data and greater ambiguity respectively; these are open gaps rather than solved problems.
• Understanding the LLaMA–Qwen3 gap. The 15.98% QCA and 43.52% QA advantage of TermGPT-Qwen3 over TermGPT-LLaMA is reported but not explained, leaving open which backbone properties make terminology-aware fine-tuning effective.

Target Audience

This paper is most useful to NLP researchers working on domain adaptation, contrastive representation learning, and embedding-space quality; to practitioners building legal or financial LLM applications where term-level precision matters; and to teams that need strong terminology discrimination from smaller open-weight models rather than large commercial APIs. Readers who want to reproduce or extend the work will need the appendices (data collection in Appendix A, model configuration and training setup in Appendix B) and the accompanying code repository, since the training hyperparameters are not present in the main text.

Authors’ abstract

Large language models (LLMs) have demonstrated impressive performance in text generation tasks; however, their embedding spaces often suffer from the isotropy problem, resulting in poor discrimination of domain-specific terminology, particularly in legal and financial contexts. This weakness in terminology-level representation can severely hinder downstream tasks such as legal judgment prediction or financial risk analysis, where subtle semantic distinctions are critical. To address this problem, we propose TermGPT, a multi-level contrastive fine-tuning framework designed for terminology adaptation. We first construct a sentence graph to capture semantic and structural relations, and generate semantically consistent yet discriminative positive and negative samples based on contextual and topological cues. We then devise a multi-level contrastive learning approach at both the sentence and token levels, enhancing global contextual understanding and fine-grained terminology discrimination. To support robust evaluation, we construct the first financial terminology dataset derived from official regulatory documents. Experiments show that TermGPT outperforms existing baselines in term discrimination tasks within the finance and legal domains.