What Transfers Under Source Shift? Definitions, Examples, and Fine-Tuning for Climate Disclosure Classification

AI summary

Overview

Research area: Natural language processing for sustainability and climate-finance disclosure analysis, specifically cross-source (out-of-distribution) adaptation of large language models.

Technical level: Intermediate. The findings are stated in plain terms, but readers will benefit from familiarity with prompting, few-shot examples, retrieval, and LoRA-style fine-tuning.

Scope: The paper reframes Task Force on Climate-related Financial Disclosures (TCFD) pillar classification as a shared-label cross-source adaptation problem and compares three adaptation strategies — definitions, examples, and fine-tuning — across eleven open- and closed-source LLMs on two corpora drawn from different disclosure sources.

What This Paper Is About

Climate disclosure classification assigns corporate text to the four TCFD pillars (Governance, Strategy, Risk Management, Metrics & Targets). Existing evaluations almost always train and test within a single disclosure source, usually annual and sustainability reports, even though practitioners also encounter press releases, earnings calls, and other investor materials that differ in length, purpose, and writing style. The paper asks which common LLM adaptation strategies still work when the classifier is moved from the source it was adapted on to a new, unseen disclosure source.

Key Contributions

1. A deployment-oriented source-shift study. The authors describe this as the first study of TCFD disclosure classification under source shift, formulating the deployment question as a shared-label cross-source evaluation problem.
2. A controlled experimental design. A single TCFD-CR adaptation pool (800 training and 200 validation instances) feeds both few-shot example selection and LoRA fine-tuning, so in-source and cross-source evaluation differ only in the evaluation set.
3. Practitioner-oriented evidence across eleven LLMs. The three strategies are swept across the Qwen, Llama, and Mistral open-weight families plus GPT-4o and GPT-5-mini, showing that strategies with the strongest in-source gains are not always the safest deployment choices.
4. Actionable defaults. The paper converts the findings into concrete guidance: match the definition prompt to the target text, prefer a small random few-shot pool over a similarity retriever, and keep a definition prompt at inference time even for fine-tuned models.

Main Findings

• All strategies help on average, but not equally. Definitions give the largest mean cross-source improvement, LoRA + D1 and random few-shot show moderate gains, and similarity retrieval barely improves over zero-shot. Per-model points show that for both random and similarity retrieval, some models fall below zero on TCFD-EX, meaning examples drawn from TCFD-CR can fail to help or even hurt on the new source.

• Strong in-source strategies lose the most under shift. LoRA + D1 produces the largest in-source gain, +14.1 pp, but retains only +3.7 pp on TCFD-EX. Similarity retrieval improves TCFD-CR by +8.0 pp but yields only +0.5 pp on TCFD-EX. The strongest in-source strategy is therefore not necessarily the strongest cross-source one.

• Definitions transfer most consistently. Definitions lie above the dashed diagonal in the in-source versus cross-source gain plane, meaning their cross-source gain exceeds their in-source gain; the authors interpret this as evidence that definitions supply TCFD label semantics rather than source-specific patterns from TCFD-CR. Random few-shot lies close to the diagonal, retaining most of its mean in-source advantage.

• More detailed definitions are not uniformly better. Averaged over the eleven prompted models, TCFD-CR peaks at D1 (the four official pillar descriptions) while TCFD-EX peaks at D2 (adding the 11 TCFD Recommended Disclosures). Both sources drop at D3 (the 27 criteria distilled from the TCFD implementation guidance), with the larger drop on TCFD-CR. At the model level, peaks scatter across D1–D3 on TCFD-CR but cluster at D2–D3 on TCFD-EX; the stable directional pattern is that no model peaks earlier on TCFD-EX than on TCFD-CR.

• Source-matched definitions help the matched source only. A CR-specific definition block (CRSEM20), constructed from 80 labelled TCFD-CR training texts at 270 words — the same length as D2 — scored 0.638 on CR versus D2's 0.563, a 7.5 pp improvement, but scored 0.638 on EX versus D2's 0.663, a −2.4 pp change. The authors note this control does not identify whether source, style, content, or annotation policy drives the effect.

• Better in-source retrieval does not transfer better. Using each model's best-on-CR (k, n) configuration for both settings, random few-shot improves macro-F1 by +2.9 pp on TCFD-CR and +2.3 pp on TCFD-EX. TF-IDF retrieval improves TCFD-CR by +8.0 pp but TCFD-EX by only +0.5 pp; dense retrieval improves TCFD-CR by +7.9 pp but TCFD-EX by only +1.0 pp. TF-IDF retrieval's gain on TCFD-CR exceeds its gain on TCFD-EX for all prompted models, and on some models TCFD-EX performance drops outright.

• The selection step, not the prompt, drives the gap. In an aligned comparison fixing the prompt, adaptation pool, and example budget at k = 8, n = 2, random few-shot shows no clearly stable in-source–cross-source gap, whereas TF-IDF and dense retrieval show a larger gap. Across the full k × n grid, similarity retrieval gives roughly 5–7 pp in-source versus at most 1 pp cross-source.

• Retrieval fails because similarity stops approximating label similarity. With coverage fixed, the benefit of similar correct-class examples decreases by 7.3 pp across sources, while the harm from similar other-class examples changes by only 1.0 pp. The net effect of full similarity retrieval falls from +7.6 to −0.7 pp. Class-omission rates were similar on both sources for dense retrieval (10.4% on CR vs. 11.2% on EX) and TF-IDF (10.1% vs. 8.9%).

• Fine-tuning is the in-source champion but not the cross-source champion. LoRA gives a large in-source gain on TCFD-CR for every fine-tuned base and outperforms the best prompted strategy for two of the four bases; for Qwen3.5-9B and Qwen3-32B the best prompted strategy is slightly higher. Evaluated on TCFD-EX with the training prompt (P0), the same adapters drop consistently across all four bases, ranging from −4.4 pp to −6.7 pp.

• Adding definitions at inference time partially recovers the loss. Keeping the trained adapters fixed and replacing P0 with the D1 definition prompt improves cross-source performance for all four bases, with gains from +0.4 to +2.4 pp. The in-source effect is mixed: D1 slightly reduces performance on Llama-3.1-8B and Qwen3-8B but improves Qwen3.5-9B and Qwen3-32B, with larger prompt-side gains on stronger base models.

• The direction of the result holds in reverse. In a reverse-direction control swapping the roles of CR and EX, same-source adaptation is higher in all six comparisons (TF-IDF, dense, and LoRA + D1 on both targets). These controls use smaller model subsets than the main study.

Methodology in Plain English

The authors use two corpora that share the same four-class TCFD label space but come from different disclosure sources. TCFD-CR, drawn from companies' annual and sustainability reports, provides 320 held-out test instances plus 1,000 instances split into 800 training and 200 validation. TCFD-EX, derived from the disclosure examples on the TCFD's example-disclosures list, supplies all 593 instances as the cross-source evaluation set. TCFD-CR items are long and discursive; TCFD-EX items are short and direct. A logistic regression on sentence embeddings separates the two with 81.2% five-fold accuracy, a proxy A-distance of approximately 1.25 out of 2.

In-source and cross-source evaluation share the same 1,000-instance TCFD-CR adaptation pool and differ only in the evaluation set, which isolates adaptations that help in-source but not cross-source. Macro-F1 is reported throughout. The authors test three strategies that inject source signal at different locations in the pipeline: definitions add TCFD label semantics to the prompt, examples insert labelled instances drawn from the pool, and fine-tuning updates adapter parameters from source labels. Each is evaluated on eleven LLMs on both the 320-instance in-source split and the 593-instance cross-source set.

The definition strategy is a four-level staircase evaluated zero-shot. D0 uses only the four label names; D1 adds the official pillar descriptions from the TCFD Recommendations; D2 adds the 11 TCFD Recommended Disclosures; D3 adds 27 criteria distilled from the TCFD implementation guidance. The prompt frame, target instance, and output constraint stay fixed, so only the definition block grows.

The example strategy draws from the adaptation pool while varying only the selection rule: zero-shot, random, TF-IDF similar, and dense similar. Random uses a fixed pool of eight examples (k = 8, n = 2, two per class). For the retrievers, k controls the total number of retrieved examples and n caps the maximum from any single class; the authors sweep k in {3, 5, 8} and n in {2, 3}, with the definition level fixed at D1. TF-IDF retrieval uses scikit-learn defaults, and dense retrieval uses all-mpnet-base-v2 sentence embeddings.

For fine-tuning, LoRA adapters are trained on four open-source bases (Llama-3.1-8B, Qwen3-8B, Qwen3-32B, and Qwen3.5-9B) using the 800 training and 200 validation instances converted to chat-format classification examples, each with a minimal label-name prompt (P0). The configuration uses 4-bit quantization (QLoRA), rank r = 16, scaling factor alpha = 32, dropout 0.05, learning rate 2 × 10⁻⁴, effective batch size 16, and up to five epochs with best-checkpoint selection on validation macro-F1, over three seeds. The trained adapters are then re-evaluated with the D1 definition prompt substituted at inference time.

Implementation runs open-weight models with vLLM 0.19.1 on 4 × NVIDIA A100 40 GB GPUs, and queries GPT-4o and GPT-5-mini through the OpenAI Chat Completions API with temperature 0. Outputs are parsed against a strict four-label format, and unparseable outputs are marked INVALID. LoRA training uses PEFT with 4-bit loading, BF16 compute, and gradient checkpointing.

Why This Matters

Impact on research. The paper shows that single-source leaderboard evaluation in climate-disclosure NLP can be misleading: the strategy that wins in-source is often the one that transfers least. It connects the climate-NLP literature to broader distribution-shift work showing that in-domain performance overestimates out-of-distribution accuracy even when the label space is held fixed, and it provides a controlled shared-label protocol that other disclosure tasks could reuse.

Real-world applications:

• Screening press releases and earnings call transcripts for TCFD-aligned climate content, where models are typically adapted on annual or sustainability reports.
• Supporting climate-finance analysts who need consistent pillar-level tagging across a heterogeneous document stream rather than a single report type.
• Informing greenwashing-oriented and ESG analysis pipelines that depend on downstream disclosure classification.
• Guiding practitioners who must pick an adaptation strategy before seeing the target source, where the choice of retriever or prompt can determine whether performance holds or degrades.

Industry relevance. The paper's practical recommendation is that simpler is often safer under source shift. Firms that fine-tune a model on their available report corpus and deploy it on new disclosure types can partially recover lost cross-source performance by keeping a definition prompt at inference time, without retraining. Choosing a small random few-shot pool over a similarity retriever reduces the risk that a strong in-source configuration becomes unreliable in deployment.

Future Directions

• Broaden the corpus pair. The authors note that their cross-source evaluation rests on a single dataset pair — TCFD-CR for adaptation and TCFD-EX for evaluation — which they describe as the only publicly available datasets sharing the TCFD four-pillar label space across sources. Additional source pairs would test whether the findings generalize.
• Disentangle what drives the source gap. The CR-specific definition control and the diagnostic retrieval intervention cannot separate style, content, and annotation-policy differences between sources. Isolating these factors remains open.
• Design weights and prompts jointly. Because cross-source robustness of fine-tuned models depends on both learned weights and the inference-time prompt, the paper argues these should be treated as a joint design problem rather than evaluated in isolation.
• Explain the definition-granularity effect. The directional finding that the TCFD-EX target benefits from more specific definitions (no model peaks earlier on TCFD-EX than on TCFD-CR) is established empirically but is illustrated only through two Qwen3-32B case examples, leaving room for a more systematic account.

Target Audience

Practitioners and researchers working on climate and sustainability disclosure NLP who need to deploy classifiers across document types; NLP researchers studying domain and distribution shift under a fixed label space; and financial or ESG analysts who consume pillar-level TCFD classifications and want to know when an adapted model can be trusted on a new disclosure source.

Authors’ abstract

Climate disclosure classification is a fundamental task for analysing corporate climate disclosures, yet such disclosures appear in many different sources -- annual reports, press releases, and earnings calls -- that differ in length, purpose, and writing style. Existing evaluations are mostly conducted within a single source, leaving open whether common LLM adaptation strategies remain effective under source shift. We reframe climate disclosure classification as a cross-source adaptation problem and study three widely used adaptation strategies -- definitions, examples, and fine-tuning -- across eleven open- and closed-source LLMs, using two corpora that share the same label space but come from different sources. We find that all strategies bring positive cross-source gains on average, but the strongest in-source strategies are not the strongest cross-source ones: similarity-based retrieval and LoRA fine-tuning gain most in-source but lose most of that advantage under source shift; randomly selected few-shot examples, a weaker in-source baseline, retain their advantage more reliably; definitions transfer most consistently, though only when their granularity matches the target text. Across these strategies, when the source changes, simpler is often safer.