ESGBench: A Benchmark for Explainable ESG Question Answering in Corporate Sustainability Reports

AI summary

Overview

Research area: Natural Language Processing / financial AI — specifically question answering over corporate sustainability disclosures, with retrieval-augmented generation (RAG) and benchmark construction. Technical level: Intermediate. Scope: The paper introduces ESGBench, a small open benchmark and reproducibility pipeline for evidence-grounded ESG question answering over ESG/TCFD reports, plus a RAG baseline evaluated with exact match, string F1, unit-aware numeric accuracy, and retrieval recall.

What This Paper Is About

Corporate ESG (Environmental, Social, Governance) and TCFD sustainability reports are long, heterogeneous PDFs mixing narrative text with numeric tables, which makes them hard to analyze at scale for both analysts and machines. Large Language Models show promise here, but there has been no standardized way to measure whether their answers are accurate and grounded in the source document. The authors build ESGBench: a public, reproducible benchmark that ingests these reports, generates questions with verbatim supporting evidence, and scores any QA system on accuracy, numeric correctness, and retrieval quality.

Key Contributions

1. An open, reproducible pipeline that collects ESG/TCFD reports, downloads the PDFs, and builds a chunked, table-aware searchable index and table cache (released at github.com/sherinegeorge21/ESGbench).
2. Automatically generated, evidence-grounded QA pairs. A constrained prompting scheme produces questions answerable only from the supplied text, with verbatim answers (numbers and units preserved) and a verbatim evidence quote with page number; pairs are de-duplicated by hashing (doc_name | question | answer).
3. A multi-metric evaluation suite reporting exact match (EM), token-level string F1, unit-aware numeric accuracy with a ±2% tolerance, and retrieval recall@K against gold evidence pages, including per-category (Environmental/Social/Governance/Strategy/Risk) breakdowns.
4. A RAG baseline and initial corpus (ESGBench v0.1): 10 companies, 12 PDF reports, and 119 QA pairs, with 40–50% of questions table-derived, spanning technology, consumer, energy, finance, and manufacturing sectors and multiple regions.

Main Findings

• Baseline accuracy is modest: Evaluating all 119 QA pairs, the RAG baseline reaches 21.0% Exact Match, 55.4% average string F1, and 45.3% numeric accuracy at ±2% tolerance.
• Category performance varies sharply: Strategy questions are comparatively easy (90.9% EM), followed by Environmental (48.0%), Governance (43.5%), and Social (35.7%), which the authors attribute to short, factual spans being easier than numeric ones.
• Numeric KPIs and tables are the weak point: Environmental numerics drawn from tables are error-prone because of formatting differences, unit conversion, and scale words such as "thousand"; unit variants like "tCO2e" and "ktCO2e," or "million," create fragile failure modes.
• Retrieval quality is reported as a first-class signal: Retrieval Recall@5 is listed as 70–80% (the table labels this "if logged"), and the authors argue retrieval gaps propagate into answer errors, so recall@K should be reported alongside generation metrics.
• Corpus composition: ESGBench v0.1 contains 10 companies, 12 reports, 119 QA pairs, and an average of 9.9 QA pairs per report; the category split across Environmental / Social / Governance / Strategy / Risk is 50 / 14 / 23 / 11 / 2, with 40–50% of QAs derived from tables.
• Deliberate design caveat: The authors state that ESGBench is "small by design but easily extensible," and the Table 1 caption instructs readers to replace counts with their latest run, signalling that the numbers reflect an initial release.

Methodology in Plain English

The authors started with a seed list of widely available sustainability reports from large, publicly known companies across several sectors and regions, recording metadata such as company, year, URL, document type, country, and industry in a CSV; dead or forbidden links are kept as metadata but skipped downstream.

Each downloaded PDF is turned into two representations. The first is passage chunks of 600–1200 characters tagged with page numbers. The second is table rows extracted with pdfplumber or Camelot and normalized with optional value and unit fields. Both keep document names and page indices so that any answer's evidence can be audited.

Questions are then generated under tight constraints: a question must be answerable only from the provided passage or table summary; the answer must be verbatim, preserving digits and units; and an evidence quote must be copied exactly. Duplicate question-answer pairs are removed by hashing.

For evaluation, the authors built a simple two-part RAG system: a retriever that embeds chunks with a general-purpose embedding model and returns the top-k chunks, and a reader that is prompted to produce a single span answer drawn from retrieved text, preserving digits and units for numeric questions. Predictions are logged with the document name, question, predicted answer, and retrieved pages, so recall@K can be computed alongside EM, string F1, and numeric accuracy.

Why This Matters

Impact on research: ESG analytics has lacked a shared, reproducible way to compare systems. ESGBench makes retrieval recall against human-locatable evidence a first-class metric, and requires verbatim evidence strings — pushing the field away from ungrounded answer generation and toward explainable, auditable extraction.

Real-world applications:

• Investment and credit analysts extracting and comparing Scope 1–3 emissions, renewable energy use, and diversity metrics across many companies.
• Portfolio and risk teams monitoring whether disclosures change year over year, where numeric precision and unit handling matter.
• Regulators, auditors, and assurance providers needing to trace a stated figure back to the exact page and passage of a report.
• ESG data vendors and rating agencies building or testing automated extraction pipelines against a common yardstick.

Industry relevance: The authors are affiliated with BNY and FedEx, and the paper is positioned for the AI and Data Science for Digital Finance Workshop at ACM ICAIF 2025 (Singapore, November 15, 2025), reflecting direct relevance to financial institutions that must process sustainability disclosures at scale. The authors also note that ESGBench measures only whether a system recovers what a report says — not whether the corporate claims themselves are true — which is an important boundary for any compliance or audit use.

Future Directions

1. Human validation of a stratified subset of the automatically generated QA pairs, to quantify generation noise and context loss.
2. Multilingual report coverage, since the seed corpus currently favors English-language reports from large multinationals and therefore biases topics and style.
3. Alignment to reporting taxonomies such as CSRD, BRSR, and ISSB, to support coverage metrics rather than only question-level accuracy.
4. Better table and unit handling, including layout-aware table parsing, plus robustness tests for prompt sensitivity and retrieval noise.

Target Audience

NLP and financial-AI researchers building or evaluating question answering and RAG systems over long, table-heavy documents; ESG data scientists and quantitative analysts at banks, asset managers, and data vendors; and practitioners in sustainability reporting, audit, or regulatory roles who need to understand what automated extraction from sustainability reports can and cannot currently deliver. Readers should be comfortable with standard QA evaluation metrics and basic retrieval-augmented generation concepts; the paper is accessible to newcomers in that it defines each metric explicitly, but its focus is on benchmark design and measurement rather than on model architecture.

Authors’ abstract

We present ESGBench, a benchmark dataset and evaluation framework designed to assess explainable ESG question answering systems using corporate sustainability reports. The benchmark consists of domain-grounded questions across multiple ESG themes, paired with human-curated answers and supporting evidence to enable fine-grained evaluation of model reasoning. We analyze the performance of state-of-the-art LLMs on ESGBench, highlighting key challenges in factual consistency, traceability, and domain alignment. ESGBench aims to accelerate research in transparent and accountable ESG-focused AI systems.