CLINB: A Climate Intelligence Benchmark for Foundational Models

AI summary

Overview

Research area: AI evaluation and benchmarking, specifically expert-grounded assessment of large language models (LLMs) in the domain of climate change science.

Technical level: Intermediate. Readers need familiarity with LLM evaluation concepts such as pairwise preference judgments, ELO ratings, LLM-as-a-Judge, rubrics, and retrieval-augmented generation.

Scope: The paper introduces CLINB (Climate Intelligence Benchmark), a benchmark built from real user climate questions and climate-scientist-validated grading rubrics, and uses it to evaluate six frontier models plus human-AI "hybrid" answers on open-ended, multimodal question answering with requirements for evidence and attribution.

What This Paper Is About

Most "hard" LLM benchmarks rely on closed-form tasks such as multiple-choice or short numeric answers, which are easy to grade automatically but represent only a fraction of real-world use. CLINB addresses this by evaluating how foundational models handle open-ended, generative, multimodal climate question answering, where answers need research, evidence assessment, and synthesis. The goal is to measure both knowledge quality and whether responses are properly grounded in verifiable references and images.

Key Contributions

1. A new expert-grounded benchmark for scientific AI. CLINB centers on a dataset of real-world climate questions paired with question-specific evaluation rubrics, curated and validated by leading climate scientists through a three-phase, human-in-the-loop process. System prompts, questions, and model judge prompts are made available at a Kaggle dataset link; the underlying questions came from chatclimate.ai logs.

2. A demonstration of "PhD-level synthesis vs. attribution failures." Frontier models show remarkably high knowledge synthesis, often described as PhD-level understanding, but this masks weak grounding: substantial hallucination rates for references (10% to 25%) and even more failures for images (50% to 80% in certain settings).

3. Insights into human-AI collaboration dynamics. Autonomous frontier models surpass "hybrid" answers curated by experts using weaker AI assistance, suggesting the assisting model's capability, not human oversight, is the primary bottleneck. Non-specialists who deeply engaged with AI tools (the "Advocates") produced higher-quality answers than domain experts who engaged less with AI.

4. A validated methodology for scalable oversight. The paper validates a rubric-based autorater and reports ablation studies showing that structured prompts and automated evidence-checking are essential for mitigating LLM judge biases, while also identifying evaluation challenges including model familiarity bias in human raters and limited rubric generalization across models.

Main Findings

• Autorater rankings. On overall side-by-side preference under the CLINB autorater, GPT-5 scored highest at 1150 ± 19, followed by Claude Opus 4.1 at 1135 ± 19, GPT o3 at 1018 ± 18, Gemini 2.5 Pro at 969 ± 18, Hybrid at 945 ± 18, Claude Sonnet 4 at 915 ± 19, and Gemini 2.5 Flash at 868 ± 18.

• Human expert rankings differ. In a manual validation over a sample of 1976 battles from the top 5 systems, using three Expert raters per battle, Claude Opus 4.1 ranked highest at 1115 ± 20, ahead of Gemini 2.5 Pro (1015 ± 18) and Hybrid (1015 ± 20), with GPT o3 at 950 ± 20 and GPT-5 at 906 ± 18. On 72 sampled disagreement battles, the Scientists sided with the autorater rather than the Experts, preferring GPT-5 and Claude Opus 4.1 for "Knowledge" and "Presentation."

• Attributed disagreement to familiarity bias. The authors attribute the Experts' preference to familiarity bias from overexposure to Gemini and hybrid outputs during data creation, and note that the Experts also penalized GPT-5 and OpenAI o3 for lacking images, which the Scientists consider a secondary epistemological factor.

• Hybrid answers beat their base model but not top models. Hybrid answers outperformed Gemini 2.5 Flash (the assisting model) and Claude Sonnet 4, and were close to Gemini 2.5 Pro, but ranked lower than top models overall and specifically on knowledge and presentation by the autorater, Scientists, and Experts.

• Evidence quality varies sharply. Claude Opus 4.1 was by far the best for quantity and validity of citations and had the highest rate of paywalled content, likely peer-reviewed sources, followed by GPT-5, Claude Sonnet 4, and Gemini 2.5 Pro. Roughly 25% of OpenAI o3's URLs were hallucinated.

• Image failures are worse than reference failures. Hybrid answers stood out for superior visuals while LLM answers often had poor visual choices or none. When image inclusion was made mandatory, hallucinated link rates ranged between 50% and 80%. The autorater's image scores for GPT-5 were described as possibly unjustified given GPT-5 provides no image links, possibly reflecting anchoring bias.

• Rubrics matter mainly at the margins. Removing question-specific rubrics from the autorater prompt changed results only in the bottom half of the ranking, with Hybrid answers overtaken by Gemini 2.5 Flash and Claude Sonnet 4, suggesting the extra resolution from rubrics applies mainly to the kinds of responses used to develop them and that rubrics are far from complete.

• Evidence-checking is hampered by inaccessible sources. The process is limited by inaccessible sources in up to 50% of cases.

• Advocates produced strong answers. In Phase 1, Experts rated the Advocates' answers higher than any other answer source, including their own, and Editor usage data showed Advocates engaged much more with all aspects of curation, including AI assistance.

• Qualitative strengths and weaknesses. The paper reports an informal, qualitative table of strengths and weaknesses across GPT-5, OpenAI o3, Claude Opus 4.1, Gemini 2.5 Pro, and Hybrid answers, derived by using Gemini 2.5 Pro to extract recurring patterns from autorater justifications; the authors explicitly label these characterizations as qualitative and informal.

• Data scale. The dataset contains 1330 candidate answers, 8654 pairwise preferences (2.17 on average per unique answer pair), and 4147 battles. Of candidate answers, 82.6% have images and 99.7% have references, averaging 6.1 references; the candidate answer set contains 8086 references and 1176 images in total.

• Best-answer analysis. Hybrid answers were the best answer 70.3% of the time, LLM answers 19.1%, and Merged answers 10.5%. PageRank-based metrics gave Hybrid a Precision@1 of 70.3 and MRR of 0.84, versus LLM at 19.1 and 0.42, and Merged at 10.5 and 0.32.

• Question composition. Among final questions, 35.7% were labeled High Confidence, 35.1% Advanced, and 29.2% Open, with a slight prevalence of climate science (WGI) questions and the Impacts, Detection, and Scenarios topics.

Methodology in Plain English

The researchers first collected real user questions from chatclimate.ai, spanning six climate topics: Weather and Climate Extremes, Mitigation Pathways, Detection, Attribution and Uncertainty, Climate Finance and Risks, Climate Impacts, Adaptation and Vulnerability, and Climate Change Scenarios. Scientists selected roughly 30 questions per topic and labeled each by difficulty (High Confidence, Advanced, Open) and by relevant IPCC working group.

They then organized three groups of people: six Scientists (five climate scientists with lead roles in IPCC and NCA reports, plus a climate finance expert), 40 Experts recruited as active academics, mostly PhD students and postdocs, and 17 Advocates from the Climate Fresk community. All work ran through a dedicated AI-assisted interface called the Editor, which used Gemini 2.5 Flash as assistant and Google for web search, with bibliographic metadata from OpenAlex.

In Phase 1, humans curated an outline and then a full first draft for each question, starting from model-generated versions. Each question received at least 3 hybrid answers (92% curated by Experts, 8% by Advocates) plus an independently generated LLM answer and a Merged answer, for N+2 answers in total. In Phase 2, humans compared answers side by side and recorded preferences, avoiding answers they had authored. In Phase 3, Gemini 2.5 Pro drafted a rubric per question, combining a Cheat Sheet and Grading Rubric, using all answers, images, references, quality guidelines, and the preference graph as input; Scientists then manually curated the final rubrics.

For evaluation, the team submitted one request per question to each model using a system prompt and evaluated answers with a Gemini 2.5 Pro judge at temperature 0.7, referred to as the CLINB autorater. Battles paired two answers, ran two evaluations with the order swapped to control for position bias, and awarded the win by majority or declared a tie. ELO scores were estimated with the Bradley-Terry model and bootstrap 95% confidence intervals. The judge also checked the validity of reference and image links, classifying them as valid or invalid and factoring that into its assessment.

Why This Matters

The paper argues that the gap between strong knowledge synthesis and unreliable attribution is a central obstacle to deploying AI in scientific workflows, and that reliable, interpretable benchmarks like CLINB are needed to track progress toward trustworthy systems.

Real-world applications:

• AI-assisted science and scientific communication, where models summarize and explain complex, debated literature.
• Decision-making and policy support that depends on traceable evidence, given the domain's reliance on institutional reports and consensus processes.
• Climate finance and risk analysis, one of the six topics covered in the question set.
• Expert-AI collaborative workflows, including training people to critically assess AI-generated content and verify claims against ground truth.

Industry relevance: The findings are directly relevant to anyone building or deploying retrieval-augmented or citation-generating systems, since hallucinated references and images undermine trust even when the underlying reasoning is strong. They also matter for organizations designing evaluation pipelines, because the paper shows that model-based judges carry biases—position bias, style-over-substance bias, AI-AI bias, anchoring bias, and familiarity bias in human raters—that structured prompts and evidence-checking can partially counteract.

Future Directions

• Evaluate search-enabled systems, since the current experiments used public APIs with default settings and search disabled.
• Make rubrics adaptive, because rubrics were found to be far from complete and did not generalize evenly across models and response types.
• Apply deeper scrutiny to evidence, extending attribution checking beyond link validity toward richer verification of claims.
• Co-evolve benchmarks with models, anticipating future systems that incorporate dataset retrieval, statistical analysis, and image synthesis.
• Design interfaces that enable genuine expert-AI collaboration, perhaps through continuous interaction and mutual questioning, to produce synergistic performance beyond what models achieve alone.

Target Audience

This paper is most useful for AI evaluation researchers and benchmark designers, climate scientists and domain experts interested in AI-assisted scientific communication, developers building citation-grounded or multimodal retrieval-augmented systems, and policy or institutional audiences concerned with whether AI systems can be trusted to represent scientific evidence accurately.

Authors’ abstract

Evaluating how Large Language Models (LLMs) handle complex, specialized knowledge remains a critical challenge. We address this through the lens of climate change by introducing CLINB, a benchmark that assesses models on open-ended, grounded, multimodal question answering tasks with clear requirements for knowledge quality and evidential support. CLINB relies on a dataset of real users' questions and evaluation rubrics curated by leading climate scientists. We implement and validate a model-based evaluation process and evaluate several frontier models. Our findings reveal a critical dichotomy. Frontier models demonstrate remarkable knowledge synthesis capabilities, often exhibiting PhD-level understanding and presentation quality. They outperform "hybrid" answers curated by domain experts assisted by weaker models. However, this performance is countered by failures in grounding. The quality of evidence varies, with substantial hallucination rates for references and images. We argue that bridging this gap between knowledge synthesis and verifiable attribution is essential for the deployment of AI in scientific workflows and that reliable, interpretable benchmarks like CLINB are needed to progress towards building trustworthy AI systems.