SciTrust 2.0: A Comprehensive Framework for Evaluating Trustworthiness of Large Language Models in Scientific Applications

AI summary

Overview

• Research area: Trustworthiness evaluation and benchmarking of large language models (LLMs) for scientific research applications.
• Technical level: Intermediate — readers need basic familiarity with LLM benchmarking, evaluation metrics (ROUGE, BERTScore, BARTScore, LLM-as-judge), and adversarial attack methods.
• Scope: The paper introduces SciTrust 2.0, a four-dimension framework (truthfulness, adversarial robustness, scientific safety, scientific ethics) used to evaluate seven LLMs — four science-specialized and three general-purpose industry models — on text-based scientific tasks.

What This Paper Is About

Scientific LLMs are increasingly used to digest literature, generate hypotheses, and solve technical problems, yet there has been no single framework that assesses their trustworthiness across factual accuracy, robustness, safety, and research ethics in a scientific context. SciTrust 2.0 fills that gap by combining existing benchmarks with two new synthetic benchmarks (open-ended scientific Q&A built through a reflection-tuning pipeline, and an ethics benchmark covering eight research-ethics areas) and applying them to seven models. The central question is whether models specifically trained on scientific content are actually more trustworthy for scientific use than general-purpose industry models.

Key Contributions

1. SciTrust 2.0 framework — a comprehensive evaluation framework for LLM trustworthiness in scientific applications across four dimensions: truthfulness, adversarial robustness, scientific safety, and scientific ethics.
2. Novel open-ended truthfulness benchmarks — synthetic scientific Q&A pairs generated via a two-stage reflection-tuning pipeline and validated by expert scientists, improving on the original SciTrust benchmarks.
3. A novel scientific ethics benchmark — synthetic ethical scenarios spanning eight areas of research ethics, including dual-use research and bias, generated with GPT-o3-mini-high and manually quality-verified.
4. A comparative analysis of seven models — four science-specialized models (SciGLM-6B, Galactica-120B, FORGE-L, Darwin1.5-7B) and three general-purpose industry models (Llama4-Scout-Instruct, Claude-Sonnet-3.7, GPT-o4-Mini), revealing strengths and limitations across all trustworthiness dimensions. The framework is open-sourced at https://github.com/herronej/SciTrust.

Main Findings

• General-purpose industry models outperformed science-specialized models across every trustworthiness dimension. Despite being trained specifically on scientific content, the specialized models scored lower on scientific knowledge, logical reasoning, adversarial robustness, safety, and ethics.
• GPT-o4-mini led on truthfulness and robustness. It achieved the highest multiple-choice accuracy (including 100.00% on MMLU College Computer Science, 97.94% on ARC-C, 97.22% on MMLU College Biology) and the highest LLM-as-judge scores on the open-ended Chemistry (0.95), Physics (0.95), Biology (0.94), and Computer Science (0.93) datasets.
• Few-shot prompting sharply changed some results. Llama4-Scout-Instruct improved dramatically from zero-shot to two-shot on several benchmarks (SciQ 31.36% to 96.73%; ARC-C 40.09% to 93.17%). Darwin1.5-7B on GPQA-Diamond went the other way, dropping from 54.26% at k=0 to 12.63% at k=2

Authors’ abstract

Large language models (LLMs) have demonstrated transformative potential in scientific research, yet their deployment in high-stakes contexts raises significant trustworthiness concerns. Here, we introduce SciTrust 2.0, a comprehensive framework for evaluating LLM trustworthiness in scientific applications across four dimensions: truthfulness, adversarial robustness, scientific safety, and scientific ethics. Our framework incorporates novel, open-ended truthfulness benchmarks developed through a verified reflection-tuning pipeline and expert validation, alongside a novel ethics benchmark for scientific research contexts covering eight subcategories including dual-use research and bias. We evaluated seven prominent LLMs, including four science-specialized models and three general-purpose industry models, using multiple evaluation metrics including accuracy, semantic similarity measures, and LLM-based scoring. General-purpose industry models overall outperformed science-specialized models across each trustworthiness dimension, with GPT-o4-mini demonstrating superior performance in truthfulness assessments and adversarial robustness. Science-specialized models showed significant deficiencies in logical and ethical reasoning capabilities, along with concerning vulnerabilities in safety evaluations, particularly in high-risk domains such as biosecurity and chemical weapons. By open-sourcing our framework, we provide a foundation for developing more trustworthy AI systems and advancing research on model safety and ethics in scientific contexts.