Evaluating Hydro-Science and Engineering Knowledge of Large Language Models

AI summary

Overview

Research area: Natural Language Processing — domain-specific evaluation of large language models, specifically hydro-science and engineering (Hydro-SE).

Technical level: Intermediate. The paper is a benchmark-and-evaluation study; it assumes familiarity with LLM terminology (parameter scale, calibration, scaling laws) but the task design and results are explained in accessible terms.

Scope: The paper introduces Hydro-SE Bench, a 4,000-question Chinese multiple-choice benchmark spanning nine subfields of hydro-science and engineering, and uses it to evaluate 16 LLMs across three cognitive task types.

What This Paper Is About

Hydro-SE covers water supply, hydropower, and flood and drought mitigation, and it requires knowledge spanning hydrology, meteorology, hydraulic structures, and geotechnical engineering. Before LLMs can be used reliably in this field, someone has to measure how much domain knowledge and practical engineering ability they actually have. The authors build the first benchmark dedicated to this problem and use it to compare 10 large commercial LLMs against 6 small open-source LLMs.

Key Contributions

1. A new benchmark: Hydro-SE Bench, containing 4,000 question-answer pairs presented in Chinese — 2,700 single-choice questions (68%) and 1,300 multi-choice questions (32%).
2. A three-dimensional task taxonomy: Every question is classified as type A (basic conceptual knowledge), type B (scenario-based engineering application), or type C (reasoning and calculation), and also assigned to one of nine Hydro-SE subfields.
3. A systematic evaluation of 16 models: 10 SOTA large-parameter commercial LLMs and 6 small-parameter open-source LLMs ranging from 7B to 70B parameters, plus a confidence-calibration analysis of two representative models.
4. A practical sampling recommendation: A difficulty-consistency analysis showing that subsets sampled at 30% or more of the full benchmark reproduce full-benchmark accuracy closely, reducing the cost of routine evaluation.

Main Findings

• Commercial model accuracy: The 10 commercial LLMs consistently achieve accuracy values of 0.74–0.80 regardless of architecture or training origin. DeepSeek-V3.2-Exp and DeepSeek-R1 have the highest overall accuracy, both exceeding 0.79, while Kimi K2 (Moonshot AI) and GLM-4.5 (Zhipu AI) exceed 0.77.
• Chinese-model lead on a Chinese benchmark: Four of the five top-performing models were developed in China, with GPT-5 (OpenAI) ranking fifth. The authors attribute this partly to the benchmark being provided in Chinese, which may align better with those models' language priors and fine-tuning corpora.
• Open-source model accuracy: The six small-parameter open-source models range from 0.41 to 0.68. GLM-4-32B is the best of the six at 0.68, close to Claude-4.5 (Sonnet). Scaling from 32B to 72B in the Qwen series yields only marginal accuracy improvements.
• Engineering application is the weak spot: Six of the 10 commercial models scored highest on type A and lowest on type B. Four models (DeepSeek-V3.2-Exp, GPT-5, Grok-4, and Gemini-2.5-Pro) scored highest on type C, even surpassing their type A results. Most small open-source models scored highest on type A and lowest on type C; GLM-4-32B-0414 was the exception, performing better on type C than type B.
• Subfield variation: Accuracy is highest in Power Systems (0.83), Meteorology (0.79), and Hydraulics and River Dynamics (0.79). It is lowest in Background Knowledge (0.70), Industry Standards (0.70), and Engineering Safety and Management (0.71).
• Scaling mainly buys reasoning: Going from small to large models raises average type C accuracy from 0.50 to 0.78, an increase of 56.0%, compared with 23.4% for type A and 32.7% for type B.
• Subfield-level scaling gains: The largest absolute gains from scaling are in Hydraulics and River Dynamics (+0.23) and Geotechnical Engineering (+0.22), both relative improvements exceeding 40%. The smallest are Industry Standards (+0.09, 15% relative) and Engineering Safety and Management (+0.13, 21% relative).
• Poor confidence calibration: GLM-4-32B rated 81.4% of questions at confidence level 5, and DeepSeek-V3.2-Exp did so for 71.2%. For GLM-4-32B, actual accuracy was 74.4% on questions it rated 5, but only 34.8% on questions it rated 4; its calibration curve lies consistently below the perfect calibration line. DeepSeek-V3.2-Exp is better calibrated at medium-to-high confidence but underconfident at low confidence, returning an average confidence of 4.64 for correct answers and 4.15 for incorrect ones.
• Benchmark is difficulty-consistent: Across sampling ratios from 10% to 100%, the difference between maximum and minimum average accuracy is less than 1.9% for DeepSeek-V3.2-Exp and 1.5% for Claude-4.5 (Sonnet). At ratios of 30% or above, the total range of the error bars does not exceed 5%.

Methodology in Plain English

The authors assembled source material from four categories: 985 Hydro-SE related books (textbooks, reference books, monographs), 600 Chinese national and industry standards, 57 water-management laws and regulations, and 1,113 statistical yearbooks and water-resource bulletins. From textbook exercises and expert-designed questions, 1,592 questions were compiled and annotated with subfield and type labels.

To expand this pool, they used a semi-automatic pipeline: source documents were parsed into smaller passages and fed to LLMs as context, with expert-designed questions serving as few-shot examples and prompts guiding question generation. Generated questions retain traceable references to their source documents. Every question was independently reviewed by at least three domain experts in addition to the original designer, with at least three rounds of expert review and refinement applied to LLM-generated questions.

For evaluation, all models received the same prompt templates. The evaluation ran in two stages: the model first produced its full reasoning or problem-solving process, and then another LLM (DeepSeek-V3.2-Exp, chosen for its text-understanding strength) extracted the final choice letters from that response. This design avoids constraining models to output only answer letters, which the authors argue would bias the assessment of their intrinsic reasoning and domain knowledge. Responses were sampled at temperature 0 for deterministic output, and models were accessed through official API endpoints or the OpenRouter platform. Failed or truncated API calls were re-sampled rather than counted as incorrect.

For confidence, the authors used verbalized confidence estimation: models were shown the question, the options, and their own selected answer, then asked to rate confidence on a scale of 1 to 5. This approach was chosen because most closed-source models such as GPT-5 do not expose logits. Questions were grouped into five subsets by reported confidence, and accuracy was computed per subset to build calibration curves.

Why This Matters

Impact on research. The paper fills a gap the authors describe as significant: highly interdisciplinary Hydro-SE knowledge has had limited representation in both the training and evaluation of existing LLMs. The results give model developers concrete training targets and give Hydro-SE researchers a picture of where current models can and cannot be trusted. The authors state that scaling alone cannot compensate for insufficient domain coverage, so targeted domain-adaptive pre-training, curated knowledge integration, or supervised fine-tuning remains necessary.

Real-world applications:

• Flood and drought mitigation planning, where models would need to reason about complex basin-scale management rather than recall textbook facts.
• Hydropower operations, including sediment management strategies for silt-laden reservoirs and analysis of hydraulic turbine failure causes — tasks the paper cites as examples where LLMs struggle.
• Infrastructure design and assessment, particularly hydraulic structures where specialized terminology and device-specific concepts are poorly represented in static training corpora.
• Checking compliance against industry standards and engineering safety regulations, the subfields where models scored lowest and where scaling produced the smallest gains.

Industry relevance. The full benchmark costs 83.9 million tokens to run for SOTA models and 15.3 million tokens for open-source lightweight models, which is a real operational expense. The finding that 30%-or-larger subsets closely reproduce full-benchmark accuracy makes routine model monitoring practical. The accompanying calibration results matter for deployment risk: the paper warns that misplaced confidence may lead to misleading or unsafe recommendations in engineering scenarios. The benchmark and implementation code are released under the MIT license at https://github.com/sheishijun/Hydro-SE-Bench.

Future Directions

• Moving beyond multiple-choice questions to capture the open-ended, dynamic decision-making processes of real engineering applications.
• Adding real-time data and scenario-based agentic tasks, since the current benchmark does not incorporate them.
• Incorporating multimodal information — satellite remote sensing imagery, station time-series data, and surveillance video streams — which many Hydro-SE applications depend on but the text-based evaluation excludes.
• Establishing standardized metrics for domain reasoning, uncertainty estimation, and human–AI collaboration, and running continuous benchmarking and longitudinal tracking of model updates to reveal performance trajectories and diagnose persistent weaknesses.

Target Audience

LLM developers and training engineers who need concrete targets for continued pre-training or supervised fine-tuning in the hydropower and water-resources domain; Hydro-SE researchers and practitioners assessing whether current models are ready for real decision support; and benchmark researchers interested in a worked example of building a vertical-domain, multi-subfield evaluation with confidence calibration and a sampling-efficiency analysis. Readers should note that the benchmark is in Chinese, which is relevant when interpreting the top-model rankings.

Authors’ abstract

Hydro-Science and Engineering (Hydro-SE) is a critical and irreplaceable domain that secures human water supply, generates clean hydropower energy, and mitigates flood and drought disasters. Featuring multiple engineering objectives, Hydro-SE is an inherently interdisciplinary domain that integrates scientific knowledge with engineering expertise. This integration necessitates extensive expert collaboration in decision-making, which poses difficulties for intelligence. With the rapid advancement of large language models (LLMs), their potential application in the Hydro-SE domain is being increasingly explored. However, the knowledge and application abilities of LLMs in Hydro-SE have not been sufficiently evaluated. To address this issue, we propose the Hydro-SE LLM evaluation benchmark (Hydro-SE Bench), which contains 4,000 multiple-choice questions. Hydro-SE Bench covers nine subfields and enables evaluation of LLMs in aspects of basic conceptual knowledge, engineering application ability, and reasoning and calculation ability. The evaluation results on Hydro-SE Bench show that the accuracy values vary among 0.74 to 0.80 for commercial LLMs, and among 0.41 to 0.68 for small-parameter LLMs. While LLMs perform well in subfields closely related to natural and physical sciences, they struggle with domain-specific knowledge such as industry standards and hydraulic structures. Model scaling mainly improves reasoning and calculation abilities, but there is still great potential for LLMs to better handle problems in practical engineering application. This study highlights the strengths and weaknesses of LLMs for Hydro-SE tasks, providing model developers with clear training targets and Hydro-SE researchers with practical guidance for applying LLMs.