FormationEval, an open multiple-choice benchmark for petroleum geoscience

AI summary

Overview

Research area: Natural Language Processing — domain-specific evaluation benchmarks for large language models, applied to petroleum geoscience and subsurface engineering.

Technical level: Intermediate. The benchmark construction pipeline and evaluation methodology are accessible to a general NLP audience, while the subject-matter content (well logging physics, reservoir characterization) assumes some geoscience background.

Scope: This paper introduces FormationEval, a 505-question, four-choice benchmark spanning seven petroleum geoscience domains, and reports accuracy results from evaluating 72 language models on it.

What This Paper Is About

Language models are used increasingly in science and engineering, but there is no public benchmark for measuring how well they understand petroleum geoscience and subsurface disciplines — a gap that general benchmarks like MMLU cannot fill because they cover such fields only shallowly. The paper builds one: a curated set of multiple-choice questions derived from authoritative textbooks and open courseware, generated so that they test conceptual understanding rather than memorized phrasing, and accompanied by source provenance and contamination-risk labels. It then uses that benchmark to compare 72 models across closed and open-weight providers.

Key Contributions

1. A concept-based question generation methodology that extracts ideas from technical source material and writes original questions from those concepts, avoiding verbatim copying or close paraphrasing of copyrighted text while still testing genuine comprehension.

2. A curated 505-question dataset covering seven domains, with per-item metadata including source provenance, chapter references, difficulty level, rationale, and an estimated contamination-risk label.

3. A large-scale evaluation of 72 language models from OpenAI, Anthropic, Google, Meta, DeepSeek, Mistral, Alibaba, Zhipu, xAI, Moonshot, MiniMax, Microsoft, and Nvidia, with breakdowns by domain, difficulty, cost, and open versus closed weights.

4. A documented bias audit and mitigation record, quantifying length bias and qualifier-word exploits in the generated questions and reporting before/after metrics, plus an interactive public quiz site for ongoing expert feedback.

Main Findings

• Frontier models exceed 97% accuracy. Gemini 3 Pro Preview leads at 99.8% (504/505 correct), followed by Gemini 3 Flash Preview at 98.2%, Gemini 2.5 Pro at 97.8%, and Grok-4.1-fast at 97.6%. The single error made by the top model occurred on a medium-difficulty question.

• Open-weight models are far more competitive than expected. GLM-4.7 ranks second overall at 98.6%, and DeepSeek-R1 (96.2%) and DeepSeek-V3.2 (94.9%) also rank in the top 25. Across all 32 open-weight models, mean accuracy is 85.7% and median 87.7%, with 11 models above 90% and 22 above 85%. DeepSeek-R1 at $0.30/M input matches GPT-5.1 variants priced at $1.25/M input.

• Petrophysics is the hardest domain for every model. Average accuracy across all 72 models is 87.5% in petrophysics versus 93–96% in other domains — a gap of 3–5 percentage points that persists across model tiers. Eight of the ten hardest individual questions come from this domain.

• Medium-difficulty questions are the hardest difficulty tier, not hard ones. Models score lower on medium questions than on either easy or hard ones. The authors attribute this to medium questions being more comparison-heavy and tool-specific, with a higher calculation rate (8.8%) than hard questions (5.1%), and less length-bias cueing (correct answer uniquely longest in 40.5% of medium versus 55.6% of hard).

• Compact models struggle specifically with technical well-logging content. The bottom ten models (3B–12B parameters or older architectures) range from 57.6% (Llama-3.2-3b-instruct) to roughly 80%. Their petrophysics accuracy drops to 51–74% while other domains stay at 62–98%.

• Most questions discriminate between models. Only 21.6% of questions were answered correctly by all 72 models, none were missed by all, and 78.4% produced mixed results.

• Length bias was a real exploit and remains partially present. Before mitigation, the correct answer was uniquely the longest in over 55% of questions (25% expected), and absolute words like "always" appeared only in distractors (49 instances, 0% correct) while "may" appeared only in correct answers (13 instances, 100% correct). After mitigation, uniquely-longest-correct fell to 43.2% and "always" was replaced with varied synonyms, but the residual bias is documented openly.

• The single hardest question exposed a systematic reasoning failure. On strike-slip fault stepovers and pull-apart basin formation, 61 of 72 models (85%) chose the wrong option, selecting left-stepping instead of right-stepping for dextral motion.

Methodology in Plain English

The authors took three authoritative sources — Ellis & Singer's Well Logging for Earth Scientists, Bjørlykke's Petroleum Geoscience, and TU Delft OpenCourseWare — converted them to Markdown with OCR, and split them by chapter into chunks of roughly 10,000–15,000 tokens.

Each chunk was fed to GPT-5.2 running with high reasoning effort, along with a long system prompt specifying the required question schema, concept-based derivation rules, difficulty targets, and explicit prohibitions. The model produced 5–12 candidate questions per chapter, which were then checked for schema compliance (no duplicate choices, valid answer index) and for actual support in the source text.

The prompt explicitly forbade questions that reference the chapter ("according to the text"), required standalone answerability from general domain knowledge, banned "All of the above" and negative phrasing, and instructed balanced answer-option lengths and even answer-position distribution. The prompt went through eight major revisions, with the prohibition list growing from 5 to 15 rules.

Quality assurance combined an LLM-based batch review with human spot-checks by the author, a petrophysicist, who verified 2–5 questions per source chapter against the original material. The full 505-question set was reviewed in batches. Corrective work included rewriting 55 questions to remove chapter self-references and fixing 12 grammar issues introduced by bias-mitigation edits.

For evaluation, each model received a zero-shot prompt: a system message framing a multiple-choice exam, then the question and four labeled choices. Answer extraction used flexible regex patterns, with reasoning tags stripped from models that expose chain-of-thought. Unparseable responses counted as incorrect. All API responses were cached per model and question so that reports could be regenerated without new API calls.

Why This Matters

Impact on research. The paper demonstrates a reusable pattern for building domain benchmarks from copyrighted technical material: extract concepts, generate original items, keep provenance metadata, and audit for exploitable surface patterns. It also provides an unusually candid bias report — documenting that the benchmark retains residual length bias rather than claiming a clean dataset. The finding that medium-difficulty items are hardest, not hard ones, challenges a common assumption in benchmark design.

Real-world applications:

• Model selection for subsurface workflows. Operators choosing an LLM to assist with well-log interpretation, core description, or reservoir characterization now have domain-specific accuracy numbers rather than general-purpose leaderboard rankings.

• Cost-efficient deployment. The cost-effectiveness data show that models at $0.05/M input tokens (Qwen3-14b, 92.9%) or $0.22/M (DeepSeek-V3.2, 94.9%) reach accuracy comparable to models costing 10–50 times more, which matters for high-volume document processing pipelines.

• Expert calibration. The public quiz at formationeval.no lets geoscientists benchmark their own knowledge against 72 models, giving a concrete reference point for how much a model can and cannot be trusted in their specialty.

• Training and certification support. The rationales and topic tags make the dataset usable as an educational or self-assessment resource, not just an evaluation artifact.

Industry relevance. Petroleum geoscience involves expensive, high-stakes decisions — drilling target selection, formation evaluation, recovery strategy — where a model that is confidently wrong carries real cost. A benchmark that shows even frontier models failing systematically on petrophysics and on structural geology edge cases gives practitioners a calibrated sense of where human expertise remains essential. The narrow gap between open-weight and closed models is also directly relevant for companies with data-sovereignty constraints that cannot send well data to third-party APIs.

Future Directions

• Multilingual expansion. The schema already includes language codes for Norwegian, Russian, and English; the authors plan to add questions in Norwegian and Russian to serve regional petroleum industries.

• Rebalancing domain coverage. Petrophysics makes up 54% of the dataset because of source availability, while drilling engineering (5%) and production engineering (3%) are thin. Additional sources would produce a more representative picture and allow stronger claims about domain-specific model weaknesses.

• Contamination detection. The current contamination-risk labels are estimates based on topic commonality and cannot be verified without training-data access. Methods for actually measuring overlap between benchmark items and pretraining corpora would strengthen the validity of the high scores.

• Investigating the medium-difficulty anomaly. The finding that medium questions are harder than hard ones deserves a dedicated analysis; if it stems from length-bias cueing rather than genuine cognitive difficulty, the difficulty taxonomy itself needs revision.

• Independent expert review. All current verification was performed by a single domain expert. Recruiting multiple independent reviewers — which the interactive quiz site is designed to encourage — would establish the benchmark's reliability more firmly.

Target Audience

This paper is most useful to NLP researchers building domain-specific benchmarks, who will find a transferable methodology for generating questions from copyrighted technical sources along with a detailed bias-audit template. It also serves applied AI teams in the energy sector deciding which models to deploy for subsurface workflows, and geoscientists who want to understand where current language models fail in their discipline. Benchmark maintainers and evaluation researchers will find the honest treatment of residual bias and contamination risk particularly instructive, since most benchmark papers do not report these limitations as explicitly. Readers need only a general familiarity with LLM evaluation to follow the methodology; the geoscience content is explained through domain labels and topic tags rather than assumed knowledge.

Authors’ abstract

This paper presents FormationEval, an open multiple-choice question benchmark for evaluating language models on petroleum geoscience and subsurface disciplines. The dataset contains 505 questions across seven domains including petrophysics, petroleum geology and reservoir engineering, derived from three authoritative sources using a reasoning model with detailed instructions and a concept-based approach that avoids verbatim copying of copyrighted text. Each question includes source metadata to support traceability and audit. The evaluation covers 72 models from major providers including OpenAI, Anthropic, Google, Meta and open-weight alternatives. The top performers achieve over 97% accuracy, with Gemini 3 Pro Preview reaching 99.8%, while tier and domain gaps persist. Among open-weight models, GLM-4.7 leads at 98.6%, with several DeepSeek, Llama, Qwen and Mistral models also exceeding 93%. The performance gap between open-weight and closed models is narrower than expected, with several lower-cost open-weight models exceeding 90% accuracy. Petrophysics emerges as the most challenging domain across all models, while smaller models show wider performance variance. Residual length bias in the dataset (correct answers tend to be longer) is documented along with bias mitigation strategies applied during construction. The benchmark, evaluation code and results are publicly available.