SciMIF: Understanding Multimodal Instruction Following in Scientific Domains

AI summary

Overview

• Research area: Evaluation of Multimodal Large Language Models (MLLMs) on scientific instruction following, sitting at the intersection of multimodal benchmark design, scientific reasoning evaluation, and instruction-adherence research.
• Technical level: Intermediate. The conceptual framing is accessible, but the paper assumes familiarity with LLM evaluation metrics and scientific task conventions.
• Scope: The paper introduces SciMIF, a benchmark of 2,527 samples that tests whether MLLMs satisfy explicit scientific and general constraints (not just produce correct answers) across chemistry, geography, biology, materials science, and physics.

What This Paper Is About

Existing scientific benchmarks mostly check whether a model's final answer is scientifically correct, which does not tell you whether the model obeyed the requested units, notation, output structure, method, or solution procedure. The authors build SciMIF, a multimodal benchmark that separates scientific correctness (is the answer scientifically valid?) from instruction adherence (were the specified constraints satisfied?), so that failures can be attributed to reasoning, instruction following, or both. The goal is to systematically measure how well current state-of-the-art MLLMs follow complex scientific instructions.

Key Contributions

1. An expert-derived taxonomy of scientific constraints spanning five disciplines and ten functional groups (Procedure, Number, Method, Unit, Format, Terminology, Precision, Letter, Structure, Selection), capturing both shared capability dimensions and discipline-specific requirements.
2. A scalable data construction framework that converts existing scientific tasks into instruction-following evaluations by recognizing implicit constraints and injecting compatible scientific and general constraints without changing the reference answers.
3. The SciMIF benchmark itself: 2,527 samples built from 13 existing scientific datasets covering 22 task types, including 42 discipline-adapted constraints and 27.50% multimodal samples.
4. A comprehensive evaluation of closed-source and open-source MLLMs revealing substantial disciplinary variation, difficulty with fine-grained constraints, and a clear gap between scientific correctness and instruction adherence.

Main Findings

• Disciplinary variation is large: The highest Instruction Satisfaction Rate (ISR) values appear in biology and materials science, while chemistry and geography are harder. GPT-5.2 reaches an ISR of 72.82% in biology and 78.07% in materials science, but falls to 46.33% in chemistry.
• Chemistry is the most challenging domain: Averaged Decomposed Requirements Following Ratio (DRFR) in chemistry is 68.2%, versus 92% for physics, which the authors attribute to chemistry's dense concentration of fine-grained numerical and structural constraints, against physics' broader methodological constraints.
• Closed-source models lead consistently: GPT-5.2 achieves an overall ISR of 65.67%, above the 57.39% of the leading open-source model, Qwen3.5-397B-A17B. The gap narrows in physics, where Qwen3.5-397B-A17B scores 62.50% ISR versus 69.56% for GPT-5.2.
• Scaling does not fix instruction following: InternVL3.5-8B scores 51.37% overall versus 51.64% for InternVL3.5-38B, and within the Qwen3.5 family the 27B model scores 51.37% while the 122B version drops to 50.41%, indicating performance saturation or an alignment tax rather than a linear scale benefit.
• General constraints are harder than scientific ones: GPT-5.2 scores 88.74% DRFR on scientific constraints versus 74.65% on general constraints, suggesting models attend to domain requirements but overlook presentation and organization requirements.
• Fine-grained symbolic groups are the weakest: GPT-5.2 achieves 51.30% DRFR on Letter, 68.62% on Number, and 73.59% on Terminology. The Number constraints require interpreting scientific structures (chemical bonds, functional groups, amino-acid residues, sequence motifs) rather than counting characters.
• Easier groups compress model differences: Gaps between the top closed-source and open-source models reach 8% to 9% in the more challenging letter, number, and format groups, but only 2% to 4% in precision, procedure, and method.
• Correctness and adherence are weakly coupled: Across all evaluated models, the Correct and Followed rate stays below 30%, roughly 20% of samples are Correct but Violated, and the Incorrect but Followed rate exceeds 30% for several models.

Methodology in Plain English

The authors began with 22 scientific tasks drawn from 13 existing datasets (including ChemEval, IMAGEO-Bench, LAB-Bench, MatCha, and PhysUniBench), and had domain experts read the tasks, reference answers, disciplinary conventions, and literature to find recurring requirements. They grouped these requirements into ten functional constraint types and kept the discipline-specific realizations as individual constraints, giving a two-level organization: domain level (one general domain plus five scientific domains) and group level (functional purpose).

To build the benchmark, each seed sample (question, optional image, reference answer, task type) is matched to a compatible constraint inventory. Annotators first identify constraints already implicit in the original query, then filter out redundant or contradictory ones. Scientific constraints are injected according to disciplinary knowledge, and general constraints are randomly sampled by N = 3 classes from the remaining inventory; each injection is retried up to k = 3 times with an alternative constraint if it fails. Injections are validated automatically twice, checking that the new constraint is actually included and that the ground-truth answer is unchanged. Two annotators then verify logical coherence, fluency, and constraint fidelity; 884 samples were revised, and unrepairable ones were discarded.

Evaluation uses three existing metrics: Constraint Satisfaction Rate (CSR), Instruction Satisfaction Rate (ISR), and Decomposed Requirements Following Ratio (DRFR). Constraint verification uses exact match, precision-based checks, and LLM-as-a-judge. Eleven models were tested — GPT-5.2, Grok-4-Fast, Gemini-3.1-Pro-Preview, Claude-Sonnet-4.6, the Qwen3.5 series (27B, 35B-A3B, 122B-A10B, 397B-A17B), and the InternVL3.5 series (8B, 14B, 38B).

Why This Matters

Separating scientific correctness from instruction adherence gives researchers a way to diagnose whether a model fails because it lacks domain knowledge or because it ignores the requested form. SciMIF fills a gap the authors identify: prior scientific benchmarks such as SciBench, MathVista, MMMU, and GeoQA measure answer correctness, while general instruction-following benchmarks such as IFEval, FollowBench, and MM-IFEngine cover formatting and linguistic requirements rather than scientific ones — and the closest prior work, SciIF, uses constraints shared across questions rather than constraints derived per discipline, and is text-only.

Real-world applications:

• Chemistry and molecular design: constraints requiring valid nomenclature such as SMILES, correct counts of atoms, bonds, and functional groups, and ordered multi-step reaction sequences matter for automated synthesis planning.
• Materials science: identifying valid characterization techniques from microscopy or property data, and formatting material properties as discrete labels or continuous values.
• Geography and remote sensing: producing valid hierarchical addresses, selecting among predefined geographical scenes, and reporting quantities with specified geographical units (distance, area, elevation, radiation).
• Biology and physics workflows: specifying protein or sequence lengths and entity relationships in biology, and requiring step-by-step derivations with physical principles and consistent physical units in physics.

Industry relevance: the scaling results argue that parameter count alone is not the bottleneck — domain-aware alignment and instruction-tuning data are. For teams building autonomous science agents, where outputs must be machine-readable and downstream-compatible, the finding that general constraints are systematically overlooked is a direct warning about pipeline reliability.

Future Directions

• Reorient training and evaluation toward practical application capabilities in constrained environments, rather than primarily injecting scientific knowledge.
• Develop domain-aware instruction alignment methods, since the paper shows scaling alone does not improve constraint adherence.
• Explore structure-aware representations, constrained decoding, and external scientific parsers or verification tools to address fine-grained symbolic and discrete constraints.
• Integrate external numerical computation tools and symbolic engines into MLLM-centric agents to guide inference and enable more precise control in real scientific workflows.
• Improve both scientific correctness and instruction adherence jointly, since the Correct and Followed rate remains below 30%.

Target Audience

Researchers and engineers working on MLLM evaluation, instruction following, and AI for science, particularly those building scientific agents or domain-specific model pipelines. It is also useful for benchmark designers interested in taxonomy-driven data construction and for practitioners who need to know where current models break down before deploying them in chemistry, geography, biology, materials science, or physics workflows.

Authors’ abstract

Understanding instruction-following capabilities in scientific domains is essential for effectively leveraging Multimodal Large Language Models (MLLMs) to advance the development of scientific fields. In this work, we introduce SciMIF, a novel benchmark designed to evaluate the capability of MLLMs in following complex scientific instructions. Specifically, based on an extensive analysis of 22 distinct tasks across 5 representative scientific disciplines, we propose a comprehensive taxonomy comprising 10 constraint groups that captures both general functional requirements and discipline-specific characteristics. Guided by this taxonomy, we develop a high-fidelity instruction injection pipeline to systematically augment existing scientific datasets. We conduct comprehensive experiments on multiple state-of-the-art closed-source and open-source MLLMs. Our findings reveal significant performance disparities across different scientific disciplines, with chemistry posing greater challenges for current MLLMs. Furthermore, we observe that increasing the model scale does not yield corresponding improvements in constraint adherence, and current models still struggle severely with fine-grained constraints and instructions requiring the deep application of disciplinary knowledge. SciMIF fills the current void in evaluating multimodal instruction adherence within scientific domains, laying a crucial foundation for future enhancements of MLLMs in rigorous scientific applications. Data and code will be released at https://github.com/shenye7436/SciMIF .