ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in Instructions

AI summary

Overview

Research area: Natural Language Processing — instruction-following and LLM evaluation.

Technical level: Intermediate. The paper assumes familiarity with LLM benchmarking, F1 scores, and prompt-based evaluation, but its core idea (instructions that contradict themselves) is intuitive.

Scope: The paper introduces ConInstruct, a benchmark that measures whether LLMs can detect and then properly respond to user instructions containing mutually contradictory constraints.

What This Paper Is About

Prior instruction-following benchmarks assume that every constraint in a user prompt can be satisfied at the same time. In practice, long or complex prompts often contain constraints that contradict each other — for example, asking for a specific phrase to appear while also forbidding that phrase. The paper asks two questions: can LLMs tell that such an instruction is internally conflicting, and once they know, do they actually tell the user?

Key Contributions

1. A new benchmark, ConInstruct, built to evaluate LLMs on instructions containing conflicting constraints, covering six NLP tasks, 35 scenario-specific domains, six constraint types, and nine conflict types (six intra-constraint, three inter-constraint).
2. A systematic study of conflict detection: the authors measure how well models can flag whether an instruction contains conflicting constraints, across single-conflict and multi-conflict settings.
3. An analysis of conflict resolution behavior: they classify model responses into four behavior types and quantify how often models silently answer instead of surfacing the conflict.
4. An examination of constraint priority: they test how newly introduced conflicting constraints affect a model's ability to satisfy the original constraints, including whether position in the instruction matters.

Main Findings

• Proprietary models detect conflicts well. Claude-4.5-Sonnet reaches the second-highest average F1-score overall at 87.3%, behind only DeepSeek-R1 at 91.5%. Claude-3.5-Sonnet reaches 86.6%, Claude-3.5-Haiku 85.0%, GPT-4o 84.9%, and GPT-4o-mini 83.6%. The random-guess baseline average is 49.4.

• Only one open-source model matches the proprietary tier. DeepSeek-R1 is the only open-source model with similarly strong performance (91.5% average F1). Models below 7B parameters generally struggle: Meta-Llama-3.2-3B-Instruct (43.8%), Qwen2.5-3B-Instruct (53.2%), Qwen2.5-1.5B-Instruct (32.8%), and Meta-Llama-3.2-1B-Instruct (32.9%) underperform relative to random guessing on most conflict types.

• Intra-constraint conflicts are easier to detect than inter-constraint conflicts. Claude-3.5-Sonnet scores 92.7% on intra-constraint subsets but only 74.6% on inter-constraint subsets. The same pattern holds for other strong models.

• More conflicts make detection easier for larger models. As the number of conflicts per instruction rises, larger models (notably Qwen2.5-7B and Qwen2.5-32B) show improved detection. Qwen2.5-32B identifies conflicts with near 100% accuracy when more than two conflicts are present. Small open-source models still show low recall even with many conflicts.

• Detection does not translate into disclosure. When instructions contain 1–2 conflicts, GPT-4o directly generates a response in 97.5% of cases without notifying the user. Even the best-performing model, Claude-4.5-Sonnet, explicitly alerts users to conflicts in only 45% of cases — 36% by requesting clarification and 9% by resolving the conflicts autonomously.

• Claude models become more conflict-aware as conflicts accumulate. For Claude-3.5-Sonnet, the combined share of clarification-requesting and autonomous-resolution behaviors rises from 32.0% on instructions with 1–2 conflicts to 64.0% on instructions with 5–6 conflicts. Behavior 1 (silently answering) still makes up a substantial share of Claude responses.

• Prompt engineering helps but overcorrects. Adding a guiding prompt pushes LLMs toward the desired behavior of acknowledging conflicts and requesting clarification — GPT-4o goes from 1% to 96% for that behavior on single-conflict instructions — but the same prompt causes models to request clarification even when no conflict exists (GPT-4o: 40% of conflict-free instructions; GPT-4o-mini: 53%; Claude-3.5-Sonnet: 28%; Claude-3.5-Haiku: 22%).

• Position matters for constraint priority. Newly introduced conflicting constraints substantially reduce satisfaction of the original constraints. A constraint appearing later in an instruction is more likely to be followed, whether it is the original or the newly introduced conflicting one.

• Length constraints are hardest to satisfy. Across the seven evaluated LLMs, the constraint satisfaction rate is notably lowest for length constraints and higher for the other five constraint types.

Methodology in Plain English

The authors started with 100 hand-written seed instructions spanning six tasks (email writing, plan generation, story generation, open-domain QA, review writing, article writing) and 35 domains. They used GPT-4o to expand each seed instruction so that it contained all six constraint types: content, keyword, phrase, length, format, and style.

Then they had GPT-4o generate conflict pairs rather than injecting conflicting constraints directly. Each pair consists of one constraint pulled from the expanded instruction and a new constraint that directly contradicts it. One pair is generated for each of nine conflict types: six within-type conflicts (content, keyword, phrase, length, format, style) and three cross-type conflicts (keyword–phrase, phrase–content, phrase–style). Each instruction ends up with 7–9 conflicts on average (8.6).

Quality control was done in two steps: two annotators refined the GPT-4o output and corrected ambiguous conflicts, and a third annotator reviewed and removed unreasonable ones.

For detection experiments, the authors appended conflicts to conflict-free instructions to build 864 single-conflict instructions, then formed nine evaluation subsets. Each subset pairs 100 conflict-free instructions with single-conflict instructions; most subsets contain 200 instructions, and the paper reports 194 and 170 instructions for the two smaller subsets. For multi-conflict experiments, they randomly sampled 1, 2, 3, 4, 5, or 6 conflicts per instruction, producing six subsets of 100 instructions each.

They evaluated seven proprietary models and eleven open-source models, with a maximum output length of 2048 tokens and temperature 0. Open-source inference ran on A100 GPUs with 40GB memory; proprietary models were accessed via official APIs. LLM outputs were judged by GPT-4o (with Gemini-2.5-Pro used as a second judge in the behavior classification), and the authors validated those judgments with human annotation.

Why This Matters

Impact on research: The paper opens a dimension of instruction-following that prior benchmarks such as IFEval-style evaluations assume away — that constraints are coherent. It shows that detection ability and disclosure behavior are separable capabilities, which reframes what "good instruction-following" should mean.

Real-world applications:

• Assistants that generate reports or emails under length, format, and keyword requirements, where silently dropping a constraint produces output that looks correct but is not.
• Enterprise and compliance tools where a model must flag an impossible specification instead of guessing which rule to violate.
• Data-extraction and structured-output pipelines where format constraints can contradict content constraints.
• Customer-facing chat systems where silently resolving a contradictory request (rather than asking which the user meant) removes user control.

Industry relevance: The results suggest that strong detection scores in evaluation dashboards can mask a usability failure. The guiding-prompt experiment shows a practical trade-off: pushing models to always ask for clarification degrades the handling of ordinary, non-conflicting requests.

Future Directions

• Implicit conflicts. The benchmark deliberately avoids ambiguous conflicts so that satisfaction can be checked automatically. Extending it to conflicts requiring multi-hop reasoning or commonsense inference is left open.
• Multimodal instructions. ConInstruct is text-only; conflicts arising in image, audio, or video instructions are not covered.
• Better training or prompting than a blanket clarification prompt. The paper shows prompt engineering alone cannot achieve both desired conflict disclosure and accurate execution of conflict-free instructions.
• Teaching models to disclose without over-asking. A key open question is how to make models surface genuine conflicts while keeping false-positive clarification requests low.

Target Audience

Researchers and engineers working on instruction-following, LLM evaluation, and benchmark design; practitioners building assistant or agent systems where user prompts may contain unsatisfiable constraints; and anyone studying transparency and clarification behavior in model outputs. Readers looking for a first introduction to LLM benchmarking will find the core idea accessible, but the experimental sections assume familiarity with F1, recall, and constraint-satisfaction metrics.

Authors’ abstract

Instruction-following is a critical capability of Large Language Models (LLMs). While existing works primarily focus on assessing how well LLMs adhere to user instructions, they often overlook scenarios where instructions contain conflicting constraints-a common occurrence in complex prompts. The behavior of LLMs under such conditions remains under-explored. To bridge this gap, we introduce ConInstruct, a benchmark specifically designed to assess LLMs' ability to detect and resolve conflicts within user instructions. Using this dataset, we evaluate LLMs' conflict detection performance and analyze their conflict resolution behavior. Our experiments reveal two key findings: (1) Most proprietary LLMs exhibit strong conflict detection capabilities, whereas among open-source models, only DeepSeek-R1 demonstrates similarly strong performance. DeepSeek-R1 and Claude-4.5-Sonnet achieve the highest average F1-scores at 91.5% and 87.3%, respectively, ranking first and second overall. (2) Despite their strong conflict detection abilities, LLMs rarely explicitly notify users about the conflicts or request clarification when faced with conflicting constraints. These results underscore a critical shortcoming in current LLMs and highlight an important area for future improvement when designing instruction-following LLMs.