RCScore: Quantifying Response Consistency in Large Language Models

AI summary

Overview

• Research area: Natural Language Processing / LLM evaluation and benchmarking
• Technical level: Intermediate
• Scope: This paper introduces RCScore, a three-dimensional metric (Structurality, Lexicality, Coherence) for quantifying how much an LLM's responses change when the same problem is asked in different instruction styles, evaluated across ten open-source instruction-tuned models on four reasoning benchmarks.

What This Paper Is About

Most LLM evaluations report accuracy on a single instruction template, which hides the fact that models can perform very differently when the same problem is phrased in a different way. This paper builds a framework that systematically rewrites the instruction prefix of benchmark problems into four syntactic styles (Declarative, Interrogative, Exclamative, Imperative) while keeping the underlying question identical, then measures both accuracy shifts and the stylistic consistency of the resulting responses. The goal is to show that instruction-style sensitivity is a real, measurable gap in current evaluation practice and that cross-style consistency tracks with task accuracy.

Key Contributions

1. RCScore, a multi-dimensional consistency metric. RCScore decomposes response similarity into Structurality (syntactic pattern preservation, measured via dependency parsing), Lexicality (TF-IDF cosine similarity combined with ROUGE-L), and Coherence (logical flow and content organization), each normalized to [0,1] and combined with equal weights of 0.33.
2. Cross-Response Similarity (CRS). A method for applying RCScore to measure a model's stylistic self-consistency across all six pairwise combinations of the four instruction styles, preserving the three-dimensional structure of the metric rather than collapsing it early.
3. Evidence that instruction style materially changes accuracy. Across ten LLMs and four reasoning benchmarks, instruction style shifted accuracy by up to 16.7 percentage points.
4. Evidence that consistency correlates with accuracy. CRS values show statistically significant positive correlations with mean task accuracy across 40 model-benchmark pairs, suggesting cross-style consistency is a useful proxy for model reliability.

Main Findings

• Instruction style can move accuracy by large margins. Under beam search, Gemma 3-27B and Qwen 2.5-72B showed gaps of 13.3% and 16.7% respectively on AIME across styles. LLaMA 3-70B showed a 3.0% gap on GPQA-Diamond. Under greedy search, LLaMA 3-70B and Qwen 2.5-72B each shifted by 6.7% on AIME.
• Deterministic decoding does not remove style sensitivity. Accuracy variations persisted under greedy search, though with reduced magnitude compared to beam search.
• Greedy search produces more stylistically stable outputs. Greedy search (temperature = 0.0) consistently yielded 7–13% higher CRS scores than beam search (temperature = 1.0) across all models. For example, LLaMA 3.3-70B's GSM8K RCScore increased from 0.44 under beam search to 0.67 under greedy search.
• Larger models tend to be more consistent. LLaMA 3.3-70B outperformed LLaMA 3.2-3B on AIME CRS (0.44 vs. 0.31 with beam search). Models above 70B generally showed higher CRS, suggesting parameter scale improves stylistic stability.
• Task complexity influences consistency. Gemma 3-27B with greedy search achieved 0.51 RCScore on AIME but 0.74 on GSM8K.
• CRS correlates with accuracy across all dimensions. For beam search, Pearson's r was 0.57 for Structurality, 0.65 for Lexicality, 0.64 for Coherence, and 0.66 for Overall RCScore. For greedy search, values were higher: 0.675 (Structurality), 0.790 (Lexicality), 0.656 (Coherence), and 0.733 (Overall), all with p-values below 2 × 10⁻⁶.
• Lexicality is the strongest single correlate. Lexicality showed Pearson's r > 0.65 under beam search and r ≈ 0.79 under greedy search.
• Greedy decoding won on most benchmarks. Greedy search achieved higher average accuracy for a majority of models on AIME, MATH-500, and GSM8K, while beam search had a slight edge on GPQA-Diamond for more models. GPQA-Diamond accuracy was typically below 10% for most models, so the authors treat those differences cautiously.

Methodology in Plain English

The researchers took four reasoning benchmarks — AIME 2024, GSM8K, MATH-500 (a curated 500-problem subset of MATH selected by OpenAI for the Let's Verify Step by Step study), and GPQA-Diamond (evaluated without answer options to test pure reasoning) — and left the problems untouched. Instead, they replaced only the instruction prefix with four syntactically distinct versions based on Huddleston et al. (2002) clause classification: Declarative, Interrogative, Exclamative, and Imperative. To avoid semantic drift, they fixed the main verbs (e.g., "solve," "suggest") and kept lexical complexity comparable across styles, with a Type Token Ratio of 0.75–0.78.

They ran ten open-source instruction-tuned models: Gemma 3 (4B, 12B, 27B), Qwen 2.5 (3B, 7B, 32B, 72B), and Llama 3.2 3B / 3.1 8B / 3.3 70B. Generation used max_new_tokens = 2048, with beam search at temperature = 1.0, top-k = 50, top-p = 0.9, and greedy search selecting the highest-probability token each step.

For each model and problem, they compared responses to the same problem across the four styles. Structurality aligns sentence pairs via BERTScore (using RoBERTa-Large) and computes Jaccard similarity over syntactic patterns of the form ⟨pos, dependency relation, head pos⟩ from dependency parsing. Lexicality combines TF-IDF cosine similarity and ROUGE-L with equal weights of 0.5 each. Coherence multiplies a structural alignment score (built from Kendall's Tau order correlation, position matching, sequential continuity, and semantic similarity, weighted 0.25 each) by a quadratic content-weighted penalty. CRS then averages these three-dimensional vectors across all six style pairs. Finally, they correlated mean accuracy (averaged across the four styles) with CRS across 40 model-benchmark pairs.

Why This Matters

Impact on research. RCScore reframes evaluation from "did the model get the right answer" to "does the model behave consistently regardless of how the question is asked." Its correlation results position consistency as a cheap, complementary reliability signal that does not require a second LLM as a judge — avoiding the circular evaluation problem the authors criticize in the LLM-as-a-judge paradigm. The framework is released with a public GitHub repository, making it reproducible.

Real-world applications:

• Robustness auditing before deployment. Teams can screen models for instruction sensitivity that single-template benchmarks would hide before shipping them into user-facing systems.
• Prompt engineering and system design. Knowing which styles degrade performance helps teams standardize prompt templates or apply style-specific safeguards.
• Model selection and comparison. CRS offers a dimension of comparison beyond accuracy, useful when two models score similarly on standard benchmarks.
• Accessibility. The authors note that inconsistent performance across formulations creates uneven experiences for users who prefer particular phrasing patterns due to cultural background or neurodivergence.

Industry relevance. Organizations that deploy LLMs in production see naturally varied user phrasing. A model that shifts by up to 16.7 percentage points on identical problems is a reliability risk. The finding that deterministic decoding improves both consistency and — for most models on AIME, MATH-500, and GSM8K — accuracy gives a concrete, low-cost deployment recommendation.

Future Directions

1. Multilingual and broader stylistic coverage. The current implementation uses four predefined styles and English-language NLP tools; the authors identify code-switching and highly informal language as unexplored territory.
2. Extension beyond mathematical and reasoning tasks. The authors state that subjective domains such as creative writing or emotional support may show different style sensitivity and are not covered.
3. Human validation of RCScore dimensions. No human evaluation was conducted to confirm that the dimensions align with human perceptions of consistency or quality.
4. Non-uniform weighting schemes. The current aggregation weights are uniform (0.33 across dimensions); task-dependent or empirically derived weights remain open.

Target Audience

LLM evaluation researchers and benchmark designers will find the metric formulation and correlation analysis most relevant. Practitioners responsible for deploying or prompt-engineering LLM systems benefit from the decoding-strategy and instruction-style findings. Computational linguists will appreciate the syntactic clause-type framing. Readers need basic familiarity with benchmarks, decoding strategies, and similarity metrics to follow the methodology fully, though the core claims are accessible without it.

Authors’ abstract

Current LLM evaluations often rely on a single instruction template, overlooking models' sensitivity to instruction style-a critical aspect for real-world deployments. We present RCScore, a multi-dimensional framework quantifying how instruction formulation affects model responses. By systematically transforming benchmark problems into multiple instruction styles, RCScore reveals performance variations undetected by conventional metrics. Our experiments across ten LLMs on four reasoning benchmarks demonstrate that instruction style can shift accuracy by up to 16.7% points. We introduce Cross-Response Similarity (CRS), a method applying RCScore metrics to measure stylistic self-consistency, and establish its strong correlation with task accuracy, suggesting consistency as a valuable proxy for model reliability. Additional findings show that deterministic decoding produces more stylistically stable outputs, and model scale correlates positively with cross-style consistency. RCScore offers a principled approach to assess instruction robustness.