Instruction Capacity: Behavior and Degradation of Language Models under Controlled Load

AI summary

Overview

Research area: Evaluation and measurement of large language model instruction-following behavior under increasing load — sitting at the intersection of LLM benchmarking methodology, psychometrics-style measurement, and applied statistics.

Technical level: Intermediate. The core task is simple to describe (word-replacement rules), but the paper's value depends on a statistical model with binned shape estimation, smooth curve fitting, a shift correction, and an inverse-problem definition of capacity with delta-method uncertainty.

Scope (one sentence): The paper builds a controlled, program-checkable instrument for measuring how many independent instructions a language model can hold at a given accuracy, and reports a calibrated "passport" for GPT-4o mini measured over 2958 requests and 642 430 trials.

What This Paper Is About

Language models get worse at following instructions as more instructions pile up, but the field lacks a single quantity — expressed in units of load, with an uncertainty — that says where a given model's limit lies. Existing benchmarks confound the number of instructions with their difficulty and interdependence, because the instructions typically share one coherent text or are graded by a judge model. This paper introduces an atomic instruction (replace one single-token word with another inside a meaningless text), fits statistical laws describing how the success rate falls along each load axis, and inverts those laws to define "instruction capacity" — the number of rules a model holds at a stated share of correct replacements.

Key Contributions

1. An instrument built on the simplest atomic instruction. A task is generated from three seeds and reproduced byte for byte; rules replace single-token words in a text without meaning; the correct answer is computed by a reference program; and answers are checked line by line, distinguishing three kinds of failure.

2. A statistical model that avoids a coding-dependent baseline. The shape of each axis's contribution is measured by bins, described by smooth laws, and then corrected by an approximation shift so that the restored reference level does not depend on the choice of reference bins.

3. A formal definition of instruction capacity. Capacity is defined as the solution of an inverse problem, with uncertainty computed by the delta method and with the spread of request difficulty taken into account.

4. An open, recomputable passport. The passport of GPT-4o mini contains laws for text length and word position, replacement density, number of rules and rule number; laws for the rate and place of answer collapse; and instruction, fatigue and guaranteed instruction capacities. The code, measurement data and passport are open, and every number is recomputed from published points.

Main Findings

• Regular laws on every load axis. The measurement of GPT-4o mini (2958 requests, 642 430 trials) produced regular laws for every load factor examined — text length, word position, replacement density, number of rules, and rule number — as well as for the collapse of the answer.

• Instruction capacity at the 0.95 level: about 37–38 rules. At a share of right replacements of 0.95, the model holds roughly 37 to 38 rules.

• Instruction capacity at full replacement density: about 18–19 rules. Under the full replacement density the figure drops to about 18 to 19 rules, showing that capacity is density-dependent rather than a single fixed number.

• Three distinct capacities are reported. Beyond the instruction capacity, the passport contains a fatigue capacity and a guaranteed instruction capacity, all computed with uncertainties.

• Collapse is characterized, not just observed. The paper reports laws for both the rate of answer collapse and the place where it occurs, not merely an aggregate accuracy drop.

• The critique of prior work is quantitative. The related-work analysis documents why earlier settings cannot serve as a load meter: form-instruction sets mix requirements of very different difficulty (for GPT-4o, per-requirement success ranges from 0.62 for an exact character count to 1.0 for a title, quotation marks or a comma ban), requirements share one text (a "no more than 79 characters per line" rule alone is met in 97% and 99% of cases for Claude 3.5 Sonnet and Gemini 1.5 Pro, but in 2% and 20% alongside five other code-style rules), and judge-based grading inflates results (0.657 versus 0.213 under programmatic checking at ten instructions).

Methodology in Plain English

The task is deliberately stripped of meaning. The model is given a nonsense text and a list of rules, each saying "replace this one-token word with that one." Each rule has exactly one correct outcome at each place it applies, the rules are independent by construction, and a program can compute the reference answer. So every occurrence of a word to be replaced becomes a Bernoulli trial whose position in the answer and whose rule number are both known.

The statistical work proceeds in two stages. First, the probability of a correct replacement is measured as a function of each load factor separately — text length, word position, replacement density, number of rules, rule number — using bins, so that the shape of the dependence is observed rather than assumed. Second, smooth laws are fitted to those shapes, and a shift correction removes the dependence of the overall level on how the bins were chosen. Capacity is then obtained by inverting the fitted law: find the number of rules at which the predicted share of correct replacements equals a target (0.95, or the full-density condition). Uncertainty on that number comes from the delta method, with an extra allowance for the fact that individual requests differ in difficulty.

The contribution list also notes that tasks are generated from three seeds and reproduced byte for byte, and that the answer is checked line by line so that three kinds of failure can be told apart. The paper's methodology section evaluates nine criteria for a load meter — atomicity, independence, homogeneity, computable check, absence of extraneous factors, resolution by place, scale, precision, and protection against memorization — and argues that no reviewed approach satisfies all of them at once. The specific equations of the model, the exact sampling schedule, and the definitions of the three failure kinds are in Sections 3 and 4, which are not included in the provided excerpt; those details are not reported here.

Why This Matters

Impact on research. The paper turns a scattered set of observations ("quality falls as instructions grow") into a comparable measurement: degradation laws with parameters and uncertainties that can be compared across models and across model generations. It also supplies a methodological checklist that exposes known weaknesses of judge-graded and shared-text benchmarks.

Real-world applications.

• Prompt-load metering: the model passport acts as a calibration so that the instruction load of an arbitrary production prompt can be expressed in the same units as a model's capacity, before deployment.
• Model selection and routing: knowing that a model holds roughly 37–38 rules at 0.95 accuracy, and roughly 18–19 at full replacement density, gives engineers a concrete threshold for when to split a prompt or escalate to a stronger model.
• Regression testing across versions: the fixed, seed-reproducible task generation lets teams re-measure a new model release against the same instrument.
• Auditing evaluation claims: the related-work analysis gives concrete evidence (for example, 0.657 versus 0.213) of how much judge-based grading can inflate reported instruction-following performance.

Industry relevance. Agentic and multi-constraint prompting is now common in production systems, and the paper's own survey of prior work records steep declines in realistic settings: AgentIF instructions (average 11.9 constraints) are fully satisfied by fewer than 30% for even the best model studied, and in Eliav's experiments the share of answers with no violation falls practically to zero by 80 rules. A capacity number with an uncertainty is directly actionable for system design in a way that a benchmark percentage is not.

Future Directions

• Calibrating the prompt-load meter. The passport is described as the calibration of a future meter, so the immediate next step is the meter itself: a procedure that takes an arbitrary prompt and returns its load in rule units, comparable to a model's measured capacity.

• Measuring more models and more generations. Only GPT-4o mini is reported in the visible content; the value of the degradation laws depends on whether other models, and later releases of the same model, produce comparable parameters.

• Extending the load axes. The current instrument covers text length, word position, replacement density, number of rules and rule number. The related work identifies other load types — accumulating requirements over dialogue turns, parallel tasks, and long self-generated answers — that the framework does not yet express in the same units.

• Reconciling with existing benchmarks. An open question is how capacity in rule units maps onto the established measures (share of fully satisfied IFEval-style prompts, IFScale-style required-word counts, judge-graded rubric scores), since those remain the numbers practitioners see reported.

Target Audience

Researchers and engineers who need to quantify instruction-following limits rather than merely observe them: evaluation and benchmark scientists, applied statisticians working on LLM measurement, and practitioners designing multi-constraint prompts or agent system prompts who want a model-specific capacity threshold with an uncertainty attached. Readers primarily interested in prompt-engineering tricks will find little here; readers interested in reproducible measurement design, and in the methodological critique of judge-graded and shared-text benchmarks, will find the related-work analysis as useful as the results.

Authors’ abstract

The quality with which language models follow instructions falls as the load grows, yet there is still no quantity that expresses the limit of a model in units of load with an uncertainty. We propose an instrument in which the load consists of rules that replace single-token words in a text without meaning: such an instruction is atomic, the rules are independent by the construction of the task, and the outcome of every application of a rule is determined by a computable reference answer. Every occurrence of a word to be replaced becomes a Bernoulli trial with a known position in the answer and a known rule number. The statistical model first measures the shape of the dependence of the probability of a right replacement on the text length, the word position, the replacement density, the number of rules and the rule number, then describes it by smooth laws and restores a reference level that does not depend on the choice of the coding. The model yields the instruction capacity - the number of rules held at a given share of right replacements - with an uncertainty. A measurement of GPT-4o mini (2958 requests, 642,430 trials) gave regular laws for every load factor and for the collapse of the answer; the instruction capacity was about 37-38 rules at a share of right replacements of 0.95, and about 18-19 rules at the full replacement density. The code of the instrument, the measurement data and the passport are open, and every number of the passport is recomputed from the published points. The passport of a model serves as the calibration of a future meter of the instruction load of prompts.