Instruction Duplication as an Inference-Time Control Primitive

AI summary

Overview

Research area: Inference-time control of large language models, specifically prompt-level intervention on procedural instruction following and its downstream use by deterministic trajectory controllers; evaluated on medical multiple-choice benchmarks.

Technical level: Intermediate. The intervention itself is minimal (repeat the instruction), but the evaluation uses a 2×2×2 placement factorial, TF–IDF recall, Holm-corrected hypothesis families, and a paired downstream reproduction. No model internals or training are required.

Scope: A single-author study (Victor Lavrenko, PeaceTech VC) testing whether duplicating only the procedural instruction — not the query — changes the explicit, machine-addressable state of a model's generated trajectory, and whether that change matters to a downstream trajectory-editing system even when final-answer accuracy does not move.

What This Paper Is About

Language-model prompts typically contain one procedural instruction telling the model how to reason (for example, list facts, give a provisional answer, name an alternative, reconsider, then commit). This paper asks whether simply repeating that instruction — without changing the question, the model, the decoder, or the reward — changes the visible intermediate state the model produces. The goal is to characterize instruction duplication as a minimal black-box "control primitive" and to show where such a change is operationally useful, even when aggregate answer accuracy is completely unchanged.

Key Contributions

1. Definition of instruction duplication as a control mechanism. A fixed instruction I and query Q are placed as ⟨I⟩⟨Q⟩ (one copy) versus, for example, ⟨I⟩⟨Q⟩⟨I⟩ (two copies). The extra copy adds no new rule, example, reward, verifier, or model access; it changes only how often and where the same procedural instruction is exposed before generation.

2. A complete 2×2×2 placement factorial. The instruction can appear in the system message (S), before the question (B), and/or after the question (A), yielding eight conditions (zero, three one-copy, three two-copy, one three-copy) and 7 × 300 × 8 = 16,800 scheduled cells across seven models and 300 medical questions.

3. A control-relevant dissociation. Explicit protocol state changes (All-8 diagnostic 90.22% → 93.17%; TF–IDF recall 73.44% → 74.81%) while final-answer accuracy stays at exactly 60.21% in both arms.

4. A blinded challenge audit and a downstream Answer Engineering (AE) demonstration. The audit yields 10/30 directional confirmations, 20/30 perceptual ties, and no reversals, missing its prespecified 28/30 criterion; the AE experiment shows why small, sometimes perceptually negligible changes can still matter to a deterministic editor.

Main Findings

• All-8 diagnostic rises. Moving from one to two instruction copies raises the deterministic All-8 diagnostic (responses passing all eight observable tests) from 90.22% to 93.17% (+2.95 percentage points; 95% CI [2.14, 3.79]), eliminating 30.2% of the failures remaining after one copy (machine-detected failures fall from 9.78% to 6.83%).

• Lexical exposure improves; accuracy does not. Pre-provisional TF–IDF recall rises from 73.44% to 74.81% (+1.38 points, Holm-adjusted p = .00012), but final-answer accuracy is 3793/6300 (60.21%) in both arms. The equality is a dissociation, not identical output: 341/2100 model–question blocks change their number of correct placements (170 favor two copies, 171 favor one), and 480 blocks change their number of All-8-complete placements, with 379/480 (79.0%) of those doing so with no accuracy-count change.

• Small protocol gains beyond recall. Contrastive discussion rises from 97.07% to 97.76% (+0.69 points, p = .00020) and deterministic role count from 7.818 to 7.845 (+0.027, p = .0356). Section count is essentially flat (7.937 → 7.922, −0.015, p = .244). The six non-bold outcomes form one exploratory Holm family.

• A real adverse effect. Premature commitment (firm answer selection before the provisional role, among completed generations) increases from 1.52% to 2.30% (+0.77 points, p_Holm = .00536) and reaches 3.12% with three copies.

• Saturation, not synergy. The factorial decomposition shows strong saturation: the first placement captures most of the available improvement and additional locations have diminishing returns (all TF–IDF location and interaction terms, sign-flip p < .001). Additivity residuals are strongly negative for TF–IDF and the other bounded protocol measures (p < .001) and null for accuracy (p = .30–.77).

• After-query placement alone is not a strong lexical lever. Relative to the mean of system-only and before-only, the after-query condition changes TF–IDF by +0.31 points (95% CI [−0.19, 0.83], p = .239) and accuracy by +0.62 points (p = .392). This rules out reading the results as a general "put the instruction last" effect.

• A trailing duplicate is the strongest tested placement. Comparing S → S+A and B → B+A, TF–IDF recall rises from 73.33% to 75.53% (+2.19 points, 95% CI [1.76, 2.64], p < .0001). All-8 rises by +3.31 points, contrastive discussion by +0.80 points, and roles by +0.040; section count and accuracy are unchanged.

• The effect is distributed across models. TF–IDF and All-8 point estimates are positive in all seven models; five and three models, respectively, survive within-endpoint Holm correction, while no model has a corrected accuracy effect.

• Length and safety checks are mostly benign. Pre-answer content length increases from 201.0 to 216.0 tokens (+7.5%); in the observed-length robustness regression the pooled two-copy coefficient is +0.36 points (95% CI [−0.01, 0.72], p = .054), and the trailing-duplicate observed-length coefficient remains +1.24 points (95% CI [0.76, 1.72], p = 4.5×10⁻⁷). Truncations by copy count 0/1/2/3 are 28/48/59/17 (two-minus-one +0.175 points, p = .166); hard failures are 0/6300 versus 2/6300 (p = .502).

• The human audit does not validate every automatic transition. Of 30 machine-positive All-8 transitions, 20 are perceptual ties; all 10 non-tied pairs favor the duplicated/machine-preferred response (one-sided exact sign p = .00098), so the prespecified 28/30 confirmation criterion is not met. The paper reads this as a human/controller boundary rather than a quality gain.

• Downstream AE results are large in the target scope. In the SSNHL scope, the published reason-first no-editing endpoint was 25.1%; published AE with trajectory editing reached 83.5%; the later system-only AE reproduction scored 842/1000 (84.2%); adding the same trailing duplicate raised it to 971/1000 (97.1%, +12.9 points, approximate 95% CI [10.4, 15.4], exact McNemar p ≈ 7.2×10⁻²⁴).

• The conductive control branch weakens. Published reason-first no-editing was 58.9%, published AE 77.9%, the system-only reproduction 786/1000 (78.6%), and duplication 738/1000 (73.8%) — a within-AE decrease, but still 14.9 points above the no-editing baseline.

• Eligibility and scoring are fixed in advance. TF–IDF is eligible for 257/300 questions; the 43 exclusions are topic-only or choice-dependent stems without a fact inventory (32), plus 11 that also trigger a very-short-stem flag. IDF weights come from a frozen global document-frequency table over 15,103,887 PubMed abstracts rather than from the 300-question sample.

Methodology in Plain English

The authors take seven instruction-tuned models with visible output (Gemma 3 12B, Llama 3.3 70B Instruct, Llama 4 Scout, Ministral 3 14B Instruct, Mistral Large 3, Qwen3 30B-A3B Instruct, and Qwen3 235B-A22B Instruct) and 300 medical multiple-choice questions — 100 each from MedQA, MedXpertQA, and AfriMed-QA. Every question is asked under all eight instruction placements, with temperature 0, cell-specific seeds, pinned model/provider routes, and model-specific output ceilings: 7 × 300 × 8 = 16,800 scheduled cells, of which the frozen run contains 16,646 completed generations, 152 truncations, and 2 hard failures.

The instruction asks for eight headings in order — Facts, Implications, Provisional answer, Best alternative, Decisive distinction, What would change the answer, Reconsideration, and Final answer — and forbids selecting an answer before the provisional section. Instead of relying on human judgment, the authors score responses mechanically: how many of the requested sections appear with non-trivial content (0–8), which deterministic roles are present, whether the contrastive elements are discussed, whether the model commits prematurely, and whether the final answer is right.

To measure how much of the question's content the model surfaces before committing, they compute a TF–IDF-weighted recall of stem terms recovered in the Facts and Implications sections, using a pre-frozen IDF table rather than one estimated from the 300 questions. Paired contrasts use 10,000 question-cluster bootstrap resamples for confidence intervals and 50,000 paired sign-flip draws for two-sided p-values, with Holm correction across the six pooled outcomes. They also fit an additive model from the zero- and one-copy cells to test whether higher-copy cells behave better than the sum of their parts.

Finally, they run a blinded human audit of the automatic transitions (hidden identity, placement, preference, stratum, gold answer, randomized A/B orientation, one non-clinician author judging only whether the displayed procedural criterion was visibly satisfied) and a separate downstream reproduction showing how an Answer Engineering editor behaves when the trajectory it edits is produced with and without the duplicated instruction.

Why This Matters

Impact on research. The paper reframes prompt-level repetition as a control variable over exposed trajectory state rather than a quality knob over answers. It separates formal compliance from human-perceived compliance, shows that the same intervention can help one clinical branch and weaken a nearby contrastive branch, and argues that controller-facing prompt evaluation must report the end task, the observability of downstream-consumed states, and adverse effects together. It also reports its own negative evidence — an audit criterion that was not met, accuracy that did not move, and premature commitment that rose.

Real-world applications.

• Clinical decision-support pipelines in which a downstream parser, monitor, or auditor must recognize an explicit alternative, qualifier, or retain/revise marker that a human reader could otherwise infer from context.
• Agentic and tool-use systems where the generated trajectory is an interface (a machine-addressable API) rather than a final answer for a person.
• Trajectory editing or repair systems, such as the Answer Engineering editor used here, which can only act on states the model makes observable.
• Prompt engineering for structured output formats, where adding an extra copy of the format instruction before generation may be cheaper than retraining, decoding constraints, or activation steering.

Industry relevance. The intervention costs only prompt tokens, requires no model internals, no retraining, and no decoder changes, which makes it easy to A/B test in production. The paper's practical rule — test duplication against the controller's own endpoint and retain an adverse-effect check — is directly usable by teams that ship verification or repair layers on top of model output. The SSNHL substring endpoint (25.1% → 84.2% → 97.1%) illustrates how large the downstream swing can look once a deterministic controller consumes the trajectory.

Future Directions

• A matched controller-on/off factorial. The authors state that the present design does not establish that TF–IDF, All-8, or any other single measured feature mediates the downstream AE effect; a factorial crossing placement with controller presence would be required for that causal decomposition.

• Semantically equivalent instruction paraphrases and a frozen measurement stack. The paper recommends that a confirmatory study freeze the measurement stack before generation and test paraphrases of the instruction rather than one fixed wording.

• Transfer beyond the current panel. The seven models are described as a selected contemporary panel, not a random sample from a defined model population, and all three datasets are medical multiple-choice benchmarks; transfer to other domains, instruction families, and serving stacks is unestablished.

• Stronger human validation. The blinded audit used one non-clinician author and oversampled machine-positive changes, so it is neither a prevalence sample nor an inter-rater study; multi-rater, prevalence-based audits would test whether threshold crossings track anything a reader would endorse.

• Replication under changed serving stacks. Verification is described at three levels — frozen-output verification, deterministic rejudging, and model regeneration — and the paper notes that hosted regeneration may differ because provider-side infrastructure and bitwise determinism are not fully exposed.

Target Audience

Researchers and engineers working on inference-time control, prompt-placement effects, and instruction following will find the factorial design and the dissociation between protocol state and accuracy most useful. Practitioners building verification, auditing, or trajectory-editing layers on top of language models — especially in clinical or other high-stakes domains where intermediate reasoning must be inspected — are the intended beneficiaries of the downstream AE results. Readers interested in evaluation methodology will value the paper's explicit separation of exploratory from confirmatory claims, its Holm-corrected families, and its reporting of a failed prespecified audit criterion. The paper is not a source of clinical safety or competence evidence, and it does not report identifiable patient data or external human participants.

Authors’ abstract

Procedural instruction following is a basic requirement for controllable language-model systems, especially when generated trajectories are inspected or repaired downstream. We introduce instruction duplication, a minimal black-box inference-time control that repeats only the procedural instruction, without retraining or decoding changes. Across seven instruction-tuned models, 300 medical multiple-choice questions, eight placement conditions, and 16,800 scheduled generations, moving from one to two copies raises the deterministic All-8 diagnostic--responses passing all eight observable tests--from 90.22% to 93.17% (+2.95 percentage points), eliminating 30.2% of the failures remaining after one copy. Pre-provisional TF-IDF recall rises from 73.44% to 74.81% (+1.38 points; Holm-adjusted p &lt; .001), while final-answer accuracy remains exactly 60.21%. Premature commitment increases from 1.52% to 2.30% (p_Holm = .00536). A blinded challenge audit yields 10/30 directional confirmations, 20/30 perceptual ties, and no reversals; its prespecified 28/30 confirmation criterion is not met. Yet this distinction can matter operationally when a downstream system acts on the generated trajectory. In Answer Engineering (AE), where explicit trajectory state determines local repair, the published reason-first no-editing SSNHL endpoint was 25.1%; system-only AE was later reproduced at 84.2%, and the same trailing duplicate raised it to 97.1%. For conductive diagnostic branch preservation, the corresponding values are 58.9% published without editing, 78.6% with reproduced AE, and 73.8% with AE plus duplication--a within-AE decrease, but still 14.9 points above the no-editing baseline. Instruction duplication is therefore a low-complexity, placement-sensitive control whose practical value can emerge through the downstream system that consumes the exposed trajectory.