Ignorance or Incompetence? Constructing Knowledge-Gated, Verifiable Tasks for LLM Agents

AI summary

Overview

Research area: Agentic AI task construction, benchmark design, and verification for LLM agents — specifically how to build tasks whose difficulty comes from deliberately withheld domain knowledge rather than from raw reasoning difficulty.

Technical level: Advanced. The paper is a methods and validation protocol paper that assumes familiarity with agentic evaluation harnesses, reinforcement learning with verifiable rewards (RLVR), pass-rate calibration, and item-response-theory-style screening.

One-sentence scope: The paper proposes and empirically calibrates a construction contract in which an agent task is paired with a small, authored "knowledge artefact," with the task instruction held byte-identical whether or not the artefact is supplied, so that dependence on private domain conventions becomes an auditable experimental variable.

What This Paper Is About

Constructing agent tasks in professional domains runs into a paradox: if the task generator already knows every rule needed to solve the task, the task may not test anything beyond the generator's own capability; if the rules are genuinely unavailable, the task becomes under-specified and hard to verify. The authors treat this as a construction problem rather than a claim about self-improvement, and build a protocol in which a human curator plants private conventions into a kilobyte-scale artefact that the agent either receives or does not. The goal is to make knowledge access an explicit, leak-audited, testable variable, and then calibrate which constructed tasks actually exhibit the intended behaviour.

Key Contributions

1. A knowledge-gated task-construction contract. Each task is paired with a kilobyte-scale artefact carrying private domain conventions, reference tables, and utility operators. Instructions are byte-identical across the provided-artefact (+A) and withheld-artefact (−A) conditions, and a static leak audit checks instructions and environments for exposed gated content. Executable per-task audit scripts cover nine of the fifteen calibration tasks.

2. A validation and attribution protocol. Deterministic witnesses give exact verification for structured outputs, named rubric criteria localise failures for open-form outputs, and provenance plus leak audits delimit what each task can claim. The protocol distinguishes non-derivable convention gates from re-derivable operator gates so the two are not presented as the same construct.

3. A calibration study of the construction protocol. Paired artefact ablations, one perturbed-artefact control, and a two-task recitation probe show where the intended dependency holds and where a full agent harness introduces additional failure modes. The resulting seven-task retained set is described as an operationally calibrated release candidate, not a demonstrated source of superior training signal.

4. A partial public release. Five representative retained tasks and supporting tooling are released at https://github.com/DatagridsAI/Knowledge-Gated-Task-Construction. The authors state that no training is performed on this data.

Main Findings

• The artefact toggles performance between total failure and majority success. Across fifteen calibration tasks with byte-identical instructions and exactly five trials per cell, pooled pass rates are 68.0% for the frontier configuration with the artefact, 0% without it, and 22.7% for the second configuration with the artefact. The aggregate table reports 0/75, 51/75, and 17/75 respectively.

• The frontier configuration is Claude Opus 4.8 via the Claude Code agent (BenchFlow's claude-agent-acp integration), and the second configuration is Qwen3.6-Plus via the OpenCode agent, always with the artefact. The authors state these are configuration comparisons, not controlled model comparisons, because both model and harness differ.

• A plausible but wrong artefact is no better than no artefact. On the bom-costing task, Opus with a perturbed artefact (three gated conventions changed: overhead rate from 12% to 15%, missing-leaf-price rule from "0.00" to "average of known sibling prices," circular-reference value from "−1.00" to "0.00") achieves a 0% pass rate across five trials, matching the no-artefact condition and falling below the 100% achieved with the correct artefact. This is reported as a single-task, single-model control.

• Seven of fifteen tasks pass the calibration screen. The retained set is legal-clause, paper-index, bom-costing, airline-booking, clinical-variant, mortgage-escrow, and dependency-audit. Three (airline-booking, mortgage-escrow, dependency-audit) sit exactly at the 40% second-configuration ceiling, and dependency-audit also sits exactly at the 60% frontier threshold — described as boundary cases under a coarse screen.

• Eight tasks are rejected in two distinct ways. Four are borderline: trial-cohort misses the frontier +A threshold, while msa-redline, fuzz-secure, and esg-synthesis exceed the second-configuration ceiling. Four fail more severely: cpi-tracker reaches only 40% with the artefact at the frontier configuration, and enterprise-search, inventory-rollforward, and sec-13f remain at 0% even with the artefact.

• A recitation probe localises one failure to the harness, not to knowledge. On bom-costing and legal-clause, Qwen recites every gated convention correctly and reaches a 100% pass rate across 10 independent conversations per task (20 total), with data inlined and no agent loop or tool use — yet it records 0% across five trials on both tasks under the full sandboxed agentic harness. The authors conclude the harness-level failure is more likely explained by tool use, file discovery, and output-format compliance than by missing knowledge or computational inability.

• Contextual public benchmarks show smaller, directionally consistent gains. On τ²-bench telecom (114 paired tasks, official four-trial results), adding a procedural workflow manual lifts GPT-4.1 from 34.2% to 51.8% and o4-mini from 42.1% to 59.4%; among tasks with zero baseline successes the manual rescues 53% for GPT-4.1 and 57% for o4-mini. SciCode's background toggle lifts Claude-3.5-Sonnet from 26.0% to 35.4% at the subproblem level and GPT-4o from 25.0% to 35.4%. The authors note these benchmarks implement none of their mechanisms and provide context rather than replication.

• The retained set shows no meaningful variance advantage. Pooling both +A configurations over all fifteen candidates gives 30 cells, of which 40.0% are mixed with mean variance 0.088; restricting to the seven retained tasks gives 14 cells, 42.9% mixed with mean variance 0.091. The authors explicitly decline to treat this difference as a headline result because of selection circularity.

• Operative content can be very small, but no compression curve is established. On paper-index, a full 4379-byte SKILL.md and a hand-written 1078-byte minimal rewrite (a 4× compression) both achieve 100% pass rate, while the 0-byte condition achieves 0%, at five trials per condition. The authors state the true breaking size lies somewhere below 1078 bytes and is untested.

• Every reported pass rate uses exact executable verification. The rubric-judging path is part of the protocol but is not exercised by any reported number, so no LLM judge is involved in the headline results.

Methodology in Plain English

The authors treat task construction as a controlled procedure built on three asymmetries the curator holds but the agent does not: the answer is planted at construction time and cannot be derived from the instruction or environment; checking a structured output against that planted answer is cheap even when producing it is not; and an all-pass verifier over named criteria lets complexity grow while each failure stays attributable.

Operationally, each curated unit is a pair — a task (inputs, output schema, formatting and ordering requirements, success criteria) and a knowledge artefact (terminology, conventions, formulas, reference tables, low-level utility operators, never an end-to-end solution script). The instruction says what is required and never mentions the artefact; the artefact says how the domain computes.

Two kinds of gates are planted. Convention gates fix private, non-derivable choices such as an internal threshold, a fixed alias or normalisation table, or one defensible definition selected among several — the paper's examples include whether a turnover metric excludes opening minutes and a baseline denominator of 235 rather than 240. Operator gates put hard-to-hand-write algorithms such as Brandes betweenness under a fixed normalisation convention, or Garman–Klass volatility, into the artefact's utility library, creating a steep capability barrier rather than a strictly non-derivable one. Orchestration — parsing, grouping, deduplication, graph construction, serialisation — is always left to the agent.

Fairness is handled by keeping instructions byte-identical across conditions, referring to conventions only as "the established in-house specification," and adding a fallback clause telling the agent to apply common conventions when no specification is present, so the −A agent submits confident answers rather than abstaining. A static audit then scans instructions and environment data for explicit gated constants and any mention of the artefact.

Each candidate is run for five trials under three conditions, and retained only if the frontier configuration scores at least 60% with the artefact, exactly 0% without it, and the second configuration scores at most 40% with the artefact. Tasks failing the rule are revised or discarded. Verification parses structured outputs, recomputes references with independent Python implementations, and checks planted values against strict tolerances, typically 10⁻⁴ for floating-point comparisons; free-form deliverables would use per-criterion all-pass rubric judging with the reward defined as the product of binary criteria.

Why This Matters

Impact on research. The paper reframes a common ambiguity in agent evaluation — was the agent ignorant or merely incompetent? — as a construction-time variable that can be planted, audited, and toggled. It also draws a sharp line between tasks that are hard because knowledge is unavailable and tasks that are hard because execution is hard, and its recitation probe shows the two can be confused by a full harness. The authors are careful that their screen is a quality-control rule, not a statistical test and not evidence of training value, which makes the paper a template for how construction-stage selection should be reported.

Real-world applications (drawn from the paper's own task domains):

• Contract review workflows, where clause risk scoring depends on in-house point values, caps, complexity bonuses, and jurisdiction multipliers that appear nowhere in public corpora.
• Financial and operational reporting, such as mortgage escrow analysis and inventory rollforward, where internal thresholds and normalisation tables determine the correct answer.
• Manufacturing cost rollup, where compounding overhead rates, missing-leaf-price rules, and circular-reference handling are company-specific conventions.
• Publication and dependency auditing, where precedence orders, alias tables, and graph-centrality conventions must be applied exactly as an organisation defines them.

Industry relevance. The protocol is aimed at teams that build agent post-training corpora and need to know whether a task's difficulty comes from withheld domain knowledge or from generic execution friction. The perturbed-artefact control is particularly relevant to deployment: a retrieved document that looks authoritative but states the wrong conventions performs no better than supplying no document at all. The authors also note that executable leak audits currently cover nine of fifteen tasks, which sets a concrete bar for how much verification infrastructure a release needs.

Future Directions

• Complete the audit and tighten the statistics. The authors' first stated next step is to complete executable leak audits for every task, increase repeated trials around decision boundaries, and rerun the screen with a common harness plus held-out agent configurations — addressing the fact that automated audits cover nine tasks and that five trials per cell make retention decisions sensitive to a single outcome.

• Run an end-to-end post-training comparison. The central unproven claim is that calibrated tasks improve learning relative to matched ungated or unscreened tasks. This requires a held-out, end-to-end post-training study, which the paper explicitly places outside its evidence.

• Test durability under test-time self-improvement. The authors conjecture that convention-gated tasks will resist self-critique loops because nothing in the instruction or environment reveals the planted thresholds, while operator gates may remain susceptible to advanced tool-use, search, or scaled test-time compute. This conjecture is untested.

• Extend the recitation probe and the compression study. The probe covers two convention gates, one model, and one reduced condition; extending it to paper-index, other models, and a controlled common harness remains future work. Establishing a real compression–performance curve would need intermediate artefact sizes, several tasks and domains, and per-rule ablations.

• Reconcile training data with evaluation. Because the same task cannot serve as both released training data and an uncontaminated evaluation item, the authors propose using physically disjoint task families, canary strings such as random GUIDs to detect contamination, and parameterised regeneration to reduce exact memorisation.

Target Audience

Readers who will benefit most are researchers and engineers who design agent benchmarks, build post-training corpora for LLM agents, or work on reinforcement learning with verifiable rewards and pass-rate filtering. It is also relevant to evaluation leads in regulated professional domains — legal, finance, healthcare, manufacturing, and software supply chain — who need tasks that test whether an agent applies an organisation's private conventions rather than merely plausible ones. Readers looking for a demonstrated training benefit, a statistical validation of the screen, or a general compression algorithm will not find those here; the paper is explicit about each of those boundaries.

Authors’ abstract

Professional agent tasks often depend on conventions that are absent from public corpora, yet benchmarks rarely control whether an agent has access to those conventions. We introduce a knowledge-gated task-construction protocol that separates a task instruction from a compact artefact containing private conventions, reference tables, and utility operators. Construction-time provenance, byte-identical task instructions across the provided- and withheld-artefact conditions, leak audits, and executable witnesses make dependence on the artefact explicit and testable. Across fifteen calibration tasks, one frontier agent configuration achieves a 68.0% pass rate with the artefact and 0% without it; on one task, a plausible but incorrect artefact also yields 0% across five trials. Deterministic solvers and rule corpora provide exact ground truth for structured tasks, while named criterion-level rubrics support outputs that cannot be checked by a single executable oracle. A configuration-relative calibration screen retains seven tasks satisfying our five-trial empirical knowledge-gating screen. These experiments validate the behavior of the construction protocol; they do not establish that the retained tasks improve post-training. We publicly release part of the task suite and supporting tooling at https://github.com/DatagridsAI/Knowledge-Gated-Task-Construction.