Cybernetic and Epistemic: A Missing Vocabulary for Trustworthy Agentic Delegation

AI summary

Overview

Research area: Trustworthy agentic AI systems, human oversight of AI agents, and the philosophy of language applied to human–agent delegation. The paper sits at the intersection of AI governance, interpretability/explainability research, and coding-agent supervision.

Technical level: Intermediate. There are no formal models, proofs, or experimental results; the argument is conceptual and draws on philosophy of language (Austin, Searle, Habermas, Frankfurt), human-factors research, and legal reasoning. Familiarity with the AI oversight and explainability literature helps but is not required.

Scope (one sentence): The paper argues that trustworthy agentic delegation needs a missing vocabulary distinguishing language that coordinates action ("cybernetic") from language that coordinates understanding ("epistemic"), and proposes a governance criterion, an operational reconstruction test, and a deliberation-recording convention (ORRCF) built on that distinction.

What This Paper Is About

As code generation is delegated to AI systems, the paper argues the bottleneck in software production is shifting from writing code to supervising the systems that write it. The problem it names is a vocabulary gap: the field asks for "human oversight" without distinguishing between language that steers action and language that conveys checkable understanding. The specific failure this gap hides is explanation-shaped output calibrated for approval rather than truth — epistemic-form language doing cybernetic work — and the goal of the paper is to name that failure, give it a testable criterion, and specify what a delegation record must contain so a third party can tell a decision from a rubber stamp.

Key Contributions

1. A named distinction with direct consequences for oversight design. The paper defines cybernetic language (coordinates action; succeeds when the world matches the words) and epistemic language (coordinates understanding; succeeds when the words answer to the world and a hearer can check that they do), and identifies the pathology as epistemic-form language doing cybernetic work. It departs deliberately from Wiener's broader use of "cybernetic," using the word narrowly for the communicative function of steering.

2. A governance criterion for agentic systems. Every consequential choice in a delegation channel should carry the condition under which it would have gone otherwise, in a form a third party can test. A decision carrying no such condition, stated or reconstructible, gives a third party no way to distinguish it from a rubber stamp.

3. An operational two-part reconstruction test. The first part gives a second reader only the record and asks, for a stated perturbation, whether the choice holds or flips and what in the record licenses that prediction. The second part applies the perturbation: the same model, at the same version, is queried afresh on the brief the record preserves — once as it stood and once perturbed, with the record withheld both times — and the prediction is scored against the difference. The paper notes the first part alone can be gamed, which is why the second part exists.

4. ORRCF, a deliberation-recording convention. Options, Recommendation, Rationale, Confidence, Falsifier — where the falsifier is the evidence, observation, failed prediction, or changed constraint that would cause the recommendation to be revised. The paper presents this as one convention with exactly the five components the criterion asks for, descended from architecture decision records and consistent with Toulmin's qualifier and rebuttal.

Main Findings

• The approval checkpoint is not a comprehension requirement. An oversight regime that only asks a human to approve or reject an output is satisfiable by a human who understands nothing about it. The paper observes that several contemporary coding agents still ask a human to approve or deny a shell command with no realistic way to assess its consequences in the time available.

• Explanations can add persuasion without adding discrimination. Bansal et al. (2021) found that explanations increased participants' acceptance of an AI's recommendation regardless of whether it was correct, without improving complementary performance. Follow-up work by some of the same authors (Chen et al. 2023) traces this to explanation form: feature-attribution displays produce this outcome, while example-based explanations that support the reader's own inductive reasoning measurably improve decisions. Kim et al. (2025) find the pattern replicates in language-model settings and that what restores discrimination is a checkable source or a noticeable inconsistency.

• A stated reversal condition does not by itself separate truth from gloss. A gloss may state a condition too; what a stated condition does is give a third party something to test, since a fabricated condition can fail a check and an absent one forecloses it.

• Composition compounds the problem in agent hierarchies. In oversight hierarchies where agents supervise agents with a human at the top, each layer optimizes against its model of the layer above's judgment, and the approximation error compounds silently because a purely cybernetic signal is testimonial. A retrievable reason, by contrast, can be examined at any layer — on a spectrum from legible (stated, readable by a third party) to contestable (stated with what would overturn it) to verifiable (checkable against what the agent actually does).

• Three delegation episodes, offered as illustrations rather than controlled evidence. All are single-practitioner and self-reported, each consistent with the rival explanation that a skilled practitioner with strong prior context would have gotten good results under any framing, and no cybernetic counterfactual was run. (a) A design problem stated as one open sentence with no implementation specified produced a working artifact on the slides three minutes later — a horizontal strip of classified, auto-generated cells tracking the session's structure, with the current position marked and click-to-jump navigation, derived from the deck's own markup; the paper attributes this to amortized prior written deliberations rather than prompting working once. (b) A whole mobile application was built by multiple AI collaborators across multiple sessions, with the author holding purpose and none of the implementation ground, exercising oversight through screenshots and reasons attached to each choice. (c) The initial architecture of a software project was completed in a few hours, in one sitting, by a different vendor's model, producing a public repository with a governance policy, a database schema, verification tooling, and six fully-formed decision records; one answered question later prompted the collaborator to find and correct a real bug in its own governance-enforcement code and to record the correction in the same record.

• A public record where a recommendation was withdrawn on its own stated terms. In a widely used GitHub Action for freeing runner disk space (the author discloses the repository is his and the maintainer is him), a deliberation record decides that an option removing the runner's swap file should default to off, because the resulting out-of-memory failure is not attributable by the person who suffers it — one user reported needing "a couple of days" to trace a failure back to the option. A first draft also recommended flipping a second option, the tool cache, to off; at implementation review a collaborator checked the code and found it had already defaulted to off since 2022. The recommendation was withdrawn and the record keeps both the mistaken premise and its correction in place. The paper claims nothing comparative — it shows what a record carrying its grounds makes possible, not that the convention caused the catch.

• The negative case: cybernetic oversight without epistemic reconstruction. In Virginia's felony sentencing of nonviolent offenders, judges encountered a recommendation as a checked box backed by a risk score findable only by paging through worksheets. Written justification was required only outside the guidelines range, and the recommendation was incorporated by expanding that range, so both diverting a low-risk defendant and ignoring the recommendation were compliant. Stevenson and Doleac (2024) find that, at a given risk score, judges sentenced Black defendants to terms 12–24% longer than White defendants — a gap robust to guidelines and judge controls and essentially unchanged by a race-inclusive alternative score. Whether adopting the tool worsened the gap is separate: the estimated change is small and not significant, and the study does not establish whether adoption increased the disparity. A second gap belongs to the score: race is never an explicit input, yet Black defendants are ten percentage points less likely than White defendants with the same guidelines-recommended sentence to be classified low-risk. Age inverts the pattern — the score penalizes youth, judges' discretion favors them.

• What the negative case does not show is stated plainly. It does not show that requiring reasons would have reduced the disparity, and the paper does not read it as doing so. Mandated justification can rationalize a decision as readily as constrain it — a reason produced after a decision for an audience whose expectations are known is exactly the situation where explanation-shaped speech does cybernetic work. Ge, Bastani, and Bastani (2024) respond with "compliance-robust fairness," which the paper calls a genuine improvement and a metric-level fix that does not ask either side to carry a reason.

• The test's outcome table. The paper presents a table crossing whether the agent flips or holds against whether the record says it should flip or hold, yielding four observations: change observed, change absent, unpredicted change, and stability. The bottom-left cell — record says hold, agent flips — is the one to watch, since an unnamed ground, missing context, or unstable behavior could all produce it, and only investigation tells them apart. The table records agreement, not causes.

• Three qualifications make the test usable. The score is on the option chosen, not on whether anything changed, and a failure to reproduce the recorded baseline is reported on its own. A predicted reopening is a qualitative result unless the protocol defines a disposition for it, and an underdetermined case is reported separately rather than scored as a hold. The model queried must be the one that wrote the record, with settings and context preserved and matched runs repeated where behavior varies.

• The strongest objection is acknowledged. Work on reasoning-trace faithfulness shows a model's stated reasoning can omit the features that drove its answer (Turpin et al. 2023), measured by intervening on the trace and watching whether the answer moves (Lanham et al. 2023). The paper calls this "a retrievable statement of reasoning is not retrievable reasoning" and treats it as the methodological precedent for scoring by intervention rather than inspection.

• Human-subjects parallels are cited as parallel, not as transfer. Across 23 experiments, a rationale attached to a request improved engagement and performance (small to moderate effects) and worked best when autonomy-supporting rather than controlling, while slightly lowering recipients' sense of their own competence (Steingut, Patall, and Trimble 2017). In organizations, the explanations a supervisor gives predicted employees' subsequent trust in the supervisor even controlling for prior trust and trustworthiness (Colquitt and Rodell 2011).

• A reception risk is named but not tested. A listener exposed often enough to explanation-shaped steering may come to treat all explanation as steering in disguise, so that genuine explanations stop landing.

Methodology in Plain English

This is a conceptual and argumentative paper, not an empirical study. Its method is:

1. Name a distinction and test its fit. The author imports a division from philosophy of language — direction of fit, Habermas's communicative versus strategic action, Maturana and Varela's "languaging" — and argues the cybernetic/epistemic pair classifies an utterance by what it does in a delegation channel rather than by the speaker's orientation, which is why the existing pairs were not adopted.

2. Illustrate at three scales. Three self-reported delegation episodes cover a single decision, a whole application, and an entire project. The paper is explicit that these are illustrations, not evidence, and that no counterfactual was run.

3. Anchor the diagnosis in a public record and a peer-reviewed study. A public GitHub record shows a recommendation withdrawn because a stated premise was checked against code. The Virginia sentencing study — peer-reviewed econometrics rather than self-report — motivates the question of individual-decision auditability, deliberately an asymmetry of evidence type.

4. Turn the criterion into a procedure. The reconstruction test is specified in two parts, with a four-cell outcome table and three qualifications for use, plus notes on deployment: the second part is available for AI agents as it never was for judges, a convention that keeps the originating brief verbatim makes re-querying possible, and the test is sampled and periodic — belonging to evaluation and gap analysis in a system's lifecycle — rather than applied to every decision of a live pipeline.

5. Situate against existing literature. The proposal is positioned beside, not against, Bainbridge's automation-monitoring observation, automation bias (Skitka, Mosier, and Burdick 1999), Green's argument that oversight policies function as rubber stamps, and the finding that forcing a decision before seeing an AI's recommendation reduces overreliance but is less preferred by users (Buçinca, Malaya, and Gajos 2021). The paper's claim is that this literature names the form of the artifact and its effect on reliance, but not the communicative function it serves.

Why This Matters

Impact on research: The paper argues that auditability, explainability, and human oversight — properties trustworthy-agentic-systems research already asks for — are under-specified. An audit trail, an explanation, or an oversight checkpoint can be built to satisfy a cybernetic demand (produce an artifact that will be approved) without satisfying an epistemic one (produce an artifact from which a human can reconstruct why the system did what it did, and could have done otherwise). It proposes that evaluations ask a third question — did the reasoning survive the delegation in a form a human can retrieve and revise — that is neither redundant with "was the outcome correct" nor with "was the process constrained," and offers the gap between an approval rate and a reconstruction-accuracy rate as something an evaluation could measure on matched cases. Whether the two diverge is described as empirical.

Real-world applications (as discussed in the paper):

• Coding-agent supervision: approving or denying shell commands, merge conflicts, and PRs, where the paper's example is a record stating a reason that would have read otherwise had the older version been right.
• Criminal justice risk tools: the Virginia sentencing case, where the choice between diversion and a compliant custodial sentence was never required to carry an assessable justification, and the pattern was visible only afterward in the aggregate.
• Software maintenance and open source: a widely used GitHub Action whose design reasons previously "lived only in my head," where a stated premise about the code was checked against the code and a recommendation withdrawn.
• Multi-agent oversight hierarchies: agents supervising agents with a human somewhere at the top, where the paper argues retrievable reasons reduce dependence on the endorsements of intermediaries.
• Organizational trust: since, in human channels, a supervisor's explanations predict employees' subsequent trust in the supervisor.

Industry relevance: Any organization deploying agents under a "human-in-the-loop" design should care that such a checkpoint is satisfiable without comprehension. The ORRCF convention, being descended from architecture decision records, is intended to fit existing engineering practice. The paper also notes that provenance and assurance-case notations — W3C PROV, Goal Structuring Notation, and Toulmin's layout behind assurance cases — already give a retrievability requirement a schema, but none settles whether what fills the schema describes the decision or decorates it. Any schema, ORRCF included, "becomes ceremony the moment its fields are filled and never checked; ceremony is a stamp with more boxes." That is why the criterion is not "has a falsifier" but "has a falsifier that survives a perturbation."

Future Directions

• Measure whether approval and reconstruction diverge. The paper states that the gap between an approval rate and a reconstruction-accuracy rate is what an evaluation could measure on matched cases, and that whether the two diverge is empirical and cannot be seen by approval checkpoints either way.
• Assess the cost of the first part's second question at record creation. Whether a reader can name a revision condition at all could be checked when a record is created, at a cost the paper says is still to be assessed.
• Test the reception-failure hypothesis. Whether frequent exposure to explanation-shaped steering causes listeners to treat all explanation as steering in disguise — so genuine explanations stop landing — is named as a longer-term effect this paper does not test.
• Handle weaker evidence forms without over-reading them. The paper notes that a prediction mismatch calls for investigation rather than a verdict, since it may reflect an omitted ground, missing context, unstable behavior, or a condition the record got wrong and the model was right to ignore. Distinguishing these remains open.
• Keep the convention from becoming ceremony. The paper flags that any schema, including ORRCF, becomes a stamp with more boxes once its fields are filled and never checked — and the design tension between keeping a convention loose enough to preserve the decider's own words while making it operational is left as a live concern.

Target Audience

Readers most likely to benefit: researchers and practitioners working on agentic-system trustworthiness, AI oversight, and interpretability/explainability evaluation; engineering teams deploying coding agents or multi-agent pipelines under human-in-the-loop designs; AI governance and policy researchers interested in auditability requirements and their legal analogues (the paper cites New Jersey's duty of articulation under Curtis v. Finneran (1980) as the legal system's version of the criterion); and computing-education researchers, since the paper builds directly on the observation by Albluwi et al. (2026) that curricula should teach supervision, constraint, and auditing of semi-autonomous agents. Readers who want the argument's philosophical grounding will also find it useful, but the paper is written to be usable by engineers and evaluators without that background.

Authors’ abstract

As code generation is increasingly delegated to AI systems, the bottleneck is shifting from writing code to supervising the systems that write it --- a shift CS-education researchers have begun to name. This shift exposes a vocabulary gap: the field asks for "human oversight" without a working distinction between the two things language does in a delegation channel --- coordinate action (cybernetic: words succeed when the world comes to match them) and coordinate understanding (epistemic: they succeed when they answer to the world and a hearer can check that they do). The failure this names is not cybernetic language but epistemic-form language doing cybernetic work: explanation-shaped output calibrated for approval rather than truth. Oversight that checks only whether an output was approved is satisfiable by rubber-stamping; oversight that holds an agent accountable requires the reasoning behind its work be retrievable and checkable. We present three delegation episodes --- illustrations, not controlled evidence --- in which epistemic engagement proved practicable while remaining auditable, one public record where a recommendation was withdrawn on its own stated terms, and one failure case illustrating oversight that requires no reasons for its discretionary choices. We propose a criterion for agentic-system governance, alongside existing technical trust properties: every consequential choice should carry the condition under which it would have gone otherwise, in a form a third party can test. Without such a condition, a third party cannot distinguish a decision from a rubber stamp. We give the criterion an operational form --- a two-part reconstruction test scoring a delegation record by whether a second reader can predict what the agent does under a perturbation --- and a deliberation-recording convention, ORRCF, that makes the condition a required component of every recorded choice.