MasterControl Seventeen Every Time

AI summary

Overview

Research area: Governed enterprise data analysis, large language model (LLM) agents for analytics, deterministic policy execution, and the theory of relational/analytical query languages. The paper sits at the intersection of database theory (finite-domain first-order logic, relational algebra) and applied LLM systems evaluation.

Technical level: Advanced. The first half (Theorems 1–4 and their proofs in Appendix A) assumes comfort with first-order logic, finite domains, relational operators, and induction proofs. The second half is a controlled empirical benchmark that is readable without that background.

Scope (one sentence): The paper defines a small typed analytical language, proves it is complete for a stated class of finite analyses and replayable under fixed governed state, and reports a 440-episode experiment in which runtime tool-planning agents produced 0 exact answer-and-evidence contracts while a policy-executed analyzer produced 110 of 110.

Note on naming: the arXiv metadata lists the title "MasterControl Seventeen Every Time"; the paper content itself carries the title "From Question to Evidence: A Small Analytical Algebra for Governed Data Analysis," by Matt Bray, Logan Green, Hemant Jomraj, Shardul Pande, and Viktoria Rojkova (MasterControl AI Lab, September 2026; arXiv:2609.03209v1 [cs.AI], 02 Sep 2026).

What This Paper Is About

Producing a number like 17 is easy; preserving what 17 means — the population, time interval, measure, definitions, and supporting records — is the hard part. Two systems can print the same number while measuring different populations, so the authors treat an answer as a structured contract R = (m, v, E), where m is the accepted governed meaning, v is the value or table, and E is role-labeled supporting evidence. The paper asks whether letting a language model interpret the user's wording, while a deterministic policy selects a pre-written analytical program, sacrifices any analytical power — and then tests empirically whether giving the model control of the method at request time actually helps.

Key Contributions

1. A constructive expressiveness result for a relational core. Starting from finite relations and first-order satisfaction, the authors give a constructive translation (Theorem 1) showing that for every finite-domain first-order specification there is a finite program built from RESTRICT, SHAPE, and RELATE with an equal result — and conversely, that every program in this core has an equivalent finite-domain first-order specification.

2. A scoped completeness theorem for a stated analytical class. By adding a finite registry of analytical kernels — explicit aggregation, comparison, windows, ranking, and versioned similarity — the authors prove (Theorem 2) that every finite acyclic specification in the declared class has a finite well-typed program over eight operation families returning the same value, evidence, or declared domain error. The claim is deliberately scoped: it does not cover every useful statistic, causal model, or business question.

3. Policy and replay guarantees. Theorem 3 shows that if every template in a supported policy family belongs to the analytical class, pre-written programs plus deterministic policy can execute every correctly bound member exactly, with no model-selected composition. Theorem 4 shows evidence-preserving replay: fixed governed state Ω = (D, A, M, Π, C, K, N, O) implies the same analytical contract and invariant execution certificate on repeated successful executions. The paper also gives margin conditions for approximate kernels.

4. A controlled 440-episode comparison. Three independent 8B open-source models (Qwen3-8B, Ministral-3-8B, Granite-4.1-8B) construct SQL procedures at runtime, while a policy-executed analyzer uses Qwen3-8B for semantic interpretation only. Across 330 runtime-planning episodes, 0 produced an exact contract over all five snapshots; the policy-executed analyzer produced 110 exact contracts in 110 episodes.

Main Findings

• Restriction did not cost expressiveness inside the declared class. Theorems 1 and 2 establish that a compact relational core extended with typed analytical kernels can exactly implement a broad stated class of finite analyses. The boundary conditions: a non-monotone operation (difference) is needed for absence questions such as "sites with no deviation," and counting requires an explicit analytical extension because pure relational operations cannot generate a variable-size count as a new scalar.

• Pre-written programs plus deterministic policy execute supported requests exactly. Under deterministic policy resolution and deterministic kernels, every supported and correctly bound request is executed without the model emitting SQL, primitive names, primitive order, or a program identifier. If no policy rule covers a request, the system exposes a coverage gap rather than inventing a method.

• Replay is guaranteed from the accepted meaning onward. With fixed canonical request and governed state, repeated successful executions return the same contract and invariant execution certificate; the proof is induction over the program dependency graph. Operational telemetry such as latency and run identifiers is explicitly excluded from analytical equality.

• No runtime-planning episode produced an exact contract. Across 330 agent episodes, 55 returned a final procedure and 0 of 330 produced an exact answer-and-evidence contract over all five snapshots. Of the 330 episodes, 220 ended in rejection, 55 exhausted the tool budget, and 55 returned a final program.

• The policy-executed analyzer matched the contract 110 of 110 times.

• Supplying the meaning did not rescue runtime planning. In the fixed-request panel, where every system received the same already accepted meaning, the runtime agents produced 0 exact contracts in 165 episodes, while policy execution returned 55/55 exactly with no model inference.

• The same model on both sides isolates method ownership. In the natural-language panel, the policy analyzer used about 4.8 times fewer tokens and 29 times less online time than the Qwen3-8B runtime agent, while moving from 0/55 to 55/55 exact contracts. In the fixed-request panel, policy execution used zero model tokens and 0.0028 seconds on average, against 8,019 tokens and 10.771 seconds for Qwen runtime planning.

• A right number is not a right analysis. Five episodes returned the correct numerical answer on the visible development snapshot; all five were Ministral repetitions of the simplest count task. The generated query counted the right 17 records, but labeled evidence with the wrong role and used a grouped aggregate that returned no row for an empty population.

• Repeatability is not correctness. At temperature zero, several returned procedures were repeatable across runs but wrong. Low retry counts should not be read as efficient success, since many episodes rejected or exhausted their budget before a correct procedure was selected.

• Approximate kernels need margins. If exact scores sᵢ are approximated by ŝᵢ with maxᵢ |ŝᵢ − sᵢ| ≤ η, threshold membership at τ is unchanged whenever every score lies more than η from the threshold, and top-k membership is unchanged if the exact gap between positions k and k+1 exceeds 2η. Without such margins, small numerical changes can alter the evidence population.

• Component determinism is not analytical determinism. If two valid deterministic procedures implement different functions and a model may choose either, deterministic tools do not guarantee the same function is selected, and tool availability alone does not imply that more planning steps converge to the correct one.

Methodology in Plain English

The paper works in two halves.

Theory half. The authors start with a simple observation: a factual analyzer should not invent a new measuring method each time it answers a question. They define what an answer is — a triple of accepted meaning, value, and role-labeled evidence, plus an execution certificate recording the data snapshot, policy version, program version, kernel versions, and serialization rules. They then show that a small set of typed operations is enough. RESTRICT decides which records qualify; SHAPE decides which fields and deterministic derivations come back; RELATE combines governed relations; AGGREGATE fixes the grain; COMPARE sets the baseline or denominator; WINDOW sets the time or ordered frame; RANK fixes the total order and tie rule; SIMILAR applies a versioned semantic or numeric neighborhood. To prove the language is broad enough, they borrow the classic database-theory analogy — relational algebra is restrictive yet complete for an important class of queries — and give a constructive translation from first-order formulas into programs over these operations, then extend it with declared analytical kernels. They also prove that with fixed governed state, the same request produces the same result and evidence.

Empirical half. They build one synthetic governed quality/manufacturing dataset and eleven analytical tasks: counts, grouping including zero-count groups, absence queries, ranking with ties, rates, period change, contribution arithmetic, join multiplicity, exact means, score thresholds, and region grouping. Each task has an independent reference specification and evidence contract. They compare four setups on the same hardware (an NVIDIA RTX PRO 6000 Blackwell Server Edition, BF16 inference, temperature 0, one request at a time, tool budget of eight): three runtime-planning agents that inspect the data, write SQL, execute it, possibly revise, and pick the final procedure; and a policy-executed analyzer in which the model only emits governed intent and deterministic policy runs the pre-written program. The benchmark runs in two panels — natural-language end to end, and a fixed canonical request where the meaning is handed to everyone. Each selected runtime procedure was frozen after development-snapshot planning and then evaluated without model assistance on four held-out database variants covering multiplicity/absence, time boundaries and scope, empty populations and zero denominators, and ties/scores/nulls. The primary metric is exact contract accuracy: the procedure had to match the reference value, governed meaning, qualifying records, and evidence roles on all five snapshots.

Why This Matters

Impact on research. The paper supplies a formal bridge between "what an analysis means" (finite-domain first-order specifications) and "how it is executed" (typed analytical programs), plus a replay guarantee that most agent papers do not attempt. It also reframes the agent debate: rather than asking whether tool-using agents can analyze data in general, it isolates one variable — whether the model owns the analytical method — and holds the model constant across both conditions in one comparison.

Real-world applications:

• Regulated quality and manufacturing analytics. The synthetic dataset and tasks are drawn from deviation counting (for example, approved temperature-excursion deviations), site and batch linkage, and CAPA relationships, where the population and definitions behind a number must be auditable.
• Audit and regulated reporting, where an answer must come with the qualifying records and evidence roles that produced it, and where the same question must be replayable against a pinned snapshot and policy version.
• Operational metric reporting with zero-count and edge-case requirements, such as period-over-period rate comparisons and region groupings where empty populations or zero denominators must return a row rather than silently disappearing.
• Systems where method discovery is not the task. The authors note that runtime planning is appropriate for open-ended research, method discovery, code generation, and operational planning — and that the case against it here is narrower: a supported factual analysis whose approved method is already known.

Industry relevance. The design principle — let the model help determine what the user means, let policy determine how an approved analysis is performed, return the evidence with the answer — targets auditability, reproducibility, and cost. In the reported configuration, policy execution used zero model tokens in the fixed-request panel and 1,433 tokens in the natural-language panel versus 6,850–17,912 tokens for runtime agents. The stated cost is maintenance: approved programs must be authored, reviewed, tested, versioned, and kept up to date, and coverage grows deliberately rather than invisibly.

Future Directions

• Measure coverage on a representative production workload. The authors state the right product question is not "can the algebra express everything?" but "what fraction of real user questions map to reviewed programs, and what does it cost to extend that coverage?" — which a representative production workload is needed to answer.
• Test whether runtime planning can be made to work under other designs. The paper explicitly notes that a larger model, native function calling, different prompts, more budget, program synthesis with formal verification, or a sound search-and-acceptance procedure could perform better; each of these is an open empirical question left untested.
• Specify a search process for agents. The authors argue that stronger claims about agent success require a specified search process, a progress condition, and a sound acceptance rule — none of which the current experiment provides.
• Extend and stress-test the kernel registry and approximation margins. The completeness result is scoped to the declared registry and the stated semantics, and the approximate-kernel conditions (η for thresholds, 2η for top-k gaps) define where exactness can break; expanding the registry and characterizing margin behavior in practice remain open.

Target Audience

This

Authors’ abstract

We study a governed approach to enterprise analytics: a language model interprets the question, while deterministic policy selects and runs a pre-approved analytical program that returns both results and evidence. We show that this restriction can remain expressive within a defined analytical class, using relational operations plus aggregation, comparison, windows, ranking, and similarity. Fixed meaning, policy, data, and execution rules also make results replayable. Across 440 runs, three 8B models generated SQL and selected tools at runtime, while Qwen3-8B interpreted intent only and policy executed the approved program. None of 330 runtime-planning episodes matched the full answer-and-evidence contract across all test datasets; the policy-executed analyzer matched 110 of 110. This is a configuration-specific result, not evidence that runtime agents cannot succeed under other designs.