ARCHER: Agentic Rule and Compliance Harness for Executable Regulations

AI summary

Overview

Research area: Multi-agent systems applied to automated building-code compliance checking (agentic program synthesis over Building Information Modeling / IFC data).

Technical level: Advanced. The paper assumes familiarity with multi-agent LLM orchestration, test-driven program synthesis, and BIM/IFC concepts, though the core argument is stated plainly.

Scope: The paper evaluates six increasingly sophisticated agentic harnesses across four backbone models, spanning frontier-API to fully on-premise deployment tiers, on a new benchmark of ten real-world regulatory requirements.

What This Paper Is About

Verifying that a building design complies with building codes means checking thousands of rules against large BIM models, a process that is manual, expensive, and hard to scale. Existing Automated Compliance Checkers (ACCs) are typically hard-coded for narrow rule sets, difficult to generalize, and often proprietary, so users cannot inspect whether a checker actually captures their regulatory intent. ARCHER aims to fix this by using a deterministically orchestrated multi-agent harness that synthesizes transparent, auditable Python compliance checkers directly from experts' written rule interpretations and labelled example models.

Key Contributions

1. First application of agentic program synthesis to automated building compliance checking (to the authors' knowledge).
2. ARCHER itself, a test-driven multi-agent harness that orchestrates planner, generator, and evaluator agents to produce inspectable compliance checkers as Python code.
3. A comprehensive cost–accuracy analysis across four backbone models and four deployment tiers, from commercial third-party APIs to a privacy-preserving on-premise setup served on a single NVIDIA DGX Spark.
4. A novel benchmark dataset of ten real-world regulatory requirements (R1–R10) paired with expert-labelled BIM models, containing 245 labelled elements in the training pairs and 270 labelled elements in the test pairs. Code and dataset are stated to be released upon publication.

Main Findings

• ARCHER is uniformly best. Harness 5 (deterministic multi-agent orchestration) achieves the highest union accuracy for every one of the four backbones, raising the overall mean from 0.4681 (Harness 0, blind baseline) to 0.8498 — a gain of 38.2 accuracy points, or an 82% relative improvement.
• Deterministic orchestration beats LLM orchestration. Harness 5 improves over the LLM-orchestrated Harness 4 for all four models, by +0.008 to +0.245, with the largest gains on weaker backbones.
• Unstructured feedback hurts weaker models. Full test-driven feedback in a single-agent prompt (Harness 2) helps GPT-5.5 (rising to 0.9468) but degrades the open-weights models relative to Harness 1: DeepSeek-v4-flash drops from 0.6625 to 0.5725, and GPT-OSS-120B-Q4KM collapses from 0.4754 to 0.1719. The structured multi-agent harnesses recover these losses, with DeepSeek-v4-flash jumping from 0.6486 (Harness 3) to 0.9459 (Harness 4) and 0.9542 (Harness 5).
• Token consumption scales inversely with model capability. From Harness 0 to Harness 5, input tokens multiply by 8.0 times for GPT-5.5, 14.1 times for GPT-5.4 Mini, 20.2 times for DeepSeek-v4-flash, and 223.7 times for GPT-OSS-120B-Q4KM. Under Harness 5, GPT-OSS-120B-Q4KM consumes 21.8 times the input tokens of GPT-5.5 while reaching barely half its accuracy.
• GPT-5.5 is the only backbone whose input consumption drops from Harness 4 to Harness 5 (a mean of 1,053,584 to 762,804 tokens per scenario), attributed to the removal of orchestration chatter and earlier early-stopping.
• Output tokens remain a small fraction of input volume, at 2–5% at Harness 5 across models, so cost at high harness levels is dominated by repeated context re-reading rather than code generation.
• Harness sophistication is a better investment than model price. The Pareto frontier ascends mainly by increasing harness level within a model family. Moving from Harness 0 to Harness 5 buys 3.8 accuracy points per dollar on the full ten-scenario run for DeepSeek-v4-flash and 6.0 for GPT-OSS-120B-Q4KM, versus 0.5 points per dollar for GPT-5.5.
• A cheap per-token price does not guarantee a cheap task. GPT-5.4 Mini is priced 6.7 times below GPT-5.5 per input token, yet under ARCHER it consumes 5.4 times the input and 7.8 times the output tokens; its mean per-scenario cost ($4.00) nearly matches GPT-5.5's ($4.59) while its accuracy is 6.2 points lower. Under ARCHER it falls off the Pareto frontier entirely, with self-hosted DeepSeek-v4-flash delivering 4.1 more accuracy points at 3.3 times lower cost.
• Data-sovereign deployments are no longer a large accuracy sacrifice. The frontier provider operating point reaches 0.9759 at $4.59 per scenario. A self-hosted DeepSeek-v4-flash reaches 0.9542 — 97.8% of frontier accuracy at roughly a quarter of the cost — whereas the same model languishes at 0.50–0.66 under the single-agent harnesses.
• Only the strictest tier still pays a real penalty. The 4-bit GPT-OSS-120B served on a single DGX Spark peaks at 0.5557, though this is more than double its blind-baseline accuracy (0.2148 to 0.5557) and gives the highest accuracy-per-dollar of any configuration.
• Even the most expensive configuration is cheap in absolute terms. GPT-5.5 under Harness 4 costs $6.19 per requirement on average, negligible against the engineering hours it replaces.

Methodology in Plain English

ARCHER takes two artefacts that domain experts can write and check: a Rule Interpretation (RI) document, which restates a building-code provision as an unambiguous logic flow in plain text, and labelled BIM models, which record the intended verdict for each relevant element. The harness then treats the training labels as a test oracle and iteratively generates, executes, and refines a Python checker until its output on the training model matches the expert verdicts — a test-driven development loop.

Three agents do the work inside a sandboxed environment: a planner writes a plan, a generator writes and edits the checker code, and an evaluator runs it and reports accuracy and what to fix. A fixed control loop (not an LLM orchestrator) sequences them, allowing up to 30 iterations and up to 2 re-plans if the evaluator traces a failure to the plan rather than to a coding bug. The loop stops early when training agreement reaches 1.0. The held-out test model is never mounted during the loop; the final checker is scored on it exactly once, using union accuracy — the number of correctly labelled elements divided by the size of the union of elements the checker labelled and elements the expert labelled. That denominator penalizes a checker for silently omitting an element, not just for mislabelling one.

To isolate what makes agentic synthesis work, the team defined six harnesses of increasing sophistication — from a blind single agent with no training-data access (Harness 0), through execution-grounded generation (1), full test-driven refinement (2), a single agent with an explicit planning step (3), and LLM-orchestrated multi-agent pipelines (4), up to ARCHER's deterministic orchestration (5) — and ran all six against GPT-5.5, GPT-5.4 Mini, DeepSeek-v4-flash, and GPT-OSS-120B-Q4KM. Because no existing checker could serve as an external baseline (commercial tools require hand-encoding rules in proprietary languages and do not emit element-level verdicts), the harness ladder itself acts as the controlled comparison. Costs were computed by pricing every run at published OpenRouter per-token rates as a common market proxy.

Why This Matters

The paper shifts automated compliance checking from hand-coded, proprietary rule engines toward synthesized code that is transparent, editable, and adaptable across jurisdictions. It also provides the first cost–accuracy evidence that agentic scaffolding — not just bigger models — is what unlocks reliable code generation, and that data-sovereign deployments are viable at near-frontier accuracy.

Real-world applications:

• Regulatory submission review in dense jurisdictions. Singapore-based work referencing requirements across thousands of rules (Building and Construction Authority; CORENET X) is the motivating setting, where checkers must be regenerated as rules change.
• Government and regulated-industry compliance under data-sharing restrictions. An agency that cannot upload proprietary designs to a third-party API can self-host DeepSeek-v4-flash or run fully on-premise on a single DGX Spark and still get usable accuracy.
• Design-firm and consultant review workflows. Checkers run against IFC models before submission, so errors in the model (ALERT) and compliance failures (FAIL) are separated, with ambiguous cases escalated to MANUAL_CHECK rather than guessed.
• Auditability and dispute resolution. Because the checker is inspectable Python rather than an opaque engine, a domain expert can see which geometric assumption failed when an unusual configuration appears.

Industry relevance centers on cost and control: the most expensive configuration studied costs $6.19 per requirement on average, while the frontier operating point costs $4.59 per scenario at 0.9759 accuracy. That makes agentic codification of regulations economically favorable across all four deployment tiers examined, and it gives procurement teams a task-level cost metric (price multiplied by the tokens a model actually needs to converge) rather than a headline per-token price.

Future Directions

• Modelling dependencies and contradictions across rules. ARCHER currently treats each rule in isolation, without capturing how rules interact; semantic knowledge graphs are proposed as a route.
• Better error localization and auto-correction. Sub-1.0 accuracy shortfalls cannot always be attributed to the requirement specification, the test cases, or a coding bug; modular intermediate representations and auto-corrective post-processing are suggested remedies.
• Test-time methods for cases outside deterministic geometry. Spatial reasoning networks or vision-language models are proposed for configurations that fixed geometric primitives cannot resolve.
• Studying the human side of the loop. The upfront expert work of authoring RI documents and labelling models is not accounted for in the cost analysis, and how reliably experts author and audit these artefacts in practice remains to be studied. The paper frames ARCHER as a human–agent collaborative system, not a fully autonomous one.

Target Audience

Researchers in multi-agent LLM systems and program synthesis who want a controlled taxonomy of harness designs; AEC (architecture, engineering, and construction) professionals and code officials exploring automated compliance checking; government and enterprise architects evaluating on-premise versus API deployment of LLMs over sensitive design data; and ML practitioners interested in the cost–accuracy trade-offs of deterministic versus LLM-driven agent orchestration.

Authors’ abstract

Verifying building compliance requires validating thousands of rules against large Building Information Modeling (BIM) designs, which is laborious, capital-intensive, and unscalable. Existing Automated Compliance Checkers (ACCs) are often difficult to generalize across different scenarios, as they are typically developed for highly specific rule sets and use cases. In addition, many ACCs are proprietary, meaning the underlying verification code is not released to end users, so users cannot verify whether their regulatory intent can be accurately captured. We introduce ARCHER (Agentic Rule and Compliance Harness for Executable Regulations), a test-driven, deterministically orchestrated multi-agent program-synthesis harness that generates auditable verification code from regulatory Codes of Practice, enabling transparent, adaptable, and scalable compliance checking. To characterize what makes agentic synthesis work, we evaluate a taxonomy of six harnesses of increasing agentic sophistication across four backbone models, spanning realistic data-governance tiers (from frontier third-party APIs to a fully on-premise open-weights model) on a novel dataset derived from real-world compliance scenarios. ARCHER's deterministic multi-agent orchestration achieves the highest accuracy for every backbone, improving mean union accuracy by 82% over a naive single-pass prompting baseline. Our cost-accuracy analysis further shows that using the ARCHER harness, a self-hosted open-weights model can reach 97.8% of frontier-API accuracy at a quarter of the cost, making data-sovereign compliance checking practical.