POLARIS: Typed Planning and Governed Execution for Agentic AI in Back-Office Automation

AI summary

Overview

Research area: Agentic AI and LLM-based orchestration for enterprise back-office automation, with emphasis on governance, typed planning, policy compliance, auditability, and evaluation benchmarks.

Technical level: Advanced. The paper formalizes typed directed acyclic graphs (DAGs), feasibility constraints, a scoring rubric, a bounded repair algorithm, and MAD-based anomaly statistics, though the conceptual architecture is explained in accessible terms.

Scope in one sentence: POLARIS is a governed orchestration framework that re-casts back-office automation as typed plan synthesis plus validator- and policy-gated execution over LLM agents, evaluated on a 40-invoice synthetic stress suite and the SROIE extraction benchmark.

What This Paper Is About

Generic multi-agent LLM stacks are flexible but rarely auditable, policy-aligned, or operationally predictable: untyped tool calls, best-of-N prompting, and open-ended retries produce brittle pipelines with unclear provenance and no service-level-agreement guarantees. The paper argues that enterprise guarantees require re-casting orchestration as typed, governed planning and execution, where candidate plans are type-checked and structurally diverse, plan selection is rubric-based and policy-aware, and execution is guarded by validators and compiled policy checks that gate side effects before they occur. POLARIS ("Policy-Aware LLM Agentic Reasoning for Integrated Systems") is the framework built to deliver decision-grade artifacts and complete execution traces while reducing human intervention.

Key Contributions

1. Typed DAG planning with diversity as a first-class generation constraint. The CoAPlanner generates a set of K feasible, pairwise distinct type-checked plans, enforcing structural diversity during generation (differing agent sets, edge structures, or stage ordering) rather than sampling duplicates and pruning them afterward, with hard-coded policy invariants (e.g., policy retrieval before irreversible actions, approval sinks trailing validation and risk control).

2. A rubric-guided, policy-aware reasoning selector. A lightweight reasoning LLM scores pre-validated candidate plans on four rubric terms — compliance, sequencing, parsimony, and planner prior — and emits a structured JSON decision containing a chosen_index and a reason, enabling fail-fast rejection and auditable selection.

3. Governed execution: validator-gated repair plus compiled policy guardrails. A bounded parser–validator repair loop (at most L_max iterations, typically about 3) targets only failing fields before external side effects, while PolicyRetrieval and RiskControl compile vendor policy clauses into executable checks (unknown vendor, threshold breach, currency mismatch, plus duplicate detection, black/whitelists, graded risk scoring, and provenance requirements) that halt, escalate, or route actions. A dependency-aware scheduler runs the selected plan with completion-based semantics, allowing safe parallelism.

4. An initial benchmark for governed Agentic AI. A controlled synthetic suite that measures policy compliance, audit-trace completeness, and risk-routing accuracy beyond conventional precision and recall, alongside SROIE evaluation; the paper positions typed DAG traces, governed selection logs, and validator repair traces as reusable evaluation primitives.

Main Findings

• Synthetic field extraction is strong overall. Across the 40-invoice synthetic suite (Table 1, reported at field level with 160 total), POLARIS achieves precision 0.9453, recall 0.9680, and F1 0.9565, with 121 TP, 7 FP, 4 FN, and 28 TN.

• Per-scenario extraction varies with input quality. Violation–Unknown Vendor (VU): precision 0.9722, recall 1.0000, F1 0.9859. Violation–Layout Drift and Noise (VL): precision 0.8214, recall 1.0000, F1 0.9020. Compliant–Clean Set (CC): precision 0.9667, recall 0.9355, F1 0.9508. Compliant–Month-End Batch (CM): precision 1.0000, recall 0.9444, F1 0.9714. The paper reads the VL result as robustness favoring coverage under noise, with a precision dip.

• Policy violation detection is moderate overall and weak under noise. Across scenarios with positive cases (Table 3), totals are 9 TPV, 2 FPV, 2 FNV, 15 TNV, giving precision 0.8182, recall 0.8182, and F1 0.8182. VU reaches precision 0.8889, recall 1.0000, F1 0.9412; VL reaches precision 0.5000, recall 0.3333, F1 0.4000. For CC and CM no policy violations are present by design, so precision/recall/F1 are undefined.

• Anomaly detection shows perfect precision with slightly reduced recall. Aggregate results (Table 4) are 9 TP, 0 FP, 1 FN, 16 TN: precision 1.0000, recall 0.9000, F1 0.9474. VL records precision 1.0000, recall 0.8333, F1 0.9091 (one FN attributed to sparse vendor baselines); CM records precision 1.0000, recall 1.0000, F1 1.0000; VU and CC have no positive cases so their metrics are undefined.

• SROIE results are reported as micro-F1 0.81 in the abstract. Table 5 gives overall precision 0.8189, recall 0.8045, F1 0.8116, and field accuracy 0.8045. Per-field accuracy: Company 0.8500, Address 0.8500, Date 0.7600, Total 0.7576.

• The abstract reports 0.95–1.00 precision for anomaly routing on the controlled synthetic suite with preserved audit trails, alongside the micro-F1 of 0.81 on SROIE.

• Experimental configuration. Runs use K = 5 diversified plans, the constrained ReasoningAgent selector, and the validator-gated repair loop with L_max = 3. Anomaly detection uses vendor-wise MAD baselines with k_mad = 3.5, plus a strict future-date rule. Month-end jobs enforce Scheduler → ReportGenerator. CoAPlanner is instantiated with GPT-4o and ReasoningAgent with a GPT-5 reasoning model configured for strict JSON output.

• Baseline comparison is qualitative, not numeric. The paper states that relative to point OCR-plus-rules baselines it observes higher recall under noise and precise policy/alarm signals, but no numerical baseline results, latency measurements, or cost measurements are reported.

Methodology in Plain English

POLARIS runs a plan–select–act loop over a library of typed, GPT-backed agents (Normalizer, Planning & Selection, Data Extractors, Data Processors, Reconciliation).

1. Normalization. An InputNormalizerAgent (an Autogen ConversableAgent prompted with a strict JSON schema) converts heterogeneous inputs — schedules, prompts, PDFs, emails, API events — into a canonical task record with fields such as task_type, input_format, file_name, file_type, timestamp, origin, instruction, and meta. The last JSON object in the model response is regex-parsed; on failure the agent returns a structured error payload rather than acting.

2. Candidate plan synthesis. The CoAPlanner interprets task intent, selects relevant agents, and sequences them into a small set of K typed DAGs. Each plan must be feasible (type-compatible edges; intrinsic orderings like parse → validate → risk/approval) and minimal (no superfluous stages). Diversity is enforced at generation, so candidates cannot share the same core chain and dependency structure. DSPy compiles few-shot exemplars (positive/negative scenarios, anomaly exemplars, approval thresholds) into a reusable planning program that biases sampling toward policy-consistent chain patterns.

3. Rubric-based selection. The ReasoningAgent scores candidate plans as a weighted sum of compliance, sequencing, parsimony, and planner prior, then returns a JSON object with the chosen plan index and a justification. Plans failing hard type or policy constraints are discarded before scoring, and the selector never alters types or adds actions.

4. Guarded execution. A dependency-aware scheduler converts ordered stages into a DAG: DocumentParser runs first, independent middle-stage checks (PolicyRetrieval, RiskControl, AnomalyDetector, RecordMatcher) run concurrently, and aggregate sinks (Approval, ReportGenerator) wait for all upstream checks. Edges can be injected at runtime, for example adding an ExtraVerification prerequisite when AnomalyDetector emits a high score.

5. Validator-gated repair. If a required field is missing, a field confidence falls below a threshold, or validation fails, the loop issues localized repair actions guided by validator feedback (region-of-interest hints, OCR/layout cues, schema prompts, string normalization) and merges fixes, re-requesting only implicated fields. The loop stops on validation pass or exhausts the budget, in which case control escalates to human review or a safe lower-coverage path with no approvals and a full audit trace.

6. Policy guardrails and anomaly handling. PolicyRetrieval queries a policy store keyed by vendor, sector, and currency; RiskControl compiles those clauses into executable checks. A violation is defined as unknown vendor, threshold breach without an attached approval artifact, or currency mismatch. Anomalies use a vendor-wise robust score z_MAD = |x − median_v| / (1.4826 · MAD_v), flagged above k_mad = 3.5, with fallback to cohort (sector/currency) or global statistics when vendor data are sparse, plus date-sanity rules that flag any issue, due, or payment date strictly after "today." Flagged items proceed through a playbook: enrich → classify severity → route → act → close.

7. Outputs and feedback. Each input produces a validated JSON record with confidences and type signatures, a decision object (approve/hold/reject) with rationales tied to policy clauses and anomaly evidence, alerts and escalation targets, and a complete execution trace. Logs update planning exemplars and anomaly/policy baselines — updating exemplars rather than opaque prompts, to preserve reviewability.

Why This Matters

Impact on research. The paper treats governance, type safety, and auditability as first-class architectural properties rather than afterthoughts, and proposes evaluation constructs — typed DAG lineage, governed selection logs, validator repair traces — that others could reuse to compare agent frameworks on compliance accuracy, routing precision, and repair effectiveness. It positions the controlled synthetic suite as an initial benchmark for governed Agentic AI alongside SROIE.

Real-world applications:

• Accounts payable processing: validating invoice fields and flagging unknown vendors, threshold breaches without approval, and currency mismatches before payment actions execute.
• Month-end financial batches: enforcing Scheduler → ReportGenerator ordering so scheduling and reporting happen in a controlled sequence.
• Anomaly triage and risk routing: using vendor-wise baselines plus date-sanity checks to route items to API holds, ticketing, or human review under a documented playbook.
• Audit and compliance reporting: producing validated records, decision objects with policy-linked rationales, and replayable execution traces for independent verification.
• Contract checks and broader regulated workflows: the paper notes the design generalizes to supply chain, HR, and compliance domains, though evaluations center on finance.

Industry relevance. The framework targets the gap between research prototypes and enterprise deployment in regulated settings where errors carry financial risk, and where auditable, policy-aligned, operationally predictable behavior is a prerequisite. The work was conducted during the authors' tenure at Accenture's Center for Advanced AI, with academic co-authors from the University of California, Irvine and the University of California, San Diego, underscoring the applied enterprise framing.

Future Directions

• Cross-domain validation. The paper explicitly names broader cross-domain validation — supply chain, HR, and compliance — as future work, since current evaluations center on finance workflows.
• Integration with standard agent benchmarks. Connecting POLARIS to standard AgentBench tasks is listed as an open area.
• Standardized governance metrics and reproducibility protocols. Future benchmarks may extend the synthetic suite's dimensions with standardized governance metrics and multi-agent reproducibility protocols to form a baseline schema for comparing enterprise-grade Agentic AI systems.
• Closing the noise-robustness gap. The VL (layout drift and noise) scenario shows the weakest policy results (precision 0.5000, recall 0.3333, F1 0.4000) and the only anomaly false negative, attributed to upstream noise and sparse vendor baselines, suggesting extraction robustness and baseline fallback as open problems.
• Tuning the fixed components. The rubric weights (w1, w2, w3) are fixed in this work, and the planner's diversity equivalence relation is specified qualitatively, leaving principled tuning or learning of these as unresolved.

Target Audience

Enterprise AI engineers and architects building agentic automation in regulated settings; agentic-AI and multi-agent systems researchers interested in typed planning, governed execution, and evaluation methodology; finance-operations and accounts-payable practitioners evaluating auditable automation; and compliance, audit, and risk professionals who need traceable decision lineages for automated financial workflows.

Authors’ abstract

Enterprise back office workflows require agentic systems that are auditable, policy-aligned, and operationally predictable, capabilities that generic multi-agent setups often fail to deliver. We present POLARIS (Policy-Aware LLM Agentic Reasoning for Integrated Systems), a governed orchestration framework that treats automation as typed plan synthesis and validated execution over LLM agents. A planner proposes structurally diverse, type checked directed acyclic graphs (DAGs), a rubric guided reasoning module selects a single compliant plan, and execution is guarded by validator gated checks, a bounded repair loop, and compiled policy guardrails that block or route side effects before they occur. Applied to document centric finance tasks, POLARIS produces decision grade artifacts and full execution traces while reducing human intervention. Empirically, POLARIS achieves a micro F1 of 0.81 on the SROIE dataset and, on a controlled synthetic suite, achieves 0.95 to 1.00 precision for anomaly routing with preserved audit trails. These evaluations constitute an initial benchmark for governed Agentic AI. POLARIS provides a methodological and benchmark reference for policy-aligned Agentic AI. Keywords Agentic AI, Enterprise Automation, Back-Office Tasks, Benchmarks, Governance, Typed Planning, Evaluation