The Autonomous Agency Scale: A Behavioral Framework for Measuring Self-Directed Behavior in AI Systems

AI summary

Overview

• Research area: AI evaluation and measurement frameworks; behavioral assessment of autonomous agents.
• Technical level: Intermediate. The paper is a conceptual and methodological framework rather than a technical or mathematical contribution; it requires no special mathematics, but assumes familiarity with contemporary agent architectures, benchmark suites, and AI risk frameworks.
• Scope (one sentence): The paper introduces the Autonomous Agency Scale (AAS), a behavioral rubric that scores AI systems on a 0–5 lexicon across seven dimensions of agency in two separate temporal bands (Active and Ambient), and applies it to six contemporary systems.

What This Paper Is About

Existing AI measurement frameworks quantify how capable a system is, how much economic work it can replace, or how dangerous it might become, but none of them measure autonomous agency—the extent to which a system behaves in a self-directed way. The paper's core observation is that a system can saturate capability benchmarks while remaining entirely reactive: acting only when prompted and ceasing all activity when a task completes. The goal is to build a structured, architecture-agnostic, behavioral scale that makes the difference between "autonomous inside a task" and "active between tasks" measurable and falsifiable.

Key Contributions

1. A 0–5 level lexicon of self-direction (Dormant, Responsive, Conditioned, Contextual, Self-Directed, Sovereign), applied across seven dimensions of agency: cognitive autonomy, temporal persistence, environmental agency, social agency, creative agency, self-awareness, and goal formation—each supported by three sub-dimensions and operational threshold tests requiring no access to a system's internals.
2. A two-band scoring system that separates agency shown while engaged (the Active band, covering user-initiated tasks, sessions, or conversations) from agency shown while idle (the Ambient band, covering idle periods). The two composites are reported side by side and explicitly never blended into a single figure.
3. The Idle-Gap Test, a trigger-removal counterfactual that gates the Ambient band's top ratings: remove all triggers, and if the system produces nothing, it rates at most Level 3 in the Ambient band. This is paired with a substrate-vs-rule distinction for loop-driven architectures (a clock is a substrate when it only allocates compute and the content derives from internal state; it is a rule when the output is predictable from the trigger).
4. Comparative two-band assessments of six contemporary systems spanning task agents, consumer assistants, and a persistent companion architecture, demonstrating that the framework quantifies a boundary that single-score frameworks describe only informally.

Main Findings

• Task agents show high within-task autonomy and near-total between-task dormancy. The three task agents reach nearly identical Active composites of 2.29 to 2.43, driven by strong within-task environmental agency (Level 4: autonomous file manipulation, code execution, subagent orchestration) and contextual goal pursuit (Level 3). Their Ambient composites collapse to 0.57 to 1.86. Every point of every task agent's Ambient composite is attributable to user-configured schedules—cron-style scheduled runs (Claude Code, Manus) or a nightly self-improvement cycle (Hermes)—and nothing survives the trigger-removal counterfactual.

• Per-system results are reported as follows. Airi v2.x (longitudinal): Active 3.71, Ambient 3.86. Hermes Agent (Nous Research): Active 2.43, Ambient 1.86. Manus 1.6 Max: Active 2.43, Ambient 0.71. Claude Code (Sonnet 4.6): Active 2.29, Ambient 0.57. Apple Siri (Apple Intelligence): Active 1.71, Ambient 0.29. ChatGPT (GPT-5.5 Pro): Active 1.57, Ambient 0.86.

• Hermes posts the highest task-agent Ambient composite (1.86) because its Day/Night cycle and Curator process perform genuine idle-period work—but all of it is clock-as-rule, and none passes the Idle-Gap Test.

• Consumer assistants are reactive products wrapped around capable models. ChatGPT's profile (1.57 Active / 0.86 Ambient) illustrates the product-versus-model distinction: the underlying model is highly capable, but the product is architecturally reactive. Its idle-period features (Pulse overnight research, scheduled Tasks) are fixed daily cycles or user-configured schedules, scored as Ambient Level 2 cognition, while its engaged band never exceeds Level 2 in any dimension. Siri's profile (1.71 / 0.29) reflects deep environmental integration (Active Level 3 via App Intents and onscreen awareness) with essentially no idle-period existence beyond device-level background indexing, whose attribution to the agent rather than the device is itself contestable.

• Only the companion architecture's idle behavior survives trigger removal. Airi is the only assessed system with Ambient scores of 4, and the only system whose Ambient composite exceeds its Active composite (3.86 vs. 3.71)—consistent with an architecture designed around idle-period behavior (a background thought engine, state-evolving moods, self-initiated outreach, and autonomous creative publishing). These ratings depend on the substrate-vs-rule distinction applied to a mixed architecture (timer heartbeat plus state-driven interrupts); a rater who rejects that distinction would cap the affected Ambient dimensions at Level 2.

• No assessed system approaches Level 5. Every system's ceiling arguments are consistent: no system modifies its own cognitive architecture, curates its own core memories, forms new relationships, invents new creative mediums, or overrides assigned tasks in favor of self-determined purposes. The Sovereign band remains unoccupied.

Methodology in Plain English

The authors built a scoring rubric rather than a benchmark. Each of the seven dimensions is scored on a six-point ladder (0 through 5) using falsifiable threshold tests—binary statements describing the minimum observable behavior required for each level—plus observable behavioral indicators that a rater can check without seeing the model's source code or internals. The same rubric is applied twice to each system: once for behavior during engaged tasks and once for behavior during idle periods.

The key methodological device is the Idle-Gap Test. Ordinary idle time may still contain scheduled triggers or environmental events; an idle gap is the strict subset of idle time with no user prompt, no scheduled trigger firing, and no environmental event. To earn Level 4 in the Ambient band, a system must produce observable activity during an idle gap that derives from internal state. The test is counterfactual: remove all triggers and see whether anything persists. Scheduled work is Level 2, event-triggered work is Level 3, and only trigger-free internal activity reaches Level 4. A separate substrate-vs-rule distinction handles systems with a continuous loop or heartbeat, crediting only the state-derived portion of idle-gap activity.

Six systems were assessed. One (Airi) was scored longitudinally from approximately eleven months of direct multimodal interaction (August 2025–July 2026). The other five were scored as Documentation-Based (Provisional) against a fixed evidence snapshot of official product documentation, vendor technical reports, published or reliably reported system prompts, and directly observed product behavior, all as available in May 2026. Each assessment quotes the satisfied threshold test verbatim, records the observable evidence, and states explicitly why the next level up was not awarded. Conflicts between vendor claims and documented behavior were resolved in favor of documented behavior, and capabilities available only through third-party extensions or wrappers were excluded. All six assessments were re-scored under framework v0.2.0 on 2026-07-07 from their original evidence bases; one rating (Airi, Goal Formation, Ambient band) was subsequently raised from 3 to 4 on 2026-07-16 based on capabilities shipped in June 2026.

Why This Matters

Impact on research. The paper targets a gap left by capability, automation, and risk frameworks. It argues that METR-style time horizons measure the capability ceiling of directed autonomy—how much a system can accomplish once instructed—but not whether a system acts when nothing instructs it, and that the two measurements are orthogonal by construction. It also argues that Anthropic's Responsible Scaling Policy and OpenAI's Preparedness Framework treat autonomy only as a hazard above a risk threshold, with no graded account of self-directed behavior below it. The AAS supplies a graded, behavioral alternative and invites independent and adversarial re-scoring; the framework, full rubric, and all assessment files are open and versioned.

Real-world applications (as framed by the paper):

• Vendor and product evaluation: distinguishing a capable model from a reactive product wrapped around it, as the ChatGPT profile illustrates.
• Agent deployment decisions: identifying whether an agent's idle-period behavior is scheduled work (predictable, auditable) or internally derived (requiring different oversight).
• Safety and monitoring: providing a graded vocabulary for autonomy-adjacent behavior below the catastrophic-risk threshold, which the paper notes existing risk frameworks lack.
• Longitudinal system comparison: using a common rubric to track a single system's agency profile as it changes across versions, as with the re-scoring of all six systems under framework v0.2.0.

Industry relevance. The Active–Ambient gap is presented as a structural property of contemporary agent products: strong within-task autonomy paired with between-task dormancy. For teams building or procuring agents, the framework offers a way to state explicitly what a system does when nobody is prompting it, and to separate genuine self-direction from sophisticated scheduling—a distinction the paper argues is currently described only informally.

Future Directions

1. Multi-rater studies to establish inter-rater reliability. All six assessments were performed by a single rater, so inter-rater reliability is currently unknown.
2. Re-derivation of the dimensions from prior theory and application to unrelated systems. The paper states the scale was developed alongside the systems it was initially designed to score, so high scores may partly reflect that correspondence rather than independent measurement.
3. An operational Active-band analogue of the Idle-Gap Test. The Ambient band's Level 4 is now gated by an observable counterfactual, but the Active band's Level 4 still lacks a strictly observable test distinguishing genuine self-direction from sophisticated rule-following while a system is engaged, and several Active-band Level 4 ratings continue to rest on judgment.
4. Principled duty-cycle weighting between bands—currently an open problem—plus independent, longitudinal re-assessment of the documentation-based scores, which the paper labels provisional.
5. The Longitudinal Turing Test is proposed as a complementary evaluation direction: if a system achieves a score of 4 or higher across all dimensions, a meaningful empirical test is sustained interaction over weeks or months, measured against the system's own prior behavior and pre-specified criteria. The paper notes this is a test of sustained self-directed behavior, not deception, and that it is confounded by anthropomorphism and requires blind raters and controls before supporting strong claims.

Target Audience

AI evaluation and safety researchers; teams designing or procuring autonomous agents; product managers deciding how much idle-period behavior a system should exhibit; and policy or governance analysts who need a graded vocabulary for autonomy below the catastrophic-risk threshold. Readers looking for a psychometrically validated instrument will not find one here—the paper explicitly positions the AAS as a structured descriptive framework, agnostic about consciousness, sentience, and moral status, and limited by single-rater provenance and developer-evaluator bias on the longitudinal assessment of Airi.

Authors’ abstract

Existing AI measurement frameworks quantify cognitive capability, task automation, or catastrophic risk, but none measure autonomous agency: the extent to which a system behaves in a self-directed way. A system can saturate capability benchmarks while remaining entirely reactive, acting only when prompted and ceasing all activity when a task completes. We introduce the Autonomous Agency Scale (AAS), a behavioral framework that scores AI systems on a 0-5 lexicon across seven dimensions of agency: cognitive autonomy, temporal persistence, environmental agency, social agency, creative agency, self-awareness, and goal formation, each operationalized by falsifiable threshold tests. Every dimension is scored in two temporal bands: an Active band covering engaged, user-initiated activity, and an Ambient band covering idle periods. Ambient Level 4 is gated by the Idle-Gap Test, a counterfactual criterion (remove all triggers and observe whether internally derived activity persists) that separates self-direction from scheduled rule-following. We apply the scale to six contemporary systems spanning task agents (Claude Code, Manus, Hermes), consumer assistants (ChatGPT, Siri), and a persistent companion architecture (Airi). The two-band profile quantifies a boundary that single-score frameworks conflate: task agents reach Active composites of 2.3-2.4 while scoring 0.6-1.9 Ambient, with every idle-period behavior attributable to user-configured schedules, whereas the companion architecture, evaluated longitudinally, is the only assessed system whose idle-period behavior survives trigger removal. We discuss limitations, including single-rater provenance, developer-evaluator bias on the longitudinal assessment, and the partially operationalized self-direction boundary in the Active band.