With Great Capabilities Come Great Responsibilities: Introducing the Agentic Risk & Capability Framework for Governing Agentic AI Systems

AI summary

Overview

Research area: Technical AI governance, specifically safety and security risk management for agentic AI systems (LLM agents that can execute code, browse the internet, modify files, and act autonomously).

Technical level: Beginner-Friendly. The paper is conceptual rather than mathematical—no benchmarks, datasets, or training runs—though it assumes the reader knows roughly what an LLM agent and tools/MCP servers are, and it uses some security terminology (prompt injection, privilege escalation, DoS/DDoS).

Scope: The paper proposes the Agentic Risk & Capability (ARC) Framework, a capability-centric technical governance framework that maps the components, design, and capabilities of agentic AI systems to materialized risks and to recommended technical controls, plus a practical organizational implementation process.

What This Paper Is About

Agentic AI systems can take autonomous action—running code, interacting with the internet, modifying files—which makes them more prone to unsafe behavior than their base LLM models and exposes organizations to a much broader range of risks than traditional LLM deployments. Because risk assessments customized to each individual agentic system do not scale, the authors set out to build a reusable, systematic framework that helps organizations identify, assess, and mitigate those risks while still allowing fast innovation.

Key Contributions

1. A novel capability-centric perspective. Instead of analyzing only an agent's components and system design, the framework also analyzes capabilities—the actions the system can autonomously execute, such as running code, searching the internet, or modifying documents. Capabilities are framed as the complement of affordances (as defined by Gaver, 1991): components and design are affordances, while executing code or altering agent permissions are capabilities.

2. Three elements as sources of risk intrinsic to all agentic systems. The framework distills components (LLM, tools, instructions, memory), design (agentic architecture, roles and access controls, monitoring and traceability), and capabilities (cognitive, interaction, operational) as the elements from which risks originate.

3. A clear nexus between elements, risks, and controls. Each risk in the Risk Register must (1) originate from an element, (2) satisfy a failure mode, and (3) result in at least one safety or security hazard; each risk is then paired with recommended technical controls tiered by criticality.

4. A structured, practical implementation approach. The framework specifies steps for centralized governance teams: contextualizing risks, establishing a relevance threshold, and scaling up through standardized declarations, checklists, and a regular cadence for updating the Risk Register. The framework is described as open-sourced.

Main Findings

• Three failure modes. Agentic systems fail through agent failure (poor performance, misalignment, unreliability), external manipulation (malicious actors causing or tricking the agent to deviate), or tool or resource malfunction (tools or resources fail, are compromised, or are inadequate).

• Hazard categories. Hazards are grouped under four types: Data (failures leading to data breaches, integrity attacks, PII exposure, or ransomware); Application (system failures, service disruptions, unintended use of applications, backdoor access, resource exploitation); Security (infrastructure & network—DoS/DDoS, man-in-the-middle, network eavesdropping, lateral access; identity & access management—unauthorized control, impersonation, privilege escalation); and Safety (illegal and CBRNE activities; discriminatory or hateful content; inappropriate content; compromised user safety; misrepresentation).

• Three advance-advantages of the capability lens. Capabilities are a more holistic unit of analysis than specific tools (many tools enable similar actions, and one tool such as GitHub's MCP server enables many actions); they allow differentiated treatment at scale (riskier systems get more scrutiny); and they are intuitive to laypersons, which aids contextualization.

• Capability taxonomy. Cognitive capabilities (Planning & Goal Management, Agent Delegation, Tool Use); interaction capabilities (Natural Language Communication, Multimodal Understanding & Generation, Official Communication, Business Transactions, Internet & Search Access, Computer Use, Other Programmatic Interfaces); operational capabilities (Code Execution, File & Data Management, System Management).

• Tiered controls. Controls are categorized as Cardinal (Level 0, fundamental requirements adopted as is), Standard (Level 1, adopted or adapted meaningfully), and Best Practice (Level 2, recommended for high-risk systems). Controls either reduce potential impact or decrease the likelihood of a failure mode.

• Worked example 1 — "Researcher" (deep-research-style agent). Its capabilities are identified as Planning & Goal Management, Natural Language Communication, and Internet & Search Access. Of 38 applicable risks, only 10 remain relevant (RISK-003, RISK-009, RISK-017, RISK-023, RISK-034, RISK-035, RISK-036, RISK-038, RISK-053, and RISK-054), with 17 associated controls to adopt or adapt. The sample risk—prompt injection via malicious websites—is scored impact 4/5 and likelihood 5/5, against a company relevance threshold of 3 for impact and 4 for likelihood.

• Worked example 2 — "Vibe Coder" (natural-language web-app builder). Its capabilities include Planning & Goal Management, Tool Use, Natural Language Communication, Internet & Search Access, Code Execution, File & Data Management, and System Management. There are 48 applicable risks—double the Researcher's capability risks—and 25 relevant risks against a lower threshold of 3 for impact and 3 for likelihood. The sample risk (overwriting or deleting database tables or files) is scored impact 3/5 and likelihood 4/5.

• Residual risk is unavoidable. The authors state that no list of controls can credibly claim to neutralize all threats, so organizations must evaluate residual risk after controls are applied. Common residual risks include inherent weaknesses in controls (guardrails trained on past jailbreaks may not generalize to novel attacks) and combinatorial risks arising from the interaction of two or more capabilities.

• Comparison with other frameworks (Appendix A). The ARC framework is compared against Dimensional Governance (Engin & Hand, 2025), OWASP Agentic AI (threats and mitigations), and Google's Secure AI Agents / SAIF 2.0. The table lists the ARC framework as medium–high prescriptiveness with a capability unit of analysis, and notes its stated gap: it would benefit from empirical evaluation, and the paper reports conceptual work plus worked examples with no empirical evaluation yet.

Methodology in Plain English

The authors surveyed existing literature on agentic AI governance—including regulatory frameworks (the EU AI Act, the NIST Risk Management Framework), adapted frameworks such as the AI TRiSM adaptation by Raza et al. (2025), dimensional governance, cybersecurity-oriented frameworks (MAESTRO, OWASP's agentic AI white paper, NVIDIA's taint tracing), safety and security benchmarks (Agent Security Bench, CVEBench, RedCode, AgentHarm, AgentDojo, APIBench, ToolSword, ToolEmu), and AI control work (Progent, AgentSpec, the UK AI Security Institute's control levels, OpenAI's and Google's best practices, and Beurer-Kellner et al.'s six design patterns).

From that review they argue existing approaches are either too narrow (application-specific) or too broad (high-level and conceptual) to operationalize. They then build the framework in three layers: (1) define the elements of agentic systems—components, design, capabilities; (2) combine elements with failure modes and hazard categories to enumerate risks in a Risk Register, keeping only risks supported by academic research or industry case studies; and (3) attach tiered technical controls to each risk and define an implementation workflow. To show it works, they hand-apply the framework to two stylized systems, scoring each risk on a five-point impact scale and a five-point likelihood scale and filtering by a relevance threshold.

Why This Matters

Impact on research. The paper positions itself within technical AI governance—"technical analysis and tools for supporting the effective governance of AI" (Reuel et al., 2025)—and adds a capability-centric unit of analysis that existing regulatory frameworks (EU AI Act, NIST RMF) and threat-modeling approaches do not provide. It also supplies testable claims: a draft Risk Register and control list that future work can empirically validate.

Real-world applications:

• Deep-research agents that search the web and compile reports, where prompt injection through malicious websites is a demonstrated attack path.
• Agentic coding tools that generate code, run front-end and back-end systems locally, and deploy to staging environments, where file or database deletion is a known failure (the paper cites a Replit failure described by Nolan, 2025).
• Enterprise office productivity and sensitive use cases such as hiring or performance assessment, where the paper argues impact is higher than in routine productivity tools.
• Government and critical infrastructure systems, where system failure can be severe and widespread and therefore demands greater scrutiny.

Industry relevance. The authors are affiliated with GovTech Singapore and the Singapore University of Technology and Design, and the target implementers are centralized governance teams managing AI risk across an organization. The framework is designed to give developers forward guidance so safety work happens upfront, to avoid imposing excessive compliance on low-risk systems, and to stay current as capabilities, tools, and threat landscapes evolve—an open-source artifact organizations can adopt rather than reinventing bespoke assessments.

Future Directions

• Empirical validation. The paper explicitly calls for developing empirical approaches to validate the risks and controls in the Risk Register, since the current work is conceptual with worked examples and no empirical evaluation.
• Automated tooling. Building automated tools to support implementation and regular updating of the framework.
• Calibration of thresholds and scores. Organizations must set impact and likelihood thresholds based on risk appetite, and the paper leaves open how these should be calibrated reliably—an issue it also raises as a weakness of dimensional governance approaches.
• Keeping pace with emergent risk. The Risk Register needs a defined review cadence to absorb new threats, emergent capabilities, and regulatory changes, and the hazard and control lists need continual updating as agentic AI matures.

Target Audience

Centralized AI governance teams responsible for managing agentic system risk; product, security, and developer teams who need upfront guidance on risks and controls; CISOs and security architects deploying agents in the enterprise; and policymakers or oversight leads looking for a more operational counterpart to high-level regulatory frameworks. Researchers in AI safety and technical AI governance will also find value in the taxonomy and the open research question of how to validate it empirically.

Authors’ abstract

Agentic AI systems present both significant opportunities and novel risks due to their capacity for autonomous action, encompassing tasks such as code execution, internet interaction, and file modification. This poses considerable challenges for effective organizational governance, particularly in comprehensively identifying, assessing, and mitigating diverse and evolving risks. To tackle this, we introduce the Agentic Risk \& Capability (ARC) Framework, a technical governance framework designed to help organizations identify, assess, and mitigate risks arising from agentic AI systems. The framework's core contributions are: (1) it develops a novel capability-centric perspective to analyze a wide range of agentic AI systems; (2) it distills three primary sources of risk intrinsic to agentic AI systems - components, design, and capabilities; (3) it establishes a clear nexus between each risk source, specific materialized risks, and corresponding technical controls; and (4) it provides a structured and practical approach to help organizations implement the framework. This framework provides a robust and adaptable methodology for organizations to navigate the complexities of agentic AI, enabling rapid and effective innovation while ensuring the safe, secure, and responsible deployment of agentic AI systems. Our framework is open-sourced \href{https://govtech-responsibleai.github.io/agentic-risk-capability-framework/}{here}.