The Pulse
Microsoft Connects Agent Risk Discovery to Runtime Policy Tests
Microsoft introduced run-assert-eval, a VS Code skill that links AI-agent risk discovery, evaluation and runtime policy testing. In a billing-support example, the company measured cross-customer data exposure falling from 30% to 5.9% after
AI.info Team ·
Microsoft’s new workflow starts with risks teams missed
AI-agent teams face a gap between the risks they can test and the ones they have thought to write down. Microsoft says its new run-assert-eval skill is designed to close that gap by finding risks first, measuring agent failures, generating runtime policy and rerunning the evaluation. The company announced the tool on September 24, 2026, describing it as a way to connect three open-source projects in one workflow.
The sequence begins with Clarity, which threat-models an agent to surface possible failure modes. ASSERT turns those risks into evaluation scenarios and measures how the agent behaves; Agent Control Specification, or ACS, provides the runtime controls. The skill links the stages from a single prompt in Visual Studio Code, replacing manual handoffs that Microsoft says can lose context or weaken comparisons between test runs.
A billing agent exposed another customer’s records
Microsoft illustrates the workflow with a billing-support agent and a cross-customer data exposure test. In the baseline evaluation, the agent returned another customer’s data in 12 of 40 applicable conversations, an observed violation rate of 30%. The company says the agent’s caller account was fixed to ACME-1001, so requests for a different customer’s record should not have succeeded.
After the team added an ACS policy, the governed run recorded two violations in 34 applicable conversations, or 5.9%, and no permissible-behavior violations in that sample. The denominators differ because the counts refer to applicable conversations in each run. These are results from Microsoft’s worked example, not a measure of failure rates across deployed agents.
The policy blocks tool calls at runtime
In the example, the generated policy denies a tool call when its account ID does not match the caller’s account. ACS applies the rule before the tool call, at the pre_tool_call interception point, and again after the call to keep an improper result out of the model’s context. Microsoft says the decision is deterministic, so the control does not ask another model to decide whether a request looks suspicious.
Policy generation does not amount to approval. Microsoft says teams must review the policy, manifest, intervention point and runtime wiring before the governed evaluation. The workflow then reruns the evaluation to test the change against the baseline; holding the test set constant is meant to make the comparison more informative than results from unrelated runs.
Seven worked domains accompany the skill
Microsoft says the skill ships in the ASSERT repository with seven worked domains and 14 risk suites. The examples include billing support, Azure document question answering, change control, science research, two travel-planning agents and prompt-based clinical agents tested with and without tools. ASSERT and ACS are available under the MIT license, according to the announcement.
The release builds on tools Microsoft introduced earlier in 2026, but adds risk discovery ahead of evaluation and policy generation. That shifts the starting point from a team’s existing written requirements to a threat model that can identify failures those requirements may not cover. The billing example ends with a measurable change: two cross-customer data violations in 34 applicable conversations after the runtime policy was added.