The Pulse
AWS introduces cross-framework evaluation system for AI agents
AWS has introduced an open-source system that scores AI agents across frameworks using 36 built-in metrics. It stores evaluation results so teams can compare agent versions and investigate changes.
AI.info Team · Written with AI · fact-checked ·
A new AWS evaluation system scores AI agents built with different frameworks using a shared set of 36 metrics across seven dimensions. The Universal Agent Evaluation Framework, or UAEF, also stores results for comparison between agent versions, according to an AWS post published November 11.
Customers across 12 organizations in telecommunications, healthcare and financial services have used the system to evaluate agents with consistent metrics and compare versions before release, AWS said. The source code is available under the Apache 2.0 license.
One format for agent traces
The system uses adapters to turn records of agent runs into a common trace format. Those records can include messages, tool calls and handoffs between agents. The evaluation engine scores the normalized trace rather than the format produced by the agent's original framework.
Adapters cover LangGraph, LangChain, Strands Agents, Amazon Bedrock AgentCore and traces collected in Langfuse. A generic JSON adapter handles other frameworks. The system can evaluate an existing trace or call an agent through an endpoint or managed runtime and collect a response for evaluation.
A test dataset supplies inputs for the agent and can also contain expected answers. Reference answers are optional, although metrics that compare responses with a reference require them. For tool-calling evaluation, the ground truth must include the expected tools, their order and arguments.
Scores and safeguards
The metrics cover tool calling, response quality, responsible AI, performance, multi-turn conversations, multi-agent coordination and reasoning. Each metric returns a score from 0 to 1, with 1 the best. Users can also add custom metrics or use integrations with RAGAS and DeepEval.
Some scores are calculated directly from the trace and ground truth. Others use a large language model on Amazon Bedrock to judge qualities such as relevance or safety. To limit the effect of agent output attempting to influence a judge, the system separates the scoring rubric from the content being evaluated and checks that the judge's cited evidence appears in that content.
AWS cautioned that judge scores are evaluation signals, not safety decisions, and should not gate a security control. The system screens traces for personally identifiable information before storage, and users handling sensitive data can turn on redaction for the stored copy.
Comparing runs
When a run is persisted, experiment metadata and averages are stored in Amazon DynamoDB, while full results are written to Amazon S3. A run can record details of the design under test, including its model, tools, system prompt and parameters. A team can mark a run as a baseline for later comparisons.
The comparison engine calculates absolute and percentage changes for metrics shared by the runs being compared. The default threshold for classifying a change as a regression or improvement is 5 percent, and it can be configured. The detector also groups regressions by dimension and assigns a severity.
The web interface displays results by experiment, metric and test case, and can generate a statistical report from stored scores without using a language model. The Python library can also pass a report to a model for analysis, including hypotheses about causes and suggested next steps. Teams can then change an agent's prompt, model or tools and evaluate it again.