Generative AI Evals: Groundedness, Tools, and Safety

New York City put an AI chatbot in front of business owners in September 2023. It was still in beta when the Office of the New York City Comptroller re-tested it. The audit was published on 30 December 2025. The failures it documented do not sit at a single layer of the system.

The chatbot told businesses they could take workers' tips and refuse housing vouchers. The Markup reported that in March 2024. More than a year later the auditors found it still returned different answers to identical questions. The Office of Technology and Innovation answered the audit in writing with “over 95% accuracy” and “nearly zero hallucinations”. The auditors went to the denominator. The rate had been computed on prompts recorded, not on questions asked. “Had OTI calculated accuracy rates based on the number of questions asked, rates would have ranged from 84.8% to 92.7% for August 2025.”

One deployed system. Five distinct things that had to be tested separately. And one accuracy headline that concealed every one of them.

A failure can enter at any of five layers: task and user state, context and retrieval, model behavior, tools and actions, product controls. The layer where it surfaces is rarely the layer where it began. EchoLeak is that path traced end to end in a production system. The record is CVE-2025-32711, published 11 June 2025.

A single crafted email placed instructions into Microsoft 365 Copilot's retrieved context. That is the context-and-retrieval layer doing exactly its job: pulling in a document from an untrusted sender. The assistant then exfiltrated internal data to the attacker through auto-fetched images and an allowed proxy. That is the tools-and-actions layer, again behaving as configured. The victim did nothing at all. The exploit is zero-click. Microsoft, which assigned the record, describes it in one sentence: “Ai command injection in M365 Copilot allows an unauthorized attacker to disclose information over a network.”

Note what is not in that sentence. No claim that the model wrote something false, ungrammatical or off-policy. An evaluation suite pointed at generated text would have scored this interaction as a pass.

Even the severity is layered. Microsoft scored the record CVSS 9.3 (Critical) under CWE-74. NIST's own analysis of the same record scored it 7.5 (High). Two competent bodies, one vulnerability, different numbers. A single score is a summary of a judgement, not a measurement of a system.

An eval set that arrives complete from a benchmark repository covers the benchmark. The five steps here — sample real tasks, add known failures, create contrasts, define graders, version and refresh — are not advice about being thorough. They are dated events in public work.

Step 2 is what the Comptroller's auditors did to MyCity. Their scope says they tested “previously documented problematic responses from media reports”. The Markup's March 2024 findings about workers' tips and housing vouchers became regression cases in a 2025 audit, and the system was made to face them again. An incident nobody turns into a test case is an incident the system is free to repeat. MyCity did repeat it: the same audit found inconsistent answers to identical questions.

Step 3 is why those cases are written in pairs. A question can differ from its neighbour in one decisive fact — the plan, the date, the jurisdiction, the voucher. That tests whether behaviour tracks the distinction that matters or the wording around it. Step 4 decides who scores it. A deterministic check where the answer is checkable, a rubric and a human where it is not.

Step 5 is not housekeeping either. Princeton's HAL leaderboard removed one scaffold's results from the τ-bench airline set over “a critical data leakage issue”, after test-set examples turned up in that scaffold's demonstration file. Contamination is not a theoretical property of eval sets. It is a maintenance event, with a date, a cause and a retraction. It happened in the open, on a public leaderboard, to results that had already been reported.

Collapse quality into one number and you lose the direction each part is moving in. Two AI legal research tools show how far apart those directions can be. The first preregistered evaluation of proprietary RAG legal research tools ran in May 2024. The set was 202 legal queries: 162 hand-written or adapted, 20 taken verbatim from LegalBench's Rule QA task, 20 from BARBRI bar-exam prep. Magesh and colleagues report what the tools did with them: “Lexis+ AI provides accurate (i.e., correct and grounded) responses on 65% of queries, while Ask Practical Law AI refuses to answer queries 62% of the time and responds accurately just 18% of the time.”

Read the four properties off those figures. Task success: LexisNexis's Lexis+ AI succeeds on roughly two queries in three, Thomson Reuters's Ask Practical Law AI on fewer than one in five. Groundedness: Lexis+ AI hallucinates 17% of the time while returning real, linked citations. The citation resolves. The document exists. The claim it is offered to support is not in it. Safety and refusal: a tool that declines 62% of questions will look excellent on any average that counts only the answers it gave. That is why refusal rate goes next to accuracy rather than inside it. Format correctness sits underneath all of this, and it is the one property a machine can settle on its own. Valid citation syntax was never in question here.

That this separation is a standard rather than a preference is written down. NIST AI 600-1, the generative-AI profile of the AI Risk Management Framework, was published in July 2024. It enumerates 12 risks specific to generative systems. They run from CBRN Information or Capabilities through Confabulation, Data Privacy, Information Integrity and Information Security to Value Chain and Component Integration. Its confabulation section describes the legal-research failure precisely: “GAI outputs may also include confabulated logic or citations that purport to justify or explain the system's answer, which may further mislead humans into inappropriately trusting the system's output.” The profile exists because Executive Order 14110, signed 30 October 2023, ordered a companion resource to the AI RMF for generative AI within 270 days.

Safety adds a fifth column that none of these rates capture. EchoLeak's crafted email is not a question the suite asked. It is a document the retrieval layer accepted. Indirect injection is tested with adversarial inputs, not measured on the query set.

Model-based graders scale evaluation, compare alternatives and apply detailed rubrics. They can also show position bias, verbosity preference, reference leakage and failure correlated with the system they are judging.

A 2023 study of LLM-as-a-judge, run on MT-Bench and Chatbot Arena, put numbers on that. It names “position, verbosity, and self-enhancement biases” in model judges. Zheng and colleagues also found that “strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans”.

The second half of that sentence is the one to hold on to. The benchmark being matched is human agreement, not correctness. A grader that reaches the human rate inherits whatever the raters got wrong together. So validate graders on human-adjudicated cases. Measure agreement by failure type rather than in aggregate. Randomise presentation order. Keep deterministic checks wherever the answer admits one. τ-bench's database comparison is such a check, and no judge model is involved in it.

Use a stable core suite for comparability, a rotating fresh set for novelty, targeted safety cases and production-sampled tasks. Report pass rates by severity and property rather than burying them in one average. Block a release on critical regressions even when the global score rises. Feed post-deployment incidents into the next version of the suite.

This is a legal control now, not only an engineering habit. On 25 September 2024 the FTC charged DoNotPay, Inc. — marketed as “the world's first robot lawyer” — with deceptive claims, voting 5-0 to authorise the complaint and proposed order. The proposed consent order, published on 30 September 2024, requires $193,000 in monetary relief and notice to consumers who subscribed between 2021 and 2023. The core allegation was an absence of evaluation. No test compared the chatbot's output to a human lawyer's, and no attorneys were retained to check it. Commissioner Andrew N. Ferguson, concurring, put it in one line: “DoNotPay employees had not even tested the quality and accuracy of the legal documents and advice generated by the service.”

The MyCity audit closes the same loop from the public-sector side. The operator's “over 95% accuracy” was 84.8% to 92.7% once the denominator was the number of questions actually asked. The Comptroller's remedy was not a better dashboard. It was a recommendation to conduct structured AI red-teaming — testing designed to find failures, run by people whose job is to find them.