The Pulse
UK Asylum AI Summaries Fail 22% of First-Stage Checks
Open Rights Group says 45 of 203 Home Office AI-generated asylum interview summaries failed their first expert check after rollout. The group says only 0.3% of summaries were checked from April to July and is calling for the tools to be sus
AI.info Team · Written with AI ·
The UK Home Office says its AI tools assist asylum decision-makers, who retain responsibility for each claim. But in checks cited by Open Rights Group (ORG), 45 of 203 AI-generated interview summaries failed their first expert review—a 22% failure rate. The advocacy group’s report, published 12 October, says the live-use results outpace the 9% of summaries removed during the Home Office’s earlier pilot.
The figures concern summaries, not asylum decisions: they do not show that AI itself grants or refuses claims. They do sharpen a dispute over how much human scrutiny surrounds the tools, what applicants are told, and whether the system’s promised time savings justify its use.
45 summaries fail the first check
Home Office Subject Matter Experts reviewed 203 summaries after the Asylum Case Summarisation tool, or ACS, was rolled out to caseworkers in April 2026. ORG says 139 passed the first check, 19 were referred for another review and 45 failed. Among those 56 second-stage reviews, 30 also failed; that is 15% of the original 203, not a separate 15% failure rate among the 56 reviewed.
The Home Office’s 2025 pilot evaluation found that 9% of AI summaries were inaccurate or incomplete and removed them from the pilot. In that trial, technical specialists checked every summary before use. ORG’s later figures come from post-rollout spot checks, so the two rates describe different stages and checking arrangements. The comparison is a warning signal, not a like-for-like controlled test.
ORG says reviewers marked summaries as failed when they were not useful for orienting a caseworker or scored poorly for consistency. The group also points to a limit in the review criteria: omissions of relevant interview details were assessed for relevance, but that score was not itself a stated basis for failing a summary. A summary could therefore leave out material information and still pass if the remaining account was accurate, the report says.
Only 0.3% of summaries were checked
The 203 expert checks covered an estimated 70,000 cases processed from April through July, using the Home Office’s average of 17,515 cases a month. ORG calculates that experts reviewed about 0.3% of the summaries produced in that period. The sample provides evidence of errors in checked outputs, but it does not establish the error rate across every summary.
ORG also says caseworkers raised seven issues through a feedback inbox over those four months, while the department’s monthly quality assurance of asylum decisions does not assess the ACS summary itself. The group reports that applicants are not specifically told when AI has been used in considering their claim and cannot see or correct the generated summary.
Both tools use GPT-4o through Azure AI Foundry, according to ORG. ACS condenses asylum interview transcripts; the separate Asylum Policy Search tool, or APS, is a chat interface for finding and summarising country-policy information. ORG says the Home Office told MPs that APS expert testing had ceased and that no written summary of those test findings was produced.
Ministers say people still decide claims
In a 7 August letter to MP Bell Ribeiro-Addy, Minister for Border Security and Asylum Anna Turley said the tools are aids, not substitutes for a decision-maker. “All asylum claims are considered on their own merits and the decision written by a decision maker. It is not possible to decide an asylum case using APS and ACS,” Turley wrote in the letter.
The government’s original efficiency case rests partly on pilot results: the Home Office evaluation found test-group decision-makers reviewed transcripts 23 minutes faster on average. ORG argues the live-use evidence has not demonstrated improved throughput. Its analysis of Home Office figures found that average monthly initial substantive decisions fell about 42% from the same period in 2025, while the mean interval from interview to decision rose 29%. Those figures describe changes over time; they do not by themselves show that the AI tools caused them.
ORG Programme Manager Sara Alsherif said the department was using AI on people seeking asylum without telling them. “We urge the Government to regulate AI and outright ban its use in asylum applications, where the risk to rights are so high,” she said in the group’s statement.
ORG wants suspension and an independent audit
The group’s report draws on parliamentary questions, ministerial letters and Freedom of Information responses gathered over a six-month investigation. ORG says the department has not published the prompts used by ACS, that the tools are absent from the government’s Algorithmic Transparency Recording Standard register, and that no independent audit is planned.
In June, then-minister Alex Norris told Ribeiro-Addy that the Home Office did not plan an independent audit “at this stage,” according to the report. ORG is asking the department to suspend both tools, put measures in place for affected applicants and commission an independent live-use evaluation. Its central demand is specific: publish a fuller account of how often the tools fail, and what happens to a case when they do.