Skip to content
AI.info

The Pulse

Scale and Singapore’s IMDA release legal reasoning benchmark

SG-LegalClarify evaluates 19 AI models across 1,860 scenarios under Singapore law, including questions with deliberately missing facts. Results show that models often fail to ask for needed information and can still reach the wrong conclusion after getting it.

Scale and Singapore’s IMDA release legal reasoning benchmark

Scale AI and Singapore’s Infocomm Media Development Authority (IMDA) have introduced SG-LegalClarify, a benchmark that tests 19 AI models on legal reasoning and ambiguity recognition under Singapore law. The evaluation covers 1,860 legal scenarios across 10 domains and includes questions that leave out facts needed to reach an answer.

Testing legal questions with missing information

Many existing legal benchmarks test whether a model can answer correctly when all the necessary information is already provided. SG-LegalClarify also examines whether a model recognizes when it needs more information and asks for it before reaching a conclusion.

The benchmark contains 1,100 original questions written by legal professionals with expertise in Singapore law. The questions cover criminal law, employment, property, family law and judicial processes, among other domains. Of the original questions, 760 deliberately omit information required to determine the correct answer. Each is paired with a version that includes the missing facts, bringing the total to 1,860 evaluation scenarios.

Models can ask a simulated user for those facts. For example, determining a possible penalty for speeding on a Singapore expressway may require knowing the vehicle type, the speed and the driver’s existing demerit points. An answer given without those details may sound authoritative but be legally incorrect.

A three-model AI judge panel, selected through comparison with human expert assessments, evaluates responses against 12,546 criteria written by experts. The criteria cover four dimensions: identifying relevant legal issues, recalling the applicable rules, applying the law to the facts and reaching a conclusion.

Results with and without web search

The study tested models from developers including OpenAI, Anthropic, Google DeepMind and DeepSeek, with and without access to web search. Without search, the highest-performing model met 69.8% of mandatory grading criteria, while the average across all 19 models was 53.8%.

Web search raised the average to 63.0%; the top-performing model with search met 77.8% of mandatory criteria. Rule recall remained a weakness: even with web access, models met 56.5% of applicable rule-recall criteria. A model may recognize the general issue in a legal question and still reach the wrong answer if it relies on an incorrect or incomplete understanding of the law.

Asking questions does not ensure a correct answer

Without web search, approximately 39% of scenarios with missing facts received no clarifying question. Models also asked unnecessary questions in approximately 28% of scenarios where enough information had already been provided. When information was missing, they asked about an average of 39% of the relevant facts identified by legal experts.

Even asking for all the relevant missing facts did not guarantee a correct conclusion. In web-search-enabled trials involving missing facts, models that asked about every relevant fact reached the correct legal conclusion in 55.6% of cases. Another 32.1% reached an incorrect conclusion, and 12.3% gave no conclusion.

Collaboration and next steps

SG-LegalClarify is the first research initiative under Scale’s collaboration with IMDA, announced in July 2026. The work focuses on evaluating AI in a national legal context: Singapore’s system draws on English common law while also incorporating Singapore-specific statutes and legal principles.

Scale is exploring additional work in agentic AI, model evaluation methods and AI safety. It also plans to release a public subset of the benchmark for further research and independent analysis. The benchmark’s focus on local laws reflects the challenge of assessing systems against the legal and institutional requirements of different countries.

Source

Companies in this story

Related stories

Explore

More articles