The Pulse
RuleTrek Benchmark Finds Agents Miss Complex Rules
RuleTrek tests 11 language models on tasks that require finding and applying rules scattered across files. Its results show lower end-to-end scores as rule graphs grow larger or deeper.
AI.info Team · Written with AI ·
RuleTrek’s top-performing model scored 55.29% on end-to-end rule tasks, exposing a gap between simplified benchmark questions and scenarios where agents must find and apply rules scattered across files. The benchmark paper, submitted to arXiv on October 14, tests 11 language models across six domains, including taxation, cybersecurity and PostgreSQL. Its authors say agents struggled even when the benchmark handed them the relevant rules in advance.
Agents stumble before they reach the answer
The authors frame RuleTrek as a test of two linked abilities: locating which rules apply, then reasoning through them with details distributed across documents. They argue that many existing rule-reasoning benchmarks present only a few facts, while workplace tasks can require agents to search rule repositories and reconcile records, configuration files or logs. In RuleTrek, agents work in interactive file-and-shell workspaces rather than receiving every fact in a single prompt.
The benchmark separates that challenge into three scores. Rule Identification measures whether an agent finds applicable rules and their dependencies; Rule Deduction tests whether it can apply a supplied, verified rule graph; End-to-End combines identification and application. GPT-6.1-Sol led the average Rule Identification score at 62.47% and Rule Deduction at 60.43%, while end-to-end scores across all tested models ranged from 26.97% to 55.29%.
“These results demonstrate that current LLM agents face substantial challenges in rule reasoning over complex scenarios.”
Chenhao Yuan and co-authors, authors of the paper, write.
240 tasks span six different rule systems
RuleTrek contains 240 instances, with 30 to 48 in each domain: taxation, game rules, procurement, cybersecurity, software policy and PostgreSQL. The authors built scenarios from real-world rule repositories, then extracted rule graphs showing relationships such as exceptions, precedence, eligibility conditions and dependencies. They generated smaller scenarios around individual rules and combined them, with human checks for consistency and solvability.
Tasks require agents to search and inspect files, and they may execute code. Each model was run three times, with a 30-minute limit and a maximum of 200 agent steps per run; the paper reports scores averaged across runs and instances. For the deduction and end-to-end tasks, a separate judge scores answer correctness and reasoning, including fact grounding and rule application.
Longer rule chains bring lower scores
RuleTrek’s results point to a relationship between graph complexity and performance: the average end-to-end score across models fell from 76.5 to 23.6 as rule graphs grew from at most three nodes to at least 14. Scores also fell from 62.3 to 26.4 as graph depth increased from at most one to at least five. The figures indicate that a model’s challenge is not just finding a relevant rule; it must also track how that rule depends on others.
In an analysis of GPT-6.1-Sol’s failed end-to-end attempts, rule-selection errors made up 54.9% of failures, the largest category. The authors also describe models overlooking conditional cybersecurity controls, missing tax-year setup rules or merging game rules that applied to different travel paces. Even with the verified rule graph supplied, the best Rule Deduction score was 60.43%, suggesting that correctly applying known rules remains a separate obstacle.
The benchmark’s limits are part of the result
RuleTrek offers a controlled test, not evidence that models will fail at the same rate in every deployed workplace. Its authors limit the benchmark to English-language rules and predefined rule and scenario repositories, and note that rules and context can change in real settings. The paper describes the benchmark as a way to measure how agents handle complex, multi-file rule tasks; its results do not establish how every agent system, with different tools or safeguards, would perform.
The RuleTrek repository accompanies the paper. Its central finding is specific: even when agents have time to inspect files and use tools, performance slips when they must connect distributed evidence to increasingly dependent rules.