The Pulse
TiGanEval Finds Agents Lose Up to 67 Points on Hidden Requirements
A new benchmark tests whether AI agents can infer requirements that users leave out but that are discoverable in a project. Across eight frontier models, success rates fell by 53 to 67 percentage points when agents received partial instruct
AI.info Team · Written with AI ·
“An agent should therefore treat the instruction as evidence rather than as the objective, inferring the true requirements from the instruction and the environment and satisfying them even when the user is not yet aware of all of them.”
Kaiming Liu and co-authors, authors of “Instructions Are Evidence, Not Objectives: Measuring Outcome Alignment of Agents,” write.
The sentence captures the problem at the center of a new benchmark: an AI agent can follow a user’s request and still fail the task because the request leaves out requirements that are discoverable in the project itself. In a paper submitted to arXiv on October 13, researchers introduce TiGanEval to measure that gap, which they call “outcome alignment.”
When the request leaves out the rule
TiGanEval gives agents software engineering and research tasks with requirements that are fixed in advance but only partly described in the initial instruction. The missing requirements are meant to be discoverable from the workspace, including its code, records and runtime behavior. A simulated user can add information after seeing the consequences of an agent’s submission, letting the researchers compare what the agent infers on its own with what it learns through correction.
The paper’s motivating example is a payment retry. A user asks for a retry module after checkout times out, but project records show that the timed-out payment may already have been charged. An agent that implements retries without checking those records could charge customers twice; the full task requires handling the lost response without duplicating the charge.
The authors also examined 100 user-agent trajectories from SWE-chat. They found that 56% of initial instructions omitted requirements that later emerged as necessary, and that 52% of useful follow-up requirements could have been inferred from the project. Those findings motivate the benchmark, but the measured agent results come from TiGanEval’s separate set of constructed tasks.
Five requirements across 64 tasks
TiGanEval contains 40 software engineering tasks and 24 research tasks. Each task has five true requirements, while its partial initial instruction covers none of them in 39% of cases, one in 50%, and two in 11%; no task’s initial instruction covers more than two. Human reviewers checked that requirements matched the evaluators, could be inferred from available evidence and were not disclosed outright in agent-visible files.
The researchers evaluated 12 models, including eight they classify as frontier models, under three configurations. “Literal” treats the instruction as complete; “Partial” prompts the agent to inspect the project and infer what the task requires; “Full” gives it the complete specification. Under the partial condition, agents could make up to four evaluated submissions, with a shared limit of 120 turns and 360 tool calls per episode.
Feedback helps, but does not close the gap
For all eight frontier models, first-submission success under partial instructions fell 53 to 67 percentage points below the same model’s result with the full specification. GPT-5.6-Sol, for example, succeeded on 10.9% of partial-instruction tasks at its first submission, compared with 64.1% when it received the full specification. After four submissions, its partial-condition success rate reached 29.7%—still well below the full-information result.
Prompting agents to infer requirements performed better than asking them to follow instructions literally. Across the frontier models, the partial configuration scored 15.7 to 34.1 points higher than the literal configuration on the first submission. Yet later rounds relied heavily on user feedback: frontier models resolved at most 17.6% of their remaining unstated requirements before the user reported them.
A two-phase plan mode did not fix the shortfall in the three models tested. It raised GPT-5.6-Luna’s first-submission score by 4.3 points but did not increase its success rate; scores and success rates fell for Kimi-K3 and DeepSeek-V4.1-Flash. In the researchers’ analysis, the planning phase often ended before the agent had uncovered the full set of requirements.
The benchmark’s limits matter
The authors audited 100 failed partial-condition episodes and classified 72 as failures to use evidence about requirements. Some agents invented a rule despite contrary project records; others ran a replay that exposed a problem but did not make the corresponding fix. The finding shifts attention from whether an agent can execute a known specification to whether it can identify the specification that the task actually demands.
TiGanEval is a controlled test, not a direct estimate of performance across deployed agents or everyday user requests. Its 64 tasks reflect cost constraints, and the simulated user responds through authored observations rather than more complex patterns such as gradually revealing requirements. The benchmark’s central result is narrower: across its tested tasks, even strong models often succeeded when every requirement was spelled out, then failed when some requirements had to be inferred from the project.