The Pulse
Coding Agents Miss 40% of Hidden Enterprise Requirements
A study of LLM agents on an enterprise text-to-SQL task found 40% of withheld requirements were missing or misrepresented. Only 25% of delivery runs across eight coding-agent configurations met every acceptance criterion.
AI.info Team · Written with AI · fact-checked ·
Forty percent of withheld requirements were missed or misrepresented in a study of coding agents working on enterprise text-to-SQL, according to a paper submitted to arXiv on Nov. 2. The result captures a gap between writing code and understanding the conditions a customer expects a system to meet. Across the study’s delivery runs, only 25% passed all acceptance criteria.
From an ambiguous request to deployment decisions
The researchers examine whether large language model agents can act as forward-deployed engineers: agents that discover customer requirements and turn them into deployment decisions. Their test uses a simulated customer grounded in the requirements of a real enterprise text-to-SQL use case. The customer begins with an ambiguous request, rather than a complete specification.
In the discovery phase, an agent must identify acceptance criteria, data permissions and hardware constraints. The paper evaluates discovery across 15 models. The strongest models find more than 80% of withheld requirements, while open-weight models find 40–50%. The study’s reported 40% shortfall concerns requirements that were missing or misrepresented; it does not describe every model’s discovery rate.
Finding requirements did not ensure delivery
After discovery, agents move to a delivery phase, where they plan and execute a post-training recipe. The paper reports that only 25% of runs across eight coding-agent configurations pass all acceptance criteria. That result measures complete success against the criteria, not whether a run produced any working output.
The authors identify unreliable execution as a separate bottleneck from incomplete discovery. They also report that agents underuse compute budgets and do not make systematic comparisons across base models and post-training methods. Those findings point to challenges both in learning what a customer wants and in carrying out a plan to meet it.
Why benchmark accuracy is not enough
The paper cautions that benchmark accuracy does not guarantee production readiness. Its test is an enterprise text-to-SQL engagement in which agents must account for requirements, permissions and hardware constraints as well as execute a post-training process. The authors frame the results around that specific task; the abstract does not claim that the same rates apply to all coding-agent work.
The authors recommend human review of the requirements agents discover and monitoring their plans during execution. They also call for an explicit experiment-environment specification that covers hardware, budget scope and production checks. Those are the paper’s stated safeguards for an engagement where missed requirements and incomplete delivery remain distinct problems.
Three safeguards the paper recommends
First, review the requirements the agent extracts from the customer interaction. Second, monitor the agent’s plan while it executes the delivery work. Third, specify the experiment environment in advance, including the hardware, what the budget covers and which production checks apply.
The study’s central figures separate two stages: the strongest models discover more than 80% of withheld requirements, yet only one quarter of delivery runs across eight configurations meet every acceptance criterion. The authors’ recommendations address each stage directly: review discovery, monitor execution and define the environment against which the work will be judged.