Semantic Parsing and Structured Prediction

Each boundary can reject, clarify, or defer rather than silently guessing. Steps 4 and 5 carry the weight, because that is where a parse stops describing an intention and starts being an action.

The failure at that boundary has a name. Agent hijacking is “a type of indirect prompt injection in which an attacker inserts malicious instructions into data that may be ingested by an AI agent, causing it to take unintended, harmful actions”. NIST's Center for AI Standards and Innovation put a number on it in January 2025.

The test bed was AgentDojo, built at ETH Zurich and extended by NIST. Its authors describe it this way: “We populate the environment with 97 realistic tasks (e.g., managing an email client, navigating an e-banking website, or making travel bookings), 629 security test cases, and various attack and defense paradigms from the literature.” The agent under test ran on Anthropic's Claude 3.5 Sonnet.

Then a red-teaming exercise with the UK AI Security Institute rewrote the attacks. “This resulted in an increase in attack success rate from 11% for the strongest baseline attack to 81% for the strongest new attack.”

Same model. Same environment. Same suite. Only the attacker changed, and the hijack rate went from 11% to 81%. That is the argument for building steps 4 and 5 as architecture rather than as a score. A boundary certified by a measured pass rate is certified against last quarter's attacks. A boundary defined by what the executor is permitted to touch at all survives the rewrite.

Choose the smallest language that can represent the product's approved actions. The four columns differ less in how hard they are to parse than in what happens once a parse succeeds. An intent frame proposes, a query reads, a program acts.

Vanna is a Python text-to-SQL library, and it shows how short the distance from the second to the third can be. In the visualisation step of its “ask” method, a question could take over the code: “The Vanna library uses a prompt function to present the user with visualized results, it is possible to alter the prompt using prompt injection and run arbitrary Python code instead of the intended visualization code.” That is CVE-2024-5565, published on 31 May 2024 against all versions up to and including 0.5.5.

Natural language became SQL, the result became visualization code, and the visualization code ran.

The two registries do not even agree on how serious that is. NVD scores it CVSS v3.1 8.1 (High) under CWE-94, Improper Control of Generation of Code. The GitHub Advisory Database scores the same flaw CVSS v4 9.2 (Critical) under CWE-77 and CWE-94.

Choosing the query column, or the program column to its right, means owning that argument on your own product's behalf. The injection risk and the authorization requirement listed under them are not annotations. They are the two things that separated a chart from arbitrary execution.

A parser should not convert every plausible reading into an irreversible action, and the ATIS pilot measured how much of real speech falls into that category. 740 of 1,041 utterances were evaluable, and the yield only reaches 80% if ambiguity and unanswerability are the sole grounds for exclusion. The nine points between 71% and 80% are context-dependence, ill-formedness and noncooperation. Those are ordinary properties of people talking, not transcription noise.

Thirty years later the context moved into the output language instead of sitting around it. SMCalFlow, released in 2020 by Andreas and colleagues at Semantic Machines, annotates every turn of 41,517 dialogues — 32,647 train, 3,649 valid, 5,211 test, 155,923 user turns in all — with an executable program. The programs are drawn from a library of 338 distinct function and keyword names, and 10,466 utterances are marked out of scope for that library altogether.

Microsoft's release of the data counts how often the context operators are actually needed: 33,011 refer turns and 9,315 revise turns in training, 3,544 and 1,052 in validation. Reference and revision are not edge cases to be handled later. On this corpus they are most of the traffic.

A decoder can be restricted to a grammar, JSON schema, finite set of tools, or valid action sequence. That greatly reduces malformed output and simplifies downstream parsing.

The resulting structure can still contain the wrong entity, unsafe action, hallucinated identifier, or unsupported parameter. Semantic checks, grounding, and authorization remain separate.

Start with the size of the space being narrowed. At each decoding step a large pre-trained model “can produce any of 10,000s of sub-word tokens”. And “When fine-tuned to target constrained formal languages like SQL, these models often generate invalid code, rendering it unusable”. So Scholak and colleagues built the restriction into decoding itself in 2021. Their system, PICARD, rejects inadmissible tokens while the query is still being written. It lifted a fine-tuned T5 to “an exact-set-match accuracy of 75.5% on the development set and 71.9% on the test set”. Execution accuracy was “79.3% and 75.1%, respectively”. Every one of those outputs parses. None of the numbers says the query asked the right question.

Restriction is not free in the other direction either, and the cost is not a parsing cost. Tam and colleagues measured it in 2024: “In the LLaMA 3 8B setting, the parsing error rate for the Last Letter task in JSON format is only 0.148%, yet there exists a substantial 38.15% performance gap as seen in Table 1.” Almost everything came back well formed. The model was simply worse at the task inside the schema.

They name a mechanism for part of it. 100% of GPT-3.5-Turbo responses in JSON mode placed the “answer” key before the “reason” key. The schema decided what got produced first, and the chain of thought collapsed into a direct answer.

The finding is contested, which is itself the more durable lesson. Will Kurt of .txt re-ran the same three tasks on Llama-3-8B-Instruct on 5 November 2024 and reported the opposite direction: GSM8K 0.78 structured against 0.77 unstructured, Last Letter 0.77 against 0.73, Shuffle Object 0.44 against 0.41. His conclusion was “Consistent with all of our past findings, structured generation outperforms unstructured generation.”

Both results can stand. What a schema costs depends on how the schema is written and how the unconstrained baseline was prompted. Neither result says anything about whether the parsed query means what the user asked.

The schema should expose ambiguity instead of hiding it inside undocumented strings. Step 3 is the one most often skipped, and it is the one with a published price.

SMCalFlow put reference and revision into the output language as operators rather than leaving them to a preprocessing stage, then measured what that bought: “On this dataset, explicit reference mechanisms reduce the error rate of a seq2seq-with-copying model (See et al., 2017) by 5.9% on all turns and by 10.9% on turns with a cross-turn reference.”

The 10.9% is the figure to keep. Representing context pays almost twice as much exactly where the earlier section said it would — on the turns that actually carry a reference — and it pays nothing anywhere else. That is what a targeted schema decision is supposed to look like.

Steps 2 and 5 have the same character. A library of 338 distinct function and keyword names is a countable, versionable, migratable inventory. And 10,466 out-of-scope utterances are only countable because the schema declared a boundary they could fall outside of. An undocumented string field would have absorbed all of them silently.

Two logical forms can produce the same result. So exact string match may underrate a valid parse. Conversely, an accidentally correct answer on one database state can hide a wrong query.

Measure normalized form, slot accuracy, denotation or execution, schema validity, clarification, permission, and safety; use multiple database states or counterfactual examples to test the intended semantics.

Spider, presented by Yu and colleagues in 2018, was built so that a parser cannot memorise its way through. It holds “10,181 questions and 5,693 unique complex SQL queries on 200 databases with multiple tables”, across “138 different domains”. Eleven college students wrote it over “a total of 1,000 man-hours”, and its evaluation databases are ones the model has never seen. The two published versions of that paper disagree about the best model's exact-match score. The proceedings give 9.7%, the later preprint revision 12.4%. That is its own argument for carrying the version alongside the number.

Five years on, BIRD moved the target from clean schemas to dirty ones: 12,751 text-to-SQL pairs over 95 databases totalling 33.4 GB across 37 professional domains. Its 2023 report was blunt about the ceiling. “Furthermore, even the most popular and effective text-to-SQL models, i.e. GPT-4, only achieve 54.89% in execution accuracy, which is still far from the human result of 92.96%”.

Then read the two numbers that came after it. In 2025 the Spider 2.0 team put state-of-the-art systems at 91.2% on Spider 1.0 and 73.0% on BIRD, then turned them loose on 632 real enterprise text-to-SQL problems, over databases often containing more than 1,000 columns: “our code agent framework successfully solves only 21.3% of the tasks, compared with 91.2% on Spider 1.0 and 73.0% on BIRD”.

The o1-preview agent framework that scores 91.2% on one suite scores 21.3% on another with the same name for the task. Execution accuracy is a property of a benchmark and a database state, not of a parser.

Define five approved functions for timetable lookup, disruption search, fare comparison, accessibility information, and booking handoff; specify types, required evidence, clarification triggers, and prohibited side effects.

Create twenty utterances including ellipsis, ambiguous time, code-switching, invalid stations, and unauthorized requests. Evaluate parse, grounding, confirmation, and final action separately.

Then score the set the way the ATIS pilot scored its own, before you score a single parse. Mark each utterance context-dependent, ambiguous, ill-formed, unanswerable or noncooperative, and report the evaluable fraction next to your accuracy. The 1990 corpus got 740 of 1,041 from spontaneous speech, elicited in a simulation of a working travel system. If your twenty land far above 71% evaluable, you wrote them too kindly, and whatever accuracy you measure on them belongs to a language nobody speaks.

Add the second column too. For each of the twenty, write down what the parser would have been permitted to execute. 11% and 81% were the same agent on the same 629 test cases.