SERIAL-7: Deriving, Realizing, and Preserving Machine-Verifiable Outputs Across Representations

AI summary

Overview

• Research area: Evaluation of large language models on structured, machine-verifiable output generation (format following, structured output, constraint following, cross-format conversion).
• Technical level: Intermediate. The paper uses formal notation for its verification protocol, but the core idea—separating "do you know the answer" from "can you emit it in a form a program can parse and trust"—is accessible without deep background.
• Scope: SERIAL-7 is a Korean–English benchmark of 177 validated canonical artifacts and 3,489 evaluation instances that holds the underlying answer fixed while varying the required output representation across seven formats, with 43 executable atomic constraints and three separate evaluation settings (Derive, Realize, Preserve).

What This Paper Is About

Large language models increasingly emit outputs that are consumed directly by programs, verifiers, and downstream systems, so a semantically correct answer can still fail if it violates the syntax, structure, or interface rules of the target format. Existing benchmarks typically vary tasks, prompts, schemas, and constraints all at once, which makes it hard to tell whether a failure came from reasoning, from formatting, or from the generated artifact itself. SERIAL-7 addresses this by fixing the semantic target and asking how reliably models can derive it, realize it under explicit representation contracts, and preserve it when converting between representations—and where exactly failures occur.

Key Contributions

1. A controlled benchmark holding semantics fixed. SERIAL-7 is a Korean–English benchmark (177 validated canonical artifacts, 3,489 evaluation instances) that renders the same canonical artifact into seven machine-consumable representations: JSON, XML, Markdown, HTML, LaTeX, YAML, and TOML.
2. An executable verification framework. The paper defines 43 atomic representation constraints (each a triple of instruction clause, deterministic renderer, and executable predicate) and couples them to deterministic checks over format, structure, and task-specific semantic invariants, allowing failure localization from representation compliance up to full artifact validity.
3. Three complementary evaluation settings. Derive measures representation-free semantic derivation, Realize measures verified structured generation under explicit contracts (with independently sampled L1/L2/L3 constraint-count conditions of one, two, and three atomic constraints), and Preserve measures verified cross-representation artifact transformation.
4. Progressively stricter metrics. Rather than a single score, the framework reports Format Compliance (FC), HSR (joint contract-and-task success), and Full Compliance, so that failures hidden behind a correct final answer become visible.

Main Findings

• Answer accuracy does not transfer to structured generation. Representation-free Derive accuracy spans a narrow 82.4–90.1% across ten model configurations, while Realize-L3 HSR spans 42.0–93.9%. Similar semantic performance therefore does not imply similar machine-verifiable structured output.
• Full Compliance exposes failures beyond correct answers. The stricter artifact-level criterion reveals additional mechanically detectable failures even when the final answer is correct and the representation is valid.
• Rankings shift across conditions. Claude-Sonnet-5 has the highest observed Derive score (90.1%) but reaches 63.3% HSR at Realize-L3; Nemotron-3-Super-120B-A12B reaches 95.6% under Preserve despite 62.7% at Realize-L3. The authors describe these as descriptive cross-condition differences, not paired causal effects, because the conditions use different artifact populations and input forms.
• Preserve is generally stronger but asymmetric. Preserve HSR ranges from 85.9% to 95.6% across models, and pooled source–target asymmetry is substantial: JSON→Markdown reaches 0.994 HSR while Markdown→JSON reaches 0.647.
• Constraint count penalizes most models. Nine of ten configurations have lower HSR at L3 than at L1. GPT-5.4 differs by only about two percentage points, while Nemotron-3-Super, Motif-3-Beta, A.X-K2, Claude-Sonnet-5, and GPT-5.4-nano differ by more than 12 points; Gemini-3.7-Flash is the sole exception. Because L1–L3 rule sets are sampled independently rather than cumulatively, these are not a nested difficulty progression.
• Certain atomic constraints are frequently violated. The lowest mean pass rates occur for toml.key_order (0.743), latex.siunitx (0.785), yaml.answer_object (0.831), and markdown.heading_depth (0.839). The same abstract key_order rule scores 0.871 in JSON, 0.844 in YAML, and 0.743 in TOML, showing carrier-specific variation.
• Failures have different mechanisms. Some low-scoring rules reflect cascading failures (malformed output prevents structure recovery—e.g., a TOML answer key scoped inside the final [[steps]] table instead of the root), while others are direct rule violations inside otherwise valid documents (e.g., YAML nested mappings ordering ref before name).
• Language effects are modest and condition-dependent. Aggregate Korean–English gaps are generally small; Claude-Sonnet-5 shows the largest English advantage (+5.2 pp) and Solar-Pro4 the clearest Korean advantage (−2.5 pp). Aggregate gaps mask reversals: Gemini-3.7-Flash and K-EXAONE-236B-A23B favor English under Realize but Korean under Preserve, and Motif-3-Beta favors English in Derive and Realize but Korean in Preserve.
• Representation–model interactions are strong. GPT-5.4 varies relatively little across carriers, Gemini-3.7-Flash is weaker on LaTeX, and Solar-Pro4 achieves higher HSR on XML than on several other representations.

Methodology in Plain English

The researchers started from 500 problems drawn from four reasoning benchmarks (TabMWP, HRM8K, SciBench, and the STEM subset of MMLU-Pro) and converted each into a single "canonical artifact"—a structured record containing the final answer, the ordered input identifier–value pairs, and optional computation steps. Candidate conversions were produced with gemini-3.1-pro-preview and treated only as candidates, never as gold; each was then passed through deterministic checks for answer consistency, grounding, structural validity, graph executability, and representation compatibility, with repairable errors fixed and revalidated and everything else excluded. Korean versions were created by translating with gemini-3.7-flash and having bilingual annotators review semantic equivalence, fluency, quantity preservation, computation structure, and source grounding, while identifiers, values, operations, step structure, and answers stayed fixed.

Because one canonical artifact can be deterministically rendered into JSON, XML, Markdown, HTML, LaTeX, YAML, and TOML, the same answer can be tested under seven different formatting demands. On top of each format's base contract, the authors defined 43 "atomic constraints" (things like key ordering, numeric encoding as strings, table versus mapping layout, or LaTeX math mode); each constraint comes with both a renderer that produces the correct reference and an executable predicate that checks the model's output, so scoring is mechanical rather than similarity to a gold string.

Three evaluation settings use this machinery. Derive just asks for the answer with no format demand. Realize asks the model to produce the artifact in a given format with one, two, or three independently sampled constraints active (L1/L2/L3). Preserve hands the model an artifact in one format and asks for the same content in another, checking source–target equality of the answer, ordered inputs, and step details. Outputs are parsed back into the shared canonical form and checked across three layers—format, structure, semantic—yielding Format Compliance, HSR, and Full Compliance. Ten model configurations were evaluated, with Korean–English analyses using the canonical artifact as the statistical unit and two-sided paired permutation tests with Holm correction.

Why This Matters

Impact on research. The paper argues that reliable structured generation cannot be reduced to answer accuracy or a single aggregate format-following score. By holding the semantic target fixed and pairing representation contracts with executable predicates, SERIAL-7 lets researchers attribute a failure to derivation, to representation realization, or to artifact validity—a separation the authors note is missing when benchmarks vary tasks, prompts, schemas, and constraints together.

Real-world applications:

• Generating JSON, YAML, or TOML configuration and data files that downstream programs parse and execute without manual repair.
• Converting structured documents between formats (for example JSON to Markdown or Markdown to JSON) in data-interchange and documentation pipelines.
• Producing LaTeX, HTML, or Markdown reports and tables where syntax errors break rendering or downstream tooling.
• Serving as the output layer in tool-using or agentic systems where a verifier, not a human, inspects the model's response.

Industry relevance. The authors are affiliated with KT Corporation, and the benchmark includes both global models and Korean-developed models (Mi:dm K 3.0 Pro, Solar-Pro4, A.X-K2, K-EXAONE-236B-A23B, Motif-3-Beta), with paired Korean–English evaluation. This matters for deployment decisions in bilingual settings, since the paper shows aggregate language gaps can mask condition-specific reversals and that model rankings change depending on whether the task is derivation, constrained realization, or cross-representation preservation.

Future Directions

• Constraint interactions. Because L1–L3 rule sets are sampled independently rather than cumulatively, the paper explicitly does not establish a causal difficulty progression from one to three constraints; a matched or nested design would address how constraints interact.
• Denser conversion coverage. Preserve samples five directed transfers per artifact out of 42 possible directions, and opposite transfers may contain different artifact subsets, so the reported asymmetry is a pooled benchmark-level observation rather than a matched-pair estimate of directionality.
• Comparable condition populations. Derive, Realize, and Preserve draw on different eligible artifact subsets (162, 37, and 177 artifacts respectively), which the authors say prevents interpreting cross-condition scores as paired item-level effects; aligning populations would enable stronger claims.
• Extending beyond the current design. The framework currently covers seven representations and 43 atomic constraints with answer-only artifacts (140 of 177) excluded from the step-level Realize evaluation; broadening verifiable structure to those artifacts and additional carriers is a natural open question.

Target Audience

This paper is most useful to researchers and engineers building or evaluating structured-output systems: benchmark designers working on format following and constraint following, practitioners who need model outputs that downstream programs can parse and trust, and teams selecting models for bilingual Korean–English deployments where format reliability matters as much as answer accuracy. Readers need only a working familiarity with structured formats and LLM evaluation to follow the main results; the formal verification protocol is contained in the main text and appendices for those who want the details.

Authors’ abstract

Large language models increasingly generate structured outputs consumed by programs and downstream systems, where semantic correctness alone does not guarantee reliable execution. We introduce SERIAL-7, a Korean-English benchmark built around an executable verification framework for machine-verifiable generation across seven representations. Using 177 validated canonical artifacts, SERIAL-7 separates semantic derivation (Derive), constrained artifact generation (Realize), and cross-representation preservation (Preserve). Its 43 atomic representation constraints are paired with executable checks over format, structure, and task-specific semantics, enabling progressively stricter verification from representation compliance to full artifact validity. Across ten models, REALIZE-L3 HSR ranges from 42.0% to 93.9%, while Full Compliance exposes additional artifact-level failures beyond correct final answers and valid representations. These results show that reliable structured generation requires explicit, executable verification of how artifacts are realized and preserved across representations.