How Useful are LLMs for Grammar Engineering? Cantonese ParGram Resources and Controlled Experimental Evaluation with English Baselines

AI summary

Overview

Research area: Natural Language Processing and computational linguistics, specifically knowledge-driven grammar engineering and the evaluation of large language models as tools for formal grammar development.

Technical level: Advanced. The work assumes familiarity with constraint-based/lexicalist grammar formalisms and with grammar-development frameworks such as ParGram, where grammars are written as machine-processable rules paired with lexical entries and templates.

Scope: In one sentence, the paper introduces new Cantonese ParGram symbolic grammar resources and uses them, alongside English baselines, as gold standards in a controlled experiment testing whether two OpenAI models can generate usable machine-processable grammars from sentences and from target formal structures.

What This Paper Is About

Building a formal, machine-processable grammar for a language is slow, expert-intensive work: linguists write phrase-structure rules, lexical entries, and templates, and must keep many interacting constraints consistent with one another. The paper asks whether current large language models can meaningfully assist in that process rather than only producing text that looks like grammar. To answer this, the author treats existing Cantonese ParGram resources as gold standards, adds matching English baselines, and tests whether two OpenAI models can generate grammars under deliberately varied prompting conditions. The goal is to characterize both what LLMs can contribute and where they break down, in order to judge whether they belong in an expert grammar-engineering workflow.

Key Contributions

1. New Cantonese ParGram resources. The paper contributes new symbolic grammatical resources for Cantonese within the ParGram framework, which serve both as a linguistic resource in their own right and as the gold standard for the evaluation.

2. A controlled experimental paradigm for LLM grammar engineering. Rather than testing models in an open-ended way, the study fixes a controlled setup with systematically varied prompting conditions, allowing different models, input types, and prompt conditions to be compared against the same reference grammars.

3. English baselines for comparison. Corresponding English resources and baselines are used so that results on Cantonese can be interpreted against a better-resourced language rather than in isolation.

4. A capability and limitation profile of LLMs for grammar engineering. The paper characterizes where the models succeed and where they fail, and draws out implications for AI-assisted expert workflows rather than presenting LLMs as autonomous grammar writers.

Main Findings

• One model clearly outperformed the other. GPT-5.4 outperformed gpt-oss-120b in the grammar-generation task. The abstract does not report the size of that gap or the metric used.

• Input type mattered. Grammars generated from target formal structures generally outperformed grammars generated from sentences. In other words, giving the model the formal target to work toward was more effective than giving it natural-language sentences alone.

• Local plausibility was achievable. Both models could produce locally plausible phrase-structure rules, lexical entries, and templates. Individual pieces of grammar output looked reasonable.

• Global coordination was the weak point. The models often struggled to coordinate interacting formal constraints, and this was especially pronounced in multi-construction settings, where several grammatical phenomena must be made consistent at once. This is the central limitation the paper identifies.

• Prompting conditions were varied systematically. The study was designed so that prompting was a controlled variable, rather than an incidental detail — the abstract does not specify the individual prompt conditions or their relative effects.

• Human expertise remains central. The paper's conclusion is that LLMs may support intermediate stages of grammar development, but human linguistic expertise stays essential for analysis, validation, and refinement.

Methodology in Plain English

The researcher took an existing body of hand-built Cantonese grammar resources — rules, lexical entries, and templates in the ParGram style — and treated them as the correct answer key. Equivalent English resources were used as baselines. Two OpenAI models, gpt-oss-120b and GPT-5.4, were then asked to generate machine-processable grammars under a set of prompting conditions that were varied on purpose rather than ad hoc. In some conditions the models were given natural-language sentences as input; in others they were given the target formal structures — the formal representations the grammar is supposed to relate to. The generated grammars were then compared against the gold-standard resources to see whether the models produced rules and entries that were not only individually plausible but also mutually consistent. The abstract does not describe the scoring scheme, the number of test items, or the exact prompt variants; it reports only the comparative outcomes noted above.

Why This Matters

Impact on research. Grammar engineering is one of the most knowledge-intensive tasks in computational linguistics, and claims about LLMs "doing linguistics" are often made loosely. This paper tests the claim in a controlled setting with a gold standard, and its key negative result — that models can produce locally plausible rules but fail to coordinate interacting constraints — is a useful caution for anyone proposing LLM-driven grammar induction or formal-grammar generation. The release of new Cantonese ParGram resources also expands the pool of languages with symbolic grammar coverage, which matters for a language with substantial speaker numbers but comparatively limited formal grammatical infrastructure.

Real-world applications (as supported by the paper's framing):

• Assisted grammar authoring: LLMs could draft candidate rules, lexical entries, or templates that a linguist then checks and refines, shortening the early, repetitive stages of building a grammar.

• Low-resource language documentation: The Cantonese setting is a case study in what happens when gold-standard symbolic resources exist but are scarcer than for English — relevant to documentation efforts for other underrepresented languages.

• Language technology pipelines: Formal grammars feed into parsing, generation, and linguistic annotation tools; better or faster grammar development can support those downstream applications.

• Benchmarking and model evaluation: The controlled paradigm offers a template for evaluating generative models on tasks where a precise, checkable gold standard exists, rather than on subjective text quality.

Industry relevance. The finding that models are useful for intermediate production but not for final validation maps directly onto how AI is realistically deployed in expert tooling: as a draft generator inside a human-reviewed workflow. Companies building developer or linguistic tooling should read this as evidence against fully automated grammar generation and in favor of systems that surface candidate structures for expert confirmation.

Future Directions

• Whether targeted training or prompting can address constraint coordination. The paper's main failure mode is coordinating interacting formal constraints in multi-construction settings; whether better prompting, decomposition into smaller sub-tasks, or fine-tuning on grammar resources can mitigate this is left open.

• How far these results extend beyond the two models tested. Only gpt-oss-120b and GPT-5.4 were evaluated, so the generality of the capability and limitation profile across other models and future releases is an open question.

• Transfer to other languages and other grammar frameworks. The study pairs Cantonese with English baselines; whether the same pattern of local success and global failure appears for other languages or other constraint-based formalisms is untested here.

• Designing the human-in-the-loop workflow. The paper argues for LLMs as support for intermediate stages with humans handling analysis and validation, but the specific division of labor — which stages to automate, how to present candidates to experts — is a design question the abstract raises without resolving.

Target Audience

Computational linguists and grammar engineers working with ParGram or similar constraint-based formalisms; NLP researchers interested in evaluating LLMs on tasks with formal, checkable gold standards; researchers and developers working on Cantonese and other under-resourced languages; and practitioners designing AI-assisted expert tools who want grounded evidence about where model assistance is reliable and where it is not. Readers seeking quantitative benchmark numbers should note that the abstract reports comparative outcomes only — specific metrics, dataset sizes, and prompt conditions are not given in it.

Authors’ abstract

This paper presents new Cantonese ParGram resources and evaluates LLMs for knowledge-driven grammar engineering within a controlled experimental paradigm. Using Cantonese ParGram resources as gold standards, with corresponding English baselines, we investigate whether OpenAI's gpt-oss-120b and GPT-5.4 can generate machine-processable grammars from sentences and target formal structures under systematically varied prompting conditions. GPT-5.4 outperformed gpt-oss-120b, while grammars generated from target formal structures generally outperformed those generated from sentences. Although both models could generate locally plausible phrase-structure rules, lexical entries, and templates, they often struggled to coordinate interacting formal constraints, especially in multi-construction settings. The results characterize both the capabilities and limitations of current LLMs for potential integration into AI-assisted expert workflows: LLMs may support intermediate stages of grammar development, but human linguistic expertise remains central to analysis, validation, and refinement. The study also contributes new Cantonese symbolic grammatical resources.