Anka: A Domain-Specific Language for Reliable LLM Code Generation

AI summary

Overview

Research area: Large language model code generation and domain-specific language (DSL) design, with the work classified under Natural Language Processing (cs.CL).

Technical level: Intermediate. The paper assumes familiarity with LLMs, prompting, and basic programming concepts, but its argument and results are presented at a level accessible to readers without deep compiler or ML-systems background.

Scope (one sentence): The paper introduces Anka, a small purpose-built DSL for data transformation pipelines, and tests whether constraining the target language improves LLM code generation accuracy relative to Python.

What This Paper Is About

LLMs are good at generating code for isolated tasks but make systematic mistakes on complex, multi-step programs — particularly errors in variable handling, operation ordering, and state tracking. The authors hypothesize that these errors come from the flexibility of general-purpose languages like Python, which offer many syntactically valid ways to express the same computation and therefore many opportunities to choose wrong. Their goal is to test whether designing a language with exactly one canonical form per operation — a language built for LLM generation rather than human ergonomics — reduces those errors.

Key Contributions

1. Anka, a DSL for data transformation pipelines, designed around four principles: one canonical form per operation, named intermediate results via INTO clauses, explicit STEP block structure, and verbose English keywords instead of symbols.
2. A benchmark suite of 100 data transformation tasks spanning eight categories (filter, map, aggregate, strings, multi_step, finance, hard, adversarial), each with a natural-language description, an input schema with field names and types, and test cases with input data and expected output.
3. A demonstration that LLMs can acquire a novel DSL purely from in-context prompts, with Claude 3.5 Haiku reaching 99.9% parse success on Anka despite zero training exposure to it.
4. Evidence that constrained syntax helps most at higher complexity, with a reported 40 percentage point accuracy advantage over Python on multi-step tasks, validated across two model families (Claude 3.5 Haiku and GPT-4o-mini).
5. Release of the complete implementation, benchmark suite, and evaluation framework at https://github.com/BleBlo/Anka, which the paper states comprises approximately 6,400 lines of code, a formal grammar with 98 production rules, 68 AST node types, a tree-walking interpreter covering all 18 operations, and 322 unit tests.

Main Findings

• Novel DSL acquisition from prompts alone: Despite having zero prior training exposure to Anka, Claude 3.5 Haiku achieves 99.9% parse success on the benchmark.
• Multi-step advantage: Anka reaches 100.0% task accuracy on multi-step tasks versus 60.0% for Python, a +40.0 percentage point difference. These tasks require maintaining state across 3–5 sequential operations.
• Overall improvement: Anka achieves 95.8% overall task accuracy versus 91.2% for Python (+4.6 percentage points) on Claude 3.5 Haiku, despite Python's substantial training data advantage.
• Cross-model validation: GPT-4o-mini shows 86.7% on Anka versus 60.0% on Python (+26.7 percentage points) on multi-step tasks. The paper notes Python accuracy is identical (60.0%) across both models, suggesting systematic difficulty rather than model-specific noise.
• Category-level detail (Claude 3.5 Haiku): Anka and Python tie at 100.0% on aggregate, map, and strings; Anka leads on multi_step (+40.0) and finance (+5.0); Python leads on filter (100.0% vs 96.7%, −3.3) and hard (100.0% vs 90.0%, −10.0).
• Error pattern analysis of failing Python generations: The paper attributes failures to variable shadowing (42% of errors), operation sequencing (31%), and chaining confusion (27%).
• Complexity dependence: The reported advantage is 0% on simple tasks (1–2 operations), +5% on medium tasks (3–4 operations), and +40% on complex tasks (5+ operations).
• Where Anka does not help: The authors state Anka shows no advantage on simple tasks and a slight disadvantage on "hard" tasks, which involve nested conditionals, edge-case handling, and domain-specific reasoning.
• Stated mechanisms: Reduced decision space (the paper illustrates this as reducing 3^5 = 243 possible 5-step programs to 1), explicit state management through named intermediates, and structural scaffolding from STEP blocks.

Methodology in Plain English

The researchers built a small language, Anka, in which every data operation has exactly one written form. A filter must be written as FILTER source WHERE condition INTO target; no alternative syntax exists. Every intermediate result must be given a name, and operations are grouped into named STEP blocks. Input tables declare typed schemas using types such as INT, STRING, DECIMAL, BOOL, DATE, and DATETIME. The language covers 18 operations grouped into selection (FILTER, SELECT, DISTINCT), transformation (MAP, RENAME, DROP, ADD_COLUMN), aggregation (AGGREGATE with COUNT, SUM, AVG, MIN, MAX), ordering (SORT, LIMIT, SKIP, SLICE), combination (JOIN, LEFT_JOIN, UNION), and I/O (READ, WRITE, FETCH, POST), plus control flow constructs (IF/ELSE, FOR_EACH, WHILE, TRY/ON_ERROR). It is implemented in Python using Lark for parsing.

To test the hypothesis, the authors wrote 100 benchmark tasks across eight categories, then asked an LLM to solve each task twice: once in Anka and once in Python. The two prompts were kept structurally identical — same task description, same input schema, same expected output format — with the Anka prompt adding roughly 100 lines of syntax documentation to teach the language from scratch, while the Python prompt assumed pandas knowledge consistent with training data. They generated 10 samples per task per language at temperature 0.3, and used task accuracy — defined as the fraction of tasks where at least 50% of samples produce correct output — as the primary metric, alongside parse success, execution success, and output correctness. Claude 3.5 Haiku served as the primary model, with GPT-4o-mini used for cross-model validation.

Why This Matters

Impact on research. The paper reframes language design as an intervention for LLM reliability, alongside the more common levers of scale, fine-tuning, and decoding constraints. It distinguishes its approach from grammar-constrained decoding: rather than masking invalid tokens during generation, it constrains the target language itself while leaving standard decoding intact. It also provides a rare datapoint that a model can achieve near-native parse rates on a language it has never seen in training, learned entirely from a prompt.

Real-world applications.

• Data transformation and ETL pipelines, where the paper's benchmark categories (filtering, aggregation, joins, ordering) map directly onto common production work.
• Finance-domain calculations, one of the benchmark categories where Anka showed a reported +5.0 percentage point advantage.
• Agentic or automated coding systems, where errors compound across sequential steps and the paper's multi-step results are most relevant.
• Internal tooling where a constrained, verbose language could act as an intermediate representation that is easier to validate than free-form Python.

Industry relevance. The paper's recommendation is specific: Anka is best suited for structured pipelines with 3+ sequential operations and standard transformation patterns, and Python's flexibility remains an asset for complex conditional logic. The authors also note that LLM-generated code should be reviewed before production deployment.

Future Directions

• Broader model coverage: The paper acknowledges that only two models (Claude 3.5 Haiku, GPT-4o-mini) were evaluated, and states that evaluation on additional model families would improve confidence.
• Fine-tuning comparison: No fine-tuned Anka model was tested, so the ceiling of Anka performance relative to prompt-only learning remains unclear.
• Generalization beyond data transformation: The benchmark focuses on data transformation pipelines, and the paper explicitly states that generalization to other programming tasks is not established.
• Human and production evaluation: No user study was conducted, leaving open whether developers find Anka code readable, and the authors list production deployment evaluation and extension to other domains such as financial calculations and workflow automation as future work.

Target Audience

This paper is most useful to researchers working on LLM code generation, program synthesis, and prompting methods; to DSL and language designers interested in how language design choices interact with model behavior; and to practitioners building code-generation agents or data pipeline tooling who want concrete evidence about where constrained syntax helps and where it does not. Readers looking for a large-scale empirical study should note the paper's own stated limitations: a single benchmark suite focused on data transformation, two evaluated models, and no user study.

Authors’ abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, yet they exhibit systematic errors on complex, multi-step programming tasks. We hypothesize that these errors stem from the flexibility of general-purpose languages, which permits multiple valid approaches and requires implicit state management. To test this hypothesis, we introduce Anka, a domain-specific language (DSL) for data transformation pipelines designed with explicit, constrained syntax that reduces ambiguity in code generation. Despite having zero prior training exposure to Anka, Claude 3.5 Haiku achieves 99.9% parse success and 95.8% overall task accuracy across 100 benchmark problems. Critically, Anka demonstrates a 40 percentage point accuracy advantage over Python on multi-step pipeline tasks (100% vs. 60%), where Python's flexible syntax leads to frequent errors in operation sequencing and variable management. Cross-model validation with GPT-4o-mini confirms this advantage (+26.7 percentage points on multi-step tasks). Our results demonstrate that: (1) LLMs can learn novel DSLs entirely from in-context prompts, achieving near-native accuracy; (2) constrained syntax significantly reduces errors on complex tasks; and (3) domain-specific languages purposefully designed for LLM generation can outperform general-purpose languages on which the LLM has extensive training. We release the complete language implementation, benchmark suite, and evaluation framework to facilitate further research.