Grammar Control for Language Learning with Sequential Monte Carlo and Small Language Models

AI summary

Overview

Research area: Natural Language Processing, specifically constrained text generation for second-language (L2) pedagogy and inference-time control of small language models.

Technical level: Intermediate. The paper relies on probabilistic inference concepts (posterior distributions, importance weights, particle resampling) and assumes some familiarity with large language model decoding, but the core argument is stated in plain pedagogical terms.

Scope: The paper shows that sequential Monte Carlo (SMC) steering, driven by interpretable grammar "potential functions," lets language models under 10 billion parameters produce target grammar constructions at rates that beat much larger models and prior fine-tuning approaches.

What This Paper Is About

Second-language learners need exposure to grammar that matches their proficiency level, and an interactive tutor model should be able to reliably produce those target constructions. Frontier models can usually do this through prompting, but small, on-device models often fail. The paper treats grammar control as a constrained generation problem and uses SMC at inference time to steer small models toward specific grammar skills drawn from the English Grammar Profile (EGP), a pedagogical resource listing 1,222 grammar skills tied to CEFR proficiency levels.

Key Contributions

1. SMC steering with hand-coded potentials raises grammar accuracy dramatically in tiny models. A Qwen 2 0.5B model goes from 58% to 98% target grammar coverage when the particle budget increases from 1 to 8, exceeding the 96% reported for Llama 3.3 70B — a model 140 times larger.

2. SMC composes with additional pedagogical constraints. The authors add a vocabulary potential restricted to approximately 3,400 A2-level word forms (from a Cambridge English Profile resource) and show that higher particle budgets recover accuracy lost to the restriction.

3. Frontier models can synthesize grammar potentials via self-steering. Using GPT-5.4 as a Planner model, potentials are generated and iteratively revised (up to 5 iterations), then used to steer Llama 3.1 8B. With 8 particles, SMC outperforms prior fine-tuning and guided-decoding approaches on combinations of up to 6 grammar targets.

4. The approach scales to the full EGP. Potentials were generated for 1,222 EGP skills, with viable potentials produced for 99% (only 4 constraints excluded for nontrivial errors), requiring no supervised data and no additional training of the base model.

Main Findings

• Small models can beat large ones at grammar control. Llama 3.1 8B reaches ceiling accuracy with template prompting alone, but the largest gains appear in the smallest models: Qwen 2 0.5B climbs from 58% to 98% as particles go from 1 to 8, surpassing the 96% best reported result from Llama 3.3 70B.

• Prior prompting baselines were weak for small models. In the replicated setup, Llama 3.1 8B rises from 39% accuracy with Basic prompting to 78% with Template prompting, which SMC then improves on.

• Vocabulary restriction costs accuracy that particles can buy back. Qwen 2 0.5B reaches 95% with 4 particles and no vocabulary restriction; this falls to 85% when the A2 vocabulary restriction is added, but returns to 94% at 8 particles.

• Quality improves with particle budget, but mostly under restriction. Higher particle budgets improve response quality across the board, with the effect more pronounced for A2-vocabulary-only generations. SMC generations score high on Grammatical Correctness (4.45–4.50) but lower on Appropriateness (around 3.94–4.12) than the prior hybrid approach (3.45 on Appropriateness, 2.02 on Grammatical Correctness).

• Speed remains viable up to moderate particle counts. Generation stays above 100 words per minute up to 32 particles, with the exception of Llama 3.1 8B, which drops to 60 words per minute at 32 particles. Learners' estimated reading speed of 70 to 90 words per minute is used as the lower bound for acceptability.

• SMC outperforms fine-tuning, guided decoding, and a hybrid baseline. Against Glandorf et al.'s Llama 3.1 8B results, SMC's mean accuracy across constraint combinations is 67.5 at 4 particles (vs. 56.2 for fine-tuning), 71.6 at 8 particles (vs. 69.1 for guided decoding), and 78.1 at 32 particles (vs. 68.5 for the hybrid).

• Gains concentrate where multiple constraints must be met simultaneously. At 6 constraints, SMC reaches 68.2 (32 particles) versus 44.7 for fine-tuning, 65.3 for guided decoding, and 59.5 for the hybrid.

• Automated potentials approach a small frontier model's own output. Using GPT-4o-mini–generated potentials, Llama 3.1 8B with 16 particles scores comparably to GPT-4o-mini one-shot generation across CEFR levels (e.g., 86.0 vs. 84.1 at A1; 65.9 vs. 73.8 at C2). With GPT-5.4–generated potentials, SMC reaches 88.7 at A1, 81.6 at B1, and 74.6 at C2, remaining below GPT-5.4's own one-shot scores of 95.4, 96.5, and 95.8 respectively.

• Evaluation was validated across judges. GPT-5-nano was the primary grammar judge; Claude Haiku 4.5 showed 94.5% agreement on Experiment 1. Human crowdworkers (11 raters for grammar, 13 for quality) judged a random subset of 99 SMC generations, with results broadly consistent with the automated pipeline.

Methodology in Plain English

The researchers frame grammar control as sampling from a modified distribution: take a language model's own probabilities over text and multiply by a "potential" — a binary check that returns 1 if the text contains the target grammar construction and 0 otherwise. The normalizing constant for this distribution is intractable because it sums over all possible strings, so they approximate it with sequential Monte Carlo.

SMC generates many candidate continuations in parallel, called particles. At each step, particles are extended by the language model, reweighted according to how promising they look, and occasionally resampled so computation concentrates on strong partial strings. At the end, the top-weighted particle is taken as the output.

Potentials are implemented in the GenLM Control library (v0.3.1) as short Python programs with two methods. A prefix method runs at every generation step to give a cheap incremental signal (for instance, lowering a penalty when a regular expression detects the word "enjoy"). A complete method runs once at the end and performs strict parsing with spaCy (v3.8.14) to confirm the full construction, such as a verb gerund following "enjoy" rather than a noun.

In Experiment 1, five grammar skills are hand-coded as potentials and tested on Llama 3.1 8B, Llama 3.2 1B, Qwen 2 1.5B, and Qwen 2 0.5B, replicating a prior prompting study's 2 × 125 generations per model. In Experiments 2 and 3, potentials are instead written by a frontier Planner model through "self-steering": the Planner writes a program, tests it by steering a small model, and regenerates up to 5 times if the target grammar does not appear. A script fixes common spaCy handling errors automatically, and in Experiment 2 eight consistently low-accuracy skills were manually revised.

Comparisons are made against two baselines with matched inference budgets: standard decoding with best-of-M selection, and rejection sampling that keeps only complete generations passing the grammar check.

Why This Matters

For research, the paper demonstrates that explicit, human-readable programs can serve as a practical interface for controlling language model output, and that inference-time compute can substitute for model scale and for fine-tuning data. It also provides a large-scale stress test of self-steered potential generation across 1,222 targets.

Real-world applications include:

• On-device language tutoring, where a small model with SMC steering can produce level-appropriate grammar without cloud APIs.
• Curriculum-aligned content generation, where instructors or curriculum designers inspect and edit potentials to match specific pedagogical goals.
• Vocabulary-controlled material generation, using the composed vocabulary potential to keep text within a learner's known word list.
• Low-resource and offline learning settings, where access to frontier-model APIs is limited or costly.

Industry relevance centers on cost and deployment: replacing calls to large frontier models with a small local model plus additional inference-time computation broadens access and reduces reliance on cloud infrastructure. The interpretability of potentials also gives developers a debugging surface that neural classifiers do not provide.

Future Directions

• Multi-turn evaluation and user studies. The current work evaluates single dialogue turns; confirming pedagogical value requires extended interactions and a full study with actual language learners.

• Inference efficiency. At 32 particles, Llama 3.1 8B falls below the acceptable reading speed, motivating better proposal distributions or particle pruning.

• Beyond English. The approach depends on spaCy and the EGP, both predominantly English resources; other languages would need equivalent tooling and pedagogical frameworks.

• Reducing human intervention in self-steering. Generated potentials still require programmatic and manual repair, and improving the generation pipeline is identified as necessary for more efficient scaling.

Target Audience

Researchers and practitioners in NLP working on constrained decoding, inference-time control, or small-model deployment will find the core contribution most relevant. The paper also speaks directly to developers of educational technology and to applied linguists or curriculum designers interested in how the English Grammar Profile can be operationalized computationally. Readers seeking a rigorous treatment of SMC itself should note that the appendix defers to prior work (Lew et al., 2023; Loula et al., 2025; Lipkin et al., 2025) for fuller coverage.

Authors’ abstract

Second language learners benefit from exposure to level-appropriate grammar in interactive language use. While prompted frontier language models can usually produce target grammar structures, less resource-intensive models that can be run on device often fail in this task. We use sequential Monte Carlo (SMC) to steer dialogue turn generation in small lan- language models toward target grammar constructions from an expert-developed pedagogical resource. With a minimal set of hand-coded potential functions, SMC achieves near-perfect coverage of target grammar, enabling small models to outperform state-of-the-art results from models 140 times larger. We also use code synthesis with frontier models to generate new potentials, showing that our method scales to hundreds of target grammar skills with no need for additional training or supervised data.