A Scaled-Up Empirical Study of Syntactic Language Models

AI summary

Overview

Research area: Natural Language Processing — specifically syntactic language models (SLMs), which generate text jointly with the syntactic tree structures behind it. The paper sits at the intersection of language-model pretraining, attention-mask design, and syntactic evaluation.

Technical level: Intermediate. The core ideas (autoregressive language modeling, attention masks, constituency parses) assume some familiarity with Transformer language models, but the paper is primarily an empirical comparison rather than a derivation-heavy methods paper.

Scope in one sentence: A controlled, large-scale comparison of one terminal baseline and 11 syntactic language model variants — spanning different tree linearizations and attention masks — trained at up to 1B parameters and roughly 100B terminal training tokens, and evaluated on 11 downstream tasks plus two syntactic benchmarks.

What This Paper Is About

Earlier work on syntactic language models showed that making a model predict sentence structure alongside words can improve syntactic generalization and downstream performance, but those studies mostly evaluated models up to 300M parameters, trained on up to 9B tokens, and tested on only two or three tasks. The design space of how trees are serialized into token sequences (linearization) and how attention masks encode structure also remained underexplored. This paper scales that comparison up and disentangles those two design choices, asking whether special structural attention is actually needed and whether explicit structure still pays off as models get larger.

Key Contributions

1. Scaled-up SLM evaluation. The authors scale evaluation to 1B parameters and approximately 100B terminal training tokens, across 11 downstream tasks and two syntactic benchmarks (SG and BLiMP).

2. Systematic design-space study. They examine tree linearization and attention design on BBC News at 100M (all 12 configurations, two pause-token controls, and Pushdown) and on deduplicated BBC News at 500M (eight configurations and two pause-token controls).

3. Comparison against pause-token baselines. They compare SLMs with pause-token controls (Pause-1 and Pause-2) to separate the benefit of explicit structure from the benefit of simply having extra token positions.

4. A reproducible framework and released code. The code is available at https://github.com/qlwpc/TG-Interpolation.

Main Findings

• Causal attention is sufficient. In both BBC settings, the tree models using standard causal attention (Tree and TGTree) match or exceed structural-mask variants, and they require no architectural modifications, remaining compatible with FlashAttention.

• Tree and TGTree lead on generation at 100M. Tree-100M and TGTree-100M reach the highest XSum means (22.84 and 23.03 R-AVG) versus Terminal-100M (22.02), TG-100M (20.15), and TGNomask-100M (21.91).

• Removing TG's post-composition mask helps. TGNomask improves XSum, BoolQ, and perplexity over TG in both BBC settings, consistent with Zhao et al. (2025). At dedup 500M, Tree and TGTree exceed TGNomask by more than 1.4 XSum points and TG by more than 2.8 points.

• Hybrid attention gives the best document perplexity but not the best generation. Mixing causal and structural heads (TGTree-Mix-TG) improves document perplexity from 14.709 to 14.482 at 100M and from 13.947 to 13.785 at dedup 500M — the lowest in each setting — and improves BoolQ by 0.98 points at 500M, but lowers XSum, SG, and BLiMP.

• Explicit structure beats extra token positions on syntax. Tree and TGTree exceed their corresponding pause controls on XSum, SG, BLiMP, and document perplexity. At 500M, both tree models gain more than 11 BLiMP points over the pause controls. BoolQ favors tree models at 100M but pause controls at 500M (by 0.21 and 0.43 points).

• Document perplexity improves with tree linearization. Tree and TGTree improve over Terminal at 100M (14.736 and 14.709 vs. 15.369) and dedup 500M (13.864 and 13.947 vs. 14.029). Tree-NoONT obtains 15.079 at 100M and 13.955 at 500M; Tree-Compress obtains 14.748 at 100M.

• Linearization details matter, and differently for different objectives. Removing opening non-terminals improves SG but lowers XSum in both settings. Doubling closing tokens gives the highest XSum mean in both experiments. At 100M, tripling them further improves SG and BLiMP over TGTree but lowers XSum, while merging consecutive closing tokens lowers SG. Tree-NoONT-100M has the highest SG score (85.38) among the 100M models. Shuffling structural tokens causes a substantial BLiMP decline (Tree-Shuffle-100M BLiMP 67.77, Doc-PPL 17.852).

• At 1B on FineWeb-Edu, the advantage narrows. TGTree achieves the highest average accuracy across the 11 tasks (55.88), exceeding Terminal (53.20) by 2.68 points, with significant gains on five tasks under paired bootstrap tests. Tree improves the average by only 0.36 points (53.56) and performs particularly poorly on HellaSwag.

• TGTree and Pause-2 are nearly tied downstream, but not on syntax. Their average accuracies are 55.88 vs. 55.80. TGTree is significantly better on BoolQ and Pause-2 significantly better on HellaSwag; differences on the other nine tasks are not significant at alpha = .05. Excluding BoolQ puts Pause-2 ahead (55.04 vs. 54.71).

• Tree representations retain a syntactic edge at 1B. Tree achieves the highest scores on both syntactic benchmarks (BLiMP 81.79, SG 89.95), exceeding Terminal (79.65 and 88.01) by 2.14 and 1.94 points. TGTree also improves both (BLiMP 80.98, SG 89.00). Pause tokens improve BLiMP over Terminal (80.83 and 80.36 vs. 79.65) but lower SG (86.54 and 86.59 vs. 88.01).

• Structural attention costs throughput. Historical H800 measurements show causal models at 62,175 token/s versus structural models at 37,975 token/s at 100M (1.6x), and 12,550 versus 8,316 at 500M (1.5x). The historical 100M mixing-head measurement was 3,375 token/s, approximately 18x slower than causal.

Methodology in Plain English

The team trained all models from scratch for one epoch using the same OLMo-based decoder-only Transformer family, varying only the input representation and attention mask. A standard causal language model sees only words; the syntactic models instead see a flat sequence that interleaves words with structural tokens — opening non-terminals like (S and (VP, and closing non-terminals like S) and VP) — that encode a constituency parse tree. Different variants change how many of those structural tokens are emitted, whether opening tokens are dropped, whether consecutive closing tokens are merged, and whether the structural tokens are shuffled while keeping the words in order.

On top of the representation, the team varied attention. The baseline causal attention leaves the mask untouched and relies on the token sequence alone. Structural attention (as in Transformer Grammars) uses hand-crafted masks so that a non-terminal composes its children and stacks context. Hybrid attention gives different attention heads different masks — half structural, half causal. The twelve configurations are the one terminal baseline plus 11 SLM variants.

The non-structural controls matter for interpretation: Pause-1 inserts one learned PAUSE token after each terminal (doubling sequence length, versus roughly 2.4x for the Lin-1 linearization), and Pause-2 inserts two (tripling it, versus roughly 3.2x for Lin-2). Tree-Shuffle breaks structure while preserving token count and word order. Pushdown is included at 100M as an architectural reference.

Training follows the Step Law for learning rate and batch size, with the actual training token budget of each representation (which differs because structural tokens lengthen sequences). Evaluation uses task-specific fine-tuning on XSum and BoolQ — five independent fine-tuning seeds, reported as mean ± sample standard deviation — plus the 11-task OLMES suite with completion/cloze scoring, and the SG and BLiMP syntactic benchmarks. Document-level perplexity is approximated by summing probabilities over up to 300 candidate parses from a CRF constituency parser, taking the highest-probability sequence as the fixed history for each completed sentence. Uncertainty on the FineWeb-Edu results is quantified with 10,000 paired bootstrap resamples.

Why This Matters

Impact on research. The paper separates two design choices — how trees are linearized and how attention is masked — that prior work typically coupled together. Its central result is that structural attention masks are not required for SLMs to work well, since plain causal attention with structural tokens is compatible with FlashAttention and matches or beats masked variants. It also introduces an important control condition: the fact that a 1B-parameter syntactic model no longer clearly beats a compute-matched pause-token baseline on downstream accuracy suggests the benefit of explicit structure may shrink with scale, which reframes how the field should report SLM gains.

Real-world applications suggested by the evaluated tasks:

• Summarization systems (XSum), where tree linearization gave the best ROUGE-1/2/L means.
• Question answering and reading comprehension (BoolQ), where tree models led at 100M.
• Commonsense and multiple-choice reasoning suites (HellaSwag, WinoGrande, ARC, PIQA, MMLU and others).
• Grammar-sensitive tooling and linguistic evaluation pipelines, since tree models retained a syntactic advantage on SG and BLiMP.

Industry relevance. Because the strongest-performing causal configurations need no custom attention kernels and stay FlashAttention-compatible, they can be adopted in standard pretraining stacks without new kernels. The throughput measurements — structural masks running at roughly 1.5x to 1.6x the cost of causal attention, and historical mixing-head runs at approximately 18x slower — give practitioners a concrete reason to prefer causal tree linearization when compute is the binding constraint. The caveat that a pause-token baseline can match an SLM on downstream averages at 1B is equally relevant for anyone budgeting parser preprocessing into a large training run.

Future Directions

1. Investigate why SLM advantages appear to shrink with scale. The paper reports that at 1B on FineWeb-Edu, TGTree and Pause-2 have nearly identical average downstream accuracy, and it calls for assessing structural benefits against the additional computation structural tokens introduce.

2. Test the proposed explanatory hypotheses directly. The implicit multi-task learning, compositional bottleneck, and adaptive computation accounts are stated as hypotheses; the authors note they would require interventions on internal representations or attention patterns. The compositional bottleneck hypothesis is explicitly stated as not directly tested.

3. Fill the evaluation gaps. The authors did not evaluate structural-mask models at 1B on FineWeb-Edu, and they note that the BBC and FineWeb-Edu experiments differ in data, token budgets, and optimization settings, so they do not isolate the effects of deduplication or scale.

4. Address preprocessing and generality limitations. Constructing linearized tree inputs requires a parser; the authors do not quantify the effect of parse quality and study only English. They also note that each pretraining configuration is represented by a single run, that pause-token controls match sequence lengths rather than exact FLOPs, and that public data and checkpoint downloads remain pending.

Target Audience

This paper is most useful to NLP and machine learning researchers working on language model pretraining and inductive biases, to computational linguists interested in how explicit syntax interacts with scale, and to engineers and practitioners who need to decide whether structural attention masks are worth their throughput cost. Readers who want a purely architectural or theoretical contribution will find this is an empirical study; readers comfortable with Transformer pretraining and constituency parsing will get the most from it.

Authors’ abstract

Syntactic language models (SLMs) that generate sentences along with their syntactic tree structures have shown promise in both syntactic generalization and downstream tasks. Prior studies, however, have mostly evaluated smaller models trained on less pretraining data and tested on narrower task suites. In addition, the design space of tree linearization and attention masks remains underexplored. We conduct a large-scale empirical study of 12 controlled configurations---one terminal baseline and 11 SLM variants spanning different linearizations and masks---at scales up to 1B parameters and 100B training tokens, evaluated across 11 downstream tasks and two syntactic benchmarks. The experimental results show that SLMs with standard causal attention match or exceed structural-mask variants across downstream and syntactic benchmarks. They require no architectural modifications and remain compatible with FlashAttention. We also observe that when scaling up to the 1B-parameter scale, the top-performing SLM variant no longer shows a clear advantage over a compute-matched non-structural baseline, suggesting that SLMs may become less competitive with scale. Our code is available at https://github.com/qlwpc/TG-Interpolation.