HSKBenchmark: Modeling and Benchmarking Chinese Second Language Acquisition in Large Language Models through Curriculum Tuning

AI summary

Overview

Research area: Natural Language Processing / computational second language acquisition (SLA) — specifically Chinese as a second language — combined with LLM benchmarking and curriculum-based fine-tuning.

Technical level: Advanced. The paper assumes familiarity with fine-tuning (LoRA, causal language modeling), curriculum learning, and linguistic complexity metrics.

Scope: The paper builds HSKBenchmark, a level-structured Chinese SLA benchmark, and uses it to train and assess LLMs through a staged curriculum that mirrors how human learners progress from HSK level 3 to HSK level 6.

What This Paper Is About

Studies of how humans acquire a second language are hard to run because researchers cannot ethically control what learners read and hear. LLMs offer a controllable, reproducible substitute, but no systematic Chinese SLA benchmark existed for modeling acquisition phase by phase. This paper fills that gap by assembling HSK-level Chinese learning materials, training LLMs on them in a beginner-to-advanced sequence, and evaluating the models' writing along five linguistic dimensions.

Key Contributions

1. HSKBenchmark, described by the authors as the first benchmark for staged modeling and writing assessment of LLMs in Chinese SLA, covering HSK levels 3 to 6 with 6.76M tokens of authentic textbooks, 16,462 synthetic instruction items, 591 grammar items, and 30 test topics drawn from real HSK exams.
2. A curriculum-tuning framework that pretrains an LLM on level-specific textbooks and then instruction-tunes it on level-specific writing exercises, proceeding in ascending order from HSK 3 to HSK 6 to simulate a human acquisition trajectory.
3. A linguistically-grounded evaluation system spanning five dimensions — coverage of level-based grammar items, writing errors, lexical complexity (MATTR-50), syntactic complexity (MDD), and holistic scoring — plus HSKAgent, an automated evaluator built on Qwen3-8B and fine-tuned on the grammar dataset and 10K compositions from human Chinese L2 learners.
4. Empirical evidence that curriculum-tuned models improve steadily across stages, reach writing performance on par with advanced human learners, and show human-like acquisition characteristics, including emergent use of grammar beyond the training data.

Main Findings

• Human baselines set the upper anchor. Chinese native speakers achieve the highest overall score at 88.3333. Learners scoring 95* and 90* also exceed 80 but still show a noticeable gap from natives in writing errors and syntactic complexity. Scores rise monotonically as learner proficiency increases from 60* (63.0000) to 95* (85.0000), which the authors treat as evidence of a predictable developmental trend that HSKAgent captures reasonably.
• Curriculum tuning lifts weaker models substantially. Base LLaMA2 scores 70.0000, roughly equivalent to Learners-70*. LLaMA2 after HSK3 improves by 5.83 points (to 75.8333), and LLaMA2 after HSK6 reaches 81.8333, on par with Learners-90*. Chinese-Alpaca climbs from 77.5000 to 85.6667 and Mistral from 76.8333 to 85.3333 by HSK6.
• Strong closed models score high but are not human-like everywhere. GPT-4.1-mini, DeepSeek-Chat, and Gemini-2.5 obtain average scores exceeding 90 (91.5000, 92.3333, and 90.5300 respectively) but are inferior to humans in syntactic complexity and mastery of advanced grammar items.
• LLMs make far fewer writing errors than humans. GPT-4.1-mini, DeepSeek-Chat, and Gemini-2.5 record 0.0000 writing errors, versus 1.4000 for natives and 2.8667–4.8000 for learner groups. The authors attribute this to differing production mechanisms: humans take risks with uncertain usages, while LLMs stick to high-confidence tokens under top-k next-token prediction.
• LLMs trail humans in lexical and syntactic complexity. LLMs optimize for predictive likelihood and produce shorter, more typical sentences, whereas L2 learners deliberately deploy complex structures in tests. Mistral's MATTR-50, for example, rises from 0.5260 in the base model to 0.7886 after HSK6, still below native 0.8061.
• Curriculum order matters more than data volume. In an ablation where level-based textbooks and instruction data were shuffled and split into four stages purely by data volume, the shuffled models scored relatively higher in early stages, likely because they saw high-level data prematurely. Models trained with the ordered curriculum surpassed them in stages 3–4, which the authors connect to Krashen's i+1 input hypothesis.
• Advanced-level grammar emerges without direct training. Models at HSK5 and HSK6 produce a higher proportion of advanced grammar items not included in training data than HSK3 and HSK4 models do. For LLaMA2, HSK3 coverage is 49.25% (HSK3 grammar) and 17.38% (HSK4 grammar); after HSK4 training the HSK3 figure falls 4.08% and the HSK4 figure rises 3.10%, indicating a shift toward more complex grammar.
• No first-language forgetting, slight Chinese gains. Llama2 evaluated on MMLU and C-Eval shows no degradation in L1 (English) performance and slight L2 improvements on C-Eval.
• HSKAgent is a reliable automated rater. It reaches an F1 of 0.97 for binary classification of grammar items, 90% accuracy for error detection, and an F1 of 0.81 for holistic scoring, with agreement to human raters of QWK = 0.7969, Spearman = 0.8010, and Pearson = 0.8023.

Methodology in Plain English

The team collected 79 widely used Chinese-language textbooks organized by HSK level, covering levels 3 through 6. They stripped out images, Pinyin, and English glosses, leaving 6.76M tokens across 162,074 sentences (41.74 tokens per sentence on average). Because real teaching materials run short at lower levels, they supplemented the textbooks with synthetic writing exercises: they took 591 grammar items from the Chinese Proficiency Grading Standards, spanning six categories (word, phrase, fixed format, sentence component, sentence type, emphatic usage), and prompted GPT-4.1-mini, DeepSeek-Chat-V3, and Gemini-2.5-Flash to generate 10 instruction instances per item using two-shot in-context learning. A random sample was checked by three trained graduate annotators, yielding a Fleiss's Kappa agreement score of 0.91 and a 95% validity rate, and after proofreading and filtering, 16,462 instruction items remained.

Training follows a two-step loop per level. The model first undergoes causal-language-modeling pretraining on that level's textbook sentences, then instruction tuning on that level's prompt–completion pairs, then moves to the next level up. Level 3 comes first, level 6 last. For evaluation, the researchers drew 30 writing topics from the HSK Dynamic Composition Corpus v2.0, covering genres such as narrative and argumentative writing, and confirmed there is no overlap or contamination with the training data. Each composition is scored on five dimensions. To make scoring automatic rather than dependent on human teachers, they fine-tuned Qwen3-8B into HSKAgent using a binary classification setup built from the instruction data plus reconstructed human compositions with error annotations and scores. Complexity-related indices are computed through function calling rather than by the model directly.

Experiments ran on PyTorch 2.6.0 with 3 RTX 3090 GPUs (24GB) using LLaMA-Factory, with LoRA for both pretraining and instruction tuning, a learning rate of 5e-5, 3 epochs, and bf16 compute.

Why This Matters

The work supplies a reusable, level-annotated Chinese SLA testbed where none existed, and it gives LLM interpretability research a controlled setting for studying whether acquisition patterns can be induced and observed in a machine. Because the experiment is reproducible, it offers an alternative to studies that would require manipulating human learners' input. The reported finding that models generalize to advanced grammar items absent from training also speaks to broader debates about emergence and generalization in LLMs.

Real-world applications:

• Automated Chinese writing assessment. HSKAgent's 90% error-detection accuracy and F1 of 0.81 on holistic scoring point to lower-cost, higher-throughput grading of learner compositions than relying on human teachers.
• Adaptive language-learning platforms. The level-based textbook splits and grammar-item inventories can drive systems that sequence content by proficiency rather than dumping mixed-difficulty material on learners.
• Curricula and test design. The 591 level-tagged grammar items and the observed developmental trends can inform how HSK-aligned courses and practice materials are staged.
• Multilingual extension. The results on MMLU and C-Eval suggest the approach could transfer to other frameworks such as CEFR for languages beyond Chinese and English.

Industry relevance: the pipeline is directly applicable to edtech products that need proficiency estimation and feedback at scale, and the pairing of a generative model with a specialized evaluator agent is a template for building domain-specific assessment tooling on top of general LLMs.

Future Directions

1. Scaling beyond Chinese and English. The authors plan to extend the SLA modeling framework to a broader range of languages, potentially using frameworks like CEFR.
2. Incorporating multimodal inputs. Images were deliberately removed from this study because multimodal input was out of scope, leaving visual learning materials as an open extension.
3. Integrating additional linguistic dimensions. The current evaluation covers five dimensions; the authors propose adding more to better capture SLA development.
4. Connecting model behavior to SLA theory. Whether curriculum-tuned LLMs can substantiate or refine existing second-language acquisition theories remains an open empirical question, as does the extent to which their trajectories genuinely mirror human acquisition.

Target Audience

Researchers in NLP and computational linguistics working on LLM interpretability, acquisition modeling, or benchmark construction; second-language acquisition researchers and psycholinguists interested in machine-based verification of acquisition theories; Chinese-language educators and test developers seeking proficiency-graded resources and automated scoring; and applied scientists in edtech building level-aware language learning or writing assessment systems. Readers should be comfortable with fine-tuning terminology and linguistic complexity metrics.

Authors’ abstract

Language acquisition is vital to revealing the nature of human language intelligence and has recently emerged as a promising perspective for improving the interpretability of large language models (LLMs). However, it is ethically and practically infeasible to conduct experiments that require controlling human learners' language inputs. This poses challenges for the verifiability and scalability of language acquisition modeling, particularly in Chinese second language acquisition (SLA). While LLMs provide a controllable and reproducible alternative, a systematic benchmark to support phase-wise modeling and assessment is still lacking. In this paper, we present HSKBenchmark, the first benchmark for staged modeling and writing assessment of LLMs in Chinese SLA. It covers HSK levels 3 to 6 and includes authentic textbooks with 6.76 million tokens, 16K synthetic instruction samples, 30 test topics, and a linguistically grounded evaluation system. To simulate human learning trajectories, we introduce a curriculum-tuning framework that trains models from beginner to advanced levels. An evaluation system is created to examine level-based grammar coverage, writing errors, lexical and syntactic complexity, and holistic scoring. We also build HSKAgent, fine-tuned on 10K learner compositions. Extensive experimental results demonstrate that HSKBenchmark not only models Chinese SLA effectively, but also serves as a reliable benchmark for dynamic writing assessment in LLMs. Our fine-tuned LLMs have writing performance on par with advanced human learners and exhibit human-like acquisition characteristics. The HSKBenchmark, HSKAgent, and checkpoints serve as foundational tools and resources, with the potential to pave the way for future research on language acquisition modeling and LLMs interpretability. Code and data are publicly available at: https://github.com/CharlesYang030/HSKB.