Can Large Language Models Anticipate Behavioral Responses to Social Policies? A Case of Pension Enrollment Prediction among China's Flexible Workers

AI summary

Overview

Research area: Applied natural language processing / computational social science — using large language models as tools for social-policy impact assessment, instantiated on pension enrollment among flexible workers in China.

Technical level: Intermediate. Readers should be comfortable with classification metrics (F1), prompting and knowledge injection, parameter-efficient fine-tuning (LoRA/SFT) and knowledge distillation. Familiarity with Probit regression and Chinese pension institutions helps but the paper defines its institutional terms.

Scope: The paper builds FlexPension-LLM, a domain-specialized LLM for predicting a two-stage pension-enrollment decision, trained via a pipeline called DKI-RDistill, and evaluates it against 17 baselines on one blind CHFS 2019 split plus four external household surveys and a 27-person expert study.

What This Paper Is About

Policymakers need to know how people will respond to social-policy changes before those changes are enacted, but econometric models extrapolate poorly to hypothetical scenarios and field pilots are expensive. This paper asks whether a large language model can be adapted into a policy-assessment tool that predicts individual-level behavioral responses, using the voluntary pension-enrollment decisions of China's flexible workers as the test case. The goal is a model that outputs both a prediction (participate or not, and which pension channel) and a rationale that can be checked against policy rules.

Key Contributions

1. FlexPension-LLM, described as the first domain-specialized large language model for the hierarchical pension-enrollment prediction task among flexible workers in China. It takes a survey-grounded worker profile, household context, participation history, hukou-province policy parameters, and Probit-derived indicators as input, and outputs the participation action, the insurance channel, and a structured rationale in JSON.

2. DKI-RDistill, a two-part adaptation framework. Domain Knowledge Injection (DKI) converts empirical priors and hukou-province pension rules into model-readable prompt cues; Rationale Distillation then preserves teacher-correct reasoning traces while regenerating teacher-error cases under the ground-truth label before LoRA/SFT adaptation.

3. Policy-grounded cue design, including three derived indicators: a resident-pension burden ratio, an employee-pension burden ratio, and household pension dependency, defined relative to local minimum contribution requirements and household capacity signals.

4. A broad validation program: blind testing on an isolated CHFS 2019 split, transfer to four external household surveys, component ablations, training-efficiency and decoding-robustness checks, boundary cases, and a blind expert evaluation of rationale quality.

Main Findings

• Blind-test performance: On the CHFS 2019 blind split, FlexPension-LLM obtains 0.9271 Action F1, 0.9383 Type F1, and 0.9316 Composite F1, ranking third among 18 evaluated models. Composite F1 is defined as 0.6 × Action F1 + 0.4 × Type F1.

• Beats its own teacher and most baselines: It exceeds the Claude Sonnet 4.5 teacher by +0.0107 Composite F1 [0.0005, 0.0212] and Qwen-ZS by +0.1445 [0.1223, 0.1675], and significantly outperforms 15 of 17 baselines, including Gemini 3.1 Pro Preview, Claude Sonnet 4.6, and DeepSeek V4 Pro (blind delta +0.0240 [0.0118, 0.0366]).

• Near-frontier, not first: Claude Opus 4.6 has slightly higher blind Action and Composite point estimates (0.9355 / 0.9367), and Correct-only SFT has the highest Type F1 (0.9492); both paired intervals cross zero, so the authors frame the result as a strong near-frontier claim rather than an outright win.

• Cross-survey generalization: Across CHFS 2017, CFPS 2018, CHIP 2018, and CLDS 2018 (500 sampled cases each), it averages 0.7549 Composite F1, improving over Qwen-ZS by +0.1170 [0.0793, 0.1534] and over the teacher by +0.0595 [0.0293, 0.0895]. Its range, 0.7471–0.7687, is narrower than Claude Opus 4.6's 0.7397–0.7953, which the authors cite as the narrowest range among the strongest systems.

• DKI is the biggest single lever on the teacher: DKI raises Claude's Composite F1 by +0.0668 [0.0607, 0.0731] on the CHFS 2019 all-sample evaluation and by +0.0623 [0.0467, 0.0782] on the blind subset. Applied to Qwen as a prompt-only intervention, the blind-subset gain is small with a confidence interval crossing zero.

• Rationale supervision splits the labor: FlexPension-LLM is higher than Correct-only SFT on the external average (+0.0171, interval crossing zero), while Correct-only SFT has the higher blind Type F1. The authors interpret this as label supervision capturing much of the channel boundary and regenerated rationales adding inspectable decision traces.

• Error structure shifts: Teacher-side DKI roughly halves blind-test errors (total 468 → 246). Student adaptation gives the largest correction relative to Qwen-ZS (total 698 → 218), sharply reducing false-participation and residual/channel errors and lowering missed-participation errors relative to the teacher; the only tradeoff is a modest increase in false-participation errors against the teacher (60 → 72).

• Training stability: Full-data training gives the highest Composite F1 among tested data scales (0.9316), a statistically significant but narrow gain over the 25% setting (+0.0116 [0.0014, 0.0221]); checkpoint 1800 has the highest point estimate (0.9394) with its comparison against the final checkpoint crossing zero. Across decoding temperatures 0.0, 0.2, 0.5, and 0.9, parse success remains 1.0 and all paired comparisons against the default 0.5 cross zero.

• Expert evaluation: In a blind study, 27 respondents with economics, public-administration, or insurance backgrounds rated 12 cases. FlexPension-LLM received the highest mean soundness (3.54) and completeness (3.46) ratings and the largest share of trust preferences, 157 of 324 choices (48.5%), versus 117 (36.1%) for the Claude Sonnet 4.5 teacher and 50 (15.4%) for Qwen-ZS.

• Supervision data composition: Of the 13,322 CHFS 2019 cases in the teacher-supervision pool, 11,839 (88.9%) were retained as teacher-correct traces and 1,483 (11.1%) regenerated under ground-truth labels; the distilled data were split into 10,657 training and 2,665 validation samples.

• A concrete boundary case: Case A has very low annual income but substantial household assets, a 0.03% resident-pension burden ratio, and 14 uninterrupted contribution years. Qwen-ZS predicts non-participation; FlexPension-LLM activates affordability and contribution-history cues and correctly predicts resident pension insurance participation.

Methodology in Plain English

The team starts by formalizing the problem as a two-layer decision: a flexible worker first decides whether to enroll at all, and, if they do, whether to choose the lower-cost Urban-Rural Resident Pension or the higher-benefit but self-funded Urban Employee Basic Pension. This yields three terminal labels: non-participation, resident pension insurance, and employee pension insurance.

They build the main dataset from CHFS 2019, constructing 15,672 flexible-worker samples (8,842 participants, or 56.4%; 6,830 non-participants, or 43.6%), and hold out 2,350 cases as a blind test set that is never used for prompt tuning, supervision construction, fine-tuning, or checkpoint selection.

Two Probit models — one for the participation margin, one for channel choice conditional on participation — identify which mechanisms matter (income, assets, household enrollment, pension receipt, contribution years, hukou, education). These mechanisms become prompt-side cues: the resident-pension burden ratio, the employee-pension burden ratio (each comparing the local minimum contribution to the strongest observed capacity signal, taking the maximum of individual income, per-capita household income, and per-capita net household assets), and household pension dependency (12 × monthly household pension income divided by household annual income). Practical affordability thresholds of 1% for the resident ratio and 25% for the employee ratio are used.

A strong teacher (Claude Sonnet 4.5, chosen via a 30-sample candidate screening) then generates a prediction and a rationale for each labeled case under this DKI prompt. Where the teacher matches the true label, the trace is kept; where it does not, the case is regenerated under the ground-truth label so the student does not imitate an incorrect decision. An open-weight MoE model, Qwen 3.5-35B-A3B, is then fine-tuned with LoRA/SFT on this distilled data, with base weights frozen and only low-rank adapters trained, minimizing a standard supervised negative log-likelihood over sequences containing both the rationale and the final label. Training used 8 × NVIDIA L20 (48 GB) GPUs with DeepSpeed ZeRO-3, LoRA rank 32 and alpha 64, 3 epochs, learning rate 1e-4, cosine schedule, global batch size 16, maximum length 4096, and roughly 4.5 hours, with outputs constrained to structured JSON. Comparisons use paired differences with 95% stratified paired-bootstrap confidence intervals over 10,000 resamples with seed 42.

Why This Matters

Impact on research. The paper argues for a specific division of labor between econometrics and LLMs: fitted equations supply population-level priors, while LLM adaptation turns those priors plus institutional rules into case-level, checkable decision traces. It also offers a reusable template for evaluation in policy settings — blind historical testing, out-of-domain transfer across four separate surveys, error-transition analysis, and blind expert rating of rationales rather than only accuracy.

Real-world applications.

• Ex ante evaluation of proposed changes to contribution floors, subsidy tiers, or local pension rules before they are enacted.
• Aggregate participation forecasting for government agencies or actuaries working with voluntary-coverage populations.
• Identifying which subgroups are likely to fall through coverage gaps, given affordability thresholds and hukou-linked access constraints.
• Auditing model or program decisions by inspecting rationales against written policy rules.

Industry relevance. The pipeline is deliberately cheap and reproducible — open-weight base model, LoRA adapters, roughly 4.5 hours on eight GPUs, public code and reproduction materials — which makes policy-domain specialization accessible without frontier-model budgets. It is also relevant to insurers, pension administrators, and consulting or actuarial firms that need structured, auditable outputs rather than free-text answers. The paper stresses that the model is for aggregate policy analysis, not individual enrollment advice, eligibility screening, or automated administrative decisions.

Future Directions

• Systematic policy counterfactuals. The authors state that future work should extend the framework to systematic policy counterfactuals, moving from prediction to structured what-if simulation of contribution thresholds, subsidies, and local rules.
• Larger expert-in-the-loop policy analysis. The expert study used 27 respondents and 12 cases; scaling expert evaluation and incorporating expert feedback into the pipeline is named as a direction.
• Transfer beyond China's institutions. The authors note the study is tied to China's pension institutions, household-survey labels, and available policy parameters, and that transferring it requires rebuilding the policy inputs and validation data.
• Open question on where the gains actually live. Prompt-side DKI helps the teacher substantially but a prompt-only application to Qwen shows only a small blind-subset gain with a confidence interval crossing zero; the mechanism by which DKI value transfers through supervision rather than directly is not fully resolved.

Target Audience

Applied NLP and machine-learning researchers working on domain specialization, knowledge distillation, and evaluation of LLMs in social-science settings; computational social scientists and econometricians interested in LLM-assisted policy analysis; and policy analysts, pension administrators, or actuaries who need a realistic picture of what a policy-response simulator can and cannot support. Readers primarily interested in methodological detail will find the Probit-to-prompt mapping and the error-filtered supervision scheme most useful, while readers focused on evidence will find the blind benchmark, four-survey transfer results, and expert evaluation the central material.

Authors’ abstract

Assessing the impacts of social policy changes is a widely acknowledged challenge for policymakers. Econometric methods can be unreliable when extrapolating to hypothetical scenarios, while field pilot programs are highly costly. In this paper, we propose using large language models (LLMs) as policy-assessment tools adapted from general-purpose models. We present FlexPension-LLM, the first domain-specialized large language model for a hierarchical pension-enrollment prediction task among flexible workers in China, and introduce DKI-RDistill, which injects policy-grounded cues into the prompt, including Probit-derived marginal effects and hukou-province pension rules. The method then uses LoRA/SFT to distill rationale-augmented supervision into an open-weight MoE student, with teacher errors corrected by regenerating those cases under ground-truth labels. On a CHFS 2019 blind split, FlexPension-LLM achieves 0.9316 Composite F1, surpassing its Claude Sonnet 4.5 teacher and 15 of 17 baselines, and is statistically indistinguishable from Claude Opus 4.6. Across four external surveys, it averages 0.7549 Composite F1 and shows the narrowest performance range among the strongest systems. Component analysis shows that gains come mainly from policy-grounded cue injection and error-filtered supervision, while rationales provide decision traces that can be checked against policy rules.