CASTLE: A Comprehensive Benchmark for Evaluating Student-Tailored Personalized Safety in Large Language Models

AI summary

Overview

• Research area: AI safety evaluation for large language models in education, specifically the intersection of LLM personalization and safety.
• Technical level: Intermediate. The paper is readable without deep technical background, but assumes some familiarity with LLM safety benchmarking, professional-personality frameworks from educational psychology, and LLM-as-a-judge evaluation.
• Scope: The paper introduces CASTLE, a bilingual benchmark of 92,908 scenarios that measures whether LLM responses are safe for a specific student profile rather than for a generic user.

What This Paper Is About

LLMs tend to give the same answer to the same prompt regardless of who is asking. In education, that "one-size-fits-all" behavior is dangerous, because a response that reassures one student may trigger harmful behavior in a student with different psychological or cognitive vulnerabilities. Existing safety benchmarks score outputs for context-independent properties such as factual accuracy, bias, or toxicity, so they cannot capture the fact that the same response causes different harm to different students. The paper defines this gap as Student-Tailored Personalized Safety and builds CASTLE, a benchmark grounded in educational theory, to measure it.

Key Contributions

1. A new evaluation perspective. The authors identify that the prevailing one-size-fits-all generation paradigm in educational LLMs overlooks student heterogeneity, and formalize this as Student-Tailored Personalized Safety.
2. A large-scale theory-grounded benchmark. CASTLE covers 15 educational safety risk domains organized into four categories and 14 student attributes, comprising 92,908 Chinese–English bilingual evaluation scenarios (53,483 Chinese and 39,425 English).
3. Three purpose-built evaluation metrics. Risk Sensitivity (detecting latent risks), Emotional Empathy (recognizing student emotional states), and Student Alignment (matching responses to student attributes), each rated on a 1–5 scale and averaged into an Average Safety Score.
4. A large empirical study. Evaluation of 18 state-of-the-art LLMs showing substantial deficiencies in personalized safety assurance, plus ablation evidence that personalization, explicit profile exposure, and reinforcement-learning-based training improve safety more than model scale.

Main Findings

• All 18 models fall below 2.5 out of 5. Under the Non-Personalized setting, no model reached an average safety score of 2.5, indicating substantial deficiencies in personalized safety assurance across open-source, closed-source, and education-based models.
• The best model still scores low. Claude-Haiku-4.5 achieves the highest score at 2.42 on the Chinese subset and 2.02 on the English subset. Education-based models InnoSpark-7B and MuduoLLM-7B reach 2.21 and 2.07 respectively. GPT-4o and GPT-5.2 both score 2.00, and Gemini-2.5-Flash scores 2.16.
• Domain-specific alignment can beat scale. InnoSpark-7B's Chinese Average Safety Score (2.21) outperforms GPT-4o (2.00), GPT-5.2 (2.00), and Gemini-2.5-Flash (2.16), suggesting that domain-specific alignment is more effective than model size alone for educational safety risks.
• Chinese scores are generally higher than English. Under the Non-Personalized setting, most models score higher on the Chinese subset. The authors attribute this to Chinese queries containing denser implicit risk cues in the current evaluation setting, which helps models identify latent safety needs.
• Personalization improves safety but does not solve it. Adding personalized information raises Average Safety Scores from a 1.8–2.5 range to above 3.0 in multiple domains, with the largest gains in psychological well-being and academic integrity. No model achieves a perfect score in any dimension, so risks are not fully eliminated.
• Reinforcement learning matters more than parameter count. In the Qwen series, RL-optimized QwQ-32B significantly outperforms its instruction-tuned counterpart Qwen2.5-32B across all dimensions. QwQ-32B improves by 2.06 after receiving personalized information, whereas Qwen2.5-32B improves by only 1.14.
• Recent feedback is the single most impactful attribute. In an ablation on 500 samples, recent feedback produces the largest gain, raising the Average Safety Score from 2.18 to 3.90.
• Explicit profile exposure gives the strongest safety gains. Comparing Non-Personalized, Implicit-Personalized, and Explicit-Personalized configurations, safety performance rises consistently as personalization becomes more explicit.
• Human and automatic judgments agree reasonably well. On 2,000 Chinese and 2,000 English sampled instances rated by ten expert annotators, human-human ICC reaches 0.87 for Average Safety Score. Claude-Haiku-4.5 shows the highest agreement with averaged human references (Spearman's rho = 0.83), outperforming GPT-4o (0.72) and a two-model averaged evaluator (0.79), so Claude-Haiku-4.5 is adopted as the primary automatic evaluator.

Methodology in Plain English

The authors first define a two-level taxonomy of 15 educational risk domains grouped into four categories: Psychological and Emotional Health, Academic Integrity and Competence, Content and Information Bias, and Learning Dependence and Cognition. Representative domains include Academic Pressure Overload, Academic Misconduct, Model Hallucination Risks, and Cognitive Rigidity and Innovation Suppression. Domains were chosen based on prevalence in student-AI interactions, psychological or cognitive fragility in developmental literature, and negative pedagogical consequences when responses misalign with a student's profile.

Student profiles use 14 attributes across four categories: Background (age, gender, learning stage), Big Five Personality Traits (openness, conscientiousness, extraversion, agreeableness, neuroticism, each at low/medium/high levels), Emotion (emotional state, emotional intensity, recent feedback), and Education (ability belief type, skill, skill acquisition stage, self-regulated learning phase). These are grounded in the Big Five model, Dweck's mindset framework, the Fitts–Posner skill acquisition model, and Zimmerman's self-regulated learning model.

Data collection starts from educational data for students aged 7–22 drawn from existing datasets; missing profile dimensions are completed to produce seed profiles. Rather than relying on a single generator, the authors use a cyclic multi-LLM collaborative generation strategy with four models (GPT-4o, Gemini-2.5-Flash, DeepSeek-V3, and Claude-Haiku-4.5), rotating them across risk domains to reduce model-specific bias. Eight hard rule constraints prevent implausible combinations, such as age–grade mismatches, personality–domain conflicts, emotion–domain conflicts, and mindset–domain conflicts; violating instances are discarded and regenerated. Personalized queries are then generated from validated profiles. Refinement proceeds through language correction for bilingual consistency, semantic deduplication, and structural filtering, followed by human audits. Notably, the Chinese and English datasets are generated with independently designed prompts rather than by translation.

Each scenario is evaluated under two conditions: Non-Personalized (query only) and Personalized (query plus full student profile). Three Likert-scale metrics (1–5) are scored, and the Average Safety Score averages them. Because full human annotation at this scale is prohibitively costly, the authors run a human-AI reliability analysis on 4,000 sampled instances to validate automatic scoring before scaling up.

Why This Matters

• Impact on research: The paper shifts safety evaluation from context-independent properties toward user-conditional harm, arguing that a single response can be safe for one student and harmful for another. It provides an open benchmark (code at the linked ECNU-RAIL repository) and a validated LLM-as-a-judge protocol for a high-stakes domain.
• Real-world applications:
  • Tutoring systems that must adjust tone, caution, and guidance based on a student's emotional state, mindset, or learning stage.
  • School and university deployment review of LLM-based assistants, where blanket safety filters miss personalized risks.
  • Guardrail and alignment development for education-domain models such as InnoSpark-7B and MuduoLLM-7B.
  • Mental-health-adjacent interaction design, where responses to academic pressure, career dilemmas, or interpersonal distress need profile-aware handling.
• Industry relevance: Model providers and edtech companies can use CASTLE as a pre-deployment test for personalized safety, and the finding that RL optimization outperforms parameter scaling gives a concrete direction for training investment rather than simply building larger models.

Future Directions

• Extending CASTLE from single-turn evaluation to more realistic interactive and multi-turn educational settings, which the authors list as a core limitation.
• Broadening coverage beyond Chinese and English to more languages and cultural contexts.
• Investigating whether explicit personalization can eliminate, rather than merely reduce, the residual risks that persist even when full student profiles are supplied.
• Developing safeguards so that student profiles are not over-interpreted or used to label students, and so that distress-related scenarios are not misused as direct counseling or intervention guidance.

Target Audience

Researchers and practitioners working on LLM safety, alignment, and personalization; education technology developers and evaluation teams responsible for student-facing AI products; and educational psychologists or learning-science researchers interested in how student attributes interact with AI-generated risk. The paper is also useful for benchmark designers looking for a methodology that combines theory-grounded taxonomy construction, multi-model data generation, hard logical constraints, and validated automatic scoring.

Authors’ abstract

Large language models (LLMs) have advanced the development of personalized learning in education. However, their inherent generation mechanisms often produce homogeneous responses to identical prompts. This one-size-fits-all mechanism overlooks the substantial heterogeneity in students cognitive and psychological, thereby posing potential safety risks to vulnerable groups. Existing safety evaluations primarily rely on context-independent metrics such as factual accuracy, bias, or toxicity, which fail to capture the divergent harms that the same response might cause across different student attributes. To address this gap, we propose the concept of Student-Tailored Personalized Safety and construct CASTLE based on educational theories. This benchmark covers 15 educational safety risks and 14 student attributes, comprising 92,908 bilingual scenarios. We further design three evaluation metrics: Risk Sensitivity, measuring the model ability to detect risks; Emotional Empathy, evaluating the model capacity to recognize student states; and Student Alignment, assessing the match between model responses and student attributes. Experiments on 18 SOTA LLMs demonstrate that CASTLE poses a significant challenge: all models scored below an average safety rating of 2.3 out of 5, indicating substantial deficiencies in personalized safety assurance.