The Role of Implicit and Explicit Demographic Signals in Large Language Model-based Student Assessment

AI summary

Overview

• Research area: Natural Language Processing / AI fairness and equity in education, specifically the sensitivity of large language models used for student assessment to demographic information about the learner.
• Technical level: Intermediate. The reader needs basic familiarity with LLM prompting, regression analysis, and standard text metrics, but the paper is more about experimental design and fairness questions than about novel architecture.
• Scope: A controlled, counterfactual study of how six state-of-the-art LLMs change their scoring, feedback, and answer generation when student demographics are signalled explicitly in the prompt versus implicitly through prior conversation history.

What This Paper Is About

LLMs are increasingly used to grade essays, write feedback, and answer learners' questions, yet little is known about whether the demographic traits of the student (gender, age, education, socioeconomic status, first language, and so on) systematically alter what the model produces. In some tasks this sensitivity is desirable — a first-language learner arguably deserves different feedback than a native speaker — but in evaluative tasks such as scoring, outputs should depend only on the quality of the work. The paper asks two questions: do implicit and explicit demographic cues systematically influence LLM behaviour, and does the size and nature of that effect differ across educational tasks?

Key Contributions

1. A large controlled benchmark of demographic sensitivity. The authors generate 192,480 LLM responses across three educational tasks (Automated Essay Scoring, Formative Feedback, and Metalinguistic Question Answering), each evaluated under three conditions: no demographics (default), explicitly stated demographics, and implicitly signalled demographics via conversation history.
2. A realistic persona-based evaluation design. Rather than synthetic stereotypes, the study draws on 200 real user profiles from the AI Gap dataset, each with 25 demographic attributes and a ten-prompt conversation history, so that implicit signals reflect genuine patterns of user language rather than artificial ones.
3. Cross-model comparison. Six instruction-tuned models spanning proprietary and open-weight families and sizes (GPT-5-mini, GPT-5-nano, Llama 3.3 70B, Llama 3.2 3B, Qwen3 4B, Qwen3 30B) are compared within a fixed-task design, isolating model-level differences in susceptibility.
4. A multi-metric statistical framework. The paper combines Lasso regression on demographic attributes, two-sample Kolmogorov–Smirnov tests on response distributions, and paired t-tests on essay scores, across readability, length, sentiment, and semantic drift (BERTScore) metrics.

Main Findings

• Both explicit and implicit demographic cues change model behaviour. Across all three tasks, models adjusted their scores, feedback, and answers when demographic context was present, confirming that demographic sensitivity is a general property of current LLMs rather than an artefact of one prompting style.

• Scoring effects are model-dependent and can be large. On Automated Essay Scoring, GPT models were the most stable, showing no significant shift from human ratings under either condition. Llama 3.3 70B was the outlier: under the implicit condition it inflated scores by +1.57 points relative to its own default, the largest deviation in the study.

• Explicit demographic information triggers readability adaptation. When education level or socioeconomic status was stated outright, most models produced less readable responses for users with higher education and higher SES. The Education attribute was a significant predictor of readability for four models (Llama-70B, Llama-3B, Qwen-30B, GPT-mini). This looks like intended personalisation, though it also implies models hold assumptions about what different learners can read.

• This adaptation disappears in the implicit condition. Implicit demographic context did not reproduce the education-driven readability effect. Instead, models appeared to mimic surface features of the prior conversation history, such as producing longer answers for users whose histories contained longer prompts.

• Implicit cues produce larger and less predictable effects on response length. Response length showed more significant implicit effects than explicit ones (regression slope of 1.20 comparing implicit to explicit), with users who mention laptops receiving longer responses across four models, and Black or African American ethnicity producing longer feedback in Qwen-4B.

• Sentiment shifts in ways that raise equity concerns. Higher education levels were associated with more positive-sentiment responses under explicit conditioning in both Llama models, a gap of roughly 0.3 on a 0–4 sentiment scale. The abstract highlights the implicit counterpart: answers given to lower-education profiles received lower sentiment scores.

• Models default to an English- and US-centric reference point. In the BERTScore analysis, "language: English" and "nationality: US" were the attributes producing responses most similar to the default output, indicating that the unstated baseline persona is implicitly English-speaking and US-based.

• Conversation topics themselves are demographically patterned. Topic modelling over the 2,000 simulated prompts found systematic differences: women's prompts concentrated on career and résumé topics while men's clustered in general Q&A; younger users asked more AI-related questions; lower-SES users asked more factual-lookup questions while higher-SES users asked more finance-and-stocks questions. This complicates the implicit condition, since topic composition alone can drive output differences.

• Formative feedback effects were concentrated in one model. In feedback generation, significant effects appeared almost exclusively under implicit conditioning and largely in Llama-70B, spanning behavioural attributes (brainstorming use cases, laptop ownership, writing use cases) as well as core demographics such as gender and religion.

Methodology in Plain English

The researchers built a controlled experiment in which the educational content stays fixed and only the demographic context changes.

For each of three tasks they selected 40 items. For essay scoring and feedback, they used 40 human-written essays with expert scores from the Persuade 2.0 corpus; for metalinguistic questions, 40 questions from the ELQA dataset.

They then selected 200 real user profiles from the AI Gap dataset, each containing 25 demographic attributes and a history of ten previous prompts. Each of the 40 items was paired with every one of the 200 profiles, creating thousands of prompt variants.

Three conditions were compared. In the default condition, the model saw only the essay or question. In the explicit condition, the demographic profile was written into the prompt as a user description. In the implicit condition, the ten prior prompts and their simulated responses were prepended as conversation history, with no demographic attributes named at all.

Responses were measured with a different lens for each task. Essay scores were compared against the human expert ratings and against the model's own default. For the open-ended tasks, the authors computed readability (Automated Readability Index, Flesch Reading Ease), average age of acquisition of the words used, response length, sentiment, and BERTScore similarity to the default response.

To detect whether any demographic attribute reliably mattered, they trained Lasso regression models predicting each metric from the demographic attributes, and ran Kolmogorov–Smirnov tests comparing distributions across subgroups. Because the same profile appears 40 times, they ran separate tests per item rather than pooling all observations, then counted how often an attribute reached significance. An attribute had to be significant in at least 10 item-level tests to count under explicit conditioning, and at least 5 under implicit conditioning, on the reasoning that implicit signals are subtler and harder to detect.

Why This Matters

The paper establishes that demographic sensitivity in educational LLMs is not a hypothetical concern but a measurable, reproducible property that varies by task, model, and channel of information. For fairness research, it shows that auditing only explicit demographic prompts is insufficient: conversation history alone can shift model outputs, and often in less predictable directions than a stated persona. It also reframes some demographic effects as legitimate personalisation rather than pure bias, which complicates how fairness metrics should be defined for instructional tasks.

Real-world applications:

• Essay grading and automated scoring systems. Institutions deploying LLM graders need to know that scores can drift by more than a point depending on inferred student background, and that model choice matters enormously — GPT models were stable here while Llama-70B was not.
• Feedback generation tools for writing instruction. Readability adaptation can help or harm: tailoring text to a learner's level is useful, but removing that adaptation under implicit conditions means the same student may receive inconsistent feedback depending on how the system infers their background.
• Conversational tutoring and question-answering assistants. Sentiment and length shifts based on inferred education or SES could mean learners from disadvantaged backgrounds receive shorter or less encouraging explanations.
• Procurement and model selection in EdTech. The finding that stability varies sharply across model families gives buyers a concrete criterion — demographic robustness testing — to include in evaluation checklists.

Industry relevance is direct: any vendor or institution using LLMs for assessment, admissions support, language learning, or tutoring is affected. The open release of code and data under GPL v3.0 gives practitioners a template for running the same sensitivity tests on their own deployed models.

Future Directions

• Separating topic effects from demographic effects in implicit conditioning. The topic analysis shows that conversation histories are themselves demographically patterned, so it remains unclear whether implicit shifts come from inferred identity or simply from the subject matter of prior prompts. Designs that hold topic constant while varying demographics would disentangle these.
• Extending beyond the attribute set and task set examined. The study covers three tasks and 25 attributes, with feedback effects concentrated in a single model. Testing additional tasks such as oral proficiency, code review, or mathematics reasoning would clarify whether the patterns generalise.
• Building mitigation and calibration methods. The paper documents sensitivity but does not propose a fix. Candidate approaches include demographic-invariance training objectives, post-hoc score normalisation, or prompt designs that suppress persona inference from history.
• Investigating whether observed adaptation helps or hurts learners. Readability matching to education level may be pedagogically beneficial, while sentiment differences almost certainly are not. Empirical studies with real students are needed to determine which demographic sensitivities are desirable and which should be eliminated.

Target Audience

This paper is most valuable to AI fairness and NLP researchers working on bias evaluation, to educational data mining and learning analytics researchers studying automated assessment, and to EdTech engineers and product teams building LLM-based grading or tutoring systems. Instructors and administrators evaluating whether to adopt AI assessment tools will also benefit from the concrete evidence that model choice substantially affects demographic robustness, as will policy analysts drafting standards for algorithmic fairness in education.

Authors’ abstract

Large Language Models are now common in student assessment, but we know little about how student demographics affect their use. Sometimes, considering student demographics may be necessary -- for example, to improve readability for users with lower educational levels. However, it also risks being a cause of discrimination, e.g., when assigning lower scores to students from lower socioeconomic backgrounds. We set up controlled prompts to test 1) explicit demographic effects, where we mention demographic details directly, and 2) implicit effects, where we use conversation history as a demographic signal. We test these settings in three tasks: Automated Essay Scoring, Formative Feedback, and Metalinguistic Question Answering. We test six state-of-the-art LLMs on these tasks. In both explicit and implicit cases, the models pick up on demographic cues and can change their scoring, feedback, and answers accordingly. We find that LLMs frequently adjust the readability of feedback to education levels when these are explicitly mentioned. On the other hand, implicit conditions produce unpredictable biases, such as in question answering, where responses from lower-education levels receive lower sentiment scores. Our results provide clear evidence of demographic sensitivity in LLMs for educational assessment tasks.