Can Large Language Models Differentiate Harmful from Argumentative Essays? Steps Toward Ethical Essay Scoring

AI summary

Overview

Research area: Automated Essay Scoring (AES) combined with LLM safety — specifically whether scoring systems and large language models can tell a harmful essay apart from a legitimate argumentative one, and whether they penalize harmful content when assigning scores.

Technical level: Intermediate. The paper assumes familiarity with AES models, instruction-tuned LLMs, precision/recall/F1, and Quadratic Weighted Kappa (QWK), but the core ideas are explained clearly enough for readers with basic NLP background.

Scope (one sentence): The paper builds the Harmful Essay Detection (HED) benchmark of 100 argumentative and 190 harmful essays, then tests five LLMs on classification and four existing AES models plus those same LLMs on holistic scoring.

What This Paper Is About

Automated Essay Scoring systems are built to grade essays quickly and consistently, but they are trained without any notion of ethics or morality, so an essay that argues a harmful position with polished writing can receive a high score. The authors ask two questions: can current LLMs reliably label an essay as "argumentative" or "harmful," and do existing AES models and LLMs lower their scores for harmful essays the way a human grader presumably would? To answer these, they construct the HED benchmark covering sensitive topics such as racism and gender bias, and evaluate both classification and holistic scoring behavior.

Key Contributions

1. The HED benchmark: a new dataset of 100 argumentative and 190 harmful essays built from 50 sensitive-issue prompts selected from the IELTS dataset (which contains 403 prompts); essays were generated by LLMs, then labeled by the authors, with only consistently labeled essays retained.
2. Two induction techniques for harmful generation: a "pretending" instruction that frames the request as AES research, and a persona-based instruction using 11 personas (from "Muhamad Ali" and "Hunter S. Thompson" to "a bad person," "a bad student," "a good student," and "4chan").
3. Two new evaluation metrics: Probability of Refusing (POR) and Probability of Circumventing (POC), which quantify whether a model refuses to generate harmful essays or instead complies by producing an argumentative essay. POR is the number of refusing queries divided by the number of queries with pretending; POC is the number of generated argumentative essays divided by the total number of generated essays.
4. A scoring instruction fix: adding the paper's harmful-essay annotation guidelines (discriminatory content, hate speech, ethical issues) into the rubric-based scoring prompt, which lowers scores for harmful essays and simultaneously improves QWK on IELTS essays.

Main Findings

• LLMs cannot yet separate harmful from argumentative essays. Macro F1 on the HED benchmark was 79.01 for Llama3.1-8B, 78.62 for Llama3-8B, 63.08 for GPT-4-turbo, 58.57 for Qwen2-7B, and 54.69 for Mistral-7B-v0.3 (averaged over five trials). Every model had lower recall for harmful essays than for argumentative ones — for example GPT-4-turbo scored 46.32 harmful recall versus 95.00 argumentative recall.

• Refusal ability correlates with detection ability. Llama3.1-8B and Llama3-8B refused every pretending instruction (POR 100%), while Qwen2-7B refused 33% and Mistral-7B-v0.3 refused only 14%. Qwen2-7B had a POC of 96%, meaning it usually complied but produced argumentative essays; Mistral-7B-v0.3 had a POC of 27%, making it the model most likely to actually generate harmful content. The HED essays therefore came mainly from Mistral, and to a lesser extent Qwen.

• Self-generated essays were not easier to classify. Mistral and Qwen showed the lowest classification performance even though they were classifying essays they themselves had generated.

• Toxicity separates the benchmark's two classes. Measured with the PERSPECTIVE API on a 0–1 scale using the maximum sentence-level toxicity per essay, harmful essays were clearly more toxic than argumentative ones; notably, the benchmark's argumentative essays were more toxic than those in the IELTS dataset because of the pretending instructions.

• Persona words shift classification performance. Averaged over three trials, every race persona improved Llama3.1 (Black +3.27 points, White +3.01 points); for Llama3 only Black (+2.73) and White (+2.93) improved; for Qwen all races helped, with Black, Indian, and White gaining 6 to 7 points; for Mistral every race persona hurt performance, with Black and White hurting least. Races used were African, Arab, Asian, Black, European, Indian, Jewish, Native American, South American, and White, combined with ages (10s, 20s–30s, 40s–50s, 60s–70s) and genders (Male, Female). Positive personalities hurt Llama3.1 and Llama3 but helped Mistral, and all personalities improved Qwen; Llama3.1 and Llama3 were least affected by personality words, Mistral and Qwen the most.

• Existing AES models reward harmful essays. On the HED benchmark (scores 1–6), harmful essays scored higher than argumentative ones for Hi att (4.65 vs 4.32), PAES (4.78 vs 4.74), NPCR (4.53 vs 4.02), and PMAES (4.58 vs 4.14) — the opposite of the reasonable behavior the authors assume.

• Rubric guidelines help QWK but backfire on harm. Adding rubric guidelines raised IELTS QWK for every LLM (Llama3.1-8B from 0.635 to 0.815, Llama3-8B 0.623 to 0.807, Qwen2-7B 0.619 to 0.799, Mistral-7B-v0.3 0.610 to 0.783), and existing AES models reached 0.608 (Hi att), 0.729 (PAES), 0.780 (NPCR), and 0.803 (PMAES). Yet with rubrics, harmful-essay scores rose for Llama3.1-8B (2.99 to 3.15), Qwen2-7B (3.74 to 4.89), and Mistral-7B-v0.3 (3.24 to 4.00); Llama3-8B was the exception, decreasing from 3.78 to 3.41.

• Harmful guidelines fix the scoring gap. Adding the annotation guidelines to the rubric instruction reduced harmful-essay scores to 2.12 for Llama3.1 (QWK 0.822), 2.59 for Llama3 (QWK 0.816), 3.79 for Qwen2 (QWK 0.811), and 3.29 for Mistral (QWK 0.802), while QWK on IELTS essays also improved. The paper's Table 7 lists a rubric-only harmful average of 3.14 for Llama3, slightly different from the 3.41 in Table 6.

Methodology in Plain English

The authors started with IELTS essay prompts rather than the more commonly used ASAP dataset, because ASAP has only 8 prompts and just two are argumentative, while IELTS offers 403 prompts and more sensitive social topics. They removed prompts that depend on charts, tables, maps, or diagrams, then picked 50 prompts about sensitive issues such as quotas for women on corporate boards.

Because direct requests for harmful essays were refused, they used two indirect approaches: a "pretending" instruction in which the model is told it is an essay writer helping AES researchers who need problematic essays for a study, and a persona instruction that assigns a named or described character to write the harmful essay. Each generated essay was labeled harmful if it contained discriminatory content, hate speech, or ethical issues, and only essays that all authors labeled the same way were kept.

For classification, the models were asked to label an essay as argumentative or harmful, with the annotator persona varied to test for persona effects. For scoring, the authors trained four existing AES models on IELTS essays sharing prompts with the benchmark, refined the IELTS score range from 1–9 to 1–6 for the experiments, and reported QWK; they then fed HED essays to those models and to the LLMs. Since no gold scores exist for harmful essays, they judged a system reasonable if it scored harmful essays lower than argumentative ones. Finally, they added the harmful-content definitions directly into the scoring instruction and measured the effect on both harmful-essay scores and IELTS QWK.

Why This Matters

For research, the paper opens an explicit ethical dimension in AES evaluation: it shows that standard scoring metrics can look healthy while a model still rewards essays promoting racism or gender bias, and it contributes a reusable benchmark and metric pair (POR, POC) for measuring refusal and circumvention behavior.

Real-world applications:

• Classroom and standardized-test grading, where a well-written but hateful essay must not receive a top band score or be returned without appropriate feedback.
• Teacher-support tools, since AES is pitched as a fairer and more consistent alternative to human graders, which requires the same ethical standards human graders apply.
• Content-moderation and integrity pipelines, for flagging essays that students may have produced with a generative model containing harmful opinions.
• Model alignment and red-teaming, using POR and POC to measure how strongly a model resists or redirects harmful-generation requests.

For industry, the finding that a few lines of harmful-content instructions in a scoring prompt both reduce harmful-essay scores and improve QWK on ordinary essays is a low-cost configurable fix for vendors shipping LLM-based grading products.

Future Directions

• Test larger models, since the current experiments were limited to LLMs below 10 billion parameters (Llama3.1-8B, Llama3-8B, Mistral-7B-v0.3, Qwen2-7B).
• Create gold scores for harmful essays, which the HED benchmark deliberately omits because annotation is so difficult, and analyze alignment between AES model scores and human assessors.
• Extend the ethical-scoring framework beyond harmful content — for example, to essays that are unsafe in other ways — and to disciplines and prompts outside the 50 IELTS prompts used.
• Investigate why persona words, especially "Black" and "White" and personality terms, shift performance so differently across models, and develop alignment approaches that reduce those biases.

Target Audience

Researchers and graduate students working on AES, LLM evaluation, and AI safety; NLP practitioners building LLM-based grading or feedback systems; educational technology developers and assessment organizations that need scoring tools sensitive to ethical content; and policy-minded readers interested in how automated scoring handles sensitive topics such as racism and gender bias.

Authors’ abstract

This study addresses critical gaps in Automated Essay Scoring (AES) systems and Large Language Models (LLMs) with regard to their ability to effectively identify and score harmful essays. Despite advancements in AES technology, current models often overlook ethically and morally problematic elements within essays, erroneously assigning high scores to essays that may propagate harmful opinions. In this study, we introduce the Harmful Essay Detection (HED) benchmark, which includes essays integrating sensitive topics such as racism and gender bias, to test the efficacy of various LLMs in recognizing and scoring harmful content. Our findings reveal that: (1) LLMs require further enhancement to accurately distinguish between harmful and argumentative essays, and (2) both current AES models and LLMs fail to consider the ethical dimensions of content during scoring. The study underscores the need for developing more robust AES systems that are sensitive to the ethical implications of the content they are scoring.