Rubric-Based Assessment of Student Reports Using Small Language Models

AI summary

Overview

Research area: Computing education / AI-assisted assessment (automated grading of student writing in cybersecurity education), published at SIGCSE TS 2027.

Technical level: Intermediate — readers should be comfortable with the general idea of running language models locally, but no deep ML expertise is required.

Scope: The paper evaluates whether small, locally hosted language and vision-language models can score two types of student cybersecurity reports against strict rubrics on consumer-grade hardware, compared with human instructor scores.

What This Paper Is About

Instructors who teach large courses need help grading, and language models can apply a rubric and return structured feedback. Most existing approaches depend on cloud services or large models, which are costly, resource-hungry, and force student data off campus. The authors ask whether small open models, run on ordinary GPUs with no fine-tuning, can grade both plain-text reports and rich-text reports containing images as accurately as human graders and as well as a much larger model.

Key Contributions

1. A locally deployable, open-source assessment pipeline that combines small text models with a vision-language model to grade reports containing both text and screenshots, using rubric-aligned prompts and JSON-constrained output.
2. An empirical comparison of five models across four hardware configurations, including a system with no GPU, with each report evaluated 10 times and validated against human instructor scores.
3. Evidence that small models are competitive with a large baseline: small models reached up to 97% accuracy on text-only reports and up to 94% on rich-text reports, which the authors say rivals or outperforms the large gpt-oss-120b model used only as a remote baseline.
4. Practical guidance on rubric design, showing that criteria tied to clearly identifiable evidence grade reliably while context-heavy, synthesis-dependent criteria remain unreliable and need human review.

Main Findings

• Accuracy ceilings: Up to 97% accuracy for text-only executive summaries and up to 94% for rich-text security advisories, across the locally hosted small models and hardware configurations tested.
• Text-only beats rich-text: Executive summaries consistently achieved higher accuracy and lower evaluation time than security advisories on every system, since advisories require image interpretation and a more complex context.
• Evidence-based criteria work well: Models exceeded 90% accuracy on phishing-related criteria such as phishing_site_disabled (gemma3 and qwen3 both reached 97.14% on systems S1 and S2) and phishing_email_address (gemma3 reached 91.43%, 94.29% and 88.57% on S1, S2 and S3).
• Context-heavy criteria fail: The sections criterion in the security advisory fell below 55% (48.57–54.29% for gemma3 across the three local systems; 31.43% for llama3). The incident_reports criterion in the executive summary sat around 68.57–71.43% for most models.
• Strict rubrics amplify disagreement: Most rubric criteria were scored 0 or 1 (one criterion used 0–2), so partially correct answers receive zero and small omissions register as full disagreements with the human score.
• No clear accuracy–speed trade-off among small models: The fastest configurations (often gemma3 on S2 or S3) matched slower alternatives on most criteria; gpt-oss-120b was the fastest for executive summaries but weaker on advisories.
• Speed patterns: qwen3:1.7b was the slowest on all systems (except mistral:7b on S2 for the advisory) because it is a reasoning model producing more output tokens. gemma3:4b was among the fastest overall. S3 was the fastest system and S1 the slowest due to its older GPU. Text-only evaluation took an average of 30 minutes per model.
• CPU-only was insufficient: The system without a GPU could not complete the rich-text evaluation; output was incomplete even with a smaller image model, so it was excluded from comparisons.
• llama3:8b tended to lead in accuracy, with qwen3:1.7b close on executive summaries but noticeably slower.
• No prompt injection found: Instructors reviewed all reports and found no attempts at prompt injection by students.

Methodology in Plain English

The data came from a real-time, in-person cybersecurity incident simulation run in two semesters (2024 and 2025) of an introductory undergraduate course, "Cybersecurity in an Organization," at a Czech public university. Students played incident handlers on a fictitious university CSIRT. Of 38 teams of three, 35 consented to their data being used. Each team produced two documents: a plain-text executive summary (graded on four criteria worth 4 points) and a PDF security advisory (graded on four criteria worth 5 points, including two required screenshots). Reports ran from 56 to 305 words and were graded by five instructors in 2024 and six in 2025, with scores checked by the course guarantor.

A Python script using PyPDF2 and Pillow separated text from embedded images. For the advisory, the vision-language model llava:7b first produced textual descriptions of the screenshots, which were then merged with the extracted text so a text-only model could grade the whole document. That split avoids needing one large multimodal model and reduces computational load.

Grading used carefully iterated prompts containing a role assignment, task context, the rubric, an exemplary report, and output constraints, with responses returned as JSON containing criterion scores and brief feedback. Less-structured approaches (zero-shot, one-shot, context-free) were fast but not accurate enough. Models ran locally through Ollama with the langchain_ollama package on three consumer-grade systems plus a remote H100 machine hosting gpt-oss-120b as a comparison baseline. Performance was measured as exact match between model score and human score, plus average processing time, averaged over 10 runs per report.

Why This Matters

Impact on research: The paper shifts automated assessment research away from cloud-dependent, large-model assumptions toward low-resource, privacy-preserving deployment, and it extends rubric-based grading from short answers and essays to rich-text documents with embedded visual evidence — a case prior work had not covered.

Real-world applications:

• Grading incident reports, advisories, and summaries in cybersecurity courses and training simulations without sending student data to a third party.
• First-pass triage in large courses, where instructors review flagged submissions and finalize scores rather than grading everything from scratch.
• Deployment at under-resourced institutions that lack GPU clusters or cloud budgets, using consumer GPUs.
• Adapting the same pipeline to other report formats in computing education, since the extraction step needs only minimal modification.

Industry relevance: The work speaks directly to organizations that must comply with FERPA or GDPR-style constraints and therefore cannot transmit student or trainee data externally. It also demonstrates a practical pattern — small text model plus small vision model, with structured JSON output — for cost-effective internal document evaluation, and it shows that on-premise small models can substitute for large hosted models on rubric-driven tasks with clearly identifiable evidence.

Future Directions

• Adapting the workflow to other assignment types and document structures beyond executive summaries and security advisories.
• Improving prompt and rubric design so models can better handle contextual interpretation, document-structure assessment, and multimodal evidence.
• Determining systematically which rubric dimensions are suitable for automated first-pass scoring and which should remain with human graders.
• Investigating the observed inconsistencies and possible model biases, such as over-reliance on surface-level patterns or incomplete use of the supplied context, and addressing cases where human scores used as ground truth may themselves be wrong.

Target Audience

Instructors and teaching staff in computing education who grade large volumes of student writing; researchers in AI-assisted assessment; and IT or educational-technology staff at institutions with limited hardware budgets or strict data-privacy requirements. It is also useful for practitioners who want a concrete, replicable template for local model deployment rather than a purely theoretical argument.

Authors’ abstract

Language models can analyze and assess student texts using predefined rubrics and deliver structured feedback, reducing instructors' workload. However, existing solutions often rely on external services that are resource-intensive, costly, and raise privacy concerns, making them unsuitable for constrained environments. This study evaluates an automated assessment pipeline with three key aims: design for low-resource environments, privacy, and focus on a novel context in cybersecurity education. The pipeline integrates lightweight language models for textual evaluation and a vision-language model for image interpretation. Responses from 105 students were collected over two semesters of an undergraduate course during a cybersecurity exercise on incident response. Results across different hardware configurations show that small models in low-resource environments can achieve up to 97% accuracy in text-only and up to 94% in rich-text assessments. This rivals or even outperforms a large model while adding the benefits of locally hosted assessment. Our solution can help educators and practitioners who need grading assistance without replacing human oversight, especially in environments where cloud-based solutions may be impractical. This study contributes an open-source, locally deployable AI-assisted scoring pipeline that runs on affordable hardware and can be extended to various types of student reports in computing education. As a result, it makes automated assessment accessible to a broad range of educators and institutions.