Repurposing Annotation Guidelines to Instruct LLM Annotators: A Case Study

AI summary

Overview

Research area: Natural Language Processing — specifically text annotation, annotation guideline design, and the use of large language models as annotators.

Technical level: Intermediate. The work sits at the intersection of corpus annotation methodology and LLM prompting, and assumes some familiarity with how annotated datasets and guidelines are produced.

Scope: A case study on transforming human-oriented annotation guidelines into explicit instructions that LLMs can follow, evaluated on the NCBI Disease Corpus.

What This Paper Is About

Annotation guidelines are traditionally written for people, who absorb much of the task through training conversations, examples, and accumulated practice rather than through the written text alone. LLMs have no such training context, so they need instructions stated explicitly and in a structured form. This paper asks whether existing guidelines can be reworked into that form automatically, using an LLM-based moderation process, and what happens when the resulting instructions are used to drive LLM annotators.

Key Contributions

1. A proposed repurposing workflow. The authors introduce a "moderation-oriented guideline repurposing method" that rewrites existing annotation guidelines into clear, explicit directives suited to LLM annotators, with an LLM moderation process doing the transformation.

2. A demonstration on a real annotation task. The method is applied to the NCBI Disease Corpus, using it as the case study for testing whether repurposed guidelines can guide LLM annotation.

3. Evidence that the approach can work. The experiments indicate that repurposed guidelines can effectively guide LLM annotators, supporting the viability of the workflow.

4. An account of practical obstacles. The study surfaces several practical challenges that arise when repurposing guidelines this way, framing the method as promising rather than solved.

Main Findings

• Repurposed guidelines can guide LLM annotators. The experiments show that transforming human-oriented guidelines into explicit directives produces instructions that LLMs can follow for the annotation task. The abstract does not report specific accuracy figures or comparisons.

• Practical challenges remain. The authors report that the workflow revealed several practical challenges. The abstract does not identify them individually or quantify their severity.

• The transformation step matters. Framing the repurposing as an LLM moderation process implies that the quality of the rewritten directives is itself a variable in the pipeline, not a trivial formatting step — though the abstract gives no detail on how moderation was implemented or evaluated.

• Scalability and cost are the stated payoff. The results are described as highlighting the potential of this workflow to support scalable, cost-effective refinement of annotation guidelines and automated annotation. No cost measurements or scalability numbers appear in the abstract.

Methodology in Plain English

The starting point is that human annotators and LLMs need different kinds of instructions: people fill gaps through training and experience, while models need everything spelled out. The authors take existing written guidelines — designed for people — and run them through an LLM-based moderation process that rewrites them into explicit, structured directives.

Those repurposed guidelines are then used to instruct LLM annotators on the NCBI Disease Corpus, a biomedical text collection used for disease mention annotation. The case study setup lets the authors observe both whether the rewritten guidelines work as instructions and where the process breaks down. The abstract does not describe the moderation prompts, the number of examples, the comparison conditions, or how annotation quality was measured.

Why This Matters

Impact on research. Most annotation projects already have guidelines, often refined over years for human use. If those artifacts can be mechanically converted into LLM-ready instructions, researchers gain a path to reuse existing annotation expertise rather than writing model prompts from scratch — and a possible route to automating part of the guideline maintenance cycle itself.

Real-world applications (as directions this work points toward, not demonstrated outcomes):

• Biomedical and clinical text annotation, where the NCBI Disease Corpus example is directly relevant — identifying disease mentions in literature.
• Any annotation-heavy domain with mature written guidelines: legal text, financial documents, social science coding, content moderation.
• Scaling up dataset creation where human annotator time is the bottleneck, by using LLM annotators under repurposed instructions.
• Ongoing guideline maintenance, where LLM-assisted rewriting could help keep guidelines consistent as tasks evolve.

Industry relevance. Organizations that maintain large annotated corpora or annotation programs face constant tension between guideline quality and the cost of producing and revising them. A workflow that reuses existing guidelines and automates their adaptation to LLM annotators is directly relevant to teams building training data and evaluation sets at scale, particularly where domain expertise is expensive.

Future Directions

• Generalization beyond the case study. The paper uses one corpus. Whether the repurposing method transfers to other annotation tasks, domains, and guideline styles is left open.

• Resolving the reported practical challenges. The abstract states that challenges were revealed but does not enumerate them; addressing those specific failure modes is the obvious next step.

• Validating the adapted guidelines. Questions remain about how to verify that a repurposed guideline preserves the intent of the original and produces annotations consistent with human standards.

• Measuring the scalability and cost claims. The abstract frames the workflow as potentially scalable and cost-effective; establishing that with concrete comparison would be a natural follow-up.

Target Audience

Researchers and practitioners in NLP and corpus linguistics who build annotated datasets, write or maintain annotation guidelines, or are evaluating LLMs as annotators. It is also relevant to domain scientists in biomedicine who rely on annotated corpora, and to industry teams weighing LLM-assisted annotation against traditional human annotation pipelines. Readers looking for quantitative benchmark results will not find them in the abstract.

Authors’ abstract

This study investigates how existing annotation guidelines can be repurposed to instruct large language model (LLM) annotators for text annotation tasks. Traditional guidelines are written for human annotators who internalize training, while LLMs require explicit, structured instructions. We propose a moderation-oriented guideline repurposing method that transforms guidelines into clear directives for LLMs through an LLM moderation process. Using the NCBI Disease Corpus as a case study, our experiments show that repurposed guidelines can effectively guide LLM annotators, while revealing several practical challenges. The results highlight the potential of this workflow to support scalable and cost-effective refinement of annotation guidelines and automated annotation.