Text Annotation via Inductive Coding: Comparing Human Experts to LLMs in Qualitative Data Analysis

AI summary

Overview

Research area: Natural Language Processing applied to computational social science — specifically the automation of qualitative data analysis (QDA) through inductive coding.

Technical level: Intermediate. Readers should be comfortable with LLM fine-tuning concepts (LoRA, few-shot prompting), text generation metrics (ROUGE, BERTScore), and basic qualitative research terminology (codes, thematic analysis, open vs. axial coding).

Scope: The paper benchmarks six open-source LLMs against human expert coders on the task of open coding, the first stage of thematic analysis, and finds a counterintuitive split in competence between humans and models depending on sentence complexity.

What This Paper Is About

Qualitative researchers spend weeks manually reading interview transcripts, field notes, and reviews to assign short "codes" — labels capturing the main idea of each passage. This coding step is essential but slow, subjective, and inconsistent across coders. The authors ask whether LLMs can automate inductive coding, where labels emerge from the data rather than being chosen from a predefined list, and how good models actually are compared to trained human coders when both are judged by experts.

Key Contributions

1. An inductive open-coding benchmark built from real qualitative data. The authors assembled 600 code–quote pairs contributed by students and professors across three social science faculties (covering self-tracking devices, voice assistants, values, procrastination, doctors' experiences, and more), then augmented it with 400 manually coded review excerpts from SemEval-2014 Task 4, for a 1,000-example dataset with 680 unique codes.

2. A systematic comparison of six open-source LLMs under three adaptation strategies. Llama3, Falcon, Mistral, Vicuna, Gemma, and TinyLlama were evaluated in zero-shot, few-shot (1/3/5 examples), and LoRA fine-tuning configurations, with multiple prompt phrasings tested for each.

3. A two-stage human expert evaluation protocol. Three expert qualitative researchers first coded 15 sentences and rated their perceived difficulty, then rated the labels produced by other humans, by the best LLM configurations, and by the golden-standard consensus label on a 1–5 scale. This produces a "deviation from golden standard" (DGS) measure rarely used in NLP evaluation.

4. Evidence that dataset size has steeply diminishing returns. Performance measured by BERTScore plateaus at roughly 100 fine-tuning examples, a finding with direct practical consequences for social scientists who cannot easily gather large labeled corpora.

Main Findings

• Fine-tuning beats prompting, and Falcon and Mistral lead. Falcon achieved the best BERTScore (0.764) and Mistral the best ROUGE-L (0.235) when fine-tuned on the augmented dataset. Zero-shot performance across all models clustered around 0.67–0.71 BERT F1, with most models improving markedly at one-shot.

• Augmentation with review data helped. Models trained on the full 1,000-example set outperformed those trained on the original 600-example social science data. Falcon's BERTScore rose from 0.735 to 0.764 and Mistral's from 0.731 to 0.756.

• Roughly 100 fine-tuning examples are enough. BERTScore improvement flattened after about 100 examples for every model tested, suggesting fine-tuning data scarcity is less limiting than commonly assumed.

• Humans and LLMs have inverted competence profiles. Human coders received the highest expert ratings on difficult sentences and performed comparatively poorly on easy ones. LLMs showed the opposite pattern — strong on simple sentences, weaker on complex ones.

• Humans over-interpret simple sentences. For the sentence "I can ask the voice assistant what the weather is like," LLMs produced codes like weather forecast and weather prediction, close to the golden standard weather. Humans produced functional usage, device feature, and voice command — abstract labels that strayed from the sentence's core meaning. One coder acknowledged a tendency to over-interpret.

• LLMs deviated less from the golden standard. Human coders, especially Coder A, showed large positive deviation (rated higher than the golden standard by experts). Falcon and Mistral showed the smallest deviations, particularly on easy and medium sentences. Llama3 deviated positively on medium and difficult sentences, mimicking human-like over-elaboration.

• Expert ratings and golden-standard alignment disagree. Some LLMs matched the consensus label more closely than human coders did, yet received lower expert ratings — while human labels that diverged from the consensus were often rated more favorably by other humans.

• Perceived difficulty tracks sentence length, not readability scores. Averaged difficulty ratings correlated strongly with sentence length. Standard readability metrics (Flesch Reading Ease, Coleman-Liau) showed no correlation with coder-perceived difficulty.

• Inter-coder reliability was low (Krippendorff's alpha ≈ 0.2). Even among expert humans using a shared definition of a code, agreement was poor — the authors attribute this to the inherently interpretive nature of the task rather than to coder error.

Methodology in Plain English

The researchers assembled a dataset of passages paired with the short labels that human analysts had agreed on, treating that consensus as a "golden standard." They split it 900/100 into training and test sets and trained open-source LLMs using low-rank adaptation (a lightweight fine-tuning method) on the training portion, then measured how close generated codes were to the reference labels using BERTScore (semantic similarity) and ROUGE (word overlap).

Separately, they bypassed automatic metrics entirely. Three expert qualitative researchers were given 15 sentences to code themselves and asked to rate how hard each was to code on a 1–3 scale. In a second stage, the same experts rated — on a 1–5 scale — labels produced by the other humans, by the best-performing LLM configurations, and by the golden standard itself. From these ratings, the authors computed both a deviation score for each coder relative to the golden standard and Krippendorff's alpha for overall agreement.

Why This Matters

Impact on research. The paper challenges two assumptions in computational social science: that human expertise guarantees higher-quality codes, and that large labeled datasets are needed to usefully fine-tune LLMs for qualitative tasks. It also provides a methodological template — combining automatic metrics with blinded expert rating and difficulty stratification — that future QDA automation studies can adopt. The low Krippendorff's alpha raises a broader point: standard reliability statistics may be inappropriate for measuring interpretive work.

Real-world applications:

• Customer and product feedback analysis — coding thousands of reviews or support tickets into themes, where sentences are short and the task is relatively objective.
• Social media and media monitoring — tracking emerging narratives across posts without predefined taxonomies.
• Market and UX research — accelerating the first pass of thematic analysis on interview transcripts before human refinement.
• Survey and open-ended response analysis — summarizing and labeling free-text answers at scale.

Industry relevance. Qualitative analysis software vendors (Atlas.ti, MAXQDA, Dedoose) currently provide organization tools but not automated coding. The finding that fine-tuning is cheap and effective at 100 examples suggests a commercially viable path to "first-pass coding" features. At the same time, the paper argues LLMs are better positioned as assistive tools than as full replacements, an important framing for product design and for research ethics review.

Future Directions

• Extending to axial coding. The paper deliberately stops at open coding. The authors plan to test whether LLMs can group open codes into hierarchical higher-order categories — a task that, unlike open coding, requires knowledge of the research topic and may favor human experts more strongly.

• Evaluating hybrid human–LLM workflows. An open question is whether LLM-generated codes can reach consensus among models the way human coders do through discussion, and whether a pipeline where LLMs produce initial codes and humans refine complex cases outperforms either alone.

• Better evaluation metrics for interpretive tasks. BERTScore and ROUGE capture surface and semantic similarity but not interpretive quality. The authors call for metrics that reflect what qualitative researchers actually value in a code.

• Generalization beyond social science interviews and reviews. The current data covers interviews and online reviews; domains like medical text, legal documents, and field notes with specialized terminology remain untested.

• Revisiting human subjectivity. If human coders' open codes shift after they see axial coding results, the notion of a fixed "golden standard" for open coding may itself need rethinking.

Target Audience

This paper is most valuable to computational social scientists and NLP researchers working on text annotation without predefined label sets; qualitative methodologists curious about where LLMs can and cannot substitute for expert judgment; and product teams building research or feedback-analysis tools who need evidence on how much training data and which model choices are actually required. Researchers new to QDA will also find the paper's framing of open versus axial coding and its candid discussion of coder disagreement unusually accessible.

Authors’ abstract

This paper investigates the automation of qualitative data analysis, focusing on inductive coding using large language models (LLMs). Unlike traditional approaches that rely on deductive methods with predefined labels, this research investigates the inductive process where labels emerge from the data. The study evaluates the performance of six open-source LLMs compared to human experts. As part of the evaluation, experts rated the perceived difficulty of the quotes they coded. The results reveal a peculiar dichotomy: human coders consistently perform well when labeling complex sentences but struggle with simpler ones, while LLMs exhibit the opposite trend. Additionally, the study explores systematic deviations in both human and LLM generated labels by comparing them to the golden standard from the test set. While human annotations may sometimes differ from the golden standard, they are often rated more favorably by other humans. In contrast, some LLMs demonstrate closer alignment with the true labels but receive lower evaluations from experts.