Zero- and Few-Shot Named-Entity Recognition: Case Study and Dataset in the Crime Domain (CrimeNER)

AI summary

Overview

• Research area: Natural Language Processing, specifically Named-Entity Recognition (NER) for forensic and crime-related documents.
• Technical level: Intermediate — the paper assumes familiarity with NER, transformer-based pretrained language models (PLMs), and zero-/few-shot evaluation, though the dataset design is described in accessible terms.
• Scope: The paper introduces CrimeNER-db, an annotated crime-domain NER database of 1568 documents, and benchmarks it with fully supervised PLMs plus zero- and few-shot open-source LLMs (arXiv:2603.02150v2 [cs.CL], licensed CC BY-NC-ND 4.0, from BiometricsAI, Universidad Autónoma de Madrid).

What This Paper Is About

Law enforcement agencies face a rapidly growing volume of crime-related documents, but there is a shortage of adequately annotated data covering general, real-world crime scenarios, so manual review is not feasible. The authors build CrimeNER-db, a dataset of more than 1.5K manually annotated documents drawn from U.S. Department of Justice (DOJ) press notes and public reports of real terrorist attacks, and use it to test how well general NER models and generalist LLMs can extract criminal information. The goal is to give researchers and agencies a shared benchmark for crime-related entity extraction, including under zero- and few-shot conditions where annotated examples are scarce.

Key Contributions

1. A new dataset and case study: CrimeNER-db, described as the first and largest dataset for NER on general crime documents and terrorist reports, with a total of 1568 annotated documents from the DOJ press notes and the Global Terrorism Database (GTD).
2. A two-level entity hierarchy: 4 coarse entity types (Crime, Actor, Agent & Agency, Logistic) with their corresponding fine-grained types, summing to 21 fine-grained entity types, with tokens labeled as a coarse entity plus fine type, or as a non-entity. Nested entities are annotated, making the dataset suitable for Nested NER.
3. A fully supervised baseline: evaluations using 6 general PLMs in 36 possible model configurations, training one model as a coarse entity extractor and another as a fine entity classifier, each configuration trained jointly for 5 epochs on an 80/10/10 train/val/test split.
4. Zero- and few-shot generalization study: evaluation of 4 open-source general LLMs (Llama3.2, Gemma 4, Mistral 7B, Phi-2) under 0, 1, 5, 10, 50, 100 and 200 shots, compared against the supervised baseline on coarse entities.

Main Findings

• Dataset size and content: CrimeNER-db contains 1568 documents. The DOJ source is the Kaggle dataset "Department of Justice 2009-2018 Press Releases" (approximately 200K press releases), from which documents between 100 and 500 characters long were selected, yielding more than 1.2k documents; roughly 300 documents were added from the 2021 GTD release descriptions to correct the under-representation of terrorist attacks, organizations, and locations outside the United States.
• Coarse entity statistics: Logistic has the most entity spans (1888) and documents (927); Crime has 1377 spans, 6013 tokens and 876 documents; Actor has 1416 spans, 2634 tokens and 854 documents; Agent & Agency has 1334 spans, 5887 tokens and 879 documents. Logistic is most common because of GPEs and dates; Agent & Agency is the least common entity type, while Crime has the highest number of tokens. Document counts are described as balanced across the four classes.
• Fine-grained distribution: For Crime, the most common fine types are Terrorism (372 spans) and Other (365 spans), with Theft least frequent. For Actor, Criminal Person is by far the most frequent with more than 600 spans. For Agent & Agency, Law enforcement is most common with approximately 800 spans. For Logistic, GPE is most common with more than 1k spans.
• Best strict supervised model: XLM-RoBERTa-Base achieves the best average strict F1, with 0.650 on coarse entities and 0.650 on fine entities.
• Best flexible supervised model: DeBERTa-V3-Base achieves the best average flexible F1, with 0.902 on coarse entities and 0.892 on fine entities. Its strict scores are 0.649 (coarse) and 0.627 (fine).
• Other supervised results: RoBERTa-Base scores 0.620 strict / 0.899 flexible on coarse and 0.631 strict / 0.882 flexible on fine; AlBERT-Base-V2 scores 0.607 / 0.881 on coarse and 0.401 / 0.888 on fine; DistilBERT-Base-Cased scores 0.519 / 0.890 on coarse and 0.643 / 0.886 on fine; BERT-Base-Cased scores 0.514 / 0.839 on coarse and 0.644 / 0.889 on fine. Flexible F1 is consistently higher than strict F1.
• Zero-/few-shot trend: LLM performance improves as more shots are provided, with the growth most notable between 0 and 10 shots. Llama3.2 begins to obtain worse flexible F1 scores when more than 10 shots are used, which the authors attribute to too many shots overwhelming the model context.
• LLMs do not beat supervision: In both strict and flexible evaluation, no LLM outperforms the fully supervised baseline. The gap is more pronounced under strict F1, which the authors link to the auto-regressive training of LLMs making token and span classification difficult due to mismatches between predicted and ground-truth spans.
• Best LLM: Gemma 4 is the best performing model in both evaluations, obtaining its best flexible F1 score with 100 shots, which the authors say shows it is suitable for crime-related information extraction even with very few samples of the crime being investigated.
• Fine entities not evaluated in zero-/few-shot: The zero-/few-shot experiments evaluate only coarse entities, because some fine entities do not appear enough in CrimeNER-db to cover all the shot counts considered.

Methodology in Plain English

The authors gathered text from two real-world sources: daily press releases from the U.S. Department of Justice (using the Kaggle "Department of Justice 2009-2018 Press Releases" collection) and descriptions of terrorist events from the 2021 release of the Global Terrorism Database. They kept passages of 100 to 500 characters and removed noisy text, then annotated entities using the Doccano library. Three independent annotators worked in three rounds: independent labeling, then cross-validation of each other's entities, then a final independent sanity check, with discrepancies discussed jointly when they exceeded a margin threshold, particularly for span length. During annotation, Agent and Agency were kept separate for clarity and merged into Agent & Agency during postprocessing. Labels follow two levels: a coarse type such as Crime or Logistic, plus a fine type such as Terrorism or Date, allowing nested entities to be labeled at both levels.

For evaluation, the authors used an extract-then-classify pipeline: one PLM detects and classifies coarse entity spans, and a second PLM (possibly the same) classifies those spans into fine types. Each of the 6 PLMs could fill either role, giving 36 configurations, each trained jointly for 5 epochs on an 80/10/10 split. Two metrics were used: a strict F1 where the span, coarse label and fine label must all match the ground truth, with a 2-character offset threshold to tolerate punctuation and whitespace; and a flexible F1 where labels must match but the predicted span only needs to overlap the ground truth or be a subset of it, because strict evaluation over-penalizes small prediction errors. For the LLM study, each model was prompted to extract all coarse entity spans from definitions, with few-shot prompts adding a different number of examples of each coarse entity per entity type; all experiments were designed with Ollama.

Why This Matters

This work addresses a documented gap: existing crime-related NER efforts focus on legal entities from court decisions or on Cyber Threat Intelligence, which the authors say are not suitable for general crime extraction in day-to-day criminal cases, and no dataset previously existed for general crime-related NER from real-world scenarios. CrimeNER-db provides that resource and a reproducible evaluation protocol, including a setting that directly reflects how agencies face unseen or barely seen crime types.

Real-world applications:

• Law enforcement triage: Automatically flagging who is accused, which agencies are involved, weapons, money and locations across large repositories of crime reports and press notes.
• Terrorism monitoring: Extracting perpetrators, targets, dates and locations from incident reports such as the GTD descriptions used here.
• Forensic document analysis: Supporting casework where models must handle nested entities such as a person inside an organization, or a place inside an agency name.
• Low-resource deployment: Using few-shot LLM prompting when an agency investigates a crime type for which it has little or no annotated training data.

Industry relevance: the dataset and the reported baselines give legal-tech vendors, security analysts and public-sector NLP teams a concrete benchmark, and the finding that generalist LLMs approach but do not beat supervised fine-tuning informs build-versus-prompt decisions. The database is available on GitHub, and the reported strengths and limits of seven model families guide practical tool selection.

Future Directions

• Dataset expansion: Extend CrimeNER-db with more documents from the same sources, or from sources in languages other than English, since the authors note the dataset is still small compared with other state-of-the-art NER datasets.
• Synthetic data generation: Explore synthetic data to enlarge the database.
• Multimodal architectures: Combine the NLP models used here with visual models that process text images, to improve criminal entity detection in multimodal documents.
• Manipulation and fake detection: Detect AI-generated information, fakes and other types of manipulation in document repositories being examined for NER.

Target Audience

Researchers and practitioners in NLP and forensic computing who need annotated crime-domain data; law enforcement and security analysts interested in automated information extraction; developers building legal-tech or security tools who want benchmark numbers for transformer models versus open-source LLMs; and annotation or dataset-construction teams looking for a documented multi-round annotation protocol, including handling of nested entities.

Authors’ abstract

The extraction of critical information from crime-related documents is a crucial task for law enforcement agencies. Named-Entity Recognition (NER) can perform this task in extracting information about the crime, the criminal, or law enforcement agencies involved. However, there is a considerable lack of adequately annotated data on general real-world crime scenarios. To address this issue, we present CrimeNER, a case-study of Crime-related zero- and Few-Shot NER, and a general Crime-related Named-Entity Recognition database (CrimeNERdb) consisting of more than 1.5k annotated documents for the NER task extracted from public reports on terrorist attacks and the U.S. Department of Justice's press notes. We define 5 types of coarse crime entity and a total of 22 types of fine-grained entity. We address the quality of the case-study and the annotated data with experiments on Zero and Few-Shot settings with State-of-the-Art NER models as well as generalist and commonly used Large Language Models.