Extracting Dataset Mentions in Forced Displacement and FCV Documents: A Weakly Supervised Framework with LLM-Based Label Refinement

AI summary

Overview

Research Area: Natural Language Processing, specifically information extraction and dataset-mention detection in specialized document collections (humanitarian, displacement, and Fragile/Conflict/Violence contexts).

Technical Level: Intermediate — readers should be comfortable with NER concepts, weak supervision, and LLM prompting, though the paper explains its pipeline clearly enough for newcomers to follow the overall logic.

Scope: This paper presents a weakly supervised pipeline that adapts an existing research-literature dataset extractor to forced-displacement and FCV documents using LLM-assisted label refinement and synthetic data, without requiring a large manually annotated training corpus.

What This Paper Is About

Development and humanitarian organizations produce and fund surveys, registries, and assessments, but tracking where these datasets are actually referenced is difficult because dataset citation practices are inconsistent and references are buried in unstructured text (research papers, project documents, humanitarian reports). Building a supervised extractor for these specialized documents normally requires expensive expert annotation that rarely exists. The paper's goal is to adapt an existing dataset-mention extractor to this new domain cheaply, using a frontier LLM to clean up noisy predictions and synthetic data to fill coverage gaps.

Key Contributions

1. A weakly supervised framework that combines an existing off-the-shelf extraction model, LLM-assisted annotation adjudication, and targeted synthetic/contrastive example generation to build domain-specific training data when labeled examples are scarce.
2. Supervision transfer to a lightweight local model: the pipeline distills LLM-refined annotations back into a small deployable model (GLiNER2) so that large-scale inference does not require an LLM at runtime.
3. An independent, heterogeneous gold-standard benchmark of 1,706 chunks spanning research (PRWPs), humanitarian (UNHCR/ReliefWeb), analytical (SEIS), and operational (World Bank Project Appraisal Documents) document types, enabling evaluation under both data-rich and data-sparse conditions.
4. Empirical demonstration of cross-domain transfer: the adapted model achieves near-90% precision on passages that actually contain dataset references, and stable passage-level specificity (>86%) across all four document collections.

Main Findings

• Mention-level extraction (full benchmark): 74.1% precision and 70.5% recall across all 1,706 chunks.
• Mention-level extraction (positive-only passages): Precision rises to 89.5%, showing that once the model is inside a passage containing a dataset reference, nearly nine in ten extractions are correct.
• Consistent positive-passage precision across genres: 88.3% on World Bank Policy Research Working Papers, 88.5% on UNHCR/ReliefWeb, 89.6% on SEIS, and 100% on the small PAD subset.
• Passage-level detection: 88.2% accuracy, 87.2% recall, and 88.6% specificity in distinguishing chunks with dataset references from those without — notable given that 68% of the benchmark chunks contain no dataset.
• Operational documents are the hardest case: Of 203 PAD candidates generated during weak supervision, 145 were rejected by the LLM. Project/program titles comprised 50.3% of rejections, administrative software and monitoring systems 17.2%, with the rest being publications and headings.
• Class imbalance affects F1 interpretation: The PAD subset (only 7 positives out of 129) yields 89.1% accuracy and 91.0% specificity but a low 36.4% F1, illustrating why accuracy and specificity should be reported alongside F1 for rare-event extraction.

Methodology in Plain English

The researchers avoided the usual path of manually labeling thousands of documents. Instead, they built a four-stage pipeline:

1. Candidate generation. They took GLiNER2, an existing lightweight model already trained to spot dataset mentions in general research papers, and ran it over unlabeled FCV and forced-displacement documents at a very permissive confidence threshold (0.15). This deliberately over-generates: it flags anything that might be a dataset reference, accepting plenty of false positives in exchange for not missing real ones.

2. LLM refinement. Each candidate is passed to GPT-4o together with its surrounding context. The LLM acts as a judge: it decides whether the span really refers to a structured data resource (survey, registry, census, microdata collection), rejects it if it actually refers to a project, organization, method, publication, or administrative tool, and corrects the span boundaries when the initial extraction was too short or too long. Structured output via a Pydantic schema keeps the responses machine-readable.

3. Synthetic and contrastive augmentation. Because the real documents only cover so much ground, they prompted GPT-4o-mini with over 100 templates across six thematic domains to generate additional passages containing dataset references — including specific FCV resources like SEIS, MSNA, and mVAM. They also generated "contrastive" examples where similar phrasing should or should not be extracted (e.g., "LSMS team" describes people, not the survey; "we conducted household surveys" describes an activity, not a dataset).

4. Fine-tuning and deployment. The real-document annotations and synthetic examples were combined to fine-tune GLiNER2. The frontier LLM is discarded at this point — the smaller model handles inference independently at a stricter threshold of 0.40, making large-scale processing feasible.

Evaluation design: The gold-standard holdout was constructed separately with human adjudication, split at the document level to prevent leakage. Mentions are scored by token-level Jaccard overlap (≥0.5 counts as a match), and both mention-level extraction and passage-level binary classification are reported.

Why This Matters

Impact on research. The paper demonstrates a reusable recipe for domain adaptation when labeled data are scarce: use a weak source model to propose, use a strong LLM to adjudicate, use synthetic data to fill gaps, then distill back into a lightweight model. This is directly relevant to any low-resource extraction task where expert annotation is the bottleneck — not just dataset mentions.

Real-world applications:

• Data investment monitoring: Organizations like the World Bank and UNHCR can trace where their funded surveys and registries are actually being cited in research, policy, and operational documents, providing evidence for future funding decisions.
• Gap analysis in the displacement data landscape: Combined with information about which datasets exist and whom they cover, observed patterns of use and non-use can highlight populations or topics that are under-served by existing data.
• Humanitarian operational intelligence: Automated detection of dataset references in field reports and assessments can help coordination bodies understand which data sources inform field-level decision-making.
• Corpus curation and metadata enrichment: Extracted mentions (with acronym, publisher, geography, and reference population fields) can feed into dataset catalogues and citation-tracking systems.

Industry relevance. Any organization that produces or depends on structured data across unstructured document collections — pharmaceutical companies tracking clinical trial datasets, financial regulators tracking economic indicators, publishers tracking data availability statements — faces the same problem: references are inconsistent and manual tracking doesn't scale. The pipeline pattern (weak labels + LLM refinement + synthetic contrastive data + distillation) generalizes well beyond this specific domain.

Future Directions

1. Broader corpus construction for substantive analysis. The authors explicitly note that their benchmark is a technical validation set, not a representative corpus. Applying the framework to characterize the actual displacement data landscape would require systematic corpus building, displacement-relevance filtering, and dataset resolution/linking across research, government policy, and operational documents.

2. Testing transfer to policy corpora. The benchmark does not include national government strategies or policy documents, which differ again in style and citation conventions. Whether the model transfers there remains an open empirical question.

3. Improving recall without sacrificing precision. Recall (62.5%–71.7% across collections) is meaningfully lower than precision. Since overall pipeline recall depends on the initial candidate generator, better candidate-generation strategies — or a two-pass architecture — could close this gap.

4. Linking extracted mentions to actual datasets. The current work stops at identifying textual references. The natural next step is resolving mentions to canonical dataset records (persistent identifiers, catalogues), and then classifying how the referenced data are used, which the authors flag as required for the downstream landscape analysis they envision.

Target Audience

This paper is most valuable to:

• NLP researchers working on low-resource information extraction, who want a concrete case study in combining weak supervision, LLM adjudication, and synthetic contrastive data.
• Data scientists and engineers at development, humanitarian, and statistical organizations who need practical methods for tracking data use across document collections without building annotation pipelines from scratch.
• Research data management and open-science practitioners interested in automated approaches to dataset citation tracking and discoverability.
• Applied ML practitioners in any domain where an existing model works reasonably well on a source corpus but degrades on a target corpus, and where labeled target data doesn't exist.

Authors’ abstract

Development and humanitarian organizations produce and support surveys, administrative registries, and other data resources to inform research, policy, and operations, yet systematically identifying where these datasets are referenced remains difficult. Such references are dispersed across research papers, project documents, humanitarian reports, and other unstructured text, limiting both the ability to trace data use and to identify potential gaps in data availability or dissemination. We present a weakly supervised framework for adapting dataset extraction to forced displacement and Fragile, Conflict, and Violence (FCV) documents without first constructing a large manually labeled training corpus. A lightweight model trained on general research literature generates candidate dataset mentions from unlabeled domain documents, which a frontier large language model (LLM) reviews in context, validating or rejecting candidates and correcting their extraction boundaries. The resulting annotations are supplemented with targeted synthetic and contrastive examples and used to fine-tune the lightweight model for large-scale extraction. We evaluate the resulting model on an independent gold-standard benchmark of 1,706 text passages spanning research, humanitarian, and operational documents. Across the full benchmark, the model achieves 74.1\% precision and 70.5\% recall at the mention level; among passages containing dataset references, precision reaches 89.5\%. At the passage level, the model achieves 88.2\% accuracy and 88.6\% specificity in distinguishing passages with dataset references from those without them. These results demonstrate a practical approach for constructing domain-specific supervision when labeled data are limited, and provide a technical foundation for larger-scale analysis of data use and potential gaps in the displacement data landscape.