ClaimPT: A Portuguese Dataset of Annotated Claims in News Articles

AI summary

Overview

Research area: Natural Language Processing — automated fact-checking, claim detection, and low-resource language resources. Technical level: Intermediate (readers should know basic NLP concepts such as token classification, fine-tuning, and evaluation metrics like F1). Scope: Introduces ClaimPT, the first large-scale, openly licensed dataset of European Portuguese news articles annotated at the span level for verifiable factual claims, together with an annotation scheme and baseline detection models.

What This Paper Is About

Automated fact-checking pipelines begin with claim detection: finding the specific statements worth verifying. Almost all existing annotated data for this task is in English and comes from social media or political debates, leaving languages like Portuguese without the resources needed to build localized tools. The authors partner with Portugal's national news agency, LUSA, to build and release ClaimPT — a manually annotated corpus of professionally edited news articles that captures the distinctive ways claims appear in journalistic writing (embedded in quotations, attributed to named sources, and surrounded by predominantly non-claim text).

Key Contributions

1. The ClaimPT dataset: 1,308 European Portuguese news articles from LUSA spanning ten topics, containing 6,875 individual annotations (463 claims, 4,393 non-claims, plus metadata and claim-attribute annotations), released openly on GitHub with a persistent DOI and CC BY 4.0 license.
2. A new three-layer annotation scheme tailored to journalistic text, extending the English NewsClaims schema with a temporal dimension, claimer-type attributes, and an identity link that resolves pronouns and abbreviated references (e.g., "he" → "Prime Minister António Costa") to their antecedents.
3. A rigorous annotation methodology and quality analysis: two trained annotators per article plus a curator, with inter-annotator agreement reported using overlap-based metrics (Jaccard, Sørensen–Dice) that are better suited to span annotation than Krippendorff's Alpha.
4. Baseline benchmarks for claim detection framed as span classification, using a fine-tuned BERTimbau encoder and two Gemini generative models, with a detailed error analysis of why generative models struggle.

Main Findings

• Claims are rare in news: Only 9.53% of annotated statements are claims (463) versus 90.47% non-claims (4,393), an average of 0.42 claims per document. The authors argue this reflects the genuine editorial character of news and creates a realistic but difficult class-imbalance setting.
• Claim identification is inherently subjective: Inter-annotator agreement on Claim vs. Non-Claim is moderate (F1 = 0.50, Jaccard = 33.65%), consistent with prior work reporting 30–50% agreement. The main source of disagreement was the guideline that statements from inherently authoritative sources (e.g., the WHO) should not be annotated.
• Claim attributes are far more reliable than claim detection: Once annotators agree a claim exists, span (F1 = 0.76), claimer (F1 = 0.80), and time (F1 = 0.93) are annotated with high agreement. Claim object is the weakest attribute (F1 = 0.44) because it requires interpretation.
• Encoder models beat generative models: Fine-tuned BERTimbau with sentence-level segmentation achieved the best micro-average F1 (66.38%), versus 36.50% for Gemini-2.5-Flash and 22.52% for Gemini-2.5-Flash-Lite. On claims specifically, BERT-Sent reached F1 = 30.57.
• Generative models fail in characteristic ways: They confuse journalistic quotations with claims made by social actors, conflate unverifiable generalizations with real claims, and extend predictions beyond the quoted span. They do, however, show partial ability to separate factual content from opinion within a sentence.
• Topic distribution is skewed: International news (21.79%) and politics (13.84%) dominate at the article level; political (22.68%) and societal (21.60%) claims dominate at the claim level. Sports and technology contribute almost no claims.
• Train/test split preserves distribution: An 80/20 document-level split maintains the same Claim:Non-Claim ratio (1:9.48) and attribute proportions across both sets.

Methodology in Plain English

The authors first worked out a data-access agreement with LUSA, Portugal's national news agency. They pulled an initial batch of articles, ran a 100-article pilot to see where claims actually appear, and discovered that politics and international news were claim-rich while sports had almost none. They also found that claims concentrate in direct speech, so they refined their retrieval to target sentences containing reporting verbs like "said" or "declared." Two collection phases produced 1,308 articles covering January 2022 to December 2023.

For annotation, they designed a scheme with three layers: article metadata (topic and publication date); a claim/non-claim distinction; and fine-grained components of each claim (span, object, claimer, time, stance, claim topic). They diverged from earlier schemes by treating claims not as isolated units but as entities linked to their context — including referential links that connect pronouns back to the people they refer to.

Two annotators with communication-studies backgrounds each labeled every article independently over six months, using the INCEpTION platform. A linguistics-trained curator reviewed everything, and a senior researcher adjudicated unresolvable disagreements. Annotators went through two trial rounds first, which also established a productivity baseline of roughly 20 minutes per article.

For the benchmark experiments, the team framed claim detection as a span-classification task: given text, predict triples of (start, end, label). They fine-tuned BERTimbau, splitting long documents either by sentence or by 512-token chunks with 128-token overlap. They also prompt-engineered two Gemini models with a one-shot System 2 Attention prompt and extracted structured output via LangExtract. All models were scored with span-level precision, recall, and F1, where a prediction counts as correct only if both boundaries and the label match exactly.

Why This Matters

Portuguese is spoken by over 260 million people, yet had no large, openly licensed, professionally annotated resource for claim detection. ClaimPT fills that gap and, because it targets news articles rather than social media, addresses a domain that is both more authoritative and more underexplored than tweets or parliamentary transcripts. The inclusion of referential links, temporal annotation, and claimer-type attributes gives downstream systems richer context than a simple binary classifier would receive.

Real-world applications:

• Newsroom triage tools: Fact-checking desks could automatically surface check-worthy statements from incoming wire copy, prioritizing which claims deserve a reporter's time.
• Media monitoring and accountability journalism: Tracking what public figures and institutions assert over time, and whether those assertions align with evidence.
• Platform-level misinformation detection: Flagging verifiable factual claims in Portuguese-language news shared on social media for escalation to human reviewers.
• Low-resource language transfer: The annotation guidelines are designed to be reused, so researchers working on other under-resourced languages can adapt the scheme rather than starting from scratch.

Industry relevance: News agencies, fact-checking organizations (such as Polígrafo in Portugal), and platform trust-and-safety teams all depend on scalable claim detection. The paper's finding that a fine-tuned monolingual encoder outperforms general-purpose large language models is directly actionable: it suggests that for domain-specific span extraction in a mid-resource language, investing in annotated data and targeted fine-tuning pays off more than prompting a frontier model.

Future Directions

1. Multilingual extension: Applying the same annotation guidelines to build parallel corpora in other languages, enabling cross-lingual claim detection and transfer learning.
2. Stronger baselines: Testing newer architectures such as ModernBERT and exploring better prompt strategies or fine-tuned generative models for span extraction, since the current generative results are weak.
3. Explainable and structured models: Leveraging the annotated claim attributes (claimer, time, stance, topic) to build systems that not only detect claims but explain who made them, when, and about what — closer to how human fact-checkers actually reason.
4. Resolving the annotation ambiguity: The moderate Claim/Non-Claim agreement signals a genuine open question about how to define check-worthiness consistently, particularly around authoritative sources and quotations embedded in journalistic prose.

Target Audience

NLP researchers working on fact-checking, misinformation, and claim detection; computational social scientists studying news discourse; and practitioners building Portuguese-language media tools. The paper is also useful for researchers in other low-resource languages who want a reusable template for how to design an annotation scheme, run a rigorous multi-annotator workflow, and report agreement metrics appropriately for span-based tasks. Readers with a background in linguistics or journalism studies but limited ML experience will find the annotation sections accessible; the baseline modeling sections assume some familiarity with transformer fine-tuning and evaluation methodology.

Authors’ abstract

Fact-checking remains a demanding and time-consuming task, still largely dependent on manual verification and unable to match the rapid spread of misinformation online. This is particularly important because debunking false information typically takes longer to reach consumers than the misinformation itself; accelerating corrections through automation can therefore help counter it more effectively. Although many organizations perform manual fact-checking, this approach is difficult to scale given the growing volume of digital content. These limitations have motivated interest in automating fact-checking, where identifying claims is a crucial first step. However, progress has been uneven across languages, with English dominating due to abundant annotated data. Portuguese, like other languages, still lacks accessible, licensed datasets, limiting research, NLP developments and applications. In this paper, we introduce ClaimPT, a dataset of European Portuguese news articles annotated for factual claims, comprising 1,308 articles and 6,875 individual annotations. Unlike most existing resources based on social media or parliamentary transcripts, ClaimPT focuses on journalistic content, collected through a partnership with LUSA, the Portuguese News Agency. To ensure annotation quality, two trained annotators labeled each article, with a curator validating all annotations according to a newly proposed scheme. We also provide baseline models for claim detection, establishing initial benchmarks and enabling future NLP and IR applications. By releasing ClaimPT, we aim to advance research on low-resource fact-checking and enhance understanding of misinformation in news media.