Hierarchical Graph Convolutional Network Approach for Detecting Low-Quality Documents

AI summary

Overview

Research area: Natural language processing — automatic detection of low-quality news documents (internal inconsistency and headline–body incongruence), using graph neural networks and pre-trained language models.

Technical level: Intermediate. Readers should be comfortable with BERT-style sentence embeddings, attention mechanisms, and graph convolutional networks.

Scope: The paper proposes a hierarchical graph convolutional network (HGCN) that represents a news article's title, subheading, body text, and image caption as connected nodes in a graph, and trains it to flag two different kinds of low-quality documents.

What This Paper Is About

News articles can be low-quality in two distinct ways: their body text can drift out of sync with itself (inconsistency), or their headline can fail to match what the body actually says (incongruence). Most existing models are built to catch only one of these problems, because the datasets they are trained on only contain one type of corruption. The authors build a new 216,512-document dataset that injects inconsistency into real news articles and then design a single model that handles both failure modes by modeling the document's parts as a graph.

Key Contributions

1. An Inconsistency Dataset of 216,512 documents. Built from real Yonhap News articles published over 22 months (January 2019 to October 2020), with title, subheading, body_text, image_caption, category, class code, desk code, and send date. Inconsistent versions are created automatically by swapping sentences between two articles from the same class code, with a shuffle ratio of 0.5.

2. A universal HGCN architecture. A hierarchical graph convolutional network that treats title, subheading, body_text, image_caption, headline, body, and document as nodes in an undirected graph, so a single model detects both within-document inconsistency and headline–body incongruence.

3. State-of-the-art results on the constructed dataset. HGCN reaches 91.20% accuracy on the Inconsistency Dataset, outperforming the strongest baseline (the Headline-Body Attention Model at 83.12%) with statistical significance at p < 0.01 by a paired t-test.

4. Evidence that meta-data matters. An ablation study shows that removing subheading and image_caption drops accuracy from 91.20% to 80.78%, quantifying the value of document meta-data that prior models typically ignored.

Main Findings

• Inconsistency detection (constructed dataset): HGCN achieves 91.20% accuracy. Baselines: TF-IDF + SVM 73.96%, GRU-AVG 64.76%, GRU-LAST 60.41%, 3HAN 74.37%, AHDE 78.12%, Headline-Body Attention Model 83.12%.

• Incongruence detection (Jang et al., 2022 dataset): HGCN reaches 92.00%, second to the Headline-Body Attention Model at 93.68% — a gap of 1.68% points. The authors explain that the Jang et al. model sees only headline–body mismatch, so it performs well there, while HGCN trades a small amount of that specialization for generality.

• Cross-dataset contrast is the key argument: The Headline-Body Attention Model drops from 93.68% on the incongruent dataset to 83.12% on the inconsistency dataset, while HGCN stays above 91% on both.

• Number of graph layers: Accuracy by GCN depth was 85.60% (1 layer), 86.70% (2), 88.20% (3), 91.20% (4), and 89.00% (5). Performance rose through four layers and fell at five, consistent with over-smoothing; four layers were used throughout.

• Headline attention weighting (alpha): 87.80% at 0.1, 87.80% at 0.2, 91.20% at 0.3, 88.40% at 0.4, 89.20% at 0.5, 88.90% at 0.6, 88.10% at 0.7. Alpha was fixed at 0.3.

• Meta-data ablation: Title + body text only: 80.78%. Adding subheading: 86.10%. Adding image caption: 89.02%. All four components: 91.20%. Excluding both meta-data fields costs roughly 10 percentage points.

• Where the model fails (qualitative analysis): Three recurring error types — articles missing a subheading or image caption (or having only very short ones); documents in specialized domains such as medicine, law, and accounting; and documents that keep the same general topic but subtly swap in different content.

• Training setup: Batch size 92, learning rate 1e-05, AdamW optimizer, hidden dimension 768, dropout 0.1, alpha 0.3, 4 GCN layers. Hardware: Intel Xeon Silver 4210R CPU @ 2.40 GHz and four TITAN RTX GPUs. Train/validation/test split of 9:0.5:0.5 gives 194,860 / 10,826 / 10,826 samples.

Methodology in Plain English

Building the data. The authors start with real news articles that have already passed editorial verification, then corrupt half of them automatically. Three rules keep the corrupted data hard: the two articles used for a swap must share the same class code (so the swapped sentence isn't obviously from a different topic), they must have similar body lengths (so the model can't just learn "long documents are fake"), and their send dates must not overlap (to avoid false negatives). Sentences are then exchanged at a fixed shuffle ratio of 0.5.

Turning each document into a graph. Each part of the article is first converted into a vector using BERT, taking the last layer's [CLS] token. Title, subheading, and image caption are short enough to encode as a single vector; body text is encoded sentence by sentence, padded or truncated to k = 21, the 80th percentile of sentence counts across the corpus.

Building headline and body representations. The title and subheading vectors are concatenated and passed through a 2-layer MLP to produce one headline vector. For the body, multi-head self-attention with eight heads lets each sentence see the others, capturing internal inconsistency. That attention output is then dotted with the headline vector to score how well each body sentence matches the headline — the incongruence signal. A weighted combination controlled by alpha blends the two signals, and the result is multiplied back into the attention output to produce the body text vector. Body text and image caption are concatenated through a second MLP to form the body vector.

Graph convolution and classification. All component vectors become nodes in an undirected graph, with a document node on top. Information is propagated with graph convolution layers using the standard normalized adjacency formulation, with a 2-layer MLP for nonlinear transformation and skip-connections between layers to combat over-smoothing. After four layers, the document node goes through a 2-layer MLP for binary classification.

Training objective. A standard binary cross-entropy loss over the labels (consistent vs. inconsistent).

Why This Matters

Impact on research. Prior work treats inconsistency and incongruence as separate tasks with separate datasets, which makes models brittle in deployment where both problems coexist. This paper shows that a single architecture can handle both without collapsing on either, and it contributes a large-scale dataset plus an automated construction pipeline — addressing what prior work identified as the two biggest obstacles in this area: a lack of large datasets and the absence of automated preparation pipelines. The paper states that all code and datasets will be released at https://github.com/nowzer0/HGCN-for-Detecting-Low-Quality-Documents for research purposes.

Real-world applications:

• News aggregation and recommendation feeds — filtering or down-ranking articles whose body contradicts their headline before they reach readers.
• Publisher-side editorial QA — flagging drafts where the headline, subheading, and body drift apart during editing.
• Adversarial content monitoring — catching articles that begin on-topic and then drift into unrelated promotional material.
• Media credibility tooling — reducing the reader fatigue and loss of trust that the paper attributes to low-quality content.

Industry relevance. The model is described as resource-efficient relative to large language models, which makes it viable for high-volume screening. The authors also note that Yonhap News is one of South Korea's largest outlets, so the dataset reflects realistic editorial structure — including subheadings and image captions, which most prior clickbait models discard.

Future Directions

1. A two-phase LLM pipeline. The authors propose running the lightweight HGCN first, then routing only the samples it misclassifies to a large language model — capturing LLM accuracy without paying its cost on every document.

2. Richer contextual understanding. The qualitative analysis argues that the remaining errors require grasping sequences, numerics, dates, and domain knowledge in fields like medicine, law, and accounting, which the current architecture does not explicitly model.

3. Broader low-quality taxonomy. The paper's stated goal is to expand beyond inconsistency and incongruence to other low-quality document types, and to incorporate more of the available meta-data (category, class code, desk code, send date are present in the dataset but not all are used as model inputs).

4. Generalization beyond Korean news. The authors claim universal applicability across media, but the paper as presented reports evaluation on the Yonhap-derived dataset and the Jang et al. (2022) dataset only; other languages and media types are not reported.

Target Audience

NLP researchers and graduate students working on misinformation, clickbait, and text coherence; applied machine learning engineers building content-quality filters for news platforms or recommendation systems; and dataset builders interested in automated corruption pipelines that avoid length and topic shortcuts. Readers new to graph neural networks will need background reading, but the document-as-graph framing is intuitive enough for practitioners familiar with BERT-based classification to follow.

Authors’ abstract

Consistency within a document is a crucial feature indicative of its quality. Recently, within the vast amount of information produced across various media, there exists a significant number of low-quality documents that either lack internal consistency or contain content utterly unrelated to their headlines. Such low-quality documents induce fatigue in readers and undermine the credibility of the media source that provided them. Consequently, research to automatically detect these low-quality documents based on natural language processing is imperative. In this study, we introduce a hierarchical graph convolutional network (HGCN) that can detect internal inconsistencies within a document and incongruences between the title and body. Moreover, we constructed the Inconsistency Dataset, leveraging published news data and its meta-data, to train our model to detect document inconsistencies. Experimental results demonstrated that the HGCN achieved superior performance with an accuracy of 91.20% on our constructedInconsistency Dataset, outperforming other comparative models. Additionally, on the publicly available incongruent-related dataset, the proposed methodology demonstrated a performance of 92.00%, validating its general applicability. Finally, an ablation study further confirmed the significant impact of meta-data utilization on performance enhancement. We anticipate that our model can be universally applied to detect and filter low-quality documents in the real world.