Improving Synthetic Data Generation for Argument Mining via Adversarial Reinforcement Learning

AI summary

Overview

Research area: Natural language processing, specifically computational argumentation (Argument Mining) and LLM-based synthetic data generation with reinforcement learning.

Technical level: Advanced. The paper combines adversarial training, Group Relative Policy Optimization (GRPO), structured annotation schemas, and reward shaping (discriminator reward plus a ROUGE-L-based source similarity penalty).

Scope: The paper proposes and evaluates an adversarial reinforcement learning framework in which a generator LLM and a discriminator LLM are trained in an alternating loop to synthesize argument-mining training data, tested on three benchmark datasets under full-data and low-resource settings.

What This Paper Is About

Argument Mining (AM) models must identify argumentative components (such as claims and premises), classify their types, and map support/attack relations between them, but high-quality annotated corpora are expensive to build and typically small — the paper notes that the AAEC dataset contains only 289 training essays. Existing LLM-based synthesis approaches, notably GPT-4o-based prompting, treat data quality and diversity as separate pipelines and depend on costly closed-source models. The goal of this paper is to generate synthetic AM data that is simultaneously structurally accurate and sufficiently diverse by jointly optimizing both properties inside one framework.

Key Contributions

1. An adversarial reinforcement learning framework for data synthesis in which a generator LLM produces structured AM instances and a discriminator LLM provides learning signals by distinguishing real annotated samples from synthetic candidates, with the two models optimized in an alternating loop.
2. A reward design that combines adversarial feedback with a rule-based novelty constraint: a clipped discriminator reward (based on the negative log-probability that the discriminator assigns to the real sample in a candidate set) is combined with a source similarity penalty computed as ROUGE-L F1 between synthetic and source text.
3. Use of GRPO to optimize the generator, converting discriminator decisions into rewards that are aggregated into relative advantages across a group of K synthetic samples.
4. Comprehensive evaluation across three AM datasets, three AM models (ST, UniASA, Directly-SFT), and multiple data settings (100%, 70%, 50%, 30%, 10%, 5%), including ablations, comparisons against GPT-4o-based synthesis baselines (QOS, DOS, QOS+DOS, FTGA, JTLS), preference-optimization and RL baselines (POPri, SEAL), and structural diversity visualization.

Main Findings

• Consistent gains across all models and settings: Adding synthetic data from the framework improved all three AM models (ST, UniASA, Directly-SFT) on all three datasets (AAEC, AbstRCT, CDCP) in both full-data and low-resource settings. On the ST model under the 5% setting, average F1 gains over training on original data alone were 8.06%, 8.14%, and 5.67% on the three datasets.
• Strongest improvements in low-resource settings: Gains are especially pronounced for the PLM-based models (ST and UniASA) at 5% training data. For the LLM-based Directly-SFT model the trend is less pronounced — on AAEC it improved by 4.47% at the 5% setting versus 5.16% at full data, which the authors attribute to stronger prior knowledge in LLM-based AM models.
• Outperforms GPT-4o-based synthesis baselines: Compared with the QOS+DOS baseline, the framework achieved average F1 improvements of 2.22%, 2.6%, and 1.84% on the three datasets under the 5% setting. The authors state the framework surpasses data generated by stronger proprietary systems such as GPT-4o.
• Best absolute results at full data (ST model): AAEC 88.16 / 79.63 / 59.87 (F1_span / F1_aci / F1_ari), average 75.89, versus Origin at 84.70 / 75.88 / 54.16, average 71.58; AbstRCT average 61.74 versus Origin 56.93; CDCP average 64.50 versus Origin 60.60.
• Low-resource results (ST, 5%): AAEC average 53.27 versus Origin 45.21; AbstRCT 51.07 versus 42.93; CDCP 48.76 versus 43.09.
• Both components matter (ablation, ST model, 100% setting): Removing discriminator refinement drops AAEC average F1 from 75.89 to 72.61; removing the similarity penalty drops it to 74.20. At 5%, the corresponding averages are 53.27 (full), 51.79 (w/o discriminator refinement), and 52.17 (w/o similarity penalty).
• Gains shrink as more real data is available: At 10%, 30%, 50%, and 70% training data (ST, AAEC), the framework's averages were 53.56, 57.54, 66.20, and 73.59, versus Origin at 46.53, 51.21, 61.06, and 69.04. An exception is F1_ari on CDCP, which does not strictly follow this trend, attributed to sparsity of relation labels in CDCP.
• More evolution rounds help, then plateau: Average performance improved with more adversarial evolution rounds, then leveled off with slight fluctuations on CDCP; the paper sets r = 3 to balance effectiveness and efficiency.
• More synthetic candidates per prompt help: All evaluation metrics consistently improved as the number of synthetic samples K increased (main experiments use K = 3).
• Structurally more diverse synthetic data: Using graph2vec embeddings of argument-annotation graphs (component types and relations only, without textual semantics) and t-SNE visualization, the synthetic data showed a broader and more dispersed structural distribution than the original data on all three datasets.
• Task-specific optimization beats transferred general methods: Under the 5% setting with ST, the method reached averages of 53.27 (AAEC), 51.07 (AbstRCT), and 48.76 (CDCP), versus POPri at 49.72 / 46.64 / 45.96 and SEAL at 49.70 / 46.60 / 45.37.
• Structure-aware reward helps on two datasets but not the third: Adding a reward computed from a frozen AM model's agreement with generated annotations raised average F1 from 53.27 to 54.08 on AAEC and from 51.07 to 52.27 on AbstRCT, but lowered it from 48.76 to 48.24 on CDCP.
• Statistical significance: The paper reports that all improvements on average F1 are statistically significant with p < 0.05, with results averaged over five random seeds.

Methodology in Plain English

The framework trains two copies of the same open-source LLM (Qwen3-8B), one acting as a generator and one as a discriminator, and lets them improve each other in alternating rounds.

• Setup: Each real annotated sample pairs a text with a structured argument annotation (component spans, component types, and relations between components). The generator is first given a basic ability to produce such pairs through supervised fine-tuning, where its targets are synthetic samples initially produced with Qwen3-8B and a detailed prompt, followed by basic error filtering.
• Generator step: For each real sample, the generator produces K synthetic candidates (K = 3). These are shuffled together with one real annotated sample into a candidate set. The discriminator must guess which sample in the set is the real one.
• Reward: If the discriminator struggles to find the real sample, the generator gets a higher reward (the negative log-probability of the correct position, clipped at a threshold c). A small constant epsilon is added for numerical stability.
• Novelty penalty: Because the generator sometimes made only shallow edits or reused original phrasing, the authors add a penalty based on ROUGE-L F1 similarity between the synthetic and source text. Below a threshold tau there is no penalty; above it the penalty grows with similarity and is bounded in [-1, 0]. The final reward is the discriminator reward plus this penalty weighted by lambda_1 = 0.5.
• Optimizing the generator: GRPO turns the group of K rewards into relative advantages, and the generator is updated to favor high-reward outputs.
• Discriminator step: After each generator update, the discriminator is retrained as a multi-class classifier over positions in the candidate set, using the index of the real sample as supervision, on the newly generated (now harder) synthetic samples.
• Repeat: These two stages alternate for r = 3 rounds, so the generator makes increasingly realistic samples and the discriminator becomes increasingly sensitive to subtle structural and annotation errors.
• Data budget: The amount of synthetic data is capped at twice the size of the original training data used in the given setting. All experiments ran on a single NVIDIA H20 GPU. Evaluation uses three fine-grained F1 scores: F1_span for component span matching, F1_aci for component type classification, and F1_ari for argumentative relation mapping, with "Avg." as their arithmetic mean.

Note: the provided text is truncated. Dataset construction details and statistics (Appendix C.1), the pseudocode (Appendix B), the task definition beyond its opening lines (Appendix A), and the supplementary analyses D.1–D.5 are referenced but not included in the available content, so their specifics are not reported here.

Why This Matters

Impact on research. The work reframes synthetic data generation for structured-output NLP tasks as a joint quality-and-diversity optimization problem rather than two separate pipelines, and shows that an open-source 8B model trained adversarially can beat data generated by a proprietary system such as GPT-4o on these benchmarks. It also provides a reusable recipe for combining adversarial feedback with rule-based constraints and GRPO-style reinforcement learning for tasks where labels have internal structure, not just free text.

Real-world applications:

• Education and essay feedback: Automatically identifying claims, premises, and support/attack relations in student writing to support argumentative writing assessment and tutoring.
• Opinion and debate analysis: Structuring large volumes of public discussion — reviews, forums, policy consultations — into claims with supporting and attacking evidence.
• Fact-checking and evidence retrieval: Mapping which statements support or attack a given claim to help retrieve supporting evidence.
• Argument generation and decision support: Supplying structured argument scaffolding for drafting or summarizing arguments in legal, scientific, and journalistic settings.

Industry relevance. The approach reduces dependence on costly closed-source APIs and on scarce expert annotation, which matters for teams building AM systems where labeled corpora are small — the paper explicitly targets low-resource conditions, where its gains are largest. The trade-off is additional compute from multiple adversarial rounds and the need for a generator with stable long-text generation ability.

Future Directions

• Reduce the cost of adversarial evolution. The authors identify the multiple optimization rounds as an inherent computational overhead and call for more efficient evolution strategies that preserve synthetic data quality.
• Reduce dependence on the generator's and discriminator's base capabilities. Performance degrades if the generator lacks stable long-text generation, so improving robustness to weaker base models is an open problem.
• Generate high-quality data from scratch. The current method still relies on existing human-annotated data as conditioning; the authors state future work should explore generating high-quality data without it.
• Make structural supervision more reliable. The structure-aware reward helped on AbstRCT and AAEC but slightly hurt on CDCP, suggesting the effect is dataset-dependent and needs further study.
• Explain and mitigate optimization limits. The paper notes slight fluctuations on CDCP possibly due to an optimization bottleneck between generator and discriminator; the appendix announces analyses of adversarial training stability and mode collapse, but their contents are not in the available text.

Target Audience

Researchers and graduate students in NLP and computational argumentation, especially those working on data augmentation for structured prediction, low-resource learning, and reinforcement learning from model feedback. It is also relevant to practitioners who need to build AM or structured-annotation pipelines with open-source models and limited labeled data, and to those studying adversarial training loops between generator and discriminator models. A working familiarity with reinforcement learning objectives (policy ratios, advantage estimation, clipping) and with the AM annotation schema is needed to follow the framework and reward design in detail.

Authors’ abstract

Argument Mining (AM) is fundamentally constrained by the scarcity of high-quality structure-annotated datasets. While LLMs have shown promise in synthetic data generation, producing synthetic AM data that is both structurally accurate and sufficiently diverse remains a challenging problem. To address this problem, we revisit synthetic data generation for AM from a new perspective and propose a novel adversarial reinforcement learning framework for data synthesis. The proposed framework jointly optimizes the generator and the discriminator in an adversarial loop, in which the generator produces structured AM instances, and the discriminator provides learning signals by distinguishing real data from synthetic candidates. This enables the generator to progressively improve both the structural accuracy of generated argument data while maintaining diversity through adversarial feedback. Extensive experiments demonstrate that the proposed framework consistently improves AM performance on three benchmark datasets in both full-data and low-resource settings, validating its effectiveness and scalability.