Learning to Generate and Extract: A Multi-Agent Collaboration Framework For Zero-shot Document-level Event Arguments Extraction

AI summary

Overview

Research area: Natural Language Processing / information extraction, specifically zero-shot document-level event argument extraction (ZS-DEAE).

Technical level: Advanced. The paper assumes familiarity with reinforcement learning, autoregressive language model fine-tuning (LoRA), and event-schema formalisms.

Scope: This paper proposes a two-agent framework — a generation agent and an evaluation agent — that collaborate in a "Propose–Evaluate–Revise" loop, trained with reinforcement learning, to synthesize training data for unseen event types and simultaneously improve argument extraction on the RAMS and WikiEvents datasets.

What This Paper Is About

Document-level event argument extraction needs labeled data, but labeled data for new ("unseen") event types is scarce. Existing attempts to have large language models generate synthetic training data rely on event-type-only prompts, which produce contexts that miss the cross-sentence and structural complexity of real documents, and there is no mechanism for judging whether the synthetic data is any good. The paper's goal is to build a system where one agent generates synthetic event instances and a second agent both extracts arguments from them and scores their quality, with those scores feeding back as reward signals that improve both agents.

Key Contributions

1. A multi-agent collaboration framework for ZS-DEAE. The framework simulates a human "Propose–Evaluate–Revise" cognitive process with a generation agent that produces document-level contexts, triggers, and role–argument pairs, and an evaluation agent that extracts arguments and judges semantic consistency with the context.
2. A reward design that combines log-likelihood with event-structure constraints. The evaluation agent's log-likelihood is normalized and then penalized when a sample's proportion of empty (None) arguments deviates from the proportion observed in the training data, preventing the loop from rewarding structurally incomplete events.
3. Iterative reinforcement-learning optimization of both agents. Policy-gradient updates are applied to the generation agent and the evaluation agent using the combined reward, so the two improve together over repeated rounds.
4. Demonstrated generality on three zero-shot settings built from RAMS and WikiEvents. The method improves both synthetic data quality and extraction performance, and the generated data also raises the zero-shot performance of other DEAE models when added to their training sets.

Main Findings

• Outperforms existing DEAE models across all three zero-shot settings. On RAMS2RAMS, "Ours (LLaMA)" reaches 46.46 Seen R., 45.06 Unseen R., and 45.77 Overall Span-F1; "Ours (Qwen)" reaches 44.06, 45.11, and 44.59. The paper states that Ours (LLaMA) exceeds the strongest baseline, DEEIA, by 6.57, 5.57, and 7.82 F1 on seen roles, unseen roles, and overall in RAMS2RAMS.
• Strongest overall F1 on the other two settings. The method reports the highest overall F1 scores of 32.38 (RAMS2Wiki) and 46.96 (Wiki2Wiki). On RAMS2Wiki the "Ours (Qwen)" variant scores 31.18 overall, and on Wiki2Wiki "Ours (Qwen)" scores 47.62 overall.
• Clear gains over the strongest zero-shot baseline, Bart-Gen. The paper reports overall F1 improvements over Bart-Gen of 7.53, 3.86, and 6.14 across the three settings.
• LLMs prompted directly perform poorly under Span-F1. Phi-4, Gemma-1.1, Mixtral, LLaMA-3.1, GPT-4o, DS-V3, and DS-R1, with and without Chain-of-Thought prompting, score much lower overall — for example GPT-4o reaches 19.64 (RAMS2RAMS), 10.17 (RAMS2Wiki), and 11.36 (Wiki2Wiki) without CoT. The paper attributes this to Span-F1's requirement of exact span boundary matching.
• Both reward and structural constraint matter. In the ablation, removing the reward lowers Ours (LLaMA) from 45.28/32.38/46.96 to 42.46/24.93/38.39 on R2R/R2W/W2W; removing the constraint gives 44.72/29.03/40.98. The constraint produces the smaller drop, but its absence increases the proportion of empty arguments and harms data quality and extraction accuracy.
• Generated data transfers to other models. Adding the synthetic data to TabEAE's training data raises its scores from 36.22/26.74/30.97 to 44.43/28.42/33.36, and adding it to Bart-Gen's raises 38.53/28.52/40.82 to 46.06/32.38/46.96 — outperforming synthetic data produced directly by LLaMA 3.1 in these comparisons.
• Performance peaks early in the interaction loop. F1 reaches its peak within one to two interaction rounds and then gradually declines, which the paper links to reduced diversity of generated samples. Both agents' losses decline steadily during reinforcement learning.
• Diversity shrinks along three of four dimensions. Following Kirk et al. (2024) and the four-dimensional analysis of Guo et al. (2024) — lexical, semantic, logical, syntactic — diversity decreases in the lexical, semantic, and syntactic dimensions as rounds increase, while logical-level diversity stays relatively stable.
• The evaluation agent is sensitive to data quality. When arguments are randomly removed ("Empty") or role–argument mappings are randomly swapped ("Mismatch"), the evaluation agent assigns lower log-likelihood scores than to correctly generated ("Normal") data.
• Qualitative case study. Compared with GPT-4o and LLaMA3.1 (70B) outputs, which tend to be short, syntactically simple, and densely packed with arguments (with LLaMA3.1 70B often omitting the trigger and key arguments such as place and place of employment), the proposed method (using LLaMA3.1 8B) generates longer documents with more widely dispersed arguments; one case shows a context–event-type mismatch producing a low reward.

Methodology in Plain English

The task is split between two agents that play different roles, like a writer and an editor.

The generation agent is given only an unseen event type and its list of roles, and is asked to write a coherent document-level context containing an event trigger plus a role–argument pair for each role, marking missing roles as None. It is built on LLaMA3.1-8B or Qwen2.5-7B and fine-tuned with LoRA (rank 8, scaling factor 32, dropout 0.05, applied to query and value projections). Samples that omit the trigger are discarded; samples that omit arguments keep None in those slots.

The evaluation agent is Bart-Gen built on Bart-large. It receives the generated context plus an unfilled template with <arg> placeholders and fills in the arguments, with its output vocabulary restricted to tokens present in the document to reduce hallucination. The probability it assigns to the filled template acts as a quality score: higher log-likelihood means the generated data is semantically consistent with what the extractor can recover.

That log-likelihood is normalized by the mean and standard deviation across the whole synthetic dataset, then adjusted by a penalty term. The penalty fires when the share of empty arguments in a sample drifts outside the range observed in the training data, which counters the observed failure mode where the evaluator happily gives high scores to samples full of None values. The resulting score becomes the reward, and policy-gradient updates are applied to both agents so that the generator learns to write better data and the evaluator learns to judge better. Training runs for five interaction rounds with three random seeds, reporting the average of the best-performing round from each run.

Experiments use three zero-shot settings — RAMS2RAMS, RAMS2Wiki, and Wiki2Wiki — where training event types and test event types are disjoint, though roles may be shared. The metric is Span-F1, which counts a prediction correct only if the span exactly matches the gold span. The number of candidate samples K per unseen event type and the underlying dataset sizes are not reported in the paper content provided.

Why This Matters

• Research impact: The paper reframes zero-shot event extraction as a data-generation-and-curation problem rather than only a knowledge-transfer problem, and shows that making the evaluator part of the training loop — via a reward signal — is what makes the synthetic data usable. It also provides evidence that LLM-generated data can be transferred into other DEAE models to improve their zero-shot behavior.
• Knowledge base construction: Extracting event participants from documents without labeled examples for each event type supports building and expanding knowledge graphs.
• Information retrieval: Structured event-argument records improve retrieval quality over raw document text.
• Low-resource and new domains: New event schemas appear constantly in news, legal, biomedical, and financial text; a generation-plus-evaluation loop offers a way to bootstrap extraction without large annotation campaigns.
• Industry relevance: Organizations that need event monitoring from documents — such as compliance, financial risk, and intelligence workflows — can reduce reliance on costly manual annotation by using a generator-evaluator pair, while the structural constraint gives a practical safeguard against synthetic data that looks plausible but carries no real arguments.

Future Directions

• Address the diversity collapse. Since lexical, semantic, and syntactic diversity fall as interaction rounds increase, and performance peaks at one to two rounds then declines, future work needs mechanisms that preserve diversity across rounds.
• Reduce sensitivity to randomness. The method already relies on three random seeds and reports the best round per run; making results stable across seeds and rounds is an open problem.
• Improve empty-argument handling without penalizing valid cases. The paper notes that logical-level diversity remains relatively stable because lexical and syntactic convergence introduces inconsistencies; how structure constraints interact with that effect is unresolved.
• Extend beyond ZS-DEAE. The conclusion states that future work will extend the framework to broader information extraction tasks and other low-resource scenarios.

Target Audience

Researchers and practitioners in natural language processing and information extraction working on event extraction, zero-shot or low-resource settings, synthetic data generation with large language models, or multi-agent and reinforcement-learning-based training pipelines. It is also relevant to engineers building document-level knowledge extraction systems who need to judge whether LLM-generated training data is trustworthy enough to use.

Authors’ abstract

Document-level event argument extraction (DEAE) is essential for knowledge acquisition, aiming to extract participants of events from documents.In the zero-shot setting, existing methods employ LLMs to generate synthetic data to address the challenge posed by the scarcity of annotated data. However, relying solely on Event-type-only prompts makes it difficult for the generated content to accurately capture the contextual and structural relationships of unseen events. Moreover, ensuring the reliability and usability of synthetic data remains a significant challenge due to the absence of quality evaluation mechanisms. To this end, we introduce a multi-agent collaboration framework for zero-shot document-level event argument extraction (ZS-DEAE), which simulates the human collaborative cognitive process of "Propose-Evaluate-Revise." Specifically, the framework comprises a generation agent and an evaluation agent. The generation agent synthesizes data for unseen events by leveraging knowledge from seen events, while the evaluation agent extracts arguments from the synthetic data and assesses their semantic consistency with the context. The evaluation results are subsequently converted into reward signals, with event structure constraints incorporated into the reward design to enable iterative optimization of both agents via reinforcement learning.In three zero-shot scenarios constructed from the RAMS and WikiEvents datasets, our method achieves improvements both in data generation quality and argument extraction performance, while the generated data also effectively enhances the zero-shot performance of other DEAE models.