Extracting Events Like Code: A Multi-Agent Programming Framework for Zero-Shot Event Extraction

AI summary

Overview

Research area: Natural Language Processing, specifically zero-shot event extraction (ZSEE) with large language models and multi-agent LLM systems.

Technical level: Advanced. The paper assumes familiarity with event extraction schemas (triggers, argument roles, event types), structured prediction metrics (TI, TC, AI, AC), and LLM agent orchestration, plus Python type-validation tooling such as Pydantic and dataclass-style BaseModel definitions.

Scope: The paper proposes Agent-Event-Coder (AEC), a four-agent framework that reformulates zero-shot event extraction as an iterative, verifiable code-generation process, and evaluates it on five datasets with six LLM backbones.

What This Paper Is About

Zero-shot event extraction asks a model to find event triggers and their arguments for event types it has never seen labeled examples for, using only the event type's name or natural-language definition. The authors argue that direct prompting of LLMs fails in two ways: contextual ambiguity (polysemous triggers like "strike" being misread, since the model over-relies on the trigger word) and structural fidelity (outputs that violate the event schema through non-existent argument roles, hallucinated arguments, or wrong data types). AEC addresses both by decomposing extraction into specialized agent subtasks and representing event schemas as executable Python classes so that outputs can be deterministically validated and repaired.

Key Contributions

1. The authors reformulate zero-shot event extraction as a multi-agent code-generation task, unifying schema constraints, iterative planning, and code-based validation for structured event extraction. They state this is the first framework to do so for ZSEE.
2. They introduce the AEC multi-agent workflow, in which four specialized agents (Retrieval, Planning, Coding, Verification) collaborate to retrieve knowledge, design extraction plans, and generate structured event representations.
3. They design a schema-as-code verification loop: a dedicated verification agent applies deterministic programming-language rules to constrain outputs and supplies precise diagnostic feedback for iterative refinement.
4. They report evaluations across five domains and six LLMs, arguing AEC is a robust, generalizable, state-of-the-art ZSEE framework.

Main Findings

• AEC leads on all benchmarks in the main comparison: With Llama3-8B-Instruct and Llama3-70B-Instruct across FewEvent, ACE 2005, GENIA, SPEED, and CASIE, AEC achieves the best overall performance of the compared methods. On ACE 2005 with Llama3-8B, the paper reports gains of +7.8% and +6.0% in Trigger Identification (TI) and Trigger Classification (TC) over ChatIE, along with superior argument extraction results.
• Generalization holds across four additional LLMs: For Qwen2.5-14B, Qwen2.5-72B, GPT-3.5-turbo, and GPT-4o, AEC beats the strongest baseline (DecomposeEE) by roughly +3–5% TI, +4–6% TC, and +2–4% on argument metrics.
• Larger models do better under AEC: GPT-4o achieves the highest overall performance, followed closely by Qwen2.5-72B, which the authors attribute to stronger reasoning from larger model sizes.
• Every component contributes: Removing the Retrieval Agent causes the largest decline in trigger identification; removing Planning Rationales degrades performance; disabling the Verification Loop or the Structural Check reduces all metrics, particularly argument classification, on both Llama3-70B and GPT-4o over FewEvent and ACE.
• Hypotheses and patch attempts help up to a point: On GPT-4o, increasing the number of hypotheses k and maximum patch attempts t improves FewEvent and ACE results, but gains saturate beyond k = 3 and t = 3 (for example, FewEvent TI goes 40.5 at k=1,t=1 → 44.6 at k=3,t=3 → 45.2 at k=5,t=3 → 45.1 at k=5,t=5).
• More verification tests help up to a point: Adding test cases improves both trigger identification and argument classification, but improvements plateau beyond three cases, with additional tests adding little while increasing computational overhead.
• Agent stage effects are visible qualitatively: The Planning Agent produces plausible trigger-type hypotheses but may omit essential arguments; the Coding Agent adds schema-guided arguments, reducing role confusion; the Verification Agent corrects type errors and removes inconsistent arguments.

Methodology in Plain English

AEC treats event extraction the way a software team treats writing a program.

• Retrieval Agent: Before reasoning about the text, this agent self-generates k exemplar sentences tailored to the given event schema, modeled on the "analogical prompting" paradigm. These exemplars act as analogies that connect abstract schema roles to concrete linguistic phrasing and help disambiguate polysemous triggers.
• Planning Agent: Given the text, the schema, and the generated exemplars, this agent produces a ranked list of trigger–type hypotheses, each with a confidence score between 0 and 1 and a natural-language rationale explaining why the pair is plausible.
• Coding Agent: The highest-scoring hypothesis is converted into executable Python that instantiates a schema-compiled BaseModel class. Because the schema itself is code, the constructor enforces the expected role types, so schema compliance reduces to constructing a valid class instance.
• Verification Agent: The generated object is run through a three-stage test suite. A semantic check confirms the trigger appears in the text and is compatible with the event type; a type check confirms argument values match schema datatypes (with multiplicity constraints enforced by Pydantic); a structural check confirms the code compiles, contains exactly the fields event_type, trigger, and arguments, and produces a serializable object. The agent returns True only if all three checks pass; otherwise it names the first failed test.
• Dual-loop refinement: If verification fails, an inner loop patches the code using compiler-like diagnostics, up to t attempts. If the hypothesis is exhausted, the outer loop removes it and moves to the next-best hypothesis, exploring O(kᵗ) candidate paths.

Experimental setup: Five datasets covering five domains — FewEvent (General, 100 event types), ACE 2005 (News, 33), GENIA (Biomedical, 9), SPEED (Epidemiological, 7), and CASIE (Cybersecurity, 5). For FewEvent and SPEED, which lack argument annotations, only TI and TC are reported; ACE 2005, GENIA, and CASIE also report Argument Identification (AI) and Argument Classification (AC). Following the TextEE evaluation protocol, the authors uniformly sample 250 test instances per dataset, and 50 for the smaller CASIE dataset, under a purely zero-shot setting with no training data. Baselines are DirectEE, CEDAR, DecomposeEnrichEE, GuidelineEE, and ChatIE, all adapted to output schema-conformant objects and each given a unified Verification component for fairness. Backbones are Llama3-8B-Instruct, Llama3-70B-Instruct, Qwen2.5-14B-Instruct, Qwen2.5-72B-Instruct, GPT-3.5-turbo, and GPT-4o. Metrics are micro-averaged F1 for TI, TC, AI, and AC. No fine-tuning is performed; agents communicate through structured outputs. Results are averaged over three independent runs with exemplar count and inner-loop iterations set to k = t = 3. Open-source models run locally on NVIDIA RTX A800 machines with 4 GPUs.

Why This Matters

Impact on research: The paper argues that structured prediction with LLMs is better handled by combining step-wise multi-agent reasoning with deterministic, code-level validation, rather than prompt engineering alone. It extends multi-agent IE work — previously applied to few-shot event debate, relation extraction, clinical IE, and zero-shot NER — into the zero-shot event extraction setting with triggers, multiple arguments, and strict schemas.

Real-world applications:

• Knowledge base population, which the paper names as a motivating application for structured event records.
• Information retrieval, also named as a domain that depends on reliable event extraction.
• Question answering, listed as a downstream consumer of extracted events.
• Technical domains the evaluation covers directly: biomedical (GENIA), epidemiological (SPEED), and cybersecurity (CASIE) event monitoring.

Industry relevance: Because AEC requires no labeled training data and runs on off-the-shelf instruction-tuned LLMs without fine-tuning, it targets deployment scenarios where event types proliferate faster than annotation budgets allow. The schema-as-code design also means output validity is enforced by run-time checks rather than trust in the model, which matters for pipelines where malformed or incomplete event records would disrupt downstream processing.

Future Directions

• Tuning the refinement budget: The k and t sweep shows saturation at k = 3 and t = 3, prompting the question of how to pick these values adaptively per dataset or schema complexity instead of fixing them.
• Reducing verification cost: Performance plateaus beyond three test cases, so determining a minimal sufficient test suite — or when to stop adding checks — is left open.
• Handling multi-word triggers: The proportion of multi-word triggers varies widely across the evaluated datasets (0% in SPEED to 54.6% in CASIE), and the paper highlights this annotation disparity without proposing a dedicated solution.
• Scaling and comparing backbones further: Since GPT-4o leads and Qwen2.5-72B follows, the parameter-scaling pattern raises the question of how AEC behaves with newer or larger backbones and whether its gains persist.

Target Audience

Researchers and practitioners in natural language processing who work on information extraction, structured prediction, or multi-agent LLM systems, and who need zero-shot extraction without annotated data. It is also relevant to engineers building schema-constrained LLM pipelines in domains such as news, biomedicine, epidemiology, and cybersecurity, and to readers interested in using executable schemas and programmatic verification as a substitute for fine-tuning.

Authors’ abstract

Zero-shot event extraction (ZSEE) remains a significant challenge for large language models (LLMs) due to the need for complex reasoning and domain-specific understanding. Direct prompting often yields incomplete or structurally invalid outputs--such as misclassified triggers, missing arguments, and schema violations. To address these limitations, we present Agent-Event-Coder (AEC), a novel multi-agent framework that treats event extraction like software engineering: as a structured, iterative code-generation process. AEC decomposes ZSEE into specialized subtasks--retrieval, planning, coding, and verification--each handled by a dedicated LLM agent. Event schemas are represented as executable class definitions, enabling deterministic validation and precise feedback via a verification agent. This programming-inspired approach allows for systematic disambiguation and schema enforcement through iterative refinement. By leveraging collaborative agent workflows, AEC enables LLMs to produce precise, complete, and schema-consistent extractions in zero-shot settings. Experiments across five diverse domains and six LLMs demonstrate that AEC consistently outperforms prior zero-shot baselines, showcasing the power of treating event extraction like code generation. The code and data are released on https://github.com/UESTC-GQJ/Agent-Event-Coder.