Goldsmith: Gold-Loss-Guided Definition Optimization with an Agentic Annotation Harness

AI summary

Overview

Research area: Natural Language Processing, specifically LLM-based data annotation, prompt optimization, and structured information extraction.

Technical level: Intermediate. Readers should be comfortable with prompt optimization concepts (few-shot prompting, instruction search), sequence-labeling and relation-classification interfaces, and standard precision/recall/F1 reporting.

Scope: The paper proposes Goldsmith, a two-stage agentic pipeline that trains a structured annotation definition against a 15-example expert gold set using an executable gold loss, then deploys that definition inside a fixed harness combining retrieval, format repair, judge-based routing, and human review across three structured annotation tasks.

What This Paper Is About

Many annotation projects start before experts have settled on a boundary policy, a complete manual, or enough labels to train a task-specific model. The paper asks whether the few authoritative examples experts can provide early — a 15-example calibration set — can be used as executable supervision to learn what the task means, rather than merely as static demonstrations pasted into a prompt.

Goldsmith's goal is to turn a short concept description plus that small gold set into a reusable, optimized operational definition, and then to make that definition usable for batch annotation through a model-external harness that keeps schema, formatting, retrieval, repair, judging, and human review outside the trainable prompt.

Key Contributions

1. A gold-loss-guided definition optimizer. The operational definition is treated as the trainable textual parameter. An LLM editor converts the highest-loss gold failures into textual-gradient edit directions, but candidate definitions are accepted only when the measured structured gold loss on the same gold examples decreases.

2. An annotation harness that wraps the optimized definition. Schema, format reference, retrieval, format repair, judge routing, and human review all remain outside the trainable definition, so a revised definition becomes a stable workflow rather than a one-off LLM response.

3. Empirical validation across three structured interfaces. OpenNER (typed multilingual named entity recognition with PER/ORG/LOC spans), CrossRE (pair-level relation classification over 17 positive relation labels plus no_relation), and PHEE V2 (fixed-trigger event-argument extraction predicting Subject, Treatment, and Effect spans plus trigger-to-argument relations).

4. Comparisons against prompt optimizers and PLM baselines under matched protocols. Prompt-optimization comparisons pit Goldsmith against LLM rewrite-only, OPRO, APE, and PromptBreeder sharing the same initial description, gold set, candidate budget, gold loss, acceptance rule, and patience rule; task-stage comparisons add mBERT, XLM-R, BERT, RoBERTa, and DeBERTa references under low-budget supervision.

Main Findings

• Goldsmith solves the most gold examples under the same budget. On OpenNER prompt optimization with 15 gold examples, Goldsmith reaches 14/15 passing gold items at loss 1.8030, versus 12/15 at 8.9091 for PromptBreeder, 10/15 at 11.5606 for APE, 9/15 at 14.5909 for OPRO, and 7/15 at 22.0303 for LLM rewrite-only. Goldsmith's best round was 18; OPRO's was 16; PromptBreeder's is reported as 23 generations; APE's best round is reported as N/A.

• The learned definition drives the biggest OpenNER gain. Removing prompt optimization drops micro-F1 from 0.8174 to 0.6661 (change of -0.1513), so the learned definition is not replaceable by the initial prose description.

• Retrieval quality adds stability but matters less than the definition. Replacing structured retrieval with random references reduces F1 to 0.7915, a change of -0.0259.

• Review routing concentrates expert effort effectively on OpenNER. Goldsmith with score-routed review (15 gold + 193 review) reaches typed precision 0.8092, recall 0.8257, micro-F1 0.8174, macro-F1 0.8919, and typed exact match 0.8140. Goldsmith without review reaches 0.7293 / 0.7505 / 0.7398 / 0.8669 / 0.7749; Goldsmith zero-shot reaches 0.7133 / 0.7440 / 0.7283 / 0.8556 / 0.7613; initial-prompt zero-shot with 0 gold examples reaches 0.7040 / 0.7441 / 0.7235 / 0.8546 / 0.7565.

• Low-budget PLM baselines trail on OpenNER. mBERT fine-tuned on 208 examples scores 0.4867 / 0.5595 / 0.5206 / 0.5413 / 0.5630; XLM-R on the same 208 examples scores 0.3281 / 0.3799 / 0.3521 / 0.3735 / 0.4210. The paper frames these as budget-matched baselines, not reproductions of full-data supervised results.

• On CrossRE, accuracy hides the real difference. On the shared 29,171-pair evaluation slice, Goldsmith reaches accuracy 0.8650, positive macro-F1 0.4273, positive recall 0.7273, and positive F1 0.5439. Zero-shot LLM has higher accuracy (0.8720) but far weaker positive macro-F1 (0.1640), positive recall (0.1980), and positive F1 (0.2470). PLM anchors are weaker still: BERT 0.3402 / 0.0883 / 0.4845 / 0.1307; RoBERTa 0.3998 / 0.1108 / 0.5231 / 0.1511; DeBERTa 0.4174 / 0.1088 / 0.4834 / 0.1445. The PLM rows were trained from 200 sentences expanded into 10,593 training pairs.

• PHEE V2 improves monotonically across settings. Initial zero-shot: argument micro-F1 0.6637, relation micro-F1 0.6620, trigger pass rate 0.9988, joint exact match 0.3513. Goldsmith without review: 0.7111, 0.7066, 1.0000, 0.3958. Score-routed review: 0.7471, 0.7427, 0.9988, 0.4500.

• Results are not tied to one backend, but model quality still matters. With the same optimized definition and review protocol, GPT-5.5 gives without-review F1 0.7398, with-review 0.8174, exact match 0.8140; GPT-5.4 gives 0.7222 / 0.8012 / 0.7989; Kimi K2.5 gives 0.7160 / 0.7890 / 0.7820; DSv4p gives 0.7660 / 0.7947 / 0.7863. DSv4p has the strongest without-review F1 but loses that lead after review.

Methodology in Plain English

The approach has two stages.

Stage one — training the definition. Start with a short task description, a gold set of expert-annotated calibration examples, and an unlabeled corpus. The current definition is combined with a fixed output protocol to form an annotation prompt, then run on every gold example. Outputs are parsed and checked against structural constraints, then compared item by item with the expert annotations. The result is a structured gold loss: an executable objective computed from fixed gold annotations and comparison rules, not a subjective score from an LLM judge. Depending on the task, that loss blends normalized missed-boundary and extra-boundary rates (weighted by alpha_miss and alpha_extra), type mismatch on matched boundaries, relation-label disagreement, trigger-to-argument contract checks, plus item-level penalties for parse failure, exact mismatch, unstable output, and trigger failure. The prompt-level objective is the average item loss, optionally rescaled by a reporting-scale parameter.

The optimization loop. At each round, the definition is evaluated on all gold items, the k highest-loss failures are selected and summarized into edit directions, and an LLM editor proposes candidate definitions expressing abstract boundary rules, label priorities, and decision criteria rather than copying examples. Each candidate is rerun on the gold set through the same parser and schema, and accepted only if its measured gold loss decreases. The process is stateful: accepted and rejected candidates, their losses, the highest-loss failures before each edit, and residual failures after accepted edits are all recorded. Training stops when all gold items pass or the patience budget is exhausted.

Stage two — batch annotation. The optimized definition is applied to the unlabeled corpus through the harness. A context builder retrieves eligible references from the gold set and previously reviewed annotations. A parser checks schema fields, label inventory, text preservation, span validity, and relation endpoints. Format-only violations get conversational repair that asks the model to preserve content while correcting structure; only unrepaired failures become full retries. An LLM judge scores parseable predictions on format validity, concept fit, boundary or relation quality, missed-span risk, extra-span risk, and overall quality — used purely as review-priority signals, after checking that low-score bins concentrate true errors. Human review targets low-score or high-risk items; in benchmark runs these corrections were simulated using the available gold annotations, whereas deployment requires domain experts. Reviewed annotations can be promoted to reviewed references for later context construction and a second annotation pass.

Data setup. OpenNER used a deterministic multilingual slice (seed 9) yielding 52 languages, 2,600 tasks, and 3,833 typed spans (1,424 PER, 1,367 LOC, 1,042 ORG); gold selection picked 15 of 2,600 candidate rows with 2,585 pool rows, covering 15 language groups, all four label states, and all four length bins (thresholds 40, 100, and 220 characters). CrossRE produced 1,200 sentence-level tasks and 39,764 directed entity-pair classification items, of which 35,275 are no_relation; six domains (ai, literature, music, news, politics, science) contributed 200 sentences each, and gold selection picked 15 of 39,764 candidates, leaving 39,749 pool items, covering all six domains, 15 relation labels, and all four length bins. All LLM-based optimization, annotation, judging, and review-routing experiments used GPT-5.5 (access month May 2026, temperature 0.0/0.0 for annotation and judge, max tokens 4096, retry bound 20, concurrency 30; prompt optimization at temperature 0.3/0.0 over 30 rounds).

Why This Matters

Impact on research. The paper reframes the scarce-gold-example problem: gold examples become executable supervision for learning what the task is, not just prompt tokens. It also argues for a bounded division of labor in long-context LLM annotation — prompt training improves the operational definition, retrieval provides in-context task grounding, and judge routing plus review concentrates expert effort on likely errors. The paper explicitly does not claim to replace supervised PLM pipelines once large stable datasets exist; it positions itself as a way to reach a stable task definition more efficiently.

Real-world applications:

• Bootstrapping annotation projects in new domains, where experts recognize a phenomenon but lack a stable boundary policy or complete manual, and need a first usable workflow from a handful of examples.
• Pharmacovigilance and biomedical event extraction, illustrated by the PHEE V2 setting where a fixed trigger anchor is supplied and Subject, Treatment, and Effect argument spans must be recovered.
• Multilingual entity recognition, illustrated by the OpenNER slice covering 52 languages with PER, ORG, and LOC labels.
• Rare-relation extraction, illustrated by CrossRE, where accuracy-oriented systems miss rare positive labels that the reviewed pipeline recovers.

Industry relevance. Building labeled datasets is often the dominant cost of deploying NLP. Goldsmith targets the earliest, most under-served phase of that lifecycle, offering a workflow that produces explicit traces of which gold examples shaped the definition and where human review changed the corpus. The backend ablation is also practically relevant: optimized definitions, retrieval, repair, judge routing, and human review reduce how much final annotation quality depends on raw backend strength alone, though GPT-5.5 still gave the best reviewed F1 and exact match in the reported comparison.

Future Directions

• Scaling and cost control. The paper states that at very large scale, token costs, provider limits, and retry overhead can make direct LLM annotation inefficient, and suggests using Goldsmith to build a small high-quality labeled set and stable definition before training a supervised PLM or other task-specific model.

• Tasks beyond natural-language expressible boundaries. Goldsmith is designed for tasks whose boundaries can be expressed through natural-language definitions and examples; the authors flag as less suitable those depending on private expert knowledge absent from the gold examples, highly ambiguous labels, or settings where LLM API calls are prohibited.

• Transferring the optimized definition across backends and tasks. The paper keeps the optimized definition fixed across four backends and reports relatively close reviewed scores, raising the open question of how far a single learned definition transfers without retraining.

• Real rather than simulated review. Benchmark runs corrected selected low-score items using available gold annotations to simulate reviewed feedback; the paper notes that in deployment this step must be performed by domain experts, leaving open how real reviewer behavior and latency affect the reported gains.

Target Audience

Researchers and practitioners working on LLM-assisted data annotation, prompt and instruction optimization, and structured information extraction — particularly those building pipelines where expert labels are scarce and the task definition is still forming. It is also relevant to applied teams evaluating whether to annotate with LLMs directly versus training a task-specific model, and to those interested in human-in-the-loop design where judge scores route review rather than serve as authoritative labels.

Authors’ abstract

Many annotation projects begin before experts have a stable guideline or enough labels to train a task-specific model. We present Goldsmith, an agentic pipeline that turns a small gold set---expert-annotated calibration examples representing the intended task boundaries---into a reusable structured annotation definition. Goldsmith treats this definition as a trainable textual object. Candidate definitions are run on the same gold examples and scored with an executable structured loss, while the output schema, formatting, retrieval, repair, judging, and human review remain in an external harness. A large language model (LLM) editor converts the highest-loss failures into textual-gradient revisions, which are accepted only when the measured loss decreases. In prompt-optimization comparisons, Goldsmith improves over direct rewriting, OPRO, APE, and PromptBreeder under matched evaluation protocols. The resulting definition also improves downstream annotation when combined with retrieval, score-based routing, and human review across typed span, pair-level relation, and fixed-trigger event-argument tasks. These results show that scarce expert supervision can support both task-definition learning and scalable annotation.