Novel or Not? PatEx-Gym: Agentic Prior-Art Search for Novelty Assessment in Patent Examination

AI summary

Overview

Research area: Information retrieval (cs.IR), specifically agentic search and deep-research agents applied to patent examination and prior-art novelty assessment.

Technical level: Advanced — the paper assumes familiarity with retrieval methods (BM25, dense embeddings), ReAct-style agent loops, supervised fine-tuning, and LLM-as-judge evaluation.

Scope (1 sentence): The paper introduces PatEx-Gym, a full framework combining a 1K-task agentic patent-examination benchmark (PatEx-Bench), an annotation-free trajectory-synthesis pipeline (PASS), and a 4B-scale trained search agent (PatEx-Agent).

Note on the source text: the provided content is truncated partway through Section 5.3. Sections after that point (remaining analysis, ablations, limitations, conclusion) are not available, so any findings there are not reported here.

What This Paper Is About

Real patent examiners must hunt through enormous prior-art collections to decide whether an invention is genuinely new, refining their queries as they learn, reading the documents that look promising, and finally writing an Office Action (OA) that justifies the decision with cited evidence. In plain terms, the paper argues that existing benchmarks cheat this process: they hand the model a tiny predefined set of candidate documents, or even the correct reference document, so the hard part — the searching — is never tested. The goal is to build an environment where an agent must actually search a million-document corpus, decide what to read, and write a defensible novelty assessment, then use that environment both to measure agents and to train a small one.

Key Contributions

1. PatEx-Bench — described as the first benchmark for agentic prior-art search and novelty assessment in patent examination, with 1,000 realistic tasks drawn from USPTO prosecution records, backed by two retrieval corpora (Patent-1M and Paper-52K) and a tool set spanning Boolean keyword search, BM25 sparse retrieval, BGE-M3 dense retrieval, plus document-visit and workspace tools.

2. PASS (Prior-Art Agentic Search Trajectory Synthesis) — an annotation-free framework that generates search trajectories with a teacher model and then scores them without needing examiner-cited gold references. It treats document visits as implicit relevance signals, segments rollouts into search turns anchored on visits, scores intra-turn query behavior, inter-turn search progression, evidence grounding, and outcome quality, then ranks rollouts with a pairwise tournament, producing PatEx-SynData.

3. PatEx-Agent — a 4B-parameter agent trained by supervised fine-tuning on PatEx-SynData, offered as a lightweight baseline that performs competitively against much larger general-purpose agents.

4. A multi-dimensional evaluation protocol — combining LLM-judged OA quality, decision accuracy, statutory-ground F1, and a three-stage "funnel" of prior-art recall (Search-R, Visit-R, Select-R) that traces examiner-cited references through retrieval, inspection, and final selection.

Main Findings

• Scores span a wide range: across evaluated agents, overall Score ranges from 3.79 to 7.34 on a 0–10 scale. Claude-Opus-5-max leads on Score (7.34), Statute F1 (56.9), and all three recall metrics (Search-R 48.7, Visit-R 35.6, Select-R 32.4).

• Prior-art discovery remains hard even for frontier models: the strongest Search-R is below 50% and the strongest Select-R is below 33%, meaning even the best agent fails to surface or retain most examiner-cited references.

• Decision accuracy is a poor proxy for examination quality: Qwen3.6-27B has the highest Decision Accuracy (87.6%) but a Score of 5.96 and Select-R of 19.5, versus 7.34 and 32.4 for Claude-Opus-5-max. Decision Accuracy and Score show only moderate rank agreement (Kendall's τ = 0.550).

• The 4B trained agent beats its own backbone substantially: PatEx-Agent-4B improves Score from 3.79 to 5.41 (+42.7%), Decision Accuracy from 69.7 to 82.9 (+18.9%), Statute F1 from 38.9 to 45.7 (+17.5%), Search-R from 23.2 to 38.7 (+66.8%), Visit-R from 12.1 to 23.2 (+91.7%), and Select-R from 12.0 to 18.2 (+51.7%).

• A small trained model can out-search much larger ones on recall: PatEx-Agent-4B's Search-R of 38.7 exceeds Qwen3.5-397A17B and is described as on par with Qwen3.7-Max (36.1), despite the roughly hundred-fold size gap.

• Searching broadly is not the same as selecting well: DeepSeek-V4-Flash-0731-max reaches a competitive Search-R of 44.1 but only 24.7 Select-R, showing agents struggle to convert retrieved candidates into references they actually cite.

• Judge reliability is strong: agreement among the three LLM judges yields Kendall's W = 0.849.

• Retriever comparison on the patent corpus: dense retrievers consistently beat sparse ones; PatEmbed-Large is best at every cutoff (R@10 22.69, R@20 28.60, R@50 38.19, R@100 45.98), ahead of the best sparse retriever SPLADE-v3 (13.96 / 17.51 / 24.28 / 29.25). Under the same budget, Claude-Opus-5-max reaches R@10 19.33 and R@100 45.71.

• Trained agent improves retrieval-style metrics too: PatEx-Agent-4B gains +8.0% / +26.1% / +34.7% / +51.1% over its base at R@10/20/50/100 (16.10 / 24.20 / 31.80 / 36.36).

Methodology in Plain English

The team started from real USPTO prosecution records. For 1,000 applications they kept the application as originally filed and the first examiner-written Office Action, which serves as the authoritative "correct answer" rather than a synthetic label. Around those tasks they assembled two searchable corpora — Patent-1M (1,011,901 unique patent documents) and Paper-52K (52,105 scientific papers) — and verified that every examiner-cited reference is actually retrievable from them.

Each task runs inside an isolated sandbox harness (built on Codex) where the agent can only see titles, abstracts, and bibliographic metadata from a search; to see claims and fuller text it must explicitly "visit" a document. This two-step search-then-visit design creates a natural signal: visiting a document is an implicit judgment that it looks relevant, much like a click in web search. The runtime is split into Preparation, Execution, and Evaluation stages so the agent never sees the grading rubric.

For training data, they took 10,000 seed patents from USPTO, disjoint from the benchmark and balanced across IPC sections, and had a teacher model (GLM-5.3) run the same harness to produce 10,000 trajectories. PASS then segmented each rollout into turns anchored on visits, scored four things — how efficiently searches turned into inspections (NDCG over visited documents), whether the next query genuinely evolved rather than being rephrased, whether cited evidence was actually supported by passages the agent had read, and whether each claim was resolved with justification — and filtered out broken rollouts. 6,321 survived the feasibility gate; the top ~3K by tournament ranking became PatEx-SynData.

Finally, they fine-tuned Qwen3.5-4B on those trajectories for 3 epochs on 8 NVIDIA H20 GPUs at a learning rate of 1×10⁻⁵, masking out system, user, and tool-result tokens so the loss applies only to the model's own reasoning and tool calls. Evaluation used the Codex harness (v0.151.0) with temperature 1.0, a maximum output length of 32,768 tokens, and at most 50 steps, with three LLM judges (Claude-Opus-4.7, Gemini-3.5-Flash, GPT-5.5) scoring OA quality.

Why This Matters

Impact on research. The paper reframes patent examination as a long-horizon agentic search problem rather than a static classification or ranking task, and it supplies an annotation-free way to manufacture high-quality training data for such agents — notable because examiner-cited gold references are expensive and unavailable at scale in most domains. The finding that decision accuracy misranks systems (Kendall's τ = 0.550 against the quality Score) is a caution for anyone evaluating agents with a single accuracy number.

Real-world applications:

• Prior-art search assistance for patent examiners, where an agent drafts a candidate reference set and an evidence-grounded first-pass Office Action.
• Pre-filing novelty checks for inventors and IP counsel deciding whether to file.
• Patent-office backlog reduction, since a 4B model is small enough for local deployment inside an institution.
• Cross-domain technical literature triage, as the tasks span all eight IPC sections (A through H) and require handling both patents and non-patent literature.

Industry relevance. The compact 4B scale is the headline for practitioners: it is described as competitive with models around 30B parameters and matching Qwen3.7-Max on Search-R, which matters for cost, latency, and data-confidentiality constraints in legal and IP workflows. The auditable harness — full trajectory logging, search-tool logs, and a strict separation between execution and evaluation — also maps onto the traceability requirements of real examination.

Future Directions

• Closing the select gap. Select-R tops out at 32.4 and trails Search-R everywhere, so the open question is how to make agents commit to the right references once they have found them, rather than just retrieve broadly.
• Legal reasoning beyond decision labels. Statute F1 stays near the mid-50s even for the best model, leaving substantial room in statutory-ground identification and claim-level justification.
• Scaling trajectory synthesis and training. Whether PASS generalizes to larger backbones, other teacher models, or reinforcement learning on top of the SFT baseline is not settled by the reported experiments.
• Benchmark and domain extension. The current corpus covers patents and papers but is sourced from USPTO and Google Patent/Scholar, so coverage of other jurisdictions, languages, and prosecution practices remains untested in the provided content.

Target Audience

Researchers and engineers working on agentic search, deep-research agents, and tool-use training; information-retrieval scientists interested in realistic long-horizon retrieval benchmarks; and applied teams in IP analytics, legal tech, and patent offices who want a concrete, reproducible testbed plus a deployable small-model baseline. Readers seeking a gentle introduction to LLM agents will find it dense, as it assumes comfort with retrieval metrics and SFT recipes.

Authors’ abstract

Patent examination is a challenging real-world task that requires agents to discover sparse prior art, iteratively refine queries, inspect technical evidence, and produce novelty assessments documented in Office Actions (OAs) across heterogeneous scientific and technical domains. However, existing benchmarks largely focus on static and limited candidate sets, leaving the full search--reasoning--assessment process underexplored. We present PatEx-Gym, the first framework for developing and evaluating agents for patent examination, which integrates three components. First, PatEx-Bench provides 1K realistic examination tasks, two large-scale retrieval corpora containing over 1M patents and 52K scientific papers, and a harness runtime supporting iterative search, document visits, reasoning, and OA generation. Second, we introduce PASS, an annotation-free Prior-Art Agentic Search Trajectory Synthesis framework. Specifically, it exploits document visits as implicit relevance signals and evidence-acquisition anchors to model intra-turn query behavior, inter-turn search progression, and trajectory-level evidence and outcome quality, yielding PatEx-SynData. Third, we train PatEx-Agent at the 4B scale via supervised fine-tuning on the curated trajectories, establishing a lightweight baseline. Extensive experiments show that final decision accuracy alone does not fully reflect examination quality, while prior-art discovery remains difficult even for strong frontier models. Our PatEx-Agent achieves competitive performance against substantially larger general-purpose agents. These results show that training on PatEx-SynData enables a compact 4B model to serve as a competitive baseline for prior-art search and patent examination. We view patent examination as a demanding testbed for agentic search and deep research agents.