Benchmarking Patent Drafting from Inventor-Style Disclosures

AI summary

Overview

• Research area: Natural language processing, specifically long-form legal text generation with large language models (LLMs), applied to patent drafting.
• Technical level: Intermediate. The reader needs some familiarity with LLM benchmarks, retrieval-free generation evaluation metrics (BLEU, ROUGE, BERTScore), and basic patent document structure (claims vs. specification).
• Scope: The paper introduces Dis2Pat, a 9.4K-example disclosure-to-patent dataset built from granted patents, and Patent-MAF, a multi-agent drafting framework, and benchmarks text-only, multimodal, open-source, and closed-source LLMs on generating complete patent applications from inventor-style disclosures.

What This Paper Is About

Real patenting starts with an informal, non-legal invention disclosure written by an inventor and supported by drawings; a patent attorney then turns it into a legally structured application with claims and a specification. Existing benchmarks instead feed models later-stage, already-legalistic inputs such as full specifications (for claim generation) or claims (for specification drafting), which sidesteps the hardest part of the job. This paper builds a dataset for the realistic disclosure-to-patent setting and a locally deployable multi-agent baseline for it, then measures how current LLMs perform.

Key Contributions

1. Dis2Pat dataset. Described as the first dataset for disclosure-to-patent generation, pairing inventor-style, de-legalized disclosures (with associated figures) against complete patent applications. Built from patents filtered from the PatentDesc publication numbers, with pseudo-disclosures written by GPT-5-mini: 9,433 patents sampled, split into 8,490 training and 943 test samples, with patent-family metadata checked so no family appears in both splits.
2. Patent-MAF baseline. A multi-agent framework with a manager agent that structures the disclosure and routes subtasks, specialized drafter agents for claims and specification, and a polisher agent for cross-section consistency, legal style, and coherence. All agents are constrained to locally deployable open-source models because disclosures are confidential.
3. Systematic benchmark. Evaluation of text-only, text+image, open-source, and closed-source LLMs on both specification and claim generation, using text overlap, semantic similarity, LLM-as-a-judge content quality, and pairwise expert human evaluation on 60 randomly sampled test cases.
4. Ablation analysis. Component-level studies removing the manager, the visual input, the claim fine-tuning, and the polisher to attribute performance to specific parts of the framework.

Main Findings

• Specifications benefit from figures, claims do not. Adding images improved overall content quality for Qwen3-VL-32B (81.0 text-only to 81.3 text+image) and Qwen3-VL-72B (81.1 to 81.8), and raised GPT-5's BERT-for-Patent similarity from 95.7 to 96.4. For claims, the trend reverses: Qwen3-VL-32B went from 78.9 to 78.6, Qwen3-VL-72B from 80.7 to 79.8, and GPT-5 from 91.1 to 90.7.
• Patent-MAF leads among open-source systems on specifications. It reached an overall content quality score of 85.4, against 81.8 for Qwen3-VL-72B and 85.1 for GPT-5 (text+image). Its BERTScore was 82.3 and BERT-for-Patent 96.1, versus GPT-5's 81.8 and 96.4.
• Patent-MAF narrows but does not close the gap on claims. Its overall claim quality was 86.7, the best among evaluated open-source systems, while GPT-5 reached 91.1. Patent-MAF had higher semantic similarity on claims: BERTScore 88.3 vs. GPT-5's 83.8, and BERT-for-Patent 96.4 vs. 95.9.
• Fine-tuning is critical for claims, the polisher matters most. For claims, removing fine-tuning dropped the overall score by 5.1 points (86.7 to 81.6), and removing the polisher caused the largest degradation of 6.6 points (86.7 to 80.1). Removing the manager lowered claims from 86.7 to 84.2.
• Ablations on specifications show smaller, distributed losses. Removing the manager reduced specification overall quality from 85.4 to 83.4, with completeness falling from 85.6 to 85.0 and clarity from 88.5 to 86.9. Removing vision dropped overall to 83.4 and BERT-for-Patent from 96.1 to 93.6. Removing the polisher mainly hurt style, from 82.0 to 80.9.
• Expert pairwise evaluation favors Patent-MAF over open-source baselines. Against Qwen3-32B, Qwen3-VL-72B, and LLaMA-3.2-90B-Vision, Patent-MAF was preferred in the majority of cases across completeness, clarity, legality, and consistency. Against GPT-5, GPT-5 was preferred more often on completeness, clarity, and legality, the two were evenly matched on consistency, and a substantial proportion of ties was reported.
• The pseudo-disclosures are faithful and not lexical copies. Human evaluation of 100 randomly sampled examples on a ten-point scale gave Hallucination 9.8, Details 9.8, Contradiction 10.0, and De-legalization 9.7. Average ROUGE-L F1 against source patents was 0.08 (maximum 0.11) and average 4-gram overlap 0.006 (maximum 0.013); Cohen's d between two randomly split halves of the validated examples was approximately 0.1.
• Disclosures are far shorter and less term-dense than patents. Original patents averaged 11,207.4 tokens (1,594.1 in claims, 9,476.5 in the specification, 128.1 in the abstract, 8.7 in the title) with 6.7 figures and 19.7 claims, against 1,196.2 tokens for disclosures; term density fell from 0.05 to 0.001.
• A Llama-based configuration also improves over its base. With Llama-3.2-90B-Vision, Patent-MAF reached an overall specification score of 70.8, versus 67.1 for the same model with text+image and 66.8 text-only.

Methodology in Plain English

Real invention disclosures cannot be released publicly because of confidentiality and legal constraints, so the authors built stand-ins. They started from the publication numbers in the PatentDesc corpus, kept only granted patents, pulled titles, abstracts, claims, specifications, and figure URLs from the Google Patents Public Dataset, and required each patent to have all components plus no more than ten figures. For each surviving patent, GPT-5-mini rewrote the patent into plain, non-legal English organized into seven invention-centric sections (title, why the invention is needed, core idea, how it works, what makes it different, benefits, optional variants), producing a JSON object while preserving technical content and dropping claim language and legal structure.

On top of this data, Patent-MAF splits drafting into roles. A manager agent converts the raw disclosure into two structured artifacts, one oriented to legally essential claim features and one oriented to background, embodiments, and figure references. A claim drafter writes the claim set (at least one independent claim plus dependent claims) from the claim material, and a specification drafter writes the specification from the specification material with vision enabled to resolve spatial and structural details. A polisher then unifies terminology, checks that claims and specification agree, and normalizes style without adding new technical content. The manager and polisher run on Qwen3-32B with prompting, the specification drafter on Qwen3-VL-32B, and the claim drafter on Qwen3-32B with LoRA fine-tuning (rank 16, alpha 8, dropout 0.05, batch size 2, learning rate 5×10⁻⁵, weight decay 0.1, up to three epochs). Inference used temperature 0.3 and up to 16,384 generated tokens; all experiments ran on NVIDIA A100 GPUs for approximately 700 GPU hours. Evaluation combines BLEU, ROUGE-1, ROUGE-L, BERTScore, and BERT-for-Patent similarity with LLM-as-a-judge scores on a 0–100 scale and a pairwise expert protocol on 60 random test cases judged for completeness, clarity, legality, and claim–specification consistency.

Why This Matters

• Research impact: The paper argues that prior benchmarks bypass the core difficulty of patent drafting by assuming later-stage inputs. Dis2Pat reframes the task as conditional drafting from an informal disclosure plus figures, and Patent-MAF shows that role decomposition plus a refinement stage can let locally deployable open-source models approach closed-source performance, which the authors frame as narrowing the gap between small open models and state-of-the-art closed systems.
• Real-world applications:
  • Assisting patent attorneys who must convert inventor notes into a first draft of claims and specification.
  • Supporting corporate IP teams that need drafts prepared without sending confidential disclosures to external APIs.
  • Producing figure-grounded descriptions of mechanical or system-oriented inventions where component relationships must be described consistently.
  • Standardizing claim–specification consistency and terminology before formal filing.
• Industry relevance: Privacy is a hard constraint in patent work; the paper treats local deployment on open-source models as a non-negotiable design requirement, and the license choices (Llama 3 Community License, Apache 2.0, CC-BY-SA-4.0 for the dataset) reflect that intended use. The 9.4K-scale dataset and the released framework give practitioners a reproducible benchmark.

Future Directions

• Move beyond pseudo-disclosures. Because real disclosures cannot be shared, Dis2Pat relies on LLM-rewritten patents that may not reproduce the diversity, incompleteness, or ambiguity of genuine inventor submissions; the authors suggest partnering with industry or law firms on anonymized or partially synthetic disclosures.
• Multilingual and jurisdiction-specific drafting. The work is evaluated primarily on English-language patents; adapting to standards such as EPO or CNIPA is described as a natural extension.
• Upstream information elicitation. Dis2Pat deliberately assumes the disclosure contains enough technical information to draft from, so recovering missing information from incomplete inventor notes is left as a separate problem.
• Better claim modeling. Claims remain the weaker area relative to closed-source GPT-5, and the ablations point to fine-tuning as the largest factor, leaving room to improve how legal scope is captured.

Target Audience

Researchers in patent NLP and legal text generation, benchmark builders interested in long-form constrained generation, and applied machine learning engineers building privacy-preserving drafting tools. Patent professionals and IP operations teams will also find the task framing and the expert pairwise evaluation relevant, though they will need comfort with LLM evaluation metrics. Readers primarily seeking practical patent prosecution guidance rather than model design would find the paper's contribution to be the dataset and benchmark rather than legal advice.

Authors’ abstract

While recent large language models (LLMs) have achieved promising results on individual patent drafting tasks, they fundamentally fail to investigate the core challenge of real-world patent drafting: generating a complete and legally coherent patent application directly from early-stage invention materials. Prior work predominantly assumes later-stage, highly structured, or already legalistic inputs. However, real patenting workflows begin with informal, de-legalized disclosures authored by inventors. To bridge the gap, we introduce Dis2Pat, a disclosure-to-patent dataset that reflects realistic patenting workflows by requiring the generation of complete patent applications directly from inventor-style, de-legalized disclosures. Given the inherent difficulty of long-form, legally constrained patent drafting and the strong privacy requirements, we further propose a strong baseline named Patent-MAF. It is a multi-agent framework for locally deployable patent drafting. Benchmark results reveal that current LLMs exhibit limitations in patent drafting, while Patent-MAF provides a strong baseline that consistently outperforms evaluated open-source models and remains competitive with large closed-source models.