PatenTEB: A Comprehensive Benchmark and Model Family for Patent Text Embedding

AI summary

Overview

• Research area: Natural Language Processing, specifically domain-specific text embedding and information retrieval for patent documents.
• Technical level: Intermediate (familiarity with sentence embeddings, contrastive learning, and benchmark evaluation helps).
• Scope: The paper introduces PatenTEB, a 15-task patent text embedding benchmark of 2.06 million examples, and the patembed family of patent-specialized encoder models from 67M to 344M parameters, validated on two external benchmarks.

What This Paper Is About

Patent documents are long, highly technical, and legally structured, yet general embedding benchmarks such as MTEB contain no patent-specific evaluation, and existing patent datasets each cover only a single task. The authors build a broad evaluation suite that captures realistic patent matching situations — including cases where a short fragment (a title, problem statement, or effect) must be matched against a full document — plus a family of models trained on those tasks. The goal is both to measure patent text embedding quality reliably and to produce models that generalize beyond the benchmark they were trained on.

Key Contributions

1. PatenTEB benchmark: 15 tasks spanning retrieval (8 tasks, 5 of them asymmetric), classification (3), paraphrase (2), and clustering (2), totaling 2,057,286 data points (1,556,751 train, 181,215 validation, 319,320 test). The benchmark uses domain-stratified splits across 109 IPC3 classes, domain-specific hard negative mining, and leakage prevention.
2. patembed model family: Six core variants from 67M to 344M parameters, plus three long-context variants (1024, 2048, and 4096 tokens), produced through multi-task training and knowledge distillation from patembed-large.
3. State-of-the-art external results: patembed-base reaches 0.494 V-measure on MTEB BigPatentClustering.v2 versus 0.445 previous best, and patembed-large reaches 0.377 NDCG@100 on DAPFAM.
4. Systematic ablations: Quantification of how supervision diversity, data scale, and architectural choices affect both benchmark scores and external generalization, with all resources released at the project's GitHub page.

Main Findings

• Multi-task training trades benchmark score for generalization: Combining retrieval, classification, and paraphrase supervision marginally reduces internal benchmark scores but improves performance on external benchmarks, which the authors interpret as evidence that benchmarks can reward overfitting.
• Domain-pretrained initialization helps consistently: Initializing from BERT-for-Patents (344M parameters, pretrained via masked language modeling on patent corpora) provides advantages across task families, with the largest gains in semantic matching tasks (retrieval and paraphrase).
• Cross-domain retrieval remains hard: Performance degrades 3–6× when matching patents across disjoint technological domains, indicating that vocabulary mismatch is still a barrier.
• External validation confirms transfer: patembed-base achieves 0.494 V-measure on MTEB BigPatentClustering.v2 (previous best 0.445), and patembed-large achieves 0.377 NDCG@100 on DAPFAM.
• Asymmetric matching is a distinct challenge: PatenTEB includes five fragment-to-document tasks (title→full, problem→full, effect→full, problem→solution, effect→substance) with deterministic fragment removal from targets to prevent trivial lexical matching; four asymmetric tasks use about 16,000–16,297 training examples while three use 150,000.
• Clustering is evaluated without training data: The two clustering tasks (IPC grouping and inventor grouping) are test-only, with 47,230 and 86,834 test families respectively.
• Loss functions are matched to task structure: Eight retrieval tasks use Multiple Negatives Ranking Loss (InfoNCE) with temperature 0.05; the two paraphrase tasks use Online Contrastive Loss with margin 0.5; the two single-document classification tasks use Batch-Hard Soft-Margin Triplet Loss; the pairwise citation-direction task uses cross-entropy on concatenated embeddings. The global objective is a uniform unweighted sum over the 13 training tasks.

Methodology in Plain English

The authors start from patent data obtained from Lens.org and reconstruct "simple patent families," treating a family (the same invention filed across jurisdictions) as the unit of analysis rather than individual filings, following the DAPFAM approach. They filter for patents filed from 1980 onward, require structural completeness such as segmented abstracts, build a citation graph, and assign each family a dominant IPC3 code (the 3-digit International Patent Classification level). Requiring at least 100 families per class leaves 109 technology domains, and they split data 80/10/10 stratified by IPC3 so the same family never appears in two splits.

For each query–target pair they compute a domain relationship from IPC3 code set intersection: IN-domain (identical sets), OUT-domain (disjoint sets), FULL_MIX (one-sided containment), and PART_MIX (partial overlap). Negative examples are always drawn from outside the citation network, and the type of negative is chosen to match the task's domain situation, so models cannot succeed by relying on superficial domain cues. Each query is capped at 100 positives and 10 training triplets, and hard negative mining is applied only to retrieval tasks.

Asymmetric tasks are built by extracting problem, effect, and solution fragments from abstracts using seven predefined pattern rules, then removing the query fragment from the target document so that copying text cannot solve the task. Clustering tasks group families by complete IPC code sets (clusters of 200–1000) or by inventor identifier (clusters of 100–1000).

On the modeling side, the flagship patembed-large starts from BERT-for-Patents and is trained on 13 tasks at once using the four loss functions above, with task-specific prompt prefixes (following the INSTRUCTOR approach) and uniform task weighting. Smaller variants are created by knowledge distillation: the student keeps the teacher's 39,859-token vocabulary and 1024 hidden size but uses a regularly-strided subset of the teacher's 24 layers, projecting to lower dimensions via linear projections initialized with incremental PCA. Students are trained to minimize mean squared error against PCA-projected teacher embeddings, with 2,318,732 unique training texts encoded and cached for distillation. Training uses one epoch, paged AdamW 8-bit, cosine decay with 10% warmup, gradient clipping at 1.0, a per-device batch size of 32 with 4 gradient accumulation steps (effective 128), and learning rates of 1e-5 (patembed-large), 5e-5 (long-context variants from gte-modernbert-base), and 1e-4 (distilled variants).

Why This Matters

Impact on research. PatenTEB fills a documented gap: general benchmarks like MTEB (58 tasks) and BEIR contain no patent-specific evaluation, and existing patent resources each cover one narrow task. By releasing a 15-task suite with documented negative sampling, domain stratification, and leakage controls, the authors give the field a reproducible way to compare patent embedding models, and their ablation results raise a methodological point — that benchmark-optimal models are not necessarily the best generalizers.

Real-world applications:

• Prior art search: Matching examiner queries and fragments against large patent corpora, including the hard case where query and target come from different technology domains.
• Fragment-based patent search: Supporting users who search with a title, a technical problem statement, or a desired effect rather than a full document.
• Technology classification and landscaping: Assigning patents to one of 109 IPC3 domains and grouping patents by technical area or inventor for portfolio analysis.
• Duplicate and citation analysis: Detecting near-duplicate disclosures via the paraphrase tasks and determining citation direction via the NLI-oldnew task.
• Resource-constrained deployment: The 67M to 344M parameter range supports edge, memory-limited, or full-server settings from a single model family.

Industry relevance. The global patent system handles over three million applications annually, and patents often exceed 10,000 words. Embedding-based search and classification are core to patent offices, law firms, IP analytics vendors, and corporate R&D groups, and the authors specifically frame model-size choices around deployment constraints.

Future Directions

• Automated multi-task weighting: The authors use uniform weights because searching a 13-dimensional weight space was computationally expensive, and explicitly suggest automated weight adaptation as future work.
• Closing the cross-domain gap: The 3–6× degradation when matching across disjoint IPC domains remains unexplained and unaddressed, pointing to vocabulary-mismatch or domain-adaptation research.
• Extending task coverage: The benchmark already covers four task families, but the authors note that fragment types (problem, effect, solution) were extracted "opportunistically" from abstracts when available, implying coverage could be broadened.
• Independent evaluation of other models: Since patembed is validated externally on MTEB BigPatentClustering.v2 and DAPFAM, PatenTEB invites zero-shot evaluation of third-party encoders, and the authors position the benchmark as supporting both training and zero-shot evaluation.

Target Audience

Researchers and engineers working on domain-specific text embeddings, patent information retrieval, or scientific and technical NLP; benchmark designers interested in asymmetric retrieval and leakage-controlled evaluation; and practitioners at patent offices, IP analytics companies, or legal-tech firms who need to choose or deploy a patent-specialized encoder under size and latency constraints.

Authors’ abstract

Patent text embeddings enable prior art search, technology landscaping, and patent analysis, yet existing benchmarks inadequately capture patent-specific challenges. We introduce PatenTEB, a comprehensive benchmark comprising 15 tasks across retrieval, classification, paraphrase, and clustering, with 2.06 million examples. PatenTEB employs domain-stratified splits, domain specific hard negative mining, and systematic coverage of asymmetric fragment-to-document matching scenarios absent from general embedding benchmarks. We develop the patembed model family through multi-task training, spanning 67M to 344M parameters with context lengths up to 4096 tokens. External validation shows strong generalization: patembed-base achieves state-of-the-art on MTEB BigPatentClustering.v2 (0.494 V-measure vs. 0.445 previous best), while patembed-large achieves 0.377 NDCG@100 on DAPFAM. Systematic ablations reveal that multi-task training improves external generalization despite minor benchmark costs, and that domain-pretrained initialization provides consistent advantages across task families. All resources will be made available at https://github.com/iliass-y/patenteb. Keywords: patent retrieval, sentence embeddings, multi-task learning, asymmetric retrieval, benchmark evaluation, contrastive learning.